A look at one of Walmart's new curated nursery rooms. This one is gender-neutral.

Walmart is making a bigger push in the baby business just weeks after the last Babies R Us store went dark.

Starting Thursday, shoppers will see a revamped landing page on Walmart's website for baby items — everything from cribs and car seats to baby food and diapers. This is just the next step in Walmart's grander web redesign, which started with the retailer's fashion and home categories earlier this year.

The move toward baby follows the fall of Toys R Us, which also owned the Babies R Us chain that once operated hundreds of stores across the U.S. Buy Buy Baby-owner Bed Bath & Beyond has also been stumbling of late, with more and more parents ditching brick-and-mortar stores to ring up their purchases for kids online.

The market for baby-care products globally isn't expected to stop growing anytime soon, either. It's expected to surpass $13 billion by 2021, up from an estimated $11.4 billion this year, according to data from Statista.

Walmart said that over the past year it's added 30,000 additional items to its baby assortment online. Searches for baby-related items on Walmart.com have jumped roughly 40 percent over the same time frame, Walmart spokeswoman Danit Marquardt told CNBC.

"Since summer and early fall have historically been when most babies are born, now is the perfect time to introduce these new shopping features for busy families," Lauren Uppington, vice president of Walmart's baby division online in the U.S., said in a statement.

Now, parents will be able to browse different nursery looks on Walmart.com and buy items directly from those rooms, with themes like Boho Chic, Wanderlust and Mid-Century Modern. Walmart is also featuring a gender-neutral room. It's a similar strategy that's already been applied to Walmart's furniture category online, where the retailer is trying to make it easier for shoppers who would prefer pieces be paired together for them.

Amazon has meanwhile been making a bigger push in the baby business, though not as pronounced as Walmart's plans announced Thursday. One of Amazon's top-selling in-house brands has been Mama Bear, which sells diapers and organic baby food pouches.

Target is another top competitor in the space and hopes to use its own brands — like Cat & Jack for kids clothing — to its advantage, potentially grabbing former Babies R Us shoppers' dollars. Target and Walmart both have been revamping the baby sections within their physical stores to be more open and have products on display for parents to test on the spot.