Walmart recently introduced a new way to shop: via text. Last month, the retail giant launched its “Text to Shop” experience which allows mobile consumers across both iOS and Android devices to text Walmart the items they want to purchase from either their local stores or Walmart.com, or easily reorder items for pickup, delivery, or shipping. However, the chat experience as it stands today does not come across as fully baked, our tests found. The chatbot said confusing things and the user interface at times was difficult to navigate, despite aiming to be a simpler, text-based shopping experience.

Conversational commerce, or shopping via text, is an area that’s been seeing increased investment over the past couple of years with numerous startups entering the market. Walmart, too, has connections with this space, as its former head of U.S. e-commerce Marc Lore backed a conversational commerce startup, Wizard. And Walmart itself acquired assets from a design tool called Botmock which had built technology that allowed companies to design, prototype, test and deploy conversational commerce applications.

The new “Text to Shop” feature, meanwhile, was built in-house using internal IP in partnership with Walmart’s Global Tech team and was tested with customers ahead of its launch. The beta version was available for around a year’s time before December’s public debut, but had only been accessible on an invite-only basis.

At launch, the “Text to Shop” feature ow customers to shop Walmart’s entire assortment via chat, whether that’s your weekly grocery order from a nearby store or an e-commerce order you want shipped to your home.

Customers more recently began receiving emails to alert them to the fact that “Text to Shop” was newly available, which prompted our tests. The feature was also highlighted in Apple’s announcement of its new Apple Business Connect dashboard, which allows businesses to manage and update their information on Apple Maps. Here, Walmart partnered with Apple so customers who visit the Walmart business listing card on Apple Maps could tap on a “message us” button to get started with a “Text to Shop” session.

In theory, chat-based shopping is supposed to simplify online shopping by bringing it into a more familiar texting interface. But in practice, Walmart’s chatbot made some missteps when we tried it, making for a more cumbersome experience compared with a traditional order placed through the Walmart website or app.

The initial steps in getting started with “Text to Shop” was easy, however, as you just sign into your Walmart account and agree to its terms. The bot then sends you a helpful introduction and some tips on how the system works. It tells you, for example, that you can just type in the names of items you want, like “Great Value Oatmeal” and explains how to set your local store, among other things.

But already, it was clear the system would have a few quirks, as it informed you that items you typed in single quotes would serve as commands.

For example, typing ‘Reorder’ with quotes would allow you to buy things again. This seemed like an odd requirement, given that the word “reorder” wouldn’t likely match a product a customer wanted to buy through text-based shopping, or at least, it should be assumed that a text with that word is a command. Plus, it puts an unnecessary burden on the end-user at a time when they’ve just started trying to learn a new system.

In my tests, I ordered a few basic items, like milk, eggs, bread, and water. The system didn’t immediately warn me that I had lingering items in my cart from an online order I had abandoned weeks ago.

The system also doesn’t prompt you upon your first text to choose whether you want to start an order for delivery, pickup or shipping. Instead, it returns a selection of options that match your request. But the way it did so was confusing.

In my test, I typed “2% milk,” and it responded twice with possible options. “OK! 2 % milk, 3 choices coming down” the bot said, followed by a link that takes you to a list. But then it replied again, “These are the closest options I found for 2 % milk,” and offered another list.

After picking an item, you’re instructed to “select one of these options next” which offered choices like “search for pickup,” “search for shipping,” or “search for delivery.”

It would seem that asking the customer how they were shopping should have been the first step, especially if product availability varies by order type. In this test, I chose delivery.

That’s when the bot texted me that I now had 6 things in my cart — a surprise, since I hadn’t remembered my earlier abandoned choices.

That one was on me, though, I admitted. I tapped “View cart” to delete the weeks-old selections. The bot didn’t immediately display the cart. Instead, it responds with your item count and total. You then have to tap a link that follows to view the cart, which pops up in another screen. I expected this to operate like a web version of a Walmart checkout page, the screen was missing obvious tools to delete items or change the quantities, which you would normally find on an e-commerce shopping cart page.

In fact, the interface instructs you to “tap to view, select or remove,” but presents radio buttons to tap and then a “Send” button at the bottom to…well, I don’t know.

How would it know if I was instructing it to show me the item or remove it?, I wondered. And why would I even need to view the item elsewhere, when its full name, photo, quantity, and price are shown here?

Still, I tapped “Send” to remove the old items (which were not the newly-added milk), only to be returned to the main chat screen where I was informed, inaccurately, “Ok, all milk taken out!” Now my cart had 5 things, it said. It had only removed one of my selections.

I tried again, tapping the other 5 items to be removed, and again, the bot responded, “Ok, all milk taken out!”

In reality, the milk was the only item that remained. The bot was wrong.

Image Credits: Screenshot of Walmart Text to Shop

Now, with only the milk remaining (despite the texts to the contrary) the bot asked me what I wanted to do next — maybe view cart or checkout?

This is a very dumb bot, I thought. Does anyone get just milk delivered and nothing else?

I wasn’t ready for that so I tried another query. “Eggs,” I typed. The bot only returned three choices: all Walmart brand large white eggs but in different sizes. Odd, since I know Walmart, like most retailers, has a much larger egg selection.

Image Credits: Screenshot of Walmart Text to Shop

“Organic eggs,” I texted, hoping for better egg options. This worked, and I added Pete and Gerry’s eggs to the cart without hassle. The bot now updated me on my total. My cart had two items, milk and eggs, and my subtotal was $10.40. (I’m not sure it’s a good idea to tell the customer the running price if they don’t ask! Yikes!)

Then I tried something to intentionally confuse the system. Knowing that end-users often don’t play by the script, I scrolled back up to tap “Pickup” instead of “Delivery.” This is the kind of thing a customer might do if they think choosing pickup would offer them a different selection of eggs. But the bot didn’t make that logical leap, asking “sure, what product would you like to search for pickup?”

“Never mind,” I texted. “No problem. Talk to you later,” the bot replied.

I then went to add the next item on my list. “La Crox,” I texted.

“These are the closest options I found for la croix organic eggs for pickup,” the bot answered. Uh? What?

I had clearly confused this bot quite a lot, it seems.

It then texts me a list to view and then asks me to select the delivery method, and then texted the list again. It only returned three La Croix options to choose from. A search in the Walmart app returned 10, however.

This system isn’t useful at all, apparently, unless you enter a very specific choice.

That realization made me dread my next item: bread. I didn’t have a brand in mind, as I usually browse and look for sales on favorite types and brands. I ask for “multigrain bread” and I only have three options shown to me alongside another message telling me I can “search for pickup” or “shipping.” I understand now these delivery choices are apparently texted every time you request an item, rather than the system building you a cart for a particular delivery method. (I didn’t tap these options because I was going to have the items delivered.)

“Checkout,” I then texted — without the single quotes, just like an everyday user would likely do, having forgotten the earlier command syntax that involved using quotes.

And, it worked. You could then select to view the cart or checkout, and through a separate screen, you could book a delivery time.

There were other odd user interface choices here, as well though.

For instance, this screen presented you with an option to change the “quantity” of the selected items, when earlier that wasn’t possible. I tapped the “Change quantity” button (as I’m now rethinking those expensive eggs!). This sent an automated command, to which the system replied “Can you please rephrase that?”

Image Credits: Screenshot of Walmart Text to Shop

I wonder if some of the issues with the bot are because it didn’t know my local store, somehow, even though this is already configured under my Walmart account — which I had authenticated with.

“Set store,” I typed, even using the single quote format.

The bot told me to choose my location and texted me two options. Both were my home street address, without the house number. Both were identical options.

At this point, it feels like the process of ordering a few basic things has become an ordeal and has taken a lot longer than the traditional method of searching in Walmart’s app and adding things to the cart. If conversational commerce like this is the future, I’d say this is very much still a work in progress.

I abandoned the cart and didn’t complete the order.

When I asked Walmart about some of the issues I encountered, wondering if this was all still a beta test, a spokesperson said the company would “continue to refine and optimize Text to Shop to ensure we’re providing the best experience possible for our customers.”

Let’s hope!