Facebook debuted chatbots for Messenger this week to much fanfare, promising consumers a new way to interact with their favorite businesses over the company’s popular messaging service. The idea is that Messenger users could simply “chat” with these automated software programs to do things like read the latest news from CNN, get a weather report or even go shopping. In practice, however, the bots — at least at launch — failed to live up to the hype.

Trying to use the bots for simple tasks — like finding out if it would rain or buying a black shirt — was frustrating, disappointing and ultimately far less efficient than simply visiting the company’s website itself.

That’s not to downplay the potential for bots in the long term, or the possibilities provided by bringing Facebook’s large base of businesses to Messenger where they could better serve customers who increasingly use mobile messaging apps while on their smartphones.

However, as the chatbots for Messenger platform launches, the bot experience leaves much to be desired.

To get a good feel for Facebook’s chatbots, we demoed the three “Featured” bots that Facebook is promoting via its Messenger platform website: CNN, shopping app Spring and weather app Poncho.

Each bot had a “Try it” link provided, which opens directly on Messenger a chat interface with the business.

For starters, clicking this link — the new short URLs aimed at Facebook Page owners (in the m.me format) — merely launched the chat window; there was no greeting text from the business in question. In other words, though you know that there’s a bot to interact with, you don’t know how to begin.

Does it require a trigger word or phrase? Can you just say “hi?”

Facebook will reportedly soon address this problem when it finalizes the rollout of “Messenger Greetings,” which will allow businesses to pass along a note to customers when they kick off a chat session. These greetings could instruct users how to get started using the bot. It’s unclear why Facebook wouldn’t have this enabled for the chatbots at launch, though. After all, this whole concept is new to so many of today’s mobile messaging users who are not old enough to remember chatting with IM bots like SmarterChild from back in their PC days.

As it turns out, not all the bots operate the same way.

For example, simply saying “hi” to CNN and Poncho generated an automated response, but Spring’s bot ignored me. (I had to Google to find out that the way to talk to Spring was to say “go shopping” to it.)

Unfortunately, that was the least of my problems with using the bots.

Spring

Here’s a simple problem a chatbot could help me solve, I thought: I’m in the market for a new shirt. I like the color black and I don’t want to spend a lot. Couldn’t Spring’s new shopping bot help point me to some nice items?

I didn’t expect it to succeed in helping me narrow top shirts by details like sleeve length or fabric type at this early point, but I at least believed it could pull up a few possibilities.

I was wrong.

After getting the shopping session started by the keyword trigger, Spring’s bot interface is easy enough to use. It asks you a series of questions to narrow down what you’re looking for — Men’s or Women’s items?, “Clothing, Shoes, or Accessories?”, etc. — and you click on your response.

After I narrowed it down to tops under $75, Spring returned five items it thought I would like.

Why would I like these particular items? I don’t know. Spring knows nothing about my shopping history, what style of shirt I’m looking for, the color, the occasion (work or casual) or anything else.

It returned three white shirts, one t-shirt, and a sweater.

Uh, OK.

At this point, I’d like to ask the bot for more options. But how do you do that?

I tried “chatting” with the bot, assuming at this point, anything I said would at least trigger a new “help” message that explains how to continue using the service.

“I don’t like these” I told it, and was met with silence.

“hi?” I spoke into the void. Nothing happened.

“can you show me more?”

Finally, the bot answered, informing me that I could return to the options above and click them again to start the process over. Hopeful that the five items it returned were a randomized group and I’d see five new shirts if I repeated things, I did just that.

The bot returned the same five shirts.

Well, maybe I need to be more specific, I thought.

“Please show some black shirts,” I said.

The bot gave me a link to its website.

I know what you’re thinking. Well maybe Spring didn’t have any black shirts under $75? But it did! The link pointed to Women’s black shirts on the Spring site, and there are more than a few from which to choose.

The thing is, it just would have been easier to visit Spring’s website directly, rather than dancing through this chatty interface.

Poncho

Poncho’s snarky weather bot was even worse. Though I got it talking with a simple “hi,” its weather reports failed to deliver even the most basic information, like when it would rain.

After getting my location and asking if it could message me daily weather reports (NOPE!, I said), it told me I could still chat with it for other information, specifically:

“Feel free to ask me if it’s going to rain, if you need sunglasses or a jacket, or just say hello from time to time!”

So I asked: “is it going to rain?”

Poncho answered: “Wet. Warm. Yuck.”

I’m sorry, but what….?

Is that supposed to be a real answer?

I tried to get Poncho to clarify — did “wet” mean rain was coming, then?

“Sorry, you’re going to have to say that again in Cat. Meow?,” it said.

You’ve got to be kidding me.

“When will it rain?,” I asked. It didn’t understand.

“What time will it rain?,” I tried again. No go.

“What’s the percentage chance of rain?,” I attempted awkwardly.

“Wet. Warm. Yuck,” the bot repeated.

I threw my phone across the room.

No thanks.

CNN

Of the three Featured bots, CNN’s did better, but still had some issues. It responded to “hi” as well, thankfully, as I still have no clue what you’re supposed to say to these things.

It then directs you to pick the news you want to hear: Top Stories, Stories for you or Ask CNN.

Top Stories is a curated selection of the top news, and can offer you a summary or a link to the CNN site to read further. The summary feature is a bit reminiscent of Quartz’s standalone news messaging app, and is a good way to get briefed on the most important aspects to the day’s news.

Ask CNN lets you specify a topic you’d like to read about, while Stories for you returns suggestions based on what you’ve been reading. (Presumably, this will improve over time.)

While CNN’s bot performed these tasks fairly well, it still failed in other areas. For example, because it’s been programmed to look for topics by keyword, apparently, it didn’t seem to grasp the nuance of a query asking for “U.S. news.”

It’s plausible that someone might think to ask for U.S. news via this bot, in order to get a filtered selection of the day’s top news. However, CNN’s bot only seemed to look for the word “U.S.” in the headline when returning its suggestions. Instead of “top” stories, the bot returned exactly two articles:

CNN’s “U.S. mobster’s heirs to Cuba: You owe us — so pay up” and a CNN Money video entitled “Ken Burns: Trump divides us. Sanders does too.”

Not only are these not what one would consider the top U.S. stories, the second was returned because it had the word “us” in the headline.

Maybe I didn’t word it correctly?

Oddly enough, asking just for “us news?” returned a different set of (now three) stories with the word “us” in the headline, including an op-ed called “What Easter teaches us about heaven.”

Similarly, requests for the “latest news” stumbled — again, only looking for the keyword in the headline.

This was not helpful at all.

Do bots have a future?

While these tests were brief, it goes to show that Messenger’s chatbots are still a long way from fulfilling their potential to be a useful way to interact with businesses and brands through chat.

These bots clearly need to be able to interact with their human customers more conversationally, and have a better understanding of what people are actually saying to them. They need to have more functionality — like the ability to pull up specific items when shopping — and they need to be better programmed from the start.

Some bots are better than others, of course.

The Guardian gave different chatbots a dry run and had similarly mixed results. Simple bots like The WSJ and CNN’s news bots did better, while Operator’s shopping bot — which claims to respond to emoji queries — didn’t really work. Meanwhile, Sequel Stories’ “choose your own adventure”-style story bot was more fun, their report found.

The question for Facebook’s business users is whether or not it’s worth the time and effort to build one of these bots, or if the chatbot phase will end up being a fad.

For now, it’s still entirely unproven that customers will want to chat with bots — especially when they work so poorly. The experience, if not improved in short order, could become the mobile equivalent of punching your way through a business’s automated customer service hotline.

At the end of the day, people may want to chat with a person, not a machine, when trying to get customer service.

They may think that using the company’s website or native app is easier and faster than trying to get the bot to respond properly, so they simply ignore the bots.

And if people try bots and fail to find them useful right out of the gate, there’s a good chance businesses won’t get a second chance to win back those same customers in the future.

“SIGH” I wrote to Spring’s chatbot, ending my experience today.

The bot said it got my note and would be back with me shortly. (Spring says its bots are partly human-powered. Maybe my message was being sent to staff?, I wondered.)

Over an hour later, Spring asked me if I wanted to keep shopping.

Not via bot, I don’t think.