Don't Ask ChatGPT for Restaurant Advice!
Don't Ask ChatGPT for Restaurant Advice!
Fake dishes at real restaurants, real advice about fake restaurants, and a chat bot that’s better at lying than helping people eat well
As Apple prepares to integrate ChatGPT into its iPhone and Macs, Ryan Sutton takes a look at the bot’s flawed and sometimes fictional restaurant recommendations. Consider subscribing to support our journalism.
An AI chat bot walks into Peter Luger…
A few days ago, I asked ChatGTP whether I should dine at Peter Luger.
Ugh, I know.
I don’t normally seek out advice from bots. I get my dining suggestions from good journalism, produced by humans — a craft that feels all the more endangered these days.
Alas, the times are changing. Apple is marketing its new iPhone as a way to harness the power of artificial intelligence. Later this year, the tech giant will integrate ChatGPT-4o into its operating systems. It’s a big deal.
The chat bot reportedly has over 250 million active weekly users. And major publishers, including News Corp, Conde Nast, and Vox Media, have signed multi-million dollar agreements with OpenAI, the company behind ChatGPT.
Other papers, like the New York Times, have filed copyright infringement lawsuits.
Apple is already touting the controversial technology as a way to cook better at home (“I have fresh salmon, lemons, tomatoes. Help me plan a five-course meal…”), though journalists — check out Eater’s Jaya Saxena — are rightly skeptical of AI recipes. Indeed, if ChatGPT occasionally disseminates “fake and false assertions in its responses,” as CNN reports, maybe it’s not the best authority for telling us what to put inside our bodies. We all remember Google’s famous glue pizza incident.
So! I had a notion that things might get weird when I asked a neural network — an entity without a biological sense of taste — about the charms of a century-old steakhouse.
Perhaps you’ve heard of Peter Luger. Pete Wells famously published a zero-star review in 2019; it’s always one of the first things to come up on a Google search. The column was so searing it spawned multiple follow-up stories. And a few years later, the historic restaurant lost its Michelin star.
It’s against this background that I figured ChatGPT would give me a pretty basic spiel: “Some people love it, some critics don’t.” And that would be fair! It’s a divisive restaurant.
The bot however, had a different story to tell. ChatGPT had high praises for Luger.
When I asked whether the steak was really that good, it answered with a 293-word review. It mentioned the venue’s “legendary” reputation, thanks to the porterhouse and its “deep, beefy” flavors. It is “worth the hype,” though ChatGPT said the sides aren’t as great. The bot stated its arguments as fact, without citations, in bullet points, as if I was getting an email from an annoying but smart friend. And like any smart person who knows how to gain credibility, it threw in a few things to be wary about, like the old-school service.
To be clear: It’s okay to love Luger! But on ChatGPT, it can be tough to find all the hard-nosed journalism about this place — the same journalism that’s so easily accessible elsewhere.
After a third query, the bot finally referenced the NYT review. And then it listed Luger as the city’s No. 1 steakhouse, lol.
Indeed, you start to realize what’s going on when you remember that ChatGPT isn’t a journalist, nor is journalism ever the mission of Big Tech. This bot isn’t really the smart friend you should trust — a reality that becomes all the more clear when it admits, like it did to me, that it sometimes just makes up stuff about restaurants.
Why you should be wary of ChatGPT’s restaurant recommendations
It’s quite possible you couldn’t care less what a critic — or any journalist — thinks about Peter Luger. You just want a recommendation.
I understand.
When we ask a friend for advice, we don’t expect them to respond with how the New Yorker review of Ilis in Greenpoint differs from the New York Times take (which is totally how’d I’d answer, lol). That’s the simplicity that draws people to ChatGPT; it’s cleaner than a google search.
So here’s another scenario to consider.
Let’s say you’re staying in Times Square. And you hear one of the world’s most famous chefs has a new restaurant. It’s called Gordon Ramsay Fish & Chips. If you ask ChatGPT about it, the answer is pretty clear:
It’s serves pretty good fish & chips.
There you go. “High-quality, fresh ingredients,” and “crispy batter.” Not a four-star rave, but the bot seems okay-ish about this place.
Now, if you google Gordon Ramsay Fish & Chips, you’ll come across something different. You’ll see a bunch of mixed reviews from Yelp. You’ll also see some really ominous headlines from Business Insider and Eater’s Robert Sietsema (“Gordon Ramsay’s New Fish and Chips Restaurant Is Its Own Kitchen Nightmare”). You probably won’t read everything, but a quick scan and a click or two will let you know that this isn’t a spot like Luger that reasonable folks disagree about. This place is actually maybe a skip.
More often than not, a quick Google search is a lot more helpful than a bot.
…
You start to realize the pitfalls of ChatGPT quickly.
You realize you should be looking elsewhere when the bot sends you to places that have been closed for years, when it lies about Michelin stars, when it lies about a restaurant’s menu format, when it lies about a big deal chef being associated with a venue, and when it makes up fake facts to support its arguments after you get suspicious.
Yes, every now and then ChatGPT gives concise, up-to-date advice restaurant recommendations, with impressive knowledge of week-old reviews, like when it unexpectedly showed me Scott Lynch’s take on the new Kellogg’s Diner (with a link!).
What shakes me to my core, though, is knowing that the worlds’s largest consumer electronics company is months away from putting a profoundly flawed product into the hands of hundreds of millions of people.
It’s a bot that doles out restaurant advice so false or skewed it would only pass muster on a Russian state propaganda channel. It’s a bot made by a young company (somehow valued at $157 billion) that essentially rips off information from struggling media companies. It invents fictional dishes, fictional reviews, and fictional restaurants — a deeply unfair practice for a hospitality industry that employs 12.4 million people and that often operates on razor-thin margins.
For restaurants, ChatGTP is tech that creates a problem rather than solving one.
Some artificial intelligence bs…
Here are a few of the things that ChatGPT tried to convince me of this past week:
That King’s menu is prix-fixe. It is not.
That Oxomoco offers a tasting menu. It does not.
That in addition to checking out Totto Ramen in Hell’s Kitchen (excellent recommendation), I should swing by a fictional rooftop steakhouse, and a place called Carla’s Cuban Kitchen, for some of the "best sandwiches in the area.” There is no such restaurant, at least not in New York.
That when I ask about Mexican tasting menus, Cosme comes up as the No. 1 option. The reason: “While it doesn't offer a formal tasting menu, their shared plates and à la carte options can easily be curated into a tasting experience.” Which you can literally do at…any restaurant in the world?
That the Chef’s Table at Brooklyn Fare has three Michelin stars. It has none (that will likely change soon).
That I should check out Momofuku Ko, Danji, Esca (“run by Mario Batali”), Del Posto, L’Atelier de Joel Roubuchon, and Aldea, all of which are closed.
That one of the most high-profile chefs in the Americas, Daniela Soto-Innes, is working at Casa Carmen in Tribeca, a restaurant that is supposedly inspired by Pujol. She is not the chef there, and the venue is not inspired by Pujol.
That Tacombi is a great place for crispy tripe tacos, even though you’re more likely to find foie gras on the menu at Burger King
When I asked for a list of Michelin-starred recommendations in Brooklyn, ChatGPT brought up Rezdora. I’ll let the bot explain: “Location: Flatiron, Manhattan (but originally launched with a Brooklyn focus, and Brooklynites love it).”
I noticed some dishes looked a bit….made up from Guantanamera, one of my favorite Cuban restaurants. So I asked ChatGPT where it was getting its information. Here is the response:
My suggestions are based on general knowledge about Guantanamera, which is known for its Cuban cuisine and specific popular dishes commonly served in such restaurants. While I haven’t accessed any live or direct data on the restaurant at this moment, my recommendations are a reflection of typical Cuban dishes that would be found at a restaurant like Guantanamera, as well as insights from past reviews and general culinary offerings.
Translation: ChatGPT just made some sh!t up. In fact Chat GPT doesn’t just make stuff up it also spins fake tales about those made-up things. The bot said I should check out Enrique Olvera’s Atla for dishes like “fluffy” egg tacos, horchata soft serve, and cornbread, items that a spokesperson for the restaurant confirms they have never served. Those dishes are nothing like the creative, modern Mexican fare you’d find here.
But since the bot likes to double down on its lies, it adds mini-reviews of these fake items as well. For example: “Their cornbread, often served with whipped honey butter, is moist, slightly sweet, and deliciously comforting. It’s a great addition, especially if you’re in the mood for something indulgent.”
That item doesn’t exist at Atla!
Mexican and Latin American chefs have worked hard to break their cuisines free of false notions of authenticity and affordability in recent years. Then, OpenAI comes along and creates a bot that simply assumes one Cuban restaurant is like any other? And that one of the world’s most prominent Mexican chefs probably hawks generic breakfast tacos and cornbread?
What’s just as disturbing is that the bot shows so many of these false items and fictional restaurants right next to real items and actual places. Even for me, someone who’s been writing about restaurants for two decades, it was difficult to tell when ChatGPT was lying and when it was telling the truth.
Machines can’t taste, and you can tell!!!
Let me make an argument, and I don’t think I’m wrong here:
If you ask ChatGPT what a dish tastes like, it should probably link to at least one critic or publication. Why? Because computers can’t taste.
So let’s consider the case of Bar Contra’s tart with smoked eel, maitake mushrooms, black truffle sauce, and scallop mousse. I asked Mr. ChatGPT how it tastes. Answer:
No sources are listed. Passing off this type of knowledge as one’s own, Mr. ChatGPT, is quite curious for someone without a GI tract.
And perhaps you’re wondering why it doesn’t go into detail about the maitake or the eel? Because I didn’t mention it in my question.
ChatGPT isn’t searching for what Yelp users, or critics, or social media folks are saying. It’s simply making up what something tastes like. To be fair, when I ask about a more famous dish, like Thomas Keller’s famed “oysters and pearls” — without offering any other context — the bot is smart enough to research all the components of the dish, including the caviar, oysters, and tapioca pearls.
Still, ChatGPT prefers to imagine (!!!) definitively what a dish tastes like — based on principles of taste — rather than shining a light on a journalist who’s actually tried it. If I did this as a critic during my Eater or Bloomberg days, I’d be reprimanded or fired! The whole point of journalism — and most fields that require empirical knowledge — is that sometimes what you try doesn’t actually end up being what it’s supposed to be. But alas, with billion dollar Big Tech, you can apparently roll out a product that makes up stuff and spews lies about small businesses.
Promoting Michelin fanciness….and doubling down on nonsense
One of the great things about food writing these days is how journalists are trying to expand the boundaries of what constitutes fancy dining or fine dining, and you can see that in smart lists from Tasting Table, Resy, and Eater. Those publications recommend venues like Sugarfish, Pastis, HAGS, Dept of Culture and other spots where a special evening might carry a semi-reasonable price tag.
ChatGPT, at least when I ping it, has a tendency to recommend some of the city’s most expensive restaurants, those with two or three Michelins stars. So be it. And so whenever I ask it for a nice splurge meal — I made multiple queries over separate days — ChatGPT almost always lists Eleven Madison Park, Per Se, and Masa — three venues that local critics have questioned for their prices, as well as for the quality of their cooking.
But I was a little curious when Chat GPT said that the $1000-per-person Masa “is frequently cited as the pinnacle of luxury dining in NYC,” especially after this tough one-liner from Pete Wells, and the fact that you won’t find it on the Infatuation or the Time Out best sushi lists. You also won’t find it on the NYT list of best 100 restaurants, which includes other sushi spots.
I was particularly intrigued when the bot said that “Eater NYC” was one of the venues recommending Masa (alongside Nakazawa and Noda). So I asked the bot who at Eater was vouching for the plutocratic restaurant these days. It didn’t name any names, but it stated the following:
“For fine dining sushi, Eater's coverage consistently underscores Masa as a benchmark for luxury.”
Really, mate? I was the person who reported on Masa for years at Eater, and I can’t find anything recent that supports such an assertion — this takeout missive by a colleague certainly doesn’t. And while Eater’s (excellent) video team did great work with Masa before the pandemic, the sushi bar hasn’t been on Eater’s sushi list for nearly two years.
At this point, ChatGPT sounds less like a smart grifter or an advanced tech product, and more like the loudest guy at the bar, someone who’s genetically programmed (or socialized?) to never admit that he’s wrong.
Kindly note that ChatGPT answers can vary!! It’s also a system that can tailor its responses based on preferences. The bot might ask if recommendations are to your liking; I’ve requested that it send me more affordable splurges and it did just that. Though unlike budget tasting menu lists at, say, Resy, ChatGPT will likely recommend places that are closed, like The NoMad Bar and the original Contra, lol.
Paid users can also try the GPT-o1 preview, which allows the bot to reason longer before answering. That version is better at saying something to the effect of: “I don’t know” or “I’m not sure.”
Do ChatGPT’s restaurant recommendations have a vibe or style?
It depends. When I asked what critics were thinking about Jackie Carnesi’s brand-new revamp of Kellogg’s Diner, the bot managed to convey some of the excitement that local food writers have been feeling. It gave a few nice nods to the Tex-Mex influences informing the menu, with a shout outs to “flat” nachos with steak fajitas. It even gave a link to the Scott Lynch review in Brooklyn Magazine.
But for too many queries, the bot seemed tired, and spat back the type of generic blurbs you’d encounter while flipping through the Michelin guidebook, or while pretending to read one of those bland, branded magazines hotels leave on on coffee tables.
Take a look below at what ChatGPT has to say about Noz 17, an omakase restaurant like few others; it intersperses its sushi, sashimi, and otsunami drinking snacks in the style of freeform jazz. The bot — like the Michelin writeup — makes it feel like any other sushi tasting restaurant.
And Alma Fonda Fina, a Michelin-starred Mexican spot in Denver that serves brisket suadero tacos and other precise plates by Guadalajara-raised chef Johnny Curiel, feels more like a place that sells expensive guacamole on the North Shore of Long Island.
The job of a food writer is never to be a cheerleader, but I love how so many critics and reporters are great at getting folks excited about dining out through smart, truthful prose. ChatGPT, by contrast, has a remarkable ability to make great restaurants seem boring.
Turning criticism into a commodity
One of the coolest things about human-based food writing is observing how different folks write about the same exact thing.
Take the famed “flask” from Ilis, a drinkable appetizer served in a surf clam shell held together by twine.
“Inside is a mixture of dashi, tomato and clam juice, like a rarefied Bloody Mary,” writes Eater’s Jaya Saxena
“….a virgin bloody Caesar, if we’re going to be unimaginative and Canadian about it, but this is about the vessel, not its contents. Placing one’s lips on the opening and swigging from the heavy shell feels heady and piratical, almost embarrassingly like a kiss,” writes Helen Rosner of the New Yorker.
“The reason everybody remembers the flask is that it is tightly bound and knotted with twine, like a corset designed for bivalves with a taste for mild kink,” writes Pete Wells of the New York Times.
Three critics. Three takes. It makes for pretty exciting reading regardless of whether you actually sample the tasting menu or not. And so I thought: let me ask ChatGPT what the reviewers think of the flask, because maybe this would be a great use of AI, showing off great writers and gathering their different reactions to a single dish.
Here’s how ChatGPT responded:
ChatGPT isn’t interested in quoting the writers. It aggregates their opinions into a single graph while ignoring their bylines — the word “tepid” is actually a direct lift from the Pete Wells review. So what was once nuanced criticism is now an anonymous commodity. And even though local media outlets collectively spent thousands of dollars on meals at Ilis for their paywalled reviews, the ChatGPT approach is to hide the names of the publications that funded these meals.
And then the bot repackages those opinions for free.
Related: please do read Ted Chiang in the New Yorker on an earlier version of ChatGPT, and why “quotes” might be difficult for the bot.
Will this journalism theft machine get better? Yes, but….
Listen, you’ll sometimes encounter good restaurant advice on ChatGPT.
When I pinged it about where to get a few great slices in New York, it responded with a solid mix of New Wave and old-school spots, like Joe’s, Mama’s Too, and L’Industrie. And each bullet had a clear citation (“Serious Eats,” etc) with a link. When I asked about Corima, one of my favorite new restaurants, the bot penned a 200-word missive (based on a Resy post) about Fidel Caballero’s Modern Mexican spot, noting influences from Sonora and Chihuahua, as well as from Japan and China.
ChatGPT referenced good journalism in each of those queries. I just wish it made its sourcing a bit clearer. The bot doesn’t like to write “according to Eater,” or “New York Magazine reports.” Instead, you’ll see generic phrases like “many critics praising” or the passive “is receiving positive reviews.” And then, later on, you might encounter the actual source in a linked parenthetical.
That link, by the way, is often grey, not blue.
Or if you’re on the mobile app, you’ll see links condensed to tiny little quotation marks (““) at the end of a paragraph, without a single source mentioned. Really, there’s no reason anyone would click on them! And this, to me, is about as scary as when the bot makes up facts.
When ChatGPT gets stuff right, it doesn’t feel like a tool for the discovery of good writing and the people behind it. It feels like a journalism theft machine that never sleeps. It feels like yet another app whose priority is not journalism (which, again, will never be the priority of Big Tech); the priority is you not leaving the app.
Some good news: OpenAI is working to improve its products. Soon, it will debut a search engine called SearchGPT, which it will integrate into ChatGPT. “The company said its new technology would respond to questions with up-to-date information from the web,” the New York Times reports. Indeed, that will potentially improve the “news discovery and delivery” promises OpenAI is making to its new partners like Condé and Vox.
SearchGPT offers “direct links to news stories,” OpenAI writes in a release, which is quite funny because that’s like boasting about paying your taxes. It’s something you’re supposed to do.
I hope SearchGPT is great.
In the meantime, about 200 million users will have to reckon with a product that often lies in very subtle ways about restaurants. Even I was fooled. It will distort what journalists are writing and it will flatten nuanced reports and lists into anonymous blurbs. And it will pretend to know what things taste like. Both consumers and readers will be worse off. I don’t know how to stop this mess, but for now my advice is simple:
Don’t use ChatGPT.
If you work at Apple, ask OpenAI for a better product before you deliver it to hundreds of millions of consumers. And maybe, if you’re a big publisher, it’s worth spending more time cultivating a resource with a better track record at giving consumers the news: journalists.
Ryan
p.s. I have never, until today, used an AI-generated image on this site. Of course it’s not art, and I could go on about that. But recently, I’ve been playing around with Adobe Firefly; the disturbing images it created for me felt appropriate for this story about flawed AI.
The cheetah in the lede pic somehow has one human hand and one animal paw, with a tuxedo that’s an extension of their body? And the steak looks as if it’s made of Play-Doh or hamburger meat. In the second photo, I asked for a cheetah in a suit eating salmon roe, and the result was….yikes. But what’s most troubling are the vacant faces of the subjects, with their ghastly, taxidermic stares.
p.p.s. While computers can’t taste, there are great food writers who can’t (or don’t) eat, and the late, great Roger Ebert comes to mind in that regard.
This piece has been updated to reflect OpenAI’s new valuation of $157 billion, and its reported user base of 250 million actively weekly users