Just two days after the release of SearchGPT, some people have already tested it in grayscale.

Today, netizen Kesku's self-made demo has been widely circulated on the internet. The SearchGPT results are so speedy that they have shocked everyone.

When asking if Porter Robinson has released a new album?

In a blink of an eye, SearchGPT immediately gave the answer "SMILE", and even attached a link. The response speed of the mobile version was almost zero latency.

Looking at the response speed of the mobile version, the delay is almost zero.

Below the comments section, there's an overwhelming shock.

However, on the other hand, in OpenAI's official demonstration that day, there were problems exposed by the foreign media "Atlantic Monthly".

When asked about the music festival held in Boone, North Carolina in August, SearchGPT got the time wrong and had a hallucination.

What happened to replacing Google as promised?

A first-hand test of the entire network has arrived.

Kesku's another self-made demo, presents a small widget of SearchGPT.

"One phenomenon she discovered is that SearchGPT tends to pay strong attention to search results."

"Sometimes, you need to tell it explicitly what you want to do instead of what you want to get from the internet."

For example, when asked about the weather in London, SearchGPT first gave the weather forecast for the next seven days.

What are the best times and places to have a picnic in London? Several backup options are given in a widget form.

Kesku also tested some examples on mobile.

Searching for stocks will give you a visualization of Nvidia's overall stock market, followed by some analysis. All stock information is seamlessly integrated into the page.$NVIDIA (NVDA.US)$By entering the name of a song "Never Gonna Give You Up", YouTube's song video is directly given without jumping to other websites, you can listen to the song and watch the video on one page.

In addition to the search function, netizens are also enthusiastic about conducting "humanization tests" - - -

In addition to search function, netizens also tirelessly engage in "humanity tests"-

How are you?

As an AI language model, I have no emotions, but I am here and ready to help you answer questions. What can I help you with today?

Tell me a joke.

Thus appeared a classic English pun.

Scientists don't trust atoms. Why don't scientists trust atoms? Because they make up everything!

Users on site are providing questions one after another, asking the post author to help test the model.

Envious netizens have been putting forth questions in the comment section, asking the post author to help test the model.

The first user asked, "Does it support regions and news, such as the latest news from the TestingCatlog?"

After testing, Kesku said, "It can provide local information for you through IP address or precise location (the latter is turned off by default, but can be turned on in the settings) - searches like 'movie theaters near me' yield good results."

Help me try out its intelligent search explanation ability.

In the explanation given by SearchGPT about high-bandwidth memory, the blue-highlighted content is the reference explanation.

Can you try searching for some articles behind paywalls? Like those that recently signed a partnership with OpenAI.

Kesku provided the content of an article, but it seems that it still cannot bypass the paywall and only gives a summary of the article.

More detailed content is still not visible.

Can you try searching "?$Yandex (YNDX.US)$Monthly active users?

I want to see if, when it can't find the exact answer I'm looking for, it will admit that it found the number of daily active users (DAU) instead of monthly active users (MAU), or if it will pretend to be confused like Copilot and just copy and paste the entire search result while ignoring the actual query.

After Kesku's search, the results are shown below:

Obviously, based on the questioner's question, SearchGPT gave an answer.

"Compared to Perplexity, how?"

Kesku said she has not tested complex tasks yet, but she loves the results she has tested so far.

In the prompt below, she directly asked the niche question, "Who is Kesku?"

Surprisingly, SearchGPT gave the correct answer, while Perplexity gave the wrong one.

Some netizens commented, "Cool demonstration! Maybe SearchGPT can bring some changes in the field of local search? It can help you get things done in the real world. From the appearance, it has good data sources, concise widgets, and super fast speed. I wonder if they can lower the cost of each query compared to Google?"

Unveiling the SearchGPT Search Mechanism

Technology media TestingCatolog was also the first to conduct internal testing and revealed a corner of the SearchGPT search mechanism.

Unlike the universal Bing search function provided by current ChatGPT, SearchGPT is better at providing real-time information.

Although it still relies on Bing's index, SearchGPT will have its own web crawler (similar to Perplexity), which is used to dynamically obtain real-time data to overcome Bing's slow speed problem.

Even TestingCatalog dug out the source code of SearchGPT and confidently stated in the comments, "Absolutely accurate, I have insiders."

The source code not only revealed Bing's interface, but also found that the search results are supported by a multimodal model.

Although the specific processing flow cannot be seen, the called model should have the function of automatically understanding images.

Official demonstration failed miserably, OpenAI was beaten in the face

Just as netizens were enthusiastically trying it out, The Atlantic magazine came out and poured cold water on it - SearchGPT had obvious search result errors in the official demo.

The search query given by the user was "Music Festival held in Boone, North Carolina in August."

This question is actually difficult to show the advantages of SearchGPT over traditional search engines. The same question thrown to Google search can also give almost identical results.

For example, "An Appalachian Summer Festival," which is at the top of SearchGPT, is also the second result in Google search.

But embarrassingly, the AI summary below the title got a key piece of information wrong - the festival dates have been confirmed by the organizer as June 29th to July 27th.

If you go to buy tickets based on the information given by SearchGPT, you will get nothing - July 29th to August 16th is exactly when the ticket office is officially closed.

OpenAI spokesperson Kayla Wood has admitted the mistake to The Atlantic magazine and said, "This is just the initial prototype and we will keep improving it."

This mistake reminds people of the tragedy once caused by Bard.

In February 2023, Google launched this chat robot product to compete with ChatGPT, but the first appearance was factually inaccurate, resulting in a 9% drop in Alphabet stock price and an instant evaporation of 100 billion US dollars in market cap.

Bard claimed that James Webb Space Telescope took the first photo of an exoplanet, but in fact, the credit belongs to the European Southern Observatory's VLT.

Fortunately, OpenAI has no stock price to fall, and the cautious approach of only opening internal testing is also quite prudent. After all, following Google's example, it can be predicted that errors like this are almost impossible to avoid.

Even if OpenAI can find a way to greatly reduce SearchGPT's illusions, it will still be insignificant in the face of huge traffic.

Assuming that the illusion rate is only 1% (this ratio is difficult to achieve), according to Google's scale, it will still produce tens of millions of incorrect answers every day.

Moreover, we have not yet discovered a reliable and effective method to eliminate LLM's nonsense and illusions.

And Andrej Karpathy once expressed the view on Twitter that "Illusions are not bugs but one of the biggest features of LLM".

Karpathy likened LLM to a "dream machine": we guide the model to "dream" with prompts, and then generate results based on fuzzy memories of the training documents.

Although most of the time the generated results are useful, since it is a "dream," it is possible for it to spin out of control. When LLM dreams into a field with factual errors, we will label it as an "illusion."

This looks like a bug, but LLM is just doing what it has always been doing.

This mechanism is completely different from traditional search engines. The latter only returns the most similar documents in the database word for word after receiving the prompt, so you can say that it has "creative problems" because the search engine can never create new responses.

According to Karpathy, it is very difficult to expect AI search driven by current LLM to generate 100% true and accurate results.

So how will the search engine revolution unfold? Will LLM's "dream imagination" and traditional search engine's truth and reliability coexist, or will it be "you die and I live"?

Author: Xin Zhi Yuan, Source: Xin Zhi Yuan, Original Title: "OpenAI was hit in the face! SearchGPT official demonstration is a fiasco, and the source code reveals the search mechanism"

Editor/ping