The problem with search is not that our answers aren't summarized well, it's that the quality of information returned for those searches is getting increasingly worse, and we are getting increasingly worse at categorizing or filtering that information in any useful way. And LLMs pulling information in and summarizing it for me is... not helpful? It's summarizing the same garbage, except now sometimes it also summarizes it wrong.
But it's not even an issue with the quality (although the quality of information from LLMs is also pretty over-hyped I think). Conceptually, I don't know that this is a product that I would ever want. I can't think of any time where I've sat down to do a search on Google or DuckDuckgo and thought, "You know what I want? I want these results presented to me in a less structured format using natural language and with less granular knowledge about where each specific statement is coming from."
At least Bing seems to be trying to do inline citations in some of its answers, which is a step up over Google's AI announcement, I guess?
Maybe I'm just in the minority on that. Users seem to like this a lot. But my ideal version of the Internet is one that decreases the number of abstractions and layers and summaries between myself and primary data rather than increasing them. My ideal Internet is a tool that makes it easier for me to actually find things, not a tool that increases the layers between me and the raw source/information that I'm looking for. I already have enough trouble needing to double-check news summaries of debates, events, and research. Getting another summary of the summaries doesn't seem helpful to me?
I can think of some ways where I might use an LLM in search, even really exciting ways where maybe it could help with categorization or grouping, but it doesn't seem like Google/Bing are interested in pursuing any of that. I look at both the Bing and Google announcements and just think, "why are you making it worse?" But who knows, maybe the actual products will sell me on the concept more.
I do see LLMs as potentially more useful for “fanciful” queries, like “what can I make with kale, tomatoes, and mushrooms?”
I think that encyclopedias are cool, but I also feel like the Internet was hopefully going to be a slightly better version of that and that it's a little frustrating to be going in the opposite direction. I'm not sure how to articulate that other than that encyclopedias are in some ways a compromise around the fact that we very often don't have good search, so we accept human beings trying to pre-aggregate data for us with the hope that they are better than Google is at aggregating that data.
And I think Wikipedia is valuable not so much because of the summaries, and more because it's obsessively curated and has (or attempts to have) a very specific, predictable set of rules that it (tries) to adhere to about sources and coverage. The text-portion of Wikipedia isn't really the part that I think is most impressive about it. If GPT-3 was being used to aggressively curate search results and remove low-quality content, then yeah, that would be potentially interesting (although I'm not sure how well it could handle that task).
----
> what can I make with kale, tomatoes, and mushrooms?
I sort of see it, it's one of the more... it's fine. It wouldn't be a strict downgrade over existing search, maybe it would save time in some situations. But if I'm being completely honest, what I want as an answer to that question isn't a paragraph of text explaining multiple recipes, it's a bullet-pointed list of recipes with links to the original sites they're listed on, so I can check to see if they're worth making.
Bing's results (to its credit) seem like they're sort of headed in that direction, which, nice. Yeah, I could use that. But a bullet-pointed list of recipes is also... search results. So how much time and effort have we put into reinventing a cleaner search interface where at best it solves the same problems that search already solved, and where we in the meanwhile haven't made any progress on the really hard fundamental problem of "how do we get a good list of recipes to display in the first place and what does 'good' mean in that context?"
----
To be less cynical, one way I could see this genuinely being useful would be completely behind the scenes in a non-user-facing role just as a way of applying "tags" to webpages or doing filtering. I would love to be able to search "is spinach poisonous for rats who are pets and just because I used the word 'poisonous' does not mean I want 2 pages of links about exterminators or getting rid of wild rats."
An LLM would be a great fit for that, because we've done such a garbage job of categorizing the web that it's very difficult to know which words to type to exclude "categories" of information from a search query. So maybe an LLM helps standardize that a bit? But then I want a normal page of search results. I don't want to have a conversation with the LLM, I want the LLM's role to be exclusively "I think this webpage should be additionally included/excluded in your query. I think this webpage is about extermination even if it doesn't use that specific word."
If Bing's service ends up being able to do that kind of thing well, then yeah, that's useful. I'm a little skeptical they can, because... gestures to the current quality of search results, but maybe their integration with GPT gives them way more capabilities and ratchets up their quality.
But similar to above, it feels a little bit like reinventing the wheel. I can refine GPT queries during a conversation, great. Could I have that feature for regular search? Why do I need to do it as part of a conversation? That seems like a good thing tied to a bad UX (although again, I might be atypical in thinking that natural language conversations are often bad UX).
And I do want to couch that by saying that maybe Bing will surprise me and it will have easy options to do that kind of thing. I'm just not currently seeing it presented in a way that looks useful.