Not optimize for "most documents indexed" but "highest quality of results". One of them encourages adding spam to their index, the other encourages removing spam from their index.
HN is constantly pushing this notion that "spam" is some well-defined, solvable problem, so obviously Google wants it. That narrative just doesn't make sense from any angle. The notion that more click bait improves Google's bottom line is absurd
Product reviews alone, whether it is enterprise software or sports clothing should be something that they can easily comb through by hand, as humans, and uprank sites that are putting out more than affiliate link assemblies.
Luckily HN posters don’t exactly represent a meaningful portion of the population.
You are dealing with a moving target that has a huge financial incentive. It's a very difficult problem.
Im legitimately asking, who is responsible for Search at Google? Prabhakar Raghavan is SVP, Search, Assistant & Ads, and I click under him, he has 8 product groups reporting to him, and none of the people are responsible for Search. Yossi Matias is responsible for Search Engineering.
It may at first come off as a laughable answer, but Google Search has been in a directionless spiral since Marissa Mayer left. Her Yahoo tenure was not well received, but at Google she cared about the end quality of the product. Her title was Search Products and User Experience. Notice how we have gone from User Experience to Search Engineering, forgetting about the people who actually use the product.
You figure out a way to crowdsource certain decisions and establish who you can trust. Ask them questions with right and wrong answers. You start to tackle it one product category at a time. Instead of pagerank, which was a web of who linked to who" you start figuring which voters you have who consistently turn in good feedback.
This is some form of metamoderation that slashdot tried to implement.
If you are going to be a tastemaker, stop hiding behind "the algorithm" having some mind of its own that cant be controlled.
> The notion that more click bait improves Google's bottom line is absurd
If you don't find what you're looking for on the first try, you'll need to try again, and see more ads. What else are you going to do, go elsewhere, visit a library, ask the town elders or give up on looking for things you want to know? You don't have a choice, you know it, they know it.
I find it equally plausible that Youtube's search sucks badly because they don't care what you're looking for, they want you to watch videos that they predict will lead to the maximum time spent on the site, again so you watch more ads. What other explanation is there that the world's leading search engine has the search of one of their flagship products run at 1999 quality? Presumably they have giant teams of people working on that too?
I see two options: a) Google can't do any better than that, b) Google has a reason to keep it in the current state (I'll put "Google doesn't know because nobody at Google has used Youtube in the last 5 years" and similar options under "a").
a) sounds ridiculous, b) sounds conspiratorial. What are the other options?
And again, I'm not saying they are making search worse on purpose (no "from now on our core mission is to make search suck"). I'm saying they aren't optimizing for SERP quality. They seem to care about index size (maybe it's an internal KPI? would certainly explain their aggressive guessing at additional URLs that you might have on their page but don't link to, don't add in sitemaps etc, and their stubbornness in keeping results from the index even if they've been 301ed or 410ed ages ago (they do get downranked after a while though)), but I assume that they mostly care about paid ad clicks, and if something increases ad clicks while the result quality decreases, they'll do it.