zlacker

[parent] [thread] 1 comments
1. gpm+(OP)[view] [source] 2021-04-15 15:46:59
Huh, I thought they published a range of IP addresses they used to prevent this, but apparently they don't use an entirely consistent one and you need to do a dns request [1] to actually check if something is google's crawler. I'm willing to bet most organizations aren't doing that... so maybe.

[1] https://developers.google.com/search/docs/advanced/crawling/...

replies(1): >>leephi+ew
2. leephi+ew[view] [source] 2021-04-15 17:55:28
>>gpm+(OP)
I just installed a user agent switcher and tried it on a prominent financial news site. The offer to subscribe was replaced by the article when I reloaded using the Googlebot user agent.
[go to top]