zlacker

>>ColinW+(OP)
How does WEI work with non-browsers, like curl or python requests? I was wondering if there is some motive here to monopolize web scraping (especially with respect to harvesting AI training data)?

>>andy99+cw
I mean that’s part of the point. It’s there to exactly lock out scrapers. Or crawlers, for that matter. What a happy little coincidence.

>>AndroT+jy
This is just silly, there exist frameworks like selenium that allow you to run any browser of choice and emulate actual user behavior(clicks, keystrokes). If they go further the emulation layer will have to be moved higher, above the virtual machine running the browser for example. The truth is, this has nothing to do with scraping, scrapers will find a way. This is to stop the majority of people from using ad block.

>>Roark6+ez
If I understand this proposal correctly, this is exactly to prevent such things. Yes, of course, it’s to prevent people from using ad block. But a nice side-effect is to block crawlers, or frameworks like selenium as well, so they can „serve ads only to real people“. Of course, people will always find a way to crawl. We already have bot farms that are just remote controlled smartphones lined up somewhere. But it makes it harder for everyone who isn’t Google to compete with Google.