zlacker

>>ColinW+(OP)
How does WEI work with non-browsers, like curl or python requests? I was wondering if there is some motive here to monopolize web scraping (especially with respect to harvesting AI training data)?

>>andy99+cw
I mean that’s part of the point. It’s there to exactly lock out scrapers. Or crawlers, for that matter. What a happy little coincidence.

>>AndroT+jy
This is just silly, there exist frameworks like selenium that allow you to run any browser of choice and emulate actual user behavior(clicks, keystrokes). If they go further the emulation layer will have to be moved higher, above the virtual machine running the browser for example. The truth is, this has nothing to do with scraping, scrapers will find a way. This is to stop the majority of people from using ad block.

>>Roark6+ez
Hi, Selenium & Appium creator here. I've always been on the test automation side of things. The fact that these tools were also useful for scraping was an interesting coincidence to me. These days I make physical robots that are the "real world" equivalent of Selenium or Appium with a stylus that actually taps the screen and presses buttons. To websites and apps, taps and clicks are real, not emulated. Primary use is still test automation, especially when it also involves a real-world component like a credit card transaction with a credit card reader. The number of people contacting me who are interested in getting a physical robot as a way to circumvent software bot detection is increasing. Yes, scrapers will find a way.