What I meant with “things like this are awful for the web,” I meant that automation through AI is awful for the web. It takes away from the original content creators without any attribution and hits their bottom line.
My story was supposed to be one about responsible AI, but somehow I screwed that up in my summary.
I’ve developed my own agent for assisting me with researching a topic I’m passionate about, and I ran into the exact same barrier: Cloudflare intercepts my request and is clearly checking if I’m a human using a web browser. (For my network requests, I’ve defined my own user agent.)
So I use that as a signal that the website doesn’t want automated tools scraping their data. That’s fine with me: my agent just tells me that there might be interesting content on the site and gives me a deep link. I can extract the data and carry on my research on my own.
I completely understand where Perplexity is coming from, but at scale, implementations like this Perplexity’s are awful for the web.
(Edited for clarity)
That was an interesting read, thanks for sharing!
Do we really need a blog to tell people about stuffy air?! I’m more and more surprised by how dumb people are. Soon we’ll have blog posts reminding us to inhale and exhale.
This article is deliberately phrased to spark precisely this kind of comment thread, and we’ve all walked straight into it.