"In our piece exploring whether the AI revolution is leaving APIs behind, we wrote about some of the factors limiting the extent to which AI tools like chatbots can interface with APIs.
Some of these include:
- Limited or no access to APIs for developers
- APIs are sometimes overcomplicated, bloated, or difficult to call
- Legacy APIs (WS/RPC) lack thorough or up-to-date documentation
- APIs sometimes only cover a fraction of the functions available via the UI
It’s worth noting that many of these points impact human API consumers just as much as they do agentic ones. If you’ve ever been in the position of trying to use an API and it falling short of your expectations, you’ll know just how frustrating it can be.
While it’s possible that some of those users will get in touch to ask you to add certain endpoints or clarify things, plenty more won’t. Some developers are more likely to take the view that it’s easier to ask for forgiveness later than permission now, and find some other way to extract the data they’re looking for. In many cases, web scraping offers just such a solution.
Web scraping APIs are a natural evolution of manual scraping techniques, such as using Python to scrape websites. Used for everything from scraping search engine results, like SERP APIs, to product prices and sentiment analysis, there are various services out there that make web scraping very straightforward. And they’re big business."
https://nordicapis.com/are-web-scraping-tools-overtaking-official-apis/