Med-Mastodon

ResearchBuzz: FirehoseThe Map Room: Unauthorized Waffle House Index Disaster Maps Taken Down. “The Waffle House Index is an informal metric used to assess the severity of a storm in the U.S. South, because Waffle House restaurants don’t close unless Things Are Very Bad. But when Jack LaFond scraped Waffle House’s website to build a map tracking restaurant closures last fall, he got a cease-and-desist from […]<a href="https://rbfirehose.com/2025/05/31/the-map-room-unauthorized-waffle-house-index-disaster-maps-taken-down/" class="" rel="nofollow noopener noreferrer" target="_blank">https://rbfirehose.com/2025/05/31/the-map-room-unauthorized-waffle-house-index-disaster-maps-taken-down/</a>

PromptCloud🤖 Ever heard of a browser that works without a screen?That’s a headless browser — your invisible ally for scraping, automation, and testing at scale.In our latest Uncomplicate Series, we break it down in plain language. ✨ No jargon. Just clarity.🔗 <a href="https://tinyurl.com/42n266cn" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://tinyurl.com/42n266cn</a><a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a> <a href="https://mastodon.social/tags/Automation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Automation</a> <a href="https://mastodon.social/tags/HeadlessBrowsers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HeadlessBrowsers</a> <a href="https://mastodon.social/tags/OpenWeb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenWeb</a> <a href="https://mastodon.social/tags/UncomplicateSeries" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#UncomplicateSeries</a> <a href="https://mastodon.social/tags/PromptCloud" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PromptCloud</a>

PromptCloudTired of babysitting DIY scraping scripts that crash the moment you scale? You’re not alone.PromptCloud takes the pain out of large-scale data extraction with fully managed, reliable solutions — so you can focus on what really matters: insights.🔗 <a href="https://shorturl.at/EApIO" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://shorturl.at/EApIO</a><a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a> <a href="https://mastodon.social/tags/OpenData" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenData</a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataEngineering</a> <a href="https://mastodon.social/tags/BigData" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#BigData</a> <a href="https://mastodon.social/tags/Automation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Automation</a> <a href="https://mastodon.social/tags/PromptCloud" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PromptCloud</a> <a href="https://mastodon.social/tags/TechForGood" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TechForGood</a> <a href="https://mastodon.social/tags/DataOps" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataOps</a>

Carlo ZottmannNeed to grab specific info from a webpage regularly? 🤔 Browser Actions can help! Create a Shortcut to: Open URL ➡️ Wait for data element ➡️ Run JavaScript to extract text ➡️ Pass it back to Shortcuts!If you need help with that, just follow the Forum link on the site!<a href="https://actions.work/browser-actions?ref=mastodon-b10" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://actions.work/browser-actions?ref=mastodon-b10</a><a href="https://norden.social/tags/macOS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#macOS</a> <a href="https://norden.social/tags/Shortcuts" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Shortcuts</a> <a href="https://norden.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a> <a href="https://norden.social/tags/DataExtraction" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataExtraction</a> <a href="https://norden.social/tags/BrowserAutomation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#BrowserAutomation</a>

@reiver ⊼ (Charles) :batman:3/For more on scraping (as in web-scraping) see here: <a href="https://mastodon.social/@reiver/114353728684249608" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://mastodon.social/@reiver/114353728684249608</a>CC: <a href="https://mastodon.social/@404mediaco" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@404mediaco</a> <a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:2/Scraping (as in Web Scraping) is the act of extracting data from HTML web-pages where the data is NOT machine-legible.If the data, even in an HTML web-page, is in a machine-legible format, then it is NOT scraping....And, getting data in JSON (key-value pairs) is definitely NOT scraping — as JSON's purpose is to communicate data in a machine-legible manner.CC: <a href="https://mastodon.social/@404mediaco" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@404mediaco</a> <a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:1/If these researchers used a typical HTTP-based API that returns JSON, then —What these researchers did is NOT scraping.CC: <a href="https://mastodon.social/@404mediaco" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@404mediaco</a>RE: <a href="https://www.404media.co/researchers-scrape-2-billion-discord-messages-and-publish-them-online/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.404media.co/researchers-scrape-2-billion-discord-messages-and-publish-them-online/</a><a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:5/For example, if software request data from a web-site, and the web-site returns HTML, but parts of the HTML has semantics marked up with a machine-legible format such as microformats, microdata, RDFa, etc, then it is NOT scraping.(microformats, microdata, RDFa, etc, are machine-legible format, designed to express semantics to machines.)<a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:4/For example, if software request data from a web-site, and the web-site returns HTML, but that HTML contains a <script> tag with JSON-LD in it, and the software consumes that JSON-LD, then it is NOT scraping.(JSON-LD is a machine-legible format, designed to express semantics to machines.)<a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:3/For example, if software request data from a web-site, and the web-site returns JSON, XML, or some other machine-legible format, then it is NOT scraping.<a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:2/Scraping (as in Web Scraping) is the act of extracting data from HTML web-pages where the data is NOT machine-legible.If the data, even in an HTML web-page, is in a machine-legible format, then it is NOT scraping.<a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:1/I am understanding when a non-technical person uses the noun "scraper" (as in "web scraper") or the verb "scrape" in a way that isn't accurate.But, I am surprised when what seems to be a technical person uses the word "scraper", "scrape", or "scraping" inaccurately — either claiming things that are NOT scrapers to be scrapers, or claiming that acts that are NOT scraping are scraping....<a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

DeployHQAutomate web scraping & deployment! Python, ScraperAPI & DeployHQ tutorial: extract data & streamline workflows. No more manual entry! 🚀 <a href="https://www.deployhq.com/blog/scrape-applications-using-scraperapi-and-deployhq" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.deployhq.com/blog/scrape-applications-using-scraperapi-and-deployhq</a><a href="https://mastodon.social/tags/webscraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#webscraping</a> <a href="https://mastodon.social/tags/python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#python</a> <a href="https://mastodon.social/tags/automation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#automation</a> <a href="https://mastodon.social/tags/scraperapi" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scraperapi</a> <a href="https://mastodon.social/tags/deployhq" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deployhq</a>

Rachel RawlingsI'm having trouble figuring out what kind of botnet has been hammering our web servers over the past week. Requests come in from tens of thousands of addresses, just once or twice each (and not getting blocked by fail2ban), with different browser strings (Chrome versions ranging from 24.0.1292.0 - 108.0.5163.147) and ridiculous cobbled-together paths like /about-us/1-2-3-to-the-zoo/the-tiny-seed/10-little-rubber-ducks/1-2-3-to-the-zoo/the-tiny-seed/the-nonsense-show/slowly-slowly-slowly-said-the-sloth/the-boastful-fisherman/the-boastful-fisherman/brown-bear-brown-bear-what-do-you-see/the-boastful-fisherman/brown-bear-brown-bear-what-do-you-see/brown-bear-brown-bear-what-do-you-see/pancakes-pancakes/pancakes-pancakes/the-tiny-seed/pancakes-pancakes/pancakes-pancakes/slowly-slowly-slowly-said-the-sloth/the-tiny-seed(I just put together a bunch of Eric Carle titles as an example. The actual paths are pasted together from valid paths on our server but in invalid order, with as many as 32 subdirectories.)Has anyone else been seeing this and do you have an idea what's behind it?<a href="https://infosec.exchange/tags/botnet" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#botnet</a> <a href="https://infosec.exchange/tags/ddos" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ddos</a> <a href="https://infosec.exchange/tags/webscraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#webscraping</a> <a href="https://infosec.exchange/tags/infosec" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#infosec</a>

ResearchBuzz: FirehoseThe Markup: A Guide on How to Legally Web Scrape EU Data. “At The Markup, some of our data journalists recently had questions about the legal risks involved in scraping websites hosted in the European Union. We conducted our own research to answer this question, and offer a summary of what we learned below. Our goal is to help other journalists, researchers, and advocates come up with a […]<a href="https://rbfirehose.com/2025/04/06/the-markup-a-guide-on-how-to-legally-web-scrape-eu-data/" class="" rel="nofollow noopener noreferrer" target="_blank">https://rbfirehose.com/2025/04/06/the-markup-a-guide-on-how-to-legally-web-scrape-eu-data/</a>

Matt HodgkinsonFor Immediate Release, April 1, 2025: University of Michigan Press will publish all of the content on Meta platforms as a series of printed books. <a href="https://www.linkedin.com/posts/charles-watkinson-7553a257_amphibians-and-reptiles-of-the-great-lakes-activity-7312775744932179968-sLSu" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.linkedin.com/posts/charles-watkinson-7553a257_amphibians-and-reptiles-of-the-great-lakes-activity-7312775744932179968-sLSu</a><a href="https://scicomm.xyz/tags/MetaPlatforms" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MetaPlatforms</a> <a href="https://scicomm.xyz/tags/Instagram" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Instagram</a> <a href="https://scicomm.xyz/tags/Facebook" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Facebook</a> <a href="https://scicomm.xyz/tags/ThreadsApp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ThreadsApp</a> <a href="https://scicomm.xyz/tags/Copyright" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Copyright</a> <a href="https://scicomm.xyz/tags/BookPublishing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#BookPublishing</a> <a href="https://scicomm.xyz/tags/TextMining" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextMining</a> <a href="https://scicomm.xyz/tags/TextCorpora" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TextCorpora</a> <a href="https://scicomm.xyz/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a> <a href="https://scicomm.xyz/tags/AIethics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AIethics</a>

TechdirtNot Content With Its Billions Of Web Scrapings, Clearview Tried To Buy Millions Of Mugshots And SSNs <a href="https://web.brid.gy/r/https://www.techdirt.com/2025/03/28/not-content-with-its-billions-of-web-scrapings-clearview-tried-to-buy-millions-of-mugshots-and-ssns/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://web.brid.gy/r/https://www.techdirt.com/2025/03/28/not-content-with-its-billions-of-web-scrapings-clearview-tried-to-buy-millions-of-mugshots-and-ssns/</a>

uǝuunɹƃʇǝOThoughts: AI corps scraping dataThe corporations assert that they can utilize public data without incurring any costs, citing fair use as their justification.To address this issue, we should implement a law that compels corporations claiming fair use as a defense to make all their process data publicly available, free of charge. This would ensure that the scraped data, as well as data derived from the freely available data, is accessible to the public. <a href="https://mstdn.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mstdn.social/tags/FairUse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FairUse</a> <a href="https://mstdn.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mstdn.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a>

ResearchBuzz: FirehoseReuters: News Corp sued by Brave Software, a Google search engine rival. “News Corp has been sued by Google search engine rival Brave Software, which seeks to forestall a lawsuit by Rupert Murdoch’s company for when readers are directed to copyrighted articles from the Wall Street Journal and New York Post.”<a href="https://rbfirehose.com/2025/03/15/reuters-news-corp-sued-by-brave-software-a-google-search-engine-rival/" class="" rel="nofollow noopener noreferrer" target="_blank">https://rbfirehose.com/2025/03/15/reuters-news-corp-sued-by-brave-software-a-google-search-engine-rival/</a>

Matthew Turland<a href="https://phpc.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a> with <a href="https://phpc.social/tags/Playwright" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Playwright</a> <a href="https://wanago.io/2025/02/24/web-scraping-playwright/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://wanago.io/2025/02/24/web-scraping-playwright/</a>

Recent searches

Search options

Administered by:

Server stats:

#webscraping