@dambuildshit Step 1: Build a free DNS provider
Step 2: Collect tolls for access to sites behind their DNS
Step 3: Build a crawler that doesn't need to pay their own toll
Step 4: Profit
English
Diffbot 🤖
2.1K posts
Diffbot 🤖
@diffbot
Never write another web scraper. Diffbot structures information from the web, so you don't have to.
The Future Katılım Eylül 2009
7.7K Takip Edilen8.1K Takipçiler
@diffbot does this for ages, and with a very nice free plan.
Cloudflare might have a different endgame with their endpoint that we are missing right now.
Anubhav@Anubhavhing
Crawling an entire website used to take: A Python script. Playwright or Selenium. Proxy rotation. Rate limiting logic. Error handling. 3 hours of debugging why page 47 returned a 403. Now it's one API call. Every web scraping startup that raised millions to solve this problem just became a single endpoint. Every freelancer charging $500 to "extract website data" just lost their entire business model to a /crawl command. HTML. Markdown. JSON. Pick your format. No scripts. No browser. No headache. The entire web scraping industry just got reduced to one line of code. Someone is going to use this to clone every competitor's website by Friday. 💀
English
Diffbot 🤖 retweetledi
Diffbot 🤖 retweetledi
Ever wondered what your white name should have been? Introducing: whatismywhitename.com
Upload a picture of you, and let the puppy guess your name! Let's test out nominative determinism 🫡
(Immigrants who named themselves will correlate more highly. Give us feedback plz)
Our thanks to:
- @modal for their generous credits toward training this meme model
- @diffbot for the clean, diverse dataset!
- @leannch86920 for the training research!
- Everyone NOT named David (biggest & noisiest dataset ever)
English
@devanshu_twt Sorry! It’s not ideal but it’s the easiest way to weed out 99% of abusers. When the product makes it easy to crawl the web, you get a lot of bad actors.
Still thinking of a better way to solve this!
English
@diffbot It's a bit more than I was planning, but yes, I'll be able to pile up a few hobby projects to squeeze them into a $50 months.
(I prefer the extra credit model *because* my interests are bursty, but I also get it's not exactly your core audience :)
English
One wish for 2026: If you count usage in credits/tokens & you offer several subscription layers with vastly different prices, *please* allow folks to buy extra credits.
(@diffbot , today - would love to use it, willing to spend $20, but $0 to $229 is a bit much of a jump)
English
Diffbot 🤖 retweetledi
State of E-commerce Data Providers - Q4 2025
E-commerce runs on constant measurement: prices, promos, availability, seller changes, and "what the shelf actually looks like" across retailers and marketplaces.
The challenge is stable collection at scale, retries when sites break, anti-bot evasion, clean geo signals, and then turning messy HTML into usable structured data.
In preparation for the holiday season, we mapped the landscape of e-commerce data providers:
Competitive intel + digital shelf: @dataweavein, @Price2Spy, @bigdataNODE, @Profitero, @WiserInc
Marketplace intelligence + data: @junglescout, @H10Software, @datahawkco, @SellerSprite_EN
Trade, Supply Chain, Imports / Exports: @Trademo1, @ImportYeti, @datamyne
Scraper APIs & Extraction Platforms: @zytedata, @diffbot, @Stratalis, (AutoScraping handle?), @serpapi
Managed Data Extraction & Services: @groupBWT, @Data_Ox, @epctex, @MrScraper_
Retail Media & Ad Platforms: @Pacvue, @PerpetuaLabs, @Teikametrics
Network & runtime infra for e-com scraping: @playwrightweb, Puppeteer, @browserless
English
Yes, I know, it adds engineering overhead and might open doors to abuse if not carefully planned.
But it's Christmas, I get to make a wish, right? :)
English
Diffbot 🤖 retweetledi
YouTube, TikTok, Mastodon, & Threads are mostly there but need optimizing.
Diffbot goes incredibly far with articles & that’s also moving along well.
Reddit & Bluesky are readily available but I haven’t spent the time.
X is finished by the endpoint gets rate limited 😞
Matthew Cassinelli@mattcassinelli
I am in love with scraping using Shortcuts. I have about 8 sets of shortcuts for everyday social media sites that I'm developing in tandem. I'll be releasing them as I finish them – thread starts here.
English
Diffbot 🤖 retweetledi
San Diego developers, join us and our technical partners @neo4j, @Intuit, Eyepop.ai, @Replit , and @diffbot at our HackNight next week!
Unstructured@UnstructuredIO
We're excited to join @neo4j , @Intuit, EyePop.ai, @Replit and @diffbot as technical partners for the upcoming Startup San Diego - FirstWave Innovator HackNight happening Wednesday, February 19th at the Intuit San Diego Campus. This is going to be an epic night where 100+ developers will come together to create innovative solutions for five select startups. 🚀 Join as a developer for free or get tickets: lu.ma/4brcg3lz
English
89,886 developers are building their own Perplexity on-prem with Diffbot LLM — huggingface.co/diffbot/Llama-…
English
#Perplexity Sonar Pro API launched last week as the best performing model on factuality.
24 hours later, it's the 2nd best performing model (and it's not because of #DeepSeek).
Why? 👇
English
Diffbot 🤖 retweetledi
Diffbot launches open-source AI model that achieves 81% accuracy by querying a trillion-fact Knowledge Graph in real-time instead of relying on static training data 🧠📊
Read more: venturebeat.com/ai/diffbots-ai… #ArtificialIntelligence #Enterprise #MachineLearning @diffbot
English