@dambuildshit Step 1: Build a free DNS provider
Step 2: Collect tolls for access to sites behind their DNS
Step 3: Build a crawler that doesn't need to pay their own toll
Step 4: Profit
English
Diffbot ๐ค
2.1K posts
Diffbot ๐ค
@diffbot
Never write another web scraper. Diffbot structures information from the web, so you don't have to.
The Future ๊ฐ์
์ผ Eylรผl 2009
7.7K ํ๋ก์8.1K ํ๋ก์
@diffbot does this for ages, and with a very nice free plan.
Cloudflare might have a different endgame with their endpoint that we are missing right now.
Anubhav@Anubhavhing
Crawling an entire website used to take: A Python script. Playwright or Selenium. Proxy rotation. Rate limiting logic. Error handling. 3 hours of debugging why page 47 returned a 403. Now it's one API call. Every web scraping startup that raised millions to solve this problem just became a single endpoint. Every freelancer charging $500 to "extract website data" just lost their entire business model to a /crawl command. HTML. Markdown. JSON. Pick your format. No scripts. No browser. No headache. The entire web scraping industry just got reduced to one line of code. Someone is going to use this to clone every competitor's website by Friday. ๐
English
Diffbot ๐ค ๋ฆฌํธ์ํจ
Diffbot ๐ค ๋ฆฌํธ์ํจ
Ever wondered what your white name should have been? Introducing: whatismywhitename.com
Upload a picture of you, and let the puppy guess your name! Let's test out nominative determinism ๐ซก
(Immigrants who named themselves will correlate more highly. Give us feedback plz)
Our thanks to:
- @modal for their generous credits toward training this meme model
- @diffbot for the clean, diverse dataset!
- @leannch86920 for the training research!
- Everyone NOT named David (biggest & noisiest dataset ever)
English
@devanshu_twt Sorry! Itโs not ideal but itโs the easiest way to weed out 99% of abusers. When the product makes it easy to crawl the web, you get a lot of bad actors.
Still thinking of a better way to solve this!
English
@groby Sorry for the late reply (and happy new years!)
It's not on the immediate horizon, but implementing a credit balance model with a low minimum is something we've discussed. I personally prefer it.
Would you mind emailing me at jerome[@]diffbot?
English
@diffbot It's a bit more than I was planning, but yes, I'll be able to pile up a few hobby projects to squeeze them into a $50 months.
(I prefer the extra credit model *because* my interests are bursty, but I also get it's not exactly your core audience :)
English
One wish for 2026: If you count usage in credits/tokens & you offer several subscription layers with vastly different prices, *please* allow folks to buy extra credits.
(@diffbot , today - would love to use it, willing to spend $20, but $0 to $229 is a bit much of a jump)
English
Diffbot ๐ค ๋ฆฌํธ์ํจ
State of E-commerce Data Providers - Q4 2025
E-commerce runs on constant measurement: prices, promos, availability, seller changes, and "what the shelf actually looks like" across retailers and marketplaces.
The challenge is stable collection at scale, retries when sites break, anti-bot evasion, clean geo signals, and then turning messy HTML into usable structured data.
In preparation for the holiday season, we mapped the landscape of e-commerce data providers:
Competitive intel + digital shelf: @dataweavein, @Price2Spy, @bigdataNODE, @Profitero, @WiserInc
Marketplace intelligence + data: @junglescout, @H10Software, @datahawkco, @SellerSprite_EN
Trade, Supply Chain, Imports / Exports: @Trademo1, @ImportYeti, @datamyne
Scraper APIs & Extraction Platforms: @zytedata, @diffbot, @Stratalis, (AutoScraping handle?), @serpapi
Managed Data Extraction & Services: @groupBWT, @Data_Ox, @epctex, @MrScraper_
Retail Media & Ad Platforms: @Pacvue, @PerpetuaLabs, @Teikametrics
Network & runtime infra for e-com scraping: @playwrightweb, Puppeteer, @browserless
English
Yes, I know, it adds engineering overhead and might open doors to abuse if not carefully planned.
But it's Christmas, I get to make a wish, right? :)
English
Diffbot ๐ค ๋ฆฌํธ์ํจ
YouTube, TikTok, Mastodon, & Threads are mostly there but need optimizing.
Diffbot goes incredibly far with articles & thatโs also moving along well.
Reddit & Bluesky are readily available but I havenโt spent the time.
X is finished by the endpoint gets rate limited ๐
Matthew Cassinelli@mattcassinelli
I am in love with scraping using Shortcuts. I have about 8 sets of shortcuts for everyday social media sites that I'm developing in tandem. I'll be releasing them as I finish them โ thread starts here.
English
Diffbot ๐ค ๋ฆฌํธ์ํจ
San Diego developers, join us and our technical partners @neo4j, @Intuit, Eyepop.ai, @Replit , and @diffbot at our HackNight next week!
Unstructured@UnstructuredIO
We're excited to join @neo4j , @Intuit, EyePop.ai, @Replit and @diffbot as technical partners for the upcoming Startup San Diego - FirstWave Innovator HackNight happening Wednesday, February 19th at the Intuit San Diego Campus. This is going to be an epic night where 100+ developers will come together to create innovative solutions for five select startups. ๐ย Join as a developer for free or get tickets: lu.ma/4brcg3lz
English
89,886 developers are building their own Perplexity on-prem with Diffbot LLM โ huggingface.co/diffbot/Llama-โฆ
English
#Perplexity Sonar Pro API launched last week as the best performing model on factuality.
24 hours later, it's the 2nd best performing model (and it's not because of #DeepSeek).
Why? ๐
English
Diffbot ๐ค ๋ฆฌํธ์ํจ
Diffbot launches open-source AI model that achieves 81% accuracy by querying a trillion-fact Knowledge Graph in real-time instead of relying on static training data ๐ง ๐
Read more: venturebeat.com/ai/diffbots-aiโฆ #ArtificialIntelligence #Enterprise #MachineLearning @diffbot
English