Scrapy

👀

“scrapy” became my alter ego. it all started with my first ever python project, a simple web scraper that pulled headlines from top news sites. i used the scrapy library by @ScrapyProject. it wasn’t much, just a script. but it was the first thing i built that actually worked. and that made it special. later, i realized something funny: most of my ideas felt like fragments, messy, raw, not yet stitched together. so the word “scrapy” stuck. i even launched a few nft projects under the name “ScrapyNFTs”, none of them took off (my art was pretty scrapy too lol). nfts introduced me to web3, which led me to explore blockchain… and eventually evolve into @scrapychain.

10 years after the release of 1.0, Scrapy has become table stakes for modern web scraping projects. Creator Shane Evans talks through Scrapy's current position in the industry. #ScrapyAnniversary #OpenSource #WebScraping

Read the full editorial about the rise of Scrapy 👉 zyte.com/blog/the-rise-…

Before powering massive scraping operations globally, #Scrapy started as a workaround at a startup that needed structured data fast. @zytedata CEO and Scrapy creator Shane Evans takes a look at those early days of Scrapy 👇 #ScrapyAnniversary #WebScraping

Scrapy didn’t start in a lab. It started in a startup. From solving real-world data problems to becoming one of the most trusted scraping frameworks out there—here’s how it all began. 👉 zyte.com/blog/the-rise-… #ScrapyAnniversary #OpenSource #Python

Scrapy 2.8.0 is out. This is a maintenance release, with minor features, bug fixes, and cleanups. See the full changelog at #scrapy-2-8-0-2023-02-02" target="_blank" rel="nofollow noopener">docs.scrapy.org/en/latest/news…

@talenodigital Yes, you can download the results in CSV format as well. Available formats: #serialization-formats" target="_blank" rel="nofollow noopener">docs.scrapy.org/en/latest/topi…

@ScrapyProject Hello @ScrapyProject , is there a way to download the result as csv from the script as well?

So I asked from the @anvil_works community if it's possible to make my idea of web scraping marketplace using @ScrapyProject . Told them I'm doing a #buildinpublic 🫰

@talenodigital Try switching from CrawlerProcess to CrawlerRunner and see if that helps. For more details: #scrapy.crawler.CrawlerRunner" target="_blank" rel="nofollow noopener">doc.scrapy.org/en/latest/topi…

@rblxstd @anvil_works @ScrapyProject Still a work in progress. I actually have an issue that's why I asked help from @anvil_works community (image above)

Have you got what it takes to be called a web scraping whizz? Join the @ExtractSummit 2022 Coding Contest to find out!

Dive into the world of web data! Register for Extract Summit.

Scrapy 2.6.2 and 1.8.3 are out, addressing a security issue: #scrapy-2-6-2-2022-07-25" target="_blank" rel="nofollow noopener">docs.scrapy.org/en/2.6/news.ht… 2.6.2 also addresses the remaining regressions of 2.6.0.

Scrapy 2.6.1 is out, with security fixes around cookies also backported to Scrapy 1.8.2. But it is not just security fixes! There is official Python 3.10 support, asyncio support no longer being experimental, and new features for feed exports: #scrapy-2-6-1-2022-03-01" target="_blank" rel="nofollow noopener">docs.scrapy.org/en/latest/news…

Have you not tried Playwright and Scrapy together yet? Get started in 10 minutes! youtube.com/watch?v=0wO7K-… Thank you, @jhnwr!!!

@hatemjaber Thanks for reaching out to support. Please provide me with more information on which project that you are looking for?

Medusa: A multi-headed @torproject HTTP proxy. Plays nicely with @ScrapyProject. datawookie.dev/blog/2021/10/m…

We have released Scrapy 2.5.1 and Scrapy 1.8.1, which fix a security issue that could lead to the leak of HTTP authentication credentials to target websites. Find out more in the release notes: #scrapy-2-5-1-2021-10-05" target="_blank" rel="nofollow noopener">docs.scrapy.org/en/2.5/news.ht…

An introduction to Scrapy from @bsekiewicz , winner of the Extract Summit 2021 Coding Contest: worksmarter.pl/en/post/webscr…

Ready!

Join Extract Summit for a day dedicated to web data extraction.

There are prizes for the first 3 people with all data correctly scraped. The solutions must run on Scrapy Cloud. #scrapy #scraping #Python