Crawl Central

The weekend is finally upon us! Have a good couple of days off everyone!

Do we release on a Friday?! Of course we release on a Friday!

Distributed Crawls also have great fault tolerance if you've architected the platform correctly, allowing you to cope with failure without having to start from scratch again.

Happy friday folks. Hopefully you've had a great week!

What is your favourite programming language and why? We enjoy Java and other JVM languages, they provide the most cross platform support, but we're also big fans of Python 😍 What about if your webcrawl was language agnostic?

Sometimes its not about crawling the competitors, its about crawling your own sites and making the results available to your own staff. If you're a company with many intranet sites Webcrawling meets Enterprise Search.

How do you productionalize a basic BeautifulSoup web crawl campaign? Get in touch, we can help!

Proxy support Multiple browser support Header rotation User interaction emulation We have the bases covered. Get in touch for crawl help.

We'll have a hosted Crawl Central demo coming soon! We're looking forward to all the feedback. In the mean time, if you need crawls or support, get in contact and we'll be happy to help.

Did you know a lot of criminals hide their criminality in plain sight? This is how #DARPA leveraged web crawling to help track down criminal groups and organizations.

Taking Selenium Grid 4 for a spin. How cool is the new UI and the various updates? Also the new Helm chart! buff.ly/3wNMZRj

Question for all our crawl nerds. What cool use cases do you use webscrapes and web data extraction for?

We also wrote a second scripting language for Selenium, slightly different and a work in progress. This one is called magnesium script, feel free to give it a go and get involved! buff.ly/3x7WvyX

If you want to nerd out over crawling strategies our @jrz813 is your boy!

What do you use to extract information after a Crawl? We're huge @ApacheTika fans over here!

Morning all! 😵

Want to know more about how to build scalable crawls? Get in touch!

Price analysis & competitor analysis projects can be greatly simplified using Selenium to interact with the target sites. Give Selenium Scripter a whirl: buff.ly/3iQwrA0

What's you're favourite cloud data service and why? Can be anything and doesn't have to be super complicated. We'll start - S3, easy storage of Crawl results for post processing and analysis.

Happy Monday!