SiteOne Crawler

crawler.siteone.io - I am creating this OSS tool with the goal of making it a real Swiss Army knife for developers, testers or website owners. I believe that it will gradually reach more people around the world :) The more useful it is for the most people around the world, the greater the motivation for its development.

What’s an OSS project / contributor that deserves more recognition and support?

@ghdotxyz @ghdotxyz At this moment it is not possible to convert the entire website to Markdown. However, we want to release a new version in 2-3 weeks that will be able to provide a fully functional and linked version of the entire website, including all subpages, in Markdown.

@siteone_crawler also if I wanted to convert a crawled site into markdown pages how would I go about it? I don't really understand the export folder structure.

v1.0.8 Update: 🚀 Enhanced redirect following for initial URLs, improved 404 detection, and URL skipping behavior. Export/clone websites better on modern JS frameworks. Plus many edge-case fixes & optimizations! Full changelog github.com/janreges/siteo… #webdev #opensource #webaudit

SiteOne Crawler is featured on @10words_io today! Help support us by voting at 10words.io 🙏

@philhawksworth Thanks for this older tip. I recommend using SiteOne Crawler instead of wget - github.com/janreges/siteo… .. at the weekend I will release version 1.0.8 where the export is even more tuned based on netlify.com export. Run this command and then open tmp/netlify.com/index.html. I believe that you will be pleasantly surprised how the whole website works even offline 😉 Alternatively, use the desktop application found at crawler.siteone.io (it's just a wrapper over the CLI tool) #TechTips #WebDevelopment

@saran_pariyar1 We consider it essential that a quality open-source project contains quality documentation. We don't have it perfect yet, but we're working on it 😊 github.com/janreges/siteo…

.@siteone_crawler is a desktop app & command-line tool #madewithsvelte that will go through your website and give you an in-depth report about SEO, security, accessibility & other best practices 🤓⚙️ - madewithsvelte.com/siteone-crawler

@MadeWithSvelte 🙏 I will be very happy if you all try this tool and give us more ideas of what it could do. Our long-term goal is to create a free tool that will help improve the quality of websites around the world. As a desktop application and a tool for CI/CD pipelines or command-line usage.

@alfloyd71 @akshara_dev @sveltejs @discord Thanks man, we do this for people and improving the quality of websites around the world. I will be very grateful if you tweet about this tool with information about how the tool helped you and possibly what other features you would appreciate ;-)

@siteone_crawler @akshara_dev @sveltejs @discord very cool app you have

found this tool in @sveltejs community server in @discord. pretty good one/ crawler.siteone.io

@alfloyd71 @akshara_dev @sveltejs @discord Hi @alfloyd71. If you are using the command-line, use the `--http-cache-dir=''` (set empty string) setting to disable the HTTP cache. If you are using a desktop application, uncheck `Request caching`, see video youtu.be/rFW8LNEVNdw?si… We will add it to the documentation.

I am currently encountering an issue with the HTML Report Page for SiteOne Crawler, which does not seem to update even after I've addressed certain issues on my website. I'm curious to know if others have experienced a similar issue and if there might be any overlooked cache settings causing this discrepancy.

@sveltejs I love @sveltejs and its simplicity! After I created the useful command-line website analyzer and exporter, I managed to create also a desktop application in a few evenings thanks to Svelte and Electron. Try it, it's free ;-) crawler.siteone.io

Today, 12/12/2023, we managed to release the desktop app v1.0.6 for macOS (arm64 and x64), which is already signed and notarized by Apple. Installing the DMG should be problem-free😉 github.com/janreges/siteo… #webdevelopment #tools #SEO #performance #security #Apple #MSFT #Linux

Dear friends, we have released the first public version of our SiteOne Crawler tool. Both desktop and command-line. We believe that it will be a quality helper for many website owners or Dev/DevOps in improving the quality of their work. Try it out! :-) crawler.siteone.io