Every once in a while you should check your website for crawl errors / duplicate pages / 404 page missing errors / redirects that go to dead links etc. Luckily there’s a great tool from Screaming Frog that makes this process quite easy.
Screaming Frog SEO Spider
Screaming Frog SEO Spider has a light free version (Mac/Linux/PC) that will crawl 500 pages of your site for free.
Here’s a video overview of its features:
Some of its features include:
- Errors – Client & server errors (No responses, 4XX, 5XX)
- Redirects – (3XX, permanent or temporary)
- External Links – All followed links and their subsequent status codes
- URI Issues – Non ASCII characters, underscores, uppercase characters, dynamic uris, long over 115 characters
- Duplicate Pages – Hash value / MD5checksums lookup for pages with duplicate content
- Inlinks – All pages linking to a URI
- Outlinks – All pages a URI links out to
- Anchor Text – All link text. Alt text from images with links
- Follow & Nofollow – At link level (true/false)
- Images – All URIs with the image link & all images from a given page. Images over 100kb, missing alt text, alt text over 100 characters
- User-Agent Switcher – Crawl as Googlebot, Bingbot, or Yahoo! Slurp
- XML Sitemap Generator – You can create a basic XML sitemap using the SEO spider
Benefits of Upgrading to Screaming Frog Pro
- The 500 URI crawl limit is removed.
- You can unlock all the configuration options.
- Save and re-upload your crawls
- You can search for anything in the source code of a site (custom source code search feature).
Other Sitemap / Site Integrity Checking Options
Google Webmaster Tools and Bing Webmaster Tools will give you much of the same information for free. You just have to submit your website to them by following the above links.
XML sitemaps can be automatically generated for the search engine bots if you’re using WordPress by using plugins like Yoast SEO and Google XML Sitemaps.
Scrutiny and Weblight for On Page SEO
Scrutiny is a very similar “webmasters toolbox” that compares favorably to Sreaming Frog but is available specifically for the Mac.
- Integrity is a free tool specifically to help find your website’s broken links. You can set per-site delays for scanning and also scan for missing images.
- Reactivity is designed to test your site’s load speed.
- Scrunity does link checking, onpage SEO, HTML validation, XML sitemap generation and web page speed tests. In the SEO section there is info on page title, url, description, keywords, on page headings.
- Scrutiny will keep track of as many sites as you like, and remember the settings for each. (Integrity doesn’t have this feature).
- Scrutiny will let you save a set of settings – you can simply double-cilck that resulting settings file to start Scrutiny running with those settings and even schedule that settings file using iCal.
- You can check multiple sites with a single crawl, which is useful for those owning multiple sites.
- The actual link checking is very fast – the backlinks Scrutiny returns are actually internal links and it will also show 404 error internal links.
It’s $55 and includes a trial (5 full scans free). It’s Mac only. Weblight is another alternative to Screaming Frog that is available for $33 and is fully cross platform.