Frogs, Spiders and Crawling Your Website: A Primer on Screaming Frog SEO Spider

In the modern internet, websites, blogs and ecommerce platforms are all getting larger. Some of these sites are bloating up due to feature rich plugins and some are simply adding more video content.

Below is an example from Gigaom that tracks the difference in Kb across all file types.

As more and more scripts get pushed into these websites it becomes necessary to empower the teams behind these web properties to examine the site in it’s entirety. Considering how many pages the average ecommerce site has this would be an enormous task to undertake without the proper tools.

This is where crawlers come in, the same kind of technology that established the prominent search engine is now available to everyday users as a desktop application.

There are many alternatives to Screaming Frog, and the primary use for this application is usually in the field of SEO, but the low yearly cost and free test drive edition make this software a fan favorite.

Out of box the tool will spider through the internal links and sitemap of anything that you throw at it blazingly fast. It will list the address, the meta data like character sets, titles, and keywords. The primary reason I like this software however is the ability it has to create custom filters that will scour your page’s HTML to whatever specifications you prefer.

Below is an example of some custom searches you can set up in seconds - the spider will read all of the HTML on the pages that it scans and provide a list of each URL it occurs on and how many times it occurs.

As you can see in the above example the creation process is easy, simply specify whether you are looking for something that contains or does not contain a string in the HTML.

The above example will highlight any occurrence of “out of stock” as well as a Google Analytics tracking code. This is an especially useful way to ensure that you have tagged each page in your site with Google Analytics, beyond that it can help ensure that each tag is only firing once.

Being able to physically audit EVERY page on your site allows brands and agencies to accurately determine if a large scale implementation was carried out thoroughly or if some page types have missed the fastidious eyes of their developers.

Just to remind the reader, this tool offers a lot of utility beyond checking tags and code implementations.

Here are a list of some other eye catching features:

1. Audit redirects during large site migrations

2. Crawl path reporting to show how pages are connected and uncover broken links

3. Title and description editing through the SERP emulator with device emulation

4. Check for canonical errors (these are related to duplicate content)

5. Collect information about link anchor text

6. Find images missing alt text

7. Competitive keyword research

8. Prospect link building opportunities

9. Create an XML sitemap

10. Find flash and PDF content and where it is located

If your company hasn’t moved all of its tags to a tag container like Google Tag Manager or Tealium then you should strongly consider Screaming Frog SEO Spider to help evaluate what is on site and how consistently it is implemented.