Highlight links internal

Posted on by

Browseo is a wonderful tool for browsing pages through the eyes of a search engine. When it comes to analyzing site structure and other factors that pertain to an entire website, Screaming Frog is the tool of choice for many to use alongside Browseo. Both tools together are a perfect match.

Besides Browseo, Screaming Frog is one of the most powerful tools a technical SEO has in his toolbox. That said, the recent additions to the software have made it a "must have" for every SEO. Out of necessity I built a similar spider back in the 90's so I know how hard it is to build a spider to evaluate site architecture and structure. The new features take "the Frog" from being a mainstay in the SEO toolbox to a tool that webmasters should be looking at to monitor site health and stay ahead of what can become very large numbers, very fast, in Google Webmaster Tools.

The Sitemap Generator

Yeah! I know you are thinking "big deal there are a bunch of free tools that do that very well!". So webmasters generate it and adjust as content is added. Yes that insures indexing but does it enable you to identify new content or changed content? Likely not. Whereas Screaming Frog includes the last modified date from the header and from that you can learn:

  1. what is new on the site
  2. what has been changed
  3. when these events occured

Last modified is a very important value in the header as Google for sure uses it to partly determine what is fresh and when indexing refreshes should occur. Since Google has been known to favor freshness a few SEO's and spameisters have made their last modified dynamic. Sorry folks, Google is not a fencepost so I would be very, very careful with that stuff. So we have this XML sitemap with last modified how do we use this to our advantage?

Simple! Open Excel import the xml file and then sort (descending) on the last modified field. Voila! At the top of the list are the new and changed files! I suggest a weekly indexing and saving your last indexing for comparison. Again, When your client screws up and potentially tanks their rankings by say .... I don't know... setting the "discourage Search Engines..." setting in WordPress...  Don't laugh I've seen it happen more than once! With weekly indexing you may even be able to catch it before the Search Engines do.


This feature is amazing for testing how spiders react! Testing robots.txt and .htaccess is now as simple as adding the list of URLs you want to test into Screaming Frog! The first thing I do for site migrations or redevelopments is index the site just before the migration/changes take place. That way I have lists to work from for my robots.txt and .htaccess files. Once those are written and the development/migration is complete I use the list feature in Screaming Frog to test the links, robots.txt and redirections.


This is very cool because you can get an idea in real seconds which makes sense whereas Google tool uses some cockameme grading thing. However the Google tool is very useful in figuring out exactly what page elements are responsible for the slow load speed. I like to make decisions on "real numbers" not some formulas. How about you?

So, what do you do when you see a need for speed? Use Screaming Frog to identify the problematic pages then you run the pages that need improvements through the Google tool to see the best way to improve the load speed. This saves you time and enables you to make more informed decisions about what pages to implement changes on because they are based on real numbers that you can understand and know exactly how it affects the user experience.


There is lots of other new features in Screaming Frog I strongly suggest you learn what data it collects and as you go through that the ways that it can benefit you will become endless. That and the the never ending additions to this program like Video and Media sitemaps means this will be a tool you'll rely on for many years so learning what it does and how will make you that much better as an SEO or webmaster!


Partner, SEO Training Dojo.   Terry Van Horne has been developing and marketing websites since the early 90's in various marketing and development positions including working as internet marketing manager for one of Canada's largest real estate developers; SEO for an award winning real estate search engine and marketing manager for ecommerce stores in the apparel and musical instruments. In 2007 he developed a YouTube Marketing Strategy and to date those 300+ videos have received over 26,000,000+ downloads . He is currently a partner with David Harry in Reliable SEO, a Search and Social Media Marketing Agency, and the award winning SEO Training Dojo, a learning community as well as three other marketing and industry news sites. Terry is Director Of SEO Dojo Radio and co-hosts 2 hour long podcasts "Search Geeks Speak," an hour with luminaries and up and comers in Search Engine Marketing as well as "The Regulators," an hour of Industry News. Terry founded SeoPros.org, the first consumer advocacy organization for purchasers of search engine optimization and is currently a Director of the NFP organization OSEOP that grew out of it. Follow him on Twitter.

2 responses to “Tool Review: 3 Tips for Using Screaming Frog’s New Features”

  1. Ryan Dozier says:

    I’ve just discovered this tool and already can see the benefits it has.

  2. Doris says:

    I think it is not a disadvantage if you pay $ 99 a year for this extensive software. Other professional SEO tools cost 50, 100 or more $ every month and the Screamingfrog offers for this price incredibly many ways.