Screaming Frog
Screaming Frog SEO Spider is a desktop website crawler used for technical SEO audits. Unlike cloud based tools like Ahrefs or Semrush, Screaming Frog runs locally on your computer and crawls websites the way a search engine would, collecting detailed data about every URL, link, image, script, and directive it encounters. For SEO professionals, it is the most detailed and configurable crawling tool available.
This page covers what Screaming Frog does, how I use it for technical audits, common mistakes, and when it is the right tool for the job.
What It Is and Why It Matters
Screaming Frog crawls websites by following links from a starting URL, just like Googlebot does. As it crawls, it collects data on every element it encounters: page titles, meta descriptions, headings, images, JavaScript rendered content, hreflang tags, canonical URLs, structured data, response codes, redirect chains, and much more.
The result is a comprehensive dataset of your entire website that you can filter, sort, and export for analysis. This is fundamentally different from cloud based SEO tools that crawl the web broadly. Screaming Frog crawls your specific site deeply, finding issues that broader tools miss.
Key capabilities include: crawling up to 500 URLs free (unlimited with a license), JavaScript rendering to see pages as Google sees them, custom extraction using CSS selectors or regex, integration with Google Analytics, Google Search Console, and PageSpeed Insights APIs, XML sitemap generation, and structured data validation.
For technical SEO work, Screaming Frog is the standard tool. While Ahrefs and Semrush offer site audit features, Screaming Frog provides deeper crawl data, more configuration options, and the ability to crawl staging environments and password protected sites that cloud tools cannot access.
Common Use Cases
Common Screaming Frog use cases for SEO.
Technical SEO Audits
Crawling an entire website to identify technical issues: broken links, missing meta tags, duplicate content, redirect chains, orphan pages, and indexability problems. The comprehensive crawl data forms the foundation of any technical SEO audit.
Site Migration Validation
Crawling a site before and after migration to verify that all URLs redirect correctly, no pages are lost, and the new site structure is technically sound. Essential for domain changes, CMS migrations, and major redesigns.
Content Analysis at Scale
Extracting page titles, headings, word counts, and custom content elements across an entire site. Identifying thin content, duplicate titles, missing headings, and content quality patterns that affect SEO performance.
Structured Data Validation
Crawling a site to find and validate structured data markup (JSON-LD, microdata). Identifying pages with missing, broken, or incorrect schema markup that could affect rich snippet eligibility in search results.
Internal Link Analysis
Mapping the internal link structure of a website to identify orphan pages (pages with no internal links), crawl depth issues (important pages too many clicks from the homepage), and link equity distribution problems.
JavaScript Rendering Analysis
Crawling with JavaScript rendering enabled to see pages as Google renders them. Identifying content that only appears after JavaScript execution and verifying that Google can access all important page content.
Practical Experience
I use Screaming Frog for every technical SEO audit. It is typically the first step in understanding a website technical health because it provides the most detailed crawl data of any tool available.
My standard audit workflow starts with a full site crawl using JavaScript rendering enabled. This takes longer but shows pages exactly as Google would see them. I then analyze the crawl data across several dimensions: response codes (looking for 4xx and 5xx errors), redirect chains (more than one hop), page titles and meta descriptions (missing, duplicate, or too long), canonical tags (misconfigured or conflicting), hreflang implementation (for multilingual sites), and internal linking structure.
For site migrations, Screaming Frog is essential. I crawl the old site before migration to create a complete URL inventory, then crawl the new site after migration to verify that every old URL either has content or redirects correctly. The comparison feature makes it straightforward to identify URLs that were missed or redirect incorrectly.
I also use the custom extraction feature extensively. For example, I extract product prices, SKU numbers, or specific content elements using CSS selectors to audit content at scale. This is useful for ecommerce sites where hundreds or thousands of product pages need to be checked for data completeness.
One practical advantage of Screaming Frog over cloud tools is that it can crawl staging environments, localhost, and password protected sites. This allows me to audit a new site before it goes live, catching technical issues before they affect search performance. Cloud based crawlers like Ahrefs Site Audit can only crawl publicly accessible URLs.
Common Mistakes to Avoid
Common Screaming Frog mistakes that reduce audit quality.
Crawling Without JavaScript Rendering
Many modern websites rely on JavaScript to render content. Crawling without JS rendering means you see the raw HTML, not what Google sees. Always enable JavaScript rendering for audits, even though it makes the crawl slower. The data is more accurate and catches issues that a basic crawl would miss.
Not Configuring Crawl Settings
Using default settings for every crawl. Screaming Frog has extensive configuration options: crawl speed, URL filters, custom extraction, robots.txt handling, and more. Configure these based on the specific site and audit objectives to get the most useful data.
Not Saving Crawl Data
Running a crawl, reviewing the results, and then closing without saving. Screaming Frog crawl files can be saved and reopened later. Save crawls to track changes over time and compare pre and post migration or pre and post fix states.
Fixing Every Issue Found
Treating the crawl results as a checklist where every warning must be fixed. Not all issues have equal impact. A broken link on the homepage matters more than a missing alt text on a deep internal page. Prioritize fixes by traffic impact and technical severity.
Crawling Too Aggressively
Setting the crawl speed too high on production websites. This can overload the server and affect site performance for real users. Start with conservative settings (2 to 5 URLs per second) and increase only if the server handles it comfortably.
Frequently Asked Questions
The free version crawls up to 500 URLs per crawl with limited features. The paid license costs 259 dollars per year and removes the URL limit, adds JavaScript rendering, custom extraction, Google API integrations, and many other advanced features. For any serious SEO work, the paid license is essential. The cost is modest compared to cloud based SEO tools.
Screaming Frog provides deeper, more configurable crawl data and runs locally on your machine. Ahrefs Site Audit is cloud based, easier to set up, and provides issue tracking over time. Screaming Frog is better for detailed technical audits and migration work. Ahrefs is better for ongoing monitoring. I use both: Screaming Frog for deep audits and Ahrefs for continuous monitoring.
Yes, but with considerations. Screaming Frog stores crawl data in memory by default, so very large sites (100k+ URLs) may require database storage mode. The paid version supports SQLite storage which handles millions of URLs. Crawl time also increases with site size. For very large sites, plan crawls during off peak hours and use URL filters to focus on specific sections.
JavaScript rendering means the crawler executes JavaScript on each page, just like a browser would. This is important because many modern websites (including those built with React, Next.js, or Angular) generate content with JavaScript. Without rendering, the crawler only sees the initial HTML, which may be empty or incomplete. Google renders JavaScript, so your audit should too.
For most sites, a comprehensive audit every quarter is sufficient. After major changes (site migration, redesign, CMS update, large content additions), run an audit immediately. For very large or frequently updated sites, monthly audits may be appropriate. Between full audits, use the list mode to check specific URL groups or newly published pages.
Need a Technical SEO Audit?
I use Screaming Frog to conduct thorough technical audits that identify and prioritize the issues affecting your search performance.