We Found A Blind Spot In SEO Rank Tracking

Key Takeaways

• First Rank found a systemic issue affecting commonly used keyword tracking tools.
• We have found the source of this systemic issue; no other agency has publicly released the same finding.
• We are using this information to improve keyword tracking for all of our clients in the future. This is a massive advantage; our clients will see more accurate keyword tracking faster than our competitors’ clients.

First Rank recently identified a blind spot in the way most SEO agencies measure Google rankings.

While investigating unusual ranking volatility in September, we found that automated rank-tracking tools were sometimes being shown a substantially different type of search results page than the one a real local customer would see for the same query.

That matters because SEO reporting across the industry relies heavily on those automated searches.

We believe we’ve identified an important reason why some ranking reports have recently stopped matching customer-facing search visibility. More importantly, we’re already working on new ways to measure rankings that account for the problem.

For our clients, that’s good news.

Rather than simply reporting a sudden “ranking loss” because a dashboard says one occurred, we can investigate whether visibility actually changed, identify when the measurement itself is the problem, and build reporting around what prospective customers are really seeing.

Thanks to our discovery, our clients will soon see better keyword tracking than what is currently available on the market.

Keyword Trackers Don’t Search Google Like You Do

Keyword tracking tools work by automatically performing Google searches and recording where a website appears.

Modern trackers can simulate important characteristics of a search. They can tell Google that a search is coming from Winnipeg, for example, or that it’s being performed on a mobile device.

But Google has far more information available when an actual person searches.

Depending on the user’s settings and available data, Google may consider their current location, device, language, recent searches, previous activity, and information associated with their Google account.

A tracker can simulate “someone searching from Winnipeg.”

That’s not necessarily identical to “an actual Winnipeg resident searching for a local service.”

Until recently, that distinction usually wasn’t large enough to cause major reporting problems.

Our testing suggests that, for many searches, it now can be.

We Found The Same Search Producing Different Types Of Results

Consider a high-intent query like “manitoba solar.”

Here are the results from a popular keyword tracking tool:

And here are the results from a human Google user:

See the difference? In the first screenshot – the one using an automated tool – the results are largely informational (including a Facebook group); the second features what we would expect from the query: Web pages from companies that offer solar panel installation.

Someone searching “manitoba solar” is likely to be looking for a solar installer in Manitoba; they want to purchase a product or service. We’d therefore expect Google to prominently show local companies and commercially relevant service pages.

During our investigation, however, automated searches sometimes produced a very different SERP: Informational articles, directories, and other results were much more prominent, while relevant commercial pages appeared considerably lower.

When we repeated the same query under different circumstances (as human users), the commercial results we were expecting appeared.

This helped explain another behaviour we’d been seeing inside our ranking reports.

A client’s service page might rank extremely well one day. The tracker might then suddenly report an informational article from the same website much farther down the rankings.

Normally, that kind of URL switching can look like keyword cannibalization, which occurs when two pages on a site are competing to rank for the same keyword.

But if Google has actually interpreted the automated search as having different intent, the pages aren’t necessarily fighting each other at all. For these automated searches, an informational blog post might outperform the commercial page.

How We Figured This Out

The first signs appeared around September 17, 2026, when our rank-tracking data suddenly began behaving in ways that didn’t match what we were seeing elsewhere.

Rather than assume thousands of rankings had genuinely fallen at once, we audited the data.

We analyzed 7,621 tracked keyword records across many client projects. Keyword.com was showing 3,100 keywords down by at least five positions and another 360 showing gains of five positions or more.

We then carried out 2,950 live Google searches to investigate those movements, comparing the results with historical Keyword.com rankings and Google Search Console data.

For each keyword, we established a baseline using its median Keyword.com ranking from September 3-16. We then investigated movements of five positions or more recorded on September 27.

The live searches matched the same geographic settings used by Keyword.com and requested the top 50 organic Google results. Where results were inconsistent, we repeated the searches rather than treating one SERP as definitive.

That turned out to be important.

In one test of 60 keywords, 12 produced a materially different SERP result on the second search. Across the wider audit, we found examples where the same search from the same city could place a client at #1 on one request and fail to return it minutes later.

Once we applied repeated checks and compared the results with historical and Search Console data, the picture changed dramatically.

Of the 3,100 apparent ranking drops, only 180 were confirmed as real. The audit classified 2,733 as tracking errors, with the remainder unstable or not yet verifiable.

The historical data told the same story. Among 4,349 keywords that had ranked consistently from September 3-16, 3,188 later disappeared from Keyword.com’s results and then returned to their normal positions.

That was the point where this stopped looking like ordinary ranking volatility.

The evidence indicated that, in many cases, the measurement was changing more dramatically than the actual search visibility.

Our next question became much more interesting: What was causing automated searches to receive such different results from the ones we expected real customers to see?

That’s what led us to investigate how Google interprets search intent, user context, location, and automated searches – and why conventional rank trackers may no longer be capturing the full picture.

This Can Affect Google Search Console Too

You might assume Google’s own Search Console would settle the question; after all, it’s information coming directly from Google.

Here’s the problem: Google says Search Console does not necessarily filter known bots and spiders from its data.

That means automated searches can potentially contribute impressions. If those automated searches are served unusual results where a client’s commercial page appears much lower, they may also pull down the average position reported for that query.

The same issue can affect click-through rate because CTR is calculated from clicks and impressions.

That doesn’t make Search Console unreliable as a whole.

Clicks remain enormously valuable. Organic traffic matters. Impression trends can still tell us a great deal. And, most importantly, leads and conversions tell us whether search visibility is producing actual business.

But it does mean that average position should not automatically be treated as an objective measurement of where every real customer sees a website.

This Is Why We’re Changing How We Report Rankings

Keyword positioning remains an important part of how we measure search performance. Rankings can help us identify changes in visibility, understand which pages Google is favouring, and spot opportunities or problems that deserve a closer look.

Right now, however, the changes we’re seeing in how Google serves results mean that a single automated ranking measurement may not always tell the whole story.

Because keyword positioning is so important, we’re doing more testing on:

• Organic clicks and traffic
• Impression trends
• The pages receiving search traffic
• Repeated SERP testing
• Manual search validation
• Leads and conversions

For most clients we’ve investigated, we have not found the dramatic loss of real-world search visibility that the rank trackers initially appeared to show.

We’re Building A Better Way To Measure Search Visibility

This is the part we’re excited about.

Keyword tracking has always tried to turn a complicated, personalized search environment into one clean number: You rank #1, #4, or #17.

Search is becoming more contextual than that.

So instead of trying to force Google’s current search environment back into an old measurement model, First Rank is exploring ways to measure it more accurately.

That includes testing searches across multiple user-like profiles and contexts, comparing repeated searches instead of relying on a single snapshot, and validating ranking data against actual customer behaviour.

We don’t consider the work finished yet. There are still questions we’re actively testing, including how frequently different SERP states appear and exactly which contextual signals are responsible.

But finding the problem is what allows us to solve it.

SEO reporting should not exist to produce a nice-looking graph. It should tell you whether your business is visible to the people searching for what you sell.

If the tools stop answering that question accurately, our job is not to trust the tools. It’s to build a better measurement. That’s exactly what we’re doing.