Have you been trying to fix Crawled Currently not Indexed on your site?
This is another common issue on search console.
Since its roll-out, we use it almost daily at godzgeneralblog.com to diagnose technical issues at scale for our clients.
Within the report, there are many different “statuses” that provide webmasters with information about how Google is handling their site content such as Insufficient HTTPS Coverage on your site. While many of the statuses provide some context around Google’s crawling and indexation decisions, one remains unclear: “Crawled — currently not indexed”.
Google’s Index Coverage report is absolutely fantastic because it gives SEOs clearer insights into Google’s crawling and indexing decisions.
What is crawled – Currently not indexed?
Contents
If you’ve submitted a URL to Google Search Console and got the message Crawled – Currently Not Indexed, it means Google has crawled the page but chose to not index it. As a result, the URL won’t appear in search results for the time being.
According to Google’s official documentation, this status means: “The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.”
How to Fix Crawled – Currently not Indexed Error
The key to understanding this status is to think of reasons why Google would “consciously” decide against indexation. We know that Google isn’t having trouble finding the page, but for some reason it feels users wouldn’t benefit from finding it.
Then you should know this three factors;
Google is able to access the page
Google took time to crawl the page
After crawling, Google decided not to include it in the index
All this decision is made by google.
Do you now see it is different from discovered currently not indexed on your site.
This can be quite frustrating, as you might not know why your content isn’t getting indexed. Below I’ll detail some of the most common Crawled – Currently not Indexed Error .
Most Common Crawled – Currently not Indexed Error
RSS feed URLs
This is one of the most common examples that we see. If your site utilizes an RSS feed, you might be finding URLs appearing in Google’s “Crawled — currently not indexed” report. Many times these URLs will have the “/feed/” string appended to the end.
Google finding these RSS feed URLs linked from the primary page. They’ll often be linked to using a “rel=alternate” element. WordPress plugins such as Yoast can automatically generate these URLs.
We advice you to Do nothing! You’re good.
Google is likely selectively choosing not to index these URLs, and for good reason. If you navigate to an RSS feed URL
While this XML document is useful for RSS feeds, there’s no need for Google to include it in the index. This would provide a very poor experience as the content is not meant for users.
Expired products
When spot-checking individual pages that are listed in the report, a common problem we see across clients is URLs that contain text noting “expired” or “out of stock” products. Especially on e-commerce sites, it appears that Google checks to see the availability of a particular product. If it determines that a product is not available, it proceeds to exclude that product from the index.
This makes sense from a UX perspective as Google might not want to include content in the index that users aren’t able to purchase.
However, if these products are actually available on your site, this could result in a lot of missed SEO opportunity. By excluding the pages from the index, your content isn’t given a chance to rank at all.
It appears that Google is taking clues from both the visible content and structured data about a particular product’s availability. Thus, it’s important that you check both the content and schema.
Check your inventory availability.
If you’re finding products that are actually available getting listed in this report, you’ll want to check all of your products that may be incorrectly listed as unavailable.
301 redirects
One interesting example we’ve seen appear under this status is destination URLs of redirected pages. Often, we’ll see that Google is crawling the destination URL but not including it in the index. However, upon looking at the SERP, we find that Google is indexing a redirecting URL. Since the redirecting URL is the one indexed, the destination URL is thrown into the “Crawled — currently not indexed” report. You can read our guide on how to create 301 Redirect on WordPress.
Create a temporary sitemap.xml.
If this is occurring on a large number of URLs, it is worth taking steps to send stronger consolidation signals to Google. This issue could indicate that Google isn’t recognizing your redirects in a timely manner, leading to unconsolidated content signals.
One option might be setting up a “temporary sitemap”. This is a sitemap that you can create to expedite the crawling of these redirected URLs.
To create one, you will need to reverse-engineer redirects that you have created in the past:
Export all of the URLs from the “Crawled — currently not indexed” report.
Match them up in Excel with redirects that have been previously set up.
Find all of the redirects that have a destination URL in the “Crawled — currently not indexed” bucket.
Create a static sitemap.xml of these URLs with Screaming Frog.
Upload the sitemap and monitor the “Crawled — currently not indexed” report in Search Console.
The goal here is for Google to crawl the URLs in the temporary sitemap.xml more frequently than it otherwise would have. This will lead to faster consolidation of these redirects.
Thin content
Sometimes we see URLs included in this report that are extremely thin on content. These pages may have all of the technical elements set up correctly and may even be properly internally linked to, however, when Google runs into these URLs, there is very little actual content on the page.
Add more content or adjust indexation signals.
Next steps will depend on how important it is for you to index these pages.
If you believe that the page should definitely be included in the index, consider adding additional content. This will help Google see the page as providing a better experience to users.
If indexation is unnecessary for the content you’re finding, the bigger question becomes whether or not you should take the additional steps to strongly signal that this content shouldn’t be indexed. The “Crawled —currently not indexed” report is indicating that the content is eligible to appear in Google’s index, but Google is electing not to include it.
However, when using a site search operator, we can see that the URL is actually included in Google’s index. You can do this by appending the text “site:” before the URL.
If you’re seeing URLs reported under this status, I recommend starting by using the site search operator to determine whether the URL is indexed or not. Sometimes, these turn out to be false positives.
If anything, make sure that you don’t do anything to impact the crawling of the individual pagination. Ensure that all of your pagination contains a self-referential canonical tag and is free of any “nofollow” tags. This pagination acts as an avenue for Google to crawl other key pages on your site so you’ll definitely want Google to continue crawling it.
Duplicate content
When evaluating this exclusion across a large number of clients, this is the highest priority we’ve seen. If Google sees your content as duplicate, it may crawl the content but elect not to include it in the index. This is one of the ways that Google avoids SERP duplication.
By removing duplicate content from the index, Google ensures that users have a larger variety of unique pages to interact with. Sometimes the report will label these URLs with a “Duplicate” status (“Duplicate, Google chose different canonical than user”). However, this is not always the case.
This is a high priority issue, especially on a lot of e-commerce sites. Key pages such as product description pages often include the same or similar product descriptions as many other results across the Web. If Google recognizes these as too similar to other pages internally or externally, it might exclude them from the index all together.
Add unique elements to the duplicate content.
If you think that this situation applies to your site, here’s how you test for it:
Take a snippet of the potential duplicate text and paste it into Google.
In the SERP URL, append the following string to the end: “&num=100”. This will show you the top 100 results.
Use your browser’s “Find” function to see if your result appears in the top 100 results. If it doesn’t, your result might be getting filtered out of the index.
Go back to the SERP URL and append the following string to the end: “&filter=0”. This should show you Google’s unfiltered result (thanks, Patrick Stox, for the tip).
Use the “Find” function to search for your URL. If you see your page now appearing, this is a good indication that your content is getting filtered out of the index.
False positives
Our first step is to always perform a few spot checks of URLs flagged in the “Crawled — currently not indexed” section for indexation. It’s not uncommon to find URLs that are getting reported as excluded but turn out to be in Google’s index after all.
when using a site search operator, we can see that the URL is actually included in Google’s index. You can do this by appending the text “site:” before the URL.
If you have any issue on this, you can use our comment session or join our delegate on Facebook to solve any issues related to WordPress and subscribe to our YouTube Channel for WordPress video tutorials.