Why Google Crawled but Currently Not Indexed Your Pages-How to Fix it
(cover image: Magnifying glass over the word Google color blue, red, yellow, and green lettering)
Hi Wealthy Affiliates,
If you are like me, you are puzzled as to why you see pages under Crawled-Not Currently Indexed in Google Search Console. When you click that, you can see that your posts are still nowhere to be found in the search results. It can be frustrating when your valuable content remains hidden from potential visitors.
"Crawling" is when search engines scan and analyze web page content using automated bots. These bots follow links, collect information, and constantly update web pages. When a page is crawled, its content is indexed and organized by Google for quick retrieval during user searches.
Let's discuss why this might happen and explore solutions to get your pages in the spotlight.
What do Crawling and Indexing Mean?
Have you ever wondered how search engines like Google find and organize information online? It all starts with crawling and indexing, which is how the search engine discovers and stores web pages on its servers. That process is essential for Google to provide relevant and accurate results when users enter a query in the search bar.
1. Crawling: Google's first step is crawling. Googlebot, an automated software program, visits websites to discover new or updated content by crawling through a list of known URLs, including popular sites, submitted sitemaps, and links found on previously crawled pages.
Googlebot follows links from one page to another and collects information about those pages. It then sends this information back to Google's servers for indexing.
2. Indexing: After crawling a website, Googlebot sends the collected information to be indexed. Indexing is the process of storing and organizing this information in Google's index or database. Think of it as organizing books in a library—without proper indexing, finding what you need would be chaotic.
Ensure
When your website is crawled but not indexed, it means that Google knows about your pages but hasn't deemed them worthy of being included in its index yet. This could be due to poor-quality content or technical issues hindering proper indexing. That is why you must constantly check Google Search Console; it will look like this.
Reasons Why Google May Not Index Your Pages After You Requested Indexing
You got your post indexed, but later, Google decided to de-index your articles. Why does that happen? As you can see, there are various reasons why Google may not index your pages even after you have requested it.
- We are not focusing on links with /feed. What does /feed mean? It means that WordPress automatically creates these for blog posts. They're like individual RSS feeds. Those are not indexed, but they are still part of your site. You do nothing- there is nothing to fix when you see that; I am telling you right now, ignore it.
See this screenshot example.
- We are looking for blog posts that have been indexed, but then later, Google decided to show those posts as crawled but not currently indexed. It will show your permalink as an example: yourdomianname.com/org/net /google-article-not-inde
- We need to check those pages to ensure that Google crawls all the relevant pages on your site efficiently.
There are various reasons why Google might not index your pages even after you've requested them:
- Technical Issues: Google may have difficulty crawling your site due to slow loading speed, broken links, or a lack of quality content and relevant keywords.
- Low-Quality Thin Content: If your website has low-quality or thin content, Google may not index it. Ensure your content is original, high-quality, and relevant to your audience.
- Incorrect Robots.txt File: The robots.txt file tells search engines which pages or sections of your site they should crawl and which ones they shouldn't. If this file is set up incorrectly, it can prevent Google from indexing certain pages on your site.
- Broken Links and Faulty Redirects: Google won't index pages with broken links caused by URL changes or removed content. Faulty redirects can send users to incorrect pages and prevent Google from indexing your site.
Google might not index your pages after you request them. Therefore, it's essential to regularly check for technical issues, monitor the quality of your content, and ensure that your site meets Google's guidelines.
- Slow-Loading for Mobile Speeds: Remember, Google uses mobile page speed as a ranking factor. If your site is slow to load on mobile, it may not get indexed. Check your site's mobile loading speed using Google's PageSpeed Insights tool and make necessary optimizations.
- Improper Use of Noindex Tag: The noindex tag, like the robots.txt file, tells search engines not to index a specific page. Google will exclude them from search results if used accidentally on essential pages.
- Canonicalization Issues: “When multiple versions of the same page exist (e.g., with different URLs), Google may struggle to determine which one to index, potentially leading to duplicate content issues and causing some pages to be de-indexed by Google.
- Server Issues: Server errors or downtime can prevent Google from correctly crawling and indexing your site. Ensure your server is functioning correctly and there are no redirects or page loading speed issues.
- Domain Age and Authority: New websites with little authority may struggle to get indexed by Google. It takes time to build trust with search engines and establish yourself as a reliable source of information.
With patience and consistent effort, you can improve your site's chances of being indexed by Google and increase its visibility in search engine results.
How to Check and Fix Your Crawled-But Not Indexed Pages
So, you've noticed that Google has crawled your website, but for some reason, it's not indexing your pages. Don't panic just yet - there are steps you can take to troubleshoot this issue.
Step 1: Check if Google has crawled your website: You can type "site:yourwebsite.com" into the Google search bar. This will show you all the pages Google has indexed for your website.
Step 2: Pages are missing from the indexed list: Technical issues may prevent them from being crawled and indexed. Some common problems listed above may cause issues.
Step 3: If none of these issues seem to be causing the problem, You can use Google Search Console to request indexing again for specific URLs. This will prompt Google to recrawl and hopefully index those pages.
Step 4: Another potential issue could be a slow site speed: Google tends to prioritize faster-loading pages for indexing, so if your site takes too long to load, it may hinder the indexing process. Use a tool such as Google's PageSpeed Insights to analyze your site's speed and make improvements if necessary.
Step 5: Make sure your website has a sitemap. A sitemap is a file that lists all the pages on your website and helps search engines understand its structure. Creating and submitting a sitemap to Google can make it easier for them to crawl and index your pages.
Step 6: Check for duplicate content: If you have multiple pages with the same or similar content, Google may not index all of them. Make sure each page on your site has unique, quality content.
Step 7: Monitor for broken links: If Google encounters too many broken links while crawling your site, it may give up and not index certain pages. Regularly check for broken links on your website and fix them promptly.
Please do the following.
- Ensure that your pages' content is unique and valuable. Google prioritizes high-quality content when indexing websites. Next, check for any technical issues blocking Google from indexing your pages correctly.
- Use Google Search Console to request re-indexing of specific pages. This can prompt Google to revisit and hopefully index them accordingly. Additionally, ensure no meta robots tags or directives instructing search engines not to index certain pages.
- Monitor your website's indexing status regularly to catch any discrepancies early on. By staying proactive and addressing any issues promptly, you can improve your chances of getting your crawled but not indexed pages back in the game.
Review and fix any pages that have been crawled but not indexed. Check for technical issues, ensure unique and valuable content, request re-indexing through Google Search Console, monitor for broken links and duplicate content, and regularly check your website's indexing status. If the problem persists, consider seeking help from an SEO specialist or asking for help in the WA community.
Final Thoughts
I used to have articles de-indexed, but I improved the content, and Google indexed them. Monitoring your website's indexing status is crucial. Check "Crawled-Not Currently Indexed" regularly. It's important to troubleshoot and improve your content for indexing. If it is not indexed, consider removing or using it for internal linking.
PS. I tried hard to make it short, but it's detailed information. Have you dealt with issues like this? Tell me what you did to solve those de-indexed posts.
Recent Comments
76
That's a good question. We see both of them there. They look similar, which can be confusing for some people.
1. Crawled-currently not indexed means that Google crawled the page. It is possible the page was indexed later but decided not to index it, which means it cannot be in the search engine results. This we worry about except for /feed
2. Discovered-currently not indexed means Google knows the URL but has decided not to crawl it yet after requesting indexing. This we don't worry about-give it some time and patience wait for Google to crawl the URLs
The problem is that Google does not clarify why a page gets this crawled but not currently indexed status. We know these pages often receive this status when Google considers it doesn't reach a quality threshold to get indexed. It's as simple as that. We have to follow the steps I mentioned in the post to figure out how to improve the page and get it re-indexed again.
I had a similar post in March when I had the same questions. Google Update was going and I couldn't stop watching... Even though I was told not to watch.
Discovered - Currently Not Indexed or Crawled - Currently Not Indexed
Yes, I remember reading your article about that. I believe Partha told us not to watch, not especially when we have Google Core updates going on. That's a bad idea. I avoided looking there until way after the last May and June updates. So, your answer is pretty much the same as mine. The bad part is Google never tells you what to look for. You have to end up guessing what it is: thin content, is it helpful content, server issues, wrong links, and many other things to look at. It can be cumbersome for some people. Also, we need to emember that not every article will be indexed; some won't make it.
for a new Dutch site of mine, I don't even use Google search console anymore, I 'hide' from Google with that one, so it can't have a site-wide classifyer ever again (meaning that my whole site got punished for a number of 'bad quality posts').
For new sites, I wouldn't rely on Google anymore. When you don't connect to Google, it won't mean that you won't rank in Google. I just measure my traffic with an independent traffic plugin, away from Google.
Please don't get caught up in all these things too much, I waisted too much time as well, and most of us (niche sites) got either hit in rankings with a domain that had a bit higher authority, but not deindexed (my mini site), and my Africa site with low DA got not just hit, but almost over 50 posts deindexed. Frustrating. I checked again I think about a month ago, and they were all indexed again, besides a few review posts (yeah Google now hates reviews of products you haven't used, it's clear).
Again, don't waste too much time in trying to be 'perfect' for your blog posts, because there isn't always a correlation between the 2 and you are putting a LOT of time on it, while the next week your posts can get deindexed yet again, if Google decides to. Trust me, I've been there. And you will burn up even more and frustration will rise.
Things 'might' change again with the next core update (September, October?), but really the focus now is: keep blogging and of course 'quality' (but what Google considers quality these days isn't clear, people are stuck on outdated SEO), don't write for Google, write for your audience!
BUT most of all, and I said this before and others said this too (in one of Jays classes as well recently), get those social media links and signs girl!
If google sees that your site is mentioned everywhere else, they will notice! (I hope lol)
And if they don't, at least I wont have waisted any time no more and build up my socials, medium, and I will be working on my forum, more tools, and newsletter. That's more consistent traffic.
The only things you need to correct if google tells you so, are errors, as Google warned me that one of my plugins was down and I needed to switch. (well, they didn't say that literally, but that's what happened, so they deindexed some pages at the time of errors)
I bet you were checking every day to see if they got indexed again, etc etc?
And that killed your day and your mood to go on, no? Just don't..
Lizzy
Yeah, you are right. I remember you talking about that a while back. Yes, that's what you mentioned about "parasite" SEO using user-generated content on those SM sites the other day. I did not focus so much on fixing some posts that got de-indexed because I knew I wouldn't be able to revive some of them. I left it there.
I just focus on answering questions and solving problems that I see in unanswered user-generated content. That is the formula that works. My DA is only two and has few backlinks, yet I got one post between Reddit and Quora. It's still there on page 1. I know it's working. I'm pivoting to find a different way. You found a way that works for you, just like I have.
I am aware that posting articles in these places is vital for backlinks. I understand precisely what you are talking about. I am not like others who write reviews that don't have first-hand experience; All my reviews are from using the products myself. Im finding answers and solutions to their problems like Partha said to do. I know he told us to focus on these UGC sites. I am trying to find someone at Fiver to help me handle these technical issues for YouTube. I got a few recommendations from members here.
I agree with you about how frustrating SEO is because of Google. I don't write for Google. I never have. I always wrote for my audience. People who want to build a thriving online business, and that's what I am doing. I found many gold nuggets that are not being answered on Reddit, which I know you hate, but I go there to find questions that people gripe about, and most don't offer solutions; they provide more upvotes to get more attention. I look for content that I can write about related to the niche of building an online business. There are many ways to create an online business. Yes, that is true. The more your name is out there, the better, as you said about TikTok and YouTube Shorts.
I never knew this Brenda.
Thank you so much for taking the time to explain to us.
Bux
PS - I like strawberry cake too ...
Hi Bux, sure. Help yourself. Yes, it is essential to check Google Search Console on that page because many of mine went there. I had to work on improving it so I could get those articles re-indexed and it worked.
Cool. We call it Cool Whip here in USA. My Mom would add that, not me, not wild about having cream on my cake. Too sweet for me. The strawberries look good. Yum!
GRR, Michael! My name isn't Glenda; my parents decided not to name me that. It's B R E N D A ! There's a big difference.
Building backlinks naturally is challenging and takes a long time. It used to be much easier five years ago.
Today, I notice that getting them naturally from higher domain sites is much harder, so it is best to use "parasite SEO." Get those backlinks from social media sites; they are high-domain user-generated content, so it's important to get your content on those UGC sites so you can have a backlink to your website. Google looks at those, too.
Don't use the black hat style when buying low-quality backlinks from sources like Fivver or elsewhere. Sure, there are plenty of white hat link-building SEO agencies, but 99 percent of the time, they are very costly, such as The Hoth, Diggity Marketing, and other link-building SEO agencies. They charge a lot of money.
I find doing outreach to secure backlinks or guest posts very tough right now, all because of Google core updates and helpful content updates, and then going after sites that aren't following the rules and guidelines. I haven't done it; I did five years ago, but not this time. It's a lot of work, and most of the emails will be ignored. I think building relationships with people helps. It does take time though. My site hardly has any backlinks. Most of it is from a few from Pinterest and other SM sites.
Okay Brenda... 😉 sorry about that mistake. I have fixed it.
Yeah I have found it tough to get backlinks too.
Michael.
I am glad you spelled my name right. Thank you. Yes, getting backlinks the natural way is tough these days.
See more comments
Great info here as always Brenda!
Don't get me started on posts getting de-indexed though!!
I've got about 80 on my newest site!! They're getting de-indexed faster than I can do a quick update and submit for indexing again!
Plus, one can only request indexing through GSC on 10 different posts a day!!
Nevermind though.... life goes on!! :-)
I feel the same as you do. I realized that Google now is very picky now. It seems like they are only ranking higher domain some are junk these are sites that don't even belong in the search engine results. I know how you feel about this. Did you know their are paid services like Omega which I use for indexing its worth it GSC has its limits.
Interesting Brenda.....I have never heard about Omega before!!
I will check it out my friend! :-)
Sure thing. There are other instant indexing services, but Omega is the one I know about. Another paid service similar to Omega that has excellent results is Index Me Now Version 4. (The price is not bad, but it's USD, not Euro.) Index Me Now should not be confused with Bing's IndexNow. I learned about this from SEO YouTubers.
Appreciate the additional info here Brenda!
It matters not if the pricing is in USD or Euros, I always pay for the cost my credit card gets hit with (if I choose Euros though, it's normally a better deal) ... but not always!!!
Go figure!!
🤣👍
Ok, I understand now.
👍👍😁😊