Any idea why pages are not being indexed?

[vetofunk]

Hi Everyone,

One section on our website is not being indexed. The product pages are, but not some of the subcategories. These are very old pages, so thought it was strange. Here is an example one one:

https://www.moregems.com/loose-cut-gemstones/prasiolite-loose-gemstones.html

If you take a chunk of text, it is not found in Google. No issues in Bing/Yahoo, only Google. You think it takes a submission to Search Console?

Jeff

[Extima-Christian]

Hey Jeff,

I just ran a quick scan of the site, it looks like you have a lot of links, pages, and directories being blocked by your robots.txt file: https://www.moregems.com/robots.txt

I would make sure the pages you want to be indexed by search engines are not being blocked in your robots.txt.

[NickW816]

Hi Jeff,

In addition to Christian's recommendation (which I would do first), use Google Search Console's Fetch & Render Tool to request your non-indexed pages to Google's index. Sometimes this tool in GSC will have them indexed immediately.

It is not uncommon for deep links or internal pages of internal pages to not be immediately indexed. It is definitely important to use new pages to link our to other pages on your website, and if possible go in and link to your new pages from older (already indexed) pages on your website

[vetofunk]

Hi Christian,

Do you have any resources for a recommended Magento robots.txt file? I added this probably 6-7 years ago, and have not updated it since. I can definitely try that.

Jeff

[vetofunk]

Hi Nicholas,

I asked him this as well, but do you have any resources for a "good" Magento specific robots.txt file? I want to try updating it, as it has been the same for about 7 years.

The strange thing is the deeper product pages are indexed, but not the subcats.

[vetofunk]

So I submitted https://www.moregems.com/loose-cut-gemstones/prasiolite-loose-gemstones.html and fetched it in Google Search Console a few hours ago. Still not being indexed. I don't see any issues in the robots.txt file. Any thoughts?

[Extima-Christian]

I usually do robots.txt for Magento sites custom. But I did find a good example to use. Check out this site:https://www.magikcommerce.com/blog/set-up-robots-txt-in-magento/

I would edit anything that doesn't fit your site.

Hope this helps!

[vetofunk]

Yeah...it's very close to what I have. I also checked other websites I own with the same category structure and robots.txt file...no issues.

I even checked other subcats on www.moregems.com, and no issues. It seems to be all the pages under "GEMSTONES" that are not being indexed. Any thoughts there?

[vetofunk]

So I am testing removing some of the restrictions in the robots.txt file and see if that helps as I still can't get it to be indexed.