Welcome to the Q&A Forum

It's not a good or bad thing if you have 27,000 pages that Google has indexed on your site. What's important is, how many of your pages has Google indexed, how many of those are duplicates, and how many of those are unimportant pages?

To find how many of your pages Google has indexed:

1. Make sure that you have an XML sitemap that includes EVERY PAGE on your site.
2. Submit that to Google Search Console.
3. Check back in a day or so, and Google Search Console will show you how many of the pages on your XML sitemap are in their index.

If you only have 5,000 pages on your site, but Google shows that you have 27,000, then you're probably dealing with duplicate pages, or tags that are built by your site and have gotten out of control.

To find out if Google has indexed duplicate versions of your pages:

1. Use Google Analytics' Search Channel report (Acquisition All Traffic Channels Organic, then change the Dimension to Landing Page) to download a list of all pages on your site Google or Bing has sent visitors to.
2. Create a new column, stripping out portions of the URL that can change without changing the content of the page. Some common offenders:

• Parameters
• Different categories earlier on in the URL
• www vs non-www
• https vs http

4. Now, find the duplicates in the "unique" column. I like using the countif function for this.
5. For any page that's got duplicates, find a way to fix those! It varies by the problem, but the answer is generally to 301 redirect all versions to one canonical version.

To find out if you have too many tags, because they're autogenerated

1. Use an internal tool, like an XML sitemap creator, to figure out how many tags you have
2. Compare that to the number of tags actually bringing you traffic
3. Check to see if you're creating tag pages that are the same content as your category pages
4. Check to see how many tag pages you have with fewer than five products matching - those probably aren't great pages!

Going through these steps should let you know if your site is well optimized - again, it's not the total count that matters!

Good luck!

Kristina

Hi Jcobo,

A good way to check to see if Google understands your content is to use Google Search Console and fetch the page as Googlebot. Look at Googlebot's version of the page - does it see the content in the table?

Like Logopedia said, HTML tables are extremely SEO friendly, so I'd be surprised if Google can't read your content.

On the flip side, tables are often not particularly user friendly. Why do you have a page full of your products? How is that valuable for a visitor? If Google doesn't think a page is user friendly, it won't rank it well.

Hope this helps steer you in the right direction!

Kristina

Hi Rose,

Hopefully Donna answered your question already, but I want to jump in with some SEO prioritization advice.

Alt text like this can add to the relevance of the page, but minimally. It can also help your image rank correctly in image search, but that doesn't bring much traffic now that Google pulls images into its results page.

I had similar conversations with our compliance team when I worked for a university, and they had a similar perspective, that alt text should be determined by the flow of the reader rather than for small SEO boosts. The nice thing is, though, when images are important to the flow of the page, and are more likely for the alt text to support the keywords you're trying to target on a page.

In short: if I were you, I'd let this argument go, and just push for alt text on images that tell a story. There's no SEO penalty for not using alt text, and I doubt you're worried about ranking for "father and young son."

Best,

Kristina

Whoa, this is a weird one.

I saw that you posted this on Google's forums as well, and they suggested that this might be the Japanese keyword hack. Did you look into that? If that's not it, did you try loading the URLs that are showing up on the Wayback Machine? It's possible that someone who owned this site before your client created these pages.

Either way, the answer is to double check that your 404 pages really are 404ing. If that doesn't remove them from the index fast enough, you can actually create all of those pages, with a noindex tag, add them all to a sitemap, and submit them to Google. But the 404ing is really your long term solution.

Good luck!

Hi Nancy,

To be clear, when you say that there were 15,000 blocked pages, do you mean that Google reported that its crawler was blocked from crawling them? Or that Google was blocking them itself, thinking that they're malware or something like that?

The term "blocked" makes me think there's a technical issue here, not a penalty. Were these 15,000 pages important pages? I see that Google still has around 1,950 of your sites' pages in its index. How many should it have?

Best,

Kristina

Hi Emanuele,

Wow, that's really weird! A few questions:

• On the days where your site stops ranking for keywords, have you done a site: search on those pages to see if Google has them in its index?
• What is your competition for these keywords? Have they been changing around a lot too, or is everything else staying about the same rank, with your site just jumping in and out of the list?
• Have you been making any changes to the site during this time? Even if you didn't change the specific page in question, it's possible that changing the link structure of the site is drastically changing their ability to rank.

Best,

Kristina

Hi Eroc,

Yep, this is a super common issue for SEOs. My company's site has reviews load in JavaScript, and Google even crawls and indexes those as if they're separate pages.

Google wants to read everything it can, so you've got to figure out the cost/benefit of this situation. If you really do have all of the product information, I'd guess that visitors entering through page 2 of reviews wouldn't be the worst thing, especially since Google would probably choose the page with the reviews most relevant to the query.

The main issue from my perspective is wasting Google's crawl budget on your extra review pages. If that's an issue that will significantly help your site (likely if you have a lot of products), you'll want to noindex those reviews like Ikea. There's no other method that will stop Google from wasting it's crawl budget.

Hope this helps!

Kristina

Hi Kavit,

The short answer is no. Google can render some JS - possibly even AngularJS - so never assume that something rendered in JS is invisible to Google. You should assume that Google can see all links visitors can, and really push for a nofollow tag.

I usually check what Google can render by loading Google's cache of the page (go to Google.com and type in "cache:" in front of the exact URL of one of your pages). Look at the text-only version of the cache, and see if Google puts a link there. If they do, it's safe to assume that they can see that link. Another option is to use GSC to Fetch as Google; Google claims this is exactly what they're seeing.

If both the cache and GSC show that Google can't see a link, Google's probably not crawling it. But, Google's always getting better, and could suddenly see the links any day now. If these links are really a concern to you, I'd strongly suggest that you push your dev team to add nofollow tags to these outgoing links.

Best,

Kristina

I've responded to specific comments below, but here's the information you need to redirect: https://developers.google.com/webmasters/smartphone-sites/details

The gist is, you need:

• 301 or 302 redirects from the desktop site to the mobile site, based on user agent detection
• Rel="canonical" tag on each mobile page pointing to to each desktop page
• Rel="alternate" tag on each desktop page pointing to each mobile page
• A Vary HTTP header
• A mobile sitemap

As I mentioned below, I also recommend that you have a link at the bottom of the mobile and desktop sites pointing to the other version. That way, if you get the redirects wrong, people can self correct.

Hi Emanuele,

To clarify:

• When you say you've verified it with a site: search, do you mean that you verified that the pages that were no longer ranking were still indexed? Or, do you mean that you verified that the site itself was still indexed? I'm wondering if you're only losing certain pages.
• I'm not sure what you meant by your response about your competition. Are you saying that the competition doesn't fluctuate for the keywords that your site is fluctuating for?

Thanks,

Kristina

That's true, they append parameters tracking where you came from, which looks like it can affect the navigation you're seeing on the left. They're making sure that Google doesn't get confused by using a canonical on their pages, like Mike, Eric and I have recommended.

Weird. So, that means that this isn't because of problems with Google's crawler not finding your content, and it's not that you are ranking for keywords that just fluctuate all the time anyway. My next guess would be that this is a poorly applied penalty of some sort. A couple of questions to see if it is:

• During the times where you're not ranking for most keywords, does your homepage rank for your brand name?
• Have you gotten any other messages in Google Webmaster Tools / Search Console since the original "Improved search visibility" that you mentioned in your original question?

Best,

Kristina

Hello there!

This isn't a question that we can answer for you definitively, but I think I can help you figure out the best method here.

(Before we start, I want to clear something up: if you use a # in your URLs, Google only indexes the page as it's loaded with the URL before the #. So, if you make the page title change for t-shirts/#/?=1&filter.style=vintage_, Google won't even see that.)

To restate your question, and make sure I'm understanding you correctly: you're migrating a smaller site into a larger site. Your options are:

1. The standard option, redirect pages to the most similar page on the bigger site
2. A (slightly sneaky) SEO option, where you redirect pages to pages that are similar enough for good user experience, but are pushing link equity to the pages that need it.

My answer is: go with option #1 unless you can prove that option #2 will have an impact on rankings for the right pages.

To determine if you can make an impact, map out which pages you'd like to redirect where. For simplicity sake, assume that all inbound links to oldsite.com/vintage-tshirts and oldsite.com/hipster-tshirts will transfer full equity to biggersite.com/tshirts. Add those links to the current count of inbound links to biggersite.com/tshirts. Then, go through the top keywords you want to rank for, and see if this new number of links will cause your rankings to increase. (I like to use the Mozbar and actually search for whatever keyword I'm targeting, and look at the number of inbound links and the total DA of competing sites.)

Be sure to look for potential increases in traffic if you send oldsite.com/vintage-tshirts to biggersite.com/vintage-tshirts too - typically, you get more conversions for pages that are already better targeted to the query.

There's definitely some wiggle room here. If you think that the new inbound links are significant but they won't actually get you up a ranking, go ahead and go with option #2. If you find that you don't have a whole lot of inbound links to oldsite.com/vintage-tshirts, though, this probably isn't worth you time.

Hope this helps!

Kristina

Yep, they're a little vague here! But the answer is: Google will crawl your site at whatever rate it wants (it's probably crawling Amazon 24/7), unless you limit how much it can crawl in Google Webmaster Tools. Then, Google will crawl your site at whatever rate it wants, unless than rate is higher than the limit you put in, and then it will limit itself.

If you're anxious for Google to crawl your site more because a) you have something that's changed and you want Google to have it in their index, or b) because you're hoping it'll affect your rankings:

a) If there's specific information that you want Google to update its index with, submit the URL of the page that's new or changed into "Fetch as Googlebot" and then, once you fetch it, hit the "Submit to index" button to the right. I work on a site that's a DA 58 and fetching something as Googlebot updates the index within an hour.

b) How much Google crawls your site has to do with how important your site is; forcing Google to crawl your site more will not make it think your site is more important.

Hope this helps!

Kristina

To be clear, Googlebot will find those external links and leave your site, but not until they've used their crawl budget up on your site.

Nope, having more or less external links will have no effect on the crawl budget spent on your site. Your crawl budget is only spent on yourdomain.com. I just meant that, Google's crawler will follow external links, but it won't until it's spent its crawl budget on your site.

Google isn't going to give you a metric showing your crawl budget, but you can assume how much it is by going to Google Search Console > Crawl > Crawl Stats. That'll show you how many pages Googlebot has crawled per day.

GSC is very finicky, I'd make sure:

• You're using the www subdomain, if that's how traffic typically gets to your site
• Google has domain.com.au/us/ indexed (check by searching for site:domain.com.au/us/ to see all of the pages Google has in its index)
• This subfolder gets traffic (check for organic traffic in Google Analytics)

Hope this helps!

Kristina

Ah, gotcha. Personally, I use Google itself to find out if something is indexable: if it's my own site, I can use Fetch as Google, and the robots.txt tester; if it's another site, you can search for "site:[URL]" to see if Google's indexed it.

I think this tool could be really good if you keep it as an icon and it glows or something if you've accidentally deindexed the page? Then it's helping you proactively.

Hope this helps!

Kristina

You can definitely build your new mobile site with a different structure than your old OScommerce site, just make sure that your desktop site has an alternate tag pointing to your new site. I'd also recommend adding canonicals to your old mobile URLs pointing to the new versions; that'll allow you to keep the old mobile site alive, but stop Google from showing it in its index.

Hope this helps!

Kristina

Hi Ioannis,

You're in quite a bind here, without a good URL structure! I don't think there's any one perfect option, but I think all of these will work:

• Create a page on your site that links to every article you would like to delete, keeping those articles 404/410ed. Then, use the Fetch as Googlebot tool, and ask Google to crawl the page plus all of its links. This will get Google to quickly crawl all of those pages, see that they're gone, and remove them from their index. Keep in mind that if you just use a 404, Google may keep the page around for a bit to make sure you didn't just mess up. As Eric said, a 410 is more of a sure thing.
• Create an XML sitemap of those deleted articles, and have Google crawl it. Yes, this will create errors in GSC, but errors in GSC mean that they're concerned you've made a mistake, not that they're necessarily penalizing you. Just mark those guys as fixed and take the sitemap down once Google's crawled it.
• 410 these pages, remove all internal links to them (use a tool like Screaming Frog to make sure you didn't miss any links!), and remove them from your sitemap. That'll distance you from that old, crappy content, and Google will slowly realize that it's been removed as it checks in on its old pages. This is probably the least satisfying option, but it's an option that'll get the job done eventually.

Hope this helps! Let us know what you decide to do.

Best,

Kristina