Welcome to the Q&A Forum

With that many I would suggest redirecting them to a relevant page rather than just stopping the indexing of them by submitting the links to Google Webmaster Tools. From what I've experienced, keeping the link juice flowing through your site by redirecting them is better for your overall SEO efforts.

Of course it's faster to submit the links to GWT…but that doesn't necessarily mean it's better. Regardless of what you do or how you do it, eliminating your crawl errors is very important.

There's tons of documentation stating that 404's negatively affect SEO. It's definitely debatable and there are obviously other factors involved. My main point is that it's important to deal with any and all crawl errors.

Hello!

Although 404's will eventually stop being crawled by Google, having too many at once can negatively affect your rankings. The  most important thing is that you do not want to be linking to these 404s anywhere from within your site. If so, you want to definitely remove those links.

If I have one or two 404s in my crawl errors, I typically will just leave them be and wait for them to be dropped out of being indexed. Some other solutions I've utilized are:

1. Make an engaging 404 page so that when users find the page they will be encouraged to stay on the website. Having a search box or some of the most popular links on the page is a good place to start

2. 301 redirect the pages to relevant pages that do exist. This will help your link juice flow and will make for a good user experience since they are reaching a relevant page.

Hope that helps!

I see your issue with that…

I'd suggest moving the H1 title and the section it titles up, at the very least. Maybe under that slider? It's not good practice to bury your H1 at the bottom of your page.

Hi Pete

H tags are used to show hierarchy so I would suggest changing your first H2 tag at the top of the page to an H1. After that, you can break up your content  with H2s, 3s, etc. Google will look for a H1 tag to see what the page content is about so it is important to have the keyword you are targeting within that.

Clearly organizing your design with heading tags hierarchy will make it more user friendly and will also help search engines better understand your content. Think about it like you are looking at a newspaper… the H1 gives you the title, H2 will be the supportive subtitles, and so forth

Hope that helps!

Hey David

Once you publish a page/post in WordPress and submit a sitemap, you are stuck with those pages. I've experienced this problem a lot as I use WordPress often. Once you trash a page there and delete it permanently, it's not stored anywhere in the WordPress CMS. They are just reading as 404s since they existed and now no longer exist.

As stated above, just make sure you are not linking to your trashed page anywhere in your site.

I've done a couple things with 404 Pages on my WordPress sites:

1. Make an awesome 404 page so that people will stay on the site if they found your 404 page on accident. Google will eventually stop crawling 404s so this is a good temporary way to engage users.

2. 301 Redirect the 404s to relevant pages. This helps keep your link juice and also helps with the user experience (since they are reaching a relevant page)

Hope that helps!

Thanks David

I already had the redirect set up in the htacces file. I think the site was crawled before I had added this to the file…total "occam's razor" moment

But now there is a few thousand duplicates because of the /catalogue /search etc… I disallowed that all in the robots file so now just have to wait another week!

This is a great question! I'm curious to see what the Moz Staff response will be.

I was going to say what Monica had said…that if a person is logged in to Google while searching it will skew the results. However I have found that when someone is logged in it tends to skew in their favor, so I was surprised to hear that their result was page 2. Very interesting…

Thanks for posting!

Hello!

I would definitely suggest disallowing these pages from being indexed in your robots file. These 120 pages will be considered duplicate content, and will most likely will also be duplicate titles and meta descriptions. Duplicates of any kind is a huge SEO problem.

Once you edit your robots file, you will not need to do anything in your webmaster tools. The site will be crawled according to your robots file.

Another idea… depending on your URL structure, you may be able to use canonical tags to stop multiple pages that are seen as duplicates from being crawled. That way you won't have to completely stop the indexing of these pages by disallowing them through the robots file.

Hope that helps!

Well, if it were one of my clients sites… I wouldn't do that. While I understand your logic with a noindex, I wouldn't want to create a situation where the pages would not be about to be found at all in search engines. Although it will drop your duplicate content numbers here on Moz, it's only a temporary fix. I guess a good question to explore is how long you will need to keep them as a noindex versus how long it would take to fix the content issues.

Hey Ross!

Those pages are not "different" when it comes to search engines. Or maybe I should say, not different enough. The content is extremely thin and only switching out a word or two will absolutely make them come up as duplicate content. I would strongly suggest optimizing the page content and meta descriptions to be unique.

Hello!

Index/Follow means that you are allowing search engines to index your pages and follow links through those pages. If you didn't have this tag on the page, search engines would do it anyway. Basically, this meta tag isn't necessarily a bad thing and the search engines would be doing it even if the tag wasn't there.

You can use meta robots to do good things for your site, such as stop the crawling of blog archive pages to limit duplicate content. There's tons of resources here on MOZ about the robots file and I would suggest looking into it.  This is a good one: http://moz.com/learn/seo/robotstxt

What type of problems is your site having?

Hello Carl!

So I checked out the pages you listed and I've had similar issues on my e-Commerce stores. There isn't much text on e-Commerce site pages and there tends to be a ton of links so that always causes a problem for me. E-commerce stores and duplicate content go hand in hand, unfortunately.

I would suggest starting with adding canonical tags into your meta data. There's a few settings in Magento you can turn on and that should take care of some of the problem. Here's a good resource http://www.magentocommerce.com/knowledge-base/entry/canonical-meta-tag

From there you might want to consider making your meta descriptions on the products a bit more unique. Changing out one word (the product name) doesn't make it different or a non-duplicate. When the content is super thin, it's harder to make the pages, titles, and descriptions unique to search engines. With e-Commerce product pages, I understand the trouble with having text on filter pages…it's just not practical and doesn't look right. But it's important to optimize where you can…the meta descriptions. Here's another resource for that http://moz.com/ugc/our-forgotten-friend-the-meta-description

Hope that helps!

Hey Carl

I have this problem a lot on my Magento stores since theres a ton of ways to access the same product and it can be an SEO nightmare. If you have FTP access I would recommend adding this to the robots.txt file:

Disallow: /wishlist/

This will stop the bots from crawling everything passed /wishlist/. Here's a good resource if you need further assistance:  http://www.magentocommerce.com/knowledge-base/entry/setting-up-a-robotstxt-file

Hopefully that helps!

Good luck

Hello!

The pages that will no longer be available after the site is migrated will go directly to a 404. In doing similar site migrations, I have seen that the 404s will eventually not be crawled by search engines. I will typically make a really good 404 page in order to reduce bounce rate and improve click through.

On the other hand, If there are a ton of pages that are not being moved, I would suggest doing 301 redirects. Too many crawl errors (404s) will hurt search engine ranking. If you are redirecting, they should go to a relevant/closely related page.

Good luck

I recently installed an SSL certificate on the site: https://libertywholesalesupply.com

Moz is now reading thousands of duplicate content pages because it is reading both http and https. I set up the configuration in Magento to auto-redirect the base URL, created a permanent redirect for the URL in the SEO settings, and adjusted the canonical settings.

What am I missing??

How can I remove the duplicate page content in my Magento store from being read as duplicate.

I added the Magento robots file that i have used on many stores and it keeps giving us errors.

Also we have enabled the canonical links in magento admin

I am getting 3616 errors and can't seem to get around it .. any suggestions?

Hey Everyone!

I have an old site with hundreds of blog posts that are very spammy (duplicate content, keyword stuffed, and just plain bad content). I am going to redirect them and delete them from WordPress but I'm wondering where is the best place to redirect them to? Home page, other posts, other pages...?

Any thoughts would be appreciated!

Thanks!

Any idea on what this does or how to use it: _cdn="<a rel="nofollow" target="_blank">true</a>"

Trying to find info but can't seem to find anything!

Awesome!

Thank you!