Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
Patrice,
Any time this question comes up, it always raises a larger question of why? What is the purpose of 2 different domains if they contain the same content? It's almost always inadvisable to do this, but there are a few use cases where it's necessary. If you could provide some more details on the why, I can help answer your question a little better.
Alternatively, if you don't have a solid "why" or it doesn't come to you immediately, you might consider scrapping one of the domains 
I agree, unfortunately Moz doesn't have an internal disallow feature that gives you the option to feed them info on where rogerbot can and can't go. I haven't come across any issues with this approach, crawl depth by search engine bots will not be affected since the user-agent is specified.
Hi,
I would advise against this for a couple reasons. First of all, dropping and re-adding pages to the index isn't quick, it often takes weeks before Google will obey a noindex tag. Second, even if it were quick, is going to cause problems at the authority level - is your site a trustworthy source of information? This week, yes, next week, no. It's best to have long-term content that can build trust/authority over time without ephemeral.
An alternative approach you might take is writing content about the particular trials offered, this will help prevent thin content. You might also consider adding a call to action on empty pages that prompt users to provide an email address and you'll notify them when trials of type XYZ have opened back up.
I second what Gaston said. This usage of robots.txt is one of the most common misconceptions in SEO, so don't feel bad. Google actually explicitly says to not use robots.txt for index-prevention in their webmaster guide.
To add to Gaston's point, make sure you remove the robots.txt disallow when you add the meta noindex tag he provided. If you don't let them crawl the page, they won't see the tag.
You can also use remove these URLs temporarily in Search Console by going to the Google Index menu and selecting "Remove URLs". It'll remove from search results, then when they come back to crawl that page again (as long as you're letting them), they'll see your noindex tag and keep it out.
Wanted to jump back in on this topic as I've just confirmed my initial suspicion.
I just added a new client to our Moz account and had the exact same issue, crawler unable to access the robots.txt file. It's a secure site and was configured in Moz without the HTTPS. When I go to the robots.txt file without https://www, it redirects to the same thing as yours where the / between the TLD and page path gets removed.
Reconfigure your site and it should begin to work.
Hi there,
There's something odd going on when I try to access your robots.txt file without the www. The www gets added back on, but when it does, the slash between the TLD and page path gets deleted, see below. I'm guessing your domain in Moz is configured without the www, which means RogerBot is getting redirected to this slash-less version of the file.
Hi Zack,
Have you configured your parameters in Search Console? Looks like you've got your prev/next tags nailed down, so there's not much else you need to do. It's evident to search engines that these types of dupes are not spammy in nature, so you're not running a risk of getting dinged.
Hi,
Patience, young padawan, it's only been a week
It's basically impossible to redesign a website without some organic traffic loss. There's a lot for search engines to consider, everything they previously knew about your website is gone, they have to learn your new site architecture, URL structures, content, meta data, etc. etc.
It sounds like you covered all the most important bases and got rid of some old school tactics. Give it some time, you should start to see a rebound in a few weeks. The worst thing you can do is panic and try to fix a bunch of stuff that isn't actually broken.
Hi,
I wouldn't recommend doing this, it's going to be a lot of work and you probably won't see any benefit. The historical value of those URLs is much stronger than any boost you'd get by taking out a few date folders. It's very common for blogs to have this URL structure, search engines are well aware of this structure and aren't going to ding you for having long or deep folder URLs.
Additionally, since your blog is 10 years old, you've probably got tons of posts, which means tons of 301 redirects. You have to take Google with a grain of salt when they come out and say things like "301s pass all authority now", it doesn't mean you should go out and redirect every page on your site. At some point you're going to begin affecting page speed, particularly on mobile devices.
This question only leads me to followup questions...
Hi Johny,
I recommend only putting schema markup on actual blog posts, and not the category pages. Chances are pretty good that you're not getting much organic traffic to those pages anyway.. There's not a lot of reason for someone to search for something that would land them there, so much so that on every blog I manage I've had /category/ and /tag/ pages deindexed.
However, if you're still wanting to do this, you'd have to set the markup for category pages as a separate tag in GTM and then apply a trigger that only applies to URLs in the category folder.
Rhys,
The self referring canonical on http://www.yoursite.com/some-page will also be applied when parameters are appended to that URL. If your pages are already self-canonicalized, you don't have anything to worry about.
Also, what do you mean by this statement? "All of our pages are automatically self-canonicalised, but I've read that it's best practice to canonicalise all URLs back to the main." Are you saying canonicalize all pages to the homepage??
Hi there,
I haven't heard of using product markup on category pages until reading your question. It's a good idea and while researching I came across this other thread from a couple years ago: https://moz.com/community/q/google-rich-snippets-in-e-commerce-category-pages
My main concern is that most ecomm category pages have a lot of products showing by default, and if you've got a snippet of JSON+LD for each of those, you're going to hurt your page speeds. Just something to be mindful of and monitor if you do decide to go this route.
Hi Nichole,
You're correct in noindexing these pages, they serve little to no value from an SEO perspective. Moz is always going to alert you of noindex tags when they find them since it's such a critical issue if that tag shows up in unexpected places. If you want to remove these issues from your crawl report, add the following directive to your robots.txt file, this will prevent Moz from crawling these URLs and therefore reporting on them:
User-agent: rogerbot
Disallow: /tag/
Disallow: /category/
*edit - do not prevent all user-agents from crawling these URLs, as it will prevent search engines from seeing your noindex tag, they can't obey what they aren't permitted to see. If you want, once all tag & category pages have been removed from the index, you can update your robots.txt to remove the rogerbot directive and add the disallows for tag & category to the * user agent.
Hi,
I'd definitely recommend keeping users on the product detail page to read reviews, for both structured data and UX reasons.
You need to have reviews on the page if you want the rating to show up in search results, there might be a way around this but I wouldn't trust it in the long-term.
From a UX perspective, if people can easily read the reviews on the same page, they'll have a lot less reason to do a search for "Product A reviews" and you're not distracting them by sending them to a different page. People have very short attention spans, so keep them on the product detail page, they shouldn't have to navigate back to the product in order to complete a purchase.
Having reviews built into the page also helps different your pages from each other (in the event you have two very similar products) as well as the competition.