Welcome to the Q&A Forum

Hi Brian,

If this change is going to be permanent, I would recommend using a 301 redirect. You'll find information on the web elsewhere (even quotes from Google employees) saying that either will pass the same authority, but you should still stick with the best-practices of using a permanent (301) redirect anytime you know the redirect is going to be long-term.

I wouldn't worry too much about losing authority from your TLD, you're clearly not using this redirect for malicious or deceptive reasons. Plus, the brand power of the Smithsonian is such that it would take a LOT of really bad content and/or links to bring that down.

Hope that helps answer your question!

Disallow: /?* is the same thing as Disallow:/?, since the asterisk is a wildcard, both of those disallows prevent any URL that begins with /? from being crawled.

And yes, it is incredibly easy to disallow the wrong thing! The robots.txt tester in Search Console (under the Crawl menu) is very helpful for figuring out what a disallow will catch and what it will let by. I highly recommend testing any new disallows there before releasing them into the wild.

Disallow: /*?

This disallow literally says to crawlers 'if a URL starts with a slash (all URLs) and has a parameter, don't crawl it'. The * is a wildcard that says anything between / and ? is applicable to the disallow.

It's very easy to disallow the wrong this especially in regards to parameters, for this reason I always do these 2 things rather than using robots.txt:

1. Set the purpose of each parameter in Search Console - Go to Crawl > URL Parameters to configure for your site
2. Self-referring canonicals - most people disallow URLs with parameters in robots.txt to prevent indexing, but this only prevents crawling. A self-referring canonical pointing to the root level of that URL will prevent indexing or URLs with parameters.

Hope that's helpful!

Hi Luke,

You are correct that this was done to block URLs with parameters. However, since there's no wildcard (the asterisk) before the folder name, the URL would have to start with /french-wines/. This disallow is really only preventing crawling on the single URL www.yoursite.com/french-wines/ with any parameters appended.

I'd recommend 301 redirecting the old version of the content to its new location on the new sub-domain. That's generally the quickest way to let search engines (and people) know you've relocated important content. Hiding URLs from Search Console is temporary only and not really intended for pointing search engines to relocated content.

Anytime you relocate an entire site's worth of URLs, you should always redirect them to their new counterparts, it's the quickest way to ensure search engines and people are able to access your new content. Updating your URLs in the index can take a very long time without this step.

In addition to the benefits from a UX and SEO point of view already mentioned, you're also destroying your Google Analytics data. When users go from a HTTPS page to a HTTP on your site, the original attribution is lost and that HTTP page will show up as direct traffic. Measurement becomes a nightmare real quick when you have users going back and forth throughout their session on your site.

I took a look at your site and didn't notice any of the obvious answers like nonidex tags or a robots.txt disallow. You do have a massive code-bloat problem though. The amount of code inside your

Link building: you're doing it wrong.

No problem! I just did this a few days ago, so I'm not sure yet. I would imagine that if the URLs aren't linked anywhere or listed on XML, then IF they come back to them, they'll see a 404 response and move on.

Hi Rhys,

The page remove tool in GSC is for temporary removal, but since the pages are actually gone, it's likely they won't come back. I actually just did this a few days ago for the same reason.

Blocking the pages with robots.txt will not remove them from the index. Robots.txt disallows only prevent crawling, not indexing.

Rhys,

Your web dev team is confused. You cannot de-index by simply disallowing them in your robots.txt file. Google will still index anything they find (that doesn't have a noindex tag) from a link, this is the reason you often see search results that say "A description for this result is not available because of this site's robots.txt" as the description.

Here's a quote from Google regarding the subject: "You should not use robots.txt as a means to hide your web pages from Google Search results." - https://support.google.com/webmasters/answer/6062608?hl=en

OK I gotcha now. You can submit the sitemap in all versions of Search Console, won't hurt anything to have it referenced in multiple profiles of SC.

Another thing you can do to make sure crawlers find your XML is add this line to your robots.txt file:

Sitemap: http://yoursitecom/sitemap.xml

Hi John, always glad to help!

For your Search Console question: When you get the redirects setup and have committed to your site being all HTTPS, you'll want to move the location of your XML sitemap to https://yoursite.com/sitemap.xml. As Cyrus mentions in that article, don't update the URLs in the sitemap yet, let search engines hit them as non-secure for a while, I think he recommends 30 days, to give them a chance to learn your new protocol and for them to hit your redirects multiple times.

For your www question: There's no difference in SEO-value whether you choose www or non-www, simply a preference. The only thing that matters here is that you pick one and stick with it.

For your GA question: That is correct, you are seeing traffic from both in GA. GA will collect and report on any page/URL/website that your UA-ID is on. If someone scraped your site and took the GA script with it, you'd start seeing their traffic in your reporting view (that's why appending hostname is always a good idea ). You can specify in the View Settings of GA what your protocol is.

Hi John,

For the most part, your IT partner is correct, 2 of the most important things are to 301 all HTTP requests to HTTPS and to update canonicals. I often refer to people with questions about HTTPS to this post written by Cyrus Shepard, he covers all the bases needed for an SEO-friendly secure migration: https://moz.com/blog/seo-tips-https-ssl.

Regarding your specific comments:

• We serve http and https versions of all our pages - A 301 redirect rule will correct this
• Our canonical tags just refer back to the URL it sits on (apparently a default Drupal thing, which is not much use) - Self-referring canonicals like this serve plenty of purpose, they just need to match your preferred version www/non-www http/https, etc. etc. Self-referring canonicals help prevent duplicates caused by parameters, case-sensitive URLs, and the aformentioned HTTP/S and www/non-www.
• We don’t actually have https properties added in Search Console/GA - You should add another profile for HTTPS, verification should be simple since you've already proven you're the site owner. You want to have both profiles in GSC so you can monitor the shift of indexed URLs from HTTP to HTTPS. Also good for future troubleshooting should you see and issue with indexing of HTTP in the future for some reason.

Hi Tyler,

Yes, remove the robots.txt disallow for that section and add a noindex tag. Noindex is the only sure-fire way to de-index URLs, but the crawlers need to be allowed to crawl those pages to see the tag.

SF finding 'useless' links is actually part of its purpose, if you believe they're useless you should be asking why they're there. Your XML sitemap should have nothing but clean URLs; 200 response codes and not canonicalized to another URL. The problem isn't that you have category URLs, it's that those (like the one in my previous example) have a canonical tag that points elsewhere. Anytime this is the case, the URL is considered un-indexable. You can see the proof of this by doing a Google search for "https://www.interstellarstore.com/meteorite-jewelry/meteorite-necklaces", I just checked and this URL isn't in the index.

You mentioned the age in your original comment that your XML sitemap had been submitted for well over 6 months, that's where I got the age from, maybe I misunderstood?

You have no reason to not trust SF, it's one of the most valuable tools in an SEO's toolbox. I've used it for 5+ years to create hundreds of sitemaps and countless other SEO tasks with no problem in providing reliable, accurate data points.

Hi,

I poked around a bit on your sitemap and noticed a couple things:

1. You've got URLs on there that have canonicals to another page. For example:This page https://www.interstellarstore.com/meteorite-jewelry/meteorite-necklaces has a canonical tag that points here https://www.interstellarstore.com/meteorite-necklaces.
2. A bunch of the URLs in your sitemap redirect elsewhere or have no response - I got 13% through crawling your XML sitemap with Screaming Frog and there were zero 200 response code URLs, not good.

Both of these things combined are causing a discrepancy in the amount of submitted URLs vs. indexed URLs. If you use Screaming Frog to create your XML sitemap it's quite easy to have only clean URLs in there. You can easily remove all URLs that are not 200 status and by default Screaming Frog will exclude any URL that canonicalizes to another URL.

Also, as a side note, you should be updating your XML sitemap more frequently, a 6 month old sitemap for an ecommerce site is far too old with new products being added and products dropping off.

Yes, you should have both active in Search Console, but set the HTTPS to the preferred.

Yea, as the bots hit the URLs on your sitemap, it forces them to step through the redirect, which is what you want. They won't notice the new location if you don't point it out to them, and this is the most efficient way to do so.

*To be clear, since this gets confusing in, the URL of the location of your XML should be HTTPS://thespacecollective.com/sitemap.xml, but the URLs listed in it should be HTTP.

Also, add this line to your robots.txt file, as the first line or last line, doesn't really matter:

Site: https://thespacecollective.com/sitemap.xml