Welcome to the Q&A Forum

In that case you don't need a robots.txt file at all. Your example above is actually very close to restricting access to your entire site.  If you had a robots.txt that looked like this: User-agent: *  Disallow: / You would stop any bot or visitor from seeing the site.  Definitely not recommended! In your case, I would avoid having a robots.txt file altogether.  It's not an SEO requirement until you need to restrict access for whatever reason.

It's a trick question, right?  Only one of them would rank and the others would be at the bottom of the results because they were duplicates.  The one that would rank would probably be the one with the best links to it / highest authority If you're asking if they were all exact copies of each other with the same back links, which URL might rank higher than the others simply by virtue of it domain name, it would be a crap shoot but Moosa may be right.

Francis, I'd guess that you've changed or removed the analytics script from some pages.  Had you done any work on the site during that time?

I think the answer depends on why you are wanting them gone.  If you have been given a manual penalty and have a warning message in your Webmaster Tools then you need to try to contact each of these sites to get them removed and then disavow the rest.  If you don't have a manual penalty and are trying to avoid an algorithmic issue like Penguin then disavowing these is enough. For those urls you showed above, none of them resolve so they probably don't need to be dealt with. If you have a bunch of spam links that are not created by you then other than disavowing them you really shouldn't have to worry about them.  Check out this webmaster hangout: http://youtu.be/BAcEz_-ujCw?t=9m30s.  At the 9:30 mark John Mueller talks about a case where someone is concerned that a competitor is pointing thousands of spammy links at their site and John's answer is not to worry about them because the algorithm is already ignoring them.

Do not play with Robots as it may block out series of pages and folders out of index Correct command as stated by Lesley is /cats/ . Refer official documentation https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

Remove the products.. it adds no value. What the article is describing is if you have collisions. Like say you have a "page" in wordpress call "ipod" and you also have a product called "ipod" When you remove the category, everything will be like yoursite.com/ipod so if you have a product named the exact name as a page you will have a collision. Just ensure that you do not have this. SEO ultimate plugin can do this, I use it on my blog for the very same thing with no problems. Or you can manually do the rewrites in the htaccess file. You absolute best bet since you are not dealing with a lot of products, it sounds like it is a small site, is do it in the middle of the night and do a spider from MOZ and see if it picks up errors.

I also wanted to put in a little update that we've noticed in this niche particularly.  When we follow the basic outline of the strategy above, we notice that the onsite article gets ranked quickly, while the target landing page drops off out of results for about half a day.  No later than the next day is the landing page back in place and the onsite (blog article if you will) is removed from rankings. I suppose this is just the Google shuffle or something similar happening.  Fortunately we've removed all but 8 errors (out of 3000) and we're just waiting for GWT to update to reflect that information.

Agree with John.  I would also not 301 the old URLs to the new.  All that would do is pass the negative link equity from the old site to the new and you are right back where you started. Kill the old site with a 410, or even better, you can go into GWT and verify yourself for that sub domain.  Then you can put in a request to have that entire subdomain removed from Google.  You then put up a robot.txt and it is gone.  This will also prevent the old site from being found again by Google for all the links. https://support.google.com/webmasters/answer/1663427?hl=en We use load balancers on our servers and once had a subdomain www1 that got indexed and there were even some FB shares etc.   We ended up going this route to get it out and keep it out of the index.

One more - once you do fix the issue of the display on the page, you will want to 301 all the rogue URLs to the correct ones just to get that cleaned up for the Goog or anyone else who might visit.  That will also take care of the 404 issues in GWT.

The selling point makes sense and I could see how that would be true.  But if you are not seeing an increase then it is not worth it, especially if your focus is on the organic traffic.

Over the last year Google is way too active with updates and there are seriously like tons of updates available till date... I would advise you to see if any new penguin or panda is not attacking the website. Also you really need to be active on building quality and diversified links to your website to gain Google trust again or else you will not be able to regain the rankings again!

Numbers in the URL never hurt rankings! It’s just that your URL should be easy to read and follow the standard length... too long URLs might get you in to trouble from user experience as well as from the crawling and indexing point of view. Search Engine Journal (know blog for SEO and Online Marketing Updates) do use numbers in there URL! http://www.searchenginejournal.com/why-relying-on-organic-search-traffic-alone-is-a-risky-business/65324/ << The URL ends at the number! Hope this answer your question!

By the way, has the blog ever ranked for instabill?

No, the page authority is earned by the links it gets from other sources. If it is a guest post it is probably a new page, therefore it has no authority yet, but it will earn it over time (the most links it gets the better (shares, etc)).

Unfortuneatly you're in a pickle, and there aren't many good solutions. The first thing I would ask is if you are sure these links are hurting you. Even though they may be 'bad', it's possible Google doesn't see them that way and has passed no filter or penalty. If you determine the links are bad then basically, your options are: Removal Disavow 404 or 410 the URL It's true you need to be careful when using the disavow, but if you've been hit by a penalty or think you've been hit by Penguin and have exhausted all other option, it's generally your best bet. Cheers, Cyrus

Is it always the same number (15633)? Are those pages dynamic or static? If they are static yes, you will need to add the canonical and noindex meta to each page. If they are dynamic just in one page while build the code to display the appropriate canonical href and NOT to show the noindex when the user is seeing the page you actually want indexed.

Thanks! Sorry I freaked out, the other day I noindexed archives and some other things and I was worried my settings screwed things up. Phew!

Hi Nicholas, in addition to posting on Inbound.org, you may also consider checking out our list of Recommended Companies. Best of luck!

I agree, though I suspect I might have a looser interpretation of if it's relevant or useful. In general terms, if they're fairly solid charities, then the links can be decent. I'd add to this that if the business is geographically focused, then there's benefit and relevance in building links and goodwill (online and off) by partnering with local charities. We've had a lot of success with this, especially in more closely-knit areas.