Welcome to the Q&A Forum

Simalar question here:

http://www.seomoz.org/q/competitor-scraped-ecommerce-product-overview

Short answer, e-commerce site copying products descriptions from other sites is common place, and there is little you can do about it, but I believed google will know that its your content as it will most likely be crawled on your site before your competitors have a chance to find, copy, post and then get crawled.

But you mom's blog will get crawled far less frequently than page A as it has less authority, so the chances the your mum read copied and then posted on her blog before the main site was crawled and in time for her site to get crawled would be very unlikely.  Other wise it would be very unfair that "bigger sites" get credit simply just by being bigger.  But I get your point that being found first is the only factor, but I would think it would be a big factor if not the biggest (back links to the original article would be one I would think)

Btw i'm sure you Mom is a very nice person and would never steal a story of another site for her own blog

Note: I have come to this conclusion based on observation and logic (and others posting similar theories), but I don't have any conclusive proof, I wonder if anyone has do any research into it...

At the moment the pages are too similar so are coming up as dups, (they also will most likely compete with each other in the serps too)

My advice would be either make them more different content wise,  or have one page that covers both terms (I would guess they would be long tail terms anyway, so that might be the best option)

using canonical links it telling google they are the same page content wise and which is the "master page" to show in the serps

"p/s: I have just add robots.txt to sandbox and disallow all pages."

That will do it, just have to wait till google picks up on those instructions (might take a while)

a trick to  not get personal results is to add &pws=0  to the end of the search url

eg

https://www.google.co.uk/search?q=keyword&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official**&pws=0**

SEOmoz's bot does not take instruction from No-follow, it crawls your whole site, unless you use robots.txt

Also I think its bad practice to No-follow internally,  better to use no index on the page, if you don't want it in the serps.

"diluting your authority by having dofollow links"  I thought it was not possible to link sclupt in this way, I'm I missing something?

My understanding is that all no follow does is not pass authority and does not help retain authority ( basicly it just throws it out the window the authority it would have passed to that link)

The e-commerance system we have has a geo tagged hompage system so you can set up different homepages based on the user country IP.  But I want to test what the default homepage is, if the system can not get the user IP, does anyone know of a way to do this?

Also does googles bot, not give an IP for this, or is it always an American IP (even if your site is set to a different country)?

Thanks

Don't forget Moz's bot only crawls 10,000 pages, while Google can/will crawl more.

But to be sure manually check some of the dups that GWTs are flagging.

If google crawled and picked up the text on your site before your competitor  then you get the "credit" for it,  which logicly they should have since you had the text first, only problem if your competitor is a bigger site DA wise and they are crawled more often.

But e-commerce site copying text from other site has been happening for a long time, they are just not doing themselves any favours

edit: missed eyepaq post, but yeah, what he said

I don't know about yoast, but if wordpress and magento have their own site maps, they both can be added to sitemaps under webmaster tools.

You can also check the sitemap that yoast makes, to see if it has both wordpress and magento links in the xml file (pick 2 random urls and seach).

I would just make sure that the xml is being updated when you add new pages to your site.

My understanding is that google will " pick up"  the story/text from the original place when it crawls it (bigger sites get crawled more often), so it gets the credit.  In that case the only problem is when a big site copies a story from a smaller site before google has crawled it.

But your worrying that all theses back links will look spammy.  Story syndication or data scrapping is so common and been happening for so long, it would have been a big problem by now, but its not (not to my knowledge).  I think google can easily spot a syndicated or scrapped story from a legit source and knows not to mark the backlink it as spammy, but I don't think you would get much juice from the syndicated links (especially if they are junk site).

I know of sites were they do the same as you (get picked up by big news sites), and it all gravy baby

I got eight 404 errors with the Seomoz crawl, but the report does not says where the 404 page is linked from (like it does for dup content), or I'm I missing something?

Thanks

So you think these would hide these links from goggle's bot, but will be usable for the end user?

I know the "no follow" link sculpting does not work, and I'm sure google would be switched on to tricks to hide links to sculpt.  If the trick does work, its the type of thing google does not want you to do, so I wouldn't do it incase I get punished by google down the line.  (maybe I missunderstand what your trying to do).

I'm not familiar with that type of link, so I don't know if it would work on a technical level.

yes, use the robots file to deindex, once your ready to launch remove the disallow, and the next time google crawls your site you should get indexed.

you must have been linked from somewhere for google to find you, so it means google will find you quicker once you ready to launch ( you already have a backlink)

export as a csv file

"However i have double checked that where is no external link to the webmail pages from any other source"

the link might be so obscure that seomoz has not picked it up and google bot has.  Note in WMT there is "latest back links" but there is a few weeks delay on them, so the link might be recent.

I'm 99% sure there has to be a link from somewhere, or there is no way google can find that page.

I have added the No-index as well in  my robot.txt file on my site but yet no change.

It will not take effect until the next time google crawls your site, I think it took a week or so before my test site was removed (i did no robot.txt file too)

you have no results at all?  as perfect007 said its updated once a week, so it normally says "in progress"

But if you still get no results by tomorrow, then report to the seomoz staff,  along time ago I had a crawl problem where it was only crawling about 500 pages of my site, when we had 10,000+ pages.   There was a technical problem with the crawl with my site which they fixed very quickly

there must be a link from somewhere for google to pick it up,  I had a test site on a sub domain that got picked up to my surprise. The reason, the site had pinterest button on the product pages, and the developer had tested it to see if it worked, and later removed it, but someone (or something) repinned it and then a backlink to my test site was born

Point is there are millions of ways to get a back-link accidentally .

Best way not to list is either added it to you robots.txt file or add No-index to the webmail page

Yeah I did that after posting the question I started test like that, but its not coming up and searching the url does not show the page, but other normal pages ("lower" pages) are showing (that don't have this problem), so it seems that it is de-indexed those pages.

its weird that webmaster tools say ""We didn't detect any issues with non-indexable content on your site.", when there are.

Getting this sorted one way or another is my top priority