Welcome to the Q&A Forum

Rand has a good response to this here:

http://www.seomoz.org/q/seomoz-rocks-but-is-it-breaking-google-s-webmaster-guidelines-when-it-comes-to-check-rankings

Inside the ranking keywords list you should have a list of all your keywords. If you mouse over a specific keyword on the left listing a little clock icon and a little square graph icon should appear. If you click on the clock icon it should take you to a historic view of that keyword's performance. I have no idea if it backfills to the beginning of your campaign though (e.g. you added a keyword last week, it might not show anything before last week).

EDIT-Just tested and it doesn't backfill.

I've got a site that's brand spanking new that I'm trying to get the error count down to zero on, and I'm basically there except for this odd problem. Roger got into the site like a naughty puppy a bit too early, before I'd put the canonical tags in, so there were a couple thousand 'duplicate content' errors. I put canonicals in (programmatically, so they appear on every page) and waited a week and sure enough 99% of them went away.

However, there's about 50 that are still lingering, and I'm not sure why they're being detected as such. It's an ecommerce site, and the duplicates are being detected on the product page, but why these 50? (there's hundreds of other products that aren't being detected). The URLs that are 'duplicates' look like this according to the crawl report:

http://www.site.com/Product-1.aspx

http://www.site.com/product-1.aspx

And so on. Canonicals are in place, and have been for weeks, and as I said there's hundreds of other pages just like this not having this problem, so I'm finding it odd that these ones won't go away.

All I can think of is that Roger is somehow caching stuff from previous crawls? According to the crawl report these duplicates were discovered '1 day ago' but that simply doesn't make sense. It's not a matter of messing up one or two pages on my part either; we made this site to be dynamically generated, and all of the SEO stuff (canonical, etc.) is applied to every single page regardless of what's on it.

If anyone can give some insight I'd appreciate it!

Unless I miss my guess the press release/journalist stuff is probably just signing up for HARO.

Yeah, got 5,000 or so. Glad to see this thread because I had a mini heart attack there. The only other time I'd seen 406 come up was related to a phishing/ad/virus filtering so I thought maybe we had an ad-spewing virus or something.

Yes, a visit to example.com/dir should now return a 404 error (if you haven't done any redirecting/canonicalizing). This will increase your 404 count in Web Master tools but it's far preferable to the alternative. If you're not redirecting the robots.txt will eventually work and hopefully the links will just fall out of WMT.

This is a guess on my part, but I imagine this is also one of the reasons why SEOMoz got out of the agency business; kind of hard to avoid conflict of interest when you theoretically could have data on all of your clients' competition at your fingertips. (The other big reason would be concentrating on core competencies)

I guess it's possible they might share some of the info with Distilled but I get the impression that SEOMoz is a pretty pro organization.

Going to have to agree with Alan here. I've worked on a lot of ecommerce sites and as the categories fill up with competitors all using the same text it quickly becomes an issue of duplicate content and authorial ownership. What I've seen in the past is cost-sharing, where the manufacturer and the retailer will combine monies to hire authors to make retailer-specific content for those products. In part it depends on the category as well; something like clothing can have just about any old description, but something like a calculator is pretty hard to make sexy.

The links are another question entirely; in a way I don't see how they can do anything but be neutral, or helpful to you right now. It certainly establishes you as the originator of the content too.

I saw it mentioned in another topic and tried it out today. I like the 'numbers' approach they take, and I'm probably going to use their data someday. Their tools are good, but they're a bit light on interpretation and calls to action. This is fine for me, but might be less useful to someone just starting out in SEO/not familiar with the technical aspects of the field (e.g. most small business owners doing self SEO), which is where SEOMoz shines I think.

Basically it shouldn't really have an affect; those unformatted file listings are literally the web server automatically saying 'here's the files that are in this folder', there's no meta tags, description, on page elements, etc.

If you have these pages and they're ranking well, you generally don't want them to be. The automatic file browsing pages don't have your name, your company, etc. in them, and they're generally pretty ugly. They also theoretically could be 'stealing' juice from your 'real' pages, if your internal structure isn't flowing relevance properly.

Basically what I'm saying is that if these pages are having some kind of SEO effect, you probably don't want them to be since they're so basic.

Also I can't overstate the security concerns that directory browsing might be introducing. If someone can directory browse to where your code lives (.php, .aspx.vb, whatever) they may be able to read it. Code sometimes has important things like logins, passwords, merchant account ids, etc. in it that you definitely don't want people reading.

I would suggest 301'ing them to their /index.htm or /pdf.htm equivalents. If you don't know, a 301 is a signal to a web browser (or search crawler) saying "this page has permanently moved, please go to (otherpage.htm) instead".

Here's a good SEOMoz article explaining it a bit more:

http://www.seomoz.org/learn-seo/redirection

What might be more of a concern, is it sounds like your web server has directory browsing enabled. This could be a security issue (depending on your web server setup). Generally you don't want to expose directories if you don't have to because it gives a potential attacker insight into your system setup. Here's an example how to do it in Apache:

www.camelrichard.org/topics/Apache/Turn_OffDirectoryBrowsing

And IIS:

technet.microsoft.com/en-us/library/cc731109(v=ws.10).aspx

If you like I can confirm if you have open directories if you give me the link, either here or through private message.

Thank you for the update NIck. I've noticed that the same reports are still 'stuck' where they were 6 hours, should I cancel them and just start new ones you think?

Just going to add to what these guys have said, you're thinking of the right thing.

One thing I'd suggest (but you've probably already thought of) is you say your URL structure is changing pretty radically. Is that because you're going to more englishy, SEO friendly URLs? If the URLs all change to just numbers and URL parameters that's obviously a bad thing.

Hello All -

I've been running some Advanced Reports again lately, and they seem much slower than I remember from last time I ran some. Currently I've got one (Inbound Links) report at 2,500 out of 10,000 links retrieved through LSAPI, and it's been at that point for about 6 hours. Did something get cloggered on the reports, or is it this just the expected performance?

410 is a weird error to be having; it means 'this page is gone'. Not 'missing' (404) but just 'gone', as in the server knew it had that page at one point and now no longer does. There's only two things I can think of off-hand that might cause it:

1. MozBot is messing up somehow, and returning this weird error message when it should be reporting a 'real' error (e.g. 'blocked by robots.txt', 404, 401, etc.) Double check your robots.txt entries, if any.

2. Your server is somehow configured to give the 410 message to the MozBot. I assume you're not getting this message when you visit this page in a normal browser. On UNIX platforms you might want to check the .htaccess file and see if there are any directives that end in [G], or have custom error handlers like in this article:

https://sites.google.com/site/onlyvalidation/page/force-a-404-or-410-using-the-htaccess-file-on-apache-server

You might be able to get fancy with the Google search operators:

http://www.googleguide.com/advanced_operators.html

For example I was able to use inurl:1MnylRaQC3w to get the pages embedding that video.

EDIT-Hmm, doesn't return the gizmodo one you want though. Unfortunately each site is probably going to embed the video a bit differently. It looks like Gizmodo is doing it entirely in Javascript and I'm not sure how to search within that exclusively. intext: doesn't seem to be doing the job.

I thought this was about micro-formats and schema.org content but after some looking around I'm not sure now. There doesn't seem to be a consensus on how those links appear, but I haven't been able to find anyone discussing it after 2006 or so. This was the best writeup I could find:

http://www.seobythesea.com/2006/12/googles-listings-of-internal-site-links-for-top-search-results/

Basically it seems to be that if Google has 'learned' your site thoroughly they'll arbitrarily decide to shove those links in there. The discussions I can find don't really suggest a way to do it, and some express frustration over controlling which links go there. The impression I get though is:

• Google SiteMaps are probably involved

• The sub pages are usually linked to heavily

• The sub pages are usually pretty 'clean', e.g. full of content

I guess it's comparable to what Google classifies as sub-results when you search for your brand name but other than that it's a mystery to me.

Okay; this is my opinion based advice but I'd suggest staying away from the Open Source solutions then. Some of those solutions can do great stuff, but they're so genericized to meet every potential need that there's a lot of upfront setup and config. Also the management interface is less than ideal for that number of SKUs out of the box. It's certainly doable, but you're getting into 'the custom work might cost more than buying it' territory. I've only worked with 4-5 'off the shelf' applications, but Magento sounds closest to what you want based on my experience. There might be better ones out there though.

Donald Barnes raises a good point too; I directly manage and host all my e-commerce apps, so it's not an issue I've run into. But having a host who understands what your platform needs is an important step too.

One thing to watch out for is that this can backfire perception-wise. It might not matter, but depending on the domain name, if your biggest competitor/related business owns the .com and you buy the .net people might feel 'cheated' by ending up at your domain when they expected the other (or vice versa!). This would be kind of like if I registered seomoz.net and used it for my blog for example.

One last thing to note is that by doing this you're not 'fixing' the errors; you're preventing the search engines from seeing them. In the case posited in the blog the errors were actually caused by the search engines going after wacky URLs but there may be other problems with your site that were causing those errors. So I guess I'm saying don't stop looking into it after the pages stop getting indexed just in case.