Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
You are totally wrong guessing my path. You are going down a tunnel which doesn't have a exit. Personally I think, in this thread, you got some good advice about what you should focus on, so I would stop feeling in dismay, and confidently steer away from bad practices. Good luck.
First of all, I was just browsing and I got blocked as bot see below:
I would remove that cloaking.
Second, understanding your visitors behavior is one of the most complex task, you don't know your user behavior until you run a lot of test, survey and so on...
Well, then I totally agree with you, Ryan, thanks for the answer. With a DA of 1, you are absolutely right.
Let me say straight forward, all that bot blocking is not a good idea.
I have been there in the past few times, especially for e-commerce, scraping to compare prices is very common, and I tried blocking scrapers many times, maybe I am not that good, but at the end I gave up because the only thing I was able to do was annoy legitimate users, and legitimate bots.
I do scrape other website too for price comparison, tens of websites, since I don't want to be blocked I split the requests among different tasks, I add a random delay between each request, I fake header data like user agent pretending to be Firefox from a windows pc, and I cycle through different proxies to continuously change IP address.
So as you can see, it's much harder to block scrapers than it seems.
Neither I would use JS to block cut&paste. I have no data to base my judgement on. But it's annoying for users, it doesn't sound compliant with accessibility, it stinks and google usually doesn't like things which stinks. Plus... If someone wants to scrape your content you are not going to block him that way.
Ryan, I don't agree. It's true external factors (in other words backlinks) nowadays have the biggest impact, but on-page optimization as far as my little experience tell, still does affect ranking and it's worth working on.
And if we don't keep track of changes on pages and change on ranking how can we know what is working and what is not?
Especially since there's no gold rule and what works for one site doesn't necessarily work for another.
To make some example, I had a page which was ranking in position 1 for a search query with a volume of 50+k and very high competition. I expanded content to improve ranking for some additional queries, and it worked, it climbed from 2nd and 3rd serp page to 1st for a couple of those queries (I use both Moz ranktracker, semrush, and proracktracker to monitor ranking).
Unfortunately ranking for the search query with the highest volume moved from position 1 to postion 2, I changed the content a little bit, to add some keyword, which made sense because was re-balancing the keyword density now that the content was bigger. And in 24 hours it got back to position 1, without damaging the other search query improvement.
**In many other cases, I improved ranking on pages without any backlink, just improving the content, and I am talking about business critical pages with a high competition.
So I would say on-page optimization is still worth spending time on, to test the effect of the changes is a must and to monitor google ranking fluctuation is a must too.
Of course I am not saying off-page optimization is not important, is fundamental, I am giving that for granted.**
That's interesting because according to google own words:
Google takes a snapshot of each page examined as it crawls the web and caches these as a back-up in case the original page is unavailable. If you click on the "Cached" link, you will see the web page as it looked when we indexed it. The cached content is the content Google uses to judge whether this page is a relevant match for your query.
Source: http://www.google.com.au/help/features.html
If I look for that page using a fragment of the <title>(site:http://www.qjamba.com/ "Ferguson, MO Restaurant") I can find it, so it's in the index.</p> <p>Or maybe not, because if you search for this query <strong>"Ferguson, MO Restaurant" 19 coupons</strong> (bold part quotes included) you are not among the results. So it seems (I didn't know) that using site: is showing results which are not in the index... But I would ask in <a href="https://productforums.google.com/forum/#!forum/websearch">google search product forum</a>.</p> <p>As far as I know you can use meta tag to avoid archiving in google cache but your page doesn't have a googlebot meta tag. So <strong>I have no idea why is not showing</strong>.</p> <p>But if I was you I would dig further. By the way the html of these pages is quite weird, I didn't spend much time looking at it, but there's no H1, you are blocking cut&paste with js... Accessibility is a factor in google algo.</p></title>
I am not sure I understood your doubt but I will try to answer.
site://foo.com
is giving you a number of indexed page, is presumably the number of pages from that site in the index, it normally differs from page indexed count in GWT, so both are probably not all that accurate
site://foo.com "The quick brown fox jumps over the lazy dog"
searches among the indexed pages for that site the ones containing that precise sentence
webcache.googleusercontent.com/search?q=cache:https://foo.com/bar
check the last indexed version of a specific page
if you have a 404 for the cache: command that page is not indexed, if searching for the content of that page using site: you find a different page, it means that other page is indexed for that content (and one possible explanation for that is a duplicate content issue)
A lot of years ago (early 2000's) I had few blog on wordpress.com and they were easily getting indexed by google with good ranking, to me it looked like google was giving wordpress.com blogs a boost just because they were on wordpress.com. But things evidently changed, because at a certain point became harder and harder to to get good ranking for new blogs hosted on wordpress.com and in later years i have never been able to win that fight.
My personal and humble suggestion is to just grab a domain, install wordpress and manage your blog there, when you are ready to include it in your main website you just 301 to the new domain.
Yes it's worth it if you don't care about your traffic and you happy to see it disappear.
I would change /av into /audio-video and add a 301
I would add sub folders to the url structure for each category with a dedicated page.
"cache:" is the most update version in google index
if you fix the duplicate content next re-indexing will fix the duplicate content issue
I am assuming it's duplicate, it can be de-indexed for other reasons and the other page is returned because has the same paragraphs in it. But if you ran a couple of crawling reports like moz/semrush etc.. And they signal these pages as duplicates it may be the issue.
Yes, one more idea, if you take the content of the page and you query your site for that content specifically like this:
You find a different page. Looks like those pages are duplicate.
Sorry for missing a w.
Well, that's a http 404 status code, which means the page was not found, in other words it's not in google index.
Please note if you type site:ww.qjamba.com/restaurants-coupons/lemay/mo/all you find nothing see image below.
Again I would doubt your logs. You can also check GWT for any manual penalty you may have there.
You evidently mistyped the url to check, this is a working example:
If your new content is not there, it have not been indexed yet, if your logs says it was crawled two days ago I would start doubting the logs.
Open this url on any browser:
You can reasonably take that as the date when the page was last indexed.
You could also programmatically store the last google bot visit per page, just checking user-agent of page request. Or just analyze your web server logs to get that info out on a per page basis. And add a couple of days just to have a buffer (even google need a little processing time to generate its index).
Check my answer to similar question: http://moz.com/community/q/pages-with-rel-next-prev-still-crawling-as-duplicate
Even if you add a canonical url, those pages are technically duplicated, that's why moz report will keep reporting them as duplicate, in the report the canonical url is shown in a different column, so if you don't care about them you can just filter them out.
But, in the first place, why did you choose to NOINDEX, FOLLOW those product pages?
If you have a preferred product page among those different variants seen as duplicate why don't you just use canonical to point there?
What do you think is the benefit of noindexing? Theoretically leaving them there with the duplicate content and a canonical you are wasting some google bot crawling budget, but unless you need google to crawl your pages with a high frequency (because your content is frequently updated) I wouldn't care much.
Personally I see de-indexing as the last resort.
As I said theoretically having the keyword you are after in the leftmost position is better. So if you already analyzed which keywords are the most relevant for you, you should try to place them in the leftmost position. If for example you are fighting a battle to improve your serp position for the query "property-name", you could decide to choose the second url in your example.
But, as others have already mentioned, serp positions are influenced by many many many factors, and focusing too much one technicality can mislead you and shift your focus from the general picture.
In general you should structure your content in folders, because google algo expect to find content structured that way, but it doesn't mean you have to in your specific case or for all your pages, or all your products, etc...
You should structure your url, after analyzing your content, the keywords you are after and your visitors behavior (and I would give a look at your competitors as well).
At the end you will have to make decisions between different possible url structures; you will have to take a risk making you best educated guess based on the analysis you have done, that's why I said the best thing you can do is "test".
I can't tell which structure is best for you, because it depends on that lengthy analysis you should perform, there's not an answer which fit them all.
As of today, every time we published a new website it got indexed by MOZ at their next update. Of course there was some sort of initial PR launch and we got some backlinks in the days or weeks following the launch (from websites with an established DA and PA).