Welcome to the Q&A Forum

Sorry about your being blocked.  A day hadn't passed due to the timing of your second visit--sorry.  I just changed the ip address in the table so you aren't blocked now.

Ok, I think I figured out what happened.  You first went to the ferguson site.  You may clicked on something but the same page was reloaded.  Then in a different tab you clicked on my home page from a google search results page.  Then in a third tab you went directly to my home page.  Then you ran screaming frog and the program stopped it without a message, seeing the word 'spider' in the useragent.  Then you tried it again and it recognized that as a stopped bot and gave the message about suspicious activity.

The program wipes out sessions and cookies when a user goes to the home page (it's not even a link anywhere) since that is just a location-choosing page, so when you opened it in a different tab the sessions were wiped out.  It had nothing to do with you being in incognito or not having cookies allowed.

Does this sound like what you may have done, and in sequence?

That's what it looks like, and that, if correct, is a huge relief for me since that is not usual user activity.  (Although I may have to reconsider whether its still a poor approach).

I don't know about what happened with your second visit and the timeout.  curious that you got some 60 pages crawled or so--I don't suppose you have anything that would tell me the first 3 of those so I can look into why it timed out?  The table isn't keeping the ip on crawling so I can only look those up by the url crawled and the time.

Travis,

First of all, I absolutely appreciate all the time you are taking to address my issues here.  Second of all, it IS very tempting to join you and any others here to go build houses or do something else, especially given the last few days..:)

Ok. I'll try to keep it short:

I wasn't thinking you had any bearing on my site going down, but that maybe there was a 'moz effect'.  hope not.

Re: Chrome Incognito Settings:
I'm really worried now that there is a sessions problem since anyone with cookies allowed should have the session ID saved between pages--in which case they would have only 1 entry in my 'user' table and you had 3 in a short amount of time.  That's why it thought you were a robot.  I don't know how to duplicate the problem though because I've never had it personally, and I use a program that connects to other machines with all kinds of combinations of operating systems and browsers and computers and have never had this problem with those.  It's my problem.  I'll have to figure it out somehow.  I have many session variables and it would be a huge overhaul to not use sessions at this point.  If you have any ideas (I'm using php) I'm all ears.

re:Fun w/ Screaming Frog:  
The IP for the 8.5 hour later instance was the same as your first one. Yet, if you were spoofing it shouldn't have said screaming frog in the user agent, right?  It was in my 'bot-stopped' file as instantly stopping because it was an unexpected bot.  So, confused unless perhaps you tried it separately from running with the spoof?

<<normally screaming="" frog="" would="" display="" notifications,="" but="" in="" this="" instance="" the="" connection="" just="" timed="" out="" for="" requested="" urls.="" it="" didn't="" appear="" to="" be="" a="" connectivity="" issue="" on="" my="" end,="" so...="" yeah...="">>
Ok.

Fun w/ Scraping and/or Spoofing:
<_>
I'll have to check into it.  I've run Yslow and Gtmetrix without problems.  I see you tried to run it on the ferguson page and the home page.  I just ran the ferguson page in gtmetrix - which uses Page Speed Test (Google?) and Yslow both - and it ran ok, although not a great grade.

<<while i'm="" running="" off="" in="" an="" almost="" totally="" unrelated="" direction,="" i="" thought="" this="" was="" interesting.="" apparently="" bingbot="" can="" be="" cheeky="" at="" times.="">>  That is interesting.

I'm worried now most about the session issue, as that may be affecting a lot of my users and I've assumed multiple entries were from robots that generally don't keep sessions between page crawls (actually quite a few of the seo crawlers do -- but google, bing, yahoo don't).  If you are ok with going to my home page without incognito and clicking on a few pages and letting me know the first part of your IP when you do that it might really help me. Your shouldn't be blocked anymore (it lasts 1 day). But, no worries if you're ready to move on.</while>_</normally>

Sorry, wasn't so short after all.  Thanks again.  Ted

Thanks Ryan.  I am new to this so I appreciate the cautions. Are you saying it is a bad idea to try and run tests on specific page changes (ie like reducing the size of a menu or adding more content in paragraph form, for example) to see how they affect ranking on given searches?  I'm struggling with why that is a problem for Google..  The SEO experts who run tests--to get a sense of what factors are important -- isn't that exactly what they are doing in a lot of cases?  I plan on using the MOZ tools and have much to learn, but it just seems so logical to test things out like I want to do also.

The GWT index status page shows an increasing number of pages, but no detail on which ones have been indexed.  That's why I was looking into the site:  command and the cache command in the first place.  It seems those exist for a reason, and I thought it was to help us with specific pages.  If that isn't it, why do they exist?  Other than showing the number of indexed pages I don't see much value to the index status page. Perhaps there is a good tutorial somewhere that helps people know how to use the GWT not just to find crawl errors or see general trends but to actually implement SEO page changes to improve their site?

I hope this doesn't sound argumentative..I just don't quite understand your perspective on it.

Thanks.  I will give you a 'good answer soon', but dont want to close this out yet since there are some other issues going on with the other response.  In addition I'm trying to figure out the practical way to use all this info:

Here's what I was trying to do:  Make a program change, fetch with google, and see in a couple of days if the site has risen in the rankings.  It was complicated (for me) by the absence of the fetched site being cached (which even the one you found is not a recent cache), and the the site:www... sometimes not appearing to have changed at all even if there was a new cache.  How is a person to test the immediate effect on ranking when they make a change if they can't tell what Google is doing?

I haven't found in GWT a list of my indexed pages, or the dates they were indexed, and certainly not the content that has been indexed, yet your answer above seems to indicate there is more under the hood than what i have found there. Is there really a way in GWT to see if a specific page has been indexed (my site has thousands so I would need to search for it), and when it was last indexed?  I can tell from my dbtables what pages and when they were crawled but not whether Google has added the new info into their indexing algorithm.

Thanks for your reply Ryan..

Well, that was easy.  How come for some pages they have that code and others they don't?  It seems like a backward way to find the cached version of your own pages too..

Thanks much.

Now, for a twist, suddenly I checked just now for site:http://www.qjamba.com/restaurants-coupons/ferguson/mo/all

and it is no longer there.  It was there yesterday.  Why would it suddenly disappear?  I again checked webmaster tools--not finding any crawling errors or problems for it.   So, the question suddenly gets reversed: how now can I have a cached version without a site:www... listing?

Glad you didn't get frustrated.  Ok, lets see, there are a number of issues:

Screaming Frogs--I see the entry, and we know why it was stopped yesterday (robots.txt).

Your activity yesterday--I see 3 entries for you yesterday.  I'm very concerned I might have a sessions problem because if you had cookies enabled my tables should have shown only one entry for you with several searches.  If you didn't have cookies enabled you should have gottena message saying the site requires cookies.  Did you get that message regarding cookies for each page you went to last night?  IF you didn't get that message i may have a serious issue to address -- it would really help in this case to know which items you have checked in Chrome for the cookies settings.  I used the site this morning incognito without blocking cookies and my table showed just one entry and no blocking, so It would really help me to know what your setttings are for cookies, as I may have to program differently.

On your second visit last night:
First, did you run Screaming Frogs on that second visit?  Someone did about 8 1/2 hours after your first visit.

Re the scraping with your googlbot spoof: My program shouldn't block the IP for google ever, so are you sure it got blocked--did it give you a message?  If so I need to figure that one out too and it would help to have the IP you grabbed if possible.   As for the CSS and Javascript, I don't know enough about that--I don't think I have any hyperlinks in the javascript that it would find.  I have a both internal and external CSS and JS.

RE the robots file and googlebot: It's crawling ok and was verified in their Webmaster Tools, and the format I use is based on what I read.  While you are right that bad bots ignore robots.txt. I want to keep a lot of the neutral or even 'good' bots out, so thats why I do it that way.  I have htaccess blocking too, although its never a perfect thing..

RE: the ferguson page OP issue   While I haven't formally submitted the ferguson page (how'd you like that for a city choice?) to google via a site map (that's a long story), google has been crawling that page since at least November 15, 2014.  I don't know if a cache existed at that time as that is something I have just learned about in the last few days after I was trying to figure out if Google indexed some changes I put in before fetching just a few days ago.

Re this:<_>  I"m not sure what a noarchive directive is, so I'd have to say probably not on that one.

Thanks for whatever more help you can provide--I really hope to solve that session issue if it exists (as that has been a concern for several months but I wasn't sure how distinguish between a robot (which doesn't keep sessions) and a real user -- like you! ), and of course the actual OP issue too!_

p.s my server/site was down this morning and my host didn't know why.  Is it a bad idea to put one's website urls on the moz forum like I have (ie bad bots crawl the moz site looking for urls to torture?)

HI Travis,

Thanks you for your reply and I'm sorry for the frustration.  I looked at my db tables to try and figure out which one was you and what actions you took so I could diagnose why it gave you that message.  Sounds like I should relax the restrictions on user actions.

I know it would be tempting to ignore me at this point but it really could help me if I can find your activity in my dbase records that the program keeps so I can come up with a better solution.Could you give me part of your IP address so I can try and find your activity in my dbase table records so I can solve the problem that gave you that irritating message?  I would really appreciate it, as there actually may be an underlying problem with how it is storing sessions that I've had a real hard time pinpointing because it is very hard to distinguish real users that the program is having a sessions problem for from a returning robot that never keeps session cookies.

As for the robots.txt file, I purposefully block all but a handful of crawlers to avoid getting bombarded by crawlers which can slow down my site, and make it difficult to gauge real users.  This is nothing new for my site, and Google crawls my site every day.  So, I think perhaps you tried using a tool on it that it blocked and I'm sorry for any confusion that caused. Which tool did you use?

I'm somewhat new to robot activity and how to deal with the pesky guys without causing user problems, so am open to suggestions.

While I don't think these problems you had relate directly to the OP I certainly can see how they would relate to anyone trying to address the OP with a tool-oriented or hands on interaction with my site--again sorry for the frustration. Thanks for your feedback.

Here's an example:

I get a 404 error for this:

http://webcache.googleusercontent.com/search?q=cache:http://www.qjamba.com/restaurants-coupons/ferguson/mo/all

But a search for qjamba restaurant coupons gives a clear result as does this:

site:http://www.qjamba.com/restaurants-coupons/ferguson/mo/all

What is going on?  How can this page be indexed but not in the Google cache?

I should make clear that the page is not showing up with any kind of error in webmaster tools, and Google has been crawling pages just fine.  This particular page was fetched by Google yesterday with no problems, and even crawled again twice today by Google  Yet, no cache.

Will do.  Forgot to mention Bing is checking into it.  But for the reasons you mentioned I am still going to do the 301s.  Thanks again.

I'm going to post a question about the non-cached as upon digging I'm not finding an answer.

And, I'm reading where it seems to take a couple of days before indexing, but seeing something strange that makes it confusing:,

This page was cached a few days ago: http://webcache.googleusercontent.com/search?q=cache:http://www.qjamba.com/restaurants-coupons/wildwood/mo/all

The paragraphs wording content that starts with 'The Wildwood coupons page' was added as a test just 3 days ago and then I ran a fetch.  When I do a Google search for phrases in it, it does show up in google results (like qjamba wildwood buried by the large national chains).  So, it looks like it indexed the new content.

But if you search for wildwood qjamba restaurants cafes  the result Google shows  includes the word diners that is gone from the cached content (it was previously in the meta description tag)!  But  if you then search wildwood qjamba restaurants diners  it doesn't come up!  So, this seems to indicate that the algorithm was applied to the cached file, but that the DISPLAY by Google when the user does a search is still of older content that isn't even in the new cached file!  Very odd.

I was thinking I could put changes on pages and test the effect on search results 1 or 2 days after fetching, but maybe it isn't that simple. Or maybe it is but is just hard to tell because of the timing of what Google is displaying.

I appreciate your feedback.  I have H2 first on some pages because H1 was pretty big.  I thought I read once that the main thing isn't if you start with H1 or H2 but that you never want to put an H1 after an H2.

I'm blocking the cut and paste just to make it harder for a copycat to pull the info.  Maybe overkill though.

Thanks again, Ted

Thanks for your reply Cyrus.  Wow, so much to learn.

I will put in logic via a mod redirect to basically remove the trailing slash and go to the resulting url because otherwise all the trailing slash urls will be a different page of basically a 'no-product' business and the like.

These are all dynamically generated pages, so I think as long as I resolve to the 'proper' no-slash version then I won't need to worry about anything else, like a rel=canonical tag because there wont be any identical content.

Does that sound right to you?

Thanks.. That does help..

<<if 404="" you="" have="" a="" for="" the="" cache:="" command="" that="" page="" is="" not="" indexed,="" if="" searching="" content="" of="" using="" site:="" find="" different="" page,="" it="" means="" other="" indexed="" (and="" one="" possible="" explanation="" duplicate="" issue)="">></if>

THIS page gives a 404:

http://webcache.googleusercontent.com/search?q=cache:http://www.qjamba.com/restaurants-coupons/ferguson/mo/all

but site:http://www.qjamba.com/restaurants-coupons/ferguson/mo/all

Give ONLY that exact same page.  How can that be?

Thanks Massimiliano. I'll give you a 'good' answer here, and cross fingers that this next round will work.  I still don't understand the timing on site:www , nor what page+features is all about.  I thought site:www was supposed to be the method people use to see what is currently indexed.

I have a bigger problem than I realized:

I accidentally put duplicate content in my subcategory pages that was just meant for category pages.  It's about 100-150 pages, and many of them have been crawled in the last few days.  I have already changed the program so those pages don't have that content.  Will I get penalized by Google-- de-indexed?  Or should I be ok going forward because the next time they crawl it will be gone?

I'm going to start over with the fetching since I made that mistake but can you address the following just so when I get back to this spot I maybe understand better?:

1. When I type into the google searchbar    lemay mo restaurant coupons smoothies qjamba

the description it gives is <cite class="_Rm">www.qjamba.com/restaurants-coupons/lemay/mo/smoothies</cite>The Lemay coupons page features both national franchise printable restaurant coupons for companies such as KFC, Long John Silver's, and O'Charlies and ...

BUT when I do a site:<cite class="_Rm">www.qjamba.com/restaurants-coupons/lemay/mo/smoothies</cite>it gives the description found in the meta description tag:  www.qjamba.com/restaurants-coupons/.../smoothie...Traduci questa pagina Find Lemay all-free printable and mobile coupons for Smoothies, and more.

It looks like site:www does NOT always give the most recent indexed content since 'The Lemay coupons page...' is the content I added 2 days ago for testing!  Maybe that's because Lemay was one of the urls that I inadvertently created duplicate content for.

2. Are ANY of the cache command, page+features command, or site:www supposed to be the most recent indexed content?

thanks.

That's weird because doing the site:  command separately for that first page for the /smoothies gives different content than for /all :

site:www.qjamba.com/restaurants-coupons/lemay/mo/smoothies

site:www.qjamba.com/restaurants-coupons/lemay/mo/all

But why would that 'page+features' command show the same description when the description in reality is different?  This seems like a different issue than my op, but maybe it is related somehow--even if not I prob should still understand it.

you are missing a w there.  site:www and you have site:ww

That's why I'm so confused--it appears to be indexed from the past, they are in my dbase table with the date and time crawled -- right after the fetch --, and there is no manual penalty in webmaster tools.

Yet there is no sign it re-indexed after crawling 2 days ago now.  I could resubmit (there are 15 pages I fetched), but I'm not expecting a different response and need to understand what is happening in order to use this approach to test SEO changes.

thanks for sticking with this.  Any more ideas on what is happening?

Hi, thanks again.

this gives an error:

http://webcache.googleusercontent.com/search?q=cache:http://www.qjamba.com/restaurants-coupons/lemay/mo/all

but the page exists, AND site:www.qjamba.com/restaurants-coupons/lemay/mo/all

has a result, so I'm not sure what a missing cache means in this case..

The log shows that it was crawled right after it was fetched but the result for site:... doesn't reflect the changes on the page.  so it appears not to have been re-indexed yet, but why not in the cache?

HI Massimiliano,

Thanks for your reply.

I'm getting an error in both FF and Chrome with this in the address bar.  Have I misunderstood?

http://webcache.googleusercontent.com/search?q=cache:http://www.mysite.com/mypage

Is the command (assuming I can get it to work) supposed to show when the page was indexed, or last crawled?

I am storing when it crawls, but am wondering about the couple of days part, since it has been 2 days now and when I first did it it was re-indexing within 5 minutes a few days ago.

I have seen where it will index on of my pages within 5 minutes of fetching, but have also read that it can take a day.  I'm on day #2 and it appears that it has still not re-indexed 15 pages that I fetched.  I changed the meta-description in all of them, and added content to nearly all of them, but none of those changes are showing when I do a site:www.site/page

I'm trying to test changes in this manner, so it is important for me to know WHEN a fetched page has been indexed, or at least IF it has.  How can I tell what is going on?