Pages are Indexed but not Cached by Google. Why?

[max.favilli]

Well, then I totally agree with you, Ryan, thanks for the answer. With a DA of 1, you are absolutely right.

[max.favilli]

First of all, I was just browsing and I got blocked as bot see below:

http://imgur.com/HoKqh97

I would remove that cloaking.

Second, understanding your visitors behavior is one of the most complex task, you don't know your user behavior until you run a lot of test, survey and so on...

[RyanPurkey]

Yeup!  Indexing time varies.  You'll be able to tell the time between crawl and indexation by when Google shows your page version B in it's cache after you made changes from A, so if the 'example.html' page is already in Google's index you'll see this:

You make changes on a page, example.html (version A is now version B)Google crawls example.html (version B)
You check Google to see if example.html is version A or B in the cache
no?
no?
no?
no?
yes. That's how long it takes.

OR, you make a new page. It gets crawled. Checking if it's indexed... no, no, no, no, yes?! That's how long it takes.

Again, this time period varies and having a site with excellent domain strength and trust usually makes it a shorter time period.  It also tends to influence how many pages Google decides to keep in its index or show to users.  Pretty much everything gets better for a site the stronger its domain authority and trust are.

[friendoffood]

Thanks for sharing that.  I was only kidding above, but obviously it's no joking matter when a user gets blocked like you did.

I just looked and see that it blocks when something/someone clicks 3 times within 30 seconds. EDIT: but that's only if it isn't keeping the session between clicks--see next post

[friendoffood]

THANK YOU!

[friendoffood]

Geez, I'm so pedantic sometimes.  Just need to understand what this means:

<<or, you="" make="" a="" new="" page.="" it="" gets="" crawled.="" <strong="">Checking if it's indexed... no, no, no, no, yes?! That's how long it takes.>></or,>

How do you do the bolded?  site:www.site.com/thepage "my content change on the page" ?

And, you did say one can change and not the other yet the page really has been indexed, right?

[RyanPurkey]

The second example is talking about a new page that never existed before, i.e. new-example.html... So you created a wholly new page on your site. You see that it gets crawled, you go to Google to see if it gets indexed.

Again though, the lower your site's domain authority and trust, the higher chance of that site getting pages indexed slower, de-indexed, and not showing up in high in the rankings.

Remember my earlier suggestion video?  You're sweating the computer and details and minutiae way too much at the expense of doing what would really move the needle for optimization at your site's stage (getting reputable links from other domains).  Same goes with what you're doing with trying to block certain activity on your site. Normal user activity is getting messed up--Massimiliano and Travis' experience.

This is probably the best advice at this stage: instead of spending one more second on this Q/A thread and trying to see how many minutes transpired between your own changes and seeing them in Google, spend that time to go get 10 good links.  No need to even thank me. I'll take the silence as your newly enlightened bliss.

[friendoffood]

Massimiliano,

Can you tell me your steps that led to that error?  It looks like you went directly to www.qjamba.com/local-coupons/wildwood/mo/all   and then  you opened up a separate tab and went to www.qjamba.com  and then either refreshed the home page or opened the home page again in another tab -- all within 30 seconds.  That's the only way I have been able to reproduce this , because it looks for 3 searches without any existing session within 30 seconds by the same ip address, and the home page wipes out the session and cookies, and those are the urls the db table shows that you went to, and in that order.

Normally a user stays in the same tab, so with the 2nd search will have a session -- but your ip had no session with each search.  And, normally you can't go to the home page from a location page.  So, I'm confused as to what you did if it wasn't like what I wrote above.  If you didn't do this then I'm worried of a serious programming problem having to do with the php sessions getting dropped between pages.

I"ve put a lot of time into this website and a ton of testing of it too, and just went live a few months ago, so these kinds of problems are disheartening.  Ironically, your experience is almost identical to that of Travis, except that in your case you must have moved a little faster since you got a different message.  But, it would REALLY help me to get some feedback from both of you confirming what I wrote or setting me straight if you did something different.

[max.favilli]

You are totally wrong guessing my path. You are going down a tunnel which doesn't have a exit. Personally I think, in this thread, you got some good advice about what you should focus on, so I would stop feeling in dismay, and confidently steer away from bad practices. Good luck.

[friendoffood]

Massimiliano, my guess of your path was the most logical conclusion based on the fact that I have 3 records of the urls you went to on my site, and showing that the program didn't keep any session variables between the 3 urls you came to.  You first went to wildwood.  Then you went to the home page.  This implies that you either did that in a new tab, or you hit the back key, or you modified the url and removed the wildwood part to go to the home page, as opposed to clicking on something on the page.  Telling me I'm wrong at least lets me know I may have a serious problem to fix, but you are mistaken to think that this is a robot problem.  It is a php session variable problem, apparently, that none of my extensive (hundreds of hours) testing has ever had.

This is a serious problem unrelated to the OP and about 100 times more important than the OP that I was hoping to get some help with because it is very difficult to diagnose without feedback from those having the experience that you had with my site,. However, that's my problem I'll have to deal with.  I don't know if you just don't remember or aren't telling me because you think it is a robot problem, but if you do happen to recall the steps (or at least tell me it was all done in the same tab or you hit the back key) I'd appreciate whatever it is you can tell me.  If I can't solve the problem it probably means I'll have to shut down my website which I've put more than 4 years of my life into.  Seriously.

Thanks for your various other responses though..  Take care. Ted

[Travis_Bailey]

I can't really argue with log files, in most instances. Unfortunately, I didn't export crawl data. I used to irrationally horde that stuff, until I woke up one day and realized one of my drives was crammed full of spreadsheets I will never use again.

There may be some 'crawlability' issues, beyond the aggressive blocking practices. Though I managed to crawl 400+ URI before timeouts, after I throttled the crawl rate back the next day. Screaming Frog is very impressive, but Googlebot it ain't, even though it performs roughly the same function. Though, given enough RAM, it won't balk at magnitudes greater than the 400 or so URIs. (I've seen... things... ) And with default settings, Screaming Frog can easily handle tens of thousands of URI before it hits it's default RAM allocation limit.

It's more than likely worth your while to purchase an annual license at ~$150. That way, you get all the bells and whistles - though there is a stripped-down free version. There are other crawlers out there, but this one is the bee's knees. Plus you can run all kinds of theoretical crawl scenarios.

But moving along to the actual blocking, barring the crawler, I could foresee a number of legit use scenarios that would be comparable to my previous sessions. Planning night out > Pal sends link to site via whatever > Distracted by IM > Lose session in a sea of tabs > Search Google > Find Site > Phone call > Not Again... > Remember domain name > Blocked

Anyway, I just wanted to be sure that my IP isn't white listed, just unblocked. I could mess around all night trying to replicate it, without the crawling, just to find I 'could do no wrong'. XD

Otherwise it looks like this thread has become a contention of heuristics. I'm not trying to gang up on you here, but I would err on the side of plenty. Apt competition is difficult to overcome in obscurity. : )

[friendoffood]

Hi Travis.

Thanks for the info re Screaming Frog.

I didn't white list your IP.  I just changed the number in my files to something else, which would already be unblocked since a day has passed.

The scenario you gave would be quite rare and wouldn't create a block because that requires everything happening in 30 seconds or less (and the session wouldn't expire in that time frame), or that the same IP address also tried to crawl my site with a bot in the useragent (your scenario with screaming frog)..  But yours and Max's experience is looking more and more like it is commonplace, and I'm the fool who hasn't known that's what's happening because I can't distinguish between you and a robot (which doesn't keep sessions).

All I need is to verify that I have a sessions problem.  Here's what it takes:

1. use a desktop or laptop
2. remove all cookies related to qjamba
3. go to http://www.qjamba.com
4. choose an option (restaurants) and a location (Saint Louis) and click
5. dont do anything else - just close out the tab
6. If your ip address changes, let me know when you do this so I can find it in the logs

In all my usage it keeps the session between steps 3 and 4.  It looks like for you and Max it doesn't do that, which means many of my users would be having the same terrible experience as you, for the 3 months since it has been live.  It's a disaster.  But I have to first verify that it really is a problem and unfortunately I have to rely on strangers like you who are experiencing the problem to do that.

If you just do those steps I promise we'll be done

Thanks, Ted

[friendoffood]

This post is deleted!

[Travis_Bailey]

Good to hear you may be getting closer to the root of the problem. Apologies that it took so long to get back to you here. I had 'things'.

I followed the steps and you should be able to determine the outcome. Spoiler Alert: No block, this time.

It's a whole other can of worms, but should you need more human testing on the cheap; you may find Mechanical Turk attractive. One could probably get a couple hundred participants for under a couple hundred dollars, with a task comparable to the one above.

Just a thought...

[friendoffood]

Thanks Travis,

I have discovered that for some users the initial page loads up to 3 times nearly immediately without it being visual.  It never happens to me or any or the browser/system combinations I use on remote machine testing - even if I match someones setup who is getting the problem, but there is no question it is happening.  This was triggering the robot message I was giving.  I don't yet know the cause as the typical culprits don't apply.  I relaxed the rule by 1 more load in 30 seconds, which is why you didn't get a message.  Am going to use someone's computer tomorrow that gets the problem to try and narrow it down.

Agree on the human testing.  Thanks for the suggestions.

take care