Welcome to the Q&A Forum

The Bingpreview crawler, which I think exists in order to take snapshots of mobile friendly pages, crawled my pages last night for the first time.  However, it is adding a trailing slash to the end of each of my dynamic pages.  The result is my program is giving the wrong page--my program is not expecting a trailing slash at the end of the urls.  It was 160 pages, but I have thousands of pages it could do this to.

I could try doing a mod rewrite but that seems like it should be unnecessary.  ALL the other crawlers are crawling the proper urls.  None of my hyperlinks have the slash on the end.  I have written to Bing to tell them of the problem.

Is anyone else having this issue?  Any other suggestions for what to do?

The user agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b

Thanks.  The robots.txt file was the problem.  It originally (yesterday) excluded rogerbot (by default) and then I remembered that and put it in as rogerbot but that didn't work.  So I changed it to RogerBot  and that didn't work.  Today I removed the robots.txt file completely and it worked.  Then I put it back with rogerbot and it is working.

It APPEARS that maybe it read the robots.txt  yesterday before i put in rogerbot and for some reason didn't read it after I put it in.  Will never know but it is now working.

Thanks for the help!

Hi sure, thanks.  This page shouldn't have a speed issue but maybe you can see what the issue is:

www.qjamba.com/local-coupons/wentzville/mo/all

Thanks.

How can I find out why rogerbot won't crawl an individual page I give it to crawl for page-grader?  Google, bing, yahoo all crawl pages just fine, but I put in one of the internal pages fo page-grader to check for keywords and it gave me an F -- it isn't crawling the page because the keyword IS in the title and it says it isn't.   How do I diagnose the problem?

Thanks Ray.  Appreciate the advice!

Hi Ray-pp,

Thanks for your answer.  I'm not getting anything significant, but occasionally a bot will come with extra stuff added to the parameter names, so it got me to thinking a malicious program or nasty competitor might want to do that to cause havoc.  My understanding is that 404s don't hurt SEO ranking from Google, but I was thinking that the way things are set up now no-one would get a 404 and in fact Google would index the 'bad' pages, so maybe I needed to do something proactively to 404 or 301 such pages so they would never get put into an index at all.

Since my site has lots of dynamically generated pages, I've had my share of surprises, and am just trying to avoid any new ones!

It occurs to me that a malicious program could set up thousands of links to dummy pages on a website:

www.mysite.com/dynamicpage/dummy123

www.mysite.com/dynamicpage/dummy456

etc..

How is this normally handled?  Does a developer have to look at all the parameters to see if they are valid and if not, automatically create a 301 redirect or 404 not found? This requires a table lookup of acceptable url parameters for all new visitors.

I was thinking that bad url names would be rare so it would be ok to just stop the program with a message, until I realized someone could intentionally set up links to non existent pages on a site.

Great answer Monica -- thank you!

Thanks Amelia!

Matt, thanks.. Good points for sure.  My concern is that since something like 50% of new businesses close doors within 5 years, so the list of redirected urls will just keep getting bigger over time..Is that a concern?  I guess over time less people will link to the defunct businesses, but I will still have to track them..maybe at some point when the number of links to them is small it would make sense to then 404 them?  Of course, I'd still need to track which ones to 404, so i'm now wondering when 404 ever makes sense on prior legitimate pages..

Just to be clear -- redirecting does remove the old link from the index, right?

It doesn't make sense to me that a 404 causes a loss in link juice, although that is what I've read.  What if you have a page that is legitimate -- think of a merchant oriented page where you sell an item for a given merchant --, and then the merchant closes his doors.  It makes little sense 5 years later to still have their merchant page so why would removing them from your site in any way hurt your site?  I could redirect forever but that makes little sense.  What makes sense to me is keeping the page for a while with an explanation and options for 'similar' products, and then eventually putting in a 404.  I would think the eventual dropping out of the index actually REDUCES the overall link juice (ie less pages), so there is no harm in using a 404 in this way.  It also is a way to avoid the site just getting bigger and bigger and having more and more 'bad' user experiences over time.

Am I looking at it wrong?

ps I've included this in 'link building' because it is related in a sense -- link 'paring'.

Are there any studies that show which is best?  If you find my page but not the specific thing you want on it, you may still find something of value.  But, if you don't you may associate my site with poor results, which can be worse than finding what you want at a competitor site.  IOW maybe it is best to have pages that ONLY and ALWAYS have the content desired.

What do the studies suggest?

I'm asking because I have content that maybe 1/3 of the time exists and 2/3 of the time doesn't...think 'out of stock' products.   So, I'm wondering if I should look into removing the page from being indexed during the 2/3 or should keep it.  If I remove it then my concern is whether I lose the history/age factor that I've read Google finds important for credibility.  Your thoughts?

Martijn,  thanks for your response.  I could see Google saying that on a given page the same source image shouldn't have more than one alt description, and either penalizing such or just picking one of them or ignoring it altogether.  It's the penalty I'm concerned with, of course. . But, I can also see that if they are seeing a blank.gif for the source and some code related to lazy loading they may go ahead and give credit to each alt as though a real src was loaded--and maybe even tying it to the real src image name for image search.  Just looking for a bit more feedback from real-world experience first..

Has anyone else worked with this and determined if it is a pro or con?

I didn't find an answer on a search on this, so maybe someone here has faced this before.

I am loading 20 images that are in the viewport and a bit below.  The next 80 images I want to 'lazy-load'.  They therefore are seen by the bot as a blank.gif file.  However, I would like to get some credit for them by giving a description in the alt tag.  Is that a no-no?  If not, do they all have to be the same alt description since the src name is the same?  I don't want to mess things up with Google by being too aggressive, but at the same time those are valid images once they are lazy loaded, so would like to get some credit for them.

Thanks!  Ted

Hi Giovatto,

Thanks for your thoughts.  I had thought that too, but it looks to me like while it crawls both, it indexes only the desktop.  I think it crawls the mobile to see what else mobile links to, but that the main purpose is to see if the pages render well for mobile.  Here's my otherr thread that goes into this in more detail: http://moz.com/community/q/mobile-search-results-show-desktop-crawled-content

Thanks.  I just feel like I have to understand it better before deciding what to do.  Adding in the desktop-only pages on a mobile device is likely to create speed issues (as they are currently designed) which will hurt my ranking.  That's one reason I prefer to not have them if it doesn't hurt me in Google's eyes, but i've just about decided something is probably better than nothing as far as SEO is concerned.

That article seems to be clearly saying that there is one index, but it isn't saying they don't crawl both--I'm sure they do, otherwise they wouldn't have both for fetching, but what I think they are doing is crawling to see if the mobile page is mobile friendly--so they can list (and rank it in search results) it as such on mobile devices.  But, since with their  fetch for mobile they immediately crawl desktop too-- it seems to validate that they are using the desktop version to get the actual content for storing in the index tables.

Hi Kristina,

I just found this:

http://blog.hubspot.com/insiders/how-to-follow-googles-mobile-rules-for-awesome-seo

<<Keep in mind that Google does not have a separate mobile index, so when a user searches from a mobile device, the results are from the same index as they are from a desktop. When a search is executed from a mobile device, Google serves up the desktop version of the URL, and when the searcher clicks a result, the mobile version of the URL is loaded. The way in which the mobile version is loaded is determined by the mobile configuration you have set up. >>

This, if true, validates what I have been suspecting, -- it appears that Google expects basically equivalent content for mobile-friendly and desktop when sharing urls--it doesn't expect a 'minimal' mobile version. I assume Google does this in part because it saves billions of dollars of storage to have only one index--and in the vast majority of cases mobile content SHOULD be equivalent anyway.

It also addressed my other concerns, which validate what you were saying:

<<redirecting mobile="" users="" to="" the="" homepage="" in="" absence="" of="" a="" version="" desktop="" page="" is="" bad="" practice.="" if="" does="" not="" exist,="" should="" be="" displayed.="" additionally,="" there="" with="" no="" equivalent,="" displayed="" for="" desktops.="">></redirecting>

I still plan to run some simple tests with fetching and diff keywords for mobile and desktop pages to verify.  Then will decide what to do.  Thanks very much.

Thanks very much Kristin.  I have to think about it more..I did just notice when I fetch in Google for desktop it crawls for only the desktop, but when I fetch for mobile, it crawls with both the smartphone and desktop bots.....I'll have to test more..will report any findings if I can figure it out!  the popup was meant for when mobile users click on a non-mobile friendly page--it automatically directs them to the mobile site, but in this case since the page isn't mobile friendly they would get a message just to basically warn them that that PAGE isn't optimized for mobile--sounds like 'best practice' is to give them that page as opposed to a 404 page with options only for mobile pages..

Yes and no   First I'm sorry if my wording threw you off on my setup.  These things are so hard sometimes to clearly describe (for me at least).

As for Amazon, it DOES seem to validate the idea that Google is using only one index of content(desktop), which is what I just read at this site, in the comments by Sp4cecat: http://www.sitepoint.com/responsive-design-vs-m-sites.  So for matching urls with only reformatted content, that seems to say I'm on track--no problem....

But here is what may be a big problem for me, I think: Amazon probably is serving all the same urls for both, where I'm serving only some of the desktop pages for mobile.  So, if Google really only is keeping one index (the one for desktop), then I may be throwing them off by giving them a 404 when their mobile bot tries to go to a page that only exists for the desktop.  They may be removing that page from the 'one and only' index they have: desktop.  Then when the desktop bot crawls that url -- it gets put back into the index.  So back and forth it goes, and that can't be a good thing for SEO, can it?  And yet I can't figure out a way to test to see if that's really what happens because of the lag time between crawling and showing up (or removing) from the index!  Arrrgghh..

If all that is true, I have a real problem, as it would suggest that for seo purposes Google expects mobile content to be equivilent to non-mobile content.

Any thoughts on that issue?

Hi Kristina,

I just responded to you on the other (related) issue at http://moz.com/community/q/mobile-search-results-show-desktop-crawled-content#reply_270529  before seeing your response here.  Sorry about that.

Thanks for your response.  It really hadn't occurred to me to allow mobile users to use those pages because the experience would be pretty bad for the most part, and a lot slower for those pages, but it definitely is a thought for solving my indexing issue.  I could give the mobile user a popup the first time they venture into the 'unmobile-friendly pages' as a heads up about which parts are mobile friendly and which aren't...I wonder though if Google will penalize the site or those pages in some way since it will not be able to render them as mobile friendly for a mobile device..EDIT: just saw this: http://www.cnet.com/news/make-web-sites-mobile-friendly-or-face-google-search-wrath/....anyway,  interesting alternative, It wouldn't be as easy as it sounds because of the degree of customization in menus I've done for the mobile pages, but may well be a lot easier than other alternatives..  Thanks very much!

Ted