Google will index us, but Bing won't. Why?
-
Bing is crawling our site, but not indexing it, and we cannot figure out why -- plus it's being indexed fine in Google. Any ideas on what the issue with Bing might be?
Here's are some details to let you know what we've already checked/established:
- We have 4 301’s and the rest of our site checks out
- We’ve already established our Robots is ok, and that we are fixing our site map/it's in fine shape
- We do not see anything blocking bingbot access to the site
- There is no varnish or any load balancers, so nothing on that end that would be blocking the access
- We also don't see any rules in the apache or the .htaccess config that would be blocking the access
-
How to use Fetch as Bingbot
Fetch as Bingbot is a very useful tool for troubleshooting. You will find this tool in the Diagnostics & Tools section in the navigation menu. **Fetch as Bingbot **allows you to see exactly what the Bingbot sees when downloading a page from your site. As such, it’s a great tool for determining if a page on your site can be crawled. To use it, simply enter in the URL from your site and click Fetch. This tool will send the request to Bingbot for processing, so it’s normal that it takes a few seconds to run your request. Just below the URL entry form, the URL you requested (along with any previous requests) will be shown. When Bingbot is finished with your request, the status for the URL will change to Completed. To see what Bingbot found at your URL, simply click the **Completed **link and scroll down the page to view the page source that Bingbot found.
SEE WHAT BINGBOT SEES
Fetch as Bingbot shows you exactly what your HTTP Headers and page source look like to Bingbot. Viewing your page’s source in this manner ensures you know exactly what Bingbot is seeing when it crawls the URL. This is especially useful in helping find areas when you want to make sure content can be indexed or in cases where your site may have been compromised and is for example sending different HTML to Bingbot then to users. By scanning through the page source, you could note things like links injected into blog posts or any other manner of additive items you did not place on your website.
Fetch as Bingbot is also a reliable way to test if a URL is being blocked by your robots.txt file. It’s polite, so sending it to fetch a URL blocked via robots.txt will get you a notice explaining that this barrier is in place.
Also, if Fetch as Bingbot is not able to crawl a page because of other blocks or politeness constraints, for example the maximum amount of crawls we are allowed to make in a given time window, the tool will tell you too.
|
NOTE |
| To see the crawled content of a page using the crawlable Ajax protocol using #! notation in the URL, you need to input the static URL instead. The static URL is the URL that contains the ?escaped_fragment= portion. Note that although Bing supports this protocol we do not recommend it for frequent use. |COMMON PROBLEMS
WHAT DOES "REDIRECTION LIMIT REACHED MEAN"?
Unlike the SEO Analyzer tool, Fetch as Bingbot does not follow redirects. Instead it will let you know that the page resulted in a redirect and shows you the HTTP headers it received from your server. The line most commonly starting with HTTP/1.1 shows the status code (301, 302, or in some cases 303 or 307 - see this full list of 3xx status codes) and the location: header shows you where the server tells Bingbot (and other clients) where the redirect should go to. To fetch the page at that location you will need to perform a new fetch using that URL. This is in fact how the Bingbot crawler works: every redirect requires a brand new fetch, and this new fetch does not necessarily happen immediately after encountering the redirect.
WHAT DOES "DOWNLOAD OF ROBOTS.TXT FAILED" MEAN?
When the result of a fetch reads “Download of Robots.txt Failed”, then Bingbot was not able get a proper server response when trying to access your site’s robots.txt. This usually means that your site’s server is configured to refuse Bingbot or the IP addresses from which Bingbot operates a proper connection or simply denies access to this URL. This is a problem, since Bingbot now cannot tell whether or not a robots.txt file actually exists nor read any directives in the robots.txt file should it exist. You should investigate (or ask your ISP to investigate) whether such a configuration is in place.
WHAT DOES "REPRESENT DOWNLOAD NOT ATTEMPTED DUE TO POLITENESS ISSUE" MEAN?
When the result of a fetch reads “Represent download not attempted due to politeness issue”, then Bingbot did not try fetch the page due to restrictions on how much we can crawl the site. This usually means that Bingbot is backing off at this point in time and that it cannot fetch the page "politely", that is, honoring crawl delays or heuristically determined maximum fetches per second for your site. If this issue persists it may point to insufficient bandwidth available to Bingbot to crawl your site efficiently. First, you should check your site's robots.txt for any crawl-delay: directives. Removing or lowering crawl-delay directives can also help with this issue. If there are no crawl-delay directives present, you should check Crawl Control settings to see if they can help improve by setting higher levels (especially during times when you expect fewer visitors).
WHAT DOES "UNEXPECTED ERROR" MEAN?
If you are seeing frequent "Unexpected Error" results in Fetch as Bingbot, this may indicate that our crawler is not able to connect to your server and not getting a server response at all. Your server may be unintentionally blocking Bingbot based on user agent or IP range. To remedy this, look for server configurations or modules that may be dropping connections for Bingbot or work with your ISP to identify whether the issue is on their side.
You can find more information on troubleshooting issues at Bing Webmaster Tools Help & How-To Center
-
Can you provide your domain please?
-
Yes! The domain is http://www.pamandgela.com/
Thank you!
-
Not seeing anything overtly wrong. I submitted your homepage to Bing for indexing, so we'll see what happens there.
2 other things though.
- You advertise. Get a hold of the ads side and see if they know anything or if they can connect you with anyone. It's a long shot, but worth a try.
- Have you verified your site in Bing Webmaster Tools and submitted your sitemap manually?
-
Thanks, Kate! Unfortunately, we've tried all those idea already too.
- We've contacted ads, but on luck in connection there.
- We've submitted to webmaster tools; the site is getting crawled but not indexed.
- We’ve submitted the sitemap manually, as well.
-
This is highly confounding. I've crawled as Bingbot and nothing is in it's way.There are a few other issues though. You are using the canonical on a large number of pages. In my crawl, I am seeing the crawler going through a large number of pages and only returning a fraction of them. At this moment my crawler is finding 31k things to crawl, has crawled 1044 and returned only 29 pages that might be indexed. 65% of those are without errors.
It is possible that the site structure is causing issues. I suggest taking time to review your site structure and making it easier for bots to find the pages they need to index, telling them what not to crawl (a canonical is a suggestion and should be used when pages are duplicated), and fixing all errors on the site.
-
We are having the same issue with one of our sites. We have tried all of the suggestions we can find on the internet.
Submitted out site to Bing webmaster tools, submitted the Sitemap, checked the Robots.txt...
It is now crawling the site but still not indexing any of the pages! The only thing left for us to do is wait and hope that Bing will eventually index it, but it seems weird as its being indexed fine in other search engines and generally getting good search engine rankings for keywords.
The website is www.thriveinlife360.com for reference.
We have a client base of well over 20 websites and all of them are being indexed in Bing, except from this one.
Would be interesting to know how many websites this affects and if there is a pattern within those that Bing doesn't like.
-
Just encase anyone is still having issues with this, or happens to come across this post in the future...
We emailed Bing support who came back and said that they had found their bots were blocked from indexing our site, there was no reason for this but just an error in their system (apparently this is quite common).
They responded pretty quickly and removed the block, a couple of days later we were being indexed and seeing traffic come from Bing!
-
Yes, sorry
I used the form on their support site for the webmaster tools : https://www.bing.com/webmaster/support
Although that doesn't seem to be working at this moment so the email address that it was sent to is :
Hope this helps!
-
0875
-
09515
-
092.2k
-
02200
-
04328
-
02559
-
110795
-
05926