Welcome to the Q&A Forum

Hi,

We've got some pretty strict anti-scraping logic in our website, and it seems we accidentally snared a Googlebot with it.

About 100 URL requests were responded to with a 403 Forbidden error. The logic has since been updated, so this should not happen again.

I was just wondering if/when Googlebot will come back and try those URLs again. They are linked from other pages on the site, and they are also in our sitemap.

Thanks in advance for any assistance.

Thanks everyone! We are just going to leave it as is. Google will eventually flush it out. Ryan - because of the 90 days we can't remove the URL's. I will need them back sooner than that when we actually put products in those states. Thanks again! helpful....as usual!

I understand HOW to remove a page. I want to know whether it's better for me to manually remove it or let Google remove it on its own.

We accidentally generated some pages on our site that ended up getting indexed by google. We have corrected the issue on the site and we 404 all of those pages. Should we manually delete the extra pages from Google's index or should we just let Google figure out that they are 404'd? What the best practice here?

Doug,

Thanks for the advice but that does not answer my question regarding truncating the description meta tag. Do you have any advice there?

Our descriptions are well written and are, of course, geared towards helping the end user. Some of them are a little long though which is why we are considering truncating them and adding a '...'.

Thanks, Alex

Thanks very much for the replies. We'll avoid automatically truncating the title tag.

We are also automatically truncating the description meta tag, and adding '...' to the end if it is longer than 150 characters. Would you recommend not truncating this as well?

I've read that page titles can't/shouldn't be more than 70 characters long. Out of around 1,000 products we have about 150 that have legitimate titles that exceed this character limitation. We plan on automatically truncating these. Should I just cut the titles off at 70 characters or should I cut them off and add a "..."? Does it even matter?

We have now launched our site and are now looking to create a 20-30 second video to put on our home page to help set expectations for our users so that they know exactly how our site works. We used Angelvision Tech in the past for a 3-minute video but that was around $10k. I need short, sweet, and affordable. Thoughts?

Thanks so Much Alan! I passed it along to him. I'm sure he will find it more than useful.

Are you in Australia? The lead developer said that someone in Australia is crawling the site. We have a bot trap in place. If you are a human it should present a captcha and let you continue viewing the site.

SEOmoz Community,

I am a business owner and have spent the past year studying SEO. I've been through countless hours of videos and reading. For those questions that go beyond anything i've watched or read i've referred to this community. Everyone here has been VERY helpful and I have based some major decisions on what i've learned from everyone here. We launched our site this morning and now i'd like to see how we did from your point of view. Is my overall site/URL structure up to par? I still need to build backlinks and things of that nature but getting the site live is a major milestone for us. Any comments (good and bad) are welcome.

Site: www.Suretegrity.com

Thanks, Alex

Thanks for the replies.

I'm not sure I stated my question clearly. What I was wondering, was whether it is better to return status code 500 from the requested url, or status code 301 from the requested url. So in one case, the actual url content would be the error page, and in the other case it would redirect to the error page.

I'm not even sure if there is a difference, but if there is I'd like to learn more.

Thanks.

When there is a 500 Internal Server Error, is it better to return an HTTP 500 response and custom error page from the requested URL, or is it better to return a 302 redirect? The redirect would send the browser to the custom error page, which would return the HTTP 500 result.

We tell Google not to index or follow our error pages, so if Google sees an error at a URL, we don't necessarily want Google to think that the URL should be ignored. That's why the alternative would be to redirect to a custom error page with it's own URL.

Similarly, what's the best approach if the response is a 404? Return HTTP 404 and custom 404 page from the requested URL, or redirect?

Thanks.

Thanks everyone. This was very helpful. I will NOT include the state abbreviation. Ryan - I have an uppercase letter in every word. I will go ahead and format everything so that only lowercase letters are used. Thanks!

My title structure is properly in place. I just want to make sure there's nothing additional I can do since we are addressing URL Keywords today.

.

.

If I have an h1 tag on the page that describes the purpose of the page, is it a bad idea to make all of the sub-headings h2 tags? A designer mentioned that it was frowned upon to make all of the secondary headings into h2 tags, or to use CSS to selectively style the heading tags.

.

.