Welcome to the Q&A Forum

Yes.  Best practice is to have 1 page optimized per keyword that you are targeting.  What you are proposing would be consistent with this.  Google Keyword Slios and that will give you some interesting reading.

I think you answered your own question.  You have to compare search traffic potential for "dog training twin cities" vs "training german shepard".   If you see enough traffic on the german shepard stuff, create a new page and link to it.  Maybe focus on the top 5 types of dogs and have that as a section.

You are talking about stealing from Peter to pay Paul, but it looks like you are being smart and using analytics to try and answer your question.  There is your science.  You have to look at the long tail of search terms from analytics, and you should see some patterns there.  That will guide you on where to build content over time.

There are some cool regular expressions you can use in Google Analytics to built keyword reports around 2 and 3 word queries.  One example:

http://secretswede.net/seo/measure-longtail-traffic-google-analytics-mayday-update/

Take this and setup a segmentation to see as a whole if this type of traffic is really driving conversions.  It may be that you get traffic, but less conversions.  This adds another layer to how you look at this and if it is worth spending time building out content.

Good luck!

Nah - I say subdirectory

You get all the control, plus all the link equity is shared across all site. Rising tides raises all ships. I have done this with independent separate brands under a single brand even. It works.

Yes you can rank for subdomains. But the link equity is not shared. Unless you want all that extra work.

More reading

https://moz.com/community/q/the-great-subdomain-vs-subfolder-debate-what-is-the-best-answer

https://moz.com/blog/subdomains-vs-subfolders-rel-canonical-vs-301-how-to-structure-links-optimally-for-seo-whiteboard-friday

I like the automated sitemap answer for the cause (as this has bitten me before), but you mentioned you do not have that.  I would still bet that somewhere on your web site you are linking to the page that you do not want indexed.    It could be a tag cloud page or some other index page.  We had a site that it would accidentally publish out articles on our home page ahead of schedule.  Point here is that when you have a dynamic site with a CMS, you really have to be on your toes with stuff like this as the automation can get you into situations like this.

I would not use the noindex tag and remove it later.  My concern would be that you are sending conflicting signals to Google.  noindex tells good to remove this page from the index.

"When we see the noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it." from GWT

When I read that - it sounds like this is not what you want for this page.

You could also setup your system to show a 404 on the URL until the content is live and then let it 200, but you run into the same issue of Google getting 2 opposite signals on the same page.     Either way, if you first give the signal to Google that you do not want something indexed, you are at the mercy of the next crawl to see if Google looks at it again.

Regardless, you need to get to the crux of the issue, how is Google finding this URL?

I would use a 3rd party spider tool.  We have used Screaming Frog SEO Spider.  There are others out there.  You would be amazed what they find.  The key to this tool is that when it finds something, it also tells you on what page it found it.  We have big sites with thousands of pages and we have used it to find broken links to images and links to pages on our site that now 404.  Really handy to clean things up.  I bet it would find where there is a link on your site that contains the page (or pages) that link to the content.   You can then update that page and not have to worry about using noindex etc.  Also not that the spiders are much better than humans at finding this stuff.  Even if you have looked, the spider looks at things differently.

It also may be as simple as searching for the URL on the web with the link: attribute.  Google may show you where it is finding the link.

Good luck and please post back what you find.  This is kind of like one of those "who dun it?" mystery shows! 

Yep, this is a geotargeted page.  That said, I am not sure it is "cloaking".   I have a site where on all pages of the site we have a sidebar that shows geolocated data specific to that user.  The info in that sidebar is different for different users.  It is not "cloaking" as even though I am showing different information to different users (based on location), I am treating all users the same.

Regardless of how "safe" using geolocation is, I do not think it makes sense the way that this site/page is doing it.  Would be better to have a dedicated "$cityname apartment" page so that Google can crawl it and let you rank for that page.  Not sure that Google would penalize this as much as not being able to tell what the focus of the page is to categorize it properly for the correct location search.

What keyword is this page ranking for ?

P

Re-read the Moz blog.  It talks about why it is not a direct impact on rankings but an indirect one.  Matt Cutts is an engineer and thinks in black and white, there is nothing "direct" but that does not mean there is not an indirect effect.  The indirect effect is what the blog post is talking about.  Read the "update" part at the bottom.

It depends on your business plan.   Why are you building a second site?  Is it for insurance in case the first site gets taken out due to penalty?  I would think that focusing on #RCS and using best practices would make sure that would not happen.   Are you trying to dominate the SERPS so that you can rank #2 and #3 and #N for a given key word?   Maintaining another site is a lot of work.  Seems simpler to build up a YouTube Channel to rank or some other social profile.  Are you wanting to rank for a second set of key words?  Seems simpler just to build up an additional section of content on your current site.

Just sounds like you are doing a lot of work on a second website in the same competitive niche, where it may make more sense to just focus on the site you have, make sure it is super wicked awesome and just continue to focus on that.

I would just follow and say that using Google or any other company that large is not the best example for most businesses.

Google does not need to worry about SEO. It has one of the most authoritative websites on the internet. They have baked in traffic due to android, chrome and map users. Yes the sub domain works for them from a product and technical standpoint. They have different groups using different technologies different servers/resources so they need the subdomains to make that all work.

As I read the question, this person is not in that situation. This brand needs link equity to be shared. Links are still super important, I don't see why they would want to dilute that by using sub domains. Everything they want to accomplish can be done w a sub folder.

I would say it is preferable to go ahead and setup the site on the new host first before you take down the old one.  While you should be "ok" with the downtime, I would not recommend it.  You never know when the spiders come along.  You would probably not be de-indexed, but Google would see a bunch of errors and you could potentially see a drop in the SERPS and then traffic to your site as a result.  This should all recover.

I have seen on our sites drops in traffic when we have had technical difficulties.  I usually see issues in GWT or other tools and I get them fixed and the traffic comes back.

Here is the other thing.  What if something goes wrong during that 12 hours?  What if  12 hours becomes 24, 48 etc. due to unforseen issues.  That is just bad for business/users in general when a site is down for any amount of time.  You do not want that, let alone the search engine issue. What if something goes wrong with the new host and you need to revert back to the old?  This has happened to me and trust me, you do not want this to happen to you.  Murphy likes to play games with scenarios like this, I do not mess with Mr. Murphy and his laws.

If I were you here is what I would do

1. setup the new host

2. setup your site on the new host

3. test test test on the new host

4. change the DNS from the old to the new host

5. watch the traffic move

6. test test test on the new host

7. shut down the old host

We have overlapped for up to 2 months with old and new hosts just to make sure everything is set.  You always back up your data yes?  Why would you not want to have a backup with your entire website?

Good luck!

The turn around is hard to predict.  I would ask as many Moz forum members to do what you and I did from a desktop and mobile device.

Several ways to go about this. Here is my take.

Buying links - "For only $50 you can get 10,000 links on sites with a PR of 3 or more". People were setting up junk sites and generating a lot of low quality links. It worked for a while, but then Google figured it out and penalized folks for it. Now the reverse has happened, people are having to pay to get links removed or for someone to help them with a disavow, also people have used negative link techniques to build bad links to competitors to hurt them (this has happened to one of my sites).

Buying followers - Similar on twitter, pay to get followers! People setup fake accounts and then sell them so that people can say they have a big number of followers.

Buying likes and shares - I have no idea who Quuu is. No idea who these people are that they use to promote/share something. For the sake of conversation, let's assume that every single person who shares content for Quuuu is legit and that I can pay to have 30,000 shares - will they buy from me? are they really relevant to my niche? On the Quuu website they list that they have followers categorized by topic, but again, unless you are a large company with a wide appeal 99% of the people that they reach will not be relevant IMHO.  Why else do you get a reach of 10k people for $10? You are going to only touch a very small percentage of that 10k who will be interested and buy from you. You can probably curate those people just as easily through your standard social media work etc.

For all the examples above, relevancy is just flushed down the toilet. Sure I have 10,000 links or followers or shares - but if none of those people are interested in buying my product, I have gained nothing but a failed ranking strategy or exposure to people who do not care about my company and my product.

I would check your title tags.

Agree with Lesley, one other thing.

We moved to using Bootstrap http://twitter.github.io/bootstrap/  It was developed by Twitter and is open source for our front end development.  This gives you a bunch of building blocks for putting a page together with "parts" that have been vetted to be as cross browser/platform friendly as possible.

Just a couple of under the hood things to check.

1. Are you sure your robots.txt is setup correctly. Check in GWT to see that Google is reading it.

2. This may be a timing issue.  Errors take 30-60 days to drop out (as what I have seen) so did they show soft 404 and then you added them to robots.txt?

If that was the case, this may be a sequence issue.  If Google finds a soft 404 (or some other error) then it comes back to spider and is not able to crawl the page due to robots.txt - it does not know what the current status of the page is so it may just leave the last status that it  found.

3. I tend to see soft 404 for pages that you have a 301 redirect on where you have a many to one association.  In other words, you have a bunch of pages that are 301ing to a single page.  You may want to consider changing where some of the 301s redirect so that they going to a specific page vs an index page.

4. If you have a page in robots.txt - you do not want them in Google, here is what I would do.   Show a 200 on that page but then put in the meta tags a noindex nofollow.

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710

"When we see the noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it"

Let Google spider it so that it can see the 200 code - you get rid of the soft 404 errors.  Then toss in the noindex nofollow meta tags to have the page removed from the Google index.  It sounds backwards that you have to let Google spider to get it to remove stuff, but it works it you walk through the logic.

Good luck!

I would Rx Screaming Frog - it does good work and will tell you what the Spider sees.

As far as what it follows - that depends on what page you start on.  You can watch "live" in the ScreamingFrog spider as it runs through the sites and finds pages.

Overall, you just want to make sure it can find all your pages via a crawl and then focus on any errors found such as 404s or even 301s.

I have to digress on 301s and then bring up soft 404s

You have to watch about sending too many 301s to a single page.  Sometimes you have to do this, but I have also seen Google showing soft 404 errors when you have a bunch of many paged 301ing to a single page on sites we manage.

There is a subtle thing we have found on how Google thinks about 301s as it relates to 404s

http://googlewebmastercentral.blogspot.com/2010/06/crawl-errors-now-reports-soft-404s.html

They state that to correct soft 404s one thing you should look at is 2.b "Should redirect to a more accurate URL"

Google prefers that a 301 is more of a one-to-one or several to one.

There was also a post in SEOmoz on this

http://www.seomoz.org/blog/301-redirect-or-relcanonical-which-one-should-you-use

If you send too many 301s to a single page - like the home page, this may look like you are trying to manipulate link juice.

I would be so bold as to say Google prefers a one to one relationship for 301s vs a many to one.

Options

1. If you want to just get rid of pages already in the index

After 20 days, show a 200 on the page, update the message to say that this product is no longer available with a link to your search page but then add the noindex meta tag to that page.  This will allow Google to spider, but remove the page from the index.   You would also need to remove all links on your site to this page after 20 days as well.

Leave up the page for 6 months - 1 year and then setup a 404 as the page is dead and out of the Google index and the 20 day deal has been long gone.

This will get those pages out of the index, tell users where they need to go in case they land on the page and minimize any 404 or soft 404 errors in Google webmaster tools.

2. Keep them out of the index to start with

I am assuming that since these pages are only up for 20 days, they do not have time to really gain any search traction to start with and so would not show up in the SERPs (or rank that high if they did).

If that is the case, why not put all these pages in a separate folder that you block with robots.txt and then no follow all links to them.  Keep them out of the index to start with.

Sounds like you are optimizing a category type page above the product pages anyway and so you just focus on optimizing the category page vs the product pages themselves.

Beyond that, would need more specifics on the how and the why of what you are doing to try and figure out the if and the when of next steps

You did not lose the traffic - it is just being classified as something else. I am going to predict that you saw an increase in Organic traffic. Right?

http://searchengineland.com/60-direct-traffic-actually-seo-195415

Above is a classic article of how Groupon deindexed page to look at this and they found for their traffic 60% of direct traffic was organic. Your % will vary but you get the idea.

Your specific issue is due to the change in http to https - as mentioned in the Groupon article https://tools.ietf.org/html/rfc2616#section-15.1.3 specifies that when moving from http to https referral data is supposed to be stripped out.

You did not lose traffic but maintained it or gained it. If I could borrow $20 from EGOL I would bet that your revenue increase is possibly due to the fact that your site is now 100% https and people trust it more, which can impact purchase intent.

Cheers!

I have not used this tool in the way you mention, but I love this tool and it is worth every penny

http://www.screamingfrog.co.uk/seo-spider/

It states the paid version will "search for anything in the source code of a website with the custom source code search feature".   Email the owner and he can confirm.

FYI - this tool will also find all those annoying 301s from menus with typos and all kinds of good stuff.

You can use the analytics to tell where the traffic drops are coming from but I think that you need to get to why Google or some other search engine traffic is dropping.  So I totally agree with what EGOL mentions, but I think you already have a global grasp that traffic has dropped.  Here are what I would suggest are next steps to then fix the issues of why the traffic has dropped.

Odds are is that technically you may have a problem going on.

Go through your GWT reports for crawl errors and HTML optimization etc.  We relaunched and it gave us all kinds of clues on what to fix.

You need to double check all of your 301 paths - I bet that there are some holes.  This is down and dirty detail work.   I bet that they are probably not as correct as you think.  Don't assume that if you test a few the rest are ok.  Dont assume that the developer says that they are working that they are working.  Run them through a tool to make sure that you are sending a 301 response.  I had a site that was using 302 responses vs 301 (a temporary vs a permanent redirect).  Verify, verify, verify.

Did you change title tags on your pages.  Titles are a big signal and if you change them, you can see a drop.  You can look at your traffic and ranking analysis to look at sample pages.  If you did do a radical change, you may want to go back to your original title tag methods.

Run site speed tools

https://developers.google.com/speed/pagespeed/insights#url=www.brickhousesecurity.com&mobile=false

See if your speed has dropped.  The GWT will also show over time if your site is speeding up or slowing down.  You may need to look at your server setup.

Verify the HTML

http://validator.w3.org/

Did you setup

It looks like you may have changed a bunch of things at once and so it is hard to tell what you changed so see what was impacted.  Usually in cases like this, Google is trying to figure out the changes and so it may take a while to sort out.  I just gave some examples, but you need to review everything that has changed and see what the differences are.   There are some things, even with a 301 redirects that are correct, if you changed the URL structure and title tags and the new site has HTML that does not validate, Google may take a while to sort it out.  I had a site that we did a complete overhaul and it was 6 months before the traffic came back and that was with pretty good controls in place.

Good luck!

It sounds like you have something going on like what Matt Cutts talks about here

http://www.youtube.com/watch?v=KBdEwpRQRD0

You have a result showing up in the SERPs, even though it is in robots.txt.   Basically, the reason it is still in the SERPs is because other pages are linking to the URL on your page.

I am going to assume that you want to keep these pages out of the index.

As you already have pages in the index, you need to get them removed, not just block them.  I would suggest using  a noindex meta tag and then letting the crawler crawl the page.  The robots.txt stops the bot cold and does not let it read anything else.  It does not let the bot read the meta tag.  If you let it read the noindex meta tag that tag directs Google to take the page out of the search results.

https://support.google.com/webmasters/answer/93708?hl=en

"When we see a noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it."

That said, if you have made a mistake and have been blocking Google when you did not mean to.  Make sure that you do not use the noindex meta tag on those pages, and make sure you are not blocking Google in your robots.txt.  If that is the case and you are still seeing the wrong info in the SERPs, you do need to wait a little while.  The updates will not be instantaneous and may take a few weeks before being updated in the SERPs.   In the mean time, just double check that everything in your robots.txt is correct etc.

Be patient and good luck!