Welcome to the Q&A Forum

First rule of hiring an SEO firm, don't hire a firm that reaches out to you randomly and tells you that you have something broken and they can fix it.  FYI, I also know a prince in Nigeria who needs access to your bank account and would be happy to pay you for it.

That said, the whole pages on site vs pages ranking in Google is a valid measure.  If most of the pages of your site are not ranked in Google, then that may be an indicator that you have a lot of low quality pages and this could be an issue.  This type of metric is a good one for check in the health of your site.

That said, your have to figure out if this matters to your site or not and it is pretty easy to do. Various ways to look at this.

Log into Search Console and if you have a sitemap submitted on your main dashboard for sitemaps you will see URLs submitted vs URLs indexed. At a basic level, how many pages did you show to Google and then how many pages did Google say, yes, these are good enough to be indexed.

Use ScreamingFrog or the MozCrawl, see how many URLs are found by crawling your website.  How does this total compare to what you show in your sitemap?  Do you have a bunch of pages that are just tag cloud pages or a page that is just a resorted or printer friendly version (aka duplicate) of another page?  You probably need to block access to those (and make sure they are not in your sitemap).

ScreamingFrog has a API to connect to your GA data.  You can then also pull what type of search traffic do these pages get.  This gets back to your ranking question.  If you have a page that Google is ranking for something, it should get some search traffic to it.  Depending on how much traffic you get overall to your site, you may need to look at a 3 to 6 month time period.  You can now say, out of all of the URLs on your site, if there is a group of URLs that is not getting any traffic from Google for the past 6 months, they are probably not ranking for any key words.  Why is that?  Look at those pages.  Are they crappy pages with very little or duplicate content?  Updating a meta tag will not do anything if you have a crappy page to start with.  Maybe you need to noindex them or 404 them and get them off your site.  Maybe you need to add content, write more, include some graphics etc.   Maybe you have duplicate or similar pages you need to combine into one super page and redirect the other page to it.

Good luck!

Just as a follow-up to Tom's great post.  If you were wanting to test a robots.txt setup, especially if you were using a wildcard or using an allow combined with a disallow, Google Search Console under the Crawl section has a robots.txt Tester.  You will see your most recent robots.txt file there that Google has a copy of.  You can then modify that version and then enter a URL at the bottom to see if everything is set correctly or not.  It is pretty handy, especially if you have a big robots.txt file.   Note that this tool does not change how Google crawls your site or your robots.txt file, it is just for testing.  Once you find the configuration that works, you would still need to update the robots.txt on your server.

It is my understanding it checks its own version and it does this weekly (Moz staff please chime in to correct and/or clarify further).  You can run a page report "on the fly" using Moz as well.  Update your page then run the grader to see the changes.

I think the key point there is not the schema markup per se, but the fact that the reviews can be original content on that and that can help the page rank.  That is the angle you want to start with with the Schema being the icing on the top.   Improving the thin pages is the key first step.  Lots of stuff out there on how Google does not like thin/duplicate content.

Markup for reviews will not help you rank for SEO directly.   That said, if the page already ranks, then Schema markup for reviews that then shows in a rich snippet helps with your CTR on that position (and this helps solidify your position as Google sees the action on your result).

I cant find the reference, but wanted to mention, there was a recent article showing that you actually get a higher CTR in position #2 with some sort of rich snippet vs something in #1 position without a rich snippet.    See if you can find it (or maybe someone else here can) for some reason my Google skills are failing me right now and I cannot find it. I must need more coffee. :-).

I cant release the actual data for client privacy reasons, but I can say from my own experience, we had a bunch of our pages on page 1 (usually top 3) for various local searches with review data marked up and the snippets would show for the reviews.  There was a change in how the schema validated and we saw a ~6% drop in traffic to those pages when the snippets on the search results went away - this was with no significant ranking changes.  We corrected the markup, Google crawled the site like crazy within 24 hours and we had our snippets and traffic back.  It was pretty amazing to watch and validated everything I had seen others talk about.

Separately, I just got back from PubCon.  Ton of people talking about using markup, most notably Duane Forrester from Bing. He spent half his keynote talking about advances in machine learning and the other half talking about how we all needed to use Schema.org.  In other words, computers are getting smarter and smarter, but if you can, please help them out with markup.

Hope this helps you figure things out.

The turn around is hard to predict.  I would ask as many Moz forum members to do what you and I did from a desktop and mobile device.

Looks like Google is pulling from a US based Yelp profile for a company with the same name.  Strange considering this is Google.com.au

http://www.google.com.au/imgres?imgurl=http://s3-media1.fl.yelpcdn.com/bphoto/zNjdBcEiNvc0YuXfuJvOJw/ls.jpg&imgrefurl=http://www.yelp.com/biz/community-first-credit-union-jacksonville-8&h=250&w=250&tbnid=ctUt5usQbxTaNM:&q=cfcu&tbnh=151&tbnw=151&usg=__mX0EGbrF8xnXjSeNBMRlSGiUn9U=&docid=Tww1toT0IL7f8M&itg=1

I checked your logo markup on your site http://www.communityfirst.com.au/Assets/1333/CFCU_logo_cent_RGB_square.png and it checks out ok in the structured data testing tool https://developers.google.com/structured-data/testing-tool/

Here is what you need to do.  If you look at your search result, right under the logo and Wikipedia snippet there is a link labeled "Feedback".  Click that and follow the instructions to let Google know about the error.

Remember these are all computers making assumptions and nobody is perfect.

Google Keyword Planner - https://adwords.google.com/KeywordPlanner

Keyword IO - http://keywordtool.io/

Word Stream - http://www.wordstream.com/keywords

Word Tracker - http://www.wordtracker.com/

More great ideas here - http://www.seerinteractive.com/blog/spark-new-ideas-with-these-keyword-research-toolstips/

and here - https://moz.com/beginners-guide-to-seo/keyword-research

and here - https://moz.com/blog/category/keyword-research

Invisible to who?  If they are invisible to the search engines (e.g. blocked via robots.txt) they could not be harmful, but if you gain any link or other quality equity through those pages, your will not get any additional domain authority or organic traffic to them.

That said, if the hidden categories were resorts of the main categories on your site then you would want to hide them from the search engine as they would be seen as low quality duplicate content.  Otherwise if they were visible they could negatively impact your site.

If Hubspot is preventing the redirects then that is your issue.  Google ranked (and sends traffic to) the Hubspot pages.  Unless you can setup the 301s from the old site to the new one, your new site will just look like duplicates of the old one.

Another option to see if you cannot get the 301s to work is to setup canonical links from the old content to the new.  Google most often treats this like a 301.    This could be a good interim step before getting 301s setup.   If Google updates with the new URLs in the SERPs and if you can get most of any external links pointing from the old blog to the new blog, you could then simply shut down the old block.  The canonical link would have updated the SERPs and the external links to the old blog have been updated to the new one.  Therefore, your old blog posts would not have any value to you (as they are not ranked in Google and have little or no links to them) and it would be ok to terminate.

Good luck!

Agree with Martijn.  Technically, Google does not want any pages in the sitemap that 4xx or 3xx or have any other result than a 200.

I would say this, if your site is being rebuilt, having a sitemap that is accurate and that updates when you update the site is a basic requirement.   The fact that there is a high cost for fixing this issue is baloney. It sounds like the devs did not build the site correctly the first time if they do not have a way to update the sitemap automatically.

You could generate the sitemap yourself

You can use tools like

http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/

Or read tutorials on how to use Screaming Frog to create a sitemap

http://www.hmtweb.com/marketing-blog/dirty-sitemaps-how-to-download-crawl/

Frankly, the annual cost of Screaming Frog (about $150 a year) gets you so much more than just sitemaps.  Buy Screaming Frog, have it generate your sitemap and ask the devs to upload it.  If you have a site with several thousand pages, just running Screaming Frog monthly would help you find issues on your site that is well worth the cost.    Search "Screaming Frog" here in the forums and you can see that this is one of the "swiss army knives" of technical SEO.

This only works if you have downloaded all the HTML files to your local computer.  That said, it works quite well!  I am betting this is a database driven site and so would not work in the same way.

Regex: href=("|'|)http:(?:/{1,3}|[a-z0-9%])|[a-z0-9.-]+.

This allows for your link to have the " or ' or nothing between the = and the http   If you have any other TLDs you can just keep expanding on the |

I modified this from a posting in github https://gist.github.com/gruber/8891611

You can play with tools like http://regexpal.com/ to test your regexp against example text

I assumed you would want the full URL and that was the issue you were running into.

As another solution why not just fix the https in the main navigation etc, then once you get the staging/testing site setup, run ScreamingFrog on that site and find all the 301 redirects or 404s and then use that report to find all the URLs to fix.

I would also ping ScreamingFrog - this is not the first time they have been asked this question.  They may have a better regexp and/or solution vs what I have suggested.

Google ignores everything after the hash to start with, so you do not need to block it to finish with.  It is a clever way to pass parameters without having to worry about Google getting lost.

Two things I found when using the Wayback Machine to look at your site

1. Did you setup 301 redirects from old to new content?

https://web.archive.org/web/20150225025506/http://advanced-driving.co.uk/

I pulled links to various and random advanced driving groups and they 404ed vs 301

http://www.advanced-driving.co.uk/advanced-driving-lessons/region5/

http://www.advanced-driving.co.uk/advanced-driving-lessons/region6/

Also on the page for driving lessons

https://web.archive.org/web/20150219123145/http://www.advanced-driving.co.uk/driving-lessons/

I pulled links to various and random advanced driving lessons and they 404ed vs 301

http://www.advanced-driving.co.uk/driving-lessons/norwich/

http://www.advanced-driving.co.uk/driving-lessons/liverpool/

Looks like you did not properly migrate the URLs from the old site to the new site.  You should be seeing 404 errors in Google Search Console and so that can be a starting point to find what pages need to be updated.  It sounds like you tracked rankings for pages on the old site, start with those URLs and make sure they are 301ed to the correct new URL.

2. Site structure

On the new home page, you are not linking into any of the pages such as your Driver Instruction and Traffic Reports page.  Likewise, your header and sidebars used to link into your content.  Your new design is cleaner, but you pretty much blew up your previous internal linking structure.  Google will use your internal links to find pages to crawl and also to determine what pages are important on the website.  I see much much less of this on the new site.  You may want to consider updating how your internal linking is structured so that you are showing your users and Google what your most important pages are.

Good luck!

Just seeing the other responses.  Agree with what EGOL mentions.  A content audit would be even better to see if there was any value at all on those pages (GA traffic, links, etc).  Odds are though that there was not any and you already killed all of it with the noindex tag in place.

Couple of things here.

1. If a second Panda update has not occurred since the changes that were made then you may not get credit for the noindexed content.   I don't think this is "cheating" as with the noindex, it just told Google to take 350K of its pages out of the index.  The noindex is one of the best ways to get your content out of Google's index.

2. If you have not spent time improving the non-syndicated content then you are missing the more important part and that is to improve the quality of the content that you have.

A side point to consider here, is your crawl budget.  I am assuming that the site still internally links to these 350K pages and so users and bots will go to them and have to process etc.  This is mostly a waste of time.  As all of these pages are out of Google's index thanks to the noindex tag, why not take out all internal links to those pages (i.e. from sitemaps, paginated index pages, menus, internal content) so that you can have the user and Google focus on the quality content that is left over.  I would then also 404/410 all those low quality pages as they are now out of Google's index and not linked internally.  Why maintain the content?

Thanks Donna!

Here is the information you need on qualification

https://support.google.com/ads/answer/6230381

Google will be performing a background and license check in addition to performing a "reputation assessment" of the business online.  I would assume that if this business has good reviews on Yelp or Google that this would help.

A good overview of the program is here

http://searchengineland.com/google-home-services-ads-plumbers-locksmiths-san-francisco-beta-226469

The service just launched the end of August so it is pretty new

http://www.buzzfeed.com/carolineodonovan/googles-home-services-product-finally-officially-launches-to

keep that in mind when taking your client through the process.

My experience with subtopics in URL structure is that they are over-rated.  Use the main category to help the user know where they are in the site and potentially about the topic that the page might be about.  If you want to drop keywords, do it in the category or in the slug for the name of the page.  You can work it in there and it gives you more flexibility.  This also helps with making your page be closer to the root folder vs ending up being too far down in a folder structure.

When I have used sub-categories you always end up with content that could be in two different ones and so then you have to decide which one is the better one, etc etc.  You will end up having to rework your URLs later due to issues with your sub-categories.   The only way I have seen subcats work well is when you have something like /state/city/zip or something else where your end item is only in one cat, subcat, etc.

I would not do the /peanut-butter/ to redirect to /peanut-butter/subtopic-1 - that makes no sense from an organizational perspective.  If peanut butter is not a strong enough category by itself, it should not be a category to start with.   You need to rethink what your category topics are.  Ideally, /peanut-butter/ is a keyword combo you want to rank for and has great traffic potential that converts.  It should be a hub page for your site for that topic.

Find good categories, work in the keyword into that category or if not work it into the slug for the name of the page.  If you want a good example, look at how the Moz site is setup.  Also, remember that keywords in the URL are good for SEO, but you really get more bang for the buck for a good title and content and links into that content.  Dont overthink the URL.

Good luck!