Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
Hi Nancy,
To be clear, when you say that there were 15,000 blocked pages, do you mean that Google reported that its crawler was blocked from crawling them? Or that Google was blocking them itself, thinking that they're malware or something like that?
The term "blocked" makes me think there's a technical issue here, not a penalty. Were these 15,000 pages important pages? I see that Google still has around 1,950 of your sites' pages in its index. How many should it have?
Best,
Kristina
Hi Alicja,
When you say "some of your content," how much do you mean? If brandB copied a paragraph or a few sentences and used it as an intro to a unique piece of content, you should be fine. Google understands that there's naturally going to be some overlap on the same domain.
If they copied an entire page, yes, I would recommend using canonical tags, or removing the content. I assume that brandB understands that a canonical tag will mean their version of the page probably won't rank on Google?
Best,
Kristina
Hey Shawn,
1. If GWT is showing a 404 without a "Linked from," I wouldn't worry about it too much. Sometimes Google tries to recrawl URLs it crawled once in the past, but that doesn't mean it's causing a problem. I would double check that you don't have any internal or external links to "blog/blog," then just "mark as fixed." There's no point to disallow Google from crawling a page that isn't there in the first place.
2. Nope. The only danger of having a long robots.txt is that you may get lost while checking your rules and accidentally disallow a page you want in the index. Whenever you add a rule to robots.txt, make sure you go to Google Search Console and use the robots.txt Tester to check if you've disallowed any important pages.
Good luck!
Kristina
Nope, having more or less external links will have no effect on the crawl budget spent on your site. Your crawl budget is only spent on yourdomain.com. I just meant that, Google's crawler will follow external links, but it won't until it's spent its crawl budget on your site.
Google isn't going to give you a metric showing your crawl budget, but you can assume how much it is by going to Google Search Console > Crawl > Crawl Stats. That'll show you how many pages Googlebot has crawled per day.
Hi Shawn,
If these are from crappy scraper sites (which it looks like they are), then I'd add it to your disavow list, yes. I don't think that'll get Google to remove your error in GWT, but remember: GWT is a separate algorithm from the actual algorithm because Google doesn't want us to see how sophisticated they are. If GWT sees "errors" in links, but you've properly disavowed, I'm fairly confident the actual Google algorithm won't penalize you.
For anyone else finding this thread: make sure that you check the sites that are linking to your broken pages before disavowing! If they're good links but out of date or mistyped, just update them. 
Best,
Kristina
To be clear, Googlebot will find those external links and leave your site, but not until they've used their crawl budget up on your site.
Hmm, a main menu link is usually great. Have you double checked to make sure those links are in HTML, so Google can read them?
Hi there,
There are a lot of components to your question, so I'm going to answer each as thoroughly as I can. Apologies if I say something that you already know (which will probably be a lot of this, since I'm trying to be thorough).
Why aren't my links showing up in GWT/Search Console?
There are some things to note about GWT/Search Console:
Which links are counting for Google's ranking algorithm?
If you read through Matt's post, he shows you how you can use tools to find which links should be passing value to your site. If you only have one or two links that you want to check, I like to follow these steps, to really see what Google sees:
Why aren't I ranking when I have better inbound links?
Inbound links, while incredibly important, are not the be-all end-all to SEO. If you're not ranking as well as you think you should be, check these things:
I know this is probably more info than you needed, but I'm not sure why you feel that you're not ranking where you should be, so I wanted to share my methods so you can tell me where you have questions, or where you disagree.
Let me know your thoughts!
Kristina
Yeah, the 301ing could be creating an issue. Google Webmaster Tools really isn't good at following through redirect paths. After hearing that, and reading everything that Matt researched, it's probably worth reaching out to your .edu partners and asking them to update their links to the proper URL for your site (including if it's https now!).
Also, forgive me if I just missed your answer, but have you set up an https profile of your site in Google Webmaster Tools? Google absolutely distinguishes between the two - my site upgraded about a year ago, and I watched traffic drop in one and jump in the other. I keep both now to monitor inbound links that are to either. It's worthwhile creating a profile with and without www as well.
Weird! Quick question: does your site have a decent DA? I wonder if this site is selling links to weak sites, and they're trying to mask that by also linking to high quality sites, like yours.
Hi Issa,
You're right, duplicate content and bad usability could be triggering the slow rolling Panda 4.2, but I'd dig in a little more (apologies if you already did this research):
You mentioned 200 pages are potentially duplicate; how many are on the site in total? If you have thousands of pages indexed, 200 duplicates probably aren't going to cause a Panda penalty.
How similar are these postings? Just the page title? Or is the entire page extremely similar in content? (To answer this: if you made a keyword cloud for these similar job descriptions, would they show roughly the same mapping?)
If it's just the page title that's similar, make sure to set the pages apart by including the name of the hiring company (which I assume makes the different positions unique) towards the beginning of the page title
If the entire page is similar, then add more content to make the pages more unique, like a blurb about the hiring company, how long the job has been up, how many applicants the job has (if available), etc.
Either way, make sure you don't have any old jobs that still have live pages! If possible, I'd redirect them to a similar job posting.
Like John asked, did your traffic drop dramatically one day, or has it been tapering off? If it's tapering off, I'd guess it's not Panda.
And, last, which pages lost traffic and rankings? Which keywords dropped in rankings? You may be able to tell how you were penalized by which keywords were most affected.
Hope this helps,
Kristina
Hi Sam,
First, to be clear, campaigns will be overwritten if visitors come from any other source, it's just direct traffic that the campaign parameter holds on to. Here's Google's direct quote from its article on campaigns and traffic sources:
Existing campaign or traffic source data for a particular user will be overwritten by new campaign or traffic source data, regardless of the configured timeout period.
If you look at the flowchart a bit below that quote, you'll see that Google starts by looking for new campaign data, then looks for traffic source data, then, if it doesn't find either of those, uses existing campaign data.
That means your theory could still be correct, but only if all of the visits you're still seeing come in are just from direct visits. You can check this theory by using the % New Sessions column - if it's 0%, you're right, these are just returning visitors, and the best I can recommend is that you set up filters to make these show up as "direct." If it's not, though, (and I'm suspecting it's not, because I doubt this would make a large enough number for you to be concerned and reach out for help), you've still got some of those campaign URLs floating around for the public.
Here's how I'd go looking for them:
I hope this helps! Let me know if you still have any questions, or if anything stumps you along the way.
Best,
Kristina
This is something I've seen on websites of all sizes. I've worked with a couple of well-respected colleges - the type of site that you would think Google would automatically trust - and it took anywhere from 2 weeks to 3 months from new pages being indexed to ranking as high as they can go without more help. We actually charted the rise of one article, written by a professor, with a link off the homepage of one of the college sites, and it took over a month to stop rising in rankings. For reference, this was about a year ago.
So, I think Alan is exactly right: I think that Google wants a lot of confirmation signals that a page is legit, and it takes awhile for its crawler to identify all of them. Google values how long a page has been around, and click through rate, which comes over time.
Hi Schmeetz,
I've had a rough time getting reviews to work for schema as well. The structured data testing tool can give you a positive when something's actually missing that'll stop Google from displaying your stars. The nice thing is, if you're using structured data, Google will update their listings as soon as they recrawl the page, so you can test things fairly easily. Use Google Webmaster Tools to "Fetch as Google" and "Submit to Index" to get a page with new code recrawled within an hour or so.
I don't think you need dates in order to get star ratings on SERPs. I haven't needed them on my company's site. But, I noticed that all of your markup is using content tags rather than marking up actual content on the page. My guess is, that's the issue, that Google doesn't trust content that isn't seen on the page. Have you tried marking up the actual content?
_Note: _Google will also sometimes blacklist domains from displaying reviews. This isn't an issue for you, but for others reading this: you can test it by searching for "site: [product URL]". If Google displays the ratings then, it's not a technical issue, it's a blacklisting issue.
Test out marking up visible HTML text and let us know how it goes!
Best,
Kristina
Yep, they're a little vague here! But the answer is: Google will crawl your site at whatever rate it wants (it's probably crawling Amazon 24/7), unless you limit how much it can crawl in Google Webmaster Tools. Then, Google will crawl your site at whatever rate it wants, unless than rate is higher than the limit you put in, and then it will limit itself.
If you're anxious for Google to crawl your site more because a) you have something that's changed and you want Google to have it in their index, or b) because you're hoping it'll affect your rankings:
a) If there's specific information that you want Google to update its index with, submit the URL of the page that's new or changed into "Fetch as Googlebot" and then, once you fetch it, hit the "Submit to index" button to the right. I work on a site that's a DA 58 and fetching something as Googlebot updates the index within an hour.
b) How much Google crawls your site has to do with how important your site is; forcing Google to crawl your site more will not make it think your site is more important.
Hope this helps!
Kristina
Hi Gemma,
Strange! Typically, %20 is the symbol that content management systems use to convert spaces into allowable characters in URLs. Have you found any URLs that were written in the HTML with an accidental space?
That said, I know I'm a Moz associate and all, but Moz and Ahrefs are not nearly as good at understanding the web as Google; it's completely possible that these are errors that their crawlers are picking up, but Google isn't having a problem. Try searching for "site:[duplicate URL]" to see if Google is indexing this "duplicate content." I just checked with the example you provided, and it's not in Google's index.
If some other duplicate content URLs are in Google's index, then I'd use Google Analytics to determine where the traffic is coming from to these pages, in order to find where the URLs are written incorrectly.
Hope this helps!
Kristina
Hi Mitesh,
I know that Omniture offers hourly alerts, and is the most commonly used analytics platform after Google Analytics, as far as I know.
Personally, though, I'd stick with GA and hire a developer to use GA's API to create the alerts you're asking for. Pretty much any analytics tool other than GA is going to cost you an arm and a leg, and all have their limitations. A good dev will be able to create the exact report you need with GA's API.
Sorry this wasn't more helpful!
Kristina
I've heard theories that Google doesn't like you changing your page title too much, but I've never seen that substantiated. So, I recommend testing this for yourself:
**Paid option: **For brand searches during the holidays, run an AdWords Search ad that uses a seasonal message. Compare the metrics of the seasonal messaging vs the metrics of the ad for the rest of the year to see what searchers are more likely to click on. This will tell you searchers' preferences, although it won't tell you Google's reaction to you changing homepage content.
Unpaid Option:
Identify the non-seasonal vs seasonal keywords that you'd like your homepage to rank for. Start tracking them in a tool like Moz. (I don't think that just changing the page title will drastically change rankings, but you want to be sure.)
Change your page title, and have Google recrawl your page immediately using Fetch as Google. I've been able to get new page titles up within an hour that way.
Keep the page title up for the shortest time span you can to get a significant amount of traffic.
Ideally, you'd get enough traffic each day that a day's worth of traffic is fairly predictable, so a change in the page title will show an obvious spike or drop in traffic, depending on rankings and click through rate.
If your traffic varies a lot day by day, though, change it once a week.
Look at the changes in traffic, rankings, and conversions.
Hope this helps, and let us know what you find out!
Kristina