Welcome to the Q&A Forum

Just a heads-up that you'll want to get this cleaned up as quickly as possible, Matthew. Time really is of the essence here.

Once this issue is recognised by the crawler as being widespread enough to trigger a warning in GSC, it can take MONTHS to get the hacked warning removed from the SERPS after cleanup.

Get the hack cleaned up, then immediately start submitting the main pages of the site back to Fetch as Google tool to get them recrawled and detected as clean.

I recently went through a very similar situation with a client and was able to get the hacked notification removed for most URLs within 3 and 4 days of cleanup.

Paul

I'm assuming you've been working on this, Frans? The redirect for http://www.bankstellenshop.com//bankstellen/u ends up on page http://www.bankstellenshop.com/bankstellen/goedkope/u.html. Is that the behaviour you want?

Note that the above is actually a chain of 301 redirects. It goes
http://www.bankstellenshop.com//bankstellen/u 301 redirects to
http://www.bankstellenshop.com/bankstellen/u.html which 301 redirects to
http://www.bankstellenshop.com/bankstellen/goedkope/u.html

It would be cleaner, more efficient and preserve rank influence better if the middle redirect was removed and the initial URL redirected to the final in one hop. (Some "rank juice" is lost with each redirect in a chain.)

Hope that helps?

Paul

Search results pages should pretty well always be no-indexed. Definitely the case in this instance.

Which makes sense if you think about it. What value would there be in a search result in the SERPS for every variation of date, time, number of beds etc? None. And your site would have a literally infinite number of those pages for the search engine to try to index, as every combination of variables creates orders of magnitude of page variations.

So long story short - keep 'em no-indexed.

Hope that helps?

Paul

It looks like the devs have cleaned up most of the obvious stuff, Matthew, so I'd get to work resubmitting the pages that were marked as hacked but now longer show that issue.

Do make sure the devs keep working on finding and cleaning up attack vectors (or just bite the bullet and pay for a year of Sucuri cleanup and protection) but it's important to get those marked pages discovered as clean before too much longer.

Also of note - your site's server's Apache install is quite a bit out of date and you're running a very old version of PHP as well that hasn't been getting even security updates for over a year. Those potential attack vectors need to be addressed right away too.

Good luck getting back into Big G's good graces!

Paul

P.S. Easy way to find the pages marked as hacked for checking/resubmission is a "site:" search e.g. enter **site:brolik.com **into a Google search.

P.P.S. Also noted that you have many pages from brolik-temp.com also still indexed. The domain name just expired yesterday, but the indexed pages showed a 302-redirect to the main domain, according to the Wayback Machine. These should be 301s in order to help get the pages to eventually drop out of the SERPS. (And with 301s in place, you could either submit a "Change of Address" for that domain in Webmaster Tools/GSC or you do a full removal request. Either way, I wouldn't want those test domain pages to remain in the indexes.

Having your own sandbox to screw around in is a great place to start, Jeff. It will let you begin to get a feel for some of the things that need to be taken into account as you build an SEO strategy for your site.

But the difficulty is your new site will have very little traffic so it will be very hard to tell how much impact your changes and optimizations or actually having.

Once you get to the point where you feel reasonably comfortable with the basics, and are pretty sure you can operate without creating major risk for the site you're working on, a great next step is to adopt a favorite local charity which has a website in need of assistance and optimization.

Total win-win, as the website gets some help that is more experienced than what they can themselves provide, and you get the opportunity to work on a site that has enough traffic that you can start to see the actual effect of what you're doing.

Obviously you are not getting paid for any of the work, so you can make it clear that their expectations should be in line with the fact that you are learning your way along in the process. But they are much more likely to allow you the opportunity than a commercial website that is depending on their site for their income and would be unwilling to trust a newbie with the keys.

This approach helps address the standard conundrum in SEO: you shouldn't take on commercial clients until you have enough experience, but you can't get enough experience until you actually can take on some bigger websites.

Paul

Setting redirects to apply only to search crawlers is pretty much their definition of cloaking, Felip3. The site could end up in a world of hurt if/when caught.

What's the reason for wanting the crawlers to be redirected differently than users? Maybe there's a better way to accomplish what you need in a way that doesn't break their terms of service.

Paul

In the "old days" (yup, I go back that far), Google's search index crawler wasn't all that powerful. So it would ration itself on each page and simply quit trying to process all the content on the page after a certain number of links and certain character count. (That's also why it used to be VERY important that your content was close to the top of your page code, not buried at the bottom of the code).

The crawler has been beefed up to the point where this hasn't been a limiting factor per page for a long time, so the crawler will traverse pretty well any links you feed it. But I +1 both Andy and Mike's advice about considering the usability and link power dilution of having extensive numbers of links on a page. (This is especially important to consider for your site's primary pages, since one of their main jobs is to help flow their ranking authority down to important/valuable second-level pages.)

Paul

Another vote for Screaming Frog. Although you'll need the paid version to so this, it's such an incredibly useful tool, you'll want a license anyway.

Paul

That's what's now typically referred to as a mega-menu, taysir. It uses JavaScript for the fancy display but If it's coded properly, there's a fallback version that uses regular html/CSS. (You can test by disabling JS in your browser then revisiting the page.)

Unlike Andy, I'm never willing to "trust" that search engines can handle JS-based navigation. It's just too big a risk for me. (Though his opinion is perfectly valid too). So I always make sure something JS-based like this has the CSS fallback version properly coded as well to be safest. Then I can certain it's SEO-friendly in all cases.

Jut my $0.02, adjusted for exchange rate and inflation

Paul

This issue is resolved by adding a single self-referential rel-canonical tag to the header of each page of the site, Catherine. Once you've done that, the URLs that contain the parameters will automatically contain the canonical to the primary URL (because the pages' code are actually the same - it's just the URL itself that is changing. By which I mean - there aren't separate pages for each of the currencies. They're are all the same page code, with just the parameter added to the URL and prices dynamically changed.)

This does mean that the search engines would index the page with the default prices, which appears to be Euros.

For example, if your home page had a self-referential canonical tag, it's canonical tag would be

<link rel="<a class="attribute-value">canonical</a>" href="<a class="attribute-value">http://www.marcb.com</a>" />

While this may seem redundant, it also means that this URL https://www.marcb.com/?setCurrencyId=2 would also contain the above canonical tag, since the page is actually built from the same code. So it's canonical would point to the correct URL automatically, without having to do anything specific for all those variations. This is a core function of how CMSs (Content Management Systems) templates work. This time it works in your favour.

You definitely don't want to no-index those parameter-based variations even if you could. Once you get the canonicals properly implemented, you want the search crawlers to keep crawling those pages URLs so they can discover the corrected canonicals and understand that they are intentional dupes of the core page. They'll eventually drop the parameter-based URLs out of the index, which you can monitor in your Google Search Console, for example. There's a major benefit to the site if the search crawlers aren't wasting their time on duplicate/useless pages, as well as reducing potential issues with Panda/Quality algorithms, so well worth getting this corrected right away.

Hope all that makes sense?

Paul

Just to update, Taysir - here's a screenshot of the same page with JavaScript disabled. Notice how the menu gracefully falls back to just a CSS-based regular dropdown. This is much more likely to be consistently crawled by search engines than the JS-dependent version.

This is the standard best practice for this type of situation, both for usability and SEO. A webpage should "just work" in a basic browser, and then can add additional appearance/functionality improvements for "enhanced" browsers. This is obviously what's best for users, and also gives the (very basic) search crawlers the best option as well.

In other words - mega menus are certainly NOT "better" for SEO, but they don't have to a problem either, if implemented correctly.

Hope that helps?

Paul

zakc0AS

While theoretically logical, Google's own John Mueller stated last week that code to text ratio has absolutely no effect on crawling of a site, and in a followup question, he directly told me text/code ratio has no effect as a ranking factor either.

These used to be very minor considerations back when the search engine crawlers weren't as powerful, but no longer.

Fully agree with Pia that the idea of "above the fold" influencing ranking is nonsense as well. Given that the sweet spot for consistently high-ranking pages is ~2200 words, the idea that only the first paragraph or two are more important is unsupportable.

Hope that helps?

Paul

You absolutely do NOT want to redirect these links to the home page, Ian! These are spam links, coming from completely unrelated sites. They are Google's very definition of unnatural links and 301-redirecting them to your home page also redirects their potential damage to your home page.

You want them to return 404 status as quickly as possible. I'd also be tempted to use the Webmaster Tools remove tool to try to speed up the process, especially if these junk links currently form a large percentage of your overall link profile. (You'll need to find & remove the redirect that currently re-points them to the home page too, for the 404 header to do it's job of telling the search engines to drop the page from their indexes.)

As far as rankings issues, this isn't a potential dupe content issue, it's a damaging unnatural links issue, which is even more significant. These are the kinds of links that could lead to at least algorithmic penalty, or worst case, manual penalty. Either way, these penalties are vastly harder to fix after the fact than to avoid them in the first place.

In addition to the steps above designed to make it clear those links don't belong to your site, I'd keep a good record of the links, their originating domains, and when & how they were originally created due to the malware attack and your fix. That way you have essential documentation should you receive a penalty and need to submit a reinclusion request.

Hope that answers your questions?

Paul

Lazy load and infinite scroll are absolutely not the same thing, as far as search crawlers are concerned.

Lazy-loaded content, if it exists in the dom of the page will be indexed but it's importance will likely be reduced (any content that requires user interaction to see is reduced in ranking value).

But because infinite scroll is unmanageable for the crawler (it's not going to stay on one page and keep crawling for hours as every blog post rolls into view) Google's John Mueller has said the crawler will simply stop at the bottom of the initial page load.

This webinar/discussion on crawl and rendering from just last week included G's John Mueller and a Google engineer and will give you exactly the info you're looking for, right from the horse's mouth, Victoria.

To consider though - the blog's index page shouldn't be the primary source for the blog's content anyway - the individual permalinked post URLs are what should be crawled and ranking for the individual post content. And the xml sitemap should be the primary source for google's discovery of those URLs. Though obviously linking from authoritative pages will help the posts, but that's going to change every time the blog index page updates anyway. Also, did you know that you can submit the blog's RSS feed as a sitemap in addition to the xml sitemap? It's the fastest way I've found of getting new blog posts crawled/indexed.

Hope that helps!

Paul

Is there a reason you don't want the existing site to be available to users while the redesign is going on? the usual process in this kind of situation is to leave the current site alone, and built the redesign site on a dev subdomain which you block from the search engines via password or IP exclusions. Then when the redesign is done, you promote the dev version to live with all the new 301 redirects already in place.

An alternative, if the site won't be out of commission for very long, is to have the site return a 503 server header code. This simply tells the search engines the site is temporarily unavailable and NOT to index anything until the 503 is removed. This ensures that the temporary landing page doesn't get indexed as the new content for the site's pages.

But still, option number 1 is preferable. Would it work in this instance?

Paul

As usual, Mueller's answers can be problematic because they're actually kind of vague. (e.g. his use of "if you use one of the other methods, make sure to follow those instructions separately" in that seroundtable article) Because the question asked in that article is specifically about responsive sites, non m. separate URL versions.

Here's the best I can give you... On that guidelines page you ,inked, Google specifically provides instructions for how to either include the mobile-URL versions of pages in the rel-alternate tag or by annotating the desktop sitemap to include rel-alternate info for the mobile URLS.

It does not make any mention of saying "or you can simply include the mobile URLs in the sitemap as well." Google's usually pretty good about telling us when there is more than one alternate method, while indicating which one they prefer. in this vase, I have to assume the conspicuous absence of any mention of including mobile URLs separately means it shouldn't be done.

Still conjecture, but does that make sense?

I'd definitely say it's imperative that the rel-alternate/rel-canonical treatment must be in place. Beyond that, I suspect it's a crawl budget/crawl efficiency issue, not an actual "indexing will break if mobile URLs are in sitemap" situation. As such, I wouldn't want to prioritise an expensive solution to this over whatever other more high-impact projects might be awaiting funding.

Just for the hell of it, I'll tweet at the Google guys to see if I can get a direct response to "will it cause harm" and let you know if I hear back.

I know this is just another perspective, not anything definitive, but hope it helps?

Paul

You have a much bigger problem than what can be solved just with a robots.txt file, Mark.

All of those URLs are showing up because of a misconfiguration of your theme installation (likely caused by the theme developer) is allowing full display of all of the content of each of those directories. In addition to polluting your search results, as you've noticed, it's a also a pretty major security risk. You can see this in action by going to http://baltimoreelite.com/wp-content/themes/sintia/wpv_theme/assets/css/ What should happen when you go to that URL is you see a blank page, or receive a 403-Forbidden warning. Instead, you're seeing a full listing of the directory contents - bad news.

Since I don't know your hosting configuration, the easiest way to fix this issue is to add a line to your .htaccess file at the root of your site. This should correct for all such instances, You need to add this line:

<code>Options -Indexes</code>

If you're not familiar, the .htaccess file is a text file which you can edit with any text editor. You'll need to use an FTP program or the file manager in your hosting control panel to access it at the root of your site. (You may also need to enable "show hidden files" in your program). I always recommend backing up the existing file before editing just in case. You'll add this new line on its own line with a blank line between it and any other lines in the file. It can go near or at the top of your htaccess file.

Now getting those URLs out of the search index is going to take a bit more work. You'll want to implement the robots.txt exclusions like Andy suggested, and then you'll need to go into Google Webmaster Tools and use the Remove URLs tool to specifically request removal of the directories you blocked with the robots.txt and that you also want removed from the search results. (The robots.txt is a critical part of this process, as Google requires it be in place in order to process the removal requests.)

This, combined with the htaccess edit mentioned above, should keep those URLs from showing up again in teh future.

Hope that all makes sense. If not, be sure to ask!

Paul

The issue of the internal links of your site going through 301-redirects isn't being caused by the htaccess file, John - in fact that file is partially saving you from the bigger issue.

The problem is that your WordPress migration to HTTPS was incomplete.

After the migration, you missed the step of going through the WP database and changing all the static URLs to the HTTPS versions as well. This can be done using a mySQL query through phpMyAdmin, but the easiest way is to use the Better Search Replace plugin. (You need to be sure you're correctly handling serialised data, which the plugin does automatically.)

In addition, you're going to want to insure that you've updated the WordPress Address URL and the Site URL in Settings > General to the HTTPS versions of the URLs.

The final check will then be to make certain all the static assets loaded by the site (and any referenced from 3rd parties) are also being loaded over HTTPS as well. Watch especially for 3rd party form codes like newsletter subscription forms, social media widgets, and other widgets in sidebars as these aren't caught by the rewrite plugin I mentioned earlier. This step is essential to insure you don't have "mixed content" on the pages which cause you to get a non-secure warning from the browser.

You may have used a "Switch to HTTPS" plugin for this process initially, but the majority of those don't accomplish their task in the most SEO-effective way. Using the method outlines above will both make the search engines happier,and will make for faster webpage performance as well.

Let me know if you have further questions.

Paul

Michael - you are _incorrect, _I'm afraid! You need to read up on the specifics of the robots exclusion protocol.

A blank Disallow directive absolutely does NOT match all URLs on the site. In order to match all URLs on the site, the configuration would have to be:

User-agent: *
Disallow: /

Note the slash denoting the root of the site. If the field after disallow: is blank, that specifically means no URLs should be blocked. To quote www.robotstxt.org:

Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record.

The second part of that statement is equally important. For a record to be valid, it must include at least one user agent declaration and at least one disallow statement. If you want the file to not block any URLs, you must include the disallow: statement, but leave its value empty.

For more proof of this, here's the exact example, also from robotstxt.org:

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

The main reason for including a robots.txt which doesn't block anything is to help clean up a server's error logs. With no robots.txt in place, an error will be inserted into the logs every time a crawler visits and can't find the file, bloating the logs and obscuring the real errors that might be present. A blank file may lead someone to believe that the robots.txt just hasn't been configured, leading to unnecessary confusion. So a file configured as above is preferable even if no blocking is desired.

Hope that clears things up?

Paul

[edited to replace line breaks in the code examples that were stripped out by Moz text editor]

The old "text to code ratio" thing was always a myth, Becky, but it keeps popping up periodically.

I asked John Mueller of Google about it directly during a Crawling and Indexing Hangout and his answer was unequivocal, short and sweet:

Paul: Followup to the code density question - does code to text ratio make any difference for ranking?

John Mueller> Paul Thompson: No.

Paul Thompson> John Mueller: Much appreciated. Have been railing against this idea for so long, but folks making this claim keep popping up regularly

Here's the page with the actual comment exchange. Good video in general on crawling/indexing questions.

So - straight from the horse's mouth.

Paul