Welcome to the Q&A Forum

Hi Gaston,

I just wanted to follow up with you with one last question if possible. Would this allow my images and PDF's to be crawled & indexed still?

Thanks!

Awesome. Thanks, Gaston!

Thank you for the response. I'm still a little uncertain, does the version you wrote allow the bots to crawl the css and js as well?

Best

Hello,

I would like my website to remain crawlable to bots, but to block my wp content and media. Does the following robots.txt work? I worry that the * user agent may conflict with the others.

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/

User-agent: GoogleBot
Allow: /

User-agent: GoogleBot-Mobile
Allow: /

User-agent: GoogleBot-Image
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Slurp
Allow: /

Using the htaccess I 404'd all the pages using "Options All -Indexes". Will this resolve the issue?

We use Worpdress as our CMS and do indeed use Yoast. We have never had an issue with /wp-content/ being indexed before, and I have been very conscientious about keeping our index clean.

Why I am confused is that this is an index of our wp-content, similar to a sitemap. I do not have robots.txt blocked for this as I do not know what is making the index.

Thanks!

Hi all,

I have suddenly noticed a massive jump in indexed pages. After performing a "site:" search, it was revealed that the sudden jump was due to the indexation of many pages beginning with the serp title "Index of /wp-content/uploads/" for many uploaded pieces of content & plugins.

This has appeared approximately one month after switching to https. I have also noticed a decline in Bing rankings. Does anyone know what is causing/how to fix this? To be clear, these pages are **not **normal /wp-content/uploads/ but rather "index of" pages, being included in Google.

Thank you.

Thanks for the great responses, Donna and Trenton. I just had one follow up question - I am still seeing small amounts of search console data on my http property. While I have read that this is not uncommon, it is mildly concerning as I force https server-side. Should I be alarmed by this?

Thanks for the help, Nicholas!

Sounds good, Ill give it a shot. Thank you guys!

Thanks for the input, Nicholas. This is what I was thinking, however, it seems that the blog post is now ranking for the last four days, and my solutions page isn't ranking at all for the keyword. Usually, the blog post would rank 1-2 days a week while the product page would rank the rest.

Would you still suggest de-optimizing the blog? Ranking for the keyword has been a months-long initiative, and I don't want to ruin my efforts.

Or should I wait and see if the product page begins ranking instead of the post again before de-optimizing the post?

Thanks for the response, I do have the keyword as the anchor text linking from my blog post to my product page. I don't know why it is when one ranks, the other does not, rather than alongside each other.

Would de-optimizing my blog post allow for my product page to rank all the time - or will it cause a lack of coverage when the blog post would otherwise rank?

Hello,

We are ranking for an acronym, which I understand can lead to fickle rankings. However, we have two pages ranking page one - two for the same keyword, but they do so in spite of each other.

By this I mean, one page will rank, while the other is nowhere to be found. It seems that the one page (a blog post) is more likely to rank on the weekends while the product page is more likely to rank on the weekdays.

I would like the product page to rank all the time, and to target another keyword with the blog post. Would removing the keyword from the blog post allow the product page to rank all the time - or would it lead to no pages ranking during times when the blog post would otherwise be ranking?

I should note the blog post has more external links and is not exactly optimized for the keyword, while the product page has more internal links and is optimized for the keyword.

Awesome, thank you for answering everything!

Hello,

We have moved our site to https, Google Analytics seems to be tracking correctly. However, I have seen some conflicting information, should I create a new view in analytics?

Additionally, should I also create a new https property in Google search console and set it as the preferred domain? If so, should I keep the old sitemap for my http property while updating the sitemap to https only for the https property?

Thirdly, should I create a new property as well as new sitemaps in Bing webmaster?

Finally, after doing a crawl on our http domain which has a 301 to https, the crawl stopped after the redirect, is this a result of using a free crawling tool or will bots not be able to crawl my site after this redirect?

Thanks for all the help in advance, I know there are a lot of questions here.

you guys are awesome! Thank you.

Thanks Niglel, after doing a little investigating, I believe google search console may have added in the backslash for formatting reasons. It appears with a backslash in home view, where you can see domains, however when viewing preferred domain, it does not appear with a backslash. To test this I used a practice site and added it in without a backslash, following my submission google added in a backslash under the domain view.

So I should be set?

Thanks!

Thanks Nigel, what will happen to the existing data under the view of the current preferred domain with the backslash if I switch the preferred domain to no backslash? I worry that the existing data will be erased or not transferred.

Thank you for the fast responses.

Currently, "www.domain.com/" has been claimed and set as preferred, all search console data appears on this account. (www and backslash)

"domain.com/" has also been claimed, with no data on this view.---(non www)

However, as stated, "www.domain.com/" (Preferred and with backslash)  redirects to www.domain.com. So as per suggestions I should add "www.domain.com", should this now be my preferred domain?

Thanks guys!

We are currently HTTP, however the page domain.com/ seems to redirect to domain.com, as I can not access domain.com/ without it bringing me to domain.com (sorry for the redundancy). However, the moz crawl did not reveal a 301. Does this resolve the duplicate content issue? Thanks for the fast answers.

-So far www and non www have been claimed only.