Site structure: Any issues with 404'd parent folders?

[Guest]

This post is deleted!

[dsbud]

Seems odd that indexers would care if a parent directory page exists or not. Is there any proof that Google will attempt crawl parent folder pages that aren't in sitemaps.xml and aren't linked to anywhere else?

Perhaps I'm slowly building out my site. Depending on the material/approach, it might make sense to release a page talking about a sub-category (lassie) before releasing content about a parent category (famous dogs). Or maybe "famous dogs" is such low search volume that it doesn't make sense to spend time creating a parent "famous dogs" page.

If I'm understanding correctly, with the above you're effectively telling me to:

1. Build a parent category page. If I don't plan on investing much time/effort into the parent page, noindex it.

2. Reorganize my site folder structure.

Neither seem like a great option.

[GastonRiera]

This post is deleted!

[willcritchlow]

The short answer is that there should be no harm going with your proposed approach.

Longer version: I believe there are cases where Google has tried to crawl a directory like "/famous-dogs/" in your example purely because it appears as a sub-folder in the paths of other pages even though there are not any direct links to it. But even if it does crawl it, if you don't have or intend to have a page there, a 404 is a perfectly valid response.

In general, while there could be a case that it's worth creating a "/famous-dogs/" page if there is search demand you can fulfil, until or unless you do, there is no harm in it returning a 404 response.

[willcritchlow]

PS - if you're worried about the crawling, you could always block it in robots.txt if you really wanted (but unless it's a huge site I wouldn't bother). Note - if you do go this route, do it carefully so as not to block all contents of the folder at the same time!

[dsbud]

Thanks for the response. This is what I expected.

I swear I read somewhere that Google may pass some form of value from a child to a parent. i.e. "/famous-dogs/lassie/" could pass some value to "/famous-dogs/", absent any links. Can't find the source, but I suppose I'm a bit worried that I'd permanently lose out on some value if the parent does not exist initially. Considering I may add a "famous dogs" parent page at some point.

[dsbud]

In my scenario, considering I might add a parent "famous dogs" page at some point, it'd probably best to leave robots.txt alone, right?

[Guest]

This post is deleted!

[willcritchlow]

Yeah. I'd just leave it as a 404 in that case

[willcritchlow]

Yeah - there is various speculation about how signals or authority traverse folder structures (see for example this whiteboard Friday ) but I haven't seen anything suggesting it's permanent - all of this may be an argument for adding /famous-dogs/ at some point, but I wouldn't personally stress about it not being there at launch.