Welcome to the Q&A Forum

While I have nothing to add in regards to facebook ad cliks not showing up in Analytics, I do want to add caution that facebook, while produces a lot of clicks, impressions, likes and even website clicks, unless you have such a broad site that you can do the fly paper traffic tactic and lure some of those people in,  I've found facebook's user base widely impulse driven rather than focus driven. For broad niches this works well, for narrow niches it's a risk.

As Eric hinted, the best method to prevent any pages being indexed would be to use htaccess password protection dialog on your development site. It's fairly easy to implement. You can find instructions to do so here: http://www.htaccesstools.com/articles/password-protection/

Thanks for the answer:)

Is there is any other option available.? Please advice.

If the pages are already indexed and you want them to be completely removed, you need to allow the crawlers in robots.txt and noindex the individual pages. So if you just block the site with robots.txt (and I recommend blocking via folders or variables, not individual pages) while the pages are indexed, they will continue to appear in search results but have a meta description of (this page is being blocked by robots.txt). However, it will continue to rank and appear because of the cached data. If you add the noindex tags to your pages instead, the next time crawlers visit the pages they will see the new tag and remove the page from the search index (meaning it won't show up at all). However, make sure your robots.txt isn't blocking the crawlers from seeing this updated code.

Hi Anoop, It looks like a hosting issue. Might have something to do with your DNS. I would check with your hosting provider to start with and then work backwards, checking your pages one by one to determine which ones are throwing you 400 error codes. You can use Screaming Frog or SEO SpyGlass from SEO Powersuite for this. A basic website audit through SEMrush or Site Auditor would also work. Based on the results of that research, you will know if it is sitewide (almost definitely hosting) or page-specific (almost definitely coding and site architecture). Let me know if you need any further help. Cheers, Rob

Hi Anoop! I agree with VERBInteractive—this is one of those questions that requires us to be able to see the site and the pages that aren't being indexed.

In your sitemap you only have to link to the pages on your site - there is no need to add a link to the robots.txt file. Your robots.txt is always on the root of the domain (yourdomain.com/robots.txt). In your robots.txt you can put the location of your sitemap(s) - just put the following line(s) at the end of the file. It is possible to add multiple lines if you have multiple sitemaps. Alternative would be to put a link to the index type of sitemap which contains the links to the different sitemaps you have. Sitemap: http://www.yourdomain.com/name_of_your_sitemap.xml It is not strictly necessary to add the link to the sitemap in your robots.txt - you can also just indicate the location in the Search Console (Google) / Web master tools (Bing) Dirk

Hi, Yes & No . I'm quoting Google on this read it carefully you will get the whole idea about 404. Q: Do the 404 errors reported in Webmaster Tools affect my site’s ranking? A: 404s are a perfectly normal part of the web; the Internet is always changing, new content is born, old content dies, and when it dies it (ideally) returns a 404 HTTP response code. Search engines are aware of this; we have 404 errors on our own sites, as you can see above, and we find them all over the web. In fact, we actually prefer that, when you get rid of a page on your site, you make sure that it returns a proper 404 or 410 response code (rather than a “soft 404”). Keep in mind that in order for our crawler to see the HTTP response code of a URL, it has to be able to crawl that URL—if the URL is blocked by your robots.txt file we won’t be able to crawl it and see its response code. The fact that some URLs on your site no longer exist / return 404s does not affect how your site’s other URLs (the ones that return 200 (Successful)) perform in our search results. Q: So 404s don’t hurt my website at all? A: If some URLs on your site 404, this fact alone does not hurt you or count against you in Google’s search results. However, there may be other reasons that you’d want to address certain types of 404s. For example, if some of the pages that 404 are pages you actually care about, you should look into why we’re seeing 404s when we crawl them! If you see a misspelling of a legitimate URL (www.example.com/awsome instead of www.example.com/awesome), it’s likely that someone intended to link to you and simply made a typo. Instead of returning a 404, you could 301 redirect the misspelled URL to the correct URL and capture the intended traffic from that link. You can also make sure that, when users do land on a 404 page on your site, you help them find what they were looking for rather than just saying “404 Not found." Hope that helps. Thanks

Hello, Could you make a nofollow link or copy url to your site? Thanks, - Mike Bean