Welcome to the Q&A Forum

The sites are massive and we are talking massive numbers: Google reports in WMT that site B still has 259,157,970 links to site A, although when you filter into the report it only shows a few The current state is that nothing is blocked on Site B, and ALL pages point to the landing page of Site B. In WMT for site B, G still shows data for all the reports, like search queries, keywords, crawl errors (very old and all fixed)  and so on. The reports and data does not bother me as much as the 259,157,970 links it reports on Site A. On the 11th of April when I started the process of getting rid of these links, there were 554,066,716, this jumped up to 603,404,378 on the 28th of April. It started dropping and was as low as 122,405,100 on the 17th of May, and then started growing again up to where it is now 259,157,970 I also noticed that when the pages was giving 404s that the crawl rate of google dropped to zero, now that its redirecting to the landing page, the crawl rate is back up to about 1,800 per day, which is still very low, considering the numbers we are talking about. The crawl rate on Site A is okay, at 220,000 per day, but it was as high as 800,000 per day at one stage.

We had 301s for about 6 months, and the old URLs did not disappear from google. Thats why we decided to change them to 404s, with the thinking that Google might remove them quicker. But the number of links from sub-domains just keeps on growing. I am worried that by having these problem urls listed in the robots.txt actually prevents google from following them and seeing that it should be removed and that it returns a 404