Large robots.txt file

[ThomasHarvey]

We're looking at potentially creating a robots.txt with 1450 lines in it. This will remove 100k+ pages from the crawl that are all old pages (I know, the ideal would be to delete/noindex but not viable unfortunately)

Now the issue i'm thinking is that a large robots.txt will either stop the robots.txt from being followed or will slow our crawl rate down.

Does anybody have any experience with a robots.txt of that size?

[ThomasHarvey]

Answered my own questions:

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?csw=1#file-format

"A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb."