FWIW, turning off validation helps a bit but not too much. For example, my laptop parsing 24 consensuses with validation takes 25 seconds, and if we disable validation it takes 22 seconds.
This means that to reach the rate of 120~ consensuses a second with parse_file(), we need to make it 100 times faster or so. This sounds much harder than 30% performance increase :/
Yup. I'm growingly kinda in agreement with your earlier sentiment that your use case (bulk processing hundreds of consensuses per second) is a bit specialized. I've occasionally thought that if I was to write Stem's parsers again I'd make them lazy load attributes. That would give you what you're after here.
But as they're written nowadays they're eager, and parsing all the consensus attributes takes time. So assuming you don't care about validation or having yet another thing to maintain a small shell script might serve you just as well.
Cheers! -Damian