This would allow it to begin tokenising mid way through downloading content, rather than requiring it all to be in memory before beginning.
This would allow it to begin tokenising mid way through downloading content, rather than requiring it all to be in memory before beginning.