fix: come back to the listing page an import stopped in

[oleksandr-nc]

Part of #139, after #142, #143 and #144: the half of it that is about files that never arrive.

What a batch that stops in a folder leaves behind

The import job downloads 500 MB, then hands over to the next run through the import_tree user setting. A directory was taken out of that tree as soon as its first listing page had been walked, and the import folder itself was never put in it — only sub-folders were. So when a batch reached its download size on the second or a later page of a listing, nothing remembered that the listing was unfinished: the next batch went through the remembered sub-folders, found nothing left to do, and the import reported itself as finished. The files behind the page it stopped in were never downloaded, nothing was written to the log, and the notification said "N files were imported from OneDrive storage."

A drive larger than one batch with at least one sub-folder is enough to hit this, which makes it a good candidate for the reports of imports that silently leave files out.

What changes

• A directory stays in the import tree while its listing is being walked, and the value carries the page to resume at, so the next batch continues at the page the last one stopped in rather than skipping the rest of the folder. The import folder itself is in the tree like any other directory.
• A page whose token is no longer accepted — they are short lived and a batch may run an hour later — starts its directory over from the first page, once. The files it already brought are skipped as existing ones.
• A listing that fails for another reason is logged; it was silently swallowed.
• The files a page skipped because they were already there are counted even when the batch runs out on that same page. The counter is written once per page, and the page that ended a batch never got to write it.

The new setting

import_batch_size says how much a single run of the import job downloads before it lets the next run continue, 500 MB by default. Servers that cannot afford that in one job can lower it, and the integration test sets it to 20 bytes so that the import has to stop in the middle of a listing page and come back to it.

Where it is covered

The stubbed drive of the Graph API test is imported a second time with the small batch size, and the job checks the drive's second listing page was fetched again, that the files behind it arrived, and that the finish notification counts the file the first batch had already brought as one that was already there.