Expand file rotation conditions in HKAggregator

[BrianJKoopman]

Originally from https://github.com/simonsobs/ocs-deployment-configs/issues/172.

For HK, specifically, we're sort of aiming for ~1GB files, but without crossing ctime % 100000 boundaries. So it might be good if aggregator had a mode where it could change files based on a variety of conditions; e.g. if any of:

• file size reaches some threshold
• some time has passed (current implementation)
• some ctime boundary (e.g. ctime % 100000 == 0) is crossed.

[jlashner]

What requirement is the ctime boundary coming from here? I've always thought that this boundary was soft and it was ok for files to go over. Is this just to make sure small files are eventually written within a day?

[BrianJKoopman]

What requirement is the ctime boundary coming from here? I've always thought that this boundary was soft and it was ok for files to go over. Is this just to make sure small files are eventually written within a day?

I'll defer to @mhasself on that.

[mhasself]

What requirement is the ctime boundary coming from here? I've always thought that this boundary was soft and it was ok for files to go over. Is this just to make sure small files are eventually written within a day?

I'll defer to @mhasself on that.

Yes, it definitely help in making sure small files are eventually flushed. If you're going to do that, it makes sense to do it on the ctime5 boundaries because a lot of our data organization is by ctime5. For example, HK books can be finalized and archived if we make a point of finalizing data files around ctime5 boundaries.