Add Data Packaging Documentation - #1723
Conversation
JBorrow
left a comment
JBorrow
left a comment
There was a problem hiding this comment.
A bit verbose but generally helpful if we can keep this in-sync with code.
| Six scripts do the work. None of them run themselves on a timer: in | ||
| production each one is launched by `Prefect <https://docs.prefect.io>`_ — the | ||
| workflow automation service that runs SO's automated pipelines — as a | ||
| recurring *Prefect deployment*, one set per DAQ node (see |
There was a problem hiding this comment.
This is an overcomplicated way of saying "The six scripts are launched through Prefect as recurring (timed) deployments".
| The ordering matters, but it is enforced by status checks rather than by when | ||
| each script happens to be launched: ``make_book`` only touches ``UNBOUND`` | ||
| Books, ``update_librarian`` only touches ``BOUND`` Books, and | ||
| ``cleanup_level2`` refuses to delete anything that is not at least | ||
| ``UPLOADED``. Running them out of order is safe; it just means nothing |
There was a problem hiding this comment.
Re-states what is stated above in a confusing way. "Ordering is enforced through data status checks in the data packaging databases and not through direct task dependencies. Running scripts 'out of order' is safe."
| Timecodes | ||
| --------- | ||
|
|
||
| A **timecode** is the first five digits of a ctime, i.e. ``int(ctime // 1e5)``. |
There was a problem hiding this comment.
I really don't like our usage of 'ctime' in this way as it is non-standard. ctime typically means a formatted string representing the current datetime (see e.g. https://www.geeksforgeeks.org/c/ctime-function-in-c-c/, something like Sat Jul 11 08:25:28 2026). What we call 'ctime' is a unix time stamp (seconds since Jan 01 1970).
| ``stray`` ``stray_<timecode>_<daq_node>`` | ||
| ============ ========================================================== | ||
|
|
||
| For ``obs``/``oper``, ``<timestamp>`` is the ctime taken from the *first* |
| However the cron schedules are set, they have to respect the dependencies | ||
| described in the :ref:`overview <data-packaging>`: | ||
|
|
||
| * ``update_g3thk_database`` before ``update_g3tsmurf_db`` before | ||
| ``update_book_plan``, since the finalization time is computed from | ||
| housekeeping data. | ||
| * ``update_delay`` (and ``update_delay_timecodes``) should comfortably exceed | ||
| the interval between Prefect runs, so that a missed run is repaired by the | ||
| next one rather than leaving a gap. | ||
| * ``make_book`` is the element that does real I/O; it is usually run less | ||
| often than the indexers. | ||
| * ``cleanup_level2`` is long-running and deletes data. Give it | ||
| ``max_runtime``, keep its completion range wider than its deletion ranges | ||
| (the script enforces this), and expect one instance at a time per node. |
There was a problem hiding this comment.
I mean the cron schedules don't have to respect that ordering, but it is better if they do.
2 participants
I had a whole bunch of free tokens to burn, so I had Claude write-up documentation for all the pieces data packaging.
I have read through it all and believe it is all correct. (I have not checked the sphinx rendering because I'm working in a system that won't let me do that).