Airflow - #3

Open

tikk3r wants to merge 143 commits into

mainfrom

Commits on May 23, 2026

Initial DAG for LINC+delay
tikk3r
committed
7818ddf

Commits on May 28, 2026

Add download step
tikk3r
committed
da28933
Fix small errors
tikk3r
committed
e3c5ae1

Commits on May 31, 2026

delay cal, rundir and outdir
tikk3r
committed
e9612d6

Commits on Jun 5, 2026

dd cal dag entry
tikk3r
committed
5bd78a3
Tweak dd cal
tikk3r
committed
9910c17

Commits on Jun 9, 2026

Return field from linc target validation
tikk3r
committed
834d888
Get dd status column
tikk3r
committed
0b698a5
Fix target retrieval in ddcal
tikk3r
committed
125de55
Set whole field to finished once ddcal finishes successfully
tikk3r
committed
abd97e8
One target list per field
tikk3r
committed
245b360
Sort priority descending
tikk3r
committed
137c456

Commits on Jun 10, 2026

Empty explicit paths
tikk3r
committed
f1966ca
Fix delay sols search path
tikk3r
committed
cc5421b
Fix suffix search
tikk3r
committed
b72eec1
Tweak most recent dir
tikk3r
committed
ee81117

Commits on Jun 15, 2026

Fix target search path in delay
tikk3r
committed
e36d327
Fix suffix for delay calibration
tikk3r
committed
28037f2
More power: restarts and bad nodes
tikk3r
committed
16781d2
Remove outdated LINC dag
tikk3r
committed
2135952

Commits on Jun 16, 2026

Attempt to retrieve rundir on the fly
tikk3r
committed
e9741f4
Small updates
tikk3r
committed
0644f6e

Commits on Jun 18, 2026

Remove explicit paths
tikk3r
committed
74dd7ef

Commits on Jun 21, 2026

Fix dd cal and update success check
tikk3r
committed
b761683
Add some documentation
tikk3r
committed
f59f56f
Implement calibrator2 and homogenise success triggers
tikk3r
committed
062e8bc

Commits on Jun 24, 2026

Fix adding a field
tikk3r
committed
3e181ca
Draft widefield dag
tikk3r
committed
f99d192
Update readme
tikk3r
committed
52a36d4
Expand readme with expected folder setup
tikk3r
committed
354b154
Allow manual approval of delay solutions for widefield
tikk3r
committed
e50a28d

Commits on Jul 16, 2026

Update widefield DAG with subtract and ddcal
tikk3r
committed
7869c95

Commits on Aug 3, 2026

Update readme
tikk3r
committed
c48cb92
Add intermediate resolution imaging workflow
tikk3r
committed
a69c0a4
Merge pull request #4 from FLOCSoft/airflow-interm-img
tikk3r
authored
01dd50a
Update tmux commands
tikk3r
committed
f31ff4c
Fix database addition
tikk3r
committed
b5a59f5
Update database column creation
tikk3r
committed
95a4d74
Add facet subtract task
tikk3r
committed
99a2199
Merge pull request #5 from FLOCSoft/airflow-facet-subtract
tikk3r
authored
8434d2f
Add facet imaging for 0.3 arcsec initially
tikk3r
committed
c478d2f
Merge pull request #6 from FLOCSoft/airflow-facet-img
tikk3r
authored
9a88b89
Fix ddf subtract status column query
tikk3r
committed
0e7b476
Fix frequency range for downloading
tikk3r
committed
9ddbc4a

Commits on Aug 4, 2026

Update readme with secret
tikk3r
committed
0944be7
Update readme with DB initialisation
tikk3r
committed
6a6c62d
Don't succeed delay if processing
tikk3r
committed
ef6892c
Fix target staged flag
tikk3r
committed
0532dcb
Run lofar-vlbi-plot if no delay catalogue is found
tikk3r
committed
f65bff5
Make downloading a bit more robust with restarts
tikk3r
committed
d67066f
Add more download guards
tikk3r
committed
598fa46
More dl guards
tikk3r
committed
ba85ba7
Be more verbose about downloading
tikk3r
committed
cd7d4c5
Fix target dl reporting
tikk3r
committed
d1225be
Fix string formatting
tikk3r
committed
467bf79
Shell in subprocess
tikk3r
committed
823d7bc
Only obtain field that are not currently processing
tikk3r
committed
cfaa858

Commits on Aug 5, 2026

Wait for both calibrators to finish
tikk3r
committed
68e7167
Do not return field for calibrator if processing
tikk3r
committed
6b2adf0
ruff
tikk3r
committed
9abb427
Initialise DB with enum values
tikk3r
committed
790892b
Download products to output directory
tikk3r
committed
05730ed
Move delay download check inside command
tikk3r
committed
e19cf9c
Do not try to restart delay if log file not present
tikk3r
committed
c444c54
Skip calibrators if they don't exist
tikk3r
committed
9a3a57a
Make resumes more robust for the PILOT pipelines
tikk3r
committed
e26583a
Skip rest of DAG if no unprocessed fields
tikk3r
committed
1fc9622
Read DAG settings from a config file
tikk3r
committed
c109630
Convert row to dict
tikk3r
committed
53df96d
Print summary of config on launch
tikk3r
committed
0d18311
evaluate bool eval for delay
tikk3r
committed
e41f83b
Try to fix unprocessed fields
tikk3r
committed
3bb5988
Break if unprocessed field found
tikk3r
committed
b4d6ac7
Warn inside the dag if config not found
tikk3r
committed
51e14f9
Make a user-friendly frontend for the airflow setup
tikk3r
committed
46ede5b
Actually launch different services
tikk3r
committed
478f372

Commits on Aug 6, 2026

Add calibrator selection based on Tim's code
tikk3r
committed
05aefc4
Update readme
tikk3r
committed
29108eb
Merge pull request #7 from FLOCSoft/airflow-calibrator-template
tikk3r
authored
c047e24
Mark ddf pipeline as finished in database
tikk3r
committed
033cb65
Implement both cwltool and toil for linc calibrator
tikk3r
committed
16dff1f
Run calibrator 2 in 2...
tikk3r
committed
1750810
Merge pull request #8 from FLOCSoft/airflow-cwltool
tikk3r
authored
88da78a
Move DB logic to submodule
tikk3r
committed
22b39d3
Merge pull request #9 from FLOCSoft/isolate-db-functions
tikk3r
authored
be36071
Add DB utils interface
tikk3r
committed
7c9f0cb
Implement cwltool for linc target
tikk3r
committed
4a0bbe0
default toil for target for now
tikk3r
committed
7efc522

Commits on Aug 7, 2026

Update score choice to handle identical scores
tikk3r
committed
ee66040
Add initial db manipulation commands
tikk3r
committed
802f050
Add nothing status
tikk3r
committed
fa5e48f
More db manipulation
tikk3r
committed
6eaea01
Merge pull request #11 from FLOCSoft/db-manipulation
tikk3r
authored
f08db75
Merge pull request #10 from FLOCSoft/airflow-cwltool-linc-target
tikk3r
authored
f9c8b9a
Implement cwltool for delay cal
tikk3r
committed
2ff0a67
Fix update if statement
tikk3r
committed
00aa1ed
Rename airflow deployment
tikk3r
committed
9effc08
Change python version bound to make ty happy
tikk3r
committed
df04743
Merge pull request #12 from FLOCSoft/airflow-cwltool-pilot-delay
tikk3r
authored
17ff461
Implement cwltool for ddcal
tikk3r
committed
4ab880c
Add cores to cwltool calls
tikk3r
committed
f537031
Merge pull request #13 from FLOCSoft/airflow-cwltool-pilot-ddcal
tikk3r
authored
55fbbbf
Extract run commands into submodule
tikk3r
committed
2e7696c
Extract intermediate resolution image
tikk3r
committed
2f0ac7a
Extract facet subtract
tikk3r
committed
8905008
Extract facet imaging
tikk3r
committed
7bae777
Add missing db arguments
tikk3r
committed
21dd0d3
Extract ddf subtract prep step
tikk3r
committed
20e3c09
Extract ddf subtract prep step
tikk3r
committed
17a654d
Extract ddf subtract
tikk3r
committed
16b2405
Implement a logs directory for most steps
tikk3r
committed
8cb3991
Extract initial ddf pipeline and actually make a logs sub directory
tikk3r
committed
3966274
Fix unprocessed target assignment
tikk3r
committed
2eb8e57
Fix unprocessed target assignment
tikk3r
committed
9b2508b

Commits on Aug 14, 2026

Check user home for a config to be a bit more flexible for screens/tmuxes
tikk3r
committed
4945af2
Resolve ddf launching
tikk3r
committed
b6b7e6e
Output logs to the proper directory everywhere
tikk3r
committed
fce2eaa
Cleanup remaining ddf logs
tikk3r
committed
8f2bc2a
Better boolean parsing
tikk3r
committed
2b57ab5
Safeguard directory creations
tikk3r
committed
04690d6
No crash if dir exists
tikk3r
committed
afc7cef
Bunch of fixes
tikk3r
committed
d74264e
More fixes
tikk3r
committed
7c2cba3
More fixes
tikk3r
committed
7ded59e
Fix model image glob
tikk3r
committed
c1fa056
More fixes
tikk3r
committed
30d5370
Really fix loop this time
tikk3r
committed
bd2f269
Add oom slurm failure
tikk3r
committed
90af5ba
Merge pull request #14 from FLOCSoft/cleanup-runners
tikk3r
authored
b23ad37
Add missing status columns
tikk3r
committed
aa179ae

Commits on Aug 15, 2026

Output err log for intermediate img
tikk3r
committed
f5c7649

Commits on Aug 16, 2026

Make facet imaging resolution dynamic
tikk3r
committed
eab657f

Commits on Aug 17, 2026

Set processing status in prepare steps
tikk3r
committed
96e5da3

Commits on Aug 24, 2026

Image at multiple resolutions
tikk3r
committed
b240964
Make facet imaging naming consistent
tikk3r
committed
7777b09

Commits on Aug 25, 2026

Import calibrator assessment from autoPILOT
tikk3r
committed
3dbaff1
Fix calibrator select logic
tikk3r
committed
a84f6b3
Fix enum value comparisons
tikk3r
committed
cbc359b
Remove toil stats argument for cwltool run in delay cal
tikk3r
committed
2d39079

Commits on Aug 28, 2026

Add some retries on partial success requests (#16)
tikk3r
authored
e6d160b

Commits on Sep 12, 2026

Also write config variable
tikk3r
committed
cc98e7e
Always select all columns
tikk3r
committed
320e472
Update single target dag
tikk3r
committed
a72a8ff