Layered direction-dependent calibration

[tikk3r]

Many directions are currently rejected based on phasediff score or will fail either self-calibration or validation of the dd calibration.

This PR revises the direction-dependent calibration to follow a layered approach such that (hopefully) more sources survive in one go and thus less compute is wasted on redoing/refining things later.

This also facilitates single science target cases where a small catalogue of the target + nearby calibrator candidate sources can be given.

The outline:

• calibrate a set "strong" calibrators
• calibrate a set of "weak" calibrators with the nearest strong solution pre-applied
• calibrate the remaining sources with the nearest overall solution applied.

Additional changes:

• the solution pre-apply has been made more generic, with an explicit toggle for indicating NL station solutions
• moved the validation from dd-calibration to the ddcal_calibrators subworkflow
• clarified step name for solution plot flattening

[tikk3r]

I think it's ready for a first review. I have tested it on a Lockman Hole pointing and on a single target. Thusfar it seems to work in both the whole field case and the case where there is no strong source, for example (so it'll jump straight to the weak ones instead of crashing). There are just two warnings with my cwltool that I don't know yet how to solve

WARNING Workflow checker warning:
workflows/dd-calibration.cwl:128:11: Source 'h5parms' of type ["null", {"type": "array", "items":
                                     "File"}] may be incompatible
workflows/dd-calibration.cwl:312:7:    with sink 'calibration_solutions' of type ["null",
                                       {"type": "array", "items": "File"}]
workflows/dd-calibration.cwl:311:7:    source has linkMerge method merge_flattened
                                       pickValue is: all_non_null
workflows/dd-calibration.cwl:188:11: Source 'h5parms' of type ["null", {"type": "array", "items":
                                     "File"}] may be incompatible
workflows/dd-calibration.cwl:312:7:    with sink 'calibration_solutions' of type ["null",
                                       {"type": "array", "items": "File"}]
workflows/dd-calibration.cwl:311:7:    source has linkMerge method merge_flattened
                                       pickValue is: all_non_null
workflows/dd-calibration.cwl is valid CWL.

so any suggestions for that are also welcome.

[jurjen93]

This is a really great update! Don't have much to comment, as it looks good.
I only see some remaining warnings when I validate the workflow:

WARNING Workflow checker warning:
workflows/dd-calibration.cwl:142:11: Source 'multidir_h5' of type ["null", "File"] may be                                                                                                             
                                     incompatible                                                                                                                                                     
workflows/dd-calibration.cwl:199:11:   with sink 'h5parms' of type ["null", {"type": "array",                                                                                                         
                                       "items": "File"}]                                                                                                                                              
                                       source has linkMerge method merge_flattened                                                                                                                    
                                       pickValue is: all_non_null                                                                                                                                     
workflows/dd-calibration.cwl:128:11: Source 'h5parms' of type ["null", {"type": "array", "items":                                                                                                     
                                     "File"}] may be incompatible                                                                                                                                     
workflows/dd-calibration.cwl:322:7:    with sink 'calibration_solutions' of type ["null",                                                                                                             
                                       {"type": "array", "items": "File"}]                                                                                                                            
workflows/dd-calibration.cwl:321:7:    source has linkMerge method merge_flattened                                                                                                                    
                                       pickValue is: all_non_null   

I will approve it when these are resolved and if you can confirm here that you have successfully ran a complete end-to-end run of the new workflow!

[jurjen93]

This is ok for now, but I will make a reminder that I have to fix this in facetselfcal :)

[tikk3r]

The warnings are now all resolved with some fiddling of the final output types. I added one more step to select only the needed images (i.e. if a source trickled down from strong->weak only return weak).