I'll make a decision post about this, but we need a tool to orchestrate the pipeline, rather than relying on using __main__ in scripts and running things through justfile. There's Dagster https://docs.dagster.io which is a formal orchestration tool that looks like it would fit our needs well (compared to others I've been looking over).
I'll make a decision post about this, but we need a tool to orchestrate the pipeline, rather than relying on using
__main__in scripts and running things through justfile. There's Dagster https://docs.dagster.io which is a formal orchestration tool that looks like it would fit our needs well (compared to others I've been looking over).