datadeps: Parallelize analysis and scheduling via hierarchical scheduling

[jpsamaroo]

This allows Datadeps analysis and scheduling (ordinarily sequential operations) to be parallelized by multiple threads and/or multiple workers, greatly improving performance and scalability.

Written by Claude

[github-actions]

Dagger benchmarks: dirty vs master

	master	dirty	master / dirty
array/dagger/N=1024 (block 512)/add (X + X)	10 ± 0.65 ms	17.3 ± 13 ms	0.579 ± 0.43
array/dagger/N=1024 (block 512)/alloc (rand)	3.17 ± 0.57 ms	4.52 ± 1.9 ms	0.703 ± 0.32
array/dagger/N=1024 (block 512)/broadcast (X .+ 1)	6.91 ± 0.75 ms	2.23 ± 0.13 ms	3.1 ± 0.38
array/dagger/N=1024 (block 512)/map (sin.(X))	9.52 ± 1.4 ms	6.63 ± 0.37 ms	1.44 ± 0.22
array/dagger/N=1024 (block 512)/norm	3.97 ± 0.11 ms	1.29 ± 0.38 ms	3.08 ± 0.91
array/dagger/N=1024 (block 512)/reduce (sum)	8.01 ± 1.9 ms	2.9 ± 0.64 ms	2.76 ± 0.89
array/dagger/N=1024 (block 512)/transpose (permutedims)	8.86 ± 1.7 ms	14.8 ± 2.4 ms	0.6 ± 0.15
array/dagger/N=256 (block 256)/add (X + X)	3.68 ± 0.6 ms	1.49 ± 0.29 ms	2.46 ± 0.62
array/dagger/N=256 (block 256)/alloc (rand)	1.15 ± 0.083 ms	1.05 ± 0.41 ms	1.1 ± 0.44
array/dagger/N=256 (block 256)/broadcast (X .+ 1)	3.24 ± 0.13 ms	0.487 ± 0.065 ms	6.66 ± 0.93
array/dagger/N=256 (block 256)/map (sin.(X))	3.67 ± 0.14 ms	0.932 ± 0.076 ms	3.94 ± 0.35
array/dagger/N=256 (block 256)/norm	2.2 ± 0.2 ms	0.507 ± 0.044 ms	4.33 ± 0.55
array/dagger/N=256 (block 256)/reduce (sum)	4.09 ± 0.95 ms	0.904 ± 0.45 ms	4.53 ± 2.5
array/dagger/N=256 (block 256)/transpose (permutedims)	2.71 ± 0.1 ms	1.07 ± 0.043 ms	2.52 ± 0.14
linalg/dagger/N=1024 (block 512)/cholesky	0.0362 ± 0.0023 s	24.5 ± 9.2 ms	1.48 ± 0.56
linalg/dagger/N=1024 (block 512)/lu	8.15 s	1.58 ± 0.012 s	5.15
linalg/dagger/N=1024 (block 512)/matmul (A*A)	0.043 ± 0.002 s	0.0388 ± 0.0027 s	1.11 ± 0.092
linalg/dagger/N=1024 (block 512)/matvec (A*x)	9.51 ± 0.97 ms	6.04 ± 2.6 ms	1.58 ± 0.7
linalg/dagger/N=1024 (block 512)/qr	0.336 ± 0.0027 s	0.336 ± 0.018 s	1 ± 0.056
linalg/dagger/N=1024 (block 512)/solve (A\b via lu)	11.3 s	2.73 s	4.13
linalg/dagger/N=1024 (block 512)/syrk (A'*A)	0.0411 ± 0.0032 s	0.0343 ± 0.0052 s	1.2 ± 0.21
linalg/dagger/N=256 (block 256)/cholesky	6.69 ± 0.094 ms	3.02 ± 0.35 ms	2.21 ± 0.26
linalg/dagger/N=256 (block 256)/lu	1.4 ± 0.023 s	0.261 ± 0.0062 s	5.37 ± 0.16
linalg/dagger/N=256 (block 256)/matmul (A*A)	4.4 ± 0.094 ms	2.68 ± 0.019 ms	1.64 ± 0.037
linalg/dagger/N=256 (block 256)/matvec (A*x)	3.55 ± 0.12 ms	1.99 ± 0.11 ms	1.78 ± 0.12
linalg/dagger/N=256 (block 256)/qr	9.38 ± 1.7 ms	6.87 ± 0.56 ms	1.37 ± 0.27
linalg/dagger/N=256 (block 256)/solve (A\b via lu)	2.33 ± 0.021 s	0.635 ± 0.036 s	3.67 ± 0.21
linalg/dagger/N=256 (block 256)/syrk (A'*A)	6.6 ± 0.98 ms	4.17 ± 0.64 ms	1.58 ± 0.34
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap)	0.0607 ± 0.001 s	0.0506 ± 0.0045 s	1.2 ± 0.11
stencil/dagger/N=1024 (block 512)/assign (const)	4.16 ± 0.046 ms	2.49 ± 1.2 ms	1.67 ± 0.81
stencil/dagger/N=1024 (block 512)/multi-expr	10.7 ± 0.55 ms	19.8 ± 6.3 ms	0.54 ± 0.17
stencil/dagger/N=1024 (block 512)/neighbors (Clamp)	0.0524 ± 0.0037 s	0.0419 ± 0.0032 s	1.25 ± 0.13
stencil/dagger/N=1024 (block 512)/neighbors (Pad)	0.0585 ± 0.0022 s	0.0511 ± 0.01 s	1.15 ± 0.24
stencil/dagger/N=1024 (block 512)/neighbors (Reflect)	0.0572 ± 0.011 s	0.0406 ± 0.0024 s	1.41 ± 0.28
stencil/dagger/N=1024 (block 512)/neighbors (Wrap)	0.0616 ± 0.0034 s	0.0411 ± 0.0052 s	1.5 ± 0.21
stencil/dagger/N=1024 (block 512)/update (+)	6.81 ± 0.72 ms	3.95 ± 3.8 ms	1.72 ± 1.6
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap)	14.5 ± 0.73 ms	9.05 ± 2.5 ms	1.61 ± 0.45
stencil/dagger/N=256 (block 256)/assign (const)	2.53 ± 0.03 ms	0.85 ± 0.083 ms	2.98 ± 0.29
stencil/dagger/N=256 (block 256)/multi-expr	4.96 ± 0.69 ms	1.65 ± 0.17 ms	3 ± 0.52
stencil/dagger/N=256 (block 256)/neighbors (Clamp)	14.2 ± 0.24 ms	8.89 ± 2.3 ms	1.59 ± 0.41
stencil/dagger/N=256 (block 256)/neighbors (Pad)	14.1 ± 1 ms	6.77 ± 1.2 ms	2.08 ± 0.4
stencil/dagger/N=256 (block 256)/neighbors (Reflect)	14.1 ± 0.2 ms	8.99 ± 2.7 ms	1.57 ± 0.47
stencil/dagger/N=256 (block 256)/neighbors (Wrap)	14.2 ± 1 ms	6.5 ± 2.4 ms	2.18 ± 0.81
stencil/dagger/N=256 (block 256)/update (+)	3.38 ± 0.26 ms	1.1 ± 0.16 ms	3.06 ± 0.49
time_to_load	0.992 ± 0.0049 s	1 ± 0.0015 s	0.992 ± 0.0051

Plots

⚠️ Regressions (> 25.0%)

• stencil/dagger/N=1024 (block 512)/multi-expr: +85.2%
• array/dagger/N=1024 (block 512)/add (X + X): +72.6%
• array/dagger/N=1024 (block 512)/transpose (permutedims): +66.6%
• array/dagger/N=1024 (block 512)/alloc (rand): +42.3%

Improvements (> 25.0% faster)

• array/dagger/N=256 (block 256)/broadcast (X .+ 1): -85.0%
• linalg/dagger/N=256 (block 256)/lu: -81.4%
• linalg/dagger/N=1024 (block 512)/lu: -80.6%
• array/dagger/N=256 (block 256)/reduce (sum): -77.9%
• array/dagger/N=256 (block 256)/norm: -76.9%
• linalg/dagger/N=1024 (block 512)/solve (A\b via lu): -75.8%
• array/dagger/N=256 (block 256)/map (sin.(X)): -74.6%
• linalg/dagger/N=256 (block 256)/solve (A\b via lu): -72.8%
• array/dagger/N=1024 (block 512)/broadcast (X .+ 1): -67.8%
• array/dagger/N=1024 (block 512)/norm: -67.5%
• stencil/dagger/N=256 (block 256)/update (+): -67.3%
• stencil/dagger/N=256 (block 256)/multi-expr: -66.7%
• stencil/dagger/N=256 (block 256)/assign (const): -66.4%
• array/dagger/N=1024 (block 512)/reduce (sum): -63.8%
• array/dagger/N=256 (block 256)/transpose (permutedims): -60.3%
• array/dagger/N=256 (block 256)/add (X + X): -59.4%
• linalg/dagger/N=256 (block 256)/cholesky: -54.8%
• stencil/dagger/N=256 (block 256)/neighbors (Wrap): -54.2%
• stencil/dagger/N=256 (block 256)/neighbors (Pad): -52.0%
• linalg/dagger/N=256 (block 256)/matvec (A*x): -44.0%
• stencil/dagger/N=1024 (block 512)/update (+): -42.0%
• stencil/dagger/N=1024 (block 512)/assign (const): -40.0%
• linalg/dagger/N=256 (block 256)/matmul (A*A): -39.0%
• stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap): -37.8%
• stencil/dagger/N=256 (block 256)/neighbors (Clamp): -37.2%
• linalg/dagger/N=256 (block 256)/syrk (A'*A): -36.8%
• linalg/dagger/N=1024 (block 512)/matvec (A*x): -36.6%
• stencil/dagger/N=256 (block 256)/neighbors (Reflect): -36.3%
• stencil/dagger/N=1024 (block 512)/neighbors (Wrap): -33.3%
• linalg/dagger/N=1024 (block 512)/cholesky: -32.2%
• array/dagger/N=1024 (block 512)/map (sin.(X)): -30.4%
• stencil/dagger/N=1024 (block 512)/neighbors (Reflect): -29.1%
• linalg/dagger/N=256 (block 256)/qr: -26.8%

Full results and plots (download the benchmark-results artifact).