305 enable post processing on each rank - #309
Conversation
There was a problem hiding this comment.
Pull request overview
This PR targets issue #305 by adding an option to run post-processing across MPI ranks (instead of only rank 0), and adds an MPI test that compares serial vs parallel post-processing outputs.
Changes:
- Add a
parallel_pprocflag toSimulation.pproc()/PostProcessorand introduce rank-aware loops + reductions in post-processing. - Add an MPI regression test comparing results from serial vs parallel post-processing.
- Adjust simulation startup logging and modify initial weight-reset behavior in the Vlasov–Ampère model.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| src/struphy/simulation/sim.py | Adds parallel_pproc option to pproc() and updates run-start logging. |
| src/struphy/post_processing/post_processing_tools.py | Implements parallel post-processing path (per-rank processing + MPI reductions) and rank-0-only output writes. |
| src/struphy/post_processing/tests/test_pproc.py | Adds an MPI test comparing serial vs parallel post-processing results (currently includes interactive plotting). |
| src/struphy/models/vlasov_ampere_one_species.py | Comments out particle weight reset after initial Poisson solve. |
Comments suppressed due to low confidence (9)
src/struphy/post_processing/post_processing_tools.py:674
- This
print()logs the entire reducedtemp_valarray; this is very noisy and can severely impact performance for large grids. Preferlogger.debugand only log metadata (e.g., shape).
print(f"{self.rank = } after\n {temp_val = }")
src/struphy/post_processing/post_processing_tools.py:677
- This
print()logs the full contents oftemp_val[j]during MPI reduction. For realistic grid sizes this will overwhelm stdout and slow post-processing substantially. Preferlogger.debugand log only shape/summary.
print(f"{self.rank = } before \n{temp_val[j].shape = }\n{temp_val[j] = }")
src/struphy/post_processing/post_processing_tools.py:692
- This
print()logs the full reduced component arraytemp_val[j], which is extremely verbose and can degrade performance. Preferlogger.debugwith shape/summary only.
print(f"{self.rank = } after\n {temp_val[j] = }")
src/struphy/post_processing/post_processing_tools.py:958
- Unconditional
print()in library code will spam stdout for MPI jobs and makes output hard to parse. Preferlogger.debug/info(and avoid interpolating expensive values) so users can control verbosity.
print(f"{self.rank} starting post-processing of distribution functions for {path_kinetic_species} ...")
src/struphy/post_processing/post_processing_tools.py:1010
- This unconditional
print()inside the per-slice loop will create a lot of noisy output for multi-rank runs. Preferlogger.debugor remove it.
print(f"{rank = } ----------------------------")
src/struphy/post_processing/post_processing_tools.py:1023
- Avoid computing and printing
xp.sum(...)in the hot post-processing path: it can be expensive (and forces materialization of arrays) and spams stdout. If progress logging is needed, uselogger.debugwithout expensive summaries.
print(f"{self.rank =} with {xp.sum(data) =} and {xp.sum(data_df) =}")
src/struphy/post_processing/post_processing_tools.py:1053
- This second
print()withxp.sum(...)repeats expensive reductions/logging and will spam stdout. Preferlogger.debugwithout forcing array reductions.
print(f"{self.rank =} with {xp.sum(data) =} and {xp.sum(data_df) =}")
src/struphy/post_processing/post_processing_tools.py:1055
- Unconditional
print()in post-processing adds noisy output in MPI jobs. Preferlogger.debug/infoso verbosity is configurable.
print(f"{self.rank =} done.")
src/struphy/post_processing/tests/test_pproc.py:8
- Tests should not depend on an interactive Matplotlib backend; importing
pyplotwithout forcing a non-interactive backend can fail/hang in headless CI. Force theAggbackend before importingpyplot.
from matplotlib import pyplot as plt
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # setup post processor and plotting | ||
| if not hasattr(self, "_post_processor") and self.rank == 0: | ||
| self._post_processor = PostProcessor(sim=self) | ||
| self._post_processor = PostProcessor(sim=self, parallel_pproc=parallel_pproc) |
| self.comm.reduce( | ||
| temp, | ||
| op=MPI.SUM, | ||
| root=0, | ||
| ) |
| if self.parallel_pproc: | ||
| assert imported_sim is not None, "Parallel post-processing only supported when the sim object is provided." | ||
| self.derham = sim.derham | ||
| self.comm = self.derham.comm | ||
| self.comm_size = self.comm.Get_size() | ||
| self.rank = self.comm.Get_rank() |
| print(f"{type(temp_val) = }") | ||
| if self.parallel_pproc: | ||
| if isinstance(temp_val, xp.ndarray): | ||
| print(f"{self.rank = } before \n{temp_val.shape = }\n{temp_val = }") |
| # plot and compare results from serial and parallel pproc | ||
| if MPI.COMM_WORLD.Get_rank() == 0: | ||
| r1_mpi, r2_mpi, r3_mpi, r4_mpi = do_plotting(sim, from_parallel=True) | ||
| plt.show() |
|
|
||
| import numpy as np | ||
| import pytest | ||
| from feectools.ddm.mpi import MockComm |
| # reset particle weights | ||
| particles.weights = particles.weights_at_t0.copy() | ||
| # particles.weights = particles.weights_at_t0.copy() |
Solves the following issue(s):
Closes #305
Test with
mpirun -n 2 python src/struphy/post_processing/tests/test_pproc.py.