The updateLayout() call in the BareField class only updates the domain decomposition associated to a field, but does not update the field values accordingly. This has not been needed so far in IPPL since previous time-step values of the simulation are not needed in the next time-step for most Poisson solvers, except for iterative solvers. In the CG solver, the previous time-step solution is used as an initial guess for the next solve. This works well when the layout of the field is not updated, however when using load balancing the solution field's layout is updated and hence the solution becomes stale. As a solution, the solution field is simply zeroed out, causing the initial guess for the iterative solve to be 0. This potentially increases the number of iterations required for convergence, and incurs a cost.
Furthremore, in the case of Maxwell solvers, load balancing will cause issues due to time history being kept. For the Maxwell equations, fields from the previous time-step need to be saved. When load balancing, the layout of these fields will be updated, causing field values to be wrong if they are not redistributed among ranks accordingly.
This exposes a problem in the updateLayout functionality, which is the lack of field update such that field values are in the correct rank when the domain decomposition changes.
The updateLayout() call in the BareField class only updates the domain decomposition associated to a field, but does not update the field values accordingly. This has not been needed so far in IPPL since previous time-step values of the simulation are not needed in the next time-step for most Poisson solvers, except for iterative solvers. In the CG solver, the previous time-step solution is used as an initial guess for the next solve. This works well when the layout of the field is not updated, however when using load balancing the solution field's layout is updated and hence the solution becomes stale. As a solution, the solution field is simply zeroed out, causing the initial guess for the iterative solve to be 0. This potentially increases the number of iterations required for convergence, and incurs a cost.
Furthremore, in the case of Maxwell solvers, load balancing will cause issues due to time history being kept. For the Maxwell equations, fields from the previous time-step need to be saved. When load balancing, the layout of these fields will be updated, causing field values to be wrong if they are not redistributed among ranks accordingly.
This exposes a problem in the updateLayout functionality, which is the lack of field update such that field values are in the correct rank when the domain decomposition changes.