Conversation
ET0 fix: both full workflows now divide the daily ET0 by its interval in hours (period_et = 24) before writing //Kurven/ET0, mirroring the rain conversion. The kernel reads the ET0 curve as a mm/h rate, so unconverted daily values were integrated 24x too high. RAM: run_one() now thins each run to its single optimisation row immediately (get_simulation_results_optim(lean = TRUE) + add_overflow_events_and_waterbalance) and returns it; run_scenarios() collects the one-row tibbles and the analyse chunk just binds them. This removes the get_simulation_results_optim_parallel() pass that loaded every run's full time series into memory at once. New 'lean' arg on get_simulation_results_optim() reads only the fields the optimisation summary needs (element rates + both water balances), nulling states/meta/ connected-area rates; its intro message is gated behind debug.
Apply the same memory fix as Wien/BadAussee: run_one() now thins each run to its optimisation row immediately (get_simulation_results_optim(lean = TRUE) + add_overflow_events_and_waterbalance) and returns it; run_scenarios() collects the rows and the analyse chunk binds them, replacing the get_simulation_results_optim_parallel() pass that held every run's full time series at once. No ET0 unit change here: the Eisenstadt workflows use the base.h5 constant ET0 placeholder (0.2 mm/h) and the template rain (already mm/h), so there is no mm/d daily series to convert.
Summary page linking the per-site brute-force outputs (workflow html, result tables, CSVs, interactive plots). Added to .Rbuildignore so R CMD check does not build/execute it; render it manually into the assembled results directory. Result-file links match each workflow's paths$modelname (Wien, BadAussee, Eisenstadt_2005).
Since the workflows now thin per run inside run_one(), reading happens inside the future_lapply batch. A result file that exists but cannot be opened/read (engine crashed mid-write for a scenario, or a transient lock) previously threw H5File.open() 'unable to open file' and aborted the whole render at the run_model chunk. Wrap the open+read in an inner function (own on.exit for handle cleanup) plus tryCatch: such a file is now treated like a missing one -- warn, name the scenario, return NULL -- so add_overflow_events_and_waterbalance() emits an NA row and the batch completes.
R CMD check failed with a codoc WARNING because the .Rd still documented the
old signature (no 'lean'). Regenerate the usage block and add the \item{lean}
documentation to match R/get_simulation_results_optim.R (devtools::document()
equivalent), clearing the only WARNING (the 3 NOTEs are pre-existing).
to do: adapt Eisenstadt to the same parameter
Consistency: - workflow_eisenstadt-2005.Rmd and workflow_eisenstadt-2005_neu.Rmd now unconditionally set //Massnahmenelemente/Mulde_Rigole/Parameter_Evapotranspiration/LAI_LeafAreaIndex = 3.9 (Hoernschemeyer grass value, Water 2023, 15, 2840, Tab. 6, plant type 5). Wien already sweeps LAI over c(3.9, 8.5); Bad Aussee uses the same sweep grid as Wien; Eisenstadt was the outlier still running on the base.h5 default 8.5. New feature: - plot_cost_vs_overflow_volume() -- new exported ggplot helper mirroring plot_wb_tradeoff_overflows() for cost-aware optimisation. x = cost_total (EUR), y = overflow volume in m3 (computed from sum_overflows [mm] * mulde_area [m2] / 1000), points coloured discretely by n_overflows using the same 0..x / ">x" palette and top legend. Plotly tooltip carries the cost breakdown (cost_excavation / _profiling / _filter / _storage / _total) plus the varying param_grid entries. - Each of the four case-study workflow vignettes (workflow_wien, workflow_badaussee, workflow_eisenstadt-2005, workflow_eisenstadt-2005_neu) now renders the plot as simulation_results_optimisation_<site>_cost-vs-overflow-volume.html right after the existing water-balance render, matching its PDF / saveWidget pattern exactly. - vignettes/index.Rmd gains a new "Kosten vs. Überlaufvolumen" section under "Interaktive Visualisierungen" that links to the three top-level sites (Eisenstadt 2005, Wien, Bad Aussee). https://claude.ai/code/session_014QrjF51tg7cMVmsgjmfPNG
Builds on the cost-vs-overflow-volume plot (PR #15) with a new cost-by-overflow-count boxplot (three best-selection variants), richer tooltips and i18n parameter labels, and fixes the Eisenstadt cost pipeline the PR review flagged. Bug fix (PR #15 review blocker): - vignettes/workflow_eisenstadt-2005.Rmd now pipes the joined optimisation results through kwb.raindrop::compute_costs(), like the Wien and Bad Aussee workflows already did. Without it the vignette's plot_cost_vs_overflow_volume() call aborted with "missing column(s): cost_excavation, ..." on Windows, so the cost PDF/HTML was never produced and the exported CSV lacked the cost columns. New plot -- plot_cost_overflow_boxplot() (exported): - Boxplot of total construction cost (EUR, y) per number of overflow events (x). Counts 0..x each get their own box (x = max_n_overflows, as in the sibling plots); higher counts collapse into a single ">x" catch-all box (furthest right, red), keeping the axis readable for the long-tailed 15-year runs (Wien / Bad Aussee reach several hundred overflow events). The ">x" box highlights the scenario with the fewest overflow events above x (closest to valid), best_by breaking ties. - Individual scenarios are overlaid as jittered points whose size scales with a chosen variable (size_by): the overflow volume (m3, default) or the element evapotranspiration share (%). The size scale is calibrated to the valid region (0..x) and capped, with a minimum size, so the many-overflow outliers of the ">x" box do not shrink the valid-region points to invisible dots. - One best scenario per box is highlighted with a black-outlined diamond in that box's group colour (so its tooltip inherits the group colour) and the best of all boxes are joined by a frontier line (mark_best / connect_best). best_by picks the objective, cost as tie-breaker -- "min_cost" (cheapest), "min_overflow" (smallest overflow volume) or "max_evapotranspiration" (highest evapotranspiration) -- so the three variants trace three different frontier lines; label_best annotates the marker ("NN m3 / NN %" or "NN %"). - Each case-study vignette loops the three variants into *_cost-by-overflows-boxplot-{cheapest,min-overflow,max-evap}.html, all linked from vignettes/index.Rmd, which now groups the cost plots under one "Kosten" heading with sub-points (toc_depth 4). Tooltip enrichment (shared helpers in R/cost_tooltip.R): - The plotly tooltip of plot_cost_vs_overflow_volume() now carries the element water balance (evapotranspiration / infiltration / overflow, all in %) in addition to the cost breakdown, and names the chosen storage type on its own bold line, bilingually ("Sickerbox / Infiltration box" or "Schotterrigol / Gravel trench"). - The "varying parameters" block is translated via the new exported default_param_labels() helper -- a hovered point shows e.g. "Muldenflaeche [m2]=125" (de) / "Swale area [m2]=125" (en) instead of the raw "mulde_area=125"; override with param_labels =. - cost_tooltip_labels() / cost_tooltip_text() / build_varying_param_html() are shared so both cost plots emit byte-identical tooltips. Code hygiene: - All non-ASCII characters in R code (string literals in the plot functions and in the vignette code chunks) are unicode-escaped (\uXXXX); the few non-ASCII code comments were rewritten in plain ASCII ("Hoernschemeyer", "2xN"). Markdown prose keeps UTF-8. New exports: plot_cost_overflow_boxplot(), default_param_labels().
Both plot_cost_vs_overflow_volume() and plot_cost_overflow_boxplot() now append the share of scenarios meeting the validity criterion (n_overflows <= x) to the plot title, e.g. "Kosten vs. Ueberlaufvolumen (39 % mit <= 5 Ueberlaeufen)" / "Cost vs. overflow volume (39 % with <= 5 overflows)". It goes in the title, not a ggplot subtitle: ggplotly drops subtitles, so the share would be lost in the interactive HTML the vignettes export. The scatter title drops its old "(Anzahl Ueberlaeufe <= x)" parenthetical, which the share now supersedes. vignettes/index.Rmd now spells out the per-site validity threshold that drives this share: Eisenstadt <= 1 (1-year simulation), Wien / Bad Aussee <= 5 (15-year rain/ET series).
…flow-plot Cost-vs-overflow-volume plot + Eisenstadt LAI 3.9 consistency
Storage types
New optimise_swale_design_simultaneous(): penalised Nelder-Mead search (stats::optim, no new dependency) over mulde_area, mulde_height and storage_height at once, instead of per-parameter bisection. Infeasible designs are graded-penalised rather than excluded, so the simplex can trade parameters against each other and the method does not rely on the per-parameter monotonicity assumption. Engine runs stay bounded via tolerance snapping (shared evaluation cache across all x targets), a deterministic multistart (prior warm start, previous-target optimum, one anchor start per storage level, space-filling points; equal budget slices per start) and a final lattice polish. Same interface and result schema as optimise_swale_design(); a pairwise dominance check replaces the bisection's volume referee. - tests mirroring the bisection suite (brute-force reference, implicit storage escalation, infeasibility, warm start, max_total_depth) plus a mutual cross-check between both optimisers - workflow_optimisation vignette: new section running the simultaneous search for all sites and tabulating cost deltas vs. bisection - NEWS entry, docs and cross-references Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
Separate the two optimisation workflows: workflow_optimisation.Rmd is restored to bisection-only (plus a pointer paragraph), the simultaneous search moves into the new vignette workflow_optimisation_simultaneous.Rmd (Nelder-Mead sweep for all sites, cell-by-cell comparison against the bisection CSV export when present, and a method benchmark at x = 1). optimise_swale_design_simultaneous() gains a method argument with two alternative optimisers for comparison purposes: - diff_evolution: compact DE/rand/1/bin, deterministic via an internal Park-Miller generator (seed argument); R's global RNG stays untouched - halton_search: quasi-random space-filling baseline (Halton sequence) All methods share the penalised objective, the evaluation cache, the tolerance snapping and a new multi-valley lattice polish (accelerated 8/4/2/1-tolerance pattern descent from the cheapest feasible design of every storage level visited - the storage axis separates cost valleys that single coordinate steps cannot cross). Results carry a method column. Tests: per-method brute-force reference comparison, DE determinism and .Random.seed invariance, start-configuration agreement within search tolerances; 137 tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
Confirmed findings from a multi-lens review (algorithms, R semantics, doc consistency, test quality), all reproduced before fixing: - both optimisers crashed in the final arrange() when every cell was analytically infeasible (e.g. a strict max_total_depth): the empty evaluation cache produced a 0-column tibble; now they return the documented infeasible_within_bounds rows with a 0-row evaluations attribute - a max_total_depth that compresses the gravel-trench bounds to a single admissible storage height (gb[2] == gb[1]) was misreported as infeasible without any evaluation; the degenerate axis is now solved (encode() guards the 0/0 division) in both optimisers - prior_start_design(): NA rows in prior results injected all-NA rows via NA logical indexing and crashed compute_costs(); all filter conditions are NA-guarded now, and an all-NA-cost selection returns NULL instead of a 0-row frame that broke encode() - DE mutation indices r1/r2/r3 could coincide (P ~ 1/11 for a zero difference vector); now pairwise distinct as DE/rand/1/bin requires - NEWS: the simultaneous-vignette bullet had split the optimiser sub-bullet list, re-parenting make_swale_runner()/stack_levels() under the wrong bullet; docs now also state the 6-seed cap of the multi-valley polish instead of claiming "every storage level" New tests: empty search space (both optimisers), degenerate gravel axis, NA-containing prior, max_evals budget effect, seed-effect canary on the search path, result attributes, single-type storage_spec for all three methods. 160 tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
The optimisation vignettes died on the user's machine with HDF5
errno = 28 ("No space left on device"): every engine run creates its
own scenario directory (a copy of base.h5 as input plus all output
HDF5s) under tempdir() and nothing was ever removed. The bisection's
~15 runs per cell masked this; the simultaneous search (60-120 runs
per cell, ~124 scenarios per task across 6 parallel tasks with the
15-year Wien/Bad Aussee models) fills the temp drive mid-sweep.
The runner only ever uses the thinned one-row result, so it now
deletes the scenario's input copy and output directory right after
that row has been read (new argument cleanup = TRUE; failed runs keep
their files for debugging). Both optimisation vignettes document the
disk behaviour and the recovery step after an aborted run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
A full Nelder-Mead sweep is ~400-700 engine runs per task (2-3 h per task at 15 s/run for the Wien / Bad Aussee models, more wall time when fewer cores than tasks are free) and ran completely silent inside the future workers - indistinguishable from a hang. Both compute chunks now show a progressr bar ticking once per engine run across the worker boundary (per-task quota topped up on completion so the bar ends at exactly 100%). The runtime paragraph states the real expectation (2-5 h), the site list gains a commented quick-test switch (Eisenstadt only, minutes instead of hours) and max_evals is exposed in the search-space chunk as the runtime lever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
…trip per run An Eisenstadt-only quick test still took close to an hour: the ~2 s engine time per run was dwarfed by per-run overhead - every single run copied base.h5, read ALL datasets via h5_read_values(), rewrote ALL of them via h5_write_values() (for Wien / Bad Aussee including the 15-year rain series), and spawned a new process, with Windows virus scanning on top of each new file. The runner now prepares a site master file once on first call (base.h5 + calculation settings + ET/rain time series + Growth/Shading end-time fix) and each run copies that master and writes only its ~15 small parameter datasets (geometry, storage soil preset, kf/Psi, LAI, result path; the base.h5 rain curve is cached once and rescaled only when rain_factor != 1). Written file contents are identical to before; untouched datasets now stay bit-identical to base.h5 instead of going through a read/write round trip. rain_factor stays ignored when own rain series are provided, as documented. The vignette's quick-test note states realistic timing and points to max_evals and a virus-scanner exclusion for the temp folder as the remaining levers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
Comparing the Eisenstadt quick-test results against the bisection showed the simultaneous NM optimiser landing up to 8.8% above the bisection optimum in three cells (box x=0, trench x=0, trench x=5). Two structural gaps in the polish, not search-budget issues: - it could only step each axis straight down, so it could not slide ALONG the feasibility boundary (trade the expensive lever, area, down against the cheap one, mulde_height, up) - box x=0 got stuck at (61 m2, depth 200) where (55 m2, depth 300) is cheaper - single steps could not cross a +1 counting-wobble band even when the whole lower mulde_height range is feasible again below it (trench x=5: stuck at depth 270 while depth 100 is feasible and ~800 EUR cheaper) Each polish round now additionally proposes a boundary slide (area - step, mulde_height at its maximum) and a mulde_height floor probe (area, height_bounds[1]) - both plain evaluated candidates, so no monotonicity assumption enters the simultaneous optimiser. On the synthetic brute-force benchmark all three methods now converge to the same lattice optimum (worst ratio 1.012, previously NM 1.016 / DE 1.028 / Halton 1.081) at ~12% more runs for NM, and a synthetic wobble-band scenario lands within 0.9% of the reference. 160 tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
The bisection makes two distinct assumptions, and only one was documented: besides hydraulic monotonicity (needed for interval halving, immune to cost changes) its search ORDER encodes a cost hierarchy - shrink area first at maximum depth, escalate storage only when area is pinned at its bound, shrink depth last. That order picks the corner of the feasibility boundary that is optimal for the default rates (per mm of capacity and m2: depth ~0.07 EUR, box storage ~0.44 EUR, area pays all four cost components at once); cost_rates only prices the found designs afterwards, it does not steer the search. With strongly different rates the optimal corner moves to places the order never visits: with box material at 5 EUR/m3 the bisection returns 20832 EUR (155 m2, storage 300) where the simultaneous search - which carries cost_rates inside its objective - finds 12682 EUR (63 m2, storage 1200), 39% cheaper. Added as a regression test. Docs now state this in optimise_swale_design(), the bisection vignette's cost-rates section, and the simultaneous vignette's Einordnung, which also distinguishes the two causes of "simultaneous systematically cheaper": violated monotonicity (model alarm) vs. a cost hierarchy that no longer matches the chosen rates (use the simultaneous optimiser as the primary method for cost sensitivity studies). 164 tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
Follow-up to documenting the hard-coded cost hierarchy: for the current three levers the order does not need to be hard-coded at all. With the capacity model V ~ area * (mulde_height + porosity * storage_height) and every cost component proportional to area, each lever has a marginal cost per mm of capacity, the area cancels out of the comparison, and the cost-optimal corner of the feasibility boundary is computable analytically from the rates: - maximising mulde_height first is optimal for ANY rates under this cost model (it costs only excavation while area pays all four components plus the capacity-free filter depth) - so that part of the order is now provably rates-independent, not assumed - the starting storage level is the rate-dependent choice: solve_cell picks the level with minimal specific cost (EUR per mm capacity) instead of always the smallest; the continuous trench compares its endpoints (linear-fractional, endpoint-optimal). Under default rates the proxy picks the smallest level, reproducing the previous behaviour exactly. default_storage_spec() now carries the layer porosity (box 0.95, trench 0.3) that the proxy needs; specs without it keep the legacy order. With box material at 5 EUR/m3 the bisection now finds the 63 m2 / storage 1200 corner for 12.7k EUR in 15 runs - previously 20.8k EUR (155 m2 / storage 300), and on par with the simultaneous optimiser's 12.7k EUR at ~130 runs. The regression test now asserts both optimisers agree under changed rates, the legacy fallback, and unchanged default-rate behaviour. The proxy is first-order and only ranks capacity-additive levers - parameters with nonlinear hydraulic effects (e.g. variable filter kf) remain the domain of the simultaneous optimiser; docs and both vignettes updated accordingly. 167 tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
The final mulde_height trim has two distinct sources of slack, and only one of them is real: at low overflow targets it merely harvests the area search's tolerance overshoot (a search artefact worth ~18% of what finer area resolution would recover, since height only saves excavation while area pays all four cost components), at high targets it exploits the wide overflow-count plateaus where height has almost no hydraulic effect (real savings, e.g. 300 -> 100 mm at Eisenstadt x = 5, ~600 EUR, unreachable via finer area search because the next area step down immediately overshoots the target). Document the cheap fix for the artefact part: thanks to the bisection, halving area_tol costs exactly one additional engine run per area search while halving the worst-case cost overshoot (~area_tol x specific cost per m2, a few percent at the default 2 m2). Noted in the area_tol roxygen docs and as a comment on the vignette's search-space knob. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
- DESCRIPTION: version 0.0.0.9000 -> 0.1.0 - NEWS.md: development-version header becomes the 0.1.0 release header - CITATION.cff: citation metadata for GitHub's "Cite this repository" and the Zenodo GitHub integration (author with ORCID, MIT, keywords, abstract noting that the proprietary Tandler engine is downloaded from the separate kwb.raindrop.binaries repository and is not part of this package) - .zenodo.json: explicit Zenodo deposit metadata (creators, license, keywords, funding note for RAINDROP / GZ C300428) - inst/CITATION: R-style citation() entry - .Rbuildignore: exclude the repo-level citation files from the built package (inst/CITATION ships with it) The GitHub release v0.1.0 itself is cut from the default branch after merging; enable the Zenodo webhook for KWB-R/kwb.raindrop beforehand so the release is archived and receives a DOI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
- NEWS.md: the make_swale_runner() entry now leads with an explicit behaviour note that cleanup = TRUE (the default) deletes each scenario's output HDF5s (Mulde_Rigole.h5, Dach.h5, Fehlerprotokoll.h5, ...) right after the thinned result row is read, and that cleanup = FALSE keeps them - so consumers are not surprised that only the returned tibble survives a run. - test-optimise_swale_design_simultaneous.R: float comparisons (cost_total, mulde_area, storage_height, mulde_height) use expect_equal(tolerance = 0) instead of expect_identical - exact values without bitwise/type strictness; integer counts, strings and .Random.seed stay expect_identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
prepare_master() read //Kurven/Regen (and Growth_1/Shading_1) unconditionally, so a base.h5 template without these datasets crashed with a cryptic low-level error. Now the available datasets are listed once and only existing curves are touched: - timeseries_rain given: //Kurven/Regen must exist (clear error naming the template path otherwise); the Growth/Shading end-time fix is applied only to curves that exist and have the expected time/value shape - no timeseries_rain: the template rain curve is cached only if present; a per-run rain_factor != 1 without a usable template curve now throws a clear error instead of silently not scaling Requirements documented in the roxygen docs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017npvdq8XW1Lg1t2dNX9sGH
for simulataneous optimisation testing
…ng-3fg5o9 Add simultaneous optimisation
Optimisation
vignettes/index.Rmd (brute-force link hub) and vignettes/monotonicity_analysis.Rmd are deleted (they remain in the git history): their relative links only work in the deploy structure on raindrop.kompetenz-wasser.io and their result files never exist on GitHub Actions, so the pkgdown-built copies were link-dead shells. The complete rendered pages live on that server. - New "Ergebnisse" navbar menu linking the published results (brute force, monotonicity analysis, optimiser vs. brute force). - The two optimisation vignettes and the roxygen docs of find_min_feasible(), optimise_swale_design() and optimise_swale_design_simultaneous() now link the monotonicity analysis on that server instead of the removed vignette. - Drop the stale vignettes/index.Rmd entry from .Rbuildignore and the mono_* mention from .gitignore. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017E2RpwmBRnnDHRmxjXRh4x
…d8ab Remove deploy-only articles from pkgdown site, link results externally
The pkgdown navbar chrome is English (Get started, Reference, Articles, Changelog), so the new menu is now "More results" with English item labels (brute force, monotonicity analysis, optimiser vs. brute force). The German link texts inside the German-language vignettes are kept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017E2RpwmBRnnDHRmxjXRh4x
…d8ab Use English labels for the external-results navbar menu
| prepare_master <- function(paths) { | ||
| mp <- file.path(paths$dir_input, | ||
| sprintf("%s_master.h5", scenario_prefix)) | ||
| fs::dir_create(paths$dir_input, recurse = TRUE) | ||
| fs::file_copy(path = paths$path_base, new_path = mp, overwrite = TRUE) |
There was a problem hiding this comment.
Potential race condition on the shared master file when parallel workers target the same site.
The master path is derived only from paths$dir_input and scenario_prefix, so both future_lapply workers of the same site (different storage_type) point to the same <site>/input/o_master.h5. First call in each worker (there are two, one per storage type) will:
fs::file_copy(base.h5 → o_master.h5, overwrite = TRUE)hdf5r::H5File$new(mp, mode = "a")— opens the shared master in append mode
If both prepare_master calls interleave, worker B's file_copy(overwrite=TRUE) can stomp on the master file A currently has open in append mode, and/or the second H5File$new(..., mode="a") may fail due to HDF5 file locking. The vignettes in workflow_optimisation.Rmd fan out over expand.grid(site, type), so both workers of a site race here.
Suggested fixes (in order of decreasing intrusiveness):
- Include a per-worker discriminator in the master filename (e.g.
sprintf("%s_master_%s.h5", scenario_prefix, Sys.getpid())), so each worker has its own; or - Serialise creation via a lockfile (
fs::file_touch+while(...)polling) — brittle; - Document that
make_swale_runner()is not safe to reuse adir_inputacross concurrent workers.
The one-master-per-worker approach doubles disk use temporarily but eliminates the race entirely.
| timeseries_et$time[nrow(timeseries_et)] <- | ||
| ceiling(timeseries_et$time[nrow(timeseries_et)]) |
There was a problem hiding this comment.
Dead ceiling() on an already-integer column.
timeseries_et$time was computed above (line 34–36) as as.integer(difftime(..., units = "hours")), which is already an integer. ceiling() on integer values is a no-op — this line does nothing (compare with line 53–54 for rain, where time is a double and ceiling actually rounds up).
Two options:
- If the intent was to round up after
as.integer's truncation, replace the earlieras.integer(...)withas.double(...)so the truncation doesn't happen first, thenceilinghere matters. - If the intent is a no-op (ET is daily and always integer hours by construction, so nothing to round), drop lines 40–41 entirely.
| evap_all <- simulation_results_optimisation[["element.WB_Evapotranspiration_"]] | ||
| valid_mask <- !is.na(simulation_results_optimisation$n_overflows) & | ||
| simulation_results_optimisation$n_overflows <= x_int & !is.na(evap_all) | ||
| ref_idx <- if (any(valid_mask)) which(valid_mask) else seq_along(evap_all) | ||
| evap_min <- suppressWarnings(min(evap_all[ref_idx], na.rm = TRUE)) | ||
| evap_min_scenario <- simulation_results_optimisation$scenario_name[ | ||
| ref_idx[which.min(evap_all[ref_idx])]] | ||
|
|
||
| # Share of scenarios meeting the validity criterion (n_overflows <= x), | ||
| # appended to the auto-generated title (a plotly-safe place -- ggplotly | ||
| # drops ggplot subtitles). | ||
| valid_pct <- round(100 * mean( | ||
| simulation_results_optimisation$n_overflows <= x_int, na.rm = TRUE)) | ||
| share_txt <- switch(lang, | ||
| de = paste0(valid_pct, " % mit <= ", x_int, " \u00dcberl\u00e4ufen"), | ||
| en = paste0(valid_pct, " % with <= ", x_int, " overflows")) | ||
| if (is.null(title)) { | ||
| title <- paste0(def_title, " (", share_txt, ")") | ||
| if (y_var == "cost_per_evap_pct") { | ||
| # name the reference of the marginal measure in the title | ||
| title <- paste0(title, "\n", switch(lang, | ||
| de = paste0("Referenz: minimale Evapotranspiration der g\u00fcltigen ", | ||
| "Szenarien (<= ", x_int, " \u00dcberl\u00e4ufe): ", | ||
| round(evap_min, 1), " % (Szenario ", | ||
| evap_min_scenario, ")"), | ||
| en = paste0("Reference: minimum evapotranspiration of the valid ", | ||
| "scenarios (<= ", x_int, " overflows): ", | ||
| round(evap_min, 1), " % (scenario ", | ||
| evap_min_scenario, ")"))) | ||
| } | ||
| } |
There was a problem hiding this comment.
evap_min_scenario becomes character(0) when every evap_all entry is NA, and the title collapses to character(0).
When no scenarios are valid AND every value in evap_all is NA (pathological but possible for a fully-failed batch), the fallback ref_idx <- seq_along(evap_all) fires, then which.min(evap_all[ref_idx]) returns integer(0) (min on all-NA), so ref_idx[integer(0)] = integer(0) and evap_min_scenario = character(0).
At line 283–291 the German/English branch does paste0(..., evap_min_scenario, ...). paste0 with a character(0) argument returns character(0), which then flows into title <- paste0(title, "\n", character(0)) → character(0). Passing a character(0) title to ggplot2::labs() produces an unhelpful downstream error rather than the intended graceful degradation.
Suggested guard (minimal patch, right at the definition):
| evap_all <- simulation_results_optimisation[["element.WB_Evapotranspiration_"]] | |
| valid_mask <- !is.na(simulation_results_optimisation$n_overflows) & | |
| simulation_results_optimisation$n_overflows <= x_int & !is.na(evap_all) | |
| ref_idx <- if (any(valid_mask)) which(valid_mask) else seq_along(evap_all) | |
| evap_min <- suppressWarnings(min(evap_all[ref_idx], na.rm = TRUE)) | |
| evap_min_scenario <- simulation_results_optimisation$scenario_name[ | |
| ref_idx[which.min(evap_all[ref_idx])]] | |
| # Share of scenarios meeting the validity criterion (n_overflows <= x), | |
| # appended to the auto-generated title (a plotly-safe place -- ggplotly | |
| # drops ggplot subtitles). | |
| valid_pct <- round(100 * mean( | |
| simulation_results_optimisation$n_overflows <= x_int, na.rm = TRUE)) | |
| share_txt <- switch(lang, | |
| de = paste0(valid_pct, " % mit <= ", x_int, " \u00dcberl\u00e4ufen"), | |
| en = paste0(valid_pct, " % with <= ", x_int, " overflows")) | |
| if (is.null(title)) { | |
| title <- paste0(def_title, " (", share_txt, ")") | |
| if (y_var == "cost_per_evap_pct") { | |
| # name the reference of the marginal measure in the title | |
| title <- paste0(title, "\n", switch(lang, | |
| de = paste0("Referenz: minimale Evapotranspiration der g\u00fcltigen ", | |
| "Szenarien (<= ", x_int, " \u00dcberl\u00e4ufe): ", | |
| round(evap_min, 1), " % (Szenario ", | |
| evap_min_scenario, ")"), | |
| en = paste0("Reference: minimum evapotranspiration of the valid ", | |
| "scenarios (<= ", x_int, " overflows): ", | |
| round(evap_min, 1), " % (scenario ", | |
| evap_min_scenario, ")"))) | |
| } | |
| } | |
| evap_all <- simulation_results_optimisation[["element.WB_Evapotranspiration_"]] | |
| valid_mask <- !is.na(simulation_results_optimisation$n_overflows) & | |
| simulation_results_optimisation$n_overflows <= x_int & !is.na(evap_all) | |
| ref_idx <- if (any(valid_mask)) which(valid_mask) else seq_along(evap_all) | |
| evap_min <- suppressWarnings(min(evap_all[ref_idx], na.rm = TRUE)) | |
| wm <- which.min(evap_all[ref_idx]) | |
| evap_min_scenario <- if (length(wm) == 0L) { | |
| NA_character_ | |
| } else { | |
| simulation_results_optimisation$scenario_name[ref_idx[wm]] | |
| } | |
| # Share of scenarios meeting the validity criterion (n_overflows <= x), | |
| # appended to the auto-generated title (a plotly-safe place -- ggplotly | |
| # drops ggplot subtitles). | |
| valid_pct <- round(100 * mean( | |
| simulation_results_optimisation$n_overflows <= x_int, na.rm = TRUE)) | |
| share_txt <- switch(lang, | |
| de = paste0(valid_pct, " % mit <= ", x_int, " Überläufen"), | |
| en = paste0(valid_pct, " % with <= ", x_int, " overflows")) | |
| if (is.null(title)) { | |
| title <- paste0(def_title, " (", share_txt, ")") | |
| if (y_var == "cost_per_evap_pct") { | |
| # name the reference of the marginal measure in the title | |
| title <- paste0(title, "\n", switch(lang, | |
| de = paste0("Referenz: minimale Evapotranspiration der gültigen ", | |
| "Szenarien (<= ", x_int, " Überläufe): ", | |
| round(evap_min, 1), " % (Szenario ", | |
| evap_min_scenario, ")"), | |
| en = paste0("Reference: minimum evapotranspiration of the valid ", | |
| "scenarios (<= ", x_int, " overflows): ", | |
| round(evap_min, 1), " % (scenario ", | |
| evap_min_scenario, ")"))) | |
| } | |
| } |
| for (curve in c("//Kurven/Growth_1", "//Kurven/Shading_1")) { | ||
| if (!curve %in% existing) next | ||
| cv <- h5_read_values(h5m, paths = curve)[[curve]] | ||
| if (is.data.frame(cv) && length(cv$time) >= 2) { | ||
| cv$time[2] <- max(timeseries_rain$time) | ||
| static_vals[[curve]] <- cv | ||
| } | ||
| } |
There was a problem hiding this comment.
Only cv$time[2] is updated — silently corrupts any Growth/Shading curve with more than 2 points.
The extension logic assumes the shipped Growth_1 / Shading_1 curves are exactly two-point ramps (start → end). For any curve with 3+ knots, points 3..n retain their original time values while point 2 is jumped to the rain end, producing a non-monotone (or otherwise physically wrong) curve that the engine reads without complaint.
Given the site templates the vignettes exercise are all 2-point ramps this hasn't bitten yet, but the length(cv$time) >= 2 guard reads as if any length were handled. Either:
- tighten the guard to
length(cv$time) == 2and stop/skip otherwise, so the assumption is enforced; or - rescale all inner knots proportionally, e.g.
cv$time <- cv$time * (max(timeseries_rain$time) / cv$time[length(cv$time)]).
At minimum, add a comment stating the 2-point assumption.
| (length(starts) - si + 1)) | ||
| stats::optim(starts[[si]], objective, method = "Nelder-Mead", | ||
| control = list(maxit = 200, reltol = 1e-4, | ||
| warn.1d.NelderMead = FALSE)) |
There was a problem hiding this comment.
warn.1d.NelderMead = FALSE does nothing here — the search is 3-D.
That control is only consulted when par has length 1 (see ?stats::optim), so on a 3-parameter starting point it is silently ignored. Not a bug, but the flag reads as if it were suppressing something and future-maintainers will wonder what. Suggest removing it, or replacing with a brief comment noting that the 1-D branch is intentionally impossible here.
| warn.1d.NelderMead = FALSE)) | |
| stats::optim(starts[[si]], objective, method = "Nelder-Mead", | |
| control = list(maxit = 200, reltol = 1e-4)) |
| if (!feasible(axis_hi)) { | ||
| n_hi <- eval_at(axis_hi)$n_overflows | ||
| anchor <- NA_real_ | ||
| if (!is.na(n_hi) && n_hi <= x_max + wobble) { | ||
| offset <- axis_tol | ||
| repeat { | ||
| p <- axis_hi - offset | ||
| if (discrete) p <- ceiling(p) | ||
| if (p <= axis_lo) break | ||
| if (feasible(p)) { | ||
| anchor <- p | ||
| break | ||
| } | ||
| offset <- offset * 2 | ||
| } | ||
| } | ||
| if (is.na(anchor)) { | ||
| status <- "infeasible" | ||
| } else { | ||
| axis_hi <- anchor | ||
| } | ||
| } |
There was a problem hiding this comment.
Wobble ladder only searches a geometric offset sequence (1, 2, 4, 8, …) below the upper edge — a strictly-monotone-but-decreasing-in-large-steps violation between the edge and lower bounds can be missed.
In discrete mode with axis_hi = length(grid) = 4 and wobble = 1, the ladder tests indices 3 then 2 (offset 1 then 2), then p = 0 <= axis_lo breaks. If only index 1 is feasible while 2 and 3 are not, the ladder declares infeasible even though a feasible level exists.
This is documented as a monotonicity assumption (the volume referee catches most violations) and the description at line 12–23 spells this out, so it's likely intentional. But the ladder's exponential search means the tolerance window shrinks the further below axis_hi the feasible anchor sits — worth calling out in the docstring so users know the wobble is only a neighbour fix, not a global rescue. Consider adding one sentence to the doc:
The ladder walks downward in offsets
axis_tol, 2·axis_tol, 4·axis_tol, …; feasible anchors more than a few tolerance steps below the edge will only be found if strict monotonicity holds up to that step size.
| df <- df %>% | ||
| dplyr::mutate( | ||
| cost_per_evap_pct = dplyr::if_else( | ||
| .data[["element.WB_Evapotranspiration_"]] - evap_min > 0, | ||
| .data$cost_total / | ||
| (.data[["element.WB_Evapotranspiration_"]] - evap_min), | ||
| NA_real_ | ||
| ) | ||
| ) %>% | ||
| dplyr::filter(!is.na(.data[[y_col]])) |
There was a problem hiding this comment.
cost_per_evap_pct uses .data$cost_total in the numerator regardless of the active y_var.
The definition is cost_total / (evap - evap_min), but when the caller is going to filter by y_col = "cost_per_evap_pct" anyway, this is fine. Just worth verifying: is cost_total the intended numerator, or should it be the raw y_col value? If y_col == "cost_per_evap_pct" gets fed back through this expression on a subsequent invocation, the divide-by-evap would compound.
In current usage the numerator is cost_total unconditionally, so this is not a bug today — flagging as a defensive-review concern only. Consider a short comment: # always cost_total (not y_col) — the *derivation* of the marginal cost measure.
| } else { | ||
| 0 | ||
| } | ||
| u1 <- (area - area_bounds[1]) / diff(area_bounds) |
There was a problem hiding this comment.
u1 = (area - area_bounds[1]) / diff(area_bounds) — silently NaN if degenerate area_bounds = c(v, v) are ever passed.
Unlike u2 (guarded at line 423 with up > height_bounds[1]) and u3 (guarded for discrete via index; guarded for continuous via gb[2] > gb[1]), there's no diff(area_bounds) > 0 guard on the area axis. Passing area_bounds = c(100, 100) would produce u1 = NaN, which then propagates through pmin/pmax (both return NaN) and into stats::optim() where it errors with an opaque message.
Not encountered in the shipped defaults, and the stopifnot() at line 255 would arguably be the right place to add diff(area_bounds) > 0 if you want to fail loudly rather than by silent NaN.
| test_that("Optimierer findet das Kostenminimum (Vergleich mit Brute-Force)", { | ||
| run <- synthetic_run_factory(demand = 3.6e5) | ||
| out <- optimise_swale_design(run, x_targets = 0:3, | ||
| fixed = test_fixed, verbose = FALSE) | ||
|
|
||
| expect_true(all(out$status == "ok")) | ||
| expect_false(any(out$monotonicity_warning)) | ||
| # Zulaessigkeit: Ueberlaufziel eingehalten | ||
| expect_true(all(out$n_overflows <= out$x)) | ||
|
|
||
| for (i in seq_len(nrow(out))) { | ||
| type <- out$storage_type[i] | ||
| stor <- if (type == "infiltration_box") c(300, 600, 900, 1200) | ||
| else seq(900, 3600, by = 25) | ||
| ref <- reference_optimum(run, type, out$x[i], stor) | ||
| expect_false(is.null(ref)) | ||
| # innerhalb 5 % des (quasi-kontinuierlichen) Brute-Force-Optimums | ||
| expect_lte(out$cost_total[i], ref$cost_total * 1.05) | ||
| } | ||
|
|
||
| # Kosten-Wirksamkeits-Kurve: lockereres Ziel ist nie teurer | ||
| for (type in unique(out$storage_type)) { | ||
| cc <- out$cost_total[out$storage_type == type][order(out$x[out$storage_type == type])] | ||
| expect_true(all(diff(cc) <= 1e-9)) | ||
| } | ||
|
|
||
| # Laufbudget: alle 8 Zellen zusammen deutlich unter Brute-Force-Groesse | ||
| expect_lte(attr(out, "n_runs_total"), 200) | ||
| }) |
There was a problem hiding this comment.
Nice reference-optimum design — worth extending one edge that isn't currently covered.
The synthetic-model tests are impressively thorough. One case that isn't exercised in either test-optimise_swale_design.R or test-optimise_swale_design_simultaneous.R: the volume referee firing end-to-end (a monotonicity violation flowing all the way to monotonicity_warning = TRUE in the output tibble). test-find_min_feasible.R covers it for the primitive, but the plumbing that assigns res_a$monotonicity_violation and res_s$monotonicity_violation and OR's them into mono_warn in solve_cell() (R/optimise_swale_design.R:268, 287, 304) is not tested — a synthetic run_fn that returns a real non-monotonicity (n_overflows AND overflow volume increase with area) would catch a regression in that wiring.
Non-blocking; just a suggestion for a follow-up commit.
| kwb.utils::catAndRun( | ||
| messageText = sprintf("(%d/%d)) Reading results files for model run %s", | ||
| which(simulation_names == s_name), | ||
| length(simulation_names), | ||
| paths$dir_target_output), |
There was a problem hiding this comment.
which(simulation_names == s_name) inside the lapply returns multiple indices if simulation_names has duplicates, then sprintf errors out.
Purely defensive: shipping code and tests never pass duplicate scenario names, so this only matters for hand-written calls. A simple fix that stays byte-identical for the common (unique) case:
messageText = sprintf("(%d/%d)) Reading results files for model run %s",
match(s_name, simulation_names),
length(simulation_names),
paths$dir_target_output),match() returns just the first index — always length 1.
| # Point size: calibrate the scale to the valid region (0..x) and cap the | ||
| # extreme ">x" values, otherwise the many-overflow outliers (overflow | ||
| # volumes of several thousand m3) shrink every valid-region point to an | ||
| # invisible dot. pmin() caps; scale_size() below adds a minimum size so even | ||
| # zero-volume points (the 0-overflow box) stay visible. | ||
| valid_size <- df[[size_col]][!is.na(df$n_overflows) & df$n_overflows <= x_int] | ||
| size_cap <- suppressWarnings(max(valid_size[is.finite(valid_size)])) | ||
| if (!is.finite(size_cap) || size_cap <= 0) { | ||
| size_cap <- suppressWarnings(max(df[[size_col]], na.rm = TRUE)) | ||
| } | ||
| if (!is.finite(size_cap) || size_cap <= 0) size_cap <- 1 | ||
| df$size_plot <- pmin(df[[size_col]], size_cap) |
There was a problem hiding this comment.
size_cap semantics are only correct for size_by = "overflow_volume" — they invert the "larger = better" meaning for size_by = "evapotranspiration".
The clip-to-valid-region logic is justified for overflow volume: invalid scenarios have very large volumes that would compress the whole scale, so capping at the valid-region max keeps the valid points readable.
But for size_by = "evapotranspiration" (line 60–63 of the docstring: "larger points then mean more evapotranspiration, which is desirable"), invalid scenarios can have evapotranspiration higher or lower than valid ones — and if higher, capping them makes the desirable outliers look like the valid-region maximum, hiding the "outperformer that overflows too often" signal that the size scale is meant to convey.
Suggested guard: only apply the cap when size_by == "overflow_volume".
if (size_by == "overflow_volume") {
valid_size <- df[[size_col]][!is.na(df$n_overflows) & df$n_overflows <= x_int]
size_cap <- suppressWarnings(max(valid_size[is.finite(valid_size)]))
if (!is.finite(size_cap) || size_cap <= 0) {
size_cap <- suppressWarnings(max(df[[size_col]], na.rm = TRUE))
}
if (!is.finite(size_cap) || size_cap <= 0) size_cap <- 1
df$size_plot <- pmin(df[[size_col]], size_cap)
} else {
df$size_plot <- df[[size_col]]
}| #' @importFrom rlang .data | ||
| plot_cost_overflow_boxplot <- function(simulation_results_optimisation, | ||
| param_grid, | ||
| x = 5, |
There was a problem hiding this comment.
Sibling default mismatch: x = 5 here vs. x = 1 in plot_cost_vs_overflow_volume / plot_cost_vs_evaporation.
The docstring (line 22–27) advertises the tooltip as identical to plot_cost_vs_overflow_volume, and both plots derive evap_min from "scenarios that satisfy n_overflows <= x", so the same tooltip line will actually differ between the two plots at their defaults. If a user renders both in a vignette without overriding x, they'll see the same scenario reported with different cost_per_evap numbers side by side.
Either align the defaults (both to x = 1 or both to x = 5), or add a note to both docstrings that the derived tooltip line depends on x and users should pass the same value to both.
|
Two more findings for this PR that I couldn't attach as inline comments because the target lines fall outside this PR's diff hunks (they're in unchanged prose / defaults of a heavily-changed file): HIGH —
|
No description provided.