CMIP7: Identify whether existing submission code could be updated to work with CMIP7 data

[matthew-mizielinski]

[mo-gill]

I'll post some comments with summaries of various aspects of the submission code that were generated with the help of copilot. I imagine the text will be more helpful than the diagrams.

[mo-gill]

main_move_in_mass — Overview

Entry point

main_move_in_mass is the CLI entry
function behind the move_in_mass bin script (cdds/bin/move_in_mass, not shown here — see
cdds/cdds/deprecated/transfer/command_line.py). It lives in the cdds.deprecated.transfer
subpackage — a legacy component that predates the modern cdds.common.plugins / cdds.common.request
framework.

Purpose in the submission pipeline

CDDS publishes CMOR-compliant NetCDF output to MASS (the Met Office Massive Archive Storage System) in
stages, tracked as a small state machine per "atomic dataset" (one variable within one dataset id).
Full transition rules are in 03-state-model.md; summary:

stateDiagram-v2
    [*] --> Embargoed : send_to_mass (not move_in_mass)
    Embargoed --> Available : move_in_mass
    Available --> Withdrawn : move_in_mass
    Withdrawn --> Available : move_in_mass
    Available --> Superseded : move_in_mass
    Superseded --> Withdrawn : move_in_mass

Loading

• embargoed: data has landed in MASS (via send_to_mass, a separate tool) but is not yet public.
• available: data has been approved (QC passed) and is published/visible to CEDA/ESGF.
• withdrawn: data has been pulled from publication (e.g. errata, replaced version).
• superseded: an older version has been replaced by a newer one.

move_in_mass is the tool that performs the state transition step: given a request config, an
--original_state, a --new_state, and a file listing QC-approved variables, it:

1. Locates the matching atomic datasets already sitting in MASS in original_state.
2. Filters them down to only the variables that passed QC (--approved_variables_path).
3. Physically moves each dataset's directory in MASS from the old state's location to the new state's
   location (moo mkdir / moo mv / moo rmdir).
4. If the new state is one CEDA/BADC cares about (available or withdrawn), publishes a message to a
   RabbitMQ queue so CEDA's harvesting/ESGF-publishing side picks up the change.

See 01-call-stack.md for the full function-by-function trace,
02-architecture.md for diagrams, 03-state-model.md for the
state machine detail, and 04-external-dependencies.md for the MASS/moo
and RabbitMQ specifics.

Why this matters for the CMIP7 rewrite

This code path is CMIP6/CMIP6Plus-only in practice today, even though the request/plugin layer above
it already understands CMIP7 (cdds.common.plugins.cmip7.Cmip7Plugin, MipEra.CMIP7). Two concrete,
verified gaps block CMIP7 use of move_in_mass as-is — see
06-gotchas-and-showstoppers.md for details:

1. No general config file exists for CMIP7 (CMIP7/general/CMIP7.cfg), which
   CDDSConfigGeneral requires to build the transfer
   facet maps.
2. KNOWN_RABBITMQ_QUEUES hardcodes only
   CMIP6/CMIP6Plus queue names, so no CMIP7 message queue is recognised.

A source comment in common.py itself
describes the general-config-to-transfer-config bridge as "a nasty hack, as the actual replacement of
the config objects within cdds.deprecated.transfer will take too much work" — i.e. the original
authors already anticipated this package would eventually need replacing rather than patching.

Document index

File	Contents
00-overview.md	This file
01-call-stack.md	Full call tree, file:line references
02-architecture.md	Mermaid flowchart + sequence diagram
03-state-model.md	State machine (embargoed/available/withdrawn/superseded)
04-external-dependencies.md	moo/MASS commands, RabbitMQ, filesystem paths
05-docstring-mismatches.md	Docstring vs code discrepancies
06-gotchas-and-showstoppers.md	Hidden assumptions, dead code, CMIP7 blockers

[mo-gill]

Full call stack for main_move_in_mass

All paths relative to the repo root unless otherwise noted. Line numbers captured during this analysis
session (July 2026) — re-verify if the source has since changed. See
00-overview.md for context and 02-architecture.md for diagrams.

Tree

flowchart TD
    A["main_move_in_mass()<br/>command_line.py:31-56"]
    A --> B["parse_arguments_move_in_mass()<br/>command_line.py:59-92"]
    B -->|"if -o given"| B1["check_directory()<br/>common/__init__.py:359-385"]
    A --> C["read_request()<br/>request.py:190-217"]
    C --> C1["read_request_config()<br/>request.py:220-260"]
    C --> C2["load_cdds_plugins()"]
    C2 --> C2a["MipEra.is_cmip7 → load_cmip7_plugin()<br/>plugin_loader.py:47-48,101-117<br/>base_plugin.py:82-95,128"]
    C --> C3["Calendar.default().set_mode()"]
    C --> C4["Request.from_config()"]
    C --> C5["validate_request()<br/>(unless relaxed_cmor)"]
    A --> D["configure_logger()<br/>common/__init__.py:33-104<br/>NOTE: output_dir never passed"]
    A --> E["run_move_in_mass()<br/>state_change.py:22-72"]

    E --> F["CDDSConfigGeneral()<br/>deprecated/config.py:41-55"]
    F --> F1["root_config()<br/>general_config/__init__.py:7-16<br/>only CMIP6/ CMIP6Plus/ exist"]

    E --> G["cfg_from_cdds_general_config()<br/>deprecated/transfer/common.py:51-84"]

    E --> H["load_rabbit_mq_credentials()<br/>common.py:14-48<br/>raises RuntimeError if fail & not simulation"]

    E --> I["drs_facet_builder_from_request()<br/>common.py:87-112"]
    I --> I1["DataRefSyntax()<br/>drs.py:80-107"]
    I1 --> I1a["_mandatory_section('project')<br/>drs.py:642-648"]
    I1 --> I1b["_optional_section('options')<br/>drs.py:649-661"]
    I1 --> I1c["_check_config_consistency()<br/>drs.py:553-586"]
    I1 --> I1d["_define_handlers()"]
    I --> I2["fill_facets_from_dict()<br/>drs.py:141-172"]

    E --> J["DataTransfer()<br/>dds.py:21-49<br/>if simulation: _simulation = LS_ONLY"]

    E --> K["find_mass()<br/>common.py:170-192"]
    K --> K1["state.make_state(current_state)<br/>state.py:16-30"]
    K --> K2["find_mass_facets()<br/>dds.py:308-345"]
    K2 --> K2a["_check_constraints_valid()"]
    K2 --> K2b["_find_all_mass_facets()<br/>dds.py:531-542"]
    K2b --> K2b1["mass_max_path()<br/>drs.py:412-427"]
    K2b --> K2b2["moo_cmd.ls_tree()<br/>moo_cmd.py"]
    K2b2 --> K2b2a["moo.run_moo_cmd('ls')<br/>moo.py:66-141"]
    K2b --> K2b3["_parse_urls_from_xml()"]
    K2b3 --> K2b3a["moo.parse_xml_output()<br/>moo.py:144-166"]
    K2b --> K2b4["_mass_to_facets()"]
    K2b4 --> K2b4a["fill_facets_from_mass_dir()<br/>drs.py"]
    K -->|"none found"| K3["raise RuntimeError('Found no matching facets')"]

    E --> L["read_variables_list_file()<br/>state_change.py:98-138"]
    E --> M["filter_filesets()<br/>drs.py:936-953"]
    E --> N["log_filesets()<br/>common.py:115-133"]

    E --> O["move_in_mass()<br/>state_change.py:76-93"]
    O --> O1["state.make_state(original_state)"]
    O --> O2["state.make_state(new_state)"]
    O --> O3["change_mass_state()<br/>dds.py:139-156<br/>raises ValueError if invalid transition"]
    O3 --> P["_run_move() per fileset<br/>dds.py"]
    P --> P1["_latest_ts_for_state()"]
    P1 --> P1a["_list_atoms()"]
    P1a --> P1a1["moo_cmd.ls()"]
    P1a1 --> P1a1a["moo.run_moo_cmd('ls', ['-d',...])"]
    P --> P2["_mass_path_to_timestamp() old_state"]
    P --> P3["_mass_path_to_timestamp() new_state"]
    P --> P4["_move_atom()"]
    P4 --> P4a["moo_cmd.dir_exists()"]
    P4 --> P4b["moo_cmd.mkdir() if absent"]
    P4 --> P4c["moo_cmd.mv()"]
    P4 --> P4d["moo_cmd.rmdir()"]
    O3 --> Q["inform() per fileset<br/>dds.py:230-251"]
    Q -->|"new_state.inform() True (Available/Withdrawn)"| Q1["_prepare_message()"]
    Q1 -->|"simulation"| Q2["log 'Simulating publishing message...'"]
    Q1 -->|"else"| Q3["Communication()<br/>msg.py:507-518"]
    Q3 --> Q3a["RabbitMqManager().start()<br/>rabbit.py:39-64"]
    Q3 --> Q4["publish_message()<br/>msg.py:520-544"]
    Q4 --> Q4a["PersistentPublish()<br/>rabbit.py:80-105"]
    Q4 -->|"if not published"| Q4b["store_message()"]

    E --> R["finally: log 'Moving complete' or log+re-raise"]

Loading

Notes on scope

• send_to_mass, rerun_send_to_mass, rerun_change_mass_state, copy_from_mass,
  find_local_facets and everything under search_local_directories in
  dds.py are not called by main_move_in_mass — they
  belong to the sibling send_to_mass/rerun tooling and are out of scope for this trace (though relevant
  if the CMIP7 rewrite touches the whole cdds.deprecated.transfer package).
• main_list_queue and main_resend_failed_msg (also in
  command_line.py) are separate CLI entry
  points, not part of main_move_in_mass's call graph — out of scope.

[mo-gill]

Architecture diagrams

See 01-call-stack.md for the authoritative function-by-function listing with
file:line references. Diagrams here are a visual summary only.

Control-flow overview

flowchart TD
    A[main_move_in_mass] --> B[parse_arguments_move_in_mass]
    A --> C["read_request(request_path)"]
    C --> C1[read_request_config]
    C --> C2[load_cdds_plugins]
    C --> C3["Request.from_config / validate_request"]
    A --> D[configure_logger]
    A --> E[run_move_in_mass]

    E --> F[CDDSConfigGeneral]
    F -.reads.-> F1["{mip_era}/general/{mip_era}.cfg"]
    E --> G[cfg_from_cdds_general_config]
    E --> H[load_rabbit_mq_credentials]
    H -.reads.-> H1["$HOME/.cdds_credentials"]
    E --> I[drs_facet_builder_from_request]
    E --> J["DataTransfer(...)"]
    E --> K[find_mass]
    K --> K1[find_mass_facets]
    K1 --> K2["moo_cmd.ls_tree (moo ls -xR)"]
    E --> L[read_variables_list_file]
    L -.reads.-> L1[approved_variables_path file]
    E --> M[filter_filesets]
    E --> N[log_filesets]
    E --> O[move_in_mass]
    O --> P[change_mass_state]
    P --> Q["_run_move (per fileset)"]
    Q --> Q1["moo mkdir / moo mv / moo rmdir"]
    P --> R["inform (per fileset)"]
    R -->|new_state.inform==True| S[publish_message to RabbitMQ]
    R -->|new_state.inform==False| T[skip]
    S -->|publish fails| U[store_message locally]

    classDef external fill:#fee,stroke:#933;
    class F1,H1,L1,K2,Q1,S external;

Loading

Sequence diagram (successful available → withdrawn example)

sequenceDiagram
    participant CLI as main_move_in_mass
    participant RM as run_move_in_mass
    participant DT as DataTransfer
    participant MOO as moo CLI (MASS)
    participant RMQ as RabbitMQ (moose.* queue)

    CLI->>RM: run_move_in_mass(request, args)
    RM->>RM: CDDSConfigGeneral(request)
    RM->>RM: load_rabbit_mq_credentials(cfg)
    RM->>DT: DataTransfer(cfg, mip_era)
    RM->>DT: find_mass_facets(facets, "available")
    DT->>MOO: moo ls -xR -p1-1000:25000 {mass_path}
    MOO-->>DT: XML directory listing
    DT-->>RM: AtomicDatasetCollection (matching filesets)
    RM->>RM: read_variables_list_file(approved_variables_path)
    RM->>RM: filter_filesets(filesets, approved_vars)
    RM->>DT: change_mass_state(filesets, Available, Withdrawn)
    loop for each fileset
        DT->>MOO: moo test -d {new_dir}
        DT->>MOO: moo mkdir -p {new_dir}  (if missing)
        DT->>MOO: moo mv {old_dir}/* {new_dir}
        DT->>MOO: moo rmdir {old_dir}
        DT->>DT: inform(fileset, new_dir, Withdrawn)
        alt Withdrawn.inform() == True
            DT->>RMQ: publish MooseMessage to moose.{mip_era}_withdrawn
            RMQ-->>DT: ack / failure
            opt publish failed
                DT->>DT: store_message(message) locally
            end
        end
    end
    RM-->>CLI: (exit code 0)

Loading

Data flow: request → facets → MASS path

flowchart LR
    REQ["Request cfg\n(request.metadata.mip_era,\nrequest.flattened_items)"] --> DRS["DataRefSyntax\n(drs_fixed_facet_builder)"]
    GENCFG["{mip_era}.cfg\n[transfer_facetmaps]\n(valid|atomic|name|dataset_id|\nlocal|sublocal|mass|pattern)"] --> DRS
    DRS -->|mass_max_path| MASSPATH["MASS wildcard path\nmoo:/.../.../*"]
    MASSPATH --> LSTREE["moo ls -xR"]
    LSTREE --> FACETS2["facets parsed back\nfrom MASS dir names"]
    FACETS2 --> COLLECTION["AtomicDatasetCollection\n(filesets to move)"]

Loading

[mo-gill]

State model

Defined in cdds/cdds/deprecated/transfer/state.py.
Referenced from the call stack at 01-call-stack.md.

States

Constant	Value	Defined at
EMBARGOED	"embargoed"	state.py:6
AVAILABLE	"available"	state.py:7
WITHDRAWN	"withdrawn"	state.py:8
SUPERSEDED	"superseded"	state.py:9

KNOWN = [EMBARGOED, AVAILABLE, WITHDRAWN, SUPERSEDED] (state.py:11) is returned by
known_states() (state.py:33-35) and used directly as the choices= list for both
--original_state and --new_state in
parse_arguments_move_in_mass — i.e.
the CLI's set of valid state names is this list, so adding/renaming a state only requires editing
state.py.

make_state(name: str) -> State (state.py:16-30) is the factory used everywhere a state name string
(from argparse or from a MASS directory path) needs to become a State object; raises ValueError for
unrecognised names.

What original_state / new_state mean operationally

• --original_state: state the target atomic datasets are currently in, in MASS. Used by
  find_mass (common.py:170-192) to build
  the MASS path used for the moo ls listing (i.e. it's a search filter, not a validated pre-check —
  if nothing is in that state matching the facets, you get RuntimeError('Found no matching facets'),
  not a state-specific error).
• --new_state: state to move matched datasets into. Combined with --original_state, the pair is
  validated as a legal transition only at the point change_mass_state runs
  (old_state.can_move_to(new_state),
  dds.py:139-142) — i.e. after MASS has
  already been searched and the approved-variables file has already been read. An illegal combination
  (e.g. --original_state available --new_state embargoed) is discovered late, not at argument-parsing
  time.

Transition graph

Each State subclass declares _valid_from_state, the set of states it can be reached from
(can_come_from); can_move_to(state) is just state.can_come_from(self) (state.py:73-78).

stateDiagram-v2
    [*] --> Embargoed : send_to_mass (not move_in_mass)
    Embargoed --> Available : move_in_mass
    Available --> Withdrawn : move_in_mass
    Withdrawn --> Available : move_in_mass
    Superseded --> Withdrawn : move_in_mass
    Available --> Superseded : move_in_mass

Loading

State	can_come_from (_valid_from_state)	can_be_put()	inform()
Embargoed	(none — unreachable via move_in_mass)	True	False
Available	Embargoed, Withdrawn	False	True
Withdrawn	Available, Superseded	False	True
Superseded	Available	False	False

(state.py:99-149)

• can_be_put(): gates send_to_mass (dds.py) — only Embargoed allows fresh data to be put into
  MASS. Irrelevant to move_in_mass itself but explains why Embargoed has no valid "come from" states
  (you can't move_in_mass something into embargoed; it only gets there via initial ingest).
• inform(): gates whether a RabbitMQ message is sent on transition — see
  DataTransfer.inform. Only transitions
  landing on Available or Withdrawn notify CEDA/BADC. Landing on Superseded (or Embargoed, though
  unreachable here) is silent.

Docstring vs code note

The move_in_mass() function docstring
(state_change.py:92) lists the states as
"'embargoed', 'available', 'withdrawn' or 'superceded'" — misspelled; the real constant is
superseded (state.py:9). See 05-docstring-mismatches.md.

[mo-gill]

External dependencies

Ask me for info on this and i'll send it to you private. The info looks useful - but i'm concerned some of the info around RabbitMQ should be kept off the private/off the repo, so playing it safe.

[mo-gill]

Gotchas and showstoppers

Same situation as above.

[mo-gill]

Docstring mismatches

Every entry below quotes the docstring, states the code reality, and explains the discrepancy. See
01-call-stack.md for the surrounding call context.

1. "superceded" typo

Location: move_in_mass() docstring, state_change.py:92

Notes

The original and new states must be in cdds.deprecated.transfer.state.KNOWN,
i.e. 'embargoed', 'available', 'withdrawn' or 'superceded'.

Code reality: the actual constant is SUPERSEDED = "superseded"
(state.py:9).

Mismatch: plain misspelling (superceded vs superseded). Harmless but should be fixed if this
docstring is retained/ported.

2. DataTransfer.__init__ — simulation typed as bool, actually tri-state

Location: DataTransfer.__init__ docstring, dds.py:44-49

Parameters

...
simulation: bool
print moose commands rather than run them

Code reality: moo_cmd.py defines LS_ONLY = 'LS_ONLY'
(moo_cmd.py:26), a string sentinel that several
moo_cmd functions (ls, ls_file_sizes, ls_tree) explicitly check for with
if simulation == LS_ONLY: to force a real moo ls even while "simulating". run_move_in_mass
directly sets transfer_service._simulation = LS_ONLY
(state_change.py:53) — a string, not a
bool.

Mismatch: the type is documented as bool but the real value space is False | True | 'LS_ONLY'.
Additionally, the docstring only describes the constructor parameter, but the actual "simulation" value
used in move_in_mass is set via direct attribute mutation from outside the class
(transfer_service._simulation = LS_ONLY) rather than through the constructor at all — the docstring
gives no indication this attribute is meant to be externally reassigned post-construction.

3. move_in_mass() (function) — new_state/original_state typed as State, called with str

Location: move_in_mass() docstring, state_change.py:76-93

Parameters

...
original_state : str
State to move data sets from.
new_state : :class:cdds.deprecated.transfer.state.State
State to move data sets to.

Code reality: the sole caller,
run_move_in_mass, calls
move_in_mass(filesets, transfer_service, args.original_state, args.new_state) where both
args.original_state and args.new_state are plain strings from argparse (constrained by
choices=known_states() in
command_line.py:88-95). The function
body itself does start_state = state.make_state(original_state) and
end_state = state.make_state(new_state) — i.e. it converts both params from str to State
internally.

Mismatch: original_state is (correctly) documented as str, but new_state is documented as
State even though it is always passed and used as a str right up until the internal
state.make_state() call — the two params are handled identically by the code but documented with two
different types. This is an internal inconsistency, not just wrong vs. reality.

4. DataRefSyntax class docstring — self-flagged uncertainty

Location: DataRefSyntax class docstring, drs.py:40-44

The following may no longer be true;
All the fill_facets_from_dict methods will first clean out any
defined facets before doing a fill, so you can safely re-use a
DataRefSyntax object inside a loop if you wish.

Code reality: fill_facets_from_dict(facets_dict, update=False)
(drs.py:141-172) does call _init_facets()
only if not update — so the "clean out defined facets" behavior is conditional on update=False, which
is the default, so the claim is roughly still accurate for default usage, but several call sites pass
update=True (e.g. fill_facets_from_drs_name, fill_facets_from_local_path,
fill_facets_from_message, fill_facets_from_serialisable all accept an update param that is
sometimes True), where facets are explicitly not cleaned first.

Mismatch: the docstring itself admits doubt about its own accuracy ("may no longer be true") rather
than describing the actual update flag behavior — i.e. it's an acknowledged-stale docstring left
unresolved rather than fixed. Flagging as-is per task instructions (don't trust docstrings blindly).

5. state.State class docstring — stale/unverifiable external link

Location: State class docstring, state.py (class
State, abstract base)

Child classes implement the state transitions documented in:
https://github.com/MetOffice/climate-dds/blob/master/doc/dds_states.pdf

Code reality: cannot verify this URL is still live or that climate-dds is the current/accessible
repository (this is a documentation reference, not something checkable from within this repo).

Mismatch: not a code/docstring contradiction, but a likely-stale external reference that should be
re-pointed at whatever canonical spec is used post-rewrite (or replaced with the state-transition table
now captured in 03-state-model.md, which was derived directly from the
_valid_from_state code rather than this external doc).