K8s: Hacky changes for TPC

[wanyunSu]

Description

Fixes issue #872

This PR enables a hacky way to readout the full TPC on np02-srv-005, addresses step2 in #895

Changes:
Change k8s configurations to specify resources for both TDE and BDE apps.
For BDE apps, strip '-1' from the hostname in the OKS config to schedule them on the actual np02-srv-005, but keep them for resource allocation from the k8s PM config.

To test it on the emulation mode, uses ehn1-daqconfigs

Example output:

((dbt) ) [wasu@np04-srv-019 ~]$ drunc-unified-shell k8s-CERN sessions/np02-emu-session.data.xml np02-emu-session claudia-k8s
                                                      claudia-k8s status                                                      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                            ┃ Info ┃ State      ┃ Substate   ┃ In error ┃ Included ┃ Endpoint                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ root-controller                 │      │ configured │ configured │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:31991 │
│   df-controller                 │      │ configured │ configured │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:37657 │
│     df-s02-d0                   │      │ configured │ idle       │ No       │ Yes      │ rest://np04-srv-002.cern.ch:53677 │
│     df-s02-d1                   │      │ configured │ idle       │ No       │ Yes      │ rest://np04-srv-002.cern.ch:59401 │
│     dfo-01                      │      │ configured │ idle       │ No       │ Yes      │ rest://np04-srv-011.cern.ch:39025 │
│   tpc-controller                │      │ configured │ configured │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:39745 │
│     emu-crp23-controller        │      │ configured │ configured │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:38235 │
│       runp02srv005tde-crp23-emu │      │ configured │ idle       │ No       │ Yes      │ rest://np02-srv-005.cern.ch:40431 │
│     emu-crp4-controller         │      │ configured │ configured │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:42981 │
│       runp02srv005eth0-emu      │      │ configured │ idle       │ No       │ Yes      │ rest://np02-srv-005.cern.ch:41737 │
│     emu-crp5-controller         │      │ configured │ configured │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:42257 │
│       runp02srv005eth1-emu      │      │ configured │ idle       │ No       │ Yes      │ rest://np02-srv-005.cern.ch:41381 │
│   trg-controller                │      │ configured │ configured │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:45491 │
│     mlt                         │      │ configured │ idle       │ No       │ Yes      │ rest://np04-srv-018.cern.ch:43383 │
└─────────────────────────────────┴──────┴────────────┴────────────┴──────────┴──────────┴───────────────────────────────────┘
[2026/06/30 12:29:41 UTC] INFO       shell_utils.py:365                       drunc.utils.ShellContext                           Current FSM status is configured. Available transitions are start, scrap. Available 
sequence commands are shutdown, start-run.

Type of change

• New feature / enhancement
• Optimization
• Bug fix
• Breaking change
• Documentation

List of required branches from other repositories

N/A

Change log

see above

Suggested manual testing checklist

LIST COMMANDS TO DEMONSTRATE CHANGE

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: np04-srv-019 from release NFD_DEV_260629_A9

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

• Unit tests (pytest --marker) passed
  • With relevant marker
  • Without marker
• Integration tests passed
  • Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
  • Full daqsystemtest_integtest_bundle.sh
• Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows
• Drunc integration tests pass (./scripts/drunc_integtest_bundle.sh)

Final checklist prior to marking this as "Ready for Review"

• Code is clearly commented.
• New unit tests have been added, or is documented in # ISSUE NUMBER
• A suitable reviewer has been chosen from this list.

Reviewer checklist

• This branch has been rebased with develop prior to testing.
• Suggested manual tests show changes.
• CI workflows fails documented (if present)
• Integration tests passed (on either np0x or IC HEP clusters)
  • Use the following guidelines to determine which of the integration tests you need to run
    • You do not need to run any integration tests if
      • Code changes are not associated with src/
      • PR changes only affect docstrings
      • In this case, be sure to validate any suggested manual testing.
    • Run only the minimum integration test as daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py if
      • PR changes only affect a few log entries
      • PR changes are small, and do not have a large impact on the workflow (use carefully)
    • Otherwise run the full integration test bundle as daqsystemtest_integtest_bundle.sh
  • What to do if the integration tests fail?
    • Only concern yourself if failures related to drunc are in the log files
    • If non-drunc failure appears:
      • Validate failure in fresh working area
      • Contact Pawel if unsure
• If you have ran the full integration test bundle, leave a comment on the PR stating
  • Which host the integration tests have ran on
  • [Optional] A copy of the test summary
• Drunc integration tests pass (scripts/drunc_integtest_bundle.sh)

Once the above boxes are checked, the PR(s) can be merged following the steps below.

Prior to merging

Choose one of the following an complete all substeps

• Changes only affect the Run Control, are in a single repository, and do not affect the end user.
  • Changes are documented in docstrings and code comments
  • Wiki has been updated if architectural or endpoint changes
• Otherwise
  • Workflow changes demonstrated in the Change Log (if necessary)
  • Wiki has been updated (if necessary)
  • #dunedaq-integration Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

[wanyunSu]

The k8s configurations for np02-srv-005 is currently defined this way to allow the TDE and BDE applications to run. It is not optimised due to the current tuning of thenp02-srv-005 server, which is outside the scope of this PR.

[PawelPlesniak]

As this PR is a complete hack, I would suggest leaving the long-term correct implementation in the code, but commented out with a relevant specifier as to why it was commented out. This should also have a child issue written that informs about the importance of reverting this change.

[PawelPlesniak]

As above

[PawelPlesniak]

Thank you Claudia.
For this process manager, did you run a full TPC readout or what is just in emulation mode? This was unclear to me from the description of this PR.

[wanyunSu]

Thank you Claudia. For this process manager, did you run a full TPC readout or what is just in emulation mode? This was unclear to me from the description of this PR.

Hi Pawel, thanks, I have only tested it in emulation mode, I plan to test it with the actual BDE readout this week (as the TDE is still not available).

[PawelPlesniak]

To validate the use of the emulation mode, I ran the following steps

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n last_fddaq $today_date
cd NFD_DEV_260706_A9
. env.sh
dbt-build
dbt-workarea-env
cd pythoncode
git clone git@github.com:DUNE-DAQ/drunc.git
cd drunc
git switch wanyunSu/k8s-tpc
pip install .
cd ../..
git clone ssh://git@gitlab.cern.ch:7999/dune-daq/online/ehn1-daqconfigs.git
cd ehn1-daqconfigs
git switch wanyunSu/k8s-005-emu
. setup_db_path.sh
cd ..
runconf-shifter-ui -a np04-emu -c ehn1-daqconfigs -l
(left everything as default, generated the "np02-emu-session.data.xml" config)
drunc-unified-shell k8s-CERN shifter-configs/2026-07-06_17-52-36/np04-emu_2026-07-06_17-52-36.data.xml np02-emu-session pplesnia
boot

Which got me the expected status table hanging on Waiting on tree initialization as

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                       ┃ Info ┃ State        ┃ Substate     ┃ In error ┃ Included ┃ Endpoint                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ root-controller            │      │ initialising │ initialising │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:31991 │
│   df-controller            │      │ initial      │ initial      │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:44195 │
│     df-s02-d0              │      │ initial      │ idle         │ No       │ Yes      │ rest://np04-srv-002.cern.ch:55209 │
│     df-s02-d1              │      │ initial      │ idle         │ No       │ Yes      │ rest://np04-srv-002.cern.ch:58569 │
│     dfo-01                 │      │ initial      │ idle         │ No       │ Yes      │ rest://np04-srv-011.cern.ch:36843 │
│   tpc-controller           │      │ initialising │ initialising │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:33795 │
│     emu-crp23-controller   │      │ initialising │ initialising │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:36485 │
│     emu-crp4-controller    │      │ initial      │ initial      │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:46587 │
│       runp02srv005eth0-emu │      │ initial      │ idle         │ No       │ Yes      │ rest://np02-srv-005.cern.ch:44065 │
│     emu-crp5-controller    │      │ initialising │ initialising │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:46411 │
│   trg-controller           │      │ initial      │ initial      │ No       │ Yes      │ grpc://np04-srv-024.cern.ch:43561 │
│     mlt                    │      │ initial      │ idle         │ No       │ Yes      │ rest://np04-srv-018.cern.ch:60103 │
└────────────────────────────┴──────┴──────────────┴──────────────┴──────────┴──────────┴───────────────────────────────────┘
Waiting on tree initialisation... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━  92% 0:00:06

for multiple minutes, ultimately leading to gRPC timeouts as

Traceback (most recent call last):
  File "/nfs/home/pplesnia/nightlyDev/260706/.venv/lib/python3.12/site-packages/drunc/controller/controller_driver.py", line 116, in status
    response = self.stub.status(request, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/home/pplesnia/nightlyDev/260706/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/home/pplesnia/nightlyDev/260706/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1009, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.DEADLINE_EXCEEDED
        details = "Deadline Exceeded"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:4, grpc_message:"Deadline Exceeded"}"
>

The above exception was the direct cause of the following exception:

drunc.utils.grpc_utils.ServerTimeout: Deadline Exceeded

This was taken 3 times to ensure reproducibility. When doing this, I was running as myself starting from np02-srv-028. Can you provide additional documentation on the process you took to deploy this emulation mode? I am unable to continue with other testing until I can reproduce your intended behaviour.

For documentation, I was able to see the pod manifests by viewing the namespace pplesnia on the headlamp cluster view, but these pods were left in a pending state, and were not booted despite leaving multiple minutes for this to be completed.

I also find the naming path of shifter-configs/2026-07-06_17-52-36/np04-emu_2026-07-06_17-52-36.data.xml quite confusing - I selected to use an np02 emulation mode configuration, but the file name suggests that this is an np04 emulation mode, and the application names are those following the np02 emulation mode.