Project Home Page: http://dmagic.readthedocs.org/

dmagic (Data Magic) is a tool developed at Argonne National Laboratory for managing
experiment metadata at APS (Advanced Photon Source) beamlines. It bridges the APS
scheduling system (which tracks who is running experiments and when) with the EPICS
control system used to operate beamline equipment, and the APS Data Management (DM)
system (Sojourner) for experiment and user access management.

Key Commands
------------

- dmagic init          : Creates ~/dmagic.conf with default values for the beamline.

- dmagic show          : Queries the APS scheduling REST API to display the currently
                         active experiment's information: PI name, affiliation, email,
                         badge number, proposal ID/title, and start/end times.

- dmagic tag           : Fetches the same scheduling data and writes it into EPICS
                         Process Variables (PVs) on the beamline's TomoScan IOC.
                         Populates UserName, UserEmail, ProposalNumber, ProposalTitle, etc.
                         If no proposal is found in the scheduling system, logs a clear
                         message and suggests running 'dmagic create-manual' followed by
                         'dmagic tag-manual'.

- dmagic tag-manual    : Interactively lists all DM experiments for the station (both
                         scheduling-based and manually created, sorted newest first) and
                         lets the operator choose which one to write to the EPICS PVs.
                         Useful for commissioning runs with no scheduling proposal, or
                         when sneaking in a sample from a different user group. For
                         non-zero GUP experiments it fetches full PI info from the
                         scheduling system; for manually created ones (GUP=0) it derives
                         the PI last name from the experiment name.

- dmagic create        : Creates a DM experiment on Sojourner for a proposal-based
                         beamtime. Lists all beamtimes in the current run, prompts for
                         selection, then creates the experiment and adds all proposal users.

- dmagic create-manual : Creates a DM experiment manually for commissioning runs or
                         staff experiments that have no scheduling proposal. PI name,
                         title, and optional badge numbers are provided on the command
                         line. Optional arguments:
                           --gup   GUP number (default: 0 = no proposal/commissioning)
                           --date  year-month in yyyy-mm format (default: current month)
                           --start experiment start date in yyyy-mm-dd format
                                   (default: first day of --date month)
                           --end   experiment end date in yyyy-mm-dd format
                                   (default: 14 days after --start)
                         Experiment name format: yyyy-mm-{LastName}-{gup}
                         Example: dmagic create-manual --name DeCarlo
                                    --first-name Francesco --gup 1
                                    --start 2026-05-01 --end 2026-05-07
                                    --title "2-BM commissioning"

- dmagic delete        : Deletes a DM experiment from Sojourner. Lists all experiments
                         for the station (last 2 years, including manually created ones)
                         and requires double confirmation before deleting.

- dmagic email         : Sends a data-access notification email with a Globus link to
                         all users on a DM experiment. Lists station experiments and
                         prompts for selection.

- dmagic daq-start     : Starts automated file transfer (DAQ) to Sojourner. On start,
                         the DM system uploads every file already present in the
                         experiment directory (raw and _rec), then continues to sync
                         new/changed files as they arrive. Runs until stopped with
                         'dmagic daq-stop'. Because it also handles pre-existing files,
                         it can be started at any time — before, during, or after the
                         acquisition — and will still catch every file.

- dmagic daq-stop      : Lists experiments that currently have running DAQs for the
                         station (skipping the full experiment list), and stops all
                         DAQs for the one you select. Exits cleanly if none are running.

- dmagic daq-status    : Lists all currently running DM DAQs for the station: experiment
                         name, source directory, files processed / total, percentage
                         complete, runtime, and DAQ id. Read-only.

- dmagic upload        : One-shot upload of all existing files in an experiment
                         directory to Sojourner. Now essentially redundant with
                         'dmagic daq-start' (which also uploads pre-existing files);
                         kept as a fallback for cases where daq-start was not running
                         during data collection.

- dmagic add-user      : Adds one or more users to an existing DM experiment by badge
                         number. Prompts interactively if no badges are provided on the
                         command line.

- dmagic remove-user   : Removes one or more users from an existing DM experiment by
                         badge number. Shows the current user list before prompting.

- dmagic list-users    : Lists all users currently granted access to a DM experiment,
                         including their DM username, full name, and email address.

- dmagic list-esafs    : Lists ESAFs for the beamline station in a date range using the
                         DM EsafApsDbApi. Options:
                           --start-date  range start in yyyy-mm-dd format
                                         (default: first day of current month)
                           --end-date    range end in yyyy-mm-dd format (default: today)
                         Output: one line per ESAF with id, status, start/end dates, title.

Per-beamline configuration (2-BM, 7-BM, 32-ID, ...)
---------------------------------------------------

Defaults baked into config.py are 2-BM values. To install at another beamline,
run 'dmagic init' and then edit the [site] and [local] sections of
~/dmagic.conf with beamline-specific values (globus-server-uuid,
experiment-type, globus-message-file, primary/secondary contact badges and
emails, tomoscan-prefix, tomolog-home, plus [local] data-host and data-top-dir).

Full per-beamline lookup table (with Globus UUIDs and staff contacts for
2-BM, 7-BM, and 32-ID) and instructions for adding a new beamline are in
docs/source/usage.rst under 'Configuring for a different beamline'.

DM data-directory format
------------------------

The [local] config knob 'dm-direct-mount' controls how dmagic hands the
source path to the DM DAQ:

  - dm-direct-mount = True (default at 2-BM): dmagic passes the bare local
    path '{data-top-dir}/{exp-name}'. Correct when the DM VM already
    mounts the data filesystem directly (2-BM DM VM has both
    tomodata2:/data2 and tomodata3:/data3 mounted).

  - dm-direct-mount = False: dmagic passes
    '@{data-host}:{data-top-dir}/{exp-name}' — DM rsyncs from the
    data host over SSH. This form has a bug on the 2-BM DM installation
    where directory scans silently return countFiles=0 with
    "no new files for upload" even for directories with matching files,
    so leave the default (True) unless a future DM release fixes that.

Passwordless SSH prerequisite
-----------------------------

'dmagic upload' and 'dmagic daq-start' pre-check the source directory before
dispatching to DM. On beamline nodes that already mount the data top-dir
(tomo1, tomo3, tocai, ...) no setup is needed. On a control computer that does
NOT mount /data2 or /data3 (e.g. arcturus), dmagic falls back to an SSH probe
on the data host — set up passwordless SSH once:

    ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519    # only if no key yet
    ssh-copy-id tomodata2
    ssh-copy-id tomodata3

If SSH is not set up, dmagic prints a warning with the exact commands above
and dispatches to DM anyway (the pre-check is a safety net, not required).

How It Works
------------

1. Authenticates against the APS scheduling REST API using stored credentials.
2. Determines the current APS run (e.g., "2026-1") based on the current date.
3. Finds the proposal(s) scheduled for the current beamline in that run.
4. Extracts PI and experiment metadata from the proposal.
5. Optionally writes that metadata to EPICS PVs so downstream data acquisition software
   (like TomoScan) can tag collected data with the correct experiment/user info.
6. Optionally creates a DM experiment on Sojourner, populates it with users from the
   scheduling proposal, starts automated file transfer (DAQ), and notifies users of
   their Globus data access link.

Use Case
--------

Primarily used at tomography beamlines (e.g., 2-BM, 7-BM, 32-ID) so that when a new
user group arrives to run their approved experiment, the beamline control system is
automatically updated with their identity and proposal information, ensuring data files
are properly attributed without manual entry. The DM integration automates creation of
data management experiments and notifies users of how to access their data via Globus.