Jackhammer agent

[tristpinsm]

Here is a first draft for a jackhammer agent. It should allow individual slots to fail and track the results.

Currently, I think you could set up an OCS-web panel that would display the result for each slot. This would only update to display the last jackhammer run however (and I guess only show the slots that were requested in that run).

If we wanted to have a continuously updating panel, we could add a monitor process to the agent that queries the SmurfApplication.SystemConfigured register, modeled on the ACU agent. I'm not super familiar with OCS, so let me know if that is a good idea.

Requires simonsobs/sodetlib#514

[tristpinsm]

actually, if it's possible for an OCS panel to display data from an ocs feed, we wouldn't need another monitor process and could get this info from the det-controller agents (with a config change for the streamers).

[tristpinsm]

@BrianJKoopman How do you usually proceed for testing new agents? Can I run this on a LAT smurf server in parallel to the production OCS and try hammering slots?

[mhasself]

Non-exhaustive review -- looks good for the most part but:

• You'll need to add a page to the docs, especially including configuration information.
• Consider starting agent name with Smurf or Pysmurf so that it sorts better in agent lists (e.g. in ocs sidebar; hostmanager lists). E.g. SmurfHammerAgent
• It's nice to require privs=2, but functionally that will make this inert right now because we don't run access control at the site! Can be changed later.

[msilvafe]

Non-exhaustive review -- looks good for the most part but:

• You'll need to add a page to the docs, especially including configuration information.
• Consider starting agent name with Smurf or Pysmurf so that it sorts better in agent lists (e.g. in ocs sidebar; hostmanager lists). E.g. SmurfHammerAgent
• It's nice to require privs=2, but functionally that will make this inert right now because we don't run access control at the site! Can be changed later.

I don't think we want to release this to the world without access control. What's the timescale for that?

[tristpinsm]

Thanks for taking a look! I renamed the agent and added to the docs, but holding off on changing the permissions until we decide on the path forward.

If escalating permissions is not supported right now, I don't feel like it's very risky to deploy this without locking it down. I seems to me like anyone with access to ocs-web can already do a lot of damage (potentially a lot worse than hammering SMuRF).

btw I'll be out on vacation next week, so if this converges on that timescale and someone wants to push it through, don't wait on me!

[msilvafe]

Thanks for taking a look! I renamed the agent and added to the docs, but holding off on changing the permissions until we decide on the path forward.

If escalating permissions is not supported right now, I don't feel like it's very risky to deploy this without locking it down. I seems to me like anyone with access to ocs-web can already do a lot of damage (potentially a lot worse than hammering SMuRF).

btw I'll be out on vacation next week, so if this converges on that timescale and someone wants to push it through, don't wait on me!

Fair point @tristpinsm!

[tristpinsm]

In anticipation of needing to bring up all the smurf crates at site sometime in the next week, I'd like to be ready to deploy and test this agent when that happens. Here's how I think that would work:

On a given smurf server:

1. edit $OCS_CONFIG_DIR/default.yaml and add

{'agent-class': 'SmurfHammerAgent',
     'instance-id': 'smurf-hammer',
     'manage': 'host',
     'arguments': []},

to the list of agents.

2. start the agent with the following command:

$ ocs-agent-cli --site-file=$OCS_CONFIG_DIR/default.yaml --instance-id=smurf-hammer --site-host={smurf-etc}

3. get a client instance and run the hammer command

$ ocs-client-cli shell smurf-hammer

Does that look correct?

[mhasself]

In anticipation of needing to bring up all the smurf crates at site sometime in the next week, I'd like to be ready to deploy and test this agent when that happens. Here's how I think that would work:

On a given smurf server:

1. edit `$OCS_CONFIG_DIR/default.yaml` and add

{'agent-class': 'SmurfHammerAgent',
     'instance-id': 'smurf-hammer',
     'manage': 'host',
     'arguments': []},

to the list of agents.

Should probably use unique instance id per server, even at this stage.

Btw, you don't have to edit the default.yaml in situ. In similar situations, I'll often duplicate the default.yaml to a scratch directory, set OCS_CONFIG_DIR to point to that scratch dir, and launch. So as not to mess with the stable configuration.

2. start the agent with the following command:

$ ocs-agent-cli --site-file=$OCS_CONFIG_DIR/default.yaml --instance-id=smurf-hammer --site-host={smurf-etc}

Only --instance-id arg would be needed. It should auto-detect the other things (assuming you've set OCS_CONFIG_DIR to something sensible for this user).

3. get a client instance and run the hammer command

$ ocs-client-cli shell smurf-hammer

Does that look correct?

Yes, should work. And note you could run this on any machine as long as it knows how to find a default.yaml that connects to the right crossbar server -- doesn't have to be on the smurf server in question. Could be daq-lat.

[tristpinsm]

Some notes from testing today:

I found some bugs in jackhammer, fixed in simonsobs/sodetlib#516

I had tried to set things up for the agent to work in a docker container, which required mounting the docker socket and adding the docker group so that jackhammer in the container can control containers on the host. This aspect seemed to be working, but I was running into a whack-a-mole of missing server configuration (missing random scripts in the path, ssh authentification to the shelf manager, ...) so I gave up on that for now. If we want it in a docker container, I think that would be doable, but need to spend more time identifying all the things it needs.

Running as a process on the host, it worked fine, except the monitor process could not query the state of slots because epics is not installed on the host (I had debugged this yesterday in a docker container and the monitor is working). Another nice thing about the docker option is it was easy to pipe logs to Loki. Presumably you can do this for host agents as well, but I didn't know how. Options then for deploying this are:

1. In a docker container, but need to finish sorting out all the configuration so that jackhammer has everything it needs.
2. On the host, but need to install epics and sort out Loki.

I also wanted to test out the ocs-web panel for the agent. I built a docker image from my ocs-web branch and picked just the LAT crossbar entry out of the ocs-deployment-configs repo to use as a config file. I was able to start the instance of ocs-web on smurf-so8-lat, but in the web browser it didn't display any agents and complained that it was not connected to the crossbar. I checked that I could reach the crossbar address from within the container, but didn't investigate any further.

[mhasself]

I also wanted to test out the ocs-web panel for the agent. I built a docker image from my ocs-web branch and picked just the LAT crossbar entry out of the ocs-deployment-configs repo to use as a config file. I was able to start the instance of ocs-web on smurf-so8-lat, but in the web browser it didn't display any agents and complained that it was not connected to the crossbar. I checked that I could reach the crossbar address from within the container, but didn't investigate any further.

The ocs-web docker doesn't need to contact crossbar -- it's actually just serving files (html/js/css) to your web browser. It is your web-browser that needs to be able to reach crossbar. So if that's running on the host system, then use the same addresses that you'd use for an agent on the host system.

I usually develop ocs-web on a local machine, and ssh port forward the one OCS port that is needed to get the connection to crossbar.

[tristpinsm]

OK, with Matthew's guidance I was able to start a test ocs-web instance and validate that the panel can display the monitoring state of the slots. I didn't trigger a hammer yet via ocs-web because it wasn't a good time.

I think the only remaining thing to sort out is whether to deploy this in a docker container or directly running on the host.

1. To run on the host I think we just need to install/configure EPICS on each smurf server.
2. To run in docker I would still need to do some work configuring things so that jackhammer has access to everything it needs (docker control, scripts installed to servers, ssh access to shelf manager, ...).

Option 2 is more work, but it may make it easier to maintain / deploy changes in the future. On the other hand, we probably want to continue supporting jackhammer on the host (unless we make it an alias that just drops you into a docker container).

Any thoughts on how to deploy the agent?

[msilvafe]

I vote we run it on host and install epics on all the smurvers.

[tristpinsm]

I had another opportunity to test now with the power briefly having gone down. This time with

• EPICS installed on the host (so8)
• an agent running as the cryo user on the host
• an instance of ocs-web with the jackhammer panel

I hammered using the ocs-web interface and it behaves as expected.

Still need to figure out how to deploy the agent / send logs to Loki, but I think this is ready for a final review.