Fast Lock On - FLO

FLO is a method for videography using a camera system that automatically moves to follow the subject.

FLO is described in the following publication:

Vo-Doan TT, Titov VV, Harrap MJM, Lochner S, Straw AD. High Resolution Outdoor Videography of Insects Using Fast Lock-On Tracking. Science Robotics 9(95), eadm7689 (2024) Free-access link DOI: 10.1126/scirobotics.adm7689.

What is in this repository

This repository contains the source code for the primary FLO executable, called flo. flo is written in the Rust language. flo receives image coordinates from Strand Cam imops and controls motors. This is the main, high-level FLO program.

While FLO was originally designed as a generic camera tracking system for all FLO variants, it has over time accumulated substantial specialized code for drone-based operation. That configuration adds MAVLink flight-controller integration and pilot video display via OSD. Non-drone use cases — such as Mini-FLO with PWM servo motors, Trinamic stepper motors, or SimpleBGC gimbal motors — remain fully supported.

Other FLO-related repositories

• https://github.com/strawlab/flo-data-analysis data analysis for FLO data
• https://github.com/strawlab/flo-hardware build instructions and manual for the FLO BYO-camera insect tracking/filming setup.
• https://github.com/strawlab/red-button-trigger-timestamp hardware to record times of external trigger events.

Overview of multiple pieces when using PWM servo motors

This diagram shows the various software pieces, and the means with which they communicate with each other, when using the Mini-FLO variant. In Mini-FLO, the rpipico-pantilt software running on a Raspberry Pi Pico board controls PWM servo motors. Other variants trade rpipico-pantilt and the PWM servo motors for other types of motors, such as Trinamic stepper motors or SimpleBGC gimbal motors. Strand Camera is available here.

+-------------+
|             |         HTTP
|   FLO UI    +<----------------------+
|    (browser)|                       |
+-------------+                       |
                                      v
+-----------------------+      +-------------+     +-----------------+
|                       | UDP  |             | USB |                 |
| Strand Camera `imops` +----->|    flo      +<--->| rpipico-pantilt |
|                   (PC)|      |         (PC)|     |        (RP Pico)|
+-----------------------+      +-------------+     +-------+---------+
            ^                                              |
            |                                              |
            |                                              |
            |                                              v
       +----+---+                                      +--------+
       |        |                                      | PWM    |
       | camera |<-------------------------------------+ servo  |
       |        |                                      | motors |
       +--------+                                      +--------+

(Ascii art made with http://asciiflow.com/ )

Overview of top-level directories

• crates Rust crates (libraries) used by flo and other software
• firmware/beamdriver Firmware for IR LED source
• firmware/flo-tilta-dongle Firmware for USB dongle to control Tilta motors.
• firmware/rpipico-pantilt Firmware for RPi Pico to control PWM servo motors.
• src Rust source code for flo

How to connect to Strand Camera

1. Run strand-cam.

Specify the networking port using command-line argument: --http-server-addr 127.0.0.1:5555 (5555 is the port number). Specify the camera to use with --camera-name Basler-40522040 (40522040 is the serial number of the camera). Thus, your command line would be:

strand-cam-pylon --http-server-addr 127.0.0.1:5555 --camera-name Basler-40522040

To obtain distance estimates using stereo cameras, it is necessary to run two instances of strand-cam, one for the main camera and one for the secondary camera. These will have different ports. For example, 5555 for the main camera, and 5556 for the secondary camera.

2. Obtain the link for strand-cam.

Strand-cam will dump a lot of info into the terminal. Look for a line like * predicted URL http://127.0.0.1:5555/.

When running with two cameras, each will have a different link. Keep the strand-cam instances running for the next step.

3. Put the link into the FLO config file, and run FLO once.

Look for url fields in the FLO config file:

  strand_cam_main:
    url: http://127.0.0.1:5555
    on_attach_json_commands:
    - '{"ToCamera":{"SetImOpsCenterX":960}}'
    - '{"ToCamera":{"SetImOpsCenterY":600}}'
    - '{"ToCamera":{"SetImOpsThreshold":200}}'
    - '{"ToCamera":{"ToggleImOpsDetection":true}}'
    - '{"ToCamera":{"SetMp4MaxFramerate": "Fps60"}}'
  strand_cam_secondary: # If you have a secondary camera for stereopsis, also use this section
    url: http://127.0.0.1:5556
    on_attach_json_commands:
    - '{"ToCamera":{"SetImOpsCenterX":960}}'
    - '{"ToCamera":{"SetImOpsCenterY":600}}'
    - '{"ToCamera":{"SetImOpsThreshold":200}}'
    - '{"ToCamera":{"ToggleImOpsDetection":true}}'

Then run FLO with something like this:

flo --config config-mini.yaml --pwm-serial /dev/ttyACM0

Note that in the FLO config file you can configure imops and other aspects of strand-cam automatically in the on_attach_json_commands section.

camshow binary

camshow is a separate desktop process that shows a USB webcam feed and draws an OSD overlay received from flo over TCP. Keeping it as a separate process means camera preview can keep running even while flo is restarted.

Build and run

Build:

cargo build --release -p camshow

Run with defaults:

cargo run --release -p camshow

Useful options:

• --listen <ADDR> TCP listen address for flo connections. Default: 127.0.0.1:2224.
• --windowed Run in a window instead of fullscreen.
• --fpv-cam <HUMAN_NAME> Prefer a specific webcam name. If not set, the first available camera is used.
• --test-pattern Show a fixed OSD test pattern whenever flo is not currently sending OSD updates.
• --log-dir <DIR> Directory for log files (defaults to the home directory).

Example:

cargo run --release -p camshow -- --windowed --listen 127.0.0.1:2224

Connect flo to camshow

Configure camshow from FLO config:

osd_config:
  camshow_addr: 127.0.0.1:2224
  # Optional: override webcam MP4 encoder settings sent to camshow.
  # camshow_mp4_cfg: ...

Or override from CLI when running flo:

flo --config config-mini.yaml --camshow 127.0.0.1:2224 --pwm-serial /dev/ttyACM0

When recording is enabled in flo, recording start/stop is forwarded to camshow. camshow writes webcam MP4 output (and a matching OSD SRT sidecar) into the recording directory selected by flo.