Refactor to use workers for all file access

[krokicki]

This is an experimental branch which refactors Fileglancer to use user-specific worker processes for all setuid-based access, including all file operations and cluster tasks. This abandons the usage of seteuid in the Uvicorn workers, which has been problematic. Any async/await code can break that model, and it's easy to introduce hard-to-debug issues by accident. This new model is more similar to JupyterHub (as we had in the beginning). However, the workers are managed automatically and don't require users to manually start a server.

I've gone through code reviews with both Claude and Codex (they caught different issues). I've also tested it with a personal dev deployment on fileglancer-dev, but the code is quite complex and low-level and needs more testing and validation.

Claude PR Summary

Refactor all user-scoped file access to run on per-user persistent worker subprocesses. The main Uvicorn process never calls seteuid/setegid/setgroups — instead, each user gets a long-lived worker subprocess that runs with their real UID/GID/groups, communicating over a Unix socketpair with length-prefixed JSON. File descriptors for opened files are passed back to the main process via SCM_RIGHTS so streaming responses work without copying data through the worker.

• Worker pool (worker_pool.py): Manages per-user workers with LRU eviction, idle timeout, configurable pool size, and graceful shutdown. Workers are spawned on demand and cleaned up automatically.
• Worker subprocess (user_worker.py): Handles all user-scoped operations (file I/O, cluster jobs, git ops, SSH keys, S3 proxy) in a synchronous request/response loop. Action handlers are registered in a dispatch table.
• Server integration (server.py): All endpoints that previously used EffectiveUserContext now dispatch through _worker_exec(). In dev/test mode (no use_access_flags), actions run directly in-process with no subprocess overhead.
• Streaming (filestore.py): New _stream_contents and _stream_range static methods that stream from an already-opened file handle, used by the server to stream file content from worker-passed file descriptors.
• Tests (test_worker.py): IPC protocol tests, UserWorker integration tests, async execute tests (including concurrency serialization), action handler tests, and worker main loop tests.

Notable fixes included in this branch

• Enforce max_workers cap: raise 503 when pool is full instead of unbounded subprocess growth
• Use os.getgrouplist() for supplementary groups (fixes missing groups on LDAP/NSS systems)
• File descriptor leak protection on error paths and for unexpected extra fds from SCM_RIGHTS
• Socket timeout (120s) to prevent indefinite thread hangs if a worker is stuck
• Proper error handling and status codes in all action handlers
• Dev-mode parity: action handler exceptions are caught and logged consistently

@JaneliaSciComp/fileglancer @StephanPreibisch

[neomorphic]

I am concerned about memory usage with multiple uvicorn workers in production.

Each uvicorn worker process creates its own independent WorkerPool. Since uvicorn uses multiprocessing.get_context("spawn"), every worker process re-imports the module and creates a fresh pool — there is no shared state between them.

This means the number of user worker subprocesses scales as active_users × uvicorn_processes, not just active_users. Claude estimated a user worker consumes ~69 MB RSS (~50 MB PSS) on my dev linux server. With the production config (--workers 10), I tested two users browsing casually and produced 6 user worker processes across 4 uvicorn workers within a few minutes.

At scale with 50 active users and 10 uvicorn workers, the theoretical maximum is 500 user worker processes consuming ~25 GB. This is mitigated by the 300-second idle timeout and connection-level load balancing, but it will still be significantly more than the 50 workers the max_workers setting suggests in the configuration.

This should either be documented, or the architecture should be adjusted — for example, by running the worker pool as a separate broker process shared across all uvicorn workers, or by reducing max_workers to (max_workers / uvicorn_workers) automatically.

[krokicki]

@neomorphic Great points, this is something I started to think about but probably needs more experimentation and discussion. A separate shared broker process would solve some issues while introducing others (single point of failure, additional multiprocessing code, etc.) so there are trade-offs.

[krokicki]

Notes:

• Optimize workers so that they import as little as possible

Test plan:

• Test all Fileglancer features
• Kill worker and see what happens
• Deploy on fileglancer-dev and test simultaneous multi-user