podman container hardening

[0xlua]

Run Rootless Podman

• no privilege, no run as root PUID/GUID “0:0”, each have own network, no default bridge
• podman when web service, nix flake when not
• Reverse Proxy has to run as root to bind to Port 80/443
  • look into alternatives
• postgres seems to be a problem for some
  • needs host directory ownership sorted out

Podman Networks

• don't put everything on the same network
• Use Reverse Proxy for HTTPS
• expose only containers which are made for being exposed
• databases have own networks: internal: true (zero internet access, only specific app)
• Reverse proxy gets its own network
• Inter-service communication goes through a separate network

    # after: database isolated, no internet
    networks:
      default:
        name: myapp_db
        internal: true
      web_ingress:
        external: true

Related: use per-service database container. I already do this though. Just in case, two services should share a database: Dedicated database + role per service, connection limits per role, and then revoked CONNECT from PUBLIC on every database.

Run unprivileged & read_only Filesystem

• tbd?? sec_options unprivileged
• some need tmpfs paths, need mapping

Drop Capabilities

• run podman inspect (check if containers have all capabilities: NET_RAW, SYS_CHROOT, MKNOD, ...)
• add cap_drop: ALL / DROP_CAP to everything
• restart one at a time, most will come back fine with zero capabilities, exceptions:
  • PostgreSQL: entrypoint needs to chown data directories (CHOWN, SETUID, SETGID, ...)
  • {REVERSE PROXY} needs NET_BIND_SERVICE for 80/443
  • Everything else should run with nothing
• Add it, restart, read the error if it crashes, add back the minimum

Resource limits

• disable swap per container (memswap_limit = mem_limit): only container gets killed, not whole box
  • mem_limit: educated guess from stats or do real profiling?
• add PID limits: prevent fork bomb
• tiered CPU with cpu_shares:
  • Reverse proxy and databases get highest priority
  • App services get medium
  • Background workers get lowest

Health checks

• replace "is the process alive" with real HTTP check
• each runtime needs its own approach:
  • Node: no curl, use Node's http module inline
  • Python slim: no curl, use urllib
  • Postgres: pg_isready just works

Use pinned versions

This adds little to no security because version tags are still mutable. However it would make it safer/easier to use podman auto-update (see #138).

For maximum security I'd have to pin every image to SHA256 digests, Tradeoff: manual updates

Reverse Proxy

Currently Caddy, but need a Reverse Proxy, that's supports the PROXY Protocol, see #133.
Maybe nginx with lego or certbot for HTTPS certs.

• TLS 1.2 minimum, Restricted ciphers
• Catch-all that returns nothing for unknown hostnames (stops bots from enumerating subdomains)
• Blocked .env, .git, wp-admin, phpmyadmin at high priority so they never reach any backend
• Rate limiting on all public routers
• Headers, e.g. HSTS

Secrets

• I already use sops for secrets, when possible no plain-text env vars
• Prefer Files / Podman Secrets
• API Keys: Use scoped tokens where possible

Host, OpSec & other apps

• enable firewall and block all access from countries you are not from
• use OIDC: 2FA/Passkeys
• rustypaste: server.hardening = true, see: https://github.com/orhun/rustypaste/releases/tag/v0.17.0

Sources

• https://reddit.lua.one/r/selfhosted/comments/1sqi04i/beyond_the_basics_what_are_your_nonnegotiable/
• https://reddit.lua.one/r/selfhosted/comments/1sgvwep/after_my_last_post_blew_up_i_audited_my_docker/
• see also: https://mroczek.dev/articles/hardened-arr-stack/