feat: WebDAV asset upload to public_dir

[bltavares]

This PR introduces an (experimental) WebDAV asset uploading to public_dir.
This make it much easier to keep the service running and upload just HTML, CSS
and Javascript assets using something like rclone or Cyberduck.

I've tried to use the WASM guest API to implement it, but WebDAV has multiple
additional HTTP Methods. Changing the WASM API to support all of them (or ANY
matcher) seemed more intrictate.

The WASM guest API also seems limited to change the Config of the service (or
I didn't fully understand the scope of KV functions). So instead I've
implemented it directly on the service core with conditional feature = webdav
to allow opt-out.

For now this is covered:

• WebDAV handling under '/_/dav' prefix
• Required Bearer $token authentication to upload assets
• Configuration of WebDAV upload token on Admin UI
• Documentation
• End to end tests

I'm not so familiar on how to write end-to-end tests in the project, and I'll
spend more time over the week trying to udnerstand it. Meanwhile, it can be tested locally using rclone:

mkdir -p dist
echo hello > dist/index.html
& trail run --public-dir dist

# Visit the UI and add a token under settings

mkdir -p demo
echo bye > demo/index.html

rclone sync --verbose --webdav-url http://localhost:4000 --webdav-bearer-token secret demo :webdav:/_/dav

Closes #255

PS: the large diff seems to be caused by a more recent protoc updating many .ts lines. I'll figure out if I can downgrade my version to reduce the noise Done

[bltavares]

Feel free to close it as not intended addition to the codebase. Integrating directly on the core crate felt the best for now, in comparison the the trailbase::api or wasm extension to the config UI and router handling.

I'd understanding not wanting to add such feature to core tho, but at least it can be an example if anyone has this niche idea in the future as well.

[ignatz]

Let me start with a bit of a disclaimer:

• I haven't really worked with WebDAV before
• I'm concurrently having conversations (e.g. Allow WASM modules to extend the TrailBase admin panel #252 (comment)) about ripping more things out of the core :)
• I'm concurrently having an inner battle on the deployment model: wit PG support and maybe distributed DB backends on the way, the question becomes how would a deployment look like if there were multiple TB instances running? Deploying multiple instances considering public_dir static may be very different from using WebDAV and having to sync 🤷‍♀️

Ok, with my confusion out of the way, what's the ultimate goal here? E.g. I'm wondering what's the benefit of rcloning assets over WebDAV rather than some other protocol? Just curios.

I've tried to use the WASM guest API to implement it, but WebDAV has multiple
additional HTTP Methods. Changing the WASM API to support all of them (or ANY
matcher) seemed more intrictate.

That could be cool. Happy to push changes to make components generally more useful. I would have zero concerns adding a WebDAV component if we can make it work. If you have a list of issues, I can try to pay them down.

The WASM guest API also seems limited to change the Config of the service (or I didn't fully understand the scope of KV functions).

There isn't really a way to write the config. Heck there isn't even a way to read the config. We should certainly add the latter (currently we use a hack for auth-ui). I'd be a bit more careful on the former. In principle we should allow components to have their own isolated configs somewhere. Would that be enough to read the webdav token from?

[bltavares]

Ok, with my confusion out of the way, what's the ultimate goal here? E.g. I'm wondering what's the benefit of rcloning assets over WebDAV rather than some other protocol? Just curios.

Yeah, I know this might be a niche scenario, and I'd also prefer to extend using WASM to not clobber the core design with this extension.

My idea was to have a Trailbase backend hosted on my container orchestrator accepting new front-end deployments, without requiring container restarts or new base image deployments. Something like old-school PHP shared host, where someone can drag frontend files using CyberDuck.

I know a more typical deployment scenario would have a JS Fronted repo building a base image baked with the compiled assets, trigger CI, upload the image, and trigger a new deploy version. But this project I'm working on with someone that might be able to edit the JS/HTML/CSS themselves, but might not be very versed with the Git commit/CI/wait-a-while/redeploy flow.

So the idea of the workflow is:

• Have a local JS development repo with a local Trailbase server
• Edit and iterate locally
• Drag and Drop the /dist folder in the production Trailbase server (or run a sync.sh script)

As this project uses Sqlite, it's not very happy to run on a NFS mount. So I'm using a block storage with exclusive access to the "pod" deployment. There are several other ways I could achieve this flow tho:

• Run a typical Git commit frontend repo, trigger CI and publish a new image (but then I actually have to start CI workers which I don't have setup now haha)
• Run tb_depot on a block storage mount, and separate the /dist into a NFS mount (but then I have to open NFS to others and it's only available in LAN/VPN)
• Run with nginx or caddy proxy, with WebDAV extension writing to /dist and proxying to the trail server (but then I have to build a base image with the server extension, and it lacks admin UI to change tokens)
• Run 2 separate deployments, Trailbase as API only without --public-dir and Frontend separately (but then CORS tends to be annoying, and I still have to build a base image for the server)
• Build trailbase using the trailbase::api custom-binary method (but then I have to replicate the CLI parsing almost 1:1)
• Run a file server "sidecar" service (eg: dufs) that handles webdav/drag-n-drop to the --public-dir (viable, and my initial plan, but 2 processes needed)

None of the options are terrible by themselves, or not viable to be done, but I thought it would be nice to have a single process that serves public assets and accept updates. It would look a lot similar to an old-school XAMPP server and more similar to anyone used to customize their MySpace profile.

That could be cool. Happy to push changes to make components generally more useful. I would have zero concerns adding a WebDAV component if we can make it work. If you have a list of issues, I can try to pay them down.

I much rather extend it with WASM, as I mentioned before, and I've started tackling it this way also, but then it started to become a bigger changeset, which requires some additional alignment on how to approach:

• The WIT specifies a limited set of HTTP Methods, so it would need an extension (either a special ANY, or accept strings)
• The WASM handlers are keyed by <http::Method, Path, Handler>, so adding the ANY or extended methods become "hacks" with special values
• It lacks a way to extend the Admin UI (as discussed on Allow WASM modules to extend the TrailBase admin panel #252) to manage configs
• The WASM Host limits the filesystem access to Read-Only (if I understood it correctly)

I think this extension would stress some of the current design strategies of the WASM extension, but I'm not sure if it's good stress or bad stress hahaha

In principle we should allow components to have their own isolated configs somewhere. Would that be enough to read the webdav token from?

Yup, it would work fine as isolated config, as long as there is a method to integrate with the Admin UI. I'd rather have a way to expose the configs to the UI editor, otherwise I guess each WASM extension would also need to bundle their own frontend bundling and it can get inconsistent

It would also be fine to have the filesystem read/write isolated to the WASM process, as long as the main process can also read the --public-dir from the WASM plugin target folder.

[ignatz]

Thanks a ton for the thoughtful reply, sharing the overall intent and the issues you ran into with the WASM approach. And sorry for the latent reply.

You got me at CyberDuck, which I haven't used for 20 years but is absolutely a blast from the past :).

Nostalgia aside, my biggest (and arguably wishful) concern is complicating a distributed deploy going forward. I'll also only play devils advocate once: if you have a partner who's not comfortable with git, maybe having a CI/CD-verification step is an especially good idea, since uploading assets straight to prod does introduce an additional failure mode. That said, ultimately you need to do what works for you :yolo:.

As for your other options, you may already run an nginx/caddy for TLS termination, so letting it serve your assets probably wouldn't add much complexity 🤷‍♀️

Now, back to your original proposal. Personally, I'd still favor a WASM component approach. Looking at your list of per-requisites:

The WIT specifies a limited set of HTTP Methods, so it would need an extension (either a special ANY, or accept strings)

We should allow wildcard matches anyway (if we don't already do).

The WASM handlers are keyed by <http::Method, Path, Handler>, so adding the ANY or extended methods become "hacks" with special values

There might be some nuance, naively this looks like the previous point. I would expect the Path to be able to contain wildcards.

It lacks a way to extend the Admin UI (as discussed on #252) to manage configs

Would this even be needed, feels more like a nice to have? Naively you'd configure your access once and never change again? That said, we're discussing the ability to add admin sub-pages via WASM components on a parallel thread: feel free to chime in (#252).

The WASM Host limits the filesystem access to Read-Only (if I understood it correctly)

The WASM Host limits the filesystem access to Read-Only (if I understood it correctly)

Good point. I'd be very happy to extend it to include write access. Read or write, longer term we should have system to grant capabilities to WASM components (similar to mobile apps), since both are priviledged operations which may be sensitive for 3rd party components. At the moment we're really only concerned with 1st and 2nd party components, where both read and write are less of a concern.

I think this extension would stress some of the current design strategies of the WASM extension, but I'm not sure if it's good stress or bad stress hahaha

It's much appreciated 🙏 - this is exactly what is needed to make the component system more useful.

[bltavares]

Thank you for considering this approach, anyway. I understand the concerns of extending the core crate. I'm glad we had the chance to discuss it anyway. I'll close this PR and explore further what is needed from the WASM extension method instead, if that is ok.

---

once: if you have a partner who's not comfortable with git, maybe having a CI/CD-verification step is an especially good idea, since uploading assets straight to prod does introduce an additional failure mode. That said, ultimately you need to do what works for you :yolo:.

Yeah, from a reliability perspective a CI pipeline would be better. I was leaning towards autonomy of editing on this project tho, closer to what MySpace used to offer on this project than correctness and safety. I want to foster a WYSIWYG dev flow instead of production reliability industry I'd apply on other scenarios 😄

As for your other options, you may already run an nginx/caddy for TLS termination, so letting it serve your assets probably wouldn't add much complexity 🤷‍♀️

Yup, I could design it different ways. But the all-in-one single-binary looked as the nicest option IMO

Would this even be needed, feels more like a nice to have? Naively you'd configure your access once and never change again?

Yeah, setting it up once on deployment would be fine also. I've looked over the settings method as a more general integration imagining others could want to enable it temporarily without restarting the service.

It's a bit unclear to me how the WASM plugin can read config options at this moment also, even if it's env environments/cli args.

Good point. I'd be very happy to extend it to include write access. Read or write, longer term we should have system to grant capabilities to WASM components (similar to mobile apps),

Maybe in the manifest of the WASM (being discussed on the other issue) it could declare permissions, and these are reviewed on trail component add where the user has to accept the requirements or not? Just an idea, I didn't spend too much time thinking on it.

[ignatz]

Don't get me wrong, I wasn't trying to tell you how to set up your environment.

It's a bit unclear to me how the WASM plugin can read config options at this moment also, even if it's env environments/cli args.

This conversation has carried into #252. Independent of FS access, settings could be handled separately. For example, if you look at mobile development you can read/write settings through a persistent KV store (e.g. shared preferences) w/o FS access. FS access whether it's read or write is strictly more priviledged. I think it would make sense for me as one of the next steps to add a per-component persistent KV store - appreciated

Maybe in the manifest of the WASM (being discussed on the other issue) it could declare permissions, and these are reviewed on trail component add where the user has to accept the requirements or not? Just an idea, I didn't spend too much time thinking on it.

Yeah, that's pretty much what would be needed (longer term) - appreciated

[bltavares]

It's all good 😄

I fully understand the maintainer-side concerns on adopting such deployment model. I'm not taking it personally.

I'll watch over the discussion on the other issue, and see what I can get along the way on the WASM refactoring. Is it ok to open draft PRs for WASM changes as discussion points, even if they don't land?

[ignatz]

Is it ok to open draft PRs for WASM changes as discussion points, even if they don't land?

Absolutely. That said, it may save some time and/or tokens if we first have a quick chat on more major things. Also always happy to help, get my hands dirty and do upfront work.