A Torrent ZarrStore

[alxmrs]

Coming back from Cloud Native Geo, a major theme was how to build resilient data infrastructure (or ecosystem). A common idea that came up in many discussion groups was could we somehow store scientific data via bit-torrent. I don't think this idea seems too far fetched; we simply need to figure out how the Zarr protocol p2p torrent protocols align.

Projects like https://academictorrents.com/ (https://github.com/academictorrents) seem to validate that a peer-to-peer, public, global store of data is highly desirable – especially in a time when critical medical and environmental data is potentially being lost.

This repo in their project, in particular, seems very xpublish shaped! https://github.com/academictorrents/torrentify. A deeper look into the intersection of these protocols is warranted right now (maybe, even urgently).

Update: @samapriya is definitely one of the people at the conference who first brought up this idea!

[alexgleith]

Is IPFS a plausible alternative?

https://ipfs.tech/

[alxmrs]

I do think that block constraints make IPFS a deal-breaker for Zarr performance, however, IPLD is a really interesting protocol and I think an integration with it for a torrent-based Zarr store would be really cool to see.

Thanks for sharing the dclimate project; it's exciting and validating to see such an effort!

[Faolain]

Thanks so much for sharing and it's exciting to see that others see value in this effort! On the performance note, what are the speeds you're looking at achieving? We've seen without any optimizations 10-20mb/s when making requests to remote gateways and 260mb/s when running on local ipfs nodes. We think there are many areas for improvement around here such as divide and conquer algorithms with RAPIDE

[alxmrs]

For climate, weather, and geospatial datasets, read speeds in the ballpark of 3 GBPS are what we're shooting for: https://earthmover.io/blog/icechunk#:~:text=new%20async%20API.-,performance,-At%20this%20stage

I don't know the network characteristics of torrents, but I imagine large block sizes would help achieve this goal.

[Faolain]

Ah fair enough, this is a good and I think achievable goal. Appreciate you sharing that link and hope to have something to share in the not too distant future. For reference IPFS team members have reported in the low GB/s range with the previous numbers reported above using the IPFS public network on a home network.

Edit: With a private cluster and/or an enterprise server pipe we expect to see speeds exceeding that of S3. tl;dr your download bandwidth can be completely saturated given enough peers.

[Faolain]

I wasn't too familiar with Bittorrent w.r.t its implementation but it seems like the "block" size is essentially equivalent for torrents

With that said I'm not sure what the advantage would be for torrents over IPFS (given the latter already exists), or rather the drawback to IPFS necessitating a torrent store if blocksize is a concern. Open to any suggestions or ideas here though.

[alxmrs]

The file structure of .torrent seems very Zarr-like to me: https://en.wikipedia.org/wiki/Torrent_file#File_Structure

[alxmrs]

I think aira2 may be a useful component to managing IO of torrent-based Zarr stores. It already has BitTorrent support from the docs: https://aria2.github.io/manual/en/html/aria2c.html#bittorrent-download

[samapriya]

We literally had this discussion on academic torrents in our group so exciting to see this form

[rsignell]

FWIW, a bunch of data has been backed up on sciop.net (example) and I've been using aria2 to grab certain datasets and copy them to object storage to make them more accessible, like the NCEI Estuarine Bathymetry: https://nbviewer.org/gist/rsignell/7339b3a4aa8d39eff4cd766e127aa77e

[alxmrs]

Really interesting Rich! The sciop.net catalog is really validating. Further, from reading your notebook, I wonder if what we need is not a new Zarr store by a torrent-based filesystem (say, in fsspec).

[fmigneault]

Would this torrent approach work hand-in-hand with Zarr stores that get extended (i.e.: Modifying existing Zarr stores)? Would that break the integrity of the torrent shared between peers?

[alxmrs]

Sorry I missed this. Extending Zarr stores with torrents for p2p distribution is an interesting idea. Do I understand you correctly? Can you say more here?

[fmigneault]

No problem. I am trying to understand if using Torrent ZarrStore would imply that the data represented this way would be inherently read-only / constant, given that modifying it (e.g.: incrementally adding new variables at later time) would change the .torrent metadata and pieces it contains, therefore making it a "different torrent"?

[alxmrs]

I see what you mean. I am hoping for the Zarr stores to be writeable, but optimized for read access. It's possible that the torrent format is not amenable to writes/updates. If so, thanks for calling this out early.

That said, I bet there could be some interesting tricks for constancy and mutability under distributed replication.

[alxmrs]

I just got a tip from my boss: Torrent Zarr stores may have performance advantages related to higher available bandwidth. See the "transfer speed" section of https://web.archive.org/web/20090613010209/https://daniel.haxx.se/docs/bittorrent-vs-http.html