Currently, the entire database lives in memory and is persisted to a single file on disk. Consider making each registered id be persisted to a separate file. By loading/unloading the file (containing the data from a single id) on user request(s) we will likely make the memory footprint a lot smaller.
Pros:
- Smaller, more manageable files.
- Data from each
id is isolated making them easier to move around and share
- Data is loaded/saved based on ID, reducing how
- Its harder to corrupt the entire data set just because of one unclean shutdown.
Cons:
- Although we already persist to disk with every request, PyXIE will be doing a lot more reads from disk, making things slower
- The
DATABASE_FILE configuration would change to something like DATA_PATH which some may consider a breaking change, so a major version bump would be recommended.
- A data migration process from the single file database to multiple a multi-file database would need to be defined.
This ticket may lend itself well to #25 where each thread that PyXIE runs would receive a request from a queue, then loads the correct file for the id, and processes the request. Unclear how well that strategy would work with Flask, however.
Currently, the entire database lives in memory and is persisted to a single file on disk. Consider making each registered
idbe persisted to a separate file. By loading/unloading the file (containing the data from a singleid) on user request(s) we will likely make the memory footprint a lot smaller.Pros:
idis isolated making them easier to move around and shareCons:
DATABASE_FILEconfiguration would change to something likeDATA_PATHwhich some may consider a breaking change, so a major version bump would be recommended.This ticket may lend itself well to #25 where each thread that PyXIE runs would receive a request from a queue, then loads the correct file for the
id, and processes the request. Unclear how well that strategy would work with Flask, however.