Skip to content

ebookmaker should run jobs in multiple threads for better concurrency #330

Description

@cpeel

ebookmaker accepts a "job queue" for PG-specific usage, but it runs the jobs in series not in parallel. This only makes use of a single CPU as observed by the hours ebookconverter takes to run but only one CPU is ever maxed out at a time.

Image

Ideally ebookmaker would accept a --concurrency value and use a ThreadPool of that size to run multiple jobs in parallel. Care must be taken to ensure that ebookmaker resources are either threadsafe or that each thread owns its own copy of the resource. The latter is easier but may require restructuring the code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions