Hi team,
When attempting to download the MIDAS v3 db (UHGG v2 catalog) for a metagenomic phasing pipeline, the midas CLI and manual https/aws downloads have not been working from a few days ago. Here is a summary:
- midas database --init hangs with no output and leaves target directories at 0 bytes.
- Direct AWS S3 calls return 403 access denied for bucket listing and file requests.
- Gladstone http returns 404 not found.
For background, I've been processing metagenomes through the core MIDAS modules (run_species, run_genes, and run_snps). The first 21 samples were processed without issues. Processing failed for sample 22 during SNP calling from unpopulated species directories in the database reference path. I tried to skip this taxon thinking that it's not a prevalent one given that the first 21 samples were complete. But there were 2 additional taxa with the same issue so I tried to re-initialize and download the db.
Specifically, when running midas v3 (run_snps), the pipeline attempts to fetch missing rep genomes and annotation via wget/tar -xz. This step fails when downloading from
https://midasdb.pollard.gladstone.org/v3/uhgg/,
and tar crashes with
gzip: stdin: unexpected end of file and throwing subprocess.CalledProcessError: Command returned non-zero exit status 2.
Subsequent attempts to reinitialize/redownload via midas database --init produces 0 bytes at the target directory. Have the paths/permissions under ttps://midasdb.pollard.gladstone.org/v3/uhgg/ changed recently, and is there an updated URL that I can configure during file fetching? I appreciate your input.
Below is an example log snippet:
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
1786399348.4: Deleting untrustworthy outputs due to error. Specify --debug flag to keep.
Traceback (most recent call last):
File "/users/jlim/miniconda3/envs/midasv3/bin/midas", line 10, in
sys.exit(main())
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/main.py", line 25, in main
return subcommand_main(subcommand_args)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/subcommands/run_snps.py", line 888, in main
run_snps(args)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/subcommands/run_snps.py", line 882, in run_snps
raise error
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/subcommands/run_snps.py", line 827, in run_snps
midas_db.fetch_files("repgenome", species_ids_of_interest)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/models/midasdb.py", line 190, in fetch_files
return self.fetch_tarball(filename, list_of_species)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/models/midasdb.py", line 204, in fetch_tarball
_fetched_files = multithreading_map(_fetch_file_from_s3, args_list, self.num_cores)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 540, in multithreading_map
return _multi_map(func, items, num_threads, ThreadPool)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 520, in _multi_map
return p.map(func, items, chunksize=1)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/models/midasdb.py", line 354, in _fetch_file_from_s3
return download_tarball(s3_path, local_dir)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 468, in wrapped_operation
return operation(*args, **kwargs)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 675, in download_tarball
command(f"set -o pipefail; wget -q -O- {ref_path} | {uncompress_cmd} {local_dir}")
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 246, in command
return subprocess.run(cmd, shell=shell, **subproc_args)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'set -o pipefail; wget -q -O- https://midasdb.pollard.gladstone.org/v3/uhgg/gene_annotations/104426/GUT_GENOME274158.tar.gz | tar -xz -C /proj/xxxx/xxxx/old_batch/data/metagenomics/my_midas3db_uhgg/gene_annotations/104426' returned non-zero exit status 2.
Best,
Joe
Hi team,
When attempting to download the MIDAS v3 db (UHGG v2 catalog) for a metagenomic phasing pipeline, the midas CLI and manual https/aws downloads have not been working from a few days ago. Here is a summary:
For background, I've been processing metagenomes through the core MIDAS modules (run_species, run_genes, and run_snps). The first 21 samples were processed without issues. Processing failed for sample 22 during SNP calling from unpopulated species directories in the database reference path. I tried to skip this taxon thinking that it's not a prevalent one given that the first 21 samples were complete. But there were 2 additional taxa with the same issue so I tried to re-initialize and download the db.
Specifically, when running midas v3 (run_snps), the pipeline attempts to fetch missing rep genomes and annotation via wget/tar -xz. This step fails when downloading from
https://midasdb.pollard.gladstone.org/v3/uhgg/,
and tar crashes with
gzip: stdin: unexpected end of file and throwing subprocess.CalledProcessError: Command returned non-zero exit status 2.
Subsequent attempts to reinitialize/redownload via midas database --init produces 0 bytes at the target directory. Have the paths/permissions under ttps://midasdb.pollard.gladstone.org/v3/uhgg/ changed recently, and is there an updated URL that I can configure during file fetching? I appreciate your input.
Below is an example log snippet:
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
1786399348.4: Deleting untrustworthy outputs due to error. Specify --debug flag to keep.
Traceback (most recent call last):
File "/users/jlim/miniconda3/envs/midasv3/bin/midas", line 10, in
sys.exit(main())
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/main.py", line 25, in main
return subcommand_main(subcommand_args)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/subcommands/run_snps.py", line 888, in main
run_snps(args)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/subcommands/run_snps.py", line 882, in run_snps
raise error
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/subcommands/run_snps.py", line 827, in run_snps
midas_db.fetch_files("repgenome", species_ids_of_interest)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/models/midasdb.py", line 190, in fetch_files
return self.fetch_tarball(filename, list_of_species)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/models/midasdb.py", line 204, in fetch_tarball
_fetched_files = multithreading_map(_fetch_file_from_s3, args_list, self.num_cores)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 540, in multithreading_map
return _multi_map(func, items, num_threads, ThreadPool)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 520, in _multi_map
return p.map(func, items, chunksize=1)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/models/midasdb.py", line 354, in _fetch_file_from_s3
return download_tarball(s3_path, local_dir)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 468, in wrapped_operation
return operation(*args, **kwargs)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 675, in download_tarball
command(f"set -o pipefail; wget -q -O- {ref_path} | {uncompress_cmd} {local_dir}")
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/site-packages/midas/common/utils.py", line 246, in command
return subprocess.run(cmd, shell=shell, **subproc_args)
File "/users/jlim/miniconda3/envs/midasv3/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'set -o pipefail; wget -q -O- https://midasdb.pollard.gladstone.org/v3/uhgg/gene_annotations/104426/GUT_GENOME274158.tar.gz | tar -xz -C /proj/xxxx/xxxx/old_batch/data/metagenomics/my_midas3db_uhgg/gene_annotations/104426' returned non-zero exit status 2.
Best,
Joe