Using minhasher type: Ordinary CpuMinhasher can use 0 maps Cannot construct a single cpu hashtable. Abort!

[YuanfengZhang]

resources/CARE/care-cpu \
  -i result/BS3_D6_1/trim-galore/no-pre-dedup/BS3_D6_1.R1.fq.gz -i result/BS3_D6_1/trim-galore/no-pre-dedup/BS3_D6_1.R2.fq.gz --pairmode PE \
  -d ${tmp_dir} -o BS3_D6_1.calibrated.fq \
  -c 30 \
  -m 48G -t 32 \
  -p

got:

Output directory: result/BS3_D6_1/trim-galore/no-pre-dedup/care
Estimated dataset coverage: 30
Input files: result/BS3_D6_1/trim-galore/no-pre-dedup/BS3_D6_1.R1.fq.gz result/BS3_D6_1/trim-galore/no-pre-dedup/BS3_D6_1.R2.fq.gz 
Output file names: BS3_D6_1.calibrated.fq 
Paired mode: PairedEnd
Number of hash tables / hash functions: 48
K-mer size for hashing: auto
Enforce number of hash tables: false
Threads: 32
Use quality scores: false
Bits per quality score: 8
Exclude ambigious reads: false
Alignment absolute required overlap: 30
Alignment relative required overlap: 0.3
Alignment max relative number of mismatches in overlap: 0.2
errorfactortuning: 0.06
coveragefactortuning: 0.6
Show progress bar: true
Output directory: result/BS3_D6_1/trim-galore/no-pre-dedup/care
Temporary directory: result/BS3_D6_1/trim-galore/no-pre-dedup/care
Save preprocessed reads to file: 
Load preprocessed reads from file: 
Save hash tables to file: 
Load hash tables from file: 
Maximum memory for hash tables: 50465865728
Maximum memory total: 51539607552
Hashtable load factor: 0.8
Fixed number of reads: 0
GZ compressed output: false
Correct candidate reads: false
Output correction quality labels: false
Max shift for candidate correction: 15
Correction type (anchor): 0 (Classic)
Correction type (cands): 0 (Classic)
ml-forestfile: 
ml-cands-forestfile: 
classification thresholds: 0.5 | 0.5
anchor sampling rate: 1
cands sampling rate: 0.01
pairedFilterThreshold: 0.06
maxForestTreesAnchor: 4294967295
maxForestTreesCands: 4294967295
ml-print-forestfile: anchorfeatures.txt
ml-cands-print-forestfile: candidatefeatures.txt
----------------------------------------
Running CARE CPU
STEP 1: Database construction
Processed 0 reads in file. Elapsed time: 0 seconds.
Converting paired reads of files result/BS3_D6_1/trim-galore/no-pre-dedup/BS3_D6_1.R1.fq.gz and result/BS3_D6_1/trim-galore/no-pre-dedup/BS3_D6_1.R2.fq.gz, storing them in memory
Processed 458752 reads in file. Elapsed time: 1.00057 seconds.
Processed 1048576 reads in file. Elapsed time: 2.00083 seconds.
Processed 2490368 reads in file. Elapsed time: 4.00113 seconds.
Processed 5439488 reads in file. Elapsed time: 8.0017 seconds.
Processed 11206656 reads in file. Elapsed time: 16.0028 seconds.
Processed 23199744 reads in file. Elapsed time: 32.0045 seconds.
Processed 51445760 reads in file. Elapsed time: 64.0099 seconds.
Processed 103809024 reads in file. Elapsed time: 128.017 seconds.
Processed 207618048 reads in file. Elapsed time: 256.03 seconds.
Processed 410124288 reads in file. Elapsed time: 512.07 seconds.
Processed 792026954 reads in file. Elapsed time: 1009.85 seconds.
# elapsed time (init readstorage after construction): 9.92929s
# elapsed time (build_readstorage): 1019.79s
Determined the following read properties:
----------------------------------------
Total number of reads: 792026954
Minimum sequence length: 20
Maximum sequence length: 151
----------------------------------------
Will use k-mer length = 20 for hashing.
Reads with ambiguous bases: 290054
reads memory usage: 30.2445 GB on host
maxMemoryForTables = 19064824396 bytes
# elapsed time (build_minhasher): 0.0294233s
Using minhasher type: Ordinary
CpuMinhasher can use 0 maps
Cannot construct a single cpu hashtable. Abort!

It's weird. Given the documentation:

-h, --hashmaps arg            The requested number of hash maps. Must be
                              greater than 0. The actual number of used hash
                              maps may be lower to respect the set memory
                              limit. Default: 48

Does this mean memory limit? But my log said there is no such thing:

[fkallen]

Yes, it is a memory limit. You specify 48 GB memory limit. 30 GB are used for the dataset which leaves only 18 GB for hash tables and hash table construction, which is not enough. If my calculations are correct, for your dataset a single hash table needs around 9 GB, and constructing needs also needs temporary storage.
For 792 Million reads I would suggest 250GB memory limit or more, as you need many hash tables for good results.

[YuanfengZhang]

Thx. How about GPU? Do I need to allocate it with ~250GB VRAM?

[fkallen]

By default, the GPU version puts hash tables in CPU memory. However, with option --useGpuTables hash tables will be placed in GPU memory. This requires large VRAM.