Support larger alphabets and k via generic kmer_t

Hi @jermp ! This is a continuation of my work that I mentioned in #35.
With this change, I rewrite current implementation of kmer_t in a full-fledged structure that handles bit manipulations over kmers. I think this should help greatly to compartmentalize current code that deals with alphabet characters <-> bits conversion, and also provides better semantics to some existing pieces of code, potentially improving their readability, and also allowing to use it with other alphabets.
I tried my best to avoid performance sacrifices, while making it (hopefully) generic enough for the needs of Metagraph. As such, new kmer type would still assume that the underlying storage for kmers is an unsigned integer, with the number of bits in it divisible by 64, but it should now support arbitrarily large integer types (so, 128-bit integers or some custom types with e.g. 256 bits should also work). To accommodate for larger alphabets, I still stick to the assumption that each alphabet character uses a fixed amount of bits, but now the specific number of bits is a template parameter, rather than the hardcoded number 2. I tried to highlight it in all the previous places where it was used as a hard-coded value.
I also compared two implementations using the first example command from the readme file:
./sshash build -i ../data/unitigs_stitched/salmonella_enterica_k31_ust.fa.gz -k 31 -m 13 --bench -o salmonella_enterica.index

You may see the resulting stats below:

 before Pull Request   After Pull Request (with uint64_t)

k = 31, m = 13, seed = 1, l = 6, c = 3, canonical_parsing = false, weighted = false
reading file '../data/unitigs_stitched/salmonella_enterica_k31_ust.fa.gz'...
m_buffer_size 29411764
sorting buffer...
saving to file './sshash.tmp.run_1705435490373836938.minimizers.0.bin'...
read 647 sequences, 4806944 bases, 4787534 kmers
num_kmers 4787534
num_super_kmers 479171
num_pieces 648 (+0.00812109 [bits/kmer])
=== step 1: 'parse_file' 0.538684 [sec] (112.518 [ns/kmer])
=== step 2: 'build_minimizers' 0.221001 [sec] (46.1618 [ns/kmer])
bits_per_offset = ceil(log2(4806975)) = 23
m_buffer_size 20833333
sorting buffer...
saving to file './sshash.tmp.run_1705435491133617083.bucket_pairs.0.bin'...
num_singletons 419549/447082 (93.8416%)
=== step 3: 'build_index' 0.099825 [sec] (20.851 [ns/kmer])
max_num_super_kmers_in_bucket 51
log2_max_num_super_kmers_in_bucket 6
num_buckets_in_skew_index 0/447082(0%)
=== step 4: 'build_skew_index' 0.003277 [sec] (0.684486 [ns/kmer])
=== total_time 0.862787 [sec] (180.215 [ns/kmer])
total index size: 2.85224 [MB]
SPACE BREAKDOWN:
  minimizers: 0.300206 [bits/kmer]
  pieces: 0.00246641 [bits/kmer]
  num_super_kmers_before_bucket: 0.15285 [bits/kmer]
  offsets: 2.30209 [bits/kmer]
  strings: 2.00815 [bits/kmer]
  skew_index: 1.33681e-05 [bits/kmer]
  weights: 0.000307465 [bits/kmer]
    weight_interval_values: 5.34722e-05 [bits/kmer]
    weight_interval_lengths: 0.000200521 [bits/kmer]
    weight_dictionary: 5.34722e-05 [bits/kmer]
  --------------
  total: 4.76611 [bits/kmer]
avg_nanosec_per_positive_lookup 710.753
avg_nanosec_per_negative_lookup 927.569
avg_nanosec_per_positive_lookup_advanced 731.852
avg_nanosec_per_negative_lookup_advanced 952.384
avg_nanosec_per_access 384.55
iterator: avg_nanosec_per_kmer 30.6924
2024-01-16 21:05:11: saving data structure to disk...
2024-01-16 21:05:11: DONE

k = 31, m = 13, seed = 1, l = 6, c = 3, canonical_parsing = false, weighted = false
reading file '../data/unitigs_stitched/salmonella_enterica_k31_ust.fa.gz'...
m_buffer_size 29411764
sorting buffer...
saving to file './sshash.tmp.run_1705435769505641298.minimizers.0.bin'...
read 647 sequences, 4806944 bases, 4787534 kmers
num_kmers 4787534
num_super_kmers 479171
num_pieces 648 (+0.00812109 [bits/kmer])
=== step 1: 'parse_file' 0.50195 [sec] (104.845 [ns/kmer])
=== step 2: 'build_minimizers' 0.204642 [sec] (42.7448 [ns/kmer])
bits_per_offset = ceil(log2(4806975)) = 23
m_buffer_size 20833333
sorting buffer...
saving to file './sshash.tmp.run_1705435770212350381.bucket_pairs.0.bin'...
num_singletons 419549/447082 (93.8416%)
=== step 3: 'build_index' 0.094723 [sec] (19.7853 [ns/kmer])
max_num_super_kmers_in_bucket 51
log2_max_num_super_kmers_in_bucket 6
num_buckets_in_skew_index 0/447082(0%)
=== step 4: 'build_skew_index' 0.002644 [sec] (0.552268 [ns/kmer])
=== total_time 0.803959 [sec] (167.928 [ns/kmer])
total index size: 2.85224 [MB]
SPACE BREAKDOWN:
  minimizers: 0.300206 [bits/kmer]
  pieces: 0.00246641 [bits/kmer]
  num_super_kmers_before_bucket: 0.15285 [bits/kmer]
  offsets: 2.30209 [bits/kmer]
  strings: 2.00815 [bits/kmer]
  skew_index: 1.33681e-05 [bits/kmer]
  weights: 0.000307465 [bits/kmer]
    weight_interval_values: 5.34722e-05 [bits/kmer]
    weight_interval_lengths: 0.000200521 [bits/kmer]
    weight_dictionary: 5.34722e-05 [bits/kmer]
  --------------
  total: 4.76611 [bits/kmer]
avg_nanosec_per_positive_lookup 772.495
avg_nanosec_per_negative_lookup 1013.68
avg_nanosec_per_positive_lookup_advanced 770.203
avg_nanosec_per_negative_lookup_advanced 1025.71
avg_nanosec_per_access 385.194
iterator: avg_nanosec_per_kmer 30.9183
2024-01-16 21:09:52: saving data structure to disk...
2024-01-16 21:09:52: DONE

I guess there is some slight slowdown, especially if you change to __uint128_t, but it also seems to me that the slowdown is not very significant compared to generality that this approach brings.
And, of course, I also tried out to run all the checks in the debug mode, both on the example query below, and also on datasets se.ust.k47.fa.gz and se.ust.k63.fa.gz. If you think that this change aligns with your vision for SSHash, I would be happy if you could review it, so that it can potentially be merged in the future.

Hi @adamant-pwn and thanks for this! I quickly skimmed through the commits and the code seems really well done.
So, as far as I see, you use a template for the internal storage of a kmer.
I wonder whether you could test SSHash with __uint128_t for k=63 for larger datasets. For example with the datasets I have here: https://zenodo.org/records/7239205. You could try human.63 or something similar to make a larger test.
The performance should be compared with the numbers here https://github.com/jermp/sshash/tree/master/benchmarks.
On the same line, do you think one could use __uint256_t ? As far as I know, it is not yet standard nor well supported as an "integer" data type since it is a vector actually...
I'm curious to see what could happen with, say, k=96 or k=127. The space of SSHash should improve a lot!
Thanks,
-Giulio

Thanks for the pointers, I will go over the datasets that you shared, and get back with results here.
In the meantime, could you elaborate a bit on which __uint256_t are you referring to exactly? I think Metagraph uses the type from sdsl, do you want to check that one?
Theoretically, I can also make a bit of changes to support std::bitset<256>, as we're still mostly working with bit operations, except for when we have to compute (1<<b)-1, but this mask can as well be constructed in increments of 64 bits, like the reverse-complement works. I'd expect there may be a significant slowdown with standard bitsets, but it should allow arbitrarily large kmers.
Also do you have any datasets to check against 256-bit kmers?

Hi @adamant-pwn,
thanks for your willingness in testing this out!
You're right, sorry for the confusion, I was referring to __m256i. In general, it would be very nice to be able to support kmers larger than 63 (hence, needing more than 128 bits. For example: can we try this lib for 256 bits https://github.com/calccrypto/uint256_t?). That's why in my previous answer here https://github.com/jermp/sshash/issues/35 I was suggesting to look for C++ libraries for arbitrary-long kmers.
-Giulio

Hi @adamant-pwn,
thank you very much for the update! Nice that the generalization does not compromise anything for k <= 63 (i.e., when using __uint128_t as internal representation).
Of course a 15X slowdown is not acceptable. I guess this comes from the uint256 type being bad rather than your code since it works very well for k <= 63. We have to come up with a more efficient way of handling larger kmers...

Hi @jermp,
I spent a bit more time investigating possible ways to use a 256-bit type. At first, I did some changes to accommodate possible usage of std::bitset<256>, but it obviously also was quite slow. I also couldn't find a way to speed up calccrypto's implementation, and as I only really needed some bit operations and not all the other arithmetic, I decided to just implement something ad hoc.
I ended up making a new structure bitpack<Int, height> that represents $k\cdot 2^{h}$-bit integers by putting builtin $2^k$-bit integers into leaves of a templated full binary tree of height $h$.  This way, in particular, we can use bitpack<__uint128_t, 1> or bitpack<uint64_t, 2> to emulate $256$-bit integers with bit operations, or more generally, bitpack<__uint128_t, h> for $128 \cdot 2^h$-bit integers.
I think for really long numbers, it would be more efficient to just store them in a plain array or vector rather than in a tree (like std::bitset does), but the tree structure is probably fine as long as $k$-mers are not super large (and if they were, we would have other concerns about computational complexity as a bottleneck anyway).
I removed any strings that is shorter than 127 from human.k63, and ran the benchmark again:

Bench log with bitpack<__uint128_t, 1> and k=127
k = 127, m = 31, seed = 1, l = 6, c = 3, canonical_parsing = false, weighted = false
reading file '../data/unitigs_stitched/human.k127.unitigs.fa.ust.fa.gz'...
m_buffer_size 29411764
read 100000 sequences, 182189115 bases, 169589241 kmers
read 200000 sequences, 353905567 bases, 328705693 kmers
read 300000 sequences, 535877434 bases, 498077560 kmers
read 400000 sequences, 717529176 bases, 667129302 kmers
read 500000 sequences, 894938693 bases, 831938819 kmers
read 600000 sequences,<PHONE_NUMBER> bases, 993555783 kmers
read 700000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 800000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
sorting buffer...
saving to file './sshash.tmp.run_1707337830575510935.minimizers.0.bin'...
read 900000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 1000000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 1100000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 1200000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 1300000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 1400000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 1500000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 1600000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 1700000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
sorting buffer...
saving to file './sshash.tmp.run_1707337830575510935.minimizers.1.bin'...
read 1797875 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
num_kmers<PHONE_NUMBER>
num_super_kmers 55359550
num_pieces 1797876 (+0.172498 [bits/kmer])
=== step 1: 'parse_file' 1869.21 [sec] (711.673 [ns/kmer])
 == files to merge = 2
num_written_tuples = 50000000
num_written_tuples = 55359550
=== step 2: 'build_minimizers' 26.5036 [sec] (10.0909 [ns/kmer])
bits_per_offset = ceil(log2(2853028173)) = 32
m_buffer_size 20833333
sorting buffer...
saving to file './sshash.tmp.run_1707339726346319196.bucket_pairs.0.bin'...
num_singletons 50839844/51714801 (98.3081%)
=== step 3: 'build_index' 33.2175 [sec] (12.6471 [ns/kmer])
max_num_super_kmers_in_bucket 29204
log2_max_num_super_kmers_in_bucket 15
num_buckets_in_skew_index 5287/51714801(0.0102234%)
num_partitions 7
computing partitions...
num_kmers belonging to buckets of size > 64 and <= 128: 10007008
num_kmers belonging to buckets of size > 128 and <= 256: 7471182
num_kmers belonging to buckets of size > 256 and <= 512: 6508079
num_kmers belonging to buckets of size > 512 and <= 1024: 5386165
num_kmers belonging to buckets of size > 1024 and <= 2048: 3677016
num_kmers belonging to buckets of size > 2048 and <= 4096: 2186595
num_kmers belonging to buckets of size > 4096 and <= 29204: 4977749
num_kmers_in_skew_index 40213794(1.53108%)
building PTHash mphfs (with 8 threads) and positions...
lower 64; upper 128; num_bits_per_pos 7; keys_in_partition.size() 10007008
  built mphs[0] for 10007008 keys; bits/key = 2.26646
  built positions[0] for 10007008 keys; bits/key = 7.00004
lower 128; upper 256; num_bits_per_pos 8; keys_in_partition.size() 7471182
  built mphs[1] for 7471182 keys; bits/key = 2.30375
  built positions[1] for 7471182 keys; bits/key = 8.00004
lower 256; upper 512; num_bits_per_pos 9; keys_in_partition.size() 6508079
  built mphs[2] for 6508079 keys; bits/key = 2.32488
  built positions[2] for 6508079 keys; bits/key = 9.00005
lower 512; upper 1024; num_bits_per_pos 10; keys_in_partition.size() 5386165
  built mphs[3] for 5386165 keys; bits/key = 2.35027
  built positions[3] for 5386165 keys; bits/key = 10.0001
lower 1024; upper 2048; num_bits_per_pos 11; keys_in_partition.size() 3677016
  built mphs[4] for 3677016 keys; bits/key = 2.26738
  built positions[4] for 3677016 keys; bits/key = 11.0001
lower 2048; upper 4096; num_bits_per_pos 12; keys_in_partition.size() 2186595
  built mphs[5] for 2186595 keys; bits/key = 2.34717
  built positions[5] for 2186595 keys; bits/key = 12.0002
lower 4096; upper 29204; num_bits_per_pos 15; keys_in_partition.size() 4977749
  built mphs[6] for 4977749 keys; bits/key = 2.36104
  built positions[6] for 4977749 keys; bits/key = 15.0001
num_bits_for_skew_index 476511808(0.181425 [bits/kmer])
=== step 4: 'build_skew_index' 49.42 [sec] (18.816 [ns/kmer])
=== total_time 1978.35 [sec] (753.226 [ns/kmer])
total index size: 1026.85 [MB]
SPACE BREAKDOWN:
  minimizers: 0.0575382 [bits/kmer]
  pieces: 0.00957263 [bits/kmer]
  num_super_kmers_before_bucket: 0.0321532 [bits/kmer]
  offsets: 0.674475 [bits/kmer]
  strings: 2.1725 [bits/kmer]
  skew_index: 0.181425 [bits/kmer]
  weights: 5.60443e-07 [bits/kmer]
    weight_interval_values: 9.74683e-08 [bits/kmer]
    weight_interval_lengths: 3.65506e-07 [bits/kmer]
    weight_dictionary: 9.74683e-08 [bits/kmer]
  --------------
  total: 3.12766 [bits/kmer]
avg_nanosec_per_positive_lookup 4598.98
avg_nanosec_per_negative_lookup 5773.25
avg_nanosec_per_positive_lookup_advanced 4263.72
avg_nanosec_per_negative_lookup_advanced 5329.6
avg_nanosec_per_access 1585.56
iterator: avg_nanosec_per_kmer 38.8011

So, while there is some slowdown, I think it's reasonable this time, as there is no builtin support for 256-bit integers, and we have to emulate them with 128-bit integers. Also the size indeed reduced quite significantly, from 1617.41 [MB] (4.67 bits/kmer) to 1026.85 [MB] (3.13 bits/kmer), so it should also make up for the slowdown per kmer.
I also tried running it with --check on some smaller datasets, and it seemed to work correctly. That being said, there is always a possibility that I overlooked something, and I'd appreciate any independent verification of the results.

Hi @adamant-pwn,
wow that's very cool! So, if I'm understanding correctly, you use some template magic to let the compiler define recursively the bitwise operators working over two halves. Sweet.
And very nice space result: increasing k improves the space of SSHash. I wonder, where it goes in the limit :)
Although storing a human genome in 3.13 bits/kmer is remarkable, I do not think the experiment is truly correct
because the kmers should come from a de Bruijn graph built with k=127 and not from parsing the unitigs
of a de Bruijn graph of order k=63. So, we should build a dBG with k=127 using BCALM2+UST as explained in the README,
and then index it with SSHash. This will give the true number of bits/kmer (which should be close to the one you had).
Let me know if something is unclear.
Thank you very much!
-Giulio

Thanks for the quick response! Sure, the space result look good. And I do hope that they're genuine and not a result of some bug in the code that misses something or not accounts for something when k goes beyond 63 :smiling_face_with_tear:
Regarding testing on unitigs for k=127, I assume the appropriate raw data is Homo_sapiens.GRCh38.dna.toplevel.fa.gz?

And I do hope that it is genuine and not a result of some bug in the code that misses something or not accounts for something when k goes beyond 63.

I think there must necessarily be some errors when parsing a dBG of order 63 with k=127, but the space result should be quite close though.
Yes, that is the raw data: Homo_sapiens.GRCh38.dna.toplevel.fa.gz.
I would suggest to use GGCAT with the flag --eulertigs to output eulertigs in fasta format that will then be indexed by SSHash.

Ok, below is the log on the "toplevel" dataset. I prepared the input the same way as described in the README:

Bench log on the "toplevel" human DNA, k=127
k = 127, m = 31, seed = 1, l = 6, c = 3, canonical_parsing = false, weighted = false
reading file '../data/unitigs_stitched/Homo_sapiens.GRCh38.dna.toplevel.fa.unitigs.fa.ust.fa.gz'...
m_buffer_size 29411764
read 100000 sequences, 641461998 bases, 628862124 kmers
read 200000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
sorting buffer...
saving to file './sshash.tmp.run_1707385092832615700.minimizers.0.bin'...
read 300000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 400000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 500000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
read 600000 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
sorting buffer...
saving to file './sshash.tmp.run_1707385092832615700.minimizers.1.bin'...
sorting buffer...
saving to file './sshash.tmp.run_1707385092832615700.minimizers.2.bin'...
read 608208 sequences,<PHONE_NUMBER> bases,<PHONE_NUMBER> kmers
num_kmers<PHONE_NUMBER>
num_super_kmers 58916081
num_pieces 608209 (+0.0536296 [bits/kmer])
=== step 1: 'parse_file' 2035.8 [sec] (712.338 [ns/kmer])
 == files to merge = 3
num_written_tuples = 50000000
num_written_tuples = 58916081
=== step 2: 'build_minimizers' 35.5332 [sec] (12.4333 [ns/kmer])
bits_per_offset = ceil(log2(2934546360)) = 32
m_buffer_size 20833333
sorting buffer...
saving to file './sshash.tmp.run_1707387164255862420.bucket_pairs.0.bin'...
num_singletons 51221382/52580188 (97.4157%)
=== step 3: 'build_index' 43.1616 [sec] (15.1025 [ns/kmer])
max_num_super_kmers_in_bucket 28977
log2_max_num_super_kmers_in_bucket 15
num_buckets_in_skew_index 9426/52580188(0.0179269%)
num_partitions 7
computing partitions...
num_kmers belonging to buckets of size > 64 and <= 128: 21978798
num_kmers belonging to buckets of size > 128 and <= 256: 17792929
num_kmers belonging to buckets of size > 256 and <= 512: 16696090
num_kmers belonging to buckets of size > 512 and <= 1024: 18053731
num_kmers belonging to buckets of size > 1024 and <= 2048: 17294062
num_kmers belonging to buckets of size > 2048 and <= 4096: 14098609
num_kmers belonging to buckets of size > 4096 and <= 28977: 20797100
num_kmers_in_skew_index 126711319(4.4337%)
building PTHash mphfs (with 8 threads) and positions...
lower 64; upper 128; num_bits_per_pos 7; keys_in_partition.size() 21978798
  built mphs[0] for 21978798 keys; bits/key = 2.2951
  built positions[0] for 21978798 keys; bits/key = 7.00002
lower 128; upper 256; num_bits_per_pos 8; keys_in_partition.size() 17792929
  built mphs[1] for 17792929 keys; bits/key = 2.32146
  built positions[1] for 17792929 keys; bits/key = 8.00002
lower 256; upper 512; num_bits_per_pos 9; keys_in_partition.size() 16696090
  built mphs[2] for 16696090 keys; bits/key = 2.32747
  built positions[2] for 16696090 keys; bits/key = 9.00002
lower 512; upper 1024; num_bits_per_pos 10; keys_in_partition.size() 18053731
  built mphs[3] for 18053731 keys; bits/key = 2.31951
  built positions[3] for 18053731 keys; bits/key = 10
lower 1024; upper 2048; num_bits_per_pos 11; keys_in_partition.size() 17294062
  built mphs[4] for 17294062 keys; bits/key = 2.32405
  built positions[4] for 17294062 keys; bits/key = 11
lower 2048; upper 4096; num_bits_per_pos 12; keys_in_partition.size() 14098609
  built mphs[5] for 14098609 keys; bits/key = 2.35036
  built positions[5] for 14098609 keys; bits/key = 12
lower 4096; upper 28977; num_bits_per_pos 15; keys_in_partition.size() 20797100
  built mphs[6] for 20797100 keys; bits/key = 2.30088
  built positions[6] for 20797100 keys; bits/key = 15
num_bits_for_skew_index<PHONE_NUMBER>(0.557064 [bits/kmer])
=== step 4: 'build_skew_index' 209.907 [sec] (73.4478 [ns/kmer])
=== total_time 2324.4 [sec] (813.321 [ns/kmer])
total index size: 1199.73 [MB]
SPACE BREAKDOWN:
  minimizers: 0.0537208 [bits/kmer]
  pieces: 0.00327841 [bits/kmer]
  num_super_kmers_before_bucket: 0.0309644 [bits/kmer]
  offsets: 0.659683 [bits/kmer]
  strings: 2.05363 [bits/kmer]
  skew_index: 0.557064 [bits/kmer]
  weights: 5.15061e-07 [bits/kmer]
    weight_interval_values: 8.95759e-08 [bits/kmer]
    weight_interval_lengths: 3.3591e-07 [bits/kmer]
    weight_dictionary: 8.95759e-08 [bits/kmer]
  --------------
  total: 3.35834 [bits/kmer]
avg_nanosec_per_positive_lookup 4558.75
avg_nanosec_per_negative_lookup 5392.28
avg_nanosec_per_positive_lookup_advanced 4565.12
avg_nanosec_per_negative_lookup_advanced 5390.89
avg_nanosec_per_access 1563.39
iterator: avg_nanosec_per_kmer 34.9164

To summarize, total size is now 1199.73 [MB] at 3.35834 [bits/kmer].
Sorry, I already started BCALM2+UST yesterday and it took some time to construct, so I didn't have an opportunity to try GGCAT yet. If it's still relevant, could you please provide specific commands I should use to build the input dataset with GGCAT?

Hi @adamant-pwn, ok that makes sense! I think this is the correct result. I would still perform a check to see if all kmers are found, etc. (just add --check at index construction, as you know).
Thanks!
For GGCAT, it should be very easy to use: just run the construction (no colored dBG needed, so without option -c) with -k 127 and --eulertigs.

I'm sorry, I'm still a bit confused as to what I have to do with ggcat exactly.
I built ggcat, and tried using the following command:
$ gzip -d Homo_sapiens.GRCh38.dna.toplevel.fa.gz
$ ggcat build -k 127 -j 8 --eulertigs Homo_sapiens.GRCh38.dna.toplevel.fa
...
Compacted De Bruijn graph construction completed.
TOTAL TIME: 1387.88s
...
Final output saved to: output.fasta.lz4
$ lz4 -d output.fasta.lz4
Decoding file output.fasta
Decompressed : 135 MB  Error 68 : Unfinished stream
$ ./sshash build -i ../data/unitigs_stitched/output.fasta -k 127 -m 31
...
total index size: 70.1673 [MB]
SPACE BREAKDOWN:
  minimizers: 0.058696 [bits/kmer]
  pieces: 0.0261435 [bits/kmer]
  num_super_kmers_before_bucket: 0.032964 [bits/kmer]
  offsets: 0.628415 [bits/kmer]
  strings: 2.53139 [bits/kmer]
  skew_index: 0.51633 [bits/kmer]
  weights: 9.9489e-06 [bits/kmer]
    weight_interval_values: 1.73024e-06 [bits/kmer]
    weight_interval_lengths: 6.48841e-06 [bits/kmer]
    weight_dictionary: 1.73024e-06 [bits/kmer]
  --------------
  total: 3.79395 [bits/kmer]

Clearly, something goes wrong here, but I can't be sure what exactly. The error during the .lz4 decompression looks very weird, and sshash doesn't seem to work with .lz4 files. I also tried to run ggcat on compressed Homo_sapiens.GRCh38.dna.toplevel.fa.gz, but trying to decompress lz4 also fails when I do this. Should I apply ggcat to the output of BCALM2+UST, rather than the raw file?
Running--check on human genome file seems to take very long. I can try to do it over the weekend, but I also tried running it on smaller datasets and everything seemed okay.

I think you did everything right. The output of GGCAT is some lz file that should be decompressed and given to SSHash as input. The point is that there must be a bug in GGCAT since the file did not uncompress without errors.
Perhaps we should open an issue on the GGCAT repo.
Can you check if with some other small file GGCAT works just fine?
If so, then it is definitely a bug in GGCAT occurring for larger files.

The error actually persists even with smaller datasets:
$ gzip -d se.ust.k31.fa.gz
$ ggcat build -k 31 -j 8 --eulertigs se.ust.k31.fa
...
Final output saved to: output.fasta.lz4
$ lz4 -d output.fasta.lz4
Decoding file output.fasta
Error 68 : Unfinished stream

I opened a new issue about this with ggcat (https://github.com/algbio/ggcat/issues/42), but maybe I'm just misuse ggcat somehow :thinking:

Thanks. I'll try it myself later and see what happens. Perhaps is just the --eulertigs flag?

I think I tried out without it as well, and got the same results. But surely would be nice to verify independently.

In the meantime, here is the full log with --check and --bench on k=127 file that I built with BCALM2+UST. I also built sshash in Debug mode for it, so that if any asserts triggered it would fail. But so far it seems that the check completed successfully.

Beautiful, thanks Oleksandr!

Hi Giulio! I was on vacation for last two weeks, but am now back and ready to work further on this. That is, if further work if required 🙂
I saw that you accepted a pull request from Harun that creates a slight merge conflict with this PR. I suppose I should resolve it now, which I'll try to do soon. Are there any other actions related to this PR that you'd like me to take?

Hi @adamant-pwn, sorry for the late reply. I do not think there are further actions :)
SSHash on such larger k is so nice and powerful!
If you're interested (and given that you worked on this) we can discuss a potential SSHash-v2 paper with extensions and engineering. CC: @rob-p  and @RagnarGrootKoerkamp who are definitely interested.

I added a new commit that merges upstream changes from Harun's PR. Should be ready for merge now 🙂
Regarding SSHash-v2 paper: Sure, I would be happy to contribute! Please note though that I don't really have any formal publication experience, so I might need quite a bit of explanation on what exactly is needed from me 🙂
As a side note, current implementation for large k is really more like a proof of concept than something that should work for really large k, as with larger k we eventually break the assumption that operations with k-mers are $O(1)$, and the way they are now, I think there are a lot of places where the number of operations would scale with $\frac{k}{64}$ or even $(\frac{k}{64})^2$. Which is still fine when $\frac{k}{64}$ is at most a very small constant, but scaling it further would probably require a different k-mer model (if it is ever of interest, which as well might not be the case for biology applications).

Hi @jermp ! I noticed that there were some new commits to the master branch on your side, which created a new marge conflict for this PR. I can update the PR to resolve them too, but I'd appreciate it if you could share your plans regarding merging this PR, so that I don't get stuck in a loop when I need to constantly update it to resolve new potential merge conflicts :slightly_smiling_face:

Hi @adamant-pwn, sorry you're absolutely right!
What I merged yesterday into the master branch was something I began earlier and that I wanted to make the default but just delayed its integration for no good reason.
You did an absolutely great work, and I'm looking forward to integrate this into the master branch. Before doing that, I will perform some benchmarks to be sure we are not sacrificing speed nor space, at least for the current most important application of SSHash: indexing DNA (so 2-bit encoded strings). But it looks from your reports that your implementation does not sacrifice anything.
As I said before, with @rob-p we're discussing plans for a SSHash-v2 paper and you're more than welcome if you want to be part of that.
Thanks!
-Giulio

Hi @jermp , thanks for the update! I've merged your changes into the PR, and also briefly re-run check and bench on se.ust.k63.fa.gz to make sure I didn't break something while doing so. Everything still looks good to me:

Run log

okulkov@compute-biomed-05:~/apps/sshash/build$ ./sshash build -i ../data/unitigs_stitched/se.ust.k63.fa.gz -k 63 -m 31 --bench --check
k = 63, m = 31, seed = 1, l = 6, c = 3, canonical_parsing = false, weighted = false
reading file '../data/unitigs_stitched/se.ust.k63.fa.gz'...
m_buffer_size 29411764
sorting buffer...
saving to file './sshash.tmp.run_1711546442838258499.minimizers.0.bin'...
read 238 sequences, 4958815 bases, 4944059 kmers
num_kmers 4944059
num_super_kmers 290990
num_pieces 239 (+0.00599427 [bits/kmer])
=== step 1: 'parse_file' 1.29271 [sec] (261.467 [ns/kmer])
=== step 2: 'build_minimizers' 0.234789 [sec] (47.4891 [ns/kmer])
bits_per_offset = ceil(log2(4958878)) = 23
m_buffer_size 20833333
sorting buffer...
saving to file './sshash.tmp.run_1711546444365949536.bucket_pairs.0.bin'...
num_singletons 289962/290421 (99.842%)
=== step 3: 'build_index' 0.032482 [sec] (6.56991 [ns/kmer])
max_num_super_kmers_in_bucket 8
log2_max_num_super_kmers_in_bucket 3
num_buckets_in_skew_index 0/290421(0%)
=== step 4: 'build_skew_index' 0.001391 [sec] (0.281348 [ns/kmer])
=== total_time 1.56137 [sec] (315.807 [ns/kmer])
total index size: 2.25106 [MB]
SPACE BREAKDOWN:
  minimizers: 0.189121 [bits/kmer]
  pieces: 0.00109384 [bits/kmer]
  num_super_kmers_before_bucket: 0.0921154 [bits/kmer]
  offsets: 1.35377 [bits/kmer]
  strings: 2.00601 [bits/kmer]
  skew_index: 1.29448e-05 [bits/kmer]
  weights: 0.000297731 [bits/kmer]
    weight_interval_values: 5.17793e-05 [bits/kmer]
    weight_interval_lengths: 0.000194172 [bits/kmer]
    weight_dictionary: 5.17793e-05 [bits/kmer]
  --------------
  total: 3.64245 [bits/kmer]
checking correctness of access and positive lookup...
checked 4944059 kmers
EVERYTHING OK!
checking correctness of negative lookup with random kmers...
EVERYTHING OK!
checking correctness of navigational queries for kmers...
checked 4944059 kmers
EVERYTHING OK!
checking correctness of navigational queries for contigs...
checked 238 contigs
EVERYTHING OK!
checking correctness of kmer iterator...
EVERYTHING OK!
checking correctness of contig iterator...
EVERYTHING OK!
avg_nanosec_per_positive_lookup 639.533
avg_nanosec_per_negative_lookup 884.007
avg_nanosec_per_positive_lookup_advanced 639.922
avg_nanosec_per_negative_lookup_advanced 899.111
avg_nanosec_per_access 153.53
iterator: avg_nanosec_per_kmer 24.37

Looking forward to your benchmarks and merging it into the master branch! Please let me know if there are any further questions or requests about the PR on your end.
Re: SSHash-v2 paper. As I wrote above, I'll be happy to contribute! Just let me know which actions are expected on my end for this :slightly_smiling_face:

Fantastic! Thank you so much. I'll be running some benchmarks soon and keep you posted here.
So, what's the largest kmer length we could use in your opinion and still get some fast queries? k=127 perhaps?
In a follow-up paper, we should definitely include results for k in range(47, 127+1, 16).

For the largest kmer length, it's hard to tell because we didn't really try anything beyong k=127 yet. For me, k=127 seemed reasonably fast, but I don't know if it adheres with your scale. We can also try k=255 or k=511 just to see how much time it takes. It should be quite slow, but sufficiently fast to at least gather benchmark data?
On the other hand, what's the largest "practically interesting" value of k?
I mean e.g. if you wanted to lookup with k=1023, would it even make sense to build sshash for such k instead of build it for k=127 and only lookup for the first 127 characters? Or maybe the threshold is with even smaller k? I know that on random strings building such structure for k=20 should be enough to almost never have any errors whatsoever, but I don't know what's the smallest k with such property for real-life data.
Trying something with proteins would be cool, of course, but there is no test coverage for this case yet, and it could be that I still missed some places where e.g. the size of the alphabet is hard-coded as a constant number 2, rather than kmer_t::bits_per_char (as you see with one of two latest commits).
So, I guess we would need to double-check everything on small inputs with some naive algorithms when we try alternate alphabets. Or we can wait a bit until Metagraph integrates sshash to the degree that also includes usage of generic alphabets, which should hopefully happen in a foreseeable future :slightly_smiling_face:

For the largest kmer length, it's hard to tell because we didn't really try anything beyong k=127 yet. For me, k=127 seemed reasonably fast, but I don't know if it adheres with your scale. We can also try k=255 or k=511 just to see how much time it takes. It should be quite slow, but sufficiently fast to at least gather benchmark data?

k=127 seems very reasonable to try. I've heard someone mentioning applications for very large k, although now I cannot recall.

On the other hand, what's the largest "practically interesting" value of k?

I think nobody has the answer right now :) k=31 and k=63 are used just because they are handy to handle
and are sufficiently long to have good specificity. On the other hand, a very large k can break sensitivity in case
of in/dels.

Trying something with proteins would be cool, of course, but there is no test coverage for this case yet, and it could be that I still missed some places where e.g. the size of the alphabet is hard-coded as a constant number 2, rather than kmer_t::bits_per_char (as you see with one of two latest commits).

Of course!
Further steps would be:

do some benchmarks first;
integrate your changes to the master branch;
start an email thread with folks potentially interested in a SSHash-v2 paper/software.

Hi @adamant-pwn and @jermp,
There was a "very large k" request in the Cuttlefish repo some time back (https://github.com/COMBINE-lab/cuttlefish/issues/22).  I think the most common use case for such a large k would be when one is doing comparative genomics on many very similar genomes. In that case, you'd like k to be large enough to span most repeats to disentangle the graph, and you expect relatively less branching from variation (because the genomes are very similar).  Likewise, I'm not sure about 1000, but larger k-mers could become more useful for mapping as we move toward "ultra high quality" long reads.
--Rob

Hi @jermp any updates on merging this (and also on communications re: preparing a new paper)? 👀
I added two minor pull requests from Harun to our fork, they are very straightforward and just fix some warnings. But I expect that some further work might be needed in the future for integration of MetaGraph with SSHash, and it'd be easier if the current PR is merged, so that further changes in our fork won't interfere unduly with the PR.

Hi @adamant-pwn, I've been busy with other work recently and didn't have time to start benchmarking this.
But I will soon. So the plan I wrote two months ago (!) hasn't changed. My intention is to first replicate these benchmarks https://github.com/jermp/sshash/tree/master/benchmarks and see if there are any differences, etc.
Best,
-Giulio

Hi @adamant-pwn,
I'm in the process of merging this PR. Thank you again for the nice job! (I read again the changes and they seem really great.) However, to start benchmarking it and include some more changes I have in mind (see below), I'd like this PR to be for the dev branch rather than the master branch. Can you rebase? Thank you!
After this PR is merged into the dev branch, here is what I'd like to do:

Use the library https://github.com/jermp/bits to handle bitvectors, rank&select, elias-fano, etc. (this comes with the current version of PTHash; I've already done this in the bits branch if you want to have a look at it).
Externalize the minimizer logic into a separate library. Would be great to have a library, say kmers_and_minimizers, that contains your kmer implementation and minimizer computations. With you consent, can I re-use your kmer impl?
Modify the SSHash logic to store the positions of the minimizers in the strings rather than the position of the starting nuc of each super-kmer. (This simplify the logic a little bit and speeds up construction.)

If you're interested in this future development, just let me know.
Thanks!

@NPSDC —- we should try and get our PR in before this is ultimately merged so that we can benefit from these updates!

Hi @jermp, thanks for the update!
I merged your latest changes to master into the pull request, and also changed the target branch to dev.

With your consent, can I re-use your kmer impl?

Sure! I think generally contributions to the project should be governed by one license. SSHash is already licensed by MIT License, I think it's fine to keep using it, but it'd make sense to update the list of authors.

If you're interested in this future development, just let me know.

Sure, please let me know if there is anything in particular that I can help with.

Hi @jermp,
This sounds great!  I was wondering if we can figure out a merge order between the submodule branch where the changes @NPSDC and I made are hosted and the changes that are being proposed here.  They should be largely orthogonal. However, to avoid the very issue that Noor so painfully solved for he and I in his most recent PRs, I'd very much like to avoid having the branch used in piscem diverge from what is happening upstream.  What are your thoughts?
Thanks!
Rob

Hi @adamant-pwn and @rob-p, and thank you for your prompt interactions.
@adamant-pwn: thanks for the rebase. You'll definitely feature as one of the authors, as well as Rob and all the other people involved in a SSHash follow-up paper :)
@rob-p: yes, you're right. I think it would probably be more difficult to include all Oleksandr's changes to the submodule-integration-for-piscem than the other way round. I just opened the PR from submodule-integration-for-piscem to dev: https://github.com/jermp/sshash/pull/50. The changes are not many; so, perhaps it makes sense to first merge this PR into dev and then incorporate the additional changes required for piscem. What do you think?
( Or is @adamant-pwn willing to also incorporate those changes to his own PR?! :D )

@rob-p, let me know if the above plan makes sense to you. I think merging this PR first into dev and then adding Noor's changes (I'll do that) is the way to go.

Ok perfect; thanks @rob-p for confirming. So, I'll merge this PR first and then integrate Noor's changes there!

Woot. Congrats @adamant-pwn and @jermp on the first major steps toward sshash 2!

All credit so far goes to @adamant-pwn! Thank you again for this.
Now I can work on the dev branch to make my own changes listed above.


Implemented Fork on ViewSnippetHeader

Fixes #112

Thanks @seveibar . Btw understanding the codebase was a challenging task for me. Lots of components, hooks, routes.

@ni9999 it can be quite tricky, glad you were able to figure it out. Do you see any areas where it could be improved?

Not much. Also I have little idea as being relatively new in frontend. I thought about reducing hooks and components by defining some function of these directly in the components and pages. ie reducing the numbers of files. But it will decrease reusability.
Also make some documentation of snippet-api usage


Install gems in Dockerfile instead of in zip and upload job

The zip and upload job is currently installing gems every time it is run. This takes a lot of time and isn't necessary. This PR installs the gems in the Dockerfile.
@orcutt989  I tested this over a week ago. It should work, but as part of your work on templating everything in the jekyll theme, please make sure it works fully.

@harryttd i think we are good now can you double check

Looks good thank you. I will merge now.


OHBM 2020 - BIDS-Derivatives poster

@effigies is planning to submit an abstract to OHBM2020 for BIDS-Derivatives and inviting contributors to provide feedback and sign on as co-authors
More information can be found here: https://groups.google.com/d/msg/bids-discussion/2ti6ev7isX0/-hIYV_JQAQAJ
cc: @bids-standard/derivatives

Thanks, @franklin-feingold and @effigies ! Do you think it would make sense to additionally ping everyone who attended the BIDS derivatives sprint last year ? Just to make sure they've seen this !


idf.py build fails due to  fatal error: esp_event.h: No such file or directory

I am running IDF version 5.1.2  on MacOS Sonoma on a 2020 Intel MacBook Pro following the instructions to install the toolchain and I get this error on the idf.py build call:
n file included from /Users/nikhelkrishna/esp/sscma-example-esp32/components/sscma-micro/porting/espressif/el_device_esp.cpp:37:
/Users/nikhelkrishna/esp/sscma-example-esp32/components/sscma-micro/porting/espressif/el_network_esp.h:32:10: fatal error: esp_event.h: No such file or directory
32 | #include "esp_event.h"
|          ^~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Compilation failed because el_network_esp.h (in "sscma-micro" component) includes esp_event.h, provided by esp_event component(s).
However, esp_event component(s) is not in the requirements list of "sscma-micro".
To fix this, add esp_event to REQUIRES list of idf_component_register call in /Users/nikhelkrishna/esp/sscma-example-esp32/components/sscma-micro/CMakeLists.txt.
ninja failed with exit code 1, output of the command is in the /Users/nikhelkrishna/esp/sscma-example-esp32/examples/yolo_detection_demo/build/log/idf_py_stderr_output_7789 and /Users/nikhelkrishna/esp/sscma-example-esp32/examples/yolo_detection_demo/build/log/idf_py_stdout_output_7789

Hello, please delete the components/sscma-micro/CMakeLists.txt file first, and then try the latest commit f90997c8cb4104a8995d277fa78790497357bb40 (run git pull in the root directory of the project).
I hope this solves your problem, thanks.

Hello, please delete the components/sscma-micro/CMakeLists.txt file first, and then try the latest commit 4acdaa3 (run git pull in the root directory of the project).
I hope this solves your problem, thanks.

It is now saying CMakeLists.txt not found in project directory /Users/nikhelkrishna/esp/sscma-example-esp32
when I run idf.py set-target esp32s3


Change download_golang script for RPM Build download path

Issue #, if available:
Description of changes: This PR is the second part of this PR. The version.yml file has also updated with the latest golang versions as per of golang new release PR.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

it looks like the 1.20 artifacts are being uploaded to the S3 bucket with the .0 path version truncated -- and up as 1.20 rather than 1.20.0

/test builder-base-tooling-presubmit-al-2

/retest

/lgtm

/approve

/unhold


remove link to internal document

As per requested, remove link to internal Accessible ICT Procurement Guide

Review the PR and it looks good


Harvester Integration - Harvester List Page

New Global Navbar link for Harvester and a page to list Harvester host clusters and functionality to import them

Harvester host clusters are a k8s cluster with a provider of "Harvester"
Ability to click on link of harvester cluster and open Harvester UI

@deniseschannon the Harvester UI team will work on this feature.

And just to clarify the design, from the Hamburger menu >> Virtualization Management >> Harvester Listing Page ---click one to go to the single cluster view-->> single Harvester cluster view . cc @yasker

additionally, @deniseschannon can u please help to ask about the design of Import Existing Harvester Cluster, or it will just be similar to the Import the external k8s cluster, thanks

This feature is already included in the PR https://github.com/rancher/dashboard/pull/3726, u can test by replacing the ui-dashboard-index value with the https://releases.rancher.com/harvester-ui/dashboard/latest/index.html, let us know if there is any design enhancement needed, thanks

@guangbochen Is your team doing this? I see in the Harvester PR on dashboard that you have a list page for Harvester clusters - I don't see the ability to click on one and open the Harvester UI.

Yep the harvester team has already include this feature in the https://github.com/rancher/dashboard/pull/3726, since this is not merged so u won't be able to test it without replacing the ui-dashboard-index  settings with the custom build UI.
I believe there is actual work left for the Rancher UI team except the PR review which is already WIP, thanks

@guangbochen "I believe there is actual work left for the Rancher UI team except the PR review which is already WIP, thanks" - there is, or is NOT?

Based on what was Richard has shown, this looks to be done but we'll allow @richard-cox to review.

I don't think there's an issue here. Kube resources underpin everything shown in the dashboard, including the continues delivery, cluster management and now virtualisation management pages. In terms of the later two the user will manage the lifecycle of the general rancher or harvester cluster from those pages. The cluster explorer on the other hand is a view of the raw kube resources inside a specific cluster. So management views to manage the clusters, explorer view to dive into the cluster.
As per slack comment, moving this back to test

Based on comments from prospects, the user expectation is:
When navigating to the virtualisation management page,  i can click on a cluster name in the list and that directly allows me to view to manage the VM resources in that cluster.

Ah, ok I think I've pin-pointed my issue with this: in short, the explore and manage buttons are swapped on the virtualization page.
In cluster management, when the user clicks the name of the cluster, they are brought to the manage page (node pools, related resouces, etc.) and there is a button on the far right to explore the cluster

however, in virtualization management, these are swapped.  There is a manage button on the far right, and clicking the name now brings you to the equivalent of the explore button from the previous example.

Can we make this more consistent between the two management panes?

I also feel a little weird that in the virtualization management page, click the name brings users to the VM management instead of resources. I remember it was the same as explorer a few days back, e.g. click the name bring to the resource page, and click the Harvester (the button was called Harvester instead of Manage at the time) brings to the Harvester UI page.
We still need the resource page for downloading kubeconfig etc.
It's more depends on how Rancher UI team wants it. Personally, I would prefer the way to be consistent across different pages, but I am not the designer for this page. cc @gaktive

Per @deniseschannon it's better to say Harvester Clusters in the Virtualization Management page to distinguish from Kubernetes clusters which are showing on the Cluster Management page.

I think the UX is okay - clicking on the name takes you a level down where you are - in Cluster Manager, clicking shows cluster detail. In VM, clicking drills down to the harvester UI for that cluster.

@n313893254 @WuJun2016 can you help to tweak the title of the cluster list inside virtualization management per https://github.com/rancher/dashboard/issues/3466#issuecomment-924467964 cc @guangbochen

on v2.6-head (06e19b0):

virtualization management page now shows title as harvester clusters -- pass
create button on the virtman page also shows title as harvester clusters: create -- pass

warning is not on harvester create page -- pass

explore and manage button/links are now working and match behavior on cluster management page -- pass


Non-working datasets in the first page of hf.co/datasets

As of today (2023/05/03), the dataset viewer does not work for the following datasets, in the list of the 30 most downloaded datasets (https://huggingface.co/datasets):

[ ] https://huggingface.co/datasets/allenai/nllb, Error code:   JobRunnerCrashedError
[ ] https://huggingface.co/datasets/facebook/flores, Error code:   JobRunnerCrashedError
[ ] https://huggingface.co/datasets/allenai/c4, Error code:   JobRunnerCrashedError
[ ] https://huggingface.co/datasets/lukaemon/mmlu, Error code:   StreamingRowsError
[x] https://huggingface.co/datasets/piqa, [Errno 2] No such file or directory: '/datasets-server-cache/all/datasets/2023-05-03-18-48-58-config-parquet-and-info-piqa-f5651569/piqa/plain_text/1.1.0/6c611c1a9bf220943c4174e117d3b660859665baf1d43156230116185312d011/piqa-train.parquet'
[x] https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt, Error code:   JobRunnerCrashedError
[x] https://huggingface.co/datasets/hendrycks_test, Couldn't get the size of external files in _split_generators because a request failed: HTTPSConnectionPool(host='people.eecs.berkeley.edu', port=443): Max retries exceeded with url: /~hendrycks/data.tar (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f15f029aac0>: Failed to establish a new connection: [Errno 111] Connection refused')) Please consider moving your data files in this dataset repository instead (e.g. inside a data/ folder).
[x] https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing, error in some cells: ERROR: type should be images list, got [ null, { "src": "https://datasets-server.huggingface.co/assets/HuggingFaceM4/cm4-synthetic-testing/--/100.repeat/100.unique/0/images/image-1d300ea.jpg", "height": 19, "width": 32 }, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null ] - see https://github.com/huggingface/moon-landing/pull/6218 (internal)
[x] https://huggingface.co/datasets/GEM/wiki_lingua, Error code:   JobRunnerCrashedError

cc @albertvillanova fyi

Yes!
I think we should regularly monitor that the main / trending datasets are working, but surely not through this issue. Thanks for looking


The meaning of the "fn+dt" couples.

In Section 4.2, -EAB is described as "Count of attached qualified Base64 first seen replay couples fn+dt". However, the specification does not explain what exactly is "fn" nor explains why it is necessary.
One actual example I found from the reference implementation is -EAB0AAAAAAAAAAAAAAAAAAAAAAA1AAG2023-07-20T17c16c19d807575p00c00. In this case, "fn" is 0AAAAAAAAAAAAAAAAAAAAAAA, which implies it is a 24-character number with a value of zero. What would be the scenario that makes "fn" not equal to zero?

Sorry @nkongsuwan, we moved the spec to trustoverip/tswg-acdc-specification, can you close and reopen there? This repo will be archived after the IPR update window closes at ToIP.

sure


[MCS-DE-GO] Access to Skill Manifest Warning causes uncaught error type

When clicking on the Warning on the "Notifications" page for the following warning

We receive the following error:

@joaollq @goergenj @garypretty @benbrown @cwhitten

@goergenj can you provide a bit more information? What version of Composer, repro steps, etc. Thank you!

@joaollq Please provide a quick update on the issue,

Hi all, we found what was causing the issue.
The skill manifest URLs on our customer's environment require authentication.
We changed the getSkillManifest method to add a Bearer token to the HTTP request and we lost that code when merging to the newest version.
After adding again the Authorization header, the error disappeared.

@cwhitten @garypretty Can you please add the support for authenticaiton as a feature request please.
@joaollq Please create a pull request for our solution for the team to review.

@joaollq Please create a pull request for our solution for the team to review.

@joaollq is this on your plate? If/when please make sure to link the PR to this issue so we can track it correctly. Thanks

@joaollq Please create a pull request for our solution for the team to review.

@joaollq is this on your plate? If/when please make sure to link the PR to this issue so we can track it correctly. Thanks


async ... await ... does not work under IE

Hi
problem is, if it is used, comes Exception
ERROR [app-router] TypeError: Object.getOwnPropertyNames: az argumentum nem objektum
[object Error]{description: "Object.getO...", message: "Object.getO...", name: "TypeError", number: -2146823281, stack: "TypeError: ..."}

Can you provide more info why you think aurelia-polyfills is at wrong and how?
async await is a new syntax and as such can not be polyfilled, it is transpiled, so what are you using for that - babel, ts, ...?


Make keybase.desktop Linux shortcut not kill Keybase by default

When you install Keybase on Linux it runs in the background on system boot. The run_keybase kills all the Keybase processes and starts them up again, so if Keybase is running it restarts it.
When I click the Keybase shortcut, I know Keybase is already running in the background and I'd expect it to just show the GUI and not restart Keybase. However with the current keybase.desktop shortcut on Linux, when you click on the shortcut, it runs run_keybase. Because this command involves killing all Keybase processes, it can be problematic and surprising, as I've found on one occasion.
The biggest problem is, what if you're uploading something to KBFS? If you've moved the files to KBFS, and then say someone's sent you a message on Keybase, and you go to click the shortcut in the app launcher to see it, Keybase will restart and the files you were uploading will be lost. This is especially a problem because some distros are either planning to or already have removed the systray. I recently upgraded to elementary OS Juno and they've already done this, so the only way I can open Keybase in a user friendly way is to click on the launcher.
The PR basically changes the launcher to point to the Keybase GUI executable instead. It now just brings up the Electron UI. If the Keybase GUI is already up, it does nothing. I don't think this would create any problems, unless there are any circumstances where the Keybase GUI executable won't get installed to /opt/keybase/Keybase. As a bonus, showing the GUI is slightly quicker than it was previously.
run_keybase is still there in the form of a "Restart" action (which more accurately represents what it does) which (at least on my distro) shows as an option when you right click the icon.
Thank you for Keybase, by the way!

That's sort of broken, since the keybase GUI depends on the daemonized kbfs and keybase helper processes and this change would result in them never being started at all.
The appropriate way to deal with this is, of course, that the run_keybase script should stop trying to do altogether too many things at once. I solved this by replacing run_keybase with my own wrapper script https://git.archlinux.org/svntogit/community.git/tree/trunk/keybase-gui?h=packages/keybase and shipping this for my distro package.


[Feature Request] Warn when changing a relationship end and other views are impacted

When editing a view, a user can move the ends of a relationship. This updates the underlying relationship in the model and immediately affect other views in which the same relationship was present. This can lead to views being no more valid without the user noticing it.
When moving a relationship's end, I'd suggest that Archi checks if the relationship is used on other views and (if any) show a warning dialog similar to this one:

Unfortunately, this is not possible because of the way GEF works in ArchimateDiagramConnectionPolicy.
When reconnecting, for each drag (and the completed reconnect) action, the Policy asks for a Reconnection Command (that in itself is a very expensive operation). When the reconnection is completed and the Command is executed it is then too late to display a dialog. I.e there is no way to veto the execution of the Command.
I tried some tricks like interrupting the Command's execute() method but this messes up the Command stack.
The best I've been able to do is show an information dialog that the connection is used in other Views and has changed after the operation. At this point the user can Undo.

Committed to show-connection-warning branch.

The best I've been able to do is show an information dialog that the connection is used in other Views and has changed after the operation. At this point the user can Undo.

Seems good. But I wonder if at this stage we could not pretend that it still hasn't been applied and then if the user choose "no" to the confirmation dialog, then we undo the whole operation ?

Tried that. At that point the command has been executed but not put on the Command Stack so Undo is not possible. I tried lots of things ;-)

Tried that. At that point the command has been executed but not put on the Command Stack so Undo is not possible. I tried lots of things ;-)

Actually, if we call the Undo command with Display#asyncExec() we can do this. So I've committed this change.

Closing this as it is implemented for the next version. Please re-open if any issues are found when released,


Simple XML/JSON export

I was wondering if it is possible to include an export function which allows you to save an XML/JSOn (or whatever you use to store the notes) and push it as an download at the users request.
This is in particular usefull if you want to save your notes (either just to have it or when migrating to a new phone).
Or perhaps an link to the location of the notes/xml (in the File Manager) so users can decide themself what to do with the data...

Good idea. It should be checked whether this could be combined somehow with #25 .


mypyc in setup.py

mypyc is used by mypy and black (latest version) for various optimisation. I wonder how does it compares to nuitka

@anki-code I found PyOxidizer interesting. It can static link and compile modules to be loaded from memory.


topandas() seems subtly broken

this works fine
a = [
 [1.,1, 1.,1., 1.],
 [1.,1.,1.,1., 1.,1.]
]

awkward.topandas(awkward.Table(pt = awkward.fromiter(a)))

and continues to work for any two arrays except for those that are exactly the same size
this also works
a = [
 [1.,1, 1.,1., 1.],
 [1.,1.,1.,1.]
]

awkward.topandas(awkward.Table(pt = awkward.fromiter(a)))

while this breaks
a = [
 [1.,1, 1.,1.],
 [1.,1.,1.,1.]
]

awkward.topandas(awkward.Table(pt = awkward.fromiter(a)))

and does this
a = [
 [1.,1.],
 [1.,1.]
]

awkward.topandas(awkward.Table(pt = awkward.fromiter(a)))

PR #198.
This is another case of hitting a Numpy corner case. (If the JaggedArray happens to be regular, Numpy turned it into a multidimensional array, rather than an object array of arrays, despite the dtype being object, when Pandas tries to print the contents to the screen. Note that your examples fail on __repr__, not on constructing the DataFrame.)

I had to switch into bug-bashing mode a few days ago because they started piling up. Sorry that I can't be more consistent.  :)


Return nonvolatile records from zngio.zngScanner.Pull

zngio.zngScanner.Pull returns volatile records, but driver.mux.run sends
them to another goroutine to be read, presenting a potential data race.
Fix that by returning nonvolatile records from zngio.zngScanner.Pull.
Fixes #1087.

Confirming that this fixes my cli repro case: https://github.com/brimsec/zq/issues/1087#issuecomment-672299522

@nwt : would you please a ztest for this?

I've also confirmed that a current master of Brim (commit def9cbb) when pointed at a zqd at the commit 958b7ed (this branch) no longer has errors during ingest as originally reported. 👍
Since this seems to prove @alfred-landrum's theory that the root cause was back when #1063 was merged, it's still a strange mystery to me why the ingest errors can't be reproduced on Brim commit ce37016 which was when Brim's zq pointer was first advanced to include the change in #1063. But since the app breakage is fixed I guess I'm content not knowing. 🤷♂

@nwt : would you please a ztest for this?

Done in 0c0ffd0d07f19518a61585c514577572c269d786.


Support Jakarta

Inject
Provider
Scope
Qualifier

Should fall out naturally in supporting the above

Singleton
Named

Mostly done but dagger's support seems incomplete

Seems this isn't quite working in the most recent anvil-ksp release during factory gen. Will look more into it


WIP: Python packages cleanup

Motivation for this change
pythonPackages is a mess and needs lots of cleanup and removal of unused unmaintained packages.
This is an attempt at starting the effort of cleaning things up.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

cc @domenkozar @bjornfor @lovek323 @nlewo @cillianderoiste

@bjornfor I know a bunch of the packages are not dead but since no one is maintaining them in nixpkgs I figure that they do more harm than good.
I started this by doing a git blame | grep sha256 and starting from the oldest packages.
This is all stuff that hasn't been updated since early 2014 and older.
I see no point in keeping around packages that no one has the intention of maintaining.

I have updated bitstring, demjson, pyodbc, robotsuite in master (0b93dbe9d63895e1ba845bc15dcd5f107958097a and parents). Please rebase.

@bjornfor Please also split them out from python-packages.nix.

@adisbladis: Done (489cdc23fca3562d757295bf9a9b4ee01e45aee5 and 3 parents).

Thank you @bjornfor

@bjornfor Thanks :) Packages dropped from PR.

@adisbladis:

@bjornfor I know a bunch of the packages are not dead but since no one is maintaining them in nixpkgs I figure that they do more harm than good.

Harm to whom? Nixpkgs users or maintainers?

@bjornfor I have a few reasons for this.

It's harmful to users. if I'm on the unstable channel I would expect to get the latest stable release of a given package, not something that is 3-7 years out of date.

It's harmful for security. Lagging several years behind on security updates can never be a good thing.

It's harmful for maintainers. A lot of the python packages are abandoned by their maintainer. You yourself is listed as a maintainer on 25 python packages, another maintainer with a bunch of abandoned packages are listed on 65 python packages. Can you honestly say that you are maintaining all of these?

It's harmful to new contributors. I have on multiple occasions tried to improve and update packages with python dependencies and ended up giving up on the endeavour because it turns out every single dependency is horribly out of date. I'm pretty sure I'm not the only one.

@adisbladis I mostly agree with you, but I do want to add the following. Many of us that have been using Nixpkgs for some time are listed as maintainers for a lot of packages. That's mostly because we have (or had) a need for that package, and simply want to be warned when they are touched. Ideally, each of us would maintain fewer packages, but as is I think obvious, we just do not have that many maintainers, but still we have a need of the packages.

@adisbladis:

It's harmful to users. if I'm on the unstable channel I would expect to get the latest stable release of a given package, not something that is 3-7 years out of date.

The alternative is not having the package at all. (I'd rather have old package than no package.)

It's harmful for security. Lagging several years behind on security updates can never be a good thing.

Good point.

It's harmful for maintainers. A lot of the python packages are abandoned by their maintainer. You yourself is listed as a maintainer on 25 python packages, another maintainer with a bunch of abandoned packages are listed on 65 python packages. Can you honestly say that you are maintaining all of these?

(That doesn't explain how it's harmful to maintainers. I think it's harmful when there are packages that require maintenance, but nobody really cares about the packages anymore. Then it is better to remove, instead of "forcing" people to spend energy on things that don't matter.)
No, I cannot claim that I maintain all my packages, if that means always having the latest version (unfortunately). I could remove myself as maintainer and/or remove all "my" packages if they lack maintainers, but I don't see that as a good solution either.

It's harmful to new contributors. I have on multiple occasions tried to improve and update packages with python dependencies and ended up giving up on the endeavour because it turns out every single dependency is horribly out of date. I'm pretty sure I'm not the only one.

But the alternative would be not having any dependency packages to begin with. Isn't it better to update a few old packages when needed than having to write all expressions from scratch?
To sum up, I think it's good that nixpkgs receives some "clean-up" from time to time (thanks!), but we must also find a balance in how eager we are when cleaning/removing stuff. Being too eager or too lazy is both bad.

Since bpython is well underway in https://github.com/NixOS/nixpkgs/pull/30922 I have dropped it from the PR.

I only noticed this PR when it was merged: it was not ready.

This merge has broken the evaluation of master (for the purpose of nox-review and Travis, and hydra advancing the channel), fixed in #31340.

RedNotebook is a diary application, its users will not be reconciled with its sudden removal, and its old version does not show its age and works perfectly well. I'm restoring and updating it in #31341.

I do not understand or appreciate this approach to packages (particularly applications) that build fine, and just were not updated in a few years. First, they may have satisfied users who did not notice anything lacking and do not care to update; or the old version may even work better than the new. Second, it is often easier to update existing packages than to add new ones. (For instance you've removed Task Coach 1.3.22 when the latest version is 1.4.3. I guess that it has not changed much, and that its old expression is useful to package the update.)

@orivej  Ideally we have expressions, and preferably of the latest versions, but that requires maintainers. We just don't have enough people maintaining all of the expressions.


Add multi-line comment syntax option with [ and ] delimiters as well as current { and } delimiters

When I have clashing #define's but one of them is in a COMMENT [] block, the preprocessor still bombs out on it.
#define x 0
comment [
#define x 0
]

The correct syntax for multiline comment is
comment {
}

A block works, so I would still like it to work correctly.

The spec clearly states that the syntax for a multiline comment is comment {}. So I changed the title of the issue to reflect that this will be a change to the specified behaviour.

Now the compiler will return an error if a string is not passed after comment. Current Red/System loader cannot support blocks as comments.


feat: Add getTuple helper function to ad-sizes.ts

As outlined in this ticket, now that ad sizes have moved over to commercial core it makes sense for some helper functions to join them. This adds a getTuple function and AdSizeTuple type to ad-sizes.ts.

Like this, @mxdvl?


Support for XBRL and iXBRL

Is your feature request related to a problem? Please describe.
Label Studio does not support XBRL (eXtensible Business Reporting Language) or iXBRL (Inline XBRL) input formats.
Describe the solution you'd like
As a data labeler, I want to import XBRL and iXBRL files as tasks, so I can better analyze the U.S. Securities and Exchange Commission (SEC) data.
Describe alternatives you've considered
I assume templating for XBRL/iXBRL Document NER is similar to HTML Document NER.
Additional context
For Abbott Laboratories, here is the iXBRL document and equivalent HTML document.

@slavakurilyak Do you know any converters from XBRL to HTML? Is it ok to get HTML elements positions and offsets as NER labeling results instead of XBRL positions? Because it's not trivial to support XBRL directly.

@makseq I opened an issue with Pandoc here. I assume that treating XBRL as an HTML page is the easiest way to support this file format.


fix(tests): abstract test time intervals to constants with explanations

Our tests pass in random time intervals very often.
Originally posted by @distractedm1nd in https://github.com/celestiaorg/celestia-node/pull/1256#discussion_r1003437837

As we work through extracting testing timing constants we should also consider making a number of the timeout constants build.Vars so that we can have production and testing values. In production these values can be used to prevent long running events from hanging and just return an error instead of waiting.
Or to generalize it more, we should minimize testing only constraints.


Autostart the service

Its probably just a ordinary linux question or related to another app, but will the service autostart if the server is rebooted?
if not, is there an easy way to get it to autostart?
like add to the file /etc/supervisor/conf.d/netbox.conf
"autostart = true" ?

The example supervisor configuration provided in the getting started guide should suffice. autostart is true by default.


I think the game crash because of Third Person

Describe the bug
I keep randomly crashing i think, and i see your mod name in the Mixins Stacktrace, I don't really know, it has been crashing for months now
Reproduce
Steps to reproduce the behavior:
I just loaded the game's worlds with many mods
Expected behavior
I expected it doesn't crash
Additional context
Log: https://mclo.gs/fNtJnfg
Crash log: crash-2024-11-01_20.30.59-client.txt

Thank you for submitting this, but it seems to be an issue with the architectury api rather than this mod. Does this problem occur every time you enter the world, or is it random?

Thank you for reporting this, but it seems to be an issue with the architectury api rather than this mod. Does this problem occur every time you enter the world, or is it random?

It's very random and I've asked some people in the Community, they said it maybe because of the Sounds mod, can you take a look at the log?, I sent them the same log that I sent you above.

they said it maybe because of the Sounds mod

I find that mod Sounds includes some dependencies in its released jar file, including:

Architectury-api
mixinextras
yet-another-config-lib

Usually when a mod depends another mods, it should only claim the dependencies but not include them.
I don't know if there is a special reason why Sounds do this. But you can try removing architectury-api file from sounds and see if it work.
Its path is META-INF/jars/architectury-fabric-9.2.14.jar. It's inside the file Sounds-2.1.0+1.20.1+fabric.jar

Ok thank you, i'll try


konga docker startup happen error

Hi all,
When I startup konga container, there has error happened as below:

TypeError: Cannot read property 'collections' of undefined.
Could you help to check why cause this error.
Thanks.

envirment as below:
OS : CentOS 7.9 x64
Docker : Docker version 20.10.17, build 100c701
Postgresql : postgresql-12
kong: kong-3.0.1.x64

konga yml file:

check db and account is OK

error as below:
docker ps -a
docker logs a2d64ea3dec3
(1) start konga

(2) check konga log error

This project is almost abandoned.