Specify download location

Hi @gjeunen, I'm getting started with crabs (db_download), and I wonder if there is an easy way to point the file downloads to a specific location? I'm working on my personal laptop (Mac), but I have an external SSD where I'd like the files to go. I'm worried about not having enough space on my computer for the downloads. In the same general vein - are there any particular computational/storage hazards I should be aware of while running crabs? Are all modules manageable with a personal laptop? For context, I'm looking to set up COI, 16S, and 18S reference databases for eukaryotes. Any suggestions would be much appreciated - thanks for such a great tool!

Hi Mary,
Thanks for your interest in Crabs! There are a couple of things you could do. I think for downloading the NCBI etc. databases, you should be able to add a path to the --output parameter (e.g. --output /Volumes/ssd/path/ncbi.fasta). However, for downloading the taxonomy files (crabs db_download --source taxonomy) I am not sure if this is possible. Gert-Jan should be able to answer that. Another way to do this is to move into the external hard drive on your terminal. On macs, these are found in the /Volumes folder. So you could run `ls /Volumes' and you should see your hard drive.
One tip to quickly find file paths on Macs is to find the folder you want in the Finder window. You can then drag that folder to a terminal and the path to the folder should appear in the terminal prompt. You can select and copy that and paste it in to your command.
I am in the process of moving to a new Mac, so need to install Crabs on their, but I will do that soon and can test it if you are still having issues. Some of the database files can be quite large so this is an important question. Thanks
Cheers,
Hugh

Thanks, Hugh! Unfortunately it didn't work when added a path to the --output parameter, but it did work by just switching my working directory to the external drive folder where I wanted the files. I only tried with a small test dataset for ncbi download, but I'll keep going and see what happens doing it this way!

Good to know. Thanks. We will look into expanding some of these functions to future versions.
When you are running your pipeline to use the Crabs database you are building, you should be able to just input the path to the reference on the SSD. I have done some testing with Crabs on a server where the reference files were on another drive, but let me know if you run into issues. Getting to know how to use paths and linking all your data together can be one of the trickiest things when starting out.
I will leave this issue open for now, as Gert-Jan may like to weigh in. Keep us posted
cheers

Hi again, just coming back with a few db download issues that I haven't been able to resolve. Let me know if you can help! I did successfully grab seq data from BOLD and NCBI for 16S and 18S eukaryotes, but I'm having the following issues with NCBI/COI, MitoFish, EMBL, and taxonomy:

I don't have the terminal output to copy/paste but my NCBI download query for eukaryote COI (below) found >4mil sequences, but only wrote ~800,000 to the output files. No error returned and the batch count looked right (over 40 fasta output files), but I'm wondering about this mismatch in seq # when the smaller NCBI queries (16s and 18s) wrote all sequences found. crabs db_download --source ncbi --database nucleotide --query '("CO1"[GENE] OR "COI"[GENE] OR "COX1"[GENE] OR "COXI"[GENE]) AND Eukaryota[Organism] AND mitochondrion[filter]' --output COI_ncbi.fasta --keep_original yes --email<EMAIL_ADDRESS>--batchsize 100000

Mitofish query returned this:
crabs db_download --source mitofish --output mitofish.fasta --keep_original

downloading sequences from the MitoFish database
dyld[17353]: missing symbol called
Traceback (most recent call last):
  File "/opt/anaconda3/envs/crabs/bin/crabs", line 1372, in <module>
    main()
  File "/opt/anaconda3/envs/crabs/bin/crabs", line 1369, in main
    args.func(args)
  File "/opt/anaconda3/envs/crabs/bin/crabs", line 96, in db_download
    dl_file = mitofish_download(url)
  File "/opt/anaconda3/envs/CRABS/lib/python3.6/site-packages/function/module_db_download.py", line 139, in mitofish_download
    os.remove('complete_partial_mitogenomes.zip')
FileNotFoundError: [Errno 2] No such file or directory: 'complete_partial_mitogenomes.zip'

Several EMBL queries returned this:
crabs db_download --source embl --database 'inv*' --output embl_inv.fasta --keep_original yes

downloading sequences from EMBL
dyld[17340]: missing symbol called
Combining all EMBL downloaded fasta files...
formatting intermediary file to CRABS format
0it [00:00, ?it/s]
found 0 sequences with incorrect accession format
written 0 sequences to embl_inv.fasta

taxonomy grab returned this:
crabs db_download --source taxonomy

downloading taxonomy information
dyld[17346]: missing symbol called
unzipping nucl_gb.accession2taxid.gz...
gunzip: can't stat: nucl_gb.accession2taxid.gz (nucl_gb.accession2taxid.gz.gz): No such file or directory
dyld[17348]: missing symbol called
removing intermediary files

Traceback (most recent call last):
  File "/opt/anaconda3/envs/crabs/bin/crabs", line 1372, in <module>
    main()
  File "/opt/anaconda3/envs/crabs/bin/crabs", line 1369, in main
    args.func(args)
  File "/opt/anaconda3/envs/crabs/bin/crabs", line 66, in db_download
    os.remove(file)
FileNotFoundError: [Errno 2] No such file or directory: 'citations.dmp'

Hi Mary,
These issues all share in common some missing libraries (dyld). I am not sure, but this may be a problem with newer macs with newer M1 processors. Can you tell me what kind of mac and processor you have?

Hi Hugh, thanks for getting back to me. I'm working on a 2017 Macbook Pro with a 2.3 GHz Dual-Core Intel Core i5 processor and OS Monterey 12.3.1.

Hi Mary, Your computer should not be a problem. I am using the same OS and have not seen that issue. I am sorry, I am not sure what is wrong. It may be related to the conda environment or a larger system issue with your mac. you could try installing the conda env again, or doing the custom install option. We are working on improving both of those options. I will keep looking into it and see if I can find a solution.
cheers,
hugh

Thanks, Hugh. I'll play around and see if I can figure it out. Any ideas about the potential issue with the NCBI COI query (#1 above)? The NCBI db_download seemed to work just fine for all 16S and 18S eukaryote seqs found (albeit with much fewer seqs), but the crabs output indicated only ~800,000 out of >4mil 'found' COI seqs were successfully downloaded.
I guess I could try moving forward with only NCBI+BOLD downloads, but I would still have I problem if I can't get the taxonomy download to work.

Hi Mary,
I had a little more time today to dig a little deeper into your issues. First off for the NCBI download, I ran the same command and it seems to be working (still going, as it is a lot of sequences). One thing I noticed is that checking the temp files, there are only 10,000 sequences in each, not 100,000. I am not sure why, but it might be good to lower the batchsize. You can probably find guidelines on the NCBI website. Though, you have to check the number of sequences in each temp file and not just the number of files.
One thing that may be happening is that the request is just timing out. NCBI puts limits on how much data any one user can download, and depending on the internet connection, large demands may be curtailed, often without any notice or warning. Try running it overnight when there is less traffic.
I would suggest curtailing your search parameters. Four million sequences is a lot to download at once. Do you really need all Eukaryotes? Your EMBL search was for only invertebrates. Just changing the Eukaryota[Organism] to Insecta[Organism] brought it down to 2.7 million. Still a lot, but might help. Lowering the batch size and constricting the parameters may work. Try running it again and see if it helps.
As for the other download commands, they all share use of the wget tool, which is installed with Crabs. We have had  issues between Linux and Mac versions of wget, but we have not seen any issues yet within Mac. In the crabs environment, try running wget -h and check the version number at the top of the output. It should be 1.20.3. You could try to reinstall this within the Crabs environment: conda install wget --force-reinstall. This will install a new version of wget and perhaps also any missing libraries. It may not help, but is worth a shot and should not affect anything else.
That is all I can think of for now. Let me know how it goes and I will keep the thinking cap on.
cheers,

Following up with progress on these issues:
Reducing the ncbi query batch size to 10000 helped recover more COI sequences - looks like I was able to get 3.8 out of 4.1M while running it overnight. Still not sure why I can't get 100% seq recovery, but maybe a time-out issue like you
My wget version was correct, but unfortunately a force reinstall in crabs still did not resolve the previous missing symbol errors during db_download. Tried manual installation of crabs according to the github instructions - that did work! I was able to complete the remaining db_downloads.
I tried moving on to the insilicopcr step 4 with the mitofish sequence subset and got stuck again. Looks like a problem with locating cutadapt, even though I have it installed already:
crabs insilico_pcr --input mitofish.fasta --output mitofish1.fasta --fwd GGWACWRGWTGRACWITITAYCCYCC --rev TAIACYTCIGGRTGICCRAARAAYCA --error 4.5
reading mitofish.fasta into memory
100%|██████████████████████| 686140221/686140221 [00:01<00:00, 386896578.11it/s]
found 751440 sequences in mitofish.fasta
running in silico PCR on fasta file containing 751440 sequences
Traceback (most recent call last):
  File "/Users/marymcelroy/reference_database_creator/crabs", line 1372, in <module>
    main()
  File "/Users/marymcelroy/reference_database_creator/crabs", line 1369, in main
    args.func(args)
  File "/Users/marymcelroy/reference_database_creator/crabs", line 230, in insilico_pcr
    sp.call(cmnd_cutadapt_1)
  File "/opt/anaconda3/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/opt/anaconda3/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/anaconda3/lib/python3.8/subprocess.py", line 1706, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cutadapt'

Some troubleshooting looking into cutadapt:
python3 -m pip install --user --upgrade cutadapt
Requirement already satisfied: cutadapt in ./.local/lib/python3.8/site-packages (4.1)
Requirement already satisfied: xopen>=1.1 in ./.local/lib/python3.8/site-packages (from cutadapt) (1.6.0)
Requirement already satisfied: dnaio>=0.7.1 in ./.local/lib/python3.8/site-packages (from cutadapt) (0.9.1)
Requirement already satisfied: isal>=1.0.0 in ./.local/lib/python3.8/site-packages (from xopen>=1.1->cutadapt) (1.0.1)

cutadapt --version
zsh: /usr/local/bin/cutadapt: bad interpreter: /usr/bin/python: no such file or directory

cutadapt -h
zsh: /usr/local/bin/cutadapt: bad interpreter: /usr/bin/python: no such file or directory

Thanks for the update Mary.
Because we had you follow the pip install of Cutadapt instead of the conda install (to avoid any potential issues with conda), you have to add the path to cutadapt for Crabs (and your system) to find it. I checked this on my system, and it worked. So add this line to your .bash_profile file (or .zshrc if you are using zsh shell).
export PATH="/Users/<username>/.local/bin:$PATH"

This is the same process you went through to add the path to Crabs to your system. In the above line, change  to your own username. If in doubt, go to your home directory and check where you are (cd; pwd should do it). Remember to be careful modifying your .bash_profile or .zshrc files.
More information can be found on the Cutadapt online guide.
For everyone: if further problems occur with the conda install option, we will put more explicit instructions for the manual install, and start recommending that one instead.

Hello Mary,
With regards to batch size downloading and the missing sequences, this is due to NCBI timing out. NCBI prefer a limit of 5000 sequences to be downloaded each time, the reason we set this value as the default. Could you please try the db_download function with a --batchsize value of 5000 or less?
Please let me know if you encounter any issues.
Best regards,
Gert-Jan

Thanks, Hugh and GJ! I added the cutadapt path to my .zshrc file and that resolved the error. I'll also re-try the NCBI downloads with batchsize=5000. Testing with the mitofish sequence data for steps 1, 4-8.2 all seem be working great now.
Thanks for your patience and guidance so far!

Hi again - I've got most features working now, so that's great! I wanted to ask about a couple of new things:

I was experiencing timeout problems with crabs+bold download, so I used a workaround for acquiring bold seq records that aren't mined from GenBank by manually downloading fasta files from their website with more complex queries than crabs could accommodate. I then used db_import without any issues:
crabs db_import --input bold.fasta --output bold_crabs.fasta --seq_header accession --delim '|'
However, the subsequent insilico pcr and assign_tax features are not working, possibly because bold uses process ids and not accession numbers? It seems like maybe there's a formatting issue. I also tried the above code with 'species' instead of 'accession' and same problem. After db_import, the fasta record headings are formatted like >CRABS:CQCS058-10

crabs insilico_pcr --input bold_crabs.fasta --output bold_pcr.fasta --fwd GGWACWRGWTGRACWITITAYCCYCC --rev TAIACYTCIGGRTGICCRAARAAYCA --error 2
Fatal error: Illegal character '-' in sequence on line 6 of FASTA file
running in silico PCR on 50368 reverse complemented untrimmed sequences
counting the number of sequences found by in silico PCR
0it [00:00, ?it/s]
found primers in 0 sequences

crabs assign_tax --input bold_crabs.fasta --output bold_tax.tsv --acc2tax nucl_gb.accession2taxid --taxid nodes.dmp --name names.dmp
assigning a tax ID number to 40118 accession numbers from bold_crabs.fasta
did not find a tax ID found for 40118 entries, most likely due to spelling mistakes.
did not find 0 accession numbers in nucl_gb.accession2taxid
0 accession numbers resulted in 0 unique tax ID numbers
generating taxonomic lineages for 0 tax ID numbers
assigning a taxonomic lineage to 40118 accession numbers
written 0 entries to bold_tax.tsv

I tried the db_completeness feature. It looked like the code was running successfully but it never produced a visualization? No errors either. This is the code I used:
crabs visualization --method db_completeness --input COIsyn_geocurate.tsv --species CAsynonyms.txt --taxid nodes.dmp --name names.dmp

Hi @hughcross and @gjeunen - do either of you have any thoughts about these new issues I'm experiencing (described above)? Thanks!

Hi Mary,
It sounds like the bold fasta file just needs some cleaning up before further processing. For the illegal character, a short script or one-liner can probably clean these up. The '-' character usually means a gap in an alignment file. Is that what you have? These can be taken out. Other illegal characters can be replaced with an 'N'. Try outputting the first 20 lines or so of the file and paste it in the response:
head -n 20 bold_crabs.fasta > bold_example.fasta

For the second issue, we have to look at the sequence headers and see if they are not NCBI- compatible or annotated in some way that the assign_tax does not recognize. The example as I suggest above might show us what we need.
We should probably start new issues for these new problems and close this one. This will help other people find them. For now, though, let's solve the current dilemmas.

Thanks Mary. It is clear now. The gaps can be fixed with a few lines of code, or a clever sed or awk line. I will add that here soon.
For the taxonomy assignment, that is more of a problem with the manual BOLD download. The sequences are named with BOLD accession numbers, and as the Crabs assign_tax tool uses NCBI accession numbers to match, that won't work. You can query the BOLD website to get the taxonomy, but I don't know how to do that for a bunch of sequences. They may have a tool for that. I will try to look more into it.

Hello Mary,
Apologies for the silence, it's been a bit hectic lately. Please find my responses below for the two queries you have posted below. If I have missed something, please let me know. Also, might I please ask that if you have queries for a different topic, as being discussed in this thread, that you place them as separate issues? Currently this thread is named "Specify download location", but none of the topics discussed below are dealing with this issue. Using separate issue threads will make it easier for people to find potential answers if they come across a similar issue.

With regards to the BOLD formatting. The issue here is that the CRABS db_import function is a very simple line of code that can only handle something in the format of:

>species name
ACGT
>acc_number
ACGT

The db_import function doesn't have the capability for people to download BOLD sequences outside of CRABS and then use CRABS db_import to format the data appropriately. Please give me a some time to write the code to add in this functionality into the db_import function.

With regards to the db_completeness function, you will need to specify an output file name with ".txt" extension using the -o or --output parameter.

Hello Mary,
You should now be able to import your BOLD sequences into CRABS using the following command:
crabs db_import -i filename.fasta -o outfile.fasta -s BOLD

This should now give you the correct CRABS format to be used in the remainder of the pipeline. Please remember to update CRABS to the latest version.

Hi Gert-Jan, thank you so much for your help. My BOLD seqs are working with the pipeline now.
My apologies for using the same thread - I edited the issue title to be more informative so hopefully it will still be useful for people seeking help on similar issues!

I will close this issue now as this has covered a lot. For anyone having similar issues, please open a new issue


Do I have to maintain a git clone of youtube-dl for latest version?

Or does youtube-dl-gui download the most recent version?

Hi @ThebestwaytodesignJoinGitHub
youtube-dl-gui uses the latest binary from https://yt-dl.org/latest/.
You can also read this issue https://github.com/MrS0m30n3/youtube-dl-gui/issues/28
which refers to the gap between releases.
Let me point that if for some reason you want to use your own repository you need to edit the code at https://github.com/MrS0m30n3/youtube-dl-gui/blob/master/youtube_dl_gui/updatemanager.py#L48 to point to your own binary or you can build the binary locally and change the 'youtubedl-path' unders settings.json file to point to your own binary.


Usage with Scala version 2.13

When running with Scala 2.13 it turns up with java.lang.ClassNotFoundException: scala.Serializable using monix shade version "io.monix" % "shade_2.12" % "1.10.0".
But works if we use with Scala version 2.12.
Do we have a way to use with Scala 2.13 or this library to be compiled to 2.13?

This'll add support for Scala 2.13 - https://github.com/monix/shade/pull/67


Add fallback for construction points using a Collection type

At runtime, when an autowired member needs a Collection or Map and no such beans exist in the context, an empty collection is injected, see https://github.com/spring-projects/spring-framework/blob/3c2dfebf4ec5e9d791c1f3c9fa0ad35a5a9fcd6b/spring-beans/src/main/java/org/springframework/beans/factory/support/ConstructorResolver.java#L893-L907
This isn't part of the standard resolution algorithm so we need to apply the same in our resolver

Closed by ac225a6dd16e03bb


Got error while runing python -m cc_net -l my -l gu

Following example at https://github.com/togethercomputer/RedPajama-Data/tree/main/data_prep/cc/cc_net#pipeline-overview but got following error. Did I forget to run any preparation?
(racoon) t@medu:~/repos/NAM/red-pajama/data_prep/cc/cc_net$ python -m cc_net -l my -l gu
usage: __main__.py [-h] [-c CONFIG_NAME] [-d DUMP] [-o OUTPUT_DIR] [-m MINED_DIR] [-e EXECUTION] [-n NUM_SHARDS] [--min_shard MIN_SHARD]
                   [--num_segments_per_shard NUM_SEGMENTS_PER_SHARD] [--metadata METADATA] [--min_len MIN_LEN] [--hash_in_mem HASH_IN_MEM]
                   [-l LANG_WHITELIST] [--lang_blacklist LANG_BLACKLIST] [--lang_threshold LANG_THRESHOLD] [-k KEEP_BUCKET]
                   [--lm_dir LM_DIR] [--cutoff CUTOFF] [--lm_languages LM_LANGUAGES] [--mine_num_processes MINE_NUM_PROCESSES]
                   [-t TARGET_SIZE] [--cleanup_after_regroup] [--task_parallelism TASK_PARALLELISM] [-p PIPELINE]
                   [--experiments EXPERIMENTS] [--cache_dir CACHE_DIR] [--config CONFIG]
__main__.py: error: argument -l/--lang_whitelist: invalid Sequence value: 'my'

Hi @tiendung, are you trying to run the pipeline on non-english languages ("my" and "gu")?
If you're goal is to create the english cc slice of the RP dataset, you can follow the steps in the readme in data_prep/cc: https://github.com/togethercomputer/RedPajama-Data/tree/main/data_prep/cc

awesome that you're working on more languages!
We haven't run the ccnet pipeline on non-english data as of yet, so we haven't run into this error error previously -- I will look into it and try to repduce the error you got.
Can you show me the steps you have run so far?

I believe I got the same error trying to provide -l en.  So I removed the language parameter and was able to proceed.

Did you run make lang=my dl_lm prior to running the cc_net pipeline? This is supposed to download the lm used in the ccnet paper.
Also, let me know which steps you ran so that I can try to reproduce your error.

I got same error when running below command. I think @danielpclark sum it up here https://github.com/togethercomputer/RedPajama-Data/issues/23#issuecomment-1520547829
make lang=en dl_lm
python -m cc_net -l en

I did some more digging on this, and I could reproduce your error with python versions > 3.8. Can you try if downgrading to 3.8 solves the issue for you? We ran the pipeline with 3.8 and did not run into this error.

Yes, it works when I used python 3.7.11.  Thanks a lot!


Port ctr to use client package

This ports over ctr to use the client package for running containers.

Codecov Report

Merging #962 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #962   +/-   ##
=======================================
  Coverage   59.55%   59.55%
=======================================
  Files           5        5
  Lines         759      759
=======================================
  Hits          452      452
  Misses        199      199
  Partials      108      108

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25cc761...75ce408. Read the comment docs.

LGTM

Note: this is probably breaking windows, going to open an issue for it :)

@mlaventure it makes some things better


Issue while Loading trained model

I trained model for a while & cancelled training
Now i'm trying it test. i'm getting below issue. kindly please help. thanks in advance
Command:
!python test.py --checkpoint saved/models/PICK_Default/test_1012_064804/model_best.pth
--boxes_transcripts {out_box_path}
--images_path {out_img_path} --output_folder /content/output/
--gpu 0 --batch_size 2
Error:
Loading checkpoint: saved/models/PICK_Default/test_1012_064804/model_best.pth
with saved mEF 0.0000 ...
RuntimeError: Error(s) in loading state_dict for PICKModel:
size mismatch for decoder.bilstm_layer.mlp.mlp.0.weight: copying a param with shape torch.Size([49, 1024]) from checkpoint, the shape in current model is torch.Size([11, 1024]).
size mismatch for decoder.bilstm_layer.mlp.mlp.0.bias: copying a param with shape torch.Size([49]) from checkpoint, the shape in current model is torch.Size([11]).
size mismatch for decoder.crf_layer.transitions: copying a param with shape torch.Size([49, 49]) from checkpoint, the shape in current model is torch.Size([11, 11]).
size mismatch for decoder.crf_layer._constraint_mask: copying a param with shape torch.Size([51, 51]) from checkpoint, the shape in current model is torch.Size([13, 13]).
size mismatch for decoder.crf_layer.start_transitions: copying a param with shape torch.Size([49]) from checkpoint, the shape in current model is torch.Size([11]).
size mismatch for decoder.crf_layer.end_transitions: copying a param with shape torch.Size([49]) from checkpoint, the shape in current model is torch.Size([11]).

Hi, @karthikesh2020 , please check your config file. The numbers of classes for train and test are likely to be different.

Hi, @karthikesh2020 , please check your config file. The numbers of classes for train and test are likely to be different.

Hi, @karthikesh2020 , please check your config file. The numbers of classes for train and test are likely to be different.

Thank you so much @tengerye, It solved the problem.


2 commits

First commit solves the crash when Mycroft-core config is not found, which I think it should be fixed in ovos_utils and not here.
Second commit replaces memory_tempfile with tempfile as the first one is only compatible with Linux systems.
Hopefully ovos_utils gets updated to fix the Mycroft-core config problem and we can discard the first commit.

it's desired that ovos_utils complains if mycroft config is missing, after all the whole purpose of the method is to read it. It's up to whatever is calling the method to handle a missing config, the main utils requires it to exist and should throw an error if it doesn't. So that fix is in the correct place and looks good

might need another rebase to solve the merge conflicts

thanks for all your PRs, keep them coming :muscle:

I think you accepted the pull before I finished rebasing and squashing, but everything seems to work just fine.

i squashed when merging, no worries there :)


Finish MVP for mid-term presentation

We assert that input is correct (which is signed by private key).

[ ] #96
[ ] #97
[ ] #95

Closed and tracked in smaller tasks in #95, #96 & #97


Support for converting manifests to in-toto statements

Description
Tracking bug for manifest to in-toto statement conversion.
In order to sign and store model signatures in a sigstore bundle the manifest must be converted to an in-toto statement.

Included in my draft work, will send a draft PR

I've had some thoughts about the intoto format. In https://github.com/sigstore/model-transparency/issues/111, we expressed concern over listing each file as intoto subjects, because tooling like cosign can be used to (wrongly) verify only a subset of subjects. Here's an idea that may be able to address this concern:

We list the (file, (custom?) hash) inside the predicate, not the intoto subject
We output a single subject in the intoto subject list by serializing the list of (file, hash), like we currently do in our existing PoC. We need not serialize json (which requires canonicalization), but a simple format like base64(path).hash-value as shown here. Other metadata for each file need not be serialized but are still protected by the manifest signature. Serialization schemes may vary, so we'll need a way to convey this as well. There are 2 ways to achieve this:
a. In subject, by changing the hash scheme scheme-sha256.  This is a bit harder to support with sigstore-python](https://github.com/sigstore/sigstore-python/issues/1018), unless we build the protobuf ourselves.
b. We add the scheme type under the predicate. I lean towards this solution.

So overall it would look like the following:
{
  "_type": "https://in-toto.io/Statement/v0.1",
  "predicateType": "https://something/model-signing/v1",
  "subject": [
    {
      "digest": {
        "sha256": "blabla"
      }
    },
  ],
  "predicate": {
     "serialization": "some-name/v1",
     "files": {
           {
              "path": "to/file1",
              "digest": {
                 "sha256": "bla",
              }
           },
          {
              "path": "to/file2",
              "digest": {
                 "custom": "bla",
              }
           },
      },
   }
}

I agree with you. It only feels like "abusing" the format which is mostly because sigstore has hard coded the acceptable hashes in their code base.
Perhaps it would be the best to go down the route of our own format to avoid additional headache.
So we can serialize the manifest to an on disk format and utilize sigstore's hashed record. We then store the bundle and manifest together and be done with it.


about version

what is the version of django supported by chatterbot?
thanks.

Versions 1.8, 1.9, 1.10, and 1.11 are currently supported. Support for Django 2.0 has been planned but is not yet available.
Using Django 1.11 is recommended.


LuckPerms-Bukkit-4.3.49 and OpenJDK 11.0.1

Hi  @lucko - I'm in the process of switching out my Oracle JDK for the Oracle OpenJDK on my servers.  Upon startup, I'm only getting an exception with LuckPerms.  I still have other tests to run, but it would be great if you can add support for this especially due to Oracle's license changes that go into effect in 2019.  I expect there will be a big push for server owners to get on OpenJDK.  - Thanks, Steve
System Info: Java 11 (OpenJDK 64-Bit Server VM 11.0.1+13) Host:  Windows 10 10.0 (amd64)
Loading libraries, please wait...
[14:59:50 INFO]: Loaded 0 recipes
[14:59:50 INFO]: Starting minecraft server version 1.13.2
[14:59:50 INFO]: Loading properties
[14:59:51 INFO]: Default game type: SURVIVAL
[14:59:51 INFO]: This server is running Paper version git-Paper-481 (MC: 1.13.2) (Implementing API version 1.13.2-R0.1-SNAPSHOT)
[14:59:52 INFO]: Debug logging is disabled
[14:59:52 INFO]: Server Ping Player Sample Count: 12
[14:59:52 INFO]: Using 4 threads for Netty based IO
[14:59:52 INFO]: Generating keypair
[14:59:52 INFO]: Starting Minecraft server on *:25565
[14:59:52 INFO]: Using default channel type
[14:59:58 INFO]: Set PluginClassLoader as parallel capable
> WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by me.lucko.luckperms.common.dependencies.classloader.ReflectionClassLoader (file:/D:/MC-Aquatic/plugins/LuckPerms-Bukkit-4.3.49.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of me.lucko.luckperms.common.dependencies.classloader.ReflectionClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

See #952 and #1084


Always add trailing .0 in Float#inspect; ref #4225

Trailing .0 is removed from Float#to_s and Float#inspect at
9d08025b. However, I think the more human-readable format is better
for Float#inspect.
For example, in the Float#to_s format, the failure message is not
well understood when testing values including types by eql? (e.g.
Numeric#step test.
assert "example" do
  exp = 1.0
  act = 1
  assert_operator(exp, :eql?, act)  #=> Expected 1 to be eql? 1.
end

This PR is merged to mryby3 branch (will be pushed soon).


Bug 1989723: Improve ensure methods

Problem: When calling a registry's ensure methods, if the resource is
deleted between the get and delete calls the ensure method fails due
to a 404 error.
Solution: When deleting an object, ignore 404 errors.
Signed-off-by: Alexander Greene<EMAIL_ADDRESS>Upstream-repository: operator-lifecycle-manager
Upstream-commit: 1b5afdceabb7a130f3f3d923b9784c007f9f4f2e

It looks like you need to run make vendor locally and commit the results.

You need to run make vendor and add those changes.

Committed the vendor changes @dinhxuanvu and @timflannagan

/retest

/bugzilla-refresh
/retest

The team has decided that this bug is not sever enough to warrant backporting, closing this PR.


Jasmine never getting called

Been trying to debug a very strange issue.  I'm loading gulp-jasmine like normal:
gulp.src([
  './build/common/**/tests/**/*.js',
  './build/server/**/tests/**/*.js'
])
.pipe(jasmine())

When I use gulp-debug, I see 16 files are loaded.  However, jasmine is never called.  I had added console.log statements into jasmine's Jasmine.prototype.execute function to verify.  No error messages, no exit codes.
However, if I delete 5 files (any 5, it doesn't matter which), then Jasmine executes fine.  I have no idea what to make of this.
I've tried adding the following line to gulp-jasmine, and the tests execute fine, so I know it's not an issue with the tests themselves:
var resolvedPath = path.resolve(file.path);
var modId = require.resolve(resolvedPath);
deleteRequireCache(modId);

jasmine.addSpecFile(resolvedPath);

// HACK BELOW
if(jasmine.specFiles.length === 16) {
  jasmine.execute();
}

cb(null, file);

But for some reason, without this hack, the test don't run.  Any ideas?

@KurtPreston Hi,
I installed the latest versions of both gulp and gulp-jasmine on the latest lts node version, created 16 test files, and ran gulp-jasmine and the specs ran fine.
My best guess would be that you are not returning your gulp stream, and not calling the async callback. You haven't posted your full gulp task, but if you try the following, does it work:
const gulp = require('gulp')
const jasmine = require('gulp-jasmine')

gulp.task('default', () => // no brackets here, so we are returning the stream.
  gulp.src([
    './build/common/**/tests/**/*.js',
    './build/server/**/tests/**/*.js'
  ]).pipe(jasmine())
)

@jbblanchet Oh wow, thanks for the incredibly quick response!  You're 100% right, I was missing a return statement on my task.  Returning the gulp stream fixed the issue.
Also wanted to document the issue stemmed from gulpjs/gulp#716


Manually set current session credentials.

Do you want to request a feature or report a bug?
Support Request / Feature Request
What is the current behavior?
I'm using Nativescript, and session is cleared after app refresh. I get no user from Auth.currentSession()
However I store the jwt from login and can access endpoints manually by passing that as header.
I would like to use this to make Amplify believe that a user is logged in to use it in other parts of the application.
Is there a way to manually set credentials to use all the functionality of Amplify?
And how is successful login handled? Can that behaviour after successful login be emulated to set current session credentials manually?

Looking for a solution. Auto Login after APP relaunch, so they have no need to manually sign in again n again (until they sign out manually). Just like any other APP, AMAZON etc.

@aaayushsingh we don't have official support for NS yet. The way the persistence is set is with storage on a PER platform basis. So on react native it uses AsyncStorage, in browser local storage etc. Similarly some bridge would need to be provided to NS to use to store the session and the library would need modifications to support it. Looks like there is a module here:
https://www.npmjs.com/package/nativescript-sqlite
and would need to be implemented somewhere in cognito sdk similar to:
https://github.com/aws/aws-amplify/blob/master/packages/amazon-cognito-identity-js/src/CookieStorage.js

tracking support feature request in #1035


Added google sheet feature and moved features to the export drop-down

Signed-off-by: Agha Saad Fraz<EMAIL_ADDRESS>FIX #2453
I have added the feature of exporting the OpenRefine project to google sheets, and have moved the export features out of dialogs and placed into the export menu.

@wetneb I have made the suggested changes. Let me know if there is something needs to be done.


logout_worlds.yml grows to a very large size

On my server, logout_worlds.yml contains data for 5563 players and is 437k bytes in size. What purpose does this file serve?
It appears that it's only used if a player joins the server and his old world no longer exists, correct? And that's so his inventory can be adjusted? e.g., so he doesn't join a survival world with his old creative inventory?
It appears the file is saved every 2 minutes. Is that save done synchronously, or asynchronously? Saving such a large file probably takes a long time.
I'd like to come up with a better way to handle this.

logout_worlds.yml contains data for 5563 players and is 437k bytes in size.

I'm seeing times of 150 msec to save the file. That's 3 whole ticks. (I expected it to be worse!)
There are ways to do it asynchronously. The current async save is unsafe. There are ways to make it safe, but that doesn't solve the problem of having a half-megabyte file.
See my experience with a similar problem here - https://www.spigotmc.org/threads/read-write-the-same-file-in-events.539624/#post-4330472
The best way to modify xInventories is probably to write out individual .yml files for each UUID. Or making it into an SQLite database. I'm not a fan of databases that grow forever.

The best way to modify xInventories is probably to write out individual .yml files for each UUID.

OK, so I have this working. It even converts logout_worlds.yml to individual uuid.yml files. I just need to test it more thoroughly.


Can't get command palette to open when , , or  have focus

I couldn't figure out how to get cmd+k or other keyboard shortcuts to work when these elements had the focus within ninja-keys.
However, I was able to get around this by importing the hotkeys-js library myself and following the guidance in their docs (see https://github.com/jaywcjlove/hotkeys#filter)
  import "ninja-keys"
  import hotkeys from "hotkeys-js"
  hotkeys.filter = _event => true

How exactly you implement this for your project is going to depend on what framework you're using, but this might get you close.
Ideally, this would be exposed as an attribute off of ninja-keys or a configuration option, but I wanted to create this issue in case anyone else ran into it.

Also, be aware that if you do this AND some of your hotkeys don't have a modifier key such as command, ctrl, or option you could be in for a world of hurt and bad UX. YMMV


Shared Workers and Storage Partitioning

Hello,
To my understanding, when Chrome releases Storage Partitioning, this will impact how Shared Workers work. I have a use case where an user may embed an app into their own sites. Now with Storage Partitioning, there would be a shared worker initialized for each top-level site that has the app embedded.
As an example, an user embeds chat.com, who uses Shared Workers to maintain a session, to two of their sites site-a.com and site-b.com. Before Storage Partitioning, there would be only one Shared Worker managing both sites if both are opened. However after Storage Partitioning, a Shared Worker would get spun up for the chat.com in site-a and the chat.com in site-b. This will lead to the two sites being out of sync.
Is there a proposal/ solution for this use case?

cc @adamziel

@kevinkiklee

Hi @andywang219, there's no solution for this shipping in Chrome as of today, but we're discussing options to extend the Storage Access API (aiming to ship in Chrome soon) with support for "storage", which could include Shared Workers, see https://github.com/privacycg/storage-access/issues/102. I don't think we've explicitly discussed the Shared Workers case yet, but off-hand it seems feasible to me. I'll check with the group in that issue.
This would require you to call requestStorageAccess() (which would show a user permission prompt) in both iframes to connect the two sites in line with the privacy goal of preventing tracking between site-a.com and site-b.com.
cc @arichiv

cc @rowan-m

Thank you for the response @johannhof! Do you think First Party sets would be a viable solution since FPS define which top-level sites can share resources (or is that just for cookies)?

So, no, FPS would not allow Shared Workers by itself, but since SAA integrates with FPS your users would not see a prompt from SAA if both site-a.com and site-b.com are in the same FPS. You'd still be dependent on Shared Workers support in SAA though. Sorry that you're experiencing this feature gap right now, I hope we can close it sooner rather than later.

Got it! Excited to see the proposed solution when it is available!

cc @dmsnell and cross-linking to this comment about what the privacy sandbox means for service workers (not shared workers).

@adamziel if possible, would you mind sharing a summary of the comment for those of us who don't have a WP Slack account?

@johannhof I'm not sure what comment Adam meant to link to but that one is him announcing he'll be out for a few months on sabbatical.
Possibly he meant to link to his post in the WordPress Playground repo https://github.com/WordPress/wordpress-playground/issues/586

Hey @andywang219, we're still working on this and, to set expectations, this is more something in the order of months rather than weeks or days, even if everything goes according to plan. Making changes to Web APIs takes time :)
If you want to follow along, we're figuring out the details here: https://github.com/privacycg/storage-access/issues/102. Feedback is very welcome!

As far as the WordPress Playground goes I don't think there's any problem with Shared Workers, as having separate Service Workers per partition seems like it would be fine. Any Storage API access should be coming from the partitioned domain and I think it would make sense to isolate them: if someone were performing work on Site A that embed the Playground, we would expect different content on Site B that embed the same.
For example, an interactive tutorial site might store progress on a learning goal with user submissions. Another site might offer an environment to build WordPress plugins. Both sites are embedding the Playground from playground.wordpress.net, and each should maintain a separate storage partition.
Not sure if this comment is useful, but in summary, Shared Worker implications wouldn't impact it because it doesn't use those.

Is there any momentum on this? Last week, as noted elsewhere, Chrome rolled out storage partitioning to 100% of users, which has broken our WebRTC application, as there is now no way to establish a Shared Worker between two separate domains, across multiple windows.
This application is used by radio broadcasters (on a large number of subdomains), and consequently we have had to advise customers to use Firefox (who have built their cross-site isolation in a manner where this still works), or a downloadable Electron-wrapper that is locked to a lower version of Chromium.

https://github.com/privacycg/proposals/issues/41 is probably the issue to follow for this, which links to the Intent to Prototype on blink-dev.

@Jamie0 do you have any details on how/why this still works in Firefox? Otherwise, is there an easy way for me to test your feature?

Sadly I'm not too familiar with the details of how it's implemented in Firefox - from my observation, for most users it "just works" but, where it doesn't (often if someone has the privacy settings customised), Firefox gives users the option to "disable tracking protection" for a specific domain, which makes it work again.
At risk of hijacking this issue, I'll try and give a (brief as possible) run-down of how our app uses Shared Workers:

Radio stations have a unique subdomain on myriadanywhere.com (e.g. example.myriadanywhere.com)
User clicks a button in the menu to "Connect To Audio"
Application launches a pop-up window to a specific URL on live.broadcast.radio, and embeds an iframe to live.broadcast.radio. For a good reason I won't go in to, the first pop-up launches a second pop-up (using document.write to ensure there is immediately content) and closes the first/itself.
The WebRTC connection then takes place in the second pop-up window.
The pop-up and iframe both have a Shared Worker, and the iframe uses postMessage to act as a proxy for messages between the main application and the pop-up window.
The use case is so that the microphone controls inside the main application reflect the state from the pop-up window, without having to depend on a WS or some other network connection.

I would make an educated guess that it works in Firefox because it tracked the origin that the pop-up window was launched from - so the pop-up exists inside a silo for the 'top' domain as it sees it - example.myriadanywhere.com. I notice that - for example - audio device IDs will change depending on the top-level domain that opened live.broadcast.radio.
We've worked around this for customers simply by mirroring live.broadcast.radio on live.myriadanywhere.com. Give me a bit of time and I'll try and set up a specific instance running an older build where the issue is present. Do you have an email I can send some test credentials to?

Hi @Jamie0, thanks for that extra detail, I highly suspect that this is unbroken by Firefox's popup heuristics, which IIRC also affects storage.
Do you mind filing a new bug at https://goo.gle/report-3pc-broken? If your test credentials are not sensitive you could attach them in that bug (maybe there's also a way to make a hidden comment? not sure if that's available to external users), otherwise feel free to email them to me at johannhof at chromium dot org.

There's a proposed extension to the existing origin trial to add shared workers. It might be in Chrome M123.

Shared Workers were added in Chrome 123! This will be a part of the existing origin trial: https://developers.google.com/privacy-sandbox/blog/saa-non-cookie-storage


Questionnaire Definition Parser

As a platform administrator, I want to specify a set of questions which users can answer to automatically select a relevant set of test-cases. The questions and mapping logic will be defined in YAML for now - in future we may expand this to include a visual interface.
This issue is to define the data structure used, and covers the import work as far as defining the questionnaire type. The work of creating the interface for users to provide answers to the questions as part of session creation is covered in #293.
Sample Questionnaire Definition File
In the example below, I'm using Laravel's validation logic for preconditions. A list of preconditions is defined to be valid if all properties validate
name: Sample Questionnaire
----
questions:
    section:
	    name: "Section One"
	    description: "Some section description goes here"
	    questions:
	      - property: category
	        question: What category do you want?
	        type: select
	        values:
	          first: The first one
	          second: The second one
	      - property: subcategory-first
	        question: Which subcategory of "first" do you want?
	        preconditions:
	          category: "in:first"
	        values:
	          A: Subcategory A
	          B: Subcategory B
	      - property: sync
	        question: Synchronous or Asynchronous?
	        type: multiselect
	        values:
	          sync: Synchronous
	          async: Asynchronous

test_cases:
    A-async:
      subcategory-first: "in:A"
      sync: "in:async"
    B-async:
      subcategory-first: "in:B"
      sync: "in:async"
    B-sync:
      subcategory-first: "in:B"
      sync: "in:sync"
    second:
      category: "in:second"

Requirements

Questionnaire YAML will be comprised of two sections: questions and testcases.

The questions field is separated into multiple sections. The sections are used to separate logical parts of the questionnaire to improve user experience. Each section may optionally define a preconditions field, which behaves as for individual questions below.
questions defines a list of questions to be presented to a user. Each question is defined by:

property The name of a property which may be referred to in other parts of the document
preconditions A set of validation rules which must be true in order for the question to be displayed. This allows us to have a dynamic interface, where questions which are not relevant based on previous answers are hidden from the user.
type The type of the question. To start with, two options will be possible: select and select-multiple.
Other properties will depend on the selected type. For both select and select-multiple, two other properties are defined:

question A string representation of the question to show the user
values. Key-value pairs indicating the options that a user may choose. The key is used as a logical ID for the answer, while the value is displayed to the user.

test_cases defines a list of test cases which will be selected, depending on the users answers. This takes the form of a set of key-value pairs, where the key refers to the slug of a test case (see below), and the value is a set of validation rules, following the same logic as preconditions in the questions section.

Preconditions are used to represent arbitrary mapping logic from the user's answers to a set of test cases. They are represented as a set of key-value pairs, where the key corresponds to the property field of one of the other questions defined. The value needs to represent some kind of boolean predicate (if X then true else false) on the user's answer: for now, I suggest that we use Laravel validation rules for simplicity, although I am open to other suggestions too. We can expand the expressiveness of these preconditions in future development.
Test case data model should be augmented to add a slug field, which is defined in the test case YAML. Some existing test cases define a test_case field which is inconsistently used - it would be acceptable to use this instead of slug. We cannot use a test-case ID directly to refer to a test-case from a questionnaire, since this may be different from one deployment to another depending on the order of insertion of test cases. Instead, the slug field is enforced to be unique across all test-cases, and provides a lookup index which can be used in the questionnaire mapping logic.

hey Christopher @cjol, hope you can assist us with the next pack of questions:

A single quiestioonaire per system is enough? I.e. superadmin has an option to upload a Questionnaire file as YAML like a Test Case one time. The next upload will just rewrite it.
In case of multiple Questionnaires support, do we need to have a list of them on UI, select the active one or edit them via UI like Test Cases (#303 )
The type of the session (test or complience) selection can be taken on Session create button (example) or it should be an additional step on Session creation steps (example)?
What do you think if the Questionnaire to be the first form/section (example) after the Complience session selection and all the other steps to remain the same i.e. Select SUT, Session name and list of non-editable TestCases, and Configuration step?
Can you please specify why we need to use test_case or slug values instead on name that are already used on Test Cases YAML for Use Cases and Components? We guess it would be easier to find the name of the Test Case during the Questionnaire creation rather than slug that is not visible on UI.. But maybe we are missing something?
Thanks!

One questionnaire is enough. I can imagine a future where we will want multiple, but for now let's not worry about it.
I'd rather have it as a separate step, because we might want to include some text to explain to the user what the different session types are.
That sounds fine.
I'm happy to use the name but then name needs to be forced to be unique among all test cases (I don't think it currently is). This will have an impact on uploading new test-cases which have the same name as an existing one (my preference would be to append a version number to the new version if there is a conflict with an existing test case), and on editing test case names.

Thank you for your comments! @cjol
1-4 seems to be resolved.

Just a few thoughts on it.
Got your point re name. Right now all test cases from /seeds looks like unique, but, yes, in some time (especially when we have new version of MM API or Mojaloop API), a new Test Case called e.g. "Authorized Transaction by Payer FSP" may be required, so we have 2 TC with the same name, but assigned to a different Components.
Seems like some unique field like slug should help with it (I guess an example of it is authorized-transaction) and we'll need to add an additional validation on TC YAML upload that will check if a TC with the same slug exists and deny duplication.
Later on once we start #281 - TC versions will be added so admins will have an option to upload a newer version of the existing TC and on its single page admins will have an option to select an active version of TC. This way we'll have technically many versions of a single TC, but only one will be active, so no conflict for Questionnaire.
Please let me know if this makes sense?

Meantime, one more qiestion came up...
Can we change the format of adding preconditions from:

 preconditions:
	          category: "in:first"

to
preconditions:
   category:
      - first

?
This will be easier for the fronted that don't work with "in:first" format to generate next steps on fly.
Thanks!

5 and 6 both look fine to me!

Actually, thinking a little harder about 6:
For now, I think all validations will be of the form in:X but in the future we might like something more powerful (like gt:Z). To futureproof ourselves, I would prefer this format:
preconditions:
   category:
      - in: first

For the purposes of this task, I am happy for the frontend code to assume that all entries are of the form in: . Does that make sense?

Hi, Christopher @cjol
I think, this is a good format for us:
preconditions:
   category:
      in: first


Added login to google funcanality

**Title: Login signup with google
**Issue No.: #223
Close #223
https://github.com/user-attachments/assets/9d74c3e2-840e-45d7-833d-bb73b720d122
Checklist:

[X ] I have mentioned the issue number in my Pull Request.
[ X] I have commented my code, particularly in hard-to-understand areas.
[ X] I have created a helpful and easy-to-understand README.md if it's a new page/tech stack.
[ X] I have gone through the contributing.md file before contributing.

Additional context (Mandatory):
Are you contributing under any Open-source programme?

[ X] I'm a GSSOC-EXT contributor
[ X] I'm a HACKTOBERFEST contributor

Well done @Sawan-Kushwah


Could not accept connection when run both FIX server and FIX client

Hi @esanchezros,
When I run both FIX Client and FIX Server in the same app,
And whenever FIX client connection is not connected yet, FIXServer will never accept any request from client.
It should be separated thread, FIX Server does it job and FIX client does it own job.
Thanks
Sang.

Hi Sang,
Could you confirm what version you're using?
Thanks

I am using latest one I think, version 2.5.0

I just created an example project that uses client and server. It seems to be working okay
https://github.com/esanchezros/quickfixj-spring-boot-starter-examples/tree/add-client-and-sender-example/simple-client-and-server
Could you share your project so I can have a look?

Thanks you
little bit busy these days, I will try your sample and let you know about it

@duongvansang did you have the chance to test the sample project I added?
The spring boot starter creates separate beans for client and server, which instantiate separate instances for the Initiator and Acceptor. Both client and server should work together okay on the same application.


Update repository config

Signed-off-by: 1gtm<EMAIL_ADDRESS>

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 13 committers have signed the CLA.:white_check_mark: tamalsaha:white_check_mark: 1gtm:x: DirectXMan12:x: markusthoemmes:x: dvaldivia:x: vincepri:x: Hellcatlk:x: k8s-ci-robot:x: joelanford:x: erikgb:x: Porges:x: alvaroaleman:x: JoelSpeedYou have signed the CLA already but the status is still pending? Let us recheck it.


Update Grunt Peer Dependency

grunt 1.0.1 has been released.

Fixed by https://github.com/palantir/grunt-tslint/pull/46


Addition to styleguide

Add a sentence to the styleguide saying to use US English spelling.

i'lll take this one up.

@ksigler7 , Please have a look.


Fixes for React 15 warnings and hex hash problems

Fixes some of the issues called out by https://github.com/casesandberg/react-color/issues/114 and by https://github.com/casesandberg/react-color/issues/152:

New React warnings caused by unitless string style values have been solved by giving those styles explicit units.
Various pickers have been updated to expect that color state .hex will include a leading #, as specified in https://github.com/casesandberg/react-color/issues/51 and implemented in upstream/master but only shipped with the latest version.
Hex display is now covered by tests for pickers that show hex.
Behavior of the color utilities is now covered by tests.  Note, the tests reveal the behavior of some of these utilities (particularly simpleCheckForValidColor) to be unuseful and possibly incorrect.  These tests should be not be taken as spec for correct behavior.

As I'm fairly new to this package, I'd appreciate review from anyone interested, including @casesandberg, @kennedy0527, @jaeh and @vdh.  You should be able to check out this branch and use run npm run docs to check behavior of the various color picker types.

So is this ever going to be merged or should I just stick to the code-dot-org version?

would love to see this merged and published soon too! 😃

This is killer man, thanks for this 🙌


geneus: 2.2.6-0 in 'jade/distribution.yaml' [bloom]

Increasing version of package(s) in repository geneus to 2.2.6-0:

upstream repository: https://github.com/jsk-ros-pkg/geneus
release repository: https://github.com/tork-a/geneus-release.git
distro file: jade/distribution.yaml
bloom version: 0.5.26
previous version for package: 2.2.5-0

geneus
* [geneus_main.py] suppress warning (#52 <https://github.com/jsk-ros-pkg/geneus/issues/52>)
  * [geneus_main.py] do not write timestamp that makes different md5
  * [geneus_main.py] write depends by alphabetical order in manifest.l
* [.travis.yml] : add jade test  (#49 <https://github.com/jsk-ros-pkg/geneus/issues/49>)
  * .travis.yml: add BUILDER=catkin_make_isolated
  * .travis.yml: add indigo/jade and catkin_make/catkin test
* [geneus_main.py]: add comment : 152683d depends on comment line (#44 <https://github.com/jsk-ros-pkg/geneus/issues/44>)
* Contributors: Yuki Furuta, Kei Okada

this is small improvement and not urgent, so merge whenever possible

@wjwwood I'll leave this up to your discretion to merge. I just merged the kinetic version.

It's fine, I haven't made any EOL announcements yet.


Feat: Their should be a feature of total repository view and contributor

What feature?
This feature can help the contributor to know how many forks this repository have and also lt us know that how many contributor does this repository have..
I have added an screenshot for more clearlity..
Add screenshots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSoC'23 contributor
[X] I want to work on this issue

ig it's unnecessary


Keylime tenant cannot parse IPv6 url, when try to add agent

Environment

OS / version: f37
Processor architecture:  x86_64
TPM Manufacturer: SWTPM
Keylime version: Upstream for rust agent and other keylime components

Description
When I tried to run keylime in dual stack (IPv4 for v registrar and IPv6 for other components) the adding of keylime_agent via tenant CLI has failed. IPv4 was setup due to actual fail in keylime registrar. IPv6 ISSUE
Expected behavior vs. actual behavior
Adding keylime_agent will be succesfull. vs. Adding keylime_agent has failed.
Steps to reproduce problem
1.Setup IPv6 localhost for keylime_agent, keylime_verifier ( for example IPv6 localhost)
2.Run all keylime components
3.Add keylime_agent via keylime_tenant
Relevant logs
:: [ 05:45:02 ] :: [  BEGIN   ] :: Running 'keylime_tenant -v ::1 -t ::1 -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000 --runtime-policy policy.json -f /etc/hostname -c add'
INFO:keylime.config:Reading configuration from ['/etc/keylime/logging.conf']
2023-03-02 05:45:02.738 - keylime.keylime.cmd_exec - DEBUG - Executing command "tpm2_startup --version"
2023-03-02 05:45:02.745 - keylime.tpm - INFO - TPM2-TOOLS Version: 5.4
2023-03-02 05:45:02.747 - keylime.tenant - WARNING - The configuration upgrade templates path /usr/share/keylime/templates does not exist
2023-03-02 05:45:02.748 - keylime.config - INFO - Reading configuration from ['/etc/keylime/tenant.conf']
2023-03-02 05:45:02.748 - keylime.tenant - INFO - Setting up client TLS...
2023-03-02 05:45:02.748 - keylime.tenant - INFO - Using default client_cert option for tenant
2023-03-02 05:45:02.748 - keylime.tenant - INFO - Using default client_key option for tenant
2023-03-02 05:45:02.749 - keylime.tenant - INFO - No value provided in client_key_password option for tenant, assuming the key is unencrypted
2023-03-02 05:45:02.771 - keylime.tenant - INFO - TLS is enabled.
2023-03-02 05:45:02.773 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): <IP_ADDRESS>:8891
2023-03-02 05:45:02.804 - urllib3.connectionpool - DEBUG - https://<IP_ADDRESS>:8891 "GET /v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000 HTTP/1.1" 200 None
2023-03-02 05:45:02.827 - keylime.tenant - ERROR - Failed to parse: https://::1:9002/version
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/requests/models.py", line 434, in prepare_url
scheme, auth, host, port, path, query, fragment = parse_url(url)
^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/urllib3/util/url.py", line 397, in parse_url
return six.raise_from(LocationParseError(source_url), None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://::1:9002/version
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/keylime-6.6.0-py3.11.egg/keylime/cmd/tenant.py", line 10, in main
tenant.main()
File "/usr/local/lib/python3.11/site-packages/keylime-6.6.0-py3.11.egg/keylime/tenant.py", line 1637, in main
mytenant.init_add(vars(args))
File "/usr/local/lib/python3.11/site-packages/keylime-6.6.0-py3.11.egg/keylime/tenant.py", line 237, in init_add
res = get_version.get("/version")
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/keylime-6.6.0-py3.11.egg/keylime/requests_client.py", line 47, in get
return self.session.get(self.base_url + url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 573, in request
prep = self.prepare_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 484, in prepare_request
p.prepare(
File "/usr/lib/python3.11/site-packages/requests/models.py", line 368, in prepare
self.prepare_url(url, params)
File "/usr/lib/python3.11/site-packages/requests/models.py", line 436, in prepare_url
raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: https://::1:9002/version

You may have to use [::1] instead of ::1 :
$ python
Python 3.11.1 (main, Jan  6 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib3.util import parse_url
>>> parse_url("https://::1:8080/foo")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/site-packages/urllib3/util/url.py", line 397, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://::1:8080/foo
>>> parse_url("https://[::1]:8080/foo")
Url(scheme='https', auth=None, host='[::1]', port=8080, path='/foo', query=None, fragment=None)

After manual testing, I can confirm that this approach works. I'm just not sure if it would be better for the user environment to implement IPv6 without square brackets. And it would also be good to standardize the IPv6 address format. In the verifier configuration file, IPv6 is set without square brackets but for other components we use the format with square brackets.

Relative to my testing that may not cover all the cases:
The verifier's server IP and port can be specified with simple '::1'. I could specify the registrar_ip in the verifier.conf with with '::1' and it worked but I am not sure whether it actually initiated a connection to the registrar (likely not).
The registrar's server IP address could also be specified using '::1' in its config file.
The tenant tool (python) does require square brackets "[::1]" on the command line (for -t and -r options for example).
The agent (Rust) seems to require the "[::1]" notation for its server IP address.
Python: For python programs it seems that server IP addresses (= local listening IP addresses) can be specified by '::1' and anything that goes into a URL requires "[::1]".
Rust: For Rust the server IP addresses the only choice seems to be square brackets. I think there it is related to how the HttpServer (library) handles the IP addresses. @lkatalin @ansasaki
Command lines: For a user to specify the non-bracket version on the command line would be better and we can add the brackets.
Config files: Same thing -- no brackets ?

I agree. Thanks for the summary, I definitely know that brackets are sometimes necessary for IPv6 processing, but we should definitely try to implement brackets in keylime code to increase usability and clarity for keylime users.

@Koncpa I updated the PR with another patch to add brackets onto IPv6 IP addresses when they are going into a URL.

Has this issue been resolved now?

@Koncpa Will close this issue for now, fell free to open if needed.

Hi guys, I can confirm that IPv6 should work for now, here is PR for our test suite. I will just disscuss to possibility to paste IPv6 into rust agent.conf without brackets, but from python side is OK.


runtime error occurs when assign value to array with index greater than its length.

It seems a bug in vm.set_member() function.
    ValueBase::Number(n) if n - n.floor() == 0.0 => {
        if n as usize >= map.length as usize {
            map.length = n as usize;
            map.elems.set_len(n as usize);
        }
        map.elems[n as usize] = val;    // <= index out of bounds
    }

ex)
var ary = []
console.log(ary[5] = 4)

$ node array.js
4
$ cargo run array.js
    Finished dev [unoptimized + debuginfo] target(s) in 0.15s
     Running `target/debug/rapidus array.js`
thread 'main' panicked at 'index out of bounds: the len is 5 but the index is 5', /rustc/96064eb61dc703c289fae61bdf90593d3e7b2449/src/libcore/slice/mod.rs:2454:14
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'exited. status: 101', src/main.rs:177:25
note: Run with `RUST_BACKTRACE=1` for a backtrace.

fixed


Error pulling with GitHub application whilst brackets is open

Have brackets open and sync a change from the GitHub using the GitHub for Windows application.

The errors don't actually affect the files as far as I can see so possibly have errors in the console rather than onscreen?

Brackets 1.2.0-15697 (win), Git 1.9.5.msysgit.1 & Brackets-Git 0.14.18
Error: Failed parsing branch name from a15752d2f103fbe17061fc9c31727a001a8a8821

Error: Failed parsing branch name from a15752d2f103fbe17061fc9c31727a001a8a8821
    at file:///C:/Users/Reece/AppData/Roaming/Brackets/extensions/user/zaggino.brackets-git/src/Branch.js:386:40
    at File.eval (/filesystem/File.js:125:13)
    at eval (/filesystem/impls/appshell/AppshellFileSystem.js:378:25)

This should be fixed in latest version 0.14.19, please test/confirm. Thanks!

Seems to be fixed, thankyou :)


For Auto number settings this page needs update

Hi Team,
The settings has a lot of changes for example the Auto-numbering for all the entities, the attributes are gone and we need to use the #Auto Numbering Button and need to select entities and provide relevant values.

Please update documentation. Also update for other changes as well on field service settings

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: f6c4cc24-2dd8-908a-aee2-3b52360eeaa7
Version Independent ID: 0dfc2d9a-ad3d-e7dd-3a7a-38da87565272
Content: Set defaults for work orders, bookings, the schedule board, and agreements
Content Source: ce/field-service/configure-default-settings.md
Service: dynamics-365-field-service
GitHub Login: @jshotts
Microsoft Alias: jasonshotts

Thanks for sharing this issue @pavansarma93. I'm going to look into it soon. If you have discovered other settings that are off, please let me know so I can address them in the same go.
Thanks,
Michael

I've created a PR that addresses the feedback and updates the setting and automatically closes this issue on merge. Thanks again for sharing. If you still see things being off, please open a new issue.
Thanks,
Michael


CORE-7629: Re-enable disabled e2e tests requiring forceUpload

With fixes pushed for CORE-7629, this will enable the tests that rely on forceUploading

Looks like there is some flakiness that need to be investigated: https://ci02.dev.r3.com/blue/organizations/jenkins/Corda5%2Fcorda-runtime-os-combined-worker-e2e-tests/detail/PR-2584/3/tests

I think that since it's the missing flow for holdingId, then it should be resolved by this PR: https://github.com/corda/corda-runtime-os/pull/2580. Will rebase and check once it's merged


Implement blockage in private channel

If userA block userB,

any future msgs from userB to userA is invisible, unblocked will not take these msgs back
publish msg endpoint will return 403 when userB sends msg to userA (actually do not persist/deliver msgs)
userB can still get new msgs from userA until userB block userA

@joejoe2 - still Looks good, you can merge it :)


Support OpenID Connect (oidc) for database user login

This is being moved to be a commercial feature from Celerdata.


Re-write of non-finetuned in-context learning sections

Can we guide developers away from fine-tuning since it is largely unavailable and maturity of generative AI approaches with later models are steering us toward using turbo for in-context learning and better system messaging to improve model steerability. It is important to share how gpt-35-turbo can even be used for non-conversational tasks too such as summarization.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 3026912b-cf94-e0fb-0b83-6f78219b24e0
Version Independent ID: 3026912b-cf94-e0fb-0b83-6f78219b24e0
Content: Query-based document summarization - Azure Architecture Center
Content Source: docs/guide/ai/query-based-summarization.md
Service: architecture-center
Sub-service: azure-guide
GitHub Login: @raniabayoumy
Microsoft Alias: rabayoum

@nicholasdbrady
Thanks for your feedback! We will investigate and update as appropriate.

Thank you for your feedback.  We have created internal ticket 88755 to track and investigate this issue.

#please-close


Dicussions RTL Support

1 - On discussion topics screen 'Search all posts' search bar is not inverted.
Do we attempt to invert native controls?
2 - On discussion Responses screen # responses ,# should be displayed first then text responses.
3 - On discussion Responses screen response should be on right
4 - On discussion Responses screen #of comments in responses are not appearing complete (as well as not inverted) along with discussion icon.
5 - On comments screen # comments ,# should be displayed first then text comments. (Icon is missing too)
6 - On discussion Responses screen Votes/Unvotes, Follow/Unfollow are not appearing properly, they should get inverted properly.
7 - On posts screen where posts appear under any topic the filters through which user refine any post, on tapping those filters the tick mark that indicates the selected filter is not inverted.
8 - On create a new post screen cancel button that is appearing on left of top header is not inverted.
Do we ignore the nav bar positioning because it will automatically be taken care of with iOS9?
9 - On create a new post screen 'Discussion and question' are not inverted.
10 - On create a new post screen 'Topic:' in drop down is not inverted.
11 - On create a new post screen drop down in which topics appears when user tap on that drop down the topics are not inverted.
12 - On response screen cancel button that is appearing on left of top header is not inverted.
13 - On Comments screen response text is not inverted.
14 - On Comments screen cancel button that is appearing on left of top header is not inverted.

@aleffert @mikekatz pl review

@aleffert updated

:+1:


taobao data loaded on Hudi shows different results than taobao data loaded on StarRocks internal format

https://forum.starrocks.io/t/retail-ecommerce-funnel-analysis-demo-with-1-million-members-and-87-million-record-dataset-using-starrocks/269
VS
mysql> select count(*) from user_behavior;
+----------+
| count(*) |
+----------+
| 86953525 |
+----------+
1 row in set (0.14 sec)

mysql> with tmp1 as (
    ->   with tmp as (
    ->     select
    ->       t.level as level,
    ->       count(UserID) as res
    ->     from
    ->       (
    ->         select
    ->           UserID,
    ->           window_funnel(
    ->             18000,
    ->             `Timestamp`,
    ->             0,
    ->             [BehaviorType = 'pv' ,
    ->             BehaviorType = 'cart',
    ->             BehaviorType = 'buy' ]
    ->           ) as level
    ->         from
    ->           user_behavior
    ->         where `Timestamp` >= '2017-12-02 00:00:00'
    ->             and `Timestamp` <= '2017-12-02 23:59:59'
    ->         group by
    ->           UserID
    ->       ) as t
    ->     where
    ->       t.level > 0
    ->     group by
    ->       t.level
    ->   )
    ->   select
    ->     tmp.level,
    ->     sum(tmp.res) over (
      order by
        tmp.level rows between current row
        and unbounded following
    ) as retention
  from
    tmp
)
select
  tmp1.level,
  tmp1.retention,
  last_value(tmp1.retention) over(
    order by
      tmp1.level rows between current row
      and 1 following
  )/ tmp1.retention as retention_ratio
from
  tmp1;    ->       order by
    ->         tmp.level rows between current row
    ->         and unbounded following
    ->     ) as retention
    ->   from
    ->     tmp
    -> )
    -> select
    ->   tmp1.level,
    ->   tmp1.retention,
    ->   last_value(tmp1.retention) over(
    ->     order by
    ->       tmp1.level rows between current row
    ->       and 1 following
    ->   )/ tmp1.retention as retention_ratio
    -> from
    ->   tmp1;
+-------+-----------+---------------------+
| level | retention | retention_ratio     |
+-------+-----------+---------------------+
|     1 |    913314 | 0.34725078122091635 |
|     2 |    317149 | 0.23266981765668504 |
|     3 |     73791 |                   1 |
+-------+-----------+---------------------+
3 rows in set (1.94 sec)

mysql> with tmp1 as (
    ->   with tmp as (
    ->     select
    ->       ItemID,
    ->       t.level as level,
    ->       count(UserID) as res
    ->     from
    ->       (
    ->         select
    ->           ItemID,
    ->           UserID,
    ->           window_funnel(
    ->             1800,
    ->             timestamp,
    ->             0,
    ->             [BehaviorType = 'pv',
    ->             BehaviorType ='buy' ]
    ->           ) as level
    ->         from
    ->           user_behavior
    ->         where timestamp >= '2017-12-02 00:00:00'
    ->             and timestamp <= '2017-12-02 23:59:59'
    ->         group by
    ->           ItemID,
    ->           UserID
    ->       ) as t
    ->     where
    ->       t.level > 0
    ->     group by
    ->       t.ItemID,
    ->       t.level
    ->   )
    ->   select
    ->     tmp.ItemID,
    ->     tmp.level,
    ->     sum(tmp.res) over (
    ->       partition by tmp.ItemID
    ->       order by
    ->         tmp.level rows between current row
    ->         and unbounded following
    ->     ) as retention
    ->   from
    ->     tmp
    -> )
    -> select
    ->   tmp1.ItemID,
    ->   tmp1.level,
    ->   tmp1.retention / last_value(tmp1.retention) over(
    ->     partition by tmp1.ItemID
    ->     order by
    ->       tmp1.level desc rows between current row
    ->       and 1 following
    ->   ) as retention_ratio
    -> from
    ->   tmp1
    -> order by
    ->   level desc,
    ->   retention_ratio
    -> limit
    ->   10;
+---------+-------+-----------------------+
| ItemID  | level | retention_ratio       |
+---------+-------+-----------------------+
|   59883 |     2 | 0.0003616636528028933 |
|  394978 |     2 | 0.0006357279084551812 |
| 1164931 |     2 | 0.0006648936170212766 |
| 4622270 |     2 | 0.0007692307692307692 |
|  812879 |     2 | 0.0009121313469139556 |
| 1783990 |     2 | 0.0009132420091324201 |
| 3847054 |     2 |  0.000925925925925926 |
| 2742138 |     2 | 0.0009881422924901185 |
|  530918 |     2 | 0.0010193679918450561 |
|  600756 |     2 | 0.0010319917440660474 |
+---------+-------+-----------------------+
10 rows in set (3.67 sec)

mysql> select
    ->   log.BehaviorType,
    ->   count(log.BehaviorType)
    -> from
    ->   (
    ->     select
    ->       ItemID,
    ->       UserID,
    ->       window_funnel(
    ->         1800,
    ->         timestamp,
    ->         0,
    ->         [BehaviorType = 'pv' ,
    ->         BehaviorType = 'buy' ]
    ->       ) as level
    ->     from
    ->       user_behavior
    ->     where timestamp >= '2017-12-02 00:00:00'
    ->         and timestamp <= '2017-12-02 23:59:59'
    ->     group by
    ->       ItemID,
    ->       UserID
    ->   ) as list
    ->   left join (
    ->     select
    ->       UserID,
    ->       array_agg(BehaviorType) as BehaviorType
    ->     from
    ->       user_behavior
    ->     where
    ->       ItemID = 3563468
    ->       and timestamp >= '2017-12-02 00:00:00'
    ->       and timestamp <= '2017-12-02 23:59:59'
    ->     group by
    ->       UserID
    ->   ) as log on list.UserID = log.UserID
    -> where
    ->   list.ItemID = 3563468
    ->   and list.level = 1
    -> group by
    ->   log.BehaviorType
    -> order by
    ->   count(BehaviorType) desc;
+--------------------------------------+-------------------------+
| BehaviorType                         | count(log.BehaviorType) |
+--------------------------------------+-------------------------+
| ["pv"]                               |                    1589 |
| ["pv","pv"]                          |                      52 |
| ["pv","pv","pv"]                     |                      10 |
| ["cart","pv"]                        |                       8 |
| ["cart","pv","pv"]                   |                       6 |
| ["fav","pv"]                         |                       6 |
| ["fav","pv","pv"]                    |                       3 |
| ["pv","pv","pv","pv"]                |                       2 |
| ["cart","pv","pv","pv"]              |                       2 |
| ["cart","pv","pv","pv","pv"]         |                       1 |
| ["pv","pv","pv","pv","pv","pv","pv"] |                       1 |
| ["fav","pv","pv","pv","pv"]          |                       1 |
| ["fav","pv","pv","pv","cart"]        |                       1 |
| ["pv","cart","pv"]                   |                       1 |
| ["pv","pv","cart"]                   |                       1 |
+--------------------------------------+-------------------------+
15 rows in set (3.10 sec)

mysql>

I cannot duplicate the results in https://github.com/StarRocks/starrocks/discussions/26075


Quality of editing video

Can the explain the quality of editing video compared to the original video?

Yes, we provide the demo video. It shows some qualitative comparisons together.
I think you may be interested in this work, why don't you come to our research and join us? It will also be nice.

If I send you a video or picture, can you editting that?

sure send me a video now


Update QuickStart guides to support PSTN Dial-Out Feature

Update QuickStart guides to support PSTN Dial-Out Feature

@Mrayyan

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

@alexokun  - can you review the proposed changes?
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Hi @American-Dipper, do you know why this PR is kept getting closed by the BOT? I have recently updated with another QuickStart Readme change but it is closed right after. There is no change to Readme header section (Author or Dates) though. This happens 2nd time for those changes.

@mikehang-msft - since you're an FTE, the bot must think you should be using the private repo. I'll include instructions for that below; it might be a better option if you're contributing frequently. Meanwhile I'll open this PR again. Meanwhile, I recommend not making any new contributions to this PR until it has been reviewed / approved by @alexokun  and merged.
Thanks!

Microsoft authors and contributors need to make content contributions through the private repository so the content can be staged and validated by the current validation rules.
After you make your changes in the private repo, review the articles on staging, and fix any validation issues, you can add the sign-off comment to let PR reviewers know that the updates are ready to be merged.
Read about authors working in the public repo for more information.
If you need help getting set up to work in the private repo, see Get started contributing to the Microsoft Learn platform.

Microsoft authors and contributors need to make content contributions through the private repository so the content can be staged and validated by the current validation rules.
After you make your changes in the private repo, review the articles on staging, and fix any validation issues, you can add the sign-off comment to let PR reviewers know that the updates are ready to be merged.
Read about authors working in the public repo for more information.
If you need help getting set up to work in the private repo, see Get started contributing to the Microsoft Learn platform.


Add 24bit color aka "direct color" aka "truecolor"

In addition to truecolor support based on TERM, this also adds some new 256 color TERMs to Mode::TERM_256 and used the mode Regexps inside the support Regexp. This required swapping the require order, because Support now depends on those Regexps from Mode.

I did make "ansi" support more conservative and added "^" to a few of the other support cases which might lead to some false negatives, but tput should still be a reliable fallback for those. A few other matches became more liberal and might lead to new false matches, but only when it correctly matches popular modern terminals though it may incorrectly match very old (presumably uncommon) releases.
To query terminfo, I ran the following bash script and several like it:
for term in $(toe -a | awk '{ print $1 }' | sort -u | xargs); do
    printf "%3d %s\n" "$(TERM="$term" tput colors)" "$term"
done | sort -n

Coverage decreased (-10.5%) to 80.198% when pulling 8ab6217da1c293248b4ccd54f314e70f1d33a342 on nevans:add-truecolor-and-some-others into 1a85486ec86c08abb6535ca0461dd07d18eb59a9 on piotrmurach:master.

Great PR! 🌟  Thank you for expanding 256 detection and adding truecolor support - this will defo come handy in the pastel gem. I'll try to release a new version soon.


Keep existing tailscale subnet routes when deploying

Is your feature request related to a problem? Please describe.

When deploying k3s with tailscale, we are resetting the tailscale subnet routes and adding the ones required for flannel communication. Therefore, if a user is already operating tailscale with subnet routes, his/her traffic will get disrupted.
Describe the solution you'd like

Before resetting, we should read the current subnet routes, save them, and then, after the reset add them back together with the new one for flannel
Describe alternatives you've considered

Additional context

It seems that Tailscale does not appear to expose the interface of the current configuration. Is there any good way to solve this problem?

It seems that Tailscale does not appear to expose the interface of the current configuration. Is there any good way to solve this problem?

What do you mean by the interface of the current configuration? AFAIK, there is only one tailscale interface and if you change the config, that interface will use the new configuration

Sorry for my mistake, I mean there is no API from tailscale to get current subroute config. And tailscale set would overwrite current subroutes.
I also encountered this issue. I tried to fix it, but I couldn't find a way to get the current subnet router.

I thought this output would return the subroute config https://github.com/k3s-io/k3s/blob/master/pkg/vpn/vpn.go#L123

You are right, This is really good news.


No such file or directory  #include "nan.h"

notebook/node_modules/ws/node_modules/utf-8-validate/build'
../src/main.cc:1:17: fatal error: nan.h: No such file or directory
#include "nan.h"
^
compilation terminated.
make: *** [Release/obj.target/runas/src/main.o] Error 1
Missing file when installing via atom package manager.

same here

Here is the full error on ubuntu 14.04
><EMAIL_ADDRESS>install /tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/pathwatcher/node_modules/runas
> node-gyp rebuild

make: Entering directory `/tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/pathwatcher/node_modules/runas/build'
  CXX(target) Release/obj.target/runas/src/main.o
make: Leaving directory `/tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/pathwatcher/node_modules/runas/build'

><EMAIL_ADDRESS>install /tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/ws/node_modules/utf-8-validate
> node-gyp rebuild

make: Entering directory `/tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/ws/node_modules/utf-8-validate/build'
  CXX(target) Release/obj.target/validation/src/validation.o
  SOLINK_MODULE(target) Release/obj.target/validation.node
  COPY Release/validation.node
make: Leaving directory `/tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/ws/node_modules/utf-8-validate/build'

><EMAIL_ADDRESS>install /tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/ws/node_modules/bufferutil
> node-gyp rebuild

make: Entering directory `/tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/ws/node_modules/bufferutil/build'
  CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
  SOLINK_MODULE(target) Release/obj.target/bufferutil.node
  COPY Release/bufferutil.node
make: Leaving directory `/tmp/apm-install-dir-11642-5711-v0zwdt/node_modules/jupyter-notebook/node_modules/ws/node_modules/bufferutil/build'

npm WARN peerDependencies The peer dependency<EMAIL_ADDRESS>included from jupyter-js-services will no
npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency
npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.
npm WARN peerDependencies The peer dependency<EMAIL_ADDRESS>included from jupyter-js-services will no
npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency
npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly.
../src/main.cc:1:17: fatal error: nan.h: No such file or directory
 #include "nan.h"
                 ^
compilation terminated.

...

I also tried installing nan header, but this didn't help.
I find similar issues around github e.g. https://github.com/TooTallNate/node-time/issues/54, https://github.com/Automattic/node-canvas/issues/588, but I'm not clear what to get from that

Similar: https://github.com/jupyter/atom-notebook/issues/41

This repo is not being actively maintained 🙁 In the meantime, I suggest that you look at hydrogen to run code inline in Atom's text editor, nteract (a new Jupyter notebook client built using Electron, React, Redux, RxJS), Jupyter Notebook, and JupyterLab.
Having said that, nteract is currently extracting its many components into separate packages which will allow them to be reused and assembled to create new apps. When I have the time, I would like to use them to completely refactor this project.


Annual Review for the OpenWallet Foundation Technical Advisory Council

Hello, @openwallet-foundation-labs/sd-jwt-rust-maintainers : Please note that your project annual review was due on Nov 27, 2024. Please do submit this to the TAC.

@Abdulbois @akarabashov @ashcherbakov please confirm if you saw this thread and let me know if you will have a chance to file the report.

@jovfer Thank you for the reminder. We will prepare and submit the report.


bug(client): test failed in package list mode, success in local directory mode

client package test failed in package list mode:
ide@sshd-lazy:~/develop/aprilsh $ go test -tags utmps github.com/ericwq/aprilsh/frontend/client
--- FAIL: TestQueryTerminal_Func (0.00s)
    --- FAIL: TestQueryTerminal_Func/query_func (0.00s)
        client_test.go:595: query func expect nil err, got set raw mode for /dev/stdout error: inappropriate ioctl for device
FAIL
FAIL	github.com/ericwq/aprilsh/frontend/client	0.044s
FAIL

client package test failed in package list mode:
ide@sshd-lazy:~/develop/aprilsh $ cd frontend/client/
ide@sshd-lazy:~/develop/aprilsh/frontend/client $ go test .
--- FAIL: TestQueryTerminal_Func (0.00s)
    --- FAIL: TestQueryTerminal_Func/query_func (0.00s)
        client_test.go:595: query func expect nil err, got set raw mode for /dev/stdout error: inappropriate ioctl for device
FAIL
FAIL	github.com/ericwq/aprilsh/frontend/client	0.041s
FAIL

client package test passed in local directory mode:
ide@sshd-lazy:~/develop/aprilsh/frontend/client $ go test
PASS
ok  	github.com/ericwq/aprilsh/frontend/client	0.198s

it's a go test bug. see this


Update Native Chat Among Federated Orgs

Native chat with people in other orgs hasn't been limited to 1:1 chat for some time now. Need to update to reflect multi-party chat and expectations among multi-org chats as it relates to federation allow/block list.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1ac8f101-bdb5-2b55-d5f4-e070b70768d4
Version Independent ID: 964d836e-d00b-b585-8f02-e03c5ec8dab9
Content: Native chat experience for external (federated) users in Microsoft Teams - Microsoft Teams
Content Source: Teams/native-chat-for-external-users.md
Service: msteams
GitHub Login: @MikePlumleyMSFT
Microsoft Alias: mikeplum

@BrianMcNeese have you found any official document to confirm this that I could add to this article? Does this article help at all - https://docs.microsoft.com/microsoftteams/teams-and-skypeforbusiness-coexistence-and-interoperability? Do you recommend removing that sentence completely about 1:1 chats, or changing it?

Hi @lindspea the only thing published by us that I have seen is the public roadmap item https://www.microsoft.com/en-us/microsoft-365/roadmap?filters=&searchterms=51126 (Microsoft Teams: Chat with External Users in Group Chats). As for the additional context about how that works in mutli-org chats (think 3+ orgs), I think we need a feature PM or whoever owns this to provide official comment.

@tiburd please can you assist?

@MikePlumleyMSFT - Hi, you're listed as the author of this article - are you able to help resolve this issue?

@lindspea @tiburd @BrianMcNeese  Given the high level scope of the article, I don't think we need to go into details around multi-org chats. If the 1:1 limitation is not longer true, let's just remove the reference.

I could agree with @MikePlumleyMSFT given the context of the article; perhaps the details of multi-org chats belong elsewhere. @tiburd - yes I'll create a pull request removing the 1:1 chat reference.

@BrianMcNeese - thank you! ...closing this issue.


fips: tikv heap profiling failed after enable tls and set require_secure_transport = on

Bug Report
Please answer these questions before submitting your issue. Thanks!
What did you do?

What did you expect to see?
No error, works fine.

What did you see instead?
Profile Error
failed to fetch and write to temp file: failed to fetch profile with *.prof format: failed to fetch tikv heap profile: Use of uninitialized value $line in substitution (s///) at /dev/stdin line 3323. http://tikv1-peer:20180/debug/pprof/symbol doesn't exist

What version of TiDB Dashboard are you using (./tidb-dashboard --version)?
master

/assign @mornyx

/severity major

TiKV heap profile depends on jeprof which does not support mTLS yet.
https://github.com/pingcap/tidb-dashboard/blob/706489ff610dcccc238335b26e0bc97cbf486c6e/pkg/apiserver/profiling/fetcher.go#L78


Feature Request: Consider mitigating possiblity of calling Process with envconfig.Config as argument.

func Process(ctx context.Context, i any, mus ...Mutator) error and func ProcessWith(ctx context.Context, c *Config) error have similar names and signaures, and it would be easy to accidentally call Process(ctx, *Config) which returns no error, but will not properly fill the target the user intended.
I can think of two paths to preventing such an error:

Add a guard clause to Process which will return an error or panic in the case that it is called with *envconfig.Config for i. There is no reason I can think of a user would intentionally call Process with config, so I think failing is a reasonable course of action.
Convert calls of Process with a *envconfig.Config into an equivalent ProcessWith call. The main issue here is dealing with the mus ... which could be done by adding any Mutators in the vararg to the Config provided, but the behavior may be somewhat unexpected.

#109 is an implementation of 2.)


[Distributed] LocalDAS cannot gain this constraint, even though the DAS protocol has it

This requirement was added in LocalTestingDistributedActorSystem but only on main. Technically this is an ABI break (!), because one could have passed a different type here - if calling internal and not through the witness I AFAICS.
We cannot make this addition and must undo and keep it as it was in Swift 5.7.
The PR introducing the issue was 1675669e4382b86083f0557df84fb824de753f75 which was undoing a bunch of workarounds, but pushed it too far with this one change.
This seems to work @DougGregor, so we got lucky

@swift-ci please smoke test

Thanks for review!
Resolves rdar://101858553


Remove duplicate assignment in Task::toJson method

summary:
The Task::toJson method currently contains a duplicate assignment of the numDriversUngrouped_ attribute, which is unnecessary and redundant.
This commit removes the superfluous line to clean up the code and prevent any potential confusion.

@Yuhta could you please take a look ?


Flaky test: ClosingConnectionMidChunkPrefixThrows

This test fails occasionally with the following error:
Assert.Equal() Failure
          ↓ (pos 0)
Expected:
Actual:   HTTP/1.1 200 OK\r\nDate: Wed, 25 Jul 2018 0···
          ↑ (pos 0)
   at Microsoft.AspNetCore.Testing.TestConnection.<ReceiveEnd>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ChunkedRequestTests.<ClosingConnectionMidChunkPrefixThrows>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

@halter73 this should have been fixed by https://github.com/aspnet/KestrelHttpServer/pull/2734 right?

Correct. It looks like the failed test was from this commit which predates the fix.


How to adjust the detection range？

Nice work! I got 0.39mAP in range 0-50m , but only 0mAP in range 50-100m . Can I adjust detection range by changing "--detection_range" in file named anchor_generator.py  when initializing anchors? Or there is another way? Thank you !

In addition to modifying the anchor, the loss weight also needs to be adjusted.
For a 100m perception range, it is recommended to set the regression loss weight for xyz to 1.
Additionally, for different datasets, parameters such as the learning rate may also need to be adjusted.


[BEAM-7036] Spark portable runner: stage files to run on 'remote' clu…

…ster
This change lets us run portable pipelines on non-local Spark masters (e.g. spark://host:port). For now, this is just for Java; I haven't tried running any Python yet.

Stage files from class path if not manually set in pipelineOptions.
Move prepareFilesToStageForRemoteClusterExecution method from SparkRunner to SparkPipelineOptions and reuse in portable runner.
prepareFilesToStageForRemoteClusterExecution should fall back on the JVM's temp dir if not already specified.

Run Python PreCommit

To make it similar to Flink runner, we should use BeamFileSystemArtifactStagingService to stage user code.

To make it similar to Flink runner, we should use BeamFileSystemArtifactStagingService to stage user code.

Can you point me to where this is used in the Flink runner? I'm not seeing it

This one needs a rebase @ibzib

rebased

Run Portable_Python PreCommit

Run Spark ValidatesRunner