[RFC] Long context fine tuning in torchtune

Goal

Many of the new LLMs models support long context. For example, lamma 3.1 and Mistral 2 support 128k;
The trend is upwards, e.g. Gemini support 1M - 10M. Claude supports 200k;
Torchtune supports ~24k for llama3 8b with bsz=1;

This RFC aims to define which type of solution to prioritize so we can bridge this gap.
What number do we hope to achieve, with which model and which GPU?
There are three ways I propose we think about it:

Prioritize techniques that enable scaling, regardless of the model/sequence length, since the goal may shift, e.g. model X with context length Y;
Getting llama-3.1-8b to 128k, using 8xA100, since 8b is the most used SKU, smallest llama, supports 128k and none of the papers about long context used only one GPU;
Same as above, but aim for 64k, and rely on RoPE scaling to extend context further;

Sanity check - bsz, num_steps, memory required
Using llama2-7b (~half of the memory llama3-8b needs):
LongLora (https://arxiv.org/pdf/2309.12307) finetunes for 1000 steps (bsz=64) using 8x100, DDP,

YaRN paper (https://arxiv.org/pdf/2309.00071) does it for 400 steps (bsz=64, seq_len=64k, +200 steps for seq_len=128k)
Ring attention and Memory Efficient Attention (https://arxiv.org/pdf/2310.01889)

Why

Source: https://www.reddit.com/r/LocalLLaMA/comments/18t63ub/is_context_length_32k_actually_useful_to_you/
Below are some use cases, but an argument we can make is that supporting long-context will enable new cases, even if they don't exist today.
Possible data flavors:

Code: torchtune has 60k lines of code;
Multimodality:

Image: 512x512, patch_size=14 → 1300 tokens;
Video: 1min, 1fps → 78k tokens;
Audio:

1s audio (~3 words) → ~25 tokens.
1 minute → 1500 tokens.
60min → 90k tokens;

Long texts (books, financial documents):

Albania Wikipedia Page: 43k tokens.
Second page of “Attention is all you need”: 837 tokens, i.e. 100 pages = ~83k tokens;

Raw html:

Apple’s website, copy/paste text: 997 tokens
Apple’s website, raw html: 92091 tokens

Possible use cases:
* Coding;
* Multimodality understanding, e.g. text books with images, video+audio+captions, call transcripts, user web navigation;
* Creative writing;
* Summarization of multiple complex long documents, e.g. financial documents;
* Complex HTML understanding for clients that lack APIs;
* In-context learning, e.g. many examples in the input;
* Personalized search, e.g. all of the user history;
What about RAG?
RAG works in some scenarios and is cheap, but:

Adds complexity (e.g. parsers, hierarchical retrieval, embeddings, etc);
Possible bad retrieval / hallucinations;
Sometimes you need the whole picture;
"The bitter lesson": gemini can answer difficult questions about a document with 170k tokens: https://www.llamaindex.ai/blog/towards-long-context-rag.

However, long context and RAG can be orthogonal approaches: with long context, people can return longer documents, instead of small paragraphs. In other words, long context enables new RAG capabilities.
Our current snapshot:

Currently we support at most ~24k with llama 8b, on A100-80GB, for LoRa, QLoRA, FFT with or without FSDP. After some memory optimizations, we can probably increase it by a little, but probably not enough for 32k.

You may think: Shouldn’t QLoRA support much more than FFT? At higher context lengths, the activations consume most of the memory.
Our levers

Inference solutions, such as:

Attention sink (https://arxiv.org/pdf/2309.17453)
Clustered head attention (https://arxiv.org/pdf/2403.08058)

Add memory/recurrency modules to the transformer, such as:

Activation Beacon (https://arxiv.org/pdf/2401.03462)
infinattention (https://arxiv.org/pdf/2404.07143)

Memory-efficient methods, such as:

Parameter efficient (LoRA, GaLore claims to require 30% less memory than LoRA (https://arxiv.org/pdf/2407.08296));
Quantization (QLoRA, Q-GaLore);
Optimizers that require less memory (MiniAdam, Adafactor, 8bitAdam);

Custom efficient triton implementation that reduce memory:
Attention approximation methods, such as:

LongLora (https://arxiv.org/pdf/2309.12307)
Mixture of sparse attention (dynamic masking of attention: https://github.com/thu-nics/MoA)

Parallelism

Ring attention (context parallelism): https://arxiv.org/abs/2310.01889
Other types of parallelism (TP, PP, MP);

(1) is not relevant, since our focus is finetuning. (2) not widely adopted and it is risky to add new modules that change its inference behavior. (3) We already support QLoRA, 8bitoptmizers. The issue is the quadratic growth of activation, and such methods don't tackle it. (4) We rely on pytorch for it. This leaves us with (5) and (6).
LongLora:

LoRA, but also finetunes embeddings + normalizations, otherwise results are not good;
Split the sequence into chunks. Each attention head pays attention to a different chunk of the text, with some overlapping. Other grouping strategies were tested, e.g. stride and dilation, and they didn’t perform as well.

Pros:

Works with flash attention;
Paper shows promising perplexity and retrieval, comparable with full finetuning;
Inference can run with FULL attention (thats so nice);
Also applies to single GPU;

Cons:

It is an approximation;
It doesn’t scale (Upper bound is num_heads and each device needs to gather the whole sequence). In their implementation they use DDP, and their maximum was 100k with llama2-7b, which requires less memory than llama3;
Unclear if it works with block causal masking;

Pseudocode

Ring Attention

Ring Attention is a type of data parallelism, where the tokens are split across GPUs;
Every token can be processed individually, except when computing the attention;
Full attention requires softmax over all tokens, but we want to avoid materializing the full quadratic matrix. How can this be done? Answer: compute the softmax incrementally, by processing the attention in blocks/chunks, and then “undoing” the normalization with the stored x.exp().sum(). Ring attention parallelizes it.

Pseudocode
def naive_soft_max(x):
	return x.exp() / x.exp().sum()

# done in vanilla attention
target = naive_softmax(x)

# softmax over chunks
x1,x2 = torch.chunk(x,2)
softmax1 = naive_softmax(x1)
softmax2 = naive_softmax(x2)

# incremental softmax
sum_exp_1 = x1.exp().sum()
sum_exp_2 = x2.exp().sum()

softmax1_corrected = softmax1*sum_exp_1 / (sum_exp_1 + sum_exp_2)
softmax2_corrected = softmax2*sum_exp_2 / (sum_exp_1 + sum_exp_2)

softmax_combined = torch.cat([softmax1_corrected, softmax2_corrected])
torch.all_close(target, softmax_combined)

Source:  https://www.youtube.com/watch?v=ws7angQYIxI
Pros:

Scales linearly with number of devices. E.g. If you can do batch_size = 1, seq_length=24k in one device → with context_parallel you can do batch_size = 1, seq_length=24k*8 when using 8x devices;
Complexity is 6bch, where c is the block size. It is independent of sequence length. Other types of parallelism, such as tensor parallelism, would be overwhelmed by the activations of long sequences;
Is (probably) future-proof, as long as we use attention;
No approximation! It is equivalent to the full attention;
No overhead, since communication time < computation time;
Pytorch is working to support it and it integrates with FSDP by default, so according to our PoC in pytorch, it should also handle quantization and torchao well;

Cons:

Block causal mask, or other types of mask, makes it more complicated and slower. We would have to support it initially only for full causal mask;
Still experimental on pytorch;

Code pointers:
Pytorch: https://github.com/pytorch/pytorch/pull/129515
Torchtitan: https://github.com/pytorch/torchtitan/pull/433
Current constraints of pytorch implementation that they are actively working on:

Doesn't work with compile yet;
Doesn't support arbritary masking yet (e.g. block causal);
Doesn't support striped attention yet (makes ring attention faster https://arxiv.org/abs/2311.09431);
UI may change;

Suggestion
My suggestion is to focus on ring attention experimental implementation by pytorch, since it scales well and will be supported by them. If their CP implementation is optimal, then 8xA100 would allow us to get to 8x24k = 192k context length. Most likely LongLora wouldn't be enough to reach 64k with llama3.
Timeline

According to our PyTorch PoC, the code to implement context parallelism is short, and can be done in a week. Therefore, I propose the following:
Week July 29th - Formalize script to, given a model, find max_seq_length for different settings, so we can test at 16GB, 24GB, 40GB, 80GB
Week Aug 5th - Understand pytorch CP implementation
Week Aug 11th - Proof-of-concept for llama3.1 8b, stress test max seq len
Week Aug 19th - Propose and discuss implementation in torchtune
Week Aug 26th - Evaluation on key long context datasets
Week Sept 2nd - Optimize + unit test (maybe do it before eval?)
Week Sept 9th -  Implement for the rest of the models
Week Sept 16th - Tutorials + best practices
The timeline should look similar for LongLora. There is a risk in focusing too much on llama 3.1 and having something that needs refactoring to work for other models. We should try to minimize this risk and make it model independent, but prioritizing a family of models will allow us to iterate faster, learn and make adjustments later.
Evaluation

Multimodality is relevant, but not supported by torchtune yet. Possible dataset with 1T tokens: https://arxiv.org/pdf/2406.11271;
There is a lack of benchmarks, even for <32k, but it is an active area;
Most of the models report passkey retrieval tasks. Those don't necessarily correlate with task performance;
Furthermore, long-context training may hurt model performance in short-context. An interesting strategy is to evaluate across different context lengths and tasks

source: https://arxiv.org/pdf/2404.02060
Therefore, for evaluating our trained models I propose three strategies (text only):

Passkey retrieval, as a sanity check;
Improve long-context performance;
Retain short-context performance;

As a starter, we can use LongAlpaca (https://huggingface.co/datasets/Yukang/LongAlpaca-12k), since it was also used by LongLora. It contains 3k short examples and 9k long examples from the Alpaca dasetet, ranging from 35 characters to 191k.
This strategy can be improved during the evaluation focus time.

right! This is more about our "success criteria". Do we want to finetune 128k max, or would we be comfortable to get to 64k max, and 128k can be achieved through scaling.

reddit.com/r/LocalLLaMA/comments/18t63ub/is_context_length_32k_actually_useful_to_you

This is awesome motivation :)

right! This is more about our "success criteria". Do we want to finetune 128k max, or would we be comfortable to get to 64k max, and 128k can be achieved through scaling.

Ahhh I see, I see. Personally, I think the latter is what I consider success.

Albania Wikipedia Page: 43k tokens

I was just thinking about how I desperately want to finetune a model on the Albania Wikipedia page...

Maybe a dumb question, but it seems like Ring Attention really only helps us improve context length based on num of GPUs...so a single GPU QLoRA would never be able to get to 128k under this plan. Is that correct? Are we just saying we don't have the BW to take on single GPU long context?

How does this RFC (#1183) on RoPE scaling fit in here? Again, in my mind they are related, but it doesn't seem like this is accounted for in your work timeline.

How does this RFC (#1183) on RoPE scaling fit in here? Again, in my mind they are related, but it doesn't seem like this is accounted for in your work timeline.

Good question! Since we released llama 3.1 that has support for 128k, I thought that this was a solved problem, and that RFC was more about design discussion. Are you still going to implement it? If so, would it be ready before the timeline: "Week Sept 9th - Implement for the rest of the models"

Thanks for the detailed RFC! Overall I agree with the proposal to start trying out context parallel support using the latest APIs from distributed. I do have two main comments on a first pass (though I reserve the right to come back later with more 😅 ).
(1) Sequencing: I would like to get to "Proof-of-concept for llama3.1 8b, stress test max seq len" as fast as possible. This is most likely a prerequisite for any asks we have to distributed or any other memory-focused work we need to do. So I would suggest frontloading this portion of the work, imo the script and design discussions are moot until we have that. Even understanding the distributed CP implementation can be done in parallel
(2) We discussed this offline a bit already, but I would love to see an effort to increase the max context length on a single device as well. 24k is not great imo so we should explore some of the other techniques you described (along with other memory-saving techniques) in parallel. This is not necessarily something you need to consider as part of the context parallelism effort so nbd for this particular RFC, just want to call it out.

So I would suggest frontloading this portion of the work

sounds good, I changed it to go before the design

So is my understanding correct: for long context, ring attention / CP is the most promising but will require X number of GPUs, and if you want to use a single GPU you need to rely on RoPE scaling?
Also, do you have an idea of how ring attention will be called in torchtune? Does it happen under the hood with SDPA if we're using DTensors (based on my brief skim through the code snippets)?

for long context, ring attention / CP is the most promising but will require X number of GPUs. Whatever context length you can fit in one gpu, you can make it times n_gpus.
Correct

if you want to use a single GPU you need to rely on RoPE scaling
It doesnt fit in memory. Its like if you wanted to fit 405b in a single GPU. Its just too much, unless you do a lot of cpu/gpu wizardry, but then its slow

Also, do you have an idea of how ring attention will be called in torchtune? Does it happen under the hood with SDPA if we're using DTensors (based on my brief skim through the code snippets)?
I will be able to answer it better next week

Any move in here ? We truly need long context says 128k. Any enhancement would be highly appreciated. We tent to use LlaMA-3.1-8B
Thanks,
Steve

@thusinh1969, the library is much more memory efficient now. If you use all the memory flags, you should be able to get to it depending on how much memory and gpus you have.
keep the loss as chunked cross entropy
use compile=True
enable_activation_checkpoint=True
tokenizer.max_seq_len = your_seq_len
dataset.packed=True # makes it faster
then if you training lora, you can use offloading. I need to implement it for the full finetune recipes
enable_activation_offloading=True
if you are doing distributed training, you can also use the fsdp cpu offloading option in the config

Is this still actual? I don't see any related PRs, but things that are proposed here are pretty reasonable.

hey @krammnic , not really. We ended up going down a path of optimization, implemented activation offloading, chunked cross entropy and better compile. This saved tons of memory, allowing larger context. In we future we may come back to parallelism.

Hey @felipemello1, very interesting issue with a lot of material, thank you for taking the time of writing it.
I ended up here because I am also an unsloth user and I was wondering how the context length was handled in torchtune as I never provide infos about the MAX_SEQ_LEN.
What is the state of it in december 2024 ?
Do you need help for something ?

hey @lulmer , glad to hear you are interested!
I don't have the exact number, but depending on the model size and tokenizer.max_seq_len, i believe you can get up to 128k in a single device (80GiB) using all of our flags. You can see them here: https://github.com/pytorch/torchtune#optimization-flags
Regarding needing help, we always love when people contribute! We don't have any easy / low hanging fruit for long context specifically, but we have many tickets marked as "community help wanted" and "good first issue". If you would like to get used to the codebase and submitting some PRs, then I am sure that complex projects will appear. Let me know if you had any in mind. :)

@lulmer, if are interested in exploring LongLora (or another technique), please feel free to open an RFC with some naive code snippets. We can review it, give some pointers, and if it makes sense, it would be great to have you as a contributor.

Hey @felipemello1
I just got some bandwidth to work on this issue. I am mostly familiar with the codebase. Can you explain what you mean by opening an RFC do you mean a PR ? If we combine longlora with the optimization you were pointing out, is it possible to go past the 128k limit ?

Hey @felipemello1
I just got some bandwidth to work on this issue. I am mostly familiar with the codebase. Can you explain what you mean by opening an RFC do you mean a PR ? If we combine longlora with the optimization you were pointing out, is it possible to go past the 128k limit ?

RFC means Request For Comments, basically a document with your plan. We want to avoid someone spending a lot of effort doing the implementation in some way, and then us not feeling comfortable merging it into main, wasting all that work. So, instead, we always request for a PR with some minimalist dummy non-working code + analysis of pros/cons, example of how the user would use it, how it would look like in the config, etc. Then, if the maintainers are comfortable and sign off, it will be easy to merge the actual code when its ready. Here is a great example by Philip: https://github.com/pytorch/torchtune/pull/1283
PR is just a pull request. If you are not familiar with how people can submit changes to a repository in github, there are many short youtube videos explaining. Here is our guide on how to contribute: https://github.com/pytorch/torchtune/blob/main/CONTRIBUTING.md
let me know if you have any questions. Feel free to join our discord and ask them there too! Thanks for being interested in it :)

If we combine longlora with the optimization you were pointing out, is it possible to go past the 128k limit ?
If you use all of our optimizations, you can already go past the limit, depending on the model size and number of gpus you have. LongLora should help even more. Another front that is more promising, because it would for not only for LoRA, but way more complex, is context parallelism, which we will address at some point next year.


[BugFix] Fix MaterializedView refresh failed when two base table from…

… different catalog have the same partition column name.
What type of PR is this：

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #21612
Problem Summary(Required) ：

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto backported to target branch

[x] 3.0
[x] 2.5
[ ] 2.4
[ ] 2.3

@Mergifyio rebase

[FE PR Coverage Check]
:disappointed: fail : 2 / 4 (50.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMaterializedViewRefreshProcessor.java
2
4
50.00%
[853, 854]

@Mergifyio backport branch-3.0

@Mergifyio backport branch-2.5


Sparse rewards are converted to dense arrays

Due to the way that scipy works, if the rewards are given in sparse format they are at the moment converted to dense arrays. Specifically sparse.multiply(dense) will result in a dense numpy.matrix and sparse.sum(1) will result in a numpy.matrix.
It may be dersirable to have a way of keeping them sparse.

The same is true for probability matrices. For example, when checking for stochasticity of P matrices in ValueIteration class, the sparse matrices are converted to dense ones.


Try integrating capnp rrgraph to symbiflow-arch-defs

Issue to look into for 2020.
There is preliminary capnp rrgraph support in VPR off of branch https://github.com/SymbiFlow/vtr-verilog-to-routing/pull/340
Investigate:

Speed of python capnp library
Try replacing xc7 routing graph import with capnp version

This has been completed, resulting in a memory drop and faster import


PhantomJS web automation with Mastodon

May I kindly check if anyone have success logging into Mastodon and performing web interactions using PhantomJS? This is not a bug report and there isn't a duplicate issue on this question. I'm aware that Vitaly is stepping down, am still posting to see if any users have encountered this.
Someone on Mastodon has a use case to automate posting her comic posts to her Mastodon account. Source is a RSS feed. Ideally it should be done using Mastodon API. I'm more familiar with web automation so I tried doing it using a general purpose web automation tool I made that is based on CasperJs/PhantomJS/SlimerJS.
I was able to use below script to automate posting a private toot when running through Firefox/SlimerJS. But running through PhantomJS I get a blank dark grey screen after the login step. I also tried changing User Agent to make sure it is not due to explicit blocking by Mastodon design. I would not want to use headless browser approach if app/website owner explicitly do not want to serve headless browsers.
Accessing a Mastodon status post directly without login works and image capture is alright. Copying @DavidLibeau as he is working on Mastodon status sharing to other networks which I believe going through PhantomJS for screen capture.
https://mastodon.social/auth/sign_in
enter user_email as your_email
enter user_password as your_password
click Log in
click Adjust status privacy
click Post to mentioned users only
enter autosuggest-textarea__textarea as Automated test message to @yourself@your_instance
click Toot
click Logout

I've also cross-posted the question to Mastodon issues page.

Closing this issue as there is no update from any Mastodon+PhantomJS users for 2+ weeks.
This is probably an application specific issue which I don't want to add on to the load of the next PhantomJS maintainer. Am aware that Vitaly is stepping down. Many exciting changes in this domain! Headless Chrome is also coming, and Firefox is coming up with headless version as well.


Compatibility with ink! 5.0.0

The problem
Some newer contracts are written using ink! 5.X.X, and bundled version of @polkadot (currently "10.9.1")  panics with: Unable to handle version 5.
Additional context
It would be possible to use the newer version of @polkadot , however
import {options} from "@astar-network/astar-api";

doesn't support updated WsProvider from @polkadot/api "^10.12.4".
    const provider = new WsProvider('ws://localhost:9944');

    // Below will not work if `WsProvider` comes from @polkadot/api "^10.12.4"
    const api = new ApiPromise(options({ provider }));

I have a pretty big contract written with !ink 5.0.0, and downgrading to 4.X.X would be quite painful. Thanks.

Closing because found another solution, thanks.


Wheel does not use correct file name format

See https://www.python.org/dev/peps/pep-0427/#file-name-convention

Installation does not use wheel anymore


Video Tutorial Added to the game

PR Description 📜
I have added the video file in the index.html file and styled it accordingly in the style.css file
Fixes #86

Mark the task you have completed ✅

[X] I follow CONTRIBUTING GUIDELINE & CODE OF CONDUCT of this project.
[X] I have performed a self-review of my own code or work.
[X] I have commented my code, particularly in hard-to-understand areas.
[X] My changes generates no new warnings.
[X] I have followed proper naming convention showed in CONTRIBUTING GUIDELINE
[X] I have added screenshot for review.

## Add your screenshots(Optional) 📸

Thank you soo much for contributing to our repository 💗

As it is my first pull request, if there are any further changes do let me know. Also, feedback is welcome.

@Durgesh4993
Kindly review the changes and let me know


QA: /documents/ebook, /infographic, /white-paper

These are all using the same template. They are using the request consult CTA fragment.
https://main--merative2--hlxsites.hlx.page/documents/ebook/child-welfare-forging-stronger-connections
https://main--merative2--hlxsites.hlx.page/documents/ebook/zelta-enablement-bundle-client-presentation
https://main--merative2--hlxsites.hlx.page/documents/infographic/2023-payer-trends-in-healthcare
https://main--merative2--hlxsites.hlx.page/documents/infographic/chso-infographic
https://main--merative2--hlxsites.hlx.page/documents/infographic/mcd-3936774507-zelta-ig
https://main--merative2--hlxsites.hlx.page/documents/white-paper/coordinated-health-and-social-outcomes
https://main--merative2--hlxsites.hlx.page/documents/white-paper/medical-episode-grouper-applications-and-methodology

@sachinmesh ready for your review.

Text inside the CTA shouldn't be underlined.

I just also noticed that the Inline link style should only have 1px bottom underline, per:
https://www.figma.com/file/vw24IPXXt4vCh9BNumka9A/Web--Merative-Digital-Design-System-2.0?node-id=272%3A62&t=6AEt8K8PTmecZmjo-1

@sachinmesh the CTA underline was fixed, thanks for flagging, that was an accidental regression. Please review again?

@venturewise I'm noticing in all of the related documents block they're missing the eyebrow. I think @amol-anand  said they were pulling from AssetType. I can go through the documents and ensure all the /document pages have those items in there.

Ok, great. So we just need to author those values on the documents then. I can start doing that today.

@keith-kaplan This is ready for your re-review.

@venturewise  thanks. Just noticed this on all the PDF viewer pages. Looks like a misalignment of the tags on Desktop devices...

ok, I see it was changed today. I think we'll separate the .readtime margin onto it's own style and that should fix it. Will let you know when ready.

@keith-kaplan this readtime was fixed, can you take another look?

Much better. Thank you.


[AIRFLOW-2989] Add param to set bootDiskType in Dataproc Op

Make sure you have checked all steps below.
Jira

[x] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-2989

Description

[x] Here are some details about my PR, including screenshots of any UI changes:
Add param to set bootDiskType for master and worker nodes in DataprocClusterCreateOperator

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
Modified DataprocClusterCreateOperatorTest

Commits

[x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[x] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

cc @Fokko @fenglu-g

Codecov Report

Merging #3825 into master will decrease coverage by 0.28%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3825      +/-   ##
==========================================
- Coverage   77.43%   77.15%   -0.29%
==========================================
  Files         203      203
  Lines       15840    15840
==========================================
- Hits        12266    12221      -45
- Misses       3574     3619      +45

Impacted Files
Coverage Δ

airflow/hooks/hdfs_hook.py
27.5% <0%> (-65%)
:arrow_down:

airflow/utils/decorators.py
85.41% <0%> (-6.25%)
:arrow_down:

airflow/hooks/dbapi_hook.py
79.67% <0%> (-3.26%)
:arrow_down:

airflow/task/task_runner/base_task_runner.py
77.96% <0%> (-1.7%)
:arrow_down:

airflow/operators/docker_operator.py
96.51% <0%> (-1.17%)
:arrow_down:

airflow/www_rbac/app.py
96.66% <0%> (-1.12%)
:arrow_down:

airflow/configuration.py
82.96% <0%> (-1.12%)
:arrow_down:

airflow/www/app.py
98.97% <0%> (-1.03%)
:arrow_down:

airflow/bin/cli.py
64.53% <0%> (-0.26%)
:arrow_down:

airflow/www_rbac/views.py
72.47% <0%> (-0.15%)
:arrow_down:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8245447...486efa8. Read the comment docs.


Can these numbers be rounded?

Way too long to make viewing easy

great question!  absolutely, this data is notional and in fact will be
replaced with something special later today :+1:
On Mon, May 4, 2015 at 10:54 AM, EvanBerner<EMAIL_ADDRESS>wrote:

Way too long to make viewing easy
[image: numbers]
https://cloud.githubusercontent.com/assets/11509735/7455626/e92f1c1a-f24b-11e4-97c0-5f2258c73866.png
—
Reply to this email directly or view it on GitHub
https://github.com/projectjellyfish/api/issues/608.

well done @AllenBW. closing this as its now rounded.


Fibaro FGMS 001 motion sensor fails to add with S0 legacy

I've migrated from OZW and I have a few Fibraro FGMS 001 devices, all on the same firmware, which moved across fine.  Afterwards, I noticed that one of them had never been added securely.  I've tried to exclude and include it multiple times, including reseting the device to factory settings but I can't get it to add with legacy S0.   This is what I see in the log:
2022-06-06T19:29:03.363Z CNTRLR » [Node 112] Querying securely supported commands (S0)...
2022-06-06T19:29:03.372Z SERIAL » 0x0109001370029840252842                                            (11 bytes)
2022-06-06T19:29:03.373Z DRIVER » [Node 112] [REQ] [SendData]
                                  │ transmit options: 0x25
                                  │ callback id:      40
                                  └─[SecurityCCNonceGet]
2022-06-06T19:29:03.374Z SERIAL « [ACK]                                                                   (0x06)
2022-06-06T19:29:03.378Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2022-06-06T19:29:03.379Z SERIAL » [ACK]                                                                   (0x06)
2022-06-06T19:29:03.379Z DRIVER « [RES] [SendData]
                                    was sent: true
2022-06-06T19:29:03.393Z SERIAL « 0x0107001328000002c1                                                 (9 bytes)
2022-06-06T19:29:03.393Z SERIAL » [ACK]                                                                   (0x06)
2022-06-06T19:29:03.394Z DRIVER « [REQ] [SendData]
                                    callback id:     40
                                    transmit status: OK
2022-06-06T19:29:03.405Z SERIAL « 0x0110000400700a9880f26f904778f9cc2da3                              (18 bytes)
2022-06-06T19:29:03.406Z SERIAL » [ACK]                                                                   (0x06)
2022-06-06T19:29:03.406Z DRIVER « [Node 112] [REQ] [ApplicationCommand]
                                  └─[SecurityCCNonceReport]
                                      nonce: 0xf26f904778f9cc2d
2022-06-06T19:29:03.408Z SERIAL » 0x011d001370169881fc70fd2ce7e77bd8cd1e66f270fc4ca40ad9c41a252952    (31 bytes)
2022-06-06T19:29:03.409Z DRIVER » [Node 112] [REQ] [SendData]
                                  │ transmit options: 0x25
                                  │ callback id:      41
                                  └─[SecurityCCCommandEncapsulation]
                                    │ nonce id: 242
                                    └─[SecurityCCCommandsSupportedGet]
2022-06-06T19:29:03.411Z SERIAL « [ACK]                                                                   (0x06)
2022-06-06T19:29:03.415Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2022-06-06T19:29:03.416Z SERIAL » [ACK]                                                                   (0x06)
2022-06-06T19:29:03.416Z DRIVER « [RES] [SendData]
                                    was sent: true
2022-06-06T19:29:03.432Z SERIAL « 0x0107001329000002c0                                                 (9 bytes)
2022-06-06T19:29:03.432Z SERIAL » [ACK]                                                                   (0x06)
2022-06-06T19:29:03.437Z DRIVER « [REQ] [SendData]
                                    callback id:     41
                                    transmit status: OK
2022-06-06T19:29:04.495Z CNTRLR   [Node 112] Timed out while waiting for a response from the node (ZW0201)
2022-06-06T19:29:04.496Z CNTRLR   [Node 112] The node was not granted the S0 security class. Continuing intervie
                                  w non-securely.
2022-06-06T19:29:04.499Z CNTRLR   [Node 112] Interviewing Manufacturer Specific...

What am I doing wrong?

Can I see some of the later log? The device doesn't seem to respond to the query Z-Wave JS uses to determine which commands are supported with encryption. Maybe it is just too slow, maybe it doesn't respond.
I'd advise against using S0 for sensors anyways. S0 triples the number of messages needed to communicate, therefore tripling the failure chances. So if you don't require S0 for some reason, I'd just include this without encryption.


Light Engine: support ingest logs by choosing an S3 location

Describe the feature
Allow ingest custom application logs from an S3 location into Light Engine.
Use Case
With this feature, I can use to ingest logs from on-premise, other cloud providers, and unsupported AWS Service logs in CLO.
Proposed Solution
No response
Other Information
No response
Solution version used
v2.1.1

fixed in v2.2.0 https://github.com/aws-solutions/centralized-logging-with-opensearch/releases/tag/v2.2.0


How to create the continous responder task ??

Hello,

I have multiple initiator which initiate initiator task at different times. I want to one responder which respond continuously to every initiator and vice-versa. I tried to create the task which only response but It now working. Can you guys tell me how do i achieve this ?

Instead of using the protocomm to create the device private and public for secure connection at run time, I want to use the my own prv and pub key to secure connection. How can I do this??

I am not very clear with your question.

There is no api to set own priv and pub key in security handshake. But you can use apis in this file [espnow_security.c] (https://github.com/espressif/esp-now/blob/master/components/security/src/espnow_security.c) to set your own security key directly.


Muxing items with embedded recipes should be able to map handles from context.

We're now aiming for the Social app to use recipes embedded in Post entities to render each post. As part of this we agreed that it can make sense for the recipe to be able to map in handles accessible to the arc's context, vs. just those in the arc.
This is beneficial as it means the recipe containing PostMuxer needn't explicitly map in handles on behalf of the item-embedded recipes with particles (of unknown implementation and so handle data needs in the future) that may want to render an item.
Initial exploration with recipe detailed below fails with 'BOXED_avatar' handle not available. I think this is due to the logic in multiplexer-dom-particle.js for _mapParticleConnections taking a list of handles, and those are only the handles in the arc (this._views). Whereas, in outer-PEC.js onArcLoadRecipe we retrieve handles via Arc.findHandleById which will in turn fall back to _context.findStorageById.
I believe we may need to also grab that _context list of handles and include them in the views we proxy, rename, and update as otherConnections in multiplexer-dom-particle.js.
Sample recipe pre-interpolation by PostMuxer.js:
particle ShowSinglePost in 'https://$cdn/artifacts/Social/source/ShowSinglePost.js'
  ShowSinglePost(in Post {Number createdTimestamp, Text message, Text author, URL image, Number imageWidth, Number imageHeight, Text renderRecipe, Text renderParticleSpec, Text owner} post, in [Avatar {URL url, Text owner}] avatars)
  affordance dom
  consume item
  description `show ${post}`
recipe
  map 'BOXED_avatar' as avatars
  use '{{item_id}}' as v1
  slot '{{slot_id}}' as s1
  {{other_views}}
  ShowSinglePost
    post <- v1
    {{other_connections}}
    consume item as s1

Sample post-interpolation:
particle ShowSinglePost in 'https://$cdn/artifacts/Social/source/ShowSinglePost.js'
  ShowSinglePost(in Post {Number createdTimestamp, Text message, Text author, URL image, Number imageWidth, Number imageHeight, Text renderRecipe, Text renderParticleSpec, Text owner} post, in [Avatar {URL url, Text owner}] avatars)
  affordance dom
  consume item
  description `show ${post}`
recipe
  map 'BOXED_avatar' as avatars
  use '!1116802698706397:demo-yi4w7bofltv7z6hz:61' as v1
  slot '!1116802698706397:demo-yi4w7bofltv7z6hz:63' as s1

  ShowSinglePost
    post <- v1

    consume item as s1

The error we get on attempt to load the interpolated recipe into an inner arc:
Recipe is not resolvable recipe
  use '!1116802698706397:demo-yi4w7bofltv7z6hz:61' as view0 // Post {Number createdTimestamp, Text message, Text author, URL image, Number imageWidth, Number imageHeight, Text renderRecipe, Text renderParticleSpec, Text owner}
  map 'BOXED_avatar' as view1 // [Avatar {URL url, Text owner}]
  slot '!1116802698706397:demo-yi4w7bofltv7z6hz:63' as slot0
  ShowSinglePost as particle0
    avatars <- // unresolved handle-connection: missing handle
    post <- view0
    consume item as slot0

FYI @mmandlis @shans

@mmandlis from my summary above, this part:

I believe we may need to also grab that _context list of handles and include them in the views we proxy, rename, and update as otherConnections in multiplexer-dom-particle.js.

Is this possible in some manner from multiplexer-dom-particle.js, like, right here?

the views there aren't just all the handles in the arc, but rather just the handles that are connected in the transformation particle.
what imo we do need to do, is to extend mapHandle (or create a new method) that will map handles into the inner arc from the real arc (or context) by IDs or tags. @shans WDYT?

Or per chat discussion with @mmandlis she notes maybe it's fine to just add avatars <- avatars in the Particle definition where the connection is noted as missing.

Essentially obsolete, and we'll be discussing further in person at next meetup.


Resolve TODO in flaskutils.py

https://github.com/orchestracities/ngsi-timeseries-api/blob/master/src/wq/ql/flaskutils.py#L16

will reopen the issue soon.

@pooja1pathak @NEC-Vishal hi guys, this is one of the most cryptical TODOs in our code, my fault, I was the one who wrote it and I don't think it can possibly make sense to anyone else than me. Sorry about that!  So here goes.
Stream processing techniques in Work Queue
First off, a little digression about Work Queue performance. We designed WQ hoping to get an implementation with decent performance---well, for Python. Our approach was to put streams at the core of the design to be able to then perform most computations in linear time and constant space. This is key to get decent performance, b/c WQ could potentially handle very large datasets. So sucking any such a beast into RAM or iterating over it multiple times isn't an option. (I know we do exactly that all over the show in the rest of the code base, but eventually we should rewrite the rest of the code to adopt the same stream-based approach.)
In particular queries stream data directly from the DB to the client in constant space and time. As the DB streams each row in the result set, WQ processes it, converts it to JSON and writes that JSON right away as a text line to the HTTP response stream as shown below---r[n] is a DB row, j[n] is the corresponding JSON object.
DB              WQ            client
---             ---           ------
 |---- r1 -----> |               |
 |               |---- j1 -----> |
 |---- r2 -----> |               |
 |               |---- j2 -----> |
 | ...           | ...           |

The JSON response is actually an array where each entry contains the JSON corresponding to a DB row. But how can we write a JSON array in the HTTP response body without holding all its entries in memory? One entry at a time? Sorry for the silly humour, but that's actually close to what happens. We start writing the response with a line containing the JSON array start delimiter [, then as each row comes in we convert it to JSON, append a comma and write the line to the response stream---e.g. {"id": 31, "speed": 76.9},\n. After streaming all the rows in the result set, we write the last line containing the JSON array end delimiter ].
JSON array streaming algo
Let's flesh out the actual algorithm to convert rows to JSON and stream them to the client as a JSON array. I'm going to use Haskell as a spec language b/c it is very close to maths notation and has the advantage you can actually run the code---yes, we're going to put together an executable spec!
Conceptually we'll think of a stream as a sequence of items---sequence as in maths, i.e. a function from the natural numbers to the set of items to stream. This maps cleanly to the Haskell list algebraic data type---think potentially infinite array or Python "iterable", etc. The items we want to stream are DB rows and let's call R the set of all possible rows know to man. We'll assume there's a given function json : R --> String that converts a row to a JSON object.
Now we're ready to give an exact definition of the JSON array streaming process we sketched out earlier. The key ingredient is the function arr : [R] ---> [String] which takes a stream of rows and turns it into a stream of strings in such a way that if you concatenate all the strings in the stream you get a valid JSON array.
arr rs = "[" : go rs
    where
        go []       = [ "]" ]
        go [r]      = [ json r, "]" ]
        go (r:rest) = (json r ++ ",") : go rest

With that under our belt, all we need is a write function to send the client each item arr outputs as a text line. This kind of function you typically get from your Web framework, but we'll implement a stub here to be able to run the Haskell spec.
write [] = send ""
write (line:rest) = do
    send line
    write rest

send is a function to write data to a socket, but for demo purposes, we'll just make it an alias to putStrLn which is the Haskell function that writes a line to the stdout stream. With all that in place, we can easily evaluate expressions in the REPL
> write (arr [1, 2, 3])
[
1,
2,
3
]

Notice that b/c Haskell evaluates expressions lazily, the argument to arr is consumed one item at a time as shown below
write (arr [1, 2, 3]) =
write ("[" : go [1, 2, 3]) =
send "["
write (go [1, 2, 3]) =
write ((json 1 ++ ",") : go [2, 3]) =
write ("1," : go [2, 3]) =
send "1,"
write (go [2, 3]) = ...

So if the example list [1, 2, 3] was actually a sequence of rows streamed from the DB, all would still work as expected.
Python implementation
Now you'd think you'd have an easy time translating the above executable spec into Python. In fact, Python 3 comes with iterators and generators that were pretty much modelled after Haskell lists. But I couldn't come up with a quick way of doing that, so I gave up and hacked together something similar which is still based on iterators & generators but it outputs an extra JSON null value as last element of the streamed array. So for example, the Python algo implementation would produce
[
1,
2,
3,
null
]

when given an iterable containing 1, 2, 3.

@pooja1pathak @NEC-Vishal, the TODO in the code asks if there's any easy way to get rid of that extra JSON null value in the Python implementation that doesn't take more than a couple of lines of code to write.
Also another thing we can look into is streamlining the JSON response format, maybe adopt one of the formats Wikipedia mentions

https://en.wikipedia.org/wiki/JSON_streaming


[Bug] Tutorial has old api

Describe the bug
https://developer.tbd.website/blog/issue-verifiable-credential-manually/ is linked from readme, but mentions issuerKid which is no longer valid.
To Reproduce
Try the tutorial
I think we can have a more end to end flow in the readme as well as the tutorial (I will look at this as well).

the issuerKid property in the CURL examples is now verificationMethodId

We've discussed this in the past, the tutorials need to point to a specific version of the service or this will keep happening. I'll update the tutorial and point it to the latest commit hash in https://github.com/TBD54566975/developer.tbd.website

@decentralgabe tutorials yes, but ideally not a blog (which is a point in time post, and will naturally get stale).

We've removed references to this post in the service.


Disable re-sync for deploy.sh

Description
Disable re-sync when deploying using deploy scripts. It should work for both k8s and OpenShift.
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

Testing Performed
Many for the last 6 months

The same CSVTest test seem to be failing for all other PRs in this job.


Add extension to force generated classes to implement additional interface(s)

Original author<EMAIL_ADDRESS>(July 08, 2012 11:47:30)
It may be convenient to ask generator to place super class (or interface) reference into generated DTO, for instance
Schema
{
...
"javaType":"model.json.Buffer",
"javaSuper":"model.AbstractModel",
"javaInterfaces": ["java.io.Serializable","api.CustomContract"],
"properties":{
"id":{
"type":"integer",
"id":"id"
},
"title":{
"type":"string",
"id":"title"
}
...
"type":"object"
}
Generated
...
public class Buffer extends AbtractModel implements Serializable, CustomContract {
private int id;
private String title;
...
It may also generate stub implementations of interface methods automatically if any.
Original issue: http://code.google.com/p/jsonschema2pojo/issues/detail?id=60

Can someone please provide an example how to add already existing class into classpath with gradle? Thanks

Hi! Does this feature exist in the maven-plugin? I cannot seem to find it here.


Explaining sklearn tree prediction

Based on the treeinterpreter decision paths idea, this adds support for explaining prediction of ExtraTreesClassifier, ExtraTreesRegressor, GradientBoostingClassifier, GradientBoostingRegressor, RandomForestClassifier, RandomForestRegressor.
AdaBoostClassifier, AdaBoostRegressor are not supported yet: the code runs but produces suspicious results, I would rather check them separately if I won't manage to make them work soon :)

Current coverage is 97.09% (diff: 94.78%)

Merging #122 into master will decrease coverage by 0.16%

@@             master       #122   diff @@
==========================================
  Files            34         35     +1
  Lines          1792       1895   +103
  Methods           0          0
  Messages          0          0
  Branches        342        364    +22
==========================================
+ Hits           1743       1840    +97
- Misses           24         26     +2
- Partials         25         29     +4

Diff Coverage
File Path

•••••••••  94%
eli5/sklearn/explain_prediction.py

•••••••••• 100%
eli5/sklearn/utils.py

•••••••••• 100%
new eli5/_decision_path.py

•••••••••• 100%
eli5/xgboost.py

Powered by Codecov. Last update 0954d7c...5d18555

@kmike I implemented AdaBoostRegressor support but then decided that it's better to leave it out for now, because it selects a tree with median score and uses it for prediction. So we report it's weights and they sum to score, but it's misleading, because only few features will be in this tree, and other trees also influenced the prediction. Another option is to report features from all trees, but it's again misleading. So it requires a ton of special caveats and care.
AdaBoostClassifier has even more complex prediction function that comes in two variants, and though it doesn't have this problem with median estimator, but it will be hard to support it outside of scikit-learn.
So I would rather leave these two estimators out for now. Apart from that, everything is ready.

Looks great, thanks @lopuhin!

Thanks @kmike !


main.js:149 Uncaught TypeError: Failed to execute 'define' on 'CustomElementRegistry': The callback provided as parameter 2 is not a function.     at main.js:149 Uncaught TypeError: note.setMessage is not a function     at StickyNotesApp.displayNote (main.js:74)     at new StickyNotesApp (main.js:34)     at main.js:88

Can this be fixed asap?


[release-1.17] fix CSR signing panic

Manual cherrypick required.
#42121 failed to apply on top of branch "release-1.17":
Applying: fix CSR signing error
Using index info to reconstruct a base tree...
M	security/pkg/pki/ra/k8s_ra.go
Falling back to patching base and 3-way merge...
Auto-merging security/pkg/pki/ra/k8s_ra.go
CONFLICT (content): Merge conflict in security/pkg/pki/ra/k8s_ra.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 fix CSR signing error
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

HI @johnma14
The fix has been the release 1.17 branch.  https://github.com/istio/istio/blame/release-1.17/security/pkg/pki/ra/k8s_ra.go#L142


Allow build time variables VUE_APP_*

There is an issue when using latest release all VUE_APP_* variables coming from .env ignored.
To make this work we checkout dispatch, set context to ../dispatch and add these variables to ../dispatch/src/dispatch/static/dispatch/.env
To reproduce:

create .env file
add VUE_APP_DISPATCH_AUTHENTICATION_PROVIDER_SLUG=dispatch-auth-provider-pkce
run ./install.sh
debug in browser show undefined for VUE_APP_DISPATCH_AUTHENTICATION_PROVIDER_SLUG value and dispatch falls back to basic auth 🤷🏻♂

This is because docker uses env file only during run process, not build process.
Additional change is required on Dockerfile side: https://github.com/Netflix/dispatch/pull/1533

@kevgliss 🙏

lgtm thanks!


make test: Don't track NOTICEs

There's little value in tracking the noise of NOTICEs emitted during loading pg_sphere.test.sql, so just mute them.
The gain from this change is that future patches will not have to update the changed line numbers in the output tracked. (Cf. c2a235160)
\set ECHO all at the end of the file can be removed since a new psql session will be created for the next file anyway.

Cool. Thank you!

Rebased to master.


update readme

@CodiumAI-Agent /review

@CodiumAI-Agent /review

PR Review

 ⏱ Estimated effort to review [1-5]
1, because the PR contains only minor text corrections in a Markdown file, which are straightforward and do not involve complex code changes.

 🧪 Relevant tests
No

 🔍 Possible issues
No

 🔒 Security concerns
No

 Code feedback:
relevant fileReadme.md
suggestion

Consider using consistent terminology for "bartending person" and "bartini guys." It might be better to use "bartenders" for both instances to maintain professionalism and clarity. [medium]

relevant lineIt's a destination where you find all types of drinks also Mocktails (for non-alcoholic person). We have a collection of all types of Cocktail from Ancient to Modern. We also think the cooperation and community have great power. So, experience bartending person also share their own recipe/tricks for new bartini guys.

 ✨ Review tool usage guide:
Overview:
The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.
The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

See the review usage page for a comprehensive guide on using this tool.


Unauthorized message when publishing module to container registry

Following the documentation, I get the following error message when I try to push my bicep module to the registry:
Publish-AzBicepModule: Unhandled exception. Azure.RequestFailedException: Service request failed.
Status: 401 (Unauthorized)
System.Management.Automation.RemoteException
Content:
{"errors":[{"code":"UNAUTHORIZED","message":"retrieving permissions failed"}]}
System.Management.Automation.RemoteException
Headers:
Server: openresty
Date: Mon, 07 Feb 2022 09:56:53 GMT
Connection: keep-alive
X-Ms-Correlation-Request-Id: e3a73ff3-0c81-4aa4-a34f-220b69b454d5
x-ms-ratelimit-remaining-calls-per-second: 166.65
Strict-Transport-Security: REDACTED
Content-Type: application/json
Content-Length: 78
I'm running pwsh core 7.2.1.
I have owner and AcrPsuh permissions on the ACR resource.
I've set the context to the proper subscription.
Searching for a solution, I found a suggestion to logon to the ACR first. But, when I tried to run 'connect-azcontainerregistry' locally on my laptop, I get the error:
Connect-AzContainerRegistry: The term 'docker' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
However, when I try to push to the registry using Cloud Shell and first run 'connect-azcontainerregistry', the command succeeds.
Is this step (connect-azcontainerregistry) missing from the documentation?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 41371dc7-e64d-9169-16da-12d1fe463a40
Version Independent ID: 5e8d9278-d73c-6399-0ff6-6da2961ce054
Content: Create private registry for Bicep module - Azure Resource Manager
Content Source: articles/azure-resource-manager/bicep/private-module-registry.md
Service: azure-resource-manager
Sub-service: bicep
GitHub Login: @mumian
Microsoft Alias: jgao

@GitHubUser1339 Thanks for your comment. We will review the issue and get back to you shortly.

@GitHubUser1339  Have you got chance to look into the previous comment by mumian ? Do revert if you have further questions.

@GitHubUser1339 As we haven't heard from you, we will close the issue for now. If there are further questions , please revert and we will be glad to assist you.

@mumian - apologies for the late response. I just tried your suggestion and these are the results:

Get-AzContainerRegistry -> lists my ACR
Publish-AzBicepModule -> WARNING: Unhandled exception. Azure.Identity.AuthenticationFailedException: The ChainedTokenCredential failed due to an unhandled exception: Azure PowerShell authentication timed out.
---> Azure.Identity.AuthenticationFailedException: Azure PowerShell authentication timed out.
at Azure.Identity.AzurePowerShellCredential.RequestAzurePowerShellAccessTokenAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)

When I do the same from a Cloud Shell it works.

@DonKoning Which version of Bicep CLI are you using? Also could you try running Disconnect-AzAccount and Connect-AzAccount to refresh the auth token?

Bicep CLI version 0.4.1124 (66c84c8ee5)
In Windows Terminal I first connect-azaccount, switch to the right context (as owner), then run the Publish-AzBiccepModule command.
Again, I get errors:

Again, running the same command in Cloud Shell succeeds.

When you ran Get-AzContainerRegistry it listed your ACR and it throwed a time out error which is different. Since it works in cloud shell, I think this has something to do with your local Azure PowerShell modules. Do you mind trying the following steps again?

Run Disconnect-AzAccount
Run Connect-AzAccount and select context
Run Get-AzContainerRegistry to check if it lists your ACR
Run Publish-AzBicepModule with the -Debug switch

If it still doesn't work, can you open an issue in the Bicep repo and copy and paste the output from the last step?

I had the same issue. I got it resolved by also doing az login and az account set --subscription <subId> after the Connect-AzAccount

I too am having issues with this (via Powershell 7.3.2. I followed the instructions from this page: https://learn.microsoft.com/en-us/training/modules/share-bicep-modules-using-private-registries/3-exercise-create-registry?pivots=powershell
Everything in that phase completes without issues, and I'm on the Concierge sub. However, if I try and publish the template as the 5th page demonstrates, I get this error:
bicep publish website.bicep --target 'br:ssllcregistry909606.azurecr.io/website:v1'
Unable to publish module "br:ssllcregistry909606.azurecr.io/website:v1": Service request failed.
Status: 401 (Unauthorized)
Content:
{"errors":[{"code":"UNAUTHORIZED","message":"retrieving permissions failed"}]}
Headers:
Server: openresty
Date: Sun, 19 Feb 2023 18:18:18 GMT
Connection: keep-alive
X-Ms-Correlation-Request-Id: 0a20c78a-080c-4256-ab46-848e27950cf0
x-ms-ratelimit-remaining-calls-per-second: 166.65
Strict-Transport-Security: REDACTED
Content-Type: application/json
Content-Length: 78
Furthermore, if I do Get-AzContainerRegistry, I get this:
Registry Name     Sku        LoginServer               CreationDate               Provisioni AdminUserE StorageAccountName
ngState    nabled

ssllcregistry909… Basic      ssllcregistry909606.azur… 2/19/2023 6:15:40 PM       Succeeded  False
It's a bit hard to read that, but I'm wondering if because in the "AdminUserEnabled" column it says "false" it means I don't have access? I don't get it. Every other example so far has worked up until now. I tried shenglol's suggestions above but it doesn't work. Here is the output from the -Debug switch when using Publish-AzBicepModule:
ublish-AzBicepModule -Debug
cmdlet Publish-AzBicepModule at command pipeline position 1
Supply values for the following parameters:
(Type !? for Help.)
FilePath: templates\6-website.bicep
Target: br:ssllcregistry909606.azurecr.io/website:v1
DEBUG: 1:22:38 PM - PublishAzureBicepModuleCmdlet begin processing with ParameterSet '__AllParameterSets'.
DEBUG: 1:22:38 PM - using account id<EMAIL_ADDRESS>DEBUG: 1:22:38 PM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
VERBOSE: Using Bicep v0.14.46
Confirm
Continue with this operation?
[Y] Yes  [A] Yes to All  [H] Halt Command  [S] Suspend  [?] Help (default is "Y"): a
DEBUG: 1:22:48 PM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].
Publish-AzBicepModule: Unable to publish module "br:ssllcregistry909606.azurecr.io/website:v1": Service request failed.
Status: 401 (Unauthorized)
System.Management.Automation.RemoteException
Content:
{"errors":[{"code":"UNAUTHORIZED","message":"retrieving permissions failed"}]}
System.Management.Automation.RemoteException
Headers:
Server: openresty
Date: Sun, 19 Feb 2023 18:22:42 GMT
Connection: keep-alive
X-Ms-Correlation-Request-Id: 854dc143-3870-4cd5-95f3-c267521b6541
x-ms-ratelimit-remaining-calls-per-second: 166.65
Strict-Transport-Security: REDACTED
Content-Type: application/json
Content-Length: 78
System.Management.Automation.RemoteException
DEBUG: 1:22:48 PM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: AzureQoSEvent:  Module: Az.Resources:6.5.1; CommandName: Publish-AzBicepModule; PSVersion: 7.3.2; IsSuccess: False; Duration: 00:00:09.5650752; Exception:
Unable to publish module "br:ssllcregistry909606.azurecr.io/website:v1": Service request failed.
Status: 401 (Unauthorized)
System.Management.Automation.RemoteException
Content:
{"errors":[{"code":"UNAUTHORIZED","message":"retrieving permissions failed"}]}
System.Management.Automation.RemoteException
Headers:
Server: openresty
Date: Sun, 19 Feb 2023 18:22:42 GMT
Connection: keep-alive
X-Ms-Correlation-Request-Id: 854dc143-3870-4cd5-95f3-c267521b6541
x-ms-ratelimit-remaining-calls-per-second: 166.65
Strict-Transport-Security: REDACTED
Content-Type: application/json
Content-Length: 78
System.Management.Automation.RemoteException;
DEBUG: 1:22:48 PM - PublishAzureBicepModuleCmdlet end processing.

Can this issue be reopened? Problem still exists

Same issue, doing az logout before Publish-AzBicepModule resolved

I had the same issue today using the following:
Az: 11.2.0
Az.Accounts: 2.15.0
Az.ContainerRegistry: 4.1.2
az cli version: 2.57.0
I couldn't push to ACR on which I am owner with the UNAUTHORIZED 401 message, while Connect-AzContainerRegistry was providing me with Login Succeeded message.
I had to logout my az cli with "az logout"
After that I had to run Connect-AzContainerRegistry again and then I could publish successfully, so it seems to be something with preventing Azure PowerShell to get the access token from the dockerconfig?

Please reopen. Furthermore, the disconnect between the tutorial (as I posted above on Feb 19, 2023) and the ability to execute it is a pretty big obstacle for anyone new and attempting to try out Bicep. Reopening would at least serve as an acknowledgement of the issue. It seems repeatable as well: follow the steps in my prior post regarding following the tutorial and running into said issue. That and people keep posting here that the issue still exists.

Try configuring a profile in bicepconfig.json per the Microsoft instructions here:
https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-config-modules#configure-profiles-and-credentials

Also with configuring a profile the problems exists


Updates to latest zipkin/brave

Zipkin now has Span.timestamp/duration, so no more need for
"aquire/release".
This also corrects the host/endpoint logged for timeline annotations, as
it should always be the local process.
See https://github.com/openzipkin/zipkin/blob/master/zipkin-thrift/src/main/thrift/com/twitter/zipkin/zipkinCore.thrift

No more fake acquire/release annotations needed when not using core annotations.

:+1:


Structured data retrieval

Use cases for structured data retrieval:

Calendar queries
Database info
Ability to extract data from unstructured documents via Google DocAI

Gonna use the new Vertex plugins


Test case for #3950

Test case for #3950 as requested by @vasily-kirichenko. I assume it passes, but can't run these tests on OSX.

Thanks! :)


Is this supposed to be a PR, or a suggestion? Can you provide more detail?


Selection bug

Fixes #1765
Please check my logic but I think this is the correct fix

Fixed branch


Feature/fdk 1103

Jira issue link: link

check applicable boxes

[ ] I have made tests to match the user stories
[ ] This code does not require tests that match user stories

Coverage increased (+2.6%) to 65.556% when pulling 59f50a576313a7bcae99b8c99d9836d536101d5e on feature/FDK-1103 into a444c8c5318d85577ed4431cc6b90994e95b5d98 on develop.


Allow owning user and group of hyperdrive folder as well as force flag for install

Needed these options as I have a setup with a privileged and unprivileged user, and the current setup needs sudo but I want to use hyperdrive from my unprivileged user. I also added a force flag, since all steps seem to be idempotent (correct me if I'm wrong)

Yep the steps are all idempotent, so this looks solid! Merging


Add Brazilian Portuguese translation (pt-BR)

Subject says it all :) . If anything else needs to be done regarding this, please let me know.

Hi,
Super thanks, I've merged these, but for future updates, could you use https://crowdin.com/project/zotero-better-bibtex? My translation workflow uses that toolchain.


tests/VectorException fails sporadically in CI

Specifically in the Kabylake configuration, we've seen the VectorException test fail a couple of times when running CI that recovers on rerun:
=== This program is used to test basic vector exception functionalities.
TestCpuidInGlobalConstructors: completed successfully.
TestVectorException: will generate a hardware exception inside enclave!
TestVectorException: hardware exception is handled correctly!
TestGetsecInstruction stack parameters are ok.
Success-Illegal GETSEC raised 2nd chance exception.
The value of cpuidRAX is now: 8
.Success-Unsupported CPUID leaf 00000008 raised 2nd chance exception.
The value of cpuidRAX is now: -2147483648
.Success-Unsupported CPUID leaf 80000000 raised 2nd chance exception.
The value of cpuidRAX is now: 21
.The value of cpuidRAX is now: 591593
.The value of cpuidRAX is now:<PHONE_NUMBER>
.The value of cpuidRAX is now: 469762337
.TestSigillHandling: completed successfully.
Test failed: /home/jenkins/workspace/Bors_staging-SAVQQOGG4TMIJW6ZHSGG2MHERTL45R6RNECLBYNXEIQ6O4GKH54Q/tests/VectorException/host/host.c(78): TestSigillHandling cpuid_info[j] == args.cpuid_table[i][j]

The reason the test fails is due to the use of cpuid with leaf 0, which returns the current executing processor (initial APIC id) in the higher 8 bits of the ebx register.
When the enclave is created, we compute the cpuids on the host side and cache it in the enclave.
The cached value of cpuid.1 will have the current processor id in ebx.
At a later point, the OS could schedule the enclave to execute in another processor, at which point cpuid.1 when called from the host will have the new processor id where as the cached value in the enclave will have the old processor id.
The test performs such a comparison, and sporadically when the enclave gets scheduled across multiple processors, the test fails.
If is possible to reproduce this by adding a sleep(20) after enclave creation, then changing the affinity of the process via (taskset -p 1,1) so that when the process resumes, it is scheduled to a different processor.

Note that intel caches leaves 0, 1, 4, and 7, so they should have the same problem too:
https://github.com/intel/linux-sgx/blob/984c04c696bddaff7fb1621b79760fabcb8c94d7/psw/urts/cpu_features.cpp

Excellent info Anand.  The test should be modified to ignore the APIC id 8-bits in the EBX register or just issue a message if the comparison fails, i.e. no fail the test.

Anita

Excellent info Anand.  The test should be modified to ignore the APIC id 8-bits in the EBX register or just issue a message if the comparison fails, i.e. not fail the test.

Anita

Fixed by #905.


Add work item user friendly API url to space links [Follow up #1490]

Expected behavior
In this issue #1490 we created a user friendly work item URL but there is no way to reach that URL in UI without constructing. The link section of space response should have that URL so that we can use it to construct api URL to get a particular work item.
Something like,  /api/namedspaces/:user_name/:space_name/workitems/{%s}. So, the UI will just replace the {%s} with a work item number.
Actual behavior
No such link is provided

@sanbornsen can you please update this issue based on the discussion we had?


[QUERY] Blobs saved as octet-stream instead of image/jpeg

Query/Question
Given IFormFile in a ASP.NET controller
[HttpPost]
public async Task<IActionResult> Upload(IFormFile formFile, CancellationToken cancellationToken)

and blob upload using new ContainerClient
var containerClient = blobServiceClient.GetBlobContainerClient("foo");
await containerClient.UploadBlobAsync(formFile.FileName, formFile.OpenReadStream(), cancellationToken);

My files end up being saved as
application/octet-stream

instead of
image/jpeg

I've verified that the content I'm trying to upload has the correct content-type set (image/jpeg in this case)
formFile.ContentType; //image/jpeg

Is this behavior expected?
Environment:

Name and version of the Library package used: Azure.Storage.Blobs 12.9.0-beta.1
Hosting platform or OS and .NET runtime version :  Windows 10 .NET Framework 5
IDE and version : JetBrains Rider 2021.1 EAP 6

Thank you for your feedback.  Tagging and routing to the team best able to assist.

@charconstpointer, you need to set the content type of the blob yourself, it is not autodetected.
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("foo");
BlobClient blobClient = containerClient.GetBlobClient(formFile.FileName);
BlobUploadOptions options = new BlobUploadOptions
{
    HttpHeaders = new BlobHttpHeaders
    {
        ContentType = "image/jpeg"
    }
};
await blobClient.UploadAsync(formFile.OpenReadStream(), options);

-Sean


[Maintenance] Fix multi-environment .env files in test application

Thanks, Jan! :1st_place_medal:


LC003 bugfix for false positive on objects with namespaces

This bugfix should restore the behaviour prior to the introduction of namespaces. Currently when you specify a full qualified object name the rule always triggers.
With runtime 12 a node can contain multiple tokens with namespaces but the last token in this context should always be the object name.

Thank you for the contribution!


Compare LDAP users in CLA groups with actual contributions

We do have some users in EasyCLA groups in LDAP which doesn't seem to be authorized by EasyCLA itself. That may be an issue because users in LDAP should ideally match with authorized users in EasyCLA. However, there is a chance that these users were added to LDAP CLA groups through different channels or they were not properly removed by older version of EasyCLA.
What needs to be done:
We need to review these LDAP users in EasyCLA groups which are not authorized by EasyCLA itself if they ever contributed or if they are even contributing now.
List of LDAP users in EasyCLA groups which not authorized by EasyCLA:

ONAP
OPNFV
ORAN-SC (OSS)
ORAN-ASC

We need to check if these users contributed (or are contributing) to Gerrit servers:

gerrit.onap.org
gerrit.opnfv.org
gerrit.o-ran-sc.org

We need to know:

If these users ever contributed
what kind of contribution they did (we mostly care only about code contribution)
when was last contribution
how much they contributed
to which server and repo they contributed to

See #4348 for more details.
This is related to #4394.

On it as requested by @mlehotskylf

Unique users from those 4 files (to check):
('AlexandruAvadanii', 'BFrazer', 'ChristopherPrice', 'GaoSong', 'HWillson', 'HuabingZhao', 'InoueReo', 'Isaac.manuelraj', 'Itohan', 'IvanADAM', 'Jallolo', 'Katel34', 'KennyPaul', 'LucProvoost', 'MagnusB', 'Manamohan', 'MattDavis', 'MehreenKaleem', 'Nagendra90287', 'PANTHEON.tech', 'Pavithra', 'Pooja03', 'PremkumarAarna', 'Ray_NTUST', 'RehanRaza', 'SandeepAarna', 'SantoshB', 'SindhuXirasagar', 'SnehaD', 'SunilB', 'ThamlurRaju', 'TianL', 'VincentDanno', 'YCJict', 'YatianXU', 'YoonsoonJahng', 'a.czajkowski', 'adetalhouet', 'akanshaDua', 'akapadia', 'aleemraja', 'aleksandrtaranov', 'alexeyaleynikov', 'allison4nordix', 'alokbhatt', 'amitagh', 'andreasgeissler', 'anil1', 'ankitbhatt', 'aribeiro', 'arjunmgupta', 'atassi', 'babejmat', 'bdavis', 'bdfreeman1421', 'bhagyalakshmi', 'bhedstrom', 'brilldav', 'brindasanthm', 'chaitanyakadiyala', 'chsailakshmi', 'codechinatelecom', 'cramstad', 'cryptomaster', 'cschowdam', 'dacher', 'dafuse', 'daniesilamdocs', 'debbiemedina', 'demx8as6', 'deswali07', 'djhunt', 'ediaz101', 'enyinna1234', 'ezhil', 'fpaquett', 'francistoth', 'frank123', 'fujihiro16', 'fzhang', 'ggarudapw', 'gmittal', 'gordonkoocommscope', 'gregory.hayes', 'gseiler', 'guyjacobson', 'halil.cakal', 'hujie', 'hwcm', 'ilanap', 'jamesgu', 'jingjincs', 'jkbecker', 'jsulliva', 'kaihlavi', 'kamezawa', 'karbon', 'kavi2021', 'kbanka', 'kevin.brown1viavi', 'ktimoney', 'kwiatrox', 'liuwenyu', 'mabelgaumkar1', 'manoj1', 'marian.vaclavik', 'mbrunner', 'mdolan', 'melliott', 'mharper', 'michal', 'mit3301', 'mizunoami123', 'nandkumar', 'neelesh.durgapal', 'niharika.sharma', 'o-ran-sc-release', 'ojasdubey', 'onapbot', 'pau2882', 'pceicicd', 'pleigh', 'prabhjot', 'preethams', 'projitaarna', 'rajeevme', 'rajiv.v', 'ramagp', 'rannyh', 'ranvijays', 'ravi.setti', 'ravikanth.p', 'ray.ntust', 'rgadiyar1', 'rp5811', 'rsrinivas', 'rsriraman', 'sainiashok', 'sanchitap', 'sanjaymekhale', 'sblimkie', 'sdevaraj665', 'seshukm', 'shalomb', 'shangyuxiang', 'shaoqiu', 'sharathprakash', 'shormancorigine', 'shrek2000', 'singh.sunil', 'singhrishipratap', 'sitedata', 'sridharkn', 'ssteve', 'subhash_singh', 'sudhakar.ndc', 'sumitc29', 'sunqiong.bri', 'surajchalapathy', 'swaminathans', 'swapnalipode', 't.seshu', 'talig', 'talio', 'thakurveerendra', 'tperala', 'tragait', 'vamshi.nemalikonda', 'vharish', 'vikaskumar', 'vikram.barate.gslab', 'vivemuthu', 'vmuthukrishnan', 'vvarvate', 'wangy122', 'wanyama', 'ychacon', 'yingyingwang', 'yogendrapal', 'z00245565')

Now guessing starts: - is this "displayName" or username" or what?

I will assume this is LFID because this is the only one that is supposed to be unique, other types are related to a specific platform, so I cannot guess which platform it is...

There are 180 distinct names across those 4 links, but when I search for them in fivetran_ingest.crowd_prod_public.memberidentities by LFID I'm getting 95 hits, and when I'm checking by gerrit platform username I have 125 hits:

When checking by member's displayname I have 81 hits and when checking by all possible username value regardless of platform I'm getting 155 hits. So nothing I can gives me all 180 users.

I will use the one giving biggest number of hits and then generate result for those users (I will list what member ID it is and what username/platform hit)

I will provide report for all hits in any of those 180 usernames - for all platforms - you can later discard anything that is not needed (report will contain both username and platform and will be ordered by platform).

Report generated
Also attaching CDV here:
easycla-gerrit.csv
Please read the Info: in the generated google spreadsheet, also pasting it here:
I've found any of those members by looking for "username" in memberidentities field for any platform
The exact criteria that foudn given member is in "Found via ..." column
Then we consider member ("Member Id" column) that any such identity belongs to
And for that member we are generating report for all its identities/platforms that has any activity
On requested gerrit servers, those are "Platfrom", "User Name" columns
Four each such identity we also group by "Activity type" (as requested) and display
first/last activity date, number of given type activities, and which gerrit servers identity contributed to
We order this by platform, username, activity type and member Id
At the bottom there are rows where only "user Name" is set - those are rows
For which usernames I've found no hits in member identities - meaning those users didn't contributed anything (not only to given gerrit servers, but overall)

cc @mlehotskylf @dealako

@lukaszgryglicki I had a question regarding the platforms. I see some platforms are tagged as git rather than gerrit . Would this make a difference ?

Noted . I asked because of the context of gerrit specifically

Working on version that only searches by gerrit username as requested by @mlehotskylf - I'll add this new version as a separate tab in the already exiting google sheet.

Report updated added sheets: Report by gerrit username only and SQL - by gerrit username only.
Also uploading CSV here:
easycla-gerrit-v2.csv
cc @mlehotskylf

Also uploading CSV here:
easycla-gerrit-v2.csv


not ok with android support appcompat v7:26.1.0

Dear TangoAgency,
Your material-intro-screen has error with
dependencies {
...
implementation 'com.android.support:appcompat-v7:26.1.0'
...}
please, kindly fix it

What errors :)?

It isn't related to our library.

As you can see we use 25.0.1 version of the Android Support Libraries.

To run I added the following
implementation 'com.android.support:appcompat-v7:27.1.0'
implementation 'com.android.support:design:27.1.0'


[New] no-duplicates: support inline type import with prefer-inline option

close https://github.com/import-js/eslint-plugin-import/issues/2470
ref https://github.com/typescript-eslint/typescript-eslint/pull/5050
Typescript import { type A } from 'util/cool-linter' was introduced in 4.5 https://devblogs.microsoft.com/typescript/announcing-typescript-4-5/#type-on-import-names
This PR seeks to add TypeScript support to the existing rule no-duplicates.
❌ Invalid `["error", "prefer-inline"]`

import { AValue, type AType } from './mama-mia'
import type { BType } from './mama-mia'

✅ Valid with `["error", "prefer-inline"]`

import { AValue, type AType, type BType } from './mama-mia'

Would it be possible to extend this rule to also test exports?

@jasongerbes Great q! I'm not seeing inline export type throw any warnings so I'm guessing it is possible!  I didn't see any mention in the release notes.  Either it is an option or a different rule.  After this PR is merged, I imagine we could consider it then.  I don't have an answer right now.
export { type A };

cc @ljharb @bradzacher This is ready for your review.

Hi @ljharb!  Just seeing if I could get another friendly review when/if you have time!

@ljharb Thanks for the reviews and comments!  So this PR is part of a larger effort.

fixes a Type import to Inline-Type via prefer-inline or prefer-top-level - https://github.com/import-js/eslint-plugin-import/pull/2473.
Fixes a Value import to Inline Type - https://github.com/typescript-eslint/typescript-eslint/pull/5050.
(this PR) adds support for resolving duplicate Type imports to Inline Types to a single import source.

I think 1) is what you are referencing, correct?

I updated to prefer-inline to be consistent with https://github.com/import-js/eslint-plugin-import/pull/2473.  Does that seem ok?

Yep, 1 is what I'm referencing

I'm not worried about the node 6 travis failure (it's due to npm 6 breaking it, and travis not yet updating nvm to work around it) nor about the uncovered line.


Prerequisite to move a work item to another project missing

Hi!
I was trying to help a co-worker moving work items between project looking at his permissions he seemed to have everything needed including the Move work items out of this project permission but it didn't work. Since the audit logs didn't show that kind of error message I had to look elswhere. I finally figured that he needed more than the stakeholder license to do the move. It worked for me (Visual Studio Subscriber) and for him when I set his license to Basic.
I know there's a documentation page listing all the differences between licenses but it should be more clear on a page like this one here when you have a prerequisite section. Even on the access level page it's not entirely clear on this subject.
Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e6d7da26-5052-70ad-4b2a-7ccd2f02f012
Version Independent ID: 7a0b1331-7ac9-9927-b96e-36c6f014391a
Content: Move work items and change the work item type in Azure Boards - Azure Boards
Content Source: docs/boards/backlogs/move-change-type.md
Product: devops
Technology: devops-agile
GitHub Login: @KathrynEE
Microsoft Alias: kaelli

@Bliyaal - Thank you for your inquiry and feedback. Stakeholders don't have access to the Move work items to another project. I will update the article later today to clarify this restriction.


Pictogram is too small in pictogram item

Detailed description
** Describe in detail the problem you're having**
The pictograms are displaying too small in the pictogram item, the specs show the pictogram at 80px
Recently James discovered that carbon changed pictogram sizes from 32px to 48px, it looks like the canvas size was increase but the artwork was not increased.
We might need to re-evaluate the size of the pictograms and adjust per the new canvas/artwork size
OR
Submit a feature request to Carbon to remove/change the excess canvas size to artwork.
Is this a feature request (new component, new icon), a bug, or a general issue?
design defect (might be a feature request to Carbon team)
Is this issue related to a specific component?
Pictogram item
What did you expect to happen? What happened instead? What would you like to see changed?
Intended design

What's happening

What browser are you working in
Chrome

What version of the IBM.com Library are you using?

What offering/product do you work on? Any pressing ship or release dates we should be aware of?
Web simplification
Steps to reproduce the issue

Step one
Step two
Step three
etc.

Additional information

Screenshots or code
Notes

Intended Design

Current

@wonilsuhibm  There are implications if dev adjusts the size differently than Carbon.  Engineering agrees that the fix may require changes on the Carbon side.  A design decision is needed around whether less padding is needed.  Engineers feel this is not a bug, but a feature request.

Link to the issue on Carbon board to track the problems we hare having with pictograms.

Update
I spoke with Russ, Conrad and Josh on March 20, they explained that the SVGs are output with extra padding for non-power users that might be using the pictograms within Powerpoint.
They agreed that there is still a need for pictograms to have no padding and are working on how they can implement that.

New issue number on the Carbon board

Dependency on Carbon.  Moving to icebox to wait until they have available resources to work on it.

@oliviaflory @jeffchew did this one get resolved?
https://percy.io/538fc19a/Carbon-for-IBM.com-React/builds/7957132?utm_campaign=538fc19a&utm_content=Carbon-for-IBM.com-React&utm_source=github_status_public


Added scikit-learn

[x] I followed the contributing guidelines: https://github.com/github/explore/blob/master/CONTRIBUTING.md

I am:

[ ] Suggesting edits to an existing Topic page
[x] Curating a new Topic page

CURATING A NEW TOPIC PAGE

[x] I've formatted my changes as a new folder directory, named for the topic as it appears in the URL on GitHub (e.g. https://github.com/topics/[NAME])
[x] My folder contains a *.png image (if applicable) and index.md
[x] All required fields in my index.md conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

Please explain why you think this Topic page should be curated:
A lots people build their libraries upon sklearn extending its functionality.
Please note that there is no logo added. The logos are in the repo, https://github.com/scikit-learn/scikit-learn/tree/master/doc/logos , the repo seems to be under BSD, but I don't want to take any legal risks about it, because they may say "the code is under bsd, but the artwork is proprietary".

@KOLANICH Gentle ping on the above comments, thanks.

@KOLANICH Any chance you'd consider reopening this? It's a good PR that it would be a shame to lose just over a non-working HTTPS page. Thanks ❤!

No, I'm not going to promote using non-encrypted non-authenticated insecure protocols in 2018. If you wanna promote it, feel free to take the files from my PR, modify them as you like and post your own PR attributing it to yourself. But I'm not going to be a part of that PR.

@KOLANICH Ok, disappointing but up to you.

Opened https://github.com/github/explore/pull/162.


Building multiple permutations of bundle

Describe the problem
Concept
A permutation is a way to build multiples variants of the same Svelte application, according to various criteria.
A permutations will have an impact on the generated code by allowing certain files to be 'replaced' by others at build time.
This will allow to have a common codebase for multiple target, each of which could have its own specificities, like dedicated CSS/components/code/ressources/pages...
Some use-case :

Sites for different audiences.
Sites for different localisations.
Sites for different clients, with specific template/customisation.

Describe the proposed solution
Permutations keys
Permutations would be defined by custom keys and their values, declared on the svelte.config.js
/// svelte.config.js
permutations: {
	keys: {
		/* any records as name: string[] */
	}
},

Permutation keys will be project-specific key/values that define variants of our project.
For example, if we want to use a common project to build 3 specifics site/app for our customers (b2c), our partners (b2b), and our internal teams (int), we could define something like this :
/// svelte.config.js
permutations: {
	keys: {
		target: ['b2c', 'b2b', 'int']
	}
},

And if, in addition to that, we want to generate 4 differents languages ('en', 'fr', 'de', 'it'), we could use :
/// svelte.config.js
permutations: {
	keys: {
		target: ['b2c', 'b2b', 'int'],
		lang: ['en', 'fr', 'de', 'it' ]
	}
},

We have now 3 targets variants in 4 differents langs, which will produce 12 distincts permutations builds :

b2c,en
b2c,fr
b2c,de
b2c,it
b2b,en
b2b,fr
b2b,de
b2b,it
int,en
int,fr
int,de
int,it

By default, each build will have his own subdirectory in the svelte-kit folder :
.svelte-kit/
    b2c,en/
        generated/
        output/
        types/

    b2c,fr/
        generated/
        output/
        types/

    ...

    int,de/
        generated/
        output/
        types/

    int,it/
        generated/
        output/
        types/

And all npm run command will use the first-permutation ('b2c,en' in the current case) :
# Launch dev mode for permutation 'b2c,en' (default)
npm run dev

But we might use a flag to select another permutation, for example --perm :
# Launch dev mode for permutation 'b2c' in 'fr' (explicitly)
npm run dev --perm b2c,fr
# Launch dev mode for permutation 'int' in 'it' (explicitly)
npm run dev --perm int,it

Special case : the check and build task will accept an --all-perms flag to check/build all the permutations :
# Launch check for all permutations
npm run check --all-perms
# Launch build for all permutations
npm run build --all-perms

Permutations files
All files of the project will accept a permutation rules in order to choose the more specific file.
This could be represented by a mark as an extension suffix like .(permutations), juste before the real file extension.
For example for the logical file mycode.js, we can have :

mycode.(b2c,en).js which represents the file for the 'b2c,en' permutation.
mycode.(b2c).js which represents the file for any of the 'b2c' permutations.
mycode.(en).js which represents the file for any of the 'en' permutations.
mycode.js which represents the default file, when any other match.

By using unique permutation name and some naming convention, we can even make more precise targeting.
Ex:

mycode.(b2c,b2b).js represents all the permutations with target 'b2c' or 'b2b'.
mycode.(b2c,b2b,en).js represents all the permutations with target 'b2c' or 'b2b', AND the lang 'en'.
mycode.(-int).js represents all the permutations where target is NOT 'int'

Import file
The permutation should work when we import a file, for example :
<script>
	import Header from './Header.svelte';
	import Footer from './Footer.svelte';
	import messages from './messages.js';
	import './styles.css';
</script>

<Header />
<slot/>
<Footer />

The importeds files will depends from the current permutation and the permutations rules of the file's variants.
For example, if we have theses files :
	Header.(int).svelte
	Header.(b2b).svelte
	Header.(b2c).svelte

	Footer.(b2c,fr).svelte
	Footer.(b2c).svelte
	Footer.(fr).svelte
	Footer.(it).svelte
	Footer.svelte

	messages.(en).js
	messages.(fr).js
	messages.(de).js
	messages.(it).js

	styles.(b2c).css
	styles.(b2b).css
	styles.(intra).css

For permutation 'b2c,en', it will use Header.(b2c).svelte, Footer.(b2c).svelte, messages.(en).js and styles.(b2c).css
For permutation 'b2c,fr', it will use Header.(b2c).svelte, Footer.(b2c,fr).svelte, messages.(fr).js and styles.(b2c).css
For permutation 'b2b,en', it will use Header.(b2b).svelte, Footer.svelte, messages.(en).js and styles.(b2b).css
For permutation 'b2b,fr', it will use Header.(b2b).svelte, Footer.(fr).svelte, messages.(fr).js and styles.(b2b).css
...
The compiler should produce an error if the permutation mark is used on import :
// ERROR : invalid import with a permutation mark
// Should use 'Header.svelte' instead
import Header from 'Header.(b2c).svelte';

Static file
The same rule should apply to the static file.
For example in order to have distinct favicons :
   favicon.(b2b).png
   favicon.(b2c).png
   favicon.(int).png

When serving/building the apps, the file will be renamed with his base name.
Routing
The svelte routing files (+page, +layout, +server) should follow the same rule, with one exception : a missing permutation will not throw an error, but simply consider that the file does not exist.
Example, with this with this route tree :
routes/
     stuff/
         +page.svelte
     news/
         +page.svelte
         +page.(int).svelte
     contact/
         +page.(b2c).svelte
         +page.(b2b).svelte
     about/
         +page.(b2c,fr).svelte
         +page.(b2c,b2b).svelte
     +layout.(b2c).svelte
     +layout.(b2b).svelte
     +layout.(int).svelte

The 'stuff' page  will be present on all permutations.
The 'news' page will be present on all permutations, but the 'int' permutations will use a specific template.
The 'contact' page will only be present on 'b2c' and 'b2b' permutations, with a specific template for each target.
The 'about' page only be present on 'b2c' and 'b2b' permutations, with a specific template for the 'b2c,fr' permutation.
Each target ('b2c', 'b2b' and 'int') will use a specific layout template.

Permutations directories
Another solution will be to use a permutation directory for src and static.
For example :
   src/               # Will contains code shared by all permutations.
   src.(b2c)/         # Will contains code for 'b2c' permutation.
   static/            # Will contains static file for all permutations.
   static.(b2c)/      # Will contains static file for 'b2c' permutation.

Perhaps we can event use both, so :
import Component from '$lib/Component.svelte';

For the b2c,en permutation, it will return the first file found in this order :

src.(b2c)/lib/Component.(en).svelte or src/lib/Component.(b2c,en).svelte
src.(b2c)/lib/Component.svelte or src/lib/Component.(b2c).svelte
src.(en)/lib/Component.svelte or src/lib/Component.(en).svelte
src/lib/Component.svelte

By generating an error in case of duplicate file at the same level.
Server side processing
The handle() hook in hooks.server.js|ts will receive a new property 'permutation'.
This property will be a object representing the current permutation configuration.
For example, for the b2c,en permutation, this will match an object of the following form :
{
    target: 'b2c',
    lang: 'en'
}

Build mode
The build mode will define the way that Sveltekit will build the apps.
I see two main build mode.
build: 'distinct'
/// svelte.config.js
permutations: {
	keys: {
		target: ['b2c', 'b2b', 'int'],
		lang: ['en', 'fr', 'de', 'it' ]
	},
	build: 'distinct'
},

In 'distinct' build mode, Sveltekit wil produce a distinct build for each permutation.
This is the default behavior described since the beginning of this document.
It should be compatible with all current adapters.
We just need to specify the build we want to dev/build on the npm run commands.
build: 'bundle'
/// svelte.config.js
permutations: {
	keys: {
		target: ['b2c', 'b2b', 'int'],
		lang: ['en', 'fr', 'de', 'it' ]
	},
	build: 'bundle'
},

In ‘bundle’ build mode, Sveltekit wil produce a common build for all the permutations.
This build will include all the permutation code, and will generate an app.js for each permutation, with it's own dictionary.
The server will serve the correct client code based on the request.
Obviously this mode will be incompatible with some adapters without server-side code (like adapter-static)
In order to do that, it will require a mapping property that define all the mapping between the URL and the permutations.
For example :
/// svelte.config.js
permutations: {
	keys: {
		target: ['b2c', 'b2b', 'int'],
		lang: ['en', 'fr', 'de', 'it' ]
	},
	build: 'bundle',
	mapping: {
		'www.my-site.com': 'b2c,en',
		'www.my-site.fr': 'b2c,fr',
		'www.my-site.de': 'b2c,de',
		'www.my-site.it': 'b2c,it',
		'www.my-site.pro/en': 'b2b,en',
		'www.my-site.pro/fr': 'b2b,fr',
		'www.my-site.pro/de': 'b2b,de',
		'www.my-site.pro/it': 'b2b,it',
		'admin.my-site.com/en': 'int,en',
		'admin.my-site.pro/fr': 'int,fr',
		'admin.my-site.pro/de': 'int,de',
		'admin.my-site.pro/it': 'int,it',
	}
},

Here www.my-site.com will serve the 'b2c,en' permutation, www.my-site.fr will serve 'b2c,fr', and so on...
We can also imagine a mapping_dev allowing to use different domain for dev mode.
An alternative/complement would be to use a server hook for that, allowing us to handle more special case :
/// hooks.server.ts
export  function  get_mapping(url: URL) {
	switch (url.hostname) {
		case  'www.my-site.com': return  'b2c,en';
		case  'www.my-site.fr': return  'b2c,fr';
		case  'www.my-site.de': return  'b2c,de';
		case  'www.my-site.it': return  'b2c,it';
		...
	}
	return  null;
}

In case of null/incorrect permutation name, the server will return an error.
And in this case we can use the handle() to redirect to the best domain...
build: 'group'
/// svelte.config.js
permutations: {
	keys: {
		target: ['b2c', 'b2b', 'int'],
		lang: ['en', 'fr', 'de', 'it' ]
	},
	build: 'group',
	group_keys: ['target']
},

The ‘group’ build mode is an intermediate version.
Sveltekit will generate distinct build based on the specified group_keys, which will bundle the other permutation keys.
In this example, using group_keys: ['target'] will generation 3 main permutations :

b2c that bundle b2c,en, b2c,fr, b2c.de and b2c.it
b2b that bundle b2b,en, b2b,fr, b2b.de and b2b.it
int that bundle int,en, int,fr, int.de and int.it

Alternatives considered
No response
Importance
nice to have
Additional Information
No response

Some of this could also be accomplished with

#5703


Support using the folder name in search

Hello,
In a multi-folder workspace, I have a few folders, and I only want to search in one of them: common. But including the folder name in the query doesn't work:
Query: common/src/*/tsconfig.json
Finds no results because the folder name common doesn't match.

+1, could definitely use this. I need to search for paths like this, for example:
<customerName>\**\*appsettings.QA*.json


[KeyVault-Secrets] The service started sending different error messages

The KeyVault service started sending different error messages, which makes our live tests fail. I haven't had time to dig more into this issue, but this PR fixes the problem for master.

I still need to push the browser recordings.

Left comments on #8830. I think we should search for the substring to be maximally flexible here.