Phi 1.5 evaluation problem

System info
Google colab free version

The command I used:
!accelerate launch  main.py \
  --model microsoft/phi-1_5 \
  --tasks humanevalsynthesize-rust \
  --limit 10 \
  --temperature 0 \
  --do_sample True \
  --n_samples 100 \
  --batch_size 1 \
  --allow_code_execution \
  --save_generations \
  --trust_remote_code

I got this error:
The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_processes` was set to a value of `1`
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2023-10-18 18:05:56.298975: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Selected Tasks: ['humanevalsynthesize-rust']
Loading model in fp32
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
Downloading pytorch_model.bin: 100% 2.84G/2.84G [00:17<00:00, 163MB/s]
/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:374: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
Downloading (…)neration_config.json: 100% 69.0/69.0 [00:00<00:00, 404kB/s]
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:655: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
Downloading (…)okenizer_config.json: 100% 237/237 [00:00<00:00, 1.48MB/s]
Downloading (…)olve/main/vocab.json: 100% 798k/798k [00:00<00:00, 14.6MB/s]
Downloading (…)olve/main/merges.txt: 100% 456k/456k [00:00<00:00, 2.32MB/s]
Downloading (…)/main/tokenizer.json: 100% 2.11M/2.11M [00:00<00:00, 16.9MB/s]
Downloading (…)in/added_tokens.json: 100% 1.08k/1.08k [00:00<00:00, 5.61MB/s]
Downloading (…)cial_tokens_map.json: 100% 99.0/99.0 [00:00<00:00, 492kB/s]
Downloading builder script: 100% 6.22k/6.22k [00:00<00:00, 25.5MB/s]
Downloading readme: 100% 7.59k/7.59k [00:00<00:00, 26.3MB/s]
Downloading data: 100% 497k/497k [00:00<00:00, 9.41MB/s]
Generating test split: 164 examples [00:00, 2090.55 examples/s]
number of problems for this task is 10
  0% 0/1000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/content/bigcode-evaluation-harness/main.py", line 356, in <module>
    main()
  File "/content/bigcode-evaluation-harness/main.py", line 342, in main
    results[task] = evaluator.evaluate(task)
  File "/content/bigcode-evaluation-harness/lm_eval/evaluator.py", line 76, in evaluate
    generations, references = self.generate_text(task_name)
  File "/content/bigcode-evaluation-harness/lm_eval/evaluator.py", line 55, in generate_text
    generations = parallel_generations(
  File "/content/bigcode-evaluation-harness/lm_eval/generation.py", line 127, in parallel_generations
    generations = complete_code(
  File "/content/bigcode-evaluation-harness/lm_eval/utils.py", line 243, in complete_code
    for step, batch in tqdm(
  File "/usr/local/lib/python3.10/dist-packages/tqdm/std.py", line 1182, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.10/dist-packages/accelerate/data_loader.py", line 560, in __iter__
    next_batch, next_batch_info = self._fetch_batches(main_iterator)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/data_loader.py", line 523, in _fetch_batches
    batches.append(next(iterator))
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 677, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/content/bigcode-evaluation-harness/lm_eval/utils.py", line 53, in __iter__
    prompt_contents = self.task.get_prompt(self.dataset[sample])
  File "/content/bigcode-evaluation-harness/lm_eval/tasks/humanevalpack.py", line 657, in get_prompt
    return super().get_prompt(prompt_base, instruction)
  File "/content/bigcode-evaluation-harness/lm_eval/tasks/humanevalpack.py", line 229, in get_prompt
    raise NotImplementedError
NotImplementedError
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 986, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 628, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'main.py', '--model', 'microsoft/phi-1_5', '--tasks', 'humanevalsynthesize-rust', '--limit', '10', '--temperature', '0', '--do_sample', 'True', '--n_samples', '100', '--batch_size', '1', '--allow_code_execution', '--save_generations', '--trust_remote_code']' returned non-zero exit status 1.

It seems like a NotImplementedError, so is this evaluation not available for Phi models?

Hello, the harness supports all decoder transformers models. The error is because you didn't provide a --prompt argument in you command, I'll update the code to show a more informative message.
HumanEvaSynthesize requires providing a prompt (see docs) type which says how the instruction should be formatted, you can use --prompt instruct for example which concatenates the english instruction with function signature (see)
Also, are you trying to do greedy since you set temperature to 0? In that case I advice you do --do_sample False instead, and you won't need a 100 samples per problem since we don't sample in greedy and generations will be the same, you can try this for evaluation on the first 10 problems in greedy:
accelerate launch  main.py \
  --model microsoft/phi-1_5 \
  --tasks humanevalsynthesize-rust \
  --limit 10 \
  --do_sample False \
  --prompt instruct \
  --n_samples 1 \
  --batch_size 1 \
  --allow_code_execution \
  --save_generations \
  --trust_remote_code \
  --max_length_generation 2048 \

Hello, the harness supports all decoder transformers models. The error is because you didn't provide a --prompt argument in you command, I'll update the code to show a more informative message.
HumanEvaSynthesize requires providing a prompt (see docs) type which says how the instruction should be formatted, you can use --prompt instruct for example which concatenates the english instruction with function signature (see)
Also, are you trying to do greedy since you set temperature to 0? In that case I advice you do --do_sample False instead, and you won't need a 100 samples per problem since we don't sample in greedy and generations will be the same, you can try this for evaluation on the first 10 problems in greedy:
accelerate launch  main.py \
  --model microsoft/phi-1_5 \
  --tasks humanevalsynthesize-rust \
  --limit 10 \
  --do_sample False \
  --prompt instruct \
  --n_samples 1 \
  --batch_size 1 \
  --allow_code_execution \
  --save_generations \
  --trust_remote_code \
  --max_length_generation 2048 \

Thanks and it worked. At the time I put the issue, I was new here. But yes, implementing those strategies worked. Thanks


Patching mechanism in CFE doesn't allow to patch members with members of a different kind

Currently both factories and redirecting factories are represented as Procedures in Kernel.  The latter has special bodies that encode the redirection information, that is, the target constructor and the type arguments.  There's also a specialized subclass of Member to represent redirecting factories, namely RedirectingFactoryConstructor.
Now, some platform-specific patches to core libraries patch non-redirecting factories with redirecting factories.  It does work currently, as both are represented as Procedures, but it doesn't work when we try to change the representation of redirecting factories to the specialized one.
The patching mechanism needs to be update to work in that case.

Punting to 2.1 per comment in https://github.com/dart-lang/sdk/issues/29989#issuecomment-400035123 that this is not critical for stable

This is unrelated to the Dart 2.1 release.

Note, this is blocking another 2.3 issue.

Specifically, this blocks #33041

I think we can punt this to 2.4 we realize the front end team is busy and we have lived with #33041 for almost a year now.


Upgrade to centralised NodeJS codegen

Looks like we lost a couple of files here. Is that intentional? E.g. https://github.com/pulumi/pulumi-newrelic/pull/73/files#diff-a3f5e7f64125afb39dcb761e05f5ff4b

Looks like we lost a couple of files here. Is that intentional? E.g. #73 (files)

Yes, that will be expected on a provider or 2 - there were generation changes as we went through and we tried to cleanup these files
As long as there are no removals from sdk/nodejs/index.ts, then it's save to assume these were incorrect files anyway

As long as there are no removals from sdk/nodejs/index.ts, then it's save to assume these were incorrect files anyway

That's a fair point.


Add Acceptable Use Policy to Website

Review first draft, get sign-off, then add page.

I think the proper place for this is the FAQ. So all that needs to be done for this is to migrate this into markdown and add it.
Assigning to @Feanor78 (please review the doc, as well).

Can I suggest linking it in the footer? There are no legal disclaimers or copyright info on the bottom of the site right now. Need privacy policy, too, but that should be its own issue.

Let's treat that separately https://github.com/lbryio/lbry.io/issues/369

@jsigwart can we close this issue?

Yes, please close.


increase role session duration

Description of changes:
Test parallelism in prow CI is low and it runs only 2-3 tests. sagemaker-controller tests do not finish in one hour as a result. see - https://github.com/aws-controllers-k8s/sagemaker-controller/pull/34. the tests won't be able to clean up the resources as well since the credentials expire which is a big issue
Can we increase the number of threads in addition to this change? CI experience is bad if the developer needs to wait for 2 hours to know their test results
Another solution is to make the session duration and num threads configurable parameter per controller repo if we configure a file that has these env variables in each controller repo
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Currently pytest detects number of usable cores (virtualised or otherwise) and runs one threaded test per core. I think that this should be the default behaviour, but from inside a container (and that container inside a DIND Prow job) it doesn't make much sense. We'll have to play around with CPU and memory limits on the pods to be sure we don't cap out when multithreading, but I can look into it.

@surajkota some questions for you:

How long does the slowest SageMaker e2e test take?
What kind of parallelism are you looking for? pytest automatically spawns a number of workers related to the Pod's number of CPUs, so we could increase the size of the ProwJob CPU requests for some of the service controller e2e jobs.
When you run the SageMaker e2e tests "locally" (against a pre-setup EKS cluster with your controller installed already), how long do the e2e tests take to run?
We could look into a fail-fast option so that a test failure can prematurely kill other running tests.

How long does the slowest SageMaker e2e test take?

about 22-25 mins

What kind of parallelism are you looking for? pytest automatically spawns a number of workers related to the Pod's number of CPUs, so we could increase the size of the ProwJob CPU requests for some of the service controller e2e jobs.

Yes, I am looking for more cpus so pytest xdist can spawn more threads. I used the incorrect term in the issue, I meant to say Can we increase the number of cpus/vcpus

When you run the SageMaker e2e tests "locally" (against a pre-setup EKS cluster with your controller installed already), how long do the e2e tests take to run?

Codebuild size - 15 GB memory, 8 vCPUs
Pytest num threads - 10,
~30 minutes only for the pytest command excluding bootstrap, cleanup, building container
~40 minutes end to end

We could look into a fail-fast option so that a test failure can prematurely kill other running tests.

Even if we add this configuration, resources will leak from tests and service_cleanup script if credentials expire


Slider responses using keyboard keys

Hi,
So I'm running an experiment that asks participants to press one of two keys to indicate which of two stimuli is the target stim, and then to give a confidence response on a slider, which in JsPsych requires the user to click. I think responses would be much quicker and user friendly if they could just give responses for a slider on a keyboard.
I've got a working example of how this can look using vanilla javascript here: https://jsfiddle.net/vjs4ykb8/
I'd like to add this feature to an experiment I'm currently running, would this be doable by editing a JsPsych plugin myself, do you think?
Thanks!

Hi @Max-Lovell,
That's a great idea. I'd be open to adding this as a parameter for all slider plugins. I think it should be quite doable by editing the existing plugins.

thanks @jodeleeuw, implementation that can sit anywhere in the main body of the html-slider-response plugin is below, with enter key triggering submit too:
    document.addEventListener('keydown', key_listener)
    function key_listener(e){
      if(["1","2","3","4","5","6"].includes(e.key)){
        display_element.querySelector('#jspsych-html-slider-response-response').value = e.key
        display_element.querySelector('#jspsych-html-slider-response-next').disabled = false;
      } else if(e.key==="Enter" && display_element.querySelector('#jspsych-html-slider-response-next').disabled === false){
        display_element.querySelector('#jspsych-html-slider-response-next').click()
      }
    }

and the event listener is then removed in the next button click event handler.


Debug platform and build plan

I can render the platform via holos render platform ./platform, but this is all or nothing.
When something goes wrong, I would like to inspect intermediate stages to be able to pinpoint misconfigurations.
In the tutorial I found cue export --out yaml ./platform to render the platform with all components, and cue export --expression holos --out=yaml ./components/podinfo to render an individual component, but those feel clunky to use.
It would be nice if holos could provide a command to print both the platform definition and a build plan for all components to stdout without actually rendering them.

Agreed, something like holos show platform ./platform and holos show component ./components/podinfo are commands I'd like to add sooner rather than later.
FYI, right now, if you run in full debug mode you can get the tags injected into the component which gives you the complete context necessary to export the build plan with CUE.
For example:
holos render platform ./platform --log-level=debug --log-format=json 2> /tmp/log.json

jq -r 'select(.args != null) | .args | join(" ")' /tmp/log.json

This will give you:
render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo components/podinfo

Clunky for sure, you can copy and paste this remembering to add ./ to the component path:
holos cue export --out=yaml --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo

Is it a requirement to get all BuildPlan resources in one YAML stream?  What's the use case for this, so you can slice and dice it with yq?

@msteinhoff FYI I added holos show buildplans and holos show platform commands in #349 released in v0.100.0.
❯ holos show buildplans --help
Show BuildPlans produced from Platform.spec.components

1. Selectors are applied to the Platform.spec.components list.
2. Results are output in the same order as listed in the Platform spec.

Usage:
  holos show buildplans [flags]

Aliases:
  buildplans, buildplan, components, component

Flags:
      --platform string     platform directory path (default "./platform")
      --format string       yaml or json format (default "yaml")
  -l, --selector selector   label selector (e.g. label==string,label!=string)
  -t, --inject tags         set the value of a cue @tag field from a key=value pair
      --concurrency int     number of concurrent build steps (default 8)
  -v, --version             version for buildplans

Global Flags:
      --log-drop strings    log attributes to drop (example "user-agent,version")
      --log-format string   log format (text|json|console) (default "console")
      --log-level string    log level (debug|info|warn|error) (default "info")


Fix cognitive mutagens

From TikaelSol: The lore skill check should be predicated on action:recall-knowledge. There is at least one official lore skill that is not used to recall knowledge (Piloting Lore, https://2e.aonprd.com/Feats.aspx?ID=3140). All of the cognitive mutagens will need to get fixed.

Looks like the rule elements are correct. Just the effect descriptions need updating.


Fetching the id from the model being saved.

I'm curious if there is a way to fetch the id from the model being saved without querying the database.

after the save you can obtain the id from the model in the then function of the promise


Remove duplicate code and check the remainder only once

Second part of a followup from https://github.com/huggingface/accelerate/pull/1709, given a dataloader never really should change its remainder after iterating, the remainder isn't re-set after the end of an __iter__, nor is it recalculated during an __iter__. Instead we check for the default (-1), then recalculate.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Can't Unhide Boss Bar

All the other ones seem to work except for the boss bar when unhiding from the tray icon.

Fixed in v0.1.2


I want to attach a realsense sensor to the world. Can someone please help? Thanks in advance.

I have tried using the plugin provided by pal-robotics but it didn't worked for me. Also i have used the official plugin from intel realsense but it doesn't shows the published topic in ros2 topic list.

Hi @8LTB8smile8angel8,
You are getting that error message because you haven't installed the IFRA_LinkAttacher repository, which is needed for the object pick and place in Gazebo.
Instructions to install it (included in the README.md file):
cd ~/dev_ws/src
git clone https://github.com/IFRA-Cranfield/IFRA_LinkAttacher.git
cd ~/dev_ws
colcon build

Regards,
Mikel (IFRA-Cranfield)


2: text to css

This switches text to use css to increase perf

@keybase/react-hackers bump

cc: @cecileboucheron @adamjspooner to sanity check

Shouldn't we try to use our globalColors constants instead of hard-coded rgb values?

this is a real css file so i can't pull in those vars. we can make the pipeline more complicated later i guess but it didn't seem worth it right now

Ok. We'll have to remember to change this file if we change our colors.

sure, also the text colors change very very rarely

There are some things that use text's getStyle (mostly input components afaict), so they're only getting partial styles on this branch

@buoyad i added back getStyle but its only called by external components, not text. we should reconcile that better soon

What about generating these at load like icons? Not super important but might avoid some of the issues mentioned

Yeah lets ticjet  this


validate that ids are not negative

closes https://github.com/graphhopper/graphhopper/issues/1222
with the discussions here and here it might be a good idea to restrict ids to be >0, which would also conform to the OSM structure.
As we now specify explicitly that all ids are >0, the toWayID >= 0 && viaNodeID >= 0 assumption in OSMReader#createTurnRelation is perfectly fine.
A try to create a graph with negative OSM ids would lead to a massage as
Caused by: com.graphhopper.reader.InvalidIdException: Invalid Id: -138082; Ids must be bigger or equal to 1

See https://github.com/graphhopper/graphhopper/pull/2648/files#r951647819
negative OSM IDs are used to add custom nodes/ways/relations to OSM data: https://wiki.openstreetmap.org/wiki/Import/Software#Negative_IDs
We should probably follow osm2pgsqls approach and switch to multiple input files instead: https://github.com/openstreetmap/osm2pgsql/issues/1097

TBH I'm not that familiar with GH to be the man with the experience to tell if the Turn Restriction case is the only case which might be affected from negative IDs. If it is, it should be enough to refactor createTurnRelation()
However, till we know there are no side effects in working with negative IDs, we should not allow them.
If you use e.g. JOSM to make your custom nodes/ways/relations it's very easy to make the IDs positive using an editor before creating a graphhopper graph. However wasting multiple hours as I did or he https://github.com/graphhopper/graphhopper/issues/1222

After spending many hours I eventually found the cause: negative IDs.

did should not be the case.

We should probably follow osm2pgsqls approach and switch to multiple input files instead: openstreetmap/osm2pgsql#1097

If I understand correctly, they deprecated the use of negative IDs which is what I suggest in this PR?

So far we have three options:
Allow negative IDs
aka the status quo.
Pro: no change from the user perspective
Contra: their might still be bugs somewhere which need  to be patched; dealing with negative IDs requires additional logic and is potentially error prone
Prohibit negative IDs
Importing a file with negative IDs would fail and the user has to patch it with an external program.
Pro: easy
Contra: outcry from users which rely on the feature
Prohibit negative IDs  and allow to import multiple OSM files instead
Pro: flexible
Contra: refactoring the import is still a big TODO and would be mandatory for this IMHO

If I understand correctly, they deprecated the use of negative IDs which is what I suggest in this PR?

osm2pgsql went with option 3. You can e.g import the default planet.osm.pbf and craft an additional pbf with your own data.

In option 3, are there negative IDs allowed in any of the OSM files?

https://github.com/openstreetmap/osm2pgsql/pull/1339
Explains how osm2pgsql handles it.
tl;dr all files must be sorted and have only positive IDs

So basically for this PR it's only option 1 or option 2, is it?

allow to import multiple OSM files instead

IMO this is a feature not related to this issue

IMO this is a feature not related to this issue

Negative IDs are currently (abused) to import non-OSM data as well. If we remove this feature, we should be able to provide users with an alternative. If external tools are good enough to do this, I think that is fine.

I think it's REALY EASY for users to assure that the data contains only valid ids.
I created an external tool for standard osm files.
However it's REALY hard for developers of such a large system to test the edge cases of negative ids.

If you use e.g. JOSM to make your custom nodes/ways/relations it's very easy to make the IDs positive using an editor before creating a graphhopper graph.

@lukasalexanderweber
Do you mean your replace-invalid-osm-ids script, or can JOSM also do this? And what about larger (pbf?) files?

Negative IDs are currently (abused) to import non-OSM data as well. If we remove this feature, we should be able to provide users with an alternative.

@otbutz
Is this really a feature people use? And why can they not simply use positive IDs for their non-OSM data? So far the only use-case I can think of is creating a graph using some UI tool like JOSM, which (unfortunately in this case) uses negative IDs.

Do you mean your replace-invalid-osm-ids script, or can JOSM also do this?

in my case I just replaced "-" with "" within an editor. But I noticed that this is not robust. replace-invalid-osm-ids, however, handles this correctly.

And what about larger (pbf?) files?

osmconvert invalid.pbf -o=invalid.osm
python replace_invalid_osm_ids.py invalid.osm valid.osm
osmconvert valid.osm -o=valid.pbf

osmconvert invalid.pbf -o=invalid.osm
python replace_invalid_osm_ids.py invalid.osm valid.osm
osmconvert valid.osm -o=valid.pbf

Ok yes this works for small files, but for larger files it is very inefficient, because it uses a DOM parser to read the xml. Anyway, of course it is not a terribly hard task to replace the negative IDs, I was just wondering if there are already tools that do this.

Is this really a feature people use?

¯\_(ツ)_/¯
I'm just pointing out that someone might get mad at us for breaking their workflow.

but for larger files it is very inefficient, because it uses a DOM parser to read the xml

Yeah I know. With the script I rather target users of JOSM which work on a local dataset.
I just tested it with a real life junction and it worked well.
Processing a 250MB subset of a medium sized city (Karlsruhe) already needs 90s.
@otbutz wouldn't it be fun to find out what abnormalities people do with negative ids? I love the comic xD

It seems like so far we only suspect negative IDs to be useful for something, so I think we should reject them until we have an actual use case for negative IDs that cannot be fixed by some easy preprocessing.

there is a realy big bunch of test data which uses 0 as ID. We could maybe limit this PR only to negative Ids?

there is a realy big bunch of test data which uses 0 as ID. We could maybe limit this PR only to negative Ids?

Yes, I thought the same.

lukasalexanderweber requested review from easbar and removed request for otbutz 19 seconds ago

sorry for that!

Thanks!

Thanks for the constructive Feedback and for maintaining Graphhopper!


Add a way to get replication links for a SQL Azure database

It would be handy to be able to retrieve the replication role of a SQL Azure database with the client.
Looks like the replication links are available through the SDK yet.

@couellet You can find the replication link operations here: https://github.com/Azure/azure-sdk-for-net/blob/master/src/ResourceManagement/Sql/SqlManagement/Generated/ReplicationLinkOperationsExtensions.cs


Scale cursors on scaled outputs

[x] Keep one xcursor theme per seat
[x] Integrate roots_xcursor_theme into wlroots

Test plan: check that compositor cursors (default, move, resize, rotate) still work with different output scales.
Fixes #401

I want cursor theme per seat. Is that possible?

I want cursor theme per seat. Is that possible?

That would make things more complicated because roots_xcursor_theme has to listen for outputs to load scaled xcursor themes. But I guess I can make the desktop loop through all xcursor themes and load new scaled cursors when a new output is plugged in.
I prefer to listen for new outputs instead of reading output scales from the config because:

It allows to lazy load scaled xcursor themes (if an output is setup with a scale of 2 and is never plugged in, xcursor theme with scale of 2 is never loaded)
It will allow to autodetect the scale of an output, if not specified in the config file

I added back roots_seat.xcursor_theme and the seat now uses this theme for its cursor. However, there's still a global theme (used for xwayland), and currently the global theme is used in all seats. What it means is that for now it's still not possible to configure per-seat themes, but it's just a matter of setting roots_seat.xcursor_theme when initializing the seat.

This PR is ready for another review.
The last commit introduces wlr_xcursor_manager, which manages multiple XCursor themes loaded with different scale factors. Note that wlr_xcursor and wlr_cursor can be used without wlr_xcursor_manager, and wlr_xcursor_manager can be used without wlr_cursor.

I like it. Will merge after testing.

https://travis-ci.org/swaywm/wlroots/jobs/300891992

I don't get any cursor at all on my hidpi output.

I don't get any cursor at all on my hidpi output.

Fixed!

Thanks!


[WIP] transform primary heating

Just a WIP about extracting the heating systems. There's a lot of unknowns in this data, we'll probably need to experiment with the Hot2000 applications to see all of the potential values in the XML.

FYI @buckley-w-david You might be interested in this approach to the multiple languages.


Documentation of modules with sphinx

Create documentation of modules with sphinx
https://github.com/sphinx-doc/sphinx

Update - would like to remove below exceptions (and passes all tests) from flake8 config before marking this as complete.
D102    # Missing docstring in public method
DAR401  # Missing exception(s) in Raises section: -r Exception


Optimizations

removed anons
normalized to simple fors

I think we are good to go. Do you have any means for testing that all of this improves performance?

I have a python script, which is multithreaded and kinda hacky . Do you want me to mail it to you ?

please do. Do this PR help with that?

Hold on. Let me verify.

I kinda prepared a small js script which mimics the way he was load testing.
https://gist.github.com/GavinDmello/4636114be7f11f36a9778c186470108a
On my machine, using this script
10k connections - 17 secs approx
20k connections - 35 secs approx
30k connections -  50 - 60 secs approx

We should use something like fastq to connect just X concurrently

@GavinDmello what are the numbers before/after  the change?

@mcollina I'm update the gist with fastq, I couldn't connect more that 5k clients with this approach. Could you check and tell me what I'm doing wrong here

You need to call the cb there: https://gist.github.com/GavinDmello/4636114be7f11f36a9778c186470108a#file-bench-js-L19

Still can't connect more than 4083 clients. The server gets only 4083 connect packets

You are probably exhausting the file descriptors. The best thing would be to set up a server on AWS (m3.medium) and do https://gist.github.com/mcollina/23c788bf2f7e8da10b9c

Merging, as it should improve things anyway.


Remove pre-built container section from getting_started.md

Description

The quickstart guide recommends users to use the pre-built containers, however these containers don't have the ability to launch Triton as documented.
Have tried the solution that adding ./external/utilities/docker/install_docker.sh to Dockerfile to enable spinning up another docker container that runs triton server inside the release container, but it cannot mount /models directory due to the nested docker mount issue, making the solution to be invalid.
For 24.06 release, just remove the pre-built container section from getting_started.md and let users to build the container from source. For further improvement, plan to publish a separate models container in the next release.
Closes #1507
By Submitting this PR I confirm:

I am familiar with the Contributing Guidelines.
When the PR is ready for review, new or existing tests cover these changes.
When the PR is ready for review, the documentation is up to date with these changes.

/merge


Possible incompatibility with Playwright?

I'm trying to use this validator with the Playwright testing framework. I've just done clean installs (version information below) and tried to run a basic test case. Here is the code for the test case:
import { w3cHtmlValidator } from 'w3c-html-validator';
import { test } from '@playwright/test';

const URL = "/path/to/local/index.html";

test("Validation check", async () => {
  const options = { filename: URL };
  w3cHtmlValidator.validate(options).then(console.log);
});

Whenever I run Playwright, I get the following error:
$ npx playwright test
Error: No "exports" main defined in /Users/.../node_modules/w3c-html-validator/package.json

   at example.spec.ts:1

> 1 | import { w3cHtmlValidator } from 'w3c-html-validator';
    | ^
  2 | import { test } from '@playwright/test';
  3 |
  4 | const URL = "/path/to/local/index.html";

    at Object.<anonymous> (/Users/.../tests/example.spec.ts:1:1)

Error: No tests found

Here's the relevant version information:
$ node --version
v20.3.0
$ npm --version
9.6.7
$ npm info playwright
<EMAIL_ADDRESS>| Apache-2.0 | deps: 1 | versions: 3902
...
$ npm info w3c-html-validator
<EMAIL_ADDRESS>| MIT | deps: 5 | versions: 32

Is this validator just incompatible with Playwright, or is there a fix for this?

I've never used Playwrite, but I assume it supports ES Modules.
w3c-html-validator is an ESM package.  node has a collection of rules based on package.json settings and file extensions to determine when to use ESM mode.  Something in your environment looks to be causing node to use the old CommonJS mode instead of ESM mode.
Check your package.json file to see if you have ESM enabled:
"type": "module",

More information is at:
https://nodejs.org/api/packages.html#determining-module-system
To run a little test yourself you can do the following:
mkdir my-proj
cd my-proj
npm init --yes
npm install w3c-html-validator

Edit the package.json file to add: "type": "module",
Create a file named index.js:
import { w3cHtmlValidator } from 'w3c-html-validator';

console.log(w3cHtmlValidator.version);
w3cHtmlValidator
   .validate({ website: 'https://pretty-print-json.js.org/' })
   .then(w3cHtmlValidator.reporter);

Then execute the command: node index.js

That was it. Thanks!


Added support for saving/loading perks

Added support for saving/loading and moving perks.
Perks are loaded from a yml file.
Also adds intial support for loading skills and spells

Nice work!

Excellent implementation!


[@sanity/client] running a script with Bun gets the process hanged.

If you find a security vulnerability, do NOT open an issue. Email<EMAIL_ADDRESS>instead.
Describe the bug
Running a simple script using the @sanity/client with bun gets the process hanging forever. This doesn't work with tsx.
To Reproduce
Steps to reproduce the behavior:

Access this repository https://github.com/devraul/bun-sanity-process-hang
Clone it
Follow the README instructions
See error

Expected behavior
Works with bun without problems
Screenshots
https://github.com/sanity-io/sanity/assets/12464600/c560e750-e531-4404-af2f-f60d85abe596
Which versions of Sanity are you using?
Run sanity versions in the terminal and copy-paste the result here.
What operating system are you using?
"@sanity/client": "6.15.11"

Which versions of Node.js / npm are you running?
❯ bun --revision
1.1.3+2615dc742

Additional context
Since this is an integration problem, I opened the same problem in Bun. Maybe they need to fix something in their side:
https://github.com/oven-sh/bun/issues/10138

Hi, we tried adding bun conditions in https://github.com/sanity-io/client/releases/tag/v6.15.12 that redirected bun to use the node version of get-it, which uses node:http and node:https under the hood as transport instead of global.fetch.
It created different problems:

https://github.com/sanity-io/get-it/blob/1c17809502af0fffdeaefd1b19d87d69c98c34e6/src/request/node-request.ts#L198-L199
https://github.com/sanity-io/client/actions/runs/8647489271/job/23709097678

Surprisingly, changing the bun import conditions to point to the transport that uses fetch appears to solve it:

https://github.com/sanity-io/client/commit/ac37787241bd15de1ec4a0a8d49593a5d1db6a20
https://github.com/sanity-io/client/actions/runs/8648445067/job/23712318406

It's surprising because the assumption has been that bun has used the worker condition until now and always used fetch under the hood.
Here's the basic e2e test we're using:
https://github.com/sanity-io/client/blob/main/runtimes/bun/client.fetch.test.js
The CI tests on Bun 1.1.3, and I've also run the same test locally with Bun 1.1.3 and it works there too.
Give https://github.com/sanity-io/client/releases/tag/v6.15.13 and tell us if that works for you 🙌

Checking the test rig, it started happening last week:
https://github.com/sanity-io/client-runtimes-compatibility/actions/runs/8516205488/job/23324913908

<EMAIL_ADDRESS><EMAIL_ADDRESS>
Last passing test: https://github.com/sanity-io/client-runtimes-compatibility/actions/runs/8502608803/job/23287010458

<EMAIL_ADDRESS>1.0.36

Based on this, and the error on the CLI that started happening with Bun 1.1.0 seems to be it started using the node version of get-it and @sanity/client indeed.
Here's how you can check what transport is in use:
import {unstable__adapter, unstable__environment} from '@sanity/client'
import {adapter, environment} from 'get-it'

console.log({unstable__adapter, unstable__environment, adapter, environment})

In bun prior to 1.1.0 adapter reports fetch, afterwards it started using node 🤔
In any case that shouldn't be a problem anymore as we now tell Bun exactly what version to use ☺

Closing as it should be fixed as of<EMAIL_ADDRESS>

Really nice. Thank you for fixing this 🙌


Simply traits

resolve #23 #25
Remove self-consuming style traits.

This is large breaking change. We have to bump up version to v0.6.0

Time associated type is redundant, but we cannot infer the type of Time in tests/view.rs.


15.0.4 doesn't work on ARM64 Macs.

I am getting the following error: dyld[3244]: Library not loaded: '/usr/local/opt/zstd/lib/libzstd.1.dylib'
/usr/local/ is where Homebrew installs stuff on x64, on ARM64, it is /opt/homebrew/: https://github.com/Homebrew/install/blob/684032061383ab14862240aaff4273cde4e1847c/install.sh#L129-L138
But to my understanding, it shouldn't try to load globally installed libraries.

Thanks for reporting @Cyberbeni. I have identified an issue with linking ZLib (previous versions did not link against Zlib) and fixed it by disabling ZLib support (I doubt it is required for clang-format). Could you try to install clang-format from these wheels and run the executable? I have no access to an ARM64 Mac to try myself.

Still crashes for me.
> python3 -m pip install --user ~/Downloads/wheels_without_zlib/clang_format-15.0.4-py2.py3-none-macosx_10_9_universal2.whl
Processing /Users/benedek.kozma/Downloads/wheels_without_zlib/clang_format-15.0.4-py2.py3-none-macosx_10_9_universal2.whl
Installing collected packages: clang-format
Successfully installed clang-format-15.0.4
> python3 -c 'from clang_format import clang_format; clang_format();'
dyld[71144]: Library not loaded: '/usr/local/opt/zstd/lib/libzstd.1.dylib'
  Referenced from: '/Users/benedek.kozma/Library/Python/3.10/lib/python/site-packages/clang_format/data/bin/clang-format'
  Reason: tried: '/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/usr/local/lib/libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib' (no such file)

If I install clang-format through Homebrew, that works correctly, the brew formula has uses_from_macos "zlib" which I think means that the OS should already have zlib installed (and there is also no sign of it installing zlib when I install clang-format through brew). Don't know if it helps but here is how they build it: https://raw.githubusercontent.com/Homebrew/homebrew-core/e0433f7392fb27e1aa51df6df1a6e3dfae2a31b1/Formula/clang-format.rb

Yes, the 14.0.6 version works fine.

I don't really understand why the official releases don't have every platform/arch for every release. 15.0.3 is the last one that has both x64 and arm64 for macOS, the later ones (15.0.4-15.0.6) only have arm64: https://github.com/llvm/llvm-project/releases
But if it had both, it would probably be better to just download those instead of rebuilding them IMO. (The lipo CLI tool can merge the 2 binaries)

@Cyberbeni @psobot thanks for the feedback, if possible could you try this test version from the test pypi repo?:
pip install -i https://test.pypi.org/simple/ clang-format==<IP_ADDRESS>

Both the libzstd library issue and the missing arm64 architecture should be fixed, but I don't have access to an arm64 mac to test it.

@Cyberbeni thanks for testing it, there's now a 15.0.6 release on pypi with these fixes


Inquiry About Killer Applications Utilizing HPX

Hello, HPX team. I am curious to know if there are any applications or projects currently utilizing HPX that can be considered "killer applications". Specifically, I'm looking for examples where HPX has made a significant impact in terms of performance, scalability, or solving complex problems that were not feasible with other technologies.
Thank you for your time and assistance.

Hi @zuoyanzhang,
Here are some links that may be of interest:

https://hpx.stellar-group.org/applications/
https://github.com/STEllAR-GROUP/hpx/discussions/5368

Many Thanks to you, Panos.


ion-refresher slow after the first time

<ion-view title="Browse">
  <ion-nav-buttons side="left">
    <button menu-toggle="left"class="button button-icon icon ion-navicon"></button>
  </ion-nav-buttons>
  <ion-content class="has-header" ng-controller="ListCtrl">

Browse

	<ion-refresher
		pulling-text="Pull to refresh..."
		on-refresh="doRefresh()"
		refreshing-text="Refreshing!"
        refreshing-icon="ion-loading-c">
	</ion-refresher>
	<ion-list can-swipe="true">
		<ion-item ng-repeat="item in items">
			{{item.title}}
		</ion-item>
	</ion-list>
  </ion-content>
</ion-view>

.controller('ListCtrl', function($scope) {
  $scope.doRefresh = function() {
	$scope.$broadcast('scroll.refreshComplete');
  };

  $scope.items = [
    { title: 'Reggae', id: 1 },
    { title: 'Chill', id: 2 },
    { title: 'Dubstep', id: 3 },
    { title: 'Indie', id: 4 },
    { title: 'Rap', id: 5 },
    { title: 'Cowbell', id: 6 },
	{ title: 'Cowbell', id: 7 },
	{ title: 'Cowbell', id: 8 },
	{ title: 'Cowbell', id: 9 },
	{ title: 'Cowbell', id: 10 },
	{ title: 'Cowbell', id: 11 },
	{ title: 'Cowbell', id: 12 },
	{ title: 'Cowbell', id: 13 },
	{ title: 'Cowbell', id: 14 },
	{ title: 'Cowbell', id: 15 },
	{ title: 'Cowbell', id: 16 },
	{ title: 'Cowbell', id: 17 },
	{ title: 'Cowbell', id: 18 },
	{ title: 'Cowbell', id: 19 }
  ];
})

Only works fine(fast) the first time. I have tested it  with a Samsung Galaxy Nexus Android 4.3.

calling this fixed then. let me know if this is not the case

this was good in beta.14 but now with rc.0 the animation is laggy again on every pull-to-refresh on a ng-repeat list with moderate content on iOS.
I will check if I could make video like @danicomas did

Same issue here android 5.0 using ng-repeat. The problem does not appear when using collection repeat but I want to use native scrolling.
Any way to disable animation?

Same here, Android 5.1.1 with ng-repeat. As @abeninskibede I want to use native scrolling as well as ionic material, which is not compatible to collection-repeat.

OMG THE ANIMATION LAG LIKE MAD, I'm not even having big data inside and my phone aint that lousy. How :( Isit because NG-Repeat is lagging it?


Test failure: HubConnectionCanSendAndReceiveMessages

This test fails occasionally with the following error:
[Test Collection Cleanup Failure (RedisEndToEndTestsCollection)]: System.Exception : Command '/usr/bin/docker stop redisTestContainer' failed with exit code '1'. Output:
Error response from daemon: No such container: redisTestContainer
   at Microsoft.AspNetCore.SignalR.Redis.Tests.Docker.RunProcessAndThrowIfFailed(String fileName, String arguments, ILogger logger, TimeSpan timeout) in /_/test/Microsoft.AspNetCore.SignalR.Redis.Tests/Docker.cs:line 90
   at Microsoft.AspNetCore.SignalR.Redis.Tests.Docker.Stop(ILogger logger) in /_/test/Microsoft.AspNetCore.SignalR.Redis.Tests/Docker.cs:line 74
   at Microsoft.AspNetCore.SignalR.Redis.Tests.RedisServerFixture`1.Dispose() in /_/test/Microsoft.AspNetCore.SignalR.Redis.Tests/RedisServerFixture.cs:line 60

Other tests within that build may have failed with a similar message, but they are not listed here. Check the link above for more info.
This test failed on 2.1.
CC @muratg
This issue was made automatically. If there is a problem contact ryanbrandenburg.

Please use this workflow to address this flaky test issue, including checking applicable checkboxes and filling in the applicable "TODO" entries:

Is this actually a flaky test?

No, this is a regular test failure, fix the test/product (TODO: Link to commit/PR)
Yes, proceed below...

Is this test failure caused by product code flakiness? (Either this product, or another product this test depends on.)

[ ] File a bug against the product (TODO: Link to other bug)
Is it possible to change the test to avoid the flakiness?

Yes? Go to the "Change the test!" section.
No?

[ ] Disable the test (TODO: Link to PR/commit)
[ ] Wait for other bug to be resolved
[ ] Wait for us to get build that has the fix
[ ] Re-enable our test (TODO: Link to PR/commit)
[ ] Close this bug

Is it that the test itself is flaky? This includes external transient problems (e.g. remote server problems, file system race condition, etc.)

Is there is a way to change our test to avoid this flakiness?

Yes? Change the test!

[ ] Change the test to avoid the flakiness, for example by using a different test strategy, or by adding retries w/ timeouts (TODO: Link to PR/commit)
[ ] Run the test 100 times locally as a sanity check.
[ ] Close this bug

No?

Is there any logging or extra information that we could add to make this more diagnosable when it happens again?

Yes?

[ ] Add the logging (TODO: Link to PR/commit)

No?

[ ] Delete the test because flaky tests are not useful (TODO: Link to PR/commit)

This comment was made automatically. If there is a problem contact ryanbrandenburg.

CanSendAndReceiveUserMessagesWhenOneConnectionWithUserDisconnects failed with about the same error on 2.1.
This comment was made automatically. If there is a problem contact ryanbrandenburg.

@BrennanConroy can we make the stop a little more resiliant and check if the container is even running first.

This comment was made automatically. If there is a problem contact ryanbrandenburg.
There were 10 failures with about the same error on release/2.2 at 6:03:17 PM:

HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages
HubConnectionCanSendAndReceiveMessages

Closing as we made some changes to the redis tests, RAAS will open a new issue if there is still flakiness.


Embed.set_author creates an invalid embed payload if you set icon_url=None

Summary
Embed.set_author creates an invalid embed payload if you set icon_url=None
Reproduction Steps
Where author is a member with no avatar set, meaning author.avatar is None, the code
embed.set_author(
  name=author.name,
  icon_url=author.avatar
)

await send(embed=embed)

Will cause a 400 as embed payload being sent has an author with icon_url equal to the string 'None', as Embed.set_author simply do str(icon_url) if icon_url is not EmptyEmbed.
The issue here is that User.avatar can be null, and if passing in None as icon_url into Embed.set_author, the actual url will be set to the string 'None'.
One can solve this in userland by doing:
embed.set_author(
  name=author.name,
  icon_url=author.avatar or disnake.Embed.Empty
)

await send(embed=embed)

-- but surely that's not a recommended solution?
Minimal Reproducible Code
No response
Expected Results
not applicable
Actual Results
not applicable
Intents
not applicable
System Information
disnake 2.3.0
Checklist

[X] I have searched the open issues for duplicates.
[X] I have shown the entire traceback, if possible.
[X] I have removed my token from display, if visible.

Additional Context
No response

This is due to some questionable legacy features left over from discord.py.
In my opinion we should either get rid of empty where it isn't applicable or turn all None into empty by default.

After looking more at this, I now realise what I want to use is display_avatar, as that will automatically handle the case where avatar is None, and use the value of default_avatar.
I suppose this isn't really a bug, though the combination of User.avatar being optional and Embed.set_author freaking out if setting icon_url to None is unfortunate.
I'll let repo owners close the issue if they feel this isn't a real bug, or the discussion of whether Embed.Empty is reasonable belongs in another issue.

While this issue was indeed caused by the improper usage of User.avatar it still doesn't change the fact that embeds are very annoying to use due to how empty is handled. We'll be keeping this open until the user experience is improved.


set class library platforms

Without setting the platforms, the project is created default targeting to AnyCPU, and then you get a build error because WinUI doesn't support AnyCPU.

/azp run

this was late for 0.8-rc. For 0.8 GA, the WinUI will allow AnyCPU.


Include fixed findings stats in the request for GenerateDigestReport

This PR modifies the GenerateDigestReport endpoint to include information about FIXED findings

Tricky question: should we include also the expired findings?

If I remember correctly, the Vulnerability DB does not return EXPIRED findings if you query for FIXED findings.
(see: https://github.com/adevinta/vulnerability-db-api/blob/a37db7fe77b0851973776524cf16ef2b60f08659/pkg/storage/postgresql/findings.go#L75)
Still, I see your point. I don't have a strong opinion on this. If we really want to include the EXPIRED findings in the weekly digest, then we will need to modify the Vulnerability DB also.
How do you want to proceed?

I think we can go with this approach and discuss further during the metrics discussion.


Fix #360. Interpret scopes as space-separated

In my efforts to make reddit's OAuth2 more spec compliant, I broke PRAW.

Thanks for fixing.

Sure thing! Sorry for breaking stuff!

If it makes things better in the long-run I'm all for it.


Run selected code snippet

VSCode Version: 1.5.3
OS Version: macOS Sierra

Missing a button to RUN the code.
The extension Code Runner can do run the code but you need to press a set of keys or press F1 and type run. A simple button could be introduced to let us run the code edited like the Code Runner App available on Mac App Store.

You could set up a launch config or a task to do that.

Yes. I use CodeRunner and a key combination to run the code, but I miss the button that make it more visual and is the standard at any IDE to run any code.
Having the RUN button you know that the IDE can do that. When it is not present appear that we are working on a text editor only.
In fact, the best is have 3 buttons: one button to RUN and another to STOP and one to RUN SETTINGS. Because we can need to change the run settings and params passed for each run to test different parts of software, then this 3 buttons are missing.

There is a run button and 'launch settings' button in the Debug viewlet. You get to that by clicking the debug icon in the top left of the window. Once you have an application running in debug mode you'll also have a stop button. See: https://code.visualstudio.com/docs/introvideos/debugging

Found it, thanks.

This button doesn't run the code calling CodeRunner extension to run.
When I try do this I get the following message "Cannot find runtime 'node' on PATH."

I guess I misunderstood. You cannot launch CodeRunner from launch.json. You can however use it to launch an entire application. It complains about not finding node because "type": "node" and nodejs is not on your PATH. Depending on what language you are working with you need to choose a different "type" in your launch.config, it might be easiest to look up an example for your programming language and stack.

Is it possible integrate this debug tab with Code Runner extension as a solution to we have the desired buttons to control the process of run an application, in a future version of VSCode?

Using this to track the request for a "Run selected code snippet" action.

Debug tab don't work with Code Runner extension.
Is it possible make it work in a future VSCode version?

Thank you


feat: add support for notifications to feedback plugin

What do you want to improve?
Please add support for Backstage notifications to the feedback plugin.
What is the current behavior?
No idea if new feedback has been given unless navigating to the right page.
What will the new behavior be?
Entity owner(s) will be notified when new feedback is given for their entity. The integration should be quite straight-forward, see Backstage documentation here https://backstage.io/docs/notifications/#backend or example usage at my own plugin here: https://github.com/drodil/backstage-plugin-qeta/blob/main/plugins/qeta-backend/src/service/NotificationManager.ts
NOTE: it's possible to just pass the entityRef of the feedback target to notificationService - it will resolve the recipients internally.

"Thank you for the input, @drodil. We'll definitely consider this request for the plugin and look into it. Also, if you're interested, we welcome any contributions you might want to make to the plugin."


IE11 tooltip comming with no data on 2nd click

Hi ,
I have implemented tool tip on click of an image . But there is an issue coming in IE11 that when I am clicking on the image 2nd time the tool tip is coming with no data .
it is working fine in Firefox and chrome .
Any help is welcomed .

Hello, this sounds like an implementation issue, not an issue with Tooltipster, so unfortunately I'm not sure I can help you.

can u please tell me when the HTML element for the tooltip is being maid , I mean is it maid at every click event (I have implemented the tooltip on click event ) or we r just doing a display operation (showing and hiding ).
This will help me a lot in debugging .

Hi, everytime you show the tooltip.

Hi ,
found the issue ,was passing the content as object .

Ok good.


Display "Playing (Song Title) by (Artist)" instead of "Playing Apple Music"

Is this possible?

Not possible, "Apple Music" is the name of the application registered on the Discord Developer Portal and it cannot be changed dynamically.


In-memory secret store

I need the ability to create new Secrets in the v0.3 API for dagger-classic.
I suggest this change (or similar) to the API:
query {
  secret { // make the id: SecretID optional
    new(contents: String!) Secret!
  }
}

Another possibility:
query {
  secret(contents: String) Secret! // make the id: SecretID optional
}

@vito mentioned https://github.com/dagger/dagger/pull/3124

Discussed this briefly with @aluzzardi today to gather context and unblock @jlongtine.
Looks like there are two options:

Option 1: (long-term), bigger effort: some sort of secret extension API to plug things like Vault, etc... as I understand that the ability to add secrets from the API (like in 0.2) is not gonna happen (for valid reasons). Also this API needs to be designed, reviewed and implemented. We know what it takes.
Option 2: (short-term): have a private API (under engine, hidden from the Go SDK) to allow for creating secrets. This will be intended to disappear (risk to leak, and to stay forever).

I guess option 2 is more viable to unblock dagger-cue, depending on the effort for option 1. If we go option 2, we should find a way to derisk, for instance hiding the call behind a feature flag (eg. if this weird env var isn't set, the code path is disabled).
Would it be possible for @sipsma or @vito to own this one? Note that I am not hinting any option here, the priority is to remove the blocker. There might be an option 3 that I don't see.

@jlongtine how absurd and awful of an idea is this: you currently can load secrets from env vars, so whenever you need to load a secret in dagger-cue you export an env var and then use that to load the secret in? Would need to be careful if dagger-cue ever forks off child processes of course. But could be a path to avoiding any changes to the API if it's plausible.

Maybe we should revisit the “no direct creation of secrets” rule, in case the circumstances have changed?
The initial motivation for “no direct creation” was to:

avoid leaking plaintext secrets into the “id” (serialized recipe). Because IDs are typically understood to be safe to copy paste, share etc.
avoid sending plaintext secrets on the wire

One major change is that we plan on making the engine remote; and our understanding of IDs and their role has evolved. So maybe it’s OK to create secrets directly after all?
TLDR my first guess would be either option 2 (hidden api call) or “option 2+” (regular api call). Skeptical about option 1 because it means rushing a pretty big design discussion for a short term problem.

Bunch of edge cases we run into. The model of API-based creation broke down in a few places. For instance, when added in an extension it only works on un-cached runs (cached runs return a SecretID without the extension having run AddSecret, resulting in a broken secret)
To add to the above -- edge cases are caused by "AddSecret" being very special. It's the only call that alters the internal API state (whereas all other calls are idempotent, multi tenant, etc -- AddSecret is not)

I'd prefer we don't rush into neither Option 1 (needs design) nor Option 2+ (ripple effects, not easily reversible). @vito is going to post a summary of Option 3 (do nothing in core, work around with the existing API)

You can actually create in-memory secrets already in a kind of round-about way:
client.Directory().WithNewFile("sekrit", dagger.DirectoryWithNewFileOpts{
  Contents: "im always angry",
}).File("sekrit").Secret()

Caveat: this will result in a SecretID that embeds the secret value in plaintext (albeit obfuscated below multiple layers of base64 :sparkles: security :sparkles:). But maybe that's enough for now? As long a the SecretID itself isn't leaked this should still accomplish the goal of keeping its value out of the container layers.
This feels roughly the same level of risk as option 2 or introducing a dedicated "create secret" API, but the benefit is it already exists.
If you want to avoid leaking values into the SecretID you could instead leak them to the host filesystem and clean them up after:
os.WriteFile("/sekrits/bruce", []byte("im always angry"), 0600)
defer os.Remove("/sekrits/bruce")

client.Host().Directory("/sekrits").File("bruce").Secret()

// ... do API calls ...

For extra pizazz you could try using a fifo/pipe instead but we're skeptical that'll actually work. :P
Alternatively you could use env vars, but that's pretty sketchy too.
Between these two my preference would be the first one, since it's essentially what this issue proposes but feels like a workaround (which really it is).

If I understand correctly, Option 3 would be basically "all userland", no new API call either private or public? If so, that would be an ideal short term solution :)
Thanks for the additional details.

@jlongtine how absurd and awful of an idea is this: you currently can load secrets from env vars, so whenever you need to load a secret in dagger-cue you export an env var and then use that to load the secret in? Would need to be careful if dagger-cue ever forks off child processes of course. But could be a path to avoiding any changes to the API if it's plausible.

@sipsma I've heard worse ideas in my time. :)
I think I'm going to start with this... see if works out, then move onto one of @vito's suggestions if it doesn't for whatever reason.
I appreciate you all digging into this with/for me!

An update: Using environment variables works! I've got that all up and running, and it unblocked me on a bunch of stuff. Thanks for all the help, everyone!

I had a good discussion today with @sipsma and @vito about this.
Our proposal as a first pass (with subsequent designs for extensions, etc) is to add a GraphQL API to allow the creation of new Secrets. This will add those secrets to the secret.Store, and allow for retrieval by Buildkit.
Everything feels pretty solid on implementation, but we still have some open questions around the API.
API Design
I had previously suggested one or the other of these:
extend type Query {
  "Load a secret from its ID"
  secret(id: SecretID): Secret!
}

type Secret {
  "The identifier for this secret"
  id: SecretID!

  "The value of this secret"
  plaintext: String!

  "Create a new secret"
  new(plaintext: String!) Secret!
}

extend type Query {
  "Load a secret from its ID"
  secret(id: SecretID, plaintext: String): Secret!
}

With the following behavior:
if id == nil && plaintext == nil {
  return err
} else if id != nil && plaintext == nil {
  return the secret with id = id
} else if id != nil && plaintext != nil {
  create a new secret with this id and plaintext, unless a secret with this id already exists (and return an error in this case)
} else if id == nil && plaintext != nil {
  create a new secret with this plaintext, setting the id to sha256(plaintext)
}

@sipsma Also suggested this as a possibility:
extend type Query {
  withSecret(id: String!, plaintext: String!): Query!
}

which would allow queries like:
query {
  withSecret(id: "mysecret", plaintext: "supersecretsecret") {
    container {
      from(address: "alpine") {
        withSecretVariable(name: "MYSECRET", secret: "mysecret") {
          withExec(args: ["printenv", "MYSECRET"]) {
            stdout
          }
        }
      }
    }
  }
}

@shykes, @aluzzardi Do you like either of these, or have any other ideas for this API?

For completeness, here's another API we talked about, drawing a parallel to the cacheVolume constructor:
extend type Query {
  inMemorySecret(plaintext: String!, name: String): Secret!
}

The name param is optional and defaults to a sha256 of the plaintext value. If specified, it can be used to prevent credential rotation from busting Buildkit caches.
There may be a better name than inMemorySecret but the general idea is that this API mutates the in-memory session.Attachable to register the secret and returns a Secret whose SecretID references its name. We also considered sessionSecret but avoided it since it forces us to decide whether the term "session" is official.
Note that this means the existing Query { secret } continues to have a required id param, whereas the Secret { new } proposal above implies that we make it optional (which also implies that there's an 'empty secret' representation).

There's some big pros+cons to the approach of withSecret(id: String!, plaintext: String!): Query!.
In code it would ideally look like:
c, err := dagger.Connect(ctx)
if err != nil {
  panic(err)
}

secretName := "mysecret"
c = c.WithSecret(secretName, "supersecretsecret")

ctr := c.Container().WithSecretVariable("SECRET", secretName)
// rest of your actual pipelines

Basically, the secret would always be provided as part of the queries you submit. In our codegen clients this would not be much overhead since it can be setup with a single line and then apply to all subsequent queries if you want.
Pros:

Secrets are only available for the duration of a single pipeline. Minimal scope, very explicit control, reflects the implementation where they exist only ephemerally in memory
Avoids questions about whether loading a secret should be a mutation or not; it just uses the same with* pattern we have elsewhere in our API
Relatively convenient to use on the codegen'd client level

Cons:

Painful to use on the raw graphql level since you have to explicitly wrap all your queries in withSecret when you want them to have access to those secrets.

Might be ways of alleviating this but goes off on other tangents

I don't know yet if our codegen clients handle the case of returning Query, if not they might need updates and thus increase implementation overhead

Overall, I like the pros of this option a lot, but I'm totally on board w/ other options too if there's concerns with this one. Any of them work for me 👍

Not sure what the desired end state is going to look like.
Apologies for not having read carefully through all the comments, but I see 2 approaches mentioned:

A dedicated new secret endpoint (inMemorySecret(plaintext: String!, name: String): Secret!, new(plaintext: String!) Secret!)
Carrying secrets on each request (session) (c = c.WithSecret(secretName, "supersecretsecret"))

IIRC last time we considered this, there were pitfalls on both approaches:

Dedicated new secret endpoint breaks isolation

Generally speaking, every API query is currently self contained: order doesn't matter, everything is fully lazy, etc
The "new secret" endpoint would break this model
At the time, the problem was extensions (e.g. an extension call is cached, the secret ID is returned but the extension did not add secret)
Problem still exists, we just don't have extensions anymore. It does apply to dagger in dagger calls though, and will be relevant at some point

Session

Fixes all the problems mentioned above
... but IMHO it defeats the point of secrets a little bit

Might as well just pass plain text secrets around rather than SecretID, the end result wouldn't be so different

At this point I'm wondering what we want from secrets longer term. I'm worried that we might adopt an API that will make it harder to switch over to the "end goal"

Might as well just pass plain text secrets around rather than SecretID, the end result wouldn't be so different

The major difference is that the ID of objects wouldn't include the plaintext (what we want to avoid), it would only include the name of the secret. The query would have the plaintext (same as other options) but that only ever lives in memory.
It also could be expanded in the future to accept secrets that aren't initialized from plaintext; I think the same pattern should still be usable.
That being said, overall atm I'm actually leaning more towards inMemorySecret as I think that session statefulness is probably inevitable and we should just embrace it fully. Still thinking about it though.

Shouldn’t we be talking about the “pluggable secret stores” idea that we keep discussing as a future possibility? If we’re going to introduce a major new API design for secrets, it seems like a good time to explore that option. I know @aluzzardi that’s been on your mind.
I’m picturing a design (strawman) that leverages session state and attachables, without exposing them directly in the API. For example: l

start with cache volume style: no IDs, just names that map directly to buildkit secret id (plus maybe mangling)
on secret lookup by buildkit, session helper passes lookup to a secret store
builtin secret store just looks up the host environment. in an interactive tty it can prompt you interactively
secret store can be customized by the operator with a simple CLI hook, that can be set in ~/.config/dagger or something. For example in GHA you would configure the CLI to look up gha secrets instead.

TLDR:

leverage session state yes
expose it in API no
there is a dimension missing here which is separation of concerns between pipeline dev & ops

@shykes We actually started the discussion somewhere along those lines but figured the proposals so far (I'll call it 'static secrets') would be a faster way to deliver incremental value and unblock the Cue SDK.
Concretely, we discussed something like this (I'll call it 'dynamic secrets'):
client := dagger.Connect(ctx, dagger.WithSecretProvider(func(ctx context.Context, secret string) ([]byte, error) {
  // fetch the secret somehow (either native library, or run an extension, whatever)
  return []byte("sekrit"), nil
}))

Personally I like this because it's super simple and avoids having to figure out Dagger's relationship with various secret providers, or whether they're extensions, etc. etc.; it's just a function. The architectural challenge is this requires a new call path from the engine to the client SDK.
On the other hand, we're pretty certain static secrets will be easy to implement and will immediately unblock the Cue SDK. It's also newly unblocked by the introduction of the session helper, which feels like a nice return on investment, but you could also read that as unnecessary bias. :wink:
All that being said, I agree that we want dynamic secrets, so it would be nice if we can avoid painting ourselves into a corner just because we want to ship the Cue SDK sooner.
Do we need to decide between "static OR dynamic secrets" or do we actually need "static AND dynamic secrets"?
If it's "AND" I'd argue we continue with static secrets but make any adjustments necessary to be forward-compatible. i.e. if we think the advent of dynamic secrets deprecates SecretID, we should try to avoid it for static secrets too.
(I'm not sure if we can actually remove SecretID though - we also use it for e.g. File { secret } and Host { envVariable { secret } }.)
Are there use cases served by static secrets that would be awkward if we only had dynamic secrets?
I think so. Simple scripts might just take secrets from env vars and it'll be awkward to force that into a callback path. There are also cases where the client is constructed early in the lifecycle of a program at a point in time where it doesn't know what secrets will be needed in the future.
For example, Bass constructs the Dagger client once on boot - it'll be awkward to use dynamic secrets here since the secrets don't exist until you use the runtime. I could move the Dagger client construction to each runtime call, but that might worsen performance (assuming dagger.Connect does engine session setup/etc.)

For example, Bass constructs the Dagger client once on boot - it'll be awkward to use dynamic secrets here since the secrets don't exist until you use the runtime. I could move the Dagger client construction to each runtime call, but that might worsen performance (assuming dagger.Connect does engine session setup/etc.)

Also just realized this is actually an argument in favor of withSecret (https://github.com/dagger/dagger/issues/3442#issuecomment-1341787281) since I wouldn't need to carefully ensure secret names don't collide in shared session state.

Here's my proposal: https://github.com/dagger/dagger/pull/4193

I don't have a lot of knowledge on BuildKit secret management. @jlongtine helped understand better but I feel I'm still not completely sure how it works.
I spent some time today in creating some diagram to share and be able to have feedback on my comprehension.
I had a first diagram that I presented to @jlongtine again.
(somehow, the mermaid interpreter from GitHub can't parse the code, so I put it in the annex)

So I thought the dagger-session would completely run in the Docker container, with the graphql server as well. But @jlongtine told me that the graphql server runs in the SDK (from the provisionning logic).
So the connection between the SDK and the Dagger Engine is not GraphQL based, but more gRPC for BuildKit Gateway?

Then, the graph would look like that:

So my questions are:

What is right? What is wrong in those diagram?
Is the Dagger Session in the SDK client, or in a container? (there was a dagger-engine-session container back in the old days, like, last week)
Related: Is the session attachable client or server side?
Are the Secrets fetched from the host or from buildkit
how is the secret provider communicate with the host?

Annex: mermaid code
1st Graph: GraphQL Server in the Dagger Engine

graph LR
    subgraph Host
    subgraph UserCode[User Code]

        code --> nsfe(["core.NewSecretFromHostEnv('MY_SECRET')"])
    end

    UserCode --> SDK

    subgraph SDK
        gqlbuilder[GraphQL query builder]

        gqlbuilder -- build query --> hvs

        hvs[ HostVariable  secret ] ~~~ spc
        spc[Secrets Provider Connection] --> pt
        pt(["secret.PlainText()"]) --> ge(["os.GetEnv('MY_SECRET')"])
    end

    end

    SDK -.- provision -.-> DaggerEngine
    SDK -- GraphQL --> DaggerEngine
    subgraph Docker
        subgraph DaggerEngine
            direction LR

            subgraph GraphQLServer

                gs[GraphQL Server] -- GraphQL to LLB --> LLB[LLB DAG]
                LLB --- as(["llb.AddSecret( 'MY_SECRET' )"])

                as ~~~ sa

                sa[Session Attachable] ---> sp
                sp[Secrets Provider] --> getSecret(["GetSecret('MY_SECRET')"])
            end

            GraphQLServer -- query gRPC --> BuildKit

            subgraph BuildKit

                bk[BuildKit]

            end

            GraphQLServer -- session connection --- BuildKit
            BuildKit -- inject secrets from\n provider as file mounts --> Containers

            subgraph Containers

                c1[Container 1] --> c2[Container 2]
            end

        end

    end
    SDK == secrets provider session attachable connection === DaggerEngine

2nd Graph: GraphQL server in the SDK/Client

graph TD
    subgraph Host
        direction LR

        subgraph UserCode[User Code]
            direction TB

            code --> wev(["WithEnvironmentVariable('MY_SECRET')"])
        end

        wev --> gqlbuilder

        subgraph SDK
            direction TB

            gqlbuilder[GraphQL query builder]

            gqlbuilder -- build query --> gs

            subgraph provisioner

            end

            subgraph GraphQLServer

                gs[GraphQL Request] -- GraphQL to LLB --> LLBDAG{{LLB DAG}}

            end

            subgraph DaggerSession
                bkgwClient[BuildKit Gateway client]
                sa[Session Attachable] --- sp
                sp[Secrets Provider] --> getSecret(["GetSecret('MY_SECRET')"])
                getSecret --> pt
            pt(["secretIDPayload{FromHostEnv: 'MY_SECRET'}.PlainText()"]) --> ge(["os.GetEnv('MY_SECRET')"])

            end

        end
    end

    bkSession -- "session connection (gRPC)" --- sa

    provisioner -- provision --> DaggerEngine

    subgraph Docker
        subgraph DaggerEngine
            direction LR

            subgraph BuildKit
                bkSession[Session]
                solver

                bkgwClient -- "query (gRPC)" --> bkSession
                bkSession --> llbDAG{{"LLB DAG"}}
                LLBDAG --> bkgwClient
                llbDAG --> as(["llb.AddSecret( 'MY_SECRET' )"])
                as -- GetSecret --> sp
            end

            ge ---> solver
            solver -- inject secrets from\n provider as file mounts --> Containers

            subgraph Containers

                c1[Container 1] --> c2[Container 2]
            end

        end

    end

Note, I gave this design proposal a nickname ("in-memory secret store") to make it easier to distinguish from other design proposals for handling dynamic secrets. See #4426 (this issue is listed as "option B").

My 2c, the original name made more sense to me. This one implies a specific secret provider implementation, which wasn't the intent when @jlongtine opened it.

My 2c, the original name made more sense to me. This one implies a specific secret provider implementation, which wasn't the intent when @jlongtine opened it.

The problem is that we have several possible ways of "creating a new secret via the API", and we need a way to distinguish them. Right now we have "in-memory store" (store the value of the secrets in memory) + "helpers" (retrieve the value of secrets from helpers). What names did you have in mind?


Small updates to the TypeScript types

This doesn't change anything about the implementation. It just changes the types to match the current reality a little bit more strictly.

Looks like the tsc script failed due to the readonly addition. If you remove that, I think the other change is good for this PR.

Hey @jameskerr! Thanks for the feedback, I just noticed the tests were not passing, so I made a few more internal annotations to fix those. The benefit of expressing that the data you pass won't be mutated by marking it readonly on the outside is that you can feed both mutable and readonly arrays to the Tree component this way.
In my use case, I found myself writing <Tree data={[...myReadonlyArray]} /> instead of just <Tree data={myReadonlyArray} />.

Great! That makes sense to me.


How to create admin / root user and set its password after a freshly installed InfluxDB 2.3?

Maybe somebody has a step-by-step-manual for that. No matter what i do, i always get a 401 error.
root@test:/var/lib/influxdb# influx auth create --org my-org --all-access Error: failed to lookup org with name "my-org": 401 Unauthorized: unauthorized access
root@test:/var/lib/influxdb# influx auth list Error: could not find authorization with given parameters: 401 Unauthorized: unauthorized access
`
root@test:/var/lib/influxdb# curl http://localhost/api/v2/authorizations

404 Not Found

Not Found
The requested URL was not found on this server.

Apache/2.4.25 (Debian) Server at localhost Port 80

`
root@test:/var/lib/influxdb# influx user ls Error: failed to list users: 401 Unauthorized: unauthorized access
root@test:/var/lib/influxdb# influx user list Error: failed to list users: 401 Unauthorized: unauthorized access
root@test:/var/lib/influxdb# influx user create -n root -p Test12345 -o myorg-com Error: failed to lookup org with name "myorg-com": 401 Unauthorized: unauthorized access
Thank you very very much for your feedback(s) :-)
With best regards from Switzerland,
Patrick

-> same problem when i write noauth into config file
root@xili:/etc/influxdb# cat config.toml bolt-path = "/var/lib/influxdb/influxd.bolt" engine-path = "/var/lib/influxdb/engine" auth-enabled = false

@kosuodhmwa  @jayjupdhig
You need to run the influx setup command to initialize the operator token/user.


Data-only container constantly restarting

I set up some data only containers as sidekicks, and for some reason they continually restart. The app still works, so it's not too concerning, but I do wonder why it's happening. Here's what it looks like in docker-compose.yml:
ps-mongo-data:
  image: busybox
  volumes:
    - /data/db
ps-mongo:
  image: mongo
  labels:
    io.rancher.sidekicks: ps-mongo-data
  volumes_from:
    - ps-mongo-data
  ports:
    - "3001:27017"
# more stuff that's not relevant continues...

You need to add in tty: true to your busybox container.
ps-mongo-data:
  image: busybox
  volumes:
     - /data/db
   tty: true

@alena1108 pointed out that since it's a data container, you can add this label to cause the container to only start once and stay stopped.
ps-mongo-data:
  image: busybox
  volumes:
     - /data/db
  labels:
    io.rancher.container.start_once: true


⚠ Samsung has degraded performance

In 957e987, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2317 ms

Resolved: Samsung performance has improved in 9311790.


Fix brew fetch --force for GitHub Packages bottle tab download.

Have to implement some custom logic here as there's two resources rather than one.

Review period skipped due to critical label.


Link non-root nodes

link(v,w) where v is not the root of its represented tree.

flipped flags can be used following Sleator's code:
https://codeforces.com/contest/117/submission/860934

The basic idea is to reroot(v) to make v the root of its tree using flipped state:
https://codeforces.com/blog/entry/75885


Bug: "maximum display length" setting is not taking effect

We have lengthy option, by default it will be displayed with first 35 char.
We configured "maximum display length" but it is not working as expected.
Note: it used to work, but recently notice it is not working anymore.

We are also experiencing this issue. It used to work, and now it does not. The custom value we have entered on the control extension is 75. I attempted to correct this issue by clearing out the value and saving (thus reverting to the default 35 character limit), and then going back into the control extension and setting the value back to 75 and then saving. This made no difference.
The display is showing 30 characters, plus the periods (i.e. ...), so not even the 35 which is supposed to be the default value.
I have confirmed this issue is affecting our multivalue controls in two separate Azure DevOps processes (both within the same organization).

@AminTi my colleague found below:
The error seems to be in the file \src\MultiValueControl.tsx
Line 109: label={this._wrapText(o.length > 30 ? ${o.slice(0, 30)}... : o)}
this line should use the property named _labelDisplayLength which can be configured (default value 35).

I think Line 109 should probably just be: label={this._wrapText(o)}
because the _wrapText function handles adding the ellipsis if the length of the label text is greater than the _labelDisplayLength property.
private _wrapText(text: string){
    return text.length > this._labelDisplayLength ? `${text.slice(0,this._labelDisplayLength)}...` : text;
}

I think Line 109 should probably just be: label={this._wrapText(o)} because the _wrapText function handles adding the ellipsis if the length of the label text is greater than the _labelDisplayLength property.
private _wrapText(text: string){
    return text.length > this._labelDisplayLength ? `${text.slice(0,this._labelDisplayLength)}...` : text;
}

using _labelDisplayLength can be configured (using config files located in config folder. By default it is 35 characters.
Line 34 in same file has the initiator of _labelDisplayLength:
private readonly _labelDisplayLength: number = VSS.getConfiguration().witInputs.LabelDisplayLength ? VSS.getConfiguration().witInputs.LabelDisplayLength : 35;


Highlight type variables as types

Currently, type variables are highlighted as variables. It would make more sense for them to be highlighted as types. This could be done by checking if the variable is an instance of a typing.TypeVar or similar types.
Some motivating examples:
T = typing.TypeVar("T")
Number = typing.Union[int, float]
IntList = typing.List[int]

Currently, these are highlighted as variables. Highlighting them as types would be really helpful for readability.

The three symbols in your example above are all examples of "type aliases". Most languages provide a syntax for defining type aliases, but Python uses the same syntax for variables and type aliases. PEP 613, which will be in Python 3.10, starts to address this shortcoming. In the meantime, type analyzers like pylance need to use heuristics to distinguish between a normal variable and a type alias.
@bschnurr, do we use a different semantic token for symbols that pyright's heuristics identify as a type alias? I think that's what this feature would require.

The three symbols in your example above are all examples of "type aliases". Most languages provide a syntax for defining type aliases, but Python uses the same syntax for variables and type aliases. PEP 613, which will be in Python 3.10, starts to address this shortcoming. In the meantime, type analyzers like pylance need to use heuristics to distinguish between a normal variable and a type alias.
@bschnurr, do we use a different semantic token for symbols that pyright's heuristics identify as a type alias? I think that's what this feature would require.

looks like we don't have this logic in yet.
There is a "type" token we should be returning instead.

looks like we don't have this logic in yet.
There is a "type" token we should be returning instead.

Any updates?

This enhancement request has received only a small number of upvotes (thumbs up reactions). We generally use that feedback to prioritize feature work in Pylance. Once a feature request has reached a sufficiently high priority level and the team decides to pursue it, the issue will receive a "Backlog" label.

Moving this issue to discussion as an enhancement request for comments and upvotes.


Setting up CI/CD, ESLint and Jest for the NomandCraft project

Description
To ensure code quality and process automation, you need to set up CI/CD, linting and testing. This will allow:

Automate code review for every push and pull request.
Maintain a consistent coding style using ESLint.
Ensure project functionality is verified through automated tests using Jest.

Tasks:

Set up GitHub Actions for CI/CD:

Create a file .github/workflows/ci.yml.

Set up automatic steps:

Cloning a repository.
Installing dependencies.
Code linting.
Running tests.

Configure ESLint:

Create a configuration file for ESLint.
Make sure ESLint works for backend and frontend.
Add global settings for Jest.

Set up Jest for testing:

Install Jest as a dev dependency.
Create a test file to test the functionality of Jest.
Ensure that tests run correctly locally and in CI.

Add scripts to package.json:

Add commands for linting and testing:
"scripts": {
  "lint:backend": "cd backend && npm run lint",
  "lint:frontend": "cd frontend && npm run lint",
  "lint:all": "npm run lint:backend && npm run lint:frontend"
}

Check CI/CD execution:

Make sure that Workflow successfully passes all stages.
Fix possible configuration errors.

Commit  d623abfe0dbc94a56fc4deb4997da75646aa106e is related to this issue.


Get available cast devices

Is it possible to get a list with the available cast devices with this library?

Hey @tehuster, it is currently limited to cast to only 1 case device on the network ( the one you choose ). But this is an interesting idea, I might add this in the future 👍

Hey @Fenny, thanks for the reply. I think my question was a little confusing because my goal is not to cast to multiple devices.
I was wondering if it's possible to get a list with all available devices without using the "google cast tab". So i can make an UI myself for all available devices and let the user select a device without using that previous named tab.

Ah I see, that is not possible. I remember trying that back in 2018, maybe they updated their API ( not sure ). But the user must select the cast device from the chromium engine.

Hi @Fenny, any news on this ?
Is it still a limitation from the Chromcast Web SDK ?
And is it possible to cast two videos silmutaneously on two chromecast devices ?
Thank you for you work


Build meme buffer

Problem:
meme command takes a lot of time (reddit api slow)
Proposed solution:

Request multiple post either when

a certain time has passed
a certain number of posts is left in the buffer

cache them in a buffer
serve answers from the buffer instead of requesting them. directly from reddit

clear outdated posts (threshold to be defined) from the buffer on a regular basis

 [Example task]


MINOR: Retry finding coordinator in testResetOffsetsNotExistingGroup …

…test
This patch adds retries when finding the coordinator in ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup. This test has intermittently failed due to a CoordinatorNotAvailableException, which is a retriable error. We should retry on these errors and rely on the TestUtils.waitUntilTrue timeout to bound the test run time.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Example of this failure: https://builds.apache.org/job/kafka-trunk-jdk11/736/testReport/kafka.admin/ResetConsumerGroupOffsetTest/testResetOffsetsNotExistingGroup/

Fixing flaky tests is great. I don't understand the fix though. We were already retrying, it seems. The PR seems to add a maxRetries parameter.

I don't think we were retrying. The default maxRetries value is 0, and the test run only lasted 4 seconds, while the default time waitUntilTrue will wait is 15 seconds. I changed the test condition to just throw, and it only ran once. I wasn't comfortable changing the waitUntilTrue default without getting a second opinion, since a ton of tests use it, but that's another approach we could take here.
While investigating that more today, I also noticed that waitUntilTrue won't correctly handle a RetriableException wrapped by a concurrent.ExecutionException, so I added a util to handle that.

Interesting. I have no idea why we have that default, it seems pretty unintuitive. Maybe @hachikuji can provide some context on why we have that.

Looks like retrying on exceptions was added by this PR: https://github.com/apache/kafka/pull/5032. It will always retry when the predicate evaluates to false, but by default won't retry exceptions, presumably to maintain previously-existing behavior. Maybe the parameter just needs to be changed to something like maxExceptionRetries.

OK, I see how it works now. It will propagate the exception by default. So, I'm not sure if this is the right fix. The retries should be happening internally, why is it being bubbled up?

@ijuma I know one issue is that the admin client still fails when retries are exhausted even if there is time remaining in the timeout. I filed a jira about this: https://issues.apache.org/jira/browse/KAFKA-8503. My suggestion was to ignore the retries config if a timeout was provided.

Interesting. I'll comment on the JIRA. For this PR, maybe we can increase the number of retries explicitly.

testResetOffsetsExistingTopic failed recently as well, which only relies on internal retries, so I'm following @ijuma's suggestion and maxing out the retries.


Query<> FirstOrDefault Error

In current Master branch (8334a03ff06d2bbe8dcd4caf0c342cd4730202c7)
Calling
    Database.Query<>.FirstOrDefault(i=>i.id) throws

In SQLCE (and probably other providers although I didn't verify) generates
SELECT...
WHERE ([A].[Id] = @0)
OFFSET @1 ROWS FETCH NEXT @2 ROWS ONLY

Which throws OFFSET clause is invalid unless ORDER BY clause is specified.

Branch dev2.0 is the latest on NuGet and this should not occur in that version.

Thanks


Any ETA for 1.1.1 ?

Great job on this library! Is there any ETA for the stable release of 1.1.1 ? It has been several months since the last beta.

Hello! I have no clear estimation, but I can share my plans on the stable release.

The main goal is to deprecate security-crypto-datastore artifacts because of #31 and provide API allowing usage of security-crypto keys. Provide smooth plan for migration.
Currently datastore-encrypted artifacts are pure Kotlin/JVM libraries. So another task is to keep simplicity of adding the library into a project providing a single artifact for JVM and Android targets (done in #77)


Design Meeting Notes, 7/12/2024

Making Type Error Elaboration More Consistent
https://github.com/microsoft/TypeScript/pull/58859

Problem: you may get a different error elaboration in different locations depending on the order in which a program is checked.

There are various reasons for this,  with the class of differences we're concerned about here, but we try to avoid reporting the full context if you've already seen it.

The main reason we're focusing on this is region-based diagnostics. If you first check statements in the middle of the file, it is weird to get a different level of error detail than top-down checking.
In this PR, every time we report a type error, we will now provide the full elaboration.

Deferred Parameters and Control Flow Analysis for Function Expression Arguments
https://github.com/microsoft/TypeScript/pull/58729

Feel it is a valuable fix for a long-standing problem.

Core idea is to assume that callback function expressions are possibly called, and to factor that into the control flow analysis of variables used afterwards.

Means that in cases like
let x: number | undefined;
x = 42;
someArray.forEach(() => { x = undefined; });
x++; // Should be an error. Was not previously, is an error with this PR.

now correctly fail to type-check.

To opt out, the idea is to introduce a deferred modifier; but there's a compatibility problem with all the tools in the ecosystem understanding that. Declaration files will need to introduce it and split between typesVersions, existing code may need an annotation even in .ts files.

PR also has support for /** @deferred */ JSDoc comment that is understood in .ts files - not just .js files.

Also a question of a new parameter modifier being possibly in conflict with JavaScript.

We don't see one, but it might be worth reaching out to the committee to better understand concerns.

What's the relationship between a deferred callback and await?

It doesn't really factor in; we don't analyze deferred callbacks other than at the use-site.

How does declaration emit work for the JSDoc tag?

What you write is what you get.

What happens when a library is not upgraded to use deferred?

You can just use a type assertion to the expected type to opt out.

Alternatively, and maybe more preferred, is to write a function that takes a deferred callback and returns it.
function deferred<F extends (...args: never[]) => unknown>(deferred callback: F): F {
    return callback;
}

We want to make sure that we're not missing anything. Will do the work to get Node.js and lib.d.ts/DOM libraries to have deferred.

type DataOrNull<T> = {
    data: T | null;
};

declare function check(s: DataOrNull<string>): void;

declare let foo: DataOrNull<string | null>;
check(foo); // Error?

We "smartly" infer that T is covariant (please disregard the soundness aspect here).

When we ask if DataOrNull<string | null> is assignable to DataOrNull<string>, we try to just ask "is string | null assignable to string?" and answer "no".

One idea: give people a new opt-out for variance calculation?
type DataOrNull<unmeasurable T> = {
    data: T | null;
};

"Unmeasurable" is the concept we use internally to say "this type parameter must always be checked with respect to the structure of the containing type".

What do we name this?
type DataOrNull<sus T> = {
    data: T | null;
};

structural?

linear?

inconsistent?

nonvariant

Could we infer this better?

The pattern is common, and if we don't infer some variance in these scenarios, things are likely to get quite a bit slower.

Isn't a similar problem going to come up with libraries prioritizing correctness?

Unmeasurable variance is pretty infectious with type parameters. A type reference that takes a type variable for an unreliable type parameter makes the type variable also unreliable with respect to its containing type.

Could view this as "unreliable" - another internal view of type parameters, where they might have a variance, but uses the structural fallback.

What about this on the use-site?
declare let foo: DataOrNull<structural string>;

Hard to think about what that means/explain to developers.

There are hacks to induce unreliable variance. @RyanCavanaugh will post this.

Ref #59049, #9998

I work with Hack (also created at Facebook) which shares the same conventions as Flow, invalidating all assertions when calling impure functions.
Hack & Flow's absolute approach is impractical for TypeScript, but you might want to introduce a config option for people with small projects to opt into stricter behaviour themselves.
I took the config approach with Psalm (a SA tool for PHP). On a ~1MLOC legacy codebase that option was not enabled, but many smaller projects have that option enabled.


HLS (.m3u8) format video from file not playing on iOS devices

I'm trying to play videos offline, and this works on Android but on iOS, even when run on real device it's showing the following error:
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(VideoError, Failed to load video: The operation couldn’t be completed. (CoreMediaErrorDomain error -12865.), null, null)
I'm wondering if there's anything we missed to make these files play?

Same issue with me too, it is playing smoothly in Android but not in iOS. Did you find any solution ?

@amani27 this issue is squarely at the feet of video_player, since chewie is merely a UI wrapper around that library.
Please raise an issue in the Flutter Github page.


[CINFRA-508] Feature/leader election set write concern on time

This patch causes the supervision to set effectiveWriteConcern and assumedWriteConcern right when the leader election happens;
Scope & Purpose
(Please describe the changes in this PR for reviewers, motivation, rationale - mandatory)

[ ] :hankey: Bugfix
[ ] :pizza: New feature
[ ] :fire: Performance improvement
[ ] :hammer: Refactoring/simplification

Checklist

[ ] Tests

[ ] Regression tests
[ ] C++ Unit tests
[ ] integration tests
[ ] resilience tests

[ ] :book: CHANGELOG entry made
[ ] :books: documentation written (release notes, API changes, ...)
[ ] Backports

[ ] Backport for 3.10: (Please link PR)
[ ] Backport for 3.9: (Please link PR)
[ ] Backport for 3.8: (Please link PR)

Related Information
(Please reference tickets / specification / other PRs etc)

[ ] Docs PR:
[ ] Enterprise PR:
[ ] GitHub issue / Jira ticket:
[ ] Design document:

This still needs a test.