Metric evaluation problems in multi-node, shared file system

Describe the bug
Metric evaluation fails in multi-node within a shared file system, because the master process cannot find the lock files from other nodes. (This issue was originally mentioned in the transformers repo https://github.com/huggingface/transformers/issues/17412)
Steps to reproduce the bug

clone this huggingface model and replace the run_speech_recognition_ctc.py script with the version in the gist here.
Setup the venv according to the requirements of the model file plus datasets==2.0.0, transformers==4.18.0 and torch==1.9.0
Launch the runner in a distributed environment which has a shared file system for two nodes, preferably with SLURM. Example here

Specifically for the datasets, for the distributed setup the load_metric is called as:
    process_id=int(os.environ["RANK"])
    num_process=int(os.environ["WORLD_SIZE"])
    eval_metrics = {metric: load_metric(metric,
                                        process_id=process_id,
                                        num_process=num_process,
                                        experiment_id="slurm")
                    for metric in data_args.eval_metrics}

Expected results
The training should not fail, due to the failure of the Metric.compute() step.
Actual results
For the test I am executing the world size is 4, with 2 GPUs in 2 nodes. However the process is not finding the necessary lock files
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 841, in <module>
    main()
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 792, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/trainer.py", line 1497, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/trainer.py", line 1624, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/trainer.py", line 2291, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/trainer.py", line 2535, in evaluation_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 742, in compute_metrics
    metrics = {k: v.compute(predictions=pred_str, references=label_str) for k, v in eval_metrics.items()}
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 742, in <dictcomp>
    metrics = {k: v.compute(predictions=pred_str, references=label_str) for k, v in eval_metrics.items()}
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/datasets/metric.py", line 419, in compute
    self.add_batch(**inputs)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/datasets/metric.py", line 465, in add_batch
    self._init_writer()
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/datasets/metric.py", line 552, in _init_writer
    self._check_rendez_vous()  # wait for master to be ready and to let everyone go
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/datasets/metric.py", line 342, in _check_rendez_vous
    ) from None
ValueError: Expected to find locked file /home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-0.arrow.lock from process 3 but it doesn't exist.

When I look at the cache directory, I can see all the lock files in principle:
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-0.arrow
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-0.arrow.lock
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-1.arrow
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-1.arrow.lock
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-2.arrow
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-2.arrow.lock
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-3.arrow
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-3.arrow.lock
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-rdv.lock

I see that there was another related issue here https://github.com/huggingface/datasets/issues/1942, but it seems to have resolved via https://github.com/huggingface/datasets/pull/1966. Let me know if there is problem with how I am calling the load_metric or whether I need to make changes to the .compute() steps.
Environment info

datasets version: 2.0.0
Platform: Linux-4.18.0-147.8.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
Python version: 3.7.4
PyArrow version: 7.0.0
Pandas version: 1.3.0

If you call metric.compute in a distributed setup like yours, then metric.compute is called in each process. metric.compute first calls metric.add_batch, and it looks like your error appears at that stage.
To make sure that all the processes have started writing their predictions/references at the same time, each process waits for process 0 to lock slurm-{world_size}-0.arrow.lock. Process 0 locks this file when metric.add_batch is called, so here when metric.compute is called.
Therefore your error can happen when process 0 takes too much time to call metric.compute compared to process 3 (>100 seconds). I haven't tried running your code but could it be the case ?
I guess it could also happen if you run multiple times the same distributed job at the same time with the same experiment_id because they would collide.

We've finally been able to isolate the problem, it wasn't a timing problem, but rather a file locking one.
The locks produced by calling flock where not visible between nodes (so the master node couldn't check other node's locks nor the other way around).
We are now having issues with the pre-processing in our runner script, but are not related with the rendezvous process during the evaluation phase. We will let you know about it once we address it.
Our solution to the rendezvous is as follows:

We solved the problem by calling lockf instead of flock.
We had to change slightly the _check_all_processes_locks method so that the main process (i.e. process 0) didn't check it's own lock (because lockf permits recursive locks and thus checking it only replaced the current lock with a new one).

We use a shared file system between nodes using GPFS in our cluster setup. Maybe the difference between the behavior we see with respect to your usage in multi-node executions comes from that fact. Which file system scheme do you use for the multi-node executions?
lockf seems to work in more settings than flock, so maybe we could write a PR so you could test it in your environment.

Cool, I'm glad you managed to make evaluation work :)
I'm not completely aware of the differences between lockf and flock, but I've read somewhere that flock is preferable over lockf in multithreading and multiprocessing situations. Here we definitely are in such a situation so unless it is super important I don't think we will switch to lockf

We had to change slightly the _check_all_processes_locks method so that the main process (i.e. process 0) didn't check it's own lock (because lockf permits recursive locks and thus checking it only replaced the current lock with a new one).

Hi, Can you share your _check_all_processes_locks function? thanks!

def _check_all_processes_locks(self):
        expected_lock_file_names = [
            os.path.join(self.data_dir, f"{self.experiment_id}-{self.num_process}-{process_id}.arrow.lock")
            for process_id in range(self.num_process)
        ]
        #for expected_lock_file_name in expected_lock_file_names: # OUR CHANGE process 0 shouldn't check its own lock
        for expected_lock_file_name in expected_lock_file_names[1:]:
            nofilelock = FileFreeLock(expected_lock_file_name)
            try:
                nofilelock.acquire(timeout=self.timeout)
            except Timeout:
                raise ValueError(
                    f"Expected to find locked file {expected_lock_file_name} from process {self.process_id} but it doesn't exist."
                )
            else:
                nofilelock.release()

Changed files:

metric.py file in the datasets library
filelock.py file in the datasets/utils library.

Changes we made:

We changed the flock for lockf
flock and lockf both perform a lock over a file (like the lock for writing).
The difference is that flock only works in local file systems, but if you have a shared file system (like what we have in the clusters) the flock fails to “see” the lock of another node. The only disadvantage we had was that a single process couldn’t detect it’s own lock so we did the second change.
We prevented the process 0 (which is the one that coordinates the rendezvous) from checking its own lock on its arrow because it didn't work with lockf (as stated in the previous change).
We made a second rendezvous so that all the process had the results of the metrics (other than the loss) and not only the process 0.
What happened was that only process 0 computed the metric and that didn’t present any problem if you are using the loss. However, if you are using another metric, the only process which had the information to choose the best checkpoint at evaluation time was the process 0. But since the evaluation was performed over all processes, every process except the process 0 chose a bad check point (bad meaning it wasn’t the best one) because they didn’t have the information of the metric of the best checkpoint.
The consequence was that the evaluation was different from what would result if using only the best checkpoint, because each process chose a different checkpoint to run the evaluation and thus the numbers were often worse than the numbers that would be obtained if all processes choose the best checkpoint (correct one) to perform the evaluation of their samples.
We performed a second rendezvous so that all processes had the same best_metric and best_model as process 0 after the evaluation cycle.

Metrics are deprecated in datasets and evaluate should be used instead: https://github.com/huggingface/evaluate


Can get any file

i downloaded the plugin using package control.
In the console i get:
Exception in thread Thread-5:
Traceback (most recent call last):
File "ftpsyncwrapper in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 778, in action
File "ftpsyncwrapper in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 290, in retryingCommand
File "ftpsyncwrapper in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 284, in call
File "lib3.ftplib in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 456, in retrlines
File "lib3.ftplib in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 396, in transfercmd
File "lib3.ftplib in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 356, in ntransfercmd
File "lib3.ftplib in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 334, in makepasv
File "lib3.ftplib in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 877, in parse227
FTPSync.lib3.ftplib.error_reply: 220 Firewall Authentication required before proceeding with service
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./threading.py", line 901, in _bootstrap_inner
File "FTPSync in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 2946, in run
File "FTPSync in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 2098, in execute
File "FTPSync in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 2152, in listFiles
File "ftpsyncwrapper in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 819, in list
File "ftpsyncwrapper in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 949, in __execute
File "ftpsyncwrapper in C:\Users\raulmatias\AppData\Roaming\Sublime Text 3\Installed Packages\FTPSync.sublime-package", line 783, in action
FTPSync.ftpsyncexceptions.FileNotFoundException
b'FTPSync > Closing connection'
and no result at all.  Can´t do anything.

Hi,
"Firewall Authentication required before proceeding with service"
Either on your hosting's side or your PC there's a firewall, that needs to be configured so that Sublime Text can send requests through...


Proposed new PR template

Description
Proposal for a much more concise PR template (I dug up my old branch, and trimmed the previous proposal even more).

I can't add Pat Nichols as a reviewer for some reason; guessing he doesn't have access to this repo?

All: I've added a [very short] section with a checklist (suggested by Dom in one of the comments above) -- what do you think?

We'll need one more approval from an org owner ... @ytremolet now that the demo is over we can bug you again ;-)

@shlyaeva I have no strong opinion on @ytremolet's suggestion, but I will note that not all repositories have unit tests (either because it's simply not applicable, or because none have been added). Up to you!

I think I've resolved all the comments, and this is good to go now. (I don't have the rights to merge)


How to calculate depth accuracy of D435 and D455?

In the datasheet, the depth accuracy of D435 at 2 meter working distance is less than 2%. And D455's depth accuracy at 4 meter is less than 2%.
So, if there is an object 1 meter away, depth accuracy is 20mm? But in the specification, it is normally 2~5mm depth accuracy at 1 meter working distance.
How can I calculate right depth accuracy?

@JANGSOONMYUN
The error rate of depth "2%" is lough. There are not exist about absolute depth accuracy of D4xx series. Depth error of RealSense can be changed by light source, shape of object, and so on. So I recommend to measure error of RealSense by setting your camera environment and application.

Thanks @wodndb for your advice to @JANGSOONMYUN !
Hi @JANGSOONMYUN  The above advice of @wodndb is right to say that accuracy can be affected by a range of factors, such as depth resolution, distance of the camera from the object and environmental conditions.
I would clarify that the percentage represents % of the distance from the camera.
Yes, at 1 meter distance an accuracy of 2.5 mm to 5 mm is expected with 400 Series cameras.
D455 has 2x the accuracy over distance of the D435 models.  So at 6 meters distance, the D455 should have the same depth accuracy as the D435 models do at 3 meters.
You can get feedback about your camera's current depth accuracy using the Depth Quality Tool program.

Thank you @wodndb and @MartyG-RealSense
However, you said "the percentage represents % of the distance from the camera". It means if the distance from the camera is 1 meter, then 2% of the distance is 20 mm. But why it is 2.5mm to 5mm in the datasheet? It is too different.
I understood accuracy can be affected by environments. I just want to know rough accuracy not exact accuracy.

@MartyG-RealSense
Can we roughly calculate depth accuracy like this?
D435

0 ~ 1 m depth depth error : 2.5 ~ 5 mm

Interpolate error value based distance.
ex: 0.8 meter --> depth error ≤ (2.5 + 0.8 × 2.5 = 4.5 ㎜)

1 ~2 m depth accuracy : 2% of distance.

Multiply percent to distance.
ex: 1.5 meter --> depth error ≤ (1500 × 0.02 = 30 ㎜)

@JANGSOONMYUN   The full version of the statement is "less than 1% of the distance from the camera".  The camera is officially capable of accuracy of < 1%, though in real-world conditions 2% may be more easily achievable.  Even then though, there is no published explanation as far as I know for how the quoted 2.5 to 5 mm range of values is arrived at, so I cannot help with an explanation for that.
The official formula for calculating error over distance (RMS Error) is provided in point 5 of the section of Intel's camera tuning guide linked to below.
https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#section-verify-performance-regularly-on-a-flat-wall-or-target

Thank you very much @MartyG-RealSense @wodndb

Thanks very much @JANGSOONMYUN for the update!


Optimizing TagBuilder.CreateSanitizedId to create buffer only when needed.

https://github.com/aspnet/Mvc/issues/2512

/cc: @rynowak

#2512 mentions "a few megabytes". But I see nothing about how much of that memory is saved with this fix. Since the id will need to be sanitized a large percentage of the time, my expectation is the added complexity does not buy us much.

:watch:

Perf measurements:
Used the following View:
@model CreateSanitizedId.User

<form asp-action="Create" asp-controller="Home">
    @Html.Label("Hello¡World", "Hello¡World", null)
    @Html.Label("¡HelloWorld", "Hello¡World", null)
    @Html.EditorForModel()
    <input type="submit" value="ok" />
</form>

Used the following model:
public class User
{
    [Required]
    public string Name { get; set; }

    [Required]
    public string Occupation { get; set; }
    }

Around 33% of the page has invalid Ids which go through TagBuilder.CreateSanitizedId. I ran 10,000 requests and measured perf.
Data without the change:
System.String allocated by CreateSanitizedId: 1.84 mb
System.Text.StringBuilder allocated by CreateSanitizedId: 1.68 mb
Data after the change:
System.String allocated by CreateSanitizedId: 0.720 mb
System.Text.StringBuilder allocated by CreateSanitizedId: 0.560 mb
This is around 60% improvement from the original case.

Looks good to me :shipit:

:shipit:

Thanks! https://github.com/aspnet/Mvc/commit/57f1ae755f1cf4afbcc5d7dd5436b33befdbdfa3


Merge in master changes since 10/09/17

Merge in changes from master into the discogapic branch.
The diff of this PR's code against origin/master for the baseline tests contains only comments to the generated config (specifically, a comment that explains config.proto:resource_name_treatment).

LGTM

Codecov Report

Merging #1621 into discogapic will increase coverage by 0.04%.
The diff coverage is 96.17%.

@@               Coverage Diff                @@
##             discogapic    #1621      +/-   ##
================================================
+ Coverage          85.5%   85.54%   +0.04%
- Complexity         4762     4775      +13
================================================
  Files               457      457
  Lines             19029    19123      +94
  Branches           2049     2053       +4
================================================
+ Hits              16270    16359      +89
- Misses             2140     2142       +2
- Partials            619      622       +3

Impacted Files
Coverage Δ
Complexity Δ

...le/api/codegen/viewmodel/CredentialsClassView.java
100% <ø> (ø)
2 <0> (ø)
:arrow_down:

...i/codegen/viewmodel/metadata/VersionIndexView.java
100% <ø> (ø)
5 <0> (ø)
:arrow_down:

...e/api/codegen/util/csharp/CSharpNameFormatter.java
73.68% <ø> (ø)
11 <0> (ø)
:arrow_down:

...oogle/api/codegen/util/java/JavaNameFormatter.java
72% <ø> (+4%)
14 <0> (+1)
:arrow_up:

.../codegen/viewmodel/testing/SmokeTestClassView.java
80% <ø> (ø)
3 <0> (ø)
:arrow_down:

...om/google/api/codegen/util/go/GoNameFormatter.java
86.36% <ø> (+4.54%)
15 <0> (+1)
:arrow_up:

.../google/api/codegen/viewmodel/GrpcEnumDocView.java
100% <ø> (ø)
3 <0> (ø)
:arrow_down:

...gle/api/codegen/viewmodel/DynamicLangXApiView.java
63.63% <ø> (ø)
9 <0> (ø)
:arrow_down:

...om/google/api/codegen/util/js/JSNameFormatter.java
81.81% <ø> (ø)
14 <0> (ø)
:arrow_down:

.../google/api/codegen/util/php/PhpNameFormatter.java
77.27% <ø> (+4.54%)
14 <0> (+1)
:arrow_up:

... and 27 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f2edd58...2ed13be. Read the comment docs.


Why do you need to read a spec file at runtime?

I'm trying to understand why this library has runtime file dependencies. It seems like you could just construct an element of ConfigSpec in your Haskell source and avoid the trouble of locating files at runtime and including Aeson as a (massive) dependency. What's the advantage to having a spec file? Or am I misunderstanding something?

Hi Langston!
Thanks for your feedback, I'll try to give some rationale around the why of some of these:

It seems like you could just construct an element of ConfigSpec in your Haskell source and avoid the trouble of locating files at runtime and including Aeson as a (massive) dependency.

Yes, in theory, you may be able to create the ConfigSpec record in Haskell, there is nothing stopping you to do that, you will still require aeson for the default value though. I've been pondering with the idea of providing a more generic Spec.ConfigValue and ConfigValue types that do not rely on JSON.Value for their default value, but most of the API will lose it benefits rapidly, also for the ConfigValue side of things, the compositionality may suffer a bit.
In regards to locating files at runtime, you are right, having to locate the configuration spec file at runtime is a bit silly, I've been doing recent work to make this step a compilation step (via TemplateHaskell). For every other file though, it makes sense to look them up at runtime, the place where you run your application is unlikely to be the same place where you compile it, and you may have sensitive values in those configuration files that you don't want to include in the binary.
In regards to aeson being a massive dependency, it may be massive, but it is also ubiquitous (you'll likely have it already on your package local db); I went with that fact as something that made it a sensible choice.

What's the advantage to having a spec file?

It's the whole point of the library really, the spec file is the canonical place where all configuration things are considered, currently you define:

All runtime configuration files of your program
All environment variables that your program may need (and where they should be bound to in the configuration map)
All CLI options your program needs (and where they should be bound to in the configuration map)
Other capabilities in the future (AWS ParameterStore is in the roadmap)

With this spec, you separate the source and precedence of your configuration values, from their usage. When you are using a Config map that is derived from the ConfigSpec, you have a pure data-structure that knows (and hides out from your program) where values come from and how to prioritize them.
Hope this helps, if it does, if so, would you be able to rename this ticket as an update to the FAQ with your questions (or even better a Pull Request).
Cheers.


Need styling that fits a theme

The website style should be crisp, clean, and cool.  I'm a programmer at heart, so carefully measured, well laid-out pages is a must.

Closed with commit fdd2dc88c6990e9339512910ee2a0d1616eb31fc.


Add TCP stats to deployment dashboards

This PR adds the TCP metrics added in #785 and #790 to the Grafana deployment dashboards. I've added three new graphs in the "Inbound Traffic" and "Outbound Traffic" headings:

"TCP Connection Closes Per Second": plots the number of opened and accepted connections that have closed successfully or failed.
"Bytes Per Second": plots the number of bytes sent and bytes received on successfully closed TCP connections
"TCP Connection Duration": plots the p50, p95, and p99 durations of closed TCP connections

Here's a screenshot of the new graphs, generated using simulate-proxy output:

(note that the unrealistically high latencies/connection durations are just what simulate-proxy currently generates)
I'm planning on adding similar graphs to other dashboards as well in subsequent PRs.

@olix0r

Ultimately, I think we need to get clearer on exactly what we want to record...

Yeah, I agree completely. I wasn't sure exactly what graphs we ought to have in the dashboard, so this just represents my best guess. Happy to have further discussions on what should change.

(because we don't have read and write events.. we probably want to add them but these could come with a real cost).

I've been wondering about this as well. Maybe, rather than sending an event on every read/write, we might send an occasional "heartbeat" type event so we can get bytes sent/received stats out of the connection with enough resolution to make useful metrics, but without having to handle an event on every read or write. On the other hand, maybe the extra overhead of accounting for whether or not to send an event would be worse than just sending every event?

Also @olix0r, I'm considering making the following changes based on your feedback:

put TCP charts in a collapsed row
rename BYTES PER SECOND to BYTES PER CONNECTION
change CONNECTIONS CLOSED PER SECOND to CONNECTIONS CLOSED and make it display a total rather than a rate
maybe change the TCP CONNECTION DURATION chart to a bar graph type histogram rather than a plot of durations over time?

What do you think?

@olix0r I've made some additional changes based on your feedback. In particular, I've:

put TCP charts in a row that's collapsed by default
renamed BYTES PER SECOND to BYTES PER CONNECTION
changed CONNECTIONS CLOSED PER SECOND to CONNECTIONS CLOSED and made it display as a heatmap (histogram over time) rather than as a line graph

Please let me know what you think of these changes!

@hawkw if it's not inconvenient, can you include a screencap of the new charts (with ~real data ;)?

I just tore down Minikube, so it might take a minute, bit yes.
On Tue, Apr 24, 2018 at 4:41 PM Oliver Gould<EMAIL_ADDRESS>wrote:

@hawkw https://github.com/hawkw if it's not inconvenient, can you
include a screencap of the new charts (with ~real data ;)?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/runconduit/conduit/pull/824#issuecomment-384114234,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACqrsqoG1L2WXbZbchd4191OJZBW1BKxks5tr7gvgaJpZM4TeBEu
.

--
Eliza Weisman
Systems Software Engineer
Buoyant, Inc.

@olix0r

Note that the TCP charts are collapsed by default, so when you open the page it looks like this:

I've added some additional changes based on the changes to metrics @olix0r made in d4d0e579c2dc626ff72f2403505c59efb0fe0379. The dashboard now looks like this:

Hopefully these new charts are much more usable.
Additionally, I've added (collapsed by default) TCP charts to the Conduit Health dashboard as well.


MajoBOT Brazilian Portuguese translation and new commands [WIP]

After thoughts - I decide to finally close this PR
But no worries - the translation will be used at 100% in Majo.exe. This code is not suitable for merging into production.
The translation will be used in Majo when whe start translation process into other languages.
✨ Small translation roadmap
+ Portuguese
+ Polish
+ Russian
- German

I'd like to thank You for the wonderful effort you put into the translation. When the translation process starts (I hope it will start soon) I will put your translation into one file.


Aboobakersiddiqr63/eks docs changes

Notes for Reviewers

Updated EKS Docs

This PR fixes #9082
Signed commits

[ ✓] Yes, I signed my commits.

🚀 Preview for commit 80b1e3bf8de70c188146a226b9c5b8665d81b698 at: https://65283e71530b2c60235553b4--meshery-docs-preview.netlify.app


MONDO:0020347 and MONDO:0007691 [Merge? or fix x-ref?]

Mondo term (ID and Label)
MONDO:0020347 acute inflammatory demyelinating polyradiculoneuropathy
Reason for deprecation
Orphanet:98916 Acute inflammatory demyelinating polyradiculoneuropathy maps 1:1 with OMIM: 139393 GUILLAIN-BARRE SYNDROME, FAMILIAL; GBS
The SNOMED code on MONDO:0007691 'Guillain-Barre syndrome, familial' is SCTID:716723000, PT "Acute inflammatory demyelinating polyradiculoneuropathy"
OMIM 139393 has Preferred name of GUILLAIN-BARRE SYNDROME, FAMILIAL; GBS and Alternative titles; symbols of POLYNEUROPATHY, INFLAMMATORY DEMYELINATING, ACUTE; AIDP

[x] (Also please update the abbreviation on MONDO:0007691 Guillain-Barre syndrome, familial: remove "CIDP" since that refers to the Chronic form, which is an "included" but separate concept on the OMIM entry, as discussed in ticket #2808) https://github.com/monarch-initiative/mondo/issues/2808

Term to be merged with
MONDO:0007691 Guillain-Barre syndrome, familial  ?? Not sure if this is a merge or X-ref fix, see notes below.
Your nano-attribution (ORCID)
If you don't have an ORCID, you can sign up for one here
Additional notes
I imagine that AIDP could present without a genetic predisposition, so maybe this is an X-ref remapping rather than merge, but at a minimum, the Orphanet X-refs suggest it should be mapped with the OMIM record in Mondo.  The mapping of NCIT entries in UMLS seems to be at odds with NCI's own mapping of the concept to UMLS, so I cannot figure out from those hierarchies which concept goes where...
C116926 and C26790 both map to UMLS's Acute Infective Polyneuritis, UMLS CUI: C3542501 (depending on whether you are looking at the NCIT browser or UMLS browser).
UMLS has split the OMIM records across multiple terms, Guillain-Barre Syndrome, Familial C4083008 and Acute Inflammatory Demyelinating Polyneuropathy C4551910 are the ones relevant to this ticket.  So maybe it's just a matter of cleaning up Orphanet 98916 mapping in Mondo (or asking if Orpha wants to review their entry for AIDP)?

Adding to Feb release to review this ticket and decide the best course of action. If we decide to merge, we'll need to add an obs tag to it and leave it open for two months.

Orphanet:98916 Acute inflammatory demyelinating polyradiculoneuropathy maps 1:1 with OMIM: 139393 GUILLAIN-BARRE SYNDROME, FAMILIAL; GBS

Orphanet:98916 does not have a 1:1 mapping with OMIM:139393. It is BTNT (ORPHA code's Broader Term maps to a Narrower Term).

I don't think these two terms should be merged, they seem like different classes to me.

I will close this when I merge the PR but please feel free to reopen if more discussion is needed, thanks!


Cannot register tunnel

While doing some refactoring in a branch, I finally got it to a running state but now I have this error. I checked out the original branch (master) just to be sure, and the error still appears (I added a single print line to show the error message):
$ go build && ./cloudflare-warp --hostname my.hostname --hello-world --no-autoupdate
INFO[0000] Proxying tunnel requests to https://<IP_ADDRESS>:53287
INFO[0000] Starting Hello World server at <IP_ADDRESS>:53287
INFO[0000] Starting metrics server                       addr="<IP_ADDRESS>:53288"
ERRO[0000] handle return: received return for unknown question id=0  subsystem=rpc-transport
INFO[0000] Connected to LAX
pogs: insert @0xc793e50592935b4a: can't insert into pogs.RegistrationOptions: pogs.RegistrationOptions has unknown field PoolName, maps to poolName
ERRO[0000] Cannot register
INFO[0000] Retrying in 1s seconds

Does this have something to do with an inconsistency between the client and the backend maybe? I actually have no idea.

Commenting out the fields PoolName and IsAutoupdated in pogs.RegistrationOptions struct resolves the problem. I just don't comprehend at what cost. :) It's working now, with that change. Will leave open for advisement.

Hi mholt, thanks for reporting the issue! We will tunnelrpc.capnp.go.

tunnelrpc.capnp.go has been updated. You shouldn't need to comment out PoolName and IsAutoupdated in pogs.RegistrationOptions anymore.


Allow rounded rectangles with rectangle node

Currently it is possible to create a rectangle by inserting a rectangle node in the node graph ("Node Graph" button in top right corner then right click in graph and select "Rectangle"). However it is not possible to make the corners round.
An option for the border radius should be added to the properties (on the right when the rectangle node is selected) that allows for creating a rounded rectangle. Note that it is not possible to create a perfect arc using bézier curves, so an approximation will need to be used (like for the circle currently).
Here is the a screenshot of the current rectangle node for reference:

Hi, I'd like to be assigned this issue.

Hey is it done? cause i want to get assigned

@devharsh2k4 this is already in progress.


Write a Java program to print all prime numbers between two numbers

Description
Write a Java program to print all prime numbers between two numbers
Input  : 10 20
Output : 11 13 17 19

How to contribute

Save the solution in PrintAllPrimeNumbersBetweenTwoNumbers.java file
Add PrintAllPrimeNumbersBetweenTwoNumbers.java file in print-all-prime-numbers-between-two-numbers folder

Can you assign this to me, I will we happy to work on this issue.

!assign


Dockerfile: Implement least privileged user

So it's not run as root, it runs as a "headlamp" user.
It's a docker security best practice according to these guides:

https://snyk.io/blog/10-docker-image-security-best-practices/
https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md

Also, make the container workflow run when Dockerfile is changed.
How to use
It should run as before.
Testing done

[x] CI passes e2e test
[x] works locally on minikube

@illume , is the discussion over? Do these changes need to be updated?
Also, it'd be nice if these changes were tested with podman too.

@illume , is the discussion over? Do these changes need to be updated?
Also, it'd be nice if these changes were tested with podman too.

No, the conversation is still unresolved. Last week I tried following the tutorial, but got lost. Since there's no tests for using mounts that way for plugins (or using podman either)... I think it's best some tests for those are added to this PR.
I've updated the todo list, and put this into draft mode.

@illume if it's okay, I can look into this?

Hi @knrt10 That would be great, thank you :)

I don't know why values.yaml changes are not applying in the CI, it's working fine when I install locally using helm. Let me debug more.
Also regarding CI on running charts test, I think it should build dockerfile and test against it. Since if any changes to dockerfile is made it still pulls latest headlamp-image. What are your thoughts folks?
cc @headlamp-k8s/devs

Failing charts test is unrelated, once we merge this PR and new docker image is pushed. I think they would pass. Since e2e tests are passing.

Failing charts test is unrelated, once we merge this PR and new docker image is pushed. I think they would pass. Since e2e tests are passing.

@yolossn can you please have a look at what to do about the charts failing in here?

@yolossn can you please have a look at what to do about the "Lint and Test Charts" CI job failing in here?

This is because the Helm test pulls the latest docker image from ghcr and the changes added in the Dockerfile is not reflecting. I discussed this with @knrt10  He is changing the testing logic to use the local docker image.

Great work :) Thanks @knrt10 (and also thanks @yolossn )


Unexpected error from cudaGetDeviceCount() error

System Info
I met a trtllm-build issue.
GPU: RTX 3090
I followed official script of the below steps.

I ran the below code after installing nvidia container toolkit.

docker run --rm --runtime=nvidia --gpus all --entrypoint /bin/bash -it nvidia/cuda:12.4.0-devel-ubuntu22.04

Then, I installed python, pip, and tensorrt_llm libraries using the below codes.

# Install dependencies, TensorRT-LLM requires Python 3.10
apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev git git-lfs

# Install the latest preview version (corresponding to the main branch) of TensorRT-LLM.
# If you want to install the stable version (corresponding to the release branch), please
# remove the `--pre` option.
pip3 install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com

# Check installation
python3 -c "import tensorrt_llm"

At this point, torch.cuda.is_available() does not work and showed the below error log despite nvidia-smi working.
/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:118: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
False

Then, I converted phi3 model to TensorRT-LLM chekcpoint format as follows.

# Get the source code required for running the example
git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM
git lfs install

# Log in to huggingface-cli
# You can get your token from huggingface.co/settings/token
huggingface-cli login --token *****

# Convert the model into TensorRT-LLM checkpoint format
cd examples/phi
export MODEL_TYPE="Phi-3-mini-128k-instruct"
pip install -r requirements.txt
python3 ./convert_checkpoint.py --model_type ${MODEL_TYPE} --model_dir "microsoft/${MODEL_TYPE}"  --output_dir ./phi-checkpoint  --dtype float16

Model was converted successfully.

But when I was compiling the model using the below codes, cuda error was occured.

# Compile model
trtllm-build --checkpoint_dir phi-checkpoint/ --gemm_plugin float16 --output_dir ./phi3-engine

[TensorRT-LLM] TensorRT-LLM version: 0.11.0.dev2024060400
[06/11/2024-01:01:19] [TRT-LLM] [I] Set bert_attention_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set gpt_attention_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set gemm_plugin to float16.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set gemm_swiglu_plugin to None.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set nccl_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set lookup_plugin to None.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set lora_plugin to None.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set moe_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set mamba_conv1d_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set context_fmha to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set context_fmha_fp32_acc to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set paged_kv_cache to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set remove_input_padding to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set use_custom_all_reduce to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set multi_block_mode to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set enable_xqa to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set attention_qk_half_accumulation to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set tokens_per_block to 64.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set use_paged_context_fmha to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set use_fp8_context_fmha to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set multiple_profiles to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set paged_state to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set streamingllm to False.
[06/11/2024-01:01:19] [TRT-LLM] [W] remove_input_padding is enabled, while max_num_tokens is not set, setting to max_batch_size*max_input_len.
It may not be optimal to set max_num_tokens=max_batch_size*max_input_len when remove_input_padding is enabled, because the number of packed input tokens are very likely to be smaller, we strongly recommend to set max_num_tokens according to your workloads.
Traceback (most recent call last):
  File "/usr/local/bin/trtllm-build", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 441, in main
    cluster_config = infer_cluster_config()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/auto_parallel/cluster_info.py", line 531, in infer_cluster_config
    device_name = torch.cuda.get_device_name(torch.cuda.current_device())
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 778, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 293, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW

Who can help?
No response
Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

docker run --rm --runtime=nvidia --gpus all --entrypoint /bin/bash -it nvidia/cuda:12.4.0-devel-ubuntu22.04

apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev git git-lfs

# Install the latest preview version (corresponding to the main branch) of TensorRT-LLM.
# If you want to install the stable version (corresponding to the release branch), please
# remove the `--pre` option.
pip3 install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com

# Check installation
python3 -c "import tensorrt_llm"

# Get the source code required for running the example
git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM
git lfs install

# Log in to huggingface-cli
# You can get your token from huggingface.co/settings/token
huggingface-cli login --token *****

# Convert the model into TensorRT-LLM checkpoint format
cd examples/phi
export MODEL_TYPE="Phi-3-mini-128k-instruct"
pip install -r requirements.txt
python3 ./convert_checkpoint.py --model_type ${MODEL_TYPE} --model_dir "microsoft/${MODEL_TYPE}"  --output_dir ./phi-checkpoint  --dtype float16

trtllm-build --checkpoint_dir phi-checkpoint/ --gemm_plugin float16 --output_dir ./phi3-engine

Expected behavior
Build sucessfully
actual behavior
[TensorRT-LLM] TensorRT-LLM version: 0.11.0.dev2024060400
[06/11/2024-01:01:19] [TRT-LLM] [I] Set bert_attention_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set gpt_attention_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set gemm_plugin to float16.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set gemm_swiglu_plugin to None.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set nccl_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set lookup_plugin to None.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set lora_plugin to None.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set moe_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set mamba_conv1d_plugin to auto.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set context_fmha to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set context_fmha_fp32_acc to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set paged_kv_cache to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set remove_input_padding to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set use_custom_all_reduce to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set multi_block_mode to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set enable_xqa to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set attention_qk_half_accumulation to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set tokens_per_block to 64.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set use_paged_context_fmha to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set use_fp8_context_fmha to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set multiple_profiles to False.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set paged_state to True.
[06/11/2024-01:01:19] [TRT-LLM] [I] Set streamingllm to False.
[06/11/2024-01:01:19] [TRT-LLM] [W] remove_input_padding is enabled, while max_num_tokens is not set, setting to max_batch_size*max_input_len.
It may not be optimal to set max_num_tokens=max_batch_size*max_input_len when remove_input_padding is enabled, because the number of packed input tokens are very likely to be smaller, we strongly recommend to set max_num_tokens according to your workloads.
Traceback (most recent call last):
  File "/usr/local/bin/trtllm-build", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 441, in main
    cluster_config = infer_cluster_config()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/auto_parallel/cluster_info.py", line 531, in infer_cluster_config
    device_name = torch.cuda.get_device_name(torch.cuda.current_device())
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 778, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 293, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW

additional notes

nvidia-smi works in nvidia-cuda docker environment

This can be resolved by installing nvidia driver version 500 or higher.


GEODE-8250: Create new custom log config acceptance tests

New tests to discover and verify that use of custom log configs works
when using GFSH to start Locator and Server.

@jinmeiliao I'm going to request a re-review. I'm not able to remove my manipulation of TemporaryFolder and I don't want to spend more time trying to debug why GfshRule doesn't stop its processes.


Instructions on creating my own dataset?

Are there any instructions on creating my own dataset and starting to train from scratch (rather than loading pretrained model). Also do you think the image to image translation (Style transfer) would work well in medical images? I'm trying to use histopathological images (H&E) images.
Also, like others have said, when would the color translation model be released? I'm really looking forward to applying this.
I would appreciate a response. Thank you!

May I ask if you have used this model to train your own dataset of medical images, I am also working on this.

@xiayhh Unfortunately, I haven't tried yet, still hoping that the authors of the repository @suxuann @chenlin9 could release instructions and also the color translation model...

Hello, I am also doing image to image conversion, I would like to ask if you found the method of ddib, I found some problems in the SyntheticModel framework


Comment why log eviction isn't implemented via a log field.

This comment explains the logging of alternate eviction which went into master (10.0.0) and 9.2.x. Thus this comment applies to the 9.2.x and 10.0.0 releases and no other releases.


about fail node resume

I have two rack(rack1 rack2) and each have six node,each node have one redis service on backend.
my client set & read data with phpredis extension.
test case 1:
rack1 & rack2 all up, random set & get data to all node, all is ok,no problem.
test case 2:
rack1 up , rack 2 some node down, backend redis up,random set & get data all rack1 node(avoid all rack 2 node),still no problem.
but rack1 and rack2 node data is diffent.
teste case 3:
rack1 up,resume rack 2 down node,random get data to all node, some test case 2 set data is missing.
my question：
fail node resume, rack1 and rack2 node data is diffent,How can avoid to get some key is missing?
my config
node 1:
dyn_o_mite:
datacenter: default_dc
dyn_listen: <IP_ADDRESS>:8101
dyn_port: 8101
dyn_seed_provider: simple_provider
dyn_seeds:

<IP_ADDRESS>:8101:rc0:default_dc:3346660200
<IP_ADDRESS>:8101:rc1:default_dc:1656170642
<IP_ADDRESS>:8101:rc0:default_dc:278415870
<IP_ADDRESS>:8101:rc1:default_dc:2691937952
<IP_ADDRESS>:8101:rc1:default_dc:2565460974
gos_interval: '500'
listen: <IP_ADDRESS>:6379
rack: rc0
redis: 'true'
servers:
<IP_ADDRESS>:6380:1
timeout: 150000
tokens: '400517191'

node 2:
dyn_o_mite:
datacenter: default_dc
dyn_listen: <IP_ADDRESS>:8101
dyn_port: 8101
dyn_seed_provider: simple_provider
dyn_seeds:

<IP_ADDRESS>:8101:rc0:default_dc:400517191
<IP_ADDRESS>:8101:rc1:default_dc:1656170642
<IP_ADDRESS>:8101:rc0:default_dc:278415870
<IP_ADDRESS>:8101:rc1:default_dc:2691937952
<IP_ADDRESS>:8101:rc1:default_dc:2565460974
gos_interval: '500'
listen: <IP_ADDRESS>:6379
rack: rc0
redis: 'true'
servers:
<IP_ADDRESS>:6380:1
timeout: 150000
tokens: '3346660200'

.....
node 6
dyn_o_mite:
datacenter: default_dc
dyn_listen: <IP_ADDRESS>:8101
dyn_port: 8101
dyn_seed_provider: simple_provider
dyn_seeds:

<IP_ADDRESS>:8101:rc0:default_dc:400517191
<IP_ADDRESS>:8101:rc0:default_dc:3346660200
<IP_ADDRESS>:8101:rc1:default_dc:1656170642
<IP_ADDRESS>:8101:rc0:default_dc:278415870
<IP_ADDRESS>:8101:rc1:default_dc:2691937952
gos_interval: '500'
listen: <IP_ADDRESS>:6379
rack: rc1
redis: 'true'
servers:
<IP_ADDRESS>:6380:1
timeout: 150000
tokens: '2565460974'

The reason that you have multi-rack deployment is for failover. Evidently, there will be cases that the two racks may be out of sync. If you do not have a lot of writes,  you can stop the writes on that database, and quickly sync up the nodes using the persistence feature in Redis: redis-cli bgrewriteaof. We are also working on a scalable approach.


Refactor/styles

Pull Request Test Coverage Report for Build 15

7 of 7 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 7:
0.0%

Covered Lines:
21

Relevant Lines:
21

💛  - Coveralls

Pull Request Test Coverage Report for Build 15

7 of 7 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 7:
0.0%

Covered Lines:
21

Relevant Lines:
21

💛  - Coveralls

I think reduce function has ugly syntax.
const state = items.reduce(() => {
	/*
	...
	*/
}, initialState);

When we read this code, we see initialState after reducer definition. I don't like it.

@Lisss13 ?


Trust Issue - revived

This question has popped up twice now ( #54 and #60 ) and both times there was no answer concerning the trust certificate issues. I guess you have used a selfsigned certificate to sign the AddIn.
Does that mean we can not trust this AddIn?
Please respond to this standing issue.
Thank you.

In the properties of the vsto file one can change the trust of the file but then one gets
Name:
From: file:///C:/Users/bernardt/AppData/Local/Temp/Temp1_outlook-matters-1.3.0.zip/OutlookMatters.vsto
************** Exception Text **************
System.Deployment.Application.InvalidDeploymentException: Deployment and application do not have matching security zones.
at Microsoft.VisualStudio.Tools.Applications.Deployment.ClickOnceAddInDeploymentManager.GetManifests(TimeSpan timeout)
at Microsoft.VisualStudio.Tools.Applications.Deployment.ClickOnceAddInDeploymentManager.InstallAddIn()

To get rid of the "Deployment and application do not have matching security zones" error you need unblock all the files you unpacked one by one (in the right click -> properties dialog). Not only the .vsto file. After that it will be installed correctly.


Travis: Add gcc 4.4 and 4.5

I'm not opposed to this, but I'm curious if there was a specific motivation here. Did you run into problems with a particular GCC version or do you think reptyr is particularly likely to run into such problems?

It's something I tried while trying to track down #67, #68, and #69.
-Werror in reptyr's CFLAGS does make it prone to build errors with different compiler versions (and other stuff in CFLAGS, like demonstrated by #68). Maybe the Travis builds should in addition to different compilers be build both without any -O (like at the moment) as well as with e.g. -O2?

My suspicion is that 4.4 and 4.5 aren't disparate enough to be likely to surface bugs, and that a lot of your problems are related to older kernel headers, which unfortunately aren't easy to test via Travis.
I'd be happy to take a change to optionally disable -Werror in the Makefile, if that would make your life easier in that regard.

Well, there's not much difference in say make WERROR= and sed -i -e 's/ -Werror//' Makefile, especially since I'm dealing with the Fedora/EPEL reptyr package builds which means I won't be typing that often at all. But it would be cleaner if the CFLAGS I set anyway would be appended to the reptyr defaults instead of the other way around; then I could do make CFLAGS="$whatever_i_do_anyway -Wno-error", maybe something like:
--- a/Makefile
+++ b/Makefile
@@ -1,4 +1,5 @@
-override CFLAGS+=-Wall -Werror -D_GNU_SOURCE -g
+ALL_CFLAGS := -Wall -Werror -D_GNU_SOURCE -g $(CFLAGS)
+override CFLAGS = $(ALL_CFLAGS)

How do you feel about that fix I just pushed?

Seems fine, thanks.
On Wed, Jan 27, 2016 at 8:10 AM, Nelson Elhage<EMAIL_ADDRESS>wrote:

How do you feel about that fix I just pushed?
—
Reply to this email directly or view it on GitHub
https://github.com/nelhage/reptyr/pull/66#issuecomment-175422061.


Complete Export

Is your feature request related to a problem? Please describe.
Sometimes clients do not want to give long term access to their org, this makes it difficult to review data after the fact.
Describe the solution you'd like
It would be wonderful if it could be possible to have a export button which would save all of the existing reports into a single file (or zip file)
Describe alternatives you've considered
Saving each object individually, etc is an option, but not very efficient.
Additional context
N/A

That may be available in the next version, where Org Check will be available as a SFDX plugin.
:D


Update our fork with Code Base

we need to update our fork with the last updated the repo has right now, they fixed some issues we have in the app, and we don't need to handle by our own.

Sorry,

I'm going to close this PR and create a new one with different name and fixing all the conflicts


scrap_to_json() returns error

Hi,
unfortunately I am getting following error:
[WDM] - Driver [/Users/dino/.wdm/drivers/geckodriver/macos/v0.34.0/geckodriver] found in cache
new layout loaded
2024-06-14 15:55:50,841 - facebook_page_scraper.driver_utilities - ERROR - Error at close_modern_layout_signup_modal: Message: Element  is not clickable at point (892,121) because another element  obscures it
Stacktrace:
RemoteError@chrome://remote/content/shared/RemoteError.jsm:12:1
WebDriverError@chrome://remote/content/shared/webdriver/Errors.jsm:192:5
ElementClickInterceptedError@chrome://remote/content/shared/webdriver/Errors.jsm:291:5
webdriverClickElement@chrome://remote/content/marionette/interaction.js:166:11
interaction.clickElement@chrome://remote/content/marionette/interaction.js:125:11
clickElement@chrome://remote/content/marionette/actors/MarionetteCommandsChild.jsm:204:29
receiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.jsm:92:31
Traceback (most recent call last):
File "/Users/dino/Codings/python/FacebookRSSInformer/.venv/lib/python3.11/site-packages/facebook_page_scraper/driver_utilities.py", line 74, in __close_modern_layout_signup_modal
close_button.click()
File "/Users/dino/Codings/python/FacebookRSSInformer/.venv/lib/python3.11/site-packages/selenium/webdriver/remote/webelement.py", line 81, in click
self._execute(Command.CLICK_ELEMENT)
File "/Users/dino/Codings/python/FacebookRSSInformer/.venv/lib/python3.11/site-packages/selenium/webdriver/remote/webelement.py", line 710, in _execute
return self._parent.execute(command, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dino/Codings/python/FacebookRSSInformer/.venv/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
self.error_handler.check_response(response)
File "/Users/dino/Codings/python/FacebookRSSInformer/.venv/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message: Element  is not clickable at point (892,121) because another element  obscures it
Stacktrace:
RemoteError@chrome://remote/content/shared/RemoteError.jsm:12:1
WebDriverError@chrome://remote/content/shared/webdriver/Errors.jsm:192:5
ElementClickInterceptedError@chrome://remote/content/shared/webdriver/Errors.jsm:291:5
webdriverClickElement@chrome://remote/content/marionette/interaction.js:166:11
interaction.clickElement@chrome://remote/content/marionette/interaction.js:125:11
clickElement@chrome://remote/content/marionette/actors/MarionetteCommandsChild.jsm:204:29
receiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.jsm:92:31
all_posts length: 3
no post_url, skipping
no post_url, skipping
no post_url, skipping
all_posts length: 3
all_posts length: 7

Same here

I think facebook have changed their layout. I have managed to fix this error, but I am running into many more. Below are fixes that I have applied:
`def scrap_to_json(self, minimum_timestamp = None):
# call the __start_driver and override class member __driver to webdriver's instance
self.__start_driver()
starting_time = time.time()
# navigate to URL
self.__driver.get(self.URL)
# only login if username is provided
Finder._Finder__accept_cookies(self.__driver)
self.username is not None and Finder._Finder__login(self.__driver, self.username, self.password)
    self.__layout = Finder._Finder__detect_ui(self.__driver)
    # sometimes we get popup that says "your request couldn't be processed", however
    # posts are loading in background if popup is closed, so call this method in case if it pops up.
    Utilities._Utilities__close_error_popup(self.__driver)
    # wait for post to load
    elements_have_loaded = Utilities._Utilities__wait_for_element_to_appear(
        self.__driver, self.__layout, self.timeout)
    # scroll down to bottom most
    Utilities._Utilities__scroll_down(self.__driver, self.__layout)
    self.__handle_popup(self.__layout)
    # timestamp limitation for scraping posts
    timestamp_edge_hit = False
    while (not timestamp_edge_hit) and (len(self.__data_dict) < self.posts_count) and elements_have_loaded:
        self.__handle_popup(self.__layout)
        # self.__find_elements(name)
        timestamp_edge_hit = self.__find_elements(minimum_timestamp)
        current_time = time.time()
        if self.__check_timeout(starting_time, current_time) is True:
            logger.setLevel(logging.INFO)
            logger.info('Timeout...')
            break
        Utilities._Utilities__scroll_down(
            self.__driver, self.__layout)  # scroll down
    # close the browser window after job is done.
    Utilities._Utilities__close_driver(self.__driver)
    # dict trimming, might happen that we find more posts than it was asked, so just trim it
    self.__data_dict = dict(list(self.__data_dict.items())[
                            0:int(self.posts_count)])

    return json.dumps(self.__data_dict, ensure_ascii=False)`

Accept cookies before try to login
`def __accept_cookies(driver):
try:
# Use JavaScript to find the button containing the text "Allow all cookies"
buttons = driver.execute_script("""
return Array.from(document.querySelectorAll('div[role="none"] span'))
.filter(span => span.textContent.includes('Allow all cookies'));
""")
        # Check if any elements were found
        if buttons:
            ActionChains(driver).move_to_element(buttons[-1]).click().perform()  # Click the last one if multiple are found
        else:
            logger.info("No 'Allow all cookies' button found.")
    except NoSuchElementException:
        logger.info("No such element exception occurred.")
    except IndexError:
        logger.info("Index error occurred.")
    except Exception as ex:
        logger.exception("Error at accept_cookies: {}".format(ex))
        sys.exit(1)`

Change cookie selector
`def __login(driver, username, password):
try:
        wait = WebDriverWait(driver, 4)  # considering that the elements might load a bit slow

        # NOTE this closes the login modal pop-up if you choose to not login above
        try:
            element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '[aria-label="Close"]')))
            element.click()  # Click the element
        except Exception as ex:
            logger.debug(f"no pop-up")

        time.sleep(1)
        #target username
        username_element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='email']")))
        password_element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='pass']")))

        #enter username and password
        username_element.clear()
        username_element.send_keys(str(username))
        password_element.clear()
        password_element.send_keys(str(password))

        #target the login button and click it
        try:
            # Try to click the first button of type 'submit'
            WebDriverWait(driver, 2).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[role='button'][aria-label='Accessible login button']"))).click()
        except TimeoutException:
            # If the button of type 'submit' is not found within 2 seconds, click the first 'button' found
            WebDriverWait(driver, 2).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button"))).click()
    except (NoSuchElementException, IndexError):
        pass
    except Exception as ex:
        logger.exception("Error at login: {}".format(ex))
        # sys.exit(1)`

Change Login selector


Using Github App for check-repos workflow

Signed-off-by: Vacha Shah<EMAIL_ADDRESS>Description
Coming from https://github.com/opensearch-project/project-meta/pull/38#issuecomment-1055669522, using Github App to trigger CI workflows on generated PRs. Also adding an option for signoff on the auto-generated PR.
Issues Resolved
[List any issues this PR will resolve]
Check List

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dblock Can you review this when you get a chance? I am not able to add reviewers to this PR. Thank you!

@VachaShah looks like the workflow is failing.
I merged the changes, but could you take a look?


add a setting to restrict which languages get downloaded

We currently store several partial translations in our OneSky project that we don't want to download. This PR adds an optional setting that allows the user to specify which languages to download.

bumping this to see if there is interest before I spend the effort to publish an internal fork for usage in our projects


Cookie length should be less then or equal to 4096 characters

I am using the userAgentApplication.loginRedirect(config.b2cScopes) function to authenticate the users for my app.
Occasionally when I login, I receive the following message:
Set-Cookie header is ignored in response from url: https://login.microsoftonline.com/xxxxxxx.onmicrosoft.com/login. Cookie length should be less then or equal to 4096 characters.
This prevents msal.js from authenticating the user until the browser is closed or the cookies are cleared.

My organization and our clients are regularly seeing this issue (or something similar).  Error message is "Bad Request - Request Too Long.  HTTP Error 400. The size of the request headers is too long."  A number of posts across the web (GitHub and Stack Overflow) all suggest that cookies need to be cleared because too many MSFT websites are causing a "cookie overflow".  While this may be true, asking our customers to clear their cookies isn't a realistic expectation to impose on them long term just because we decided to use AAD B2C for our apps.

@RobARichardson : thanks for the feedback. Do you think it's a cookie issue? or is it because the query parameters on the URL is too long (which also happens with B2C) ?
For the moment proposing to close this issue as this is not an issue with msal.js, but with the B2C requests (I added [Feedback] to the title, BTW). Have communicated the feedback to the B2C team.

@jmprieur  We are developing an app that lives in Azure and uses AAD for authentication. All five of us (developers) see the exact same error message that cookies exceed 4096.
We cannot just clear all cookies all the time, as we don't only work in this project (think about Google account, Microsoft, Stack Overflow, any app that uses cookies really).
Isn't there anything we can do?

@navyasric can you please check with the MSAL.JS team what @CamiloTerevinto can do?


fix: Decrease MyC artwork high demand icon size

Addresses CX-3077
Description
Decrease MyC artwork high demand icon size.

Before
After

nice


Unit test to deploy, execute, and undeploy CWL DAG

Purpose

Create unit tests to test the deployment, execution, and un-deployment of this CWL DAG.
Execute with the default parameters: CWL workflow and argument of {"message": "Hello Unity"}.

Proposed Changes

[ADD] Unit tests: deploy, execute, status, and undeploy for CWL DAG.

Issues

#131 - Create test to deploy, execute and undeploy the CWL DAG

Testing

Local unit tests executed as expected:

collected 16 items

unity-test/test_main.py::test_get_landing_page PASSED                          [  6%]
unity-test/test_main.py::test_get_health PASSED                                [ 12%]
unity-test/test_main.py::test_get_conformance_declaration PASSED               [ 18%]
unity-test/test_main.py::test_post_deploy_process PASSED                       [ 25%]
unity-test/test_main.py::test_delete_undeploy_process PASSED                   [ 31%]
unity-test/test_main.py::test_post_execute_process PASSED                      [ 37%]
unity-test/test_main.py::test_delete_dismiss_execution PASSED                  [ 43%]
unity-test/test_main.py::test_get_process_description PASSED                   [ 50%]
unity-test/test_main.py::test_get_process_list PASSED                          [ 56%]
unity-test/test_main.py::test_get_job_list PASSED                              [ 62%]
unity-test/test_main.py::test_get_status PASSED                                [ 68%]
unity-test/test_main.py::test_get_results PASSED                               [ 75%]
unity-test/test_main.py::test_post_deploy_process_cwl_dag PASSED               [ 81%]
unity-test/test_main.py::test_post_execute_process_cwl_dag PASSED              [ 87%]
unity-test/test_main.py::test_get_status_cwl_dag PASSED                        [ 93%]
unity-test/test_main.py::test_delete_undeploy_cwl_dag PASSED                   [100%]

================================== warnings summary ==================================
../../../environments/../lib/python3.12/site-packages/httpx/_models.py:1
  /../../../../../../lib/python3.12/site-packages/
    httpx/_models.py:1: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
    import cgi
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== 16 passed, 1 warning in 1.60s ============================

Unit tests passed via GitHub Action. Results located here.

@drewm-jpl - This is a great idea. I am thinking that I can parameterize the four unit tests that I created to except different CWL/DAGs and point to where applicable the different execution request and process description JSON files alongside the expected process ID.

Do you think you could parameterize the existing functions, so that we can exercise all of them with a set of CWLs/DAGs?

test_post_deploy_process
test_delete_undeploy_process
test_post_execute_process
test_delete_dismiss_execution
test_get_process_description
test_get_process_list
test_get_status
test_get_results

The parameterization could look something like this:
# Parametrize for different process descriptions
@pytest.mark.parametrize("process_filename", [
    "test_data/process_descriptions/EchoProcess.json",
    "test_data/process_descriptions/AnotherProcess.json"
])
@pytest.fixture(scope="module")
def deploy_process(test_directory, client, process_filename):

Or even more generically like this to avoid hardcoding filenames:
# Use glob to get all files in the process descriptions directory
process_files = glob.glob("test_data/process_descriptions/*.json")

@pytest.mark.parametrize("process_filename", process_files)

@drewm-jpl and @LucaCinquini - I really liked Drew's idea of using glob to discover all of the files in either the process_descriptions and execution_requests directories and then parametrize the unit tests using those files. I did need to implement a little bit of structure for this to work:

JSON files must be named after the process ID which is located in the process description JSON files.
Execution requests and process description JSON files must be named the same but organized into their respective directories: execution_requests and process_descriptions.
To add a new CWL DAG you will need to create a process description file with the same name as the id property and then create an execution request file to describe the expected inputs and outputs with the same name as the process file then the unit tests will automatically discover and test that new CWL DAG.

I also feel like we may want to move the four unit tests that I created into more of an integration test set up since they rely on each other to a certain extent and follow the expected order of CWL DAG deployment, execution, and un-deployment but I don't think that is necessary for now.
Let me know if you have any feedback and/or if this is ready to be merged.

@LucaCinquini - I believe that I added the workflow_dispatch trigger and it should take effect when this is merged into main. At least I think that is how it works but let me know if I need to do anything else.
@drewm-jpl - Great suggestion, I felt like things were getting redundant so I removed the three tests you recommended.

@nikki-t please go ahead and merge, great progress.


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/114864

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/117053

Triggering TagBot for merged registry pull request
This extra notification is being sent because I expected a tag to exist by now, but it doesn't.
You may want to check your TagBot configuration to ensure that it's running, and if it is, check the logs to make sure that there are no errors.

Triggering TagBot for merged registry pull request
This extra notification is being sent because I expected a tag to exist by now, but it doesn't.
You may want to check your TagBot configuration to ensure that it's running, and if it is, check the logs to make sure that there are no errors.

Triggering TagBot for merged registry pull request
This extra notification is being sent because I expected a tag to exist by now, but it doesn't.
You may want to check your TagBot configuration to ensure that it's running, and if it is, check the logs to make sure that there are no errors.

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/120875


StackExchange.Redis.ConfigurationOptions exists in 2 libraries

Not sure if I'm just setting this up incorrectly, but I used nuget to install the IdentityServer4.Contrib.RedisStore (v1.0.0).  It looks like it has dependencies on StackExchange.Redis (1.2.6) and StackExchange.Redis.StrongName (1.2.4).  I tried to new up a ConfigurationOptions object in the Startup.cs and I get an error that ConfigurationOptions exists in both of the StackExchange libraries.  I even pulled down the StackExchange.Redis (1.26) and updated StackExchange.Redis.StrongName to 1.26 via nuget and still get the same error.
From what I remember the IdentityServer3 contrib library didn't have a dependency on the StrongName library.  If I then try to remove the StrongName nuget package it just reverts to the Contrib dependency of 1.2.4.
I'm not sure what version of IdentityServer4 this project was compiled for.  I'm running near latest (2.0.5) which uses Asp.Net Core 2.

Hi,
I tried to replicate the issue and I was able to see the same error message.
This library was created while Identity server was supporting .Net core 1.1 only, with that, it relies on StackExchange.Redis library(v 1.2.6).
While Microsoft.AspNetCore.All package includes Microsoft.Extensions.Caching.Redis, which in turns relies on StackExchange.Redis.StrongName (v1.2.4)
So from here, the conflict is happening, I will update the library to rely on StackExchange.Redis.StrongName (v1.2.4), which is still compatible with .Net core 1.1, and Identity Server 4 (v 1.0).
With that I think you will be able to use this library in Asp.Net core 2 and Latest Identity Server version.

@gilm0079 a release is deployed to fix your issue, please let me know if it not fix your problem.

@AliBazzi your update fixed the build error for me. Thanks.
Would you be able to increment up the assembly version and publish a new version to nuget?  Right now I just have your project from here added to my solution.
Not related to this problem. I was going to setup the StackExchange.Redis.ConfigurationOptions object to be used in the startup.cs.  Do you know what properties are important to get the connection setup?  The structure is a little different than how it worked in IdentityServer3 with the ConnectionMultiplexer.  I'm assuming at a minimum I need to put a host name, port, and prefix for the redis keys in the ConfigurationObject?

Sorry @gilm0079
my mistake, maybe I didn't clarify enough what I meant by release deployed; the Nuget Package is update to 1.0.1 since yesterday, no need to add the project as reference.
for the other question, you still have an overload that allows you to use connection string style initialization, you can use that, for what to use while you initializing ConfigurationOptions object, you can refer to StackExchange.Redis docs.
Please let me know if I can close the issue now, thanks.

Thanks. My VS2017 must have just been not syncing with nuget. I wasn't finding 1.0.1 yesterday now it finds it today.
Just for notes for others. It looks like if you want to use the ConfigurationOptions class to setup the operational store the syntax is:
StackExchange.Redis.ConfigurationOptions redisOptions = StackExchange.Redis.ConfigurationOptions.Parse("<redisServerIp>:<port>");
it can then be used on the IIdentityServerBuilder object as
.AddOperationalStore(redisOptions)
Additional properties on the redisOptions object can be appended before this step.


Maximum policy size of 10240 bytes exceeded for role

Describe the bug
When we run amplify push on our project, we get an error Maximum policy size of 10240 bytes exceeded for role mspbeLambdaRole53878999-staging (Service: AmazonIdentityManagement; Status Code: 409; Error Code: LimitExceeded.
We have 51 types in our schema and 27 lambda triggers, this error came up when we wanted to push the 27th lambda function.
Amplify CLI Version
4.20.0
Expected behavior
Push to the cloud without any hiccups.
Desktop (please complete the following information):

OS: Windows and Mac
Node Version. 12.14.0

@varunpsr We have a way off minifying CF templates can you try using amplify push --minify-stack-template. Let me know if this helps

Closing as a duplicate of #3815


Repeated logs in plugins

We are seeing repeated logs for the plugins, might be a good first issue to clean the up. Also need to investigate why dropreason's Generate is being called twice implied from the logs

ah, we are invoking Generate twice for each plugin, once in Reconcile

and another one in Compile for packetparser and dropreason


Unescape character function does not return correct value

If the data from OLAP source has & in it, SOAP message has it correctly as & But rowset from xmla4js shows undefined in place of &.
By looking at source code, the function for unescape characters, correctly identifies the value, but does not return it and hence end result has undefined.

What line do you see this on @ajkaushik ?

I can't reproduce this. I have cubes with & in captions, and it works for me. Please provide me sample Xml/A response so I may test?

It’s possible I could have figured this out. Probably safe to close this issue.
Chad Hollman

On Apr 21, 2018, at 16:59, Roland<EMAIL_ADDRESS>wrote:
I can't reproduce this. I have cubes with & in captions, and it works for me. Please provide me sample Xml/A response so I may test?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi Chad! Thanks for the reply - much appreciated! I will close the issue, but feel free to re-open if it pops up.


Delete package-lock.json

Please delete https://github.com/theowenyoung/obsidian-template-gatsby-theme-primer-wiki/blob/main/.layouts/package-lock.json. It causes the template to build with old dependencies and incorrect inbound and outbound references to be generated.

You can see that [[https://demo-obsidian.owenyoung.com/How to/Embed files/]]. There are no backlinks generated. //cc @theowenyoung


feat: new chat protocol extension points

Purpose
Update chat protocol data points structure to match Python sample's updated implementation, following their support of GPT4-V: https://github.com/Azure-Samples/azure-search-openai-demo/commit/36015895f14c8569e3a3989fcf5744a9b5bd4845
Does this introduce a breaking change?

[x] Yes
[ ] No

Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

@anfibiacreativa I just redeployed the branch from codespaces, everything works fine on my side.
Could this be merged so we have compatibility with Python backend for today's stream?


Dark mode toggle

Added dark mode toggle button. Linked to issue #53

amazing work, I changed some stuff in the summary component to make it consistent! Thanks a lot!
done in #84


Update Composer dependencies (2023-03-11-00-09)

Visual regression test passed!
View the visual regression test report


.NET 6 launch

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.924%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
2109

Relevant Lines:
2927

💛  - Coveralls


<EMAIL_ADDRESS>hash check failed

Checking hash of CheatEngine70.exe ... ERROR Hash check failed!
App:         extras/cheat-engine
URL:         https://github.com/cheat-engine/cheat-engine/releases/download/7.0/CheatEngine70.exe
First bytes: 4D 5A 50 00 02 00 00 00
Expected:    ff73dfcc885d4e9f2978d15cd03124d760b0c23f0ad9fd1de4e62a87df01e314
Actual:      932705995da1091f5465b5ef8a41fb764184816efe125354a3a28275ef9cb528

Can confirm.
Updating 'cheat-engine' (6.8.3 -> 7.0)
Downloading new version
Downloading https://github.com/cheat-engine/cheat-engine/releases/download/7.0/CheatEngine70.exe (17.0 MB)...
Checking hash of CheatEngine70.exe ... ERROR Hash check failed!
App:         extras/cheat-engine
URL:         https://github.com/cheat-engine/cheat-engine/releases/download/7.0/CheatEngine70.exe
First bytes: 4D 5A 50 00 02 00 00 00
Expected:    ff73dfcc885d4e9f2978d15cd03124d760b0c23f0ad9fd1de4e62a87df01e314
Actual:      932705995da1091f5465b5ef8a41fb764184816efe125354a3a28275ef9cb528

Please try again or create a new issue by using the following link and paste your console output:
https://github.com/lukesampson/scoop-extras/issues/new?title=cheat-engine%407.0%3a+hash+check+failed


Support directly designating a gitSSHKey instead of File for launcher

What this PR does / why we need it:
Add --git-ssh-key-data for launcher.  This flag enables passing a sshKey without preparing a file.
Usecase:

You can use the same launcher command for AWS and GCP (+ etc.)
If you use AWS Secrets Manager to store the sshKey, Secrets Manager will write it to an env (e.g. YOUR_ENV) and you can read it by --git-ssh-key-data $YOUR_ENV

Which issue(s) this PR fixes:
N/A
Does this PR introduce a user-facing change?: N/A

How are users affected by this change:
Is this breaking change:
How to migrate (if breaking change):

@Warashi
Sorry, would you please review it again?
I changed from --git-ssh-key-env to --git-ssh-key-data since it's easier to use.


I can't input ikj when the lua-snip visible when I mapping ikj for moving

for my neovim keymaps, I perfer to map 'i-j-k-l' for moving cursor
buf I found that it doesn't work for luasni
sorry for my poor English, and I make a video for explain what I meaning, But I get 'Something went really wrong, and we can't process that file' about that, I don't know how to fixed that ......

Luasnip.zip
I can' upload this video directly....

Could you upload another video, I can't find the issue in the first one (it seems a bit broken for me :/)
It would be great if you could convert the video to mp4 (or make it possible to watch in github in some other way), that makes it easier to watch it.
But besides that, are you sure this issue originates with Luasnip?

Sorry about that, I have found the result of that!
It's just because I do some mappings with 'v' mode

like 'i' mapped  to 'k'

and the 'v' mode means the 'x' and 's' modes
I guess you guys set select mode so that we can press any key to change the visual select section,  so I change some mappings then it works well!

Ahh of course :facepalm:
Glad you figured it out :+1:


[MINOR] Fix issue template name

What changes were proposed in this pull request?

When I wanted to create an issue, I found that only the feature-request template was available

According to the information, it is found that the templates (bug-report and custom) has the same name

So it was renamed based on https://github.com/TuGraph-family/tugraph-analytics/commit/f69aa453120890ae42acf487d739f6602f2ce1a7
How was this PR tested?

[ ] Tests have Added for the changes
[ ] Production environment verified

Thanks @liuxiaocs7  for correct me. It will merged soon as the ci has passed.


Segmentation fault (core dumped)

DTIAtlasFiberAnalyzer crashes when computing some cases, I'm not sure if the problem is from the the program or maybe something wrong with the case itself.
**Command Line :  /BAND/USERS/skp78-dti/DTIAtlasFiberAnalyzer-1.8.1/bin/dtitractstat --input_fiber_file /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/8188_time3_DWI_s013_QCed/8188_time3_DWI_s013_QCed_CC_3seg_body_extracted_clean_processed.vtk --ouput_stats_file /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/8188_time3_DWI_s013_QCed/8188_time3_DWI_s013_QCed_CC_3seg_body_extracted_clean_rd.fvp --remove_clean_fiber --auto_plane_origin median --parameter_list rd --step_size 1
Unable to open the input fiber file...exit without results!
End of Dti Tract Stat.
Work done : 3
Child finished
Threads are executed
Applydti_tract_stat_mt finished
Segmentation fault (core dumped)**

Please do you mind in the terminal checking du - hsc /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/* CC_3seg_body_extracted_cleaprocessed
So we may compare what files exist, what file sizes are in comparison to other cases, what files appear to be incomplete or unusual. It is possible that some cases did not register properly or completely.
A good rule of thumb and the ideal researchers’ approach to any imaging study is to visually QC registration at every step of the process. You can load every subjects’ registered vtk file up into a program like Slicer with the subject’s original input DTI file to visually quality control the registration of fiber file into the subjects’ anatomical space. You’d be surprised sometimes. This step lets you know the final numbers you’re getting are real so you can be confident in what you’re studying.
I suggest opening 3D Slicer GUI, and seeing if you can open the original subject image .. not registered to the atlas, just the original input image  Likely /ASD/Autism2/IBIS2/SchoolAge_
Load into slicer your original subject input image which I’m guessing is levels up somewhere in here in an original DTI folder  here’s the path: /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/
8188_time3_DWI_s013_QCed_DTI.nrrd or something. Make sure it’s the original image. Not registered to the atlas.
Load into slicer and set as a VTK fiber file the following path:
/ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/8188_time3_DWI_s013_QCed/8188_time3_DWI_s013_QCed_CC_3seg_body_extracted_clean_processed.vtk
This will let you 1. Test that the input to DTI tract stat fiber file exists. 2. Test that it is in the same space as the original subject image (where it should be) 3. Let you test that the fiber has RD information included within the image or within the vtk file if you.
Sometimes the computer runs out of memory or you lose internet access to the server and some fibers are incompletely processed. The trick is to throw those out and run the tool again, skipping over the files that you’ve already created and just running the tool on the files you had to remove.
If you have thousands of cases or you’re short in time, you can use the du - hsc * trick.  The files that are unusually small sizes compared to the same parameter file for the same tract in every case can be removed and rerun as a likely fix. (so check that CC Seg RD vtk should be roughly be the same size in case 8188 as all other cases...if it’s much smaller, it must be incompletely processed. Remove just the vtk for that case and let the tool try again to create it with fiberprocess then see if dtitractstat fails.  I can’t remember the language in the GUI or the most current command line flag, but their is a setting to tell the tool to skip over existing files when you rerun DTIFiberAnalyzer. Let me know if you’d like me to find that detail for you.
I hope this helps.

On Jan 5, 2021, at 12:44 PM, KhalidwAlAli<EMAIL_ADDRESS>wrote:

DTIAtlasFiberAnalyzer crashes when computing some cases, I'm not sure if the problem is from the the program or maybe something wrong with the case itself.
Command Line : /BAND/USERS/skp78-dti/DTIAtlasFiberAnalyzer-1.8.1/bin/dtitractstat --input_fiber_file /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/8188_time3_DWI_s013_QCed/8188_time3_DWI_s013_QCed_CC_3seg_body_extracted_clean_processed.vtk --ouput_stats_file /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/8188_time3_DWI_s013_QCed/8188_time3_DWI_s013_QCed_CC_3seg_body_extracted_clean_rd.fvp --remove_clean_fiber --auto_plane_origin median --parameter_list rd --step_size 1
Unable to open the input fiber file...exit without results!
End of Dti Tract Stat.
Work done : 3
Child finished
Threads are executed
Applydti_tract_stat_mt finished
Segmentation fault (core dumped)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Please do you mind in the terminal checking du - hsc /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/* CC_3seg_body_extracted_cleaprocessed
So we may compare what files exist, what file sizes are in comparison to other cases, what files appear to be incomplete or unusual. It is possible that some cases did not register properly or completely.
A good rule of thumb and the ideal researchers’ approach to any imaging study is to visually QC registration at every step of the process. You can load every subjects’ registered vtk file up into a program like Slicer with the subject’s original input DTI file to visually quality control the registration of fiber file into the subjects’ anatomical space. You’d be surprised sometimes. This step lets you know the final numbers you’re getting are real so you can be confident in what you’re studying.
I suggest opening 3D Slicer GUI, and seeing if you can open the original subject image .. not registered to the atlas, just the original input image  Likely /ASD/Autism2/IBIS2/SchoolAge_
Load into slicer your original subject input image which I’m guessing is levels up somewhere in here in an original DTI folder  here’s the path: /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/
8188_time3_DWI_s013_QCed_DTI.nrrd or something. Make sure it’s the original image. Not registered to the atlas.
Load into slicer and set as a VTK fiber file the following path:
/ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/8188_time3_DWI_s013_QCed/8188_time3_DWI_s013_QCed_CC_3seg_body_extracted_clean_processed.vtk
This will let you 1. Test that the input to DTI tract stat fiber file exists. 2. Test that it is in the same space as the original subject image (where it should be) 3. Let you test that the fiber has RD information included within the image or within the vtk file if you.
Sometimes the computer runs out of memory or you lose internet access to the server and some fibers are incompletely processed. The trick is to throw those out and run the tool again, skipping over the files that you’ve already created and just running the tool on the files you had to remove.
If you have thousands of cases or you’re short in time, you can use the du - hsc * trick.  The files that are unusually small sizes compared to the same parameter file for the same tract in every case can be removed and rerun as a likely fix. (so check that CC Seg RD vtk should be roughly be the same size in case 8188 as all other cases...if it’s much smaller, it must be incompletely processed. Remove just the vtk for that case and let the tool try again to create it with fiberprocess then see if dtitractstat fails.  I can’t remember the language in the GUI or the most current command line flag, but their is a setting to tell the tool to skip over existing files when you rerun DTIFiberAnalyzer. Let me know if you’d like me to find that detail for you.
I hope this helps.

On Jan 5, 2021, at 12:44 PM, KhalidwAlAli<EMAIL_ADDRESS>wrote:

DTIAtlasFiberAnalyzer crashes when computing some cases, I'm not sure if the problem is from the the program or maybe something wrong with the case itself.
Command Line : /BAND/USERS/skp78-dti/DTIAtlasFiberAnalyzer-1.8.1/bin/dtitractstat --input_fiber_file /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/8188_time3_DWI_s013_QCed/8188_time3_DWI_s013_QCed_CC_3seg_body_extracted_clean_processed.vtk --ouput_stats_file /ASD/Autism2/IBIS2/SchoolAge_Projects/DTI_Processing/Active_SchAgeDTI/7_DTIAFA/CC_3seg_body_extracted_KA/Cases/8188_time3_DWI_s013_QCed/8188_time3_DWI_s013_QCed_CC_3seg_body_extracted_clean_rd.fvp --remove_clean_fiber --auto_plane_origin median --parameter_list rd --step_size 1
Unable to open the input fiber file...exit without results!
End of Dti Tract Stat.
Work done : 3
Child finished
Threads are executed
Applydti_tract_stat_mt finished
Segmentation fault (core dumped)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

I think you need to check if your original fiber file is valid. Seems like fiber file is not valid for the input.

I think you need to check if your original fiber file is valid. Seems like fiber file is not valid for the input.


⚠ ClamAV has degraded performance

In 377412b, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 67 ms

Resolved: ClamAV performance has improved in f6d8927.


docs: issues while following the getting started on a apple silion mac

I followed the getting started guide for potpie development, I noticed that it does not successfully start running on my mac m2 pro. To start off I created a service account in a firebase account and installed it locally. A few packages weren't getting installed locally, got the following error when running pip install -r requirements.txt
pip install -r requirements.txt                                                                       ─╯
Collecting uvicorn (from -r requirements.txt (line 2))
  Using cached uvicorn-0.32.1-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: sqlalchemy in /Users/vibhavbobade/miniconda3/lib/python3.12/site-packages (from -r requirements.txt (line 3)) (2.0.36)
Requirement already satisfied: alembic in /Users/vibhavbobade/miniconda3/lib/python3.12/site-packages (from -r requirements.txt (line 4)) (1.14.0)
Collecting gunicorn (from -r requirements.txt (line 5))
  Using cached gunicorn-23.0.0-py3-none-any.whl.metadata (4.4 kB)
...
ERROR: Ignored the following versions that require a different python version: 0.0.1 Requires-Python <3.12,>=3.10; 0.0.10 Requires-Python <3.12,>=3.9; 0.0.10a0 Requires-Python <3.12,>=3.9; 0.0.11 Requires-Python <3.12,>=3.9; 0.0.12 Requires-Python <3.12,>=3.9; 0.0.13 Requires-Python <3.12,>=3.9; 0.0.14 Requires-Python <3.12,>=3.9; 0.0.15 Requires-Python <3.12,>=3.9; 0.0.16 Requires-Python <3.12,>=3.9; 0.0.17 Requires-Python <3.12,>=3.9; 0.0.18 Requires-Python <3.12,>=3.9; 0.0.19 Requires-Python <3.12,>=3.9; ...
ERROR: Could not find a version that satisfies the requirement blar-graph==1.0.11 (from versions: none)
ERROR: No matching distribution found for blar-graph==1.0.11

Based on the command above I understand that I have the wrong version of python installed which is right, I have
python --version                                                                                ─╯
Python 3.12.4

a  < 3.12 version installed.
The docs is also missing dev dependency on albemic which is used for database migrations.
Maybe, support for devcontainer could help here

Thanks @waveywaves , will add this in the documentation


Fixed "Go to stitch x/y" block in Kente Computing and Navajo Weaving sims

Fixed blocks in order to accurately represent the 0.5 shift in y-values when the x-value is odd numbered. This makes the "Go to stitch x", "Go to stitch y" and "Go to stitch x, y" blocks go to the accurate spot in the stage (even when scaling the costume size).

Just BTW, it's now on the community site, though you will have to clear your cache to see it.


fix: minor css fixed along with small UI issue

https://issues.redhat.com/browse/DFMS-161
https://issues.redhat.com/browse/DFMS-162
https://issues.redhat.com/browse/DFMS-163
https://issues.redhat.com/browse/DFMS-164
https://issues.redhat.com/browse/DFMS-181

plz add all the screenshots...

/approve
/lgtm


macro elasticity run not completing

http://www.ospc.org/dynamic/macro_results/381/

@brendancol @PeterDSteinberg

just polls forever http://www.ospc.org/dynamic/macro_results/381/

There are two bugs here:

the worker node was incorrectly calling code in dropq
the webapp was not properly reading the FAIL status of the jobs, which results in endless polling

I have fixed both. PRs coming up.


feat: add @stdlib/iter/cartesian-product

Resolves #1335
Description

What is the purpose of this pull request?

This pull request:

Adds implementation of iterator equivalent of cartesian-product

Related Issues

Does this pull request have any related issues?

#1335
Questions

Any questions for reviewers of this pull request?

No.
Other

Any other information relevant to this pull request? This may include screenshots, references, and/or implementation notes.

TESTS

BENCHMARKS

EXAMPLES

Checklist

Please ensure the following tasks are completed before submitting this pull request.

[x] Read, understood, and followed the contributing guidelines.

@stdlib-js/reviewers

@shubhexists hey, I cannot seem to run benchmark.js on the feat I added (@stdlib/utils/parse-ndjson). I'm getting this error. Did you encounter it too?

Hi @steff456 , my idea of using linspace and zeroTo was that the iterCartesianProduct function takes in an Array instead of an Iterator, as mentioned in the Issue.
Since, randu returns an Iterator object, I would eventually have to run a while loop to collect all it's values into an Array.
What are your opinions?

@shubhexists you are right, let's leave the examples with linspace

@steff456 Kindly have a look now


S3: Pre-signed Download URLs

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue, please leave a comment

Tell us about your request
Presigned urls for AWS S3 Upload
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Generate a pre-signed URL for an Amazon S3 objects to give users temporary access to objects.
Additional context
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html
https://github.com/awslabs/aws-sdk-rust/issues/139

Tracking this in #139 since the pre-sign functionality will be implemented generically across all services and operations.