Performance with Breakout

Have you trained Breakout with your a3c by any chance?  I wonder that kind of scores you have gotten.
John

It's not as good as DeepMind's implementation.
After several hours of training it gets reward around 300 and stops there.

Have you tried the rmsprop optimizer with shared parameters (this is what the authors use) instead of adam?

No, but it should be relatively easy to try.

On this issue, are you aware of this discussion (https://github.com/dennybritz/reinforcement-learning/issues/30) ?
[It's on dqn / tensorflow performance issue but the guess is that a3c tensorflow's performance issue has the same causes ]
Here cgel suggests the following makes the difference -
Important stuff:
Normalise input [0,1]
Clip rewards [0,1]
don't tf.reduce_mean the losses in the batch. Use tf.reduce_max
initialise properly the network with xavier init
use the optimizer that the paper uses. It is not same RMSProp as in tf
Not really sure how important:
They count steps differently. If action repeat is 4 then they count 4 steps for action. So divide all pertinent hyper-parameters by 4.
Little difference (at least in breakout):
pass terminal flag when life is lost
gym vs alewrap. Learning rate is different but If one works so will the other
Among important stuff, what are incorporated into your code ?

Everything except the optimizer. But I posted a link to the same one as in DM's paper in the description of the repo.

Wonderful.  Thank you.
Just one more question.  In terms of params settings, are they same as this
https://github.com/muupan/async-rl/wiki ?

No, I decided to use parameters from the open ai starter agent.

Not yet, I may try in the future. In my experience, for a fixed model and a fixed dataset meta optimizer tends to overfit. However, it's probably not a problem for atari.

@ikostrikov thanks for the heads up on how the meta-optimiser performs !!!
Regarding models, this looks promising, XNOR-Net. As far as I know it's not been ported over to PyTorch yet, (they released it in Torch last year) ??? But if I do convert it, I'll let you know if I get it working with with your meta-optimiser, and how it performs.

@dylanthomas
Update:
After 10h with 16 threads it achieves reward of >400.

@ikostrikov   Super !!

A3C and Breakout:
How did you get reward > 400? (using the same code or did you make some changes?)
I want to run some code and get > 400 rewards, what should I do?
Regards

The same code.
10 hours with 16 threads on xeon 2650 v4.

Thank you
I will clone the code again from:
https://github.com/dennybritz/reinforcement-learning/tree/master/PolicyGradient/a3c
and try .... correct?
This should replicate DeepMind's results, correct?

The numbers are for my code, not from that repo.
I'm not sure whether it's physically possible to replicate DeepMind's results.

I am very sorry for asking many questions ...
How many frames did you process in 10 hours?
Is this your code? https://github.com/ikostrikov/pytorch-a3c
Do you have any clue why is repo do not work as expected?! (rewards are around 30 to 35 for Breakout?!)

I didn't count the number of frames.
Yes.
Which one?
Because A3C is extremely sensitive to hyper parameters. And DeepMind ran massive grid search to find the best hyper parameters. Then in evaluation they run 50 trials with fixed hyper parameters for each game and average top 5 performances. It's rather difficult to replicate that.

Thank you!
I mean, this  (dennybritz/reinforcement-learning#30)
It seems something is wrong with DQN implementation and A3C too ....
Performance is only around 30 to 35 for Breakout ... any clue why?

@dylanthomas , I find the current repo does not learn as expected.  Did you make it work?
Time 00h 08m 56s, episode reward -2.0, episode length 106
Time 00h 09m 58s, episode reward -2.0, episode length 111
Time 00h 11m 04s, episode reward -2.0, episode length 113
Time 00h 12m 08s, episode reward -2.0, episode length 104
Time 00h 13m 13s, episode reward -2.0, episode length 111
Time 00h 14m 17s, episode reward -2.0, episode length 107
Time 00h 15m 22s, episode reward -2.0, episode length 110
Time 00h 16m 26s, episode reward -2.0, episode length 105
Time 00h 17m 31s, episode reward -2.0, episode length 104
Time 00h 18m 37s, episode reward -3.0, episode length 156
Time 00h 19m 44s, episode reward -3.0, episode length 156
Time 00h 21m 13s, episode reward -21.0, episode length 764
Time 00h 22m 43s, episode reward -21.0, episode length 764
Time 00h 24m 07s, episode reward -21.0, episode length 764
Time 00h 25m 15s, episode reward -4.0, episode length 179
Time 00h 26m 44s, episode reward -21.0, episode length 764
Time 00h 28m 02s, episode reward -11.0, episode length 425
Time 00h 29m 36s, episode reward -21.0, episode length 764
Time 00h 31m 29s, episode reward -21.0, episode length 1324
Time 00h 32m 58s, episode reward -21.0, episode length 764
Time 00h 34m 30s, episode reward -21.0, episode length 764
Time 00h 36m 01s, episode reward -21.0, episode length 764
Time 00h 37m 30s, episode reward -21.0, episode length 764
Time 00h 39m 32s, episode reward -21.0, episode length 1324

How many threads did you use?

@ikostrikov  Thanks for your quick reply.
I am using 8 threads

Did you run with OMP_NUM_THREADS=1 ?

@ikostrikov  Wow, maybe that's the reason. I didn't notice this. Why is this important?

Otherwise it uses multiple cores for OMP within a thread.

Do you think it is necessary to add:
model.zero_gradient()

at https://github.com/ikostrikov/pytorch-a3c/blob/master/train.py#L108 ?

@ikostrikov  Thanks!, I will try it again~

optimizer.zero_gradient() already zeros the gradients.

@ikostrikov  but it only zeros the gradient of the shared_model right?

https://github.com/ikostrikov/pytorch-a3c/blob/master/train.py#L18 They share gradients within this thread.

@ikostrikov  many thanks!
But I still do not get it why using multiple cores for OMP within a thread will be a problem. Won't that make the algorithm faster?

It will make some threads sequential and you will effectively collect less data.

@ikostrikov  I see, do you mean that all the threads are very likely to process the same frame, thus useless?

I think it happens for many reasons. Just try to run it this way :)

I see, this really bothers me in the last two days.
Do you think this issue is with pytorch or  general python code?
Cause I saw many tensorflow implementations does not pose this constraint.

I think it's just the way multiprocessing is organized in PyTorch. I think the author of PyTorch have a better answer. I also found this thing surprising.

Updates, it starts learning this time :D  You saved my day. Thank you!
@apaszke Is it true that OMP_NUM_THREADS=1 is a necessary thing to run pytorch code?
Thanks!

@ikostrikov  When doing the optimizer.step(), I notice that there is no mutex to protect the shared_model weights, do you think it is safe?

In DM's paper they say they perform async updates without locks.

Oh I see, thanks!

Will the following code restrain the shard_model grad only bound with one local_model?
Cause share_model.grad will not be None after running the following function.
def ensure_shared_grads(model, shared_model):
    for param, shared_param in zip(model.parameters(), shared_model.parameters()):
        if shared_param.grad is not None:
            return
        shared_param._grad = param.grad


Reading API Definition from ConfigMap support

Description:
Changes will include:

Operator reading definition file referred in API CR
Sending it to Mgt server and calling back office internal service

This is implemented in https://github.com/wso2/apk/pull/509

https://github.com/wso2/apk/pull/509 merged


Literature not correctly using write-mode

Describe the bug
The write-mode config is being ignored by the literature step for the output W2VModel.
Observed behaviour
When re-running the ETL with already existing outputs and literature output already generated the ETL fails because it's trying to write to an already existing output. This is because we're not writing a DF but the outputs of an AI model.
Expected behaviour
When re running the ETL with an already generated W2VModel the output should be ignored if the write-mode set to ignore and overwritten if set to overwritte
To Reproduce
Steps to reproduce the behaviour:

Run the ETL and wait for it to be over
Run the ETL with the write-mode set to ignore
You should see the error when the ETL tries to write the output for W2VModel

this got added in the latest release I'm closing this item


About loading the checkpoint file for generative task

Hi,
After I tried preprocessing and fine-tuning, I found that two checkpoint files were generated, one is called 'mp_rank_00_model_states.pt' and the other one is called 'zero_pp_rank_0_mp_rank_00_optim_states.pt'.  It seems that both of them could be used for the following generative task as the checkpoint_path, and give me similar generative results. It would be highly appreciated if more details or explanations could be given.
Many thanks in advance!
Song

Dear Song,
I'm happy to hear that you successfully ran our project. When you open the model directory, you will find a zero_to_fp32.py script. This script extracts fp32 consolidated weights from zero stage 2 and 3 DeepSpeed checkpoints. You can run this script, which will automatically generate a model file according to your zero stage during finetuning. For example,
> python zero_to_fp32.py "../model" "global_step0/0model.pkl"

Furthermore, when you use multiple GPUs to finetune the model, you will get multiple checkpoint files starting with zero_pp_rank, the number of which is the same as the number of GPUs. At this point, if you want to further use the model for inference or other tasks, you need to use zero_to_fp32.py to merge these files into a usable single file. The details involve model saving in DeepSpeed.  You can refer to https://github.com/microsoft/DeepSpeed for more information.
Best,
Yin Fang


Create a About Us Page

🚀 Feature
Need to create an about us page following the theme of the website.

I want to work on this issue.

Assigned @anushruti11 , go ahead.


Fix internal state machine of action node

The implicit state machine in the action node was not implemented correctly.
This PR implements the state machine explicitly and fixes the invalid transitions in the previous one.

Codecov Report
Attention: Patch coverage is 24.44444% with 68 lines in your changes are missing coverage. Please review.

Project coverage is 33.02%. Comparing base (17e31b8) to head (c30152f).

Files
Patch %
Lines

ros_bt_py/src/ros_bt_py/ros_nodes/action.py
24.44%
68 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##              dev      #63      +/-   ##
==========================================
+ Coverage   32.87%   33.02%   +0.15%
==========================================
  Files          37       37
  Lines        5415     5453      +38
  Branches     1250     1251       +1
==========================================
+ Hits         1780     1801      +21
- Misses       3567     3585      +18
+ Partials       68       67       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


enhance resource deletion flow.

This PR enhance the resource deletion flow by:

Soft delete the resource record when the request reach to Maestro Server
Publish delete cloud event to Maestro agent by setting deletionTimeStamp
Agent delete the resource from cluster and publish status update with ManifestsDeleted condition
Maestro check the ManifestsDeleted condition in the status update and delete the resource record permanently.

/assign @clyang82

fixed: https://github.com/openshift-online/maestro/issues/101


Update bootstrap.sh

ifupdown package is necessary to add private_network to allow two or more vagrant VMs to communicate on their own network

Thank you!


Add ./bootstrap.sh file, refactor Makefiles

Note
I'm going to add information about this in the README but at the minute I'm just pushing up the code so that it's visible.  Anything looks odd, let me know.

The "get up and going" process is (should be) much easier now.
The ./bootstrap.sh file installs all the things we need to install once (pyenv, mongodb, etc), and then the Makefiles have all the commands we need to run multiple times.

The /python/Makefile runs everything without tripping over virtualenv problems now.
The /api/Makefile looks for an NRCAN_ENGINE_API_KEY to be set or else it won't run.

Also, I've tested this on my Macbook the best that I can. In theory, it should work on everyone's computers as well, but I haven't been able to confirm that it does.

Closing this until I work out what do do about mongo in the Makefile (ie, right now it relies on homebrew).


Web Support ?

Hello i like the package it's helpful can you add support for web
Thanks

Already web supported https://simformsolutionspvtltd.github.io/flutter_chatview/
This package just some widgets nothing special to support on web platform

@Baghdady92 as shared by @Saifallak Web support is already available, so I'm closing this issue for now. Please feel free to reopen it if you need any further information.


Consistency of return types

Spun off from an http4s issue.

char returns a Parser1[Unit]. We know what it captured, so we don't return it.
charIn returns a Parser1[Char]. It captured one of a set of characters, and we want to know which.
ignoreChar returns a Parser1[Unit]. Arguably like charIn in that we don't know what it cpatured, but the docs tell us to call .string if we need the result.
string1 returns a Parser[Unit].  Seems consistent with .char.

Did we get this right, or should everything return what it captured?

A few notes:

Returning the captured String from string is free. We don't have to allocate (we know what we expect already).
Returning the string in ignoreCase is not free, we need to allocate a substring if we are capturing. If we aren't capturing there is an additional Boolean check we don't have now, probably not a big deal.
Returning the Char in char is cheap since we have the char, there is likely a boxing of the char but I assume the JVM has a cache of those so we are hitting a lookup table that might even be in the CPU cache.

To summarize, I doubt this is a perfect question to add the return values and since we have using *> and void as recommended style I guess it doesn't do much harm either.
On the other hand, matching a constant string and wanting that exact thing back is rare IME. Usually you combine it a bit more then capture a larger block with string or you use .as to change it. So I haven't felt the need.
I think we are consistent now (except arguably the ignoreCase variants maybe should return what they capture), so I think the discussion would be helped if we had some examples where one did want to capture directly the results of these parsers.

As I get deeper into this, I'm feeling more comfortable with the way it is.  A mild discomfort: in the spirit of "return Unit if constant, return the value when variable", this rule from RFC5234 should return a Char. To do so would require a couple annoying .ases, because it's composed of two Unit rules.
  val wsp: Parser1[Unit] =
    sp.orElse1(htab)

I made a judgment call that which kind of whitespace is found is generally uninteresting, and left it as Unit.


⚠ ClamAV has degraded performance

In 57ebd9e, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 95 ms

Resolved: ClamAV performance has improved in a994e60.


split peewee-async.py

split peewee-async.py to peewee-async dir
asyncpg requires Python 3.5

Although I also prefer to have the project in separate files as it becomes more manageable in me eyes it was recently declined in #72


Shikiji support

Adding support for shikiji (currently shiki) will help users to use dual themes with expressive code
https://github.com/antfu/shikiji#lightdark-dual-themes

Hello Akash, thank you for this feature request!
I love the approach that Shikiji uses to support multiple themes. As Expressive Code is doing all the rendering itself already, it wouldn't benefit from using Shikiji as-is, but I'll use it as inspiration to add similar efficient multi-theme rendering.


Clear handling tracked events of Http2Stream

We faced some crashes on canceling events of Http2Stream on 8.1.x branch.
crash 1
(gdb) bt
#0  0x00007f70fd4d2199 in waitpid () from /lib64/libpthread.so.0
#1  0x000055f06daffe74 in crash_logger_invoke(int, siginfo_t*, void*) (signo=11, info=0x7f70f250aa70, ctx=0x7f70f250a940) at traffic_server/Crash.cc:165
#2  <signal handler called>
#3  0x00007f70d6a13a00 in ?? ()
#4  0x000055f06dbc791c in clear_inactive_timer (this=0x7f68464d68a0) at Http2Stream.cc:878
#5  clear_timers (this=0x7f68464d68a0) at Http2Stream.cc:895
#6  Http2Stream::initiating_close() (this=0x7f68464d68a0) at Http2Stream.cc:436
#7  0x000055f06dbbd972 in Http2ConnectionState::delete_stream(Http2Stream*) (this=0x7f49eab7d7d0, stream=0x7f68464d68a0) at Http2ConnectionState.cc:1371
#8  0x000055f06dbbea72 in Http2ConnectionState::send_data_frames(Http2Stream*) (this=0x7f49eab7d7d0, stream=0x7f68464d68a0) at Http2ConnectionState.cc:1580 (<- stream state: closed)
...
(gdb) frame 4
#4  0x000055f06dbc791c in clear_inactive_timer (this=0x7f68464d68a0) at Http2Stream.cc:878
878         inactive_event->cancel();

crash 2
(gdb) bt
#0  0x00007f10ca1f4199 in waitpid () from /lib64/libpthread.so.0
#1  0x000055950ad4fe74 in crash_logger_invoke(int, siginfo_t, void) (signo=11, info=0x7f10bfb099b0, ctx=0x7f10bfb09880) at traffic_server/Crash.cc:165
#2  <signal handler called>
#3  0x00007f0f88544c60 in ?? ()
#4  0x000055950ae179b2 in clear_io_events (this=0x7f0e297a5670) at Http2Stream.cc:917
#5  Http2Stream::initiating_close() (this=0x7f0e297a5670) at Http2Stream.cc:437
#6  0x000055950ae0d972 in Http2ConnectionState::delete_stream(Http2Stream) (this=0x7f0fd51062d0, stream=0x7f0e297a5670) at Http2ConnectionState.cc:1371
#7  0x000055950ae0ea72 in Http2ConnectionState::send_data_frames(Http2Stream) (this=0x7f0fd51062d0, stream=0x7f0e297a5670) at Http2ConnectionState.cc:1580
...
(gdb) frame 4
#4  0x000055950ae179b2 in clear_io_events (this=0x7f0e297a5670) at Http2Stream.cc:917
917         this->_read_vio_event->cancel();

Some tracked events are not set nullptr when Http2Stream handles them. This leads Http2Stream to cancel invalid events on shutdown.
We did some refactorings for these tracked events, so we need to modify this patch for 9.0.x, 8.1.x, and 8.0.x branch.

[approve ci autest]

@zwoop Although we can cherry-pick this without conflicts, I'll open PRs to backport this.
Because 9.0.x, 8.1.x & 8.0.x has members need to be set nullptr which are removed on the master branch.

@zwoop Please ignore previous my comments. I misunderstood about inactive_event.
For crash 1 (inactive_event->cancel()), this approach doesn't work. I'll figure out in another PR.
For crash 2 (_read_vio_event->cancel()), this PR works for all branches. Let's just cherry-pick this.

Cherry-picked to 8.1.x
Cherry-picked to v9.0.x branch.


Using openssl doesn't work on a Mac, and is overkill on Linux

Hi!  Thanks for this training example; it's really helpful!
I found that the Makefile line which sorts the list of training examples breaks on a Mac, presumably because it doesn't have named pipes.  But it is also somewhat of an overkill: sort -R --random-source only reads in 16 bytes from the random source to initialise the sorting, so sort -R is probably adequate here.  shuf reads in much more randomness than this, using the random-source for every step of the permutation, but it will fail on a Mac as the shuf command doesn't exist there.
Best wishes,
Julian

Maybe I should reactivate my Python script to sort and shuffle the data. Obviously that would not only be much faster but also improve the portability. @wrznr, what do you think?

that would not only be much faster but also improve the portability.

+1

@juliangilbey, did you see my report in the Wiki? Maybe it also helps. And of course you are invited to share your results there, too.

Thanks - that's a nice report!
I wouldn't go to the effort of using python; I'd just replace
find $(GROUND_TRUTH_DIR) -name '*.lstmf' | sort | \
	  sort -R --random-source=<(openssl enc -aes-256-ctr -pass pass:"$(RANDOM_SEED)" -nosalt </dev/zero 2>/dev/null) > "$@"

(and I just noticed: there's an extra unnecessary sort in that pipeline!) with
find $(GROUND_TRUTH_DIR) -name '*.lstmf' | sort -R > "$@"

The only difference between this and the original version is that sort -R reads from /dev/urandom if it can, and if not, it does something else reasonably intelligent.  The original version uses the --random-source to get a 16-byte seed, and then uses some form of md5 hashing to generate later randomness.  So it will still be at least as good as the Python version.  (And MacOS supports sort -R.)
Best wishes,
Julian

Just a quick clarification: sort -R reads its seed from /dev/urandom if it can, and if not, it does something else reasonably intelligent to get a random seed.

We want reproducible training results. Therefore we can only use pseudo random sequences. Using /dev/urandom is therefore not possible.

Ah, I missed that!
If all you then want is a reproducible result, you could have a 16+ byte file called seed.txt (eg "this is the reproducible seed file; do not modify"), and call --random-source=seed.txt.  That would work without the hassle of calling openssl or python.  You could even use one of the files already present for that, but its content might change at some point.
Best wishes,
Julian

Does that work for Linux, too? I had a similar solution before, but with less than 16 bytes because I did not know that.

I would assume so, as both MacOS and Linux use GNU sort (from coreutils) as their implementation of /usr/bin/sort.

I am pretty sure that the BSD version of coreutils (which is the default on MacOS) does not offer -R.

After a little research, I fear that we have to go with Python if we want to be portable. shuf and sort -R are not available on MacOS (you have to install coreutils via brew or macports). Python is always present. I think the PR proposed by @stweil is sufficient.

I think that it may depend on what version of MacOS you are running; macOS Mojave (macOS 10.14) allows sort -R --random-source, and that's the BSD version of sort.  (I've just checked that on a fresh install of macOS, without brew or macports.)  Of course, the Python solution works too, but it assumes that python3 is installed, which it is not by default on a Mac :-(

If we are talking about portability, we should consider MacOS versions which are still officially supported. Concerning Python3, cf. https://github.com/tesseract-ocr/tesstrain/pull/86#discussion_r320618926

@juliangilbey, Python 3 is required anyway for generating the line boxes, so that is not a new dependency.

True, true!

PR #86 is merged now, so this issue should be fixed.


Inconsistencies between Get-ChildItem and Remove-Item with respect to -Recurse and -Include

Update: Based on @StingyJack's feedback, part of this issue may be related to a longstanding known Remove-Item problem currently documented as:

When it is used with the Include parameter, the Recurse parameter might not delete all subfolders or all child items. This is a known issue.

Get-ChildItem allows the following idioms to look for matching files on all levels of the target directory's hierarchy:
Get-ChildItem *.txt -Recurse
Get-ChildItem .  -Recurse -Include *.txt

By contrast, these idioms currently do not work with Remove-Item:
Steps to reproduce
Set-Location ~

Get-ChildItem *.txt -Recurse
Get-ChildItem . -Recurse -Include *.txt

Remove-Item *.txt -Recurse -WhatIf
Remove-Item . -Recurse -Include *.txt -WhatIf

Expected behavior
All 4 commands should output / target the same set of files, across the directory hierarchy.
Actual behavior
The Get-ChildItem commands behave as expected.
The Remove-Item commands do not:

Remove-Item *.txt -Recurse -WhatIf only targets items directly in the target directory (does not recurse).

Remove-Item . -Recurse -Include *.txt -WhatIf fails.
E.g., on macOS: Remove-Item : Cannot remove the item at '/Users/jdoe' because it is in use.; i.e., an attempt is made to blindly remove ., without considering the -Include wildcard pattern.

Environment data
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on macOS 10.12.4
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Microsoft Windows 10 Pro (64-bit; v10.0.14393)

Remove-Item also does not appear to either sort the items for deletion by leaf-first or at least check if the item exists before deleting it.
The example pattern is to use Get-ChildItem and pipe the results
Get-ChildItem -Path "c:\temp\folder\* -Recurse | Remove-Item

but this will get "directory is not empty" and other errors, and you have to run the same thing 2 or more times to actually clean the folder out.
Its also does not carry over the behavior of rd /s or rmdir where a directory and all its contents must be removed. These two causes lead to code like the following being necessary to have a better chance (but not guarantee) of working.
if (Test-Path $folderPath)
{
  $items = Get-ChildItem -Path $folderPath -Recurse
  foreach ($item in $items)
  {
    if (Test-Path $item)
    {
      Remove-Item $item -Recurse -Force -Confirm:$false
    }
  }
  Remove-Item $folderPath -Force -Confirm:$false
}

Hello @StingyJack
It seems that the issue is bigger than that (prior to Windows 10 1903, if I understood correctly)
This video seems to explain well the bug: https://www.youtube.com/watch?v=uhRWMGBjlO8&feature=youtu.be&t=455
And someone made a terrifying script 🤯 about it: https://stackoverflow.com/questions/53553729/cannot-remove-item-the-directory-is-not-empty/53561052#53561052
function Remove-FileSystemItem {
  <#
  .SYNOPSIS
    Removes files or directories reliably and synchronously.

  .DESCRIPTION
    Removes files and directories, ensuring reliable and synchronous
    behavior across all supported platforms.

    The syntax is a subset of what Remove-Item supports; notably,
    -Include / -Exclude and -Force are NOT supported; -Force is implied.

    As with Remove-Item, passing -Recurse is required to avoid a prompt when
    deleting a non-empty directory.

    IMPORTANT:
      * On Unix platforms, this function is merely a wrapper for Remove-Item,
        where the latter works reliably and synchronously, but on Windows a
        custom implementation must be used to ensure reliable and synchronous
        behavior. See https://github.com/PowerShell/PowerShell/issues/8211

    * On Windows:
      * The *parent directory* of a directory being removed must be
        *writable* for the synchronous custom implementation to work.
      * The custom implementation is also applied when deleting
         directories on *network drives*.

    * If an indefinitely *locked* file or directory is encountered, removal is aborted.
      By contrast, files opened with FILE_SHARE_DELETE /
      [System.IO.FileShare]::Delete on Windows do NOT prevent removal,
      though they do live on under a temporary name in the parent directory
      until the last handle to them is closed.

    * Hidden files and files with the read-only attribute:
      * These are *quietly removed*; in other words: this function invariably
        behaves like `Remove-Item -Force`.
      * Note, however, that in order to target hidden files / directories
        as *input*, you must specify them as a *literal* path, because they
        won't be found via a wildcard expression.

    * The reliable custom implementation on Windows comes at the cost of
      decreased performance.

  .EXAMPLE
    Remove-FileSystemItem C:\tmp -Recurse

    Synchronously removes directory C:\tmp and all its content.
  #>
    [CmdletBinding(SupportsShouldProcess, ConfirmImpact='Medium', DefaultParameterSetName='Path', PositionalBinding=$false)]
    param(
      [Parameter(ParameterSetName='Path', Mandatory, Position = 0, ValueFromPipeline, ValueFromPipelineByPropertyName)]
      [string[]] $Path
      ,
      [Parameter(ParameterSetName='Literalpath', ValueFromPipelineByPropertyName)]
      [Alias('PSPath')]
      [string[]] $LiteralPath
      ,
      [switch] $Recurse
    )
    begin {
      # !! Workaround for https://github.com/PowerShell/PowerShell/issues/1759
      if ($ErrorActionPreference -eq [System.Management.Automation.ActionPreference]::Ignore) { $ErrorActionPreference = 'Ignore'}
      $targetPath = ''
      $yesToAll = $noToAll = $false
      function trimTrailingPathSep([string] $itemPath) {
        if ($itemPath[-1] -in '\', '/') {
          # Trim the trailing separator, unless the path is a root path such as '/' or 'c:\'
          if ($itemPath.Length -gt 1 -and $itemPath -notmatch '^[^:\\/]+:.$') {
            $itemPath = $itemPath.Substring(0, $itemPath.Length - 1)
          }
        }
        $itemPath
      }
      function getTempPathOnSameVolume([string] $itemPath, [string] $tempDir) {
        if (-not $tempDir) { $tempDir = [IO.Path]::GetDirectoryName($itemPath) }
        [IO.Path]::Combine($tempDir, [IO.Path]::GetRandomFileName())
      }
      function syncRemoveFile([string] $filePath, [string] $tempDir) {
        # Clear the ReadOnly attribute, if present.
        if (($attribs = [IO.File]::GetAttributes($filePath)) -band [System.IO.FileAttributes]::ReadOnly) {
          [IO.File]::SetAttributes($filePath, $attribs -band -bnot [System.IO.FileAttributes]::ReadOnly)
        }
        $tempPath = getTempPathOnSameVolume $filePath $tempDir
        [IO.File]::Move($filePath, $tempPath)
        [IO.File]::Delete($tempPath)
      }
      function syncRemoveDir([string] $dirPath, [switch] $recursing) {
          if (-not $recursing) { $dirPathParent = [IO.Path]::GetDirectoryName($dirPath) }
          # Clear the ReadOnly attribute, if present.
          # Note: [IO.File]::*Attributes() is also used for *directories*; [IO.Directory] doesn't have attribute-related methods.
          if (($attribs = [IO.File]::GetAttributes($dirPath)) -band [System.IO.FileAttributes]::ReadOnly) {
            [IO.File]::SetAttributes($dirPath, $attribs -band -bnot [System.IO.FileAttributes]::ReadOnly)
          }
          # Remove all children synchronously.
          $isFirstChild = $true
          foreach ($item in [IO.directory]::EnumerateFileSystemEntries($dirPath)) {
            if (-not $recursing -and -not $Recurse -and $isFirstChild) { # If -Recurse wasn't specified, prompt for nonempty dirs.
              $isFirstChild = $false
              # Note: If -Confirm was also passed, this prompt is displayed *in addition*, after the standard $PSCmdlet.ShouldProcess() prompt.
              #       While Remove-Item also prompts twice in this scenario, it shows the has-children prompt *first*.
              if (-not $PSCmdlet.ShouldContinue("The item at '$dirPath' has children and the -Recurse switch was not specified. If you continue, all children will be removed with the item. Are you sure you want to continue?", 'Confirm', ([ref] $yesToAll), ([ref] $noToAll))) { return }
            }
            $itemPath = [IO.Path]::Combine($dirPath, $item)
            ([ref] $targetPath).Value = $itemPath
            if ([IO.Directory]::Exists($itemPath)) {
              syncremoveDir $itemPath -recursing
            } else {
              syncremoveFile $itemPath $dirPathParent
            }
          }
          # Finally, remove the directory itself synchronously.
          ([ref] $targetPath).Value = $dirPath
          $tempPath = getTempPathOnSameVolume $dirPath $dirPathParent
          [IO.Directory]::Move($dirPath, $tempPath)
          [IO.Directory]::Delete($tempPath)
      }
    }

    process {
      $isLiteral = $PSCmdlet.ParameterSetName -eq 'LiteralPath'
      if ($env:OS -ne 'Windows_NT') { # Unix: simply pass through to Remove-Item, which on Unix works reliably and synchronously
        Remove-Item @PSBoundParameters
      } else { # Windows: use synchronous custom implementation
        foreach ($rawPath in ($Path, $LiteralPath)[$isLiteral]) {
          # Resolve the paths to full, filesystem-native paths.
          try {
            # !! Convert-Path does find hidden items via *literal* paths, but not via *wildcards* - and it has no -Force switch (yet)
            # !! See https://github.com/PowerShell/PowerShell/issues/6501
            $resolvedPaths = if ($isLiteral) { Convert-Path -ErrorAction Stop -LiteralPath $rawPath } else { Convert-Path -ErrorAction Stop -path $rawPath}
          } catch {
            Write-Error $_ # relay error, but in the name of this function
            continue
          }
          try {
            $isDir = $false
            foreach ($resolvedPath in $resolvedPaths) {
              # -WhatIf and -Confirm support.
              if (-not $PSCmdlet.ShouldProcess($resolvedPath)) { continue }
              if ($isDir = [IO.Directory]::Exists($resolvedPath)) { # dir.
                # !! A trailing '\' or '/' causes directory removal to fail ("in use"), so we trim it first.
                syncRemoveDir (trimTrailingPathSep $resolvedPath)
              } elseif ([IO.File]::Exists($resolvedPath)) { # file
                syncRemoveFile $resolvedPath
              } else {
                Throw "Not a file-system path or no longer extant: $resolvedPath"
              }
            }
          } catch {
            if ($isDir) {
              $exc = $_.Exception
              if ($exc.InnerException) { $exc = $exc.InnerException }
              if ($targetPath -eq $resolvedPath) {
                Write-Error "Removal of directory '$resolvedPath' failed: $exc"
              } else {
                Write-Error "Removal of directory '$resolvedPath' failed, because its content could not be (fully) removed: $targetPath`: $exc"
              }
            } else {
              Write-Error $_  # relay error, but in the name of this function
            }
            continue
          }
        }
      }
    }
}

@mickaeltr Your referring to #8211, which will ultimately only be resolved once the WinAPI file-removal calls become synchronous (you can use the function as a workaround, but it has limitations).

Thanks @mklement0 for the link and explanation.


perf: use rel=preload for pages that perform fetching

closes #515.
applied links as suggested, yet additionally had to add crossOrigin="anonymous" to resolve the following warning in the console:

also homogenized the components using fetch to have an endpoint prop

Can you please run deno fmt?


add relation quantity value of

#700

is this ready for review?

It's just that the relations between the quantity values and the "entities in reality" are not yet added. I was wondering, if this sould be done automatically or manually.


Fixes #159221

Fixes https://github.com/microsoft/vscode/issues/159221

Thanks for the PR, will take it in next week after we finish prepping for this months release

@lramos15 this is a regression from this PR https://github.com/microsoft/vscode/pull/157099 for the august iteration so it'll be good to be merged before it hits stable


Support DouYin Live Crawler in China？

Hello
I have a requirement to crawl Douyin data in the China region, similar to TikTok's data. May I ask what we need to do if our project wants to switch to Douyin's live streaming?
Thank you

I don't know what is Douyin


Fix/update sentencepiece dep

The version was failing on my local install:
Installing collected packages: sentencepiece, torch, numpy, pandas, blis, transformers, thinc, spacy, presidio-analyzer, obsei
  Running setup.py install for sentencepiece ... error
  error: subprocess-exited-with-error

  × Running setup.py install for sentencepiece did not run successfully.
  │ exit code: 1
  ╰─> [41 lines of output]
      running install
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.3-arm64-3.9
      creating build/lib.macosx-11.3-arm64-3.9/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.macosx-11.3-arm64-3.9/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.macosx-11.3-arm64-3.9/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.macosx-11.3-arm64-3.9/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Cloning into 'sentencepiece'...
      Note: switching to 'd8711f55d9b2cb9c77a00adcc18108482b29b675'.

      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by switching back to a branch.

      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -c with the switch command. Example:

        git switch -c <new-branch-name>

      Or undo this operation with:

        git switch -

      Turn off this advice by setting config variable advice.detachedHead to false

      ./build_bundled.sh: line 15: cmake: command not found
      ./build_bundled.sh: line 16: nproc: command not found
      make: *** No targets specified and no makefile found.  Stop.
      make: *** No rule to make target `install'.  Stop.
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Failed to find sentencepiece pkg-config
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> sentencepiece```

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Closing, sorry intended to open this against our fork. Not here.

@pramttl Thank you for your PR.
@GirishPatel is already working on fixing this in PR#257
With that, we are going to relax dependencies pinning


Use spark 2.1 and spark SQL

Since we use Spark 2.1, we need a full rebuild of the project with spark 2.1
Changes are mostly due to new version of EventLog listeners
We also need to upgrade to sbt 0.13.9 for an obscure build error
This change is compatible with Spark 1.4> spark logs

Can you test it against our spark 1.6 application? you can launch it from spark-cli

@Willymontaz please amend your first commit
we do not want to merge the "Remove code that should not be in this branch" commit

LGTM


Add --pr to open build preview a python/peps PR

Add --pr to open the ReadTheDocs build preview of PRs at https://github.com/python/peps.
For example, for PR https://github.com/python/peps/pull/2440:
$ pep 594 --pr 2440
https://pep-previews--2440.org.readthedocs.build/pep-0594/

Codecov Report

Merging #1 (e8ee57a) into main (08fe89e) will increase coverage by 3.12%.
The diff coverage is 90.90%.

@@            Coverage Diff             @@
##             main       #1      +/-   ##
==========================================
+ Coverage   87.50%   90.62%   +3.12%
==========================================
  Files           2        2
  Lines          24       32       +8
==========================================
+ Hits           21       29       +8
  Misses          3        3

Flag
Coverage Δ

GHA_Ubuntu
90.62% <90.90%> (+3.12%)
:arrow_up:

GHA_Windows
90.62% <90.90%> (+3.12%)
:arrow_up:

GHA_macOS
90.62% <90.90%> (+3.12%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/pepotron/__init__.py
85.71% <83.33%> (+2.38%)
:arrow_up:

tests/test_pep.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 08fe89e...e8ee57a. Read the comment docs.


Fix: Update incorrect documentation link

What changed? Why?
Updated an incorrect documentation link to the correct one to ensure accurate navigation for developers.
Notes to reviewers
Replaced the outdated link with the current one from the LayerZero documentation.
How has it been tested?
Manually verified that the updated link navigates to the intended documentation page.

🟡 Heimdall Review Status

Requirement
Status
More Info

Reviews

🟡
0/1

Denominator calculation

Show calculation

1 if user is bot
0

1 if user is external
0

From .codeflow.yml

1

Additional review requirements

Show calculation

Max
0

0

From CODEOWNERS

0

Global minimum
0

Max

1

1

1 if commit is unverified
0

Sum
1

@brendan-defi @wbnns Hey ! Sorry to bother you, but could you please review my PR when you have time? There are a few of them because the improvements and fixes were made at different times. I'm leaving for a conference in a day and will be unavailable, so thank you very much in advance for your help !
https://github.com/base-org/web/pull/1384
https://github.com/base-org/web/pull/1383
https://github.com/base-org/op-viem/pull/189

@brendan-defi this pls ?


Content structure suggestion

At the moment the page is a wall of text, which is good because it covers a lot of detail that we need to capture. But it's not as SEO-friendly as it could be.
My suggestion is to break up the sections a bit more and use more conversational phrases for headings. Because doing audits has trained me to think in terms of headings, I've put my idea as a heading structure below.
This would be a big change to the current page so I won't make a pull request until I've had some feedback on it.

H1 - Overlay Fact Sheet

H2 - What's an accessibility overlay?
H2 - Strengths and weakness of overlays. (This section could be a nice side-by-side bullet point list, followed by a sentence about how if you want to know more about the potential benefits you should check with the overlay vendors, but we'll be focusing on the weaknesses.)

H3 - Overlays will not make you WCAG compliant
H3 - Overlays create more access problems than they solve (could include In Their Own Words here, or give it it's own visually-distinct highlighted section).
H3 - Overlays invade privacy
H3 - Overlays cannot fix content that isn't in HTML, CSS or JavaScript
H3 - Overlays create performance problems

H2 - What do we recommend instead of overlays? (Could have some subsections here depending on how much content we have. See karlgroves/overlayfactsheet#36 )
H2 - Conclusion
H2 - Statement from Sponsors and Signatories to this Fact Sheet

H3 - Signed by

H2 - Additional reading

Potential Strengths of Overlays list:

Most accessibility overlays are easy to install
People without disabilities will feel good about doing business with an organisation which cares about people with disabilities
May be useful to people who have newly acquired disabilities and are not yet aware of the assistive technology that is available to them

This is sort of "damn them with faint praise", and the second one throws a fair amount of shade. I don't want to be sarcastic but I think it's important to our credibility to mention that there are some limited advantages. Hopefully they will be outweighed by other considerations.

@stringyland I apologize for letting this rot, but can you issues a PR with the changes you propose?


[Formatter] Fix space between call name and parenthesized arg

When you have a call with a space between the method name and the arguments: foo (1). It is parsed as a call without parenthesis with a single arg which is itself wrapped in parenthesis.
The space is probably not intended here and the parenthesis of the argument is strictly not necessary. Most likely, this should have been written without a space as a parenthesized call.
This patch to the formatter removes that space. There should not be any semantical change because it only applies to a call with a single arg. Calls with multiple args like foo (1), 2 are not affected.
Related note: A call like foo (1, 2) is invalid syntax, so this only applies to calls that have a single argument before and after the space was removed.
This has been spotted in https://github.com/crystal-lang/crystal/pull/11485#discussion_r759539399

It's actually a surprising number of instances where this change has triggered in this repo. Interestingly, the vast majority is eq calls in specs.
229d6a4 (#11523)

The Expressions must also have a size of 1, otherwise foo (1; 2) will be formatted to foo(1; 2), which is invalid code.


Simplify installation and CI

Hopefully this will fix some build issues and duplicities we had in the previous codebase. As with every change that interacts with CI, this is only the first iteration of many...
Hopefully we will fix #185 along the way too.

I try to switch back to pip and this is what happens:

No SciPy wheel on Windows https://github.com/scipy/scipy/issues/5461 and the Christoph Gohlke binaries cannot be downloaded automatically http://www.lfd.uci.edu/~gohlke/pythonlibs/
No pandoc, and therefore I cannot build the docs. One can choose the pandoc version in Travis CI... if using R https://docs.travis-ci.com/user/languages/r/#Pandoc
And of course, no numba.

This is why we can't have nice things...
I'll be going back to conda to setup the environment, but still make sure that the project is installable with pip in a simpler way and fix #187.

I need to update the docs, but that will go in a separate PR.


create github page (re-use guoda.github.io ?) with "big button"

Functionality:

enter a id pattern
create a list of all emls urls of datasets that contain the id pattern

I create a first pass at a simple page where you can enter your identifier, then retrieve associated datasets. See https://guoda.bio/linkreg/ . and https://github.com/bio-guoda/bio-guoda.github.io/blob/master/linkreg.md and attached screenshot. The button turned out to be not as big as I hoped.

@mielliott please review.

that that ideally we'd have a websocket implementation here to let results stream in as they are available.

I did some experimentation with websockets. On server-side, we might want to consider using https://github.com/vi/websocat , proxied by nginx. On client-side, standard Web API's WebSocket can be used - I've added an example here in the linkreg.md .
@mielliott let me know if you are interested taking a stab at this . . . if we get streaming working via websockets, I can imagine a wide range of powerful webapps . . .


OpenTelemetry exporter fails with message Cannot read property 'Resolver' of undefined

Environment

Telemetry Streaming Version: 1.36.0-1
BIG-IP Version: <IP_ADDRESS>

Summary
Opentelemetry_Exporter type of consumer fails to send the collected metrics.
Verifying by tcpdump on the target shows that no attempt to open the tcp connection to send the report is made.
In the /var/log/restnoded/restnoded.log file this line is printed:
Thu, 26 Sep 2024 08:25:09 GMT - severe: [telemetry.service.ConsumersService.OpenTelemetry_Exporter.f5telemetry_default::SRV01_OTEL_GRPC_NOSSL] error: Cannot read property 'Resolver' of undefined

Impact is that I cannot use gRPC to send Opentelemetry reports from BIG-IP.
Steps To Reproduce
Steps to reproduce the behavior:

Submit the following declaration:

{
    "$schema": "https://raw.githubusercontent.com/F5Networks/f5-telemetry-streaming/master/src/schema/latest/base_schema.json",
    "class": "Telemetry",
    "schemaVersion": "1.36.0",
    "controls": {
        "class": "Controls",
        "logLevel": "debug",
        "debug": false,
        "memoryMonitor": {
            "logFrequency": 60
        },
        "memoryThresholdPercent": 90
    },
    "Listener": {
        "class": "Telemetry_Listener",
        "port": 6514,
        "enable": true,
        "trace": false,
        "match": "",
        "actions": [
            {
                "setTag": {
                    "tenant": "`T`",
                    "application": "`A`"
                },
                "enable": true
            }
        ]
    },
    "defaultPoller": {
        "class": "Telemetry_System_Poller",
        "interval": 60,
        "host": "localhost",
        "port": 8100,
        "protocol": "http",
        "allowSelfSignedCert": true,
        "enable": true,
        "workers": 5,
        "chunkSize": 30
    },
    "telemetrySystem": {
        "class": "Telemetry_System",
        "enable": true,
        "systemPoller": [
            "defaultPoller"
        ],
        "host": "localhost",
        "port": 8100,
        "protocol": "http",
        "enableHostConnectivityCheck": false,
        "allowSelfSignedCert": true
    },
    "SRV01_OTEL_GRPC_NOSSL": {
        "class": "Telemetry_Consumer",
        "type": "OpenTelemetry_Exporter",
        "enable": true,
        "trace": true,
        "host": "IP-address-redacted",
        "port": 4317,
        "exporter": "grpc",
        "convertBooleansToMetrics": false,
        "useSSL": false
    }
}

Declaration is accepted, response is 200 OK:

{
    "message": "success",
    "declaration": {
...

Wait for the poller to collect data and observe the restnoded.log file for result. Also, verify with tcpdump if any connection is attempted to the consumer.

Expected Behavior
The consumer of type OpenTelemetry_Exporter would be succesful in sending telemetry reports using gRPC.
Actual Behavior
No report is sent out. Based on logs, data is collected, but sending to the consumer fails even before a TCP connection is attempted:
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Starting polling cycle
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "WAITING" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "WAITING" to "COLLECT"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.Loader[localhost]] Task queue "TaskQueue_9dd6d" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.context.Collector] Starting stats collection
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.context.Collector] Task queue "TaskQueue_4667f" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.stats.Collector] Starting stats collection
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.stats.Collector] Task queue "TaskQueue_918df" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully collected stats
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "COLLECT" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "COLLECT" to "SEND_REPORT"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully processed stats report
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "SEND_REPORT" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "SEND_REPORT" to "DONE"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully collected and processed stats. Polling Cycle duration - 0 sec.
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "DONE" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "DONE" to "SCHEDULE"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Next polling cycle starts on 2024-09-26T08:26:09.029Z (in 59 s.)
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully scheduled next execution date
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "SCHEDULE" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "SCHEDULE" to "WAITING"
Thu, 26 Sep 2024 08:25:09 GMT - severe: [telemetry.service.ConsumersService.OpenTelemetry_Exporter.f5telemetry_default::SRV01_OTEL_GRPC_NOSSL] error: Cannot read property 'Resolver' of undefined

Hi @jussisjostrom,
New versions of OTEL library can't be used because node.js is too old and OTEL libs has been updated and it fails now. So it even fails on the version that worked in the past: the server side was updated and as result the client side (TS) tries to trigger the code that even does not exist in the library or does not work on node.js 8.11.1

@jussisjostrom
you can try http/protobuf instead of grpc if you server configured to accept it

Same problem here after upgrading to Telemetry 1.37
"f5-collector": {
"class": "Telemetry_Consumer",
"type": "OpenTelemetry_Exporter",
"host": "<IP_ADDRESS>",
"port": 4317,
"trace": true,
"enable": true,
"headers": [
{
"name": "x-access-token",
"value": "YOUR_TOKEN"
}
],
"convertBooleansToMetrics": true,
"exporter": "grpc",
"useSSL": false,
"allowSelfSignedCert": true
},
What's the plan? should we revert back to previous version or will someone fix issue?

Telemetry 1.36 has the same issue.

Hi @PatricDahl
New versions of OTEL library can't be used because node.js is too old and OTEL libs has been updated and it fails now. So it even fails on the version that worked in the past: the server side was updated and as result the client side (TS) tries to trigger the code that even does not exist in the library or does not work on node.js 8.11.1.
Can you try http/protobuf instead of grpc if you server configured to accept it?


Add SSL Option to Busola backend

Description
Add the option to secure the backend service with SSL.
It is completely optional, if the environment variables are not set the service will start up as normal without SSL.

Reasons

Attachments

Done in https://github.com/kyma-project/busola/pull/1011


[libclocale] Initial Port

[x] Changes comply with the maintainer guide.
[x] The name of the port matches an existing name for this component on https://repology.org/ if possible, and/or is strongly associated with that component on search engines.
[x] Optional dependencies are resolved in exactly one way. For example, if the component is built with CMake, all find_package calls are REQUIRED, are satisfied by vcpkg.json's declared dependencies, or disabled with CMAKE_DISABLE_FIND_PACKAGE_Xxx.
[x] The versioning scheme in vcpkg.json matches what upstream says.
[x] The license declaration in vcpkg.json matches what upstream says.
[x] The installed as the "copyright" file matches what upstream says.
[x] The source code of the component installed comes from an authoritative source.
[x] The generated "usage text" is accurate. See adding-usage for context.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is in the new port's versions file.
[x] Only one version is added to each modified port's versions file.

Pinging @eyalgolan1337 for response. Is work still being done for this PR? There are conflicts error on CI pipeline as below.
Installing 9/9<EMAIL_ADDRESS>Building<EMAIL_ADDRESS>-- Downloading https://github.com/libyal/libclocale/releases/download/20240414/libclocale-alpha-20240414.tar.gz -> libclocale-alpha-20240414.tar.gz...
-- Extracting source D:/downloads/libclocale-alpha-20240414.tar.gz
-- Using source at D:/b/libclocale/src/a-20240414-f70f96bd93.clean
-- Using cached libyal-vstools-f412b5f4347839c31a6f7ff2a631990f84d81b40.tar.gz.
-- Extracting source D:/downloads/libyal-vstools-f412b5f4347839c31a6f7ff2a631990f84d81b40.tar.gz
-- Using source at D:/b/libclocale/src/0f84d81b40-78707ee12a.clean
-- Found external ninja('1.11.0').
-- Getting CMake variables for x86-windows
-- Building vs2022/libclocale.sln for Release
-- Building vs2022/libclocale.sln for Debug
-- Installing: D:/p/libclocale_x86-windows/include/libclocale/definitions.h
-- Installing: D:/p/libclocale_x86-windows/include/libclocale/error.h
-- Installing: D:/p/libclocale_x86-windows/include/libclocale/extern.h
-- Installing: D:/p/libclocale_x86-windows/include/libclocale/features.h
-- Installing: D:/p/libclocale_x86-windows/include/libclocale/types.h
-- Installing: D:/p/libclocale_x86-windows/include/libclocale.h
-- Installing: D:/p/libclocale_x86-windows/share/unofficial-libclocale/unofficial-libclocale-config.cmake
-- Installing: D:/p/libclocale_x86-windows/share/libclocale/copyright
-- Performing post-build validation
error: The following files are already installed in D:/installed/x86-windows and are in conflict with libclocale:x86-windows
Installed by libcerror:x86-windows
debug/lib/libcerror.lib
    lib/libcerror.lib

You can get the more logs from https://dev.azure.com/vcpkg/public/_build/results?buildId=102483&view=logs&jobId=878666d5-db33-5b27-9e7d-b0c7ee352005&j=878666d5-db33-5b27-9e7d-b0c7ee352005&t=dc897974-8f7e-5107-e5db-0a01be8e9d4f, could you please resolve this error?

Closing this PR since it seems that no progress is being made. Please ping us to reopen if work is still being done.


Support non-explicit output argument calls to Numpy ufunc's with NRT.

This PR builds on PR #1077 (sklam/numba_runtime), and also requires PR #1078.
This PR enables the following example to work:
from numba import *
import numpy as np

@njit(nrt=True)
def myadd(a0, a1):
    return np.add(a0, a1)

x, y = np.linspace(0,1,10), np.linspace(1,2,10)
print(myadd(x, y))

Now that NRT is merged, this PR needs to remerge master again, I think.

Can you add some test cases for Fortran-ordered arrays?  After a few experiments with Numpy, it seems that operations where all the arguments are in Fortran-order result in a Fortran-order output.  If any argument is in C-order, then the output is in C-order.

I added one unit test, and found us lacking.  Fix is in.  More required, or is one enough?

That test looks like it covers the two important cases.


Explore adopting multi file diff editor for notebook diff

### Tasks
- [x] New Editor behind a setting
- [ ] Ability to edit cells on the right hand side
- [ ] Ability to edit outputs/metadata on the right hand side (ensure metadata changes are validated, as valid JSON before saving)
- [x] Ability to revert input, metadata and output
- [x] Address issue with not being able to open editor more than once
- [ ] Toggle between rich and plain text outputs
- [ ] View white space differences in cells
- [x] Collapse unchanged cells (perhaps we can display them and just collapse by default, instead of hiding them?)
- [x] Navigation icons in editor toolbar to navigate to next/previous changed cell
- [x] Menu to show/hide metadata/output differences
- [ ] Breadcrumbs in new Editor (for some reason its not displayed)
- [x] Possibly display all output/metadata and suffix with (changed) and expand changed items by default (instead of hiding them)
- [ ] Change progress lable from `No Changed Files` to something else (e.g. `Computing Notebook Diffs`)

Internal Notes
Features not available in multi-diff that will need to be added

Diff overview ruler
Display unmodified cells (akin to displaying un-modified files) & ability to hide/collapse these

Will be required for diff overview ruler as well

Custom toolbar actions

Revert a cell
Revert changes to metadata
Revert changes to outputs

Add margin for metadata & output diff view (but not allow reverting individual parts, as we only allow reverting everything)

I.e. we do not want to give fine grained control of diff editing of metadata changes.

Grouping a cell (i.e. ensure sub-sections under the cell resource for output/metadata, etc)

Current approach of treating them as separate resources works, but feels wrong (no separation between cells)
Perhaps having the ability to group resources would work

Custom names of the resources (currently file name cannot be removed)
Ignore white space (per resource/cell)
Accessibility (e.g. keyboard navigation between cells)
Optionally: Ability to have custom content in each diff view item

Current work around of using editor insets works best (before we build something more generic)

Other changes

Ability to create read-write editor models (currently all virtual text models are readonly), only file backed models are read-write.

Need to create a custom impl of TextResourceEditorModel & modify textModelResolverService.ts

Output Diffing

Using individual web views for each output seems to work

However still using the backview layer components with messages being sent back n forth).
I.e. given we're using the same infra structure, we might as well use the single webview approach.

There are some delays in loading the webviews

Multi-Diff Editor View Model
Current approach of using a custom impl of IMultiDiffSourceResolver works,

However we need access to the current Diff Editor (IEditorGroup)
We need the ability to store local disposables (the source resolver doesn't provide that option)

Perhaps a custom EditorInput class will work (but requires changes to multi-diff editor class)

Ability to mark an editor as editable (right now cells cannot be edited as they are virtual documents, unless we make a change in core text resource provider)
custom context for toolbar actions (currently hardcoded to pass uri)
Changes to expose a custom context object via view model (currently need to perform lookup via uri, similar to previous request)
Custom collapsible sections (with a hierarchy)

Custom borders to  clearly distinguish between cells, metadata, etc)

Toggle white space differences per item (or resource type)
Change progress lable from No Changed Files to something else (e.g. Computing Notebook Diffs)
ability to hide/collapse unchanged cells (like unchanged lines)
ability to navigate to each item (keyboard navigation to next changed cell)

ie ficus next item if not visible in viewport

Custom editors (webview support in outputs)


sticky cache: fix resource leaks on worker.Stop()

Currently, entire sticky cache is leaked when worker is stopped. This is because the stickyCache is a global singleton which makes it impossible to clear it on worker.Stop() (because there could be other workers using the cache and there is no way to tell which entries belong to which worker). Leaking sticky cache results in memory & go routine leaks.
The right fix here is to create stickyCache per worker and purge it completely on worker.Stop()

Ported from: https://github.com/uber-go/cadence-client/issues/739

Fixed this a while back https://github.com/temporalio/sdk-go/pull/310


Add icon toolbars in capture view.

Add stb_image to load png files
Add icons from Google's https://material.io/resources/icons/
Add post-build steps to package icons folder next to executable
Add common actions, track filter, search filter and process info
toolbars in capture view

Short question upfront regarding priorities: Are you planning on merging this into 1.48?

Are you planning on merging this into 1.48?

Ideally yes, but I understand it's a relatively big change. I submitted into master in case there are issues with the PR.

Are you planning on merging this into 1.48?

Ideally yes, but I understand it's a relatively big change. I submitted into master in case there are issues with the PR.

I consider this a fix for b/156245865. Agree that it's a larger fix that I would normally not consider purely a "bugfix", but the quality delta due to this change is quite large IMO, so I think it's worth the risk.

Hi @beckerhe , thanks for the feedback!  I added the license for the icons, and modified the stb license to only show the MIT section.  Concerning the conan package, I had actually checked and all I found was a java port of the library.  Good to know that it is indeed available for C++.  Using the conan version will have to be done in an other PR though due to time constraints for release.


Is this repo realy "CSS only"?

The Javascript files are in the repo's root as well, but the README sais "CSS only version of Semantic UI" ... I'm confused. I can e.g. initialize a dropdown by adding the class simple, but calling $('.dropdown').dropdown() causes TypeError: $(...).dropdown is not a function

OK, the error is resolved. Was accidentally loading jQuery a second time after semantic. So I'd state that this is not a CSS only version.

It would be nice to have a CSS only package for npm to use it with stardust (official Semantic-UI + React integration) or semantic-react.
They both don't need jquery or semantic-ui-js.
At least we should make the jquery npm dependency optional or get rid of it, as it's not necessary for this use-case.

I think that is better to call "assets-only" or "static-files-only".

or just "compiled"?

I would like to continue this discussion.  An example of why this is a problem is that, for example, in the React Semantic-UI docs, the very first line of the homepage says: jQuery Free
However, in the installation section it suggests using this repo, which imports jQuery as a transitive dependency, into the project.  While semantic-ui-react is not using jQuery, your project bundle size has increased by ~100 KB (assuming no tree shaking).
I would suggest a new CSS-Only repo that has literally that: CSS Only.

So are there any packages that are css only? In my project I would just like to use the semantic-ui css classes, and have no need for the JS.
Adding it as a dependency is preferable to adding the raw css (even minified), as it just bloats my repo.
Happy to help if I can :upside_down_face:


Struct Values Assignment

I may be just over looking something but how would i go about assigning values to a struct?

Once you create the struct type, just instantiate an instance and assign properties unless there's a portion of your question that I'm missing.
$Mod = New-InMemoryModule -ModuleName Test

$SampleStructType = struct $Mod TestStruct @{
    Field0 = field 0 UInt16
    Field1 = field 1 UInt16
    Field2 = field 2 UInt32
    Field3 = field 3 UInt32
    Field4 = field 4 UInt32
    Field5 = field 5 UInt16
}

$SampleStructInstance = New-Object -TypeName TestStruct
$SampleStructInstance.Field0 = 1

has this been tested on PS version 2? Trying to work with LSA_Policy
`
Import-Module ".\PSReflect.psm1"
$Mod = New-InMemoryModule -ModuleName Win32
$LSA_OBJECT_ATTRIBUTES_TYPE = struct $Mod LSA_OBJECT_ATTRIBUTES @{
Length                   = field 0 UInt32
RootDirectory            = field 1 IntPtr
ObjectName               = field 2 IntPtr
Attributes               = field 3 UInt32
SecurityDescriptor       = field 4 IntPtr
SecurityQualityOfService = field 5 IntPtr
}
$LSA_OBJECT_ATTRIBUTE = New-Object -TypeName LSA_OBJECT_ATTRIBUTES;
$LSA_OBJECT_ATTRIBUTE.Length = 0;

`
Tried it on v5 worked great but v2 no luck :/

Ah okay yeah. There's a quirk with v2 that prevents you from calling New-Object on types created with PSReflect. You have to use the Activator class to get an instance.
$Mod = New-InMemoryModule -ModuleName Win32

$LSA_OBJECT_ATTRIBUTES_TYPE = struct $Mod LSA_OBJECT_ATTRIBUTES @{
Length = field 0 UInt32
RootDirectory = field 1 IntPtr
ObjectName = field 2 IntPtr
Attributes = field 3 UInt32
SecurityDescriptor = field 4 IntPtr
SecurityQualityOfService = field 5 IntPtr
}

$LSA_OBJECT_ATTRIBUTE = [Activator]::CreateInstance($LSA_OBJECT_ATTRIBUTES_TYPE)
$LSA_OBJECT_ATTRIBUTE.Length = 1

Let should do it. I hope that helps!

Yes it helped a lot. I was reading the comments and saw something about GetSize. I tried using the method but get an error
$LSA_OBJECT_ATTRIBUTE = [Activator]::CreateInstance($LSA_OBJECT_ATTRIBUTES_TYPE);
$LSA_OBJECT_ATTRIBUTE.Length = $LSA_OBJECT_ATTRIBUTE.GetSize();
GetSize : Method invocation failed because [LSA_OBJECT_ATTRIBUTES] doesn't contain a method named 'GetSize'.
At C:\Users\Raul Martinez\Source\Repos\PowerExporter\PowerExporter\PowerExporter\PowerExporter.ps1:54 char:62

$LSA_OBJECT_ATTRIBUTE.Length = $LSA_OBJECT_ATTRIBUTE.GetSize <<<< ();

CategoryInfo          : InvalidOperation: (GetSize:String) [], RuntimeException
FullyQualifiedErrorId : MethodNotFound

GetSize is a static method so you need to invoke it as one (w/ ::).
$LSA_OBJECT_ATTRIBUTE = [Activator]::CreateInstance($LSA_OBJECT_ATTRIBUTES_TYPE)
$LSA_OBJECT_ATTRIBUTE.Length = $LSA_OBJECT_ATTRIBUTE::GetSize()

Thank makes sense. Thank you very much 👍


Addition of the ability to decorate with custom fixed tags

prometheus dropwizard bundle PR
<High level description of the PR>
This PR adds the ability to decorate the metrics with a set of custom fixed tags that do not resolve from the metric name.
Added

Ability to configure the applcation to use custom tags
Custom decorator to add the tags via the SampleBuilder interface
Tests

Changed

Readme changes describing the feature

LGTM


Support for passing along options.data to node-sass

I have a use case where I need to be able to pass along the option data to node-sass. This PR fixes this.

Hmm, those failing tests seems to not be related to this PR. The tests pass when node-sass v3.4.1 is used (as specified in package.json) but not in the latest v3.4.2.

I'm unclear what the usecase is here. Please elaborate.

I want to be able to use the data option that exists in node-sass, but through the gulp-sass plugin.
I think the unit test I added shows the use case pretty good; to be able to pass arbitrary strings from JS to Sass.

This PR would be great for us. We would like to be able to pass our static asset path (as defined in our case by a django instance) to sass. In our case we cannot use relative paths as the CSS is being inlined into the HTML after being compiled to CSS (for performance and IE9 file size issue reasons).

This feature has been discussed at length and reject until a valid use case can be proposed. Please read the discussion in https://github.com/dlmanning/gulp-sass/issues/302 for more context.
If you believe you still believe you have use cases that doesn't violate the gulp semantics please leave a thorough description on that #302 and we'll reopen the discussion.


🐦 Handle tweet event task

📝 Description
Add handler for all NewTweetEvent. Each tweet received by the event handler, the subscriber will find in the current phase if there is a tweet task, If tweet task has been found and if the task is in progress, the subscriber will mark the task has completed for the corresponding validator.
A check is also done to verify if the tweet has been posted between the start and end date of the task.
🐛 Fix
Include the created at date of tweet that was not working.

:tada: This PR is included in version 1.2.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Unable to reuse db connections in API due to a non-empty event loop

As documented here https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html, for non-async lambdas which is what this plugin creates the lambda terminates only when the event loop is empty.
An example is to create a connection with mysql and keep the connection open for the next invocation to reduce the overhead of creating new connections. The lambda will continue to run after called res.end leading to timeouts.
I can see two likely solutions to this:

Refactor the entry point to be an async function.
Set context.callbackWaitsForEmptyEventLoop = false by default in the entry point.

Option 2. seems to be pretty easy, but I'm not sure if there are any other side effects compared to using async. We could also just set it for APIs since that's the only place were I image people would need it.

@Deadleg thanks for the awesome contributions in the last couple of days! I would be very interested in making the entry point async! It should be relatively straightforward to do that. Feel free to PR if you got time.

In fact, I've found today making it async might not be a great idea after all.
From https://docs.aws.amazon.com/lambda/latest/dg/limits.html:

That would limit the response size significantly.

I've refactored the compat layer to support the async signature. It also maintains backward compat with the callback approach. @Deadleg I've not tested it, but this issue should be fixed in the new release. Feel free to reopen if you encounter any problems.


Exclude commons-logging deps coming transitively + replace with jcl-over-slf4j where needed

@mbarkley this is the changes we talked about yesterday. It was much easier than I anticipated :)

Thanks, Petr!
I've also cherry-picked this to master: 79770b0

Thanks Max.


ipam: Fix overlapping/duplicate PodCIDR allocation when nodes are added while operator is down

This PR fixes an edge case in the NodesPodCIDRManager.
If there were any nodes on operator startup which have no PodCIDRs, the
operator would sometimes assign PodCIDRs to these nodes which have
already been allocated to other nodes.
The operator assumed that when k8sCiliumNodesCacheSynced closes, all
node events have been processed. And it proceeds to call Resync on
the nodeManager.
The NodesPodCIDRManager will queue any nodes
without PodCIDRs to be allocated once the canAllocatePodCIDRs variable
is set. This variable is set by the Resync.
So, the assumption/expected behavior is that the
NodesPodCIDRManager.Update function has been called for all nodes in
the cache before Resync is called. However, this wasn't the case.
The startSynchronizingCiliumNodes function starts the informer
and connects the nodeManager to it. But instead of handling the events
at once, the callbacks enqueue the events, to be handled by a separate
go routine. This means that k8sCiliumNodesCacheSynced is closed once
all of the node events are enqueued, not when they have been processed
by the nodeManager.
This commit fixes this behavior by processing all events at once
in the informer callbacks until the full sync is complete, at which
point we will switch over to using the workqueue.
Fixes: #21482
Fix overlapping/duplicate PodCIDR allocation when nodes are added while operator is down

/test

note I'm not very familiar with this logic and don't understand all consequences this change can have

Me neither. This change bypasses a queuing / rate limiting mechanism on startup. So it might cause more load on the API server on startup. Digging into the commit history, that doesn't seem to be the reason for introducing the queue.
This PR introduces the queue, the commit says "this commit introduces the right
way of handling Kubernetes events, with a workqueue" but I am not sure why this is the right way. @aanm could you clarify?

@aanm Is unavailable for two weeks.
An alternative solution which would preserve rate limiting is to enqueue the "k8s cached synced" event as well. Since the queue already allows any type to be enqueued, we could enqueue a special "resync" key and invoke nodeManager.Resync when we dequeue the key.

An alternative solution which would preserve rate limiting is to enqueue the "k8s cached synced" event as well.

Yes, that is also something that crossed my mind. I initially went for this approach because it is the minimal change, and reverts to the behavior of <=v1.10. I don't see the added benefit of the worker queue personally since events are already queued internally in the informer.
I would personally like to refactor this code at some point to use the new Resources. The event handling of the NodeManagers is currently split across packages and files and relies on these global variables to synchronize across at least 3 goroutines. Having each node manager be its own Cell with a dependency on Resource[CiliumNode] and Invoking one of them depending on config. Then making the KVStore synchronization its own Cell would remove a lot of complexity.

/test

All tests green, except for the connectivity test, which has the following error:
curl: (22) The requested URL returned error: 503

So I guess cloudflare has a bad day, seem unrelated to these changes.
We have decided to merge the fix into master now without waiting for André's feedback. Removing the backport labels for now in case there turns out to be an issue with this. Will re-add the labels as soon as we get the all clear next week.

curl: (22) The requested URL returned error: 503

In this case it's okay, but there's one case where we can cause a 503, that's when L7 policies are enforced.

The reason why there's a workqueue instead of processing the event directly from within the informer callbacks is because that processing will actually delay the informer itself. With the workqueue, you receive events and push them into a queue for the work to be done later, without blocking the informer itself.

That is fair, but for the node sync we have our own dedicated ciliumNodeInformer. So the only thing we are impacting is the sync to the KVStore which should be handled after nodeManagerSyncHandler anyways. So I don't think it matters that much, unless I am missing something

Changed the fix to use the solution proposed by @gandro, we now insert a special key in the queue and added a second channel. So we now have the original cacheInSync signal which is closes as soon as the informer has perform the initial sync. The new signal closes as soon as the nodeManager is in sync, which is triggered by consuming the special key from the queue.

/test

FYI, resolving the failures in 1.25-net-next would require a rebase. There are some new/updated tests that depend on recent cilium changes (ie. another case of https://github.com/cilium/cilium/issues/15474).

/test

Now the k8s-1.16-kernel-4.9 fails, which passed before the rebase. The  Transparent encryption DirectRouting Check connectivity with transparent encryption and direct routing test failed, which doesn't seem to be related to this PR. So marking as ready-to-merge.

@aanm

Unfortunately the test in the first commit only fails on my side 6 out of 10 times.

 Didn't run into this myself, it looks like this is because the nodes are stored in NodesPodCIDRManager.nodesToAllocate which is a map and then iterated over so the order is randomized each time. I have tried increasing the amount of nodes to make it more likely that we get a map order that triggers the bug, but I still can't get it 100% reliable. Also can't rerun the test x times until it fails due to global state and never stopping goroutines. So I think I can't make it reliable.
Do we leave the test in, in which case it will flake if we ever regress or do we remove the test which leaves us with no regression detection?

@aanm

Unfortunately the test in the first commit only fails on my side 6 out of 10 times.

 Didn't run into this myself when initially making the test, it looks like this is because the nodes are stored in NodesPodCIDRManager.nodesToAllocate which is a map and then iterated over so the order is randomized each time.
I have tried increasing the amount of nodes to make it more likely that we get a map order that triggers the bug, but I still can't get it 100% reliable. Also can't rerun the test x times until it fails due to global state and never stopping goroutines. So I think I can't make it reliable.
Do we leave the test in, in which case it will flake if we ever regress or do we remove the test which leaves us with no regression detection?

@dylandreimerink lets keep the test in. Please add a comment in the test stating that if it's flaking it's likely a real failure because its not possible to make it reliable.

@dylandreimerink

Also can't rerun the test x times until it fails due to global state and never stopping goroutines. So I think I can't make it reliable.

You can store the global state at test startup and then revert it at test teardown. We do this in a few tests already throughout the code.
And which goroutines are never stopping? I think we should change the code to make testing more reliable, no?

This was a hint to leave a comment in the code since it's not obvious :)

Oh, missed that hint. Added a verbose comment explaining the mechanism, should be clear now.

@dylandreimerink Are you considering https://github.com/cilium/cilium/pull/21526#issuecomment-1276690839?

You can store the global state at test startup and then revert it at test teardown. We do this in a few tests already throughout the code.
And which goroutines are never stopping?

I will take a gander at it. Might indeed not be a big an issue as I initially thought. There are two never ending goroutines: here and here. Though both seem to only modify local variables or arguments and do not modify the global state, so I think I can keep them running without issues.

I think we should change the code to make testing more reliable, no?

Yes. Agreed. This PR was mainly for the actual bug fix, not to improve the operators structure so it is more testable. I would honestly prefer to do that in a separate PR in line with Jussi's modularization work

I updated the test, to run up to 5 times now which seems to make this test reliable enough to remove the flake warnings. In my testing it typically triggers the bug now in run 1 or 2. This still leaks goroutines, but it doesn't impact this test case. I am writing an issue for the refactoring now

@aanm I added my detailed comment on the wrong commit, changed that now, does this address your previous review?

@dylandreimerink IMO the best way to make sure you didn't miss anything is to mark resolved everything that you think you addressed. If something isn't marked resolved then maybe it was missed?

/test


Missing ContentDialog in WinUI 3 template

This issue has been moved from a ticket on Developer Community.

I am using VS 2019 v16.11.3 to build a WinUI 3 desktop project (packaged) and are missing the possibility to add a custom ContentDialog using "Add item".
Is there something wrong with my VS installation or is this coming in a future update?
! [Knipsel.PNG] (https://aka.ms/dc/image?name=Bbddae67e40254e24b7d413e57b24c971637692102444732142_Knipsel.PNG&tid=bddae67e40254e24b7d413e57b24c971637692102444732142)

Original Comments
Feedback Bot on 10/7/2021, 08:16 PM:
We have directed your feedback to the appropriate engineering team for further evaluation. The team will review the feedback and notify you about the next steps.

Original Solutions
(no solutions)

This is a known shortcoming of the current winui3 package.  As a work around you could add a new user control and then change the type from usercontrol to ContentDialog in the .xaml and the .xaml.cs file, which should work.

This template is missing again after updating VS 2022 Professional to 17.2.2


Edit profile page changes for Shibboleth

Remove the shibboleth link/unlink functionality from the edit profile page.
If the user has logged in via shibboleth (they have a record in user_identifiers for shib) then hide the passwords tab.

closing this old one as I think its no longer applicable


Switch PDB from 'embedded' to 'portable' for rzls

### Summary of the changes

Switch PDB from 'embedded' to 'portable' for rzls
This is an attempt to make builds more reliable, so that conversion from Windows to Portable is unnecessary for Linux and Mac.

Embedded PDBs are portable though so curious what problem this solves.


Modify robin-map download to look in system first

Instead of downloading unconditionally, look for system install.
Sorry, this looks like a bigger change than it really is. Most of the "changed lines" are just indented differently because they are enclosed in an if.

Had some other packages that needed attention. Testing now.

Ok, build completed. There are a lot of warnings generated with GCC 8.1 but not related to this PR so I'll post it separately.


Document RenderingServer.get_video_adapter_name() may report a fixed name

Firefox may[^1] report the user's GPU as a GeForce GTX 980 in an attempt to make fingerprinting more difficult. This is not the case in Chromium-based browsers though.
[^1]: I've noticed this on 3 different setups (GTX 1080, RX 6900 XT, RTX 4090). I don't know if GPUs older than the GTX 980 are affected by this (probably not).

Thanks!

Cherry-picked for 4.1.2.


Root certificate expired

HTTP Strict Transport Security certificate expired; can't connect or add exception (Firefox).

The same for me (Firefox)


[Feature Request] - Separate Data Storage Files per Metric Category

Your plugin is awesome btw!!
Request Summary
I am using IOS Shortcuts to track my habits.  When I trigger a shortcut, it appends a text string to my logseq markdown file. I use your plugin to monitor progress on properties such as ::weight, ::supplements, etc but I can't use the Data Storage because of IOS Shortcut limitations.
By using IOS Shortcuts, I am not able to leverage the full plugin because I am not able to append Data Points to the metrics-plugin-data.md with multiple properties.
I would like to request the option to split up the metrics-plugin-data.md file into multiple files (each representing one property).  With this type of setup, I could append my habits directly to the md page and leverage the full plugin.
Use-Case

Thanks for the feature request!  I think this is definitely doable.  But let me ask – would it be possible to instead have the shortcut append a block to your journal and use block properties to store the metrics you want to track?  The Metrics Plugin allows you to create Properties Charts from entries in your journal rather than pulling data from the metrics-plugin-data page.  Can you give that a try to see if will meet your needs?

I've been using the Journal pages to append the metrics as you recommended.  I can successfully view the graphs :) However, I'm having trouble with getting the Cards to work.  I'm interested in having a view where I can view the cards of the metrics on the journal pages.
Here is an example of the card that I can't get to work:
{{renderer :metrics, weight, -, latest}} (KO)
{{renderer :metrics, weight, -, properties-line}} (OK)
Maybe my syntax is incorrect?
Also I really appreciate you reading and responding to my request.  I love your plugin!!!!

Got it. The cards currently don’t work with data stored in the journal / block properties. There is a similar feature request for the bar graph, so I’ll go ahead and implement support for all visualizations using block properties.

That would be awesome!  Thank you very much!!

I'm working on this now.  I hope to have it released soon.

Implemented in version 0.17.

Thank you 🙌


Fix pattern-match warning on GHC 8.2

template-haskell-2.12 added an Info constructor for pattern synonyms, so let's catch them here.

Thanks!

Uploaded as version 0.1.8


test(amplify-util-mock): add e2e tests for new transformers

Description of changes
Port non-auth mock util tests over for v2 transformers. (Auth tests to follow.)
Checklist

[x] PR description included
[x] yarn test passes
[x] Tests are changed or added
[x] Relevant documentation is changed or added (and PR referenced)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

A small generic comment: can we put the v2 tests in their own directory. It will make future maintenance a lot easier, especially for people not working on the initial v2 implementation.

Codecov Report

Merging #8238 (acfa82c) into master (acfa82c) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head acfa82c differs from pull request most recent head 3b2910f. Consider uploading reports for the commit 3b2910f to get more accurate results

@@           Coverage Diff           @@
##           master    #8238   +/-   ##
=======================================
  Coverage   55.39%   55.39%
=======================================
  Files         885      885
  Lines       50141    50141
  Branches    10814    10814
=======================================
  Hits        27774    27774
  Misses      20314    20314
  Partials     2053     2053

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acfa82c...3b2910f. Read the comment docs.

Ready for merge once it's approved.

It looks like there are some lgtm bot issues to address.

@cjihrig fixed the bot alerts. ready for rereview!


Support setup.cfg

Setup.py has been replaced by setup.cfg and is now moving towards pyproject.toml. Probably there aren't many projects on setup.py anymore.
While I was cleaning up an old setup.py & miniver project, migration seemed straight forward (with https://github.com/pypa/setuptools/issues/2570 now implemented).
Except that cmdclass dict values must be module paths to a public, top-level class. Since the change 1c9f8ca1dfc759488a078b67f14c8ed41300eb47, the build command classes can only be created through function calls, which does not work for the declarative setup.cfg.
I tried working around by creating a helper object with lazy properties (so I can preserve the lazy import of super classes), but that is not detected by setuptools and gives an error.

_version.py
…
PACKAGE_SOURCE_PATH = os.path.dirname(__file__)

class CmdClass:
    build_py = None
    sdist = None

    def __init__(self, pkg_source_path):
        self._pkg_source_path = pkg_source_path

    @property
    def build_py(self):
        from setuptools.command.build_py import build_py as build_py_orig

        pkg_source_path = self._pkg_source_path

        class _build_py(build_py_orig):
            def run(self):
                super().run()

                src_marker = "".join(["src", os.path.sep])

                if pkg_source_path.startswith(src_marker):
                    path = pkg_source_path[len(src_marker):]
                else:
                    path = pkg_source_path
                _write_version(os.path.join(self.build_lib, path, STATIC_VERSION_FILE))

        return _build_py

    @property
    def sdist(self):
        from setuptools.command.sdist import sdist as sdist_orig

        pkg_source_path = self._pkg_source_path

        class _sdist(sdist_orig):
            def make_release_tree(self, base_dir, files):
                super().make_release_tree(base_dir, files)
                _write_version(os.path.join(base_dir, pkg_source_path, STATIC_VERSION_FILE))

        return _sdist

cmdclass = CmdClass(pkg_source_path=PACKAGE_SOURCE_PATH)

def get_cmdclass(pkg_source_path=PACKAGE_SOURCE_PATH):
    cmds = CmdClass(pkg_source_path=pkg_source_path)
    return dict(sdist=cmds.sdist, build_py=cmds.build_py)

# Unlike expected, setup.cfg cannot access cmdclass.build_py, but the following would work:
# build_py = cmdclass.build_py
# sdist = cmdclass.sdist

setup.cfg
…
[options]
cmdclass =
    build_py = my_package._version.cmdclass.build_py
    sdist = my_package._version.cmdclass.sdist

ModuleNotFoundError: __path__ attribute not found on 'my_package._version' while trying to find 'my_package._version.cmdclass'

In short, first I think it would be good to support setup.cfg, and second, I am not sure whether it is feasible without moving the build command classes to the top level.

I have only played with standalone setup.py and with setup.py + pyproject.toml, so I'm not very familiar with setup.cfg
From my understanding, in a setting that has  setup.py + pyproject.toml you can specify under the [project] section that "version" is dynamic:
#### pyproject.toml
[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"

[tool.setuptools.packages.find]
where = ["."]

[project]
name = "some_package"
# ... some other unrelated fields
dynamic = ["version", "readme"]

[tool.setuptools.dynamic]
version = {attr = "some_package.__version__"}
readme = {file = "README.md", content-type = "text/markdown"}

#### setup.py
from setuptools import setup

pkg = "some_package"

# Loads _version.py module without importing the whole package.
def get_version_and_cmdclass(package_name):
    import os
    from importlib.util import module_from_spec, spec_from_file_location

    spec = spec_from_file_location("version", os.path.join(package_name, "_version.py"))
    module = module_from_spec(spec)
    spec.loader.exec_module(module)
    return module.__version__, module.get_cmdclass(pkg)

version, cmdclass = get_version_and_cmdclass(pkg)
setup(
    version=version,
    cmdclass=cmdclass,
)

Is it not the case that you can do something similar with a setting if you have setup.py and with setup.cfg ?


Nomai text line still visible in village snowman

I'll put there some other text that explain why there's the nomai wires and nomai cairn.


test Array: Remove too big size test

On large memory machines, if machine have enough memory,
tests failed because it doesn't return null.

@junsu testInitializer -> REQUIRE(Initializer::init(CanvasEngine::Sw, -1) == Result::FailedAllocation);
I think it's the same issue

@mgrudzinska please see this: https://github.com/Samsung/thorvg/pull/1016


Geometric Distortions

When using the triangulation, we observe strong geometric distortions for some data sets. This is particularly the case with official LOD2 datasets. These distortions can usually be avoided by skipping the triangulation.
The following graphic gives an example of these distortions

Experiments have shown that a translation of the CityGML data set can lead to an improvement of the results. However this is not the case for all data sets with this issue.
An examination of the CityGML data sets has shown that the error occurs with data sets in which the coordinates only have 2-3 decimal places.

The issue was a problem with the surface normal and plane parameter calculation.


Rect.Decoration and NetGraph

Here's how it looks

clip = True can eat the bottom and top part, but we lose a good chunk of the Net.Graph display
I guess it's more an issue of resizing the widget to the size of Rect.Decoration?

Closed as margin_y can fixed it.


L_PREFER_CANVAS and display vector object

Hi,
I have web application with AngularJs, Leaflet 0.x.x and Postgres/Symfony2 server side for GeoJson vector data. I'm going to work to export part of map (tiles and vector) in pdf, so i use leaflet-image plugin. After a lot of tests, i have this issue.

if i put L_PREFER_CANVAS = true; in script tag in my html file that manage all my application, polygons and linestring vector object are in canvas format (not markers) in map and when i click on button to fire to export image from map all objects are displayed good in image.

if i put L_PREFER_CANVAS = true; in script tag in map html, polygons and linestring vector object are displayed good (i think vector object are not in canvas format) in map (markers also), but when i click on button to fire to export image from map, i have error message in chrome debug console :
leaflet-image.js:185 Element could not be drawn on canvas <svg class=" leaflet-zoom-animated" width="476" height="357" viewBox="-38 -29 476 357" style="transform: translate3d(-38px, -29px, 0px);">…handlePathRoot @ leaflet-image.js:185start @ leaflet-image.js:307poke @ leaflet-image.js:295(anonymous function) @ leaflet-image.js:323tileQueueFinish @ leaflet-image.js:162maybeNotify @ leaflet-image.js:348(anonymous function) @ leaflet-image.js:324im.onload @ leaflet-image.js:138 .....

Can you help me
Thanks.
Christophe

Yep, L_PREFER_CANVAS must be set before Leaflet or Mapbox.js is included, hence this problem.

Thanks,
but why i have polygons and linestring vectors layers in block canvas and points layers in map.
I tried with simple example, that is all in html file, tile layer and polygon vector layer with L_PREFER_CANVAS=true. Polygon is displayed in map and also in image created by leaflet-image.
Do you have an idea on this issue ?
I'm going to simplify my code lines for not having vectors layers in canvas object on map.
Thanks
Christophe

Hi,
I solved my problem. I had a canvas tag with attribute in my css file, therefore i removed this tag and it's ok.
Other problem resolved. My point vector are leaflet icon. In leaflet-image i had an error message : 'not read property 0 of undefined'. leaflet-image could not get iconSize and iconAnchor properties in handleMarkerLayer function. I added iconSize and iconAnchor properties in my leaflet icon (example : iconSize: [20, 20], iconAnchor: [10, 10])
Leaflet-image is a very good plugin allowing me to get image from map with IGN Geoportail tiles and vector layers (point, linestring and polygon), that will be cropped and inserted in PDF file with pdfmake.
Thanks
Christophe