[opendistro-1.2][backport] Add validation for null elements in JSON array (#1157)

(cherry picked from commit 960d4209282027adedc0b37b28d74df8913dfafe)
opensearch-security pull request intake form
Please provide as much details as possible to get feedback/acceptance on your PR quickly

Category: (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Github Issue # or road-map entry, if available:

Description of changes:

Why these changes are required?

What is the old behavior before changes and new behavior after changes? (Please add any example/logs/screen-shot if available)

Testing done: (Please provide details of testing done: Unit testing, integration testing and manual testing)

TO-DOs, if any: (Please describe pending items and provide Github issues# for each of them)

Is it backport from main branch? (If yes, please add backport PR # and commits #)

By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1350 (c366f80) into opendistro-1.2 (8f92f50) will increase coverage by 0.04%.
The diff coverage is 88.23%.

@@                 Coverage Diff                  @@
##             opendistro-1.2    #1350      +/-   ##
====================================================
+ Coverage             62.57%   62.62%   +0.04%
- Complexity             2904     2914      +10
====================================================
  Files                   224      224
  Lines                 15967    15991      +24
  Branches               2996     3004       +8
====================================================
+ Hits                   9992    10015      +23
+ Misses                 4419     4418       -1
- Partials               1556     1558       +2

Impacted Files
Coverage Δ

...arch/security/dlic/rest/api/AbstractApiAction.java
76.21% <60.00%> (+1.32%)
:arrow_up:

...est/validation/AbstractConfigurationValidator.java
80.82% <100.00%> (+2.25%)
:arrow_up:

...csearch/security/auditlog/sink/InternalESSink.java
69.23% <0.00%> (-11.54%)
:arrow_down:

.../dlic/auth/ldap2/LDAPConnectionFactoryFactory.java
57.25% <0.00%> (+0.76%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d8920e...c366f80. Read the comment docs.


Broken links

The forensicswiki links are broken and the site is moving. More info here:
https://osdfir.blogspot.com/2022/11/transitioning-forensics-wiki-to-github.html

Thanks @Beercow all links are updated now.


containsWildcards always returns false

Coverage remained the same at 65.12% when pulling<PHONE_NUMBER>cd7197bd6758e8273061c19f1b167c on janisz:patch-1 into 6d86abe7b4fe8d4fb5c7acabfc8e4a38a5023701 on fstab:master.

Nice catch. Thanks!


(bug) Panic when call getVariation with rules evaluated in parallel

Observed behaviour
We've encountered panic when http requests come in parallel and retrieving flag at the same time.

Expected Behavior
Retrieve flag usecase should be concurrent safe
Steps to reproduce
Just make a test and run it in parallel. Or you also can reproduce it via http load test tools with the code should be calling any function that trying to retrieve flag with rules successfully evaluated.

This is issue also causing the relay proxy to panic when trying to serialize flag value from allFlags and flagEval handler.
The root cause is because of same reference of metadata (map), and also the global context enrichment.


Remove invalid trailing comma in ARM template

Remove invalid trailing comma in ARM template sample for Log Analytics workspace.

@BrianTJackett : Thanks for your contribution! The author, @mgoedtel, has been notified to review your proposed change.

@BrianTJackett - Thanks for making the time to submit this feedback for consideration.  I've reviewed and agree the extra comma in the template needs to be removed.  #sign-off


Save variants

Couple of issues which could be broken into separate issues if desired. I generated these comments working from the demo data.
Saving vcf

Mark the RAI1 variant as significant and add a note
Go to export variants and select VCF. Click Save and there is an error

What is undefined?
Saving csv
Same steps as above, but save as csv. After you click save, we get notified that variants are being saved, and then I'm returned to this modal:

Save has been replaced by Download File, and I have to click this again to save the file. Why are there 2 steps? Surely when I click save, I should be asked where to save the file and the file is saved. This is not intuitive.
Import variants

Refresh gene
Use the file selector to choose the demo data files
gene will open and highlight that you need to add genes
Import variants and select the csv from above

The RAI1 variant has the significant flag and the added note. There are also a couple of variants that I didn't add significance to in PDHA1 and PDGFB. But in the genes tab, there is only one gene (RAI1). It needs to be clear what is being exported in the steps above (all variants, all variants in one of the filter categories, only variants that have been reviewed?) and the genes/variants tabs need to be consistent (e.g. if we have variants from 3 genes, there need to be 3 genes).

There is another open issue on testing saving


fix(vite): fix hmr issue in vite v1.0.5 and later

resolve https://github.com/kuma-ui/kuma-ui/issues/273
I have successfully resolved the Hot Module Replacement (HMR) issue that was occurring in Vite versions v1.0.5 and later when using Kuma UI. The issue was related to CSS styles not being applied correctly during HMR.
Changes Made:

Updated the Vite plugin to ensure proper CSS application during HMR.
Added a hash to the imported CSS URL to ensure cache busting.

This PR addresses the problem reported in issue https://github.com/kuma-ui/kuma-ui/issues/273, which was caused by changes introduced in PR https://github.com/kuma-ui/kuma-ui/pull/231. The issue was traced back to the way CSS was being handled during HMR.

I confirmed that HMR is functioning correctly after installing the version of the Vite plugin for this pull request as follows in the repository where the issue provided by @nojiritakeshi is reproduced. Thank you very much @nojiritakeshi
pnpm i @kuma-ui/vite<EMAIL_ADDRESS>


Better error handling for adding Auth Sources

Description
this line could get an internal error: https://github.com/go-gitea/gitea/blob/54e9ee37a7a301dbe74d46fd3c87712e6120e9bf/routers/web/admin/auths.go#L295
if for example, adding a new OpenID config and the openid auto discovery URL returns unexpected response.
Maybe inspect error and improve the UI form error render to avoid confusion.
Screenshots
I already fixed the URL and didn't capture a screenshot but it's basically the 500 error page.
Gitea Version
1.16.7
Can you reproduce the bug on the Gitea demo site?
Yes
Operating System
No response
Browser Version
Latest Chrome.

It really is much much much much better to give us logs.

sorry I lost the logs inside the container, thought it was a general error checking. but anyways it's now working, thanks!

Closed as outdated.


net: never check IPv4-on-IPv6 map support on OpenBSD

Certain operating systems may refuse to implement IPv4 addresses
mapped to the IPv6 address space, and a runtime check for this support
can be avoided.  OpenBSD is one such OS, and others may exist as well.

@sthen fyi


[CL]: Attack vector repro to drain LP funds from any CL pool

Closes: #XXX
What is the purpose of the change
This PR demonstrates an attack vector to drain LP funds from a pool.
Testing and Verifying
N/A
Documentation and Release Note

[ ] Does this pull request introduce a new feature or user-facing behavior changes?
[ ] Changelog entry added to Unreleased section of CHANGELOG.md?

Where is the change documented?

[ ] Specification (x/{module}/README.md)
[ ] Osmosis documentation site
[ ] Code comments?
[ ] N/A

Closing since this was fixed by #5541


Add Skydio to Adopters list

Thank you for the contribution. Please sign Adobe CLA before we will be able to merge the change.

Ah, unfortunately I don't have the time to review that, I thought a simple readme addition promoting our use of this tool wouldn't require it. Feel free to add us if you wish.


Adjust existing position with newly entered position when "Add options to Trade" is used.

I am selling 5 puts.
I close 1 put via close leg. Remain -4.
If I now select Add Options to Trade and buy 2 puts, these are not deducted from the current position. Instead, a long position is opened.

It might make sense if it is the same strike, has the same expiration date and there are enough units available.

Ok, I understand what you are doing. The program does not currently have that logic built in where it determines that you buying long Puts to essentially Buy To Close (BTC) part of your existing position. As you know, it would be better to use the "Close Leg" functionality and close 2 of your 4 Puts, however, I do need to build your buying long puts scenario into the program. Thanks for the example. I will do my best to incorporate this logic into the program.

Can you perhaps also add a closing of the entire position as with a leg? Maybe in the menu like in the picture.

Sounds good to me. I should be able to add that option pretty easily.

Okay, I have added your suggestion to the popup menu. It was not as simple as it sounded because the Trade could involve BOTH shares/futures AND options, so simply having a "Close Trade" menu option would not work. In situations where there only exists just SHARES, or just FUTURES, or just OPTIONS, then I display the "Close Trade" option. In mixed case situations I display multiple "close" menu options.


dma::WriteDma::write is unsound.

The trait Static doesn't provide the invariants necessary for a buffer to be safe to use with DMA, it actually doesn't provide any invariant at all, since it's public and safe to implement.
The quickest solution would be to mark the trait as unsafe and document it. But a better solution would be to move this API to use the embedded-dma traits.
This is all breaking changes and we should also discuss if we want to yank previous versions with this problem.

Thanks for the report, is this the same unsoundness that was discussed in https://github.com/rust-embedded/embedonomicon/issues/64? I'm not sure what the best practice for DMA unsoundness is, I imagine not a whole lot of people use DMA so maybe yanking things causes more harm than good

Thanks for helping me resolve this @thalesfragoso :)

I imagine not a whole lot of people use DMA

@TheZoq2 I just wanted to mention that the SPI DMA is especially useful for working with large numbers of LEDs which I'd imagine is not an uncommon use case (this is our use case anyway). But I do agree that yanking might cause more harm than good either way.

Oh yea, I use the SPI DMA for just that myself :)
I haven't had time to read up on the specifics here, but I'd say wether or not to yank depends on how hard it is to run into this unsoundness. If it happens every time you use DMA, then we should probably yank, otherwise I guess it would depend on the severity

Good point! We spent a couple of days investigating noisey SPI LEDs on our PCB prototype, trying to work out what was going on by using an oscilloscope to check all the clocks and data lines but could not find the issue. It turns out that the unsoundness somehow lead to an issue where writing to the SPI DMA would cause packet loss in our Ethernet, and similarly Ethernet activity would result in the LEDs lighting up in seemingly random ways.
Here's the story I posted to the stm32-rs matrix last night for the record:

We seem to have found the source of the noisey LED behaviour on our PCB and it turns out to be pretty strange! We noticed that our LEDs lit up seemingly randomly whenever there was Ethernet activity. E.g. we could send a message from my laptop to the board once per second, and very reliably a random group of LEDs would light up. For some context, these are ~150 SPI addressable LEDs.
We started checking all the clock, Ethernet and SPI pins across the whole board with an oscilloscope expecting to see a noisey line, but alas everything looked very clean
Even the SPI lines both at the MCU and at the other end of the 20 mm cable looked really clean, though we noticed that there was some odd 1s on the SPI line even though we were just writing zeroes. I figured it was just the end frame which expects some Fs, but then remembered that around the same time we received our new PCBs was also around the same time I got the SPI3 DMA working for the F107
Turns out that if I write to SPI3 Synchronously rather than via the DMA, the LED noise goes away!
Not only does it go away, it also solved an odd issue that we were running into where our MCU would drop lots of TCP packets while running the main firmware ( which uses SPI3 ), but would work flawlessly  in the bootloader
Seems there is some funky interaction between the SPI3 DMA and the Ethernet DMA on the F107?
/ story time

Following this, @thalesfragoso dug into the issue with me and we discovered that my DmaBuffer struct was the cause of the issues. The type was a wrapper around a heapless::Vec<u8, U1024> that implemented the necessary traits for use as a DMA buffer including Static. Seeing as no unsafe was required, it didn't occur to me it would lead to any issues, and while I did think Static was an odd name I thought it just implied that the buffer should be owned as it was very similar to the std Borrow trait and the buffer was passed by value to the WriteDma::write.
After removing this type in favour of using a &'static mut [u8], all of our strange LED noise issues are gone and we no longer appear to get the Ethernet packet loss that we were getting before!

 I'll try and have a look at this tomorrow evening
Sidenote: Maybe that could explain why the backlight on one half of my keyboard flickers while pressing keys on the other half :thinking:

I wonder if this could be addressed (in stm32f1xx-hal at least) by requiring that the Static trait's borrow method returns a &'static B rather than &B?

A friend of mine also suggested just that. I've got to admit I'm a bit out of my comfort zone with DMA soundness issues, but that might work.
I think a better solution is to just use embedded-dma as suggested

I wonder if this could be addressed (in stm32f1xx-hal at least) by requiring that the Static trait's borrow method returns a &'static B rather than &B?

I think that would make things very inflexible, the new embedded-dma API is a better solution imo, and I don't think it would be that hard to change to it. I might do a PR in the coming days (not sure if before the weekend though).

Yea, I agree that we should use embedded-dma.

I might do a PR in the coming days (not sure if before the weekend though).

Sounds good, I could do it but I think you have more DMA experience than me :) Though let me know if you can't find the time, it feels like we should try to get this done fairly quickly given the severity :D


Google Meet not working

Bug description:

When trying to join a new meeting in Google Meet, I get an error saying that the video cannot be started due to an error.
Steps to reproduce:

Add a Google Meet service.
Login with your Google account.
Click on the button to join a meeting, leave the room name empty and click on continue.
See the error.

Expected behavior

A meeting room should be joined.
Screenshots

Environment:

Operating System: Fedora 32

Ferdi Version: 5.5.0

Server: Ferdi

Debug information:

https://debug.getferdi.com/65461606-d5eb-4223-bb3d-2832377498f9

@yann-soubeyrand - A fix has been put up in the recipes for this issue. Until then open your Google Meet recipe's index.js and replace it with the following piece of code and restart Ferdi
var os = require("os");

module.exports = (Franz) =>
  class Gmail extends Franz {
    modifyRequestHeaders() {
      return [
        {
          headers: {
            "user-agent": window.navigator.userAgent.replace(/(Ferdi|Electron)\/\S+ \([^)]+\)/g, "").trim(),
          },
          requestFilters: {
            urls: ["*://*/*"],
          },
        },
      ];
    }
  };

Hi @mahadevans87, thanks for your answer. However it doesn't work for me, but adding .replace(/Ubuntu/, "Fedora") before .trim() (I'm on Fedora) does the trick.

@mahadevans87 Btw, what is the difference between using modifyRequestHeaders this way vs overrideUserAgent?

@kris7t  overrideUserAgent is called on a webview when the window initially loads up. But modifyRequestHeaders will send the user-agent to each and every request that we specify in the requestFilters

@mahadevans87 the following is working for me (not sure which is better between overrideUserAgent and modifyRequestHeaders but based on what you said, I guess the former doesn't imply modifying each request and is thus better)
var os = require('os')

module.exports = Franz =>
  class Gmail extends Franz {
    overrideUserAgent() {
        return window.navigator.userAgent.replace(/(Ferdi|Electron)\/\S+ \([^)]+\)/g, "").replace(/Ubuntu/, "Fedora").trim();
    }
  };

Also on Ubuntu 20.04 LTS is not working with the same behavior.

Hi there. I have the same here and the recipe solution didn't work. I'm running the appimage release of Ferdi 5.5.0 on Arch Linux.

Hi there. I have the same here and the recipe solution didn't work. I'm running the appimage release of Ferdi 5.5.0 on Arch Linux.

Add an alert(window.navigator.userAgent) before return in overrideUserAgent methode.
Replace ubunthu with your OS. (use return window.navigator.userAgent.replace(/(Ferdi|Electron)\/\S+ \([^)]+\)/g, "").replace(/Ubuntu/, <your Os here>).trim();
Restart Ferdi

Work for me, i'm on Manjaro (Arch Linux base)

@philwronski Thank you, but I couldn't make it work yet. My index.js is a little different than the examples here so I couldn't figure exactly where to put those. I tried replacing the content with your accretions over what's been posted by @yann-soubeyrand and @mahadevans87 but didn't work neither. Here's how my index.js looks like originally:
var os = require('os')

module.exports = Franz =>
  class Gmail extends Franz {
    overrideUserAgent() {
      if (os.platform() == 'linux')
        return "Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0"
      else
        return "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0";
    }
  };

Anyway, thanks for helping!

@philwronski Thank you, but I couldn't make it work yet. My index.js is a little different than the examples here so I couldn't figure out if I was putting those exactly where they should be. I tried replacing the content with your accretions over what's been posted by @yann-soubeyrand and @mahadevans87 but didn't work neither. Here's how my index.js looks like originally:
var os = require('os')

module.exports = Franz =>
  class Gmail extends Franz {
    overrideUserAgent() {
      if (os.platform() == 'linux')
        return "Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0"
      else
        return "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0";
    }
  };

Anyway, thanks for helping!

replace return in if.

I'm having the same issue on macOS 10.15.5. How do I edit a recipe?

Thank you, @philwronski it finally worked for me!
@akira28

How do I edit a recipe?

At Ferdi, go to "available services" and them to the "custom services" tab. You'll see a field with the recipes directory, where you'll see the googlemeet folder and the index.js file inside it. This is the one you'll edit.
You may want to copy that folder inside the dev folder, so your modified Google Meet will be treated as a custom service (separate from the "original").

Still happening

This is what I did for MacOS to make it work on 5.6.0-beta2
var os = require('os')

module.exports = Franz =>
  class Gmail extends Franz {
    overrideUserAgent() {
    	return "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36";
    }
 };

@yann-soubeyrand - A fix has been put up in the recipes for this issue. Until then open your Google Meet recipe's index.js and replace it with the following piece of code and restart Ferdi

Thank you.
This worked for me. 👍🏻

Please try with the nightly builds from here - this exposes the ability to the user to control the user-agent via the UI as opposed to editing files in the Ferdi profile. Please do be aware of the unsigned nature of the builds in this repo - while we work on fixing the issue with signing of our build artifacts.

It seems this issue is fixed by using custom user-agent strings.

I try change user-agent in index.js but it don't work.
module.exports = Ferdi =>   class GoogleMeet extends Ferdi {     overrideUserAgent() {       return "Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36.0 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36.0";     }   };
or return window.navigator.userAgent.replace(/(Ferdi|Electron)\/\S+ \([^)]+\)/g, "").trim();


Electron is too slow to download

electron-packager must download electron framework form GitHub every time when I'm packaging an app.
BUT THE PROBLEM IS, it is so slow to connect the GitHub CDN (githubusercontent) in China, which is not convenient.
It looks like this:
Downloading electron-v8.2.5-darwin-x64.zip: [============----------------------------------------------------------------------------------------] 12% ETA: 7258.9 seconds

The only way to connect githubusercontent in China is to change the electron repo url to a GitHub mirror URL. So how can I change it? Or, can I use a local folder as the electron framework repo.?

This isn't a problem with Electron Packager itself. The Electron documentation indicates how to change the download location for folks in China.

I'm NOT DOWNLOADING ELEECTRON FROM NPM!
I mean: because I add electron in the package.json dependencies, so electron-packager is default downloading the electron package from GitHub (not from other mirrors), every time when I'm packing an app, and you cannot change it's mirror or download link.

Both Electron Packager and the electron npm package use the same mechanism for downloading the prebuilt binaries (by design), so the environment variables will work the same.

Both Electron Packager and the electron npm package use the same mechanism for downloading the prebuilt binaries (by design), so the environment variables will work the same.

Do you mean that electron-packager is using npm commands to download the dependencies and devDependencies in package.json?

No, they both use the @electron/get node module to download the Electron prebuilt binaries, which is where the environment variables come into play.


improve createResource types

JSDoc comments for better IDE integration

Exporting the internal types of createResource makes extending it by tertiary primitives (e.g. https://github.com/davedbase/solid-primitives/tree/main/packages/fetch) easier.
JSDoc comments allow us to improve the developer experience by helping them directly in their IDE. To try it, hover over a function signature e.g. of createSignal or createResource in visual studio code or similarly advanced IDEs.

Any particular reason as to why initialValue has a different type from the success value?
There's also some potential Resource type change to handle idle state (source = null), please check this issue thread.

I'm pretty stoked to have the JSDocs in.
TypeScript need to look at a bit more to understand. I know work was done to make sure that the type could be inferred correctly from initial value. And it's important that resources that don't start with an initial value are nullable, since it is possible for them to not have a value. That being said it going null afterwards I'm fine being explicit about in end user types and not including here as it would ever go null unless you set the source that way. But just want to ensure that is retained, the type inference and the fact when not initially set it can be nullable.

Looking primarily at the createResource type changes

I think it makes sense to create named types for the argument and return types. Like you said this will make it easier for someone to build on top.
I see little value in adding a second generic to ResourceReturn and Resource to specify the initial value type. In Resource they ultimately get combined in a union for the function return type and the individual types are never needed separately. Therefore, the upstream type user can just pass the type union in as a single generic.
Same question as @LXSMNSYC, do we want to allow the initial value to have a different type as the fetcher return type outside of the undefined initialValue case?

Thanks for your feedback. While the idea was that I either extends T or T | undefined, this does not seem to work as intended. I have refactored the types to fix the failing cases and removed ResourceReturn as you suggested.

I actually thought ResourceReturn was fine to keep, I just wanted to remove the second generic like you have done with Resource. It might make sense to at codify the actions object to make it easier for things to build on top and tighten the other definitions.
interface ResourceActions<T> { mutate: Setter<T>; refetch: () => void; }

I think createResource can be simplified from 6 overloads down to 4 be combining the cases where options is defined but initialValue is not defined and where options is not defined. This means overload 1 is removed and 3 has its options parameter made optional with a ?. Similarly overload 4 is removed and 6 has options made optional. As far as I can tell we don't lose anything by doing this.

Thanks again for your time and your valuable feedback. I have implemented all the suggested changes. Yes, it makes sense to bundle the ResourceActions and this makes it easier to extend, too.

I'll keep the return types as is, because you'll see easier that the result is an array then.

@atk Your latest updates look good to me. Those test-cases I provided are all working as expected and the types seem succinct. From a createResource types standpoint, I think this is in a good state to merge now.

Hmmm I was thinking, can we consider this? https://github.com/solidjs/solid/issues/608

@LXSMNSYC let's not draw this PR out even longer, shall we? Incremental improvements is the way to go.
@rturnq We forgot about fixed generics.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

12 of 16 (75.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.005%) to 90.0%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

packages/solid/src/reactive/signal.ts
12
16
75.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.005%

Covered Lines:
1109

Relevant Lines:
1178

💛  - Coveralls

Thank you so much. Thank you for all the effort put in here.


Add GPTQ quantization

Describe your changes
Implemented GPTQ quantization utilizing Hugging Face Optimum and packed quant linear with ONNXRuntime-supported format to enable the latest 4-bit kernel (MatmulNBits), reducing quantization time to half an hour compared to several hours in INC.
Tested 4-bit quantization on winograde and humaneval with llama2 7b.

Model
Winogrande
Humaneval

Baseline
49.64
12.80

GPTQ 4bit
49.17
12.20

All unittest passed.
Checklist before requesting a review

[x] Add unit tests for this change.
[ ] Make sure all tests can pass.
[x] Update documents if necessary.
[x] Lint and apply fixes to your code by running lintrunner -a
[x] Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

(Optional) Issue link

/azp run

https://github.com/microsoft/Olive/commit/c093dd94628510a21e183ff0cf313f03124b13ab
Could you help also update the docs to tell this new pass? Please refer to above PR.

/azp run

/azp run

/azp run

/azp run

I am tring to using PytorchModelHandler to suport more general input models, leaving the model check to this class. And also use olive data loader to support custom dataset, it may take one or two days. Thanks for your patience.

/azp run

/azp run

/azp run

/azp run

/azp run

@jambayk I've made some changes to remove optimum dependencies and use optimum_conversion pass to do export. GPTQ pass would save pytorch model like TorchTRTConversion pass. Could you help me review these changes?


Parsing error

Describe the bug
Hi, I'm developing a GitHub action based on your library.
But I'm receiving a parse error in the following line:
<?php
$custom = new self::$allowed_providers[ $source ]['class']();

Error: Parse Error : syntax error, unexpected '[', expecting ';' on line 131 | Unable to parse ../widgets/proxio-sso-app-widget.php

Module versions
18:17 $ npm ls wp-pot php-parser
<EMAIL_ADDRESS>/Users/breno-proxio/action-wp-pot
└─┬<EMAIL_ADDRESS>  └──<EMAIL_ADDRESS>

Thanks for your report, it seems to be a known issue in php-parser. Please follow https://github.com/glayzzle/php-parser/issues/589 for updates. I will release a new version of wp-pot when this issue is resolved.

Thanks @rasmusbe


Can now run Mocha tests from the browser.

Added the possibility to run (and debug!) JS Mocha tests from the browser.
H/T https://stackoverflow.com/questions/33080182/how-do-i-get-node-js-mocha-to-run-test-in-browser.
This really is the easiest way I found to look into evaluating Rego's IR interactively, step by step.

Forgot to paste the run instructions:
mkdir clean
cd clean
git clone -b mocha_in_chrome git@github.com:dkorolev/asbyrgi.git
cd asbyrgi
./scripts/gen_all_goldens.sh && ./scripts/gen_all_js.sh && ./scripts/gen_all_js_tests.sh
firefox tests/mocha.html

No node_modules / npm i required, and git log is clean after this.


How to know, if the consumer is idle for x amount of time(using c#). I have a task I need to perform. If the consumer is idle for 2 minutes and then insert some data in DB.

Description
How to know, if the consumer is idle for x amount of time(using c#). I have a task I need to perform. If the consumer is idle for 2 minutes and then insert some data in DB.
How to reproduce
Not a problem, just asking if we have seetings imilar tp python in c# like "consumer_timeout_ms = 5000"
Checklist
Please provide the following information:

[ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
ConsumerConfig consumerConfig = new ConsumerConfig
{
GroupId = "Consumer23" + _kafkaOptions.GroupIdSuffix,
ClientId = _kafkaOptions.ClientId,
ApiVersionRequest = true,
BootstrapServers = _kafkaOptions.Authenticated.Brokers,
SecurityProtocol = SecurityProtocol.SaslSsl,
AutoOffsetReset = AutoOffsetReset.Earliest,
SaslKerberosServiceName = _kafkaOptions.Authenticated.KerberosServiceName
};
     if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
     {
         consumerConfig.SaslMechanism = SaslMechanism.Gssapi;
         consumerConfig.SslCaLocation = _kafkaOptions.Authenticated.CaFilePath;
     }
     else
     {
         // Linux and MacOS
         consumerConfig.SaslKerberosKeytab = _kafkaOptions.Authenticated.KeyTabFilePath;
         consumerConfig.SaslKerberosPrincipal = _kafkaOptions.Authenticated.Principal;
     }

[ ] Confluent.Kafka nuget version.
1.2.1

[ ] Apache Kafka version.
1.1.1

[ ] Client configuration. -

provided in description

[ ] Operating system. - Windows
[ ] Provide logs (with "debug" : "..." as necessary in configuration).

provided in description

[ ] Provide broker log excerpts.

provided in description

[ ] Critical issue.

provided in description

You could always provide a TimeSpan of 2 minutes to the .Consume() method call.  Then if the message is null, that should indicate that the consumer hasn't processed anything in the last 2 minutes.

Thanks for your reply @caj-johnson , but that sounds like polling right. It checks for the message every 2 minutes, right?
How different between this two:
consumer.Consume(TimeSpan.FromMinutes(2));
consumer.Consume(cancellationToken);
The second consume option will block and wait for messages to come in right?

The second one you have to manually cancel it.
What I'm suggesting is:
var msg = consumer.Consume(TimeSpan.FromMinutes(2));
if (msg == null)
{
  InsertDataIntoDB();
}

The only polling is the polling happening internal to the consumer.

Thanks @caj-johnson  !!


Fix the stop-flag setting: no arguments are expected

Issue: #142, introduced in #147

Description
Sadly so, in #147, an error sneaked into the code:
Event.set() does not accept any arguments, but they are passed into a callback. The callback was originally implemented as a separate async-function, which took a list of tasks and cancelled all of them. Then, the callback was replaced to a flag-setting. But the argument with a list of tasks went unnoticed.
This only affects the operators on its termination stage (Ctrl+C or pod deletion/restarting), and does not affect the behaviour itself.
Types of Changes

Bug fix (non-breaking change which fixes an issue)

Codecov Report

Merging #151 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master     #151   +/-   ##
=======================================
  Coverage   82.56%   82.56%
=======================================
  Files          30       30
  Lines        1572     1572
  Branches      229      229
=======================================
  Hits         1298     1298
  Misses        229      229
  Partials       45       45

Impacted Files
Coverage Δ

kopf/reactor/queueing.py
56% <0%> (ø)
:arrow_up:


10442 response csv export modal

Spec failure due to \n vs \r\n in fixture file, I fixed it locally and committed but I think git automatically converted it. Will look into better conversion tomorrow.
  1) Results::Csv::Generator with long_text_behavior exports truncated values
     Failure/Error: is_expected.to match_user_facing_csv(prepare_response_csv_expectation("truncated_values.csv"))
       expected "ResponseID,Shortcode,Form,Submitter,DateSubmitted,Reviewed,GroupName,GroupLevel,TextQ1,TextQ2\r\n7b...866-31d4612fca00,vv-oqv-ql8g2,Sample Form 1,A User 1,2015-11-20 06:30:00-06,false,,0,foo,foo ba\r\n"
       to match user facing csv "ResponseID,Shortcode,Form,Submitter,DateSubmitted,Reviewed,GroupName,GroupLevel,TextQ1,TextQ2\n7b8ad...-9866-31d4612fca00,vv-oqv-ql8g2,Sample Form 1,A User 1,2015-11-20 06:30:00-06,false,,0,foo,foo ba\n"
       Diff:
       @@ -1,3 +1,3 @@
       -ResponseID,Shortcode,Form,Submitter,DateSubmitted,Reviewed,GroupName,GroupLevel,TextQ1,TextQ2
       +ResponseID,Shortcode,Form,Submitter,DateSubmitted,Reviewed,GroupName,GroupLevel,TextQ1,TextQ2
        7b8ad3ee-fe20-4262-9866-31d4612fca00,vv-oqv-ql8g2,Sample Form 1,A User 1,2015-11-20 06:30:00-06,false,,0,foo,foo ba
     # ./spec/models/results/csv/generator_spec.rb:315:in `block (3 levels) in <top (required)>'


Refactoring

Hi there
I was (once again) reading some code here and there then noticed clippy issues + some duplication.
So I decided to do something
Take care.

So are you happy if I merge this and then do the fail threshold in another PR? :eyes: Just wanted to make sure you're not working on it before I hit merge

Initially I did not planned to do it.
And I was typing a corresponding response here.
But then I tried.....
And didn't managed that :disappointed:
So you can take over :)

A note:
Could you check this test https://github.com/zhiburt/tarpaulin/blob/1722d63a99a0017bc07babab75b338f471c4c9bb/tests/failures.rs#L88-L104?
I run 2 configs guessing that 2 of them must be run.
But I see only output of 1 config.
I am not very versed in the code base and did not investigate things deeply.

So I see the output of both configs, and if I remove the failing config the test fails, if I remove the good config it passes. I'll merge this though, issue has been raised in https://github.com/xd009642/tarpaulin/issues/995

test do_nothing is in success, and test it_works are in failing. Both project tests are in there. And the type is an error type instead of reporting it and ending with success - that looks as expected

ah see you spotted that in an edit :sweat_smile:


Glyph of Subjugate Demon - Warlock - Causing import issues.

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to https://wowsims.github.io/wotlk/warlock/
Click on Import
Use Import String {"talents":"2350002030023510253500331151-000000000000000000000000000-55000005010000000000000000","glyphs":{"major":["Glyph of Haunt","Glyph of Life Tap","Glyph of Quick Decay"],"minor":["Glyph of Unending Breath","Glyph of Subjugate Demon","Glyph of Drain Soul"]},"class":"warlock","race":"Gnome","name":"Bookofmormon","gear":{"items":[{"enchant":3820,"gems":[34220,40051],"id":39409},{"id":40680,"gems":[]},{"enchant":3810,"gems":[40051],"id":40424},{"enchant":3825,"gems":[],"id":40251},{"id":39497,"gems":[]},{"enchant":2332,"gems":[],"id":41908},{"id":40197,"gems":[]},{"id":40696,"gems":[40051,40051]},{"id":42002,"gems":[40050,41467]},{"id":39254,"gems":[]},{"id":40719,"gems":[]},{"id":39389,"gems":[]},{"id":40255,"gems":[]},{"id":40682,"gems":[]},{"enchant":3855,"gems":[],"id":37360},null,{"id":40284,"gems":[]}]},"professions":[{"name":"Enchanting","level":235},{"name":"Tailoring","level":250}],"level":80,"spec":"","realm":"Grobbulus"}
See error

Desktop (please complete the following information):

OS: Microsoft
Browser: Chrome
Version: [e.g. 22]

Additional context
Add any other context about the problem here.

To add to this, I removed the glyph and now I am getting error.

Temp fix is to rename it to "Enslave Demon" in import text.

Thank you for the response. I will do that! Luckily I was able to manually
input my gear using the pre-raid Bis list.
On Fri, Oct 21, 2022 at 3:27 AM Anipsy @.***> wrote:

Temp fix is to rename it to "Enslave Demon" in import text.
—
Reply to this email directly, view it on GitHub
https://github.com/wowsims/wotlk/issues/1371#issuecomment-1286763645,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A3WYVMUVCC3DWBJQEEQP6RDWEJVZDANCNFSM6AAAAAARJS4DU4
.
You are receiving this because you authored the thread.Message ID:
@.***>


Fix mixin's this context

Fix the issue reported by @Avnerus
See https://github.com/muut/riotjs/pull/694#issuecomment-100705684

@emehrkay I've extended extend method. I think there's no side effects, but it's so fundamental one. Pls review well. Or it would be also OK to make new method. How do you think?
https://github.com/cognitom/riotjs/blob/fix-mixin-this-context/lib/browser/tag/util.js#L17

Changes Unknown when pulling cd2684ee12f0f6576b44eeb75ba747fac4cac137 on cognitom:fix-mixin-this-context into ** on muut:dev**.

Maybe just do the extend inline rather than a new function?

Yeah, agreed. I'll fix it next morning. Thank you!

Changes Unknown when pulling 01a816c7681463fee26ae91d7f67ce5c825c11c1 on cognitom:fix-mixin-this-context into ** on muut:dev**.

Clean code. I often forget about bind being a native method now

@emehrkay thanks! me too, haha.

thanks @cognitom


Visualization of a Docmap Graph

It would be useful for any Docmap graph to have an ergonomic way of being displayed/viewed. It is easy to convert an RDF graph to a generic node visualization, but these are full of meaningless metadata, such as blank nodes and pieces of information that may be too verbose/not interesting. We would like to enable a highly readable, static image or interactive visualization of a DocMap that makes opinionated summarizaitions while being configurable.

off to a great start https://github.com/source-data/render-rev


fix(OpenAPI): YAML schema dump

Description
Fixes an issue in the OpenAPI YAML schema dump logic of OpenAPIController where the endpoint for the OpenAPI YAML schema file returns an empty response if a request has been made to the OpenAPI JSON schema previously due to an incorrect variable check.

Closes
Edit from @Alc-Alc: https://discord.com/channels/919193495116337154/919193495690936353/1246024918487793744 explains the need for this

does this need a test? if so how to test it? but its a go from me 🙂

Added a test for this in efa4fd1aa2e2cad282699349d4678714383be417 on the OpenAPIController, as the new system using OpenAPIRenderPlugin (JsonRenderPlugin and YamlRenderPlugin) does not have this issue.
I suspect #2634 is related but it got mistakenly closed due to the nature of this bug, so one more reason to add a test I guess :D

thanks for contributing again :)


rename FVT to custom in Dashboard

related: https://github.com/AdoptOpenJDK/openjdk-test-tools/issues/331
Signed-off-by: OscarQQ<EMAIL_ADDRESS>


Component/select

#10
Made some progress with the select component. Initial styling - border, basic layout, padding inside component. Work still to be done on drop down arrow styling.

Good progress! Before I recommend changes though can you target this pull request at the develop branch as opposed to main?

Good progress! Before I recommend changes though can you target this pull request at the develop branch as opposed to main?

All sorted!

All sorted!

Thank you Rob! I will look into all of these ahead of Wednesday.

Thank you Rob! I will look into all of these ahead of Wednesday.


Update Travis CI settings

@sebastienros @Jetski5822
Fixed build in Travis CI

Still failing?
Should we have our own scripts btw, because we are currently using the ones from the aspnet projects, but we don't need all the complexity.
For instance on appveyor I changed it to just be dotnet restore and dotnet build, knowing the I load a Visual Studio 2017 RC image first that contains dotnet core runtime and SDK.
I assume a simple script for travis would be to load the runtime/sdk manually, and do the same.
And run the tests also, this is something I forgot on appveyor.

At the moment don't pass tests. I should simplify logic of build? Maybe we need to configure a build through MSBuild?

Yes, we need to use msbuild, and also remove mono dependency.

Ok, I'll do it

Looks good so far, I can see some green on travis right now

Looks good so far, I can see some green on travis right now

I need to look at some moments of build


Invalid frames from animated gifs

When I try to resize the following animated gif, the resizer seems to read a wrong frame size and it looks like the image moves from top to bottom.
Source: https://cloudflare-ipfs.com/ipfs/QmSf3CvdGbKgfYTVpEkwDLLs1zsh1Tc2BJfb1UW4t4Mr4B
Resized: https://images.weserv.nl/?url=https://cloudflare-ipfs.com/ipfs/QmSf3CvdGbKgfYTVpEkwDLLs1zsh1Tc2BJfb1UW4t4Mr4B&h=370&w=370&fit=inside&we&q=50&n=-1&output=webp

Mmh I notice that it has been fixed in the live version but I still have this problem on my own server. Is there any branch with this fix?

Sorry for the delay, backlog was a bit large last week. This animated image reaches the limit of 256 pages, see:
$ vipsheader -f n-pages x.gif
300

I've increased it to 320 on the production server. But ideally in this case we should print an error instead of truncating the image (which does not work as you have noticed).
If you want to increase this limit on your server; try setting the weserv_max_pages directive in your nginx configuration. For example:
weserv_max_pages 320;

Thanks for your answer. Is there a list with all of these options documented somewhere?

It's documented here: https://github.com/weserv/images/tree/5.x/ngx_conf.

Commit https://github.com/weserv/images/commit/8f6d213c8baada23eb92c810b837a21c2aeca7a0 should ensure that an error is returned instead of the image being truncated. Thanks for reporting this!


[Common] Reduce duration of locks when processing items held in ConcurrentOpenHashMaps

Motivation
Currently ConcurrentOpenHashMap.forEach keeps a read lock in each section for the duration of the processing. This can lead to various issues when I/O operations are done in the processing function. This is a common patter in Pulsar code base. The I/O usually happens to Zookeeper and it leads to threads piling up in waiting for locks in ConcurrentOpenHashMap when there are both reads and writes for the same section in concurrently executing threads.
It is also possible that long living locks lead to dead locks.
This is the current implementation of ConcurrentOpenHashMap.forEach
https://github.com/apache/pulsar/blob/34ca8938ca13ea60b05d164c27d9755855caf87c/pulsar-common/src/main/java/org/apache/pulsar/common/util/collections/ConcurrentOpenHashMap.java#L157-L161
and the referenced Section.forEach
https://github.com/apache/pulsar/blob/34ca8938ca13ea60b05d164c27d9755855caf87c/pulsar-common/src/main/java/org/apache/pulsar/common/util/collections/ConcurrentOpenHashMap.java#L356-L395
Modifications

Add a new method ConcurrentOpenHashMap.forEachInSnapshot which makes a copy of the entries in each section before processing the entries.

    public void forEachInSnapshot(BiConsumer<? super K, ? super V> processor) {
        for (Section<K, V> s : sections) {
            List<Entry<K, V>> entries = new ArrayList<>(s.size);
            s.forEach((k, v) -> entries.add(new Entry<>(k, v)));
            entries.forEach(entry -> processor.accept(entry.key, entry.value));
        }
    }

Replace all current usages of forEach with forEachInSnapshot

There are tradeoffs involved. Some memory allocations will have to be made to make a copy. However, allocations aren't a real issue. JVMs are really efficient for such short time allocations and the allocations aren't causing new bottlenecks or performance issues.
The other tradeoff is a related to data consistency. The entry might have already been removed from the collection at the time it is processed. This is also a non-issue in most cases since it would be a bad solution in the first place to determine state based on the membership in the collection.
Since the previous forEach implementation remains, it's possible to choose between the usage of forEach or forEachInSnapshot in each specific use case. If this feature isn't needed, it is possible to replace the forEach implemention with the snapshotting implemention completely.

But I am not sure we can touch so many sensible parts of the codebase in one single patch.
We could break things without knowing and it will be hard to track down to the cause.
I don't know which is the best strategy to incorporate this work safely

that's a valid concern when a major change is made. However, the problems that the long living locks cause are nondeterministic and much hard to track down. One of the issues with StampedLocks used by ConcurrentOpenHashMap is that a dead lock doesn't get detected by the JVM. I made a thread dump of an experiment where I caused a dead lock by modifying the testDeadlockPreventionWithForEachInSnapshot to use forEach and a timeout (@Test(timeOut = 10000L)). I cherry-picked #9766 to get a standard thread dump with locks information. The StampedLocks information don't seem to be covered.
StampedLocks aren't re-entrant so that is yet another way how deadlocks can occur since a single thread can deadlock.

Here's an example of a suspicious stacktrace from one of the ReplicatorTest timeouts (full threaddump):
"ForkJoinPool.commonPool-worker-2" daemon prio=5 tid=503 in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1286)
        at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:367)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2170)
        at org.apache.pulsar.broker.service.BrokerService.lambda$null$62(BrokerService.java:1674)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$922/1818690322.accept(Unknown Source)
        at java.util.Optional.ifPresent(Optional.java:159)
        at org.apache.pulsar.broker.service.BrokerService.lambda$null$63(BrokerService.java:1674)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$921/148576384.accept(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
        at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683)
        at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010)
        at org.apache.pulsar.broker.service.BrokerService.lambda$onUpdate$64(BrokerService.java:1669)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$920/1723403002.accept(Unknown Source)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
        at org.apache.pulsar.broker.service.BrokerService.onUpdate(BrokerService.java:1665)
        at org.apache.pulsar.broker.service.BrokerService.onUpdate(BrokerService.java:175)
        at org.apache.pulsar.zookeeper.ZooKeeperDataCache.lambda$4(ZooKeeperDataCache.java:138)
        at org.apache.pulsar.zookeeper.ZooKeeperDataCache$$Lambda$538/1964134182.accept(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$18(ZooKeeperCache.java:386)
        at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$188/1103281887.accept(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$14(ZooKeeperCache.java:366)
        at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$198/880015051.run(Unknown Source)
        at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

whenever Policies get updated, this code will get executed:
https://github.com/apache/pulsar/blob/dbf29e95f7a4003bd905a7bd2ad13a5c46ae1a5e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L1659-L1682

There are tradeoffs involved. Some memory allocations will have to be made to make a copy. However, allocations aren't a real issue. JVMs are really efficient for such short time allocations and the allocations aren't causing new bottlenecks or performance issues.

The primary goal of having these custom concurrent maps implementation was actually to avoid memory allocations to the max possible extent :)
Even if these allocations are short lived, they still contribute to generate garbage that will fill up the new-gen space at a faster rate. That in turn will cause more "in-flight", reasonably short-lived objects to get caught in a collection event and copied over to old-gen spaces, where they'll have to get compacted later on.
On the class there are also already 2 methods, keys() and values(), that are taking copies, thought they were put there mostly to simplify unit tests.

The I/O usually happens to Zookeeper and it leads to threads piling up in waiting for locks in ConcurrentOpenHashMap when there are both reads and writes for the same section in concurrently executing threads.

We shouldn't be doing blocking calls from within the process method. I think we should rather refactor that code instead.

The primary goal of having these custom concurrent maps implementation was actually to avoid memory allocations to the max possible extent :)
Even if these allocations are short lived, they still contribute to generate garbage that will fill up the new-gen space at a faster rate. That in turn will cause more "in-flight", reasonably short-lived objects to get caught in a collection event and copied over to old-gen spaces, where they'll have to get compacted later on.

Yes, it makes sense to avoid memory allocations in general, but here there could be reason to make a compromise in order to make Pulsar more stable.
In this case, the memory allocated in forEachInSnapshot is a tiny fraction of all memory allocations happening. The ConcurrentOpenHashMap.forEach calls aren't on the fast path of processing where a lot of garbage could be generated. The amount of memory allocations are in the order of a few kilobytes per second since it's only about a few object references that consume memory (about 28 bytes per item: 16+4+4 bytes for Entry, 4 bytes for each item in the ArrayList). Is GC pressure really a problem in this case and how .forEach is used in the Pulsar code base?

We shouldn't be doing blocking calls from within the process method. I think we should rather refactor that code instead.

This is easier said than done. It's a major refactoring to refactor the code in that way. It won't happen quickly. There's plenty of code that does blocking calls within forEach or putIfAbsent methods. My assumption is that Zookeeper is involved in a lot of the cases.
After reviewing how the current forEach method relies on optimistic locking, I don't think there is risk from data consistency perspective in switching to use the forEachInSnapshot method for processing elements of ConcurrentOpenHashMaps. That was something that was my major concern initially that the behavior would be a lot different. It's no more a concern for me.
I have added a few signs of deadlocks from ReplicatorTest in comments of this PR. All such issues go away after applying the changes in this PR. That's why I'm confident that this PR is useful.
In the past, I have been investigating some problems in real environments where the symptoms observed from the thread dumps look similar to thread dumps from stalled ReplicatorTests. A certain set of problems seem to happen only in Pulsar deployments where there's a large amount of topics. When there's a lot of topics, the .forEach iterations take a long time.  There's a lot more changes that the nondeterministic deadlocks occur. That is something that the changes in this PR would prevent.
Perhaps we could evaluate each individual usage of .forEach, case-per-case, and decide whether to use .forEach or .forEachInSnapshot? When there's a change the the processing uses blocking methods, we would choose .forEachInSnapshot and if it's computation-only, .forEach would be used? @merlimat  WDYT?

/pulsarbot run-failure-checks

@lhotari
what about changing to the new method only in the cases that are relevant to the deadlocks you see in the test ?

what about changing to the new method only in the cases that are relevant to the deadlocks you see in the test ?

Why limit to the cases that happen to show up? Matteo commented earlier: "We shouldn't be doing blocking calls from within the process method. I think we should rather refactor that code instead." Based on this rule, it can be determined from the code whether a certain usage of .forEach needs addressing or not.
If it's possible to refactor the code to not use blocking calls, that should be done. Since it's really hard to get rid of the blocking calls in processing methods, it leaves the only choice of switching to use .forEachInSnapshot instead of .forEach.
Makes sense?

/pulsarbot run-failure-checks

In this case, the memory allocated in forEachInSnapshot is a tiny fraction of all memory allocations happening. The ConcurrentOpenHashMap.forEach calls aren't on the fast path of processing where a lot of garbage could be generated. The amount of memory allocations are in the order of a few kilobytes per second since it's only about a few object references that consume memory (about 28 bytes per item: 16+4+4 bytes for Entry, 4 bytes for each item in the ArrayList). Is GC pressure really a problem in this case and how .forEach is used in the Pulsar code base?

This map is used in many places throughout the code base and, depending on the use cases, there can be instances with a lot of entries or many instances with few entries. Also, the rate of this scanning through the map could be quite high in some cases.
In light of that, I'm not convinced that a model where we always defensively copy the entire map when we need to do a map scan is a good general option, or a good replacement in the current code where most places assumed the scanning not be expensive (allocations-wise), even though there might be few specific cases in which it's appropriate.
If we do have blocking in that code that process the map items, and that cannot be easily removed, I believe a better alternative would be to make the scanning more flexible.
For example, we're not expecting to do a scan over a consistent snapshot of the map (for the simple reason that the map is broken down into independent sections). We only need to maintain the consistency between a key and it's associated value.
With that in mind, we don't strictly need to get a read lock for the entire duration of scanning the whole section.
We could instead use different approaches:

Grab the stamp once per section and upgrade to read-lock only if there are write occurring
Do the same as a get, with stamp and fallback to read-lock per each item we're reading

Another alternative could be to measure the time spent in the process function and decide based on that wether to make a copy or keep the read lock during the scan.

This map is used in many places throughout the code base and, depending on the use cases, there can be instances with a lot of entries or many instances with few entries. Also, the rate of this scanning through the map could be quite high in some cases.

That's the reason why I have suggest that we evaluate each usage of ConcurrentOpenHashMap.forEach call and decide on a case-by-case basis how to fix the handling.

In light of that, I'm not convinced that a model where we always defensively copy the entire map when we need to do a map scan is a good general option, or a good replacement in the current code where most places assumed the scanning not be expensive (allocations-wise), even though there might be few specific cases in which it's appropriate.

It's not really a defensive copy of the entire map. The copy is done one section at a time. Memory allocations and GC is extremely efficient in modern JVMs. Since it's about 28 bytes per entry, there would have to have millions of entries for the memory allocation overhead to have any noticeable significance. If necessary, this could be benchmarked.
Since there aren't millions of entries, but usually a few thousand of entries, it has even less significance. This is why I don't see a reason to avoid the copying in this case.

If we do have blocking in that code that process the map items, and that cannot be easily removed,

We do have blocking in most of the code that processes the map items and it cannot be easily removed.

I believe a better alternative would be to make the scanning more flexible.
...
Grab the stamp once per section and upgrade to read-lock only if there are write occurring

The .forEach method in master branch is already implemented this way "Grab the stamp once per section and upgrade to read-lock only if there are write occurring".
It reduces the duration of the locks in the case that there are no writes. When there are writes, the .forEach processing will hold the read lock until all items have been processed. This is one of the reasons why the problems are rare with small number of items or low amount of write operations. However it doesn't help if the problems are rare, if the model is such that it could lead to a deadlock that isn't detected or resolved after some timeout. Currently the Pulsar broker locks up and doesn't resume operations if there's a deadlock. That's what makes the problem severe. Since the locks could be in specific sections of a ConcurrenOpenHashMap, it leads to partial unavailability which is hard to detect.
Improving Pulsar stability is high priority and this should be addressed. Please don't take me wrong that I would think that this PR resolves all such issues. I know that it will only resolve the issues that are caused by a dead lock which could be avoided by not locking at all when processing items of a ConcurrentOpenHashMap.
It would be interesting if @Vanlightly could take a look from a formal verification / modelling perspective on what would be practical in this case. @merlimat WDYT?

@lhotari
Let me step back, in order to understand better the issue.
if we execute the loop over the sections always in the same order and keep/release the locks in the appropriate order.
how can we fall into a deadlock ?

how can we fall into a deadlock ?

Not an easy question. I don't have an answer to that. Perhaps @Vanlightly could help us?

I see now, this is a case of deadlock, we are issuing a blocking call to ZooKeeper into a forEach, and the ZooKeeper library handles every operation in only one single thread.
So the chain is pretty hard do understand
"pulsar-backlog-quota-checker-146-1"  prio=5 tid=258 TIMED_WAITING
java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1709)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
        at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1788)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
        at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97)
        at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:66)
        at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:78)
        at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuotaLimit(BacklogQuotaManager.java:94)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.isBacklogExceeded(PersistentTopic.java:2227)
        at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$53(BrokerService.java:1502)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$453/1035521358.accept(Unknown Source)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$858/1296923805.accept(Unknown Source)
        at java.util.Optional.ifPresent(Optional.java:159)
        at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$52(BrokerService.java:1490)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$454/965168673.accept(Unknown Source)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:387)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
        at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1488)
        at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1499)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$280/2017773859.run(Unknown Source)
        at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32)
        at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

I've not spent much time in the Pulsar codebase, so don't know much about the wider context. But if it helps, I could model this ConcurrentOpenHashMap in TLA+ and see if it has any deadlock scenarios. Couldn't give any commitment on timing though.

It's not really a defensive copy of the entire map. The copy is done one section at a time.

By defensive, I mean that we'd be taking a copy of the entire map (section by section) each time we need to traverse it, even though there would no need to do it (eg: when a section is not being modified). In case of a map which is not updated super-frequently, that would impose a size-able overhead.

Memory allocations and GC is extremely efficient in modern JVMs.

I've been battling GC and allocations for many years now and while that might be true to a certain extent, there are also many bad GC behaviors that we had fixed doing changes like switching from the regular ConcurrentHashMap to this custom implementation.

Since it's about 28 bytes per entry, there would have to have millions of entries for the memory allocation overhead to have any noticeable significance. If necessary, this could be benchmarked.
Since there aren't millions of entries, but usually a few thousand of entries, it has even less significance. This is why I don't see a reason to avoid the copying in this case.

The problem is that there can be easily millions of entries (aggregated from 100s of thousands of maps) on a single instance. That has been the case in many production deployments for many years.

We do have IO operations (interactions with Zookeeper) in most of the code that processes the map items and it cannot be easily removed.

Sure, but we should take care of them (eg: during the refactoring to port these components to use MetadataStore instead of direct ZK access.
Also, I was commenting that we could do it such that we automatically switch to individual locks (instead of whole section lock) if the processing is taking more than X amount of time.

It reduces the duration of the locks in the case that there are no writes. When there are writes, the .forEach processing will hold the read lock until all items have been processed. This is one of the reasons why the problems are rare with small number of items or low amount of write operations. However such a solution won't completely prevent deadlocks, if the model is such that it could lead to a deadlock that isn't detected or resolved after some timeout. A deadlock will eventually happen even if the probability is low. Currently the Pulsar broker locks up and doesn't resume operations if there's a deadlock. That's what makes the problem severe. Since the locks could be in specific sections of a ConcurrenOpenHashMap, it leads to partial unavailability which is hard to detect.

While it's clear that a slow scan will provoke a spike in write latency into the map, I'm still not clear on the exact case of deadlock here.
Typically the way to get deadlocked on the map is if, while doing a scan, you're trying to access it again from a different thread (or even same thread, since the lock is non-reentrant) and that blocks the processing function.

I see now, this is a case of deadlock, we are issuing a blocking call to ZooKeeper into a forEach, and the ZooKeeper library handles every operation in only one single thread.
So the chain is pretty hard do understand

@eolivelli Even if do a ZK blocking call (which we shouldn't), that doesn't necessarily mean that there will be a deadlock. It will only be the case if the ZK callback thread is blocked trying to read/write on the same map. Do you have the other threads stacks?
To conclude, in my opinion the easiest/lower-risk/lower-overhead change here would be to switch the forEach() to do the stamped+read-lock combination for each entry, instead of the current per-section.
That will ensure that the process function is always call without holding a lock on the map. It can lead to spurious inconsistencies during re-hashing time, but I don't think that will be a problem. And anyway that can be solved by requesting to behave like the current implementation for these specific cases.

It can lead to spurious inconsistencies during re-hashing time,

Scratch that. it's actually possible to do it with the current concurrency level, no funky stuff.

To conclude, in my opinion the easiest/lower-risk/lower-overhead change here would be to switch the forEach() to do the stamped+read-lock combination for each entry, instead of the current per-section.

Thank you for the detailed feedback @merlimat . This makes sense. I believe that it would solve most of the problems. I'll revisit this PR with that approach.

I have a WIP solution already. Need to look it a bit more to see if I'm missing anything and I'll create a PR.

I have a WIP solution already. Need to look it a bit more to see if I'm missing anything and I'll create a PR.

Ok great. I'm looking forward to that.
I'll close this PR.

@merlimat

@eolivelli Even if do a ZK blocking call (which we shouldn't), that doesn't necessarily mean that there will be a deadlock. It will only be the case if the ZK callback thread is blocked trying to read/write on the same map. Do you have the other threads stacks?

I didn't paste the full dump of competing stacktraces sorry.
Btw I believe that it is no more so important to the discussion. Thanks for answering my comment

@eolivelli @lhotari Created #9787


Update gspread_client.py

Inside google, we  are removing support for from gspread import gspread
Description of the change
remove support for from gspread import gspread
Motivation
When gspread is installed through pip, it can be used as import gspread
Type of change
Choose one: Bug fix
Checklist

[x] I have performed a self-review of my code.
[x] I have added detailed comments to my code where applicable.
[x] I have verified that my change does not break existing code.
[x] My PR is based on the latest changes of the main branch (if unsure, please run git pull --rebase upstream main).
[x] I am familiar with the Google Style Guide for the language I have coded in.
[x] I have read through the Contributing Guide and signed the Contributor License Agreement.
I am already a Googler so I don't need to sign the CLA as the CLA text says.

Ah, I see what happened (cl/572251129).
Thanks for the fix, but we can't merge this until you sign the CLA.

@google-cla check again please.


Import CircuitInstruction to fix Pylance complaint

Summary
Using CircuitInstruction in a type annotation causes Pylance to complain
Variable not allowed in type expression

This fixes that.
Details and comments

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.001%) to 88.96%

Files with Coverage Reduction
New Missed Lines
%

crates/qasm2/src/lex.rs
5
92.23%

Totals

Change from base Build<PHONE_NUMBER>2:
-0.001%

Covered Lines:
79435

Relevant Lines:
89293

💛  - Coveralls

If we change this, it would be good to have a guideline and a check by CI/linter. Otherwise we cannot ensure we're consistent here and this change seems a bit arbitrary.
I've used Pylance as well, aren't there also other things it complains about? E.g. using (TypeA, TypeB) instead of tuple[TypeA, TypeB] or something?

If we change this, it would be good to have a guideline and a check by CI/linter. Otherwise we cannot ensure we're consistent here and this change seems a bit arbitrary.

Does the change still seem arbitrary after the latest commit? I think it brings it more in line with the rest of Qiskit, since we normally import stuff rather than setting a variable like it was before.

I've used Pylance as well, aren't there also other things it complains about? E.g. using (TypeA, TypeB) instead of tuple[TypeA, TypeB] or something?

I don't know, this is the only one that I noticed.


Db/add 2004 container

What does this PR do?
Adds new targets for generating windows-2004 based containers
Motivation
customer request

Moved back to 7.25 as discussed in channel, more than one customer request at this time


Refactor user needs

Started of by realising we could compute a need’s title from its fileSlug… and then I kinda went to town! 🤪
Layouts

Remove separate user-need layout and use post layout for user needs instead. Related posts now show in the sidebar, like elsewhere on the site.
Update user-needs layout so that in inherits from page layout. Remove references to unused macros.

Content and data

Compute a user need’s title from its file slug (and prefix with Need ). This saves us having repeat this information twice (once in the file name, and again in front matter)
Update all needs to move need out from front matter and into post content. Most needs now don’t require any front matter data.
Use eleventyComputed to generate related posts, rather than calculate this in the template.
Update a post that was missing tags that prevented the related need linking back to it.

Components

Update appUserNeed to be an article with a uniquely named title that is linked. Capitalise status tag text. Refer to item.data.content instead of item.data.userneed. Adjust styles so that title appears below need and the entire card is selectable (darkening slightly on hover).
Update appArticle component so that it can show a tag in its header. This seems like a thing you might want to do separate from this particular feature (see this example of a tag shown on an article on the design system website). Currently have this set up so that the component accepts a tag parameter, and in our layouts we give this a post’s status value.

Comparison
User need component before

User need component after

User need layout before

User need layout after

❤

Spotted this – is this a bug:

@adamsilver:

Returned underline links on user needs card
Fixed disappearing text on user need card within a post

and also fixed appearance of parent section in search results (which is broken currently).

As to related links, I agree, we might want to visit this presentation later. However, for now, most needs have one or two related posts, so I think we can stick with the existing pattern for now.


Flake8 test failures

I'm sure I'm doing something wrong, but basically every test file is failing the Flake8 check in the same way. I'm running this on Alpine Linux with Python 3.8 and all the latest versions of everything.
___________________________________________________________ FLAKE8-check ____________________________________________________________
/usr/lib/python3.8/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
/usr/lib/python3.8/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/lib/python3.8/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
/usr/lib/python3.8/site-packages/_pytest/runner.py:143: in pytest_runtest_call
    raise e
/usr/lib/python3.8/site-packages/_pytest/runner.py:135: in pytest_runtest_call
    item.runtest()
/usr/lib/python3.8/site-packages/pytest_flake8.py:119: in runtest
    found_errors, out, err = call(
/usr/lib/python3.8/site-packages/py/_io/capture.py:150: in call
    res = func(*args, **kwargs)
/usr/lib/python3.8/site-packages/pytest_flake8.py:191: in check_file
    app.parse_preliminary_options_and_args(args)
E   AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'

Presumably this issue is no longer an issue, as tests are run using flake8 as part of the CI tests. Please comment back if this issue still manifests for you.

I just upgraded to 5.0.1 and it isn't a problem anymore indeed, thanks!


lineDashOffset has no effect

Tested on ol.style.Stroke. LineDash works correctly, but offset is ignored.
See live example: https://codepen.io/gkresic/pen/gRVXgL
To test browser support (works on mine Chrome 60): https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/lineDashOffset
OpenLayers 4.2.0

lineDashOffset is not supported by the immediate renderer, which is used for rendering regular shapes. That functionality would have to be added to ol.render.canvas.Immediate.prototype.setContextStrokeState_ in src/ol/render/canvas/immediate.js. We'd accept a pull request for that.

fixed with #7042


further_info_url should include https, rather than http

Ref PCMDI/cmor#247
The template should be https:// whereas it is currently http:// following v6.2.5 of the CMIP6 Global Attributes, DRS, Filenames, Directory Structure, and CV’s document

https://github.com/PCMDI/cmor/commit/773119d3bdff897190dd604c477a5a0e37856b65

I forced https:// in further_url_info.


Add support for INCLUDE NMODL directive

This pull-request tackles #300

handle the INCLUDE NMODL directive

parser now follows the included files and store the result AST
in the nmodl::ast::Include AST node.

update NmodlVisitor generatede class so that it ignores the program
member of the nmodl::ast:Include AST node.
NmodlDriver:

new method parse_include in charge of parsing
move reporting of parsing error in NmodlDriver class.
remove useless member variables lexer and parser to make it completely reentrant.
all parse_* methods now raise exceptions upon error.

Improve performance of methods:

less lookup in datastructure:

NmodlDriver::get_defined_var_value
SingletonRandomString::get_random_string

return const&:

SingletonRandomString::get_random_string

I am very puzzled. I was trying to get more log using:
`nmodl /home/katta/projects/nmodl/test/integration/mod/cabpump.mod passes --json-ast
[NMODL] [info] :: Processing /home/katta/projects/nmodl/test/integration/mod/cabpump.mod
[NMODL] [info] :: Running symtab visitor
[NMODL] [info] :: Running LOCAL to ASSIGNED visitor
[NMODL] [info] :: Running code compatibility checker
[NMODL] [info] :: Writing AST into /home/katta/projects/nmodl/test/integration/mod/cabpump.mod
Segmentation fault (core dumped)
`
seg fault. I am checking the debugger but the node looks fine except for some modTokens that are nullptr. I am not sure, may they be nullptr? I am not sure

seg fault. I am checking the debugger but the node looks fine except for some modTokens that are nullptr. I am not sure, may they be nullptr? I am not sure

I can reproduce the issue. the segfault happens in
json_printer.cpp:75: back->front().push_back(*current);

the current local variable is nullptr

Here is the sequence of things that lead to this issue:

traversing AST
...
JSONVisitor::visit_include

...
JSONVisitor::visit_program // new attribute added in this PR
JSONPrinter::flush()  // special case for Program*block = nullptr`

Suggestion: in visit_program, only call flush is the parent node is nullptr

seg fault. I am checking the debugger but the node looks fine except for some modTokens that are nullptr. I am not sure, may they be nullptr? I am not sure

I can reproduce the issue. the segfault happens in
json_printer.cpp:75: back->front().push_back(*current);

the current local variable is nullptr
Here is the sequence of things that lead to this issue:

traversing AST

...

JSONVisitor::visit_include

...
JSONVisitor::visit_program // new attribute added in this PR
JSONPrinter::flush()  // special case for Program

block = nullptr

Suggestion: in visit_program, only call flush if the parent node is nullptr

179f5a9 fixes the issue.

wow, this is the first time we use parent pointers!

defined_var became a umap while open_files remained a map (while I asked the change on open_files). Is maybe a lapsus or is there a reason behind?

I changed defined_var to unordered map before your comment. You were right for open_files, key ordering is not required.

defined_var became a umap while open_files remained a map (while I asked the change on open_files). Is maybe a lapsus or is there a reason behind?

I changed defined_var to unordered map before your comment. You were right for open_files, key ordering is not required.

Never mind, defined_var was changed after your comment @cattabiani :)

Hello @tristan0x , I will review this tomorrow but one thing that I would like to change here is to have only one Program node for whole AST i.e. if I do:
./bin/nmodl ../test/integration/mod/ispc_rename.mod passes --nmodl-ast --json-ast

then we get:
# tmp/ispc_rename.ast.json

{
  "Program": [
    {
      "NeuronBlock": [
        {
          "StatementBlock": [
....
    {
      "Include": [
        {
          "String": [
            {
              "name": "\"var_init.inc\""
            }
          ]
        },
        {
          "Program": [
            {
              "BlockComment": [
                {
                  "String": [
                    {
                      "name": "\n    Example file to test INCLUDE keyword\n    in NMODL. This file is included in\n    cabpump.mod and takes care of   initializing\n    the dummy variable named \"var\"\n"
                    }
                  ]
                }
              ]
            },
            {
              "FunctionBlock": [

we have two Program nodes in whole AST. For internal AST of included node, I suggest to either replace Program node with something else (e.g. Block) or just attach all children inside Include. The reason for this is : At conceptual level, the included file is not the whole "Program" but typically just helper functions. I would like to keep a notion of "Program" representing whole resultant mod file and hence single Program node in whole AST.

I totally subscribe @pramodk view. I think that the problem highlighted by the integration tests and ioannis is due to having a program node that is not root in the ast. symtab visitor is confused (probably). It would be much easier (even conceptually) to just replace program with its blocks.

I totally subscribe @pramodk view. I think that the problem highlighted by the integration tests and ioannis is due to having a program node that is not root in the ast. symtab visitor is confused (probably). It would be much easier (even conceptually) to just replace program with its blocks.

I applied your suggestion @pramodk and @cattabiani

New error tests to add :

[ ] filename is empty$ echo '#include ""' | | g++ -x c++ -
<stdin>:1:10: error: empty filename
#include ""
         ^
1 error generated.

[ ] filename does not exist
[ ] included filename has parsing issues

reset this please

retest this please

please retest

retest this please Jenkins

please retest

retest this please Jenkins

This branch passed and is ready to be merged. Do you think we need to recheck it? @tristan0x

This branch passed and is ready to be merged. Do you think we need to recheck it? @tristan0x

It is ok to be merged on my side. @pramodk had additional questions I just answered, so it is up to him.


Allow test.sh CI script to run locally

So that it is easier to run the CI test scripts on a developer machine, a few changes are made to the script. As well as making some functional changes, shellcheck is run and some escaping in the script is cleaned up.
Running the script locally makes it easier to check that the tests are running as expected when porting to other architectures, before pushing changes to a pull request.
The main changes are:

Made paths relative to the oneDAL root directory, to allow an out-of-source run
Only source miniconda in the case that the file exists. This means that the CI runs are the same, but that a developer doesn't need to install miniconda on their own system
Write stderr from tests to file. The redirects were in the wrong order for this to happen

Minor changes are:

Added help text
Changed the naming of command-line arguments to use '-' instead of '_' to make naming consistent
Iterate over a list of elements, rather than a string
Update escaping, as suggested by shellcheck
Add some exit paths (e.g. if directories do not exist)

/intelci: run

on the side note - similar scripts exist in scikit-learn-intelex.  At some point it would worth porting them there


fix compilation with latest TLS >= 1.3 and remove deprecated crypto packages

remove deprecated cryptohash and cryptorandom in favor of cryptonite
remove base16-bytestring in favor of memory that can convert more elaborate type like (HMAC)
adapt to work with tls 1.3 (removal of the rng parameter)

Thanks a lot for this! Just one request before merging: seems base64-bytestring and memory overlap. Can we completely replace the former and remove it from the build depends?


[doc] Fix a typo in KubeVirt troubleshooting script

What this PR does / why we need it:
Fix a typo in KubeVirt troubleshooting script
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[X] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[X] This change includes docs.
[ ] This change includes unit tests.

@davidvossel FYI


Update  cryptoauthlib to include atecc608b support ?

Hi,
Great code!
Will you be updating the cryptoauthlib module to the latest version to support the Rev B of the atecc608?
Thanks!

sorry, we don't have ATECC608B at the moment. According to public documentation, there is virtually no differences between ATECC608B and ATECC608A.
Please try https://github.com/wirenboard/atecc-util/pull/15 , I've added ATECC608B signature to IC auto detection method.


Bump version to 1.7.4

@theotherjimmy Don't merge yet.

@screamerbg No problem. What are we waiting on?

@theotherjimmy Let's get this in and make a new release.

@screamerbg Would you like me to make the release?

Thanks. It's OK. I'll sort it out shortly.


Update crutch.rb to allow crutch inter-dependency with efficiency

Adds support for passing all crutches into a crutch definition, to be able to form a chain of crutches such that a field can depend and load only what it needs, instead of loading multiple objects at times, which might not all be required always.
Earlier:
class MyIndex

    crutch :my_large_crutch do |models|
        model_rel1_ids = models.map { |m| m.rel1_id }
        rel1_objs = <Load objects of Rel1 from Mongo>
        rel2_ids = rel1_objs.map{ |r| r.rel2_id }
        rel2_objs  = <Load objects of Rel2 from Mongo>
        ..... Relations can go any nested levels
        data = {}
        data[:rel1_objs] = <hash_of rel1_objs>
        data[:rel2_objs] = <hash_of rel2_objs>
        data
    end

    field :field_belongs_to_rel1_obj # <also ends up loading rel2 objects in case of partial update>
    field :field_belongs_to_rel2_obj
end

Now:
class MyIndex

    crutch :my_rel1_objs do |models|
         model_rel1_ids = models.map { |m| m.rel1_id }
        rel1_objs = <Load objects of Rel1 from Mongo>
        <hash of rel1_objs>
    end

    crutch :my_rel2_objs do |_, crutches|
         rel1_objs = crutches.my_rel1_objs
         rel2_ids = rel1_objs.map{ |r| r.rel2_id }
        rel2_objs  = <Load objects of Rel2 from Mongo>
        <hash of rel2_objs>
    end

    field :field_belongs_to_rel1_obj # <now can depend only on my_rel1_objs crutch, avoiding loading of rel2 objects, in case of a partial update>
    field :field_belongs_to_rel2_obj
end

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[ ] Wrote good commit messages.
[ ] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[ ] Added tests.
[ ] Added an entry to the changelog if the new code introduces user-observable changes. See changelog entry format for details.

Thanks @akshay58538 for your PR
Overall I think looks good, but you miss 2 things

Documentation
Specs

Thanks!

Yeah, and fix CI please (rubocop specifically)

@konalegi Unrelated, but still asking.
Do you know why mongoId adapter was removed in https://github.com/toptal/chewy/pull/776 ?
We were trying to update to latest version of chewy and realized much later, that mongoid adapter itself has been dropped from chewy.
Isnt mongo+ES a very standard architecture in many companies (including mine)?

Hmm I just came across this issue hoping this was a resolution to a problem I'm having which I noticed when you have more than one crutch defined (eg. crutch a, crutch b). It seems to evaluate both crutches during the update_fields use case (where the update_fields is specific to field :a, i would only expect crutch a to evaluate).


[-] MO: sekeywords / Use new hook with PSCleaner module

Hello @PrestaEdit , thanks for your contribution, would you mind to rebase your pull request (for fixing conflicting file)? Thanks


Files with little text emit a description rather than just the text

I have a debug PDF I use to test cropping to specific text ranges. It consists of some textboxes with the text A1 to A4. When I pass it to Xerox it returns the following description: The page is blank except for the labels in the corners: A1, A2, A3, and A4., rather than the text contents A1 A2 A3 A4.

I've also noticed this with pages that are blank. Which we can probably fix with the tesseract pre-process step.
For this one, probably a prompt change...

to fix your trouble check this solution click
maybe this will solve your problem.


GH-5 Throw automatic errors if authors submits screenshots which are not png

[x] I understand that contributing to this repository will require me to agree with the CLA

Description
If screenshots are not png format, label "changes_requested" and comment about the problem are added.

What type of PR is this? (check all applicable)

[ ] 🍕 Feature
[x] 🐛 Bug Fix
[ ] 📝 Documentation Update
[ ] 🎨 Style
[ ] 🧑💻 Code Refactor
[ ] 🔥 Performance Improvements
[ ] ✅ Test
[ ] 🤖 Build
[x] 🔁 CI

Related Tickets & Documents
https://linear.app/anytype/issue/GH-5/throw-automatic-errors-if-authors-submits-screenshots-which-are-not
Mobile & Desktop Screenshots/Recordings

Added tests?

[ ] 👍 yes
[ ] 🙅 no, because they aren't needed
[ ] 🙋 no, because I need help

Added to documentation?

[ ] 📜 README.md
[ ] 📓 tech-docs
[ ] 🙅 no documentation needed

[optional] Are there any post-deployment tasks we need to perform?

@any contributor @jmetrikat tooling


Question

Hello,
Please, I want to ask you about the implemented Bidirectional LSTM (Bi-LSTM), is it a deep one? because it is mentioned in the RAM paper that the proposed approach uses 2 layers of (Bi-LSTM) but I can't see that in your code!
Thank you,

Actually, I implemented BiLSTM with single layer here. I have tried to implement a BiLSTM network with two  layers but achieve a worse result. Maybe it's because some details I implemented (i.e. hyperparameter setting) are different from the author did.

Okay, thank you for your efforts.


Live Preview Opening Up Multiple Tabs

Hello,
Every time I edit something in my html file multiple previews open of my website. Why is this happening and if there is a solution how can I fix it?
Thanks!

I'm also seeing this.  It makes this otherwise awesome plugin almost unusable ... :-(  Please fix!

I'm also seeing this.  It makes this otherwise awesome plugin almost unusable ... :-(  Please fix!

I'm only having that issue when I launch the app and it restores my project.

Same issue here. Otherwise, working great. Thanks!

Im having the same issue

same issue

the same issue

Same problem here

I'm noticing this as well, I can reproduce it when I hit "CTRL + S" to do a quick save, it will spawn 2 more tabs of the same preview every time.  I hope that helps.

I found this fixed it for me.
Close all current files.
Uninstall Live Preview.
Close Atom
Re-open Atom and reinstall Live Preview.
Close again and re-open Atom.
Hopefully it works for others too.


Update MD file on new branch 2017-11-15_19-24-38

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


BUG:user hover on the card then it is not in perfect layout

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
@Suchitra-Sahoo  user hover on the card then card img width is cut the right section ,thats why proper border radius not display

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Is it happening in mobile device?

@Suchitra-Sahoo  ,it is happenig in all devices

But I just checked and everything is working properly.


deepintent: ccpa support header added

Fix for mentioning CCPA support.

Hi
Is this just a docs update?

Yes this is just a doc update

confirmed that the bid adapter code does look for bidderRequest.uspConsent. merging.


WIP: Minor fixups for recipe and try to build with new compilers (4.4.x)

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


feat: added nockStripHeaders option

:tada: This PR is included in version 2.19.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket: