RobustPhaseEstimation vs QuantumPhaseEstimation

Is your feature request related to a problem? Please describe.
RobustPhaseEstimation and QuantumPhaseEstimation return quite different values as "phases", which makes switching between the two non-trivial.

QPE returns (after measurement) an integer which, when divided by 2^(bits of precision), produces a floating-point number between 0 and 1, which needs to be multiplied by 2π to get a phase e^{iϕ}. The API documentation does not describe the return, and the only range in the docs is ϕ∈[0,2π) which seems to suggest that no multiplication by 2π is necessary.
Robust PE returns a floating point phase between -π and π which corresponds to e^{iϕ} directly.

Describe the solution you'd like
It would be great to unify the outputs of these operations or at least to document them better, with specific output descriptions and examples.

The different in implementation can be observed in this sample:
https://github.com/microsoft/Quantum/blob/783f4a7ba71e6f2f9d4db0861473ebfa2c3cf1f2/samples/algorithms/integer-factorization/Shor.qs#L243--275

Ideally the documentation would be sufficient to use the library operation without looking at the samples repo and the source code of the operation... I just got burned by this again, since the docs for QuantumPhaseEstimation still hint that the result is ϕ∈[0,2π).

@tcNickolas: I agree that this issue leads to problems and needs to be addressed. My comment about the sample was only meant as additional information to the problem itself.


cmake: compiler: Add framework for disabling C++ standard includes

This commit adds a new C++ compiler property `nostdincxx` which
specifies the C++ compiler flag for excluding the C++ standard library
include directory from the include paths.

Signed-off-by: Stephanos Ioannidis<EMAIL_ADDRESS>
Fixes #36644.

@evgeniy-paltsev @abrodkin Does ARC MWDT have something like -nostdinc++ from GCC?

@tejlmand - can you take a look?

@nashif - this is a fairly straight-forward PR that mainly touches the build system. Are you able to ack?

Does ARC MWDT have something like -nostdinc++ from GCC? If so, we should specify the nostdincxx property for the MWDT as well.

@stephanosio I guess we do similar thing for MWDT but in a slightly different way:
cmake/compiler/arcmwdt/target.cmake
if(CONFIG_ARCMWDT_LIBC AND CONFIG_LIB_CPLUSPLUS)
  list(APPEND NOSTDINC ${TOOLCHAIN_HOME}/arc/lib/src/c++/inc)
endif()

foreach(isystem_include_dir ${NOSTDINC})
  list(APPEND isystem_include_flags -isystem "\"${isystem_include_dir}\"")
endforeac


Unable to start the daemon process with Gradle 3.5

I'm trying to build a Java 9 based modular project using Gradle 3.5. But, it fails with "Unable to start the gradle daemon process" message.
Expected Behavior
build should succeed
Current Behavior
Starting a Gradle Daemon (subsequent builds will be faster)
FAILURE: Build failed with an exception.

What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the user guide chapter on the daemon at https://docs.gradle.org/3.5/userguide/gradle_daemon.html
Please read the following process output to find out more:

FAILURE: Build failed with an exception.

What went wrong:
java.lang.ExceptionInInitializerError (no error message)

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

Steps to Reproduce (for bugs)

export PATH to OpenJDK b160 build.
export PATH to Gradle 3.5 downloaded from https://gradle.org.
export GRADLE_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED"
gradle build

It's enough to have only "apply plugin: 'java'" in the build.gradle to reproduce this problem.
As a work around, currently, i'm disabling the daemon by adding -Dorg.gradle.daemon=false option to GRADLE_OPTS.
Your Environment
Gradle 3.5
Build time:   2017-04-10 13:37:25 UTC
Revision:     b762622a185d59ce0cfc9cbc6ab5dd22469e18a6
Groovy:       2.4.10
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          9-internal (Oracle Corporation 9-internal+0-adhoc.jenkins.openjdk.jdk9.master.build.espresso.linux.x8664)
OS:           Linux 4.4.0-66-generic amd64

Thanks for the well written issue @sabkrish. This is a duplicate of https://github.com/gradle/gradle/issues/1549. We plan on working on improved Java 9 support for Gradle 4.1.


Register the package

Done - I've made this public. Please go ahead and register this as a new package.

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/37502
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.0 -m "<description of version>" 48fd6c6b36c1737acd13618646d917cafbd09d4c
git push origin v0.1.0


Marseille , Abudhabi and Singapore not available in Regions

We from OCI Vision team are trying to deploy few applications to the listed regions and found the Regions missing here

This has  been fixed in the 5.3.1 release


Add more controls on historyBlocks in manifest

@incrypto32 looking good, one last thing, can we change the grapman prune default history blocks to also be 2 * reorg_threshold? For consistency across ways of pruning.

Chatting to @Data-Nexus, we don't want the syntax to spook subgraphs if they understand 'historical' as meaning entities that are just old, and that we'd delete their entities just because they are old. An alternative syntax could be:
indexerHints:
  prune: [auto|never|<number>]

This way the default
indexerHints:
  prune: auto

Looks as friendly and reassuring as a piece of YAML can.

I agree that "prune" is probably the better word. See: https://dictionary.cambridge.org/us/dictionary/english/prune.

to reduce something by removing things that are not necessary


Add warp-sync zombienet test with validators

This is followup of #11471.
The warp-sync test for full-nodes was implemented in https://github.com/paritytech/substrate/pull/12675.
The warp-sync test with validators should also be added. The initial try was taken in #12751, however the test is failing with following errors:
2022-11-18 17:05:10.577  WARN tokio-runtime-worker sync: 💔 Verification failed for block 0xf0756734abbabde16cae212d5f238994707d69fa2fe7e4d095a281638b2998fc received from peer: 12D3KooWRkZhiRhsqmrQ28rt73K7V3aCBpqKrLGSXmZ99PTcTZby, "Could not fetch epoch at 0x5448eda4b7963ee85d28ea523149317dc39ce41173e198126b7ba67b7e123854"

or
2022-11-18 16:57:55.224  WARN tokio-runtime-worker sync: 💔 Verification failed for block 0x105866ce153d80fff9d67ea804cd43543bffd1e66c0714e40a79ad6c9f8d0008 received from peer: 12D3KooWRkZhiRhsqmrQ28rt73K7V3aCBpqKrLGSXmZ99PTcTZby, "VRF verification failed: EquationFalse"

The warp-sync test for validators could be added to #11727. More sophisticated scenario could be tested e.g. sync-warp validators and check if they can produce blocks.

2022-11-18 16:57:55.224  WARN tokio-runtime-worker sync: 💔 Verification failed for block 0x105866ce153d80fff9d67ea804cd43543bffd1e66c0714e40a79ad6c9f8d0008 received from peer: 12D3KooWRkZhiRhsqmrQ28rt73K7V3aCBpqKrLGSXmZ99PTcTZby, "VRF verification failed: EquationFalse"

But this should be solved now?

Should be, but test is still not passing: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2084982
(PR with recent master merged, test toml)

Root cause: https://github.com/paritytech/substrate/pull/12751#issuecomment-1328032781


[camera] flip/change camera while recording - platform interface

This is the split out PR for the platform interface changes for this PR: https://github.com/flutter/plugins/pull/6478

@bparrishMines Wondering if you have a chance to look at this PR?

Forgot about secondary reviewer.
@camsim99 @stuartmorgan Could you do a secondary review?


Correction for raw affiliation School of Management, Economics and Mathematics, King’s University College at Western University Canada, London, Canada

Correction needed for raw affiliation School of Management, Economics and Mathematics, King’s University College at Western University Canada, London, Canada
raw_affiliation_name: School of Management, Economics and Mathematics, King’s University College at Western University Canada, London, Canada
new_rors: ;02grkyz14
previous_rors: 048f5b183;02grkyz14
works_examples: W4387872219;W4390237365
contact: a35349993ad8ccddb57958aa08e2a280:fe29cb5095ac82fc @ uwo.ca

This issue was accepted and ingested by the OpenAlex team on 2024-12-21. The new affiliations should be visible within the next 7 days.


Improved the running and capturing output of idevice_id.exe. Removed expired cert.

How is that better? Also the certificate isn't expired. Run Unblock-File on the script, make sure you've ExecutionPolicy set to RemoteSigned.

Your method of capturing the output from idevice_id.exe was returning false positives after a device had been found then gone back to being not found again.
I tried unblocking the file and setting the ExecutionPolicy  to RemoteSigned and it still didn't work for me.

Strange, it worked for me on another machine without issues. Did you clone to desktop? When you copy from raw, the line endings will be wrong.

Done. Retracted the earlier pull request and created a new one. Thanks

From: Discord Digital @.>
Sent: 23 May 2023 10:56
To: DiscordDigital/AltCheck-Reborn @.>
Cc: CrazeeGhost @.>; Author @.>
Subject: Re: [DiscordDigital/AltCheck-Reborn] Improved the running and capturing output of idevice_id.exe. Removed expired cert. (PR #3)
Feel free to create a pull request where only the method of capturing the output from idevice_id.exe is changed. Then I'll accept the pull request.
—
Reply to this email directly, view it on GitHubhttps://github.com/DiscordDigital/AltCheck-Reborn/pull/3#issuecomment-1559605474, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFCIIF2WW2T76ELHIG67NQTXHTFYPANCNFSM6AAAAAAYL64SDY.
You are receiving this because you authored the thread.Message ID: @.***>


Update StakingNFT to include required skimming of slush

Scope
We are including calls to _slushSkim to ensure that all rewards are appropriately distributed.
Why?
Without this change, profits may be distributed to Stakers who were not staked at the time of deposit.
Todos
Write additional tests in StakingNFT to ensure appropriate behavior.

Codecov Report

Merging #191 (1684ce9) into candidate (3b9bdd8) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           candidate     #191   +/-   ##
==========================================
  Coverage      45.02%   45.02%
==========================================
  Files            220      220
  Lines          29131    29131
==========================================
  Hits           13116    13116
  Misses         14067    14067
  Partials        1948     1948

Impacted Files
Coverage Δ

badgerTrie/smt_merkle_proof.go
67.50% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b9bdd8...1684ce9. Read the comment docs.

This PR was moved over to AliceNet. It is no longer current and has been superseded.


Improve user experience

Research possible ways to simplify the use of the interface. One example is the abi parameter in deploy_contract_v3, which shouldn't be required.

Hi @ddoktorski can you please assign this issue to me?


Figure out which images we are missing on AWS and copy them up

The readme says there are ~1.5 million images. This is probably just the images for the subguide cards. Figure out how many there actually are, and update that. As part of this, validate that we successfully got all the images
Do all of the following, numbers should align:

Some kind of command line file listing count executed on the machine that originally was storing them. (they were rsynced to the imagecat staging box and that's all we have access to now; count them there)
do some kind of aws list command
wait for the card image load to finish and then do CardImage.count.

If we do all 3 of these we can validate that we successfully copied everything to AWS and then successfully loaded all of those into the prod database.
If those 3 counts are off, figure out what's missing and get it copied / loaded.

I did a very rough estimate based on the progressbar currently loading cardimages for guide cards, and I got 4,375,000. I think the original number reflects the number of images for subguide cards, so the total may be something like 5.5 - 6 million.

@imagecat-staging1:/var/tmp/imagecat# find . -type f -name "*.tiff" | wc -l
Gives 5,786,727

deploy@imagecat-staging1:/var/tmp/imagecat$ for i in {1..22}; do echo disk${i}; find disk${i} -type f -name "*.tiff" | wc -l ; done
disk1
277228
disk2
275260
disk3
261405
disk4
276321
disk5
257479
disk6
264898
disk7
257121
disk8
274647
disk9
283706
disk10
263320
disk11
278984
disk12
252882
disk13
277719
disk14
279365
disk15
284386
disk16
262902
disk17
269623
disk18
252032
disk19
253633
disk20
193619
disk21
242875
disk22
247322

from aws:
I, [2023-10-10T15:22:44.202695 #77846]  INFO -- : Fetching disk 1 file list
277227
I, [2023-10-10T15:24:17.118391 #77846]  INFO -- : Fetching disk 2 file list
275259
I, [2023-10-10T15:25:55.267232 #77846]  INFO -- : Fetching disk 3 file list
261405
I, [2023-10-10T15:27:26.129265 #77846]  INFO -- : Fetching disk 4 file list
276216
I, [2023-10-10T15:29:05.928898 #77846]  INFO -- : Fetching disk 5 file list
257479
I, [2023-10-10T15:30:36.616389 #77846]  INFO -- : Fetching disk 6 file list
264898
I, [2023-10-10T15:32:12.400218 #77846]  INFO -- : Fetching disk 7 file list
257121
I, [2023-10-10T15:33:44.087002 #77846]  INFO -- : Fetching disk 8 file list
274647
I, [2023-10-10T15:35:21.771344 #77846]  INFO -- : Fetching disk 9 file list
283706
I, [2023-10-10T15:36:58.456716 #77846]  INFO -- : Fetching disk 10 file list
263320
I, [2023-10-10T15:38:28.430788 #77846]  INFO -- : Fetching disk 11 file list
278984
I, [2023-10-10T15:40:06.554338 #77846]  INFO -- : Fetching disk 12 file list
252882
I, [2023-10-10T15:41:35.291431 #77846]  INFO -- : Fetching disk 13 file list
277719
I, [2023-10-10T15:43:11.833596 #77846]  INFO -- : Fetching disk 14 file list
279365
I, [2023-10-10T15:44:53.257739 #77846]  INFO -- : Fetching disk 15 file list
284386
I, [2023-10-10T15:46:36.283309 #77846]  INFO -- : Fetching disk 16 file list
262902
I, [2023-10-10T15:48:07.166952 #77846]  INFO -- : Fetching disk 17 file list
269623
I, [2023-10-10T15:49:42.001741 #77846]  INFO -- : Fetching disk 18 file list
252032
I, [2023-10-10T15:51:09.708023 #77846]  INFO -- : Fetching disk 19 file list
253633
I, [2023-10-10T15:52:41.115495 #77846]  INFO -- : Fetching disk 20 file list
187169
I, [2023-10-10T15:53:49.957428 #77846]  INFO -- : Fetching disk 21 file list
242875
I, [2023-10-10T15:55:12.637343 #77846]  INFO -- : Fetching disk 22 file list
247322

There's a report of all the content that's in the S3 bucket which will get generated to the dls-transfer bucket, but it might take up to two days to generate. After that it's daily.

Ran the following, ignore the ;nil, that was me running this in IRB:
files = Dir.glob("tmp/*.csv")
all_lines = files.flat_map do |f|
  File.read(f).split("\n").map do |entry|
    corrected = entry.split(",").last.gsub("imagecat-","").gsub('"', "").gsub("-", "/").gsub(/tif$/, "tiff")
    "./#{corrected}"
  end
end; nil
set = Set.new(all_lines); nil
local_files = File.read("tmp/file_list.txt").split("\n"); nil
local_files = Set.new(local_files); nil
diff = local_files - set; nil

6,557 missed files.
missing_files.txt

Those 6,557 have been uploaded to puliiif-production via #192. There were some weird permission things (files in tmp were owned by pulsys, but I had to run commands as deploy) so I had to do some permission stuff to get it to upload, but they're there now.

Ah - the files weren't missing, they just aren't called imagecat-disk - they're imagecat-temp or imagecat-temp2 or imagecat-temp3, so the above s3 prefix command wasn't right.

The reason is there are temp directories that look like old staging directories. For example: /var/tmp/imagecat/disk20/temp2/0048/A2380/0000.0045.tiff.
There's also a /var/tmp/imagecat/disk20/0048/A2380/ and they're exactly the same. I think we can safely ignore the temp dirs:
diff <(ls /var/tmp/imagecat/disk20/temp2/0048/A2380/) <(ls) (empty)


Helm 3.5.0 releases disappear if timed out

How to reproduce:

Do helm upgrade command, and set --timeout (with some value) and --wait flags.
Press CTRL+C to terminate the helm upgrade operation.
The helm upgrade command will fail.
Do helm ls command, and now you won't be able to see the release.
Try to do helm upgrade command again for this release.
You will get the following error: another operation (install/upgrade/rollback) is in progress.

What I would expect:
I would expect that the release will be in a "failed" state, and won't disappear from the releases list. In addition, I would expect that the second helm upgrade would work, and fix the release to the desired state.
Why I would expect that:
Releases may fail from time to time due to timeouts or abruptions, and retrying helm upgrade command is the desired solution in these cases.
A workaround:
If you get another operation (install/upgrade/rollback) is in progress error, you can rollback/delete the release, and then do helm upgrade command again. However, as I stated, I don't think this is the desired behavior.
Cloud Provider/Platform (AKS, GKE, Minikube etc.): AKS

Can you check the status of the release with helm history?

Can you check the status of the release with helm history?

@bacongobbler, yes:

I checked it more deeply, and I think that the issue happens when the helm client closes the connection with the cluster (for example, when you press CTRL+C during helm upgrade with wait).
I'm updating my issue with a more accurate description.

@bacongobbler, yes:

I checked it more deeply, and I think that the issue happens when the helm client closes the connection with the cluster (for example, when you press CTRL+C during helm upgrade with wait).
I'm updating my issue with a more accurate description.

Also noticed that behavior on my side. The worst thing that happens that the deployment is not longer listed.

Also noticed that behavior on my side. The worst thing that happens that the deployment is not longer listed.

My history looks like this:
817     	Fri Jan 29 03:10:43 2021	superseded	middleware-1.0.0	           	Upgrade complete
818     	Fri Jan 29 10:23:58 2021	failed    	middleware-1.0.0	           	Upgrade "middleware-master" failed: timed out waiting for the condition
819     	Fri Jan 29 10:46:01 2021	deployed  	middleware-1.0.0	           	Upgrade complete

My history looks like this:
817     	Fri Jan 29 03:10:43 2021	superseded	middleware-1.0.0	           	Upgrade complete
818     	Fri Jan 29 10:23:58 2021	failed    	middleware-1.0.0	           	Upgrade "middleware-master" failed: timed out waiting for the condition
819     	Fri Jan 29 10:46:01 2021	deployed  	middleware-1.0.0	           	Upgrade complete

@Alien2150 helm rollback worked for me as an alternative to deleting the secret.

Will try next time @guy-microsoft. Was not seeing the release anymore

You're right @Alien2150, it disappears, but if know the name of the missing release, rolling it back will restore it. It's not the ideal solution, but it's the best we can do until the issue is fixed.

I fixed this with helm delete $RELEASE. Still, annoying that these broken releases don't show up in helm list.

Some news on this?

still having this issue on helm v3.6.3

This issue is still there with 3.7.0


Host Fee Share Percent

Problem
Foundation is working with Platform Tips for Credit Cards and PayPal. As part of their custom plan, the host fee is 5% and we don't want to  have it shared following hostFeeSharePercent.
Solution
We need to  implement custom settings credidCardHostFeeSharePercent and paypalHostFeeSharePercent and set them to 0 for the Foundation host.
Todo

[x] support credidCardHostFeeSharePercent and paypalHostFeeSharePercent
[x] update foundation settings with credidCardHostFeeSharePercent and paypalHostFeeSharePercent
[ ] update Transactions in March for foundation hostFeeSharePercent=0 isSharedRevenue=false, delete settled
[ ] verify that hostFeeShare and pendingHostFeeShare metrics are accurate for foundation
[ ] calculate how much is owed to foundation
[ ] add the owed money to foundation settlement invoice on April 1st

Let's talk about this in our meeting on Wednesday. The exceptional setup for OCF came about because of all the experimentation we were doing. Now we've clarified how platform tips and fees will work for all the other hosts, and it could be worth it to discuss if OCF should actually be treated as such an exception, or should just go on a much simpler option of the regular host plan with a 50% rev share instead of 15%. If we keep going with the unique plan where OCF has different fees depending on payment method, we should be clear about our justification for why that is.

Based on this morning discussion, we're moving forward with the original plan. Specs are up to date.

To calculate the amount owed to Foundation this month.
SELECT
  t."HostCollectiveId",
  DATE_TRUNC('month', t."createdAt") AS "month",
  SUM("hostFeeInHostCurrency" * CAST(t."data"->>'hostFeeSharePercent' AS NUMERIC) / 100) AS "owed"
FROM
  "Transactions" t
WHERE
  t."createdAt" > '01-03-2021'
  AND t."HostCollectiveId" = 11049
  AND t."data"->>'isSharedRevenue' = 'true'
GROUP BY
  t."HostCollectiveId", "month"


fix: flush buffer before stopping

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.XLPE seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Fix uninitialized constant when checking for Homebrew (Fixes #119)

Recipe crashes if homebrew cookbook isn't referenced. See #119.
Additionally, adds support for Homebrewalt.

Closing due to inactivity.
If this is still an issue please reopen or open another PR.
Alternatively drop by the #sous-chefs channel on the Chef Community Slack and we'll be happy to help!
Thanks,
Sous-Chefs


[Issue] CVE-2023-23397 script - Failed to clear property

Describe the issue
CVE-2023-23397 script version <IP_ADDRESS>3
Expected behavior
When running the script in cleanup mode with "ClearProperty" parameter, the cleanup failed
With the parameter "ClearItem", the object is correctly deleted.
Script Output

Unfortunately, the script doesn't return any explanation
Additional context
Exchange servers are installed with french language (system and UI). Same for Exchange 2019.
I have followed the prerequisites (throttling policy and impersonation role)
My admin account has a mailbox and is member of these accounts

It also has the correct throttling policy

The csv file I am using for the tests

The cleanup command I am running directly from an Exchange server
.\CVE-2023-23397.ps1 -Environment Onprem -DLLPath .\Microsoft.Exchange.WebServices.dll -SkipVersionCheck -CleanupAction ClearProperty -CleanupInfoFilePath .\AuditResults_20230321_090848.csv -EWSServerURL https://<IP_ADDRESS>/EWS/Exchange.asmx -IgnoreCertificateMismatch
I have tried without the EWSUrl parameter, same result.
Please note that with a previous version of the script, I had these error messages (a quite bit explicit)

Translated error message : Exception calling 'Update' with '1' argument(s): 'At least one recipient is invalid., A message cannot be sent because it does not contain a recipient. »

I hope I haven't missed something obvious. Feel free to ask me anything.
Thank you for the script and for your help

@GeoCookie I am not sure why you are still facing this issue. It should have been resolved with this PR #1600
Based off the error message, it seems like we are not determining that the entry is a Calendar type. But the image does report it as one.

It looks like this occurs when the CSV is opened in Excel on a machine where the default delimiter is not a comma. When choosing Save As, Excel sees the file as having only one column and surrounds the entire line with a quote.
We can undo this with some PowerShell script. To fix an existing file in this state:
$inputFile = "C:\bad.csv"
$newFileName = "C:\fixed.csv"
$importedData = Import-Csv $inputFile
$headerLine = ($importedData[0] | Get-Member | ? { $_.MemberType -eq "NoteProperty" }).Name
$headerLine | Out-File $newFileName
$importedData | % { $_."$headerLine" } | Out-File $newFileName -Append

We might be able to fix these on the fly in the CVE script by doing this:
$inputFile = "C:\bad.csv"
$importedData = Import-Csv $inputFile
$headerLine = ($importedData[0] | Get-Member | ? { $_.MemberType -eq "NoteProperty" }).Name
$fixedData = $importedData | % { $_."$headerLine" } | ConvertFrom-Csv -Header ($headerLine.Split(","))

I tried again with two new calendar appointments and two new tasks.
I ran the script one time in audit mode, change the Cleanup values to "Y" with notepad and ran the script in "CleanProperty" mode. This time, everything went fine. All four objects are correctly cleaned!
If I run the script in cleanup mode a second time with the same csv file, I got the same errors.

So maybe at some point, the script correctly removed the property and I didn't check correctly the appointments on Outlook. But I am sure the script throws an error at each run.
Anyway, I am now able to run the script against our mailboxes.
Thanks a lot for your help!

Leaving this one open until we figure out how to address this in the script.


respects hide_last_name flag when showing to a non-admin user

a few places (users/index and users/show) will show the full name, but only to admins

Resolves #153

+1


DOC-3006 add metric snapshot bytes sent/recv

Addresses: DOC-3006

Adds range.snapshots.rcvd-bytes and range.snapshots.sent-bytes to v22.1 metrics listing. This is the only place in the corpus we list / refer to these raw metrics, so this is a tiny PR.

Note: the table housing these metrics is prefaced by a note that explains: This list is taken directly from the source code ... Does this indicate that this list is programmatically generated elsewhere and that I should instead be editing the source script?
metric-names.md

This change is

@aayushshah15 friendly ping for review. Thanks!

@MichaelTrestman sweet, TYFR!


Protractor Reporter not following config

Hello,
var SpecReporter = require('jasmine-spec-reporter');     jasmine.getEnv().addReporter(new SpecReporter({       displayStacktrace: 'summary',       displayFailuresSummary: false     }));
Doesn't seem to be working anymore. The failures summary still show.
Also, it seems like I'm getting the dot report even though I have print: function() {} in the jasmineNodeOpts.
Does this have to do with the newest version of protractor being 4.0.9, at the time of this issue?

Started an issue at protractor, seeing if it's a problem on their end, or JSP.
https://github.com/angular/protractor/issues/3609

I have same problem

It looks like this is an issue with Jasmine. Either downgrade to Protractor 4.0.8 or wait until 4.0.11 is released as they downgraded Jasmine until the fix is issued there.


Update reference docs for kubectl commands.

This PR updates the reference docs for the kubectl commands. The kubectl commands are documented in the one topic.
This file is generated by kubernetes-incubator/reference-docs.

This change is

@mengqiy Can you take a quick look at this generated topic? kubectl commands 1.9

@mengqiy Can you take a quick look at this generated topic? kubectl commands 1.9

I think it looks fine.


[Bug] progress_bar_refresh_rate causes error with pytorch-lightning>=1.7

For a new user installing neuralforecast and all dependencies, they will hit an error when calling model.forecast() when building a model or even just following an example step for step (ie the mqnhits example)
The error is: TypeError: init() got an unexpected keyword argument ‘progress_bar_refresh_rate’
From the pytorch-lightning docs: "progress_bar_refresh_rate has been deprecated in v1.5 and will be removed in v1.7. Please pass TQDMProgressBar with refresh_rate directly to the Trainer’s callbacks argument instead."
Seems like progress_bar_refresh_rate is passed to many pl.Trainer objects in this repo - I think it'd make sense to remove those so that the newest version of pytorch-lightning is still valid, and I'm not sure it would hurt performance in any way.
Alternatively, limit the lightning version to <1.7.
Either way, should solve the issue. I am really enjoying this library, thanks for all the good work.

I met the same problem and solved it by pip uninstall pytorch-lightning and pip install pytorch-lightning==1.6.
Thanks for @chillerobscuro.
And this library helps me a lot, thanks for Nixtla/neuralforecast and its contributors.

您好，我已经收到您的邮件，祝生活愉快。


Expressions using builtins.fromTOML + builtins.readFile can't be evaluated on the installer image.

In the installer image, we copy a modified Nixpkgs into the Nix store, to use as the initial version of the "nixos" channel:
https://github.com/NixOS/nixpkgs/blob/fd57780d9c3c311d600a531fb91dbfac17be8d0e/nixos/modules/installer/cd-dvd/channel.nix#L15-L29
The problem is that, because we modify Nixpkgs in a derivation, we create a derivation with references.  So if, in Nixpkgs, read a file using builtins.readFile, we will end up with a string with context, which cannot be passed to e.g. builtins.fromTOML.
As a practical example, it is not possible to evaluate makeInitrdNGTool on the installer image, the expression for which tries to import and parse a Cargo.lock file:
error: the string '# This file is automatically @generated by Cargo.
       # It is not intended for manual editing.
       [[package]]
       name = "make-initrd-ng"
       version = "0.1.0"
       ' is not allowed to refer to a store path (such as '/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh')

       at /nix/store/c1032pb33hdykvby2layk3h27ckn3czr-nixos-22.05.git.a323570a264/nixos/pkgs/build-support/rust/import-cargo-lock.nix:36:15:

           35|
           36|   packages = (builtins.fromTOML lockFileContents).package;
             |               ^
           37|
(use '--show-trace' to show detailed location information)

This also causes problems in some NixOS tests, that use the same module for installing a Nixpkgs channel into the test VM.  For example, since https://github.com/NixOS/nixpkgs/pull/164943 the containers-imperative test fails, because of the same makeInitrdNGTool error.
I don't think it would be safe to just discard the string contexts here, so we need to find some way to make sure that the Nixpkgs channel in the image does not have derivation dependencies.  So it needs to be either outside the store, or a fixed-output derivation.  I've thought of a couple of ways to do that so far, but neither is a universally suitable solution:

We could not modify Nixpkgs, meaning that the store path would be the result of simply copying Nixpkgs to the store (after passing it through a source filter, which is still fine), and would therefore be fixed-output.  This would be fine for the tests, but wouldn't work for the installer image because Nixpkgs would be missing the .git-revision, .version-suffix, and svn-revision files that we'd have no way to add.  Even using a symlink farm wouldn't work because it creates a derivation with dependencies.

We could re-copy Nixpkgs into a new store path, using nix-store --add, making it fixed-output.  This would solve our problem but it's slow and requires a lot of memory.

So neither of my solutions so far would be an acceptable fix for the installer.  Possibly we need some new mechanism in the image building code to turn a non-fixed-output derivation on the build system into a fixed-output path in the generated image, or perhaps such a thing already exists, and I'm not aware of it?

It looks like this behaviour was introduced in Nix 2.6.0 with PR https://github.com/NixOS/nix/pull/1643.

Then maybe ping @abbradar https://github.com/NixOS/nix/pull/1643#issuecomment-1009398264

https://github.com/NixOS/nixpkgs/pull/182980 will fix this for Cargo lockfiles.  I don't know if there are other uses of fromTOML in Nixpkgs we'd also need to evaluate.

This also happens with builtins.fromJSON -- after applying the above patch to fix the TOML issue, I can't build because of matrix-appservice-irc. 😕

Would this be fixed by filtering out the refs that don't appear in the string, in prim_readFile? Are there any downsides to doing that?
Implementation-wise, this could probably reuse some code from the reference scanner.

Can someone see if this is fixed with nix/daf1423a4a07f3d8a0c88a590b4ec454d64233cc? (or tell me how to test it)

I'm going to assume this is fixed for now.

How did we fix it in 22.11 without either discarding the Cargo lockfile context or adding that Nix patch (which seemingly landed in 2.12)?

I don't know. I couldn't reproduce the issue.

What environment were you unable to reproduce it under, a clean copy of 22.11?

This was probably 24 days ago when I asked for instructions to reproduce this, and I probably built the ISO from master then.
But we're taking this backwards: can you provide instructions to reproduce the issue (at any point in the history, even if it's fixed now)?
Also the OP says containers-imperative is failing but it doesn't seem to have failed on Hydra since before this issue was created.

But we're taking this backwards: can you provide instructions to reproduce the issue (at any point in the history, even if it's fixed now)?

nixos-install is how I reproduced it, see https://github.com/NixOS/nixpkgs/issues/175125#issuecomment-1229692308
(Apparently) running the os-prober NixOS test also suffices

Also the OP says containers-imperative is failing but it doesn't seem to have failed on Hydra since before this issue was created.

Yeah, ack.
@alyssais, do you happen to remember the circumstances surrounding this failure -- maybe it was unrelated to this (since it's not failing on Hydra)?

(Apparently) running the os-prober NixOS test also suffices

That test also runs fine for me (even on https://github.com/NixOS/nixpkgs/pull/182980 's base), and has been running fine on Hydra.

This also happens with builtins.fromJSON -- after applying the above patch to fix the TOML issue, I can't install from an ISO built on master because of matrix-appservice-irc.

Could you clarify this? What call to builtins.fromJSON were you referring to? Do I need to enable matrix-appservice-irc to reproduce this?

@ncfavier So I've spent some time trying to repro this, and was only able to because I had an ISO laying around that I originally reproduced the issue on.
Since ISO builds use the local source folder, this is a somewhat dirty tree, but I can't pinpoint what exactly in that tree would be causing this error. If you'd like a copy of the tree (and optionally the ISO), I can send it over if you're still interested in looking into this.

Sure, I'll take anything at this point.

@ncfavier I extracted this tree from the ISO (SHA-512: 416a25a1ebefde65a7b6d16da022a246b7fd97bdcd7be04b012ad4c13b09184960c880673b9e79eeb578dd10993d4db6f97ff3ad2efcb73fa24460f25c30d4d3).
Here's some info that I found when debugging this yesterday:

This is d9a636e9c634940b8afc1aaa71069d9e5c2a71be
nix/.../nixos/nixpkgs is a recursive symlink, delete it to save yourself from pain
Building the docs from a fresh checkout and trying to repro this does not work, so I'm guessing it may be the random derivations for aarch64-darwin laying around in the tree?
To reproduce: nix-build nixos/release.nix -A iso_minimal.<your system>, boot the ISO, setup /mnt, and run nixos-install. Since the Cargo lockfile context discarding patch has been applied, you should see an eval error relating to matrix-appservice-irc's usage of lib.importJSON.

Alright, I know what's happening. The channel derivation only turns out to have references (and hence readFile <nixpkgs/foo> to have a non-empty context) if some of the build inputs to the channel derivation are referenced in the nixpkgs tree. In your case there are files named one and two containing derivations that reference /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh, and since the default builder is also used by runCommand, it gets included in the context.
I have confirmed that Nix 2.12 (https://github.com/NixOS/nix/pull/7260) fixes the underlying problem on the Nix side. On the nixpkgs side, the problem is that random untracked files can end up on the ISO or in the tests, which harms reproducibility as I mentioned in https://github.com/NixOS/nixpkgs/pull/204178#issue-1473475374 (I'm guessing that people who observed tests failing also had derivations lying around in their nixpkgs).
Flakes would solve this issue, but it would be nice if we could also just tell cleanSource not to include untracked files somehow.
Thanks for your help in troubleshooting this!


Doesn't work on Windows machines.

Set up all the paths, however doesn't work on windows machines due to how arguments are formatted.

What version of Matlab are you using?

I apologize but the documentation indicates that Matlab will not run in the command prompt. I will update the extension to say Mac only. Sorry for the inconvenience.

Thanks, I was editing the extension and was able to get a working copy after hacking it together. Though my solution isn't convenient, it is better than dealing with the matlab editor.
Thanks for making this. I'll likely use it on another platform.

 If you don't mind me asking, what did you do to get it to work? Maybe I can work it into the extension or invite you to the repo to implement it yourself if you'd like.


Reduce allocations when writing

PR Details
Fix #494
If a row is full, don't bother allocating a new one, just return it.
Use the last populated row as a hint for the size of new rows.
Simplify checkSheet to remove row map (tiny performance improvement, mostly just easier to follow now)
Related Issue
#494
Motivation and Context
Reduce memory consumed while creating a document
How Has This Been Tested
Added a new benchmark, ensured tests continue to pass
Master
BenchmarkWrite-8   	       2	 601920270 ns/op	207277296 B/op	 1893770 allocs/op
This fix
BenchmarkWrite-8   	       2	 567284257 ns/op	127115028 B/op	 1833744 allocs/op
Rows:
Master
BenchmarkRows-8   	   10000	    362571 ns/op	 2010561 B/op	      30 allocs/op
This fix
BenchmarkRows-8   	   10000	    349607 ns/op	 2010560 B/op	      30 allocs/op
Performance wise, the change I made to rows barely makes a difference, most of that was likely stack allocated anyway. This is just simpler.
Types of changes

[ ] Docs change / refactoring / dependency upgrade
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Codecov Report

Merging #514 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master    #514      +/-   ##
=========================================
+ Coverage   96.89%   96.9%   +<.01%
=========================================
  Files          26      27       +1
  Lines        5546    5549       +3
=========================================
+ Hits         5374    5377       +3
  Misses         94      94
  Partials       78      78

Impacted Files
Coverage Δ

xmlWorksheet.go
100% <100%> (ø)

sheet.go
96.13% <100%> (+0.05%)
:arrow_up:

excelize.go
94.93% <100%> (-0.22%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6abf8bf...bf9a835. Read the comment docs.

I should clarify, the performance of reading a file barely changed which is what I was referring to. This cuts the memory used while writing a file by about 40%.

LGTM, thanks @mlh758.

I tested the version 1.4.1 and 2.1.0 of excelize, and the writing time improved a lot, the memory use is significantly better, thanks for these, but I think is not enough, because for a lot of information the memory use is a lot too. For example, to write 5 million rows with 16 columns each row the program takes around 4GB of ram.
Here is the example I used: https://gist.github.com/mperezguendulain/393d59f285608fa4b7142ba5120b5109
It will great if it excelize could use a small chunk of memory.


Issue 66

Description

Motivation and Context

How Has This Been Tested?

Checklist:

[ ] I have submitted a CLA form
[ ] If applicable, I have updated the documentation accordingly.
[ ] If applicable, I have added tests to cover my changes.

Codecov Report

Merging #67 into master will decrease coverage by 0.54%.
The diff coverage is 9.67%.

@@            Coverage Diff             @@
##           master      #67      +/-   ##
==========================================
- Coverage   37.33%   36.78%   -0.55%
==========================================
  Files           5        5
  Lines         608      598      -10
==========================================
- Hits          227      220       -7
+ Misses        381      378       -3

Flag
Coverage Δ

#CouchDB
36.78% <9.67%> (-0.55%)
:arrow_down:

#CouchDBSample
36.78% <9.67%> (-0.55%)
:arrow_down:

Impacted Files
Coverage Δ

Sources/CouchDB/ConnectionProperties.swift
40.9% <ø> (ø)
:arrow_up:

Sources/CouchDB/CouchDBUtils.swift
43.1% <ø> (ø)
:arrow_up:

Sources/CouchDB/UsersDatabase.swift
0% <0%> (ø)
:arrow_up:

Sources/CouchDB/Database.swift
42.58% <25%> (-0.87%)
:arrow_down:

Sources/CouchDB/CouchDBClient.swift
33.63% <6.25%> (-0.44%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3609112...e7d0780. Read the comment docs.


FAQ styles updates

Applied fixes suggested by @yumeforever in #34.

Looks good to me. Merge into the feature branch.


Adding Hover Effect to all the Buttons

Adding Hover Effect to all the Buttons
Assign this to me!!

can you please assign this to me?

can you please assign this to me?

can you please assign this to me?

Hi, which of our websites is this issue for?

Hi, which of our websites is this issue for?

https://chimoney.io/     this one

CAN YOU PLEASE ASSIGN THIS ISSUE TO ME

The website is not in a public repo


Yeah, i know. it's huge. Update module and 1.19 tests

Hi team!
I know, this PR is huge, but it contains a lot of new stuff:
Main changes of this HUGE pr:

Add license checks at CI
Add 1.19.0 testing
Update all modules (including major changes in Grafana)
Add kube-proxy-exporter (daemonset to expose metrics working in all kube setups) (suggested by @lnovara )
Add eks-sm (suggested by @lnovara )
Add grafana internal dashboard
Refactored all dashboards: (suggested by @lnovara )

Before this PR, all dashboards got deployed with Grafana. This can cause some issues having dashboards without metrics/product
With this PR, every product/package has its own serviceMonitor + grafana Dashboard (as configmap).

This caused a lot of changes and headaches. In addition, this update provides a tested upgrade path described in the release notes: docs/releases/v1.10.0.md

PD: Special thanks to @lnovara rises some issues/ways to improve the module.
PD2: Pipeline passing: https://ci.sighup.io/sighupio/fury-kubernetes-monitoring/900

LGTM, awesome work! :shipit: ❤
Minor non-blocking comments follow:
* I think that GlusterFS and Kong stuff need to be moved to a separate repository (which already exists AFAIK) because they're not shipped by default

I was thinking about it. I don't know what to do. Maybe I can delete it, and if someone needs the dashboards, they are available in the previous releases of this module.

* the `kube-proxy-exporter` doesn't fit 100% as a name to me, I would have called it `kube-proxy-metrics` but I do realize I'm just nipicking 😅

Renaming, I think it makes more sense kube-proxy-metrics. my fault!

* did you have any chance to check if dashboards and alerts from the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project got updated as well?

Nop, I'll take a look. If there is something really interesting, I'll include it.

P.S. I do understand the huge effort you put into this PR but I think we should avoid this big PRs since it makes really difficult to do a thorough review even though the PR title really makes up for it! 😄

Ye... I know, it was a huge PR because of various reasons... The next release should be smaller.

I'll remove the kong and glusterfs dashboards. If someone requires it, feel free to recover it from a previous release :)


[WIP, ENH] Refactoring of IO and processing

A relatively dramatic refactoring of the API, including better I/O handling of files, improved preprocessing and cleaning of expression data, and integration of many of the fantastic recommendations laid out in Arnatkevičiūte et al., 2018.
Notable changes:

Downloading expression files now converts the two largest (MicroarrayExpression.csv, and PACall.csv) to parquet format for dramatically faster loading in the future.
Functionality has been added for intensity-based filtering of microarray expression probes based on whether the probe exceeds background noise above a specified threshold. Refer to documentation in abagen.processing.filter_probes() for more details.
Functionality has been added for probe selection based on "differential stability" (Hawrylycz et al., 2015) across donor samples. Refer to documentation in abagen.processing.get_stable_probes() for more details.
Functionality has been added to perform better correction for donor variation using a scaled robust sigmoid (SRS) normalization procedure. Refer to documentation in abagen.processing.normalize_expression() for more details.

This is still a work-in-progress, so I'll be updating this more as I continue to integrate the full pipeline in the paper referenced above.

[ ] Ready to merge

Codecov Report

Merging #4 into master will decrease coverage by 3.25%.
The diff coverage is 25.78%.

@@            Coverage Diff             @@
##           master       #4      +/-   ##
==========================================
- Coverage   34.29%   31.04%   -3.26%
==========================================
  Files           7        8       +1
  Lines         242      364     +122
==========================================
+ Hits           83      113      +30
- Misses        159      251      +92

Impacted Files
Coverage Δ

abagen/info.py
100% <100%> (ø)
:arrow_up:

abagen/utils.py
38.96% <18.18%> (-15.59%)
:arrow_down:

abagen/allen.py
12.84% <20%> (+0.97%)
:arrow_up:

abagen/processing.py
23.63% <23.63%> (ø)

abagen/io.py
22.8% <24.44%> (-9.02%)
:arrow_down:

abagen/datasets.py
30.3% <40%> (-0.74%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c4ce79...4adc10f. Read the comment docs.

Pull Request Test Coverage Report for Build 6

41 of 159 (25.79%)  changed or added relevant lines in 6 files are covered.
6 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-3.3%) to 31.044%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

abagen/datasets.py
2
5
40.0%

abagen/allen.py
3
15
20.0%

abagen/utils.py
6
33
18.18%

abagen/io.py
11
45
24.44%

abagen/processing.py
13
55
23.64%

Files with Coverage Reduction
New Missed Lines
%

abagen/datasets.py
1
30.3%

abagen/io.py
5
22.81%

Totals

Change from base Build 4:
-3.3%

Covered Lines:
113

Relevant Lines:
364

💛  - Coveralls


Browse Dataset: Sorting BG Color

Is your enhancement related to a problem? Please describe.
Nothing urgent, but after changing the selection of the Sorting, a light grey background is visible around the button. This shouldn't be visible.
Describe the solution you'd like
Delete this grey background
Describe alternatives you've considered
None
Additional context
https://user-images.githubusercontent.com/59613691/203264860-a776a7d4-ec71-4468-b64c-17a54d38216d.mp4

So the bug is more that relevance is displayed when we have not searched anything right ?


Self-organizing hash table to improve the performance of bucket aggregations

Description
Some aggregations know ahead of time about the number of buckets and owning ordinals (eg. range aggregations), thus, are able to map densely and directly. Most other aggregations cannot predict this, thus, rely on a hash table lookup to identify the owning bucket ordinal.
Aggregations in the "nyc_taxis" and the "http_logs" workloads revealed that a major share of the CPU time is spent in these lookups. As these lookups are performed for each document matched by the query, even a small improvement in the hash table can have a large overall improvement.
Latest Approach: FastLongHash
This PR introduces FastLongHash which is a drop-in replacement for LongHash. It organizes itself by moving keys around dynamically in order to reduce the longest probe sequence length (PSL), which makes lookups faster as keys are likely to be found in the same CPU cache line. It also optimizes lookups for recently added keys, making it useful for aggregations where keys are correlated across consecutive hits.

The expected longest PSL in a full table is Θ(ln n).
Inserts are slightly expensive due to additional bookkeeping, but lookups are faster. As the number of hits >> number of buckets, most keys only require a lookup (i.e. the number of unique inserts is less). This makes the "add" operation faster overall.
Keys are likely to be correlated across consecutive hits (eg. belonging to the same minute/hour/day in the case of date histogram aggregation). By placing a "recent" key closer to its ideal home slot, we can reduce the number of probes on such workloads.
Faster bit-mixers were evaluated (#7345), but their distribution were unacceptable.

Probe Sequence Length
It is the number of probes needed to find the desired key in the hash table.
The average PSL is nearly identical. This is expected as the sum of PSLs is preserved after keys are rearranged.

The longest PSL is reduced significantly, thus improving the worst-case performance of certain keys.

Memory Usage
The memory usage is identical. This is expected as the initial capacity, growth rate, and the underlying table implementation remains the same. Only the encoding and placement strategy has changed.

JMH Benchmarks
Uncorrelated Adds
This represents workloads where keys arrive in random order. Performance is similar on small number of keys, but gets better with large number of keys, mainly due to the reduced number of memory lookups.

Correlated Adds
This represents workloads where adjacent keys are likely to be similar (eg. bucketing on timestamp where two consecutive hits are likely to share the same "hour"). Performance is significantly better as keys are likely to be found at their ideal home slot.

Distinct Adds
This represents workloads with high cardinality, i.e., each hit maps to a different key. Performance gets better due to the reduced number of memory lookups.

OSB Benchmarks
nyc_taxis
There is around 15-23% improvement on Intel and 8-12% on Graviton.
c6i.2xlarge

Metric
Task
Baseline
Contender
Diff
Unit

Min Throughput
autohisto_agg
1.5042
1.50298
-0.00122
ops/s

Mean Throughput
autohisto_agg
1.50684
1.50485
-0.00199
ops/s

Median Throughput
autohisto_agg
1.50625
1.50443
-0.00182
ops/s

Max Throughput
autohisto_agg
1.51209
1.50857
-0.00351
ops/s

50th percentile latency
autohisto_agg
345.824
267.783
-78.041
ms

90th percentile latency
autohisto_agg
354.344
274.595
-79.7489
ms

99th percentile latency
autohisto_agg
357.427
292.599
-64.8274
ms

100th percentile latency
autohisto_agg
359.794
296.043
-63.7512
ms

50th percentile service time
autohisto_agg
344.754
266.345
-78.4085
ms

90th percentile service time
autohisto_agg
353.373
273.745
-79.6275
ms

99th percentile service time
autohisto_agg
356.595
291.844
-64.7506
ms

100th percentile service time
autohisto_agg
358.902
294.78
-64.1225
ms

error rate
autohisto_agg
0
0
0
%

Min Throughput
date_histogram_agg
1.50502
1.50575
0.00072
ops/s

Mean Throughput
date_histogram_agg
1.50819
1.50949
0.0013
ops/s

Median Throughput
date_histogram_agg
1.50746
1.50865
0.00119
ops/s

Max Throughput
date_histogram_agg
1.51452
1.51704
0.00252
ops/s

50th percentile latency
date_histogram_agg
330.51
281.653
-48.8575
ms

90th percentile latency
date_histogram_agg
337.787
287.68
-50.1066
ms

99th percentile latency
date_histogram_agg
347.023
307.381
-39.6416
ms

100th percentile latency
date_histogram_agg
353.087
324.471
-28.6164
ms

50th percentile service time
date_histogram_agg
329.366
280.401
-48.9651
ms

90th percentile service time
date_histogram_agg
336.458
286.567
-49.8908
ms

99th percentile service time
date_histogram_agg
345.665
305.908
-39.7566
ms

100th percentile service time
date_histogram_agg
352.358
323.602
-28.7562
ms

error rate
date_histogram_agg
0
0
0
%

c6g.2xlarge

Metric
Task
Baseline
Contender
Diff
Unit

Min Throughput
autohisto_agg
1.50132
1.50246
0.00114
ops/s

Mean Throughput
autohisto_agg
1.50213
1.50401
0.00188
ops/s

Median Throughput
autohisto_agg
1.50195
1.50365
0.0017
ops/s

Max Throughput
autohisto_agg
1.50376
1.50705
0.00329
ops/s

50th percentile latency
autohisto_agg
558.64
493.889
-64.7505
ms

90th percentile latency
autohisto_agg
560.574
503.981
-56.5928
ms

99th percentile latency
autohisto_agg
561.542
519.913
-41.6294
ms

100th percentile latency
autohisto_agg
562.369
519.959
-42.4109
ms

50th percentile service time
autohisto_agg
557.821
492.454
-65.3662
ms

90th percentile service time
autohisto_agg
559.619
502.671
-56.9488
ms

99th percentile service time
autohisto_agg
560.985
518.241
-42.7436
ms

100th percentile service time
autohisto_agg
561.36
518.883
-42.4771
ms

error rate
autohisto_agg
0
0
0
%

Min Throughput
date_histogram_agg
1.50196
1.50266
0.00071
ops/s

Mean Throughput
date_histogram_agg
1.5032
1.50433
0.00113
ops/s

Median Throughput
date_histogram_agg
1.50293
1.50395
0.00102
ops/s

Max Throughput
date_histogram_agg
1.50562
1.50764
0.00202
ops/s

50th percentile latency
date_histogram_agg
531.506
486.349
-45.157
ms

90th percentile latency
date_histogram_agg
532.87
488.923
-43.9464
ms

99th percentile latency
date_histogram_agg
533.273
511.675
-21.598
ms

100th percentile latency
date_histogram_agg
534.678
514.076
-20.6019
ms

50th percentile service time
date_histogram_agg
530.393
485.487
-44.906
ms

90th percentile service time
date_histogram_agg
531.645
487.663
-43.9819
ms

99th percentile service time
date_histogram_agg
532.286
510.785
-21.5009
ms

100th percentile service time
date_histogram_agg
532.896
513.15
-19.7462
ms

error rate
date_histogram_agg
0
0
0
%

http_logs
There is around 23% improvement on Intel and 11% on Graviton.
c6i.2xlarge

Metric
Task
Baseline
Contender
Diff
Unit

Min Throughput
hourly_agg
0.20066
0.200747
9e-05
ops/s

Mean Throughput
hourly_agg
0.200913
0.201033
0.00012
ops/s

Median Throughput
hourly_agg
0.200878
0.200994
0.00012
ops/s

Max Throughput
hourly_agg
0.201311
0.201485
0.00017
ops/s

50th percentile latency
hourly_agg
1675.88
1284.36
-391.512
ms

90th percentile latency
hourly_agg
1726.11
1336.14
-389.966
ms

99th percentile latency
hourly_agg
1834.83
1373.69
-461.141
ms

100th percentile latency
hourly_agg
1866.72
1412.86
-453.857
ms

50th percentile service time
hourly_agg
1672.5
1280.77
-391.733
ms

90th percentile service time
hourly_agg
1721.53
1332.07
-389.458
ms

99th percentile service time
hourly_agg
1830.75
1370.11
-460.638
ms

100th percentile service time
hourly_agg
1862.45
1408.22
-454.229
ms

error rate
hourly_agg
0
0
0
%

c6g.2xlarge

Metric
Task
Baseline
Contender
Diff
Unit

Min Throughput
hourly_agg
0.200508
0.200526
2e-05
ops/s

Mean Throughput
hourly_agg
0.200702
0.200727
2e-05
ops/s

Median Throughput
hourly_agg
0.200676
0.200699
2e-05
ops/s

Max Throughput
hourly_agg
0.201008
0.201043
3e-05
ops/s

50th percentile latency
hourly_agg
2445.65
2167.05
-278.599
ms

90th percentile latency
hourly_agg
2448.21
2168.8
-279.413
ms

99th percentile latency
hourly_agg
2462.3
2173.48
-288.816
ms

100th percentile latency
hourly_agg
2543.93
2187.5
-356.429
ms

50th percentile service time
hourly_agg
2442.78
2163.35
-279.429
ms

90th percentile service time
hourly_agg
2444.92
2165.02
-279.904
ms

99th percentile service time
hourly_agg
2459.01
2170.67
-288.345
ms

100th percentile service time
hourly_agg
2540.64
2183.31
-357.334
ms

error rate
hourly_agg
0
0
0
%

Previous Approach: LongRHHash
This PR introduces LongRHHash which is a drop-in replacement for LongHash. It combines ideas from Robin Hood hashing and Cuckoo hashing to improve performance. Here's a summary:

Keys are moved around dynamically in order to reduce the probe sequence length (PSL). The expected longest PSL in a full table is Θ(ln n). Inserts are slightly expensive due to additional bookkeeping, but lookups are faster.

This reduces the worst-case lookup time of keys with a large PSL.
Keys are also more likely to be found in the same CPU cache line (typically 64 bytes).

As the number of hits >> number of buckets, most keys only require a lookup (i.e. the number of unique inserts is less). This makes the "add" operation faster overall.
Keys are likely to be correlated across consecutive hits (eg. belonging to the same minute/hour/day in the case of date histogram aggregation). A cheap hash function is used to look up recent keys, then falling back to a good hash function with linear probing.
Faster bit-mixers were evaluated (#7345), but their distribution were unacceptable.

Probe Sequence Length
It is the number of probes needed to find the desired key in the hash table.
The average PSL is nearly identical. This is expected as the sum of PSLs is preserved after keys are rearranged.

The longest PSL is reduced significantly, thus improving the worst-case performance of certain keys.

JMH Benchmarks
These benchmarks were performed on c6i.2xlarge (Intel) / c6g.2xlarge (Graviton) instances running OpenJDK 17.0.6. These results represent the time to perform 16.7M "adds" with varying number of distinct buckets.
Uncorrelated Adds
This represents workloads where keys arrive in random order. "Recent" lookups are ineffective for large hash tables, so the improvement solely comes from the rearrangement of keys.
(A) is the baseline, (B) is the contender without "recent" lookups, and (C) is the contender with "recent" lookups.

(B) was better than (A) throughout the spectrum.
(C) was much better at low occupancy, but performed poorly with larger table sizes (~150+) due to a cache-miss followed by a full linear probe.
(C) uses a fixed size (256) "recent" table which performs well until 60% occupancy. It is possible to grow this table or switch to (B) if the hash table grows larger. While it may improve performance on synthetic benchmarks, it may not translate to any real-world performance benefits.

Correlated Adds
This represents workloads where adjacent keys are likely to be similar (eg. bucketing on timestamp where two consecutive hits are likely to share the same "hour").
(A) is the baseline, (B) is the contender with "recent" lookups. Performance is consistent (no sawtooth pattern) and keeps improving as the number of buckets grow. This is because, with two independent hash functions, it is unlikely that both collide.

OSB Benchmarks
nyc_taxis
There is 14-17% improvement on Intel and 8-12% improvement on Graviton.

c6i.2xlarge details

Metric
Task
Baseline
Contender
Diff
Unit

Min Throughput
autohisto_agg
1.5042
1.50502
0.00082
ops/s

Mean Throughput
autohisto_agg
1.50684
1.50818
0.00134
ops/s

Median Throughput
autohisto_agg
1.50625
1.50746
0.00121
ops/s

Max Throughput
autohisto_agg
1.51209
1.51452
0.00243
ops/s

50th percentile latency
autohisto_agg
345.824
298.608
-47.2159
ms

90th percentile latency
autohisto_agg
354.344
307.344
-46.9998
ms

99th percentile latency
autohisto_agg
357.427
316.633
-40.7939
ms

100th percentile latency
autohisto_agg
359.794
326.847
-32.9466
ms

50th percentile service time
autohisto_agg
344.754
297.394
-47.3594
ms

90th percentile service time
autohisto_agg
353.373
305.879
-47.4936
ms

99th percentile service time
autohisto_agg
356.595
315.587
-41.0082
ms

100th percentile service time
autohisto_agg
358.902
326.042
-32.8601
ms

error rate
autohisto_agg
0
0
0
%

Min Throughput
date_histogram_agg
1.50502
1.50586
0.00084
ops/s

Mean Throughput
date_histogram_agg
1.50819
1.50967
0.00148
ops/s

Median Throughput
date_histogram_agg
1.50746
1.5088
0.00134
ops/s

Max Throughput
date_histogram_agg
1.51452
1.51738
0.00286
ops/s

50th percentile latency
date_histogram_agg
330.51
275.304
-55.2067
ms

90th percentile latency
date_histogram_agg
337.787
285.986
-51.8007
ms

99th percentile latency
date_histogram_agg
347.023
307.172
-39.8502
ms

100th percentile latency
date_histogram_agg
353.087
310.265
-42.8225
ms

50th percentile service time
date_histogram_agg
329.366
273.975
-55.3916
ms

90th percentile service time
date_histogram_agg
336.458
285.122
-51.3364
ms

99th percentile service time
date_histogram_agg
345.665
306.428
-39.2366
ms

100th percentile service time
date_histogram_agg
352.358
308.634
-43.7241
ms

error rate
date_histogram_agg
0
0
0
%

c6g.2xlarge details

Metric
Task
Baseline
Contender
Diff
Unit

Min Throughput
autohisto_agg
1.50132
1.50251
0.00119
ops/s

Mean Throughput
autohisto_agg
1.50213
1.50408
0.00195
ops/s

Median Throughput
autohisto_agg
1.50195
1.50373
0.00179
ops/s

Max Throughput
autohisto_agg
1.50376
1.50722
0.00346
ops/s

50th percentile latency
autohisto_agg
558.64
491.169
-67.471
ms

90th percentile latency
autohisto_agg
560.574
492.553
-68.021
ms

99th percentile latency
autohisto_agg
561.542
494.173
-67.3688
ms

100th percentile latency
autohisto_agg
562.369
510.678
-51.6918
ms

50th percentile service time
autohisto_agg
557.821
490.021
-67.7991
ms

90th percentile service time
autohisto_agg
559.619
490.908
-68.7118
ms

99th percentile service time
autohisto_agg
560.985
492.691
-68.2933
ms

100th percentile service time
autohisto_agg
561.36
508.64
-52.7199
ms

error rate
autohisto_agg
0
0
0
%

Min Throughput
date_histogram_agg
1.50196
1.50268
0.00072
ops/s

Mean Throughput
date_histogram_agg
1.5032
1.50435
0.00115
ops/s

Median Throughput
date_histogram_agg
1.50293
1.50397
0.00103
ops/s

Max Throughput
date_histogram_agg
1.50562
1.50768
0.00205
ops/s

50th percentile latency
date_histogram_agg
531.506
487.024
-44.4823
ms

90th percentile latency
date_histogram_agg
532.87
488.058
-44.8119
ms

99th percentile latency
date_histogram_agg
533.273
489.811
-43.4617
ms

100th percentile latency
date_histogram_agg
534.678
496.169
-38.5092
ms

50th percentile service time
date_histogram_agg
530.393
486.082
-44.311
ms

90th percentile service time
date_histogram_agg
531.645
486.96
-44.6849
ms

99th percentile service time
date_histogram_agg
532.286
488.544
-43.7412
ms

100th percentile service time
date_histogram_agg
532.896
495.547
-37.3498
ms

error rate
date_histogram_agg
0
0
0
%

http_logs
There is 25% improvement on Intel and 12% improvement on Graviton.

c6i.2xlarge details

Metric
Task
Baseline
Contender
Diff
Unit

Min Throughput
hourly_agg
0.20066
0.200759
0.0001
ops/s

Mean Throughput
hourly_agg
0.200913
0.20105
0.00014
ops/s

Median Throughput
hourly_agg
0.200878
0.20101
0.00013
ops/s

Max Throughput
hourly_agg
0.201311
0.201509
0.0002
ops/s

50th percentile latency
hourly_agg
1675.88
1249.67
-426.202
ms

90th percentile latency
hourly_agg
1726.11
1294.06
-432.045
ms

99th percentile latency
hourly_agg
1834.83
1346.8
-488.03
ms

100th percentile latency
hourly_agg
1866.72
1362.41
-504.306
ms

50th percentile service time
hourly_agg
1672.5
1244.76
-427.744
ms

90th percentile service time
hourly_agg
1721.53
1289.75
-431.78
ms

99th percentile service time
hourly_agg
1830.75
1342.43
-488.317
ms

100th percentile service time
hourly_agg
1862.45
1358.18
-504.272
ms

error rate
hourly_agg
0
0
0
%

c6g.2xlarge details

Metric
Task
Baseline
Contender
Diff
Unit

Min Throughput
hourly_agg
0.200508
0.20057
6e-05
ops/s

Mean Throughput
hourly_agg
0.200702
0.200788
9e-05
ops/s

Median Throughput
hourly_agg
0.200676
0.200758
8e-05
ops/s

Max Throughput
hourly_agg
0.201008
0.201131
0.00012
ops/s

50th percentile latency
hourly_agg
2445.65
2141.59
-304.058
ms

90th percentile latency
hourly_agg
2448.21
2145.17
-303.046
ms

99th percentile latency
hourly_agg
2462.3
2149.36
-312.941
ms

100th percentile latency
hourly_agg
2543.93
2159
-384.936
ms

50th percentile service time
hourly_agg
2442.78
2138.38
-304.401
ms

90th percentile service time
hourly_agg
2444.92
2141.43
-303.493
ms

99th percentile service time
hourly_agg
2459.01
2145.47
-313.546
ms

100th percentile service time
hourly_agg
2540.64
2155.71
-384.934
ms

error rate
hourly_agg
0
0
0
%

Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@nknize @reta Can you help review this change? :)

How does this compare with LongHash where every key need to be moved at least once to achieve optimal PSL?

How does this compare with LongHash where every key need to be moved at least once to achieve optimal PSL?

Experimentally, I observed that the cost of key movement is pretty small. It's worth noting that the use of PageCacheRecycler (implementation detail) may provision the backing array in multiple pages at non-contiguous memory locations. Movement of keys that cross the page boundary have a higher effect than the ones moved in the same page. But this overhead is compensated by other improvements.
I'll craft a workload that simulates the case you mentioned in order to quantify the overhead.

Thanks @ketanv3 are there memory profiles that you can share to understand the memory overhead?

Thanks @ketanv3 are there memory profiles that you can share to understand the memory overhead?

Thanks for the feedback, @Bukhtawar! I'll share some memory profiles w.r.t. the number of distinct keys stored. Though, I don't expect the allocated memory to change in any way—the initial capacity and the growth rate remains the same, just the memory layout and the encoding strategy changes.

Looks good!

Have you audited other uses of LongHash and whether replacing those with FastLongHash makes sense?
Is there a better name than FastLongHash that is more specific, e.g. DynamicallyOrganizingKeysHash or LowestSequenceProbingKeysHash or something like that? I fear we may need to supersede it with FasterLongHash :)

The goal was to make this implementation perform similar/better but not worse. To avoid any unknown-unknowns, I'll still put this behind an experimental/sandboxed version until we're fully confident that it works across the board.

@ketanv3 - Given that the latency is same or better across correlated, uncorrelated and distinct adds. Can we flip this as default?
@dblock - Thoughts?
If in case we decide to make it experimental/sandbox, what would be the exit criteria to make it default?

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

URL: https://build.ci.opensearch.org/job/gradle-check/17747/
CommitID: bc6f294
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Unrelated to this PR. This flaky-test is being fixed in https://github.com/opensearch-project/OpenSearch/pull/8063.
Codecov/patch failed as it's expecting test coverage in the benchmark file. This should probably be excluded.

Hi @dblock, can you check the latest revision and comments?

I'm good with this. There's a weird changelog conflict, care to resolve? I tried to use the GitHub UI and I didn't like its suggestion that had nothing to do with this PR :)

@dblock Updated!

Good work. Watch the backport to 2.x, might need some manual work.

@ketanv3 want to backport manually to 2.x?

@dblock Sure! Working on it.

@dblock Here's the backport PR: https://github.com/opensearch-project/OpenSearch/pull/8337


Add support for AAC radio streams

Solves the following warning when playing a radio station using AAC compression:
[2018-11-17 12:18:21] [ZP] Kitchen Sonos: unknown track metadata {"$":{"id":"-1","parentID":"-1","restricted":"true"},"res":[{"_":"aac://http://prem2.di.fm:80/station?<DI.fm ID>","$":{"protocolInfo":"aac::application/octet-stream:"}}],"r:streamContent":["Phats & Small - Turn Around"],"r:radioShowMd":[""],"dc:title":["station?<DI.fm ID>"],"upnp:class":["object.item"]}

Included in v0.3.1.


New floating windows are not interactive if they appear above maximized windows.

Steps to reproduce:

Maximize a windowed program such as Firefox.
Have the program open a new floating window while maximized. (For instance open a page in Firefox which downloads a file. Firefox will then open a floating window asking what it should do with the corresponding file.)
The new floating window is visible in front of the maximized tiled window. But is not interactive unless the maximized program is no longer maximized or the floating window is moved to a sepperate workspace.

Expected outcome:
The new floating window is interactive on top of the maximized tiled window.
OS: Gentoo Linux x64 5.15.41
Hyprland version: 0.7.1beta

fixed in c2215e4efbe4e7351b8ccd12341e3c8dfe23cc39

Can confirm that this commit fixed the issue. Thank you for your help!


New formula audit doesn't check for name conflicts with casks

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

[x] are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
[x] ran a brew command and reproduced the problem with multiple formulae? If it's a problem with a single, official formula (not cask) please file this issue at Homebrew/homebrew-core: https://github.com/Homebrew/homebrew-core/issues/new. If it's a brew cask problem please file this issue at https://github.com/caskroom/homebrew-cask/issues/new. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
[x] ran brew update and can still reproduce the problem?
[x] ran brew doctor, fixed all issues and can still reproduce the problem?
[ ] ran brew config and brew doctor and included their output with your issue?

To help us debug your issue please explain:

What you were trying to do (and why)

Run  audit --new-formula on a formula that has a name conflict with a cask.

What happened (include command output)

$ brew audit --new-formula handbrake
handbrake:
  * New formulae should not have a HEAD spec
Error: 1 problem in 1 formula

What you expected to happen

  * Formula name conflicts with caskroom/cask/handbrake

Step-by-step reproduction instructions (by running brew commands)

brew audit --new-formula handbrake

The audit was removed in https://github.com/Homebrew/brew/pull/3236

I believe we're intentionally not adding any more --new-formula audits until post https://github.com/Homebrew/brew/pull/3835.

#3236 was not about removing a check for cask duplicates.


Add Campsite extension

Description

Screencast

Checklist

[ ] I read the extension guidelines
[ ] I read the documentation about publishing
[ ] I ran npm run build and tested this distribution build in Raycast
[ ] I checked that files in the assets folder are used by the extension itself
[ ] I checked that assets used by the README are placed outside of the metadata folder

Oops - not ready to ship yet! I should have read the publishing docs all the way through first, I missed the bit about needing the owner field to make it private 🤦


Add field configuration loading

Because issue field configuration in JIRA is highly dynamic and customizable, Bugjira needs to be able to retrieve field configuration data from an external source. This patch adds support for that operation, and also defines a very minimal set of field configuration data. The classes in bugjira/field.py will be heavily extended.
Once we can make Bugjira aware of the available fields in the associated JIRA and Bugzilla backends, we will be able to add support for querying and updating those field attributes in BugjiraIssue objects.
The module that loads the field configuration data is designed as a plugin. The plugin included in the source code loads json from a local file when the Bugjira instance is created. Another approach would be to load field configuration by querying a JIRA instance directly, possibly dynamically updating the field info periodically or even each time a BugjiraIssue's fields are accessed. These alternate field info access approaches could be coded into an external library that implements the plugin interface, thus allowing us to decouple the public Bugjira code from any user's particular JIRA instance configuration.

Just curious, what was the reason for specifying the config supplied by the user as json vs yaml, like we do in other projects? It is fine, I just personally find yaml much easier to edit and work with.

Mostly just unfamiliarity with yaml. I agree it'd be more user-friendly for the config file stuff. I'll create a task to switch over to using yaml in the future.


Help handling errors

Hi
So I have this line:
local transcribe_json = JSON.decode(transcribe_result);
how should I code the error verification without breaking the Lua script? Sometimes, transcript_result is not a valid JSON.
local ok, err = pcall(JSON.decode, transcribe_result)
?? :)

Sorry for very very late reply. You're correct, you can use pcall to catch error. I suppose you already solved the problem by yourself.
Lunajson is written solely in Lua, so any error could be caught unless you are encountering an interpreter bug. In version 1.2 or later, stack overflow detection (during encoding/decoding too deeply nested JSON) is also added. If there is any infinite-loop, this is a bug of lunajson, so please raise an issue.


Fix the db_type issue

In infi.clickhouse_orm v1.0.0 the structure of infi.clickhouse_orm.models.AdHocModel has changed making current version of sqlalchemy-clickhouse incompatible.
The purpose of this pull request is to up infi.clickhouse_orm version and fix a compatibility issue: https://github.com/cloudflare/sqlalchemy-clickhouse/issues/22.
It was tested with newest clickhouse installation.

Any update?

@vavrusa could you verify the PR please?

Looks like it's working, thanks for fixing!


[1.1] Error on commissioning step 'FindOperational', CHIP Error 0x000000C9: No shared trusted root

Reproduction steps
Hi all,
I am trying to commission nxp's matter end-device(lighting-app) using chip-tool as matter controller.
I followed the steps in this link to make nxp's k32w dongle as end-device. And I took v1.0-branch, compiled chip-tool example, and used it to commission nxp's matter end-device.
I copied all the paa-root-certs and cd-certs from connectedhomeip GitHub repository and put them at my preferred path and passed the preferred path to chip-tool as an argument.
After that, I connected nxp's matter end-device(lighting-app) and matter controller in the same thread network using ot-ctl and then tried to commission nxp's matter end-device using chip-tool as matter controller using the below command:

./chip-tool pairing onnetwork-long 0x17 20202021 3840 --paa-trust-store-path /usr/local/etc/matter-certs/paa-root-certs --cd-trust-store-path /usr/local/etc/matter-certs/cd-certs

I got the below error in commissioning:

Error on commissioning step 'FindOperational', CHIP Error 0x000000C9: No shared trusted root

I have attached the full logs herewith.
no_shared_trusted_root.txt
Let me know in case more information is required.
Thanks in advance for any help/support.
Bug prevalence
consistent
GitHub hash of the SDK that was being used
v1.0-branch
Platform
k32w
Platform Version(s)
No response
Type
Platform Issue
Anything else?
No response

No idea, sorry.  Normally that sort of thing would mean mismatched root certs or IPK, but here chip-tool is doing the commissioning and the operational setup...
The only thing I can think of is that there is already another device on the operational network with the same node id for the same fabric, but using a different IPK.
But looking at the IPs, when commissioning the device we were sending packets to fd7f:a9ce:b5bf:1:325b:f31b:7c5f:a47e.
And the operational Sigma1 is being sent to fd7f:a9ce:b5bf:1:325b:f31b:7c5f:a47e
So that looks lke the same device...

Thank you very much for the response @bzbarsky-apple

The only thing I can think of is that there is already another device on the operational network with the same node id for the same fabric, but using a different IPK.

I factory reset the device and then did commissioning with an unused node id, but the error persists.
I connected the matter controller and matter end-device over the same thread network and then I tried to commission the matter end-device using the onnetwork-long method.

This matter end-device has BLE and DNS-SD advertisement enabled [advertisementMode == chip::CommissioningWindowAdvertisement::kAllSupported] and I do commissioning using onnetwork-long method. Will this cause the above problem?

Normally that sort of thing would mean mismatched root certs or IPK

How can I validate this?
Any help/support is much appreciated!!

How can I validate this?

If you have access to the code of the device, step through or add logging to see exactly where the CASE bits fail....  The device is claiming there is no trusted root.  The question is why.

Thanks for the great support always @bzbarsky-apple
For Matter Controller: Now I am using  v1.1-branch of the connectedhomeip GitHub repository.
For Matter End-Device: Now I am using v<IP_ADDRESS>.1 of the NXP matter GitHub repository.
I connected nxp's matter end-device(lighting-app) and matter controller in the same OpenThread network using OpenThread-CLI and then tried to commission nxp's matter end-device using chip-tool as matter controller with onnetwork commissioning method using the below command:

./chip-tool pairing onnetwork-long 0x17 20202021 3840 --paa-trust-store-path /usr/local/etc/matter-certs/paa-root-certs --cd-trust-store-path /usr/local/etc/matter-certs/cd-certs

I got the below error in commissioning:

Error on commissioning step 'FindOperational', CHIP Error 0x000000C9: No shared trusted root

I have attached the full logs herewith for matter controller and matter end-device.

bzbarsky-apple commented Oct 19, 2023
How can I validate this?
If you have access to the code of the device, step through or add logging to see exactly where the CASE bits fail.... The device is claiming there is no trusted root. The question is why.

I observed the end-device logs and come to know that "No shared trusted root" event has been generated from end-device.
At the End-device side code, error is returned in GenerateCaseDestinationId() function which will result in CHIP_ERROR_KEY_NOT_FOUND for FindLocalNodeFromDestinationId() which sends kProtocolCodeNoSharedRoot arguement to SendStatusReport() function.
Do i need to make any changes in end-device code to support onnetwork commissioning method?
Further help/support is much appreciated!!

GenerateCaseDestinationId does not return a "no shared trusted root" status....  Where did GenerateCaseDestinationId() actually fail, if it failed?

Thanks for helping me debug the issue @bzbarsky-apple
Root Cause Analysis for the above error:
I have flashed the lighting-app example to NXP K32W061 and commissioned it into the matter network using the matter controller.
After that, I erased and re-flashed the binary to NXP K32W061, but NXP 's programmer(DK6Programmer.exe) did not erase the persistent data stored previously in external flash. After this, I started to commission the same device into the matter network using the matter controller but it failed due to the previously stored persistent data in external flash.
Solution:
"Doing the Factory Reset" NXP K32W061 resolves the "No shared trusted root" as it will erase the previously stored persistent data.
Is this the expected behavior when we didn't factory reset the device or it shouldn't happen?
@bzbarsky-apple you can close the ticket and mark it as resolved.

Is this the expected behavior when we didn't factory reset the device or it shouldn't happen?

@doru91 can you take a look at this part?


Add codespell as a github action

Thanks again @luzpaz! If you can help us with adding this as an action in .github/workflows/repo-check.yml, we will appreciate it.

Originally posted by @tzanio in https://github.com/mfem/mfem/issues/3134#issuecomment-1242236631
https://github.com/codespell-project/actions-codespell

@tzanio there should be a streamlined way to do this through github, see codespell-project/actions-codespell

I tried the above ☝ quickly in #3205, but it doesn't seem to work.
@luzpaz, any ideas what is wrong?

Discussion continued in #3205.


Haxe implementation of Go map and need for unique pointers

The TardisGo implementation:
class GOmap {
	public var baseMap:Map<String,{key:Dynamic,val:Dynamic}>;
	public var kz:Dynamic; // "key" zero value
	public var vz:Dynamic; // "val" zero value
	public function new (kDef:Dynamic,vDef:Dynamic) {
		//trace("DEBUG new",kDef,vDef);
		baseMap = new Map<String,{key:Dynamic,val:Dynamic}>();
		kz = kDef;
		vz = vDef;
	}
	#if cpp
		static var setDefaultFormat:Bool=true;
	#end
	public static function makeKey(a:Dynamic):String{
		#if cpp
			if(setDefaultFormat){ // TODO rewrite this so that we don't check every time
				cpp.Lib.setFloatFormat("%.17g");
				setDefaultFormat=false;
			}
		#end
		//trace("DEBUG makeKey",a);
		if(a==null) return "<<<<NULL>>>>"; // TODO how can this be more unique?
		if(Reflect.isObject(a)){
			if(Std.is(a,String))
				return a;
			if(Std.is(a,Pointer))
				return cast(a,Pointer).toUniqueVal();
			// NOTE Object could be an abstract
			#if !abstractobjects
			if(Std.is(a,Object)){
				//trace("DEBUG makeKey Object found");
				var r=cast(a,Object).toString();
				//trace("DEBUG makeKey Object="+r);
				return r;
			}
			#end
			if(Std.is(a,Complex)){
				return Complex.toString(a);
			}
			if(Std.is(a,Interface))
				return cast(a,Interface).toString();
			if(Std.is(a,Slice))
				return a.toString();
			if(Std.is(a,Channel))
				return a.toString();
			if(Std.is(a,GOmap)||Std.is(a,Closure)) {
				Scheduler.panicFromHaxe("haxeruntime.GOmap.makeKey() unsupported haxe type: "+a);
				return "";
			}
			#if abstractobjects
				return a.toString(); // must be an Object or Int64
			#else
				return GOint64.toString(a);
			#end
		}
		#if cs
			if(Std.is(a,Float)) {
				// in cpp & cs, Std.string(1.9999999999999998) => "2"
				// TODO consider how to deal with this issue in the compound types above
				return GOint64.toString(Go_haxegoruntime_FFloat64bits.callFromRT(0,a));
			}
		#end
		return Std.string(a);
	}
	public function set(realKey:Dynamic,value:Dynamic){
		var sKey = makeKey(realKey);
		//trace("DEBUG set",sKey,realKey);
		if(baseMap.exists(sKey)){
			if(!Force.isEqualDynamic(baseMap.get(sKey).key,realKey))
				Scheduler.panicFromHaxe("haxeruntime.GOmap non-unique key for: "+sKey);
		}
		baseMap.set(sKey,{key:realKey,val:value});
	}
	public function get(rKey:Dynamic):Dynamic {
		var sKey = makeKey(rKey);
		//trace("DEBUG get",sKey,rKey);
		if(baseMap.exists(sKey))	return baseMap.get(sKey).val;
		else 						return vz; // the zero value
	}
	public function exists(rKey:Dynamic):Bool {
		var sKey = makeKey(rKey);
		//trace("DEBUG exists",sKey,rKey);
		return baseMap.exists(sKey);
	}
	public function remove(r:Dynamic){
		var s = makeKey(r);
		//trace("DEBUG remove",s,r);
		baseMap.remove(s);
	}
	public function len():Int {
		var _l:Int=0;
		var _it=baseMap.iterator();
		while(_it.hasNext()) {_l++; _it.next();};
		//trace("DEBUG len",_l);
		return _l;
	}
	public function range():GOmapRange {
		var keys = new Array<String>();
		var k = baseMap.keys(); // in C# and Java, this may not work if new items are added to the map
		while(k.hasNext())
			keys.push(k.next());
		return new GOmapRange(keys,this);
	}
}

class GOmapRange {
	private var k:Array<String>;
	private var m:GOmap;
	public function new(kv:Array<String>, mv:GOmap){
		k=kv;
		m=mv;
	}
	public function next():{r0:Bool,r1:Dynamic,r2:Dynamic} {
		var _hn:Bool=k.length>0;
		if(_hn){
			var _nxt=k.pop();
			if(m.baseMap.exists(_nxt))
				return {r0:true,r1:m.baseMap.get(_nxt).key,r2:m.baseMap.get(_nxt).val};
			else
				return next(); // recurse if this key is not found (deleted in-between?)
		}else{
			return {r0:false,r1:m.kz,r2:m.vz};
		}
	}
}

The "key" issue I would draw your attention is the need to store the key value as part of the map's data structure to allow easy iteration.

Note also that pointers need to have unique values (rather than always being 1, as now), as pointers are often used as map keys.
From the above code:
	return cast(a,Pointer).toUniqueVal();

Pointers are also often cast to uintptr and then processed and compared.
In TardisGo, each Go object in Haxe contained an integer field uRef assigned a unique value when the object was created, which acted as its "pointer". This could be cast to uintptr when required and to string for use as map keys.
// as part of new object creation...
// uniqueCount is a global variable & would need a mutex round it for go2hx use
	uniqueCount += 1;
	uRef = uniqueCount;

@elliott5 What is the advantage of having a unique number, when reference equality can be used for pointer comparison?

The TardisGo code above has to mess about with the precision of printing Float values inside Haxe to make the map keys work reliably.
If I were making a map-key from a float64 now, I would cast the float64 to uint64, so that precision when converting to string was not an issue.

There are 3 places where a unique number might be helpful:

Creating keys for a Haxe map, assuming Haxe maps cannot have objects as their keys;
Casting pointers to uintptr and getting unique values per object when you do; and
Potential unsafe casting of uintptr to pointer, assuming that a memory map exists:

public static var memory = new Map<Int,Object>()

Where a new value is created every time a pointer is cast to uintptr.
This last one is for the future I know.

I see there is haxe.ds.ObjectMap available on some platforms.

Borrowing and amending the Haxe comments for Map key values:

if K is a String, haxe.ds.StringMap is used => Go string
if K is an Int, haxe.ds.IntMap is used =>  Go int32
if K is an EnumValue, haxe.ds.EnumValueMap is used  => (no direct Go equivalent, but see below)
if K is any other class or structure, haxe.ds.ObjectMap is used => Go *something (a pointer)
if K is any other type, it causes a compile-time error

haxe.ds.EnumValueMap is the only one of the above implementations that is written in Haxe, and that by extending haxe.ds.BalancedTree. The extension only requires two functions to be written:
	override function compare(k1:EnumValue, k2:EnumValue):Int {}
	override function copy():EnumValueMap<K, V> {}

Clearly, if the Go map key is string, int32 or *something those target-specific implementations of Map should be used.
But for everything else, either:

all keys could be expressed as enums (unsurprisingly, I have ideas for how to do this); or
for each Go type used as a map key, create a custom extension of haxe.ds.BalancedTree for that Go type; or
as with TardisGo,  haxe.ds.StringMap could be used by stringifying the key; or
the map code could be written in Go, for example using https://pkg.go.dev/github.com/google/btree

What approach do you think is best to try first in your view? @elliott5
I'm uneasy about a stringifying method, 2 and 4 look the best to  me.

I think option 2, creating custom extensions of haxe.ds.BalancedTree  is most likely to be successful @PXshadow.
This requires creating an extension to  haxe.ds.BalancedTree for every Go type. The extensions probably need to be declared immediately after the type is declared, with a consistent naming convention, so that whatever the typeName is, the code generator knows that it should be using typeName_map (or whatever naming convention) as the map type with that type as the key.
Special handling will be required for the built-in string / haxe.ds.StringMap, int32 / haxe.ds.IntMap and *something / haxe.ds.ObjectMap maps.
Maps with having an interface{} key also require special handling. The override function compare can use reflect.Equal() to test equality, but would need to use two fmt.Sprintf("%v",value) values to create comparable strings to define the ordering (see comment on reflect.String). For speed, these generated string key values should  be kept in each haxe.ds.BalancedTree extension node, to prevent multiple (slow) calls to fmt.Sprintf().
In terms of sequencing this work, you should consider getting the system working correctly, then speeding it up. To get the map system working quickly but correctly, implementing all go2hx maps as having interface{}  keys (using the technique in the paragraph above) would require least code changes I imagine.
A second phase could then roll out custom maps for every Go type, which would make them faster.

This approach is so clean and I think it will work great, I'm really happy you thought about this your expertise is invaluable!
Here are to the 2 most complex maps (still prototyping before I commit):
class GoObjectMap<K,V> extends BalancedTree<K,V> {
	public var t:_Type;
	public function new(t) {
		super();
		this.t = t;
	}
	override function compare(k1:K, k2:K):Int {
		final key = new AnyInterface(k1,t);
		final objKey = new AnyInterface(k2,t);
		return key == objKey ? 0 : 1;
	}
}

class GoAnyInterfaceMap<AnyInterface,V> extends BalancedTree<AnyInterface,T> {
	override function compare(k1:AnyInterface, k2:AnyInterface):Int {
		return k1 == k2 ? 0 : 1;
	}
}

For AnyInterfaceMap I think I need to write if the types can be compared first, but the general idea is there. I hope to share more progress soon!

The logic of the compare functions you have written is not quite right @PXshadow :
override function compare(k1:K, k2:K):Int { // should return a -ve, +ve, or 0 result:
   // If k1 is less than k2, the result is negative.
   // If k2 is less than k1, the result is positive.
   // If k1 and k2 are equal, the result is 0.
}

See code for compare and documentation for Reflect.compare.

To see why this ordering is important, look at the code for get(key:K):Null<V>.

I think any map key that is a Go go2hx type that fits into Haxe Int:
typedef GoByte = GoUInt8;
typedef GoRune = GoInt32;
typedef GoInt = GoInt32;
typedef GoUInt = GoUInt32;
private typedef Int8 = #if simulate_num Int; #elseif cpp cpp.Int8; #elseif cs cs.Int8; #elseif java java.Int8; #else Int #end
private typedef Int16 = #if simulate_num Int; #elseif cpp cpp.Int16; #elseif cs cs.Int16; #elseif java java.Int16; #else Int; #end
private typedef Int32 = haxe.Int32; // #if cpp cpp.Int32 #elseif cs cs.system.Int32 #else haxe.Int32 #end;
private typedef UInt8 = Int; // #if hl hl.UI8 #elseif cpp cpp.UInt8 #elseif cs cs.UInt8 #else Int #end;
private typedef UInt16 = #if simulate_num Int; #elseif hl hl.UI16; #elseif cpp cpp.UInt16; #elseif cs cs.UInt16; #else Int; #end
private typedef UInt32 = #if simulate_num UInt; #elseif cpp cpp.UInt32; #elseif cs cs.system.UInt32; #elseif eval eval.integers.UInt64; #else UInt; #end

Should (maybe eventually) use haxe.ds.IntMap, because the ordering is not important so long as it is consistent (so unsigned values will not matter).

@elliott5 my apologizes, you're right I will use sprintf and compare strings with one another, and use IntMap for Int values that fit into Int32 space.

@elliott5 I ran into a slight problem https://pkg.go.dev/reflect#MapIter.Value I think requires all Maps to have the type data passed inside of them, do you think there's a way around it?
The map iterates on values inside of the map, which means that the value's reflect type is also needed from my understanding.

Also here's my current write up of GoAnyInterfaceMap (map[any]unknown):
class GoAnyInterfaceMap<AnyInterface,T> extends BalancedTree<AnyInterface,T> {
	override function compare(k1:AnyInterface, k2:AnyInterface):Int {
		#if nolinkstd
		return 0;
		#else
		final k1Str = stdgo.fmt.Fmt.sprintf("%v", k1);
		final k2Str = stdgo.fmt.Fmt.sprintf("%v",k2);
		return if (k1Str == k2Str) {
			0;
		}else if (k1Str > k2Str) {
			1;
		}else{
			-1;
		}
		#end
	}
}

@PXshadow, your code for GoAnyInterfaceMap should work, but be slow because of recreating the string version of the key for every comparison.
To get round this, the stdgo.fmt.Fmt.sprintf("%v",key) value needs to be created only once for each entry in the map.
There are two approaches to doing this, either:
class GoAnyInterfaceMapKey {
  var ai:AnyInterface;
  var ais:String;

  function new(k:AnyInterface) {
    ai = k;
    ais = stdgo.fmt.Fmt.sprintf("%v",k);
  }
}

class GoAnyInterfaceMap<AnyInterfaceMapKey,T> extends BalancedTree<AnyInterfaceMapKey,T> {
  // code as you have it but comparing AnyInterfaceMapKey.ais
}

Or (as TARDISgo did):
class GoAnyInterfaceMapEntry<T> {
  var key:AnyInterface;
  var val:T;

  function new(k:AnyInterface,v:T) {
    key = k;
    val = v;
  }
}

// Then use the normal target-specific haxe.ds.StringMap
// with the  stdgo.fmt.Fmt.sprintf("%v",GoAnyInterfaceMapEntry.key) as the key
// and GoAnyInterfaceMapEntry as the value.
// The proper key value is required in the entry in order to do Go map iteration.

var myMap : haxe.ds.StringMap<GoAnyInterfaceMapEntry<myValueType>>;
myMap[ stdgo.fmt.Fmt.sprintf("%v", myKey) ] = GoAnyInterFaceMapEntry(myKey, myValue);
x = myMap[ stdgo.fmt.Fmt.sprintf("%v", y) ].val; // null checks also needed here

I'll look into the MapIter.Value issue and get back to you.

@elliott5 I ran into a slight problem https://pkg.go.dev/reflect#MapIter.Value I think requires all Maps to have the type data passed inside of them, do you think there's a way around it?
The map iterates on values inside of the map, which means that the value's reflect type is also needed from my understanding.

reflect.MapIter is created by Value.MapRange() which is a method of reflect.Value which already contains the type information.
The Go implementation of  MapIter  includes the map's Value (containing the Type information). The map's value can be obtained using MapIter.Value() and the map's type by calling MapIter.Value().Type().
All of this only relates to Go maps accessed via the reflect package, normal Go maps do not need their type information stored within them.

@elliott5 thanks for the help, from first glance I don't see how the first approach works because that would only sort the type string and not the type+value string that is created from fmt sprintf. Am I missing something?

I don't believe any type information ever gets formatted as a string @PXshadow, only the value of the key.
Fuller code (in my dreadful Haxe) for the haxe.ds.BalancedTrree method would be:
// define a key type that includes both the original key and its string representation
class GoAnyInterfaceMapKey {
  var ai:AnyInterface;
  var ais:String;

  function new(k:AnyInterface) {
    ai = k;
    ais = stdgo.fmt.Fmt.sprintf("%v",k);
  }
}

// The composite key type defined above forms the key to the map
class GoAnyInterfaceMap<T>
extends haxe.ds.BalancedTree<AnyInterfaceMapKey,T> {
         // code as you have it but comparing AnyInterfaceMapKey.ais
	override function compare(k1:AnyInterfaceMapKey, k2:AnyInterfaceMapKey):Int {
	         if (k1.ais==k2.ais) return 0;
		return if (k1.ais > k2.ais) {
			1;
		}else{
			-1;
		}
	}
}

// example use:
var myMap : GoAnyInterfaceMap<MyType>;
myMap[ GoAnyInterfaceMapKey.new(someAnyInterfaceKey) ] = someMyTypeValue;
x= myMap[ GoAnyInterfaceMapKey.new(y) ]; // needs null handling

I hope the above code ideas make this approach a little clearer.

I'll work on implementing the key value string caching and other things now, just wanted to get an initial commit through.
Your Haxe code is a great reference thank you, and I get it now, you don't want to have to remake the string values every time the map iterates for an exists/set/get function. It should be a massive difference, thanks @elliott5 !

Current benchmark holding up GoMap performance
package main

import (
	"fmt"
	"time"
	"unicode"
)

func main() {
	stamp := time.Now()
	for i := 0; i < 1_000_000; i++ {
		unicode.IsUpper('ű')
	}
	fmt.Println(time.Since(stamp))
}

Haxe: 10.365405952s
Go: 16.157997ms
There is an allocation occuring in slice -> ref and I'm not sure how to get rid of it yet.

I'm not sure if an allocation can be avoided when making a ref to a slice, it would be faster if it were not required.
However, in this case the issue is probably optimisation. Since the following code has no effect (the result of IsUpper() is not used and IsUpper() has no side-effects) the call to IsUpper() and possibly even the whole for loop is probably optimised away by Go:
	for i := 0; i < 1_000_000; i++ {
		unicode.IsUpper('ű')
	}

See this go playground code showing if IsUpper() is in or out makes no difference to the execution time - https://go.dev/play/p/XKvlnBAwAnM

@elliott5 here

I don't think an allocation to hold a sub-slice's "pointer-and-offsets" is avoidable.
The optimisation question is, why is this unicode.IsUpper('ű') code creating a sub-slice at all? Could it's creation be avoided?

Go map implementation is drastically different now, closing and made a new post for the benchmark issue.