Include a new class called ownership which separates autoproducer power plants from public power plants

Description of the issue
All types of power plants (also the renewable ones) can be devided into public power plants or autoproducer power plants. Public power plants belong to the public electricity sector and autoproducer plants belong to the industry sector. I cannot find such a differenciation in the ontology. In energy sytem models there are usually both types of power plants represented, but it might be that in electricity market models there are just the public power plants represented.
Ideas of solution
There are two ideas of solution:

We could include a class called ownership and devide the power plants into public and autoproducer power plants.
We could use the existing class sector devision and devide the power plants into belonging to electricity sector and industry sector.

Workflow checklist

[x ] I discussed the issue with someone else than me before working on a solution
[x ] I already read the latest version of the workflow for this repository
[ x] I added this issue to the Project 'Issues'. If suitable, I add it to further Projects.
[ x] The goal of this ontology is clear to me

I am aware that

[x ] every new entry in the ontology should have an annotation
[x ] classes should arise from concepts rather than from words

We could include a class called ownership and devide the power plants into public and autoproducer power plants.

Makes sense to me, although I don't think "ownership" would be a useful term for this.  I think the distinction is quite arcane to people not familiar with it and can safely be ignored by models that do not represent the distinction.  So the term does not have to be immediately clear to everyone.  But "ownership" will generate false associations for a lot of people.
From the IEA "Energy Balances of OECD Countries: Beyond 2020 Documentation":

Main activity producers generate electricity and/or heat for sale to third parties, as their primary activity. They may be privately or publicly owned. Note that the sale need not take place through the public grid.

Autoproducer undertakings generate electricity and/or heat, wholly or partly for their own use as an activity which supports their primary activity. They may be privately or publicly owned.

So maybe something like "activity focus"?

We could use the existing class sector devision and devide the power plants into belonging to electricity sector and industry sector.

I think that would wreck the normal distinction between supply and demand sectors (it would for REMIND) and probably wouldn't make life easier for models representing only energy supply.

Yes thats true, the suggested names could be missleading. I just found a nice definition from the European Commission:

Types of Producer:
Producers are classified according to the purpose of production:
Main Activity Producer (formerly known as public) undertakings generate electricity and/or heat for sale to third parties, as their primary activity. They may be privately or publicly owned. Note that the sale need not take place through the public grid.
Autoproducer undertakings generate electricity and/or heat, wholly or partly for their own use as an activity which supports their primary activity. They may be privately or publicly owned.

So let's go with type of producers, main activity producer and autoproducer?

I don't now if the source of the definition is needed? It's this one: https://ec.europa.eu/eurostat/documents/38154/41386/Ele_AQ_documentation.pdf/a06bf650-1830-4fcb-9376-7be1f8ab080c

Those definitions from the European Commission and IEA help a lot. But in the end we need an Aristotelian definiton.

I don't now if the source of the definition is needed? It's this one: https://ec.europa.eu/eurostat/documents/38154/41386/Ele_AQ_documentation.pdf/a06bf650-1830-4fcb-9376-7be1f8ab080c

Sources are nice.  The PDF is actually archived.  The IEA documentation doesn't seem to be publicly available.

Being proper Aristotelian and camel-casing, we get:

A MainActivityProducer is a ProducerType that generates electricity and/or heat for sale to third parties as their primary activity.

and

An AutoProducer is a ProducerType that  generates electricity and/or heat, wholly or partly for their own use as an activity which supports their primary activity.

Browsing around, I would stick it all beneath "organisation".

An organization is a structure with multiple people that has a collective goal.

Which would give

A ProducerType is an Organisation that genereates electricity and/or heat.

Reads a quite contrived, but I get the feeling that's what ontologies are like.
Can we get some feedback/input from @OpenEnergyPlatform/ontology-experts ?

CamelCasing just got replaced with normal lower case because of the change to numerical identifiers. Templates are updated in roughly 10 seconds from now.

Templates are updated, new rules, so new defs:

A MainActivityProducer is a ProducerType that generates electricity and/or heat for sale to third parties as their primary activity.

An AutoProducer is a ProducerType that  generates electricity and/or heat, wholly or partly for their own use as an activity which supports their primary activity.

A ProducerType is an Organisation that genereates electricity and/or heat.

"an organisation that generates electricity and/or heat" sounds a bit misleading as an organisation consists mainly of a group of people and those people don't generate the electricity, their machines etc do.
btw the upcoming def of organisation is: "A role depending on a society that has a collective goal."
so what is meant is that the goal of the producers is to generate electricity and/or heat?

These definitions read good. As this is related to markets and economic activities and I would assume this goes into the oeo-social module? I added the respective label

definitely social

do we need the Type suffix? this reads a bit cumbersome

so what is meant is that the goal of the producers is to generate electricity and/or heat?

The ultimate goal of these organisations is earn money. :) Some of them sell electricity and/or heat, these are the main activity producers. Other have power plants which assist their production process with electricity and/or heat to produce goods (or services). The latter don't sell the electricity and/or heat but the produced products.
(This is probably obvious to you, but sometimes it helps to spell this out in such an explicit way.)

then a producer is an organisation that is active in the energy production domain (pr sth in that sense?)

or we just leave it as "an organisation that generates electricity and/or heat" as long as the domain experts understand what is meant ;)

I just really don't like the "Type" suffix

do we need the Type suffix? this reads a bit cumbersome

I just really don't like the "Type" suffix

Yes, "type" is also weird too me. It is somehow an objectification of a property.
@jannahastings : Any thoughts on this?

I agree, we should not use "type" in the name or definition. After all, everything in the ontology is a type, unless it is an individual.
How about introducing "organisational energy producer": def=An organisation that undertakes generating electricity and/or heat to sell to third parties.
SubClassOf: organisation and has_role some 'producer' (role)
I would also recommend adding "energy" in the other labels as well, because otherwise, for example, "auto producer" reads as though is is cars that are being made.
"main activity energy producer": def=An organisational energy producer in which energy or heat production is their primary activity.
Comment: They may be privately or publicly owned. Note that the sale need not take place through the public grid.
"auto energy producer": def=An organisational energy producer in which energy or heat producition wholly or partly for their own use as an activity which supports their primary activity.
Comment: They may be privately or publicly owned.

How about introducing "organisational energy producer": def=An organisation that undertakes generating electricity and/or heat to sell to third parties.
SubClassOf: organisation and has_role some 'producer' (role)

I think the last part of the definition for "organisational energy producer" is not correct (to sell to third parties). That applies only to "main activity energy producers". We should delete that part:
def=An organisation that undertakes generating electricity and/or heat.
"main activity energy producer" sounds good to me.
The term "autoproducer" is one word and cannot be seperated as far as I know, so I would go with "energy autoproducer".

then a producer is an organisation that is active in the energy production domain (or sth in that sense?)

We already have a class producer which is a subclass of agent which might also fit as classification (better that organisation?).
Its def is at the moment: A producer is an agent that makes goods to sell them.
The def could be extended like that A producer is an agent that makes goods for selling or self consumption.

Is the for selling or self consumption part needed? A producer is an agent that makes goods is more generic. In my view someone who produced something but does not sell something is still a producer.
A relevant example in energy modeling: There are people who own a small rooftop PV system and generate (produce) electricity for own consumption and do not sell the electricity to someone else. In my view these people are still producers.

Btw, do we have a proper definition of good? Are electricity and heat goods?

Btw, do we have a proper definition of good? Are electricity and heat goods?

no but we have an issue to include economic goods #340.

so in conclusion:

producer: "an agent that makes goods"

organisational energy producer: "a producer that undertakes generating electricity and/or heat."

"main activity energy producer": "An organisational energy producer whose primary activity is energy or heat production."
Comment: They may be privately or publicly owned. Note that the sale need not take place through the public grid.

"auto energy producer": "An organisational energy producer who produces energy or heat wholly or partly for their own use as an activity which supports their primary activity."
Comment: They may be privately or publicly owned.

I don't see many advantages of either implementing it as a subclass of producer or organisation.
Maybe that as a subclass of producer it allows auto energy producers to be direct subclass of the organisation fitting to their primary activity and to just have 'auto energy producer' as a role?

Two small changes:

* **organisational energy producer:** "a producer that undertakes generating electricity and/or heat."

Just a minor point, maybe change to "a producer that undertakes the generation of electricity and/or heat"

* **"auto energy producer":** "An organisational energy producer who produces energy or heat wholly or partly for their own use as an activity which supports their primary activity."

I think this should be  named energy autoproducer as per @Vera-IER's comment above.

I changed my comment :)

I don't see many advantages of either implementing it as a subclass of producer or organisation.
Maybe that as a subclass of producer it allows auto energy producers to be direct subclass of the organisation fitting to their primary activity and to just have 'auto energy producer' as a role?

My thinking was that the definition of agent is as a role, and in this case it is the organisation that has the role, so "organisational energy producer" should be defined as a child of organisation and has_role producer

Two points:

How would these people with small rooftop PV systems be classified? They are energy autoproducers but are not organisational.
If we made organisational energy producer a subclass of organisation, we should call them energy producing organisation instead?!

The ones having a small rooftop PV system also need to be registerd with a trade license, so they are also tiny business organisations. This is true for Germany, I don't know how it works in other countries. We call them prosumers, but I think we don't need this new term in the ontology, because it is a special subclass of autoproducers.

We have prosumer in the oeo:
A prosumer is an agent who is producer and consumer at the same time.
The def of organisation, as it is right now, refers to multiple persons, though: An organization is a structure with multiple people that has a collective goal.

can an organisation be a structure with just one person?
That def is not aristotelean btw, we should fix that:
" An organization is a generically dependent continuant ..."

Most definitions that I read an organisation had to be multiple people.
So do we implement the organisational energy producer now instead as a producer to include those small rooftop PV system owners?

An organisation can definitely consist of just one person, even though they typically have more. For example, you can start a business by yourself :-). As per comment above, even small rooftop energy producers have to be registered and have a license, giving them organisational paperwork. So we should fix the definition of organisation.

OBI defines 'organisation' as follows:
An entity that can bear roles, has members, and has a set of organization rules. Members of organizations are either organizations themselves or individual people. Members can bear specific organization member roles that are determined in the organization rules. The organization rules also determine how decisions are made on behalf of the organization by the organization members.

ok we wanted to make Organisation a role (#301, 5. dev meeting).
I proposed there: "An organisation is a roal depending on a society that has a collective goal."
This could be extended, inspired by OBI, to:
"An organisation is a roal depending on a society that has a collective goal and a set of organization rules."

The extended def sounds good to me.

sounds also good to me. I think there is a tiny misspelling: roal = role?

sounds also good to me. I think there is a tiny misspelling: roal = role?

oh yes I fixed it

@Vera-IER do you want to implement or shall I do it?

Can you do it and I try to learn from you by following the review process?


[aws-cpp-sdk-s3-crt]: avoid race condition in Get/PutObjectAsync (#2024)

Issue #, if available: Resolves #2024.
Description of changes:
The underlyingS3Request can be accessed (read/write) from different threads, due to the use of callbacks.
Use a mutex to avoid a race condition in concurrently accessing the aws s3 meta_request structure.
Check all that applies:

[X] Did a review by yourself.
[X] Added proper tests to cover this PR. (If tests are not applicable, explain.)
Tested with clang thread sanitizer that race condition described in #2024 no longer occurs.
[X] Checked if this PR is a breaking (APIs have been changed) change.
[X] Checked if this PR will not introduce cross-platform inconsistent behavior.
[X] Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

[X] Linux
[ ] Windows
[ ] Android
[ ] MacOS
[ ] IOS
[ ] Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks for the contribution!


Create language selection

Create an input form where the user can select what languages that the site will be supporting (the public facing site).

Can add any number of languages
Can remove languages
Must choose one default language

This is preparing for localization of content later on the public facing site so that the content can be served as:

example.com/blog/my-blog-post
example.com/en/blog/my-blog-post
example.com/id/blog/blog-post-aku

This is the related component: https://github.com/oddbit/tanam/tree/angular/src/app/admin/settings/site-form

👍


Allow storing build scripts somewhere other than the project root

I'd like to be able to hide my CI/build scripts away from the root directory, purely so it doesn't become too cluttered. Other services allow using hidden directories like .github and .buildkite - it'd be great it AppCenter could search for the various pre/post scripts in .appcenter (or allow us to specify an alternate location).

I also would like the build scripts to be able to go into another folder but I don't particularly want it to be hidden. I'd prefer bin/ or app center/ at the root of the project.


Add C17a to microdata output list

Description
C17a, a revision of C17, was introduced in Wave 8 but not added to the microdata output list.
Fixes
Closes #987.
@krivard We'll need to reissue the microdata going back to Feb 8 so we can include this item.

I don't think so... I am able to produce the backfill files locally and can presumably add them to the SFTP site. But since the automatically-run code needs to be updated manually as far as I'm aware, it will continue making the old version of the microdata.

@nmdefries Reminder that this will require a reissue of microdata; can you coordinate with @krivard?

Pulled, built, and installed; first issue of microdata with this code will go out tomorrow morning. @nmdefries you're clear to upload backfilled files through today at will.


Update dependencies week 48/2018

@webcomponents/webcomponentsjs 2.1.3 -> 2.2.1
chromedriver 2.43.1 -> 2.44.0
npm-run-all 4.1.3 -> 4.1.5
prettier 1.14.3 -> 1.15.2
rollup 0.67.1 -> 0.67.3
webpack 4.25.1 -> 4.26.1
@types/node 10.12.9 -> 10.12.10
firebase 5.5.8 -> 5.5.9
ts-loader 5.3.0 -> 5.3.1
ts-node-dev 1.0.0-pre.30 -> 1.0.0-pre.31

various point releases.  addresses one npm audit issue.

Is there a particular catalyst for this? From recent discussion in chat we were considering whether to do these ongoing or more reactionarily. @shans to comment as well.


fix(checkpoint-validation): fix broken initial release

Unfortunately the initial release of @langchain/langgraph-checkpoint-validator is broken for two reasons.

A bug in @swc-node/register attempts to resolve SWC packages from the CWD rather than the installed package's node_modules directory.
The jest config used by the CLI wasn't included in the release because I forgot to add it to the files array in package.json.

I fixed the first issue by applying the workaround mentioned here: https://github.com/swc-project/swc-node/pull/805#issuecomment-2257146304.
I fixed the second issue by moving the runtime jest config into the src dir and making it a TS file.
I tested this locally using npm link and it seems to fix both issues.
Sorry for the hiccup!

Thank you!


Add remove

In this milestone I:

Add Edit Function
Add add function
Add remove function

STATUS: APPROVED 👍🎯
Hello @Mahdi-Niazi 👋
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉
Highlights ⭐
✔ item indexes are fixed
Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


Numpy needs to be installed before installing this package

This issue stems from:
https://github.com/moble/quaternion/issues/42
The current solution is to install numpy before installing this package.

Fixed with #6


Add module for registering arbitrary events

See #1123 for more context.

Currently it's only possible to make the World contract emit arbitrary events by modifying the World contract directly or via a root system. Both is only possible with root access to the World, and there is no default tooling to support this.
We initially designed ephemeral table's as a replacement for what events are used outside MUD, but the compromise between "events" and "offchain tables" resulted in a design that's suboptimal for either use case.
Events other than the default Store(Set|Splice|Delete)Record events and ephemeral table events are useful to implement existing interfaces (e.g. ERC20, ERC721, etc) in a World contract.
The proposal is to implement an optional module and corresponding client tooling to register arbitrary events in an existing World contract.

An event has to be registered (with its signature and an eventId consisting of its namespace and name) before it can be emitted.
Each signature can only be registered once per World and only by the owner of the namespace.
After an event is registered it can be emitted by anyone with write access to the namespace (or the specific eventId).
Client tooling can add listeners for the registration of new events, and create decoders based on the event's signature. Then it can provide a stream of decoded events.

Not everyone might want this feature in their World. By implementing this functionality as a (root) module, we give developers the choice of whether they want this functionality in their World or not.

Since this is an optional additional module, it doesn't necessarily have to be part of the first stable release.

Are you able to dynamically register/emit event types in pure Solidity? Or will this need some TS/codegen so we can modify the world contract before deploy to include the custom events?

Are you able to dynamically register/emit event types in pure Solidity?

It should be possible via assembly (eg LOG1, see https://www.evm.codes/#a1?fork=shanghai), but haven't tried yet

Do we think our puppet approach is good enough for this both as an existing implementation folks can use and also an example of how to do this if folks want more control?

Do we want to allow any event via https://github.com/latticexyz/mud/pull/2171 ?

Marking as done via puppet approach


don't compile if XML files already present

Let's check if they exist and skip processing

@rultor release, tag is 0.4.0

@rultor release, tag is 0.4.0

@yegor256 OK, I will release it now. Please check the progress here

deleted comment

@yegor256 Done! FYI, the full log is here (took me 10min)

Job gh:cqfn/eo#322 is not assigned, can't get performer


Make tabs Hound-compliant

It was a poor experience to add tabs to a project and then
have Hound provide lots of style feedback.
https://github.com/thoughtbot/hub/pull/411
This pull request attempts to address all that feedback.

@joshuaogle Can I get a review?

Small note, but otherwise this looks good :+1:
There are a few lines in the JavaScript that are too long too, but that could be another PR

Another minor note; otherwise :+1:


phonegap run ios --device fails silently if ios-deploy is not installed

I am on a brand new machine setting things up from scratch. OSX 10.10.2. I installed phonegap via npm (though it required the use of sudo), and had to jump through a few hoops to get ios-sim to install through npm, too (sorry, didn't write them down and that was late last night)...
Now I'm trying to run phonegap run ios --device and nothing happens:
$ phonegap run ios --device
[phonegap] executing 'cordova run ios --device'...
[phonegap] completed 'cordova run ios --device'

Since nothing happened on my connected ios device, I decided to try again with the --verbose flag and saw this error:
Running command: [snip]/mobile/platforms/ios/cordova/run --device

Error: ios-deploy was not found. Please download, build and install version 1.2.0 or greater from https://github.com/phonegap/ios-deploy into your path. Or 'npm install -g ios-deploy' using node.js: http://nodejs.org/

Command finished with error code 1: [snip]/mobile/platforms/ios/cordova/run --device

ERROR running one or more of the platforms: Error: [snip]/mobile/platforms/ios/cordova/run: Command failed with exit code 1
You may not have the required environment or OS to run this project

[phonegap] completed 'cordova run ios --device --verbose'

I will be installing that dependency manually and trying again, but I thought you should know so you can surface that error without the --verbose flag.

+1

+1

+1

+1

+1

+1

+1

Thanks for the heads up!

Thanks for the heads up @atuttle!
I've went ahead and created a milestone to improve this experience. The main consideration is non-OS X users because they cannot install/compile ios-sim and ios-deploy. I'll use this issue to track the progress of bundling these tools and ensuring that Cordova is able to leverage the local copies.

When this error happen, I am desesperated, but I just desplugged my iPhone device from my MacBook and.. SURPRISE, all works fine!

Just an update for this issue: in the latest phonegap-cli, if you run without the -d or --verbose flag, this is what you will see (on macOS Sierra):
Error: spawn ios-deploy ENOENT

With -d, you will see:
Error: spawn ios-deploy ENOENT
    at exports._errnoException (util.js:1018:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
    at onErrorNT (internal/child_process.js:367:16)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

So, I am not sure that the original issue with respect to 'failing silently' is still applicable here. Even in non-verbose mode, ENOENT error with respect to ios-deploy makes it relatively clear what the issue is, I think.
Perhaps worth closing the issue?
Thoughts @mwbrooks / @surajpindoria ?

Agreed, if it is displaying an error message then I think it is safe to close it.

Alright, if others disagree and would like to see more specific error messaging here, please speak up + reopen the issue. In the mean time, closing.


Mean substraction

In the demo file, I can't see any mean subtraction from the input images on the webcam. Is that correct?

Yes, I did not perform any preprocessing like mean substraction or scale normalization in training.
Thus, the input to the pretrained model should be a standard "RGB" image patch.

Okey! Thanks for replying


bump up to 0.23.2 version for release

bump up to 0.23.2 version for release
PLUGIN-1780 changes done for GCS multisink issue

Can you please add JIRA in description which changes needs to be released?

Added in description

Can we hold this release, untill https://github.com/data-integrations/google-cloud/pull/1423 is merged !?

Can we hold this release, untill #1423 is merged !?

Yes, we can wait. Customer is not waiting on this. @itsankit-google please confirm.

@vikasrathee-cs #1423 is merged you can go ahead with this release ! 😃


🐛 [firebase_dynamic_link] Not clear dynamic links after retrieved

Bug report
Describe the bug
Not clear dynamic links after retrieved
Steps to reproduce
Steps to reproduce the behavior:

I call the initial dynamic link at splash screen and home screen
open my app from dynamic link and open a specific page
back to the home page
at the home page, I call again the dynamic link initial and  dynamic link open my link again and go to my specific page before

Expected behavior
My expectation is when I call the dynamic link on the home page, it should not call my previous link because I already used it.so it should be clear from the cache
Sample project
`Future init(BuildContext context) async {
FirebaseDynamicLinks.instance.onLink(
onSuccess: (PendingDynamicLinkData dynamicLink) async {
final Uri deepLink = dynamicLink?.link;
if (deepLink != null) {
handleRoute(context, deepLink.query);
}
},
onError: (OnLinkErrorException e) async {},
);
final PendingDynamicLinkData data =
    await FirebaseDynamicLinks.instance.getInitialLink();
final Uri deepLink = data?.link;
if (deepLink != null) {
  handleRoute(context, deepLink.query);
}

}`
Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 2.5.1, on macOS 11.3 20E232 darwin-x64, locale en-ID)
[!] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.
[✓] Xcode - develop for iOS and macOS
[✓] Chrome - develop for the web
[✓] Android Studio (version 4.1)
[✓] VS Code (version 1.60.2)
[✓] Connected device (2 available)

!
Thank you


Unable to Use "export-zone" Command with Required  Argument

Hi Team,
I’m encountering an issue when using the export-zone command in the Akamai Terraform CLI. Despite providing the <zone> argument as required, I keep getting the following error:
Invalid arguments usage, next arguments are required: <zone>

Command Used:
akamai terraform export-zone --resources existing_zone.com

Expected Behaviour:
The command should generate the json formatted resource import list file.
Observed Behaviour:
The CLI throws an error indicating that the <zone> argument is required, even though it was provided.
Environment Details:

Akamai CLI Version: 1.6.0
Akamai Terraform CLI Plugin Version: 1.19.0
Operating System: macOS 15.1
EdgeRC File: Configured with default section (used it before for property import)

Troubleshooting Steps Taken:

Verified that the <zone> argument is correctly provided as a positional argument.
Attempted using various combinations of flags (e.g., --createconfig, --resources) along with the <zone> argument.
Updated Akamai CLI and Terraform plugin to the latest version.
Reviewed the documentation for the export-zone command.

Questions:

Have you come across a similar issue with the export-zone command?
Is there an additional flag or step required that is not mentioned in the documentation?
Could this be a bug in the CLI or the Terraform plugin?

Steps to Reproduce:

Run the following command:akamai terraform export-zone existingdomain.com

Observe the error message:Invalid arguments usage, next arguments are required: <zone>

Any guidance or suggestions to resolve this issue would be greatly appreciated. Thank you for your support!

Hi @macalac,
Unfortunately I can't reproduce your issue:
mstojano@krk-mpbff cli-terraform % akamai --version
akamai version 1.6.0
mstojano@krk-mpbff cli-terraform % akamai terraform --version
akamai terraform version 1.19.0
mstojano@krk-mpbff cli-terraform % akamai terraform export-zone *********.net
Configuring Zone
Zone configuration completed

What terminal you use? Is it default or custom one?
BR,
Marcin


Decode URL params

Since query params are passed in the URL, the data is encoded (e.g. @ becomes %40).
This breaks the validators in cases like having a Regex to validate an email.
Also the data received in the handlers is encoded.
Current Behavior
const query = c.req.valid('query')
// { email: 'example%40gmail.com' }

Expected Behavior
const query = c.req.valid('query')
// { email<EMAIL_ADDRESS>}

Hi @sergiocarneiro !
Thanks for creating the issue. I've released the new version v3.1.3, which includes the change to resolve this issue. Try it!


Use ENS name as account identifier if it exists

Show ENS name instead of truncated address in transactions pill if one exists for the address.
Before

After

Thanks for the review @kangarang . Updated now.


Automatically figure out optimum processes to parallelize on

From: https://github.com/bundler/bundler-features/issues/44

Done in https://github.com/bundler/bundler/pull/5986


nvhpc-only environment won't work because of xz package

This is a simplified DFT dev.env https://github.com/eth-cscs/alps-spack-stacks/tree/main/pe-dft-nvhpc
It won't work becuase spack install xz%nvhpc won't work.

An additional gcc compiler is needed to build the dependencies that can't be built by the nvhpc toolchain, such as xz.
Out of curiosity, does xz%nvhpc not work because it tries to build with nvhpc and fails?
    nvhpc-env:
      compiler:
          - toolchain: llvm
            spec<EMAIL_ADDRESS>          - toolchain: gcc
            spec: gcc@11
      unify: true
      specs:
      - libxc
      -<EMAIL_ADDRESS>      - quantum-espresso +cuda +libxc ^intel-oneapi-mkl+cluster
      - xc@gcc11
      mpi:
          spec: cray-mpich-binary
          gpu: Null
      packages:
      - perl
      - git


Fix issue 191

The issue is not yet closed, but these tests can now start to be removed in aurora -- merging in.


docs: update parquet_sql_multiple_files.rs with a relative path ex

Which issue does this PR close?
Related to #9280, but not sure if it'd technically close it.
Rationale for this change
It's easy to get tripped up when querying the local file system if you aren't using absolute paths because by default there's a local file system registered at file://, so nothing needs to be done for absolute paths. But if you want to query with relative paths, you first need to register an ObjectStore, so this example shows that.
An alternative might be to also register file://./ by default, but I'm not sure all the consequences of that. Or perhaps something easy like ctx.register_cwd()?
What changes are included in this PR?
Updates the example.
Are these changes tested?
Yes, running:
arrow-datafusion update-p… ➜ cargo run --example parquet_sql_multiple_files
    Finished dev [unoptimized + debuginfo] target(s) in 0.38s
     Running `target/debug/examples/parquet_sql_multiple_files`
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col  | string_col | timestamp_col       |
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
| 4  | true     | 0           | 0            | 0       | 0          | 0.0       | 0.0        | 30332f30312f3039 | 30         | 2009-03-01T00:00:00 |
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col  | string_col | timestamp_col       |
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
| 4  | true     | 0           | 0            | 0       | 0          | 0.0       | 0.0        | 30332f30312f3039 | 30         | 2009-03-01T00:00:00 |
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+

Are there any user-facing changes?
Insofar as it's a doc-related update.

I'm not sure if it fixes, as the example change is introduced, but more examples its for better

Yeah, agreed. Hopefully the OP will follow-up as to if they've resolved their issue.

Related to https://github.com/apache/arrow-datafusion/issues/9280, but not sure if it'd technically close it.


Syntax for database.mmones.io/GrantMariaDB

Hi,
The Grant example shows how to grant basic rights to a user, but how to list them all ? Like what should I declare to grant all rights to a database to a user ?
Thanks in advance,

Hi @krptg0 !
You can grant ALL PRIVILEGES using the Grant CRD:
apiVersion: database.mmontes.io/v1alpha1
kind: GrantMariaDB
metadata:
  name: user
spec:
  mariaDbRef:
    name: mariadb
  privileges:
    - "ALL PRIVILEGES"
  database: "*"
  table: "*"
  username: user
  grantOption: true


User-reported unhelpful error message

Program: https://code.world/haskell#PdE2EGgt7PRQNBPaNuB9jWQ
Message:
program.hs:5:12-15: error:
    Variable not in scope: path :: [(Integer, Integer)] -> Picture

Uses a deprecated symbol, path, that is many years old by now.  This is too bad, but I've chased down everything I can, and there's nothing more to be done here.


WYSIWYG field : Use custom toolbar

Expected Behavior
Hello everyone,
According to the ACF documentation, you can define a custom toolbar in WYSIWYG field. Currently, you can define toolbar in extended-acf with ->toolbar(string $toolbar) in Wysiwyg class.
If you create a new toolbar with the name Very Simple (as in the ACF cookbook), and you try to pass the same name in Wysiwyg toolbar method, nothing happens, because ACF use snake case version of name to define which element is used.

My question is : it is the code's responsibility to turn all names into a snake case version, or the documentation for this project should be more explicit ?
Actual Behavior
->toolbar(string $toolbar) supports only snake case names.
Versions

WordPlate: 8.5.0
PHP: 7.3

Thanks for reporting this Jean-Baptiste. In this case I think we can simply add a comment to the documentation. Do you mind sending a pull request? Maybe something like:
->toolbar('simple') // toolbar name in snake_case

@vinkla , of course PR is open! Have a nice day.

Fixed in #85


Interactive Parameters Demo 2

X -> Y

Show your :heart:  for vegalite by adding a tip!

*Tips will only be transferred when the issue gets resolved

Show your :heart:  for vegalite by adding a tip!

*Tips will only be transferred when the issue gets resolved


Problems when Adding New Benchmark

I am trying to add new benchmarks on uPIMulator but I've encountered some issues.
1.Do we still need to write host-side code for the new benchmarks?
2.How can we avoid generating DPU-side code with instructions that you haven't implemented? For example, what code should we avoid writing, or how should we modify DPU side code after unimplemented instructions show up?
Thank you!

Hello,

No, you don't need to write host-side code for simulation. However, you have to prepare input/output data to the device-side code.
Sorry, we don't have an answer for this question since we used UPMEM's compiler as-is. We didn't implement the compiler. But we found that the compiler only generates a handful of instructions that we can simulate with uPIMulator. As a general rule of thumb, if you don't use int64, uint64, float, or double data types, those unimplemented instructions won't be generated. If the simulator encounters unimplemented instructions, it will crash. We are currently working on extending uPIMulator so it can simulate complete UPMEM ISA.

I hope this can answer your question, and if you have further questions, don't hesitate to reach out to us.
Thank you,
Bongjoon


chore(package): update dependencies

https://greenkeeper.io/

Current coverage is 82.48% (diff: 100%)

Merging #243 into master will decrease coverage by 1.98%

@@             master       #243   diff @@
==========================================
  Files            18         18
  Lines           934        748   -186
  Methods         214        144    -70
  Messages          0          0
  Branches        286        214    -72
==========================================
- Hits            789        617   -172
+ Misses          145        131    -14
  Partials          0          0

Powered by Codecov. Last update 779590d...87fe7dd

Getting the error in the build
Error: Could not find source map for: "/home/travis/build/formly-js/ng2-formly/src/core/test-utils.ts" ...........


for #858 the encoding issue

Codecov Report

Merging #861 into dev will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              dev    #861      +/-   ##
=========================================
- Coverage   63.75%   63.7%   -0.06%
=========================================
  Files         500     500
  Lines        9811    9811
  Branches     1616    1616
=========================================
- Hits         6255    6250       -5
- Misses       3200    3208       +8
+ Partials      356     353       -3

Impacted Files
Coverage Δ

...spring/datasource/SpringMasterSlaveDataSource.java
83.33% <0%> (-16.67%)
:arrow_down:

...tion/internal/config/ConfigMapListenerManager.java
60% <0%> (-5%)
:arrow_down:

...al/state/datasource/DataSourceListenerManager.java
50% <0%> (-4.55%)
:arrow_down:

...ternal/state/instance/InstanceListenerManager.java
38.46% <0%> (-3.85%)
:arrow_down:

.../internal/config/ConfigurationListenerManager.java
71.42% <0%> (-3.58%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7ed360...0db190c. Read the comment docs.


fixI(inline-select): Updates invalid text alignment

Overview
Resolves #921
Ref #926
Added
Note This does not solve the issue for IE11, however, I don't see any reason why this fix can't be merged in and we can tackle the IE issue separately. IE11 does not support auto when using display: grid;

@emyarod Take a look at my solution here using grid. The fallback for IE is to have the error text all the way left aligned.

Thanks for the heads up @alisonjoseph. I think if that fallback method is alright then grid is probably the way to go here. I synced my branch with your feature branch so there should be no conflicts merging to master. Your changes will fix the validation text alignment issue and I've committed the focus outline fix beforehand

closing this since updates were merged in to #926


code updates to use override.json file..

code updates to use override.json file..

/run pipeline

/run pipeline

@ocofaigh - Please review this PR which is only for latest override.json file usage instead of json string in terraform code. I created symbolic link to override.json file but it didn't work after checking into branch. I will look at that issue & fix it in next PR.

/run pipeline

:tada: This PR is included in version 1.7.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


.net 8.0.302 is not installed on all agents

Description
Some of the agents we get from azure devops when running windows-latest does not have .net 8.0.302
So Image 20240618.1 is not deployed to all agents yet, normal you only updated the wiki page after all agents are rolled updated
Platforms affected

[ ] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] Ubuntu 24.04
[ ] macOS 11
[ ] macOS 12
[ ] macOS 13
[ ] macOS 13 Arm64
[ ] macOS 14
[ ] macOS 14 Arm64
[ ] Windows Server 2019
[X] Windows Server 2022

Image version and build link
Azure devops
Runner Image Provisioner
2.0.370.1
Current image version: '20240610.1.0'
Is it regression?
20240618.1.0
Expected behavior
I expect that all agent run the version on the wiki page or a new version.
Actual behavior
We get the version list on the wiki page or older versions
Repro steps
Create a pipeline that need .net 8.0.302
Run it until you get an agent with version '20240610.1.0'

Assume you need planId
"orchestrationPlan":{"planId":"82f32fd5-932a-403c-9a5e-5101575888ed"}
Since you needed it last time it was a problem:
https://github.com/actions/runner-images/issues/5976
Let me know if you need more planId

Hi @HenrikSommer-Energinet  , We will be analysing the issue  and provide solution.. Thanks

Hi @HenrikSommer-Energinet  , the issue is resolved and image is deployed in all agents . Thanks


right mouse click issue

i have a remote webpage i am loading that allows for context menus to be shown on right click, currently they work if i do a left long click + right click. I am looking to make them work on long right click.
I know in android the right click acts as a back button, i have disabled the back button in my webview.
Is there a way to make the back work as a long left click?

Sorry, don’t know anything about this, and it seems this is not specific to this library.


feat: bittwister sample code added to the app sync test

This PR adds a traffic shape test to the app (TestPoolSync_WithTrafficShape).
It basically creates one validator and two full nodes out of which one is not restricted and the other one has a limitation on bandwidth and then let them sync and measures the elapsed time of each and compares them.
This code is a sample code to show app developers how to apply traffic shape conditions on their tests using BitTwister.

I just realized we combined DisableNetwork() and Bittwister. For me, this does not make sense. Disabling the network does not allow connections to that Instance, but as we want to check Bittwister, this should not be used here. I propose to move the Bittwister networking test to a separate file and keep the existing one with disabling network.

Good point, I 'll create an issue to keep track of it.


Change Marshal.SizeOf to Unsafe.SizeOf

This currently doesn't break anything but Unsafe.SizeOf is the correct way to do it.

What is the difference between Marshal.SizeOf and Unsafe.SizeOf that makes Unsafe.SizeOf the "correct" way to do it?

What is the difference between Marshal.SizeOf and Unsafe.SizeOf that makes Unsafe.SizeOf the "correct" way to do it?

They both produce the same result in most cases. Seems like Marshal.SizeOf is intended for when sending raw data between C# and C/C++, while Unsafe.SizeOf is for internal C# use. The "marshal" can add padding bytes sometimes if necessary. At least this is what I remember reading.

They both produce the same result in most cases. Seems like Marshal.SizeOf is intended for when sending raw data between C# and C/C++, while Unsafe.SizeOf is for internal C# use. The "marshal" can add padding bytes sometimes if necessary. At least this is what I remember reading.

This was fixed in the latest commit already though.

This was fixed in the latest commit already though.


fwd: KeyCode is not working, Debug Camera does not react for fwd key (default W)

fwd: KeyCode::W is not working.
key "W" press or any other key set for "fwd" is ignored. Debug Camera does not react.
rest works very nice, thank you for sharing !!

Found the line causing the error:
    .add_plugins(DebugCameraPlugin {
        // As a demonstration, this example shows how you can override key bindings. We keep as
        // default to keep a consistent experience
        keyboard_bindings: KeyboardBindings {
            //fwd: KeyCode::W,  // <- has to be ***commented out***, then ***fwd works***  as well :-)
            ..default()
        },
        ..default()

see above


ARROW-8504: [C++] Add BitRunReader and use it in parquet

Adds two implementations of a BitRunReader, which returns set/not-set
and number of bits in a row.

Adds benchmarks comparing the two implementations under different
distributions.

Adds the reader for use ParquetWriter (there is a second
location on Nullable terminal node that I left unchanged because
it showed a performance drop of 30%, I think this is due the issue
described in the next bullet point, or because BitVisitor is getting
vectorized to something more efficient).

Refactors GetBatchedSpaced and GetBatchedSpacedWithDict:

Use a single templated method that adds a template parameter
that the code can share.
Does all checking for out of bounds indices in one go instead
of on each pass through th literal (this is a slight behavior
change as the index returned will be different).
Makes use of the BitRunReader.

Based on bechmarks BM_ColumnRead this seems to make performance worse by
50-80%.  This was surprising to me and my current working theory is
that the nullable benchmarks present the worse case scenario every
other element is null and therefore the overhead of invoking the call
relative to the existing code is high (using perf calls to NextRun()
jump to top after this change).  Before making a decision on reverting
the use of BitRunReader here I'd like to implement a more comprehensive
set of benchmarks to test this hypothesis.
Other TODOs

[x] Need to revert change to testing submodule hash
[x] Add attribution to wikipedia for InvertRemainingBits
[x] Fix some unintelligible comments.
[x] Resolve performance issues.

A fair chunk of RLE-related code came out of Impala originally, it might not be a bad idea to peek at what's in apache/impala to see if it has gotten worked on perf-wise since the beginning of 2016. In any case, we wouldn't want to take on a change that causes a perf regression in cases with many short null/non-null runs

Agreed.  I'm think I'll have to add a special case for this.  The code can also be simplified a bit further for cases with runs

@wesm interesting data point, I updated performance benchmarks to generate random values/nullability (and kept deterministic one).  It seems like the bad regression is really only deterministically alternating nullability.  Randomly generated nullability (at 50%) shows improvements.  I still have some cleanup to do but I'd be curious on peoples thoughts on the need to try to detect/special case deterministic patterns in nullability (its possible the maybe I just chose a bad seed?):

OK, any code cleanup I've tried to do seems to make performance worse.  So I think the one remaining question is if I should try to special case alternating null/non-null values because the regression is bad.
I think the run calculation is still more expense in some cases, but looks very bad when nulls follow a pattern the CPU branch predictor can get correct all the time.  So the only way of special casing this is try to estimate when alternating values is happening and revert back to the old one at a time code path.  I'm -0 on make such a special case because it isn't really clear to me how much this occurs in "real" data.
I think another way of framing this is, if we had pseudo-random benchmark data to begin with would anybody notice the particularly bad performance?  Would we try to special case this under that scenario?

I agree that we shouldn't be basing the decision on the performance of random / worst-case scenario data. If the perf is better on "realistic" datasets (> 80/90% non-null data with consistent consecutive runs) then it doesn't seem like a bad idea to me.

@wesm OK, I did a little bit more in depth sampling.  And it looks like this new algorithm is a win for 0-5% nulls, then a regression until someplace between 45-50% nulls then a likely a win with a larger percentage of nulls.  I'll add a special case to estimate which algorithm to use (this one or 1 by 1 based on percentage of nulls and sampling the first N elements of the bitmap vector).

OK, I was actually able to make this faster with a little bit of refactoring.  The only major regression is now the exact alternating case (~50% worse).  I'll wait till https://github.com/apache/arrow/pull/7175 is merged to publish new benchmarks, but I no longer think special casing is necessary.

OK I think this is ready for review CC @pitrou @wesm
Benchmarks show mostly 20% improvement for any random null values across the board.  The worse case scenario shows 40-60% regression.  This seems to occur when we have deterministically alternating null/not-null.  I believe this to be because modern branch predictors can detect this pattern so there is minimal branch cost relative to the extra work that is happening inside of BitRunReader

Cool, I will try to look at this tomorrow and kick the tires a bit with benchmarks

Will fix CI issues tomorrow

Need to look into the windows failures some more.

I will review and do some perf tire-kicking in the meantime

If this is not urgent, you can wait for next week and I'll review as well. As you prefer, of course :-)

@pitrou yes, I can wait for you to code review, I will have a brief look and I'm mostly interested in trying it out on some datasets (such as the ones from https://ursalabs.org/blog/2019-10-columnar-perf/)

I fan on the "Fannie Mae 2016 Q4" dataset from that blog post
before
In [1]: %timeit pq.read_table('/home/wesm/code/notebooks/20190919file_benchmarks/2016Q4_uncompressed.parquet', use_threads=True)
1.01 s ± 18.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [2]: %timeit pq.read_table('/home/wesm/code/notebooks/20190919file_benchmarks/2016Q4_uncompressed.parquet', use_threads=False)
5.27 s ± 104 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

after
In [1]: %timeit pq.read_table('/home/wesm/code/notebooks/20190919file_benchmarks/2016Q4_uncompressed.parquet', use_threads=True)
1.01 s ± 9.35 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [2]: %timeit pq.read_table('/home/wesm/code/notebooks/20190919file_benchmarks/2016Q4_uncompressed.parquet', use_threads=False)
4.86 s ± 39.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

So there seems to be a perf improvement in single-threaded decoding but it seems that it's effectively bottlenecked on other issues in multithreaded decoding

@emkornfield if it's OK I'm going to let @pitrou review this when he is back from vacation next week since he's better with these code paths

No rush on review.  The failures on windows are little concerning to me (i.e. there still might be a bug in the code).   @wesm not sure if is is noise, but it looks like variance is reduced with the new code.  Can you share the encodings/schema for the parquet file that you are running?  Would it pay to check-in the file (or a sampled version) and write a c++ benchmark against it?

Also @wesm have you looked at improvements from https://github.com/apache/arrow/pull/6985?

Ah, I will try to look next week on my machine with AVX512

It should be a performance win on SSE4 or AVX as long as the data isn't nested.

  Comments inline.

Ok, I find the implementation quite convoluted. I would like to see the following changes:

Make all critical functions inline

If this is AdvanceTillWord, this hurts performance.  LoadInitialWord in the constructor can be done but I don't think it will have a large impact on performance.

Load a single byte at a time (it probably doesn't bring much to process an entire 64-bit word at a time, and it adds complication)

I'm not sure I understand what direction you would like me to take here.  I would expect word level handling to be much faster for cases this is intended for (when there are actual runs).  In practice, once integrated into a system you are likely correct though.  But I'm not sure exactly what algorithm you are thinking of for byte level handling.  If you provide more description I can try to implement it and compare.

Try to simplify mask handling
Is this mostly InvertRemainingBits? could you provide some more feedback on the algorithm structure you were thinking of?

BTW, with GCC (Clang numbers are generally more favorable to BitRunReader)  I get the following numbers:
BitRunReader/-1                             17084 ns        17083 ns        40761 bytes_per_second=28.5828M/s
BitRunReader/0                                316 ns          316 ns      2208352 bytes_per_second=1.51007G/s
BitRunReader/10                              3899 ns         3898 ns       179454 bytes_per_second=125.263M/s
BitRunReader/25                              7548 ns         7548 ns        91984 bytes_per_second=64.6927M/s
BitRunReader/50                              9519 ns         9518 ns        72676 bytes_per_second=51.3026M/s
BitRunReader/60                              9203 ns         9202 ns        76441 bytes_per_second=53.0612M/s
BitRunReader/75                              7116 ns         7115 ns        98615 bytes_per_second=68.6234M/s
BitRunReader/99                               654 ns          654 ns      1059813 bytes_per_second=747.001M/s
BitRunReaderScalar/-1                       13243 ns        13242 ns        52663 bytes_per_second=36.8737M/s
BitRunReaderScalar/0                         4956 ns         4956 ns       140715 bytes_per_second=98.5307M/s
BitRunReaderScalar/10                        7650 ns         7649 ns        90971 bytes_per_second=63.8322M/s
BitRunReaderScalar/25                       15947 ns        15946 ns        43569 bytes_per_second=30.6216M/s
BitRunReaderScalar/50                       23469 ns        23466 ns        29645 bytes_per_second=20.8077M/s
BitRunReaderScalar/60                       22177 ns        22174 ns        31488 bytes_per_second=22.0202M/s
BitRunReaderScalar/75                       14378 ns        14376 ns        48980 bytes_per_second=33.9642M/s
BitRunReaderScalar/99                        5376 ns         5376 ns       130274 bytes_per_second=90.8295M/s

If this is AdvanceTillWord, this hurts performance.

I don't think compiler-specific micro-optimization is a good practice. Generally, the compiler should make a better job if you expose all the code, rather than hide it.

Is this mostly InvertRemainingBits? could you provide some more feedback on the algorithm structure you were thinking of?

It is. Basically, you could have a running word mask (or byte mask) of the remaining bits in the current word (or byte).

I would expect word level handling to be much faster for cases this is intended for (when there are actual runs). In practice, once integrated into a system you are likely correct though.

Yes, I think you're gonna hit diminishing returns when handling more than one byte at a time. Byte level handling would also alleviate the endianness concerns.

I need to take a closer look to compare the API of this versus what is in #7346, but for the specific use case in #7346, that code is going to be much faster. It would be nice to have some apples-to-apples benchmarks

From a brief look this is a lot more complicated than #7346. Does it have additional features? I can try to set up benchmarks to compare the two performance wise

OK I adapted the benchmark here to use the BitmapScanner from #7346
https://github.com/wesm/arrow/tree/bit-runner
------------------------------------------------------------------
Benchmark                           Time           CPU Iterations
------------------------------------------------------------------
BitRunReader/-1                  9890 ns       9890 ns      70683   49.3705MB/s
BitRunReader/0                    108 ns        108 ns    6250693     4.423GB/s
BitRunReader/10                  2101 ns       2101 ns     334686    232.36MB/s
BitRunReader/25                  4072 ns       4072 ns     173114   119.915MB/s
BitRunReader/50                  5221 ns       5221 ns     133040   93.5178MB/s
BitRunReader/60                  5042 ns       5042 ns     138099   96.8386MB/s
BitRunReader/75                  3933 ns       3933 ns     179857   124.152MB/s
BitRunReader/99                   291 ns        291 ns    2412105    1.6376GB/s
BitRunReaderWithScanner/-1         47 ns         47 ns   15059881   10.2331GB/s
BitRunReaderWithScanner/0          46 ns         46 ns   15078363   10.2704GB/s
BitRunReaderWithScanner/10         47 ns         47 ns   15118172   10.2299GB/s
BitRunReaderWithScanner/25         47 ns         47 ns   15033144   10.2528GB/s
BitRunReaderWithScanner/50         47 ns         47 ns   14947443   10.1964GB/s
BitRunReaderWithScanner/60         47 ns         47 ns   14668505   10.0837GB/s
BitRunReaderWithScanner/75         46 ns         46 ns   15045334   10.2918GB/s
BitRunReaderWithScanner/99         46 ns         46 ns   14961067   10.2813GB/s
BitRunReaderScalar/-1           13089 ns      13088 ns      51449   37.3063MB/s
BitRunReaderScalar/0             3844 ns       3844 ns     176221   127.024MB/s
BitRunReaderScalar/10            6621 ns       6621 ns     104648   73.7517MB/s
BitRunReaderScalar/25           12397 ns      12397 ns      55998    39.388MB/s
BitRunReaderScalar/50           17099 ns      17099 ns      41378    28.556MB/s
BitRunReaderScalar/60           16606 ns      16606 ns      42580   29.4046MB/s
BitRunReaderScalar/75           11431 ns      11431 ns      61744   42.7165MB/s
BitRunReaderScalar/99            4265 ns       4265 ns     167402   114.484MB/s

If this is AdvanceTillWord, this hurts performance.

I don't think compiler-specific micro-optimization is a good practice. Generally, the compiler should make a better job if you expose all the code, rather than hide it.

OK moved everything to the header.  I looked again at the performance numbers inlining seems to improve benchmarks for randomly distributed bits slightly (eyeballing looks like ~20ish percent over existing improvements) but makes the worst case regression 50-100% worse (i.e. in the worse case it is 100% slower then before).

Is this mostly InvertRemainingBits? could you provide some more feedback on the algorithm structure you were thinking of?

It is. Basically, you could have a running word mask (or byte mask) of the remaining bits in the current word (or byte).

Removed InvertRemainingBits and generate remaining bit mask on the fly.

I would expect word level handling to be much faster for cases this is intended for (when there are actual runs). In practice, once integrated into a system you are likely correct though.

Yes, I think you're gonna hit diminishing returns when handling more than one byte at a time. Byte level handling would also alleviate the endianness concerns.

I would rather not expend the effort to refactor this to byte at a time.  It will likely take a performance hit at least on microbenchmarks, and I'd rather spend the effort debugging and fixing the windows builds if that is OK.

Also for endiannes, I was conservative in the implementation I think its possible most of the code will just work with a single ToLittleEndian call in the appopriate place.

This isn't apples-to-applies at all because the scanner just popcounts 256-bits at a time, it doesn't segment null- from non-null runs. If the goal is to accelerate the writing of mostly-non-null data, is it worth going to all this trouble of exactly delimiting the start and end point of each run?

@wesm I think the parquet benchmarks show the answer to this question is yes.  They also show there is a hybrid solution between the two where we first check for larger blocks of non-null and then use the BitRunReader.  But for instance on the parquet benchmarks with 1% null rate we see ~20-30% speedup.  Using the BitBlockCounter at this rate of nulls we would expect to see ~8 percent of blocks non have at least 1 null in them.  Maybe 1% null rate is still higher then most datasets have in practice though?  We should probably document what performance parameters we want to optimize for as it would help the decisions (i.e 0.01%, 0.1%, 0.5%, 1% or 5%).
If we didn't use BitRunReader at all and only used the BitBlockCounter then we would like avoid the bad regression on deterministically alternative nulls.
All that being said, if we don't think the approach in this PR is promising I'm happy to abandon it.

My anecdotal experience suggests that 1% is a higher-than-average rate of nulls, but I don't know how accurate that is. The popcount strategy could be adapted to do 64 bits at a time (instead of the 256 that's implemented now) which would mean a higher incidence of no-nulls blocks in the 1% case.

My anecdotal experience suggests that 1% is a higher-than-average rate of nulls, but I don't know how accurate that is. The popcount strategy could be adapted to do 64 bits at a time (instead of the 256 that's implemented now) which would mean a higher incidence of no-nulls blocks in the 1% case.

@wesm Yeah, it would still expect some performance improvements with this approach at 1% at 64 bit blocks (roughly 48% of blocks would still use this path).  I don't think there is necessarily a scientific way of determining this.  We should probably just draw a line in the sand.  I'm willing to revisit the approach in this PR to use BitBlockCounter once it is checked in if you and @pitrou would prefer to avoid complexity here.

I could try to adapt the Parquet code to use BitBlockCounter and see what the benchmarks look like?

I could try to adapt the Parquet code to use BitBlockCounter and see what the benchmarks look like?

@wesm I think if you can take a look and potentially revise the benchmarks at https://github.com/apache/arrow/blob/7ad49eeca5215d9b2a56b6439f1bd6ea38888ea9/cpp/src/parquet/arrow/reader_writer_benchmark.cc#L238 to make sure we are aligned on what we are trying to improve, I can update the this PR accordingly.  I think there are really two options:

Remove BitRunReader entirely and use BitBlockCounter
Use BitBlockCounter in addition to BitRunReader

The way to go really depends on what percentage of values we expect to be null.  My intuition is that very high rates and very low rates are likely, but I think you probably have a better intuition as to the exact definition of high or low.

@emkornfield I can take a look at this once https://github.com/apache/arrow/pull/7356 is merged which adds a single-word function to BitBlockCounter

@emkornfield I'm sorry that I've been neglecting this PR. I will try to rebase this and investigate the perf questions a little bit

I rebased. I'm going to look at the benchmarks locally and then probably merge this so further performance work (including comparing with BitBlockCounter) can be pursued as follow up

Here's what I see with gcc-8:
                                benchmark          baseline         contender  change %  regression
1       BM_ReadColumn<true,Int32Type>/1/1     1.336 GiB/sec     1.880 GiB/sec    40.701       False
17      BM_ReadColumn<true,Int64Type>/1/1     2.224 GiB/sec     2.926 GiB/sec    31.567       False
14     BM_ReadColumn<true,Int32Type>/99/0     1.599 GiB/sec     2.001 GiB/sec    25.128       False
23    BM_ReadColumn<true,Int32Type>/99/50     1.601 GiB/sec     1.990 GiB/sec    24.300       False
21    BM_ReadColumn<true,Int64Type>/50/50  1002.325 MiB/sec     1.206 GiB/sec    23.171       False
12    BM_ReadColumn<true,Int32Type>/50/50   619.919 MiB/sec   758.845 MiB/sec    22.410       False
33    BM_ReadColumn<true,Int64Type>/45/25  1016.908 MiB/sec     1.212 GiB/sec    22.095       False
30     BM_ReadColumn<true,Int32Type>/50/0   620.068 MiB/sec   757.008 MiB/sec    22.085       False
34      BM_ReadColumn<true,Int64Type>/5/5     1.586 GiB/sec     1.935 GiB/sec    21.981       False
27     BM_ReadColumn<true,Int64Type>/99/0     2.539 GiB/sec     3.081 GiB/sec    21.308       False
13     BM_ReadColumn<true,Int64Type>/10/5     1.361 GiB/sec     1.644 GiB/sec    20.791       False
18     BM_ReadColumn<true,Int64Type>/50/1     1.002 GiB/sec     1.204 GiB/sec    20.174       False
26     BM_ReadColumn<true,Int64Type>/75/1     1.182 GiB/sec     1.419 GiB/sec    20.005       False
15    BM_ReadColumn<true,Int64Type>/99/50     2.532 GiB/sec     3.036 GiB/sec    19.890       False
11    BM_ReadColumn<true,Int64Type>/30/10     1.123 GiB/sec     1.322 GiB/sec    17.754       False
35    BM_ReadColumn<true,Int64Type>/25/10     1.187 GiB/sec     1.396 GiB/sec    17.558       False
25     BM_ReadColumn<true,Int32Type>/25/5   741.612 MiB/sec   870.530 MiB/sec    17.384       False
3     BM_ReadColumn<true,Int32Type>/10/10   859.032 MiB/sec  1005.688 MiB/sec    17.072       False
31    BM_ReadColumn<true,Int64Type>/35/10     1.095 GiB/sec     1.254 GiB/sec    14.528       False
20   BM_ReadColumn<true,BooleanType>/-1/1   712.571 MiB/sec   808.480 MiB/sec    13.460       False
5    BM_ReadColumn<true,DoubleType>/25/25     1.244 GiB/sec     1.399 GiB/sec    12.397       False
9    BM_ReadColumn<true,BooleanType>/5/10   460.741 MiB/sec   484.654 MiB/sec     5.190       False
7   BM_ReadColumn<false,BooleanType>/1/20   397.103 MiB/sec   412.517 MiB/sec     3.882       False
29  BM_ReadColumn<false,BooleanType>/-1/0   984.767 MiB/sec   994.483 MiB/sec     0.987       False
19   BM_ReadColumn<false,DoubleType>/-1/0    13.609 GiB/sec    13.738 GiB/sec     0.944       False
28    BM_ReadColumn<false,Int32Type>/-1/1     4.910 GiB/sec     4.702 GiB/sec    -4.236       False
8     BM_ReadColumn<false,Int64Type>/-1/1     8.631 GiB/sec     8.000 GiB/sec    -7.310        True
6    BM_ReadColumn<false,Int64Type>/-1/10     2.872 GiB/sec     2.329 GiB/sec   -18.912        True
32   BM_ReadColumn<false,Int32Type>/-1/10     1.457 GiB/sec     1.099 GiB/sec   -24.615        True
22   BM_ReadColumn<true,DoubleType>/10/50     1.433 GiB/sec   964.501 MiB/sec   -34.267        True
24     BM_ReadColumn<true,Int64Type>/-1/0     2.325 GiB/sec     1.400 GiB/sec   -39.804        True
2     BM_ReadColumn<true,DoubleType>/-1/0     2.412 GiB/sec     1.422 GiB/sec   -41.059        True
10     BM_ReadColumn<true,Int32Type>/-1/0     1.454 GiB/sec   864.019 MiB/sec   -41.959        True
16  BM_ReadColumn<false,DoubleType>/-1/20     3.826 GiB/sec     2.184 GiB/sec   -42.899        True
0    BM_ReadColumn<false,Int64Type>/-1/50     4.669 GiB/sec     1.865 GiB/sec   -60.055        True
4    BM_ReadColumn<false,Int32Type>/-1/50     3.184 GiB/sec   862.581 MiB/sec   -73.546        True

I agree that the alternating NA/not-NA is pretty pathological so IMHO this performance regression doesn't seem like a big deal to me.

Here's with clang, even less of an issue there it seems
                                benchmark         baseline        contender  change %  regression
32      BM_ReadColumn<true,Int32Type>/1/1    1.656 GiB/sec    2.689 GiB/sec    62.384       False
15     BM_ReadColumn<true,Int64Type>/99/0    2.861 GiB/sec    4.184 GiB/sec    46.245       False
21    BM_ReadColumn<true,Int32Type>/99/50    1.890 GiB/sec    2.744 GiB/sec    45.220       False
20    BM_ReadColumn<true,Int64Type>/99/50    2.898 GiB/sec    4.199 GiB/sec    44.891       False
6      BM_ReadColumn<true,Int64Type>/50/1    1.530 GiB/sec    2.210 GiB/sec    44.402       False
30    BM_ReadColumn<true,Int64Type>/50/50    1.586 GiB/sec    2.233 GiB/sec    40.787       False
28     BM_ReadColumn<true,Int32Type>/50/0  881.102 MiB/sec    1.196 GiB/sec    39.025       False
22    BM_ReadColumn<true,Int64Type>/45/25    1.598 GiB/sec    2.221 GiB/sec    38.951       False
14      BM_ReadColumn<true,Int64Type>/1/1    2.967 GiB/sec    4.122 GiB/sec    38.919       False
29    BM_ReadColumn<true,Int32Type>/50/50  887.206 MiB/sec    1.203 GiB/sec    38.854       False
35     BM_ReadColumn<true,Int32Type>/99/0    1.928 GiB/sec    2.661 GiB/sec    38.045       False
17    BM_ReadColumn<true,Int64Type>/35/10    1.656 GiB/sec    2.248 GiB/sec    35.689       False
0    BM_ReadColumn<false,Int32Type>/-1/50    2.765 GiB/sec    3.750 GiB/sec    35.628       False
26    BM_ReadColumn<true,Int64Type>/30/10    1.692 GiB/sec    2.221 GiB/sec    31.244       False
19     BM_ReadColumn<true,Int64Type>/75/1    1.712 GiB/sec    2.217 GiB/sec    29.505       False
2      BM_ReadColumn<true,Int32Type>/25/5  991.022 MiB/sec    1.229 GiB/sec    27.031       False
31    BM_ReadColumn<true,Int64Type>/25/10    1.744 GiB/sec    2.164 GiB/sec    24.091       False
16   BM_ReadColumn<false,Int64Type>/-1/50    5.892 GiB/sec    7.252 GiB/sec    23.084       False
24    BM_ReadColumn<true,Int32Type>/10/10    1.003 GiB/sec    1.225 GiB/sec    22.073       False
23   BM_ReadColumn<true,DoubleType>/25/25    1.790 GiB/sec    2.159 GiB/sec    20.659       False
10      BM_ReadColumn<true,Int64Type>/5/5    1.995 GiB/sec    2.371 GiB/sec    18.853       False
5      BM_ReadColumn<true,Int64Type>/10/5    1.796 GiB/sec    2.122 GiB/sec    18.128       False
3   BM_ReadColumn<false,DoubleType>/-1/20    3.488 GiB/sec    4.062 GiB/sec    16.453       False
1    BM_ReadColumn<false,Int64Type>/-1/10    2.740 GiB/sec    3.055 GiB/sec    11.497       False
11   BM_ReadColumn<false,Int32Type>/-1/10    1.362 GiB/sec    1.486 GiB/sec     9.086       False
9    BM_ReadColumn<false,DoubleType>/-1/0   12.984 GiB/sec   13.310 GiB/sec     2.509       False
13   BM_ReadColumn<true,BooleanType>/5/10  515.781 MiB/sec  523.838 MiB/sec     1.562       False
7     BM_ReadColumn<false,Int32Type>/-1/1    4.505 GiB/sec    4.559 GiB/sec     1.198       False
34  BM_ReadColumn<false,BooleanType>/1/20  400.254 MiB/sec  403.780 MiB/sec     0.881       False
25  BM_ReadColumn<false,BooleanType>/-1/0  870.109 MiB/sec  875.827 MiB/sec     0.657       False
18    BM_ReadColumn<false,Int64Type>/-1/1    8.237 GiB/sec    8.045 GiB/sec    -2.331       False
27   BM_ReadColumn<true,BooleanType>/-1/1  816.651 MiB/sec  793.763 MiB/sec    -2.803       False
8    BM_ReadColumn<true,DoubleType>/10/50    1.649 GiB/sec    1.587 GiB/sec    -3.792       False
12     BM_ReadColumn<true,Int64Type>/-1/0    3.044 GiB/sec    1.432 GiB/sec   -52.958        True
33    BM_ReadColumn<true,DoubleType>/-1/0    3.161 GiB/sec    1.479 GiB/sec   -53.209        True
4      BM_ReadColumn<true,Int32Type>/-1/0    1.863 GiB/sec  754.619 MiB/sec   -60.448        True

I'm not sure what the MSVC failure is about but I'll debug locally

I'm able to reproduce the error in VS and set breakpoints, I got this far to see that GetBatchWithDictSpaced has decoded more values than it was asked to

there seems to be a situation where the bit run has more values then are needed to fulfill the call to GetSpaced

@emkornfield I'm sort of at a dead end here, hopefully the above gives you some clues about where there might be a problem

The bug seems to be in the BitRunReader

Thanks, I'll take a look tonight.  Hopefully this should be enough of a clue.

I fixed the bug and some other issues.  I also incorporated BitBlockCounter in rle_encoder.h.  Latest benchmarks.

This is with SSE4_2

FWIW gcc benchmarks (sse4.2) on my machine (ubuntu 18.04 on i9-9960X)
$ archery benchmark diff --cc=gcc-8 --cxx=g++-8 emkornfield/ARROW-8504 master --suite-filter=parquet-arrow
                                benchmark         baseline        contender  change %            counters
35     BM_ReadColumn<true,Int64Type>/75/1  267.930 MiB/sec  275.083 MiB/sec     2.670   {'iterations': 2}
6      BM_ReadColumn<true,Int32Type>/99/0  238.381 MiB/sec  244.286 MiB/sec     2.477   {'iterations': 3}
10   BM_ReadColumn<true,DoubleType>/10/50  270.775 MiB/sec  275.249 MiB/sec     1.652   {'iterations': 2}
31     BM_ReadColumn<true,Int32Type>/25/5  171.549 MiB/sec  174.237 MiB/sec     1.567   {'iterations': 2}
29    BM_ReadColumn<false,Int32Type>/-1/1  856.194 MiB/sec  866.945 MiB/sec     1.256  {'iterations': 11}
7        BM_WriteColumn<true,BooleanType>   31.593 MiB/sec   31.845 MiB/sec     0.797   {'iterations': 1}
44             BM_ReadIndividualRowGroups  849.469 MiB/sec  856.209 MiB/sec     0.793   {'iterations': 6}
36    BM_ReadColumn<true,DoubleType>/-1/0  408.388 MiB/sec  411.490 MiB/sec     0.759   {'iterations': 3}
3      BM_ReadColumn<true,Int32Type>/50/0  151.600 MiB/sec  152.399 MiB/sec     0.527   {'iterations': 2}
32    BM_ReadColumn<true,Int64Type>/30/10  268.512 MiB/sec  269.849 MiB/sec     0.498   {'iterations': 2}
8   BM_ReadColumn<false,DoubleType>/-1/20  772.878 MiB/sec  776.716 MiB/sec     0.497   {'iterations': 5}
17   BM_ReadColumn<true,DoubleType>/25/25  277.705 MiB/sec  278.957 MiB/sec     0.451   {'iterations': 2}
23    BM_ReadColumn<false,Int64Type>/-1/1    1.472 GiB/sec    1.478 GiB/sec     0.414  {'iterations': 13}
30      BM_ReadColumn<true,Int32Type>/1/1  235.809 MiB/sec  236.730 MiB/sec     0.391   {'iterations': 3}
45         BM_WriteColumn<true,Int32Type>   45.067 MiB/sec   45.232 MiB/sec     0.366   {'iterations': 1}
41    BM_ReadColumn<true,Int64Type>/45/25  241.545 MiB/sec  242.427 MiB/sec     0.365   {'iterations': 2}
40   BM_ReadColumn<false,Int64Type>/-1/10  710.485 MiB/sec  712.181 MiB/sec     0.239   {'iterations': 5}
22  BM_ReadColumn<false,BooleanType>/-1/0  211.939 MiB/sec  212.433 MiB/sec     0.233  {'iterations': 15}
39   BM_ReadColumn<false,Int32Type>/-1/50  629.632 MiB/sec  630.955 MiB/sec     0.210   {'iterations': 9}
15  BM_ReadColumn<false,BooleanType>/1/20  145.490 MiB/sec  145.701 MiB/sec     0.145  {'iterations': 10}
9    BM_ReadColumn<true,BooleanType>/-1/1  162.373 MiB/sec  162.497 MiB/sec     0.077   {'iterations': 4}
11     BM_ReadColumn<true,Int32Type>/-1/0  257.190 MiB/sec  257.278 MiB/sec     0.034   {'iterations': 3}
13        BM_WriteColumn<false,Int32Type>   32.378 MiB/sec   32.386 MiB/sec     0.025   {'iterations': 1}
37    BM_ReadColumn<true,Int32Type>/99/50  238.918 MiB/sec  238.975 MiB/sec     0.024   {'iterations': 3}
20       BM_WriteColumn<false,DoubleType>   54.200 MiB/sec   54.209 MiB/sec     0.017   {'iterations': 1}
19      BM_ReadColumn<true,Int64Type>/1/1  369.522 MiB/sec  369.199 MiB/sec    -0.087   {'iterations': 2}
0    BM_ReadColumn<false,Int64Type>/-1/50    1.144 GiB/sec    1.143 GiB/sec    -0.090  {'iterations': 10}
34    BM_ReadColumn<true,Int64Type>/35/10  262.271 MiB/sec  261.890 MiB/sec    -0.145   {'iterations': 2}
21     BM_ReadColumn<true,Int64Type>/50/1  242.510 MiB/sec  242.067 MiB/sec    -0.183   {'iterations': 2}
5     BM_ReadColumn<true,Int32Type>/50/50  152.344 MiB/sec  152.000 MiB/sec    -0.225   {'iterations': 2}
1     BM_ReadColumn<true,Int32Type>/10/10  179.063 MiB/sec  178.583 MiB/sec    -0.268   {'iterations': 2}
28        BM_WriteColumn<false,Int64Type>   64.162 MiB/sec   63.990 MiB/sec    -0.269   {'iterations': 1}
38        BM_WriteColumn<true,DoubleType>   69.641 MiB/sec   69.446 MiB/sec    -0.280   {'iterations': 1}
18      BM_WriteColumn<false,BooleanType>   26.643 MiB/sec   26.557 MiB/sec    -0.324   {'iterations': 2}
16   BM_ReadColumn<false,DoubleType>/-1/0    2.291 GiB/sec    2.281 GiB/sec    -0.397  {'iterations': 20}
43         BM_WriteColumn<true,Int64Type>   75.453 MiB/sec   75.050 MiB/sec    -0.534   {'iterations': 1}
33   BM_ReadColumn<true,BooleanType>/5/10  115.534 MiB/sec  114.574 MiB/sec    -0.832   {'iterations': 3}
14    BM_ReadColumn<true,Int64Type>/50/50  242.071 MiB/sec  240.005 MiB/sec    -0.854   {'iterations': 2}
4                BM_ReadMultipleRowGroups  826.026 MiB/sec  818.709 MiB/sec    -0.886   {'iterations': 5}
24   BM_ReadColumn<false,Int32Type>/-1/10  386.383 MiB/sec  382.213 MiB/sec    -1.079   {'iterations': 6}
12     BM_ReadColumn<true,Int64Type>/-1/0  410.877 MiB/sec  404.700 MiB/sec    -1.503   {'iterations': 3}
25     BM_ReadColumn<true,Int64Type>/10/5  289.612 MiB/sec  284.777 MiB/sec    -1.670   {'iterations': 2}
2     BM_ReadColumn<true,Int64Type>/25/10  277.045 MiB/sec  271.925 MiB/sec    -1.848   {'iterations': 2}
26      BM_ReadColumn<true,Int64Type>/5/5  311.137 MiB/sec  295.823 MiB/sec    -4.922   {'iterations': 2}
42     BM_ReadColumn<true,Int64Type>/99/0  371.803 MiB/sec  347.872 MiB/sec    -6.436   {'iterations': 2}
27    BM_ReadColumn<true,Int64Type>/99/50  381.029 MiB/sec  351.329 MiB/sec    -7.795   {'iterations': 3}

Here's clang-11
$ archery benchmark diff --cc=clang-11 --cxx=clang++-11 emkornfield/ARROW-8504 master --suite-filter=parquet-arrow
                                benchmark         baseline        contender  change %            counters
37    BM_ReadColumn<true,Int64Type>/35/10  242.396 MiB/sec  259.801 MiB/sec     7.180   {'iterations': 2}
17   BM_ReadColumn<false,DoubleType>/-1/0    2.160 GiB/sec    2.276 GiB/sec     5.363  {'iterations': 20}
0       BM_ReadColumn<true,Int64Type>/5/5  304.868 MiB/sec  313.829 MiB/sec     2.939   {'iterations': 2}
10     BM_ReadColumn<true,Int64Type>/-1/0  398.515 MiB/sec  408.179 MiB/sec     2.425   {'iterations': 3}
3      BM_ReadColumn<true,Int64Type>/10/5  286.868 MiB/sec  289.629 MiB/sec     0.962   {'iterations': 2}
18             BM_ReadIndividualRowGroups  845.304 MiB/sec  851.353 MiB/sec     0.716   {'iterations': 5}
33     BM_ReadColumn<true,Int64Type>/99/0  372.125 MiB/sec  374.716 MiB/sec     0.696   {'iterations': 2}
12     BM_ReadColumn<true,Int32Type>/50/0  151.242 MiB/sec  152.137 MiB/sec     0.591   {'iterations': 2}
8                BM_ReadMultipleRowGroups  819.229 MiB/sec  823.423 MiB/sec     0.512   {'iterations': 5}
1     BM_ReadColumn<true,Int64Type>/25/10  270.997 MiB/sec  272.218 MiB/sec     0.450   {'iterations': 2}
16         BM_WriteColumn<true,Int32Type>   44.766 MiB/sec   44.946 MiB/sec     0.402   {'iterations': 1}
11     BM_ReadColumn<true,Int64Type>/50/1  241.717 MiB/sec  242.539 MiB/sec     0.340   {'iterations': 2}
45    BM_ReadColumn<true,Int64Type>/50/50  241.083 MiB/sec  241.646 MiB/sec     0.233   {'iterations': 2}
27      BM_WriteColumn<false,BooleanType>   26.050 MiB/sec   26.095 MiB/sec     0.170   {'iterations': 2}
20    BM_ReadColumn<false,Int32Type>/-1/1  864.068 MiB/sec  865.475 MiB/sec     0.163  {'iterations': 11}
7         BM_WriteColumn<true,DoubleType>   69.439 MiB/sec   69.545 MiB/sec     0.152   {'iterations': 1}
19   BM_ReadColumn<true,DoubleType>/25/25  275.396 MiB/sec  275.811 MiB/sec     0.151   {'iterations': 2}
23    BM_ReadColumn<true,Int64Type>/99/50  370.614 MiB/sec  371.017 MiB/sec     0.109   {'iterations': 2}
9      BM_ReadColumn<true,Int32Type>/25/5  173.728 MiB/sec  173.861 MiB/sec     0.076   {'iterations': 2}
5     BM_ReadColumn<true,DoubleType>/-1/0  410.275 MiB/sec  410.482 MiB/sec     0.051   {'iterations': 3}
44       BM_WriteColumn<false,DoubleType>   54.559 MiB/sec   54.586 MiB/sec     0.050   {'iterations': 1}
36       BM_WriteColumn<true,BooleanType>   31.427 MiB/sec   31.426 MiB/sec    -0.003   {'iterations': 1}
28   BM_ReadColumn<true,BooleanType>/5/10  115.041 MiB/sec  114.909 MiB/sec    -0.115   {'iterations': 3}
2     BM_ReadColumn<true,Int64Type>/45/25  242.615 MiB/sec  242.257 MiB/sec    -0.148   {'iterations': 2}
6     BM_ReadColumn<false,Int64Type>/-1/1    1.471 GiB/sec    1.469 GiB/sec    -0.157  {'iterations': 13}
4   BM_ReadColumn<false,BooleanType>/1/20  145.277 MiB/sec  145.008 MiB/sec    -0.185  {'iterations': 10}
24    BM_ReadColumn<true,Int32Type>/99/50  238.739 MiB/sec  238.281 MiB/sec    -0.192   {'iterations': 3}
34        BM_WriteColumn<false,Int32Type>   32.013 MiB/sec   31.937 MiB/sec    -0.237   {'iterations': 1}
30  BM_ReadColumn<false,BooleanType>/-1/0  213.488 MiB/sec  212.948 MiB/sec    -0.253  {'iterations': 14}
39     BM_ReadColumn<true,Int32Type>/-1/0  259.936 MiB/sec  259.218 MiB/sec    -0.276   {'iterations': 3}
14    BM_ReadColumn<true,Int64Type>/30/10  270.662 MiB/sec  269.891 MiB/sec    -0.285   {'iterations': 2}
26        BM_WriteColumn<false,Int64Type>   62.818 MiB/sec   62.576 MiB/sec    -0.385   {'iterations': 1}
13     BM_ReadColumn<true,Int64Type>/75/1  275.538 MiB/sec  274.461 MiB/sec    -0.391   {'iterations': 2}
32   BM_ReadColumn<false,Int32Type>/-1/50  631.052 MiB/sec  627.655 MiB/sec    -0.538   {'iterations': 9}
38  BM_ReadColumn<false,DoubleType>/-1/20  771.752 MiB/sec  766.959 MiB/sec    -0.621   {'iterations': 5}
22   BM_ReadColumn<false,Int64Type>/-1/10  712.603 MiB/sec  708.037 MiB/sec    -0.641   {'iterations': 5}
21      BM_ReadColumn<true,Int64Type>/1/1  370.007 MiB/sec  367.621 MiB/sec    -0.645   {'iterations': 2}
25    BM_ReadColumn<true,Int32Type>/10/10  182.975 MiB/sec  181.714 MiB/sec    -0.689   {'iterations': 2}
29   BM_ReadColumn<false,Int64Type>/-1/50    1.144 GiB/sec    1.135 GiB/sec    -0.713  {'iterations': 10}
41   BM_ReadColumn<true,DoubleType>/10/50  271.556 MiB/sec  269.579 MiB/sec    -0.728   {'iterations': 2}
42   BM_ReadColumn<false,Int32Type>/-1/10  390.676 MiB/sec  386.926 MiB/sec    -0.960   {'iterations': 6}
35         BM_WriteColumn<true,Int64Type>   74.108 MiB/sec   73.364 MiB/sec    -1.004   {'iterations': 1}
40      BM_ReadColumn<true,Int32Type>/1/1  236.583 MiB/sec  233.838 MiB/sec    -1.160   {'iterations': 3}
43   BM_ReadColumn<true,BooleanType>/-1/1  162.242 MiB/sec  160.101 MiB/sec    -1.320   {'iterations': 4}
31     BM_ReadColumn<true,Int32Type>/99/0  236.270 MiB/sec  232.278 MiB/sec    -1.690   {'iterations': 3}
15    BM_ReadColumn<true,Int32Type>/50/50  152.712 MiB/sec  148.673 MiB/sec    -2.645   {'iterations': 2}

+1, thanks for fixing this @emkornfield!


runtime cmake error:add_subdirectory given source "kaldi" which is not an existing directory.

Describe the bug
runtime cmake error:add_subdirectory given source "kaldi" which is not an existing directory.

To Reproduce
Steps to reproduce the behavior:

Go to 'wenet/runtime/server/x86'
mkdir build && cd build && cmake -DINTTYPES_FORMAT:STRING=C99 .. && cmake --build .

Desktop :

OS: [CentOS Linux release 7.6.1810 (Core) ]

由于是从windows平台拷贝到centos的，如果直接clone到centos平台没有这个错误


SP5: SharedLayoutManager is not respecting ShareX and ShareY

this section needs to be updated to only update the axes enabled by the two properties
https://github.com/ScottPlot/ScottPlot/blob/42987b60b6fb90e94d0fb67a60c7dfafc73d8d9a/src/ScottPlot5/ScottPlot5/SharedLayoutManager.cs#L7-L8
https://github.com/ScottPlot/ScottPlot/blob/42987b60b6fb90e94d0fb67a60c7dfafc73d8d9a/src/ScottPlot5/ScottPlot5/SharedLayoutManager.cs#L32-L43

Actually, fixing this probably requires adding logic inside FixedLayout
https://github.com/ScottPlot/ScottPlot/blob/main/src/ScottPlot5/ScottPlot5/LayoutEngines/FixedDataArea.cs

I'll probably rollback the shared layout manager in favor of a simpler solution I've discovered. I'll document it by adding it to the SP5 demo app.


Switch to Maven Central since jCenter quit

@asm0dey Obviously, if you want to use another repo, let me know!
But since jCenter quit (https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/), we need to switch to another repo for the dependencies.
Maven Central seems to work fine!

Already deleted in my local repo :)

@asm0dey
Oh it can actually be entirely deleted and it will default to maven Central?
I switched from jcenter to see if it could be upgraded to a newer scala version but it seems TraversableLike from Scala is causing issues after 1.12.10 :/

And that's why we're waiting for Kotlin 1.5.20

@asm0dey Oh, I think it just arrived!

Fixed in upstream


[BEAM-4809] Build javadoc on :javaPreCommit and :javaPostCommit

Build javadoc on :javaPreCommit and :javaPostCommit to check it builds ok
Please add a meaningful description for your change here

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---
---

Javadoc should be fixed by https://github.com/apache/beam/pull/5972

@boyuanzz

Do you think it would be better to ensure that the check task depends on javadoc in each project. This way during the course of a normal build a user would get javadoc failures instead of only getting the failure during the precommit run?

Run Java PreCommit

Hey Alan, do we have enabled java spotless tasks in precommit and postcommit? If not, do you think we can add them into at least one of the check job?

Run Java PreCommit

Run Java PreCommit

Run Seed Job

Run Java PreCommit

Run Java PreCommit

Run Java PreCommit

What is the state of this PR?

@aaltay It is ready to be merged;
precommits passed including the new javadoc check.

Very nice. Merging.


[BUG] definitions of global constants

Describe the bug
I've never been able to define a global cdef constant value using Cython. I've always assumed the feature was not supported due to the way code was generated.
However, I just read in the documentation: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#deprecation-of-def-if

Usages of DEF should be replaced by:
global cdef constants

Am I the only one to not be able to define cdef constants ?
Code to reproduce the behaviour:
Error compiling Cython file:
------------------------------------------------------------
...
from libc.math cimport sqrt as csqrt
cdef const int int_zero = 0
     ^
------------------------------------------------------------
choreo/scipy_plus/cython/blas_consts.pyx:11:5: Assignment to const 'int_zero'

Expected behaviour
No compile time error.
OS
WSL2
Python version
3.11.2
Cython version
3.0.8
Additional context
No response

There is currently Draft PR #5242 implementing global const variables. The issue there is that it translate cdef const directly to global const C variable which does not support initialisation of arrays - see https://github.com/cython/cython/pull/5242#issuecomment-1419810587
Possible solution is to not allow to use const variables for initialisation of array size but we need to figure out how we can communicate it to the users properly...


Add an expandconfig utility to add default values

This patch adds an expandconfig binary which is able to load a
configuration file, and write it back (in either TOML or YAML format)
with the default values expanded.  This is useful in a variety of
scenarios:

Check that a configuration file is valid with:
expandconfig /path/to/config.toml

See default field values and structure with:
expandconfig

See all available fields with (YAML displays fields with a default
None value while TOML skips them):
expandconfig -f yaml

@ulysseB expandconfig now uses structopt.  I'll try converting the argument parsing code in explorer/config to structopt in a separate PR when I have time.

Ok. Can I merge this ?

bors r=ulysseB


RFC: make lattice concrete

@aviatesk As we discussed, I was curious to see how this would work out. Seemed like this was a good test case for it, since the change was very simple with the current EscapeAnalysis design. As an example, I added some discussion of what ArgEscape would look like in this framework. I think ThrownEscape would also be simple to add to this (instead of making a dual throws/returns lattice, as we'd discussed somewhat on a recent call).
This will hopefully allow us to express a more complex lattice, and avoid unnecessary abstract dispatch in the analysis pass itself.

Looks great. I will try to implement ThrowEscape on top of this design, and see how it feels to develop a lattice with this design.
Maybe we can even encode flow sensitivity into this lattice, by tracking the statement number of escape sites ?

I'm going to merge this PR for now. Thanks Jameson for taking the initiative on this.


Nested if can be replaced with else-if

Signed-off-by: arcolife<EMAIL_ADDRESS>Many thanks for submitting your Pull Request :heart:!
What this PR does / why we need it:
Nested if can be replaced with else-if.
why we need it?: 🤷🏼♂
Special notes for reviewers:
not a bug? 🤷🏼♂
Additional information (if needed):

Hi! Thanks for your contribution, but we are about to merge #25, which completes rewrite this feature, deprecating this file.


[BUG] Negative Search Stats

Describe the bug
We have identified a bug where search statistics are being incorrectly reported as negative, which is causing failures when accessing the nodes/stats API. This issue is specifically related to the handling of negative values in the SearchStats class. The exception occurs when the API attempts to serialize the statistics, causing a java.lang.IllegalStateException due to the use of VLong encoding, which does not support negative values.
Example Response from nodes/stats API:
{
    "_nodes": {
        "total": 6,
        "successful": 5,
        "failed": 1,
        "failures": [
            {
                "type": "failed_node_exception",
                "reason": "Failed node [gxu1I2AFSo2Co-gL3Cq14A]",
                "node_id": "gxu1I2AFSo2Co-gL3Cq14A",
                "caused_by": {
                    "type": "illegal_state_exception",
                    "reason": "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-3]"
                }
            }
        ]
    },
...

Exception in elasticsearch.log:
[2024-10-16T19:55:29,039][WARN ][o.o.t.OutboundHandler    ] [8da3ab95536131cd3252613bab350bd7] send message failed [channel: Netty4TcpChannel{localAddress=/<IP_ADDRESS>:9300, remoteAddress=/<IP_ADDRESS>:40652}]
java.lang.IllegalStateException: Negative longs unsupported, use writeLong or writeZLong for negative numbers [-140]
    at org.opensearch.common.io.stream.StreamOutput.writeVLong(StreamOutput.java:307)
    at org.opensearch.index.search.stats.SearchStats$PhaseStatsLongHolder.writeTo(SearchStats.java:88)
    at org.opensearch.index.search.stats.SearchStats$Stats.lambda$writeTo$0(SearchStats.java:393)
    at org.opensearch.common.io.stream.StreamOutput.writeMap(StreamOutput.java:637)
    at org.opensearch.index.search.stats.SearchStats$Stats.writeTo(SearchStats.java:390)
    at org.opensearch.index.search.stats.SearchStats.writeTo(SearchStats.java:588)
    at org.opensearch.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:959)
    at org.opensearch.action.admin.indices.stats.CommonStats.writeTo(CommonStats.java:273)
    at org.opensearch.indices.NodeIndicesStats.writeTo(NodeIndicesStats.java:207)
    at org.opensearch.action.admin.cluster.node.stats.NodeStats.writeTo(NodeStats.java:429)
    at org.opensearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:104)
    at org.opensearch.transport.OutboundMessage.serialize(OutboundMessage.java:81)
    at org.opensearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:234)
    at org.opensearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:220)

Root Cause
The issue occurs due to the use of StreamOutput.writeVLong() in the SearchStats class, which does not support negative long values. The stack trace shows that when a negative value is encountered, it triggers the exception in elasticsearch.log shown above.
The problematic line in the code is:
out.writeVLong(current);

Source: SearchStats.java#L89
Related component
Search
To Reproduce
This bug does not occur consistently across all domains or search queries. We are still investigating the specific conditions or types of searches that trigger this issue.
Expected behavior
Search stats should not be negative.
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):

OS: [e.g. iOS]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Probably introduced in https://github.com/opensearch-project/OpenSearch/pull/8386
Similar issue https://github.com/opensearch-project/OpenSearch/issues/13343

The current stat can be decremented in two places, on phase end (success):
https://github.com/opensearch-project/OpenSearch/blob/e68838819710d7040cf2b591590285f1b86f0da0/server/src/main/java/org/opensearch/action/search/SearchRequestStats.java#L82
and on phase failure:
https://github.com/opensearch-project/OpenSearch/blob/e68838819710d7040cf2b591590285f1b86f0da0/server/src/main/java/org/opensearch/action/search/SearchRequestStats.java#L89
I looked through and didn't find any cases where they might both be called.
However, I did notice that current stat is stored in an unsynchronized map:
https://github.com/opensearch-project/OpenSearch/blob/e68838819710d7040cf2b591590285f1b86f0da0/server/src/main/java/org/opensearch/index/search/stats/SearchStats.java#L120
And the key is just the phase name, which are common strings:
https://github.com/opensearch-project/OpenSearch/blob/e68838819710d7040cf2b591590285f1b86f0da0/server/src/main/java/org/opensearch/action/search/SearchPhaseName.java#L19-L25
I suspect possibly incrementing a phase stat in one thread while simultaneously decrementing it in another.

onPhaseStart() and onPhaseEnd() are both wrapped in a conditional:
https://github.com/opensearch-project/OpenSearch/blob/e68838819710d7040cf2b591590285f1b86f0da0/server/src/main/java/org/opensearch/action/search/AbstractSearchAsyncAction.java#L464-L466
https://github.com/opensearch-project/OpenSearch/blob/e68838819710d7040cf2b591590285f1b86f0da0/server/src/main/java/org/opensearch/action/search/AbstractSearchAsyncAction.java#L457-L459
However, onPhaseFailure() is not:
https://github.com/opensearch-project/OpenSearch/blob/e68838819710d7040cf2b591590285f1b86f0da0/server/src/main/java/org/opensearch/action/search/AbstractSearchAsyncAction.java#L487-L491
This means that failed phases that do not have a lifecycle can decrement a counter that was never incremented.


Add Bitbucket repository support.

Are you sure?
have a look: https://confluence.atlassian.com/bitbucket/use-the-bitbucket-cloud-rest-apis-222724129.html


Update MusicHandler.js

i'll merge this to a branch for some adjustments
btw, are the PRs have been tested?

i'll merge these to a branch for some adjustments
btw, are the PRs have been tested?

Yes I have tested it works fine .you even just have a look and check it

i'll merge these to a branch for some adjustments
btw, are the PRs have been tested?

Yes I have tested it works fine .you even just have a look and check it

thanks. i was about to implement it myself, but im too lazy tbh