[Failing Test]: postgres-service-for-local-dev action configuration issue

What happened?
Example run: https://github.com/apache/beam/actions/runs/6380840947/job/17318812300
Log:
service/postgres-for-dev created
replicationcontroller/postgres created
error: unrecognized condition: "jsonpath=***.status.loadBalancer.ingress[0].ip***"
Error: Process completed with exit code 1.

Issue Failure
Failure: Test is continually failing
Issue Priority
Priority: 2 (backlog / disabled test but we think the product is healthy)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

CC: @volatilemolotov

Similar issue for hdfs related performance tests:
https://github.com/apache/beam/actions/runs/6376132302/job/17320192049
Run kubectl apply -f /runner/_work/beam/beam/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml

service/hadoop created
service/hadoop-datanodes created
statefulset.apps/datanode created
pod/namenode-0 created
error: unrecognized condition: "jsonpath=***.status.loadBalancer.ingress[0].ip***"
Error: Process completed with exit code 1.

Similar issue for hdfs related performance tests:
https://github.com/apache/beam/actions/runs/6376132302/job/17320192049
Run kubectl apply -f /runner/_work/beam/beam/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml

service/hadoop created
service/hadoop-datanodes created
statefulset.apps/datanode created
pod/namenode-0 created
error: unrecognized condition: "jsonpath=***.status.loadBalancer.ingress[0].ip***"
Error: Process completed with exit code 1.


Adding a missing isosceles case

pointed out by @plinto in http://exercism.io/submissions/1a614864d2dc44bdb18e1be9f5f6970e

Thank you for the new test case, but I'm sorry, I think that test case is already covered. In fact, I realized we already have two similar cases (third with your proposed one).
You propose: new Triangle(2, 2, 3) should be isosceles.
On line 30 in your commit, we already have: new Triangle(4, 4, 3) expecting it to be isosceles.
On line 35, we also have: new Triangle(10, 10, 2) expecting it to be isosceles as well.
I'm probably missing something, but they look to me identical. @matthewmorgan has been involved in previous modifications of this exercise. Do you have some comments?

I believe the existing tests are incomplete because they include only acute isosceles triangles and no obtuse isosceles triangles. If the tests were more complete, I would have noticed the bug in my submission.

@mikegee I see what you're saying.  There's no overall spec for Triangle in x-common, but the neither Ruby and Java track have tests for obtuse isosceles triangles.
I don't have any objection, @rchavarria ?  Maybe it would make sense to add this test to those tracks too.

In my opinion, the 3 triangles...:

new Triangle(2, 2, 3)
new Triangle(4, 4, 3)
new Triangle(10, 10, 2)

...test the same idea. An isosceles triangle must have two sides equal, and a different third side. The exercise says nothing about acute or obstuse isosceles triangles. For me it doesn't matter if the triangle is acute or obstuse, it's just isosceles. And tests already cover the three cases for an isosceles triangle: [3, 4, 4], [4, 3, 4] and [4, 4, 3].
Sorry, but I don't see the value of adding a new test case that doesn't test anything different. If tests checked for acute/obstuse triangles, I would totally agree. But it's not the case, sorry.

@rchavarria So you feel it's OK if an iteration that doesn't check for obtuse isosceles can still pass the tests?
I see both sides-- @mikegee 's solution was incomplete because it didn't test all three sides, but the tests also allow it to pass.

Whoops! My fault, sorry. I knew I was missing something.
Of course @matthewmorgan I'm not ok with a wrong solution. I didn't see that sides variable in @plinto's solution was sorted. I tested @plinto's code and it fails with @mikegee's test.
My apologies @mikegee, I didn't understand completely the issue :cry:
I'm looking forward to see this PR merged. And, I'm ok with creating a issue in x-common to discuss with other track maintainers if they think it would be necessary to add this spec to the exercise.
Sorry @mikegee for not taking the time to fully understand you, thanks @matthewmorgan for opening my eyes.

Thanks @mikegee @rchavarria !


Initial delivery of Markdown for DB2 on Cloud Getting Started Documention in Bluemix

@spackows you might want to test the service doc on staging first before enabling it in production.
Here is the staging doc repo  https://github.com/IBM-Bluemix/docs-staging/tree/master/services


Updates docstring of MZgate

Add useful meaningful info for MZgate in its docstring.

You can see the rendered version of the maths in this link:
https://strawberryfields--624.org.readthedocs.build/en/624/code/api/strawberryfields.ops.MZgate.html?highlight=mzgate#strawberryfields.ops.MZgate

Thanks @tymomboe !


2-step zip import

Second try for 2-step zip import (#1950)
Should work well, but would be nice if someone can double test (@christianguenter2 seems like you were interested in the feature last time? :)) ).
Notes:

No changes were done to: jump_display_transport - it's se80 auto reordering :) Also just  colspan fixes in render_parent_dir
completely removed deserialize from zip. So no new actions compared to previous approach in #1953 (which is good)
renamed methods of ZCL_ABAPGIT_REPO_CONTENT_LIST for sematics reasons

merge conflicts...

Conflict after 'Open in master language'. Did a rebase onto latest master

"Import zip": hmmm. I added "zip" to emphasize that all these operations (Pull/Import/Export) are related to external files and not to remote repo. I think this is clearer. But indeed a bit bulky. Technically "Attach zip" will be more correct. But probably less understandable. Or we can arrows instead of works (+ some hint). Kind of:

or (icon based)

... or leave as is for the time being

just leave as is for now 😄


drop Python 3.7, and PyPy 3.7 and 3.8

Change Summary
Time to clean up some old technologies!
I've also pushed a version bump to 2.15.0 in this PR, so that pydantic main branch can start building against a newer core.
Related issue number
N/A
Checklist

[ ] Unit tests for the changes exist
[ ] Documentation reflects the changes where applicable
[ ] Pydantic tests pass with this pydantic-core (except for expected changes)
[ ] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

please review


Webcam works in the editor, but not in an exported project on macOS

Godot version
3.3.3 stable
System information
Mac OSX 10.15.7, GLES3, Intel Iris Graphics 6100
Issue description
I create a simplest project to test webcam realtime display, it could work well in Editor environment. If I run the project with "Play" button in the Editor, the webcam frames could display as the following screenshot (with a fps label on the upper left corner).

But when I export the project as a runnable MacOSX app, it could not work as expected. The screenshot is as follows. And every time execute the app the display will change, sometimes a blank screen only with default background, sometimes a full blink screen, sometimes a blink region above the background. In the mean while, the fps label could display correctly indict the webcam itself works well.

Steps to reproduce
Almost there is no special code I could paste here, I attached this simplest project for anybody to evaluate and reproduction.
In addition (1),  when I "play" the project in Godot Editor, sometimes (one in five or six) the Sprite, which stand for the CameraTexture，will disappear in the program.  I doubt it may be caused by a tiny renderer bug. As it is only a test in Editor, I think some random abnormal is acceptable. But  when the project exported to a runnable App, I think the abnormal effect (even sometimes abnormal effect) could not be acceptable.
In addition (2),   this is only a test project, in my real world project, I will have the WebCam frames to be re-processed as textures. As I know there is a demo CameraServer project which could set the 3D spacial environment background as the WebCam input in an easy way to set the CameraFeed ID, but in such situation I could not reuse the CameraTexture again, so I want to attach the CameraTexture to a Sprite for convenience.
The WebCam function is very important in my real world project, so I hope anybody could help my on this issue. Thank you very much!!!!!!!!!!!
Minimal reproduction project
The simplest project archive attached here, only 12KB.
The described situation could be reproduction either on Godot version 3.3.2 stable and the latest 3.3.3, on MacOSX 10.15.7 macbook pro machine.
TestWebCam.zip
Sincerely thanks for any help !

Did you enable camera access in the macOS export preset before exporting the project? You need specify an entitlement for this, which is a string that describes why you need to access the camera in your project. This string will be displayed to the user when the permission is requested.

There are two possible issues (it's probably both):

Camera texture have a bug (on all platforms, relevant for 3.x and 4.x) and won't activate if it's initialized in the starting scene - try resizing the sprite from the GDScript or resizing window to test. I have no idea how to fix it.
Make sure code signing is enabled during the export (require installed Xcode, you do not need Apple account, if you do not have one leave "identity" empty), valid application "identifier" is set, "camera" entitlement enabled and "camera usage description" is filled with text.

There's also a third issue with the camera, it will not initialize correctly on the first start (when user confirmation dialog is shown) - this should be fixed by #51964

Thank you @bruvzg !  Your first suggestion did resolve the problem. The workaround works on my project.
Thank you very much!


Update to Serilog v4, remove reference to Serilog.Sinks.PeriodicBatching

From the Serilog.Sinks.PeriodicBatching readme:

Serilog 4.x and later versions support batching natively. New projects should use Serilog's IBatchedLogEventSink and WriteTo.Sink(IBatchedLogEventSink), not this package which is now only maintained for compatibility reasons.

Therefore I suggest to release a new major version of this sink which directly uses IBatchedLogEventSink from Serilog and doesn't have a reference to Serilog.Sinks.PeriodicBatching any more.

The PR was merged. Thank you very much for the contribution @cancakar35!
@cremor, @cancakar35 and everyone interested: now the dev branch was updated with the contribution a dev packages was built and published on nuget.org. Could you please give this version a spin and let me know if it works fine or you see any new issues with it?
https://www.nuget.org/packages/Serilog.Sinks.MSSqlServer/7.0.0-dev-00087
If the feedback on the dev package is good, I will merge dev into main and produce a latest release 7.0.0.

I've ran a quick test in my solution and everything seems to work fine. Thanks!

@cremor Thanks for testing with the prerelease.

This seems to have crept back in on 7.0.1, getting errors about missing PeriodicBatching on a deployed application that's been updated to this version.

@elylv See https://github.com/serilog-mssql/serilog-sinks-mssqlserver/issues/566#issuecomment-2356973725

Hi @elylv , https://github.com/serilog-mssql/serilog-sinks-mssqlserver/issues/564#issuecomment-2351559452 this should help. Also if your project targets .net framework, please check your assembly bindings in web.config/app.config file.


Connect with National Library of Scotland’s DDS (Digital Deposit Scotland) interface

This appears to be focusing particularly on Legal Deposits: "For electronic publications, our new web-based deposit tool, Digital Deposit Scotland, makes it quick and easy for Scottish publishers to send us publications for our collections.  Electronic titles sent to the National Library of Scotland in this way are distributed to the other Legal Deposit Libraries via our shared infrastructure, benefitting all parts of the UK. " (source)

Team WP7 met with NLS to discuss Digital Deposit Scotland - DDS is a file drop web interface offered for small Scottish publishers to deposit PDFs; NLS currently do not take any kind of metadata, records are created from PDF in-house by NLS cataloguers/archivists.


v1.0 is removed from npm?

We did not expect people to start using v1 in production when it was still a release candidate. We'll find a way to prevent this in the future.
The result is the channels are heavy with questions.
We've decided to return npm to v0.7 to stop more people going down the v1 route by default.
v1 is stable. ekryski is working on freeing up time to finish permissions and docs. Keep using v1 if you're happy with it. It will be returned to npm shortly.

You can still install it from npm using the version number.  It's just no longer the default, for a little while.

Wbat @eddyystop said. You can always install 1.x with
npm install feathers-authentication@pre

or
npm install<EMAIL_ADDRESS>
And any ^1.0 dependency in the package.json.

got it, thanks, guys~ lookibg forward to the final permissions and docs.

I had I question about version. It seems the code on github is different from the code from npm with the same version tag. Is it because the package delivered to npm was compiled by babel?

@jcyh0120 yep, it's pre-compiled.


Question about how to prepare the input of numbers

Hi! I am a bioinformatic student who is new to Transformer models, and my project is to predict a number according to a long sequence of numbers(a csv with 10000 columns, most of them are 0, some of them are int numbers) So it is a simple regression task.
As I learned in your README, the dim is the embedding vector's dim for words, like a word has a vector matrix to stand for it. However, my seq is not a language sentence, it is int numbers, so how can I set the dim?

example csv data:
home/usr/Protein_interaction_1.pdb,0,0,0,0,2,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,9,0,0,0,0,0,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,010,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1...

My classmate told me that I should consider each sample as a word, so that 10000 features could be the dim of the sample, because one sample only refer to one number result. And I need to input multiple samples at once, he calls it 'each time step'.
What do you think of his suggestion? I would be glad to get your suggestions~~

Hi dear developer, I wanna share more specific info about my question for you to discuss~
Each biology sample in my input csv file contains 10,000 (125 features * 80 distance) features columns, and one float number as the label. My friend suggested that every 125 features can be considered as a block representing a 'word', so I need to find a way to represent 125 features into a embedding vector? And each feature is an integer (most of which are zeros). Then, each sample is a 'sentence' with seq_lens = 80 which corresponds to a label.
In summary, I am wondering if this task suitable for Linformer, and how to set the 125 features into a 'word'.
Sorry to bother you but our biotech group doesn't have anyone who is majored in AI.Thanks!


add postinstall into DSW IPM that will setup IRIS web server prefix

Sometimes production servers can have a prefix for all the REST-API web apps.
IT's a setting for Web Gateway.
So, if the production server is example.com, then normally MDX2JSON will be at:
example.com/MDX2JSON
but with this setting it will sit at:
example.com/prefix/MDX2JSON
The command that returns the prefix:
w $system.CSP.GetConfig("CSPConfigName")
There is a config.json file in DSW root installation that this postinstall call should change in a following way:
{
"endpoints": {
"mdx2json": "prefix/MDX2JSON"
}
}

Fixed


How to use connect-history-api-fallback with fusebox?

I'm using create react app and - unless I'm on the homepage- can't refresh a page without it breaking. This issue only occurs locally and not when I deploy to production. In that case Firebase takes care of the routing.
I've read about this issue and it seems like I have to use connect-history-api middleware. My question is how I would go about using connect-history-api-fallback within my fuse.js file?
const { FuseBox, Sparky, WebIndexPlugin, ImageBase64Plugin, QuantumPlugin } = require("fuse-box");
const { src, task, watch, context, fuse } = require("fuse-box/sparky");

// I'm assuming I need these two packes to insert the middleware?
const { history } = require('connect-history-api-fallback')
const { connect } = require('connect');

context(class {
    getConfig() {
        return FuseBox.init({
            homeDir: "src",
            output: "dist/$name.js",
            target : "browser",
            hash: this.isProduction,
            useTypescriptCompiler : true,
            plugins: [
                ImageBase64Plugin(),
                WebIndexPlugin({
                    template : "src/index.html"
                }),
                this.isProduction && QuantumPlugin({
                    bakeApiIntoBundle: "app",
                    uglify: true,
                    extendServerImport: true
                })
            ]
        })
    }
    createBundle(fuse) {

       // Should I use history and connect here?
        const app = fuse.bundle("app")
        if (!this.isProduction) {
            app.watch()
            app.hmr()
        }
        app.instructions(">Index.jsx");
        return app;
    }
});

task("clean", () => src("dist").clean("dist").exec() )

task("default", ["clean"], async context => {
    const fuse = context.getConfig();
    fuse.dev();
    context.createBundle(fuse);
    await fuse.run();
});

task("dist", ["clean"], async context => {
    context.isProduction = true;
    const fuse = context.getConfig();
    fuse.dev(); // remove it later
    context.createBundle(fuse);
    await fuse.run();
});

You can find it documented here, alongside other options that will make you a happy person 👍

That is a whole lot simpler then what I had in mind. Missed that part of the docs. Thanks!


How to reuse a mqtt connection?

Hi,
first, thanks for your work. I was flashed 10 minutes ago as I comanded Siri to switch on my living room light. I'm fairly new to nodejs but as a full-time web developer I know JavaScript itself. So my problem is simply the structure I think.
Can anyone tell me how to reuse a mqtt-connection through different accessories?
I have many sensors and actuators which work over mqtt but I don't want to open an new mqtt connection for every accessory or put everything in one file.
Maybe someone of you got it working or has a good starting point to improve my understanding.
Thanks and cheers,
Patrik

Install Domoticz, let Domoticz index all your MQTT devices, use homebridge-edomoticz plugin and homebridge to shunt it all out to Siri via HAP-NodeJS (part of Homebridge)?
On Saturday, 11 June 2016, 12:44, Patrik Mayer<EMAIL_ADDRESS>wrote:

Hi,first, thanks for your work. I was flashed 10 minutes ago as I comanded Siri to switch on my living room light. I'm fairly new to nodejs but as a full-time web developer I know JavaScript itself. So my problem is simply the structure I think.Can anyone tell me how to reuse a mqtt-connection through different accessories?
I have many sensors and actuators which work over mqtt but I don't want to open an new mqtt connection for every accessory or put everything in one file.Maybe someone of you got it working or has a good starting point to improve my understanding.Thanks and cheers,
Patrik—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Hey,
thanks for that fast reply. That could be a very interessting way.
Currently I run openHAB with homegear to controll many Homematic and MAX! devices. The goal is to bring all these devices to mqtt via homegear but I'm not there yet.
I will have a look into the domoticz-plugin to get my head around how to reuse a mqtt-connection.
Cheers,
Patrik

It does if you look into the MQTT-integration branch.

perhaps this is something for you: https://github.com/hobbyquaker/homekit2mqtt


Upgrade Phase 2 for iOS 17 (more x86 prod bots)

Prod pool:
x86 - Upgrade 3 devices (6 / 12)

[x] Upgrade device on https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-7
[x] Upgrade device on https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-8
[x] Upgrade device on https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-9

Please quarantine the bot before upgrading.
Steps after upgraded to iOS 17 on iPhone devices.

Open Xcode 15 from /opt/flutter/xcode/15a240d/XCode.app
In the top bar, open Window > Device and Simulators
The window that opens should show the connected device
Accept the Trust dialog on the iPhone device
If you get the dialog on the device twice, you may need to accept it once, disconnect, reconnect, and accept again.
Wait for the yellow bar at the top of the Device and Simulators window to go away - It should take a minute or so to Copy cache symbols

Once all steps above are complete, the device can be taken out of quarantine.

@kentnguyen99 lets start with these devices. Lets complete each device before moving on to the next. Confirm with @vashworth if you need to verify the upgrade is correct.

@vashworth The following iPhone devices have been upgraded IOS 17 for mac-7, mac-8 & mac-9. Will you check these Bots before these Bots into productions. Thanks.

All devices look good and working. Thanks!


Push image to Docker Hub on releases and pushes to master

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Re: https://github.com/toniblyx/prowler/issues/476
This adds github actions for building and pushing the image to Docker Hub. One action for releases and one for latest.
To use: add the DOCKER_USERNAME and DOCKER_PASSWORD secrets to your GitHub repo settings

see issue  #476

Closing in favor of using Docker Hub builds

Hey @stone-z look at this docker pull toniblyx:2.2.0, new version with new tag as you asked. I'll do that from now on each new release. Thanks your your suggestion!

Awesome, this is a big help - thanks again for all your work here!


Bucket Pattern

A pattern that can be created with a filled bucket + pattern to ease fluid automation.

Do I understand that correctly that this pattern should take a bucket of fluid (as fluid in the system) + a bucket and produce an bucket of fluid as item?

Do I understand that correctly that this pattern should take a bucket of fluid (as fluid in the system) + a bucket and produce an bucket of fluid as item?

Yes, this pattern is an alternative to having a bucket filling machine being necessary like RS 1.
This pattern would create the fluid bucket by just extracting a regular bucket from the storage network and a fluid from the storage network.

That is brilliant, that was always a pain in RS1 and AE2 in mod packs.

with this addition will I need 100 buckets to make 100 cakes even though the cake recipe returns the bucket?

I assume it won't be necessary, as for example if you only have 1 bucket, the system will reserve that one, start crafting the first cake, but stop until more buckets are available, and that will resume once the first bucket is back to the system, which might be slow, but it is the same thing that happens with RS1 when you use an external bucket filling machine.

Will this also be able to empty buckets into the system

Would an alternative be to have it called an upgraded pattern with a allow fluids option on the crafting recipe menu which allows the fluid that is in the bucket to be used instead of the bucket as it would remove the need for extra return items of the bucket

The system could reserve the amount of buckets required to craft one item, and if such amount exists, act as if there are enough buckets to craft the whole request.
E.g: I want to craft 100x Fooses, and one Foo needs one water bucket. RS2 would reserve one bucket and act as if it had 100 water buckets.


TensorFlow pickle version too low

E:\aimv2\venv\Scripts\python.exe E:/aimv2/main.py
Preparing style...
Loading networks from abstract_photo.pkl...
Traceback (most recent call last):
File "E:\aimv2\main.py", line 7, in
L.hallucinate(file_name = 'E:\FFOutput\qby90.mp4')
File "E:\aimv2\lucidsonicdreams\main.py", line 708, in hallucinate
self.stylegan_init()
File "E:\aimv2\lucidsonicdreams\main.py", line 179, in stylegan_init
self.Gs = self.legacy.load_network_pkl(f)['G_ema'].to(device) # type: ignore
File "E:\aimv2\stylegan2\legacy.py", line 26, in load_network_pkl
G = convert_tf_generator(tf_G)
File "E:\aimv2\stylegan2\legacy.py", line 111, in convert_tf_generator
raise ValueError('TensorFlow pickle version too low')
ValueError: TensorFlow pickle version too low
Why???

I know this is late, but this is because your model is a StyleGAN1, and the convert_tf_generator cannot convert between StyleGAN1 and StyleGAN2.

I had this error when I plugged in a stylegan2 config model trained with stylegan3


singularity: 4.0.1 -> 4.0.2

Description of changes
Changelog: https://github.com/sylabs/singularity/releases/tag/v4.0.2
cc @jbedo @ShamrockLee
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg eval

Result of nixpkgs-review pr 267971 run on x86_64-linux 1

4 packages built:

singularity
singularity-overriden-nixos
udocker
udocker.dist


Upgrade to bitflags 0.7

Clean up lmdb-rs code after https://github.com/rust-lang-nursery/bitflags/issues/39 got resolved.

Thanks!


Fixed broken env target

This fixes issue #71

Hi,
Thank you for sending the pull request! It's awesome to see you contributing to this project. However, before I can look at this request you need to add DCO sign-off to commits in this repository.
We require DCO sign-off for all commits to this repository (including all authors). Please see the following URL to see more information and BeeWare also have a Beginners guide to DCOs.
You can use git commit --amend --signoff and then a git push --force to add the signoff. You can also use git rebase --signoff.
Thanks!

We really should have a CI step which runs make env and the developer setup...

@acomodi - Can you approve and merge?


Add incomplete bundles to item lookup

When looking up an item, show incomplete community bundles which need it. This should only be shown if the player hasn't demolished the community centre with Joja.
Discussed on the forums (see 2016-10-14 and 2016-10-17).

Done in the upcoming 1.6 release via 5992aca24dcf5fada715c6421976eb326212e3e0:


Ability to define the border gradient (for the 'bevel' effect)

Would be lovely to have this feature, now the border is one-coloured. The gradient will create nice bevel effect.

Hi, any updates here?


Not compiles wasm target on Windows

error: failed to run custom build command for 'zstd-sys v2.0.1+zstd.1.5.2'

Caused by:
  process didn't exit successfully: 'F:\work\web-cycling\target\release\build\zstd-sys-a05007c57adc943b\build-script-build' (exit code: 1)
  --- stdout
  cargo:rustc-cfg=feature="std"
  cargo:rerun-if-changed=wasm-shim/stdlib.h
  cargo:rerun-if-changed=wasm-shim/string.h
  TARGET = Some("wasm32-unknown-unknown")
  HOST = Some("x86_64-pc-windows-msvc")
  CC_wasm32-unknown-unknown = None
  CC_wasm32_unknown_unknown = None
  TARGET_CC = None
  CC = None
  CFLAGS_wasm32-unknown-unknown = None
  CFLAGS_wasm32_unknown_unknown = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  running: "**clang**" "-O3" "-ffunction-sections" "-fdata-sec.....

should use cl not clang for this host.

The compiler is chosen by the cc-rs crate.
May be related to https://github.com/rust-lang/cc-rs/issues/681
TLDR: You might need to install clang and ar.

@Fedoresko if you use Visual Studio Tools (2022) with cmake component installed, you can run Developer Command Prompt for VS 2022  you just need to set ar=llvm-ar and it will work:

@gyscos is it expected that wasmtime cannot run the example zstd-safe/zstd-sys/examples/it_work.rs on commit 83d21974bf1d91431f63f57a82ad6ff28dd57e2d
root@DESKTOP-FONP7QD:/mnt/c/repos/github/gyscos/zstd-rs/zstd-safe/zstd-sys# git log -n 1
commit 83d21974bf1d91431f63f57a82ad6ff28dd57e2d (HEAD -> main, origin/main, origin/HEAD)
Author: Chris Burgess<EMAIL_ADDRESS>Date:   Sat Feb 25 15:30:11 2023 -0500

    Fix OutBuf around_pos invariant (#215)
root@DESKTOP-FONP7QD:/mnt/c/repos/github/gyscos/zstd-rs/zstd-safe/zstd-sys# clang --version
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
root@DESKTOP-FONP7QD:/mnt/c/repos/github/gyscos/zstd-rs/zstd-safe/zstd-sys# wasmtime --version
wasmtime-cli 6.0.1
root@DESKTOP-FONP7QD:/mnt/c/repos/github/gyscos/zstd-rs/zstd-safe/zstd-sys# cargo build --target wasm32-unknown-unknown --no-default-features --example it_work
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
root@DESKTOP-FONP7QD:/mnt/c/repos/github/gyscos/zstd-rs/zstd-safe/zstd-sys# ls -la target/wasm32-unknown-unknown/debug/examples/
total 19328
drwxrwxrwx 1 root root    4096 Mar 12 16:44 .
drwxrwxrwx 1 root root    4096 Mar 12 11:50 ..
-rwxrwxrwx 1 root root     390 Mar 12 16:27 it_work-0e1a13e5f0f45118.d
-rwxrwxrwx 2 root root 6606331 Mar 12 16:27 it_work-0e1a13e5f0f45118.wasm
-rwxrwxrwx 1 root root     382 Mar 12 16:40 it_work-baeeb6a126d4dc5b.d
-rwxrwxrwx 1 root root 6577359 Mar 12 16:40 it_work-baeeb6a126d4dc5b.wasm
-rwxrwxrwx 1 root root     676 Mar 12 16:44 it_work.d
-rwxrwxrwx 2 root root 6606331 Mar 12 16:27 it_work.wasm
root@DESKTOP-FONP7QD:/mnt/c/repos/github/gyscos/zstd-rs/zstd-safe/zstd-sys# wasmtime target/wasm32-unknown-unknown/debug/examples/it_work.wasm  --invoke test_compress
Error: failed to run main module `target/wasm32-unknown-unknown/debug/examples/it_work.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-unknown-unknown/debug/examples/it_work.wasm"
    1: unknown import: `env::malloc` has not been defined
root@DESKTOP-FONP7QD:/mnt/c/repos/github/gyscos/zstd-rs/zstd-safe/zstd-sys#

I went back to git history and found that works 417d4420e3e7a8c406b8641e731caa427d98398d


Validation group not applied correctly when validating a CollectionInputFilter

While debugging a problem when validating a complex form containing collections and validation groups to limit the fields that are being validated I notice the following code in CollectionInputFilter:
        foreach ($this->data as $key => $data) {
            // ....
            if (null !== $this->validationGroup) {
                $inputFilter->setValidationGroup($this->validationGroup[$key]);
            }

If the CollectionInputFilter has a validation group then the validation group is indexed using the same key as the data. A validation group is assigned like this:
$collection = new CollectionInputFilter(...);
$collection->setValidationGroup(array(
    'a', 'b', 'c'
));

Now, for the first element in the list only property a is validated, property b for the second, etc.
Am I missing something in the way I assign my validation group? Or should the line where the validation group is assigned read:
                $inputFilter->setValidationGroup($this->validationGroup);

I can work around this by calling $collection->getInputFilter()->setValidationGroup(...) instead, however this prevents me of performing a single setValidationGroup call at the root of my form with a single nested array for the entire form.

Yep really strange, but can be used like this:
$inputFilter = new InputFilter();
$inputFilter->add([
    'name' => 'foo',
    'required' => true,
]);
$inputFilter->add([
    'name' => 'bar',
    'required' => true,
]);

$collection = new CollectionInputFilter();
$collection->setInputFilter($inputFilter);
$collection->setValidationGroup([0 => ['bar'], 1 => ['foo']]);

$collection->setData([
    0 => ['bar' => 'value'], // valid because only 'bar' in validation group
    1 => ['foo' => 'value2'] // valid because only 'foo' in validation group
]);
$collection->isValid();

Okay I am not sure this is desired behavior nor use case for this, but well can't think otherwise why setValidationGroup() is called in every foreach loop. And also there is a test for this so can't be fixed?
also data being array of 3 elements will throw Undefined offset: 2
@weierophinney what do you think? If this is a bug I can create PR with a fix.

This repository has been closed and moved to laminas/laminas-inputfilter; a new issue has been opened at https://github.com/laminas/laminas-inputfilter/issues/3.


Home assistant to be able display in a iframe

Description
locall http not showing as well as Nabu Casa Account
<IP_ADDRESS> refused to connect
Thank you
Priority
Low (Nice-to-have)

Hello, we are unable to help you without further information. The requirements for the iframe widget are quite clearly documented here:
https://homarr.dev/docs/widgets/iframe/
Please read through the page and try out if it resolves your problem.

Many thanks but I do have iframes that work but my local HASS install on a local ip will not display for some reason other IP do work just fine


higher level API

Currently, the API exposed by shell_completion is very low level. Many goals of this crate will require the design and implementation of a higher level API.
This is a request for proposals for the shape of the public API of shell_completion. All ideas are welcome, and they don't have to be fully baked. Even something as simple as "I wish there was a method that does X" is useful feedback.

Hackernews user chubot suggested reviewing the [shellac protocol proposal] for API inspiration.
$ git --git-dir . a<TAB>

{ "SHELLAC_ARGV": ["git", "--git-dir", ".", "a"]
  "SHELLAC_ARGV_INDEX": 3,
  "SHELLAC_CHAR_INDEX": 1,
}

I'm going to move forward with a trait modeled after this protocol proposal.

Closing as #6 takes care of this in the short term. Feel free to create a new issue to suggest any changes or additions.


' characters in item name cause errors when setting return date

Will need to figure out how to fix this

Everything has been sanitized and shouldn't be a problem now


[23665] - Store & Show Challenge using the NotificationService extension.

Ticket

[23665]
[23687]

Description

Store & Show Challenge using the NotificationService extension.

Commit message

Screenshot
https://user-images.githubusercontent.com/4898461/151061482-5ea1b39a-1523-4aa4-b74d-101e1c3c20da.MP4
The video showcase the following scenarios:

The store of a challenge using the NotificationService extension & showing a challenged pre-processed & stored in the app. During background.
The store of a challenge using the NotificationService extension & showing a challenged pre-processed & stored in the app. During foreground.
The current management of multiple push notifications for the same challenge, the first time it will load using the cache, but after it will require the fetch, since we are deleting that information once we display the information for first time.

Testing

[ ] Added unit tests
[ ] Ran unit tests successfully
[ ] Added documentation for public APIs and/or Wiki

Code coverage
Total coverage: 94.68%

Powered by Slather

  Generated by :no_entry_sign: Danger

Code coverage
Total coverage: 94.62%

File
Coverage

TwilioVerifySDK/TwilioVerify/Sources/Data/Storage.swift
95.56%

Powered by Slather

  Generated by :no_entry_sign: Danger


chore(main): release 0.3.1

:robot: I have created a release beep boop
0.3.1 (2023-04-30)
Bug Fixes

set correct default values for DeletionPolicy, when importing resources (#111) (c8be9a4)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/udondan/cfn-teleport/releases/tag/v0.3.1 :sunflower:


Postinstall fail on windows

Its seems that there is some problem when resolving the path while creating the generated file.
Sharing below a SS from the TG group facing an issue :

Its seems its resolving c:\c:\ instead of c:\

We should have a proper way to test this. VM on linux/macOS? XD
@ZakGriffith are you still on windows? If so, could you try?

Also, maybe we should just un-gitignore this file? @sverps mentioned at some point.

If you connect your GH repo to vercel, and want to interact with a live contract with your dapp....you need to un-gitignore and commit the file

Thoughts?

Yeah maybe that's better. We'll need to worry a lot less about build pipelines and the postinstall script etc.
When we make changes to the contract for testing stuff, we need to be mindful of not committing that one anayway, might as well keep an eye out for changes to hardhat_contracts file from being committed by accident.

Ok, let's go with it and see how it feels.
https://github.com/scaffold-eth/se-2/issues/278

i'm on windows and just needed to change this. not sure if it'll affect mac users though
const __dirname = path.dirname(new URL(import.meta.url).pathname).slice(1);

This issue will be fixed with https://github.com/scaffold-eth/se-2/issues/278

Closing this in favour of #278

Closing this in favour of #278


GIP issue when navigation is triggered from model

DEMO: https://github.com/7iomka/nextjs-effector-issue-with-router-bindings

Go to My profile page (Gip)
Click to Go to home button on bottom of any page
Instead of navigation - nothing happened

I try to make a research and I found line with issue
https://github.com/7iomka/nextjs-effector-issue-with-router-bindings/blob/master/library/fabrics/get-initial-props.ts#L64
When state.clientScope is used as a scope on navigation on client side allSettled stacks and never resolved.
The only question is - why and how to figure out?
From how I know - allSettled stacks in execution when we have an issue with logic of effects execution, for example - infinite loops.
But in this case, I don't know.
Please help me to solve this problem, bro!

Any news?(

Fixed in<EMAIL_ADDRESS>Thanks


Remove the direct dependency on rdflib

…to prevent duplication of it and solid-auth-client
Two instances of solid-auth-client in the build are bad news.

I don't think this is necessary anymore, so I'll close this issue.


[Marker.io] can we please improve the spacing in the text section of "key numbers"

Steps to reproduce:

go to key numbers section
compare with figma

Actual result: text seems congested . notice how the text is shown in figma..
Expected result: this should be improved.
https://go.screenpal.com/watch/cZQ1jFVRzMt

Reported by: vinod<EMAIL_ADDRESS>- Contact via Marker.io
Source URL: https://proav-stg.webflow.io/about
Issue details: Open in Marker.io
Device typedesktopBrowserChrome <IP_ADDRESS>Screen Size1366 x 768OSWindows 10 (10.0.0)Viewport Size1366 x 641Pixel Ratio@1x

🤖 Marker.io bot - This issue was marked as ✅ Resolved in Marker.io


Removes bullet points and the term "guide" from quickstart.md

Description
Based on some feedback from @hdhalter I changed the way extra info was presented in the numbered list for the overall processes. Previously, there were bullet points nested within the numbered steps. Now, that information is part of the bullet list and no longer nested. This improves readability greatly. It also looks a lot cleaner and less fragemented.
Issues Resolved
Relates to #789
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Just need an approval for the 1.3 backport


94 convenience function to return qualitycontrol object

This PR refactors the utilities and adds to new ones, and then uses those to support a helper function that can return a valid QC object (if it exists, and deserializes properly)

From teams discussion:

Either we don't validate it or we move this into a different repo to avoid pulling aind-data-schema as a dependency
Use aind-metadata-validator for the validation?

Maybe we can organize it like: utils/
* s3.py

* docdb.py

* quality_control.py (or aind_data_schema.py, since helpers seems similar enough to utils)

Helen pointed out that there's a bunch of downstream dependencies on this stuff, so I think I should revert it and just put my utilities into their own file

Maybe we can organize it like: utils/
* s3.py

* docdb.py

* quality_control.py (or aind_data_schema.py, since helpers seems similar enough to utils)

Helen pointed out that there's a bunch of downstream dependencies on this stuff, so I think I should revert it and just put my utilities into their own file

I reckon it would introduce a breaking change. We can re-organize things later.

Maybe util/ isn't the best name since it conflicts with utils.py. We could go back to helpers/?

Maybe util/ isn't the best name since it conflicts with utils.py. We could go back to helpers/?

helpers is probably okay

Thanks Dan. Let me know if you need any help or clarifications. Are you still intending to rename the folder to utils/helpers?
Side note for next time- Contributor Guidelines contains branch naming conventions. It's the same across the core services. You can also link the issue in the PR message by writing "closes/fixes #{issue_num}"

I use Github to open branches directly linked to issues, this also auto-links the issue in the PR without needing to write the closes statement. If the contributor guidelines don't support that then we should update the guidelines because that workflow is much better than manually writing this stuff out.
I'll start a thread in the sci comp channel about this.


Updated to v2.2.1

Hi @winsento !
Thanks for your PR. I merged into master. We have automated the builds in DockerHub; now this new version is building: https://hub.docker.com/r/geographica/gdal2/builds/bccfuwmsfnmqe44kyqpanrn/

Sorry, hand trembled. I did not expect that the docker hub would not be able to untar the archive.
Small fix here https://github.com/GeographicaGS/Docker-GDAL2/pull/2

Ok @winsento , I answer you in #2


NPE com.google.cloud.bigquery.BigQueryException.getError()

java.lang.NullPointerException: Cannot invoke "com.google.cloud.bigquery.BigQueryError.getMessage()" because the return value of "com.google.cloud.bigquery.BigQueryException.getError()" is null
at no.nrk.bigquery.testing.BQSmokeTest.bqCheckFragmentTestFailing$$anonfun$1$$anonfun$1$$anonfun$1$$anonfun$1(BQSmokeTest.scala:179)

Not seen since this was created, maybe this was fixed in upstream. Reopen if it occurs again


Refactored encoder

Encoder v2 is the re-written encoder (specaugment on non-input stacking).
Added support for WPM, specaugment, CNN, bidi LSTMs - refactored librispeech_ctc.py with different classes. Unit tests modified accordingly.
New baseline is the 0.8DO, 6x1024, specaugment.
Added Lingvo's conformer model (TF2.3) but TPUs seem to have issues with TF2.3.

Added Lingvo's conformer model (TF2.3) but TPUs seem to have issues with TF2.3.

Some thoughts:

We will need to update to the latest version of lingvo to use TF2.3.
You may have specified --tf-version 2.2 in the "ctpu up" command on accident.

Linted and formatted. I've addressed and resolved most of the comments. Lets get this merged! I can make the further changes in a separate PR.

Please go ahead and merge it!


图例筛选项配置

[ ] I have searched the issues of this repository and believe that this is not a duplicate.

What problem does this feature solve?
使用图例筛选图表时至少保留一个筛选项，当前可以取消所有图例的选中，图表内容为空
What does the proposed API look like?
使用图例筛选图表时至少保留一个筛选项，期望完善相关文档或者支持配置

ref https://github.com/antvis/G2/issues/2899

ref https://github.com/antvis/G2/issues/2899


review findings - nil fix

With this fix we handle all cases of nil the same way
we dropped the nil check in getTopologyManagedFields as we never call it with nil (see the switch on the call site)

not required anymore already cherry-picked


Some variable names in libtorch's logging and glog are conflicting.

version
Pytorch: 1.13.1
Cuda: 11.7
CMAKE: 3.16.3
make -j failed:
I have seen a lot of similar bugs in pytorch issues and there is no good solution. It seems to be due to a link conflict between torch and glog.Have you encountered similar problems?
/usr/local/miniconda3/envs/gsampler/lib/python3.9/site-packages/torch/include/c10/util/logging_is_not_google_glog.h:133: note: this is the location of the previous definition
133 | #define DLOG(n)
|
In file included from /gSampler/src/cuda/batch/../../logging.h:29,
from /gSampler/src/cuda/batch/../utils.h:6,
from /gSampler/src/cuda/batch/list_sampling_probs.cu:5:
/gSampler/build/third_party/glog/glog/logging.h:1193: warning: "DLOG" redefined
1193 | #define DLOG(severity)  \

Hello, thank you for the bug report.
The issue stems from both glog and torch defining some identical macros (such as DLOG, LOG, and others). One straightforward solution is to undefine one before including the other. This is precisely what logging.h accomplishes.
It appears that PyTorch seems introduce a new macro DLOG, which causes compilation to fail. I've created a new branch fix_dlog to address this issue. Feel free to try it out. It has successfully resolved the problem for me using PyTorch version 2.2 and CUDA version 12.3.


feat(northlight): datePicker-calendar fiscalStartDay prop

Adds a new prop fiscalStartDay as prop to all the calendars/date pickers, which solves the issue of having no exact starting day from the fiscal year settings  when creating a new plan
closes: DEV-8742

https://github.com/mediatool/northlight/assets/521837/56aa30b8-4e71-4122-bb11-c54fffef5323
There is something wrong with the end date calculation, it should end at the 10th in this example but ends a few days earlier.

Well done, you gained a follower!

Well done, you gained a follower!

Damn!! Finally, my very own 5 minutes of fame. Thank you for the follow. f4f.


[BUG] heads is not an argument in Vim

Hi,
I'm simply trying to run the 'Usage' code, but this outputs a traceback:
Traceback (most recent call last):
File "/home/fnowak/Mamba Reproduction/mamba.py", line 8, in
model = Vim(
^^^^
File "/home/fnowak/Mamba Reproduction/model.py", line 228, in init
VisionEncoderMambaBlock(
TypeError: VisionEncoderMambaBlock.init() got an unexpected keyword argument 'heads'
If I remove heads as an input parameter it runs, but I'm not sure how that influences the model initialisation.
Best,
Franek

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.

It seems to run fine with 'heads' commented out
# Model
model = Vim(
    dim=256,  # Dimension of the transformer model
    #heads=8,  # Number of attention heads
    dt_rank=32,  # Rank of the dynamic routing matrix
    dim_inner=256,  # Inner dimension of the transformer model
    d_state=256,  # Dimension of the state vector
    num_classes=1000,  # Number of output classes
    image_size=224,  # Size of the input image
    patch_size=16,  # Size of each image patch
    channels=3,  # Number of input channels
    dropout=0.1,  # Dropout rate
    depth=12,  # Depth of the transformer model
)


AzureKeyVault task not passing schema validation

Schema validation errors were encountered.
tasks/azdo-pat.yaml::task: 'AzureKeyVault@1' does not match '^PowerShell@2$'

parameters:
  KEY_VAULT_NAME: ""
  SERVICE_CONNECTION: ""

steps:
  - task: AzureKeyVault@1
    inputs:
      azureSubscription: ${{ parameters.SERVICE_CONNECTION }}
      keyVaultName: ${{ parameters.KEY_VAULT_NAME }}
      secretsFilter: "azdo-pat-token"
    displayName: "Get AZDO PAT"

I'm using the latest schema file, 1.191.0.
Tried with both tasks @1 and @2.

Known issue with the schema; this is being tracked at #410. Thanks for reporting, though!


Moved: PostgreSQL.PostgreSQL.16 version 16.1-1

Checklist for Pull Requests

[x] Have you signed the Contributor License Agreement?
[ ] Is there a linked Issue?

Manifests

[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[x] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Possible duplicate package entry.
Similar installer SHA256 hash found in manifest:

InstallerSha256: 16eaa194fcb94d6aa856a14080892233058c6725744324fcbf436aea6585e931

manifests\p\PostgreSQL\PostgreSQL\16.1

Verify by hash.


Tokotron: Tokenized TTS for the SpeechBrain benchmark (single speaker)

@flexthink  any update on this PR?

Thanks @flexthink, everything looks really neat .. I  did the review... I have fetched the latest update from the DASB main branch... These are the few comments that I have before merging the PR:

There is some recommit error mostly related to unused lib.
The utils and Custim_model should be moved to TTS folder, especially custom_model should be renamed or avoid the link to the main custim_model because the wrappers ae not used for other tasks and it could be confusing,.. we could add your changes later to all tasks, but for now let's keep it only for TTS.
I have added the run_generative_benchmark.sh for SE and SS, please complete it for TTS and make sure that it works.

Thanks @flexthink  for this PR... everything works now..It merged


Writing resource data to sqlite DB raises "sqlite3.OperationalError: too many SQL variables"

Hi,
writing sufficiently "wide" (i.e. number of fields/colums) and "long" (as in number of rows) table data easily runs into "sqlite3.OperationalError: too many SQL variables" for SQLite.

I.e. this uses separate parameters for each field, for each row.
According to https://www.sqlite.org/limits.html the max. number of params in an SQL statement is "[...] SQLITE_MAX_VARIABLE_NUMBER, which defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0. [...]".
I'm running on 3.22.0 so that's why I've immediately stumbled into the limit. While this has obviously been lifted somewhat in newer versions I still think it's not a viable approach, as you'll just run into the problems again for bigger (more rows) tables.
It does look like there's some code that addresses this to a degree (the buffer and buffer_size parts) but IMHO this is not sufficient (think tables with 50 colums and 800 rows = 40000 params > 32766, again).
I've taken a look at SQLAlchemy possibilities and it seems to me one should rather use
self.__connection.execute(sql_table.insert(), buffer)

instead of
self.__connection.execute(sql_table.insert().values(buffer))

(see https://docs.sqlalchemy.org/en/14/tutorial/dbapi_transactions.html#tutorial-multiple-parameters) and make SQLAlchemy apply cursor.executemany.

I'm aware this might have performance implications - depending on if the SQL engine implements executemany efficiently. Haven't measured.
I'm not convinced the "params for each field for each row" is faster in the first place; but if it is one could also use the  executemany approach as a fallback, in case of running into the exception(?).
So a naive fix to not run into the error would be
$ diff frictionless/plugins/sql//storage.py.BAK frictionless/plugins/sql//storage.py
333c333
<                     self.__connection.execute(sql_table.insert().values(buffer))
---
>                     self.__connection.execute(sql_table.insert(), buffer)
336c336
<                 self.__connection.execute(sql_table.insert().values(buffer))
---
>                 self.__connection.execute(sql_table.insert(), buffer)

(no changes to the buffer and buffer_size parts here since I didn't really understand their full intention ;-)).
If helpful I could come up with a PR.
Again, awesome library, best regards,
Holger

Thanks @hjoukl!
@shashigharti can you please take a look?

Thank you @hjouki!
SQLITE_MAX_VARIABLE_NUMBER is the max number of fields/columns not (rows x columns) and max limit is 32766 for SQLite version > 3.32.0.
I am using version 3.38.5 and tested with 191(columns) X 1000 (rows)  and it worked fine. I used this excel sheet https://docs.google.com/spreadsheets/d/12VbraN-O_gBpP246wTWdaK4BkvLYNIUm1LkXSRQzbaE/edit#gid=566358228
And I think 32766(column number limit) is more than enough.
Regarding using
self.__connection.execute(sql_table.insert(), buffer)
instead of
self.__connection.execute(sql_table.insert().values(buffer))
I tested as shown in the screenshot and I think both use prepared statements. So it would be better to keep everything as it is for now because 32766 fields (limit) should be enough. What do you think?

Hi @shashigharti,
thanks for getting back!

SQLITE_MAX_VARIABLE_NUMBER is the max number of fields/columns not (rows x columns) and max limit is 32766 for SQLite version > 3.32.0.

I kindly disagree ;-)
According to  https://www.sqlite.org/limits.html: "A host parameter is a place-holder in an SQL statement that is filled in using one of the sqlite3_bind_XXXX() interfaces."
So as you yourself show the number of parameters applied in a statement is indeed rows x columns in the  self.__connection.execute(sql_table.insert().values(buffer)) case:
>>> from sqlalchemy import table, column
>>> t = table('foo', column('x'), column('y'))
>>> stmt1 = t.insert().values([(3, 'd'), (4, 'e')])
>>> print(stmt1)  # 2 columns, 2 rows => 2*2 = 4 parameters
INSERT INTO foo (x, y) VALUES (:x_m0, :y_m0), (:x_m1, :y_m1)
>>> t = table('foo', column('x'), column('y'), column('z'))
>>> stmt1 = t.insert().values([(3, 'd'), (4, 'e'), (5, 'f')])
>>> print(stmt1)  # 3 columns, 2 rows => 3*2 = 6 parameters
INSERT INTO foo (x, y) VALUES (:x_m0, :y_m0), (:x_m1, :y_m1), (:x_m2, :y_m2)

So IMHO this doesn't scale, per definition - unless you have other mechanisms in place to avoid creating a prepared statement with an immense amount of named/numbered parameters.
As I mentioned I didn't quite grasp the buffer and buffer_size parts of the code, maybe they keep the number of params under the 32766 limit? Will take a deeper look.
Just from looking at it I'd say you still run into the restriction with e.g. 33 columns and 1001 rows since 33 * 1001 > 32766. Have to check...

I am using version 3.38.5 and tested with 191(columns) X 1000 (rows) and it worked fine. I used this excel sheet https://docs.google.com/spreadsheets/d/12VbraN-O_gBpP246wTWdaK4BkvLYNIUm1LkXSRQzbaE/edit#gid=566358228 And I think 32766(column number limit) is more than enough.

I have yet to test with your sample spreadsheet. But looking at it, are you sure it actually applies 191 columns, since the column name "field12" is duplicated in columns L-Gi?
Unfortunately several "enterprise" systems ship with sqlite < 3.32.0 and thus have a drastically lower SQLITE_MAX_VARIABLE_NUMBERof 999:

RHEL 7.9: sqlite3 3.7.17
RHEL 8.5: sqlite3 3.26.0

So this may be a problem in an enterprise context where you might not be easily able to install newer versions. So frictionless-py might not be practically usable there unless you're able to upgrade sqlite.

Regarding using self.__connection.execute(sql_table.insert(), buffer)
instead of self.__connection.execute(sql_table.insert().values(buffer))
I tested as shown in the screenshot and I think both use prepared statements. So it would be better to keep everything as it is for now because 32766 fields (limit) should be enough. What do you think?

Both use prepared statements, but the currently implemented variant prepares statements where the number of used named/numbered parameters depend on both the number of columns and the number of rows.
I wonder how other databases handle a huge number of named/number parameters. Personally, I find it a bit surprising by SQLAlchemy to "create a dedicated variable for every single input value". It feels a bit like having a function f(x, y) and instead of invoking this with several (x, y) input tuples [(1,'a'), (2, 'b'), (3, 'c')] like f(1, 'a'), f(2, 'b'), f(3, 'c') you're instead rewriting to a function f_mod(x1, y1, x2, y2, x3, z3) and calling f_mod(1,'a', 2, 'b', 3, 'c').
But I take it there might be valid reasons for doing that with .insert().values(...) (e.g. execeutemany is not performant).

Hi @shashigharti ,

So IMHO this doesn't scale, per definition - unless you have other mechanisms in place to avoid creating a prepared statement with an immense amount of named/numbered parameters.
As I mentioned I didn't quite grasp the buffer and buffer_size parts of the code, maybe they keep the number of params under the 32766 limit? Will take a deeper look.
Just from looking at it I'd say you still run into the restriction with e.g. 33 columns and 1001 rows since 33 * 1001 > 32766. Have to check...

Have checked now :-):
With a slightly modified test program for modifying the number for columns/rows:
#  frictionless_sqlite_params_error.py

import os

from frictionless import Resource
from frictionless.plugins.sql import SqlDialect

# sufficiently 'wide' and 'long' table data to provoke SQLite param
# restrictions
number_of_fields = int(os.environ.get('FRICTIONLESS_TEST_FIELDS', '90'))
number_of_rows = int(os.environ.get('FRICTIONLESS_TEST_ROWS', '100'))

data = '\n'.join([
    ','.join(f'header{i}' for i in range(number_of_fields)),
    '\n'.join(
        ','.join(f'row{r}_col{c}' for c in range(number_of_fields))
        for r in range(number_of_rows)
        )
    ]).encode('ascii')

with Resource(data, format='csv') as resource:
    resource.write('sqlite:///app.db', dialect=SqlDialect(table='datatable'))

To not run into the very restricted limits of the sqlite versions on RHEL7 and RHEL8 I tested on my Kubuntu 20.04 machine. FWIW this also still features an sqlite version < 3.32.0, namely 3.31.1:
holger@server:~/dev$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"
holger@server:~/dev$ sqlite3 -version
3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837balt1

However, (K)Ubuntu / Debian seem to drastically raise the (compiled) limits for their sqlite package:
holger@server:~/dev$ sqlite3 app.db 'pragma compile_options;'|grep MAX_VARIABLE
MAX_VARIABLE_NUMBER=250000

Still, one can easily provoke the error:
# <= 250.000 is fine...
holger@server:~/dev$ FRICTIONLESS_TEST_FIELDS=240 FRICTIONLESS_TEST_ROWS=1001 ./frictionless-py-venv/bin/python frictionless_sqlite_params_error.py

# > 250.000 is not
holger@server:~/dev$ FRICTIONLESS_TEST_FIELDS=250 FRICTIONLESS_TEST_ROWS=1001 ./frictionless-py-venv/bin/python frictionless_sqlite_params_error.py
raceback (most recent call last):
  File "/home/holger/dev/frictionless-py-venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/home/holger/dev/frictionless-py-venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: too many SQL variables

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "frictionless_sqlite_params_error.py", line 21, in <module>
    with Resource(data, format='csv') as resource:
...
  File "/home/holger/dev/frictionless-py-venv/lib/python3.8/site-packages/frictionless/plugins/sql/storage.py", line 333, in __write_convert_data
    self.__connection.execute(sql_table.insert().values(buffer))
...

I wouldn't say that 250 columns makes a lot of sense but I don't think it should crash, either.

I am using version 3.38.5 and tested with 191(columns) X 1000 (rows) and it worked fine. I used this excel sheet https://docs.google.com/spreadsheets/d/12VbraN-O_gBpP246wTWdaK4BkvLYNIUm1LkXSRQzbaE/edit#gid=566358228 And I think 32766(column number limit) is more than enough.

I have yet to test with your sample spreadsheet. But looking at it, are you sure it actually applies 191 columns, since the column name "field12" is duplicated in columns L-Gi?

Also, as I've had to learn you need to check the actual compiled restrictions for your sqlite installation since these could well not be the defaults. One can check with sqlite3 app.db 'pragma compile_options;'.

Regarding using self.__connection.execute(sql_table.insert(), buffer)
instead of self.__connection.execute(sql_table.insert().values(buffer))
I tested as shown in the screenshot and I think both use prepared statements. So it would be better to keep everything as it is for now because 32766 fields (limit) should be enough. What do you think?

Both use prepared statements, but the currently implemented variant prepares statements where the number of used named/numbered parameters depend on both the number of columns and the number of rows.
I wonder how other databases handle a huge number of named/number parameters. Personally, I find it a bit surprising by SQLAlchemy to "create a dedicated variable for every single input value". It feels a bit like having a function f(x, y) and instead of invoking this with several (x, y) input tuples [(1,'a'), (2, 'b'), (3, 'c')] like f(1, 'a'), f(2, 'b'), f(3, 'c') you're instead rewriting to a function f_mod(x1, y1, x2, y2, x3, z3) and calling f_mod(1,'a', 2, 'b', 3, 'c').
But I take it there might be valid reasons for doing that with .insert().values(...) (e.g. execeutemany is not performant).

What about combining the 2 variants for the prepared statements? I.e.

keep the currently implemented self.__connection.execute(sql_table.insert().values(buffer)) variant as the default and
fall back to self.__connection.execute(sql_table.insert(), buffer) in case of the exception?

Unfortunately it doesn't look like there's a programmatic sqlite3 Python interface to get at MAX_VARIABLE_NUMBER; if there was frictionless-py could dynamically optimize buffer_size. But reading this information through sqlite3 app.db 'pragma compile_options;' seems brittle.
I'm also not convinced that the "named/numbered parameters for all row values" is actually faster than running through executemany but that probably depends heavily on the database and the database library/engine used. I.e. this needs benchmarking...
(For completeness'  sake: Looks like https://github.com/rogerbinns/apsw (an alternative sqlite Python wrapper) seems to indeed feature getting at that information through it's connection.limit method)

I got curious so hacked together a rather naive benchmark (timeit-based, measurements without any precautions to keep the machine idle apart from the benchmark process).
# benchmark_sqlite_params.py

# Pretty naive timeit benchmark
# I'm using globals all over the place here - don't do this at home, kids.

import os
import timeit

from sqlalchemy import (
        Column, MetaData, String, Table, create_engine, delete, func,
        select)
from sqlalchemy.orm import Session

# some outside configurability through env vars
number_of_fields = int(os.environ.get('FRICTIONLESS_TEST_FIELDS', '90'))
number_of_rows = int(os.environ.get('FRICTIONLESS_TEST_ROWS', '100'))
timeit_number = int(os.environ.get('FRICTIONLESS_TEST_TIMEIT_NUMBER', '1000'))
buffer_size = int(os.environ.get('FRICTIONLESS_TEST_BUFFER_SIZE', '1000'))

col_pad = len(str(number_of_fields)) - 1
row_pad = len(str(number_of_rows)) - 1

columns = [Column(f'col{i:0{col_pad}}', String(col_pad + row_pad + 1)) for i in range(number_of_fields)]
rows = [
    {col.name: f'row{r:0{row_pad}}_col{c:0{col_pad}}' for (c, col) in enumerate(columns)}
    for r in range(number_of_rows)
    ]

engine = create_engine('sqlite:///app.db', echo=False)
meta = MetaData()
Mytable = Table('mytable', meta, *columns)
meta.create_all(engine)

def insert_values_params(session):
    stmt_values_params_base = Mytable.insert()
    buffer = []
    for row in rows:
        buffer.append(row)
        if len(buffer) >= buffer_size:
            stmt_values_params = stmt_values_params_base.values(buffer)
            session.execute(stmt_values_params)
            buffer = []
    if len(buffer) > 0:
        stmt_values_params = stmt_values_params_base.values(buffer)
        session.execute(stmt_values_params)
        buffer = []
    session.commit()

def insert_values_executemany(session):
    stmt_executemany = Mytable.insert()
    buffer = []
    for row in rows:
        buffer.append(row)
        if len(buffer) >= buffer_size:
            session.execute(stmt_executemany, buffer)
            buffer = []
    if len(buffer) > 0:
        session.execute(stmt_executemany, buffer)
        buffer = []
    session.commit()

def check_and_clean(session):
    count_stmt = select(func.count()).select_from(Mytable)
    result = session.execute(count_stmt)
    print(f'{count_stmt}: {result.all()}')
    last_row = session.query(Mytable).order_by(Mytable.c[0].desc()).first()
    print(f'last row/col value: {last_row[-1]}')
    session.execute(delete(Mytable))
    session.commit()

with Session(engine) as session:
    print('\n--------------------------------------------------------------------')
    print("timeit.timeit('insert_values_params(session)', "
           "number=timeit_number, globals=globals())")
    print(timeit.timeit('insert_values_params(session)',
           number=timeit_number, globals=globals()))
    check_and_clean(session)

with Session(engine) as session:
    print('\n--------------------------------------------------------------------')
    print("timeit.timeit('insert_values_executemany(session)', "
          "number=timeit_number,  globals=globals())")
    print(timeit.timeit('insert_values_executemany(session)',
          number=timeit_number, globals=globals()))
    check_and_clean(session)

In my initial tests this seems to clearly hint - for the sqlite3 engine at least - that applying executemany is (vastly) superior to "named/numbered parameters for all row values":
 rm -f app.db && FRICTIONLESS_TEST_FIELDS=50 FRICTIONLESS_TEST_ROWS=10000 FRICTIONLESS_TEST_TIMEIT_NUMBER=100 FRICTIONLESS_TEST_BUFFER_SIZE=1000 ./frictionless-py-venv/bin/python benchmark_sqlite_params.py

--------------------------------------------------------------------
timeit.timeit('insert_values_params(session)', number=timeit_number, globals=globals())
547.7838764779999
SELECT count(*) AS count_1
FROM mytable: [(1000000,)]
last row/col value: row9999_col49

--------------------------------------------------------------------
timeit.timeit('insert_values_executemany(session)', number=timeit_number,  globals=globals())
20.852090813999894
SELECT count(*) AS count_1
FROM mytable: [(1000000,)]
last row/col value: row9999_col49

(unless I've made some really stupid mistake in the benchmarking code, obviously :-))
So please take with a grain of salt - but I think this indicates that some proper benchmarking should be done, preferably with other DB backends/engines, too.
Best regards, Holge

Thank you again for the detailed explanation :) and benchmark code.
After some testing again, I agree with you that SQLITE_MAX_VARIABLE_NUMBER is not total fields. I was assuming that it was the total limit for the variables/fields in "single sql statement for one row", based on this question https://stackoverflow.com/questions/64419397/sqlite-max-variable-number-increase-or-break-sql-query-into-chunks
However, regarding your analysis about SQLITE_MAX_VARIABLE_NUMBER being rows * columns, I tested it as well and got following results. I am sharing the google collab files with sqlite <=3.32 and > 3.32 and found results that didnot match both the assumptions.
My configuration
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"
max_variable_number=250000
sqlite version < 3.32
https://colab.research.google.com/drive/17qq9i4PyqUl-_mra2IiWVvClHazIAdjH?usp=sharing
Fields    Rows    Chunk     Variable size(per chunk)  Total Variables      Status
400       500         1              200000                           200000           no-error
500       500         1              250000                           250000           no-error
600       500         1              300000                           300000           error(too many SQL variables)
10        30000      30              10000                           300000           no-error(chunks using buffer size, handled in frictionless)
700       500         1              350000                           350000           error(too many SQL variables)
70        5000        5              70000                             350000           no-error
sqlite version > 3.32
https://colab.research.google.com/drive/12JA3ZdFRh-6T_o7Lr_Wyx07Qg0rs0u0v?usp=sharing
Fields    Rows    Chunk    Variable size(per chunk)    Total Variables    Status
400       500         1              200000                                200000          no-error
500       500         1              250000                                250000          no-error
600       500         1              300000                                300000          error(too many SQL variables)
10        30000      30              10000                                300000          no-error(chunks using buffer size, handled in frictionless)
700       500         1              350000                                350000          error(too many SQL variables)
70        5000        5               70000                                 350000          no-error
To answer your question regarding buffer size, it breaks insert statements into multiple chunks of 1000 rows if row size is greator than 1000.
I am not quite sure about two options that you have mentioned, which I have to test
self.__connection.execute(sql_table.insert().values(buffer))
fallback: self.__connection.execute(sql_table.insert(), buffer)
From the above result rows * columns also doesn't quite fit to what we assumed to be SQLITE_MAX_VARIABLE_NUMBER.
WDYT?
Could you please provide your feedback @roll  @aivuk ?

@shashigharti thanks for investigating further!
One quick remark:

To answer your question regarding buffer size, it breaks insert statements into multiple chunks of 1000 rows if row size is greator than 1000.

That's what I also figured out in the meantime, thanks for confirming. :-)
I applied the same approach in the benchmark of my last comment.

I am not quite sure about two options that you have mentioned, which I have to test self.__connection.execute(sql_table.insert().values(buffer)) fallback: self.__connection.execute(sql_table.insert(), buffer)
From the above result rows * columns also doesn't quite fit (999 for sqlite < 3.32 and 32766 > 3.32) to what we assumed to be SQLITE_MAX_VARIABLE_NUMBER. WDYT?

From sqlite3 app.db 'pragma compile_options;', do you get MAX_VARIABLE_NUMBER output? Then that's what's relevant instead of the sqlite version number. E.g. my Kubuntu (Debian) sqlite package (< 3.32) has drastically raised limits, i.e. you don't run into the 999 limit there, even though that's "vanilla" sqlite's default.
So you really need to check the actual compiled limits of the sqlite under test to make proper expectations of when the test will fail or run through, not breaching the limits.
If I interpret your configuration output correctly that's also the case for you, so the results look expectable to me (e.g. no-error for 250.000 variables, error for 300.000 variables without chunking)?

Thanks for the amazing input @hjoukl!
@shashigharti and I will be discussing it soon with the team

Hi @roll and @shashigharti,
great to see you considering this issue! Just a note wrt the classification change from bug to enhancement:
Since stock RHEL8 seems to have below sqlite3 defaults (sqlite version + compile options) the usability of frictionless-py is severely limited on this platform, which I think is in very widespread use in "enterprisey" Linux environments (probably predominant). Same goes for the older RHEL7.
I.e. you'd basically need to be able install a newer sqlite version or compile sqlite and use this instead of system sqlite. Which might mean you'd also need to recompile Python (at least it's sqlite3 extension)...
This is something that most users won't be able/allowed to do in a corporate environment, from my experience.
0 $ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.5 (Ootpa)
0 $ sqlite3 --version
3.26.0 2018-12-01 12:34:55 bf8c1b2b7a5960c282e543b9c293686dccff272512d08865f4600fb58238alt1
0 $ sqlite3 app.db 'pragma compile_options;'
COMPILER=gcc-8.4.1 20210423 (Red Hat 8.4.1-2)
DISABLE_DIRSYNC
ENABLE_COLUMN_METADATA
ENABLE_DBSTAT_VTAB
ENABLE_FTS3
ENABLE_FTS3_PARENTHESIS
ENABLE_FTS4
ENABLE_FTS5
ENABLE_JSON1
ENABLE_RTREE
ENABLE_STMTVTAB
ENABLE_UNKNOWN_SQL_FUNCTION
ENABLE_UNLOCK_NOTIFY
HAVE_ISNAN
SECURE_DELETE
THREADSAFE=1

You run into those limits with a table with 20 fields and 50 rows, i.e. not only tables with an unrealistically high number of columns:
0 $ FRICTIONLESS_TEST_FIELDS=20 FRICTIONLESS_TEST_ROWS=50 python3 frictionless_sqlite_params_error.py
Traceback (most recent call last):
  File "venv/rhel8-frictionless-venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1820, in _execute_context
    cursor, statement, parameters, context
  File "venv/rhel8-frictionless-venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: too many SQL variables

...
(Background on this error at: https://sqlalche.me/e/14/e3q8)

So I basically think a change like
$ diff frictionless/plugins/sql//storage.py.BAK frictionless/plugins/sql//storage.py
333c333
<                     self.__connection.execute(sql_table.insert().values(buffer))
---
>                     self.__connection.execute(sql_table.insert(), buffer)
336c336
<                 self.__connection.execute(sql_table.insert().values(buffer))
---
>                 self.__connection.execute(sql_table.insert(), buffer)

is both a bugfix (the 1000-lines chunking is not enough to avoid exceptions in general) and an enhancement (it looks like it's dramatically faster for SQLite, but of course other DB backends might completely differ - needs proper benchmarking).
Hopefully not getting on anybody's nerves here - of course it's entirely your call how you handle this. I just wanted to bring it to your attention that frictionless-py might not be properly usable with an SQLite DB backend on these mentioned platforms, at all. Which would be a shame IMHO since it's so cool. :-)
Best regards,
Holger

If I interpret your configuration output correctly that's also the case for you, so the results look expectable to me (e.g. no-error for 250.000 variables, error for 300.000 variables without chunking)?
Thank you very much @hjoukl. Yes it is because I am using ubuntu the SQLITE_MAX_VARIABLE_NUMBER is set to 25000.

I got curious so hacked together a rather naive benchmark (timeit-based, measurements without any precautions to keep the machine idle apart from the benchmark process).
# benchmark_sqlite_params.py

# Pretty naive timeit benchmark
# I'm using globals all over the place here - don't do this at home, kids.

import os
import timeit

from sqlalchemy import (
        Column, MetaData, String, Table, create_engine, delete, func,
        select)
from sqlalchemy.orm import Session

# some outside configurability through env vars
number_of_fields = int(os.environ.get('FRICTIONLESS_TEST_FIELDS', '90'))
number_of_rows = int(os.environ.get('FRICTIONLESS_TEST_ROWS', '100'))
timeit_number = int(os.environ.get('FRICTIONLESS_TEST_TIMEIT_NUMBER', '1000'))
buffer_size = int(os.environ.get('FRICTIONLESS_TEST_BUFFER_SIZE', '1000'))

col_pad = len(str(number_of_fields)) - 1
row_pad = len(str(number_of_rows)) - 1

columns = [Column(f'col{i:0{col_pad}}', String(col_pad + row_pad + 1)) for i in range(number_of_fields)]
rows = [
    {col.name: f'row{r:0{row_pad}}_col{c:0{col_pad}}' for (c, col) in enumerate(columns)}
    for r in range(number_of_rows)
    ]

engine = create_engine('sqlite:///app.db', echo=False)
meta = MetaData()
Mytable = Table('mytable', meta, *columns)
meta.create_all(engine)

def insert_values_params(session):
    stmt_values_params_base = Mytable.insert()
    buffer = []
    for row in rows:
        buffer.append(row)
        if len(buffer) >= buffer_size:
            stmt_values_params = stmt_values_params_base.values(buffer)
            session.execute(stmt_values_params)
            buffer = []
    if len(buffer) > 0:
        stmt_values_params = stmt_values_params_base.values(buffer)
        session.execute(stmt_values_params)
        buffer = []
    session.commit()

def insert_values_executemany(session):
    stmt_executemany = Mytable.insert()
    buffer = []
    for row in rows:
        buffer.append(row)
        if len(buffer) >= buffer_size:
            session.execute(stmt_executemany, buffer)
            buffer = []
    if len(buffer) > 0:
        session.execute(stmt_executemany, buffer)
        buffer = []
    session.commit()

def check_and_clean(session):
    count_stmt = select(func.count()).select_from(Mytable)
    result = session.execute(count_stmt)
    print(f'{count_stmt}: {result.all()}')
    last_row = session.query(Mytable).order_by(Mytable.c[0].desc()).first()
    print(f'last row/col value: {last_row[-1]}')
    session.execute(delete(Mytable))
    session.commit()

with Session(engine) as session:
    print('\n--------------------------------------------------------------------')
    print("timeit.timeit('insert_values_params(session)', "
           "number=timeit_number, globals=globals())")
    print(timeit.timeit('insert_values_params(session)',
           number=timeit_number, globals=globals()))
    check_and_clean(session)

with Session(engine) as session:
    print('\n--------------------------------------------------------------------')
    print("timeit.timeit('insert_values_executemany(session)', "
          "number=timeit_number,  globals=globals())")
    print(timeit.timeit('insert_values_executemany(session)',
          number=timeit_number, globals=globals()))
    check_and_clean(session)

In my initial tests this seems to clearly hint - for the sqlite3 engine at least - that applying executemany is (vastly) superior to "named/numbered parameters for all row values":
 rm -f app.db && FRICTIONLESS_TEST_FIELDS=50 FRICTIONLESS_TEST_ROWS=10000 FRICTIONLESS_TEST_TIMEIT_NUMBER=100 FRICTIONLESS_TEST_BUFFER_SIZE=1000 ./frictionless-py-venv/bin/python benchmark_sqlite_params.py

--------------------------------------------------------------------
timeit.timeit('insert_values_params(session)', number=timeit_number, globals=globals())
547.7838764779999
SELECT count(*) AS count_1
FROM mytable: [(1000000,)]
last row/col value: row9999_col49

--------------------------------------------------------------------
timeit.timeit('insert_values_executemany(session)', number=timeit_number,  globals=globals())
20.852090813999894
SELECT count(*) AS count_1
FROM mytable: [(1000000,)]
last row/col value: row9999_col49

(unless I've made some really stupid mistake in the benchmarking code, obviously :-))
So please take with a grain of salt - but I think this indicates that some proper benchmarking should be done, preferably with other DB backends/engines, too.
Best regards, Holger

Thank you again for the benchmark code! 'executemany' seems to be faster than 'named/numbered parameters'!.
Please feel free to create a PR for it or I can create one whichever is fine for you.

is both a bugfix (the 1000-lines chunking is not enough to avoid exceptions in general) and an enhancement (it looks like it's dramatically faster for SQLite, but of course other DB backends might completely differ - needs proper benchmarking).

Current solution that I can think of, would be reducing the chunk size and the optimizing the insert statement using execute many(benchmarking code above). And also investigate it with other DB backends. WDYT? @hjoukl @roll @aivuk

Hi,

Please feel free to create a PR for it or I can create one whichever is fine for you.

I'll be glad to come up with a PR with the proposed simple fix (switch to executemany, not touching the current chunk size implementation or trying to further optimize) and a simple test.
Can't promise benchmarking other DB backends though. I'll try to take a look but no promises.
One caveat: I'll be travelling for 3 weeks from next week on. So if I'm not getting to land the PR this week there'll be delay from my side - I'm planning to generally stay away from computers. :-)

Current solution that I can think of, would be reducing the chunk size and the optimizing the insert statement using execute many(benchmarking code above). And also investigate it with other DB backends. WDYT? @hjoukl @roll @aivuk

What DB backends do you minimally aim for in test coverage and/or benchmarking? I've noticed Postgres and Mysql in your actions pipeline. So these (and sqlite) are probably covered by CI wrt to testing?

Can't promise benchmarking other DB backends though. I'll try to take a look but no promises.
One caveat: I'll be travelling for 3 weeks from next week on. So if I'm not getting to land the PR this week there'll be delay from my side - I'm planning to generally stay away from computers. :-)

I think that is fine :).

What DB backends do you minimally aim for in test coverage and/or benchmarking? I've noticed Postgres and Mysql in your actions pipeline. So these (and sqlite) are probably covered by CI wrt to testing?

Mysql, postgresql and sqlite for now
@roll @aivuk WDYT?

@shashigharti :+1: mysql, portgresql and sqlite
Thanks for helping with this issue @hjoukl!

Hi there @shashigharti @roll,
PR #1255 is there (finally) - sorry for the long silence.
That's the minimal code change discussed in previous ticket comments plus accompanying test.
As a little extra and compensation for your patience I've benchmarked the principal executemany approach against the one previously used in frictionless-framework storage.py. You can find it all here: https://github.com/hjoukl/bench-sqlalchemy-executemany :-)
The gist of it is that executemany is superior to the previous approach performance-wise, sometimes vastly (especially with SQLite). I've run the sample benchmark for the DB engines tested in your CI (SQlite, PostgreSQL, MySQL).
Btw I've stumbled over some rough edges in the v5 docs (e.g. invalid examples using not-existing-anymore plugins imports) and some missing bits that might be useful for the development/contribution docs (like "copy .env.example to .env before running make test"). Would you like some suggestions wrt this in form of tickets or otherwise?
Greetings,
Holger

Hi @roll, @shashigharti,
please reopen:
Sadly, somehow the (merged) PR #1255 never made it to framework v5:
https://github.com/frictionlessdata/framework/blob/01689f923560cf6d5aa6b6b0fd44229118fec75f/frictionless/formats/sql/adapter.py#L124-L126
(Accompanying tests test_sqlite.py  and conftests.py additions also missing, see e.g.: https://github.com/frictionlessdata/framework/tree/main/tests/formats/sql)
So I'm running into this issue again on frictionless 5.3.0. Since this was merged in Oct. 2022 after the last v4 release, it looks like there's now no released version that contains the fix. :-(
Let me know if I can help...
Best, Holger


fix(compiler): Clarify Finished message

What does this PR try to resolve?
As highlighted on zulip, many users think "Rust is slow" because of the dev profile.
While a perfect solution is still being worked out, this attempts what will hopefully be smaller, incremental step that hopefully maintains balance of the different needs.
We are changing the message from:
Finished dev [unoptimized + debuginfo] target(s) in [..]s

to
Finished `dev` profile [unoptimized + debuginfo] target(s) in [..]s

where dev profile is a link to the Cargo book.
The intent is

Clarify what dev even means

Add profile to give it context
Quote it to highlight this is something mutable

Make the profile content stand out on hover by being a link
Help people learn more by following the link

For now, this leaves the profile description alone.
Part of #9446
How should we test and review this PR?
Additional information

I also changed the link to point directly to the built-in profile definitions so they can see what dev means and then browse further for what those fields mean.

I also changed the link to point directly to the built-in profile definitions so they can see what dev means and then browse further for what those fields mean.

Sounds like we can make it smarter by linking to each profile?

Anyway, this is a two-way door. Going to merge this.
@bors r+

:pushpin: Commit 69f38a08d139da36185be5d1a7832aad5a411dc0 has been approved by weihanglo
It is now in the queue for this repository.

:hourglass: Testing commit 69f38a08d139da36185be5d1a7832aad5a411dc0 with merge<PHONE_NUMBER>7f7524f657e4692d672354000fb202...

:sunny: Test successful - checks-actions
Approved by: weihanglo
Pushing<PHONE_NUMBER>7f7524f657e4692d672354000fb202 to master...

I dont like this change-- its rather annoying, and not very useful.
Im pretty sure a total of two people have clicked on the link intentionally (not to mention low terminal support for non XTerm feature), and the people who click on it intentionally are simply annoyed, as theres no need for this guide. Also, the pr description says that \dev` profileis a link, but the entirety of`dev` profile [unoptimized + debuginfo]` is a link.

@bend-n could you clarify why you find this change to be annoying?  We expect to be adding a lot more links to Cargo and Rustc's outputto better connect the commands to their documentation.

as theres no need for this guide

This runs a bit counter to our experience talking to users as we found they didn't know what dev meant.  Granted, also clarified it is a "profile" which also helps.

its disruptive, as you can accidentally click on it, and in certain terminals has pop ups that are annoying. however, i have since solved my problems:
set -x FORCE_HYPERLINK 0

Note that FORCE_HYPERLINK is not officially support and is a side effect of the implementation.
You should probably be using CARGO_TERM_HYPERLINKS=false

oh, thats cool. thanks!

You set
[term]
hyperlinks = false

and it didn't work?

well, i set
[build]
rustflags = ["-Ctarget-cpu=native"]
target-dir = "/mnt/HDD500/.target"

[unstable]
gc = true

[term]
hyperlinks = true
unicode = true

and it didntt work.

Would this be improved by shrinking the range of the hyperlink?

The point of the range of the hyperlink was to cover everything profile related.  The most likely alternative (what I wish I had done but hadn't felt motivated enough to implement after the PR) was to link directly to the documentation for the referenced fields, instead of one monolithic link.
Again, as I said, this also isn't expected to be isolated.  Both Cargo and Rustc are looking to take advantage of more links in the output to help bridge the tool / documentation divide.  Shrinking in a one off case will have little impact if we have a lot more links.  Its more of needing to decide if the general vision for links needs to be adjusted and, if so, to what.


Fix amortized asymptotics of BytesMut

Closes #524. See that issue for a description of what this PR changes and why.

Interesting test case that fails here; there's room for improvement so this doesn't have to fail.

So, I somehow only considered the case of reserving more space when the buffer is currently full to capacity. I suppose that it's not actually an important metric that the capacity is doubled when reserving new size, but rather it's sufficient to consider the current length. After all, the cost of moving the data is only as high as the current length.
Let me try to remember the amortized analysis (for Vec<u8>) for why pushing bytes one-by-one is (amortized) constant time (for each push operation)... the re-allocating+moving a whole, full, Vec is accounted for by previous push operations. If the capacity of a Vec doubles every time, than always the cost of (the latter) half of the data that had been pushed to the Vec accounts for moving the whole thing another time. Each byte being pushed allows for two bytes being copied in the future, thus we get constant time push. After resizing, the latter half of the Vec is empty and would need to be filled by push operations again, before more resizing could happen.
The test case in question uses reserve though, requiring additional capacity without a previously completely filled buffer, which complicates the matter because I'm not 100% certain how that interacts with the whole analysis.
Well, at least, one main point in the analysis above was that when we know that some amount of amortized-constant-time operations have happened since the last time all the data was (possibly re-allocated and) copied, it ought to be okay to copy twice the amount of data around, compared to how many of those amortized-constant-time operations operations we had, as long as we and up with an at-most half-full buffer in the end. (The last condition to make sure that ordinary sequences of push will write half the buffer before resizing / copying again.)
So in this test case
fn bytes_buf_mut_reuse_when_fully_consumed() {
    use bytes::{Buf, BytesMut};
    let mut buf = BytesMut::new();
    buf.reserve(8192);
    buf.extend_from_slice(&[0u8; 100][..]);

    let p = &buf[0] as *const u8;
    buf.advance(100);

    buf.reserve(8192);
    buf.extend_from_slice(b" ");

    assert_eq!(&buf[0] as *const u8, p);
}

there's, for the second reserve (which is the operation in question), an empty buffer with an offset of 100. We know from an offset of 100, that 100 bytes have been read (using get_u8 or similar) since the last time the data was moved to the front of the Vec. Reading a byte in this manner is an amortized-constant time (well, even an actual constant time) operation, so we can use that as justification to move up to 200 bytes, as long as we end up with a capacity >= 2*len situation in the end.
The 2 is actually an arbitrary constant factor, coming from the arbitrary rule that "each constant-time operation accounts for two bytes being copied in the future" let's generalize to C
This suggests a rule like...

C * off >= len should be checked (this is weaker than the current [in this PR!] condition off >= cap, provided C >= 1) to make sure we can afford moving all the data
2 * len <= off + cap to make sure that the buffer is half-empty after the operation (off + cap is the new capacity), so that future resizes due to actually reaching the limit can be afforded. This is also weaker than the current [in this PR!] condition off >= cap. Not quite as obvious, but: if off >= cap, then 2 * len <= 2 * cap == cap + cap <= off + cap

Also, I like keeping the copy_nonoverlapping, and the test case that failed doesn't need to break it. Let's see how the calculation here works out then:

Overlap is avoided if off >= len.
This implies that C == 1 is a great choice, actually!

So with C == 1, the checks are

off >= len
2 * len <= off + cap

except, the second one is redundant now. len + len <= off + cap is clear from len <= off and len <= cap.
So let me change the PR to just use a off >= len check instead of off >= cap check.

I just noticed the part of the documentation that says

Before allocating new buffer space, the function will attempt to reclaim space in the existing buffer. If the current handle references a small view in the original buffer and all other handles have been dropped, and the requested capacity is less than or equal to the existing buffer’s capacity, then the current view will be copied to the front of the buffer and the handle will take ownership of the full buffer.

Is shockingly explicit about the “the requested capacity is less than or equal to the existing buffer’s capacity” condition. I’m expecting a more vague statement similar to the preceding “More than additional bytes may be reserved in order to avoid frequent reallocations.” in order to leave the implementation details unspecified. As it stands, this sounds as if only frequent reallocations are bad, but frequent "copy everything to the front of the buffer" is totally okay.

I’m open for suggestions of how to best improve this.

As long as we have C * len ≤ cap for some constant C greater than one, then the amortization will work since you have to perform at least (C-1) * len get+put operations between each time it copies the data backwards.

Interesting point. So checking 2 * len <= off + cap alone could be enough?

Yes, though arguably we might want to pick a smaller C such as 1.5.

I'm trying to evaluate whether it's important to make sure that something like
let mut bytes = BytesMut::with_capacity(10_000);
bytes.put_bytes(0, 5_000);
loop {
    bytes.get_u8();
    bytes.put_u8(0);
    bytes.reserve(5_000);
}

remains efficient.
The buffer is always half-full at the point of reserve.
An approach that allows shifting the data when 2 * len <= off + cap will shift the whole (5_000 bytes) data for each reserve, while the PR as it stands currently would increase the size of the buffer once by growing the Vec (so it goes to 20_000), and then it only needs to shift all the 5_000 bytes once every 20_000 iterations.

increase the size of the buffer once by growing the Vec (so it goes to 20_000), and then it only needs to shift all the 5_000 bytes once every 20_000 iterations.

That seems fine to me.

What's the status here? If you wanted to update the text in the documentation, then go ahead and do that as well.

Oh, well I guess I kind-of forgot about this PR entirely. Alright, so I’m going to think of a way to reword the documentation, and I’ll keep the implementation as it currently is in this PR.

Sounds good!

@steffahn I think you forgot to also fix the amortized asumptotics for cases where BytesMut holds a unique reference to KIND_ARC.

@NobodyXu I will take a look at this this weekend.

Sorry for the delay. The new documentation got somewhat lengthy, but should be okay now. I will probably also re-read it for any mistakes tomorrow.
@NobodyXu, I indeed didn’t look at the KIND_ARC case before, and I think it makes sense to apply the same logic there. I have adapted the same logic into that part of the code now (hopefully correctly).

Fixed the trailing whitespace. I’ll have to figure out what the most proper way of calculating the offset without offset_from is. I could convert into usize and subtract; I’m not sure whether or not maybe the BytesMut type itself already stores the offset somehow. I won’t look back into this before tomorros.

Fixed the trailing whitespace. I’ll have to figure out what the most proper way of calculating the offset without offset_from is. I could convert into usize and subtract; I’m not sure whether or not maybe the BytesMut type itself already stores the offset somehow. I won’t look back into this before tomorrow.

I think you can substract vec.capacity() - self.cap and that would give you the offset.
Converting to usize is probably UB because of pointer provenance.

Converting into usize is definitely fine. The trouble is when you convert the usize back into a pointer.

Converting into usize is definitely fine. The trouble is when you convert the usize back into a pointer.

Yes, I think converting it back to a pointer might trigger UB, or at least involves some dark corner of the compiler.

This PR seems to be staling for a long time...Will it be a good idea for me to fork @steffahn 's branch and create a separate PR as I want this optimization to be in bytes @Darksonn

I haven't looked at this for a while and I don't remember what the status is.

IIRC, I only have to rephrase/improve the documentation. Sorry for the delay; in the meantime, I had to finish a master’s thesis due to the end of March; since that’s done now, I suppose I should get around to finish up this PR.

Sounds good. And congratulations on finishing your master's thesis!

So, I’ve rewritten the section in the docs now. Maybe still not the perfect balance in terms of lengthiness, but it seems necessary to give at least enough information for the subsequent examples to make sense, and the condition is somewhat complex. I’m of course still open to suggestions how to make it shorter and/or better; but maybe it's already acceptable for now?

Not sure why, but it seems like some tests have started failing.

I can reproduce the test failure and miri UB locally when merging with master, I will need to take a look into this.

It may be due to some changes in miri? Note that #542 exists, which may affect things when it comes to miri.

Though it seems like the failure also exists on non-miri.

There's the rest failure unrelated to miri though. And the miri error is reading of uninitialized memory. Tests and miri do succeed (with current miri) both in master and on this PR branch, but not on the result of merging the two. Maybe the diff just results in logically broken code somehow due to other changes (yet its not too broken to be a merge conflict), who knows... I hope I get around to looking into this tomorrow.

Pinging this PR.

@steffahn Pinging this PR.

@steffahn Pinging since I'd really want this PR to be merged.