Allow exceptions to modify debugging tracebacks.

Extremely simple, checks for a "exc.werkzeug_debug_traceback" method and invokes that instead of the Traceback constructor when the corresponding exception is raised. This is useful for packages (like Mako) that autogenerate code and need to separately annotate tracebacks to translate source lines.
For an example of this project's usefulness, see this commit in the flask-mako extension: AnIrishDuck/flask-mako@ea6a05171caba6ec2e094467cb8087a148e391e3. This commit vastly improves the debugging environment when errors occur inside Mako templates.

Just found my way here from benselme/flask-mako#2 and sorry to see this was just a testcase and changelog entry away from getting merged but it never happened.
@untitaker and other maintainers, if someone does that last bit of work, would you still merge this?

Jinja2 already fixes tracebacks, doesn't mako have the same something

Thanks for taking a look @RonnyPfannschmidt. (Are you a werkzeug maintainer?) I'm new to this code (so @AnIrishDuck might have more to say) but from https://github.com/benselme/flask-mako/commit/0b537235fb125f427de7ab4acbbe5f6f6f2a258d it looks like the answer is yes, but flask-mako still needs this change to take advantage of it.
Would you still be willing to merge if someone adds the testcase @mitsuhiko requested?

Is this still needed? I'm not clear how the linked commit in Flask-Mako needs this. If someone adds a simple test case demonstrating what this does, it's fine to merge.

Is this still needed?

Yes, for the exception formatting in flask-mako to work. I'm not sure that flask-mako still works with new versions of werkzeug and mako, but that's not really your problem :)

I'm not clear how the linked commit in Flask-Mako needs this.

It allows templating libraries (i.e. flask-mako) to modify the exception handling behavior in werkzeug. In particular, we need to be able to translate the stack differently. The comment by @RonnyPfannschmidt threw a bit of a wrench in the works - I was not aware that there was a way to modify error stacks directly in the templating library. That seems like a more appropriate long-term fix.
Unfortunately, I am no longer working on flask-mako (or indeed python) with enough frequency to pick this back up again. So the way forward is to close this, and wish whoever is audacious enough to do the necessary work on Mako good luck.

I'm not sure that flask-mako still works with new versions of werkzeug and mako

Running the latest versions of Flask (0.12.1), Werkzeug (0.12.1), Mako (1.0.6), and flask-mako (0.4) together is working fine for me.
(I'm using flask-mako because I'm using Plim, which depends on Mako. Pretty big fan of Plim; surprised it (or something like it) doesn't seem to be more popular among Flask/Werkzeug users.)

@jab if you're interested, you can apply this PR as a patch from https://github.com/pallets/werkzeug/pull/206.patch, add a test, and submit a new PR.

For the record, I still believe that this is the wrong solution for the problem. The right solution is to fix Mako to generate tracebacks like Jinja does.
 The problem is that it's not guaranteed to stay that way. The matching changes in flask-mako introduce a lot of coupling between that extension and how werkzeug handles errors internally. werkzeug's error handling could presumably change in the future, which would be likely to break that code.
I also wouldn't be surprised if future Mako changes break that code.
It's much less likely said breakage will occur if Mako uses the same approach as Jinja. Moving the error handling logic closer to where the errors happen is just less brittle. But, ultimately my opinion isn't worth much because I can't help do the work.

Yeah, that makes sense. So let's leave this be then.


Using process scoped environment variables to populate NBi variable

Hi @Seddryck ,
I will not be able to do any real C# code changes but at least I can contribute the full description and reasoning.
This discussion originally started in #410 . As I assume this is just a one line code check I have created PR #718
background
environment variables
Windows systems seem to have three kind of environment variables process, system and user scope.
System and user scope are in general set with the configuration dialog from the start menu:

Process based environment variables are in PowerShell just set by $Env:MYVAR="value".
NBi
The usage of environment variables is explained at http://www.nbi.io/docs/variable-define/#environment-variable .
Automation
I assume NBi is most of the time used in automatic testing and in DevOps pipelines.
Moreover, most likely NBi scripts contain references that are created as part of an Azure DevOps pipeline and thus connections strings and server names must be dynamic.
Azure DevOps and most likely any other build agent Jenkins/Bamboo/etc. have the option to pass in environment variables. In Azure DevOps this is easly done and the suggested way to pass in any secret values. ( https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#secret-variables )
Currently my Azure DevOps pipelines do hack the nbits file to adjust several settings instead....
solution
The relevant code is at: EnvironmentScalarResolver.cs
The documentation of the API is at https://learn.microsoft.com/en-us/dotnet/api/system.environment.getenvironmentvariable?view=net-7.0#system-environment-getenvironmentvariable(system-string) .
Which states that:
All environment variables that are provided to it by the parent process that created it. For example, a .NET application launched from a console window inherits all of the console window's environment variables.
If there is no parent process, per-machine and per-user environment variables are used instead. For example, a new console window has all per-machine and per-user environment variables defined at the time it was launched.
As a result I conclude, that we just have to switch from
            var value = Environment.GetEnvironmentVariable(args.Name, EnvironmentVariableTarget.User);

to
            var value = Environment.GetEnvironmentVariable(args.Name);

As a result it should be backwards compatible and in addition work for variables set at process scope...

I have tested release 1.24 on Azure DevOps today and it works very very well. Thank you.


Parametrize opentelemetry service.name via --otel-service / SCOPE_OTEL_SERVICE

When we use scope in several different applications we need to distinguish between them.
This PR introduces the new CLI argument
      --otel-service <OTEL_SERVICE>
          [env: SCOPE_OTEL_SERVICE=]
          [default: scope]

Test:
SCOPE_OTEL_SERVICE=myapp-scope SCOPE_OTEL_ENDPOINT=tcp://<IP_ADDRESS>:4317 cargo run doctor run

Hey @maratkomarov , I'm not very familiar with those part of the code, but think automatically setting a new KeyValue property on the OTEL resource based on the current directory is a better approach here.
This has the benefits of:

Being set automatically, and not requiring scope consumers to set this
Not risking app generated metrics (likely with a service name of $APP_NAME) conflicting with dev environment originated metrics

Would that address your original issue?

@noizwaves  Current directory is not stable. For a same application repository - i might clone it into foo, and another user into bar. My change doesn't add any new burden for a scope user - if service is not passed, then the default scope is used like it was before.

Thanks @maratkomarov. Could you elaborate more on why this needs to be a service name override, and not an attribute or tag on the trace?
I think there may be an underlying concept (like app name) that we could automatically set for all traces, and that may be the config we want to override here.

I would shift the focus on why you're so resistant to this change?

The following thread has a lot of context https://github.com/Gusto/terraform/pull/13041#discussion_r1871773153

Just to chime in here, OTEL_SERVICE_NAME is the standard OTEL environment variable.
OTEL expects users to set this.
https://opentelemetry.io/docs/languages/sdk-configuration/general/#otel_service_name

Thanks for the extra information here @rubberduck203 & @maratkomarov - this was useful context. Adding this standard configuration for overriding the service name makes sense to me. This could be useful for purposes other than per-application uses.
I do still believe there's a better way to accomplish per-application observability here, that:

could be automatically derive a "project name" from a working directory, git origin, or an explicit YAML config, and add this as a filterable attribute onto spans
can work across all scope entry points, like Gusto's gdev wtf

Let's follow up with this later.


NPM install failing

What type of issue is this?
Question
What SharePoint development model, framework, SDK or API is this about?
💥 SharePoint Framework
Target SharePoint environment
SharePoint Online
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[ ] 💥 Microsoft Edge
[ ] 💥 Google Chrome
[ ] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[ ] not applicable
[X] other (enter in the "Additional environment details" area below)

Additional environment details

browser version:  NA
SPFx version: 1.12.1
Node: v12.18.0

Issue description
Running yo @microsoft/sharepoint  builds the scaffolding, but seems to fail on npm install bit.
Running npm install (--save-dev or not) fails also.
Running npm install --save-dev '@microsoft/sp-build-web' fails aswell
node_modules folder not built
Getting error like this:
npm ERR! notarget No matching version found for<EMAIL_ADDRESS>npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of 'event-stream'

I'm wondering if you have some old version of something kicking around, pinned, etc.  on a brand new 1.12.1 install, event-stream looks like this:

npm list event-stream
<EMAIL_ADDRESS>xxxxx\code\1121
--<EMAIL_ADDRESS> --<EMAIL_ADDRESS>--<EMAIL_ADDRESS>     --<EMAIL_ADDRESS>
and

npm list flatmap-stream
<EMAIL_ADDRESS>xxxxx\code\1121
`-- (empty)

I cant do a npm list, as the node_modules folder doesnt get built out :(

If you take a new full directory structure, and don't have a customized .npmrc file / etc. does this still happen?  I've run the install in numerous versions, and I'm not getting any errors, and the flatmap-stream vulnerability was reported / removed more than 3 years ago.

Thanks for the reply @patmill

If you take a new full directory structure
Yes, tried a few times.
and don't have a customized .npmrc file / etc.
That has got to be it.  My company uses it own artifact repo, instead of github, etc.
Something old there I imagine, ugh.
Would you happen to know what package might have used flatmap-stream in the past?  so I can track down it down and get it updated?  Any other tips appreciated :)

From your initial error, it looks like event-stream.
npm ERR! notarget It was specified as a dependency of 'event-stream'

Yes, you are right.  Sorry should have been more specific.   How can I find what uses event stream?

For 1.12.1, it's this chain:
xxxx\code\1121>npm list event-stream
<EMAIL_ADDRESS>xxxx\code\1121
--<EMAIL_ADDRESS> --<EMAIL_ADDRESS>--<EMAIL_ADDRESS>     --<EMAIL_ADDRESS>for a much older build (1.4.1) it's this:
xxxx\code\2019_1100>npm list event-stream
<EMAIL_ADDRESS>xxxx\code\2019_1100
--<EMAIL_ADDRESS> +--<EMAIL_ADDRESS> | --<EMAIL_ADDRESS>|   --<EMAIL_ADDRESS> --<EMAIL_ADDRESS>--<EMAIL_ADDRESS>     --<EMAIL_ADDRESS>

closing now.


DOC: Include parent object in sidebar links

Issue with current documentation:
Once you navigate down to the docs for methods of a class, there is no link that will bring you back to the top level object. For  example, in the page for scipy.spatial.transform.Rotation.from_quat, there isn't a link that I can see to go back up to scipy.spatial.transform.Rotation.
Idea or request for content:
It'd be great to have the top level object as the first item in the sidebar list:

Additional context (e.g. screenshots, GIFs)
No response

Thank you for reporting @scottshambaugh. We actually just changed the sidebar to remove all nesting. See the dev docs.
The difficulty is that dynamic nesting cannot be done in the sidebar (html size issue as we need the full nesting of everything and cannot filter out). It's something the they are thinking about on the theme side.
Having another button or UI element for that would also be an issue for the theme.
I will close for now as it a theme thing.


I can't make the 17th. Moving to following Saturday

I can't make the 17th either. The 24th works better for me too.

How about the Sunday? I have quidditch practice on Saturday...


The reason for using two datasets together?

Hi Tony,
Thank you for sharing the code publicly. I read the paper and I am wondering if there are any specific reasons for using these two ACCORD and SPRINT datasets together, especially for the sensitivity analysis part, the result for using only one dataset is not as good as the combined one. So, how to ensure the generability?
Thank you very much.
Suxi

Thanks for reaching out.
The primary reason we wanted to combine the two datasets together was to improve the amount of training data available, as the combination of SPRINT (N=9361) and ACCORD BP (N=4733) yielded significantly more patients. A secondary reason was to explore to what degree the method could show the positive ARR in SPRINT and null result in ACCROD BP (Supplemental Figure 5).

Please don't hesitate to reach out with further questions.

Hi Tony,
Thanks for the detailed explanation.
Suxi


SwaggerDefinition should be completely supported (add support for @Info)

Right now, it seems that swagger-play only partially support the core annotation @SwaggerDefinition.
If I use the annotation provided in the documentation over one of my constructor:
 @SwaggerDefinition(
		info = @Info(
				description = "Gets the weather",
				version = "V12.0.12",
				title = "The Weather API",
				termsOfService = "http://theweatherapi.io/terms.html",
				contact = @Contact(
						name = "Rain Moore",
						email =<EMAIL_ADDRESS>						url = "http://theweatherapi.io"
				),
				license = @License(
						name = "Apache 2.0",
						url = "http://www.apache.org/licenses/LICENSE-2.0"
				)
		),
		consumes = {"application/json"},
		produces = {"application/json"},
		schemes = {SwaggerDefinition.Scheme.HTTP, SwaggerDefinition.Scheme.HTTPS},
		tags = {
				@Tag(name = "Private", description = "Tag used to denote operations as private")
		},
		externalDocs = @ExternalDocs(value = "Meteorology", url = "http://theweatherapi.io/meteorology.html")
)

It will only generate this:
{
  "swagger" : "2.0",
  "info" : {
    "version" : "1.0",
    "title" : "",
    "contact" : {
      "name" : ""
    },
    "license" : {
      "name" : "",
      "url" : "http://licenseUrl"
    }
  },
  "host" : "localhost:9000",
  "basePath" : "/",
  "tags" : [ {
    "name" : "Private",
    "description" : "Tag used to denote operations as private"
  }, {
    "name" : "Actors"
  }, {
    "name" : "Affiliates"
  } ],
  "schemes" : [ "http", "https" ],
  "consumes" : [ "application/json" ],
  "produces" : [ "application/json" ],
  "externalDocs" : {
    "description" : "Meteorology",
    "url" : "http://theweatherapi.io/meteorology.html"
  }

It seems only these fields are supported:

consumes
produces
schemes
tags

The more important field, called @Info is completely ignored and the value we see are the default values.
I'm using swagger-play2 1.5.3

@JFCote You can use application.conf for that instead: https://github.com/swagger-api/swagger-play/tree/master/play-2.6/swagger-play2#applicationconf---config-options

@jorge-sanz
Is there any way I can put the email in the contact from application.conf ? Currently I don't see any documentation for that.

@ultrasonex Me neither. I would put it in the swagger.api.info.contact property.
One of the Swagger staff member told me that would change a lot of stuff in Swagger Core v3 including the annotations thing. :-)

@jorge-sanz  Thanks for replying !   I tried to put the contact and it shows in the swagger.json as "contact" : {       "name" : "name"     },  but seems to me missed in the swagger-ui . Not sure how why it is not visible. Do you know when v3 is releasing and when will be ready for play from your source?  Anyways looking forward to it.


PRs not showing up

🐞 Problem
It looks like some PRs are not showing up on the dashboard, even after manually syncing. For me in particular it is https://github.com/rust-lang/crates.io/pull/1186

I'm investigating this now

not sure if relevant, but the first commit of the PR branch was done yesterday already, but the PR was only opened today after the second commit was done too 🤔

@Turbo87 I think there's been a small change in one of the GitHub API responses that's causing this issue, I've replicated it, just putting together a fix now

Just deploying a fix now, will go back through and pull in any unsynced prs shortly

awesome, thanks for the fast response!

Code deployed, and it now doesn't throw exceptions when downloading prs, although appears to still not find that pr, will investigate further after some lunch with @BenJam

So this is weird, https://github.com/rust-lang/crates.io/pull/1186 doesn't show up in your event feed https://api.github.com/users/Turbo87/events?per_page=100, so there's no way for 24pr to find using that API.

hmm, that is strange indeed... I wonder if I'm the only user with that problem or if that repo is somehow messed up and nobody gets credit for contributing to it 🤔
sounds like a GitHub bug to me...

FWIW some other PRs in that repo have now made their way into 24PRs (e.g. https://github.com/rust-lang/crates.io/pull/1192), so apparently this was only a temporary issue 🤷♂

I've been having this issue, too! I'm a grinch but shouldn't be :( !


[Docs] Contributing: Breaking out Meshery Server into separate page

create meshery-server page
suggested reading; errors

Description
This PR fixes #
Notes for Reviewers
Signed commits

[ ] Yes, I signed my commits.

🚀 Preview for commit 43a426030ae5e00f8beff80c4c6b7e085cf3c1cd at: https://60f72b6982110100c0ba09f7--meshery-docs-preview.netlify.app

🚀 Preview for commit b3fe10b5256fd798a0b521d5a1f70c3653c66ccf at: https://60f72f32022fc500a586d60c--meshery-docs-preview.netlify.app


installer images: Add available modules to stage-1 on ARM platforms

See relevant PRs

#119974
#121450

Motivation for this change
There are two main classes of modules this adds:

graphics enablement
storage enablement

Graphics is to give end-users feedback about the current boot. If and when things fail in stage-1, it is helpful for users.
Storage is to allow loading stage-2! All of the additions here are for USB support in stage-1. This allows using a USB drive to write either of the "SD" image or the UEFI iso installer.
This was tested on

Pinebook (A64)
Pine A64-LTS
Raspberry Pi 3B
Raspberry Pi 4B
Libre Computer AML-S805X-AC (La Frite)

Special note:

Pinebook Pro (the UEFI iso seems to break rockchipdrm or something else in the display chain, SD image on USB works fine)
Orange Pi PC (armv7l)

Testing, for aarch64 included installing from the UEFI iso (except for the Pinebook Pro).
Testing, for armv7l was limited to running the UEFI iso. Installing requires a complete native build due to the lack of native compilation.
But... how were the images built?
Using this commit (branch tip might move), which includes some cross-compilation fixes:

https://github.com/samueldr/nixpkgs/commit/b2a67fa784c819b15e21e0072842f564f67964ec

Using my recently refactored "cross-system" repo.
 # Assuming <nixpkgs> is correct, or `--arg pkgs` is given
 $ nix-build -A aarch64-linux -A armv7l-linux

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

This isn't touching anything #121450 touches, so it can be reviewed in parallel!

This PR enables ARM_LPAE on armv7l, which has the unfortunate side effect of disabling support for the Cortex-A5, A8, and A9 processors. In my case, this prevents the standard NixOS kernel from booting on the Beaglebone boards (TI AM335x).
There's no explanation given in this PR, but was this was done to support 32-bit kernels on the RPi 4?

There's no explanation given in this PR, but was this was done to support 32-bit kernels on the RPi 4?

I guess it's for future (I hope) native armv7l builders (the ones that use qemu-kvm on aarch64).

Perhaps they should just apply that option in their config, rather than in the standard kernel.

At the time I tried searching information about which CPUs supported LPAE, and was led to believe that all "relevant" (read: armv7) CPUs supported LPAE.
@lopsided98 We can remove this. armv7l as a whole is more important than only some.
When we'll need it for builders, we'll see how we'll handle it. Whether we add a new attribute (for more broad use) or "just" customize the kernel config in the builder's config.

#132100


Add dummy package AzureTemplate for EngSys development

Just needs the SPM repo setup (broken link validation failure).

/azp run


[FEATURE] Sequential model deep dive notebook

Description
Do we need a deep dive notebook for sequential model?
@Leavingseason @miguelgfierro @anargyri
Expected behavior with the suggested feature

Other Comments

yes, we agreed to do this: https://github.com/microsoft/recommenders/issues/1013


Column searchable via collections

Using $table->filter(function($instance) {}) i can filter out what rows for my search, but passing it back to $instance->collection seems to be getting ignored??
            /*
            TODO: this filters out the correct rows, but doesnt pass it back to the datatable???
            $table->filter(function ($instance) use ($request, $columns) {

                foreach ($request->get('columns') as $idx => $info) {
                    if (empty(array_get($info, 'search.value', null))) {
                        continue;
                    }

                    if (array_get($info, 'searchable', false) === false) {
                        continue;
                    }
                    $key = array_get($info, 'data', null);

                    $instance->collection = $instance->collection->filter(function ($row) use ($info, $key) {
                        return str_contains(strtolower($row->$key), strtolower(array_get($info, 'search.value'))) ? true : false;
                    });

                }

            });*/

I think this has something to do with Laravel version. What versions are you using please? Will try to replicate the issue on my free time. Thanks!

Laravel 5.1

Seems to work well on demo but it uses Laravel 5.0 though.
BTW, are you sure that filtering is getting reached on your code? This code might be blocking?
if (array_get($info, 'searchable', false) === false) {
  continue;
}

@yajra nah ive had some debug happening below that, that ive seen, the filter is working properly its just not being used :S

@xLink sorry for overlooking this. I just realized that I already upgraded my demo to Laravel 5.1 and seems to work well for me. Are you still having this issue?

yup, just tried using the latest version, still on L5.1, and the filter is working, but setting $instance->collection is having no effect & the datatable is showing zero results too

@yajra any ideas on this?

@xLink, sorry but I can't replicate this issue. Looking at this Laravel 5.1 Demo, everything seems to work as expected.

okay looking at your demo some more, mebe i screwed up somewhere

hey @yajra, okay so i rebuilt the JS some, and managed to fix it :D thanks for the eg and your awesome work on this package
You can close this now


Add xd-plugin-boilerplate-react to the list

Submit your project for inclusion in XD-Awesome
Hey! We're happy you want to submit your project to XD-Awesome. Please create a pull request and ensure that you fill out the following template:
xd-plugin-boilerplate-react
Description of your project: A fork of xd-plugin-boilerplate which includes React Library, Including Babel and necessary lint rules to create plugins with User Interface easily.
Link to your project: https://github.com/Ashwinvalento/xd-plugin-boilerplate-react
License: MIT
Category: Templates
Why is it awesome?
pklaschka has done an amazing Job of creating the boilerplate code, but it lacked features for building a UI. Since React is a great way to build a great library for building User Interfaces, I've added all necessary tools to build the Plugin UI using react, Including babel and necessary linting rules for React.
Checklist:

[x] I have signed the Adobe Open Source CLA.
[x] I have read the CONTRIBUTING document.

sorry for the delay! Thanks for including your project


test: add test for scalar agg PushDownFilterThroughAggregation

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


python312Packages.flask-migrate: 4.0.5 -> 4.0.7

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.flask-migrate is: SQLAlchemy database migrations for Flask applications using Alembic
meta.homepage for python312Packages.flask-migrate is: https://github.com/miguelgrinberg/Flask-Migrate
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 4.0.6 with grep in /nix/store/g17kq7x265cllp2gdy3jjx1aqgnfss21-python3.12-flask-migrate-4.0.6
found 4.0.6 in filename of file in /nix/store/g17kq7x265cllp2gdy3jjx1aqgnfss21-python3.12-flask-migrate-4.0.6

Rebuild report (if merged into master) (click to expand)

10 total rebuild path(s)

10 package rebuild(s)

First fifty rebuilds by attrpath
fit-trackee
pgadmin4
pgadmin4-desktopmode
powerdns-admin
python311Packages.automx2
python311Packages.flask-dramatiq
python311Packages.flask-migrate
python312Packages.automx2
python312Packages.flask-dramatiq
python312Packages.flask-migrate

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/g17kq7x265cllp2gdy3jjx1aqgnfss21-python3.12-flask-migrate-4.0.6 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.flask-migrate https://github.com/r-ryantm/nixpkgs/archive/fe33a59238f2ff5dfd4b195d7a10e4e05352d6c0.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/fe33a59238f2ff5dfd4b195d7a10e4e05352d6c0#python312Packages.flask-migrate

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/g17kq7x265cllp2gdy3jjx1aqgnfss21-python3.12-flask-migrate-4.0.6
ls -la /nix/store/g17kq7x265cllp2gdy3jjx1aqgnfss21-python3.12-flask-migrate-4.0.6/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

6 packages failed to build:

pgadmin4
pgadmin4-desktopmode
pgadmin4-desktopmode.dist
pgadmin4.dist
python312Packages.flask-dramatiq
python312Packages.flask-dramatiq.dist

13 packages built:

fit-trackee
fit-trackee.dist
powerdns-admin
python311Packages.automx2
python311Packages.automx2.dist
python311Packages.flask-dramatiq
python311Packages.flask-dramatiq.dist
python311Packages.flask-migrate
python311Packages.flask-migrate.dist
python312Packages.automx2
python312Packages.automx2.dist
python312Packages.flask-migrate
python312Packages.flask-migrate.dist

Maintainer pings
cc @gador for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.flask-migrate is: SQLAlchemy database migrations for Flask applications using Alembic
meta.homepage for python312Packages.flask-migrate is: https://github.com/miguelgrinberg/Flask-Migrate
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 4.0.7 with grep in /nix/store/dnc91xnv1yy4wp5332kc0l13faij1nsm-python3.12-flask-migrate-4.0.7
found 4.0.7 in filename of file in /nix/store/dnc91xnv1yy4wp5332kc0l13faij1nsm-python3.12-flask-migrate-4.0.7

Rebuild report (if merged into master) (click to expand)

10 total rebuild path(s)

10 package rebuild(s)

First fifty rebuilds by attrpath
fit-trackee
pgadmin4
pgadmin4-desktopmode
powerdns-admin
python311Packages.automx2
python311Packages.flask-dramatiq
python311Packages.flask-migrate
python312Packages.automx2
python312Packages.flask-dramatiq
python312Packages.flask-migrate

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/dnc91xnv1yy4wp5332kc0l13faij1nsm-python3.12-flask-migrate-4.0.7 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.flask-migrate https://github.com/r-ryantm/nixpkgs/archive/73fb28244f30cd94aa3ef9129de9da983929c8d8.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/73fb28244f30cd94aa3ef9129de9da983929c8d8#python312Packages.flask-migrate

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/dnc91xnv1yy4wp5332kc0l13faij1nsm-python3.12-flask-migrate-4.0.7
ls -la /nix/store/dnc91xnv1yy4wp5332kc0l13faij1nsm-python3.12-flask-migrate-4.0.7/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

6 packages failed to build:

pgadmin4
pgadmin4-desktopmode
pgadmin4-desktopmode.dist
pgadmin4.dist
python312Packages.flask-dramatiq
python312Packages.flask-dramatiq.dist

13 packages built:

fit-trackee
fit-trackee.dist
powerdns-admin
python311Packages.automx2
python311Packages.automx2.dist
python311Packages.flask-dramatiq
python311Packages.flask-dramatiq.dist
python311Packages.flask-migrate
python311Packages.flask-migrate.dist
python312Packages.automx2
python312Packages.automx2.dist
python312Packages.flask-migrate
python312Packages.flask-migrate.dist

Maintainer pings
cc @gador for testing.

@ofborg build pgadmin4 pgadmin4.passthru.tests


[WIP]docs: create docs using rushstack

fix: build failure on it-drain
docs: attempt to create documentation using api-extractor
docs: temporarily show generated docs

check out docs at https://github.com/ipfs/helia/blob/test/new-api-docs/docs-temp/index.md
Note that api-documenter website recommends accessing these markdown files directly on github, using github pages, or with docusaurus. see "What do we do with these generated Markdown files? There are various options:" at https://api-extractor.com/pages/setup/generating_docs/

@whizzzkid @achingbrain I think this was significantly to configure than tsdoc. It also generated much better documentation by default than tsdoc.
There is also https://docusaurus.io/docs/installation#monorepos which makes it seem like adding a prettier website is possible.

trying to build to gh-pages on my fork via https://github.com/SgtPooki/helia/actions/runs/5548479221

removing gh-pages from sgtpooki fork so i can push up changes from https://github.com/ipfs/helia/pull/181

There is some weirdness on linking, e.g. https://sgtpooki.github.io/helia/interfaces/helia.internal.RmOptions.html

That link doesn't work when the docs are generated with api-documenter. I think you reviewed the old site. The rushstack one should be up again now.
However, it doesn't seem like the generated docs are getting all the types.. searching the docs folder with rg 'RmOptions' docs ends up empty.

closing this since we won't have the bandwidth to nail this down for a while.


[HUDI-416] improve hint information for cli

What is the purpose of the pull request
Right now, cli always give error information:
Command 'desc' was found but is not currently available (type 'help' then ENTER to learn about this command), but it is confused to user. We can give a hint clearly like:
Command failed java.lang.NullPointerException: There is no hudi dataset. Please use connect command to set dataset first
Brief change log
(for example:)

use exception instead of CliAvailabilityIndicator

Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
Committer checklist

[x] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@nsivabalan Please review this when you get a chance

Two minor comments. @leesf you can merge once addressed.

Big thanks for your @nsivabalan  detail review. Will merge it once addressed.

IMO, It would be better if we could hide hidden the exception trace, just keep the hint info.
Command failed java.lang.NullPointerException: There is no hudi dataset. Please use connect command to set dataset first
There is no hudi dataset. Please use connect command to set dataset first
java.lang.NullPointerException: There is no hudi dataset. Please use connect command to set dataset first
    at org.apache.hudi.cli.HoodieCLI.getTableMetaClient(HoodieCLI.java:98)
    at org.apache.hudi.cli.commands.StatsCommand.writeAmplificationStats(StatsCommand.java:72)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
    at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
    at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
    at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
    at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
    at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
    at java.lang.Thread.run(Thread.java:748)
hudi->

@lamber-ken hudi-cli mainlu for ops,  so the exception trace may is ok here.

@nsivabalan @leesf thank for your review. All is be fixed and please point out if any ignored.


Articulate converse service error

Issues

If you're having trouble with Articulate we definitely want to help you. If you can provide a little bit of information it will make our troubleshooting job easier.
Make an agent
Train agent
Test agent
Receive error
Error: I'm sorry. An error occurred calling Articulate's converse service. This is not an issue with your agent.

Please describe the issue you are having here and answer the question below.

Did you download a release or pull from Master? Which release?
latest
What operating system are you running?
centos 7.7
Are you running locally or in the cloud?
locally
Which browser are you using?

Feature Request

If you've got a feature request we want to hear it

Please desribe the feature you would like us to add.
If possible provide the use case for the feature and how you would use it.

can you provide an export of your agent so we can try to identify the problem?
How much memory does your VM have?

Also, there may be an error in the terminal that may provide us some extra info. Can you paste/upload the output of docker-compose logs api in a gist?

Yes
qa is agent
vm.zip
8GB RAM
Screen attached
qa.zip

thanks, we'll take a look.

Thank you

@aermike can you tell me what the program category was for? Did it at some point in time have sayings in it? As soon as I deleted it the error went away.
I tried to create a new agent and get it to fail in the same way by having an empty category, or a category where I had deleted stuff out of, but was unable to reproduce.

Hi
First agent's name was q&a, after see this error, I was rename the agent.
Categorii was hello, and action to respond at hello

I can rename agents just fine. :( still can't find a way to reproduce.

ok
I put vm on line again, and retest
Come back in ticket after

After reinstall is not reproduced
robot.zip

thank you for suport


Showing stack trace in debug log.

I did this for HEDAP and it helped me debug an issue that was otherwise hard to track. I thought it might be worth it adding it to NPSP.

@davidhabib ~ would you mind taking a quick look?

@jlantz why didn't this pull request generate a build?

It did build: http://ci.salesforcefoundation.org/view/feature/job/Cumulus_feature/3555/
I think it's just that @njjc just merged another pull request, and it's now building the dev-to-feature jobs.

why isn't it showing the "waiting for build xxxx to complete" message?

Because it's not correctly referencing the issue number perhaps? feature/1234-foo


Named / multiple vector support for Qdrant

The current Qdrant implementation by default only supports a single vector. By supporting named vectors we enable two highly desired features:

Different metadata embedding modes Currently we add the metadata to the chunk and then embed. Ideally we can also choose not to, or do both, and create and add vectors for each field. This might be a requirement for this issue.
Sparse vector support Sparse vectors are great for lightweight hybrid search looking for more exact matches.

See https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors.

Would such implementation be ok:

Introduce public transformers::embed::EmbedMode { WithMetadata, WithoutMetadata, Both } type.
Update transformers::embed::Embed by creating a Builder (optionally replacing with_concurrency with concurrency derive_builder generated setter), and adding embed_mode field/builder setter with a default value WithMetadata.
Update ingestion::IngestionNode to have vectors: Option<HashMap<String,<Vec<f32>>>> instead of a vector field (it will do for now, but might require some VectorType to support Sparse vectors)
Make impl BatchableTransformer for Embed::batch_transform to create embeddable chunks depending on selected EmbedMode and add embeddings vectors to the map.
Make impl Qdrant::create_index_if_not_exists to create qdrant_client::qdrant::CreateCollection struct with vectors_config field of Config::ParamsMap value containing named vectors. Its value would depend on selected EmbedMode. There could be a default vec named "WithMetadata", or "WithoutMetadata", or both vectors. Their length would come from Qdrant::vector_size (but it could as well come from impl EmbeddingModel - if I understand correctly the length of dense vecs depends on selected embed model).
Tests/examples/etc.

?
Will it be enough to configure named vectors using only an EmbedMode field in Embed struct, or will users need to configure e.g. a different Distance for each vector, so introduction of some vector_config field in Qdrant builder will be necessary?
It could not even give user control over vector names, but instead it could have some dedicated config types for each EmbedMode.
Disclaimer: after a short break from Rust I need to do some tasks as an reminder. I am unfamiliar with RAG pipelines, so I do not know e.g. how much control over vectors users may need.

Hi Pwalski, thanks for being interested and offering your help! Sounds really good.
Design wise I'd like to make sure we do the following things:

Sane and minimal config defaults, get started without a builder, then use a builder when more configuration is required. If even more is needed, build the client and users can add it to the wrapper.
See if we can get to a point where there is minimal duplication of configuration; for embedding for instance, the transformer needs to know what to embed and how and storage needs to know what to store and the dimensions; it's interesting to think about which part is responsible for what.
Modularity should be maintained such that when adding more databases and other embedding models, the storage can decide how it should persist the Node.

Introduce public transformers::embed::EmbedMode { WithMetadata, WithoutMetadata, Both } type.

👍 Perhaps naming wise we can get it more inline with what happens, something like SingleBlob, NamedFields, Both. Feels like a mouthful though, perhaps you have a better idea.

Update transformers::embed::Embed by creating a Builder (optionally replacing with_concurrency with concurrency derive_builder generated setter), and adding embed_mode field/builder setter with a default value WithMetadata.

That's an interesting view. Initially I was thinking of having the EmbedMode as a property on the IngestionNode, perhaps inheriting it from the pipeline as a global default. So then the IngestionNode owns its data, it's up to an Embedder to embed depending on the mode, and it's up to the storage to persist depending on the mode and any other config it needs. I'm very open to alternative solutions.

Make ingestion::IngestionNode to have vectors: Option<HashMap<String,<Vec>>> instead of a vector field (it will do for now, but might require changes to support Sparse vectors)

👍 I think it will work fine with sparse, as long as the dimensions can differ. How will this work if there is a single embedding? There is also a type alias for Embeddings.

Make impl BatchableTransformer for Embed::batch_transform to create an embeddable chunks depending on selected EmbedMode and add embeddings vectors to the map.

👍 Sounds good. If it's complicated to have one do both, creating a new transformer is also an option.

Make impl Qdrant::create_index_if_not_exists to create qdrant_client::qdrant::CreateCollection struct with vectors_config field of Config::ParamsMap value containing named vectors. Its value would depend on selected EmbedMode. There could be a default vector named "WithMetadata", or "WithoutMetadata", or both vectors. Their length would come from Qdrant::vector_size (but it could as well come from impl EmbeddingModel - if I understand correctly the length of dense vecs depends on selected embed model).

This is food for thought, I can imagine multiple use cases:

Querying on the aggregated metadata instead of the full document
Querying on a single metadata field
Querying on a sparse vector of a metadata field
Querying on a sparse vector that's on it's own
Which to me also raises the question if everything should be stored as is, or if we want (in the future) some way to define that genericly. Maybe it's possible to derive (part of) the schema of a node in the schema, with the option to override, i.e. wait until the first node comes through, then create a derived schema lazily. Just brainfarting here, probably all future material.

For now though, I think:
I'd prefer it if the default is a single vector (EmbedMode::WithMetadata). On EmbedMode::Without it will create named vectors for each metadata field and the text itself. For Both it does both with the metadata also embedded. Then for Qdrant, in the builder we can allow full customization, and we can provide some default or helper to make the setup a bit easier (Qdrant rust api is imo pretty rough). I.e. having a '.vector_names(vector_names: &[&str])` on the builder that takes the already given dimensions with a default distance. If users then want to fully configure it, they can provider their own implementation.

Disclaimer: after a short break from Rust I need to do some tasks as an exercise. I am unfamiliar with RAG pipelines, so I do not know e.g. how much control over vectors users may need.

Hell yeah! Sounds like you already got it under control!
Thanks again for offering your support <3. Named vectors are very high on the priority list as it will open up a lot of ways to improve search. Let me know if there is anything I can help with.

Would just like to add that the embedding transformer currently assumes that the ordering of vectors is the same as the requested text. For OpenAI I haven't seen it differently, but that might actually be a potential problem when testing if this works. There's some performance considerations here as well.

Initially I was thinking of having the EmbedMode as a property on the IngestionNode, perhaps inheriting it from the pipeline as a global default. So then the IngestionNode owns its data, it's up to an Embedder to embed depending on the mode, and it's up to the storage to persist depending on the mode and any other config it needs.

Makes sense.

I'd prefer it if the default is a single vector (EmbedMode::WithMetadata). On EmbedMode::Without it will create named vectors for each metadata field and the text itself. For Both it does both with the metadata also embedded.

The only problem I see is that it could be confusing having a vector WithMetadata sometimes as a default unnamed vector, and sometimes as a one of named vectors. Having said that I have no idea if it would be weird to store a single named vector in Qdrant, so I am fine with proposed approach.

For now though, I think:
I'd prefer it if the default is a single vector (EmbedMode::WithMetadata). On EmbedMode::Without it will create named vectors for each metadata field and the text itself. For Both it does both with the metadata also embedded. Then for Qdrant, in the builder we can allow full customization, and we can provide some default or helper to make the setup a bit easier (Qdrant rust api is imo pretty rough). I.e. having a '.vector_names(vector_names: &[&str])` on the builder that takes the already given dimensions with a default distance. If users then want to fully configure it, they can provider their own implementation.

Will do that. You can assign me to the issue.

The only problem I see is that it could be confusing having a vector WithMetadata sometimes as a default unnamed vector, and sometimes as a one of named vectors. Having said that I have no idea if it would be weird to store a single named vector in Qdrant, so I am fine with proposed approach.

Agree, it does sound weird, maybe a better name pops up when working on it. What I dislike about the with/without metadata is that it doesn't really communicate what it does, maybe something like:
enum EmbedMode {
  SingleWithMetadata,
  PerField,
  Both
}

Naming always is the hardest part 👶
I was thinking about the config api a bit. Looking at the docs, the python / TS api is actually really concise. Beats me why the Rust api isn't (audience and priorities I bet). I see no reason why the Rust api cannot be just as concise. https://qdrant.tech/documentation/concepts/collections/

Will do that. You can assign me to the issue.

<3 Let me know if there is anything I can help with

@pwalski heads up, the qdrant client had a significant update, I'll update it in swiftide asap

I noticed qdrant_client::client::QdrantClient is deprecated in favor of qdrant_client::Qdrant.
You can update version only, and I could migrate to new client.
BTW I have some problem with deciding how to configure default vectors.
Initially I wanted to change
Persist::setup(&self) -> Result<()> into Persist::setup(&self, embed_mode: EmbedMode) -> Result<()> so the pipeline could pass EmbedMode it was started with which would let Qdrant integration to configure default vectors. The problem is only Qdrant would use this param.

I noticed qdrant_client::client::QdrantClient is deprecated in favor of qdrant_client::Qdrant. You can update version only, and I could migrate to new client.
I'll update it properly asap :-)

BTW I have some problem with deciding how to configure default vectors. Initially I wanted to change Persist::setup(&self) -> Result<()> into Persist::setup(&self, embed_mode: EmbedMode) -> Result<()> so the pipeline could pass EmbedMode it was started with which would let Qdrant integration to configure default vectors. The problem is only Qdrant would use this param. EDIT: Nvm, Passing EmbedMode in Persist::setup would require storing it somewhere in IngestionPipeline, while method with_embed_mode I added only adds mapping to ingestion stream which sets node.embed_mode. Instead for now I will force user to configure EmbedMode both in pipeline, and Qdrant client.

I think that you might not need it at all in the storage. IngestionNodes will have a hashmap in vectors with <name, vector>. Maybe it can try to map that to the collection it got configured with and error if it does not match?


Fallback when mysql goes down

Has anyone figured out a logic that takes care of operations when MYSQL goes down.
Ideally if mysql is not available and we have data, we would write to a log file and then when mysql is available we would push things from the log file back to the database.
What would be the right way to do it?
If we write to a log file with all the queries and when mysql connection is restored - how do we remove the entries from the log file which got successfully fired?

Hi @astutesoftware, that question is probably too broad for the issue tracker of this module. It is definitely something you could implement using this module for the underlying connection, yes, but going into the detail of a management implementation is just beyond the scope of here, I'm sorry.
This module is just meant to be a low-level driver module, simply providing connections out when asked and shuttling the data over the protocol.

understood. meanwhile i did come up with a solution. will leave it as a note on this thread for anyone with similar needs.
On non-availability of mysql server we write the SQL statements to a taffydb persistent store.
On restoration of connectivity we push all the SQL statements in batches of X number of queries back to the mysql. Batching several queries is recommended over several individual queries fired one by one.
http://www.taffydb.com/


[camera] IllegalStateException io.flutter.embedding.engine.dart.DartMessenger$Reply in reply

Steps to reproduce
Steps to reproduce isn't so clear, this exception is reported using Sentry and judging from the exception stack, it's coming from camera plugin and it's only happening on Samsung devices.
It probably happens after using the camera.
io.flutter.embedding.engine.dart.DartMessenger$Reply in reply at line 435
io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler$1 in error at line 268
io.flutter.plugins.camera.DartMessenger in lambda$error$2 at line 198
io.flutter.plugins.camera.DartMessenger$$ExternalSyntheticLambda2 in run at line 8
android.os.Handler in handleCallback at line 958
android.os.Handler in dispatchMessage at line 99
android.os.Looper in loopOnce at line 230
android.os.Looper in loop at line 319
android.app.ActivityThread in main at line 8893
java.lang.reflect.Method in invoke
com.android.internal.os.RuntimeInit$MethodAndArgsCaller in run at line 608
com.android.internal.os.ZygoteInit in main at line 1103

It'll be nice if there's a way to catch native exception that is coming from plugins
Expected results
the exception should not happen and if it does there should be at least a way to catch native exceptions.
or we can have this following suggestion
MethodChannel(flutterEngine.dartExecutor.binaryMessenger, METHOD_CHANNEL).setExceptionHandler(

Actual results
the exception happens and it's crashing the app with no way to reproduce it
Code sample
Code sample
[Paste your code here]

Screenshots or Video

Screenshots / Video demonstration
[Upload media here]

Logs
Logs
java.lang.IllegalStateException: Reply already submitted
    at io.flutter.embedding.engine.dart.DartMessenger$Reply.reply(DartMessenger.java:435)
    at io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler$1.error(MethodChannel.java:268)
    at io.flutter.plugins.camera.DartMessenger.lambda$error$2(DartMessenger.java:198)
    at io.flutter.plugins.camera.DartMessenger$$ExternalSyntheticLambda2.run
    at android.os.Handler.handleCallback(Handler.java:958)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loopOnce(Looper.java:230)
    at android.os.Looper.loop(Looper.java:319)
    at android.app.ActivityThread.main(ActivityThread.java:8893)
    at java.lang.reflect.Method.invoke(Method.java)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:608)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)

Flutter Doctor output
Doctor output
Doctor summary (to see all details, run flutter doctor -v):
[!] Flutter (Channel [user-branch], 3.13.8, on Microsoft Windows [Version 10.0.22631.3374], locale fr-FR)
    ! Flutter version 3.13.8 on channel [user-branch] at C:\Users\oussa\flutter
      Currently on an unknown channel. Run `flutter channel` to switch to an official channel.
      If that doesn't fix the issue, reinstall Flutter by following instructions at
      https://flutter.dev/docs/get-started/install.
    ! Upstream repository unknown source is not a standard remote.
      Set environment variable "FLUTTER_GIT_URL" to unknown source to dismiss this error.
[✓] Windows Version (Installed version of Windows is version 10 or higher)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
[✓] Chrome - develop for the web
[✓] Visual Studio - develop Windows apps (Visual Studio Build Tools 2022 17.5.1)
[✓] Android Studio (version 2022.1)
[✓] VS Code (version 1.88.1)
[✓] Connected device (3 available)
[✓] Network resources

! Doctor found issues in 1 category.

We are encountering the same exception in one of our applications.

Hi @melWiss, @khalilyamoun
Which camera_android plugin version are you using? (Please find it in pubspec.lock file).
Could you also share the entire crash stack trace if any? (to see if this is similar to a case at https://github.com/flutter/flutter/issues/88598)

for the camera plugin, it's camera_android: 0.10.8+16
for the stack trace, this is all what we have:
java.lang.IllegalStateException: Reply already submitted
    at io.flutter.embedding.engine.dart.DartMessenger$Reply.reply(DartMessenger.java:435)
    at io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler$1.error(MethodChannel.java:268)
    at io.flutter.plugins.camera.DartMessenger.lambda$error$2(DartMessenger.java:198)
    at io.flutter.plugins.camera.DartMessenger$$ExternalSyntheticLambda2.run
    at android.os.Handler.handleCallback(Handler.java:958)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loopOnce(Looper.java:230)
    at android.os.Looper.loop(Looper.java:319)
    at android.app.ActivityThread.main(ActivityThread.java:8893)
    at java.lang.reflect.Method.invoke(Method.java)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:608)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)

Thanks for the update. Does this occur if you run your project in release mode? If it does, please share the entire crash stack trace that may help us investigate this further.

the problem is it didn't occur on our end, our users faced the exception and it was reported by sentry, what we can is that after taking the picture the exception happens and i tried the fix mentioned here https://github.com/flutter/flutter/issues/88598 but with no luck

https://github.com/flutter/flutter/issues/88598 is still open and needs to investigate further. I would suggest you follow up on that issue. It would be helpful if you can find a pattern to reproduce it. Closing this in favor of https://github.com/flutter/flutter/issues/88598.


Patched functions add events even if not wrapped

Once epsagon is required, it patches the event handlers, e.g. for http.request etc.
Now, if requests are made before the tracer is restarted, it will add events to the tracer, which can lead to a high memory consumption.
a solution could be to set the __epsagonCallback for all requests that should not be traced, but this looks like an implementation detail.
ideally, the API would export some way to disable the tracer until tracer.restart is called.

this was fixed with 3b44a17c96af61573130e69227a98d516ffd50d7


Add section to README about Transformers development branch

What does this PR do?

This PR adds a small section to the README to explain how to use our Transformers development branch.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Samples need to be updated to support Helm 3

Now that Helm 3 is GA, the sample apps need to be updated, such as: https://github.com/Azure/dev-spaces/tree/master/samples/BikeSharingApp. The instructions and charts, etc, do not work with Helm 3.

Dev Spaces does not yet support Helm 3, but when we do we'll be sure to update documentation and samples.


Protecting user information

Protecting user information (name, phone and address)
Just a thought. How do we plan to protect user details. Name and phone numbers are visible for everyone to see. I understand the urgency of the situation but at some point we would have to remove it from public source.
One suggestion, is to hide a request from the website the moment it is resolved.
Thus completely removing the user details (name, phone, address) from the website.

Ideally we should allow only rescuers/admins to see this information.

Authorisation and authentication is the solution from front end. Database side we can encrypt the data.

Well, I do think like creating a :--subdomain/section or page/ populating a Google sheet :-- of someone who is safe can enable people to check out who is safe and we can also manage a simple portal which can result names from this list as safe.

Probably no, as the general public are essentially the volunteers. May not be feasible at this point!

I think the best way would be to prevent scraping. Maybe limit the number of requests per minutes or something?

we will be addressing this

duplicate #308 ?


Indepently verify pip installation

@kolibril13 would you mind giving this a try?
I just renamed my project again and would like confirmation that it installs properly.
https://buckaroo-data.readthedocs.io/en/latest/
What do the docs and initial experience seem like to you?
Thanks

I did:
jan-hendrik@JanHendriksAir okapi % python3.11 -m venv .venv
jan-hendrik@JanHendriksAir okapi % source .venv/bin/activate
(.venv) jan-hendrik@JanHendriksAir okapi % pip install buckaroo
(.venv) jan-hendrik@JanHendriksAir okapi % pip install pandas
(.venv) jan-hendrik@JanHendriksAir okapi % python -m pip install jupyterlab
(.venv) jan-hendrik@JanHendriksAir okapi % python -m jupyterlab
And it works out of the box in JupyterLab
import pandas as pd

# get a pandas dataframe from the web

df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/2014_usa_states.csv')
from buckaroo.buckaroo_widget import BuckarooWidget
BuckarooWidget(df=df)  #df being the dataframe you want to explore

In VS Code I get an error:

[Open Browser Console for more detailed log - Double click to close this message]
Failed to load model class 'DCEFWidgetModel' from module 'buckaroo'
Error: No version of module buckaroo is registered
    at ph.loadClass (https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-renderers-1.0.15/out/node_modules/%40vscode/jupyter-ipywidgets8/dist/ipywidgets.js:2:4099813)
    at ph.loadClass (https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-renderers-1.0.15/out/node_modules/%40vscode/jupyter-ipywidgets8/dist/ipywidgets.js:2:4403287)
    at ph.loadModelClass (https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-renderers-1.0.15/out/node_modules/%40vscode/jupyter-ipywidgets8/dist/ipywidgets.js:2:4097773)
    at ph._make_model (https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-renderers-1.0.15/out/node_modules/%40vscode/jupyter-ipywidgets8/dist/ipywidgets.js:2:4094616)
    at ph.new_model (https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-renderers-1.0.15/out/node_modules/%40vscode/jupyter-ipywidgets8/dist/ipywidgets.js:2:4092246)
    at ph.handle_comm_open (https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-renderers-1.0.15/out/node_modules/%40vscode/jupyter-ipywidgets8/dist/ipywidgets.js:2:4091039)
    at https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-renderers-1.0.15/out/node_modules/%40vscode/jupyter-ipywidgets8/dist/ipywidgets.js:2:4402511
    at g._handleCommOpen (https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-2023.4.1011241018-darwin-arm64/out/webviews/webview-side/ipywidgetsKernel/ipywidgetsKernel.js:2:36493)
    at async g._handleMessage (https://file+.vscode-resource.vscode-cdn.net/Users/jan-hendrik/.vscode/extensions/ms-toolsai.jupyter-2023.4.1011241018-darwin-arm64/out/webviews/webview-side/ipywidgetsKernel/ipywidgetsKernel.js:2:38387)

What do the docs and initial experience seem like to you?

Cool project, I think this can be very useful!
The UI feels a bit clunky, but the logic is great.
The docs are great, but are missing an example, I'd say.
Maybe it's worth thinking of adding a interactive demo with jupyterlite/binder.
https://github.com/scikit-hep/awkward/blob/main/docs/getting-started/try-awkward-array.md
JupyterLite actually has a sphinx directive that you can try here:
https://awkward-array.org/doc/main/getting-started/try-awkward-array.html
https://github.com/scikit-hep/awkward/blob/main/docs/getting-started/try-awkward-array.md


Add DFE sign in integration for login

Context
We need to allow our users to login into our service using DFE sign in.
This PR adds the configuration to accommodate this, we find the user by the dfe_sign_in_uid. This should work for placements and claims through env variables. And for different env, production, staging, local etc.
Currently, there is no way to create a user in dfe sign in through our app so I had to create a user through the seeds file.
There is no system test for the dfe sign in path, this is because we need to refactor most of our tests as they use persona sign in, they cannot work together because the omniauth config loads only once
Dfe sign-in works with localhost even if their documentation says it doesn't 😄. There is a bit of service configuration that needs to happen in here https://dev-manage.signin.education.gov.uk, mainly the url of the app, the redirect path and the log out path
Changes proposed in this pull request

Added awesome_print Helps me a lot when debugging
Migrations to user table, dfe sign in uid and last_signed_at
omniauth.rb
Sessions controller
DFE sign in user file
Update user after sign in service
Updated the Notice banner to allow other messages beside success ones

Guidance to review

This is a bit tricky 😄
Because we don't have a way to create users in dfe sign in through our app, we need to create a user with a dfe_sign_in_uid. This uid is returned in the omniauth payload after you successfully sign in with your account on the dfe sign in form.
If a user with a dfe_sign_in_uid exists in our DB we allow the user to access our app, otherwise we redirect to sign_in_path

Screenshots
https://github.com/DFE-Digital/itt-mentor-services/assets/11318084/38c46a46-a303-44df-9699-c0affdd6a442

 DfE sign in

 Investigate how other DfE Rails apps integrate with DfE Sign-in

Code looks good. I'm reserving approval until after tomorrows demo. Placement also needs to setup their service config. Perhaps we could do that as part of the demo?

We can do it as part of the demo, what can go wrong right? :D But someone from Placements needs to pull this branch down as I don't have access to the placement side of the dfe console

 As a User, I should be able to sign into the service using DfE sign-in

 As a Support User, I should be able to sign into the service using DfE sign-in


Metadata schema validation

Based on YAML schema validation.

Resolved by 03051b50125a5303bd563ca5e4244d55a496e2c7.
In the implementation, we used JSON schema for validation.


Add HIP-745: Optionally send transaction data without required transaction fields

Problem
HIP-745 is an application level HIP which introduces a way to create transactions without the notion of throwing errors if the transaction being created does not have all the required fields before signing. The required fields would need to be in place before the transaction is signed, but the dapp should be able to send, for example, an incomplete transaction to the wallet and allow the wallet to assign other transaction properties before it presents it to the user to sign.
HIP: https://hips.hedera.com/hip/hip-745
Solution

Design the feature for the SDK
Review the design with the team
Implement the feature
Release the feature

Alternatives
No response
### Tasks
- [ ] https://github.com/hashgraph/hedera-sdk-java/issues/1556
- [ ] Review the design with the team
- [ ] Implement the feature
- [ ] Release the feature

duplicate


parsing escaped characters in strings

There are some cases where escaped characters do weird stuff.

https://github.com/elm-lang/elm-compiler/issues/1123
https://github.com/elm-lang/elm-compiler/issues/1279
https://github.com/elm-lang/elm-compiler/issues/1263

Follow along in #1374


Random - Error: The value “” is invalid.

ArgumentParser version: 1.2.3
Swift version: ~5
I have been using Argument Parser since the day 1 (since 2020) with one of my apps (published on App Store). It works perfectly. But on small number of user devices they get
Error: The value “” is invalid.

The only place I narrowed it down could be
https://github.com/apple/swift-argument-parser/blob/1.2.3/Sources/ArgumentParser/Usage/UsageGenerator.swift#L426
Where key and value should be empty/nil. But how do I get there? And why would it happen only on such a small number of users, what can cause that?
Sorry for such a vague question/bug report. I have verified with the user, they don't have anything unusual in their terminal/environment variables.
And the way I use the library:
struct ShHist: ParsableCommand {
    static var configuration = CommandConfiguration(
        commandName: "shhist",
        abstract: "A utility to save an extended history",
        subcommands: [
            ShHistVersion.self,
            ShHistInsert.self,
            ShHistSearch.self,
            ShHistSearchUI.self,
            ShHistImport.self
        ]
    )
}

extension ShHist {
    struct ShHistInsert: ParsableCommand {
        static var configuration = CommandConfiguration(
            commandName: "insert",
            abstract: "Insert history in database. Expects only one line."
        )

        @Option(name: [.short, .long], help: "set session identifier" ) var session: String = ProcessInfo.processInfo.environment["TERM_SESSION_ID"] ?? ""
        @Option(name: [.short, .long], help: "set username" ) var username: String = ProcessInfo.processInfo.environment["LOGNAME"] ?? ProcessInfo.processInfo.userName
        @Option(name: [.short, .long], help: "set hostname" ) var hostname: String = ProcessInfo.processInfo.hostName
        @Option(name: [.long], help: "set shell" ) var shell: String = ProcessInfo.processInfo.environment["SHELL"] ?? "/bin/zsh"
        @Option(help: "set exit code of inserted command", transform: {v in Int64(v)}) var exitCode: Int64? = nil
        @Option(help: "set the duration (in seconds) of inserted command",transform: {v in Int64(v)}) var durationSeconds: Int64? = nil
        @Option(help: "set the duration (in milliseconds) of inserted command",transform: {v in Int64(v)}) var durationMilliseconds: Int64? = nil
        @Flag(name: [.long], help: "verbose output") var verbose: Bool = false

        mutating func run() throws {
            // the code ...
        }

        // the code
    }
}

ShHist.main()

Unrelated, I have figure it out, similar message in the error, but a different library.


Release to master

:tada: This PR is included in version 5.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


update sample datasets

Title: Update sample datasets
Description

Category: other
JIRA issue: MIC-3990

I generated new sample data with the latest prl repo.
Testing
na

Testing should be applicable... there are tests that use the hdf files. Now they'll need paths updated.

Failing tests

https://github.com/ihmeuw/pseudopeople/blob/a30dab8701560e70d8b2e9b84c935f13a6edf337/src/pseudopeople/constants/paths.py#L11-L17


mocha.css contains :is() and broke in older browsers

According to MDN, :is() is supported since Chromium 88 and Firefox 78.
Not sure what browser versions do mocha intend to support. Our project currently targets Chromium >= 73 and Firefox >= 68, and this CSS issue prevents us from upgrading mocha library.

Not sure what browser versions do mocha intend to support.

Good question. I filed #5226 just now to make this more explicit. The only explicit version indicator I could find was that Mocha 10 supports Node 14.
Some details on Node 14:

It generally supports up through ES2020: https://compat-table.github.io/compat-table/es2016plus/#node14_6
It was released in April 21st, 2020: https://endoflife.date/nodejs > Show more unmaintained releases

For reference, equivalent browser versions end of April 2020 were:

Chrome/Chromium 81 released April 7, 2020: https://developer.chrome.com/blog/new-in-chrome-81
Firefox 75 released April 7, 2020: https://www.mozilla.org/en-US/firefox/75.0/releasenotes

I also asked in our Discord if anybody knows the explicit browser version support range of Mocha: https://discord.com/channels/1169379988978536528/1169381309144109177/1294295572852772887. cc @mochajs/maintenance-crew - surely this is documented somewhere?
For context, #5027 - there's a lot of info that's been lost to time & we need to pick up.

prevents us from upgrading mocha library.

Can you be more specific, please? What's the actual visual bug that's preventing you? It'd be helpful for us as maintainers to triage this if you could provide, for example, screenshots showing the problem and/or a way for us to quickly reproduce the issue.

Not sure what browser versions do mocha intend to support.

Good question. I filed #5226 just now to make this more explicit. The only explicit version indicator I could find was that Mocha 10 supports Node 14.
Some details on Node 14:

It generally supports up through ES2020: https://compat-table.github.io/compat-table/es2016plus/#node14_6
It was released in April 21st, 2020: https://endoflife.date/nodejs > Show more unmaintained releases

For reference, equivalent browser versions end of April 2020 were:

Chrome/Chromium 81 released April 7, 2020: https://developer.chrome.com/blog/new-in-chrome-81
Firefox 75 released April 7, 2020: https://www.mozilla.org/en-US/firefox/75.0/releasenotes

I also asked in our Discord if anybody knows the explicit browser version support range of Mocha: https://discord.com/channels/1169379988978536528/1169381309144109177/1294295572852772887. cc @mochajs/maintenance-crew - surely this is documented somewhere?
For context, #5027 - there's a lot of info that's been lost to time & we need to pick up.

Can you be more specific, please? What's the actual visual bug that's preventing you? It'd be helpful for us as maintainers to triage this if you could provide, for example, screenshots showing the problem and/or a way for us to quickly reproduce the issue.

I think saying that all the issue is :is()-related is specific enough—simply replacing all :is() with a version without it will fix. FYI the current mocha version our project use is v10.3.0, and when using the current version (v10.7.3) there is no other CSS or script error besides this.

PR provided. No matter what older browsers this project intend to support, removing :is() can at least fix the current downward compatibility issue and make the code more consistent with older existing code.


Add admin and export links to nav bar (take #2)

Coverage decreased (-0.2%) to 32.834% when pulling 24ad33c238a62d3414e2ee1041876446f7ac27a5 on admin-navitems-take-2 into 45bcd1b867d41eedac08233e31a7c3a99c36c9ef on master.

Coverage decreased (-0.2%) to 42.546% when pulling c702abb59f8f9aabf631aa366b00b72247bda98b on admin-navitems-take-2 into 0d26d64ccc88c27862e4a7893d77eb7077d0d9c5 on master.

Coverage decreased (-0.2%) to 42.546% when pulling 0a7f63fbd515839d9c0bec9582555e235222933c on admin-navitems-take-2 into 0d26d64ccc88c27862e4a7893d77eb7077d0d9c5 on master.

Coverage decreased (-0.2%) to 42.546% when pulling 5063fcc276190814e99939f582317123999a169d on admin-navitems-take-2 into 0d26d64ccc88c27862e4a7893d77eb7077d0d9c5 on master.


ssl SailthruClientError with python 2.7

Hi All,
After a bunch of successfull calls, the client calls fails with the following error. Digging this, I found that this might be an issue with the use of request module in sailthru_http_request method in sailthru_http.py.
File "../lib/python2.7/site-packages/sailthru/sailthru_client.py", line 678, in api_post     return self._api_request(action, data, 'POST')   File "../lib/python2.7/site-packages/sailthru/sailthru_client.py", line 710, in _api_request     return self._http_request(self.api_url+'/'+action, self._prepare_json_payload(data), request_type)   File "../lib/python2.7/site-packages/sailthru/sailthru_client.py", line 714, in _http_request     return sailthru_http_request(url, data, method, file_data)   File "../lib/python2.7/site-packages/sailthru/sailthru_http.py", line 47, in sailthru_http_request     raise SailthruClientError(str(e)) SailthruClientError: EOF occurred in violation of protocol (_ssl.c:581)  ERROR - EOF occurred in violation of protocol (_ssl.c:581)
Could you please look into this. We have tried this with the latest sailthru client version as well.
Thanks !!

Also some discussion here
https://github.com/kennethreitz/requests/issues/3006

I haven't been able to replicate the issue on my end, going to close it for now, please reach out if this occurs again.


Deprecate this repo

Requires migrating all of the custom styles, either by supporting them or removing them and using the appropriate CSS classes that are already defined by the ASU Web Standards WordPress Theme.

:+1:

Dependency on #1


fix: remove namespaces

We shall not specify namespaces, because otherwise it's impossible to deploy to cad and cad-stage namespaces with the different pipelines.
For dev deployments we can either do oc apply -f -n cad or set the current project as context in oc
via oc project cad.

I just needed to change one thing in the conversation script, and it would have worked :(
K


Turn off site registration feature

Add the feature of closing site registration to facilitate the management of users in the background
Add invitation registration link

You can tun off registrations in Admin -> Settings -> Login.

You can tun off registrations in Admin -> Settings -> Login.

Thank you. I saw it


Raise TypeError if key is not str

Trying to use the database with a non-str key will currently result in a KeyError. This adds a check so that it will result in a TypeError instead, so that it is clearer and does not need to make a failing request.

What is the motivation behind this? What type of object are you trying to use a key? I think that the current state is a more pythonic way of doing things. Why not check this in your program instead of in the library?

Just wanted to clarify KeyErrors when people try to put other stuff into it.

What other stuff?

ints, generally


Fix indentation of the workflow

Signed-off-by: Hugues de Valon<EMAIL_ADDRESS>

Should we specify a version of Rust to use, or is sticking to the latest OK?

Rust is stable, but I think sticking to a specific version is better in case new lints that might break CI are added. In my opinion, it is better to increase the version manually and fix all the problems in the same PR.


Update README.md

Added star and language badges for Dark Reader tool.

Thank you for PR! I'll merge and updating data.json.
Enjoy the rest of your day :D


Update troubleshoot-required-resource-access-limits.md

Removed ambiguous suggestion that customers use multiple app registrations as a workaround, since we have no supporting information or docs for that scenario.
Minor comma fix.

@henrymbuguakiarie
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


trivy-operator error download-policy

What steps did you take and what happened:
Deploying a aquasecurity/trivy-operator helm chart with version
---
apiVersion: v2
name: trivy-operator
description: trivy-operator
type: application
version: 0.22.0
dependencies:
  - name: trivy-operator
    version: 0.24.1
    repository: https://aquasecurity.github.io/helm-charts/

and using the following values file
trivy-operator:
  targetNamespaces: ""
  operator:
    namespace: "security"
    scanJobsConcurrentLimit: 2
    builtInTrivyServer: false
    builtInServerRegistryInsecure: false
  trivy:
    slow: true
    resources:
      requests:
        cpu: 500m
        memory: 800Mi
      limits:
        memory: 800Mi
        cpu: 1500m
    dbRepositoryInsecure: "true"
    debug: false
  podAnnotations:
    fluentbit.io/exclude: "true"
  resources:
    requests:
      cpu: 100m
      memory: 800Mi
    limits:
      memory: 800Mi
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: topology.kubernetes.io/region
            operator: In
            values:
            - eu-south-2
        topologyKey: "kubernetes.io/hostname"
  policiesBundle:
    insecure: true

We're getting errors in trivy-operator that failed to download policies
{"level":"error","ts":"2024-11-06T10:00:31Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: f │ │ ailed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 168.77µs, allowed: 44000/minute","stacktrace":"github.com/a  │ │ quasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runne │ │ r/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operato │ │ r/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller- │ │ runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sig │ │ s.k8s.io/controller-runtime<EMAIL_ADDRESS>│ │ controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/cont │ │ roller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}
scan-vulnerabilities pods get the following output
57475ada-1905-46a4-872e-6a76fc6688f2 2024-11-06T10:02:11Z    INFO    Adding schema version to the DB repository for backward compatibility    repository="ghcr.io/aquasecurity/trivy-db:2"                                                           │
57475ada-1905-46a4-872e-6a76fc6688f2 2024-11-06T10:02:11Z    INFO    Need to update DB
57475ada-1905-46a4-872e-6a76fc6688f2 2024-11-06T10:02:11Z    INFO    Downloading DB...    repository="ghcr.io/aquasecurity/trivy-db:2"

What did you expect to happen:
I expect to not get download policies errors and pods scan-vulnerability finish scaning, the last output that we get is "Downloading DB..."
Anything else you would like to add:
We've added the insecure flag to allow the DB to download if not we get error downloading db with TOOMANYREQUESTS
[Miscellaneous information that will assist in solving the issue.]
Environment:

Trivy-Operator version (use trivy-operator version): "0.22.0"
Kubernetes version (use kubectl version): v1.30.4-eks-a737599
OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 24.04

I am also unable to deploy trivy-operator.
Warning  FailedPullImage        10s              kubelet            Failed to pull image "ghcr.io/aquasecurity/trivy-operator:0.19.4": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: retry-after: 22.917µs, allowed: 44000/minut

I have the same problem. I was able to solve a part of it with the following settings in values.yaml to use mirrors for querying the images:
global:
  image:
    registry: "public.ecr.aws"

trivy:
  registry:
    mirror:
      "mirror.gcr.io": mirror.gcr.io
      "public.ecr.aws": public.ecr.aws
      "docker.io": docker.io
      "ghcr.io": ghcr.io

With that, trivy-operator can be deployed and will create report resources.
What remains in the logs are TOOMANYREQUESTS errors for the policiesBundle aquasecurity/trivy-checks. And I was not able to find any mirror for that.

That is good to hear. I temporarily deactivated the policiesBundle with trivy.useBuiltinRegoPolicies: "false" to get rid of the "useless" errors (and to reduce pressure on the repo).

That is good to hear. I temporarily deactivated the policiesBundle with trivy.useBuiltinRegoPolicies: "false" to get rid of the "useless" errors (and to reduce pressure on the repo).

good decission, I did the same change. but it seems it doesn't work for image scan

I saw by chance that trivy-check is available at Docker Hub. So I gave it a try, but recognized something weird:
trivy:
  useBuiltinRegoPolicies: true

policiesBundle:
  # -- registry of the policies bundle
  registry: docker.io
  # -- repository of the policies bundle
  repository: aquasecurity/trivy-checks

But in the trivy-operator logs it seems that the change in registry is ignored:
{"level":"error","ts":"2024-11-21T14:40:49Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 363.439µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/operator.(*TTLReportReconciler).applicableForDeletion\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/operator/ttl_report.go:152\ngithub.com/aquasecurity/trivy-operator/pkg/operator.(*TTLReportReconciler).DeleteReportIfExpired\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/operator/ttl_report.go:105\ngithub.com/aquasecurity/trivy-operator/pkg/operator.(*TTLReportReconciler).SetupWithManager.(*TTLReportReconciler).reconcileReport.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/operator/ttl_report.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-11-21T14:40:50Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 482.171µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func2\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-11-21T14:40:53Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 35.881µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func2\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-11-21T14:40:54Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 406.265µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Hash\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:114\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func2\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:208\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-11-21T14:40:54Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:cba49b6781cfcdeb6b063283a711ce0ddb1f36d6e2a5db69ef7d2e3f13998149: TOOMANYREQUESTS: retry-after: 239.849µs, allowed: 44000/minute","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).loadPolicies\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:144\ngithub.com/aquasecurity/trivy-operator/pkg/policy.(*Policies).Eval\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/policy.go:199\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.evaluate\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/helper.go:45\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*ResourceController).SetupWithManager.(*ResourceController).reconcileResource.func2\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/resource.go:229\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}

@pfaelzerchen thanks for the report.
Could you recheck with a correct repository aquasecurity/trivy-checks:1?
  repository: aquasec/trivy-checks

I can confirm that it is fixed in V0.23.0. Everything works fine by default now.

Hi,
thanks, I also confirm that's working now.


Add strMap

I regularly need to encode a Map String a (or, equivalently, Map k a where k can be printed / parsed as a string). There's no easy way to do this right now besides converting to a Foreign.Object, but it's nice to skip the ceremony if I can simply convert the keys to/from strings as needed but retain a map structure.

I also frequently want to work with a map that has some type as the key that can be serialized to a string, so it should be represented in JSON as an object, which would perhaps work out to something like this:
strMap :: forall k a. Ord k => String -> (String -> Maybe k) -> (k -> String) -> JsonCodec a -> JsonCodec (Map.Map k a)
strMap ty parse print valueCodec = Codec.basicCodec decode encode
  where
  encode :: Map k a -> Json
  encode m = Codec.encode CA.jobject $ Object.runST do
    obj <- Object.ST.new
    forWithIndex_ m \k v -> Object.ST.poke (print k) (Codec.encode valueCodec v) obj
    pure obj

  decode :: Json -> Either CA.JsonDecodeError (Map k a)
  decode json = do
    array :: Array _ <- Object.toUnfoldable <$> Codec.decode CA.jobject json
    parsed <- array # traverse \(Tuple k v) -> do
      key <- note (CA.Named k (CA.TypeMismatch ty)) (parse k)
      val <- lmap (CA.AtKey k) (Codec.decode valueCodec v)
      pure $ Tuple key val
    pure $ Map.fromFoldable parsed

I thought I was onto something for this that would be more flexible and handle both cases here as well as foreign objects.
The general idea being:

you'd have a codec type that represents things with key/value entries (and functions that create codecs of this type for Map and Object) - this KVCodec type would be similar to the existing JPropCodec but needs a little more flexibility, it would probably replace JPropCodec for the existing usages also.
a codec that can remap the key part of a KVCodec so you could handle the non-String-key Map case you outlined above, where they can still be represented as Strings
two functions that accept KVCodec that dictate the final JSON representation (object being restricted to accepting KVCodec String a, list of tuples would accommodate KVCodec k a in general)

But I'm not quite satisfied with it yet. If I can't get it figured out tonight I'll just merge this 🙂

No rush! I'm using this strMap implementation in a codebase for now, but I'll tear it out if we come up with something nicer.


feat(oui-progress): new component

Signed-off-by: Kevin Manson<EMAIL_ADDRESS>

Here is a screenshot of this feature.