Make Docker image run as non-root.

There's still a lot of duplication going on because we're having to do this under time pressure without having sorted out our base images. These Dockerfiles will look much better once we can factor out the common stuff.

Use Debian 11 (bullseye) because it makes it way easier to install compatible versions of Node.js and Yarn that are still supported (for now).
Make the bulk upload directory configurable via an environment variable instead of hardcoding it to write to the same place that the app runs from. Defaults remain the same.
Fix a bunch of lints (found using hadolint, which is too rough around the edges to enable as a pre-commit/pre-merge but somewhat useful if taken with a pinch of salt).
Use the right version of bundler. Turns out the feature where it tries to reinstall the right version of itself doesn't always do the right thing. (App fails at runtime because the default bundler is still the wrong version.)
Avoid installing perl during build. (We don't need it.)
Have Sidekiq log to stdout.

Also contains all the same fixes as alphagov/feedback#1432:

Add some files to .dockerignore.
Don't run as root by default.
Set RAILS_LOG_TO_STDOUT because we always want that when running in a container.
Symlink /app/tmp and /home/app to /tmp so that gems don't try to write to the (read-only) image filesystem. (Ruby/Rails doesn't seem to have a consistent convention for telling gems where to write temp files.)
Run rails assets:precompile in the build stage.
Avoid installing "suggested" and "recommended" dependencies of Debian packages.
Avoid installing gem documentation.
Avoid including the APT cache in the output image.
Increase build parallelism (saves a couple of minutes on a 2019 MBP).

https://trello.com/c/Jef1t4xU
Tested: docker build -t whitehall && docker run -it --rm -e GOVUK_APP_DOMAIN=x -e GOVUK_WEBSITE_ROOT=x whitehall
Rollout: this app is push-on-green and test coverage is sketchy, so look out when merging.

Still haven't been able to figure out why publishing-e2e-tests Uploading an attachment on Whitehall Uploading an attachment on Whitehall is failing. Can't get it to log anything useful.

The test failure looks like this, but I'm pretty sure it's failing because something's going wrong with the file upload. Damned if I can get it to log anything useful from either the worker or the -admin process.
  1) Uploading an attachment on Whitehall Uploading an attachment on Whitehall
     Failure/Error: check "Test taxon"

     Capybara::ElementNotFound:
       Unable to find checkbox "Test taxon" that is not disabled
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/finders.rb:303:in `block in synced_resolve'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/base.rb:77:in `synchronize'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/finders.rb:292:in `synced_resolve'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/finders.rb:53:in `find'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/actions.rb:367:in `block in _check_with_label'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/base.rb:83:in `synchronize'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/actions.rb:366:in `_check_with_label'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/actions.rb:150:in `check'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/session.rb:762:in `check'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/dsl.rb:53:in `call'
     # /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/dsl.rb:53:in `check'
     # ./spec/whitehall/uploading_attachment_spec.rb:30:in `given_i_have_a_draft_document_with_attachment'
     # ./spec/whitehall/uploading_attachment_spec.rb:9:in `block (2 levels) in <top (required)>'

I think it's very likely that it's the removal of this line which "fixes" the e2e tests and the base image is probably a red herring.

I suppose it could be, but on multiple tries and tests - both User and Image was required to be updated for the test to pass.

So the e2e test fails with the full-fat image when running as non-root. The failure is the same as before. Unfortunately that means there's a real bug in this PR; we can't just keep the e2e test running as root and pretend there's no problem, because then we'll break production.
Next step is to (continue to) figure out what's really failing when it's running as non-root. Which is easier said than done, but it's very likely to be some filesystem permissions / path issue relating to uploading assets.

I don't know why the other tests are failing since the rebase, but I'm pretty sure that's unrelated. Also seemed kinda flaky with a couple of different tests failing each time, but hasn't completely passed at all in four runs so who knows.

At least the other testsuite is now passing again (didn't change anything - must have been something else that broke it).
I'm starting to suspect that the e2e test failures are specific to the e2e environment after all, since I haven't been able to get asset upload to fail when running this branch under govuk-docker. I'm able to manually follow along with the failing e2e test scenario but under govuk-docker it successfully creates the edition and uploads the asset (and I see the page containing the taxonomy tags string that the test looks for.
Also the error in the docker.log from the e2e test artefacts contains this, which kinda sounds like /tmp isn't writable in the e2e whitehall_admin container, somehow (I've no idea why that should be yet though) :
whitehall-admin_1                       | 2022-07-11T15:45:14.703245782Z system temporary path is not writable: /tmp
whitehall-admin_1                       | 2022-07-11T15:45:14.703270483Z /tmp is not writable: /tmp
whitehall-admin_1                       | 2022-07-11T15:45:14.703274315Z . is not writable: /app

looks good, I don't know if you want a ruby developer to have a look too. Great work.

@theseanything could you have a quick look over my Ruby changes here, just in case I'm doing anything whacky? thanks! 🙏

Rebased against origin/main to update and to clean up the commit history.


Genshin won't start

Describe the bug
Launcher opens fine, but when i click to launch, it loads while saying (the game is running), then after stops. i dont know if there are any screenshots for it but it just wont start
macOS: [Sonoma Version 14.4.1]
Device: [MacBook Air 13" 2020]
CPU: [Apple M1 (i cant find anymore info on the cpu)]
RAM: [8GB]
Additional context
Weird other thing but the app logo background is white now, normally transparent.

If it's related to the launcher, upload neutralinojs.log in Yaagl Data Directory  (i still dont know how to do this)

It's written in the readme of the main page, needs Sonoma 14.4 for ARM.

Sorry about that GitHub didn't update the edit, this seems to be a Rosetta 2 issue, go to Terminal and type -
softwareupdate --install-rosetta
Follow through the prompts and see what happens.

softwareupdate --install-rosetta

it not help, still with the problem.

it not help, still with the problem.

Any errors it is putting out?

it not help, still with the problem.

Any errors it is putting out?

1st off, how do i do your first request, im extremely bad at computer stuff so if you can be as detailed as possible that would be super awesome.
and 2nd, im not getting any errors, it literally just wont launch.

it not help, still with the problem.

Any errors it is putting out?

OH SORRY, i get what your first reply meant now. I installed rosetta and it still isnt working. after installing i also reset my computer and force quit installer. Also checked file integrity, nothing working so far.
im gonna try deleting yaagl and redownloading though.

its all good I will be more detailed
Did u delete Yaagl OS folder in /Applications Support?
You can Locate the folder in Library(it is a hidden folder)
when you are on a Finder window > Press "Go" on the left Menu bar > Then press "Go to Folder" > Type Library .
Locate a Folder called "Applications support" > inside that folder should be Yaagl OS.

its all good I will be more detailed
Did u delete Yaagl OS folder in /Applications Support?
You can Locate the folder in Library(it is a hidden folder)
when you are on a Finder window > Press "Go" on the left Menu bar > Then press "Go to Folder" > Type Library .
Locate a Folder called "Applications support" > inside that folder should be Yaagl OS.

i did that now, just redownloaded it after fully wiping yaagl off my laptop. finally at least getting an error message .
{"code":"NE_FS_FILRDER","message":"Unable to open file: /Applications/GenshinImpact_Data/globalgamemanagers"}
cant pick a file

Guys, remember to just restart the pc. it helps

Guys, remember to just restart the pc. it helps

i did a few times, no clue why it wont work

what are you opening it with?
crossover, wine, whisky?

its all good I will be more detailed
Did u delete Yaagl OS folder in /Applications Support?
You can Locate the folder in Library(it is a hidden folder)
when you are on a Finder window > Press "Go" on the left Menu bar > Then press "Go to Folder" > Type Library .
Locate a Folder called "Applications support" > inside that folder should be Yaagl OS.

i did that now, just redownloaded it after fully wiping yaagl off my laptop. finally at least getting an error message .
{"code":"NE_FS_FILRDER","message":"Unable to open file: /Applications/GenshinImpact_Data/globalgamemanagers"}
cant pick a file

Make sure Yaagl has the needed access and game is in a single folder not spread across ur home

what are you opening it with? crossover, wine, whisky?

the default, i haven't tried anything different for it. everything is just the default

its all good I will be more detailed
Did u delete Yaagl OS folder in /Applications Support?
You can Locate the folder in Library(it is a hidden folder)
when you are on a Finder window > Press "Go" on the left Menu bar > Then press "Go to Folder" > Type Library .
Locate a Folder called "Applications support" > inside that folder should be Yaagl OS.

i did that now, just redownloaded it after fully wiping yaagl off my laptop. finally at least getting an error message .
{"code":"NE_FS_FILRDER","message":"Unable to open file: /Applications/GenshinImpact_Data/globalgamemanagers"}
cant pick a file

Make sure Yaagl has the needed access and game is in a single folder not spread across ur home

Also mentioning @sineoxan as they seem to know about wine (i have no clue how to work wine.)
i just realized that these are my only wine options. normally its a LOT. idk what this is

For me specifically I have a few options as well. Personally I want to run crossover as my emulator but the option won't pop up. I am currently running genshin on a version shown below

For me specifically I have a few options as well. Personally I want to run crossover as my emulator but the option won't pop up. I am currently running genshin on a version shown below
its very finniky not being able to alt tab and change windows through my first and second monitor. I hope this is just a me problem and that you won't encounter it when you boot it up. (very funky but this wine version works best for me)

AH THANK YOU!! the game actually started and is continually staying on. Im going to download it all and if anything happens ill mention it, but if nothing happens ill close this. Thank you guys so so much for helping


Take min(popularity, authority) for boosting purposes

Child of https://github.com/creativecommons/cccatalog/issues/431
Description
Boosting entire sources as a bloc is not effective, as it results in wiping out gains from popularity boosting. Instead we should only boost by the minimum of either popularity or authority.

We're indexing max and min of (popularity, authority) for ranking experiments; whether it is useful is yet to be seen


Similar project

FYI, strsim is a similar project with a few more algorithms (I'm not the author). 'distance' is a far better name. I've struggled to remember 'strsim' every time I wanted to use it!

Oh..and i thought i finally found a useful first crate to publish /:
Thanks for the info though ;)

There's room for more than one crate that does this :) Even if you just re-export the other algorithms form strsim with a unified API, it would help as your crate is far easier to find.

Yeah, you're right.
But i think i'm going to reinvent the wheel and implement the algorithms on my own. New ones will follow soon ;)

Still awesome. Implementing cool algorithms in Rust never hurt anybody (hopefully)!


Test failure when building with Python v 3.5

Running scipy.test() yields an assertion error under Python3.5. It looks like a new DeprecationWarning preempted the expected OptimizeWarning. Am I correct to assume that this is probably nothing to worry about?
Cython, numpy, nose installed with conda, scipy built from source. OS X 10.10, clang v 6.1.0 (clang-602.0.53), gfortran 5.1.0, FFLAGS=-ff2c
FAIL: test_indeterminate_covariance (test_minpack.TestCurveFit)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/lwoodscn/anaconda/envs/scipy_dev/lib/python3.5/site-packages/scipy/optimize/tests/test_minpack.py", line 397, in test_indeterminate_covariance
    _assert_warns(OptimizeWarning, curve_fit, lambda x, a, b: a*x, xdata, ydata)
  File "/Users/lwoodscn/anaconda/envs/scipy_dev/lib/python3.5/site-packages/numpy/testing/utils.py", line 1603, in assert_warns
    "%s( is %s)" % (func.__name__, warning_class, l[0]))
AssertionError: First warning for curve_fit is not a <class 'scipy.optimize.optimize.OptimizeWarning'>( is {message : DeprecationWarning('inspect.getargspec() is deprecated, use inspect.signature() instead',), category : 'DeprecationWarning', filename : '/Users/lwoodscn/anaconda/envs/scipy_dev/lib/python3.5/site-packages/scipy/optimize/minpack.py', lineno : 545, line : None})```

This is mostly harmless and is going to be fixed by either gh-4928 or gh-5248, whichever lands first.

Good to hear. I figured as much.

On Oct 6, 2015, at 1:53 PM, Evgeni Burovski<EMAIL_ADDRESS>wrote:
This is mostly harmless and is going to be fixed by either gh-4928 or gh-5248, whichever lands first.
—
Reply to this email directly or view it on GitHub.

gh-5248 now merged, so this is fixed.


i ｢wds｣: 404s will fallback to //index.html

🐞 Bug report
Command (mark with an x)

- [ ] new
- [ ] build
- [x ] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Is this a regression?
I don't have any issue in angular 7 with cli 7.3.8 i guess.

 Yes, the previous version in which this bug was not present was: ....
Description
I just added two routes and angular theme css in my project. Now i encountered to this error when i run ng serve.
 A clear and concise description of the problem...
🔬 Minimal Reproduction
app-routing.module.ts
const routes: Routes = [
  {path: '', redirectTo:'/login', pathMatch:'full'},
  {path: 'login', loadChildren: () => import('../auth/auth.module').then(m => m.AuthModule)}
];

auth-routing.module.ts
const authRoute: Routes = [
    {path:'', component: LoginComponent}
]

base href in index.html bydefault
 <base href="/">

🔥 Exception or Error

ng serve
 10% building 3/3 modules 0 activei ｢wds｣: Project is running at http://localhost:4200/webpack-dev-server/
i ｢wds｣: webpack output is served from /
i ｢wds｣: 404s will fallback to //index.html
chunk {auth-auth-module} auth-auth-module.js, auth-auth-module.js.map (auth-auth-module) 279 kB  [rendered]
chunk {main} main.js, main.js.map (main) 16.4 kB [initial] [rendered]
chunk {polyfills} polyfills.js, polyfills.js.map (polyfills) 251 kB [initial] [rendered]
chunk {runtime} runtime.js, runtime.js.map (runtime) 8.94 kB [entry] [rendered]
chunk {styles} styles.js, styles.js.map (styles) 1.95 MB [initial] [rendered]
chunk {vendor} vendor.js, vendor.js.map (vendor) 4.16 MB [initial] [rendered]
Date: 2019-07-29T05:43:05.908Z - Hash: d30fdab03b1f2a3e4fa3 - Time: 20301ms
** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
i ｢wdm｣: Compiled successfully.

🌍 Your Environment

ng version

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/

Angular CLI: 8.1.2
Node: 10.16.0
OS: win32 x64
Angular: 8.1.3
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.801.2
@angular-devkit/build-angular     0.801.2
@angular-devkit/build-optimizer   0.801.2
@angular-devkit/build-webpack     0.801.2
@angular-devkit/core              8.1.2
@angular-devkit/schematics        8.1.2
@angular/cli                      8.1.2
@ngtools/webpack                  8.1.2
@schematics/angular               8.1.2
@schematics/update                0.801.2
rxjs                              6.4.0
typescript                        3.4.5
webpack                           4.35.2

Anything else relevant?

similar issue:

https://stackoverflow.com/questions/56954866/can-any-one-help-me-to-understand-this-message-in-cmd-when-run-ng-serve-angular

This is no bug. It is an automatic created failback, if the url is unknown.
Do you have any issues?

Hi @indraraj26, The message you are seeing is for informational purposes as explained in the above linked SO issue.
Thanks.


Respect a .nvmrc file

I'm not sure if this already exists, but can nodenv support .nvmrc files in the same way it supports a .node-version file - it may allow for easier migrations in the future.

Bump?

@joelzimmer @jasonkarns have a look at https://github.com/ouchxp/nodenv-nvmrc. I was after the same thing and found that plugin.

closing with @ouchxp's plugin as the solution.


Invalid url for method GoogleApi.Storage.V1.Api.Objects.storage_objects_insert

I believe it should be /upload/storage/v1/b/{bucket}/o instead of /storage/v1/b/{bucket}/o
https://github.com/googleapis/elixir-google-api/blob/dac4877b5da2694eca6a0b07b3bd0e179e5f3b70/clients/storage/lib/google_api/storage/v1/api/objects.ex#L432

You actually want to use storage_objects_insert_simple which has the correct URL and can specify the uploadType parameter.

So the storage_objects_insert function has incorrect URL and is not working? Or is it somehow intended for different use cases? I am a bit confused.
I am still not sure how to address the #1197
As a workaround, I am using a modified fork of this repo as a local dependency with the upload URL prefix, which is working OK for me.
There was no update after the GCP Next conference as @dazuma said in my issue, so I am not sure which approach to choose.
Thanks a lot for directions.

You actually want to use storage_objects_insert_simple which has the correct URL and can specify the uploadType parameter.

storage_objects_insert_simple() is designed to work with file names only while storage_objects_insert() allows you to upload a string from memory, which is valuable. Please fix the url with the next regen.

@chingor13 I believe the problem is a disconnect between the REST specification as returned by the discovery API and the underlying API implementation.
TL;DR
The storage API needs to accept POST /b/{bucket}/o as a valid path and, ideally, use uploadType to determine which protocol to use (simple, multipart, and resumable) rather than having multiple paths that map to different upload types (simple and multipart use the same path but resumable uses a different one). But because it will likely take a long time for the team managing that API to make the necessary fixes, I have two possible solutions for the interim:

It's not my ideal solution, but it would be rather simple to look for an id as a key in a configuration map (e.g. storage.objects.insert) and merge those overrides on top of the discovered metadata. This would require us to maintain a configuration map somewhere but it allows us to alter the metadata used to generate code without messing up the actual discovery or code generation components, i.e. if no overrides exist then behavior would remain exactly as it is now, and if overrides do exist then the behavior still remains the same except the input is transformed before generation starts.

A perhaps more dynamic solution would be to allow practically any value to be overridden from an optional keyword list or through application configs. For example, the opts keyword list could allow for things like base_path, path_prefix, and/or library_version that would override the default values set by the generator.

Ultimately it is a matter of whether we want the output (i.e. client code) to remain the same (option 1) or to change to provide more dynamic behavior (option 2).
The Whole Story
I understand these clients are being generated from a Swagger API specification, and looking at the specification I see the following:
"insert": {
  "id": "storage.objects.insert",
  "path": "b/{bucket}/o",
  "httpMethod": "POST",
  "..."
}

The official API reference says that path should theoretically be correct:

POST
https://www.googleapis.com/upload/storage/v1/b/bucket/o
and
POST  /b/bucket/o

But the error message returned by the server says the latter version (/b/bucket/o) is not valid:

"message": "Upload requests must include an uploadType URL parameter and a URL path beginning with /upload/"

Also, uploadType does not appear to be a required query parameter despite the specification and error message both mentioning that it is. The endpoint generation code will manually add uploadType as a required query parameter for the simple and resumable protocols, but the basic endpoint does not have uploadType set as a required parameter in the specification; only bucket is a required parameter. In fact, the specification doesn't have uploadType in the parameter list at all, not even as an optional parameter, so you can't set it in storage_objects_insert/4 even if you wanted to. Despite this, the call still succeeds as long as the path is correct.
Ultimately, Google is not respecting their own API specification (they fail on the resource path listed in their own specification) and have misleading error messages (they communicate rules about valid requests that aren't completely respected by the server). Either the specification needs to be corrected, the implementation needs to be corrected, or both need to be adjusted to bring it to a happy medium. Barring that, the generator might need hard-coded logic to create a workaround for this and other instances where the specification and implementation do not agree.
In case there's any disagreement on whether the specification and implementation do not agree, or whether we should be "hard-coding" solutions to this problem, I would like to redirect any interested parties to the base service object for Ruby. If you look at lines 105 and 131, you can see that the non-generated BaseService class definition has an "override" attribute called upload_path that is specifically aimed at modifying the path for media uploads. The value is hard-coded to use the /upload/ prefix. So clearly this problem exists in other versions of the client libraries but are being overridden manually to adjust for the path discrepancy.

I did some digging here, and the issue is both a bit simpler and more complicated. Simpler in that we've done away with Swagger and are now generating directly from the source Discovery document, which does in fact have the upload URLs available (so we should be able to fix this without requiring user override of the path.) But more complicated in that we can't just change the URL for storage_objects_insert since it's being generated for a different case (metadata-only upload—and just to make it more confusing, Storage doesn't actually support this operation, unlike other similar upload APIs such as Drive and YouTube.) To actually upload data, the intent was to generate an auxiliary function storage_objects_insert_simple, as @chingor13 points out. Unfortunately, as you point out, this function as written requires you to upload from a file.
So what we're going to do is generate a second auxiliary function storage_objects_insert_iodata. This will be the same as storage_objects_insert_simple except it will take an iodata (i.e. either a raw string/binary, or an iolist) instead of a filename. A PR is open for this change; we'll try to get the storage client regenerated with the addition later this week.

The google_api_storage package should be released by Monday. Use the storage_objects_insert_iodata function.

I did some digging here, and the issue is both a bit simpler and more complicated. Simpler in that we've done away with Swagger and are now generating directly from the source Discovery document, which does in fact have the upload URLs available (so we should be able to fix this without requiring user override of the path.) But more complicated in that we can't just change the URL for storage_objects_insert since it's being generated for a different case (metadata-only upload—and just to make it more confusing, Storage doesn't actually support this operation, unlike other similar upload APIs such as Drive and YouTube.) To actually upload data, the intent was to generate an auxiliary function storage_objects_insert_simple, as @chingor13 points out. Unfortunately, as you point out, this function as written requires you to upload from a file.
So what we're going to do is generate a second auxiliary function storage_objects_insert_iodata. This will be the same as storage_objects_insert_simple except it will take an iodata (i.e. either a raw string/binary, or an iolist) instead of a filename. A PR is open for this change; we'll try to get the storage client regenerated with the addition later this week.

This is still not working.,
Whats the deal?? some of us actually want to use the libraries you guys spend so much time working on.
this path issue still hasn't been figured out?
like... why is this issue Closed? The issue still exists!!

@ESearcy What issue are you referring to? "Not working" is not a helpful bug report.
If you're talking about #3407, please comment there, or on the PR #3638 that is open to address it.
If you're not talking about #3407, please open a new issue and explain what your problem is.
Thanks.

I think what the previous person is pointing out is that the function storage_objects_insert is still defined and is still using the incorrect url. So even though there is a different function to use, it would stand to reason that the function that doesn't work should be fixed or removed.


[v1.2][ISSUE-374] deltatuner denas-ssf multiple model validation and model merge

What changes were proposed in this pull request?

refine the code to support multi LLM models for ssf, and update the readme
refactor the code to support easy-to-use model merge utility for ssf

Why are the changes needed?

To prove the deltatuner's compatible with multiple models
To improve the user experience

How was this patch tested?

deltatuner ci/cd has covered this PR's change

@XinyaoWa @tianyil1 Please review this PR. Feel free to leave your comments.

@zhouyu5 , I will merge this PR. BTW, reading through comments, I think there are some design changes happened, can you also wrap up a design doc or point me existing design doc, so I can understand the workflow and class diagram from there?

@zhouyu5 , I will merge this PR. BTW, reading through comments, I think there are some design changes happened, can you also wrap up a design doc or point me existing design doc, so I can understand the workflow and class diagram from there?

Actually, the API keep unchanged, just some small internal changes. We don't have a design doc yet, but we will begin to prepare a design doc. Please stay tuned.


updates local executor

Updates local executor to address dependency vulnerabilities.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Michael Namoff seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Update ui-components to newest version

in https://github.com/ManageIQ/manageiq-ui-classic/pull/26 we need newest version of ui-components, so update it in bower.

@miq-bot add_label ui, enhancement, euwe/no


Easy dev environment setup with Gitpod

Is your feature request related to a problem? Please describe.
This problem is actually not faced by the consumers of this package, but the contributors to it.
Cloning a repo, installing tools, and setting up shop for development is always tedious. Additionally, dealing with issues that only happen on one version of one OS is a pain. Both of these problems only get worse over time as more people contribute and the tool and the code gets bigger.
Describe the solution you'd like
I think we can give future maintainers of this repo an easier setup and development experience by adding a few files to support Gitpod. They of course wouldn't need to ship with the package. This is purely a nice-to-have that helps significantly speed up the onboarding, development, and review process.
Describe alternatives you've considered

Supporting GitHub CodeSpaces instead, which is basically the same product but closed-source and proprietary.

Sorry for the delay.
I do not have any experience of this and therefore can't see the benefits this would bring.


chore: use t.Errorf(...) instead of t.Error(fmt.Sprintf(...))

@testwill thx for your contribution. Is there a concrete reason for this change since I can't find an issue? If it's about performance improvements, please also file a test benchmark.

@testwill please see comment: https://github.com/cloudevents/sdk-go/pull/908#issuecomment-1618167395
and a rebase is needed


Observables

So far, when taking expectation values, I convert to quimb operators manually.  For instance, given a CircuitMPS:
mps.local_expectation(quimb.pauli('Z') & quimb.pauli('Z'), (3, 4)).real.item()

I wonder if there's a better way, or a way we could enable users to use SparsePauliOp and its successor and have the conversion to quimb be automatic.

I dug up this function I wrote at some point, but I don't remember how I used it. Not sure if it's useful.
def quimb_paulis(pauli_list: PauliList):
    result = []
    for qiskit_pauli in pauli_list:
        quimb_pauli_list = []
        for i, p in enumerate(reversed(qiskit_pauli.to_label())):
            if p != "I":
                quimb_pauli_list.append((quimb.pauli(p), i))
        if quimb_pauli_list:
            paulis, qubits = zip(*quimb_pauli_list)
            pauli = functools.reduce(lambda a, b: a & b, paulis)
        else:
            pauli = 1.0
            qubits = []
        result.append((pauli, qubits))
    return result

OK, that is probably going to be useful to reference.


How to see flow of a chatterbot and queries that are hitting database ?

How to see flow of a chatterbot and queries that are hitting database ?

By adding a logging config to your bot, here is a sample snippet
# -*- coding: utf-8 -*-
from chatterbot import ChatBot

# Uncomment the following lines to enable verbose logging
import logging
logging.basicConfig(level=logging.INFO)

# Create a new instance of a ChatBot
chatbot = ChatBot(
    'SQL Example',
    trainer='chatterbot.trainers.ChatterBotCorpusTrainer'
)

chatbot.train(
    "chatterbot.corpus.english.ai"
)

# Get a response to the input text 'How are you?'
response = chatbot.get_response('What is AI?')

print(response)

@ayushecps What type of database are you using? There are a number of excellent database debugging tools out there, many of them support the functionality that you are describing.


BLD: fix include order, Python.h before standard headers

Python.h must always come before any standard headers, or it may result in warnings like:
/opt/hostedtoolcache/Python/3.9.6/x64/include/python3.9/pytime.h:153:60: warning: ‘struct timespec’ declared inside parameter list will not be visible outside of this definition or declaration
  153 | PyAPI_FUNC(int) _PyTime_FromTimespec(_PyTime_t *tp, struct timespec *ts);
      |                                                            ^~~~~~~~
/opt/hostedtoolcache/Python/3.9.6/x64/include/python3.9/pytime.h:158:56: warning: ‘struct timespec’ declared inside parameter list will not be visible outside of this definition or declaration
  158 | PyAPI_FUNC(int) _PyTime_AsTimespec(_PyTime_t t, struct timespec *ts);

The warning may not show up for all build systems; it is visible in the Meson Linux CI job.

These are the last changes needed to make a build with -Werror pass (xref https://github.com/rgommers/scipy/pull/60).

Hmm, there are 3 failures even:
________________ test_cosine_cdf[-3.1415-2.109869685443648e-14] ________________
[gw1] linux -- Python 3.8.0 /usr/bin/python3.8
scipy/special/tests/test_cosine_distr.py:42: in test_cosine_cdf
    assert_allclose(_cosine_cdf(x), expected, rtol=5e-15)
E   Not equal to tolerance rtol=5e-15, atol=0
E
E   Mismatch: 100%
E   Max absolute difference: 6.86510247e-29
E   Max relative difference: 1.38508612e-10
E    x: array(4.956444e-19)
E    y: array(4.956444e-19)
        expected   = 4.956444476505336e-19
        x          = -3.14159
__________ test_cosine_cdf[-3.141592653589793-4.871934450264861e-50] ___________
[gw1] linux -- Python 3.8.0 /usr/bin/python3.8
scipy/special/tests/test_cosine_distr.py:42: in test_cosine_cdf
    assert_allclose(_cosine_cdf(x), expected, rtol=5e-15)
E   AssertionError:
E   Not equal to tolerance rtol=5e-15, atol=0
E
E   Mismatch: 100%
E   Max absolute difference: 3.41274945e-49
E   Max relative difference: 7.0049166
E    x: array(3.899943e-49)
E    y: array(4.871934e-50)
        expected   = 4.871934450264861e-50
        x          = -3.141592653589793

There's something quite funky going on with that code. I already had another problem with it when just switching the includes into what should have been the correct order:
/usr/include/x86_64-linux-gnu/bits/mathcalls.h:79:1: error: expected ‘;’ before ‘extern’
   79 | __MATHDECL_VEC (void,sincos,,

That's why the _cosine.c change is the only one that's different (I explicitly added the Python.h include at the top).
The errors are still small, so I'll look into them; maybe the reference values are just derived from the implementation rather than being known good.

The code only uses sin, cos and pow. The only problem seems to be 32-bit Linux, and the problem is minor. I'm inclined to bump the tolerance for 32-bit for the first two tests and mark the third one as xfail. It looks like some obscure problem where pymath.h doesn't get along with glibc-specific optimizations in math.h.

@WarrenWeckesser you wrote that code not too long ago, so if you could have a quick look and see if you have any concerns, that would be great.

Okay, removed the problematic change to _cosine.c to make this a straightforward merge. Can for now explicitly silence the warning to make a -Werror CI job work.

Yeah this looks good now so I'll put it in, thanks Ralf.
No idea about the special stuff--maybe Josh or Warren can check it eventually.

@rgommers and @tylerjereddy thanks.
For the failure of the cosine CDF that occurred on the 32-bit build, I can reproduce the reported relative errors (to within 3 significant digits, anyway) of the first two failures with a calculation (in Python--I don't have a build system set up for the failing platform) in which I replace this line in the C code
https://github.com/scipy/scipy/blob/6925c378bf5766079708420c6caa880612022bca/scipy/special/_cosine.c#L98
with
h = x + (M_PI + 1.2246467991473532e-16)

which should be equivalent to
h = x + M_PI

because 1.2246...e-16 is smaller than the ULP of M_PI.    Do we use the compiler flag -ffast-math or any other unsafe math optimization flags when we compile C code?

Thanks Warren, nice analysis. So maybe fiddling with that line (e.g. (x + M_PI) -> (M_PI + x)) may fix things?

Do we use the compiler flag -ffast-math or any other unsafe math optimization flags when we compile C code?

We shouldn't be. Here are the flags from the failing 32-bit Linux build:
gcc-5 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

extra options: '-std=c99'

And here are the ones from a local build with conda on 64-bit Linux:
C compiler: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-dev/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-dev/include -fPIC

extra options: '-std=c99 -msse -msse2 -msse3'

There are no flags added in special/setup.py aside from -std=c99.

Oh wait, maybe we are leaving something out. There is this in the setup.py:
    define_macros = []
    if sys.platform == 'win32':
        # define_macros.append(('NOINFINITIES',None))
        # define_macros.append(('NONANS',None))
        define_macros.append(('_USE_MATH_DEFINES',None))

_USE_MATH_DEFINES is related to enabling M_PI. So somehow we do that on Windows, and on Linux _cosine.c has:
#if !defined(M_PI)
#define M_PI 0x1.921fb54442d18p+1
#endif

There's probably some combination of these things that ends up being slightly different numerically on 32-bit Linux + GCC 5.

Oh wait, it's simpler. The 1.2246467991473532e-16 is simply not working on 32-bit platforms, the code comment and calculation are for 64-bit only:
    // M_PI is not exactly π.  In fact, float64(π - M_PI) is
    // 1.2246467991473532e-16.  h is supposed to be x + π, so to compute
    // h accurately, we write the calculation as:
    h = (x + M_PI) + 1.2246467991473532e-16;

@WarrenWeckesser this test on 32-bit platforms is mostly a dont-care I think, shall we just mark it knownfail? For values that precise it doesn't make much sense to use 32-bit Python.

How does the 32-bitness of the platform matter to these floating-point calculations? The N-bitness is just about the sizes of integers. Now, when you change CPUs, there might well be differences in how the FPU deals with the precision of intermediates, for example, but that's just a coincidence with changing the 32/64-bitness.

You're right, that made little sense - it's all double calculations. I'm not sure how the hexadecimal M_PI value conversion to double works exactly, but it should be unrelated to 32-bit Python yes/no.

I set up a docker image like the one used by the Azure test setup that was failing, and I was able to do some debugging on that platform (i.e. 32 bit ubuntu bionic, gcc-5, Python 3.8).  It turns out that the problem is this line in the C code in the file _cosine.c:
h = (x + M_PI) + 1.2246467991473532e-16;

combined with this definition of M_PI:
#if !defined(M_PI)
#define M_PI 0x1.921fb54442d18p+1
#endif

With the order that the header files are included in the master branch, the macro M_PI ends up not being defined by the included headers, so we get the definition above. That value is the 64 bit floating point approximation to π (corresponding to, for example, np.pi and math.pi).  When M_PI has that value, h is computed correctly, and there are no test failures.
When "cephes/polevl.h" is included first, we end up getting M_PI defined by one of the included files, and the value of M_PI in that case is 3.14159265358979323846.  That numerical value has a few more digits than can be represented with 64 bit floating point.  The calculation of h is then equivalent to
h = (x + 3.14159265358979323846) + 1.2246467991473532e-16;

The problem is that, even though x and h are double precision, the compiler is maintaining the extra precision specified in 3.14159265358979323846, presumably by using 80-bit extended precision.  That breaks the calculation of h, because the trick of adding 1.2246467991473532e-16 is based on M_PI being the 64-bit representation.  (The value 1.2246...e-16 is the correction to the 64-bit value of π.)
I didn't figure out why changing the order of the headers resulted in M_PI being defined.  It looks like the macro is defined in math.h if either __USE_MISC or __USE_XOPEN are defined.  If something in the scipy and numpy header files defines one of those, it would result in M_PI being defined when math.h is included.  I didn't see where either of those might end up being defined, but I didn't dig too deep.
Whatever the reason, a simple fix is to just unconditionally define M_PI locally with the 64-bit value.  That is, replace the #if ... #endif lines with the one line
#define M_PI 0x1.921fb54442d18p+1

@rgommers, see https://github.com/scipy/scipy/pull/14712


Can Zero Values be included in the mean and median calculation?

I am working with MMSE scores, in which zero is a valid score that needs to be included in the median and mean calculation. What can I do with your box and whisker app that will allow zeros to be included?

I will add the option to include zero's and/or nulls in the calculations in the next update.

Thank you very much. We appreciate that your box plot includes "number of samples", which others do not.

thanks for your contribution!

Hi Jan,
Any expectation for when the version 2.2 will be released?
It includes several changes, like this one (#10) that make-or-break the possibility to use this visual.
I don't want to sound pushy, I just would like to know if will be released soon or if we will have to wait a little longer.
Thanks for the awesome work.


Use the Assistant Library for OK Google hotwording

This requires installation of the wheel for the Google Assistant
Library, which includes a closed-source hotword implementation.

also note that the google-assistant-sdk supports "Hey Google" as well.  noticed that most of the wording and triggers are set only to "OK Google" in the PR.

Codecov Report

Merging #64 into master will increase coverage by 0.19%.
The diff coverage is 66.66%.

@@            Coverage Diff            @@
##           master     #64      +/-   ##
=========================================
+ Coverage    9.76%   9.95%   +0.19%
=========================================
  Files           3       3
  Lines        1720    1727       +7
  Branches      295     297       +2
=========================================
+ Hits          168     172       +4
- Misses       1542    1543       +1
- Partials       10      12       +2

Impacted Files
Coverage Δ

src/actionbase.py
90.32% <66.66%> (-9.68%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 250b8cb...7fd0903. Read the comment docs.

PTAL

I am a newcomer - so this question might not make a lot of sense but the sentence above "This requires installation of the wheel for the Google Assistant Library, which includes a closed-source hotword implementation" has me worried.
I associate wheel with updating python libraries via pip.
People are aware that there is a new version of the google assistant SDK is available BUT that it is for  armv7 capable processors. People are also aware that it supports "hotword"
Many of us are running the AIY software on Pi Zero's and Pi W's which are armv6 based and therefore we cannot use that new version of the the assistant SDK.
My worry is that the change you are making here means that once it is merged - if I was to git pull this repository my installation will stop working because the google assistant will want an armv7 processor.
But as we say in england - I may be (entirely) barking up the wrong tree.

You have chased the cat up the correct tree.
The SDK update in this PR will restrict installation in ARMv7 going forward.
The binary is specific to detecting the hot words, OK Google and Hey Google.  They have rigorously tested it for false positives.  But why they kept it closed sourced we don't know.  For ARMv7 though, they mention they are leveraging some of the newer CPU instructions to speed things up.
To address your concern though, just don't upgrade.  Or, fork it and use your on branch.  You can merge changes in, but kept the wheel out of your branch.

We could remove google-assistant-library from requirements.txt so that it's still usable on ARMv6, (maybe ok_google_requirements.txt?) and amend the error message to say "ok-google triggering is not available on your platform" for ARMv6 etc, and "Please pip install -r ok_google_requirements.txt" for ARMv7.

ummmmmhhhhhhhhhhh.  Is it realistic for the newcomers to start using Git to maintain their own fork/branch and maintain it. We are struggling with how to update it.  A compromise such as that outlined by Drigz seems a better solution and only a little inconvenient for people using the armv7 processor. I would hate to think that some kids using this on a Zero or W suddenly need to become Git gurus.

I know the struggles quite well, as I am not a Python guru myself and stumble through most of this.
I think this discussion really belongs over at the Assistant SDK repo though, not in this PR:
https://github.com/googlesamples/assistant-sdk-python
You can read a marketing overview of what the Assistant does (or will do soon):
https://developers.google.com/assistant/sdk/
Basically, the Assistant SDK does a lot more than just hot wording.  It is what does the interrupting of the vocal commands, for example.  Removing it would pretty much render this "Voice Recognizer" project dead and useless.

I see - and I can see the clamour for new features.

Hi, I've closed my issue above


PhoneGap jank?

Trying to use the Modal component in a PhoneGap app.  It works fine in the browser during testing, but when building via PhoneGap Build and testing on the device, the closing of the modal, along with the background overlay, do pretty strange things..flickering, changing size, etc..before finally closing.
Any ideas on this?  The code I used was pretty much copied form the Elemental Modal page verbatim.
Thanks.

The modal component has a fairly unique composition compared to other components:

It uses a portal to inject code into the DOM
Accessibility enhancement for web use

I suspect the accessibility stuff is the culprit; recommend duplicating the component in your codebase and replacing the relevant sections to see how it effects your build. All that being said, PhoneGap does some wacky stuff sometimes...


fix: utils - change way to compute selector hash

Closes #247
Introduced changes
Changes way to get component from event hash.
This will not fix the recs issue about Map receiver.

Checklist

[x] Linked relevant issue
[x] Added relevant tests
[x] Performed self-review of the code

Summary by CodeRabbit

New Features

Introduced enhanced logging and error handling in entity management functions.
Added a new test suite using Vitest for utility functions to ensure reliability.
Implemented new constants and functions for improved event filtering and component handling.

Bug Fixes

Refined error handling in component setting logic to prevent issues during execution.

Chores

Updated the testing framework from Jest to Vitest, enhancing testing capabilities.
Added coverage reporting for utility functions to improve quality assurance.

Looks good other than that, nice iteration


Revert "Don't throw away transaction metadata in CDATA."

This reverts commit d1fac8cfe40d3c6d0c1f1375b1ef7faf40e29ef4.
This is no longer needed after #108

Tests are passing: https://travis-ci.org/nathangrigg/ofxparse/builds/167237795

The travis-ci stuff looks neat, looking forward to setting it up. :)


TabView: "Element is already the child of another element" thrown when a TabViewItem is switched to

Describe the bug
I migrated my project over to the WinUI version of TabView, but noticed that clicking a different TabViewItem will crash the app with the error: Element is already the child of another element. This bug occurs only when running with a debugger in Visual Studio.
Steps to reproduce the bug

Create a project with a WinUI TabView
Run Project with debugger
Add Multiple TabViewItems to the control
Click one of the unselected items

Expected behavior
Clicking an item should simply display its content. The crash and error is unexpected because it only occurs when running the app with a debugger.
Screenshots

Version Info

NuGet package version:
Microsoft.UI.Xaml 2.2.190830001

Windows 10 version
Saw the problem?

Insider Build (18362.10015)
Yes

May 2019 Update (18362)

October 2018 Update (17763)

April 2018 Update (17134)

Fall Creators Update (16299)

Creators Update (15063)

Device form factor
Saw the problem?

Desktop
Yes

Mobile

Xbox

Surface Hub

IoT

Additional context
The problematic lines of code may be related to line 222 of TabView.cpp

Thanks for reporting! Do you have a repro app you can share?

@jevansaks Unfortunately, I can't reproduce this weird bug in another project. Confusing!

Seems that it is duplicate of https://github.com/microsoft/microsoft-ui-xaml/issues/1245

@eugenegff Agreed.

Thanks! I'll close it and we can use #1245 to track.


⚠ Bofrost.lu has degraded performance

In 7954b99, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2191 ms

Resolved: Bofrost.lu performance has improved in 2e7d4e5 after 35 minutes.


[backport] Eliminate read past end of array in percentile_weightnening

Bring changes from PR 4453 into v8 to eliminate read past end of array in percentile calculations

Given that many consumers of cupy are currently on version 8 and that this issue has caused some difficult-to-mitigate bugs in (e.g.) Rapids, it would be helpful to have the fix from #4453 backported to version 8. Is this something that would be possible to get included in a patch release?

backport of #4453

Jenkins, test this please

Jenkins CI test (for commit 44590cb8069ca8ead97f1b91e88284c077d15372, target branch v8) failed with status FAILURE.

Thanks everyone! 😀


Icon Widget Isn't working correctly in flutter web

Steps to Reproduce

Create a flutter web app
Make a Iconbutton or use Icon widget.
Run and Check the widget in Browser

Problem
Icon Widget isn't working correctly in flutter web
Output

This issue was resolved multiple times here. Try search before you open a new one.  Thank you for your cooperation and have a nice day. :)


Set sun energy to a float insted of int

fixed #76677
please be aware its my first ever contribution to godot or any project really.
even though it was easy fix it looks to good to be true ,please check if ther is any problem.

You can come over to our RocketChat to talk about how to fix this. But it's totally acceptable to close the PR if you feel like you can't salvage it with your level of Git skills. And reapply the changes, this time following the [PR workflow].(https://docs.godotengine.org/en/stable/community/contributing/pr_workflow.html)


Inconsistencies in how sanitization is handled

If the user's fullname is changed from their profile page, script tags appear to be escaped as HTML characters. For example, <script>console.log('Robert')</script> becomes <script>console.log('Robert')</script>.
If the user's fullname is instead changed from their settings page, script tags are removed completely, leaving console.log('Robert').
It is important to note that CenterForOpenScience/osf/pull/812 likely resolves this specific issue, but these multiple methods of sanitization are still both actively used. For example, renaming a project from the project's dashboard follows the pattern of the first, while renaming from the new project organizer follows the second.
Do we/should we have a standard for sanitization?

Dropping back to "working" status, and this is now planned for 0.24
We've had a lot of broad changes go in over the past few weeks, and after discussing with @lbanner and @GaryKriebel, the best course of action here seems to be to defer until 0.23 is live, and re-implement this change instead of trying to merge. So many files have been moved, renamed, and refactored, attempting to reconcile the differences is taking longer than starting fresh.

Agreed - Thanks

I concur... either a .23-dot release or push into .24 release

This is fixed for Ampersands, but only partially fixed for < and >.

My PR above never got merged; Andy is now working on this.

New tracking issue: #2933

Got it -- Thanks.

Close this one and continue tracking on #2933


Add ability to define base_locator as class attribute

Could you please extend ComponentWithBaseLocator and provide ability to define base_locator as class attribute like
class Header(ComponentWithBaseLocator):
    base_locator = locators.ClassLocator("navigation-bar")

	def open_index_page(self):
        ...

Currently it's required to override __init__ method to do this

@ron8mcr I have extended ComponentWithBaseLocator and ListComponent. You can update the pomcorn version and try the changes.


Update components-slots.md

I guess this is a typo: slot-props -> slot-scope. @chrisvfritz

Thanks!


Uncaught TypeError: Cannot redefine property: ontrack if adapter.js included twice on the page

Versions affected
Chrome 63.0.,239.111 on Android 6.0.1
Chrome 63.0.,239.111 on Android 8.0.0
latest adapter.js (https://webrtc.github.io/adapter/adapter-latest.js)
Description
If adapter.js is included twice (more than once) in the page and the browser does not provide window.RTCPeerConnection.prototype.ontrack by default then the following error is thrown: "Uncaught TypeError: Cannot redefine property: ontrack"
Before first insertion window.RTCPeerConnection.prototype.ontrack is undefined, after first insertion it is defined but with configurable: false thus redefinition at second insertion fails.
N.B.:
Double inclusion is a real use case when once uses the adapter.js plus some other package that has webrtc-adapter npm package as dependency.
Steps to reproduce
Reproduction case can be seen on any of the following URLs:
https://s3-eu-west-1.amazonaws.com/tempfilesbucket/COL-1281/ontrack.html
http://jsfiddle.net/az5npxt0/5/
Expected results
No exception should be thrown
Actual results
Exception thrown: "Uncaught TypeError: Cannot redefine property: ontrack"
Suggested solution
"enumerable: true, configurable: true" should be added to https://github.com/webrtc/adapter/blob/master/src/js/chrome/chrome_shim.js#L64 as it was added to https://github.com/webrtc/adapter/blob/master/src/js/utils.js#L75 in https://github.com/fippo/adapter/commit/4f89e1eef675c9391b5d80b953811200726adf70

sigh
the code around line 64 is guarded by a !('ontrack' in RTCPeerConnection.prototype) but the second include attempts to modify via the other path
When using npm and require() double inclusion is not a problem so I don't really see the use-case. But its easy to fix so #833 should take care of that


Bring PHP functionality in line with recent JS changes and add integration tests for third parties

Fixes #66.
This PR makes several enhancements:

Add support for optionalParams for scripts, as was recently introduced in the JS codebase (see #58).
Add third party implementation for Google Tag Manager, as it was recently added in the JS codebase (see #36).
Add PHPUnit integration test coverage to ensure the output for all third parties works as expected (see #66).
Fix various bugs that were exposed by the new integration tests.

It may be helpful for the review process to review the individual commits as part of this PR, rather than all changes at once.
Effectively, this PR does everything needed to bring it up to date with the JS codebase, and it includes test coverage to make sure in the future it doesn't get out of date without being noticed.

Pushed one more fix here in https://github.com/GoogleChromeLabs/third-party-capital/pull/70/commits/b38ca1214a4742d20037b83c7b3d699ba9eda2e3 because the schema changes from #71 broke PHPUnit tests again.


Is it possible to use --service-account?

I'd like to move some backups from a dedicated server to this container, however they are configured to use --service-account (https://github.com/GAM-team/got-your-back/wiki#--service-account).
The command I use is as follows:
gyb --email<EMAIL_ADDRESS>--service-account --local-folder<EMAIL_ADDRESS>Is it possible to configure this container to call gyb in this manner, or is it fixed to the standard 3-legged OAuth authentication?
Thanks

It should work, this container is just a wrapper around gyb.
I haven't explicitly tested it but try the following:

Per the GYB wiki, add/edit the oauth2service.json file in <local config folder>/data
Try running docker run -it -e<EMAIL_ADDRESS>-e NOCRON=1 -v <local config folder>:/config awbn/gyb /app/gyb --action estimate --service-account

If that works you can add this as part of the default cron backup jobs using container env variables:

JOB_FULL_CMD=/app/gyb --action backup --service-account
JOB_INC_CMD=/app/gyb --action backup --search "newer_than:3d" --service-account

If it all works correctly, it would be easy to add this as an extra environment variables (e.g., -e USE_SERVICE_ACCOUNT=1?) to have the wrapper append --service-account automatically without overriding the cron command defaults. Feel free to submit a PR.


add type support

Happy to help out with this if you'd like :)
here is the module I'm declaring for a personal project, but would be glad to help with something like writing in TS to export the types automatically 😃
declare module 'aimless.js' {
	export function seedFunc(salt: number): () => number;
	export function sequence<T>(array: T[], engine?: () => number): T[];
	export function oneOf<T>(array: T[], engine?: () => number): T;
}

@kungfooman Let’s break that down a little bit.  I would assert that:

So you are happy to rewrite everything in useless TS

TypeScript, with over 30 million downloads per week, is not useless.  I would encourage you to read of the companies and open-source libraries on this page that migrated to TypeScript.

destroying the readability of pure JS code

TypeScript code readability compared to JSDoc is subjective at best.  I personally find that JSDoc typings are more verbose and less documented.  Developers that I have worked with are more familiar with TypeScript too.

ALSO requiring an extra build step

We already need a build step to support multiple output formats—ESM and UMD.  Plus, a build step is needed to polyfill older browsers.

impossible to use <script type="module">

TypeScript can be compiled to ESM, CJS, and IIFE.  Of which, ESM is designed to work with <script type="module">
While JSDoc can be used for type information, you will likely still need // @ts-check at the top of all your files and additional linter rules (such as eslint-plugin-jsdoc) to receive the benefits of static typing. That way, you will have errors if your code is missing JSDoc types, or if the JSDoc types don’t actually match your code.

Since the maintainer of this repository @ChrisCavs himself created this issue, I am confident that this library will end up typed.  The only question is which implementation he prefers—TypeScript, JSDoc, or handcrafted types.
I think @chrskerr’s offer to help out is very kind.  I hope that TypeScript is adopted, but having solid types would be a net benefit regardless.

"subjective at best" :joy:
It's like arguing that everyone should use the elevator because one person has two broken legs and sitting in a wheel chair... or a diver telling everyone to carry oxygen flasks while shopping in a super market.
The cherry on top is that most TypeScript developers are hopelessly lazy. As soon as a type becomes just a little bit more complicated, they mentally drop out because they can't handle the task (search for any and ts-ignore...).
Shall we test your TypeScript skills and collect some subjective-at-best test data? :wink:
Type this: const concat = (a = [], b = []) => [...a, ...b];
Pro-tip: don't get it wrong. Some people need help to see reality, so please bear with me and I'm sure we can get to some sort of consensus.

@kungfooman I’ve experienced and used all three methods of adding types to libraries.  In order to be effective:

TypeScript needs a tsconfig file with strict: true (I usually add other rules too)
JSDoc needs // @ts-check and eslint-plugin-jsdoc
Handcrafted types need to be tested (with a library like tsd)

Of all three implementations, I have found TypeScript to be the best experience:

Widely supported with documentation, libraries, communities, issues/Q&A, and more
Widely known by developers and preferred in job interviews

In my (subjective) personal experience, strong typing in TypeScript has:

Made working in highly collaborative codebases easier
Made refactoring easier with more confidence
Made reading and understanding code easier
Made writing new code easier with strong auto-complete

It's like arguing that everyone should use the elevator because one person has two broken legs and sitting in a wheel chair... or a diver telling everyone to carry oxygen flasks while shopping in a super market.

While TypeScript may seem heavy-handed at times, I usually need to dive into more advanced types for library code.  That’s what makes autocomplete like zod and others so powerful.  I’d rather setup TypeScript at the beginning, so that I can avoid converting the whole library when it outgrows JSDoc or hand-crafted types.  Starting off with a strong foundation can avoid technical debt in the future.

The cherry on top is that most TypeScript developers are hopelessly lazy. As soon as a type becomes just a little bit more complicated, they mentally drop out because they can't handle the task (search for any and ts-ignore...).

It’s important to remember that TypeScript is not, in fact, JavaScript.  TypeScript and strongly typed languages in general have unique patterns that need to be learned.  While adding types to the JavaScript runtime world is most common, a lot of the difficulty comes in understanding the TypeScript type world or combining the two worlds together.  I encourage you and others to study more advanced TypeScript with:

Type Challenges
Type Workshops (1, 2, 3, 4)
Type Level TypeScript

These exercises will be helpful if you aren’t familiar with generics, mapped types, and other type-specific concepts.

Shall we test your TypeScript skills and collect some subjective-at-best test data? 😉
Type this: const concat = (a = [], b = []) => [...a, ...b];
Pro-tip: don't get it wrong.

Speaking of Type Challenges, implementing a Concat type is actually one of their easy challenges.  First, let’s look at the official TypeScript Array.concat method:
interface ConcatArray<T> {
    readonly length: number;
    readonly [n: number]: T;
    join(separator?: string): string;
    slice(start?: number, end?: number): T[];
}

interface Array<T> {
    /**
     * Combines two or more arrays.
     * This method returns a new array without modifying any existing arrays.
     * @param items Additional arrays and/or items to add to the end of the array.
     */
    concat(...items: ConcatArray<T>[]): T[];
    /**
     * Combines two or more arrays.
     * This method returns a new array without modifying any existing arrays.
     * @param items Additional arrays and/or items to add to the end of the array.
     */
    concat(...items: (T | ConcatArray<T>)[]): T[];
}

Next, let’s type your original JavaScript function:
const concat = (a = [], b = []) => [...a, ...b];

with TypeScript using generics:
function genericConcat<A extends unknown[], B extends unknown[]>(a: A, b: B): [...A, ...B] {
    return [...a, ...b]
}

We can even make a more complex generic that supports more specific types like [1, 2, ...string[]] instead of just (string | number)[].  Making the result mutable is optional, but I have included it for reference.
type RecursiveMutable<T> = {
    -readonly [K in keyof T]: RecursiveMutable<T[K]>;
}
function constConcat<const A extends readonly unknown[], const B extends readonly unknown[]>(a: A, b: B): RecursiveMutable<readonly [...A, ...B]> {
    return [...a, ...b]
}

I think JSDoc has the @template tag for generics, but I believe it is more limited.  When I’m diving this deeply into types, I’d much rather use TypeScript.
You may say that the syntax is poor or the example is too complex, but it’s uncommon to use this level of TypeScript, especially in application code.  As a library author, I find the very specific TypeScript types improve the developer experience.

so please bear with me and I'm sure we can get to some sort of consensus

Again, it’s ultimately up to @ChrisCavs to decide which of the three implementations (TypeScript, JSDoc, or handcrafted types) he prefers to end up with .d.ts files for type support.
If he chooses JSDoc, would you be willing to contribute and add JSDoc annotations in the same way chrskerr volunteered to help with a TypeScript transition?  Will you put your money time where your mouth is?

While TypeScript may seem heavy-handed at times, I usually need to dive into more advanced types for library code. That’s what makes autocomplete like zod and others so powerful. I’d rather setup TypeScript at the beginning, so that I can avoid converting the whole library when it outgrows JSDoc or hand-crafted types. Starting off with a strong foundation can avoid technical debt in the future.

TypeScript IS the technical debt. You are actively hindering developers to run your code without installing and running an elevator first. Or to take the other metaphor, you are binding people to a wheelchair and call it decreased technical debt because no one can take the stairs any longer.
Also companies seek people with 5 years of experience in whatever is brand-new on the market. :+1:
You are hunting shadows. If you need advanced types, just add a superMegaAdvanced.d.ts.
Whatever you can do in TypeScript, you can do in JSDoc aswell.
And you didn't actually implement the given concat function; the default arguments influence the typing process, which you simplified (maybe because you already lost track of what you were doing while trying to type this one-liner in TypeScript - which by now is a three-liner).
Wanna try again - this simple example of an easy challenge?

And you didn't actually implement the given concat function; the default arguments influence the typing process, which you simplified

Ahh, I missed that since I was coding late last night when I should have gone to bed 😂 in hindsight, I think that was off-topic.  I don’t think you’re actually curious about the answer, but one potential solution is function overrides, similar to the original TypeScript Array.concat definition.
@ChrisCavs I have created a PR to convert this library to typescript in https://github.com/ChrisCavs/aimless.js/pull/8, with additional changes in the description.
Let’s see which one you end up choosing 😄

Hey folks.  Lot of heated discussion on this issue.  Here are my thoughts:
I've always considered TypeScript to be the only real option when it comes to typing JavaScript.  This is probably entirely due to ignorance on the matter.  However I've used TypeScript almost exclusively for work and felt most comfortable with it as an option.  It also seems to be the option with the most flexibility (in typing) and the most features.
However, this project was never about flexing my expertise.  It was about getting out of my comfort zone and creating something awesome that people could use.
I was curious about JSDoc and what it could do, so I've spent about 20 hours over the past week exploring what that might look like and how I could get to a typed result.
I just published a beta version to npm, which utilizes JSDoc and an updated rollup config to generate a aimless.d.ts file.  I believe the idea behind this approach is to write JSDoc comments and then utilize TypeScript to infer the types and generate the files, then use rollup to package the emitted type folder into a single directory file.
I would like for y'all to test what I have by installing the beta version 1.0.3-beta.0, and let me know if this provides sufficient typing for the library.  Feel free to post here with feedback!  If this result is insufficient, I'm happy to go back to the drawing board and refactoring for TypeScript.

@ChrisCavs I cloned the kungfooman:jsdocDevToolsTesting PR and installed the repo, but received this error:
types/aimless.d.ts → dist/aimless.d.ts...
[!] RollupError: Could not resolve entry module "types/aimless.d.ts".

With a cursory glance, here are notes from my PR if you end up continuing the JSDoc route:

avoid the use of * (equivalent of the any type)

it makes the types from functions like uniqFuncSequence and oneOf useless
consider the use of the @template or similar generic tag instead

right click > Go to Definition is broken in multiple editors
add // @ts-check to the top of the js files
add eslint-plugin-jsdoc
devDependencies needs to be updated to the latest versions
formatting from standard needs to be updated
the package.json syntax needs to be updated

Here are additional notes that may be considered out-of-scope:

add CJS build is output
use serve to drop the python dependency
refactor uuid away from the code golf implementation (hard for types and linter)

@ChrisCavs definitely your prerogative to choose JSDoc.  I encourage you to jump in, ask questions, and set a direction early and often to avoid duplication of efforts.
Have a great weekend everyone!  That’s all from me.

@jlarmstrongiv you should be able to install the beta version from npm, or just pull my latest main branch since it's the same code as the beta version.  Are these notes from kungfooman's branch or from my beta release?

@ChrisCavs the build error is from the kungfooman:jsdocDevToolsTesting.  All notes are applicable to<EMAIL_ADDRESS>

@jlarmstrongiv You are easily confused about what you are doing and then blaming me twice? I didn't touch the rollup config, the reason you got that error is the new rollup config:
https://github.com/ChrisCavs/aimless.js/blob/3b5a3ca0e5d226912951ed710379808f2f67c0b5/rollup.config.mjs#L39-L43
The reason why Chris didn't get this error is because he run npm run typegen first.
TLDR: rollup needs to run the typegen command first
Simple as this in package.json:
  "scripts": {
    <snip>
    "prebuild": "rm -rf dist types && npm run typegen",
    <snip>
  },

Ahh, I missed that since I was coding late last night when I should have gone to bed  in hindsight, I think that was off-topic. I don’t think you’re actually curious about the answer, but one potential solution is function overrides, similar to the original TypeScript Array.concat definition.

No worries, everything is fine. I actually do care, otherwise I wouldn't invest time here. I realized very early on that you didn't have too much experience with JSDoc, so I just want to freshen up your perspective a bit.
The funny thing is I used TypeScript for years myself, diving into type programming and telling everyone the world would be a better place once they adopted TypeScript. Until I realized that I can't read my JavaScript any longer once I'm done with the type programming. :joy:
But yea, I'm sure this line alone makes you - at least internally - admit what I'm talking about:
function constConcat<const A extends readonly unknown[], const B extends readonly unknown[]>(a: A, b: B): RecursiveMutable<readonly [...A, ...B]> {

Such thoughts require a certain degree of maturation before they are generally accepted. :wink:

@jlarmstrongiv thanks for the feedback on the beta version!  I apologize for not commenting earlier, I appreciate the effort you put in to the typescript PR.
regarding CJS support: I currently have rollup spitting out a UMD which should be compatible with CJS, AMD, and IIFE.  I added a second file for ES definition to allow more flexibility outside of UMD.
I like @template.  interesting that it is not in the JSDoc documentation.  I see it in the typescript JSDoc documentation however.  I assume the // @ts-check is required to use features like @template?
regarding right click > Go to Definition errors, could you provide a bit more info or maybe an example?  I'm wondering if there's anything I can do about that.  in my editor it seems to work fine.  I wonder i the issue is that it tries to go to the dist definition by default?

Hey folks, and important update:
After having successfully typed the repo using JSDoc and figured out the bundling required via Rollup, I've decided that this isn't a good direction for the repo.  I will be refactoring to TypeScript and Parcel and ditching many of the dependencies currently listed.
My reasoning is as follows: it will be hard to maintain the repo moving forward without modern build tools (Parcel).  Modern build tools are built around TypeScript, and I believe it is the superior option for typing JS.  There are just too many workarounds required to support JSDoc, and JSDoc requires TypeScript support to work in any meaningful way regardless.
Additionally, I do not want to support users importing src files directly into their repos.  I believe this is a bad practice that moves counter to the direction of the entire JavaScript community.  I will continue to support ES modules alongside CJS.
You can expect a patch in the next week (v1.0.3) containing the TypeScript refactor.

I lied, I had time to do this today.  Closing since addressed in v1.0.3

@ChrisCavs no worries!  I’m glad to see v1.0.3 supporting types.  It seems you’ve added the type map too, which is a nice touch.  I appreciate your reasoning throughout this issue.

@kungfooman not blaming you for the build error, I just thought I ran into https://github.com/ChrisCavs/aimless.js/pull/7#discussion_r1215671554 or something else entirely, like having the wrong version/branch.  I decided to look into<EMAIL_ADDRESS>instead of the error because I was short on time.  It’d be good to put npm run typegen into a prebuild script.

Finally, @kungfooman, you are right that I am not as well-versed in JSDoc as you.  I usually use TSDoc with Typescript.  If the package is very small, then I sometimes author my code in .js and add .d.ts files, which are then checked with tsd or tsc.
Again, JSDoc, TypeScript, and Handcrafted Types are all valid solutions to the problem.  I still prefer TypeScript due to the reasons above, but you are more than welcome to prefer JSDoc.  Similarly to how several companies and libraries migrated to TypeScript, I do know that eslint may be sticking with JSDoc https://github.com/eslint/eslint/discussions/16557
While TypeScript can get messy, such as that line you quoted, I find that many more of those situations are when converting JavaScript to TypeScript, rather than starting with TypeScript at the beginning.  And, in library code, I usually need to dive down into more TypeScript-specific syntax, so it’s easier to have TypeScript support at the get-go too.  As a library author, I also try to make my types as specific as possible, as I think it improves the developer experience of the end user.
I am also probably biased due to the code bases I have worked with.  The pure JavaScript ones have been much harder to work with, compared to the TypeScript ones.  And one of the larger JSDoc codebases I have worked with did not use // @ts-check or eslint-plugin-jsdoc, which led to a lot of issues down the line—it was not possible to generate types from that code base, despite the JSDoc hints—plus other time sinks.  Luckily, Aimless.js did not have nearly as many issues as that code base.  At the end of the day, I have also spent significantly more time with TypeScript code bases, and have become much more familiar with its syntax and idiosyncrasies.
If JSDoc improves their documentation/resources and supports more type features, I’d be glad to take another look at it someday.  For now though, I’ll be sticking with TypeScript, warts and all.
I think that tone in this asynchronous chat has also been difficult to get right, and I apologize for any remarks that came across poorly.  At the end of the day, I’m just glad that this library now has solid types.


Live scan feature

It would be nice if this lib could have a live scan/live stream feature - just like this lib for barcodes -, so you can easily use it to read QR codes in real time.
Also related to/required for https://github.com/edi9999/jsqrcode/issues/29

Actually a problem for this is how to efficiency get the image data from a video element without saving it in a canvas first. @edi9999, do you have any suggestions/ideas?

Okay, I found some ways, but actually decoding does not work at all on mobile devices (tested with Firefox for Android). It always fails with: Couldn't find enough finder patterns:0 patterns found
The code I used is here: https://gist.github.com/rugk/44e9cd89f3b67adfcb229843fffed02
And even when using your test file on the desktop with some photos of a QR code and some surrounding elements, only 2 of 4 images were recognized in my case. (If you want the test pics, I can mail them to you)

The URL file gist is a 404.
Yes, I'd be interested in the images, but if possible, it would be good to add them in the testsuite. Do you have confidential data on those images ?

No, but it's easy to create new images. (I only wanted to share them, so you can reproduce the issues…) The real link is https://gist.github.com/rugk/44e9cd89f3b67adfcb229843fffed02f.

Here's what I'm using to live-scan (extracted and approximately rewritten from my vue files, it's now pseudo-code for some parts and I messed up indentation when pasting), I'm mid-development and will refactor & temper everything, so it's a mess right now.
I'll probably post the whole thing when it'll be clean, though. It includes live scan & a fallback for older browsers / Safari / IE / devices without a camera.
Hope it'll help and you won't mind this mess.
main file
<video id="videoTarget"></video>
<!-- I'm binding the width & height of the canvas to document.documentElement.clientHeight/Width, since it's in a PWA -->
<canvas id="canvasTarget" :width="w" :height="h"></canvas>

#videoTarget {
        position:fixed;
        left:-9999px;
        top:-9999px;
}

#canvasTarget {
        position:fixed;
        left:0;
        top:0;
}

import Loop from './loop';
const FPS = 45;
const FRAMESKIP = 17;
const constraints = {
  audio: false,
  video: {
    width: { min: 1280},
    height: { min: 720},
  },
};

const state = {
  streamRef : {},
  w: 0,
  h: 0,
  workerAnswered: true,
};

let refToVideo;
let refToCanvas;
let refToContext;
let loop;
let worker;

created() {
  // again, I'm really sorry, this is vue-specific but easily adaptable.
  state.w = document.documentElement.clientWidth;
  state.h = document.documentElement.clientHeight;
  worker = new Worker('/js/worker.js');
  worker.onmessage = (message) => {
     state.workerAnswered = true;
     if (message.data !== 'NORESULT') {
       stopStream();
       // do whatever you want with the result
     }
  };
}

mounted() {
  if (!'getUserMedia' in navigator) {
    this.$router.push({path: '/legacy'});
  }
  refToVideo = this.$el.querySelector('#videoTarget');
  refToCanvas = this.$el.querySelector('#canvasTarget');
  refToContext = refToCanvas.getContext('2d');
  startStream();
}

startStream() {
  navigator.mediaDevices.getUserMedia(constraints).then((stream) => {
  state.streamRef = stream;
  refToVideo.srcObject = stream;
  refToVideo.play();
  loop = new Loop( // { params }, callback, frameskipped callback
    { fps: FPS, frameskip: FRAMESKIP},
    (frame) => { if (refToVideo.readyState === refToVideo.HAVE_ENOUGH_DATA) {
      refToContext.drawImage(refToVideo, 0, 0, this.w, this.h);
      }
    },
    (frame) => {
      if (this.workerAnswered) {
         this.workerAnswered = false;
         const imageData = refToContext.getImageData(0, 0, this.w, this.h);
         worker.postMessage(imageData);
       }
    });
    loop.start();
  }).catch((err) => {
    console.log(err);
  });
}

stopStream() {
  state.streamRef.getTracks().forEach((track) => {
     track.stop();
  });
  loop.stop();
}

### loop.js (worst file)
class Loop {
   constructor({ fps, frameskip}, callback, rareCallback) {
       this.params = {
           fps: fps || 60,
           frameskip: frameskip || 1
       };
       this.callback = callback;
       this.rareCallback = rareCallback;
       this.interval = 1000 / fps;
       this.rafRef = 0;
       this.skipper = 0;
       let then = performance.now();
       this.loop = (now) => {
           const delta = now - then;
           this.rafRef = requestAnimationFrame(this.loop);
           if (delta >= this.interval) {
               then = now - (delta % this.interval);
               this.skipper += 1;
               if (this.skipper % this.params.frameskip === 0) {
                   this.skipper = 0;
                   this.rareCallback(this.rafRef);
               }
               this.callback(this.rafRef);
           }
       }
   }

   start() {
       this.rafRef = requestAnimationFrame(this.loop);
   }

   stop() {
       cancelAnimationFrame(this.rafRef);
   }
};

export default Loop;

worker.js
import QrCodeReader from './qrcode-reader'; // my fork, no big changes.
const qr = new QrCodeReader();

qr.callback = (err, res) => {
    if(err) {
        postMessage('NORESULT');
        return;
    }
    postMessage(res);
};

onmessage = function(e) {
    qr.decode(e.data);
};

Is there a proof of concept?

I'll make a github page with a runnable version this afternoon.

@rugk here's a demo : https://lucassifoni.github.io/qr-code-live-scan/
I'm really sorry for the packaging though, you currently both have to run
npm run watch

And serve it accordingly, with
php -S localhost:8080

or
caddy

or whatever suits your use. Without mentioning that the code is currently a mess and needs to be refactored.
I added qrcode-reader to the repo, because I removed a case invoking new Image() in my worker script, that made it fail.
Regardless, the demo works, and is configured to run at 45 FPS and ask for a scan (if the worker isn't busy) every 17 frames. With this setup, my testing Motorola G (1gb ram, quite weak computing power) manages to keep everything fluid.
Feel free to ask anything, I'll have to elaborate on this and cover more use-cases for a current project, this is our first draft.

Salut Lucas, ta démo avec support WebRTC fonctionne très bien. J'ai besoin de ce système pour un projet mais je n'y arrive pas avec le projet de LazarSoft, c'est très instable. Alors que ta démo me détecte facilement et rapidement les QRCodes ! Merci beaucoup, je vais récupérer tes sources, on verras si j'arrive à le faire fonctionner :)

Le repo est très sale parce que je n'arrivais pas à configurer webpack pour compiler mon worker séparément avec la base fournie par vue-cli, d'où les dépendances demandées par vue-cli ET laravel-mix (qui n'est qu'un wrapper autour de webpack). J'ai l'impression que réécrire la lib en s'inspirant de ça https://github.com/cozmo/jsQR/pull/25 pourrait être bon pour les perfs, cependant.
C'est pour un gros soft interne, donc c'est possible que je prenne le temps de le faire.
//
The repository is really messy because I couldn't configure webpack to separately compile my worker with the vue-cli generated configuration, so that's why both vue-cli/webpack & laravel-mix dependencies are required. Anyway, laravel-mix is only a webpack wrapper. I feel like rewriting the library following this trail https://github.com/cozmo/jsQR/pull/25 could improve performance. We'll maybe put the time for it.


MINIFI-478 Fix broken OrderedProperties and OrderedPropertiesTest

This fix is not specific to a platform or Java version.
This test was previously passing by plain coincidence. Here's a more detailed explanation:

The only ordered data structure in OrderedProperties was linkedHashSet happened to be updated but never read and therefore it was never consulted for reading properties in ordered manner, i.e. in the same order they were put/set.
The expectation and the contract of OrderedProperties is that overridden store(OutputStream out, String comments) method stores properties in ordered manner. However, this doesn't happen without this PR.
The order in which properties are stored does not depend on keys() method but entrySet() method as it can be witnessed along the following call path OrderedProperties.store -> Properties.store -> Properties.store0 where the entries (Ordered)Properties are read via entrySet(). Without this PR the default Properties entrySet() implementation is used to read properties.
The order in which properties are read in Properties entrySet() which is based on underelying ConcurrentHashMap.entrySet() where entries are unordered.
The reason why OrderedPropertiesTest was ever passing (on earlier Java versions) is the combination of a small test data set - only three properties with specific property names and the hash function implementation within the underlying ConcurrentHashMap.putVal, that stored those property names in such a way that the corresponding ConcurrentHashMap.entrySet() returned them in order by coincidence. The fact that OrderedProperties.sets() and any order wasn't explicitely used was not realized by neither implementor nor anyone else by this same coincidence.
Due to limitations of my current computer setup, I didn't bother to compare side-by-side the hash function implementations of ConcurrentHashMap.putVal across Java versions so this is left as an exercize to the reader. However, note that hash function implementations may change from Java version to version since their role is to provide statistically normalized data (i.e. hash key) distribution across a hash map without having to guarantee the same mapping from name to a hash map slot.

Thank you for submitting a contribution to Apache NiFi - MiNiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Hi @shonzilla
Code looks fine to support both 8 and 11.  The implementation of the store0 method does actually change from 8->11 as seen in the the relevant sources:
8: http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/Properties.java#l832
11: https://github.com/unofficial-openjdk/openjdk/blob/jdk/jdk/src/java.base/share/classes/java/util/Properties.java#L926
It is my belief leaving keys() as is and then introducing the overridden entryset as provided by you should cover both.  Does that thought process seem reasonable to you as well?

Hi @shonzilla,
Just wanted to see if you had any thoughts on the comments I left so we can get this wrapped up.  Thanks!

Going to make the suggested changes and include as I believe the functionality to be correct as discussed.

Hi @apiri, sorry for not getting back in timely fashion. After some frantic time at work and subsequent vacation (mostly offline, no-laptop), I got back into the pickle with no time for open-source. :wink:
Thanks for your patience and for removing the superfluous comment for me with your amended merge commit.
I've re-checked the JDK source code changes between versions 8 and 11 and indeed - Properties.store (i.e. the underelying private store0 method) has migrated from iterating over its keys to iterating over its entrySet. This storing of key-value pairs with incl. comments to an output stream is part of the current unit test before it loads already ordered properties from the said output stream. If I remember correctly, this test coverage is sufficient for the little use of OrderedProperties in the MiNiFi project.
That's why one needs both to keep both the original overridden keys and the newly overridden entrySet method, contrary to what I originally thought. However, one should keep in mind that in both cases the requirement is that either keys (in JDK 8) or entries (key-value pairs, used in JDK 11) must be returned in ordered manner, i.e. in the same order that the individual properties/entries were put into an OrderedProperties instance. Without additional testing (on JDK 8) I cannot verify and hence cannot guarantee that the keys method returns keys in order that they were put into OrderedProperties.
I'm temporarily down to Rasberry Pi 3+ with just 1GB (incl. OpenJDK 11 though) so it may be problematic to verify this by building NiFi... and it would certainly hurt without closing the web browser first. :laughing:
Speaking of Rasberry Pi - it might be an interesting experiment though to get MiNiFi or NiFi (to build and) run on it. One could build for example a self-hosted and highly flexible service comparable to IFTTT but isolated from the public internet, used to make one's home automation smarter etc.
Are you aware of anyone doing this with MiNiFi or NiFi?
A quick web search revealed that (obviously) I'm not the first man to think about running NiFi on a RasPi and it's great to already know that it's possible and maybe even feasible. :nerd_face:

@shonzilla No worries.  These things happen, thanks for following up and your contributions.  I think we are on the same page and thus why I moved forward with the merge (and cleaned up the errant comment).
We've had several folks that have done some neat projects and exercises centered around Pis.  I would encourage you to scope out our lists and optionally post to reach out to the broader community (http://nifi.apache.org/mailing_lists.html).  Builds that support ARM are even enabled via our bootstrap process on our C++ impl (https://github.com/apache/nifi-minifi-cpp/).  A unique twist on integration with other services that comes to mind is http://www.opensourcedad.com/apache/minifi-cpp/2016/12/18/poop-scale.html


Fixed h2 link to not bounce to random h2 section

Docs Build status updates of commit bd50373:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-edge/devtools-guide-chromium/3d-view/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Feature Request - Add Support for Diablo 4

Diablo 4 is coming/is downloadable
It is found under the Path of /tpr/fenris. Due to the nature of the Naming Scheme of other Activision-Blizzard Games it seemse like this is the Productname on their CDN

Postpone till 06.06.2023, because currently only the Tact Code fenris_beta is accessible (due to the fact that fenris is not out yet)

Released in v1.6.0


Quickstart not working

This guide does not work.
I met the prerequisites, created a function app project. The next step run the function locally throws a lot of errors. I suspect there are some steps missing in the guide which need to be added.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 6e88ea08-81ef-2b5a-b9d6-539e85592672
Version Independent ID: f2d8b7b4-906a-d1f5-5201-fc22a7890955
Content: Quickstart: Create your first C# function in Azure using Visual Studio
Content Source: articles/azure-functions/functions-create-your-first-function-visual-studio.md
Service: azure-functions
GitHub Login: @ggailey777
Microsoft Alias: glenga

Thank you for your feedback! We will review and update as appropriate.

Hello @danielniccoli - Sorry for the late reply. I've gone through the steps on this doc and didn't run into any issues. Based on the screenshot you've included, it looks like the Nuget packages aren't being installed on your machine. You can resolve this problem by right-clicking on your project in the left nav, choosing the Manage Nuget Packages option, and installing the required dependencies. Typically, Visual Studio would automatically install these packages (when you press the F5 key), and it's not a required manual step. I suspect the issue may have to do with the specific version of Visual Studio you may be running or its missing updates. If you continue to encounter this problem, please let me know and I'd be happy to investigate further.
Since we didn't determine any changes/faulty steps in this documentation, we will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.

Hi @mike-urnun-msft ,
Thanks. The problem was that the Nuget repository (nuget.org) was not available in my fresh install of VS 2022. Before I noticed this, I did a clean uninstallation and tried installing VS 2022 Preview, in which the repository was also not available. In both versions, I had to add the repo manually and then I was able to follow the guide.
I'm unfamiliar with VS and it took a while to figure this out. If the Nuget repo is not added to VS at the time of installation, maybe it would be good to include that into the quick guide.


⚠ Secret Website 2 has degraded performance

In b4ba9b9, Secret Website 2 ($SECRET_WEBSITE_2) experienced degraded performance:

HTTP code: 200
Response time: 5955 ms

Resolved: Secret Website 2 performance has improved in 81f5501.


Fix Asset Build Input File cannot be found while Archiving

This Pull Request fixes solution referencing #134. This is achieved by using xmark symbol image. NOTE: this bumps macOS version to .v11

Hello @Awesomeplayer165,
im not comfortable bumping the version, that's why I exported the x.mark SFSymbol and used it as an image instead of using it via code in the first place. As said in the related issue, im not even sure if this is an issue with this library and not just your local build. Again, if multiple people start having this issue I would rethink this. Anyway, thank you for your work!


Very interesting

Describe the bug
So I've been learning quite a bit about signatures the past few months and have barely scratched the surface it seems, so please bare with me. I've been trying to manipulate user installed apks that have the sharedUserId="android.uid.system" flag set. This one happens to have the sharedUserId="android.uid.shell". I'm mainly trying to get the debuggable flag set to true and preserve the signature. Obviously it uses the platform sig. It installs no problem on its own without anything done to it yet. In my experimentation one of my attempts was this: "apksigner sign --debuggable-apk-permitted --append-signature --ks platform.p12 --verbose LabelChecker_1.0.apk". The platform.p12 I use is another cert and private key that match the regular platform sig in quite a few areas, mainly these:
"X509v3 Subject Key Identifier:
93:2C:3A:F7:0B:62:7A:0C:76:10:B5:A0:E7:42:7D:6C:FA:EA:3F:1E
X509v3 Authority Key Identifier:
keyid:93:2C:3A:F7:0B:62:7A:0C:76:10:B5:A0:E7:42:7D:6C:FA:EA:3F:1E
DirName:/C=KR/ST=South Korea/L=Suwon City/O=Samsung Corporation/OU=DMC/CN=Samsung<EMAIL_ADDRESS>serial:D2:09:95:A7:9C:0D:AA:D6".
Except the private key doesn't match because of key rotation I believe. Anyways, after I use that first command I get this output using your tool:
"java -jar APKEditor-1.3.8.jar info -signatures -i downloads/LabelChecker_1.0.apk"
"Certificates="1"
Subject<EMAIL_ADDRESS>CN=Samsung Cert, OU=DMC, O=Samsung Corporation, L=Suwon City, ST=South Korea, C=KR
Issuer<EMAIL_ADDRESS>CN=Samsung Cert, OU=DMC, O=Samsung Corporation, L=Suwon City, ST=South Korea, C=KR
Validity FROM: Wed Jun 22 12:25:12 GMT 2011, TO: Sun Nov 07 12:25:12 GMT 2038, PERIOD:<PHONE_NUMBER>00
Serial: 0xd20995a79c0daad6
OID: 1.2.840.1135<IP_ADDRESS>"
Looks great, but fails verification since it didn't preserve the original signature. Now with the same command on the original app that does install I get this output:
"java -jar APKEditor-1.3.8.jar info -signatures -i downloads/LabelChecker_1.0.apk"
"certificates="null""
It's pretty strange behavior and was hoping I might be able to get the reasons why the original apk isn't recognizing the signature at even even though it is signed and installs with no issues. I'll be available if you need anymore information at all. Thank you.
To Reproduce
Steps to reproduce the behavior:

Use the apksigner command
Check the signature using APKEditor
Check the signature of the original unedited apk
Output is "null"

Used version '1.3.8'
Operating system 'samsung s22 android 14'
Command 'java -jar APKEditor-1.3.8.jar info -signatures -i "LabelChecker_1.0.apk"'

Log/Stacktrace
I'm not sure what logcat command I could use for this. Please provide an appropriate one and I'll give the output afterwards.
Used apk file
Github won't allow me to attach an apk.
Additional context
Add any other context about the problem here.

HAH! I can't tell if you're being sarcastic or not about the brief description. All good anyways. And you nailed it with the V1, I actually was going to bring it up but forgot when I submitted it. This app would be great to run, it's just samsung is being samsung again. I'm also pretty new to this type of stuff, last year around this time I was still using my phone as a phone instead of exploring everything about them and now it's taken over everything haha. Anyways here's the app if you want to mess around with it.....
LabelChecker_1.0.apk.zip