Detect differences between KCP and SKR CRDs if they have the same version

Description
Changes proposed in this pull request:

Store Kyma and ModuleTemplate CRD generation as annotations in the Kyma CR for the SKR and KCP clusters
Upgrade the CRD in SKR if the stored generation is not the same as the latest CRD generation

Related issue(s)
Resolves #534

/retest


Nelson - https://github.com/cncf/landscape/pull/3071

https://github.com/cncf/landscape/pull/3071

@Nelson-He please review the PR

LGTM

@Nelson-He please review the PR
The contents are ok, I just add the LGTM
Thank you very much, bro.  :)


The github branches are disorganized and messed up.

Today, the master branch was deleted and merged into the main branch.


Git tfs clone never working

git tfs clone http://your-tfs-server:8080/tfs/your-collection $/your-tfvc-repo --export --export-work-item-mapping=C:\temp\git_migration_mapping.txt
the above command was ran and after running always resulting the below error.
Usage: git-tfs clone [options] tfs-url-or-instance-name repository-path <git-repository-path>
  ex : git tfs clone http://myTfsServer:8080/tfs/TfsRepository $/ProjectName/ProjectBranch

  -h, -H, --help
  -V, --version
  -d, --debug                Show debug output about everything git-tfs does
  -i, --tfs-remote, --remote, --id=VALUE
                             The remote ID of the TFS to interact with
                               default: default
  -A, --authors=VALUE        Path to an Authors file to map TFS users to Git
                               users (will be kept in cache and used for all
                               the following commands)
      --template=VALUE       Passed to git-init
      --shared[=VALUE]       Passed to git-init
      --autocrlf=VALUE       Normalize line endings (default: false)
      --ignorecase=VALUE     Ignore case in file paths (default: system
                               default)
      --bare                 Clone the TFS repository in a bare git repository
      --workspace=VALUE      Set tfs workspace to a specific folder (a
                               shorter path is better!)
      --ignore-regex=VALUE   A regex of files to ignore
      --except-regex=VALUE   A regex of exceptions to '--ignore-regex'
      --gitignore[=VALUE]    Use .gitignore to ignore files on download from
                               TFS. Alternatively, provide path toward the-
                                .gitignore file which will be used to ignore
                               files
      --no-gitignore         Do not use .gitignore to ignore files on
                               download from TFS
  -u, --username=VALUE       TFS username
  -p, --password=VALUE       TFS password
      --no-parallel          Do not do parallel requests to TFS
      --all, --fetch-all     Fetch TFS changesets of all the initialized tfs
                               remotes
      --parents              Fetch TFS changesets of the parent(s)
                               initialized tfs remotes
  -l, --with-labels, --fetch-labels
                             Fetch the labels also when fetching TFS
                               changesets
  -b, --bare-branch=VALUE    The name of the branch on which the fetch will
                               be done for a bare repository
      --force                Force fetch of tfs changesets when there is
                               ahead commits (ahead commits will be lost!)
  -x, --export               Export metadata
      --export-work-item-mapping=VALUE
                             Path to Work-items mapping export file
      --branches=VALUE       Strategy to manage branches:
                               * auto:(default) Manage the encountered merged
                               changesets and initialize only the merged
                               branches
                               * none: Ignore branches and merge changesets,
                               fetching only the cloned tfs path
                               * all: Manage merged changesets and initialize
                               all the branches during the clone
      --batch-size=VALUE     Size of the batch of tfs changesets fetched (-1
                               for all in one batch)
  -c, --changeset, --from=VALUE
                             The changeset to clone from (must be a number)
  -t, --up-to, --to=VALUE    up-to changeset # (optional, -1 for up to
                               maximum, must be a number, not prefixed with 'C')
      --ignore-branches-regex=VALUE
                             Don't initialize branches that match given regex
      --ignore-not-init-branches
                             Ignore not-initialized branches
      --ignore-restricted-changesets
                             Ignore restricted changesets
      --resumable            if an error occurred, try to continue when you
                               restart clone with same parameters

Find more help in our online help : https://github.com/git-tfs/git-tfs/blob/master/doc/commands/clone.md

this was executed based on the steps provided in the page
https://www.scrum-tips.com/agile/migrate-from-tfvc-to-git/

Did you run this perhaps in a git bash? If so, try prefixing it with MSYS_NO_PATHCONV=1,
e.g. git tfs clone http://myTfsServer:8080/tfs/TfsRepository $/ProjectName/ProjectBranch
or try running the clone in a normal windows cmd.
The reason for that is that the underlying MSYS2 environment tries to be smart and assumes that $/ProjectName/ProjectBranch
is a windows path and mangles it somehow so that git-tfs doesn't understand it anymore.
cmd doesn't have that behaviour, and so does deactiving that behaviour with MSYS_NO_PATHCONV=1 in git bash

Is the answer from @siprbaum solved your issue?
I will close but if not, ask for reopening...

MSYS_NO_PATHCONV=1

'MSYS_NO_PATHCONV' is not recognized as an internal or external command,
operable program or batch file.
The above is the error message which I got when I tried executing the command in both powershell and windows cmd

Is the answer from @siprbaum solved your issue?
I will close but if not, ask for reopening...

Please Reopen as I did not get the correct result

The above is the error message which I got when I tried executing the command in both powershell and windows cmd

Indeed, 'MSYS_NO_PATHCONV' is for git bash only...
But I have no clue why the command failed.
What the log file is showing (in %AppData%\Local\git-tfs\git-tfs_log.txt)?

@TheIntruder I edited your post to make the output of th cmd more readable.
I also tried your example.
git-tfs clone  http://your-tfs-server:8080/tfs/your-collection $/your-tfvc-repo  Foo --export --export-work-item-
mapping=C:\temp\git_migration_mapping.txt

It started to fetch the changesets when I specified a directory (Foo) and manually created an empty file C:\temp\git_migration_mapping.txt. I am preetty sure that this file should not be created by the user

@TheIntruder: Could you retry this (adapted accordingly with your TFS server) in a cmd:
REM Create an empty file
type NUL > C:\temp\git_migration_mapping.txt

git-tfs clone  http://your-tfs-server:8080/tfs/your-collection $/your-tfvc-repo  Foo --export --export-work-item-
mapping=C:\temp\git_migration_mapping.txt

Please report back if it does work for you when the directory is specified and (currently as a workaround) an empty work-item-mapping file is provided.
If it doesn't work, we need the git tfs version and the  traces in %AppData%\Local\git-tfs\git-tfs_log.txt.
Otherwise there is no chance to analyse this  any further

Closing due to inaction of the reporter.
In case further information is provided, we can revisit that and reopen the issue again.

Closing due to inaction of the reporter.
In case further information is provided, we can revisit that and reopen the issue again.


Add a new API for Apps Browser to use

The old deprecated custom CLI command interface that was exposed as an API (this one https://github.com/fyne-io/fyne/blob/master/cmd/fyne/commands/command.go) has been removed from cmd/fyne tool here. We should introduce a new, more sensible, API for use in https://github.com/fyne-io/apps.

Hmm. This might actually not be a blocker now that I think about it. The apps browser can still use the deprecated API until we have added a new API. Nothing from the new commands here is exposed.


Mscolab: collaborator can access version history

Fixes: #920

@joernu76 @ReimarBauer please review this PR

try to resolve this conflict

I have also refactored permissions in def set_active_pid(self): as suggested by @joernu76

I've called handle_mscolab_buttons inside set_active_pid and handle_update_permission,  now it is completely working.


Merge the 3 repeating Python binary compilations

This PR affects "Basic Tests Python" on Linux/Windows/macOS. We used to re-compile Python binaries for different IO platforms (native/gevent/asyncio). But they are actually the exact same build. In recent effort of improving grpc/grpc's presubmit tests efficiency, @jtattermusch has brought this problem up.
This PR merges the build stage of different IO platforms. Test coverage should be the same.
This PR also adds Python3.10 support to run_tests.py.

TIL that the parallelism in test running was not achieved by the customized test runners, but run_tests.py itself. run_tests.py encapsulates every single test into a job, then handle the scheduling. Each test case will be run as a different process. (We did work hard to implement an OSS Blaze back then.

Optimized Time Cost for "Basic Tests Python":

Linux: 30 minutes -> 17 minutes
macOS: 30 or 60 minutes -> 14 minutes (we have 2 types of macOS worker)
Windows: 80 minutes -> 38 minutes

Non of the failed presubmit checks is related to this PR.

Known failures: https://github.com/grpc/grpc/issues/28483 https://github.com/grpc/grpc/issues/28389


allow usage of multiple gopaths

Fixes #123
Requires https://github.com/src-d/go-parse-utils/pull/8

Amended with the change, @Serabe


[RFC]: alias core-js in jest-preset to correct version

Not sure if we should wait for https://github.com/babel/babel/pull/10146
to be released until we release this?

We could also investigate whether it makes sense to extract this into a separate babel plugin and replace the webpack aliases in untool with that babel plugin too: https://github.com/untool/untool/blob/next/packages/webpack/lib/configs/build.js#L103-L106

Btw, v7.7.0 already contains the fix that you've mentioned: https://github.com/babel/babel/releases/tag/v7.7.0

Btw, v7.7.0 already contains the fix that you've mentioned: https://github.com/babel/babel/releases/tag/v7.7.0

Yes, let's remove it from this PR and set the new minimum required version for babel to v7.7.0


feat(impl):[TRI-1414] update dependency manually

Update dependency manually to fix 3 vulnerabilities

Created IP Review Issues:

https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/9097
https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/8919
https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/9098


GDExtension library not found: meta/plugin.gdextension

Using Godot 4.2 stable downloaded today.
Using the release copy of this plugin here: https://github.com/GodotVR/godot_openxr_vendors/releases/tag/2.0.1-stable
I've also tried with 2.0.0.
I get this error on startup with the plugin enabled:
  core/extension/gdextension.cpp:688 - GDExtension library not found:
  Failed loading resource: res://addons/godotopenxrvendors/meta/plugin.gdextension. Make sure resources have been imported by opening the project in the editor at least once.

(the project has been opened at least once)
Inspecting the file, I see this:
[libraries]

android.debug.arm64 = "res://addons/godotopenxrvendors/meta/.bin/debug/arm64-v8a/libgodotopenxrmeta.so"
android.release.arm64 = "res://addons/godotopenxrvendors/meta/.bin/release/arm64-v8a/libgodotopenxrmeta.so"
macos.debug = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.macos.template_debug.framework"
macos.release = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.macos.template_release.framework"
windows.debug.x86_64 = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.windows.template_debug.x86_64.dll"
windows.release.x86_64 = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.windows.template_release.x86_64.dll"
linux.debug.x86_64 = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.linux.template_debug.x86_64.so"
linux.release.x86_64 = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.linux.template_release.x86_64.so"
linux.debug.arm64 = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.linux.template_debug.arm64.so"
linux.release.arm64 = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.linux.template_release.arm64.so"
linux.debug.rv64 = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.linux.template_debug.rv64.so"
linux.release.rv64 = "res://addons/godotopenxrvendors/meta/.bin/libgodotopenxrmeta.linux.template_release.rv64.so"

Inspecting the res://addons/godotopenxrvendors/meta/.bin/ path however in the release ZIP file, I see:

debug

godotopenxrmeta-debug.aar

release

godotopenxrmeta-release.aar

More of a traceback:
Godot Engine v4.2.stable.official.46dc27791 - https://godotengine.org
OpenGL API 3.3.0 NVIDIA 546.17 - Compatibility - Using Device: NVIDIA - NVIDIA GeForce GTX 1070 Ti

ERROR: Parameter "profile_def" is null.
   at: _update_interaction_profile (modules/openxr/editor/openxr_interaction_profile_editor.cpp:260)
ERROR: GDExtension library not found:
   at: open_library (core/extension/gdextension.cpp:688)
ERROR: Failed loading resource: res://addons/godotopenxrvendors/meta/plugin.gdextension. Make sure resources have been imported by opening the project in the editor at least once.
   at: (core/io/resource_loader.cpp:274)
ERROR: GDExtension library not found:
   at: open_library (core/extension/gdextension.cpp:688)
ERROR: Failed loading resource: res://addons/godotopenxrvendors/meta/plugin.gdextension. Make sure resources have been imported by opening the project in the editor at least once.
   at: (core/io/resource_loader.cpp:274)
ERROR: GDExtension library not found:
   at: open_library (core/extension/gdextension.cpp:688)
ERROR: Failed loading resource: res://addons/godotopenxrvendors/meta/plugin.gdextension. Make sure resources have been imported by opening the project in the editor at least once.
   at: (core/io/resource_loader.cpp:274)

Version 2.0.2 (https://github.com/GodotVR/godot_openxr_vendors/releases/tag/2.0.2-stable) solved the problem in my case.


Clean up conditional git CMAKE args

Resolves #336
I can add this too to PR #350 if that gets merged.
However this still has the issue that @rijobro mentioned: In order to change the git checkout, you need to have the repo downloaded already, otherwise it will not work. I do not know how to fix that, I am open to suggestions.

@KrisThielemans  updated


CIS 1.6 scan ( profile k3s-cis-1.6-profile-hardened ) failing on K3S agent nodes

Issue description:
While installing the K3S cluster , set the hardening parameters for both master and agent nodes. Nevertheless, the CIS scan fails for worker nodes.The default Rancher CIS Benchmark with the profile k3s-cis-1.6-profile-hardened is used.
Business impact:
Unable to produce any CIS scan report and its making the impact on security.
Troubleshooting steps:
Done the manual scan from the K3S cluster and could see its failing for agent nodes. Looks like the CIS scan template needs to be adjusted for agent nodes. As per the template below checking is happening and its waiting for a true flag. But in agent node it should not be the same as master.
Chart is trying as below and returns no result
root@v0003773 ~]# journalctl -u k3s | grep "Running kubelet" | tail -n1 | grep "protect-kernel-defaults"
[root@v0003773 ~]#
But it should be below in agent node and then it works.
[root@v0003773 ~]# journalctl -u k3s-agent | grep "Running kubelet" | tail -n1 | grep "protect-kernel-defaults"
Feb 15 16:18:52 v0003773.kne.bund.drv k3s[1572]: time="2022-02-15T16:18:52.543130502+01:00" level=info msg="Running kubelet -address=<IP_ADDRESS> --anonymous-auth=false --authentication-tokenon-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=<IP_ADDRESS> --cluster-domain=cluster.local --cni-s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerdr-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.availablhealthz-bind-address=<IP_ADDRESS> --hostname-override=v0003773.kne.bund.drv --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels= --podcher/k3s/agent/pod-manifests -protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/servte-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
[root@v0003773 ~]#
Chart details for agent node : https://github.com/rancher/security-scan/blob/master/package/cfg/k3s-cis-1.6-hardened/node.yaml
Repro steps:
Create the K3S cluster with one master and one agent node. Use the doc for hardening the nodes(https://rancher.com/docs/k3s/latest/en/security/hardening_guide/,
https://rancher.com/docs/k3s/latest/en/security/hardening_guide/,
       https://rancher.com/docs/rancher/v2.6/en/cis-scans/

 2. Install the CIS benchmark 1.6 chart from the Apps and market place.

 3. Run the scan and see the result (With the profile k3s-cis-1.6-profile-hardened)

Workaround:
Is work around available and implemented? No
What is the workaround: No workaround so far
Actual behavior:
CIS scan is not working for agent nodes where there is no issue with the Master node scanning.
Expected behaviour:
CIS scan should work for both master and agent nodes.
Files, logs, traces:
Attached
Additional notes:
This needs to be adjusted in chart for agent nodes.

Tracking internally as SURE-4123


GH-43535: [C++] support the AWS S3 SSE-C encryption

Rationale for this change
What changes are included in this PR?
Extend S3Options to support server-side encryption with customer-provided keys (SSE-C).
Are these changes tested?
Yes

Are there any user-facing changes?
yes, the new member sse_customer_key added for S3Option

GitHub Issue: #43535

@pitrou @mapleFU @felipecrv can you help to review the PR, Thanks

I took a quick look and here are some assorted thoughts:

there are no unit tests
only the encryption key needs to be on S3Options, not the MD5 and algorithm string (which can be computed on the fly)

Also, for the other readers wondering, I've found a useful (though possibly outdated?) comparison of S3 encryption options here:
https://www.linkedin.com/pulse/delusion-s3-encryption-benefits-ravi-ivaturi/

I took a quick look and here are some assorted thoughts:

@pitrou Thanks for your review, for #1, i would try to add ut for the change, for #2, yes, actually ONLY the Key is needed, that's why i only expose one SetSSECKey function, and three Get function, and private the new added datamember.
do you suggest to only add one datamember for S3Options? but if there is only one memeber, we may needs to calculate everywhere?

@pitrou I've submitted another commit to only expose the ssec-key in S3Options, and then calculate the MD5 every time on the fly, can you help to review whether there are other comments? I would try to add the ut if there is no interface change required, Thanks

@pitrou Can you help to review the PR again?Thanks

@kou @mapleFU can you help to review the change, Thanks

In your PR, it appears that S3Options.sse_customer_key expects a base64-encoded private key. Instead, why not expect the raw 256-bit key?

It appears that boto3 (the officiel Python bindings for AWS) expects the 32-byte raw key, example:
https://github.com/boto/boto3/issues/1941#issuecomment-487211835

In your PR, it appears that S3Options.sse_customer_key expects a base64-encoded private key. Instead, why not expect the raw 256-bit key?

@pitrou though the boto3 exposed the 32-bytes key, the offcial c++ aws sdk expected the base64 encoded key, I would prefer to just passthrough the input to the aws sdk. Meanwhile, during  the integration between the arrow  and  my own project, using the base64encoded key may be a little more easy for integration/debug/troubleshooting purpose.
Anyway, if you insist on  exposing the 32 bit raw key, I'm also fine with that.
@kou @mapleFU  any insights?

If we accept raw key instead of base64 encoded key, we don't need base64 encoded key validation in CalculateSSECKeyMD5(), right?
If so, I like raw key approach.

Meanwhile, during the integration between the arrow and my own project, using the base64encoded key may be a little more easy for integration/debug/troubleshooting purpose.

BTW, why is this? Is it simply because your project already has base64 encoded key not raw key?

If we accept raw key instead of base64 encoded key, we don't need base64 encoded key validation in CalculateSSECKeyMD5(), right? If so, I like raw key approach.

Meanwhile, during the integration between the arrow and my own project, using the base64encoded key may be a little more easy for integration/debug/troubleshooting purpose.

BTW, why is this? Is it simply because your project already has base64 encoded key not raw key?

@kou readable ascii string may has some benefits, some system may needs to use the environment variable or even the Registry/Configure fille to configure the key, though the usage for the configure fie to store such a key is NOT the security practice, there do has the possibility to use the environment variable to store such information.
cc @pitrou

@pitrou @kou @mapleFU any more comments for the PR? Thanks

I'll be out until 9.21, sorry for delaying. I think kou and pitrou can well review this

@kou visible ascii string may has some benefits, some system may needs to use the environment variable or even the Registry/Configure file to configure the key, though the usage for the configure fie to store such a key is NOT the security practice, there do has the possibility to use the environment variable to store such information.

OK. I feel that decoding base64 data is a responsibility of application side not Apache Arrow C++. Because how to manage key is also a responsibility of application side.
Could you accept raw key not base64 encoded key?

@kou @pitrou I've changed the code to expect the 32 bytes raw key as the input. please review again.
Thanks

@kou @pitrou May i know is there any other comments for the change?

@kou Can you help to review and help to trigger the CI again? Meanwhile, I've rebased the main branch,thanks

Could you rebase on main?
Could you check S3 filesystem related CI failures?

@kou  The reason for the failure is, the minio test server is start with http, while the SSEC-Key would require https.
do you know why the minio server is started with http rather than https? is there any concern for the self-signed certificate?
I tend to start the minio server with self-signed certificate for the new test case only, and keep the current http case unchanged, do you think it's acceptable?Thanks

I think it's ok to switch the tests to access Minio using HTTPS with a self-signed certificate.

do you know why the minio server is started with http rather than https?

Just because it was easier :-) No particular reason otherwise.

We can always use HTTPS Minio.
Can we prepare a self-signed certificate with portable way? (Do we need openssl command for it?)

FYI: I created https://github.com/apache/arrow-flight-sql-postgresql/blob/main/dev/prepare-tls.sh for other Arrow related project but this is not for Windows...

There is no need to generate a self-signed cert everytime, it can just be stored in the repository.

In general, it's not a good idea.
If we add it to this repository, someone may report it a security vulnerability. We need to explain that it's not danger.
FYI: GH-6399
(If we don't have another approach, we can use the approach.)

If someone is silly enough to report a self-signed certificate used for testing purposes as a security vulnerability, then they deserve to be ignored.
CPython has stored a ton of fake certificates in its repository for decades and we don't get any complaints:
https://github.com/python/cpython/tree/main/Lib/test/certdata

@pitrou regarding to use the self-signed certificate file, i've serval questions that may needs your further confirm:
where should the certificate file be stored? I didn't see a proper resource folder to store such information, could it be  put it in the same folder as the test code, say arrow\cpp\src\arrow\filesystem\certs ?  Or, do you think it's acceptable to hard code the static string(certificate's content) in the test code, and then write the pem file into the tempory directly(in this way, it could avoid the relative path handling in different platform)
Thanks

Or, do you think it's acceptable to hard code the static string(certificate's content) in the test code, and then write the pem file into the tempory directly

Ah, yes, that's a good idea. You will need two strings: one for the private key, one for the cert.

Or, do you think it's acceptable to hard code the static string(certificate's content) in the test code, and then write the pem file into the tempory directly

Ah, yes, that's a good idea. You will need two strings: one for the private key, one for the cert.

yes, according to https://github.com/minio/minio/tree/0c53d860173e98f154ce9fc117725652bd90f840/docs/tls, the file name also need to be fixed.
Since you have no concern on this approach, i would try to do it in this way, Thanks

@kou @pitrou I've push the commit to start the minio with https, can you help to review and trigger the ci again, Thanks

@pitrou the previous ci failed due to the specific minio version could NOT accept the --certs-dir at the end of the args,I could reproduce the failure locally, and after upgrade the minio to the latest, the failure  would disappear. Meanwhile, the minio version is two years ago, i wonder whether we should upgrade the minio version regularly
Can you help to trigger the CI action again, Thanks

bash-5.1$ ./minio.2022 server --quiet --compat --address <IP_ADDRESS>:9000 /home/admin/minio --certs-dir /home/admin/minio
ERROR Invalid command line arguments: Incorrect number of endpoints provided [/home/admin/minio --certs-dir /home/admin/minio]
> Please provide an even number of endpoints greater or equal to 4
HINT:
For more information, please refer to https://docs.min.io/docs/minio-erasure-code-quickstart-guide
bash-5.1$ ./minio.2024  server --quiet --compat --address <IP_ADDRESS>:9000 /home/admin/minio --certs-dir /home/admin/minio
WARN: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables

For the record: https://github.com/apache/arrow/pull/44225 is bumping the Minio version, this PR could wait for it

For the record: #44225 is bumping the Minio version, this PR could wait for it
Thanks, I've move the --certs-dir parameters in the mid of the args, suppose this is also supported by the old minio.
Meanwhile, I've also set the global option to trust the self-signed certificate, can you help to trigger the CI pipeline again.
Thankyou

continuous-integration/appveyor/pr

@kou Can you help to review and trigger the CI again, Thanks

Triggered.
FYI: You can run CI on your fork by enabling GitHub Actions on your fork.

Triggered.
FYI: You can run CI on your fork by enabling GitHub Actions on your fork.

Cool, glad to know that!

D:/a/arrow/arrow/cpp/src/arrow/filesystem/s3_internal.h:327:11: error: 'class Aws::S3::Model::CompleteMultipartUploadRequest' has no member named 'SetSSECustomerKeyMD5'

@kou can you give some hint/guidance on how to check the version of the aws sdk in arrow, if there are some other similar example code, that would be much preferred. honestly, i'm a freshman to the arrow, Thanks.

@kou can you give some hint/guidance on how to check the version of the aws sdk in arrow, if there are some other similar example code, that would be much preferred. honestly, i'm a freshman to the arrow, Thanks.

We have macros for this, example:
https://github.com/apache/arrow/blob/61c99a5bc05a7436a468c4420b9b8d30fb0307ea/cpp/src/arrow/filesystem/s3fs.cc#L100

@pitrou Can you help to review the last commit, Thanks.

[ RUN      ] TestS3FS.GetFileInfoGeneratorStress /arrow/cpp/src/arrow/filesystem/s3fs_test.cc:949: Failure Failed 'AllFinished(tasks)' failed with IOError: When completing multiple part upload for key '0/53' in bucket 'stress': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. /arrow/cpp/src/arrow/filesystem/s3fs.cc:1830  CleanupIfFailed(FinishPartUploadAfterFlush()) [  FAILED  ] TestS3FS.GetFileInfoGeneratorStress (2969 ms)
@kou @pitrou do you have any insight on the error? suppose the workflow in this case should NOT impacted by the newly added SSEC feature, may it caused by the minio with tls could not support so much concurrency write/read?

@kou can you help to trigger the CI again, the latest error seems related to the specific ci environment.
https://github.com/apache/arrow/actions/runs/11304742428/job/31457900241?pr=43601
`WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
python3 is already the newest version (3.12.3-0ubuntu2).
python3-dev is already the newest version (3.12.3-0ubuntu2).
python3-pip is already the newest version (24.0+dfsg-1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
Error: Process completed with exit code 1.
0s`

They are unrelated to this change. They are caused by becoming ubuntu-24.04 as ubuntu-latest. Could you open a new issue for these failures?

@kou  Thanks, I've submited https://github.com/apache/arrow/issues/44396 to report the issue.

@kou seems the https://github.com/apache/arrow/issues/44396 has been solved, can you help to trigger the CI again, or do i need to rebase the main branch, Thanks.

@kou The failure in https://github.com/apache/arrow/actions/runs/11357304238/job/31590378994?pr=43601 seems caused by the instable networkaccess to the https://auth.docker.io/token, is it possible to trigger this specific pipeline again?Thanks
 DEBUG:archery:Executing['docker', 'compose', '--file=/home/runner/work/arrow/arrow/docker-compose.yml', 'build', '--build-arg', 'BUILDKIT_INLINE_CACHE=1', '--progress=plain', 'ubuntu-cpp']  --progress is a global compose flag, better usedocker compose --progress xx build ...
#0 building with "default" instance using docker driver
#1 [ubuntu-cpp internal] load build definition from ubuntu-24.04-cpp.dockerfile
#1 transferring dockerfile: 7.06kB done
#1 DONE 0.0s
#2 [ubuntu-cpp auth] amd64/ubuntu:pull token for registry-1.docker.io
#2 DONE 0.0s
#3 [ubuntu-cpp internal] load metadata for docker.io/amd64/ubuntu:24.04
#3 ERROR: failed to authorize: failed to fetch oauth token: unexpected status from POST request to https://auth.docker.io/token: 502 Bad Gateway
failed to solve: amd64/ubuntu:24.04: failed to resolve source metadata for docker.io/amd64/ubuntu:24.04: failed to authorize: failed to fetch oauth token: unexpected status from POST request to https://auth.docker.io/token: 502 Bad Gateway

[ubuntu-cpp internal] load metadata for docker.io/amd64/ubuntu:24.04:

Error: docker compose --file=/home/runner/work/arrow/arrow/docker-compose.yml build --build-arg BUILDKIT_INLINE_CACHE=1 --progress=plain ubuntu-cpp exited with a non-zero exit code 17, see the process log above.
The docker-compose command was invoked with the following parameters:
`

@kou @pitrou Can you help to review the PR again, Thanks a lot.

Could you update the PR description to reflect the current implementation?

@kou do you think we can merge the PR now. Those requested changes from @pitrou  are also resolved.

I'll wait for re-review from pitrou in this week.
I'll merge this in the next week if nobody reviews this.

@github-actions crossbow submit -g cpp

@pitrou is it possible to help review the change again, Thanks.

@pitrou Can you help to take a review again, let me know if there is any concerns, thanks

@kou Do you think we could merge the pr since there is no response from @pitrou  for two weeks.

I'll merge this in the next week if nobody objects it.

@github-actions crossbow submit -g cpp

@github-actions crossbow submit -g cpp


remote write: Handle float histogram and gauge histograms (and float gauge histograms ;)

Once #11199 and #11200 are done, we also have to propagate float histograms and gauge histogram (and the combination of both) via remote write. Since this is all prepared in the protobuf already, it should be a fairly trivial change. (Maybe the code is even written in a way that it will "just work" once TSDB supports both properties. Still need to make sure, therefore this issue.)

Looks like support for float histograms was added in https://github.com/prometheus/prometheus/pull/11817 so it's just support for gauge histogram needed now

Yes, I believe this is all done. @codesome if you find the time, please verify. But I'm confident enough to close this as done.


Fixed eigen3 version.

The CONTROL file said 3.3.0 but the version being downloaded is 3.3.3. This updates the CONTROL file to 3.3.3.

@dada2048,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

Good catch, thanks!


Move mocks into folders in package directories

How to categorize this PR?

/area dev-productivity
/kind cleanup
/priority normal
What this PR does / why we need it:
Instead of having mocks in ./pkg/mock/... representing the directory structure we are now generating the mocks in dedicated folders in the package directories themselves (i.e., if package foo is in ./pkg/path/to/foo then the mock would be in pkg/path/to/foo/mock instead of ./pkg/mock/gardener/path/to/foo.
Hence, the ./pkg/mock/gardener folder is eliminated completely. Only the mocks for third-party/vendored packages remain in ./pkg/mock now.
Special notes for your reviewer:
/cc @BeckerMax
Release note:

The mocks for Gardener packages were moved to dedicated folders in the respective package directories, i.e., if there is package `foo` in `./pkg/path/to/foo` then the mock would be in `pkg/path/to/foo/mock` instead of `./pkg/mock/gardener/path/to/foo`. Only the mocks for third-party/vendored packages remain in `./pkg/mock`.

/squash

https://github.com/gardener/gardener/pull/3641 will conflict with this.
Can we merge this PR, so that I can rebase https://github.com/gardener/gardener/pull/3641?

I think we can, it's only a little restructuring of the mocks. So I'll merge it, please rebase your PR :) Thanks!


test_chat.ipynb halts with missing 'self.exclude_ida' in file wangchanglm.py

Your test code halts with missing 'exclude_ids' in the file wangchanglm.py.
Also see here, AttributeError: 'WangChanGLM' object has no attribute 'exclude_ids'
Any solution available?
jupyter lab notebooks/test_chat.ipynb
http://localhost:8888/lab/tree/test_chat.ipynb
 93     if thai_only:
 94         output_tokens = self.model.generate(
 95             input_ids=batch["input_ids"],
 96             max_new_tokens=max_new_tokens, # 512

---> 97             begin_suppress_tokens = self.exclude_ids,
98             no_repeat_ngram_size=no_repeat_ngram_size,
99             #oasst k50
100             top_k=top_k,
101             top_p=top_p, # 0.95
102             typical_p=typical_p,
103             temperature=temperature, # 0.9

Thanks bot for your reply. Run the code sample test code provided on the github site as jupyter lab notebooks/test_chat.ipynb. The file is located here: https://github.com/PyThaiNLP/pythainlp/blob/dev/notebooks/test_chat.ipynb. As I pointed out earlier, the code malfunctions. The test code halts with missing a 'exclude_ids' in the file wangchanglm.py.
The error is also described as, AttributeError: 'WangChanGLM' object has no attribute 'exclude_ids' If you can not find the file, and can not test for the error by simply running the exisiting test code, don't bother responding to this error report. Screen prints, with pixels, to help you - will not help you. There is a minimum skill level here.


If possible can we have a button to update metadata so that when user switches from cennznet network it works fine

It sometimes throws bad signature issue. which can be prevented by a updating meta from UI..
Can be solved once - https://github.com/cennznet/extension/pull/27 and https://github.com/cennznet/api.js/pull/398 are merged..and extensions new version is release on chrome/ff
but if we have update meta button, won't need to wait for other things..

let's fix the extension, the update metadata thing is not a nice ux


[FLINK-28869] Emit a warning event for ClusterDeploymentException

What is the purpose of the change
Emit warning events for ClusterDeploymentException
Brief change log

Trigger a K8s warning event when ClusterDeploymentException occurs.

Verifying this change

This change added tests
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changes to the CustomResourceDescriptors: (no)
Core observer or reconciler logic that is regularly executed: (yes / no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Please add a test case for this to the FlinkDeploymentControllerTest . There are other examples there that test event triggering.

ut added :)


use native RTCPeerConnection promises in Chrome 51

Description
does not attempt to shim promises in Chrome 51+
Purpose
make adapter superflous

cc @jan-ivar

getStats-promises are still shimmed

I did a git reset, cherry-picked jib's stuff, then reapplied my change. I think that should merge without problems... other than confusing @googlebot of course :-)

Merged his change but a conflict arose, I can fix it but rather you do it since it's your PR ;)

LGTM


LS: Make diagnostics state be FileId based instead of Uri

Stack:

#6615
#6614 ⬅
#6613

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.


CI: Debug upgrade test hang

Description
WIP
Checklist

[ ] Investigated and inspected CI test results

Testing Performed
CI is sufficient

/test gke-upgrade-test

/test gke-upgrade-tests

/test gke-upgrade-tests

/swan

/honk

Welp, I didn't realize auto-merge was on 🙃


Typo in section "Critical Path Rendering"

The section says "Do you understanding" when it should say "Do you understand".

Do you understanding.


What is "email_in_database" and how to use it?

I'm getting error using devise on master branch.
Rails 5.1
Ruby 2.4.1

You need to give us more information on how to reproduce this issue, otherwise there is nothing we can do. Please read CONTRIBUTING.md file for more information about creating bug reports. Thanks!

Sorry about that.
I've just searched in devise. I see email_in_database is used here https://github.com/plataformatec/devise/blob/master/lib/devise/models/confirmable.rb#L260
But cannot find where it's defined.

In Rails. It was named email_was before Rails 5.1.


Demo: Web frontend update and addition of demo attack scripts for it to call via XMLRPC

Ensure that the most recent version of the web frontend code (external, on git.uptane.org) works with demo code here.
Write functions here to execute the attacks @vladimir-v-diaz wrote into the README for demo_director and demo_image_repo.
Register those functions as XML-RPC calls.
Call those functions appropriately from the web frontend
Adjust buttons and labels and so on on the web frontend for those attacks

Would it be possible to move, or mirror, the web frontend code available on git.uptane.org?

I think that would be best, personally, but this is up to @cameronmott and @eacain Allen. (I don't know his GitHub handle.)

I'd expect mirroring to be a quick and easy way to do it, and possibly in the long run determine if a migration makes sense. I'll look into it.

Pushing it here looks like this:

On GitHub, create uptane/uptane_web_app (or whatever name)
From a local repository checked out from https://git.uptane.org/eacain/uptane_web_app, do this:

git remote add mirror_on_github https://github.uptane.org/uptane/uptane_web_app (or whatever name)
git push mirror_on_github


Add Custom package-info Descriptions Configuration

Adds a new Autorest configuration custom-package-info-descriptions which allows for a map of packageName-packageInfoDescription to be passed to code generation which overrides the default package-info javadoc descriptions.
Also fixes a bug with stream-style serialization for Unix long times and Unix datetime times where they didn't have serialization mappings configured.

Closing as this won't be done and code customizations will be used instead. In the future there are plans to make this a formal part of code generation instead of a one-off configuration.


fix(core): use -mfr only for cc2022+

This fixes a regression introduced in the PR #656 two years ago, it causes invalid argument -mfr error when trying to render with aerender version older than 2022.
In #656 @chrisapplegate added new commandLineRenderer-2022.jsx for AE 2022+, and kept commandLineRenderer-default.jsx for pre-2022, they were patched correctly. But Chris forgot to check the version and applied the cc2022-only argument -mfr in all cases, resulting a regression.
It seems this regression has lasted a long time and no one noticed. Recently I tried to upgrade the dependencies of my project which stays at using old version of aerender and found that a -mfr error throws when I update @nexrender/core to any version greater then 1.32.x. #656 was shipped in 1.33.0, all versions of @nexrender/core starting from 1.33.0 are buggy.

Oh, great catch! Amazing job


Support openwrt-23.05 on armv7l

Hi,
I have a Netgear Orbi RBK50 pair, and I would like to use OpenWrt with Pi-Hole.
These devices use an armv7l processor, and openwrt-23.05 can be installed on them.
https://openwrt.org/toh/hwdata/netgear/netgear_rbr50
I just tried to install your Pi-Hole packages and encountered these:

[ ] architecture armv7l is not available
[ ] signature file download failed
[ ] package cron is not available
[ ] package net-tools is not available

I tired to work around these and see how far I can get:

use armv7sf-k3.2 architecture
disable check_signature option

and finally I have been blocked: Package pi-hole version 2024.08.06-134720-c6ff1d52 has no valid architecture, ignoring.
Can you help me contribute back the needed changes ?

What prevents you from installing Entware on your device?
With this package I'm targeting Entware, not OpenWRT.

Hi,
I am just discovering the whole Entware/OpenWrt landscape...
I just added Entware sources to opkg to provide the missing packages.
src/gz entware https://bin.entware.net/armv7sf-k3.2
Still the install does not go further with many 'xxx has no valid architecture, ignoring.' messages, then fails: Unknown package 'pi-hole'.
I finally found https://github.com/Entware/Entware/wiki/Alternative-install-vs-standard, which installation type do you recommend for Pi-Hole?

Entware != OpenWRT
The package repositories are not compatible

Thank you for you help and the good work you do and share. I will continue on discovering Entware and OpenWRT.


[Search v1] Add empty search List skeleton

Details

introduces new EmptySearchView component that displays skeleton view of search list loading
refactored OptionsListSkeletonView partially according to design doc to extract some common behavior to a component
temporarily left commented calls to EmptySearchView because otherwise there is no way to see how this behaves
only after the SearchPage actually displays results we will correctly use the new skeleton component

Fixed Issues

$ https://github.com/Expensify/App/issues/39882
PROPOSAL:
Tests

This adds a new component to be displayed when there are no search results, but the search doesn't yet work, so the only way to test is to temporarily display the component.
go to SearchPage and SearchPageBottomTab and uncomment the usages of new component to see if it works

Offline tests

QA Steps

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
https://github.com/Expensify/App/assets/3929868/20ccac81-48d5-4ec1-9b35-347de731c3ba

Android: mWeb Chrome

iOS: Native
https://github.com/Expensify/App/assets/3929868/424d86e6-17eb-427f-ad01-2541c90ad64d

iOS: mWeb Safari

MacOS: Chrome / Safari
https://github.com/Expensify/App/assets/3929868/4f27bc14-b082-4d90-957c-d094241fa8a3

MacOS: Desktop

No need for C+ review on this one. We're still developing the feature and this is unused in App.

🚀 Deployed to staging by https://github.com/luacmartins in version: 1.4.65-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/mountiny in version: 1.4.65-5 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


WireMock with JUnit 5

In JUnit 5, @Rule and @ClassRule is superseded by @ExtendWith, which requires extension class.
And it seems like extension for WireMock is not yet supported.
Is there any alternative way to use WireMock with stubbing and verification via JUnit 5?

You can programmatically start and stop WireMockServer. That's all the rule is doing really.
I'll mark this as a feature request as JUnit 5 support should start to be a priority soon.

Thank you for your comment, and marking.
Actually, I tried to use WireMock with JUnit, and there was no other way to do it without using JUnit 4.x Rule.
Thanks !

You can use  @Before and @After annotated methods to start and stop a WireMockServer in JU4.

Some of WireMock's own acceptance tests actually do this in their base class, so might be worth checking that out if you're not sure what I mean.

Thanks for your kind comment. I'll check it.

Please look at https://github.com/lanwen/wiremock-junit5 (will be available in maven central soon)

@lanwen nice! Could you do this as class level fields instead of (or additionally) method parameters?

@sta-szek Like a class rule?

@lanwen exactly - just to simplify as much as possible.

@lanwen thanks for the nice code!

https://github.com/JensPiegsa/wiremock-extension

What the difference?

It's an approach that uses annotated fields. See ExampleTest

@lanwen can you help me how to wiremock-junit5 for custom port.
As in your code, as per my understanding, we can get any random port, but if you want a specific port for wiremock, can you specify how to do that.
void<EMAIL_ADDRESS>WireMockServer server) throws IOException {
....
}
but in this, i can't set server port for which server need to start,
I was checking the code of wiremockResolver, this was the code,
this.server = new WireMockServer(((WiremockConfigFactory)mockedServer.factory().newInstance()).create());
i guess, due to this.. i am unable to specify some specific port.
Is there is any way to do that ?
Please unicast me

@ghilainm for me rewriting rule to Junit5 extension looks like this: https://github.com/jmayday/wiremock/commit/908cbb20d19cf7dcabcca3de28a2712fd895e472 (I'm not sure if it satisfies requirements for pull request though, @tomakehurst?)
As server starts on random port, we need to build also a webclient with correct url. It might be achieved in beforeAll method:
  @BeforeAll
  static void setup(@ServerUrl String serverUrl) {
    // build webclient using server url
  }

I'm a bit reticent about making WireMock dependent on both JUnit 4 and 5. Dropping 4 isn't an option at the moment as it's still pretty widely used.
Do you think there's much disadvantage in adding JUnit 5 support via a separate library?

Do you think there's much disadvantage in adding JUnit 5 support via a separate library?

Yes, an official but optional library should be quite helpful.

What about splitting Wiremock library in two -- wiremock-core and wiremock-junit4 and adding wiremock-jupiter for JUnit 5 support?
The wiremock library could then simply have a dependency on core and junit4 for the sake of legacy users?

@luolong that doesn't sound much different from what we have now, if you consider @lanwen's library to be the jupiter version.

Having an 'official' Wiremock JUnit5 extension would be valuable. I actually expected to find a JUnit 5 extension when I have a JUnit 4 rule. And I think an optional dependency on JUnit 5 would be no problem at all, would it?

@tomakehurst Sure, I realize that this is not so much different from what you have now. This is by design.
The only difference is that JUnit 4 dependency is explicitly divorced from the core dependency.
The separate modules could even share the same repository and release schedule.
Should not be difficult to achieve.

I'm coming around to the idea of making both JUnit 4 and 5 optional/provided dependencies. I'm going to look at this (with the usual caveats about being busy and it not happening quickly).

Any update on this?
Currently, I am using @JensPiegsa implementation, but would be great to have an "official" artifact from Wiremock repository.

@tomakehurst Is this something any of us could help with? I can imagine splitting out the junit4 dependencies can be a hairy task with some architectural impact, so I would understand if you want to do it yourself since Wiremock is your baby :-)
But otherwise, let us know!

@fransflippo @tomakehurst I would recommend using:
@AutoConfigureWireMock from our friends from spring's 'org.springframework.cloud:spring-cloud-contract-wiremock' - it's working great with Junit5 and you can customize it it you want :)

@fransflippo @tomakehurst I would recommend using:
@AutoConfigureWireMock from our friends from spring's 'org.springframework.cloud:spring-cloud-contract-wiremock' - it's working great with Junit5 and you can customize it it you want :)

Thanks @blocha , will take a look!

@fransflippo the JUnit 4 dependency is already optional, so it might be quite straightforward to do something similar with JUnit 5.
I'd definitely welcome a PR to add this if you'd like to contribute.
I like the approach @lanwen has taken i.e. creating JUnit 5 extensions to inject WireMock classes as test method parameters, although I'd be inclined to inject an instance of WireMock rather than WireMockServer.

Canned rules are sometimes more trouble than they are worth. My workaround:
private WireMockServer wireMockServer;

@BeforeEach public void proxyToWireMock() {
    wireMockServer = new WireMockServer(WireMockConfiguration.options()/*.port(etc)*/);
    wireMockServer.start();
}

@AfterEach public void noMoreWireMock() {
    wireMockServer.stop();
    wireMockServer = null;
}

There is one aspect the rules were quite good at that are not possible anymore with @lanwen's or @JensPiegsa's implementations: Having more than one mock server. We have one server that talks to several others, and having different mock servers made the tests more readable. Theoretically, we can only use one mock server, but then it is hard to understand where each request is supposed to be sent.
We could also do this like @jglick suggests, but this scales badly, too.

@fdw Of course, there is support for multiple server instances. Please have a look at the example.

@fdw this is a very good point, and I think any solution should attempt to cleanly support multiple instances. I haven't looked at JUnit5 extension architecture at all closely, so I'm open to suggestion as to how this could be done.

@JensPiegsa You are right, I hadn't seen that example. Thank you :)

If it helps anyone, we are doing multiple wiremock servers with this single extension class:
class WireMockExtension extends WireMockServer implements BeforeEachCallback, AfterEachCallback {

    @Override
    public void beforeEach(ExtensionContext context) {
        this.start();
    }

    @Override
    public void afterEach(ExtensionContext context) {
        this.stop();
        this.resetAll();
    }
}

Use it with multiple servers in a test like this:
        @RegisterExtension
        WireMockExtension mockServer1 = new WireMockExtension(8081)

        @RegisterExtension
        WireMockExtension mockServer2 = new WireMockExtension(8082)

I'm getting this odd error, if I try to use wiremock with JUnit 5...
@AfterAll
public void after() throws Exception {
	super.tearDown();
	wireMockRule.shutdown();
}

error: cannot access MethodRule
[ERROR]   class file for org.junit.rules.MethodRule not found
It only happens if I try to do the shutdown.  Is there another way I can shut down the server without triggering this error?

ahh, how silly.  I had copied and pasted some snippits from demos, and was trying to do new WireMockRule(8089) instead of new WireMockServer(8089), which obviously hits a class that has dependencies on the Junit 4 API.

@jterhune In my case I also had to add WireMock.configureFor to your solution:
 @Override
    public void beforeEach(ExtensionContext context) {
        this.start();
        WireMock.configureFor("localhost", port());
    }

Otherwise WireMock used port 8080 for admin mappings and can't connect to it the tests.

Hello,
If you're ok with that, I can try to add this feature in WireMock.
Recently, I have setup WireMock in a project at work. I was a little bit surprised that this is not already part of WireMock and we need to add an other library to add this kind of extension.
First at all, I took a look of some repositories such as :

https://github.com/lanwen/wiremock-junit5
https://github.com/JensPiegsa/wiremock-extension
https://github.com/phxql/wiremock-junit5
...

It gave my some inspiration but I found that each extensions had some missing features. For example, none of these 3 JUnit extensions allow you to have an alternative between constructor injection, field injection or method injection.
For the configuration, somes extension didn't seem very clear to me. However, there are great ideas in each of these community extensions.
I can try to put all ideas together and maybe to add new one to have an official JUnit 5 extension for WireMock.
What do you think of that ?

If you're willing to work on this, that would be awesome. JUnit 5 is very high on my priority list, but I've just not had time to devote to it.
Happy to provide support and design input as you need it.

Great to see @Patouche step in to add a JUnit Jupiter extension 🚀
In the meantime and for those who are looking for a solution to start WireMock before all test methods (as far as I can see it, no community exertion allows this as they are implementing the BeforeEachCallback) especially when writing test that involve a SpringTestContext where one wants to set e.g. the base URL prior to starting the application, I used the following setup over the past months:
public class WireMockInitializer
  implements ApplicationContextInitializer<ConfigurableApplicationContext> {

  @Override
  public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
    WireMockServer wireMockServer = new WireMockServer(new WireMockConfiguration().dynamicPort());
    wireMockServer.start();

    configurableApplicationContext
      .getBeanFactory()
      .registerSingleton("wireMockServer", wireMockServer);

    configurableApplicationContext.addApplicationListener(applicationEvent -> {
      if (applicationEvent instanceof ContextClosedEvent) {
        wireMockServer.stop();
      }
    });

    TestPropertyValues
      // override any base URL of your HTTP clients here
      .of(Map.of("todo_base_url", "http://localhost:" + wireMockServer.port()))
      .applyTo(configurableApplicationContext);
  }
}

With the initializer above you'll start WireMock as part of the bootstrapping phase of the Spring Context, register it as a Spring Bean and can override base URLs for your HTTP clients.
Next, you need to activate/include this initializer with @ContextConfiguration:
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ContextConfiguration(initializers = {WireMockInitializer.class})
class TodoControllerIT {

  @Autowired
  private WireMockServer wireMockServer;

  @AfterEach
  public void afterEach() {
    this.wireMockServer.resetAll();
  }
}

You can find a detailed explanation as part of this blog post.

Hello,
If you're ok with that, I can try to add this feature in WireMock.
Recently, I have setup WireMock in a project at work. I was a little bit surprised that this is not already part of WireMock and we need to add an other library to add this kind of extension.
First at all, I took a look of some repositories such as :

https://github.com/lanwen/wiremock-junit5
https://github.com/JensPiegsa/wiremock-extension
https://github.com/phxql/wiremock-junit5
...

It gave my some inspiration but I found that each extensions had some missing features. For example, none of these 3 JUnit extensions allow you to have an alternative between constructor injection, field injection or method injection.
For the configuration, somes extension didn't seem very clear to me. However, there are great ideas in each of these community extensions.
I can try to put all ideas together and maybe to add new one to have an official JUnit 5 extension for WireMock.
What do you think of that ?

So, any news?

@tomakehurst I just gave it a try. Unfortunately, the beta version is not available on central and JitPack has issues building the project. So I downloaded the JAR and added it to my local repo:
mvn install:install-file \
        -Dfile=wiremock-jre8-standalone-2.31.0-beta.jar \
        -DgroupId=com.github.tomakehurst \
        -DartifactId=wiremock-jre8 \
        -Dversion=2.31.0-beta \
        -Dpackaging=jar

Works like a charm, almost like a drop-in replacement – great work!
Side note: it's JUnit 5, so you can remove the public modifiers from the docs.

Thanks for the feedback @beatngu13!
It's on my list to get JitPack working so should be easier with future betas.

@tomakehurst one suggestion: maybe use Options#DEFAULT_PORT (i.e. 8080) here in the extension, rather than defaulting to a random port. Then, the extension IMHO better aligns with WireMock's default behavior and the existing JUnit 4 rules; also users don't have to "fiddle" with @RegisterExtension and can use the more popular @ExtendWith annotation. (Convention over configuration, so to say…)

I agree it's less surprising to default to 8080, but OTOH it prevents you from running your tests in parallel without switching everything over to the programmatic form, which you tend to want to do only when you've got 100s of tests and your build is starting to feel slow.
Also, defaulting to the random port nudges developers to use the WireMock's base URL methods rather than "http://localhost:8080" + myPath, which seems like better practice to me.
I'm genuinely in two minds though, not least because I can see myself spending the next 5 years telling every other mailing list poster about the random port.

There's another possibility, arguably a middle way.
I've been thinking it'd be good to support automatic JVM proxy setup in a similar fashion to Hoverfly, whereby forward proxying is enabled and the JVM system properties for proxy config are set to point to it so that you can mock multiple domains in a single instance.
I wouldn't want to turn this on by default, but it'd be nice to have it available declaratively so I was wondering about creating a subclass to the extension that turns this option on e.g.
@ExtendWith(WireMockExtensionWithJvmProxy.class)

Perhaps we could also have something like:
@ExtendWith(WireMockExtensionRandomPort.class)

or if random port is kept as the the default:
@ExtendWith(WireMockExtensionPort8080.class)

Thoughts?

Ah, I see. Using a random port definitely makes sense when executing the tests concurrently. You also know your user base way better than I do and what default makes more sense.
Regarding WireMockExtensionRandomPort and WireMockExtensionPort8080, you could also do something like:
@Target( ElementType.TYPE )
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(WireMockExtension.class)
public @interface WireMockTest {

    // null means random port
    Integer httpPort;

    // more core config properties

}

Which can be used like:
@WireMockTest
class DeclarativeWireMockTest { }

If a static port is desired:
@WireMockTest(httpPort = 8080)
class DeclarativeWireMockTest { }

Yes, that's much nicer isn't it. I'll go with that.

I've updated the branch to include this ☝ now.
https://github.com/tomakehurst/wiremock/blob/junit-jupiter/docs-v2/_docs/junit-jupiter.md
I'm still trying to figure out a better way to publish a beta build than putting the standalone JAR on Releases. So far:

JitPack won't work because there are build steps that rely on NPM.
GitHub Packages won't work because it won't let you make Maven repos public (unauthenticated).
Sonatype's OSS snapshots repo won't work because it's giving me unexplained 400 errors when I try to publish to it.

If anyone has any other ideas I'm all ears...

Sonatype's OSS snapshots repo won't work because it's giving me unexplained 400 errors when I try to publish to it.

Seems to be the client side problem. I have snapshots publishing in a few projects and it used to work fine.
Do you have some CI logs to take a look (and configuration in a branch)?

@szpak there's nothing useful in the logs really, just Could not PUT 'https://oss.sonatype.org/content/repositories/snapshots/com/github/tomakehurst/wiremock-jre8/2.31.0-beta-2/wiremock-jre8-2.31.0-beta-2.jar'. Received status code 400 from server: Bad Request.
Do you have a Gradle build file with working sonatype snapshot publishing you could share so I can compare?

OK, looks like the issue is that you have to actually have SNAPSHOT in the version (duh!).
I've pushed a 2.31.0-SNAPSHOT to the Sonatype snapshots repo now for those wanting to try it out. You'll need to use https://oss.sonatype.org/content/repositories/snapshots/ as the repo URL in your build.

OK, looks like the issue is that you have to actually have SNAPSHOT in the version (duh!).

Yup, they have separate repos for snapshot and release versions and are strict about content. I'm glad you solved it!

Thanks for this! Just playing with the new SNAPSHOT build - looks like the noargs constructor on WireMockExtension is private and the example docs have newInstance() calls to get an instance programatically.
It might be good to make the API more "friendly" for programmatic use? Perhaps include some static factory methods or a builder?

There seems to be some issue with the setup of Gradle for the JUnit BOM. If you look at the pom you can see the following
<dependency>
    <groupId>org.junit</groupId>
    <artifactId>junit-bom</artifactId>
    <version>5.7.1</version>
    <scope>runtime</scope>
</dependency>

i.e. it adds the BOM as the runtime dependency rather than in the dependencyManagement section and the scope is runtime not import.
Not sure if this is a Gradle bug, but it makes Gradle builds unhappy (running Gradle 7.2)
> Could not resolve all files for configuration ':testRuntimeClasspath'.
   > Could not resolve org.junit:junit-bom:5.7.1.
     Required by:
         project : > com.github.tomakehurst:wiremock-jre8:2.31.0-SNAPSHOT:20210829.210337-1
      > No matching variant of org.junit:junit-bom:5.7.2 was found. The consumer was configured to find a runtime of a library compatible with Java 16, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm' but:
          - Variant 'apiElements' capability org.junit:junit-bom:5.7.2:
              - Incompatible because this component declares an API of a platform and the consumer needed a runtime of a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 16)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about org.jetbrains.kotlin.platform.type (required 'jvm')
          - Variant 'enforcedApiElements' capability org.junit:junit-bom-derived-enforced-platform:5.7.2:
              - Incompatible because this component declares an API of an enforced platform and the consumer needed a runtime of a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 16)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about org.jetbrains.kotlin.platform.type (required 'jvm')
          - Variant 'enforcedRuntimeElements' capability org.junit:junit-bom-derived-enforced-platform:5.7.2 declares a runtime of a component:
              - Incompatible because this component declares an enforced platform and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 16)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about org.jetbrains.kotlin.platform.type (required 'jvm')
          - Variant 'runtimeElements' capability org.junit:junit-bom:5.7.2 declares a runtime of a component:
              - Incompatible because this component declares a platform and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 16)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about org.jetbrains.kotlin.platform.type (required 'jvm')

This is because Gradle "knows" that the BOM is a "platform" but it's being added as a dependency.
Excluding the dependency from Wiremock makes things happy
    testImplementation("com.github.tomakehurst:wiremock-jre8:2.31.0-SNAPSHOT") {
        exclude(group = "org.junit", module = "junit-bom")
    }

Thanks for reporting this.
Would you mind sharing your Gradle build file (or enough of it) so I can reproduce this?
Also, does this happen with Java versions < 16?

It's publicly available on the competing code hosting platform - https://gitlab.com/bmorris591/volvooncall-mysql
I've pushed up a branch without the exclusion to show the error, mainline has the exclusion and compiles happily.
CI/CD shows the error

Thanks for the detailed info @bmorris591!
I think I've figured out the solution, so I'll push another snapshot tomorrow.

@bmorris591 I could fix the Gradle synchronization with ::exclude, but if I try to run a test, I still get:

e: /home/elect/IdeaProjects/gradle-download-task/src/test/kotlin/de/undercouch/gradle/tasks/download/testBases.kt: (89, 22): Cannot access 'org.junit.rules.MethodRule' which is a supertype of 'com.github.tomakehurst.wiremock.junit.WireMockClassRule'. Check your module classpath for missing or conflicting dependencies
e: /home/elect/IdeaProjects/gradle-download-task/src/test/kotlin/de/undercouch/gradle/tasks/download/testBases.kt: (89, 22): Cannot access 'org.junit.rules.TestRule' which is a supertype of 'com.github.tomakehurst.wiremock.junit.WireMockClassRule'. Check your module classpath for missing or conflicting dependencies
e: /home/elect/IdeaProjects/gradle-download-task/src/test/kotlin/de/undercouch/gradle/tasks/download/testBases.kt: (94, 22): Cannot access 'org.junit.rules.MethodRule' which is a supertype of 'com.github.tomakehurst.wiremock.junit.WireMockClassRule'. Check your module classpath for missing or conflicting dependencies
e: /home/elect/IdeaProjects/gradle-download-task/src/test/kotlin/de/undercouch/gradle/tasks/download/testBases.kt: (94, 22): Cannot access 'org.junit.rules.TestRule' which is a supertype of 'com.github.tomakehurst.wiremock.junit.WireMockClassRule'. Check your module classpath for missing or conflicting dependencies

New 2.31.0-SNAPSHOT pushed. This seems to fix the issue in your project @bmorris591.

That got a green build when I reran the spike linked above. Mainline build has also completed successfully after merge.
Thanks!

Seems fixed and this works for me!
http://wiremock.org/docs/junit-jupiter/
At first I thought we still had to make our own extensions because this issue was still open. Posting the link above so others can find it too.

Good point, closed!


Fetch the data per page. Consecutive calls will fetch the next page

Use  ->setOperationInvoker(new LdapOperationInvokerForPaging()) to
activate this change. ->setUsePaging(true) is needed to get data per
page, otherwise all the data will be returned.
Example code:
$config = new Configuration();

// A domain configuration object. Requires a domain name, servers, username, and password.
$domain = (new DomainConfiguration('mydomain.com'))
   ->setBaseDn('dc=domain,dc=com')
   ->setServers(['<IP_ADDRESS>'])
   ->setPort(11111)
   ->setUsername('cn=admin,dc=domain,dc=com')
   ->setPassword('password')
   ->setOperationInvoker(new LdapOperationInvokerForPaging())
   ->setUsePaging(true)
   ->setPageSize(250);

$config->addDomain($domain);
// Defaults to the first domain added. You can change this if you want.

$ldap = new LdapManager($config);

$query = $ldap->buildLdapQuery();
$ldap_query = $query->select(['dn'])
   ->where(["objectClass" => "inetOrgPerson"])
   ->setBaseDn('ou=people,dc=domain,dc=com')
   ->getLdapQuery();

while ($users = $ldap_query->getArrayResult()) {
  print_r($users);  // or do other things
}

Note that caching hasn't been tested but I don't think it will work properly, so I guess it should be disabled when it's used with the new operation invoker.

Thanks for taking the time to work on this! I think the separate OperationHandler is the way to go. There should be one for paging interactively and one for other queries I guess. Though I think the separate operation invoker could be removed. You could add an option to the QueryOperation called pageInteractively or something like that and use that as a check in the supports() method of the OperationHandler. Then you just need to add the operation handler to the existing OperationInvoker. There's also a lot of duplication between the query operation handler and this. We could probably use a trait between the two for certain common things.
Anywho, I still need to take a closer look at this later.

I think I tried to go with a custom PagedQueryOperation but I discarded that approach because there were too many changes to do, which also involves more testing. There is also the problem of adding like getPagedLdapQuery or something like that to distinguish between a full query and a paged one, which also involves more code changes. Too many changes for someone that just scratched the code's surface 😃
The current approach is pretty much isolated and can be switch on or off just by changing the operation invoker, which I think it's quite handy.
Regarding the duplication, you're completely right. Something left to do. I just wanted to get the things done with the least amount of changes. That's good enough for me at the moment, but feel free to improve it.

Yeah, as I look back at this I wish I really would have split the query operations into two different ones. One for paging based operations and another for normal non-paged operations sigh :( Now there would be a lot to change to make that happen. Still doable I guess.
Another thing I just realized with this approach is that there doesn't seem to be a way to tell it that paging has ended if you get to a certain result and want to stop. It would also be nice to modify the paging size before the next result set is retrieved (such as in the loop).
I almost feel like a separate getPagedLdapQuery() would be the best route to go here as well. But that would probably require a lot of changes to the existing LdapQuery class, which is already in need of refactoring/improvement :/


frontail and nginx

Hi, I have a issue I'm not sure of how to solve.
I have a nginx server on <IP_ADDRESS>
and have the log file on server <IP_ADDRESS>  I just want to see..
it works fine on local machine to run frontail but via nginx it just shows med frontail webpage but not the output from that log file?
Help is much appreciated I would like to have this solution because it's very elegant one to show others... :-)

Solved:
server {
listen 80;
server_name your.domainname.com;     #change to your domain name
    location /frontail/socket.io/ {
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_pass "http://servername:9001/frontail/socket.io/";
    }

    location /frontail/ {
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $host;
      proxy_pass http://servername:9001;
       # proxy_http_version 1.1;
       # proxy_set_header Upgrade $http_upgrade;
       # proxy_set_header Connection "upgrade";
    }

}


add Tweetman to app list

Adding the Tweetman to Electron apps section.
I submitted it to Mac App Store.
https://tweetman.kksg.net/en

:sparkles: Thanks!


Add variable to override default instance AMI

Currently the node_group module uses the aws_ami data source to determine the newest AMI ID
published by Ubuntu within a major version and use that in our launch configurations. This parameter
could be overriden in each project to set a specific image name, and this is going to be the needed
to avoid having to recreate the launch_configuration every time we get a new image patch.
This commit adds a new variable to all the projects to make sure we set the same image everywhere.
The default image version should be set in the common data file.

I updated the commit to leave the node_group as it was. It's expected a "default" variable in the data directory to set the image name.


⚠ Bofrost.be has degraded performance

In 39f9005, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2118 ms

Resolved: Bofrost.be performance has improved in dd85146 after 6 minutes.


AVRO-3093: Stop releasing avro-python3 package

With the deprecation notice in place for some time, we can stop maintaining and releasing avro-python3 package along with other SDKs
Closes: https://issues.apache.org/jira/browse/AVRO-3093
Make sure you have checked all steps below.
Jira

[x] My PR addresses the following Avro Jira issues and references them in the PR title.

https://issues.apache.org/jira/browse/AVRO-3093

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
NA

Commits

[x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.
NA

It's hard to tell from iOS GitHub. Did the interop tests run? Can we do something to explicitly trigger those tests?

I can confirm that all file interop tests pass, but the RPC interop tests break :/  I don't think these are taken into account in the GitHub Actions yet, but you can launch them after building the world: ./share/test/interop/bin/test_rpc_interop.sh
+ py_tool rpcsend http://<IP_ADDRESS>:45815 share/test/schemas/simple.avpr add -file share/test/interop/rpc/add/onePlusOne/request.avro
+ PYTHONPATH=lang/py
+ python3 -m avro.tool rpcsend http://<IP_ADDRESS>:45815 share/test/schemas/simple.avpr add -file share/test/interop/rpc/add/onePlusOne/request.avro
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/rskraba/avro/lang/py/avro/tool.py", line 166, in <module>
    sys.exit(main(sys.argv))
  File "/home/rskraba/avro/lang/py/avro/tool.py", line 157, in main
    send_message(uri, proto, msg, datum)
  File "/home/rskraba/avro/lang/py/avro/tool.py", line 91, in send_message
    url_obj = urllib.parse(uri)
TypeError: 'module' object is not callable

:point_up: OK, it looks like this is a bug in the avro.tool left over from python2, and we never noticed because the tool hasn't been used for rpc interop tests.
We can merge this and take care of the py2 tool error later, the CI will be fine -- it's probably even a good idea to have a JIRA for that problem in case anyone ever runs into it (although I can't imagine how...)


Create automerge pyup GitHub action

Description
Fixes #2852
Supersedes #2868
Adds auto-merging pip updates to this repository using dependabot edit: pyup to alleviate @browniebroke 's workload. Includes CI checking
Rationale
Make updating this repository less stressful. Instead of updating dependabot, update pyup-bot. Hopefully that GitHub actor is correct!
Here's a test of CI checking here: https://github.com/Andrew-Chen-Wang/test-dependabot-automerge/pull/11 in which the test failed.
Here's a test of CI checking here which everything passed: https://github.com/Andrew-Chen-Wang/test-dependabot-automerge/commit/a3c7db1fb5a96a43c8f611d0fd313e31505f2a25
Made interval checking the CI 300 seconds or 5 minutes to avoid any rate limit blockage. If anything, Travis CI does take awhile so...
Also added WIP just in case because that did make the action get stuck for a long time.

@browniebroke Resolved comments and committed!

Sorry for being late in this discussion:
With the proposed auto-merge how to manage when an update to a dependency requires a change and that change does not necessarily break the build in CI?
I personally read the release notes and see the diff changes for the libraries before merging a pull-request that updates a dependency and I don't see how it can be automated with 100% reliability in the final result of the merge. Yes, changes like these are few, but they do exist.

@luzfcb  Hm I mean take #2829 for example. Someone from the community reported celery being funky in #2827, so Celery had to be downgraded.

Good point @luzfcb. Our CI is not running very extensive tests. Especially things which are only used in prod (e.g. django-storages) might have some breaking changes that will go unnoticed.

I think the amount of breaking changes combined with the amount of time checking each package takes plus the amount of packages makes this worthwhile. #2745 came 5 months after the error popped up from ca7fcb8f625af000693bc1f1e66d0730850af62b
Perhaps a fix could be to mark all production.txt requirements to not auto merge? Would it also be necessary for local.txt? I think that part is covered by the current test cases.

As you pointed out, we already have some issues with manual reviewing. Some bugs appear due to changes in this project, others from some of our dependencies.
At least when we merge things manually, I might see a correlation between a bug report and a recent merge, which might be a bit more difficult to keep track if this is all automated. Merging dependencies updates is not taking a lot of time at the moment, but it's not done blindly either.
For example, for the recent pytest update  (#2859), I double checked their changelog and noticed a breaking change. I quickly checked the codebase to make sure our fixtures were only using keyword arguments. All fixtures are all executed, in our tests, so this was probably not necessary, but this tells how much I trust our tests 😄
The main driver for this change is to reduce maintenance burden on maintainers, and I really appreciate the effort. However, Fábio's comment made me realise that it might have the opposite effect.

So as discussed, closing this. Thanks for the good intentions!


unity3d debugging broken in vscode insiders

Problem: when debugging unity3d game, I cannot see values of variables in editor. Need to open another editor to debug.

VSCode Version: 1.43.0-insider
OS Version: macos mojave 10.14.6

Steps to Reproduce:

Add breakpoint
Run unity project in editor
Run > Start Debugging
Game stops on breakpoint but "Variables" is empty.

C# right? Please file this on the repo for the C# extension: https://github.com/OmniSharp/omnisharp-vscode


Spawnpoints not working in some worlds

I'm trying to make an arena world where you spawn on a platform in the sky. I have the world spawn set there and the player spawnpoint and the Multiverse spawnpoint there, but when I die I respawn at a completely different location high enough in the air to die of fall damage.
It works properly in some other worlds, but not all.
Video: https://drive.google.com/file/d/1lm-2hqSfMeXL1UnrEtnN1rOjI9rN1Qrb/view?usp=sharing

Do you have other hub/lobby plugins that alter spawn? Or essentialsXspawn?
Do send your full server startup log as well.

Do you have other hub/lobby plugins that alter spawn? Or essentialsXspawn?
Do send your full server startup log as well.

I have EssentialsX, and I didn't know EssentialsXSpawn was a thing

I had a similar problem, and changing these settings in /plugins/Essentials/config.yml solved it:
Note: EssentialsXSpawn.jar controls these settings, and will only take effect if you have it installed.
# What priority should we use for handling respawns?
# Set this to none, if you want vanilla respawning behaviour.
# Set this to lowest, if you want Multiverse to handle the respawning.
# Set this to high, if you want EssentialsSpawn to handle the respawning.
# Set this to highest, if you want to force EssentialsSpawn to handle the respawning.
respawn-listener-priority: lowest

# What priority should we use for handling spawning on joining the server?
# See respawn-listener-priority for possible values.
# Note: changing this may impact or break spawn-on-join functionality.
spawn-join-listener-priority: lowest

I also added this alias in commands.yml to override the Essentials /spawn command in favor of /mv spawn.
  spawn:
  - mv spawn $1-

So if you're not using EssentialsXSpawn.jar, then it's most likely a plugin that handles /spawn or your world/player spawn point that's conflicting with Multiverse.

Thank you!


Can 2080Ti GPU support the computation power?

Just for confirm, I try to run default adt_expam algorithm on a leasehold server, however, the connection is closed. I want to know whether it is the problem of server itself or whether the single-GPU of 2080Ti cannot support the batch-size of GPU memory on the default setting.
Moreover, thanks for you sharing and contribution.
Regards!
Momo

A single 2080Ti is sufficient to run the experiments. We also ran our experiments on 2080Ti.


HDDS-11326. Speed up TestBlockOutputStreamWithFailures

What changes were proposed in this pull request?
Speed up TestBlockOutputStreamWithFailures. Reduce execution time from 1000s (581 + 418) to 670s Currently:
Tests without flaky
[INFO] Running org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 581.401 s - in org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

flaky tests
[INFO] Running org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 418.331 s - in org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

After changes:
[INFO] Running org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 671.981 s - in org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Things change in this pr:

Remove flaky tag, since it was passed for 1000 iterations without any failures. See https://issues.apache.org/jira/browse/HDDS-6113
Setup th cluster only once before all the tests.
Restart the datanodes after shutting them down to keep the cluster healthy.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11326
How was this patch tested?
CI:
https://github.com/chungen0126/ozone/actions/runs/11561813369

I triggered 20x10 repeated run for the latest commit.
https://github.com/chungen0126/ozone/actions/runs/11629416951/job/32397413670
The test is not stable now. I will try to fix it.

This test is not stable because the datanode restarts fail intermittently. It is related to RATIS-2194.


Not able to install the chart

Hi, I am trying to install this chart using the below command:
helm install my-n8n oci://8gears.container-registry.com/library/n8n --version 0.24.3

But I am getting this error:
Error: INSTALLATION FAILED: failed to download "oci://8gears.container-registry.com/library/n8n" at version "0.24.3"

try 0.24.4
try

❯ helm pull oci://8gears.container-registry.com/library/n8n --version 0.24.3
Pulled: 8gears.container-registry.com/library/n8n:0.24.3
Digest: sha256:66c2d5d28300efbb82b8e23d5d2814d34e877ac443234155e14c18a1fa4a9af0

I'm going ahead and close this issue, please reopen if the issue is not resolved.

Thanks for the help.
Getting this error now ->
helm pull oci://8gears.container-registry.com/library/n8n --version 0.24.4
Error: no cached repo found. (try 'helm repo update'): open /Users/shashank/Library/Caches/helm/repository/timescaledb-index.yaml: no such file or directory

Thanks a lot for the help, it works now.


Plugin sendmessages

Pull Request Test Coverage Report for Build 33

32 of 119 (26.89%)  changed or added relevant lines in 6 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.5%) to 25.291%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/discord/bot.go
0
4
0.0%

src/matrix/api.go
0
5
0.0%

src/discord/handleevents.go
0
12
0.0%

src/discord/messagefunctions.go
0
12
0.0%

src/plugins/sendmessagesplugin/sendmessages.go
24
49
48.98%

src/api/api.go
8
37
21.62%

Files with Coverage Reduction
New Missed Lines
%

src/discord/handleevents.go
1
0.0%

Totals

Change from base Build 26:
0.5%

Covered Lines:
239

Relevant Lines:
945

💛  - Coveralls


Cirrus CI

As opposite to #2178, this change configures Cirrus CI to run Puppeteer in both Windows and Linux containers.
Here is a link to a corresponding Cirrus CI build:

As you can see there are 4 auto-generate Docker Builder tasks to build custom Linux and Windows Docker containers. And 4 tasks to actually run tests in the cached containers. If interested read more about Cirrus CI's container builds in this blog post.

@fkorotkov interesting! Are there any re-tries on Cirrus CI? I wonder how come our builds always flake on travis/appveyor and are green on Cirrus.

@aslushnikov there are no re-tries on Cirrus CI. I've seen some tests failing with timeouts during testing. Maybe just once of twice.
The only possible explanation I can come up with is that the environment that tests are executed is more performant. Travis executes builds on AWS VMs that run not the latest Docker. AppVeyour uses not the latest Windows VMs on Azure.
Cirrus on the other hand uses Google Cloud and specifically Kubernetes Engine to run Linux-based container builds. Kubernetes nodes that run containers are using the latest Docker and by the way Chromium OS 😅 For Windows containers, at the moment, Cirrus CI uses Compute Engine to run the latest Windows Server optimized for Docker container. We are planning to move Windows container to GKE once GKE supports Windows.
So since Cirrus CI builds are executed on Google Cloud Platform, maybe you can tell me why tests are more stable? 🤣


Produce documentation  a readme for cluster processing

Originally posted by @chrisbc in https://github.com/GNS-Science/nshm-nz-opensha/issues/16#issuecomment-686084863

At least add a readme for this


[Improvement-15713][api]DataSource And UdfFunc list query use Enum code value rather than ordinal

Purpose of the pull request
fix #15713
Brief change log

DataSource list query use DbType enum code value rather than ordinal

UdfFunc list query use UdfType code value rather than ordinal

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

@songwenyong please link this PR to an issue

#15713

@songwenyong please link this PR to an issue

done

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 39.06%.  Consider uploading reports for the commit d8a47bf to get more accurate results

Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #15714      +/-   ##
============================================
- Coverage     39.07%   39.06%   -0.01%
  Complexity     4849     4849
============================================
  Files          1316     1316
  Lines         45014    44978      -36
  Branches       4819     4795      -24
============================================
- Hits          17589    17572      -17
+ Misses        25517    25498      -19
  Partials       1908     1908

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


GHA: save-state and set-output commands are deprecated

See https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

#874