Automatically set sync_batchnorm for training_type_plugin

What does this PR do?
This PR automatically sets the sync_batchnorm attribute for the training_type_plugin in the accelerator_connector. This is useful for custom plugins when sync_batchnorm is not known during plugin instantiation.

Fixes #<issue_number>
Before submitting

[x] Was this discussed/approved via a GitHub issue? (not for typos and docs)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
[x] Did you make sure to update the documentation with your changes? (if necessary)
[x] Did you write any new necessary tests? (not for typos and docs)
[x] Did you verify new and existing tests pass locally with your changes?
[x] Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

[x] Is this pull request ready for review? (if not, please submit in draft mode)
[ ] Check that all items from Before submitting are resolved
[x] Make sure the title is self-explanatory and the description concisely explains the PR
[x] Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?
Make sure you had fun coding 🙃

We may want a small test for this, just to make sure we catch this in case! Would you be able to write a small test using the BoringModel using a custom plugin? Just to ensure that sync_batchnorm is set correctly if not present

Hello @amogkam! Thanks for updating this PR.

In the file tests/plugins/test_custom_plugin.py:

Line 14:34: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
Line 22:34: E712 comparison to True should be 'if cond is True:' or 'if cond:'
Line 22:41: W292 no newline at end of file

Please, do not ignore the Before submitting/PR review headers at the top.

Is this ready to get merged in?

Is this ready to get merged in?

seems you have failing GPU test test_sync_batchnorm_set

@Borda which job is the failing test on? I’m not able to find it.

@amogkam You could see the failing tests, here.

I skipped the test on GPU, but the CI is still failing. Any suggestions here? It doesn't look like it's related to this PR.

Though all the required jobs are passing. Is this enough to merge this in?

@kaushikb11 it was failing with this error RuntimeError: Tensors must be CUDA and dense on trainer.fit(model). So I thought that BoringModel just didn't work with GPU and skipped it. I'll remove it and see if it's passing now.


Provide transport packages as releases

I have recently tried to import the sflight objects very unsuccessfully, as described in this issue on abapGit: https://github.com/abapGit/abapGit/issues/6924, leading to many broken objects on our system.
Would it be possible to provide transport packages as releases for this project that can be imported by a Basis person as an alternative to the abapGit installation method?

@VolkerDrees What prevents you from providing transport packages? Is it a technical problem? Presumably you can add everything to a transport, release it, and then copy the files from the OS and provide those? That is the classical way that SAP has provided packages and updates for many years?
As you may have seen from https://github.com/SAP-samples/abap-platform-refscen-flight/issues/36, I have twice now unsuccessfully tried to import with abapGit, spending many hours in the process.
SAP spends a lot of time promoting abapGit, which is being developed independently, so I suppose SAP makes no guarantees about it working or provides support for it? This seems to me an awful dilemma.


Travis build fails with default config due to old Node version

Describe the bug
The default config for Travis CI on this page (running-tests.md) suggests the use of Node 8.
Create React App, however, has moved on and some of its dependencies require a later Node version so the CI build fails with very strange and unhelpful warnings.
Setting Node version to 12 in .travis.yml resolves the issue.
Did you try recovering your dependencies?

No.
Which terms did you search for in User Guide?

"Travis CI"
Environment

Environment on my local system:
Environment Info:

  current version of create-react-app: 4.0.2
  running from C:\Users\<censored-username>\AppData\Roaming\npm-cache\_npx\10984\node_modules\create-react-app

  System:
    OS: Windows 10 10.0.19041
    CPU: (6) x64 Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
  Binaries:
    Node: 12.18.0 - C:\Program Files\nodejs\node.EXE
    Yarn: Not Found
    npm: 6.14.4 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Chrome: 88.0.4324.146
    Edge: Spartan (44.19041.423.0), Chromium (88.0.705.68)
    Internet Explorer: 11.0.19041.1
  npmPackages:
    react: ^17.0.1 => 17.0.1
    react-dom: ^17.0.1 => 17.0.1
    react-scripts: 4.0.2 => 4.0.2
  npmGlobalPackages:
    create-react-app: Not Found

Steps to reproduce

Create a new react app npx create-react-app app-name
Add .travis.yml from the docs to the project
Push the changes to GitHub

Expected behavior

I expected a clean CI build on a fresh project.
Actual behavior

I don't think this is relevant but pasting it anyway. Build fails with:
> react-scripts build
/home/travis/build/Glusk/app-name/node_modules/fs-extra/lib/mkdirs/make-dir.js:85
      } catch {
              ^
SyntaxError: Unexpected token {
    at createScript (vm.js:80:10)
    at Object.runInThisContext (vm.js:139:10)
    at Module._compile (module.js:617:28)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/travis/build/Glusk/app-name/node_modules/fs-extra/lib/mkdirs/index.js:3:44)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>build: `react-scripts build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/travis/.npm/_logs/2021-02-14T13_39_49_905Z-debug.log
The command "npm run build" exited with 1.
0.41s$ npm test
><EMAIL_ADDRESS>test /home/travis/build/Glusk/app-name
> react-scripts test
/home/travis/build/Glusk/app-name/node_modules/jest/node_modules/jest-cli/build/cli/index.js:227
    } catch {
            ^
SyntaxError: Unexpected token {
    at createScript (vm.js:80:10)
    at Object.runInThisContext (vm.js:139:10)
    at Module._compile (module.js:617:28)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/travis/build/Glusk/app-name/node_modules/jest/node_modules/jest-cli/build/index.js:13:12)
npm ERR! Test failed.  See above for more details.
The command "npm test" exited with 1.

Reproducible demo

/

Same issue. The react build script fails when generating the build within docker;
Tried deleting the local node_modules and everything. Some stack overflow answers indicate that it can be fixed with upgrading the local npm and node versions to something above 10. Both my local env and the docker env is 14; On macOS big Sur.
Reference:

https://stackoverflow.com/questions/65894427/react-scripts-build-works-local-but-fails-in-codebuild


Add basic implementation of the ComWrappers

Added tests, but they are not working yet due to wrong targeting.
See #4219

Added tests, but they are not working yet due to wrong targeting.

Fixing the targeting for the tests is non-trivial with the build system in the repo. It would be best to wait with automating these tests after we move CoreRT into dotnet/runtimelab, and we would probably just want to reuse the CoreCLR tests for this. For now, I think it is good enough if you just do a manual testing.

I have mentioned earlier that the implementation should be 99% C# ... I take that back. I think you want to reuse a good chunk of https://github.com/dotnet/runtime/tree/master/src/coreclr/src/interop from CoreCLR. You can reimplement some of it in C#, but it would not be that much of an improvement.

I see that. Plus one more place

https://github.com/dotnet/runtime/blob/master/src/coreclr/src/vm/interopconverter.cpp (Maybe can be implemented in c#)
https://github.com/dotnet/runtime/blob/master/src/coreclr/src/vm/interoplibinterface.cpp
https://github.com/dotnet/runtime/blob/master/src/coreclr/src/vm/rcwrefcache.cpp (Maybe can be implemented in c#)

Given that there some interesting scenarios in the test, I decide add implementation blindly trying make tests to pass just to excercise the code paths where it blow up.
I have set of questions

What is QCall
How do I handle GC stuff like that https://github.com/dotnet/runtime/blob/master/src/coreclr/src/vm/interoplibinterface.cpp#L259 or https://github.com/dotnet/runtime/blob/master/src/coreclr/src/vm/interoplibinterface.cpp#L369
Another interesting piece https://github.com/dotnet/runtime/blob/master/src/coreclr/src/vm/runtimehandles.cpp#L1739 where I decide to ignore type specific memory loading, and unloading context, until I have answer for you. Not important for dirty implementation which I'm want to achieve.

@kant2002 RE QCalls see https://github.com/dotnet/runtime/blob/f23e2796ab5f6fea71c9fdacac024822280253db/docs/design/coreclr/botr/mscorlib.md#calling-from-managed-to-native-code

I do not think there is much to directly reuse for CoreRT from the files under vm.

QCall

The equivalent of QCall in CoreRT is [DllImport(RuntimeLibrary, ExactSpelling = true)]. You should be able to fine number of examples.
The most significant difference between CoreCLR QCalls is that CoreRT does not have exception interop. If you need to throw exception in CoreRT, you need to return an error code and then throw the exception in C#.

There is an implementation of the identity cache in managed code that you could borrow. https://github.com/microsoft/CsWinRT/blob/master/WinRT.Runtime/ComWrappersSupport.netstandard2.0.cs has an implementation of the identity caching entirely in managed. There's likely some optimizations that could be done.

I will try to revitalize my attempts to make some progress.
@jkotas if I try to borrow idea/impl from @jkoritzinsky link, would you think is this good plan?

It has quite a bit of CsWinRT-specific code that you will need to fix up, otherwise sounds reasonable to me.

If you would live to continue working on this, could you please open a new PR in https://github.com/dotnet/runtimelab/tree/feature/NativeAOT ? Thank you!
The new place should make reusing the coreclr implementation at https://github.com/dotnet/runtime/tree/master/src/coreclr/src/interop easier.


[SQL] Fix #27300: az sql vm group create cluster_subnet_type should only be passed into WsfcDomainProfile

…SqlVirtualMachineGroup
Related command

Fix #27300
Stop passing cluster_subnet_type to SqlVirtualMachineGroup class
Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

SQL


Chore: setup CI and CD

What does this PR do?
Sets up CI with circleCI, CD with circleCI and heroku. It also implements code coverage check using coveralls and readme badges.
How should this be manually tested?
CD setup can't be tested manually. Only new scripts (start and coveralls) can be tested manually, run them in your console to test them. You can also check that the app is deployed here.
Any background context you want to provide?
My TravisCI free trial has ended, so I used CircleCI instead
Relevant screenshots:

Questions:
I have some questions about environment variables, test coverage reports and CD pipelines, but I'll ask those questions during our meeting :)

@Avuidrauxs I implemented changes we were talking about yesterday and added relevant screenshots to PR description. Ready for review


add :Gina branch command support

Hi!
:Gina branch is a command like agit branch support by gina.vim.
This patch is to add supporting this feature.
Prease review this Pull Request!
screenshot
before

after

thanks!

@chrisbra
Thank you!


Update federation in community-operators

This PR removes the community-operators/federationv2 folder entirely, since the package name is changing.

/hold
I will make another PR that adds files for the new package; holding until then

Actually, I decided to just add the new package in this same PR.

@aravindhp @SamiSousa This is ready to go in from my side if there are no further feedbacks / changes I need to make from your side.

/approve

/cc @jeff-phillips-18
Adding Jeff to this PR for feedback relating to OperatorHub.io

This is what operatorhub.io would show:

I see a couple things that I want to change that I previously glossed over.

Updated, PTAL

Drive by rewrite of the description:
  annotations:
    description: Gain Hybrid Cloud capabilities between your clusters with Kubernetes Federation.

  description: |-
    Kubernetes Federation is a tool to sync (aka "federate") a set of Kiubernetes objects from a "source" into a set of other clusters. Common use-cases include federating Namespaces across all of your clusters or rolling out an application across several geographically distributed clusters.

  	The Kubernetes Federation Operator runs all of the components under the hood to quickly get up and running with this powerful concept. Federation is a key part of any Hybrid Cloud capability.

    ### Supported Features

    * **Control Placement & Scheduling** - Federation allows you to control the set of cluster(s) an object is federated to.

    * **Override Settings** - Override certain parameters of an object based on the cluster, eg. on clusters in USA, set a specific environment variable like `GEO=usa`.

    * **Configure Cross-Cluster DNS & Ingress** - Orchestrate changes to DNS records and Ingress in order for traffic to flow to the clusters.

    ### Using Federation

    After configuring and starting your Federation instance, use the command line tool to enable each type of object you would like federated to all of your clusters.

    This deploys Federation in a namespace-scoped configuration. This means that only objects the share the Namespace where you Federation instance is deployed can be federated. You must enable Federation support for Namespaces in this configuration before other types will work. Use `kubefed2 enable namespaces`.

    Next, use `kubefed2 enable <object>` for the other objects you would like to federate.

    Finally, [register the other clusters](https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md#join-clusters) that will be federated to.

    See the [full user guide](https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md) for other capabilities of Federation.

Question @pmorie: How does the CLI address this instance of Federation? What address do you use and what flags need to be set on the binary to make that happen. Are there any other manual steps you need to take?

/hold cancel

/lgtm

Changes pushed to quay's community-operators namespace.


Remove 'self-DoS'ing tokio task for testing 'zebra seed' command.

Request::Peers not GetPeers
Fix 'dos' feature for seed command, and Buffer the seed service

In the future I think it would be good to split that code out into a separate binary rather than using features at all (since if it's in a load-generator tool, it's much more useful for testing, rather than a static build configuration option), in which case another alternative would be to just do that rather than fixing the CI.

:shipit:

why is this a squirrel????

:shipit:

why is this a squirrel????

Someone on the internet says


Re-export A11yDialogLib type to simplify dialogRef typing for consumers.

Based on this issue: https://github.com/KittyGiraudel/react-a11y-dialog/issues/275, I am adding a new types export for the convenience of consumers of the library.
The use case I have in mind is setting up a ref like so: const dialogRef = useRef<A11yDialogInstance>();, with the issue being that A11yDialogInstance is not easily available.
So far, that type has had to be deduced either by pulling it out of a11y-dialog itself (risking falling out of sync if versions mismatch for some reason), or by playing around with the existing types in creative ways.
By exposing the type directly, users don't need to do mental gymnastics, and also don't need to pull types out of the parent library.

Actually I‘m realizing my suggestion will cause conflicts with the component name. I’ll rename it.


lifecycle hook apis

Looking at the Go lang tempaltes, I am seeing some nice hooks to plug into the lifecycle of a function:
https://github.com/knative/func/blob/main/templates/go/.instanced-http/function.go#L34-L69
It would be nice if other SDKs/languages, such as Node.js, quarkus would have these APIs:

Start
Stop
Alive
Ready

Seems pretty handy

@matzew JS/TS have this too https://github.com/rh-max/openshift-docs/blob/a500f4874d3c0a14a88265d451ca1ea39d6fcd29/modules/serverless-nodejs-functions-overriding-liveness-readiness.adoc
With Quarkus you can do it too albeit it might be little bit more complicated. You just need to use standard Java lifecycle thingies @PostConstruct @PreDestroy. For health probes check out https://quarkus.io/guides/smallrye-health.

Yes, however we should probably leave this as an open issue because I think we should discuss how these are implemented and invoked to ensure we're being consistent.


GlusterFS missing in OKD 4.12

FCOS in OKD 4.12 is missing GlusterFS packages, so the gluster PVs can't be mounted anymore and the containers are stuck in ContainerCreating state, with glusterfs mount errors in event log.
I'm aware that GlusterFS support has been stale for a while now both in Kubernetes and OpenShift, but this abrupt undocumented removal could cause issues for some users.
Kubernetes 1.25 did deprecate GlusterFS (more details here), but release notes for OKD 4.12 don't mention anywhere that glusterfs and glusterfs-fuse packages are not installed by default so this behavior is unexpected.
This can be worked around by installing glusterfs and glusterfs-fuse packages via FCOS image layering mechanism described here. Example configuration for 4.12.0-0.okd-2023-03-05-022504:
FROM quay.io/openshift/okd-content@sha256:{HASH_FOR_INSTALLED_VERSION}

RUN rpm-ostree install -y glusterfs glusterfs-fuse && \
    rpm-ostree cleanup -m && \
    ostree container commit

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: os-layer-hotfix
spec:
  osImageURL: {CUSTOM_IMAGE_URL}

This change should either be reverted to include GlusterFS until upstream Kubernetes removes it, or clearly document it and warn users before the upgrade.

These RPMs were installed to pass conformance tests, so its a coincidence that you could use GlusterFS in-tree driver without additional packages. This was never mentioned in the documentation afaik, if it is we'll need to remove it


[5.2][SourceKit/CodeCompletion] Remove unnecessary sorting in completion

Cherry-pick #28786 into swift-5.2-branch originally reviewed by @benlangmuir
rdar://problem/57926003

@swift-ci Please test

Build failed
Swift Test Linux Platform
Git Sha -<PHONE_NUMBER>548409bb40c5ddba173d324855acc0

Build failed
Swift Test OS X Platform
Git Sha -<PHONE_NUMBER>548409bb40c5ddba173d324855acc0

@swift-ci Please test


docs: couple of ReferenceGrant cleanups

Remove deprecated ReferencePolicy using the hideTypePatterns config for the API docs generator.
ReferencePolicy has been renamed to ReferenceGrant, but we have retained ReferencePolicy as a deprecated type in order to make migration easier.
Keeping ReferencePolicy around in the docs is confusing for readers and probably doesn't provide any value.
Also fix path of ReferenceGrant image.
/kind cleanup

Thanks!
/lgtm


Alternative Recent Items menu in MacOS Dock

Hi vscode team!
This PR is my proposal of an alternative "Recent Documents" menu in the Mac Dock. It is based on the recent items submenu that you can find under the "File" menu. Today I sometimes find myself a bit confused by the default recent documents menu built with the Electron api app.addRecentDocument as it feels different from the "File" recent items menu and I would prefer if the menu worked like in this PR. (which is basically just like the recent items menu under "File" except the other options like "Clear Recently Opened").
A screenshot of how it looks:

I make this PR as a proposal and it would be really cool to hear what your thoughts are. This is my first time looking at the vscode codebase so I might very well have missed something that needs to be done. There is also one outstanding question that have commented in the code. This change seems to work for me though and I would really like to have this behaviour in vscode.
Thanks,
Pontus

Isn't this submenu duplicating the existing recent entries:

I think the default macOS behaviour is to show recently opened files and folders directly in the dock menu.


Couldn't create temp directory for images

Describe the bug
I rendered simple test scene, but blender couldn't render AI Render and showed such a messege "Couldn't create temp dirctory for images"  and didn't funtion rendering (only render base blender render. Didn't rendered AI render)
I relocated other directory for render, but same happen.

AI Render node is mute too !

To reproduce
Install AI Render
Make simple test scene
Did Render (with AI Render)
Renedered blender Render, but no AI Render
show error log at Prompt :Couldn't create temp dirctory for images
Expected behavior
Render currectly and save images,
Screenshots/video
No response
Environment

AI Render version:
Blender version:
Operating system: (Windows, OS X, etc)

AI Render 0..2.6
Blender 3.1
Window 10
Logs
No response
Additional information
No response

Thanks for posting this. I will do an update that will let you choose a temp directory, which should solve this. So far, this seems like a rare bug, but I will squash it!

@coolwant I think this update should fix the issue. Could you try testing this for me before I release it to everyone else? Remove the existing add-on and install this one (attached). Let me know if it works!
AI-Render-v0-2-7.zip

Hi Thank you for quick update !
First time it work! but second when i did render again (Changed parameter...etc), The AI render muted.  same as before bugs list "AI Render nodes muted"

@coolwant Gotcha. Yeah, at the moment, it will likely time out if you do 1024x1024. I plan to add AI upscaling as soon as I can get to it.

Fixed in the latest release. Get it from Blender Market, Gumroad or the releases page
Or update in Blender under 'Preferences' > 'Add-ons' > 'AI Render' > 'Updates'
Fixed in a8d18a059e24442f95104e080c71c6ed2d106ae1 and 2db43b2a93a10a3034db9eb6d7e725dfab4db4f3

Hi ben, I'm also getting the error, version 0.10, Windows 10, Blender 3.6.5. Where is the setting or how do you set up the temp folder?

@afecelis - I'm going to open a new issue for this. It has happened to a couple others since the latest version in the past few days, and I need a bit more info to track it down.

@afecelis just in case you didn't get a notification on my new issue, could you see #136


Fix class name in boundary generator

When the namespace of the given component contains a hyphen, the
generated boundary cannot be compiled due to a class not found
exception. The reason is the generated component name does not point
to a Java class, since the corresponding class name contains
underscores instead of hyphens.

Made unnecessary by 0.9.0.


qpid-cpp: switch to python3

qpid-cpp only supports python2 bindings, so they've been removed. #148779
(Also, the current derivation on master doesn't build; this one does)
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 169816 run on x86_64-linux 1

1 package built:

qpid-cpp


[WIP] Maintenance release

Codecov Report

Merging #2728 (230e421) into master (caf3164) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2728      +/-   ##
==========================================
+ Coverage   93.32%   93.34%   +0.02%
==========================================
  Files          46       53       +7
  Lines        2712     2886     +174
==========================================
+ Hits         2531     2694     +163
- Misses        181      192      +11


Implement inputFunctionName feature for correctnessTest

Resolving #144

Also don't make a PR for code that doesn't pass tests.

@googlebot  can you check the cla again?

Closing and reopening to try retriggering cla/google checks.


Use resource get_queryset in ModelInstanceLoader

Problem
ModelInstanceLoader class method get_queryset returns a different queryset from the resource's get_queryset (if overloaded).
Solution
Having access to the resource object, the instance loader (ModelInstanceLoader) should return the same queryset as the resource is using. If needed this default behavior can be overloaded.
Acceptance Criteria
No new tests needed as this is a simple change of default behavior (and for the default behavior both classes return the same queryset).

Coverage remained the same at 95.659% when pulling 80ab9a4cda756a2910f8e63f9c0bdc5ec8402493 on adibo:feature-resource-queryset into f062d38922be2b4d574be562d0830e06bf8784ab on django-import-export:master.

Ah much better diff! Thanks 👏

I will try to have a look over it this weekend and contemplate the implications of this change. Can you think of any? Breaking changes, etc.?

Sure, please do.
I don't see any breaking issues, though. To my eye, the default behavior of get_queryset of  both  ModelInstanceLoader and ModelResource will remain unchanged (before this change both return _meta.model.objects.all()).
This PR actually aligns/couples the instance loader method get_queryset with the ModelResource get_queryset. If one overloads the resource method, I'd say it would be logical to expect that the instance loader get_qeuryset should follow (i.e. return the same queryset, even if overloaded).
However, if one would like them to behave differently, with this change it is still possible, by overloading each of them separately.
But, please, do check! I'll try to go through the code once more myself too. I'll report if I spot any issues.

After a review of this, I don't see any negative implications.
@bmihelac Do you have an opinion one way or another?
Otherwise, I feel comfortable merging this.

I also went again through the code in places where get_queryset is used and couldn't spot any harmful potential.

This looks like right thing to do. Thanks @adibo and @andrewgy8 for PR and review

Thanks @adibo  for the contribution!


Dark mode

Add dark mode
Can be activated manually
Or based on time
(Maybe also based on ambient light)

added in darkmode
can be activated manually


opae: update 1.4.0  release notes

I do not believe this is what we have tested.
We should specify OS and Kernel that we have tested.  Also I think this "through" word is very miss-leading.  We should not use this word.
SUSE SLE 12 SP3 and CentOS 7.4, with Linux kernels 3.10 through 4.19

SUSE SLE 12 SP3 and CentOS 7.4, with Linux kernels 3.10 through 4.19

This is a good point. Maybe this should say "SUSE SLE 12 SP3, CentOS 7.4, and the 4.19 LTS kernel"


OngoingStubbing failing to call doReturn value for covariant types

Code sample:
interface Foo {
  fun bar(): Single<Bar<Int>>
}

sealed class Bar<T> {
  data class A<T>(val a: T) : Bar<T>()
}

class Test {
  val foo = mock<Foo> {
    on { bar() } doReturn Single.just(Bar.A(1))
  }
}

Received error:
error: none of the following functions can be called with the arguments supplied:
public infix fun <T> OngoingStubbing<???>.doReturn(t: ???): OngoingStubbing<???> defined in com.nhaarman.mockito_kotlin
public fun <T> OngoingStubbing<???>.doReturn(t: ???, vararg ts: ???): OngoingStubbing<???> defined in com.nhaarman.mockito_kotlin
public inline infix fun <reified T> OngoingStubbing<Single<Bar<Int>>>.doReturn(ts: List<Single<Bar<Int>>>): OngoingStubbing<Single<Bar<Int>>> defined in com.nhaarman.mockito_kotlin
    on { bar() } doReturn Single.just(Bar.A(1))

Calling thenReturn() from original Mockito library however compiles just fine:
on { bar() }.thenReturn(Single.just(Bar.A(1))

It's seems that the type is getting lost along the way so I tried to define my own infix function with covariant type like so but that gave type mismatch:
infix fun <E> OngoingStubbing<out E>.myReturn(t: E): OngoingStubbing<out E> = thenReturn(t)

I think the problem here is that Single doesn't have covariant type which doesn't allow us to establish relationship between the types. Initial code works as expected if the Bar value is being wrapped in something with covariant types, e.g. Kotlin's List:
interface Foo {
  fun bar(): List<Bar<Int>>
}

sealed class Bar<T> {
  data class A<T>(val a: T) : Bar<T>()
}

class Test {
  val foo = mock<Foo> {
    on { bar() } doReturn listOf(Bar.A(1))
  }
}

I'm not sure if that's org.mockito:mockito-core or com.nhaarman:mockito-kotlin-kt issue.
Mockito: org.mockito:mockito-core:2.7.5
Mockito Kotlin: com.nhaarman:mockito-kotlin-kt1.1:1.5.0
Kotlin: 1.2.71
JDK: 1.8

Related to #274.

@kirillzh Have you tried using onGeneric?

This is the correct answer I think:
interface Foo {
    fun bar(): Single<out Bar<Int>>
}

sealed class Bar<T> {
    data class A<T>(val a: T) : Bar<T>()
}

@Test
fun covariantTest() {
    val foo = mock<Foo> {
        on { bar() }.doReturn(Single.just(Bar.A(1)), Single.just(Bar.A(2)))
    }
    val obj = foo.bar().blockingGet()
    val obj2 = foo.bar().blockingGet()
    assertEquals(Bar.A(1), obj)
    assertEquals(Bar.A(2), obj2)
}

So change the signature of your interface should work

@kirillzh Did the above solution work for you?

Yes, I ended up using Mockito's original thenReturn as a workaround.


Minor, change Kyligence nexus repository port to 8081

Can one of the admins verify this patch?

Can one of the admins verify this patch?


fix(action): Freeze the Transport instance returned

Coverage remained the same at 79.446% when pulling 414c326d1001da1fccb4e7b24d4ee598f7312971 on orestes:master into e077b9c98cdab859c9938d930d7f9f26c50520db on kusanagi:master.


garbled ansi logs

OrientDB Version: 3.0RC1
Java Version: 8
OS: MacOs
Expected behavior
[WARNING ] Storing configuration for property:'server.database.path' not existing in current version
Actual behavior
[WARNING ] $ANSI{green {db=OSystem}} Storing configuration for property:'server.database.path' not existing in current version
Steps to reproduce
I am getting this $ANSI {} stuff in log messages. I am assuming someone wanted to color things on the screen (assuming ANSI terminal always) making a huge mess in the process.
Second question: Why am I getting 'Storing configuration for property:'server.database.path' not existing in current version'

Hi @hrstoyanov,
there is some autodetection of ansy console, but probably in your case is not working as expected, you can run the server with ./server.sh -Dlog.console.ansi="false" and the ansi log will be disabled, the server.database.path logging is some missing internal mapping, is going to be fixed in future version.
Regards


rcode is 5 bits wide

rcode should be 5 bits wide, not 4. Thus the mask is 0x1F, not 0xF.
See http://www.zytrax.com/books/dns/ch15/#rcode
See https://github.com/tjfontaine/native-dns-packet/pull/34

Just kidding. It is 4 bits https://www.ietf.org/rfc/rfc1035.txt


Configurable acceptable project score

Hi
It would be good if we could set either

a minimum project score
minimum required severity

and have dbt-score return a non-zero return code so that we can stop CI pipelines. At the minute, only the inability to find a manifest file returns a non-zero return code. Duplicate rules raise an exception
e.g. for minimum required project score
[tool.dbt-score]
min_required_project_score=7.5

or passed on command line --min_required_project_score 7.5
for maximum allowed severity
[tool.dbt-score]
min_required_severity=MEDIUM

or passed on command line --min_required_severity MEDIUM
Should it raise an exception or simply log a message and exit with non-zero return code ?
Regards
Rob

We should absolutely do this, yes! That's definitely how we envision it.
I would suggest something like
[tool.dbt-score]
fail_project_under = 7.5
fail_any_model_under = 6.0

meaning dbt-score would exit with 1 if either is true:

the project score is below 7.5
any model score is below 6.0

I think that would allow better control rather than min_required_severity.
Would such an approach work with your use-case?

@matthieucan that would absolutely work !!


Problem with escaped characters

Hello Ryan,
Nice work on this, I found it because I'm working on the same thing, except using native FM. I did some testing and found some issues. One improvement I would suggest is to do a full round trip on the expression, meaning that once the JS calculates the expression, it then calls a FM script that then validates the expression to make sure it's valid.
Here's the issue...
This is valid json:
{
  "backslash": "\\bar\\",
  "quote": "\"foo\""
}

But, the resulting expression is not a valid FM calc:
JSONSetElement (
    "{}";
    [ "backslash" ; "\bar\"; JSONString];
    [ "quote" ; ""foo""; JSONString]
)

The expected result would be:
JSONSetElement (
    "{}";
    [ "backslash" ; "\\bar\\"; JSONString];
    [ "quote" ; "\"foo\""; JSONString]
)

Hi there Micah, thanks for pointing this out! looks like the / and " are being lost in the javascript before it gets put back together again. I'll take a look at supporting this and validating it!

hopefully this is now fixed in the latest pull request! thanks for reporting the issue with escaped characters!
#8

Did a quick test, works as expected!


Readding Autoassess landsurface recipes for scientific review and inclusion in ESMValTool v2.3

Description

Closes #issue_number
Link to documentation:

Before you get started

[ ] ☝ Create an issue to discuss what you are going to do

Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[ ] 🛠 This pull request has a descriptive title
[ ] 🛠 Code is written according to the code quality guidelines
[ ] 🛠 Documentation is available
[ ] 🛠 Tests run successfully
[ ] 🛠 The list of authors is up to date
[ ] 🛠 Any changed dependencies have been added or removed correctly
[ ] 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

[ ] 🧪 Recipe runs successfully
[ ] 🧪 Recipe is well documented
[ ] 🧪 Figure(s) and data look as expected from literature
[ ] 🛠 Provenance information has been added

New or updated data reformatting script

[ ] 🛠 Documentation is available
[ ] 🛠 The dataset has been added to the CMOR check recipe
[ ] 🧪 Numbers and units of the data look physically meaningful

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

Autoassess landsurface metrics/diagnostics
Battle plan to include the Autassess landsurface recipes in ESMValTool v2.3:

[x] resurrect them from the now extremely old all_autoassess_recipes branch into a new branch (this one) and create Draft PR
[x] test run them on JASMIN with the latest dev ESMValCore and obtain results, stock datasets from recipe
[ ] rerun them with UKESM1 data for comparisons with PP data -> @alistairsellar needs to tell me the data specs
[ ] store results somewhere easily available for checking
[ ] run the equivalent Autoassess ones on UKESM pp data
[ ] compare and do scientific review
[ ] check documentationand fix missing plots in gallery
[ ] mergy-merge and grab a brewski :beer:

OK - smashing news: I ran all four landsurface recipes with the latest esmval- code from current development branches (both Core and current Tool branch, this one). Specs are:
esmval code
(release220) [valeriu@sci1 esmvaltool_var_test]$ conda list esmval
# packages in environment at /home/users/valeriu/anaconda3/envs/release220:
#
# Name                    Version                   Build  Channel
esmvalcore                2.2.0                     dev_0    <develop>
esmvaltool                2.2.0                     dev_0    <develop>
esmvaltool-sample-data    0.0.3                    pypi_0    pypi

Python, conda, iris and matplotlib
(release220) [valeriu@sci1 esmvaltool_var_test]$ conda list python
# packages in environment at /home/users/valeriu/anaconda3/envs/release220:
#
# Name                    Version                   Build  Channel
antlr-python-runtime      4.7.2           py39hde42818_1002    conda-forge
msgpack-python            1.0.2            py39h1a9c180_1    conda-forge
python                    3.9.2           hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-stratify           0.1.1           py39hce5d2b2_1003    conda-forge
python-xxhash             2.0.0            py39h3811e60_1    conda-forge
python_abi                3.9                      1_cp39    conda-forge

(release220) [valeriu@sci1 esmvaltool_var_test]$ conda -V
conda 4.10.1

(release220) [valeriu@sci1 esmvaltool_var_test]$ conda list iris
# packages in environment at /home/users/valeriu/anaconda3/envs/release220:
#
# Name                    Version                   Build  Channel
iris                      3.0.1            py39hf3d152e_1    conda-forge
(release220) [valeriu@sci1 esmvaltool_var_test]$ conda list matplotlib
# packages in environment at /home/users/valeriu/anaconda3/envs/release220:
#
# Name                    Version                   Build  Channel
matplotlib                3.3.4            py39hf3d152e_0    conda-forge
matplotlib-base           3.3.4            py39h2fa2bec_0    conda-forge

@alistairsellar could you please give me the UKESM1 data specifications that I should be rerunning these recipes with? If you look at the autoassess_landsurface_* recipes in the current recipes dir, none of the recipes use UKESM1, so if you could tell me what the exact datasets per recipe should be so we can do the comparisons with the PP data, that's be grand! :beer:

Hi V, please use these runs to allow direct comparison:

HadGEM3-GC31-LL r1i1p1f3
UKESM1-0-LL r5i1p1f3

The period you have already is fine: start_year: 1992,  end_year: 2002
but can I just check that this means 1992-01-01 to 2002-12-31

excellent, cheers @alistairsellar :beer: are the other data parameters and variables good?

Yes, they look correct to me.

excellent @alistairsellar - I'll run those tomorrow after 10am (JASMIN is having FS updates 830am-10am tomorrow, I can have coffee :coffee: ) :beer:

Hi again. I've now been through the original code in more detail, and picked up 3 changes to variable names needed in the recipes (see comments above).
I also noticed your comment about the missing soil moisture variables. I'm consulting with our local expert on this.

Data issues: CMIP6 data is patchy on JASMIN: so far I have noticed mrsos data missing for HadGEM3-GC31-LL for experiments, and other bits and bobs missing for the setup you asked for @alistairsellar - for permafrost I could get data (apart for mrsos if I change to historical or piControl, would either of those do?

Any experiment with HadGEM3 or UKESM1 will be fine - I can make the equivalent AutoAssess in a couple of hours. You could even compare two UKESM exps, or two HadGEM3 exps if that's more complete.

@alistairsellar can we not do it for the exact same model and experiment - comparing the individual metrics should probably do the trick? At least for permafrost mrasos is giving me terrible headaches for HadGEM missing everywhere. If I do two different experiments for UKESM1 that's no good either because the fx files (sftlf) lives only in piControl (this is a massive gripe me and @zklaus and @ledm have been having, replication of fx files across experiments is nonexistent in CMIP6 UKESM1 data)

Sure, that's fine. Whatever you can make work for available CMIP6 data, I can replicate here.

this is a massive gripe...
Is fx thing this a Met Office-specific issue? If so, do you want me to pick it up with our data publishing experts?

OK @alistairsellar I did manage to run the permafrost diag (not dig :rofl: ) with full data complement but for a very very restricted data call, see the recipe I used, if you could run the same I have the results at hand :+1:

this is a massive gripe...

Is fx thing this a Met Office-specific issue? If so, do you want me to pick it up with our data publishing experts?

we spoke to @matthew-mizielinski and the take was (at the time we spoke, about a year ago) that these files, since they are identical across exps, should stay in one exp only, so not a serious problem but just a local annoyance :grin:

if you could run the same I have the results at hand

Great! Setting it off now...

since they are identical across exps, should stay in one exp only

Do other modelling centres take the same view? Or do they replicate fx files across all exps?

Permafrost RESULTS
Recipe
recipe
Metrics: UKESM1
permafrost area,1354.5278365343268
fraction area permafrost over zerodeg,29.150328299094394
offset 1m minus surface,1.275604248046875
offset surface minus air,8.726593017578125
attenuation 1m over surface,0.49492978649195407
attenuation surface over air,0.4920951365355539

Plots

Great stuff! That's a damn good start. I'm still waiting for my piControl AutoAssess to run, but doing an initial comparison to hist results: http://gws-access.jasmin.ac.uk/public/mo_gc3/dev2/autoassess/u-az513_vs_u-bg466/land_surface.html (should be visible to you if you're going in through Reading IP address - let me know if not).
For this rough comparison, the plots look very very good. However, the first two metrics too large by about 2 orders of magnitude. Could there be a mixup between fraction and % in what the code is expecting? I'll take a look...

not sure yet, but there are two problems I can already see in my run:

first off, the mrsos file we have off ESGF has depth coord at one single pint, and that is well below 2m:

/badc/cmip6/data/CMIP6/CMIP/MOHC/UKESM1-0-LL/piControl/r1i1p1f2/Lmon/mrsos/gn/latest/mrsos_Lmon_UKESM1-0-LL_piControl_r1i1p1f2_gn_196001-204912.nc
Depth coordinate:
DimCoord(array([0.05]), bounds=array([[0. , 0.1]]), standard_name='depth', units=Unit('m'), long_name='depth', var_name='depth', attributes={'positive': 'down'})

the second issue is that mrsos is time varying, and the Autoassess code needs it to be 2d lat-lon only, so to do that I had to collapse time on MEAN, which is a hacky hack

you if you're going in through Reading IP address

at home today so I have a Reading West IP :grin: - so no, I can't sees it. Anyways, off the grid for a bit, need lunch :hamburger:

it's that depth that's killing us

Sorry, I hadn't spotted that mrsos is just the shallowest soil layer. After more hunting I see that mrsol is the 3D equivalent and has 4 layers for the UK models, with the deepest at 2m.

And a time mean is correct - the original code uses a mean over the whole period.

Reading West IP

We definitely don't put those on the allow-list! I'll email you the password...

it's that depth that's killing us

Sorry, I hadn't spotted that mrsos is just the shallowest soil layer. After more hunting I see that mrsol is the 3D equivalent and has 4 layers for the UK models, with the deepest at 2m.

very cool! lemme try mrsol then

Ooh, also mrsol has actually been published for HadGEM3-GC31-LL piControl

Ooh, also mrsol has actually been published for HadGEM3-GC31-LL piControl

I can rerun if I find the data, just shipped the run for UKESM1 only for now

That's all we need to verify the metrics anyway.

OK I just reran with mrsol and extracting the 2m layer and...the metrics are identical down to the last digit :worried:

Hrrm, that's a shame.
I have an idea - looking into this idea now...

just saw the plots you linked, they look almost identical to mines, but look at the scale, yours are 0-1, mines are 0-100, that may explain the two orders of magnitude (BTW I don't see the metrics values anywhere?)

In the meantime, here are the AutoAssess permafrost metrics for UKESM1 piControl 1997/12/01- 2002/12/01:
attenuation 1m over surface,0.49573169134648043
attenuation surface over air,0.4925018141311794
fraction area permafrost over zerodeg,0.28940865677944033
offset 1m minus surface,0.7005462646484375
offset surface minus air,5.88677978515625
permafrost area,13.561272172616677

OK clearly there is an extra factor of 100 in some of my results, divide those by 100 and we get pretty dam close numbers

I have an idea - looking into this idea now...

The idea came to naught. I might need to run with some debugging prints. Is it possible for me to run your recipe + branch on Jasmin?

yes, I'll set up an executable script for you, but not now, I'm off for the day, will do it Monday first thing. But look at the metrics - they are almost identical (well, if you divide some of mines by 100)

Yes, it looks really really promising!
Have a great weekend!

FWIW, the piControl (aw310) permafrost plots should appear here soon:
http://gws-access.jasmin.ac.uk/public/mo_gc3/dev2/autoassess/u-az513_vs_u-aw310/land_surface.html

this is a massive gripe...

Is fx thing this a Met Office-specific issue? If so, do you want me to pick it up with our data publishing experts?

we spoke to @matthew-mizielinski and the take was (at the time we spoke, about a year ago) that these files, since they are identical across exps, should stay in one exp only, so not a serious problem but just a local annoyance 😁

Hi @valeriupredoi, @alistairsellar
I'll have a think about whether we can reconsider this, but a quick back of the envelope calculation gives ~3,000 simulations* x 9 fixed field per simulation = ~27,000 datasets which will all be duplicates.
The concept of a symbolic link within ESGF has been discussed, which would allow us to avoid duplication like this, but I don't expect progress in the near term.
*DCPP & PAMIP account for ~85% of this

Thanks @matthew-mizielinski. What have other modelling centres done in this regard? Or is it the usual hodge-podge of different approaches?

Thanks @matthew-mizielinski. What have other modelling centres done in this regard? Or is it the usual hodge-podge of different approaches?

If you do a search for fx/orog on esgf you can get a feels for the number. I think about a third have published it for everything, another third have a minimal set (including MOHC), and a third have a random selection.

cheers @matthew-mizielinski - don't forget the third that have not published it at all :grin:

I'll have a think about whether we can reconsider this, but a quick back of the envelope calculation gives ~3,000 simulations* x 9 fixed field per simulation = ~27,000 datasets which will all be duplicates.

Indeed, it's a bit silly to replicate duplicates, but maybe include them at top ensemble (r1i1p1fX?) for each experiment? The CMIP5 r0i0p0 method was better IMHO :+1:

cheers @matthew-mizielinski - don't forget the third that have not published it at all 😁

Good point!

I'll have a think about whether we can reconsider this, but a quick back of the envelope calculation gives ~3,000 simulations* x 9 fixed field per simulation = ~27,000 datasets which will all be duplicates.

Indeed, it's a bit silly to replicate duplicates, but maybe include them at top ensemble (r1i1p1fX?) for each experiment? The CMIP5 r0i0p0 method was better IMHO 👍

Not a bad idea actually, this brings the number of required datasets down to below 2,500.

maybe include them at top ensemble (r1i1p1fX?) for each experiment

Would that enable ESMValTool to pick them up for a different ensemble member of the same experiment? If not, I guess it would at least allow use of r1.

cheers again @matthew-mizielinski - feel free to untag yourself from this PR, Alistair and myself will be exchanging quite a lot of communication here and I don't want you to get pinged everytime we post stuffs about Autoassess :grin:
@alistairsellar I found out where we have that extra factor of 100 - using an sftlf mask that is in percents x 100, difiding the result of pf_periods by 100 to account for absulte land fraction (and not percent) gives me these metrics:
permafrost area,13.54527836534327
fraction area permafrost over zerodeg,0.29150328299094397
offset 1m minus surface,1.275604248046875
offset surface minus air,8.726593017578125
attenuation 1m over surface,0.49492978649195407
attenuation surface over air,0.4920951365355539

which are very very close to your UKESM1 '97-'02 period (great period that one BTW, some of the best years in me life haha)
attenuation 1m over surface,0.49573169134648043
attenuation surface over air,0.4925018141311794
fraction area permafrost over zerodeg,0.28940865677944033
offset 1m minus surface,0.7005462646484375
offset surface minus air,5.88677978515625
permafrost area,13.561272172616677

the plots look sane and almost identical to yours as well:

Brilliant! Well spotted.
So I think we're scientifically verified for:

Permafrost plots
attenuation 1m over surface
attenuation surface over air
fraction area permafrost over zerodeg,0.28940865677944033
permafrost area,13.561272172616677

But these two are still different:

offset 1m minus surface, 0.70 vs 1.3
offset surface minus air, 5.9 vs 8.7

I'll look at the code for these after lunch...

awesomepossum! Naively those two metrics differ since the analysis periods differ - my analysis is 1992-2002 and yours is 1997-2002 I can run with 1997 right now :+1:

awesomepossum! Naively those two metrics differ since the analysis periods differ - my analysis is 1992-2002 and yours is 1997-2002 I can run with 1997 right now +1

Nevermind, I just reran with 97-02 and I got pretty much the same metrics, yeah maybe have a look at the code, beats me why they'd be different, bon appetit :+1:

@alistairsellar I've realized I was not applying the median in masked mode (hence shifting counts with masked elements) and I have also changed the interpolator for the case when we extract layer 0 to Nearest since that makes more sense given that the data doesn't actually have a layer=0, so with these changes I got:
offset 1m minus surface,0.86416626
offset surface minus air,7.6981506

The problem is that the interpolation/extrapolation mechanism can change results quite significantly, so am pretty sure the results you get differ coz of this now (the major source of issues was not using the masked median though)

What do you get if you apply the median in masked mode but leave the interpolator as linear? For comparison with the original this would be the most direct comparison.
We can then discuss as a separate issue whether Nearest is more correct.

I get:
permafrost area,13.54527836534327
fraction area permafrost over zerodeg,0.29150328299094397
offset 1m minus surface,0.9465637
offset surface minus air,7.673065
attenuation 1m over surface,0.49492978649195407
attenuation surface over air,0.4920951365355539

with the setup identical to my previous run that we compared with

OK, thanks for trying. I reckon I need to dig in a bit with print statements then. Are you still up for making an executable script that I could run on Jasmin?

sorry, got distracted with other shtuffs :giraffe: try this: on JASMIN: in a completely clean environment (no sys vars declared, no Python virtual envs activated):
eval "$(/home/users/valeriu/anaconda3/bin/conda shell.bash hook)"
conda activate release220
git clone https://github.com/ESMValGroup/ESMValTool.git
cd ESMValTool
git checkout autoassess_landsurface_recipes
pip install -e .[develop]

set up your config file in ~/.esmvaltool to be like mine in /home/users/valeriu/.esmvaltool/config-user.yml (may as well copy it) and the you should be able to run, and while running edit the script in your local copy of ESMValTool/esmvaltool/diag_scripts/autoassess/land_surface_permafrost/permafrost.py

Thanks V. I get the following error from the pip command:
'''
The following error occurred while trying to add or remove files in the
installation directory:
    [Errno 13] Permission denied: '/home/users/valeriu/anaconda3/envs/release220/lib/python3.9/site-packages/test-easy-install-28528.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /home/users/valeriu/anaconda3/envs/release220/lib/python3.9/site-packages/

'''

Full output attached in case the above doesn't point to an obvious mistake.
error.log
Presumably I made some schoolboy error?

@alistairsellar - no, that's actually my schoolboy mistake for forgetting that pip needs to write to the root site-packages even if you perform a local, editable installation, can you run the pip install command but --user flag added to it? Sorry about that :relaxed:

found another bug (self-bug, mind you) - the Koven sites locations need to be adjusted for lon: 0-360 data (they come on a -180-180 deg grid) so the interpolations can be correct! You may want to check that too in native Autoassess, @alistairsellar but I think all there is on a -180-180 deg grid.  This thing is sensitive like heck on masks and interpolation mechanisms (as it should be :grin: )

can you run the pip install command but with --user flag added to it?

Thanks V. I tried that now (pip install -e .[develop] --user), and I'm afraid I get the same error message. Should I try manually setting --install-dir to override what it's trying to use?

ugh, try that (altho I've never used that option), alternatively, it's time you installed it fully in your JASMIN home :computer:

Sorry, choka with meetings all day. I'll try it now.
And yes you're right it's high time I did my own proper installation. Off to the tutorial for me...

no worries, I been to get jabbed :syringe: - yes, it'll be much more convenient if you have your own installation, you can port these metrics at a crazy speed :grin:

frenly ping @alistairsellar about new rseults :beer:

Thanks V, that's great. Sorry - another bad day for me. Tomorrow looks better...

no worries, man, get some rest! :beer:

Sorry for the radio silence. I've now got a personal install working and have reproduced your results for the permafrost metrics - so I should be able to debug that now.
But first I'll take a look at the metrics for the other sections that you made yesterday...

...OK, here's what I reckon:
surfrad is close enough.
SurfRadNLW MedAbsErr ann,3.8124046325683594
SurfRadNLW MedAbsErr djf,5.2260589599609375
SurfRadNLW MedAbsErr jja,5.695487976074219
SurfRadNLW MedAbsErr mam,4.456073760986328
SurfRadNLW MedAbsErr son,4.6931610107421875
SurfRadNSW MedAbsErr ann,4.9277801513671875
SurfRadNSW MedAbsErr djf,6.69140625
SurfRadNSW MedAbsErr jja,6.107902526855469
SurfRadNSW MedAbsErr mam,4.683586120605469
SurfRadNSW MedAbsErr son,5.502941131591797

(Sorry for the change in order)
Differences 5% or less, which could be explained by differences processing (e.g. regridding)

Snow is not quite right:
snow MedAbsErr djf,5.917015075683594
snow MedAbsErr mam,4.424624443054199
snow MedAbsErr son,0.43985575437545776

mam in particular is out by >50%. I'll dig into this after the permafrost.

soilmoisture is way out:
soilmoisture MedAbsErr djf,0.07084117829799652
soilmoisture MedAbsErr jja,0.0563998743891716
soilmoisture MedAbsErr mam,0.0671340823173523
soilmoisture MedAbsErr son,0.06610548496246338

but I know why. I got an answer back from our local expert on how to deal with the missing variables. I now know how to convert mrso into the units needed for comparison with the obs, and will code this up after the others.

In short, leave these with me and I'll let you know how I get on...

Permafrost is now OK, including the offset metrics.
AutoAssess (including the bugfix noted below):
attenuation 1m over surface,0.49573169134648043
attenuation surface over air,0.4925018141311794
fraction area permafrost over zerodeg,0.28940865677944033
offset 1m minus surface,0.94659424
offset surface minus air,7.7096863
permafrost area,13.561272172616677

ESMValTool (after the above two commits):
attenuation 1m over surface,0.49573173498262546
attenuation surface over air,0.4920951365355539
fraction area permafrost over zerodeg,0.29002045066872484
offset 1m minus surface,0.9465637
offset surface minus air,7.673065
permafrost area,13.54527836534327

The fixes needed were the above two commits, plus there was a bug in AutoAssess that you must have fixed in the ESMValTool code when you ported it: using np.median() instead of np.ma.median() on masked arrays in koven_temp_offsets().

I converted mrsos to volumetric soil moisture following some advice from the original author of the AutoAssess code. That makes the metrics agree...
AutoAssess:
soilmoisture MedAbsErr djf,0.07084117829799652
soilmoisture MedAbsErr mam,0.0671340823173523
soilmoisture MedAbsErr jja,0.0563998743891716
soilmoisture MedAbsErr son,0.06610548496246338

ESMValTool:
soilmoisture MedAbsErr djf,0.0708264410495758
soilmoisture MedAbsErr mam,0.06645134091377258
soilmoisture MedAbsErr jja,0.05707845464348793
soilmoisture MedAbsErr son,0.06563765555620193

Surfrad was already OK in your test above, and permafrost and soil moisture are now sorted, so just snow to go. I should be able to fix that today.

wohooo - awesome man, these metrics now look absolutely spot-on and I'm happy you didn't have to refactor too much :grin: Excellent, all that's left to be done now is to create the documentation for these metrics and upload the (few) figures, am afraid that's mostly you since I haven't the foggiest what to write. Let me know what I can do to help :beer:

we need to find the equivalent of mrsol for CMIP5 - this is why the test is failing, it can't find the CMIP5 CMOR table for mrsol - we may have to create a derived variable? @jvegasbsc @zklaus  do you know what's the map of that? - I couldn't find it in the usual CMIP5->CMIP6 mapping dict we use

Look at the CMIP6 data request and the CMIP5 standard output it seems that mrsol is the CMIP6 replacement for the CMIP5 variable mrlsl, possibly to homogenize the nomenclature among the soil moisture variables.

shweet, cheers for the superquick answer @zklaus :beer: - @alistairsellar what say you? Also, for documentation purposes, for the new recipes, I have included a template in this commit whic is a stripped-down version of the one we did for Stratosphere, so it's easier to copy along and modify for each of the four recipes we need to include :+1:

Yes, I agree that mrlsl = mrsol. How do you implement this kind of flexibility?

Thanks for the documentation template V - leave that with me. I can pull the relevant bits out of AutoAssess.
I'm going to try and get the snow metrics right first though.

Yes, I agree that mrlsl = mrsol. How do you implement this kind of flexibility?

OK, we'll have to write a new recipe only for CMIP5 data I reckon

Update: I've got to the bottom of the snow difference, and it lies with the input data. Looking at a few sample points, snw in LImon has much smaller values (e.g. factor of 4) than the snow variable used by AutoAssess. I've contacted our local snow expert, but since it relates to something quite fundamental, I think that we should plough (pun intended) ahead without it. Are you OK for me to delete the snow recipes from this branch?

@valeriupredoi I've updated the battle plan in the PR description. Is anything missing? I'm not sure what the "missing plots" one was about - is it still a thing?

@valeriupredoi I've updated the battle plan in the PR description. Is anything missing? I'm not sure what the "missing plots" one was about - is it still a thing?

@alistairsellar great, cheers! OK, let's forget about snow, summer's here anyway :grin: The missing plots in gallery bit was to remind us to add the plots to recipe figures (I think only permafrost outputs plots anyway, but we could add the metrics plots, if we run the recipes with two different models, probably would be nice to have those too?). And yes, we should remove the MOHC test recipes too (I've updated the battle plan as well)

Hey @valeriupredoi, I think I've done everything that I know how to! If there are outstanding tasks, please tell me.
Key updates:

Documentation for permafrost, soil moisture & surface radiation now complete, including metrics plots and permafrost maps
Fixed the permafrost recipe that was failing due to mrsol missing, by using mrsos instead (as you suggested at one point I think). Results unaffected due to the limited way in which this field is used.

I've ticked off most items in the PR description. Outstanding are:

closes issue: should I open an issue retrospectively, or reference #1931?
code quality guidelines: is it enough that the automated tests pass?
recipe is well documented: given that I'm the one adding the documentation, this should be reviewed by someone else
provenance information: I think I need technical help on this one

Fantastic work @alistairsellar :beer: :tophat:

I've ticked off most items in the PR description. Outstanding are:
* closes issue: should I open an issue retrospectively, or reference #1931?

referencing it is enough, we should keep that open for the next stuffs (inc. snow which is under snow now)

* code quality guidelines: is it enough that the automated tests pass?

that's perfect, documentation builds fine and we don't really have functional tests for recipes anyway (yet)

* recipe is well documented: given that I'm the one adding the documentation, this should be reviewed by someone else

good point, shall we say @zklaus maybe - since he's the release manager anyway?

* provenance information: I think I need technical help on this one

we should just see what sort of provenance info we provided for Stratosphere, that should suffuce I believe -> @bouweandela what do you think?

I changed the PR from Draft to RfR, @zklaus would you be a peach and have a look over the documentation pls mate :peach: Ali, it's you the technical/scientific reviewer :grin:

Fab, thanks V!

we should just see what sort of provenance info we provided for Stratosphere

Sounds sensible - please can you point me to that? Sorry for the basic question.

@alistairsellar you're a star :star: For the references/provenance here's how the documentatuion tells you to do it - I am not sure if we need to add any here, @bouweandela is the provenancespecialist (it's his baby after all) maybe he can help out here? :beer:

Provenance tags now added to the recipes. I think that completes everything from my side.
Status summary:

I've reviewed the science aspects, having run equivalent AutoAssess diagnostics to check for consistency of results (comparisons logged above).
The documentation should ideally be reviewed by someone else, since I added this to ESMValTool (although it was based on the original AutoAssess documentation for these metrics). Documentation pages are:

Permafrost
Surface radiation
Soil moisture

Technical review required

@remi-kazeroni noted that the documentation for these recipes was not following the template. They have now been reformatted to follow it much more closely. The only deviation from the template is an additional section at the end explaining about the AutoAssess wrapper script, which could be helpful to some users if they need to make changes.

as for provenance output - let me do that in a separate PR after we've merged this one, not all diags have provenance output set, so it's not a make or break, let's just get this one over the hill and then I'll add it for all the AA diags (incl Stratosphere which is currently missing it too)

Brilliant - thanks @remi-kazeroni!
I think that all of the points have been answered now, apart from this one:

Also, I couldn't reproduce exactly the figure Permafrost_Metrics.png. It seems to me that there is a scaling issue for the x-axis which extends from 0 to 200 in my case. The permafrost area and the fraction area permafrost over zerodeg seem to be correct assuming a factor 100 (% ?) while other quantities seem to be fractional ([0-1] ?). Could you please check that?

That's odd. I've just re-run this recipe from the latest revision and the plot is the same as https://esmvaltool--2170.org.readthedocs.build/en/2170/_images/Permafrost_Metrics.png. Please could you send or point me to your metrics.csv files from the output dirs? These should be
plots/aa_landsurf_permafrost/autoassess_landsurf_permafrost/*/land_surface_permafrost/*/metrics.csv.

That's odd. I've just re-run this recipe from the latest revision and the plot is the same as https://esmvaltool--2170.org.readthedocs.build/en/2170/_images/Permafrost_Metrics.png. Please could you send or point me to your metrics.csv files from the output dirs? These should be
plots/aa_landsurf_permafrost/autoassess_landsurf_permafrost/*/land_surface_permafrost/*/metrics.csv.

You're welcome @alistairsellar! I will review your changes now.
I have just rerun the recipe_autoassess_landsurface_permafrost.yml with the latest changes to this branch and still get the same behaviour. Here are the metrics files you asked (renamed *.txt to send them through GitHub). I hope I didn't do anything wrong to run this recipe.

E3SM-1-0_metrics.csv.txt
ACCESS-CM2_metrics.csv.txt

we should just see what sort of provenance info we provided for Stratosphere, that should suffuce I believe -> @bouweandela what do you think?

as for provenance output - let me do that in a separate PR after we've merged this one, not all diags have provenance output set, so it's not a make or break, let's just get this one over the hill and then I'll add it for all the AA diags (incl Stratosphere which is currently missing it too)

Provenance is required for new diagnostics, the fact that you got away once without adding it doesn't mean it's not needed. Could you please add provenance in this pull request for those diagnostics that are new in this pull request?

OK OK, I'll a function now :hammer_and_wrench:

Thanks!

Thanks V! :beers: 2u
Let me know if you need any input information for the provenance, or any 🍺 .

on it @alistairsellar - should be fine, I'll just output stuff that's already in the code, I'll ping you if I gets stuck :stop_button:

dunit @alistairsellar - could you pls have a quick look at the provenance parameters (eg statistics, domains etc) if they're correct? If all's good I reckon we done and done :grin:

@remi-kazeroni thanks for the files. Looking into this now...
One question - where are you running it? Or perhaps more importantly, which ESGF node is it pointing to?

@remi-kazeroni, OK so your metrics.csv file for E3SM-1-0 is identical to mine down to the last digit, but ACCESS-CM2 differs in those two metrics by a factor of 100
Me:
    permafrost area,11.943403914017559
    fraction area permafrost over zerodeg,0.2789722958859204
You:
    permafrost area,0.11943403918041137
    fraction area permafrost over zerodeg,0.002789722959799026

(and is identical in all others).
These normalised assessment metrics are a ratio between the control and the experiment, so if one of those is out by 100 then so is the ratio, and hence the metrics plot.
So why is ACCESS-CM2 wrong in your test, just for these two metrics? Your hunch has got to be right that there is a fraction / percent mixup here. These are the only metrics that make use of the sftlf land area mask, so my guess is that your ESGF node has ACCESS-CM2 sftlf as a fraction (range 0-1) while the Jasmin/CEDA node has it (correctly) as a percentage. Is it possible that ACCESS initially delivered sftlf with the wrong units and subsequently corrected it, and that Jasmin has a newer version of that data than the ESGF node you are pointing to?

So why is E3SM-1-0 wrong in your test, just for these two metrics? Your hunch has got to be right that there is a fraction / percent mixup here. These are the only metrics that make use of the sftlf land area mask, so my guess is that your ESGF node has E3SM-1-0 sftlf as a fraction (range 0-1) while the Jasmin/CEDA node has it (correctly) as a percentage. Is it possible that E3SM initially delivered sftlf with the wrong units and subsequently corrected it, and that Jasmin has a newer version of that data than the ESGF node you are pointing to?
[Edit - I got the models the wrong way round initially. E3SM is the one that is too low by 100.]

Thanks for your investigations @alistairsellar! I'm running the recipes at DKRZ which has an older version of the data you are referring to: /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/CMIP/E3SM-Project/E3SM-1-0/historical/r1i1p1f1/fx/sftlf/gr/v20190919/sftlf_fx_E3SM-1-0_historical_r1i1p1f1_gr.nc. That file contains indeed only [0-1] values... But I see that the version v20201015 is made available on Jasmin but not on Mistral. That must explain the discrepancies. Sorry for the inconvenience!

That must explain the discrepancies.

Phew! That's good news.

So apart from the small matter that my last commit has broken the tests, I think we're done. Provenance now added, and metrics issue resolved. Anything I've missed?
I'll fix that final error now...

Test now fixed. Back to you @remi-kazeroni - I think we've covered everything raised by your review.

[Edit (by Alistair): correcting the quote since I corrected my original post that it was quoting. Also, I think you meant lower on DKRZ, rather than higher. I don't know if it's bad form to edit someone else's post, but I didn't want to confuse Remi...]

good point and call, Alistair! I am working in 30C at home, I don't even know how me brain is not mush :hot_face:

Forgot to ask: do you want to edit the PR's title or is it fine like this for the merge and the changelog?

cheers @remi-kazeroni (sorry @alistairsellar I re-edited the title :grin: ) - I have also updated https://github.com/ESMValGroup/ESMValTool/issues/1931 that should not be closed yet since snow still needs attention but now it contains the correct pointers to the work here

Thanks for adding the provenance @valeriupredoi & @alistairsellar! For each three recipes, a diagnostic_provenance.yml file is created. But 2 of these recipes crash before the end because of the provenance. There seems to be a last bug for soilmoisture and surfrad... Could you please have a look? I attached the main_log_debug.txt and log.txt for one of these.
log.txt
main_log_debug.txt

eek! I'll have a look now, cheers mate :beer:

OK @remi-kazeroni - fixed it, silly code was trying to re-write the record when running the bit to plot the metrics, my bad, I should have tested by running the thing yesterday (I have now :grin: )

brilliant! cheers ever so much, Remi! since you helped so much with it, how about you merge it :medal_sports:

brilliant! cheers ever so much, Remi! since you helped so much with it, how about you merge it 🏅

You're welcome @valeriupredoi! Thanks to you and @alistairsellar for all the work! Great to have this merged for the next release and the IS-ENES3 deliverable D9.2 👍

That's great! Thank you @valeriupredoi & @remi-kazeroni for your time on this! Great to be another step closer to using ESMValTool in operations at the UK Met Office.


Check for Updates button not doing anything on VSCode snap

The Check for Updates button in the Manage menu does not do anything on the Snap version of VSCode.

VSCode Info
Version: 1.67.2
Commit: c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5
Date: 2022-05-17T18:23:40.286Z
Electron: 17.4.1
Chromium: 98.0.4758.141
Node.js: 16.13.0
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 5.15.41-1-MANJARO snap

Can you run F1, Developer: Show Logs..., Main and then show me what comes up when you run Check for Updates...?

It seems to be checking for updates in the background, but there is no visual feedback for the user.
Here are the logs:
[2022-06-08 19:39:41.890] [main] [info] update#setState checking for updates
[2022-06-08 19:39:41.893] [main] [info] update#setState idle

Right, but other VSCode packages show a pop-up saying that the latest version of VSCode is installed when the Check for Updates button is clicked.
It would be nice if the Snap showed that pop-up as well imo. Or maybe remove that option entirely since all Snaps update automatically.


V0.12.1.x - reduce IS/PS fees 10x

instantsend min fee to 0.001
PrivateSend fee to 0.001

@bertlebbert I overlooked 1K limit. changed to 999.999. Thank you.

utACK


feat: consensus mechanism to write to the CAS/ledger

Problem
Currently, writing to the ledger of the network is a process that is not synchronized. That is to say that there is no consensus between the peers, and as such data could be lost during the process as multiple actors could try to write at the same time to the ledger - and only one of them would win. This is a problem because we cannot guarantee data integrity, and moreover, since we want to have a CAS ( https://github.com/masa-finance/masa-oracle/issues/381 ) this would lead to invalid data across the nodes.
Proposed solution
As a first pass, we could implement raft consensus among the peers. There is already a well known library ( https://github.com/libp2p/go-libp2p-raft ) which can work with libp2p to have a basic raft consensus.
The idea is to have raft to guarantee that there is only one leader in a specific moment that can write to the ledger, and all the other nodes would have to accept new blocks only by the leader.
Additional context
See also an example of using raft and libp2p here: https://github.com/mudler/go-libp2p-simple-raft
Acceptance criteria

[ ] Nodes should agree on leadership before trying to attempt to write to the network
[ ] only the leader will write new blocks to the network when needed
[ ] new blocks are going to be accepted by the peers only if written by the current leader
[ ] the leader should accept data to write on the ledger by other nodes via libp2p protocol

[ ] if non-leader nodes needs to write data, they should try to acquire leadership, or send the data to the leader

blocked by https://github.com/masa-finance/roadmap/issues/66 as we want to make sure we have all the requirements first


Update to Rails 4.2.1 from 4.2.0

https://github.com/eliotsykes/rspec-rails-examples/pull/44

Closed via #44


Feature/peoplegroups

💬 Description
Didn't realise this had gotten quite so large...
This adds a People & Groups webapp story and adds a few needed extra props/helper classes here and there.
All component changes are really minor and wont effect current functionality, like i said above, its just all classes/props that ended up being required.
Taking a look at the gif is probably a good idea to get the gist of what we're after for the People & groups story!
📹 GIF (optional)
http://g.recordit.co/FRVFfkOg59.gif
🚪 Start Points
stories/webapp/peoplegroups/PeopleGroupsStory.js
✅ Checklist

[ ] Tests written
[ ] Browser tested
[ ] Added to documentation
[ ] Added to storybook

 Set up finder panel view components in gather-ui


Code Clean Up

Separating the code into the 3 folders: Objects, Screens, Widgets

The code has been organized into lib/objects/, lib/pages/, and lib/widgets/ folders. A few further cleanup steps should be taken:

[ ] Remove the lib/notNeeded/ and water_quality_app/ folders
[ ] Remove any unused Firebase files; this may include .firebaserc, firebase.json, firestore.indexes.json, and firestore.rules
[ ] Remove the functions/ folder (unless we are using Firebase cloud functions)
[ ] Remove unused assets from the assets/ folder


Agenta web failing to start correctly

Describe the bug
Agenta web fails to start
To Reproduce
Steps to reproduce the behavior:
Spin up Docker using docker compose
Go to http://localhost:3000/ in your browser
See error below

Expected behavior
Agenta web should display as normal

try simply localhost

As @aakrem mentioned, can you please try running http://localhost.
If this does not work, can you please provide us with the following information to be able to debug the issue:

the Network logs in the developer console in the browser.
A screenshot with running docker containers
The logs for the agenta-backend docker container

@MohammedMaaz Can you please look into improving the error handling for this error.

@aakrem @mmabrouk running http://localhost/ face the same issue. here is some info:

**agenta-backend logs: **
INFO:     Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://<IP_ADDRESS>:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [1] using StatReload
INFO:numexpr.utils:Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:numexpr.utils:NumExpr defaulting to 8 threads.
INFO:     Started server process [8]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 221, in connect_tcp
await event.wait()
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 597, in aexit
raise exceptions[0]
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_sockets.py", line 167, in try_connect
stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1627, in connect_tcp
await get_running_loop().create_connection(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection
sock = await self._connect_sock(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 500, in sock_connect
return await fut
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 121, in connect_tcp
stream._raw_socket.setsockopt(*option)  # type: ignore[attr-defined] # pragma: no cover
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 119, in exit
raise TimeoutError
TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 92, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 69, in handle_async_request
stream = await self._connect(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 117, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
return await self._backend.connect_tcp(
File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 121, in connect_tcp
stream._raw_socket.setsockopt(*option)  # type: ignore[attr-defined] # pragma: no cover
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectTimeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 677, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in aenter
return await self.gen.anext()
File "/app/agenta_backend/main.py", line 47, in lifespan
tags_data = await retrieve_templates_from_dockerhub_cached(cache=cache)
File "/app/agenta_backend/services/cache_manager.py", line 37, in retrieve_templates_from_dockerhub_cached
response = await retrieve_templates_from_dockerhub(
File "/usr/local/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry
ret = await target(*args, **kwargs)
File "/app/agenta_backend/services/container_manager.py", line 97, in retrieve_templates_from_dockerhub
response = await client.get(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1757, in get
return await self.request(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1530, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectTimeout
ERROR:    Application startup failed. Exiting.

@MachineWei Did you change the port in the docker from 80 to 90 by chance?
@aybruhm Maybe we can add a section in the documentation on how to run on different port than 80.

@MachineWei Did you change the port in the docker from 80 to 90 by chance? @aybruhm Maybe we can add a section in the documentation on how to run on different port than 80.

Yes, we can.

@mmabrouk yes, i change it to 90 in agenta-web/.env as i met the same error before.
@aybruhm it doesn't help restarting the backend container

@mmabrouk @aakrem   i hava fixed the backend (i have some issues accessing the dockerhub online):

but i still face the issue, the web container log as follow( I really don't understand this)

@mmabrouk @aybruhm
i have fixed the backend (There is a problem accessing the Docker Hub on my network)
Now it seems to be running normally

today i update it to 0.2.7, but still face the issue

here is some info:

localhost is the point.
if someone want to deploy this project on the server
step 1. change agenta-web/.env  localhost to your server ip
step 2. change agenta-backend/main.py  set origins = ["*"]

Thanks @MachineWei for the solution.
@aakrem what's your feedback? Don't we have a docker file that does this automatically? Can we add this to the documentation?

@MachineWei actually you didn't mention anywhere that you want to deploy it on a server.
I could invited you to use terraform to launch it in an aws instance or simply follow this script below launch it, which will do the setup you did.
@mmabrouk We can add to the docs how to run it on a random machine.
To deploy on EC2 machine
https://docs.agenta.ai/installation/self-hosting/self-hosted-installation-aws
Script to deploy on whatever server (you need to populate the domain):
https://github.com/Agenta-AI/agenta/blob/main/self-host/aws/instance-setup.sh

@aakrem thank you and sorry
yes , i did not mention anywhere that i want to deploy it on a server
@mmabrouk
thank you for your support, too


Handle escaping for different SQL_MODE parameters

mysql.escape() and prepared statements do not escape strings properly with different SQL_MODE parameters are used.
My company has our MySQL database default to NO_BACKSLASH_ESCAPES mode. In this mode ' must be escaped as '', rather than \'. We have found out (the hard way) that this library always assumes MySQL is operating in standard mode.
This is especially surprising in prepared statements (or what appear to be prepared statements but just borrow the syntax):
dbconn.query('select * from mytable where tableName = ?', [userName], function(e, r) { ...
As a workaround, I'm now running SET SESSION SQL_MODE = '' immediately after creating the DB connection. But it would be much better if this were somehow incorporated as an option. Either a global config for the mysql object, or a config option on the DB connection. Or at the very least there should be a big warning somewhere in the documentation.
Example of a potential vulnerability:
var username = "' OR userid=17 --";
dbconn.query('select * from users where username = ?', [username], function(e,r) { ... });

In this case, query is executed like:
select * from users where username = '\' OR userid=17 --'

In NO_BACKSLASH_ESCAPES mode, that means:  where username is a backslash or userid=17.  Assuming there's no user named backslash, but userid 17 is somehow interesting, then you've exposed data.

Pull requests for any of the options are certainly welcome!

Created pull request 1830. This is just a warning in the ReadMe.
I may try to implement something. Ideally this would be a configuration on the DB connection, but mysql.escape() doesn't take a connection object. I wouldn't want to require passing an optional flag to mysql.escape() everytime either, as developers would be likely to forget it once in a while. So I'm thinking the best course of action would be to add a global config for the module, something like:
var mysql = require('mysql');
mysql.config({
  noBackslashEscapes: true
});

From then on, any calls to mysql.escape(), or when running .query() with ? replacement variables, would escape strings using the rules of NO_BACKSLASH_ESCAPES mode.
What do you think of this idea? I don't want to go through the effort of learning your code and coding it if this is just going to be rejected. :)
If there is another approach you would prefer please let me know.

It's possible, but then you'll be in a similar (or worse?) situation if you are connecting to two servers in the same process where one has NO_BACKSLASH_ESCAPES and one does not.

Yes I thought about that too. In my experience that would be uncommon, but maybe for someone else it is not.
I just noticed in re-reading your docs that there is also a connection.escape() method. So I think you could make it a connection-specific flag. And add a warning around the part in the ReadMe where you suggest doing mysql.escape(). Something like this:
var mysql      = require('mysql');
var conn1 = mysql.createConnection({
  host     : 'example.org',
  user     : 'bob',
  password : 'secret'
});
var conn2 = mysql.createConnection({
  host     : 'example.org',
  user     : 'bob',
  password : 'secret',
  noBackslashEscapes : true
});

console.log(conn1.escape("hello ' world \\ goodbye"));   //prints:  'Hello \' world \\ goodbye'
console.log(conn2.escape("hello ' world \\ goodbye"));   //prints:  'Hello '' world \ goodbye'
console.log(mysql.escape("hello ' world \\ goodbye"));   //prints:  'Hello \' world \\ goodbye'
console.log(mysql.escape("hello ' world \\ goodbye", {noBackslashEscapes:true}));   //prints:  'Hello '' world \ goodbye'

Without telemetry, it's impossible to know what is common and what is not; best just to support as much as possible. I think that makes sense. I'm not very familiar with the NO_BACKSLASH_ESCAPES and didn't find a section on how things like 0x00 0xa0 and the like are supposed to be escaped when that is enabled. Do you know all the escaping rules when NO_BLACKSLASH_ESCAPES is on? Maybe some of those just don't need escaping in that case?

In fact, it looks like the server will indicate in the initial handshake if it is in NO_BACKSLASH_ESCAPES mode, so I would say use that from the server to set an internal flag instead of requiring the user to specify it as a connection option so the handling is always correct.

Hmm okay. I don't think I've ever encountered those kinds of characters in strings. My company uses the flag because of porting from SQL Server to MySQL some years back, and the no backslash mode makes it behave like SQL Server.
Using the handshake value might actually break the code for anyone using the same workaround as me: In my production code I'm calling SET SESSION SQL_MODE = '' after connecting.  If the handshake indicates no_backslash_escapes mode, but then I change it for my session, I'm not sure if the connection object would know about that. (I don't know much about the low-level stuff.)

Ah, yea. So that would mean that after every query the status flags from the OK packet should be checked for the NO_BACKSLASH_ESCAPES and update as necessary, which would automatically account for that I believe.
Re chars in string, in order to implement escapes without backslashes, probably need to understand what the escaping rules actually are. I think the rules for ' is clear: it becomes ''. What about the " character? I assume that becomes "" as well? Perhaps each character that needs to be escaped is just doubled?

Why are you escaping " anyway? You always add the single-quote before and after the string, so you know your string is always going to use single-quote as the delimiter.
console.log(mysql.escape("'Test'")); //prints:   '\'Test\''
console.log(mysql.escape('"Test"')); //prints:   '\"Test\"',  but you could just return: '"Test"'

That's how the module was before I was here, so don't have an answer for you on the why.


[VL][CI] Add a nightly build job

What changes were proposed in this pull request?
This PR proposes a nightly job to verify gluten build script from a clean environment.

Gluten's existing CI jobs are using docker images with pre-installed dependencies to accelerate validation process, but it is possible the code doesn't work in a clean environment.
E.g., Gluten changes velox's setup script by using "sed" command to match some code, then modifying it. The expected change sometimes doesn't happen due to upstream's code changes.
dev/package.sh is still necessary to some users, even though static build is recommended. It lacks CI verification.

To avoid slowing down PR verification process, this job will be triggered only if the added yml file is changed or scheduled nightly test time comes.
How was this patch tested?
N/A

@rui-mo, do you have any comment?


Warn about supported locales that lack localizations

Fixes #13482, #14890, #16405, #22205
Added missing debugCheckHasMaterialLocalizations() calls.
Added a check to WidgetsApp that prints a warning if the app's resolved locale isn't supported by all of the LocalizationsDelegate types specified by localizationDelegates.
For example if a MaterialApp was configured like this:
MaterialApp(
  localizationsDelegates: [
    GlobalMaterialLocalizations.delegate,
    GlobalWidgetsLocalizations.delegate,
  ],
  supportedLocales: const <Locale>[Locale('foo', 'BAR')],
  locale: Locale('foo', 'BAR'),
  // ...
)

The following console warning would appear.
════════
Warning: This application's locale, foo_BAR, is not supported by all of its
localization delegates.
> A delegate that supports MaterialLocalizations was not found.
See https://flutter.io/tutorials/internationalization/ for more
information about configuring an app's locale, supportedLocales,
and localizationsDelegates parameters.
════════

This is only a warning because an app configured this way could still run correctly, if it didn't create any widgets that depended on MaterialLocalizations.

Also, the PR description promised "Added missing debugCheckHasMaterialLocalizations() calls." - I don't see any new calls in the PR. :(

That's true. I seem to have lost them in the midst of local git stash, git pop etc. thrashing.  Will restore them.

Sorry I missed this before.
I don't think we should have warnings. It will lead to log spam where we output warnings that aren't errors and that are therefore ignored and eventually people ignore all our output even the important errors.
Case in point: it turns out this warning has been firing for our tests for weeks now but nobody has done anything about it.
I think we should turn this either into a fatal error, or remove the warning entirely.

I made it a warning to avoid having existing apps and tests suddenly start failing because the cupertino localizations only exist for english. Once the cupertino translations are in (#23847), I'm happy to make it an error instead of a warning.

Ah, sounds good. Thanks.


GO: implement PING command

Describe the feature
Keeping the focus on adding the missing commands to the GO client
As a good first issue for people that wish to contribute to the GO client can be the implementation for the command: PING
See command details here:
https://valkey.io/commands/ping/
Use Case
Missing functionality
Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

Client version used
1

@MikeMwita Wanna take it?


A bug in the module

I'm running some tests over the ecosystem and found a bug in the module. Sub gray-decode defines its return value as Int but returns a Str instance. To fix it line 9 must be either $n.Int or +$n.

Hmm. The problem was actually incorrectly implemented coercion signatures. Or more accurately, a half-assed attempt on my part to use coercion signatures without fully understanding how they should be used.
Fixed with commits 236125ec5dc4922ec771f962a22b3ef4fdab796e and 09bbc37ce208c95b74c8d71d8551c9d6fdf95d6c
Expanded testing to explicitly test passing String or Int arguments.
Thanks


perf(cdk): replace setTimeout with rAF and skip change detection

PR Checklist

[x] The commit message follows Conventional Commits

PR Type

[x] Refactoring (no functional changes, no api changes)

Does this PR introduce a breaking change?

[x] No

I don't wanna create breaking changes, but I'm not sure if we need to run tick() when the element gets focused since it's done via the native browser's API, which doesn't require Angular to run change detection. That's why I've also removed markForCheck() to prevent Angular running change detection down to the autofocus directive after setTimeout fires.

Hi @arturovt. Any particular reason you came up with this PR? Were you seeing any issues related to autofocus feature?

@arturovt pinging you to see if this is still relevant. This directive is messy as is, thanks to brilliant iOS webkit. I wouldn't want to introduce yet another bunch of workarounds unless there are strong reasons to do so.


remove all references to iOS 9 frameworks from podfile (Xcode 6 compa…

…tibility)
@aaustin after approval, hold on branch until release

:+1:

GM seeds of Xcode 7 are now available, probably don't need to remove these. In fact, could probably move them back into the primary Frameworks section.

Unfortunately some users are still on Xcode 6.4 which is why we're having to pull this.


ENHANCING THE USER INTERFACE OF THE PASSWORD GENERATOR

### ENHANCING THE USER INTERFACE OF THE PASSWORD GENERATOR
Improving the User Interface (UI) of the Password Generator is essential to provide users with a seamless and efficient experience. A well-designed interface not only makes the tool more appealing but also enhances usability and functionality.

Hi , I hope you're doing well. I I believe I can contribute effectively, and I'm eager to help resolve it. If it's possible, could you please assign me this task? I would really appreciate the opportunity. Thank you!"

Sir please. I humbly request to you to please reassign this task to me . Cause I had successfully solved this issue and also requested the PR to merge. But you told me that you are not assigned to this issue and hence by mistake I clicked on the unassign button . Please assign me and accept the solution. I had taken it to the next level. Allready all the work hard done just need to merge. #81 @iamrahulmahato .
Please sir respect my hardwork . You can see the changes in the PR request that I had created..