Not really an issue:   Just another approach to parsing the scrape of rarbg

Thank you for all the info I found in your MagnetMagnet python set up.
I have played around a little with the rarbg and found a different way to parse the json coming back.
Just have pasted the code below to see if you can use it in the future:
`
import requests
import os
main_link = 'https://torrentapi.org/pubapi_v2.php?mode=search&search_string=almost paradise s01e08&token=lnjzy73ucv&format=json_extended&app_id=lol'
main_request = requests.get(main_link, headers={'User-Agent': 'Mozilla/5.0'}).json()
if main_request == {'error': 'No results found', 'error_code': 20}:
quit("Not Found")
titles = main_request['torrent_results']
dl_options = []
for title in titles:
name = title['title']
magnet = title['download']
seeders = title['seeders']
size = title['size']
dl_options.append((size, seeders, name, magnet))
dl_options.sort(reverse=True)
print()
for dl in dl_options:
print(dl[0], dl[1], dl[2], dl[3])
print()
print(dl[0][0])
quit()
print('Starting Download via RARBG', dl_options[0][3])
command_str = 'open -a transmission ' + dl_options[0][4]
os.system(command_str)
`

Wow. This is way more efficient! MagnetMagnet was my first project, so I didn't learn JSON yet (still haven't) and went for web scraping with beautiful soup and requests first.
I'll read into your code and try to understand it within the next 2 days.
Thank you for your interest in my project 😃

I've tried to use you're code, but it added complications with how I set up my code.
So I've just spared myself the headache and didn't bother swapping out the code.
Hey, if it ain't broke don't fix it.
Sorry for not using your great code.


feat: allow sending queries with GET method

This PR introduces a new preferGETQueries flag which when enabled, sends all query requests using the HTTP GET method.
Closes #219

I think there might be a bug with this not working server side… These requests seem to be using POST despite having declared preferGETQueries: true in nuxt.config.


[TRACKER] GPU/driver-specific issues

This tracker lists GPU driver-specific issues that affect the new renderer in Godot 3.x.

[ ] OpenGL error: GL_INVALID_OPERATION in DisableVertexAttribArray [intel, Windows] #7906
[ ] Stream of errors when enabling refraction and SSS [radeon, Windows] #9139
[ ] GL error spam when using a viewport (Sampler object 0 is bound to non-depth texture 0) [nvidia, Linux] #9335
[ ] OpenGL fragment and vertex shader error spam [nvidia, Windows] #9434
[ ] graphical issues with material_tester demo [intel, Linux] #9525

I have #7906 and #9139. I assume they're driver related too.

I have #7906 when I try to use 3.0 - the same stream of errors that make the application unusable as it becomes very slow.
ERROR: _gl_debug_print: GL ERROR: Source: OpenGL        Type: Error     ID: 1282        Severity: High  Message: Error has been generated. GL error GL_INVALID_OPERATION in DisableVertexAttribArray: (ID:<PHONE_NUMBER>) Generic error    At: drivers/gles3/rasterizer_gles3.cpp:123

#9725 maybe?

#7906 and #9525 are fixed.

This will probably be solvable mostly in 3.2

All these issues are fixed, so closing.
That doesn't mean that there aren't any GPU specific issues left, but this tracker has outlived its initial purpose.


Artemis image throws unable to expand property AUTHENTICATION_SERVICE_HOST

Artemis image throws unable to expand property AUTHENTICATION_SERVICE_HOST when authorization is disabled.
keycloak.log
keycloak-controller.log
agent.log
artemis.log

@kornys is this issue still reproducible?  If so what are the steps?


Server rules open when starting a dev client

Description
It would be nice if we can put this off for dev instances of the startup.
Reproduction

Startup local/dev client/server.
Get shown empty rules

Screenshots

Additional context
NA

Yeah it was kind of silly when i loaded it up and saw that

fixed in #28752


Tags not properly updated to 1.21

As part of 1.21, Mojang changed the directories for various tags to use singular names instead of plural.
Your mod uses tags in one or more of the old directories that was changed! This means those tag files will no longer load in 1.21 unless you update the directories!
The affected directories and their new counterparts are:

tags/blocks -> tags/block
tags/items -> tags/item
tags/entity_types -> tags/entity_type
tags/fluids -> tags/fluid
tags/game_events -> tags/game_event

Please ensure that any of these that you are using is updated to the new name in all namespaces (including your own, minecraft, and c).

Oh wow thanks for letting me know. I don't know why I didn't notice, because all of my tag translations are using singular form there.
I've updated the mod now with these changes. Thanks again for notifying me about it!


Make suggested keyboard shortcuts clickable

Last time we discussed this we agreed that it would either be hard to discover and maybe unexpected if the mouse cursor does not change on hover or the changing mouse cursor would be distracting during regular work.

@chrmarti About your points:

For me, it was unexpected when I clicked and nothing happened.
I do not think the mouse cursor should change on hover (see https://github.com/microsoft/vscode/issues/55346).
To solve the discoverability problem they could be styled as buttons.

Overthinking here, maybe the user could be able to configure his set of "quick actions".


Two UPFs on same host not working (any more)

Hi,
I'm running two UPFs on the same host listening on the same interface (but on different IPs).
Unfortunately only the UPF started second will receive GTP packets. I assume the second UPF is kicking out the first UPF. I already tried the newest version of upf from the main branch together with the newest kernel module (v0.3.1).
Can someone confirm this bug?
I started them as follows:
sudo -E NFs/upf/build/bin/free5gc-upfd -f config/upfcfg_ralf.yaml &
PID_LIST+=($!)
sleep 1

sudo -E NFs/upf/build/bin/free5gc-upfd -f config/upfcfg_ims_ralf.yaml &
PID_LIST+=($!)
sleep 1

upfcfg_ims_ralf.yaml.txt
upfcfg_ralf.yaml.txt

Hi @ralfkundel,
Your confguration of two UPF looks good.
Could you please capture the two UPF run log with "trace".

Hi,
attached the two log files with trace. Note that both UPFs are listening on the same interface with diferent IP addresses. I have not tested it with two interfaces on one machine.
upf1.txt
upf2.txt
For me, there is no error detectable. Note that the following script works:
run_dual_upf.sh.txt
the difference is this script creates a namespace and starts the second UPF within the namespace. This makes setting up all the routes more complicated but is a workaround for now ;-)
best and thanks,
Ralf

Hi Ralf ,
Thanks for the update, I will let you know once i verified the given log
Thanks & Regards,
Muhturaman

Hi Ralf,
I verified your work around with UPF source code.
In function  _gtp_dev_create(), by default the dest_ns is -1, and it may cause your problem. Because, both interface referenced same namespace which results forwarded the packets to one interface as you mentioned.

Unfortunately only the UPF started second will receive GTP packets. I assume the second UPF is kicking out the first UPF. I >>>already tried the newest version of upf from the main branch together with the newest kernel module (v0.3.1).

static int _gtp_dev_create(int dest_ns, const char *gtp_ifname, int fd, enum ifla_gtp5g_role role)

if (dest_ns >= 0)

mnl_attr_put_u32(nlh, IFLA_NET_NS_FD, dest_ns);

It can be fixed by passing the dest_ns (for the flag IFLA_NET_NS_FD) value in function Gtp5gDeviceAdd(),
status = gtp_dev_create(-1, dev->deviceID, gtp5gDevice.sock->fd);
This fix required to modify the config of UPF YAML as well parse it.

@ralfkundel
If you want to run 2 UPFs on the same machine, you need to create new network namespace.
Or run UPFs on different VMs/docker containers.
This is deployment issue, not UPF issue.
You can try and report us which way is workable.

Hi,
Thanks. Yes, i can confirm all 3 approaches to work.

Namespaces are the easiest solution, however, the kernel version must be the right one for the gtp kernel module.
Using VMs worked on our lab as well. This solution is very nice as you can run any kernel on the hypervisor Independent to the VM. However, configuring this takes some time ;-)
Docker we only tested with the same kernel on the hypervisor. But I don‘t see any issues in using different kernel versions. Configuration overhead is similar to a VM but you can script it much better/faster.
Alternative approach 4: just take two servers. This is the easiest one to debug ;-)

best and thanks


Save encoding warning does not appear for project files

Start VS, do not open a project.
Open a python file, paste this at the top:
#!/usr/bin/env python
# -*- coding: cp866 -*-
print('Бóльшая')

Save As, Save With Encoding, Cyrillic (DOS) - Codepage 866
Result: Correct behavior, you get a warning message that says you will lose data and you should change encoding (that 2nd char cannot be encoded).
Now repeat the same operation, but this time, with a file inside of a Python project.
Result: NO warning message. When you close and reload the file, you'll see the 2nd char passed to print is replaced with ?.
FYI RTVS had a similar issue, and it was discovered that the bug was in CPS.
https://github.com/Microsoft/RTVS/issues/2471 (scroll down to the bottom)
Also we discussed the lack of warning in https://github.com/Microsoft/PTVS/issues/74 but the no warning issue was not fixed.

Should be covered by #1952


OnlyOffice stops saving document back to NextCloud

Problem:
OnlyOffice in some case doesn't save document back to NextCloud. It happens time-to-time, with different documents, I can't identify, which situation causes it.
If needs, I have example of such document, and can collect all info needed for diag.
I have last versions of OnlyOffice and NextCloud 12.
I tried to set force save for every 5 minutes in default.json, but it seems, that OnlyOffice totaly ignores this param.

Hello @duckhawk,
Please specify if JWT is on in Document Server and NextCloud.

Sorry, I don't know, can you tell, where I can check it?

Hello @duckhawk,
Check the advanced server settings in NextCloud administrator section. When JWT is off the field 'Secret key' is blank (see attached screenshot).

No, this field is empty.

@duckhawk,
Please send us logs of NextCloud in which we can check both cases (when the document was saved and when it was not saved). You can send them to<EMAIL_ADDRESS>

Hello @duckhawk,
Please specify if the issue still exists.

Which logs I can send to you?
I can't find any logs, that belongs to nextcloud/onlyoffice on my server.

Hello @duckhawk,
Please specify the OS of the server where the document server is installed, so we can tell you where to find logs. You can get logs of Nextcloud in Nextcloud Logging section. Please see the attached screenshot.

There is Ubuntu 16.04 LTS installed on server (Onlyoffice is running in Docker, if it matters)

@duckhawk, the logs should be in the following directory: /app/onlyoffice/DocumentServer/logs/

Hello @duckhawk, please specify if the issue still persists, as we did not receive the logs.

This issue was closed due to no response.


Auth. fixtures for auth unit tests

Description
Auth. fixtures for auth unit tests
Type of change

[x] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library
[x] Unit tests

Checklist before requesting a review

[x] I have performed a self-review of my code.
[x] PR has passed all pre-merge test jobs.
[ ] If it is a core feature, I have added thorough tests.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.15%. Comparing base (59dab1d) to head (55c6a83).
Report is 20 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #901   +/-   ##
=======================================
  Coverage   95.15%   95.15%
=======================================
  Files          55       55
  Lines        2045     2045
=======================================
  Hits         1946     1946
  Misses         99       99

/lgtm


bigfloat doesn't work

If I try to use bigfloat variables I get an error:
ERROR: MethodError: no method matching randn(::Random.TaskLocalRNG, ::Type{BigFloat})
OT question: Can I use SymbolicRegression to generate random math functions?

Hi @qwertyjl,
Run this code to add the randn method for BigFloats and it will work:
using Random: AbstractRNG

try
    # In case randn(BigFloat) gets defined in the future
    randn(BigFloat)
catch e
    if isa(e, MethodError)
        Base.randn(::Type{BigFloat}, args::Integer...) = BigFloat.(randn(Float64, args...))
        Base.randn(rng::AbstractRNG, ::Type{BigFloat}, args::Integer...) = BigFloat.(randn(rng, Float64, args...))
    else
        throw(e)
    end
end

(I experienced the same issue here: https://github.com/JuliaLang/julia/issues/27858#issuecomment-1232264692 - apparently this will be added in Julia 1.9)

By the way - I'd be happy to accept a PR putting this code directly into SymbolicRegression.jl, if you'd be interested!
(Maybe a new Extensions.jl file - which could be included by the main SymbolicRegression.jl file.)


Ci/publish

Addresses #527 by introducing a method to publish crates using cargo-workspaces, a tool that publishes all crates in a workspace in the correct dependency order. This serves as an alternative approach to #569.
Changes Included in This PR

Consolidated package metadata into the workspace configuration.
Added a new workflow to automate the publishing process for crates upon a new version release.
Migrated the semantic release configuration from .releaserc.yml to .releaserc.cjs. This allows for a more specific configuration of the semantic-release-replace-plugin, which is used to update internal dependency versions. While this approach is somewhat more fragile than the one proposed by @amimart in #569, it is simpler and effective. Further refinements can be made if needed.

Summary by CodeRabbit

New Features

Introduced a new GitHub Actions workflow for automating the publishing of Rust crates.
Added a configuration for semantic release to streamline versioning and changelog generation.

Bug Fixes

Adjusted formatting rules for .cjs files in the editor configuration.

Documentation

Updated README for the axone-wasm package to reflect its new focus on CosmWasm URIs.

Chores

Updated multiple Cargo.toml files to utilize workspace configurations for various packages.
Added a directive to prevent the testing package from being published.

size-limit report 📦

Path
Size

target/wasm32-unknown-unknown/release/axone_objectarium.wasm
458.07 KB (0%)

target/wasm32-unknown-unknown/release/axone_law_stone.wasm
593.03 KB (0%)

target/wasm32-unknown-unknown/release/axone_cognitarium.wasm
882.05 KB (0%)

target/wasm32-unknown-unknown/release/axone_dataverse.wasm
697.14 KB (0%)

@coderabbitai pause


xcode10?

Any update for Swift 4.2? RealmSwift is failing.

On Xcode 10 and Swift 4.2 - I overcame the RealmSwift issue by reinstalling but I am having a strange problem in Extensions where the showBottomHairline function is failing, navigationBarImageView is nil and the forced unwrapping of it is failing.

internal extension UINavigationBar {

    func showBottomHairline() {
        let navigationBarImageView = hairlineImageViewInNavigationBar(self)
        navigationBarImageView!.isHidden = false
    }

Commented out the call to showBottomHairline and could get the reader to open successfully.  Unfortunately, clicking on Text resizing button resulted in another error
So yeah, I believe this needs to be updated for 4.2.  I simply can't really figure out why these failures are happening in my existing app.

Hi, any update for your dependency? RealmSwift is a bit to old, and its failing on XCode 10.

Hi @johnbanks, how did you fix it on XCode 10?

I think we should update Realm to the newest release or at least 3.7.1 version.
the 3.1.1 version is too old for Xcode 10

On Xcode 10 and Swift 4.2 - I overcame the RealmSwift issue by reinstalling but I am having a strange problem in Extensions where the showBottomHairline function is failing: navigationBarImageView is nil and the forced unwrapping of it is results in an exception. Any ideas would be appreciated.
internal extension UINavigationBar {

    func showBottomHairline() {
        let navigationBarImageView = hairlineImageViewInNavigationBar(self)
        navigationBarImageView!.isHidden = false
    }

@johnbanks how did you solve this issue? i also encountered this.

To update realm , remove FolioReaderKit and then install latest Realm version.
pod 'Realm/Headers' , '~> 3.11.0'
after that you install realm. It will solve realm issue. But still you will find alot of other issue as FolioReaderKit not supporting iOS12 :(

Thanks guys for troubleshooting this. If you have implemented the fixes or use this project please open a PR.

Disclaimer: I am closing GH Issues on this project, I don't work full time on it and I haven't been able to update that recently if you would like to help to maintain that and to push the development of this project forward, just reach me out or open a PR and we can discuss it there.


The icons at the top of the page are too close

On my version of Firefox, with slightly larger fonts, the text under the different icons is running together, and the icons themselves look squashed together with excess space on the margins.

Screenshot provided by @mkg33 this morning:

This also happens when using Safari on macOS (Firefox and Chrome with default font settings are fine).
Note that we're planning to overhaul the whole scipy.org site, so probably not worth a lot of effort now. But if it's a simple tweak to the spacing, then a PR is of course welcome.


Remove text stating C.ADDI4SPN is RV32C/RV64C-only

The paragraph of text describing C.ADDI4SPN states the instruction is
RV32C/RV64C-only, but the opcode map indicates the instruction is
available for RV128C too.  This change updates the descriptive text,
assuming that the opcode map is correct.

Yes, this looks like an editing error.  (Note RV128C isn't yet frozen, so anything could happen, but for the time being C.ADDI4SPN is part of it.)


Getting "AttributeError: 'list' object has no attribute 'get'" for array/list json response in

I am getting AttributeError: 'list' object has no attribute 'get' error when comparing json response with schema with jsonschema.validate().
I tried following:

When I use 0.3.8 version, the same code is working perfectly.
When I use python's requests module directly with same jsonschema.validate() its working fine.
When the response dont have list on parent, the validation works very well.

Example:
Test Schema:
schema= """
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "array",
"items": {
"type": "string"
}
}
"""
test_res="""[
"9A32SG",
"FS4EP6"
]"""
Stack trace:
python2.7/site-packages/jsonschema/validators.py", line 476, in validate
cls = validator_for(schema)
python2.7/site-packages/jsonschema/validators.py", line 431, in validator_for
return meta_schemas.get(schema.get(u"$schema", u""), default)

Greetings,
Are you using which version?
Are you using python or jython?
If possible could you share your robot test case?

I am using Python, My Robot Version is 2.9.
On Tue, Oct 11, 2016 at 8:31 PM, Mallikarjunarao Kosuri <
<EMAIL_ADDRESS>wrote:

Greetings,
Are you using which version?
Are you using python or jython?
If possible could you share your robot test case?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bulkan/robotframework-requests/issues/137#issuecomment-253109985,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVIaOhs5_j7JE2GOZr0cuhRh3nK_aov2ks5qzFSNgaJpZM4KUO9P
.

--
Chetan Kumar Dewangan
Software Engineer in Test
San Francisco
<EMAIL_ADDRESS><EMAIL_ADDRESS>Rally https://werally.com/ by Rally Health, Inc.
https://www.rallyhealth.com/
Leave your Rally STAMP on all that you do!
Stay hungry!
Teams win
Accountable: Own it. Measure it. Improve it.
Mission focused. Be it. Live it.
Proud. It’s what we are!  People matter, work matters.

@chetand24 I am not sure in what context your validating json schema, Could you please let me know how you are trying/writing Robot testcase to validate json schema, If possible could you please share a sample/replicated testcase?

@vkosuri  here is the code which I am using:
Python code which is working perfectly:
`
import json
import jsonschema
from jsonschema import validate
def should_match_schema(schema, test):
schemaJson = json.loads(schema)
test_json = json.loads(test)
validate(test_json, schemaJson, format_checker = jsonschema.FormatChecker())
sc= """
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "array",
"items": {
"type": "string"
}
}
"""
import requests
from requests.auth import HTTPBasicAuth
headers = {'Content-Type': 'application/json', 'accept':'application/json'}
req=requests.get("URL", auth=HTTPBasicAuth('user', 'password'), headers=headers, verify=False)
print req.status_code
print req.content
s = should_match_schema(sc,req.content)
`
Robot snippet, which is throwing the error:
`
Verify Test
[Documentation]     This test verifies the mobile api
set test variable    &{headers}     Content-Type=application/json       Accept=application/json
Set Schema Vars
Build Session
${resp}=   Make Request    GET
log json   ${resp.content}
Should be equal as numbers    ${resp.status_code}   200
Should match schema    ${resp.content}   ${ExpectedSchema}
`

@chetand24 Thank you,  by looking to into you testcase, It's not an issue, It's issue with Robot unicode data, For more information please go through this link http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#scalar-variables
You have to write some thing like this
json_validator.py
import json
import jsonschema
from jsonschema import validate

def should_match_schema(schema, test):
    schemaJson = json.loads(schema)
    test_json = json.loads(test)
    validate(test_json, schemaJson, format_checker = jsonschema.FormatChecker())

json_schema_validator.robot
*** Settings ***
Library           RequestsLibrary
Library           json_validator.py

*** Test cases ***
Json Schema Validator
    [Documentation]    This test verifies the JSON Schema
    Create Session    httpbin    http://httpbin.org
    ${ExpectedSchema}=    Set Variable    {"$schema": "http://json-schema.org/draft-04/schema#", "type": "array", "items": {"type": "string"}}
    &{headers}=    Create Dictionary    Content-Type=application/json    Accept=application/json
    ${resp}=    Get Request    httpbin    /get    ${headers}
    Log    ${resp.content}
    Should be equal as numbers    ${resp.status_code}    200
    Should Match Schema    ${resp.content}    ${ExpectedSchema}

@chetand24 I am closing this issue, if you have other inputs or comments reopen issue

@vkosuri my "Set Schema Vars" is
Set Schema Vars
    ${Schema}=    Set Variable      {"$schema": "http://json-schema.org/draft-04/schema#", "type": "array", "items": {"type": "string"}}
   Set suite variable     ${Schema}

Which doing exactly same thing.
As I said earlier my other endpoints are working perfectly fine. This issue is specific when the response body has  type:array in first json element.
Can you please reopen this issue.

I am not sure, could you provide an example of that response contains array strings?

It is provided in issue description, which is like:
test_res="""[
"9A32SG",
"FS4EP6"
]"""

I tried with your example it was went file. I have placed robot testcase, are you trying similar like this? If you have any questions please let me know
*** Settings ***
Library           RequestsLibrary
Library           json_validator.py

*** Test cases ***
Json Schema Validator
    [Documentation]    This test verifies the JSON Schema
    ${expected_schema}=    Set Variable    {"$schema": "http://json-schema.org/draft-04/schema#", "type": "array", "items": {"type": "string"}}
    ${test_res}=  Set Variable  ["9A32SG", "FS4EP6"]
    Should Match Schema    ${test_res}    ${expected_schema}

Starting test: Json Schema Validator.Json Schema Validator
20161014 10:41:36.083 : TRACE : Arguments: [ '{"$schema": "http://json-schema.org/draft-04/schema#", "type": "array", "items": {"type": "string"}}' ]
20161014 10:41:36.083 : TRACE : Return: '{"$schema": "http://json-schema.org/draft-04/schema#", "type": "array", "items": {"type": "string"}}'
20161014 10:41:36.083 :  INFO : ${expected_schema} = {"$schema": "http://json-schema.org/draft-04/schema#", "type": "array", "items": {"type": "string"}}
20161014 10:41:36.084 : TRACE : Arguments: [ '["9A32SG", "FS4EP6"]' ]
20161014 10:41:36.085 : TRACE : Return: '["9A32SG", "FS4EP6"]'
20161014 10:41:36.085 :  INFO : ${test_res} = ["9A32SG", "FS4EP6"]
20161014 10:41:36.086 : TRACE : Arguments: [ '["9A32SG", "FS4EP6"]' | '{"$schema": "http://json-schema.org/draft-04/schema#", "type": "array", "items": {"type": "string"}}' ]
20161014 10:41:36.087 : TRACE : Return: None
Ending test:   Json Schema Validator.Json Schema Validator

@chetand24 Any updates?

@vkosuri this is working for me now, Thanks.


paictl.py service: can't start service after stop, need to try 'stop then start' for several times

How to reproduce:

Stop a PAI service using: paictl.py service stop -n service_name
Start the PAI service using: paictl.py service start -n service_name

It won't work the first time. Take pylon/resource-manager for example, the daemon-set would up, but the pod won't be created.

closed as no further complaints.


feature/135-terms

As the issue explains, a new model called Terms has been added, which represents the terms and conditions. The get should be able to be done by anyone but only admin can create, update or delete them. As an example, I have added a simple first version.
@pabpercab1 told me that it would be a good idea to know which version the user would have signed, so I think of is to add a new attribute to the User model, for example, terms_version_accepted and add a validation that the latest version corresponding to the nearest date has been accepted.
As changing the User model was not part of this task I wanted to leave it here and if you accept the proposal I will make the appropriate changes.
I wait your feedback!

so I think of is to add a new attribute to the User model, for example, terms_version_accepted and add a validation that the latest version corresponding to the nearest date has been accepted. As changing the User model was not part of this task I wanted to leave it here and if you accept the proposal I will make the appropriate changes.
I think this is a good idea! Take into account to update the populate.json as well. Re-request my review when you're done.


File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': 'original-xarray-correlation-aa49928eb1a8c35b11cedd952f043584'}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--41b80a94332ef20c3ed709e58f7500a6', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--8f3d587a87441f5ef9a64fb660fcac10', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--0a86185bfa68c433e1d1106944d7421b', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--2381badfd2c9e4e472a4bc0bcea480af', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('getitem-3dff0c0b1297f41c4c840dab18997453', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--9081f3a009580b2bdbdf45ab29f4a636', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('array-a46f6f15799e48fad8fde929682ebaf2', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--a84c523e18bdf364a6a8c89a63f8001b', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('mean_agg-aggregate-5ba091f58b50c5152b3a23f84a632321', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--03a074e697c65c1dcfa5108783844155', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--66835645657a33f1cee70013a61950ae', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('array-497117a73dfd25c61cca2e435389967a', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--9b575535d46ac4471b0cd6e970213457', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
INFO:distributed.scheduler:Receive client connection: Client-worker-aefb4587-0397-11ef-b3ae-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:37204
INFO:distributed.scheduler:Receive client connection: Client-worker-aefbf145-0397-11ef-b3a9-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:37188
INFO:distributed.scheduler:Receive client connection: Client-worker-aefb4dc0-0397-11ef-b3b5-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:38194
INFO:distributed.scheduler:Receive client connection: Client-worker-aefae76f-0397-11ef-b3b3-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:38186
INFO:distributed.core:Event loop was unresponsive in Nanny for 22.08s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:55346 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-aefb4dc0-0397-11ef-b3b5-0242ac1c000c
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:55338 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-aefb4587-0397-11ef-b3ae-0242ac1c000c
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 767, in run
value = future.result()
File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 262, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 297, in write
raise StreamClosedError()
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 767, in run
value = future.result()
File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 307, in write
convert_stream_closed_error(self, e)
File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346>: Stream is closed
INFO:distributed.core:Event loop was unresponsive in Nanny for 22.02s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 22.02s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 767, in run
value = future.result()
File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 262, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
INFO:distributed.core:Event loop was unresponsive in Nanny for 15.45s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 15.45s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.scheduler:Close client connection: Client-worker-aefae76f-0397-11ef-b3b3-0242ac1c000c
CancelledError                            Traceback (most recent call last)
 in <cell line: 3>()
1 disp_subsett2 = disp_sbas_finish.rio.write_crs("epsg:4326", inplace=False)
2 disp_subsett2.rio.set_spatial_dims('lon', 'lat', inplace=True)
----> 3 disp_subsett2.rio.to_raster(f'disp_sbas.tiff')
12 frames
/usr/local/lib/python3.10/dist-packages/distributed/client.py in _gather()
2231                         else:
2232                             raise exception.with_traceback(traceback)
-> 2233                         raise exc
2234                     if errors == "skip":
2235                         bad_keys.add(key)
CancelledError: ('getitem-b01734d57e58971e18e041a264b58a25', 0, 0)

In case it still does not work for you you need to check your installed Python libraries.

I'm working on colab, should I try this anyway?

No, on Google Colab just check if you use "High RAM" instance.

I am using high RAM, should I turn off this feature?

I turned off the high ram feature but the error still persists.

High RAM is better, no need to disable it. Ok, you can also try to export single band rasters (for one date).

This feature was working very well, what is the reason why it is not working now? Also, how can I save them one by one from xarray?

Maybe changes you made in your notebook or updates to Google Colab's installed libraries could be causing issues with reproducibility. For consistent execution, you might want to check my examples, which are updated in response to changes in Google Colab, or use the PyGMTSAR Docker image. To export a single date raster as a single-band GeoTIFF, use disp_subset[0], disp_subset.isel(date=0), or disp_subset.sel(date=...). And, pay attention to the PyGMTSAR export functions which are well-optimized for most use cases.

Thanks, what is your most current colab notebook?

All PyGMTSAR public Google Colab notebooks are up-to-date.

Alexey, I examined your notebooks, but I could not see the code area where you saved the displacement maps, similar to my saves. How can I do that. Or am I getting an error due to an update to pygmtsar? I will send you access to my colab notebook to your e-mail address, if you deem it appropriate.

I cannot debug and support your own code for free. Use PyGMTSAR export functions as I have mentioned above (see https://github.com/AlexeyPechnikov/pygmtsar/blob/pygmtsar2/pygmtsar/pygmtsar/Stack_export.py) or you need to pay for my work on your special requirements. But what actually you reason to reinvent the available in PyGMTSAR GeoTiFF export function?…

Thank you Alexey, but I'm stuck like this again :(

@AlexeyPechnikov ,I got this error while trying a different notebook, how can I get past it?

It means your wavelength choice does not make sense. The filter size spans thousands of kilometers, even though the full size of a Sentinel-1 scene is much smaller.

 Teşekkür ederim Alexey, ama yine böyle sıkışıp kaldım :(

@AlexeyPechnikov It is very important for me to get past this problem.

The progress indicator is blue, so it is currently calculating. It's possible that yourdisp_sbas_finish variable is defined in a way that requires too much RAM for processing. Be aware that exporting to NetCDF produces a single large file, which can be quite huge. As discussed above, exporting to GeoTIFF generates a set of files and can be much more efficient. You don’t need to switch between functions; stick with the selected one and adjust your code if problems arise. Also, restarting the Dask scheduler can help resolve issues that occurred before the execution of the selected cell.

I tried using the export geotiff you mentioned above to save them in historical tif format one by one, but I still got the same error.

There is no error in your recent screenshot; it is working.

Son ekran görüntüsünde herhangi bir hata yok; çalışıyor.

INFO:distributed.scheduler:Receive client connection: Client-worker-842d6f0e-1103-11ef-b26f-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:39142
INFO:distributed.scheduler:Receive client connection: Client-worker-8630719a-1103-11ef-b272-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:39160
INFO:distributed.scheduler:Receive client connection: Client-worker-8638e296-1103-11ef-b267-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:39170
INFO:distributed.scheduler:Receive client connection: Client-worker-862be432-1103-11ef-b26d-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:39154
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.16s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 16.17s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.17s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.15s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.31s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 15.65s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 31.75s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 31.80s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 31.80s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 31.80s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.64s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
WARNING:distributed.utils_perf:full garbage collections took 19% CPU time recently (threshold: 10%)
INFO:distributed.core:Event loop was unresponsive in Nanny for 32.10s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:39170 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-8638e296-1103-11ef-b267-0242ac1c000c
INFO:distributed.core:Event loop was unresponsive in Nanny for 32.07s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 32.09s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 32.09s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 31.91s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 15.55s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 15.53s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:39154 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-862be432-1103-11ef-b26d-0242ac1c000c
INFO:distributed.core:Event loop was unresponsive in Scheduler for 15.56s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.scheduler:Close client connection: Client-worker-8638e296-1103-11ef-b267-0242ac1c000c
INFO:distributed.core:Event loop was unresponsive in Nanny for 15.56s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:43693 remote=tcp://<IP_ADDRESS>:39154>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 767, in run
value = future.result()
File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 262, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
INFO:distributed.scheduler:Close client connection: Client-worker-862be432-1103-11ef-b26d-0242ac1c000c
INFO:distributed.scheduler:Receive client connection: Client-worker-862be432-1103-11ef-b26d-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:46956
CancelledError                            Traceback (most recent call last)
 in <cell line: 2>()
6     filename = f"result/data_{date}.tif"
7     # Subset'i GeoTIFF olarak kaydetme
----> 8     subset.rio.to_raster(filename, driver='GTiff', crs='EPSG:4326')
9     print(f"{filename} başarıyla kaydedildi.")
17 frames
/usr/local/lib/python3.10/dist-packages/distributed/client.py in _gather()
2231                         else:
2232                             raise exception.with_traceback(traceback)
-> 2233                         raise exc
2234                     if errors == "skip":
2235                         bad_keys.add(key)
CancelledError: ('getitem-08f5ccff885124bb3aed4c18f43f0f97', 48, 0, 0)

@AlexeyPechnikov However, the analysis I made covered a very small area of 1 year. I don't understand why the dimensions are a problem because I have been running this problematic code for 2 months.

There are no errors in the log, and the processing can continue. The 'CancelledError' simply indicates that one of the tasks was cancelled and will be re-executed automatically. The message 'INFO:distributed.core:Event loop was unresponsive in Nanny for 32.10s.' indicates that the processing tasks are large and require substantial RAM. You should check the sizes of your processing grids. Also, subset.rio.to_raster(filename, driver='GTiff', crs='EPSG:4326') is not part of the PyGMTSAR code but your own, and it requires the complete data cube to be materialized, which is impractical for large grids. Use PyGMTSAR functions or your own well-optimized code because straightforward solutions do not work for large datasets. You would check how 'Lake Sarez Landslides, Tajikistan'  and 'Golden Valley, CA.' examples processes large data stacks efficiently.

This is the data I want to save.

As I mentioned above, you start memory-intensive computations with disp_sbas_finish. PyGMTSAR utilizes delayed computation principles, and your variable can be not actual output data but rather a formula to compute them. When you stack a lot of computation on data without materializing them, it only works for small datasets. Try materializing the data on disk first and then export it afterward.

@AlexeyPechnikov Alexey, thank you very much for your help, the notebooks you mentioned worked faster and better. Now I have a different question. I parsed the los data in vertical and east-west directions and wrote it as geotiff. However, it prints the results as pair. My expectation was that it would be like the los outputs on the left in the screenshot, that is, a single .tif file for each date.

To compute the displacements between scenes (dates) you need to apply the least-squares solution:
disp = sbas.lstsq(unwrap.phase - trend - turbo, corr)

It seems you have inconsistency between your disp_sbas_finish defined for interferograms and least-squares processed los results.

You have already done this step in the notebook, do I need to do it again?

Commonly, we use least-squares processing on phase values and convert them to LOS and EW and vertical displacements later. In the notebook the least-squares processing and LOS projection calculating are merged into a single command, you need to split them.

I didn't fully understand. Also I couldn't find the turbo variable in the notebook, where do I pull it from?

Do not apply east-west or vertical projection to your disp_sbas variable because it is already LOS projection. See 'CENTRAL Türkiye Mw 7.8 & 7.5 Earthquakes Co-Seismic Interferogram, 2023.' example for the functions usage. If you don't calculate turbulent atmosphere correction turbo, just omit it.

I opened the notebook you mentioned and changed the initial variables only for my own field, but even though I tried twice, I always get stuck here, what should I do?

Change 300000 meters (300 km) filter size to a reasonable value.

What does this value represent? Will changing it myself to make it equal to 79 negatively affect the results? ValueError: The overlapping depth 507 is larger than your array 79.

Do not apply east-west or vertical projection to your disp_sbas variable because it is already LOS projection. See 'CENTRAL Türkiye Mw 7.8 & 7.5 Earthquakes Co-Seismic Interferogram, 2023.' example for the functions usage. If you don't calculate turbulent atmosphere correction turbo, just omit it.

@AlexeyPechnikov ,Is it right for me to do this?

@AlexeyPechnikov, Do you see any abnormalities in the images?

Do not apply east-west or vertical projection to your disp_sbas variable because it is already LOS projection. See 'CENTRAL Türkiye Mw 7.8 & 7.5 Earthquakes Co-Seismic Interferogram, 2023.' example for the functions usage. If you don't calculate turbulent atmosphere correction turbo, just omit it.

Also, how can I calculate the turbo variable you mentioned here? Can you show me an example line?

Your interferograms look affected by strong atmospheric noise. Try to cleanup (detrend) them. While I don't know your ground truth and it potentially could be valid surface deformation, I'm doubtful.
I think I haven't shared public examples with turbulence correction. You can use Gaussian filtering as in the 'Imperial Valley SBAS Analysis, 2015' notebook.

@AlexeyPechnikov ,Even though I change the wavelength, the expected array size also changes. How can I choose the appropriate wavelength?

For your area with a side of about 10 km, you selected a filter wavelength that is too long. What is the reason? You should compare this filter wavelength and area size with the example in the 'Imperial Valley SBAS Analysis, 2015' notebook. And, by the way, the ValueError on your screenshot has the exact information about the maximum possible wavelength for your array size.


Added email_validator in requirements.txt

On running make the following error occurred:
$ make
git submodule update --recursive --init --rebase
./trackerctl setup bootstrap
Traceback (most recent call last):
  File "./trackerctl", line 5, in <module>
    cli.main()
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/flask/cli.py", line 425, in decorator
    with __ctx.ensure_object(ScriptInfo).load_app().app_context():
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/flask/cli.py", line 381, in load_app
    app = call_factory(self, self.create_app)
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/flask/cli.py", line 113, in call_factory
    return app_factory(*arguments, script_info=script_info)
  File "/home/grijul/Projects/arch-security-tracker/tracker/__init__.py", line 111, in create_app
    from tracker.view.error import error_handlers
  File "/home/grijul/Projects/arch-security-tracker/tracker/view/__init__.py", line 2, in <module>
    from .admin import *
  File "/home/grijul/Projects/arch-security-tracker/tracker/view/admin.py", line 11, in <module>
    from tracker.form.admin import UserForm
  File "/home/grijul/Projects/arch-security-tracker/tracker/form/admin.py", line 26, in <module>
    class UserForm(BaseForm):
  File "/home/grijul/Projects/arch-security-tracker/tracker/form/admin.py", line 28, in UserForm
    email = EmailField(u'E-Mail', validators=[DataRequired(), Length(max=User.EMAIL_LENGTH), Email()])
  File "/home/grijul/Projects/arch-security-tracker/.venv/lib/python3.8/site-packages/wtforms/validators.py", line 332, in __init__
    raise Exception("Install 'email_validator' for email validation support.")
Exception: Install 'email_validator' for email validation support.
make: *** [Makefile:26: setup] Error 1

This PR fixes this issue.

can you check if thats the case for the system packages as well and extend the README.md as well as the travis file?

Added python-email-validator to both travis.yml and README.md files

applied in 9aa4e0eee3f399a0d23a294e77eebfa9ca8052bd
thanks for the contribution

applied in 9aa4e0eee3f399a0d23a294e77eebfa9ca8052bd
thanks for the contribution


Added German Translation

Hi
This is my first Github PR. Sorry for the messup with the ~ files.
Hope to find it useful.
Have a nice week
Manuel

Hmm
I don't know what it wrong here. Anyway as described in: https://github.com/dularion/streama/issues/246 i can not compile a grails3 branch.

yup sure :)
thanks!

Manually merged!
Thank you for the contribution!
One note though, I am not sure I like the word "Tafelansicht", cant think of anything better either at the moment though.

Since these are many small changes, I'd appreciate another PR for this. Thanks for your help on this! Please be mindful to do a fresh fork or pull from the current status of the grails3 branch

Alright. I'm on it!

Love the enthusiasm! :)

Thanks for the help on this! I was planing on refining the translation when i get my grails3 branch to build but that is even better to help together :-)


Support OptInStatus for EC2 describe_region calls

The describe_regions call now includes an OptInStatus key in the return.
Examples:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.describe_regions
https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-regions.html

Added this capability given these constraints:

Regions can have a status of opt-in-not-required, not-opted-in, or opted-in
Regions introduced by AWS after 3/20/19 must be enabled (ie, status of not-opted-in, or opted-in)

Source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html (under "Available Regions" section)
Far as I know, this should be me-south-1 and ap-east-1

Regions before 3/20/19 should have status of opt-in-not-required

I am not exactly sure what China or Gov regions should return, but I set it to opt-in-not-required.

Coverage increased (+0.0006%) to 94.344% when pulling 59c1f0e1313ae362b192dfbc98b698129157a660 on getglad:support_optin_regions into eb1728850349badcb8d85e8d22ab0d534639f6ec on spulec:master.

Hi @getglad, based on this blog post it looks like all new regions will be not-opted-in:
https://aws.amazon.com/blogs/security/setting-permissions-to-enable-accounts-for-upcoming-aws-regions/
This is consistent with my personal account, where the new Capetown region af-south-1 is disabled by default - so I think the table is wrong.
I've created a PR at AWS (https://github.com/awsdocs/amazon-ec2-user-guide/pull/108) to verify this.
Related issue in boto to expose the Opt-In-Status: https://github.com/boto/boto3/issues/2022

@bblommers makes sense. So I guess there are two questions that stem from that:

Should I swap how I've done this, so that there is a static set of opt-in-not-required vs a static set of not-opted-in? More typing for me short term, but long term would eliminate a need to open a PR for every new region. I would imagine the regions will roll in slowly, but still... something I could do here.

Given my implementation, you would never get a opted-in result. Should we consider a method to simulate enabling a region? It's not clear to me if this can actually be done programmatically (https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/manage-account-payment.html#disable-region), so I don't know what the best way to get to that functionality would be. I could see figuring out how we want to at least do opt-in-not-required vs not-opted-in for now, and opening and issue to see if we can get a proposal rolling.

Thoughts?

Hey @getglad, the PR at AWS was just merged, so we're correct that the Capetown should also be opt-in-required.

Yes, I think it makes sense to have a static set of opt-in-not-required, so we don't have to be on top of it.
As far as I know it's not possible to programmatically disable/enable regions, so lets not worry about that now. If/when people need that functionality we can always think of how to expand this

@bblommers done

Looks great, thanks!

@spulec or @bblommers - just wondering when/if there is a plan to tag a new release for pypi so that this would be available?

Hi @getglad, dev releases are done automatically after every PR merge: https://pypi.org/project/moto/#history

@bblommers thanks!


Use team owned public repo

Description
This PR introduces the use of the newly created public ECR registry, to be used for PCM distribution and release.
This PR should not be merged right away, but we can wait till release time to merge it.

Testing
The ECR repo has been manually tested to prove only the owner can push images to the repo.
PR Quality Checklist

[ ] I added tests to new or existing code
[ ] I removed hardcoded strings and used our i18n solution instead (see here)
[ ] I checked that infrastructure/update_infrastructure.sh runs without any error
[ ] I checked that npm run build builds without any error
[ ] I checked that clusters are listed correctly
[ ] I checked that a new cluster can be created (config is produced and dry run passes)
[ ] I checked that login and logout work as expected

In order to increase the likelihood of your contribution being accepted, please make sure you have read both the Contributing Guidelines and the Project Guidelines
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Marked as Draft since it's not to be merged

Have you tested it somehow? Some ideas:

push the current publicly accessible image to the new repo
pull the image locally from the new repo (via one of the scripts or with cli commands)
test the run script locally