internal: specify the example ID within the generated snippet

the merge conflicts are brutal and i can take a stab at them for slowing this down


Create option for scope overwrite exception

Mongoid handles scope overwrites with an exception instead of only logging these overwrites if option is set.
I think, this is very useful while development of an app with many methods and scopes in a model. In most cases it was not intended to overwrite and only happend accidentally. A raised error will be more helpful than a log entry and prevent possible failures or misbehavior.

I like it. Thanks, Christoph. What do you think, @durran?

Looks good to me... Will pull into 2.4

Sounds great!


Print warning when unsigned and signed integer types used for binary operators

Problem description
When we compare unsigned integer and signed integer, as an example, Slang currently casts unsigned integer to signed integer.
But this is opposite to how other languages do such as C/C++, GLSL and HLSL.
In C, when unsigned integer is used with a signed integer, the signed integer is promoted to unsigned integer.
C99 defines the rule as following, called "usual arithmetic conversion":

One type between T1 and T2 is an signed integer type S, the
other type is an unsigned integer type U. Apply the following rules:

If the integer conversion rank of U is greater than or equal to the integer conversion rank of S, C is U.
Otherwise, if S can represent all of the values of U, C is S.
Otherwise, C is the unsigned integer type corresponding to S.
https://en.cppreference.com/w/cpp/language/usual_arithmetic_conversions

We should print a warning when binary operators are used with different sign-ness of integer types.
And we should also follow the "usual arithmetic conversion".
The reason why Slang converts from unsigned to sign is because it follows the "ConversionCost" rule.
But we should apply a different rule for the binary operators.
Following lines show the Conversion costs for the sign-ness cost.
        // Conversions that are lossless, but change "kind"
        kConversionCost_UnsignedToSignedPromotion = 200,

        // Conversion from signed->unsigned integer of same or greater size
        kConversionCost_SignedToUnsignedConversion = 300,

Note that the cost of "uint -> int" is lower than "int -> uint".
The binary operators subject to this issue are following:

operator+
operator-
operator*
operator/
operator%
operator<
operator<=
operator>
operator>=

Goal
We should print a warning when binary operators are used with different sign-ness of integer types.
And we should also follow the "usual arithmetic conversion".
Repro step
Following test case can generate the HLSL/GLSL source code for the operator<(int,uint) case.
//TEST:SIMPLE:-target hlsl -entry PSMain -stage pixel
//TEST:SIMPLE:-target glsl -entry PSMain -stage pixel

StructuredBuffer<int32_t> i32Input;
StructuredBuffer<uint32_t> u32Input;
RWStructuredBuffer<int> outputBuffer;

void PSMain()
{
    bool result = (i32Input[0] < u32Input[1]);
    outputBuffer[0] = int(result);
}

Slang generates HLSL as following,
StructuredBuffer<int > i32Input_0 : register(t0);
StructuredBuffer<uint > u32Input_0 : register(t1);
RWStructuredBuffer<int > outputBuffer_0 : register(u0);

void PSMain()
{
    outputBuffer_0[0U] = int(i32Input_0.Load(0U) < int(u32Input_0.Load(1U)));
}

In order to inspect the DXIL, you can copy the code above and put it in the Shader Playground
When the cast to "int" is removed, DXIL looks different.
  // With cast to int,
  // outputBuffer_0[0U] = int(i32Input_0.Load(0U) < int(u32Input_0.Load(1U)));
  %8 = icmp slt i32 %5, %7

  // Without cast to int,
  // outputBuffer_0[0U] = int(i32Input_0.Load(0U) < u32Input_0.Load(1U));
  %8 = icmp ult i32 %5, %7

Similarly, Slang generates GLSL as following,
#version 450
layout(row_major) uniform;
layout(row_major) buffer;

layout(std430, binding = 0) readonly buffer StructuredBuffer_int_t_0 {
    int _data[];
} i32Input_0;

layout(std430, binding = 1) readonly buffer StructuredBuffer_uint_t_0 {
    uint _data[];
} u32Input_0;

layout(std430, binding = 2) buffer StructuredBuffer_int_t_1 {
    int _data[];
} outputBuffer_0;

void main()
{
    outputBuffer_0._data[0U] = int(i32Input_0._data[0U] < int(u32Input_0._data[1U]));
}

When the cast to "int" is removed, SPIR-V asm looks different.
  // With cast to int,
  // outputBuffer_0._data[0U] = int(i32Input_0._data[0U] < int(u32Input_0._data[1U]));
         %27 = OpLoad %uint %26
         %28 = OpBitcast %int %27
         %30 = OpSLessThan %bool %18 %28

  // Without cast to int
  // outputBuffer_0._data[0U] = int(i32Input_0._data[0U] < u32Input_0._data[1U]);
         %28 = OpLoad %uint %27
         %30 = OpULessThan %bool %20 %28

Note
I suspected that the ConversionCost might be set incorrectly.
But it doesn't seem that is the case.
When there are overloading with "int" and 'uint", DXC prefers "int" over "uint".
The following example shows the different DXIL between two cases where the first case defines only "USE_INT" and another case defines only "USE_UINT".
When both "USE_INT" and "USE_UINT" are defined, the resulting DXIL was same to the case that defined only "USE_INT".
StructuredBuffer<int > i32Input : register(t0);
StructuredBuffer<uint32_t > u32Input : register(t1);
RWStructuredBuffer<int > outputBuffer : register(u0);

#define USE_INT
#define USE_UINT

#ifdef USE_INT
bool testOverload(int lhs, int rhs)
{
    return lhs < rhs;
}
#endif

#ifdef USE_UINT
bool testOverload(uint32_t lhs, uint32_t rhs)
{
    return lhs < rhs;
}
#endif

void PSMain()
{
    outputBuffer[0U] = int(testOverload(i32Input.Load(0U), u32Input.Load(1U)));
    return;
}

Removing from Q3. Setting P2.

Closing because #4102 is closed for the same reason.


Move from gulp to fly

https://github.com/flyjs/fly
Mentioned by @rauchg here: https://github.com/zeit/next.js/pull/435

I am down with anything that works. But I personally prefer simply to use shell scripts (may be with shelljs) along with NPM scripts.

@arunoda As do I. IMO, everything a repo can do (re: builds, test) should be easily described by an npm script...and its definition shouldn't extend beyond the viewport. Pointless at that point.
I'm finishing up Fly 2.0 right now, and initial benchmarks show that it's 3 to 8x faster than Gulp. And, its flyfile.js syntax is much simpler & cleaner than Gulp's -- we're chaining methods (extremely minimal wrappers around any API) directly instead of asking you to write 1,001 pipe()s on the page. 😆


ZS-189, removed MicrogridsResponseMessageFinder and updated Messaging…

…Config

All is well.

All is well.

All is well.

All is well.

All is well.


Json path functionality

Coverage remained the same at 100.0% when pulling f3dc673cb27261eb9376328c713d3234a14fd476 on JSONPath-Functionality into 88e32ee4189e44b73c53a43dc67b2ba207137f12 on master.


Debug Addin: New Method and Documentation

Feature Request 🛍

Due to an update of the  Edge Chromium, the current debug method for add-in is facing issues. Basically, the add-in reloads after 5 seconds if the debugger is attached to it. A new debug method can be used for such cases.
Use Case

Debugging is an important part of development so the debugger should work.
Possible Solution

Using this we can implement a solution.
Context

Whenever an update occurs, there might occur similar changes which lead to breaking the working of debugging the add-in. In such cases, one can opt the other method to debug and continue the work.
Detailed Description

A launch.json file would need to be created along with some documentation changes for the same describing its working.
Would you like to work on the issue??
Yes

If you and @Aniruddha-Shriwant are facing this issue, feel free to work on this. I am not facing it yet. That could be probably because I haven't updated my edge.

@algomaster99
I have created a PR. Let me know if there are any changes required.


Add facter

Please rename commit's name following our guide from CONTRIBUTING.md

thanks for looking into this @jagajaga. i've just rebased the changes.

But you forgot to change the commit's name :)

Commit message updated. Thanks @jagajaga

Travis built, thanks.


i18n(tr): add css-and-tailwind

What kind of changes does this PR include?
Updates of Starlight docs site content
Description
This pull request adds translated in Turkish css-and-tailwind.mdx file into project.

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

tr
guides/css-and-tailwind.mdx
Localization added, will be marked as complete.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.


Fix: Creatability of page types incorrectly affecting the can_reorder_subpages() check

Fixes https://github.com/wagtail/wagtail/issues/3465
Where all of the page types in subpage_types have is_creatable=False on them, it is currently not possible to reorder pages that exist below instances of that page type.
I'm classifying this as a bug, because I have always interpreted the is_creatable=False mechanism as a way to prevent manual creation ONLY. Not only do I feel it is perfectly valid for programatically-created pages to exist at certain locations (and remain editable/publishable/deletable). But, projects also have a life outside of the current state of the codebase... it's quite possible for pages to have been created before the current code state (where is_creatable or subpage_types have changed); In which case, the ability to manage those pages (including 'reordering') should not be impacted.

Hi @ababic, are you still keen on picking this up? No worries if not, I'll try to get to it soon. Thanks!

@laymonage I should have some headspace for this next week, unless you get to it sooner?

@ababic I likely won't get to this until the week after, so if you have time next week that'd be great!

New tests aren't behaving quite as expected

The tests are hitting the 'set_page_position' URL, which reaches the set_page_position view
The only thing that could cause those views to redirect is the raising of a PermissionDenied error, which could only be raised by this line
The tests are run as a superuser, so permissions shouldn't be a problem in the updated PagePermissionTester.can_reorder_children() method

Confused :(

Superseded by #12156.


Add Microsoft Store REST Source as default option and fix telemetry gaps

Add Microsoft Store REST Source as default option
Fix gaps found in telemetry from data validation

[ ] Have you signed the Contributor License Agreement?
[ ] Are you working against an Issue?

Microsoft Reviewers: Open in CodeFlow

Probably might as well pull in the com interface changes as well in this checkin.

Refers to: src/AppInstallerRepositoryCore/RepositorySource.cpp:1098 in 5e6c3af. [](commit_id = 5e6c3afb0962446db1e9ac0ef476150a6f8aab77, deletion_comment = False)

Add Microsoft Store REST Source as default option Fix gaps found in telemetry from data validation

[ ]  Have you signed the Contributor License Agreement?
[ ]  Are you working against an Issue?

Microsoft Reviewers: Open in CodeFlow


Zebra stable release binaries should be linked from ZFND website

Is your feature request related to a problem? Please describe.
Whenever we have a Zebra stable release ready, some people will expect to find it from the main ZFND website.
Describe the solution you'd like
We should, at the very least:

[x] link to the Zebra dockerfile on docker hub.

We might additionally wish to:

[ ] Link to the Arch Linux package in the AUR
[ ] Create and maintain a Debian package
[x] Link to our github repo and build instructions

Describe alternatives you've considered
Do nothing, people will use a search engine to find Zebra and/or will go directly to our github repository
Additional context
Although this did not come up as an issue during the user studies (#1501) for our first alpha release, we do know that some people prefer to click through rather than search so we would like to provide them with this option. For reference this is what is provided from z.cash/download/:

Debian package install instructions (the ECC maintains a package repo for 64-bit Debian-based distributions)
Binary tarball
Link to the source code, build instructions and signing keys
A link to their dockerfile on docker hub

@winniendela when we have a stable release ready, we'll probably want to have a section in the new ZFND website where users can download the zebra binary (if we package one up instead of asking users to build from source) cc: @ZcashFoundation/zebra-team

Not strictly necessary for release candidate

I'm working on this one with Elise

This is done for now. Follow up tasks to add a command for running from crates.io and Debian and Arch Linux packages can be done by myself and Elise outside of github.


On job submission, record job_name & job_id in recipe_run.message

Copied from https://github.com/pangeo-forge/staged-recipes/pull/194#issuecomment-1261487532:

@andersy005, I was thinking about how to explain to you how to determine which compute job on Dataflow actually failed here, because if we know which job failed, then we can get logs, and try to diagnose the problem.
Here are the backend server logs from immediately after you called /run in https://github.com/pangeo-forge/staged-recipes/pull/194#issuecomment-1261431878:

2022-09-28T20:30:11.667158+00:00 app[web.1]: 2022-09-28 20:30:11,667 DEBUG - orchestrator - Running command: ['pangeo-forge-runner', 'bake', '--repo=https://github.com/yloskvi4/staged-recipes', '--ref=9cdf5ef31a5a321705eee3aa91c881a47ca60c88', '--json', '--prune', '--Bake.recipe_id=casm-soil-moisture', '-f=/tmp/tmp14ejr6q9.json', '--feedstock-subdir=recipes/casm']
2022-09-28T20:30:18.562676+00:00 heroku[web.1]: Process running mem=694M(128.1%)
2022-09-28T20:30:18.567781+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2022-09-28T20:30:27.651843+00:00 app[web.1]: 2022-09-28 20:30:27,651 DEBUG - orchestrator - Command output is {"message": "Config option `public_url` not recognized by `TargetStorage`."}
2022-09-28T20:30:27.651851+00:00 app[web.1]: {"message": "Config option `public_url` not recognized by `InputCacheStorage`."}
2022-09-28T20:30:27.651851+00:00 app[web.1]: {"message": "Config option `public_url` not recognized by `MetadataCacheStorage`."}
2022-09-28T20:30:27.651864+00:00 app[web.1]: {"message": "Target Storage is FSSpecTarget(S3FileSystem(client_kwargs=<dict>, default_cache_type=<str>, default_fill_cache=<bool>, use_listings_cache=<bool>, key=<str>, secret=<str>, root_path=\"Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1162/casm-soil-moisture.zarr\")\n", "status": "setup"}
2022-09-28T20:30:27.651865+00:00 app[web.1]: {"message": "Input Cache Storage is CacheFSSpecTarget(GCSFileSystem(bucket=<str>, root_path=\"pangeo-forge-prod-cache\")\n", "status": "setup"}
2022-09-28T20:30:27.651865+00:00 app[web.1]: {"message": "Metadata Cache Storage is MetadataTarget(GCSFileSystem(, root_path=\"pangeo-forge-prod-cache/metadata/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1162/casm-soil-moisture.zarr\")\n", "status": "setup"}
2022-09-28T20:30:27.651866+00:00 app[web.1]: {"message": "Picked Git content provider.\n", "status": "fetching"}
2022-09-28T20:30:27.651866+00:00 app[web.1]: {"message": "Cloning into '/tmp/tmpqnfbznkd'...\n", "status": "fetching"}
2022-09-28T20:30:27.651867+00:00 app[web.1]: {"message": "HEAD is now at 9cdf5ef [pre-commit.ci] auto fixes from pre-commit.com hooks\n", "status": "fetching"}
2022-09-28T20:30:27.651867+00:00 app[web.1]: {"message": "Parsing recipes...", "status": "running"}
2022-09-28T20:30:27.651868+00:00 app[web.1]: {"message": "Baking only recipe_id='casm-soil-moisture'"}
2022-09-28T20:30:27.651868+00:00 app[web.1]: {"message": "Using Any for unsupported type: typing.Callable[[typing.Any], NoneType]"}
2022-09-28T20:30:27.651868+00:00 app[web.1]: {"message": "Using Any for unsupported type: typing.Callable[[typing.Any], NoneType]"}
2022-09-28T20:30:27.651868+00:00 app[web.1]: {"message": "Downloading source distribution of the SDK from PyPi"}
2022-09-28T20:30:27.651874+00:00 app[web.1]: {"message": "Executing command: ['/usr/bin/python3.9', '-m', 'pip', 'download', '--dest', '/tmp/tmpmoukbms8', 'apache-beam==2.41.0', '--no-deps', '--no-binary', ':all:']"}
2022-09-28T20:30:27.651874+00:00 app[web.1]: {"message": "Staging SDK sources from PyPI: dataflow_python_sdk.tar"}
2022-09-28T20:30:27.651875+00:00 app[web.1]: {"message": "Downloading binary distribution of the SDK from PyPi"}
2022-09-28T20:30:27.651875+00:00 app[web.1]: {"message": "Executing command: ['/usr/bin/python3.9', '-m', 'pip', 'download', '--dest', '/tmp/tmpmoukbms8', 'apache-beam==2.41.0', '--no-deps', '--only-binary', ':all:', '--python-version', '39', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux1_x86_64']"}
2022-09-28T20:30:27.651875+00:00 app[web.1]: {"message": "Staging binary distribution of the SDK from PyPI: apache_beam-2.41.0-cp39-cp39-manylinux1_x86_64.whl"}
2022-09-28T20:30:27.651876+00:00 app[web.1]: {"message": "Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild"}
2022-09-28T20:30:27.651876+00:00 app[web.1]: {"message": "Default Python SDK image for environment is apache/beam_python3.9_sdk:2.41.0"}
2022-09-28T20:30:27.651876+00:00 app[web.1]: {"message": "Using provided Python SDK container image: pangeo/forge:8a862dc"}
2022-09-28T20:30:27.651877+00:00 app[web.1]: {"message": "Python SDK container image set to \"pangeo/forge:8a862dc\" for Docker environment"}
2022-09-28T20:30:27.651877+00:00 app[web.1]: {"message": "==================== <function pack_combiners at 0x7f9239c3faf0> ===================="}
2022-09-28T20:30:27.651877+00:00 app[web.1]: {"message": "==================== <function sort_stages at 0x7f9239bc0310> ===================="}
2022-09-28T20:30:27.651877+00:00 app[web.1]: {"message": "Defaulting to the temp_location as staging_location: gs://pangeo-forge-prod-dataflow/temp"}
2022-09-28T20:30:27.651878+00:00 app[web.1]: {"message": "Setting socket default timeout to 60 seconds."}
2022-09-28T20:30:27.651878+00:00 app[web.1]: {"message": "socket default timeout is 60.0 seconds."}
2022-09-28T20:30:27.651878+00:00 app[web.1]: {"message": "Starting GCS upload to gs://pangeo-forge-prod-dataflow/temp/a6170692e70616e67656f2d666f7267652e6f7267251162.1664397023.078565/dataflow_python_sdk.tar..."}
2022-09-28T20:30:27.651879+00:00 app[web.1]: {"message": "Completed GCS upload to gs://pangeo-forge-prod-dataflow/temp/a6170692e70616e67656f2d666f7267652e6f7267251162.1664397023.078565/dataflow_python_sdk.tar in 0 seconds."}
2022-09-28T20:30:27.651879+00:00 app[web.1]: {"message": "Starting GCS upload to gs://pangeo-forge-prod-dataflow/temp/a6170692e70616e67656f2d666f7267652e6f7267251162.1664397023.078565/apache_beam-2.41.0-cp39-cp39-manylinux1_x86_64.whl..."}
2022-09-28T20:30:27.651880+00:00 app[web.1]: {"message": "Completed GCS upload to gs://pangeo-forge-prod-dataflow/temp/a6170692e70616e67656f2d666f7267652e6f7267251162.1664397023.078565/apache_beam-2.41.0-cp39-cp39-manylinux1_x86_64.whl in 1 seconds."}
2022-09-28T20:30:27.651885+00:00 app[web.1]: {"message": "Starting GCS upload to gs://pangeo-forge-prod-dataflow/temp/a6170692e70616e67656f2d666f7267652e6f7267251162.1664397023.078565/pipeline.pb..."}
2022-09-28T20:30:27.651886+00:00 app[web.1]: {"message": "Completed GCS upload to gs://pangeo-forge-prod-dataflow/temp/a6170692e70616e67656f2d666f7267652e6f7267251162.1664397023.078565/pipeline.pb in 0 seconds."}
2022-09-28T20:30:27.651886+00:00 app[web.1]: {"message": "Create job: <Job\n clientRequestId: '20220928203023079305-9232'\n createTime: '2022-09-28T20:30:26.472569Z'\n currentStateTime: '1970-01-01T00:00:00Z'\n id: '2022-09-28_13_30_25-9666562633575851936'\n location: 'us-central1'\n name: 'a6170692e70616e67656f2d666f7267652e6f7267251162'\n projectId: 'pangeo-forge-4967'\n stageStates: []\n startTime: '2022-09-28T20:30:26.472569Z'\n steps: []\n tempFiles: []\n type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>"}
2022-09-28T20:30:27.651887+00:00 app[web.1]: {"message": "Created job with id: [2022-09-28_13_30_25-9666562633575851936]"}
2022-09-28T20:30:27.651887+00:00 app[web.1]: {"message": "Submitted job: 2022-09-28_13_30_25-9666562633575851936"}
2022-09-28T20:30:27.651887+00:00 app[web.1]: {"message": "To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-09-28_13_30_25-9666562633575851936?project=pangeo-forge-4967"}
2022-09-28T20:30:27.651887+00:00 app[web.1]: {"message": "Submitted job 2022-09-28_13_30_25-9666562633575851936 for recipe casm-soil-moisture", "recipe": "casm-soil-moisture", "job_name": "a6170692e70616e67656f2d666f7267652e6f7267251162", "job_id": "2022-09-28_13_30_25-9666562633575851936", "status": "submitted"}

And, the most important last line:
2022-09-28T20:30:27.651887+00:00 app[web.1]: {"message": "Submitted job 2022-09-28_13_30_25-9666562633575851936 for recipe casm-soil-moisture", "recipe": "casm-soil-moisture", "job_name": "a6170692e70616e67656f2d666f7267652e6f7267251162", "job_id": "2022-09-28_13_30_25-9666562633575851936", "status": "submitted"}

This gives us two identifiers we can use to find the job on Dataflow:
"job_name": "a6170692e70616e67656f2d666f7267652e6f7267251162"
"job_id": "2022-09-28_13_30_25-9666562633575851936"

I'll think bit more about the most efficient way to actually retrieve logs based on these identifiers, but first, a suggestion for how we could make these values easier to recover. We have access to this log line in the backend application code, here:

https://github.com/pangeo-forge/pangeo-forge-orchestrator/blob/4b008d2eba4602dcc2681e7c538738c6b5ad79a9/pangeo_forge_orchestrator/routers/github_app.py#L621

Right after that line, we could do something like this:
out = subprocess.check_output(cmd)
# new code from this point on
for line in out.splitlines():
    p = json.loads(line)
    if p["status"] == "submitted":
        recipe_run.message = json.dumps(dict(job_name=p["job_name"], job_id=p["job_id"]))
        db_session.add(recipe_run)
        db_session.commit()

Then, we would have these values directly available to us in https://pangeo-forge.org/dashboard/recipe-run/1162?feedstock_id=1, without going to the server logs.

I wrote https://github.com/yuvipanda/mybinder-analytics/blob/main/logs.py to which you can give a dataflow job id and it'll show you fun logs! You can even filter based on type, etc at source.

I wrote it to be async so we can fairly easily lift it into the orchestrator code as an endpoint if necessary.

Should be slightly wary of secrets leaking though :)

@andersy005, using the job_id above, you can fetch error logs as follows:
$ gcloud logging read 'resource.type="dataflow_step" AND resource.labels.job_id="2022-09-28_13_22_15-5900460635961685595" AND logName=("projects/pangeo-forge-4967/logs/dataflow.googleapis.com%2Fjob-message" OR "projects/pangeo-forge-4967/logs/dataflow.googleapis.com%2Flauncher") AND severity="ERROR"' --limit=1

This gives a really long trace!

---
insertId: fjjmtqc4qj
labels:
  dataflow.googleapis.com/job_id: 2022-09-28_13_22_15-5900460635961685595
  dataflow.googleapis.com/job_name: a6170692e70616e67656f2d666f7267652e6f7267251167
  dataflow.googleapis.com/log_type: system
  dataflow.googleapis.com/region: us-central1
logName: projects/pangeo-forge-4967/logs/dataflow.googleapis.com%2Fjob-message
receiveTimestamp: '2022-09-28T20:27:04.722067318Z'
resource:
  labels:
    job_id: 2022-09-28_13_22_15-5900460635961685595
    job_name: a6170692e70616e67656f2d666f7267652e6f7267251167
    project_id: '13667658525'
    region: us-central1
    step_id: ''
  type: dataflow_step
severity: ERROR
textPayload: |
  Workflow failed. Causes: S13:Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/Reshuffle_000/ReshufflePerKey/GroupByKey/Read+Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/Reshuffle_000/ReshufflePerKey/GroupByKey/GroupByWindow+Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/Reshuffle_000/ReshufflePerKey/FlatMap(restore_timestamps)+Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/Reshuffle_000/RemoveRandomKeys+Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/prepare_target+Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/Reshuffle_001/AddRandomKeys+Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/Reshuffle_001/ReshufflePerKey/Map(reify_timestamps)+Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/Reshuffle_001/ReshufflePerKey/GroupByKey/Reify+Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/Reshuffle_001/ReshufflePerKey/GroupByKey/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers:

    a6170692e70616e67656f2d66-09281322-kh7r-harness-qzf5
        Root cause: Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 110, in _error_wrapper
      return await func(*args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiobotocore/client.py", line 265, in _make_api_call
      raise error_class(parsed_response, operation_name)
  botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/mapping.py", line 137, in __getitem__
      result = self.fs.cat(k)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 86, in wrapper
      return sync(self.loop, func, *args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 66, in sync
      raise return_result
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 26, in _runner
      result[0] = await coro
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 398, in _cat
      raise ex
    File "/srv/conda/envs/notebook/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
      return await fut
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 955, in _cat_file
      return await _error_wrapper(_call_and_read, retries=self.retries)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 137, in _error_wrapper
      raise err
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 110, in _error_wrapper
      return await func(*args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 942, in _call_and_read
      resp = await self._call_s3(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 332, in _call_s3
      return await _error_wrapper(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 137, in _error_wrapper
      raise err
  FileNotFoundError: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 384, in open_group
      zarr_group = zarr.open_consolidated(store, **open_kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/convenience.py", line 1304, in open_consolidated
      meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/storage.py", line 2853, in __init__
      meta = json_loads(self.store[metadata_key])
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/storage.py", line 717, in __getitem__
      return self._mutable_mapping[key]
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/mapping.py", line 141, in __getitem__
      raise KeyError(key)
  KeyError: '.zmetadata'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 502, in prepare_target
      ds = open_target(config.storage_config.target)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 115, in open_target
      return xr.open_zarr(target.get_mapper())
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 789, in open_zarr
      ds = open_dataset(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py", line 531, in open_dataset
      backend_ds = backend.open_dataset(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 837, in open_dataset
      store = ZarrStore.open_group(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 401, in open_group
      zarr_group = zarr.open_group(store, **open_kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/hierarchy.py", line 1347, in open_group
      raise GroupNotFoundError(path)
  zarr.errors.GroupNotFoundError: group not found at path ''

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
    File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1877, in <lambda>
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/executors/beam.py", line 14, in _no_arg_stage
      fun(config=config)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 524, in prepare_target
      with open_chunk(chunk_key, config=config) as ds:
    File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
      return next(self.gen)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
      ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
      response = task()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
      lambda: self.create_worker().do_instruction(request), request)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
      return getattr(self, request_type)(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
      bundle_processor.process_bundle(instruction_id))
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
      input_op_by_transform_id[element.transform_id].process_encoded(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
      self.output(decoded_value)
    File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
    File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1877, in <lambda>
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/executors/beam.py", line 14, in _no_arg_stage
      fun(config=config)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 524, in prepare_target
      with open_chunk(chunk_key, config=config) as ds:
    File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
      return next(self.gen)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
      ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/prepare_target-ptransform-56']
  ,

    a6170692e70616e67656f2d66-09281322-kh7r-harness-qzf5
        Root cause: Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 110, in _error_wrapper
      return await func(*args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiobotocore/client.py", line 265, in _make_api_call
      raise error_class(parsed_response, operation_name)
  botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/mapping.py", line 137, in __getitem__
      result = self.fs.cat(k)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 86, in wrapper
      return sync(self.loop, func, *args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 66, in sync
      raise return_result
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 26, in _runner
      result[0] = await coro
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 398, in _cat
      raise ex
    File "/srv/conda/envs/notebook/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
      return await fut
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 955, in _cat_file
      return await _error_wrapper(_call_and_read, retries=self.retries)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 137, in _error_wrapper
      raise err
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 110, in _error_wrapper
      return await func(*args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 942, in _call_and_read
      resp = await self._call_s3(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 332, in _call_s3
      return await _error_wrapper(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 137, in _error_wrapper
      raise err
  FileNotFoundError: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 384, in open_group
      zarr_group = zarr.open_consolidated(store, **open_kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/convenience.py", line 1304, in open_consolidated
      meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/storage.py", line 2853, in __init__
      meta = json_loads(self.store[metadata_key])
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/storage.py", line 717, in __getitem__
      return self._mutable_mapping[key]
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/mapping.py", line 141, in __getitem__
      raise KeyError(key)
  KeyError: '.zmetadata'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 502, in prepare_target
      ds = open_target(config.storage_config.target)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 115, in open_target
      return xr.open_zarr(target.get_mapper())
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 789, in open_zarr
      ds = open_dataset(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py", line 531, in open_dataset
      backend_ds = backend.open_dataset(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 837, in open_dataset
      store = ZarrStore.open_group(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 401, in open_group
      zarr_group = zarr.open_group(store, **open_kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/hierarchy.py", line 1347, in open_group
      raise GroupNotFoundError(path)
  zarr.errors.GroupNotFoundError: group not found at path ''

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
    File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1877, in <lambda>
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/executors/beam.py", line 14, in _no_arg_stage
      fun(config=config)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 524, in prepare_target
      with open_chunk(chunk_key, config=config) as ds:
    File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
      return next(self.gen)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
      ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
      response = task()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
      lambda: self.create_worker().do_instruction(request), request)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
      return getattr(self, request_type)(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
      bundle_processor.process_bundle(instruction_id))
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
      input_op_by_transform_id[element.transform_id].process_encoded(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
      self.output(decoded_value)
    File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
    File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1877, in <lambda>
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/executors/beam.py", line 14, in _no_arg_stage
      fun(config=config)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 524, in prepare_target
      with open_chunk(chunk_key, config=config) as ds:
    File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
      return next(self.gen)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
      ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/prepare_target-ptransform-56']
  ,

    a6170692e70616e67656f2d66-09281322-kh7r-harness-qzf5
        Root cause: Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 110, in _error_wrapper
      return await func(*args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiobotocore/client.py", line 265, in _make_api_call
      raise error_class(parsed_response, operation_name)
  botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/mapping.py", line 137, in __getitem__
      result = self.fs.cat(k)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 86, in wrapper
      return sync(self.loop, func, *args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 66, in sync
      raise return_result
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 26, in _runner
      result[0] = await coro
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 398, in _cat
      raise ex
    File "/srv/conda/envs/notebook/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
      return await fut
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 955, in _cat_file
      return await _error_wrapper(_call_and_read, retries=self.retries)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 137, in _error_wrapper
      raise err
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 110, in _error_wrapper
      return await func(*args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 942, in _call_and_read
      resp = await self._call_s3(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 332, in _call_s3
      return await _error_wrapper(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 137, in _error_wrapper
      raise err
  FileNotFoundError: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 384, in open_group
      zarr_group = zarr.open_consolidated(store, **open_kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/convenience.py", line 1304, in open_consolidated
      meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/storage.py", line 2853, in __init__
      meta = json_loads(self.store[metadata_key])
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/storage.py", line 717, in __getitem__
      return self._mutable_mapping[key]
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/mapping.py", line 141, in __getitem__
      raise KeyError(key)
  KeyError: '.zmetadata'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 502, in prepare_target
      ds = open_target(config.storage_config.target)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 115, in open_target
      return xr.open_zarr(target.get_mapper())
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 789, in open_zarr
      ds = open_dataset(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py", line 531, in open_dataset
      backend_ds = backend.open_dataset(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 837, in open_dataset
      store = ZarrStore.open_group(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 401, in open_group
      zarr_group = zarr.open_group(store, **open_kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/hierarchy.py", line 1347, in open_group
      raise GroupNotFoundError(path)
  zarr.errors.GroupNotFoundError: group not found at path ''

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
    File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1877, in <lambda>
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/executors/beam.py", line 14, in _no_arg_stage
      fun(config=config)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 524, in prepare_target
      with open_chunk(chunk_key, config=config) as ds:
    File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
      return next(self.gen)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
      ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
      response = task()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
      lambda: self.create_worker().do_instruction(request), request)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
      return getattr(self, request_type)(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
      bundle_processor.process_bundle(instruction_id))
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
      input_op_by_transform_id[element.transform_id].process_encoded(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
      self.output(decoded_value)
    File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
    File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1877, in <lambda>
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/executors/beam.py", line 14, in _no_arg_stage
      fun(config=config)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 524, in prepare_target
      with open_chunk(chunk_key, config=config) as ds:
    File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
      return next(self.gen)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
      ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/prepare_target-ptransform-56']
  ,

    a6170692e70616e67656f2d66-09281322-kh7r-harness-qzf5
        Root cause: Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 110, in _error_wrapper
      return await func(*args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiobotocore/client.py", line 265, in _make_api_call
      raise error_class(parsed_response, operation_name)
  botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/mapping.py", line 137, in __getitem__
      result = self.fs.cat(k)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 86, in wrapper
      return sync(self.loop, func, *args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 66, in sync
      raise return_result
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 26, in _runner
      result[0] = await coro
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 398, in _cat
      raise ex
    File "/srv/conda/envs/notebook/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
      return await fut
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 955, in _cat_file
      return await _error_wrapper(_call_and_read, retries=self.retries)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 137, in _error_wrapper
      raise err
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 110, in _error_wrapper
      return await func(*args, **kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 942, in _call_and_read
      resp = await self._call_s3(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 332, in _call_s3
      return await _error_wrapper(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/s3fs/core.py", line 137, in _error_wrapper
      raise err
  FileNotFoundError: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 384, in open_group
      zarr_group = zarr.open_consolidated(store, **open_kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/convenience.py", line 1304, in open_consolidated
      meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/storage.py", line 2853, in __init__
      meta = json_loads(self.store[metadata_key])
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/storage.py", line 717, in __getitem__
      return self._mutable_mapping[key]
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/mapping.py", line 141, in __getitem__
      raise KeyError(key)
  KeyError: '.zmetadata'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 502, in prepare_target
      ds = open_target(config.storage_config.target)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 115, in open_target
      return xr.open_zarr(target.get_mapper())
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 789, in open_zarr
      ds = open_dataset(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py", line 531, in open_dataset
      backend_ds = backend.open_dataset(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 837, in open_dataset
      store = ZarrStore.open_group(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py", line 401, in open_group
      zarr_group = zarr.open_group(store, **open_kwargs)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/zarr/hierarchy.py", line 1347, in open_group
      raise GroupNotFoundError(path)
  zarr.errors.GroupNotFoundError: group not found at path ''

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
    File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1877, in <lambda>
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/executors/beam.py", line 14, in _no_arg_stage
      fun(config=config)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 524, in prepare_target
      with open_chunk(chunk_key, config=config) as ds:
    File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
      return next(self.gen)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
      ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 284, in _execute
      response = task()
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 357, in <lambda>
      lambda: self.create_worker().do_instruction(request), request)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 597, in do_instruction
      return getattr(self, request_type)(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
      bundle_processor.process_bundle(instruction_id))
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
      input_op_by_transform_id[element.transform_id].process_encoded(
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
      self.output(decoded_value)
    File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
    File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
    File "apache_beam/runners/common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
    File "apache_beam/runners/common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
    File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
    File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
    File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
    File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
    File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
    File "apache_beam/runners/common.py", line 983, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    File "/usr/local/lib/python3.9/dist-packages/apache_beam/transforms/core.py", line 1877, in <lambda>
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/executors/beam.py", line 14, in _no_arg_stage
      fun(config=config)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 524, in prepare_target
      with open_chunk(chunk_key, config=config) as ds:
    File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
      return next(self.gen)
    File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
      ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/prepare_target-ptransform-56']
timestamp: '2022-09-28T20:27:03.809417850Z'

TL;DR
File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 421, in open_chunk
ds = config.process_chunk(ds)
    File "<string>", line 20, in postproc
  NameError: name 'np' is not defined [while running 'Start|cache_input|Reshuffle_000|prepare_target|Reshuffle_001|store_chunk|Reshuffle_002|finalize_target|Reshuffle_003/prepare_target-ptransform-56']

I've confirmed that this works with the creds stored in secrets/dataflow-job-submission.json, so to replicate, any Pangeo Forge Orchestrator admin can:
$ sops -d -i secrets/dataflow-job-submission.json
$ gcloud auth activate-service-account --key-file=`pwd`/secrets/dataflow-job-submission.json
$ gcloud logging read 'resource.type="dataflow_step" AND resource.labels.job_id="2022-09-28_13_22_15-5900460635961685595" AND logName=("projects/pangeo-forge-4967/logs/dataflow.googleapis.com%2Fjob-message" OR "projects/pangeo-forge-4967/logs/dataflow.googleapis.com%2Flauncher") AND severity="ERROR"' --limit=1

https://github.com/pangeo-forge/pangeo-forge-orchestrator/issues/145#issuecomment-1261535709

@yuvipanda, oh cool! Didn't realize this.

I wrote it to be async so we can fairly easily lift it into the orchestrator code as an endpoint if necessary.

I think we maybe want to add a gcloud logging call to DataflowBakery.get_logs instead? And then wrap that in a subprocess call here, rather than going to Dataflow directly from orchestrator? https://github.com/pangeo-forge/pangeo-forge-runner/issues/20

@cisaacstern putting this in the location you suggested makes sense!
Folks can already use this to debug though!

Folks can already use this to debug though!

How would one use this to debug, and is it easier than calling gcloud logging directly, as described in https://github.com/pangeo-forge/pangeo-forge-orchestrator/issues/145#issuecomment-1261544227 ?
If so, should it be including in the recommendations for https://github.com/pangeo-forge/pangeo-forge-orchestrator/issues/146?

@cisaacstern the additional features it logs.py offers over just doing gcloud directly is:

The --follow flag, that you can specify and it will show you new logs as they arrive! gcloud doesn't do that
It'll show you logs from all components - dataflow (reshuffler, harnass, etc) as well as the individual nodes (kubelet, docker, etc).
You can filter to see what exactly is you wanna see as well so you don't get overwhelmed.

I think the most useful thing it has is --follow though, as otherwise you have to run gcloud logging read many times over. However, I don't really have too many strong opinions on this, other than thinking that --follow is neat :D

Noted! https://github.com/pangeo-forge/pangeo-forge-orchestrator/issues/146#issuecomment-1261610539


CC2 - Optimizing lambda equations

Description
The current implementation of the lambda equations for CC2 still contains some N6-terms.
Todos
Notable points that this PR has either accomplished or will accomplish.

[x] Rewrite the code to calculate the CC2 lambda amplitudes in order to avoid N6-terms.
[x] Add H2 as a test case

Status

[x] Ready to go

Codecov Report

Merging #29 (1af74f6) into main (6bcac2e) will increase coverage by 0.10%.
The diff coverage is 100.00%.

The N6-terms are located in the H[o,o,v,v] and H[o,v,o,v] intermediates. I removed these terms from the intermediates in the case of CC2 and placed them directly in the loop where the λ1-amplitudes are calculated. By doing this, it is possible to contract t2 with λ1 first which avoids doing the  N6 contractions.


add pep pointer tool

This is @pravs3683's PepPointer tool to classify novel peptides based on their genomic coordinates.

We're using PepPointer in a publication, so it would be useful to have it released (even if we determine later on that it duplicates some BEDtools functionality)


FATAL ERROR: OUT OF MEMORY

Hello
I am trying to debug a python load test package called Locust.
It creates a ROUTER and several DEALERs. The ROUTER often crashes due to this assert: warning : FATAL ERROR: OUT OF MEMORY (C:\projects\libzmq\src\decoder_allocators.cpp:85)
It is setup to use zmq.green.
This is running on a windows machine using localhost.
pyzmq version 22.1.0
I doubt this is something wrong with pyzmq. What sort of misuse could cause such an error to be thrown?
Thanks

I have the same issue with Locust and version 22.1.0 of pyzmq.  I am new to using it so I don't have any idea if other versions have worked.
python 3.9.5
locust 1.5.3
pyzmq 22.1.0
FATAL ERROR: OUT OF MEMORY (C:\projects\libzmq\src\decoder_allocators.cpp:85).

The call that's failing is malloc. I'm not an expert in Windows memory management. But possible causes include:

allocating a very large message (larger than _HEAP_MAXREQ), which is ~ 4 GB when run under win32
failure to deallocate messages (they could be queued in memory if they cannot be delivered, or they might not be freed properly)

Unfortunately, it's hard to say much without reproducible test cases.

Hello
I believe the issue is std::size_t shared_message_memory_allocator::_max_counters is being overridden by mistake from somewhere.
I wrote some debug to output the values of _max_counters when calls to shared_message_memory_allocator are made. The normal value should be 249 however it changes to large values such as<PHONE_NUMBER>464. The value it changes to is never the same each test but is always very large. The assert does not trigger when _max_counters changes. The value can be in a malformed state for most of the applications lifetime. If an allocation is made it will use the large _max_counters which triggers the assert due to trying to allocate an incredibly large amount of memory.
Note: _max_counters can actually be made const. When doing so the value stops become malformed but I suspect this is the compiler hiding the memory override somehow.
I have attached a small folder for VSCode that will reproduce the issue. It is about 1/5 repro rate for the assert. However most runs will malform _max_counters.
libzmq_repro.zip

@bluca this sounds like a libzmq bug to me. Any idea what could be writing weird values to _max_counters?

check the usual suspects first - multithreading, etc

https://github.com/zeromq/pyzmq/pull/1560
I have added the above pull request to fix this issue.

I have implemented your change to core.py locally on my box in C:\Program Files\Python39\Lib\site-packages\zmq\green\core.py.  After doing so, the out of memory issues that I was getting with locust (uses pyzmq) are gone.  Thanks for finding and fixing (or at least working around)!


Last x Days on Home Page

If you open the home page the system defaults to Last 30 Days, however, I would just like to see the last 7 days, so I change it to 7 and next time I open the page again I have to change it again.
If the Last X Days field is change, could this not be saved in either the postgress DB of even a config file and loaded back up with the page to only show that info. This value can then also be later used to filter through the entire site, to show last X days activity, statistics, recently added etc.
The value could also be pulled from the settings screen.
Additional context

Note to self, review where possible to store values in local storage for per device preferences.

Feature has been pushed to unstable


feat: Add a way to check if a file is ready

So that a client can let the user know the file is fully uploaded.
Maybe this could be added in the GET /files/metadata/{file_uuid} endpoint.

Why is this needed? Right now the GET /files/metadata/{file_uuid} endpoint returns a 202 status and a custom message when the file isn't ready yet.
Maybe we can add a GET method to the /ready/:fileUUID endpoint to just return the status of a file (if needed).

@PedroChaparro You are right. Thanks, I missed it.


Add self-hosted DNS Validation plugin

Adds a self-hosted DNS validation plugin as an option for DNS validation. The plug-in requires three one-time initial steps:
-Open port 53 on the machine that runs win-acme
-Create an A record in your DNS manager that points to the external IP of your win-acme machine
-Create individual NS records for each domain that you will validate.
For example, if you are trying to validate test.mydomain.com, you would create the following records:
NS: _acme-challenge.test.mydomain.com  ==> acme.mydomain.com
A: acme.mydomain.com ==> external IP of your win-acme machine
[note: you may already have an A record set up for your win-acme machine]
The plug-in will verify that you have correctly opened the DNS server port and set up the correct records during interactive mode of setup, giving you an opportunity to re-test or bypass the tests if problems occur.
During both setup and actual validation, a stripped-down DNS server will run on your machine and will shut down as soon as verification completes.
Given all of the other mechanisms for DNS validation, this one might not be completely necessary. Compared to the acme-dns server option, the self-host option is a lot easier to maintain and setup, and it doesn't require running a full-time server listening on port 53.

Given all of the other mechanisms for DNS validation, this one might not be completely necessary. Compared to the acme-dns server option, the self-host option is a lot easier to maintain and setup, and it doesn't require running a full-time server listening on port 53.

I see your point, though I'm worried about duplication of efforts, especially related to DNSSEC, which would very nice to have for this purpose (and acme-dns also lacks, though they may be working on it). Wouldn't it be possible to create some kind of wrapper around acme-dns? E.g. firing up a Powershell script that downloads and spins up an acme-dns docker image?
Anyway, in the meantime it looks like a nice tool to have on the utility belt, but I think it may be best to spin it off into a separate project like the Route53 plugin and others. Not just because it might not be the "ultimate" self-hosting solution, but also because I want to follow the rule of thumb that plugins that bring on new package dependencies (which this one does) should be separate projects, to keep the main distribution lean and more prepared for future .NET platforms (I'm eyeing .NET Core 3.0).
Of course that means that I will eventually want to push grandfathered plugins like FTP and WebDav to their own projects as well, but that would be a breaking change kind of off topic here :).

Completely reasonable. I can do some looking into how the Route 53 plug-in works and do something analogous, but I'm also fine just holding off until we get some clear sense that there is a need for this solution.


newly installed articulate package , clicking on categories or tags gives --> NullReferenceException: Object reference not set to an instance of an object.

I've freshly installed Articulate 5.1.0 and have umbraco version 13. Package is successfully installed however everytime I click on categories or tags from the blog home page it redirects me to localhost/categories and gives null reference exception.
What am I missing?

—
Reply to this email directly, view it on GitHub
https://github.com/Shazwazza/Articulate/issues/449#issuecomment-2155316683,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A6MABF2ZMS75HLBXW3AVQYLZGH3HHAVCNFSM6AAAAABI7CSJKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVGMYTMNRYGM
.
You are receiving this because you authored the thread.Message ID:
@.***>

Interesting, i can replicate, looking into it

Fixed in 5.1.1 https://github.com/Shazwazza/Articulate/releases/tag/v5.1.1


issues after reconnection

Looks like the Plugin Manager didn't get instantiated correctly, as well as the irc.invite method never got de-registered when Cardinal disconnected. (Basic guess, would have to look deeper in the code to verify.)
2015-03-06 02:25:51,214 - cardinal.bot - INFO - Could not connect ([Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionRefusedError'>: Connection was refused by
 other side: 111: Connection refused.
]), retrying in 300 seconds
2015-03-06 02:30:52,594 - cardinal.bot - INFO - Signed on as Cardinal
2015-03-06 02:30:52,595 - cardinal.bot - INFO - Attempting to identify with NickServ
Unhandled Error
Traceback (most recent call last):
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 120, in dataReceived
    self.wrappedProtocol.dataReceived(data)
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2430, in dataReceived
    basic.LineReceiver.dataReceived(self, data.replace('\r', ''))
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2438, in lineReceived
    self.handleCommand(command, prefix, params)
--- <exception caught here> ---
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2482, in handleCommand
    method(prefix, params)
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 1846, in irc_RPL_WELCOME
    self.signedOn()
  File "/home/john/files/Cardinal-DH/cardinal/bot.py", line 85, in signedOn
    self.event_manager.register("irc.invite", 2)
  File "/home/john/files/Cardinal-DH/cardinal/plugins.py", line 674, in register
    raise EventAlreadyExistsError("Event already exists: %s" % name)
cardinal.exceptions.EventAlreadyExistsError: Event already exists: irc.invite
2015-03-06 02:30:52,608 - cardinal.plugins - INFO - Calling callbacks for event: irc.mode
2015-03-06 02:30:52,680 - cardinal.plugins - INFO - Calling callbacks for event: irc.notice
2015-03-06 02:30:52,774 - cardinal.plugins - INFO - Calling callbacks for event: irc.notice
2015-03-06 02:30:52,851 - cardinal.plugins - INFO - Calling callbacks for event: irc.privmsg
Unhandled Error
Traceback (most recent call last):
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 120, in dataReceived
    self.wrappedProtocol.dataReceived(data)
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2430, in dataReceived
    basic.LineReceiver.dataReceived(self, data.replace('\r', ''))
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2438, in lineReceived
    self.handleCommand(command, prefix, params)
--- <exception caught here> ---
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2482, in handleCommand
    method(prefix, params)
  File "/home/john/files/Cardinal-DH/cardinal/bot.py", line 139, in irc_PRIVMSG
    self.plugin_manager.call_command(user, channel, message)
exceptions.AttributeError: 'NoneType' object has no attribute 'call_command'
2015-03-07 15:06:26,477 - cardinal.plugins - INFO - Calling callbacks for event: irc.notice
2015-03-12 20:13:51,596 - cardinal.bot - INFO - Connection lost ([Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
]), reconnecting in 10 seconds.
2015-03-12 20:14:08,986 - cardinal.bot - INFO - Signed on as Cardinal
2015-03-12 20:14:08,986 - cardinal.bot - INFO - Attempting to identify with NickServ
Unhandled Error
Traceback (most recent call last):
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 120, in dataReceived
    self.wrappedProtocol.dataReceived(data)
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2430, in dataReceived
    basic.LineReceiver.dataReceived(self, data.replace('\r', ''))
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2438, in lineReceived
    self.handleCommand(command, prefix, params)
--- <exception caught here> ---
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 2482, in handleCommand
    method(prefix, params)
  File "/home/john/files/Cardinal-DH/local/lib/python2.7/site-packages/twisted/words/protocols/irc.py", line 1846, in irc_RPL_WELCOME
    self.signedOn()
  File "/home/john/files/Cardinal-DH/cardinal/bot.py", line 85, in signedOn
    self.event_manager.register("irc.invite", 2)
  File "/home/john/files/Cardinal-DH/cardinal/plugins.py", line 674, in register
    raise EventAlreadyExistsError("Event already exists: %s" % name)
    cardinal.exceptions.EventAlreadyExistsError: Event already exists: irc.invite
2015-03-12 20:14:08,992 - cardinal.plugins - INFO - Calling callbacks for event: irc.mode
2015-03-12 20:14:46,450 - cardinal.plugins - INFO - Calling callbacks for event: irc.notice

On second look, I believe this is a two-part issue. It looks like the PluginManager hadn't been instanced by the time a private message was received. Which is odd, considering it also looks like the EventManager instance was kept around from the last connection.


FindpFUnit.cmake missing

Is there a reason why you removed cmake/modules/FindpFUnit.cmake in e8df281, although you have replaced all the pFUnit stuff in CMakeLists.txt by find_package(pFUnit REQUIRED)?

pFUnit comes with a pFUnitConfig.cmake which is the modern way to use find libraries. You need to do cmake . -DpFUnit_DIR=<path to pFUnit install directory> and it should be found.

It works, thanks a lot. I had to update pFUnit first.


Please provide a clean example for Core 2.1 that doesn't use ApplicationUser and Scaffolding (or Roles)

In general, we're not supposed to use Roles now (vs Claims moving forward except in limited cases), and everything is tucked away nice and clean in Core 2.1, only for all the examples (that I can find) to tell us to use Roles or scaffold everything out again and revert back to ApplicationUser. I realize this example will probably work in Core 2.1. However, can you please provide an example that does something similar as the above example and conforms to how Core 2.1 leaves everything tucked away clean with IdentityUser? In other words, for those that don't need to customize their Identity user, what is the solution for authorization and policies? How would you do like an admin or owner type of user? Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b66f8ffa-699c-8b11-0cab-805f5953805f
Version Independent ID: 7c151ed5-fd78-84fd-a08a-88912990c4ca
Content: Create an ASP.NET Core app with user data protected by authorization
Content Source: aspnetcore/security/authorization/secure-data.md
Product: aspnet-core
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

We're scheduled to update this topic/sample for 2.1 in #6290

@sumnone  the 2.1 version uses IdentityUser
See #7651


Include partners into this project

UCT?
Seed Initiative?
Farmer's network?

Computer Science Department: who is the person in charge?
Seed and Knowledge Initiative:  Funding could be use for hosting and domain fees, around ZAR 1000 per year.
Umthunzi Farming Community: Let's go to them with the MVP and iterate from there.

Computer Science Department: (TBC)
SARChI & SKI Project: Jaci van Niekerk - will update her on our progress; she is the link to the Computer Science Dpt.
Umthunzi Farming Community: Kim & Emma - best to approach with MVP
Abe Gordon: retired investor; current sharehold in several agricultural projects in WC; has potential connections & advice on the agricultural landscape and economic potential of Polycrox - perhaps we should approach him once we have the business outline? (before the MVP)
First People Group - (not sure of the specific name): Carlo - a representative of an indigenous community in Ocean View; can support the medicinal and indigenous herb elements of the simulation; strong activist for Coloured Rasta & Indigeneous growers
Organic Matters: Xola - known as the 'Green Ranger'; works as a food activist and networker by channeling funding into community projects; potential collaboration as he is currently working on 'Bots' for ecological, agroecological, organic agriculture questions for his personal website (maybe integrate that into Polycrox?)

Current Pitch:
A web and mobile app designed as an online library tool and crop rotation simulator,
Geared towards areas under 1 ha for community, subsistence, medicinal & herbal (indigenous), and small-scale market growers,
Framed around research from regenerative soil science, agroforestry, and Square Foot Gardening


Slight UI Issue in the dashboard

I hope this message finds you well.
I am currently facing a challenge with a bug reported by our content editors in our Optimizely project. We have installed the SEO Sitemaps (Geta.Optimizely.Sitemaps) to manage all SEO-related features on our CMS site. Functionally, this is working fine; however, there is a slight issue with the UI.
When editors add a significant number of entries (10-15), clicking on the "New Sitemap" button opens an additional section at the bottom for inputting details. Unfortunately, this section is no longer visible within the fold, as there is not enough scrolling available. As a result, editors cannot input sitemap details.
To resolve this, I think a small CSS styling is needed to ensure the scrollbar is set up correctly on the UI. Could anyone help me solve this as this is currently impacting the user/editor experience?
Thank you for your assistance.

I know this is an old entry, and I do not really have a permanent solution, but I would want one as well since our number of sitemaps will be around 50+ when we're finished ... so it would be great to have this fixed.
However, what you can do is:

press F12
Click Tab "Elements" in the top row of the developer tools
second entry on the right side should be "body", remove the checkmark next to "overflow:hidden"
Now you can edit everything and scroll the whole page for this session.


Compiler::get_shader_resources() does not return all uniforms

I have the following uniforms in my fragment shader:
layout(location = 0) uniform mat4 u_model;
layout(location = 4) uniform mat4 u_view;
layout(location = 8) uniform mat4 u_projection;

There is no way to get the above uniforms using Compiler::get_shader_resources() method (the struct ShaderResources does not contain uniforms member). The ones which does not have base type SPIRType::Image, SPIRType::Sampler, SPIRType::SampledImage or SPIRType::AccelerationStructure are not getting collected, I think they should be collected as uniforms.

Created PR https://github.com/KhronosGroup/SPIRV-Cross/pull/2190.

As per @HansKristian-Work, since the plain uniforms are supported only in legacy GL, we will use the name gl_plain_uniforms instead of uniforms.