[Bug]: Python ReadFromMongoDB/RangeTracker raises OverflowError

What happened?
Our dataflow job started to fail few days ago (apache beam 2.44) without any changes. After investigating logs and execution details I came to conclusion that the issue is with a part of code that is related to ReadFromMongoDB:
Error message from worker: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 287, in _execute
    response = task()
  File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 360, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
  File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 596, in do_instruction
    return getattr(self, request_type)(
  File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 634, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1003, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 1021, in apache_beam.runners.worker.operations.SdfProcessSizedElements.process
  File "apache_beam/runners/worker/operations.py", line 1030, in apache_beam.runners.worker.operations.SdfProcessSizedElements.process
  File "apache_beam/runners/common.py", line 1432, in apache_beam.runners.common.DoFnRunner.process_with_sized_restriction
  File "apache_beam/runners/common.py", line 817, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 988, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/sdf_utils.py", line 111, in check_done
    return self._restriction_tracker.check_done()
  File "/usr/local/lib/python3.8/site-packages/apache_beam/io/iobase.py", line 1590, in check_done
    return self.restriction.range_tracker().fraction_consumed() >= 1.0
  File "/usr/local/lib/python3.8/site-packages/apache_beam/io/range_trackers.py", line 285, in fraction_consumed
    return self.position_to_fraction(
  File "/usr/local/lib/python3.8/site-packages/apache_beam/io/range_trackers.py", line 432, in position_to_fraction
    return float(ikey - istart) / (iend - istart)
OverflowError: int too large to convert to float

Code:
pipeline | ReadFromMongoDB(uri=..., db=...,  coll=..., bucket_auto=True)
Looks like RangeTracker that is used underneath makes the code fail (ikey is probably choosen as a too large number).
The current collection size is 221418 elements which should not be a problem for float capacity.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

Also just wanted to mention that  the same code in different environment succeeds (it has different number of items in mongo collection).
Also the same code and the same pipeline and environment worked just a few days ago.
So it really should be the issue with the number of items in the mongo collection (And the Error place also suggests that too)

Just verified that I see the same issue on 2.46


"[" sometimes resets cursor position (textmode)

Typing "[" in expression bodies will sometimes cause the cursor to reset to the beginning of the line. A very basic example can be seen by typing "a=[" or "a = [". The end square bracket will be inserted automatically and the cursor will reset to the beginning of the line.
Browser: Chrome

I have a half-baked solution in the form of the following code (swap out in modify.ts). But it is whitespace-dependent (works for a = [ but not a=[) and overall a hack. I'd rather address the underlying cause, #393.
function insertWithIndentation(
  view: EditorView,
  pos: TextAST.Pos,
  insert: string
) {
  const indentation = getIndentation(view, pos.from);
  insert = insert.replace(/\n/g, "\n" + indentation);
  const head = view.state.selection.main.head;
  if (
    pos.from <= head &&
    head <= pos.to &&
    view.state.sliceDoc(pos.from, head) === insert.slice(0, head - pos.from)
  ) {
    return {
      from: head,
      to: pos.to,
      insert: insert.slice(head - pos.from),
    };
  }
  return {
    from: pos.from,
    to: pos.to,
    insert,
  };
}


Lpoptions instances working again.

Issue https://github.com/OpenPrinting/cups/issues/71
cups_enum_dests() was not properly accounting the instances, so passed the num_dests again to cups_get_dests() after passing to _CupsGetDests() to get the required instances from the ~/.cups/lpoptions file.
@zdohnal @michaelrsweet Please review it. Thanks !

Will review, still not sure this is the correct fix.

[master b53ad5bf3] Fix printer instances (Issue #71)


.sublime-syntax

Hi :wave:
I made an entirely new syntax definition for groff files (with a focus on man macros) in the new sublime-syntax format and was wondering whether you are interested in replacing your dated syntax with that one.
It follows scope naming conventions as well as possible and supports all font modifiers that I could find, with a couple extra commands etc. The only thing to note is that it doesn't support ST2, but less than 5% of Package Control users are on that, so it shouldn't be an issue. If anything, the current tmLanguage can still be used by older builds, though I would suggest getting rid of it.
I haven't used anything else of your package, i.e. the preview mode, but if I find areas to improve I won't hesitate to propose them.
By the way, for better searchability you should include "groff" and "troff" in your repo's description.

Thanks for the input! Make a PR with the replacement and I would be happy to test it and pull it in.


Resolve package types in interfaces

Fixes #2114

Thank you for fixing this issue! I believe this change is correct. I would appreciate it if you could also add a test (as you've kindly done elsewhere), ideally covering each of the three branches. There are a couple of interface-related tests in tests/svinterfaces, but they are lacking.

OK, I've added a test, which checks both a normal typedef and a typedef in a package.
As far as I can see, the two first branches modified by the patch are part of the same thing (enter and exit module or interface).
The third branch just considers a (* mem2reg *) attribute for both modules and interfaces. I couldn't find any test for (* mem2reg *) on modules to build on.
I think I have now contributed as much as I can reasonably be expected to on this one - I just stumbled across the issue and thought it could be an easy fix. I don't have any interest in interfaces whatsoever myself - I tried to use them at one time, but decided that in my case, they couldn't provide anything that couldn't be done better with structs :smile:


.maf file not produced ?

Hello,
First of all, thanks for this amazing fast genome alignment tool!
The program does not output .maf file for an alignment between two .fasta files. However, all other files are produced after alignment is done.
Attached screenshot shows some of the last lines of console output.

Please help or comment if there is some mistake in using the tool efficiently.
I really need the .maf file.
Many thanks in advance!
Command used:
.../GSAlign -t 4 -slen 20 -alen 2000 -i  -q  -o  -fmt 1 -dp

Hi! It looks like GSAlign could not find alignments between the two sequences. You may use different parameters to lower the thresholds and try again. Since your sequences are not very long, I'll suggest to use -slen 15 -alen 100 or -sen (sensitive mode). If there should be alignments and GSAlign could not detect them, please let me know. Thank you!


[Travis]: fix 3.7/docs build for external PRs

External PRs will fail on Travis due to the way the docs are deployed. We should add some sort of hook that only deploys the docs from the vtki master branch (not other people's forked master branches)

You have access as admin, right?  Or do you need me to fix this?

I have admin access and can fix it just not sure what needs to be done at the moment

Fixed in 4d5a9a8c5693255a353b2408bdca06a5dd7156cd

Note that I changed:
https://github.com/vtkiorg/vtki/blob/919f096e2d9ce1b527abe42434ebab815dd14dc5/.travis.yml#L82
to have and conditions && because we only want to deploy the docs if all three of those conditions are met

Ah, that's right.  Still not that familiar with bash I'm afraid.

Down the road when the docs are more complete and the API is more stable, we should likely change this to only update the docs on tagged releases

Agreed.

Looks like this is still an issue.  I thought it would have been solved by my commit.  Guess not.

 If that variable is set correctly, then the docs will deploy.
see ecbe734
Eventually we should set it up in a deploy script but the output from deploy scripts are not shown so they are ridiculously difficult to debug

We should wait until the next external PR to close this

#178 shows this is working! Hack of a solution but it'll do for now.


Old screen shot

Please update to show the current screens for this functionality
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 35b52f01-3dee-6d41-f9f6-7710e461ffeb
Version Independent ID: c8ef08a8-d503-b52f-9de8-21b13cfa3d72
Content: Azure Active Directory Connect Health operations
Content Source: articles/active-directory/hybrid/how-to-connect-health-operations.md
Service: active-directory
Sub-service: hybrid
GitHub Login: @billmath
Microsoft Alias: billmath

@DeanGross
Thanks for your feedback! I've assigned this issue to the author who will investigate and update as appropriate.

@DeanGross
Thank you for the feedback. I have created a work item to update the screenshots. I am going to close this issue now and will update it once I have additional information and an update. Thanks!
Bill
#please-close


Feature/disable menu

Disabled buttons on menu header for both desktop and mobile visualizations.
Added home button to navigate back to the currently selected base homepage.

 2.0 Menu: hide QR-Scanner, Mobile Distribution, Boxes and Admin menu. Only the User thumbnail should be there.

Can you also remove the Profile link in the menus?

@johnsmol It does what it should. Thank you very much Giovanni. Was really nice working with you.

@pylipp @vahidbazzaz ready to be merged when you want it to

 Add Menu Item "Classic View" with links to go to dropapp


Add master volume sensor.

Because #90 has gone stale, I converted the code into a new sensor and update GUI and documentation to reflect this.

Hi! Thanks for creating a new PR. The sensor should also be configurable through the UI. Would you mind adding that?

After making a new directory and cloning my develop branch, I have failed to replicate this issue. Here is the list of commands I used to recreate the test build :
git clone --branch develop https://github.com/SecretiveShell/hass-workstation-service
cd .\hass-workstation-service\
mkdir build
dotnet build
dotnet publish --output .\build\
cd .\build
explorer .

After closing the installed version of HWS using task manager I was able to initialize the new one and successfully configure the comman
build log.txt
d

This is a great addition. When will there be an official release/build for those of us who do not have the dotnet build tools.

Were you also planning to add a SetVolumeComand from your end @SecretiveShell or is it something you would consider? My goal is to update some HA automations when I click certain "Activities" on a harmony remote, so for "Play Games" I wanted to set TV HDMI inputs (done), set wireless speaker inputs to say Optical (done), as well as set the wireless speaker volume (done) and set an appropriate windows master volume (to do).

That would be a nice addition, although I currently have no plans to implement this. It would require a rework of how the app handles MQTT messages.


Add Befunge to syntax list

Name of an extension you are adding
Befunge
Why do you think this extension is awesome?
Adds language support and alignment guides for the Befunge-93 and Funge-98 esolangs - alignment guides are quite helpful in a language which takes advantage of directionality
Make sure that:
[ ] Screenshot/GIF included (to demonstrate the plugin functionality)

not included, as it doesn't look like any other language syntaxes included one. However, here is a screenshot:

[x] ToC updated

👍


Issue while calling the GUI

I had deployed the code as per your instructions on apache karaf 4.0.4 and when i hit the url
http://localhost:8181/personui
I am getting the exception
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '405: HTTP method POST is not supported by this URL' when communicating with http://localhost:8181/cxf/personService

Can you check the URL http://localhost:8181/cxf to see if the service is present at all?
I also recently updated the Tutorial and the code. So please checkout the newest versions.

With the latest code am getting a 404 for all the urls you mentioned. Also I am installing cxf version 3.1.6, will that be an issue. But all the bundles are active.Please check the screenshot for more details.

I extracted a plain karaf 4.0.4 and installed:
feature:repo-add cxf 3.1.6
feature:install http cxf-jaxws http-whiteboard
install -s mvn:net.lr.tutorial.karaf.cxf.personservice/personservice-model/1.0-SNAPSHOT
install -s mvn:net.lr.tutorial.karaf.cxf.personservice/personservice-server/1.0-SNAPSHOT
install -s mvn:net.lr.tutorial.karaf.cxf.personservice/personservice-proxy/1.0-SNAPSHOT
install -s mvn:net.lr.tutorial.karaf.cxf.personservice/personservice-webui/1.0-SNAPSHOT
After that I can see the cxf servlet on /cxf and also the service. If that does not work for you can you show a list of all bundles:
la -s
When I do this I see that I have a lot less bundles than you. In my karaf the highest bundle id is 116 while you have a lot more bundles installed.
Make sure you freshly extract karaf or delete the data dir before start.

I have done like you told, but still I am getting the previous error. The cxf is still showing 404. Please find the screenshot

Can you do:
dev:dump-create --no-heap-dump
and attach the created zip here?

Please find the dump attached.
2016-04-04_133529.zip

I downloaded a fresh installation of karaf 4.0.4 and its working fine on it
thank you.

Also can you create a sample where we are consuming a rest service from a javascript ?

http://localhost:8181/cxf/
http://localhost:8181/personui are finr
http://localhost:8181/cxf/person/ is giving no service found error

http://localhost:8181/personuirest is giving 404

The rest and soap examples are separate now. Did you install the rest example?
There already is such a javascript example. See https://github.com/cschneider/Karaf-Tutorial/tree/master/tasklist-blueprint-cdi/angular-ui
The rest service is exported in a bit unusual way in this example as I wanted to keep it annotation only but you can also use the blueprint approach.

I installed the rest one as mentioned in your readme.txt
feature:repo-add cxf 3.1.5
feature:install cxf-jaxrs http
install -s mvn:net.lr.tutorial.karaf.cxf.personrest/personrest-model/1.0-SNAPSHOT
install -s mvn:net.lr.tutorial.karaf.cxf.personrest/personrest-server/1.0-SNAPSHOT
install -s mvn:net.lr.tutorial.karaf.cxf.personrest/personrest-webui/1.0-SNAPSHOT
The rest service is up, but the ui is giving 404

Can you to add the http-whiteboard feature. I think I accidently removed it in the last changes but it is needed by the UI.

Can you please create an example in which we have a wab which have a rest service in it with the same context root as the WAB?

I know this works for spring and there it works simply like in plain java. For blueprint I am not sure if it is possible at all. You should ask this on the CXF user list.

you mean I have to create the rest service using spring ? do you have a sample ? also can u give link to cxf user link ?

Its working fine after adding the http-feature. Please add it in the readme txt too.
Also do you have an example for basic form auth ?

I added the feature to the readme. Unfortunately I have no example for a form based auth.

thank you.
I created it.


Bookmarks plugin needs a rewrite?

Hi, the bookmarks  plugin code is older and dysfunctional.
I'd like rewrite the plugin, the localstorage structure would remain as it is now (so don't make too many changes), but many functions and the rest would be differents, so many third-party tools will be broken (if not updated from their developers).
It's better the plugin will remains as it or that is updated?
Or create a second bookmarks plugin (same storage structure, same or different namespace), so users and developers can make a smooth transition from old to new (developers update their plugins and the users can wait to update to the new version)?

I think a new namespace would be appropriate for significant changes - existing 3rd party integrations will act as if it was not loaded, rather than being broken.

I think a new version would be more appropriate.
What you might also consider: the drive sync can only handle concurrent
edits at the top level, so if I add a bookmark on device A and another one
on B, one change will get overwritten since bookmarks are currently at
third level (top level is portals and maps, second level are folders).
Instead, one might rather add different files for maps and portals (like in
the missions plugin for waypoints) so that at least different folders are
independent. Or use a data structure where all items are on the top level
and store their parent's ID, so the tree can be reconstructed on the fly.

Storing the parent's ID still has a race condition.  You'd need an integrity check to verify all parent IDs are still valid, and a way of dealing with the ones that are not.

@ZasoGD
Is there any progress on new plugin?


add StateDuration node

This is a proposal for a solution to #1211
It provides a node which evaluates an expression, and adds a field which tracks how long the expression has evaluated as true.
Example usage which alerts when a host has not had any primary ntp peers for over 5 minutes looks like:
stream
  |from().measurement('ntpq')|default().tag('state_prefix',' ')
  |groupBy('host')
  |stateDuration(lambda: "state_prefix" != '*').unit(1m)
  |default().field('state_duration',-1)
  |alert()
    .warn(lambda: "state_duration" > 5)
    .crit(lambda: "state_duration" > 60)

If this is an acceptable solution to the problem, I can finish up the PR by adding tests, docs, etc.
Required for all non-trivial PRs

[x] Rebased/mergable
[x] Tests pass
[x] CHANGELOG.md updated
[X] Sign CLA (if not already signed)

What should be the behavior when we hit a point where the expression doesn't evaluate (such as referencing a field which is null)? The options I see

Do nothing and pass the point through unchanged.
Treat it as if the expression evaluated as false (reset duration).
Drop the point.

I don't like option 2. I'm torn between options 1 & 3.

I would go with option 3 as that is how the eval node behaves. In addition to dropping the point the error is logged and a counter is incremented indicating there was an error.

Also as a heads up this PR https://github.com/influxdata/kapacitor/pull/1175 is about to be merged which changes how the error should be counted. So I would recommend rebasing your work off that branch for a bit or just wait till that PR is merged. I expect the PR will be merged either tonight or tomorrow sometime.

Just merged  #1175

Adjusted for comments, and added tests & docs.

Updated to set the resulting duration to -1 if expression evaluates as false.
How do we want to handle #1225? It's still under discussion, but do we want to implement any of the proposals in here?

@phemmer I think include it at a part of this PR.

@desa @phemmer
Quote from @desa in the other issue

My only concern is if I'm a user, looking for this functionality, would I think to use stateDuration? I think not.

As I result I think having a separate node that is similar in functionality but mirrors the behavior is best.
It keeps it simple so that each node has a single purpose. And since they could mirror each other in property names etc then it makes it so that learning one teaches you how to use the other.

What should it be called?

I think stateCount works.

I'm not fond of the name stateCount as it's not indicative of the fact that it's a consecutive count, instead of a total count. But meh, naming is easy to change so I've updated the PR to include the feature with that name.
Also instead of a massive copy/paste of code with a tiny amount of different lines, I redesigned them to share the same core code. There are unfortunately a dozen different ways of doing this, so the implementation I have up now is just what I thought would be easiest to understand & maintain. But open to other ideas.

@nathanielc Requested change have been made.


[Docs] Long Windows paths in FilePathUrlConverter notebook

Part of OpenAssetIO/OpenAssetIO#1257. The
FilePathUrlConverter.pathFromUrl function now converts long Windows file URLs, which exceed the Windows MAX_PATH limit, to UNC device paths.

Note that this has a depdenency on https://github.com/OpenAssetIO/OpenAssetIO/pull/1269l. Tests pass without the changes in that PR because there are no assertions of the output (it just prints).


update vault-helm to 0.0.2 for upstream

Signed-off-by: Brett Johnson<EMAIL_ADDRESS>Thanks submitting your Operator. Please check below list before you create your Pull Request.
New Submissions

[x] Does your operator have nested directory structure?
[x] Have you selected the Project Community Operator Submissions in your PR on the right-hand menu bar?
[x] Are you familiar with our contribution guidelines?
[x] Have you packaged and deployed your Operator for Operator Framework?

[x] Is your submission signed?
[x] Is operator icon set?

Updates to existing Operators

[x] Did you create a ci.yaml file according to the update instructions?
[x] Is your new CSV pointing to the previous version with the replaces property if you chose replaces-mode via the updateGraph property in ci.yaml?
[x] Is your new CSV referenced in the appropriate channel defined in the package.yaml or annotations.yaml ?
[x] Have you tested an update to your Operator when deployed via OLM?
[x] Is your submission signed?

Your submission should not

[x] Modify more than one operator
[x] Modify an Operator you don't own
[x] Rename an operator - please remove and add with a different name instead
[x] Submit operators to both upstream-community-operators and community-operators at once
[x] Modify any files outside the above mentioned folders
[x] Contain more than one commit. Please squash your commits.

Operator Description must contain (in order)

[x] Description about the managed Application and where to find more information
[x] Features and capabilities of your Operator and how to use it
[x] Any manual steps about potential pre-requisites for using your Operator

Operator Metadata should contain

[x] Human readable name and 1-liner description about your Operator
[x] Valid category name1
[x] One of the pre-defined capability levels2
[x] Links to the maintainer, source code and documentation
[x] Example templates for all Custom Resource Definitions intended to be used
[x] A quadratic logo

Remember that you can preview your CSV here.
--
1 If you feel your Operator does not fit any of the pre-defined categories, file an issue against this repo and explain your need
2 For more information see here

@SDBrett, please add replaces directive to your CSV. More info here

@SDBrett, please add replaces directive to your CSV. More info here

Added

@SDBrett we found out that same problem you had in openshift version. Can you please make new pull request to fix it also in community-operators dir? Then we will merge both. Thanks
PS: We are working on fix to catch this error

@SDBrett we found out that same problem you had in openshift version. Can you please make new pull request to fix it also in community-operators dir? Then we will merge both. Thanks
PS: We are working on fix to catch this error

Thanks for letting me know :)
Created PR #3115 to fix the community-operators CSV


Is there a standard approach to adding Select or Clear All options?

In the list of choices, is there a way to add an option that adds/removes all options to/from the selected group? Has anyone implemented something like this?
I suppose I could probably hack around this with ng-change -- I'm just wondering if there's a better way.

Clear all has been implemented in more recent versions of the library.
To simulate select all, simply add a button to add all the options to your ng-model array.


South Korean postal code format has changed

According to an issue filed against twitter-cldr-rb, postal codes in South Korea have changed from 6 digits to 5. I believe the appspot API data needs to be updated with this change.
Is there anything I can do to speed this process along?

Currently the widget accepts both 5 digit and 6 digit postal codes.  We will be deprecating support for 6 digit postal codes in the near future.  I'll update this issue and close it out once we do.

Thanks for the feedback.
Closing this bug out regex for South Korea has been updated and it's now reflected in the data served from Chromium.


Update akka-actor, akka-stream, ... to 2.6.6

Updates

com.typesafe.akka:akka-actor
com.typesafe.akka:akka-stream
com.typesafe.akka:akka-stream-testkit
com.typesafe.akka:akka-testkit

from 2.6.3 to 2.6.6.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.typesafe.akka" } ]

labels: library-update, semver-patch

We get a runtime exception introduced by https://github.com/akka/akka/pull/29112


Suspend to RAM doesn't work with NVidia Prime / Thinkpad P1 Gen2

Describe the bug
On the new Lenovo Thinkpad P1 Gen 2 series (suspect its the same as X1 Extreme), when using NVidia Prime suspend to RAM no longer works. If you disable hybrid graphics in the BIOS and just use the NVidia graphics card then suspend to RAM works fine (albeit it completely kills your battery and makes the fans on your laptop go crazy).
When you do suspend to RAM in this configuration and then resume, the screen just stays black (the laptop does properly power up and the keyboard lights up). Unlike other suspend to RAM black screen problems, the only way to get out of this black screen is to physically restart the laptop with the power button, manually switching to TTY with Ctrl+Alt+F1 and doing systemctl restart display-manager does not work.
To Reproduce
Steps to reproduce the behavior:

Set up NVidia Prime in static mode as described here https://nixos.wiki/wiki/Nvidia
Press suspend to RAM
Wait for laptop to suspend to RAM and then resume by hitting the power button. The screen is now stuck being black.

Expected behavior
Resuming from suspend to RAM works (i.e. it opens your login manager)
Additional context
The relevant nixos configuration is here
boot.blacklistedKernelModules = [ "nouveau" ];
hardware.nvidia = {
  modesetting.enable = true;
  optimus_prime = {
    enable = true;
    nvidiaBusId = "PCI:1:0.0";
    intelBusId = "PCI:0:2.0";
  };
};
services.xserver.videoDrivers = [ "intel" "nvidiaBeta" ];

Archlinux has a good resource https://wiki.archlinux.org/index.php/Lenovo_ThinkPad_X1_Extreme_(Gen_2) . According to their documentation the latest nvidia beta drivers should just work fine (Prime Offloading should also work although that is covered by this PR https://github.com/NixOS/nixpkgs/pull/66601 )
@eadwu Noted that the most likely reason behind this is that Nixos uses default power management rather than one based on systemd. For nvidia its recommended to use the systemd power management (more info here https://download.nvidia.com/XFree86/Linux-x86_64/435.17/README/powermanagement.html)
Metadata
 - system: `"x86_64-linux"`
 - host os: `Linux 5.3.11, NixOS, 20.03pre201791.c1966522d7d (Markhor)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.1`
 - channels(root): `"nixos-20.03pre201791.c1966522d7d"`
 - channels(mdedetrich): `"nixpkgs-20.03pre201329.f1682a7f126"`
 - nixpkgs: `/home/mdedetrich/.nix-defexpr/channels/nixpkgs`

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:  [ "services.xserver.videoDrivers" "hardware.nvidia" "boot.blacklistedKernelModules" ]

Personally for me, suspend for RAM on NVIDIA has always been "glitchy". I suppose that this issue will remain as long as we use the default power management instead of systemd-based one NVIDIA recommends.

@eadwu

I suppose that this issue will remain as long as we use the default power management instead of systemd-based one NVIDIA recommends.

Can you expand on this, what power management does NixOS use?

Maybe it makes sense to add an hardware.nvidia.power_management.enable attribute along with powerManagement.systemd where if you set hardware.nvidia.power_management.enable to true then it will set a powerManagement.sysetmd to true?
Sounds like a big feature though.

I don't know of any other drivers that use systemd-based power management, so if it is just a nvidia thing, keeping it under hardware.nvidia is probably the better way to do this.
Also on the expansion of the "glitchy" behavior, I don't usually have problems resuming within a short time (<1 min) of suspending with NVIDIA online, but when I leave it in suspend for long periods of time, I experience the same problem. Before this use to also effect it when the monitor brightness was 0 but this was fixed by HardDPMS and fixes in the driver I believe.

I don't know of any other drivers that use systemd-based power management, so if it is just a nvidia thing, keeping it under hardware.nvidia is probably the better way to do this.

Agreed

Before this use to also effect it when the monitor brightness was 0 but this was fixed by HardDPMS and fixes in the driver I believe.
I have an OLED screen so I am not sure if this is making an impact (OLED screens technically don't have brightness so the standard kernel way of configuring brightness doesn't work on them)

Also judging from what NVidia is saying about this, i.e.

However, these allocations are collectively large, and typically cannot be evicted. Since the amount of system memory available to drivers at suspend time is often insufficient to accommodate large copies of video memory, the NVIDIA kernel drivers are designed to act conservatively, and normally only save essential video memory allocations.

The resulting loss of video memory contents is partially compensated for by the user-space NVIDIA drivers, and by some applications, but can lead to failures such as rendering corruption and application crashes upon exit from power management cycles.

Another theoretical reason why the default power management doesn't work is due to limitations of memory in the kernel memory and the memory of the graphics card that I have is quite high (4 GB of VRAM)

Before this use to also effect it when the monitor brightness was 0 but this was fixed by HardDPMS and fixes in the driver I believe.

Is HardDPMS on by default now?

The limitation of memory is the most likely cause, on the Arch Linux front, seems like they use it [1], though not sure whether or not it's enabled by default.
[1] https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/nvidia-utils#n166

Okay so it seems like this is a first good step, is this something you would be willing to look at or should I do an attempt? Note that I am very new to nix/nixos so it would likely take some time for me to learn how it works properly.
I don't think it should be that difficult since we already have a reference to work on (i.e. arch).

I'll see how much time I can pull, though yeah by the next stable the implementation should be ready if nobody else picks it up, (since at least I'll work on it over winter break).

So update on this, I am no longer getting this issue from a couple of months ago although I am not sure if this is due to an update in NVIDIA/Kernel or the mix module itself

the screen just stays black

I have this problem with my ThinkPad T25 since I upgraded from NixOS 19.09 to 20.03.

@nh2
What is your config (i.e. NVidia/XServer hardware conf?)

@mdedetrich
  services.xserver.videoDrivers = [ "nvidia" ];
  hardware.nvidia.optimus_prime.enable = true;
  # Bus ID of the NVIDIA GPU. You can find it using lspci, either under 3D or VGA
  hardware.nvidia.optimus_prime.nvidiaBusId = "PCI:2:0:0";
  # Bus ID of the Intel GPU. You can find it using lspci, either under 3D or VGA
  hardware.nvidia.optimus_prime.intelBusId = "PCI:0:2:0";
  #hardware.nvidia.modesetting.enable = true; # tried both, makes no difference

  services.xserver.displayManager.defaultSession = "xfce+i3";

niklas:~/ $ lspci | grep -i nvidia
02:00.0 3D controller: NVIDIA Corporation GM108M [GeForce 940MX] (rev a2)

This still does not work for me, even with these patches applied on top of 20.03 in order:

#86094
#77054
#73530

In both the new offload mode, and before in sync mode, do I get a black screen in Xorg when resuming from standby. I'm pretty sure the screen is fully off, that is no backlight is on (it turns on when switching to the a virtual terminal). I then have to switch to the virtual terminal with Ctrl+Alt+F1, and restart Xorg with sudo systemctl restart display-manager. Of course that loses my desktop session.
I also tried with adding "modesetting" to services.xserver.videoDrivers = [ "modesetting" "nvidia" ]; and hardware.nvidia.modesetting.enable = true. The nely added nvidia-resume.service runs thorough successfully according to journalctl output, but it doesn't help.
@eadwu is this working for you?

I found that this helps: https://askubuntu.com/questions/512192/turn-monitor-back-on-after-xrandr/553944#553944
From the VT, run:
sudo chvt 7; sleep 3; xrandr --display :0.0 --auto

That turns the screen back on.
Isn't that what nvidia-resume.service from #73530 is supposed to do? It does a chvt, but no xrandr.

I see https://github.com/NixOS/nixpkgs/blob/9d0c3ffe6783d59b427d018e8341e0084737fde9/nixos/modules/hardware/video/nvidia.nix#L220-L224
Should something like that also be run upon resume?

I found an ugly workaround that makes it work on my laptop:
{
  # Workaround to make standby resume work with nvidia without getting a black screen because the display is off.
  # See https://github.com/NixOS/nixpkgs/issues/73494
  systemd.services.nvidia-resume.serviceConfig = {
    # Requires `xhost +local:` in `sessionCommands` so that root can run X commands.
    ExecStartPost = "${pkgs.xorg.xrandr}/bin/xrandr --display :0.0 --auto";
  };

  services.xserver.displayManager.sessionCommands = ''
    # Needed to fix resume on nvidia, see `nvidia-resume` section.
    # TODO: This is suboptimal but I haven't figured out yet how to make root-commands work with XAUTHORITY
    ${pkgs.xlibs.xhost}/bin/xhost +local:
  '';
}

I'm not entirely sure how the suspending works since I rarely suspend my laptop anyway since there seems to be bugs with waking up from suspend (hardware and/or software?).
The few times I did test it, I never had any problems, though it might be dependent on the situation (nearly all of those times the video card wasn't really being used).

This seems to be fixed for me with 21.05, can anyone confirm?


Fix incremental hitting end_value throwing out whole batches

Raise a custom StopGenerator exception after filter so whole batch isn't thrown out.
StopIteration turns into a RuntimeError when raised from generator so use custom exception instead.

@steinitzu I will do a pre-release of dlt when this is merged so we can update the zenpy source


cannot find module jest-cli on cli tool

Error: Cannot find module '/Users/sibelius/Dev/me/flow-typed/cli/node_modules/jest-cli'
at Function.Module._resolveFilename (module.js:485:15)
at Function.Module._load (module.js:437:25)
at Module.require (module.js:513:17)
at require (internal/module.js:11:18)
at getJest (/Users/sibelius/Dev/me/flow-typed/cli/node_modules/jest/node_modules/jest-cli/build/cli/getJest.js:22:12)
at Object.run (/Users/sibelius/Dev/me/flow-typed/cli/node_modules/jest/node_modules/jest-cli/build/cli/index.js:39:48)
at Object. (/Users/sibelius/Dev/me/flow-typed/cli/node_modules/jest/node_modules/jest-cli/bin/jest.js:14:25)
at Module._compile (module.js:569:30)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:503:32)

@sibelius Were you able to get this working eventually?

Only installing jest-cli as well

Thanks for the update, in that case I'm going to re-open the issue.

I think we can close in fact, it is not that bad. I’ll try again with latest version, tks


[request] could/should the announce field be optional?

I used lava_torrent::torrent::v1::TorrentBuilder to build a torrent file and then tried to parse it with serde_bencoded.
I know the BitTorrent specification is notoriously sloppy and ambiguous, so I presumed this would have to follow the Robustness Principle.
git clone https://github.com/knightpp/serde_bencoded.git
cd serde_bencoded
cargo run --example torrent -- ../../mk_torrent/sample.torrent

    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/examples/torrent ../../mk_torrent/sample.torrent`
Error: Message("missing field `announce`")

"If it's missing then just display an empty Vector?"

serde_bencoded crate is just about bencoding. So it has nothing to do with torrent specification.
If you need announce to be optional you can change it.
Here we have announce: String
And you want it to be
#[derive(Debug, Serialize, Deserialize)]
struct MetaInfo {
// ...
    announce: Option<String>,
// ...
}

https://wiki.theory.org/BitTorrentSpecification#Metainfo_File_Structure
That website doesn't say it is optional, so I assume it is required.

I'll go and report this to lava_torrent.  Thank you for your help and patience.

You are welcome :)


Added Extractor for Athena metadata

Added Athena sample dag that uses Athena metadata extractor
Added Unit test for Athena metadata extractor
Summary of Changes
Added Extractor for Athena metadata
Added sample dag for Athena metadata extractor
Added requirement PyAthena[SQLAlchemy]
Tests
Added Unit test for Athena metadata extractor
Documentation
CheckList
Make sure you have checked all steps below to ensure a timely review.

[V ] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"

In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

[V ] PR includes a summary of changes.
[V ] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
[ V] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain docstrings that explain what it does

[ ] PR passes make test
[V ] I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message"

will take a look tomorrow. Thanks @levyitay

@levyitay FYI, the CI seems to fail though.

@feng-tao looks like bigquery is failing... didn't do anything with that piece of code..
E       DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

Also when testing on Python3 test_table_column_usage_aggregate_extractor.py is failing..
not sure how it's related to my code addition

@levyitay correct, currently the column_usage extractor is not available for py3 as it uses a antlar_py2 dependency. We have a plan to migrate it to py3, but just haven't had time yet :)
Per CI log, it seems the issue is due to flakey8
flake8 .
./example/dags/athena_sample_dag.py:68: [E302] expected 2 blank lines, found 1
./example/dags/athena_sample_dag.py:82: [E303] too many blank lines (2)
./example/dags/athena_sample_dag.py:91: [E231] missing whitespace after ':'
./example/dags/athena_sample_dag.py:110: [E231] missing whitespace after ','
./example/dags/athena_sample_dag.py:110: [E501] line too long (132 > 120 characters)
./example/dags/athena_sample_dag.py:114: [E302] expected 2 blank lines, found 1
./example/dags/athena_sample_dag.py:162: [E303] too many blank lines (5)
./tests/unit/extractor/test_athena_metadata_extractor.py:49: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:59: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:65: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:71: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:77: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:83: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:89: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:96: [E126] continuation line over-indented for hanging indent
./tests/unit/extractor/test_athena_metadata_extractor.py:116: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:123: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:124: [E126] continuation line over-indented for hanging indent
./tests/unit/extractor/test_athena_metadata_extractor.py:130: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:131: [E126] continuation line over-indented for hanging indent
./tests/unit/extractor/test_athena_metadata_extractor.py:140: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:146: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:152: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:158: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:164: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:170: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:176: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:182: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:188: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:194: [E231] missing whitespace after ':'
./tests/unit/extractor/test_athena_metadata_extractor.py:265: [E303] too many blank lines (3)
./databuilder/extractor/athena_metadata_extractor.py:15: [E303] too many blank lines (3)
./databuilder/extractor/athena_metadata_extractor.py:27: [E501] line too long (168 > 120 characters)
./databuilder/extractor/athena_metadata_extractor.py:27: [W291] trailing whitespace
./databuilder/extractor/athena_metadata_extractor.py:38: [E303] too many blank lines (2)
./databuilder/extractor/athena_metadata_extractor.py:51: [E303] too many blank lines (2)
./databuilder/extractor/athena_metadata_extractor.py:90: [E501] line too long (134 > 120 characters)
./databuilder/extractor/athena_metadata_extractor.py:94: [E303] too many blank lines (2)

Codecov Report

Merging #105 into master will increase coverage by 0.21%.
The diff coverage is 94.11%.

@@           Coverage Diff            @@
##           master   #105      +/-   ##
========================================
+ Coverage   82.78%    83%   +0.21%
========================================
  Files          52     53       +1
  Lines        2591   2642      +51
  Branches      271    276       +5
========================================
+ Hits         2145   2193      +48
- Misses        359    361       +2
- Partials       87     88       +1

Impacted Files
Coverage Δ

databuilder/extractor/athena_metadata_extractor.py
94.11% <94.11%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c6073c...95b9ed7. Read the comment docs.

@feng-tao thanks, didn't know about flakey8.
Maybe we should add it to the checklist?

thansk @levyitay , will take a look later today or tomorrow. Yeah I think we should update the pull request template to reflect that :)


Where shall I put this file "terraform_azure.tf"

Where shall I put this file "terraform_azure.tf"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: dc3a8a34-f0f0-7aba-b45b-2c59d722b7de
Version Independent ID: 667d19a1-ff34-3979-105c-c0a918d29707
Content: Create a Linux VM with infrastructure in Azure using Terraform
Content Source: articles/terraform/create-linux-virtual-machine-with-infrastructure.md
Service: terraform
GitHub Login: @TomArcherMsft
Microsoft Alias: tarcher

@kind03 The file name and location are user-specific. You could put in pretty much any directory from which you want to call the Terraform commands.

Thanks!


[Feature] Add user-agent for requests to start.spring.io

Is your feature request related to a problem? Please describe.
When requesting a new project on start.spring.io, we prefer that a User-Agent header to be set, see the reference guide.
Describe the idea you'd like
A User-Agent to be set according to the docs for every request to the service.

Hi @snicoll!
Thanks for reaching out about this. I was indeed not aware of that guide.
I will start implementing the enhancement right away.
PS: by the way, I love ❤ the work you do on Spring (core, boot, initialzr, etc)! Keep up 💪🏾

Sweet, thanks for the fast turnaround and the kind words!
Just to make sure I got this right, the User-Agent is something like @nxrocks_nx-spring-boot/<version>, e.g. @nxrocks_nx-spring-boot/1.0.0?

Yes exactly, as mentioned in the guide:

@nxrocks_nx-spring-boot is the clientId (i chose to replace the / by _, to avoid issue with the version part)
x.y.z is the clientVersion, and follows   the semantic versioning


fix: package seletor REGEX

From REGEX="Package ID: (.*), Label: $LABEL" To REGEX="Package ID: ([^,]*), Label: $LABEL"
REGEX default is "Greedy search"

Once change #138 is merged you'll need to do a rebase on the master branch to fix this issue.
You'll also need to sign off your commit following these instructions: https://github.com/hyperledger/fabric-samples/pull/135/checks?check_run_id=483880620

#138 was merged, please rebase on the tip of master


Added BPM-Analyser

Project URL
https://github.com/Luccifer/BPM-Analyser
Description
Little lib to help get BPM from audio files
Why it should be included to awesome-ios (optional)
No other libs to get BPM from audio-files
Checklist

[x] Only one project/change is in this pull request
[x] Addition in chronological order (bottom of category)
[x] Supports iOS 9 / tvOS 10 or later
[x] Supports Swift 3
[x] Has a commit from less than 2 years ago
[x] Has a clear README in English

          1 Error

:no_entry_sign:
Found 3 link issues

Link issues by awesome_bot

Line
Status
Link

2305
Error
http://www.sensiblecocoa.com  getaddrinfo: Temporary failure in name resolution

2552
Error
https://www.facebook.com/groups/designcode/  Network is unreachable - connect(2) for "www.facebook.com" port 443

2555
Error
https://www.facebook.com/groups/framerjs/  Network is unreachable - connect(2) for "www.facebook.com" port 443

  Generated by :no_entry_sign: Danger

Nice lib! Thanks for contributing! 🎉


Add workflow execution summary to trace

What was changed
Added workflow execution summary to trace command.
Depends on https://github.com/temporalio/cli/pull/233.
Example:

Why?
To provide important information about what workflow execution is traced. This could be extended to include other information as StartTime (or a diff with now).
Checklist

Closes texp upstream

How was this tested:
Running temporal (see example provided above)

Any docs updates needed?
None

@Sikian can you please rebase?

Recreated the PR from a new branch: https://github.com/temporalio/cli/pull/251


enhancement? add screenshot to readme

I concur, I'd like to know how it looks without installing

+1


Default support for the netip package

Now that netip is in the stdlib, it would be nice if go-cmp supported it by default. Today you need to add:
cmp.Comparer(func(x, y netip.Addr) bool { return x == y }),
cmp.Comparer(func(x, y netip.Prefix) bool { return x == y }),

to make cmp.Diff work with types that include netip types.

At present, cmp has no special treatment of any type in the Go stdlib, including time.Time, which instigated the invention of this package. I think this is a reasonable design goal as special treatment of any particular type begs the question of where to draw the line. Even if we limit cmp to just the stdlib, that's a lot of types that would be linked in.
I see a few options:

Have netip.Addr (and friends) implement an Equal method.
Provide a cmpopts.EquateShallow helper that shallow compares the types passed to it (e.g., cmpopts.EquateComparable(netip.Addr{}, netip.Prefix{})).
Propose a standard way for comparable types to self identify that their comparability is intentional (and guaranteed to not be lost in the future)
Have cmp stop panicking on unexported fields, and just compare fields recursively as it would otherwise.

At minimum, we should probably do option 2 as it makes this easier.


Pin the routes version

https://bugs.launchpad.net/neutron/+bug/1563157

Tested on UAT, appears to work, merging to get CI working again.

LGTM


Changing the style of the 'Send Message' button in the Contact Us page.

What's the style idea?
Currently the 'Send Message' button is not properly placed in the Contact Us page. It is overlapping with other sections. I will fix this issue. Please assign me this issue under GSSOC24. I will also make it's UI better for user.

Checklist

[X] I have checked the existing issues.
[X] I follow Contributing Guidelines of this project.
[ ] I am a IWOC'24 contributor.
[ ] I am a JWOC'24 contributor.
[ ] I am a SWOC'24 contributor.

Code of Conduct

[X] I agree to follow this project's "Code of Conduct".

Sir please assign me this issue.

Proceed @samruddhikurhe


fix(font-encoding): fix invalid characters on windows, fix #639

set default shell from cmd to powershell.[^1]
set fileendoing to use utf-8 only

All screenshots are taken in windows terminal
TODO:

icon overlapping (if anyone have this issue, just try another nerdfont)
icon won't show in toggleterm.nvim, it only works in the neovim built-in terminal.

Results:

[^1]: pwsh would be better, but it's not installed on windows by default. Only cmd and powershell are.

for the icon overlapping issue could be fixed if use wezterm as a workaround. it seems like a long term issue in windows terminal.

Wezterm results on windows:
Built-in terminal:

toggleterm:

Some it seems the remain issues are all about windows terminal...

Actually, It's a font issue. After trying other fonts from nerdfonts on windows terminal. There's no more icon overlapping!

But the icon still doesn't show in toggleterm in windows terminal...

I think it's better to use wezterm to save our sanity.

@CharlesChiuGit Sorry for the late reply. I just messed up my Windows 10 VM and did some testing. The known issues are:

cmd.exe requires some tweaks to fully support UTF-16LE (by setting codepage to a more appropriate value, which might need further discussion).
The flags for launching pwsh may need to be further adjusted to support command mode IO. Not tested yet.
Looks like those incorrect behaviors are all related to the terminal invoking nvim. This is not related to the configuration of nvim itself (well, except for the options: 'shell*')
The "icon overlapping issue" should be related to using incorrect font variants. AFAIK some terminal emulators will automatically adjust it, but most will not.

For the first three items, I'll propose a draft solution ASAP.
For the last item, manually installing all fonts ending in Mono Windows Compatible can address this problem on my VM (and on any terminal), probably b/c the mono variant of Nerd Fonts should be more suitable for supporting terminal icons (as its name "mono" suggests). e.g.:

Then use JetBrainsMono Nerd Font Mono as the font name.

@CharlesChiuGit Sorry for the late reply. I just messed up my Windows 10 VM and did some testing. The known issues are:

cmd.exe requires some tweaks to fully support UTF-16LE (by setting codepage to a more appropriate value, which might need further discussion).
The flags for launching pwsh should be further adjusted to support command mode IO. Not tested yet.
Looks like those incorrect behaviors are all related to the terminal invoking nvim. This is not related to the configuration of nvim itself (well, except for the options: 'shell*')
The "icon overlapping issue" should be related to using incorrect font variants. AFAIK some terminal emulators will automatically adjust it, but most will not.

For the first three items, I'll propose a draft solution ASAP.
For the last item, manually installing all fonts ending in Mono Windows Compatible can address this problem on my VM (and on any terminal), probably b/c the mono variant of Nerd Fonts should be more suitable for supporting terminal icons (as its name "mono" suggests). e.g.:

Then use JetBrainsMono Nerd Font Mono as the font name.

@CharlesChiuGit Sorry for the late reply. I just did some testings on my Windows 10 VM. The known issues are:

cmd.exe requires some tweaks to fully support UTF-16LE (by setting codepage to a more appropriate value, which might need further discussion).
The flags for launching pwsh should be further adjusted to support command mode IO. Not tested yet.
Looks like those incorrect behaviors are all related to the terminal invoking nvim. This is not related to the configuration of nvim itself (well, except for the options: 'shell*')
The "icon overlapping issue" should be related to using incorrect font variants. AFAIK some terminal emulators will automatically adjust this behavior, but most will not.

For the first three items, I'll propose a draft solution ASAP.
For the last item, manually installing all fonts ending in Mono Windows Compatible can address this problem on my VM (and on any terminal), probably b/c the mono variant of Nerd Fonts should be more suitable for supporting terminal icons (as its name "mono" suggests). e.g.:

Then use JetBrainsMono Nerd Font Mono as the font name. Not sure if this method can solve the problem on your end.

cmd.exe requires some tweaks to fully support UTF-16LE (by setting codepage to a more appropriate value [CCSID 850 by default], which might need further discussion).

I think supporting cmd is unnecessary, since I have every windows ship with windows powershell by default. We can just set it as default shell in the nvim config.

The flags for launching pwsh should be further adjusted to support command mode IO. Not tested yet.

Can u explain more on "command mode IO"? I not sure what do u mean. I can run most windows command with :!.

Looks like those incorrect behaviors are all related to the terminal invoking nvim. This is not related to the configuration of nvim itself (well, except for the options: 'shell*')

Ys, I follow the step in :h shell-powershell to set those things when the the detected OS is windows.

The "icon overlapping issue" should be related to using incorrect font variants. AFAIK some terminal emulators will automatically adjust this behavior, but most will not.

Ys, I also discovered that installing nerd-font via third-party package manager like scoop might cause issue (at least it looks different then downloading the ttf files directly from https://www.nerdfonts.com/) as I mentioned in https://github.com/ayamir/nvimdots/issues/639#issuecomment-1508496620.

For the last item, manually installing all fonts ending in Mono Windows Compatible can address this problem on my VM (and on any terminal), probably b/c the mono variant of Nerd Fonts should be more suitable for supporting terminal icons (as its name "mono" suggests). e.g.:

Ys, use the mono or the NFM variants did the same work on solving the icon overlapping issues. The result of mono variant is exactly the result in my above comment.

I think supporting cmd is unnecessary, since every Windows ships with windows powershell by default. We can just set it as default shell in the nvim config.

IMO we could vim.notify those users who haven't installed PowerShell on their machine? To avoid some extremely rare situations (such as running nvim on Windows Vista or accidentally uninstalled pwsh 🤣)

Can u explain more on "command mode IO"? I not sure what do u mean. I can run most Windows command with :!.

I mean we should further specify how PowerShell works - e.g., whether to start in non interactive-mode/read user-configuration/set useful environment variables, with the goal of simulating bash's -c flag.

Ys, I also discovered that installing nerd-font via third-party package manager like scoop might cause issue as I mentioned in #639 (comment). (at least it looks different then downloading the ttf files directly from https://www.nerdfonts.com/)

I just tested this and it seems feasible to install the mono variant directly:
scoop install JetBrainsMono-NF-Mono

btw, are the icons displaying in Windows Terminal (toggleterm) now? Looks like the byte string wasn't properly pasted to toggleterm. Try using pure ASCII:
Write-Host "`u{ebb1}"

actually "pwsh" is not the same as "windows powershell" lol

scoop install JetBrainsMono-NF-Mono

interesting! didn't knw that lol

btw, are the icons displaying in Windows Terminal (toggleterm) now? Looks like the byte string wasn't properly pasted to toggleterm. Try using pure ASCII:

Write-Host "`u{ebb1}"
will try once i got home.

I just tested this and it seems feasible to install the mono variant directly:
scoop install JetBrainsMono-NF-Mono

This works! Cool! It will install these two font family.

btw, are the icons displaying in Windows Terminal (toggleterm) now? Looks like the byte string wasn't properly pasted to toggleterm. Try using pure ASCII:
Write-Host "`u{ebb1}"

In windows terminal(pwsh):

In wezterm(pwsh):

I just re-install my win11, so I will test the above command later.

In windows terminal(nvim built-in terminal, pwsh):
TODO

In windows terminal(nvim toggleterm, pwsh):
TODO

In wezterm(nvim built-in terminal, pwsh):
TODO

In wezterm(nvim toggleterm, pwsh):
TODO

IMO we could vim.notify those users who haven't installed PowerShell on their machine - to avoid some extremely rare situations (such as running nvim on Windows Vista or accidentally uninstalled windows powershell 🤣)

Agreed.

I mean we should further specify how PowerShell works - e.g., whether to start in non interactive-mode/read user-configuration/set useful environment variables, with the goal of simulating bash's -c flag.

Got it, but I'm not sure how to test it lol. We can discuses more later.

This works! Cool! It will install these two font family.

Great! NL here probably stands for no ligature, AFAIU. IMO we can add this to the wiki.

Got it, but I'm not sure how to test it lol. We can discuses more later.

We're gonna need massive reference to about_PowerShell_exe and about_Pwsh  😄

In windows terminal(nvim built-in terminal, pwsh):

Updated. It seems fine using pure ASCII, no matter where u use it. But
Write-Host ("E492：不是编辑器的命令", "", "", "", "XXXXXXXXX") -Separator " || " is still not valid in windows terminal(nvim toggleterm, pwsh).

But <...> is still not valid in windows terminal(nvim toggleterm, pwsh).

Hmm maybe this is because those multi-byte icons were not properly pasted to toggleterm. Is pasting  (fa-lock) directly to the terminal supported? (i.e., is the icon displayed before execution?)

@CharlesChiuGit The following are all possible flags that we could further discuss:

-NoLogo - Hides the copyright banner at startup.
-ExecutionPolicy - Sets the default execution policy for the current session.

I prefer RemoteSigned -Scope Process -Force for maximum security & portability.

-Command - Executes the specified commands (and any parameters) as though they were typed at the PowerShell command prompt, and then exits, unless the NoExit parameter is specified.

I prefer New-Object and full module name for the script block:

- -Command [Console]::InputEncoding=[Console]::OutputEncoding=[System.Text.UTF8Encoding]::new();$PSDefaultParameterValues['Out-File:Encoding']='utf-8';
+ $OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding

[ ] -NoProfile - Does not load the PowerShell profile.
[ ] -NonInteractive - This switch is used to create sessions that shouldn't require user input. This is useful for scripts that run in scheduled tasks or CI/CD pipelines. Any attempts to use interactive features, like Read-Host or confirmation prompts, result in statement terminating errors rather than hanging.

I'm prone to add this flag since :! (bang) and system() are not interactive in neovim on *nix like systems.

LEGEND:

- should be added, but details need further discussion;
[ ] - whether to add this option requires further discussion.

@CharlesChiuGit The following are all possible flags that we could further discuss:

-NoLogo - Hides the copyright banner at startup.
-ExecutionPolicy - Sets the default execution policy for the current session.

I prefer RemoteSigned -Scope Process -Force for maximum security & portability.

-Command - Executes the specified commands (and any parameters) as though they were typed at the PowerShell command prompt, and then exits, unless the NoExit parameter is specified.

I prefer New-Object and full module name for the script block:

- -Command [Console]::InputEncoding=[Console]::OutputEncoding=[System.Text.UTF8Encoding]::new();$PSDefaultParameterValues['Out-File:Encoding']='utf-8';
+ -Command $OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding

[ ] -NoProfile - Does not load the PowerShell profile.
[ ] -NonInteractive - This switch is used to create sessions that shouldn't require user input. This is useful for scripts that run in scheduled tasks or CI/CD pipelines. Any attempts to use interactive features, like Read-Host or confirmation prompts, result in statement terminating errors rather than hanging.

I'm prone to add this flag since :! (bang) and system() are not interactive in neovim on *nix like systems.

LEGEND:

- should be added, but details need further discussion;
[ ] - whether to add this option requires further discussion.

https://github.com/ayamir/nvimdots/pull/659/commits/8282c80d8c330a6a96fbd932273e2234c9ae2046 cc @CharlesChiuGit If you have time to review & discuss this!

Sorry! I totally forgot to test this. Will test soon!

But <...> is still not valid in windows terminal(nvim toggleterm, pwsh).

Hmm maybe this is because those multi-byte icons were not properly pasted to toggleterm. Is pasting  (fa-lock) directly to the terminal supported? (i.e., is the icon displayed before execution?)

This works!

-NoLogo somehow doesn't work, the copyright banner still appears.
-ExecutionPolicy RemoteSigned -Scope Process -Force seems problematic, nvim process got shut down immediately after I open toggleterm, -ExecutionPolicy RemoteSigned -Scope Process cause the toggleterm to shut down immediately, but user will still remain in nvim.
-Command LGTM, but I also doesn't really know the syntax. The original one is from the nvim help :h shell_powershell.
NoProfile: I don't feel like adding it, b/c ppl might define aliases in their profile. Also, loading profiles is more like using shell in *nix-like OS.
-NonInteractive LGTM

I also notice sth. I'm not sure why, but current settings seem to increase noticeable startup time comparing to my original draft.
Maybe some api is a bit slower, will check some doc to analyze.

-NoLogo somehow doesn't seem to work, the copyright banner still appears.

It's actually working, notice that:
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/pscore6-docs
Type 'help' to get help.

have disappeared.

-ExecutionPolicy RemoteSigned -Scope Process -Force seems problematic, nvim process got shut down immediately after I open toggleterm, -ExecutionPolicy RemoteSigned -Scope Process cause the toggleterm to shut down immediately, but user will still remain in nvim.

Oh yes, I forgot toggleterm will also use these options. "nvim process got shut down immediately" is b/c, from the documentation:

Executes the specified commands (and any parameters) as though they were typed at the PowerShell command prompt, and then exits, unless the -NoExit parameter is specified.

We probably need to add -NoExit, but I'm not sure if the emulator will work properly in :! (shell bang).

NoProfile: I don't feel like adding it, b/c ppl might define aliases in their profile. Loading profiles is more like using shell in *nix-like OS.

Sounds very reasonable 😄 Will remove this flag.

Also this doesn't seem effective, nothing changed after I add this flag.

Maybe u are missing something from about_Profiles?

but current settings seem to increase noticeable startup time comparing to my original draft.

Hmm that's strange... The current settings work fine with PowerShell core on macOS, will test this issue on Windows later ;)

https://github.com/ayamir/nvimdots/pull/659/commits/e5485ee44c6a0513316ae354ad4c927bf58172e8

It's actually working, notice that:

I probably had already hidden it lol. my bad.

I think I found the reason why toggleterm shutdown immediately. -Scope Process is not valid in current version of pwsh, i think.

Maybe u are missing something from about_Profiles?

I think I have it in the right place?

And I found a bug. Current -Command $OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding won't display icons as icon, only in utf8 code. I think we need to change it back, until we got basecmd settled.

From wpsh -h, and about Scopes, I think we can't set -Scope as a flag to pwsh. -Scope seems to only used for limiting the scopes when using Get-Variable, not for limiting current pwsh process.

-NoExit will somehow output an extra line of weird raw utf8 code straing in :!.

I think I found why toggleterm will still output profile loading time and copyright, that's b/c we need to append the flags after vim.o.shell like so: shell = string.format("%s %s;", vim.o.shell, vim.o.shellcmdflag),. However, flags after -Command seems not been taken and it caused errors. Still need to research on that more.

The extra delay time seems to related to waiting for lsp to startup, not directly related to the PR.

After hours of debugging and reading the docs, I just don't think manual-imputed icons can be display correctly on toggleterm on windows terminal.
But everything just works fine in wezterm.
I think we need to cut the lost, we had already invested too much time in this trial font issue.

-NoExit will somehow output an extra line of weird raw utf8 code straing in :!.

This is exactly what I was concerned about before:

We probably need to add -NoExit, but I'm not sure if the emulator will work properly in :! (shell bang).

That "extra line" containing ANSI escape sequences is actually the user's PowerShell custom prompt. If -NoExit is not supplied PowerShell would exit right after the execution of :! with no trailing user prompt. Not sure if toggleterm will work properly if we ignore -NoExit.
Also, we should consider users who use powershell.exe, AFAIK powershell.exe provides fewer options than pwsh.exe. A proposed workaround:
local basecmd = "-NoLogo -MTA -ExecutionPolicy RemoteSigned"
local ctrlcmd = "-Command [console]::InputEncoding = [console]::OutputEncoding = [System.Text.Encoding]::UTF8"

The following configuration will handle the issue of toggleterm shutting down immediately right after the execution of -Command (tested on macOS, not sure if we also need to make this change on Windows):
vim.api.nvim_set_option_value("shellredir", '-RedirectStandardOutput %s -NoNewWindow -Wait', {})
vim.api.nvim_set_option_value("shellpipe", '2>&1 | Out-File -Encoding UTF8 %s; exit $LastExitCode', {})

I think we need to cut the lost, we had already invested too much time in this trial font issue.

💯💯 agreed. If the above proposal works properly, we should just merge this PR. As for the actual behavior of toggleterm (such as whether to ignore copyright information or not), users can customize this by setting shell = <...> in toggleterm.setup(). And since the :term feature of nvim was originally proposed to simulate system terminals, IMO displaying all these information happens to be the way native terminals work, out-of-the-box, so there is no need to make any changes to the appearance of the terminal buffer(s) 😃 The same applies to localized strings.

Not sure if toggleterm will work properly if we ignore -NoExit.

Ys, toggleterm would work without -NoExit.

Also, we should consider users who use powershell.exe,

That's reasonable. Agreed.

As for the actual behavior of toggleterm (such as whether to ignore copyright information or not), users can customize this by setting shell = <...> in toggleterm.setup().

Agreed.

local basecmd = "-NoLogo -MTA -ExecutionPolicy RemoteSigned"
local ctrlcmd = "-Command [console]::InputEncoding = [console]::OutputEncoding = [System.Text.Encoding]::UTF8"

vim.api.nvim_set_option_value("shellredir", '-RedirectStandardOutput %s -NoNewWindow -Wait', {})
vim.api.nvim_set_option_value("shellpipe", '2>&1 | Out-File -Encoding UTF8 %s; exit $LastExitCode', {})

Both of those new settings works fine on my win11.

A minor patch for consistent code style: https://github.com/ayamir/nvimdots/commit/f2f3c2c60b9768815fc28f47e53d5562b032065a.

I just observe some interesting things.
Even in wezterm, sth the icons won't show up too!
wezterm, toggleterm:

This often happens when I just opened wezterm, as if wezterm itself haven't been "warm up".
If I open a second wezterm tab, everything just work again.

Such a mystery...

@CharlesChiuGit I just ran into a similar problem, and after some investigation, this also seems to be an encoding issue b/c:

Pasting a glyph directly worked (i.e., let the terminal convert the encoding if necessary);
Providing an Unicode code point explicitly (via `u{x}) worked (i.e., let the terminal handle the encoding details);
These characters were not replaced by the "replacement character" � (*nothing was translated), indicating that they are interpreted as something else, probably functional keys (like ins) that won't be fed to PowerShell.

Perhaps this is indeed related to the way WezTerm (or other similar emulators), PowerShell, and PSReadLine handle the input stack. The reason why everything works fine except for the glyphs provided by NF is probably that they all have fixed representation. E.g.:
Windows          終端機                      <- Glyph representing PowerShell
|Latin| |CJK Unified Ideograph|   |Private Use Area|

After WezTerm "warms up", all components should be configured as UTF-8 compatible, so everything will work normally from that time on.

Cool! Let's really interesting! Thanks for your observation!


Unable to use @searchable with sort param, nextToken has wrong value

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?
yarn
If applicable, what version of Node.js are you using?
v17.0.1
Amplify CLI Version
4.50.0
What operating system are you using?
Mac
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
No update
Amplify Categories
api
Amplify Commands
Not applicable
Describe the bug
When calling the search query on a searchable model generated from the API, it inaccurately returns an empty array when sorting AND paginating. If I remove the sort param, it returns all the values as expected.
nextToken always return the value of the field I try to sort.
I have this schema:
type Brand @model
  @auth(rules: [
    { allow: public, operations: [read]},
    { allow: private, operations: [read]},
    { allow: groups, groups: ["Admin"], operations: [create, update, delete]}
  ])
  @key(name: "pushDate", fields: ["pushDate", "createdAt"], queryField: "brandByPushDate")
  @key(name: "votesNumber", fields: ["votesNumber", "createdAt"], queryField: "brandByVotesNumber")
  @key(name: "ByName", fields: ["name"])
  @searchable
  @aws_api_key
  @aws_cognito_user_pools
{

And I try to search with this query:
API.graphql({ query: queries.searchBrands, variables: { limit: 50, sort: { field: 'votesNumber', direction: 'desc' }}})

And on the result payload I get the nextToken value 0.0 (equal to the votesNumber value of the last element).
Expected behavior
I would like to obtain a unique nextToken.
And not the value of the field of the last element.
Reproduction steps

Add the schema
push the schema with amplify push
get the searchable query with sort and limit.

GraphQL schema(s)

type Brand @model
  @auth(rules: [
    { allow: public, operations: [read]},
    { allow: private, operations: [read]},
    { allow: groups, groups: ["Admin"], operations: [create, update, delete]}
  ])
  @key(name: "pushDate", fields: ["pushDate", "createdAt"], queryField: "brandByPushDate")
  @key(name: "votesNumber", fields: ["votesNumber", "createdAt"], queryField: "brandByVotesNumber")
  @key(name: "ByName", fields: ["name"])
  @searchable
  @aws_api_key
  @aws_cognito_user_pools
{
  id: ID!
  name: String!
  link: String
  image: String
  logo: String
  priority: Int
  votesNumber: Int
  bio: String
  deliveryReturnAddress: String
  deliveryDays: Int
  deliveryOfferFrom: Float
  isInSale: Boolean
  isInSaleAfterFlashSale: Boolean
  deliveryPrice: DeliveryPrice @connection(fields: ["deliveryPriceID"])
  deliveryPriceID: ID
  deliveryServices: [DeliveryServiceBrand] @connection(keyName: "byBrand", fields: ["id"])
  flashSaleDescription: String
  flashSaleStartDate: AWSDateTime
  flashSaleEndDate: AWSDateTime
  notificationDate: AWSDateTime
  topics: [BrandTopic] @connection(keyName: "byBrand", fields: ["id"])
  productTypes: [BrandProductType] @connection(keyName: "byBrand", fields: ["id"])
  votes: [Vote] @connection(keyName: "byBrand", fields: ["id"])
  flashSales: [FlashSale] @connection(keyName: "byBrand", fields: ["id"])
  products: [Product] @connection(keyName: "byBrand", fields: ["id"])
  pushDate: AWSDateTime
  createdAt: AWSDateTime! #important to keep it to have filter with keys
  updatedAt: AWSDateTime! #important to keep it to have filter with keys
}

export const searchBrands = /* GraphQL */ `
  query SearchBrands(
    $filter: SearchableBrandFilterInput
    $sort: SearchableBrandSortInput
    $limit: Int
    $nextToken: String
    $from: Int
  ) {
    searchBrands(
      filter: $filter
      sort: $sort
      limit: $limit
      nextToken: $nextToken
      from: $from
    ) {
      items {
        id
        image
        name
        bio
        votesNumber
        deliveryDays
        deliveryOfferFrom
        isInSale
        isInSaleAfterFlashSale
        deliveryPriceID
        productTypes {
          items {
            productType {
              name
              id
            }
          }
        }
        topics {
          items {
            topic {
              id
              name
              icon
              color
            }
          }
        }
        votes {
          items {
            id
          }
        }
      }
      nextToken
      total
    }
  }
`;

Log output

# Put your logs below this line

Additional information
No response

Hello @renanbronchart, Thank you for reaching out. On replicating the schema in my application I was unable to reproduce the behaviour of nextToken value being same as votesNumber. In order to dive deeper into the issue, could you please check your resolver mapping template?

Hello @renanbronchart, wanted to followup on this to see if you are experiencing any issues.

Hello @ykethan , what is the file of resolver mapping template please?

Hey @renanbronchart, I was able to get some additional information on this. Currently, v1 the @searchable directive uses Search After to perform pagination. causing it to return a value of the last item in the response.
in v1 the application uses util.toJson($context.args.nextToken) causing the return of last item.
in v2 it utilizes the util.base64Decode($args.nextToken) which returns a base64 token.
The issue seems to be similar to https://github.com/aws-amplify/amplify-cli/issues/3280

Furthermore, on additional testing I was able to replicate the null behaviour for search query in my AppSync API. I utilized the following in my application.

Added the following schema in my application:

type Post @model @auth(rules: [{ allow: private }]) @searchable {
  id: ID!
  title: String!
  createdAt: String!
  updatedAt: String!
  upvotes: Int
}

Performed amplify push
in my AppSync API logged in via Cognito authorization.
I first created a mutation to add a record.
On performing a Search Query immediately, I observed the data come in as null.
I added another record via mutation.
Ran the search query. I observed that my first record was displayed but not my second record.

example of mutation Query that I ran first.

Then immediately i ran the search query.

running the query again.
This appears to be a delay in the data prorogation to searchable instance created. rerunning the query will retrieve the data.

Hey @renanbronchart, following up to see if you are experiencing any issues.


"use custom marking algorithm" should always be turned on for extension parts

'extension' parts have no built-in marking algorithm, so the "use custom marking algorithm" and "extend base marking algorithm" options are redundant.

i want to do on this issue, please add me as a contributor


Proposed Recipes for ERA5

There are currently a few subset's of the ERA5 dataset on cloud storage (example but none are complete or updated regularly. It wont be a trivial recipe to implement with Pangeo-Forge but it would be a good stretch goal to support such a dataset.

Source Dataset

Link to the website / online documentation for the data: https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5, https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=overview
The file format: NetCDF/GRIB
How are the source files organized? One file per api request.
How are the source files accessed: Copernicus Climate Data Store, API Request (https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=form)
Any special steps required to access the data: access through the cdsapi or cdstoolbox apis

Transformation / Alignment / Merging

Most likely, the best way to access and arrange the data is in 1-day chunks, concatenating along the time dimension. Given the large user pool for this dataset, I would suggest this recipe does as little data processing as possible.
Output Dataset
One (or more?) Zarr stores. Hourly data for all available variables, all pressure levels, etc.

I was just talking about this with @spencerahill. We also had a meeting with ECMWF about this last spring.
It is a really big job.

Yes.  Context: 3-yr NSF CLD grant starting hopefully within next month or two, 6 mo / yr my time and hopefully next year a graduate student.  We're doing wavenumber-frequency spectral analysis of energy transports in low latitudes using ERA5.  So that requires 6-hourly or higher resolution, up to a dozen or so vertically defined variables.  Many TBs.
My default plan was to just use the CDSAPI to download it to local cluster but talking w/ Ryan sounds like this could plug into pangeo efforts nicely!  Which would be fun for me, having been mostly watching from the sidelines for a few years now :)

Here are some notes from Baudouin Raoult after our meeting last April

I would still like another virtual meeting with you, during which we present to you our data handling system, because I don’t think we have yet a common understanding of the work to be done, and discussing it in writting would be very inefficient. Below is a short list of things that comes to mind immediately:

The MARS system does not handle files, but provide a direct access to 2D fields using an hypercube-based index (https://confluence.ecmwf.int/download/attachments/45754389/server_architecture.pdf?api=v2 , a paper from 1996, before “data cubes” where popular 😊. The whole archive can be seen has a 12-dimensions single dataset of 300PB/3e+12 fields . A 2D fields is the smallest accessible object (think chunk), and users can access the archive along any dimensions (dates, parameters, levels,…). A lot of the data is on tape, so one cannot just scan a directory full of files. Any recipe will have to access MARS.
Most of the upper-air fields are in spherical harmonics, and the few upper-air grid-point fields are on a reduced gaussian grid. All surface fields are also on the reduced gaussian grid. These are not supported by CF/Xarray. The data needs to be interpolated to a regular lat/lon grid. This requires a non-trivial amount of time and resources.
We still have to address the issue of the two dimensions of time of forecasts fields (although this is not so bad with ERA5). Same issue for all accumulations, fluxes and other maximum (e.g. the ‘cell-methods’). That discussion has been going on from a couple of decades (see https://www.ecmwf.int/sites/default/files/elibrary/2014/13706-grib-netcdf-setting-scene.pdf, some slides I did in 2011 that illustrate most of the points I am making here)
We need to decide how to organise the dataset, not all variables having the same time frequency (although I think is may be OK with ERA5), and how much metadata are we ready to provide.
The full ERA5 is around 5 PB using 16 bits per values. The size may explode if we are not careful with the packing/compression of the target. There will be a lot of trial and error at the beginning, and this is something we cannot afford to much, considering the volumes involved.

Having done similar exercises in the past (such as delivering previous reanalysis to NCAR and other institutes), I think the “recipe” will have to run for many months to extract, interpolate, reformat, transfer and store the data. This needs to be done in parallel and having some checkpoint/restart facility as one cannot run a single process for months (planned system sessions, network issues, disks issues, etc). Furthermore, the data transfers will certainly be interrupted when our data centre move from the UK to Italy later this year.
So before we start writing some code, I would like that we have a clear definition of the scope of that project. We also need to decide who is going to be part of that activity and where the heavy lifting is happening

That was sufficiently intimidating that we tabled the discussion for the time being. Now that Pangeo Forge is farther along, and we have people who are interested in working on it, I think we can pick it up again.

Exciting! A few questions/comments:

Do I understand correctly that the proposed temporal extent for this recipe is 1950 - present, plus regular appending? If so, noting that appending is remains an open (but presumably resolvable) issue https://github.com/pangeo-forge/pangeo-forge-recipes/pull/81
I assume there is no way around moving ~26,240 (Jan 1, 1950 to present) daily NetCDFs to a cloud cache? That is, the CDS API probably won't work with https://github.com/pangeo-forge/pangeo-forge-recipes/pull/218 ?
Assuming we do need to cache the files, noting that the CDS API appears to diverge from the Pangeo Forge FilePattern assumption that datasets will be accessible via a human-readable URL. Instead, most of the identifying information about the requested dataset is encoded into a dictionary (e.g., here) which becomes a JSON POST request (👈 here, request is the user-defined dictionary).
I wonder if this API design pattern (dataset identifiers in JSON, rather than URL) is common enough to consider supporting it in pangeo-forge-recipes, or if it's preferable just to handle caching with some one-off scripts outside of Pangeo Forge, and then write the recipe to point to the cloud cache directly?
If we do attempt file transfer via Pangeo Forge, we'll also have to consider if/how fsspec can somehow be used as wrapper for CDS API requests.

To clarify, what Baudouin was proposing was that we go around the CDS API and talk directly to MARS, their internal archival system.

As a heads up, there is a full copy of the Copernicus 0.25 degree version of the ERA-5 Reanalysis dataset maintained at NCAR under the following dataset collections, which are preserved as time-series NetCDF-4 files:
https://doi.org/10.5065/YBW7-YG52
https://doi.org/10.5065/BH6N-5N20
These may be easier to access and stage to public cloud resources unless you need the raw spherical harmonics and reduced gaussian grids at model level resolution, which are only available through ECMWF MARS.  You can also access the NCAR maintained datasets by direct read from NCAR HPC systems as an NSF funded researcher. https://www2.cisl.ucar.edu/user-support/allocations/university-allocations

Cross-reference: #22

Question (which may reveal just how little I've worked with and understand the cloud): would it be useful to this effort to have some nontrivial chunk of the ERA5 data downloaded to the cluster at Columbia (berimbau) we're using for our project, to subsequently be uploaded to the cloud?  My big concern w/r/t tying my project's science tightly to this pangeo-ERA5 effort is our project's science potentially getting held up, maybe in a big way, if there end up being unforeseen delays etc. in getting the data onto the cloud.  Whereas I already have a functional pipeline for downloading the ERA5 data I'll need directly to that cluster via the CDS API, as well as the computational power I'll need at least for the preliminary analyses.
So, in this scenario, I'd start downloading the data I need basically right away to our cluster, and then once on the pangeo side things are ready I could upload from our cluster to the cloud.  The upside for pangeo of this direct transfer from us would be no waiting on the CDS system queue etc.
Thoughts?

would it be useful to this effort to have some nontrivial chunk of the ERA5 data downloaded to the cluster at Columbia (berimbau) we're using for our project, to subsequently be uploaded to the cloud?

Short answer: no, it would not be particularly useful for you to manually download data and store in on your cluster. That is sort of the status quo that we are trying to escape with Pangeo Forge. The problems with that workflow are

it is probably hard to reproduce because it involves manual steps
it may be hard to update, meaning that the data will eventually go stale

The goal with Pangeo Forge is to develop fully automated, reproducible pipelines for building datasets.
However, I recognize the tension here: you want to move forward with your science, and you need ERA5 data to do so. You can't wait a year for us to sort out all of these  issues.
Here is an intermediate plan that I think might balance the two objectives. You should write a Pangeo Forge recipe to build the dataset you need on berimbau. The process of writing this recipe will be very useful for the broader effort. Note that this won't be possible until https://github.com/pangeo-forge/pangeo-forge-recipes/pull/245 is done, since that will be required to get data from the CDS API.
@alxmrs has also been working on a "ERA5 downloader" tool which could be very useful here.. Alex is that released yet?
So a possible order of operations would be:

Spencer gets some basic familiarity with Pangeo Forge recipes by going through the docs / tutorials (feedback welcome)
In the meantime, Ryan finishes https://github.com/pangeo-forge/pangeo-forge-recipes/pull/24
Alex releases his ERA5 downloader tool
Spencer starts creating the recipe with a custom opener for downloading ERA5
Spencer runs this recipe on berimbau to generate the data he needs
We use this experience to inform the broader, more generic effort to bring ERA5 into the cloud

However, I recognize the tension here: you want to move forward with your science, and you need ERA5 data to do so. You can't wait a year for us to sort out all of these issues.

Exactly.
Thanks @rabernat.  That all makes sense.  I'm subscribed to the relevant pangeo repos now and in particular will keep an eye on https://github.com/pangeo-forge/pangeo-forge-recipes/pull/245.
And going through the docs/tutorials sounds like a great task when I'm procrastinating on a review / revision / etc. in the near-ish term future.

@spencerahill, once you've started on your recipe, please feel free to @ me in a comment here with any questions. The documentation is far from exhaustive, so don't be discouraged if there's something that doesn't make sense. I'll make sure any questions you have get answered, and we can use any insights we gain to improve the official docs.

Excellent, thanks!  Also IIRC you are at least sometimes in-person at Lamont(?) If so would be fun to meet+chat in person too

@alxmrs has also been working on a "ERA5 downloader" tool which could be very useful here.. Alex is that released yet?

Hey Ryan! The release is in progress – I have just submitted the codebase for internal approval. Not sure about the ETA, since we are in late December. Usually, this last part of the process takes about ~1 week.
As soon as it's public, I will post about it here.

at least sometimes in-person at Lamont(?)

Sadly I'm rarely there as I work out of a home office in California. Even if you sail through the recipe development process without any issues, I'd love to set aside some time to catch up over video either way. 😊

I'm happy to announce that the aforementioned tools to help download era 5 data are public, as of today! Please check out weather-dl at https://github.com/google/weather-tools.
@spencerahill @jhamman I'm happy to answer and questions you have along the way.
CC: @rabernat


Admonition Summary Tool

Description
Adds a tool that extracts all admonitions from the Kong documentation and allows you to view them by product or by type.
This can be useful to see at a glance what the most used admonitions are, or to find all warning blocks for a given product.
Testing instructions
cd tools/admonition-summary
npm ci
node run.js
open http://localhost:8000

Don't read the code 😬


js-interop: DateTime parsed as invalid if printed

I have a small interop class with moment.js
If I print the datetime passed to moment it fails to verify (only in dart2js)
Dart VM version: 1.21.1 (Fri Jan 13 01:50:41 2017) on "windows_x64"
See: https://github.com/pinnisi/moment-test
on Dartium I have:
true
true
true
on Chrome:
true
true
true
on Chrome with print(date); uncommented
false
true
true

This might be related to https://github.com/dart-lang/sdk/issues/28329


Modify options for darwin platforms

It is known that we need a different set of options for the darwin platforms as seen in:

https://github.com/protocolbuffers/protobuf/blob/v4.0.0-rc2/kokoro/release/ruby/macos/ruby/ruby_build_environment.sh#L27-L50
https://github.com/sparklemotion/nokogiri/blob/v1.11.0.rc3/scripts/setup-osx-native-builders#L34-L39

This patch backports the modifications above, and also fixes https://github.com/rake-compiler/rake-compiler/issues/177.

cross-ruby.rake is for building Windows binaries by Mingw-w64.
Why do you want to use it for macOS?

cross-ruby.rake is for building Windows binaries by Mingw-w64.
Why do you want to use it for macOS?

In fact, I'm not sure about the exact reason why google-protobuf, grpc, or nokogiri uses rake-compiler for non-Windows platforms. I made this patch only because I wanted to easily reproduce build environments for google-protobuf.
Nevertheless, I thought it's a common usecase because rake-compiler-dock provides builds for Linux.

I wanted to easily reproduce build environments for google-protobuf.

How about using https://github.com/protocolbuffers/protobuf/blob/v4.0.0-rc2/kokoro/release/ruby/macos/ruby/ruby_build_environment.sh#L27-L50 instead of using raw rake-compiler?

Nevertheless, I thought it's a common usecase because rake-compiler-dock provides builds for Linux.

I'm not sure about it's a common use case but cross-ruby.rake isn't for it.

How about using https://github.com/protocolbuffers/protobuf/blob/v4.0.0-rc2/kokoro/release/ruby/macos/ruby/ruby_build_environment.sh#L27-L50 instead of using raw rake-compiler?

Yeah, that's what I'm doing now. And it took a while to figure out we need the following additional patch:
-    'LDFLAGS=-pipe -s',
+    'LDFLAGS=-pipe',

So I made this PR to share the knowledge with whoever needs it.

I'm not sure about it's a common use case but cross-ruby.rake isn't for it.

I thought rake-compiler-dock shares philosophy with rake-compiler (because they're in the same org), but is it not?

Philosophy...?
I'm just the current maintainer not the original author but I don't have philosophy of this project... What do you want to say by it? "You should accept this patch"?
Anyway, we don't list the feature in our feature overview: https://github.com/rake-compiler/rake-compiler#feature-overview
If we add support for building different version Rubies for the current platform, we need to redesign and rename the current cross-ruby.task. It's not "cross-compiling".
We can't accept this patch because this approach is too ad-hoc. MINGW_TARGET =~ /darwin/ is strange. MinGW is "Minimalist GNU for Windows". It's not related to Darwin.

What do you want to say by it? "You should accept this patch"?

Depends on the answer. It's fine if rake-compiler is totally for MinGW only and rake-compiler-dock's Linux builds are only working by chance. Or if Linux is supported but darwin is out of support, then that would be fine too, although unfortunate to me. I wanted to clarify which of them are supported.

If we add support for building different version Rubies for the current platform, we need to redesign and rename the current cross-ruby.task. It's not "cross-compiling".

I agree that the current use-case is not a cross-compilation. However, I think supporting darwin hosts or darwin targets does sometimes imply cross-compilation because darwin runs on several ISAs.

We can't accept this patch because this approach is too ad-hoc.

I'm confused again. Then is there a chance such a change is accepted if cross-ruby.rake is refactored in a non-ad-hoc way? Like renaming MINGW_* into some other names and supporting both .exe and non-.exe files...

I'm currently adding support for MacOS to rake-compiler-dock. It mostly works already - I expect to have it ready when ruby-3.0.0 comes out. First I aim to support x86_64-apple-darwin, maybe later arm64-apple-darwin as cross targets.


Fix local_file target result unstable.

Backport #140 to v0.7
Flush before move to file from tempfile.
Fix https://github.com/tumugi/tumugi/issues/139

Coverage increased (+0.006%) to 92.971% when pulling d8dc48d1046ddca799f2a5b4b77399816781f19a on backport-140 into 2151c077ef32b6ed195cf028c955de0ed39b4d3c on v0.7.


Adaption for Asteroid

I am extending my video chat extension to work as an Asteroid (Alternative Meteor Client) mixin and at present, this package is a dependency. Is there any desire for this to be extended to work with Asteroid (I have done most of the work, I can create a PR if need be).

Here is the video chat extension for anyone interested https://github.com/elmarti/meteor-video-chat

A simple fix would be to inject the what is currently the Meteor object, which should be reusable verbatim by the Asteroid instance.

Tested on Meteor, still awaiting testing in the MeteorVideoChat Asteroid conversion. I get that this is a big change to the project, this could be released to NPM as a fork if you have no desire to merge it.

Closing due to lack of interest. If anyone needs advice on integrating this package with Asteroid, just ask.

@elmarti There are very useful contributions and discussions that go stagnant on here sometimes for a year or more. I wouldn't take the 7 days of inactivity as a sign its unwanted. There's a lot going on currently with the core team and I'm sure if you left this open it would get some attention at some point. I am also working on a project that implements Asteroid Node.js SDK for bot adapters. Currently it's using Asteroid v1 because that was already in use and I didn't have time to update and test with the v2 mixin for DDP. So I'd be keen to see examples of other Asteroid mixins when I come back to that update.

Ok great, I didn't want to persevere if there's no desire at all, in the mean time I dropped the client code straight into my extensions, but I guess I'll try and get this merged once it's ready :)