maintainState could have a clearer description

See https://github.com/felangel/bloc/issues/669 for sample and examples.
Today I spent a lot of time trying to figure out why maintainState = false was maintaining the state. Basically, if I open 5 widgets, mainTainState = true maintains all five, mainTainState = false maintains only the first and last, and Navigator.pushReplacement only the last.
The issue is that maintainState (from MaterialPageRoute) explicitly says:
https://github.com/flutter/flutter/blob/bcc93bca2388401d57d5cc2ba70c4a33bfde7066/packages/flutter/lib/src/material/page.dart#L22-L24
This isn't true. When maintainState = false and you are using push, it is not freeing everything. I recommend you update the description to say what it is doing.
➜ flutter doctor -v
[✓] Flutter (Channel dev, v1.10.16, on Mac OS X 10.15.1 19B88, locale en-BR)
    • Flutter version 1.10.16 at /usr/local/Cellar/flutter/dev
    • Framework revision 798e4272a2 (7 days ago), 2019-11-06 21:04:17 -0500
    • Engine revision 34a40cfbfd
    • Dart version 2.7.0

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at /Users/bernardoferrari/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling
      support)
    • Platform android-29, build-tools 29.0.2
    • Java binary at: /Users/bernardoferrari/Library/Application
      Support/JetBrains/Toolbox/apps/AndroidStudio/ch-1/192.5984562/Android
      Studio 4.0 Preview.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_212-release-1586-b4-5784211)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.2, Build version 11B52
    • CocoaPods version 1.8.3

[!] Android Studio
    • Android Studio at /Users/bernardoferrari/Library/Application
      Support/JetBrains/Toolbox/apps/AndroidStudio/ch-1/192.5916306/Android
      Studio 3.6 Preview.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    ✗ Android Studio not found at /Users/bernardoferrari/Library/Application
      Support/JetBrains/Toolbox/apps/AndroidStudio/ch-1/192.5916306/Android
      Studio 3.6 Preview.app/Contents
    • Try updating or re-installing Android Studio.

[!] Android Studio
    • Android Studio at /Users/bernardoferrari/Library/Application
      Support/JetBrains/Toolbox/apps/AndroidStudio/ch-1/192.5984562/Android
      Studio 4.0 Preview.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    • Dart plugin version 192.7402
    • Java version OpenJDK Runtime Environment (build
      1.8.0_212-release-1586-b4-5784211)

[✓] IntelliJ IDEA Ultimate Edition (version 2019.2.4)
    • IntelliJ at /Users/bernardoferrari/Applications/JetBrains Toolbox/IntelliJ
      IDEA Ultimate.app
    • Flutter plugin version 38.2.4
    • Dart plugin version 192.6459

[✓] VS Code (version 1.40.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.6.0

[✓] Connected device (1 available)
    • Android SDK built for x86 64 • emulator-5554 • android-x64 • Android 10
      (API 29) (emulator)

! Doctor found issues in 2 categories.

@shihaohong

The documentation on this should be clearer. Do you want to submit a PR to improve it?

From https://github.com/flutter/flutter/pull/51748 (closed PR) for future reference if anyone wants to work on this.

When maintainState is set to false on a particular route, other routes below it may still be kept in memory (e.g. because they may have maintainState set to true). And even if this route has maintainState set to false and is not the topmost route it may be kept in memory when the route above it is not opaque. See also https://master-api.flutter.dev/flutter/widgets/OverlayEntry/maintainState.html.


Could pagination use  instead of ?

This let user can use contextmenu "Open in newtab" or "Copy link url".

@hackwaly sounds good!
I'd like to support both; could you create a PR for this?


Invite to server button isn't updated after adding bot to a guild

Steps to replicate

Sign in on the dashboard
Click invite to server on a new guild
Add the bot to the server
Go back to the dashboard
The dashboard will still display the "Invite to server" button

Similar behavior when kicking the bot form a guild. The button should change  from "Manage server" to "Invite to server" when the bot is kicked from the guild and opening the control panel fails for the first time.

This is a side-effect of attaching the user's current guilds to the session when they log in and never refreshing them after. This is also why logging out and back in does update the dashboard. This was done to limit the number of Discord API requests which sometimes resulted in the backend being rate limited (#24).
To tackle this bug but also prevent the issue of rate limiting, the guilds should be updated each time a user loads the guild selection page. This prevents too many API requests being made but also ensures the latest information is shown at each time the user could possibly need it (there is no need to update the guilds when the user is on a guild's dashboard as they don't need other guilds other than the current one).

There are two solutions possible to fix this bug: update the joined property on each dashboard refresh OR do a total refresh of the list of guilds.
The second option was chosen as this also ensures that if the user joins a new guild, it will pop up on the guild selection page without having to reauthenticate. In addition, any changed guild data such as its name or icon will also be displayed properly.
However, by making this choice one has to be aware of the effect it will have on rate limiting. Refreshing the list of guilds on each page refresh is quite costly and if spammed, can run into rate limiting right away. Therefore, the additional measure was taken to have an expires field which defines the time the cache will expire and a complete refresh is necessary.
The current cache expiration is set to 5 seconds which prevents unnecessary rate limiting but also ensures that the most up to date information is shown up to 5 seconds old and thus will not have an effect on the user.


Reformat Models: Tag

This change is being split out from #420 and it:

Cleans up the model look a bit
Fixes the documentation header

@guidow, PTAL

Looks ok


Adobe ID

Please add Adobe ID to list

Hmm. I think something went wrong here... I don't see Adobe ID in this pull request--did you mean to open this as an issue?

I was trying to add Adobe ID to the list, per:
https://twofactorauth.org/  "Add your own favorite site by submitting a
pull request on the GitHub repo."
Not familiar with GitHub procedures, though...
ᐧ
--

Mike Shupp
w:  www.mikeshupp.com
♫   download.mikeshupp.com
subscribe: sign up for email list http://www.mikeshupp.com/email-list/
[image: http://www.facebook.com/MikeShuppPage]
http://www.facebook.com/MikeShuppPage [image:
https://twitter.com/MikeShupp] https://twitter.com/MikeShupp [image:
http://www.linkedin.com/in/mikeshupp]
http://www.linkedin.com/in/mikeshupp [image:
https://plus.google.com/u/0/b/106091738946393217068/106091738946393217068/posts]
https://www.google.com/+Mikeshupp [image:
http://www.youtube.com/user/MikeShuppOfficial]
http://www.youtube.com/user/MikeShuppOfficial
On Thu, Apr 30, 2015 at 7:14 PM, Seth Holloway<EMAIL_ADDRESS>wrote:

Hmm. I think something went wrong here... I don't see Adobe ID in this
pull request--did you mean to open this as an issue?
—
Reply to this email directly or view it on GitHub
https://github.com/jdavis/twofactorauth/pull/1100#issuecomment-97997213.

@mikeshupp you should not have made a pull request into gh-pages branch but rather into master.
Please open a new pull request into the correct branch.


Create VesselMover.netkan

Adds Vesselmover back to the CKAN.

Need to delete VesselMover.kerbalstuff as part of this.

Looks good, merging with my thanks.


GUI - Add daterange picker within drawer tabs

We would like to be able to show a daterange picker in the tabs view.

Notes
Step functionality as well as the chips (Dag, Uge, Måned, Kvartal, År) should be out of scope for now, since the component is currently on track to be implemented in Watt.
Tasks

[x] Add daterange picker to tabs view (should not be connected to table data yet)
[x] Feature flag chips.

Clarifications

Should the date selection carry over between tabs? Or should they be considered entirely separate?
Should the daterange picker have a default? Is it possible to search for all items (no date range)?

Acceptance Criteria

[ ] As a GUI user, I want to be able to filter my results based on a date range.

Out of scope
The chips is not a part of this story because the frontend team is making the functionality.

udeståender:
mangler at få changes fra datepickeren over på de andre faner
datepickeren viser ikke initial value korrekt, Bjørn kigger på problemet,
der er angivet nogle todo's på tests jeg mener der skal laves inde i dh-drawer-datepicker.component.spec.ts

October
T-001

Tab is showing as expected. Calenderfunctionn works. Out of scope ( another storie) functionality is first comming later,


chore: update generation workflow

In this PR:

Add a workflow, when generation_config.yaml is changed in a pull request, to generate changed libraries and push commits to the corresponding pull request.
Update workflow only to update googleapis commit.
Update workflow only to add library configuration to generation_config.yaml.

Follow up of https://github.com/googleapis/sdk-platform-java/pull/2616
For the goal of series of PRs, please refer to improvement proposal.

The workflows are tests in a forked repo:
https://github.com/JoeWang1127/google-cloud-java/pull/14 (update a library)
https://github.com/JoeWang1127/google-cloud-java/pull/13 (update googleapis commit)

Should we remove "Verify GAPIC client library generation" or rewrite it using hermetic build script?

Should we remove "Verify GAPIC client library generation" or rewrite it using hermetic build script?

I think we can remove it as the logic of "new client generation" is part of the hermetic build scripts now.

I think we can remove it as the logic of "new client generation" is part of the hermetic build scripts now.

I'll remove this check in another PR.


Fix deprecation warning from OrchestrationStack, remove unnecessary flatten

This fixes a deprecation warning I've been seeing in the dev.log file:
WARN -- : DEPRECATION WARNING: Passing an argument to force an association to reload is now deprecated and will be removed in Rails 5.1. Please call `reload` on the result collection proxy instead. (called from retirement_check at /Users/dberger/Repositories/manageiq-djberg96/app/models/orchestration_stack/retirement_management.rb:8)

I also removed the .flatten call since I can see no way that could be a nested array.

👍

The previous version brings all the records back.
Could we instead use a subquery:
def retirement_check
  ems_ids = MiqServer.my_server.zone.ext_management_systems.select(:id)
  table   = OrchestrationStack.arel_table
  stacks  = OrchestrationStack.where(:ems_id => ems_ids)
                              .where(table[:retires_on].not_eq(nil).or(table[:retired].eq(true)))
  stacks.each(&:retirement_check)
end

The previous version brings all the records back.
Could we instead use a subquery:
def retirement_check
ems_ids = MiqServer.my_server.zone.ext_management_systems.select(:id)
table   = OrchestrationStack.arel_table
stacks  = OrchestrationStack.where(:ems_id => ems_ids)
.where(table[:retires_on].not_eq(nil).or(table[:retired].eq(true)))
stacks.each(&:retirement_check)
end

@kbrock Maybe we could do this in a follow up PR?  It's mostly unrelated to the deprecation.

FWIW @kbrock the explain plan looks identical to me either way:
stacks1 = OrchestrationStack.where(table[:retires_on].not_eq(nil).or(table[:retired].eq(true)).and(table[:ems_id].in(ems_ids)))

stacks2 = OrchestrationStack.where(:ems_id => ems_ids).where(table[:retires_on].not_eq(nil).or(table[:retired].eq(true)))

Output:
> stacks1.explain
...
Seq Scan on orchestration_stacks  (cost=0.00..14.69 rows=1 width=491)
   Filter: (((retires_on IS NOT NULL) OR retired) AND (ems_id = ANY ('{3,6,7,4,5,1,2}'::bigint[])))

> stacks2.explain
...
Seq Scan on orchestration_stacks  (cost=0.00..14.69 rows=1 width=491)
   Filter: (((retires_on IS NOT NULL) OR retired) AND (ems_id = ANY ('{3,6,7,4,5,1,2}'::bigint[])))

Maybe we could do this in a follow up PR? It's mostly unrelated to the deprecation.

👍

@kbrock I do not understand all your comments about this PR and its focus on fixing a deprecation.  Please elaborate or explicitly state you are discussing a future PR.

thanks - yes, unrelated

@djberg96 the ems_id line changed - it no longer had the collect. But locally, I'm not sure if running as 1 vs 2 queries is that much different.
Looks like were missing a few indexes here

@kbrock true, we could remove that .collect call, too. How's the explain plan look if we add an index (temporary or otherwise) to ems_id?

That index ext_management_systems(zone_id) helps with the first line:
MiqServer.my_server.zone.ext_management_systems.select(:id)

Since it is a common pattern, we may want to add an index on zone_id to all models accessed via my_server.zone.X. On my system (and most devs?) the ems table is small enough that it will always do a sequential scan, even with an index:
 Seq Scan on ext_management_systems  (cost=0.00..1.01 rows=1 width=8)
   Filter: (zone_id = 1::bigint)

removing the collect up top produces the following plan:
explain analyze SELECT "orchestration_stacks".*


Blocking paths/files inside an exported path

{
  "name": "foo-bar",
  "exports": {
    "./timezones/": "./data/timezones/",
    "./timezones/private.js": false,
    "./timezones/private/": false
  }
}

Has this type of functionality been discussed/considered?  The idea is this would allow import of foo-bar/timezones/edt.js but forbid import of foo-bar/timezones/private/edt.js or foo-bar/timezones/private.js.  Even if not documented I wouldn't be surprised to see people try to do this or using "./timezones/private/": "./suppress-direct-import-of-privates-with-invalid-path/".

I think it was considered but not in the concrete sense that those are private modules.
That said, and while I completely dig the simplicity and clarity, I feel more inclined to think of this out of immediate scope (ie worth discussing in due time if so) — Sorry for being blunt, but that is my personal take, and others might prove me completely off here :)

By specifying an exports map, every file is already implicitly false - so you can achieve what you want merely by not listing those entry points.

Package-private modules are definitely something that would be nice to offer. Even if they are currently out of scope, I feel that they could belong to the package export config: it describes the public interface of the package.
A common situation where I wish it was supported is when I have re-exports in my lib package and my IDE uses the wrong imports.
I think that a even simpler design would be to use the config as a whitelist: only listed packages would be allowed (edit: I see that @ljharb posted the same idea). Maybe a flag could be added to ask Node to check this policy during the import?

@demurgos it’s not an idea, it’s how the proposal already works

@ljharb the value "./timezones/": "./data/timezones/" causes everything within data/timezones/ to be exposed under foo-bar/timezones/**.  So import something from 'foo-bar/timezones/private.js' would be allowed by that exports key, the question here is about the ability to block specific paths within a folder that is exported.

aha, my apologies. I think then that that should still work as is, because the value you pass on the RHS can’t be resolved.

You could also put a package.json in that folder that had its own exports map, I’d think?

I would assume that the behavior will align with the approach taken by import maps (once that's more stable). This means looking at the situation today:
{
  "name": "foo-bar",
  "exports": {
    "./timezones/": "./data/timezones/",
    "./timezones/private.js": null,
    "./timezones/private/": null
  }
}

But just in terms of project maintenance, mixing private files inbetween things that are publicly accessible seems like asking for trouble. So I'm not sure we should prioritize it at this point.

@jkrems what does import maps do for null mappings?

It's an alternate syntax for [] ("no mapping"): https://github.com/WICG/import-maps#denying-access-to-a-built-in-module

Interesting, I guess we could support it similarly then, if/when this is demonstrated to be a stable feature.

I think this will come up when we add support for arrays in general since then it falls out pretty natural. But for that I want to punt on what exactly the fallback semantics end up being for import maps (e.g. if multiple arbitrary URLs can be used for example).

@jkrems I agree that it's probably best to avoid structuring a project as I suggested in this ticket but it will happen.  My hope here is to eliminate the edge case where projects might decide to use potentially unspecified behavior to suppress parts of an exported directory.

Empty arrays should work as a way to selectively disallow/un-export files inside of an exported directory. Adding support for the null sugar would at most be a minor cosmetic change.

@jkrems it's a useful thing to consider, though - have we specified what happens with a non-array, non-string value?

non-array, non-string values will be ignored in the current implementation. We're trying to define a syntax where bundlers and/or future versions of node could use them for advanced features like differential mappings based on the environment / feature support. So to allow package owners to publish backwards-compatible packages, we have to completely ignore those (reserved) values.


[TC-DM-4.*] Need YAML support for access XPAN ID form OperationalDataset

Summary:
How do we access XPAN ID form OperationalDataset’s need to save the value as "Userth_op"
Also need way to check if a specific value is present in the "ServerList" attribute
Affecting following tests:
1.TC-DM-4.6
2.TC-DM-4.10
3.TC-DM-4.12
4.TC-DM-4.8
5.TC-DM-4.16
6.TC-DM-4.7
Testplan screenshot:

Spec screenshot:

@manjunath-grl - Can you tag your previous issues with your new ID so folks don't ignore the tickets please

This issue was created by me using a account that i had to delete due to access issues, I will continue to follow these issues with my current id.


Create Haul Webpack Resolver for eslint-plugin-import

Currently, it's impossible to use eslint-plugin-import together with Haul. We should fix this by introducing our very own resolver.
If no one is super interested in I will do it.

@jukben Grab one I've done some time ago for Jest: https://github.com/facebook/jest/blob/master/eslint_import_resolver.js

Would be even nicer to make it npm module and use it in both projects 👍

Shall we keep it in Haul's repo or create separate one?

I'm fine with copying this resolver and use here, but if we want to make it a package, then a separate repo

@jukben can you pick this up?

@thymikee I've been a little bit busy this week but I'll look into this ASAP.

Cool, no rush!

we don't even need a separate package. could do haul/eslint-import-resolver or similar


[WIP] Create initial swift/ & objc/ content files and initial test CI

Create initial contents for new onnxruntime-swift-package-manager repo

Create and pull initial objective-c header files and SPM support from ORT main repo : rel-1.15.0 branch.

Package.swift file is in sync with rel-1.15.0version

Set up initial azure CI test pipelines
Initial README.md
Test swift project: https://github.com/YUNQIUGUO/ORT-SPM-Test-App

Created a simple swift app via Xcode and tried to add Package dependency from this new repo and consumes OnnxruntimeBindings target in the test app. Looks to work.
More screenshots in the test repo about how swift user will add this new repo in their project.

Need to see how we want to sync the ORT main objc/swift files later and the release process

See how to set up jenkins or github actions, etc. to monitor changes from objc files in ORT main repo and automatically syncing the changes to this separate repo.
Or manually pull changes from main ORT repo in a period of time/before release.

TODO:  - copy all the header files from rel-1.15.0 branch
-  run tests for Package.swift in two configurations
-  1. Set ENV and testing the latest (upcoming)
-  2. Test latest release version (doesn't break the current)

thanks for all the valuable comments! will try to address them all.

The first pull request is not configured as squash and commit
I just elevated my repo manage access and changed the settings for pull requests here. Hopefully starting from the next one, it will all be squashed and then check-in as one commit.


set up @embroider/test-setup

Totally missed that this is part of the Ember v3.26 blueprint already. Done!


Branch bug fix edit order

Fix ordering of shows after edit. Issue 215

Codecov Report

Merging #253 (d349c30) into master (b54530d) will decrease coverage by 0.67%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master     #253      +/-   ##
============================================
- Coverage     47.21%   46.54%   -0.68%
+ Complexity      368      364       -4
============================================
  Files            91       91
  Lines          1724     1721       -3
  Branches        193      193
============================================
- Hits            814      801      -13
- Misses          842      852      +10
  Partials         68       68

Impacted Files
Coverage Δ

...va/seedu/trackermon/logic/commands/AddCommand.java
100.00% <ø> (ø)

src/main/java/seedu/trackermon/model/Model.java
100.00% <ø> (ø)

...main/java/seedu/trackermon/model/ModelManager.java
48.33% <0.00%> (-17.25%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b54530d...d349c30. Read the comment docs.


update new zuclopenthixol measure

pushed commit to master branch in error. @sebbacon  has reverted and I have now changed title and creating a new branch for a new zuclopenthixol measure supporting a new outlier alert

Closing in favour of #2142.


Add basic support for YugabyteDB.

Added support for YugabyteDB by extending the existing API implementation of PostgreSQL.
APIs overridden in YugabyteDB:

Database.ensureSupported()
Database.supportsDdlTransactions() - YugabyteDB does not support it (even though it does not throw an error if a DDL is run inside an explicit transaction).
BaseDatabaseType.handlesJDBCUrl()
BaseDatabaseType.handlesDatabaseProductNameAndVersion()
BaseDatabaseType.getPriority()
BaseDatabaseType.getName()

No need to update pom file to add maven coordinates of YugabyteDB-specific driver, since YugabyteDB works with PostgreSQL driver which is already shipped with Flyway distribution.
Tested the changes with commands: info, migrate and clean - as outlined in this section.
Screenshot of the test run via IDE is available here.

I'll close this PR and open another one from yugabyte org and incorporate the couple of suggestions.


Improve Istio security (mTLS, jwt, RBAC, etc.) debuggability

We have seen many users asking for help about why does Istio auth policy (mTLS, jwt, etc) doesn't work or not working as expected in their environment. Usually we have to look deep into the policy applied, pilot logs, envoy logs and the actual request being sent to determine the root cause (e.g. policy misconfiguration, bug in the code, broken environment, expired cert, etc.)
Sometimes, this is a very time consuming process even for expert Istio developers, not mention for the normal users.
We should look into this area and improve the debuggability of Istio security features which would benefit both normal users and developers. This issue is used to track efforts in this area. Feel free to comment your thoughts here.
Related issue to improve error message for TLS: #11017
/cc @costinm @duderino @liminw @rshriram @mandarjog

We have a CLI command, istioctl authn tls-check <pod> that is supposed to report conflicts.  However, it doesn't always do what is expected.  For example, if a namespace is set up for mTLS but not for auto-inject the tool doesn't detect it. https://github.com/istio/istio/issues/11697
I have an idea for a CLI command to check pod connectivity.  The proposal is to automate the steps I take manually when diagnosing a problem.  I would like feedback. https://docs.google.com/document/d/1yoq1RLJbfljXfRmbaV4HAVoog1LzHgeHkbbR9wdKRW8/edit
There is an Istio configuration "linter" project at https://github.com/aspenmesh/istio-vet

/cc @incfly

/cc @wattli

Great work and it's really important if we can do it right. @yangminzhu /@incfly, does Ed's summary match what's in your mind?

/cc @nishanth-pinnapareddy

@wattli The situation (pod without sidecar) in Ed's comment could be improved with my ongoing PoC work. The other idea (checking connectivity) mentioned is not related to my work as I'm currently more focused on checking the static config first.

My command is a good start.  We also need to restore the RBAC tool that was removed for 1.1 because it didn't work.  https://github.com/istio/istio/issues/12075
It would also be good to understand why openssh cannot be used to debug Istio when mTLS is on (see https://github.com/istio/istio/issues/12513#issuecomment-474082391 )

While I find checking connectivity is a nice idea, I feel it is a challenge to implement it right. At the same time, it may somewhat overlapped with other add-ons (like Kiali and Grafana), which have graphical UI and present only the interested connections based on real traffic. For the command line tool, may be we should focus on the mechanism to investigate the source policy and the actual configs that envoy received. Of course, it would be great if we have all of these :), just my 2c to prioritize.

@diemtvu Kiali and Grafana only show working connections.
The problem isn't always the configuration Envoy receives.  It could be KubeDNS crashing, or an application that doesn't bind to localhost, or Network Policy.  The problem could be Envoy configuration, but unrelated to Istio CRDs, perhaps all the Citadels have crashed and the certs have expired.
When there is an unexpected outage it will be very stressful.  Anything we can do to automate some of the troubleshooting steps will help.  It may be that our CLI is not the right place for connectivity validation tools.  Let me turn the question around.  When you have a 503, do you have any kind of checklist or procedure you use to identify the problem?

To give a quick update on this, we were busy with a lot of other higher priority works (vulnerability issue, code mauve, 1.2 release, etc.) and didn't work on this actively so far.
We did check-in some prototype change in istioctl experimental auth with the target to make it easier to inspect and check security policy, but there is much more to do than this. We will have more time to work on this once we get 1.2 out, which is likely going to happen this month.

May I close this because of the output of istioctl x describe pod? and describe service?

@esnible Let's close this issue as istiocto x describe has already been in the release.


Fix flexbox layout of the news section on GPP for safari.

Fixes https://github.com/artsy/force/issues/253
The row on GPP may contains 0-2 sections. Let's simplify the layout by setting percentage to the flexbox-basis for each section. This also provides us better responsiveness.
Screenshots
With About and News

With only About

With only News (hide the title when if only 1 entry)

          1 Warning

  :warning:
  Please assign someone to your PR

  Generated by :no_entry_sign: danger

Thanks @starsirius, nice solution.


Include files not copied on install

Ds is not usable from other PHP extensions, because the header files are not copied to the PHP include directory.
This should be done using PHP_INSTALL_HEADERS in config.m4 and config.w32.

The following should work, though it can probably be done nicer.
It also puts everything in include/ext/ds/src/. It doesn't seem like it possible to set a custom base dir with the PHP_INSTALL_HEADERS macro.
PHP_INSTALL_HEADERS(ext/ds, [
    php_ds.h
    src/common.h
    src/ds/ds_deque.h
    src/ds/ds_htable.h
    src/ds/ds_map.h
    src/ds/ds_priority_queue.h
    src/ds/ds_queue.h
    src/ds/ds_set.h
    src/ds/ds_stack.h
    src/ds/ds_vector.h
    src/php/arginfo.h
    src/php/parameters.h
    src/php/classes/php_collection_ce.h
    src/php/classes/php_deque_ce.h
    src/php/classes/php_hashable_ce.h
    src/php/classes/php_map_ce.h
    src/php/classes/php_pair_ce.h
    src/php/classes/php_priority_queue_ce.h
    src/php/classes/php_queue_ce.h
    src/php/classes/php_sequence_ce.h
    src/php/classes/php_set_ce.h
    src/php/classes/php_stack_ce.h
    src/php/classes/php_vector_ce.h
    src/php/handlers/php_common_handlers.h
    src/php/handlers/php_deque_handlers.h
    src/php/handlers/php_map_handlers.h
    src/php/handlers/php_pair_handlers.h
    src/php/handlers/php_priority_queue_handlers.h
    src/php/handlers/php_queue_handlers.h
    src/php/handlers/php_set_handlers.h
    src/php/handlers/php_stack_handlers.h
    src/php/handlers/php_vector_handlers.h
    src/php/iterators/php_deque_iterator.h
    src/php/iterators/php_htable_iterator.h
    src/php/iterators/php_map_iterator.h
    src/php/iterators/php_priority_queue_iterator.h
    src/php/iterators/php_queue_iterator.h
    src/php/iterators/php_set_iterator.h
    src/php/iterators/php_stack_iterator.h
    src/php/iterators/php_vector_iterator.h
    src/php/objects/php_deque.h
    src/php/objects/php_map.h
    src/php/objects/php_pair.h
    src/php/objects/php_priority_queue.h
    src/php/objects/php_queue.h
    src/php/objects/php_set.h
    src/php/objects/php_stack.h
    src/php/objects/php_vector.h
])

I'm not sure if we should be installing all headers this way. I think we should consider this as a priority in 2.0 to have a public internal API that is designed with other extensions in mind. This is solid though - could you please PR this and confirm it works? Happy to merge.

@jasny I'd like to consider this specifically as part of v2 - I'm not comfortable to expose this currently because I didn't consider it when I was writing 1.x. Nice to know about PHP_INSTALL_HEADERS :+1:


Unpublished package: Correct way to remove it from index?

Hi folks 👋
Over at our repository we got an issue https://github.com/gatsbyjs/gatsby/issues/30058 that points to https://github.com/Burnsy/gatsby-plugin-ackee-tracker/issues/69. Reading https://github.com/algolia/npm-search/issues/556 I understand that normally the deletion on npm should account for that.
Is it still in the index because yarnpkg has it? https://yarnpkg.com/?q=gatsby-plugin-ackee
npm doesn't have it: https://www.npmjs.com/search?q=gatsby-plugin-ackee
Would deleting/unpublishing on yarnpkg resolve the issue? I want to spare y'all some work and get it done by robots 🤖 as manually asking for deletions doesn't scale.
Thanks!

I've deleted that package now manually, although it should have been done automatically following this:
https://github.com/algolia/npm-search/blob/b40c6b21aa7d354c7e0fa938651916e52d132436/src/watch.js#L187-L195
I haven't had time to debug this yet, but I saw that not every package on npm actually gets deleted. In the past we did a full reindex every week or two, but that became much too slow (the full reindex takes >1 week now, during which new packages don't get processed). If you could debug what's going on here, I'd gladly get you up to speed with the code base and credentials!

Thanks for the very quick answer, much appreciated!
While I'd love to dig into this and help resolve it, we're quite busy with our v3 release from last week and it's not a high priority at all for us. When we revisit that plugins page of our website it might come up again, but until then feel free to close this issue or keep it open (& not prioritized).


DEV-14765 Upgrade and pin terraform-aws-modules/key-pair/aws

https://datatamr.atlassian.net/browse/DEV-14765

This PR doesn't appear to be linked to a DevOps/SRE jira ticket

This PR doesn't appear to be linked to a DevOps/SRE jira ticket

This PR doesn't appear to be linked to a DevOps/SRE jira ticket

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Errors with GATK calling on exomes

Hi,
I've been trying to run GATK variant calling on 250 whole-exome samples, and I keep running into errors. I get several error messages like the one below in the log files, and the job stops logging at some point, but the processes keep running for some days afterwards, although they don't seem to produce output.
I've tried increasing memory allocation (since it's an OutOfMemoryError). I've currently got jvm_opts: ["-Xms1g", "-Xmx32g"], and the problem persists. Any help would be much appreciated!
06:55:13.193 INFO  HaplotypeCaller - Shutting down engine
[August 22, 2018 6:55:13 AM AEST] org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller done. Elapsed time: 794.88 minutes.
Runtime.totalMemory()=7265714176
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at org.broadinstitute.hellbender.tools.walkers.genotyper.GenotypeAlleleCounts.alleleCountsByIndex(GenotypeAlleleCounts.java:514)
        at org.broadinstitute.hellbender.tools.walkers.genotyper.afcalc.GeneralPloidyExactAFCalculator.computeLofK(GeneralPloidyExactAFCalculator.java:276)
        at org.broadinstitute.hellbender.tools.walkers.genotyper.afcalc.GeneralPloidyExactAFCalculator.calculateACConformationAndUpdateQueue(GeneralPloidyExactAFCalculator.java:187)
        at org.broadinstitute.hellbender.tools.walkers.genotyper.afcalc.GeneralPloidyExactAFCalculator.fastCombineMultiallelicPool(GeneralPloidyExactAFCalculator.java:148)
        at org.broadinstitute.hellbender.tools.walkers.genotyper.afcalc.GeneralPloidyExactAFCalculator.combineSinglePools(GeneralPloidyExactAFCalculator.java:112)
        at org.broadinstitute.hellbender.tools.walkers.genotyper.afcalc.GeneralPloidyExactAFCalculator.computeLog10PNonRef(GeneralPloidyExactAFCalculator.java:25)
        at org.broadinstitute.hellbender.tools.walkers.genotyper.afcalc.AFCalculator.getLog10PNonRef(AFCalculator.java:33)
        at org.broadinstitute.hellbender.tools.walkers.genotyper.GenotypingEngine.calculateGenotypes(GenotypingEngine.java:255)
        at org.broadinstitute.hellbender.tools.walkers.genotyper.GenotypingEngine.calculateGenotypes(GenotypingEngine.java:210)
        at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerGenotypingEngine.assignGenotypeLikelihoods(HaplotypeCallerGenotypingEngine.java:150)
        at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerEngine.callRegion(HaplotypeCallerEngine.java:565)
        at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller.apply(HaplotypeCaller.java:218)
        at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:295)
        at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:271)
        at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:893)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:134)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:179)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:198)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
        at org.broadinstitute.hellbender.Main.main(Main.java:289)
' returned non-zero exit status 1

Bennet;
Sorry about the issues and thanks for the report. For the error you're reporting here, you're asking for 32g of memory per process which should be more than enough for this many samples, although I haven't run larger pooled variant calling with GATK4 recently since we're instead using joint calling for larger samples. A couple of questions/suggestions:

Is joint calling with gVCF intermediates an option? That should be more memory efficient approach as sample sizes increase (https://bcbio-nextgen.readthedocs.io/en/latest/contents/pipelines.html#population-calling)
What version of GATK are you running for this?
How are you running this? Is it distributed on a cluster or otherwise allocated with respect to resources? One potential issue might be that you're running out of memory due to overlapping allocation between processes. It would help to know what type of machines you run on (cores and memory) to understand how a 32Gb process allocates.

Hope this helps some with debugging.

Hi Brad,
Thanks for the suggestion - we'll give joint calling a go. This run is actually for a PhD student in the group I support, who's getting desperate for results, so I hope this solves the issue!
We're running on a single machine with 62 cores and 512Gb of RAM, which shares storage with a few other machines as part of a cluster. A somewhat idiosyncratic setup, and we'll be moving to a more standard HPC cluster soon, but that's what we've got for now. Running GATK <IP_ADDRESS>.


MultiZarrToZarr.translate: TypeError: shape is None from zarr

I want to use kerchunk to open climate model netcdf output stored 1 or 12 timestep/s per file in many files as one huge zarr.
I do not care about the cloud capabilities. I just need to work this out locally.
I do not want to convert nc files to zarr, because I dont have the storage to keep both and need to keep nc.
Therefore I thought kerchunk could help me.

I wrote a small example below but it fails with an zarr shape error which I do not understand. I get SingleHdf5ToZarr running, but MultiZarrToZarr fails. Does anyone see a fix for this? Is this example supposed to work?
import os
import zipfile
import kerchunk.hdf
import fsspec
import json
import xarray as xr
import numpy as np

# dummy data
ds = xr.DataArray(name='var', data=np.random.random((100,100)), dims=['lon','lat'], coords={'lon':range(100), 'lat':range(100)}).to_dataset()#.expand_dims('time')
urls =[f'test_{y}.nc' for y in range(2000,2011)]
for u in urls:
    ds.to_netcdf(u)

so = dict(
    anon=True, default_fill_cache=False, default_cache_type='first'
)
with zipfile.ZipFile("out.zip", mode="w") as zf:
    for u in urls:
        print(u)
        print(os.path.exists(u))
        with fsspec.open(u, **so) as inf:
            h5chunks = kerchunk.hdf.SingleHdf5ToZarr(inf, u, inline_threshold=100)#, mode='r')
            with zf.open(os.path.basename(u) + ".json", 'w') as outf:
                outf.write(json.dumps(h5chunks.translate()).encode())

from kerchunk.combine import MultiZarrToZarr
mzz = MultiZarrToZarr(
    "zip://*.json::out.zip",
    remote_protocol="file",
    xarray_open_kwargs={
       # "preprocess": None,#drop_coords,
        "decode_cf": False,
        "mask_and_scale": False,
        "decode_times": False,
        "decode_timedelta": False,
        "use_cftime": False,
        "decode_coords": False
    },
    xarray_concat_args={'dim':'time'}
)
#test_dict = mzz.translate()

# open single json as zarr
test_dict='test_2000.nc.json' # after unzip out.zip
m = fsspec.get_mapper('reference://', fo=test_dict, remote_protocoll='file')
print(xr.open_dataset(m, engine='zarr')) # works

mzz.translate("output.zarr") # see Traceback below

# This can also be written as a json
mzz.translate("output.json") # same Traceback

Output
(xr) aaron.spring@MacBook-Pro:~/Coding/kerchunk/my_kerchunk$ python run.py
test_2000.nc
True
test_2001.nc
True
test_2002.nc
True
test_2003.nc
True
test_2004.nc
True
test_2005.nc
True
test_2006.nc
True
test_2007.nc
True
test_2008.nc
True
test_2009.nc
True
test_2010.nc
True
/Users/aaron.spring/Coding/kerchunk/my_kerchunk/run.py:49: RuntimeWarning: Failed to open Zarr store with consolidated metadata, falling back to try reading non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:
1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.
  print(xr.open_dataset(m, engine='zarr'))
<xarray.Dataset>
Dimensions:  (lat: 100, lon: 100)
Coordinates:
  * lat      (lat) float64 0.0 1.0 2.0 3.0 4.0 5.0 ... 95.0 96.0 97.0 98.0 99.0
  * lon      (lon) float64 0.0 1.0 2.0 3.0 4.0 5.0 ... 95.0 96.0 97.0 98.0 99.0
Data variables:
    var      (lon, lat) float64 ...
Traceback (most recent call last):
  File "/Users/aaron.spring/Coding/kerchunk/my_kerchunk/run.py", line 52, in <module>
    mzz.translate("output.zarr")
  File "/Users/aaron.spring/Coding/kerchunk/kerchunk/combine.py", line 79, in translate
    out = self._build_output(ds, ds0, fss)
  File "/Users/aaron.spring/Coding/kerchunk/kerchunk/combine.py", line 185, in _build_output
    acc_len = make_coord(fss, z, accum_dim)
  File "/Users/aaron.spring/Coding/kerchunk/kerchunk/combine.py", line 301, in make_coord
    zz = zarr.open_array(fs.get_mapper(accum_dim))
  File "/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/zarr/creation.py", line 526, in open_array
    init_array(store, shape=shape, chunks=chunks, dtype=dtype,
  File "/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/zarr/storage.py", line 353, in init_array
    _init_array_metadata(store, shape=shape, chunks=chunks, dtype=dtype,
  File "/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/zarr/storage.py", line 390, in _init_array_metadata
    shape = normalize_shape(shape) + dtype.shape
  File "/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/zarr/util.py", line 42, in normalize_shape
    raise TypeError('shape is None')
TypeError: shape is None

Versions:

zarr                      2.10.3
kerchunk                  0.0.5+47.gf42b0c2 (current main)
fsspec                    2022.1.0
xarray                    0.20.2

I do not see the variable/coordinate "time" in the input data

I got the same error even with ds = xr.DataArray(name='var', data=np.random.random((100,100)), dims=['lon','lat'], coords={'lon':range(100), 'lat':range(100)}).to_dataset().expand_dims('time') also with multiple time indices in ds

.expand_dims('time')

This just adds "time" to the list of dims but doesn't assign any values, so there's no way to determine what the output "time" coordinate should be.
Interestingly, in #122 I am developing a new version of combine, where you can define how to extract the concat coord(s) from each dataset.
The following works, for example
In [20]: from kerchunk.combine import MultiZarrToZarr
    ...: mzz = MultiZarrToZarr(
    ...:     "zip://*.json::out.zip",
    ...:     remote_protocol="file",
    ...:     concat_dims=["time"],
    ...:     coo_map={"time": "INDEX"}
    ...: )
In [22]: m = fsspec.get_mapper('reference://', fo=test_dict, remote_protocoll='file')
In [23]: xr.open_dataset(m, engine='zarr', backend_kwargs={"consolidated": False})
Out[23]:
<xarray.Dataset>
Dimensions:  (lat: 100, lon: 100, time: 11)
Coordinates:
  * lat      (lat) float64 0.0 1.0 2.0 3.0 4.0 5.0 ... 95.0 96.0 97.0 98.0 99.0
  * lon      (lon) float64 0.0 1.0 2.0 3.0 4.0 5.0 ... 95.0 96.0 97.0 98.0 99.0
  * time     (time) float64 nan 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0
Data variables:
    var      (time, lon, lat) float64 ...

where we use the coo_map argument we define the value of "time" to be equal to the sequential index of the inputs. The output dtype of "time" should be int64, not float64, but this is pretty good.

great. #122 works for me. only the final step fails File "/Users/aaron.spring/Coding/kerchunk/run.py", line 44, in <module>     mzz.translate("output.json") TypeError: translate() takes 1 positional argument but 2 were given. looking forward to when this PR is merged!

Yep, for now it returns the references for you to deal with - but it should write directly as the previous version did.

I want to use kerchunk to open climate model netcdf output stored 1 or 12 timestep/s per file in many files as one huge zarr.

After discussing with @d70-t, I found out that my climate model output is usually stored with no spatial chunks and the unlimited time dimension. Given those current defaults in creating climate model output, kerchunk cannot help me speeding up. Also my internal test case is slower using kerchunk compared to just using nc. Nevertheless #122 works for me and will resolve the issue title.

my climate model output is usually stored with no spatial chunks and the unlimited time dimension

That's OK!
See this branch, where I've made a start on netCDF3:

we can grab arrays as chunks (which could pre sub-chunked #124)
the records part (for the unlimited dimension(s)) can either be read as a single numpy record array and sub-selected, or one chunk per record if they are big enough - or even something inbetween. The choice would be in the hands of the person making the references.

test case is slower using kerchunk compared to just using nc

Can you share?

I create from MultiZarrToZarr and compare this performance in xr.open_dataset(json) again xr.open_mfdatasets(nc_file_list). To get some more random read accesses shuffle the data and multiply with itself.
Script:
import os
import zipfile
import kerchunk.hdf
import fsspec
from timeit import default_timer as timer

import xarray as xr
import numpy as np
from kerchunk.combine import MultiZarrToZarr
import json

recreate=True
use_dummy=True # False
nvar=20
ntime=50
nlev=1
if use_dummy:
    ds = xr.DataArray(name='var', data=np.random.random((220,256,12,nlev,nvar)), dims=['x','y','time','depth','var'], coords={'x':range(220), 'y':range(256), 'depth':range(nlev), 'var':[f'var{i}' for i in range(nvar)]}).to_dataset('var')

    urls =[f'test_{y}.nc' for y in range(2000,2000+ntime)]
    if recreate:
        for i,u in enumerate(urls):
            #ds = ds.assign_coords(time=xr.cftime_range(start=str(2000+i),freq='MS', periods=ds.time.size))
            ds = ds.assign_coords(time=range(2000+i*12,2000+i*12+12))
            ds.astype('float32').to_netcdf(u)

else:
    # use MPIESM output
    urls = [i for i in os.listdir('.') if 'asp_esmControl' in i]

start=timer()
so = dict(
    anon=True, default_fill_cache=False, default_cache_type='first'
)

with zipfile.ZipFile("out.zip", mode="w") as zf:
    for u in urls:
        print(u)
        with fsspec.open(u, **so) as inf:
            h5chunks = kerchunk.hdf.SingleHdf5ToZarr(inf, u, inline_threshold=100)
            with zf.open(os.path.basename(u) + ".json", 'w') as outf:
                outf.write(json.dumps(h5chunks.translate()).encode())

remote_protocol="memory"
from kerchunk.combine import MultiZarrToZarr
mzz = MultiZarrToZarr(
    "zip://*.json::out.zip",
    remote_protocol=remote_protocol,
    concat_dims=["time"],
    #coo_map={"time": "INDEX"},
)
print(mzz)
test_dict = mzz.translate()
end = timer()
print('create json '+str(round(end-start,3))+'s')

#print(test_dict)
#mzz.translate("output.zarr")

# This can also be written as a json
#mzz.translate("output.json")

combine='nested'
concat_dim='time'

m = fsspec.get_mapper('reference://', fo=test_dict, remote_protocoll=remote_protocol)
ker = xr.open_dataset(m, engine='zarr', backend_kwargs={"consolidated": False})
x = xr.open_mfdataset(urls, concat_dim=concat_dim, combine=combine)
print(f'{ker.sizes} {ker.nbytes*1e-9} GB')
print(f'{x.sizes} {x.nbytes*1e-9} GB')

def shuffle(ds):
    idx = np.arange(ds.time.size)
    np.random.shuffle(idx)
    return (ds.isel(time=idx)*ds).mean('time')

# time xr.open_mfdataset(, engine='zarr')
print('start shuffle(xr.open_dataset(json, engine="zarr")).compute()')
start = timer()
#m = fsspec.get_mapper('reference://', fo=test_dict, remote_protocoll=remote_protocol)
shuffle(xr.open_dataset(m, engine='zarr', backend_kwargs={"consolidated": False, "storage_options": {"fo":test_dict, "remote_protocol":remote_protocol}})).compute()
end = timer()
print('took '+str(round(end-start,3))+'s')

combine='nested'
concat_dim='time'
# time open xr.open_mfdataset(urls)
print('start shuffle(xr.open_mfdataset("*").compute()')
start = timer()
shuffle(xr.open_mfdataset(urls, concat_dim=concat_dim, combine=combine)).compute()
end = timer()
print('took '+str(round(end-start,3))+'s')

print('start shuffle(xr.open_mfdataset("*", parallel=True, coords="minimal", data_vars="minimal", compat="override").mean().compute()')
start = timer()
shuffle(xr.open_mfdataset(urls, combine=combine, concat_dim=concat_dim, parallel=True, coords="minimal", data_vars="minimal", compat="override")).compute()
end = timer()
print('took '+str(round(end-start,3))+'s')

print('start shuffle(xr.open_mfdataset("*", parallel=False, coords="minimal", data_vars="minimal", compat="override").mean().compute()')
start = timer()
shuffle(xr.open_mfdataset(urls, concat_dim=concat_dim, combine=combine, parallel=False, coords="minimal", data_vars="minimal", compat="override")).compute()
end = timer()
print('took '+str(round(end-start,3))+'s')

print("")
print("end")

Output:
(xr) aaron.spring@MacBook-Pro:~/Coding/kerchunk$ python zarrify_nc.py
test_2000.nc
test_2001.nc
test_2002.nc
...
test_2048.nc
test_2049.nc
<kerchunk.combine.MultiZarrToZarr object at 0x100d53f70>
create json 1.029s

Frozen({'depth': 1, 'time': 600, 'x': 220, 'y': 256}) 2.703368616 GB
Frozen({'x': 220, 'y': 256, 'time': 600, 'depth': 1}) 2.703368616 GB

start shuffle(xr.open_dataset(json, engine="zarr")).compute()
took 22.922s

start shuffle(xr.open_mfdataset("*").compute()
/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing with an out-of-order index is generating 12 times more chunks
  return self.array[key]
...
/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing with an out-of-order index is generating 12 times more chunks
  return self.array[key]
took 5.966s

start shuffle(xr.open_mfdataset("*", parallel=True, coords="minimal", data_vars="minimal", compat="override").mean().compute()
/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing with an out-of-order index is generating 12 times more chunks
  return self.array[key]
...
/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing with an out-of-order index is generating 12 times more chunks
  return self.array[key]
took 6.157s

start shuffle(xr.open_mfdataset("*", parallel=False, coords="minimal", data_vars="minimal", compat="override").mean().compute()
/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing with an out-of-order index is generating 12 times more chunks
  return self.array[key]
...
/Users/aaron.spring/mambaforge/envs/xr/lib/python3.9/site-packages/xarray/core/indexing.py:1227: PerformanceWarning: Slicing with an out-of-order index is generating 12 times more chunks
  return self.array[key]
took 5.958s

end

The biggest difference here, is that the kerchunk-zarr dataset is not being opened with dask. You need to add chunks={} for that. I get the following results. Note that I saved my output references to a file, "test_out.json" (62kB) so that I could run everything with a fresh python session (I also added skip_instance_cache for this reason)
import xarray as xr
import numpy as np

ntime = 50
combine='nested'
concat_dim='time'

urls =[f'test_{y}.nc' for y in range(2000,2000+ntime)]

%time ds = xr.open_mfdataset(urls, concat_dim=concat_dim, combine=combine, parallel=True, coords="minimal",
                             data_vars="minimal", compat="override")
# 1.56 s

%time ds2 = xr.open_dataset("reference://", engine='zarr', backend_kwargs={"consolidated": False, "storage_options": {
    "fo":"test_out.json", "remote_protocol":"file", "skip_instance_cache": True}}, chunks={})
# 57 ms

def shuffle(ds):
    idx = np.arange(ds.time.size)
    np.random.shuffle(idx)
    return (ds.isel(time=idx)*ds).mean('time')

%time shuffle(ds).compute()

# 10.3 s

%time shuffle(ds2).compute(0)

# 14.3 s

import dask.distributed
client = dask.distributed.Client()

%time shuffle(ds).compute()

# 29.5 s

%time shuffle(ds2).compute(0)

# 20.6 s

So with default threaded Dask, kerchunk is somewhat slower (I suspect xarray is doing some caching of its own here, cc @rabernat ?) but with distributed it is somewhat faster (I suspect distributed's cache is efficient here). Kerchunk is always much faster to open the dataset, which is a small part of the overall cost in this case.
Note that kerchunk allows you to choose chunk sizes bigger that the native ones in the data (e.g., chunks={"time": 36}), which makes a big difference to remote data, but no real difference here.

On second thoughts, since the data is uncompressed, it may well be that h5py can read it directly without copy and without the GIL, whereas kerchunk/referenceFS deals with bytes. We could implement opening as mmap and passing around memorybuffers.


[Fix #10613] Allow parallel inspection for autocorrect

Allow the -P/--parallel flag together with the autocorrection flags --[safe-]auto-correct/-a --auto-correct-all/-A and  --fix-layout/-x. Also make this the default, so that --no-parallel has to be given for a single core run.
The algorithm for normal parallel inspection is followed, so that a cache warm-up phase is run first. This phase only does caching of inspection results. It doesn't update any files. Then it does a single core run that takes advantage of the cached results and does the corrections. The speed gain from this approach is probably smaller than writing files in parallel, but it's a pretty small change that still outputs correction results in the same way as before.
Also fix the documentation for --parallel, which did not mention --no-parallel.

What gain in speed you will get from this change depends on two things:

the state of the cache
the percentage of files with no offenses

With a cold cache and offenses in 10% of the files, it can cause rubocop -a to execute in about half the time in a large project.
You can get the same effect (guarantee a warm cache) in the current version by running rubocop; rubocop -a.

Looks like a nice improvement to me. Thanks!
P.S. Probably we should document somewhere how the autocorrect works, so that people are not confused it writes the corrections in parallel.


404 link at EOF

[Enter feedback here]

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d7c93e65-86c1-fd8b-a888-c57d3b26e2bd
Version Independent ID: 3f31fd84-7dba-2444-2344-ec70113ba694
Content: Use Azure Databricks for sentiment analysis
Content Source: databricks/scenarios/databricks-sentiment-analysis-cognitive-services.md
Service: azure-databricks
GitHub Login: @lenadroid
Microsoft Alias: alehall

@oscarmandley Thank you for your feedback! We will review and update as appropriate.

Hi @leifbro - Could you please review this feedback and update the doc as needed? - Thank you

Fixed in Databricks source. Will bed deployed to docs.microsoft.com between now and Monday night

Thanks @leifbro for the quick fix.
@oscarmandley - Thanks for reporting this issue. PR for the fix has been submitted and changes should reflect by end of Monday.
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply and reopen the issue. We will gladly continue the discussion.


RKE cluster fails to deploy

What kind of request is this (question/bug/enhancement/feature request): bug
Steps to reproduce (least amount of steps as possible):

Deploy an rke cluster - DO/AWS - 5 nodes - 1 etcd, 1 control plane, 3 worker nodes (tried with a custom - 1 node - all roles also)
The cluster fails to come up.
Cluster is stuck in provisioning state, error: Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [] on the UI
On checking rancher logs - rancher/rancher-agent:master-head keep restarting
logs of the agent container:

ERROR: Please check if the correct certificate is configured at https://rancher-server/v3/settings/cacerts
INFO: Arguments: --server https://rancher-server --token REDACTED --ca-checksum 48a041033e42d49f56b748b579f3aa6172a787523f673c94224dcb9c131256f5 -r -n m-g95gr
INFO: Environment: CATTLE_ADDRESS=ip CATTLE_AGENT_CONNECT=true CATTLE_INTERNAL_ADDRESS= CATTLE_NODE_NAME=m-g95gr CATTLE_SERVER=https://rancher-server CATTLE_TOKEN=REDACTED
INFO: Using resolv.conf: nameserver <IP_ADDRESS> nameserver <IP_ADDRESS>
INFO: https://rancher-server/ping is accessible
INFO: Value from https://rancher-server/v3/settings/cacerts is an x509 certificate
ERROR: Configured cacerts checksum (fb5552604ef8b0b0ad24f98afb37712f77ad5bb836e8a36dcfc4be410d9cdbee) does not match given --ca-checksum (48a041033e42d49f56b748b579f3aa6172a787523f673c94224dcb9c131256f5)
ERROR: Please check if the correct certificate is configured at https://rancher-server/v3/settings/cacerts

Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): master-head - commit id: 537896429
Installation option (single install/HA): single

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): rke DO/ AWS
Machine type (cloud/VM/metal) and specifications (CPU/memory):
Kubernetes version (use kubectl version):

1.17.2

Tested with latest build from master - Rancher version e9b7af681.
The issue is still reproducible .
Not able to bring up any RKE cluster.
rancher/rancher-agent:master-head keep restarting
logs of the agent container:
INFO: Using resolv.conf: nameserver <IP_ADDRESS> nameserver <IP_ADDRESS>
INFO: https://<rancher-server ip>/ping is accessible
INFO: Value from https://<rancher-server ip>/v3/settings/cacerts is an x509 certificate
ERROR: Configured cacerts checksum (48d5c3abd34aba1709c309490b8a602e38c50c3f0f64d720925f4c7f41e6d1e9) does not match given --ca-checksum (3dd229922de70a8574ebdcb7b35660a89bf406a472b3963611fcd98d7032517e)
ERROR: Please check if the correct certificate is configured at https://<rancher-server ip>/v3/settings/cacerts

verified on master-head latest - commit id 92117c9bd

RKE cluster comes up fine. No error seen.
RKE custom cluster also is deployed successfully.


currency number

https://github.com/opensquare-network/paid-qa/issues/773

toLocaleString ?

BigNumber.js did it


Update to 1 in STEP and README.md

A new pull request by NChavan

A new pull request comment by NChavan


Activating Cortana Channel in Bot Framework

When I attempt to add Cortana as a channel in a Bot via Azure, I receive a number of different errors errors.

When I 1st click deploy, I get the incorrect image error.

Once I upload another image for the Bot, I receive an internal server error. Below is a screenshot of it along with what's shown with the api call.

Hi @007-fr
Cortana Skills have been retired, except for members of the TAP program:
https://docs.microsoft.com/en-us/cortana/skills/third-party-deprecation#how-do-i-build-cortana-skills-in-the-future
https://docs.microsoft.com/en-us/cortana/enterprise/overview
https://github.com/MicrosoftDocs/bot-docs/issues/1959
I'm not sure how to contact someone from the TAP program.  Does anyone on the ABS team know if this is documented anywhere?

I have seen it suggested in forums that Enterprise-level customers need to reach out to a Microsoft Account rep in order to start the process of joining a TAP program. But, I don't know if that is still the case and no link / reference was provided in those cases.

Hi @007-fr, we have fixed the errors in the screenshot above and you should be able to make progress now.  However, please keep in mind, that Cortana Skills have been retired except for TAP program: https://docs.microsoft.com/en-us/cortana/skills/third-party-deprecation#how-do-i-build-cortana-skills-in-the-future.

Hi @007-fr, we have fixed the errors in the screenshot above and you should be able to make progress now.  However, please keep in mind, that Cortana Skills have been retired except for TAP program: https://docs.microsoft.com/en-us/cortana/skills/third-party-deprecation#how-do-i-build-cortana-skills-in-the-future.

@p-nagpal The company I work for is member of the tap program, but I don’t think it’s a good idea to start Cortana if it’s already retired.
What the recommendation for using voice interaction in the Bot Framework?

@p-nagpal The company I work for is member of the tap program, but I don’t think it’s a good idea to start Cortana if it’s already retired.
What the recommendation for using voice interaction in the Bot Framework?

Hi @007-fr, the roadmap for Cortana Skills will be updated on https://docs.microsoft.com/en-us/cortana/skills/third-party-deprecation#how-do-i-build-cortana-skills-in-the-future as we make changes. Additionally, you might also be interested in looking at Direct Line Speech Channel. Here is some documentation on building voice assistants with Direct Line Speech Channel: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/direct-line-speech

Hi @007-fr, the roadmap for Cortana Skills will be updated on https://docs.microsoft.com/en-us/cortana/skills/third-party-deprecation#how-do-i-build-cortana-skills-in-the-future as we make changes. Additionally, you might also be interested in looking at Direct Line Speech Channel. Here is some documentation on building voice assistants with Direct Line Speech Channel: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/direct-line-speech


fix: github actions are run on PRs from forked repositories too

Codecov Report

Merging #45 into master will increase coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #45      +/-   ##
==========================================
+ Coverage   24.83%   24.89%   +0.05%
==========================================
  Files          11       11
  Lines         467      466       -1
==========================================
  Hits          116      116
+ Misses        351      350       -1

Impacted Files
Coverage Δ

eventually-test/src/order.rs
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a11073c...72a7225. Read the comment docs.


Upstream repo moved again

Hello!
The upstream repository seems to have moved again under the Aqua Security organization:  https://github.com/aquasecurity/tfsec

Thanks for the heads up @aspring! It should be fixed now, thanks to #4


⚠ Bofrost.at has degraded performance

In fc5e797, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1744 ms

Resolved: Bofrost.at performance has improved in 252ebe1 after 2 hours, 2 minutes.


[Brackets auto-update Windows] Update notification button is still visible after brackets has been updated to latest version.

Description
Update notification button is still visible after brackets has been updated to latest version.
Steps to Reproduce

Launch brackets 1.14.
Update Notification Button is still visible.

Expected behavior: Update notification button should not be available once the update has been done.
Actual behavior: Update notification button is still visible after brackets has been updated to latest version.
Versions
Windows 10 64 Bit
Release 1.14 build 1.14.0-17639

Working fine now.
Verified in latest build - Release 1.13 prerelease build 1.13.0-17689.


Chore/extract codegen parser more than one module exception

Summary
This PR is part of https://github.com/facebook/react-native/issues/34872
This PR extracts MoreThanOneModuleInterfaceParserError exception to a separate function inside an error-utils.js file
Changelog
[Internal] [Changed] - Extract MoreThanOneModuleInterfaceParserError to a seperate function inside error-utils.js
Test Plan

Can you rebase on top of main? The #34896 finally landed!

Done @cipolleschi
Let me know any feedback, thanks!

Hi @Marcoo09!
The problem is in how you are trying to export the ParserType. As the compiler tells you, Types are erased at runtime. If you want to reuse that type, you have to export it with this syntax in the errors.js
export type ParserType

and then you have to use it with the import type syntax:
import type {ParserType} from

In this way, it should work!

Great @cipolleschi
I've just done the changes. Thanks!

@motiz88 could you please reimport this PR?

/rebase

@Marcoo09 could you rebase and resolve the conflicts, please?

@cipolleschi sorry, just rebased

Sorry to ask again, could you rebase, please? 🙏


Update public documentation regarding deprecation of operators

Highlight 1.19 related notes in public documentation for astronomer-providers.
It's essential for us to closely coordinate with the documentation and CRE teams
throughout the release to ensure seamless communication. In our GitHub repo docs,
we should suggest using Async operators from OSS repo instead of using ours.

With the SnowflakeSqlApiHookAsync, we neglected to specify the region while constructing the URL. In the connection, we were appending the region to the account name only. However, in the OSS hook, the URL is constructed using the region. Therefore, we can now update the region field in the connection and remove the region from the account field.

Created draft PR: https://github.com/astronomer/astronomer-providers/pull/1468


Static Redis cache

Hi Ankit,
The RedisStore pretended to be non-Static; I have taken the liberty to make a static implementation of the cache (in fact, it was already static before).
Also, I felt the integration to the bot framework could be simplified by just configuring the Redis Bot cache. I think, by explicitly configuring the bot cache, we can assume that the user wants to use it with the bot framework, so we can do the configuration for him.

Hi vossccp,
Overall this is good. However the StreamWriter in Serialize method is by default buffered. That is why I had it explicitly closed. If you can fix it it would be great.

Yes, you are right. I was a bit confused since Dispose should close the stream. I hope, its ok for you if we just flush before returning the content

Sorry for late follow-up. The GZipStream Flush method is not implemented to flush underlying buffer as mentioned here. We need to explicitly close the stream. Just call Close on streamWriter instead of flush and it should work properly.

Hi there. We never really came to finish this. Hope, its now ok


Extra vertical lines (white) inside the bar chart

I have a stacked bar chart with the following settings (chart.js - 2.5.0):
scales: {
  xAxes: [{
    barPercentage: 1.0,
    categoryPercentage: 1.0,
  }]
}

I don't expect to see any vertical lines, but if resize the browser, I see them.
See this fiddle and try to resize the browser window.
https://jsfiddle.net/3q7npu1n/5/
You will see the following issues. How can I avoid them?

Closing since I think this is a duplicate of #3916. What's happening is that the edges of the bars are moving due to numerical rounding.
Have you tried the latest code from master?


Batched rollout

Vectorize the rollout loop. I could fit batch_size=10 of 2D RPF on a 48GB GPU at seemingly no additional cost. Larger batches also fit, but then computation is linearly slower.
This should not break any previous functionality as we essentially just vectorize, i.e. batch_size=1 corresponds to what we had before.

Could you add to the PR how much does this speed up evaluation?

Could you add to the PR how much does this speed up evaluation?

done


added FollowRedirects and AllowInsecure to LoadBalancerMonitor

Missing fields in LoadBalancerMonitor
see api doc

@FnuGk Do you have any time to wrap this one up? Looks like the Terraform provider wants this functionality and it would be good to finish this one as you're quite close to getting this over the line.

cc @patryk - this is ready for review 🍰

Many thanks, @FnuGk!


Updates to printing paths

The printed paths only contain variables that exist in the original binary. For example, if variable #1 existed in only the modified binary, it would not be printed. This made it difficult to trace through the printed path of the modified binary.

Although WP first prints out the original path and then the modified path, it is difficult to determine when the switch occurs. Having headers for this would be nice.

Within a certain path, the order of the jumps printed out seem to be arbitrary. It would be great if it was in order.

Closed with #151.


Throw if no component is passed to connect

Currently, if you fail to pass a component to the function returned by connect it fails in a confusing way (usually inst.render is not a function) that can be hard to track down if you're new to Redux.
This PR adds a check, making sure a function is passed to connect, throwing a useful error if not.

Superseded by #416


Add instructions how to do commit signing

What type of PR is this?

Uncomment only one  /kind line, and delete the rest.
For example, > /kind bug would simply become: /kind bug

/kind documentation
[skip ci]
What does does this PR do / why we need it:
Adds documentation with regards to creating a pull request / commit
signing. It is good practice to get into the habit of signing each
commit.
Which issue(s) this PR fixes:
N/A
How to test changes / Special notes to the reviewer:
N/A
Signed-off-by: Charlie Drage<EMAIL_ADDRESS>

/retest

/retest


Bug Report

Environment
Windows build number: [run "Version 10.0.18912.1001" at a command prompt]
Windows Terminal version (if applicable): unsure how to check, fresh install from the Microsoft Store

Any other software?

Steps to reproduce

Start windows terminal,
Open couples of tabs
Try to select and then close final tab
Expected behavior

Every tab should be able to select, and close
Actual behavior

#857


flynn release use cases

Hi, I don't fully understand the use cases for the flynn release  command. Is this to run arbitrary docker images in flynn? or to dockerize an app that already runs in flynn?
If you can please elaborate the use cases that would be helpful. Thanks!

flynn release is used to create a release with an arbitrary image, typically from the default Docker registry.
It is currently under-documented as we are not happy with the current workflow (see #1085 for thoughts on improving it).
See here for an example of using flynn release to deploy Redis.

Thanks @lmars I'll be warching the feature as it evolves.


Change hero background colour to an image

Swap the blue hero background to an image. Maybe add some sort of blur to it too.

@wrussell1999 I'll work on this enhancement.


Update WaypointManager 2.7.3 hashes

Replaced download, see https://forum.kerbalspaceprogram.com/index.php?/topic/154922-ckan-the-comprehensive-kerbal-archive-network-v1254-kennedy/&do=findComment&comment=3503578

#rebuild


Cooldown

Hey,
Why  do you remove cooldown? when using other permission system we can't set cooldown :/

Cooldown should work in all permission systems

Ok, so in my configuration i use LPX permission system.
How can i add cooldown on command ?

I don't know how lpx does it, but it HAS to implement cooldown

Cooldown should work in all permission systems
(If plugin is update and if the developper know the rocket update)

I think, is not a good idea, to do critical update, all developers are not ready and don't have information of this update . . . It HAS to implement cooldown, yes is sure ^^ But need time to do it.
Need to stop doing critical update, without preventing Dev & User. Why don't use the forum in this case ?
In this time server owner can't provide good service to her player.
Why don't do an option for enable or disable cooldown ?

#1 you don't have to enable updates
#2 there is no need to update until friday, thats 4 days to implement, way more time then i have to cope with nelsons surprising changes

<3 Nelson's Surprise ^^
Now i need to wait LPX update :((( 8 Servers without perm :/
Have a nice day.


Style loading indicator

Add styling for loading panel following the figma UX

Fixes #662

Added screenshot and translation of string.


#53 properly test next-on-netlify in onBuild

Fixes https://github.com/netlify/netlify-plugin-nextjs/issues/53

thought: we could maybe try to get a dist from /sample

re: the conversation around the sample project/fixture - i hear what youre saying but dont see nearly a big enough issue to justify why certain files need to be removed. next-on-netlify has a ton of sample projects embedded in its cypress and test directories, with respective package.json filees and .gitignore and files. when /sample was on the root level of the project, it existed with these same files (package.json required to, well, run the project to test the plugin as a local plugin and .gitignore to avoid committing files relevant to that project). if you'd really like, we can combine the .gitignore with the top-level, but because the sample project requires different dependencies to run as a local plugin, i don't think it makes sense to combine package.json files. i hope this makes sense!

What you say makes sense. However, when it comes to having multiple package.json, we do need to consider the following drawbacks:

This requires adding cd ... && npm install in the CI, which makes CI much slower. It also makes local development setup (since multiple directories node_modules would now need to be installed)
Dependencies upgrades would now need to be done in multiple package.json
At the moment, this is a single sample project. But new tests might add more of them, which would increase that maintenance cost.

Separating which dependencies are needed in specific samples/fixtures from the main dependencies is nice, but it feels to me that benefit would be outweighed by those drawbacks. Just adding samples/fixtures to the top-level package.json devDependencies would just work without any of those drawbacks. What do you think?

removing the package.json from the sample dir seems to interfere with testing the local plugin within the /sample project, which is and has been its primary purpose (see TESTING.md from original commit of the sample dir). we cant test with netlify cli without a package.json and the standard Next.js essentials. this is essential to plugin development and so unless you have a way to run netlify build and netlify deploy from sample without a package.json and the files you claim to be unnecessary, i'm going to assert the need to keep things as is.

separate issue: the plugin won't install as a local plugin with the tests, it seems. it complains about the jest keywords being undefined. unclear at this moment if this is just local plugin specific or an issue with installing the plugin in prod as well (that wouldn't make sense, though, since we have tests in main/published). what's the fix here for local plugin installation? not sure if it's obvious or a plugin issue, happy to open an issue as well.

not sure if it's obvious or a plugin issue, happy to open an issue as well.

Yes, that might be good (including build logs and steps to reproduce), thanks!

i think this should be good now!!!!!!!!! @ehmicky


CSP errors in Microsoft Edge

So, I was just trying out Edge and I get these:
HTML1300: Navigation occurred.
www.bootstrapcdn.com
CSP14309: Unknown directive 'child-src' in Content-Security-Policy - directive will be ignored.
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: https://ghbtns.com/github-btn.html?user=MaxCDN&repo=bootstrap-cdn&type=watch&count=true. Resource will be blocked.
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: https://platform.twitter.com/widgets/follow_button.f7908d4abf5ce27173c69bdbb93aedb6.en.html. Resource will be blocked.
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: https://platform.twitter.com/widgets/follow_button.f7908d4abf5ce27173c69bdbb93aedb6.en.html. Resource will be blocked.
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: https://platform.twitter.com/widgets/tweet_button.f7908d4abf5ce27173c69bdbb93aedb6.en.html. Resource will be blocked.
XML5632: Only one root element is allowed.
Line: 1, Column 1
XML5632: Only one root element is allowed.
Line: 1, Column 1
XML5632: Only one root element is allowed.
Line: 1, Column 1
XML5632: Only one root element is allowed.
Line: 1, Column 1
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: data:image/svg+xml,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2072%2072%22%3E%3Cpath%20fill%3D%22none%22%20d%3D%22M0%200h72v72H0z%22%2F%3E%3Cpath%20class%3D%22icon%22%20fill%3D%22%23e1e8ed%22%20d%3D%22M68.812%2015.14c-2.348%201.04-4.87%201.744-7.52%202.06%202.704-1.62%204.78-4.186%205.757-7.243-2.53%201.5-5.33%202.592-8.314%203.176C56.35%2010.59%2052.948%209%2049.182%209c-7.23%200-13.092%205.86-13.092%2013.093%200%201.026.118%202.02.338%202.98C25.543%2024.527%2015.9%2019.318%209.44%2011.396c-1.125%201.936-1.77%204.184-1.77%206.58%200%204.543%202.312%208.552%205.824%2010.9-2.146-.07-4.165-.658-5.93-1.64-.002.056-.002.11-.002.163%200%206.345%204.513%2011.638%2010.504%2012.84-1.1.298-2.256.457-3.45.457-.845%200-1.666-.078-2.464-.23%201.667%205.2%206.5%208.985%2012.23%209.09-4.482%203.51-10.13%205.605-16.26%205.605-1.055%200-2.096-.06-3.122-.184%205.794%203.717%2012.676%205.882%2020.067%205.882%2024.083%200%2037.25-19.95%2037.25-37.25%200-.565-.013-1.133-.038-1.693%202.558-1.847%204.778-4.15%206.532-6.774z%22%2F%3E%3C%2Fsvg%3E. Resource will be blocked.
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: data:image/svg+xml,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2054%2072%22%3E%3Cpath%20class%3D%22icon%22%20fill%3D%22%23e1e8ed%22%20d%3D%22M38.723%2012c-7.187%200-11.16%207.306-11.723%208.13-.563-.824-4.496-8.13-11.723-8.13C8.79%2012%203.533%2018.163%203.533%2024.647%203.533%2039.964%2021.89%2055.907%2027%2056c5.11-.093%2023.467-16.036%2023.467-31.353C50.467%2018.163%2045.21%2012%2038.723%2012z%22%2F%3E%3C%2Fsvg%3E. Resource will be blocked.
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: data:image/svg+xml,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2056.8%2072%22%3E%3Cpath%20fill%3D%22none%22%20d%3D%22M0-.7h48v72H0z%22%2F%3E%3Cpath%20class%3D%22icon%22%20fill%3D%22%23e1e8ed%22%20d%3D%22M52.8%2032.9L42.2%2022.3c-1.4-1.4-3.6-1.4-5%200s-1.4%203.6%200%205l4.6%204.6H20.9c-1.9%200-3.5%201.6-3.5%203.5s1.6%203.5%203.5%203.5h20.9l-4.6%204.6c-1.4%201.4-1.4%203.6%200%205%20.7.7%201.6%201%202.5%201s1.8-.3%202.5-1l10.6-10.6c1.4-1.4%201.4-3.6%200-5z%22%2F%3E%3Cpath%20class%3D%22icon%22%20fill%3D%22%23e1e8ed%22%20d%3D%22M24.4%2049.4h-13c-.6%200-1.1-.5-1.2-1.1V22.4c0-.7.5-1.2%201.2-1.2h12.9c1.9%200%203.5-1.6%203.5-3.5s-1.6-3.5-3.5-3.5H8.9c-3.1%200-5.7%202.5-5.7%205.7v31c0%203.1%202.5%205.7%205.7%205.7h15.5c1.9%200%203.5-1.6%203.5-3.5.1-2.1-1.5-3.7-3.5-3.7z%22%2F%3E%3C%2Fsvg%3E. Resource will be blocked.
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: data:image/svg+xml,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2064%2072%22%3E%3Cpath%20fill%3D%22none%22%20d%3D%22M0%200h64v72H0z%22%2F%3E%3Cpath%20fill%3D%22%2388c9f9%22%20d%3D%22M3%2037.315c0%204.125%202.162%207.726%205.363%209.624-.056.467-.09.937-.09%201.42%200%206.103%204.72%2011.045%2010.546%2011.045%201.295%200%202.542-.234%203.687-.686C24.22%2062.4%2027.827%2064.93%2032%2064.93c4.174%200%207.782-2.53%209.49-6.213%201.148.45%202.39.685%203.69.685%205.826%200%2010.546-4.94%2010.546-11.045%200-.483-.037-.953-.093-1.42C58.83%2045.04%2061%2041.44%2061%2037.314c0-4.37-2.42-8.15-5.933-9.946.427-1.203.658-2.5.658-3.865%200-6.104-4.72-11.045-10.545-11.045-1.302%200-2.543.232-3.69.688-1.707-3.685-5.315-6.216-9.49-6.216-4.173%200-7.778%202.53-9.492%206.216-1.146-.455-2.393-.688-3.688-.688-5.827%200-10.545%204.94-10.545%2011.045%200%201.364.23%202.662.656%203.864C5.42%2029.163%203%2032.944%203%2037.314z%22%2F%3E%3Cpath%20fill%3D%22%23FFF%22%20d%3D%22M17.87%2039.08l7.015%206.978c.585.582%201.35.873%202.116.873.77%200%201.542-.294%202.127-.883.344-.346%2015.98-15.974%2015.98-15.974%201.172-1.172%201.172-3.07%200-4.243-1.17-1.17-3.07-1.172-4.242%200l-13.87%2013.863-4.892-4.868c-1.174-1.168-3.074-1.164-4.242.01-1.168%201.176-1.163%203.075.01%204.244z%22%2F%3E%3C%2Fsvg%3E. Resource will be blocked.
CSP14312: Resource violated directive 'default-src 'none'' in Content-Security-Policy: https://syndication.twitter.com/i/jot. Resource will be blocked.
SEC7120: Origin https://www.bootstrapcdn.com not found in Access-Control-Allow-Origin header.
www.bootstrapcdn.com
SCRIPT7002: XMLHttpRequest: Network Error 0x80700013, Could not complete the operation due to error 80700013.
www.bootstrapcdn.com
SEC7120: Origin https://www.bootstrapcdn.com not found in Access-Control-Allow-Origin header.
www.bootstrapcdn.com
SCRIPT7002: XMLHttpRequest: Network Error 0x80700013, Could not complete the operation due to error 80700013.
www.bootstrapcdn.com

I don't use Edge, nor do I like it at all, so any ideas are welcome.
/CC @ScottHelme just in case you have any hints.

This is Edge failing hard at CSP. To be fully backwards compatible I'd advise duplicating your child-src into the frame-src too.
child-src 'self' platform.twitter.com syndication.twitter.com ghbtns.com;connect-src *.init.cedexis-radar.net radar.cedexis.com rpt.cedexis.com;
frame-src 'self' platform.twitter.com syndication.twitter.com ghbtns.com;connect-src *.init.cedexis-radar.net radar.cedexis.com rpt.cedexis.com;

@ScottHelme: thanks a lot for the explanation. Unfortunately, express-csp we currently use doesn't apply the old frame-src directive. I have a PR to switch to helmet for the security headers, but I'm having some issues with HSTS headers not being sent even if I specify it.
I'll try to fix the helmet branch


User Build - Unable to upload file to uss with Zowe profile ZOSMF

Development environment used

Z Open Editor version: 2.0.0
Editor Platform

[X] Visual Studio Code
[ ] Red Hat CodeReady Workspaces
[ ] Eclipse Che
[ ] Standalone Theia

Editor Platform Version: 1.66.2
Operating System: Windows 10 20H2
Java Version: 1.8
[ ] Related to RSE API?

RSE API Plugin version:
Zowe CLI version:
Node.js version:

Logs attached: no

Problem Description
Transferring a file to USS using the "Run IBM User build" command systematically fails when using a ZOSMF-type Zowe profile.
This seems related to interpreting the .gitattributes file to extract the codepage to use for uploading and tagging the file to USS.

Declaring encoding ibm-1147

Uploading file c:\Trv\DepotsGIT\environn-techleads-sandbox\src\S9T\COBOL\minimal.cbl to /u/J070188/vscode/environn-techleads-sandbox/src/S9T/COBOL/minimal.cbl with encoding ibm-1147
Stopping User Build. An error occurred uploading required files: Invalid character in header content ["X-IBM-Data-Type"]

Declaring encoding 1147

Uploading file c:\Trv\DepotsGIT\environn-techleads-sandbox\src\S9T\COBOL\minimal.cbl to /u/J070188/vscode/environn-techleads-sandbox/src/S9T/COBOL/minimal.cbl with encoding 1147
Stopping User Build. An error occurred uploading required files: Invalid character in header content ["X-IBM-Data-Type"]

Declaring encoding IBM-1147

Uploading file c:\Trv\DepotsGIT\environn-techleads-sandbox\src\S9T\COBOL\minimal.cbl to /u/J070188/vscode/environn-techleads-sandbox/src/S9T/COBOL/minimal.cbl with encoding IBM-1147
Stopping User Build. An error occurred uploading required files: Invalid character in header content ["X-IBM-Data-Type"]

Declaring encoding IBM-1047

Uploading file c:\Trv\DepotsGIT\environn-techleads-sandbox\src\S9T\COBOL\minimal.cbl to /u/J070188/vscode/environn-techleads-sandbox/src/S9T/COBOL/minimal.cbl with encoding IBM-1047
Stopping User Build. An error occurred uploading required files: Invalid character in header content ["X-IBM-Data-Type"]

Declaring no encoding

Uploading file c:\Trv\DepotsGIT\environn-techleads-sandbox\src\S9T\COBOL\minimal.cbl to /u/J070188/vscode/environn-techleads-sandbox/src/S9T/COBOL/minimal.cbl
Using SSH profile GMVS-SSH
Executing SSH command:
chtag -tc undefined /u/J070188/vscode/environn-techleads-sandbox/src/S9T/COBOL/minimal.cbl
@@START OF COMMAND@@
$ chtag -tc undefined /u/J070188/vscode/environn-techleads-sandbox/src/S9T/COB
$
$ /environn-techleads-sandbox/src/S9T/COBO                                    <L/minimal.cbl
FSUMF174 chtag: You must enter a codeset name or a ccsid between 0 and 65535 (USHRT_MAX).
$ exit $?
Stopping User Build. An error occurred uploading required files: Error: Something went wrong executing SSH command chtag -tc undefined /u/J070188/vscode/environn-techleads-sandbox/src/S9T/COBOL/minimal.cbl

Hi,
Could the problem encountered be related to this other problem?
https://github.com/zowe/zowe-cli/issues/1383
What type of terminal does IBM User Build use to place its Zowe orders?

Hi,
Things seem to have gotten worse with the switch to Zowe CLI V2 and the migration of Zowe profiles from V1 to V2: now the upload of elements on USS freezes.
=====================
Running User Build...
Identified the local root workspace path as c:\Trv\DepotsGIT\environn-techleads-sandbox
---------------------
Checking the ZAPP file or workspace settings...
DBB ZAPP or workspace settings should be defined in either your repository's zapp.yaml, zapp.json or .vscode/settings.json file.
    command: /bin/echo
    buildScriptArgs: --userBuild,--workspace /u/J070188/vscode,--application environ-techleads-sandbox/src,--hlq J070188.VSCODE.USERBUILD,--outDir /u/J070188/vscode/logs
    buildScriptPath: /tmp
    additionalDependencies: application-conf
---------------------
Checking user settings...
User settings should be defined in your VS Code or Theia Settings under zopeneditor.userbuild.userSettings
    localWorkspacePath: **Optional setting is missing**
    dbbWorkspace: /u/J070188/vscode
    dbbHlq: J070188.VSCODE.USERBUILD
    dbbLogDir: /u/J070188/vscode/logs
---------------------
The default Zowe CLI profile specified as GMVS in zopeneditor.zowe setting. Will try loading that.
Using Zowe profile GMVS
(frozen...)

Hi @phaumer,
We are starting work on setting up IBM DBB. It is imperative for us that the problem described here is solved very quickly
Thanks.

@FALLAI-Denis we are not able to reproduce this. We asked the local tech sales people to contact you to explore this with you directly.

Hi @phaumer,
I discussed with the IBM France representatives who are supporting us in setting up IBM DBB about the problems encountered.
We agreed to analyze with the IBM team the problem in detail with the implementation of IBM DBB, from the beginning of June.
I will already see why the switch to Zowe CLI V2 caused a new problem...may be a problem in profile management, (Lots of issues around profile management with the move to Zowe CLI V2...).
Looking forward to, the tests made with Zowe CLI V1 show that the problem is related to the formatting of the "X-IBM-Data-Type" header from the zos-working-tree-encoding information extracted from the .gitattributes file.
If I don't provide a zos-working-tree-encoding value the problem disappears (but another one appears).
So it is probably around the analysis of the .gitattributes file that the problem lies.
The workspace files use crlf line terminators, (we are on a Windows platform), the zos-working-tree-encoding parameter is the last of the line. Wrongly extracting the "cr" character in the value?
VS Code workspace uses UTF-8 encoding by default.
The .gitattributes file is encoded in iso-8859-1.
Is there a way to see what Z Open Editor builds as a command and sends to z/OSMF (because it's z/OSMF that refuses the command), either locally on the workstation or by analyzing the network traffic, on the workstation, or on the z/OS system (in the past I have already taken network frame captures on z/OS... I am looking to see if I can find the procedure for dumping network frames) .

@FALLAI-Denis you are right CRLF seems to be the problem. As a workaround you could convert your .gitattributes file locally to LF before running a build. (Just click on the CRLF in VS Code at the bottom-right, select LF and Save. You do not need to commit it as it just reads locally and use VS Code to revert the change once you finished building.)
We want to provide a fix.

Hi @phaumer
I adapted my .gitattributes file to be with "lf" line endings:

It appears in VS Code as "Windows 1252", but it is "iso-8859-1".
I am under Zowe CLI V2.
I declared a zowe.config.json file and a zowe.config.user.json file at the root of the Workspace.
Confer Zowe CLI V2 - Using environment variables in profile properties
Apparently Z Open Editor ignores these files present in the root of the Workspace, and uses those declared in my home directory, or combine both:
  - Zowe CLI API version used for profile management: 7
  - Zowe CLI Secure Credentials enabled: true
  - Zowe CLI profiles of types rse, zosmf or ssh found: GMVS(zosmf),EMVS(zosmf),EMVS-JENKINS(zosmf),GMVS-JENKINS(zosmf),SYSB-JENKINS(zosmf),TMVS-JENKINS(zosmf),SYSB(zosmf),TMVS(zosmf),GMVS-SSH(ssh),TMVS-SSH(ssh)
  - Zowe CLI files profile name specified in VS Code user settings: GMVS
  - Zowe CLI SSH profile name specified in VS Code user settings: GMVS-SSH
  - Zowe CLI files profile used for remote include files: GMVS
  - Zowe CLI SSH profile used: GMVS-SSH

Only "GMVS**" profiles are present in the zowe.config.json file of the Workspace... the other profiles are declared in the zowe.config.json file of my home directory, and should not be taken into account?
Anyway, the IBM User Build commands asks for credentials on each activation, even if they are saved, and freeze on the z/OSMF profile usage test, and this since the switch to Zowe CLI V2.

After going through Zowe Explorer, the credentials were again requested and saved, and no longer requested afterwards (stopping / restarting VS Code).
Back in Z Open Editor, credentials are no longer requested... but IBM User Build commands still freeze on z/OSMF profile test.
I deleted the zowe.config.json file present in my home directory.
Z Open Editor now only sees profiles defined in the Workspace zowe.config.json file:
  - Zowe CLI API version used for profile management: 7
  - Zowe CLI Secure Credentials enabled: true
  - Zowe CLI profiles of types rse, zosmf or ssh found: GMVS(zosmf),GMVS-SSH(ssh)
  - Zowe CLI files profile name specified in VS Code user settings: GMVS
  - Zowe CLI SSH profile name specified in VS Code user settings: GMVS-SSH
  - Zowe CLI files profile used for remote include files: GMVS
  - Zowe CLI SSH profile used: GMVS-SSH

IBM User Build commands always freeze... and don't ask for credentials...

I found the reason for the freezing the procedure!
After a very long time, a message appears in the log: Timed out waiting for password.
There is no prompt for entering user/password in VS Code...
Welcome to IBM User Build. Run User Build from the pop-up menu in an open editor with a COBOL, PL/I, or HLASM program.
Visit our IBM Docs to learn about IBM User Build here: https://www.ibm.com/docs/en/z-modernization-stack/2022.1.1?topic=code-building-cobol-pli-hlasm-programs-user-build.
=====================
Running setup for User Build...
Setup will upload files or folders that are defined in your ZAPP file or in .vscode/settings.json under "zopeneditor.userbuild.userSettings.additionalDependencies".
Checking the ZAPP file or workspace settings...
DBB ZAPP or workspace settings should be defined in either your repository's zapp.yaml, zapp.json or .vscode/settings.json file.
    command: /bin/echo
    buildScriptArgs: --userBuild,--workspace /u/J070188/vscode,--application environ-techleads-sandbox/src,--hlq J070188.VSCODE.USERBUILD,--outDir /u/J070188/vscode/logs
    buildScriptPath: /tmp
    additionalDependencies: application-conf
---------------------
Checking user settings...
User settings should be defined in your VS Code or Theia Settings under zopeneditor.userbuild.userSettings
    localWorkspacePath: **Optional setting is missing**
    dbbWorkspace: /u/J070188/vscode
    dbbHlq: J070188.VSCODE.USERBUILD
    dbbLogDir: /u/J070188/vscode/logs
---------------------
The default Zowe CLI profile specified as GMVS in zopeneditor.zowe setting. Will try loading that.
Using Zowe profile GMVS
An error occurred: Timed out waiting for password.

Fixed in v2.0.3

Hi @phaumer,
I update to 2.0.3.
Sorry, but IBM User Build functions are still freezing since I upgraded to Zowe CLI V2.
The initial problem may have been solved but I can't test it (except to go back to Zowe CLI V1).
Do you want me to open a different issue to track the crash case with Zowe CLI V2?

@FALLAI-Denis yes, please. Let us know how to reproduce it.

Hey @FALLAI-Denis there is a new feature in the Zowe CLI that allows the use of home and project dir config file profiles at the same time and is why you see all available.  If the names are the same only the project level will be available of that particular name due to the precedence.
For the prompting of credentials in Zowe Explorer, if the autoStore: true in config file you should only be prompted the one time and then the values are saved to the keystore and in turn will be used across both Zowe Explorer and Z Open Editor.  The issue I am seeing with the User Build is the ssh profile password.  Zowe Explorer is saving the zosmf one for you but if your config file is like the one shared in the Zowe CLI issue the ssh password is stored in it's own secure array and not shared across profiles like if using the base profile.  What you can do to store the ssh password in the keystore you can run the command while in the project directory zowe config set --secure "profiles.host1.profiles.service2.properties.setting" "value" --user-config changing the set path to match the config file in place and value for the password, for example using the config shared in the Zowe CLI issue should be profiles.GMVS-SSH.properties.password


Add EFS example

After deploying EFS myself and seeing https://github.com/cloudtools/troposphere/issues/580 decided to give back and add an EFS example.
I purposely made it a a reasonably interesting example with security groups, IAM artifacts etc since that's probably a more realistic real-world example. The examples have helped me tremendously in the past but you often have to pull info from several different ones to get a full picture.

Awesome...thanks!


feat: Make git commits more readable

Git commits should list all files created/updated (with newlines breaks for readability)

So we actually recently just removed specifying files all together as part of a refactor, which might be a more intuative outcome. Leveraging git CLI or your git providers website to see what files have changed as part of a commit feels better than reading the list in the commit message

cc @catmo-syntasso


Add Java 9+ modules

There are two distinct modules in this project, the scram-common which contains the SCRAM messages and utilities/functions, and the scram-client module.
Add Java modules using MR-JARs:

com.ongres.scram.common
com.ongres.scram.client

@jorsol  This is one of the dependencies I need to have a stable module name to resolve https://github.com/pgjdbc/pgjdbc/issues/2657
Let me know if there is any way i can help

Hi @bowbahdoe, there will be a release sooner than later that will be fully modular.
Yet, if you look closely at PgJDBC, this dependency is shaded, so it will not "require" the module-info for the shaded distribution.