Support for multiple Terraform modules.

Because of its performance characteristics, a best-practice at every org I've been involved in is to have multiple separate Terraform modules.  That could be per-environment, or per-environment-per-system.  In my case, I have 3 modules for AWS (one per env), 3 for GCP (one per env; soon to be 4), and so on (we also use the DataDog, and Github providers with plans to add a couple more).  Looking at the source to this action, I don't see a means to handle this.
If such a mechanism exists, can you point me at relevant documentation?  If such a mechanism does not exist, I'd love to see it added.

@MrJoy thanks! I added this doc to explain what happens just now (https://www.infracost.io/docs/terraform_modules)
Are you wanting to see a cost difference for a specific module or are you wanting to see how the change affects each different module, i.e. "This change will increase the monthly cost on prod by X%" and "This change will increase the monthly cost on dev by X%".? To show a diff for each module individually one workaround at the moment could be to run the action twice with different inputs.

@MrJoy thanks! I added this doc to explain what happens just now (https://www.infracost.io/docs/terraform_modules)
Are you wanting to see a cost difference for a specific module or are you wanting to see how the change affects each different module, i.e. "This change will increase the monthly cost on prod by X%" and "This change will increase the monthly cost on dev by X%".? To show a diff for each module individually one workaround at the moment could be to run the action twice with different inputs.

Showing change per-module (on modules whose source or dependencies have changed) seems a reasonable option.  Can the same action be added multiple times with different inputs?  I'll see about giving that a shot once I have time to circle back around to CI/CD stuff.

Showing change per-module (on modules whose source or dependencies have changed) seems a reasonable option.  Can the same action be added multiple times with different inputs?  I'll see about giving that a shot once I have time to circle back around to CI/CD stuff.

Ah, sorry.  I think I've confused the issue by abusing terminology.
When I said "module", I misspoke.  I mean separate Terraform projects -- I.E. things with separate state files.  There are modules shared between them, and I don't especially care about the modules individually.  I only care about seeing changes as they pertain to the array of separate Terraform states that I have.

Ah, sorry.  I think I've confused the issue by abusing terminology.
When I said "module", I misspoke.  I mean separate Terraform projects -- I.E. things with separate state files.  There are modules shared between them, and I don't especially care about the modules individually.  I only care about seeing changes as they pertain to the array of separate Terraform states that I have.

To be a bit more explicit, this is a subset of what our infra repo's layout looks like (the .terraform directories are not versioned, they're included for illustrative purposes):
...
./gcp
./gcp/environments
./gcp/environments/development
./gcp/environments/development/.terraform # Has its own state on GCP.
./gcp/environments/meta
./gcp/environments/meta/.terraform # Has its own state on GCP.
./gcp/environments/stage
./gcp/environments/stage/.terraform # Has its own state on GCP.
./gcp/modules # Shared modules used by multiple environments...
...
./aws
./aws/environments
./aws/environments/prod
./aws/environments/prod/.terraform # Has its own state on AWS.
./aws/environments/meta
./aws/environments/meta/.terraform # Has its own state on AWS.
./aws/environments/stage
./aws/environments/stage/.terraform # Has its own state on AWS.
./aws/modules # Shared modules used by multiple environments...
...

To be a bit more explicit, this is a subset of what our infra repo's layout looks like (the .terraform directories are not versioned, they're included for illustrative purposes):
...
./gcp
./gcp/environments
./gcp/environments/development
./gcp/environments/development/.terraform # Has its own state on GCP.
./gcp/environments/meta
./gcp/environments/meta/.terraform # Has its own state on GCP.
./gcp/environments/stage
./gcp/environments/stage/.terraform # Has its own state on GCP.
./gcp/modules # Shared modules used by multiple environments...
...
./aws
./aws/environments
./aws/environments/prod
./aws/environments/prod/.terraform # Has its own state on AWS.
./aws/environments/meta
./aws/environments/meta/.terraform # Has its own state on AWS.
./aws/environments/stage
./aws/environments/stage/.terraform # Has its own state on AWS.
./aws/modules # Shared modules used by multiple environments...
...

So I would ideally see, like you said, GCP Development changed by X% if I change source files in gcp/environments/development, or anything in gcp/modules that's used from gcp/environments/development.

So I would ideally see, like you said, GCP Development changed by X% if I change source files in gcp/environments/development, or anything in gcp/modules that's used from gcp/environments/development.

hmm this smells like https://github.com/infracost/infracost/issues/188, so if report_all.sh could be run in CI with table output, then we could do the git diff on that combined report, and show the "changed by X%". @MrJoy am I right or am I right? (lol, I'm probably missing something)

hmm this smells like https://github.com/infracost/infracost/issues/188, so if report_all.sh could be run in CI with table output, then we could do the git diff on that combined report, and show the "changed by X%". @MrJoy am I right or am I right? (lol, I'm probably missing something)

Mostly.  In our case, we're using Terraform providers other than just GCP and AWS (Github, Datadog), so including all directories not named modules containing .tf files is going to lead to a lot of needless CPU-tim and complexity around secrets management / errors, etc.

Mostly.  In our case, we're using Terraform providers other than just GCP and AWS (Github, Datadog), so including all directories not named modules containing .tf files is going to lead to a lot of needless CPU-tim and complexity around secrets management / errors, etc.

@MrJoy we're looking at adding a config file you can pass to Infracost where you can specify multiple projects and combine them into one report/PR comment with any changes grouped by the project. Would something like this help your use case?
log_level: warn
no_color: true

projects:
  terraform:
    - name: AWS Development
      dir: aws/environments/development

    - name: AWS Stage
      dir: aws/environments/meta

    - name: AWS Production
      dir: aws/environments/prod

    - name: GCP Development
      dir: gcp/environments/development

    - name: GCP Stage
      dir: gcp/environments/meta

    - name: GCP Production
      dir: gcp/environments/prod

outputs:
  - format: table
    show_skipped: true
    group_by: project

That would be perfect, yes.

@MrJoy we shipped the config-file feature in the CLI, docs are here. I've added support in this GH Action but haven't shipped the docs for it yet, can you possibly test it? We haven't added the "breakdown by project or module yet" feature yet, we're working on change to the underlying data structures to enable that...
In your github action yml, you can use the config like this to test it:
    - name: Run infracost diff
      uses: infracost/infracost-gh-action@master
      env:
        INFRACOST_API_KEY: ${{ secrets.INFRACOST_API_KEY }}
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        config_file: infracost.yml # path to the config file in the repo, file needs to be in the master branch too
      with:
        entrypoint: /scripts/ci/diff.sh # Do not change

Awesome, thank you!

A cursory glance looks good.  I still need to make time to finish the relevant setup here.  That mostly boils down to doing the leg-work to figure out the minimum privilege IAM setup needed for this.

Makes sense. One of the users was using AWS assume-role and noted this in case it helps.

I appreciate the tip, but that's not really where I'm at.  I basically am trying to come up with a policy that allows all Describe*/List*/etc commands, plus read-access to the state file without accidentally allowing reading of any actual application data.  I'm not entirely sure that all services follow the naming convention strictly enough for pattern-matching on Describe* and List* to be both complete and safe.  I'm operating on the assumption that such a policy would be sufficient to run terraform plan.  Worst-case I just make a policy that's tailored for the tiny subset of services we use, but I was hoping to make something more general so I don't get tripped up every time we make use of a new service.

.... however, this appears to be super useful: https://github.com/iann0036/iamlive

Ok!  I found something.  You might wanna add this to the docs -- fleshed out for when DynamoDB is in use, etc.
This applies if you are storing TF state in S3, not using DynamoDB for locking, and using a non-default KMS key to encrypt said bucket (that last part had me pulling my hair out for 2 hours, because the error said I lacked permission for GetObject!).
resource "aws_iam_user" "infracost" {
  name = "infracost"
  path = "/operations"
}

# Generate API key / security key:
resource "aws_iam_access_key" "infracost" {
  user = aws_iam_user.infracost.name
}

# Give read-only access to a KMS-encrypted S3 bucket containing Terraform state:
resource "aws_iam_user_policy" "infracost" {
  name = "infracost-terraformstate-readonly"
  user = aws_iam_user.infracost.name

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        "Effect": "Allow",
        "Action": "kms:Decrypt",
        "Resource": "<relevant ARN>"
      },
      {
        "Effect": "Allow",
        "Action": [
          "s3:GetBucketLocation",
          "s3:ListAllMyBuckets"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:ListBucket",
        "Resource": "<relevant S3 bucket ARN>"
      },
      {
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "<relevant S3 bucket ARN>/*"
      }
    ]
  })
}

# Give access to enumerate every kind of resource on AWS:
resource "aws_iam_user_policy_attachment" "infracost_everything_audit" {
  user       = aws_iam_user.infracost.name
  policy_arn = "arn:aws:iam::aws:policy/SecurityAudit"
}

Arg.  So.  Because the TF state contains secrets, and the GH action involves running code I can't control/audit reliably...  Basically, I can't use Infracost for the time being.  We have to deal with a security audit before we ship, and using the GH action basically breaks chain-of-custody.  It would be fine if I had the time to set things up on our own infrastructure where I could use artifact management to deploy only trusted builds of the Infracost code and the like, but we don't have time for that right now.  Sorry.  I hope the above IAM learnings are useful in helping onboard people, and maybe you can lean on the Terraform folks to find a way to move secrets out of the state file, but in the meantime I'm just gonna have to stick with some basic budget alerts.

@MrJoy would something like this work:

use the Terraform GH Action to generate a plan JSON file.
use these greps ("Finding hard-coded credentials in your code") to remove AWS creds from the plan.json.
Run the Infracost GH Action with path=plan.json.

But step 2 won't remove all secrets, just AWS creds, so maybe that's not good enough.

@alikhajeh1 No, using the Terraform GH action to generate a plan file requires having access to the TF state file, and we have no chain of custody with HashiCorp either.  Code we do not control, running on infrastructure we do not control is only a viable option if we have a contractual agreement with the owner of said code/infrastructure that obligates them to us in certain relevant ways.  (E.G. they pass certain certifications, are obliged to inform us promptly of breaches, etc.)
At some point I might set something up where we have our own GH hook that runs terraform plan from our infrastructure and makes it available somewhere that a GH action could subsequently access it but that's not something we have the time to implement right now.  At that point I might as well just have the server run Infracost itself, of course.  I can always audit the Infracost code, and build a binary from a known-good commit.
As for removing secrets:  The TF state is complex, and the exact shape of things is provider-specific AFAICT.  Those regexes will only cover strings that look like an AWS access key / secret key but won't necessarily cover every secret that needs to be protected.  Making something that can comprehensively scrub the state file is an uphill battle and runs the risk of corner-cases going unnoticed.
For example, one of things in our state file is the body of a Secrets Manager document.  It's used to define the passwords for RDS instances, among other things.  Terraform ensures the RDS instance PW matches what's in that secret.  But it's a JSON document embedded as a string field in another JSON document, so (double-)escaping comes into play given that some of our auto-generated passwords have characters that have semantic meaning in JSON.  That prevents us from just blindly doing a search-and-replace of all known secret values.  And even if we did account for escaping and double-escaping, what happens if Terraform changes how strings are escaped in the state file to, say, base64 encode things in order to sidestep escaping entirely?  Any attempt to strip secrets is intrinsically fragile.


Mobile View Select language drop down hidden

Describe the bug
hey i want to address a issue that in the mobile view of the website navbar which is covering all the screen in that the select language drop down menu is not apprearing
To Reproduce
Steps to reproduce the behavior:

Right Click on the page
Click on the hamburger button of the navbar
See at last the select language drop down menu button is not showing

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots

Additional context
I am a GSSOC'23 contributor
I want to work on this issue please allow me to resolve this issue @akshitagupta15june

I am interested to work on this issue under GSSoC 2023 as a contributor


[Dev comm][Link in see also for people picker component]

Learn Build status updates of commit 0fad395:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/concepts/device-capabilities/people-picker-capability.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


[DROOLS-1010] fix equality of MvelConstraints when constraints are used in rules with different packages

+1 to merge

Merged by https://github.com/droolsjbpm/drools/commit/e67d1ca3c


Add QR code scanner example to dreamhouse

What does this PR do?
Adds an example of the mobile Barcode (QR Code) Scanner functionality
The PR fulfills these requirements:
[x] Tests for the proposed changes have been added/updated.
[x] Code linting and formatting was performed.
Functionality Before
No QR code scanning available
Functionality After
On the Properties tab:

When on desktop, user is shown instructions to open Dreamhouse on their mobile phone to access the scanner
When on mobile, user is able to use the barcode scanner to scan a QR code to view that the corresponding property's record:

@tessaSAC yes please, this is the preferred merge mode.


[Feature Request] OpenCV

Checklist

[X] I searched exising issues and no one else requests similar feature.
[X] I think that 25%+ users are positive towards this feature.

Describe the feature
https://github.com/bytedeco/javacv
https://blog.csdn.net/qianlixiaomage/article/details/108416400
Additional context
No response

https://mvnrepository.com/artifact/org.bytedeco/opencv/4.10.0-1.5.11

https://blog.csdn.net/eguid_1/article/details/108010456

https://www.cnblogs.com/dongye95/p/17773211.html

https://www.cnblogs.com/stjwy/p/13678635.html

https://blog.csdn.net/lilinhai548/article/details/143068899

https://github.com/bytedeco/javacv-examples

Hello @unknowIfGuestInDream. This issue is marked as deprecated and closed.


add continueOrElseRoot, cross-compile for 2.11

This adds a continueOrElseRoot method that tries to continue a trace, and if the headers aren't present it will fall back to creating a new root span. Also adds 2.11 support, which I will be happy to ditch at the slightest provocation but it works fine for now.

Nice addition! :)


chore: fix linting in config/redis template

Proposed changes
When testing out the Redis 4.0.1 package I noticed a linting error in the config/redis.ts template with a missing comma. This tiny PR adds it.
Types of changes
N/A
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have read the CONTRIBUTING doc
[x] Lint and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works.
[ ] I have added necessary documentation (if appropriate)

Further comments
Nothing to add.

Thanks 😄


BUG validating collective with GH

Message from David:

I tried to create a new collective but the form was not able to validate and go to the next page.

When I reloaded the page, I found that few projects were created (I clicked on the button few times).
Are you able to remove:

https://opencollective.com/front-end-checklist-211
https://opencollective.com/front-end-checklist-330
https://opencollective.com/front-end-checklist-417
https://opencollective.com/front-end-checklist-703
https://opencollective.com/front-end-checklist-8

closing this since it duplicates #555


Add unifi

Ran Plan for project: local-lhr-unifi dir: infrastructure/local/lhr/unifi workspace: terragrunt
Show Output
17:53:48.116 INFO   Downloading Terraform configurations from ../../.. into ./.terragrunt-cache/XZA3y1f-u_Qoa7vNbD_TACwt1Iw/OX1egdcJ2n6cYa8KkD2-Uf-O7KA
17:53:48.485 STDOUT tofu: Initializing the backend...
17:53:48.730 STDOUT tofu:
17:53:48.730 STDOUT tofu: Successfully configured the backend "s3"! OpenTofu will automatically
17:53:48.730 STDOUT tofu: use this backend unless the backend configuration changes.
17:53:48.811 STDOUT tofu: Initializing provider plugins...
17:53:48.811 STDOUT tofu: - Reusing previous version of integrations/github from the dependency lock file
17:53:48.991 STDOUT tofu: - Reusing previous version of vexxhost/uptimerobot from the dependency lock file
17:53:49.040 STDOUT tofu: - Reusing previous version of akerl/unifi from the dependency lock file
17:53:49.107 STDOUT tofu: - Reusing previous version of hashicorp/aws from the dependency lock file
17:53:49.194 STDOUT tofu: - Reusing previous version of backblaze/b2 from the dependency lock file
17:53:49.247 STDOUT tofu: - Reusing previous version of cloudflare/cloudflare from the dependency lock file
17:53:50.357 STDOUT tofu: - Installing cloudflare/cloudflare v4.43.0...
17:53:51.751 STDOUT tofu: - Installed cloudflare/cloudflare v4.43.0 (signed, key ID C76001609EE3B136)
17:53:52.829 STDOUT tofu: - Installing integrations/github v6.2.2...
17:53:53.899 STDOUT tofu: - Installed integrations/github v6.2.2 (signed, key ID 38027F80D7FD5FB2)
17:53:54.979 STDOUT tofu: - Installing vexxhost/uptimerobot v0.8.2...
17:53:55.851 STDOUT tofu: - Installed vexxhost/uptimerobot v0.8.2. Signature validation was skipped due to the registry not containing GPG keys for this provider
17:53:56.958 STDOUT tofu: - Installing akerl/unifi v0.41.10...
17:53:57.930 STDOUT tofu: - Installed akerl/unifi v0.41.10. Signature validation was skipped due to the registry not containing GPG keys for this provider
17:53:58.951 STDOUT tofu: - Installing hashicorp/aws v5.70.0...
17:54:17.751 STDOUT tofu: - Installed hashicorp/aws v5.70.0 (signed, key ID 0C0AF313E5FD9F80)
17:54:18.904 STDOUT tofu: - Installing backblaze/b2 v0.8.12...
17:54:21.261 STDOUT tofu: - Installed backblaze/b2 v0.8.12. Signature validation was skipped due to the registry not containing GPG keys for this provider
17:54:21.262 STDOUT tofu: Providers are signed by their developers.
17:54:21.262 STDOUT tofu: If you'd like to know more about provider signing, you can read about it here:
17:54:21.262 STDOUT tofu: https://opentofu.org/docs/cli/plugins/signing/
17:54:21.264 STDOUT tofu: OpenTofu has made some changes to the provider dependency selections recorded
17:54:21.264 STDOUT tofu: in the .terraform.lock.hcl file. Review those changes and commit them to your
17:54:21.265 STDOUT tofu: version control system if they represent changes you intended to make.
17:54:21.283 STDOUT tofu: OpenTofu has been successfully initialized!

17:54:38.816 STDOUT tofu: No changes. Your infrastructure matches the configuration.
17:54:38.816 STDOUT tofu: OpenTofu has compared your real infrastructure against your configuration and
17:54:38.816 STDOUT tofu: found no differences, so no changes are needed.

:arrow_forward: To apply this plan, comment:atlantis apply -p local-lhr-unifi

:put_litter_in_its_place: To delete this plan and lock, click here
:repeat: To plan this project again, comment:atlantis plan -p local-lhr-unifi

No changes. Your infrastructure matches the configuration.

:fast_forward: To apply all unapplied plans from this Pull Request, comment:atlantis apply

:put_litter_in_its_place: To delete all plans and locks from this Pull Request, comment:atlantis unlock

atlantis apply

Ran Apply for project: local-lhr-unifi dir: infrastructure/local/lhr/unifi workspace: terragrunt
Show Output
17:55:01.782 INFO   Downloading Terraform configurations from ../../.. into ./.terragrunt-cache/XZA3y1f-u_Qoa7vNbD_TACwt1Iw/OX1egdcJ2n6cYa8KkD2-Uf-O7KA
17:55:02.369 STDOUT tofu: Initializing the backend...
17:55:03.403 STDOUT tofu: Initializing provider plugins...
17:55:03.403 STDOUT tofu: - Reusing previous version of vexxhost/uptimerobot from the dependency lock file
17:55:03.515 STDOUT tofu: - Reusing previous version of akerl/unifi from the dependency lock file
17:55:03.571 STDOUT tofu: - Reusing previous version of hashicorp/aws from the dependency lock file
17:55:03.636 STDOUT tofu: - Reusing previous version of backblaze/b2 from the dependency lock file
17:55:03.711 STDOUT tofu: - Reusing previous version of cloudflare/cloudflare from the dependency lock file
17:55:03.772 STDOUT tofu: - Reusing previous version of integrations/github from the dependency lock file
17:55:03.875 STDOUT tofu: - Using previously-installed cloudflare/cloudflare v4.43.0
17:55:03.892 STDOUT tofu: - Using previously-installed integrations/github v6.2.2
17:55:03.905 STDOUT tofu: - Using previously-installed vexxhost/uptimerobot v0.8.2
17:55:03.922 STDOUT tofu: - Using previously-installed akerl/unifi v0.41.10
17:55:04.459 STDOUT tofu: - Using previously-installed hashicorp/aws v5.70.0
17:55:04.499 STDOUT tofu: - Using previously-installed backblaze/b2 v0.8.12
17:55:04.499 STDOUT tofu: OpenTofu has been successfully initialized!
17:55:04.499 STDOUT tofu:
17:55:04.499 STDOUT tofu: You may now begin working with OpenTofu. Try running "tofu plan" to see
17:55:04.499 STDOUT tofu: any changes that are required for your infrastructure. All OpenTofu commands
17:55:04.499 STDOUT tofu: should now work.
17:55:04.499 STDOUT tofu: If you ever set or change modules or backend configuration for OpenTofu,
17:55:04.499 STDOUT tofu: rerun this command to reinitialize your working directory. If you forget, other
17:55:04.499 STDOUT tofu: commands will detect it and remind you to do so if necessary.
17:55:09.094 STDOUT tofu:
17:55:09.095 STDOUT tofu: Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
17:55:09.095 STDOUT tofu:

Automatically merging because all plans have been successfully applied.

Locks and plans deleted for the projects and workspaces modified in this pull request:

dir: infrastructure/local/lhr/unifi workspace: terragrunt


Bump storybook from 6.3.8 to 7.0.6

Francesco had been bumping dependencies in https://github.com/rancher/dashboard/issues/4534
Storybook is left to bump: https://github.com/rancher/dashboard/pull/8715

Later on, we can look at more recent versions (as of writing: 8.2.1 is the latest) but this solves our dependencies issue.

Storybook is currently ^7.6.17.


Figure out how to ensure GPG Key installs work properly on a restrictive network

GPG Keys are installed by communicating with -- for instance -- keyserver.ubuntu.com over port 80, but they're then sent over port 11731. If port 11731 is blocked Very Bad Things occur, like PHP 5.5 being installed from Ubuntu rather than ondrej's PPA, and Percona will super refuse to install properly.
We have a few ways around this, like there's a way to bounce through a proxy, which we could provide . . or we could just ensure that a packed version of Precip comes with the keys all baked-in.
Either way, when this fails the messaging really sucks, so we should find a way to fix it.

Pull Request #40 introduces "Packaging Mode", which starts us on the path to fix this issue. Once we introduce a conditional switch to use the partially-provisioned packaged box instead of the puppetlabs box this issue should be resolved.

I get a slightly different issue related to GPG key which seem to prevent the provisioning form running successfully, it fails on all apt packages with an error like this:
DEBUG ssh: stderr: [1;31mError: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install git' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  git-man liberror-perl
Suggested packages:
  git-daemon-run git-daemon-sysvinit git-doc git-el git-email git-gui gitk
  gitweb git-arch git-bzr git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  git git-man liberror-perl
0 upgraded, 3 newly installed, 0 to remove and 127 not upgraded.
Need to get 3400 kB/3421 kB of archives.
After this operation, 21.9 MB of additional disk space will be used.
Err http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main git-man all 1:1.9.1-1ubuntu0.2
  404  Not Found [IP: <IP_ADDRESS> 80]
Err http://security.ubuntu.com/ubuntu/ trusty-security/main git-man all 1:1.9.1-1ubuntu0.2
  404  Not Found [IP: <IP_ADDRESS> 80]
Err http://security.ubuntu.com/ubuntu/ trusty-security/main git amd64 1:1.9.1-1ubuntu0.2
  404  Not Found [IP: <IP_ADDRESS> 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git-man_1.9.1-1ubuntu0.2_all.deb  404  Not Found [IP: <IP_ADDRESS> 80]

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git_1.9.1-1ubuntu0.2_amd64.deb  404  Not Found [IP: <IP_ADDRESS> 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
[1;31mError: /Stage[main]/Precip/Package[git]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install git' returned 100: Reading package lists...

Then when manually running:
sudo apt-get update
The following error pops-up:
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://apt.puppetlabs.com trusty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7F438280EF8D349F

To fix, I manually ran this on Precip box:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7F438280EF8D349F
sudo apt-get update

Then, vagrant provision on host machine.

Unfortunately we can't really test the original issue anymore because we no longer have access to a restricted network. @en-vcracan, the similar issue you encountered here should be resolved on the ubuntu-xenial branch, see PR #53.


[Core] Added some TabView unit tests

Description of Change
Added TabView unit tests to validate the BindingContext propagation.
PR Checklist

[x] Has tests (if omitted, state reason in description)
[ ] Has samples (if omitted, state reason in description)
[ ] Rebased on top of main at time of PR
[ ] Changes adhere to coding standard

[ ] Updated documentation

@jsuarezruiz can you check why the build failed?

@jsuarezruiz can you check why the build failed?

Seems it was stopped manually

The unit tests are not running for dome reason. I am going to review what is happening in this case.

@jsuarezruiz can it be related to .NET Core version we running the unit tests on?

Adding tab item causes test hang
tabView.TabItems.Add(tabViewItem1);

Did some attempts to fix the tests but I'm starting to think that something is wrong in this actual test. Because: none of the other PRs fail with these tests, so there has to be something with this one. Is something cause an infinite loop?


Incorrectly reporting unused imports for the 'as' syntax

Versions

Flake8: 3.0.4 (flake8-docstrings: 1.0.2, pydocstyle: 1.0.0, pycodestyle: 2.0.0, mccabe: 0.5.2, pyflakes: 1.2.3) CPython 3.5.1 on Darwin
Linter-flake8: 1.13.4

Code to reproduce
from sys import path as sys_path

# Nothing here, just need to room to move add lines

Reported error
F401 — 'sys.path as sys_path' imported but unused at line 5 col 1 in test.py
Expected error
F401 — 'sys.path as sys_path' imported but unused at line 1 col 17 in test.py
With the whole of path as sys_path underlined.

Just as an FYI, as part of the ES2017 re-write I'm working on I'm investigating this. I'm still not sure whether I want to keep the mass of code that is the "position fixing" and get it working properly, or just go back to the incorrect results from flake8...

Fixed in #255. Although the specs don't show it, it does work with that syntax, it's just that the hacking module forces an older version of flake8 that only marks the alias.


NetCDF: Not a valid ID when trying to retrieve values from Dask array

Hi, I am attempting to pull values from an xarray dataset to accumulate rainfall at specific times over a large number of dimensions. The dataset, concat_floods_all, is as follows:
<xarray.Dataset>
Dimensions:  (south_north: 1015, west_east: 1359)
Coordinates:
    XLAT     (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)>
    XLONG    (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)>
Dimensions without coordinates: south_north, west_east

With 658 variables (all accumulated rainfall at different times over the same domain):
Data variables:
    var0     (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)>
    var1     (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)>
...
    var658    (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)

The issue is when I sum all the variables up, using the following:
sum_floods = concat_floods_all.sum(skipna = True, dim='variable').compute()
I get the following error message:
RuntimeError: NetCDF: Not a valid ID
Based on #https://github.com/pydata/xarray/issues/1001, I believe this error is due to opening numerous files I search, and then appending them to a list in a for loop (I chose this method over mfdataset, due to combining some files and deleting redundant ones).
var_list = []
### Find files that match start and end year/month of flood (fflood_start_end) and subset based on flood duration

for c, item in enumerate(fflood_start_end):
glob.glob(os.path.join('/glade2/collections/rda/data/ds612.0/CTRL/**/wrf2d_d01_CTRL_'+var+ '_*')):
        if item in name:
            wrf_match_fflood = xr.open_dataset(name, chunks = {'Time':10})
            # pull only times of floods
            var= wrf_match_fflood[var].sel(Time = slice(date_fflood_st_dt2[c], date_fflood_end_dt2[c]))
            var_list.append(var)

I am wondering how to get the actual 1015x1359 array of values for sum_floods and work around this issue.
Thanks!

@edougherty32 - I'm not exactly sure what your problem is but I have some ideas for you:

open_mfdataset can take a list of filenames in addition to a glob string. You may be able to filter your list of filenames ahead of time and give that to open_mfdataset.
I'm thinking it may be possible to create a time dimension/coordinate in your dataset of length 659. This tends to be a more common workflow. It will also generally play nicer than 600 individual variables.
If you can switch back to open_mfdataset, you may try the autoclose=True flag. This will help with the too many open files.

@jhamman–Thanks for the suggestions, I'll definitely try some of these out.
In regards to #2, I'm guessing that involves defining a new xarray in which a new coordinate is defined using variables 0–658?

In regards to #2, I'm guessing that involves defining a new xarray in which a new coordinate is defined using variables 0–658?

Yes. My reading of your comment above made me think that each variable corresponded to a specific event/time. If that is the case, you could populate your coordinate values with the corresponding time stamps.

Thanks, @jhamman! I attempted using open_mfdataset, but that will not work for what I need to do with my data, even with filtering filenames ahead of time.
How would I go about defining a new coordinate and populating with corresponding time stamps? Would that involve using Dataset.set_coords() to create a new dataset with a coordinate of length 659 and using the data from the 659 variables?
I am still worried that will not work, since I am still receiving the same error message as before that will not load data from dask arrays into numpy arrays, as shown by this test:
print(concat_floods_all[0].values())
Which results in the following error message:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-49-a698f5985450> in <module>()
----> 1 print(eu_list_combine[0].values)

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/dataarray.py in values(self)
    403     def values(self):
    404         """The array's data as a numpy.ndarray"""
--> 405         return self.variable.values
    406
    407     @values.setter

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/variable.py in values(self)
    385     def values(self):
    386         """The variable's data as a numpy.ndarray"""
--> 387         return _as_array_or_item(self._data)
    388
    389     @values.setter

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/variable.py in _as_array_or_item(data)
    209     TODO: remove this (replace with np.asarray) once these issues are fixed
    210     """
--> 211     data = np.asarray(data)
    212     if data.ndim == 0:
    213         if data.dtype.kind == 'M':

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    490
    491     """
--> 492     return array(a, dtype, copy=False, order=order)
    493
    494

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/array/core.py in __array__(self, dtype, **kwargs)
   1190
   1191     def __array__(self, dtype=None, **kwargs):
-> 1192         x = self.compute()
   1193         if dtype and x.dtype != dtype:
   1194             x = x.astype(dtype)

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/base.py in compute(self, **kwargs)
    152         dask.base.compute
    153         """
--> 154         (result,) = compute(self, traverse=False, **kwargs)
    155         return result
    156

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/base.py in compute(*args, **kwargs)
    405     keys = [x.__dask_keys__() for x in collections]
    406     postcomputes = [x.__dask_postcompute__() for x in collections]
--> 407     results = get(dsk, keys, **kwargs)
    408     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    409

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, **kwargs)
   2096             try:
   2097                 results = self.gather(packed, asynchronous=asynchronous,
-> 2098                                       direct=direct)
   2099             finally:
   2100                 for f in futures.values():

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in gather(self, futures, errors, maxsize, direct, asynchronous)
   1506             return self.sync(self._gather, futures, errors=errors,
   1507                              direct=direct, local_worker=local_worker,
-> 1508                              asynchronous=asynchronous)
   1509
   1510     @gen.coroutine

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in sync(self, func, *args, **kwargs)
    613             return future
    614         else:
--> 615             return sync(self.loop, func, *args, **kwargs)
    616
    617     def __repr__(self):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, *args, **kwargs)
    251             e.wait(10)
    252     if error[0]:
--> 253         six.reraise(*error[0])
    254     else:
    255         return result[0]

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
    692                 raise value.with_traceback(tb)
--> 693             raise value
    694         finally:
    695             value = None

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/utils.py in f()
    236             yield gen.moment
    237             thread_state.asynchronous = True
--> 238             result[0] = yield make_coro()
    239         except Exception as exc:
    240             error[0] = sys.exc_info()

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py in run(self)
   1097
   1098                     try:
-> 1099                         value = future.result()
   1100                     except Exception:
   1101                         self.had_exception = True

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py in run(self)
   1105                     if exc_info is not None:
   1106                         try:
-> 1107                             yielded = self.gen.throw(*exc_info)
   1108                         finally:
   1109                             # Break up a reference to itself

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1383                             six.reraise(type(exception),
   1384                                         exception,
-> 1385                                         traceback)
   1386                     if errors == 'skip':
   1387                         bad_keys.add(key)

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
    690                 value = tp()
    691             if value.__traceback__ is not tb:
--> 692                 raise value.with_traceback(tb)
    693             raise value
    694         finally:

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/array/core.py in getter()
     74         c = a[b]
     75         if asarray:
---> 76             c = np.asarray(c)
     77     finally:
     78         if lock:

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/numpy/core/numeric.py in asarray()
    490
    491     """
--> 492     return array(a, dtype, copy=False, order=order)
    493
    494

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/indexing.py in __array__()
    600
    601     def __array__(self, dtype=None):
--> 602         return np.asarray(self.array, dtype=dtype)
    603
    604     def __getitem__(self, key):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/numpy/core/numeric.py in asarray()
    490
    491     """
--> 492     return array(a, dtype, copy=False, order=order)
    493
    494

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/indexing.py in __array__()
    506     def __array__(self, dtype=None):
    507         array = as_indexable(self.array)
--> 508         return np.asarray(array[self.key], dtype=None)
    509
    510     def transpose(self, order):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/backends/netCDF4_.py in __getitem__()
     64         with self.datastore.ensure_open(autoclose=True):
     65             try:
---> 66                 array = getitem(self.get_array(), key.tuple)
     67             except IndexError:
     68                 # Catch IndexError in netCDF4 and return a more informative

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.shape.__get__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dimension.__len__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Not a valid ID
Future exception was never retrieved
future: <Future finished exception=AllExit()>
Traceback (most recent call last):
  File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py", line 1357, in wait
    raise AllExit()
distributed.client.AllExit
Future exception was never retrieved
future: <Future finished exception=AllExit()>
Traceback (most recent call last):
  File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py", line 1357, in wait
    raise AllExit()
distributed.client.AllExit
Future exception was never retrieved
future: <Future finished exception=AllExit()>
Traceback (most recent call last):
  File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py", line 1357, in wait
    raise AllExit()
distributed.client.AllExit

I only get this error message when opening the files in a loop with chunking (which I need to do for efficiency and methodological purposes):
### Get subset of files based on flood duration
var_list = []

### Find files that match start and end year/month of flood and subset based on flood duration
for c, item in enumerate(fflood_start_end_Ym2_unique):
    for name in glob.glob(os.path.join('/glade2/collections/rda/data/ds612.0/CTRL/**/wrf2d_d01_CTRL_'+var+ '_*')):
        if item in name:
            wrf_match_fflood = xr.open_dataset(name, chunks = {'Time':10}, lock=False)
            # pull only times of floods
            var= wrf_match_fflood[var].sel(Time = slice(date_fflood_st_unique[c], date_fflood_end_unique[c]))
            var_list.append(var)

Again, I'm not sure how to get around these issues in my current framework, so please let me know if you have any more suggestions!

If you're interested in testing out development versions of xarray, there's a decent chance that this pull request will fix this issue:
https://github.com/pydata/xarray/pull/2261
I would be curious to know if this works.

@shoyer– Thanks, I'll try this out and let you know how it works.

@shoyer–Sorry for my own ignorance, but how do I implement the xarray.backends.file_manager within the framework of my code? Do I need to download the .py files to my directory and call them in my own script?
The functionality looks promising, but I admit that the use of these tools is somewhat new to me and I would appreicate any additional guidance. Thanks!

@edougherty32 - from inside your environment (conda or virtual env), you'll want to run something like:
source activate pangeo
pip install --upgrade --no-deps git+git://github.com/shoyer/xarray@file-manager

Thanks @jhamman!
Then to implement this in my script, should I follow something like that xarray/tests/test_backends_file_manager.py under #2261 using the test_file_manager_write_consecutive?
Again, I apologize for being new to this.

@edougherty32 - I think using @shoyer's branch (as installed using the pip command above), you should just try rerunning your failing example. The pip command above should update your version of xarray to #2261.

Great, thanks @jhamman!

Hi @jhamman and @shoyer, updating my version of xarray to #https://github.com/pydata/xarray/pull/2261 solved the issue I had mentioned above!
However, now I am having a new issue when trying to access values from a datarray, concat_floods_all_ar with the following dimensions:
<xarray.DataArray 'stack-10cc15f05bda9db8e96c68120fb29f3a' (variable: 658, south_north: 1015, west_east: 1359)>
dask.array<shape=(658, 1015, 1359), dtype=float32, chunksize=(1, 1015, 1359)>
Coordinates:
    XLAT      (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)>
    XLONG     (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)>
  * variable  (variable) <U6 'var0' 'var1' 'var2' ... 'var655' 'var656' 'var657'
Dimensions without coordinates: south_north, west_east

When I access the first variable, 'var0' values:
print(concat_floods_all_ar[0].values)
I receive a numpy array, as expected:
[[nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]
 ...
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]]

Yet, when I try to access some other variables, I receive the following error message:
print(concat_floods_all_ar[540].values)
--------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-139-58d1d3a67d8c> in <module>()
      1 #print(concat_floods_all_ar[0])
      2 print(concat_floods_all_ar[540].values)
----> 3 print(concat_floods_all_ar[541].values)
      4 print()

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/dataarray.py in values(self)
    403     def values(self):
    404         """The array's data as a numpy.ndarray"""
--> 405         return self.variable.values
    406
    407     @values.setter

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/variable.py in values(self)
    385     def values(self):
    386         """The variable's data as a numpy.ndarray"""
--> 387         return _as_array_or_item(self._data)
    388
    389     @values.setter

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/variable.py in _as_array_or_item(data)
    209     TODO: remove this (replace with np.asarray) once these issues are fixed
    210     """
--> 211     data = np.asarray(data)
    212     if data.ndim == 0:
    213         if data.dtype.kind == 'M':

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    490
    491     """
--> 492     return array(a, dtype, copy=False, order=order)
    493
    494

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/array/core.py in __array__(self, dtype, **kwargs)
   1190
   1191     def __array__(self, dtype=None, **kwargs):
-> 1192         x = self.compute()
   1193         if dtype and x.dtype != dtype:
   1194             x = x.astype(dtype)

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/base.py in compute(self, **kwargs)
    152         dask.base.compute
    153         """
--> 154         (result,) = compute(self, traverse=False, **kwargs)
    155         return result
    156

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/base.py in compute(*args, **kwargs)
    405     keys = [x.__dask_keys__() for x in collections]
    406     postcomputes = [x.__dask_postcompute__() for x in collections]
--> 407     results = get(dsk, keys, **kwargs)
    408     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    409

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, **kwargs)
   2096             try:
   2097                 results = self.gather(packed, asynchronous=asynchronous,
-> 2098                                       direct=direct)
   2099             finally:
   2100                 for f in futures.values():

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in gather(self, futures, errors, maxsize, direct, asynchronous)
   1506             return self.sync(self._gather, futures, errors=errors,
   1507                              direct=direct, local_worker=local_worker,
-> 1508                              asynchronous=asynchronous)
   1509
   1510     @gen.coroutine

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in sync(self, func, *args, **kwargs)
    613             return future
    614         else:
--> 615             return sync(self.loop, func, *args, **kwargs)
    616
    617     def __repr__(self):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, *args, **kwargs)
    251             e.wait(10)
    252     if error[0]:
--> 253         six.reraise(*error[0])
    254     else:
    255         return result[0]

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
    692                 raise value.with_traceback(tb)
--> 693             raise value
    694         finally:
    695             value = None

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/utils.py in f()
    236             yield gen.moment
    237             thread_state.asynchronous = True
--> 238             result[0] = yield make_coro()
    239         except Exception as exc:
    240             error[0] = sys.exc_info()

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py in run(self)
   1097
   1098                     try:
-> 1099                         value = future.result()
   1100                     except Exception:
   1101                         self.had_exception = True

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py in run(self)
   1105                     if exc_info is not None:
   1106                         try:
-> 1107                             yielded = self.gen.throw(*exc_info)
   1108                         finally:
   1109                             # Break up a reference to itself

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1383                             six.reraise(type(exception),
   1384                                         exception,
-> 1385                                         traceback)
   1386                     if errors == 'skip':
   1387                         bad_keys.add(key)

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
    690                 value = tp()
    691             if value.__traceback__ is not tb:
--> 692                 raise value.with_traceback(tb)
    693             raise value
    694         finally:

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/zict/buffer.py in __setitem__()
     80             self.fast[key] = value
     81         else:
---> 82             self.slow[key] = value
     83
     84     def __delitem__(self, key):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/zict/func.py in __setitem__()
     40
     41     def __setitem__(self, key, value):
---> 42         self.d[key] = self.dump(value)
     43
     44     def __contains__(self, key):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/zict/file.py in __setitem__()
     78
     79     def __setitem__(self, key, value):
---> 80         with open(os.path.join(self.directory, _safe_key(key)), 'wb') as f:
     81             if isinstance(value, (tuple, list)):
     82                 for v in value:

FileNotFoundError: [Errno 2] No such file or directory: '/gpfs/fs1/work/doughert/flooddata/dask-worker-space/worker-e19vqltp/storage/%28%27open_dataset-getitem-576ab71123d7113592b2d6da59777734%27%2C%200%2C%200%2C%200%29'

This only happens for variables for which I implement
xarray.Dataset.combine_first()
To combine two consecutive files that contain information on single flood.

Hi @jhamman and @shoyer–updating my version of xarray to #https://github.com/pydata/xarray/pull/2261 mostly solved the issue I mentioned above!
However, I am now having a new issue when trying to access values from a dataarray,
concat_floods_all_ar, with the following dimensions:
<xarray.DataArray 'stack-10cc15f05bda9db8e96c68120fb29f3a' (variable: 658, south_north: 1015, west_east: 1359)>
dask.array<shape=(658, 1015, 1359), dtype=float32, chunksize=(1, 1015, 1359)>
Coordinates:
    XLAT      (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)>
    XLONG     (south_north, west_east) float32 dask.array<shape=(1015, 1359), chunksize=(1015, 1359)>
  * variable  (variable) <U6 'var0' 'var1' 'var2' ... 'var655' 'var656' 'var657'
Dimensions without coordinates: south_north, west_east

Where each variable is accumulated precipiation over the U.S. for a particular flood case.
When I access the first variable, var0 values:
print(var_val_conus[0].values)
I receive a numpy array, as expected (thus solving the issue from above).
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]]

Yet, when I try to access other variables, I receive the following error message:
print(var_val_conus[542].values)
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-125-cc538b653f54> in <module>()
----> 1 test = concat_floods_all_ar[542].values
      2 print(np.shape(test))

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/dataarray.py in values(self)
    403     def values(self):
    404         """The array's data as a numpy.ndarray"""
--> 405         return self.variable.values
    406
    407     @values.setter

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/variable.py in values(self)
    385     def values(self):
    386         """The variable's data as a numpy.ndarray"""
--> 387         return _as_array_or_item(self._data)
    388
    389     @values.setter

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray/core/variable.py in _as_array_or_item(data)
    209     TODO: remove this (replace with np.asarray) once these issues are fixed
    210     """
--> 211     data = np.asarray(data)
    212     if data.ndim == 0:
    213         if data.dtype.kind == 'M':

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    490
    491     """
--> 492     return array(a, dtype, copy=False, order=order)
    493
    494

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/array/core.py in __array__(self, dtype, **kwargs)
   1190
   1191     def __array__(self, dtype=None, **kwargs):
-> 1192         x = self.compute()
   1193         if dtype and x.dtype != dtype:
   1194             x = x.astype(dtype)

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/base.py in compute(self, **kwargs)
    152         dask.base.compute
    153         """
--> 154         (result,) = compute(self, traverse=False, **kwargs)
    155         return result
    156

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/base.py in compute(*args, **kwargs)
    405     keys = [x.__dask_keys__() for x in collections]
    406     postcomputes = [x.__dask_postcompute__() for x in collections]
--> 407     results = get(dsk, keys, **kwargs)
    408     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    409

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, **kwargs)
   2096             try:
   2097                 results = self.gather(packed, asynchronous=asynchronous,
-> 2098                                       direct=direct)
   2099             finally:
   2100                 for f in futures.values():

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in gather(self, futures, errors, maxsize, direct, asynchronous)
   1506             return self.sync(self._gather, futures, errors=errors,
   1507                              direct=direct, local_worker=local_worker,
-> 1508                              asynchronous=asynchronous)
   1509
   1510     @gen.coroutine

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in sync(self, func, *args, **kwargs)
    613             return future
    614         else:
--> 615             return sync(self.loop, func, *args, **kwargs)
    616
    617     def __repr__(self):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, *args, **kwargs)
    251             e.wait(10)
    252     if error[0]:
--> 253         six.reraise(*error[0])
    254     else:
    255         return result[0]

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
    692                 raise value.with_traceback(tb)
--> 693             raise value
    694         finally:
    695             value = None

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/utils.py in f()
    236             yield gen.moment
    237             thread_state.asynchronous = True
--> 238             result[0] = yield make_coro()
    239         except Exception as exc:
    240             error[0] = sys.exc_info()

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py in run(self)
   1097
   1098                     try:
-> 1099                         value = future.result()
   1100                     except Exception:
   1101                         self.had_exception = True

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py in run(self)
   1105                     if exc_info is not None:
   1106                         try:
-> 1107                             yielded = self.gen.throw(*exc_info)
   1108                         finally:
   1109                             # Break up a reference to itself

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1383                             six.reraise(type(exception),
   1384                                         exception,
-> 1385                                         traceback)
   1386                     if errors == 'skip':
   1387                         bad_keys.add(key)

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
    690                 value = tp()
    691             if value.__traceback__ is not tb:
--> 692                 raise value.with_traceback(tb)
    693             raise value
    694         finally:

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/zict/buffer.py in __setitem__()
     80             self.fast[key] = value
     81         else:
---> 82             self.slow[key] = value
     83
     84     def __delitem__(self, key):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/zict/func.py in __setitem__()
     40
     41     def __setitem__(self, key, value):
---> 42         self.d[key] = self.dump(value)
     43
     44     def __contains__(self, key):

~/miniconda3/envs/pangeo/lib/python3.6/site-packages/zict/file.py in __setitem__()
     78
     79     def __setitem__(self, key, value):
---> 80         with open(os.path.join(self.directory, _safe_key(key)), 'wb') as f:
     81             if isinstance(value, (tuple, list)):
     82                 for v in value:

FileNotFoundError: [Errno 2] No such file or directory: '/gpfs/fs1/work/doughert/flooddata/dask-worker-space/worker-ndjdgimu/storage/%28%27open_dataset-getitem-05742037c2598486787e2b5accee36e9%27%2C%200%2C%200%2C%200%29'

This only happens for variables for which I previously utilized the
xarray.Dataset.combine_first()
function to combine two consecutive files that contains information on the same flood (files are grouped in 3 month increments, so it's possible that a flood starting at the end of May ends at the beginning of July, which is in a different file).
Is there a work-around on this new issue? Thanks.

Thanks for testing it out!
That pull request still needs a bit of work with dask.distributed -- it's own tests are still failing. When we get that working, it will probably be ready for another test.

@shoyer–No problem!
Ok, thanks for letting me know! Do you when that pull request would be working for another test? No rush, but I'm just curious. Thanks again for all the help!

@edougherty32 This took a while, but I think https://github.com/pydata/xarray/pull/2261 is ready for another test now.


IE not removing whitespace from email fields

I'm testing in IE and Edge on windows 10 and it seems that spaces are not being removed.  I see that spaces are getting ignored in chrome, safari, and firefox on OSX but the Microsoft browsers are sending the spaces in the actions.

I couldn't find a bug filed for IE or Edge, but I found these:

https://bugs.chromium.org/p/chromium/issues/detail?id=423785
http://jsfiddle.net/4dh8gaf9/3/
facebook/react#6368

And even a bug filed for redux-form itself: #417
Most of the time, I would say that email inputs should be trimmed.
But there are folks (like that one from the Chrome bug, and some over the React bug) which believe the input should show the same the user typed.
However, trimming is declared in the HTML specs:

The value sanitization algorithm is as follows: Strip line breaks from the value, then strip leading and trailing whitespace from the value.

So IE and Edge are not behaving the way they should;
I even created a fiddle to check this: https://jsfiddle.net/ujyo162t/2/
The best we can do is automatically add a normalizer to trim it and cope with their inconsistencies.
cc @erikras

what we ended up doing is adding a normalizer to deal with the browser inconsistency.  I do think it isn't ideal that if a user types a space that it just won't be shown but it is important to us to be consistent.

Because this issue saw almost no activity for a few months, I'm closing it.


Validation of Detected Pose w/ VISP

Once the board pose detection software approaches robustness, it may be worth doing some validation and comparison testing against existing tools. VISP may be one such tool to compare against.
@agillman-afrl suggested that @makella6315 's work on VISP over the 2021 summer may be useful in this.
Is VISP a helpful tool for what you were describing for tracking tic tac toe board?
https://visp-doc.inria.fr/doxygen/visp-daily/tutorial-tracking-mb-generic-rgbd.html

Note: This was discussed again at 2022-01-21 lab meeting with @agillman-afrl
Investigating the website, it looks like there is a lot of potential for deploying this tool in this and other projects.


Set project_level bool in CLI and/or in config file

Currently, in branch sqlmodel, there is not a way to set pipestat to report sample level or project level results using either CLI or in the config file. However, this can be accomplished when calling pipestat in python:
psm.report(values={"percentage_of_things": "10"}, strict_type=False, project_level=False)
If you give pipestat a sample_output.yaml file with both project and sample items and attempt to report a value it will give the following error:
'Exception: Cannot determine table suffix with 2 models present and no project-level flag'
Example sample output yaml:
pipeline_id: test_pipe
project:
  number_of_things:
    type: integer
    description: "Number of things"
samples:
  percentage_of_things:
    type: number
    description: "Percentage of things"

Example database_config.yaml:
namespace: test
record_identifier: sample1
schema_path: drc_sample_output_schema.yaml
database:
  dialect: postgresql
  driver: psycopg2
  name: pipestat-test
  user: postgres
  password: pipestat-password
  host: <IP_ADDRESS>
  port: 5432

Suggested solutions/additions:

Add function that will parse project_level: true within the config file such that it is passed to the pipestat object for correctly reporting either sample or pipeline results.
Add optional CLI flag to do the above.

For solution 1, add function in pipestat.py here:
https://github.com/pepkit/pipestat/blob/faad7b1780cbb397046ad734f2e492fe6d65aaf2/pipestat/pipestat.py#L185-L206

Addressed in: https://github.com/pepkit/pipestat/pull/36/commits/8cddca050c7cfae1805efa9ef98e0ecc80d8a074

Check TO DO comments:
-Ensure set_status() is using sample_level/project_level appropriately.

Some areas within pipestat do not disambiguate between sample-level and project-level

This works for API but not CLI reporting. CLI reporting will always assume sample level.

This is now fixed in Dev.


Improve test setup and reenable two tests

This is merging some changes of outdated PR https://github.com/getappmap/appmap-intellij-plugin/pull/674
This should make tests more reliable and easier to debug. Async process execution is disabled for tests now and this should avoid some of the unstable tests we've observed in the past.
The AppMap report is showing nicely that there this PR un-ignored two of the previously flaky tests :)
A user-visible fix is https://github.com/getappmap/appmap-intellij-plugin/pull/762/commits/96cc6c2cfbfc7c86fb07504eba50d207e98dd310, which fixes the restart of the JSON-RPC server after too many terminations of the process.

:tada: This issue has been resolved in version 0.73.0 :tada:
The release is available on:

v0.73.0
GitHub release

Your semantic-release bot :package::rocket:


No user:file found on centos7.4

Hi Guys:
In my targetcli, I only got:
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 0]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
| o- user:fbo ............................................................................................... [Storage Objects: 0]
| o- user:rbd ............................................................................................... [Storage Objects: 0]
| o- user:zbc ............................................................................................... [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 0]
o- loopback ......................................................................................................... [Targets: 0]
o- srpt ............................................................................................................. [Targets: 0]
I don't know why there is no user:file.
I'm sure that I take steps as readme.md described.
And thanks mike for giving me some clues. issues 571
But I didn't find the handler directory. I have no idea about how to add user:file to the targetcli.
After "make install" operation:
There exist "handler_file.so" in tcmu-runner directory.And I am not sure that if I need to eidt the config in CMakeLists.txt. I used the default configuration.
And my system info:
[root@localhost /]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
Linux localhost.localdomain 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Could anyone please give me some instructions?
Thanks a lot!
Best Regards!

@zjs1224522500
As Mike said, the cmakefile won't install it as defualt.
The following should work for you:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 0ddcdde..6dbae02 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -168,6 +168,7 @@ set_target_properties(handler_file
 target_include_directories(handler_file
   PUBLIC ${PROJECT_SOURCE_DIR}/ccan
   )
+install(TARGETS handler_file DESTINATION ${CMAKE_INSTALL_LIBDIR}/tcmu-runner)

 if (with-fbo)
        # Stuff for building the file optical handler

Thanks,
BRs

@lxbsz Thanks firstly!
Before solving the problem issued in this issue, I met another trouble.
When I run the ./tcmu-runner, the command line was like (pending):
[root@localhost tcmu-runner]# ./tcmu-runner
log level now is DEBUG
Attempting to use '/var/log/tcmu-runner.log' as the log file path
log file path now is '/var/log/tcmu-runner.log'
Starting...
And I vi the log file in /var/log/tcmu-runner.log. There exist one error.
2019-07-14 21:22:06.479 15495 [DEBUG] load_our_module:537: Module 'target_core_user' is already loaded
2019-07-14 21:22:06.479 15495 [DEBUG] main:1106: handler path: /usr/lib64/tcmu-runner
2019-07-14 21:22:06.479 15495 [DEBUG] main:1112: blocking netlink
2019-07-14 21:22:06.480 15495 [ERROR] tcmu_cfgfs_set_str:294: Kernel does not support configfs file /sys/module/target_core_user/parameters/block_netlink.
2019-07-14 21:22:06.480 15495 [DEBUG] main:1115: blocking netlink done
2019-07-14 21:22:06.492 15495 [DEBUG] main:1133: 5 runner handlers found
2019-07-14 21:22:06.492 15495 [DEBUG] dyn_config_start:427: Inotify is watching "/etc/tcmu", wd: 1, mask: IN_MODIFY
2019-07-14 21:22:06.496 15495 [DEBUG] dbus_bus_acquired:450: bus org.kernel.TCMUService1 acquired
2019-07-14 21:22:06.497 15495 [DEBUG] dbus_name_acquired:466: name org.kernel.TCMUService1 acquired
After I rebuild the project for serveral times, it still exits. I am really confused.
Could you please help give me some clues?
Thx & BR

@zjs1224522500
This should due to your kernel version is a little bit low, which doesn't contain the following patch:
https://patchwork.kernel.org/patch/10483079/
You can upgrade you kernel version to >= kernel-3.10.0-927.el7, or some other version of 3.10.0-862.X.Y.el7.x86_64 and I am not sure from which version has it.
But this ERROR just one warning message, without it the tcmu-runner should still work for you and it shouldn't block your basic use normally. It is just fixing some bugs.
Thanks,
BRs

But I didn't find the handler directory. I have no idea about how to add user:file to the targetcli.

When you do make install you should see where things are installed to if you did not pass in a path:
make install
...
-- Installing: /usr/lib64/tcmu-runner/handle_xyz.so
..
If you are already did a make install then you see
-- Up-to-date: /usr/lib64/tcmu-runner/handler_xyz.so
So in both cases about the tcmu-runner handler dir is /usr/lib64/tcmu-runner

@lxbsz Thanks for your help! But this ERROR blocked my basic use. If I kept the old version of the kernel, I can't start the tcmu-runner successfully. And I can't use the functions provided by tcmu-runner.
In targetcli, the backstores will be like this (without user:xyz):
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- lun1 ........................................................................... [/dev/sdb (278.5GiB) write-thru activated]
| |   o- alua ................................................................................................... [ALUA Groups: 1]
| |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 2]
| | o- FILE1 .................................................................... [/root/tcmu_file (50.0MiB) write-back activated]
| | | o- alua ................................................................................................... [ALUA Groups: 1]
| | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| | o- FILE2 .................................................................. [/root/tcmu_file_2 (50.0MiB) write-back activated]
| |   o- alua ................................................................................................... [ALUA Groups: 1]
| |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
So I can't use the self-define tcmu_handler.
Must I update the version of Kernel to solve the problem?
Thx & BR

@zjs1224522500
To be support the tcmu better, you'd better upgrade the kernel. We are fixing many bugs and adding many new features in later release versions.
Yours is 3.10.0-693.el7.x86_64, this is two low, not sure will it work well. We usually limit to use >= 3.10.0-862.14.4.el7.x86_64 for some reason.
As Mike mentioned, please check your install location:
# ls  /usr/lib64/tcmu-runner/
If you are not doing it like:
# cmake -DSUPPORT_SYSTEMD=ON -DCMAKE_INSTALL_PREFIX=/usr .
Then please make sure you setup is clean, it should be nothing in:
# ls /usr/local/lib64/tcmu-runner/
Thanks.

@lxbsz
I am really sorry to have a misunderstanding about the tcmu-runner process.
Because it will end if I execute ./tcmu-runner when the process is already running, so I thought the bash will run in backend automatically. And the function is ok in my old version kernel. And I will try to upgrade our kernel version.
Now all issues have been fixed! Thanks for your help and enthusiasm.
Also thank @mikechristie. Thanks for your great work!
Thx & BR


NUMBERS-122: Make assertFraction(long, long, BigFraction) compare the…

… full numerator and denominator

Coverage remained the same at 94.196% when pulling 3697cffeeb966556bd9a815dd75ec07ffd0fa159 on Schamschi:NUMBERS-122 into 860adcdb865842f673ba5010ebcbfd501abcff8b on apache:master.


CertAuth support

Added CertAuth support
Added tests for CertAuth and some tests for BasicAuth
Modified README.md with an example using CertAuth and how to use it on the command-line

Note: When the SSL certificate is generated for winRM the UPN extension must not be marked as critical because it can not be validated by GO.

Thanks for your contribution.
Can you please squash all the commits in one?
Also, make sure you change the commit message so that it shows what this change brings, why it's useful, and how it works.
I'll then do a proper review of the commits.
Thanks!

@masterzen: Is there any problem?

@claudiu-coblis, I'm very sorry I haven't had the time to do the review I promised you. I will do it ASAP.

@masterzen, can you do the review please?

How necessary is replacing tls and x509 with their gwacl forks?  I ask because, although I want to use this code (since the cloudbase fork of winrm, which does have cert auth), it was a pain replumbing other parts of my project with tls/x509 from gwacl.
The other thing I would suggest (which is in my own code) is adding the ability to just pass a *http.Transport into a winrm.NewClient call (mine's called NewClientWithParametersAndTransport).  This makes it easy to sling your own tls.Config object around.

Hi @claudiu-coblis,
First, thank you for this contribution, still I have two concerns about this patch:

use of a fork of http and tls golang library
a new parameter for NewClient

I think we can solve the second issue quite easily, but I'm quite concerned about the first one, and I'd like to hear your reasons of using this fork.
Thanks!

@masterzen Well, I'm not @claudiu-coblis , but I can address the "why the fork" problem.
https://github.com/golang/go/issues/5742 basically has a pretty complete history; the short answer is, for Certificate Auth to work against WinRM, TLS needs to support renegotiation, and it does not.  Prior to @claudiu-coblis pointing to the issue page I didn't know the history, which is why I had the same question yesterday.
Essentially, the Golang maintainers hate supporting renegotiation because it's broken, weird, and basically only there to support Microsoft.  Which, well, all true, but talking to MS systems is something our customers are demanding, and I'm sure we're not alone here.

Yes, I wasn't aware of the ticket too, so thanks @claudiu-coblis for pointing us in this direction. I understand that managing Azure is something important, so I'd like this PR to be merged in.
I'd like to gather @dylanmei , @sneal, @mefellows advice on this particular topic. What do you all think about using this forked http/tls subsystem until Golang decide to fix the issue?

I agree we don't want to modify the API beyond adding to Endpoint.
Regarding TLS renegotiation:  The winrm package is used by Packer and Terraform. At this time there are no asks from that community for Windows HTTPS certificate authentication. While I haven't tested it yet, I don't perceive a "technical" problem. But adding the http fork as a dependency to the winrm package feels like something that may haunt this project until golang/go#5742 is resolved upstream.
If a transport can be supplied via main.go -> winrm/client.go, as @athornton has suggested, then projects dependent on the masterzen/winrm/winrm package can't be impacted now or while the issues around golang/go#5742 evolve.

:+1: Basically what @dylanmei said. If we can make the transport pluggable so this project doesn't depend upon the gwacl, all the better.

I'm not sure being able to pass Transport helps, though: the implemented interface for Transport is http.RoundTripper, which isn't compatible with the gwacl (or azure-sdk-for-go) http.RoundTripper.
It's helpful for my use case because I want to be able to sling around my TLS client configurations, but I don't think it solves the problem of needing a forked net/http and crypto/tls.  Am I missing some obvious interface magic that lets us get away with it?

I agree with @athornton. Any ideas would be helpful.

net/http is only used in http.go (and only to do the POST), and also for the TLS transport parameters in client.go, so this is quite limited, as the code is quite decoupled (for instance Http_post returns a string and not a full http response).
The Azure fork has the same interface as net/http.
Thinking  more about this, I think that we should do the following:

abstract net/http and the TLS transport parameters in a set of interfaces/neutral structures
implement such interface for net/http as the default HTTP/TLS engine
implement the Azure implementation in another package/repo (for instance winrm-azure) that would depend on this winrm package
have the the people that want using winrm on Azure use winrm-azure instead of winrm

Would that work?

I am unclear on whether trying to do client certificate auth against a local SCVMM would require the hacked TLS renegotiation or not, or whether that is specific to Azure.  I guess if it ends up being needed, you just end up with a winrm-certauth package too?
At the moment I'm still working with Basic Auth, but I suspect certificate auth would be something our customers would be less uncomfortable with.  Obviously ultimately we'd like to do Kerberos domain auth but I am not ready to open that bucket of worms yet and I keep hoping that someone else will solve the problem before I need to.
As far as the general question of how to implement, I'd prefer to extend the API with new functions than to change existing semantics, since breaking people who are already using it seems worse to me than cluttering the API.

https://github.com/golang/go/issues/5742#issuecomment-215474624
The patch works ! We don't need the fork anymore for this patch .

@masterzen you can't close this issue because I will soon come with a new pull request resolving this.

close?

@masterzen you can close the PR because I already submitted a better one.

Closing in favor of #56


add template compile pre/post hooks

Approved but subject to change, I want to enable dependency injection for at least the template object, so a hook can inspect/retrieve values.


Add null safety note to flutter create

Note added as per discussion in https://github.com/flutter/flutter/issues/67447#issuecomment-763151291.
Fixes https://github.com/flutter/flutter/issues/67447

I think the idea here is to always --apply-changes since the freshly created template shouldn't have anything in it that requires manual intervention. Showing the UI is probably more confusing at this stage.

I think the idea here is to always --apply-changes since the freshly created template shouldn't have anything in it that requires manual intervention. Showing the UI is probably more confusing at this stage.

I was debating that with myself. I guess if you have a brand new template, you might as well skip the migration preview UI. And given we're printing it here during create, maybe that is a safe assumption?

I was debating that with myself. I guess if you have a brand new template, you might as well skip the migration preview UI. And given we're printing it here during create, maybe that is a safe assumption?

Makes sense, LGTM

Makes sense, LGTM


Request: Bump onstarjs to 2.3.26 and rebuild Docker container

Hi, I'm running this module as a Docker container, and recently noticed all requests are failing with this error:
{"error":{"message":"Request Failed with status 400 - Bad Request",

This was when POSTing to url: 'https://api.gm.com/api/v1/oauth/token'
I also found I was getting something similar when running a test script with<EMAIL_ADDRESS>Updating to 2.3.26 (released a few days ago) resolved this issue for me. It appears the key was updated: https://github.com/samrum/OnStarJS/pull/231
Would you mind updating the package (I see a Dependabot PR out there already going to 2.3.25) and push a new Docker container up?
Thank you!

Build and publish in progress. Thanks for the notice.


Paginate logs API

Paginate logs API
Requirement
We have to paginate the logs API so that it should not return all the data at once and return data in a batch.
Problem with logs API not paginated:

Currently if we fetch logs and if the logs are in bulk it would cost our DB very high amount of reads.
Rendering big amount of data at once can also affect frontend performance.
In live-site we're polling logs API so we will fetch large amount of logs every x seconds, which is not idle.

API URL
{{BASE_URL}}/logs/:type

Note: Before paginating the API make sure you check where this API is being used so that the functionality of the other features doesn't get broke. This task also includes scoping out this and changing the frontend code too by mentioning respective PRs, after that only PR for this will be merged.

Questions you can have in mind

Is this should be under feature flag?
Yes, it should be under feature dev=true

What is feature flag?
It's a simple variable which we can use to turn off/on the feature. Currently we use dev=true in URL query.

Please assign this to me

No longer required.


Update fvar instance ps names when using gen_stat_tables_from_config

@RosaWagner Please check if this PR solves #297

It works ! Thanks !!

I'll cut a release later


How to access get_page_ranges?

Is there a way to call get_page_ranges (https://docs.microsoft.com/en-us/rest/api/storageservices/get-page-ranges) from a blob client? or is there any other way to call that API?

I coded up a commit to support get_page_ranges.  How can I create a pull request for my code?

I created a pull request to resolve this issue.
#725


Where's the sample?

All I can see in the repo is the library...
Please add the sample
:(

I don't have one. The gif is based on an app I'm developing. But I've added a demo here: https://github.com/mikepenz/FastAdapter/blob/develop/app/src/main/java/com/mikepenz/fastadapter/app/ExpandableMultiselectDeleteSampleActivity.java
It does not matter which adapter you use, the usage is very simple. The readme shows tht you just have to do 3 things.
I'll add a demo soon...

OK. Can you please update this post when you have a sample?

Done. Check out this repo folder: https://github.com/MFlisar/DragSelectRecyclerView/tree/master/demo/src/main/java/com/michaelflisar/dragselectrecyclerview/demo

I think it has a bug:
I can select items, but I can't de-select them. Not by single click, and not by dragging, like on Google Photos.

It's a very simple demo to show the usage only. Deselect all via the "Clear" button in the toolbar at the top. Normally you will add an ActionMode as well and a few other things.
Just change the onClickListener to toggle the selection if you want that. Or adjust it to whatever logic you want.
Dragging over a already selected area and dragging back will unselect the items that were already selected as well for example, this may be desired or not.

Can you please modify it to show that it works like on Google Photos? That de-selecting is possible via both clicking and dragging?

That's all not part of the library. That's selection logic. The library will notify you about over which items you dragged and if the range was deselected (this means, you have already dragged over those items but decided to drag back) or selected (this means the user has not yet dragged over those items and wants to select more).
What you do with this info is up to you. Remove the range from your selected items if the user does a swipe selection in a range of selected items for example. Or remember the selection you had before the drag selection started and always invert or revert the selection of the items if the item is drag selected or unselected. Do whatever you want, it's totally up to you.
This library does not provide selection or anything like this, just the events of the drag selector. As said, you can implement any logic with this...
I changed the demo to toggle the selection if you click an item, but the rest is up to you and up to your personal needs.

ok, sorry and thanks. I just thought it would be a more fitting and whole sample if it showed that it can do exactly as the original app.

If you define the behaviour I can show you how to achieve it...

I was only curious to see a library that works like on Google Photos. I don't need it currently. Was just searching for something interesting. I even asked about it here:
http://stackoverflow.com/q/41199611/878126

I tried google photos. It does react the same way as my demo, you can't deselect items via drag selection....
I could think of the ability to start selection mode by long pressing a not selected items and starting a desolation mode by long pressing an already selected item. That should be quite easy to achieve

On Google Photos, you can deselect items, even by dragging, though by dragging it works in a bit weird way.
To reproduce, select multiple items (use dragging if you wish). Then, drag an already selected item to selected items, go back and then again to those items. Now it's deselecting them.
What you offer seems better. To drag an already selected item to start deselection...

This is working in my demo as well. That's because the logic is kept simple - dragging back means unselected and dragging away from the first selected item means select. No logic for remembering the original state.

Correct. Thought it didn't have it.
But what you offered is better I think.
Also, single click should de-select, as we both mentioned.

Single click worked as well already, it toggles the selection...
I extended the DragSelectListener a little bit and added an OnAdvancedDragSelectListener and updated the demo, it shows you how to use it with 3 different modes.

Nice. Thank you.
What's the difference between "advanced" and the "advanced first item dependant" (typo btw, should be dependent)  ?

Try it ;-).
There are some comments in the demo.
Advanced will invert the initial state on drag selection and revert to the
original state if you drag back. It will just toggle any item and revert to
initial state if you drag back.
The other one will do the same as happened to the first item to all others
and simple do the inverse on drag back. This means if you long press an
unselected item each drag selected item will be unselected as well, if you
drag back each item will be selected. And vice versa...
AndroidDeveloperLB<EMAIL_ADDRESS>schrieb am Sa., 4. März 2017,
11:38:

Nice. Thank you.
What's the difference between "advanced" and the "advanced first item
dependant" (typo btw, should be dependent) ?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/MFlisar/DragSelectRecyclerView/issues/2#issuecomment-284143298,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACUyXm_keF7-G2r6LGA3BvRuNLt-pu94ks5riT8LgaJpZM4MRKdj
.

Ohh....
Both seem logical things to choose from. Thank you.
So in other words:
Advanced :  will toggle each item you go by, no matter what is the first item.
advanced first item dependent -If first was selected, now it goes to deselection mode on all. and the opposite.

Exactly

I think I'll add a reusable helper class to the library and even add a fourth mode, a combination of the second and third mode...
Any naming suggestions for the 3 modes?

I think you can just add a TextView above the RecyclerView, saying how the current mode works.
But, here are mine:

simple -> drag to select
advanced -> drag to toggle all in the way
advanced first item dependent - drag to toggle from first item

Naming for the demo is not that important. I want to provide a configurable class that can be used in any project with any adapter, therefore this must be configurable with a mode and for this mode I need names...
Maybe
enum Mode
{
    /* simply selects each items you go by, unselects on move back */
    Simple,
    /* toggles each items original state, reverts to the original state on move back */
    Toggle, // or ToggleAndRememberOriginalState
    /* toggles the first item and applies the same state to each item you go by and inverted state on move back */
    FirstItemDependent, // or ???
    /* toggles the item and applies the same state to each item you go by and reverts to the original state on move back */
    FirstItemDependentToggle  // or FirstItemDependentToggleAndRememberOriginalState
}

Sure.
Just please use javadocs instead of comments (meaning "/** this */ "  instead of "// this"

You can check out the new demo, it uses a so called DragSelectionProcessor and offers all 4 modes and is useable via a very simple setup and just replaces the OnDragSelectListener.
It does do all the logic for you, you just have to select/deselect items if the processor tell you to do so and that's it.

Thank you so much.
You are welcomed to try out my tiny libraries and apps, if you wish:
https://github.com/AndroidDeveloperLB?tab=repositories
https://play.google.com/store/apps/developer?id=AndroidDeveloperLB

I've checked them already ;-)


ModuleNotFoundError on tqdm with linea enabled

If I have lineapy extension loaded, and have a cell with from tqdm import tqdm , and I run that cell twice in a row (works first time), I get error:
ModuleNotFoundError                       Traceback (most recent call last)
/var/folders/w8/wx98k7w1093497fd8l72lwdnj3ln52/T/tmptql1_is0linea_ipython/d6311c30-5a9a-4b9c-8236-1f57a47c8b40_7.py in <module>
----> 1 from tqdm import tqdm

ModuleNotFoundError: No module named 'tqdm.tqdm'

Error still happens if I %unload_ext lineapy , but goes away if I remove lineapy from c.InteractiveShellApp.extensions in ipython_config.py, unless I run %load_ext lineapy and the cell, then it occurs again

Hi Anting, Thanks for reporting this interesting bug. We will prioritize this issue and give you an ETA about when this will be fixed soon.

This will be available starting v0.1.5


Ionic / Crosswalk / Appbuilder problem

Hi Eddy,
I have a very strange/specific issue with my project. I'm using the latest version of this plugin 1.7.2 in project. I have an Ionic Framework based project, running of 1.3.1 of Ionic.
The problem is as follows:
I have an input element in a list on one of my templates. It was originally a 'toggle' type control from ionic, but I have changed it to a button and a normal angular checkbox, with the same results.
The result is that it looks like the event is being cancelled by some sort of error within the Crosswalk library.
The message I receive is as follows:
[chromium] [ERROR:xwalk_autofill_client.cc(116)] Not implemented reached in virtual void xwalk::XWalkAutofillClient::OnFirstUserGestureObserved()

I realize that the error is coming directly from the crosswalk files/library, but I'm trying here with you first to see if it can be resolved.
The layout I'm using is very simple, below is an excerpt from my view:
<ion-list show-reorder="true" can-swipe="false" show-delete="true">

                    ...

                    <ion-item class="item item-toggle">
                        Kennisgewings

                            <input type="checkbox" ng-model="pushSettings.Enabled" ng-change="updatePushRegistration(pushSettings.Enabled)">

                    </ion-item>

                    ...

                </ion-list>

Hi,
Googling shows a lot of similar issues.
I can't reproduce this issue, so if you're up for a bit of Java hacking you can try the suggestions on SO.
Does an older version of Crosswalk work bte?

The code in the SO answer didn't help solve my issue. Has anyone else found another solution to this?


Roman/fix dest conn method names

Description

Make sure all destination connectors implement the base abstract methods using the same signatures.
Also leverage conform dict in the base methods to make sure it's called in a consistent fashion.
Additional updates to move the common code into the base destination connector class

@ahmetmeleq I believe this should handle the issue you brought up with conform_dict()

I think you're going to hit issues with the unit tests leveraging assignment like here. should be quick bump

bump PR title (for conventional) and description to include some more info about the what and why? also maybe include a note in the changelog?

not sure if this is in scope for this PR, but calling write_elements() rather than write() will still hit issues with some of these connectors, right? i.e. with Pinecone we do formatting directly in write()

not sure if this is in scope for this PR, but calling write_elements() rather than write() will still hit issues with some of these connectors, right? i.e. with Pinecone we do formatting directly in write()

I think this PR should:

isolate all formatting logic from write methods
call the isolated formatting logic both in write , and write_elements to support both how ingest and other downstream applications

@ahmetmeleq, look at the latest changes. I created two base classes in the BaseDestinationConnector class:

conform_dict: Changes that are needed in place
normalize_dict: map to a new dict to accommodate the input needed by the destination

These are then called in the base write_raw_dict method which gets called by the write and write_elements methods.

The design looks good 👍 , waiting for the tests - also two comments on naming and azure cognitive search test script

This is great! conform_dict is something we've been wanting to address. I like the approach.

Looks all good now, waiting for tests to approve


Crash on M1 Mac

App Version: 1.0.8
Licensed: ?
Operating System: darwin 22.3.0
Error: Cannot read properties of undefined (reading 'qm-vamp-plugins')
Stack: TypeError: Cannot read properties of undefined (reading 'qm-vamp-plugins')
at w.render (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:1283075)
at Ps (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:173739)
at Is (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:173537)
at yl (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:216999)
at vd (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:205032)
at gd (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:204960)
at ud (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:204823)
at ad (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:201630)
at ld (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:202019)
at Ho (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:142937)
{"componentStack":"
at w (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:1278903)
at div
at div
at k (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:1283931)
at div
at div
at N (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:1291597)
at div
at main
at Y (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:1329488)
at oe (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:1331523)
at U (file:///Applications/ssp.app/Contents/Resources/app/.webpack/renderer/main_window/index.js:2:1303313)"}

Seeing the same on an Intel mac