Installs PHP 7.2

Installs PHP 7.2 to allow the Riprap microservice and module to install.

It appears that RipRap supports PHP 7.1.3 and greater.

True dat, but there doesn't seem to be a way to go higher than 7.1.26 on
Ubuntu without going to 7.2
On Fri, Feb 22, 2019, 4:21 PM Jared Whiklo<EMAIL_ADDRESS>wrote:

It appears that RipRap supports PHP 7.1.3 and greater
https://github.com/mjordan/riprap/blob/master/composer.json#L13.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Islandora-Devops/claw-playbook/pull/95#issuecomment-466534163,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAeNJ0pO0LIjEbdfMFH-X2ODEarjPLH2ks5vQFFKgaJpZM4bKTot
.

@mjordan can you explain why 7.1.3 is required for RipRap?
On Fri, Feb 22, 2019, 20:49 Alan Stanley<EMAIL_ADDRESS>wrote:

True dat, but there doesn't seem to be a way to go higher than 7.1.26 on
Ubuntu without going to 7.2
On Fri, Feb 22, 2019, 4:21 PM Jared Whiklo<EMAIL_ADDRESS>wrote:

It appears that RipRap supports PHP 7.1.3 and greater
https://github.com/mjordan/riprap/blob/master/composer.json#L13.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
https://github.com/Islandora-Devops/claw-playbook/pull/95#issuecomment-466534163
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAeNJ0pO0LIjEbdfMFH-X2ODEarjPLH2ks5vQFFKgaJpZM4bKTot
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Islandora-Devops/claw-playbook/pull/95#issuecomment-466609288,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACua4ef-dd-6iqn5S4NLufST9Fi23o88ks5vQKwvgaJpZM4bKTot
.

Symfony 4 dependency, https://symfony.com/doc/current/reference/requirements.html.

@seth-shaw-unlv got Riprap to work on the claw-playbook. See https://github.com/mjordan/riprap/issues/20. Not sure how he handled the PHP requirement though.

I know @seth-shaw-unlv is running Centos so maybe it is a better version of
PHP.
I wonder if the php-versions role would accept 7.1.3 as the version?
On Sat, Feb 23, 2019, 11:22 Mark Jordan<EMAIL_ADDRESS>wrote:

@seth-shaw-unlv https://github.com/seth-shaw-unlv got Riprap to work on
the claw-playbook. See mjordan/riprap#20
https://github.com/mjordan/riprap/issues/20. Not sure how he handled
the PHP requirement though.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Islandora-Devops/claw-playbook/pull/95#issuecomment-466670784,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACua4XtTe75Q0Ajhmx_Vz740Pqcm8HZ_ks5vQXjogaJpZM4bKTot
.

If @ajstanley's PR works, any reason why we shouldn't install PHP 7.2 on the claw-playbook?

Just that it will be tougher to get the minimum PHP version to do things like update composer.lock files and such.  We can always just drop 7.1 and roll out with only 7.2+ if folks don't think it's too late.

@mjordan and @whikloj I just run 7.2 on CentOS.

Looking at this again, symfony/flex:^1.1 supports PHP 7.0.

So... is there anything standing in the way of moving to 7.2?

I've found a couple of minor problems - don't merge this one just yet.

Should be good now.

@Islandora-Devops/committers are there any concerns with switching to 7.2? To respond explicitly to @whikloj 's point, which implies that this whole PR isn't necessary, it is necessary because Riprap requires Symfony 4, which requires 7.1.3+ - @mjordan pointed out.

@rosiel It was decided to go with 7.2 at this meeting: https://github.com/Islandora-CLAW/CLAW/wiki/February-27%2C-2019.  If we run into any issues, we can deal with as they arise.

Just to clarify my reasoning.

Symfony 4 has a dependency on PHP 7.1.3, but RipRap has a dependency on symfony/flex:^1.1
The composer version constraint ^1.1 means version >= 1.1.* and version < 2.*
Symfony/Flex version 1.2.0 (the most current) supports PHP 7.0

and finally
I just cloned RipRap into an existing claw-playbook instance running PHP 7.1.27, installed it and ran it.
vagrant@claw:/var/www/html/riprap$ COMPOSER_MEMORY_LIMIT=-1 php -d memory_limit=-1 /usr/local/bin/composer update
Deprecation warning: Your package name riprap is invalid, it should have a vendor name, a forward slash, and a package name. The vendor and package name can be words separated by -, . or _. The complete name should match "[a-z0-9]([_.-]?[a-z0-9]+)*/[a-z0-9]([_.-]?[a-z0-9]+)*". Make sure you fix this as Composer 2.0 will error.
Cannot create cache directory /home/vagrant/.composer/cache/repo/https---repo.packagist.org/, or directory is not writable. Proceeding without cache
Cannot create cache directory /home/vagrant/.composer/cache/files/, or directory is not writable. Proceeding without cache
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 78 installs, 0 updates, 0 removals
  - Installing symfony/flex (v1.2.0): Downloading (100%)
Cannot create cache directory /home/vagrant/.composer/cache/repo/https---repo.packagist.org/, or directory is not writable. Proceeding without cache
Cannot create cache directory /home/vagrant/.composer/cache/repo/https---repo.packagist.org/, or directory is not writable. Proceeding without cache
Cannot create cache directory /home/vagrant/.composer/cache/repo/https---flex.symfony.com/, or directory is not writable. Proceeding without cache
Cannot create cache directory /home/vagrant/.composer/cache/repo/https---repo.packagist.org/, or directory is not writable. Proceeding without cache
  - Installing ocramius/package-versions (1.4.0): Downloading (100%)
  - Installing guzzlehttp/promises (v1.3.1): Downloading (100%)
  - Installing ralouphie/getallheaders (2.0.5): Downloading (100%)
  - Installing psr/http-message (1.0.1): Downloading (100%)
  - Installing guzzlehttp/psr7 (1.5.2): Downloading (100%)
  - Installing guzzlehttp/guzzle (6.3.3): Downloading (100%)
  - Installing ramsey/uuid (3.8.0): Downloading (100%)
  - Installing symfony/yaml (v4.2.4): Downloading (100%)
  - Installing psr/log (1.1.0): Downloading (100%)
  - Installing symfony/debug (v4.2.4): Downloading (100%)
  - Installing symfony/polyfill-mbstring (v1.11.0): Downloading (100%)
  - Installing symfony/http-foundation (v4.2.4): Downloading (100%)
  - Installing symfony/contracts (v1.0.2): Downloading (100%)
  - Installing symfony/event-dispatcher (v4.2.4): Downloading (100%)
  - Installing symfony/http-kernel (v4.2.4): Downloading (100%)
  - Installing symfony/routing (v4.2.4): Downloading (100%)
  - Installing symfony/finder (v4.2.4): Downloading (100%)
  - Installing symfony/filesystem (v4.2.4): Downloading (100%)
  - Installing psr/container (1.0.0): Downloading (100%)
  - Installing symfony/dependency-injection (v4.2.4): Downloading (100%)
  - Installing symfony/config (v4.2.4): Downloading (100%)
  - Installing symfony/var-exporter (v4.2.4): Downloading (100%)
  - Installing psr/simple-cache (1.0.1): Downloading (100%)
  - Installing psr/cache (1.0.1): Downloading (100%)
  - Installing symfony/cache (v4.2.4): Downloading (100%)
  - Installing symfony/framework-bundle (v4.2.4): Downloading (100%)
  - Installing symfony/console (v4.2.4): Downloading (100%)
  - Installing nikic/php-parser (v4.2.1): Downloading (100%)
  - Installing doctrine/inflector (v1.3.0): Downloading (100%)
  - Installing symfony/maker-bundle (v1.11.5): Downloading (100%)
  - Installing monolog/monolog (1.24.0): Downloading (100%)
  - Installing symfony/monolog-bridge (v4.2.4): Downloading (100%)
  - Installing symfony/monolog-bundle (v3.3.1): Downloading (100%)
  - Installing symfony/stopwatch (v4.2.4): Downloading (100%)
  - Installing zendframework/zend-eventmanager (3.2.1): Downloading (100%)
  - Installing zendframework/zend-code (3.3.1): Downloading (100%)
  - Installing ocramius/proxy-manager (2.1.1): Downloading (100%)
  - Installing doctrine/event-manager (v1.0.0): Downloading (100%)
  - Installing doctrine/cache (v1.8.0): Downloading (100%)
  - Installing doctrine/dbal (v2.9.2): Downloading (100%)
  - Installing doctrine/lexer (v1.0.1): Downloading (100%)
  - Installing doctrine/annotations (v1.6.1): Downloading (100%)
  - Installing doctrine/reflection (v1.0.0): Downloading (100%)
  - Installing doctrine/collections (v1.6.1): Downloading (100%)
  - Installing doctrine/persistence (v1.1.0): Downloading (100%)
  - Installing doctrine/common (v2.10.0): Downloading (100%)
  - Installing doctrine/migrations (v2.0.0): Downloading (100%)
  - Installing symfony/inflector (v4.2.4): Downloading (100%)
  - Installing symfony/property-access (v4.2.4): Downloading (100%)
  - Installing symfony/security-core (v4.2.4): Downloading (100%)
  - Installing symfony/doctrine-bridge (v4.2.4): Downloading (100%)
  - Installing doctrine/doctrine-cache-bundle (1.3.5): Downloading (100%)
  - Installing jdorn/sql-formatter (v1.2.17): Downloading (100%)
  - Installing doctrine/doctrine-bundle (1.10.2): Downloading (100%)
  - Installing doctrine/doctrine-migrations-bundle (v2.0.0): Downloading (100%)
  - Installing doctrine/instantiator (1.2.0): Downloading (100%)
  - Installing doctrine/orm (v2.6.3): Downloading (100%)
  - Installing symfony/orm-pack (v1.0.6): Downloading (100%)
  - Installing symfony/security-http (v4.2.4): Downloading (100%)
  - Installing symfony/security-guard (v4.2.4): Downloading (100%)
  - Installing symfony/security-csrf (v4.2.4): Downloading (100%)
  - Installing symfony/security-bundle (v4.2.4): Downloading (100%)
  - Installing symfony/polyfill-php72 (v1.11.0): Downloading (100%)
  - Installing symfony/polyfill-intl-idn (v1.11.0): Downloading (100%)
  - Installing egulias/email-validator (2.1.7): Downloading (100%)
  - Installing swiftmailer/swiftmailer (v6.2.0): Downloading (100%)
  - Installing symfony/swiftmailer-bundle (v3.2.5): Downloading (100%)
  - Installing symfony/messenger (v4.2.4): Downloading (100%)
  - Installing symfony/dotenv (v4.2.4): Downloading (100%)
  - Installing symfony/process (v4.2.4): Downloading (100%)
  - Installing symfony/web-server-bundle (v4.2.4): Downloading (100%)
  - Installing symfony/phpunit-bridge (v4.2.4): Downloading (100%)
  - Installing symfony/dom-crawler (v4.2.4): Downloading (100%)
  - Installing symfony/browser-kit (v4.2.4): Downloading (100%)
  - Installing doctrine/data-fixtures (v1.3.1): Downloading (100%)
  - Installing doctrine/doctrine-fixtures-bundle (3.1.0): Downloading (100%)
  - Installing squizlabs/php_codesniffer (3.4.1): Downloading (100%)
Writing lock file
Generating autoload files
ocramius/package-versions:  Generating version class...
ocramius/package-versions: ...done generating version class
Executing script cache:clear [OK]
Executing script assets:install public [OK]

vagrant@claw:/var/www/html/riprap$ php -v
PHP 7.1.27-1+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Mar  7 2019 20:02:03) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.1.27-1+ubuntu16.04.1+deb.sury.org+1, Copyright (c) 1999-2018, by Zend Technologies
    with Xdebug v2.7.0, Copyright (c) 2002-2019, by Derick Rethans

vagrant@claw:/var/www/html/riprap$ php bin/console app:riprap:check_fixity --settings=sample_csv_config.yml
Riprap checked 2 resources (2 successful events, 0 failed events) in 0.123 seconds.

I'll note I had to edit the sample_csv_config.yml to point to files accessible to the vagrant machine.

@whikloj Will bumping the flex version riprap uses to symfony/flex:^1.2 provide better PHP 7.x compatibility? Flex 1.2 supports PHP ^7.0, which means that it should work with any version of PHP higher than 7.0. Is that correct?

@mjordan Bumping to ^1.2 has no effect at all. The ^ means all non-breaking changes are allowed. So you are good from version 1.1 all the way until (but not including) 2.0.0.
More information on this version constraint here
And yes, the flex PHP requirement is php: ^7.0 so again, they are saying all versions from 7.0 up to (but not including) 8.0.
Also, and this is a mea culpa, version 7.1.3 < 7.1.27 which is how I was able to install it.

@whikloj thanks for the suggestions on updating riprap. I probably haven't had enough caffeine yet today but is there anything holding us back from using PHP 7.2 in the CLAW vagrant? I am running PHP 7.2.15-0ubuntu<IP_ADDRESS> (cli) on my bare-metal laptop and riprap installs and runs fine.

@mjordan no, there is no reason why we can't do this. My point is that we would be choosing to drop support for PHP 7.1. There is no technical need, I just want people to be aware of that.
When we dropped PHP 5.6 and 7.0 support, the discussion was around Silex deprecation and that Symfony 4 required PHP 7.1.
Riprap runs under 7.2, but it also runs in claw-playbook under 7.1.
But to the question of what we need to do.
Along with this PR, the travis build scripts will need to be updated to remove 7.1 testing for the Drupal modules, Crayfish and Crayfish-Commons. Perhaps we should add in PHP 7.3 for testing. As well someone should send a note out to the community just to keep them informed.

@whikloj your clean explanation is worth more than any amount of caffeine. I get it now.

Testing this now :rocket:

Installed fine.  Could verify PHP 7.2 on the command line and went through a basic sanity check.  Ingested an image -> gave it an original file -> checked derivatives -> verified fedora, solr, and triplestore indexing.
That was for Ubuntu.  Didn't try CentOS.  I need to use this environment to work through some things, but I'll give CentOS a whirl later just for sanity's sake before merging and doing all the afterwork for this.
@seth-shaw-unlv You use 7.2 and CentOS.  Do you know if anything special needs to be done?  Or does the 7.1 -> 7.2 change in webserver.yml take care of it.

@dannylamb you need to change it in both webserver.yml and crayfish.yml.

@seth-shaw-unlv Gotcha.  Both of those are covered under this PR.

@dannylamb, yeah, that should do it. I can kick off another build if you want.

@seth-shaw-unlv++ That would be awesome.  If CentOS build succeeds we can merge this in.

@dannylamb vagrant up!


Update cc to 1.1.22

This version of cc contains a fix to prevent spurious rebuilds. Hopefully this should help avoid the CI issues rustc has been having.

@bors r+

:pushpin: Commit a5503c1480d82407b5d234ab77ef6e158cbbbd7f has been approved by epage
It is now in the queue for this repository.

:hourglass: Testing commit a5503c1480d82407b5d234ab77ef6e158cbbbd7f with merge 80d82ca22abbee5fb7b51fa1abeb1ae34e99e88a...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing 80d82ca22abbee5fb7b51fa1abeb1ae34e99e88a to master...

It sounds like rust-lang/rust build is affected. I guess we should include it in https://github.com/rust-lang/rust/pull/130929?

That would be ideal, imho. Though the CI failure rate isn't as bad at the moment so it'll still be tolerable if it slips to the next update.


IP validation fails for gateway deployment with TLS passthrough

I tried deploying a gateway with TLS passthrough to a Wireguard private network IP using Terraform. It fails with this error:
Error: couldn't deploy name gateway with error: failed to deploy gateway name Id: map[]: invalid deployment: failed to validate backend 'https://<IP_ADDRESS>': failed to parse backend https://<IP_ADDRESS> with error: invalid ip

Here's the Terraform file that I used, for reference:
variable "mnemonics" {
    type = string
}

terraform {
  required_providers {
    grid = {
      source = "threefoldtech/grid"
      version = "1.9.2-rc2"
    }
  }
}

provider "grid" {
    mnemonics = var.mnemonics
    network = "test"
}

data "grid_gateway_domain" "domain" {
  node = 12
  name = "nextcloud"
}

resource "grid_network" "net" {
    nodes = [12]
    ip_range = "<IP_ADDRESS>/16"
    name = "network"
    description = "My network"
    add_wg_access = true
}

resource "grid_deployment" "d1" {
  node = 12
  network_name = grid_network.net.name

  disks {
    name = "data"
    size = 50
  }

  vms {
    name = "vm1"
    flist = "https://hub.grid.tf/tf-official-apps/threefoldtech-nextcloudaio-latest.flist"
    cpu = 1
    memory = 2048
    rootfs_size = 15000
    entrypoint = "/sbin/zinit init"
    env_vars = {
      SSH_KEY = file("~/.ssh/id_rsa.pub")
    }
    mounts {
        disk_name = "data"
        mount_point = "/mnt/data"
    }
  }
}

resource "grid_name_proxy" "p1" {
  node            = 12
  name            = "nextcloud"
  backends        = [format("http://%s", grid_deployment.d1.vms[0].ip)]
  network         = grid_network.net.name
  tls_passthrough = false
}

output "wg_config" {
  value = grid_network.net.access_wg_config
}

output "vm1_ip" {
  value = grid_deployment.d1.vms[0].ip
}
output "vm1_ygg_ip" {
  value = grid_deployment.d1.vms[0].ygg_ip
}

output "fqdn" {
  value = data.grid_gateway_domain.domain.fqdn
}

@scottyeager can you try to remove the https:// from the gateway's backend? so it should be the ip only.

Hi @AhmedHanafy725. With the bare IP, I now get this error:
Error: expected "backends" to have a host, got <IP_ADDRESS>

The Terraform provider docs indicate that the correct form for a TLS passthrough backend includes https://:

backends (List of String) The backends of the gateway proxy (in the format (http|https)://ip:port), with tls_passthrough the scheme must be https.

I should also mention that attempting to use a custom port number with TLS passthrough fails too:
Error: couldn't deploy name gateway with error: failed to deploy gateway name Id: map[]: invalid deployment: failed to validate backend 'https://<IP_ADDRESS>:8080': failed to parse backend https://<IP_ADDRESS>:8080 with error: failed to parse backend https://<IP_ADDRESS>:8080 with error: address https://<IP_ADDRESS>:8080: too many colons in address

I think I have discovered the issue. Zos expects only port:ip when TLS passthrough is enabled, but the Terraform provider is validating for an https URL. Transferring this issue to the provider repo, since the issue is not in sdk-go.


Feature: Client support for competitions

Changelogs

Client methods for downloading a competition from the Hub and submitting predictions
New CompetitionSpecification class that inherits from the DatasetV2 class
New CompetitionPredictions class that inherits from the BenchmarkPredictions class for standardized prediction uploads
Removal of old competition code (e.g. differing URLs based on the ArtifactSubtype)
Removal of the CompetitionResults class since these results never reach the client currently
New method for loading a competition that accepts a local Zarr path for cached dataset downloads

Checklist:

[x] Was this PR discussed in an issue? It is recommended to first discuss a new feature into a GitHub issue before opening a PR.
[ ] Add tests to cover the fixed bug(s) or the newly introduced feature(s) (if appropriate).
[ ] Update the API documentation if a new function is added, or an existing one is deleted.
[x] Write concise and explanatory changelogs above.
[x] If possible, assign one of the following labels to the PR: feature, fix, chore, documentation or test (or ask a maintainer to do it for you).

Important Notes
This PR is ready in terms of features, but is currently missing:

Documentation/tutorials for these changes
Updated tests

I will leave this PR as a draft to gather comments while I work on these next.

@cwognum going to mark this ready for review despite missing those couple docs and tutorial. Everything else should be ready 💪

@jstlaurent If you have the time, it would be great if you could take a stab at this. I was planning to take over from Andrew, but there are some other things on my plate I would really like to wrap before EOY so we can hit the ground running in 2025.


Use 302 to redirect instead of 301

301 means "Moved Permanently", so browser caches this redirect, it is used when changing server URL.
302 means "Found", it is used generally (e.g. redirect top page after login).
Each of 301, 302, 303 and 307 means redict, but they are different. We could choice those status in case.

This is good :+1: But i'm hesisant about 301 vs 302

302 is indeed a better default than 301. +1

Why? Rails and Sinatra and any framework uses 302 to redirect.
In addition, when server returns 301, browser caches this redirect, and it remains when removing cache. It makes us confusing!! For example, I try to login but failed, then redirect to login page. Next day, I try to login again and succeeded, then server returns redirect to index page, but browser caches yesterday's redirect so displaying login page eternally.

Okay. Makes sense :100:
Thanks a lot :+1:


Experimental python interface for finitediff

Codecov Report
Attention: Patch coverage is 0% with 29 lines in your changes are missing coverage. Please review.

Project coverage is 92.39%. Comparing base (3f97b77) to head (4c6a6e7).

Files
Patch %
Lines

python/finitediff-py/src/lib.rs
0.00%
29 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #480      +/-   ##
==========================================
- Coverage   92.49%   92.39%   -0.11%
==========================================
  Files         178      179       +1
  Lines       24568    24597      +29
==========================================
+ Hits        22725    22726       +1
- Misses       1843     1871      +28

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Резкое падение FPS с 60 до 11(30) Всегда по разному

Время ошибки

Приблизительное время ошибки: 22.02.2018 16:16 - 16:18

Your Environment

Version used: Elementum 0.0.43 | Burst 0.0.16

Environment name and version: Kodi 17.4

Operating System and version: Android 7.1.1

Link to Kodi debug log: https://paste.ubuntu.com/p/Cr86QCY6Jt/

Well, for anyone that ever has an issue like this (regarding large thumbnails directory), here's what you do.
Delete or rename textures.db from your database folder
Delete thumbnails folder from userdata folder
This causes xbmc to re-download your thumbnails data.

Maybe you have big CPU load because of big memory size (and buffer size).
You can try limiting download speed or play with memory size.

@promidrol есть изменения с версией 0.0.44 ?

Еще не проверял, проверю отпишу тут.


How to Change icons color and crop text color?

Hey!
I start a new project to handover this library
https://github.com/CanHub/Android-Image-Cropper
The ideia is that we keep improving because this project don’t have updates since 2018
Hope I can count with your help.
Open to contribute, next pieces of work will be Android 11 permissions, refactor into Kotlin and ActivityContract

@Canato do you know how to change icons color and crop text color?

@ahmadbajwa12315 if you open the question on the new repo https://github.com/CanHub/Android-Image-Cropper I maybe be able to help more. just need more information of how are you trying to do now

ok


UPDATE: NodeJS 16.x

This updates our NodeJS version to the latest LTS release.
@pradyunsg I wasn't sure if sphinx-theme-builder allows you to specify something like node-version = 16.x so I just pinned the full version, but if you tell me that it is possible then I'm happy to make a quick docs PR to note this in the STB
closes #551 closes https://github.com/pydata/pydata-sphinx-theme/issues/537

I just pinned the full version

I always prefer to be explicit about the version being used.

With this PR I noticed we do not have visibility of what is happening on the stb side of things...
And I am actually surprised I do not see any stb compile in the workflow file... should we add something like that to test the compilation process somehow?

We do have it in our noxfile (https://github.com/pydata/pydata-sphinx-theme/blob/master/noxfile.py#L35), and I believe that the build step in our "publish to PyPI" job does this as well, but we should double check and document (here and or upstream to STB!)
Can we capture that in a new issue to tackle as part of the dev workflow improvements?

You likely need to regenerate package-lock.json and check that in too!

And I am actually surprised I do not see any stb compile in the workflow file...

It will be run as part of:
https://github.com/pydata/pydata-sphinx-theme/blob/7d64c927668ce653c9af3c62cf2bed9801be01bc/.github/workflows/tests.yml#L46
The whole design is that the NodeJS pipeline is fetched and used "transparently" as part of the wheel generation process.

Wheel generation calling generate_assets:
https://github.com/pradyunsg/sphinx-theme-builder/blob/10450cfeb65bc3610a6708ee731069c707978fef/src/sphinx_theme_builder/_internal/distributions.py#L157
stb compile calling generate_assets:
https://github.com/pradyunsg/sphinx-theme-builder/blob/10450cfeb65bc3610a6708ee731069c707978fef/src/sphinx_theme_builder/_internal/cli/compile.py#L26

Can we capture that in a new issue to tackle as part of the dev workflow improvements?

Sure thing!

The whole design is that the NodeJS pipeline is fetched and used "transparently" as part of the wheel generation process.

Yep, and you are doing a great job on that direction @pradyunsg, but as a dev for the theme, I would like to see what is happening under the hood in the CI as well 😜 ... this is why I think we should explicitly add a stb compile step in the future.

@pradyunsg you said:

You likely need to regenerate package-lock.json and check that in too!

But this file doesn't exist anywhere in the repository. If I re-run stb compile (after deleting .nodeenv so that it grabs a new NPM), it builds fine (with some warnings), but no new files are generated anywhere. Could you provide guidance on where to look for package-lock.json?

But this file doesn't exist anywhere in the repository

We do have https://github.com/pydata/pydata-sphinx-theme/blob/master/package.json (without the "lock" part). It seems that furo has checked in both files.

I would like to see what is happening under the hood in the CI as well

We could also make the install step in CI more verbose? (so that the pip install step shows the output from stb?)

I think adding a -v to pip is a good idea!

But this file doesn't exist anywhere in the repository

Ah, interesting! Never mind me then!

I wasn't sure if sphinx-theme-builder allows you to specify something like node-version = 16.x

It doesn’t. That’s a limitation inherited from nodeenv.

If I re-run stb compile (after deleting .nodeenv so that it grabs a new NPM), it builds fine (with some warnings)

@choldgraf, warnings about?

Would it be enough to improve the documentation in the sphinx-theme-builder about what happens when python -m build is invoked, as well as what happens when stb compile is called, and then add a section in our documentation that links out to those more detailed sections?

Docs are always welcome, but I still feel we need visibility on what is happening under the hood when we run the CI job.

We do have https://github.com/pydata/pydata-sphinx-theme/blob/master/package.json (without the "lock" part). It seems that furo has checked in both files.

Yep, AFAIK, the lock file is intended to be included in the repo: https://docs.npmjs.com/cli/v8/configuring-npm/package-lock-json

We could also make the install step in CI more verbose? (so that the pip install step shows the output from stb?)

+1, if that allows us to see a piece of the compilation stuff, I will be more than happy 😉

stb compile is basically running the same code as python -m build (although, the build also generates a wheel with all the stuff).

@damianavila the warnings were just things like deprecated dependencies, it didn't seem to be anything too bad. Mostly it was deprecations of some dependencies that we've got. I'll put a warning log at the bottom of this comment and maybe we should open an issue to track updating those dependencies?
re: being verbose, I agree but I can't figure out how to make stb compile or python -m build be verbose - neither of them have --verbose or -v arguments as far as I can tell. Can we push that to a new issue to track rather than blocking this PR on it?
for the lockfilee, I agree it seems like this should be checked into the repository...but I don't know why it isn't generated in the first place. Maybe something to do with our webpack config? For this as well, can we track it in an issue or do you think this should block upgrading node?
My vote is that we do the following:

convince ourselves that NodeJS16 doesn't introduce bugs for us
merge this PR
Open issues for: "more verbosity in our compile step, especially in our publishing build", "update deprecated dependencies", and "add a package lockfile to the repo".

Warnings generated by webpack
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated<EMAIL_ADDRESS>this library is no longer supported
npm WARN deprecated<EMAIL_ADDRESS>https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated<EMAIL_ADDRESS>Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated<EMAIL_ADDRESS>The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated<EMAIL_ADDRESS>Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated<EMAIL_ADDRESS>Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated<EMAIL_ADDRESS>request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated<EMAIL_ADDRESS>This SVGO version is no longer supported. Upgrade to v2.x.x.
npm WARN deprecated<EMAIL_ADDRESS>You can find the new Popper v2 at @popperjs/core, this package is dedicated to the legacy v1
npm WARN deprecated<EMAIL_ADDRESS><EMAIL_ADDRESS>is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.

My vote is that we do the following:

+1, from a quick look at the readthedocs preview, everything looks fine (I also would not expect changes, as node itself shouldn't impact the built site if we don't change any version of actual package we are using). I agree the other issues are not strictly related to upgrading the node version.

OK I've done some investigating and opened up 3 new issues:

#557
#558
#559

Can we merge this one in so that we resolve the dev installation issues on Mac, and then iterate on those next?


Error when logging in by Google on the frontend

I'm getting this error, strapi working normally, but I can't log into the strapi through the frontend.
Does anyone go through this?
 OAuthProfile: {
    id: '104265214333844458905',
    email<EMAIL_ADDRESS>    verified_email: true,
    name: 'SHARKDEV SHARKDEV',
    given_name: 'SHARKDEV',
    family_name: 'SHARKDEV',
    picture: 'https://lh3.googleusercontent.com/a-/AOh14Gh4nDcKYkBW5fcxAC4yb4SjgY8zi5QKdNJEL-B7=s96-c',
    locale: 'pt-BR'
  }
}
[next-auth][error][adapter_connection_error]
https://next-auth.js.org/errors#adapter_connection_error error: password authentication failed for user "strapi"
    at Parser.parseErrorMessage (D:\DEV\App\tmp\teste\frontend\node_modules\pg-protocol\dist\parser.js:278:15)
    at Parser.handlePacket (D:\DEV\App\tmp\teste\frontend\node_modules\pg-protocol\dist\parser.js:126:29)
    at Parser.parse (D:\DEV\App\tmp\teste\frontend\node_modules\pg-protocol\dist\parser.js:39:38)
    at Socket.<anonymous> (D:\DEV\App\tmp\teste\frontend\node_modules\pg-protocol\dist\index.js:10:42)
    at Socket.emit (events.js:315:20)
    at addChunk (_stream_readable.js:295:12)
    at readableAddChunk (_stream_readable.js:271:9)
    at Socket.Readable.push (_stream_readable.js:212:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:186:23) {
  length: 164,
  severity: 'FATAL',
  code: '28P01',
  detail: undefined,
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'd:\\pginstaller_13.auto\\postgres.windows-x64\\src\\backend\\libpq\\auth.c',
  line: '334',
  routine: 'auth_failed'
}
[next-auth][error][oauth_callback_handler_error]
https://next-auth.js.org/errors#oauth_callback_handler_error TypeError: Cannot destructure property 'manager' of 'connection' as it is null.
    at Object.<anonymous> (D:\DEV\App\tmp\teste\frontend\node_modules\next-auth\dist\adapters\typeorm\index.js:112:9)
    at Generator.next (<anonymous>)
    at asyncGeneratorStep (D:\DEV\App\tmp\teste\frontend\node_modules\next-auth\dist\adapters\typeorm\index.js:26:103)
    at _next (D:\DEV\App\tmp\teste\frontend\node_modules\next-auth\dist\adapters\typeorm\index.js:28:194)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

@marcoriesco I believe that this issue is similar to https://github.com/ghoshnirmalya/nextjs-strapi-boilerplate/issues/4. Please let me know if the discussion in that issue helps you.
I'll keep this open for the time being.

I was able to resolve by examining #4 - for me the issue was that I had a local postgres instance running, and the docker container's port mapping (5432:5432) was masked by this instance as it uses the default port.
This causes the frontend to connect to the local postgres instance, where the strapi user & db don't exist.
Either stopping the local postgres or changing the external port in frontend/.env and backend/docker-compose.yml 's db section will resolve (gotcha here was not to change the internal DATABASE_PORT used in the backend service).


Update transform function in DML Language Reference

Updated transform() documentation.
The update can be reviewed here: http://deroneriksson.github.io/incubator-systemml/dml-language-reference.html#data-pre-processing-built-in-functions

LGTM


Component name casing doesn't work in component details panel

Version
6.0.0-beta.21
Browser and OS info
Chrome 97.0.4692.71/ Win
Steps to reproduce
Set component name casing to PascalCase, go to inspector, select any component
What is expected?
Component name in the details panel should be PascalCase
What is actually happening?
Component name in the details panel is kebab-case

thank you :)


Config options to set headers for introspection query

So something like that would be possible
{
    "schema": {
        "url": "https://...",
        "headers": "x-auth-token: test; x-custom-flag: test"
    }
}

Is this meant to be able to get the schema from a web server that requires authentication or authorization? if that the case, then I would rather not add such feature to the config since it will encourage people to store their credentials in the config files and commit them to source.
If you need something like this, I would suggest to use a HTTP client like Postman to the get the schema using the introspection query, then save the result to local file schema.json and reference that from the Snowflaqe config:
{
    "schema": "./path/to/schema.json"
}

if this is too much work and you require refreshing the schema a lot, I could consider using a environment variable such as
SNOWFLAQE_INTROSPECTION_HEADERS=x-auth-token: test; x-custom-flag: test

That will be used when requesting the GraphQL schema

So basically interpolating environment variables in config. I think this allows more extensibility and reuse of pre-existing environment variables in a project.

I feel like this is a bit overkill for Snowflaqe, I would like to keep the config simple
I feel like since the config already supports getting a schema from a file which is the result of the introspection query, you can easily download a fresh authenticated schema using curl
cat introspection.gql
 | curl -H "Content-Type: application/json"
        -H "Authorization: Bearer $AUTH_TOKEN"
        -X POST --data-binary @- https://example.io/api/graphql
 > schema.json

Does this not work for your use case?

Yeah that's what i do now but it feels less straightforward than it needs to be. I have to keep this introspection.gql file around in my projects, and also i have like 4 schemas atm that i need to regularly update (this way basically manually).
My setup includes MSBuild directives that run Snowflaqe from multiple configs before build. And it seems way too hard for me to embed those manual introspection queries in the process (having troubles with the file system access from MSBuild command directives).

Alright, I see. Then I am okay adding the SNOWFLAQE_INTROSPECTION_HEADERS support with the .env file. Happy to accept PR for this as always 🙏


SVMView{Training,Test}Predictions and related impls added

This feature provides viewing predictions on training and test datasets.

Still missing "-svm_view_train_predictions" in SVMPostTrain() and "-svm_view_test_predictions" in SVMTest()

It would be; however it is not (basically).

This isn't a philosophical question. Can you invoke the new viewers from the command line? No, because there are no options for that.

You can print performance scores and other model qualities in these functions. If some sample is misclassified, it shows up in the scores and value of loss functions and other metrics (which are actually used for auto testing in PermoSVM).

Yes, I can get some aggregate information and I can invoke the viewers giving me this aggregate information from the command line (as is the case for every other viewer!). But the point is, that I can not get the information provided by the new viewers without explicitly calling them in an application.
Why you don't want to have the option to invoke the new viewers from the command line? Why I can't simply call -svm_view_train_predictions :predictions.mat:ascii_matlab and instead I need to modify my application to open a viewer, push format, call SVMViewTrainingPredictions() and recompile! my application?

This isn't a philosophical question. Can you invoke the new viewers from the command line? No, because there are no options for that.

Sure, it isn’t a philosophical discussion. A typical ML pipeline is like that: data preprocessing (standardization, feature extraction, ...), training model, validating/testing model, and saving results. I have followed this workflow and haven’t wanted to combine validating/testing and saving steps. On the other hand, I agree with you that providing an option for invoking a new viewer from a command line (without explicitly calling in an app) should be fine and helps to make an app code cleaner. So, I added these two options.


RASC on BTC

The initial creation of the RASC (Rotten Anti Social Collection) on bitcoin ordinals

did I miss anything here?


ND4J: FlatBuffers serialization issue for String arrays

Looks like there's some issue with serializing String arrays to/from FlatBuffers.
    @Test
    public void testStringSameDiffSerialization() throws Exception {

        File f = new File(testDir.newFolder(), "test.bin");

        SameDiff sd = SameDiff.create();
        INDArray arr = Nd4j.scalar("My test string here!");
        sd.constant("s", arr);
        sd.save(f, false);

        SameDiff sd2 = SameDiff.load(f, false);
        INDArray deArr = sd2.getVariable("s").getArr();
        assertEquals(arr, deArr);
    }

java.lang.AssertionError:
Expected :"My test string here!"
Actual   :"       My test stri"

Looks like the problem is this Nd4j.create(Buffer, shape) call. The String array header is not being  handled correctly, based on that result, not sure that should be fixed...
https://github.com/eclipse/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/factory/Nd4j.java#L6568

Maybe FlayBuffers doesn't use UTF-8 by default?

No, this isn't a FlatBuffers issue per-se. IIRC from looking at this, dump the INDArray buffer directly into flatbuffers as bytes, then pull it out again into a buffer to recreate the INDArray. So we're not encoding it as a String, we're encoding it as bytes.

Strange. We have test for it. https://github.com/SkymindIO/deeplearning4j/blob/b5f0ec072f3fd0da566e32f82c0e43ca36553f39/nd4j/nd4j-backends/nd4j-tests/src/test/java/org/nd4j/linalg/mixed/StringArrayTests.java#L88-L104

Might be scalar specific...
    @Test
    public void testBasicStrings_4a() {
        val arrayX = Nd4j.scalar("alpha");

        val fb = new FlatBufferBuilder();
        val i = arrayX.toFlatArray(fb);
        fb.finish(i);
        val db = fb.dataBuffer();

        val flat = FlatArray.getRootAsFlatArray(db);
        val restored = Nd4j.createFromFlatArray(flat);

        assertEquals(arrayX, restored);
        assertEquals("alpha", restored.getString(0));
    }

java.lang.AssertionError:
Expected :"alpha"
Actual   :"    "

quite possible...

Fixed in my branch.


Upload multiple files

From @pbeckmann on July 28, 2016 17:8
Hello,
when i click the upload button in the UI, the window opening allows me to select multiple files to upload.
But when i select multiple files and click ok, i would thought i could upload them together.
But i just get:
::1 - [28/Jul/2016:19:02:55 +0200] "POST /filemanager/ HTTP/1.1" 403 14
::1 - [28/Jul/2016:19:02:56 +0200] "POST /filemanager/ HTTP/1.1" 403 14
::1 - [28/Jul/2016:19:03:07 +0200] "POST /filemanager/ HTTP/1.1" 403 14
::1 - [28/Jul/2016:19:03:13 +0200] "POST /filemanager/ HTTP/1.1" 403 14
::1 - [28/Jul/2016:19:03:18 +0200] "POST /filemanager/ HTTP/1.1" 403 14
::1 - [28/Jul/2016:19:04:21 +0200] "POST /filemanager/ HTTP/1.1" 403 14
Maybe somebody can help me? Or is it just not possible to upload multiple at once?
Greetings from Germany
P.S. If you need more informations, it would nice if you provide me a link to a documentation, how i can get the information myself ;)
Copied from original issue: hacdias/filemanager-old#12

Hello @pbeckmann!
Thanks for reporting this. I have a question, when you do that, is there any error on browser's developer console (on dev tools)?
It was supposed to work...
Thanks!

From @pbeckmann on July 28, 2016 19:46
Hello,
hm, i just started the pc to look this up, but now it works :+1:
:curious:
Sorry for the disturbance.
Greetings & Thanks

No problem. :) I will always hear. Hope you enjoy it.
By the way, 403 is a forbidden error. Probably was something preventing caddy to access your files or changing its permissions.


GCE Cloud Provider doesn't display volumes or snapshots

There appears to be code for parsing/handling a disk in GCE, but this information isn't being surfaced in the UI.

@miq-bot add_label providers/google

/cc @agrare @blomquisg - this can be assigned to @selmanj

@miq-bot assign @selmanj

Volumes and snapshots should now be visible


Warnings after unit tests for dmod.core package

While the unit tests for dmod.core pass, there are some warnings that appear after test completion.  Beside the general undesirableness, the warnings are throwing off how ./scripts/run_tests.sh indicates whether dmod.core tests passed.
Current behavior
If running tests using ./scripts/test_package.sh, you will get one of the following (note that the exact number of leaked semaphores will vary per attempt):
rbartel@friday dmod % ./scripts/test_package.sh python/lib/core
Detected default virtual env directory: /Users/rbartel/Developer/noaa/dmod/venv
Activating virtual environment from /Users/rbartel/Developer/noaa/dmod/venv
===========================================================================
....................................................................................
----------------------------------------------------------------------
Ran 84 tests in 4.446s

OK
===========================================================================
Deactiving active virtual env at /Users/rbartel/Developer/noaa/dmod/venv

rbartel@friday dmod % /opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 7 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

rbartel@friday dmod %

OR
rbartel@friday dmod % ./scripts/test_package.sh python/lib/core
Detected default virtual env directory: /Users/rbartel/Developer/noaa/dmod/venv
Activating virtual environment from /Users/rbartel/Developer/noaa/dmod/venv
===========================================================================
....................................................................................
----------------------------------------------------------------------
Ran 84 tests in 4.571s

OK
===========================================================================
Deactiving active virtual env at /Users/rbartel/Developer/noaa/dmod/venv

rbartel@friday dmod % /opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:267: UserWarning: resource_tracker: '/mp-u4dy24ud': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))

Expected behavior
Tests should simply pass without the warnings.
Steps to replicate behavior (include URLs)

Run ./scripts/update_package.sh to ensure the latest package versions are installed.
Run ./scripts/test_package.sh python/lib/core one or more times.

The semaphore thing is a known issue in python itself. I don't remember which version that it goes away in.

# test.py
import multiprocessing
import unittest
rlock = multiprocessing.RLock()

def foo():
        import time
        time.sleep(1)

class Test(unittest.TestCase):
    def test_it(self):
        proc = multiprocessing.Process(target=foo)
        proc.start()
        proc.join()

python -m unittest test.py

The issue is here. I think I glossed over this and thought it had been addressed in #652. See comment.
This is not an issue if you just fork. But, since on macOS the default mp context is spawn this causes a resource leak, likewise you get the same thing if it is forkserver.

Given that it is simply an annoyance, is MacOS-only, and seems to be a subtle, difficult item to address, I'm moving this out of our immediate TODOs.  We should fix it eventually if we can, but we can live with it for now.

It's not macOS only, just the default mp context on macOS is spawn. On linux it is fork, but this will change in the future.

Just for history sake, the reason this is an issue is b.c. an multiprocessing.RLock() is created by each process (spawn = fork() -> exec()). multiprocessing.RLock() are backed by named semaphores. If fork were to be used, the same named semaphore would be used by the parent and all child processes.
spawn creates a separate resource_tracker process that tracks resources including semaphores. The resource_tracker is tracking the RLocks created in each child process and i'm not sure if the underlying named semaphore is not being cleaned up properly or if since that work is likely run as a finalizer, the resource_tracker is not being updated to reflect that the resource was relinquished.

This issue might be solved by the stream conversion ticket. I believe I removed the rlock.


Expose EndpointsServlet internals to allow customizing JSON serialization

The goal of these changes is to allow customization of the JSON serialization process of API responses, by subclassing EndpointsServlet, EndpointsMethodHandler and ServletResponseResultWriter.
An example of such a customization is available in this project.
PartialResponseEndpointsServlet provides a (hopefully) compatible implementation of the partial response feature that was dropped in Endpoints v2 (see #75).
It has some with additional goodies such as eager "fields" validation (Google APIs seem to validate it after executing backend methods), and JSON Pointer support.

Codecov Report

Merging #166 into master will decrease coverage by 0.04%.
The diff coverage is 80%.

@@             Coverage Diff              @@
##             master     #166      +/-   ##
============================================
- Coverage     80.12%   80.07%   -0.05%
- Complexity     1689     1692       +3
============================================
  Files           157      157
  Lines          5629     5637       +8
  Branches        735      735
============================================
+ Hits           4510     4514       +4
- Misses          838      843       +5
+ Partials        281      280       -1

Impacted Files
Coverage Δ
Complexity Δ

...rver/spi/response/ServletResponseResultWriter.java
94.8% <100%> (+0.06%)
18 <1> (+1)
:arrow_up:

...va/com/google/api/server/spi/EndpointsServlet.java
86.88% <50%> (-2.95%)
18 <1> (ø)

...pi/server/spi/handlers/EndpointsMethodHandler.java
92.85% <88.88%> (-7.15%)
11 <3> (+1)

.../server/spi/discovery/CommonPathPrefixBuilder.java
100% <0%> (+5.55%)
9% <0%> (+1%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 413e434...eaa5153. Read the comment docs.


[DOC] Releases infos in repositories docker-compose and dojot.

When a baseline, release ou rc was create, when the user to sees the Releases tab, he should see all infos abouts this release, something like our "Test backlog"
Maybe this should be done in the dojot / dojot, not just in the docker-compose every night.

It has donte.


RecordNotFound error from LiveQuery after removing record

Not sure who is to blame here, ember-orbit (using master branch) or orbit.js(0.16.0-beta.4), so putting this here first.
After removing a record, I am getting a RecordNotFound error, visible in the console, for this very record.
I don't have a reproduction repo, but hopefully this sequence of screenshots is able to illustrate the problem sufficiently:
The last stack trace item originating in our app itself does this:
return this.store.removeRecord({
  type: 'assumptionTree',
  id: tree.id
});

Much higher up in the stack trace, after the API source has successfully(!) deleted the record, ember-orbit`s LiveQuery, which handles a hasMany relationship for that deleted record, gets notified here:

This kicks off property change notifications on the ArrayProxy, here for arrangedContent:

Which higher up the stack trace triggers a reevaluation of LiveQuery's content getter, which in turn calls into Orbit (record here is the deleted record!):

This finally throws in Orbit's (new) RecordCache:

I guess as the error is thrown in RecordCache, this is the code that has recently changed AFAIK, and is probably causing this changed behavior. But as I said not sure who is to be blamed here, maybe this should be catched and prevented in ember-orbit?

@dgeb did you have a chance to look at this?
Sorry for my impatience, but this is currently blocking us form upgrading. And happy to help with a PR if I can, just need to better understand how a solution should look like!

I think this happens because the patch watcher on the LiveQueryArray is way too agressive. I will look in to it.

@simonihmig Sorry to delay with a reply. I'm debugging this with @tchak in person now. It seems that we need to expand Model#disconnect to also disconnect any relationship live queries.

I'm debugging this with @tchak in person now.

Sorry to bug you guys again, but has this been somehow successful?

@simonihmig yes, we identified the problem but got distracted before shipping a fix. Sorry for the delay - I'll get a fix out by tomorrow at the latest.

I have not forgotten this, but I had to merge a large PR as well as some test upgrades before getting to it. I'm working on a reproduction in the tests and a fix now ...

@simonihmig please try master - this should be fixed via #185.

@dgeb Fantastic, thank you!
I quickly tried to update it my project, but this failed due to the ember-orbit:config changes. The app uses MU, so this probably needs some fixes to our custom setup. Will try this again on Monday hopefully, and post my feedback asap!

So, I was now able to update our app to latest ember-orbit (master) and orbit (0.16-beta.x) and this indeed fixed the last remaining issue. All tests are green again :tada:
Thanks @dgeb & @tchak!

@simonihmig great! Glad you can finally update to the beta (and start using hints and the other new features) 😄


Fix version number

closes #24

Codecov Report

Merging #26 (9964b86) into master (659420c) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master      #26   +/-   ##
=======================================
  Coverage   93.61%   93.61%
=======================================
  Files           4        4
  Lines          94       94
=======================================
  Hits           88       88
  Misses          6        6

Flag
Coverage Δ

unittests
93.61% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

geojson_pydantic/__init__.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 659420c...9964b86. Read the comment docs.


Mitigate against PyYaml CVE-2017-18342

While we already use only safe_load on molecule we are better-of
forcing use of a version on PyYaml which is not affected by this
security issue.
Fixes: #1806
Signed-off-by: Sorin Sbarnea<EMAIL_ADDRESS>Please include details of what it is, how to use it, how it's been tested
PR Type

Bugfix Pull Request

I'd be inclined to leave this off based on https://github.com/ansible/molecule/pull/1808#discussion_r263610826 and https://github.com/ansible/molecule/issues/1806#issuecomment-470742420.

Awaits https://github.com/yaml/pyyaml/issues/259 :+1:

I am not sure if that closing is the right way to deal with it because is not really resolved, we are just waiting for an external dependency but our core is still affected by the use of an unsafe library.

I am not sure if that closing is the right way to deal with it because is not really resolved

Sorry if my meaning didn't come across well (or, it always feels bad to get a PR closed!) but https://github.com/ansible/molecule/issues/1806#issuecomment-470742420 remains open and is IMHO sufficient to track the progress of the new release of the library. I prefer to minimise noise in the PR/issues list where possible.

@ssbarnea we can always reopen/resubmit this once the external requirement is fulfilled. Besides, I'd like to emphasize that it is not a security issue so no need to rush.
Oh, and also it might be wise first ensure that any other dists which are usually installed along with Molecule itself will not suffer the dependencies conflict. I can imagine that if some of our (direct or transitive) deps is following typical semver practices and has pyyaml >=3,<4 to avoid unexpected breaking changes we can get pretty much unusable (impossible to install) software by locking >5.
On Molecule side we already use safe API.

cc

cc https://github.com/ansible/molecule/issues/1806#issuecomment-472506822

Closing as it should be done via https://github.com/ansible/molecule/issues/1806#issuecomment-486711379. Feel free to start doing that.


Would it be possible to introduce an option to not auto-generate a run.csx file for functions that will be deployed as DLLs

Our Deployment process involves selecting the correct function file and manually deleting the run.csx file as it is not needed for our DLL deployment.
we are deploying as a class library and only need to deploy host.json, function.json and the required DLLs. could there be a checkbox or something for the function creator to not include this file out of convenience?
Perhaps this could be in the form of a template specifically for a function that is to be deployed as a Class Library?

We have a preview project (for VS 2017) targeting this scenario, have you tried that?

no i am still using visual studio 2015. i was also referring to creating a function via the azure portal

I wanted to suggest that this be added to the portal as an enhancement.

Sorry, but today our plans are to support Functions in VS 2017 only, going forward, via the compiled Functions based project.


Allow LAN to work offline

ScratchX doesn't work offline. This is unneeded.

Will fix, this isn't just for scratchX


Incorrect rollup calculations when child records are inserted in separate transactions at the same time

Recreate issue:
Bulk Insert child records with a smaller batch size(let's say 1) and use multiple threads, the rollup calculations get messed up.
This is because the parent record update is performed in an Async Queueable context unlike the parent record being locked for sometime in native salesforce roll-ups.
Eventually, each thread tries to update the parent record rollup field parallelly but does not wait for either.
E.g. Initial Count()  - 0
Try inserting 2 records parallelly, then both try to update Count to 1 instead of 1+1 = 2.
Use Case: We have integrations that are pushing records in smaller chunks in separate transactions.

@puneetagarwal2971 what is your Max Parent Rows Updated At Once value and is that lower than your Batch Chunk Size? The parent updates shouldn't be happening in a separate queueable unless that condition is true, and if they're updated at the same time as each batch chunk is committed the proper values should be applied

@jamessimone In my scenario, I'm inserting 100 child records all linked to one single parent.
But the only tweak I'm doing is batch size = 1, and threads = 6 while inserting data.

Now, the rollup count result should be 100 but instead, it is a lot lesser.
The same happens if I delete all the related child records. Let's say we have Rollup Count = 100, and we delete all 100 child records in batch size = 1.
The Rollup Count should become 0, but again that's not the result I'm getting. Its count is decreasing but not to zero.

Have you tried setting the Is Full Record Set flag on your rollup metadata?

Yes, thanks @jamessimone! The numbers the accurate now.
For some records, I still see the error: UNABLE_TO_LOCK_ROW: unable to obtain exclusive access to this record or 1 records
which points to the parent record ID.
I'm thinking of moving to a schedule-based rollup to avoid lock-row issues. Please let me know if there's a better approach to this.
Also, thanks for building this awesome rollup package!

With a batch size that small, I could see row locks being intermittent - reason being that children associated with the same parent sent through will be contending for the update at the same time. As you said, a schedule-based rollup may be more appropriate if you can't increase your batch size for other reasons.


KeyError: 179520 ？？

while running command
bash scripts/pretrain.sh \
    --task ok --version okvqa_pretrain_1 --gpu 0

I met this problem:
Traceback (most recent call last):
  File "/root/autodl-fs/prophet-main/main.py", line 35, in <module>
    runner.run()
  File "/root/autodl-fs/prophet-main/prophet/stage1/pretrain.py", line 162, in run
    self.train(train_set, valid_set)
  File "/root/autodl-fs/prophet-main/prophet/stage1/pretrain.py", line 93, in train
    for step, input_tuple in enumerate(dataloader):
  File "/root/miniconda3/envs/prophet/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 652, in __next__
    data = self._next_data()
  File "/root/miniconda3/envs/prophet/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1347, in _next_data
    return self._process_data(data)
  File "/root/miniconda3/envs/prophet/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1373, in _process_data
    data.reraise()
  File "/root/miniconda3/envs/prophet/lib/python3.9/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/root/miniconda3/envs/prophet/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/root/miniconda3/envs/prophet/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/root/miniconda3/envs/prophet/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/root/autodl-fs/prophet-main/prophet/stage1/utils/load_data.py", line 136, in __getitem__
KeyError: 179520

whole project structure:
prophet-main
├── assets
│   ├── answer_aware_examples_okvqa.json
│   ├── answer_dict_aokvqa.json
│   ├── answer_dict_okvqa.json
│   ├── answer_dict_vqav2.json
│   ├── candidates_aokvqa_test.json
│   ├── candidates_aokvqa_val.json
│   ├── candidates_okvqa.json
│   ├── captions_aokvqa.json
│   ├── captions_okvqa.json
│   ├── examples_aokvqa_test.json
│   ├── examples_aokvqa_val.json
│   └── Untitled.ipynb
├── ckpts
│   └── epoch_6.pkl
├── CLIP
│   ├── clip
│   │   ├── bpe_simple_vocab_16e6.txt.gz
│   │   ├── clip.py
│   │   ├── __init__.py
│   │   ├── model.py
│   │   └── simple_tokenizer.py
│   ├── CLIP.png
│   ├── data
│   │   ├── country211.md
│   │   ├── prompts.md
│   │   ├── rendered-sst2.md
│   │   └── yfcc100m.md
│   ├── hubconf.py
│   ├── LICENSE
│   ├── MANIFEST.in
│   ├── model-card.md
│   ├── notebooks
│   │   ├── Interacting_with_CLIP.ipynb
│   │   └── Prompt_Engineering_for_ImageNet.ipynb
│   ├── README.md
│   ├── requirements.txt
│   ├── setup.py
│   └── tests
│       └── test_consistency.py
├── configs
│   ├── finetune.yml
│   ├── path_cfgs.py
│   ├── pretrain.yml
│   ├── prompt.yml
│   ├── __pycache__
│   │   ├── path_cfgs.cpython-39.pyc
│   │   ├── task_cfgs.cpython-39.pyc
│   │   └── task_to_split.cpython-39.pyc
│   ├── task_cfgs.py
│   └── task_to_split.py
├── datasets
│   ├── aokvqa
│   │   ├── aokvqa_v1p0_test.json
│   │   ├── aokvqa_v1p0_train.json
│   │   └── aokvqa_v1p0_val.json
│   ├── coco2014
│   │   ├── train2014
│   │   ├── train2014.zip
│   │   └── val2014
│   ├── coco2014_feats
│   │   ├── train2014
│   │   ├── train2014.zip
│   │   ├── val2014
│   │   └── val2014.zip
│   ├── coco2017
│   ├── coco2017_feats
│   ├── datasets.zip
│   ├── okvqa
│   │   ├── mscoco_train2014_annotations.json
│   │   ├── mscoco_val2014_annotations.json
│   │   ├── OpenEnded_mscoco_train2014_questions.json
│   │   └── OpenEnded_mscoco_val2014_questions.json
│   ├── old_data
│   │   ├── coco2014
│   │   └── coco2014_feats
│   ├── Untitled.ipynb
│   └── vqav2
│       ├── v2_mscoco_train2014_annotations.json
│       ├── v2_mscoco_val2014_annotations.json
│       ├── v2_OpenEnded_mscoco_train2014_questions.json
│       ├── v2_OpenEnded_mscoco_val2014_questions.json
│       ├── v2valvg_no_ok_annotations.json
│       ├── v2valvg_no_ok_questions.json
│       ├── vg_annotations.json
│       └── vg_questions.json
├── environment.yml
├── evaluation
│   ├── ans_punct.py
│   ├── aok_utils
│   │   ├── eval_predictions.py
│   │   ├── load_aokvqa.py
│   │   ├── __pycache__
│   │   └── remap_predictions.py
│   ├── aokvqa_evaluate.py
│   ├── okvqa_evaluate.py
│   ├── __pycache__
│   │   ├── ans_punct.cpython-39.pyc
│   │   ├── aokvqa_evaluate.cpython-39.pyc
│   │   └── okvqa_evaluate.cpython-39.pyc
│   └── vqa_utils
│       ├── __pycache__
│       ├── vqaEval.py
│       └── vqa.py
├── LICENSE
├── main.py
├── misc
│   ├── framework.png
│   └── tree.txt
├── outputs
│   ├── ckpts
│   │   ├── okvqa_finetune_1
│   │   ├── okvqa_heuristics_1
│   │   └── okvqa_pretrain_1
│   ├── logs
│   │   ├── okvqa_finetune_1
│   │   └── okvqa_pretrain_1
│   └── results
│       ├── okvqa_finetune_1
│       └── okvqa_heuristics_1
├── preds
├── prophet
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-39.pyc
│   ├── stage1
│   │   ├── finetune.py
│   │   ├── heuristics.py
│   │   ├── model
│   │   ├── pretrain.py
│   │   ├── __pycache__
│   │   └── utils
│   └── stage2
│       ├── prompt.py
│       └── utils
├── README.md
├── scripts
│   ├── evaluate_file.sh
│   ├── evaluate_model.sh
│   ├── extract_img_feats.sh
│   ├── finetune.sh
│   ├── heuristics_gen.sh
│   ├── pretrain.sh
│   └── prompt.sh
├── --task
├── tools
│   ├── extract_img_feats.py
│   ├── __pycache__
│   │   └── transforms.cpython-39.pyc
│   └── transforms.py
└── Untitled.ipynb

Did you prepare the image features using tools/extract_img_feats.py?

Did you prepare the image features using tools/extract_img_feats.py?

Yes. So I think it's a problem with the dataset. I'm in the process of redownloading the dataset.

Have you solve the problem?

Have you solve the problem?

I am running command: bash scripts/extract_img_feats.sh
but it seems that it will cost 50 hours or so.

So, how long do you usually need to preprocess to run this command（bash scripts/extract_img_feats.sh）

It is normal to take 50 or more hours to complete the process. And it's acceptable as it only needs to be done once.

It is normal to take 50 or more hours to complete the process. And it's acceptable as it only needs to be done once.

ok i see,thanks

Preprocessing image files may require a lot of disk space. I tried multiple times but reported an error :
 OSError: [Errno 122] Disk quota exceeded: 'datasets/coco2014_ Features/val2014/COCO_ Val2014_<PHONE_NUMBER>71. npz '.
So how much disk space does it take to run the above preprocessing command?

Hello, @ParadoxZW! First of all, I would like to express my heartfelt gratitude for your remarkable project. This project is truly fantastic and has added many highlights to my research paper. Hence, I want to take this opportunity to thank you!
Unfortunately, our laboratory has limited machine memory and can only accommodate up to 200GB of data. As a result, I couldn't reproduce your project entirely, and the final results obtained were not as satisfactory as I had hoped. Therefore, I would appreciate it if you could provide me with the performance metrics of your project without any discounts on the dataset. Thank you for your assistance!

至于上面提到的我遇到的问题，经过我的调试我发现问题出在数据集上面。因此我修改了部分代码，使得项目能够处理一部分coco2014的数据并且运行，因此不需要500G的内存就可以复现模型。但是这也意味着模型的效果不够好。
首先我在extract_img_feats.py中将main函数修改成如下的样子：
def main(__C, dataset):
    # find imgs
    img_dir_list = []
    for split in SPLIT_TO_IMGS:
        if split.startswith(dataset):
            img_dir_list.append(
                __C.IMAGE_DIR[SPLIT_TO_IMGS[split]]
            )
    print('image dirs:', img_dir_list)
    img_path_list = []
    for img_dir in img_dir_list:
        img_path_list += glob.glob(img_dir + '*.jpg')
    print('total images:', len(img_path_list))

    # load model
    clip_model, _ = clip.load(__C.CLIP_VERSION, device='cpu')
    # clip_model, _ = clip.load(__C.CLIP_VERSION, device='cuda')
    img_encoder = clip_model.visual

    model = ExtractModel(img_encoder)
    T = _transform(__C.IMG_RESOLUTION)

    failed_paths = []
    train_cnt = 0
    val_cnt = 0
    count = 5000
    train_img_idList = []
    val_img_idList = []
    for img_path in tqdm(img_path_list):
        try:
            if train_cnt > count and val_cnt <= count and 'train' in img_path:
                continue
            if val_cnt > count:
                break
            if "train" in img_path:
                train_cnt += 1
                train_img_idList.append(int(img_path.split(".")[0].split("_")[-1]))
            else:
                # print(img_path)
                val_cnt += 1
                val_img_idList.append(int(img_path.split(".")[0].split("_")[-1]))
            img_path_sep = img_path.split('/')
            img_path_sep[-3] += '_feats'
            save_path = '/'.join(img_path_sep).replace('.jpg', '.npz')
            _extract_feat(img_path, model, T, save_path)
        except Exception as e:
            print(f"_extract_feat Failed: img_path={img_path}")
            failed_paths.append(img_path)  # Append failed path to the list

    # Save failed paths to a JSON file
    json_file = './failed_paths.json'
    print(f"len_failed_paths={len(failed_paths)}")
    with open(json_file, 'w') as f:
        json.dump(failed_paths, f)

通过修改count = 5000就可以控制val和train的样本数
然后我在datasets/coco2014_feats文件夹下新增了一个python文件，用于统计处理好的数据有哪些：
import glob
import json
import os
print(os.getcwd())
def save_file_paths_as_json(folder_path, json_file_path):
    # 使用glob模块查找指定文件夹下所有以.jpg结尾的文件，并将它们存储到一个列表中
    file_list = glob.glob(folder_path + "*.npz")
    print(len(file_list))
    # 将文件路径列表保存为JSON文件
    with open(json_file_path, "w") as json_file:
        json.dump(file_list, json_file, ensure_ascii=False)

# 替换为你要查找的文件夹路径和要保存JSON数据的文件路径
img_dir = os.getcwd()+"/train2014/"
output_json_file = "train2014.json"

# 调用函数将文件路径保存为JSON文件
save_file_paths_as_json(img_dir, output_json_file)
img_dir = os.getcwd()+"/val2014/"
output_json_file = "val2014.json"

# 调用函数将文件路径保存为JSON文件
save_file_paths_as_json(img_dir, output_json_file)

然后我新建了一个python文件叫做adapt.py，放在tools文件夹下，用于将对应的json映射文件按照处理图像的情况进行修改。其内容如下：
import json
import os
import pprint
with open("../datasets/coco2014_feats/train2014.json", "r", encoding='utf-8') as f:
    train2014lst = json.load(f)
with open("../datasets/coco2014_feats/val2014.json", "r", encoding='utf-8') as f:
    val2014lst = json.load(f)
train_img_idList = []
val_img_idList = []
for i in train2014lst:
    train_img_idList.append(int(i.split(".")[0].split("_")[-1]))
for i in val2014lst:
    val_img_idList.append(int(i.split(".")[0].split("_")[-1]))

def deal():
    ####################################################################################
    # 根据处理出来的image去匹配对应的json打标情况
    ####################################################################################
    # okvqa_ann_route = "autodl-fs/prophet-main/datasets/okvqa"
    # print(f"适配ing\n\ntrain_img_idList:{train_img_idList}\ntype:{type(train_img_idList[0])}\n\nval_img_idList:{val_img_idList}\ntype:{type(val_img_idList[0])}")
    okvqa_ann_route = "../datasets/okvqa"
    okvqa_ann_route_base = os.path.join(okvqa_ann_route, "baseJson")
    annotationsJsonList = ["mscoco_train2014_annotations.json", "mscoco_val2014_annotations.json"]
    questionsJsonList = ["OpenEnded_mscoco_train2014_questions.json", "OpenEnded_mscoco_val2014_questions.json"]
    print("annotationsJsonList")
    for jsonName in annotationsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_annotations = sorted(oldJson["annotations"], key=lambda x: x["image_id"])
        newJson['annotations'] = []
        cnt = 0
        for items in sorted_annotations:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['annotations'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len ann of{jsonName}:{len(newJson['annotations'])}")
    print("questionsJsonList")
    for jsonName in questionsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_questions = sorted(oldJson["questions"], key=lambda x: x["image_id"])
        newJson['questions'] = []
        cnt = 0
        for items in sorted_questions:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['questions'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len que of{jsonName}:{len(newJson['questions'])}")
####################################################################################
    # 根据处理出来的image去匹配对应的json打标情况2
    ####################################################################################
    # okvqa_ann_route = "autodl-fs/prophet-main/datasets/okvqa"
    # print(f"适配ing\n\ntrain_img_idList:{train_img_idList}\ntype:{type(train_img_idList[0])}\n\nval_img_idList:{val_img_idList}\ntype:{type(val_img_idList[0])}")
    okvqa_ann_route = "../datasets/vqav2"
    okvqa_ann_route_base = os.path.join(okvqa_ann_route, "baseJson")
    annotationsJsonList = [
        "vg_annotations.json",
        "v2valvg_no_ok_annotations.json",
        "v2_mscoco_val2014_annotations.json",
        "v2_mscoco_train2014_annotations.json"
    ]
    questionsJsonList = [
        "vg_questions.json",
        "v2_OpenEnded_mscoco_train2014_questions.json",
        "v2_OpenEnded_mscoco_val2014_questions.json",
        "v2valvg_no_ok_questions.json"
    ]
    print("annotationsJsonList")
    for jsonName in annotationsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_annotations = sorted(oldJson["annotations"], key=lambda x: x["image_id"])
        newJson['annotations'] = []
        cnt = 0
        for items in sorted_annotations:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['annotations'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len ann of{jsonName}:{len(newJson['annotations'])}")
    print("questionsJsonList")
    for jsonName in questionsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_questions = sorted(oldJson["questions"], key=lambda x: x["image_id"])
        newJson['questions'] = []
        cnt = 0
        for items in sorted_questions:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['questions'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len que of{jsonName}:{len(newJson['questions'])}")
deal()

Regarding the issues mentioned above, after my debugging, I found that the problem lies with the dataset. Consequently, I made some changes to the code, allowing the project to handle a portion of the coco2014 data and run without requiring 500GB of memory to reproduce the model. However, this also implies that the model's performance is not as good as expected.
Firstly, in extract_img_feats.py, I modified the main function as follows:
def main(__C, dataset):
    # find imgs
    img_dir_list = []
    for split in SPLIT_TO_IMGS:
        if split.startswith(dataset):
            img_dir_list.append(
                __C.IMAGE_DIR[SPLIT_TO_IMGS[split]]
            )
    print('image dirs:', img_dir_list)
    img_path_list = []
    for img_dir in img_dir_list:
        img_path_list += glob.glob(img_dir + '*.jpg')
    print('total images:', len(img_path_list))

    # load model
    clip_model, _ = clip.load(__C.CLIP_VERSION, device='cpu')
    # clip_model, _ = clip.load(__C.CLIP_VERSION, device='cuda')
    img_encoder = clip_model.visual

    model = ExtractModel(img_encoder)
    T = _transform(__C.IMG_RESOLUTION)

    failed_paths = []
    train_cnt = 0
    val_cnt = 0
    count = 5000
    train_img_idList = []
    val_img_idList = []
    for img_path in tqdm(img_path_list):
        try:
            if train_cnt > count and val_cnt <= count and 'train' in img_path:
                continue
            if val_cnt > count:
                break
            if "train" in img_path:
                train_cnt += 1
                train_img_idList.append(int(img_path.split(".") [0].split("_")[-1]))
            else:
                # print(img_path)
                val_cnt += 1
                val_img_idList.append(int(img_path.split(".") [0].split("_")[-1]))
            img_path_sep = img_path.split('/')
            img_path_sep[-3] += '_feats'
            save_path = '/'.join(img_path_sep).replace('.jpg', '.npz')
            _extract_feat(img_path, model, T, save_path)
        except Exception as e:
            print(f"_extract_feat Failed: img_path={img_path}")
            failed_paths.append(img_path)  # Append failed path to the list

    # Save failed paths to a JSON file
    json_file = './failed_paths.json'
    print(f"len_failed_paths={len(failed_paths)}")
    with open(json_file, 'w') as f:
        json.dump(failed_paths, f)

By modifying the variable count = 5000, you can control the number of samples for both validation and training sets.
After that, I added a new Python file in the "datasets/coco2014_feats" folder to track and list the processed data：
import glob
import json
import os
print(os.getcwd())
def save_file_paths_as_json(folder_path, json_file_path):
    file_list = glob.glob(folder_path + "*.npz")
    print(len(file_list))
    with open(json_file_path, "w") as json_file:
        json.dump(file_list, json_file, ensure_ascii=False)

img_dir = os.getcwd()+"/train2014/"
output_json_file = "train2014.json"

save_file_paths_as_json(img_dir, output_json_file)
img_dir = os.getcwd()+"/val2014/"
output_json_file = "val2014.json"

save_file_paths_as_json(img_dir, output_json_file)

Then, I created a new Python file called adapt.py and placed it in the tools folder. This file is used to modify the corresponding JSON mapping file according to the image processing requirements. The contents of the adapt.py file are as follows:
import json
import os
import pprint
with open(".. /datasets/coco2014_feats/train2014.json", "r", encoding='utf-8') as f:
    train2014lst = json.load(f)
with open(".. /datasets/coco2014_feats/val2014.json", "r", encoding='utf-8') as f:
    val2014lst = json.load(f)
train_img_idList = []
val_img_idList = []
for i in train2014lst:
    train_img_idList.append(int(i.split(".") [0].split("_")[-1]))
for i in val2014lst:
    val_img_idList.append(int(i.split(".") [0].split("_")[-1]))

def deal():
    ####################################################################################
    # Match the processed images with their corresponding JSON annotations.
    ####################################################################################
    okvqa_ann_route = ".. /datasets/okvqa"
    okvqa_ann_route_base = os.path.join(okvqa_ann_route, "baseJson")
    annotationsJsonList = ["mscoco_train2014_annotations.json", "mscoco_val2014_annotations.json"]
    questionsJsonList = ["OpenEnded_mscoco_train2014_questions.json", "OpenEnded_mscoco_val2014_questions.json"]
    print("annotationsJsonList")
    for jsonName in annotationsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_annotations = sorted(oldJson["annotations"], key=lambda x: x["image_id"])
        newJson['annotations'] = []
        cnt = 0
        for items in sorted_annotations:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['annotations'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len ann of{jsonName}:{len(newJson['annotations'])}")
    print("questionsJsonList")
    for jsonName in questionsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_questions = sorted(oldJson["questions"], key=lambda x: x["image_id"])
        newJson['questions'] = []
        cnt = 0
        for items in sorted_questions:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['questions'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len que of{jsonName}:{len(newJson['questions'])}")
    ####################################################################################
    # Match the processed images with their corresponding JSON annotations. （2）
    ####################################################################################
    okvqa_ann_route = ".. /datasets/vqav2"
    okvqa_ann_route_base = os.path.join(okvqa_ann_route, "baseJson")
    annotationsJsonList = [
        "vg_annotations.json",
        "v2valvg_no_ok_annotations.json",
        "v2_mscoco_val2014_annotations.json",
        "v2_mscoco_train2014_annotations.json"
    ]
    questionsJsonList = [
        "vg_questions.json",
        "v2_OpenEnded_mscoco_train2014_questions.json",
        "v2_OpenEnded_mscoco_val2014_questions.json",
        "v2valvg_no_ok_questions.json"
    ]
    print("annotationsJsonList")
    for jsonName in annotationsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_annotations = sorted(oldJson["annotations"], key=lambda x: x["image_id"])
        newJson['annotations'] = []
        cnt = 0
        for items in sorted_annotations:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['annotations'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len ann of{jsonName}:{len(newJson['annotations'])}")
    print("questionsJsonList")
    for jsonName in questionsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_questions = sorted(oldJson["questions"], key=lambda x: x["image_id"])
        newJson['questions'] = []
        cnt = 0
        for items in sorted_questions:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['questions'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len que of{jsonName}:{len(newJson['questions'])}")
deal()

have you try to run with okvqa？

你尝试过用okvqa运行吗？

sure,but got bad result because of tiny samples.However, my current plan is to directly use the model provided by the author to execute the code for the second phase, in order to obtain the final evaluation results. (I am currently figuring out how to make the server utilize the GPT API).

Regarding the issues mentioned above, after my debugging, I found that the problem lies with the dataset. Consequently, I made some changes to the code, allowing the project to handle a portion of the coco2014 data and run without requiring 500GB of memory to reproduce the model. However, this also implies that the model's performance is not as good as expected.
Firstly, in extract_img_feats.py, I modified the main function as follows:
def main(__C, dataset):
    # find imgs
    img_dir_list = []
    for split in SPLIT_TO_IMGS:
        if split.startswith(dataset):
            img_dir_list.append(
                __C.IMAGE_DIR[SPLIT_TO_IMGS[split]]
            )
    print('image dirs:', img_dir_list)
    img_path_list = []
    for img_dir in img_dir_list:
        img_path_list += glob.glob(img_dir + '*.jpg')
    print('total images:', len(img_path_list))

    # load model
    clip_model, _ = clip.load(__C.CLIP_VERSION, device='cpu')
    # clip_model, _ = clip.load(__C.CLIP_VERSION, device='cuda')
    img_encoder = clip_model.visual

    model = ExtractModel(img_encoder)
    T = _transform(__C.IMG_RESOLUTION)

    failed_paths = []
    train_cnt = 0
    val_cnt = 0
    count = 5000
    train_img_idList = []
    val_img_idList = []
    for img_path in tqdm(img_path_list):
        try:
            if train_cnt > count and val_cnt <= count and 'train' in img_path:
                continue
            if val_cnt > count:
                break
            if "train" in img_path:
                train_cnt += 1
                train_img_idList.append(int(img_path.split(".") [0].split("_")[-1]))
            else:
                # print(img_path)
                val_cnt += 1
                val_img_idList.append(int(img_path.split(".") [0].split("_")[-1]))
            img_path_sep = img_path.split('/')
            img_path_sep[-3] += '_feats'
            save_path = '/'.join(img_path_sep).replace('.jpg', '.npz')
            _extract_feat(img_path, model, T, save_path)
        except Exception as e:
            print(f"_extract_feat Failed: img_path={img_path}")
            failed_paths.append(img_path)  # Append failed path to the list

    # Save failed paths to a JSON file
    json_file = './failed_paths.json'
    print(f"len_failed_paths={len(failed_paths)}")
    with open(json_file, 'w') as f:
        json.dump(failed_paths, f)

By modifying the variable count = 5000, you can control the number of samples for both validation and training sets. After that, I added a new Python file in the "datasets/coco2014_feats" folder to track and list the processed data：
import glob
import json
import os
print(os.getcwd())
def save_file_paths_as_json(folder_path, json_file_path):
    file_list = glob.glob(folder_path + "*.npz")
    print(len(file_list))
    with open(json_file_path, "w") as json_file:
        json.dump(file_list, json_file, ensure_ascii=False)

img_dir = os.getcwd()+"/train2014/"
output_json_file = "train2014.json"

save_file_paths_as_json(img_dir, output_json_file)
img_dir = os.getcwd()+"/val2014/"
output_json_file = "val2014.json"

save_file_paths_as_json(img_dir, output_json_file)

Then, I created a new Python file called adapt.py and placed it in the tools folder. This file is used to modify the corresponding JSON mapping file according to the image processing requirements. The contents of the adapt.py file are as follows:
import json
import os
import pprint
with open(".. /datasets/coco2014_feats/train2014.json", "r", encoding='utf-8') as f:
    train2014lst = json.load(f)
with open(".. /datasets/coco2014_feats/val2014.json", "r", encoding='utf-8') as f:
    val2014lst = json.load(f)
train_img_idList = []
val_img_idList = []
for i in train2014lst:
    train_img_idList.append(int(i.split(".") [0].split("_")[-1]))
for i in val2014lst:
    val_img_idList.append(int(i.split(".") [0].split("_")[-1]))

def deal():
    ####################################################################################
    # Match the processed images with their corresponding JSON annotations.
    ####################################################################################
    okvqa_ann_route = ".. /datasets/okvqa"
    okvqa_ann_route_base = os.path.join(okvqa_ann_route, "baseJson")
    annotationsJsonList = ["mscoco_train2014_annotations.json", "mscoco_val2014_annotations.json"]
    questionsJsonList = ["OpenEnded_mscoco_train2014_questions.json", "OpenEnded_mscoco_val2014_questions.json"]
    print("annotationsJsonList")
    for jsonName in annotationsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_annotations = sorted(oldJson["annotations"], key=lambda x: x["image_id"])
        newJson['annotations'] = []
        cnt = 0
        for items in sorted_annotations:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['annotations'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len ann of{jsonName}:{len(newJson['annotations'])}")
    print("questionsJsonList")
    for jsonName in questionsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_questions = sorted(oldJson["questions"], key=lambda x: x["image_id"])
        newJson['questions'] = []
        cnt = 0
        for items in sorted_questions:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['questions'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len que of{jsonName}:{len(newJson['questions'])}")
    ####################################################################################
    # Match the processed images with their corresponding JSON annotations. （2）
    ####################################################################################
    okvqa_ann_route = ".. /datasets/vqav2"
    okvqa_ann_route_base = os.path.join(okvqa_ann_route, "baseJson")
    annotationsJsonList = [
        "vg_annotations.json",
        "v2valvg_no_ok_annotations.json",
        "v2_mscoco_val2014_annotations.json",
        "v2_mscoco_train2014_annotations.json"
    ]
    questionsJsonList = [
        "vg_questions.json",
        "v2_OpenEnded_mscoco_train2014_questions.json",
        "v2_OpenEnded_mscoco_val2014_questions.json",
        "v2valvg_no_ok_questions.json"
    ]
    print("annotationsJsonList")
    for jsonName in annotationsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_annotations = sorted(oldJson["annotations"], key=lambda x: x["image_id"])
        newJson['annotations'] = []
        cnt = 0
        for items in sorted_annotations:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['annotations'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len ann of{jsonName}:{len(newJson['annotations'])}")
    print("questionsJsonList")
    for jsonName in questionsJsonList:
        newPath = os.path.join(okvqa_ann_route, jsonName)
        oldPath = os.path.join(okvqa_ann_route_base, jsonName)

        with open(newPath, "r", encoding='utf-8') as f:
            newJson = json.load(f)
        with open(oldPath, "r", encoding='utf-8') as f:
            oldJson = json.load(f)
        newJson = oldJson
        sorted_questions = sorted(oldJson["questions"], key=lambda x: x["image_id"])
        newJson['questions'] = []
        cnt = 0
        for items in sorted_questions:
            if items['image_id'] in train_img_idList + val_img_idList:
                newJson['questions'].append(items)
                # print(items['image_id'])
            if cnt < 0:
                cnt += 1
                print(items['image_id'], type(items['image_id']))

        with open(newPath, "w", encoding='utf-8') as f:
            json.dump(newJson, f)
        print(f"len que of{jsonName}:{len(newJson['questions'])}")
deal()

l want a question about the first sentance:
Since the author has provided the weights of the trained models, why do we still need these training sets and its features?


An arror in computing transition_matrix

When I want to run "ot_kernel.compute_transition_matrix(eps = 0.025, dt = 0.25, cost_norm_method = "mean", thresh = 0, verbose = True)", there would be an error" All array should be from the same type/backend. Current types are : [<class 'numpy.ndarray'>, <class 'pandas.core.series.Series'>, <class 'numpy.ndarray'>]". So what should I do? Thank you so much!

It looks like you're trying to pass a pandas Series where a numpy ndarray is expected. Somewhere upstream I am guessing there is a Series object (maybe the growth rates vector?). Could you try casting this to an ndarray?

I tried "adata.obs['growth_rates'] = np.array(adata.obs['growth_rates'])", but it did not work. So what to do next?

发件人: Stephen Zhang @.>
发送时间: 2022年4月3日 6:25
收件人: zsteve/StationaryOT @.>
抄送: xiangwei6963 @.>; Author @.>
主题: Re: [zsteve/StationaryOT] An arror in computing transition_matrix (Issue #3)
It looks like you're trying to pass a pandas Series where a numpy ndarray is expected. Somewhere upstream I am guessing there is a Series object (maybe the growth rates vector?). Could you try casting this to an ndarray?
―
Reply to this email directly, view it on GitHubhttps://github.com/zsteve/StationaryOT/issues/3#issuecomment-1086786073, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AN4OG4U2JMFJKWQVVJIMA4TVDETUBANCNFSM5SIVM6YA.
You are receiving this because you authored the thread.Message ID: @.***>

Then， I could not download the data"expr.npz" from github. So where could I download it? In this way, I can run StationaryOT using the example.


Crouton problems

I try to enter crouton usin sudo startunity, but it replies with this
Entering /mnt/stateful_partition/crouton/chroots/precise...
enter-chroot must be run as root.
Not unmounting /mnt/stateful_partition/crouton/chroots/precise as another instance is using it.
i have tried everything, any ideas on how to fix?

sudo startunity should work, reboot and try again.

ok thnx
On Tue, Jul 28, 2015 at 4:02 PM, DennisL<EMAIL_ADDRESS>wrote:

sudo startunity should work, reboot and try again.
—
Reply to this email directly or view it on GitHub
https://github.com/dnschneid/crouton/issues/1975#issuecomment-125431932.

changed to KDE build an have had no problems since
On Thu, Aug 6, 2015 at 10:14 AM, Samuel Gent<EMAIL_ADDRESS>wrote:

ok thnx
On Tue, Jul 28, 2015 at 4:02 PM, DennisL<EMAIL_ADDRESS>wrote:

sudo startunity should work, reboot and try again.
—
Reply to this email directly or view it on GitHub
https://github.com/dnschneid/crouton/issues/1975#issuecomment-125431932
.

good. This issue can be closed


Elements that are hidden when in the hover state come back with the hover state active

E.g. if you have some hover styles on a button. And when you click that button the button is hidden. When the button comes back since it never got the onMouseLeave it comes back with the hover state active.
JSBin
https://jsbin.com/yesisoxiti/1/edit?html,js,output
Code
<!DOCTYPE html>


Build failure with bevy_gltf and no default features

Bevy version
0.15.0
Relevant system information
cargo 1.83.0 (5ffbef321 2024-10-29)
host: x86_64-apple-darwin
What you did
cargo add bevy --no-default-features --features=bevy_gltf
cargo check

What went wrong
error[E0432]: unresolved import `bevy_animation`
 --> /Users/kpreid/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bevy_gltf-0.15.0/src/loader.rs:7:5
  |
7 | use bevy_animation::animated_field;
  |     ^^^^^^^^^^^^^^ use of undeclared crate or module `bevy_animation`

It appears that the use just needs a #[cfg(feature = "bevy_animation")].

Whoops, looks like this was fixed in #16551.


Home page

Description

On the main page, the user can see a list of Houses
When the user selects a specific item, they can see the details page

Nice changes, I'm gonna approve it :100:
There are some issues with styles that I'm gonna repair, that will be my part. I will also change the manes of files because it needs to be more clear.