Activity for the permission request is not exist.

The error.
E/flutter (15003): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(error, Activity for the permission request is not exist., null, java.lang.NullPointerException: Activity for the permission request is not exist.
E/flutter (15003): 	at com.fluttercandies.photo_manager.permission.PermissionsUtils.getPermissionsWithTips(PermissionsUtils.kt:85)
E/flutter (15003): 	at com.fluttercandies.photo_manager.permission.PermissionsUtils.getPermissions(PermissionsUtils.kt:69)
E/flutter (15003): 	at com.fluttercandies.photo_manager.core.PhotoManagerPlugin.onMethodCall(PhotoManagerPlugin.kt:209)
E/flutter (15003): 	at io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler.onMessage(MethodChannel.java:261)
E/flutter (15003): 	at io.flutter.embedding.engine.dart.DartMessenger.invokeHandler(DartMessenger.java:295)
E/flutter (15003): 	at io.flutter.embedding.engine.dart.DartMessenger.lambda$dispatchMessageToQueue$0$DartMessenger(DartMessenger.java:321)
E/flutter (15003): 	at io.flutter.embedding.engine.dart.-$$Lambda$DartMessenger$2j2MERcK825A5j1fv5sZ7xB2Iuo.run(Unknown Source:12)
E/flutter (15003): 	at android.os.Handler.handleCallback(Handler.java:938)
E/flutter (15003): 	at android.os.Handler.dispatchMessage(Handler.java:99)
E/flutter (15003): 	at android.os.Looper.loopOnce(Looper.java:233)
E/flutter (15003): 	at android.os.Looper.loop(Looper.java:344)
E/flutter (15003): 	at android.app.ActivityThread.main(ActivityThread.java:8191)
E/flutter (15003): 	at java.lang.reflect.Method.invoke(Native Method)
E/flutter (15003): 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:584)
E/flutter (15003): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1034)
E/flutter (15003): )
E/flutter (15003): #0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:653:7)
E/flutter (15003): #1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:296:18)
E/flutter (15003): <asynchronous suspension>
E/flutter (15003): #2      PhotoManagerPlugin.requestPermissionExtend (package:photo_manager/src/internal/plugin.dart:72:24)
E/flutter (15003): <asynchronous suspension>
E/flutter (15003): #3      flutterIsolateFunction (package:stack/pages/post_creation/post_creation_page.dart:36:15)
E/flutter (15003): <asynchronous suspension>
E/flutter (15003):

I'm trying to use this plugin in isolation, so i call the method below in isolation thanks to package FlutterIsolate(it gives you an ability to use flutter plugins in isolates).
@pragma('vm:entry-point')
void flutterIsolateFunction(List<int> args) async {
  final _ps = await PhotoManager.requestPermissionExtend();

  if (_ps.isAuth) {
    final paths = await PhotoManager.getAssetPathList();
    final recentPath = paths.firstWhereOrNull((el) => el.name.toLowerCase().contains("recent"));

    final assetPathEntity = recentPath ?? paths.first;

    print("HERE IS ASSET THE ASSET PATH ENTITY $assetPathEntity");
  } else {
    print("isAuth -> false");
  }
}

But it throws the error.
I tryed to use another plugin and that worked, but for photo_manager it still throws an error. I tryed many variations, but to no avil.

The exception tells everything, that in this isolate environment the plugin is not registered properly. More specifically, not with the ActivityAware. The permission must be asked with a valid Activity. Duplicate of #591 #581.
Closing as resolved.


Test for AVIF is not working in PR #26

With the current implementation, to recognize an image, it needs to be both recognized by libmagic and Pillow. And that caused test for AVIF to fail. Because while Pillow.Image can recognize it:

But libmagic cannot:

If Pillow.Image can return a meaningful result, libmagic should not block the process.

But libmagic cannot:

This line is logged by us, first i try to get mime by using magic, if magic can't get it or result is not in our supported-mime list(means we can't handle this kind of mime), i'll log this.
Actually, in the above case, application/octet-stream is result that mgaic get, magic guess mime type by the first few bytes of byte stream, same as PIL, but the types supported by PIL are limited, if PIL can recognize, magic should be able too.
It's a little strange. magic didn't get the correct mime from bytes content, i'll test and find out.

I test it by curl with cap.avif in repo, seesm normal.

I guess you are testing on macOS.
Can you try Ubuntu or Debian?

CI is now failing for the same reason:
https://github.com/v2ex/remote/actions/runs/1578163584

Sure, let me create on first😂

Seems magic not support avif now, but pillow-avif-plugin allows PIL to handle avif.

If magic gets the valid mime then use it, otherwise try to get mime by using PIL, how about that?

Maybe there is a component in my macOS that supports magic to handle avif.🤔

It seems Magic on macOS can recognize it. But the production server is running Debian with Docker.
Yes, we need to turn these tests green. Thanks!


Add configs for  PASCAL VOC datasets

🚀 Feature
It will be very convenient to add configs for  PASCAL VOC datasets, as PASCAL VOC datasets are smaller and easily trained.

I agree.
We would also need to have a more generic way of computing the mAP that doesn't require cocoAPI, or have the VOC datasets converted into COCO format.


Ansible cask needs a Python library added.

General troubleshooting steps

[X] I have retried my command with --force and the issue is still present.
[X] I have checked the instructions for reporting bugs (or making requests) before opening the issue.

[X] None of the templates was appropriate for my issue, or I’m not sure.

[X] I ran brew update-reset && brew update and retried my command.
[X] I ran brew doctor, fixed as many issues as possible and retried my command.
[X] I understand that if I ignore these instructions, my issue may be closed without review.

Description of issue
We've been using cask 'ansible' for quite some time. Lately, we've started to orchestrate hosts in Softlayer, and since then, I've been getting this error:
fatal: [localhost]: FAILED! => {"changed": false, "msg": "softlayer python library required for this module"}

I tried to add the SoftLayer library throught pip, the usual way, to no avail.
I finally found the problem, and solved as I realized this Ansible cask is using its own Python, as I executed ansible with -vvvv. I knew the python executable was /usr/local/Cellar/ansible/2.5.0/libexec/bin/python instead other python in the path.
So, then I did:
$ sudo /usr/local/Cellar/ansible/2.5.0/libexec/bin/pip install SoftLayer
Password:
The directory '/Users/mmolinac/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/mmolinac/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting SoftLayer
  Downloading SoftLayer-5.4.2-py2.py3-none-any.whl (411kB)
    100% |████████████████████████████████| 419kB 2.1MB/s
Requirement already satisfied: six>=1.7.0 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from SoftLayer)
Requirement already satisfied: requests>=2.18.4 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from SoftLayer)
Requirement already satisfied: urllib3>=1.22 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from SoftLayer)
Requirement already satisfied: pygments>=2.0.0 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from SoftLayer)
Requirement already satisfied: prompt-toolkit>=0.53 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from SoftLayer)
Requirement already satisfied: prettytable>=0.7.0 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from SoftLayer)
Requirement already satisfied: click>=5 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from SoftLayer)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from requests>=2.18.4->SoftLayer)
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from requests>=2.18.4->SoftLayer)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from requests>=2.18.4->SoftLayer)
Requirement already satisfied: wcwidth in /usr/local/Cellar/ansible/2.5.0/libexec/lib/python2.7/site-packages (from prompt-toolkit>=0.53->SoftLayer)
Installing collected packages: SoftLayer
Successfully installed SoftLayer-5.4.2

Besides this, I also checked for the library required to run GCE orchestrations, and it's there!
$ /usr/local/Cellar/ansible/2.5.0/libexec/bin/pip list | grep apache
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
apache-libcloud (2.3.0)

And from this moment, the executions are working!
*So, I'm kindly asking you to include the SoftLayer library as you're already including apache-libcloud *
Output of your command with --verbose --debug
$ ansible-playbook -i environments/Softlayer/manual_inventory -u XXXX --key-file keys/XXXX playbook-Softlayer.yml

PLAY [Create Softlayer items for Backup instances] ***************************************************************************

TASK [include_vars] **********************************************************************************************************
ok: [localhost]

TASK [Launch Backup instances] ***********************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "softlayer python library required for this module"}

PLAY RECAP *******************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1

Output of brew cask doctor
==> Homebrew-Cask Version
Homebrew-Cask 1.5.12-15-g2ec684a
caskroom/homebrew-cask (git revision 315c9b; last commit 2018-03-23)
==> macOS
10.13.3
==> SIP
Enabled
==> Java
10, 9.0.4, 9.0.1
==> Homebrew-Cask Install Location
<NONE>
==> Homebrew-Cask Staging Location
/usr/local/Caskroom
==> Homebrew-Cask Cached Downloads
~/Library/Caches/Homebrew/Cask
==> Homebrew-Cask Taps:
/usr/local/Homebrew/Library/Taps/caskroom/homebrew-cask (3944 casks)
==> Contents of $LOAD_PATH
/usr/local/Homebrew/Library/Homebrew/cask/lib
/usr/local/Homebrew/Library/Homebrew
/Library/Ruby/Gems/2.3.0/gems/did_you_mean-1.0.0/lib
/Library/Ruby/Gems/2.3.0/gems/json-2.1.0/lib
/Library/Ruby/Gems/2.3.0/extensions/universal-darwin-17/2.3.0/json-2.1.0
/Library/Ruby/Gems/2.3.0/gems/psych-3.0.2/lib
/Library/Ruby/Gems/2.3.0/extensions/universal-darwin-17/2.3.0/psych-3.0.2
/Library/Ruby/Site/2.3.0
/Library/Ruby/Site/2.3.0/x86_64-darwin17
/Library/Ruby/Site/2.3.0/universal-darwin17
/Library/Ruby/Site
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/vendor_ruby/2.3.0
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/vendor_ruby/2.3.0/x86_64-darwin17
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/vendor_ruby/2.3.0/universal-darwin17
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/vendor_ruby
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/x86_64-darwin17
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/universal-darwin17
==> Environment Variables
LC_ALL="en_US.UTF-8"
PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/Homebrew/Library/Homebrew/shims/scm"
SHELL="/bin/bash"

Output of brew tap
buo/cask-upgrade
caskroom/cask
homebrew/core

ansible isn't in Caskroom, you're looking for the Formula:
https://github.com/Homebrew/homebrew-core/blob/master/Formula/ansible.rb
I'd suggest submitting a PR to Homebrew/core.


Clarify that Omnichannel roles need to be assigned directly to users

The documentation should highlight that roles need to be assigned directly to users for the role to persona mapping to work correctly with the Omnichannel authorization logic. Omnichannel currently resolves a user's role memberships by joining the systemusers table with systemuserroles, which does not account for roles assigned through team memberships.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a472ee3d-4ca4-bfcf-2008-4e376203f493
Version Independent ID: 156e001b-25c6-7380-3956-c3b46111585c
Content: Assign roles, enable users for Omnichannel for Customer Service
Content Source: ce/customer-service/add-users-assign-roles.md
Service: dynamics-365-customerservice
GitHub Login: @neeranelli
Microsoft Alias: nenellim

@maskati,
Thank you for bringing this gap in information to my attention. It's now been addressed in the following articles:

Assign roles to users
Manage users in Omnichannel for Customer Service
Map roles to personas

I understand that there is no other action pending on the docs and therefore will close this issue. If you have any questions, please let me know.
Best regards,
Neeraja


Push new app to 0.3.23: remote: fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree

I just installed the upgrade to 0.3.23 and tried to push a new app (rq-dashboard).
From my workstation:
11:46:09 [aaron@ender:~/code] 130 $ git clone git@github.com:nvie/rq-dashboard.git
Cloning into 'rq-dashboard'...
remote: Counting objects: 945, done.
remote: Total 945 (delta 0), reused 0 (delta 0), pack-reused 945
Receiving objects: 100% (945/945), 361.93 KiB | 0 bytes/s, done.
Resolving deltas: 100% (427/427), done.
Checking connectivity... done.
11:46:13 [aaron@ender:~/code] $ cd rq-dashboard
11:46:15 [aaron@ender:~/code/rq-dashboard] $ echo 'web: PYTHONUNBUFFERED=true rq-dashboard --redis-URL=$REDIS_URL' > Procfile
11:48:32 [aaron@ender:~/code/rq-dashboard] master(+1/-1,1) ± git remote add production dokku@tetrad.utilizeit.com:rqdash
11:49:15 [aaron@ender:~/code/rq-dashboard] master(+1/-1,1) ± ssh -t<EMAIL_ADDRESS>apps:create rqdash
Creating rqdash... done
Connection to tetrad.utilizeit.com closed.
11:49:19 [aaron@ender:~/code/rq-dashboard] master(+1/-1,1) ±
11:49:29 [aaron@ender:~/code/rq-dashboard] master(+1/-1,1) 130 ± git push production master
Counting objects: 948, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (442/442), done.
Writing objects: 100% (948/948), 362.19 KiB | 0 bytes/s, done.
Total 948 (delta 428), reused 944 (delta 427)
-----> Cleaning up...
-----> Building rqdash from buildstep...
-----> Checking deploymentkeys Plugin sanity ...
-----> Installing shared SSH keys in build environment ...
-----> Python app detected
-----> Stack changed, re-installing runtime
-----> Installing runtime (python-2.7.9)
-----> Installing dependencies with pip
       Downloading/unpacking argparse==1.3.0 (from -r requirements.txt (line 7))
       Downloading argparse-1.3.0-py2.py3-none-any.whl
       Downloading/unpacking arrow==0.5.4 (from -r requirements.txt (line 8))
<snip out a lot of python build stuff...>
       Successfully installed argparse arrow click ecdsa enum34 fabric flake8 flask grin ipaddress isort itsdangerous jinja2 markupsafe mccabe natsort paramiko pep8 pies2overrides pies pycrypto pyflakes python-dateutil redis rq six werkzeug wheel
       Cleaning up...

-----> Discovering process types
       Procfile declares types -> web
-----> Preparing Supervisor ...
-----> Injecting Supervisor ...
-----> Releasing rqdash...
remote: fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
remote: Use '--' to separate paths from revisions, like this:
remote: 'git <command> [<revision>...] -- [<file>...]'
To dokku@tetrad.utilizeit.com:rqdash
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'dokku@tetrad.utilizeit.com:rqdash'
11:51:17 [aaron@ender:~/code/rq-dashboard] master(+1/-1,1) 1 ±

I tried mv /var/lib/dokku/plugins /var/lib/dokku/plugins.old and re-installing to get just the 'standard' plugins.  No effect.
I'll post some of the trace output in a minute...

'Release' output with trace turned on.
-----> Releasing rqdash...
remote: + [[ -f /home/dokku/dokkurc ]]
remote: + source /home/dokku/dokkurc
remote: + [[ -d /home/dokku/.dokkurc ]]
remote: + for f in '$DOKKU_ROOT/.dokkurc/*'
remote: + source /home/dokku/.dokkurc/DOKKU_TRACE
remote: ++ export DOKKU_TRACE=1
remote: ++ DOKKU_TRACE=1
remote: + [[ -n 1 ]]
remote: + set -x
remote: + parse_args release rqdash buildstep
remote: + for arg in '"$@"'
remote: + case "$arg" in
remote: + for arg in '"$@"'
remote: + case "$arg" in
remote: + for arg in '"$@"'
remote: + case "$arg" in
remote: + return 0
remote: + args=("$@")
remote: + [[ release =~ ^--.* ]]
remote: + has_tty
remote: ++ /usr/bin/tty
remote: ++ true
remote: + [[ not a tty == \n\o\t\ \a\ \t\t\y ]]
remote: + return 1
remote: + DOKKU_QUIET_OUTPUT=1
remote: ++ id -un
remote: + [[ dokku != \d\o\k\k\u ]]
remote: + [[ -n '' ]]
remote: + case "$1" in
remote: + implemented=0
remote: + for script in '$PLUGIN_PATH/*/commands'
remote: + set +e
remote: + /var/lib/dokku/plugins/00_dokku-standard/commands release rqdash buildstep
remote: ++ dirname /var/lib/dokku/plugins/00_dokku-standard/commands
remote: + source /var/lib/dokku/plugins/00_dokku-standard/../common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + case "$1" in
remote: + APP=rqdash
remote: + IMAGE=dokku/rqdash
remote: + IMAGE_SOURCE_TYPE=buildstep
remote: + case "$IMAGE_SOURCE_TYPE" in
remote: + pluginhook pre-release rqdash
remote: + pluginhook pre-release-buildstep rqdash
remote: ++ dirname /var/lib/dokku/plugins/20_events/pre-release-buildstep
remote: + source /var/lib/dokku/plugins/20_events/../common/functions
remote: ++ set -eo pipefail
remote: ++ [[ -n 1 ]]
remote: ++ set -x
remote: + [[ ! -n '' ]]
remote: + [[ -f /home/dokku/ENV ]]
remote: ++ docker run -i -a stdin dokku/rqdash /bin/bash -c 'mkdir -p /app/.profile.d && cat > /app/.profile.d/00-global-env.sh'
remote: + id=f522321562e478ce696a3f8435d529226f0c5b1dd706f434a97caea2b4273baf
remote: ++ docker wait f522321562e478ce696a3f8435d529226f0c5b1dd706f434a97caea2b4273baf
remote: + test 0 -eq 0
remote: + docker commit f522321562e478ce696a3f8435d529226f0c5b1dd706f434a97caea2b4273baf dokku/rqdash
remote: + [[ -f /home/dokku/rqdash/ENV ]]
remote: + pluginhook post-release rqdash
remote: + APP=rqdash
remote: + IMAGE=dokku/rqdash
remote: ++ GIT_DIR=/home/dokku/rqdash
remote: ++ git rev-parse HEAD
remote: fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
remote: Use '--' to separate paths from revisions, like this:
remote: 'git <command> [<revision>...] -- [<file>...]'
remote: + GIT_REV=HEAD
remote: + exit_code=128
remote: + set -e
remote: + [[ 128 -eq 10 ]]
remote: + implemented=1
remote: + [[ 128 -ne 0 ]]
remote: + exit 128
remote: + exit_code=128
remote: + set -e
remote: + [[ 128 -eq 10 ]]
remote: + implemented=1
remote: + [[ 128 -ne 0 ]]
remote: + exit 128
remote: + exit_code=128
remote: + set -e
remote: + [[ 128 -eq 10 ]]
remote: + implemented=1
remote: + [[ 128 -ne 0 ]]
remote: + exit 128
remote: + exit_code=128
remote: + set -e
remote: + [[ 128 -eq 10 ]]
remote: + implemented=1
remote: + [[ 128 -ne 0 ]]
remote: + exit 128
+ exit_code=0
+ set -e
+ [[ 0 -eq 10 ]]
+ implemented=1
+ [[ 0 -ne 0 ]]
+ for script in '$PLUGIN_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/link/commands git-receive-pack ''\''rqdash'\'''
+ [[ git-receive-pack == link ]]
+ [[ git-receive-pack == link:* ]]
+ case "$1" in
+ exit 10
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/nginx-vhosts/commands git-receive-pack ''\''rqdash'\'''
++ dirname /var/lib/dokku/plugins/nginx-vhosts/commands
+ source /var/lib/dokku/plugins/nginx-vhosts/../common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+ case "$1" in
+ exit 10
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/ps/commands git-receive-pack ''\''rqdash'\'''
++ dirname /var/lib/dokku/plugins/ps/commands
+ source /var/lib/dokku/plugins/ps/../common/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
++ dirname /var/lib/dokku/plugins/ps/commands
+ source /var/lib/dokku/plugins/ps/functions
++ set -eo pipefail
++ [[ -n 1 ]]
++ set -x
+++ dirname /var/lib/dokku/plugins/ps/commands
++ source /var/lib/dokku/plugins/ps/../common/functions
+++ set -eo pipefail
+++ [[ -n 1 ]]
+++ set -x
+ case "$1" in
+ exit 10
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ for script in '$PLUGIN_PATH/*/commands'
+ set +e
+ /var/lib/dokku/plugins/shell/commands git-receive-pack ''\''rqdash'\'''
+ case "$1" in
+ exit 10
+ exit_code=10
+ set -e
+ [[ 10 -eq 10 ]]
+ continue
+ [[ 1 -eq 0 ]]
+ exit 0
To dokku@tetrad.utilizeit.com:rqdash
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'dokku@tetrad.utilizeit.com:rqdash'
11:53:45 [aaron@ender:~/code/rq-dashboard] master(+1/-1,1) 1 ±

I also tried creating my own new git repo, copied and committed the files I needed and did a git push.  Same issue.  It does not appear to affect pushing existing projects.

Fixed it.
Apparently puppet was re-copying a plugin we needed (dokku-git-rev) back into /var/lib/plugins/ and it was causing the bomb.  Turns out there's an updated version that fixes the issue.  Sorry for the noise.


must override run and assemble scripts together

if you only supply one or the other, the resulting image fails to build.

@soltysh I thought we had an issue for this but i couldn't find one..feel free to close if you have one.

I have an open PR for that #167, but it's far from being ready, as I've find some bugs in it, it's on top of my todo list.

@bparees can you close this one?


Small improvements for german translation.

Sorry, I forgot to change it back. I did another pull request...


Why ping <IP_ADDRESS> to check connections?

Hello!
I just cannot install because of this error:
Checking environment...
We can't connect to the Internet, check your network!
May i suggest you remove this? Because some ppl (like me) use static routes and block <IP_ADDRESS>. Why not just ping github or pimoroni.com?
`check_network() {
sudo ping -q -w 10 -c 1 <IP_ADDRESS> | grep "received, 0" &> /dev/null && return 0 || return 1
}
launch_url() {
check_network || (error_box "You don't appear to be connected to the internet, please check your connection and try again!" && exit 1)
if command -v xdg-open > /dev/null; then
xdg-open "$1" && return 0
else
error_box "There was an error attempting to launch your browser!"
fi
}
get_install() {
check_network || (error_box "You don't appear to be connected to the internet, please check your connection and try again!" && exit 1)
if [ "$1" != diagnostic ];then
sysupdate && UPDATE_DB=true
fi
if ! command -v curl > /dev/null; then
apt_pkg_install "curl"
fi
curl -sS https://get.pimoroni.com/$1 | bash -s - "-y" $2
read -p "Press Enter to continue..." < /dev/tty
`}``

if you know what you are doing you can use:
curl https://get.pimoroni.com/speakerphat | bash -s - "-y"


crypto/x509: support DirectoryName name constraints

I would like to request for the adoption of change 3230 which is in code review for a long time.

This change extents the Name Constraint properties by adding the Excluded property for DNSDomains and both the permitted and excluded properties for EmailDomains, IPAddresses and DirectoryNames as specified for the GeneralName property in RFC5280.
The selected properties are required to create or validate a fully constrained certificate.
The change also improves the validation of Name Constraints in general and allows the parsing of certificates that have Name Constraints marked as cirtial.

https://go-review.googlesource.com/#/c/3230/
Change-Id: Idaa7abafec372d5eb444cad7ee2ea5794aee3424
To be able to validate all certificates issued according to the CA / Browser Forum Baseline Requirements the full set of name constraints need to be available in GO.
A recent version of the CA / Browser Forum Baseline Requirements states that Technical Constraints in Subordinate CA Certificates MUST be applied via Name Constraints. To support strong and strict certificate path validation and to allow users to see the actual constraints it's important that GO supports the required Name Constraints.
Section 9.7 of the baseline requirements states:

"If the Subordinate CA Certificate includes the id-kp-serverAuth extended key usage, then the Subordinate CA Certificate MUST include the Name Constraints X.509v3 extension with constraints on dNSName, iPAddress and DirectoryName as follows:-"

The full requirements can be found on: https://cabforum.org/baseline-requirements-documents/
The DirectoryName is also needed in some Microsoft environments. Forbidding all directory names would enforce domain validated certificates even if all certificates under a specific root are used and issued by the same organisation. Or could enforce all certificates to be issued with the same OV certificate details. Email addresses are not specifically handled by the CABForum because they don't cover client auth or s/mime certificates currently but likely to have the same requirements and make the set of constraints supported complete.

Assigning to @agl to decide the fate of this. His last comment on CL 3230 was:

I don't think that there's a need to handle verifying email addresses, nor to process DirectoryNames (other than to accept forbidding all DirectoryNames).

Not handling directory names is against RFC 5280 section <IP_ADDRESS>:

Applications conforming to this profile MUST be able to process name
constraints that are imposed on the directoryName name form and
SHOULD be able to process name constraints that are imposed on the
rfc822Name, uniformResourceIdentifier, dNSName, and iPAddress name
forms.

Handeling these name constraints should have minimal impact as the extension is only processed when actually present in the certificate

Has there been any more thought regarding this issue?
We are seeing public CA certificates issued with critical name constraint (per RFC 5280) and Excluded subtrees for IP addresses (per the Baseline requirements https://cabforum.org/baseline-requirements-documents/).
I think x509 should be able to parse BR-compliant CAs.

@tadukurow can you paste an example for testing?
I thought that I had done this but I see that I forgot to hit submit on https://go-review.googlesource.com/c/36900/. Also, that only supports it for names so I suspect that I'll want to extend it first.

Here is an example for a CA which does not allow ips in the san of its leaf certificates as per baseline requirements.
 X509v3 Name Constraints: critical                 Excluded:                   IP:<IP_ADDRESS>/<IP_ADDRESS>                   IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0
-----BEGIN CERTIFICATE-----
MIIFojCCA4qgAwIBAgIJAOMWrsjU/s6/MA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV
BAYTAkdCMQ8wDQYDVQQHDAZMb25kb24xEDAOBgNVBAoMB0RlbW8gQ0ExCzAJBgNV
BAsMAklUMB4XDTE3MDUwNDE1NDQyM1oXDTM3MDQyOTE1NDQyM1owPTELMAkGA1UE
BhMCR0IxDzANBgNVBAcMBkxvbmRvbjEQMA4GA1UECgwHRGVtbyBDQTELMAkGA1UE
CwwCSVQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCy6709MthSnIg6
jl9+fgSdqdok56ElUVVaWiF0skZy17skS05zkdLXT15Su4Ffqs7kRawO4IHZvNVc
oVGOkz4xnG4UT8w9dhtu10yUBbs/hi4/KWtdUFNvu2xU0vEzxuLjCZNyOmy5+rYB
TfD6RIZ3eaElE5dZv0syZfG3WoVJMCYZt3KL7O0FtXSd6lCqUNE+bcbxEZ6Av4aJ
NMGMKK+3ML1lDwmyDalUnkOlcTkxlUt2mBt4MUHzc3fmjY4GgfjlyjKp1bzrBVU5
sGrGGiZgH/kYvqABsUyaf3ejvg413YB66bAOr1sMvm1HGnqock8NpEYCJmWGDBox
2s6o/nNO1i1+eE11Iyty24wx1Wwx1VMXd08CYDPpdhkkfN9LK+8yK7CpOViOy6/a
Xc2h7W0QHvWXCHGoEPDjGPKhSVzB2YR/+Nxlei9SjiI1cOE0ThTH1ikO3/3+S3yT
KGZUJT8Mlra7IUBGfbwm63WRyEI9rlowhtGnA7bNtRPekiA95TqYmf5ZGYyksCso
wnYMU8Onm0JkDu5smeV+UGuqV5kXRiCOKoxdbGiJXWYZHNXEc1rnnDzN5TMJK3S2
qLh0IpAJtQwhS++Oi0oTD3bjsyF9RPtDFnxYbxQJYGO/ig5WG1MQ97QMcUu94ll6
vcjm8B14OUEkSzOVvxJtsaeldquRWQIDAQABo4GkMIGhMB0GA1UdDgQWBBQBMXxW
PYExW1n/j0gJPuO7idEwQDAfBgNVHSMEGDAWgBQBMXxWPYExW1n/j0gJPuO7idEw
QDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBhjA+BgNVHR4BAf8ENDAy
oTAwCocIAAAAAAAAAAAwIocgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAwDQYJKoZIhvcNAQELBQADggIBAAxNFDaTK70DtoX5NkWpfiNau1yyFGryygqb
cTQAwkm8U+qp3Ktwf59eQfCd6fwq0aZXHbF+oLTuJrF9SWrx7RG5maZjAHXQKC52
x5c6jYoWxhBiWcBGf9nYMTs16PKi0gRr5MLWaGsDWG8hMxZPeDW7S7c8Z1UXEul3
6AyHKMON56lqB7i2/b1VB18eDW8g6Y7fyWvNhJBJx5tgAq+zgQNBaKfmmwP3XoRG
2WZXN28+vX6rgpSG9oCei26A4/BYEp37tC2VMTaJ8rvAWaRr+B3jYes94a25E/q1
Vp1/IZBY8nCMZgsz5tusROI+rQPdZPV8AJVSgQsfPZFmBY9V9vMYkuiuTTtnzNCo
vrVU5sQWau+oOKYz/nsip0A7Bq1ZKhcSs3Bzpn1sh870yLWL1ALOLQf2SGrK/lRK
+jcPSifqcme7kUQqjX2+gSn9j0deM3UMoEEfVNt39jnzmDVlxyFzhszLCZFzE+X4
sKuKSsrSCVZdzqC3SKgWrJLTEJajoI5I3pFf4hM9kuinlijOQXtQjPERWeBiUhEH
BqFeLZOvDMcru4UPmaDe5G3J5vNDzk3jvNwc/w1hvESgb8/aiNZFYqNN0bL1RXAH
7LknjK/MZvCZ+nBvjWQMfLhbn6TZqiRHSo55e3DCiCCtffEdGjiPOUdjWRPd97YE
p0Oon62s
-----END CERTIFICATE-----

Hi @agl,
I see that ExcludedDNSDomains is included in the beta for 1.9, but what about the remaining name constraints?
	PermittedEmailDomains       []string
	ExcludedEmailDomains        []string
	PermittedIPAddresses        []net.IPNet
	ExcludedIPAddresses         []net.IPNet
	PermittedDirectoryNames     []pkix.Name
	ExcludedDirectoryNames      []pkix.Name

These name constraints are needed to parse publicly trusted CA certs, for example, this publicly trusted CA certificate with critical Name Constraints (redacted, I can probably share via a private message if needed).
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ???
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=BE, O=GlobalSign nv-sa, CN=GlobalSign CA for AATL - SHA256 - G2
        Validity
            Not Before: Nov 26 00:00:00 2014 GMT
            Not After : Nov 26 00:00:00 2017 GMT
        Subject: C=US, ST=California, O=???, CN=???
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                      ???
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, E-mail Protection, <IP_ADDRESS>.4.1.3<IP_ADDRESS>
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Subject Key Identifier:
                ???
            X509v3 Authority Key Identifier:
                keyid:60:C2:F1:52:3E:AD:8C:13:DC:DB:CA:0E:FA:0A:E6:2A:2C:99:4B:D8

            Authority Information Access:
                OCSP - URI:http://ocsp2.globalsign.com/gsaatlsha2g2
                CA Issuers - URI:http://secure.globalsign.com/cacert/gsaatlsha2g2.crt

            X509v3 Name Constraints: critical
                Permitted:
                  email:.???.com
                  email:???.com
                  DNS:???.com
                  DirName: C = US, ST = California, O = ???
                  DirName: C = US, O = ???
                  DirName: O = ???
                Excluded:
                  IP:<IP_ADDRESS>/<IP_ADDRESS>
                  IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.globalsign.com/gs/gsaatlsha2g2.crl

            X509v3 Certificate Policies:
                Policy: <IP_ADDRESS>.4.1.4146.1.60
                  CPS: https://www.globalsign.com/repository/
                Policy: <IP_ADDRESS>.4.1.2774.2.100

    Signature Algorithm: sha256WithRSAEncryption

As also required by the Mozilla Root Store Policy:

If the certificate includes the id-kp-serverAuth extended key usage, then the certificate MUST be Name Constrained as described in section 7.1.5 of version 1.3 or later of the Baseline Requirements.
If the certificate includes the id-kp-emailProtection extended key usage, it MUST include the Name Constraints X.509v3 extension with constraints on rfc822Name, with at least one name in permittedSubtrees, each such name having its ownership validated according to section <IP_ADDRESS> of the Baseline Requirements.

https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#technically-constrained

The other types did not make it into 1.9, which is now frozen I'm afraid. Shouldn't be a stretch for 1.10, although we do not support validating against emails or directory names so some thought is required about how that would be exposed. (I.e. don't expose and ignore them, even if critical, since we'll never use them in the stdlib and let people extract the extension if they care?)

@agl Directory names are used to chain certificates together issuer<->subject, so directory name constraints are always relevant. Also the name constraint enforcement is supposed to be done independently of what kind of name is being validated. That is, if there is an email address constraint and that constraint is violated, then the certificate chain should be rejected even if you're validating a TLS server name.

That is, if there is an email address constraint and that constraint is violated, then the certificate chain should be rejected

That's the "word of God" approach to certificate validation. I.e. that once validated, everything in the certificate is valid. Go does not take that approach and always considers validity in a specific context.

I absolutely don't like the approach of ignoring the critical flag, these constraints and the flag is there with a good reason. While I see your argument that Go does not follow the "word of God", which is fine. Let's instead verify what "God" tells us and do more on top of that where possible. If we start leaving out parts of the standards and validation logic the mess is only getting bigger.
We can't say that the users need to parse the extension them selfs as this would mean that they also have to perform the full chain validation and certificate handling them self which is not even possible in all use cases. Name constraints are a key part of the certificate chain validation.
Instead of exposing all name constraint details in the x509.Certificate struct I can imagine that we would prefer to put the name constraints in its own linked struct like pkix.Name. We could also decide to include verification options in x509.VerifyOptions that let users choose to disable name constraint checking.
Another approach is to add a hook to the certificate validation logic to handle 'unknown' or override/extend known extension handling. This way users can fulfill any type of critial extension (also thinking about OCSP must-staple) and perform full name constraints validation in the certificate chain without the need to rewrite the x509 package.

Hi Adam, you probably already know this, but the design, (lack of) documentation, and code of the Verify() and VerifyHostname() functions suggest that it's good to do it the way I suggest.
For example, IIUC, VerifyOptions.DNSName isn't supposed to be required, but IIUC it is required now if and only if there are any “permitted” name constraints, which seems weird.
It also seems reasonable to call Verify() once (perhaps with a DNSName) and then call VerifyHostname() one or more times with different hostnames, for example if one is trying to implement the HTTP/2 connection coalescing feature. However, VerifyHostname() doesn't check the name it is given against the name constraints, IIUC.
I think if you want to continue doing things as you suggest then it might be better to do the name constraint check in VerifyHostname() instead of in isValid(). Note that isValid() calls VerifyHostname() when the DNSName option is given.
Sorry if I'm misunderstanding something.

Brian, you're not misunderstanding. (Although I don't think Paul's argument that context-dependent validation is ok but ignoring these constraints is not, is sound. One implies the other unless we're adding contexts for email and so on.)
When the Go X.509 library was first written it was quite unclear whether context-dependent validations were going to be the norm or not, mostly because of DANE. With a DNSSEC-based system, it's not possible to implement a Voice of God-style validation because the authority is inherently limited.
But, as you note, the API was frozen long ago and isn't well demarcated. And, for a long time, it didn't really matter whether people got it right or not.
I've written a CL previously to switch Go over to Voice of God but it didn't end up landing because it was somewhat large and I was overtaken by other stuff. VOG is less surprising and, after Google decided to buttress X.509 rather than replace it, and with Let's Encrypt eliminating most of the friction for large numbers of people, VOG is looking fairly dominant.
So reviving that effort might be a sensible prereq here. I assume it would make Paul happy because VOG implies checking all constraints up-and-down the chain at verification time.
(Although there's still some wooly thinking happening from what I can see. If intermediates need to have negative constraints for all the common other SAN types, then how do new name types ever get added? By that scheme, it would need every constrained intermediate to be updated before it would be safe. At best, I guess, we can declare that all the existing name types got this wrong but, for new ones, any name constraints implicitly exclude them.)

I've heard off-issue from some people working with SPIFFE that need URI-based Name Constraints, and SPIFFE in turn is hopefully going to be written in Go and used by a variety of systems, including gRPC and Istio.
I'm trying to understand the current state here. @agl, it sounds like you are okay with moving to the VOG approach to X.509 validation, and he may still have an old CL doing much of the work. The issue is titled "add Name Constraints", though. How would moving to VOG address that issue, or what new API would be needed?
Thanks.

I spoke to @agl, and he confirmed that he believes moving to VOG is the right thing to do. He hopes to do that for Go 1.10, and then adding URI-based Name Constraints would follow. If there are crypto experts who want to help, please let us know.

@rsc, @agl, my previously submitted patch to implement this can't merge and anymore but I can contribute some coding and/or testing time to fix this patch or to create something new. Have there been any further thoughts about the implementation?
I was thinking that instead of creating a list for 'each' general name type in x509.Certificate it might be cleaner to create one GeneralName interface in crypto/x509/pkix?
crypto/x509
type NameConstraints struct {
  Permitted pkix.GeneralName
  Excluded pkix.GeneralName
}

// Permitted verifies if the given GeneralName is allowed to be used according to the
// name constraint rules. The name must be explicitly be Permitted or no other names
// of the given type must be listed as permitted or excluded.
func (n *NameConstraints) Permitted(pkix.GeneralName) bool {
  ...
}

type Certificate struct {
  ...
  NameConstraintsCritical bool // if true then the name constraints are marked critical
  NameConstraints NameConstraints
  ...
}

NameConstraintsCritical could also be exposed as function Critital() on NameConstraints?
crypto/x509/pkix
type GeneralName interface{
  String() string
}

type RFC822Name string
type DNSName string
type DirectoryName pkix.Name
type UniformResourceIdentifier net.URL
type IPAddress net.IP
...

@agl thanks for working on this, can we please include directoryName too?
directoryName is used a key restriction to prevent constrained sub-CAs to issue under a non-validated/allowed subject DN.

@agl thanks for working on this, can we please include directoryName too?

Based on internal feedback from people who know more about X.509 than I, directoryName constraints were less useful and so I didn't include them in the first CL. The other thing that needs to get done this cycle is EKU checking at chain building time (in order to match the constraints behaviour). After that, it's possible but November is past approaching.

@agl, what remains here? Anything for Go 1.10?

@agl Thank you for working on this, based on the milestone change, is it safe to assume this will hit with 1.11?

@jeffcpullen, that's not a safe assumption. It previously had Milestone Go1.10 and wasn't included in Go 1.10.

What's in 1.10 is everything except constraints on directory names, which seemed to be less important.
DNS, Email, and IP constraints should all be enforced, and EKUs are also enforced as a tree property. (Glancing over the big table in the test file should give some idea of what the behaviour will be.)

Per @agl's earlier comment, all that's left here is DirectoryName name constraints.
It's unclear whether that will happen or not so changing back to NeedsDecision
and updating milestone to Go 1.12.

I got hit by this issue while using Prometheus. It fails to check any TLS service, rendering the software much less useful.
Our internal CA use dirname name constraint (and also ip, email, dns name constraint). Any TLS connection using our CA fails with: "x509: unhandled critical extension", with cert.UnhandledCriticalExtensions as "[<IP_ADDRESS>] (Name Constraints)". I wasn't able to identify which name constraint failed as code simply considers the whole extension unhandled. However, checking the code, it looks like it was dirname. After that, it was simple to find this issue.
Please, consider including this improvements ASAP. Any popular TLS implementation does support Directory Name name constraints. It would be a shame to discard any golang products simply because it does not have a couple of lines of code. RFC5280 says "Applications conforming to this profile MUST be able to process name constraints that are imposed on the directoryName name form and SHOULD be able to process name constraints that are imposed on the rfc822Name, uniformResourceIdentifier, dNSName, and iPAddress name forms." golang implements all the optional restrictions ("SHOULD") but missed the required restriction ("MUST").
Now, as this is my first day with golang, would it be possible to monkey patch golang internal code with a fix?
As a suggestion, it would be nice if error message could contain the unhandled extension like: "x509: unhandled critical extension [<IP_ADDRESS>]". It would be much better if it could inform somehow which entry was not recognized appending some info to the error message or even adding it to cert.UnhandledCriticalExtensions (or turning it from an array to a map).

Besides the usage of name constraints in internal CA, there are lots of public CA using directory name constraints (https://censys.io/certificates?q=((parsed.extensions.name_constraints.excluded_directory_names%3A*+OR+parsed.extensions.name_constraints.permitted_directory_names%3A*))+AND+tags.raw%3A+"unexpired"+).
However, I could not find any certificate that sets name_constraints as critical:
https://censys.io/certificates/help?q=parsed.extensions.name_constraints.critical%3A+true
They might have opted to let it be a "best effort" restriction as there are TLS implementations that does not support all kinds of constraints. Go shouldn't be one of those to be blamed.

Hello, I'm very interested in the support of critical name constraints. I am stuck accessing a system with a cert using a critical name constraint!

@cwawak , it does not seem that go staff thinks that this is important.
If you are building the software and you can patch the go used, I have patches since 1.15 that you can apply and build your own "go" that actually works.
https://github.com/luizluca/go/tree/1.15/nameconstraint
...
https://github.com/luizluca/go/tree/1.18/nameconstraint
Just pick the version you need. That is the best I can do for now.
Those branches are outdated.
For 1.19, I need to (again) solve the conflicts.

I still think this should be supported by Go core, but maybe we can also think about a different approach, for example by extending the core implementation with a registrable extension handler. In this case a handler could be registered for each extension object identifier, if registered, that package will be used for verification and/or parsing and prevent failures for unknown critical extensions.
An Extension interface could provide these optional Verify and Parse functionalities which would be called by the relevant certificate functions or the user directly.

Hi @vanbroup

I still think this should be supported by Go core, but maybe we can also think about a different approach, for example by extending the core implementation with a registrable extension handler.

If you are suggesting a change in API, then that would need to go through the proposal process.
If this would not need to go through the proposal process, would you mind very briefly elaborating on why not?
(Sorry, just a quick comment, and I haven't looked at any of the details here).

Hi @vanbroup

I still think this should be supported by Go core, but maybe we can also think about a different approach, for example by extending the core implementation with a registrable extension handler.

If you are suggesting a change in API, then that would need to go through the proposal process.

The suggestion of an API change is only because go is failing to implement parts of RFC5280 like, in this case, the MANDATORY directoryName name constraint RFC5280:

Applications conforming to this profile MUST be able to process name
constraints that are imposed on the directoryName name form and
SHOULD be able to process name constraints that are imposed on the
rfc822Name, uniformResourceIdentifier, dNSName, and iPAddress name
forms.

Even the dNSName is optional, but not the directoryName.

However, I could not find any certificate that sets name_constraints as critical:\n\nhttps://censys.io/certificates/help?q=parsed.extensions.name_constraints.critical%3A+true

Currently this search returns at list trusted CA certificates with name constraints critical and a directory name included:
https://search.censys.io/search?resource=certificates&q=(parsed.extensions.name_constraints.critical%3A+true)+and+labels%3D`trusted`+and+parsed.extensions.name_constraints.permitted_directory_names%3A*+↩+

If you are suggesting a change in API, then that would need to go through the proposal process.\n\nIf this would not need to go through the proposal process, would you mind very briefly elaborating on why not?\n\n(Sorry, just a quick comment, and I haven't looked at any of the details here).

This issue has been open for about 7+ years, it followed the same process as for other name constraints implementations.
This is a minor modification to the implementation of name constraints which retains API compatibility. It extends the x509.Certificate struct with the permitted and excluded directory trees and validates these according RFC 5280 as supposed to.
Without proper validation of (directory) name constraints the process should fail (with critical name constraints) or would continue otherwise, putting the user at risk when the constraints are not fullfilled.

Hi Go team, what would be the next steps toward a fix for this issue?  A few comments in a PR indicate that a decision should be made in the issue tracker before proceeding.  Is there sufficient information in this thread to make that call?  Would it be better to open a separate formal change proposal?
Thanks!


Edit: How to set Working Directory

Hi there,
I;m able to use your amazing library.
However i have a question, how can i set working directory?
I tried CD path but Bin/sh always cd to wrong path.
Thanks

Hi @isolatedperson,
I'm not related to this project but I've worked with it. It seems to me that your question is actually related to Unix* systems and not to this library, am I wrong?
Are you executing /bin/sh -e "..."? your question needs more details for me to understand your actual problem.


Miscellaneous code refactoring

The aim of this refactoring is to improve code readability and maintainability.

Maven Code Style page documents the steps. It is easiest to apply the code style to the entire project. I have done that and pushed the changes. It however makes the diff huge ;). I can split the formatting changes in a different PR, if you'd like that.

Maven Code Style page documents the steps. It is easiest to apply the code style to the entire project. I have done that and pushed the changes. It however makes the diff huge ;). I can split the formatting changes in a different PR, if you'd like that.

@amCap1712 Thanks for updating the code style.  I was able to review the changes by commit - I'll go ahead and merge, but I won't squash this one to preserve the individual improvement commits.

@amCap1712 Thanks for updating the code style.  I was able to review the changes by commit - I'll go ahead and merge, but I won't squash this one to preserve the individual improvement commits.


feat(besu): enable support for deployment without vault

Description
As a developer I want to enable option to deployment bevel besu network without using external vaults.
Addition description
The crypto materials needs be stored on Kubernetes secrets and mounted to the containers. The chart needs to have conditions to use the mounts when no vault is selected
Acceptance Criteria

Deployment works without error
Each org has the option to choose or not to choose vault

I'll take this issue


Table 172 - Standard Customer Sales Code - CreateSalesInvoice

I would like to request a new puiblisher in function CreateSalesInvoice of table 172 standard customer sales code:
procedure CreateSalesInvoice(OrderDate: Date; PostingDate: Date)
var
SalesHeader: Record "Sales Header";
Handled : Boolean; //Also new
begin
//>> START
OnBeforeCreateSalesInvoice(Rec, OrderDate, PostingDate, Handled)
if Handled then exit;
//<< END
TestField(Blocked, false);
SalesHeader.Init();
SalesHeader."No." := '';
SalesHeader."Document Type" := SalesHeader."Document Type"::Invoice;
OnCreateSalesInvoiceOnBeforeSalesHeaderInsert(SalesHeader, Rec);
....

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Earnings report cookbook

This is a condensed version of the demo for extracting earnings reports to CSV.
Overview:

Shows how to use Outlines to structure CSV output
Provides simple tools for converting a table specification to regular expressions
Includes a tuned extraction prompt that performs reasonably well on income statements

Is this ready for review?

Yes, sorry!

Could I get a bump on this one? Need to start linking to it.


do not run at the lates IDEA version

It seems that it has not been maintained, but it has encountered issues in the new version. Help me!

I have the same problem


API docs?

Specifically interested in the helper class.
I see a lot of examples, which are helpful, but is there anywhere I can look (besides src code) to see the methods and properties etc. enumerated and explained?
e.g., this line from the examples:
var mail = new helper.Mail(from_email, subject, to_email, content)
It's not clear to me what arguments the constructor takes besides those included here, or which arguments are required, etc. etc.
I see the helper class doc section here, but all it has is the link to the examples and  a page in Sendgrid docs that don't seem to say anything about this specific library (only the endpoint).

Hello @brandonmp,
That constructor is only designed for that basic use case. For full usage, please check out this example: https://github.com/sendgrid/sendgrid-nodejs/blob/master/examples/helpers/mail/example.js#L15
For further documentation, check out: https://sendgrid.com/docs/API_Reference/api_v3.html
Please let me know if I can further assist. Thanks!
With Best Regards,
Elmer

thanks @thinkingserious . the examples are helpful, but was hoping for more traditional docs of the SDK itself. understand if they're not available, though.
Can you tell me, though, in general: if I want to leave all of the params of the email (subject, body, etc.) to the template, and only provide the to, from, and any %variables% in my API call, how do I do that?
I've been trying to work through the examples, but so far it seems like if I omit content or subject I get an exception.
for example:
// @flow
const Sendgrid = require('sendgrid')
const helper = Sendgrid.mail

type CreateEmailInputs = {
  from: string,
  to: string,
  subject?: string,
  content?: string,
  templateId: string,
  variables: { [varName: string]: string }[]
};

const createEmail = (
  { from, to, subject, content, templateId, variables }: CreateEmailInputs,
  sendgridInstance: typeof Sendgrid) => {

  const from_email = new helper.Email(from)
  const to_email = new helper.Email(to)

  // not clear on why i need this, but w/o actual
  // html in content, the template won't send'
  const mailContent = new helper.Content('text/html', '<div />')

  const mail = new helper.Mail(from_email, subject, to_email, mailContent)
  mail.setTemplateId(templateId)

  if (variables && Array.isArray(variables)) {
    variables.forEach((v) => {
      const varName = Object.keys(v)[0]
      mail.personalizations[0].addSubstitution(
        new helper.Substitution(varName, v[varName])
      )
    })
  }
  const request = sendgridInstance.emptyRequest({
    method: 'POST',
    path: '/v3/mail/send',
    body: mail.toJSON(),
  })

  return request

}

When I call this function with a valid to/from, 1 template variable, and undefined content/subject, I'm getting this exception:
   TypeError: Cannot read property '0' of undefined
        at variables.forEach.v (/home/bmp/storage/code/wayhome/micro-functions/src/utils/create-email.js:30:7)
        at Array.forEach (native)
        at createEmail (/home/bmp/storage/code/wayhome/micro-functions/src/utils/create-email.js:28:15)
        at /home/bmp/storage/code/wayhome/micro-functions/src/wayhome-invitation/handle-invitation-deletion.js:47:19
        at Generator.next (<anonymous>)
        at step (/home/bmp/storage/code/wayhome/micro-functions/src/wayhome-invitation/handle-invitation-deletion.js:1:191)
        at /home/bmp/storage/code/wayhome/micro-functions/src/wayhome-invitation/handle-invitation-deletion.js:1:361
        at process._tickDomainCallback (internal/process/next_tick.js:129:7)

Even in the example you linked to, the subject is provided to the constructor:
email = new<EMAIL_ADDRESS>"Example User")
Is there a way to omit subject & content & simply defer to whatever values the template has?

Hello @brandonmp,
I'll have to give that a try (without subject and content). Meanwhile, maybe this example will help: https://github.com/sendgrid/sendgrid-nodejs/blob/master/USE_CASES.md#transactional-templates

We will be updating the docs for this repo when we merge this PR: https://github.com/sendgrid/sendgrid-nodejs/pull/378


Graph Expression Object returns null for childless graph

Hi,
When I use the graphExpressionObject method on a graph that has no children, it returns null instead of $modify information. May I know the reason for this? I didn't see any information about this in the document.
My code block
.withGraphFetched(`
    [
      games(
        filterNews, filterDeleted
      )
    ]
  `)
  .modifyGraph('games', builder => {
    console.log(builder.graphExpressionObject())
 }

Actual behavior
return null
Expected behavior
{
  '$name': 'games',
  '$relation': 'games',
  '$modify': [
    'filterNews',
    'filterDeleted'
  ]
}

He is curious about the repository and expects a different result so asks a question about it. Closing the issue without a reply is not nice @koskimas


"

wont handle " correctly

What do you mean? Maybe you need to escape it in your use case (i.e: " ). Sorry for the late reply.


Howto retry out-of-order

Say, I am polling a database object to get into a particular state. At the same time an external event may come informing that the state has changed. It would be beneficial to cause OOO-retry to occur now to avoid the delay until the next retry. Is it possible to force an out-of-order retry before the next scheduled one occurs? If it is, what is the right way to do that?

There is not, really. I'd be interested to know if you have any suggestion for how this should work, in terms of the API, if you were to tweak the current API.

Closing for now. Feel free to open with some example of how this might look in the API.


[DISCUSSION] Inconsistencies in pagination of child objects.

This ticket is a discussion ticket, based on discussion from this week's DSpace 7 Mtg (on March 15).
As noted in that meeting, currently when generating paginated lists of child objects (e.g. bitstreams within an item, items within a collection, collections within a community), we currently seem to have two strategies in place.
It's unclear whether there is a reason for having two pagination strategies, or if we should standardize on one. It's also unclear whether we need to modify the REST API or Angular UI, or both.  This ticket should be seen as a way to document these strategies in order to come to a consensus on how / whether to standardize.
Strategy 1: Obtain child objects using scope parameter & paginate
In this strategy, we use the object endpoint and pass along a scope parameter in order to obtain child objects within a specific scope (i.e. parent object).  We then paginate these objects.

REST API calls look like /items?scope=[uuid] (This is how we currently obtain items under a collection, where the uuid passed in is a specific Collection's UUID)
Responses from REST API provide page information at a level above the _embedded array of objects. For example, a response for the above sample call looks like:

{
    "_embedded": {
        "items": [...]
    },
    "_links": {...},
    "page": {...}
}

Important Note on Strategy 1: Currently, the Angular UI pagination code assumes the API response will always be of the format provided by this strategy.
Known areas of Angular UI where Strategy 1 is used:

Listing top-level Communities
Listing Items within a Collection

Strategy 2: Obtain child objects using embedded objects & paginate
In this strategy, we use obtain child objects (of a parent) via embedded objects in the REST API request.  (Unfortunately UI pagination does not work in this scenario at this time. See below)

REST API look like /community/[uuid]/collections (This is how we currently obtain collections under a community).
Responses from REST API provide page information at the same level of the _embedded array of objects. For example, a response for the above sample call looks like:

{
 "collections": {
   "_embedded": [...],
   "_links": {...},
   "page": {...}
 },
 "logo": {...}
}

Important Note on Strategy 2: However, it is worth noting that the current Pagination code in Angular UI does not work properly when the page information and array of objects are at the same level of the API response. So, areas of the Angular UI that use this strategy currently do not provide pagination (but, we should find a way to allow them to provide pagination.)
Known areas of Angular UI where Strategy 2 is used:

Listing Collections in a Community
Listing Bitstreams in an Item

@abollini : Thanks for the notes there.  It seems we didn't have the full context in this discussion.
In addition, as you discovered, it looks like our Strategy #2 example is a bit mistated.
In the Angular UI, when we are finding the Collections in a Community, it looks like the code currently uses the embedded Collection returned by a Community response (https://dspace7.4science.it/dspace-spring-rest/#https://dspace7.4science.it/dspace-spring-rest/api/core/communities/9076bd16-e69a-48d6-9e41-0238cb40d863)  As you noted, in that scenario, the page information appears "nested" -- and this is the scenario where the Angular UI pagination code currently fails.
However, it sounds like if we did a second request to https://dspace7.4science.it/dspace-spring-rest/#https://dspace7.4science.it/dspace-spring-rest/api/core/communities/9076bd16-e69a-48d6-9e41-0238cb40d863/collections then the response would look like Strategy #1 (and likely the Angular pagination code would succeed, as this is the format it is looking for.)


nested Route with relative path support

Hi,
this should be support relative path on nested route
use case:

<Route path='/home' >
     <Switch>
          <Route nested path='/section1' />
          <Route nested path='/section2' />
          <Route nested path='/section3' />
     </Switch>
</Route >

is same as
<Route path='/home/section1'>

Thanks, but this is part of a larger discussion here: #5127


Cannot retrieve a value of type double w/o fraction. Zero is returned instead.

Which service(blob, file, queue, table) does this issue concern?
table
Which version of the Azurite was used?
3.12.0
Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)
Visual Studio 2022. I have both channels installed Preview and Current.
What's the Node.js version?
v10.19.0
What problem was encountered?
A value of type double is stored as Int32 when no fraction is provided.
The Int32 value is not converted back to double instead zeo is returned.
Steps to reproduce the issue?

Create a table using Microsoft Azure Storage Explorer
Create a row with PartitionKey, RowKey and a double value with fraction. Example: pk, rk, 5.7
Create a second row with a double value w/o fracktion: pk, rk, 5
Retrieve 1st row. Result: 5.7
Retrieve 2nd row. Result: 0.0

Have you found a mitigation/solution?
I have temporarily suspended Azurite and returned to Azure Storage Emulator.

Thanks for raising.
We shall take a look and address this.

Hi,
This issue appears resolved, feel free to reopen if not.
If I double click in storage explorer, I see the expected double values (5.7,5).
Please ensure that you are using a recent and supported version of NodeJs (14+), and that you are using the most recent version of Azurite.

See issue #1796 for more precise reproduction steps


install on windows

followed your instructions, but

got the same error:
File "E:_AI\ComfyUI\nodes.py", line 1872, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in call_with_frames_removed
File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone_init.py", line 1, in
from .nodes import NODE_CLASS_MAPPINGS
File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\nodes.py", line 22, in
from src.models.unet_2d_condition import UNet2DConditionModel
File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\src\models\unet_2d_condition.py", line 40, in
from .unet_2d_blocks import (
File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\src\models\unet_2d_blocks.py", line 15, in
from .transformer_2d import Transformer2DModel
File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\src\models\transformer_2d.py", line 7, in
from diffusers.models.embeddings import CaptionProjection
ImportError: cannot import name 'CaptionProjection' from 'diffusers.models.embeddings' (C:\Users****\AppData\Local\Programs\Python\Python311\Lib\site-packages\diffusers\models\embeddings.py)
Cannot import E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone module for custom nodes: cannot import name 'CaptionProjection' from 'diffusers.models.embeddings' (C:\Users****\AppData\Local\Programs\Python\Python311\Lib\site-packages\diffusers\models\embeddings.py)
I got this version of diffusers:
Name: diffusers
Version: 0.25.0
Have you tried the Tri fix button?

got the same error: File "E:_AI\ComfyUI\nodes.py", line 1872, in load_custom_node module_spec.loader.exec_module(module) File "", line 940, in exec_module File "", line 241, in call_with_frames_removed File "E:AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone__init.py", line 1, in from .nodes import NODE_CLASS_MAPPINGS File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\nodes.py", line 22, in from src.models.unet_2d_condition import UNet2DConditionModel File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\src\models\unet_2d_condition.py", line 40, in from .unet_2d_blocks import ( File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\src\models\unet_2d_blocks.py", line 15, in from .transformer_2d import Transformer2DModel File "E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\src\models\transformer_2d.py", line 7, in from diffusers.models.embeddings import CaptionProjection ImportError: cannot import name 'CaptionProjection' from 'diffusers.models.embeddings' (C:\Users****\AppData\Local\Programs\Python\Python311\Lib\site-packages\diffusers\models\embeddings.py)
Cannot import E:_AI\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone module for custom nodes: cannot import name 'CaptionProjection' from 'diffusers.models.embeddings' (C:\Users****\AppData\Local\Programs\Python\Python311\Lib\site-packages\diffusers\models\embeddings.py)
I got this version of diffusers: Name: diffusers Version: 0.25.0
Have you tried the Tri fix button?

yeah, tryied all buttons )
i think requirements need to be installed in venv, but i dont know how to do it in comfy

install via git link in manager
then python tools/download_weights.py
then restart server
working

install via git link in manager then python tools/download_weights.py then restart server
working

I wish this were true, but I've tried installing this way and every other way i can think of besides a brand new comfy ui installation and each time the nodes fail to load.

install via git link in manager then python tools/download_weights.py then restart server
working

I wish this were true, but I've tried installing this way and every other way i can think of besides a brand new comfy ui installation and each time the nodes fail to load.

delete this node folder and try again

install via git link in manager then python tools/download_weights.py then restart server
working

I wish this were true, but I've tried installing this way and every other way i can think of besides a brand new comfy ui installation and each time the nodes fail to load.

delete this node folder and try again

Yup, after moving the pretrained weights that worked. Ty

install via git link in manager then python tools/download_weights.py then restart server
working

I wish this were true, but I've tried installing this way and every other way i can think of besides a brand new comfy ui installation and each time the nodes fail to load.

delete this node folder and try again

Yup, after moving the pretrained weights that worked. Ty

Where did you move the weights? I think what happens is if you downloaded them once theyre just simlinks to the cache, so have to delete cache before attempting again. But you resolved it by moving the weights folder?

install via git link in manager then python tools/download_weights.py then restart server
working

I wish this were true, but I've tried installing this way and every other way i can think of besides a brand new comfy ui installation and each time the nodes fail to load.

delete this node folder and try again

Yup, after moving the pretrained weights that worked. Ty

Where did you move the weights? I think what happens is if you downloaded them once theyre just simlinks to the cache on subsequent installs, so have to delete cache before attempting again. But you resolved it by moving the weights folder?

Simlink it ok

install via git link in manager then python tools/download_weights.py then restart server
working

I wish this were true, but I've tried installing this way and every other way i can think of besides a brand new comfy ui installation and each time the nodes fail to load.

delete this node folder and try again

Yup, after moving the pretrained weights that worked. Ty

Where did you move the weights? I think what happens is if you downloaded them once theyre just simlinks to the cache on subsequent installs, so have to delete cache before attempting again. But you resolved it by moving the weights folder?

Simlink it ok

install via git link in manager then python tools/download_weights.py then restart server
working

I wish this were true, but I've tried installing this way and every other way i can think of besides a brand new comfy ui installation and each time the nodes fail to load.

delete this node folder and try again

Yup, after moving the pretrained weights that worked. Ty

Where did you move the weights? I think what happens is if you downloaded them once theyre just simlinks to the cache on subsequent installs, so have to delete cache before attempting again. But you resolved it by moving the weights folder?

Simlink it ok

The fix doesnt work for me, no matter how many times I try it.

install via git link in manager then python tools/download_weights.py then restart server
working

I wish this were true, but I've tried installing this way and every other way i can think of besides a brand new comfy ui installation and each time the nodes fail to load.

delete this node folder and try again

Yup, after moving the pretrained weights that worked. Ty

have you moved the pretrained weights somewhere? or you meant that you've removed and redownloaded them?

This worked for me, hope it helps.
TL;DR: On Windows use the python_embeded executable from the command line.
Prerequisite: Download ComfyUI-Moore-AnimateAnyone via the ComfyUI Manager or GitHub

Open up a new command line window.

CD to the root directory of your ComfyUI installation

Run the following command in the command line ...

python_embeded\python.exe -m pip install -r .\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\requirements.txt

Download the weights.

python .\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\tools\download_weights.py

Move the pretrained_weights dir from the root ComfyUI directory into the Moore-AnimateAnyone dir

YOUR-COMFY-ROOT\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\pretrained_weights

This worked for me, hope it helps.
TL;DR: On Windows use the python_embeded executable from the command line.
Prerequisite: Download ComfyUI-Moore-AnimateAnyone via the ComfyUI Manager or GitHub

Open up a new command line window.
CD to the root directory of your ComfyUI installation
Run the following command in the command line ...

python_embeded\python.exe -m pip install -r .\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\requirements.txt

Download the weights.

python .\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\tools\download_weights.py

Move the pretrained_weights dir from the root ComfyUI directory into the Moore-AnimateAnyone dir

YOUR-COMFY-ROOT\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\pretrained_weights

You sir, are legendary

Actually, moving the folder didnt seem to work for me, but installing the custom node from root folder fixed the 'Failed to Import' issue. But when moving pretrained weights folder from root after downloading weights, I got errors when trying to Queue Prompt. The solution in my case was to delete the weights folder again and run the .py file from the custom node AnimateAnyone folder.

Actually, moving the folder didnt seem to work for me, but installing the node through comfyUI manager, and the requirements.txt from comfyUI root folder fixed the 'Failed to Import' issue. But when moving pretrained weights folder from root after downloading weights, I got errors when trying to Queue Prompt. The solution in my case was to delete the weights folder again and run the .py file from the custom node AnimateAnyone folder.
To reiterate, the only solution I found was:
Download ComfyUI-Moore-AnimateAnyone via the ComfyUI Manager
Open up a new command line window.
CD to the root directory of your ComfyUI installation
Run the following command in the command line ...
python_embeded\python.exe -m pip install -r .\ComfyUI\custom_nodes\ComfyUI-Moore-AnimateAnyone\requirements.txt
CD to the ComfiUI-Moore-AnimateAnyone folder in your custom_nodes folder
Download the weights python tools\download_weights.py

Thanks Sire!

I've followed the steps above, and it's not working for me.  I keep getting the 'Failed to Import' error in the startup console, and the ComfyUI Manager still shows (IMPORT FAILED) in the Install Missing Plugins window.

The download_weights.py is using hf_hub_download helper.
So it creates a hidden .cache/huggingface folder in your root then attempts to symlink back to it. Then the paths the repo is using to look for pretrained_weights on start up includes linux forward slashes so thats why that fails too. This could be made a lot simpler by just listing which models to use from " animate anyone " and let users set it up without issues ..


Python versions

Codecov Report

Merging #28 (3acd79d) into master (c530c32) will decrease coverage by 0.73%.
The diff coverage is 95.23%.

@@             Coverage Diff             @@
##            master      #28      +/-   ##
===========================================
- Coverage   100.00%   99.26%   -0.74%
===========================================
  Files            4        5       +1
  Lines          125      136      +11
===========================================
+ Hits           125      135      +10
- Misses           0        1       +1

Impacted Files
Coverage Δ

isabelle_client/socket_communication.py
100.00% <ø> (ø)

isabelle_client/compatibility_helper.py
87.50% <87.50%> (ø)

isabelle_client/__init__.py
100.00% <100.00%> (ø)

isabelle_client/isabelle__client.py
100.00% <100.00%> (ø)

isabelle_client/utils.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c530c32...3acd79d. Read the comment docs.


Auto-teleport ED followup

from @hassnian

atm the autoteleport will check the transferable balance (balance - ed = transferrable balance ) and then check if that amount is greater than the one needed plus current chain balance

I was thinking of introducing some logic

If the account would be terminated, but you could save the funds by teleporting the whole amount -> pass

Enable auto-teleport if there are enough funds on the relay chain - if the account would drop below the ed after auto-teleport, we will teleport the whole amount.
The user won't technically lose funds and can finish the desired action.
worst thing that can happen is that you would teleport a max of 0.999 DOT more to the parachain
wdyt?

👋


Popover arrow renders on the wrong side

The popover component  https://tailwind.primevue.org/popover/ renders the arrow on the wrong side. I thought it was something I did wrong and tried it on your documentation example and the behaviour is the same.
I moved the button in the example to the far right and popover rendered the arrow on the left
https://prnt.sc/iSaPW3nCwQGy

Cannot replicate the issue. If you think the problem still persists, please reopen this issue with a reproducer link.

I have the same problem
for fix need to add this code into popover.css
.p-popover:after, .p-popover:before {
     left: calc(1.25rem + var(--p-popover-arrow-left))
}

cause: on original primevue u have this rule:
.p-popover:after, .p-popover:before {
     ......
     left: calc(var(--p-popover-arrow-offset) + var(--p-popover-arrow-left));
     .....
}

and tailwind version css have just
.p-popover:after, .p-popover:before {
      .....
      left: 1.25rem;
      .......
}

Thanks


Params inside iarange

the current behavior of params created inside an iarange (aka vectorized map_data) is a bit nonsensical: a vector of params of length batch_size will be created and used on every iteration. this means that a param gets reused for whatever data point happens to be selected at that index in a batch. this comes up eg when there is a mean-field model for a local variable.
(there is a reasonable work-around, by creating the vector of needed params globally, and only selecting from it in the iarange.)
a better behavior would be to automatically associate params to their index into the data tensor, pulling out the appropriate params for a given batch (and creating them as needed). it might be tricky to get this working, and especially to do it efficiently.
i'm flagging this as a low priority bug for now. we can think through a solution after launch.

It seems pretty simple to write this as:
with pyro.iarange("data", size, subsample_size) as ind:
    ps = pyro.param("ps", Variable(torch.ones(size), requires_grad=True)).index_select(ind)
    # ...do stuff with ps...

Here is a proposal for refactoring towards a more plate-friendly pyro.param statement.
I think we could modify param behavior by inspecting the cond_indep_stack. For each vectorized frame we could .expand(frame.size) at the appropriate dim. Determining the appropriate dim is tricky because param currently does not have a notion of event_dim, and we would need to add something akin to .independent() or an event_dim kwarg to param statements. For each non-vectorized frame we could do some simple autoname stuff like appending '_{}'.format(frame.counter). For consistency, we should probably also apply this autonaming to sample statements and avoid most of our current '_{}'.format(i) grossness.
To correctly handle subsampling, we would need to give pyro.param access to the subsampling tensors. This information can't live in the cond_indep_stack, but it could be looked-up in a trace via  trace.nodes['_subsample_{}'.format(frame.name)]. This is a little gross, but param could simply find that trace in the _PYRO_STACK. Once this information is available, vectorized param would slice its tensors by the subsample index, and sequential param would index into subsample_index[frame.counter] to create the correct param name.
To account for global params, we could use various autonaming hacks. E.g. we could avoid expanding in vectorize_particles if the cond_indep_stack's frame.name.startswith('_pyro_'). Or we could add a kwarg pyro.param(..., global=True) to avoid the proposed .expand() behavior. @eb8680 could probably find cleaner patterns based on pyro.contrib.autoname.
@eb8680 @neerajprad @ngoodman WDYT?

what if we add an abstract index argument to param (and sample)? ie something along the lines of pyro.param("foo", index=[I1 I2], ...) where Ij can be either literals or abstract indices introduced by plates (assuming we go with the proposal to have plate bind a name for the index it introduces).
that is, within an iteration the indices are just strings/numbers that get appended to the name. within a plate block the index is an object that tells us which dimension to expand along. if a param is created without and index it is treated as global.
this by itself doesn't tell us what to do with subsampling, though as @eb8680 pointed out we should probably rethink that anyhow.... maybe @fritzo's suggestions is an answer?

I like the idea of a more plate friendly param!

This information can't live in the cond_indep_stack, but it could be looked-up in a trace via trace.nodes['subsample{}'.format(frame.name)]. This is a little gross, but param could simply find that trace in the _PYRO_STACK.

It should mostly be fine, but I hope that the cost doesn't add up as we do this every time the model is run. In any case, I think we can explore other ideas of exposing this information to param.

E.g. we could avoid expanding in vectorize_particles if the cond_indep_stack's frame.name.startswith('pyro'). Or we could add a kwarg pyro.param(..., global=True) to avoid the proposed

Given our existing use cases, it might be easier to add a kwarg to plate, say global_params=Trueto denote that any params inside it do not need to be expanded. Then it will be easy for us to wrap a user model inside a plate without making any changes to the model code.
I have not thought through this, but I think this could also be extended to support iparam (e.g. by wrapping inside a plate with global_params=False) #1213, #1330.

Ok, I've demonstrated possible syntax changes in #1491 (based on #1486 ). The intent is to:

demonstrate current syntax in #1486;
work towards consensus on proposed syntax changes in #1491;
decide which changes we actually want to implement.

In addition to a batch_shape/event_shape distinction, it seems to me that there are at least two types of broadcasting operations we'd need to distinguish between - one that simply broadcasts and one that actually replicates the parameter tensor along a dimension to allow different values at each site.  This distinction corresponds to the distinction between torch.Tensor.expand() and torch.Tensor.repeat().
For example, consider the case of SVI with vectorized num_particles and local parameters created by a param statement within a model iarange.  Presumably we'd want to repeat along the model iarange, but only expand along the num_particles iarange.  It's easy enough to see that here, but what about more complicated settings, especially when a model may be used under multiple different broadcasting interpretations?
There's fundamentally ambiguity in the identity of each broadcasting dimension, and moreover expand/repeat decisions made in one setting will affect all future decisions that could be made about that parameter.  Is there a design decision we could make that would remove that ambiguity?  Should we only allow repeat when a parameter is first created, and assume all broadcasts thereafter are expands?  That's a possibility, but it introduces a dependence on the context in which the parameter was created, which sounds like a recipe for very tricky bugs.


Collect & Ideate User Stories

Please comment your user story ideas down below 👇
They'll be turned into individual User Story issues in our next Wed meeting.

As a host, I want to have support for >200 users, so I can actually use it for end-of-term presentation.

As a host, I want to know that there's no way for clients to cheat, so I can actually rely on the results.
This could be implemented using some kind of hash-based message authentication scheme.

As a host, I want to be able to choose between different types of answer modes, such as single-choice, multiple-choice, free-text and prioritisation, so I can make sure I get answers that fit my questions.

As a host, I want to be able to export results into a fitting format (e.g. CSV), so I can save them for later use.

As a poll participant, I want to be able to undo my choices before I submit them, so I can reconsider or fix accidental clicks.

As a host, I want to be able to ask multiple questions in one poll, so I can ask more specific questions instead of one generalised one, as well as make the process easier for my participants.

As a host, I want to be able to choose whether or not participants can see the poll results, so I can create polls where the results are only meant for internal use.

As a host, I want to have a visualization of the results, so I can easily evaluate the answers.

As a poll participant, I want to get a clear idea of the poll's characteristics (number of votes allowed / who can see the results / ...), so I don't make mistakes during voting.

As a host, I want to be able to run multiple polls in parallel.

As a host, I want to prepare polls to be held at a later time.

As a host, I want to run the same poll amongst different groups and compare the results.

As a host, I want to run the same poll multiple times, so I don't need to type it out again and again.

As a host, I want to show questions depending on answers to previous questions, so I can run complex surveys.

As a host, I want to be able to publish new polls to the same audience without having to re-send them connection details, so I can react to changes quickly & reduce administrative overhead.

As a host as well as a participant, I want to be able to see my poll history, so I can look at results at a later time and easily compare polls from within Liquid.

As a poll participant, I want to be able to undo my choices before I submit them, so I can reconsider or fix accidental clicks.

To extend this: In multi-question surveys, I want to be able to see a recap of all of my answers after the last question, and be able to go back to a specific question to change my answer, without having to restart the survey.

As a poll participant I want to see my own answers in comparison to the end result. This would be especially helpful if Liquid is used  for quizzes as in TI via Zoom polls.

As a host, I want to know that there's no way for clients to cheat, so I can actually rely on the results.
If we go with RatPack-based architecture, most of this should already be solved. For a relay-based one, this could be implemented using some kind of hash-based message authentication scheme.

Also regarding security:
As a host, I want to be able to create polls that are public (accessible by link/id/whatever we use) and private (with additional required password). Maybe even personalised passwords that can only be used once

As a host, I want to create a poll, so I can do polls.

As a host, I want to share a poll to clients, so I can do polls.

As a participant, I want to take part in polls, so I can take part in them.

As a host, I want to close polls and prevent new participants to vote to have a final poll result.

I think we are done here? If someone disagrees, please reopen^^


Rewriting last console output line dynamically breaks when resizing window

Windows Terminal version (or Windows build number)
Windows Terminal Preview Version: 1.9.1523.0
Other Software

DISM.exe

Example code that outputs to the console:

public class ClearConsoleOutput {
    public static void main(String[] args) throws Exception {
        String text = "Hello World ";
        for(int i=0; i<10000; i++) {
            System.out.println(text + i);
            Thread.sleep(10);
            int count = 1;
            System.out.print(String.format("\033[%dA",count)); // Move up
            System.out.print("\033[2K"); // Erase line content

            //System.out.print(String.format("\033[2J"));
        }
    }
}

Steps to reproduce
Disclaimer: if you run DISM.exe do not close the console until the process finishes.  ELSE compile and run the Example java code above
Run DISM command in Windows Terminal (PowerShell or cmd) or the Example Code above, and when text is being dynamically re-written, resize Windows Terminal window by dragging from one of the edges. Pressing Ctrl + Scroll_Wheel_Up/Down produces a similar result.
NOTE: the terminal needs to be resized such that while text is printed, the window height is smaller than the content height. The pictures below show the result after the window has been resized to a normal viewing size.
Expected Behavior
The "[==...==x%==...==]" loading bar should appear only once, as last line in the terminal, being re-written as the program dictates.
OR
"Hello World x" should be visible as a single line of text when dynamically re-written.
Actual Behavior
The line containing the loading bar is not completely cleared before it is rewritten.

The line containing the text, should appear only once. This shows the result of resizing content as well.

Conclusion:
Depending on the way the window is being resized, the output can appear: concatenated, incomplete, on separate lines, multiple times on the same line, even in the wrong order.
As seen in the updated version below, which includes a timestamp for each statement printed, the expected  output is one continuously updating line counting from 0 to 9999, but the actual output is unreadable.
Expected:

Actual: (This is me just going crazy with the "Ctrl + Scroll_Wheel" shortcut for resizing content)
The line marked with an arrow shows where the console is still writing. What you see below is the bottom part of the output of the console, it was unnecessary to capture the rest because it was just a mess.

I have found the other day an article saying that this is a known problem for consoles developed by Microsoft and that in some cases it is labeled "won't fix". I find it funny that for those posts, the solution was presented in the form: "Right click title bar to go to properties -> layout and disable Line wrapping". In the case of the Windows Terminal you cannot do that, and the only feature I can imagine might prevent some cases of this kind presented above is: #10466

Weird question. If you run your Java application for long enough, does it start to emit broken escape sequences to the Terminal? You may start seeing garbage like ;31 and 2J showing up as though they were missing the Esc.

Weird question. If you run your Java application for long enough, does it start to emit broken escape sequences to the Terminal? You may start seeing garbage like ;31 and 2J showing up as though they were missing the Esc.

Unfortunately I have not encountered this type of problem with that code in particular. My Windows Terminal character encoding settings are in their default state, as when the application was installed. Also, as stated in the issue description, the observations were made using PowerShell and cmd. Maybe differences between what character encoding the JVM uses and the character encoding of the console running in your Windows Terminal Application are making the strings (that could be changed to bytes for less processing of the output on the part of the console) be printed in incomplete form when something like this happens:

... or like this:

Note:
For the examples in this comment I used the following modification to the code in the issue description.
From:
for(int i = 0; i < 10000; i++)
To:
for(int i = 0; i < Integer.MAX_VALUE; i++)
In order to investigate further I need to know in what context did you run the java code in order to get the output you mentioned.
My setup:

PowerShell or cmd running in Windows Terminal
javac --release 8 ClearConsoleOutput.java
java ClearConsoleOutput

Conclusion:
Unfortunately I was unable to reproduce the effects you mentioned appeared while running the code for longer.
Last minute edit
Just as I was about to send the comment, I saw this symbol being printed in the console: ' o '.
As it turns out it is just a leftover "o" from printing "Hello World" incompletely... Ha ha!

Footnote
I will run this code until it reaches the max integer limit while periodically checking on the output and occasionally messing with the window ad resizing content. I'll post updates as I find them, but to be honest, the behaviour I mentioned in this issue is the one I'm most concerned about, even if there is always the posibility of creating a pop-up window from an application that needs something like a loading bar or whatever, instead of using the console directly... Consoles in general are a junky mess when trying to fiddle with the output.

Follow-up to the above comment
No instances of anyhthing like ;31 and 2J were found after the code ran overnight. I don't know how to make a script that makes the console window smaller and larger in a loop continuously, so that the output gets jumbled around, so the only thing the program did was to count towards Integer.MAX_VALUE. It currently reached 2,984,792+ iterations.
@DHowett

Footnote
I will run this code until it reaches the max integer limit while periodically checking on the output and occasionally messing
with the window ad resizing content. I'll post updates as I find them, [...]

I have to take back this statement. When I wrote it, it seemed like it won't take that long, but as I calculated it might take around 250 days for the program to complete. So, to conclude this test, I have to say that there are no problems with the output generated by the Java code I posted in the issue description, other then it may not be the best way of achieving its goal.
It does achieve it though, in a couple lines of code.
I observed the progress over 4,723,054 lines printed to the console (some re-written as the program wants them to be, others just dancing around the window when resizing the console size) and have seen multiple funny ways of making the text being printed all over the place. But no un-escaped control characters of any kind, nor other characters than the ones found in the literal string "Hello World " or in the number range [0,<PHONE_NUMBER>].
The main issue still stands described in the Tittle of this converation.
AND is not complicated at all to understand. Whenever the window is resized, content is being processed because line wrapping is enabled, and that moves the cursor around. The java program prints wherever the cursor is. That's all. Just make the cursor be wherever the, in this case, Java program knows it should be. If the cursor goes down by a line because console thinks **we want to have pretty text in the console so I just trew some pixy dust on the cursor and it fell a line down but it's ok nobody cares**, well then the toughts of the console might not travel fast enough for the JVM to notice them before printing the next line.
This either implies major, unnecessary changes to the way things work on the inside when showing text to the user in the console or it is something beyond my understanding, else it is not considered an issue, thus it is either an intended feature or a "Won't fix" problem. I suspect it should one of the first two..
I have just stated my issue I had one time with the Terminal, and again, to be honest, now I seem to have this exact same issue everywhere XD.
Hope this sets me in the clear from further code related questions, since I have minimum basic knowledge about how a console works internally, and since the issue was reported as observable by running a version of DISM.exe command that displays a progress bar that updates over time:
To @DHowett and anybody else that wants to question the Java code I used to provide people who want to see this problem in action a quick, easy and safe way of doing so: Maybe you should ask DISM.exe developers if they see escape character showing after they run their implementation of a progress bar for longer. I'm sure they will say that they couldn't have done anything more at the time about preventing it from happening, or they might pass the fault to compatibility problems with console in which the program is running. In the case of DISM.exe I suspect Microsoft is in charge with the program itself, and the console(s) in which it was intended to run. But the problem still appears even with the DISM.exe's really nice, updated on-demand, never moving the cursor below the line it occupies, progress bar. I bet it is not far from different then my implementation.

Hope this sets me in the clear from further code related questions ... anybody else that wants to question ...

I never meant to offend you. We found a bug in the JVM -- one that still exists today! -- that occasionally results in incoming VT being "double-processed" in a way that looks a lot like this, but with the addition of broken VT sequences. It would cause us to do line wrapping on the characters assuming that the entire control sequence was printable text, but then they would be stripped out after the line wrapper mangled them.

I never meant to offend you. We found a bug in the JVM -- one that still exists today! -- that occasionally results in incoming

Thanks for the kind words! I just questioned the direction of the question you posed, about what you have just explained now.
No offense taken, glad that a problem was actually in whatever parts of the environment, either console or JVM. The question sounded like targeted to a 5yo, so maybe I got a bit frustrated but I still ran the 250days long program to test what you asked me to XD.
As for your update, I cannot add any more to it. But it sound like afun discovery, so good luck and congrats!


Make First Release

We should make a first release of the Companion App, before we are able to put the app into the F-Droid store.
This is required, so we already have the "correct" version in place and the changelog generated, which is required by the Fastlane file structure, so it can be read out and be shown in the app store.

Wrong Repository


Support for 1.13

Minecraft 1.13 makes many changes to the engine, the main one beeing a complete rework on how blocks are stored : there is no longer a hard limit of 255 different blocks, so most of the hacks using block data to increase the number of actual blocks (like all wool blocks sharing the same ID with different blocks data).
Spigot will have a compatibility layer for most of the incompatibles changes, but we shouldn't rely on it.
Some links to read :
https://www.spigotmc.org/threads/minecraft-1-13-what-to-expect.288714/
https://www.spigotmc.org/threads/bukkit-craftbukkit-spigot-bungeecord-1-13-pre7-development-builds.328883/

Duplicate of #349 ? Thanks for the links though.

Oops yes definitively, looked too fast at the existing issues :)