OSX + homebrew<EMAIL_ADDRESS>pybuilder/_vendor/virtualenv.py change_prefix does not consider symlinks, raises invalid AssertionError

I have installed Python 3 on macOS 11.3 (Big Sur) with brew install<EMAIL_ADDRESS>Homebrew installs Python 3.9 under<EMAIL_ADDRESS>and symlinks<EMAIL_ADDRESS>-><EMAIL_ADDRESS>python3 is a symlink /usr/local/bin/python3 -><EMAIL_ADDRESS>pip3 is a symlink /usr/local/bin/pip3 -><EMAIL_ADDRESS>$ head -1 $(which pip3)
<EMAIL_ADDRESS>
I used pip3 install --user pybuilder to install pybuilder:
$ which pyb
/Users/username/Library/Python/3.9/bin/pyb
$ head -1 $(which pyb)
<EMAIL_ADDRESS>
$ pyb
PyBuilder version 0.12.10
Build started at 2021-04-28 10:36:15
------------------------------------------------------------
------------------------------------------------------------
BUILD FAILED - AssertionError: Filename<EMAIL_ADDRESS>does not start with any of these prefixes<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>'/Users/username/Library/Python/3.9/lib/python/site-packages', '/Users/username/.local/lib/python/3.9/site-packages', '/Users/username/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages'] (pybuilder/_vendor/virtualenv.py:1272)
------------------------------------------------------------
Build finished at 2021-04-28 10:36:15
Build took 0 seconds (64 ms)

AssertionError is raised from change_prefix:
https://github.com/pybuilder/pybuilder/blob/6ac1ef4e3f504fec7d4f86378fde1c5420d6a261/src/main/python/pybuilder/_vendor/virtualenv.py#L1272

Are there any workarounds? We're completely blocked.

Hello, unfortunately I'm completely swamped right now with work. We don't test with Brew python, but we do test with PyEnv python on Mac and it works: https://travis-ci.com/github/pybuilder/pybuilder/jobs/502734233
I would highly recommend using PyEnv with Brew regardless as Brew Python contains the following warning:

Important: Python may be upgraded to a newer version at any time. Consider using a version manager such as pyenv if you require stability of minor or patch versions for virtual environments.

https://docs.brew.sh/Homebrew-and-Python

I see why this is happening and it's brew-specific, unfortunately.

@hoxu could you please run the build with pyb -vX so that the full exception will be printed out?

Are there any workarounds? We're completely blocked.

@dindurthy you could use OSX-shipped python3 for pybuilder execution by prepending /usr/bin to PATH. Here is an example with which:
$ which -a python3
/usr/local/bin/python3
/usr/bin/python3
$ python3 --version
Python 3.9.4
$ PATH=/usr/bin:$PATH which -a python3
/usr/bin/python3
/usr/local/bin/python3
/usr/bin/python3
$ PATH=/usr/bin:$PATH python3 --version
Python 3.8.2

@arcivanov:
+ pyb -vX
[DEBUG] Registered Python environment 'system': CPython version 3.9.4.final.0 on darwin in ['/Users/username/XXX/vebuild/bin/python3.9']
PyBuilder version 0.12.10
Build started at 2021-05-04 08:48:51
------------------------------------------------------------
[DEBUG] Loading project module from ./build.py
[DEBUG] Setting up plugins VEnv at '/Users/username/XXX/.pybuilder/plugins/cpython-3.9.4.final.0'
Traceback (most recent call last):
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/cli.py", line 461, in main
    reactor.prepare_build(property_overrides=options.property_overrides,
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/reactor.py", line 203, in prepare_build
    self._setup_plugin_directory(reset_plugins, no_venvs)
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/reactor.py", line 517, in _setup_plugin_directory
    plugin_env = per["pybuilder"] = PythonEnv(plugin_dir, self).create_venv(with_pip=True,
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/python_env.py", line 209, in create_venv
    create_venv(self._env_dir,
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/python_env.py", line 429, in create_venv
    virtualenv.create_environment(
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/_vendor/virtualenv.py", line 1161, in create_environment
    install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/_vendor/virtualenv.py", line 1450, in install_python
    copy_required_modules(home_dir, symlink)
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/_vendor/virtualenv.py", line 1340, in copy_required_modules
    dst_filename = change_prefix(filename, dst_prefix)
  File "/Users/username/XXX/vebuild/lib/python3.9/site-packages/pybuilder/_vendor/virtualenv.py", line 1272, in change_prefix
    raise AssertionError("Filename {} does not start with any of these prefixes: {}".format(filename, prefixes))
AssertionError: Filename<EMAIL_ADDRESS>does not start with any of these prefixes: ['/Users/username/XXX/vebuild/Extras/lib/python'<EMAIL_ADDRESS>'/Users/username/Library/Python/3.9/lib/python/site-packages', '/Users/username/XXX/vebuild', '/Users/username/.local/lib/python/3.9/site-packages', '/Users/username/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages']
------------------------------------------------------------
BUILD FAILED - AssertionError: Filename<EMAIL_ADDRESS>does not start with any of these prefixes: ['/Users/username/XXX/vebuild/Extras/lib/python'<EMAIL_ADDRESS>'/Users/username/Library/Python/3.9/lib/python/site-packages', '/Users/username/XXX/vebuild', '/Users/username/.local/lib/python/3.9/site-packages', '/Users/username/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages'] (pybuilder/_vendor/virtualenv.py:1272)
------------------------------------------------------------
Build finished at 2021-05-04 08:48:51
Build took 0 seconds (64 ms)

Changing PATH won't work without breaking too many other things, unfortunately. I think the only workaround is to not use brew python.

So change_prefix function is comparing<EMAIL_ADDRESS>to sys.prefix, that points to symlink-containing<EMAIL_ADDRESS>on Homebrew python.
@arcivanov: wouldn't this be fixed by simply changing:
 def change_prefix(filename, dst_prefix):
-    prefixes = [sys.prefix]
+    prefixes = [os.path.realpath(sys.prefix)]

>>> sys.prefix
<EMAIL_ADDRESS>>>> os.path.realpath(sys.prefix)
<EMAIL_ADDRESS>
If so, would you accept PR with the above change?

If you could could you please retest with latest --pre and let me know?

First reproduction with 0.12.10:
$ pyb
PyBuilder version 0.12.10
Build started at 2021-09-06 08:48:40
------------------------------------------------------------
------------------------------------------------------------
BUILD FAILED - AssertionError: Filename<EMAIL_ADDRESS>does not start with any of these prefixes<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>'/Users/username/Library/Python/3.9/lib/python/site-packages', '/Users/username/.local/lib/python/3.9/site-packages', '/Users/username/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages'] (pybuilder/_vendor/virtualenv.py:1272)
------------------------------------------------------------
Build finished at 2021-09-06 08:48:40
Build took 0 seconds (68 ms)

Then test with --pre:
$ pip3 install --user --pre --upgrade pybuilder
<clip>
Successfully installed pybuilder-0.13.0.dev20210905004524
$ pyb
PyBuilder version 0.13.0.dev20210905004524
Build started at 2021-09-06 08:50:08
------------------------------------------------------------
------------------------------------------------------------
BUILD FAILED - StopIteration:  (virtualenv/run/plugin/discovery.py:22)
------------------------------------------------------------
Build finished at 2021-09-06 08:50:08
Build took 0 seconds (153 ms)

A different error, I call it progress :)
$ pyb -vX
[DEBUG] Registered Python environment 'system': CPython version 3.9.6.final.0 on darwin in<EMAIL_ADDRESS>PyBuilder version 0.13.0.dev20210905004524
Build started at 2021-09-06 08:54:14
------------------------------------------------------------
[DEBUG] Loading project module from ./build.py
[DEBUG] Setting up plugins VEnv at '/Users/username/XXX/.pybuilder/plugins/cpython-3.9.6.final.0'
Traceback (most recent call last):
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/cli.py", line 461, in main
    reactor.prepare_build(property_overrides=options.property_overrides,
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/reactor.py", line 203, in prepare_build
    self._setup_plugin_directory(reset_plugins, no_venvs)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/reactor.py", line 517, in _setup_plugin_directory
    plugin_env = per["pybuilder"] = PythonEnv(plugin_dir, self).create_venv(with_pip=True,
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/python_env.py", line 214, in create_venv
    create_venv(self._env_dir,
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/python_env.py", line 453, in create_venv
    virtualenv.cli_run(args, setup_logging=False)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/_vendor/virtualenv/run/__init__.py", line 30, in cli_run
    of_session = session_via_cli(args, options, setup_logging, env)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/_vendor/virtualenv/run/__init__.py", line 48, in session_via_cli
    parser, elements = build_parser(args, options, setup_logging, env)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/_vendor/virtualenv/run/__init__.py", line 69, in build_parser
    discover = get_discover(parser, args)
  File "/Users/username/Library/Python/3.9/lib/python/site-packages/pybuilder/_vendor/virtualenv/run/plugin/discovery.py", line 22, in get_discover
    default=next(iter(choices)),
StopIteration
------------------------------------------------------------
BUILD FAILED - StopIteration:  (virtualenv/run/plugin/discovery.py:22)
------------------------------------------------------------
Build finished at 2021-09-06 08:54:14
Build took 0 seconds (129 ms)

Is this a regression?

Yes and no. :smile:  This is apparently a bug in virtualenv, which is bundled in PyB. Let me see if VEnv can fix it on their end.

@hoxu
Would you kindly please:

pip3 install --user virtualenv into your homebrewed Python environment.
virtualenv  -vvv --with-traceback ~/testvenv
Capture the output and post it here? Does it fail?

{code}
$ realpath $(which pip3)
<EMAIL_ADDRESS>$ pip3 install --user virtualenv
Collecting virtualenv
Downloading virtualenv-20.7.2-py2.py3-none-any.whl (5.3 MB)
|████████████████████████████████| 5.3 MB 3.6 MB/s
Requirement already satisfied: six<2,>=1.9.0 in /usr/local/lib/python3.9/site-packages (from virtualenv) (1.15.0)
Collecting backports.entry-points-selectable>=1.0.4
Downloading backports.entry_points_selectable-1.1.0-py2.py3-none-any.whl (6.2 kB)
Collecting distlib<1,>=0.3.1
Downloading distlib-0.3.2-py2.py3-none-any.whl (338 kB)
|████████████████████████████████| 338 kB 4.0 MB/s
Collecting filelock<4,>=3.0.0
Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting platformdirs<3,>=2
Downloading platformdirs-2.3.0-py3-none-any.whl (13 kB)
Installing collected packages: platformdirs, filelock, distlib, backports.entry-points-selectable, virtualenv
Successfully installed backports.entry-points-selectable-1.1.0 distlib-0.3.2 filelock-3.0.12 platformdirs-2.3.0 virtualenv-20.7.2
$ realpath $(which virtualenv)
/Users/username/Library/Python/3.9/bin/virtualenv
$ virtualenv -vvv --with-traceback /tmp/testvenv

1790 created virtual environment CPython3.9.7.final.0-64 in 1677ms
creator CPython3Posix(dest=/private/tmp/testvenv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/username/Library/Application Support/virtualenv)
added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING main:19]
$ echo $?
0
{code}
So nope, doesn't fail. Let me know if you need the full output regardless.

Could you please post the shebang of the virtualenv file (first line)?
And yes please post the whole output.

$ head -n1 $(which virtualenv)
<EMAIL_ADDRESS>$ realpath<EMAIL_ADDRESS><EMAIL_ADDRESS>
$ virtualenv -vvv --with-traceback /tmp/testvenv
114 setup logging to NOTSET [DEBUG report:43]
140 find interpreter for spec<EMAIL_ADDRESS>[INFO builtin:62]
140 proposed PythonInfo(spec=CPython3.9.7.final.0-64<EMAIL_ADDRESS>platform=darwin, version='3.9.7 (default, Sep  3 2021, 12:37:55) \n[Clang 12.0.5 (clang-1<IP_ADDRESS>)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:69]
140 accepted PythonInfo(spec=CPython3.9.7.final.0-64<EMAIL_ADDRESS>platform=darwin, version='3.9.7 (default, Sep  3 2021, 12:37:55) \n[Clang 12.0.5 (clang-1<IP_ADDRESS>)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:71]
143 filesystem is not case-sensitive [DEBUG info:28]
186 create virtual environment via CPython3Posix(dest=/private/tmp/testvenv, clear=False, no_vcs_ignore=False, global=False) [INFO session:52]
186 create folder /private/tmp/testvenv/bin [DEBUG _sync:25]
186 create folder /private/tmp/testvenv/lib/python3.9/site-packages [DEBUG _sync:25]
187 write /private/tmp/testvenv/pyvenv.cfg [DEBUG pyenv_cfg:34]
187     home =<EMAIL_ADDRESS>[DEBUG pyenv_cfg:38]
187     implementation = CPython [DEBUG pyenv_cfg:38]
187     version_info = 3.9.7.final.0 [DEBUG pyenv_cfg:38]
187     virtualenv = 20.7.2 [DEBUG pyenv_cfg:38]
187     include-system-site-packages = false [DEBUG pyenv_cfg:38]
187     base-prefix =<EMAIL_ADDRESS>[DEBUG pyenv_cfg:38]
187     base-exec-prefix =<EMAIL_ADDRESS>[DEBUG pyenv_cfg:38]
187     base-executable =<EMAIL_ADDRESS>[DEBUG pyenv_cfg:38]
187 symlink<EMAIL_ADDRESS>to /private/tmp/testvenv/bin/python [DEBUG _sync:44]
188 create virtualenv import hook file /private/tmp/testvenv/lib/python3.9/site-packages/_virtualenv.pth [DEBUG api:95]
189 create /private/tmp/testvenv/lib/python3.9/site-packages/_virtualenv.py [DEBUG api:98]
190 ============================== target debug ============================== [DEBUG session:54]
190 debug via /private/tmp/testvenv/bin/python /Users/username/Library/Python/3.9/lib/python/site-packages/virtualenv/create/debug.py [DEBUG creator:224]
190 {
  "sys": {
    "executable": "/private/tmp/testvenv/bin/python",
    "_base_executable": "/private/tmp/testvenv/bin/python",
    "prefix": "/private/tmp/testvenv",
    "base_prefix"<EMAIL_ADDRESS>    "real_prefix": null,
    "exec_prefix": "/private/tmp/testvenv",
    "base_exec_prefix"<EMAIL_ADDRESS>    "path": [
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>      "/private/tmp/testvenv/lib/python3.9/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.9.7 (default, Sep  3 2021, 12:37:55) \n[Clang 12.0.5 (clang-1<IP_ADDRESS>)]",
  "makefile_filename"<EMAIL_ADDRESS>  "os": "<module 'os' from<EMAIL_ADDRESS>  "site": "<module 'site' from<EMAIL_ADDRESS>  "datetime": "<module 'datetime' from<EMAIL_ADDRESS>  "math": "<module 'math' from<EMAIL_ADDRESS>  "json": "<module 'json' from<EMAIL_ADDRESS>} [DEBUG session:55]
248 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/username/Library/Application Support/virtualenv) [INFO session:59]
251 got embed update of distribution setuptools from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/setuptools.json [DEBUG via_disk_folder:135]
252 got embed update of distribution wheel from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/wheel.json [DEBUG via_disk_folder:135]
252 got embed update of distribution pip from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/pip.json [DEBUG via_disk_folder:135]
256 got embed update of distribution setuptools from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/setuptools.json [DEBUG via_disk_folder:135]
256 got embed update of distribution pip from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/pip.json [DEBUG via_disk_folder:135]
256 got embed update of distribution wheel from /Users/username/Library/Application Support/virtualenv/wheel/3.9/embed/1/wheel.json [DEBUG via_disk_folder:135]
257 install setuptools from wheel /Users/username/Library/Python/3.9/lib/python/site-packages/virtualenv/seed/wheels/embed/setuptools-57.4.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
257 install pip from wheel /Users/username/Library/Python/3.9/lib/python/site-packages/virtualenv/seed/wheels/embed/pip-21.2.3-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
257 install wheel from wheel /Users/username/Library/Python/3.9/lib/python/site-packages/virtualenv/seed/wheels/embed/wheel-0.37.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
258 Attempting to acquire lock<PHONE_NUMBER> on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [DEBUG filelock:270]
258 Attempting to acquire lock<PHONE_NUMBER> on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [DEBUG filelock:270]
258 Attempting to acquire lock<PHONE_NUMBER> on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [DEBUG filelock:270]
259 Lock<PHONE_NUMBER> acquired on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [INFO filelock:274]
259 Lock<PHONE_NUMBER> acquired on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [INFO filelock:274]
259 build install image for pip-21.2.3-py3-none-any.whl to /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any [DEBUG base:52]
259 Lock<PHONE_NUMBER> acquired on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [INFO filelock:274]
259 build install image for wheel-0.37.0-py2.py3-none-any.whl to /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any [DEBUG base:52]
259 build install image for setuptools-57.4.0-py3-none-any.whl to /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any [DEBUG base:52]
287 Attempting to release lock<PHONE_NUMBER> on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [DEBUG filelock:315]
287 Lock<PHONE_NUMBER> released on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [INFO filelock:318]
288 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel-0.37.0.dist-info to /private/tmp/testvenv/lib/python3.9/site-packages/wheel-0.37.0.dist-info [DEBUG _sync:52]
335 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel to /private/tmp/testvenv/lib/python3.9/site-packages/wheel [DEBUG _sync:52]
412 Attempting to release lock<PHONE_NUMBER> on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [DEBUG filelock:315]
412 Lock<PHONE_NUMBER> released on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [INFO filelock:318]
413 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools-57.4.0.dist-info to /private/tmp/testvenv/lib/python3.9/site-packages/setuptools-57.4.0.dist-info [DEBUG _sync:52]
414 copy /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel-0.37.0.virtualenv to /private/tmp/testvenv/lib/python3.9/site-packages/wheel-0.37.0.virtualenv [DEBUG _sync:52]
417 generated console scripts wheel-3.9 wheel3 wheel wheel3.9 [DEBUG base:48]
427 copy /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/distutils-precedence.pth to /private/tmp/testvenv/lib/python3.9/site-packages/distutils-precedence.pth [DEBUG _sync:52]
432 copy /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools-57.4.0.virtualenv to /private/tmp/testvenv/lib/python3.9/site-packages/setuptools-57.4.0.virtualenv [DEBUG _sync:52]
433 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools to /private/tmp/testvenv/lib/python3.9/site-packages/setuptools [DEBUG _sync:52]
529 Attempting to release lock<PHONE_NUMBER> on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [DEBUG filelock:315]
529 Lock<PHONE_NUMBER> released on /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [INFO filelock:318]
530 copy /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip-21.2.3.virtualenv to /private/tmp/testvenv/lib/python3.9/site-packages/pip-21.2.3.virtualenv [DEBUG _sync:52]
530 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip to /private/tmp/testvenv/lib/python3.9/site-packages/pip [DEBUG _sync:52]
657 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/pkg_resources to /private/tmp/testvenv/lib/python3.9/site-packages/pkg_resources [DEBUG _sync:52]
684 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/_distutils_hack to /private/tmp/testvenv/lib/python3.9/site-packages/_distutils_hack [DEBUG _sync:52]
688 generated console scripts  [DEBUG base:48]
1776 copy directory /Users/username/Library/Application Support/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip-21.2.3.dist-info to /private/tmp/testvenv/lib/python3.9/site-packages/pip-21.2.3.dist-info [DEBUG _sync:52]
1785 generated console scripts pip-3.9 pip3 pip3.9 pip [DEBUG base:48]
1786 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:64]
1789 write /private/tmp/testvenv/pyvenv.cfg [DEBUG pyenv_cfg:34]
1789    home =<EMAIL_ADDRESS>[DEBUG pyenv_cfg:38]
1789    implementation = CPython [DEBUG pyenv_cfg:38]
1789    version_info = 3.9.7.final.0 [DEBUG pyenv_cfg:38]
1789    virtualenv = 20.7.2 [DEBUG pyenv_cfg:38]
1789    include-system-site-packages = false [DEBUG pyenv_cfg:38]
1789    base-prefix =<EMAIL_ADDRESS>[DEBUG pyenv_cfg:38]
1789    base-exec-prefix =<EMAIL_ADDRESS>[DEBUG pyenv_cfg:38]
1789    base-executable =<EMAIL_ADDRESS>[DEBUG pyenv_cfg:38]
1790 created virtual environment CPython3.9.7.final.0-64 in 1677ms
  creator CPython3Posix(dest=/private/tmp/testvenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/username/Library/Application Support/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:19]

I can't seem to reproduce this issue with BigSur and Homebrew on GitHub Actions. Could you please see these logs and tell me whether anything jumps at you as different from your system configuration?
https://github.com/pybuilder/pybuilder/commit/618512b0f356e3314b1218ae7c0192b9fea49c6b/checks/3581148294/logs

I noticed one difference:
$ which python3
/usr/local/bin/python3
$ realpath $(which python3)
<EMAIL_ADDRESS>$ l<EMAIL_ADDRESS>lrwxr-xr-x  1 username  admin  26 Sep  6 09:21<EMAIL_ADDRESS>-><EMAIL_ADDRESS>
I seem to be using python3 from /usr/local/bin/python3, while in GitHub Actions you add<EMAIL_ADDRESS>to PATH.
2021-09-12T22:24:47.3742910Z echo<EMAIL_ADDRESS>>> $GITHUB_PATH

Both symlinks seem to exist for me, though.
I don't think this is relevant.
Does the GitHub Actions workflow work with pybuilder 0.12.10? I can't test with --pre because of the StopIteration issue.

I might have supplied you with the wrong logs. One sec.

Yeah sorry, wrong logs. Correct link: https://github.com/pybuilder/pybuilder/commit/618512b0f356e3314b1218ae7c0192b9fea49c6b/checks/3581148294/logs
2021-09-12T22:25:46.2604910Z To reinstall 3.9.7, run:
2021-09-12T22:25:46.2605500Z   brew reinstall<EMAIL_ADDRESS>2021-09-12T22:25:46.2734340Z ##[group]Run pwd
2021-09-12T22:25:46.2735090Z pwd
2021-09-12T22:25:46.2735500Z which python3
2021-09-12T22:25:46.2735920Z python3 --version
2021-09-12T22:25:46.2736410Z python3 -m pip --version
2021-09-12T22:25:46.2736880Z echo GITHUB_REF $GITHUB_REF
2021-09-12T22:25:46.2737570Z echo GITHUB_HEAD_REF $GITHUB_HEAD_REF
2021-09-12T22:25:46.2738150Z echo GITHUB_BASE_REF $GITHUB_BASE_REF
2021-09-12T22:25:46.2925070Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2021-09-12T22:25:46.2925760Z env:
2021-09-12T22:25:46.2926200Z   DEPLOY_BRANCHES: refs/heads/master
2021-09-12T22:25:46.2926690Z   DEPLOY_PYTHONS: 3.9
2021-09-12T22:25:46.2927090Z   DEPLOY_OSES: Linux
2021-09-12T22:25:46.2927560Z   PYB_ARGS: -E ci -v -X analyze install
2021-09-12T22:25:46.2928230Z   TWINE_USERNAME: pybuilder-travis2
2021-09-12T22:25:46.2928770Z   TWINE_PASSWORD:
2021-09-12T22:25:46.2929180Z   SKIP_JOB: false
2021-09-12T22:25:46.2929660Z ##[endgroup]
2021-09-12T22:25:46.3201590Z /Users/runner/work/pybuilder/pybuilder
2021-09-12T22:25:46.3225010Z<EMAIL_ADDRESS>2021-09-12T22:25:46.3843640Z Python 3.9.7
2021-09-12T22:25:48.2166530Z pip 21.2.4 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
2021-09-12T22:25:48.2455010Z GITHUB_REF refs/pull/786/merge
2021-09-12T22:25:48.2456760Z GITHUB_HEAD_REF pr-769
2021-09-12T22:25:48.2459170Z GITHUB_BASE_REF master

That's the thing - StopIteration issue doesn't reproduce either on GHA macos 11 + Homebrew. I wonder if it's some standard environment on a laptop that doesn't appear on macos boxes...
I can hypothetically build a 0.12 on GHA and see what's up, but 0.12 contains an old version of bundled VEnv as well, which was never tested with Python 3.9 and latest version of setuptools and wheel, so it's a whole other can of worms...

I seem to be using python3 from /usr/local/bin/python3, while in GitHub Actions you add<EMAIL_ADDRESS>to PATH.

Now this is interesting... Did you add that to PATH? Because I think I saw Homebrew not updating that on GHA, that's why python location check is there. I'll test tomorrow if it makes a difference and will look at whether Brew updates that symlink.

Can you check what this outputs on GHA?
$ python3 -c 'import sys; print(sys.path); print(sys.prefix)'
[''<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>'/Users/username/Library/Python/3.9/lib/python/site-packages', '/usr/local/lib/python3.9/site-packages']
<EMAIL_ADDRESS>

Did you add that to PATH?

I have /usr/local/bin in PATH, but it does not make difference which way python3 is found. Build still fails with AssertionError even if prefixed with PATH=/usr/local/opt/python@3.9/bin:$PATH.

That is weird... Are you on M1 by any chance? Or Intel?

https://github.com/pybuilder/pybuilder/actions/runs/1228406688

Not M1.
macOS Big Sur 11.5.2 on MacBook Pro (13-inch, 2019, Four Thunderbolt 3 ports).

This is the homebrewed one: https://github.com/pybuilder/pybuilder/runs/3583700812?check_suite_focus=true

python3 -c 'import sys; print(sys.path); print(sys.prefix); print(sys.base_prefix); print(sys.exec_prefix); print(sys.executable);'

[''<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>'/usr/local/lib/python3.9/site-packages']
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Is your PyBuilder installed in --user or system?

@hoxu?

I run pybuilder mostly with a custom build script in a virtualenv like this:
python3 -m venv vebuild
. vebuild/bin/activate
pip --disable-pip-version-check install pybuilder

But I get the same AssertionError outside virtualenv with pybuilder installed with pip3 install --user pybuilder==0.12.10.
Apologies for not having enough time to dig properly into this.
Maybe @dindurthy could take a look at the GitHub Actions workflow logs and see if there are any differences to local execution?

Same exception with  pybuilder installed withpip3 install pybuilder as well.

[''<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>'/usr/local/lib/python3.9/site-packages']
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Here is a diff from mine to GHA:
  ['',
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>-  '/Users/username/Library/Python/3.9/lib/python/site-packages',
   '/usr/local/lib/python3.9/site-packages']
<EMAIL_ADDRESS>

Rejoice! I was able to reproduce this issue on Linux :)

@arcivanov let me know when --pre with #794 is available, so that I can try to reproduce this with the StopIteration fix. Good job fixing it :)

@hoxu go ahead please with --pre testing. :)

@hoxu did you have a chance to try it out?

Working on other projects, but a quick test with --pre PyBuilder version 0.13.0.dev20210917052200 & Python 3.9.7 seems to indicate that this issue is fixed! Awesome, good job!
I think this can be closed after a release.

Glad to hear! The release is a bit away, as an issue in virtualenv has been discovered as well.


Contours and Landmarks in Face Detection

Hi @azihsoyn
I was trying out Face Detection using your package but couldnt figure out how to use Landmarks and Contours. Can you please provide an example to do the same?
Thanks

Hi, @tperm94.
I'm sorry, Face Landmarks and Contours is not supported in this library.
Please consider use official library. https://github.com/FirebaseExtended/flutterfire/blob/master/packages/firebase_ml_vision/lib/src/face_detector.dart


Register overrides other container factories with same protocol signature

Hi Michael,
thanks for this great package! I've found an issue and maybe you can lead me to the right direction.
I have a SharedContainer similar to this:
class MyModule: SharedContainer {
    static let approve = Factory<MyProtocol<Input, Output>> { wrap(Approve()) }
    static let complete = Factory<MyProtocol<Input, Output>> { wrap(Complete()) }
    static let reset = Factory<MyProtocol<Input, Output>> { wrap(Reset()) }
}

And my ViewModel is injecting the dependencies:
final class ViewModel: ObservableObject {
        @Injected(MyModule.approve) private var approve
        @Injected(MyModule.complete) private var complete
        @Injected(MyModule.reset) private var reset
        ...
}

My issue now is that Factory is overriding my mocks in tests with the latest definition for the same protocol signature. My assumption was that this handled per container property. (Like https://github.com/hmlongco/Resolver/blob/master/Documentation/Names.md)
extension MyModule {
    static func mockApprove(data: Output?) {
        approve.register { Fake<Input, Output>(data) }
    }

    static func mockComplete(data: Output?) {
        complete.register { Fake<Input, Output>(data) }
    }

   static func mockReset(data: Output?) {
        reset.register { Fake<Input, Output>(data) }
    }

The test:
class MyTest: XCTestCase {

    override class func setUp() {
        super.setUp()
        MyModule.mockComplete()
        MyModule.mockApprove() <-- MyModule.complete behaves after evaluation like approve
        MyModule.mockReset() <-- MyModule.approve and MyModule.complete behave after evaluation like reset
    }
    ...
}

Since the definitions are static it's not really the container but the Factory that should be distinct. But yes, registering a type affected any other container of the same type.
🤦 This prevented something as simple as...
extension SharedContainer {
    static let a = Factory<String> { "A" }
    static let b = Factory<String> { "B" }

At any rate, I think it's fixed in 1.0.7.

Thanks a lot @hmlongco! Works like charm with 1.0.7 👍🏼


Spell Shield Fighter Level 7 Feature not working with Cantrips?

The War Magic Feature says that you can use your bonus action to attack using the weapon in your main hand after casting a spell. I tried with Scorching ray, and it works properly. Similarly if you use shield as a reaction. However, when I use cantrips (Resonating strike, Sunlit Blade, or even Light) the Bonus action attack does not become available.  Is this intended?
Here is the save. Using the modded campaign "Champion of Ages"
https://drive.google.com/file/d/10Eds9tLd05aXjD9LzPSGyMldeU_v55x4/view?usp=share_link

I have done some checking. i think you have misunderstood how the level 7 feature works. the cantrips cant be cast in place of one fo your attacks, so you can attack then cast a cantrip or cast a cantrip then attack. the other feature lets you just cast a leveled spell then attack using a bonus action.

I have done some checking. i think you have misunderstood how the level 7 feature works. the cantrips cant be cast in place of one fo your attacks, so you can attack then cast a cantrip or cast a cantrip then attack. the other feature lets you just cast a leveled spell then attack using a bonus action.

Alright, figured I misunderstood. My apologies.


display storm by storm type

hurricane harvey was actually tropical storm harvey for a while. can we have the legend and the dot color change over time to show how the storm is changing type?

MVP: dot changes color, add a few lines to the legend for all the possible storm types

Here is an example of how Harvey changed:

I'll start with the dot, but having the line be different colors should be doable as a simple add on.

> dput(unique(storm_line$STORMTYPE))
c("Low", "Tropical Depression", "Tropical Storm", "Tropical Wave",
"Hurricane1", "Hurricane2", "Hurricane3", "Hurricane4", "Extratropical Cyclone")

["#009900", "#ffff00", "#ff9933", "#ff0000", "#ff00ff"]
4
3
2
1
Tropical *

So -- should I change the dot size AND color?!?

Nah, just know that you have both at your disposal. Do something that looks nice. And has a color scheme that's compatible with the rest of the figure...those colors up above are really bright...


Update devise.gemspec

Accented characters cause error "ArgumentError: invalid byte sequence in US-ASCII"
They definitely are the correct way of writing names with all dignity but I believe "large scale acceptance" of the solution here should have higher priority, if I may.

Thanks but this is not fixing the real problem. If you are receiving such errors, it means something else is messing up and not Devise. Try updating your rubygems or bundler version if the problem persists.

this hotfix helped me:
http://stackoverflow.com/questions/8252160/invalid-byte-sequence-error-in-normalize-yaml-input-being-thrown


wip: minimal client

We wanted to move from the legacy client to something slimmer that could be driven by the Kernel's exposed OpenAPI definitions. This PR accomplishes that for the most part. Unfortunately CoreAPI which makes up the meat of the client has some issues with our API. As such, only CREATE and READ operations are currently supported. As those are the operations in the client that are currently used in Integration Tests and the Producer, it was decided   that we could release a limited client.
As part of this work, we've also removed the integration test suite's dependency on Mocker and Saladbar. Saladbar has been removed entirely as it's not being used or actively maintained. It will likely serve as a prototype for future schema library concepts but as it exists now it is vestigial. Mocker remains in the codebase, but  as it's no longer required for integration testing is no longer automatically built / distribuited.

Features merged as part of #378


Release 1.2.0

1.2.0 - 13.02.2024

Updated dependencies
Upgraded from node 12 to 20
Replaced deprecated @zeit/ncc @ for @vercel/ncc

Looking at the code, we might want to get rid of this action, given that it's just creating a deployment status similarly we do here: https://github.com/zattoo/aurora/pull/7741/files#diff-301753639d55ffec04dc3ced92cb1a0e096be03408776b4cd3f3570a6c69fce1R1-R24
We can just adapt the composite action to take the missing environment_url and environment

@z-merge it


update to latest ldc version

This nicely locks ddmd development to only use what's working across all browsers. It also updates automatically and we more or less agreed to not pin the base compiler version.

You mean compilers?

probably


I added the file explicitly because there are systems that look for it in
order to automatically figure out if it will be allowed in house at
commercial shops. Like mine. :-)
On Thu, Aug 10, 2017 at 07:24 Chris Carpita<EMAIL_ADDRESS>wrote:

@ccarpita commented on this pull request.
In LICENSE.md
https://github.com/ccarpita/kvstore/pull/5#discussion_r132427893:

@@ -0,0 +1,19 @@
+Copyright 2017 Chris Carpita

I've labelled as MIT license recently (see README) to clarify rights for
contributors, so this file should be consistent. Thanks for adding though!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ccarpita/kvstore/pull/5#pullrequestreview-55506041,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC-_dreccqpXpAew1LIbY_ZBnuHTq9NSks5sWuh4gaJpZM4Oy1F1
.

--
Matthew O. Persico

@matthewpersico understood about the license file, which is a good idea.  I realize now that it is the MIT license in fact.  No need to change anything 👍

That solution sounds good to me
On Thu, Aug 10, 2017 at 8:17 AM Matthew O. Persico<EMAIL_ADDRESS>wrote:

@matthewpersico commented on this pull request.
In kvstore.sh
https://github.com/ccarpita/kvstore/pull/5#discussion_r132437152:

@@ -331,3 +347,9 @@ kvstore () {
}
kvstore "$@"
exit $?

+## for emacs...
+## Local Variables:
+## sh-basic-offset: 2
+## sh-indentation: 2
+## End:
Yes. I did some research and found that I could add a .dir-locals.el file
to the directory and stuff those items in there. I'll amend the commit if
that solution is ok with you.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ccarpita/kvstore/pull/5#discussion_r132437152, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABjrzQXz_qFPDT52Hl14oCGVW0yg8aacks5sWvTvgaJpZM4Oy1F1
.

Done. Emacs config moved out to .dir-locals.el.

👍


fix: regexp_replace fails when pattern or replacement is a scalar NULL

Which issue does this PR close?

Closes #11410
Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Thank you @Weijun-H  and @jonahgao


request: API that just takes a hash value

The AddWithHasher API is awkward. It takes the value just so it can call the hasher function for us.. What if I'm hashing something that's not easily represented as a string? A function that just takes the hash would be more useful.

@RaduBerinde here you go: https://godoc.org/github.com/mynameisfiber/gohll#HLL.AddHash


[Bug] NETWORKIDLE event fired eventhough the url is not loaded completely.

This works perfectly in pupeeteer . I am trying to convert the node.js code to java , so using playwright java code to automate my test case.
Have tried all below options to stop closing the browser before taking screenshot. But without fully loading the specified url it takes screenshot and closes the browser.
NavigateOptions navigateOptions = new NavigateOptions();
navigateOptions.withWaitUntil(LoadState.NETWORKIDLE);
page.waitForResponse(url);
WaitForSelectorOptions waitForSelectorOptions = new WaitForSelectorOptions();
waitForSelectorOptions.withState(State.VISIBLE);
page.waitForSelector("view", waitForSelectorOptions);
page.waitForSelector("react-grid-layout", waitForSelectorOptions);
page.waitForSelector("view", waitForSelectorOptions);
page.waitForSelector("dashboard-content", waitForSelectorOptions);
page.waitForSelector("react-grid-item", waitForSelectorOptions);
//Even tried to fulfil the response by 200

page.route(grafanaUrl, r -> {
				r.fulfill(new FulfillResponse().withStatus(200));
			});
page.navigate(grafanaUrl,navigateOptions);
			page.screenshot(new Page.ScreenshotOptions().withPath(Paths.get("screenshot-" + browserType.name() + ".png")));

Pupeeteer equivalent code that works - await page.goto(rUrl, { waitUntil: 'networkidle0' });
Any help could be appreciated.

This should work as Playwright also waits for 500ms after last finished network request. Can you share the working puppeteer snippet or the page your are testing? Can you also print the network traffic
    page.addListener(REQUEST, event -> {
      System.out.println("Request: " + ((Request) event.data()).url());
    });
    page.addListener(RESPONSE, event -> {
      System.out.println("Response: " + ((Response) event.data()).url());
    });

to ensure there are now unexpected network requests?

This should work as Playwright also waits for 500ms after last finished network request. Can you share the working puppeteer snippet or the page your are testing? Can you also print the network traffic
    page.addListener(REQUEST, event -> {
      System.out.println("Request: " + ((Request) event.data()).url());
    });
    page.addListener(RESPONSE, event -> {
      System.out.println("Response: " + ((Response) event.data()).url());
    });

to ensure there are now unexpected network requests?

I have tried printing the above snippet but it is not reliable . I am not getting the reponse sysouts sometimes but i get the request sysouts always as shown in below logs.
Version used - 0.172.3
Also i am not able to provide the link since it is my local Grafana. I am able to take screenshot of Grafana with panels only sometimes as the backend code is still executing resulting in empty panels.
Tried options - Generated code via playwright-cli but not able to scroll the dashboard.
Please find below the java code and log . Also pasted the working puppeteer code .
Kindly let me know if i have to change any code snippet.
JAVA snippet
`
public static void main(String[] args)  {
	       Playwright playwright = Playwright.create();
                 try {

                    System.out.println("Playwright---");
		String grafanaUrl = "http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1";
		BrowserType browserType = 	playwright.chromium();
		LaunchOptions launchOptions = new LaunchOptions();
		launchOptions.withHeadless(true);
		launchOptions.withDevtools(false);
		//launchOptions.withEnv("DEBUG=pw:protocol");

		String auth = "user:pwd";
		byte[] encodedAuth = Base64.getEncoder().encode(auth.getBytes(StandardCharsets.ISO_8859_1));
		String authHeader = "Basic " + new String(encodedAuth);
		Map<String, String> headers = new HashMap<>();
		headers.put("Authorization", authHeader);

		Browser browser = browserType.launch(launchOptions);
		BrowserContext context = browser.newContext(
				new Browser.NewContextOptions().withViewport(1200, 800));
		Page page = context.newPage();

		page.setExtraHTTPHeaders(headers);
		page.setDefaultNavigationTimeout(120000);
		page.setViewportSize(1200, 800);

		NavigateOptions navigateOptions = new NavigateOptions();
		navigateOptions.withWaitUntil(LoadState.NETWORKIDLE);
		page.route("**", route -> {
			//System.out.println(route.request().url());
			route.continue_();
		});

                   /** hidding some css not working **/
		page.evaluate("() => { return document.getElementsByClassName('sidemenu').hidden; }");
                 /** scroll not working **/
		page.evaluate("() => window.scrollTo(0, document.body.scrollHeight)");

		page.addListener(EventType.REQUEST, e -> System.out.println("Request--"+((Request)e.data()).url()));
		page.addListener(EventType.RESPONSE, e -> System.out.println("Response--"+((Response)e.data()).url()));

		page.route("http://localhost:3000/api/datasources/proxy/**", r -> {
			r.fulfill(new FulfillResponse().withStatus(200));
		});
		page.waitForResponse(grafanaUrl);
		page.navigate(grafanaUrl,navigateOptions);
		Page.WaitForSelectorOptions waitForSelectorOptions = new Page.WaitForSelectorOptions();
		waitForSelectorOptions.withState(State.ATTACHED);
		Deferred<ElementHandle> event = page.waitForSelector(".view", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		event.get();
		Deferred<ElementHandle> revent = page.waitForSelector(".react-grid-layout", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		revent.get();
		Deferred<ElementHandle> devent = page.waitForSelector(".dashboard-content", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		devent.get();
		Deferred<ElementHandle> ievent = page.waitForSelector(".react-grid-item", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		ievent.get();
		Deferred<ElementHandle> cevent = page.waitForSelector(".panel-content", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		cevent.get();
		Deferred<ElementHandle> sevent = page.waitForSelector(".track-horizontal", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		sevent.get();
		page.screenshot(new Page.ScreenshotOptions().withPath(Paths.get("screenshot-" + browserType.name() + ".png")));
		PdfOptions pdfOptions =  new PdfOptions();
		pdfOptions.withPath(Paths.get("test.pdf"));
		pdfOptions.withPrintBackground(true);
		//pdfOptions.withFormat("Letter");
		//pdfOptions.withLandscape(true);
		//pdfOptions.withScale(2);
		pdfOptions.withDisplayHeaderFooter(true);
		pdfOptions.setMargin().withTop("90").withRight("10").withBottom("50").withLeft("10");
		page.pdf(pdfOptions);
		page.close();
		context.close();
		browser.close();

	} catch (Exception e) {
		e.printStackTrace();
	}finally {
		try {
			playwright.close();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

`
Logs
Failed to process descriptor at C:\Users\ADMINI~1\AppData\Local\Temp\2\playwright-java-8166996152194358923
Playwright---
2021-01-13T07:02:56.449Z pw:api navigating to "http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1", waiting until "networkidle"
Request--http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1
Response--http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1
2021-01-13T07:02:56.497Z pw:api   navigated to "http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1"
2021-01-13T07:02:57.684Z pw:api   "networkidle" event fired
2021-01-13T07:02:57.688Z pw:api waiting for selector ".view"
2021-01-13T07:02:58.638Z pw:api   "domcontentloaded" event fired
2021-01-13T07:02:58.752Z pw:api   "load" event fired
Request--http://localhost:3000/api/datasources/proxy/1
Response--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Response--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/public/fonts/fontawesome-webfont.woff2?v=4.7.0
Response--http://localhost:3000/api/datasources/proxy/1
Response--http://localhost:3000/public/fonts/fontawesome-webfont.woff2?v=4.7.0
2021-01-13T07:02:59.398Z pw:api   selector resolved to visible â€¦
2021-01-13T07:02:59.539Z pw:api waiting for selector ".react-grid-layout"
Request--http://localhost:3000/api/datasources/proxy/1
2021-01-13T07:02:59.685Z pw:api   selector resolved to visible â€¦
2021-01-13T07:02:59.706Z pw:api waiting for selector ".dashboard-content"
2021-01-13T07:02:59.829Z pw:api   selector resolved to visible â€¦
2021-01-13T07:02:59.884Z pw:api waiting for selector ".react-grid-item"
2021-01-13T07:02:59.896Z pw:api   selector resolved to visible â€¦
didn't receive response for the below highlighted urls
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
2021-01-13T07:02:59.924Z pw:api waiting for selector ".panel-content"
2021-01-13T07:02:59.929Z pw:api   selector resolved to visible â€¦
2021-01-13T07:02:59.966Z pw:api waiting for selector ".track-horizontal"
2021-01-13T07:02:59.972Z pw:api   selector resolved to hidden â€¦
Working puppeteer code
`
(async () => {
try {
const browser = await puppeteer.launch({ headless: true, devtools: true, ignoreHTTPSErrors: false });
const page = await browser.newPage();
  await page.setExtraHTTPHeaders({ 'Authorization': auth_header });

  await page.setDefaultNavigationTimeout(120000);

  await page.setViewport({
    width: width_px,
    height: 800,
    deviceScaleFactor: 2,
    isMobile: false
  })
  await page.goto(rUrl, { waitUntil: 'networkidle0' });

  await page.evaluate(() => {
    let sidemenu = document.getElementsByClassName('sidemenu');
    for (el of sidemenu) { el.hidden = true; };
  });

  var height_px = await page.evaluate(() => {
    return document.getElementsByClassName('react-grid-layout')[0].getBoundingClientRect().bottom;
  }) + 160;

  async function autoScroll(page) {
    await page.evaluate(async () => {
      await new Promise((resolve, reject) => {
        var totalHeight = 0;
        var distance = 100;
        var height_px = document.getElementsByClassName('react-grid-layout')[0].getBoundingClientRect().bottom;
        var timer = setInterval(() => {
          var scrollHeight = height_px;

          var element = document.querySelector('.view');
          element.scrollBy({
            top: distance,
            left: 0,
            behavior: 'smooth'
          });

          totalHeight += distance;

          console.log('totalHeight', totalHeight)

          if (totalHeight >= scrollHeight) {
            clearInterval(timer);
            resolve();
          }
        }, 300);
      });
    });
  }

  await autoScroll(page);

  const pdf = await page.pdf({
    //path: outfile,
    width: width_px + 'px',
    height: height_px + 'px',
    //format: 'A2',
    scale: 1,
    displayHeaderFooter: true,
    footerTemplate: footer,
    headerTemplate: header,
    printBackground: true,
    margin: {
      top: 90,
      right: 0,
      bottom: 50,
      left: 0,
    },
  });

  await browser.close();
  res.set('Access-Control-Allow-Origin', '*');
  res.contentType("application/pdf");
  res.send(pdf);
} catch (error) {
  console.log(error);
}

})()`

I have tried printing the above snippet but it is not reliable . I am not getting the reponse sysouts sometimes but i get the request sysouts always as shown in below logs.
Version used - 0.172.3
Also i am not able to provide the link since it is my local Grafana. I am able to take screenshot of Grafana with panels only sometimes as the backend code is still executing resulting in empty panels.
Tried options - Generated code via playwright-cli but not able to scroll the dashboard.
Please find below the java code and log . Also pasted the working puppeteer code .
Kindly let me know if i have to change any code snippet.
JAVA snippet
`
public static void main(String[] args)  {
	       Playwright playwright = Playwright.create();
                 try {

                    System.out.println("Playwright---");
		String grafanaUrl = "http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1";
		BrowserType browserType = 	playwright.chromium();
		LaunchOptions launchOptions = new LaunchOptions();
		launchOptions.withHeadless(true);
		launchOptions.withDevtools(false);
		//launchOptions.withEnv("DEBUG=pw:protocol");

		String auth = "user:pwd";
		byte[] encodedAuth = Base64.getEncoder().encode(auth.getBytes(StandardCharsets.ISO_8859_1));
		String authHeader = "Basic " + new String(encodedAuth);
		Map<String, String> headers = new HashMap<>();
		headers.put("Authorization", authHeader);

		Browser browser = browserType.launch(launchOptions);
		BrowserContext context = browser.newContext(
				new Browser.NewContextOptions().withViewport(1200, 800));
		Page page = context.newPage();

		page.setExtraHTTPHeaders(headers);
		page.setDefaultNavigationTimeout(120000);
		page.setViewportSize(1200, 800);

		NavigateOptions navigateOptions = new NavigateOptions();
		navigateOptions.withWaitUntil(LoadState.NETWORKIDLE);
		page.route("**", route -> {
			//System.out.println(route.request().url());
			route.continue_();
		});

                   /** hidding some css not working **/
		page.evaluate("() => { return document.getElementsByClassName('sidemenu').hidden; }");
                 /** scroll not working **/
		page.evaluate("() => window.scrollTo(0, document.body.scrollHeight)");

		page.addListener(EventType.REQUEST, e -> System.out.println("Request--"+((Request)e.data()).url()));
		page.addListener(EventType.RESPONSE, e -> System.out.println("Response--"+((Response)e.data()).url()));

		page.route("http://localhost:3000/api/datasources/proxy/**", r -> {
			r.fulfill(new FulfillResponse().withStatus(200));
		});
		page.waitForResponse(grafanaUrl);
		page.navigate(grafanaUrl,navigateOptions);
		Page.WaitForSelectorOptions waitForSelectorOptions = new Page.WaitForSelectorOptions();
		waitForSelectorOptions.withState(State.ATTACHED);
		Deferred<ElementHandle> event = page.waitForSelector(".view", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		event.get();
		Deferred<ElementHandle> revent = page.waitForSelector(".react-grid-layout", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		revent.get();
		Deferred<ElementHandle> devent = page.waitForSelector(".dashboard-content", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		devent.get();
		Deferred<ElementHandle> ievent = page.waitForSelector(".react-grid-item", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		ievent.get();
		Deferred<ElementHandle> cevent = page.waitForSelector(".panel-content", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		cevent.get();
		Deferred<ElementHandle> sevent = page.waitForSelector(".track-horizontal", new Page.WaitForSelectorOptions()
				.withState(Page.WaitForSelectorOptions.State.ATTACHED));
		sevent.get();
		page.screenshot(new Page.ScreenshotOptions().withPath(Paths.get("screenshot-" + browserType.name() + ".png")));
		PdfOptions pdfOptions =  new PdfOptions();
		pdfOptions.withPath(Paths.get("test.pdf"));
		pdfOptions.withPrintBackground(true);
		//pdfOptions.withFormat("Letter");
		//pdfOptions.withLandscape(true);
		//pdfOptions.withScale(2);
		pdfOptions.withDisplayHeaderFooter(true);
		pdfOptions.setMargin().withTop("90").withRight("10").withBottom("50").withLeft("10");
		page.pdf(pdfOptions);
		page.close();
		context.close();
		browser.close();

	} catch (Exception e) {
		e.printStackTrace();
	}finally {
		try {
			playwright.close();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

`
Logs
Failed to process descriptor at C:\Users\ADMINI~1\AppData\Local\Temp\2\playwright-java-8166996152194358923
Playwright---
2021-01-13T07:02:56.449Z pw:api navigating to "http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1", waiting until "networkidle"
Request--http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1
Response--http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1
2021-01-13T07:02:56.497Z pw:api   navigated to "http://localhost:3000/d/M-djRN2ik/4-test-dashboard?orgId=1"
2021-01-13T07:02:57.684Z pw:api   "networkidle" event fired
2021-01-13T07:02:57.688Z pw:api waiting for selector ".view"
2021-01-13T07:02:58.638Z pw:api   "domcontentloaded" event fired
2021-01-13T07:02:58.752Z pw:api   "load" event fired
Request--http://localhost:3000/api/datasources/proxy/1
Response--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Response--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/public/fonts/fontawesome-webfont.woff2?v=4.7.0
Response--http://localhost:3000/api/datasources/proxy/1
Response--http://localhost:3000/public/fonts/fontawesome-webfont.woff2?v=4.7.0
2021-01-13T07:02:59.398Z pw:api   selector resolved to visible â€¦
2021-01-13T07:02:59.539Z pw:api waiting for selector ".react-grid-layout"
Request--http://localhost:3000/api/datasources/proxy/1
2021-01-13T07:02:59.685Z pw:api   selector resolved to visible â€¦
2021-01-13T07:02:59.706Z pw:api waiting for selector ".dashboard-content"
2021-01-13T07:02:59.829Z pw:api   selector resolved to visible â€¦
2021-01-13T07:02:59.884Z pw:api waiting for selector ".react-grid-item"
2021-01-13T07:02:59.896Z pw:api   selector resolved to visible â€¦
didn't receive response for the below highlighted urls
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
Request--http://localhost:3000/api/datasources/proxy/1
2021-01-13T07:02:59.924Z pw:api waiting for selector ".panel-content"
2021-01-13T07:02:59.929Z pw:api   selector resolved to visible â€¦
2021-01-13T07:02:59.966Z pw:api waiting for selector ".track-horizontal"
2021-01-13T07:02:59.972Z pw:api   selector resolved to hidden â€¦
Working puppeteer code
`
(async () => {
try {
const browser = await puppeteer.launch({ headless: true, devtools: true, ignoreHTTPSErrors: false });
const page = await browser.newPage();
  await page.setExtraHTTPHeaders({ 'Authorization': auth_header });

  await page.setDefaultNavigationTimeout(120000);

  await page.setViewport({
    width: width_px,
    height: 800,
    deviceScaleFactor: 2,
    isMobile: false
  })
  await page.goto(rUrl, { waitUntil: 'networkidle0' });

  await page.evaluate(() => {
    let sidemenu = document.getElementsByClassName('sidemenu');
    for (el of sidemenu) { el.hidden = true; };
  });

  var height_px = await page.evaluate(() => {
    return document.getElementsByClassName('react-grid-layout')[0].getBoundingClientRect().bottom;
  }) + 160;

  async function autoScroll(page) {
    await page.evaluate(async () => {
      await new Promise((resolve, reject) => {
        var totalHeight = 0;
        var distance = 100;
        var height_px = document.getElementsByClassName('react-grid-layout')[0].getBoundingClientRect().bottom;
        var timer = setInterval(() => {
          var scrollHeight = height_px;

          var element = document.querySelector('.view');
          element.scrollBy({
            top: distance,
            left: 0,
            behavior: 'smooth'
          });

          totalHeight += distance;

          console.log('totalHeight', totalHeight)

          if (totalHeight >= scrollHeight) {
            clearInterval(timer);
            resolve();
          }
        }, 300);
      });
    });
  }

  await autoScroll(page);

  const pdf = await page.pdf({
    //path: outfile,
    width: width_px + 'px',
    height: height_px + 'px',
    //format: 'A2',
    scale: 1,
    displayHeaderFooter: true,
    footerTemplate: footer,
    headerTemplate: header,
    printBackground: true,
    margin: {
      top: 90,
      right: 0,
      bottom: 50,
      left: 0,
    },
  });

  await browser.close();
  res.set('Access-Control-Allow-Origin', '*');
  res.contentType("application/pdf");
  res.send(pdf);
} catch (error) {
  console.log(error);
}

})()`

I have tried printing the above snippet but it is not reliable . I am not getting the reponse sysouts sometimes but i get the request sysouts always as shown in below logs.

There is a chance that some of the requests fail and NETWORKIDLE starts counting from that point. It's not quite clear to me where
_didn't receive response for the below highlighted urls_

comes from. Maybe there is REQUESTFAILED event instead of RESPONSE for that request?

I have tried this code with node as well but fullscreenshot is getting trimmed and takes only the visible part.

There is a bunch of code unrelated to networkidle event. Why do you think the problem is in the networkidle event? E.g. the js snippet contains time consuming autoScroll function that is not present in the java counterpart. It'd be nice to reduce the example to just the network part as I suspect the problem might be elsewhere.

I have tried printing the above snippet but it is not reliable . I am not getting the reponse sysouts sometimes but i get the request sysouts always as shown in below logs.

There is a chance that some of the requests fail and NETWORKIDLE starts counting from that point. It's not quite clear to me where
_didn't receive response for the below highlighted urls_

comes from. Maybe there is REQUESTFAILED event instead of RESPONSE for that request?

I have tried this code with node as well but fullscreenshot is getting trimmed and takes only the visible part.

There is a bunch of code unrelated to networkidle event. Why do you think the problem is in the networkidle event? E.g. the js snippet contains time consuming autoScroll function that is not present in the java counterpart. It'd be nice to reduce the example to just the network part as I suspect the problem might be elsewhere.

Closing as there was no feedback.

@yury-s Sorry for the delay. I was doing a lot of tries .Later tried same code(above  in pupeeteer) using Playwright in node.js . In node.js it works fine since it has async/await that makes it to wait until the entire website loads. But in Java it does not wait for NETWORK IDLE  nor a url pattern to fulfill the 200 status code. Ps this works perfectly in node pupeeteer and node playwright.
Am I missing something or it is because of advantage due to async/await in typescript ?


ssh completion: skip only aliases matching *

This commit fixes a bug where lines that declare multiple hostnames get
omitted from completion entirely if one of the hostnames matches *. For
example:
Host foo.com bar.dev baz.*

This is what I was trying to "fix" in #1784.

LGTM, thanks.


:bug: ActivityFlags no longer crash

I got so annoyed that I finally fixed it
ActivityFlags are sent as a number with multiple flags in it. Pretty sure I did the bitwise to get the bool right.
Check off the following

[x] I have tested my changes with the current requirements
[x] My Code follows the pep8 code style.

Maybe i should remove the dataclass :skull:
i think the usage was unpythonic anyway

Maybe i should remove the dataclass :skull:
i think the usage was unpythonic anyway

dataclass is gone


Add a "Structuring Reducers" recipe

We really, really need a page talking about approaches, guidelines, and suggestions for organizing reducer logic.  I've been saying I want to write this for a while, but have been too busy so far.  I'm still up for it, but further suggestions and/or offers of help would be appreciated.
WIP page: https://github.com/markerikson/redux/blob/structuring-reducers-page/docs/recipes/StructuringReducers.md
Initial sketch of possible topics:

[x] Understanding that you really only have one reducer function, which is just subdivided for maintainability and simplicity

[x] Understanding "reducer composition" and that you can break things down using plain ol' functions

[x] The basic rules of reducers: (state, action) -> newState, immutable updates, and one other item I've said before but I'm not remembering at the moment

[x] Emphasis that combineReducers is simply a utility function for the common use case of delegating update logic based on state domain organization, and not a requirement

[x] That "actions are passed to all reducers" only if you're using combineReducers

[x] Passing different chunks or all of the state to a sub-reducer based on need for that action

[x] That defining object keys when using createReducers is, effectively, defining the name/shape of your state (not always clear when using ES6 object literal shorthand - naming of imported reducer functions matters!)

[x] Normalizing data

[ ] Updating normalized data

[x] Updating data immutably in general, particularly nested data and arrays, and how just using a variable assignment doesn't mean you've "made a copy"

[x] Reusing logic and creating reducers that can be "targeted"

A whole bunch of other stuff that has crossed my mind at various points and that I'm not thinking of right now, but hopefully will remember later

Random related links:

https://github.com/reactjs/redux/issues/749
https://github.com/reactjs/redux/issues/857
https://github.com/reactjs/redux/issues/1014
https://github.com/reactjs/redux/issues/1018
https://github.com/reactjs/redux/issues/1171
https://github.com/reactjs/redux/issues/1617
https://github.com/reactjs/redux/issues/1628
https://github.com/reactjs/redux/issues/1653
https://github.com/reactjs/redux/issues/1726
https://github.com/reactjs/redux/issues/1602#issuecomment-208987442
https://gist.github.com/deanius/f380994d117dffb5b625a7871a4fa893
http://bbarr.github.io/code/2015/12/23/redux-with-pattern-matching.html
https://www.reddit.com/r/javascript/comments/42ey9e/redux_reducer_composition_without_slicing_state/
http://www.code-experience.com/problems-with-flux/
https://twitter.com/dan_abramov/status/745248569599397888
http://stackoverflow.com/questions/38012852/users-sees-one-part-of-deeply-nested-state-should-visible-properties-be-at-top/38017227#38017227
http://stackoverflow.com/questions/33749759/read-stores-initial-state-in-redux-reducer/33791942

Some overlap with the performance/optimization discussions in https://github.com/reactjs/redux/issues/1783 , the store organization mention in http://redux.js.org/docs/FAQ.html#organizing-state-nested-data , and the concepts in http://redux.js.org/docs/recipes/ComputingDerivedData.html .

This may help a bit https://github.com/nosovsh/reduceless

@BananaBobby : thanks for the link, but I'm looking to document and clarify idiomatic Redux usage.

Related http://stackoverflow.com/questions/37647178/is-there-an-idiomatic-way-to-test-nested-state-branches/37647357#37647357

Reactiflux quote:

[8:33 PM] Francois Ward: where state lives is more a matter of how you plan on manipulating it than anything else.
[8:34 PM] Francois Ward: so if you have, let say a "loading: true/false" flag that -always- updates when you set some entity, it probably should live there.
[8:34 PM] Francois Ward: if its completely independant, changes separately, etc, then it should be its own thing.
[8:34 PM] Francois Ward: as you develop apps you often will refactor that stuff.
[8:35 PM] Francois Ward: since the relationship between actions, components and reducers is completely loose and maleable, it can change all the time.

I could use some advice around using combineReducers when some slices of the state are static and don't need to be reduced. Here is a contrived example:
// This is what I'd like the state to look like:
{
  isUserSignedIn: true, // static, hydrated data - doesn't need a reducer
  todoTemplates: [...], // same
  todos: [...] // hydrated then manipulated - needs a reducer
}

// This is one way of making it work:
const reducer = combineReducers({
  isUserSignedIn: (state = false) => state, // basically a noop reducer
  todoTemplates: (state = []) => state, // same
  todos: todosReducer
});

The solution above is not very practical and scalable when the number of static slices of the state increases. I can think of a couple of other ways to make this work, but neither really stands out as the right solution:

don't use combineReducer => not scalable
move all the static data under a common key, like static, and use a noop reducer on that => scalable, but static is kind of ugly and doesn't describe what the data actually is (and there may not be a name that does, since all the static sub-keys may not have anything in common).

Not really intended as a Q&A thread, but that's definitely an interesting question I haven't considered before.
My initial thought was that a static value like that shouldn't really be in Redux state, but given that it's something server-hydrated, I can see a point to it.  I've actually got some static-ish data coming back in my host page (user full name, etc), and at the moment a couple of my components are just referencing window.theVariableFromTheServer, but that's an intriguing idea now that I think about it.
I would think that putting it under a staticData key with a noop reducer would be reasonable.

Just write your own root reducer which keeps all keys of state, and then overrides them with whatever your combined reducers returns.  In other words, static keys are kept, and dynamic keys can still use combineReducers

combinedReducers = combineReducers({
  todos: todosReducer
 })

rootReducer = function(state, action) {
  return Object.assign({}, state, combinedReducers(state, action));
}

Also see this issue: https://github.com/reactjs/redux/issues/1457
In particular, Dan's comment here: https://github.com/reactjs/redux/issues/1457#issuecomment-191347804

Wow, thanks! This definitely should go in the docs. I'm going with your second solution @naw, as dealing with static data the same way as dynamic data will lead to the cleanest code in my situation. This custom root reducer is so straightforward, logical and convenient, that it comes a little as a surprise to me that it's not already a built-in feature of combineReducers (perhaps an opt-in feature). Anyway, I'm sure there are reasons why it's not, and this is definitely not the thread for it... So thanks for your help!

Two thoughts:

putting "static" data in a Redux store is a relatively less common use case.  After all, Dan's original article on "The Case for Flux" points out that a Flux-like approach is mostly useful for data that changes over time.
Per my comments up at the top of this thread: reducers are just functions, and combineReducers is just a utility function for the most common use case.  Just like any other part of your application code, you can break things down into functions any way it makes sense to you, just that with reducers as a whole have to obey the basic rules of (state, action) -> newState and immutable updates.  Certainly doesn't mean that every reducer function must have the exact (state, action) signature, though, just that the final result needs to be put together that way.

Glad that you got things working!

@markerikson Regarding the original intent of this issue you created --- I definitely agree about structuring reducers would be helpful.
I also strongly agree with your statement:

Emphasis that combineReducers is simply a utility function for the common use case of delegating update logic based on state domain organization, and not a requirement

Personally I think one thing that might help is standardizing terminology regarding things like "root reducer", "sub-reducers", "reducers", "helper functions", "reducer factories", "state", "slice", etc.  In some cases we use the same terms to refer to different things, and other case we use different terms to refer to the same thing.
For example, what should we call the functions that manage a particular slice of state?  Are they "reducers", "sub-reducers", both, neither? Is a function considered a reducer merely because it "helps" the rootReducer manage state in some way, or is it only  a reducer if it has a certain method signature?
Moving on to combineReducers in general:
I think you could make the argument that combineReducers isn't really "core" redux. It's a high-level organization pattern that plugs in to the main redux engine (single store with dispatcher and subscriptions).  There are other patterns that arguably work just as well, and as you've pointed out, all that matters is that the rootReducer itself exposes the correct method signature.
We do need to help people understand that combineReducers is just a pattern, and perhaps wrap some standard terminology around this pattern as well as other patterns, so that we have a language for the ecosystem as a whole.  For example, if a 3rd party provides a "sub-reducer", it needs a way to communicate how it expects to be embedded into the rootReducer. Is it going to be embedded via combineReducers within a given key/slice, or is it going to be embedded at the top-level, or is it something else?
It's easy for 3rd party tools just to "assume" everyone is using combineReducers, even though combineReducers is not required.  Sometimes it's feasible for a 3rd party tool to accommodate more than one pattern at once. For example, https://github.com/reactjs/react-router-redux can accommodate combineReducers and https://github.com/acdlite/reduce-reducers: https://github.com/reactjs/react-router-redux/pull/183
Maybe it would be helpful to highlight some of these other patterns alongside combineReducers to make it more obvious that combineReducers is not required?
Just trying to brainstorm with you...

@naw : excellent comments, and definitely some of the stuff that's going through my head.  Not sure I have specific or definitive answers at the moment, but please keep tossing out those kinds of ideas for discussion.

Okay.  I am finally, finally sitting down to start work on this.  To be honest, there's so many related topics involved here that I'm not exactly sure how to address things.
Initial empty doc page pushed to https://github.com/markerikson/redux/blob/structuring-reducers-page/docs/recipes/StructuringReducers.md .  I'll try to update it as I go.  Suggestions and feedback wanted!

I was able to get started on this and crank out a decent first chunk of work.  Unfortunately, it basically duplicates a large portion of what's already in the "Reducers" docs page, and Dan's videos.
There's a couple reasons for that.  First, I feel like I need to be covering things from first principles.  Stuff like what "mutations" and "immutability" are, how to update data immutably, demoing how to refactor a reducer into smaller functions, etc.
I'm probably going to take a suggestion from @naw and prefix this with a page saying "Go read articles X, Y, and Z first, and make sure you totally understand them."  (Loosely, "you must be THIS tall to ride".)  For immutability, http://reactkungfu.com/2015/08/pros-and-cons-of-using-immutability-with-react-js/ is fantastic, http://wecodetheweb.com/2016/02/12/immutable-javascript-using-es6-and-beyond/ and http://t4d.io/javascript-and-immutability/ are pretty good.
Anyway, I'm also figuring on breaking this up into subpages.  Rough sketch:

Intro concept and reducer rules

Prerequisite articles
Basic reducer structure and state shape
Splitting up reducers (terms and concepts)
Splitting up reducers (progressive refactoring example)
Using combineReducers
Using something OTHER than combineReducers
Normalizing state shape
Writing reducers for normalized data

And, uh, whatever else I come up with.

One other side thought that I'm jotting down here for reference, but won't create an issue for yet:  might be useful to have an "Idiomatic Redux Architecture" page or something, that talks about stuff like using mapDispatch and binding to keep your components "unaware" of Redux, etc.

Thank you for your effort to educate the community. It's really great to have access to so much material while trying to learn new stuff. I have read through most of the articles/posts that you have written and linked to.
I'm completely new to Redux and I'd really like to read some best practice on what to store in the store. Maybe such information would fit in your new document? I might be way off here or come out as a complete fool, but I'll try to explain what I'm having a hard time to grasp by describing the flow in a new project I'm working on.

Serverside is Node.js and Postgres.
REST-API for now, maybe websockets later.
App must be usable without network, i.e. syncronization of data needs to be done both ways (new data in app goes to server, new data on server goes to app when network gets available again)
Local storage in app not decided upon yet (SQLite, AsyncStore, Realm, or whatever)
Local storage in app needs to be encrypted
Locally cached data might get withdrawn upon synchronization if the users access rights are changed for some objects on the server.
Some data will be described with JSON Schema for dynamic form layout. The schemas will be fetched from the server as well. Would these schemas go into the store as well? They are completely static once they are fetched.

What data goes where? I think I'd like the local database to be the truth. After some usage time this database will hold most of the relevant data fetched from the server as an offline cache. After a while I will throw out old data though.
I assume that Redux will serve as kind of a secondary cache? I.e. it won't hold data that is not loaded/used in this app session, would it? Or should Redux hold all data? If that is the case the local database would be an exact copy of the Redux store and the Redux store would potentially be quite big.
I have a lot of image handling in the app. Images will be referenced from the objects in the Redux store by UUID (also a key in Postgres DB). Local disk on the device serves as an image cache with UUIDs as filenames. I shouldn't cache binary data in the Redux store should I? Or is there a balance somewhere? Maybe Base64 encoded thumbnails can live in the store? But not 5 Mb jpg files, for sure..
In what order should I fetch data?
A. Container needs props. Props fetched from local DB if available, otherwise from REST API. After fetching, the data gets duplicated in local db and Redux store.
or B. Container needs props. Props fetched from local DAO which handles local DB as cache
And what about writing local changes to local database for later synchronization with server? Should I have a DAO which subscribes to store changes and saves changes to database. And at the same time have a synchronizer subscribing to changes for sending to the server if network is available?
I'm having a hard time to envision how Redux fits in with a local database. Hopefully I'm overthinking stuff and instead there is a really simple solution to all this =)

And some thoughts about immutability (a concept quite new to me as well)..
If I understand correctly I could make a shallow copy of the state in the reducer in order to save on the time it would take to make a deep copy? This would be done only to get a new object reference and fool the data consumers that I have provided a shiny new object. But in fact I have the same old referenced data deeper down in the object. Is this really immutability? I'm perfectly fine with whatever, but I would really like to read some more on this. How much of a state-slice should be immutable for the state to be considered immutable? I might have misunderstood something here through...
Btw, thank you very much for your answer at SO about store structure.

After some sleep I now see that I might have hijacked your thread with my own questions. Please take the above questions as input and suggestion on what to include in your StructuringReducers.

Yeah, bit of a digression there.  You might want to drop into the Reactiflux chat channels and ask some of the questions there.
For immutability, the key is that you never modify the contents of an existing object reference.  If you have an = sign for an assignment, and the thing on the left side of the = isn't already a copy, you're probably directly mutating data.  So yes, that means making a shallow copy, and overwriting some of the fields in that copied object.  See the list of articles I have linked for this topic at https://github.com/markerikson/react-redux-links/blob/master/immutable-data.md.  You probably also want to read through the Redux FAQ, which includes topics like deep-cloning vs shallow-cloning: http://redux.js.org/docs/FAQ.html#performance-clone-state

While I don't want to go over the entire concept of updating data immutably, it would probably be useful to add a page that demonstrates various recipes and approaches for specific tasks.  For example, updating a nested object field using Object.assign vs object spreads vs a couple immutable update utilities vs a Ramda lens.

Another related topic: "thin" reducers vs "thick" reducers, per http://redux.js.org/docs/FAQ.html#structure-business-logic .  I very much tend towards 'thick" reducers myself, but I've seen a number of utilities and libs that treat part or all of a Redux store as a simple key/value storage, with reducer logic along the lines of return {...state, ...action.payload}.  I know @phoenixmatrix seems to prefer that sort of approach.  That obviously changes reducer logic structure considerably.

For anyone actually paying attention to this: I'm on a work trip atm, got nothing on my schedule for the weekend, and intend to spend basically the entire weekend trying to crank out more of these docs.  We'll see how much progress I make.

Immutable data update tools:

utilities like dot-prop-immutable and React Update Addons
common utilities like Lodash (in "FP" mode) and Ramda
Redux-ORM

Was able to crank out first drafts for "Using combineReducers" and "Beyond combineReducers" yesterday.  Looks like the next couple topics on my list are dealing with normalized data.

I should also include something about the approach Dan uses in his first video series, where he defines a reducer function for a single todo, and then reuses it in a couple different contexts to do updates ( https://github.com/tayiorbeii/egghead.io_redux_course_notes/blob/master/08-Reducer_Composition_with_Arrays.md )

Normalized data reading:

https://github.com/paularmstrong/normalizr/pull/80#issuecomment-206981676
https://gist.github.com/amannn/89b8f4e1c52541d323c574fcb5a80b83
http://www.robinwieruch.de/the-soundcloud-client-in-react-redux-normalizr/
https://medium.com/@mcowpercoles/using-normalizr-js-in-a-redux-store-96ab33991369
https://github.com/reactjs/redux/issues/1255
https://github.com/reactjs/redux/issues/1824

@markerikson this thread is really informative, thanks

@mmazzarolo : Thanks.  If you have any specific feedback about the current WIP versions of the doc pages, I'd definitely like to hear it.

Thanks @markerikson for this awesome overview of structuring reducers! Looking forward to reading https://github.com/markerikson/redux/blob/structuring-reducers-page/docs/recipes/reducers/07-UpdatingNormalizedData.md

@davincho : Thanks.  If you've got any other comments or suggestions, please let me know.
Anything specific relating to normalized data that you'd like to have covered?

@markerikson Maybe food for the "Updating normalized data" chapter:
http://stackoverflow.com/questions/39243075/redux-structure-for-image-capturing
I would be thrilled to hear if there is a best practice for this kind of "ownership" updates. I.e. which entity should be responsible for updating the relationship.

In a "slice reducer", is it better to name the state arg state or something closer to what it is? In other words:

function todosReducers(state, action)
function todosReducer(todos, action)
function todosReducer(todosState, action)

Since it's just naming a local variable, totally up to you.  I can see arguments both ways.  Calling it state would be sticking with a consistent convention.  Giving it a specific name could be easier to read for intent and context.

@michaelswe : Read your SO question, and I'm not sure I understand the context enough.  Could you sketch out your current store structure for me?
Also, I'm not sure I understand the phrase "which entity should be responsible for updating the relationship".  Entities are really just plain data - it's the reducer logic that's responsible for doing the updates.

@markerikson some input from me:

Best practice of sharing update logic of entities (Updating same type of entity but in different action creators)
Updating nested entities (normalized with normalizr). Probably a generic solution is possible as relationships between entities are known via Schema definition.

@markerikson Thank you for taking the time for my SO question. "I should have written which reducer should be responsible.."
It almost boils down to a master/detail scenario, but it's a 1:1 relation so I need to update the foreign key of the master after creating the detail. A view is showing the master record and I need to create a detail record and link it to the master.
In the case when images are the "details" some master entity types needs to reference the image in different ways. Some examples:
project.coverImage
profile.avatar
profile.avatarThumbnail
controlpoint.imageBefore
controlpoint.imageAfter
So, after the image is saved to disk I need to instruct a reducer to populate the master record in the correct way. This can be done in many different ways, but I thought that there might exist a best practice for this.

The "master view" could dispatch an action which pushes the cameraView onto the navigation stack and populates the props with both its own id (master id) and the id of the future detail record. This could be done in a thunk which also dispatches the needed information to update the master record with the detail ids. I would have to handle the case where the user closes the cameraView without saving an image and reset the optimistically set detail record in the master.

The master view could dispatch an action which pushes the cameraView and includes all the needed instructions for a reducer to update the master entity after the fact that an image is saved.

I'm hoping that I managed to explain myself better this time.

Definitely need to address initializing state.  Probably best if I just copy Dan's answer in http://stackoverflow.com/questions/33749759/read-stores-initial-state-in-redux-reducer/33791942 .

"Multiple instanced data" issue/discussion roundup: https://github.com/reactjs/redux/issues/1602#issuecomment-208987442

Yay.  I have FINALLY managed to crank out "Managing Normalized Data".  I also went ahead and copied Dan's answer on "Initializing State" from Stack Overflow into a separate subpage and updated the formatting.
With that, I think I have completed all the major content that I wanted to cover for this effort.  The next step is to get some actual eyes on this, and look at usefulness of content, organization, etc.
I have not yet tried to actually build the Gitbook stuff on this branch.  I figured I'd wait until I was done content-wise to give that a shot.
@reactjs/redux  , @jimbolla , @Aweary , @naw , @phoenixmatrix , @tommikaikkonen , et al: I would greatly appreciate any and all feedback on the stuff I've put together here.  I'm definitely figuring that the order of the topics will need to be shuffled a bit.  Also, the current link to "Prerequisite Concepts" on the TOC page probably needs to be reformatted better.  Beyond that... thoughts?  Comments?  Suggestions?

Can you make a PR? That will let us comment on stuff line-by-line, rather than on the document as a whole.

@timdorr : Done.  See #1930 .

Okay, so we've got the primary content merged in.  The TOC file needs to be updated to include the new pages, and I'm wanting to do some renaming as well.
As a side note, the docs seem to have some odd numbering going on.  They should be numbered "1, 2, 3", but instead it's "1.1, 1.2, 1.3", etc.  Looking at some other Gitbook examples and can't see a real difference in the TOC definition.

This is merged! Yay!

W00t!


small fix to m_mpi_proxy.fpp

Adding 'cv' and 'qvp' variables to m_mpi_proxy.fpp module such that parallel pre-processing runs correctly whenever those variables are called (e.g. phase change module).

run ./mfc.sh format


Adding disconnected socket detection

This is needed for idle connection resiliency functionality.
Statement execute -> Check for disconnected socket -> Idle session restore/reconnect -> Continue execute

this has been merged to #1656


basic synchronous Account from token and url

Description
Adds a new component that creates a local Account from a Token and Url
Type of change

New feature (non-breaking change which adds functionality).

How has this been tested?

Manual Tests (please write what did you do?)
Generated a token from my account, input this plus the address of the server. Dropped the ManualAccount component and setup inputs. Waited for 1second (canvas blocked) got an id and a remark with my new account.

Docs

Will be updated ASAP

Welcome back from holidays @psarras! I'm curious about this PR, as it's opening up a pretty big and bad security vulnerability - if that gh definition gets saved it's like... storing your password in plaintext, on a post-it note, potentially on a network drive.
What are the use-cases that need it?

Hey @didimitrie, valid concern, not sure how to overcome it. Open to alternatives as always.
The goal is to have a synchronous way to inject the Account in Rhino Compute. This should be able to serve different users, so we cannot have something on the Rhino Compute servers baked in. The way it is setup right now it would play nice with they way Rhino Compute works. Maybe we can have a very special input that gets erased when you save the gh file? Not sure if it is possible.
BTW I will update the description.

Hi Stam, sorry for the really late reply. What's the architecture planned around the Rhino Compute part? If my assumptions are correct, the end user will need to somehow upload the grasshopper definition to a rhino compute application. This is the magic moment where you can associate credentials with the gh definition safely; the application can be a registered third party app for a given speckle server, and thus hold a user's api token that it can pass on to the compute workflow.
Is this something feasible? I'll discuss this with @cristi8 tomorrow, but I'm really hesitant to merge something that allows for the equivalent of plain text token storage inside potentially portable grasshopper files by everyone...

Hey Dimitrie. Yeah something like that. However, the user will not be the one to upload the GH definition necessarily. The idea is you might have generic definitions that do X, Y, Z and that do not include any credentials, only placeholders (empty strings). The a front end app will fill those in for you after you have "logged" in with your Speckle account.
I guess the awkwardness is that RhinoCompute can edit existing params from the gh canvas (AFAIK).
Another option for this piece of work is to put it in a different library and only include it in our workflows, while also taking the risk! Let me know how the discussion with @cristi8 goes and if you prefer this separate and we can manage it! Cheers!

Would it be possible to, somehow, hardcode in the node a variable name that is then set by your frontend? Something like an environment variable, but maybe specific to Rhino Compute?

Hey @teocomi !I will check with the Rhino Compute team and get back to you. Thanks for the feedback guys

@didimitrie - As you've closed this issue, where would you like to continue this discussion?
There are some clarifications to make, and I'm not sure the environment variable would work as the webserver may receive concurrent requests, with the potential for two GH solve's to be running at the same time, in the same process space.

Hi @CapnK! I've closed this as we're certain we do not want to officially allow for plain text tokens stored in grasshopper definitions: this has the potential to undermine the trust in the whole Speckle ecosystem. Speckle Core is explicitly and carefully architected to prevent any kind of security critical information to be serialised in any files in which Speckle might be used, so it's a definitive no.
Ideally this would've been discussed first in an actual issue rather than a surprise PR 😎 IMHO, the best place would be the forum as it's not something related to the code in this repo but rather to the way you will architect your rhino compute server and its integration with Speckle. We can help you on this front and explore options together if you want to.
As a post mortem, you can much easier do what this PR intends via a few lines of C# inside a scripting component:

Agreed that env variables would not work. If your server application is registered as a third party application with your Speckle Server, it will be granted a token which it can use to act on behalf of the user. You can store this token somehow in your backend and inject it programmatically in the gh definition before it is run by the rhino compute engine.

Hey @didimitrie, I guess I was going the long way around to solve this issue. I see this as a simpler and neater solution. But you can see how I was lead in that direction if the Account class doesn't expose a minimal constructor with token / api and instead there are a bunch of public properties I cannot be certain are unnecessary?

I was a bit quick to praise this solution as it seems that most components require this account to be added / be visible on the Speckle Manager:

The proposed solution was adding it on the local accounts. Any thoughts?

Hey Stam,
Would you be able to use local accounts instead?
https://speckle.guide/user/manager.html#adding-accounts-manually
https://github.com/specklesystems/speckle-sharp/blob/main/Core/Core/Credentials/AccountManager.cs#L126
On a similar note, would RhinoCompute have access to Speckle's kit?

Let me take a step back and reframe our aims. Our high level goal is to introduce accounts at a grasshopper canvas level of scope, in a secure manner. (This is Arup's goal, I'm not suggesting that this is necessarily a Speckle-wide goals).
Question 1: do we want to assume grasshopper canvas scope is the same as computer scope? This is a Rhino.Compute set up question and I'll follow up with @CapnK. It is much easier to deal with this if each canvas is run in a separate worker with a clean environment, because then you can just write an account to the SQLite DB as @teocomi suggests.
However, for this discussion, the interesting question is: can we securely define an account on the GH canvas level?
Here's one idea: when searching for accounts, look for a SpeckleAccounts.json in the same directory as the grasshopper file.
The "specify account" component then just takes username and server URL as input and searches the json file for the rest of the info (and then the DB if it doesn't find it). The advantage to this method is that it doesn't change how all the other components deal with accounts, so we maintain the token protections in place, and the GH file can be  shared without including the token.
The Rhino.Compute environment can then authenticate the user, save the token to a separate file, run the script then delete the token file.
Thoughts?

p.s. my concern with directly adding accounts to the DB is that if multiple canvases are running in a single Rhino.Compute runtime, we are potentially exposing tokens to multiple users who shouldn't have them.

I feel this is the right direction. Ideally, the most secure way to go would be to inject somehow the token at runtime in the gh canvas through the backend service that will most likely wrap around the compute engine. E.g, user goes to web ui, logs in, uploads gh definition and sets params (which stream to post data to, which branch). The backend service then knows the identity of the user, and can inject the token in the gh canvas when it's run. I'm assuming here that the backend+frontend service are a registered third party app on the speckle server you're running this against!

yup, that's the exact approach we use. The question is how to inject them.
Rhino.Compute has a mechanism to inject data directly into the canvas (using RH_IN tags), thus the component in this PR that takes username, server and token as input. The intention is for this component to only ever be used with injected data, but the question is how to do it without creating something easily abused...

@teocomi Yes this is what this PR was doing. Anyway, the idea has been moved here: https://github.com/arup-group/SpeckleExtensions
@daviddekoning we can maybe add something that removes it once the canvas finishes?

Honestly, in this situation, GH is a programming environment, and we can't stop people from hardwiring their tokens into their Python or dotNet code and committing it to GitHub. My gut feeling is to make it easy to do this securely, but making it "impossible" for programmers to make mistakes will be counter productive.

Hey @psarras thanks!
I think it makes a lot of sense to have these components in a separate repo, given their use. In case you think they should instead be part of the core Speckle ones, maybe start a discussion on the forum, so other Speckle+RhinoCompute users can chime in.
@daviddekoning I would argue that developers have a better understanding of tokens and basic security concepts that average computational designers usually don't have ;) we don't want to make it impossible to use tokens from GH, we simply don't want to encourage it!


Deprecated implementation

PythonSIFT/pysift.py:80: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(gaussian_images)

@ra0e
Fixed in commit 78c779ae58fd95d4ea65924fd7471d8038f5a9be.
Thanks for pointing this out.

@ra0e
Fixed in commit 78c779ae58fd95d4ea65924fd7471d8038f5a9be.
Thanks for pointing this out.


Constrain ContinueButton width to 300 pixels

Addresses the issue with the button stretching too far on a wide mobile/tablet.

Rebase?