Add MoveBundle command

Objective
Moving one component from a entity to another entity is a common operation. It would make sense if this was supported out the box.
Solution
Create a move_bundle API on EntityCommands which results in a MoveBundle command. The MoveBundle command peforms EntityMut::remove_bundle::<T> on the source entity and then EntityMut::insert_bundle::<T> on the target entity.
Possible Additions
It might be reasonable to also add a CloneBundle which does the same except replaces remove::<T> with get::<T> then a clone.

IMO we should add the move_from variants in the same PR. I've been writing the clonable-entities RFC and having both directions is very useful for ergonomics.

@alice-i-cecile Should I link to the *_to methods in the *_from documentation and then omit the doc tests?

@hlb8122 do you want to clean this up so we can merge it?

This looks like it has gone stale, and required components change the landscape here significantly. Closing as not planned.

I still think this is a compelling idea, but I agree, this needs to be completely remade.


--lint failing when config file is missing

Is your feature request related to a problem? Please describe.
As described here, currently snakemake --lint fails with an error when a config file is missing. I consider this unwanted behaviour in this context, as it makes it unusable in GH Actions on a public repository (with suites like e.g. super-linter) if the config file contains absolute paths.
Describe the solution you'd like
snakemake --lint not throwing an error when the configfile is missing.
Describe alternatives you've considered
I think a warning would be a possible substitute, if the file is not strictly required by the linter

It makes sense. Another solution I was thinking of would be to allow specifying a test config file with example paths (like in a testing suite).


Don't add default flags that conflict with _FLAGS

This is an evolution of https://github.com/alexcrichton/cc-rs/pull/374. That PR was too naive with duplicate flag checking and ended up causing problems.
This change organizes flags into two groups: flags that do not carry a value and are mutually exclusive (-m64, -m32) and flags that carry a value that should only be set once (-march=, -mcpu=). Both groups are handled separately.

Thanks for the PR! After the first iteration though I'm pretty wary about this in the sense that it seems like it may cause undue difficulty keeping it working over time. Since we place user flags at the end and later flags should take precedence, is this still needed?

I'm really sorry about that. I should have better considered the implications of the naive first implementation.
The main pain point this solves is successfully building crates that set warnings_into_errors - these fail on the flag conflict warning.
I had hoped that the CC_NO_DEFAULTS environment variable would offer a useful tool to help solve these situations, but my experience so far has been that setting it causes a lot more trouble than it solves.

Oh no worries at all! This area of compiling is notoriously tricky and really tough to ever know if it's right or wrong, so sometimes it's just a "throw things out there and see what sticks" sort of situation :)
That's an interesting point though about warnings to errors, would it be possible to specifically disable warnings related to overriding command line flags though? (since it's the intended behavior that we want)


D3D Fullscreen capture

So, it was mentioned in the features that this library was able to capture screenshots of fullscreen applications, even when alt+tabbed from said applications. How would one get around to achieve this? Can't see it as an extra device, nor as an extra output, and digging around the code yielded nothing.
Some help would be greatly appreciated

Hi ianespana, thanks for trying the library out. For fullscreen applications you don't need to do anything special, just normally specify the output and device in dxcam.
The logic for alt+tab is here:
https://github.com/ra1nty/DXcam/blob/cd1c36783d206027075309d7a9be89ef5099d10a/dxcam/dxcam.py#L91-L109
Alt + Tab will yield a DXGI_ERROR_ACCESS_LOST error and the error is handled by releasing and recreating the IDXGIOutputDuplication interface.

Ah, I see. So you can't exactly target a full screen application only, but it will handle the error that occurs when you alt+tab from said application

What do you mean by "target"? I imagine for capturing a full-screen application you just specify the screen that the app is on and do the capture.


Support multiple channel (supports creating multiple RtcEngine)

Is your feature request related to a problem? Please describe.
Yes. Future create(String appId)
/// The Agora RTC Native SDK supports creating only one RtcEngine instance for an app for now.
My platform need to handle multiple channel at the same time. Could Agora's team help this? Thanks a lot !!!

dev version support multiple channel

Thanks for your reply @LichKing-2234
I'm using dev version. But RtcEngine it still using Singleton pattern. So we can use only one RtcEngine instance.
I can not use RtcEngine rtcEngine2 to join another channel (it will overload the old one). Or we have another way join multiple channel?
Could you help me check again. Thanks @LichKing-2234 !


Consistently check whether the map option as been supplied to compile

Compile returns an object instead of a string if supplied the map option, so this has to be checked before using the returned value.

We should probably add CLI tests...
Thanks!


Deploy to Heroku button not working.

It tries to load the following URL: https://dashboard.heroku.com/new?button-url=https%3A%2F%2Fgithub.com%2F&template=https%3A%2F%2Fgithub.com%2F
Nonetheless, it stays on a loading state for ever.

Do you know if it can be fixed from here? or is it a Heroku issue?
Let me know if I can be of help. I believe this is the reason why the UnityYoutubePlayer is not working

When clicking the button: current
Expected: expected


Colab cannot import piml: initialization of _internal failed

Hi,
I tried "Example_CaliforniaHousing.ipynb" for a long time but still couldn't get the initial importing of PiML working.
Tried on 2023-03-18 using Google Colab py39 runtime without a hardware accelerator.
Running the first cell !pip install piml, I got most all dependencies installed, but an error was thrown:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

I ignored the error, restarted the runtime, and moved to the next cell from piml import Experiment. I got an error (trimmed to the end):
File /usr/local/lib/python3.9/dist-packages/numba/np/ufunc/decorators.py:3
      1 import inspect
----> 3 from numba.np.ufunc import _internal
      4 from numba.np.ufunc.parallel import ParallelUFuncBuilder, ParallelGUFuncBuilder
      6 from numba.core.registry import TargetRegistry

SystemError: initialization of _internal failed without raising an exception

Is there a way to resolve this? Thanks!
Extra:
Full error of the first cell:
!pip install piml

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting piml
  Downloading PiML-0.4.3-cp39-none-manylinux_2_5_x86_64.whl (9.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.8/9.8 MB 40.6 MB/s eta 0:00:00
Collecting natsort>=8.2.0
  Downloading natsort-8.3.1-py3-none-any.whl (38 kB)
Collecting shap>=0.39.0
  Downloading shap-0.41.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (572 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 572.4/572.4 KB 19.8 MB/s eta 0:00:00
Collecting pygam>=0.8.0
  Downloading pygam-0.9.0-py3-none-any.whl (522 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 522.2/522.2 KB 25.7 MB/s eta 0:00:00
Requirement already satisfied: torch>=1.11.0 in /usr/local/lib/python3.9/dist-packages (from piml) (1.13.1+cu116)
Requirement already satisfied: numpy>=1.21.4 in /usr/local/lib/python3.9/dist-packages (from piml) (1.22.4)
Requirement already satisfied: pandas>=1.3.5 in /usr/local/lib/python3.9/dist-packages (from piml) (1.4.4)
Requirement already satisfied: scikit-learn>=0.24.2 in /usr/local/lib/python3.9/dist-packages (from piml) (1.2.2)
Requirement already satisfied: statsmodels>=0.12.2 in /usr/local/lib/python3.9/dist-packages (from piml) (0.13.5)
Requirement already satisfied: matplotlib>=3.1.2 in /usr/local/lib/python3.9/dist-packages (from piml) (3.7.1)
Requirement already satisfied: lightgbm>=3.1.1 in /usr/local/lib/python3.9/dist-packages (from piml) (3.3.5)
Requirement already satisfied: scipy>=1.5.3 in /usr/local/lib/python3.9/dist-packages (from piml) (1.10.1)
Collecting dcor>=0.5.3
  Downloading dcor-0.6-py3-none-any.whl (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.5/55.5 KB 1.2 MB/s eta 0:00:00
Requirement already satisfied: xlrd>=1.2.0 in /usr/local/lib/python3.9/dist-packages (from piml) (2.0.1)
Requirement already satisfied: xgboost>=1.4.2 in /usr/local/lib/python3.9/dist-packages (from piml) (1.7.4)
Collecting ipython>=7.12.0
  Downloading ipython-8.11.0-py3-none-any.whl (793 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 793.3/793.3 KB 29.6 MB/s eta 0:00:00
Requirement already satisfied: joblib>=1.1.0 in /usr/local/lib/python3.9/dist-packages (from piml) (1.1.1)
Requirement already satisfied: psutil>=5.9.0 in /usr/local/lib/python3.9/dist-packages (from piml) (5.9.4)
Collecting lime>=<IP_ADDRESS>
  Downloading lime-<IP_ADDRESS>.tar.gz (275 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 275.7/275.7 KB 14.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: seaborn>=0.11.2 in /usr/local/lib/python3.9/dist-packages (from piml) (0.12.2)
Collecting momentchi2
  Downloading momentchi2-0.1.8-py3-none-any.whl (11 kB)
Collecting dill>=0.3.6
  Downloading dill-0.3.6-py3-none-any.whl (110 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.5/110.5 KB 6.2 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20.5 in /usr/local/lib/python3.9/dist-packages (from piml) (23.0)
Requirement already satisfied: ipywidgets>=7.7.0 in /usr/local/lib/python3.9/dist-packages (from piml) (7.7.1)
Requirement already satisfied: numba>=0.51 in /usr/local/lib/python3.9/dist-packages (from dcor>=0.5.3->piml) (0.56.4)
Collecting stack-data
  Downloading stack_data-0.6.2-py3-none-any.whl (24 kB)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.9/dist-packages (from ipython>=7.12.0->piml) (0.7.5)
Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.9/dist-packages (from ipython>=7.12.0->piml) (4.8.0)
Requirement already satisfied: pygments>=2.4.0 in /usr/local/lib/python3.9/dist-packages (from ipython>=7.12.0->piml) (2.6.1)
Collecting matplotlib-inline
  Downloading matplotlib_inline-0.1.6-py3-none-any.whl (9.4 kB)
Collecting prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30
  Downloading prompt_toolkit-3.0.38-py3-none-any.whl (385 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 385.8/385.8 KB 20.2 MB/s eta 0:00:00
Requirement already satisfied: backcall in /usr/local/lib/python3.9/dist-packages (from ipython>=7.12.0->piml) (0.2.0)
Requirement already satisfied: decorator in /usr/local/lib/python3.9/dist-packages (from ipython>=7.12.0->piml) (4.4.2)
Requirement already satisfied: traitlets>=5 in /usr/local/lib/python3.9/dist-packages (from ipython>=7.12.0->piml) (5.7.1)
Collecting jedi>=0.16
  Downloading jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 53.3 MB/s eta 0:00:00
Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.9/dist-packages (from ipywidgets>=7.7.0->piml) (0.2.0)
Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.9/dist-packages (from ipywidgets>=7.7.0->piml) (3.0.5)
Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.9/dist-packages (from ipywidgets>=7.7.0->piml) (3.6.2)
Requirement already satisfied: ipykernel>=4.5.1 in /usr/local/lib/python3.9/dist-packages (from ipywidgets>=7.7.0->piml) (5.3.4)
Requirement already satisfied: wheel in /usr/local/lib/python3.9/dist-packages (from lightgbm>=3.1.1->piml) (0.40.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from lime>=<IP_ADDRESS>->piml) (4.65.0)
Requirement already satisfied: scikit-image>=0.12 in /usr/local/lib/python3.9/dist-packages (from lime>=<IP_ADDRESS>->piml) (0.19.3)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib>=3.1.2->piml) (1.4.4)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib>=3.1.2->piml) (1.0.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib>=3.1.2->piml) (0.11.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib>=3.1.2->piml) (3.0.9)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib>=3.1.2->piml) (8.4.0)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.9/dist-packages (from matplotlib>=3.1.2->piml) (2.8.2)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib>=3.1.2->piml) (4.39.0)
Requirement already satisfied: importlib-resources>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib>=3.1.2->piml) (5.12.0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas>=1.3.5->piml) (2022.7.1)
Collecting progressbar2<5.0.0,>=4.2.0
  Downloading progressbar2-4.2.0-py2.py3-none-any.whl (27 kB)
Collecting numpy>=1.21.4
  Downloading numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 38.2 MB/s eta 0:00:00
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from scikit-learn>=0.24.2->piml) (3.1.0)
Collecting slicer==0.0.7
  Downloading slicer-0.0.7-py3-none-any.whl (14 kB)
Requirement already satisfied: cloudpickle in /usr/local/lib/python3.9/dist-packages (from shap>=0.39.0->piml) (2.2.1)
Requirement already satisfied: patsy>=0.5.2 in /usr/local/lib/python3.9/dist-packages (from statsmodels>=0.12.2->piml) (0.5.3)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch>=1.11.0->piml) (4.5.0)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.9/dist-packages (from importlib-resources>=3.2.0->matplotlib>=3.1.2->piml) (3.15.0)
Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.9/dist-packages (from ipykernel>=4.5.1->ipywidgets>=7.7.0->piml) (6.2)
Requirement already satisfied: jupyter-client in /usr/local/lib/python3.9/dist-packages (from ipykernel>=4.5.1->ipywidgets>=7.7.0->piml) (6.1.12)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.9/dist-packages (from jedi>=0.16->ipython>=7.12.0->piml) (0.8.3)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /usr/local/lib/python3.9/dist-packages (from numba>=0.51->dcor>=0.5.3->piml) (0.39.1)
Collecting numba>=0.51
  Downloading numba-0.56.3-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 66.3 MB/s eta 0:00:00
  Downloading numba-0.56.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 83.7 MB/s eta 0:00:00
  Downloading numba-0.56.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 82.7 MB/s eta 0:00:00
  Downloading numba-0.55.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 79.0 MB/s eta 0:00:00
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 15.6 MB/s eta 0:00:00
Collecting numba>=0.51
  Downloading numba-0.55.1-1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 82.6 MB/s eta 0:00:00
  Downloading numba-0.55.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 75.4 MB/s eta 0:00:00
  Downloading numba-0.54.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 84.7 MB/s eta 0:00:00
Collecting llvmlite<0.38,>=0.37.0rc1
  Downloading llvmlite-0.37.0-cp39-cp39-manylinux2014_x86_64.whl (26.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.3/26.3 MB 52.4 MB/s eta 0:00:00
Collecting numba>=0.51
  Downloading numba-0.54.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 76.4 MB/s eta 0:00:00
  Downloading numba-0.53.1-cp39-cp39-manylinux2014_x86_64.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 77.7 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/dist-packages (from numba>=0.51->dcor>=0.5.3->piml) (63.4.3)
Collecting llvmlite<0.37,>=0.36.0rc1
  Downloading llvmlite-0.36.0-cp39-cp39-manylinux2010_x86_64.whl (25.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.3/25.3 MB 48.9 MB/s eta 0:00:00
Requirement already satisfied: six in /usr/local/lib/python3.9/dist-packages (from patsy>=0.5.2->statsmodels>=0.12.2->piml) (1.15.0)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.9/dist-packages (from pexpect>4.3->ipython>=7.12.0->piml) (0.7.0)
Requirement already satisfied: python-utils>=3.0.0 in /usr/local/lib/python3.9/dist-packages (from progressbar2<5.0.0,>=4.2.0->pygam>=0.8.0->piml) (3.5.2)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.9/dist-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=7.12.0->piml) (0.2.6)
Requirement already satisfied: imageio>=2.4.1 in /usr/local/lib/python3.9/dist-packages (from scikit-image>=0.12->lime>=<IP_ADDRESS>->piml) (2.9.0)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.9/dist-packages (from scikit-image>=0.12->lime>=<IP_ADDRESS>->piml) (1.4.1)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.9/dist-packages (from scikit-image>=0.12->lime>=<IP_ADDRESS>->piml) (2023.3.15)
Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.9/dist-packages (from scikit-image>=0.12->lime>=<IP_ADDRESS>->piml) (3.0)
Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.9/dist-packages (from widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (6.3.0)
Collecting asttokens>=2.1.0
  Downloading asttokens-2.2.1-py2.py3-none-any.whl (26 kB)
Collecting pure-eval
  Downloading pure_eval-0.2.2-py3-none-any.whl (11 kB)
Collecting executing>=1.2.0
  Downloading executing-1.2.0-py2.py3-none-any.whl (24 kB)
Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.17.1)
Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (21.3.0)
Requirement already satisfied: jupyter-core>=4.6.1 in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (5.2.0)
Requirement already satisfied: nbconvert in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (6.5.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (3.1.2)
Requirement already satisfied: pyzmq>=17 in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (23.2.1)
Requirement already satisfied: Send2Trash>=1.5.0 in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (1.8.0)
Requirement already satisfied: prometheus-client in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.16.0)
Requirement already satisfied: nbformat in /usr/local/lib/python3.9/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (5.7.3)
Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.9/dist-packages (from jupyter-core>=4.6.1->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (3.1.1)
Requirement already satisfied: argon2-cffi-bindings in /usr/local/lib/python3.9/dist-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (21.2.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/dist-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (2.1.2)
Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (1.5.0)
Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.4)
Requirement already satisfied: bleach in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (6.0.0)
Requirement already satisfied: defusedxml in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.7.1)
Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.2.2)
Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.8.4)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (4.11.2)
Requirement already satisfied: lxml in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (4.9.2)
Requirement already satisfied: tinycss2 in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (1.2.1)
Requirement already satisfied: nbclient>=0.5.0 in /usr/local/lib/python3.9/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.7.2)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.9/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (4.3.3)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.9/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (2.16.3)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (22.2.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.19.3)
Requirement already satisfied: cffi>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (1.15.1)
Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.9/dist-packages (from beautifulsoup4->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (2.4)
Requirement already satisfied: webencodings in /usr/local/lib/python3.9/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (0.5.1)
Requirement already satisfied: pycparser in /usr/local/lib/python3.9/dist-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.7.0->piml) (2.21)
Building wheels for collected packages: lime
  Building wheel for lime (setup.py) ... done
  Created wheel for lime: filename=lime-<IP_ADDRESS>-py3-none-any.whl size=283859 sha256=977759970aaaf5e3c539ceb4ea229ba921e34294642260690da4c7fac6e0c3ca
  Stored in directory: /root/.cache/pip/wheels/ed/d7/c9/5a0130d06d6310bc6cbe55220e6e72dcb8c4eff9a478717066
Successfully built lime
Installing collected packages: pure-eval, executing, slicer, prompt-toolkit, progressbar2, numpy, natsort, matplotlib-inline, llvmlite, jedi, dill, asttokens, stack-data, numba, pygam, momentchi2, ipython, dcor, shap, lime, piml
  Attempting uninstall: prompt-toolkit
    Found existing installation: prompt-toolkit 2.0.10
    Uninstalling prompt-toolkit-2.0.10:
      Successfully uninstalled prompt-toolkit-2.0.10
  Attempting uninstall: progressbar2
    Found existing installation: progressbar2 3.38.0
    Uninstalling progressbar2-3.38.0:
      Successfully uninstalled progressbar2-3.38.0
  Attempting uninstall: numpy
    Found existing installation: numpy 1.22.4
    Uninstalling numpy-1.22.4:
      Successfully uninstalled numpy-1.22.4
  Attempting uninstall: natsort
    Found existing installation: natsort 5.5.0
    Uninstalling natsort-5.5.0:
      Successfully uninstalled natsort-5.5.0
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Uninstalling llvmlite-0.39.1:
      Successfully uninstalled llvmlite-0.39.1
  Attempting uninstall: numba
    Found existing installation: numba 0.56.4
    Uninstalling numba-0.56.4:
      Successfully uninstalled numba-0.56.4
  Attempting uninstall: ipython
    Found existing installation: ipython 7.9.0
    Uninstalling ipython-7.9.0:
      Successfully uninstalled ipython-7.9.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires ipython~=7.9.0, but you have ipython 8.11.0 which is incompatible.
Successfully installed asttokens-2.2.1 dcor-0.6 dill-0.3.6 executing-1.2.0 ipython-8.11.0 jedi-0.18.2 lime-<IP_ADDRESS> llvmlite-0.36.0 matplotlib-inline-0.1.6 momentchi2-0.1.8 natsort-8.3.1 numba-0.53.1 numpy-1.24.2 piml-0.4.3 progressbar2-4.2.0 prompt-toolkit-3.0.38 pure-eval-0.2.2 pygam-0.9.0 shap-0.41.0 slicer-0.0.7 stack-data-0.6.2

Full error of the seond cell:
from piml import Experiment
exp = Experiment()

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
Cell In[2], line 1
----> 1 from piml import Experiment

File /usr/local/lib/python3.9/dist-packages/piml/__init__.py:1
----> 1 from .api import Experiment
      2 from .diagnoser.utils import twosample_test
      4 __all__ = ['Experiment']

File piml/api.py:16, in init piml.api()

File piml/dashboard/data_panel.py:12, in init piml.dashboard.data_panel()

File piml/workflow/data_api.py:8, in init piml.workflow.data_api()

File piml/data/base.py:14, in init piml.data.base()

File piml/data/data_selector.py:11, in init piml.data.data_selector()

File /usr/local/lib/python3.9/dist-packages/dcor/__init__.py:13
     10 import os as _os
     11 import pathlib as _pathlib
---> 13 from . import distances, homogeneity, independence
     14 from ._dcor import (
     15     DistanceCovarianceMethod as DistanceCovarianceMethod,
     16     Stats as Stats,
   (...)
     27     u_distance_stats_sqr as u_distance_stats_sqr,
     28 )
     29 from ._dcor_internals import (
     30     double_centered as double_centered,
     31     mean_product as mean_product,
   (...)
     35     u_projection as u_projection,
     36 )

File /usr/local/lib/python3.9/dist-packages/dcor/independence.py:14
     11 import numpy as np
     12 import scipy.stats
---> 14 from ._dcor import u_distance_correlation_sqr
     15 from ._dcor_internals import (
     16     _check_same_n_elements,
     17     _distance_matrix_generic,
   (...)
     22     u_product,
     23 )
     24 from ._hypothesis import HypothesisTest, _permutation_test_with_sym_matrix

File /usr/local/lib/python3.9/dist-packages/dcor/_dcor.py:35
     32 from dcor._dcor_internals import _af_inv_scaled
     34 from ._dcor_internals import _dcov_from_terms, _dcov_terms_naive
---> 35 from ._fast_dcov_avl import _distance_covariance_sqr_terms_avl
     36 from ._fast_dcov_mergesort import _distance_covariance_sqr_terms_mergesort
     37 from ._utils import ArrayType, CompileMode, _sqrt, get_namespace

File /usr/local/lib/python3.9/dist-packages/dcor/_fast_dcov_avl.py:18
      7 import warnings
      8 from typing import (
      9     TYPE_CHECKING,
     10     Any,
   (...)
     15     overload,
     16 )
---> 18 import numba
     19 import numpy as np
     20 from numba import boolean, float64, int64

File /usr/local/lib/python3.9/dist-packages/numba/__init__.py:43
     39 from numba.core.decorators import (cfunc, generated_jit, jit, njit, stencil,
     40                                    jit_module)
     42 # Re-export vectorize decorators and the thread layer querying function
---> 43 from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
     44                             get_num_threads, set_num_threads)
     46 # Re-export Numpy helpers
     47 from numba.np.numpy_support import carray, farray, from_dtype

File /usr/local/lib/python3.9/dist-packages/numba/np/ufunc/__init__.py:3
      1 # -*- coding: utf-8 -*-
----> 3 from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
      4 from numba.np.ufunc._internal import PyUFunc_None, PyUFunc_Zero, PyUFunc_One
      5 from numba.np.ufunc import _internal, array_exprs

File /usr/local/lib/python3.9/dist-packages/numba/np/ufunc/decorators.py:3
      1 import inspect
----> 3 from numba.np.ufunc import _internal
      4 from numba.np.ufunc.parallel import ParallelUFuncBuilder, ParallelGUFuncBuilder
      6 from numba.core.registry import TargetRegistry

SystemError: initialization of _internal failed without raising an exception

Resolved by using numpy 1.23.1 instead of 1.24.2!
Specifically, I added the following two lines after !pip install piml
!pip uninstall numpy -y
!pip install numpy==1.23.1

Thanks.
PS: love the great work of PiML in making neural nets interpretable in an easy way!

Can solve the issue by
!pip uninstall numpy -y
!pip install numpy==1.23.1

Hi @faradawn
Thanks for reporting this issue.
Temporarily, the corresponding version requirements are added in each example notebook.
In the next release, such requirements will be enforced automatically when installing PiML.


Fine-tuning without '*unfreeze.yaml' config file

Thanks for sharing nice work and source code.
I found that '*unfreeze.yaml' files in configs directory, such as faster_rcnn_R_101_FPN_ft_all3_1shot_unfreeze.yaml.
Is there experimental result with those config files?
Thanks,

Oh, i see those are *FRCN+ft-full.


Release - 0.3.0

Features

#94
#95
#96
#97
#98
#99
#100
#102
#103
#104
#105
#106
#107

Chores

#96

Bugs
N/A

@suho you forgot to put milestone link in description 💪


FanyZones last known zone doesn´t work as expected

Environment
Windows build number: Version 1803 (Build 17134.1425)
PowerToys version: v0.17.0
PowerToy module: FanyZones
Function: *Move newly created windows to their last know zone*

Steps to reproduce

Open a Window and attach it to a zone (works fine)
Close the Window (works fine)
Open it again

Now the Window is slightly shifted and it seems like its not actual in this Zone

Expected behavior
Set the Window in the correct zone in the correct size.
Actual behavior
Window is in the correct size, but not in the correct zone.
Well its slightly shifted
Screenshots

Are you still having the same issue with 0.20 or newer?

I also have this issue, when opening multiple instances of file explorer.
The first instance receives correct position, but the following floats somewhere else in my screen.

And here's another, specifically Wondershare PDFelement (I am using ver7.5.7) - doesn't cover the entire zone.

The spaces are set to 0.
The bug could not be directly related to PowerToys, that's why I don't brag this before.


Add a multicast operator

For pull based Streams this should buffer the source Stream until all reads have complete.
For push based Streams this should push the same value to all connected Streams.

Released in 2.0.0-MI3


Creating with Associations - Repository mode

Issue
Would like to know if Creation with associations is supported with repository mode?
Versions

sequelize: V6.6.2
sequelize-typescript: V2.1.0
typescript: V4.1.3

Issue type
Not sure of issue type - if it's not supported - then it would be a good feature request.
if it is supported, then It may not be working, and then it's a bug.

[ x] bug report
[ x] feature request

Actual behavior
When attempting to create a parent record and (associated (HasMany) relationship) children with a repository
that was created for the parent - pseudocode
const parentRepo = sequelize.getRepository(Parent)
parentRepo.create(Parent, { include: [Children]});
Then get an error:
Model not initialized: Member "getTableName" cannot be called. "Children" needs to be added to a Sequelize instance.
Expected behavior
Child records are created if they don't already exist.
Steps to reproduce
As above
Please let me know if this functionality is supported - it was not clear in the documentation as it did not state that this was one of the limitations of repository mode.
Thanks!

Sorry Folks, I need to RTFM - FWIW the docs specify that a repository for the included model be supplied,
change the above as follows:
const childRepo = sequelize.getRepository(Children); const parentRepo = sequelize.getRepository(Parent); parentRepo.create(Parent, { include: [childRepo]});
There do appear to be limitations on updates... But this seems to be Sequelize itself.


write a structural hasher in lambdas  to help with the no_egg transition in stitch

note it will need to be able to build up egg style analysis data too

done on no_egg branch


auto detect docker-for-mac

When using docker-machine, the DOCKER_HOST env var is typically set. Tweak the auto detection for the MacOS to avoid having ot use -t.

My fingers can’t longer take the extra “-t local”
🙏🏻🎉🙌🏻🔥💯🤩👏🏼🙀🌮🏆

Thanks - tweaked the doc.


Check if __horseman exists on window object in browser before evaluate

Fixing "ReferenceError: Can't find variable: __horseman".
Fairly complex Hapi.js API communicating with various websites via Horseman depending on user actions. The issue pops up because window.__horseman init does not run as the API answers to users killing the promise chain before it reaches this init. On subsequent actions with same Horseman instance running evaluate does not find the __horseman in browser and the whole process fails.
Possibly better future solutions could be @awlayton TODO: Consolidate into one page.evaluate? and not adding the window.__horseman before we need it (only in evaluate).
Thanks!

If _horseman isn't there, something went wrong. I do not think this change is a good idea. It just masks the real problem rather than fixing it.

I am yet to be able to isolate the issue into a simple example but working on it. What makes it tricky is that I cant reliably recreate it and did not always happen on live systems either. Only thing certain is that after adding the lines in this PR the issue is gone for me. I definitely want to be in sync with the project and not run my custom version in the long run.
Is there a reason why init of __horseman is where it is right now and not within evaluate only when its called? window.__horseman init.

Did some tests and log mining and core of the issue seams to be that phantomjs onLoadFinished triggered is not always there. The reasons are still unknown and simple recreation fails.

I guess we could move it like you want. But that won't fix the underlying problem. Right now it is part of the page setup. Your fix won't make the rest of the page setup happen as it should (e.g. injecting jQuery).
I do not know of a way to force the onLoadFinished, but even if there is one we need to know if loading it's actually finished or not somehow...

The way it is now, it issupposed to wait for that callback before moving on to something else, e.g. an evaluate. Could you post some log of when it seems to not be waiting? Please turn on DEBUG='horseman*' and BLUEBIRD_DEBUG=1 when generating the log.

{ line: 2,
  message: 'Can\'t find variable: __horseman',
  sourceId:<PHONE_NUMBER>84960,
  stack: 'ReferenceError: Can\'t find variable: __horseman\n    at waitForCb (:2)\n    at evaluatePage (phantomjs://webpage.evaluate():6)\n    at phantomjs://webpage.evaluate():11\n    at phantomjs://webpage.evaluate():11\n    at /opt/netbet/node_modules/node-horseman/lib/actions.js:992:36\n    at tryCatcher (/opt/netbet/node_modules/bluebird/js/release/util.js:16:23)\n    at Function.Promise.fromNode.Promise.fromCallback (/opt/netbet/node_modules/bluebird/js/release/promise.js:178:30)\n    at Horseman.<anonymous> (/opt/netbet/node_modules/node-horseman/lib/actions.js:990:29)\n    at Horseman.tryCatcher (/opt/netbet/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/opt/netbet/node_modules/bluebird/js/release/promise.js:504:31)\n    at Promise._settlePromise (/opt/netbet/node_modules/bluebird/js/release/promise.js:561:18)\n    at Promise._settlePromiseCtx (/opt/netbet/node_modules/bluebird/js/release/promise.js:598:10)\n    at Async._drainQueue (/opt/netbet/node_modules/bluebird/js/release/async.js:143:12)\n    at Async._drainQueues (/opt/netbet/node_modules/bluebird/js/release/async.js:148:10)\n    at Immediate.Async.drainQueues [as _onImmediate] (/opt/netbet/node_modules/bluebird/js/release/async.js:17:14)\n    at processImmediate [as _immediateCallback] (timers.js:383:17)',

this happens when i use ignoreSSLErrors = true

That is very interesting if it is related to ignoreSLLErrors. Could you please post a complete log of the output @aryeharmon (with BLUEBIRD_DEBUG=1 and DEBUG='horseman*' set if they are not already)?

I am also getting this error. @awlayton here is a full error log with debugging enabled.
DEBUG='horseman*' BLUEBIRD_DEBUG=1 node ./as-library.js
module.js:341
    throw err;
    ^

Error: Cannot find module '/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/as-library.js'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Function.Module.runMain (module.js:447:10)
    at startup (node.js:148:18)
    at node.js:405:3
➜  lib git:(horsemanjs) ✗ cd ../..
➜  node_modules git:(horsemanjs) ✗ cd ..
➜  tiyo-login git:(horsemanjs) ✗ DEBUG='horseman*' BLUEBIRD_DEBUG=1 node ./as-library.js
module.js:341
    throw err;
    ^

Error: Cannot find module '/Users/jim.cummins/projects/tiyo-login/as-library.js'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Function.Module.runMain (module.js:447:10)
    at startup (node.js:148:18)
    at node.js:405:3
➜  tiyo-login git:(horsemanjs) ✗ cd examples
➜  examples git:(horsemanjs) ✗ DEBUG='horseman*' BLUEBIRD_DEBUG=1 node ./as-library.js
  horseman using PhantomJS from phantomjs module +0ms
  horseman .setup() creating phantom instance 1 +5ms
  horseman .viewport() set +18ms 3200 1800
  horseman phantom created +506ms
  horseman phantom version 2.1.1 +37ms
  horseman page created +25ms
  horseman phantomjs onLoadFinished triggered +18ms success NaN
  horseman injected jQuery +50ms
  horseman .open() +21ms https://online.theironyard.com/users/sign_in
  horseman phantomjs onLoadFinished triggered +9s success 1
  horseman injected jQuery +24ms
  horseman .click() +1ms #new_user [type=submit]
  horseman .click() done +15ms
  horseman .waitForSelector() +1ms .dashboard
  horseman .waitFor() +1ms elementPresent .dashboard
  horseman:verbose .waitFor() iteration +87ms elementPresent false 56 1
  horseman:verbose .waitFor() iteration +48ms elementPresent false 116 1
  horseman:verbose .waitFor() iteration +54ms elementPresent false 177 1
  horseman:verbose .waitFor() iteration +48ms elementPresent false 230 1
  horseman:verbose .waitFor() iteration +77ms elementPresent false 285 1
  horseman:verbose .waitFor() iteration +43ms elementPresent false 344 1
  horseman:verbose .waitFor() iteration +50ms elementPresent false 398 1
  horseman:verbose .waitFor() iteration +65ms elementPresent false 449 1
  horseman:verbose .waitFor() iteration +44ms elementPresent false 506 1
  horseman:verbose .waitFor() iteration +54ms elementPresent false 559 1
  horseman:verbose .waitFor() iteration +53ms elementPresent false 611 1
  horseman:verbose .waitFor() iteration +62ms elementPresent false 661 1
  horseman:verbose .waitFor() iteration +38ms elementPresent false 714 1
  horseman:verbose .waitFor() iteration +55ms elementPresent false 766 1
  horseman:verbose .waitFor() iteration +56ms elementPresent false 819 1
  horseman:verbose .waitFor() iteration +43ms elementPresent false 870 1
  horseman:verbose .waitFor() iteration +50ms elementPresent false 921 1
  horseman:verbose .waitFor() iteration +68ms elementPresent false 979 1
  horseman:verbose .waitFor() iteration +49ms elementPresent false 1037 1
  horseman:verbose .waitFor() iteration +52ms elementPresent false 1088 1
  horseman:verbose .waitFor() iteration +118ms elementPresent true 1138 1
  horseman .waitFor() completed successfully +0ms
  horseman .waitForSelector() complete +1ms
  horseman .screenshot() +1ms big.png
  horseman:verbose .waitFor() iteration +43ms elementPresent true 1189 1
  horseman .waitFor() completed successfully +4ms
  horseman .evaluate() +3s <anonymous>
  horseman:verbose onError +93ms ReferenceError: Can't find variable: __horseman
        undefined: 2 in evaluate
        : 74 in <anonymous> 1
  horseman:verbose .evaluate() waiting for callback +0ms <anonymous> null 1
  horseman .waitFor() +0ms waitForCb
Warning: a promise was rejected with a non-error: [object Object]
    at /Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:1010:30
    at tryOnImmediate (timers.js:534:15)
    at processImmediate [as _immediateCallback] (timers.js:514:5)
From previous event:
    at handleErrback (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:1003:23)
    at /Users/jim.cummins/projects/tiyo-login/node_modules/node-phantom-simple/node-phantom-simple.js:60:18
    at IncomingMessage.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-phantom-simple/node-phantom-simple.js:645:9)
    at emitNone (events.js:85:20)
    at IncomingMessage.emit (events.js:179:7)
    at endReadableNT (_stream_readable.js:913:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
From previous event:
    at Horseman.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:1002:5)
    at tryOnImmediate (timers.js:534:15)
    at processImmediate [as _immediateCallback] (timers.js:514:5)
From previous event:
    at Horseman.evaluatePage (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:984:20)
    at Timeout.waitForCheck [as _repeat] (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:1646:26)
    at Timeout.wrapper [as _onTimeout] (timers.js:408:11)
    at tryOnTimeout (timers.js:224:11)
    at Timer.listOnTimeout (timers.js:198:5)

Warning: a promise was rejected with a non-error: [object Object]
    at Horseman.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:1664:8)
From previous event:
    at Horseman.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:1632:10)
    at tryOnImmediate (timers.js:534:15)
    at processImmediate [as _immediateCallback] (timers.js:514:5)
From previous event:
    at Horseman.waitForPage (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:1631:20)
    at waitForCb (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:918:25)
    at /Users/jim.cummins/projects/tiyo-login/node_modules/node-phantom-simple/node-phantom-simple.js:60:18
    at IncomingMessage.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-phantom-simple/node-phantom-simple.js:645:9)
    at emitNone (events.js:85:20)
    at IncomingMessage.emit (events.js:179:7)
    at endReadableNT (_stream_readable.js:913:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
From previous event:
    at Horseman.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:914:5)
    at tryOnImmediate (timers.js:534:15)
    at processImmediate [as _immediateCallback] (timers.js:514:5)
From previous event:
    at Horseman.exports.evaluate (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:832:20)
    at Horseman.(anonymous function) [as evaluate] (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/index.js:402:17)
    at Horseman.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/index.js:410:22)
    at /Users/jim.cummins/projects/tiyo-login/node_modules/node-phantom-simple/node-phantom-simple.js:60:18
    at IncomingMessage.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-phantom-simple/node-phantom-simple.js:645:9)
    at emitNone (events.js:85:20)
    at IncomingMessage.emit (events.js:179:7)
    at endReadableNT (_stream_readable.js:913:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
From previous event:
    at Promise.HorsemanPromise.(anonymous function) [as evaluate] (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/index.js:408:15)
    at login (/Users/jim.cummins/projects/tiyo-login/index.js:15:6)
    at Object.<anonymous> (/Users/jim.cummins/projects/tiyo-login/examples/as-library.js:12:1)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Function.Module.runMain (module.js:447:10)
    at startup (node.js:148:18)
    at node.js:405:3

  horseman .close(). +92ms
Warning: a promise was rejected with a non-error: [object Object]
    at ClientRequest.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-phantom-simple/node-phantom-simple.js:501:11)
From previous event:
    at tryOnImmediate (timers.js:534:15)
    at processImmediate [as _immediateCallback] (timers.js:514:5)
From previous event:
    at login (/Users/jim.cummins/projects/tiyo-login/index.js:35:13)
    at Object.<anonymous> (/Users/jim.cummins/projects/tiyo-login/examples/as-library.js:12:1)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Function.Module.runMain (module.js:447:10)
    at startup (node.js:148:18)
    at node.js:405:3

Unhandled rejection waitForCb
evaluatePage

➜  examples git:(horsemanjs) ✗ node ./as-library.js
Unhandled rejection waitForCb
evaluatePage

global code
evaluateJavaScript@[native code]
evaluate@phantomjs://platform/webpage.js:390:39
phantomjs://code/bridge.js:121:61
    at /Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:989:36
    at tryCatcher (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.fromNode.Promise.fromCallback (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/promise.js:178:30)
➜  examples git:(horsemanjs) ✗ node ./as-library.js
Unhandled rejection waitForCb
evaluatePage

global code
evaluateJavaScript@[native code]
evaluate@phantomjs://platform/webpage.js:390:39
phantomjs://code/bridge.js:121:61
    at /Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:989:36
    at tryCatcher (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.fromNode.Promise.fromCallback (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/promise.js:178:30)
    at Horseman.<anonymous> (/Users/jim.cummins/projects/tiyo-login/node_modules/node-horseman/lib/actions.js:987:29)
    at Horseman.tryCatcher (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/promise.js:504:31)
    at Promise._settlePromise (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/promise.js:561:18)
    at Promise._settlePromiseCtx (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/promise.js:598:10)
    at Async._drainQueue (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/async.js:143:12)
    at Async._drainQueues (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/async.js:148:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/jim.cummins/projects/tiyo-login/node_modules/bluebird/js/release/async.js:17:14)
    at tryOnImmediate (timers.js:534:15)
    at processImmediate [as _immediateCallback] (timers.js:514:5)

It would seem that the code for waiting for the page injection to finish does not always work. When I get time I will look into it. If you put a .delay() after the .open() does that improve/fix the problem?

@awlayton I put long delays between each step and it still happens with the same error. Doesn't seem to be timing related in this sense. It appears the page has no problem navigating between the two pages (my click causes a new page to be loaded). My wait selector is really just waiting for the new page to be drawn.
I even put a screenshot in there after the delay but before evaluate. It runs properly and the screenshot is correct.

FWIW jQuery is already in this page so it is not getting injected.

Ok solved my issue. I was using waitForSelector when I really wanted to waitForNextPage combined with waitForSelector. Once I used both and bumped up my timeout I was able to get it to work. I think the problem is that I was evaluating before horseman was injected.

I am not able to recreate the situation playing against my test page. My theory on the issue I am having is as follows:

User(end user using my app that is running Horseman) initiates login on Webpage (the web that causes the reference error) via the App(Hapi powered API heavily using Horseman to interact with different sites based on user actions)
Webpage is waiting for the User to finish login on their mobile device with 2 factor auth
Once the 2 factor auth is successful Webpage sometimes refreshes, other times loads the new DOM elements async. That is why I have hard time recreating it and see it in logs on some of the attempts only.
App constantly checks what the status is on the Webpage going to .evaluate() once login has been successful.

My implementation uses .waitFor() with a function checking if the evaluatable is readable and if it is not then repeats the request checking if auth status is waiting, done or timed out. Now the times that the webpage did refresh I dont have any way to get onLoadFinished to run and __horseman to be there before evaluate. I can not start waiting for next page at any point since the 2 factor auth could take any amount of time. Is there a way to force the onLoadFinished event without .waitForNextPage()? This would most likely be better for my case than the temporary solution in this PR.
PR itself does offer help in similar weird cases as protection but it does possibly mask some other issues.

I was able to resolve my issue without this PR.
On August 15, 2016 at 7:17:40 AM, Robert Pallas<EMAIL_ADDRESS>wrote:

I am not able to recreate the situation playing against my test page. My
theory on the issue I am having is as follows:

User(end user using my app that is running Horseman) initiates
login on Webpage (the web that causes the reference error) via the App(Hapi
powered API heavily using Horseman to interact with different sites based
on user actions)
Webpage is waiting for the User to finish login on their mobile
device with 2 factor auth
Once the 2 factor auth is successful Webpage sometimes refreshes,
other times loads the new DOM elements async. That is why I have hard time
recreating it and see it in logs on some of the attempts only.
App constantly checks what the status is on the Webpage going to
.evaluate() once login has been successful.

My implementation uses .waitFor() with a function checking if the
evaluatable is readable and if it is not then repeats the request checking
if auth status is waiting, done or timed out. Now the times that the
webpage did refresh I dont have any way to get onLoadFinished to run and
__horseman to be there before evaluate. I can not start waiting for next
page at any point since the 2 factor auth could take any amount of time. Is
there a way to force the onLoadFinished event without .waitForNextPage()?
This would most likely be better for my case than the temporary solution in
this PR.
PR itself does offer help in similar weird cases as protection but it does
possibly mask some other issues.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/johntitus/node-horseman/pull/208#issuecomment-239788631,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGpip6iok_2lLv6lPE324jy4bdkAZNgks5qgFjkgaJpZM4JaZl-
.

I had this problem last week. The code in the PR fixed it (or hid it).
The problem occurred when I tried to run evaluate inside an iframe. jQuery wasn't being injected into the iframe either, though I didn't need that.

@jimthedev Did you resolve this issue by calling .waitForNextPage, followed by .waitForSelector, and bumping up the timeout in your config? Otherwise, please share!

I have the same problem as @jorgeesquer .
When I call
horseman
  .switchFrame(1)
  .evaluate(function() {
    console.log('Hello');
  })

I have the error:
"Phantom error" [
  "ReferenceError: Can't find variable: __horseman",
  [
    {
      "file": "undefined",
      "line": 2,
      "function": "evaluate"
    },
    {
      "file": "",
      "line": 74,
      "function": ""
    }
  ]
]

If I include this pull request, I no longer have the error, but I also don't have jQuery injected inside the frame.
Is is possible that Horseman is not "rebuilding" the Horseman context in the frame?
Full output of commands with Horseman logs:
Tue, 14 Mar 2017 10:43:44 GMT horseman phantom created
Tue, 14 Mar 2017 10:43:44 GMT horseman phantom version 2.1.1
Tue, 14 Mar 2017 10:43:44 GMT horseman page created
Tue, 14 Mar 2017 10:43:44 GMT horseman phantomjs onLoadFinished triggered success NaN
Tue, 14 Mar 2017 10:43:44 GMT horseman injected jQuery from /home/steven/wars/groot/bin/groot/bin/lib/jquery-2.1.1.min.js
Tue, 14 Mar 2017 10:43:44 GMT horseman .viewport() set 1024 768
Tue, 14 Mar 2017 10:43:44 GMT horseman .on() consoleMessage set.
Tue, 14 Mar 2017 10:43:44 GMT horseman .on() error set.
Tue, 14 Mar 2017 10:43:44 GMT horseman .userAgent() set Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Tue, 14 Mar 2017 10:43:44 GMT horseman .url()
[Tue Mar 14 2017 11:43:44 GMT+0100 (CET)] [trace] "[ACTION][OPEN] Opening http://localhost:3009/contentInAFrame?firstFrame"
Tue, 14 Mar 2017 10:43:44 GMT horseman .open() http://localhost:3009/contentInAFrame?firstFrame
Tue, 14 Mar 2017 10:43:44 GMT horseman phantomjs onLoadFinished triggered success 1
Tue, 14 Mar 2017 10:43:44 GMT horseman injected jQuery from /home/steven/wars/groot/bin/groot/bin/lib/jquery-2.1.1.min.js
Tue, 14 Mar 2017 10:43:44 GMT horseman .url()
[Tue Mar 14 2017 11:43:44 GMT+0100 (CET)] [trace] "Opened link at http://localhost:3009/contentInAFrame?firstFrame"
[Tue Mar 14 2017 11:43:44 GMT+0100 (CET)] [trace] "Getting href from a with frame 0"
[Tue Mar 14 2017 11:43:44 GMT+0100 (CET)] [trace] "Getting links"
Tue, 14 Mar 2017 10:43:44 GMT horseman .evaluate() <anonymous> a href
Tue, 14 Mar 2017 10:43:44 GMT horseman .switchToFrame() 0
[Tue Mar 14 2017 11:43:44 GMT+0100 (CET)] [error] "Phantom error" [
  "ReferenceError: Can't find variable: __horseman",
  [
    {
      "file": "undefined",
      "line": 2,
      "function": "evaluate"
    },
    {
      "file": "",
      "line": 74,
      "function": ""
    }
  ]
]
Tue, 14 Mar 2017 10:43:44 GMT horseman .waitFor() waitForCb
{ line: 2,
  stack: 'waitForCb\nevaluatePage\n\nglobal code\nevaluateJavaScript@[native code]\nevaluate@phantomjs://platform/webpage.js:390:39\nphantomjs://code/bridge.js:121:61\n    at /home/steven/wars/groot/node_modules/node-horseman/lib/actions.js:989:36\n    at tryCatcher (/home/steven/wars/groot/node_modules/bluebird/js/release/util.js:16:23)\n    at Function.Promise.fromNode.Promise.fromCallback (/home/steven/wars/groot/node_modules/bluebird/js/release/promise.js:179:30)\n    at Horseman.<anonymous> (/home/steven/wars/groot/node_modules/node-horseman/lib/actions.js:987:29)\n    at Horseman.tryCatcher (/home/steven/wars/groot/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/home/steven/wars/groot/node_modules/bluebird/js/release/promise.js:510:31)\n    at Promise._settlePromise (/home/steven/wars/groot/node_modules/bluebird/js/release/promise.js:567:18)\n    at Promise._settlePromiseCtx (/home/steven/wars/groot/node_modules/bluebird/js/release/promise.js:604:10)\n    at Async._drainQueue (/home/steven/wars/groot/node_modules/bluebird/js/release/async.js:138:12)\n    at Async._drainQueues (/home/steven/wars/groot/node_modules/bluebird/js/release/async.js:143:10)\n    at Immediate.Async.drainQueues (/home/steven/wars/groot/node_modules/bluebird/js/release/async.js:17:14)\n    at runCallback (timers.js:666:20)\n    at tryOnImmediate (timers.js:639:5)\n    at processImmediate [as _immediateCallback] (timers.js:611:5)' }

@awlayton I believe there's a bug relating to iframes which relates to this PR, so I've opened it up here.

As I said before, this is more of a bandaid than a fix for the underlying issue. However, I do not know when I will have the time to fix it properly.
I am leaning towards accepting this since it will make things better now, even if not completely right. Do you have an opinion on merging this in @johntitus?

The part about jquery not being injected after this pull requests concerns me, but it looks like all tests are passing.  I'd say do it, and let's see if anyone complains.

The PR won't cause jQuery to not be inject any more than before. I am not entirely happy because it won't fix whatever is already causing jQuery etc. to not get injected sometimes.

I see to have confused GitHub, but the changes from this PR are merged in now.


chore: remove "dead code" files

This removes some files that aren't performing a specific action, and which we can safely ignore.
Additions to .gitignore will help default behaviors in the case that logs and/or output folders are recreated by future invocations.

Feel free to ignore/disregard. This came up in my fork and I thought it might be worth contributing back here.


Content from MDX component isn't indexed

What version of starlight are you using?
0.7.3
What version of astro are you using?
2.10.5
What package manager are you using?
npm
What operating system are you using?
Mac
What browser are you using?
Safari
Describe the Bug
Content in a .mdx file that comes from imported component (.astro) doesn't get indexed.

After build, fulltext search doesn't work on it sorry
Headings don't appear in left sidebar (table of contents)

Link to Minimal Reproducible Example
https://stackblitz.com/edit/github-xs3vfn?file=src%2Fcontent%2Fdocs%2Fguides%2Fexample.mdx
Participation

[ ] I am willing to submit a pull request for this issue.

This may be related to https://github.com/withastro/astro/issues/5084 and also this closed PR https://github.com/withastro/astro/pull/6922.
Basically, non-markdown headings do not show up in the TOC and this is used to exclude headings from the ToC.
It looks like Docusaurus also uses the same pattern and they have a discussion to gather feedback on this behavior.
If this is a behavior that is preserved, then I think this should at least be documented.

Thanks for the issue and the clear reproduction @VojtaStruhar!
I’ve hit similar cases before and usually what I do is to provide headings I want to display in the sidebar something like this:
---
title: MDX page
---

<Component>

## Heading for component

</Component>

---
// Component.astro
---

  <slot />

More component content

This keeps the heading as Markdown/MDX content so Astro can collect it for the table of contents.
As @HiDeoo mentioned, this is a behaviour in Astro’s collection of page headings in Markdown and MDX and not something we can control in Starlight, so going to close this as beyond Starlight’s control. I still really appreciate the issue — hopefully this workaround helps.
If you have ideas for how to improve Astro’s handling of headings in cases like this, a discussion in the Astro roadmap repo would be a good place to start: https://github.com/withastro/roadmap/discussions/

Thank you for the thorough answer! That's really interesting, that supplying a markdown heading into a MDX component does work.
I appreciate your time


What is URL for solvers?

We haven't "released" the catalogue yet, so projects and test cases are in situ in the setup of the ccp-wsi website i.e.:
https://ccp-wsi.ac.uk/about/linked_projects/
https://ccp-wsi.ac.uk/data_repository/test_cases/
Where do solvers sit?
https://ccp-wsi.ac.uk/catalogue/solvers?
Then at some point we'll move projects and test cases (but setup aliases as well so old links still work) to https://ccp-wsi.ac.uk/catalogue/projects (remove the linked?) and https://ccp-wsi.ac.uk/catalogue/test_cases?

Have changes URLs to from software to solver for now so it's makes more sense but will need to change them again, presumably to https://ccp-wsi.ac.uk/catalogue/solvers  - this will be done when projects and test cases go to that URL as well

Don't forget to update URLS in the various related_tools.html templates

Have temporarily changed all URLS to </catalogue/...> to help demo what the grand plan is in the commit above. I think I've made the changes in all the relevant places so to undo see commit  6e526e430a76612a512c30ed3fbaae71f235049e

Forgot to change URLS in header template (search pages need the header in the centre whereas record pages have the header left aligned)

Emailed Ed about this 2/2/24

In order to best show the potential of the catalogue to the working group we're going to implement it as a catalogue in release 2 (rather than integrating solvers into the website as we did with projects and test cases for release 1).
So, we need:

https://ccp-wsi.ac.uk/catalogue/solvers/
https://ccp-wsi.ac.uk/catalogue/projects/
https://ccp-wsi.ac.uk/catalogue/test-cases/.

We need aliases for the test cases - wordpress will have to do this.
We also need a home page for the release now, see #137

The urls were already catalogue/solvers etc from a previous commit so no further action needed here.


Document new collect_default_jvm_metrics flag for JMXFetch integrations

What does this PR do?
Added docs for collect_default_jvm_metrics. This flag allows users to disable default jvm metrics in JMXFetch
Related PRs:
https://github.com/DataDog/datadog-agent/pull/6923
https://github.com/DataDog/jmxfetch/pull/345
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

Is this option related to collect_default_metrics in the init_config? https://github.com/DataDog/integrations-core/blob/b7221e22b0672bcaac75f2529d0965977fd55bd2/datadog_checks_dev/datadog_checks/dev/tooling/templates/configuration/init_config/jmx.yaml#L7-L12

Is this option related to collect_default_metrics in the init_config?
https://github.com/DataDog/integrations-core/blob/b7221e22b0672bcaac75f2529d0965977fd55bd2/datadog_checks_dev/datadog_checks/dev/tooling/templates/configuration/init_config/jmx.yaml#L7-L12

Nope! This is a different behavior and it is specific to the given instance.


Can't Install Latest Version of Kivy using Python 3.10.0

Software Versions

Python: 3.10.0
OS: Windows 10
Kivy: 2.0.0
Kivy installation method: Git Bash, PYCHARM

Describe the bug
A clear and concise description of what the bug is.
Sure, it won;t install most of the modules
The issue tracker is a tool to address bugs only (search known bugs: https://git.io/vM1iE).
Please use the #support Discord channel at https://chat.kivy.org/ or Stack Overflow for
support questions, more information at https://git.io/vM1yQ.
Expected behavior
Expected to download and install Kivy
To Reproduce
Far too long to reproduce here as the errors run for pages,
Code and Logs and screenshots
Examples of errors:
ERROR: Could not find a version that satisfies the requirement kivy_deps.gstreamer_dev~=0.3.1 (from versions: none)
ERROR: No matching distribution found for kivy_deps.gstreamer_dev~=0.3.1
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?
ERROR: No matching distribution found for kivy[base]
Additional context
Add any other context about the problem here.

Currently supported Python versions are 3.6 - 3.9.

https://github.com/kivy/kivy/releases

Gotcha. Any timeline when you will support 3.10?
On Sun, Oct 10, 2021 at 1:52 AM RobertF @.***> wrote:

Currently supported Python versions are 3.6 - 3.9.
https://github.com/kivy/kivy/releases
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kivy/kivy/issues/7654#issuecomment-939411957, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AB2ZR4HC3CI6A5LTNKNK4XLUGES2TANCNFSM5FVKGLPQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

same issue.. i was scared for a moment and someone told me about this.. ahaha!!!

Our Nightly wheels now support Python 3.10, feel free to check it out :)
See: https://kivy.org/doc/stable/gettingstarted/installation.html#install-kivy

Closing as since Kivy 2.1.0 we now support Python 3.10 🎉


JBTM-3610

This PR is a proposal for a new logic to suspend/shut down Narayana's Recovery Manager

@jhalliday Jonathan can you help us out here? Manuel is proposing that we move graceful shutdown type functionality into ArjunaCore but I am struggling to see the benefits of doing this and I'd appreciate it if you can give us your slant on the proposal.

@jhalliday Jonathan can you help us out here? Manuel is proposing that we move graceful shutdown type functionality into ArjunaCore but I am struggling to see the benefits of doing this and I'd appreciate it if you can give us your slant on the proposal.

Lifecycle management belongs in the server, not the transaction engine. Recovery needs to expose sufficient state/callbacks for the server to determine when its done, but driving it is the server's problem.
BTW, the recovery coordinator's job is not necessarily done when the ObjectStore is empty. Stopping new transactions from being initiated before shutting down the RC is insufficient. You have to wait for in-flight tx to actually complete, which may take up to $timeout, otherwise they may write a log to the store after the RC has terminated. Likewise there may be in-doubt branches in an RM that have no corresponding record in the store. So you have to scan all the RMs and successfully get back an empty in-doubt list from each of them before you can consider that the RC no longer has any responsibilities. So you can't bring down the db conn pool (or http client, for WS) until after recovery stops. But you need the db pool to start before recovery does, so guess which way the module dependency is declared... Life gets even harder if the store contains subordinate records - you can't safely shut down at all until their external parent resolves them, which may never happen. Furthermore, you have to keep enough external endpoints alive for the external parent to make contact to drive that resolution, which can mean e.g. you can't shut down the orb/http/webservices module until after the recovery one. Tx core should not know about any of those, they're the server's problem.

Thank you very much for your message, @jhalliday!

Lifecycle management belongs in the server, not the transaction engine. Recovery needs to expose sufficient state/callbacks for the server to determine when its done, but driving it is the server's problem.

Although this proposal is fairly abstract, what I am proposing to modify (in a nutshell) are the suspend and terminate hooks [1][2] of the Recovery Manager to introduce an extra behaviour: suspending/terminating the Recovery Manager (i.e. its periodic scan) only when there are no transactions left in the Object Store. There will not be any modification to the current API of the Recovery Manager and only few configuration properties will be introduced to turn on/drive the new behaviour.
I have put together a PoC to show what I would like to do (starting from commit 2394324f8274052bccb19b3c0f79ad8710735409)
I thought Narayana was the best place to offer this alternative suspension logic as WFLY is not the only one interested in suspending/terminating the Recovery Manager with this new behaviour.

Stopping new transactions from being initiated before shutting down the RC is insufficient. You have to wait for in-flight tx to actually complete, which may take up to $timeout, otherwise they may write a log to the store after the RC has terminated.

That is a very useful :-) Thanks.

Likewise there may be in-doubt branches in an RM that have no corresponding record in the store. So you have to scan all the RMs and successfully get back an empty in-doubt list from each of them before you can consider that the RC no longer has any responsibilities.

Very interesting. I thought it was not possible to scan RMs once there were no transactions in the Object Store. In this case, I can introduce an extra step in the new suspension/termination of the Recovery Manager to run that scan.

So you can't bring down the db conn pool (or http client, for WS) until after recovery stops. But you need the db pool to start before recovery does, so guess which way the module dependency is declared... Life gets even harder if the store contains subordinate records - you can't safely shut down at all until their external parent resolves them, which may never happen. Furthermore, you have to keep enough external endpoints alive for the external parent to make contact to drive that resolution, which can mean e.g. you can't shut down the orb/http/webservices module until after the recovery one. Tx core should not know about any of those, they're the server's problem.

I agree, all these problems belong to the Application Server (which is the only place where an informed decision can be made).

Lifecycle management belongs in the server, not the transaction engine. Recovery needs to expose sufficient state/callbacks for the server to determine when its done, but driving it is the server's problem.

@jhalliday , I think your answer may have been partially misinterpreted by Manuel since he argues that, in light of your advice, he cannot add clean shutdown logic to narayana and must drive every aspect of shutdown externally. I think this is resulting in a more complex and invasive solution than is strictly necessary, requiring extra SPIs to get at internals and drive recovery externally while checking for a clean store and orphans and waiting for in flight txns - these behaviours need to be under the control of the recovery system. I'm wary of opening it up to be driven externally, potentially bypassing internal checks.
The API to cleanly shutdown the recovery manager should be as simple as setting a property to define which types should be fully recovered before allowing shutdown to complete. The shutdown method would then only move to Mode.TERMINATED when those records have recovered and the XARecoveryModule reports no more orphans (we may possibly need a default method in the RecoveryModule interface for that).
We already have methods to quiesce the TM, TxControl.disable(); or its equivalent, and TransactionReaper.terminate(true); to finish in flight transactions.
The other requirements are handled by WildFly transaction subsystem.

Hmm. I think I'd agree that it's not possible to add clean shutdown logic to narayana, but not that it's necessary to drive every aspect externally. Some aspects I think are best handled as external concerns, particularly with regard to module/service dependency graphs in the app server. Others can be handled internally. Unfortunately, many are in the messy area that may require a bit of both.

The API to cleanly shutdown the recovery manager should be as simple as setting a property to define which types should be fully recovered before allowing shutdown to complete.

What's the use case for wanting to select by type? Or, more generally, by some configurable criteria, because I don't think type is necessarily the whole story here. For example XAResource abstracts away the distinction between remote EJB participants and remote JMS or JDBC participants. If you want to distinguish those cases, knowing that you have an XA transaction is insufficient, you have to know about the implementation behind each of its enlisted resources. You may be able to e.g. bring down http if you have no XTS/LRA tx type instances left. OTOH you can safely shut down the JMS if you only have tx left that use JDBC, but there both those cases have the same tx type, and you're probably in trouble if you happen to have an XAResource whose implementation uses http...
Let's take the 'simple' case of the relationship between an XA connection pool and the recovery manager. From the app server's point of view, these are separate services with their own lifecycles. Can the app server shut down the pool before shutting down the recovery coordinator, or without asking the recovery coordinator if it's ok? Is it the recovery manager's job to shut down the pools when it's done? I think not. How about signalling when it's safe to shut them down? Perhaps.
From the recovery system's perspective, XA recovery plugins (helpers) can be dynamically registered and deregistered. So, if the pool is shut down, the recovery system sees that as the helper being deregistered. Been a while since I read the code, but IIRC if it's mid-pass that will block, otherwise it will succeed. That's not a bad compromise, but it doesn't deal with the case where there are tx in the resource manager that need resolving but the recovery system is idle. Those will be missed and, if the pool never restarts, will never be automatically resolved. For orphans, there won't be any corresponding record in the objectstore, so checking that condition won't help. For owned branches there will, but even if the RC stays active it will never be able to resolve them, because it can no longer talk to the RM. So in the case where the app server shuts down the pool before the RC and the RC won't shut down until the store is empty, the server will never stop. I don't think you can cleanly fix that entirely internally, short of having removeXAResourceRecoveryHelper block for a potentially long time. That helper registration interface then is essentially one of the 'extra SPIs' we'd prefer not to have - it's required to allow the RC to veto or hold up app server state changes based on the RC's internal state. You can flip it around and have the app server query that state instead to determine if it's safe to shut down the pool, but one way or another there is some communication going on somewhere.
This pattern extends outwards to other services - can the app server bring down the http transport? There is no recovery registration for that, so no point as which the RC can delay or veto it, even if it's potentially needed to resolve outstanding tx. The ORB is slightly different, in that (again IIRC) the RC depends on it at the module level, so the dependency graph prevents it going away. But that brings its own issues - as long as a transport is up, new requests can come in on it. TxControl.disable deals with that to an extent - new requests trying to start a tx will fail, but existing tx operations will complete. Even that breaks a bit where e.g. a compensation step for an existing saga type tx recovery tries to start a subordinate JTA tx. Essentially we (both at the tx core and more generally the server transport or cluster routing levels) can't tell when a request is totally new vs. a substep of something already inflight/recovering. Without the ability to do that, there will inevitably be cases where some tx can never complete, or we can never shut things down safely.
Going back to the idea that the API to cleanly shutdown the recovery manager should support some notion of describing what conditions should be achieved first, I think that is right, but it's probably some combination of not just tx type but also time passed, things tried / states achieved, and rules of thumb (I won't call them heuristics, because that's already something else, which incidentally may also be a factor - can a non-empty store containing only heuristic outcomes be shut down?) But at the end of the day that starts to look a lot like a policy plugin SPI that allows a rule, supplied by the user/appserver, to examine some internal state and make a decision based on it. Anything less and you're instead hardcoding that policy into scattered points in the tx code such as rc.shutdown, TxControl.disable and the removeXAResourceRecoveryHelper method, which is not necessarily worse but may make the behaviour of the app server's service dependency graph harder to reason about since the policy then will have a more scattered and subtle manifestation in the app server code.

Thanks Michael for your message.

I think your answer may have been partially misinterpreted by Manuel since he argues that, in light of your advice, he cannot add clean shutdown logic to narayana and must drive every aspect of shutdown externally.

@jhalliday’s reply in this thread was analysed during a meeting where the Narayana team discussed the next steps to implement (the concept behind) JBTM-3610 according to Jonathan’s suggestion. Following, the summary of the meeting:
JBTM-3610 was rejected as the graceful shutdown of Narayana should be guided from integrating parties (e.g. WildFly)
Narayana should provide a switch to stop the ability to create new transaction before its shutdown (which should be called by the integrating party whenever the creation of new transactions is no longer necessary)
Narayana should provide an API to inform the integrating party that it needs to wait for completion of in-flight transactions
Narayana should provide an API to inform the integrating party that it needs to wait for in-doubt transaction to complete recovery
Heuristic transactions should be solved with human intervention

I think this is resulting in a more complex and invasive solution than is strictly necessary, requiring extra SPIs to get at internals and drive recovery externally while checking for a clean store and orphans and waiting for in flight txns - these behaviours need to be under the control of the recovery system. I'm wary of opening it up to be driven externally, potentially bypassing internal checks.

Regarding the (new) PoC I put together (i.e. not the one linked in this thread), I don’t think we can say that it is an invasive solution. In fact, all it does is simply expose information from Narayana to the integrating party to allow the latter to call the shutdown hook of Narayana at the right moment. In other words, the shutdown of Narayana will not be driven externally but the integrating party will have the ability to fetch all the relevant information from Naryana to make an “educated” choice when to call the Narayana shutdown hook. Basically, my PoC does not modify the logic and internal checks of Narayana. What it does is expose the information that the Narayana team decided to expose (please, see the report of our meeting above) to allow the integrating party to call the Narayana shutdown hook at the right moment.

The API to cleanly shutdown the recovery manager should be as simple as setting a property to define which types should be fully recovered before allowing shutdown to complete. The shutdown method would then only move to Mode.TERMINATED when those records have recovered and the XARecoveryModule reports no more orphans (we may possibly need a default method in the RecoveryModule interface for that).
We already have methods to quiesce the TM, TxControl.disable(); or its equivalent, and TransactionReaper.terminate(true); to finish in flight transactions.

Conceptually, this is similar to the code I proposed last year (albeit more rounded) and that was rejected by the Narayana team.

The other requirements are handled by WildFly transaction subsystem.

Yeah, I have my doubts about that. Not saying that other subsystems in WildFly do not care about transactions but I think they just arrive at a certain point (which usually is in-flight transactions). WildFly is then responsible to know when is the best moment to call the shutdown hook of Narayana.

Jonathan, Manuel's proposal is for WildFly so the module/service dependency graphs ought to be correct, which sidesteps your points about the order in which subsystems are shutdown; recovery stops when WildFly stops the service, ie we're not proposing that anything other than the app server is in control of the shutdown.
I proposed clean shutdown by type in order to include the types corresponding to the recovery modules which are installed by WildFly so those will be known to the transaction subsystem, this is to avoid having to change every module to force them to report what they have left to recover, I believe the existing solution and certainly Manuel's proposed new one both wait for each recovery module to report that their part of the store is empty so I took their lead but restricted it by type, regarding that as a cleaner solution since the equivalent behaviour can be realised generically via the ObjectStoreAPI during shutdown. Although knowing the types might not be the whole story, as you say, the WildFly transaction subsystem should know, and if it does not then that would be something we would need to expose.
Your last paragraph is interesting and the points need to be considered, but I am not proposing a change to WildFly here, I'm just asking that shutting down the subsystem should make a best effort attempt at ensuring the store is clean since we know the server is not coming back again (unless it crashes) - this solution was proposed for pod scale down and I've been told a number of times that we cannot guarantee a server will be restarted if its pod is scaled down with a dirty object store and that platforms need to apply ad hoc/manual solutions, or veto scale down, to recover anything left dangling (remark, I'm told that our target platforms cannot always guarantee that scaled down pods with dirty stores will be restarted, it's only if they crash that they'll be restarted).

Hmm. So if I'm following this right, the idea of passing a list of types to the recovery shutdown is to deal with the case where there are records of some other type in the store that will never recover, because no recovery module is configured for them, therefore making 'the store is empty' too crude of a check?  But the recovery system already knows what modules it's got configured, isn't giving it a list of types just repeating that information? Can it not internally derive the same from walking the list of things in the store against the list of modules and seeing which ones remain unclaimed at the end?
I'm definitely on board with the idea that finishing recovery is a somewhat fuzzy decision. At some point the RC is going to have decide it can't reasonably do any more and it's time to quit. Realising (or being told) that there is nothing it can do to remove some types of tx from the store is part of that. Realising that even for types it can do, there are some instances of that type it can't is also a thing - for example, heuristic outcomes are outside the scope of recovery and can only manually be removed/resolved anyhow. Likewise tx where it can't talk to the RM - though there it has to decide if the communication problem is temporary or permanent. Reasoning over known unknowns is also a thing - e.g. has it scanned all RMs for orphans? The open question in my mind is, to what extent is it desirable to make this definition of 'best effort' flexible, or allow it to externally determined?
The 'this is for Wildfly' part is interesting too. In my mind that's a clear indication the code for it would live on the Wildfly side, not in narayana. But Wildfly's dependency model isn't nuanced enough to deal with the relationships in play here. It wants to be able to dynamically deploy datasources, which means it can't statically declare that the recovery system depends on them, even though it really does. So in that respect the dependency graph is deliberately wrong and that's a feature (or at least enables a feature - dynamically adding ds) and not a bug. Likewise the circular dependency between the web stack: servlets can do transactions, so web depends on tx, but transactions can have http participants, so tx depends on web. But the dependency system doesn't accurately reflect that. SPIs like dynamic registration/deregistration of recovery helpers are to pave over these problems, but are not well integrated with Wildfly's lifecycle model, and fixing that feels a bit out of scope?
I wonder if a reasonable first cut here may be to just reach a point where recovery shutdown can print accurate status information? For example 'INFO: Terminating with no known outstanding transactions" vs. "WARN: $n known unresolved transactions: "+list vs. "WARN: There MAY BE unresolved transactions in "+datasource.  The data gathering and reasoning needed to allow printing things like that would be the same that  is needed to change the shutdown policy, i.e. you could later replace one of the logging statements with a handler to deal with that specific case. (echoes of how CheckedAction 'aborting with multiple thread active' works) It would also serve to export at least some relevant state info from the soon to be erased objectstore to a log that (usually) is permanently stored off-node. For example, heuristic transactions with some participant id info. That would allow at least some basic after the fact tx identification and manual resolution to take place if it turns out the node went away when it ideally shouldn't have because the shutdown policy it's using doesn't quite align with what the user wanted/expected.

Hmm. So if I'm following this right, the idea of passing a list of types to the recovery shutdown is to deal with the case where there are records of some other type in the store that will never recover, because no recovery module is configured for them, therefore making 'the store is empty' too crude of a check? But the recovery system already knows what modules it's got configured, isn't giving it a list of types just repeating that information? Can it not internally derive the same from walking the list of things in the store against the list of modules and seeing which ones remain unclaimed at the end?

Inferring which types to recover based on which recovery modules get loaded is a good suggestion. Manuel, well need to add an internal compile time map of modules to the types they manage, similar to, but simpler than, the way we internally register object store record types (see Implementations.java, RecordTypeManager.java and RecordType.java, maybe we could even augment one of those, not sure).

I'm definitely on board with the idea that finishing recovery is a somewhat fuzzy decision. At some point the RC is going to have decide it can't reasonably do any more and it's time to quit. Realising (or being told) that there is nothing it can do to remove some types of tx from the store is part of that.

Force terminating the RC is an option here with a dump of the state, similar to your final point.

... Realising that even for types it can do, there are some instances of that type it can't is also a thing - for example, heuristic outcomes are outside the scope of recovery and can only manually be removed/resolved anyhow. Likewise tx where it can't talk to the RM - though there it has to decide if the communication problem is temporary or permanent.

We need a management API that quiesces the RC and TM, externalises the remaining state and stops.

... Reasoning over known unknowns is also a thing - e.g. has it scanned all RMs for orphans? The open question in my mind is, to what extent is it desirable to make this definition of 'best effort' flexible, or allow it to externally determined?

If we add the API I just mentioned then the platform can control how to stop with in doubt records and avoid trying to recover forever.

The 'this is for Wildfly' part is interesting too. In my mind that's a clear indication the code for it would live on the Wildfly side, not in narayana. But Wildfly's dependency model isn't nuanced enough to deal with the relationships in play here.

And the code does live in WildFy, I was referring to the dependency graph which needs to be managed by the server, and I think we already do a little to help facilitate that; basically we expose start and stop for the TM and RC.
Similarly for other platforms, see my next response as well. But I hear your point, perhaps there is something the RC can do to make up for the coarseness of the dependency model, but let's pursue that in a later iteration.

... It wants to be able to dynamically deploy datasources, which means it can't statically declare that the recovery system depends on them, even though it really does. So in that respect the dependency graph is deliberately wrong and that's a feature (or at least enables a feature - dynamically adding ds) and not a bug. Likewise the circular dependency between the web stack: servlets can do transactions, so web depends on tx, but transactions can have http participants, so tx depends on web. But the dependency system doesn't accurately reflect that. SPIs like dynamic registration/deregistration of recovery helpers are to pave over these problems, but are not well integrated with Wildfly's lifecycle model, and fixing that feels a bit out of scope?

This is part of a more comprehensive solution for supporting different platforms, again let's keep this advice for the next iteration of the design.

I wonder if a reasonable first cut here may be to just reach a point where recovery shutdown can print accurate status information? For example 'INFO: Terminating with no known outstanding transactions" vs. "WARN: $n known unresolved transactions: "+list vs. "WARN: There MAY BE unresolved transactions in "+datasource. The data gathering and reasoning needed to allow printing things like that would be the same that is needed to change the shutdown policy, i.e. you could later replace one of the logging statements with a handler to deal with that specific case. (echoes of how CheckedAction 'aborting with multiple thread active' works) It would also serve to export at least some relevant state info from the soon to be erased objectstore to a log that (usually) is permanently stored off-node. For example, heuristic transactions with some participant id info. That would allow at least some basic after the fact tx identification and manual resolution to take place if it turns out the node went away when it ideally shouldn't have because the shutdown policy it's using doesn't quite align with what the user wanted/expected.

I think this is a good compromise, especially given that the implementation requirements and design are still being debated. Let's provide a best effort shutdown(clean) policy to minimise how often we have to deal with "dirty shutdowns" and ensure that we externalise the remaining state so that admins can reason about and make decisions on what to do about that state.
Remark: As a general comment, we need to minimise what we expose: the design I've been encouraged to accept is adding SPIs to:

disable the transaction system;
running single recovery scans (ie single stepping from outside the RC);
asking the RC how many logs it is recovering, and how many orphans, and how long such information remains valid for;
pausing the RC and handing over contol of to the platform (which single steps recovery until the RC reports to the user that a condition has occurred);

Exposing this much SPI would need to first undergo community testing, i.e. it's out of the time frame for the needing the solution. Besides, in view of the above discussion with Jonathan, I don't believe it is justified.


i18n(ko-KR): update mdx.mdx

Description (required)

update mdx.mdx
#9027

Related issues & labels (optional)

Closes #
Suggested label:

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

ko
guides/integrations-guide/mdx.mdx
Localization changed, will be marked as complete.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.