Editable installs are broken after __editable__ and __path_hook__ changes

Description
I am starting to observe broken editable installs since past week.
Specifically, editable install now creates a __editable__.<pkg>.pth and __editable___<pkg>_finder.py files.  These paths are returned as part of namespace.__path__, which has broken a lot of tooling.

pdoc3 went broke.  Skipping __editable__ package seems to fix (hack) it.

See https://github.com/pdoc3/pdoc/pull/408

We also noticed, namespace packages that installs a CLI are now also broken, because while the binary is under .venv/bin, it is unable to import the modules since it was installed using -e.

Modifying PYTHONPATH is the only option.

We also noticed, due to these changes even VSCode/Pylance is now broken, as it is unable to resolve paths to editable installed packages.

Adding "python.analysis.extraPaths" to .vscode/settings.json is the way out

I was unable to find any search result on Google for __editable__ and __path_hook__.  So, I am unsure what I am up against here.
Expected behavior

Installed CLI from namespace package must work fine without modification of PYTHONPATH
namespace.__path__ must return actual path instead of __editable__.* which seems to break a lot of existing tooling

pip version
22.2.2
Python version
3.10.5
OS
MacOS 12.5
How to Reproduce

Create a namespace package that installs CLI (entry point)
Install the package with -e
Installed CLI is broken due to missing PYTHONPATH

Our projects are using pyproject.toml and setup.cfg.
Output
`ImportError: cannot import name 'entry_point' from 'namespace.cli' (unknown location)`

Code of Conduct

[X] I agree to follow the PSF Code of Conduct.

I can actually transfer the issue, so… here you go. :)

It looks like we started to have the same problem in Airflow.

pip install -e . after checking out basically any version (including main) and running any Airlfow cli command results in

Traceback (most recent call last):
  File "/home/jarek/.pyenv/versions/airflow-test-cli3/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/jarek/code/airflow/airflow/__main__.py", line 38, in main
    args.func(args)
  File "/home/jarek/code/airflow/airflow/cli/cli_parser.py", line 50, in command
    func = import_string(import_path)
  File "/home/jarek/code/airflow/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/home/jarek/.pyenv/versions/3.9.13/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/jarek/code/airflow/airflow/cli/commands/standalone_command.py", line 35, in <module>
    from airflow.www.app import cached_app
  File "/home/jarek/code/airflow/airflow/www/app.py", line 43, in <module>
    from airflow.www.extensions.init_views import (
  File "/home/jarek/code/airflow/airflow/www/extensions/init_views.py", line 28, in <module>
    from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
  File "/home/jarek/code/airflow/airflow/www/views.py", line 685, in <module>
    class AirflowBaseView(BaseView):
  File "/home/jarek/code/airflow/airflow/www/views.py", line 699, in AirflowBaseView
    extra_args['sqlite_warning'] = settings.engine.dialect.name == 'sqlite'
AttributeError: module 'airflow.settings' has no attribute 'engine'

This started to happen few days ago and adding Airlfow's source to PYTHONPATH solves the problem.
I did some bisecting and pin-pointed it to version 64.0.0 of setuptools. Version 63.4.3 worked perfectly fine, 64.0.0 breaks it. The reason seems to be as described in the issue.  I've added those lines to our pyproject.toml and they solved the problem:
[build-system]
requires = ['setuptools==63.4.3']
build-backend = "setuptools.build_meta"

See: https://github.com/apache/airflow/pull/25848
It is super-easy to reproduce now:

checkout main from https://github.com/apache/airflow
create and activate a new virtualenv
run pip install -e .
run airflow standalone

This works perfectly well (because of the 63.4.3 limit)

change the line in pyproject.toml to:

requires = ['setuptools==64.0.0']

create and activate a new virtualenv
run pip install -e .
run airflow standalone

You see the stack-trace above

(The error is also present in other 64.* and 65* versions BTW).

Hi @abhinavsingh,
Could you provide a minimal reproducer explaining more in details the issues you are having? (Maybe it is worthy to split multiple issues in multiple reproducers).
I tried to investigate what you are describing with the following reproducer, but I was not able to find the problematic behaviour you are referring to... (Maybe I understood something wrong, or maybe the reproducer is missing something??).
# docker run --rm -it python:3.10.6 bash

cd /tmp && rm -rf /tmp/proj_root
mkdir -p /tmp/proj_root
mkdir -p /tmp/proj_root/namespace/cli
echo "def run(): print('hello world')" > /tmp/proj_root/namespace/cli/__init__.py
cat <<EOF > /tmp/proj_root/pyproject.toml
[build-system]
requires = ["setuptools>=65.1.0"]
build-backend = "setuptools.build_meta"

[project]
name = "namespace.cli"
version = "42"

[project.scripts]
namespace-cli = "namespace.cli:run"
EOF
cd /tmp/proj_root
python3.10 -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
cd /var
/tmp/proj_root/.venv/bin/python -c 'import namespace; print(f"{namespace.__path__=}")'
# ==> namespace.__path__=_NamespacePath(['/tmp/proj_root/namespace'])
/tmp/proj_root/.venv/bin/namespace-cli
# ==> hello world

As you can see above, the reproducer show that whenever possible setuptools will add the corresponding path entry to __path__.
Cases when this does not occur are likely associated with configurations by which users explicitly opt out from certain packages or there is simply no directory in the file system for the package to be associated with...

There are a few other remarks I would like to make:

In Python, path entries do no need to be existing locations in the file system, this is mentioned in Python docs:

Path entries need not be limited to file system locations. They can refer to URLs, database queries, or any other location that can be specified as a string.
...
Most path entries name locations in the file system, but they need not be limited to this.
...
Entries in sys.path can name directories on the file system, zip files, and potentially other “locations” (see the site module) that should be searched for modules, such as URLs, or database queries.

We can also demonstrate this with the following snippet:
# docker run --rm -it python:3.10.6 bash
python3.10 -c 'import sys; print(sys.path)'
# ==> ['', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages']
ls -la /usr/local/lib/python310.zip
# ==> ls: cannot access '/usr/local/lib/python310.zip': No such file or directory

Most static analysis tools (including the ones used in code editors) have limitations in terms of aspects of the Python language they cannot handle. Import hooks (which have been part of the Python language specification for years, and are listed in PEP 660 as a perfectly viable implementation for editable installs), are notably one of these limitations. This is reported in https://github.com/pypa/setuptools/issues/3518.
Setuptools understand this limitation, and I am personally happy to contribute to an effort of improving this. However we do need someone to champion a design/architecture/standard that is well received in both packaging and static analysis community.

I would also recommend setuptools' docs on editable installs. There might be some useful information there.

@potiuk I suspect airflow's use case is different.
By having a look on the setup.py, it seems that airflow implements a custom develop command.
Setuptools's implementation of PEP 660 does not use the develop command. Instead, setuptools now rely in a complete different command. Please have a look on the docs for a few possible options on how to customise the editable installation.

By running:
# docker run --rm -it python:3.10.6 bash

apt update && apt install -y git build-essential
cd /tmp
git clone https://github.com/apache/airflow
cd /tmp/airflow
sed -i 's/==63.4.3/>=65.1.0/g' pyproject.toml
python -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
cd /var
/tmp/airflow/.venv/bin/python -c 'import airflow.settings; print(dir(airflow.settings))'
# ==> ['AIRFLOW_HOME', 'AIRFLOW_MOVED_TABLE_PREFIX', 'ALLOW_FUTURE_EXEC_DATES', 'CAN_FORK', 'CHECK_SLAS', 'COMPRESS_SERIALIZED_DAGS', 'Callable', 'DAEMON_UMASK', 'DAGS_FOLDER', 'DASHBOARD_UIALERTS', 'DEFAULT_ENGINE_ARGS', 'DONOT_MODIFY_HANDLERS', 'EXECUTE_TASKS_NEW_PYTHON_INTERPRETER', 'Engine', 'GUNICORN_WORKER_READY_PREFIX', 'HEADER', 'HIDE_SENSITIVE_VAR_CONN_FIELDS', 'IS_K8S_OR_K8SCELERY_EXECUTOR', 'KILOBYTE', 'LAZY_LOAD_PLUGINS', 'LAZY_LOAD_PROVIDERS', 'LOGGING_CLASS_PATH', 'LOGGING_LEVEL', 'LOG_FORMAT', 'List', 'MASK_SECRETS_IN_LOGS', 'MEGABYTE', 'MIN_SERIALIZED_DAG_FETCH_INTERVAL', 'MIN_SERIALIZED_DAG_UPDATE_INTERVAL', 'NullPool', 'Optional', 'PLUGINS_FOLDER', 'SASession', 'SIMPLE_LOG_FORMAT', 'SQL_ALCHEMY_CONN', 'STATE_COLORS', 'TIMEZONE', 'TYPE_CHECKING', 'USE_JOB_SCHEDULE', 'Union', 'WEBSERVER_CONFIG', 'WEB_COLORS', '__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_get_rich_console', 'atexit', 'conf', 'configure_action_logging', 'configure_adapters', 'configure_logging', 'configure_orm', 'configure_vars', 'create_engine', 'custom_show_warning', 'dag_policy', 'dispose_orm', 'exc', 'executor_constants', 'functools', 'get_airflow_context_vars', 'get_dagbag_import_timeout', 'get_session_lifetime_config', 'import_local_settings', 'initialize', 'json', 'log', 'logging', 'original_show_warning', 'os', 'pendulum', 'pod_mutation_hook', 'prepare_engine_args', 'prepare_syspath', 'reconfigure_orm', 'replace_showwarning', 'scoped_session', 'sessionmaker', 'setup_event_handlers', 'sqlalchemy', 'sys', 'task_instance_mutation_hook', 'task_policy', 'tz', 'validate_session', 'warnings']

I can see that the package is indeed installed in the editable mode, however the engine attribute does not seem to be initialised. Maybe this is related to the customisations you guys are doing in the develop command, but that are no longer invoked when you run pip install -e .?

By having a look on the setup.py, it seems that airflow implements a custom develop command.

Thanks for the pointer @abravalheri . I will take a closer look and see if I can work it out. I was kinda suspecting that we could have been doing something that PEP 660 does not really like :). I will keep you posted.

Time to do some more deep PEP reading

@potiuk Hey! Airflow's setup.py looks quite complex which intrigues me. Would you be interested in me opening a PR for a PoC switch to Hatchling (which will fix your issue)?

(Copying some opinion from the PyPA discord for eventual readers of this thread).
setup.py implementations tend to grow complex with time. Any backend that supports extensions can be used to refactor (which includes setuptools). Documentation on custom build steps can be found at https://setuptools.pypa.io/en/latest/userguide/extension.html.

@potiuk Hey! Airflow's setup.py looks quite complex which intrigues me. Would you be interested in me opening a PR for a PoC switch to Hatchling (which will fix your issue)?

@ofek  Airlfow is an ASF project and community makes decisions not me personally. If you want to make a POC and have some good arguments, you will have to start dicsussion at the airflow devlist https://airflow.apache.org/community/ . But you have to - in general - have a good reasoning and justification for such a change in build system. just fixing an error that we can fix by not switching to another build mechanism is not good enough reason.

Just a note here to say pkg_utlts.iter_modules() from the stdlib is also affected by this. It no longer returns the editable pacakges like it used to. Is this something that would have to be fixed in the stdlib ?

Hi @lonetwin, pkgutil.iter_modules does not cover the same spectrum as importlib machinery: it is much more limited.
Based on this comment I would say that (so far) the stdlib developers have no intention the increase the scope or add features to pkgutil. The phrase used was: pkgutil is on its way out as soon as people have the time to deprecate it.
If you absolutely need to use pkgutil you can try to opt into a different editable installation mode (e.g. pip install -e . --config-settings editable_mode=strict) -- I haven't tested it myself, so I am not sure it will work.

In the spirit of over-communicating: I'm unsubscribing from this since I don't think there's anything for me to do here. To the maintainers, please feel free to @-mention me in case there's something here that's on pip's end.

@abravalheri Apologies, haven't checked back on GitHub in weeks.  Missed your message altogether.  A point to note in our case is that, we have all our packages under a namespace.  As others have figured, v64 is an issue.  At our end, I had to pin setup tools to v62 to let our company workflows pass.  In our setup, we simply go by paths returned by namespace.__path__.   But ever since v64, we realise, __path__ contains entries which are actually not real file paths.  They needs to be resolved using latest mechanism adopted by setup tools.  The day I ran into it, I wasn't even able to find any formal documentation around it.
I am not on my work system today.  But, I'll give you a reproducible scenario as soon as I get to my system.  We surely don't want to pin ourself to v62 forever.

Hey @abravalheri ,
Our repo has the same issue with editable mode (for us as well the issue appears in version setuptools==64). We have an interesting case, that one of the artifacts our app creates is a folder with the same name as our package (i know :/).
To reproduce you can run this:
# docker run --rm -it python:3.10.6 bash

cd /tmp
git clone https://github.com/wandb/wandb.git
cd /tmp/wandb
git checkout a21d2622dab04ace91d76ba05b1631fff9c891e
python -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .

cd /var
#first time running the script
/tmp/wandb/.venv/bin/python -c "import wandb; run = wandb.init(mode='offline'); run.finish()"
#second time running the script
/tmp/wandb/.venv/bin/python -c "import wandb; run = wandb.init(mode='offline'); run.finish()"

note that after the first execution of our script we create a folder with the same name as our project (so the first execution works). However, the second execution of script fails, since the importer "finds" the package in the current working directory i.e. the newly created folder.
Thanks for your help!

I'm running into a related issue, where after installing -e ./foo (containing ./foo/setup.py, ./foo/foo.py), somehow import foo gives me a namespace package for ./foo rather than the module ./foo/foo.py. This is a change in behavior that breaks various CI and seems extremely weird; why isn't an installed package preferred over an implicit namespace package?
I can pin setuptools 65 but this seems like a bug on your side.

Have you tried Hatchling?

@abravalheri I apologise for missing out on it.   I checked our codebase today and we are still using "setuptools <= 62.6.0", which is not good in the long run (setuptools is now on v67).  Likely I simply missed the notifications but today's activity has brought me back here.
I really want to get past this pinned setuptools issue in our codebase.  Will investigate back into it over the coming weekend and come up with a minimal example to reproduce the issue at your end.

hi team, just checking my luck, by any chance, do we have an update on this issue? I am still facing issues with editable installs in a local virtualenv for Apache Airflow.

Hi @pankajkoti to keep investigating this issue we need a reproducer of the original problem.
I reported some thoughts in https://github.com/pypa/setuptools/issues/3548#issuecomment-1221366695 (please note the remarks about entries in sys.path).

note that after the first execution of our script we create a folder with the same name as our project (so the first execution works). However, the second execution of the script fails, since the importer "finds" the package in the current working directory i.e. the newly created folder.

Hi @kptkin, in our docs we list the limitation of coinciding names of file entries in the working directory. If you need to support that use case you can try other installation modes, e.g. pip install -e . --config-settings editable_mode=strict.

note that after the first execution of our script we create a folder with the same name as our project (so the first execution works). However, the second execution of the script fails, since the importer "finds" the package in the current working directory i.e. the newly created folder.

Hi @kptkin, in our docs we list the limitation of coinciding names of file entries in the working directory. If you need to support that use case you can try other installation modes, e.g. pip install -e . --config-settings editable_mode=strict.

Ah thanks for the reply! Yeah, I found this exact flag in the docs, good to have a verification that this is the recommended approach and it won't be something that is/should be fixed.

Just ran into this issue after updating to python 3.12, should I just revert to 3.11 and wait or is there a different approach I should take?

If you're affected by this issue to the point where even editable_mode=strict won't work then the Legacy Behavior can still be enabled as a temporary workaround:
pip install -e . --config-settings editable_mode=compat

This is not a long term solution. If any of your tools requires editable_mode= to work correctly then it should probably be reported as an issue to that tools repo.

@HexDecimal What if it's our tool having the issue? How should it be updated to conform to editable installs in python 3.12?

This issue is not caused by your Python version. It is caused by installing setuptools packages in editable mode using the latest version of setuptools.
I've only experienced this issue from the perspective of a tool user (VSCode/Pylance) so I'm not sure what to do on the development side.

For what it's worth, installing packages using pip install --editable . was adding significant (in this case 0.225s) lag to Python startup time. Happy to provide more information / a reproducer if that's useful (time spent was split across importlib.abc, subprocess, ast, and json in that order). I was a little disappointed that starting Python now required launching a new subprocess, but maybe I misunderstood (or misunderstand why that's necessary).

@rachtsingh performance degradation using editable mode can be caused by a myriad of elements many of them not under the control of setuptools (e.g. depending on how the particular package you are installing is implemented, which is the layout of that package, how many items you have in the .pth file, etc...).
Setuptools follows the implementation options described in https://peps.python.org/pep-0660/#what-to-put-in-the-wheel, and in general the performance hits associated with these implementation options are considered acceptable (the purpose of the editable mode is to help in development time and not in production).
If you manage to investigate more and isolate a specific behaviour in setuptools that you think can be optimised, PRs would be very welcome.

I'm seeing an issue that might be related to this, for reproduction I've created this Dockerfile
To me this shows that after installation the namespace Package Products is initialized i think correctly, i.e. it contains the ZMS package, but after the first time pkgutil.iter_modules() is called that is gone.
So, the package Products.zms is not part of the output, even though it is correctly installed and importable. Why is that? Am I missing something about how installation with setuptools is supposed to work?
Further if I install Zope (as of yet a package that still does not have a pyproject.toml) in editable mode beforehand everything works as expected. I think because this forces a legacy mode in pip/setuptools?
Still this smells fishy enough that I would like some advice if this is a problem in setuptools, or python - or just in my code (still the most likely scenario I think).

@dwt pkgutil.iter_modules() docs say:

Note: Only works for a finder which defines an iter_modules() method. This interface is non-standard, so the module also provides implementations for importlib.machinery.FileFinder and zipimport.zipimporter.

Maybe it is related to that?

@abravalheri How do you read that notice in the docs? At first I read this as: "the module pkgutil.iter_modules() also provides implementation for importlib.machinery.FileFinder and zipimport.zipimporter". This could make some sense if it actually meant that only import lib provides the machinery to iterate over modules?
That leaves behind the question: How am I supposed to iterate over submodules of a namespace package if they are installed with setuptools in the new mode? pkgutil.iter_modules() seems to be what the python packaging documentation seems to recommend, so I gathered it should work?

My interpretation of that passage in the Python docs is: pkgutil.iter_modules() relies on the implementation of a non-standard iter_modules() method. Then it special cases the implementation to handle importlib.machinery.FileFinder and zipimport.zipimporter even when these don't implement iter_modules().

How am I supposed to iterate over submodules of a namespace package if they are installed with setuptools in the new mode?

This might be a good question for CPython?

If you would like to propose a PR in setuptools to implement this non standard iter_modules(), that would be an option to consider (although I would like to avoid increasing the complexity of the editable finder -- which is already very high).
Another possibility is to always use src-layouts for the projects that need to be discovered that way, OR install them using --config-settings editable_mode=strict/compat.
The good part ("half-full glass point of view") is that editable installs are not really meant to be used in production, only for development. So in production you should not reach these code paths as packages are installed regularly.

I assumed that setuptools has a some test cases that show that editable installs work. Can someone point to me to where they are? I think a pull request that adds an editable install for a namespace package and then follows the recommended way to iterate it seems like a sensible first step?

My interpretation of that passage in the Python docs is: pkgutil.iter_modules() relies on the implementation of a non-standard iter_modules() method. Then it special cases the implementation to handle importlib.machinery.FileFinder and zipimport.zipimporter even when these don't implement iter_modules().

How am I supposed to iterate over submodules of a namespace package if they are installed with setuptools in the new mode?

This might be a good question for CPython?

Uhm... shouldn't the question be to setuptools? I mean, the documented way is right there, linked above?
If not, where would I ask this? Also, it may be way stronger a question if you or one of the setuptools contributors asks this, as I may just be ignored.

If you would like to propose a PR in setuptools to implement this non standard iter_modules(), that would be an option to consider (although I would like to avoid increasing the complexity of the editable finder -- which is already very high).

I would like to help review this, but I feel unable to judge wether that is the best way forward?

Another possibility is to always use src-layouts for the projects that need to be discovered that way, OR install them using --config-settings editable_mode=strict/compat.

I thought that my project does have the source layout. How would I check this?

The good part ("glass half-full" point of view) is that editable installs are not really meant to be used in production, only for development. So in production you should not reach these code paths as packages are installed regularly.

While that is true, they are also essential to (my) development workflow, either local or in docker containers, to install the module under development into a virtualenv and work on it there.

Uhm... shouldn't the question be to setuptools?

pkgutil is not maintained in the setuptools repo, so if you have a question on how that works, probably you are better speaking with the maintainers of that module. The "importlib machinery" in Python import is equally not maintained by setuptoools.
Setuptools itself also does not expose any APIs (or at least not that have not been deprecated) for iterating over namespaces.
So maybe you are looking for a feature that does not exist. If pkgutil.iter_modules implementation relies entirely on non-standard methods and special-casing, it is probably best to not take it as fully reliable/bulletproof. It likely works "to some extent" but does not fully interoperate with other importlib machinery standards, being provided "as is".
The best discovery mechanism known to work with third-party packages is entry-points, although even that has its limitations with editable installs (i.e. newly added entry-point need a full re-install).

I thought that my project does have the source layout. How would I check this?

This is setuptools' description of what a src-layout entails: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#src-layout.
Note that if you "remap" or "customise" package directories via the package-dir configuration in setuptools, the repository no longer will be considered under a src-layout.
When installed in editable mode, a src-layout project will add a simple and static .pth file into the site-packages directory, simply augmenting sys.path and leaving Python's "native import machinery" to do the rest. This provides the best interoperability.

While that is true, they are also essential to (my) development workflow, either local or in docker containers, to install the module under development into a virtualenv and work on it there.

That is a fair comment, but it is also fair to recognise that editable installations are not a silver bullet and have many problems and limitations. This is made explicit in the setuptools docs.

I would like to help review this, but I feel unable to judge wether that is the best way forward?

In my option, the best way forward if you would like this feature to be implemented would be to create a PR (ideally not a complex change). A good place to start would be having a look on the docs about Python's importlib machinery and PEP 660. After that it is nice to have a look on https://setuptools.pypa.io/en/latest/userguide/development_mode.html and https://setuptools.pypa.io/en/latest/userguide/extension.html. Finally you can read through the implementation in https://github.com/pypa/setuptools/blob/main/setuptools/command/editable_wheel.py where the changes would need to be made.
Please note however that this territory is "muddy" and that there are not many answers out there for the problems of editable installs and namespace packages:

https://github.com/pypa/packaging-problems/issues/588
https://github.com/python/cpython/issues/92054

Note that it was also very difficult to have a general agreement in terms of standards for editable installs, and one of the main issues is that there is no solution that solves all problems:

https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450
https://discuss.python.org/t/pep-660-editable-installs-for-pep-517-style-build-backends/8510
https://discuss.python.org/t/third-try-on-editable-installs/3986
https://discuss.python.org/t/specification-of-editable-installation/1564
https://discuss.python.org/t/standardising-editable-mode-installs-runtime-layout-not-hooks/4098

In general, the understanding is that editable installs are inherently "flawed", because it is currently impossible to have something to behave at the same time as if it was installed and in development at the same time, without constantly reinstalling it.
Since the mileage may vary according to project layouts, usage of namespaces, etc..., users may have to choose the best installation procedure and/or project layout and compromise in a case by case basis.

That... sounds like a lot of work. I was hoping that we could agree that this is a backwards incompatibility issue. I.e. in the past editable installs did work even for namespace packages with setuptools, and thus this should continue to work.
If I understand you correctly, adding pull request here for itermodules() could solve that particular problem?
Just so I'm not seeing things: pkgutil is part of the python standard library, right? So that should make it the authority on how to  iterate over submodules?
In this case, I cannot choose a different way to install, as the Zope-Maintainers (in their infinite wisdom) have decided that they require namespace packages for extensions. :-/

That... sounds like a lot of work. I was hoping that we could agree that this is a backwards incompatibility issue. I.e. in the past editable installs did work even for namespace packages with setuptools, and thus this should continue to work.

I don't think we can apply the concept of backwards incompatibility in this scenario because the brutal change in behaviour is motivated by a PEP, which basically made the exact previous implementation no longer viable. Also the previous behaviour and API still exists in setuptools, unchanged and bug-by-bug backwards compatible (although deprecated). Installers no longer call that API by default, though.

Just so I'm not seeing things: pkgutil is part of the python standard library, right? So that should make it the authority on how to iterate over submodules?

I don't think there is a standard about iterating over submodules. The pkgutil implementation clearly documents its "non-standardness" and clearly says it "only works" in some scenarios.

In this case, I cannot choose a different way to install, as the Zope-Maintainers (in their infinite wisdom) have decided that they require namespace packages for extensions. :-/

If you are writing a package, you can still chose its directory structure right? So you can use a src-layout in that scenario. Also you are the one installing the packages, right? So I suppose you can also choose with arguments to pass to pip install (which can include --config-settings editable_mode=..."). That might be your escape hatch.

Please note however that this territory is "muddy" and that there are not many answers out there for the problems of editable installs and namespace packages:

To be completely clear here, it is 100% possible to implement the "legacy" mechanism that setuptools used in the past for editable installs. I believe this mechanism is now available in setuptools via the editable_mode=compat config setting.
The difficulties I have seen raised with editable mode have typically been either limitations of alternative implementation methods, bugs that were also present in the legacy mechanism, or feature requests for capabilities that were mot present in the legacy mode. While it would be nice to have these improvements, I don't believe the community has ever managed to come up with a consistent, clearly specified, and implementable of what such an "improved editable mode" would look like.
If the problems reported here are ones that did not exist before setuptools added PEP 660 support, and they can't be fixed by specifying editable_mode=compat, then I would say that they are a regression in the compatibility mode for editable installs. In all other cases, they are feature requests that may or may not be possible without changes to the core Python import machinery and/or the editable install specification.
The current state of the art is that all editable install mechanisms have limitations, as @abravalheri says. Setuptools gives you a choice of which limitations you prefer (most other build backends just pick one implementation technique and accept its limitations) but that's about the best you can hope for as things stand.
@dwt does editable_mode=compat fix your issue?

To be completely clear here, it is 100% possible to implement the "legacy" mechanism that setuptools used.

Hi @pfmoore, to be frank I think that compat is merely "compatible" (optimistically speaking) with the legacy mechanism that setuptools used in the past, but not 100% identical (Setuptools used complicated things like .egg-link and importlib hacks and it apparently involved "reading" and "modifying" stuff in site-packages). I don't think that old approach is viable to implement given the current standard due to the decoupling of editable wheels and were they are going to be installed.

I think that compat is merely "compatible" (optimistically speaking) with the legacy mechanism that setuptools used in the past, but not 100% identical

OK, I wasn't aware of that. I'm disappointed that it was never raised in the PEP 660 discussions - we repeatedly characterised the existing mechanism as "just add a .pth file" and no-one contradicted that description. But that's water under the bridge now, the discussion was what informed the final specification and the various flaws in that discussion aren't something I have the appetite for revisiting (unless someone is willing to write a new PEP and manage a new discussion that way).
I'll retract the statement that editable_mode=compat not working like legacy editable installs is a regression. But it remains the closest we have to "how things used to work", and should probably still be the first thing to check if something has stopped working.

With the risk of sounding exaustive, I also would like to point out another thing that may have remained unsaid in this recent discussion, but is covered in the docs:
Legacy namespaces are not guaranteed to work in the current implementation of editable installs in setuptools.
This change is intentional, may be backwards incompatible, but I don't think there is much to do about it due to the reasons listed in the previous comment.
Support for namespace packages has been deprecated by a couple of years now and users are prompted to switch.

We may revisit editable in the future based on the new knowledge that was gained via feedback, but that should be done in a "let's look forward" way. We should probably wait for input from core devs on namespaces (and other issues open in the cpython repo) first, though.

+1. My personal view is that what we need to do in order to improve editable support is:

Get an accepted definition of what precisely is meant by an "editable install".
Ensure (via discussion, PRs, and where necessary, PEPs) that the core Python import system is able to support all the capabilities needed for that definition of an editable install (this includes things like importlib.metadata, and anything needed for typing).
Create a mechanism that implements this definition. This could be done by each backend individually, or by a common project like editables. That depends on how successful we are getting consensus on the definition in (1)...

When I accepted PEP 660, my view was that for (1) "what you get creating a .pth file pointing to a src distribution" was the community's accepted minimum spec for an editable install, and (2) and (3) followed from that (with the additional benefit that backends like setuptools could experiment with producing more capable editable installs). Where we are now, it seems like we have established that there is a desire for a better "minimum standard" and we're starting to get an idea of what core Python capabilities that will need. So we're making progress - slowly, like with everything in packaging, but steadily.

This discussion about what to do in the wider python ecosystem is probably necessary, but I would like to come back to my initial observation once more to make sure we are not missing a simple bug.
What I see there is that it seems that Products.__path__ (Product is the namespace package in this case) seems to be correct after python startup. It contains some extra items that seem to be correlated to how setuptools integrates itself, but also the real names of all the packages that are installed. That seems like it would work for tools like Zope, which just iterate that __path__ directly and filter out anything that doesn't exist in the filesystem.
However, after calling list(pkgutil.iter_modules(Products.__path__)) once, some of these entries are gone.
This seems to be part of the problem and something I might be able to contribute a fix towards.
But to do that, I need to understand: Is that actually a problem? Is there a reason why it would be 'correct' to remove these extra entries? If so, why?

ping? Could any of the setuptools maintainer give me a heads here please?

@dwt is there any PR you would like to propose/implement?
As previously noted we don't control the implementation of pkgutil, so the best we can do is to comply with the docs/specifications of the Python import system as defined in https://docs.python.org/3/reference/import.html, which indeed I believe we follow.
Otherwise please consider using a different installation mode (e.g. --config-settings editable_mode=strict or --config-settings editable_mode=compat) or migrating to a src-layout or not using editable installs at all, as they are not guaranteed to always work as stated in the docs


proposal: cmd/go: include the raw import path in the output of go list -json

Background
The ImportPath field in the output of go list -json is not quite what it says on the tin. Instead of a package import path, it contains an import path augmented with a suffix for uniqueness.
This has led to several bugs and some confusion in the past:

#26722
#43380
#44963
#60454

It also adds complexity in tools that deal with the output of go list. For example, golang.org/x/tools/go/packages renames the field to ID and parses out the package path in an ad-hoc manner:
https://cs.opensource.google/go/x/tools/+/master:go/packages/golist.go;l=544-549;drc=f60f2e6aa42c945111053771ea53938694791d83
Since the way one interacts with the go command is by passing it package import paths, the need for this kind of ad-hoc parsing requires that users building tools on top of cmd/go must implement their own ad-hoc parsing, with varying degrees of accuracy.
Unfortunately, too many tools already expect the ImportPath field to contain this extra information. We cannot feasibly rename ImportPath itself to be clearer.
Proposal
I propose that we add a new field to the Package struct format produced by go list, containing the original, unmodified path by which the package's source code was located (in GOROOT, GOPATH, or the module cache).
I suggest that we name the new field PackagePath, along the lines of the existing PkgPath field reported by x/tools/go/packages.
Specifically:

The path should not contain the suffix .test or _test unless that suffix is literally included in the directory or module path containing the package source code.
The path should not contain suffixes to denote test or PGO variants.
As a result, multiple packages reported by go list may share the same PackagePath. They can be distinguished using the (misnamed) ImportPath field, whose semantics remain unchanged.

I suggest that we name the new field PackagePath, along the lines of the existing PkgPath field reported by x/tools/go/packages.

I was hoping this would allow us to simplify the go/packages logic for inferring the package path, but there's surprisingly little of it: with strings.Cut(" "), it would amount to a 1-liner.
Seems reasonable though. The name is consistent with the names we use for this concept elsewhere. ImportPath should ideally be ID, matching the concept used in go/packages, but as you point out, we can't change it now.

Unfortunately, too many tools already expect the ImportPath field to contain this extra information. We cannot feasibly rename ImportPath itself to be clearer.

I assume we will carefully document both ImportPath and PackagePath fields in the docs, though :) I think that's a pretty important part of this proposal, actually.
We can even consider duplicating the new ImportPath field as ID , deprecating the former and discouraging its use. We can likely never remove it, and it would be duplication, but at least JSON users could use the very clear pair of fields ID and PackagePath rather than the pair ImportPath and PackagePath, which will always confuse the reader until they read the docs.


Setup project labels to match other repos

Setup the labels to match the other projects in the KinsonDigital organization

Duplicate


⚠ AGEG website (prod) has degraded performance

In b937776, AGEG website (prod) (https://www.ageg.ca/) experienced degraded performance:

HTTP code: 200
Response time: 2069 ms

Resolved: AGEG website (prod) performance has improved in 5aebe5b.


Unit tests record/playback modification: only one jetty server instance gets started for all tests.

@vladimir-shcherbakov,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Algo expert  questions

below link has algo expert questions.
https://github.com/das-jishu/algoexpert-data-structures-algorithms/blob/master/Questions/

It has already been mentioned in the README file. :)


Use Parser#hooks instead of Tapable#plugin

What kind of change does this PR introduce?
refactoring
Did you add tests for your changes?
no
If relevant, link to documentation update:
n/a
Summary
Replace deprecated Tapable#plugin calls by Hook#tap calls.
After this PR, almost all plugin calls would be replaced! 🎉
Does this PR introduce a breaking change?
no

@sokra Do you want me to rebase the PR?

I've rebased the PR. Conflicts are gone.

Everything is fine now 🎉

Thanks


mosfet props

@seveibar how can they be the same thing? I didn't get you

@seveibar defined pins for mosfet and transistor like the other components


Symbol not found: 'req.request' (expected type BOOL):

Having trouble with a rule that worked in varnish 3 with throttle
Prevent users from making excessive POST requests that aren't for admin-ajax
if(((vsthrottle.is_denied(req.http.X-Actual-IP, 15, 10s) && (!req.url ~ "admin-ajax.php") &&     (req.request == "POST")) {
       return (synth(429, "Too Many Requests"));
       # Use shield vmod to reset connection
       shield.conn_reset();
}

Hi,
req.request has become req.method in varnish 4.

Hi,
req.request becames req.method in varnish 4.
François Veux.
Le 28/10/2015 21:05, blindpet a écrit :

Having trouble with a rule that worked in varnish 3 with throttle
|Prevent users from making excessive POST requests that aren't for
admin-ajax if(((vsthrottle.is_denied(req.http.X-Actual-IP, 15, 10s) &&
(!req.url ~ "admin-ajax.php") && (req.request == "POST")) { return
(synth(429, "Too Many Requests")); # Use shield vmod to reset
connection shield.conn_reset(); } |
—
Reply to this email directly or view it on GitHub
https://github.com/varnish/libvmod-vsthrottle/issues/9.

--
François Veux - +33633267508

Thank you very much for the support.
Made good use of this for protecting Wordpress, feel free to link to the walkthrough in Wiki. I will continue to tweak as necessary but this seems good for now.


on() Method not working on iPhone Safari

Extract from Plyr readme.md...
"Mobile Safari on the iPhone forces the native player for  so no useful customization is possible."
This is very true, however it would be very helpful if Plyr's ability to watch for events, also worked for iPhone users. I'm successfully using Plyr to trigger events for Google Analytics, but it appears that I'll need a separate solution using addEventListner for iPhone users.
I'm not sure if this is a feature request for Plyr, a bug with Plyr, or user error on my part?
Thanks for your help.

Update...
I've been able to test solutions based on the standard HTML5 video player (using addEventListener) which work on the iPhone, but only limited success when using the addEventListener on the HTML video element of Plyr based solutions. Though everything appears to work for simple setups, but fails on more real world setups with more than one Plyr instance and changing media sources.
The only solution I can think of is to setup both Plyr and HTML5 video players, targeted at different devices.

This should work so is definitely a bug. Will look at fixing it shortly.

Any luck with this? I noticed this as well and haven't been able to get around it.

Same here. For me it is a blocking issue because I can't track events.

Hi !
Same issue here. I'm trying to add an event on play in order to stop a slideshow from rotating. Unfortunately it does not work on iPhone so whenever the video starts, it rotates to next slide.
Any workaround ? :)

I used native events waiting for a fix.

Same issue here. Relevant to this, the responsive design of my site (with the responsively embedded Plyr videos) seems to also be ignored because iOS insists on using its own huge, native player (It's my portfolio, so I need to have the 1080p video there) - https://sofusrose.com/portfolio .
Any ideas?

I have it in my notes to fix this in the next major release (which will be soon). Sorry for the delay.

I have the same issues :((

In case someone else needs a work-around, here is mine:
const [player] = plyr.setup(...);

if (window.YT) {
  ytReady();
} else {
  window.onYouTubeReadyCallbacks = window.onYouTubeReadyCallbacks || [];
  window.onYouTubeReadyCallbacks.push(ytReady);
}

function ytReady() {
  const embed = player.getEmbed();

  // Do whatever you want with the YT embed https://developers.google.com/youtube/iframe_api_reference
  embed.addEventListener('onStateChange', (evt) => {
    if (evt.data === window.YT.PlayerState.ENDED) { ... }
  }
}

The purpose of the code is to access the underlying YouTube embed instance. It's a bit convoluted for two reasons:

player.getEmbed() doesn't return anything until the YT API has loaded and the player is ready
player.on('ready') doesn't work as per this issue

Hi, since this is still open it isn't fixed yet?
I'm also trying to get it work with no success...

Same here. I think this is a critical issue for a lot of users. when can we see it fixed?

Everything's critical depending on who you ask 👍
It's coming. I've been doing a lot of work on v3 which has tons of bug fixes in and improvements. I can't give a firm date but I'm in a testing and optimisation kinda phase at the moment.

Thanks for the quick  response, I appreciate it. looking forward for v3

Hey @sampotts, great to hear v3 is underway, have you got any idea of when it might be ready for production?
We have added plyr support for Wistia to our version, is this something that will be baked into v3? Or something that we will need to write again? We have a client who is currently using plyr as their main player and our video tracking is impacted by this bug specifically. If it's not something that will be addressed imminently then we may need to look for alternative's until its ready.
No pressure, just need to know so we can get a plan in place, would much rather keep using Plyr, because its awesome!
Thanks

I'm in the testing + fixing phase now so it's mostly done but of course I keep finding things as I go. We're into the Christmas period as well which doesn't help (I'm in the UK at the moment with family). I'm hoping to get it out in January.
Wistia support has been requested a lot and is in the list. I really want to make a plugin framework so that things like Wistia, YouTube and Vimeo can become addons rather than core functionality as not everyone needs them. I'd like to tackle this first as potentially it could speed up development of features etc since I wouldn't have to be doing it all and it'd open it up a little more and cut down the size of JS and CSS required.

ready is not triggered on iOS for me

Closing this one as I think it's fixed but feel free to re-open if not solved in v3 👍


[SPARK-32915][CORE] Network-layer and shuffle RPC layer changes to support push shuffle blocks

What changes were proposed in this pull request?
This is the first patch for SPIP SPARK-30602 for push-based shuffle.
Summary of changes:

Introduce new API in ExternalBlockStoreClient to push blocks to a remote shuffle service.
Extend SPARK-6237 to allow responding the metadata header back to the client.
Leveraging the streaming upload functionality in SPARK-6237, it also enables the ExternalBlockHandler to delegate the handling of block push requests to MergedShuffleFileManager.
Propose the API for MergedShuffleFileManager, where the core logic on the shuffle service side to handle block push requests is defined. The actual implementation of this API is deferred into a later RB to restrict the size of this PR.
Introduce OneForOneBlockPusher to enable pushing blocks to remote shuffle services in shuffle RPC layer.
New protocols in shuffle RPC layer to support the functionalities.

Why are the changes needed?
Refer to the SPIP in SPARK-30602
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added unit tests.
The reference PR with the consolidated changes covering the complete implementation is also provided in SPARK-30602.
We have already verified the functionality and the improved performance as documented in the SPIP doc.
Lead-authored-by: Min Shen<EMAIL_ADDRESS>Co-authored-by: Chandni Singh<EMAIL_ADDRESS>Co-authored-by: Ye Zhou<EMAIL_ADDRESS>

Can one of the admins verify this patch?

Can one of the admins verify this patch?

A few clarifications on this PR:
The entire netty RPC layer change for push-based shuffle is ~4000 LOC in our current implementation. We plan to break it down into 3 PRs for easier review:

The first one in this PR focus on the foundation for supporting block push functionalities
The second PR will provide the actual implementation for the MergedShuffleFileManager, as well as the integration with YARNShuffleService
The third PR will provide the read path implementation supporting fetching a merged shuffle file as a sequence of chunks

In addition, there are some additional refactoring we could do with this PR.
For example, we reuse RetryingBlockFetcher and BlockFetchingListener for block push as well.
This makes their naming not appropriate any more.
We didn't make that change in this PR to reduce the number of files we touch, so it's easier to review.
We can either send out a separate PR just to do these refactoring or update this PR, depending on the reviewers' preferences.

ok to test

+CC @jiangxb1987

Test build #129045 has started for PR 29855 at commit 2bdf800.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33667/
Test PASSed.

+CC @attilapiros, @mccheah

Something seems off about the jenkins test. @shaneknapp can you please take a look ?

+CC @tgravescs

Test build #129046 has started for PR 29855 at commit 3e9e9e1.

Fixed the Java style issue and the 1 UT failure.
Test build should be clean now.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33668/
Test PASSed.

Test build #129045 has finished for PR 29855 at commit 2bdf800.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129045/
Test FAILed.

Test build #129046 has finished for PR 29855 at commit 3e9e9e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129046/
Test PASSed.

Test build #129122 has started for PR 29855 at commit eb366d4.

Test build #129123 has started for PR 29855 at commit 7a6ab15.

Test build #129122 has finished for PR 29855 at commit eb366d4.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129122/
Test FAILed.

Test build #129123 has finished for PR 29855 at commit 7a6ab15.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129123/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33741/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33742/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33741/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33741/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33742/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33742/
Test FAILed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33744/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33744/
Test PASSed.

Test build #129126 has finished for PR 29855 at commit 85b0de8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129126/
Test PASSed.

+CC @squito for changes related to SPARK-6237

Test build #129257 has started for PR 29855 at commit 7b48c50.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33874/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33874/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33874/
Test PASSed.

Test build #129257 has finished for PR 29855 at commit 7b48c50.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129257/
Test PASSed.

For a bit more context: I am ok with us using external shuffle service on
K8s, I just think that we should keep in mind that for many configurations
it isn’t a good fit due to limitations around volumes so we should be sure
to document it clearly and when possible support alternatives.
On Tue, Sep 29, 2020 at 2:57 PM Min Shen<EMAIL_ADDRESS>wrote:

@Victsm commented on this pull request.

In
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/MergedShuffleFileManager.java
https://github.com/apache/spark/pull/29855#discussion_r497084532:

+import java.io.IOException;

+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.StreamCallbackWithID;
+import org.apache.spark.network.shuffle.protocol.FinalizeShuffleMerge;
+import org.apache.spark.network.shuffle.protocol.MergeStatuses;
+import org.apache.spark.network.shuffle.protocol.PushBlockStream;

+/**

The MergedShuffleFileManager is used to process push based shuffle when enabled. It works

along side {@link ExternalBlockHandler} and serves as an RPCHandler for

{@link org.apache.spark.network.server.RpcHandler#receiveStream}, where it processes the

remotely pushed streams of shuffle blocks to merge them into merged shuffle files. Right

now, push based shuffle can only be enabled when external shuffle service in YARN mode

That comment might not be clear enough.
It's rather that right now we only implemented support for push-based
shuffle when external shuffle service in YARN mode is used.
Extending to standalone mode is definitely doable, considering the
similarity of the usage of ESS in that mode compared with YARN.
For K8S, we discussed briefly with @dbtsai https://github.com/dbtsai
and @holdenk https://github.com/holdenk before.
They prefer to not use external shuffle service in K8S due to the
limitation of mounting persistent volume in that environment, i.e. the
volume can only be mounted in one pod for I/O.
So some work might be needed there to extend support into K8S.
There's also the need of supporting disaggregated shuffle storage, which
we are also evaluating.
I have created SPARK-32925
https://issues.apache.org/jira/browse/SPARK-32925 for discussions on
this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/29855#discussion_r497084532, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAOT5LN25SS2CXYIAKPAPLSIJJ6HANCNFSM4RXN5ZFQ
.
--
Cell :<PHONE_NUMBER>

Test build #129312 has started for PR 29855 at commit 9f00cc3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33927/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33927/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33927/
Test PASSed.

Test build #129312 has finished for PR 29855 at commit 9f00cc3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129312/
Test PASSed.

The pyspark related test failure does not seem related to this PR.

Test build #129370 has started for PR 29855 at commit db36f3f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33979/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33979/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33979/
Test PASSed.

Test build #129370 has finished for PR 29855 at commit db36f3f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129370/
Test PASSed.

Test build #129625 has started for PR 29855 at commit e604686.

@attilapiros @jiangxb1987 @tgravescs @mridulm @Ngone51
I have resolved all pending comments, including the protocol change suggested.
Please let us know if there are any remaining concerns.
If not, want to push on merging this PR, so we can start sending out the 2nd PR for this SPIP.

Test build #129626 has started for PR 29855 at commit b45b190.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34229/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34229/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34229/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34230/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34230/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34230/
Test PASSed.

Test build #129625 has finished for PR 29855 at commit e604686.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129625/
Test PASSed.

Test build #129626 has finished for PR 29855 at commit b45b190.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129626/
Test PASSed.

Test build #129643 has started for PR 29855 at commit f11eb9b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34247/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34247/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34247/
Test PASSed.

Test build #129643 has finished for PR 29855 at commit f11eb9b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129643/
Test PASSed.

Test build #129678 has started for PR 29855 at commit 691635e.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34283/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34283/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34283/
Test PASSed.

Test build #129678 has finished for PR 29855 at commit 691635e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129678/
Test PASSed.

Test build #129699 has started for PR 29855 at commit f016b39.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34306/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34306/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34306/
Test PASSed.

Test build #129699 has finished for PR 29855 at commit f016b39.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129699/
Test PASSed.

Thanks for the additional review comments from @jiangxb1987 @Ngone51. I should have resolved all pending issues.
Ping @attilapiros @tgravescs @mridulm to see if there're any additional concerns on the PR and if we can get a +1.

The most recent test failure does not seem related to this patch.

LGTM

Thanks for the review @Ngone51 !
+CC @tgravescs, @attilapiros, @jiangxb1987, @otterc  any additional comments ? Or are all concerns resolved ?.

Looks good to me.

Test build #129765 has started for PR 29855 at commit 2c95f18.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34371/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34371/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34371/
Test PASSed.

Test build #129765 has finished for PR 29855 at commit 2c95f18.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129765/
Test PASSed.

I do not like the error message check (the string contain) but as it it is promised to be addressed in a follow-up PR https://github.com/apache/spark/pull/29855#discussion_r502788433, it is fine now.
LGTM

changes look fine

Thanks for the reviews @attilapiros, @tgravescs, @otterc, @jiangxb1987, @Ngone51 Merging to master.

Thanks for working on this @Victsm ! Looking forward to the next set of patches on push based shuffle :-)

Thank you all!


Better error handling

Currently, when an error occur inside of a waterfall function, the error is swallowed and waterfall continues to the next step.
This is pretty hard to debug.
Perhaps a better way of handling, bubbling up an exception within a waterfall flow.

Agree... There are definitely cases where I'm swallowing exceptions. I have logic to bubble it up I'm just being too aggressive in certain cases. I need to write a unit test that throws an exception in every possible callback so I can ensure they all bubble. Just haven't had time to do that yet.

I just checked in a new errors.js test and found & fixed an issue with the LuisDialog swallowing errors.

yep, looks good. Thanks a lot!

Hi. I'm a total beginner, and spent a lot of time to understand why I had these

ERROR: Too many calls to session.endDialog()
all the time, before I understood there was an underlying error swallowed .. now I understand !
any clue when this next release will happen so we can debug more easily, or any best practice alternative to actually see these errors ?

I'm trying to get it out this week. I made a pretty big change not related to this issue but it means a bunch of additional testing and documentation work.

Closing this issue. Please open a new issue for further discussion if necessary. @Stevenic - reopen if not implemented.  Thanks!


WIP: Remove cuda labels from .condarc

WIP: Not fully tested yet. Not sure what to do with file: generatedDockerfiles/Dockerfile.centos7-gpuci

Not sure what to do with file: generatedDockerfiles/Dockerfile.centos7-gpuci

I would just apply the same changes to it and resolve the conflicts.

cugraph build on branch-0.9 is broken. I see this error when building clean branch-0.9:
cugraph/utilities/column_utils.pyx:60:36: undeclared name not builtin: get_dtype
cugraph build error blocks building docker images.

There is so many errors when running test.sh using new containers built with branch-0.9 that I'm unable to determine if this patch is working as intended. Blocked on getting this merged/reviewed due to broken tests.

As mentioned in #24 lets put this on the backlog as the code changes in branch-0.9 work without the need for the .condarc file. We can troubleshoot this more after the release


checkout submodules auth failed on workflow/action(personal access tokens is not work)

Description
My repository has a private submodule. I created a Personal Access Token (PAT) and added the "read repository" permission to this token. Then, I placed this token in Actions.Secrets with the name GH_PAT.
I got errors:
::group::Setting up auth
[command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
[command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/git\.sf\.tech\/\.extraheader
[command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/git\.sf\.tech\/\.extraheader' && git config --local --unset-all 'http.https://git.sf.tech/.extraheader' || :
[command]/usr/bin/git config --local http.https://git.sf.tech/.extraheader AUTHORIZATION: basic ***
::endgroup::
::group::Fetching the repository
[command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +bd836639774617aba2efb873cafcd7d8dfcbcb4d:refs/remotes/origin/main
remote: Forbidden
fatal: unable to access 'https://git.sf.tech/C/CodeGen/': The requested URL returned error: 403
The process '/usr/bin/git' failed with exit code 128
Waiting 10 seconds before trying again

name: Build
run-name: Build by @${{ gitea.actor }}

on:
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    container:
      volumes:
        - /home/vicalloy/.ssh:/root/.ssh
    steps:
      - name: Check out repository code
        uses: https://gitea.com/actions/checkout@v3
        with:
          token: "${{ secrets.GH_PAT }}"
          submodules: true

Gitea Version
1.20.0
Can you reproduce the bug on the Gitea demo site?
Yes
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
version: "3.6"
services:
  gitea:
    container_name: gitea
    image: gitea/gitea:${GITEA_VERSION:-1.20}
    restart: unless-stopped
    environment:
      # https://docs.gitea.io/en-us/install-with-docker/#environments-variables
      - USER_UID=1026
      - USER_GID=100
      - DOMAIN=${IP_ADDRESS}
      - SSH_DOMAIN=${IP_ADDRESS}
      - HTTP_PORT=3000
      - ROOT_URL=https://${IP_ADDRESS}
      - SSH_PORT=2222
      - SSH_LISTEN_PORT=22
      - DB_TYPE=postgres
      - DB_HOST=postgres:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea
      - DISABLE_REGISTRATION=true
      - GITEA__openid__ENABLE_OPENID_SIGNIN=true
      - GITEA__service__ENABLE_NOTIFY_MAIL=true
      - GITEA__oauth2_client__ACCOUNT_LINKING=auto
      - GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION=true
      - GITEA__oauth2_client__OPENID_CONNECT_SCOPES=openid email groups
      - GITEA__oauth2_client__USERNAME=email
      - GITEA__indexer__REPO_INDEXER_ENABLED=true
      - GITEA__webhook__ALLOWED_HOST_LIST=192.168.0.*
    ports:
      - "3000:3000"
      - "2222:22"
    networks:
      - gitea
    volumes:
      - ./data/gitea:/data
    depends_on:
      - postgres
  postgres:
    image: postgres:${POSTGRES_VERSION:-13}
    restart: always
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
      - gitea
    volumes:
      - ./data/postgres:/var/lib/postgresql/data

networks:
  gitea:

Database
None

I am trying to validate the token with Git, but I am encountering the following error.
If I provide an invalid token, Git will require username and password input.
base64_token = base64("x-access-token:pat_token")

git -c http.extraheader="AUTHORIZATION: basic base64_token" clone https://git.sf.tech/Castle/Castle.git
Cloning into 'Castle'...
remote: Forbidden
fatal: unable to access 'https://git.shifei.tech/Castle/Castle.git/': The requested URL returned error: 403

You don't have to do this, runner will use an automatically generated token to clone private repo (only the repo which trigger the workflow, not all repos in your Gitea).
So I think this is enough:
    steps:
      - name: Check out repository code
        uses: https://gitea.com/actions/checkout@v3
        with:
          submodules: true

Actually, token: "${{ secrets.GH_PAT }}" should work too, I think it failed because of incorrect permission configuration.

@wolfogre If no token: "${{ secrets.GH_PAT }}" , root repository can be checkout, submodules will failed with remote: Forbidden.
After add token: "${{ secrets.GH_PAT }}", root repository will failed with remote: Forbidden.

So it's a token that can only read packages, not codes.
You need:

Sorry for my mistake. I created a new token.
The root repository can be checked out successfully, but I'm still encountering errors with the submodules.
It seems that the submodule checkout is being done through the ssh:// protocol, and the token is not being utilized.
We are using 2222 as the SSH port, which could be a potential reason for the issue.

::group::Setting up auth for fetching submodules
[command]/usr/bin/git config --global http.https://git.sf.tech/.extraheader AUTHORIZATION: basic ***
[command]/usr/bin/git config --global --unset-all url.https://git.sf.tech/.insteadOf
[command]/usr/bin/git config --global --add url.https://git.sf.tech/.insteadOf<EMAIL_ADDRESS>[command]/usr/bin/git config --global --add url.https://git.sf.tech/.insteadOf<EMAIL_ADDRESS>::endgroup::
::group::Fetching submodules
[command]/usr/bin/git submodule sync
[command]/usr/bin/git -c protocol.version=2 submodule update --init --force --depth=1
Submodule 'vendor/Castle' (ssh://git@git.sf.tech:2222/Castle/Castle.git) registered for path 'vendor/Castle'
Submodule 'vendor/Sample' (ssh://git@git.sf.tech:2222/Manuals/Sample.git) registered for path 'vendor/Sample'
Cloning into '/workspace/huxm/demo/vendor/Castle'...
Cloning into '/workspace/huxm/demo/vendor/Sample'...
Gitea: Public (Deploy) Key: 5:raspberry-pi is not authorized to read manuals/sample.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'ssh://git@git.sf.tech:2222/Manuals/Sample.git' into submodule path '/workspace/huxm/demo/vendor/Sample' failed
Failed to clone 'vendor/Sample'. Retry scheduled
Cloning into '/workspace/huxm/demo/vendor/Sample'...
Gitea: Public (Deploy) Key: 5:raspberry-pi is not authorized to read manuals/sample.
fatal: Could not read from remote repository.

Maybe the token does not match your needs easily, I think you should try using ssh-key, see https://github.com/actions/checkout#usage

Thanks, I have added a deploy key to my runner as a temporary solution.

In the Github actions checkout documentation, there's this relating to the checkout of submodules:
# Whether to checkout submodules: true to checkout submodules or recursive to
# recursively checkout submodules.
#
# When the ssh-key input is not provided, SSH URLs beginning with
#<EMAIL_ADDRESS>are converted to HTTPS.
#
# Default: false
submodules: ""
The action automatically changes the ssh prefix of the URL to https if it begins with the github string if there's no ssh-key. That step is needed if you are going to use the Access Token method. I'm not sure how this could be fixed easily because everyone's Gitea instance will have a different URL as well as a SSH port as part of the URL possibly. The only thing I can think of at the moment is to add some sort of additional option for a git ssh URL substitution string.
In the meantime, it might be better to just create an automated tool user that has an SSH key that can be specified with the 'ssh-key' option. Then give that tool user read-only access to the repos of interest.


Provide a sample extension that does NOT use TypeScript

Not everyone uses TypeScript, even though Lens does, and it's not a requirement to get an extension working. Not using TypeScript means a different configuration of ESLint and Babel, so that would be a useful time saver to get developers started quickly.
Only providing a TS sample means a steep learning curve, and a steep price to pay just to get started on things.
Also, someone may be porting a large amount of non-typed code into an extension, and then TS is just a hinderance, unless they really want to take the time to type it all.

It was decided that at this moment only TypeScript extensions are supported, so closing.

Well, I wrote an extension that doesn't use TypeScript (that is, beyond the required main.ts and renderer.tsx; I wasn't asking to remove the need for those here). Perhaps this will help others get started since it requires additional Webpack configuration to compile .js files separately from .ts[x] files: https://github.com/Mirantis/lens-extension-cc/blob/master/webpack.config.js

Instead of closing maybe we just change the milestone to a future 4.x.0

And to be clear, I was not proposing to change the fact that main.ts and renderer.tsx are TypeScript files, use TS, and those are the entry points. What I was asking for is a formal sample that shows how to incorporate non-TS (i.e. plain JavaScript) files into the mix. That requires additional Webpack configuration, with the babel-loader, and wasn't the easiest thing to get working, so I think others could benefit. All the work is done in my extension, just needs to be formalized a bit into a basic sample.


Code Execution Trace

Woul be very helpful for debugging if, code line execution tracing feature is added. Just the way it does in colab, a pointer shows which line is being executes currently.

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

This feature request has not yet received the 20 community upvotes it takes to make to our backlog. 10 days to go. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

:slightly_frowning_face: In the last 60 days, this feature request has received less than 20 community upvotes and we closed it. Still a big Thank You to you for taking the time to create this issue! To learn more about how we handle feature requests, please see our documentation.
Happy Coding!


Bot not evolving pokemon

Expected Behavior
Evolving Pidgey/caterpie/rattata/weedle when there is enough candies
Actual Behavior
Bot is simply releasing the pokemon even though there are enough candies
Your config.json (remove your credentials and any other private info)
"tasks": [
{
"type": "HandleSoftBan"
},
{
"type": "CollectLevelUpReward"
},
{
"type": "IncubateEggs",
"config": {
"longer_eggs_first": true
}
},
{
"type": "EvolvePokemon",
"config": {
"evolve_all": "Pidgey,Weedle,Rattata,Caterpie",
"first_evolve_by": "cp",
"evolve_cp_min": 10,
"evolve_speed": 20,
"use_lucky_egg": false
}
},
{
"type": "TransferPokemon"
},
Other Information
OS: linux
Git Commit: 72622b438ebd65283ee14d136e2ee610612c43f6
Python Version: Python 2.7.12

update: for some reason, the bot evolved my rattata after collecting 37 candies, there are currently 104 pidgey candies but the bot is not evolving any pidgey,

I believe that will evolve on startup only? I think you need to use '"evolve_captured": "NONE"' to evolve something when its caught. "evolve_captured" is not inside the evolve task.

hmm. is that something new? the old version that i was using was working fine without any issues. this problem came after i updated the bot today.

seems to be working now after i added "evolve_captured": "Pidgey,Weedle,Rattata,Caterpie".
seems weird to me since it was all working well before the latest update. thanks nevertheless.

Hi, I have the same problem,
Could you give me your config ? where did you put the evolve_captured": "Pidgey,Weedle,Rattata,Caterpie"  line ?
Many thanks for your help.

I had this issue too and I solve this by added
            "first_evolve_by": "cp",
            "evolve_above_cp": 0,
            "evolve_above_iv": 0,
            "logic": "or",

to evolve config


Unused '@ts-expect-error' directive in Event Listener module

What happened?
My project imports TypeScript source codes of SnabbDOM:
https://github.com/EasyWebApp/WebCell/blob/20c8fff2e8f788537c68bb6507d5af784b561e52/source/renderer.ts#L7
When I run tsc, I got an error:

node_modules/snabbdom/src/package/modules/eventlisteners.ts:17:5 - error TS2578: Unused '@ts-expect-error' directive.

https://github.com/snabbdom/snabbdom/blob/349c663b795adcc9b1b3fd28e3630e2098228c9d/src/package/modules/eventlisteners.ts#L14-L25
My Environment

software
version

Node
12.18.2

NPM
6.14.8

TypeScript
4.0.2

SnabbDOM
2.0.0

Importing TypeScript source code is not supported. The TypeScript source code of this project is provided in the package not for the purpose of being compiled by the consumer, but because it is necessary for debugging with source maps.
Use the JavaScript, instead. If you have an issue using the JavaScript, please look through the issues and if you don't find your issue, file one.

@mightyiam him, haven't known. Is it common practice? Is it difficult to provide TypeScripf?

I've never heard about packages providing TypeScript. Packages provide JavaScript plus type information in .d.ts files. And that's what we do. I don't want to support something for no good reason.

It won't bundle duplicate helper functions while importing TypeScript source directly.

It won't bundle duplicate helper functions while importing TypeScript source directly.

What is "It", please?

@mightyiam Bundlers

Answer to myself:

replace all snabbdom/src with snabbdom/build
reconfigure Jest: "preset": "ts-jest" to "preset": "ts-jest/presets/js-with-ts"
add "allowJs": true to tsconfig.json

Thanks: https://kulshekhar.github.io/ts-jest/user/config/#the-3-presets

@TechQuery could you please share why you insist on importing the TypeScript?


Improve testing

Pull Request Test Coverage Report for Build 448108968

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-5.4%) to 89.651%

Totals

Change from base Build 446911969:
-5.4%

Covered Lines:
559

Relevant Lines:
612

💛  - Coveralls


Upgrade template to PrimeReact 8

Upgrade template to PrimeReact 8 (and optionally React 18)

I know its not related to this project but I used Sakai React for my "React Security Hardening" blog post and I have a version updated for React 18 and PrimeReact 8.
See: https://github.com/melloware/react-security-hardening


Add docker-build smoke-docker docker-push smoke-package back to .travis.yml

This change is so we can continue pushing to packagecloud.io while pushing branches to travisci/worker-testing on Docker Hub.

It looks like some of the YAML is being interpreted in an unexpected way: https://travis-ci.org/travis-ci/worker/builds/274656582#L3238-L3243


dataloader: Fix clearing a missing value causing a KeyError and halting the event loop #29

Not sure if the tests are any good. If i run the test suite without the fix for "clear", the tests hang...

Yes, tests are looking good!


Introduce a cluster-aware addressing scheme

This PR is a rework of the https://github.com/cilium/cilium/pull/20754.
Introducing the cluster-aware addressing scheme which identifies the network endpoint with IP address + ClusterID. For example, we can distinguish two Pods belongs to the different cluster which has same PodCIDR range with this scheme.
This PR also contains type conversion to the several data structures. Please see the individual commits for more details.
Note that this PR only contains the infrastructure to realize the new addressing scheme. It doesn't contain any code to "use" that.
Introduce a cluster-aware addressing scheme and convert some types to use that

Fixed the suggestion from Casey and rebased with the latest master.

/test

/test

/test

/test-1.25-net-next

ci-l4lb is failing with a known issue (https://github.com/cilium/cilium/pull/21304). We can fix it by rebasing to the latest master, but since it only tests stand-alone LB use cases, it is not related to the change.
I think the error reported from CodeQL is safe to ignore. I intentionally downcast to convert uint32 to uint8 array.

Reviews are in and CI looks ok. Making this ready to merge.

I agree with Tam's suggestion above; that would be great!


feat(aux): simple yield farming contract (masterchef)

Yield farming implementation based on SushiSwap's Masterchef contract.

Similar with stretching out the reward end time: in that case it seems A will accumulate rewards faster at a grandfathered rate compared to users who enter the pool later.

Similar with stretching out the reward end time: in that case it seems A will accumulate rewards faster at a grandfathered rate compared to users who enter the pool later.

I modified the test to show these cases a bit better: https://github.com/aux-exchange/aux-exchange/blob/af9d1a7f5537fb32df4b7b84a976f870c2f1044a/aptos/contract/aux/sources/stake.move#L1246-L1352
The way the mechanism works, there is a single global rate (acc_reward_per_share) that everyone receives regardless of when they started staking, so we can be sure that these two stakers are receiving the same rate. The difference is accounted for by setting user_reward_debt = user_stake * current_acc_reward_per_share when they first deposit / update their position (e.g. by depositing again/withdrawing/claiming). This essentially zeros out their earned rewards since rewards are calculated as user_stake * acc_reward_per_share - user_reward_debt (ignoring the multiplier). So they earn the rate going forward. So if a user started staking before the pool was modified, their reward debt would be set based on the acc_reward_per_share at that time (setting rewards to zero). Then all additions to acc_reward_per_share in the future would be reflected in their reward rate. Not sure if that was a very coherent explanation, I'll add some comments to the contract as well.


⚠ Bofrost.fr has degraded performance

In d339c96, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1193 ms

Resolved: Bofrost.fr performance has improved in 717df40 after 3 hours, 25 minutes.


uibDatePicker focuses the grid of buttons after being popped up, which doesn't include the button bar at the bottom

Bug description:
When opening uibDatepicker through uibDatepickerPopup, this code gets called:
var focusElement = function() {
    self.element[0].focus();
};

because of this event listener:
// Listen for focus requests from popup directive
$scope.$on('uib:datepicker.focus', focusElement);

This will focus the table that surrounds all of the buttons for each day, or month, or year, depending on which picker is showing (because it is the root element for each picker). It seems to me that the element that contains the table plus the button bar at the bottom should be focused instead so that way the whole thing gets scrolled into view, when it gets opened. It is noticeable when using the popup-placement="bottom" or popup-placement="bottom-left" or popup-placement="bottom-right" and the input is near the bottom of the page, since the button bar is below the table.
An easy workaround is just to not use the bottom placements, but I also feel like this behavior can be easily changed too, to work the same for all placements.
Link to minimally-working plunker that reproduces the issue:
http://plnkr.co/edit/ZiszhdYfbZt8UvC2MT5y?p=info
As you can see, when the datepicker gets opened at the bottom of the page, the button bar below it is still out of view.
Version of Angular, UIBS, and Bootstrap
Angular: 1.5.5
UIBS: 1.3.2
(this is the version I'm using and the version in the plunkr, but the behavior is the same in 2.0.0)
Bootstrap: 3.3.6
More info
I have some code that I could potentially use in a PR that would instead focus the outer ul element (which would be a parent of the picker and button bar), but I'm not yet sure whether this is a bug or is as designed.

This would be very tricky to fix - I would consider this an accessibility issue, but this could potentially result in a breaking change.
We need to investigate this - the earliest I can promise to do a detailed analysis is next week sometime, as I am currently on vacation in Europe.

Ok, that will work.
I will add some suggestions/thoughts based on a potential solution that I came up with. It still has some issues (see bottom of post) and I haven't fully tested it, but it might be helpful.
I removed these lines from UibDatepickerController to stop it from focusing the element that it was previously focusing from uib:datepicker.focus:
var focusElement = function() {
    self.element[0].focus();
};

// Listen for focus requests from popup directive
$scope.$on('uib:datepicker.focus', focusElement);

Instead I wanted to listen for the event elsewhere to trigger this focus (somewhere farther up the DOM tree from UibDatepickerController).
So I added this to uibDatepickerPopupWrap:
link: function(scope, element) {
    var child;
    scope.$on('uib:datepicker.focus', function() {
        if(child = element.children()[0]) {
            child.focus();
        }
    });
}

The child of uibDatepickerPopupWrap is actually the ulat the root of this template. Then, I also added a tabindex to this element to actually allow it to be focused.
This works for me, because I used the default popup template. But the main problem I see is that since the template for the popup is configurable by datepicker-popup-template-url, the child element from the uibDatepickerPopupWrap might not even be focusable. So in that case, then maybe it could focus the picker that is showing (it would fallback to the current behavior).
This is a fork of the plunkr above with the changes I was talking about incorporated in.


dbt catalog doesn't support missing "database" fields

Describe the feature
The dbt catalog functionality doesn't support missing database fields.
In a number of places, dbt expects database to be a str, but we should allow it to be None for some databases that don't have support for databases.

Catalogs should be allowed to set the database to None for data warehouses that don't have them.
list_relations should be allowed to take None for its database

on most data warehouses, None should either be an error or mean target.database.
on Spark, anything but None or the schema would be an error.

get_relation should get the same treatment
That's probably it?

Should this be extended to schemas?
Describe alternatives you've considered
We can set database='' everywhere. Kind of ugly! I would like to be able to differentiate "this component does not exist/has no meaning" vs "this is the empty string"...
Edit: This actually doesn't work without writing a bunch of new agate type tester things because by default, '' is coerced into None (which makes sense for seeds!).
Additional context
This is mostly relevant for Spark.
Who will this benefit?
Spark users, developers.

Fixed in #2489


Double MoTD on CDC_SET_CONTROL_LINE_STATE

As seen here the MoTD displays twice, this only happens when using Web Serial API. Everything works normally when using seen.
Solution: Likely debugging in VS Code

Resolved in commit https://github.com/russeree/nyan-keys-stm32-firmware/commit/72d28e07fba2bd1719e44bc3924f10e570465fbf - added a long timer that guards against double displays of the MoTD


Update readme (#129)

Add information about eslint behaviour when configFile
is set directly.

@MoOx Sorry for that. I updated PR.
I thought that LoaderOptionsPlugin is the only way to pass options to eslint-loader but you are right query or options can be also used  here (source webpack2/docs).

query can even be used in webpack 1 ;)


[Question] 大佬你好，请问可以上传一下论文Improving Low-resource Named Entity Recognition with Graph Propagated Data Augmentation的代码吗？

What is your question?
[Question] 大佬你好，请问可以上传一下论文Improving Low-resource Named Entity Recognition with Graph Propagated Data Augmentation的代码吗？最近看到您的论文很有启发，因此希望能够在您的基础上进行研究工作。非常感谢大佬了。
What have you tried?
No response
Code (if necessary)
No response
What's your environment?

AdaSeq Version (e.g., 1.0 or master):
ModelScope Version (e.g., 1.0 or master):
PyTorch Version (e.g., 1.12.1):
OS (e.g., Ubuntu 20.04):
Python version:
CUDA/cuDNN version:
GPU models and configuration:
Any other relevant information:

Code of Conduct

[X] I agree to follow this project's Code of Conduct

请问作者上传代码了吗，在哪里可以找到


Can't find npm module 'newrelic'

Got this error when installing the package on 0.9.4
Error: Can't find npm module 'newrelic'. Did you forget to call 'Npm.depends' in package.js within the 'mycartio_newrelic' package?

@premosystems I'm seeing the same issue — did you ever get it working?