Fixed #30948 -- Changed packaging to use setuptools declarative config in setup.cfg.

setuptools allows using setup.cfg to define a package’s metadata and other options that are normally supplied to the setup(). This declarative approach avoids the need for some logic in setup.py which can help reduce boilerplate and support automation with external tools.
Available since setuptools 30.3.0 (8 Dec 2016).
https://code.djangoproject.com/ticket/30948

I was considering also removing the Python version check at the top of setup.py as I feel it duplicates the python_requires option which has been supported since pip 9.0.0 (2016-11-02). Any thoughts on this?
Same thought for the the overlay warning. Is this warning & logic still relevant with modern pip and setuptools?

Same thought for the the overlay warning. Is this warning & logic still relevant with modern pip and setuptools?

We should always be recommending pip to install Django andpip handles this automatically. For example, installing Django 2.2 over Django 2.1 shows:
Installing collected packages: sqlparse, Django
  Found existing installation: Django 2.1.13
    Uninstalling Django-2.1.13:
      Successfully uninstalled Django-2.1.13
Successfully installed Django-2.2.6 sqlparse-0.3.0

If we decide to remove the overlay warning, I could also ensure all docs recommend pip for installing.

Could you also add a pyproject.toml with the following:

Can you link or provide information as to what this will solve? In other words, when will this influence an action by a packager? I've been testing using python3 setup.py sdist bdist_wheel and other commands.

All remaining feedback from the review has been incorporated. Thanks!

+1 to getting rid of the overlay warning. Using python setup.py install has been deprecated for a long time and use of pip install . or pip install --editable . is recommended.

I think that we also need to update the following:
diff --git a/INSTALL b/INSTALL
index eb9cf6eaf2..c24c4bb028 100644
--- a/INSTALL
+++ b/INSTALL
@@ -1,10 +1,9 @@
 Thanks for downloading Django.

 To install it, make sure you have Python 3.6 or greater installed. Then run
-this command from the command prompt:
+the following commands from the command prompt:

-    python setup.py install
-
-If you're upgrading from a previous version, you need to remove it first.
+    python -m ensurepip
+    python -m pip install .

 For more detailed instructions, see docs/intro/install.txt.
diff --git a/docs/faq/troubleshooting.txt b/docs/faq/troubleshooting.txt
index ba44aa83ef..f90d0e8e6e 100644
--- a/docs/faq/troubleshooting.txt
+++ b/docs/faq/troubleshooting.txt
@@ -14,9 +14,9 @@ Problems running ``django-admin``
 -----------------------------------

 :doc:`django-admin </ref/django-admin>` should be on your system path if you
-installed Django via ``python setup.py``. If it's not on your path, you can
-find it in ``site-packages/django/bin``, where ``site-packages`` is a directory
-within your Python installation. Consider symlinking to :doc:`django-admin
+installed Django via ``pip``. If it's not on your path, you can find it in
+``site-packages/django/bin``, where ``site-packages`` is a directory within
+your Python installation. Consider symlinking to :doc:`django-admin
 </ref/django-admin>` from some place on your path, such as
 :file:`/usr/local/bin`.

diff --git a/docs/topics/auth/passwords.txt b/docs/topics/auth/passwords.txt
index 44e80911ba..134ef14583 100644
--- a/docs/topics/auth/passwords.txt
+++ b/docs/topics/auth/passwords.txt
@@ -89,7 +89,7 @@ To use Argon2 as your default storage algorithm, do the following:
 #. Install the `argon2-cffi library`_.  This can be done by running
    ``python -m pip install django[argon2]``, which is equivalent to
    ``python -m pip install argon2-cffi`` (along with any version requirement
-   from Django's ``setup.py``).
+   from Django's ``setup.cfg``).

 #. Modify :setting:`PASSWORD_HASHERS` to list ``Argon2PasswordHasher`` first.
    That is, in your settings file, you'd put::
@@ -119,7 +119,7 @@ To use Bcrypt as your default storage algorithm, do the following:
 #. Install the `bcrypt library`_. This can be done by running
    ``python -m pip install django[bcrypt]``, which is equivalent to
    ``python -m pip install bcrypt`` (along with any version requirement from
-   Django's ``setup.py``).
+   Django's ``setup.cfg``).

 #. Modify :setting:`PASSWORD_HASHERS` to list ``BCryptSHA256PasswordHasher``
    first. That is, in your settings file, you'd put::

We should probably also update scripts/rpm-install.sh, but I don't know much about the RPM packaging.

Can you link or provide information as to what this will solve? In other words, when will this influence an action by a packager? I've been testing using python3 setup.py sdist bdist_wheel and other commands.

pip will use the pyproject.toml to ensure that it has the required dependencies to build the package. In this case it'll ensure that it has a new enough setuptools and wheel. I chose the version of setuptools based on what was required for configuration in setup.cfg; the version of wheel was admittedly subjective. (It probably isn't essential, as pyproject.toml came after the changes in setuptools that we are interested in...)

Trying to specify old setuptools results in:
Some build dependencies for file:///home/jon/devel/django conflict with PEP 517/518 supported requirements: setuptools==30.2.1 is incompatible with setuptools>=40.8.0.

So I'll use this as the minimum min setuptools unless someone has a reason for a higher version.

Thanks for all the helpful feedback! Changes:

Applied patch to fix docs
Added pyproject.toml
Added Co-authored-by to acknowledge the helpful contributions! :slightly_smiling_face:

Maybe 1 & 2 can be addressed together, but I'd prefer to break things up into smaller changes, to be assessed individually.

Sure.

On 5, first of all gulp. :) — I need to look into this. I'm going to ask @ubernostrum if it's something the board might be able to advise on?

Heh. Yeah. I figured that this was an oversight w.r.t. to the dual licensing bit. But would be good to clarify. It just doesn't quite feel right...

On 3, would you think me particularly old-fashioned if I said that I'd prefer to hold-off on dropping this until well after the Python 2.7 EOL, e.g. until the end of 2020 say? With modern versions of... — yes, but I take it we're catching the case of someone who just happens to have some Python, they know not what installed and downloads Django to find an inconsistency. It costs us nothing to have this here for a season longer.

I figure that there isn't much difference between August 2020 - when Django 3.1 will be release with these changes - and December 2020... If we decide to keep it until Django 3.2, let's put a comment in to that effect so that we can remember to scrub it when development on 3.2 starts.

On 4, not sure what to say immediately, but (as per above) I'd like to address it separately if that's OK.

Yeah. So I think the main issue here is due to installing on top of an existing install in the global site packages using python setup.py install which is discouraged. Hence the update to INSTALL to use pip instead. The main outstanding thing here is the bit related to RPM installs.

On 1 and 2, I'm inclined towards being super conservative here. (For all manner of reasons, the main one being I don't want to break anything.) But it looks OK at first glance, so trimmed down, and with a bit of time to play with, no doubt you'll persuade us.

I don't believe we've changed anything of substance? Do correct me if I'm wrong... :stuck_out_tongue:

Super effort. Thank you.

A pleasure!

I don't believe we've changed anything of substance?

It doesn't look that way, but let's trim it down and then we'll see.

Clarifying the licence ...

Why don't we punt on this action item then? This PR keeps the license exactly as it was before, no change. If there is a decision made that the license needs updating, that can easily happen after this lands (or even before, really). Whether or not we're using setup.cfg makes no difference on precising the license. Do we all agree?

Yep, that's fine. (It's something to handle apart from this PR.) (Super happy to look at it but...)

On 3, would you think me particularly old-fashioned if I said that I'd prefer to hold-off on dropping this until well after the Python 2.7 EOL, e.g. until the end of 2020 say? With modern versions of... — yes, but I take it we're catching the case of someone who just happens to have some Python, they know not what installed and downloads Django to find an inconsistency. It costs us nothing to have this here for a season longer.

Yup, I can add that back, but some context:
Here is my experience installing Django with pip on Python 2. Notice it chooses Django 1.11, not Django 2.0+.
$ pip install django
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting django
  Downloading https://files.pythonhosted.org/packages/cf/19/632a613bc37bbf890f9323ba09374ce9af1d70bb4cba7ff4d3e5e0991b47/Django-1.11.26-py2.py3-none-any.whl (6.9MB)
     |████████████████████████████████| 7.0MB 3.9MB/s
Collecting pytz
  Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Installing collected packages: pytz, django
Successfully installed django-1.11.26 pytz-2019.3

Without this version checking, here is my experience installing Django 3.1 from the source directory using Python 2
$ pip install .
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing /home/jon/devel/django
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jon/test/v/bin/python2 /home/jon/test/v/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpNt7Y0Y
       cwd: /tmp/pip-req-build-85tspF
  Complete output (42 lines):
  Traceback (most recent call last):
    File "/home/jon/test/v/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/home/jon/test/v/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/jon/test/v/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 146, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 127, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 237, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 3, in <module>
      setup()
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib64/python2.7/distutils/core.py", line 124, in setup
      dist.parse_config_files()
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/dist.py", line 701, in parse_config_files
      ignore_option_errors=ignore_option_errors)
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/config.py", line 121, in parse_configuration
      meta.parse()
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/config.py", line 426, in parse
      section_parser_method(section_options)
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/config.py", line 399, in parse_section
      self[name] = value
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/config.py", line 184, in __setitem__
      value = parser(value)
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/config.py", line 514, in _parse_version
      version = self._parse_attr(value, self.package_dir)
    File "/tmp/pip-build-env-mglrZa/overlay/lib/python2.7/site-packages/setuptools/config.py", line 349, in _parse_attr
      module = import_module(module_name)
    File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
      __import__(name)
    File "/tmp/pip-req-build-85tspF/django/__init__.py", line 1, in <module>
      from django.utils.version import get_version
    File "/tmp/pip-req-build-85tspF/django/utils/version.py", line 71, in <module>
      @functools.lru_cache()
  AttributeError: 'module' object has no attribute 'lru_cache'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/jon/test/v/bin/python2 /home/jon/test/v/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpNt7Y0Y Check the logs for full command output.

With the error messaging reinstated:
$ pip install .
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing /home/jon/devel/django
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jon/test/v/bin/python2 /home/jon/test/v/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp58SeFP
       cwd: /tmp/pip-req-build-rfm95q
  Complete output (20 lines):

  ==========================
  Unsupported Python version
  ==========================

  This version of Django requires Python 3.6, but you're trying to
  install it on Python 2.7.

  This may be because you are using a version of pip that doesn't
  understand the python_requires classifier. Make sure you
  have pip >= 9.0 and setuptools >= 24.2, then try again:

      $ python -m pip install --upgrade pip setuptools
      $ python -m pip install django

  This will install the latest version of Django which works on your
  version of Python. If you can't upgrade your pip (or Python), request
  an older version of Django:

      $ python -m pip install "django<2"
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/jon/test/v/bin/python2 /home/jon/test/v/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp58SeFP Check the logs for full command output.

Installing without the check certainly has less pleasant error message, but it will not allow the user to proceed with an incompatible version. From your perspective, is this all about the user messaging?

I figure that there isn't much difference between August 2020 - when Django 3.1 will be release with these changes - and December 2020... If we decide to keep it until Django 3.2, let's put a comment in to that effect so that we can remember to scrub it when development on 3.2 starts.

I agree with this sentiment, but am happy to comply.  I can add this comment if it is agreeable with @carltongibson

I have restored the version check and overlay check in setup.py. I have also included a TODO comment to remove this at a future date. Thanks for all the feedback!

@jdufresne This check is not only for Python 2, you can try to install e.g. Django 3.0 with Python 3.5 via easy_install (it will raise two errors but finished properly) it can be harder to find small incompatibilities between Python 3.X versions. With our "unsupported" message you will get a clear information
Processing .
Writing django/setup.cfg
Running setup.py -q bdist_egg --dist-dir django/egg-dist-tmp-dxmgcrij

==========================
Unsupported Python version
==========================

This version of Django requires Python 3.6, but you're trying to
install it on Python 3.5.

This may be because you are using a version of pip that doesn't
understand the python_requires classifier. Make sure you
have pip >= 9.0 and setuptools >= 24.2, then try again:

    $ python -m pip install --upgrade pip setuptools
    $ python -m pip install django

This will install the latest version of Django which works on your
version of Python. If you can't upgrade your pip (or Python), request
an older version of Django:

    $ python -m pip install "django<2"
error: Setup script exited with 1

easy_install is deprecated. Is that really an install method Django supports?
Here is setuptools purging it from their docs:
https://github.com/pypa/setuptools/commit/4c0e204413e6d2c33662a02c73a0ad13d81af9e5
I think we should do the same. It is only referenced in internals docs:
docs/internals/howto-release-django.txt
307  #. Test that the release packages install correctly using ``easy_install``
314          $ easy_install https://www.djangoproject.com/m/releases/$MAJOR_VERSION/Django-$RELEASE_VERSION.tar.gz

Why don't we punt on this action item then? This PR keeps the license exactly as it was before, no change. If there is a decision made that the license needs updating, that can easily happen after this lands (or even before, really). Whether or not we're using setup.cfg makes no difference on precising the license. Do we all agree?

Let's just consider at least changing BSD to BSD-3-Clause so that we are explicitly specifying the correct license. There are easily 10~20 variants. I'm fine to punt the dual license issue thing to be addressed later.

OK, thanks for the comments all. I'll have another read over tomorrow and we can come to a final decision.

Right, lots going on here. I'm still reading about PEP 518 and such.
A couple of points thus far though:

Let's just consider at least changing BSD to BSD-3-Clause so that we are explicitly specifying the correct license. There are easily 10~20 variants. I'm fine to punt the dual license issue thing to be addressed later.

OK, but can we do that in a separate PR please? Should be easily mergeable and the commit history will show Clarified licence... when we're looking for it later.
Then, I would like us to keep the install warning, at least for now.

I accept the wording is sub-optimal, saying "download from pip" when you've tried to install from source, but we could improve that in another PR, and the headline news is correct:
  This version of Django requires Python 3.6, but you're trying to
  install it on Python 3.5.

The easy_install discussion is not relevant — I can quite easily imagine someone doing it — but you get the same issue using pip install . as any other method.

From your perspective, is this all about the user messaging?

Ultimately yes. I think there are many more users than we think about who don't have access to the latest Pythons, or even the option to always fetch from PyPI. I think if we can ensure clear error messages in these cases (and I know they're always going to be under-maintained...) then I think we do a good thing.
I'd just like a little more time to be 100% clear, but the scope of this is looking much more manageable. Thanks again for your efforts both.

OK, but can we do that in a separate PR please? Should be easily mergeable and the commit history will show Clarified licence... when we're looking for it later.

That's fair.

Then, I would like us to keep the install warning, at least for now.

Sure. Jon has essentially punted the removal to Django 4.0 by sticking a comment in.

The easy_install discussion is not relevant — I can quite easily imagine someone still using easy_install — but you get the same issue using pip install . as any other method.

Except pip will honor python_requires. It's not the end of the world as we've already decided to hold off until Django 4.0, by which time I imagine the whole story around packaging and the obsolescence of easy_install will have solidified. At least by 2022 it better have!

Should be easily mergeable and the commit history will show Clarified licence... when we're looking for it later.

Pull request for license clarification: #12038

On 5, first of all gulp. :) — I need to look into this. I'm going to ask @ubernostrum if it's something the board might be able to advise on?

@carltongibson I've read this and I'm still unsure what the question is. If someone can distill it plainly for me I'm happy to look into it.

@ubernostrum This stems from my https://github.com/django/django/pull/12013#discussion_r342001037. Read that for more detail.
Essentially I noticed when reviewing this that the metadata field for the license was ambiguous. It is often enough to omit this field where it is already available in the trove classifiers. The classifier for "BSD License" isn't specific enough, however, as there are many variants and the value "BSD" in the license field isn't clear either. Hence pull request #12038. That I think is fairly straightforward.
So the main issue is that some of the code in Django is copied from the Python Standard Library. We used to have a backport of weakref, but now it is just a few functions in django.utils.http. (I will add a couple of comments in another PR to highlight some of those that do not make it clear.) When the weakref backport was added, the Python-2.0 license was added to the repository.
So the question is: Does this mean Django is released under the terms of both licenses?
If so, should we change the license metadata field to be (BSD-3-Clause AND Python-2.0) and add License :: OSI Approved :: Python Software Foundation License to classifiers?
Maybe I'm making something of nothing or opening a can of worms, but I felt this need clarification.
(@carltongibson A decision on this doesn't need to hold up this PR. We can address any action separately.)

I will rebase this.

I noticed the RemovedInDjango40Warning comment was removed from setup.py. Was that intentional?

I believe for GitHub to pick up the Co-authored-by tag you'll need to remove the period at the end of the line. Notice that right now Nick's avatar isn't on the commit.

Yes, it was, and . hmmm — good spot!

I made tests, few releases, installations etc., and I think we should merge the first two commits and drop the last one with pyproject.toml. Mainly because it impacts all installations from source by adding steps with extra requirements and building wheels:
Installing build dependencies ... done
Getting requirements to build wheel ... done
  Preparing wheel metadata ... done
...
Building wheels for collected packages: Django
  Building wheel for Django (PEP 517) ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-rdpid0sx/wheels/37/5a/91/c7c6aa11614143eb9df817cdae01d30967a46a1c3673469755
Successfully built Django
...

I don't see any advantages (in our case) of wheel over sdist :man_shrugging:

So here is the documentation on how pip behaves which may explain better than I can, along with the previously linked articles.
I was trying to be conservative but misunderstood and thought that by not specifying build-backend we were opting out of the PEP 517 stuff which did have some issues with editable builds earlier this year. That amounted to the fact that editable installs were not specified under PEP 517 so some of the PyPA guys thought that it would be alright to prevent them - this didn't go down well because, while a new generic blessed way of doing them is as yet undefined, they worked. They backtracked on this though and there are no issues as far as I can tell. All of this would have only affected those who are developing Django as there is no reason to install it in editable mode if it is a dependency of your own project.
Given all that, perhaps we should just go with:
[build-system]
requires = ['setuptools>=40.8.0', 'wheel']
build-backend = 'setuptools.build_meta'

I've also notice that we'll need to add a line to tox.ini too - see here - which also highlights this as the recommended way to package.
So to my understanding, the whole purpose of this PEP 517 & 518 stuff is to formalise how packages are installed and, in the case of source packages, how those are built at install time without pip needing to know explicitly which build backend. In addition, any build time requirements are specified in pyproject.toml (setup_requires in setuptools has never worked well and is a bit chicken-and-egg) and pip will install those prior to starting the build. The source package is built as a wheel in an isolated virtualenv with all of the required build dependencies at the correct versions (something that was also questionable in the past) and that wheel is then (optionally) cached and installed. Note that pip has a single format to install from in the PEP 517 & 518 world.
As you say, it impacts all installations from source. This is intentional. Is it a negative impact however? Is it actually better because it is less error prone and reproduceable? Besides, how many installations are actually done from source distributions and not wheels these days?
If you decide to drop it be aware that this will come back again in the future.

OK, thank again @pope1ni. We can go with the first two changes and pause to think (over the weekend say) about the last.

If you decide to drop it be aware that this will come back again in the future.

Yes. It will. That might be OK though: it might not be so hot then 🔥. Anyhow... we'll have a read. 👍

Let's not worry about the last commit for now.
It's not worth getting bogged down in at this stage. I honestly didn't think it would be that contentious or too soon, but I think the real problem here is a lack of clear messaging to developers around packaging in Python - what the roadmap is and how the changes all work.
Thank you for taking the time with this 🙃

Thanks y'all :tada:


bagger: 0.1.1-0 in 'kinetic/distribution.yaml' [bloom]

Increasing version of package(s) in repository bagger to 0.1.1-0:

upstream repository: https://github.com/squarerobot/bagger.git
release repository: https://github.com/squarerobot/bagger-release.git
distro file: kinetic/distribution.yaml
bloom version: 0.6.5
previous version for package: null

bagger
* Initial Commit
* Contributors: Brenden Gibbons

Your release repository does not appear to exist.

It is marked private?

Whoops, yes it was.  Changed to public.


system cannot find the file spcified

Do these lines in the bot.py file correctly point to the location of the speed test program executable on your machine?


Discord nicknames

When players have a server nickname with WMGT we’re unable to see that nickname.

We'll use wmg_players.name for the nickname and keep it empty for most users.
wkg_players.account will always be the base Discord account name


enable esbuild for SSR

Command
build
Description
we can use @angular-devkit/build-angular:browser-esbuild but cannot use @angular-devkit/build-angular:server-esbuild
error:
Cannot find builder "@angular-devkit/build-angular:server-esbuild"

Describe the solution you'd like
No response
Describe alternatives you've considered
No response

Thanks for this feature request. This is currently in the roadmap and will be available in an upcoming minor version within the v16 timeframe.

wow, great news


server: reduce execution time while listing project if projects have many resource tags

Description
If projects have many resource tags, it will take long time to list projects.
Remove resource tags information from project_view will fix it the issue.
Fixes: #3178
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

create project,
add resource tags to project,
listprojects  lists all resouce tags.

If the issue is mostly prevalent in UI, I would like to propose an alternate -

Add a parameter showTags to list projects API
Set the flag to false to list project call in GUI.

Although if not many users need the tags as part of response this is perfectly reasonable too.

@ustcweizhou we will test it and come back :)

If the issue is mostly prevalent in UI, I would like to propose an alternate -

Add a parameter showTags to list projects API
Set the flag to false to list project call in GUI.

Although if not many users need the tags as part of response this is perfectly reasonable too.

@anuragaw In my point of view,  it would be preferable to fix an issue without api change.

not sure if the 'standard' travis smoke tests cover enough of this functionality.
@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2727

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@ustcweizhou we are tested it and it seems and feels faster but it seems there is still a small delay. is there anything to make it more faster?

@svenvogel we have similar issue while listing accounts. for project, it can be improved by
diff --git a/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java b/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java
index 91e0466..2b15113 100644
--- a/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java
+++ b/server/src/main/java/com/cloud/api/query/QueryManagerImpl.java
@@ -1298,7 +1298,7 @@ public class QueryManagerImpl extends MutualExclusiveIdsManagerBase implements Q
         Long accountId = null;
         String path = null;

-        Filter searchFilter = new Filter(ProjectJoinVO.class, "id", false, startIndex, pageSize);
+        Filter searchFilter = new Filter(ProjectJoinVO.class, null, false, startIndex, pageSize);
         SearchBuilder<ProjectJoinVO> sb = _projectJoinDao.createSearchBuilder();
         sb.select(null, Func.DISTINCT, sb.entity().getId()); // select distinct
         // ids

Do not order project_view will reduce execution time of db query.

@ustcweizhou oh i see. hmm should we add this to this pr?

@ustcweizhou we will pick up your change ne try to fix the sort of the project list with jQuery.

@ustcweizhou @svenvogel  I built and tested both the PR and the speed improvemennt for the db query (not ordering project_view). Although there seems to be a slight improvement in time for us it still takes way too long. It always feels like you miss-clicked and users then tend to click two or three times (which ends in misery because the lists closes with the second click)

@DennisKonrad did you recreate project_view in your testing ?

@ustcweizhou Yes, i executed the schema update. I think there is an issue about the project listing that's not about tags used. We have about ~30 projects now and this seems to be too much

@DennisKonrad can you login database to see how many projects do you use and how many records in project_view ?
If you do not use tag , the root cause should be different .

@ustcweizhou We have ~40 entries and 30 projects that are selectable in the list. Our database is very fast so the queries should not be a problem.
Do you see any improvments that can be done to the project listing code that would make it faster?

@DennisKonrad
I think it might be caused by the queries to get resource limit of project account.
I will have a look.

@DennisKonrad I will raise a PR to fix the issue in listProjects and listAccounts.
What cloudstack version do you use ?

@ustcweizhou We more or less use the master branch. So you can base the PR on that if you want. Thanks

@DennisKonrad PR #3331 to apache/master

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2797

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2866

Guys,
personally I suspect this is just hiding the problem, and doesn't help anyone who does want to see tags.
I created the followingissue a few days ago https://github.com/apache/cloudstack/issues/3375
I suspect that this looping through accounts in a project is causing the delay.
??

@PaulAngus I think the issue #3375 can be fixed.
in https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/api/query/ViewResponseHelper.java#L201-L211
with this PR, fillProjectDetails will do nothing, so the tags will be set only once.
    public static ProjectResponse fillProjectDetails(ProjectResponse rsp, ProjectJoinVO proj) {
        return s_projectJoinDao.setProjectResponse(rsp, proj);
    }

Can you review and test this @anuragaw

Just to confirm this does not break returning of tags in the API response, just the mechanics of how they are found/used @ustcweizhou ? cc @PaulAngus

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Just to confirm this does not break returning of tags in the API response, just the mechanics of how they are found/used @ustcweizhou ? cc @PaulAngus

in my testing , tags are still in the api response and same as before.

Thanks @ustcweizhou I'll kick some test and review/lgtm it soon.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-52

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-48)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31665 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3306-t48-kvm-centos7.zip
Smoke tests completed. 71 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

I'll wait for your final change @ustcweizhou, then it it ready for merge. The latest tests LGTM.

@rhtyd @anuragaw
rebased with latest master, and added a separate commit to remove the methods not in use.
If we need to fill more information to project response in the future, we can revert the commit.

Thanks @ustcweizhou
@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian.


Download link Provides a different version to Microsoft Download Center

The MS download center provides what appears to be a much outdated link:
https://www.microsoft.com/en-us/download/details.aspx?id=46436
(From here: https://azure.microsoft.com/en-us/updates/documentdb-data-migration-tool/)
Where as this page links to a link currently point to this address:
https://aka.ms/csdmtool -> https://cosmosdbtools.blob.core.windows.net/datamigrationtool/2019.09.23-1.8.3/dt1.8.3.zip
Are both version Officially supported? Should we use latest linked here for production?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9edf1df2-588e-01ff-6d1c-234b8e51d550
Version Independent ID: 27873c08-b6cc-a52f-f277-ff9818a5e4d2
Content: Tutorial: Database migration tool for Azure Cosmos DB
Content Source: articles/cosmos-db/import-data.md
Service: cosmos-db
GitHub Login: @deborahc
Microsoft Alias: dech

@Daniel-SW Thank you for the feedback.
This is being assigned to the content author to evaluate and update the documentation as appropriate.

@Daniel-SW Thanks for pointing out the issue, the download on this page: https://www.microsoft.com/en-us/download/details.aspx?id=46436 is outdated, I don't have access to update that page, I will reach out to our product group.
The aka link is kept up to date, so please use the version in the doc.

The 1.8.3 version is the latest and it matches with the version in the source https://github.com/Azure/azure-documentdb-datamigrationtool/blob/master/Solution Items/CommonAssemblyInfo.cs
I passed the details on updating the download page to our product team.

@Daniel-SW Because there isn't anything to fix specifically in the doc we will close this issue. I have an email thread internally to get the downloads page updated. However, it's out of the scope of the docs and I will track it there.
#please-close


Gimbals not respecting world/local angles in offset input

Please note: This issue tracker is for bug reports
If your issue isn't a bug report, or if you are unsure whether it's a bug, please go to our discord channel instead at https://discord.gg/H8UKY3Y
If you are confident your issue is a bug, please fill in as much information as you can below.
Describe the bug
Angle Offset input on wiremod gimbals is seemingly (a hypothesis at least) not offset with respect to the prop or world, but something else?
To Reproduce
Steps to reproduce the behavior:

Place a gimbal
Create two constant values with a difference of only 90 degrees between the two, any axes
Note how it seems to rotate on inconsistent axes

Expected behavior
Each angle axis should offset the gimbal on a consistent axis, but doesnt.

this is matrix stuff, https://github.com/wiremod/wire/blob/master/lua/entities/gmod_wire_gimbal.lua#L66-L69
@thegrb93 looks like you wrote this code, you wanna take a look at this?


Add notebook for Zero and Few Shot named entity recognition

Hi, can you provide a tutorial notebook for Zero and Few Shot named entity recognition?

Closing as a notebook has been added to the Readme/Doc


block_utils

Hi all,
When trying to execute find_homozygous_regions.py ran into a problem of missing block_utils module.
Could only find up42-blockutils, but that didn't seem to be correct, as there was nothing about Alignment there.
Best,
Evgeny Leushkin

@evgenyleushkin ,
Can you please show how you are running the script? Through docker?


fruit dataset

Here you go!

You kept the issue up and I figured you where going to move forward with the site and still wanted the files in the repo. I mean 6mb... its small enough to where I just wasn'tt sure.. I should have included the zip at the very least... closed

Ah, got it :)


doc: Add VSCode and openOCD launcher documentation

Summary

Added VSCode launcher to use openOCD in the flashing and debugging process.
Closes #9
Changes

Added VSCode openOCD launcher and sample images.

Notes for Reviewers

...
Checklist

[x] Scope: The referenced issue is addressed, there are no unrelated changes.
[x] Run Locally: Verified that the docs build using make server, posted screenshots, verified external links.
[x] New Features Marked: Documentation for new features is marked using the new-in-version shortcode, according to the guidelines in CONTRIBUTING.
[x] Style Guidelines: Documentation obeys style guidelines in CONTRIBUTING.
[x] Commits: Commit messages follow guidelines in CONTRIBUTING, there are no fixup commits left.

Thanks @nejraselimovic for the review, can you please take anther look.


Bug: 2022.14 update display resolution, display name gives 'unknown' state

Describe the bug
Since 2022.14. update, the display resolution is not available in home assistant. It gives "unknown" state. Right after i updated to this version, and rebooted, it stopped working correctly.
To Reproduce
i started to use hass agent earlier, and then updated to 2022.14, and then the display resolution is unknown
Expected behavior
I hope somehow i can get it work
Screenshots
screenshot: https://drive.google.com/file/d/1CmMGnyKlYwunFcGmhC-LtIUUT8_14nyW/view?usp=share_link
Misc info (please complete the following information):

Windows build (ideally screenshot/info of winver.exe output): screenshot: https://drive.google.com/file/d/1xtWKQTDTvEKTvb0Eui2m3CrK0H7idXsX/view?usp=share_link
Windows' UI language: english
HASS.Agent version: 2022.14

Please check what's applicable (multiple answers possible):

[x] Installed via installer
[ ] Installed manually
[x] Problem occurs in HASS.Agent
[ ] Problem occurs in Satellite Service

Additional context
Add any other context about the problem here.
Logs
all the logs are compressed to zip and uploaded here: https://drive.google.com/file/d/1s0ROrKUY4_FcIcQa8NyDf1xmekNl4f12/view?usp=share_link

Hi @deeguz,
There were a few small changes to the Display sensor. because combined with a non-default DPI the wrong values would show. However, I don't really recognize your screenshot - did you create that yourself with a template sensor?
You need to use any of these attributes:

Sorry, it was my fault. Before this update, the display resolution had its very own entity, now it's moved to display's attributes. I guess it's not a bug, so i guess it's solved :)


feat: workflow dispatch for releases

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 27.065%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
3642

Relevant Lines:
14187

💛  - Coveralls


Support pipeline overridable constants

wgpu now supports constants that can be set when the pipeline is created https://github.com/gfx-rs/wgpu/releases/tag/v0.20.0
It'd be cool if those were supported in a type-safe way.

This makes sense to just add as a struct parameter similar to javascript. I'll look into adding this to the example project.
https://www.w3.org/TR/webgpu/#dom-gpuprogrammablestage-constants
shader::Constants {
    _1300: 2.0,  // "gain"
    depth: -1.0,  // "depth"
}

If you need to use pipeline-overrideable constants in your project, you can use the latest commit and reference the example project code. I still need to update the generated code to make it easier to actually set the constants in the compilation options.

That's amazing, thank you very much! I'll look at the examples and use that for now.
And I'll look forward to the codegen improvements.


ui-select and ui-select multiple styling mismatch for bootstrap theme

Thanks in advance for helping direct me to the right place for this if this is not a ui-select bug...
 and  appear to have different CSS stylings when theme="bootstrap".
The official examples here demonstrate the problem. :
http://angular-ui.github.io/ui-select/demo-bootstrap.html
http://angular-ui.github.io/ui-select/demo-multiple-selection.html
Specifically the form field height is smaller and the placeholder text is floating further to the left. There also seems to be a placeholder text color mismatch between bootstrap form-control input fields, ui-select, and ui-select multiple (three unique values).
Reproduced in my own form here:

Hi, I see the difference but cannot tell why this is. Would you mind to have a look at it and find the root cause? Then we can tag it as bug (or something).
Thanks.

Here are the full screenshots of the official "with Bootstrap" examples.
Another problem, that you don't see here: The allow-clear option for the ui-select-match directive does not seem to have any effect, the clear button is not shown.
I already tried to get this to work with multiple version combinations of all bootstrap/selectize/select2, but nothing seemed to work.

Just came across this and noticed the examples looked jacked -- I'm seeing the same thing as @henk23

it is messy up when not adding the "multiple" attribute... it is only for multiple select?not for single select!


Inconsistent behaviour between InputEvent.is_action* methods

Tested versions

Reproducible in: Godot v4.3.stable

System information
Godot v4.3.stable unknown - Arch Linux #1 SMP PREEMPT_DYNAMIC Mon, 09 Dec 2024 14:31:57 +0000 - Wayland - Vulkan (Forward+) - integrated Intel(R) HD Graphics 5500 (BDW GT2) - Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz (4 Threads)
Issue description
In the documentation page for InputEvent it says that is_action* methods will ignore input modifiers if exact_match is false.
The methods is_action() and is_action_released() follow this description, but is_action_pressed() does not ignore input modifiers even if the value of exact_match is false.
I'm not sure if this is a mistake in the documentation or an error in the implementation of these methods.
Steps to reproduce

Create an action with an input modifier, Ex.: 'Ctrl+s';
Paste this code in a script:

func _input(event: InputEvent) -> void:
    prints("")
    if event.is_action("special_action"):
        prints("According to 'is_action' Ctrl+s has been pressed/released.")

    if event.is_action_pressed("special_action"):
        prints("According to 'is_action_pressed' Ctrl+s has been pressed.")

    if event.is_action_released("special_action"):
        prints("According to 'is_action_released' Ctrl+s has been released.")

Press 's', in the console it'll show that is_action() and is_action_released() returned true;
Press 'Ctrl+s', in the console it'll show the previous messages plus the is_action_pressed() message

Minimal reproduction project (MRP)
N/A

See https://github.com/godotengine/godot/issues/57943#issuecomment-1073191623.

I see, I was confused since they all have the same description, yet is_match_pressed() behaved differently.


[fatal] [bridge] Bridge must be enabled by using defineNuxtConfig to wrap your Nuxt configuration.

Environment
Nuxt project info:                                                                                15:21:34

Operating System: Windows_NT
Node Version:     v16.14.2
Nuxt Version:     2.16.0-27616340.013f051b
Package Manager<EMAIL_ADDRESS>Builder:          webpack
User Config:      loading, css, render, buildModules, modules, build, serverMiddleware, bridge, serverHandlers, devServerHandlers
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules<EMAIL_ADDRESS><EMAIL_ADDRESS>

Reproduction
https://github.com/some-user123/nuxt-bridge
Describe the bug
Error message "Bridge must be enabled by using defineNuxtConfig to wrap your Nuxt configuration." even though the nuxt configuration is wrapped with defineNuxtConfig.
Additional context
No response
Logs
yarn run v1.22.19
$ nuxi build
[log] Nuxt CLI v3.0.0-rc.4
[fatal] [bridge] Bridge must be enabled by using `defineNuxtConfig` to wrap your Nuxt configuration.
  at setup (node_modules/@nuxt/bridge/dist/chunks/module.mjs:1307:13)
  at ModuleContainer.normalizedModule (node_modules/@nuxt/kit/dist/index.mjs:582:29)
Error:  [bridge] Bridge must be enabled by using `defineNuxtConfig` to wrap your Nuxt configuration.
  at setup (node_modules/@nuxt/bridge/dist/chunks/module.mjs:1307:13)
  at ModuleContainer.normalizedModule (node_modules/@nuxt/kit/dist/index.mjs:582:29)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Process completed with exit code 1.

as a workaround add section bridge : {} into config


Make DockerCompose image waiter independent of image version

I'm going to think a bit more if I am happy with this approach

The reason I needed this has been resolved by https://github.com/shotover/shotover-proxy/pull/1127
I'm not convinced the complexity provides enough convenience to be worth it, so I'm going to close this PR.


mypy fixes

adding type-annotation marker files for matter_idl and matter_yamltests packages
more mypy fixes

@mergifyio refresh


Customizing the Ribbon Menu

I have figured out to apply customizations and publish them (eg. adding a new tab to the main screen), but haven't been able to find a way to customize the ribbon menu.  Can anyone point me in the right direction?

When you Sam customize the ribbon menu, do you then mean the buttons like ”Create New Project” and ”Delete” ?
I am looking for a way to hide those buttons.


Refactor dronecore server

some cleanup on the proto generation (build_system)
use references and smart pointers instead of old-style pointers
make a library out of dronecore_server (necessary for running from iOS)
rename dronecore_server into backend

A PR actually updating the proto files will come just after this one is merged.
Connects to #238.

Yes sorry, I got stuck on the iOS build. I'm working on this right now.

Coming back with some unit tests soon.


Making Activity Classifier updatable

Hello, I try to do my Activity Classifier updatable using this script.
`import coremltools
spec = coremltools.utils.load_spec('RawSleepClassifier.mlmodel')
spec.description.trainingInput.extend([spec.description.input[0]])
spec.description.trainingInput.extend([spec.description.output[-1]])
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)
builder.make_updatable(['dense_layer1'])
builder.set_categorical_cross_entropy_loss(name='lossLayer', input='activityProbability', target='trueActivityLabel')
from coremltools.models.neural_network import SgdParams
builder.set_sgd_optimizer(SgdParams(lr=0.01, batch=32))
builder.set_epochs(10)
from coremltools.models import MLModel
mlmodel_updatable = MLModel(builder.spec)
mlmodel_updatable.save('SleepClassifier.mlmodel')`
But I get the error:
RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "Error reading protobuf spec. validator error: Model specification version for an updatable model must be '4' or above.".
I run MacOS 10.15 Beta 6 and Turi Create 5.7 with CoreML tools v3.0b3

It looks like TuriCreate is still exporting to Core ML using specificationVersion: 1. You do indeed need version 4 in order to have an updatable model. There are also lots of other improvements made in Core ML since the first version.
We should update TuriCreate to export models to Core ML using the most recent specification version.

@TobyRoseman so there is no way to generate updatable model with Turi Create? As I remember it works before.

@mecid - There is no way to generate an updatable Core ML model with Turi Create 5.7.  I also don't think this has worked with any previous version of Turi Create.

@TobyRoseman can we expect it anytime soon?

What if you add spec.specificationVersion = 4 to the above script?

@hollance it works! Model works correctly. I will try to run on-device update task.
Is it legal to change version like this? I mean I can set version to 5 or whatever and it also works.

@mecid I'm not sure what version of coremltools you're using, but I think in the latest version, builder.set_categorical_cross_entropy_loss() will automatically fill in that version number. If not, you can fill in anything you like here, but I would use 4 as that is the correct number. ;-)

@hollance so you mean by setting version to 4 and using builder.set_categorical_cross_entropy_loss() my Model packaging will be updated to latest version?
I on the latest coreml tools beta version.

@mecid I was slightly mistaken, builder. make_updatable() will now automatically fill in the specificationVersion for you. Not sure why that didn't happen automatically in your script, as with the current beta version of coremltools it should do that. This means you shouldn't have to fill in specificationVersion yourself anymore.

@hollance I'm on CoreML tools 3.0b3 and Turi Create 5.7.
Is it safe to use that model with manual setting version to 4? I fear that it can be broken.

It is safe. Note that coremltools 3.0b4 is the latest, not b3.

@hollance thanks, just checked it now, we already have b5.

Ah nice!

Created #2860 to track the more general issue (i.e. this issue for all toolkits that support exporting to Core ML).


update notebooks

@valentina-s @leewujung This addresses issues #76, #77, and #78.
Minimal updates of notebooks, enough to get them running as was mentioned in Wednesday's meeting. The most important change was just pointing the echoregions pip installation at the main branch, instead of an older commit. Larger changes will be made once refactoring and other functionalities are added.

Looks good. Only thing is that now it is pointing to your fork which will change too. You could either pin to the current commit, or the main repo. You can also make them so they automatically work in binder. I actually tried them there but some of the last steps I could not run since the  RAM is only 4GB. Maybe we can think how to reduce the size of the datasets in the example. After the refactoring we can synchronize the datasets to be maybe of same type (nc is a bit easier for download but we currently store data in .zarr by default.)


Free accounts

Currently this package only supports Spotify Premium accounts. This can be fixed by switching:
PRODUCT__PRODUCT_PARTNER

to:
PRODUCT__PRODUCT_CLIENT

and selecting normal or low audio qualities.
https://github.com/ejurgensen/librespot-c/blob/b4412f03a30adfce44624cd6d9731b38a70ffe92/src/connection.c#L903-L905

Thanks a million for the suggestion, and also for looking at the implementation.
As you can see, I copied librespot, and they use PRODUCT_PARTNER. I'm nervous about unintended consequences from straying away from that. I guess you must be well into this subject, so perhaps you can add details about how you know that PRODUCT_CLIENT can be used? Do you know of other implementations using it?


Add event list information in .matter files

Report what events are expected to be generated by each cluster in matter files.
Reviewer notes:

I compared windowapp endpoint_config contents to validate actual output
it seems to me that only a subset of zap files have event list defined (I don't see light switch or pump examples or such). Assuming this is ok and expected.
Our golden image tests do not currently use events as no python codegen depends on it. If we add those, we probably want that as a followup.

> rg --files-without-match --glob '*.matter' 'emits event' examples
examples/light-switch-app/light-switch-common/light-switch-app.matter
examples/dynamic-bridge-app/bridge-common/bridge-app.matter
examples/log-source-app/log-source-common/log-source-app.matter
examples/contact-sensor-app/contact-sensor-common/contact-sensor-app.matter
examples/placeholder/linux/apps/app2/config.matter
examples/chef/devices/noip_rootnode_dimmablelight_bCwGYSDpoe.matter
examples/chef/devices/rootnode_colortemperaturelight_hbUnzYVeyn.matter
.....
examples/chef/devices/rootnode_doorlock_aNKYAreMXE.matter
examples/placeholder/linux/apps/app1/config.matter
examples/ota-provider-app/ota-provider-common/ota-provider-app.matter
examples/thermostat/thermostat-common/thermostat.matter
examples/ota-requestor-app/ota-requestor-common/ota-requestor-app.matter
examples/tv-app/tv-common/tv-app.matter
examples/tv-casting-app/tv-casting-common/tv-casting-app.matter
examples/temperature-measurement-app/esp32/main/temperature-measurement.matter
examples/pump-app/pump-common/pump-app.matter
examples/pump-controller-app/pump-controller-common/pump-controller-app.matter
examples/lighting-app/silabs/efr32/data_model/lighting-wifi-app.matter
examples/lighting-app/silabs/efr32/data_model/lighting-thread-app.matter
examples/lighting-app/lighting-common/lighting-app.matter

Dummy bootstrap force version

Maybe bumping version in https://github.com/project-chip/connectedhomeip/blob/master/scripts/py_matter_idl/setup.cfg#L17 will help with the bootstrap cache issue. The cache itself will not be invalidated, but newer package should be installed. The best way would be to install this package in a developer mode - in such a case only symlink will be cached.


Cleanup: Reorganize top level files to minimize clutter

[x] move azure pipelines to ci folder, reroute in azure
[ ] move js assets to js folder
[ ] configure readthedocs from the readthedocs settings instead of yaml file
[ ] do we actually use pyproject.toml?

do we actually use pyproject.toml?

@timkpaine as far as I know it's the only python way to specify "build time" dependencies ("build time" covering quite a few scenarios).
(ignoring other ecosystems e.g. conda, which has requirements/build or requirements/host in meta.yaml and what have you)

@ceball I understand the logic behind it, but im curious if we actually rely on it anywhere. all part of my considering moving to poetry

@timkpaine using poetry doesn't seem related to removing pyproject.toml, since poetry uses pyproject.toml, but I'm sure you already know that, so I'm still misunderstanding :) But I'm totally onboard with the aim of this issue.
First, I'll just describe some current stuff about celltests. Then I'll try to expand your question and you can correct me where I go wrong.
But before that, the short version is that I think your question might be (I'm really not sure):

What does nbcelltests use from setuptools/jupyter-packaging (an extension of setuptools for jupyter stuff?) that is not supported/available/possible with - or adaptable to - poetry?

current situation
What python config files do we have and why? (I didn't create them ;)

pyproject.toml: lists build deps (i.e. allow pip to know what tools it should install "for setup.py to run"); turns on pip build isolation by default, which means better reproducibility of builds (but if you have long build times, or need to ensure binary compatibility between build/run, and so on, can be annoying...).
setup.cfg:

required to specify that when a wheel's built, it should be universal (i.e. not just pure python, but python 2 and 3 compatible)
used for flake8 config (flake8 can only be configured in setup.cfg, tox.ini, or .flake8).

setup.py:

lists end user run time deps
lists developer deps
lists metadata like authors, version, classifiers, etc
defines some ?setuptools/distutils commands, for handling "jupyter nbextension and lab stuff" (more on this below)

MANIFEST.in: more control over files that end up in distributions (e.g. adds docs, jupyterlab_celltests files, etc)

Please edit and/or correct in a comment!
As an aside, we could also possibly (but probably not) count:

a conda.yaml file on conda-forge, which repeats the metadata and "build"+run+test dependencies
...and the same again if e.g. celltests were installable via apt or whatever else

what does it mean to move to poetry?
I'll try to expand your question - tell me where I go wrong :) Not only do I not know exactly what your question is, I have never actually used poetry, and I don't know what jupyter-packaging is/does (beyond guessing).
Current situation:

nbcelltests is a setuptools-based project.
our build dependencies are setuptools, wheel, and jupyter-packaging
the above build dependencies are required to create a wheel, create or install an sdist, install from github (e.g. pip install git+https://github...), do an editable/dev install (pip install -e . ...), and so on.

What you're considering:

switch nbcelltests to use poetry instead of setuptools+pip for development and package building

Note: this would have no impact on most users, as I expect most users get nbcelltests via a wheel. Wheel installation does not require executing setup.py or any of that stuff - as long as a wheel is generated, users don't care if it was via poetry or you wrote it out by hand ;) Some users will get nbcelltests via sdist (or e.g. from github source), and they will need to execute the build system - they are basically in the same position as developers.
Advantages:

metadata like authors etc would move from python code in setup.py to simpler/static pyproject.toml
dependencies would move from python code in setup.py to simpler/static pyproject.toml
? "one file", pyproject.toml, to configure everything (on the python side!) rather than 4
access to poetry features, e.g. commands to add/remove deps to the project, have a lock file, dependency solver, manage a virtual env, ... ?

Questions:

What does jupyter-packaging do/could it be used with poetry/could it be translated to poetry?
does wheel really need setup.cfg (last time I checked, a couple of years ago: yes)
can flake8 really not use pyproject.toml (...as I recall, pyproject.toml was not originally supposed to be a dumping ground for all tool config, that's just what's happened...apparently some people still resist it? ;) )
does poetry bundle tests with packages? poetry projects I've seen always seem to have tests/ outside the package
... (I've never used poetry, so I have lots of basic questions, plus I also wonder how it interacts with other useful tools such as tox)

Other misc notes:

If we switched to poetry, we/others could still support build systems other than poetry via a tool like dephell (which can e.g. generate setup.py from pyproject.toml).


Optimise RBAC index memory pressure

These are based on 10mio rules
Base is:
➜  server git:(2024.3.x-rbac-opt-v2) ✗ go test -benchmem -memprofile=base_now_10mio.pprof -run=^$ -bench ^BenchmarkIndexBuild_10000000$ github.com/cortezaproject/corteza/server/pkg/rbac
goos: darwin
goarch: arm64
pkg: github.com/cortezaproject/corteza/server/pkg/rbac
BenchmarkIndexBuild_10000000-12                1<PHONE_NUMBER>3 ns/op<PHONE_NUMBER> B/op 71882410 allocs/op
PASS
ok      github.com/cortezaproject/corteza/server/pkg/rbac       14.242s

Trying out improving memory alignment
➜  server git:(2024.3.x-rbac-opt-v2) ✗ go test -benchmem -memprofile=upd_alignment_10mio.pprof -run=^$ -bench ^BenchmarkIndexBuild_10000000$ github.com/cortezaproject/corteza/server/pkg/rbac
goos: darwin
goarch: arm64
pkg: github.com/cortezaproject/corteza/server/pkg/rbac
BenchmarkIndexBuild_10000000-12                1<PHONE_NUMBER>5 ns/op<PHONE_NUMBER> B/op 71883998 allocs/op
PASS
ok      github.com/cortezaproject/corteza/server/pkg/rbac       13.720s

this is... kind of worse... difference negligible so it's whatever
10984989333/10459754625 = 1.05
5192429296/5192614672 = 0.99

Removing some stuff from the structs we get...
➜  server git:(2024.3.x-rbac-opt-v2) ✗ go test -benchmem -memprofile=simpler_structs_10mio.pprof -run=^$ -bench ^BenchmarkIndexBuild_10000000$ github.com/cortezaproject/corteza/server/pkg/rbac
goos: darwin
goarch: arm64
pkg: github.com/cortezaproject/corteza/server/pkg/rbac
BenchmarkIndexBuild_10000000-12                1<PHONE_NUMBER>6 ns/op<PHONE_NUMBER> B/op 71887240 allocs/op
PASS
ok      github.com/cortezaproject/corteza/server/pkg/rbac       14.100s

10984989333/10744250416 = 1.02
5192429296/5028433528 =  1.03
marginally better but the bottom line... same difference

using values we get...
➜  server git:(2024.3.x-rbac-opt-v2) ✗ go test -benchmem -memprofile=no_pointers_10mio.pprof -run=^$ -bench ^BenchmarkIndexBuild_10000000$ github.com/cortezaproject/corteza/server/pkg/rbac
goos: darwin
goarch: arm64
pkg: github.com/cortezaproject/corteza/server/pkg/rbac
BenchmarkIndexBuild_10000000-12                1<PHONE_NUMBER>0 ns/op<PHONE_NUMBER> B/op 71876321 allocs/op
PASS
ok      github.com/cortezaproject/corteza/server/pkg/rbac       14.007s

10984989333/10557726250 = 1.04
5192429296/6178684144 = 0.8
yeesh... yeah no

The TL;DR current change does the following:

some wrapper around the index to hold stuff in memory
some counter to keep track of misses

Theoretically, the most used bits will be in memory with fast access and all the rest will be in the database, accessed on demand (since DB access is generally slower, we can't just go 100% database).
For now, things are indexed/grouped by role and not by resource.
The most memory-efficient version would hold the most used rules but we can't ensure that all needed rules re ready for evaluation (too many DB requests).
Indexing by most used resource vs most used role -- I might implement both as the underlying code shouldn't care too much re. what thing is used. In my head, some users/roles probably do the majority of operations so this is the most logical solution.

Now... to do some benchmarking and do it in a way which won't be a pain to do...
Right now, we're looking for these things:

memory pressure when updating the index: this should be better since we don't re-index everything anymore

memory pressure when re-evaluating the entire index: this should be better since we're holding a smaller chunk; currently there isn't some super special way of making it easier on memory.

performance for evaluating unindexed roles: this should be fairly slower because of overhead

performance for evaluating partially unindexed roles: worst case in mind it'll be the same as unindexed so I might not benchmark this at all

Benchmarking Memory pressure when updating the index
Now, I'm not entirely sure how to do this... I think I'll do it like this

get a profile of the index initializing
use the original code + whatever extra is needed
diff the two profiles
profit

But these benchmarks provide aggregate values... will it be okay if I want to see how well updating the index/reindexing works?
A problem we're solving is that if we have a bunch of rules, re-indexing causes a big spike in memory (purely due to the sheer size).
Is there even a real need to do memory profiling on this level? Since as is, the design limits the rules, the memory consumption is predictable.

Trying out some code to report memory state programmatically...

We get this:
    rbac_rules_test.go:79: Before Benchmark: Alloc = 4415 kB, TotalAlloc = 8717 kB, Sys = 19361 kB, NumGC = 6
    rbac_rules_test.go:79: Index built: Alloc = 7070 kB, TotalAlloc = 14054 kB, Sys = 24225 kB, NumGC = 8
    rbac_rules_test.go:79: Index updated: Alloc = 9552 kB, TotalAlloc = 19112 kB, Sys = 24225 kB, NumGC = 9
    rbac_rules_test.go:79: After Benchmark: Alloc = 9552 kB, TotalAlloc = 19115 kB, Sys = 24225 kB, NumGC = 10

So these numbers seem about the same as they did using pprof, but it's a bit nicer to work with (at least for me)

I added a .Clear() function to the service and I get this
    rbac_rules_test.go:80: Before Benchmark pre gc: Alloc = 4427 kB, TotalAlloc = 8737 kB, Sys = 19617 kB, NumGC = 5
    rbac_rules_test.go:84: Before Benchmark post gc: Alloc = 4427 kB, TotalAlloc = 8737 kB, Sys = 19617 kB, NumGC = 5
    rbac_rules_test.go:80: Index built pre gc: Alloc = 8064 kB, TotalAlloc = 14077 kB, Sys = 23969 kB, NumGC = 7
    rbac_rules_test.go:84: Index built post gc: Alloc = 8064 kB, TotalAlloc = 14077 kB, Sys = 23969 kB, NumGC = 7
    rbac_rules_test.go:80: Index updated pre gc: Alloc = 12137 kB, TotalAlloc = 19138 kB, Sys = 24225 kB, NumGC = 8
    rbac_rules_test.go:84: Index updated post gc: Alloc = 12137 kB, TotalAlloc = 19138 kB, Sys = 24225 kB, NumGC = 8
    rbac_rules_test.go:80: Index cleared pre gc: Alloc = 9568 kB, TotalAlloc = 19147 kB, Sys = 28577 kB, NumGC = 9
    rbac_rules_test.go:84: Index cleared post gc: Alloc = 9568 kB, TotalAlloc = 19147 kB, Sys = 28577 kB, NumGC = 9
    rbac_rules_test.go:80: After Benchmark pre gc: Alloc = 4449 kB, TotalAlloc = 19152 kB, Sys = 28577 kB, NumGC = 10
    rbac_rules_test.go:84: After Benchmark post gc: Alloc = 4449 kB, TotalAlloc = 19152 kB, Sys = 28577 kB, NumGC = 10

Looking at this:
    rbac_rules_test.go:84: Index updated post gc: Alloc = 12137 kB, TotalAlloc = 19138 kB, Sys = 24225 kB, NumGC = 8
    rbac_rules_test.go:80: Index cleared pre gc: Alloc = 9568 kB, TotalAlloc = 19147 kB, Sys = 28577 kB, NumGC = 9
    rbac_rules_test.go:84: Index cleared post gc: Alloc = 9568 kB, TotalAlloc = 19147 kB, Sys = 28577 kB, NumGC = 9

runtime.GC() does nothing... why?

... because I forgot to call runtime.ReadMemStats(&m)...
So..
func printMemStats(b *testing.B, label string) {
	var m runtime.MemStats

	runtime.ReadMemStats(&m)
	b.Logf("%s pre gc: Alloc = %v kB, TotalAlloc = %v kB, Sys = %v kB, NumGC = %v\n",
		label, bTokB(m.Alloc), bTokB(m.TotalAlloc), bTokB(m.Sys), m.NumGC)

	runtime.GC()
	runtime.ReadMemStats(&m)
	b.Logf("%s post gc: Alloc = %v kB, TotalAlloc = %v kB, Sys = %v kB, NumGC = %v\n",
		label, bTokB(m.Alloc), bTokB(m.TotalAlloc), bTokB(m.Sys), m.NumGC)
}

gives
    rbac_rules_test.go:80: Before Benchmark pre gc: Alloc = 4430 kB, TotalAlloc = 8734 kB, Sys = 19873 kB, NumGC = 5
    rbac_rules_test.go:85: Before Benchmark post gc: Alloc = 4429 kB, TotalAlloc = 8737 kB, Sys = 19873 kB, NumGC = 6
    rbac_rules_test.go:80: Index built pre gc: Alloc = 7872 kB, TotalAlloc = 14072 kB, Sys = 23969 kB, NumGC = 7
    rbac_rules_test.go:85: Index built post gc: Alloc = 7079 kB, TotalAlloc = 14078 kB, Sys = 23969 kB, NumGC = 8
    rbac_rules_test.go:80: Index updated pre gc: Alloc = 12139 kB, TotalAlloc = 19138 kB, Sys = 24225 kB, NumGC = 8
    rbac_rules_test.go:85: Index updated post gc: Alloc = 9565 kB, TotalAlloc = 19146 kB, Sys = 28833 kB, NumGC = 9
    rbac_rules_test.go:80: Index cleared pre gc: Alloc = 9568 kB, TotalAlloc = 19148 kB, Sys = 28833 kB, NumGC = 9
    rbac_rules_test.go:85: Index cleared post gc: Alloc = 4445 kB, TotalAlloc = 19149 kB, Sys = 28833 kB, NumGC = 10
    rbac_rules_test.go:80: After Benchmark pre gc: Alloc = 4448 kB, TotalAlloc = 19152 kB, Sys = 28833 kB, NumGC = 10
    rbac_rules_test.go:85: After Benchmark post gc: Alloc = 4445 kB, TotalAlloc = 19153 kB, Sys = 28833 kB, NumGC = 11

🎉

It might be worth looking into this change:
Instead of relying solely on roles, it might be worth indexing by resource and role. When I was discarding indexing by rules, the rationale was that many cases would still need database access (controlling memory consumption would be trivial then).
If we do roles and resources we could do the following:

Determine most used resources
Determine most used roles that have rules for said resources
Pull all rules for the role resource combo

Doing it like this would reduce the amount of rules while assuring fast lookups for common resources & roles.
Now... how do we change the change to do this...
Should we use two counters? One for roles, one for resources?
Should we use one big index? I think it'll be a bit of a pain to figure out what rules to yank out of the index... would it be more sane to use multiple little indexes? One per resource might be a bit insane so let's consider one per role...
So we'd have the wrapper service with two counters; one for used roles and one for used resources. When some resource is determined to be in high demand, the lowest in demand resource would be yanked out and the new one would be indexed. The index would be updated with N most in-use roles.
Will it be a pain to know what to yank out?
If we do multiple indexes, one per role, that'll be more of a pain.
If we index per resource, we'll get some duplication which will also be a pain (I think)...
So I think that one big index will be enough; since we're dealing with resources, those aren't really hard to take out.

Would it make more sense to combine the role & resource and use the resulting string to track usage?
It'll probably use a bit more space but I think it'll be easier to do; so let's go with this one for now.
But now then... how do we figure out what resources we have available?
We can't really rely on analysing rules since most of them won't be resource-specific (except for some scenarios).
A cold start?
A combination of roles and resources?
going through some records, modules, ... ?
If we go down this route we'd definitely need to store this somewhere -- it's kind of expensive to do on each cold start...

So... this is what we'll do:
The design will be as I've originally envisioned it, but instead of tracking usage based on roles (to see what needs to get indexed), we'll use roles & resources. Doing it like so should provide more control over how many RBAC rules and which specifically we keep in memory.
The only scenario (that I could think of) where using roles might be better is when each user has their own role (or a small set) and each role has a modest amount of RBAC rules attached (there isn't much composition going on). For this scenario, using roles, we'd probably get more index hits and less database usage.
With that said, mixing in resources into the equation we address a lot of the issues we'd encounter while keeping the original benefits (minus some memory reduction).
Upsides, downsides
Upsides

More control over how many RBAC rules we keep in memory
we don't need to keep unused resources even though they belong to a role

Downsides

The usage counter will take up more memory since it'll need to store ~100 runes instead of an uint64 per entry (a combination of role ID + resource identifier). If we use a hashmap, that'll be about 100MB/1mio entries. We probably won't keep that many entries to began so it should be fine. We can also use some suffix tree to compress this a bit (co pilot sais it'll reduce memory to ~30MB/1mio entries).
It'll be a bit tricky to preload what resources are available since we can't purely rely on rules. I think we can just pull out random records and a few modules, and users at the end. We'll most likely need to flush the resulting states to the database so future startups won't be so random-esk. A cold start might be fine as well but that's TBD...

Added points for improvement

The wrapper should enable alternative usage tracking methods if we need to add more to cover/improve usecases
The wrapper should let users disable (regardless, we might need to keep some bits regarding context roles in memory) the index/counters all together to solely rely on the database to reduce memory usage as much as possible.
The wrapper should let users disable usage tracking to pull everything in the index (so it works like it does now) since this improvement only applies to large(r) systems.
The wrapper should not rely on the standard store.Storer interfaces to access RBAC rules; it should be a wrapper around it so arbitrary implementations (external providers) could be plugged in.

Problems to address

Resource usage might drop off after a while so we'll need to add some TTL (or similar). Iirc there were some mechanisms to combat that (thanks uni) so I'll just use one of those bad boys.
Some bits need to check if a resource/role combo was added to the index... keeping another list would probably be insane so I'll probably offload that to the counter. I don't think that's that big of a deal since it wan't intended to be generic for anything (at least for now).

Some bits need to check if a resource/role combo was added to the index... keeping another list would probably be insane so I'll probably offload that to the counter. I don't think that's that big of a deal since it wan't intended to be generic for anything (at least for now).

Do they though? Since access to the index is much faster compared to the DB, does it matter if I fetch missing bits from the DB while collecting bits from the index? Theoretically, the gained time should be too small to notice. Will do some benchmarks and see if we want to change.

Resource usage might drop off after a while so we'll need to add some TTL (or similar). Iirc there were some mechanisms to combat that (thanks uni) so I'll just use one of those bad boys.

I'll use a LRFU counter to combat all this. Might opt-in to log/track some extra stats to show reports. Using LRFU instead of just LRU or LFU since both introduce valid cases for Corteza where they would fall flat. LRFU provides an option to generically cover all cases or can be configured to cover either one or the other with a more/less aggressive λ.
If I feel prestige I might implement, along with some stats, a model to dynamically adjust the λ based on usage.


Code error on latest build, runs fine older build

I finally updated to the latest version of LP, along with updating all of my libraries. However, running my code throws an error for a missing closing brace, when I have looked over it for about an hour, and the exact same code runs fine on my older build. Unfortunately I can't seem to isolate it, so I included both versions of LovePotion, and the game folder. I'll keep looking into it, but not tonight since it's kinda making me frustrated lol. Thanks in advanced.
old_build.zip
new_build.zip
game.zip

Alright, I'll look into it tonight

Xavyrr, I'll try to get some of my latest work in when I can. It'll have audio streaming for Ogg Vorbis, but not wave. This build should be fine when I push it (hence the fact I'm making sure it passes builds while working :P ).


switch from 'groups' to 'sections'

What is the context of this PR?
Switches to using 'sections' instead of 'groups' from API
How to review
Run tests, ensure all pass. Manually test in browser

 Author API - Rename Groups -> Sections (2)


Fix line breaks in documentation

Just fixing line wrapping everywhere.
If possible I'd like this to be merged fast-forward (no squashing) because I think there's a bit of a difference between general paragraph wrapping across the documentation (no line breaks) vs careful manual semantic line wrapping in the list of references.

I've now force-pushed a version without line breaks at all. Apparently RTD thinks it's fine, and I trust that Travis won't complain (since I didn't touch anything but the contentns of some rst files anyway).

I've now force-pushed a version without line breaks at all. Apparently RTD thinks it's fine, and I trust that Travis won't complain (since I didn't touch anything but the contentns of some rst files anyway).

I've now force-pushed a version without line breaks at all. Apparently RTD thinks it's fine, and I trust that Travis won't complain (since I didn't touch anything but the contentns of some rst files anyway).

Mostly LGTM! Just in case you missed it, though, there's one small review comment remaining here: https://github.com/neuropoly/spinalcordtoolbox/pull/3197#discussion_r565571969

I've now force-pushed a version without line breaks at all. Apparently RTD thinks it's fine, and I trust that Travis won't complain (since I didn't touch anything but the contentns of some rst files anyway).

Mostly LGTM! Just in case you missed it, though, there's one small review comment remaining here: https://github.com/neuropoly/spinalcordtoolbox/pull/3197#discussion_r565571969


Windows Support

Should look into supporting the Windows secret storage doodad.

https://stackoverflow.com/a/9228105
https://docs.microsoft.com/en-us/windows/win32/api/wincred/nf-wincred-credreadw
https://docs.microsoft.com/en-us/windows/win32/api/wincred/nf-wincred-credwritew


Some tests wrongly assume callbacks are delivered with "exactly once" semantics

An example is TckContextTests#testForget that asserts that the forget callback is delivered exactly once. For example the notification could reach the participant but the response could be lost which would cause the implementation to reissue the forget call. Our delivery semantics is "at least once".

@mmusgrov are you going to work on this or can I take it?

@mmusgrov are you going to work on this or can I take it?

I assigned the issue to you, thanks


Add Nvidium

Sometime in the future I plan on including Nvidium in Adrenaline by default as it disables itself on unsupported GPUs and is in a relatively stable state by now. Will require a lot of testing though. This issue is here to track progress.
@MCRcortex
GitHub Repository: https://github.com/MCRcortex/nvidium
Issue Tracker: https://github.com/MCRcortex/nvidium/issues
Modrinth Page: https://modrinth.com/mod/nvidium

The main issue right now is that I personally don't have a compatible Nvidia GPU to test this with. I will need to rely on community support and feedback.

Personally, ive seen performance increases up to ~2.5x with Nvidium
i think that its a very good mod, and i feel it may worth it to add?
It is much more stable than it was upon first release

Performance probably varies decently between graphics cards tho. I could do some benchmarking if you'd like (you mentioned you do not have a compatible gpu)

I could do some benchmarking if you'd like (you mentioned you do not have a compatible gpu)

Benchmarks aren't necessary but could be helpful. The main thing I'm worried about is possible (but very rare) issues people may experience during gameplay, such as crashes and whatnot.

I see. I've personally never had a crash related to Nvidium (in modpacks with >300 mods aswell)
I've played with probably around 100 hours with the mod and to my experience its stable enough, but could probably vary for other people

My experience with him is highly positive. Even though I keep fiddling with all its settings like crazy, hoping that at some point it will blow up my computer, it has never broken or even crashed in any of my uses. It's just perfect.

It is quite good as a chunked distance increase mod, it performs better than Distants Horizons, it would be nice if it was included but disabled by default.

Benchmarks aren't necessary but could be helpful.

Decided to quickly benchmark the most recent version of Adrenaline with and without Nvidium
I have a 3060 for my GPU. Tested on the same world at 32 render distance and 32 simulation distance. Elsewise all settings were modpack default

I saw a performance increase of ~35% which is a major increase. Hope it helps 👍

Thanks! This is a nice thing to have. I've heard from some of my friends that they had a pretty noticeable improvement as well

cortex said that personally it should be fine for general usage and many people have said it's good so i'll just add this now

Will this change be put into Additive as well? Curious

yes, changes from Adrenaline get put into Additive too (when possible)

👍 hope to have helped in some way with this

I've temporarily removed Nvidium until https://github.com/MCRcortex/nvidium/pull/173 is merged

Pull request is merged and version is published, will add to Adrenaline


handlers: quick fix for remote api handler when remote is string

https://its.cern.ch/jira/browse/INSPIR-3317

@zzacharo in case of an error currently it's ignoring if you want to use the a different handler than the default one

@drjova yes in case of error there was no remote at all so we had to make some changes. Personally, I would go probably for a global default error handler so it can be used in the views where no remote is available and thus you cannot use the response_handler factory function. Then, I would remove the if remote...  check from the response_handler as this was needed only to handle errors in the rest views. WDYT? Also bear in mind that @lnielsen will most probably work on this and refactor a bit the code but certainly include all the features we had agreed during our sprint :)

@zzacharo 👍  global default error handler works for us as well. It is actually how @pazembrz has implemented it, we can refactor a bit the PR to reflect this.

@zzacharo we've updated the PR

Ok, I had to add separate error_handler for response_handler as response_handler is called directly from views.
@zzacharo  @drjova

Closing in favour of #205


Mapper does not update.

Corp uses Pathfinder. No issues for anyone, except me: Mapper does not update. I can add and delete systems manually, but it wont show me the chains entered by our members. Whatever they edit, i can not see it. I see two statics and a chain of 3 wormholes, and that has been the same for me since corp started using the mapper.
Ive tried various browsers, with no luck. Deleted account and create a new one, with same result.

I´m on it thanks! Duplicate: #357

http://prntscr.com/cxj5oq
Thats what i been getting, without any change. Can the problem be on my end
somehow? My ingame name is MD74.
Thanks!
On Sat, Oct 22, 2016 at 3:13 PM, Mark Friedrich<EMAIL_ADDRESS>wrote:

Closed #356 https://github.com/exodus4d/pathfinder/issues/356.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/exodus4d/pathfinder/issues/356#event-832801503, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AV7e1bHUebJ9ktHtxa-P87Mnp3a9ymFAks5q2gvmgaJpZM4Kdy-x
.

Are you sure you're not using the -w.space pathfinder while your corp uses
an internally hosted one?
On Saturday, October 22, 2016, MDDG<EMAIL_ADDRESS>wrote:

Corp uses Pathfinder. No issues for anyone, except me: Mapper does not
update. I can add and delete systems manually, but it wont show me the
chains entered by our members. Whatever they edit, i can not see it. I see
two statics and a chain of 3 wormholes, and that has been the same for me
since corp started using the mapper.
Ive tried various browsers, with no luck. Deleted account and create a new
one, with same result.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/exodus4d/pathfinder/issues/356, or mute the thread
https://github.com/notifications/unsubscribe-auth/AJ8ssPmrWoTX-w24ytOnS09ET0vxcMkFks5q2cMqgaJpZM4Kdy-x
.


Conditional derive

When implementing REST API clients, I usually derive Serialize and Default conditionally for tests, like so:
#[derive(Debug, Deserialize)]
#[cfg_attr(test, derive(Serialize, Default))]
pub struct SlavaUkraini {
    pub field: String,
    pub something: u64,
}

This enables me to use Default and Serialize in tests to easily and quickly instantiate mock responses, while at the same time keeping the prod binary smaller and prod compile times lower.
Unfortunately conditional derive doesn't currently seem possible in nutype.
Would it be possible to enable nutype to do something similar? Like
#[nutype(derive(Debug, Deserialize, AsRef), cfg_attr(test, derive(Default, Serialize))]
pub struct HeroyamSlava(String);

#[derive(Debug, Deserialize)]
#[cfg_attr(test, derive(Serialize, Default))]
pub struct SlavaUkraini {
    pub field: HeroyamSlava,
    pub something: u64,
}

Or maybe
#[nutype(derive(Debug, Deserialize, AsRef)]
#[cfg_attr(test, nutype(derive(Default, Serialize))]
pub struct HeroyamSlava(String);

Whatever makes the most sense. I am a complete noob when it comes to Rust macros, and proc macros especially, so no idea what the best solution would be.
This request is not super important, as it works just fine without the conditional derive, it would just be nice to have.
Actually now that I think about it, it would also be nice to have this for libraries wanting to conditionally derive on feature flags, not just on tests.

Hi!
I knew this is gonna be requested some day, this day has come :)
Technically there is nothing that limits us to implement it, it will just require some effort though.
Whatever goes into #[nutype()] is pure DSL.

Is there a reason why you can't just let the user add separate derives?

Is there a reason why you can't just let the user add separate derives?

Answered in https://github.com/greyblake/nutype/issues/191#issuecomment-2434690325
I copy answer here for other readers:

Nutype takes a full control over derive(..)  to ensure that it's not possible to derive traits that possibly can mutate value avoiding the constraints. E.g. deriving DerefMut would make a problem.

I guess I need to add an FAQ section

Hi! I knew this is gonna be requested some day, this day has come :)
Technically there is nothing that limits us to implement it, it will just require some effort though. Whatever goes into #[nutype()] is pure DSL.

Is there a short term fix that can be applied in the meantime?

@LockedThread Which trait do you need?
What it is possible is to implement a trait manually in terms of other traits. (if it's very repetitive, a macro rules can be used).

@LockedThread Which trait do you need?
What it is possible is to implement a trait manually in terms of other traits. (if it's very repetitive, a macro rules can be used).

Serde traits, enabled by feature.
I wasn't able to use cfg_attr to enable them. However, I was able to make 2 modules and enable those based on the feature and then reexport the nutype's defined inside of them.
Not a very clean solution, but it works.


Select Form Validation not working

I was testing your example code but it was not working for me.
Decided to look under the hood and data-invalid is only applicable to input and textarea elements in forms.scss and carbon-component.js.
I've fixed it on my end and hopefully you can add it on the future build.

Thanks for catching that! I'll submit a PR now to fix the issue, as well as add an example to form.html


base directory for PLUGINS

Hi, we have recently insatlled netbox 2.8.4, in that version plugins directory moved to "project-static", so after installation of nextbox-ui-plugin we have copied all content from "static" directory to "project-static" and it works with some issues (not showing some icons, "hide passive" didnt work, broken font), so I think problem that some files had not correct link to directories

Please try to install the latest version.
All links on the topology page are now rendered from the {% static %} template.


config.stories does not accept an array of strings as it should

Describe the bug
globby can accept a string or an array of strings as patterns to search for, but the ladle build script assumes that config.stories is a single string and specifying an array of strings leads to a fatal error:
TypeError:  Patterns must be a string or an array of strings
This is due to the build script wrapping config.stories in an array, and thus if the user configures an array of strings we end up with an array of a single array of strings, rather than an array of strings.
See line 39 of build.js.
Reproduction
https://stackblitz.com/edit/ladle-gwkvsl?file=package.json
DEBUG=ladle* npm run dev
DEBUG=ladle* npm run build

Environment

OS: macOS Monterey 12.4
Version: 2.0.2

What's your naming scheme? Even a single glob is pretty flexible.

In a mono-repo where we'd like to aggregate stories across the project into a single page. Desired config is: stories: ['../src/src/**/*.stories.tsx', './stories/*.stories.tsx', '../../icons/src/*.stories.tsx']
I guess that could make be expressed in a single glob: ../**/*.stories.tsx, but that might capture more than intended.
It seems like a simple fix to enable multiple globs, tho, which would definitely allow this config variable to be more expressive:
const entryData = await getEntryData(await globby(Array.isArray(config.stories) ? config.stories : [config.stories]));
I was planning to submit a PR for this later today.

I was planning to submit a PR for this later today

Please do.


AndBeyond.Media adapter: update bidder code

@AndBeyondMediaHB - you have a case problem here. Unix is case sensitive and everything needs to match:

your actual PBJS bid adapter file name is andBeyondMediaBidAdapter.js -- mixed case
the biddercode in the adapter is all lowercase https://github.com/prebid/Prebid.js/pull/9005/files
the biddercode in this docs PR is lowercase

The download page works by looking for the biddercode from this file and trying to build it. There's no such adapter as andbeyondmedia.js. It's andBeyondMediaBidAdapter.js
Please update the PRs to make the case consistent however you prefer.

ok - changing the biddercode to "beyondmedia" for Prebid.js will solve the website download problem, but not you've introduced another problem -- you're not matching the biddercode for your server-side adapter.
I'm afraid I have to insist that your client-side and server-side bidder code match. Publishers need to be able to use the Server-to-Server testing module, which they wouldn't be able to do for your service if you've got different codes for these adapters. Let's get this right.
Will let you choose whether to use "beyondmedia" or "andbeyondmedia" - just settle on one please. And if desired, you can define an alias on either or both client- and server-sides to use both names!

ok - changing the biddercode to "beyondmedia" for Prebid.js will solve the website download problem, but now you've introduced another problem -- you're not matching the biddercode for your server-side adapter.
I'm afraid I have to insist that your client-side and server-side bidder code match. Publishers need to be able to use the Server-to-Server testing module, which they wouldn't be able to do for your service if you've got different codes for these adapters. Let's get this right.
Will let you choose whether to use "beyondmedia" or "andbeyondmedia" - just settle on one please. And if desired, you can define an alias on either or both client- and server-sides to use both names!

@bretg Thanks, we decided to keep the name beyondmedia.
https://github.com/prebid/prebid-server/pull/2408

Prebid.js PR - https://github.com/prebid/Prebid.js/pull/9005
Prebid Server (GoLang) PR - https://github.com/prebid/prebid-server/pull/2408
@AndBeyondMediaHB - do you have existing clients using the name "andbeyondmedia"? If so, you'll need to add an alias or this will be considered a breaking change.

@bretg
If none of our partners has used the adapter yet, can we change the bidderCode without any problems ?
If this is true, could you confirm that it is good ?)

Confirmed the code release went out with 7.19


some excluded tests are running

Some tests are excluded by labels are actually not excluded. For example:
java/net/MulticastSocket/Test.java                             https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/699    linux-s390x aix-all

are running
https://ci.adoptopenjdk.net/view/Test_openjdk/job/openjdk8_j9_openjdktest_ppc64_aix/69/#showFailuresLink

Looks like the labels are separated by spaces, which shouldn't.
The document says 'More than one label is allowed but must be on the same line comma seperated, without spaces.'


What is the use of 'url' option in site configuration

I referred to the documentation in README. There is an option named 'url' which I can not find in jekyll's options. I tried to remove it and I found my site can still work without any difference.
I wonder whether the 'url' option is required in this theme and how it take effects.

Allo,
The url is simply the url which the jekyll will be served at once built. When run locally for dev it will be accessible via localhost.
You can have a loot at it in the jekyll configuration options at the name site.url


zstd CLI log lines truncate file names to last 20 characters

Any reason why we don't print out the full file name?
davidlam-mbp:zstd davidlam$ touch my_very_long_file_name_that_is_over_20_characters_long
davidlam-mbp:zstd davidlam$ echo my_very_long_file_name_that_is_over_20_characters_long  | wc -c
      55
davidlam-mbp:zstd davidlam$ ./zstd ./my_very_long_file_name_that_is_over_20_characters_long -o /dev/null
r_20_characters_long :1300.00%   (     0 =>     13 bytes, /dev/null)

For comparison, gzip prints out full file name in verbose mode. If this is a bug, I can send up a pull request.
davidlam-mbp:zstd davidlam$ gzip --verbose my_very_long_file_name_that_is_over_20_characters_long
my_very_long_file_name_that_is_over_20_characters_long:   -99.9% -- replaced with my_very_long_file_name_that_is_over_20_characters_long.gz

This was a UI choice,
to avoid mangled output when some files end up being very long.
This is more useful when there are a lot of files of variable name length.
Though this can be changed if there are better choices.

@Cyan4973, I was confused by this truncation and personally would like to see the full file name (I don't know of any Unix CLI that shortens file params in output like this FWIW); I think that if the user has long file names already, then they would not be bothered by very long log lines. I'll send a pull request.

Fix got merged into dev, closing for you.


Request: Allow user to specify cookies

Imagine a use case where you want to import the cart page from an ecommerce site populated with items.  Allowing the user to add a "cart_id" cookie would allow the plugin to return a screenshot from an existing cart.
Other use cases:

Sites/apps protected by log ins
Sites that have dynamic content based on a cookie (e.g. popups/promos)

Ah yes, thanks for the feedback! The way I plan to handle use cases like this is to make a chrome extension where you can log in to a page (or do anything else on a page - like add items to cart and open the cart) and then parse and import that to Figma
Does that sound like a fitting solution to your request here as well?

Love it!

Chrome extension has been created, just waiting on Google to approve it now 😃

Chrome extension published https://chrome.google.com/webstore/detail/efjcmgblfpkhbjpkpopkgeomfkokpaim


Localization support

It'd be great to have the walk in Spanish and Hmong at least.

Jekyll does not support localization out of the box.

https://forestry.io/blog/creating-a-multilingual-blog-with-jekyll/
https://github.com/kurtsson/jekyll-multiple-languages-plugin
https://medium.com/@desfocado/how-to-make-jekyll-multilingual-c13e74c18f1c
https://mcpride.github.io/posts/development/2018/02/25/localized-jekyll-pages-without-plugin/ (https://github.com/mcpride/mcpride.github.io)

I used the https://mcpride.github.io/posts/development/2018/02/25/localized-jekyll-pages-without-plugin/


Cant build on windows

Describe the bug
When trying to build on windows I get the error
Missing script: "electron:build:windows"
To Reproduce
Clone repo and use
npm run electron:build:windows
Expected behavior
It will build
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

Electron application
OS: Windows
Browser chrome
Version 132.0.6834.15

I updated the README. Since the project is moving to Tauri, you need to install Rust for building.


Sync refactored API with spec

I mentioned in ooni/spec that most backends description is now obsolete. I think I don't have bandwidth to do more than that.


[MAINTENANCE] Delete or redesign of CNF configuration

Summary:
We need to evaluate the following three sample CNFs, which are currently not used in tests, are necessary to keep in the repository:

sample-coredns-cnf2
sample-coredns-cnf3
sample-coredns-cnf4

Suggestions:

DELETION of samples
If deletion is not appropriate, we need to consider redesigning the cnf-testsuite.yaml file, focusing on the helm_install_namespace parameter.

Consider if the helm_install_namespace parameter is necessary
If the parameter is necessary, propose changing its value from cnfspace[1-4] to more descriptive name such as cnf-coredns[1-4] or cnf-default[1-4]

duplicate to #2073