Include path missing

I couldn't make the "Quick Setup with CMake FetchContent" working without adding the following line:
include_directories(_deps/cpptrace-src/include)
P.S. Making a fork for one line in the README seemed like too much overhead.

Hello, thanks for the report! I'll look into this. What did you specify for GIT_TAG? I'd recommend v0.1.1 as main is somewhat unstable at the moment (working on overhauling some internals).

Thanks for the quick reply. I used the latest commit SHA. Did not try with the tag. I'll do that now.

Hi @mvitorovic, things have now stabalized with the reworking I've done and it looks like cmake + fetchcontent is working.
(I tested with:
cmake_minimum_required(VERSION 3.8)

project(demo_project VERSION 0.0.1 LANGUAGES CXX)

add_executable(main main.cpp)

include(FetchContent)
FetchContent_Declare(
  cpptrace
  GIT_REPOSITORY https://github.com/jeremy-rifkin/cpptrace.git
  GIT_TAG        main # <HASH or TAG>
)
FetchContent_MakeAvailable(cpptrace)
target_link_libraries(main cpptrace)

target_compile_features(main PRIVATE cxx_std_11)

)


KAFKA-16529; Implement raft response handling

This change implements response handling for the new version of Vote, Fetch, FetchSnapshot, BeginQuorumEpoch and EndQuorumEpoch. All of these responses were extended to include the leader's endpoint when the leader is known.
This change also includes sending the new version of the requests for Vote, Fetch, FetchSnapshot, BeginQuorumEpoch and EndQuorumEpoch. The two most notable changes are that:

The leader is going to include all of its endpoints in the BeginQuorumEpoch and EndQuorumEpoch requests.
The replica is going to include the destination replica key for the Vote and BeginQuorumEpoch request.

QuorumState was extended so that the replica transitions to UnattachedState instead of FollowerState during startup, if the leader is known but the leader's endpoint is not known. This can happen if the known leader is not part of the voter set replicated by the replica. The expectation is that the replica will rediscover the leader from Fetch responses from the bootstrap servers or from the BeginQuorumEpoch request from the leader.
To make sure that replicas never forget the leader of a given epoch the unattached state was extended to allow an optional leader id for when the leader is known but the leader's endpoint is not known.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Unrelated failures. Mergning.


Mapping

ISSUE TEMPLATE ver. 0.2.0

Which TurtleBot3 you have?

[ ] Burger
[ ] Waffle
[* ] Waffle Pi

Which SBC(Single Board Computer) is working on TurtleBot3?

[* ] Raspberry Pi 3
[ ] Intel Joule 570x
[ ] etc (PLEASE, WRITE DOWN YOUR SBC HERE)

Which OS you installed in SBC?

[* ] Ubuntu MATE 16.04 or later
[ ] Raspbian
[ ] etc (PLEASE, WRITE DOWN YOUR OS)

Which OS you installed in Remote PC?

[* ] Ubuntu 16.04 LTS (Xenial Xerus)
[ ] Ubuntu 18.04 LTS (Bionic Beaver)
[ ] Linux Mint 18.x
[ ] etc (PLEASE, WRITE DOWN YOUR OS)

Write down software version and firmware version

Software version: [x.x.x]
Firmware version: [x.x.x]

Write down the commands you used in order

roslaunch turtlebot3_slam turtlebot3_slam.launch

Copy and Paste your error message on terminal

HERE

Please, describe detailedly what difficulty you are in

I can running the slam node, but the mapping result is poor. When the turtlebot run in a straight line, the map is good as expected. But when the turtlebot is turning, the map will confuse and be built in wrong places. Is this problem caused by the odometry error? Do I need to calibrate the odomtery? Thank you very much.

@Babaevzhu
If the speed of the robot is too fast, that could happen. Would you check if the robot odom values are correct while turning using $ rostopic echo [topic_name]
If you still need more help with this issue, attach screenshots of your Rviz after mapping that can help us figure out the problem better.
Regards,
Ryan

Thanks for your replying. The turning speed is just 0.1, it should not be too fast.
The odom output look regular as following, and I upload an image of mapping result.
Regards,
Ryan

Sorry, I mean I put the "rostopic echo odom" output at the end of comment.
I face another issue now, when roslaunch the slam node, the map can't be built and always reports a warning "Failed to compute laser pose, aborting initialization (Lookup would require extrapolation into the future.  Requested time<PHONE_NUMBER>.604781301 but the latest data is at time<PHONE_NUMBER>.448156116, when looking up transform from frame [odom] to frame [base_footprint])"
DO you know how to solve this issue?
Best Regards

@Babaevzhu
Instead of just copypasting the terminal output, you should check it with the actual robot's movement. And If you think there is no problem with odometry, try below and see if it works.
(http://emanual.robotis.com/docs/en/platform/turtlebot3/faq/#timesync-between-turtlebot3-and-remote-pc)
If it does not work, reinstall Ubuntu & ROS packages following the tutorial. You should let us know if you make any modifications while following it.
(http://emanual.robotis.com/docs/en/platform/turtlebot3/getting_started/#getting-started)
Let us know if it doesn't work,
Ryan

This issue will be closed since there were no actions for a while. You can reopen this issue to show this issue to the users whenever. Thanks.


ctrl+click MouseEvent.button change in Firefox

Gecko recently landed https://bugzilla.mozilla.org/show_bug.cgi?id=1615732, which makes it so ctrl+click is no longer reported as right click on Mac (MouseEvent.button=0 as opposed to MouseEvent.button=2 before).
The behavior is put behind dom.event.treat_ctrl_click_as_right_click.disabled preference and only enabled in Firefox Nightly.
Since then we got a report about a problem with a context menu in Google Sheets where a context menu opened with ctrl+click opens and closes immediately, making it unusable (https://bugzilla.mozilla.org/show_bug.cgi?id=1638872).
This problem is not reproducible in Chrome despite the fact that ctrl+click is reported the same way as in Nightly (MouseEvent.button=0). So we tracked it down to this function where the event is handled differently based on UA detection:
https://github.com/google/closure-library/blob/719529feafc9d60eecd2f620756d3e5cc84a9250/closure/goog/events/browserevent.js#L379-L392
Since Gecko will be reporting ctrl+click the same way as in Chrome, would it be possible to change the above condition to handle it similarly?

This is the STR for the Google Sheets issue:
1. Open a google sheet document in https://docs.google.com/spreadsheets/
2. press and hold control + mouse left button.
3. Wait for the menu to show up.
4. Move mouse left a bit to make it is not over the menu.
5. The menu will be closed as soon as I release the mouse button.

My understanding here is that there should never be a situation in which we treat Ctrl+Button0 as anything but a right click on a Mac, regardless of user agent.  So the easy fix should be to just drop the WEBKIT check.
Unfortunately, this house of cards will collapse in a few months when userAgent deprecation makes the platform check no longer accessible synchronously.  So we'll need to figure out what to do with that.


added support for unit testing and test results

Can one of the admins verify this patch?

@csharon / @jpoo90
I seem to have an issue that the test-results.xml file is not seen as there by by node, although other files are.  Thoughts?

ok to test

ok to test

ok to test

ok to test

Aside from the coverage plugin not working quite right in Jenkins, this is working as expected.  Created a follow task to merge all the results into one directory - issue #5
cc: @simoami / @jpoo90 / @csharon

Looks good to me. I tested locally and it is working.

move chai to dependencies


conflict between seaborn==0.11 and recent fgclustering pulls in deprecated sklearn package

running pip install -r requirements.txt in a fresh Python 3.10 venv shows a message that the sklearn package is deprecated and aborts. It seems that the requirement seaborn==0.11.2 forces fgclustering to version 0.3 because the newer versions require seaborn >= 0.12. The old version, in turn, depends on the deprecated package. Switching the line in requirements.txt to seaborn>=0.12 installs seaborn 0.13 and fgclustering 1.0.3, which pulls in the new package scikit-learn. I haven't checked if that seaborn upgrade breaks anything.

Thank you for the message. We removed the restriction on the seaborn version and this solves the issue.


Error installing with pip

Describe the bug:
Hi there, hope it's fine for me to submit a bug here. I was looking to use this package for a project, but I seem to be hitting an error on installing it (see below for the traceback).
It looks like it may have something to do with a file in this repository, so I wanted to flag it for your attention. (error: command 'swig' failed: No such file or directory)
Reproducing code example:
pip install --upgrade pip setuptools wheel numpy
pip install --upgrade tfc

Error message: (starting from traceback)
Collecting tfc
  Using cached tfc-1.0.1.tar.gz (62 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting jaxlib
  Using cached jaxlib-0.3.25-cp310-cp310-macosx_11_0_arm64.whl (51.6 MB)
Collecting colorama
  Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting jaxtyping
  Using cached jaxtyping-0.2.9-py3-none-any.whl (18 kB)
Collecting annotated-types
  Using cached annotated_types-0.4.0-py3-none-any.whl (11 kB)
Collecting yattag
  Using cached yattag-1.14.0-py3-none-any.whl
Collecting graphviz
  Using cached graphviz-0.20.1-py3-none-any.whl (47 kB)
Requirement already satisfied: matplotlib in ./world/lib/python3.10/site-packages (from tfc) (3.6.2)
Requirement already satisfied: numpy>=1.23.0 in ./world/lib/python3.10/site-packages (from tfc) (1.23.5)
Collecting jax
  Using cached jax-0.3.25-py3-none-any.whl
Collecting opt-einsum
  Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting typing-extensions
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Requirement already satisfied: scipy>=1.5 in ./world/lib/python3.10/site-packages (from jax->tfc) (1.9.3)
Collecting typeguard>=2.13.3
  Using cached typeguard-2.13.3-py3-none-any.whl (17 kB)
Requirement already satisfied: packaging>=20.0 in ./world/lib/python3.10/site-packages (from matplotlib->tfc) (21.3)
Requirement already satisfied: pyparsing>=2.2.1 in ./world/lib/python3.10/site-packages (from matplotlib->tfc) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7 in ./world/lib/python3.10/site-packages (from matplotlib->tfc) (2.8.2)
Requirement already satisfied: contourpy>=1.0.1 in ./world/lib/python3.10/site-packages (from matplotlib->tfc) (1.0.6)
Requirement already satisfied: pillow>=6.2.0 in ./world/lib/python3.10/site-packages (from matplotlib->tfc) (9.3.0)
Requirement already satisfied: kiwisolver>=1.0.1 in ./world/lib/python3.10/site-packages (from matplotlib->tfc) (1.4.4)
Requirement already satisfied: fonttools>=4.22.0 in ./world/lib/python3.10/site-packages (from matplotlib->tfc) (4.38.0)
Requirement already satisfied: cycler>=0.10 in ./world/lib/python3.10/site-packages (from matplotlib->tfc) (0.11.0)
Requirement already satisfied: six>=1.5 in ./world/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib->tfc) (1.16.0)
Building wheels for collected packages: tfc
  Building wheel for tfc (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tfc (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      /private/var/folders/7h/w5blw4yd7z3bw6s7blkpn7780000gn/T/pip-build-env-_r566shw/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py:129: _InvalidFile: The given `pyproject.toml` file is invalid and would be ignored.
          !!

          ############################
          # Invalid `pyproject.toml` #
          ############################

          Any configurations in `pyproject.toml` will be ignored.
          Please note that future releases of setuptools will halt the build process
          if an invalid file is given.

          To prevent setuptools from considering `pyproject.toml` please
          DO NOT include the `[project]` or `[tool.setuptools]` tables in your file.

      !!

        if _skip_bad_config(project_table, orig_setuptools_table, dist):
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'tfc.utils.BF._BF' extension
      swigging src/tfc/utils/BF/BF.i to src/tfc/utils/BF/BF_wrap.cpp
      swig -python -c++ -o src/tfc/utils/BF/BF_wrap.cpp src/tfc/utils/BF/BF.i
      error: command 'swig' failed: No such file or directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tfc
Failed to build tfc
ERROR: Could not build wheels for tfc, which is required to install pyproject.toml-based projects

System / TFC version information:

OS: MacOS
Python Version: 3.10.8
TFC Version: 1.0.1

Hello,
You're in the right spot :)
swig is not a file, but an executable program. It is used to create Python bindings for the C++ basis functions. I believe what happened is we don't have a binary wheel available for your version of MacOS. As a result, it tried to install via the source distribution, which requires swig. Can you install swig (likely via brew or homebrew) and then try the pip installation again?
Out of curiosity, what version of MacOS are you using?

Glad to hear, and thank you! I am on MacOS Monterey Version 12.6 if that helps.

Yeah, we don't have binary wheels for that version of MacOS, so it must be falling back to a source distribution. Please let me know if it installs correctly after installing swig.

That did it! I'll leave this here for other Mac users, appreciate the help.
brew install swig
pip install --upgrade tfc


Update versions of logging site extensions

These ones are included when aggregate logging site extension is built.
@muratg

@pakrym What's this exactly? 2.2 is done AFAIK.

This is aggregate logging site extension that includes both 2.1 and 2.2 bits.

It's triggered and updated manually


Failed to set className in TableHeaderColumn while using a modal dialog

I got stuck in a question just as the title describled. When I tried to use this component in a modal dialog to set the table  width , I got another inline styles unexpectedly.

thanks for your reading!

@j78742, sorry for lately reply. I'll check this out. BTW, if you dont assign the width, everything is fine?


Specify the number of epochs when finetuning gpt2client

How many epochs does it take the gpt2client to fine tune and can one specify explicitly ?
I have been running on Colab and now it is at 177 and I am wondering how long I have to go yet. Thanks

Hey there,
The finetuning feature of gpt2-client uses a third-party finetuning library called gpt2-simple. All the details on finetuning can be found there. So sorry for the delayed response – I haven't had the time to work on this project for a while.
Appreciate your patience!


feat: add ECDSA signature verification

Initial PR for discussion. If the API looks OK, will add documentation and merge for now.
This PR implements also emulated short-Weierstrass package, which for now assumes the special case for secp256k1 (a4=0), but in general can be generalised to any a4. I didn't want to do it now because then we would need to have more parameters etc. and I didn't want to eagerly generalise.
A few things to do before merge:

[x] this PR is slightly larger and contains other fixes to field emulation. Should we separate this into three - this, emulated SW and field emulation bugs?
[x] add package documentation and code examples
[ ] add ECRecover?
[x] i'm leaking the internal implementation of the field emulation package by type asserting frontend.Variable to emulated.Element[T]. I think long-term would be better to have something which returns actual types not empty interface.

This implementation right now is very heavy and requires 3.7M constraints without range checks, but I think it is a good base for optimisation. There are several optimisation options:

implement separate variable and base point scalar mul. Base point scalar mul can take advantage of huge precomputation tables + lookups.
variable point scalar multiplication requires building lookup tables per-point. For a single signature this allows to implement windowed scalar multiplication and reduce the cost several times. See https://0xparc.org/blog/zk-ecdsa-2.
0xPARC blog describes an interesting idea for checking only a random linear combination of signatures. This would require applying Fiat-Shamir and a lot of hashing, but may probably be more efficient than point operations.
When we combine two lookup tables, then can build window over two scalars at a time for ECDSA.

DRAFT: should be merged after #395.

@yelhousni can you do a review of that one before merge?

Merged develop and fixed conflicts. It is good to merge from my side.
@yelhousni said that will post optimisations in separate PR.

@yelhousni should we merge now or wait for replacing the external dependency for out-of-circuit ecdsa with latest gnark-crypto ?

@yelhousni should we merge now or wait for replacing the external dependency for out-of-circuit ecdsa with latest gnark-crypto ?

To do so we have to wait for https://github.com/ConsenSys/gnark-crypto/pull/310 to be merged I guess. So either we wait or we merge this and follow with a quick PR for the dependency.


multi facility support

Needs to better handle deploying across multiple facilities.

facility var name is different in the two terraform vars
facility isn't set in the student environment - always defaults to EWR1, it should use the facility of the lab master
networks are hardcoded for EWR1 and should be dynamic

workspaces now pull over the labmaster defaults allowing different facilities to be used


feat: Create directly without schema predefined

What's changed and what's your intention?
Client and StreamClient can Create body directly into GreptimeDB without schema predefined.
	monitors := []monitor{
		{
			ID:          randomId(),
			Host:        "<IP_ADDRESS>",
			Memory:      1,
			Cpu:         1.0,
			Temperature: -1,
			Ts:          time1,
			Running:     true,
		},
		{
			ID:          randomId(),
			Host:        "<IP_ADDRESS>",
			Memory:      2,
			Cpu:         2.0,
			Temperature: -2,
			Ts:          time2,
			Running:     true,
		},
	}

	resp, err := client.Create(context.Background(), monitors)

Checklist

[x]  I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

Codecov Report
Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (34f2874) 77.62% compared to head (8c082fd) 76.76%.

Files
Patch %
Lines

client/client.go
62.50%
2 Missing and 1 partial :warning:

client/stream_client.go
62.50%
2 Missing and 1 partial :warning:

schema/schema.go
0.00%
2 Missing and 1 partial :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #19      +/-   ##
==========================================
- Coverage   77.62%   76.76%   -0.87%
==========================================
  Files           7        7
  Lines         371      383      +12
==========================================
+ Hits          288      294       +6
- Misses         63       67       +4
- Partials       20       22       +2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Fix stale PR message to use the correct number of days

An example of an incorrect message: https://github.com/LazyVim/LazyVim/pull/3503#issuecomment-2240829662

ty!


Lab 14: Small typo, simply should be simple.

Module: 00
Lab/Demo: 14
Task: Add a trigger for the dialog
Step: 03 (Grey box)
Description of issue
Small typo Simply instead of Simple. Screen shot highlighed.

Repro steps:

Another small one.

Another small one.


Cache policy attached to runner instead of docker machine instance

https://github.com/npalm/terraform-aws-gitlab-runner/blob/3ea16b04df7129c813f46cdcbf42aba3da90e65a/main.tf#L406
If docker machine runners are used, I believe this should be attached to the docker machine instance role instead.

@hntrmrrs I think this issue is done as explained in #410. Could you please close it?


Feature Request:

Will it be possible to change the tool such that it triggers the github checks only for modified commits?
Example:
Say I have commits 1 <- 2 <- 3 which were already submitted (using git spr update).
Then I make edits to commits 2 and 3 (using interactive rebasing), resulting in:
1 <- 2' <- 3'
Currently, running git spr update re-triggers github checks for commit 1 just as well, though it hasn't changed.
Is there a way to disable this redundancy?
Much appreciated,
Itay
p.s.,
Loving this tool :-)

Hi Itay,
My guess is that this is due to the rebase on origin. If there are new changes from upstream all the commits in the stack will end up re-triggering github checks.
If this is the case, we can add a flag such as git spr update --norebase which will not pull in the latest changes, in this case I think only the commits in your stack starting from the one you updated will trigger the checks.
Can you check if this is the case for you?
Eitan

Hi,
The repo I'm working with is frequently updated and our github checks take a while, I doubt I'd manage to squeeze in an uninterrupted update like that.
I don't really mind about the first commit being "outdated" w.r.t. to the current rebased origin -- this already happens since time passes between the checks and the actual merge.
The suggested flag seems like a great idea -- can this work based on the commit ids?
Something in the sort of:

Maintain a list of commit ids of the prs in the last update (or merge)
When a new update is invoked, only triger the githut checks for prs with different commit ids

Thanks (and hag sameh),
Itay

You can now run 'git spr update --no-rebaseorgit spr update -nr`.
Also there is a configuration in the user config to disable rebase by default.
Thank you @adambenhassen for adding this flag!


pass operator-sdk scorecard test

https://github.com/operator-framework/operator-sdk/blob/master/doc/test-framework/scorecard.md
Test command:
operator-sdk scorecard -o text --selector='test in (checkspectest,checkstatustest,writingintocrshaseffecttest,bundlevalidationtest,crdshavevalidationtest,crdshaveresourcestest,specdescriptorstest,statusdescriptorstest)'

Current result:
basic:
	Spec Block Exists                   : pass
	Labels:
		"necessity":"required"
		"suite":"basic"
		"test":"checkspectest"

	Status Block Exists                 : fail
	Labels:
		"necessity":"required"
		"suite":"basic"
		"test":"checkstatustest"
	Suggestions:
		Add a 'status' field to your Custom Resource
		Add a 'status' field to your Custom Resource
		Add a 'status' field to your Custom Resource

	Writing into CRs has an effect      : fail
	Labels:
		"necessity":"required"
		"suite":"basic"
		"test":"writingintocrshaseffecttest"
	Suggestions:
		The operator should write into objects to update state. No PUT or POST requests from the operator were recorded by the scorecard.
		The operator should write into objects to update state. No PUT or POST requests from the operator were recorded by the scorecard.
		The operator should write into objects to update state. No PUT or POST requests from the operator were recorded by the scorecard.

olm:
	Status fields with descriptors      : pass
	Labels:
		"necessity":"required"
		"suite":"olm"
		"test":"statusdescriptorstest"

	Bundle Validation Test              : fail
	Labels:
		"necessity":"required"
		"suite":"olm"
		"test":"bundlevalidationtest"
	Errors:
		unable to find the OLM 'bundle' directory which is required for this test
		unable to find the OLM 'bundle' directory which is required for this test
		unable to find the OLM 'bundle' directory which is required for this test

	Provided APIs have validation       : pass
	Labels:
		"necessity":"required"
		"suite":"olm"
		"test":"crdshavevalidationtest"

	Owned CRDs have resources listed    : fail
	Labels:
		"necessity":"required"
		"suite":"olm"
		"test":"crdshaveresourcestest"
	Suggestions:
		If it would be helpful to an end-user to understand or troubleshoot your CR, consider adding resources [pods/v1] to the resources section for owned CRD MetaOperatorCatalog
		If it would be helpful to an end-user to understand or troubleshoot your CR, consider adding resources [pods/v1] to the resources section for owned CRD MetaOperatorConfig
		If it would be helpful to an end-user to understand or troubleshoot your CR, consider adding resources [pods/v1] to the resources section for owned CRD MetaOperatorSet

	Spec fields with descriptors        : fail
	Labels:
		"necessity":"required"
		"suite":"olm"
		"test":"specdescriptorstest"
	Suggestions:
		Add a spec descriptor for operators
		Add a spec descriptor for services
		Add a spec descriptor for services


OSSMC does not show Istio Config names

OSSMC (left) does not show Istio Config names as standalone Kiali (right)
OpenShift version 4.15.18
Kiali Community Operator 1.86.0 provided by Kiali
istiod Version: 1.22.1
Namespace: istio-system

I cannot reproduce the issue with the latest version of OSSMC. Could you check again please?

cannot reproduce, closing


fix(aspect): count query only for relevant aspect index

Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

I merged in because I believe the sql write failure is the separate from this PR that the team is trying to eliminate.


Issue #215: Do not contribute SCB layer if already there

prefer the spring-cloud-bindings jar provided by the user

Summary
Add a check on the presence of an existing spring-cloud-bindings jar in the Spring libs (BOOT-INF/libs)
Use Cases
When a user already added to its application a spring-cloud-bindings version, the spring-boot buildpack should not add another one.
Checklist

[X] I have viewed, signed, and submitted the Contributor License Agreement.
[X] I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
[X] I have added an integration test, if necessary.
[X] I have reviewed the styleguide for guidance on my code quality.
[X] I'm happy with the commit history on this PR (I have rebased/squashed as needed).

hi there @dmikusa @pivotal-david-osullivan !
I try re requesting your reviews via GH UI, but apparently only one can be set as a reviewer!
Anyway, please feel free to re review; I should have addressed all concerns with my updated commit

Can you please add the following labels please ?  semver:patch and type:enhancement (or type:bug even?!)
thank you!


Goerli chain selection fails and uses the mainnet URL

Goerli network is not available on version 10.2.0 on NPM.
var etherscan = require('etherscan-api');
console.log(etherscan.pickChainUrl("goerli")); >> "api.etherscan.io"

This issue has already been resolved in PR: https://github.com/sebs/etherscan-api/pull/113 but no new NPM package is available.

closed with 10.2.1


training issue

Hi, I am interested in your work published in MICCAI2022, but when I try to reproduct the result presented in your paper, the training process is very slow, like the training loss drops slowly. I am wondering if it's normal or any pretrained weight of transformer should be loaded? I just downloaded Brats2020 data and loaded it  into the network as your description without addtional processing,  I am looking forward to your reply, thank you.

Thank you for your attention, because the nestedformer network uses 4 encoders to process the brats2020 dataset, so it is normal to run a bit slower, it should not be very slow. What device are you using? I am using a V100 or A100 graphics card. Of course you can also try to adjust the network parameters to reduce the number of parameters in the network.


Maximum Resolution Change Yields Failed

Required Info

Camera Model
{ d435f }

Firmware Version
<IP_ADDRESS>

Operating System & Version
Ubuntu 20.04

Kernel Version (Linux Only)
5.14.0-1059-oem

Platform
NVidia Jetson Orin 32GB

SDK Version
{ librealsense2.53 }

Language
{C++}

Segment
{ Robotics }

Issue Description
What changed:
I recently upgraded our code to capture at 1920 x 1080 for color stream and  depth streams from our intel realsense d435f. In the past we ran 848x400 for both cameras. And then we started noticing the issue.
note all of the below are using our own internal frame numbers (not RS frame number)
The Issue:
It has caused a very peculiar issue where after exactly 3 frames, we drop frames 4 - 17 inclusive. After which the next wait_for_frames(), repeats frame 3 forever. When this issue occurs the depth frames are always black and 0 valued everywhere. This has happened twice so far, and its always 3 good frames, then 4 - 17 are null, and then frame 18 returns the same data as frame 3.
This issue will persist for 10000+ frames.
The settings we use are applied here:
FPS = 10
depth_width: 1280,
depth_height: 720,
color_width: 1920,
color_height: 1080

My capture code and frame checking occurs here:

Here's how I can tell that the frames continue to be stale and how my frames 4 - 17 are dropped. Notice how the file sizes stop changing on the color frames (C)

I call my wait_for_frames in a loop that fires at about 9.5Hz.
The issue fixes itself if I stop my binary and restart it. This issue happens about 5% of my binary runs so far, over at least two runs.
Request
Is there any way to fix this issue without restarting my binary? (and closing and reopening the pipeline) I am running this code on agricultural equipment and once we have started capturing frames its imperative that the depth code actually works, otherwise, the equipment will fail to do its job.
Please let me know what other information I can provide!

Hi @shihaocao  Does the problem still occur if you use 1280x720 color instead of 1920x1080, please?
When auto-exposure is enabled, the first few frames after starting a program are spent settling down the auto-exposure.  So it may be worth testing disabling auto-exposure to see whether it is the cause of your issue when using 1920x1080 color.  It can be disabled using code like this:
sensor.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, 0);

Hi Marty!
Unfortunately, configuration changes like that are a bit hard to test at the moment since all of our realsenses are out on production units... and we need the full resolution of the color sensor and auto exposure. I can definitely try at my next chance though.
New information:
Most interestingly though, this appears to happen around about 1/25 to 1/50 captures, where capture is a pipeline start, followed by calls to wait_for_frames(). This also makes it very hard to test if configuration/if the resolution change actually caused the issue or if I just didn't notice it before...
The "issue" seems slightly more likely to strike if the previous binary was interrupted/killed instead of naturally shutting down and perhaps letting destructors be called properly?
I have seen it occur on a fresh jetson boot + realsense powerup + binary run. Could anything persist on the realsense despite a full system power cycle?
Also, ctually checked my code, and I don't ever call .stop() on my pipeline. Do I need to make sure that is called even if I only call pipeline .start() once per program?
Thanks!

If your program needs to run continuously then don't call stop().  A program should be able to run continuously for a very long time period anyhow, as the pipeline can only hold up to 16 frames at a time in its 'queue' for each stream type by default and drops old frames from the queue as new ones arrive, releasing memory resources.
If a program is improperly closed then problems can occur in a subsequent session of that program when it is run again.
When rebooting a computer to try to resolve a problem, a cold shutdown (full power off and start-up with the power button) is more likely to resolve the issue than a reset, where the issue may persist after reboot.

Hi @shihaocao  Do you require further assistance with this case, please?  Thanks!

Case closed due to no further comments received.


Completed tasks showing in autocomplete

We've got a couple of completed tasks which are still showing in the auto-complete drop down list. This list should be refreshed on load of the app (or at least daily). Some of the tasks were completed a long time ago.

Cannot reproduce the issue. Please check that you have latest app version, if you do, please contact<EMAIL_ADDRESS>and tell us the exact task name(s) that are marked as done but still show up.


Total number of reports in organisation metric

Guidance on the legacy report (our reports state 22k) suggests:

Count of all dashboards or reports in the entire organization that had at least one view in the past 90 days.

...vs guidance on the new report (our reports say 1.5k) which suggests:

Based on view count, the rank shows the popularity of a report in comparison to all other reports in the organization.

Have the guidance pages been mixed up?
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: f16f558c-a397-7d10-b953-6c8972df9478
Version Independent ID: ffc58ae8-f232-f6d8-f0ea-e4d6001e0e3d
Content: Monitor usage metrics in workspaces (preview) - Power BI
Content Source: powerbi-docs/collaborate-share/service-modern-usage-metrics.md
Service: powerbi
Sub-service: pbi-collaborate-share
GitHub Login: @maggiesMSFT
Microsoft Alias: maggies

Thanks for your comment, @adam-pope-arup. This issue has been assigned to the doc author for review.


Improve some documents: release version, logging, NFS lock (#25202)

Backport #25202 by @wxiaoguang
Close #23654
Close #24684
@techknowlogick I still think we need to rename https://dl.gitea.com/gitea/1.20/ to https://dl.gitea.com/gitea/1.20-nightly/
/gitea/1.20/ is quite confusing, it needs these words to explain why. If we call it 1.20-nightly, the FAQ can be simplified a lot.

Additional info: I updated gitea on my k3s cluster to 1.20.3-rootless, and in addition I updated with k3sup to k3s to v1.27.4+k3s1. After rebooting gitea db (sqlite3) was looked. gitea db is located on an NFS share mapped to the default container location for rootless. After reading the comments of "LevelDB/LevelQueue could get stuck on NFS when NFS locking doesn't work properly", I tried a
root@k3s01:/etc/init.d# ./rpcbind restart
on the k3s host, which ran the gitea container. After that the gitea db lock on the nfs share was gone and it worked again. So that might help as an addition "try that" in the documentation.
Cheers and thanks for your great work... ;-)


add stripQuery method and strip the query from the extension

Given a url https://example.com/file.yml?foo=bar, the extension will be parsed as .yml?foo=bar and the relevant parsers will miss it.
This PR fixes this issue by stripping the query from the extension.

done 👍

Thank you @mathieudutour!

Thank you @mathieudutour!


[serverless-1.26.0] patch func-deploy task to use kn client midstream built

Patching func-deploy task to use midstream specific kn client image.

/hold

/unhold

/override "On Cluster RT Test (1.18.x, ubuntu-latest)"
/override "On Cluster Test (1.18.x, ubuntu-latest)"


pretrained weights

I want to train Sigma on my device. But I can not find suitable vmamba pretrained weights in the provided link. Can you provide ''vssmsmall_dp03_ckpt_epoch_238.pth'' and ''vssmbase_dp06_ckpt_epoch_241.pth'' for me. Thanks!

Hi, have you tried to download the weights here by clicking at the download button?

Oh! I didn't notice there was a link. Thanks！


How can I output the debug information onto the terminal in Rust ( Beginner in Rust! )

Hi, so I was recently introduced to quiche and Rust, but I was wondering how I can output the debug information onto the terminal for example from the client.rs file:

Is there a flag for cargo that would be able to handle this?

You can set RUST_LOG environment variable. It takes different levels such as race, debug, warn or error.

You can set RUST_LOG environment variable. It takes different levels such as race, debug, warn or error.

@LPardue that does not work because the examples are missing a logger implementation, https://docs.rs/log/latest/log/#available-logging-implementations
I can send a PR to add env_logger to the examples


Is the "cant duplicate this flag" error (v3) makes sense?

Hi there, team of the best CLI package 😄
I have a question that might be a bit silly, but I was confused when I encountered the error can't duplicate this flag while using only one flag with the OnlyOnce: true property for a cli.Command.
https://github.com/urfave/cli/blob/c4cd0a51cc113a90eca318cc8f7d328f05a5f2b7/flag_impl.go#L166-L168
Is this a mistake or not? If not, what is the purpose of this guard? Could someone explain it to me?

Ouch, I could provide a bit more context on how exactly I encountered this error.
Usually, if the app I develop has more than one sub-command that requires the same flags, I prefer to define every flag that I could reuse in a separate package named shared:
package shared

import (
	// ...
	"github.com/urfave/cli/v3"
)

var ListenPortFlag = &cli.UintFlag{
	Name:     "port",
	Aliases:  []string{"p"},
	Usage:    "TCP port number",
	Value:    8080, // default port number
	Sources:  cli.EnvVars("LISTEN_PORT"),
	OnlyOnce: true,
	Required: true,
	Validator: func(port uint64) error {
		if port == 0 || port > 65535 {
			return fmt.Errorf("invalid TCP port number [%d]", port)
		}
		return nil
	},
}

// ...

And use it in my commands in this manner:
import (
	// ...
	"github.com/urfave/cli/v3"
)

func NewCommand() *cli.Command {
	return &cli.Command{
		Name:    "some_command",
		Aliases: []string{"some", "command"},
		Action: func(ctx context.Context, c *cli.Command) error {
			runtime.Gosched() // <-- important, can be replaced with something else

			return nil // some code goes here
		},
		Flags: []cli.Flag{
			shared.ListenPortFlag, // <-- important
		},
	}
}

Everything works fine until I write a unit test to test the command execution:
func TestCommand_RunSuccess(t *testing.T) {
	t.Parallel()

	var cmd = healthcheck.NewCommand()

	require.NoError(t, cmd.Run(context.Background(), []string{"", "--port", "1234"}))
}

func TestCommand_RunFail(t *testing.T) {
	t.Parallel()

	cmd := healthcheck.NewCommand()

	require.NoError(t, cmd.Run(context.Background(), []string{"", "--port", "4321"}))
}

If you run each test one by one (not together - that's important), each of them will pass. But, if you run them in parallel (go test ./...), the test will fail with the error:
=== RUN   TestCommand_RunFail
=== PAUSE TestCommand_RunFail
=== CONT  TestCommand_RunFail
    command_test.go:47:
        	Error Trace:	/path/to/command_test.go:47
        	Error:      	Received unexpected error:
        	            	invalid value "4321" for flag -port: can't duplicate this flag
        	Test:       	TestCommand_RunFail
--- FAIL: TestCommand_RunFail (0.00s)

To fix this, all I need to do is replace
// file: shared/flags.go

var ListenPortFlag = &cli.UintFlag{
//                   ^

with
var ListenPortFlag = cli.UintFlag{

This was so strange that I decided to create this issue to highlight this workaround.

[!NOTE]
TL;DR - flags have their own state, so don't reuse them.

@tarampampam The OnlyOnce field is used to tell the cli library to expect this flag on the cmd line only once. So if it is specified twice , for example
--port 1234 --port 4566

then you will get that error. This is to protect users from themselves. Without this option in above example port would be set to 4566


Playground: IndexError halts execution with no obvious feedback

Given a file test.cr with the contents:
array = [1, 2, 3, 4, 5]
array[0]
array[10]

And then I run crystal run test.cr
Then I receive:
Index out of bounds (IndexError)
[4351419234] *CallStack::unwind:Array(Pointer(Void)) +82
[4351419137] *CallStack#initialize<CallStack>:Array(Pointer(Void)) +17
[4351419096] *CallStack::new:CallStack +40
[4351428553] *Exception@Exception#initialize<IndexError, String>:CallStack +41
[4351428477] *IndexError#initialize<IndexError, String>:CallStack +29
[4351428420] *IndexError::new<String>:IndexError +100
[4351428300] *IndexError::new:IndexError +28
[4351415216] *Array(Int32)@Array(T)#at<Array(Int32), Int32>:Int32 +96
[4351415113] *Array(Int32)@Array(T)#[]<Array(Int32), Int32>:Int32 +9
[4351405496] __crystal_main +2600
[4351413032] main +40

The same code in playground doesn't show anyhting in the UI to help indicate the breakage, rather only the modal icon goes red, and it appears that execution halts after the second line.
My expectation was that there would either be some more "red" in the UI to denote the code breakage/where execution stopped.

@miketheman until #1270 we are unable to show exactly in which line a runtime exception was thrown.
I understand that the current red might not be enough.
An option would be to show the output panel directly when an exit status is 1. (other exit status might be if the process is kill by the stop button for example)
In more complex programs there will be loops, blocks, functions, etc. But an approximate line could be highlighted with the error trace if, for example we show that on the line following the last value sent to the sidebar. That could be more red, but might not be accurate.

@miketheman we changed how the instrumentation is done and now we surround the expressions that leads to the values in the sidebar with a begin/rescue so the runtime exceptions are informed at the same line that triggers the issue. Thanks for playing!

@bcardiff Thanks! I tested the same off master and it now behaves the way I'd expect.
Great response time, keep up the good work!


kotlin: Replace most query param classes with generated ones

… and clean up the remaining ones to prepare for future replacement.
Also leave the default for the limit on pagination APIs up to the server. I don't think we need to mention this in the changelog since so far the default has been the same as on the server side; however if we were to change the limit for some API on the server side, it makes much more sense to me that the new default would apply immediately and for everyone who doesn't specify it, rather than requiring a libs update for Kotlin users.
Part of https://github.com/svix/monorepo-private/issues/9576.

Yeah, no need to mention that. It's an implementation detail that was never public, and anyhow we won't change the server value anyway.


Make Linux packages C++14 compatible

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Is there an upstream issue/PR for this?

Is there an upstream issue/PR for this?

No, this is due to the nature that we compile everything with C++17 but Tensorflow doesn't support C++17, see https://github.com/tensorflow/tensorflow/issues/51925
The diff here makes no difference if you compile with std=c++17 but if you compile with std=c++14 objects with static constexpr inline will be inlined while with static constexpr they are exported and linked explicitly. With C++17, static constexpr is always inlined.


⚠ Bofrost.de has degraded performance

In 5bd1daa, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 916 ms

Resolved: Bofrost.de performance has improved in 8e1abdc after 44 minutes.


Add IntelStitch

@hslatman

Thank you! Good luck on developing your product! 😄

Hi @himynamesdave, I wanted to test the IntelStitch and went to the website you provided in the PR, but the domain does not resolve to your website. It resolves to IP "<IP_ADDRESS>" which is serving different ad-like websites.
You better check your configuration.


In man pages, short flags should follow long flags.

Description
In the man pages we list the short version of the flag before the long version.  For instance in the buildah bud man page we list:
**-c, --cpu-shares**=*0*

The -c should be listed after -cpu-shares like:
**--cpu-shares, -c**=*0*

Describe the results you received:
See description
Describe the results you expected:
See description
Output of buildah version:
Builda v1.0

Fixed with #675


C++ Code generation

What
Generate C++ Trait/Specification from yaml definition.
Why
We anticipate an up-tick in C++ hosts in Y23.
Requirements
User

As a Host integrating OpenAssetIO into my project, I need to be able to define additional traits that are specific to my integration. I need to ensure these traits are available to any other OpenAssetIO user, regardless of their implementation of choice, and that they can be generated for new languages as they become available. This ensures that all parties involved in the ecosystem have the same source of truth to avoid interoperability issues.

As a host integrator, it should be simple to incorporate custom trait definitions into my existing build [and compile] workflow without being required to adopt a different build system.

As an OpenAssetIO integrator (Host or Manager) I wish to be able to easily use Traits defined in common libraries such as the MediaCreation project, without complex and onerous build steps.

As a maintainer of a common trait collection, I wish to be able to publish the trait collection (using a mechanism of my choice) as a library for C++ in the same way as we do for PyPI etc. such that it can be easily consumed by downstream projects.

Functional

The autogenerated code should not be opinionated about which build or packaging approach is used by a project incorporating traitgen code into their pipeline.

Generated traits should fail to link to ABI incompatible OpenAssetIO version.

Generated trait code should contain its own (traitgen centric) ABI namespace using the conventions established with the OpenAssetIO ABI namespace mechanism (ie most recent major version that broke ABI).

Generated trait code should use the same trait package namespace not openassetio.

Generated code should share the same namespace structure as Python unless otherwise dictated by the lanuguage.

Generated code should contain Doxygen formatted comments to describe the code, the nature of the traits/properties.

Technical

C++17/CY22 support
Linux/Mac/Windows

ACs

C++ code can be generated using the cpp generator
Autogenerated code has parity with Python in terms of detail of docstrings
Test coverage/CI has parity with Python (+ any new language specific considerations)
Code generation produces a src only package with no compiled output
README.md lists supported languages.
C++ code namespace(s) mirror python package structure
TraitGen ABI version encoded in inline namespace
C++17
API feature/naming parity with Python (flag any incongruous consequences).

Out of scope

No need extend documentation/examples beyond what already exists for Python.
No need to expose version numbers programtically
No build infrastructure or tooling to be generated

Tasks

[ ] [Core][C++] Initial C++ codegen parity with Python
[ ] [Core][C++] More granular dependencies in hoisting headers
[ ] [CI][C++] Run ctest tests on CI

Rather than add a bunch of tasks to this ticket that aren't related only to C++ codegen, I've added #14 to track the additional tasks discovered/decided during this work.


Edit's to make demo work on ARM architecture reliably

Previously the demo in /scriptswould fail when I tried to run it on ARM architecture. I made some edits to specify platform so that the docker environment knows to use virtualised hardware as no implmentation exits on native ARM. If it's of use to anyone I'd be happy to make a PR. Here's my branch for reference: https://github.com/H4LL/traction/tree/arm_virtualisation.
Thanks for creating traction. It's an awesome tool. I'm loving it so far!

Awesome and much appreciated if you could put in a PR.  Thanks!


Write a blog post about the Safari text-wrapping bug

What
Write a blog post about the issue we had with text wrapping unnecessarily in Safari Technology Preview, including:

how we found out about the bug
how we understood the bug enough to be able to report it
what happened
why the bug occurred
what we think could have happened if we hadn't reported it

We can use the 'The Safari bug that never was' presentation as a starting point.
Why
To promote the work that the Design System does, and to promote a culture of contributing to the web platform both internally and externally.
Who needs to work on this
Developer, designer (for illustrative images), content designer?
Who needs to review this

Contacts at Apple (WebKit)
Tech blog editor

Done when

[x] Draft written
[x] Draft shared with team
[x] Draft shared with WebKit engineer who worked on the bug
[ ] Draft shared with tech blog editor for approval
[x] Blog post published

Closing this issue as this blog was published.


Create a Responsive and beautiful About page

Create a beautiful About page and linked it with the navbar . Text content can be of any poem and image must be of Pokemon realated.

I want to contribute on it but what I am going to write on it?. Can you please said to me that. Because I am not from JGEC.

I want to contribute on it but what I am going to write on it?. Can you please said to me that. Because I am not from JGEC. Assign me under Aperture'24

Design by you own but Page content can be any story/poem text and if you want to add image in this page then must add Pokemon images only.

can i use ui library or only HTML,CSS

Whatever you want ....

Go ahead @AnitSarkar123

Whats the update

I am intrested in this issue

Whats the update

@rko0211 assigned it to anyother because of some reason I don't able to contribute

Whats the update

@rko0211 assigned it to anyother because of some reason I don't able to contribute

Actually you have to inform me other wise how i can come to know it

OK Go ahead @Sneha123-zudo


Local packages

Ability to use a local airpackage, i.e. not downloaded from the repository.

I've added in a first pass at this to the latest in develop.
apm install path/to/some_package.airpackage


Enable scripts to update and format packages

This PR adds the update_packages.sh bash script for helping to update changes on cert-manager, cluster-api and multus packages as well as the make fmt instruction which fixes the script and YAML formats.

@electrocucaracha Is there a way we can get this branch tested again test-infra? Probably for this we need to make another test-infra branch which pulls from here. That will be good just to verify that nothing has been affected.

@electrocucaracha Is there a way we can get this branch tested again test-infra? Probably for this we need to make another test-infra branch which pulls from here. That will be good just to verify that nothing has been affected.

There is a way but I think that it requires some manual adjustments, let me try to document the steps to do it

@electrocucaracha Is there a way we can get this branch tested again test-infra? Probably for this we need to make another test-infra branch which pulls from here. That will be good just to verify that nothing has been affected.

There is a way but I think that it requires some manual adjustments, let me try to document the steps to do it

Thank you. It is not necessary to test it but more of a precaution and it will be easier for everyone to review.

@electrocucaracha Is there a way we can get this branch tested again test-infra? Probably for this we need to make another test-infra branch which pulls from here. That will be good just to verify that nothing has been affected.

There is a way but I think that it requires some manual adjustments, let me try to document the steps to do it

Thank you. It is not necessary to test it but more of a precaution and it will be easier to review.

Well, eventually it's something that we'll have to implement for other catalog changes as well.

@electrocucaracha Is there a way we can get this branch tested again test-infra? Probably for this we need to make another test-infra branch which pulls from here. That will be good just to verify that nothing has been affected.

+1

@tliron @henderiw I think that this is ready to be merged

/approve
/lgtm


Support mem_reservation service property

Docker supports --memory-reservation="", this adds support for that property.
See https://github.com/docker/compose/pull/3723 for relevant docker-compose equiv.
I looked for where to test this, but couldn't find a good spot. Suggestions?

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "support-memory-reservation" git@github.com:lox/libcompose.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

I wrote a test for this feature in our fork. Feel free to reuse :smile:
Can you also add this field to ServiceConfigV1?

Done, thanks @joshwget

Perfect, thanks! LGTM.


fix: Use SSE for storage control updates

Fixes Implemented:

[x] Implemented immediate display of error messages when admins change storage rules for the cloud application.
[x] Added listener for the new StorageRuleChanged message.
[x] Display the server error messages for PROJECT_STORAGE_RESTRICTION errors.
[x] Addressed the issue where attempting to create a new cloud project as a member displayed an error modal, which reappeared when trying to create a local vault project subsequently.
[x] Always show a warning message when creating or updating a project if the storage rule is not cloud_plus_local.

Notes for Team Discussion:

The fourth fix on the list is a quick solution. If preferred, I can revert this change and address it more efficiently in a separate PR.
For handling storage rules, I utilized network cache instead of localStorage. If there are concerns about this approach, let's use localStorage or discuss about alternative solutions.

Closes INS-4113
Related BE PR INS-4108

@pavkout we rebased your branch with latest develop to help test for one of @marckong 's PRs in the API

@CurryYangxx Just finished the changes we discussed. Feel free to test it and review them one more time, and let's have a try it session one more time tomorrow morning.


CI: Updated build with setuptools's embedded distutils

Trying to build with upcoming setuptools changes related to PEP 632 and checking numpy.distutils compatibility.

#3581 superseeds this PR


can not run the mode

Hello,
I have edited my compose part for transmission as follows:
transmission:
image: lscr.io/linuxserver/transmission:latest
container_name: Transmission
environment:
- PUID=1000
- PGID=1000
- DOCKER_MODS=michaukrieg/docker-mods:transmission-gluetun-port-update|linuxserver/mods:universal-cron
- TRANSMISSION_WEB_HOME=/config/flood-for-transmission
- PEERPORT=30399
volumes:
- /Users/mediaserver/Desktop/Docker/MediaServer/transmission/config:/config
- /Volumes/torrent_storage:/torrent_storage
- /Users/mediaserver/Downloads:/downloads
- /Volumes/media_storage/temp:/temp_storage
restart: unless-stopped
network_mode: "service:gluetun"
However, I am getting the folllwing error when I run the container: [mod-init] (ERROR) michaukrieg/docker-mods:transmission-gluetun-port-update digest could not be fetched from registry-1.docker.io
Do I miss something here?
Thank you!

Everything looks correct, not sure why you can't fetch it. Could you try ghcr.io instead?
DOCKER_MODS=ghcr.io/michsior14/docker-mods:transmission-gluetun-port-update|linuxserver/mods:universal-cron

It did not work as well. However, I realised something. I am running docker on mac (m2 chip). When I try to manually get the the mod via 'docker pull ghcr.io/michsior14/docker-mods:transmission-gluetun-port-update-20240430121441' I get the following error: transmission-gluetun-port-update-20240430121441: Pulling from michsior14/docker-mods
no matching manifest for linux/arm64/v8 in the manifest list entries
Can it be something related to this?

According to previous issues reported https://github.com/Michsior14/docker-mods/issues/7, this shouldn't be the case, but maybe linuxserver implementation has changed. Unfortunately I don't have arm anywhere to test things out. I will have a look if there is something I can deduct from theirs images.

Thank you for taking the time to respond and check this. I am more than happy to test anything you want on my systems (I've got linux, macos, windows 11).


Error while running Flutter project using package.

I am using dash_chat_2 Version 0.0.19, while running the app it gives below error. Can anyone tell me how to solve it ??
Using Flutter version : 3.3.10
Launching lib\main.dart on sdk gphone64 x86 64 in debug mode...
Running Gradle task 'assembleDebug'...
/D:/Flutter%20Sdks/flutter_windows_3.3.10-stable/flutter/.pub-cache/hosted/pub.dartlang.org/dash_chat_2-0.0.19/lib/src/widgets/input_toolbar/input_toolbar.dart:49:32: Error: The getter 'View' isn't defined for the class 'InputToolbarState'.

'InputToolbarState' is from 'package:dash_chat_2/dash_chat_2.dart' ('/D:/Flutter%20Sdks/flutter_windows_3.3.10-stable/flutter/.pub-cache/hosted/pub.dartlang.org/dash_chat_2-0.0.19/lib/dash_chat_2.dart').
Try correcting the name to the name of an existing getter, or defining a getter or field named 'View'.
final double bottomInset = View.of(context).viewInsets.bottom;
^^^^
/D:/Flutter%20Sdks/flutter_windows_3.3.10-stable/flutter/.pub-cache/hosted/pub.dartlang.org/dash_chat_2-0.0.19/lib/src/widgets/input_toolbar/input_toolbar.dart:177:42: Error: A value of type 'OverlayState?' can't be assigned to a variable of type 'OverlayState' because 'OverlayState?' is nullable and 'OverlayState' isn't.
'OverlayState' is from 'package:flutter/src/widgets/overlay.dart' ('/D:/Flutter%20Sdks/flutter_windows_3.3.10-stable/flutter/packages/flutter/lib/src/widgets/overlay.dart').
final OverlayState overlay = Overlay.of(context);
^

FAILURE: Build failed with an exception.

Where:
Script 'D:\Flutter Sdks\flutter_windows_3.3.10-stable\flutter\packages\flutter_tools\gradle\flutter.gradle' line: 1159

What went wrong:
Execution failed for task ':app:compileFlutterBuildDebug'.

Process 'command 'D:\Flutter Sdks\flutter_windows_3.3.10-stable\flutter\bin\flutter.bat'' finished with non-zero exit value 1

Try:

Run with --stacktrace option to get the stack trace.
Run with --info or --debug option to get more log output.
Run with --scan to get full insights.

Get more help at https://help.gradle.org

BUILD FAILED in 7s

Look at the pinned issue: https://github.com/SebastienBtr/Dash-Chat-2/issues/55


[doc-enhancement][Build features]

Update for What's new and Landing page - format and links

Acrolinx Scorecards
We currently enforce a minimum score of 80.
Click the scorecard links for each article to review the Acrolinx feedback on grammar, spelling, punctuation, writing style, and terminology:

Article
Score
Issues
Scorecard
Processed

msteams-platform/mstdd-landing.yml
100
0
link
:white_check_mark:

msteams-platform/whats-new.md
92
32
link
:white_check_mark:

More info about Acrolinx
Use the Acrolinx extension, or sidebar, in Visual Studio Code to check spelling, grammar, style, tone, clarity, and key terminology when you're creating or updating content. For more information, see Use the Visual Studio Code extension to run Acrolinx locally.

Docs Build status updates of commit 9eac5fe:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/mstdd-landing.yml
:white_check_mark:Succeeded
View

msteams-platform/whats-new.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 4f589a5:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/mstdd-landing.yml
:white_check_mark:Succeeded
View

msteams-platform/whats-new.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Acrolinx Scorecards
We currently enforce a minimum score of 80.
Click the scorecard links for each article to review the Acrolinx feedback on grammar, spelling, punctuation, writing style, and terminology:

Article
Score
Issues
Scorecard
Processed

msteams-platform/mstdd-landing.yml
100
0
link
:white_check_mark:

msteams-platform/whats-new.md
92
32
link
:white_check_mark:

More info about Acrolinx
Use the Acrolinx extension, or sidebar, in Visual Studio Code to check spelling, grammar, style, tone, clarity, and key terminology when you're creating or updating content. For more information, see Use the Visual Studio Code extension to run Acrolinx locally.


Merge basetheme

What is the context of this PR?
This PR merges changes made in the new "basetheme" into the "default" theme. Going forward, all changes will made to "default" unless a need for a specific theme arises (eg. census).
No new styles are introduced in this PR.
How to review
The effect of this PR should be that everything looks exactly as it did previously, as all schemas were using the "basetheme" anyway.

Start any questionnaire (not Star Wars)
Complete questionnaire, confirming there are no visual defects.

Backend looks good to me, needs front-end testing from @yoganandkunche before merging

Front End looks good to me. QA Passed


Update GitHub pages

Remove all unsued files
Add latest path for documentation
Website only shows documentation now

I am a bit confused. Isn't this something that should automatically be generated?
Also, I am not completely sure what I should review here.

Only the documentation will be automatically generated. The pages currently includes more than the documentations, the homepage for example, and other things that aren't used at all (https://iguana-benchmark.eu/sc/).
The home page is also quite outdated and hard to maintain (there are also no source files available).
That's why I think, that the website should be exclusively used for the documentation for now. The readme of the github repository should also be enough as an replacement for a website in my opinion. It doesn't really add anything extra anyway.


Question about usability of generic structure definition

This is stdlib's HashMap:
pub fn HashMap(comptime K: type, comptime V: type,
    comptime hash: fn(key: K)->u32,
    comptime eql: fn(a: K, b: K)->bool) -> type
{ ... }

It is constructed like this:
var map = HashMap(i32, i32, hash_i32, eql_i32).init(&debug.global_allocator);
Am I supposed to write function with hash map parameters this way:
fn foo(map1 : &HashMap(i32, i32, hash_i32, eql_i32), map2 : &HashMap(i32, i32, hash_i32, eql_i32), map3 : &HashMap(i32, i32, hash_i32, eql_i32))
{
...
}

This would be usability nightmare. Some types passed into the constructor may not be visible where HashMap is used.
Is there some way to define short alias for complex generics?

const I32Map = HashMap(i32, i32, hash_i32, eql_i32);

fn foo(map1 : &I32Map, map2 : &I32Map, map3 : &I32Map)
{
...
}

@thejoshwolfe: thanks!


New users - remove guest user limitation temporarily

AT the moment, new user accounts get limited as guest user accounts.
As we don't have implemented yet the registration process updates, users are not informed about this limitation, not able to request access either. Therefore, the limitation for new users should not be applied.
'Limit access to guest user permissions' should be 0.
Later when the UI will be updated, affected users will get the limited access by default.
#1079

@tovari , are we no longer assigning new users as guest users until the UI is updated?

@tovari, are we no longer assigning new users as guest users until the UI is updated?

Correct. Only after the registration flow will be also updated.

I think there's more urgency on this as I just noticed that every new user - even users from whitelisted domains - gets the "Limit access to guest user permissions" flag! So, e.g. an IFRC staff signing up can't see anything non-public.

Deployed to Staging and is ready for UAT.
Following up on your comment @esario , we must manually visit the users and remove the guest user permissions for now. But now the newly registered users will not be tied up with guest user permissions.


Adds support for generating projects that use AndroidX support libraries

Description
This PR modifies the Flutter tool with support for generating projects that use AndroidX support libraries rather than the legacy support library.
This is a source of pain in Flutter development and now that AndroidX is 1.0 all new Flutter projects should be using it.
Projects generated with AndroidX libraries also use Jetifier which allows any dependencies (e.g. plugins) that still use the legacy support libraries to be upgrade to use AndroidX.
With this change, the flutter tool has the following changes:

Adds an --androidx flag when creating flutter projects. This is disabled by default, and can be enabled via --androidx
Adds a new androidX property under the module section of a pubspec.yaml file. This is to control whether the generated module project uses AndroidX.
e.g.

flutter:
  uses-material-design: true

  module:
    androidX: true
    androidPackage:
    iosBundleIdentifier:

Related Issues

https://github.com/flutter/flutter/issues/28805

Tests

create_test.dart/androidx app project
create_test.dart/non androidx app project
create_test.dart/androidx plugin project
create_test.dart/non androidx plugin project

Checklist

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] I signed the [CLA].
[x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
[x] I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require Flutter developers to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (Please read [Handling breaking changes]). Replace this with a link to the e-mail where you asked for input on this proposed change.
[x] No, this is not a breaking change.

Why haven't merged yet?
Is it necessary use support library by default and Optional androidx support before code merge?

@dnfield I've seen you comment about AndroidX support a few other places. Is it possible this PR may be merged or is there other internal work happening around it?

Why should this PR not be merged?

@matthew-carroll would you be the right person to review this?

@xster you're probably in the best position to make a call on this particular solution.

#28805

@dnfield @mklim, could you guys take a pass at this?
I'm out for a week. I'll check this wrt add-to-app when I'm back.

Thanks for the contribution!
We've had a number of issues with jetifier not working with our current gradle setup.  I'm a bit curious about how this impacts that - are you able to consume a plugin that uses the support libraries with this project structure successfully?
My main concern here is that it's not clear what benefit this really offers to someone developing a Flutter app at this point - unless it's helping with that issue.

Hey Dan,
With this setup, a project will work with both AndroidX and Support dependencies.
Without this, I believe Flutter projects that use plugins with AndroidX dependencies need to be upgraded to AndroidX.
I'm using this branch in a Flutter project using Jetifier and all is working well.
So the benefit is that new Flutter projects using AndroidX should just work with any plugin. Do you have any more information about the issues with Jetifier that you've had? I'd like to test this branch to make sure it solves those issues.

@athornz
I think it should use support library as the default then androidX library is optional for old support library flutter module project update

Since support library is deprecated I think AndroidX should be the default. If a Flutter project with AndroidX works with Jetifier then there is no reason to use the support libraries.
We should be making use of Jetifier as Google implemented as a good way to update to AndroidX and be compatible with support libraries.

Thanks for the contribution! @athornz I agree, AndroidX with Jetifier should ideally be our default to get rid of any support issues. The problem we've run into so far is that Jetifier doesn't work with how we build Flutter apps and include plugins: flutter/flutter#32714. You can see this by making a new template with this PR or by migrating an app manually, then depending on a an older plugin (like firebase_core: 0.1.0), then running flutter build apk.
Even with Jetifier being broken for us @dnfield I think this is probably an improvement. At least all of flutter/plugins are migrated to AndroidX at this point, so with that it makes sense to me to have the default template be migrated too. Right now people still need to migrate manually.

Thanks @dnfield and @tvolkert for the review - I think I've addressed all the comments now but let me know if there is anything else.

LGTM.  Will land on green.
Thanks again!

No problem, thank you!

Looks like you're synced to a bad commit of master, you'll need to fetch and update

@jonahwilliams I've rebased on master now.

Flutter.jar is not compatible with androidX

@mcxinyu

Flutter.jar is not compatible with androidX

Currently I have a the app crashed:
    java.lang.NoClassDefFoundError: Failed resolution of: Landroidx/lifecycle/DefaultLifecycleObserver;
        at io.flutter.embedding.engine.FlutterEngine.<init>(FlutterEngine.java:149)

is it related to flutter.jar? cc @athornz @dnfield @tvolkert

That is because we are missing AndroidX transitive dependencies from the engine. We have a fix in progress. cc @matthew-carroll

In the meanwhile, you can the dependencies manually to android/build.gradle as indicated on https://developer.android.com/jetpack/androidx/releases/lifecycle#declaring_dependencies

The crash still happens even after I added the following snippet to app/build.gradle

    // known issue https://github.com/flutter/flutter/pull/31028#issuecomment-516902006
    def lifecycle_version = "2.0.0"
    // ViewModel and LiveData
    implementation "androidx.lifecycle:lifecycle-extensions:$lifecycle_version"
    // alternatively - just ViewModel
    implementation "androidx.lifecycle:lifecycle-viewmodel-ktx:$lifecycle_version"
    // alternatively - just LiveData
    implementation "androidx.lifecycle:lifecycle-livedata:$lifecycle_version"
    implementation "androidx.lifecycle:lifecycle-runtime:$lifecycle_version"
    kapt "androidx.lifecycle:lifecycle-compiler:$lifecycle_version"
    // optional - ReactiveStreams support for LiveData
    implementation "androidx.lifecycle:lifecycle-reactivestreams-ktx:$lifecycle_version"
    // optional - Test helpers for LiveData
    testImplementation "androidx.arch.core:core-testing:$lifecycle_version"

@truongsinh would you mind creating an issue with a repro test case? Thanks!

@truongsinh would you mind creating an issue with a repro test case? Thanks!

I might take a while, but sure.

Actually, trying to reproduce the error on a sample, I encoutered a different problem (here's the code https://github.com/truongsinh/android_flutter_host/tree/built-in-androidx)
Cannot open a library at 'FileMapping(from=/Users/truongsinh/Dev/flutter/android_flutter_host/flutter_embedding/.android/Flutter/build/intermediates/flutter/debug/libs.jar, to=/Users/truongsinh/.gradle/caches/transforms-1/files-1.1/libs.jar/9a5b6ea5d0ab9a291353d9eb6a6ec26b/jetified-libs.jar)'

@athornz @blasten any update?

update from my side,
Cannot open a library at 'FileMapping(from=/Users/truongsinh/Dev/flutter/android_flutter_host/flutter_embedding/.android/Flutter/build/intermediates/flutter/debug/libs.jar, to=/Users/truongsinh/.gradle/caches/transforms-1/files-1.1/libs.jar/9a5b6ea5d0ab9a291353d9eb6a6ec26b/jetified-libs.jar)'

is solved by work around, I have to cd flutter_embedding/.android && ./gradlew assembleDebug to generate those jars file, and then I can reproduce
E/AndroidRuntime: FATAL EXCEPTION: main
    Process: pro.truongsinh.flutter.android_flutter_host.free, PID: 11100
    java.lang.NoClassDefFoundError: Failed resolution of: Landroidx/lifecycle/DefaultLifecycleObserver;
        at io.flutter.embedding.engine.FlutterEngine.<init>(FlutterEngine.java:149)
        at pro.truongsinh.flutter.android_flutter_host.FlutterEmbeddingActivity$Companion.init(FlutterEmbeddingActivity.kt:168)
        at pro.truongsinh.flutter.android_flutter_host.MainActivity.onCreate(MainActivity.kt:17)
        at android.app.Activity.performCreate(Activity.java:7802)
        at android.app.Activity.performCreate(Activity.java:7791)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1299)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3243)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3407)
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:83)
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2016)
        at android.os.Handler.dispatchMessage(Handler.java:107)
        at android.os.Looper.loop(Looper.java:214)
        at android.app.ActivityThread.main(ActivityThread.java:7343)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:933)
     Caused by: java.lang.ClassNotFoundException: Didn't find class "androidx.lifecycle.DefaultLifecycleObserver" on path: DexPathList[[zip file "/data/app/pro.truongsinh.flutter.android_flutter_host.free-R3e6unzzN8OPsBvhFb5V2w==/base.apk"],nativeLibraryDirectories=[/data/app/pro.truongsinh.flutter.android_flutter_host.free-R3e6unzzN8OPsBvhFb5V2w==/lib/arm64, /data/app/pro.truongsinh.flutter.android_flutter_host.free-R3e6unzzN8OPsBvhFb5V2w==/base.apk!/lib/arm64-v8a, /system/lib64, /system/product/lib64]]
        at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:196)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
        at io.flutter.embedding.engine.FlutterEngine.<init>(FlutterEngine.java:149)
        at pro.truongsinh.flutter.android_flutter_host.FlutterEmbeddingActivity$Companion.init(FlutterEmbeddingActivity.kt:168)
        at pro.truongsinh.flutter.android_flutter_host.MainActivity.onCreate(MainActivity.kt:17)
        at android.app.Activity.performCreate(Activity.java:7802)
        at android.app.Activity.performCreate(Activity.java:7791)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1299)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3243)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3407)
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:83)
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2016)
        at android.os.Handler.dispatchMessage(Handler.java:107)
        at android.os.Looper.loop(Looper.java:214)
        at android.app.ActivityThread.main(ActivityThread.java:7343)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:933)
Disconnected from the target VM, address: 'localhost:8606', transport: 'socket'

Figured out, with some caveats:

I have to use lifecycle_version = "1.1.1", not 2.0.0

    def lifecycle_version = "1.1.1"
    implementation "android.arch.lifecycle:common-java8:$lifecycle_version"

Even with this, my example code still crashed (something like attempt to use destroyed engine), because I use shared cached Flutter enginer, and to fix this, have to override:

    override fun retainFlutterEngineAfterHostDestruction(): Boolean {
        return true
    }

Still caveats, it seems that the performance of transitioning between Kotlin activity and Flutter activity is much worse than before. Previously, with cached engine, it's really smooth. Even though low performance happens on debug mode only (not profile nor release), it might still be a performance degradation from previous version (1.5.4-hotfix)

It's hard for us to make sure we follow up on comments on previously merged PRs. If there are still remaining issues or inconveniences with the flows, please file a new issue with repro steps.

@truongsinh Btw, we will start shipping Maven artifacts. As a result, these artifacts will become the source of truth for the engine dependencies such as android.arch.lifecycle:common-java8. Once this happens, there won't be more NoClassDefFoundError.

@truongsinh Btw, we will start shipping Maven artifacts. As a result, these artifacts will become the source of truth for the engine dependencies such as android.arch.lifecycle:common-java8. Once this happens, there won't be more NoClassDefFoundError.

Do we have a github issue or PR to track this?

There's https://github.com/flutter/flutter/issues/11439


Remove needed patches

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (06405a8) to head (aa267af).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##                main      #203   +/-   ##
===========================================
  Coverage     100.00%   100.00%
  Complexity       594       594
===========================================
  Files             54        54
  Lines           1832      1832
===========================================
  Hits            1832      1832

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-yml-archive-non-versioning-archive/pull/326 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-yml-archive-non-versioning-archive.git

Docs Build status updates of commit f16665f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

docs/TOC.yml
:white_check_mark:Succeeded
View

docs/index.yml
:white_check_mark:Succeeded
View

docs/to-be-archive/level1/article3-1-1.md
:white_check_mark:Succeeded
View

docs/to-be-archive/level1/article3-1-2.md
:white_check_mark:Succeeded
View

docs/to-be-archive/level1/article3-1-3.yml
:white_check_mark:Succeeded
View

docs/to-be-archive/level1/sql-database-engine.png
:white_check_mark:Succeeded

docs/to-be-archive/level2/article3-2-1.md
:white_check_mark:Succeeded
View

docs/to-be-archive/level2/article3-2-2.md
:white_check_mark:Succeeded
View

docs/to-be-archive/level2/article3-2-3.yml
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Add support for optimizers and learning rate schedulers to LightningCLI

What does this PR do?
New feature for LightningCLI to make optimizers and learning rate schedulers easily configurable. It implements the proposal mentioned in https://github.com/PyTorchLightning/pytorch-lightning/issues/7576#issuecomment-852387071 except for minor differences required during development. How it should be used is explained in lightning_cli.rst.
Fixes #7576
Before submitting

[x] Was this discussed/approved via a GitHub issue? (not for typos and docs)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
[x] Did you make sure to update the documentation with your changes? (if necessary)
[x] Did you write any new necessary tests? (not for typos and docs)
[x] Did you verify new and existing tests pass locally with your changes?
[x] Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

[x] Is this pull request ready for review? (if not, please submit in draft mode)
[x] Check that all items from Before submitting are resolved
[x] Make sure the title is self-explanatory and the description concisely explains the PR
[x] Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?
Make sure you had fun coding 🙃

Hello @mauvilsa! Thanks for opening this PR.

In the file pytorch_lightning/utilities/cli.py:

Line 413:44: E226 missing whitespace around arithmetic operator
Line 413:48: E226 missing whitespace around arithmetic operator

Do see the Hitchhiker's guide to code style

It all sounds great!
I couldn't quite tell what the CLI would look for for choosing an which optimizer and which lr schedulers (and arguments) and how the default ones and values are set... but maybe I just can't read the diffs correctly.
The only other thing I would say is that whatever that way is, I would doublecheck that it works fine on the grid CLI with escaping characters/etc.  Since trying different permutations of optimizers, LR schedulers, and hyperparameters for them is a key use-case.  If grid has trouble with the escaping I think it is important to know early.

However, note that for SGD the learning rate by design is a required parameter and has no default. So in this case only giving the class path would not work. Someone could subclass SGD to give lr a default and make it work. But this is not recommended because for SGD there is no good default. The range of its optimal value depends on the task.

Given the comments above, I am worried that this has a terminal design flaw that makes it unusable in practice.
This sounds like there is no way to set reasonable defaults that are problem specific. And hence in practice an entire dictionary needs to be passed in each time. That is verbose. And "users" need to know what the reasonable problem dependent defaults are for those all as well. At that point I should be using a json file rather than the cli
If that is the design and I am not misunderstanding it, and there is no way to change the defaults in my code, I just dont think I can use this feature. Creating my own subclasses for all optimizers /etc is not a solution.
Isn't there a design or tweak that can make it more like the nice, hierarchical CLI that come up in the rest of this package?

@jlperla I believe that you have some expectations for the cli that I don't think are justified. What makes something a cli is being able to run something from the command line. It is not that every minute detail should be grid run sweepable using individual command line arguments. Don't confuse current cli limitations with limitations of grid which should be resolved there. The focus of LightningCLI from the start has been configuration files. Config files allow to overcome limitations that individual command line arguments have. Not sure what you mean by "json file rather than the cli" but that just makes me think of configuration file for a cli.
Luckily the optimizer example is perfect to illustrate why this should be done via a config file instead of individual command line arguments. Imagine that for some task you want to try different optimizers. Each optimizer has its own parameters that could be varied. Even if multiple optimizers have a parameter with the same name, e.g. lr, in general it does not make sense to sweep the same range of values all optimizers. If you are also going to try SGD it does not make sense to try it with a single "default" learning rate. Multiple values should be tried. That is simply the way SGD works, not related to a design of a cli or a sweeper. It is not possible to know a priori what a good value will be. That is why in pytorch it does not have a default value for it.
I have no good idea how the example above could be easily done using individual command line arguments. But with a config file this can be defined quite clearly, for example:
optimizer:
  sweep:
    - class_path: torch.optim.SGD
      init_args:
        lr:
          sweep: [0.00001, 0.0001, 0.001, 0.01, 0.1, 1.0]
        momentum:
          sweep: [0, 0.1]
    - class_path: torch.optim.Adam
      init_args:
        lr:
          sweep: [0.001, 0.01]
        amsgrad:
          sweep: [True, False]
    - class_path: torch.optim.RMSprop

This would be for some hypothetical sweeper that would take care of trying out all the combinations defined in the config file. The cli itself would not be aware of the sweeping and only get a config file with individual values without any sweep:.

@jlperla I believe that you have some expectations for the cli that I don't think are justified. What makes something a cli is being able to run something from the command line. It is not that every minute detail should be grid run sweepable using individual command line arguments.

To me, that is really the only reason I want to use the CLI.  If I can't make it sweepable with grid and then see the results in the grid webview then I need to use a different system.  And just to be clear, this is not narrowly for JUST grid.  I would be doing the identical thing on a cluster myself.  Come up with the cartesian product of these things, run a whole bunch of aruments with CLI, etc.  Not modify individual config files).  So this isn't any sort of "lets change PL to be more grid-friendly" as you would want the same thing even if you weren't on grid.
I should also point out that this is the only sane way to work with debugging in PL.  In vscode, you set the CLI arugments you want to work with in a new "launch" type and then hit F5.

Don't confuse current cli limitations with limitations of grid which should be resolved there. The focus of LightningCLI from the start has been configuration files.

I don't see a limitation of grid.  Except that with grid I hope to one day be able to execute these things programmatically instead of literally calling from a shell - but for that you would need just as much control over default values for execution.
I see a limitation of the CLI.  If the config files are the first and foremost goal, then I missed things (and I think it should be renamed as the name LightningCLI suggests otherwise).  What is the point then vs. Hydra, which does an amazing job of handling config files?

That is simply the way SGD works, not related to a design of a cli or a sweeper. It is not possible to know a priori what a good value will be. That is why in pytorch it does not have a default value for it.

Exactly!  That is why you need to find the value for your problem and then set it.  This shows why managing default values is so essential.

Luckily the optimizer example is perfect to illustrate why this should be done via a config file instead of individual command line arguments. I

If that is the assumption and the starting point, then that is fine - but it means that the feature is useful for people like me, and most low-tech users of grid.

Even if multiple optimizers have a parameter with the same name, e.g. lr,
Exactly.  That is why you need to tie CLI parameters as usual.

Each optimizer has its own parameters that could be varied.

Exactly, that is why you need to be able to change the defaults for when you choose that parameters.

I have no good idea how the example above could be easily done using individual command line arguments.

I think this is the problem.  It hasn't been designed for the CLI.  And just to be clear, for this, there is absolutetly nothing specific to my problem.  Every single project has the same need for choosing optimizers/learning rate schedulers... tinnkering arorund until you find a nice default... baking them in... and then moving onto the other parts of the project.
I have discussed this with @tchaton  and @zippeurfou at various points and they might have some insights on this general topic, but I think the core of the problem is that (for me at least), I want the LightningCLI as a very simple CLI solution for simple projects.  I want to be able to write a bunch of calls tot he function in either grid on on my cluster etc. without manually messing around with YAML files.  If I wanted a more complciated config file solution, I would look to hydra.  But I don't.

Being able to provide init arguments of a subclass as independent command line arguments is a general feature that would be implemented in jsonargparse. Regarding the defaults for subclasses, again this is a new feature for jsonargparse and probably with a small change in LightningArgumentParser to be able to specify them. I don't think these are difficult to do.
Neither of these features are specific to optimizers and learning rate schedulers which is the purpose of this pull request. And what is currently implemented here is not incompatible with these additional features. For me it is fine if this pull request does not close #7576 and it is closed only when the rest is ready. But I don't see any more comments that require changes for this pull request

base on explanation of complexity in https://github.com/PyTorchLightning/pytorch-lightning/pull/7508#issuecomment-874489470 pulling back my approval


Loading JSON throws ArrowNotImplementedError

Describe the bug
I have created a dataset of GitHub issues in line-separated JSON format and am finding that I cannot load it with the json loading script (see stack trace below).
Curiously, there is no problem loading the dataset with pandas which suggests some incorrect type inference is being made on the datasets side. For example, the stack trace indicates that some URL fields are being parsed as timestamps.
You can find a Colab notebook which reproduces the error here.
Edit: If one repeatedly tries to load the dataset, it eventually works but I think it would still be good to understand why it fails in the first place :)
Steps to reproduce the bug
from datasets import load_dataset
from huggingface_hub import hf_hub_url
import pandas as pd

# returns https://huggingface.co/datasets/lewtun/github-issues-test/resolve/main/issues-datasets.jsonl
data_files = hf_hub_url(repo_id="lewtun/github-issues-test", filename="issues-datasets.jsonl", repo_type="dataset")
# throws ArrowNotImplementedError
dset = load_dataset("json", data_files=data_files, split="test")
# no problem with pandas ...
df = pd.read_json(data_files, orient="records", lines=True)
df.head()

Expected results
I can load any line-separated JSON file, similar to pandas.
Actual results
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
<ipython-input-7-5b8e82b6c3a2> in <module>()
----> 1 dset = load_dataset("json", data_files=data_files, split="test")

9 frames
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: JSON conversion to struct<url: timestamp[s], html_url: timestamp[s], labels_url: timestamp[s], id: int64, node_id: timestamp[s], number: int64, title: timestamp[s], description: timestamp[s], creator: struct<login: timestamp[s], id: int64, node_id: timestamp[s], avatar_url: timestamp[s], gravatar_id: timestamp[s], url: timestamp[s], html_url: timestamp[s], followers_url: timestamp[s], following_url: timestamp[s], gists_url: timestamp[s], starred_url: timestamp[s], subscriptions_url: timestamp[s], organizations_url: timestamp[s], repos_url: timestamp[s], events_url: timestamp[s], received_events_url: timestamp[s], type: timestamp[s], site_admin: bool>, open_issues: int64, closed_issues: int64, state: timestamp[s], created_at: timestamp[s], updated_at: timestamp[s], due_on: timestamp[s], closed_at: timestamp[s]> is not supported

Environment info

datasets version: 1.11.0
Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
Python version: 3.7.11
PyArrow version: 3.0.0

Hi @lewtun, thanks for reporting.
Apparently, pyarrow.json tries to cast timestamp-like fields in jour JSON file to pyarrow timestamp type, and it fails with ArrowNotImplementedError.
I will investigate if there is a way to tell pyarrow not to try that timestamp casting.

I think the issue is more complex than that...
I just took one of your JSON lines an pyarrow.json read it without problem.

I just took one of your JSON lines an pyarrow.json read it without problem.

yes, and for some peculiar reason the error is non-deterministic (i was eventually able to load the whole dataset by just re-running the load_dataset cell multiple times 🤔)
thanks for looking into this 🙏 !

I think the error is generated by the pyarrow.json.read() option: read_options=paj.ReadOptions(block_size=block_size)...

The code works fine on my side.
Not sure what's going on here.
I remember we did a few changes in the JSON loader in #2638 , did you do an update datasets when debugging this ?

OK after upgrading datasets to v1.12.1 the issue seems to have gone away. Closing this now :)

Oops, I spoke too soon 😓
After deleting the cache and trying the above code snippet again I am hitting the same error. Here's a Colab notebook which reproduces the error: https://colab.research.google.com/drive/1DMV2sYro9GZjtmgOnSd859nfNBGwFNim?usp=sharing

Apparently it's possible to make it work by increasing the block_size, let me open a PR

I just opened a PR with a fix, feel free to install datasets from source from source and let me know if it helps

@zijwang did PR #3000 solve the problem for you? It did for me, so it all is good on your end we can close this issue. Thanks again to @lhoestq for the pyarrow magic 🤯


The road map lacking more core skills for developer, especially in China.

I just read the map originally and find it lack many fields, such as in the big data, which is not involved.
And the core part also lacking many things.
For example, in the ORMS part, Mybatis is a core skill for the developers who can work in China.
In the Framework part， Spring， SpringMVC，Springboot, Spring Cloud are independent, though it's based on the same principle.
In the testing part, JMeter is a more popular way than you have listed.
That's true for china's developers.
I am not sure whether it's correct for the developers outside China. You can consider adding it to your roadmap.

@oneWalker feel free to contribute


Test hover lacks padding

This is the glyphMarginHoverMessage, cc alex

This is the glyphMarginHoverMessage, cc alex

It looks like this happens because the hover label doesn't contain a node with .hover-contents, which effectively would add padding. Adding that class to all .hover-row nodes here solves that issue. Can I make a PR?

It looks like this happens because the hover label doesn't contain a node with .hover-contents, which effectively would add padding. Adding that class to all .hover-row nodes here solves that issue. Can I make a PR?

@goldst Thank you for figuring that out! There is some debt in this area that I would like to refactor. GlyphHoverWidget should be collapsed into ModesGlyphHoverWidget and the outcome of that needs to be refactored to reuse the HoverWidget from src/vs/base/browser/ui/hover/hoverWidget.ts. It would not be good to add class names to a widget that refer to CSS defined in other files. Given the needed refactorings, and the high chance of conflicts, I would say please hold off on a PR in this area, I plan to tackle this probably next week.

@goldst Thank you for figuring that out! There is some debt in this area that I would like to refactor. GlyphHoverWidget should be collapsed into ModesGlyphHoverWidget and the outcome of that needs to be refactored to reuse the HoverWidget from src/vs/base/browser/ui/hover/hoverWidget.ts. It would not be good to add class names to a widget that refer to CSS defined in other files. Given the needed refactorings, and the high chance of conflicts, I would say please hold off on a PR in this area, I plan to tackle this probably next week.

/verified


for End Users, include their user names within the "work requester" select options

@sabrine33 It doesn't specifically prevent it; it just doesn't specifically allow it. The list of possible work requesters is a list that the client gets from the core. If the client specifies a requester that isn't on that list, the core will not accept it, unless I change the core to allow it.

merged to the devel