updating nlp notebook

Why are these changes needed?
Update the nlp notebook:

Replaced the sentiment classification & model selection example with Colab built in GPU experiment.
Rerun multiple choice and text summarization on Ray Tune with 4 V100 GPU.

Related issue number

Checks

[x] I've used pre-commit to lint the changes in this PR, or I've made sure lint with flake8 output is two 0s.
[x] I've included any doc changes needed for https://microsoft.github.io/FLAML/. See https://microsoft.github.io/FLAML/docs/Contribute#documentation to build and test documentation locally.
[x] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[x] I've made sure all auto checks have passed.

When the PR is approved, can you please also make sure changes in this notebook are also included in the tutorial branch? Also can you add a version number at the end of this line pip install flaml[nlp,notebook,blendsearch] in the notebook (similar to the pip install in this notebook: https://github.com/microsoft/FLAML/blob/tutorial/notebook/automl_classification.ipynb) in the tutorial branch?
Thank you!


Cutroch + Magma breaks on update

Hello! I installed MAGMA
make all
make install

Then reinstalled torch from distro. At this point everything worked, but OMP went mad, and I had to limit process number for each of my tasks.
distro does not install latest versions of packages, so I casually started to update everything with luarocks install, It was fine till I did
luarocks install cutorch

now require 'cutorch' fails with
/home/lidl/torch/install/bin/luajit: /home/lidl/torch/install/share/lua/5.1/trepl/init.lua:384: /home/lidl/torch/install/share/lua/5.1/trepl/init.lua:384: /home/lidl/torch/install/share/lua/5.1/trepl/init.lua:384: /home/lidl/torch/install/share/lua/5.1/cutorch/init.lua:2: cannot load '/home/lidl/torch/install/lib/lua/5.1/libcutorch.so'
stack traceback:
        [C]: in function 'error'
        /home/lidl/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require'
        main.lua:14: in main chunk
        [C]: in function 'dofile'
        ...lidl/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00406670

And the only thing helps is to reinstall from distro. But other code throws a error with distro torch. Can you try to reproduce this error please?
Thank you!

What is the best practice for keeping two separate torch builds on one system?

I am sorry, removed ~/torch folder completely, reinstalled with ~/torch/install.sh, all updates luarocks install .. went smooth.
Now it throws a warning
warning: could not load nccl, falling back to default communication

Probably it was something related to nccl.


Django admin forms issues for employee

There is a bug in django admin form for custom employee user, password field keeps password hashing saving, so when save() method is used, password hash is overwritten again.

Fixed in this commit


Exiting VR does not switch to the previous active camera

Playground for repro:
http://playground.babylonjs.com/#WSMD93
Steps to reproduce:

Move the camera a bit (note you're using an ArcRotateCamera)
Switch to VR using the goggles button
Exit VR

Observe that you're no longer using the camera you were before.

By default the vrHelper creates it's own non-vr camera. Recently the feature to use a custom camera has been added. See http://doc.babylonjs.com/how_to/webvr_helper.
Create your own camera before creating the vr helper and then set the new camera creation to false.
scene.createDefaultVRExperience({createDeviceOrientationCamera: false}

See this playground and let me know if you still have an issue.
http://playground.babylonjs.com/#WSMD93#1
For even more custom logic you can listen to the VRHelper.onExitingVR event and manually switch cameras.


Description of user roles should be collapsible

The description of user roles should be collapsible rather than dismissable. It should be easy to view the description once it has been collapsed. Or use a tooltip that will describe the roles when rolling over the role.
To replicate the issue-

Click on 'Your organizations'
Click on 'Manage' for any organization
Dismiss the role description

Once it has been dismissed, it is not possible to get the description back

Priority 2/3.


⚠ Bofrost.lu has degraded performance

In 99b1a45, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 853 ms

Resolved: Bofrost.lu performance has improved in b7e1194 after 8 minutes.


No .corgidrp folder

I installed corgidrp on my Windows 10 computer, tried to run l1_to_l2a_e2e.py, and it crashed because there is no .corgidrp directory, let alone files within it, in my home directory.
I installed most dependencies listed in requirements.txt using Anaconda (e.g., conda install numpy), and then executed pip install -e . to get the rest. There was no report of a problem on the Anaconda Prompt.
After reading the Readme, I saw that there should be a .corgidrp folder in my home directory, checked, and saw none, so I knew something was wrong.
I downloaded the main branch on Tuesday, October 23. I think Jason Wang's "Merge pull request #211 from roman-corgi/cosmics_kgain" (9489f00fa1ed05b6469506e4446ac8c6e4a7fada) was the last commit when I installed.

Can you print out what you get for the following? import corgidrp; print(corgidrp.config_filepath) and confirm that file does not exist?
We've never formally tested corgidrp on windows, so you may need to use Windows Subsystem for Linux.

It looks like the .corgidrp folder was created when I ran the script in Windows, but only with an empty default_calibs/ folder and a corgidrp.cfg file, nothing else.
I made double sure to install git lfs ahead of time
I set up WSL and ran corgidrp from a completely native installation, and it looks like everything worked there.
As I learn the package more, I'll try and find clues as to why it crashes in Windows.
A peripheral detail - the Readme states that the ~/.corgidrp directory is created upon installation of the package, but even on WSL it only appeared when I ran a script for the first time.

thanks, that used to be the behavior, but we changed it (for pip-related reasons) to create .corgidrp on install.

regarding your windows install, try deleting the whole .corgidrp folder and try again.

Deleted every trace of the repo, the environment, and the home directory folder on Windows and started over.
Installed everything over again, and again tried running l1_to_l2a_e2e.py, which worked on WSL.
Crashed in the process of importing everything
Specifically, on line 10, import corgidrp.walker as walker
Traceback from Spyder:
File ~\anaconda3\envs\CorgiDRP_p312\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
exec(code, globals, locals)
File ~\documents\github\corgidrp\tests\e2e_tests\l1_to_l2a_e2e.py:10
import corgidrp.walker as walker
File ~\documents\github\corgidrp\corgidrp\walker.py:11
import corgidrp.caldb as caldb
File ~\documents\github\corgidrp\corgidrp\caldb.py:353
default_detparams.save(filedir=corgidrp.default_cal_dir)
File ~\documents\github\corgidrp\corgidrp\data.py:453 in save
hdulist.writeto(self.filepath, overwrite=True)
File ~\anaconda3\envs\CorgiDRP_p312\Lib\site-packages\astropy\io\fits\hdu\hdulist.py:1032 in writeto
fileobj = _File(fileobj, mode=mode, overwrite=overwrite)
File ~\anaconda3\envs\CorgiDRP_p312\Lib\site-packages\astropy\io\fits\file.py:218 in init
self._open_filename(fileobj, mode, overwrite)
File ~\anaconda3\envs\CorgiDRP_p312\Lib\site-packages\astropy\io\fits\file.py:651 in _open_filename
self._file = open(self.name, IO_FITS_MODES[mode])
OSError: [Errno 22] Invalid argument: 'C:\Users\user\.corgidrp\default_calibs\DetectorParams_2023-11-01T00:00:00.000.fits'
It stumbled on opening a FITS file from ~/.corgidrp
Checking that directory reveals that the default_calibs/ folder is (once again) empty
It looks like the code which populates that folder slips up on Windows and does not report any error.

Looking more closely, it appears that the default values for calibration are successfully put together, and the problem is in the attempted file name.
ChatGPT's first guess is that the problem lies with the colons in the file name, which Windows does not allow. The recommendation is to replace them with dashes.

Changing ':' to '-' in line 352 of caldb.py (second line of block below) is not a quick escape.
if not os.path.exists(os.path.join(corgidrp.default_cal_dir, "DetectorParams_2023-11-01T00:00:00.000.fits")):
    default_detparams = data.DetectorParams({}, date_valid=time.Time("2023-11-01 00:00:00", scale='utc'))
    default_detparams.save(filedir=corgidrp.default_cal_dir)

Error response to change:
  File ~\documents\github\corgidrp\corgidrp\caldb.py:352
    default_detparams = data.DetectorParams({}, date_valid=time.Time("2023-11-01 00-00-00", scale='utc'))

  File ~\anaconda3\envs\CorgiDRP_p312\Lib\site-packages\astropy\time\core.py:2021 in __init__
    self._init_from_vals(

  File ~\anaconda3\envs\CorgiDRP_p312\Lib\site-packages\astropy\time\core.py:559 in _init_from_vals
    self._time = self._get_time_fmt(

  File ~\anaconda3\envs\CorgiDRP_p312\Lib\site-packages\astropy\time\core.py:655 in _get_time_fmt
    raise ValueError(message)

ValueError: Input values did not match any of the formats where the format keyword is optional:
- 'datetime': Input values for datetime class must be datetime objects
- 'ymdhms': input must be dict or table-like
- 'iso': Time 2023-11-01 00-00-00 does not match iso format
- 'isot': Time 2023-11-01 00-00-00 does not match isot format
- 'yday': Time 2023-11-01 00-00-00 does not match yday format
- 'datetime64': Input values for datetime64 class must be datetime64 objects
- 'fits': Time 2023-11-01 00-00-00 does not match fits format
- 'byear_str': Time 2023-11-01 00-00-00 does not match byear_str format
- 'jyear_str': Time 2023-11-01 00-00-00 does not match jyear_str format
- 'astropy_time': Input values for astropy_time class must all be the same astropy Time type.

Clearly the datetime needs to be recognizable in a standard form.
Therefore, it appears to me at this moment that it will be necessary to add another step to each of the classes in data, processing the datetime values and turning them into strings that are acceptable within Windows filenames.

Thanks for the detailed investigation on this. I think this is fixable, but are you ok with using WSL?

For now, I think I can use WSL

Ok great. I think the addition we need to do is to do a check and replace on the filename when reading in the calibration file. It's a pretty straightforward fix, when we get around to it.

Note, the README still says, at the beginning of the first solid paragraph in the Install section, "The installation will create a configuration folder in your home directory called .corgidrp."
If that is out of date, there may be other parts to update as well.

@Post169 was this resolved? I'd like to close the issue or identify the next steps.

The original issue was that I was not able to run corgidrp on Windows, but I was pointed to a successful workaround for that.
A likely fix was identified by @semaphoreP.
My last comment was that the README needed some updating.


Frontend Setup

Setting up the frontend in React. Would need a basic react setup to kickstart the project.

i would like to do it


[Backport 2.x] Integrate translog cleanup with snapshot deletion and fix primary term deletion logic (#15657)

Backport 7b0846ecb1ec3707cf996b81e74559bff1442dd5 from #15657.

Duplicate of https://github.com/opensearch-project/OpenSearch/pull/15742


add Microsoft Open Source CoC to our Readme

Following the instructions here;
https://opensource.microsoft.com/codeofconduct/faq/
cc @mairaw

The change LGTM. But the page states this:

Projects in the .NET Foundation are already covered by the .NET Foundation Contributors Code of Conduct. As such, there is no need for .NET Foundation projects to make any changes.

So do we need to add this here?

Agree with @mairaw - as this is a .NET Foundation repo then the .NET Foundation Code of Conduct should apply.  Guidance on how to apply the code in your readme is given here but basically its:
This project has adopted the code of conduct defined by the Contributor Covenant
to clarify expected behavior in our community.
For more information see the
[.NET Foundation Code of Conduct](https://dotnetfoundation.org/code-of-conduct).

Thanks @martinwoodward. @BillWagner, I've addressed the feedback for you.

Thanks @mairaw I'll :shipit:


fix: Transfer page showing wrong tokens

PR Type

[x] Bugfix
[x] Feature
[x] Refactoring

Context

[x] fix: /transfer /telport /migrate permission based redirects
[x] Closes #10808
[x] Closes #10836

Did your issue had any of the "$" label on it?

[x] Fill up your DOT address: Payout

Screenshot 📸

[x] My fix has changed something on UI;

missing review fro, @kodadot/internal-dev


Firmware 4.0 for go-e-charger with new parameter "amx" for PWM Signalisierung.

Hello colleagues,
first of all: thank you for providing this adaper for iobroker - this is working great on my side.
I am not sure if you saw the latest 4.0 firmware change.  According to
https://github.com/goecharger/go-eCharger-API-v1/commit/3d46503129e4c37cf7c55a7b70b9b56e34464daa
there was a new set parameter "amx" created in parallel to "amp" to set one and the same value. The basic difference between amx and amp is, that since firmware 040.0 there is a possibilty to change the max_load parameter only in "ram" of the machine and not writing it to the flash (and yes, it will be copied from flash during a re-boot). Flash write access is limited both from space also from lifetime, so it makes sense to use amx instead of amp for setting max_load.
Is it possible to implement something like this -that shouldn´t be a big thing. I could offer to code if my help is needed.
if firmware >= "040.0"
use set (amx)
else
use set (amp)
Bests,
Markus

Sounds like a very good idea - I will check this

You could do me a favour.... my charger is still on firmware 033 - I would need someone to test on 040 if the object info.firmware is filled correctly.

I could of course offer check that. To reach that out two options are possible:
(1) Either there is a possiblity to read and display that amx value in iobroker (most convenient way for all users, but the you have to code both a setter and a getter :-)
(2) I could use the generated set method and set something different, see the result in the app and after reboot it should be restored to amp (which is different)

At the moment it's sufficient if you could verify that info.firmware in objects is 040 in your system

OK - my vault - two agents tried to write o the same box simultaneously.
It's working in my setup. Adapter version 0.4.0

Release of version 0.4.0


Implement Abort according to RFC 4960

Currently the Close methods immediately closes the connection.
We should send an abort packet before closing the connection.

@jeremija My apologies for my long absence. I'd like to help resolving this. Let me review the PR.


Dynamic Programming algorithms

I would like to work on dynamic programming algorithms. I can provide all the important algorithms such as knapsack, coin change, staircase, etc.  Please assign this issue to me as a SWOC 2021 participant.

Please don't consume an entire programming department for your issue.
And the most important thing, we add problems, their solutions and explanations, please create an issue which has only one problem stated.
Hence closing the issue.

Please don't consume an entire programming department for your issue.
And the most important thing, we add problems, their solutions and explanations, please create an issue which has only one problem stated.
Hence closing the issue.


Add Ujjal Sharma to Contributors list

Hi  @ujjalstark, I'm quite elated about your pull request. I wanna evolve this project to addresses various problems faced by first-time contributors. I'd love to learn about your journey in open source community, the problems, pain points you had etc.
Could you explain how you felt when you went through the tutorial, made a pull request and learned that I merged it?
We’ve recently added social share  to our web app. Could you please go to https://roshanjossey.github.io/first-contributions/#social-share and share your first contribution to open source?
Also, check out projects with easy issues while you’re there.

It was great to work first time with git command line especially and to have gained the mere sense of joining the open-source community after u merged it has given a lot of pleasure. Thanks again for your initiative. I learned python and have done some mini-projects but I had no idea of how to get involved in the open-source community but this page has been a wonderful first time guide. Thanks again for this.

I’d urge you to continue contributing to other open source projects.
We also have a slack team to help/mentor people trying to contribute to open source projects.
You can join the team by going to https://goo.gl/PM1Jdh


Fix/offscreen tooltip

resolves #1793

Went back and looked at the PR that led to clipAdjust, #1046.  I realized this type of change would break a lot of edge cases for people using tooltip such as that listed in  #932. So, I switched it around in a different PR ,  #1889, as this fix is not linked to a known bug in ours users code bases but is something we need in cases where we want to render the tooltip on page load and it's offscreen.


[bug] some mangas say they havd no chapters

Describe the bug
A clear and concise description of what the bug is.
Some mangas dont load chapters and just says they're empty
To Reproduce
Steps to reproduce the behavior:

Go to 'mob psycho 100'
Open it
See error

Expected behavior
A clear and concise description of what you expected to happen.
It should have shown a chapter list
Screenshots
If applicable, add screenshots to help explain your problem.
Please complete the following information:

Device: samsung galaxy s8
Android Version: 9
Version 1.6.4

Additional context
Add any other context about the problem here.

Are you logged in?
Also have you tried going to the chapter list and swipe down refresh. It shows for me no issue

Try the yaoi manga "psycho" that also doesnt work. Or the unordinary webtoon.

And yes, I tried it

Do you have settings on the website potentially blocking?

Also psycho has no English chapters. So I don't know what you are expecting there


Watchtower don't work

Description
root@lenovo2 / cat /etc/docker/daemon.json
{
        "proxies": {
                "https-proxy": "socks5://proxy.huecker.io:1080",
                "no-proxy": "ghcr.io,registry.rocket.chat,<IP_ADDRESS>/8"
        },
        "registry-mirrors": [
                "https://huecker.io",
                "https://mirror.gcr.io/",
                "https://cr.yandex/mirror",
                "https://noohub.ru",
                "https://public.ecr.aws",
                "https://quay.io",
                "https://dockerhub1.beget.com",
                "https://dockerhub.timeweb.cloud",
                "https://jockerhub.com/",
                "https://dcr-px.ru",
                "https://daocloud.io",
                "https://c.163.com",
                "https://registry.docker-cn.com"
        ]
}

Error in telegram:
Watchtower updates on lenovo2
Could not do a head request for "mariadb:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "bbilly1/tubearchivist:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "grafana/grafana:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "gitea/gitea:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "impworks/bonsai:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "docker.io/library/redis:alpine", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "jc21/nginx-proxy-manager:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "postgres:alpine", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "docker.io/library/postgres:12-alpine", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "wordpress:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "prom/prometheus:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "prom/snmp-exporter:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "mongo:5.0", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "registry:2", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "portainer/portainer-ce:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "gitea/act_runner:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "bbilly1/tubearchivist-es:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "redis/redis-stack-server:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "rocketchat/hubot-rocketchat:latest", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "jitsi/jicofo:stable-9111", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "jitsi/jvb:stable-9111", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "jitsi/prosody:stable-9111", falling back to regular pull.
Reason: unsupported challenge type from registry
Could not do a head request for "jitsi/web:stable-9111", falling back to regular pull.
Reason: unsupported challenge type from registry

Hi! To make huecker.io work with Watchtower, you can pass the HTTPS_PROXY and NO_PROXY environment variables similar to the Docker Engine.
Example Compose configuration:
services:
  watchtower:
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      HTTPS_PROXY: "socks5://proxy.huecker.io:1080"
      NO_PROXY: "*.docker.io"

I think I should add this to the documentation. Thanks for helping to find this issue!


"cycle error" regression with let fields between 0.4.3 and 0.5.0

What version of CUE are you using (cue version)?

$ cue version
cue version v0.6.0-alpha.1.0.20230418084629-59080b65a5ba

go version devel go1.21-9d35ebba06 Thu Apr 20 01:07:29 2023 +0000
      -buildmode exe
       -compiler gc
        -ldflags -w -s
  DefaultGODEBUG panicnil=1
     CGO_ENABLED 1
          GOARCH amd64
            GOOS linux
         GOAMD64 v3

Does this issue reproduce with the latest stable release?
Yes; tested with v0.5.0 and master per the version output above.
What did you do?
$ cue export -
#GetF1: {
	in:  _
	out: in.f1
}

#PutOut1: {
	in: _
	out: out1: "\(in)"
}

export: {
	let data = {f1: "foo"}
	(#PutOut1 & {
		in: (#GetF1 & {
			in: data
		}).out
	}).out
}

What did you expect to see?
Same output as v0.4.3:
{
    "export": {
        "out1": "foo"
    }
}

What did you see instead?
invalid interpolation: cycle error:
    -:8:13

This bug is very similar to https://github.com/cue-lang/cue/issues/2351; it's entirely possible that both will be fixed with the same commit. However, that other bug was replacing one error with a slightly less helpful error, whereas this bug is turning a successful export into an error. I also think the cycle isn't valid; just like with #2351, embedding a struct shouldn't break let definitions.
Thanks to @massive for reporting this bug and helping investigate it.

Not just let fields
#Build: {
	steps: [#Step, ...#Step]
	output: #Image

	_dag: {
		for idx, step in steps if idx == 0 {
			"\(idx)": step
		}

		for idx, step in steps if idx > 0 {
			"\(idx)": {
                                 // here throw cyclic task dependency in cueflow
                                 // it works well with v0.4.3 and v0.5.0-beta.1
				_prev: _dag["\(idx-1)"].output

				step & {
					input: _prev
				}
			}
		}
	}

	if len(_dag) > 0 {
		output: _dag["\(len(_dag)-1)"].output
	}
}

#Step: {
	input?: #Image
	output: #Image
	...
}

@morlay could you please raise a new bug following the bug report template? That will include enough information for us to reproduce your problem and be able investigate it. What you shared here isn't enough, and on a first look, the lack of let fields means that it's a different issue.

@myitcv Sorry.
My case may a issue of tools/flow
I have to add new config DisableDynamicTasks to disable initTasks when after task.Fill() for a quick fix.
https://github.com/morlay/cue/commit/b9dd73e5a30879bdb2d5bab3e2fcaa113840bb77
Since v0.5.0-beta.2 (v0.5.0-beta.1 works well) the task deps will be changed after task.Fill() in complex workflow, even all tasks are static.
I will try to create another issue to post this, when I have enough time to make a simplier test case.

@myitcv just let you know.
Since I upgrade to v0.6.0-rc.1, my issue is gone without any patch.

@myitcv just let you know.
Since I upgrade to v0.6.0-rc.1, my issue is gone without any patch.

Thanks very much for confirming, @morlay

Thanks very much for confirming, @morlay

cycle error is gone, but deps issue coming 🤣
https://github.com/cue-lang/cue/issues/2516 https://github.com/cue-lang/cue/issues/2517


cmd/pgopack: cache packed pgo profile during build

The PGO-enabled Go compilation shows the significant latency improvement of generated binary which benefits from profile-guided inlining and code specialization. However, the workflow of processing and extracting profiling data is surprisingly suboptimal in the community Go compiler. Based on the observation and instrument timing data from small benchmark and real service, we notice that the overall pprof data parsing time varies from 70% to over 95% of total time when PGO flow is enabled in the compilation. Furthermore, the existing PGO flow reads and parses the pprof file in every single package compilation and the cumulative time of this process is long and unnecessary. One-shot processing profiling data and cache information in a well-designed format for optimization transformation among all packages is critical.
To optimize the existing flow and reduce build time of multiple Go services and projects, we propose to create a standalone tool, PGO preprocessor, to extract information from collected profiling files and to cache the WeightedCallGraph in one time fashion. By adding the new tool to the Go compiler, it reduces the time of repeated profiling file parsing in current Go Compiler. The new tool is capable of all existing PGO enabled optimizations including the inlining and devirtualization in the Go compiler.
In summary, we propose adding a standalone preprocess tool to reduce the compile time when the PGO is enabled. Inputs are welcome.

For the record, speeding up the underlying work is #58102.

@jinlin-bayarea Are your .pgo files in version control, so that all builds use PGO? I know that's the Go team's recommendation (here, for example).
At my company we've only started playing around with PGO a bit but our current approach is to only build with PGO as part of production deploys, not local development/testing. So having PGO builds be a bit slower doesn't really matter much.
(We don't want to put the .pgo files in our repo because we are collecting very large profiles and we are updating them with automated processes.)

The pgo files are not inversion control. We have collected our pgo profiles daily and always use the most recent profile to build the service code.

cc @aclements @cherrymui

Discussed briefly offline. The idea of preprocessing sounds good overall.

In the usual case we'd want the go command automatically runs the preprocessor and cache the result. So this step is transparent to the user, and go build -pgo still works as before.
The preprocessor can be a standalone command, so external build tools like Bazel can invoke it (and do its own caching).

It would be good if you could share a prototype CL. Then we can discuss the details from there, like the exact format of the preprocessed profile. Thanks.

Hi Cherry. I have submitted the prototype in this CL. https://go-review.googlesource.com/c/go/+/529738.
Please help review the changes. Thanks.

I would suggest calling it pgopack or something like that instead of "preprofile" since Go has lots of profiles. Otherwise this seems fine, and it should be invisible to users, so I think it can proceed with compiler/runtime team agreement instead of a proposal.
/cc @bcmills

Taking out of proposal as this is invisible to users using the go command. The tool will be internal (the go command will run it automatically if needed), so is the preprocessed profile format. Users are not expected to check in profiles generated by the preprocessing tool.

Agree with @cherrymui . For bazel build users, they need to use the preprocess command separately to retrieve the preprocessed profile.


Uncaught TypeError: Cannot read property '0' of undefined

[Enter steps to reproduce:]

...
...

Atom: 1.14.4 x64
Electron: 1.3.13
OS: Mac OS X 10.12.3
Thrown From: learn-ide package 2.4.2
Stack Trace
Uncaught TypeError: Cannot read property '0' of undefined
At /Users/kelton/.atom/packages/learn-ide/node_modules/learn-term.js/src/term.js:1306

TypeError: Cannot read property '0' of undefined
    at TerminalWrapper.Terminal.refresh (/packages/learn-ide/node_modules/learn-term.js/src/term.js:1306:18)
    at TerminalWrapper.Terminal._cursorBlink (/packages/learn-ide/node_modules/learn-term.js/src/term.js:1422:8)
    at _blinker (/packages/learn-ide/node_modules/learn-term.js/src/term.js:1439:10)

Commands

Non-Core Packages
atom-material-syntax 1.0.2
atom-material-syntax-dark 0.2.7
atom-material-syntax-light 0.4.6
atom-runner 2.7.1
autocomplete-ruby 0.2.6
learn-ide 2.4.2
learn-ide-material-ui 1.3.14
learn-ide-tree 1.0.22
linter 1.11.23
linter-rubocop 1.1.0

This was addressed in #409 an will be released with v2.5 🎉


implement stable unique inode for passthroughfs

In the current passthroughfs implementation, temporary inodes are allocated and then stored in memory, But a FORGET request causes the inode to be removed from memory, new anodes are reassigned, resulting in the instability of the inodes of a file. this PR implements a stable inode, Limited by the current implementation limitations of VFS, Host inode + mnt + dev needs to be controlled in 56bit, and it is not supported beyond the range.

Forget does not delete the internal inode state, in this way can also ensure a stable inode, but it requires some memory overhead, 100w different files occupy 60M


Make it easier for adding custom selector utilities in typescript

Discussed in https://github.com/modulz/stitches/discussions/969

Originally posted by joseDaKing February 24, 2022
Add custom selector utilities in typescript will yield the wrong type.
This wont work because of self refercing type
import {  createStitches, CSS as BaseCSS } from "@stitches/core";

const stitches = createStitches({
    theme: {
        space: {
            sm: "16px",
            md: "32px",
            lg: "64px"
        }
    },
    utils: {
        hover: (styles: CSS) => ({
            "@media (hover: hover)": {
                "&:hover": styles
            }
        })
    }
});

type CSS = BaseCSS<typeof stitches.theme>;

The only way to make it work is by using the CSS type without passing in the theme, but the theme tokens won't be typed anymore
import {  createStitches, CSS } from "@stitches/core";

const stitches = createStitches({
    theme: {
        space: {
            sm: "16px",
            md: "32px",
            lg: "64px"
        }
    },
    utils: {
        hover: (styles: CSS) => ({
            "@media (hover: hover)": {
                "&:hover": styles
            }
        })
    }
});
```</div>

@joseDaKing will this not work and automatically pick up the type for your custom hover util?
export const { css, theme, createTheme, getCssText, globalCss, keyframes, config } =
	createStitches({
		theme: {
			colors: {},
			fonts: {},
			shadows: {},
			sizes: {},
			space: {},
			fontSizes: {},
			radii: {},
			zIndices: {},
		},
		media: {},
		utils: {},
	});

export type CSS = Stitches.CSS<typeof config>;


Add MT State Public Health

Looks like the state of Montana is piping all the county health department clinics through a prepmod site.
https://www.mtreadyclinic.org/clinic/search
It would be a great addition for our community.

@GUI @elocke I can take this one on!


[BUG] kafka consumers stop receiving new data from kop

Describe the bug

we create a partitioned-topic "BD_YD_OrgInsight_STEP", with 10 partitions, using ./pulsar-admin command.
we deploy a kafka producer to send data to pulsar continuously and deploy a kafka consumer.
Kafka consumer can normally consume part of the data after starting, and then stops receiving new data.
At this time, pulsar consumers work well and recevice data normally！

To Reproduce
Steps to reproduce the behavior:

Start a consumer  by running a program or using kafka commands .
the consumer will receive some data after it is started immediately, and then stops receiving.
This phenomenon can recur stably.

Expected behavior
kafka consumers should receive data normallly.
Screenshots
If applicable, add screenshots to help explain your problem.

Kafka CLI 工具看到的group 信息如图

This is the broker logs (<IP_ADDRESS>-209)
2022-08-22_18_25.zip

Look like the broker logs cannot download. Can you check if the link is correct?

In addition to the broken download url, could you describe it in detail?
I see there are 3 consumers from the group describe result, however, you said

Start a consumer by running a program or using kafka commands .

Could you provide your application code or the full Kafka commands? Could you try only one consumer to see if the problem could happen again?
How long will this issue happen since your application started? Could you adjust the log level to debug?
Could you try reproducing it with only 1 broker available?

Look like the broker logs cannot download. Can you check if the link is correct?

@Demogorgon314 I have uploaded again, thanks.

In addition to the broken download url, could you describe it in detail?
I see there are 3 consumers from the group describe result, however, you said

Start a consumer by running a program or using kafka commands .

Could you provide your application code or the full Kafka commands? Could you try only one consumer to see if the problem could happen again?
How long will this issue happen since your application started? Could you adjust the log level to debug?
Could you try reproducing it with only 1 broker available?

1、We use command ” ./kafka-console-consumer.sh --bootstrap-server  <IP_ADDRESS>:9092,<IP_ADDRESS>:9092,10..100.100.209:9092 --topic BD_YD_OrgInsight_STEP “ to start a consumer.
the problem  still happens even if only one consumer.
2、It happended  when consumer started immediately, in several second.
3、we will try when only one broker.

2022-08-23 13_47_01.zip
@BewareMyPower this is the debug logs after i run command "./kafka-console-consumer.sh --bootstrap-server <IP_ADDRESS>:9092,<IP_ADDRESS>:9092,10..100.100.209:9092 --topic BD_YD_OrgInsight_STEP "

[root@kxnhhj6f bin]# ./pulsar-admin topics partitioned-stats persistent://public/default/BD_YD_OrgInsight_STEP
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" :<PHONE_NUMBER>1947303,
"msgThroughputOut" :<PHONE_NUMBER>678956,
"bytesInCounter" : 801274020,
"msgInCounter" : 2846790,
"bytesOutCounter" :<PHONE_NUMBER>,
"msgOutCounter" : 4278205,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" :<PHONE_NUMBER>7,
"backlogSize" : 46843493,
"publishRateLimitedTimes" : 0,
"earliestMsgPublishTimeInBacklogs" : 0,
"offloadedStorageSize" : 0,
"lastOffloadLedgerId" : 0,
"lastOffloadSuccessTimeStamp" : 0,
"lastOffloadFailureTimeStamp" : 0,
"publishers" : [ {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"averageMsgSize" : 0.0,
"chunkedMessageRate" : 0.0,
"producerId" : 0,
"supportsPartialProducer" : false
} ],
"waitingPublishers" : 0,
"subscriptions" : {
"dsj_cashflow_service_test_node" : {
"msgRateOut" :<PHONE_NUMBER>1947303,
"msgThroughputOut" :<PHONE_NUMBER>678956,
"bytesOutCounter" :<PHONE_NUMBER>,
"msgOutCounter" : 4278205,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0,
"msgBacklog" : 68544,
"backlogSize" : 0,
"earliestMsgPublishTimeInBacklog" : 0,
"msgBacklogNoDelayed" : 68544,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"lastMarkDeleteAdvancedTimestamp" : 0,
"consumers" : [ {
"msgRateOut" :<PHONE_NUMBER>1947303,
"msgThroughputOut" :<PHONE_NUMBER>678956,
"bytesOutCounter" : 648862028,
"msgOutCounter" : 2266642,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"availablePermits" : -642,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0
} ],
"isDurable" : true,
"isReplicated" : true,
"allowOutOfOrderDelivery" : false,
"consumersAfterMarkDeletePosition" : { },
"nonContiguousDeletedMessagesRanges" : 30,
"nonContiguousDeletedMessagesRangesSerializedSize" : 740,
"subscriptionProperties" : { },
"durable" : true,
"replicated" : true
}
},
"replication" : { },
"nonContiguousDeletedMessagesRanges" : 30,
"nonContiguousDeletedMessagesRangesSerializedSize" : 740,
"compaction" : {
"lastCompactionRemovedEventCount" : 0,
"lastCompactionSucceedTimestamp" : 0,
"lastCompactionFailedTimestamp" : 0,
"lastCompactionDurationTimeInMills" : 0
},
"metadata" : {
"partitions" : 10
},
"partitions" : { }
}
[root@kxnhhj6f bin]# ./pulsar-admin topics partitioned-stats persistent://public/default/BD_YD_OrgInsight_STEP
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" :<PHONE_NUMBER>1947303,
"msgThroughputOut" :<PHONE_NUMBER>678956,
"bytesInCounter" : 801274020,
"msgInCounter" : 2846790,
"bytesOutCounter" :<PHONE_NUMBER>,
"msgOutCounter" : 4278205,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" :<PHONE_NUMBER>7,
"backlogSize" : 46843493,
"publishRateLimitedTimes" : 0,
"earliestMsgPublishTimeInBacklogs" : 0,
"offloadedStorageSize" : 0,
"lastOffloadLedgerId" : 0,
"lastOffloadSuccessTimeStamp" : 0,
"lastOffloadFailureTimeStamp" : 0,
"publishers" : [ {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"averageMsgSize" : 0.0,
"chunkedMessageRate" : 0.0,
"producerId" : 0,
"supportsPartialProducer" : false
} ],
"waitingPublishers" : 0,
"subscriptions" : {
"dsj_cashflow_service_test_node" : {
"msgRateOut" :<PHONE_NUMBER>1947303,
"msgThroughputOut" :<PHONE_NUMBER>678956,
"bytesOutCounter" :<PHONE_NUMBER>,
"msgOutCounter" : 4278205,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0,
"msgBacklog" : 68544,
"backlogSize" : 0,
"earliestMsgPublishTimeInBacklog" : 0,
"msgBacklogNoDelayed" : 68544,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"lastMarkDeleteAdvancedTimestamp" : 0,
"consumers" : [ {
"msgRateOut" :<PHONE_NUMBER>1947303,
"msgThroughputOut" :<PHONE_NUMBER>678956,
"bytesOutCounter" : 648862028,
"msgOutCounter" : 2266642,
"msgRateRedeliver" : 0.0,
"chunkedMessageRate" : 0.0,
"availablePermits" : -642,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0
} ],
"isDurable" : true,
"isReplicated" : true,
"allowOutOfOrderDelivery" : false,
"consumersAfterMarkDeletePosition" : { },
"nonContiguousDeletedMessagesRanges" : 30,
"nonContiguousDeletedMessagesRangesSerializedSize" : 740,
"subscriptionProperties" : { },
"durable" : true,
"replicated" : true
}
},
"replication" : { },
"nonContiguousDeletedMessagesRanges" : 30,
"nonContiguousDeletedMessagesRangesSerializedSize" : 740,
"compaction" : {
"lastCompactionRemovedEventCount" : 0,
"lastCompactionSucceedTimestamp" : 0,
"lastCompactionFailedTimestamp" : 0,
"lastCompactionDurationTimeInMills" : 0
},
"metadata" : {
"partitions" : 10
},
"partitions" : { }
}

2022-08-23 13_47_01.zip
@BewareMyPower this is the debug logs after i run command "./kafka-console-consumer.sh --bootstrap-server <IP_ADDRESS>:9092,<IP_ADDRESS>:9092,10..100.100.209:9092 --topic BD_YD_OrgInsight_STEP "

The debug log level was not enabled. See the instruction here, if the logs are written into the file, e.g. start the broker in background (pulsar-daemon start broker), you need to change the ref from Console to RollingFile.
BTW, I found the following log appeared frequently in broker-log-2.txt:

WARN  org.apache.pulsar.broker.namespace.NamespaceService - Broker http://<IP_ADDRESS>:8080 (<IP_ADDRESS>:8080) couldn't be found in available brokers <IP_ADDRESS>:8080,<IP_ADDRESS>:8080
WARN  org.apache.pulsar.broker.namespace.NamespaceService - The current leader broker LeaderBroker(serviceUrl=http://<IP_ADDRESS>:8080) isn't active. Handling load manager decisions in a decentralized w       ay. NamespaceBundle[gmtp/sse_SHL2/0xc0000000_0xffffffff]

The HTTP endpoint of broker in <IP_ADDRESS> is available. There might be something wrong with the load manager, could you help take a look? @Demogorgon314

2022-08-24 kop-log.zip
This the broker's log (kop related logs are printed in pulsar-broker-kop-xxx.txt)

This broker configs's example :
broker-01.conf.txt

broker-configs-08-24.zip
config files for all brokers.


Trailing white space in SR300 serial number

Required Info

Camera Model
SR300

Firmware Version
<IP_ADDRESS>

Operating System & Version
Ubuntu 14.04.4 LTS

Kernel Version (Linux Only)
4.4.0-040400-generic

Build System
Makefile

The serial number of few SR300 cameras have trailing white space (E.g. "60920300464 "). It is returned by rs_get_device_serial().

Hi @reaganlo -- seems like an issue the ROS team could fix/pull request easily.

@dorodnic Confirmed that PR #240 fixes this bug.


Possible strange behaviour with HDMI on first boot

Apologies for duplicating my post on the Pine64 forum. Unfortunately I'm unable to reply further due to an anti-spam measure that they have introduced on the forums (according to my IRC conversation, as a new user I have to wait three days before I can make my second post).
For completeness I'm going to include my original text:

I've observed a weird issue with the xenial-pine64-bspkernel-20161218-1.img image whilst trying to get it to run headless on my Pine64. Based on an evening of flashing and re-flashing SD cards I have concluded that:
If an HDMI display is connected on the first ever boot then it seems that the OS will NEVER boot without an HDMI display.
If NO HDMI display is connected on the first ever boot then the OS will boot happily - with or without a display for ever more.
This has tripped me up on an OpenHABian derivative image that exhibits the same behaviour (see issue at https://github.com/openhab/openhabian/issues/105).
I figure there is a script that is running on the first ever boot that sets a piece of configuration differently depending on whether a display is connected or not. Thus far I've not figured out what that is or how to fix it so that a system booted with HDMI the first time can later be booted headless.
Sadly I do not have a serial cable for my P64 so am unable to see the console and figure out what's happening.
Sounds suspiciously like unintended behaviour - if anyone has any suggestions then I'd be glad to hear them.

@longsleep kindly replied thus:

Well this sounds strange. The only thing that happens on the first boot is generating keys. This takes a lot of computing power. May be the power supply is not sufficient for this and when HDMI is connected extra power is available through HDMI.
If the board does not not, do you know what the error is? Where is it stuck? How did you find out that it did not boot?

I'm using a Raspberry Pi 2A PSU that I had to hand so I'm reasonably confident that power is not an issue. Also, the issue only occurs on subsequent boots if an HDMI display was attached on the first boot - and it doesn't sound like it should be generating keys on subsequent boots.
My testing setup has been brutally simple: have it plugged into an ethernet port. My criteria as to whether it has booted or not is whether the interface comes up and I see traffic on the port. I've been leaving my laptop pinging the IP address. Crude, but effective and reproducible many times.
I was looking at dmesg output and noticed that the sunxi disp2 is initialised once on first boot and twice on subsequent boots. I have no idea if that's connected.
Sadly it's impossible to tell where it is stuck without a display or console attached. I've just ordered a USB/UART cable so I can do that (been regretting not buying the Pine64 adaptor in the first place). I might try seeing if I can connect it to the serial port of a raspberry pi tonight rather that waiting for that delivery.
I'd be intrigued to know if anyone else was able to re-produce it (or not able to re-produce it) - would give me more confidence that this is actually a thing rather than it being something silly that I've done or my particular board.
I'll write more when I discover anything new.

Well, just to be clear. I have flashed my images many times and usually do not have HDMI connected at all ever. I gues the issue is specific to your particular setup.

Yes, and that works. Unfortunately I built a machine that happened to be connected to HDMI on first boot and now I can't unplug the display to hide it in a cupboard as it won't boot :(
The simplest answer for me is to rebuild it and start over (which is now my plan for tonight), but that won't solve it for future users and violates the principle of least surprise.

I'll test that tonight, as I can't say with certainly I've done exactly that... connected with HDMI in the first instance, and then run the pine64 headless afterwards. I have mostly run it with HDMI connected all the time as it was a GUI image, or with no HDMI connected right from the start as I have run it with a console cable connected for the initial configuration.
btw, you should be able to post 1 message per day during the settling in period. If not, please send me a PM (same handle on the forum), as it means something has been misconfigured.

@pfeerick I am able to post again. It would be really helpful if you could add another line of text to the error page that indicates that rate limiting might be the reason.
I'm really interested to hear what your results are :)

I wasn't able to reproduce that behaviour. Here was my test methodology so we can verify were are on the same page.
I have booted a fresh image of Ubuntu (https://www.stdin.xyz/downloads/people/longsleep/pine64-images/ubuntu/xenial-pine64-bspkernel-20161218-1.img.xz). I plugged in a wireless USB keyboard/mouse dongle, ethernet, and HDMI. Powered up the pine64, let it boot up, logged in, rebooted. I pulled the HDMI as the pine64 was shutting down. Watched the ethernet lights, the pine64 came back up again, and I was able to log in via SSH.
So it has booted up with HDMI in the first instance, and had no problems. Booting up without the HDMI also appear to be fine. I tried powering up the pine64 up and down a few times, and it continued to start up flawlessly, so it wasn't a one off brought about by rebooting it.
My power supply is a 5A capable 12v to quad-usb converter, and it is tuned to the slightly higher voltage of 5.2v. Hopefully that will start to determine what is the cause of the problem. If you have a similar setup bar the power supply, then it does start sounding like it is power related.

Hmmm, curious. That does sound similar - except I have not plugged in a mouse or keyboard, just HDMI (that sounds ridiculous now I'm writing it down, but none the less).
I'll have another go tonight.

Thanks for testing this. I am very interested in getting this resolved. @sihil do you have an alternative power supply which you could try? Preferably power via the PINs on the Euler connector.
Also connecting any extra USB devices like keyboard or mouse require even more power unless they are connected via a powered USB hub which then might in turn feed power to Pine64.

Doesn't sound too ridiculous... you can always plug in the keyboard/mouse after the pine64 has booted and you can see stuff on the screen... or you might have the screen connected just to see boot messages ;)
Another thing to consider is kernel/uboot updates. If you had done that on the first boot, and something went wrong (it can happen, but it is likely to be power or sd card corruption related), that could be the cause, not the first boot with HDMI. In other words, don't do it (just in case that is the issue).  And as longsleep said, alternate power supply to the euler pins would be great also, as that will provide more reliable power to the pine64.

I experienced the same issue again. I'll see if I can borrow a workbench PSU and do as you suggest.

I am seeing similar behaviours that you are @sihil when I flashed the xenial-pine64-bspkernel-20161218-1.img. In my case my goal is to run headless, only access the board by ssh.
After flashing the board, I did not connect any cables except power (5V 2A) and ethernet. The board sometimes would come up though other times it would not. I read your post on the forum that it had some success when connecting an HDMI display so I tried that. And to my luck it came up just fine. I then unplugged the HDMI cable and used it headless.
However, if I reboot the board or power is lost, there is a good chance it won't come back up unless I connect an HDMI monitor and power cycle it a few times.

@RyanRamchandar - so far i have seen no indication that there is a general issue with my image. I strongly suggest you get a better power supply or a lower AWG cable as i still thing you guys suffer from a voltage drop which makes things go sideways on boot and HDMI just gives the extra juice to cope with that.

I didn't want to think it was a power supply issue either, but when running off a bench power supply (5A, good filtering), my previously 100% repro crash went away.
Possible solution: A 10µF tantalum (low ESR) capacitor soldered between the DC IN and GND pins of the Euler connector (via a 2x3 female header).  Result: It's not 100% successful, but I've had 4 successful boots out of 5 now.  Maybe a bigger cap will do it.

I didn't want to think it was a power supply issue either, but when running off a bench power supply (5A, good filtering), my previously 100% repro crash went away.
Possible solution: A 10µF tantalum (low ESR) capacitor soldered between the DC IN and GND pins of the Euler connector (via a 2x3 female header). Result: It's not 100% successful, but I've had 4 successful boots out of 5 now. Maybe a bigger cap will do it.

So what are you saying. It does not crash with your bench PSU? What is the reason for the capacitor? Did you try to slightly increase voltage with the bench PSU to 5.1V or 5.2V?

Yes, with my bench supply (set at 5.00v as exactly as possible) no crash.  With all my other power supplies it crashed.  Didn't try a higher voltage on the bench supply, because it works fine.
Adding a capacitor between DC IN and GND on the Euler connector gets booting working on several of those supplies... most of the time (roughly 80%).

Hi, i do encounter the same issue using headless image with kernel 3.10.105. However, it is not caused by HDMI but the ethernet. It cannot boot up at all and shows "BUG: soft lockup - CPU#0 stuck for 22s! " without ethernet plugged in but it sometimes can boot up successfully with ethernet plugged in. So, is it related to power supply issue too?

@whongx yes - Ethernet draws quite some power and Gigabit Ethernet even more.

@longsleep ok!
Ya! I forget to mention that it does not encounter the issue when using kernel 3.10.104.

@whongx what does it mean "cannot boot up" ? Do you have logs or at least an error message?

@longsleep
Most likely related: similar issue can be reproduced with Armbian builds (your BSP kernel source with slightly different configuration). Kernel randomly stalls on boot with different stall to success rate depending on connected/disconnected Ethernet, connected/disconnected HDMI display, etc., but there is no clear conection between these factors.
Dmesg logs with stack traces can be found in attachments in this thread, I'm attaching one of them here:
BOOTFail_2017-04-15-C1.txt
According to my understanding it locks up somewhere here when setting up IRQ for the DE2 HDMI driver:
[   45.232803] [<ffffffc000083dc0>] el1_irq+0x80/0xe4
[   45.241520] [<ffffffc000125844>] __setup_irq+0x318/0x3e0
[   45.250792] [<ffffffc000125a84>] request_threaded_irq+0xe0/0x124
[   45.260858] [<ffffffc00041280c>] disp_sys_register_irq+0x88/0x98
[   45.270936] [<ffffffc000420610>] disp_hdmi_enable+0x1d4/0x278
[   45.280724] [<ffffffc000414540>] disp_device_attached_and_enable+0x1bc/0x1d4
[   45.291985] [<ffffffc0004146f8>] bsp_disp_device_switch+0xbc/0xe4
[   45.302194] [<ffffffc00040b50c>] start_work+0x174/0x1f0
[   45.311445] [<ffffffc0000cb788>] process_one_work+0x27c/0x42c
[   45.321274] [<ffffffc0000cc76c>] worker_thread+0x208/0x320
[   45.330810] [<ffffffc0000d27ec>] kthread+0xb4/0xbc

Part of the stack trace above this must be related to the watchdog that detects the lockup, but in case it doesn't it may be related to the arch timer bug referenced in https://github.com/longsleep/linux-pine64/issues/44
I am using modified ATX power supply for tests connected to the pin header, so underpowering should not be an issue in my setup.

I was able to reproduce a boot-up panic with a specific USB device connected. PR https://github.com/longsleep/linux-pine64/pull/56 seems to fix that. If you can please try if that change also fixes your particular issue.

I'm getting these lockups with no USB devices connected (even got one today with another good power supply when I was testing u-boot changes). While the problem can be power related stack traces look too strange to me,
Also one time I got this log pine64-lockup-debug3.txt - it didn't happen in initrd as usual but much later in the boot process.
Anyway I'll try to test the PR changes later.

Yes - i doubt that the USB change does fix lock-ups which happen later. I will also merge your backport-fsl-errata.patch now after reading up on the issue. But as you probably use a Kernel with that patch already this also does not fix every issue. That FSL fix might resolve https://github.com/longsleep/linux-pine64/issues/44 though.

Yes - i doubt that the USB change does fix lock-ups which happen later.

The stack traces for the "stuck" kworker look too similar in both cases, so it looks like the same issue. And since I enabled a lot of debugging options for spinlocks and mutexes, each time HDMI lock was still held by disp_hdmi_enable() function.
Unfortunately it's still not clear what IRQs correspond to lines like el1_irq+0x84/0xec.

I was able to reproduce a boot-up panic with a specific USB device connected. PR longsleep/linux-pine64#56 seems to fix that. If you can please try if that change also fixes your particular issue.

longsleep/linux-pine64#56 makes USB crash less often but it still crashes a lot on boot with "MOSART Semi. Rapoo 2.4G Wireless Touch Desktop" plugged in. Also the FSL fix does not help.

Btw, on Pinebook with exactly same Kernel - it works just fine every time.

@longsleep
Are you getting lockups with stack traces similar to posted previously with disp2 HDMI functions in them?

@longsleep
Are you getting lockups with stack traces similar to posted previously with disp2 HDMI functions in them?

@zador-blood-stained - Yes, very similar to pine64-lockup-debug3.txt - it has
[   39.838477] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:1:30]
[   39.851912] Modules linked in:
[   39.861726]
[   39.869831] CPU: 0 PID: 30 Comm: kworker/0:1 Not tainted 3.10.105-- #35
[   39.883727] Workqueue: events start_work
[   39.894722] task: ffffffc078b52f80 ti: ffffffc078b54000 task.ti: ffffffc078b54000
[   39.909764] PC is at __do_softirq+0xb4/0x2d8
[   39.921341] LR is at __do_softirq+0x30/0x2d8

and
[   44.313504] [<ffffffc000083dc0>] el1_irq+0x80/0xe4
[   44.323414] [<ffffffc00012584c>] __setup_irq+0x318/0x3e0
[   44.333885] [<ffffffc000125a8c>] request_threaded_irq+0xe0/0x124
[   44.345147] [<ffffffc00040f004>] disp_sys_register_irq+0x88/0x98
[   44.356431] [<ffffffc00041cf9c>] disp_hdmi_enable+0x1d4/0x278
[   44.367423] [<ffffffc000410d38>] disp_device_attached_and_enable+0x1bc/0x1d4
[   44.379876] [<ffffffc000410ef0>] bsp_disp_device_switch+0xbc/0xe4
[   44.391253] [<ffffffc000407d04>] start_work+0x174/0x1f0
[   44.401655] [<ffffffc0000cb784>] process_one_work+0x27c/0x42c
[   44.412623] [<ffffffc0000cc768>] worker_thread+0x208/0x320
[   44.423315] [<ffffffc0000d27f0>] kthread+0xb4/0xbc
[   44.433240] kworker/1:1     S ffffffc0000853b8     0

and
   45.225365] [<ffffffc0000853b8>] __switch_to+0x7c/0x88                           [445/9673]
[   45.235455] [<ffffffc0007244f4>] __schedule+0x4fc/0x714
[   45.245628] [<ffffffc000724780>] schedule+0x74/0x7c
[   45.255409] [<ffffffc000722564>] schedule_timeout+0x34/0x27c
[   45.266012] [<ffffffc000723cbc>] wait_for_common+0x118/0x158
[   45.276588] [<ffffffc000723d24>] wait_for_completion+0x28/0x34
[   45.287325] [<ffffffc0000cb108>] flush_work+0xf8/0x11c
[   45.297312] [<ffffffc0000cccd4>] schedule_on_each_cpu+0xf8/0x124
[   45.308281] [<ffffffc00016c5f0>] lru_add_drain_all+0x1c/0x24
[   45.318875] [<ffffffc0001a4d54>] migrate_prep+0x14/0x20
[   45.328979] [<ffffffc000167d78>] alloc_contig_range+0xb8/0x26c
[   45.339729] [<ffffffc000493884>] dma_alloc_from_contiguous+0xa4/0x12c
[   45.351152] [<ffffffc0000928cc>] __dma_alloc_coherent+0xb0/0x118
[   45.362088] [<ffffffc000092a00>] __dma_alloc_noncoherent+0xcc/0x158
[   45.373319] [<ffffffc00019979c>] dma_pool_alloc+0xf0/0x1c4
[   45.383705] [<ffffffc0004ef388>] ehci_qh_alloc+0x4c/0xc4
[   45.393894] [<ffffffc0004f1408>] ehci_init+0x13c/0x3b8
[   45.403875] [<ffffffc0004f16a4>] sunxi_ehci_setup+0x20/0x38
[   45.414303] [<ffffffc0004de7a8>] usb_add_hcd+0x1c8/0x5a8
[   45.424417] [<ffffffc0004f5560>] sunxi_insmod_ehci+0x118/0x218
[   45.435096] [<ffffffc0004f56d8>] sunxi_usb_enable_ehci+0x78/0x88
[   45.445982] [<ffffffc00051144c>] usb_msg_center+0x88/0x104
[   45.456307] [<ffffffc00051057c>] usb_host_scan_thread+0x54/0x68
[   45.467110] [<ffffffc0000d27f0>] kthread+0xb4/0xbc

and
[   47.357995] [<ffffffc0000853b8>] __switch_to+0x7c/0x88
[   47.368085] [<ffffffc0007244f4>] __schedule+0x4fc/0x714
[   47.378228] [<ffffffc000724780>] schedule+0x74/0x7c
[   47.387959] [<ffffffc000722564>] schedule_timeout+0x34/0x27c
[   47.398562] [<ffffffc000723cbc>] wait_for_common+0x118/0x158
[   47.409169] [<ffffffc000723d24>] wait_for_completion+0x28/0x34
[   47.419962] [<ffffffc0000cb108>] flush_work+0xf8/0x11c
[   47.429992] [<ffffffc0000cccd4>] schedule_on_each_cpu+0xf8/0x124
[   47.440953] [<ffffffc00016c5f0>] lru_add_drain_all+0x1c/0x24
[   47.451515] [<ffffffc0001e5b24>] invalidate_bdev+0x30/0x4c
[   47.461872] [<ffffffc0002453b4>] ext4_put_super+0x264/0x2ec
[   47.472336] [<ffffffc0001b24d8>] generic_shutdown_super+0x68/0xd4
[   47.483396] [<ffffffc0001b27c0>] kill_block_super+0x30/0x7c
[   47.493872] [<ffffffc0001b2b44>] deactivate_locked_super+0x44/0x74
[   47.505016] [<ffffffc0001b2fb4>] deactivate_super+0x68/0x74
[   47.515443] [<ffffffc0001cdbd0>] mntput_no_expire+0x158/0x168
[   47.526039] [<ffffffc0001cef48>] SyS_umount+0x34c/0x36c

I have a rather relyable setup to reproduce this. With the new USB drivers it is less likely to trigger. I boot to initrd only (have simpleimage without rootfs). It just booted 4 times in a row without issue and then crashed twice in a row like this.
I am powering through euler and have HDMI connected (but that does not seem to matter). When i disconnect the USB Keyboard/Mouse dongle it never crashes. Also i can connect the dongler at any time later and it also does not crash.

I tested this in detail yesterday. It still can crash exactly like with even when powered at 5.2V via Euler. It never draws more than 400mA during bootup either.

I did some more tests and compiled the kernel with debug info. Looks like it's actually stuck in a softirq, but it's relatively hard to debug since the stack trace is be incomplete in this case and I'm not sure if the info I got after applying an extra patch is correct
[   42.584359] Last softirq was rcu_process_callbacks+0x0/0x3f8

P.S. it seems that this behavior also occured on my SoPine w/ Baseboard, running mainline kernel w/ HDMI driver patched. Strange.

I am experiencing a HDMI bug too - if a HDMI cable is plugged in to the HDMI port, the A64 boots fine after a power cycle. If there is no HDMI cable, it may or may not boot.
There is nothing connected at the other end of the HDMI cable.
Workaround: Keep a HDMI cable plugged in.

I am experiencing a HDMI bug too - if a HDMI cable is plugged in to the HDMI port, the A64 boots fine after a power cycle. If there is no HDMI cable, it may or may not boot.
There is nothing connected at the other end of the HDMI cable. I am running Xenial with Longsleep kernel.
Workaround: Keep a HDMI cable plugged in.

Most likely the HDMI cable feeds enough extra power to the device that the voltage does not drop on load. Means your power supply solution is to blame and not sufficient.

Unlikely, since there's nothing plugged in at the other end of the HDMI cable.
The power supply is the model recommeded in the Pine64 store at the time of purchase.

I don't think this is a power supply issue -- I see this happening on two of my boards (bought from separate lots) with about a 30% successful boot rate sometimes.  Both boards exhibit this behavior while running off a bench supply powered through the Euler bus at as high as 6 volts (I've not risked going any higher).  The crashes happen on all the images I've tried though the behavior is different on each one.  Sometimes I can get things to boot more reliably on an image and it will stay about 80% reliable once it boots successfully a few times.  I can post output from the serial console if there is any interest.

Well this still is an issue - so feel free to post your findings here in case someone is willing to take a detailed look. If it is HDMI related it might be an idea to get rid of this driver and all related to it.

Let me try some experiments and see what I come up with.  Is there a way to turn off the HDMI driver completely?   The most reliable boot image has been Android, but I've been using debian and xubuntu since I want to run a headless server with these units.  I have successfully upgraded one unit to bionic beaver (haven't tried with the other one) but the /boot partition has to be enlarged for the do-release-upgrade to work (I can open another issue to cover that if you like).  The bionic beaver image also exhibits this behavior.


QOL fixes + recursion restructure

New features:

Avoid lowsec option
User-facing notification of crest connectivity issues

Other changes:

The main recursive loop that generates routes has been restructured to know more about where it currently is in the route generation process. This allows a degree of introspection that lets us tell if we're putting lots of the same solar system in the route, and avoid it. As a result, routes now take noticeably longer to generate but are much more squiggly and go into more nooks and crannies in the map. I've got the niggling feeling that the logic isn't quite up to scratch, behavior needs more testing and review.

That's really neat!  The idea seems to be to daisy chain routes together to generate a longer one.

I think I've figured out the logic error actually, probs submitted this too soon. I think line 43 in RouteGenerator.java is causing it to add all eligible systems to the path, which makes it starburst at each actual node on the system.
It would be ok behavior I guess, if it didn't cause you to end up with way more jumps than you asked for. (sidenote, with this structure "jumps" should probably be "systems").


How to use filter function on this inventory plugin?

How would i go about to filter inventory on data that Netbox presents?
I can see "name", "platform", "primary_ip" in your code but i was looking for "role" as in first version

Going to assume you are talking about the NetBoxInventory2 and that you mean Nornir inventory filters. Since those details are missing from your report.
NetBoxInventory2 stores the complete JSON reponse from the NetBox devices API endpoint into the data attribute of a Nornir host.
"role" doesn't exist in the data model of NetBox, NetBox's data model refers to this as a "device_role". You can find an example here https://github.com/wvandeun/nornir_netbox/blob/070c09282f341ae43907e068544668699310998e/tests/NetBoxInventory2/2.8.9/expected.json#L20
To filter for that particular device_role you would need to use an F object filter, since we want to filter on data that is nested in the device_role https://nornir.readthedocs.io/en/latest/tutorial/inventory.html#Advanced-filtering
You can filter the inventory for the device_role with name "Router" like this:
nr.inventory.filter(F(device_role__name="Router"))

or slug rt:
nr.inventory.filter(F(device_role__slug="rt"))

thank you so much for this clarification. i'm new to all this and I tried some different approaches with F object filter but always got an empty dict. Now it works great.
From what i understand about "Role" is that you can call it from config,yml under "filter_parameters" of your plugin. This is what i used prior to this.
"filter_parameters": {
              "role": "distribution",
 },

Yes, you can still do that through the Nornir configuration file or when you instantiate a Nornir object with InitNornir()

Somewhat related to this issue, in Netbox >2.9 the tags on devices are now stored as an array of dictionaries, instead of a simple array of strings, do you have any examples how to use F object to filter on tags such as:
nrRun = nr.filter(F(device_type__manufacturer__name="Cisco") & F(tags__slug__contains="switch-stack")
The line above does not work with 2.9 version of NB, since tags is an array of dictionaries
example data:
nrRun.inventory.hosts['zzzzzzzz'].data.get('tags') [{'color': '9e9e9e', 'id': 5, 'name': 'switch-stack', 'slug': 'switch-stack', 'url': 'https://xxxxxx/tags/5/'}]
Probably missing something simple, any help would be greatly appreciated.

I have an ugly workaround that uses existing transform function to flatten the tags back to a simple list so that my filter object can continue to work, but I was hoping to figure out a cleaner way in the future:
Nornir 3.0 changed the transform as well to make it much simpler (I think)
https://github.com/twin-bridges/nornir_course/blob/master/nornir3_changes.md#transform-function-is-now-an-entry-point
for host in nr.inventory.hosts.values():
    adapt_host_data(host)

def adapt_host_data(host):

    tag_list = host.data.get('tags')
    tags=[]
    for tag in tag_list:
        tags.append(tag.get('slug'))

    host.data['tags']=tags

Not sure if F filters are still in a option in that case. It should be possible if you know the exact contents of the tag dictionary you are looking for, which doesn't seem to be to practical.
nr.filter(F(tags__contains={'color': '9e9e9e', 'id': 5, 'name': 'switch-stack', 'slug': 'switch-stack', 'url': 'https://xxxxxx/tags/5/'}))

Alternative approaches:

use a filter function

def my_filter(host: Host):
    value_to_search_for = "something"
    return value_to_search_for in[ tag.name for tag in host.get("tags")]

nr.filter(filter_func=my_filter)

use filter_parameters

Not sure if F filters are still an option in that case. It should be possible if you know the exact contents of the tag dictionary you are looking for, which doesn't seem to be to practical.
nr.filter(F(tags__contains={'color': '9e9e9e', 'id': 5, 'name': 'switch-stack', 'slug': 'switch-stack', 'url': 'https://xxxxxx/tags/5/'}))

But I could be missing some functionality that I'm not aware off.

This method would work, but way too fragile as tag colors can change, for example, and it would break the code.

Alternative approaches:
* use a filter function

def my_filter(host: Host):
    value_to_search_for = "something"
    return value_to_search_for in[ tag.name for tag in host.get("tags")]

nr.filter(filter_func=my_filter)

This is probably a much cleaner way to handle this, I will probably adopt this, however for compatibility reason in my scripts, I may stick to the transform function library to flatten the tags attribute for now.  Just wondering if we should look at potentially adding an option to flatten it as part of the inventory plugin.  If you are open I can look into creating a PR.

* use filter_parameters

We already use some of that, but that is also a great suggestion.

Just wondering if we should look at potentially adding an option to flatten it as part of the inventory plugin. If you are open I can look into creating a PR.

Not convinced that we need to offer an option to flatten this. The whole point of NetBoxInventory2 is to have more relevant data available in the data attribute of a Nornir host. There is a lot of benefit in keeping the data structure as close to the original data structure returned by the NetBox API.

Just wondering if we should look at potentially adding an option to flatten it as part of the inventory plugin. If you are open I can look into creating a PR.

Not convinced that we need to offer an option to flatten this. The whole point of NetBoxInventory2 is to have more relevant data available in the data attribute of a Nornir host. There is a lot of benefit in keeping the data structure as close to the original data structure returned by the NetBox API.

After thinking about it more, I tend to agree with your statement and using filter_parameters should be a much cleaner way to filter out the results.


Ingress gateway caches connection to backend in ambient mode when creating/deleting waypoint proxy

Bug Description
following the ingress gateway docs steps from here, after verifying traffic routes to httpbin through ingress gateway,
$ curl -i -HHost:httpbin.example.com "http://$INGRESS_HOST:$INGRESS_PORT/status/200"
HTTP/1.1 200 OK
server: istio-envoy
date: Wed, 18 Jan 2023 20:20:55 GMT
content-type: text/html; charset=utf-8
access-control-allow-origin: *
access-control-allow-credentials: true
content-length: 0
x-envoy-upstream-service-time: 19

if you deploy a waypoint proxy for the httpbin service:
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: gateway
  annotations:
    istio.io/service-account: httpbin
spec:
  gatewayClassName: istio-mesh
EOF

the connection at the ingress gateway appears to be cached, and it still tries to call httpbin directly, instead of routing through waypoint:
$ curl -i -HHost:httpbin.example.com "http://$INGRESS_HOST:$INGRESS_PORT/status/200"
HTTP/1.1 503 Service Unavailable
content-length: 95
content-type: text/plain
date: Wed, 18 Jan 2023 20:20:21 GMT
server: istio-envoy

Request succeeds if you restart the ingress gateway.  Similarly, if you delete the waypoint proxy, requests start failing again
Version
experimental-ambient branch

Additional Information
No response

this happens when there is only one httpbin service instance.  If you try to re-create with the cross node helloworld example from here, and modify the istio GW+VS for hello.example.com, requests are routed as expected

We have a DR with maxRequestsPerConnection=1 to work around an issue - not sure its this one, though. For integ tests

Just to clarify - this is a gross hack to make the test work well, we definitely should not recommend this and should fix it. Just stating it may (or may not) be the same issue as that one

Just to clarify - this is a gross hack to make the test work well, we definitely should not recommend this and should fix it. Just stating it may (or may not) be the same issue as that one

Ok, we will wait for @GregHanson to confirm if the workaround fixes it. It does sound it would.   Do we have any plan to fix this? hot restart?

creating a DR such as :
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: gihanson
spec:
  host: "*.local"
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
EOF

only fixes the problem once, I'm assuming because it pushes a CDS update for the cluster in question.  But as soon as the waypoint is created or deleted once more for the sleep and httpbin examples where I have reproduced, the same error occurs on the next request

re: draining - this might be the setting:close_connections_on_host_set_change, from @peterj

I don't think it is related to EDS info on the ingress gateway being out of date.  I have the httpbin and helloworld examples deployed.  helloworld has a waypoint and IS working
$ istioctl pc endpoint deploy/istio-ingressgateway -n istio-system --cluster "outbound|5000||helloworld.default.svc.cluster.local"
ENDPOINT                                    STATUS      OUTLIER CHECK     CLUSTER
envoy://outbound-tunnel/<IP_ADDRESS>:5000     HEALTHY     OK                outbound|5000||helloworld.default.svc.cluster.local
envoy://outbound-tunnel/<IP_ADDRESS>:5000     HEALTHY     OK                outbound|5000||helloworld.default.svc.cluster.local

and checking the workload info for the ztunnels, the helloworld waypoint address is <IP_ADDRESS> but does not appear in the output above:
$ istioctl pc workload ds/ztunnel -n istio-system --address <IP_ADDRESS> -o json
[
    {
        "workloadIp": "<IP_ADDRESS>",
        "waypointAddresses": [
            "<IP_ADDRESS>"
        ],
        "gatewayIp": null,
        "protocol": "HBONE",
        "name": "helloworld-v1-cross-node-55446d46d8-69d2j",
...
...

I and can see access logs getting generated in the waypoint proxy on every request
httpbin is deployed with a waypoint proxy which is currently NOT working, despite showing similar config
$ istioctl pc endpoint deploy/istio-ingressgateway -n istio-system --cluster "outbound|8000||httpbin.default.svc.cluster.local"
ENDPOINT                                  STATUS      OUTLIER CHECK     CLUSTER
envoy://outbound-tunnel/<IP_ADDRESS>:80     HEALTHY     OK                outbound|8000||httpbin.default.svc.cluster.local

and the workload info showing the httpbin waypoint address as <IP_ADDRESS>
$ istioctl pc workload ds/ztunnel -n istio-system --address <IP_ADDRESS> -o json
[
    {
        "workloadIp": "<IP_ADDRESS>",
        "waypointAddresses": [
            "<IP_ADDRESS>"
        ],
        "gatewayIp": null,
        "protocol": "HBONE",
        "name": "httpbin-847f64cc8d-6qq5d",

no logs get generated on the httpbin waypoint proxy during failed requests

I am confused why the ztunnel config is relevant here?  The hop that has issue is from ingress gateway to waypoint, which doesn't go through ztunnel to my knowledge.  Does ingress gateway knows the next hop is waypoint to get to the httpbin?

and checking the workload info for the ztunnels, the helloworld waypoint address is <IP_ADDRESS> but does not appear in the output above:
$ istioctl pc workload ds/ztunnel -n istio-system --address <IP_ADDRESS> -o json
[
    {
        "workloadIp": "<IP_ADDRESS>",
        "waypointAddresses": [
            "<IP_ADDRESS>"
        ],
        "gatewayIp": null,
        "protocol": "HBONE",
        "name": "helloworld-v1-cross-node-55446d46d8-69d2j",
...
...

I and can see access logs getting generated in the waypoint proxy on every request
httpbin is deployed with a waypoint proxy which is currently NOT working, despite showing similar config
$ istioctl pc endpoint deploy/istio-ingressgateway -n istio-system --cluster "outbound|8000||httpbin.default.svc.cluster.local"
ENDPOINT                                  STATUS      OUTLIER CHECK     CLUSTER
envoy://outbound-tunnel/<IP_ADDRESS>:80     HEALTHY     OK                outbound|8000||httpbin.default.svc.cluster.local

and the workload info showing the httpbin waypoint address as <IP_ADDRESS>
$ istioctl pc workload ds/ztunnel -n istio-system --address <IP_ADDRESS> -o json
[
    {
        "workloadIp": "<IP_ADDRESS>",
        "waypointAddresses": [
            "<IP_ADDRESS>"
        ],
        "gatewayIp": null,
        "protocol": "HBONE",
        "name": "httpbin-847f64cc8d-6qq5d",

no logs get generated on the httpbin waypoint proxy during failed requests

@linsun the workload command from ztunnel is just sanity checking that waypoint proxy with IP address XX.XX.XX.XX is known/associated with a particular workload.

The hop that has issue is from ingress gateway to waypoint, which doesn't go through ztunnel to my knowledge. Does ingress gateway knows the next hop is waypoint to get to the httpbin?

I could use some clarification on this one as well.  ingress gateway and httpbin are on the same node, so I would NOT expect there to be any traffic through the ztunnel in this case (verified by no logs getting generated).  However, in the helloworld example, with an instance on each node, ztunnel will and does generate a log for the requests that occur across nodes.  If I similarly break the helloworld example, calling it through ingress gateway before deploying its waypoint, bad access logs get generated in the corresponding ztunnel
working:
2023-01-20T16:31:33.125137Z  INFO inbound{id=4e0da5bbd358758f91f4baf8dd4b68d8 peer_ip=<IP_ADDRESS> peer_id=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account}: ztunnel::proxy::inbound: got CONNECT request to <IP_ADDRESS>:5000
2023-01-20T16:31:33.261485Z  INFO inbound{id=4e0da5bbd358758f91f4baf8dd4b68d8 peer_ip=<IP_ADDRESS> peer_id=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account}: ztunnel::proxy::inbound: complete dur=136.043718ms

broken
2023-01-20T16:32:17.117267Z  INFO inbound{id=87850dafb9200d565e4033897f208fba peer_ip=<IP_ADDRESS> peer_id=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account}: ztunnel::proxy::inbound: got CONNECT request to <IP_ADDRESS>:5000
2023-01-20T16:32:17.117413Z  INFO inbound{id=87850dafb9200d565e4033897f208fba peer_ip=<IP_ADDRESS> peer_id=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account}: ztunnel::proxy::inbound: bypassed waypoint conn=<IP_ADDRESS>(spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account)-><IP_ADDRESS>:5000

@GregHanson the prob with checking ztunnel config is that it won't impact ingress gateway's config. they are using different xDS APIs.
So the hop will go through ztunnel for sure but my understanding is what is broken is the hop from ingress gateway to waypoint.
Basically, the usual flow is source -> source ztunnel -> dest waypoint (if exists) -> destination ztunnel -> destination.
In the ingress gateway scenario, the flow is ingress gateway -> dest waypoint (if exists) -> destination ztunnel -> destination.  I don't think ingress gateway goes through ztunnel like a normal source.  In this case, it would be important to check the Envoy config of the ingress gateway - does it aware of the dest waypoint as an endpoint to connect next when traffic goes to the destination service (httpbin in your example)?
Happy to be corrected!

ingress gateway scenario, the flow is ingress gateway -> dest waypoint (if exists) -> destination ztunnel -> destination

but in the broken helloworld example, I see the same broken ztunnel log in the source and destination node's ztunnel

I don't think this is correct. Even if it is on the same node, traffic needs to go through httpbin's ztunnel to do rbac check at the minimal.

I could use some clarification on this one as well. ingress gateway and httpbin are on the same node, so I would NOT expect there to be any traffic through the ztunnel in this case (verified by no logs getting generated).

When hairpinning is implemented (https://github.com/istio/ztunnel/issues/73), I think dest ztunnel will attempt to forward the traffic to dest waypoint if it exists. However, I do think even without hairpinning, it would be best to send to waypoint in the first place if we could program ingress gateway to do so. @howardjohn pls correct me if i am wrong.

Ingress should go direct to waypoint, yes. Not sure that is 100% working properly though, I haven't looked into the Envoy HBONE impl for a bit. btw, hairpin is merged in ztunnel but there is a small bug I had to fix before can import it into istio/istio. Should be done tomorrow though

to clarify some of my statements above:

when ingress gateway on node 2 calls hello-v2 on node2, the request should (and does) get routed through ztunnel on node 2 and then the waypoint
when ingress gateway on node 2 calls hello-v1 on node1,  the request should (and does) get routed through ztunnel on node 1 and then the waypoint

I thought the comments above suggested traffic from ingress gateway to an endpoint on the same node should NOT be directed through the co-located ztunnel.  But requests from ingress gateway should always go through a ztunnel regardless in order to apply any L4 policies - so that part is behaving as expected

The issue is the waypoint IP Is stored as metadata in the endpoint config for the httpbin endpoint.  The waypoint and its cluster and endpoint are correctly added/deleted when the waypoint is added and deleted - but the metadata of the httpbin endpoint is never updated
working httpbin with waypoint:
$ istioctl pc endpoint deploy/istio-ingressgateway -n istio-system
ENDPOINT                                                STATUS      OUTLIER CHECK     CLUSTER
...
<IP_ADDRESS>:15008 ## IP of waypoint                      HEALTHY     OK                outbound-tunnel
...
envoy://outbound-tunnel/<IP_ADDRESS>:80 ##IP of httpbin   HEALTHY     OK                outbound|8000||httpbin.default.svc.cluster.local

looking at envoy config_dump for ingress gateway
   {
     "endpoint_config": {
      "@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
      "cluster_name": "outbound-tunnel",
      "endpoints": [
       {
        "locality": {},
        "lb_endpoints": [
         {
          "endpoint": {
           "address": {
            "socket_address": {
             "address": "<IP_ADDRESS>",  ## IP of waypoint
             "port_value": 15008
            }
           },
           "health_check_config": {},
           "hostname": "outbound-tunnel<IP_ADDRESS>:15008"
          },
          "health_status": "HEALTHY",
          "load_balancing_weight": 1
         }
 ...
...
   {
     "endpoint_config": {
      "@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
      "cluster_name": "outbound|8000||httpbin.default.svc.cluster.local",
      "endpoints": [
       {
        "locality": {},
        "lb_endpoints": [
         {
          "endpoint": {
           "address": {
            "envoy_internal_address": {
             "server_listener_name": "outbound-tunnel",
             "endpoint_id": "<IP_ADDRESS>:80". ## IP of httpbin
            }
           },
           "health_check_config": {}
          },
          "health_status": "HEALTHY",
          "metadata": {
           "filter_metadata": {
            "envoy.transport_socket_match": {
             "tunnel": "http"
            },
            "tunnel": {
             "destination": "<IP_ADDRESS>:80",  ## IP of httpbin
             "address": "<IP_ADDRESS>:15008" ## IP of waypoint
            },
            "istio": {
             "workload": "httpbin;default;httpbin;v1;Kubernetes"
            }
           }
          },

deleting waypoint, breaking traffic from ingress gateway to httpbin
$ istioctl pc endpoint deploy/istio-ingressgateway -n istio-system
ENDPOINT                                                STATUS      OUTLIER CHECK     CLUSTER
...
envoy://outbound-tunnel/<IP_ADDRESS>:80 ##IP of httpbin   HEALTHY     OK                outbound|8000||httpbin.default.svc.cluster.local

and the config_dump:
    {
     "endpoint_config": {
      "@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
      "cluster_name": "outbound|8000||httpbin.default.svc.cluster.local",
      "endpoints": [
       {
        "locality": {},
        "lb_endpoints": [
         {
          "endpoint": {
           "address": {
            "envoy_internal_address": {
             "server_listener_name": "outbound-tunnel",
             "endpoint_id": "<IP_ADDRESS>:80"
            }
           },
           "health_check_config": {}
          },
          "health_status": "HEALTHY",
          "metadata": {
           "filter_metadata": {
            "tunnel": {
             "address": "<IP_ADDRESS>:15008",  ## still IP of waypoint
             "destination": "<IP_ADDRESS>:80"
            },
            "istio": {
             "workload": "httpbin;default;httpbin;v1;Kubernetes"
            },
            "envoy.transport_socket_match": {
             "tunnel": "http"
            }
           }

and finally in the case of no waypoint is deployed and traffic from ingress to httpbin IS working, this is what the endpoint config from the config_dump looks like for httpbin:
    {
     "endpoint_config": {
      "@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
      "cluster_name": "outbound|8000||httpbin.default.svc.cluster.local",
      "endpoints": [
       {
        "locality": {},
        "lb_endpoints": [
         {
          "endpoint": {
           "address": {
            "envoy_internal_address": {
             "server_listener_name": "outbound-tunnel",
             "endpoint_id": "<IP_ADDRESS>:80"
            }
           },
           "health_check_config": {}
          },
          "health_status": "HEALTHY",
          "metadata": {
           "filter_metadata": {
            "istio": {
             "workload": "httpbin;default;httpbin;v1;Kubernetes"
            },
            "tunnel": {
             "address": "<IP_ADDRESS>:15008", ## IP of httpbin
             "destination": "<IP_ADDRESS>:80". ## IP of httpbin
            },
            "envoy.transport_socket_match": {
             "tunnel": "http"
            }
           }
          },

once the ingress -> httpbin traffic is broken, the only surefire way to fix env is to unlabel and re-label the namespace for ambient

I tried the maxConnection=1 DestinationRule and an EnvoyFilter for setting close_connections_on_host_set_change on the cluster, but neither of those resolve the problem
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: gihanson
spec:
  host: "*.local"
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
EOF

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gihanson
  namespace: istio-system
spec:
  configPatches:
  - applyTo: CLUSTER
    match:
      context: ANY
    patch:
      operation: MERGE
      value:
        common_lb_config:
          close_connections_on_host_set_change: true
EOF


Publish on PyPI

This can also be automated during handling of #2

Release 0.1.3 has been released on PyPi, next release/tag will be published by Travis CI


Config changes to reflect the default branch from master to main. Please merge to complete the default branch renaming job.

Auto branch renaming content.

Docs Build status updates of commit 5473b24:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.publish.config.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


"nullable: True" ignored if used together with allOf or oneOf

I have the following issue. drf-spectacular generates the following spec for enums (for example, for a building_type field which can be either "house" or "apartment" or null):
    WhateverObject:
      type: object
      properties:
        building_type:
          nullable: true
          oneOf:
          - $ref: '#/components/schemas/BuildingTypeEnum'
          - $ref: '#/components/schemas/NullEnum'

    BuildingTypeEnum:
      enum:
      - house
      - apartment
      type: string

    NullEnum:
      enum:
      - null

but when trying to validate data containing None for the building_type field I get this error:
Failed validating 'oneOf' in schema['properties']['building_type']:
    {'nullable': True,
     'oneOf': [{'$ref': '#/components/schemas/BuildingTypeEnum',
                'nullable': False,
                'x-scope': ['', '#/components/schemas/HomeProfile']},
               {'$ref': '#/components/schemas/NullEnum',
                'nullable': False,
                'x-scope': ['', '#/components/schemas/HomeProfile']}]}

On instance['building_type']:
    None
None is not valid under any of the given schemas

I think that None should be allowed if nullable: True is specified.

I ran into this with anyOf in my project. Is there a workaround for this?

I ran into this with anyOf in my project. Is there a workaround for this?

@mmerickel I haven't found a workaround yet. For now I'm just evaluating whether to use this library so I haven't worked on these issues yet. Are you using drf-spectacular to generate your schema too?

@mmerickel I haven't found a workaround yet. For now I'm just evaluating whether to use this library so I haven't worked on these issues yet. Are you using drf-spectacular to generate your schema too?

@idesoto I am not, we hand-create the yaml right now.

@idesoto I am not, we hand-create the yaml right now.

Seems like its an issue with "openapi_schema_validator" lib.
https://github.com/p1c2u/openapi-schema-validator/blob/master/openapi_schema_validator/validators.py#L69
May be it should check whether nullable property is present with anyOf, oneOf before adding nullable: false

It is still an issue with this library.
The current workaround is
field:
  nullable: true
  oneOf:
    - type: object
      nullable: true
    - $ref: '#/components/schemas/Field'

It should not be required to define a nullable object in the oneOf

It still does not work with this library. (openapi 3.0 validator)
The only workable solution at this moment is
field:
  nullable: true
  oneOf:
    - type: object
      nullable: true
    - '$ref': '#/components/schemas/Field'

The expected schema is
field:
  nullable: true
  allOf:
  - $ref: '#/components/schemas/Field'


feat: VSCode extension

Validation based on the models present in the schema.prisma is implemented.
Issue: #7

https://github.com/SamagraX-Stencil/stencil/assets/84929607/af468031-b56b-4b46-bff1-e7deab80235a

Thanks for the video @Apurv428 this is super work! Thanks for the PR.


SetSpecialMapping in ratbagd should accept an enum instead of a string

I suppose this one was simply glanced over, but this function too should take an enum ratbag_button_action_special instead of a string. Right now Piper will crash over this (in https://github.com/libratbag/piper/pull/47), because it already assumes the enum while the bindings and ratbagd still want the string.

Yeah, this simply has not been implemented yet. See https://213-41709447-gh.circle-artifacts.com/0/root/libratbag/build/meson-logs/testlog.txt -> the last skipped test is related to set action special :)

I'll fix this