Include the location of the mistake in case of a "Could not resolve X" error

Is your feature request related to a problem? Please describe.
I'm trying to generate documentation for my project with mkdocstrings, and griffe constantly crashes with weird "Could not resolve X" errors that are impossible to track down and fix.
Describe the solution you'd like
The error message should include the exact location of the symbol that couldn't be resolved.
Additional context
This is the error I get running mkdocs serve on my project:
ERROR    -  Error reading page 'api_reference.md': Could not resolve typing._no_init_or_replace_init
Traceback (most recent call last):
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\griffe\dataclasses.py", line 1111, in _resolve_target
    resolved = self.modules_collection[self.target_path]
               ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\griffe\mixins.py", line 28, in __getitem__
    return self.members[parts[0]][parts[1:]]  # type: ignore[attr-defined]
           ~~~~~~~~~~~~^^^^^^^^^^
KeyError: 'typing'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocs\__main__.py", line 301, in <module>
    cli()
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocs\__main__.py", line 234, in serve_command
    serve.serve(dev_addr=dev_addr, livereload=livereload, watch=watch, **kwargs)
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocs\commands\serve.py", line 83, in serve
    builder(config)
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocs\commands\serve.py", line 76, in builder
    build(config, live_server=live_server, dirty=dirty)
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocs\commands\build.py", line 308, in build
    _populate_page(file.page, config, files, dirty)
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocs\commands\build.py", line 181, in _populate_page
    page.render(config, files)
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocs\structure\pages.py", line 270, in render
    self.content = md.convert(self.markdown)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\markdown\core.py", line 264, in convert
    root = self.parser.parseDocument(self.lines).getroot()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\markdown\blockparser.py", line 90, in parseDocument
    self.parseChunk(self.root, '\n'.join(lines))
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\markdown\blockparser.py", line 105, in parseChunk
    self.parseBlocks(parent, text.split('\n\n'))
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\markdown\blockparser.py", line 123, in parseBlocks
    if processor.run(parent, blocks) is not False:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings\extension.py", line 116, in run
    html, handler, data = self._process_block(identifier, block, heading_level)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings\extension.py", line 202, in _process_block
    rendered = handler.render(data, options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\handler.py", line 255, in render
    return template.render(
           ^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\jinja2\environment.py", line 1301, in render
    self.environment.handle_exception()
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\jinja2\environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\templates\material\module.html", line 1, in top-level template code
    {% extends "_base/module.html" %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\templates\material\_base\module.html", line 58, in top-level template code
    {% include "children.html" with context %}
^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\templates\material\children.html", line 1, in top-level template code
    {% extends "_base/children.html" %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\templates\material\_base\children.html", line 45, in top-level template code
    {% include "class.html" with context %}
^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\templates\material\class.html", line 1, in top-level template code
    {% extends "_base/class.html" %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\templates\material\_base\class.html", line 30, in top-level template code
    {%- include "signature.html" with context -%}
^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\templates\material\signature.html", line 1, in top-level template code
    {% extends "_base/signature.html" %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\mkdocstrings_handlers\python\templates\material\_base\signature.html", line 12, in top-level template code
    {%- for parameter in function.parameters -%}
^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\jinja2\environment.py", line 485, in getattr
    return getattr(obj, attribute)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\griffe\dataclasses.py", line 1035, in parameters
    return cast(Function, self.target).parameters
                          ^^^^^^^^^^^
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\griffe\dataclasses.py", line 1078, in target
    self.resolve_target()
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\griffe\dataclasses.py", line 1105, in resolve_target
    self._resolve_target()
  File "C:\Users\Aran-Fey\AppData\Roaming\Python\Python311\site-packages\griffe\dataclasses.py", line 1113, in _resolve_target
    raise AliasResolutionError(self.target_path) from error
griffe.exceptions.AliasResolutionError: Could not resolve typing._no_init_or_replace_init

My project doesn't contain typing._no_init_or_replace_init anywhere. I have no idea what's causing this error or how to fix it. I would like to see a more helpful error message, something along the lines of

Could not resolve "typing._no_init_or_replace_init" in class "Foo" from "my_project/foo.py"

Hello, thanks for the report! And this is a good suggestion. Do you have a public repository I could use to replicate and fix this particular issue?

Unfortunately this particular crash is pretty tricky to reproduce. I still haven't quite figured it out yet, but this one in particular is probably my own fault (I'm doing something unusual with my PYTHONPATH and other things). I'll probably open another issue related to that, because there's something weird about how griffe discovers modules, and it's making it very difficult for me to debug this.

Ok, I give up. I'm sorry, but I can't for the life of me isolate this bug. I have a demo, which you can download here, but it's far from minimal - I've had to include no less than 4 of my projects. (Instructions for reproducing the issue are included in the zip file.)
I've discovered that the issue is indeed related to my crazy setup - I have an __init__.py file in the project directory. Deleting that file makes the error go away, but in exchange the generated API documentation is completely blank. So I figured it's still worth reporting. Something very weird is going on there.

Thanks, I'll have a look at it.

Yep, I confirm the behavior you describe. I'm able to replicate the issue in my shell, but not in VSCode, so I can't debug :rofl:

About the docs being blank: you need to enable showing submodules:
::: animation_editor
    options:
        show_submodules: true

Oh, huh. Ok, rookie mistake on my part. But I swear I have seen documentation being generated today even without that setting enabled. Nothing makes sense to me anymore...

It's more clear after a break usually :wink:
I've pushed a commit that should improve the messages of alias resolution errors. You can try it by installing from master branch. Otherwise I'll publish a new release soon.

What about the griffe.exceptions.AliasResolutionError: Could not resolve typing._no_init_or_replace_init error caused by the presence of an __init__.py file in the project root directory? Should I open a separate issue for that, or is that already taken care of?

Reading the traceback, it seems that the __init__ method in some class of yours is actually an alias to typing._no_init_or_replace_init. This is the first time I encounter such a scenario. Maybe we could add a condition in the template to skip the __init__ method if it's an alias to typing._no_init_or_replace_init (or just an alias).

The weird thing is that the presence of typing._no_init_or_replace_init alone doesn't make it crash. For some reason, renaming or deleting the project-level __init__.py file makes the crash go away. I've observed this behavior in multiple projects of mine, but I can't reproduce it in a minimal demo project...

Did you try to isolate the issue by starting from your complete setup and removing small parts one by one? (trying to help the best I can :sweat_smile: )

Hi @Aran-Fey, do you still have this issue with latest Griffe version?

I honestly can't say. It's been so long, I can't even reproduce the error anymore.
But anyway, feel free to close this issue. The thing I originally made the bug report/feature request for has been taken care of, after all.
I'll probably run into problems with mkdocs/mkdocstrings/griffe again in the future, but I'll just open a new issue then.

Alright, thanks!


layer number bug fixes

Fixes #294, UnfusedDotProductAttention for fp16, and KV inference cache.

/te-ci

@timmoon10 in the second scenario, layer_number is used as a scaling technique for better convergence with fp16 training.

Can you give some intuition or documentation? All layers should have QKV values around the same range due to layernorm, so the QK GEMM should also have a consistent range and not require scaling. Perhaps in bprop there is some growth or decay in the gradients, but why do we scale by layer_number instead of 1/layer_number or exp(+/- layer_number)?


Makefile uses "uname -p" instead of "uname -m"

ifeq ($(origin GOARCH), undefined)
		UNAME_P := $(shell uname -p)
		ifeq ($(UNAME_P),x86_64)
			export GOARCH = amd64
		endif
		ifneq ($(filter %86,$(UNAME_P)),)
			export GOARCH = 386
		endif
	endif

The correct use is "uname -m" to get x86_64. -p will show unknown or i386 depending on operating system.
$ uname --help
Usage: uname [OPTION]...
Print certain system information.  With no OPTION, same as -s.

  -a, --all                print all information, in the following order,
                             except omit -p and -i if unknown:
  -s, --kernel-name        print the kernel name
  -n, --nodename           print the network node hostname
  -r, --kernel-release     print the kernel release
  -v, --kernel-version     print the kernel version
  -m, --machine            print the machine hardware name
  -p, --processor          print the processor type (non-portable)
  -i, --hardware-platform  print the hardware platform (non-portable)
  -o, --operating-system   print the operating system
      --help     display this help and exit
      --version  output version information and exit

modified Makefile with pull request #444

merged into milestone 0.4.0


User permission problem with mounted volume

We try to mount /var/lib/artemis/data to a directory on the host but we have a permission conflict. The user artemis created in the container has the id 1000 and the user we created on the host has the id 1003. So they don't match.
We tried to run the container with --user 1003:1003 but then we get a permission conflict in the container itself because in the dockerfile
RUN chown -R artemis.artemis /var/lib/artemis
was executed which is the container user with id 1000.
The only way we got it running was when we gave the user with the id 1000 on the host the ownership of the directory we want to mount in the container. But as this user is used for other things we are not very happy with the solution.
Is there any other way to solve this problem?

To the best of my knowledge there is not other solution generally speaking. If the other things you need to do are done with containers you can always try data containers or the other tricks with volumes created later.


feat: adding number chart

Fixes Issue

Closes #1612
Changes proposed

This PR adds a number chart and a few revisions. This addition would help new contributors, especially those who are new to open source and tech, have an idea of how they can contribute to LinksHub's project.
Screenshots

Note to reviewers
I need help with hyperlinking the number chart in line 141.

Points system is not needed unless we plan to use these points to make a leaderboard or something. Since, GSSoC is over.

Hey @Anmol-Baranwal. I was heavily inspired by what EddieHub did with them. It's for contributors to determine what they feel they can contribute to at the moment and to grow from there

Points system is not needed unless we plan to use these points to make a leaderboard or something. Since, GSSoC is over.

Hey @Anmol-Baranwal. I was heavily inspired by what EddieHub did with them. It's for contributors to determine what they feel they can contribute to at the moment and to grow from there

Yeah, I have seen them implement this. Just discuss with @rupali-codes

@rupali-codes, can you review this?

it looks fine tho, but can you add the labels we are using and place the table at the end. Also, i'd want to rename this variables like, level 1 -> priority: low level 2 -> priority: medium level 3 -> priority: high

Can you put your suggestions as in-line comments @rupali-codes? I'm having trouble visualizing your ideas.

@CBID2
Just replace the variables

From level 1 to priority low, for other two as well.

@CBID2 Just replace the variables

From level 1 to priority low, for other two as well.

Thanks for your advice @Anmol-Baranwal. Btw, I made a PR for #1464. Check out #1650

Now it's your turn @Anmol-Baranwal

sorry for being late, lgtm

It's ok @rupali-codes. What matters is that you're well! 😊


Logic Flow Improvements for Dependencies

Details of the scenario you try and problem that is occurring:
Now that we have a Graphical Interface where users can pretty much cherry pick what component they want extracted, we need to make sure we tight up our logic flow for the dependencies. For example, if users choose to not include Site Collections, but to include result sources, we need to make sure that the Read-SPSearchResultSource function only extracts the Global Result Source, not those at the site collection level.  A complete review of components dependencies need to be done.
The DSC configuration that is using the resource:
N/A
Version of the Operating System and PowerShell the DSC Target Node is running:
N/A
Version of the DSC module you're using:
N/A

Will not be implemented now that we have the new Gui


Using different latent space and knn for milo analysis

Thanks for the wonderful tool!
I was wondering whether it would be okay to perform MILO analysis based on a different latent space and KNN than those used for building the initial UMAP and clusters.
For example, using X_scVI for the analysis, and then using PCA and a new KNN for MILO analysis.
When tested on a couple of samples, it seems that X_scVI-based MILO may be slighly so too conservative when used on identical sample types with two different conditions, especially when the sample size is only 20~30K, whereas batch-corrected weighted PC-based KNN seems to work slightly better (it does not compress the data too much in terms of spatial FDR).
On one hand, I feel like this approach is acceptable, but on the other hand, I have some doubts.

Hi @revolvefire, this is entirely up to the user and to what you think makes most sense for the dataset you are analysing. I personally find that using the same KNN graph for multiple analyses helps for coherent interpretation.
The best way to safeguard your analysis from false positives coming from imperfect dimensionality reduction is to include confounders (i.e. labels that you use for batch correction) in the DA testing model, either as samples (columns of the count matrix) or as additional factors in the linear model.

Thank you for the detailed reply!


some other networks do not converge in the second stage.

Hi! Thank you for your excellent work, I just replaced the network which is vgg in your original network with Resnet or some others for my interest. during the first-stage training, the network shows almost the same performance with vgg. But in the second-stage training. it seems that the network no longer converges, and the effect is even worse than that in the first stage. According to the concept of heatmaps and feature maps in your paper, there should be some improvement in the accuracy in the second stage, but during my training in the second stage, the validation accuracy has decreased. By the way, in the second stage, I deleted the face alignment operation, the connected section only contains the heatmaps and feature map process second. whether this is the cause of the problem? In my opinion, it does not matter. Looking forward for your responds!!!
Thanks!!!

Hi,
Thanks for the email.
Do you mean that the second stage no longer gets a face image as input? If that's the case then it probably won't work.
You can provide the second stage with a non-aligned image, essentially the same as intput to the first stage. However, if you do that, remember that the heatmap should also be generated from landmarks that have not been aligned.
Best regards,
Marek

hi! Thank you for your quick respond. The feed to the second stage is a cropped face image which is not aligned, the feature map and the heat maps generate from the non-aligned landmarks . But in that case the network doesn’t converge. By the way, the input size, I have modified to 2562563, does the different input sizes cause the problem? can you provide me some advice to solve this situation? Thank you!

sorry to take your time again! I saw some other network including yours, when the images change from the original size to 112 size, the groundtrue landmarks should minus 1 because of the affine tranforms. when should I minus 1 during training? when changing the original size to 256 size, whether the groundtrue need to minus 1?


Converted AbsoluteLayoutDemos to Maui

Converted Android platform. It hangs on splash screen.

Note: It's super involved because of the namespace move. xmlns needs to be set wherever it's used..


fuzz/corpus: Update

This adds more corpus to fuzz as run by me and which found a bit more
crashers now collected in #30.

Thanks for applying.


Bug: openSUSE Leap: standard use of "sdk list java" suspends itself

Bug report
The command sdk list java results in displaying this error:
master ~/dev/kotlin/truesight> sdk list java
[1]  + 13675 suspended (tty output)  sdk list java
exit 150

It's possible to get the list of JREs by using the command fg.
To reproduce

Install SDKMAN with curl -s "https://get.sdkman.io" | bash, openSUSE Leap
Open a new terminal
Run command: sdk list java.

System info

Linux openSUSE Leap 15.5
zsh 5.6 (x86_64-suse-linux-gnu)

Hi @antekone!
Do you have any aliases defined which run a command in the background? Maybe for less?


Select with AngularJS - can' unselect and checkbox doesn't display

I'm using <select multiple> in order to select various options, but it doesn't display the checkbox and I can't un-select items.

Hi, I fixed your issue in the version I'm currently working on. I'll link your issue once I posted the PR.
Anyway thanks to reporting this.


Broken links

In "Paid Lightning Channel providers" the links to "Zero Fee Routing" and "BlockTank" point to a 404 page. Github link too.

In "Paid Lightning Channel providers" the links to "Zero Fee Routing" and "BlockTank" point to a 404 page. Github link too.

Update your app

Solved! Thanks


Regression?

import { assert } from "tsafe" does not work in latest versions
Last known working point (is not the bisection point, might be working on later versions): 1.2.1
@garronej
If you need minimum repro, let me know

Thank you for the report.

Fixing asap

Hi @nopeless,
So 1.3 is when I started to release for ESM alongside CJS.
I though I did everything right but apparently no 😟
I tried to reproduce in every environement that I know of but I was unable to get any error.
If you could please give me some more infos, a reproduction repo would be great obviously.
Thank you for your help!

@garronej thanks for the quick reply. I'll send a repro in a few hours

thank you very much !

@garronej
https://stackblitz.com/edit/github-emokqj?file=test%2Fmain.spec.ts
Here is the reproduction

Thank you.
It's next in my stack

Additional context: it seems that adding "type": "module" breaks

I'ts not fixed.
I'm going to solve this one and for all

Hi @nopeless,
I'm finally done,
Gosh releasing in CJS + ESM + Deno + featuring short import path + enabling cherry-picking + having intelissence suggestion when writing import from "tsafe/*" is HARD.
Can't believe I had to write my own bundler for this.
Anyway, thank you for taking the time to report.
If you could confirm that it's indeed working with tsafe: 1.4.2 that'll be great.

Works perfectly!
Great job 👍


API to evaluate the property of an element

Proposed behavior
The user should be able to perform any js action on element.
Current behavior
No API to get this information
Tech notes
As discussed one approach could be
evaluate(link("something"), (element) => element.style.backgroundColor)

Here we are getting the bgColor of an element

Fix should be available in b664dd3


runtime: investigate possible Go scheduler improvements inspired by Linux Kernel's CFS

First, please let me pay tribute to your contributions. You guys are awesome!
It has been more than ten years, and Go has already been very successful. So, I think it is time to give Go a better scheduler -- maybe like the CFS scheduler of Linux kernel.
The current implementation of the scheduling strategy of the Go runtime scheduler is basically Round-Robin scheduling. But in practical application scenarios, priority relationships may need to exist between different types of tasks or even between tasks of the same type. For example, if we want to use Go to implement a storage engine, in general, I/O tasks should have higher priority than any other CPU intensive tasks. Because in this case, the system bottleneck is more likely to be in disk I/O, not CPU. For another example, we may use Go to implement a network service, which is required to ensure the latency of some certain lightweight tasks even while other goroutines are quite CPU intensive. However, in the current implementation of Go runtime, if there are a certain number of CPU intensive tasks that need to run for a long time, it will inevitably impact the scheduling latency. Therefore, we need a real mature Go scheduler like the CFS scheduler in kernel.
From another perspective, we could compare the thread model provided by kernel with the goroutine model provided by Go. In addition to the inherent low-cost and high-concurrency advantages of goroutine over threads, some very useful mechanisms in the kernel thread model cannot be found in the goroutine model of Go, at least not yet. For example, dynamic modification of scheduling policy and priority mechanism including adaptive priority adjustment.
The scheduler of the initial versions of Linux kernel, e.g. v0.01, is quite primitive. But with the continuous development of the kernel, more and more applications have higher and higher requirements for schedulers, and the kernel scheduler has been keeping evolving until today's CFS scheduler.
That should also apply to Go.
I have already developed a customized scheduler over Go runtime as a demo and which proves my proposal is feasible and such scheduler would benefit many applications which have high requirements from the perspective of latency and throughput.
Terminology
Event intensive task: Most of the time in its life cycle is waiting for events (just name it off CPU), and only a very small part of the time is doing CPU computing.
CPU intensive task: Most of or even all of its life cycle is doing CPU computing (just name it on CPU), and only a very small part of its time or has never been in the state of waiting for events.
Description
Basically, the idea of CFS scheduling is to give every thread a logical clock which records the duration of thread's on-cpu time. Different priority settings of threads, different speed time passes of its logical clock. And CFS scheduler would prefer to choose the thread which has the most behind logical clock to run. Because the scheduler thinks it is quite unfair to make such thread even more behind. After all, the scheduler's name CFS stands for Completely Fair Scheduler. So, if one thread is event intensive, then it would have a higher de facto priority. And if one thread is cpu intensive, then it would have a lower de facto priority. That we could call it adaptive priority adjustment. That is a quite important feature which could ensure the scheduling latency of event intensive threads will not be very high even current system is under high cpu load due to the existence of some cpu intensive threads.
Although this demo only implements some part of CFS scheduling, the result is quite promising.
Demo
cpuworker/example/demo.go:
package main

import (
	"crypto/rand"
	"fmt"
	"hash/crc32"
	"log"
	_ "net/http/pprof"

	mathrand "math/rand"
	"net/http"
	"runtime"
	"time"

	"github.com/hnes/cpuworker"
)

var glCrc32bs = make([]byte, 1024*256)

func cpuIntensiveTask(amt int) uint32 {
	var ck uint32
	for range make([]struct{}, amt) {
		ck = crc32.ChecksumIEEE(glCrc32bs)
	}
	return ck
}

func cpuIntensiveTaskWithCheckpoint(amt int, checkpointFp func()) uint32 {
	var ck uint32
	for range make([]struct{}, amt) {
		ck = crc32.ChecksumIEEE(glCrc32bs)
		checkpointFp()
	}
	return ck
}

func handleChecksumWithCpuWorkerAndHasCheckpoint(w http.ResponseWriter, _ *http.Request) {
	ts := time.Now()
	var ck uint32
	cpuworker.Submit1(func(checkpointFp func()) {
		ck = cpuIntensiveTaskWithCheckpoint(10000+mathrand.Intn(10000), checkpointFp)
	}).Sync()
	w.Write([]byte(fmt.Sprintln("crc32 (with cpuworker and checkpoint):", ck, "time cost:", time.Now().Sub(ts))))
}

func handleChecksumSmallTaskWithCpuWorker(w http.ResponseWriter, _ *http.Request) {
	ts := time.Now()
	var ck uint32
	cpuworker.Submit(func() {
		ck = cpuIntensiveTask(10)
	}).Sync()
	w.Write([]byte(fmt.Sprintln("crc32 (with cpuworker and small task):", ck, "time cost:", time.Now().Sub(ts))))
}

func handleChecksumWithoutCpuWorker(w http.ResponseWriter, _ *http.Request) {
	ts := time.Now()
	var ck uint32
	ck = cpuIntensiveTask(10000 + mathrand.Intn(10000))
	w.Write([]byte(fmt.Sprintln("crc32 (without cpuworker):", ck, "time cost:", time.Now().Sub(ts))))
}

func handleDelay(w http.ResponseWriter, _ *http.Request) {
	t0 := time.Now()
	wCh := make(chan struct{})
	go func() {
		time.Sleep(time.Millisecond)
		wCh <- struct{}{}
	}()
	<-wCh
	w.Write([]byte(fmt.Sprintf("delayed 1ms, time cost %s :)\n", time.Now().Sub(t0))))
}

func main() {
	rand.Read(glCrc32bs)
	nCPU := runtime.GOMAXPROCS(0)
	cpuP := cpuworker.GetGlobalWorkers().GetMaxP()
	fmt.Println("GOMAXPROCS:", nCPU, "DefaultMaxTimeSlice:", cpuworker.DefaultMaxTimeSlice,
		"cpuWorkerMaxP:", cpuP, "length of crc32 bs:", len(glCrc32bs))
	http.HandleFunc("/checksumWithCpuWorker", handleChecksumWithCpuWorkerAndHasCheckpoint)
	http.HandleFunc("/checksumSmallTaskWithCpuWorker", handleChecksumSmallTaskWithCpuWorker)
	http.HandleFunc("/checksumWithoutCpuWorker", handleChecksumWithoutCpuWorker)
	http.HandleFunc("/delay1ms", handleDelay)
	log.Fatal(http.ListenAndServe(":8080", nil))
}

The server cpuworker/example/demo.go is running at an aws instance c5d.12xlarge and with the env GOMAXPROCS set to 16.
$ GOMAXPROCS=16 ./cpuworker-demo

GOMAXPROCS: 16 cpuWorkerMaxP: 12 length of crc32 bs: 262144

The benchmark tool is running at an aws instance c5d.4xlarge. The two machines is running at the same cluster placement group.
# please complete the server IP
SeverIP=x.x.x.x

# cmd 1
while true; do sleep 1;ab -s10000000 -c 100 -n 60000 http://$SeverIP:8080/delay1ms; done

# cmd 2
while true; do sleep 1;ab -s10000000 -c 100 -n 10000 http://$SeverIP:8080/checksumWithoutCpuWorker; done

# cmd 3
while true; do sleep 1;ab -s10000000 -c 100 -n 10000 http://$SeverIP:8080/checksumWithCpuWorker; done

# cmd 4
while true; do sleep 1;ab -s10000000 -c 100 -n 10000 http://$SeverIP:8080/checksumSmallTaskWithCpuWorker; done

Step 1: Killall already existing cmd x, then run the cmd 1 (run the standalone benchmark of delay1ms).
$ ab -s10000000 -c 100 -n 60000 http://$SeverIP:8080/delay1ms
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking <IP_ADDRESS> (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:
Server Hostname:        <IP_ADDRESS>
Server Port:            8080

Document Path:          /delay1ms
Document Length:        37 bytes

Concurrency Level:      100
Time taken for tests:   0.225 seconds
Complete requests:      10000
Failed requests:        1066
   (Connect: 0, Receive: 0, Length: 1066, Exceptions: 0)
Total transferred:      1538813 bytes
HTML transferred:       368813 bytes
Requests per second:    44413.06 [#/sec] (mean)
Time per request:       2.252 [ms] (mean)
Time per request:       0.023 [ms] (mean, across all concurrent requests)
Transfer rate:          6674.16 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:     1    2   0.4      2       4
Waiting:        1    2   0.4      1       4
Total:          1    2   0.5      2       5
ERROR: The median and mean for the waiting time are more than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      3
  95%      3
  98%      4
  99%      4
 100%      5 (longest request)

Step 2: Killall already existing cmd x, then run the cmd 1 and cmd 2 simultaneously (run the benchmark of delay1ms with a very heavy cpu load which is scheduled by the original Go scheduler).
Current CPU load of the server-side (and please note that the load average is already reaching the GOMAXPROCS, i.e. 16 in this case):

$ ab -s10000000 -c 100 -n 60000 http://$SeverIP:8080/delay1ms
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking <IP_ADDRESS> (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:
Server Hostname:        <IP_ADDRESS>
Server Port:            8080

Document Path:          /delay1ms
Document Length:        38 bytes

Concurrency Level:      100
Time taken for tests:   31.565 seconds
Complete requests:      10000
Failed requests:        5266
   (Connect: 0, Receive: 0, Length: 5266, Exceptions: 0)
Total transferred:      1553977 bytes
HTML transferred:       383977 bytes
Requests per second:    316.80 [#/sec] (mean)
Time per request:       315.654 [ms] (mean)
Time per request:       3.157 [ms] (mean, across all concurrent requests)
Transfer rate:          48.08 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    50  314  99.3    293    1038
Waiting:       11  305 102.5    292    1038
Total:         50  314  99.3    293    1038

Percentage of the requests served within a certain time (ms)
  50%    293
  66%    323
  75%    353
  80%    380
  90%    454
  95%    504
  98%    604
  99%    615
 100%   1038 (longest request)

As we can see, the latency becomes very high and unstable.
Step 3: Killall already existing cmd x, then run the cmd 1 and cmd 3 simultaneously (run the benchmark of delay1ms with a very heavy cpu load which is scheduled by our own scheduler over the original Go scheduler).
Current CPU load of the server-side (and please note that the load average is near the cpuWorkerMaxP, i.e. 12 in this case, and you could set this parameter by yourself):

$ ab -s10000000 -c 100 -n 60000 http://$SeverIP:8080/delay1ms
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking <IP_ADDRESS> (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:
Server Hostname:        <IP_ADDRESS>
Server Port:            8080

Document Path:          /delay1ms
Document Length:        37 bytes

Concurrency Level:      100
Time taken for tests:   0.234 seconds
Complete requests:      10000
Failed requests:        1005
   (Connect: 0, Receive: 0, Length: 1005, Exceptions: 0)
Total transferred:      1538877 bytes
HTML transferred:       368877 bytes
Requests per second:    42655.75 [#/sec] (mean)
Time per request:       2.344 [ms] (mean)
Time per request:       0.023 [ms] (mean, across all concurrent requests)
Transfer rate:          6410.35 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:     1    2   0.5      2       4
Waiting:        1    2   0.4      2       4
Total:          1    2   0.5      2       5

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      3
  90%      3
  95%      4
  98%      4
  99%      4
 100%      5 (longest request)

Now the latency becomes fine again even it is running with many CPU intensive tasks!
Step 4: Killall already existing cmd x, then run the cmd 1, cmd 3, and cmd 4 simultaneously (run the benchmark of delay1ms and checksumSmallTaskWithCpuWorker with a very heavy cpu load which is scheduled by our own scheduler over the original Go scheduler).
Current CPU load of the server-side (and please note that the load average is near the cpuWorkerMaxP, i.e. 12 in this case, and you could set this parameter by yourself):

$ ab -s10000000 -c 100 -n 60000 http://$SeverIP:8080/delay1ms

This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking <IP_ADDRESS> (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:
Server Hostname:        <IP_ADDRESS>
Server Port:            8080

Document Path:          /delay1ms
Document Length:        37 bytes

Concurrency Level:      100
Time taken for tests:   0.238 seconds
Complete requests:      10000
Failed requests:        1038
   (Connect: 0, Receive: 0, Length: 1038, Exceptions: 0)
Total transferred:      1538857 bytes
HTML transferred:       368857 bytes
Requests per second:    42031.11 [#/sec] (mean)
Time per request:       2.379 [ms] (mean)
Time per request:       0.024 [ms] (mean, across all concurrent requests)
Transfer rate:          6316.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:     1    2   0.5      2       5
Waiting:        1    2   0.4      1       5
Total:          1    2   0.6      2       5
ERROR: The median and mean for the waiting time are more than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      3
  90%      3
  95%      4
  98%      4
  99%      4
 100%      5 (longest request)

$ ab -s10000000 -c 100 -n 10000 http://$SeverIP:8080/checksumSmallTaskWithCpuWorker

This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking <IP_ADDRESS> (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:
Server Hostname:        <IP_ADDRESS>
Server Port:            8080

Document Path:          /checksumSmallTaskWithCpuWorker
Document Length:        71 bytes

Concurrency Level:      100
Time taken for tests:   0.469 seconds
Complete requests:      10000
Failed requests:        9157
   (Connect: 0, Receive: 0, Length: 9157, Exceptions: 0)
Total transferred:      1889624 bytes
HTML transferred:       719624 bytes
Requests per second:    21333.56 [#/sec] (mean)
Time per request:       4.687 [ms] (mean)
Time per request:       0.047 [ms] (mean, across all concurrent requests)
Transfer rate:          3936.76 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       2
Processing:     1    4   3.3      3      13
Waiting:        1    4   3.3      3      13
Total:          2    5   3.4      3      13

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      4
  75%      6
  80%      9
  90%     11
  95%     11
  98%     12
  99%     12
 100%     13 (longest request)

The latency of both of them are fine :-)
This demonstration is only to prove that this proposal is feasible and will have obvious benefits to those applications which cares about latency. Moreover, for many applications, throughput is directly affected by latency, so this proposal can also optimize the throughput of those applications
Proposal
Option 1: Bring Go a better scheduler just like Linux Kernel's CFS Scheduler. Support adaptive priority adjustment. Goroutine Setpriority like syscall.Setpriority could also be supported which only influences the speed time passes of the logical clock.
Option 2: Give users the ability to customize their own scheduler without the need to modify their already existing Go codes. I didn't get a very detailed idea yet. Look forward to your excellent ideas.
You might say that one could achieve this goal in the application layer like the way of cpuworker did, but for most applications with already a very large amount of Go codes, such change is just too expensive. And such change in runtime would be more efficient. Therefore, I tend to hope that Go could provide a similar mechanism rather than expect users to achieve this goal by themselves in the application layer.
Thanks a lot for your time to watch this proposal :-)

There are no API changes here so this isn't really a proposal. Retitling to reflect the intent as mentioned in Option 1 above.

Reworking the scheduler is something we've thought about on and off for quite a while. I haven't taken a close look at your demo scheduler yet, but it looks nifty.
In addition to the issues you describe with IO-intensive tasks, overall CPU overhead of the scheduler is something that comes up frequently as problematic for applications that happen to stress corners of the scheduler. So another point to consider.
Any major change we make to the scheduler will likely need to be paired with an investment in better testing of the scheduler. The current scheduler has evolved to become quite complex, but with few explicit, deterministic tests due to the difficult nature of testing a scheduler deep in the runtime, instead relying on larger Go programs to reveal problems. This makes it difficult to have high confidence in major changes, so investing in better testing is definitely something I'd want to see (whether or not we make large changes to the scheduler, but definitely if we do).

Reworking the scheduler is something we've thought about on and off for quite a while. I haven't taken a close look at your demo scheduler yet, but it looks nifty.

Thanks a lot, @prattmic.

In addition to the issues you describe with IO-intensive tasks, overall CPU overhead of the scheduler is something that comes up frequently as problematic for applications that happen to stress corners of the scheduler. So another point to consider.

Any major change we make to the scheduler will likely need to be paired with an investment in better testing of the scheduler. The current scheduler has evolved to become quite complex, but with few explicit, deterministic tests due to the difficult nature of testing a scheduler deep in the runtime, instead relying on larger Go programs to reveal problems. This makes it difficult to have high confidence in major changes, so investing in better testing is definitely something I'd want to see (whether or not we make large changes to the scheduler, but definitely if we do).

Yes, I agree with you. I investigated the performance of Go priority heap:
 Tested on RockyLinux 8.5, AMD 5950x

 1000,000 max size of the heap

 push with random priority
 	~ 30 ns/op - 50.9 ns/op
 pop
 	~ 4.2 ns/op - 4.5 ns/op

Such cost might be affordable.
I think maybe we should decouple the specific scheduler implementation from other parts of runtime like Linux Kernel does:
(https://github.com/torvalds/linux/blob/e6251ab4551f51fa4cee03523e08051898c3ce82/kernel/sched/sched.h#L2123-L2186)
// definition of `sched_class` in kernel
struct sched_class {

#ifdef CONFIG_UCLAMP_TASK
	int uclamp_enabled;
#endif

	void (*enqueue_task) (struct rq *rq, struct task_struct *p, int flags);
	void (*dequeue_task) (struct rq *rq, struct task_struct *p, int flags);
	void (*yield_task)   (struct rq *rq);
	bool (*yield_to_task)(struct rq *rq, struct task_struct *p);

	void (*check_preempt_curr)(struct rq *rq, struct task_struct *p, int flags);

	struct task_struct *(*pick_next_task)(struct rq *rq);

	void (*put_prev_task)(struct rq *rq, struct task_struct *p);
	void (*set_next_task)(struct rq *rq, struct task_struct *p, bool first);

#ifdef CONFIG_SMP
	int (*balance)(struct rq *rq, struct task_struct *prev, struct rq_flags *rf);
	int  (*select_task_rq)(struct task_struct *p, int task_cpu, int flags);

	struct task_struct * (*pick_task)(struct rq *rq);

	void (*migrate_task_rq)(struct task_struct *p, int new_cpu);

	void (*task_woken)(struct rq *this_rq, struct task_struct *task);

	void (*set_cpus_allowed)(struct task_struct *p,
				 const struct cpumask *newmask,
				 u32 flags);

	void (*rq_online)(struct rq *rq);
	void (*rq_offline)(struct rq *rq);

	struct rq *(*find_lock_rq)(struct task_struct *p, struct rq *rq);
#endif

	void (*task_tick)(struct rq *rq, struct task_struct *p, int queued);
	void (*task_fork)(struct task_struct *p);
	void (*task_dead)(struct task_struct *p);

	/*
	 * The switched_from() call is allowed to drop rq->lock, therefore we
	 * cannot assume the switched_from/switched_to pair is serialized by
	 * rq->lock. They are however serialized by p->pi_lock.
	 */
	void (*switched_from)(struct rq *this_rq, struct task_struct *task);
	void (*switched_to)  (struct rq *this_rq, struct task_struct *task);
	void (*prio_changed) (struct rq *this_rq, struct task_struct *task,
			      int oldprio);

	unsigned int (*get_rr_interval)(struct rq *rq,
					struct task_struct *task);

	void (*update_curr)(struct rq *rq);

#define TASK_SET_GROUP		0
#define TASK_MOVE_GROUP		1

#ifdef CONFIG_FAIR_GROUP_SCHED
	void (*task_change_group)(struct task_struct *p, int type);
#endif
};

This is not only convenient for testing but also convenient for users to add their own scheduler implementation if they want.

Thank you for working on this.
I know nothing of the Go scheduler, but a few questions came to mind when reading your proposal:

I suspect goroutines are much shorter-lived (i.e. more fine-grained) than kernel threads. Does that affect the applicability of the Linux approach in Go?
Would this benefit from grouping goroutines by inferred "type of work" to bootstrap the priority?
If goroutines are indeed short-lived, does that increase the possibility of starving old goroutines?

Maybe MLFQ could be of use? Goroutines of the same priority level would be run in a round robin style, perhaps not too different from the current approach. Demotions would occur after time allotments are used up. And periodic priority boosts could prevent starvation for longer-running tasks.

@adyanth While your concern is fair, I believe this may not be the case eventually. Kernel threads are oblivious to Go runtime or what kind of code the thread is running ( ignoring IO wait and CPU fairness ). A new runtime scheduler is in turn a pure userspace construct, how it manages its own goroutines should not compete with kernel scheduler which operates on the other side of system boundary.

Thank you for working on this.

It's very kind of you, @tommie :)

I know nothing of the Go scheduler, but a few questions came to mind when reading your proposal:
I suspect goroutines are much shorter-lived (i.e. more fine-grained) than kernel threads. Does that affect the applicability of the Linux approach in Go?

I'm afraid that would not make a big difference. Round-Robin is a kind of scheduling strategy, and so is CFS. The key difference between them is CFS could do some adaptive priority adjustment and try to achieve the Completely Fair scheduling. As we already saw in the demo above, if we run a number of event intensive tasks and cpu intensive tasks together under the Round-Robin scheduling strategy, it would be never fair for event intensive tasks -- because these cpu intensive tasks would consumes much more cpu time than event intensive tasks.

Would this benefit from grouping goroutines by inferred "type of work" to bootstrap the priority?

Do you mean something like go fp() type-of-work-or-priority which could set a constant priority when the goroutine is created? Yes, there would be some benefits. But that would be not very friendly to those Go applications which already have a very large codebase. And furthermore, think about those hybrid tasks which combine event & cpu types together, or tasks might change their types from time to time. So, I think it's best to give runtime this job, then our applications do not need to change a single line of code.
I think the API designing of kernel syscalls related to the scheduler is also worth learning:
setpriority // set the "literal" priority, then scheduler could adaptive adjust the real priority based on this "literal" priority
sched_setscheduler // set scheduling strategy, rr or cfs?
sched_setparam // tuning

If goroutines are indeed short-lived, does that increase the possibility of starving old goroutines?

To be honest, I thought about this problem myself before, and my conclusion is that such problem would unlikely happen in the real production environment. And if it happens, that means the system is already far more overload.
Please take a look on the benchmark below:
func BenchmarkIntChan(b *testing.B) {
    ch := make(chan int)
    go func() {
        for {
            <-ch
        }
    }()
    for i := 0; i < b.N; i++ {
        ch <- 1
    }
}

func BenchmarkIntChanX(b *testing.B) {
    ch := make(chan int)
    go func() {
        for {
            <-ch
        }
    }()
    for i := 0; i < b.N; i++ {
        myCh := make(chan int)
        go func() {
            ch <- 1
            myCh <- 1
        }()
        <-myCh
    }
}

Which yields:
$ # rockyLinux 8.5 & AMD 5950x
$ GOMAXPROCS=1 go test -bench=.
goos: linux
goarch: amd64
pkg: echo
cpu: AMD EPYC-Milan Processor
BenchmarkIntChan         8433139               139.5 ns/op
BenchmarkIntChanX        2393584               495.8 ns/op

Let's assume it takes about 1us on-cpu time to run one of those short-lived goroutines, and we have 32 logical cores, then we could get ~32,000,000 op/sec which means a very huge load. If such applications really exist, that means either the design of such application is not very good, or other components (kernel, network, disk ...) will reach their bottlenecks before our Go. This benchmark also proves the excellent performance of goroutine and chan.

If goroutines are indeed short-lived, does that increase the possibility of starving old goroutines?

@tommie, and furthermore, there are strategies both in CFS and MLFQ (which were mentioned by @tfaughnan above) to prevent this from happening.

Maybe MLFQ could be of use? Goroutines of the same priority level would be run in a round robin style, perhaps not too different from the current approach. Demotions would occur after time allotments are used up. And periodic priority boosts could prevent starvation for longer-running tasks.

Very good point, @tfaughnan. My only concern is that a mature MLFQ may not be much simpler or maybe even more complex than CFS, so I think an interface like sched_class in kernel could be added to decouple the specific scheduler implementation from the scheduler interface. In this way, Go could even support multiple different schedulers.

I am not aware of how the interactions between the two schedulers (go vs OS) is being tested now, but making the two behave similarly might have drastic behaviour causing them to work against each other (I can think of TCP over TCP meltdown). Which means this would be a slow process.

Hi @adyanth, I agree with @singhpradeep's reply above. Go runtime scheduler is running on the top of kernel scheduler, so they are not in a competitive relationship. Furthermore, they could "work together".

I wrote a similar package to play with different scheduling algos: https://github.com/tommie/go-coopsched

It implements FIFO and something similar to hnes/cpuworker, see function Waitiness.
It uses Go benchmarks.
The "initial results" are meant as a rough analysis to prove to myself the scheduler probably works. :)
It only shows average latency (wait overhead), not percentiles.
It passes the task in a context.Context rather than directly. Overheady, but felt nice.
More notable differences.

Assuming the code works, and that Waitiness is a reasonable simplification of calcEIfactorAndSumbitToRunnableTaskQueue, it does seem to lower the wait overhead (i.e. the time from Sleep being done to the task running again).

I'd like to post here my partial skepticism for this change. I believe as a goal, it might be a good thing. I am not equipped to evaluate whether adopting CFS-scheduler algorithm in the goroutine clockwork is a good idea moving forward or not.
What I do like to point out is that Go has a surprising small number of adjustment knobs considering the complexity of the runtime (we do see a lot of knobs tweaking the compilation or part of the behavior of the standard library though), I might be forgetting something here, but to the best of my recollection we have just GOGC, GOMAXPROCS, GORACE, and GOTRACEBACK - and these are all scalar values.
That leads to a significantly easier experience from operations standpoint. Therefore, adopting a custom goroutine scheduler interface, or adding "just another switch" for an alternative operation mode might shave against the grain in the long term stance of Go having as few knobs as possible.
In other words, CFS Scheduler good. More knobs, I am not so sure.

Agreed about Go's lack of knobs being a great feature.
Here's a self-contained cpuworker/example/demo.go. The percentile printout isn't as nice as ab, but it seems to show @hnes's point just by looking at the value ranges.
I wonder if this could go as an add-on to the existing runq FIFO. If priority levels are fairly coarse, the lowest priority g:s could be in runq, while the others are in a priority queue, sitting between runq and globrunq. That, way, the top of the priority queue could just be batch-taken into runq as needed. Introducing that priority queue would leave the field open for experimenting with different algorithms for selecting priority, and yet would be easy to disable until the feature is finished.

If the internal working of the Go scheduler is to be reconsidered then an obvious question arises. Should any new scheduler take account of the heterogeneous nature of modern CPUs. Arm has had big.LITTLE for over ten years, Apple's M1 processors have Firestorm and Icestorm cores and Intel Alder Lake CPUs have Golden Cove high-performance cores and Gracemont power-efficient cores.
I too appreciate the lack of "knobs" in the Go runtime but the asymmetric nature of current CPUs may require an extra knob or two to schedule Goroutines as efficiently as possible.

Naive question: Would this interact at all with package sync's fairness algorithms?

I think these two behaviors of the Go scheduler must fix in the new version (If any schedule):

randomly scheduled or rescheduled goroutines: A goroutine may be arbitrarily and randomly scheduled or rescheduled on different running threads(CPU cores), i.e., the same piece of code will be called from different threads over time, even without evolving the go keyword. In this case, that talks more here, in many logic developers need to care more about syncing the operations that produce by this random behavior. for example, when you want to implement workers logic to handle income network services, you can't be sure that if you have e.g. 10 workers for a user connection, the workers run sync in order, so need to use locking or atomic mechanism to implement connections metrics, But if goroutines can be more predictable it is easy to forget about concurrency because we are sure that all same connections workers run on same CPU cores. something like network Tx and Rx queue in kernels that you can easily predict same sender packets just goes to the same queue in same CPU core over the times. No packet can process before the last one finishes its path logic even if the CPU core is scheduled to another process and back again later.
Hybrid kernel/userspace thread manager/scheduler: I know for blocking syscalls we need some mechanism to block a thread that needs to kernel know about threads, but In fundamental thinking, Why hybrid mechanism be the first-class citizen? Why not pure userspace scheduler be the first-class citizen? and have some mechanism in some rare situation to schedule a new OS thread of course if OS kernel support it (some new kernels like Unikernels suggest dropping kernel-level threads manager at all in favor of pure userspace threads). I suggest just having one thread per CPU core and having a priority mechanism for goroutines. Also, as said above a goroutine can lock to one CPU core.
Some other problems like #14592 must be considered in the re-written Go scheduler.

Both suggestions don't change the Go behavior and Go apps easily work out of the box, just change the way advanced developers write frameworks to developed software by Go in a better way.

See also #33803


Migrate Connection Events and Lifecycle

This needs to be filled out with more detail. References the Partially Complete Connection feature from the Readme.

Quick update here. The connection_events branch that resolves these and other issues finally has all its tests passing. It's not complete, but things are closing in on an RC.


Problem with export to csv

Dear Matt,
I'm using the latest version of Larix (larch version 0.9.74). I noticed that I cannot export some processed data to csv files - especially from EXAFS.  For the same file I can export normalized, flattened data but not in kspace or Rspace.
I got such error:
Could not export CSV File C:\Users\Sample1_Ni3B_Rspace.csv
Traceback (most recent calls last):
File "C:\Users\anna.wach\xraylarch\Lib\site-packages\larch\wxxas\xasgui.py", line 686, in onExportCSV
groups2csv(savegroups, outfile, x=res.xarray, y=res.yarray,
File "C:\Users\anna.wach\xraylarch\Lib\site-packages\larch\io\csvfiles.py", line 94, in groups2csv
buff.append(delim.join([gformat(s[i]) for s in columns]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\anna.wach\xraylarch\Lib\site-packages\larch\io\csvfiles.py", line 94, in
buff.append(delim.join([gformat(s[i]) for s in columns]))
~^^^
IndexError: index 392 is out of bounds for axis 0 with size 392
Any ideas why? I guess this may be a problem only on my computer.
Best,
Anna

Hi @AnnaDragonfly,
The error shows that the length of the x axis and y axis of the data is not matching.
Is there any chance that you are using "energy" as a x axis?
If so, you can specify "k" for the "X Array" when you save it. (same for R)
length of 392 corresponds to k = 19.6, which is relatively large for k space.

Hi @Ameyanagi
You're right. I was so used to Athena, that I didn't notice this.
Thanks for your prompt answer.
Best,
Anna

@AnnaDragonfly @Ameyanagi Thanks -- I was just looking at this too.  I will try to fix it so that selecting "chi(k)" will change the default X-axis to "k" (and also for chi(R)).  And I'll try to give a better warning.

@newville
I think it would be better to automatically select k when we choose "chi(k)" and r or q when we select the r or q space.
Thank you.

@Ameyanagi done - well, at least the easy part ;)


Add cross depiction to LineRepresentation

References: #354. Let me know if there's anything else that needs adding or fixing.

thanks!!!


Basic grammar for LaTeX letters with regexes fails

from PyQt5.QtCore import QObject, QThread, pyqtSignal
from lark import Lark, Transformer

grammar = """start:
              LETTER: "\" (VARGREEK | GREEK | LATIN)
              GREEK: /"alpha" | "beta" | "gamma" | "delta" | "epsilon" | "zeta" | "eta"
                 | "theta" | "iota" | "kappa" | "lambda" | "mu" | "nu" | "xi" | "omicron"
                 | "pi" | "rho" | "sigma" | "tau" | "upsilon" | "phi" | "chi" | "psi" | "omega"/i
              VARGREEK: "varepsilon" | "vartheta" | "varkappa" | "varpi" | "varrho" | "varsigma" | "varphi"
              LATIN: /[a-z]/i
              NATURAL: /[0-9]+/
        """

parser = Lark(grammar)

tree = parser.parse("some input string")

class MyTransformer(Transformer):
    def rule1(self, matches):
        return matches[0] + matches[1]

    # I don't have to implement rule2 if I don't feel like it!

new_tree = MyTransformer().transform(tree)

Exception:
￼ File "C:\Users\FruitfulApproach\Desktop\zoomspace_app\latex_parsing.py", line 14, in <module>
  parser = Lark(grammar)
File "c:\Python35-32\Lib\site-packages\lark\lark.py", line 153, in __init__
  self.grammar = load_grammar(grammar, source)
File "c:\Python35-32\Lib\site-packages\lark\load_grammar.py", line 596, in load_grammar
  raise GrammarError("Unexpected input %r at line %d column %d in %s" % (e.context, e.line, e.column, name))

lark.common.GrammarError: Unexpected input '/"alp' at line 3 column 21 in <string>

Another approach is to use raw strings: r""" ... """


Assumptions behind assembly and resource names, namespaces and paths

(Using .NET 7)
I've been banging my head trying to get the embedded localization json files to load...
I have a project where the default namespace has no relation with the assembly name. As such, a type's FullName is decorated with a namespace that the ResourcePathHelper has no idea how to handle.
Also, you're replacing all the '.' with '/', but the EmbeddedFileProvider is a flat provider, and all the resources are given by accessing their complete type name in the root folder, so using '/'s doesn't seem to work at all...
So, in order for this to work at all with my setup, I had to:

change line 113 of EmbeddedJsonLocalizationExtensionService to: return new EmbeddedFileProvider(assembly, "");
comment out lines 39-45 of ResourcePathHelper
leave the resourcespath property empty and put the json files in the same folder as the razor components

Mind you, I quite like having done point 3, as it's just easier to refer to both the razor component and the json file when they're side by side, but not everyone is like me... ;) Also, there should be some way to fix it so that other people don't have to do 1 and 2.
If I follow the setup for having a dedicated json folder, the resources get stored in the assembly as somthething like:
part.of.namespace.path.to.resources.rest.of.namespace.typename.json
and there's no reliable generic way to get at them...
I propose you do the following changes:

Keep my change to EmbeddedJsonLocalizationExtensionService, with the "" argument to the EmbeddedFileProvider constructor
Add a DefaultNamespace option to EmbeddedJsonLocalizationOptions
In EmbeddedJsonLocalizationExtensionService.TryLoadAsync (lines 39-60), if DefaultNamespace is not null, take the ResourcesPath property, replace any '/' with '.', and insert the value of the property in between the default part of the type's full name and the specific part

Otherwise, you're basically forcing the client assembly's name to be the same as its project's default namespace, and you're forcing all its types to be somewhere in that namespace.

Hi,
Thanks for this feedback!
What version of the project do you use?
I will provide a fix for this point.
I think I am going to add a new option in EmbeddedJsonLocalizationOptions:
        /// <summary>
        /// Gets/Sets Assembly Root NameSpace resolver.
        /// </summary>
        /// <remarks>
        /// By default the root name space will be resolved as the assembly name.
        /// </remarks>
        public Func<Assembly, string?>? RootNameSpaceResolver { get; set; }

This way you will be able to resolve yourself the root name space for the given assembly.

I downloaded your main branch. :)
The new property will certainly help, and the default is reasonable.

One more thing I did: I named all my json files .razor.json and changed the extension service accordingly. That way, VS2022 lists them neatly under the associated component, as if it were a CSS file. :)
So, I might suggest you actually go larger and instead of (or in addition to) RootNameSpaceResolver, add a ResourceNameResolver:
public Func<Type, string?>? ResourceNameResolver{ get; set; }

Ok I will add an option to handle this.
Didn't you get any trouble to load embedded json named with *.razor.json or *.razor.en.json or *.razor.fr.json?
The fact is that there are some limitations on the embedded file name with several "dot": see https://learn.microsoft.com/en-us/dotnet/core/resources/manifest-file-names
On my side I am experiencing issues with embedded file name like *.razor.fr.json and I have to name it like this *.razor-fr.json.

Didn't you get any trouble to load embedded json named with *.razor.json or *.razor.en.json or *.razor.fr.json?
The fact is that there are some limitations on the embedded file name with several "dot": see https://learn.microsoft.com/en-us/dotnet/core/resources/manifest-file-names

They're working like a peach. :)
From what I gathered, these points are specifically about .resx files implicitly included by MSBuild. Files explicitly added added as embedded resources by the user (seem to) have no such limitations.
This article is also almost a year old, so this may well be a .NET6 vs .NET 7 issue, or it could even be an MSBuild vs VS2022 issue.

Ok I think I have a solution to solve the dot issue. I will be back with a PR soon!...

I updated the examples to show how to handle the new NamingPolicy option.
in project SoloX.BlazorJsonLocalization.Example.Components.Embedded3
branch dev

Well it's merged in main.

available in nuget version 2.0.0-alpha.4

Works like a charm! :)
Here's the config I'm using:
builder.Services.AddJsonLocalization(builder => builder.UseEmbeddedJson(options =>
{
    options.RootNameSpaceResolver = _ => typeof(App).Namespace!;
    options.NamingPolicy = (path, culture) =>
        $"{path.Replace('/', '.')}.razor{(string.IsNullOrEmpty(culture) ? string.Empty : $"-{culture}")}.json";
}));

Guess you can close this issue. Thank you kindly for this! :)


Standardize list inputs

The following changes are implemented
TODO: Summary
Changes in the user interface:
TODO: Add screenshots, recordings or remove this section
Checklist when submitting a final (!draft) PR

[x] Commits are tidied up, squashed if needed and follow guidelines in CONTRIBUTING.md
[x] Code builds
[x] All existing tests pass
[x] All new critical code is covered by tests
[x] PR is linked to the relevant issue(s)
[x] Rebased with the target branch

Make all tables that use the wrapper be small


Current state of this project?

Hi,
I'd like to better understand the current state of this project. Sorry for asking this in GitHub but it looks like the google groups and slack channel are no longer active. Which is one reason I am even asking the question.
I understand that it has recently been forked from the oracle speedle project, but I can also see that there has been little activity on it. There doesn't seem to be a published docker image despite docker being listed as a deployment option in the docs. The helm chart is not published either.
These are all red flags for me signifying a dead or dying project. However, given that it was recently forked I figured I would at least ask before ruling it out.
We are currently looking at OPA and Casbin. I am currently leaning more towards Casbin because of the persistent storage options. The fact that OPA runs all in memory is a limitation that we don't want to deal with. But then I came across Speedle. The API looks a lot nicer than Casbin and (at least from the surface) it seems a bit more flexible as well. So I would like to know if Speedle is even an option, given it's current state of development, before I sink too much time into it.
So what is the current state of the project and where is it going?
Thanks,
Jordan

hi Jordan,
Thanks for asking this in GitHub! :-) The project is still alive. Actually we are monitoring both Slack and Google groups. If you asked the question there, you would get a prompt response too.
Currently the project is maintained by Cynthia, Bill, myself, and other ex-Oracle colleagues who worked on Speedle project. We maintain it in our spare time as volunteers. There is no commercial companies behind the project. So it doesn't look as "active" as other options you mentioned. However, I'd like to share a few facts here.

Speedle project is adopted by quite a few companies, including but not limited to the largest reinsurance company in Asia, a nation-wide real estate company in China, an ISV offering DevOps system, etc. We spent zero-effort on promoting the project. These companies made the decision to adopt Speedle rather than other options based on their own analysis/testing on the players in this field.

We are adding new features to Speedle project, slowly (I have to admit it). After moving to the new repo, we added MongoDB and policy management console. More features are planned too. I don't think we can speed up in short time unless we get more volunteers to help out or find sponsors.

As I said above, we are monitoring Slack channels and google groups and providing prompt response to any questions. We are really happy to see people show their interest in the project and appreciate any questions, issues, or even challenges. The latest question in Slack is Brian's question about MongoDB support. Cynthia answered the question yesterday. We commit to providing fix to all reported bugs in time.

Speedle is very stable, scalable, and high performance. We built it up based on our 20-year experience in fine-grained access control field. I guess that probably partially results in the "inactive" community. :-) Frankly speaking, I checked with several Speedle users in the past a few months, "did you encounter any issues with Speedle?" They all answered "so far so good". When you compare Speedle with other alternatives, just try them in the toughest scenarios. Think about there are 1 million policies defined the policy store, can we manage the policies efficiently and get authorization result in a few milliseconds? (Don't laugh, it is a real world use case. I once met a customer who created 4 millions of policies. It is crazy I agree though); Can it handle 100,000 authorization checks per second? how many resource will it consume? how about 1,000,000 authorization calls? is it linear scalable? does the project offer tools helping you managing policies or extending the authorization engine? Is the policy definition language intuitive enough such that system administrators, business administrators, internal auditors, lawyers or non-technical senior management all understand it? Is it possible to build a flexible policy model with the project/product? etc etc

I'll leave the issue open such that others who are interested in the project can get some clues too.
Please feel free to let me know (on Slack, github issues, etc) for any further questions.
Thanks,
William

There doesn't seem to be a published docker image despite docker being listed as a deployment option in the docs. The helm chart is not published either.

Please check https://speedle.io/docs/deployment/ for details.

@caiwl Thank you for the detailed and prompt response. I really appreciate it.
My comment on the slack community was only because when I clicked the link on the website it says that it is no longer active. I can't actually join. Same when I click the google groups link on the website; it says "Content unavailable". So I assumed they had been shut down.
That deployment page is what I was referring to. It says

Before deploying Speedle manually, you need to build Speedle and build and push the Speedle Docker images.
Which tells me that there is no official docker image.
And the same is true for the helm chart. It's just part of the repo. I can't do helm repo add ... and then helm install ... and get up and going easily.
 I'm perfectly capable of cloning the repo, building the images, customizing the helm chart, and deploying it myself. But together with the other things I mentioned at least caused me to hesitate and ask this question.
I'm glad that the project is stable and under active development. I will definitely not be crossing it off my list yet.

Thanks again.

Hi Jordan,

My comment on the slack community was only because when I clicked the link on the website it says that it is no longer active. I can't actually join. Same when I click the google groups link on the website; it says "Content unavailable". So I assumed they had been shut down.

My bad! I forgot to update Slack invitation link in speedle.io web page. Just corrected the link and Google group setting. Both Slack and google group should work now.

Which tells me that there is no official docker image. And the same is true for the helm chart. It's just part of the repo. I can't do helm repo add ... and then helm install ... and get up and going easily.

@bill828 I recollect that you planned to add the docker build/publishing logic to pipeline. Any progress so far?

I'm glad that the project is stable and under active development. I will definitely not be crossing it off my list yet.

Thank you very much. If you need any help, please feel free to let us know in Slack.
Thanks,
William


DefaultApp.tsx missing in build

🐛 Bug Report
Cannot find module 'node_modules/next-page-tester/dist/_app/DefaultApp.tsx' from 'node_modules/next-page-tester/dist/loadPage.js'
To Reproduce
install from npm and run example test
Run npx envinfo --preset jest
Paste the results here:
System:
    OS: macOS 11.1
    CPU: (8) x64 Apple M1
  Binaries:
    Node: 14.15.2 - /usr/local/bin/node
    Yarn: 1.22.10 - ~/Desktop/projekty/publisher_pwa/node_modules/.bin/yarn
    npm: 6.14.9 - /usr/local/bin/npm
  npmPackages:
    jest: 26.6.3 => 26.6.3

Hi @tomaszrondio 👋 Could you also share NextJS version you are using?

Hi @tomaszrondio 👋 Could you also share NextJS version you are using?

that is easy :) "next": "10.0.3". I just switched to 10.0.5 and same happens

that is easy :) "next": "10.0.3". I just switched to 10.0.5 and same happens

Nice one! Since we don't import DefaultApp.tsx (but manually require it), Typescript won't include it in its dist folder.
Thanks @tomaszrondio
We should add tests targeting the bundled code.

Nice one! Since we don't import DefaultApp.tsx (but manually require it), Typescript won't include it in its dist folder.
Thanks @tomaszrondio
We should add tests targeting the bundled code.

My pleasure. Always happy to break stuff.

My pleasure. Always happy to break stuff.

I believe temporary fix would be to create a custom app file until this is fixed.

I believe temporary fix would be to create a custom app file until this is fixed.

I do have custom _app.jsx  with lots of stuff

I do have custom _app.jsx  with lots of stuff

I would have expected this issue not to happen when a custom _app file is available. @tomaszrondio are you maybe testing with useApp option set to false?
BTW, I'll publish a patch for the failing require today.

I would have expected this issue not to happen when a custom _app file is available. @tomaszrondio are you maybe testing with useApp option set to false?
BTW, I'll publish a patch for the failing require today.

Published a fix with version 0.16.1 🚢 . Thanks for reporting @tomaszrondio!
Would mind letting us know if the issue is solved?

Published a fix with version 0.16.1 🚢 . Thanks for reporting @tomaszrondio!
Would mind letting us know if the issue is solved?

I updated to 0.16.1 and it works!

I updated to 0.16.1 and it works!


grpc support in fortio (grpc load testing)

fortio support for grpc
fixes #451
fortio now supports grpc testing as well

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
8aa9d2ccef978f579ab42dd71db69d7bd4a4fe34
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Jenkins job istio/presubmit passed

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
a444f5629368478edcccbd816c31fb959ac1e66f
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Jenkins job istio/presubmit passed

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
5b11f83614814bfe3e5916230727cd469ba85bf0
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

https://github.com/istio/istio/commit/a1f87a35acfe7da4b44b70820787a928cc651743#diff-1960f6ed47cfa60dd75b0f30a997add9 in #520 is easier/better to review
will keep this one as WIP though if you want to approve it too that works for me as well

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
e8cf9492032a69786542e809ca730ad6901575a5
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
2019f6d1b83229a2268135c4588b9f26a62855f3
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
b9f687994c783f75407e788513c387404715a876
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

it's ready for review now, please take a look, thanks in advance @douglas-reid

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
aa989f3b21173c2511f1f774dc02211e74f05809
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
2128d04bc848d16623ce27a45e5430733962eae3
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
97dd49370091cb45a5b9cbfc814c4e63ac460968
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
35c1036a1fd1ffe723155b9faebb9174f16670ba
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
afad4e8be011d08a0f14b1bc75a43ccd32ccab3f
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
384d66404805dd1b89667dbe048297d31014490e
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
69e9c6f6f421d91e8358553f78403a856abba747
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
34711c5e652d9281512b88e8d92a2fdca07344eb
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

prow/istio-presubmit.sh and makefile look legit beside the typo

the presubmit part needs https://github.com/istio/test-infra/pull/365

/retest

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
34711c5e652d9281512b88e8d92a2fdca07344eb
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
93915da4bb4de5e8befdc0a4869a6eee2e69a1fd
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
93915da4bb4de5e8befdc0a4869a6eee2e69a1fd
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
93915da4bb4de5e8befdc0a4869a6eee2e69a1fd
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
93915da4bb4de5e8befdc0a4869a6eee2e69a1fd
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ldemailly: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
1f03365bc2edafdb11f29bd98575f08ac760b8c2
link
@istio-testing bazel test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Thanks a lot @douglas-reid for all the reviews!


Can't ignore removing blacklisted libraries

Can't build appimage in docker with necessary libs. In Function delete_blacklisted() all blacklisted libs are being removed but, I needed some of them. Guess we need some way to skip delete_blacklisted or skip removing certain libs form recipe.

https://github.com/AppImage/pkg2appimage/pull/394

Which libraries do you think should not be blacklisted, and why? In that case, we should discuss and solve this for everyone. I don't want that each pkg2appimage user has to think individually about what to bundle and what not to bundle - this should be figured out by the pkg2appimage developers and users, and should be documented in https://github.com/AppImage/pkg2appimage/blob/master/excludelist. I don't want a situation where every 2nd AppImage is broken because someone excluded (or did not exclude) something that we know must be/must not be bundled.

There should be an option in the yml to always include some libraries. My use case is that I want to bundle libstdc++6 from GCC 7. I can not use the older GCC 5 from the oldest supported Ubuntu LTS (16.04) because the project uses C++17 thus it must use at least GCC 7.
Ok, I just noticed that such patch was refused.
The GCC's C++ standard library has new features with every GCC release, and the C++ standard gets updated every 3 years, therefore a better solution may be to remove this library from the exclusion list.

Unfortunately what you suggest would lead to crashes in some situations, and hence is not something we can do.
The best solution is to develop applications using the compiler that comes with the oldest still-supported LTS release of the distribution, in order to ensure that the binaries will run on all still-supported distribution releases. If this is not possible, it likely means that during development no one took care to ensure that the software needs to build and run on older distributions, too.
The second-best solution may be something like https://docs.appimage.org/packaging-guide/advanced/troubleshooting.html
In very rare circumstances (almost never) it is impossible to develop an application using the compiler that comes with the oldest still-supported LTS release of the distribution. In those cases, it may be needed to develop and build on a newer distribution version. In this case, almost every library needs to be bundled privately with the application, also those which normally come with all distributions (e.g., glibc).
If you are interested in this, then ping @azubieta who is working on a solution for that.

Closing this.
If you think that a library should not be blacklisted, then please open a new issue in this project about that library. Because then it should not be blacklisted for everyone. In most cases, the blacklisted libraries are blacklisted for a reason though, and there should not be a way to circumvent the blacklist or else there will be breakage.

I don't have the option to use older GCC than v7. Which issue can be used for further discussion about libstdc++? These two seem good https://github.com/AppImage/AppImageKit/issues/445 https://github.com/AppImage/AppImageKit/issues/225

First see whether https://github.com/darealshinji/AppImageKit-checkrt/ can solve your problem.

this is absolutely horrible, how about you let me worry about crashes?

I don't understand your comment.

I was trying to build something with pkg2appimage and blacklist was getting in a way of not getting the library because you "assumed" it will be installed, it wasn't even even when it was it was missing symbols. There should be a quick way to force include blacklisted deb packages...

Please discuss such issues so that they can be solved for everyone, not just for you. Thanks for sharing!

That's not how any of that works. There are situations when I need the solution right now and for a different linux. I don't have time to discuss it on github and wait for someone to add it. There should simply be a way to unblacklist packages. Simple as that. I did it with running pkg2appimage then stopping it in the middle, editing blacklist and running it again, fortunately it did not "redownload" blacklist.