Model robustness

Hello,
I have checked the facemesh demo of Mediapipe (https://google.github.io/mediapipe/solutions/face_mesh) and face landmark detection (https://blog.tensorflow.org/2020/11/iris-landmark-tracking-in-browser-with-MediaPipe-and-TensorFlowJS.html). The second one really works better and more robust even when I change my head pose extremely and quickly.

But which one did you use in your model? If you used the second one why it is not robust and the facemesh disappears suddenly when I change my head pose extremely?
hand tracking is not working for both hands. Is there a reason for that since in the mediapipe's demo it works very well?
For the age prediction which model did you used? a pre-trained network?
For mood prediction, did you used deep leaning model or just it takes the differences between nodes around eyebrow, lips and eyes?

Thank you

But which one did you use in your model? If you used the second one why it is not robust and the facemesh disappears suddenly when I change my head pose extremely?

it's a tradeoff between precision and performance and it's fully tunable in configuration.
how it works is that library tries to re-use previously detected face bounding box without running it again and just runs the mesh and iris detector instead. so if the face moves suddenly, face will disappear and bounding box gets invalidated so it re-runs on the next frame. again, behavior is fully tunable.
also disabled by default for performance reasons is face angle calculation so if head tilts more than ~20 degrees, it will get invalidated. also can be enabled via configuration.

hand tracking is not working for both hands. Is there a reason for that since in the mediapipe's demo it works very well?

yes, it does, you just have to set configuration parameter hand.maxHands to a value higher than 1. again, it's about performance vs precision.

For the age prediction which model did you used? a pre-trained network?

Pre-trained network, trained on imdb dataset.

For mood prediction, did you used deep leaning model or just it takes the differences between nodes around eyebrow, lips and eyes?

pre-trained network, not just simple math.
math is used to calculate gestures such as mouth open, raise left hand, looking down, looking at camera, etc..
list of all models used are in credits page: https://github.com/vladmandic/human/wiki/Models
list of pre-defined math-based gestures that are included (others can be added by user) is in https://github.com/vladmandic/human/wiki/Gesture

But which one did you use in your model? If you used the second one why it is not robust and the facemesh disappears suddenly when I change my head pose extremely?

it's a tradeoff between precision and performance and it's fully tunable in configuration.
how it works is that library tries to re-use previously detected face bounding box without running it again and just runs the mesh and iris detector instead. so if the face moves suddenly, face will disappear and bounding box gets invalidated so it re-runs on the next frame. again, behavior is fully tunable.
also disabled by default for performance reasons is face angle calculation so if head tilts more than ~20 degrees, it will get invalidated. also can be enabled via configuration.

hand tracking is not working for both hands. Is there a reason for that since in the mediapipe's demo it works very well?

yes, it does, you just have to set configuration parameter hand.maxHands to a value higher than 1. again, it's about performance vs precision.

For the age prediction which model did you used? a pre-trained network?

Pre-trained network, trained on imdb dataset.

For mood prediction, did you used deep leaning model or just it takes the differences between nodes around eyebrow, lips and eyes?

pre-trained network, not just simple math.
math is used to calculate gestures such as mouth open, raise left hand, looking down, looking at camera, etc..
list of all models used are in credits page: https://github.com/vladmandic/human/wiki/Models
list of pre-defined math-based gestures that are included (others can be added by user) is in https://github.com/vladmandic/human/wiki/Gesture

Thank you very much for the detailed answer.
I still have lots of questions about mediapipe also. I don't want to bother you, so is there any forum for asking these questions?

Thank you very much for the detailed answer.
I still have lots of questions about mediapipe also. I don't want to bother you, so is there any forum for asking these questions?

for mediapipe, author state that general questions should be posted via https://stackoverflow.com/questions/tagged/mediapipe

for mediapipe, author state that general questions should be posted via https://stackoverflow.com/questions/tagged/mediapipe

closing the issue as the original question is answered.

closing the issue as the original question is answered.

Hello Vladimir,
I am sorry for asking alot, I am amateur but very curious about this library.
What do you mean by performance? precision is also a concept of performance. you mean if I make the model to act more accurate, the performance will decrease? but what is performance here? is it the speed of algorithm?
mediapipe (the new version) also does the face detection/bounding box/mesh/iris jobs as well but why it doesn't have lag, shaking and disappearing even when I rotate my head more that 30 or 40 degrees? in its demo I don't tune anything and it works very well.
For the age prediction have you done any fine tuning over the pre-trained network or any other changes?
For emotion detection, why you didn't use simple math (calculating difference in distances between eyebrow etc.) what will be its deficiency or problem?
If simple math won't work for emotion detection correctly, would it work for mouth opening and gaze direction?!!
Thx
@vladmandic

Hello Vladimir,
I am sorry for asking alot, I am amateur but very curious about this library.
What do you mean by performance? precision is also a concept of performance. you mean if I make the model to act more accurate, the performance will decrease? but what is performance here? is it the speed of algorithm?
mediapipe (the new version) also does the face detection/bounding box/mesh/iris jobs as well but why it doesn't have lag, shaking and disappearing even when I rotate my head more that 30 or 40 degrees? in its demo I don't tune anything and it works very well.
For the age prediction have you done any fine tuning over the pre-trained network or any other changes?
For emotion detection, why you didn't use simple math (calculating difference in distances between eyebrow etc.) what will be its deficiency or problem?
If simple math won't work for emotion detection correctly, would it work for mouth opening and gaze direction?!!
Thx
@vladmandic

you mean if I make the model to act more accurate, the performance will decrease? but what is performance here? is it the speed of algorithm?

yes, if you make model to act more accurately it will decrease performanse. this can be due to many things.
for example (and there are other examples as well):

input resolution: higher resolution, better accuracy but more data has to be processed by model so it's slower.
pre&post processing: for example, after face detect model returns bounding box, there is additional math to calculate the rotation angle of the face and rotate the image within bounding box before passing it to face mesh model. but that rotating image is computationally intensive, so it's optional - it increases precision, but decreases performance.
how often are some steps skipped: i've mentioned that face detect does not run on every frame by default, my library tries to re-use previously detected bounding box to skip re-running that model before passing bounding box to face mesh. but if face moves too fast, it will get out of frame. optimally, you can run face detect on each frame, but it's slower.

mediapipe (the new version) also does the face detection/bounding box/mesh/iris jobs as well but why it doesn't have lag, shaking and disappearing even when I rotate my head more that 30 or 40 degrees?

new version of mediapipe is written in C++ and compiled to WASM. so entire pre and post processing that i've mentioned above is much faster than in JavaScript.
also, JavaScript is single threaded, so any calculation will block processing - thus more lag. mediapipe can perform asynchronous interpolations between model runs. meaning, it runs the model prediction, but also calculates averages between each run so it can draw much smoother frames.
i have added async processing in human library as well, but due to overal nature of JavaScript, it doesn't help much.
in reality, mediapipe is a superior way of doing it, but it's proprietary - you can only do what they allow you and not much else, not room for user modifications/improvements/tunings.
also, mediapipe models are pre-compiled to tflite format and cannot be easily re-used elsewhere.

For the age prediction have you done any fine tuning over the pre-trained network or any other changes?

no additional fine tuning.

For emotion detection, why you didn't use simple math (calculating difference in distances between eyebrow etc.) what will be its deficiency or problem?

it's possible, but emotions are complex - it's easy to differentiate between angry vs happy, but not so easy to mathematically describe surprise vs digust. i just found using model to be easier and more reliable.

If simple math won't work for emotion detection correctly, would it work for mouth opening and gaze direction?!!

it does work for anything you can clearly describe mathematically. so things like 'mouth open' or 'looking left' or 'leaning down' are easy.

you mean if I make the model to act more accurate, the performance will decrease? but what is performance here? is it the speed of algorithm?

yes, if you make model to act more accurately it will decrease performanse. this can be due to many things.
for example (and there are other examples as well):

input resolution: higher resolution, better accuracy but more data has to be processed by model so it's slower.
pre&post processing: for example, after face detect model returns bounding box, there is additional math to calculate the rotation angle of the face and rotate the image within bounding box before passing it to face mesh model. but that rotating image is computationally intensive, so it's optional - it increases precision, but decreases performance.
how often are some steps skipped: i've mentioned that face detect does not run on every frame by default, my library tries to re-use previously detected bounding box to skip re-running that model before passing bounding box to face mesh. but if face moves too fast, it will get out of frame. optimally, you can run face detect on each frame, but it's slower.

mediapipe (the new version) also does the face detection/bounding box/mesh/iris jobs as well but why it doesn't have lag, shaking and disappearing even when I rotate my head more that 30 or 40 degrees?

new version of mediapipe is written in C++ and compiled to WASM. so entire pre and post processing that i've mentioned above is much faster than in JavaScript.
also, JavaScript is single threaded, so any calculation will block processing - thus more lag. mediapipe can perform asynchronous interpolations between model runs. meaning, it runs the model prediction, but also calculates averages between each run so it can draw much smoother frames.
i have added async processing in human library as well, but due to overal nature of JavaScript, it doesn't help much.
in reality, mediapipe is a superior way of doing it, but it's proprietary - you can only do what they allow you and not much else, not room for user modifications/improvements/tunings.
also, mediapipe models are pre-compiled to tflite format and cannot be easily re-used elsewhere.

For the age prediction have you done any fine tuning over the pre-trained network or any other changes?

no additional fine tuning.

For emotion detection, why you didn't use simple math (calculating difference in distances between eyebrow etc.) what will be its deficiency or problem?

it's possible, but emotions are complex - it's easy to differentiate between angry vs happy, but not so easy to mathematically describe surprise vs digust. i just found using model to be easier and more reliable.

If simple math won't work for emotion detection correctly, would it work for mouth opening and gaze direction?!!

it does work for anything you can clearly describe mathematically. so things like 'mouth open' or 'looking left' or 'leaning down' are easy.

ok, thank you
Is this library (Human) available for both academic and industry use? I know it is open-source but is it for both purpose freely?

ok, thank you
Is this library (Human) available for both academic and industry use? I know it is open-source but is it for both purpose freely?

Re: license
Library is released under MIT license.
Some models are originally under Apache 2.0 license.
Both are permissive - so in short, yes.

Re: license
Library is released under MIT license.
Some models are originally under Apache 2.0 license.
Both are permissive - so in short, yes.

I really appreciate your detailed answer.
In the Human library is there any black box module? can I use/change/modify all the functions and modules?

I really appreciate your detailed answer.
In the Human library is there any black box module? can I use/change/modify all the functions and modules?

In terms of functionality, human/TFJS solution is obviously better than the current MediaPipe JS soltuion. The current MediaPipe JS solution is quite buggy. It doesn't work on web worker. It is not well documented and sometimes you have to do trial-n-error just to make things work. Its output is bugged. For example, MediaPipe JS facemesh output doesn't give the z coordinates for some unknown reasons.

In terms of functionality, human/TFJS solution is obviously better than the current MediaPipe JS soltuion. The current MediaPipe JS solution is quite buggy. It doesn't work on web worker. It is not well documented and sometimes you have to do trial-n-error just to make things work. Its output is bugged. For example, MediaPipe JS facemesh output doesn't give the z coordinates for some unknown reasons.

@lghasemzadeh other than the fact that included models are pre-trained, there is nothing you cannot change/modify.
and if you want to add additional modules or swap out existing ones, that is also pretty easy as design is modular.

@lghasemzadeh other than the fact that included models are pre-trained, there is nothing you cannot change/modify.
and if you want to add additional modules or swap out existing ones, that is also pretty easy as design is modular.

Hello Vladimir,
Hope you are well.

It seems that the Iris direction (Gaze Direction) is not working, it always print 'Looking at Camera' while I am completely looking at Left or Right. Is there any way to correct or improve it?
The head pose works very well but there are two face description on the screen, one for left and right, and another for up and down. what is the reason to separate them?

Thank you

@lghasemzadeh

Hope you are well.

Yes, thank you! :-)
Btw, can you open a new issue for this? Both questions are valid, but unrelated to this issue that is already closed.

It seems that the Iris direction (Gaze Direction) is not working, it always print 'Looking at Camera' while I am completely looking at Left or Right. Is there any way to correct or improve it?

I just tried using webcam and it works for me.
Can you send some sample images where its misdetected along with details of your configuration?
Note that mathematically 'looking at camera' is defined as difference between area size of left and right iris and if they are within 25% (see src/gestures/gestures.ts:iris(), then human will return looking at camera. This threshold can be changed from 25% to anything, but it's not exposed as user-configurable value as I don't want to have too complex of configuration file (it's already pretty complex), so I just use some value that looked ok.

The head pose works very well but there are two face description on the screen, one for left and right, and another for up and down. what is the reason to separate them?

Face can be up/center/down and left/center/right at the same time, so gestures returns two separate results.
But gestures return object is just that - an object - and If you want to combine them, it's just a question of printing them together instead separately.

Ok, I will open a new issue


[bug] tauri-build build script always rebuild if tauri.conf.json is missing

Describe the bug
When only using Tauri.toml for the config file, the build script always rebuilds the final executable even nothing is changed. This is due to the usage of  unconditional cargo:rerun-if-changed=tauri.conf.json in the build function. Since there is no tauri.conf.json file, and rerun-if-changed considers missing file the same as "stale file", it always triggers the rerun.
Removing the unconditional println!("cargo:rerun-if-changed=tauri.conf.json") line, or placing a duplicate tauri.conf.json in the project bypasses the problem.
Reproduction
Simply switch to use the Tauri.toml file (and of course the config-toml feature). Build the project, then build it again without changing anything.
Expected behavior
There should be no rebuild triggering.
Full tauri info output
[✔] Environment
    - OS: Windows 10.0.22631 X64
    ✔ WebView2: 121.0.2277.83
    ✔ MSVC: Visual Studio Build Tools 2022
    ✔ rustc: 1.75.0 (82e1608df 2023-12-21)
    ✔ cargo: 1.75.0 (1d8b05cdd 2023-11-20)
    ✔ rustup: 1.26.0 (5af9b9484 2023-04-05)
    ✔ Rust toolchain: stable-x86_64-pc-windows-msvc (environment override by RUSTUP_TOOLCHAIN)
    - node: 20.11.0
    - pnpm: 8.15.1
    - npm: 10.2.4

[-] Packages
    - tauri-cli [RUST]: 2.0.0-alpha.21

Stack trace
No response
Additional context
I think the default tauri.conf.json should have its own feature, just like config-toml, and make it default in Cargo. Then we can conditionally include the rerun line with a #[cfg(feature)].

In my opinion, adding an empty tauri.conf.json file at the build time would be a better option, because tauri.conf.json is the default config file, and the cargo.toml or tauri.conf.json5 are just additional features.
So, if a user deletes the tauri.conf.json file.
And thanks for reporting the bug btw.

In principle, I really dislike ambiguous logic/environment/etc, because it causes confusion. 3 months later when I revisit this project and see both tauri.conf.json and Tauri.toml, especially the former being {} (but can't be completely empty because then the build complains unable to parse JSON Tauri config file at D:\Development\Rust\Aoi\src-tauri\tauri.conf.json because EOF while parsing a value at line 1 column 0 for some reason), I will force myself to spend some time recollect the memory we are having here. I will have to ask myself why there are two config files where one is completely unused just for arbitrary Tauri rule that can be found nowhere else.
But if you guys really want to keep the empty tauri.conf.json, please document this behavior in https://tauri.app/v1/references/configuration-files. There is currently no mentioning that by switching the non-default option and deleting the old file, it causes side-effect.

In principle, I really dislike ambiguous logic/environment/etc, because it causes confusion. 3 months later when I revisit this project and see both tauri.conf.json and Tauri.toml, especially the former being {} (but can't be completely empty because then the build complains unable to parse JSON Tauri config file at <path>\src-tauri\tauri.conf.json because EOF while parsing a value at line 1 column 0 for some reason), I will force myself to spend some time recollect the memory we are having here. I will have to ask myself why there are two config files where one is completely unused just for arbitrary Tauri rule that can be found nowhere else.
But if you guys really want to keep the empty tauri.conf.json, please document this behavior in https://tauri.app/v1/references/configuration-files. There is currently no mentioning that by switching the non-default option and deleting the old file, it causes side-effect.

But if you guys really want to keep the empty tauri.conf.json, please document this behavior in https://tauri.app/v1/references/configuration-files. There is currently no mentioning that by switching the non-default option and deleting the old file, it causes side-effect

Yeah, because that's a bug :sweat_smile:
Don't worry, we will fix it ASAP and consider all the odds.
Thank You

I think the default tauri.conf.json should have its own feature, just like config-toml, and make it default in Cargo. Then we can conditionally include the rerun line with a #[cfg(feature)].

I agree, this should be the solution

Actually, according to my test, if tauri.conf.json exists, Tauri will completely ignore the config-toml feature, and always use tauri.conf.json. So having an empty tauri.conf.json won't work. It needs to be an exact duplicate at the moment.
I see you are making https://github.com/tauri-apps/tauri/pull/8732. Thank you so much! However, I do not know where in tauri-build is the logic that the presence of tauri.conf.json overrides everything. It might be a potential source of bug. Just FYI.


can it support rtx 4090 (24 gb) ?

How many Mistral 7b adapters can be supported on a rtx 4090 or single T4? Will be good to have such benchmark.


Issue generating Appwrite database models using flutter extension

👟 Reproduction steps
Using flutter extension from feat-flutter branch
when i use the cli command
appwrite flutter generate
👍 Expected behavior
generate model files with no errors
👎 Actual Behavior
When i use the cli command
appwrite flutter generate
i get the following error:
no such file or directory, open '/usr/local/lib/node_modules/appwrite-cli/lib/commands/../../flutter_res/template_model.dart'
i tried
appwrite flutter generate --vebose and i got the following logs:
Error: ENOENT: no such file or directory, open '/usr/local/lib/node_modules/appwrite-cli/lib/commands/../../flutter_res/template_model.dart'     at Object.openSync (node:fs:601:3)     at Object.readFileSync (node:fs:469:35)     at generate (/usr/local/lib/node_modules/appwrite-cli/lib/commands/flutter.js:69:31)     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {   errno: -2,   syscall: 'open',   code: 'ENOENT',   path: '/usr/local/lib/node_modules/appwrite-cli/lib/commands/../../flutter_res/template_model.dart' }
after further investigation it seems like the file exists in /generator/template_dart.dart and it does work when the path points to it in /usr/local/lib/node_modules/appwrite-cli/lib/commands/flutter.js:69:31
🎲 Appwrite version
Version 1.3.x
💻 Operating system
MacOS
🧱 Your Environment
AWS EC2 amazon linux 2023
👀 Have you spent some time to check if this issue has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Hey, we are working on generator for multiple language support so, instead of using appwrite flutter generate please use appwrite generate dart


906 start end date validation

Description
Talent start and end date validation is done with this PR.
An onchange() function is implemented in #start_day which sets the min value of #end_day to #start_day.value with setAttribute() method.
Two javascript functions are added - one at the bottom with all js functions and one inline with html edit page.
Fixes #906
Review Time Estimate
Please give your idea of how soon this pull request needs to be reviewed by selecting one of the options below. This can be based on the criticality of the issue at hand and/or other relevant factors.

[ ] Immediately
[x] Within a week
[ ] When possible

Type of changes
Please select a relevant option:

[ ] Bug fix (non-breaking change which fixes an issue).
[x] New feature (non-breaking change which adds functionality).
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected).
[ ] Other (any another change that does not fall in one of the above categories.)

Checklist:
Please select all applicable options:

[ ] I have signed the Rokwire Contributor License Agreement (CLA). (Any contributor who is not an employee of the University of Illinois whose official duties include contributing to the Rokwire software, or who is not paid by the Rokwire project, needs to sign the CLA before their contribution can be accepted.)
[x] I have updated the CHANGELOG.
[ ] I have read the Contributor Guidelines.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[ ] My change requires updating the documentation.
[ ] I have made necessary changes to the documentation.
[ ] I have added tests related to my changes.
[ ] My changes generate no new warnings.
[ ] New and existing unit tests pass locally with my changes.
[ ] Any dependent changes have been merged and published in downstream modules.

@minump, please fix the merge conflicts when you get a chance. The CHANGELOG entry for this PR needs to go at the end of the Added section. Thanks.

Hi, @minump,
Add Talent/Capability button is submitting the Contribution instead of displaying relevant form entries. I'm not sure if something got messed up during the merge process. Please look into this when you get a chance. Thanks.

Converting to draft as PR#924 is created for the same.

Closing this #924 is merged. Thanks.


Clicking a node highlights its nonmonophylic pair

This fixes #42 by refactoring what clicking does. The old code was throwing an error because it was attempting to get input from an html input field that no longer exist.
The new behavior accepts clicks on any leaf node, and will pop up an alert telling you its pair. It will also set all other nodes except for the pair to opacity 0.2 so that it is visually easy to find the pair. Clicking again will reset everything to opacity 1 (clicking on a blue dot will also set opacity to 1).
I would personally prefer to chuck the alert and just have it toggle the opacity, but I'm keeping it for now in case the biologists' use case is more about just getting the number than seeing it in the tree.

Should be all good now. Tested and it still works!


bug fixes

One is for duplicated posts display in LatestPostsPlugin.
Another one is for the author selector in admin for AuthorPlugin.

This breaks, because if an author have say 20 published posts, his name will show up in the admin list 20 times. Also it makes it possible to choose an author who has not yes published anything but will do in the future without having to change the plugin instance. The we can override the template instead to include a conditional on whether or not to show authors that have not published anything yet.

For instance like so:
{% for author in authors_list %}
	    {% if author.get_full_name %}
		    {% if author.count > 0 %}

		      {{ author.get_full_name }}
		      (
		        {% if author.count > 0 %}
		        {% blocktrans count articles=author.count %}1 news story{% plural %}{{ articles }} news stories{% endblocktrans %}
		        {% else %}{% trans "0 news stories" %}{% endif %}
		        )

		     {% endif %}
	     {% endif %}
	      {% empty %}

{% trans "No news stories found." %}

      {% endfor %}

@zoe-stockholm @smaaland still interested in having this in?


BITMAKER-2542 estela web: DataPersistence field on Project/Spider models

Description
estela web: Define field DataPersistence on Project/Spider models
Issue
https://tasks.bitmaker.dev/issues/2542?tab=time_entries
Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] My code follows the style guidelines of this project.
[ ] I have made corresponding changes to the documentation.
[ ] New and existing tests pass locally with my changes.
[ ] If this change is a core feature, I have added thorough tests.
[ ] If this change affects or depends on the behavior of other estela repositories, I have created pull requests with the relevant changes in the affected repositories. Please, refer to our official documentation.
[ ] I understand that my pull request may be closed if it becomes obvious or I did not perform all of the steps above.

@Adriana618 This PR is ready for Review
Issue: https://tasks.bitmaker.dev/issues/2542


pip install carla - ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I try to use pip install carla and I get this error :
Collecting carla
Using cached carla-0.9.5.tar.gz (3.9 kB)
ERROR: Command errored out with exit status 1:
command: 'c:\users\f67886\appdata\local\programs\python\python37\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\F67886\AppData\Local\Temp\pip-install-9x5wm3pa\carla\setup.py'"'"'; file='"'"'C:\Users\F67886\AppData\Local\Temp\pip-install-9x5wm3pa\carla\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\F67886\AppData\Local\Temp\pip-pip-egg-info-6e19xs3q'
cwd: C:\Users\F67886\AppData\Local\Temp\pip-install-9x5wm3pa\carla
Complete output (7 lines):
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\F67886\AppData\Local\Temp\pip-install-9x5wm3pa\carla\setup.py", line 114, in
ext_modules=get_libcarla_extensions(),
File "C:\Users\F67886\AppData\Local\Temp\pip-install-9x5wm3pa\carla\setup.py", line 75, in get_libcarla_extensions
libs = [x for x in os.listdir('dependencies/lib') if any(d in x for d in required_libs)]
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'dependencies/lib'
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
I'm trying to do this because there is no "carla" folder in "\ Python \ Python37 \ Lib \ site-packages" and I think that's why when I run a matlab script it can't connect to the carla server and I get this error: "Unable to resolve the name py.carla.Client. "
There is someone who has encountered this problem ?

HI @craciunescumarian  i faced th same problem can u help me to resolve it if u already find a solution
thanks in advance


[FEATURE REQUEST] Support custom scheme field values in server object.

Is your feature request related to a problem? Please describe.
Server Object has a scheme field (which will be renamed to protocol in version 2) which supports only limited amount of protocols. The problem is that there are custom protocols exist which are not covered by specification. In case of custom protocol defined it is impossible to generate API html or any other view because of error.
Can't it be tackled using specification extensions?
scheme field is a mandatory field in specification and can not be replaced with extension.
Describe the solution you'd like
The solution is to make scheme field required in specification but not limited to specific list of values. It will allow to define custom protocols in API.
Also, it will "release" specification from implementation details. Current implementation requires to create new specification version with every new supported protocol.
Describe alternatives you've considered
I have not found any alternative solutions.
Additional context
We are using custom event bus built on top of another popular open source event bus solution. The custom solution have a custom protocol implemented. We would like to use Async API specification to describe service API. Unfortunately, it is not possible due to scheme field limitation.

Hey @vadeg! It's great to see you came to the same conclusion as us. It's already implemented in version 2.0.0-rc1. Check it out: https://www.asyncapi.com/docs/specifications/2.0.0-rc1/#serverObject.
We're close to releasing a new version of the AsyncAPI generator that supports AsyncAPI 2.0.0-rc1. This will allow you to generate documentation and code.
Would you like to participate in the beta test? That would be very helpful for us.

Hi @fmvilas .
Good news!
Yes, I would like to participate in the beta test.

Awesome! Can you please send me an email to fmvilas at gmail.com? It's just so that I can send the details whenever the beta is ready for sharing.


[fix#12170] If the worker's clock early than master node, will ignore dispatch time.

Purpose of the pull request
fix #12170
Brief change log

Add a validotor, if the clock of worker node early than the master node, just ignore the dispatch time.
Added requirement for clock synchronization in doc.

Verify this pull request
This change added tests

Codecov Report

Merging #12219 (7a84691) into dev (1a4d7a8) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                dev   #12219      +/-   ##
============================================
- Coverage     39.44%   39.42%   -0.03%
+ Complexity     4142     4138       -4
============================================
  Files          1011     1011
  Lines         37845    37815      -30
  Branches       4345     4333      -12
============================================
- Hits          14929    14908      -21
+ Misses        21185    21175      -10
- Partials       1731     1732       +1

Impacted Files
Coverage Δ

...pache/dolphinscheduler/common/utils/DateUtils.java
76.07% <100.00%> (+1.69%)
:arrow_up:

...eduler/server/worker/task/WorkerHeartBeatTask.java
69.38% <0.00%> (-8.17%)
:arrow_down:

...erver/master/processor/queue/TaskEventService.java
75.00% <0.00%> (-5.36%)
:arrow_down:

...e/dolphinscheduler/remote/NettyRemotingClient.java
50.00% <0.00%> (-2.86%)
:arrow_down:

...rver/master/runner/task/BlockingTaskProcessor.java
75.86% <0.00%> (-0.55%)
:arrow_down:

...heduler/api/service/impl/ResourcesServiceImpl.java
51.68% <0.00%> (-0.47%)
:arrow_down:

.../dolphinscheduler/plugin/task/spark/SparkTask.java
72.56% <0.00%> (-0.25%)
:arrow_down:

...eduler/api/dto/workflow/WorkflowCreateRequest.java
100.00% <0.00%> (ø)

...eduler/api/service/impl/DataSourceServiceImpl.java
30.06% <0.00%> (ø)

...cheduler/server/master/utils/DependentExecute.java
0.00% <0.00%> (ø)

... and 13 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@EricGao888 @caishunfeng could u help to review this PR? thx

This should also be released in 3.1.1, 3.0.2.


Reuse floating windows [work-in-progress]

[work-in-progress]
This update results in floating windows being reused where possible (e.g., when scrolling).
However, as is, this currently results in scrollbar artifacts in some scenarios. For example, here's a screenshot after performing a vertical then horizontal split. The artifacts appear after the second split. They disappear when pressing <ctrl-l> or entering the :mode command.
Example artifacts (circled)

No artifacts after <ctrl-l>

Closing this for now, and will possibly revisit in the future.

The artifacts may be related to the issue reported in https://github.com/neovim/neovim/issues/13991, where drawing artifacts arise and can be cleared with <c-l>.

The changes from this PR were incorporated in d15b750c584b69f33071239d76ce624c5fe28dc2.


remove the log line for OTLP Metrics (too noisy)

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area config

/area outputs

/area tests

What this PR does / why we need it:
remove noisy log lines from OTLP Metrics (even when not set)
2024/12/03 19:59:27 [INFO]  : OTLP Metrics - OK
2024/12/03 19:59:27 [INFO]  : OTLP Metrics - OK
2024/12/03 19:59:27 [INFO]  : OTLP Metrics - OK
2024/12/03 19:59:27 [INFO]  : OTLP Metrics - OK
2024/12/03 20:00:39 [INFO]  : OTLP Metrics - OK
2024/12/03 20:00:39 [INFO]  : OTLP Metrics - OK
2024/12/03 20:00:39 [INFO]  : OTLP Metrics - OK
2024/12/03 20:00:39 [INFO]  : OTLP Metrics - OK
2024/12/03 20:01:01 [INFO]  : OTLP Metrics - OK
2024/12/03 20:01:01 [INFO]  : OTLP Metrics - OK

Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Issif
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Issif]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: cpanato, Issif
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Issif,cpanato]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 313f395dd8987fefbe27527ee1dc7fe91982a828


Time-to-live (TTL) for Functions

Does Time-to-live (TTL) for Functions of roadmap feature mean that to create FunctionTtlConfig supports automatically dropping function instances that have not been messaged for a defined amount of time, the way like StateTtlConfig of Flink, by default using RocksdbCompactFilterCleanupStrategy?

It means that, once enabled, function state that was not accessed would be TTLed. Can you share a little about your use case for function TTL?

@igalshilman I haven't used the function TTL which includes in roadmap features. I just want to implement the TTL for functions feature. Therefore, I have confusion about this feature description.


constructors of classes derived from VersionCreateRequest and VersionLoadRequest broken

The commit 80b249b introduced this change to the end of the constructor of SingleEntityVersionCreateRequest and ComplexVersionCreateRequest:
-        VersionCreateRequest.__init__(self, branch, type, version_name)
+        VersionCreateRequest.__init__(self, *args, **kwargs)

This is at the end of the constructors of these classes. The first thing the superclass constructor does here is to set _branch, _type, and _version_name to None, and since these are not in args or in kwargs, they don't get set to the values they are meant to be (this is what the subclass constructor did just a few lines earlier).
The same problem is also in the VersionLoadRequest superclass and its derived classes.

This is quite possibly a problem in the swagger definition that spring exports for thingsboard. I have not seen anything that explicitly speaks against it, but this doesn't seem correct:
      "ComplexVersionCreateRequest": {
        "title": "ComplexVersionCreateRequest",
        "properties": {
          "branch": {...},
          "entityTypes": {...},
          "syncStrategy": {...},
          "type": {...},
          "versionName": {...}
        },
        "allOf": [
          {
            "$ref": "#/components/schemas/VersionCreateRequest"
          },...],
... }, ...
      "VersionCreateRequest": {
        "title": "VersionCreateRequest",
        "type": "object",
        "properties": {
          "branch": {...},
          "type": {...},
          "versionName": {...}
        }},

Is it okay to have the branch, type, and versionName properties duplicated in the parent and child classes? At least the swagger api spec doesn't have it like that in their examples.

a random find, SmsTwoFaAccountConfig also has this problem with the use_by_default parameter, which is probably much harder to actually notice is not working

AlarmNotificationRuleTriggerConfig as well. It's probably a good idea to just go through every mention of "allOf" in the swagger definition json and check the constructors for this pattern

Perhaps you are using an older version of https://github.com/swagger-api/swagger-codegen/pull/11968 which at least now passes all the arguments to the superclass constructor and that should fix this issue


[Bug report] Default theme broken in width 719

Description
Home Page Sidebar Broken in 720

Navbar broken
The navbar behavior is incorrect in 720

Search broken
The search button is actually expanded in 720, instead of the actually layout change in 719
Reproduction
https://v2.vuepress.vuejs.org/
Used Package Manager
pnpm
System Info
Newest version

In V1 the breakpoint was 719 / 720 change, instead of 720 / 721 change, so I will add these changes:

fix search plugin breakpoint to 719 / 720
fix 720 homepage sidebar issue
fix navbar 720 issue.

@meteorlxy  cc:


[Question]: MSBuild Target Build Swagger CLI with ConfigurationProvider

What are you wanting to achieve?
Hello guys,
I'm trying to generate the documentation through the Swagger CLI, but my application has a call to the AWS Secret Manager. Therefore, when I'm building the application, the Swagger ToFile returns an error when querying the AWS service. Is there a way to generate the document without necessarily running the application?
What code or approach do you have so far?
Project csproj
<Target Name="ExportSwagger" AfterTargets="AfterBuild">
        <Exec Command="dotnet tool restore" />
        <Exec Command="dotnet swagger tofile --output .\swagger.json $(TargetPath) v1" EnvironmentVariables="DOTNET_ROLL_FORWARD=LatestMajor" />
</Target>

Output error:
  Done executing task "Exec".
  Task "Exec"
    Environment Variables passed to tool:
      DOTNET_ROLL_FORWARD=LatestMajor
    dotnet swagger tofile --output .\swagger.json ..\...\API.dll v1
    Unhandled exception. System.AggregateException: One or more errors occurred. (The security token included in the request is expired)
     ---> Amazon.SecretsManager.AmazonSecretsManagerException: The security token included in the request is expired
     ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
       at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
       at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
       at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
       at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
       at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)

My Program.cs, I'm placing the call to the aws secret manager as an IConfigurationSource
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.Host.AddSecretsManager("CampaignManagerSecrets"); <= Here

builder.Services
    .AddInternalService(builder.Configuration);

Additional context
No response

Is there a way to generate the document without necessarily running the application?

There is not - this is fundamentally how OpenAPI generation works. The application is run so that all your filters, versioning, etc. is applied to the document. Only reflection isn't sufficient.
You'll need to make your application "generator aware" so that it doesn't use/need the bits that won't work outside of your real deployed application when the OpenAPI document is generated.
Possible solutions:

Run with a specific environment name instead of Development/Production
Run with a specific environment variable defined that indicates the same (e.g. OPENAPI_GENERATOR=true)
Check the name of the entrypoint assembly to see if it's the generator

Then given which way you want to check for "is generator", conditionally not include SSM as a configuration source.

tks @martincostello


Increased status timeout for LinuxFilesystemIsolatorTest.ROOT_Recover…

…OrphanedPersistentVolume
Fix the following issue when doing make check on RedHat7:
../../src/tests/containerizer/filesystem_isolator_tests.cpp:697: Failure
Failed to wait 15secs for status
../../src/tests/containerizer/filesystem_isolator_tests.cpp:685: Failure
Actual function call count doesn't match EXPECT_CALL(sched, statusUpdate(&driver, _))...
Expected: to be called at least once
Actual: never called - unsatisfied and active

Hi @Djidiouf , thanks for this PR. This is a known issue. Currently we have some tests in LinuxFilesystemIsoaltorTest are flaky. We used to trying fix them by increasing the await time from 15 sec to 60 sec, but it the tests are still flaky, because that may not be the root cause for this failure.
We usually track on mesos issues in apache mesos JIRA (https://issues.apache.org/jira/browse/MESOS/), which makes it easier to classify diff types and priority of  any issue. If you are interested in working on any issue, please follow this document and submit patches to apache mesos reviewboard. :)

Closing per request at https://s.apache.org/V8r3


.NET integration follow-up considerations

Here is an issue regarding a recent integration effort into the .NET SDK & providing SBOMs for NuGet packages:

[ ] Investigate a passthrough for future command line args.
[ ] Provide response file support.
[ ] Check if file paths are normalized.
[ ] Ensure logging is localized.

https://github.com/gustavoaca1997/dotnet-sdk/pull/1
https://github.com/microsoft/sbom-tool/pull/674

We hear you loud and clear about publishable projects - the first iteration of this integration in .NET for 9.0.100 is aimed squarely at NuGet packages, but we hope to follow in a subsequent release for published applications. This is great feedback to hear.
@KalleOlaviNiemitalo thank you for kicking the tires! I generally agree with everything you've said here (and especially thank you for trying it in VS).

Microsoft.Sbom.Targets 2.2.8 doesn't find the names of referenced NuGet packages when I use it with .NET SDK 8.0.304 in a project that specifies artifacts output layout in Directory.Build.props:
  <PropertyGroup>
    <UseArtifactsOutput>true</UseArtifactsOutput>
    <ArtifactsPath>$(MSBuildThisFileDirectory).artifacts</ArtifactsPath>
  </PropertyGroup>

Specifically, the NuGet and NuGetProjectCentric component detectors do not detect any components in this case.
It seems this can be fixed by setting <SbomGenerationBuildComponentPath>$(BaseIntermediateOutputPath)</SbomGenerationBuildComponentPath>, but I'm not sure whether that could break something else.

I think this is a single instance of a general problem - all of the properties here should be lazily computed when the SBOM generation task runs instead of being 'pinned' when the targets file is read. This lets user logic/packages/etc influence those properties safely.

My org has a number of packages with metadata configured along the lines of
<Company>Acme Inc</Company>
<Authors>Team Name</Authors>

In my view it would make more sense to base SbomGenerationPackageSupplier on the Company field than the Authors.  (or at least to prefer Company over Authors.  This could also provide a bit more consistency when support is implemented for publishing as Authors may not be defined on non packable projects (as it doesn't do anything outside of a nuget package)
https://github.com/microsoft/sbom-tool/blob/08ba73d303228eb4d92a6a5f75350d78230bca30/src/Microsoft.Sbom.Targets/Microsoft.Sbom.Targets.targets#L21C51-L21C80

hi folks,
i have filed a handful of issues here to track various things so far. i would encourage y'all who feel strongly about certain topics to file issues you're passionate for and we can add them to this mini-epic to track for now. thanks again for all the feedback so far!


Remove extraneous line from generated markdown

What type of PR is this?
(REQUIRED)

[ ] bug
[x] cleanup
[ ] documentation
[ ] feature

What this PR does / why we need it:
(REQUIRED)

This PR removes the extraneous man-page line from markdown rendering. It does this by only including the line if SectionNum > 0
Which issue(s) this PR fixes:
(REQUIRED)

Fixes #1275
Special notes for your reviewer:
(fill-in or delete this section)

Testing
(fill-in or delete this section)

go test -v ./...
Release Notes
(REQUIRED)

None.

thanks a lot!


Fixed #29275 -- Print "(actual errors: none)" instead of "(actual errors: )" in assertFormError.

https://code.djangoproject.com/ticket/29275


[stable/keycloak] Bug report: Redirection loop when replicas > 1

I am using chart with Keycloak 4.5.0.Final which is provided by keycloak chart version 4.0.0. I am trying to deploy HA version on Istio enabled cluster (1.0.2). I managed to run one replica without any troubles but when I set 2 (>1) replicas I get a problem with infinite redirections. It looks like there may be some problem with Infinispan cluster? As I understand it is used for distributed session management?
# keycloak-0 instance
[0m06:15:58,460 WARN [org.keycloak.events] (default task-5) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=<IP_ADDRESS>, error=expired_code, restart_after_timeout=true

# keycloak-1 instance
[0m06:15:57,974 WARN [org.keycloak.events] (default task-3) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=<IP_ADDRESS>, error=expired_code, restart_after_timeout=true

I deployed istio Gateway & Virtual Service with HTTPS in front of service provided by keycloak Helm Chart.
Details
Chart maintainer: @unguiculus
K8s version: 1.10.7-gke.2
I use let's encrypt ssl certificate

Sticky sessions on networking.istio.io/v1alpha3/DestinationRule solved issue but is sticky session required for keycloak cluster @unguiculus?

They weren't in the 3.4.0 version of the chart (which uses 4.2.1.Final of Keycloak).  We only started seeing this behavior when we updated to 4.0.0.
We are actually seeing an issue with only one replica when using the following ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    app: keycloak
    chart: keycloak-4.0.0
    heritage: Tiller
    release: keycloak
  name: keycloak
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: keycloak-http
          servicePort: http
        path: /auth

This works fine using the 3.4.0 version of the chart, but in 4.0.0 we get an error when we go to the admin console -- "invalid parameter redirect_uri".  This is with a single replica.

To clarify, we get the above error when we attempt to navigate to the admin console using the link on the "home page" (the one you get via /auth).

I tried the latest version on Google Kubernetes Engine and found severe issues when replicas > 1.
(I haven't tried the previous version though. I will try it ASAP)
When replicas =1 everything is fine.
I had included all the details in my conversation here
https://github.com/helm/charts/pull/8192#issuecomment-429592731

To @sfitts point, I'm also having this issue with only one replica with an ingress controller with HTTPS.
It was working fine when using a simple service as a LoadBalancer.

Seems like #4066 is related.

Based on the keycloak docs, I think this has everything to do with the way the helm chart is configured or rather the way that keycloak understands host names / protocols, etc.
I did find a fix, but I thought I'd document my lengthy process, maybe to give hope to others.
https://www.keycloak.org/docs/latest/server_installation/index.html#verify-configuration says to go to https://keycloak.example.com/auth/realms/master/.well-known/openid-configuration in order to verify the configuration.
On my server, this looks like:
{
    "issuer": "http://keycloak.example.com/auth/realms/master",
    "authorization_endpoint": "http://keycloak.example.com/auth/realms/master/protocol/openid-connect/auth",
    "token_endpoint": "http://keycloak.example.com/auth/realms/master/protocol/openid-connect/token",
    "token_introspection_endpoint": "http://keycloak.example.com/auth/realms/master/protocol/openid-connect/token/introspect",
    "userinfo_endpoint": "http://keycloak.example.com/auth/realms/master/protocol/openid-connect/userinfo",
    "end_session_endpoint": "http://keycloak.example.com/auth/realms/master/protocol/openid-connect/logout",
    "jwks_uri": "http://keycloak.example.com/auth/realms/master/protocol/openid-connect/certs",
    "check_session_iframe": "http://keycloak.example.com/auth/realms/master/protocol/openid-connect/login-status-iframe.html",
    "grant_types_supported": ["authorization_code", "implicit", "refresh_token", "password", "client_credentials"],
    "response_types_supported": ["code", "none", "id_token", "token", "id_token token", "code id_token", "code token", "code id_token token"],
    "subject_types_supported": ["public", "pairwise"],
    "id_token_signing_alg_values_supported": ["ES384", "RS384", "HS256", "HS512", "ES256", "RS256", "HS384", "ES512", "RS512"],
    "userinfo_signing_alg_values_supported": ["ES384", "RS384", "HS256", "HS512", "ES256", "RS256", "HS384", "ES512", "RS512", "none"],
    "request_object_signing_alg_values_supported": ["none", "RS256"],
    "response_modes_supported": ["query", "fragment", "form_post"],
    "registration_endpoint": "http://keycloak.example.com/auth/realms/master/clients-registrations/openid-connect",
    "token_endpoint_auth_methods_supported": ["private_key_jwt", "client_secret_basic", "client_secret_post", "client_secret_jwt"],
    "token_endpoint_auth_signing_alg_values_supported": ["RS256"],
    "claims_supported": ["sub", "iss", "auth_time", "name", "given_name", "family_name", "preferred_username", "email"],
    "claim_types_supported": ["normal"],
    "claims_parameter_supported": false,
    "scopes_supported": ["openid", "phone", "address", "email", "profile", "offline_access"],
    "request_parameter_supported": true,
    "request_uri_parameter_supported": true,
    "code_challenge_methods_supported": ["plain", "S256"],
    "tls_client_certificate_bound_access_tokens": true,
    "introspection_endpoint": "http://keycloak.example.com/auth/realms/master/protocol/openid-connect/token/introspect"
}

You can see that there's only http:// values in there.
So what I seem I need to do is tell Keycloak to use HTTPS for all connections:
https://www.keycloak.org/docs/latest/server_installation/index.html#setting-up-https-ssl
But then here I see that I need to actually use the admin to turn on SSL for the admin: https://www.keycloak.org/docs/4.5/server_admin/#_ssl_modes
So, my approach to fixing it was to use kubectl port-forward svc/keycloak-http 8888:80 to browse to http://localhost:8888 on my local machine to turn on SSL.
But then I hit a dead end when I refreshed the page and discovered the error message: "HTTPS required".
I need some way of telling the environment variables that I already am behind HTTPS....
So then I discovered additional environment config inside the docker specifications: https://github.com/jboss-dockerfiles/keycloak/blob/master/server/README.md#enabling-proxy-address-forwarding
I discover PROXY_ADDRESS_FORWARDING, which seems promising.
Here's the config that worked form me:

## https://github.com/helm/charts/tree/master/stable/keycloak

keycloak:
  extraEnv: |
    - name: PROXY_ADDRESS_FORWARDING
      value: "true"

  persistence:
    deployPostgres: true
    dbVendor: postgres

  ## Ingress configuration.
  ## ref: https://kubernetes.io/docs/user-guide/ingress/
  ingress:
    enabled: true
    path: /

    annotations:
      kubernetes.io/ingress.class: nginx

    ## List of hosts for the ingress
    hosts:
      - keycloak.example.com

    ## TLS configuration
    tls:
      - hosts:
          - keycloak.example.com
        secretName: keycloak-ssl

The part that I had to enable specifically was:
keycloak:
  extraEnv: |
    - name: PROXY_ADDRESS_FORWARDING
      value: "true"

Now, another issue is that trying to helm upgrade breaks things, but I'll save that for tomorrow.
cc: @sfitts

@weisjohn Thanks for sharing that.
I got the single replica working with similar config as I mentioned here
https://github.com/helm/charts/pull/8192#issuecomment-429592731
Did you try with replicas > 1 ? If so, what's the behavior?
The ultimate goal is to have replication working for prod..

@vsomasvr I haven't tried with more than one replica yet.  I'm currently unable to scale the deployment at all right now.

FWIW, we have the 3.4.0 chart deployed in multiple clusters, fully replicated and behind multiple ingress definitions using the following values file:
keycloak:
  replicas: 3

  ## Username for the initial Keycloak admin user
  username: keycloak

  ## Password for the initial Keycloak admin user
  ## If not set, a random 10 characters password will be used
  password: ""

  ## Ingress configuration.
  ## ref: https://kubernetes.io/docs/user-guide/ingress/
  ## This is the default ingress -- realm specific access is created as part of the Vantiq deploy
  ingress:
    enabled: true
    path: /auth

    annotations:
       kubernetes.io/ingress.class: nginx

    ## List of hosts for the ingress
    hosts:
      - ""

    ## TLS configuration
    tls: []
    # - hosts:
    #     - keycloak.example.com
    #   secretName: tls-keycloak

  persistence:

    # Disable deployment of the PostgreSQL chart
    deployPostgres: false

    # The database vendor. Can be either "postgres", "mysql", "mariadb", or "h2"
    dbVendor: postgres

    ## The following values only apply if "deployPostgres" is set to "false"

    # Optionally specify an existing secret
    existingSecret: keycloakdb
    existingSecretKey: password

    dbHost: <ourDBHost>
    dbPort: 5432 # 5432 is PostgreSQL's default port. For MySQL it would be 3306
    dbName: keycloak
    dbUser: keycloak
    dbPassword: <ourDBPassword>

  resources:
    limits:
      cpu: 1
      memory: 4Gi

    requests:
      cpu: 100m
      memory: 1Gi

In addition to the ingress created by the chart (which we only use for master realm access) we also deploy an ingress that looks like this (in any given cluster there may be multiple of these, each for a different v-host):
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: vantiq-alik8s
    namespace: alik8s
  spec:
    rules:
    - host: alik8s.vantiq.com
      http:
        paths:
        - backend:
            serviceName: keycloak-http
            servicePort: 80
          path: /auth/
        - backend:
            serviceName: vantiq
            servicePort: 8080
          path: /
    tls:
    - hosts:
      - alik8s.vantiq.com
      secretName: vantiq-cert

Other details that may (or may not) matter:

We use the Nginx Ingress controller (installed from the stable chart hosted here).
We use a wildcard certificate for our domain
We have deployed this on multiple Amazon EKS clusters, an Alicloud CS for K8s cluster, and an OpenStack cluster (created via Magnum).
We've used several different PostgreSQL servers as well, so I doubt that's a factor.

All of this works perfectly using 3.4.0.
Switching to 4.0.0 while leaving everything else the same results in:

Cluster formation failure when running replicated (exactly as reported by @vsomasvr )
"invalid parameter" failure when running non-replicated (when accessed via the initial Ingress, we haven't tested it with the second Ingress).

Not sure if this will help anyone, but...  For now we are sticking with 3.4.0 since it is working with our configuration.

@weisjohn
Thanks for sharing. I can confirm that on single replica install adding this extraEnv to Keycloak helm chart (version 4.0.0) values.yaml,  fixes issue with Invalidparameter: redirect_uri for HTTPS ingress to access Keycloak admin interface.
  keycloak:
  extraEnv: |
    - name: PROXY_ADDRESS_FORWARDING
      value: "true"

Will deploy 2 node replica soon and update.

@sfitts
I guess

Nginx Ingress controller (installed from the stable chart hosted here).

would make a lot of difference. I will try this today
Thank you very much for sharing this

@elvinasp thanks for the update on that.
Based on the comments in #8192 it seems that the cluster issue may be related to the switch from JDBC_PING to DNS_PING.  Has anyone tried switching back?  Haven't looked at this in detail yet.

@axdotl is this the problem you're also facing?

I had tried it (version: 3.4.0) using nginx ingress  (as mentioned by @sfitts) , and it worked on GKE, great to have keycloak cluster deployed successfully..

I tried deploying the latest version 4.0.0 kc helm chart (with 4.5.0 Final keycloak) with JDBC_PING settings from 3.4.0 kc helm chart (with 4.2.1 Final keycloak) and found issues with the deployment.
Clearly, the JGroups JDBC_PING settings from 3.4.0 version don't work for the 4.0.0 version.
I had posted this question on stackoverflow here
https://stackoverflow.com/questions/52901704/keycloak-4-5-0-final-cluster-deployment-jdbc-ping-cli-config

# keycloak-0 instance
[0m�[33m06:15:58,460 WARN [org.keycloak.events] (default task-5) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=<IP_ADDRESS>, error=expired_code, restart_after_timeout=true

# keycloak-1 instance
[0m�[33m06:15:57,974 WARN [org.keycloak.events] (default task-3) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=<IP_ADDRESS>, error=expired_code, restart_after_timeout=true

@swistaczek I'm getting the same expired_code error in a deployment that used to work before the update to 4.5.0.Final. This is in a clean installation and trying to login in the Admin console. In my case I'm not using Istio and using 2 replicas. Does anyone else was able to resolve it?

I'm hitting the same on kube with haproxy ingress, on aws and postrgres in rds (this bit doesn't matter most likely. I found this excerpt from documentation ref https://www.keycloak.org/docs/4.5/server_installation/#troubleshooting

If you try to authenticate with Keycloak to your application, but authentication fails with an infinite number of redirects in your browser and you see the errors like this in the Keycloak server log:

2017-11-27 14:50:31,587 WARN  [org.keycloak.events] (default task-17) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=aa.bb.cc.dd, error=expired_code, restart_after_timeout=true

it probably means that your load balancer needs to be set to support sticky sessions. Make sure that the provided route name used during startup of Keycloak server (Property jboss.node.name) contains the correct name used by the load balancer server to identify the current server.

This might be related: https://github.com/helm/charts/issues/7679#issuecomment-430254416

Rebalancing and initial discovery works when I added that line to keycloak.cli.ha.

I opened a PR for this but in the meantime you should be able to add this line to your keycloak.cli.custom or ha section: /subsystem=jgroups/channel=ee:write-attribute(name=stack, value=tcp) to force jgroups to use tcp instead of udp

Cool, this has worked! Keycloak was able to start in cluster mode successfully..
I tried this on GKE, worked as expected with gce loadbalancer

I had the same issue.  Sticky sessions and enabling proxy as extra env var resolved it.
From the official documentation https://www.keycloak.org/docs/3.4/server_installation :
If you try to authenticate with Keycloak to your application, but it failed with the infinite number of redirects in your browser and you see the errors like this in the Keycloak server log:

2017-11-27 14:50:31,587 WARN  [org.keycloak.events] (default task-17) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=aa.bb.cc.dd, error=expired_code, restart_after_timeout=true
it probably means that your load balancer needs to be set to support sticky sessions. Make sure that the provided route name used during startup of Keycloak server (Property jboss.node.name) contains the correct name used by the load balancer server to identify the current server.

TL;DR
The ingress/ load balancer/ proxy configuration needs to support sticky sessions.

Sticky sessions shouldn't be required and we are not using them in our deployment behind nginx-ingress.
The sticky session is not mandatory for the cluster setup, however it is good for performance for the reasons mentioned above. https://www.keycloak.org/docs/3.4/server_installation/#sticky-sessions
The proxy_address_forwarding is absolutely required in the latest version of keycloak (I believe this is because the intenal keycloak config file changed) when you are behind an ingress with tls so the server knows to convert http to https.

I have got it to run without PROXY_ADDRESS_FORWARDING, see https://github.com/helm/charts/issues/7679#issuecomment-430254416. IMHO, the reason is a mismatch between this chart and the keycloak image.

Thanks @mborst I think I see now why it works based on your comment if you include the standalone.xml.  Looks like #8777 should fix this issue without having to include the extraArgs. As for the Ha configuration I think I have a solution that works without having to include the PROXY_ADDRESS_FORWARDING and will open a PR for that

Still seeing the redirect loop problem with v4.0.4 of the chart. Works ok with a single replica, but 2+ fails to set up infinispan correctly.
Setup:

Google Kubernetes (two node cluster)
Traefik as ingress handler with L4 gcloud load balancer in front
PROXY_ADDRESS_FORWARDING enabled

SOLVED.
I'm using pulumi to deploy the helm chart and for some reason it was deploying to default but .Release.Namespace was set to platform which meant it was using the wrong hostname for DNS_PING (keycloak-headless.platform.svc.cluster.local). Explicitly defining the namespace as default fixed it and it's now working correctly.
I think this is related to pulumi's handling of namespaces ref: https://github.com/helm/helm/issues/3553

I solved this by making sure my JGROUP_ env vars for the KEycloak docker image were correctly set. As this manages the service discovery + Clustering via "Infinispan".
There are two (actually 3 counting JDBC) discovery options, in K8s:  DNS_PING or KUBE_PING
if using DNS_PING you need to set the "property" to the proper DNS record of the service  (considering namespace and cluster domain), e.g.:
        - name: JGROUPS_DISCOVERY_PROTOCOL
          value: dns.DNS_PING
        - name: JGROUPS_DISCOVERY_PROPERTIES
          value: dns_query=keycloak-discovery.security
        - name: CACHE_OWNERS_COUNT
          value: "2"
        - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
          value: "2"

Note these are env variables passed to the Keycloak container (the  CACHE_OWNERS_*  are always present and need to be >=2 for HA)
in the case of using KUBE_PING, it looks like this:
            - name: JGROUPS_DISCOVERY_PROTOCOL
              value: kubernetes.KUBE_PING
            - name: JGROUPS_DISCOVERY_PROPERTIES
              value: 'namespace=${KUBERNETES_NAMESPACE:<your_namespace_name>}'
            - name: JGROUPS_TRANSPORT_STACK
              value: tcp
            - name: CACHE_OWNERS_COUNT
              value: '2'
            - name: AUTH_CACHE_OWNERS_COUNT
              value: '2'

or even:
            - name: JGROUPS_DISCOVERY_PROTOCOL
              value: kubernetes.KUBE_PING
            - name: KUBERNETES_NAMESPACE
              value: <your_namespace_name>
            - name: CACHE_OWNERS_COUNT
              value: '2'
            - name: AUTH_CACHE_OWNERS_COUNT
              value: '2'

JGROUPS_TRANSPORT_STACK seems to be optional (probably defaults to tcp), and the  KUBERNETES_NAMESPACE is actually read by Jgroups to figure the ns out.

I have the same issue and find a solution without the use of localhost or 127.0.01
I put the solution here


histogram calculate

What way can I use to calculate the histogram of image classes of my own datasets to get the weights in the train/main.py?

def compute_class_weights(self, histogram):
    """to compute the class weights
    Args:
        histogram: distribution of class samples
    """
    normHist = histogram / np.sum(histogram)
    for i in range(self.classes):
        self.classWeights[i] = 1 / (np.log(self.normVal + normHist[i]))

·```python
def compute_class_weights(self, histogram):
"""to compute the class weights
Args:
histogram: distribution of class samples
"""
normHist = histogram / np.sum(histogram)
for i in range(self.classes):
self.classWeights[i] = 1 / (np.log(self.normVal + normHist[i]))

    def compute_class_weights(self, histogram):
        """to compute the class weights
        Args:
            histogram: distribution of class samples
        """
        normHist = histogram / np.sum(histogram)
        for i in range(self.classes):
            self.classWeights[i] = 1 / (np.log(self.normVal + normHist[i]))


Secure removal of files

Description of the new feature / enhancement
Any file should have an option to be wiped clean from the disk, not leaving a recoverable version laying around.
Scenario when this would be used?
Secure removal of privacy information in files downloaded.
Supporting information
No response

/dup #13055


FW-73 Internal migration updates

Changelog
Updated previous PR to merge latest upstream/master and address edge cases highlighted in QA: Users will not be migrated if they do not have an email address, if their username is not an email address, or if their password does not meet the current password policy. In these cases, an InvalidMigrationException is thrown (and caught and logged at WARN level).

lol @plasticviking "sonar boolean unboxing silliness" 😆


Ability to remove a workload from a physical cluster

Is your feature request related to a problem? Please describe.
When I'm programming my global load balancer and I want to take a physical cluster out of service, I need to start draining a backend (connections routed to a pod on a physical cluster), wait for the drain to finish, and then remove the pod from that physical cluster.
Describe the solution you'd like
I'd like to be able to tell kcp when to remove a workload from a specific physical cluster.
Describe alternatives you've considered
Additional context

@chirino please let me know if I captured this correctly

I think the important thing is that "wait for drain to finish" is also based on an event that is external to the physical cluster.  Our ingress control plane will in charge of emitting it.

The functionality here feels very similar to that of a finalizer, except it is applied by a controller at the KCP level and then synced down to the physical cluster. Control plane decides when that finalizer is removed so that the resources can then be removed from the physical cluster.

Clearing milestone to re-triage

Could be done with some of the finalizer work in #914

@davidfestal @sttts @jmprusi action item: please come back and provide an example of how this could be achieved with finalizers

Yes, this is exactly what an ingress coordination controller will do by hooking into the deletion via workload finalizers, compare https://github.com/kcp-dev/kcp/blob/main/pkg/apis/workload/v1alpha1/types.go. We might be ready in v0.5 to actually demo this.

The PR https://github.com/kcp-dev/kcp/pull/1228 has made available per-location virtual-finalizers by default, so your controller can apply backpressure, and block the deletion of a resource by adding an annotation to the resource:
finalizers.workload.kcp.dev/"<workloadClusterName>": "finalizer1,finalizer2,..."

The syncer expects that annotation to be empty, before deleting the resource.
Does that cover your use-case?

As we talked via chat, this issue can be closed as this seems to cover the use-case! Thanks!


serde_codegen missing some carraige returns

This:
/// An uploaded file that was received as part of `multipart/form-data` parsing.
///
/// Files are streamed to disk because they may not fit in memory.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct UploadedFile {
    /// The temporary file where the data was saved.
    pub path: PathBuf,
    /// The filename that was specified in the data, unfiltered. It may or may not be legal on the
    /// local filesystem.
    pub filename: Option<String>,
    /// The unvalidated content-type that was specified in the data.
    pub content_type: Mime,
    /// The size of the file.
    pub size: usize,
    // The temporary directory the upload was put into, saved for the Drop trait
    tempdir: PathBuf,
}

is turning into this:
/// An uploaded file that was received as part of `multipart/form-data` parsing.////// Files are streamed to disk because they may not fit in memory.#[derive_Clone]
#[derive_PartialEq]
#[derive_Debug]
pub struct UploadedFile {
    /// The temporary file where the data was saved.
    pub path: PathBuf,
    /// The filename that was specified in the data, unfiltered. It may or may not be legal on the
    /// local filesystem.
    pub filename: Option<String>,
    /// The unvalidated content-type that was specified in the data.
    pub content_type: Mime,
    /// The size of the file.
    pub size: usize,
    tempdir: PathBuf,
}
impl ::serde::de::Deserialize for UploadedFile {
...

Causing one of the derives to be commented out.
(serde_codegen 0.6.6, syntex 0.22.0)
Any sort of workaround would be appreciated.

Just ran into this today myself. Is this an issue in serde_codegen or in syntex?

@mtdowling: I think this was supposed to be fixed in the latest syntex and serde_codegen. Which version are you on?

This is happening to me using these versions:

serde_codegen 0.6.10
syntex 0.26.0

Anyone have a workaround for this in the interim?

Ugh, I was able to reproduce this. One workaround would be to use the /** ... */ style comments, as in:
/**
An uploaded file that was received as part of `multipart/form-data` parsing.

Files are streamed to disk because they may not fit in memory.
*/
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct UploadedFile {
    /// The temporary file where the data was saved.
    pub path: PathBuf,
    /// The filename that was specified in the data, unfiltered. It may or may not be legal on the
    /// local filesystem.
    pub filename: Option<String>,
    /// The unvalidated content-type that was specified in the data.
    pub content_type: Mime,
    /// The size of the file.
    pub size: usize,
    // The temporary directory the upload was put into, saved for the Drop trait
    tempdir: PathBuf,
}

That still has the missing-newline, but it won't comment out critical code.

Turns out this is a rust pretty printing bug that syntex inherited.

Moving this bug over to serde-rs/syntex#31.


Various improvements to Ionide article

Fixes #5605
Also:

Removes some superfluous images to help with load times for less fast internet connections
Adds a tab control for using the command palette
Adds more formatting for code
Simplifies a few explanations

Thanks @mairaw!


Incompatable with "Dynamic Sidebar" snipets

Do you have the latest versions of Spicetify, Spotify and Lucid?

[X] I'm using the latest versions of Spicetify, Spotify and Lucid

Is there already an issue for your problem?

[X] I have checked older issues, open and closed

Is your problem definitely caused by the theme?

[X] I have checked the app behavior without the theme and it is different

Environment & Computer Info
- Installed from: Spotify site
Spotify for Windows (32 bit)
<IP_ADDRESS>9.gfae492c7
Spicetify v2.38.4
Theme: Lucid (Beta) / catppuccin-darkrose
Extensions: keyboardShortcut.js, bookmark.js, skipStats.js, fullAlbumDate.js, adblock.js, SleepTimer.js, noControls.js, quickQueue.js, section-marker.js, copytoclipboard.js, whatsThatGenre.js, find-duplicates.js, playback-bar-waveform.js, volumePercentage.js, track-trim.js, theme.js
Custom apps: visualizer, name-that-tune, marketplace
- Lucid installation method: Spicetify Marketplace

Description
All sidebar spinets use higher z-index to stay on top of screen
Screenshots

This might be caused by other snippets. Also, don't expect all snippets to work perfectly, as the theme applies different styling. However, the dynamic left sidebar snippet does work well with the theme.

https://github.com/user-attachments/assets/fcf6a984-6967-47bc-8cde-46fd1489789c


DMPTool Rails 6 infrastructure changes

We will be upgrading the DMPTool system from Rails 5.2 to 6.1.
Need to update puppet hiera file to use bundler 2.3.20 and ruby 2.7.6.
Proposed approach:

Deregister 02x2 host from the ALB
Disable puppet on 02x2
Update the Puppet hiera file to tag 4.x and bump the bundler and ruby versions
Run puppet deploy on 01x2
Test/verify that the application is working
Leave 02x2 offline for a few days (just in case)
Once satisfied with performance of the upgraded system, re-enable puppet on 02x2, run puppet deploy, and register with ALB

Rails upgrade notes ... thing to watch for

completed Nov 18


Editing Private Environments

Expected Behavior
When am editing any key in the environment, the editor shouldn't lose cursor focus on the line am editing
Actual Behavior
When editing the cursor disappears and loses focus on the input
Reproduction Steps

Go to environments
Create a private environment
Add in some environment variables
Click Done
Go Back to environments
Edit recent environment variable

Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information
No response
Insomnia Version
2022.6.0
What operating system are you using?
macOS
Operating System Version
Monterey 12.4 Beta (21F5058e)
Installation method
insomnia.rest
Last Known Working Insomnia version
No response

@elemanhillary this has been fixed since, and should not be reproducible in our latest beta build


Add additional default columns to poliy table

Required columns (the ones that can't be deselected).  Name, Namespace and Cluster compliance
Default columns (the columns that are selected by default and that you reset to when selecting reset to defaults).  This must always include the required columns, but can include 1 or more additional columns.   Remediation action, Policy Set and Source along with #1 columns.
The rest (not selected) columns

Ref: https://issues.redhat.com/browse/ACM-7156

/cc @gparvin  @mprahl

/test unit-tests-sonarcloud

/cc @fxiang1 @KevinFCormier

/lgtm


Make the Router mock automatic.

Hey @RobertBroersma I made some changes to your PR that allows it to work without having to create the mock file (with some additional fixes.)

Hey @peterp, that looks great! Nicely cleaned up as well.
I'm guessing this is the relevant snippet for making it work without the manual mock:
'^@redwoodjs/router$': path.join(
  NODE_MODULES_PATH,
<EMAIL_ADDRESS>),

I was thinking of doing this as well, but (afaict) this removes the ability for anyone to test the entire Router itself (or the entire app including the Router). Perhaps with some unsavory hacks in a config extension?
Now I'm not sure if that's something anyone would want to ever do tho, so if you think this is not an issue I'm all for it!

I was thinking of doing this as well, but (afaict) this removes the ability for anyone to test the entire Router itself (or the entire app including the Router). Perhaps with some unsavory hacks in a config extension?

Yup, the MockRouter does re-xport all the things, except for this top-level Router mock. You're right that if someone wanted to do something that isn't the "golden path" that it would require some configuration, and I think that's OK for now.
I would like to add a way for a user to include ./web/configs/jest.setup.js and ./web/configs/jest.config.js so that they could override this.

I would like to add a way for a user to include ./web/configs/jest.setup.js and ./web/configs/jest.config.js so that they could override this.

Certainly!


Option for correcting file modified timestamps of already downloaded media

Provide us a use case of the feature
The user would invoke it by a new option (e.g. --correct-timestamps). It would not only check, if a file already exists, but also whether it's modified date is that of the post. In that way erroneous blog downloads from other tools could be refreshed without the need to re-download everything, especially if meta data is missing and an external script cannot be used.
Describe the solution you'd like
A new option that modifies the file check/download behavior to not only check if a file has already been downloaded before, but also compare the already existing file's modified date to the post's date and adjust the file time if necessary.
Describe alternatives you've considered
If meta data would exist, an external script could be used. But even it would exist, that may be too difficult to do for the average user.
If the feature request is accepted, would you be willing to submit a pull request?
Yes

Additional context

Timestamps do not get changed once created. Therefore it would be a barely useful feature to be able to change them afterwards. It might be better to fix the other tools rather than having Instaloader support a feature that is not useful for itself.


Type Invariants Examples

Pick one example from s2n-quic that uses Vec and implement their type invariants then move on to Vec and its invariants and how that composes with the customer invariant.

This is no longer a priority for us. s2n-quic can use the kani::Invariant to annotate their own type invariants once it's available.


+0 SPS shop item desc. changed, formula formatting, changing pony images (#48)

The formula now has a bigger, serif font which fits better for a formula
The SPS increase text was changed to better reflect the fact that no SPS will be gained upon purchase. Also moved the remaining part of the text into a new variable to make the code cleaner.
Credits updated with the new image links (new format, see #53)
Re-compressed ground.svg while keeping the width & height attributes
Princess Cadence and Sunset Shimmer weren't changed, I only removed the unnecessary standalone="no" from the beginning of the files

Merging, but might reword the no SPS gain message.


to_jsonld()

I'm testing the Python package and noticed the following for "to_jsonld()":
If @context with @vocab is set the properties do not have to (should not ?) identify the vocab.
Also @type should not be given as a list but as a string (turning it into a list is EBRAINS KG specific and not a requirement by JSONLD).
Example for a correct openMINDS JSONLD:
{
  "@context": {
    "@vocab": "https://openminds.ebrains.eu/vocab/"
  },
  "@id": "http://localhost/openminds/instance/person/ZaphodBeeblebrox",
  "@type": "https://openminds.ebrains.eu/core/Person",
  "familyName": "Beeblebrox",
  "givenName": "Zaphod"
}

The current implementation uses the key "vocab" in the @context, not "@vocab", so the current output is valid JSON-LD.
However, I think it's a good idea to use "@vocab"

This has been separated into two issues:

@vocab -> #11
@type should not be a list -> this issue


Small typo in the Emmet 2.0 blog post

Someone wrote "It's features include" whereas it should be "Its features include".
Just thought to point it out. :)

Thanks for reporting @eyalsk
Fixed with https://github.com/Microsoft/vscode-docs/commit/1b471b70e6008b8761aa013911cdb9b18124084b
cc @gregvanl for publishing the website
The website will be updated early next week, as its the weekend now.
@eyalsk As a side note, all our docs are in https://github.com/Microsoft/vscode-docs. If you find any mistakes in https://code.visualstudio.com you can create an issue in that repo. PRs are welcome as well. We can use all the help we can get in improving our docs :)

@ramya-rao-a ah okay, thanks. ;)


subgraph_callable failure with dask 0.20.1

This fails on python 2.7.14, dask 0.20.1,  but works on dask 0.19.4.
As far as I can tell, a task like (operator.mul, 0j, [np.ndarray(...)]) gets constructed by subgraph_callable which produces the TypeError: can't multiply sequence by non-int of type 'complex' exception below
from __future__ import print_function

import dask.array as da
import numpy as np

row_chunks = (25, 25, 25, 25)
chan_chunks = (4, 4, 4, 4)
corr1_chunks = (2,)
corr2_chunks = (2,)

rows = sum(row_chunks)
chans = sum(chan_chunks)
corr1 = sum(corr1_chunks)
corr2 = sum(corr2_chunks)

shape = (rows, chans, corr1, corr2)
chunks = (row_chunks, chan_chunks, corr1_chunks, corr2_chunks)

vis = (da.random.random(shape, chunks=chunks) +
       da.random.random(shape, chunks=chunks)*1j)

def _grid2(vis):
    return np.ones(vis.shape[:2], dtype=vis.dtype)

grid2 = da.core.atop(_grid2, ("row", "chan"),
                     vis, ("row", "chan", "corr1", "corr2"),
                     dtype=vis.dtype)

print(grid2.compute().shape)

Produces the following:
Traceback (most recent call last):
  File "dask_atop_issue.py", line 38, in <module>
    print(grid.compute().shape)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/base.py", line 156, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/base.py", line 397, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/threaded.py", line 76, in get
    pack_exception=pack_exception, **kwargs)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/local.py", line 501, in get_async
    raise_exception(exc, tb)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/local.py", line 272, in execute_task
    result = _execute_task(task, data)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/local.py", line 253, in _execute_task
    return func(*args2)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/optimization.py", line 942, in __call__
    dict(zip(self.inkeys, args)))
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 132, in _get_recursive
    res = cache[x] = _get_recursive(dsk, dsk[x], cache)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 136, in _get_recursive
    args2 = [_get_recursive(dsk, k, cache) for k in args]
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 126, in _get_recursive
    return [_get_recursive(dsk, k, cache) for k in x]
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 132, in _get_recursive
    res = cache[x] = _get_recursive(dsk, dsk[x], cache)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 136, in _get_recursive
    args2 = [_get_recursive(dsk, k, cache) for k in args]
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 132, in _get_recursive
    res = cache[x] = _get_recursive(dsk, dsk[x], cache)
  File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 137, in _get_recursive
    return func(*args2)
TypeError: can't multiply sequence by non-int of type 'complex'

It looks to me like you want to specify concatenate=True to the atop
function.  I'm not sure why this would have changed in recent versions
though.  Is this something that you would be willing to investigate?
On Mon, Nov 12, 2018 at 4:36 AM Simon Perkins<EMAIL_ADDRESS>wrote:

This fails on python 2.7.14, dask 0.20.1, but works on dask 0.19.4.
As far as I can tell, a task like (operator.mul, 0j, [np.ndarray(...)])
gets constructed by subgraph_callable which produces the TypeError: can't
multiply sequence by non-int of type 'complex' exception below
from future import print_function
import dask.array as daimport numpy as np
row_chunks = (25, 25, 25, 25)
chan_chunks = (4, 4, 4, 4)
corr1_chunks = (2,)
corr2_chunks = (2,)
rows = sum(row_chunks)
chans = sum(chan_chunks)
corr1 = sum(corr1_chunks)
corr2 = sum(corr2_chunks)
shape = (rows, chans, corr1, corr2)
chunks = (row_chunks, chan_chunks, corr1_chunks, corr2_chunks)
vis = (da.random.random(shape, chunks=chunks) +
da.random.random(shape, chunks=chunks)*1j)
def _grid(vis, ny=None, nx=None):
print("_grid")
return np.ones(shape=(1, ny, nx), dtype=vis[0][0][0].dtype)
ny, nx = 1024, 1024
grid = da.core.atop(_grid, ("row", "ny", "nx"),
vis, ("row", "chan", "corr1", "corr2"),
new_axes={"ny": ny, "nx": nx},
adjust_chunks={"row": 1},
ny=ny, nx=nx,
dtype=vis.dtype)
print(grid.compute().shape)
Produces the following:
Traceback (most recent call last):
File "dask_atop_issue.py", line 38, in
print(grid.compute().shape)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/base.py", line 156, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/base.py", line 397, in compute
results = schedule(dsk, keys, **kwargs)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/threaded.py", line 76, in get
pack_exception=pack_exception, **kwargs)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/local.py", line 501, in get_async
raise_exception(exc, tb)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/local.py", line 272, in execute_task
result = _execute_task(task, data)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/local.py", line 253, in _execute_task
return func(*args2)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/optimization.py", line 942, in call
dict(zip(self.inkeys, args)))
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 132, in _get_recursive
res = cache[x] = _get_recursive(dsk, dsk[x], cache)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 136, in _get_recursive
args2 = [_get_recursive(dsk, k, cache) for k in args]
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 126, in _get_recursive
return [_get_recursive(dsk, k, cache) for k in x]
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 132, in _get_recursive
res = cache[x] = _get_recursive(dsk, dsk[x], cache)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 136, in _get_recursive
args2 = [_get_recursive(dsk, k, cache) for k in args]
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 132, in _get_recursive
res = cache[x] = _get_recursive(dsk, dsk[x], cache)
File "/home/sperkins/tmp/python/dask/venv/local/lib/python2.7/site-packages/dask/core.py", line 137, in _get_recursive
return func(*args2)
TypeError: can't multiply sequence by non-int of type 'complex'
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/4201, or mute the thread
https://github.com/notifications/unsubscribe-auth/AASszFXA4-9q6IZ2cAUU5IvCj11LpgV5ks5uuUEugaJpZM4YZRUs
.

@mrocklin I've reduced the example further. concatenate=True does indeed ameliorate this issue, but its preferable not to rely on this for performance reasons. Unfortunately I don't have time to delve deep into the new dask internals.

Here is a perhaps even simpler failure that shows the problem you're running into
import dask.array as da
import numpy as np

x = da.ones((10, 10), chunks=(5, 5))

def f(chunk):
    assert isinstance(chunk, np.ndarray)

y = da.core.atop(f, 'i',
                 x, 'ij',
                 dtype=x.dtype)
y.compute()

      6 def f(chunk):
----> 7     assert isinstance(chunk, np.ndarray)
      8
      9

AssertionError:

You are missing dimensions in your output, so they get concatenated into a list of elements.  If you want to treat that list as a numpy array then you should pass concatenate=True.  Really the question here is why this would have worked before.
I'm curious, how is passing concatenate=True a performance issue for you?

You are missing dimensions in your output, so they get concatenated into a list of elements. If you want to treat that list as a numpy array then you should pass concatenate=True

Correct this is a mistake in my updated example, I've re-updated it, and the complex multiplied by list issue still occurs.

I'm curious, how is passing concatenate=True a performance issue for you?

I use this style based on @shoyer's comments in https://github.com/dask/dask/pull/3412#issuecomment-381742007

Hrm, interesting, I've reduced this to the following minimal example
import dask.array as da
import numpy as np

x = da.ones((10, 10), chunks=(5, 5)) * 1j

def f(L):
    pass  # never actually get here

y = da.core.atop(f, 'i',
                 x, 'ij',
                 dtype=x.dtype)

y.compute()

Yeah, looks like a genuine bug.  My guess is that this comes up in the atop fusion code, my guess is somewhere around here:
https://github.com/dask/dask/blob/df4c6dc05461c77d568e747dfceb5043c065fb4d/dask/array/top.py#L599-L617
If anyone has time to take a look that would be welcome.  I'm busy the next couple days.


fix fix total task exec time is far more than actual

Proposed changes
The properties totalTaskExcutionTimeMs is far more than actual, which make load rate is too low than actual.
The actual load rate is 30w/s, but show routine load info is 1w/s.

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


Add support for asdf

Add support for asdf for easier python version switching (this also adds python's version 3 as the requirement to run the project).
Add details about downloading the model required by spaCy (without it, an error is thrown when running python scrape.py).

I feel like asdf is an unnecessary requirement, since none of the libraries we're using are python (3.X) version dependent. We now have a docker image anyway so we can directly change the python version there. Thanks for the PR though!


API: improve database queries

Purpose
The request user is injected in every API endpoint. Not prefetching related objects such as groups and operational units generates a huge number of unexpected queries and slows down API response time.
Proposal

[x] prefetch user-related groups and operational units
[x] improve bulk endpoints permissions checking

This is work is related to #179


PowerShellGet module edition metadata isn't working as expected

PowerShellGet is compatible with PowerShell Core and Windows PowerShell (PSEdition desktop and core). But this doesn't show up in all the expected places.
Current Behavior

PS C:\Users\edyoung> ipmo PowerShellGet
PS C:\Users\edyoung> get-module powershellget | select Version,CompatiblePSEditions

Version CompatiblePSEditions
------- --------------------
2.0.1   {}

Expected Behavior
CompatiblePSEditions should include Core,Desktop
Also,
Get-Module -listavailable PowerShellGet -PSEdition Core

does not return anything - expect to see the PowerShellGet module listed
Add Get-Module -Listavailable shows the Edition as 'Desk' - see below.
Your Environment

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      6.1.0
PSEdition                      Core
GitCommitId                    6.1.0
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

> Get-Module -ListAvailable PowerShellGet,PackageManagement

    Directory: C:\Program Files\PowerShell\Modules

ModuleType Version    Name                                PSEdition ExportedCommands
---------- -------    ----                                --------- ----------------
Script     <IP_ADDRESS>    PackageManagement                   Desk      {Find-Package, Get-Package, Get-PackageProvider,...
Script     2.0.1      PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-Module, Fi...
Script     1.6.6      PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-Module, Fi...
Script     1.6.5      PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-Module, Fi...
Script     1.6.0      PowerShellGet                       Desk      {Install-Module, Find-Module, Save-Module, Updat...

    Directory: C:\program files\powershell\6\Modules

ModuleType Version    Name                                PSEdition ExportedCommands
---------- -------    ----                                --------- ----------------
Script     <IP_ADDRESS>    PackageManagement                   Desk      {Find-Package, Get-Package, Get-PackageProvider,...
Script     1.6.7      PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-Module, Fi...

Once a module manifest is specified with the CompatiblePSEditions key or uses $PSEdition variable, it can not be imported on lower versions of PowerShell. The supported PSEditions are specified as tags. Some additional details can be found at https://docs.microsoft.com/en-us/powershell/gallery/concepts/module-psedition-support .

can't be fixed while keeping compat with older PS versions, so won't fix for the time being


replaceable encoder for converters

Hey,
If you can't afford to rely the built in iconv library directly, for example your server does not supports iconv library, now it possible to make your Encoder class / method easily, and replace them whatever you want.
No tests supplied now, maybe later.

@retnek  thanks


Save Cookie

I'm saving cookie to a text file so once again log in to get the file and upload the cookie again, however this is giving error in the ea.com domain.
Would he have any alternative?

I found the Solution in closed questions.
tks