Add parser support for Tiled hex colors as ui.Color

Adding direct support for parsing color hex strings as dart.ui.Color objects, which is the primary type the interoperates with Dart rendering libraries and most Canvas methods use.
Also, should eventually replace _parseTiledColor in flame_tiled RenderableTileMap.

I didn't get parts of this, what is the point of having two color fields everywhere?

I didn't get parts of this, what is the point of having two color fields everywhere?

Just trying to avoid breaking clients of this library that may be using the hex string version of the color fields. So, backwards compatibility 🤷
Happy to rename the color_ fields to a better name if you have an idea, but there is prior art for this naming convention like class_ https://github.com/flame-engine/tiled.dart/pull/50

Just trying to avoid breaking clients of this library that may be using the hex string version of the color fields. So, backwards compatibility 🤷

Since this lib is still pre 1.0 I think it would be better to just do a breaking change here than introducing this kind of "code smell".

Happy to rename the color_ fields to a better name if you have an idea, but there is prior art for this naming convention like class_ https://github.com/flame-engine/tiled.dart/pull/50

The difference between that field and this one is that both those fields exist in the specification though?

And the main rationale: hex strings are just super cumbersome to use as colors in the Dart/Flutter ecosystem, so it makes sense to parse them to ui.Color in my opinion.

Agree.

Since this lib is still pre 1.0 I think it would be better to just do a breaking change here than introducing this kind of "code smell".

Gotcha, I agree it'll be better to just make the breaking change, sounds good!
I almost forgot that flame_tiled can pick when it updates to the bumped version and deal with the breaking changes smoothly.

@spydon I pushed some changes that keep both the String and the Color version of the fields, but with colorHex and color as the name of the fields for clarity. I think it makes sense to keep exposing the hex string to clients in case they want to integrate with a different API (especially a web client). What do you think?

Adding my two cents to this PR: This change added a dependency on flutter's ui library, making this library no longer a dart only lib. With this, I don't think this library can be used anymore in command line dart programs to parse tiled maps if so needed. I feel like the bridge to flutter specific structures should be in flame_tiled perhaps? Perhaps some external extension methods to build Color objects from the string hex values?

Adding my two cents to this PR: This change added a dependency on flutter's ui library, making this library no longer a dart only lib. With this, I don't think this library can be used anymore in command line dart programs to parse tiled maps if so needed. I feel like the bridge to flutter specific structures should be in flame_tiled perhaps? Perhaps some external extension methods to build Color objects from the string hex values?

It doesn't seem like Flutter was added as a dependency in this PR? The Color object here is from dart:ui right? But I can see that Flutter is in the pubspec, and I agree that we should try to remove that.

Ah, I thought dart:ui was from flutter, but maybe I'm mistaken? Either way, I've been experimenting locally with removing the dependency to dart:ui and was able to get my code generator working, which was failing before from the dart:ui import. And I'm glad there's agreement that the flutter dependency should be taken out!

Curious, why was the code generator failing due to that one?


Cant Send Collected Fee

I am using parity for erc20 with peatio 2.0.32 then i get error like this

Failed to collect fee transfer deposit 80. See exception details below.
#<NoMethodError: undefined method address' for nil:NilClass> /home/app/app/services/wallet_service/ethereum.rb:53:in deposit_collection_fees'
/home/app/app/workers/worker/deposit_collection_fees.rb:27:in block in process' /opt/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.11/lib/active_record/locking/pessimistic.rb:72:in block in with_lock'
/opt/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in block in transaction' /opt/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/transaction.rb:184:in within_new_transaction'
/opt/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in transaction' /opt/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.11/lib/active_record/transactions.rb:220:in transaction'
/opt/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.11/lib/active_record/transactions.rb:277:in transaction' /opt/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.11/lib/active_record/locking/pessimistic.rb:70:in with_lock'
/home/app/app/workers/worker/deposit_collection_fees.rb:15:in process' lib/daemons/amqp_daemon.rb:60:in call'
lib/daemons/amqp_daemon.rb:60:in block (2 levels) in <main>' /opt/vendor/bundle/ruby/2.5.0/gems/bunny-2.11.0/lib/bunny/consumer.rb:56:in call'
/opt/vendor/bundle/ruby/2.5.0/gems/bunny-2.11.0/lib/bunny/channel.rb:1728:in block in handle_frameset' /opt/vendor/bundle/ruby/2.5.0/gems/bunny-2.11.0/lib/bunny/consumer_work_pool.rb:107:in block (2 levels) in run_loop'
/opt/vendor/bundle/ruby/2.5.0/gems/bunny-2.11.0/lib/bunny/consumer_work_pool.rb:102:in loop' /opt/vendor/bundle/ruby/2.5.0/gems/bunny-2.11.0/lib/bunny/consumer_work_pool.rb:102:in block in run_loop'
/opt/vendor/bundle/ruby/2.5.0/gems/bunny-2.11.0/lib/bunny/consumer_work_pool.rb:101:in catch' /opt/vendor/bundle/ruby/2.5.0/gems/bunny-2.11.0/lib/bunny/consumer_work_pool.rb:101:in run_loop'`

fixed


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v4.9.285",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=6fc4476dda58f6c00097c7ddec3b772513f57525"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "6fc4476dda58f6c00097c7ddec3b772513f57525",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "HID: betop: fix slab-out-of-bounds Write in betop_probe\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.9.285 by commit 6fc4476dda58f6c00097c7ddec3b772513f57525. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1001512


1.12

This question requires that you evaluate the underlined text to determine if it is correct.
An Azure region contains one or more data centers that are connected by using a low-latency
network.
Instructions: Review the underlined text. If it makes the statement correct, select "No change is
needed". If the statement is incorrect, select the answer choice that makes the statement correct .
A. No change is needed
B. Is found in each country where Microsoft has a subsidiary office
C. Can be found in every country in Europe and the Americas only
D. Contains one or more data centers that are connect by using a high-latency network

D


Minify Angular Material dialogs

Hi,
I am currently using gulp-cssnano to minify my application that uses Angular Material. I have noticed that, after minification, angular material dialogs cannot receive input from the user by any means. The only action that can be done is pressing ESC to close the dialog.
I have also noticed that other people face the same problem in this issue.
Angular version: 1.3.20
Angular Material version: 1.0.3
Gulp-CssNano version: 2.1.0
CssNano version: 3.4.0

Can you try with {safe: true} and report back?

@ben-eb That worked! Thanks very much.

No problem. :smile:

+1


Widget not loading, getting error

Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: null.

// Or you can replace latest with a specific version

Uncaught Error: Minified React error #130; visit https://reactjs.org/docs/error-decoder.html?invariant=130&args[]=null&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at Mu (webchat-latest.js:95)
at p (webchat-latest.js:95)
at m (webchat-latest.js:95)
at webchat-latest.js:95
at Ha (webchat-latest.js:95)
at Ss (webchat-latest.js:95)
at mu (webchat-latest.js:95)
at du (webchat-latest.js:95)
at iu (webchat-latest.js:95)
at webchat-latest.js:95

I Have Also observed same issue. It was running smoothly suddenly i am getting this issue.

Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: null.
 // Or you can replace latest with a specific version
Uncaught Error: Minified React error #130; visit https://reactjs.org/docs/error-decoder.html?invariant=130&args[]=null&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at Mu (webchat-latest.js:95)
at p (webchat-latest.js:95)
at m (webchat-latest.js:95)
at webchat-latest.js:95
at Ha (webchat-latest.js:95)
at Ss (webchat-latest.js:95)
at mu (webchat-latest.js:95)
at du (webchat-latest.js:95)
at iu (webchat-latest.js:95)
at webchat-latest.js:95

In my case it gets solved by clearing cache

Clearing cache does solve the issue here, The format of data kept in localStorage changes over different versions. We are currently experimenting with different solutions to that problem for future releases.

change localstorage to session storage like this:
params: { "storage": "session" }

Hi, concerning this issue we've added a mechanism that automatically wipes the webchat local storage when it's loaded with a new version. In effect, this will make sure this issue never arise again. Sorry for the inconvenience. I'm going to close the issue now.


Parsers fix

The FindBugsXMLParser needs to be changed to work with the output produced. The reason for the change is because the output that is produced after running FindBugs is different than the result that is produced by Maven.

Nice to see that the ProjectInfoFinder fixes the path problems with findbugs warnings.
Can we please remove one of the the parser for FindBugs? I mean I understand that it was written for the system before we added the Maven restriction, but now there we need only one parser. And having a NewFindBugsParser makes things more confusing. (or at least for me)

yes, I agree, I removed NewFindBugsParser, and now there is only 'FindBugsParser' in the code.

I think you should just call it FindBugsParser

yeah, mistake in my previous comment, it has the right name in the commits.

Good to see that the parser is now working again with FindBugs! Now we could filter also on FindBugs warnings in the visualizer!


wizard breaks appearance of kendo and bootstrap editors

For some reason Kendo Datepickers, Kendo Mutliselects and Bootstrap Datepicker are not displayed correctly within the wizard.

found the problems:

css rules affect (most) other plugins

render method


Ignore Unknown keys

I found this issue when pressing the Ctrl key while using Input, which causes it to return Err(Not a terminal) under windows (but works fine on Linux). In this case, I was in the process of pressing Ctrl + C in order to terminate the program.
There are a few keystrokes which cause console-rs to produce Unknown keys when running under Windows. At the time of this commit this includes pressing the Control key, but also previously included Shift and Alt.

https://github.com/console-rs/console/issues/83
https://github.com/console-rs/console/issues/102

Pressing these keys under a *nix terminal does not appear to produce a "key" at all - rather the user must press a complete key chord before Term::read_key() will produce a result.
Looking at the console-rs source, from what I can see that it produces an Unknown key under Windows in only two cases:

When the console is not "attended": https://github.com/console-rs/console/blob/master/src/term.rs#L274-L276
When the key code does not map to a known key in the Key enum: https://github.com/console-rs/console/blob/master/src/windows_term.rs#L337

Since dialoguer is already checking for non-attended consoles and executing an early return we can rule out needing to handle that scenario:
https://github.com/mitsuhiko/dialoguer/blob/f7d02321f5eda89c15b76ffbefbaebdd2c9f0bcb/src/prompts/input.rs#L276-L278
For the second scenario, it seems reasonable to ignore any "key" not recognized by console-rs for the purposes of the Input component.

Related PR on console: https://github.com/console-rs/console/pull/124

Pressing AltGr also triggers this behaviour on Windows, which can be more preblematic to enter special characters.

This also includes Caps Lock (as mentioned in the issues above). Any chance this could be merged?


How to reconstruct original data with Pyspod, in the time domain?

I am trying to apply spectral POD to a mean-normalized data matrix Q(t,x)∈ℝN×M, where N is the number of variables times the number of spatial grid points and M is the number of snapshots.
In classical POD, we find the K modes Ψ(x)∈ℝN×K, either through the eigenvectors of the covariance matrix or of the correlation matrix, and then define the time-amplitudes A(t)∈ℝK×M that approximates Q via A=Ψt Q.
Thus, the reconstructed (mean-normalized) data in classical POD is Q̃=Ψ A
But this is not clear to me in SPOD. Specifically, I have two main questions on using Pyspod, which I believe are missing in the tutorials:
1) How does one obtain the equivalent (spectral?)matrix Â in SPOD? Is there one matrix for each frequency fk?
I imagine it would be Âk=Ψ̂tk Q̂k
2) How do I get the approximation of Q̃ in the time domain, once I have the equivalent reconstructed matrix in the frequency domain?
Do I need to perform a "block inverse FFT"?
PS.: I really wish Github supported LateX notation hahaha

@bva99 thank you for the note. I am trying to get that implemented that shortly. To answer to your questions, yes, that is what you need to do, and there is a paper about to appear from @olivertschmidt that will explain that in detail.
1.) for every frequency:  A=Ψt W Q.
2.) q(k) = (1 / W) * ifft((k, freq))

@bva99  this is now present in the main branch of the code. You can check the code in spod_base.py, methods: reconstruct_data and reconstruct_data_freq. These two methods follow "Frequency–time analysis, low-rank reconstruction and denoising of turbulent flows using SPOD".


chore!: release 4.2.0

About
This PR lines up all of the commits on unstable and moving to stable to effectively lead us up to the changes we need to bring for version 4.2.0
There's multiple breaking changes from the RC candidate of 4.1.1 which lead to the decision to effectively bump the expected version as per semantic versioning.
Checklist

[X] I've ran pre-commit to format and lint the change(s) made.
[X] I've checked to make sure the change(s) work on 3.8.6 and higher.
[] This fixes/solves an Issue.

(If existent):

[X] I've made this pull request for/as: (check all that apply)

[X] Documentation
[X] Breaking change
[X] New feature/enhancement
[X] Bugfix

I should note that we are absolutely not following semantic versioning to the letter right now - we kinda already broke it many times 😅.
But it was decided that it was significant enough for a higher version bump .

I'd like to see this file deletion gone from the pr if possible but that's not required

what was it exactly? github didn't pick up on any file comments

I'd like to see this file deletion gone from the pr if possible but that's not required

what was it exactly? github didn't pick up on any file comments

The deletion of the duplicated bot.py, it's in the pr but it was deleted from stable already


I got bad results with my datasets

Thank you for your code
I try to train my own model using my datasets.But I got bad results, and I dont know how to fix it.

Then I try to figure out whats wrong. I got the results of lv1 and lv2 network.
lv1. It seems not bad.

But lv2 seems deviated to one side

Can you help me with that?
Thank you

Hi @hpbzxxxx,
This phenomenon usually happens when the background intensity of your input image is not equal to zero or the input images are not normalized within [0, 1].
If you are performing resize/interpolation using sklearn package, please make sure the background intensity of the input images is equal to zero. Otherwise, you may want to switch to masked NCC similarity function, when measuring the distance of the images.

Thank you for answering.
The background intensity of image is 0 and I did do the normalized within [0, 1] using the function in your code.
Is there any other problem that cause my issues?
Besides, the loss seems strange too.

total_loss, NCC_loss, Jet_loss, reg_loss，from top to bottom，respectively.

Could you share a few samples of the training dataset and your training code with me?

I used OASIS2. I ran freesurfer -autorecon1, and got brainmask.mgz. And normalized ,cropped to 160*192*144
OAS2_0002_MR1.nii.gz
OAS2_0009_MR1.nii.gz
OAS2_0010_MR1.nii.gz
OAS2_0012_MR1.nii.gz
The training code was copied from your code, just changed datapath, and sorted '*.nii.gz'

Hi @hpbzxxxx ,
Your preprocessed data looks fine to me. I decide to reproduce your error by training a new model with your preprocessed data. I will get back to you very soon.
If possible, could you please provide the exact modified Train_cLapIRN.py file to me, so that I can minimize the discrepancy between your training code and my training code.

There are all data and code I used for training.Thank you again!
code.zip
OAS2_0002_MR1.nii.gz
OAS2_0009_MR1.nii.gz
OAS2_0010_MR1.nii.gz
OAS2_0012_MR1.nii.gz
OAS2_0013_MR1.nii.gz
OAS2_0016_MR1.nii.gz
OAS2_0027_MR1.nii.gz
OAS2_0030_MR1.nii.gz
OAS2_0052_MR1.nii.gz
OAS2_0061_MR1.nii.gz
OAS2_0067_MR1.nii.gz
OAS2_0077_MR1.nii.gz
OAS2_0081_MR1.nii.gz
OAS2_0087_MR1.nii.gz
OAS2_0089_MR1.nii.gz
OAS2_0098_MR1.nii.gz
OAS2_0102_MR1.nii.gz
OAS2_0105_MR1.nii.gz
OAS2_0111_MR1.nii.gz
OAS2_0124_MR1.nii.gz
OAS2_0126_MR1.nii.gz
OAS2_0143_MR1.nii.gz
OAS2_0146_MR1.nii.gz
OAS2_0147_MR1.nii.gz
OAS2_0156_MR1.nii.gz
OAS2_0158_MR1.nii.gz
OAS2_0169_MR1.nii.gz
OAS2_0174_MR1.nii.gz

Hi @hpbzxxxx,
I have tried your preprocessed dataset with my method. Unfortunately, I have the same results as yours and I am not able to locate the bug.
Yet, I want to share the debugging process with you.
Potential problems: PyTorch version problem and the instance normalization makes the training become unstable.
Results: Tested with the latest Pytorch and old Pytorch, removed instance normalization. Yet, the problem persisted.
Potential problems: The contrast of your data is significantly lower than my OASIS training dataset.
Results: Improve the contrast using windowing technique, i.e. np.clip(img, 0, 0.65). Yet, the problem persisted.
Potential problems: The size of your training dataset is too small, i.e., 28 image scans.
Results: Sampled 28 image scans from my OASIS training dataset. It works, which implies our method can train with a small dataset.
Potential problems: The conditional module is somehow not compatible with your preprocessed dataset.
Results: Training LapIRN  without conditional modules using your preprocess dataset. The model was still collapsing at the first stage.
Conclusion: There is something wrong with your preprocessed dataset or in my code, but I cannot tell what exactly the problem is.
Suggestions:

Try another image registration method such as the Voxelmorph to see if the problem persists. If the problem persists, revisit the preprocess pipeline of your training data. If the problem is solved by another deep learning-based method, try not to crop the data but downsample it when using our method instead, because the gradient in the image boundary is not well defined in the deep learning-based method.
Use the preprocessed OASIS dataset provided by Adrian Dalca in link.

Please let me know if you have made any progress. I apologize I cannot locate the errors.

Im turely grateful for helping me!
Now Im training LapIRN with these data to see if it works. If it does not work, I will try data in link next.
As for voxelmorph, I tried it before which did work. The difference between these data and data for voxelmorph is that I didnt do normalization training voxelmorph and trained with another size(160*192*224). I will try downsample instead of crop as you said.
I will let you know if I get any progress.
AND THANK YOU AGAIN

Hi @hpbzxxxx,
I have updated the code and added an example using the preprocessed OASIS dataset without cropping/resize. You may want to check out the section in "(Example) Training on the preprocessed OASIS dataset without cropping" in the "README.md" file.


i18n(zh-cn): Translate from-create-react-app.mdx

What kind of changes does this PR include?

New or updated content
Translated content

Description
Translate from-create-react-app.mdx content to Chinese. #2982 PR is referred in this PR. and #4109 updates has fixed in this PR
Note: this PR should be merged after #2982 else the CI will be failed for broken links.

Rerunning CI / updating branch after merge of https://github.com/withastro/docs/pull/2982


Detect field contains base64

This seems like a useful one. I am always sticking base64-encoded things in JSON strings...

Ok. How to annotate them?

I guess with a string regex format: http://stackoverflow.com/questions/475074/regex-to-parse-or-validate-base64-data

No, problem not with detection.
URL, IP, etc are valid values for format field. "Base64" however isn't a valid value http://json-schema.org/latest/json-schema-validation.html#anchor107

With the pattern annotation: https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/elasticsearch_enriched_event/jsonschema/1-0-1#L90

Ok. Good point.

Actually, string recognitions are pretty straightfoward to implement.
If we need them, we can add them in nearest release.

Nice - let's bring this one forward then!


How to generate embeddings using Semantic Kernel

Hello. I want to create a custom memory index on Azure Cognitive Search (i.e. not using the structure that SK offers).
How can I go about uploading my data into a custom ACS index with embeddings and use ACS's vector search.
Using kernel.Memory.SaveInformationAsync or kernel.Memory.SearchAsync only seem to work with the index created by SK.
Is there any way to do it with SK or should I use Azure Search client and Azure openAPI clients to generate embeddings and store it on ACS.
Thanks!

@animeshj9 - thanks for raising this. We will look into this supporting this in a future release.
FYI: @dluc

Thank you @nacharya1. For now, we are using the 'AdditionalMetadataField' to store a Json Serialized text for the information we want to retrieve based on the searchable fields.
@nhandangt

@nacharya1 @dluc - Do we have any links to work items or something for me to follow? Thanks! :)

@nacharya1 Any update on this feature asks? Kindly share eta in case there is a plan to fix it.
Also wondering why we made AdditionalMetadataField type as string and not complex type, as we have a need to store json object. This is limiting us to perform filter on this field in case we want to use any of the properties of the complex object type.

I would suggest starting from our new memory solution at https://github.com/microsoft/semantic-memory - Custom schemas are not there yet but we have a plan to support them and many more features coming.
If you prefer using the current SK Memory, I think the fastest way would be using something like SK Indexer (part of Chat Copilot) to bulk upload your data. The indexer is now using the new memory, but earlier versions used the SK Memory I believe.

@animeshj9 Does the AzureCognitiveSearchMemoryStore not do what you are looking for? It connects to an existing/custom ACS, but I'm not sure how to integrate it with your own ACS index's custom schema; from here it looks like the schema is hardcoded.
Related announcement with link to video: https://devblogs.microsoft.com/semantic-kernel/announcing-semantic-kernel-integration-with-azure-cognitive-search/

@brendenpetersen - We were not able to use the AzureCognitiveMemorySearchStore for this - the schema is hardcoded and there isnt an easy way to create a custom schema. We used https://github.com/microsoft/semantic-memory as suggested by @dluc .
@lallgaurav will have more information as he is the one who implemented this for us.

@lallgaurav That would be great if you could provide any input, we are in the same boat but have not yet tried the semantic-memory repository.

@brendenpetersen yes, we are leveraging Semantic Memory (I believe it got renamed to Kernel Memory). It has a provision for specifying tags which is a kind of work around for Custom schema though not very efficient as it does in-memory filtering. I am hopeful that Semantic Memory team will fix it in its upcoming release.
@dluc any eta by when this issue will get fixed?

The work to support custom schema is in progress, here is a link to the ADR https://github.com/microsoft/semantic-kernel/issues/5887
Closing this issue as we have backlog tasks to track this enhancement.


feat: Initial FrameTime API

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#14  👈
#13
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @tommy-xr and the rest of your teammates on  Graphite


KVM NFS disk IO driver supporting IO_URING

Description
Currently there is no disk IO driver configuration for VMs running on KVM. That's OK for most the cases; however, recently there have been added some quite interesting optimizations with the IO driver io_uring.
Note that IO URING requires:

Qemu >= 5.0, and
Libvirt >= 6.3.0.

By using io_uring we can see a massive I/O performance improvement within Virtual Machines running from Local and/or NFS storage.
This implementation enhances the KVM disk configuration by adding workflow for setting the disk IO drivers. Additionally, if the    Qemu and Libvirt versions matches with the required for having io_uring we are going to set it on the VM. If there is no support for such driver we keep it as it is nowadays, without any IO driver configured.

Fixes: #4883

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[X] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

Locally debugging with breakpoints on remote debug and asserting the IO driver as well as qemu/libvirt versions.

Thanks for the review, @GutoVeronezi. Code has been updated.

I will check if I can install the required Libvirt/Qemu packages on Ubuntu 20.04 in order to test this implementation. I am not sure if they will be installable in 20.04 as we can find them on 21.04.
If it works I can run some tests to see how it behaves with RBD, NFS, and Filesystem.

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_multiplication_x: centos7 :heavy_multiplication_x: centos8 :heavy_check_mark: debian. SL-JID 271

@blueorangutan package

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: centos7 :heavy_check_mark: centos8 :heavy_check_mark: debian. SL-JID 357

@blueorangutan test

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1126)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33399 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5012-t1126-kvm-centos7.zip
Smoke tests completed. 88 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

I see unit tests @GabrielBrascher :+1: are integration tests for io driver-types in the planning/possible? or do we need hardware? (i don't think so, only kernel features and maybe qemu/libvirt version supporting it)

@DaanHoogland @rhtyd I've executed some manual tests on hardware that has the required Qemu & Libvirt versions to support IO_URING.
The VM has the IO_URING added as expected in its XML when the versions match the requirements.
Executed some simple IO tests on the same host before and after the IO driver being applied (same hardware, VM, and OS/Kernel).
data in/out  1GB   2GB   4GB   5GB   10GB
default      104   101   102    92    102 MB/s
io-uring      92    95    96    96    100 MB/s

@DaanHoogland Do you mean integration tests that would validate the domain XML according to the host qemu/libvirt version?.

@DaanHoogland @rhtyd I've executed some manual tests on hardware that has the required Qemu & Libvirt versions to support IO_URING.
The VM has the IO_URING added as expected in its XML when the versions match the requirements.
Executed some simple IO tests on the same VM and host before and after the IO driver being applied (same hardware, VM, and OS/Kernel).
data in/out  1GB   2GB   4GB   5GB   10GB
default      104   101   102    92    102 MB/s
io-uring      92    95    96    96    100 MB/s

@DaanHoogland Do you mean integration tests that would validate the domain XML according to the host qemu/libvirt version?.

@GabrielBrascher it looks the performance are quite close, right ?

I agree, @weizhouapache. I think that this is due to the tests being too simple, probably not the best way of exploring the IO optimizations with some "dd" tests. I think that I need to run some elaborated tests.
There are some interesting tests thet show a great improvement in performance. For example: https://github.com/apache/cloudstack/issues/4883#issuecomment-838523087

I agree, @weizhouapache. I think that this is due to the tests being too simple, probably not the best way of exploring the IO optimizations with some "dd" tests. I think that I need to run some elaborated tests.
There are some interesting tests thet show a great improvement in performance. For example: #4883 (comment)

@GabrielBrascher
the testing results in https://github.com/apache/cloudstack/issues/4883#issuecomment-838523087 are exciting.
do you know any other setting other than io_uring ? ( I did not look into the pdf).

@DaanHoogland Do you mean integration tests that would validate the domain XML according to the host qemu/libvirt version?.

yes, and does a copy/mv/del to verify io is functioning, maybe. I do not say we need it, just wondering. might be excessive.

this PR prevent me from using Cloudstack
I get the following in cloudstack logs
May 17 16:21:48 app.sy.sa java[1982]: INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-77:ctx-da4144a8 job-481/job-559) (logid:f2176072) Add job-559 into job monitoring
May 17 16:21:48 app.sy.sa java[1982]: INFO  [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-77:ctx-da4144a8 job-481/job-559 ctx-a0d804e3 FirstFitRoutingAllocator) (logid:3d720a9a)  Guest VM is requested with Custom[UEFI] Boot Type false
May 17 16:21:48 app.sy.sa java[1982]: INFO  [c.c.s.StorageManagerImpl] (Work-Job-Executor-77:ctx-da4144a8 job-481/job-559 ctx-a0d804e3) (logid:3d720a9a) Storage pool Primary1 (1) does not supply IOPS capacity, assuming enough capacity
May 17 16:21:49 app.sy.sa java[1982]: INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-77:ctx-da4144a8 job-481/job-559 ctx-a0d804e3) (logid:3d720a9a) Unable to start VM on Host {"id": "1", "name": "app.sy.sa", "uuid": "512fa84c-237d-4cdb-b86b-1bac252eaffc", "type"="Routing"} due to unsupported configuration: io uring is not supported by this QEMU binary

May 17 16:21:51 app.sy.sa java[1982]: INFO  [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-82b6a15f) (logid:792d27bd) allocating virtual machine from template:0d91769b-c74f-4614-92a3-759f26af8c62 with hostname:v-182-VM and 3 networks
May 17 16:21:51 app.sy.sa java[1982]: INFO  [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-82b6a15f) (logid:792d27bd) adding disk object ROOT-182 to v-182-VM
May 17 16:21:52 app.sy.sa java[1982]: INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-78:ctx-8d72fa13 job-483/job-560) (logid:3bbd911f) Add job-560 into job monitoring

same thing for router:
May 17 21:21:29 app.sy.sa java[1975]: INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-9643ce8b job-481/job-1600 ctx-501df6a9) (logid:3d720a9a) Unable to start VM on Host {"id": "1", "name": "app.sy.sa", "uuid": "512fa84c-237d-4cdb-b86b-1bac252eaffc", "type"="Routing"} due to unsupported configuration: io uring is not supported by this QEMU binary
May 17 21:21:29 app.sy.sa java[1975]: INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-038727fd job-483/job-1601 ctx-cd372811) (logid:e68ba882) Unable to start VM on Host {"id": "1", "name": "app.sy.sa", "uuid": "512fa84c-237d-4cdb-b86b-1bac252eaffc", "type"="Routing"} due to unsupported configuration: io uring is not supported by this QEMU binary

@xlmnxp, can you share the libvirt/qemu versions and the OS? Also, do you use advanced virtualization? I had the same problem on Rocky with the advanced virtualization, and there is a fix which will be included in CS versions - <IP_ADDRESS> and 4.17.
Reported issue

as @slavkap said, #6253 introduced a new config enable.io.uring in agent.properties.
it will be included in upcoming <IP_ADDRESS>.
if you use latest libvirt, #6244 is also required which will be included in <IP_ADDRESS> as well.

@xlmnxp, can you share the libvirt/qemu versions and the OS? Also, do you use advanced virtualization? I had the same problem on Rocky with the advanced virtualization, and there is a fix which will be included in CS versions - <IP_ADDRESS> and 4.17. Reported issue

I indeed think this is due to the OS. Without knowing the OS and Qemu version we can't tell the root cause.

OS: Rocky Linux 8.6 (Green Obsidian)
Libvirt: libvirtd (libvirt) 8.0.0

as @slavkap said, https://github.com/apache/cloudstack/pull/6253 introduced a new config enable.io.uring in agent.properties.
it will be included in upcoming <IP_ADDRESS>.
if you use latest libvirt, https://github.com/apache/cloudstack/pull/6244 is also required which will be included in <IP_ADDRESS> as well.

I already disabled it in agent config but the router didn't start because of is_uring which I disabled
/etc/cloudstack/agent/agent.properties:
#Storage
#Wed May 18 05:08:47 EDT 2022
cluster=1
pod=1
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
guest.cpu.mode=host-passthrough
router.aggregation.command.each.timeout=600
guest.network.device=cloudbr0
enable.io.uring=false
keystore.passphrase=EaPsawRuTKRF7km3
hypervisor.type=kvm
port=8250
zone=1
public.network.device=cloudbr0
local.storage.uuid=ca0e32d1-a818-49fb-bf4d-9e73aef1f19e
host=<IP_ADDRESS>@static
guid=28e45ca9-ac69-3278-8703-3e81566b0575
LibvirtComputingResource.id=1
workers=5
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600

@xlmnxp, can you share the libvirt/qemu versions and the OS? Also, do you use advanced virtualization? I had the same problem on Rocky with the advanced virtualization, and there is a fix which will be included in CS versions - <IP_ADDRESS> and 4.17. Reported issue

I indeed think this is due to the OS. Without knowing the OS and Qemu version we can't tell the root cause.

OS: Rocky Linux 8.6 (Green Obsidian) (Inside Virtualbox and I enabled nested virtualization hypervisors) Libvirt: libvirtd (libvirt) 8.0.0 Cloudstack: <IP_ADDRESS>

as @slavkap said, #6253 introduced a new config enable.io.uring in agent.properties.
it will be included in upcoming <IP_ADDRESS>.
if you use latest libvirt, #6244 is also required which will be included in <IP_ADDRESS> as well.

I already disabled it in agent config but the router didn't start because of is_uring which I disabled
/etc/cloudstack/agent/agent.properties:
#Storage
#Wed May 18 05:08:47 EDT 2022
cluster=1
pod=1
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
guest.cpu.mode=host-passthrough
router.aggregation.command.each.timeout=600
guest.network.device=cloudbr0
enable.io.uring=false
keystore.passphrase=EaPsawRuTKRF7km3
hypervisor.type=kvm
port=8250
zone=1
public.network.device=cloudbr0
local.storage.uuid=ca0e32d1-a818-49fb-bf4d-9e73aef1f19e
host=<IP_ADDRESS>@static
guid=28e45ca9-ac69-3278-8703-3e81566b0575
LibvirtComputingResource.id=1
workers=5
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600

@xlmnxp
<IP_ADDRESS> does not have the changes in #6253
nothing will be changed if you set enable.io.uring to false.

@xlmnxp, can you share the libvirt/qemu versions and the OS? Also, do you use advanced virtualization? I had the same problem on Rocky with the advanced virtualization, and there is a fix which will be included in CS versions - <IP_ADDRESS> and 4.17. Reported issue

I indeed think this is due to the OS. Without knowing the OS and Qemu version we can't tell the root cause.

OS: Rocky Linux 8.6 (Green Obsidian) (Inside Virtualbox and I enabled nested virtualization hypervisors) Libvirt: libvirtd (libvirt) 8.0.0 Cloudstack: <IP_ADDRESS>

as @slavkap said, #6253 introduced a new config enable.io.uring in agent.properties.
it will be included in upcoming <IP_ADDRESS>.
if you use latest libvirt, #6244 is also required which will be included in <IP_ADDRESS> as well.

I already disabled it in agent config but the router didn't start because of is_uring which I disabled
/etc/cloudstack/agent/agent.properties:
#Storage
#Wed May 18 05:08:47 EDT 2022
cluster=1
pod=1
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
guest.cpu.mode=host-passthrough
router.aggregation.command.each.timeout=600
guest.network.device=cloudbr0
enable.io.uring=false
keystore.passphrase=EaPsawRuTKRF7km3
hypervisor.type=kvm
port=8250
zone=1
public.network.device=cloudbr0
local.storage.uuid=ca0e32d1-a818-49fb-bf4d-9e73aef1f19e
host=<IP_ADDRESS>@static
guid=28e45ca9-ac69-3278-8703-3e81566b0575
LibvirtComputingResource.id=1
workers=5
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600

@xlmnxp <IP_ADDRESS> does not have the changes in #6253 nothing will be changed if you set enable.io.uring to false.

Then how I can upgrade to 4.17?

@xlmnxp <IP_ADDRESS> does not have the changes in #6253 nothing will be changed if you set enable.io.uring to false.

Then how I can upgrade to 4.17?

@xlmnxp
<IP_ADDRESS> is not released yet.  it will be released in coming weeks.

@xlmnxp, I could suggest until there isn't a released fix, you can downgrade the qemu/libvirt versions to libvirt 6.0.0 and qemu 4.2 or something lower than the supported ones:
Qemu >= 5.0, and
Libvirt >= 6.3.0

@xlmnxp, I could suggest until there isn't a released version with the fix, you can downgrade the qemu/libvirt versions to libvirt 6.0.0 and qemu 4.2 or something lower than the supported ones: Qemu >= 5.0, and Libvirt >= 6.3.0

Is there a way other then downgrade libvirtd?

@xlmnxp, I don't have anything else on my mind :/ ... The most painless solution for me is downgrading the versions of libvirt/qemu. Another option is to switch to another OS (which supports io_uring) on the hypervisor or downgrade the CS version.
Or wait for 4.17

@xlmnxp, I don't have anything else on my mind :/ ... The most painless solution for me is downgrading the versions of libvirt/qemu. Another option is to switch to another OS (which supports io_uring) on the hypervisor or downgrade the CS version. Or wait for 4.17

then best option is to wait 4.17

I have same problem after I update to Rocky Linux 8.6
I had tried to downgrade, but it is difficult to downgrade with some dependence issue.
@xlmnxp , You can rebase #6253, #6399, and #6402 to tag <IP_ADDRESS> then replace jar file cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar and cloud-agent-<IP_ADDRESS>.jar if you urgently need.
My steps:
Install requirement packages (on RockyLinux)
$ sudo dnf install git java-11-openjdk-devel maven
$ sudo alternatives --config javac
$ export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-<IP_ADDRESS>.10-2.el8_6.x86_64/

clone the repository
$ git clone https://github.com/apache/cloudstack.git
$ cd cloudstack
$ git remote add shapeblue https://github.com/shapeblue/cloudstack.git
$ git fetch shapeblue

checkout #6399 and rebase to <IP_ADDRESS>
only pick #6253 and #6399, drop others commit (note that PR #6399 isn't merged yet)
$ git checkout -b fix-iouring shapeblue/disableiouringbydefault
$ git rebase -i <IP_ADDRESS>
:1,29s/^pick/drop/g
:31,39s/^pick/drop/g

drop bc70535ee5 Updating pom.xml version numbers for release <IP_ADDRESS>-SNAPSHOT
drop c366511294 UI: Missing message on VM import for not found networks (#6055)
drop 2820a36f86 Check the network access when deploying VM in Advanced Security Group. (#6050)
drop 84f19d8f36 UI: update vm with userdata (#6051)
drop e4d70d4214 UI: Fix Autogenview cleared resource (#6066)
drop 6b913a76cf ui: Add user initials as avatar if no icon present (#6070)
drop 7fbfd4c6ea UI: Fix navigation to domains (#6072)
drop c2bcad8571 ui: Set icon to osdisplayname to avoid multiple api calls (#6075)
drop 00c1bdb365 UI: Reload page on closing Bulk Action modal (#6077)
drop 3a456f1b31 server: mark volume snapshots as Destroyed if it does not exist on primary and secondary storage when delete a volume (#6057)
drop 067c1de080 Fix get upload params NPE (#6079)
drop 6ff378f6a3 travis: run nosetests-2.7 (#6113)
drop 75b54171ae Travis - fix test failures observed (#6119)
drop 4be99fe971 api: Allow updating VM settings when custom contrained offering is used (#6136)
drop f8b648b938 Fix migration of VM with volume on Ubuntu (#6116)
drop fb43076f9e Fix linux native bridge for SUSE in cloudutils (#6134)
drop f4b9ab034b UI: fix create l2 network offering with userdata (#6174)
drop bcd1a3274a api: Fix reset configuration (#6168)
drop 908f594f00 configDrive: Fix failure to delete (unstarted) VM (#6146)
drop a69ab3b28f Ensure configdrive path is edited properly during live migration (#6173)
drop ee27708ffb SAML: replace first number with random alphabet if request ID starts with a number (#6165)
drop 66a6671e0b ui,refactor: fix missing label in update network form (#6181)
drop 47454eca7d VR: add '-m <protocol>' for tcp or udp protocol (#6188)
drop c61ea9f96d VR: Do not add iptables rules for the revoked ip addresses (#6189)
drop e7071ec196 server: increment deviceid while importing vm data volumes (#6123)
drop ee2ded8200 potential null pointer in condition; AYAI9l8k5Irk9_td-cXb (#6237)
drop b6072fc826 Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service (#6234)
drop 19a7774cab VR: add rules for traffic between static nat and private gateway static routes (#6153)
drop 91a5f0e285 server: honor global setting system.vm.default.hypervisor as first option when deploy VRs (#6160)
pick 42a92dcdd3 Extract the IO_URING configuration into the agent.properties (#6253)
drop fbf77978e1 Fix: Allow disabling the login attempts mechanism for disabling users (#6254)
drop bbb4ffa593 UI: fix dedicate public ip range to domain (#6258)
drop 365966dd0a UI: Fix custom unconstrained for a zone does not show CPU speed (#6285)
drop b2338f7158 Updated reset configuration, to return the updated config value in the response (#6284)
drop 5fa8fa5580 Fix upload volume format (#6297)
drop efb1f2b719 UI: Fix templates page redirection after delete job is finished (#6345)
drop 556f9dac0f ui: Network offerings not listed if listVPCs not available in the account Role (#6354)
drop d373f973ba Update VM name, when the new name provided in updateVirtualMachine API in different case. (#6379)
drop 006473ca19 Log exception on keystore build for custom certificate (#6394)
pick f9aa91c642 [KVM] Disable IOURING by default on agents
pick fad8a5752c Refactor
pick 1d6d1b09fb Remove agent property for iouring
pick c11cd1bf4e Restore property
pick 261e275c38 Refactor suse check and enable on ubuntu by default
pick 0e76ca64d7 Refactor irrespective of guest OS
pick 9e0574683d Improvement
pick b349e75614 Logs and new path
pick a74a4a83df Refactor condition to enable iouring
pick 09c1f44093 Improve condition
pick a3ab1a33e4 Refactor property check
pick 88b3666d8c Improvement
pick 0c8a6aa03a Doc comment
pick 3835e8b8cf Extend comment

# Rebase cad9332082..3835e8b8cf onto 0c8a6aa03a (53 commands)

:wq

and also need to fix VNC password as #6402
checkout #6402 and rebase to fix-iouring
only pick #6402, drop others commit
$ git checkout -b fix-vncpassword 363a2cff8231a31c3c9def329805910e43358b1d
$ git rebase -i fix-iouring
:1,39s/^pick/drop/g

drop bc70535ee5 Updating pom.xml version numbers for release <IP_ADDRESS>-SNAPSHOT
drop c366511294 UI: Missing message on VM import for not found networks (#6055)
drop 2820a36f86 Check the network access when deploying VM in Advanced Security Group. (#6050)
drop 84f19d8f36 UI: update vm with userdata (#6051)
drop e4d70d4214 UI: Fix Autogenview cleared resource (#6066)
drop 6b913a76cf ui: Add user initials as avatar if no icon present (#6070)
drop 7fbfd4c6ea UI: Fix navigation to domains (#6072)
drop c2bcad8571 ui: Set icon to osdisplayname to avoid multiple api calls (#6075)
drop 00c1bdb365 UI: Reload page on closing Bulk Action modal (#6077)
drop 3a456f1b31 server: mark volume snapshots as Destroyed if it does not exist on primary and secondary storage when delete a volume (#6057)
drop 067c1de080 Fix get upload params NPE (#6079)
drop 6ff378f6a3 travis: run nosetests-2.7 (#6113)
drop 75b54171ae Travis - fix test failures observed (#6119)
drop 4be99fe971 api: Allow updating VM settings when custom contrained offering is used (#6136)
drop f8b648b938 Fix migration of VM with volume on Ubuntu (#6116)
drop fb43076f9e Fix linux native bridge for SUSE in cloudutils (#6134)
drop f4b9ab034b UI: fix create l2 network offering with userdata (#6174)
drop bcd1a3274a api: Fix reset configuration (#6168)
drop 908f594f00 configDrive: Fix failure to delete (unstarted) VM (#6146)
drop a69ab3b28f Ensure configdrive path is edited properly during live migration (#6173)
drop ee27708ffb SAML: replace first number with random alphabet if request ID starts with a number (#6165)
drop 66a6671e0b ui,refactor: fix missing label in update network form (#6181)
drop 47454eca7d VR: add '-m <protocol>' for tcp or udp protocol (#6188)
drop c61ea9f96d VR: Do not add iptables rules for the revoked ip addresses (#6189)
drop e7071ec196 server: increment deviceid while importing vm data volumes (#6123)
drop ee2ded8200 potential null pointer in condition; AYAI9l8k5Irk9_td-cXb (#6237)
drop b6072fc826 Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service (#6234)
drop 19a7774cab VR: add rules for traffic between static nat and private gateway static routes (#6153)
drop 91a5f0e285 server: honor global setting system.vm.default.hypervisor as first option when deploy VRs (#6160)
drop fbf77978e1 Fix: Allow disabling the login attempts mechanism for disabling users (#6254)
drop bbb4ffa593 UI: fix dedicate public ip range to domain (#6258)
drop 365966dd0a UI: Fix custom unconstrained for a zone does not show CPU speed (#6285)
drop b2338f7158 Updated reset configuration, to return the updated config value in the response (#6284)
drop 5fa8fa5580 Fix upload volume format (#6297)
drop efb1f2b719 UI: Fix templates page redirection after delete job is finished (#6345)
drop 556f9dac0f ui: Network offerings not listed if listVPCs not available in the account Role (#6354)
drop d373f973ba Update VM name, when the new name provided in updateVirtualMachine API in different case. (#6379)
drop 006473ca19 Log exception on keystore build for custom certificate (#6394)
drop b62b5c96e8 Prevent NPE on reboot stopped VM and startVM output with null displayname (#6397)
pick 363a2cff82 Backport: kvm: truncate vnc password to 8 chars (#6244) (#6402)

# Rebase bab903af16..363a2cff82 onto 363a2cff82 (40 commands)

:wq

build jar files and copy to your agent
$ mvn install
$ scp agent/target/cloud-agent-<IP_ADDRESS>.jar leo@kvm-agent:
$ scp plugins/hypervisors/kvm/target/cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar leo@kvm-agent:

You need to update CloudStack to <IP_ADDRESS> first
Then you can replace those two jar file and restart agent
Of course you can backup the original jar file if you want
$ sudo mv /usr/share/cloudstack-agent/lib/cloud-agent-<IP_ADDRESS>.jar /usr/share/cloudstack-agent/lib/cloud-agent-<IP_ADDRESS>.jar.origin
$ sudo mv /usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar /usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar.origin
$ sudo cp ~/cloud-agent-<IP_ADDRESS>.jar /usr/share/cloudstack-agent/lib/
$ sudo cp ~/cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar /usr/share/cloudstack-agent/lib/
$ sudo systemctl restart cloudstack-agent.service

You can use my jar file if you don't mind
CloudStack-<IP_ADDRESS>-agent_patch_for_rockylinux8.6.zip

I have same problem after I update to Rocky Linux 8.6 I had tried to downgrade, but it is difficult to downgrade with some dependence issue.
@xlmnxp , You can rebase #6253, #6399, and #6402 to tag <IP_ADDRESS> then replace jar file cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar and cloud-agent-<IP_ADDRESS>.jar if you urgently need.
My steps: Install requirement packages (on RockyLinux)
$ sudo dnf install git java-11-openjdk-devel maven
$ sudo alternatives --config javac
$ export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-<IP_ADDRESS>.10-2.el8_6.x86_64/

clone the repository
$ git clone https://github.com/apache/cloudstack.git
$ cd cloudstack
$ git remote add shapeblue https://github.com/shapeblue/cloudstack.git
$ git fetch shapeblue

checkout #6399 and rebase to <IP_ADDRESS> only pick #6253 and #6399, drop others commit (note that PR #6399 isn't merged yet)
$ git checkout -b fix-iouring shapeblue/disableiouringbydefault
$ git rebase -i <IP_ADDRESS>
:1,29s/^pick/drop/g
:31,39s/^pick/drop/g

drop bc70535ee5 Updating pom.xml version numbers for release <IP_ADDRESS>-SNAPSHOT
drop c366511294 UI: Missing message on VM import for not found networks (#6055)
drop 2820a36f86 Check the network access when deploying VM in Advanced Security Group. (#6050)
drop 84f19d8f36 UI: update vm with userdata (#6051)
drop e4d70d4214 UI: Fix Autogenview cleared resource (#6066)
drop 6b913a76cf ui: Add user initials as avatar if no icon present (#6070)
drop 7fbfd4c6ea UI: Fix navigation to domains (#6072)
drop c2bcad8571 ui: Set icon to osdisplayname to avoid multiple api calls (#6075)
drop 00c1bdb365 UI: Reload page on closing Bulk Action modal (#6077)
drop 3a456f1b31 server: mark volume snapshots as Destroyed if it does not exist on primary and secondary storage when delete a volume (#6057)
drop 067c1de080 Fix get upload params NPE (#6079)
drop 6ff378f6a3 travis: run nosetests-2.7 (#6113)
drop 75b54171ae Travis - fix test failures observed (#6119)
drop 4be99fe971 api: Allow updating VM settings when custom contrained offering is used (#6136)
drop f8b648b938 Fix migration of VM with volume on Ubuntu (#6116)
drop fb43076f9e Fix linux native bridge for SUSE in cloudutils (#6134)
drop f4b9ab034b UI: fix create l2 network offering with userdata (#6174)
drop bcd1a3274a api: Fix reset configuration (#6168)
drop 908f594f00 configDrive: Fix failure to delete (unstarted) VM (#6146)
drop a69ab3b28f Ensure configdrive path is edited properly during live migration (#6173)
drop ee27708ffb SAML: replace first number with random alphabet if request ID starts with a number (#6165)
drop 66a6671e0b ui,refactor: fix missing label in update network form (#6181)
drop 47454eca7d VR: add '-m <protocol>' for tcp or udp protocol (#6188)
drop c61ea9f96d VR: Do not add iptables rules for the revoked ip addresses (#6189)
drop e7071ec196 server: increment deviceid while importing vm data volumes (#6123)
drop ee2ded8200 potential null pointer in condition; AYAI9l8k5Irk9_td-cXb (#6237)
drop b6072fc826 Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service (#6234)
drop 19a7774cab VR: add rules for traffic between static nat and private gateway static routes (#6153)
drop 91a5f0e285 server: honor global setting system.vm.default.hypervisor as first option when deploy VRs (#6160)
pick 42a92dcdd3 Extract the IO_URING configuration into the agent.properties (#6253)
drop fbf77978e1 Fix: Allow disabling the login attempts mechanism for disabling users (#6254)
drop bbb4ffa593 UI: fix dedicate public ip range to domain (#6258)
drop 365966dd0a UI: Fix custom unconstrained for a zone does not show CPU speed (#6285)
drop b2338f7158 Updated reset configuration, to return the updated config value in the response (#6284)
drop 5fa8fa5580 Fix upload volume format (#6297)
drop efb1f2b719 UI: Fix templates page redirection after delete job is finished (#6345)
drop 556f9dac0f ui: Network offerings not listed if listVPCs not available in the account Role (#6354)
drop d373f973ba Update VM name, when the new name provided in updateVirtualMachine API in different case. (#6379)
drop 006473ca19 Log exception on keystore build for custom certificate (#6394)
pick f9aa91c642 [KVM] Disable IOURING by default on agents
pick fad8a5752c Refactor
pick 1d6d1b09fb Remove agent property for iouring
pick c11cd1bf4e Restore property
pick 261e275c38 Refactor suse check and enable on ubuntu by default
pick 0e76ca64d7 Refactor irrespective of guest OS
pick 9e0574683d Improvement
pick b349e75614 Logs and new path
pick a74a4a83df Refactor condition to enable iouring
pick 09c1f44093 Improve condition
pick a3ab1a33e4 Refactor property check
pick 88b3666d8c Improvement
pick 0c8a6aa03a Doc comment
pick 3835e8b8cf Extend comment

# Rebase cad9332082..3835e8b8cf onto 0c8a6aa03a (53 commands)

:wq

and also need to fix VNC password as #6402 checkout #6402 and rebase to fix-iouring only pick #6402, drop others commit
$ git checkout -b fix-vncpassword 363a2cff8231a31c3c9def329805910e43358b1d
$ git rebase -i fix-iouring
:1,39s/^pick/drop/g

drop bc70535ee5 Updating pom.xml version numbers for release <IP_ADDRESS>-SNAPSHOT
drop c366511294 UI: Missing message on VM import for not found networks (#6055)
drop 2820a36f86 Check the network access when deploying VM in Advanced Security Group. (#6050)
drop 84f19d8f36 UI: update vm with userdata (#6051)
drop e4d70d4214 UI: Fix Autogenview cleared resource (#6066)
drop 6b913a76cf ui: Add user initials as avatar if no icon present (#6070)
drop 7fbfd4c6ea UI: Fix navigation to domains (#6072)
drop c2bcad8571 ui: Set icon to osdisplayname to avoid multiple api calls (#6075)
drop 00c1bdb365 UI: Reload page on closing Bulk Action modal (#6077)
drop 3a456f1b31 server: mark volume snapshots as Destroyed if it does not exist on primary and secondary storage when delete a volume (#6057)
drop 067c1de080 Fix get upload params NPE (#6079)
drop 6ff378f6a3 travis: run nosetests-2.7 (#6113)
drop 75b54171ae Travis - fix test failures observed (#6119)
drop 4be99fe971 api: Allow updating VM settings when custom contrained offering is used (#6136)
drop f8b648b938 Fix migration of VM with volume on Ubuntu (#6116)
drop fb43076f9e Fix linux native bridge for SUSE in cloudutils (#6134)
drop f4b9ab034b UI: fix create l2 network offering with userdata (#6174)
drop bcd1a3274a api: Fix reset configuration (#6168)
drop 908f594f00 configDrive: Fix failure to delete (unstarted) VM (#6146)
drop a69ab3b28f Ensure configdrive path is edited properly during live migration (#6173)
drop ee27708ffb SAML: replace first number with random alphabet if request ID starts with a number (#6165)
drop 66a6671e0b ui,refactor: fix missing label in update network form (#6181)
drop 47454eca7d VR: add '-m <protocol>' for tcp or udp protocol (#6188)
drop c61ea9f96d VR: Do not add iptables rules for the revoked ip addresses (#6189)
drop e7071ec196 server: increment deviceid while importing vm data volumes (#6123)
drop ee2ded8200 potential null pointer in condition; AYAI9l8k5Irk9_td-cXb (#6237)
drop b6072fc826 Allow expunging a VM on a deleted host when using host cache and ConfigDrive userdata service (#6234)
drop 19a7774cab VR: add rules for traffic between static nat and private gateway static routes (#6153)
drop 91a5f0e285 server: honor global setting system.vm.default.hypervisor as first option when deploy VRs (#6160)
drop fbf77978e1 Fix: Allow disabling the login attempts mechanism for disabling users (#6254)
drop bbb4ffa593 UI: fix dedicate public ip range to domain (#6258)
drop 365966dd0a UI: Fix custom unconstrained for a zone does not show CPU speed (#6285)
drop b2338f7158 Updated reset configuration, to return the updated config value in the response (#6284)
drop 5fa8fa5580 Fix upload volume format (#6297)
drop efb1f2b719 UI: Fix templates page redirection after delete job is finished (#6345)
drop 556f9dac0f ui: Network offerings not listed if listVPCs not available in the account Role (#6354)
drop d373f973ba Update VM name, when the new name provided in updateVirtualMachine API in different case. (#6379)
drop 006473ca19 Log exception on keystore build for custom certificate (#6394)
drop b62b5c96e8 Prevent NPE on reboot stopped VM and startVM output with null displayname (#6397)
pick 363a2cff82 Backport: kvm: truncate vnc password to 8 chars (#6244) (#6402)

# Rebase bab903af16..363a2cff82 onto 363a2cff82 (40 commands)

:wq

build jar files and copy to your agent
$ mvn install
$ scp agent/target/cloud-agent-<IP_ADDRESS>.jar leo@kvm-agent:
$ scp plugins/hypervisors/kvm/target/cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar leo@kvm-agent:

You need to update CloudStack to <IP_ADDRESS> first Then you can replace those two jar file and restart agent Of course you can backup the original jar file if you want
$ sudo mv /usr/share/cloudstack-agent/lib/cloud-agent-<IP_ADDRESS>.jar /usr/share/cloudstack-agent/lib/cloud-agent-<IP_ADDRESS>.jar.origin
$ sudo mv /usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar /usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar.origin
$ sudo cp ~/cloud-agent-<IP_ADDRESS>.jar /usr/share/cloudstack-agent/lib/
$ sudo cp ~/cloud-plugin-hypervisor-kvm-<IP_ADDRESS>.jar /usr/share/cloudstack-agent/lib/
$ sudo systemctl restart cloudstack-agent.service

You can use my jar file if you don't mind CloudStack-<IP_ADDRESS>-agent_patch_for_rockylinux8.6.zip

Great, That fix the issue 👍🏼

Thank you, I'm facing other issues, I will try to fix it
Why there no discourse community forum or discord server that will help on troubleshooting and learn Cloudstack

finally everything fixed and Cloudstack work great 👍🏼
thanks all


fix image vulnerability

fix this vulnerability in kraken-tools image: https://quay.io/repository/samsung_cnct/kraken-tools/image/cc736fb576a7bc62f6e8d295de24b657064fa38951a2d607136b3e5c52b5f458?tab=vulnerabilities&fixable=true

This is passing 100%, version 0.7, and latest as of Nov 3 look good.
https://quay.io/repository/samsung_cnct/kraken-tools?tab=tags


Paper doll renders with low frame rate

The mini player model is very choppy, this is because InventoryScreen::renderEntityInInventory sets the partialTick parameter to 1.0F, instead of using the actual value for the current render tick.
Just duplicating the vanilla method and passing the current partialTick value is enough to make the player model render smoothly.
https://github.com/Wilyicaro/Legacy-Minecraft/blob/e491d807db7c0666926b8ad03a571a220c382ce4/common/src/main/java/wily/legacy/mixin/GuiMixin.java#L130

It's supposed to be like this, as this is also what it does in legacy console

And unfortunately, if I pass the current partialTick to a duplicate of the InventoryScreen::renderEntityInInventory method, this will need a workaround, because it needs to be changed certain rotations of the LocalPlayer, which causes a "jitter" in the model when rendering with the current partialTick in the EntityRenderDispatcher::render method


Completed task from 1 to 5

@juliankohlman Finished all tasks that were overdue because of being unable to assist class Saturday.

Great

Very thorough and clear answers to questions
Completed tasks 1-5.
Both index.html/about.html match the design files

Requested Improvements

Use markdown to format answers.md file
In the future try and commit a little more often.
Create a develop branch

Add some hover effects to links on nav
Merge develop branch with master

Pin footer to bottom of index/about.html

Questions

Saturday during our 1:1, we'll just touch on some TrainingKit Objectives for UI-I, UI-II, and Git

Rating: {1-3}

3

Git-Comments

[x] Utilizing Meaningful Commit Messages
[x] Committing Often: On Every Breakthrough or Feature Completion.
[ ] Submitted a Pull Request with your First & Last Name in the title.

Comments
Comments

I am glad that you are using meaningful commit messages, and committing often.
I would like to see you work on future projects towards using the methods mentioned above. It will help in your future life as a developer and will help you better leverage git version control.
 If you have not yet tried it
Using git from the command line with bash you can do the following:

git commit -m "Implemented Desktop Break-Points
> CSS Break-Points with mediaquery at 1024px max-width
> Tested in Chrome, Firefox
> Fluid transition between breakpoints."

Press Enter when you want a new line. When you see the `>` symbol this means that you have pressed Enter and there is a new line. Remember to close your quotes at the end of your message and press Enter to submit the command.


Add range support

Range support by adding options.startOffset and options.endOffset

Would it be okay to drop 0.8 travis check? I have this running on 7.4.0. Also can I add some newer nodes to travis? Which do you suggest?

Awesome work @Climax777! Published as v0.1.3 (I guess we could bump this up to v1.0.0, it's been pretty stable?)
I'd be happy to add you as a collaborator if you would like?

@jeffbski sure, I'd be happy to help out. Thanks for the package!
On Tue, 18 Apr 2017, 17:56 Jeff Barczewski<EMAIL_ADDRESS>wrote:

Awesome work @Climax777 https://github.com/Climax777! Published as
v0.1.3 (I guess we could bump this up to v1.0.0, it's been pretty stable?)
I'd be happy to add you as a collaborator if you would like?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jeffbski/redis-rstream/pull/3#issuecomment-294890333,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAu7nR5BYn2yHoyVHlaMH0SrfqgYBrZ4ks5rxN0lgaJpZM4NAax1
.


Fix custom webpack config warnings

Issue: Fix the #3981 + a bit more refactoring
What I did
I've refactored loadCustomWebpackConfig.js and now we have a serverRequire.js that will be able to load client's scripts based on their preferred compiler on server (Currently the only usecase is webpack.config, but I assume will be more, for example: presets, single config file, etc')

Any ETA on when this fix will be released? 😇


Update the provider version to use minimum 1.65.0

In the CBR module, the provider constraints are updated to >= 1.65.0, hence the need to update.

Available in release https://github.com/terraform-ibm-modules/terraform-ibm-event-notifications/releases/tag/v1.6.6, closing this.


Primary Key column has always NOT NULL after using update command

Descripton:
Primary Key column is always not nullable after using update command but according to documentation on Cloud Spanner:
https://cloud.google.com/spanner/docs/schema-and-data-model#notes_about_key_columns
Primary Key columns can be defined to store NULLs.
Steps to reproduce:
Use changeLog as below:
databaseChangeLog:
- changeSet:
    id:<PHONE_NUMBER>323-1
    author: author
    changes:
    - createTable:
        columns:
        - column:
            constraints:
              primaryKey: true
              primaryKeyName: PRIMARY_KEY
            name: FIRST_1
            type: STRING(1024)
        - column:
            constraints:
              nullable: false
              primaryKey: true
              primaryKeyName: PRIMARY_KEY
            name: FIRST_2
            type: STRING(MAX)
        - column:
            constraints:
              primaryKey: true
              primaryKeyName: PRIMARY_KEY
            name: FIRST_3
            type: INT64
        - column:
            name: FIRST_4
            type: INT64
        tableName: FIRST_TABLE

In command:
liquibase --url="jdbc:cloudspanner:/projects/<project_name>/instances/<instance_name>/databases/<db_name>" --changeLogFile=changeLog.yml update
After that in schema you can see all Primary Keys as a not nullable but should be only FIRST_2 column.
Even adding "nullable: true" does not help.

Hmm... The default behavior in Liquibase seems to be to assume that any primary key column should also be NOT NULL:
https://github.com/liquibase/liquibase/blob/99308bdebc8cd45142c3e2b8824838d9bd22ea21/liquibase-core/src/main/java/liquibase/statement/core/CreateTableStatement.java#L121


Rules not being overridden?

I'm checking parity between xo and eslint warnings and I get 58 warning / errors with xo and 1 with eslint. Obviously, most of these are because xo has its own rules.  I'd like to override all of these with my own rules from my shareable config, but can't seem to get that to happen, even individual rules are not being overridden when I use my custom shareable config.
//package.json

  "xo": {
    "envs": [
      "node",
      "mocha"
    ],
    "extend": "myConfig",
    "rules": {}
  }

can I nullify the rules, like above?
Thanks!

You can't nullify the default rules, only extend. The whole point of using XO is embracing most if not all the rules, and override the ones you don't agree with. If you want to replace all the rules you're probably better of just using ESLint directly.
As for:

but can't seem to get that to happen, even individual rules are not being overridden when I use my custom shareable config.

That should be working. It's tested here: https://github.com/sindresorhus/xo/blob/c8283cd0df59a9a8ef57b74033cf0184d2a0b735/test/api.js#L43

Gotcha, Thanks Sindre.  Not sure what I was doing wrong, I'll figure it out.


Avoid showing cargo-dist release body

cargo-dist's release body, which shows up when you have changelogs enabled in oranda, as well as cargo-dist integration, but don't actually have a CHANGELOG/RELEASE.md, break the page layout by going over the viewport's width. I'm not sure if this is a bug (as the defined fallback for this case is an empty string), but we should avoid showing this either way.

probably related to #253

@shadows-withal can you add screencaps and/or a repro repo- this is interesting and i'm wondering what exactly it does. when you have cargo-dist enabled, the changelog feature uses the announcement changelog, which is different than the release body - but i am not sure i have tested this with a repo that doesn't have a changelog/releases.md

will do as soon as i remember what this issue is about

circling back up on this, @shadows-withal

looks like this was fixed since it happened, building my test site where it occurred with the latest main doesn't make it show anymore :+1:


Any news regarding the patent?

In Readme is mentioned, that there is an ongoing patent application. Do you happen to have any news on that? In which countries in the world was the patent request submitted?
I'd like to use this implementation (or at least get some inspiration) e.g. in Nuklear or gg, but am afraid of patents :cry:.

A patent application was filed in the USA (February 1, 2018) and recently granted (August 6, 2019). You can look in the USPTO database for "Method for rendering resolution-independent shapes directly from outline control points". As for other countries, I do not know. You should definitely not be using this if you intend to redistribute your work.
Now, there are many alternatives whose viability will greatly vary based on your tech/performance/quality/legal needs:

Multi-channel signed distance fields: https://github.com/Chlumsky/msdfgen/files/3050967/thesis.pdf
nVidia's path rendering OpenGL extension: https://www.khronos.org/registry/OpenGL/extensions/NV/NV_path_rendering.txt
16xAA font rendering with coverage masks: https://www.superluminal.eu/2018/10/29/16xaa-font-rendering-using-coverage-masks-part-i/
Real-Time Analytic Antialiased Text for 3-D Environments: https://www.highperformancegraphics.org/wp-content/uploads/2019/session1/real-time_analytic_antialiased_text.pdf
https://www.superluminal.eu/2018/10/29/16xaa-font-rendering-using-coverage-masks-part-i/

Wow, thanks for the thorough answer! I'll take a look which of those "alternatives" are easier to implement or have wider cross-platform potential.


Fix size calculation logic in PooledByteBufAllocator

Motivation:

When set "io.netty.allocator.pageSize" to 4MiB or larger, the PooledByteBufAllocator.DEFAULT instance throws java.lang.IllegalArgumentException in the construction stage:

public class TestDemo {
    static {
        System.setProperty("io.netty.allocator.pageSize", (1 << 22) + ""); // Set page size to 4MiB
    }

    public static void main(String[] args) {
        System.out.println(PooledByteBufAllocator.DEFAULT.metric()); // Throws `pageSize` overflow exception.
    }
}

The pageSize for PooledByteBufAllocator.DEFAULT is initialized and validated in the static block, and it is supposed to resolve the overflow problem in the static block initialization stage.

After the static block initialization finished, the PooledByteBufAllocator.DEFAULT should not throw pageSize overflow exception in the construction stage.

Some size-validation logics in the PooledByteBufAllocator's constructor and static block can be optimized.
For example:
Some validation code in the constructor should move into method validateAndCalculatePageShifts().

Modification:
Correct the size validation logic.
Result:
Fix the problem above, and optimize size-validation related code.

The reason you're getting an error when using a 4MiB page size and the default maxOrder of 9 is because it causes an int32 overflow. To fix this, you can manually set the io.netty.allocator.maxOrder to a value less than 9.
By the way, Could you tell me why you need such large page and chunk sizes? AFAIK, using such large sizes can often hurt performance and make it harder to manage memory efficiently, with the current implementations.

using such huge chunk size almost certainly can degrade performance and make it harder to manage memory efficiently.

I agree with you above. But I take this issue from a pure code-logic perspective.

for the issue # 1. I think it is correct behavior. The reason you're getting an error when using a 4MiB page size and the default maxOrder of 9 is because it causes an int32 overflow. To fix this, you can manually set the io.netty.allocator.maxOrder to a value less than 9.

The following code piece from PooledByteBufAllocator 's static initializer block:
try {
      validateAndCalculateChunkSize(DEFAULT_PAGE_SIZE, defaultMaxOrder);
 } catch (Throwable t) {
      maxOrderFallbackCause = t;
      defaultMaxOrder = 9;
 }

The method validateAndCalculateChunkSize(DEFAULT_PAGE_SIZE, defaultMaxOrder) already did the overflow check and rise an Exception, then the catch block catch the Exception, and fall back the defaultMaxOrder to 9, at this point, the defaultMaxOrder = 9 is supposed to resolve the overflow Exception, but  it actually not, so the following code goes overflowed:
// `defaultChunkSize became` became a negative value.
final int defaultChunkSize = DEFAULT_PAGE_SIZE << DEFAULT_MAX_ORDER;

So, we should solve the overflow problem in the catch block.
The overflow Exception in the PR's code sample is triggered in the PooledByteBufAllocator 's constructor, but it does not change the fact that the default static initializer's overflow problem should be take care earlier, which means in the static block initializer.

I'm unsure if the proposed approach is the most suitable one for this situation. It seems that setting a large value for pageSize also affects the default value of maxOrder. Perhaps we should explore alternative solutions that include a fallback to the default page size.

I'm unsure if the proposed approach is the most suitable one for this situation. It seems that setting a large value for pageSize also affects the default value of maxOrder. Perhaps we should explore alternative solutions that include a fallback to the default page size.

Based on current PR's committed code, this can be done by simply change following code in method PooledByteBufAllocator.validateAndCalculatePageShifts():
if (pageSize > MAX_CHUNK_SIZE) {
        throw new IllegalArgumentException("pageSize: " + pageSize + " (expected: " + MAX_CHUNK_SIZE + ')');
}

changes to :
if (pageSize > MAX_PAGE_SIZE) {
        throw new IllegalArgumentException("pageSize: " + pageSize + " (expected: " + MAX_PAGE_SIZE + ')');
}

We can set MAX_PAGE_SIZE to a smaller & reasonable value, if pageSize exceed MAX_PAGE_SIZE, a IllegalArgumentException exception will be thrown up and be caught in PooledByteBufAllocator's static block code, then the pageSize will fallback to default page size.

Thank you for taking the time to reply to my comment. I would like to kindly request that we wait for the maintainers to weigh in. Their expertise and input could provide valuable insights into this issue.

I think it's better to fail early and explicitly, as an exception from a static initializer would.

Agree, then we can remove the two try...catch blocks in the static initializer, and let the exception throws once it failed the validation.

Let us close this for now... If someone is feeling strong about it we can reopen


Dataset

I used the dataset you organized, but I get an error when sampling, do I need to do anything further?

 I used the dataset you organized, but I get an error when sampling, do I need to do anything further?

It seems that you forgot to unzip the folder B_norm.zip and C_norm.zip.


Add GetBrowseNodes operation

Implement the GetBrowseNodes operation
https://webservices.amazon.com/paapi5/documentation/getbrowsenodes.html

Hi,
Thanks for the good work. I noticed the Sales Rank in BrowseNodes always return 0. Is this a bug? Or am I doing something wrong.
var searchRequest = new SearchRequest
{
Keywords = "9780060004873|9780999120842",
SearchIndex = SearchIndex.All,
Resources = new[]
{
"Images.Primary.Medium",
"BrowseNodeInfo.BrowseNodes",
"BrowseNodeInfo.WebsiteSalesRank",
"ItemInfo.ExternalIds",
"ItemInfo.Title",
"ItemInfo.ByLineInfo",
"ItemInfo.ManufactureInfo",
"ItemInfo.ProductInfo",
"ItemInfo.Classifications",
"Offers.Listings.Price",
"Offers.Listings.MerchantInfo"
},
};

Hi @Jammasternam,
You are trying to use the SearchItemsAsync method, which I think I didn't change with my pull request.
Anyways, I've tested the same values as you did in my solution and You're right. The response coming from Amazon actually has SalesRanks and our serialized objects don't, so there seems to be a bug.
Have you found this error with the version from my fork or with the currently available NuGet package?
I'd recommend you to open an own issue for the SearchItemsAsync operation and the bug you found.


TimeSinceEpoch is not correct

It seems that the time should multiply time.Millisecond (1e6) rather than time.Second (1e9)
The corresponding code in cdproto are https://github.com/chromedp/cdproto/blob/8501c74e8228308e0dbd4aedb1f24ea1c103dd6e/network/types.go#LL1027
and
https://github.com/chromedp/cdproto/blob/8501c74e8228308e0dbd4aedb1f24ea1c103dd6e/cdp/types.go#L742
The following code snapshot is what I tried
package main

import (
	"fmt"
	"time"
)

func main() {
	// a response time from cdp log
	t := 1.685446691349539e+12

	// correct time 2023-05-30 19:38:11.349539072 +0800 CST
	fmt.Println(time.Unix(0, int64(t*float64(time.Millisecond))))

	// wrong time 1677-09-21 08:12:43.145224192 +0800 CST
	fmt.Println(time.Unix(0, int64(t*float64(time.Second))))
}

@RickyWei Thanks for bringing this to my attention. I'll look at this. I don't recall specifically why I implemented this as I did at the time, except to mention that the behavior has at time been incorrect on Chromium's code base, and documented incorrectly.
In the interim, I'd suggest just multiplying the time value if you need the different value at the moment.

@RickyWei I've changed this, but I'm not yet going to update this on the chromedp package. In the interim, you can update to the latest cdproto package. I will need to do some testing against the many other places TimeSinceEpoch is used to determine if this is sound/correct, or it's only with the network.Response type where this differs.

Good morning Ken!
I checked the Network.requestWillBeSent and Network.responseReceived events, only response.responseTime in Network.responseReceived contains the number of milliseconds (which is conflicted with the document).

Network.requestWillBeSent

[x] wallTime

Network.responseReceived

[ ] response.responseTime
[x] response.securityDetails.validFrom
[x] response.securityDetails.validTo

Network.requestWillBeSent (truncated)
{
  "requestId": "4393E3633F10D1113C22E7A368A550C4",
  "loaderId": "4393E3633F10D1113C22E7A368A550C4",
  "documentURL": "https://bing.com/",
  "request": {
    "url": "https://bing.com/",
    "method": "GET"
  },
  "timestamp": 13720.870149,
  "wallTime":<PHONE_NUMBER>.297646
}

Network.responseReceived (truncated):
{
  "type": "Document",
  "response": {
    "url": "https://cn.bing.com/",
    "status": 200,
    "statusText": "",
    "responseTime":<PHONE_NUMBER>757.947,
    "securityDetails": {
      "protocol": "TLS 1.2",
      "keyExchange": "ECDHE_RSA",
      "keyExchangeGroup": "P-384",
      "cipher": "AES_256_GCM",
      "certificateId": 0,
      "subjectName": "www.bing.com",
      "issuer": "Microsoft RSA TLS CA 02",
      "validFrom":<PHONE_NUMBER>,
      "validTo":<PHONE_NUMBER>
    }
  }
}

@ZekeLu much appreciated. I haven't had time to go through all the places where there are timestamps in the cdproto code to check everything. Obviously before tagging anything, this should be done, and confirmed. My greatest concern is with the input.* events, as those are likely being used by actual complex programs.

I will try to go through all of them today.

@ZekeLu you're amazing man. I appreciate the help!

@ZekeLu @kenshaw Thanks for all of your effort. I really appreciate it. 👍👍

Below is the full list of the places where Network.TimeSinceEpoch or Input.TimeSinceEpoch are used.
Among them, only Response.responseTime does not carry the correct value (in respect of the CDP doc).
Please note that I'm not familiar with Chromium, and the listed Chromium Implementation could be wrong.
But I think it's enough for finding out whether the implementation is consistent with the CDP doc.
Network.TimeSinceEpoch

CDP doc: Network.TimeSinceEpoch
cdproto package: cdp.TimeSinceEpoch

Seq
CDP doc
cdproto package
Chromium
Chromium Implementation

1
NetworkResponseresponseTime
networkResponseResponseTime
BuildObjectForResourceResponse
response.ResponseTime().ToJsTimeIgnoringNull()

2
NetworkSecurityDetailsvalidFromvalidTo
networkSecurityDetailsValidFromValidTo
BuildSecurityDetails
ssl_info.cert->valid_start().ToDoubleT()ssl_info.cert->valid_expiry().ToDoubleT()

3
NetworkrequestWillBeSentwallTime
networkEventRequestWillBeSentWallTime
InspectorNetworkAgent::WillSendRequestInternal
base::Time::Now().ToDoubleT()

4
PageOriginTrialTokenexpiryTime
cdpOriginTrialTokenExpiryTime
CreateOriginTrialToken
blink_trial_token.expiry_time().ToDoubleT()

5
PageFrameResourcelastModified
pageFrameResourceLastModified
InspectorPageAgent::BuildObjectForResourceTree
last_modified.value().ToDoubleT()

6
PageScreencastFrameMetadatatimestamp
pageScreencastFrameMetadataTimestamp
BuildScreencastFrameMetadata
base::Time::Now().ToDoubleT()

7
SecurityCertificateSecurityStatevalidFromvalidTo
securityCertificateSecurityStateValidFromValidTo
CreateCertificateSecurityState
state.certificate->valid_start().ToDoubleT()state.certificate->valid_expiry().ToDoubleT()

8
StorageinterestGroupAccessedaccessTime
storageEventInterestGroupAccessedAccessTime
StorageHandler::OnInterestGroupAccessed
access_time.ToDoubleT()

9
StoragesharedStorageAccessedaccessTime
storageEventSharedStorageAccessedAccessTime
StorageHandler::NotifySharedStorageAccessed
access_time.ToDoubleT()

10
StorageSharedStorageMetadatacreationTime
storageSharedStorageMetadataCreationTime
SendSharedStorageMetadata
metadata.creation_time.ToDoubleT()

11
StorageStorageBucketInfoexpiration
storageBucketInfoExpiration
BuildBucketInfo
bucket.expiration.ToDoubleT()

12
BackgroundServiceBackgroundServiceEventTimestamp
backgroundserviceEventTimestamp
ToBackgroundServiceEvent
timestamp.ToJsTime() / 1'000

13
NetworkReportingApiReporttimestamp
networkReportingAPIReportTimestamp
NetworkHandler::BuildProtocolReport
(report.queued - base::TimeTicks::UnixEpoch()).InSecondsF())

14
PerformanceTimelineLargestContentfulPaintrenderTimeloadTime
performancetimelineLargestContentfulPaintRenderTimeLoadTime
BuildEventDetails
ToProtocolTime(timeOrigin, lcp.renderTime())ToProtocolTime(timeOrigin, lcp.loadTime())

15
PerformanceTimelineLayoutShiftlastInputTime
performancetimelineLayoutShiftLastInputTime
BuildEventDetails
ToProtocolTime(timeOrigin, ls.lastInputTime())

16
PerformanceTimelineTimelineEventtime
performancetimelineTimelineEventTime
BuildProtocolEvent
ToProtocolTime(timeOrigin, entry->startTime())

17
EmulationsetVirtualTimePolicyinitialVirtualTime
emulationSetVirtualTimePolicyParamsInitialVirtualTime
InspectorEmulationAgent::setVirtualTimePolicy
base::Time::FromDoubleT(initial_virtual_time.fromJust())

18
NetworksetCookieexpires
networkSetCookieParamsExpires
NetworkHandler::SetCookie
MakeCookieFromProtocolValues

19
NetworkCookieParamexpires
networkCookieParamExpires
NetworkHandler::SetCookies
MakeCookieFromProtocolValues

ToProtocolTime:
double ToProtocolTime(DOMHighResTimeStamp timeOrigin,
                      DOMHighResTimeStamp time) {
  return time ? ConvertDOMHighResTimeStampToSeconds(timeOrigin + time) : 0.0;
}

MakeCookieFromProtocolValues:
  if (expires >= 0) {
    expiration_date =
        expires ? base::Time::FromDoubleT(expires) : base::Time::UnixEpoch();
  }

Input.TimeSinceEpoch

CDP doc: Input.TimeSinceEpoch
cdproto package: input.TimeSinceEpoch

Seq
CDP doc
cdproto package
Chromium
Chromium Implementation

1
InputdispatchKeyEventtimestamp
inputDispatchKeyEventParamsTimestamp
InputHandler::DispatchKeyEvent
GetEventTimeTicks(timestamp)

2
InputemulateTouchFromMouseEventtimestamp
inputEmulateTouchFromMouseEventParamsTimestamp
InputHandler::EmulateTouchFromMouseEvent
GetEventTimeTicks(timestamp)

3
InputdispatchMouseEventtimestamp
inputDispatchMouseEventParamsTimestamp
CreateWebMouseEvent
GetEventTimeTicks(timestamp)

4
InputdispatchTouchEventtimestamp
inputDispatchTouchEventParamsTimestamp
CreateWebTouchEvents
GetEventTimeTicks(timestamp)

GetEventTimeTicks
base::TimeTicks GetEventTimeTicks(const Maybe<double>& timestamp) {
  // Convert timestamp, in seconds since unix epoch, to an event timestamp
  // which is time ticks since platform start time.
  return timestamp.isJust() ? base::Seconds(timestamp.fromJust()) +
                                  base::TimeTicks::UnixEpoch()
                            : base::TimeTicks::Now();
}

We can add some code to correct responseTime for the cdproto users. But since this issue is not noticed for a long time, I think it's OK to leave it as what it is now and just wait for the upstream (Chromium) to fix it. BTW, I have just reported the issue to the Chromium team: https://bugs.chromium.org/p/chromium/issues/detail?id=1451082.

@ZekeLu in your opinion should we revert/change back the Millisecond resolution on pdlgen/cdproto-gen?

@ZekeLu There are other possibilities as well: we can manually mark certain fields in the generator's fixup package, and add an additional field to the TimeSinceEpoch that tells the resolution. Alternately, I can write some code that tries to "guess" what the time is supposed to be, based on how large/small the value is.

Hi Ken. Yes I think we should revert the Millisecond resolution on pdlgen because the initial implementation is correct. And only responseTime in the Response type is wrong.

Alternately, I can write some code that tries to "guess" what the time is supposed to be, based on how large/small the value is

This is what came up to my mind too.
I think it's better to limit the change to the responseTime field only. But I can not find a easy fix to pdlgen to do this.

I just pushed a commit to cdproto that uses a different type just for the responseTime field (cdp.TimeSinceEpochMilli). It has the same interface as cdp.TimeSinceEpoch, but uses millisecond resolution. Should the Chromium code change, this will need to be change accordingly.
@ZekeLu if you can check this out, assuming all the various time times are correct, we should tag a new version of chromedp with the updated cdproto dependency.

Thank you @kenshaw ! I have sent a PR to upgrade cdproto. And I have modified an existing test to check the responseTime so that we will be noticed once chromium fixed the issue.
If you're OK with the PR, I will tag a new release as 0.9.2.

https://bugs.chromium.org/p/chromium/issues/detail?id=1451082 is closed as "WontFix". So it's good to go with cdp.TimeSinceEpochMilli. I'm going to tag a new release.


fix: types to devDependencies

Please merge this back as the types should be only dev dependencies.

Coverage remained the same at 100.0% when pulling e702189fbf837267d43f645ae87d0739eea33f1c on B3zo0:master into d272b76cb3f247b27261feae8a1642a8b827273c on Romakita:master.


Implement nested errors in errorMessagesForSchema

Purpose
When a domain function received nested input data, errorMessagesForSchema should mirror that structure so we do not lose nested error information.
This should address #24

@danielweinmann I don't think this will cause any issues for remix-forms, but I haven't done any integration testing. Thoughts?

@diogob if you feel like waiting for my opinion here I'd ask you to wait for my return on the 29th bc I need to wrap my head around it to understand the implications.
That said I trust you guys to take that decision regardless bc I know some changes must happen as I could see that request coming when we designed the shallow version ;)

#31 seems to be a simpler approach that uses only the data contained in SchemaError thus avoiding inspection of the schema to generate the nested structure.


Include non-required header and query param constants

Storage, and other services, contain optional header and query params
that are constants so include them in codegen.
Replaced 'TODO' type descriptions with actual descriptions.  They aren't
actually used but cleans up noise when searching for TODO comments.

Fixes https://github.com/Azure/autorest.go/issues/294


Slideshow

Currently, the primary image changes only when there's a new prompt submission. It can be made more attractive by showing previously generated images.
This can either be something like recent 20 images, or we can add some way to vote & feature popular ones.

https://github.com/tinkerhub/tinkerspace-stable-diffusion/pull/14

https://user-images.githubusercontent.com/36979685/211189051-c0548a39-bc13-4ae5-8561-250057458fc3.mp4


Add support for testing code which needs to run in background function

In the tests for https://github.com/AllenInstitute/MIES we have the need to test data acquisition which runs in a background function. I've hacked a solution together but that is, well, hacky. See run in UTF_HardwareMain.ipf.
Currently we have two test cases, the first sets upDAQ and starts it and the second does the tests. The background function WaitUntilDAQDone_IGNORE starts the second test case after DAQ has finished.
Current issues with that approach:

It is outside of the UTF
It creates one JUNIT xml file for each test case
We can't use autodiscovery of test cases but have to manually list them
Fatal errors in one of the test cases don't stop the full run

A proper support would entail:

one JUNIT XML file per Igor experiment
full autodiscovery support
user configurable condition when the second test case needs to be run

These two testcases are run in order, runTest stops after the first and continues after the listed bkg functions to RegisterToBeMonitoredBkgFunctions are done via a monitoring background task.
Function TPDuringDAQOnlyTP_1()

	STRUCT DAQSettings s
	InitDAQSettingsFromString(s, "DAQ_MD1_RA0_IDX0_LIDX0_BKG_1_RES_1")
	AcquireData(s, preAcquireFunc=TPDuringDAQOnlyTP_IGNORE)

	RegisterToBeMonitoredBkgFunctions("Testpulse;TestpulseMD;...", mode = <AND or OR>)
End

Function TPDuringDAQOnlyTP_2()
End

One possible solution:

New flag for runtest to stop/continue its run
New flag for finding test cases with numeric suffix which will then be run in that order

Closed via #67.


ModuleNotFoundError: scipy

compiling this:
import sys

import scipy

if(__name__ == '__main__'):
    evals, V = scipy.linalg.eigh([[1, 0], [1, 0]], overwrite_a=True, turbo=True)

    sys.exit(1)

fails with
Traceback (most recent call last):
  File "C:\Users\geoff\Code\rom\OASISA~1\SSCCE~1.DIS\sscce.py", line 3, in <module>
ModuleNotFoundError: No module named 'scipy'

python -m nuitka --version

1.7rc2
Commercial: None
Python: 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)]
Flavor: CPython Official
Executable: C:\Python311\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10
Version C compiler: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\bin\Hostx64\x64\cl.exe (cl 14.3).

nuitka installed via pip globally
I will be purchasing the commercial package if i can get this proof-of-concept working!

I believe this is a problem with caching. I believe it was able to compile a project with scypi once correctly.
I may be misremembering, but this is whats happened:
sequence of events:

built my POC (of my whole project, using numpy, scipy, etc) with nuitka
I ran the program and got our argparse screen
I started wrapping the nuitka calls in a script to do things like versions, copyrights, etc
I deleted the 255mb output.dist folder,
called my script, it produced a 67mb output folder, missing the scipy output binaries and more
went back and ran the simple original nuitka bigproject --standalone, it produced a 67 mb folder

Thus I believe the problem is in caching logic; some part of nuitka seems to have understood its need to compile/copy scipy output previously, it did, set some cache, and now no longer believes it needs to do that anymore.
To try and address this:

I ran nuitka bigproject --standalone --clear-cache=all => no affect, still produces binary without scipy folder
created the sscce.py file above, created this issue.

If there is some kind of cache treatment logic stronger than --clear-cache please let me know.

full output:
C:\Users\geoff\Code\rom\OASISAIROM [main ≡ +2 ~1 -0 | +2 ~3 -0 !]> nuitka .\sscce.py --standalone
Nuitka-Options:INFO: Used command line options: .\sscce.py --standalone
Nuitka:INFO: Starting Python compilation with Nuitka '1.7rc2' on Python '3.11' commercial grade 'not installed'.
Nuitka:INFO: Completed Python level compilation and optimization.
Nuitka:INFO: Generating source code for C backend compiler.
Nuitka:INFO: Running data composer tool for optimal constant value handling.
Nuitka:INFO: Running C compilation via Scons.
Nuitka-Scons:INFO: Backend C compiler: cl (cl 14.3).
Nuitka-Scons:INFO: Backend linking program with 6 files (no progress information available for this stage).
Nuitka-Scons:INFO: Compiled 6 C files using clcache with 3 cache hits and 3 cache misses.
Nuitka:INFO: Keeping build directory 'sscce.build'.
Nuitka:INFO: Successfully created 'sscce.dist\sscce.exe'.
C:\Users\geoff\Code\rom\OASISAIROM [main ≡ +2 ~1 -0 | +4 ~3 -0 !]> .\sscce.dist\sscce.exe
Traceback (most recent call last):
  File "C:\Users\geoff\Code\rom\OASISA~1\SSCCE~1.DIS\sscce.py", line 3, in <module>
ModuleNotFoundError: No module named 'scipy'
C:\Users\geoff\Code\rom\OASISAIROM [main ≡ +2 ~1 -0 | +4 ~3 -0 !]> nuitka --version
1.7rc2
Commercial: None
Python: 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)]
Flavor: CPython Official
Executable: C:\Python311\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10
Version C compiler: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\bin\Hostx64\x64\cl.exe (cl 14.3).
C:\Users\geoff\Code\rom\OASISAIROM [main ≡ +2 ~1 -0 | +4 ~3 -0 !]> pip list
Package           Version
----------------- --------
distlib           0.3.6
filelock          3.12.0
mpmath            1.3.0
networkx          3.1
Nuitka            1.7rc2
numpy             1.24.3
opencv-python     <IP_ADDRESS>
ordered-set       4.1.0
pip               23.1.2
platformdirs      3.5.1
setuptools        65.5.0
sympy             1.12
typing_extensions 4.6.3
virtualenv        20.23.0
zstandard         0.21.0
C:\Users\geoff\Code\rom\OASISAIROM [main ≡ +2 ~1 -0 | +4 ~3 -0 !]> python -m pip freeze
distlib==0.3.6
filelock==3.12.0
mpmath==1.3.0
networkx==3.1
Nuitka @ https://github.com/Nuitka/Nuitka/archive/develop.zip#sha256=7dac8e149e96e024a7cb19c2f9f5317c15066563290f8af216b6a713a89d841e
numpy==1.24.3
opencv-python==<IP_ADDRESS>
ordered-set==4.1.0
platformdirs==3.5.1
sympy==1.12
typing_extensions==4.6.3
virtualenv==20.23.0
zstandard==0.21.0

Nuitka needs to be run with a Python that runs your program. Using nuitka to run it is not recommened, and works only if you know what you are doing, which maybe you are not. In the User Manual it asks you to use python -m nuitka where Python is the version to use.

Forgive me, I'm used to compilers like rustc or javac or msbuild that emit errors like error: symbol 'scipy' not defined when they encounter... well references to things that don't exist. But this is my mistake!
To reframe what keyhayen said:

scipy is not on the python path (aka include-path, libbpath, etc --nuitka simply hasnt been told about where scipy is)
I got in to this confusion because I now have two different environments. I thought I was using a local venv/scripts/python.exe but actually I have been using my global copy of python installed to C:\python311\bin. oops!

to fix this problem: install (or use if already installed) the scipy copy in ./venv
virtualenv venv
./venv/scripts/pip install scipy
./venv/scripts/nuitka yoursource.py --standalone

Well, in Python imports of modules that are not installed are very much the norm. The compilation report can be used to see them, usually a lot of them.


When I create a workitem how do I select the template I want to start with?

This seems to only cover things to use after the workitem has been created.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1debd814-eb48-c984-a4f9-7606093e9617
Version Independent ID: f48ef796-c2e3-8fa3-6c41-326a5b252a3d
Content: Add and use work item templates to update user stories, tasks, issues, and other work items - Azure Boards
Content Source: docs/boards/backlogs/work-item-template.md
Product: devops
Technology: devops-agile
GitHub Login: @KathrynEE
Microsoft Alias: kaelli

Thanks for your question. It looks like you're working through an issue with your scenario or implementation, rather than an issue with the documentation. Here are a couple of options where you might consider asking your question:

Azure DevOps on Stack Overflow
Azure DevOps Support Bot

I am closing this issue now.  You are welcome to @ mention me for any followup.
We hope to hear from you again.

@WilliamAntonRohm - please explain what part of my post gives you the impression it's got anything to do with implementation.
If you don't know the answer to a question, or don't understand the question, why aren't you asking for additional information or clarification instead of slamming the door in someone's face, so to speak?
If you think I'm being rude, salty, or even hostile, strongly consider that I'm only reflecting what has been directed at me.

@StingyJack -- Andrew, thank you for clarifying.
@KathrynEE -- Kathryn, please look into adding a template selection step.

@WilliamAntonRohm - this is at least the third consecutive interaction we have had like this. I am really not looking forward to there being another. Please let's try to end that streak.

@StingyJack - Thank you for your inquiry. You apply a template to a new work item in the same way you do for an existing work item. The following image shows a doc bug template being applied to a Bug work item.  I will update the topic to reflect that the steps are the same for existing and new work items. Let me know if more needs clarification.

@KathrynEE   - Nothing else needed, just that little bit of info will do.
Can the support request tag be removed?


e2e: Run e2e against shared server only a single time.

Multiple times is broken by #796 and even a broken non-gating CI job
has a negative impact. A fix for #796 should update the count back to
2.

Rebased