Failing to add mdns remote component

Hi 👋 I'm trying to use esp_idf_svc::mdns module but I'm having trouble actually getting the build process to pull the remote component. I'm using an esp32s3 and my project seems to be building fine when I don't import this module in src/main.rs.
I'm familiar with this build option:
https://github.com/esp-rs/esp-idf-sys/blob/master/BUILD-OPTIONS.md#remote-components-idf-component-registry
My Cargo.toml file looks like this:
[package]
name = "esp32s2-server2"
version = "0.1.0"
authors = ["Pat O'Connor<EMAIL_ADDRESS>edition = "2021"
resolver = "2"
rust-version = "1.71"

[package.metadata.esp-idf-sys.extra_components]
remote_component = { name = "espressif/mdns", version = "1.2" }

[profile.release]
opt-level = "s"

[profile.dev]
debug = true    # Symbols are nice and they don't increase the size on Flash
opt-level = "z"

[features]
default = ["std", "hal", "esp-idf-sys/native"]

pio = ["esp-idf-sys/pio"]
all = ["std", "nightly", "experimental"]
hal = ["esp-idf-hal", "embedded-svc", "esp-idf-svc"]
std = ["alloc", "esp-idf-sys/std", "esp-idf-sys/binstart", "embedded-svc?/std", "esp-idf-hal?/std", "esp-idf-svc?/std"]
alloc = ["embedded-svc?/alloc", "esp-idf-hal?/alloc", "esp-idf-svc?/alloc"]
nightly = ["embedded-svc?/nightly", "esp-idf-svc?/nightly"] # Future: "esp-idf-hal?/nightly"
experimental = ["embedded-svc?/experimental", "esp-idf-svc?/experimental"]
#embassy = ["esp-idf-hal?/embassy-sync", "esp-idf-hal?/critical-section", "esp-idf-svc?/embassy-time-driver", "esp-idf-svc?/embassy-time-isr-queue"]

[dependencies]
log = { version = "0.4.17", default-features = false }
esp-idf-sys = { version = "0.33.7", default-features = false }
esp-idf-hal = { version = "0.42.5", optional = true, default-features = false }
esp-idf-svc = { version = "0.47.3", optional = true, default-features = false }
embedded-svc = { version = "0.26.4", optional = true, default-features = false}
anyhow = "1.0.72"
futures = "0.3.28"
heapless = { version = "0.8", default-features = false }

[build-dependencies]
embuild = "0.31.2"

I don't know if this is necessary but my build.rs file looks like this:
fn main() -> Result<(), Box<dyn std::error::Error>> {
    embuild::espidf::sysenv::output();
    Ok(())
}

Is there something else I need to do to get the build to download mdns? Thanks for all the hard work on this project!

your syntax is a bit wrong in the Cargo.toml. You need double [] around the definition like the following ->
[[package.metadata.esp-idf-sys.extra_components]]
remote_component = { name = "espressif/mdns", version = "1.2" }

this is similar to how a bin and a lib definitions are written. So essentially its a array-of-tables of TOML configs so its technically possible to write more than one of this definitions. For more info look here

Thanks for the response. I did try that as well. It seems to have the same result. Here's what the output of cargo run --release looks like:
cargo run --release
   Compiling esp32s2-server2 v0.1.0 (/Users/patrickoconnor/workspace/esp32s2-server2)
error[E0432]: unresolved import `esp_idf_svc::mdns`
  --> src/main.rs:3:5
   |
3  | use esp_idf_svc::mdns;
   |     ^^^^^^^^^^^^^^^^^ no `mdns` in the root
   |
note: found an item that was configured out
  --> /Users/patrickoconnor/.cargo/registry/src/github.com-1ecc6299db9ec823/esp-idf-svc-0.47.3/src/lib.rs:70:9
   |
70 | pub mod mdns;
   |         ^^^^

you may need to run a cargo-clean to make sure the bindgens are build correctly. You can check that the flag is correctly set and included with  cargo rustc -- --print cfg | grep esp_idf_comp_espressif__mdns_enabled

Short story: it seems to be fixed now. Thanks!
Long story: I ended up running cargo clean, rm -r .embuild, and git submodule deinit --all. The error I seemed to be getting was a problem cloning the esp-idf repo from espressif. The git error included invalid index-pack output. After searching around that seemed to be caused by a network issue. So I restarted my computer, ran the commands above, then ran cargo run --release and everything worked as expected. I'm now able to build while importing mdns.
Thanks for the help!


Bumping up the normal label's background colors

All has been ran by Sassy and these are the improvements we make bumping all default labels' colors a bit

I will wait until our next sync to обнародвам and then merge this :)


Test for OPTIONS *

Hopefully I will get a chance to try more of these scenarios. Right now I only know about kestrel:
Kestrel: Passes * through to PathString which throws
We need to know what the other servers behavior is and decide on the right implementation to handle *.

IIS running nothing (i.e. Default Site with default content) returns this:
HTTP/1.1 200 OK
Allow: OPTIONS, TRACE, GET, HEAD, POST
Server: Microsoft-IIS/10.0
Public: OPTIONS, TRACE, GET, HEAD, POST
X-Powered-By: ASP.NET
Date: Fri, 18 Mar 2016 21:09:33 GMT
Content-Length: 0

From the RFC:

If the Request-URI is an asterisk (""), the OPTIONS request is intended to apply to the server in general rather than to a specific resource. Since a server's communication options typically depend on the resource, the "" request is only useful as a "ping" or "no-op" type of method; it does nothing beyond allowing the client to test the capabilities of the server. For example, this can be used to test a proxy for HTTP/1.1 compliance (or lack thereof).

I'm not sure there's any reason to pass OPTIONS * to the app. We could just return 200 OK with no other info, or something basic like Allow: GET, HEAD, POST.

RFC link?
How about WebListener?

https://tools.ietf.org/html/rfc7231#section-4.3.7
Text is a little different because I had copied from RFC2616. But the essence is the same.
WebListener just passes it to the app, but I think the * special case shouldn't be.

If we handle it in the server the response needs to be configurable. If we don't then it can be handled by middleware / app code.

Discussed offline with @lodejard and we're recommending that servers surface Options * requests to the app but leave both Path and PathBase empty. If in the future there's a raw url field that would contain the *.

Filed https://github.com/aspnet/KestrelHttpServer/issues/709 and https://github.com/aspnet/WebListener/issues/183. I'll leave this issue open for adding a shared test.


Fix reserve over allocating underlying buffer

Fixes calls to reserve when the underlying shared buffer was already
big enough to fit the requested capacity. Previously a new even larger
buffer was created anyways. This could eventually lead to an OOM
condition.
Fixes #559

This appears to have been introduced by #555

Here is an example of how you could cause an OOM error
#[test]
fn reserve_in_arc_unique_does_not_overallocate_after_multiple_splits() {
    let mut bytes = BytesMut::from(LONG);
    let orig_capacity = bytes.capacity();
    for _ in 0..100 {
        drop(bytes.split_off(LONG.len() / 2));

        // now bytes is Arc and refcount == 1

        let new_capacity = bytes.capacity();
        bytes.reserve(orig_capacity - new_capacity);
    }
    assert_eq!(bytes.capacity(), orig_capacity);
}

Thanks for the quick merge! And I see you are already prepping a new release (1.2.1), thank you!!


Fix README.md markdown syntax for python module

Thanks!

No problem, I just created a gui for debugging CMSIS values on real MCU's. Wouldn't have been possible without cmsis-svd.


Remove storybook

Summary | Résumé
Removes StoryBook as it isn't being utilized by the team.
For discussion:
I think we can drop the documentation pages for:
Api
Schema
Form Viewer JSON Specification
I don't think these pages provide much value at the moment and are bound to fall out of date
I converted a11y + System Architecture to converted to plain markdown
Pull Request Checklist
Please complete the following items in the checklist before you request a review:

[x] Have you completely tested the functionality of change introduced in this PR? Is the PR solving the problem it's meant to solve within the scope of the related issue?
[x] The PR does not introduce any new issues such as failed tests, console warnings or new bugs.
[ ] If this PR adds a package have you ensured its licensed correctly and does not add additional security issues?
[ ] Is the code clean, readable and maintainable? Is it easy to understand and comprehend.
[ ] Does your code have adequate comprehensible comments? Do new functions have docstrings?
[ ] Have you modified the change log and updated any relevant documentation?
[ ] Is there adequate test coverage? Both unit tests and end-to-end tests where applicable?
[ ] If your PR is touching any UI is it accessible? Have you tested it with a screen reader? Have you tested it with automated testing tools such as axe?

For discussion:
I think we can drop the documentation pages for:
Api
https://cds-snc.github.io/platform-forms-client/refs/pull/1911/merge/?path=/story/admin-docs-api-documentation--page
+
Schema
https://cds-snc.github.io/platform-forms-client/refs/pull/1911/merge/?path=/story/forms-docs-form-builder-database-schema--page
I don't think this page provides much value in it's current form / current API users
a11y
https://cds-snc.github.io/platform-forms-client/refs/pull/1911/merge/?path=/story/forms-docs-accessibility-report-for-forms-alpha--page
perhaps this page should be more "built-in" to the product so we can collect feedback
System Architecture
https://cds-snc.github.io/platform-forms-client/refs/pull/1911/merge/?path=/story/forms-docs-architecture--page
I'll update this to just Markdown

Path does not exist /home/runner/work/platform-forms-client/platform-forms-client/cypress/screenshots

@bryan-robitaille any idea why this happens with Cypress sometimes?

@timarney in this case it looks like a package didn't get installed properly so cypress didn't even launch.  That's why it couldn't find the screenshots folder.  I know github was actions was having issues this afternoon.  I'll relaunch the action to see if it self heals or a sign of a bigger problem.

Looks like there's an issue with the cypress workflow where sub-modules aren't being installed as part of the cache.  Will open a separate PR to fix.


[BUG-UI/UX] no-warning of loosing information when leaving page after not saving edited settings as owner

Describe the bug
I just updated the annotation guidelines and pressed the leave dataset button but did not get a warning about loosing information.
To Reproduce
Steps to reproduce the behavior:

Go to a dataset
Click on settings
Edit the setting but don't press the save button
Leave the page

Expected behavior
I would expect to get a warning
Screenshots
N.A.

Environment (please complete the following information):

OS [e.g. iOS]: MaxOS
Browser [e.g. chrome, safari]: Firefox
Argilla Version [e.g. 1.0.0]: 1.20
ElasticSearch Version [e.g. 7.10.2]: N.A.
Docker Image (optional) [e.g. argilla:v1.0.0]: 1.20

Additional context
N.A.

Hi @davidberenstein1957 🙌
I added your feature request in this PR https://github.com/argilla-io/argilla/pull/4461
If you try to close the tab with unsaved changes you can see the before unload event with default browser message.
Also if you try to go to dataset with see your dataset button you can see the custom notification toaster with our default behavior, and if you click on X button on toaster you'll move automatically unsaved data tab; if you click on 'ignore and cotinue` button you'll go to dataset with your latest filters applied.
Thanks for this great feature!!


Autplay

Hi is there some example how to do autoplay just after page is rendered  in Blazor Server?

i did try
autoplay="autoplay"
loop="loop"
But is simply not start

You can try adding "muted" to the attributes, but even then it is not guaranteed to autoplay.
Browsers are in control of things like this and they can all handle it differently.

Quick and dirty solution.
I've added the following javascript to my index.html
    <script>
        function autoPlay(id)
        {
            document.getElementById(id).play();
        }
    </script>

In the BlazoredVideo.LoadedData event i call the javascript using:
        <BlazoredVideo
                id="myId"
                muted="muted"
                autoplay="autoplay" LoadedData="LoadedData">
            <source src="@path" type="video/mp4" />
        </BlazoredVideo>

    private void LoadedData(VideoState state)
    {
        JS.InvokeVoidAsync("autoPlay", "myId");
    }

You need to inject the IJSRuntime at the top of the page like:
@inject IJSRuntime JS

Note, you do need to have some interaction with the browser in order for it to play automatically. Or else you will see this error in Chrome's console: Uncaught (in promise) DOMException: play() failed because the user didn't interact with the document first. See: https://developer.chrome.com/blog/autoplay/

This seems to work:
index.html
    <script>
        function autoPlay(id)
        {
            document.getElementById(id).play();
        }
    </script>

index.razor
@inject IJSRuntime JS
<BlazoredVideo
            id="myId"
            muted="muted"
            autoplay="autoplay"
            LoadedData="LoadedData">
        <source src="@path" type="video/mp4" />
</BlazoredVideo>

    private void LoadedData(VideoState state)
    {
        JS.InvokeVoidAsync("autoPlay", "gource");
    }

Just make sure you interact with the page first (by clicking somewhere for example). Or else you get Uncaught (in promise) DOMException: play() failed because the user didn't interact with the document first.


sub-module isn't considered for import codefix

i have a sub-module called xxx in the zzz.ts file
// zzz.ts
module xxx {
}

when i open yyy.ts and type xxx in there i expect the codefix to offer importing the xxx module from the zzz.ts but it is not happening

is it exported? or do you mean module "xxx" { .. ?

ok,
i have z.ts file
in it i have xxx sub module:
module xxx {
//  ...
}
then in another file a.ts i write xxx and nothing happens
On Thu, Dec 22, 2016 at 6:57 PM, Mohamed Hegazy<EMAIL_ADDRESS>wrote:

is it exported? or do you mean module "xxx" { ..?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/13137#issuecomment-268916869,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA5PzccSYNJZDUfNsUDhTTPr31AK6CfNks5rKw54gaJpZM4LUeDp
.

--
Aleksey Bykov

so xxx is a namespace?

xxx is exported:
export module xxx {
// ...
}
On Thu, Dec 22, 2016 at 7:58 PM, Aleksey Bykov<EMAIL_ADDRESS>wrote:

ok,
i have z.ts file
in it i have xxx sub module:
module xxx {
//  ...
}
then in another file a.ts i write xxx and nothing happens
On Thu, Dec 22, 2016 at 6:57 PM, Mohamed Hegazy<EMAIL_ADDRESS>wrote:

is it exported? or do you mean module "xxx" { ..?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/13137#issuecomment-268916869,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA5PzccSYNJZDUfNsUDhTTPr31AK6CfNks5rKw54gaJpZM4LUeDp
.

--
Aleksey Bykov

--
Aleksey Bykov

can you share a sample?

In 40 mins

nailed it:
// file x.ts
export module xxx {
   export interface I{
   }
}

then
// file y.ts

                           xxx.I
put your carret here         ^

observe no codefix offering


npm file size pretty big

Just noticed that your published npm bundle is pretty large, almost 1MB (824KB).
$ du -h node_modules/xml-js

184K    xml-js/artwork
 12K    xml-js/bin
 32K    xml-js/lib
156K    xml-js/test/browse-jasmine/lib/jasmine-2.5.2
156K    xml-js/test/browse-jasmine/lib
452K    xml-js/test/browse-jasmine
520K    xml-js/test
824K    xml-js

$ ls -lh

-rw-r--r--   1 mac  staff   1.1K May  2  2016 LICENSE
-rw-r--r--   1 mac  staff    17K Nov  4 00:15 README.md
-rw-r--r--   1 mac  staff   456B Jun  7  2016 appveyor.yml
drwxr-xr-x   5 mac  staff   170B Jan 24 10:07 artwork
drwxr-xr-x   4 mac  staff   136B Jan 24 10:07 bin
-rw-r--r--   1 mac  staff    34B Sep  7 03:42 index.js
drwxr-xr-x   8 mac  staff   272B Jan 24 10:07 lib
-rw-r--r--   1 mac  staff   5.2K Jan 24 10:07 package.json
drwxr-xr-x  13 mac  staff   442B Jan 24 10:07 test

Your core javascript files (index.js + lib/) is only 32KB.
Inside your test cases you've got 520KB that you ship with your npm bundle also artwork is 180KB, might be worth adding an .npmignore file to your repo and exclude some of the files that are not needed.
**Proposed .npmignore
artwork/
test/

Or you can include files in your package.json.
Example
  "files": [
    "index.js",
    "index.d.ts"
  ],

@DenisCarriere  Thanks for this great tip. Your pull request has been merged.

Awesome stuff! Let me know when you publish a patch release, I'll update my dependencies

v1.01 is published to address this issue.
Thanks again for your suggestion. I wouldn't have realized it by myself.

@nashwaan Thanks,
However, now you've got new files 😢 _playground which is 364K.
$ du node_modules/xml-js -h
4.0K	./_playground/node-xml2json-master/bin
 20K	./_playground/node-xml2json-master/lib
104K	./_playground/node-xml2json-master/test/fixtures
132K	./_playground/node-xml2json-master/test
184K	./_playground/node-xml2json-master
 44K	./_playground/sax-js-master/lib
 76K	./_playground/sax-js-master
364K	./_playground
 12K	./bin
 32K	./lib
 16K	./types
640K	.

It's kinda tricky when you publish with npm, it literally takes your entire folder and publishes that. I forgot when you add .npmignore it stops following the .gitignore rules.

Pull request has been merged. Published in v1.0.2.
Hope it is ok now.

@nashwaan Yay! that works, it's a lot smaller now 👍
Now v1.0.2
 44K	./sax/lib
 76K	./sax
 12K	./xml-js/bin
 32K	./xml-js/lib
 16K	./xml-js/types
120K	./xml-js
200K	.

Before v0.9.6
4.0K	./.bin
 44K	./sax/lib
 76K	./sax
184K	./xml-js/artwork
 12K	./xml-js/bin
 32K	./xml-js/lib
156K	./xml-js/test/browse-jasmine/lib/jasmine-2.5.2
156K	./xml-js/test/browse-jasmine/lib
452K	./xml-js/test/browse-jasmine
516K	./xml-js/test
820K	./xml-js
900K	.

I'd say that's a 450% filesize reduction.


Updates DaemonSet Name

Previously, the DaemonSet name was derived from contour.Name + "envoy" . This PR does the following:

Aligns the DaemonSet name with upstream until multiple Contour instances per namespace are supported.
Moves the ReadinessProbe from the shutdown-manager container to the envoy container (where it belongs).
Updates container port definitions and other port references to use port variables.
Sets readOnly: false for Envoy's config volume (as it should be).
Updates unit tests as needed.

Requires https://github.com/projectcontour/contour-operator/pull/65 for port variables.
/assign @jpeach @Miciah
/cc @stevesloka @skriss
Signed-off-by: Daneyon Hansen<EMAIL_ADDRESS>

@jpeach regarding https://github.com/projectcontour/contour-operator/pull/64#pullrequestreview-514312562, this PR depends on #65 for port variables.


update the layer list

In the layer list things are missing :

description of the layer : it should include both the description and the technical input to reproduce it locally
names : some name are way too long to be displayed in the proposed dashboard, do you think we could shorten them
layers : lots of them are out of date if not fake
the other subtheme is not referenced in the wireframe so I assume it has been abandoned

In short if you have an updated google sheet I would be happy to update the csv file

Spoke with Jeff and Karis about shorting the layer names and they're happy to do it for us. Is there a character limit or ideal range I can give them/us to work with?

I think we're using the subthemes for assigning the weights via the priorities panel (at least for now) so I say we keep them in.

Spoke with Jeff and Karis about shorting the layer names and they're happy to do it for us. Is there a character limit or ideal range I can give them/us to work with?

between 30 and 40 character is a must. In fact I canot guarantee that all the letters will be displayed in all conditions after 30. but for big screens you can go to 40


[Bug Report] docker镜像触发构建时机不正确

Checks

[X] I have searched the existing issues
[X] I have read the documentation
[X] Is it your first time sumbitting an issue

Current Behavior
注意到 https://github.com/daeuniverse/daed/actions/workflows/publish-docker-image.yml 这里没有被 0.8.0 和 0.7.4 的release触发构建
参考 https://docs.github.com/zh/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#release
Expected Behavior
每次发布release都应当触发docker镜像构建和推送
https://github.com/daeuniverse/daed/blob/main/.github/workflows/publish-docker-image.yml#L9 这里应当是 [published] 而不是 [prereleased,released]
Steps to Reproduce
检查 https://github.com/daeuniverse/daed/actions/workflows/publish-docker-image.yml ，和release发布对照
Environment
N/A
Anything else?
No response

感谢报告，可以帮忙修改吗，谢谢！

@wanlce 已经在pr中提交修改


Update testing guidelines

Which issue(s) are closed by this pull request?
Closes #403
Changes proposed in this pull request:
Update testing guidelines

maybe we should be clear if we use parameter or arguments. If I get it right in python they are called arguments


What are the differences in Blazor for .net 6 from .net 5 requiring a new project?

You mention here: https://docs.microsoft.com/en-us/aspnet/core/migration/50-to-60?view=aspnetcore-6.0&tabs=visual-studio#blazor

To adopt all of the new 6.0 features for Blazor apps, we recommend the following process:
Create a new 6.0 Blazor project from one of the Blazor project templates. For more information, see Tooling for ASP.NET Core Blazor.
Move the app's components and code to the 6.0 app making modifications to adopt the new 6.0 features.

But you never mention what is lacking if we don't update the project. So if we change the dotnet version on our blazor serverside project from .net5.0 to .net6.0, are we still lacking? And if so, what and why?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5bec61f9-7298-170d-5e12-abb27cde584e
Version Independent ID: 27a43e3f-149d-9058-f937-0eda1d6dd4f2
Content: Migrate from ASP.NET Core 5.0 to 6.0
Content Source: aspnetcore/migration/50-to-60.md
Product: aspnet-core
Technology: aspnetcore-migration
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Yes, the app doesn't receive most new features without developer intervention automatically when you merely update the target framework. The new features are listed in the What's New for 6.0 doc and will be listed in the 6.0 GA release blog post when it comes out.

https://docs.microsoft.com/aspnet/core/release-notes/aspnetcore-6.0#blazor
https://devblogs.microsoft.com/dotnet/

We used to describe how to migrate an app from one release to the next for Blazor step-by-step, but it's just too much churn/work to cover this time around. Devs can adopt each feature using the docs and by looking at an app created from a new 6.0 app or by creating a 6.0 app and migrating their components, code, etc. to the new app.
What I'll do here to help surface the new features is cross-link the What's New topic to this. It wasn't possible before because the What's New topic wasn't available until recently, but I can make the cross-link now.
Thanks for commenting on the guidance. Leave this issue open, as it will close automatically when the PR merges.


Pure c10d example

Description
Small example that only uses c10d and no rpc.
Test
$ torchrun --nproc_per_node 4 local_test_c10d.py
# cpu:
equivalence test passed 95468232.0 ref 95468232.0
# cuda:
equivalence test passed 95261472.0 ref 95261472.0

Can you educate me on what does pure c10d refer to? Do you mean SPMD style or?


Storm Flux

This switches from the old TopologyRunner model written in Java code, to use Storm Flux to configure the topologies entirely in YAML.
This makes the topologies more flexible and the config files are more apparent. It also reduces the number of config files we need in order to run a topology.
I've verified that all these topologies run in local mode.
The command to run a topology has changed:
storm jar metron-streaming/Metron-Topologies/target/Metron-Topologies-0.6BETA.jar org.apache.storm.flux.Flux --local metron-streaming/Metron-Topologies/src/main/resources/Metron_Configs/topologies/bro/local.yaml --filter metron-streaming/Metron-Topologies/src/main/resources/Metron_Configs/etc/env/config.properties

Note that the "main class" is always the same now, and you specify which topology to run entirely through the YAML.

+1
Nice work migrating everything to flux. That will hopefully make topology configuration more manageable.

I expect this will have a substantial effect on the Ansible scripts, which now have to construct entirely different configuration files.

thanks for checking in. we are in the process of converting our scripts to encompass your changes. will be checking in shortly


WIP: DOC: Add note on features

Codecov Report

Merging #541 (b8e3772) into main (0460355) will decrease coverage by 0.13%.
The diff coverage is 84.00%.

@@            Coverage Diff             @@
##             main     #541      +/-   ##
==========================================
- Coverage   98.49%   98.35%   -0.14%
==========================================
  Files          51       51
  Lines        2063     2072       +9
==========================================
+ Hits         2032     2038       +6
- Misses         31       34       +3

Impacted Files
Coverage Δ

pfhedge/features/_getter.py
95.55% <0.00%> (ø)

pfhedge/nn/modules/hedger.py
100.00% <ø> (ø)

pfhedge/features/features.py
97.67% <80.00%> (-2.33%)
:arrow_down:

pfhedge/features/__init__.py
100.00% <100.00%> (ø)

pfhedge/features/_base.py
100.00% <100.00%> (ø)

pfhedge/features/container.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0460355...b8e3772. Read the comment docs.


Fix issue #2951

Do not depth test against terrain in 2D. Only draw the depth plane in 3D.

Looks good.


Add check for invalid EntityUid when dropping

About the PR
Fixes error when Drop is passed an invalid EntityUid.
Fixes #3894

Due to the new viewport, the item will now drop at the player's feet instead of where they're pointing in the world behind the UI like it did previously. Adding the previous functionality requires changes to UI input handling.

Due to the new viewport, the item will now drop at the player's feet instead of where they're pointing in the world behind the UI like it did previously. Adding the previous functionality requires changes to UI input handling.

Never really liked that anyway tbh

Due to the new viewport, the item will now drop at the player's feet instead of where they're pointing in the world behind the UI like it did previously. Adding the previous functionality requires changes to UI input handling.

Never really liked that anyway tbh

Just to clarify the dropping at feet is only when hovering over a UI with the cursor. When the mouse is over the viewport it'll drop at the cursor still.

Ah, lame.
Current behaviour makes sense then, d/w about it. (even if I dislike it)

I literally told you how to solve it on discord though? If there's no viewport hovered over, use the main viewport instead.

I literally told you how to solve it on discord though? If there's no viewport hovered over, use the main viewport instead.

I'll do that separately. This is server side for invalid client input.


FAQ on how stats are calculated.

Thanks for contributing to the Harness Developer Hub! Our code owners will review your submission.
Description

Please describe your changes: __________________________________
Jira/GitHub Issue numbers (if any): ______________________________
Preview links/images (Internal contributors only): __________________

PR lifecycle
We aim to merge PRs within one week or less, but delays happen sometimes.
If your PR is open longer than two weeks without any human activity, please tag a code owner in a comment.
PRs must meet these requirements to be merged:

[ ] Successful preview build.
[ ] Code owner review.
[ ] No merge conflicts.
[ ] Release notes/new features docs: Feature/version released to at least one prod environment.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://662aedbd8b83d4355625295a--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://662c1e7b52d90f34de9077d3--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://662c20e325ed7e36078aaa4f--harness-developer.netlify.app


Broken link

there is a broken link at
"For a complete list and descriptions of items in a manifest, see the manifest specification"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d657479f-c0b7-6358-c7d0-0402215edaa5
Version Independent ID: 8976699b-cdba-5206-bdb0-8a0875acfe45
Content: Create your package manifest
Content Source: hub/package-manager/package/manifest.md
Product: dev-environment
Technology: package-manager
GitHub Login: @KevinLaMS
Microsoft Alias: kevinla

Thanks for bringing this to our attention @birbilis. I'm investigating and have reached out to the PMs. I'll add an update once I've fixed this issue. Thanks again.

@mattwojo looks like this is correct link is https://github.com/microsoft/winget-pkgs/tree/master/doc/manifest/schema
Problem is manifest schema is located in the microsoft/winget-pkgs but doc assumes it's located at  microsoft/winget-cli
Source: https://github.com/microsoft/winget-cli/blob/master/doc/windows/package-manager/package/manifest.md#manifest-contents
Do you want me to open a pr to fix the link?

Thanks @kdpuvvadi!  The PM responded with these two links as well:
https://github.com/microsoft/winget-pkgs/blob/master/AUTHORING_MANIFESTS.md
https://github.com/microsoft/winget-pkgs/tree/master/doc/manifest/schema/1.2.0
This all looks like information worth including in the docs, so I'm setting aside some time to do a bit of a read-through and update to ensure that all of the relevant info and links are included.

Took me a bit, but got this update in: https://github.com/MicrosoftDocs/windows-dev-docs/commit/dfeb79d6a19efc0c30783e5ed79d93a2de54d76f
Thank you @birbilis for bringing this issue to my attention! And thanks @kdpuvvadi for contributing as well.


Import from Zim

It would be great to have a way to import notes from Zim. Its syntax is described here.
There are currently several ways to convert Zim notes to Obsidian, but all still need some tweaking:

This script is an attempt to directly convert Zim notes to Obsidian.

This script is post-processing Zim notes that were exported to Markdown (through Pandoc) with the built-in exporter of Zim. Some issues need attention there, f.i. indentations, checkboxes and code blocks aren't converted properly, as are Markdown and WikiLinks links.

Zim's built-in exporter also allows exporting to HTML, but it's currently unclear how the HTML should look like in order to work with the HTML import feature.

Please upvote this issue, if you would like to import your notes from Zim.

For what it's worth (little) here's the script I used to take zim-exported markdown and make it more suitable for use in Obsidian by removing the (redundant) title heading and converting setext to atx headers. I hadn't found the first and second scripts myself; they are probably better.
I still had to manually tweak things once I was in Obsidian, but the script got me started. Plus, zim's markdown export is not great, so I found some manual tweaks were going to necessary in any case.

For what it's worth (little) here's the script I used to take zim-exported markdown and make it more suitable for use in Obsidian by removing the (redundant) title heading and converting setext to atx headers. I hadn't found the first and second scripts myself; they are probably better.

This is exactly what the second script from above is doing and a little more.

I still had to manually tweak things once I was in Obsidian, but the script got me started. Plus, zim's markdown export is not great, so I found some manual tweaks were going to necessary in any case.

How did you solve the issue with indentations after converting?

I did a lot of regex across my vault in Sublime Text.

Very good suggestions. My choice (I still need some testing) could be export from Zim and apply option 2.
What is missing, however, (if I am not wrong) is the possibility to add YAML front-matter in the Obsidian file, keeping the creation timestamp, which is present in Zim notes and lost in the conversion.

@ricir

My choice (I still need some testing) could be export from Zim and apply option 2.

While the second option has its elegance, the first one seems to be the better one regarding the potential end result, because the second option has some in-built limitations, like the proper conversion of indented text or new lines before and after lines starting with checkboxes.

What is missing, however, (if I am not wrong) is the possibility to add YAML front-matter in the Obsidian file, keeping the creation timestamp, which is present in Zim notes and lost in the conversion.

This would be of course helpful as well, but it would already help a lot to have a reliable conversion of only the content, without the metadata.
Roughly two weeks ago the developer of the first script expressed his interest in improving it, but since then no actions were taken.

I also found markdown2zim (https://github.com/Xunius/markdown2zim) which despite the name also converts zim to markdown, but I see no specific "Obsidian dialect" support so I don't think it'll work well on internal links and we'll have to see for attached pictures (but should be good for the simple notes).
I have yet to try any of those tools, but I will start a huge conversion of my Zim notes to Obsidian soon, starting with a sample page containing all kinds of elements.
I will post the results and compare them to pick the best option - or wait if none is good enough and open corresponding issues on each repository (but all notes without pictures/internal links should probably be fine).

A reminder that this repo accepts PRs, see contribution guidelines.


Bug: [Lighthouse][PWA] Remove property display: -ms-flexbox; from components

Ionic version:  (check one with "x")
[] 1.x
[x] 2.x
I'm submitting a ...  (check one with "x")
[x] bug report
[ ] feature request
[ ] support request => Please do not submit support requests here, use one of these channels: https://forum.ionicframework.com/ or http://ionicworldwide.herokuapp.com/
Current behavior:
When deploying my app as PWA and I run Google Chrome's Lighthouse plugin to audit my app it throws an error Avoids old CSS flexbox Ionic components contain the display: -ms-flexbox property and that property seems to be deprecated.

The 2009 spec of Flexbox is deprecated and is 2.3x slower than the latest spec.
Expected behavior:
I expect the Lighthouse plugin to pass the test.
Steps to reproduce:
Setup a new Ionic blank v2 template and run lighthouse
Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
Cordova CLI: 6.5.0
Ionic Framework Version: 2.2.0
Ionic CLI Version: 2.1.18
Ionic App Lib Version: 2.1.9
Ionic App Scripts Version: 1.1.4
ios-deploy version: 1.9.0
ios-sim version: 5.0.11
OS: macOS Sierra
Node Version: v6.9.1
Xcode version: Xcode 8.2.1 Build version 8C1002

We use autoprefixer to generate it:
.page-wrap {
  display: -webkit-box;  /* OLD - iOS 6-, Safari 3.1-6, BB7 */
  display: -ms-flexbox;  /* TWEENER - IE 10 */
  display: -webkit-flex; /* NEW - Safari 6.1+. iOS 7.1+, BB10 */
  display: flex;         /* NEW, Spec - Firefox, Chrome, Opera */
}

http://shouldiprefix.com/#flexbox
Eventually it would not be required, but you know... this is not a perfect world with everybody using Chrome 57


Metadata + more

Hello, first of all thank you so much for your library which is so simple and nice !
I'd like to know if it is possible to retrieve the metadata (ie. title and artist) from the bluetooth ?
Also, at least on my iPhone I usually get a "headphones" symbol in the status bar when I connect a bluetooth speaker. Here I got nothing, would it be possible to have it ? I guess I would have to tell the phone what type of bluetooth device is connected.
Finally, but maybe that's too much to ask, with some bluetooth devices, my iPhone is capable of showing the battery level. Since my project is battery powered and I already know the level, would it be possible to send that as well ?
Best regards,
Mael

Simple answer is yes. And it's already implemented in this library.
Here (https://github.com/pschatzmann/ESP32-A2DP/blob/master/src/BluetoothA2DPSink.cpp#L397) you're telling what you want to get.
And here (https://github.com/pschatzmann/ESP32-A2DP/blob/master/src/BluetoothA2DPSink.cpp#L434) you're getting that.
More information about that you can be read in ESP32 documentation (https://docs.espressif.com/projects/esp-idf/en/v3.3/api-reference/bluetooth/esp_avrc.html).

Thank you so much for your answer! I went trough the code and it seems that everything I asked is indeed possible. Now I have to find the time and courage to go deeper in the code to understand how it works, it'a maybe a bit over my skills!
Cheers.

I  think it would be cool to have some callbacks to make it easy to use.

I think it would be cool to have some callbacks to make it easy to use.

Yes, it should. But I would also like a callback for connection state changes, so I could notify the user about this, e.g. let a LED blink or so ..

Support for a metadata callback has been added with the latest commit


Released image tags should include version

Expected Behavior
If a user of Tekton wants to know what version they have installed, they should be easily able to get that information.
Actual Behavior
To find this now you need to:

Find the names of the images you are running, by looking at your tekton Deployments and/or finding the release.yaml you installed (e.g. release.yaml attached to https://github.com/tektoncd/pipeline/releases/tag/v0.6.0)
Navigate in your browser (or use some kinda other tool I don't know about) to an image with its digest, e.g.  https://gcr/images/tekton-releases/GLOBAL/github.com/tektoncd/pipeline/cmd/webhook@sha256:59be4e291f510bced7c138e92cbfee1ea58e0a48969fbaa3a055c1537d77bfd9/details?tab=info) and see the tag (this one is v0.5.2)

Additional Info

Can we somehow include both the digest + the tag in the release.yaml? e.g. something like:

gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/kubeconfigwriter:v0.6.0@sha256:43050320e994156a2683ab5d30551a52a4a59fe20b0ee50161af5d524b9aa5c6

I'm not sure this is actually a valid way to refer to an image tho :(
We might want to consider also including 'alpha'  / 'beta' / etc. in the tag #1231

It is valid for an image name to include both tag and digest. In that case the digest wins but the tag can be useful informationally as in this case.

As @ImJasonH, this is possible, we need to create an issue upstream google/ko to add support for having the full image reference (tag + digest) written in the resolved yaml :angel:

This has been fixed since 0.10 at least, closing.
/close


Fix: Keep original name of field

Currently, if I have the following .proto file:
// The request message containing the user's name.
message HelloRequest {
  string my_name = 1;
}

And the following stub:
{
    "service": "helloworld.Greeter",
    "method": "SayHello",
    "in": {
        "equals": {
            "my_name": "hi"
        }
    },
    "out": {
        "data": {
            "message": "lemon"
        }
    }
}

And I were to make the following request:
grpc -plaintext -d @ <IP_ADDRESS>:22222 helloworld.Greeter/SayHello <<! | jq
{
	"my_name": "hi"
}
!

The mock server would fail to match the stub to the request, because once data is parsed into its internal map m, it's converted to camelCase and instead searches for "myName": "hi". The GRPC services I work with have snake cased fields, so when mocking, the the in's must be camel, and the outs must be snake.
I propose doing this, having a jsonpb marshaller which marshels the keynames down to the original name defined in the .proto file, to allow the above scenario to pass.

Hi, will this be merged?

I've another PR to open for this branch too to add header support @theflyingcodr , since the container you guys use has both of these changes implemented.


Add pre-commit filters with ruff

Lots of python codebases are now using ruff to standardize and check their codebase, and pre-commit to run those checks before each git commit.

Sounds like a good idea. I've already been using ruff for linting and black for formatting, but I've got no objection to using ruff for formatting as well.
To make this happen, I think the following would be needed:

Update requirements-dev.txt

Remove black
Add pre-commit
Maybe set a minimum version for ruff that has the formatter?

Configure pre-commit hooks for linting and formatting
Update pyproject.toml to configure the ruff formatter the same way black is configured at the moment
Update the development section of the README

You're welcome to open another PR, otherwise I'll take a look at doing this in a few days.


Using vs-repeat with a sliding-window slice of an extremely large data-set

I am trying to use vs-repeat with an extremely large data-set, potentially running into millions of records. For this to work, I will have to retrieve the data from the server incrementally as and when it becomes necessary, and also discard unused data on the client-side, to conserve browser memory.
To implement this I am using client-side "vs-scrolled-to-beginning" and "vs-scrolled-to-end" callbacks to initiate server calls to retrieve more data, and also ensure that only records within visible bounds (+20 records on each side) are retained in the browser memory.
Records that are beyond those limits will be discarded. For example, when the user scrolls to startIndex = 40 and endIndex = 50, the records from 0 - 20 are discarded. These will be fetched again from the server if the user starts scrolling up again. The basic idea is to keep the browser memory usage low, given that the total data-set can run into a huge size.
I looked at the approach used on https://github.com/kamilkp/angular-vs-repeat/issues/10 and tried to enhance that to fit my purpose. It works OK.
A working example of the issue is at https://jsfiddle.net/kjholla/8tytptbb/
My issue is that whenever records are removed from the start of the array, there is a very noticeable jump in the scroll position. I tried multiple approaches to smooth this out but this is something I have been unable to overcome. Any suggestions on how this could be made more smooth in terms of the scrolling experience?

Turns out that the issue causing the "jump" was my code and not anything to do with the library. I have updated the example at https://jsfiddle.net/kjholla/8tytptbb/ and it now works fine.
However, the solution still involves modifying the startindex and endindex that is calculated by the vs-repeat directive. It would be great to know if there is a better way to do this.


Prepares 3.10.2 release

Description of changes:
Bump AWS X-Ray Node SDK components to version 3.10.2 and update CHANGELOG.md
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Curious if these changes are being done manually or is there a script?

Curious if these changes are being done manually or is there a script?

Actually NVM - I know that these are done manually. I got confused with something else. 😆

Everything package.json and package-lock.json is updated via a script, it's the CHANGELOG that is updated manually.


kong reload , a few hours later, cpu 100% ( luajit table rehash/resize by stapxx profiling),help...

Is there an existing issue for this?

[X] I have searched the existing issues

Kong version ($ kong version)
all
Current Behavior
kong reload , a few hours later, cpu 100% ( luajit table rehash/resize by stapxx profiling)，this problem no longer occurs after，get everything back to normal
We have multiple custom plugins in kong and use kong.cache(mlcache/lru/shm) for data cache.
Single-core CPU monitoring chart:
single-core cpu 100% lasts for 3 minutes

Flame Graph kong-c-on-cpu-stack:
profiling command:
./samples/sample-bt.sxx --arg time=10  --skip-badvars  -D MAXACTION=2000000 -D MAXSKIPPED=10000 -x 13349 >kong-c-on-cpu-stack-29414-20231027131138.bt

kong-c-on-cpu-stack-29414-20231027131138-gt-100.00.bt.txt
@Tieske  help,Thank you very much.

The reason may be as follows, but I don't know how to solve it?
LuaJIT is a high-performance Lua interpreter that includes a data structure called "table" for storing key-value pairs. When the number of elements in a table increases, it may trigger a rehash operation, which involves resizing the table. This process can lead to high CPU usage.
In LuaJIT, the rehash operation is triggered when the number of elements in the table exceeds twice the current size of the table. During the rehash operation, the elements in the table are rehashed into a new memory space, and the size of the table is adjusted.
If the table rehash resize operation in LuaJIT continues for one minute, it could be due to several reasons:
1.Large number of elements: If the table contains a large number of elements, the rehash operation needs to process a significant amount of data, resulting in a longer operation time.
2.Memory allocation: When performing a rehash operation, LuaJIT needs to allocate new memory space to store more key-value pairs. Memory allocation itself requires computational resources, especially when dealing with large tables, the overhead of memory allocation can be significant.
3.Data migration: During the rehash process, LuaJIT needs to migrate the old key-value pairs to the new memory space. This involves a substantial amount of data migration operations, including recalculating hash values and determining new index positions, which can consume a considerable amount of CPU resources.
4.Collision resolution: During the rehash operation, there may be key collisions, where different keys map to the same index position. To resolve these collisions, LuaJIT needs to perform additional operations such as maintaining and updating linked lists or tree structures, which can increase CPU workload.
5.Concurrent competition: If multiple threads are simultaneously performing rehash operations on the same table, it may lead to concurrent competition issues. This can result in additional overhead due to lock contention, thread switching, and other factors, leading to increased CPU load.

but why does single-core CPU 100% last for 3 minutes?
Modify resty.lrucache in mlcache to resty.lrucache.pureffi?,
-- alternatively: local lrucache = require "resty.lrucache.pureffi"
local lrucache = require "resty.lrucache"

hi @fairyqb

I believe there is an issue with your stapxx flame graph; all the code paths seem to be merged into a single stack, making it impossible to analyze Lua CPU usage. examination. Hi @ADD-SP, could you please take a look at why the flame graph appears to be broken?
Additionally, you can use perf top or pstack <100%-cpu-nginx-pid> to obtain a C-level stack trace at that moment, which would be helpful for further
We also need some basic information about how many services or plugins are included in your Kong Gateway.

kong-luajit-rehash.bt.txt
kong-c-on-cpu-stack-29414-20231027131138-gt-100.00.bt.txt

cd /root/FlameGraph
export PATH=/root/FlameGraph:$PATH
./stackcollapse-stap.pl kong-luajit-rehash.bt  > kong-luajit-rehash.cbt
./flamegraph.pl kong-luajit-rehash.cbt > kong-luajit-rehash.svg

perf top  -p  [pid]

Update luajit version to the latest v2.1-20231006, currently using v2.1-20180420  ???
String hashing
This optimization only applies to Intel CPUs supporting the SSE 4.2 instruction sets. For such CPUs, and when this branch is compiled with -msse4.2, the string hashing function used for strings interning will be based on an optimized crc32 implementation (see lj_str_new()).
This optimization still provides constant-time hashing complexity (O(n)), but makes hash collision attacks harder for strings up to 127 bytes of size.
https://github.com/openresty/luajit2/tags

Flame graph fixed

hi @fairyqb

I believe there is an issue with your stapxx flame graph; all the code paths seem to be merged into a single stack, making it impossible to analyze Lua CPU usage. examination. Hi @ADD-SP, could you please simply take a look at why the flame graph appears to be broken?
Additionally, you can use perf top or pstack <100%-cpu-nginx-pid> to obtain a C-level stack trace at that moment, which would be helpful for further
We also need some basic information about how many services or plugins are included in your Kong Gateway.

Flame graph fixed

hi @fairyqb

I believe there is an issue with your stapxx flame graph; all the code paths seem to be merged into a single stack, making it impossible to analyze Lua CPU usage. examination. Hi @ADD-SP, could you please simply take a look at why the flame graph appears to be broken?
Additionally, you can use perf top or pstack <100%-cpu-nginx-pid> to obtain a C-level stack trace at that moment, which would be helpful for further
We also need some basic information about how many services or plugins are included in your Kong Gateway.

you can upload your .svg file here.

kong-luajit-rehash-c.svg.txt
kong-luajit-rehash-lua.svg.txt

Related to this question?  but,It doesn’t seem to be the same problem
https://github.com/Kong/kong/issues/6347
https://github.com/Kong/lua-resty-worker-events/pull/37/files

@Tieske  Please help with this issue, thank you very much.

Related to this question? but,It doesn’t seem to be the same problem #6347 https://github.com/Kong/lua-resty-worker-events/pull/37/files

@Tieske Please help with this issue, thank you very much.

Looks like your custom plugin yields too many worker events.

This is not the main reason. This is just writing cache data to mlcache.
Why Luajit table rehash/resize/hashkey?

Related to this question? but,It doesn’t seem to be the same problem #6347 https://github.com/Kong/lua-resty-worker-events/pull/37/files

@Tieske Please help with this issue, thank you very much.

Looks like your custom plugin yields too many worker events.

This is not the main reason. This is just writing cache data to mlcache.
Why Luajit table rehash/resize/hashkey?

Related to this question? but,It doesn’t seem to be the same problem #6347 https://github.com/Kong/lua-resty-worker-events/pull/37/files

@Tieske Please help with this issue, thank you very much.

Looks like your custom plugin yields too many worker events.

LuaJIT may rehash or resize a table if it becomes larger or smaller.
How many entries are you writing into the cache?

why single-core cpu 100% lasts for 3 minutes ？

why single-core cpu 100% lasts for 3 minutes ？

To determine the root cause, we require a minimal reproduction of this issue.
Based on the Lua-land flamegraph, it appears your custom plugin is storing an excessive number of entries in the mlcache. How many entries have you written into the cache?

After restarting Kong in the production environment, there is such a problem; in the test environment, we cannot reproduce it。
You can try to continuously set data to mlcache, or lua table.  different lengths, greater or less than 128
cache:set(key, opts?, value)
cache:get(key, opts?, callback?, ...)

After restarting Kong in the production environment, there is such a problem; in the test environment, we cannot reproduce it。
You can try to continuously set data to mlcache, or lua table. different lengths, greater or less than 128 cache:set(key, opts?, value) cache:get(key, opts?, callback?, ...)

Does this issue still occur after disabling the pass-external-authentication-plus-for-gray plugin?
I suspect that this plugin is storing too many cache entries.

pass-external-authentication-plus-for-gray
only:
`
local function callback(key)
local obj, err = http_client.execute(key)
return obj, nil, obj["TTL"]
end
function plugin:access(conf)
plugin.super.access(self)
local token = kong.request.get_header("Authorization")
local method = kong.request.get_method()
local path = kong.request.get_path()
local key = string.format("%s_%s_%s", token, path,  method)
local value ,err  = kong.cache:get(key, {}, callback, key)
kong.cache:set(token, { ttl = auth_value["TTL"] }, true)

end
`

kong.cache:get and kong.cache:set in the hot code path, so if key design failure will result in different keys for each request and the cache will lose its significance(It looks like this situation from flame graph.)

kong.cache:get and kong.cache:set in the hot code path, so if key design failure will result in different keys for each request and the cache will lose its significance(It looks like this situation from flame graph.)

We cannot predict the user's request content and behavior. Even so, why does table rehash/resize take so long? Which version of luajit needs to be replaced? Any suggestions? Thanks

We cannot predict the user's request content and behavior.

Yes, we cannot predict the end user, but the logic in your plugin triggered by ur user has exhausted the cpu usage.

From your source code, we already know that the pressue to the mlcache comes from the request.
Before you have cpu usage 100% problem, you need to do some tests on that scenario, like using wrk to check it. I think whatever you try to improve, there is always a limit of your plugin. So the perf top / flamegraph result is ok

Even so, why does table rehash/resize take so long?
I think this is not a problem because your QPS is already very high. Pushing the system to its limits will result in a hot function. Regardless of what this function is, your system will always have its limits.

So, I also wonder why you cannot trigger this problem(not problem, but bottleneck) in your test environment. After a quick look at the source code of your plugin, I think that if you stress test it with high pressure (just using wrk for benchmarking), you can easily encounter the bottleneck of Kong and your plugin. At that point, the QPS result from wrk becomes the limit.
If you want to improve this situation, you may need to enhance your plugin.

By the way, testing your plugin's scenario is similar to testing the performance of mlcache + pure proxy. This is because every request coming to your system will trigger one call to mlcache:get/set.
So, if you conduct performance tests on memory cache software like a wrapper for mlcache, Redis, or Memcache, you definitely find a hotpath function, similar to what we see here in your perf top or flamegraph results, such as rehash/resize. This isn't a problem to question why it takes so long; rather, it's a result that shows your system has encountered a bottleneck.

In the test environment wrk stress test immediately the cpu is 100%, while in the production environment the cpu is normal at first, and after a few hours the cpu rises to 100%, lasting 1 to 3 minutes.
Use wrk in the test environment to stress test the CPU stress to 50%. After waiting for a long time, the CPU will not rise to 100%.
I can't reproduce，you can try the stress testing process and observe the results.
Thanks.

The issue at hand isn't solely the speed of rehashing/resizing, but rather the necessity of frequent rehashing/resizing in a production environment. Could there be room for optimization? It's likely that bottlenecks exist within your plugin.
Reproducing this issue in a development environment should be straightforward by simulating high traffic volumes with tools like wrk.
After reproducing the issue, it's crucial to profile your system to identify and address these bottlenecks.
There are bottlenecks in your system, it's crucial not to concentrate solely on the flame graph.

Mlcache is based on lru and shm. There is a large table in lru. Shouldn't it be automatically eliminated or GC? Why is it 2 times resize and rehash take very long time?
Lru length is set to 512000
this is the problem,WYH?

Flame Graph

CPU resources are the most occupied ( do ... while)

The function is to look up the given string key in the hash table and return the corresponding value if it exists. If it does not exist, create a new key-value pair and return a pointer to its value.

Lru length is set to 512000. this is the problem,WYH?
You can add some trace log to do some metrics in your LRU/mlcache hot path to check how many entries has been added into LRU, kong developer may not answer this specific question.

I know that here we have two problems from this thread

which table was used to do rehash? need to locate the source of that table
why does this lua table try to do rehash? I think it is easy to answer if we can answer the first quetions.

Attach to the nginx process and execute the lbt tool for analysis. There is no permission in the production environment.
Results of profiling of lj-lua-stacks （stapxx#lj-lua-stacks）
https://github.com/openresty/stapxx#lj-lua-stacks

resty.events.lua.txt
@chobits

Attach to the nginx process and execute the lbt tool for analysis. There is no permission in the production environment.
Results of profiling of lj-lua-stacks （stapxx#lj-lua-stacks）
https://github.com/openresty/stapxx#lj-lua-stacks

resty.events.lua.txt
@chobits

Thats pretty cool. I'll take a look later.

hi @fairyqb If you have new updated info, you can post it in new comment. Because if you modified your original message, our previous discussion has become invalid and people who join the discussion later will find it strange.
From your new flamegraph, we found there is no problem with high usage of rehash, which was talked before. It shows that your plugins triggered too many search in lua-resty-worker-events:handler_list

The key to the problem is that it cannot be reproduced in the test environment, and multiple tools are used for stress testing.

The key to the problem is that it cannot be reproduced in the test environment, and multiple tools are used for stress testing.

WIthout enough information, we could not provide effective help. I'm tracing this issue because I have interests on the reason why there is large rehash operations, which might be beneficial for our future perforamnce improvement.
However, from my previous reply, I think the key point is that your plugin directly pass the QPS pressure to the mlcache. Assuming we have identified the rehash hotspot issue, you may still not be able to address the actual problem.

The information has been very comprehensive and provides possible solutions and pointers. I don't know what else can be done? What information do you want me to provide?

Rehash is a normal expansion behavior of the Luajit virtual machine, but what I can't understand is why it lasts for a long time and several minutes, and the CPU single core is very high. It is because the elements in the table are rehashed, the memory is reallocated, and the memory elements are moved to new Area???

The information has been very comprehensive and provides possible solutions and pointers. I don't know what else can be done? What information do you want me to provide?

There are two flamegraph, one is about rehash, another one is about hotkey.
The backtrace of  hotkey flamegraph is complete, and I have answered why it is hot.
But you focus on why rehash is hot, however, your rehash flamegraph has no complete backtrace from this https://github.com/Kong/kong/issues/11877#issuecomment-1794051703. We could not get to know which lua file or function trigger it.

Running the following statement in the plugin on the hot path, the rehash is very frequent and the CPU is very high
kong.cache:set(auth_cache_key1, { ttl = 0 }, version)
https://github.com/Kong/lua-resty-worker-events/blob/b90f067144fdc708dab6d29475979dab4d59a821/lib/resty/worker/events.lua#L227C16-L227C24

Modify the code variable name as follows and the problem is solved：
local function do_event(source, event, data, pid)
  log(DEBUG, "worker-events: handling event; source=",source,
    ", event=", event, ", pid=", pid) --,", data=",tostring(data))
    -- do not log potentially private data, hence skip 'data'

local list = _callbacks
do_handlerlist(list, source, event, data, pid)
local list_subs1 = list.subs[source]   ---*****Modify variable name*****
do_handlerlist(list_subs1 , source, event, data, pid)
local list_subs2  = list.subs[event]   ---*****Modify variable name*****
do_handlerlist(list_subs1 , source, event, data, pid)
end

@chobits  @Tieske
Check, please，extremely grateful

Because performance issue, we have deprecated lua-resty-worker library, see this pr for replacing this lua-resty-worker library https://github.com/Kong/kong/pull/8890.

The issue has been resolved and closed. Thank you for your attention!


Highlight network upgrade proposals

Story
As a governance dapp user
I want to know when a validator has proposed a network upgrade
So that I have visibility of what's happening with the network, can assess how it might effect me
Acceptance Criteria

[ ] I can understand that network upgrade proposals are made by validators (and not Vega)
[ ] Network upgrade proposals are visible and highlighted on the landing page
[ ] If there are more than one open network upgrade proposal they are all shown on the landing page
[ ] Network upgrade proposals are visible and highlighted on the proposals page
[ ] There is a way to see all the details of the release that will be upgraded to on github https://github.com/vegaprotocol/vega/tags (until this ticket is done https://github.com/vegaprotocol/frontend-monorepo/issues/3260)
[ ] Network upgrade proposals are pinned to the top of the proposals list on the proposals page
[ ] Once the network proposal has closed, it goes into the closed list, sorted by whatever parameters already sort that list (closed date)

Tasks

[X] Finalise copy
[ ] Visual design
[ ] Implement copy change on the landing page to indicate that network upgrades being proposed by validators
[ ] Implement new network proposals module on the landing page
[ ] Implement copy change on the proposals page to indicate that network upgrades being proposed by validators
[ ] Get upgrade network proposals showing on the proposals page
[ ] Implement visual differentiation of network upgrade proposal in the proposals list
[ ] Implement pinning of network upgrade proposals to the top of the proposals list
[ ] QA test (Verify meets acceptance criteria, exploratory testing for bugs, check adherence to visual design, add automated regression tests for the A/Cs)
[ ] Release

Sketch

Additional details / background info
Details of what data is available from the API to facilitate this story can be found in this spike #2992

Note: there's no closed date available for the proposals currently, and no way to map block height to a timestamp. So for now, closed proposals will sit at the top of the closed proposals list. @pennyandrews @ckpringle


[windows] add a new bundle upgrade code for 5.10 release

Testing a build first.

This appears to work with compnerd/swift-build :)


Monitor not changing state

Hi Zvi,
Here my question that I also posted in the community of squaredup.
Hi,
Has anyone experience with the SQL Query MP. I did some testing with a count query that returns values around 58 and set the monitor at ‘greater than 55’. I do the monitoring every 5 minutes and will give an alert by 2 samples. The monitor stays on the OK status. I need to do a recalculate to get this in error and after a refresh health it stays on OK.
Has anyone a solution for that?
Thanks,
Luc

I have done some changes.
Sorry but you would not to delete the current MPs in your SCOM environment and import new MPs
version <IP_ADDRESS> to keep it more easy to maintain I change library mp to version <IP_ADDRESS>
QueryOleDbMonitorTemplate.mpb
QueryOleDbMonitorLibrary.mp
I hope this will help.
If not, I will keep looking for the root cause of this problem

Have just updated to <IP_ADDRESS>, now trying to set this up again, performance is working as expected but as  Monitors / Alerts dont seem to be working :(
I have it set to check every 5 minutes and if greater than 0 then raise an alert (For 1 sample) but no alerts are generated

I fixed the issue with the monitor not change state
QueryOleDbMonitorLibrary.mp
QueryOleDbMonitorTemplate.mpb
I will wait for your feedback before I will close the issue.
thanks

Have installed the pack and now gone to edit and save and now get this :(
Note:  The following information was gathered when the operation was attempted.  The information may appear cryptic but provides context for the error.  The application will continue to run.
: Processing the template failed. See inner exception for details.
Verification failed with 1 errors:
Error 1:
Found error in 2|UnknowMP|<IP_ADDRESS>|UnknowMP|| with message:
XSD verification failed for the management pack. [Line: 0, Position: 0]
System.Xml.Schema.XmlSchemaValidationException: The 'AlertParameter7' element is invalid - The value 'Provider=SQLOLEDB;Server=SERVER;Initial Catalog=DB;User Id=$RunAs[Name="OleDb.f32b5c9916d5482cbe098bc48fe21b48.SimpleAuthenticationAccount"]/UserName$;Password=$RunAs[Name="OleDb.f32b5c9916d5482cbe098bc48fe21b48.SimpleAuthenticationAccount"]/Password$' is invalid according to its datatype 'ManagementPackString' - The actual length is greater than the MaxLength value. ---> System.Xml.Schema.XmlSchemaException: The actual length is greater than the MaxLength value.
--- End of inner exception stack trace ---
at System.Xml.Schema.XmlSchemaValidator.SendValidationEvent(XmlSchemaValidationException e, XmlSeverityType severity)
at System.Xml.Schema.XmlSchemaValidator.CheckElementValue(String stringValue)
at System.Xml.Schema.XmlSchemaValidator.ValidateAtomicValue(String stringValue, XmlSchemaSimpleType& memberType)
at System.Xml.Schema.XmlSchemaValidator.InternalValidateEndElement(XmlSchemaInfo schemaInfo, Object typedValue)
at System.Xml.XsdValidatingReader.ProcessEndElementEvent()
at System.Xml.XsdValidatingReader.Read()
at Microsoft.EnterpriseManagement.Configuration.XSDVerification.ValidateManagementPack(TextReader mpcontents, ManagementPack mp, Boolean throwError)
XSD verification failed for the management pack. [Line: 0, Position: 0]
: Verification failed with 1 errors:
Error 1:
Found error in 2|UnknowMP|<IP_ADDRESS>|UnknowMP|| with message:
XSD verification failed for the management pack. [Line: 0, Position: 0]
System.Xml.Schema.XmlSchemaValidationException: The 'AlertParameter7' element is invalid - The value 'Provider=SQLOLEDB;Server=SERVER;Initial Catalog=DB;User Id=$RunAs[Name="OleDb.f32b5c9916d5482cbe098bc48fe21b48.SimpleAuthenticationAccount"]/UserName$;Password=$RunAs[Name="OleDb.f32b5c9916d5482cbe098bc48fe21b48.SimpleAuthenticationAccount"]/Password$' is invalid according to its datatype 'ManagementPackString' - The actual length is greater than the MaxLength value. ---> System.Xml.Schema.XmlSchemaException: The actual length is greater than the MaxLength value.
--- End of inner exception stack trace ---
at System.Xml.Schema.XmlSchemaValidator.SendValidationEvent(XmlSchemaValidationException e, XmlSeverityType severity)
at System.Xml.Schema.XmlSchemaValidator.CheckElementValue(String stringValue)
at System.Xml.Schema.XmlSchemaValidator.ValidateAtomicValue(String stringValue, XmlSchemaSimpleType& memberType)
at System.Xml.Schema.XmlSchemaValidator.InternalValidateEndElement(XmlSchemaInfo schemaInfo, Object typedValue)
at System.Xml.XsdValidatingReader.ProcessEndElementEvent()
at System.Xml.XsdValidatingReader.Read()
at Microsoft.EnterpriseManagement.Configuration.XSDVerification.ValidateManagementPack(TextReader mpcontents, ManagementPack mp, Boolean throwError)
: XSD verification failed for the management pack. [Line: 0, Position: 0]

I did not know but it turns out that the AlertParameter value has a max size of 256 characters.
which means I will have to remove the connection string and query string from an alert description.
Thanks

DONE! current alert example

QueryOleDbMonitorLibrary
QueryOleDbMonitorTemplate

Seems to be working as expected now ! :)

Great !!


Fix bug with no-op coerce in the metadata table

Coverage increased (+0.001%) to 95.284% when pulling b4b0c56a8cf09d5b231537a4ef3fe4934440e486 on no_op_coerce_metadata_fix into 9511cd3da09e74eb362e65cff05a11c9508f28a9 on master.

I'll merge and release in v1.8.1 as soon as CI finishes.


Logging of Jetty verbose

AFAICS there is no option to set Jetty to a less verbose logging mode. Running Spark I get a lot of output. I currently pipe system.out. into a file, but a specific logging function would be nice.
See also: http://stackoverflow.com/questions/9235809/how-to-enable-debug-level-logging-with-jetty-embedded

Hello team
what is the status of this feature? Is there a way now I can turn off console logs ?
Thank you
Mritunjay Dubey

Not sure if fixed. Closing this because we need to clean up. Please create a new issue if this is still a problem.


Easy way to change arrow color ? Not the background color

I want to change the inner arrow color. In the demo the browser has the css as
font-size: 20px;
    line-height: 1;
    color: #00558B;
    opacity: .75;

how do i replicate this? Is there an easy way to change this color property? I have no issue with custom arrow changing background, but i only want to change arrow color.
https://jsfiddle.net/9raxke45/
Cheers!

Just target the button's contents directly like this:
.slick-prev:before,
.slick-next:before {
  color: yellow;
}

https://jsfiddle.net/ss6LedtL/

Ah thanks for the quick response :)

@radiovisual
Dear sir,
this method dosenot work with me.

@radiovisual
When i add style to component directly dosenot work and when but it in styles in assets folder it work
thanks a lot

If you want to change the component directly without changing anything in slick.css, you can add !important next to the color like this:
.slick-prev:before,
.slick-next:before {
color: red!important;
}

which place include this? like my style.css or nodemodules ?

Try adding it to style.css

@ieatcrayolaaa sorry but it not working directly any other way to solve this issue

This is incorrect when using react and typescript. You have to make an override stylesheet, CSS, not SCSS and include that to over write the styles.
import './YourCss.css'
and then you can over ride the default styling.

If you want to change the component directly without changing anything in slick.css, you can add !important next to the color like this:
.slick-prev:before,
.slick-next:before {
  color: red!important;
}

!important is really important!

hi there, what about change color of the arrow when we set (infinite:false) (so the slide will come to an end)

//How to change the color for this code

   //how to change the color for this

Checkout my answer on custom Slider Carousel on stack overflow for React:
https://stackoverflow.com/questions/29876185/change-the-arrow-buttons-in-slick-slider/77772400#77772400


Apply networkpolicy namespaceselector cause can't access service via ClusterIP

It's getting timeout when access the service via ClusterIP
try to deploy the example
https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/06-allow-traffic-from-a-namespace.md
There are two conclusions was found after test:

It CAN NOT access service via ClusterIP across the node but inter node is well.
It CAN access PodIP acording the networkpolicy.

Expected Behavior
Can access the service via ClusterIP from arbitary node in cluster acording the networkpolicy
Possible Solution
I think maybe the issue comes from the kube-proxy, because the networkpolicy only effects pods that hostNetwork=false.
Steps to Reproduce (for bugs)

Acording the example above, try to access url from different node

Your Environment

k8s: v1.17.16
calico: v3.18.0
kube-proxy :
mode: ipvs
hostnetwork: true
core-dns: enabled

Can access the service via ClusterIP from arbitary node in cluster acording the networkpolicy

This should absolutely work.  The way this is supposed to work is that:

source pod sends to cluster IP
kube-proxy on the source pod's node DNATs the packet to one of the pods backing the service
packet is sent exactly as if it were a pod-pod packet
packet arrives at destination pod - network policy is applied (exactly as if the packet was pod-pod originally).

Does connectivity work if you send to the pod IP direct?  If not, need to fix that first.
If pod-pod works, then look at service endpoints - check that the destination pod is in the list of endpoints for the service.  If it is, check kube-proxy logs - did it create the iptables rule to NAT the packet to the service?
If none of those work, perhaps join us in the calico slack http://slack.projectcalico.org/ and we can try to help you there?

Thanks @lwr20 .
Here is the context :
Namespace NS1:
(label app=net)

Service A (<IP_ADDRESS>:8080) -> Endpoint(<IP_ADDRESS>:8766) -> Pod A1
Pod T1 (a test pod)

Namespace NS2

Pod T2 (a test pod)

Networkpolicy
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-allow-prod
  namespace: ns1
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          app: net
    - namespaceSelector:
        matchLabels:
          name: kube-system
  egress:
  - {}

Test:

Pod T1 -> Endpoint(<IP_ADDRESS>:8766)     ✔
Pod T1 -> Service A (<IP_ADDRESS>:8080)  ❌
Pod T2 -> Endpoint(<IP_ADDRESS>:8766) ❌   (acording to the policy)

ipvs:
ipvsadm -Ln  | grep <IP_ADDRESS> -B1

TCP  <IP_ADDRESS>:8080 rr
  -> <IP_ADDRESS>:8766          Masq    1      0          0

test in pod:
# curl -m 2 <IP_ADDRESS>:8766
OK
# curl -m 2 <IP_ADDRESS>:8080
curl: (28) Connection timed out after 2001 milliseconds

endpoint:
kubectl -n ns1 get ep
service-a         <IP_ADDRESS>:8766                       3d

service:
kubectl -n ns1 get svc
service-a         ClusterIP   <IP_ADDRESS>   <none>        8080/TCP   3d

Hi @lwr20
After modify the cidr to <IP_ADDRESS>/16 still can't access via clusterIP.

modify the config and flag --cluster-cidr and restart kube-proxy's daemonset.

apiVersion: kubeproxy.config.k8s.io/v1alpha1
    kind: KubeProxyConfiguration
    bindAddress: <IP_ADDRESS>
    clientConnection:
      acceptContentTypes: ""
      burst: 10
      contentType: application/vnd.kubernetes.protobuf
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 5
    # Use the ippool of calico
    clusterCIDR: <IP_ADDRESS>/16
    configSyncPeriod: 15m0s
    conntrack:
      maxPerCore: 32768
      min: 131072
      tcpCloseWaitTimeout: 1h0m0s
      tcpEstablishedTimeout: 24h0m0s
    enableProfiling: false
    healthzBindAddress: <IP_ADDRESS>:10256
    iptables:
      masqueradeAll: true
      masqueradeBit: 14
      minSyncPeriod: 0s
      syncPeriod: 30s
    ipvs:
      minSyncPeriod: 0s
      # rr: round-robin
      # lc: least connection
      # dh: destination hashing
      # sh: source hashing
      # sed: shortest expected delay
      # nq: never queue
      scheduler: rr
      syncPeriod: 30s
    metricsBindAddress: <IP_ADDRESS>:10249
    mode: ipvs
    oomScoreAdj: -999

Acording to the IPIP rules access the service IP is corssing the subnet, it pass through the tunl0.

Just spotted this:

I think maybe the issue comes from the kube-proxy, because the networkpolicy only effects pods that hostNetwork=false.

pods running with hostNetwork=true are effectively running directly on the host.  Their traffic is indistinguishable from traffic from the host itself, so kubernetes networkpolicy will not apply to them.  Instead, you have to use Calico's Global Network Policy and Host Endpoints for this. https://docs.projectcalico.org/security/tutorials/protect-hosts

In addition, a host will always be able to talk to pods on the same node regardless of network policy (because kubelet healthchecks need to work).

Thanks @lwr20
Issue still here and all pod are hostNetwork=false

After modify the cidr to <IP_ADDRESS>/16 still can't access via clusterIP.

Exactly how did you modify the cidr?  https://docs.projectcalico.org/networking/migrate-poolsDid you restart all pods after that, so that they get new IPs in that CIDR?

Can I recommend joining us in Slack for this?  I think the discussion will go faster there.  http://slack.projectcalico.org/

After modify the cidr to <IP_ADDRESS>/16 still can't access via clusterIP.

Exactly how did you modify the cidr? The correct steps are in https://docs.projectcalico.org/networking/migrate-pools

Yes, I modify the kube-proxy's cidr

My cluster has the same issues.
After I changed the ESXi, this fault returned to normal.
So I suspect that there is a problem with esxi, or there is a problem with the NIC driver.

Thanks @lwr20 @lindong629
Problem has been resolved.
Modify kube-proxy confuguration masqueradeAll: false and restart all kube-proxy.


Rotation is a significant factor in runtime

From profiling the function generate_polycubes, for larger values of n the time spent on rot90 is a significant factor, with 45 of the 108 second runtime being spent in it for n=9 (see table at bottom of text).
So far I've been thinking about a couple of potential solutions, but I haven't reach a point where either solution could be implemented:
Solution A - Writing a purpose-specific rot90 or all_rotations
Numpy appears to use index swapping to achieve 90 degree rotation of an array (definitely faster than a transformation matrix I would imagine), but it has significant overhead as it is written to work for n-dimensional arrays. In the case of polycubes, we know that the array we want to rotate is 3D, and that we want all 28 rotations. Writing a purpose specific version of this function could save on overhead time, but I think it would most likely need to be done in another language.
I haven't really been pursuing this as I don't know any low level or performant languages to try this in, and I think we won't save time with a pure python implementation.
Solution B - Representing the polycube as three 2-D views
This solution appeals to me far more than solution A, as it reduces either our comparisons between hashes, or our memory used on storing hashes (if we store rotations). As we conveniently have a boolean array aligned with the axes, I think we can represent any polycube with three, fast to calculate, 2-D views of the polycube:
def view_2d(polycube):
    view1 = np.sum(polycube, axis=0)
    view2 = np.sum(polycube, axis=1)
    view3 = np.sum(polycube, axis=2)

    return view1, view2, view3

For the simple L shaped (cropped) polycube with a length of 4 and 1 cube off the a side, the views would be represented this way:
[4, 1], [1, 1, 1, 2], [1, 0, 1, 0, 1, 0, 1, 1]
We can avoid worrying about the order in which these appear by using collections.Counter or generally by treating it as an unsorted list. We could then compare the view to a rotated version of the polycube by ensuring that all entries match only once, hence finding that both polycubes are identical...
Unfortunately, though this works for comparing the two 1-D views of a polysquare with some additional checks, I haven't found a way to ensure these views match for polycubes after any of the 28 arbitrary rotations:
polycube1_views         -> [4, 1], [1, 1, 1, 2], [1, 0, 1, 0, 1, 0, 1, 1]
polycube1_rotated_views -> [1, 4], [1, 0, 1, 0, 1, 1, 1], [1, 1, 1, 2]

We can see from this that some rotations will give us outputs that are very hard to compare to the original polycube for equality.
Perhaps a solution to this is to rebuild the arbitrary polycube in a specific manner to always give the same output? I don't have a solution for this yet.
Profiling
cumtime is the value we're worried about for rot90 in this case, as looking at the source code for this function, it calls a number of other functions to achieve rotation by 90 degrees, with flip being the main point of concern.
With optimisations from the PRs on the original repo (bertie2, georgedorn, and mine), the cProfile.run output looks like this for n=9:
         67515198 function calls (64901683 primitive calls) in 108.627 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   669114    1.084    0.000    8.019    0.000 <__array_function__ internals>:177(any)
     8151    0.013    0.000    0.055    0.000 <__array_function__ internals>:177(around)
        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:177(copyto)
  1693830    2.401    0.000   21.180    0.000 <__array_function__ internals>:177(flip)
  1578102    2.321    0.000    4.637    0.000 <__array_function__ internals>:177(packbits)
     8151    0.018    0.000    0.794    0.000 <__array_function__ internals>:177(pad)
  1637369    2.486    0.000   45.442    0.000 <__array_function__ internals>:177(rot90)
     8151    0.014    0.000    0.091    0.000 <__array_function__ internals>:177(round_)
  1338228    1.863    0.000    6.435    0.000 <__array_function__ internals>:177(swapaxes)
   903376    1.308    0.000    6.317    0.000 <__array_function__ internals>:177(transpose)
        1    0.055    0.055  108.627  108.627 <string>:1(<module>)
    32604    0.023    0.000    0.023    0.000 arraypad.py:109(<genexpr>)
    32604    0.021    0.000    0.021    0.000 arraypad.py:120(<genexpr>)
    24453    0.125    0.000    0.155    0.000 arraypad.py:129(_set_pad_area)
    48906    0.030    0.000    0.030    0.000 arraypad.py:33(_slice_at_axis)
    16302    0.086    0.000    0.220    0.000 arraypad.py:454(_as_pairs)
     8151    0.002    0.000    0.002    0.000 arraypad.py:521(_pad_dispatcher)
     8151    0.169    0.000    0.746    0.000 arraypad.py:529(pad)
    24453    0.033    0.000    0.033    0.000 arraypad.py:58(_view_roi)
     8151    0.105    0.000    0.157    0.000 arraypad.py:86(_pad_simple)
  1578102    8.903    0.000   17.019    0.000 cubes.py:152(rle)
   223038    0.341    0.000    2.796    0.000 cubes.py:174(cube_exists_rle)
  1693830    4.488    0.000   40.548    0.000 cubes.py:24(single_axis_rotation)
   223038    7.844    0.000   22.298    0.000 cubes.py:44(crop_cube)
   231189    1.430    0.000   24.981    0.000 cubes.py:64(expand_cube)
  1411525    7.328    0.000   57.259    0.000 cubes.py:9(all_rotations)
      8/1    8.456    1.057  108.572  108.572 cubes.py:96(generate_polycubes)
   669114    0.179    0.000    0.179    0.000 fromnumeric.py:2328(_any_dispatcher)
   669114    1.046    0.000    5.503    0.000 fromnumeric.py:2333(any)
    16302    0.004    0.000    0.004    0.000 fromnumeric.py:3241(_around_dispatcher)
     8151    0.010    0.000    0.031    0.000 fromnumeric.py:3245(around)
     8151    0.011    0.000    0.067    0.000 fromnumeric.py:3754(round_)
  2249755    1.892    0.000    4.635    0.000 fromnumeric.py:51(_wrapfunc)
  1338228    0.290    0.000    0.290    0.000 fromnumeric.py:546(_swapaxes_dispatcher)
  1338228    1.331    0.000    3.073    0.000 fromnumeric.py:550(swapaxes)
   903376    0.196    0.000    0.196    0.000 fromnumeric.py:597(_transpose_dispatcher)
   903376    1.029    0.000    3.903    0.000 fromnumeric.py:601(transpose)
   669114    1.598    0.000    4.458    0.000 fromnumeric.py:69(_wrapreduction)
   669114    0.590    0.000    0.590    0.000 fromnumeric.py:70(<dictcomp>)
  1637369    0.357    0.000    0.357    0.000 function_base.py:154(_rot90_dispatcher)
  1637369   11.148    0.000   40.823    0.000 function_base.py:158(rot90)
  1693830    0.342    0.000    0.342    0.000 function_base.py:248(_flip_dispatcher)
  1693830    8.105    0.000   16.712    0.000 function_base.py:252(flip)
  3387660    0.842    0.000    0.842    0.000 index_tricks.py:765(__getitem__)
        1    0.000    0.000    0.000    0.000 multiarray.py:1079(copyto)
  1578102    0.333    0.000    0.333    0.000 multiarray.py:1175(packbits)
  1693830    5.360    0.000    7.474    0.000 numeric.py:1348(normalize_axis_tuple)
  1693830    1.112    0.000    1.469    0.000 numeric.py:1398(<listcomp>)
        1    0.000    0.000    0.000    0.000 numeric.py:150(ones)
  1693830    0.227    0.000    0.227    0.000 {built-in method _operator.index}
     8151    0.001    0.000    0.001    0.000 {built-in method builtins.callable}
        1    0.000    0.000  108.627  108.627 {built-in method builtins.exec}
  2249755    0.394    0.000    0.394    0.000 {built-in method builtins.getattr}
  1693830    0.290    0.000    0.290    0.000 {built-in method builtins.hasattr}
  5025116    0.608    0.000    0.608    0.000 {built-in method builtins.len}
       94    0.009    0.000    0.009    0.000 {built-in method builtins.print}
       87    0.000    0.000    0.000    0.000 {built-in method builtins.round}
   903376    1.746    0.000    1.746    0.000 {built-in method numpy.arange}
   247491    0.432    0.000    0.432    0.000 {built-in method numpy.array}
  1637369    0.242    0.000    0.242    0.000 {built-in method numpy.asanyarray}
    16302    0.008    0.000    0.008    0.000 {built-in method numpy.asarray}
7844473/5230965    8.905    0.000   56.396    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
  1693830    0.357    0.000    0.357    0.000 {built-in method numpy.core._multiarray_umath.normalize_axis_index}
     8152    0.008    0.000    0.008    0.000 {built-in method numpy.empty}
  1355064    0.495    0.000    0.495    0.000 {method 'add' of 'set' objects}
    56461    0.012    0.000    0.012    0.000 {method 'append' of 'list' objects}
     8151    0.021    0.000    0.021    0.000 {method 'astype' of 'numpy.generic' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
  1578102    3.479    0.000    3.479    0.000 {method 'flatten' of 'numpy.ndarray' objects}
     8151    0.002    0.000    0.002    0.000 {method 'get' of 'dict' objects}
   669114    0.090    0.000    0.090    0.000 {method 'items' of 'dict' objects}
    16302    0.037    0.000    0.037    0.000 {method 'nonzero' of 'numpy.ndarray' objects}
     8151    0.008    0.000    0.008    0.000 {method 'ravel' of 'numpy.generic' objects}
     8151    0.002    0.000    0.002    0.000 {method 'ravel' of 'numpy.ndarray' objects}
   669114    2.180    0.000    2.180    0.000 {method 'reduce' of 'numpy.ufunc' objects}
     8151    0.009    0.000    0.009    0.000 {method 'round' of 'numpy.ndarray' objects}
  1338228    0.499    0.000    0.499    0.000 {method 'swapaxes' of 'numpy.ndarray' objects}
   903376    1.842    0.000    1.842    0.000 {method 'transpose' of 'numpy.ndarray' objects}

I think your onto something with the views, the hard part will be proving that the views are unique, i'm going to run a test with the n=12 data to check.
as for an efficient way to compare the views, I think we just need to come up with a quick to compute canonical direction and order for the vectors.
so in your example:
polycube1_views         -> [4, 1], [1, 1, 1, 2], [1, 0, 1, 0, 1, 0, 1, 1]
polycube1_rotated_views -> [1, 4], [1, 0, 1, 0, 1, 1, 1], [1, 1, 1, 2]
the problem I foresee is after doing all these checks to get the right order, is it any faster than the rot90?
more testing to be done.

ah wait i'm dumb, i was packing to bits and changing 2's into 1's

ok so you cannot in fact do a naive re-arrangement of the views based on their properties, the problem you hit is that certain rotations and flips swap 2 axis,
the simplex example I could find was:

the rotation essentially swaps 2 of the views because it is the same shape along different axis.
the interesting part is ... that's a mirror!, and we have identified it as the same, the current code doesn't identify mirrors as the same, so is this actually an improvement in behavior ? or a regression ?

some of these are really hard to tell if they are a mirror or not just by looking at them, any idea on how to make proper code to check for mirrors?

I cant seem to get this to work, it does in fact collide more than just mirrors, at least with my naïve approach of sorting the vectors, (picking the largest out of each view and its reverse, and then ordering the views by size). I have pushed my code at
https://github.com/bertie2/opencubes/tree/view_test
this is not saying this idea cant work, just that I couldn't make it work

I had opened an issue in cubes and had the idea of using radii to make it rotation independent. Maybe we could look into it. Or a multihash solution. Like not for 28 or so rotations but some specific ones and change all to a uniform structure, like to prefere some axis in the uniform structure. So we need less hashes and still count all shapes.

@bertie2 ,

any idea on how to make proper code to check for mirrors?

# polycube_1 and polycube_2 are both truncated `np.int8` ndarrays

id_1 = get_canonical_packing(polycube_1, set())
id_2 = get_canonical_packing(polycube_2, set())
id_2_mirror = get_canonical_packing(np.flipud(polycube_2), set())

are_mirrors = id_1 != id_2 and id_1 == id_2_mirror

I cant seem to get this to work, it does in fact collide more than just mirrors

The collision you have in test.ipynb is actually from a mirror operation! Consider mirroring the polycube along the X/Y plane, where the furthest cube (along our Z, or depth of the image) is our starting point. This is the same polycube but rotated.
I think the fundamental issue we will likely run into is that this allows "illegal" rotations, or mirroring, which are actually just 4D rotations, or p(n) as described by Kevin Gong. The same actually happens in 2D, representing the two skew tetrominoes S, Z this way (3D rotation to achieve mirroring). This could be useful for calculating p(n), which tends towards 2 * r(n). This could still be useful for r(n) as we could then do a search specifically for mirrored cubes, and "unmirror" them. I'll keep looking into any other collisions and see if we run into any other collisions that aren't mirrors.

Just an afterthought, it makes heaps of sense that p(n) and r(n) appear to be linked by just a factor of 2. As n grows, we get more polycubes with chirality, relative to the number of achiral polycubes, and for each chiral polycube we will get exactly 1 other polycube with the inverse chirality. This is a lot easier to think about in 2D, where we will find more polysquares for larger values of n that we can mirror (3D rotate) in such a way that no 2D rotation can achieve the same shape.

@ana-096 ,

Solution B - Representing the polycube as three 2-D views

for example, if shape is (4,4,4), then there are 64 variables (that can be 0 or 1), but only 3*16=48 equations that constrain them -> for some system of these equations more than 1 solution could be possible.
This probability grows with shape.
Is this not a problem? I'll try to generate a collision and update this.

@ana-096 ,
https://gist.github.com/VladimirFokow/c314ab45dbe1c97d763b4e81a0a3728e

@VladimirFokow your example is for a polycube n=33, did you find any smaller collisions? I did some quick tests for arbitrary shapes 2 < x,y,z < 10 (yes this is taking a long time) and haven't found any smaller collisions yet. Kevin Gong only got to n=16, so if there's no issues for collision, or we account for it, up to n=17, this could help us get further.

@ana-096 https://gist.github.com/VladimirFokow/aebc3dc937d0585018690a8aceba3ebf for n=9 and n=10

@VladimirFokow are you checking for mirrors? The examples you provided for n=9 I think are mirrors, but it's hard to tell when it's represented as an array.

for example, if shape is (4,4,4), then there are 64 variables (that can be 0 or 1), but only 3*16=48 equations that constrain them -> for some system of these equations more than 1 solution could be possible. This probability grows with shape.
Is this not a problem? I'll try to generate a collision and update this.

For a (4,4,4) boolean array, there are 2 ^ 64 values, represented as 2D views, they would each be (4,4) integer arrays, where the value of any given cell is 0 <= x <= 4, hence 5 ^ 16 values. This is approximately 2 ^ 37 values.
In general for 2 dimensions, x, y, the number of possible values is 2 ^ (xy), while the 1D views would be (x+1) ^ (y) and (y+1) ^ (x). The number of possible values in the x,y sized boolean array grows faster than the views, however I haven't yet figured out how many of the values can be attributed to mirrors and rotationally symmetric arrays.


Coordinators Update in Coordinators Team Page

Some coordinators needed to be removed from this page.
Coordinators to be removed :

Ankur Pratap Singh
Mhd Sahil
Karim Mohammad
Harsh Kumar
Anshika Jain
And comment out Mehul Singhal (do not delete the card of Mehul Singhal from the code, only comment it out).

Also, Add/Update senior coordinators details from this link.

Hi @Hardik-Mittal,
I would like to fix this issue. Can you please assign it to me?
Thanks,
Pramod

Hi @pramod74 ,
Sorry, this issue is only for club members. So, I can't assign it to you.

No problem, Hardik. Thanks for the update.

I am Abhinav from Robotics club. I will take up this issue.
Regards,
Abhinav

Okay @agg-geek, I have assigned this issue to you. You can start working on it.

I would like to fix this issue. Can you please assign it to me?

@Lalit1471 I have assigned this issue to you.


[BUG]: Cannot restore the lr_scheduler correctly

🐛 Describe the bug
After saving optimizer and learning rate scheduler states after 6,000 steps, and successfully restore the states , I observe an incorrect learning rate at the first step after restoration (step 6,001). Specifically, the learning rate(lr_scheduler.get_last_lr()[0]  or lr_scheduler.get_lr()[0]) seems to be reset to the initial value.

plugin=gemini
booster = Booster(plugin=plugin)
....
optimizer = HybridAdam(model.parameters(), lr=args.lr, betas=(0.9, 0.95), weight_decay=args.weight_decay)

lr_scheduler = CosineAnnealingWarmupLR(
    optimizer, total_steps=args.max_steps, warmup_steps=args.warmup_steps, eta_min=0.1 * args.lr
)
....

for step in dataloader:
    ....
    optimizer.step()
    lr_scheduler.step()
    optimizer.zero_grad()
    ....
    log.info('lr', lr_scheduler.get_last_lr()[0])

Environment
Pytorch: 2.0.1
ColossalAI: 0.3.4

It appears that the issue you're encountering is due to the param_group['lr'] not being saved correctly. This is a common challenge when restoring the state of an optimizer and learning rate scheduler in deep learning models. As of now, I haven't been able to determine the exact reason why param_group['lr'] is not saved as expected. However, to address this, I've developed a temporary solution that ensures the learning rate is correctly set upon restoration. Here's the refined code snippet:
def load_checkpoint(booster, model, optimizer, lr_scheduler):
    args = get_args()
    load_dir = args.load_dir

    # Load model, optimizer, and lr_scheduler states
    booster.load_model(model, os.path.join(load_dir, "model"))
    booster.load_optimizer(optimizer, os.path.join(load_dir, "optimizer"))
    booster.load_lr_scheduler(lr_scheduler, os.path.join(load_dir, "lr_scheduler"))

    # Correctly set the learning rate for each param_group
    for i, data in enumerate(zip(optimizer.param_groups, lr_scheduler.get_last_lr())):
        param_group, lr = data
        param_group['lr'] = lr

This code ensures that each parameter group's learning rate is updated to match the last recorded learning rate from the scheduler. By doing this, we can maintain consistency in the learning rate when resuming training from a checkpoint. This approach is a workaround until a more permanent solution is identified for the underlying issue of param_group['lr'] not being saved properly.

It appears that the issue you're encountering is due to the param_group['lr'] not being saved correctly. As of now, I haven't been able to determine the exact reason why param_group['lr'] is not saved as expected. However, to address this, I've developed a temporary solution that ensures the learning rate is correctly set upon restoration. Here's the refined code snippet:
def load_checkpoint(booster, model, optimizer, lr_scheduler):
    args = get_args()
    load_dir = args.load_dir

    # Load model, optimizer, and lr_scheduler states
    booster.load_model(model, os.path.join(load_dir, "model"))
    booster.load_optimizer(optimizer, os.path.join(load_dir, "optimizer"))
    booster.load_lr_scheduler(lr_scheduler, os.path.join(load_dir, "lr_scheduler"))

    # Correctly set the learning rate for each param_group
    for i, data in enumerate(zip(optimizer.param_groups, lr_scheduler.get_last_lr())):
        param_group, lr = data
        param_group['lr'] = lr

This code ensures that each parameter group's learning rate is updated to match the last recorded learning rate from the scheduler. By doing this, we can maintain consistency in the learning rate when resuming training from a checkpoint. This approach is a workaround until a more permanent solution is identified for the underlying issue of param_group['lr'] not being saved properly.

Thanks so much @imgaojun

Perhaps using lr_scheduler.step(step) after restoring the learning rate scheduler is a simpler approach, if we know the step when saving lr_scheduler.


%WindowsSdkDir%Include\cppwinrt\winrt not correct

There is no idl here, just cppwinrt generated code,
One usually wants to write idl and then invoke cppwinrt.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2dacbdf2-172f-dd41-829f-94295cd0c621
Version Independent ID: b511accc-e0b1-9b30-d044-231d5204024d
Content: Introduction to Microsoft Interface Definition Language 3.0 - Windows UWP applications
Content Source: winrt-related-src/midl-3/intro.md
Product: uwp
GitHub Login: @stevewhims
Microsoft Alias: stwhi

Hi, @kingofthebongo2008! You're right--thank you for pointing that out, that's my mistake. I'll go ahead and fix that error.
~Steve

Thanks again, @kingofthebongo2008, I've fixed the issue you pointed out. I'll close this one now. ~Steve


add FalkorDB support

Fixes # (issue)

@jacoblee93 this PR is replacing the original PR #2774

@jacoblee93  I don't have access to the vercel logs, so I’m unable to investigate the failure directly. Could you please take a look and let me know if anything is wrong? Let me know if there’s anything else I can do to help get this PR merged