panic on most methods to select/examine/fetch from gmail

First off, thank you for your work on this crate.
I'm having trouble getting almost all session methods to work with authenticating with an app password with gmail
let domain = "...";
let tls = TlsConnector::builder().build().unwrap();

let client = imap::connect((domain, 993), domain, &tls)
            .expect(&format!("failed to connect to {}", domain));

let mut session = client.login(username, app_password).expect("failed to authenticate with domain");

println!("{:#?}", session. select("INBOX").expect("failed to select inbox"));

In this example and many others I'll get Parse(Invalid(...)) errors. When I view the bytes as a string in this case I see
 "* FLAGS (\\Answered \\Flagged \\Draft \\Deleted \\Seen $NotPhishing $Phishing OIB-Seen-INBOX OIB-Seen-OIB/Business OIB-Seen-OIB/Entertainment OIB-Seen-OIB/Finance OIB-Seen-OIB/Groups OIB-Seen-OIB/Home OIB-Seen-OIB/Jobs OIB-Seen-OIB/News OIB-Seen-OIB/Shopping OIB-Seen-OIB/Social Networking OIB-Seen-OIB/Travel OIB-Seen-[Gmail]/All Mail OIB-Seen-[Gmail]/Important OIB-Seen-[Gmail]/Spam OIB-Seen-[Gmail]/Trash)\r\n"

I'm not familiar with imap at the protocol level but am I missing something here?

Hmm, that's very interesting... Thanks for reporting it!
Could you please post the full error + backtrace that you get?

as far as i can see, imap-proto chokes on the ']' chars (in its parser, flag_char references atom which is everything that is not an atom special and those include b']' in resp_specials.
not sure what the best path forward is, though i'd think towards having the parser be more liberal here.

Ah, good catch! Let's see what @djc has to say here ^

Thanks for the super fast response folks. I'll post a stacktrace soon. There's actually a Gmail example in this repo but uses oauth tokens instead of an app secret. I imagine that may be a repoducable case as well

Speaking of, we should probably change that example to use app secrets too, since that's probably the recommended approach these days anyways. Maybe you could submit a PR with that once you get a working example?

ok so this isn't that helpful but this is the output of RUST_BACKTRACE=full cargo run in my case. I actually need to update the title of this issue. the library doesn't panic. I panic ( I'm calling result.expect(...) on the fetch to select my inbox because I always expect that to succeed.
The issue is more like many methods are result in Err(Parse(...)) is more what I was going for
stack backtrace:
   0:        0x10b342c8f - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::haab3e9e4ad6801d1
                               at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1:        0x10b337e8d - std::sys_common::backtrace::print::ha116a70121383601
                               at libstd/sys_common/backtrace.rs:71
                               at libstd/sys_common/backtrace.rs:59
   2:        0x10b345583 - std::panicking::default_hook::{{closure}}::h111a5999ffc43b19
                               at libstd/panicking.rs:211
   3:        0x10b34530c - std::panicking::default_hook::h68d40004cf0facbc
                               at libstd/panicking.rs:227
   4:        0x10b345cb7 - <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get::h860b1aff5e2bb311
                               at libstd/panicking.rs:476
   5:        0x10b34581c - std::panicking::continue_panic_fmt::h604d00ec956509ef
                               at libstd/panicking.rs:390
   6:        0x10b345708 - std::panicking::try::do_call::ha38138adc359e978
                               at libstd/panicking.rs:325
   7:        0x10b37fc11 - core::ptr::drop_in_place::h17bb2eb4974d8208
                               at libcore/panicking.rs:77
   8:        0x10b2873ef - core::ptr::drop_in_place::hffa41ab4c16c3d8e
                               at libcore/macros.rs:26
   9:        0x10b286b7c - core::ptr::drop_in_place::hffa41ab4c16c3d8e
                               at libcore/result.rs:835
  10:        0x10b288005 - core::ptr::drop_in_place::hffa41ab4c16c3d8e
                               at lib/src/lib.rs:62
  11:        0x10b27ce93 - checkbox::main::h2180fa4ab32da1ee
                               at cli/src/main.rs:11
  12:        0x10b27d161 - std::rt::lang_start::{{closure}}::h8e17a3b02492aefd
                               at libstd/rt.rs:74
  13:        0x10b345687 - std::panicking::try::do_call::ha38138adc359e978
                               at libstd/rt.rs:59
                               at libstd/panicking.rs:310
  14:        0x10b351a2e - panic_unwind::imp::find_eh_action::{{closure}}::hd5dfb9ab0206071d
                               at libpanic_unwind/lib.rs:102
  15:        0x10b33bc0c - std::alloc::default_alloc_error_hook::hc04de1c966debfb0
                               at libstd/panicking.rs:289
                               at libstd/panic.rs:392
                               at libstd/rt.rs:58
  16:        0x10b27d141 - std::rt::lang_start::h4ae009f28048594c
                               at libstd/rt.rs:74
  17:        0x10b27cf11 - checkbox::main::h2180fa4ab32da1ee

I can add a new gmail the example. It's not going to be that different from the basic example because you just use the login method but I think it will be helpful for those getting started

I haven't looked to see if they are doing anything different ( I think they are actually using the same parser ) but this library also exists .

Pushed up a pull for that. This may help finding a fix simpler. I was able to reproduce this issue using the new example.
The example includes a link on how to create your own app password for gmail
cargo run --example gmail_app_password
   Compiling imap v0.10.0 (/Users/dougtangren/code/rust/rust-imap)
    Finished dev [unoptimized + debuginfo] target(s) in 3.86s
     Running `target/debug/examples/gmail_app_password`
Error selecting INBOX: Unable to parse status response
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[97, 50]`,
 right: `[97, 51]`', src/client.rs:1175:25
note: Run with `RUST_BACKTRACE=1` for a backtrace.

https://support.google.com/mail/forum/AAAAK7un8RUB3i6PX4aWp0/?hl=en some info on folder names in gmail

I think the same bug exists in the other project as well. I'll try to cross post the gh for visibility


#19 mariadb time type final (#8)

#19 Automatic tests for MariaDB time based types
Currently there is no "timestamp with timezone" ( https://jira.mariadb.org/browse/MDEV-10018 )

Codecov Report

Merging #26 (e8fb882) into main (137f7bd) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main      #26      +/-   ##
============================================
+ Coverage     96.11%   96.13%   +0.02%
- Complexity      265      267       +2
============================================
  Files            44       44
  Lines           771      775       +4
  Branches         56       55       -1
============================================
+ Hits            741      745       +4
  Misses           26       26
  Partials          4        4

Impacted Files
Coverage Δ

src/main/java/org/stdg/ColumnValueFormatter.java
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 137f7bd...e8fb882. Read the comment docs.

Thank you @raychenon


Added portfolio About and Contact section with CSSstyles

Added portfolio section and stylized it.
Added about section and stylized it.
Added contact section and stylized it.

Hi @YugynDprodigy10,
While you made a great effort in this project, unfortunately, I cannot proceed to review your code.
Invalid Code Review Request

Please open a Pull Request from new-feature into main. You have done it the other way around.

Your Code Review Request will be marked as invalid in your Dashboard, so please submit a new one once you are ready 🙏
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

Invalid Code Review Request does not count into the code reviews limit.


[Snyk] Fix for 1 vulnerabilities

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.
Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:

requirements/base.txt

⚠ Warning
python3-openid 3.2.0 has requirement defusedxml<=0.4.1, but you have defusedxml 0.6.0.
marshmallow-enum 1.5.1 requires marshmallow, which is not installed.
importlib-metadata 2.1.1 requires zipp, which is not installed.
croniter 0.3.36 requires natsort, which is not installed.

Vulnerabilities that will be fixed
By pinning:

Severity
Priority Score (*)
Issue
Upgrade
Breaking Change
Exploit Maturity

658/1000  Why? Proof of Concept exploit, Recently disclosed, Has a fix available, CVSS 5.3
Regular Expression Denial of Service (ReDoS) SNYK-PYTHON-JINJA2-1012994
Jinja2:  2.11.2 -> 2.11.3  jinja2:  2.11.2 -> 2.11.3
No
Proof of Concept

(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings
📚 Read more about Snyk's upgrade and patch logic

Codecov Report

Merging #80 (b0d43bf) into master (51195af) will decrease coverage by 3.66%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #80      +/-   ##
==========================================
- Coverage   67.04%   63.38%   -3.67%
==========================================
  Files        1022      489     -533
  Lines       50186    30180   -20006
  Branches     5204        0    -5204
==========================================
- Hits        33647    19129   -14518
+ Misses      16404    11051    -5353
+ Partials      135        0     -135

Flag
Coverage Δ

cypress
?

javascript
?

python
63.38% <ø> (-0.71%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset/db_engines/hive.py
0.00% <0.00%> (-85.72%)
:arrow_down:

superset/databases/commands/create.py
32.65% <0.00%> (-59.19%)
:arrow_down:

superset/sql_validators/postgres.py
50.00% <0.00%> (-50.00%)
:arrow_down:

superset/views/database/mixins.py
59.64% <0.00%> (-22.81%)
:arrow_down:

superset/db_engine_specs/hive.py
73.84% <0.00%> (-17.31%)
:arrow_down:

superset/databases/commands/update.py
85.71% <0.00%> (-8.17%)
:arrow_down:

superset/db_engine_specs/presto.py
81.38% <0.00%> (-7.15%)
:arrow_down:

superset/databases/schemas.py
93.40% <0.00%> (-6.05%)
:arrow_down:

superset/databases/dao.py
94.11% <0.00%> (-5.89%)
:arrow_down:

superset/databases/api.py
86.55% <0.00%> (-5.47%)
:arrow_down:

... and 553 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 51195af...b0d43bf. Read the comment docs.


exclude non-js manifest files from being imported

This came up when I tried some angular packages and got a lot off .d.ts files along with some .scss files. Potentially the sass files could be imported, but as of now it doesn't work anyhow. This makes it not break

Awesome. Thank you!

Released on 1.1.2:
https://www.npmjs.com/package/meteor-imports-webpack-plugin

that was fast!


chore: Add more obvious dependencies into README

Close #445

This PR is included in version 1.76.1 :tada:


[BUG] Unable to delete images from Train Menu

Describe the bug
I use Compreface and am whilst i can train and untrain images, I um unable to delete from the interface
Version of Double Take
<IP_ADDRESS>
Expected behavior
Expect the image to be deleted
Additional context
log:
error: TypeError: db.query is not a function
at module.exports.delete (/double-take/api/src/controllers/storage.controller.js:145:8)
at newFn (/double-take/api/node_modules/express-async-errors/index.js:16:20)
at Layer.handle [as handle_request] (/double-take/api/node_modules/express/lib/router/layer.js:95:5)
at next (/double-take/api/node_modules/express/lib/router/route.js:144:13)
at /double-take/api/src/middlewares/index.js:67:3
at newFn (/double-take/api/node_modules/express-async-errors/index.js:16:20)
at Layer.handle [as handle_request] (/double-take/api/node_modules/express/lib/router/layer.js:95:5)
at next (/double-take/api/node_modules/express/lib/router/route.js:144:13)
at module.exports.jwt (/double-take/api/src/middlewares/index.js:9:14)
at newFn (/double-take/api/node_modules/express-async-errors/index.js:16:20)

fixed in v<IP_ADDRESS>


Spring cloud Contract - 1.2.6

Hi,
I am using verifier version 1.2.6 to create the stub and consumer tests. I am facing some problems with dynamic properties.
Producer Side
Contract
Contract.make {
description "getAvailablePostals - Contract : 0"
request {
method 'POST'
url '/xyz/v5/getAvailablePostals'
headers { header ('Content-Type', 'application/json;charset=UTF-8') }
body (""" {
"shipDateAndTime" : "2019-02-13T03:33:43.437-0600"
}
} """)
stubMatchers {
jsonPath('$.shipDateTime', byTimestamp())
}
}
response {
status 200
headers { contentType applicationJson() }
body (""" {
""")
}
}
In the request i have used stubMatcher to match the pattern for the property "shipDateAndTime".
Below are the snip of code from stub
{
"id" : "5a266ad5-6a9e-47aa-935e-cf9660d34516",
"request" : {
"url" : "/xyz/v5/getAvailablePostals",
"method" : "POST",
"headers" : {
"Content-Type" : {
"equalTo" : "application/json;charset=UTF-8"
}
},
"bodyPatterns" : [  {
"matchesJsonPath"<EMAIL_ADDRESS>=~ /(([0-9]{4})-(1[0-2]|0[1-9])-(3[01]|0[1-9]|[12][0-9])T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9]))/)]"
} ]
},
"response" : {
"status" : 200,
"body" : "...",
"headers" : {
"Content-Type" : "application/json"
},
"transformers" : [ "response-template" ]
},
"uuid" : "5a266ad5-6a9e-47aa-935e-cf9660d34516"
}
When i try to get the response from stub the stub is failing to validate the shipDateAndTime with the regex. Below is the response i am getting.
Request was not matched
Closest stub
POST
/xyz/v5/getCountryProfile
Content-Type: application/json;charset=UTF-8
<EMAIL_ADDRESS>=~
Request
POST
/xyz/v5/getCountryProfile
Content-Type: application/json;charset=UTF-8
{"shipDateAndTime":"2019-02-13T03:33:43.437-0600"}
From the above response it seems it is not matching the date value in the request with the regex in the stub.
Please suggest on this.

Hello @nagendramithra . Please learn how to properly format code and logs.
Also, we no longer support the 1.2.x branch. Please try upgrading to 2.0.x or 2.1.x and let us know if the issue is still there.

Hello @nagendramithra . Please learn how to properly format code and logs.
Also, we no longer support the 1.2.x branch. Please try upgrading to 2.0.x or 2.1.x and let us know if the issue is still there.

Ok. Thank you.


Getting invalid ELF header issue

Describe your issue
Hi Team
I tried running cython compile time optimization on FastAPI serving layer in python during ML Model inferencing.
Instead of loading plain python module for ML Model , I tried loading cython compile ML Module in FastAPI Server.
I am getting below error while loading ML Model while running on Mac OS Monterey ( Version 12.5)
Getting invalid ELF header issue
Command to used to compile python file
python setup.py build_ext --inplace
Content of setup.py file
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize('v2/model.pyx','v2/model_config.pyx','v2/utils.pyx'),
extra_compile_args=["-mmacosx-version-min=10.9", "-stdlib=libc++"],
extra_link_args=["-mmacosx-version-min=10.9", "-stdlib=libc++"],
)
File which I has trying to load in python code
model.cpython-310-darwin.so
Any help would be appreciated here

Are you mixing arm64 and x86_64 compilers/pythons? Can you do this, which should show which architecture you are using
file `which python`
file model.cpython-310-darwin.so

@mattip I ran above commend and received below output while my laptop is Mac M1.

FYI

I also guessed this could be issue in M1 and hence already tried in another laptop without M1. The architecture there was same but still getting same error there.

I am running this project under docker , base image which I am using in docker file is ubuntu:22.04. This docker image i am running on MAC M1. Whether this can cause any issue ?

Please copy-paste text rather than screen shots.
I don't see the output of the commands I asked for, rather the output of some other shared object, that does not seem to be created by your setup.py.

In your first comment you included contenst of a setup.py and wrote

File which I has trying to load in python code
model.cpython-310-darwin.so

So which is it: model.cpython-310-darwin.so or IMHitEnterQueryFastText.cpython-310-darwin.so?

Ohh @mattip I got your concern. Sorry for miss here.
Post raising this issue , I have change setup.py content to check if this solved issue or not.
New setup.py file content
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize('v2/IMHitEnterQueryFastText.pyx','v2/model_config.pyx','v2/utils.pyx'),
      extra_compile_args=["-mmacosx-version-min=10.9", "-stdlib=libc++"],
      extra_link_args=["-mmacosx-version-min=10.9", "-stdlib=libc++"],
      )

I hope this resolve confusion here.
New shared object file generate from setup.py file is IMHitEnterQueryFastText.cpython-310-darwin.so
`

OK, that makes more sense now.
It seems something is off with the file you are creating, your version of python does not want to open it. So in order to determine what is wrong, the operating system gives you tools to verify that the file you create matches the python you are using. One of those tools is the file command, which (as you saw above) prints the machine type for the file. Now we know the so you are creating is for x86_64, since you showed that a few comments ago. What does the same file command show for the python you are using to run the program? Is it also x86_64 or is it arm64 or hybrid?

@mattip
Command to run -
file /usr/bin/python3.10
python3.10: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=74fffe0a427c04b9b155640e5d840c17cbd0b65f, for GNU/Linux 3.7.0, stripped

Docker file used to install python
FROM [157529275398.dkr.ecr.ap-southeast-1.amazonaws.com/ci-libraries/docker/library/ubuntu:22.04](http://157529275398.dkr.ecr.ap-southeast-1.amazonaws.com/ci-libraries/docker/library/ubuntu:22.04)

RUN apt-get update \
&& apt-get install build-essential -y \
&& apt-get install gcc -y \
&& apt-get clean \
&& apt install -y python3 \
&& apt install -y python3-pip \
&& apt-get install file -y

RUN mkdir /app
RUN mkdir -p /app/resources
WORKDIR /app/

COPY . /app/
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN pip install -r requirements-doc.txt

RUN #python3 -m spacy download en_core_web_md

ENV PYTHONUNBUFFERED 1
EXPOSE 8000
CMD ["bash","run.sh"]

Your python is python3.10: ELF 64-bit LSB pie executable, ARM aarch64 and the compilation is making a shared object  that is  Mach-O 64-bit bundle x86_64. These are incompatible. You need to tell the compiler "please make me a aarch64 shared object" when compiling v2/IMHitEnterQueryFastText.pyx since your python is aarch64. There are a number of ways to do that, I think you should be able to add -arch arm64 to the compilation arguments.


don't override ct.yml in the release workflow

We have a ct.yml that configures the chart-releaser-action. So no need to have the values here.

Fingers crossed!

Hm.. That didn't work I guess..
I can only imaging that the action script checks if there are any changed files under the defined path since the last commit.
That was not the case here, so maybe it'll need an initial change to one of the chart files in order to produce a release?

@SeWieland yes I'm nearly positive that a minor bump to the chart version in Chart.yaml and subsequent merge to the main branch will trigger a successful publish.

The problem is with the chart releaser script, it cannot handle the charts dir being prefixed with ./, see https://github.com/oauth2-proxy/manifests/issues/6#issuecomment-774500210


Fixed bug initializing empty data in combined dictionary format

The DictInterface allows combining two columns into one array, however when this array is empty it currently fails and raises an error. This PR ensures that this works:
Dataset({('x', 'y'): []}, kdims=['x', 'y'])

[x] Adds unit test

Tests are green. Merging.


latest version  could not be accepted by go mod

Expected Behavior
Run go mod vendor
add github.com/DataDog/datadog-lambda-go v0.5.0 to go.mod file
Actual Behavior
github.com/DataDog/datadog-lambda-go v0.2.5 added to go.mod file

latest version "0.5.0" is not semver tag, and it should be "v0.5.0"
Steps to Reproduce the Problem

sam init
select go template
run go mod init and setup package name
add "github.com/DataDog/datadog-lambda-go"
run go mod vendor

Specifications

Go version: 1.13

Hey @ypwu1, thanks for the report. I've gone and correctly tagged release v0.5.0. Can you try again?

Works now !!!


Misaligned table cells

Table cells need to be centre aligned, in order to give better readability, and UI.

Awesome @code-reaper08


Freemoji sends double emojis and messages

Latest versions of everything.

This is usually caused because of interactions with incompatible plugins. The plugin was discontinued now though and I won't provide support for it.


find assets

get or make assets for possible hexagon grids

Nice, looks good. Those are not pointing up as in my imaginations, but I guess that doesn´t matter and this one works better for numbers.

As placeholders, these should be okay for now. I simply took the first ones that were free, and we will probably replace them with our own designed assets afterward anyway


[Software Addition] Prism Launcher

Checklist

[X] I've checked that my software suggestion hasn't already been added to one of the lists.
[X] I've checked the latest issues to make sure that this software hasn't already been suggested.

What is your software's name?
Prism Launcher
What operating systems dose this software works on?
Windows 11, Windows 10, Windows 8/8.1, Windows 7, MacOS, Ubuntu, Debain, Arch Linux, Fedora, Other Linux
Input a link to your suggested software.
https://github.com/PrismLauncher/PrismLauncher, https://prismlauncher.org/
What is your reasoning for including this software?
it is a minecraft launcher, which many people use.
It is also the go-to way of installing minecraft on steam deck.
(OPTIONAL) Give some extra information about the software.
In addition the software list could maybe use some more info on it, I made this list a few months back and there are still missing fields, but maybe some of the information there could be useful to extend your list: https://gist.github.com/TayouVR/ed585459737fc5a0afa2b02eb10ce78b

Would you like to work on our launcher lists directly?

I am not sure i'd feel comfortable operating on this list directly.
If you like I can PR changes for it though, when I get the time.

Feel free to open a pr anytime. any help is greatly appreciated

done in pr #150


Linux containers not cleaned up after VM reboot

Suspected cause: https://github.com/srl-labs/containerlab/blob/2d6d77baf9b7c9b15cfdc93c19f74bbf37679a20/runtime/docker/docker.go#L411
When the host system reboots, most containers are removed but a set of Linux containers remains running in isolation

As you can see, the reason is in the comment section. Some management
containers (like telemetry stack) may fail over time and the automatic
restart beings them back without human intervention.
Rebooting a node with a lab running is considered to be an incident, and a
manual clean up is expected. Luckily it is a single line bash script
On Fri, 7 Oct 2022 at 14:31, J vanBemmel @.***> wrote:

Suspected cause:
https://github.com/srl-labs/containerlab/blob/2d6d77baf9b7c9b15cfdc93c19f74bbf37679a20/runtime/docker/docker.go#L411
When the host system reboots, most containers are removed but a set of
Linux containers remains running in isolation
—
Reply to this email directly, view it on GitHub
https://github.com/srl-labs/containerlab/issues/1055, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABLKV5NLT4DT5PGF5IED6RTWCAJYTANCNFSM6AAAAAAQ7RSRT4
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

I would still like to be able to disable this through configuration. When building automated topologies with several simple linux host containers, I prefer them to be cleaned up like the other non-linux containers, and not stick around
Suggestion:
auto-restart: False (default True)

thanks


Change port Lavalink runs from

How can I configure Lavalink to run from a port other than 8080? I have a Jenkins pipeline that already uses port 8080. I know there is probably a simple solution but I haven't been able to find anything online to help.
Running from Ubuntu 20.04.1 LTS
Error message:
Web server failed to start. Port 8080 was already in use.
Action:
Identify and stop the process that's listening on port 8080 or configure this application to listen on another port.

My application.yml also has port 2333 as the configured port.

Sounds like your config file is not actually being picked up. Somewhat counter-intuitively the default is 8080. Make sure it is in the working directory.
You can also use the SERVER_PORT environment variable.
Closing this as I do not like help requests in the issue tracker.


Add Range operation shape inference and lowering to Krnl

Add Range operation shape inference and lowering to Krnl.

Thanks for adding the backend test

Jenkins Linux amd64 Build #1210 [push] Add Range operation shap... started at 15:32

Jenkins Linux s390x Build #1208 [push] Add Range operation shap... started at 16:32

Jenkins Linux ppc64le Build #356 [push] Add Range operation shap... started at 16:34

Jenkins Linux s390x Build #1208 [push] Add Range operation shap... passed after 41 min

Jenkins Linux amd64 Build #1210 [push] Add Range operation shap... passed after 45 min

Jenkins Linux ppc64le Build #356 [push] Add Range operation shap... passed after 53 min


[BUG] JPG fails to open

axe.zip

I can repro

The file itself definitely has problems -- other utilities also complain about it. But I think we might be treating it as a fatal error when it doesn't need to be.

Proposed fix in #2927. My assessment above was correct -- the header is indeed weird/corrupted, but not fatally so, and OIIO was treating message callbacks from jpeglib as fatal errors, when really they were more appropriate as warnings or just let it slide.
I will backport the fix to the supported release branch, but not in time for this week's release. It will have to wait for the next release.


This is an advice!

It would be really nice if it were possible to create a shortcut to open a certain application and then add the shortcut to the home screen so that you can immediately access that application

Already implemented. At app list, hold down an app and tap Add to Home Screen.


Fix circular dependency with components

This fixes the circular dependency by having two versions of some components, one with all possible sub-components and one 'compact' with just the text component.
Also, the full version is lazy loaded so that it's initialised only when needed.

Coverage increased (+0.1%) to 94.483% when pulling 46ce9775d64612d321edea596ba301a4c9f35956 on circular-dep into 14e64e896fe579651799a1de600ee66b9d4c57ad on master.

Coverage increased (+0.1%) to 94.483% when pulling ad856dd99675cd20cbc3c522bccde6fe4b49a4d5 on circular-dep into 14e64e896fe579651799a1de600ee66b9d4c57ad on master.


Check memory leak

joewandy [1:00 PM]
Got the feeling that there’s a memory leak somewhere on the server
joewandy [1:00 PM]
uploaded this image: Screen Shot 2017-04-15 at 13.00.07.png
Add Comment
joewandy [1:01 PM]
nothing’s running but high memory usage ..?
simon [1:01 PM]
I think that's Justin's mega experiment...
joewandy [1:01 PM]
but cpu is 0
simon [1:33 PM]
it was loading (and according to the log it still is)
[1:33]
but I think it’s dead
joewandy [1:34 PM]
this happened too before ..?
simon [1:34 PM]
celery rebooted before
[1:34]
not sure on the memory thing though
joewandy [1:34 PM]
hmmm
simon [1:34 PM]
when you start celery it says something about memory leaks
[1:34]
also debug mode in django def has them
[1:35]
see: [2017-04-15 12:35:12,298: WARNING/MainProcess] /Users/simon/.virtualenvs/ms2ldaviz/lib/python2.7/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn(‘Using settings.DEBUG leads to a memory leak, never ’
joewandy [1:35 PM]
SECURITY WARNING: don’t run with debug turned on in production!
DEBUG = False
[1:35]
in settings.py
simon [1:36 PM]
is that a celery thing or a django thing?
joewandy [1:37 PM]
for both i think
simon [1:38 PM]
just seen this:
[1:38]
@ask I have tried running the stress tests and even put load on the memory by creating big objects. No leak.
Is there a difference between running python manage.py celery worker and celery -A ....?
[1:38]
from :
[1:38]
https://github.com/celery/celery/issues/2927
GitHub
Celery prefork memory leak · Issue #2927 · celery/celery
We are running Celery with Django using the following command: python manage.py celery worker -Q some_queue -l INFO -n some_worker —concurrency=1 The flow is as such - (assume CELERYD_MAX_TASKS_PER...
joewandy [1:42 PM]
Thanks! Looks complex though ..
simon [1:42 PM]
and this:
[1:42]
use this param CELERYD_MAX_TASKS_PER_CHILD = 100, a worker will be killed after complete 100 tasks and a new worker will be created。
[1:43]
from:
https://github.com/celery/celery/issues/3436
GitHub
Memory leak with redis broker · Issue #3436 · celery/celery
celery leaks when redis becomes unavailable. This is easily reproducible by stopping redis while celery is working. from celery import Celery from celery.task import periodic_task from datetime imp...
[1:43]
might be an easy thing to try. Memory links in celery seem to be common.
joewandy [1:43 PM]
alright, will try
simon [1:44 PM]
two options here (including the one above):
[1:44]
http://docs.celeryproject.org/en/latest/userguide/workers.html#max-tasks-per-child-setting
[1:44]
search ‘memory leak’
[1:44]
feels a bit hacky to force the proces to die once it’s done (say) 20 jobs
[1:45]
but it will probably work

Added the parameter. Let's see how it goes.
If not, it's time to upgrade the memory of the server.


feat: pluggable consortium management consensus

Description
Currently we only have the manual consortium plugin that needs to be hand configured in advance.
Should create more plugins that allow for things like:

Achieving consortium membership consensus based on certain types of majorities (simple, super, etc.)
Arbitrary smart contract implementations that can implement any type of governance normally DAOs do

Acceptance Criteria

TBD
...
...

Note to self: this needs some more designing, specifically regarding the scope of the storage we would provide. Have to be clear whether it's a consortium level storage or just consortium member level. This can have implications in latency for example instead of everything running in the same data center it could end up spanning multiple continents within multiple data centers.
Also have to be careful not to bleed into actually running our own blockchain so triage is definitely in order on this one.

Completed in the scope of another issue where the consortium static plugin got added to the mix of options.


rocketmq 2.2.0

Which Component
spring-cloud-stream-binder-rocketmq      2.2.0.RELEASE
Describe what problem you have encountered
The application is deployed in a docker environment with the same instance name, causing multiple instances of a service to repeatedly consume messages
Describe what information you have read
I see com.alibaba.cloud.stream.binder.rocketmq.consuming.RocketMQListenerBindingContainer
consumer.setInstanceName(RocketMQUtil.getInstanceName(rpcHook,
topic + "|" + UtilAll.getPid()));
Our application is deployed in the docker environment, the pid is the same

so，when can this problem be solved? We are waiting for the release of 2.2.2 @lvITStudy

you can use the beta version #2029


Memcpy/memcmp macros

Define preprocessor macros zmemcpy/zmemcmp that force unaligned access on supported architectures
Use zmemcpy/zmemcmp in the code where necessary.

This does look very promising, I wish we could do some broader benchmarking of this though, like aarch32 for example.

Yeah I agree. I don't have any access to ARM machines. 😟

Rebased.

Oracle cloud has free amprere arm VMs

I've got an odroid n2 for aarch64 and a quad powermac g5 for ppc64

Rebased.

I plan to split some of this PR out into a PR just for moving unaligned detection into code.

Rebased.

I have moved the unaligned build changes into another PR so I have removed it from this one.

RPI 1B arm6l GCC 4.9
Baseline GCC4.9 88f331e6a64653b533721a645f8253b78e7f391a
 Tool: minigzip   Levels: 0-9
 Runs: 10         Trim worst: 6

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.180/0.188/0.190/0.005        0.220/0.232/0.240/0.010       15,737,543
 1     54.185%      3.610/3.625/3.660/0.024        0.840/0.867/0.890/0.021        8,526,745
 2     43.874%      5.070/5.112/5.160/0.040        0.850/0.885/0.910/0.026        6,904,168
 3     42.394%      6.340/6.405/6.440/0.045        0.780/0.820/0.860/0.033        6,671,197
 4     41.693%      7.200/7.253/7.340/0.061        0.800/0.823/0.840/0.017        6,560,888
 5     41.216%      7.770/7.785/7.790/0.010        0.800/0.818/0.830/0.013        6,485,936
 6     41.038%      8.870/8.925/9.020/0.067        0.790/0.807/0.820/0.015        6,457,827
 7     40.778%   11.090/11.213/11.340/0.107        0.810/0.820/0.830/0.008        6,416,941
 8     40.704%   13.340/13.497/13.600/0.111        0.740/0.780/0.800/0.028        6,405,249
 9     40.409%   12.260/12.502/12.660/0.176        0.820/0.850/0.870/0.022        6,358,951

 avg1  48.630%                        7.650                          0.770
 avg2  54.033%                        8.501                          0.856
 tot                                306.020                         30.810       76,525,445

   text    data     bss     dec     hex filename
 105272     900      16  106188   19ecc libz-ng.so.2

PR #1109 GCC4.9 727254d2e7184945e2b5477c2174cd624b753e13
 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.180/0.188/0.190/0.005        0.190/0.230/0.250/0.028       15,737,543
 1     54.185%      3.750/3.790/3.850/0.042        0.870/0.897/0.920/0.026        8,526,745
 2     43.874%      5.100/5.130/5.170/0.036        0.790/0.873/0.910/0.056        6,904,168
 3     42.394%      6.390/6.405/6.440/0.024        0.850/0.873/0.890/0.017        6,671,197
 4     41.693%      7.180/7.265/7.340/0.066        0.810/0.833/0.850/0.017        6,560,888
 5     41.216%      7.660/7.730/7.770/0.050        0.810/0.823/0.850/0.019        6,485,936
 6     41.038%      8.800/8.880/8.950/0.065        0.830/0.843/0.850/0.010        6,457,827
 7     40.778%   11.220/11.425/11.560/0.145        0.760/0.782/0.800/0.021        6,416,941
 8     40.704%   13.240/13.303/13.370/0.054        0.780/0.800/0.810/0.014        6,405,249
 9     40.409%   12.520/12.678/12.960/0.201        0.830/0.847/0.860/0.013        6,358,951

 avg1  48.630%                        7.679                          0.780
 avg2  54.033%                        8.532                          0.867
 tot                                307.170                         31.200       76,525,445

   text    data     bss     dec     hex filename
 105189     904      16  106109   19e7d libz-ng.so.2

RPI 1B, arm6l, GCC 8.3
Baseline GCC8.3 88f331e6a64653b533721a645f8253b78e7f391a
 Tool: minigzip   Levels: 0-9
 Runs: 10         Trim worst: 6

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.180/0.195/0.220/0.017        0.210/0.215/0.230/0.010       15,737,543
 1     54.185%      3.480/3.507/3.520/0.019        0.830/0.835/0.840/0.006        8,526,745
 2     43.874%      4.990/5.110/5.160/0.081        0.750/0.787/0.820/0.029        6,904,168
 3     42.394%      6.240/6.290/6.320/0.038        0.770/0.782/0.790/0.010        6,671,197
 4     41.693%      7.200/7.240/7.300/0.045        0.770/0.782/0.790/0.010        6,560,888
 5     41.216%      7.680/7.768/7.850/0.071        0.760/0.772/0.790/0.013        6,485,936
 6     41.038%      8.890/9.062/9.180/0.123        0.750/0.770/0.790/0.023        6,457,827
 7     40.778%   11.000/11.072/11.160/0.066        0.730/0.755/0.770/0.019        6,416,941
 8     40.704%   13.400/13.620/13.840/0.232        0.690/0.718/0.750/0.028        6,405,249
 9     40.409%   12.660/12.785/13.020/0.167        0.760/0.795/0.820/0.030        6,358,951

 avg1  48.630%                        7.665                          0.721
 avg2  54.033%                        8.517                          0.801
 tot                                306.600                         28.850       76,525,445

   text    data     bss     dec     hex filename
 104778     856      16  105650   19cb2 libz-ng.so.2

PR #1109 GCC8 727254d2e7184945e2b5477c2174cd624b753e13
 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.180/0.185/0.190/0.006        0.210/0.232/0.260/0.022       15,737,543
 1     54.185%      3.530/3.577/3.610/0.034        0.770/0.790/0.820/0.024        8,526,745
 2     43.874%      5.010/5.040/5.080/0.032        0.820/0.835/0.850/0.013        6,904,168
 3     42.394%      6.250/6.303/6.340/0.038        0.790/0.807/0.820/0.015        6,671,197
 4     41.693%      7.030/7.130/7.240/0.088        0.770/0.775/0.780/0.006        6,560,888
 5     41.216%      7.570/7.713/7.820/0.115        0.740/0.760/0.790/0.022        6,485,936
 6     41.038%      8.850/9.037/9.140/0.137        0.740/0.760/0.770/0.014        6,457,827
 7     40.778%   11.100/11.162/11.250/0.063        0.730/0.740/0.750/0.008        6,416,941
 8     40.704%   13.340/13.470/13.560/0.101        0.710/0.725/0.750/0.019        6,405,249
 9     40.409%   12.510/12.690/12.910/0.188        0.760/0.772/0.790/0.013        6,358,951

 avg1  48.630%                        7.631                          0.720
 avg2  54.033%                        8.479                          0.800
 tot                                305.230                         28.790       76,525,445

   text    data     bss     dec     hex filename
 104842     856      16  105714   19cf2 libz-ng.so.2

RPI 3, aarch64, GCC 10
Baseline GCC10 88f331e6a64653b533721a645f8253b78e7f391a
 Tool: minigzip   Levels: 0-9
 Runs: 32         Trim worst: 18

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.008/0.015/0.020/0.003        0.016/0.034/0.044/0.008       15,737,543
 1     54.165%      0.535/0.548/0.555/0.007        0.188/0.201/0.208/0.006        8,523,602
 2     43.877%      0.904/0.921/0.936/0.010        0.184/0.199/0.210/0.008        6,904,649
 3     42.389%      1.202/1.225/1.241/0.011        0.166/0.194/0.205/0.011        6,670,476
 4     41.647%      1.358/1.372/1.380/0.007        0.168/0.187/0.197/0.009        6,553,763
 5     41.217%      1.474/1.498/1.517/0.016        0.181/0.190/0.199/0.007        6,485,981
 6     41.037%      1.740/1.757/1.765/0.008        0.172/0.187/0.195/0.008        6,457,780
 7     40.778%      2.405/2.417/2.427/0.007        0.167/0.176/0.187/0.007        6,416,924
 8     40.704%      2.938/2.949/2.955/0.005        0.168/0.181/0.194/0.007        6,405,244
 9     40.409%      3.302/3.328/3.341/0.011        0.178/0.193/0.200/0.007        6,358,951

 avg1  48.623%                        1.603                          0.174
 avg2  54.026%                        1.781                          0.194
 tot                                224.415                         24.390       76,514,913

   text    data     bss     dec     hex filename
 116566    1664      16  118246   1cde6 libz-ng.so.2

PR #1109 GCC10 727254d2e7184945e2b5477c2174cd624b753e13
 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.008/0.015/0.020/0.003        0.016/0.032/0.040/0.008       15,737,543
 1     54.165%      0.494/0.524/0.543/0.015        0.185/0.199/0.208/0.007        8,523,602
 2     43.877%      0.897/0.913/0.924/0.008        0.192/0.204/0.218/0.009        6,904,649
 3     42.389%      1.195/1.209/1.221/0.007        0.174/0.194/0.201/0.007        6,670,476
 4     41.647%      1.342/1.360/1.371/0.010        0.182/0.191/0.200/0.006        6,553,763
 5     41.217%      1.477/1.492/1.504/0.008        0.174/0.189/0.198/0.009        6,485,981
 6     41.037%      1.735/1.748/1.756/0.007        0.167/0.179/0.189/0.007        6,457,780
 7     40.778%      2.379/2.407/2.416/0.010        0.157/0.178/0.188/0.009        6,416,924
 8     40.704%      2.925/2.936/2.943/0.006        0.164/0.172/0.180/0.005        6,405,244
 9     40.409%      3.297/3.321/3.333/0.010        0.174/0.184/0.194/0.007        6,358,951

 avg1  48.623%                        1.593                          0.172
 avg2  54.026%                        1.769                          0.191
 tot                                222.951                         24.117       76,514,913

   text    data     bss     dec     hex filename
 116198    1664      16  117878   1cc76 libz-ng.so.2

It seems like this has a slight performance regression for GCC 4.9 on arm6l, but also a slight performance increase on the same platform with GCC 8.3.
Also a slight performance increase on aarch64 with GCC 10.
Compiled code size is smaller for all ARM tests, that is a nice bonus.
So, this seems like a win on ARM at least.
Off topic observation: I think it is kind of interesting how level 9 is consistently quite a bit faster than level 8 for arm6l, the same is not true on aarch64..

Well, arm6 specifically lacks NEON and as such isn't calling the NEON implementation of slide_hash or chunkset.  Perhaps there's something happening at level 9 with the canonical examples that is less friendly to the scalar implementations at 8 than it is at 9?

Rebased. Benchmark numbers look good. I'm not sure why the slight increase/decrease in performance. It is very close to functionally equivalent to what we already have. I'm also not doing any checks based on compiler version or anything like that.

One other difference between level 8 and level 9 is the fast-zlib code.

I want to do a few checks on x86 as well, just to be sure we don't have any obvious regressions.
Other than that, is this PR considered ready to go? I don't see any objections at least.
I am aware of the fast-zlib code for level 9, it is just interesting how that ends up faster on arm6l. That might mean there is a potential for optimization of the other levels, but I have not looked at it noticing that unexpected speedup (We never benchmarked that change on 32-bit arm, so this is the first time I saw that).
My ARM test setup is pretty good and reliable now. Now I just need to spend some time to regain a good x86 test setup as well.

Yeah this PR is ready to go. I tried to keep the changes relatively straight forward. If we need to make any config changes I broke those out into another PR to be discussed separately.

Benchmarks
MSVC 17.0.5 x86-64
DEVELOP https://github.com/zlib-ng/zlib-ng/commit/748127e51cfd368923f688cfa74ae784d2f1998a
OS: Windows 10 10.0.22000 AMD64
CPU: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
Tool: ../zlib-ng/build-develop/Release/minigzip.exe
Levels: 1-9
Runs: 70         Trim worst: 40

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 1     54.166%      0.094/0.096/0.098/0.001        0.047/0.048/0.049/0.001        8,523,732
 2     43.871%      0.158/0.161/0.164/0.002        0.049/0.050/0.051/0.001        6,903,609
 3     42.387%      0.216/0.219/0.221/0.001        0.047/0.048/0.050/0.001        6,670,099
 4     41.647%      0.238/0.242/0.245/0.002        0.046/0.047/0.048/0.001        6,553,723
 5     41.216%      0.240/0.243/0.246/0.002        0.045/0.047/0.048/0.001        6,485,938
 6     41.037%      0.276/0.280/0.287/0.003        0.045/0.047/0.048/0.001        6,457,776
 7     40.778%      0.346/0.354/0.360/0.003        0.046/0.047/0.048/0.001        6,416,919
 8     40.704%      0.438/0.445/0.451/0.004        0.112/0.118/0.122/0.003        6,405,244
 9     40.409%      0.521/0.537/0.543/0.005        0.047/0.048/0.049/0.001        6,358,951

 avg1  42.913%                        0.286                          0.056
 tot                                 77.298                         15.015       60,775,991

PR 1109 https://github.com/zlib-ng/zlib-ng/pull/1109/commits/f5d680fe912ef06f4eb8f039ccc520a8f31f2581
OS: Windows 10 10.0.22000 AMD64
CPU: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
Tool: ../zlib-ng/build-1109/Release/minigzip.exe
Levels: 1-9
Runs: 70         Trim worst: 40

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 1     54.166%      0.093/0.095/0.097/0.001        0.047/0.048/0.049/0.001        8,523,732
 2     43.871%      0.157/0.159/0.161/0.001        0.049/0.050/0.051/0.001        6,903,609
 3     42.387%      0.217/0.219/0.221/0.001        0.047/0.048/0.049/0.001        6,670,099
 4     41.647%      0.236/0.240/0.242/0.001        0.046/0.047/0.047/0.001        6,553,723
 5     41.216%      0.239/0.243/0.245/0.002        0.044/0.046/0.047/0.001        6,485,938
 6     41.037%      0.274/0.279/0.284/0.002        0.045/0.046/0.047/0.001        6,457,776
 7     40.778%      0.346/0.354/0.360/0.003        0.045/0.047/0.048/0.001        6,416,919
 8     40.704%      0.437/0.442/0.447/0.003        0.111/0.117/0.120/0.003        6,405,244
 9     40.409%      0.526/0.534/0.539/0.004        0.046/0.048/0.049/0.001        6,358,951

 avg1  42.913%                        0.285                          0.055
 tot                                 76.994                         14.910       60,775,991

@nmoinvaz Whatever happened to decompression on level 8 in your benchmarks?
Those all use the same code, so how did level 8 data make it take more than twice the time compared to any other level? It might be worth profiling that, since it seems it manages to reliably hit some kind of a corner-case.

Level 8 looks almost identical..

Here is another set that I ran last night on a different data set.
Benchmarks
MSVC 17.0.5 x86-64 silesta.tar
DEVELOP https://github.com/zlib-ng/zlib-ng/commit/748127e51cfd368923f688cfa74ae784d2f1998a
OS: Windows 10 10.0.22000 AMD64
CPU: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
Tool: ../zlib-ng/build-develop/Release/minigzip.exe
Levels: 1-9
Runs: 70         Trim worst: 40

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 1     44.409%      0.960/0.972/0.979/0.005        0.489/0.499/0.504/0.003       94,127,290
 2     35.518%      1.639/1.657/1.664/0.007        0.511/0.519/0.524/0.004       75,282,961
 3     33.882%      2.298/2.332/2.346/0.013        0.493/0.499/0.503/0.003       71,816,478
 4     33.174%      2.559/2.585/2.602/0.013        0.482/0.487/0.493/0.003       70,315,668
 5     32.660%      2.610/2.633/2.647/0.010        0.474/0.483/0.488/0.004       69,225,542
 6     32.508%      3.035/3.059/3.074/0.010        0.470/0.478/0.482/0.003       68,902,222
 7     32.255%      4.075/4.119/4.149/0.020        0.471/0.478/0.480/0.002       68,366,800
 8     32.167%      6.326/6.390/6.422/0.029        0.469/0.477/0.481/0.003       68,180,776
 9     31.887%      6.910/6.955/6.981/0.019        0.466/0.473/0.476/0.002       67,586,442

 avg1  34.273%                        3.411                          0.488
 tot                                921.037                        131.820      653,804,179

PR 1109 https://github.com/zlib-ng/zlib-ng/pull/1109/commits/f5d680fe912ef06f4eb8f039ccc520a8f31f2581
OS: Windows 10 10.0.22000 AMD64
CPU: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
Tool: ../zlib-ng/build-1109/Release/minigzip.exe
Levels: 1-9
Runs: 70         Trim worst: 40

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 1     44.409%      0.960/0.971/0.977/0.005        0.486/0.497/0.504/0.005       94,127,290
 2     35.518%      1.637/1.659/1.668/0.008        0.510/0.518/0.523/0.004       75,282,961
 3     33.882%      2.315/2.339/2.354/0.010        0.492/0.499/0.502/0.003       71,816,478
 4     33.174%      2.568/2.590/2.604/0.011        0.478/0.485/0.489/0.003       70,315,668
 5     32.660%      2.614/2.634/2.644/0.009        0.475/0.482/0.486/0.003       69,225,542
 6     32.508%      3.039/3.064/3.084/0.014        0.467/0.476/0.480/0.003       68,902,222
 7     32.255%      4.091/4.124/4.151/0.017        0.471/0.477/0.481/0.003       68,366,800
 8     32.167%      6.315/6.387/6.433/0.035        0.467/0.477/0.480/0.003       68,180,776
 9     31.887%      6.885/6.943/6.977/0.025        0.463/0.470/0.474/0.003       67,586,442

 avg1  34.273%                        3.412                          0.487
 tot                                921.352                        131.426      653,804,179

@nmoinvaz I don't mean this PR changed anything. I just meant that in both your tests, level 8 is very consistently more than twice as slow at decompression than ANY other level. And your tests with another dataset confirms my suspicions that the issue is caused by that specific dataset somehow.
Taking 2.36X the time, reproducibly over at least 140 runs, with very low stddev... That has got to be some kind of bad corner case..

I see what you are saying now. It is very odd. I wonder if it is a compiler issue with that silesia-small.tar data.

Looks like it still happens with -DWITH_OPTIM=OFF.

@Dead2 it looks like it is caused by this commit: 0129e88cee5f5ac295c311d5d94f2bb6a2f475f6.

We knew limiting minimum match length to 4 instead of 3 would hurt compression in some cases...

It hurts decompression..

The compress stream changes and somehow decompression is hurt. Maybe it is having to seek far back into the window all the time?
I did a comparison of minigzip output bwith z_verbose between level 7 and 8 and their didn't seem to be that much of a difference imo.

I would check distances of 3 and 4 byte matches... It's possible that the chain lenghts are long enough to hurt performance.

Well my CPU may need to be bringing memory in and out of cache to perform these operations if they are too far back is my thinking. If somebody else can run the test using silesia-small.tar it might give a better idea.

I think I would do a profiling comparison, and see where the huge amount of extra time is spent. Not sure whether it will actually tell us what is "wrong" about the data, but hopefully it would give a hint.
My thinking is that the commit that you found did not cause this, it just modified the compressed data in a way that made the issue crop up in this specific circumstance. The compressed data is valid after all, and it would surprise me if it is very different from level 7, making the increased time spent even more surprising.
Interestingly, I do not see the same problem with the same data. But we do have several cases where we compress the same data differently depending on what arch-specific optimizations are enabled etc. Not sure how that still happens with WITH_OPTIM=OFF, but we have not really kept track of what affects the compressed output.
You could upload the compressed level 7 and level 8 files, then we could all test decompression to see whether we can at least replicate the slowdown with the same compressed data as you.

Last night I tried to do some performance profiling in Visual Studio 2022 but nothing stood out at me.
Here is my data set. I have included the original .tar and also the compressed version of the same at level 7 and 8.
silesia-small-data.zip

Nothing terribly conclusive there - just that 4% less of the total runtime is burned in zng_inflate_fast on level 8 than level 7 (inclusive, not just self).  There are also seems to be ~5% less time in the C runtime though it's hard to say if that's a good apples to apples comparison here (there's going to be some run to run variance and it's hard to get great resolution).  Do you see this same behavior with the larger version of the test data?

Here is benchmark results on my Tiger Lake laptop:
MSVC 17.0.5 x86-64 silesta-small.tar
DEVELOP https://github.com/zlib-ng/zlib-ng/commit/748127e51cfd368923f688cfa74ae784d2f1998a
 Processor: Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
 Tool: ../zlib-ng/build-x/Release/minigzip.exe Levels: 0-9
 Runs: 70         Trim worst: 40

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.016/0.017/0.017/0.000        0.085/0.104/0.114/0.008       15,737,543
 1     54.166%      0.089/0.092/0.097/0.002        0.044/0.045/0.046/0.001        8,523,732
 2     43.871%      0.143/0.151/0.170/0.007        0.046/0.047/0.048/0.001        6,903,609
 3     42.387%      0.200/0.206/0.215/0.005        0.045/0.046/0.047/0.001        6,670,099
 4     41.647%      0.224/0.234/0.265/0.011        0.043/0.045/0.045/0.000        6,553,723
 5     41.216%      0.229/0.235/0.247/0.005        0.043/0.045/0.045/0.001        6,485,938
 6     41.037%      0.267/0.276/0.289/0.006        0.043/0.044/0.045/0.000        6,457,776
 7     40.778%      0.337/0.345/0.367/0.008        0.043/0.044/0.045/0.001        6,416,919
 8     40.704%      0.430/0.444/0.465/0.010        0.105/0.121/0.128/0.005        6,405,244
 9     40.409%      0.505/0.518/0.535/0.009        0.042/0.044/0.044/0.001        6,358,951

 avg1  48.622%                        0.252                          0.059
 avg2  54.025%                        0.280                          0.065
 tot                                 75.573                         17.558       76,513,534

Oddly enough on level 0, the decompression time is also higher.

Even stranger, gz file is twice as large as the original on level 0.

Nevermind, the file size thing appears to be something with PowerShell redirection possibly. When I run minigzip.exe in cmd prompt it outputs the proper size.

As far as I can go back in the git tree, it seems that inflate stored takes that same amount of time.

Oddly enough on level 0, the decompression time is also higher.
So that's not too surprising.  At level 0 you're likely incurring much more IO if you're actually decompressing from block storage.  You're running nearly double the amount of data through memory and back out to storage.

That is what I have always seen too, that the time spent to inflate a file increases roughly linearly with its compressed size.
I have not gotten around to testing the file you uploaded yet.

Xeon E5-2650, x86-64, GCC 4.8
Baseline GCC4.8 88f331e6a64653b533721a645f8253b78e7f391a
 Tool: minigzip   Levels: 0-9
 Runs: 32         Trim worst: 18

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.014/0.015/0.017/0.001        0.013/0.019/0.023/0.003       15,737,543
 1     54.166%      0.212/0.227/0.237/0.007        0.100/0.113/0.117/0.005        8,523,732
 2     43.871%      0.434/0.445/0.450/0.005        0.106/0.118/0.122/0.005        6,903,609
 3     42.387%      0.544/0.552/0.557/0.003        0.112/0.118/0.119/0.002        6,670,099
 4     41.647%      0.596/0.607/0.614/0.006        0.099/0.109/0.115/0.005        6,553,723
 5     41.216%      0.660/0.667/0.672/0.004        0.104/0.110/0.114/0.003        6,485,938
 6     41.037%      0.745/0.749/0.756/0.003        0.099/0.105/0.112/0.005        6,457,776
 7     40.778%      0.951/0.959/0.966/0.005        0.098/0.108/0.111/0.004        6,416,919
 8     40.704%      1.167/1.174/1.177/0.003        0.087/0.105/0.114/0.007        6,405,244
 9     40.409%      1.432/1.437/1.442/0.003        0.105/0.114/0.118/0.004        6,358,951

 avg1  48.622%                        0.683                          0.102
 avg2  54.025%                        0.759                          0.113
 tot                                 95.652                         14.269       76,513,534

   text    data     bss     dec     hex filename
 130303    1456      48  131807   202df libz-ng.so.2

PR #1109 GCC4.8 727254d2e7184945e2b5477c2174cd624b753e13
 Tool: minigzip   Levels: 0-9
 Runs: 32         Trim worst: 18

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.009/0.014/0.017/0.003        0.016/0.020/0.023/0.002       15,737,543
 1     54.166%      0.216/0.227/0.234/0.006        0.100/0.109/0.117/0.006        8,523,732
 2     43.871%      0.437/0.444/0.449/0.003        0.109/0.120/0.125/0.004        6,903,609
 3     42.387%      0.538/0.553/0.557/0.005        0.109/0.118/0.122/0.004        6,670,099
 4     41.647%      0.593/0.605/0.611/0.005        0.101/0.111/0.118/0.004        6,553,723
 5     41.216%      0.662/0.672/0.677/0.004        0.100/0.111/0.116/0.004        6,485,938
 6     41.037%      0.749/0.764/0.771/0.007        0.089/0.106/0.113/0.006        6,457,776
 7     40.778%      0.926/0.937/0.944/0.004        0.091/0.103/0.111/0.006        6,416,919
 8     40.704%      1.142/1.150/1.156/0.005        0.098/0.107/0.114/0.004        6,405,244
 9     40.409%      1.405/1.416/1.422/0.004        0.101/0.112/0.118/0.005        6,358,951

 avg1  48.622%                        0.678                          0.102
 avg2  54.025%                        0.754                          0.113
 tot                                 94.948                         14.235       76,513,534

   text    data     bss     dec     hex filename
 130495    1456      48  131999   2039f libz-ng.so.2

Slightly faster on older compiler for x86-64, that is very good to see.
Increases compiled code size slightly.
That alleviates my worries about this introducing any potential regressions.

@Dead2 are you sure it's not time lost in IO for @nmoinvaz at level 0?  It'd be helpful if these were fed from a ram disk. At least that point you're only measuring the additional memory access time (which still would likely be slower but not by the same scales at least).
Is there a way to measure system time of a process in windows to narrow it down? If it's all in system time that would 100% agree with that theory.

Here is a run on WSL2 using GCC 9.3.0.
OS: Linux <IP_ADDRESS>-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64
CPU: x86_64
Tool: /mnt/c/users/nathan/source/zlib-ng/build-wsl/minigzip
Levels: 0-9
Runs: 70         Trim worst: 40

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.171/0.195/0.215/0.013        0.245/0.273/0.288/0.011       15,737,543
 1     54.166%      0.260/0.331/0.369/0.029        0.182/0.225/0.254/0.021        8,523,732
 2     43.871%      0.334/0.395/0.430/0.025        0.196/0.246/0.270/0.022        6,903,609
 3     42.387%      0.389/0.446/0.481/0.025        0.188/0.223/0.243/0.015        6,670,099
 4     41.647%      0.416/0.487/0.517/0.028        0.189/0.221/0.242/0.017        6,553,723
 5     41.216%      0.470/0.531/0.563/0.027        0.177/0.215/0.235/0.018        6,485,938
 6     41.037%      0.514/0.580/0.613/0.030        0.181/0.213/0.229/0.014        6,457,776
 7     40.778%      0.603/0.661/0.703/0.028        0.177/0.209/0.229/0.015        6,416,919
 8     40.704%      0.736/0.782/0.805/0.020        0.254/0.297/0.318/0.018        6,405,244
 9     40.409%      0.822/0.881/0.902/0.020        0.185/0.219/0.238/0.016        6,358,951

 avg1  48.622%                        0.529                          0.234
 avg2  54.025%                        0.588                          0.260
 tot                                158.660                         70.243       76,513,534

@KungFuJesus I thought maybe if I used WSL2 I could have access to all the perf tools but they were all missing or needed to be compiled manually. So I couldn't get access to the sys timings.

@KungFuJesus I thought maybe if I used WSL2 I could have access to all the perf tools but they were all missing or needed to be compiled manually. So I couldn't get access to the sys timings.

Perf is not a strictly userspace component either, though WSL2 does provide a kernel so perhaps that doesn't matter anymore.  You should be able to do a direct comparison with time -v (should give you memory utilization, system time, etc).

@KungFuJesus For sure, it could absolutely be explained by slow IO.
But I don't see how the OS manages to somehow recognize and only do slow IO for compressed files compressed by level 0 or 8.
AFAIK, there is no way the OS should be aware of what compression level was used for those files. Also, the file is compressed before each decompression test, so it is not even the same file that was decompressed 70 times, it is 70 different (and probably cached) files that should not look any different to the OS (compared to level 7 for example) than the quite minor differences in size.
The OS could be slower when doing IO because of those exact file sizes, but that would be a quite weird bug.
Also, remember the fact that using a different files for testing does not result in this phenomenon..

@KungFuJesus For sure, it could absolutely be explained by slow IO. But I don't see how the OS manages to somehow recognize and only do slow IO for compressed files compressed by level 0 or 8.
AFAIK, there is no way the OS should be aware of what compression level was used for those files. Also, the file is compressed before each decompression test, so it is not even the same file that was decompressed 70 times, it is 70 different (and probably cached) files that should not look any different to the OS (compared to level 7 for example) than the quite minor differences in size.
The OS could be slower when doing IO because of those exact file sizes, but that would be a quite weird bug.
Also, remember the fact that using a different files for testing does not result in this phenomenon..

Well given that compression is kind of a state machine with opcodes, it's not super surprising that differently compressible files would react differently here.    Perf can be used to measure LLC misses, so it'd be interesting to see from the xeon and from the tiger lake -U CPU what those were for those given compression levels.

@KungFuJesus Level 7 and Level 8 should not differ so much that it would affect speed much. And certainly level 0 should not be very challenging at all, since it does not in fact compress the data at all.
I don't think the LLC is the problem that causes the 11KB smaller level 8 file to be 2.5x slower to decompress than the level 7 file.. If anything, it should be the other way around.
The compressed file is also only about 6.4MiB, so that should leave a lot of room even in the 15MiB tiger lake cache.
I have a feeling this is OS or timing related, but it is pretty hard to do any kind of deep inspection like this on windows.
Intels Vtune Profiler could do it though https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html
It can (used to at least) be a bit of a hassle to set up, but that whole toolkit allows very powerful analysis both on Linux and Windows (and Android possibly).
Remember that even the Raspberry Pi 1B shows a much faster level 0 than level 1, and level 8 is just about as expected. And that thing does not have a lot of cache at all.

I used SysInternal's ProcessMonitor to watch the file read/writes from minigzip.exe as deflatebench ran. It seems on level 0 and level 8 a Win32 function called QueryEAFile is run when the archive is first opened. It is probably related to Windows Defender although I can't confirm because nothing really pulls up online for that function. We have seen in our own applications at work that Windows Defender can cause delays when closing files by several seconds as the system scanner checks the file before allowing it to be committed to disk. The doubling of the decompression time makes sense if the Windows Defender scanner also opens and decompresses the achieve for scanning.
Here are rough (not all my apps shutdown or videos stopped playing) benchmarks with Windows Defender on and without. For those who don't know, Windows Defender is anti-virus/anti-malware that comes with Windows - it's not something I installed.
Real-time protection enabled
OS: Windows 10 10.0.22000 AMD64
CPU: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
Tool: ../zlib-ng/build/Release/minigzip.exe
Levels: 0-9
Runs: 70         Trim worst: 40

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.015%      0.044/0.051/0.056/0.003        0.131/0.152/0.160/0.008       15,738,743
 1     54.166%      0.135/0.142/0.149/0.004        0.068/0.077/0.082/0.003        8,523,732
 2     43.871%      0.210/0.223/0.231/0.007        0.076/0.082/0.087/0.004        6,903,609
 3     42.387%      0.281/0.300/0.314/0.010        0.070/0.079/0.083/0.004        6,670,099
 4     41.647%      0.295/0.313/0.325/0.009        0.071/0.075/0.079/0.002        6,553,723
 5     41.216%      0.307/0.324/0.336/0.008        0.069/0.075/0.081/0.004        6,485,938
 6     41.037%      0.349/0.365/0.377/0.008        0.067/0.073/0.079/0.003        6,457,776
 7     40.778%      0.419/0.451/0.465/0.011        0.067/0.075/0.080/0.004        6,416,919
 8     40.704%      0.519/0.565/0.579/0.014        0.143/0.155/0.165/0.006        6,405,244
 9     40.409%      0.668/0.699/0.725/0.015        0.072/0.080/0.085/0.004        6,358,951

 avg1  48.623%                        0.343                          0.092
 avg2  54.026%                        0.381                          0.103
 tot                                102.984                         27.688       76,514,734

Real-time protection disabled
OS: Windows 10 10.0.22000 AMD64
CPU: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
Tool: ../zlib-ng/build/Release/minigzip.exe
Levels: 0-9
Runs: 70         Trim worst: 40

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.015%      0.018/0.021/0.023/0.001        0.017/0.019/0.021/0.001       15,738,743
 1     54.166%      0.095/0.103/0.111/0.005        0.049/0.053/0.056/0.002        8,523,732
 2     43.871%      0.155/0.165/0.176/0.007        0.051/0.055/0.058/0.002        6,903,609
 3     42.387%      0.218/0.227/0.240/0.008        0.048/0.053/0.057/0.002        6,670,099
 4     41.647%      0.239/0.251/0.264/0.009        0.047/0.052/0.056/0.003        6,553,723
 5     41.216%      0.245/0.265/0.276/0.010        0.047/0.050/0.054/0.002        6,485,938
 6     41.037%      0.287/0.304/0.316/0.010        0.046/0.051/0.054/0.002        6,457,776
 7     40.778%      0.353/0.372/0.392/0.015        0.046/0.051/0.054/0.002        6,416,919
 8     40.704%      0.446/0.465/0.493/0.017        0.047/0.051/0.055/0.003        6,405,244
 9     40.409%      0.525/0.546/0.580/0.020        0.047/0.051/0.057/0.003        6,358,951

 avg1  48.623%                        0.272                          0.049
 avg2  54.026%                        0.302                          0.054
 tot                                 81.599                         14.582       76,514,734

Notice that level 0 and level 8 return to normal when real-time protection is disabled.

@KungFuJesus Level 7 and Level 8 should not differ so much that it would affect speed much. And certainly level 0 should not be very challenging at all, since it does not in fact compress the data at all. I don't think the LLC is the problem that causes the 11KB smaller level 8 file to be 2.5x slower to decompress than the level 7 file.. If anything, it should be the other way around. The compressed file is also only about 6.4MiB, so that should leave a lot of room even in the 15MiB tiger lake cache.
I have a feeling this is OS or timing related, but it is pretty hard to do any kind of deep inspection like this on windows. Intels Vtune Profiler could do it though https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html It can (used to at least) be a bit of a hassle to set up, but that whole toolkit allows very powerful analysis both on Linux and Windows (and Android possibly).
Remember that even the Raspberry Pi 1B shows a much faster level 0 than level 1, and level 8 is just about as expected. And that thing does not have a lot of cache at all.

I'll contend that the level 7 & 8 different could very well be a bug or a really strange side effect.  However the level 0 being slower than level 1 despite being far less work for the CPU doesn't seem strange to me, especially if level 1 is not significantly more work and the compression reduces the traffic from memory to disk by nearly half. Even if it were just dumping it back into memory with a round trip, I could see the reduction in memory bandwidth and cache pressure being a win.  This is precisely the argument made for compressed swap or file systems with transparent compression.
Though, on my system level 0 is beating level 1 by a long shot (at least when I operating entirely out of a tmpfs backed file system):
 Tool: minigzip   Levels: 0-9
 Runs: 15         Trim worst: 5

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.000/0.001/0.002/0.001        0.000/0.001/0.003/0.001       15,737,543
 1     54.166%      0.070/0.072/0.074/0.001        0.033/0.035/0.037/0.002        8,523,732
 2     43.871%      0.134/0.137/0.139/0.002        0.032/0.037/0.039/0.002        6,903,609
 3     42.387%      0.162/0.165/0.166/0.001        0.033/0.035/0.037/0.001        6,670,099
 4     41.647%      0.192/0.193/0.194/0.001        0.031/0.034/0.037/0.002        6,553,723
 5     41.216%      0.208/0.211/0.214/0.002        0.031/0.034/0.036/0.001        6,485,938
 6     41.037%      0.253/0.255/0.258/0.001        0.032/0.034/0.035/0.001        6,457,776
 7     40.778%      0.345/0.348/0.350/0.002        0.031/0.034/0.036/0.001        6,416,919
 8     40.704%      0.452/0.455/0.456/0.001        0.032/0.034/0.038/0.002        6,405,244
 9     40.409%      0.533/0.536/0.537/0.001        0.031/0.034/0.036/0.002        6,358,951

 avg1  48.622%                        0.237                          0.031
 avg2  54.025%                        0.264                          0.035
 tot                                 23.724                          3.132       76,513,534

I do likely have cache depraved systems I could try this on, but I might even have something a step further - a slow 7200 RPM disk:
 Tool: minigzip   Levels: 0-9
 Runs: 15         Trim worst: 5

 Level   Comp   Comptime min/avg/max/stddev  Decomptime min/avg/max/stddev  Compressed size
 0    100.008%      0.014/0.016/0.018/0.002        0.017/0.020/0.022/0.002       15,737,543
 1     54.185%      0.147/0.150/0.153/0.002        0.074/0.080/0.082/0.003        8,526,745
 2     43.871%      0.270/0.275/0.277/0.002        0.076/0.081/0.083/0.002        6,903,702
 3     42.388%      0.336/0.338/0.340/0.002        0.074/0.077/0.080/0.002        6,670,239
 4     41.647%      0.382/0.384/0.387/0.002        0.072/0.076/0.079/0.002        6,553,746
 5     41.216%      0.418/0.423/0.426/0.003        0.073/0.076/0.079/0.002        6,485,936
 6     41.038%      0.489/0.491/0.493/0.002        0.072/0.075/0.077/0.002        6,457,827
 7     40.778%      0.644/0.648/0.650/0.002        0.072/0.075/0.077/0.002        6,416,941
 8     40.704%      0.810/0.812/0.814/0.002        0.071/0.074/0.077/0.002        6,405,249
 9     40.409%      0.894/0.897/0.899/0.002        0.074/0.077/0.079/0.002        6,358,951

 avg1  48.624%                        0.444                          0.071
 avg2  54.027%                        0.493                          0.079
 tot                                 44.352                          7.097       76,516,879

Ok, I'll eat my words.  What is going on with Windows?  Are you plugged in?

I used SysInternal's ProcessMonitor to watch the file read/writes from minigzip.exe as deflatebench ran. It seems on level 0 and level 8 a Win32 function called QueryEAFile is run when the archive is first opened. It is probably related to Windows Defender although I can't confirm because nothing really pulls up online for that function. We have seen in our own applications at work that Windows Defender can cause delays when closing files by several seconds as the system scanner checks the file before allowing it to be committed to disk. The doubling of the decompression time makes sense if the Windows Defender scanner also opens and decompresses the achieve for scanning.

Lol, oh yeah, that's likely it.  We've seen all kinds of weird performance issues with Windows defender.  You might want to whitelist the directory you're benchmarking to, it intercepts all writes and tries to inspect them.  Man, I wonder how many times you've had bad benchmark info due to defender chowing down on your IO performance.
This definitely justifies the Google Benchmark microbenchmarks quite a bit more.

You might want to whitelist the directory you're benchmarking to, it intercepts all writes and tries to inspect them.

Good idea! I had forgotten about that which I had done before... but I recently reimaged my machine and now need to set that up.


Revert "(feat) add generate sitemap.txt"

Reverts hexojs/hexo-generator-sitemap#136
It is causing issue #139.
@yoshinorin Please release an emergency release 3.0.1 after revertion.
cc @love4taylor

Coverage remained the same at 100.0% when pulling 49a8e7ea5429def6bf2c8b581490ce55c407de12 on revert-136-master into 11bd26daa4232ec9e4502acaa2a4e3d85c25c176 on master.

@SukkaW
I merged #141 & published 3.0.1. Now it works well.
Thank you :)


Added Checkbox code

This uses labels instead of the toggle class, giving the added benefit
of being able to click on the label as well as the icon. Added a check
into ROADMAP as well

Nice, thanks!


Move Go Integration tests into main test Github workflow

e2e tests are still separate, there's a separate ticket for those.

Going to merge this. The Rust benchmarking workflows run on this and they take a long time to run. By the time they finish, the PR is out of date already, so it's going to be very hard to get into a state where auto-merge works.


Expose MultiMeshInstance instance transform and ability to add collision

Issue description:
it will be very useful if someone expose the instance transform in MultiMeshInstance node
because The "Populate MultiMesh" window dialog is not very useful  .. this node is almost useless if we can't control the instance transform in Editor
and it will be an awesome thing if we have the Ability to add collision per instance ( if this possible )
i think this will be very useful to improve the engine performance

The way to do this is to have a script that will take a list of nodes (whether they be editor 3d gizmos, physics objects, etc.), grab the transforms of those objects, and apply to the multimesh. Multimesh is mostly meant to be interfaced with programmatically. In my game if I want to have 30 colliding instances being rendered by a multimesh, it's simple, I just use 30 rigid bodies and then I have a multimesh listen to the transformations of those 30 rigid bodies. If you want to see this in editor mode, make sure to have a "tool" at the top of your script.
@akien-mga I consider this closeable and too open-ended of an issue. @CCODE05B I can talk to you more about this if you're facing issues.

This solution might be even more efficient, maybe:
https://godotengine.org/qa/52924/how-set-collisionshape-on-multimeshinstance

Closed as this is a very old abandoned enhancement issue. If the author still needs it - this should be properly re-created at godot-proposals to receive user votes and dev suggestions.


Voldemort server cleans up HA state.

Added code so that the Voldemort server automatically cleans
the shared High Availability state from HDFS when appropriate.
Currently, this new code runs when:

An old Read-Only store-version is deleted, which usually
happens asynchronously after a new store-version is activated.
When a server transitions from OFFLINE mode to ONLINE mode.

This new benahvior is disabled by default, but can be enabled via:
push.ha.state.auto.cleanup=true
N.B.: This is not tested yet.

Left some minor comments, looks good to me.

When you are done testing this, please merge this in.

Closed by mistake, re-opened it.

I'm done testing. I squashed and merged.


Create SerialPort-background-thread-exceptions.md

@AlexGhiondea,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

/cc @JeremyKuhne

@rpetrusha What is the comment?

@AlexGhiondea, @JeremyKuhne, sorry, it was a comment in the text, and evidently wasn't saved. I'll recreate it.


LUIS Dispatch Bot sdk V3 support?

Is it possible to create LUIS Dispatch for existing V3 Bot ? We have v3 bot that's in production and we have gone so far now so its difficult to switch over v4 Bot. However, there are awesome capabilities like LUIS dispatch tool that we would like to leverage.
Ref:- https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-tutorial-dispatch?view=azure-bot-service-4.0&tabs=jsaddref%2Cjsbotconfig#integrate-multiple-luis-apps-and-qna-services-with-the-dispatch-tool

Any update on this ?

@CezaryMarcjan can you help answer this?

Yes. The dispatch CLI produces a LUIS model which can be used by V3 bot as any other LUIS model. Please keep in mind that dispatcher is not GA yet (it is pre-release).


Fix ioutil deprecation

Updated: -

- bouncer/licenses/classifier.go
- bouncer/licenses/find.go:20
- bouncer/finder.go

to remove deprecated ioutil library

Signed-off-by: Dave Hay<EMAIL_ADDRESS>

bouncer/licenses/classifier.go changes
bouncer/licenses/classifier.go:19:2: SA1019: "io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details. (staticcheck)
	"io/ioutil"
	^

bouncer/licenses/find.go changes
bouncer/licenses/find.go:20:2: SA1019: "io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details. (staticcheck)
	"io/ioutil"
	^

bouncer/finder.go changes
bouncer/finder.go:7:2: SA1019: "io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details. (staticcheck)
	"io/ioutil"
	^

Hey @wagoodman thanks for the review/approval - however I note that one of the CI build jobs failed with: -
...

Allow Rules: [BSD.* MIT.* Apache.* MPL.*]
Unallowable license (OpenSSL) from "crypto/internal/boring"
failed validation
make: *** [Makefile:98: ci-plugs-out-test] Error 1
Error: Process completed with exit code 2.
...

and that you've added crypto/internal/boring to .bouncer.yaml
Is there a way to rerun the CI for this PR and see whether that test now passes ?


Prepare 0.3.5

coverage: 72.155%. remained the same
when pulling 8afc62263c35bdf303ece9b5d24c2f0a81d89c37 on massi/0.3.5
into 93176e2daf2bdfd899ee4b8182d8001a4cd4930f on main.


K3OS VMs querying their own DNS like crazy

Hi Everyone,
I created two k3os one-node cluster VMs in Proxmox lately, and when I checked the PiHole DNS blackhole service on the LAN, which is for filtering out ads and other unwanted sites, I saw that the k3os VMs are querying themselves in an extreme measure, like 30+ thousand times in a day. After I created the first VM, it ran for some two days, then I destroyed it for experimentation purposes, then created the current one yesterday, which does the same.
Both VM's init was the following:
# get latest iso once (it was 0.2.0-rc6 previously this week)
cd /var/lib/vz/template/iso
GITHUB_REPO="rancher/k3os"
GITHUB_LATEST_RELEASE=$(curl -L -s -H 'Accept: application/json' https://github.com/${GITHUB_REPO}/releases/latest)
GITHUB_LATEST_VERSION=$(echo $GITHUB_LATEST_RELEASE | sed -e 's/.*"tag_name":"\([^"]*\)".*/\1/')
GITHUB_ORIG_FILE="k3os-amd64.iso"
GITHUB_DOWN_FILE="k3os-amd64-${GITHUB_LATEST_VERSION//v/}.iso"
GITHUB_URL="https://github.com/${GITHUB_REPO}/releases/download/${GITHUB_LATEST_VERSION}/${GITHUB_ORIG_FILE}"
wget -O $GITHUB_DOWN_FILE $GITHUB_URL

qm create 200 --agent 1 --cores 2 --ide2 local:iso/${GITHUB_DOWN_FILE},media=cdrom --memory 3072 --name k-node-1 --net0 virtio,bridge=vmbr0,firewall=1 --numa 0 --onboot 1 --ostype l26 --scsi0 local-lvm:8 --scsihw virtio-scsi-pci --sockets 1
qm start 200

# login in VNC as rancher
sudo passwd rancher
# type rancher
# exit VNC

# ssh login using a DHCP allocated address
ssh<EMAIL_ADDRESS># type rancher

sudo os-config
# Install to disk: 1
# Cloud-init: N
# Github SSH: N
# Type new pass 2 times
# Wifi: N
# Server: 1
# Cluster secret: generate new with `openssl rand -hex 20` in a new tab
# Continue: y
# reboot

# host info changes, so remove it
ssh-keygen -f "/root/.ssh/known_hosts" -R <IP_ADDRESS>
ssh<EMAIL_ADDRESS># using new pass

Nothing else was set up, I just looked around inside the VMs. The process could be easier on the setup side (no VNC interaction, automated network settings, installed image by default), but there are other issues open for cloud-init, etc. so I won't bother with it.
My only problem now is the VM going berserk on querying its own DNS, it amounts to ~90% of the local DNS traffic these days. Can it be stopped somehow? I would like to use fixed IPs, but I just haven't got the time to set it up yet with conman (I also need to look into conman's config options, new stuff to me). Could it stop the flood?

I tried to assign a fix IP to the one-node K3OS cluster VM to stop the DNS query flood but it doesn't help, the VM still bombards the DNS server.
ETH0=$(sudo connmanctl services | awk '{ print $3 }' | while read -r s1; do sudo connmanctl services $s1 | grep -q "eth0" && echo "$s1"; done)
echo $ETH0
# ethernet_abcdef01234_cable
sudo connmanctl config $ETH0 --ipv4 manual <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS> --nameservers <IP_ADDRESS>
sudo connmanctl config $ETH0 --domains my.local
sudo connmanctl config $ETH0 --ipv6 off
sudo service connman restart

Last 24 hours, 31000+ queries:

The interval seems to be 10s according to the log, if it helps.

I created a run_cmd file to set up networking through cloud-init as can be seen in my other comment here https://github.com/rancher/k3os/issues/68#issuecomment-492089070 and this DNS query issue still exists.

This should be fixed with a recent change; this should work much better now. Could you try with the latest RC? https://github.com/rancher/k3os/releases/tag/v0.3.0-rc2

Thanks, sure, though I'll need a couple of days to test it, will report back when ready.

I was too curious, so allocated time in the morning to test it with v0.3.0, it seems the DNS querying is reduced to 1 per hour at every XX:00:00, wonderful, thanks for the fix!


Custom attributes extraction

Looks like the test failure on min is because of Pydantic 1.x not producing additional metadata parts (that we don't use anyway) for Json fields.

Pydantic 1.x not producing additional metadata parts

Actually it's Pydantic 2.0, (we require 2.0+ and that code was added in 2.1). We can change the test expectations to be different on different Pydantic versions, or we can bump the minimum Pydantic version (no idea if that makes a difference for anything except the test itself).

I think bumping Pydantic 2.x version is fine. It seems we still should keep 1.8 support for now, right?

We don't have 1.8 support in this project, as we have pydantic>=2 since the beginning.

But isn't the fix in scrapy-spider-metadata, not here, which supports old pydantic?


OAK-9339 - Image Similarity: LSH based search

Changes committed in http://svn.apache.org/viewvc?view=revision&revision=1886318


chore: install wasm-opt automatically

Description
Updates the Makefile to install the wasm-opt command automatically in the bin directory. Following a similar approach to the go linter. Therefore, the user does not need to install it manually.

I've mixed feelings about that, since I install wasm-opt via binaryen and keep the whole binaryen package updated via package manager.
I fear this binary could become stale over the time

@flavio @jvanz, not a strong opinion but I would also try to avoid maintaining another tool version as it doesn't change so much.

Fair enough. Closing this.


Project name same as dependency name - cycle detected and restore failed

Description
Restore fails when project is name is the same as project's dependency. Tested on windows which is case insensitive.
Repro steps
Follow steps described in https://saturnframework.org/docs/guides/how-to-start/ inside saturn folder:

install .net core sdk version 3.1.201
mkdir saturn && cd ./saturn
dotnet tool install fake-cli -g
dotnet new -i Saturn.Template
dotnet new saturn -lang F#
fake build

Expected behavior
Build succeeded
Actual behavior
Build failed with error:
Restore completed in 83,95 ms for C:\projects\saturn\src\saturn\saturn.fsproj.
C:\projects\saturn\src\saturn\saturn.fsproj : error NU1108: Cycle detected.
C:\projects\saturn\src\saturn\saturn.fsproj : error NU1108:   saturn -> Saturn (>= 0.12.1).
Restore failed in 1,09 sec for C:\projects\saturn\src\saturn\saturn.fsproj.
Known workarounds
Name root project folder with different name than referenced saturn dll - for example saturnexample

that seems to be a NuGet / dotnet SDK related issue and not related to
Paket. I think
Am Fr., 3. Apr. 2020 um 00:41 Uhr schrieb padzikm<EMAIL_ADDRESS>
:

Description
Restore fails when project is name is the same as project's dependency.
Tested on windows which is case insensitive.
Repro steps
Follow steps described in
https://saturnframework.org/docs/guides/how-to-start/ inside saturn
folder:

install .net core sdk version 3.1.201
mkdir saturn && cd ./saturn
dotnet tool install fake-cli -g
dotnet new -i Saturn.Template
dotnet new saturn -lang F#
fake build

Expected behavior
Build succeeded
Actual behavior
Build failed with error:
Restore completed in 83,95 ms for
C:\projects\saturn\src\saturn\saturn.fsproj.
C:\projects\saturn\src\saturn\saturn.fsproj : error NU1108: Cycle detected.
C:\projects\saturn\src\saturn\saturn.fsproj : error NU1108: saturn ->
Saturn (>= 0.12.1).
Known workarounds
Name root project folder with different name than referenced saturn dll -
for example saturnexample
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/3821, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAOANCNPJADOWHOFL26XITRKUIDBANCNFSM4L3ALEFA
.

Can you explain why do you think that? Paket is responsible for restoring packages and I can't see any logical connection to dotnet sdk.

You are not allowed to name your project as the dependency. this is a requirement that dotnet SDK / NuGet brings. NU1108 is an NuGet error that is raised in such cases by NuGet itself (which is still running even when you use paket).

Ok, thanks for explanation


[release-1.15] istioctl x proxy-status via istio agents.

Manual cherrypick required.
#40623 failed to apply on top of branch "release-1.15":
Applying: `istioctl ps` via istio agents.
Applying: Fix lint errors
Applying: Fix lint errors
Applying: Use k8s client_go instead of RESTClient
Applying: Update pkg/kube/client.go
Applying: Use constant for "status.phase=Running"
Applying: Use the revision in the annotation instead of labels for the injected pods
Applying: Add xds-via-agents-limit flag
Applying: Reflect the comments and refactoring
Using index info to reconstruct a base tree...
M	istioctl/cmd/internal-debug.go
M	istioctl/cmd/proxystatus.go
M	istioctl/pkg/multixds/gather.go
Falling back to patching base and 3-way merge...
Auto-merging istioctl/pkg/multixds/gather.go
CONFLICT (content): Merge conflict in istioctl/pkg/multixds/gather.go
Auto-merging istioctl/cmd/proxystatus.go
Auto-merging istioctl/cmd/internal-debug.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0009 Reflect the comments and refactoring
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

/help

I think this is fixed by https://github.com/istio/istio/pull/41041


Less file compile issue

Hi, when I use gulp.spritesmith in my project, I found something was wired.
I tried to use less as the sprite css output file, but after gulp run less task, the sprite file was compiled to empty.
gulpfile.js
var runSpriteTask = function(projectName){
    var sliceSourceFile = projectName + '/slice/*.png';
    var retinaSliceSourceFile = projectName +<EMAIL_ADDRESS>

    var spriteData = gulp.src(sliceSourceFile)
                            .pipe(spritesmith({
                                imgName: projectName + '.png',
                                cssName: 'mod-sprite.less',
                                retinaSrcFilter : [retinaSliceSourceFile],
                                retinaImgName: projectName +<EMAIL_ADDRESS>                            }));
    var imgStream = spriteData.img.pipe(gulp.dest(OutputImagePath));
    var cssStream = spriteData.css.pipe(gulp.dest(projectName + '/css/'));
    return merge (imgStream,cssStream)
}

And this is gulp.spritesmith export sprite file  mod-sprite.less
/*
LESS variables are information about icon's compiled state, stored under its original file name

.icon-home {
  width: @icon-home-width;
}

The large array-like variables contain all information about a single icon
@icon-home: x y offset_x offset_y width height total_width total_height image_path name;

At the bottom of this section, we provide information about the spritesheet itself
@spritesheet: width height image @spritesheet-sprites;
*/
@approval-detail-reject-name: 'approval_detail_reject';
@approval-detail-reject-x: 0px;
@approval-detail-reject-y: 0px;
@approval-detail-reject-offset-x: 0px;
@approval-detail-reject-offset-y: 0px;
@approval-detail-reject-width: 15px;
@approval-detail-reject-height: 15px;
@approval-detail-reject-total-width: 30px;
@approval-detail-reject-total-height: 15px;
@approval-detail-reject-image: 'worknote-desktop.png';
@approval-detail-reject: 0px 0px 0px 0px 15px 15px 30px 15px 'worknote-desktop.png' 'approval_detail_reject';
@approval-detail-uncheck-name: 'approval_detail_uncheck';
@approval-detail-uncheck-x: 15px;
@approval-detail-uncheck-y: 0px;
@approval-detail-uncheck-offset-x: -15px;
@approval-detail-uncheck-offset-y: 0px;
@approval-detail-uncheck-width: 15px;
@approval-detail-uncheck-height: 15px;
@approval-detail-uncheck-total-width: 30px;
@approval-detail-uncheck-total-height: 15px;
@approval-detail-uncheck-image: 'worknote-desktop.png';
@approval-detail-uncheck: 15px 0px -15px 0px 15px 15px 30px 15px 'worknote-desktop.png' 'approval_detail_uncheck';
@spritesheet-width: 30px;
@spritesheet-height: 15px;
@spritesheet-image: 'worknote-desktop.png';
@spritesheet-sprites: @approval-detail-reject @approval-detail-uncheck;
@spritesheet: 30px 15px 'worknote-desktop.png' @spritesheet-sprites;

/*
The provided classes are intended to be used with the array-like variables

.icon-home {
  .sprite-width(@icon-home);
}
.icon-email {
  .sprite(@icon-email);
}

Example usage in HTML:

`display: block` sprite:

To change `display` (e.g. `display: inline-block;`), we suggest using a common CSS class:

// CSS
.icon {
  display: inline-block;
}

// HTML

*/
.sprite-width(@sprite) {
  width: extract(@sprite, 5);
}

.sprite-height(@sprite) {
  height: extract(@sprite, 6);
}

.sprite-position(@sprite) {
  @sprite-offset-x: extract(@sprite, 3);
  @sprite-offset-y: extract(@sprite, 4);
  background-position: @sprite-offset-x @sprite-offset-y;
}

.sprite-image(@sprite) {
  @sprite-image: extract(@sprite, 9);
  background-image: e(%('url(%a)', e(@sprite-image)));
}

.sprite(@sprite) {
  .sprite-image(@sprite);
  .sprite-position(@sprite);
  .sprite-width(@sprite);
  .sprite-height(@sprite);
}

/*
The `.sprites` mixin generates identical output to the CSS template
  but can be overridden inside of LESS

This must be run when you have at least 2 sprites.
  If run with a single sprite, then there will be reference errors.

.sprites(@spritesheet-sprites);
*/
.sprites(@sprites, @i: 1) when (@i <= length(@sprites)) {
  @sprite: extract(@sprites, @i);
  @sprite-name: e(extract(@sprite, 10));
  .@{sprite-name} {
    .sprite(@sprite);
  }
  .sprites(@sprites, @i + 1);
}

style-test.less
@import "mod-sprite";
body {
    color: red;
    font-size: 20px;
    left: 0;
    top: 0;
}

This is compiled file and all the sprite css code were gone.
style-test.css
/*
LESS variables are information about icon's compiled state, stored under its original file name

.icon-home {
  width: @icon-home-width;
}

The large array-like variables contain all information about a single icon
@icon-home: x y offset_x offset_y width height total_width total_height image_path name;

At the bottom of this section, we provide information about the spritesheet itself
@spritesheet: width height image @spritesheet-sprites;
*/
/*
These "retina group" variables are mappings for the naming and pairing of normal and retina sprites.

The list formatted variables are intended for mixins like `retina-sprite` and `retina-sprites`.
*/
/*
The provided classes are intended to be used with the array-like variables

.icon-home {
  .sprite-width(@icon-home);
}
.icon-email {
  .sprite(@icon-email);
}

Example usage in HTML:

`display: block` sprite:

To change `display` (e.g. `display: inline-block;`), we suggest using a common CSS class:

// CSS
.icon {
  display: inline-block;
}

// HTML

*/
/*
The `retina-sprite` class sets up rules and a media query for a sprite/retina sprite.
  It should be used with a "retina group" variable.

The media query is from CSS Tricks: https://css-tricks.com/snippets/css/retina-display-media-query/

@icon-home-group: 'icon-home' @icon-home @icon-home-2x;

.icon-home {
  .retina-sprite(@icon-home-group);
}
*/
/*
The `.sprites` mixin generates identical output to the CSS template
  but can be overridden inside of LESS

This must be run when you have at least 2 sprites.
  If run with a single sprite, then there will be reference errors.

.sprites(@spritesheet-sprites);
*/
/*
The `.retina-sprites` mixin generates a CSS rule and media query for retina groups
  This yields the same output as CSS retina template but can be overridden in LESS

.retina-sprites(@retina-groups);
*/
body {
  color: red;
  font-size: 20px;
  left: 0;
  top: 0;
}

I have tried run lessc mod-sprite.less aaa.css, and it returnd the same output.

Spritesmith only provides variables and mixins which can generate CSS. This
is because some people might be adapting legacy systems, others might have
custom says of handling sprites
We do provide a convenience mixin for you to use. Try the following after
@import:
.sprites(@spritesheet-sprites);
I'm afk at the moment but will close this issue as a question when I get access
On Wed, Sep 25, 2019, 5:25 AM Meters<EMAIL_ADDRESS>wrote:

I have tried run lessc mod-sprite.less aaa.css, and it returnd the same
output.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/twolfson/gulp.spritesmith/issues/148?email_source=notifications&email_token=AAG4KWDB463K55LLKJ6NFIDQLNKFLA5CNFSM4I2MCLW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7RWRHA#issuecomment-534997148,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAG4KWGNWAH72ZOXDCTN3ILQLNKFLANCNFSM4I2MCLWQ
.

Oh, I got. Thank you.