Solved incident "llm_lazy" - "llm_summary"  running against coolstore and llama3 is seeing multiple issues

run_demo.py against coolstore with llama3 served from IBM BAM.
I am running with an alternate configuration where I explicitly enabled the solution_producers "llm_lazy".
As I did this I have not seen a single fix come back to the run_demo.py client, it's been close to 60 minutes now since I first kicked this off.
I am seeing multiple problems:

~10 minutes of no activity in server logs, and then I saw:

"The user message is too long. Send a short message or decrease property min_new_tokens" for pom.xml

waited an additional ~40 minutes past the prior error and have not see any other activity from other files in the server console

I do see DEBUG activity in logs/kai_server.log, but it is sparse, all I see are multiple kai.models.file_solution - [    file_solution.py:23   -       guess_language()]
Because I do see that 1 line outputted multiple times for various files I can see that for the past 40 minutes the server is processing more files for solved incidents.  We do not output any other information to logs to know what is happening.

Server console output:  https://gist.github.com/jwmatthews/a60f2a36b5691b466d9964386c61d04b
Full kai_server log:  https://gist.githubusercontent.com/jwmatthews/7451f65a05dd38e5c22651d7f53e451a/raw/38cd9e0ebdb8d0e7b411aa57a813f262d854bf6b/kai_server.log

Note that I kicked off this experiment ~930am, there are some prior runs earlier in the log file which can be ignored.

Looking at one example of pom.xml traceback:   https://gist.github.com/jwmatthews/7c5a6e4d24bf445c35d972013cad4729
Sample error from pom.xml:
Failed to handle request to https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10.
{
  "error": "Bad Request",
  "extensions": {
    "code": "INVALID_INPUT",
    "state": null
  },
  "message": "The user message is too long. Send a short message or decrease property min_new_tokens",
  "status_code": 400
}

The config I am running with
$ cat ~/git/jwmatthews/kai/kai/config.toml
# Default configuration file for Kai. For a better understanding of the
# configuration options, please refer to `build/example_config.toml`

log_level = "info"
file_log_level = "debug"
log_dir = "$pwd/logs"
demo_mode = false
trace_enabled = true

solution_consumers = ["diff_only", "llm_summary"]

[incident_store]
solution_detectors = "naive"
solution_producers = "llm_lazy"

[incident_store.args]
provider = "postgresql"
host = "<IP_ADDRESS>"
database = "kai"
user = "kai"
password = "dog8code"

[models]
provider = "ChatIBMGenAI"

[models.args]
model_id = "meta-llama/llama-3-70b-instruct"
parameters.max_new_tokens = 2048

Below is an example from above run that completed.  It's been ~75 minutes since I kicked off run_demo.py.
Most of the files request to be fixed have timed out and are retrying, below is 1 example that succeeded
$ gist ShoppingCart.java.prompts.md
https://gist.github.com/jwmatthews/481ed0ced17b81588e5a9362c663ec26
$ gist ShoppingCart.java.llm_result.md
https://gist.github.com/jwmatthews/881a6f06dcc630d3b5b12733d8ce9c16
$ gist ShoppingCart.java.llm_response_metadata.json
https://gist.github.com/jwmatthews/8a1d0f55d680c64a4aabd50a01d1c51a

Example of what I'm seeing with run_demo.py
$ time ./run_demo.py
26 files with a total of 150 incidents.
INFO - 2024-09-21 09:33:57,004 - __main__ - [         run_demo.py:227  -             run_demo()] - Running in parallel with 8 workers
INFO - 2024-09-21 09:33:57,004 - __main__ - [         run_demo.py:192  -         process_file()] - File #1 of 26 - Processing src/main/webapp/WEB-INF/web.xml which has 1 incidents.
INFO - 2024-09-21 09:33:57,005 - __main__ - [         run_demo.py:192  -         process_file()] - File #2 of 26 - Processing pom.xml which has 12 incidents.
INFO - 2024-09-21 09:33:57,005 - __main__ - [         run_demo.py:192  -         process_file()] - File #3 of 26 - Processing src/main/java/com/redhat/coolstore/model/Order.java which has 10 incidents.
INFO - 2024-09-21 09:33:57,005 - __main__ - [         run_demo.py:192  -         process_file()] - File #4 of 26 - Processing src/main/java/com/redhat/coolstore/model/OrderItem.java which has 6 incidents.
INFO - 2024-09-21 09:33:57,005 - __main__ - [         run_demo.py:192  -         process_file()] - File #5 of 26 - Processing src/main/webapp/WEB-INF/beans.xml which has 5 incidents.
INFO - 2024-09-21 09:33:57,006 - __main__ - [         run_demo.py:192  -         process_file()] - File #6 of 26 - Processing src/main/resources/META-INF/persistence.xml which has 8 incidents.
INFO - 2024-09-21 09:33:57,006 - __main__ - [         run_demo.py:192  -         process_file()] - File #7 of 26 - Processing src/main/java/com/redhat/coolstore/model/InventoryEntity.java which has 6 incidents.
INFO - 2024-09-21 09:33:57,007 - __main__ - [         run_demo.py:192  -         process_file()] - File #8 of 26 - Processing src/main/java/com/redhat/coolstore/model/ShoppingCart.java which has 1 incidents.
INFO - 2024-09-21 09:34:18,062 - __main__ - [         run_demo.py:208  -         process_file()] - Response StatusCode: 200 for src/main/webapp/WEB-INF/web.xml

INFO - 2024-09-21 09:34:18,062 - __main__ - [         run_demo.py:110  -        write_to_disk()] - Writing updated source code to ./coolstore/src/main/webapp/WEB-INF/web.xml
INFO - 2024-09-21 09:34:18,062 - __main__ - [         run_demo.py:122  -        write_to_disk()] - Writing prompts to ./coolstore/src/main/webapp/WEB-INF/web.xml.prompts.md
INFO - 2024-09-21 09:34:18,063 - __main__ - [         run_demo.py:132  -        write_to_disk()] - Writing llm_response_metadata to ./coolstore/src/main/webapp/WEB-INF/web.xml.llm_response_metadata.json
INFO - 2024-09-21 09:34:18,064 - __main__ - [         run_demo.py:145  -        write_to_disk()] - Writing llm_result to ./coolstore/src/main/webapp/WEB-INF/web.xml.llm_result.md
INFO - 2024-09-21 09:34:18,064 - __main__ - [         run_demo.py:192  -         process_file()] - File #9 of 26 - Processing src/main/java/com/redhat/coolstore/persistence/Resources.java which has 6 incidents.
INFO - 2024-09-21 09:34:18,064 - __main__ - [         run_demo.py:239  -             run_demo()] - Result:  21.05963397026062s to process src/main/webapp/WEB-INF/web.xml with 1 violations
INFO - 2024-09-21 09:34:18,064 - __main__ - [         run_demo.py:246  -             run_demo()] - 25 files remaining from total of 26
INFO - 2024-09-21 09:44:04,445 - __main__ - [         run_demo.py:57   -         generate_fix()] - [pom.xml] Received status code 500
ERROR - 2024-09-21 09:44:04,445 - __main__ - [         run_demo.py:64   -         generate_fix()] - [pom.xml] Failed to get a '200' response from the server.  Retrying 6 more times
INFO - 2024-09-21 09:44:04,948 - __main__ - [         run_demo.py:57   -         generate_fix()] - [pom.xml] Received status code 500
ERROR - 2024-09-21 09:44:04,948 - __main__ - [         run_demo.py:64   -         generate_fix()] - [pom.xml] Failed to get a '200' response from the server.  Retrying 5 more times
INFO - 2024-09-21 09:44:05,278 - __main__ - [         run_demo.py:57   -         generate_fix()] - [pom.xml] Received status code 500
ERROR - 2024-09-21 09:44:05,278 - __main__ - [         run_demo.py:64   -         generate_fix()] - [pom.xml] Failed to get a '200' response from the server.  Retrying 4 more times
INFO - 2024-09-21 09:44:05,614 - __main__ - [         run_demo.py:57   -         generate_fix()] - [pom.xml] Received status code 500
ERROR - 2024-09-21 09:44:05,614 - __main__ - [         run_demo.py:64   -         generate_fix()] - [pom.xml] Failed to get a '200' response from the server.  Retrying 3 more times
INFO - 2024-09-21 09:44:05,956 - __main__ - [         run_demo.py:57   -         generate_fix()] - [pom.xml] Received status code 500
ERROR - 2024-09-21 09:44:05,956 - __main__ - [         run_demo.py:64   -         generate_fix()] - [pom.xml] Failed to get a '200' response from the server.  Retrying 2 more times
INFO - 2024-09-21 09:44:06,287 - __main__ - [         run_demo.py:57   -         generate_fix()] - [pom.xml] Received status code 500
ERROR - 2024-09-21 09:44:06,287 - __main__ - [         run_demo.py:64   -         generate_fix()] - [pom.xml] Failed to get a '200' response from the server.  Retrying 1 more times
INFO - 2024-09-21 09:44:06,288 - __main__ - [         run_demo.py:192  -         process_file()] - File #10 of 26 - Processing src/main/java/com/redhat/coolstore/rest/CartEndpoint.java which has 9 incidents.
INFO - 2024-09-21 10:28:26,933 - __main__ - [         run_demo.py:208  -         process_file()] - Response StatusCode: 200 for src/main/webapp/WEB-INF/beans.xml

INFO - 2024-09-21 10:28:26,935 - __main__ - [         run_demo.py:110  -        write_to_disk()] - Writing updated source code to ./coolstore/src/main/webapp/WEB-INF/beans.xml
INFO - 2024-09-21 10:28:26,935 - __main__ - [         run_demo.py:122  -        write_to_disk()] - Writing prompts to ./coolstore/src/main/webapp/WEB-INF/beans.xml.prompts.md
INFO - 2024-09-21 10:28:26,935 - __main__ - [         run_demo.py:132  -        write_to_disk()] - Writing llm_response_metadata to ./coolstore/src/main/webapp/WEB-INF/beans.xml.llm_response_metadata.json
INFO - 2024-09-21 10:28:26,944 - __main__ - [         run_demo.py:145  -        write_to_disk()] - Writing llm_result to ./coolstore/src/main/webapp/WEB-INF/beans.xml.llm_result.md
INFO - 2024-09-21 10:28:26,944 - __main__ - [         run_demo.py:192  -         process_file()] - File #11 of 26 - Processing src/main/java/com/redhat/coolstore/rest/OrderEndpoint.java which has 8 incidents.
ERROR - 2024-09-21 10:35:09,636 - __main__ - [         run_demo.py:61   -         generate_fix()] - [src/main/java/com/redhat/coolstore/model/ShoppingCart.java] Received exception: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Read timed out. (read timeout=3600)
ERROR - 2024-09-21 10:35:09,636 - __main__ - [         run_demo.py:61   -         generate_fix()] - [src/main/java/com/redhat/coolstore/model/OrderItem.java] Received exception: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Read timed out. (read timeout=3600)
ERROR - 2024-09-21 10:35:09,637 - __main__ - [         run_demo.py:64   -         generate_fix()] - [src/main/java/com/redhat/coolstore/model/ShoppingCart.java] Failed to get a '200' response from the server.  Retrying 6 more times
ERROR - 2024-09-21 10:35:09,637 - __main__ - [         run_demo.py:61   -         generate_fix()] - [src/main/resources/META-INF/persistence.xml] Received exception: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Read timed out. (read timeout=3600)
ERROR - 2024-09-21 10:35:09,638 - __main__ - [         run_demo.py:64   -         generate_fix()] - [src/main/resources/META-INF/persistence.xml] Failed to get a '200' response from the server.  Retrying 6 more times
ERROR - 2024-09-21 10:35:09,637 - __main__ - [         run_demo.py:61   -         generate_fix()] - [src/main/java/com/redhat/coolstore/model/Order.java] Received exception: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Read timed out. (read timeout=3600)
ERROR - 2024-09-21 10:35:09,640 - __main__ - [         run_demo.py:64   -         generate_fix()] - [src/main/java/com/redhat/coolstore/model/Order.java] Failed to get a '200' response from the server.  Retrying 6 more times
ERROR - 2024-09-21 10:35:09,637 - __main__ - [         run_demo.py:61   -         generate_fix()] - [src/main/java/com/redhat/coolstore/model/InventoryEntity.java] Received exception: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Read timed out. (read timeout=3600)
ERROR - 2024-09-21 10:35:09,641 - __main__ - [         run_demo.py:64   -         generate_fix()] - [src/main/java/com/redhat/coolstore/model/InventoryEntity.java] Failed to get a '200' response from the server.  Retrying 6 more times
ERROR - 2024-09-21 10:35:09,637 - __main__ - [         run_demo.py:64   -         generate_fix()] - [src/main/java/com/redhat/coolstore/model/OrderItem.java] Failed to get a '200' response from the server.  Retrying 6 more times
ERROR - 2024-09-21 10:35:30,642 - __main__ - [         run_demo.py:61   -         generate_fix()] - [src/main/java/com/redhat/coolstore/persistence/Resources.java] Received exception: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Read timed out. (read timeout=3600)
ERROR - 2024-09-21 10:35:30,643 - __main__ - [         run_demo.py:64   -         generate_fix()] - [src/main/java/com/redhat/coolstore/persistence/Resources.java] Failed to get a '200' response from the server.  Retrying 6 more times
INFO - 2024-09-21 10:35:57,218 - __main__ - [         run_demo.py:208  -         process_file()] - Response StatusCode: 200 for src/main/java/com/redhat/coolstore/model/ShoppingCart.java

INFO - 2024-09-21 10:35:57,219 - __main__ - [         run_demo.py:110  -        write_to_disk()] - Writing updated source code to ./coolstore/src/main/java/com/redhat/coolstore/model/ShoppingCart.java
INFO - 2024-09-21 10:35:57,219 - __main__ - [         run_demo.py:122  -        write_to_disk()] - Writing prompts to ./coolstore/src/main/java/com/redhat/coolstore/model/ShoppingCart.java.prompts.md
INFO - 2024-09-21 10:35:57,219 - __main__ - [         run_demo.py:132  -        write_to_disk()] - Writing llm_response_metadata to ./coolstore/src/main/java/com/redhat/coolstore/model/ShoppingCart.java.llm_response_metadata.json
INFO - 2024-09-21 10:35:57,222 - __main__ - [         run_demo.py:145  -        write_to_disk()] - Writing llm_result to ./coolstore/src/main/java/com/redhat/coolstore/model/ShoppingCart.java.llm_result.md
INFO - 2024-09-21 10:35:57,222 - __main__ - [         run_demo.py:192  -         process_file()] - File #12 of 26 - Processing src/main/java/com/redhat/coolstore/rest/ProductEndpoint.java which has 3 incidents.
ERROR - 2024-09-21 10:45:18,913 - __main__ - [         run_demo.py:61   -         generate_fix()] - [src/main/java/com/redhat/coolstore/rest/CartEndpoint.java] Received exception: HTTPConnectionPool(host='<IP_ADDRESS>', port=8080): Read timed out. (read timeout=3600)
ERROR - 2024-09-21 10:45:18,913 - __main__ - [         run_demo.py:64   -         generate_fix()] - [src/main/java/com/redhat/coolstore/rest/CartEndpoint.java] Failed to get a '200' response from the server.  Retrying 6 more times


[FEATURE REQUEST] Capture entire change as raw JSON to display on Trail admin screen

Similar to:

Thanks, me.


doc/storage: add link to video about benchmarking storage drivers

Added as a related link instead of "embedded" video, because the
video isn't illustrating the same content as the page, but presents
additional, related information.
Signed-off-by: Ruth Fuchss<EMAIL_ADDRESS>

Documentation preview available at: https://linuxcontainers.org/lxd/docs/pr.10698/


Support valhalla and corretto build

support strinf in JDK_VERSION
fix the display order

Signed-off-by: lanxia<EMAIL_ADDRESS>


Add Loading State to Follow Button

In the user page and user card views, when the follow button is clicked and it is waiting for the follow call to work, it's not clear anything is happening.
Add a loading/in progress state to the button when it is clicked, which is cleared when the call completes, similar to what was used in v1.
Ideally this is done in a way that can easily be applied to any button.

There's already a 'processing' class for this. It's styling is a bit of a placeholder, but we could tweak that once we settle on a spinner.

#355 fixes this in the User Card view.
It still needs to be fixed on the user card on the Home tab in the user page, since that isn't rendering a User Card view, but is just using the template. The more robust fix would be for that to use the actual User Card view, so we don't always need to double-do functionality. The User Card view needs to be adjusted to not necessarily always fetch a Profile model, since in some cases (like this one) you would already have one. At the least that fetch should be opt-out.

FWIW, with the new batch / async profile API, we probably want to make the user card view so that it could place nice with that.
I think the user card should be able to accept a profile model at creation (since the parent may already have one) or have one provided at a later time (e.g. with lists of users, the parent would probably get the profile via the batch API).
If there are use-cases where the user card should create and fetch the profile on it's own, I echo my sentiments above that that functionality should be optional.

I agree that the userCard view should be updated to accept a profile, and only fetch it on it's own if a guid is provided with no profile.

this was fixed.


Add Support for bzip2 decompression in Decompressor

🚀 The feature
As of now Decompressor does not have support for .bz2 files. This could be added.
Motivation, pitch
I am currently working on https://github.com/pytorch/vision/issues/5352 with the USPS dataset which uses bzip2. I had to use bz2 directly. Would've liked to use Decompressor.
Alternatives
No response
Additional context
No response

Thanks for the suggestion. We will be open to accepting a PR that adds bzip2 to Decompressor.

@NivekT I can take it up :)


LS Extension UI contributions, ownership, and maintenance

Spike on figuring out how LS extension can contribute its unique UI experience. How this will be maintained for Pylance? What is the ownership model of this component?

Pylance currently is its own extension, the language client is created in the Python extension by taking the launch parameters from Pylance. This allows all the features from Pylance to be registered via Python extension. LS features and UI happens via LS Client, other extension features, and UI are handled in Pylance since it is just another extension from that perspective.


error using painters, ghoscript?

I'm trying to run the following code to export a figure:
export_fig('figure1ricardian_cleanA2.png','-m2','-painters',figout1)
I get the error output that is posted below, and have verified that I have the latest version of export_fig, have ghostscript installed, and that there are not multiple versions of export_fig.
Error: /undefined in bdef
Operand stack:
dom   --nostringval--
Execution stack:
%interp_exit   .runexec2   --nostringval--   --nostringval--   --nostringval--   2   %stopped_push   --nostringval--   --nostringval--   --nostringval--   false   1   %stopped_push   1967   1   3   %oparray_pop   1966   1   3   %oparray_pop   1950   1   3   %oparray_pop   1836   1   3   %oparray_pop   --nostringval--   %errorexec_pop   .runexec2   --nostringval--   --nostringval--   --nostringval--   2   %stopped_push   --nostringval--
Dictionary stack:
--dict:1203/1684(ro)(G)--   --dict:0/20(G)--   --dict:78/200(L)--
Current allocation mode is local
Current file position is 64
GPL Ghostscript 9.18: Unrecoverable error, exit code 1
export_fig error. Please ensure:
that you are using the latest version of export_fig
and that you have Ghostscript installed
and that you do not have multiple versions of export_fig installed by mistake
and that you did not made a mistake in the expected input arguments
If the problem persists, then please report a new issue.
Error using imread (line 349)
File "/private/tmp/tpd6423327_b5ba_4f3a_bbf4_c38d50bf5d6e.tif" does not
exist.
Error in print2array (line 94)
A = imread(tmp_nam);
Error in export_fig (line 518)
[A, tcol] = print2array(fig, magnify, renderer);

I believe that you are using an old version of export_fig. The latest version should fix this issue for you.


[master] Upgrade to latest dependencies

Cron. -knative-prow-robot
/cc knative/networking-wg-leads
/assign knative/networking-wg-leads
Produced via:
./hack/update-deps.sh --upgrade --release v0.21
./hack/update-codegen.sh

Details:
bumping knative.dev/pkg 7101e9d...ca02ef7:
  > ca02ef7 Genreconciler properly generates reconciler for Resources with Status (# 2004)
  > 0d31134 Fix nil pointer panic in kvstore (# 2002)
  > 6040b3a improve client error message when our webhook doesn't have a valid cert (# 2000)

/lgtm
/approve


Github Actions build of ExoPlayer demo APK

Use case description
ExoPlayer demo app is very useful for verifying playback on a device, against specific reference / test streams etc. Sometimes it's useful to have the built app ready to be pulled and installed, without setting up local build and Android development environment every time.
It's also useful to validate external code pull requests to make sure they build and produce a valid binary.
Proposed solution
I propose adding basic Github Actions code into the repository here to quickly build debug-signed APK that would be available on at least build artifacts page ( see first screenshot below), possibly also on github "Releases" page ( see second screenshot below )
Actions takes very little code to set up and maintain, and it shouldn't get in a way anywhere in the rest of the codebase, e.g. it's quite non-intrusive. It really just needs this .github/workflows/ yaml file, i have an example here.
Alternatives considered
Not many, really :)
I'd be happy to PR and tweak this to expectations, if it would be found useful

That makes sense - thanks.


Minor Docker fixes & podman "rootless" container support (see comments)

I was intrigued by the Docker support and decided to try it.  When it comes to containers, I always prefer Podman running "rootless" rather than Docker running as root, so made a few changes to support this as well.
This was tested on both Podman and Docker.
Notes:

To run w/Podman, just set CONTAINER_ENGINE="podman"  (default is "docker") on build.sh and run.sh.  Otherwise, everything should hopefully run as before.
For fun, it was also built on an Raspberry Pi 4 w/debian 11.  No, I couldn't generate any images as the 8GB ram quickly filled up, but at least it built and got the web interface up and running.
I do have Podman running at full-speed with cuda on Podman on my machine, but I didn't want to dramatically break any of the current docker behavior and (and wasn't really sure how cuda was even working as-is).  My cuda Dockerfile uses the nvidia/cuda ubuntu image as a base, not the python base image.  If there is interest I can clean up the other Dockerfile and provide it.  I'm guessing it will work with Docker as well.)
To support Podman, the last commit f70fb02 pulls two Docker features that I really wanted to keep but don't yet have Podman support. These changes shouldn't break Docker's build, but may make subsequent builds less-efficient. I hope Podman 4.0 will support at least some of them.  See the notes in that commit.

Could someone with podman and/or docker test it?  Even if y'all don't want ALL the commits here, hopefully some of it will be of value for Docker users too.
Enjoy!

I was intrigued by the Docker support and decided to try it. When it comes to containers, I always prefer Podman running "rootless" rather than Docker running as root, so made a few changes to support this as well.

Docker can also be used rootless: https://docs.docker.com/engine/security/rootless/

I do have Podman running at full-speed with cuda on Podman on my machine, but I didn't want to dramatically break any of the current docker behavior and (and wasn't really sure how cuda was even working as-is).  My cuda Dockerfile uses the nvidia/cuda ubuntu image as a base, not the python base image.  If there is interest I can clean up the other Dockerfile and provide it.  I'm guessing it will work with Docker as well.)

People where already using this image with the cuda runtime: https://invoke-ai.github.io/InvokeAI/installation/040_INSTALL_DOCKER/

To support Podman, the last commit f70fb02 pulls two Docker features that I really wanted to keep but don't yet have Podman support. These changes shouldn't break Docker's build, but may make subsequent builds less-efficient. I hope Podman 4.0 will support at least some of them.  See the notes in that commit.

But it is rly no option to remove the build cache which is not only used by our CI/CD. Also it would be nice to have the linked copy job and not need to change default values for one build argument in three stages.

Does runpod use podman or docker?
I can try to pull the full image from docker hub and see what happens.  The build issues at least should not be a facgtor.  But I think I'll be stuck w/cpu until I can figure out how to use the existing image with the cuda runtime.
In the meantime, I may as well push the requested changes here, and you can decide if you want to use any parts of it.  If not, I can always just host a "Podman/CUDA"-specific version for my own use.  No big deal.

Does runpod use podman or docker?
I can try to pull the full image from docker hub and see what happens. The build issues at least should not be a facgtor. But I think I'll be stuck w/cpu until I can figure out how to use the existing image with the cuda runtime.

The latest tag is built for CUDA

[...] (and wasn't really sure how cuda was even working as-is). My cuda Dockerfile uses the nvidia/cuda ubuntu image as a base, not the python base image [...]

@fat-tire: The torch cuda-enabled distribution bundles the entirety of its CUDA dependencies (cuda,cudnn, cublas etc.) That is the reason for its ~1.8GB installed size for Linux (w/ cuda), vs mere megabytes for Mac (and Windows is somewhere in between). So you'll be happy to know that the massive nvidia/cuda image is redundant when using cuda-supporting torch - you can use any minimal image as needed. hope this saves you 2GB of base layer :)

Does runpod use podman or docker?

Runpod uses Kubernetes as far as I can tell (so its containerd most likely), but generally any OCI-compliant image should work equally well with either docker, podman, or k8s. Not sure of the rootless implications specifically, but could build-time compatibility issues (buildkit syntax support, etc) be avoided by building with docker and running with podman? Or does that undermine anything for your use case, specifically needing to run docker as root?

@fat-tire: The torch cuda-enabled distribution bundles the entirety of its CUDA dependencies (cuda,cudnn, cublas etc.) That is the reason for its ~1.8GB installed size for Linux (w/ cuda), vs mere megabytes for Mac (and Windows is somewhere in between). So you'll be happy to know that the massive nvidia/cuda image is redundant when using cuda-supporting torch - you can use any minimal image as needed. hope this saves you 2GB of base layer :)

Hey thanks for the response!  I did try building with "cuda" set as the flavor (manually, just to be sure), and did notice a ton of packages coming in and as you suggest at first I assumed I had everything I needed to run w/cuda (as the docs said).  But- no matter what I did and how I started it (again, this is with Podman) it kept coming up "cpu".  Going to the container shell, running python and importing torch and checking if gpu is available, I constantly got "False".  I thought maybe the problem was that the container didn't have accesss to the nvidia hardware, so I tried adding to the run.sh file all the
   --device /dev/dri \
     --device /dev/input \
     --device /dev/nvidia0 \
     --device /dev/nvidiactl \
     --device /dev/nvidia-modeset \
     --device /dev/nvidia-uvm \
     --device /dev/nvidia-uvm-tools \

stuff, and I played with giving it --privileged permissions, added the GPU_FLAGS=ALL, manually added the --gpus set to all as well.    Nothing was working.
As a last resort, I tried the nvidia/cuda repo as a base and boom it came up.  Well, once i also installed the nvidia-driver matching my host driver, that is.
As for your suggestion to not worry re building in Podman and just use the prebuilt images-- sure that would be fine and that way there's no need to pull all those things that aren't working in Docker yet.  I've yet to try pulling/running the pre-built docker-built container rather than building it myself but I'll give it a try in the next day or so and report back.
Thx again!

Had some time to do a Podman test with the officially prebaked image (latest tag) on docker hub-- here's the command to run it (note I'm using both ./mounts/outputs and ./mounts/invokeai for /data/outputs and /data just because I like to organize things in local folders rather than as a volume but this should make no difference really.)
Command to run:
GPU_FLAGS=all podman run --interactive --tty --rm --name=invokeai --hostname=invokeai \
--mount type=bind,source=./mounts/invokeai,target=/data \
--mount type=bind,source=./mounts/outputs,target=/data/outputs \
--publish=9090:9090 --user=appuser:appuser --gpus=all \
--device /dev/dri --device /dev/input --device /dev/nvidia0 \
--device /dev/nvidiactl --device /dev/nvidia-modeset \
--device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools \
--cap-add=sys_nice invokeai/invokeai

Note I'm including explicit access to the nvidia devices just in case it's needed when running rootless (it's what I use when I'm running the nvidia/cuda image so I didn't want to make any changes since I know they provide access the container needs,  For all I know --gpus=all might be enough but I didn't want to introduce anything different from the known-working command).
Running this command downloads the image from docker hub and runs, resulting in the (expected) error:
PermissionError: [Errno 13] Permission denied: '/data/models'
This was expected due to the uid/guid ownership issue discussed above.
The workaround was to run this 1x.  It can only be run after the image is downloaded and the files/volumes are created:
#!/bin/bash
CONTAINER_ENGINE="podman"

# Podman only:  set ownership for user 1000:1000 (appuser) the right way
# this fixes PermissionError: [Errno 13] Permission denied: '/data/models'
if [[ ${CONTAINER_ENGINE} == "podman" ]] ; then
   echo Setting ownership for container\'s appuser on /data and /data/outputs
   podman run \
      --mount type=bind,source="$(pwd)/mounts/invokeai",target=/data \
      --user root --entrypoint "/bin/chown" "${CONTAINER_IMAGE:-invokeai}" \
      -R 1000:1000 /data
   podman run \
      --mount type=bind,source="$(pwd)"/mounts/outputs,target=/data/outputs \
      --user root --entrypoint "/bin/chown" "${CONTAINER_IMAGE:-invokeai}" \
      -R 1000:1000 /data/outputs
fi

Now that the mounted directories are correct, the above podman run command can work, the various models are loaded and the web server starts.
The problem though--
$ ./run.sh
Setting ownership for container's appuser on /data and /data/outputs
* Initializing, be patient...
>> Initialization file /data/invokeai.init found. Loading...
>> Internet connectivity is True
>> InvokeAI, version 2.3.1
>> InvokeAI runtime directory is "/data"
>> GFPGAN Initialized
>> CodeFormer Initialized
>> ESRGAN Initialized
>> Using device_type cpu
>> xformers not installed
>> Initializing NSFW checker

CPU.   not CUDA.
From the container:
root@invokeai:/usr/src# python3
Python 3.9.16 (main, Feb  9 2023, 05:40:23)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False

Also:
oot@invokeai:/usr/src# pip list | grep torch
clip-anytorch               2.5.0
pytorch-lightning           1.7.7
torch                       1.13.1
torch-fidelity              0.3.0
torchdiffeq                 0.2.3
torchmetrics                0.11.1
torchsde                    0.2.5
torchvision                 0.14.1
root@invokeai:/usr/src#  pip list | grep nvidia
nvidia-cublas-cu11          <IP_ADDRESS>
nvidia-cuda-nvrtc-cu11      11.7.99
nvidia-cuda-runtime-cu11    11.7.99
nvidia-cudnn-cu11           <IP_ADDRESS>

Contrast with my nvidia/cuda image-based container:
root@invokeai:/usr/src# python3
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True

and
* Initializing, be patient...
>> Initialization file /data/invokeai.init found. Loading...
>> Internet connectivity is True
>> InvokeAI, version 2.3.1
>> InvokeAI runtime directory is "/data"
>> GFPGAN Initialized
>> CodeFormer Initialized
>> ESRGAN Initialized
>> Using device_type cuda

and
root@invokeai:/usr/src# pip list | grep torch
clip-anytorch               2.5.0
pytorch-lightning           1.7.7
torch                       1.13.1+cu117
torch-fidelity              0.3.0
torchdiffeq                 0.2.3
torchmetrics                0.11.1
torchsde                    0.2.5
torchvision                 0.14.1+cu117
# pip list | grep nvidia
root@invokeai:/usr/src#

Most obviously torch/torchvision have cu117 on my version but for some reason this didn't make it to the upstream container (or it wasn't properly d/l'd if it was supposed to be).
Thoughts?

For a moment I thought perhaps I was just using the wrong tag and getting the cpu version as a result,, but trying the `:main-cuda tag came up "cpu" as well:
>> Using device_type cpu
If I pip uninstall torch torchvision and then pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu117 I can get the right version of torch in there.  But it's not there by default.
root@invokeai:/data# pip list | grep cu117
torch                       1.13.1+cu117
torchaudio                  0.13.1+cu117
torchvision                 0.14.1+cu117

Unfortunately, this didn't work.  I also tried installing cu113 (no dice) and the cuda package from nvida directly to the base image per nvidia's instructions.
This didn't work either.

I don't know podman at all, sadly, just looked over some of the docs... but this is very curious indeed if the nvidia/cuda based images work correctly for you, but not an image with torch..+cu117 installed. Because as I mentioned above, Torch bundles all of the required cuda dependencies and works out of the box - that's been well tested albeit in Docker and Kubernetes.
I see you're mapping quite a lot of devices in one of the above commands. Are you certain to be running the invoke image identically to the known working nvidia/cuda?
Curious as to the mininal set of podman arguments required to get the nvidia/cuda image to see the GPU, and whether the same works with the invoke image... does podman run --device nvidia.com/gpu0  work for you? the docs seem to suggest this is the correct way

Thanks for taking a look.  Yeah I'm sure I am running it the same way as I copy/pasted the RUN command from the nvidia/cuda run.sh, minus some QT_FONT_DPI cruft I had that I'm pretty sure isn't needed.  I literally added an echo to the front of the RUN command, then used the output of that with the invokeai prebaked image so I knew I was running it just as it was running the run.sh file.  Also tried prepending with GPU_FLAGS=all.  It always comes up "cpu" no matter what I seem to do.
Trying with --device nvidia.com/gpu0 returned a Error: stat nvidia.com/gpu0: no such file or directory failure.  I hadn't seen this syntax before, but didn't seem to work.
Curious why would the main-cuda tagged image includes versions 1.13.1 and 0.14.1 of torch/torchvision by default rather than the cu117 versions?  Again, after running, then manually removing these versions and re-installing with pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu117 they did appear in pip, but again, no dice getting "cuda" to appear when running invokeai --web right afterwards:
appuser@invokeai:/usr/src$ pip list | grep torch
clip-anytorch               2.5.0
pytorch-lightning           1.7.7
torch                       1.13.1+cu117
torch-fidelity              0.3.0
torchdiffeq                 0.2.3
torchmetrics                0.11.1
torchsde                    0.2.5
torchvision                 0.14.1+cu117
appuser@invokeai:/usr/src$ python
Python 3.9.16 (main, Feb  9 2023, 05:40:23)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False
>>> quit()

Thanks for taking a look. Yeah I'm sure I am running it the same way as I copy/pasted the RUN command from the nvidia/cuda run.sh, minus some QT_FONT_DPI cruft I had that I'm pretty sure isn't needed. I literally added an echo to the front of the RUN command, then used the output of that with the invokeai prebaked image so I knew I was running it just as it was running the run.sh file. Also tried prepending with GPU_FLAGS=all. It always comes up "cpu" no matter what I seem to do.
Trying with --device nvidia.com/gpu0 returned a Error: stat nvidia.com/gpu0: no such file or directory failure. I hadn't seen this syntax before, but didn't seem to work.
Curious why would the main-cuda tagged image includes versions 1.13.1 and 0.14.1 of torch/torchvision by default rather than the cu117 versions? Again, after running, then manually removing these versions and re-installing with pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu117 they did appear in pip, but again, no dice getting "cuda" to appear when running invokeai --web right afterwards:
appuser@invokeai:/usr/src$ pip list | grep torch
clip-anytorch               2.5.0
pytorch-lightning           1.7.7
torch                       1.13.1+cu117
torch-fidelity              0.3.0
torchdiffeq                 0.2.3
torchmetrics                0.11.1
torchsde                    0.2.5
torchvision                 0.14.1+cu117
appuser@invokeai:/usr/src$ python
Python 3.9.16 (main, Feb  9 2023, 05:40:23)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False
>>> quit()

I don't know if it's still necessary, but the last time I needed to use graphics acceleration inside a podman container (fedora silverblue, to use an old version of invoke) I had to install the nvidia-container-runtime package (which only has a version for RHEL 8.3, but it works on fedora) on the host system, plus the respective gpu drivers inside the container as well. Also, I had to edit /etc/nvidia-container-runtime/config.toml and set no-cgroups = true.
I don't have an Nvidia gpu anymore, I don't have anything to test with, but maybe this is the way to go.

Well I have some good news.. I got it running from the pre-built on podman/rootless!  What's nice is that the image is significantly smaller, even with the nvidia drivers installed, than the previous container I was using, and rebuilding will be VERY fast now.
Steps:

Modify Dockerfile to start FROM the upstream prebuilt image.  Then apt-get install curl and kmod packages, then curl/download and install the Nvidia driver into the container.  (The container driver version  must match the host driver version.) Then clean up the installer.
Create a new build.sh to determine in the correct nvidia driver version since it will change in future, then build the image with buildah bud, passing in the driver info.
Create a run.sh script which MUST explicitly allow access to the various nvidia devices (or else nvidia-smi gives a "can't connect" error.  Before every run it makes sure the mounted volumes/binds have correct 1000:1000 ownership for the local user to avoid fixes PermissionError: [Errno 13] Permission denied: '/data/models' podman errors.

I have all of the above working with the latest "3.0.0+a0" version.
Does anyone want this code... or what should I do with it?  Since the real work is done in the prebuilt, these are all very small, simple files.  With a $CONTAINER_ENGINE flag, it could be integrated into the source in this repo.. but I can also make a dedicated podman_invokeai repository solely for this purpose (it would be maybe 4 files and a README).  I don't know if there other other podman users who would be interested or not.
I didn't have to touch /etc/nvidia-container-runtime/ anything, I didn't even have to install the nvidia-container-runtime package on the host at all, though I am running the 525 driver, so I think it's included now.

I don't know if it's still necessary, but the last time I needed to use graphics acceleration inside a podman container (fedora silverblue, to use an old version of invoke) I had to install the nvidia-container-runtime package (which only has a version for RHEL 8.3, but it works on fedora) on the host system, plus the respective gpu drivers inside the container as well. Also, I had to edit /etc/nvidia-container-runtime/config.toml and set no-cgroups = true.
I don't have an Nvidia gpu anymore, I don't have anything to test with, but maybe this is the way to go.

Great to hear you got it working! If there was a way to run this without installing the nvidia driver into the image, that would be ideal, in my opinion. Generally you really want to be using the driver that is already loaded by the kernel. But perhaps that's a hard limitation due to podman's rootless nature - i'm not sure.
I think your work here is valuable for supporting users who wish to run in a rootless container. Is there a way to do this without maintaining a separate Dockerfile and build/run scripts? Will leave it up to @mauwii to make the call on how to proceed next.

I already addressed a lot of changes (see the 11 unresolved conversations) and made clear that I would not want to remove the caching 😅

Thanks for addressing the changes in the other convos--
You wouldn't have to remove the caching as podman does now run with the prebaked image (built with caching, --linking etc).  My working Dockerfile builds on top of that by adding the NVIDIA drivers on top.  It basically looks like this:
FROM docker.io/invokeai/invokeai:main-cuda

ARG ARCH=x86_64
ARG NVIDIA_VERSION=525.85

USER 0
RUN apt update && apt install -y kmod curl
RUN cd /tmp && curl https://us.download.nvidia.com/XFree86/Linux-${ARCH}/${NVIDIA_VERSION}/NVIDIA-Linux-${ARCH}-${NVIDIA_VERSION}.run -o /tmp/NVIDIA-Linux-${ARCH}-${NVIDIA_VERSION}.run \
       && bash /tmp/NVIDIA-Linux-${ARCH}-${NVIDIA_VERSION}.run --no-kernel-module --no-kernel-module
-source --run-nvidia-xconfig --no-backup --no-questions --accept-license --ui=none \
       && rm -f /tmp/NVIDIA-Linux-${ARCH}-${NVIDIA_VERSION}.run \
       && rm -rf /tmp/*
RUN apt remove --purge kmod curl -y && apt-get clean

The build.sh looks something like:
#!/bin/bash

CONTAINER_BUILD="buildah bud"
TAG=latest

if [ -z "$RESOLVE_NVIDIA_VERSION" ]; then
   export NVIDIA_VERSION=`nvidia-smi --query-gpu=driver_version --format=csv,noheader`
else
   export NVIDIA_VERSION="${RESOLVE_NVIDIA_VERSION}"
fi

${CONTAINER_BUILD} -t "invokeai:${TAG}" -t "invokeai" --build-arg ARCH=`uname -m` --build-arg NVIDIA_VERSION="${NVIDIA_VERSION}"

As you can see it passes the current NVIDIA driver and ARCH to the build command.  The container has to match the host, so this may be an issue for making any generic image for rootless.
I did try avoiding installing the nvidia-driver and instead tried using only the nvidia-runtime-container package in the Dockerfile.  (Notes for automating that installation are here.)
RUN apt update && apt install gpg curl -y
RUN distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
RUN apt update && apt install -y nvidia-container-runtime
RUN apt remove --purge kmod curl gpg -y && apt-get clean

But this did test NOT work, at least not for me.
The run.sh is pretty much the same, except these --device lines were needed and the couple lines in this PR that verify the user permissions of the mount/bind volumes/directories, which I guess is a rootless thing to make sure that the userid & usergroup for the user running in the container can access the volume/directories.
  --device /dev/dri --device /dev/input --device /dev/nvidia0 \
  --device /dev/nvidiactl --device /dev/nvidia-modeset \
  --device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools \

That's basically all I got! Just happy it's working now, and, while I don't know what if anything above can be integrated, hopefully someone else running rootlessly can find value in it.

Hey it's been a few weeks and I'm inclined to close this just to not clutter up the PR area.  Is there anything anyone wants from here?   I've got podman running locally w/the method outlined above-- one thing I did add recently is:
  --env TRANSFORMERS_CACHE=/data/.cache \

to the podman run command, as it wasn't set anywhere else.
I've also noticed I get this:
 Server error: [Errno 18] Invalid cross-device link:

When trying to delete an image via the trash can icon in the web ui, because image files can't apparently be moved from  /data/outputs/ to /data/.Trash-1000/files/.  I had this problem previously.  Dunno if it's podman only or an issue with the mounts or what.

@fat-tire I'm going to close this PR as outdated - we'll have it for reference if/when implementing Podman support, as discussed in #3587


Browser extension can't download MIDI, 404 error

I thought it was just on one score, but checking other one returns same result :( I can't really give any of useful information, and sorry about that

The file is downloading correctly for me, this score: https://musescore.com/user/11512151/scores/5581047, right?
Please try again and see if it works

I run the Tampermonkey with the Brave Browser on PC.
Specifically, I block with very specific settings. Not allowing anything to track me.
I do it in the order that follows:

Open the desires musescore site
Select the Brave logo icon in the right side of the search bar.
In advanced controls: [Block trackers & ads], [Upgrade connection to HTTPS], [Do not touch the block scripts yet], [Block Fingerprinting], [Block all cookies], and [Forget me when I close this site] on.

After that, continue with the following as fast as possible.

Turn on Tampermonkey with the loaded script (using the latest version that came out 3 weeks ago)
Reopen the Brave logo in the search bar and turn on [Block all scripts] until the screen turns white.
Once the screen is all white, turn off [Block all scripts] and select Midi.
The MIDI should be downloaded in a few seconds.

Thanks for the information. Maybe one of MuseScore's backend servers was added to a tracker list? I use uBlock Origin which also blocks trackers but it works for me. I'll contact the others who reported this problem on Discord and see what their browser situation is

You're welcome. I'm so glad I was able to help! :3

For others: You can get the data on about:support

Name: Firefox
OS Version: Linux 6.7.2-arch1-1 # 1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 19:10:20 +0000
Version: 122.0
Build ID:<PHONE_NUMBER>1424
Distribution ID: archlinux
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0
Safe Mode: false
Memory Size (Bytes):<PHONE_NUMBER>0
Disk Available (Bytes):<PHONE_NUMBER>8
Support URL: https://support.mozilla.org/1/firefox/122.0/Linux/en-US/
OS Theme: Breeze / Adwaita
Number of Total Windows: 1
Number of Fission Windows: 1
Number of Remote Windows: 1
Fission Auto Start: true
Fission Decision Status: enabledByDefault
Remote Auto Start: true
Policies Status: 0
Key Location Service Google Found: true
Key Safebrowsing Google Found: true
Key Mozilla Found: true

Extensions:

7TV Nightly
base64 decoder
Bitwarden - Free Password Manager
ClearURLs
Dark Reader
HistoryBlock
I still don't care about cookies
Return YouTube Dislike
SponsorBlock for YouTube - Skip Sponsorships
Tampermonkey
TTV LOL PRO
uBlock Origin
User-Agent Switcher and Manager
YouTube NonStop

Some possible culprits based on messages on Discord:
Firefox itself doesn't work
For chromium, it's hit or miss, I can't reproduce it

Reopening to track PDF download issue after fallback used in https://github.com/LibreScore/dl-librescore/commit/fc99d7fd9b32f0eeaec79a657268f6060bdb341b patched

Fixed in https://github.com/LibreScore/dl-librescore/commit/772e7b37a13c93244077149c3c24378ac0099d86


[T4A4][T10-B2] Li Zhiyuan

interface

Ready for review

@lzyuan168 Some comments added. Please close the PR after reading comments.


Too many false positives

I have created the following rule for generic secrets:
[[rules]]
    description = "Generic Credential"
    regex = '''(?i)(dbpass|dbuser|secret|key|pass|guid|pw|token)(.{0,40})?([0-9a-zA-Z-_\/+!{}/=]{4,120})'''
    tags = ["Generic"]
    [[rules.Entropies]]
        Min = "3.9"
        Max = "5.5"
        Group = "2"

The following two configurations are used for testing the rule (contains dummy secrets):
# test real secrets
env:
  PASSWORD: dY^5Ke76gdSiF
  OPSGENIE_API_KEY_URL: https://api.opsgenie.com/v1/json/cloudwatch?apiKey=182663ax-3ccb-20c3-1097-3ol15d0wfu45
  OKTA_CLIENT_SECRET: LjBlsCPNJNDQ7gazunVyZu3f2y5sF69RJmdxec6v
  ANODOT_TOKEN: uvqcblfxi1d3ohvwjftgb59e3mba1x98
  GITHUB_TOKEN: 1mlcyz92suir2qzfimaeekzb1goo5lveam15hebm

# test false positives
env:
  PASSWORD: test
  OPSGENIE_API_KEY: test
  OPSGENIE_API_KEY_URL: https://api.opsgenie.com/v1/json/cloudwatch?apiKey=test
  OKTA_CLIENT_SECRET: test
  ANODOT_TOKEN: $(echo $ASSUME_ROLE_OUTPUT | jq -r .Credentials.SessionToken)
  GITHUB_TOKEN: os.getenv('AWS_SECRET_ACCESS_KEY')

When running the rule on the second configuration, the rule alerts on false positives:
{
	"line": "          OPSGENIE_API_KEY_URL: https://api.opsgenie.com/v1/json/cloudwatch?apiKey=test",
	"lineNumber": 17,
	"offender": "KEY_URL: https://api.opsgenie.com/v1/json/cloudwatch",
	"offenderEntropy": 4.325,
	"commit": "",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "Generic Credential",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": ".",
	"date": "0001-01-01T00:00:00Z",
	"tags": "Generic"
}
{
	"line": "          ANODOT_TOKEN: $(echo $ASSUME_ROLE_OUTPUT | jq -r .Credentials.SessionToken)",
	"lineNumber": 19,
	"offender": "TOKEN: $(echo $ASSUME_ROLE_OUTPUT | jq -r .Credentials",
	"offenderEntropy": 4.465,
	"commit": "",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "Generic Credential",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": ".",
	"date": "0001-01-01T00:00:00Z",
	"tags": "Generic"
}
{
	"line": "          GITHUB_TOKEN: os.getenv('AWS_SECRET_ACCESS_KEY')",
	"lineNumber": 20,
	"offender": "TOKEN: os.getenv('AWS_SECRET_ACCESS_KEY",
	"offenderEntropy": 4.123,
	"commit": "",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "Generic Credential",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": ".",
	"date": "0001-01-01T00:00:00Z",
	"tags": "Generic"
}

Raising the min entropy value would cause the rule to not detect all secrets in the configuration with the "real" secrets.
In addition, it seems that the rule is calculating the entropy for the key as well, even though i've used Group = "2" within the rule.
What can be done in order to both find the real secrets and not find false positives?
Thanks

@dudicoco you could add an allowlist to the rule that ignores lines with os.getenv, echo, jq -r, https://api.opsgenie.com
[[rules]]
    description = "Generic Credential"
    regex = '''(?i)(dbpass|dbuser|secret|key|pass|guid|pw|token)(.{0,40})?([0-9a-zA-Z-_\/+!{}/=]{4,120})'''
    tags = ["Generic"]
    [[rules.Entropies]]
        Min = "3.9"
        Max = "5.5"
        Group = "2"
    [[rules.Allowlist]]
        regexes=['''os.getenv''','''echo''','''jq -r''']

Basically, it's hard to create a generic rule and not get a ton of false positives

In addition, it seems that the rule is calculating the entropy for the key as well, even though i've used Group = "2" within the rule.

What makes you say that?

updated config provides better rules and advice on how to write rules. closing this issue


remove Abstracttest->startDate member variable

reuse Timer functionality to get information about elapsed time instead of the local variable

removed code in 6.0-beta3+. we agreed to remove elapsed from local emailable report at all


[FEATURE] boolean inversion generator

Describe the feature
A boolean toggle setter, that generates a function that, when called, flips the boolean's state. This can be done either with a field on the @Setter (maybe @Setter(generateInverter = true)), or with a separate  annotation extending Setter.
Lomboked Code:
public class Main {

	@lombok.Setter
	@lombok.Getter
	@Lombok.Inverter
	private boolean test;

	public static void main(final String[] args) {

	}
}

Or, alternatively
public class Main {

	@lombok.Setter
	@lombok.Getter(generateInverter = true)
	private boolean test;

	public static void main(final String[] args) {

	}
}

Delomboked Code:
public class Main {

	private boolean test;

	public static void main(final String[] args) {

	}

	public boolean isTest() {
		return test;
	}

	public void setTest(final boolean $test) {
		test = $test;
	}

	public void setTest() {
		test = !test;
	}

}

Describe the target audience
This would be useful for setting flags. Especially if you are only going to set the flag once, and you know the predefined value, calling the inverter function is a bit cleaner. Obviously it is also useful if you want to flip the value of the boolean, as instead of doing main.setTest(!main.getTest()) you can just do main.setTest(). Perhaps the overloaded setTest may not be the best name, I am open to ideas for other names (perhaps flipTest or allowing you to customize the name?).
Additional context
None.

Please keep your de-constructive opinions in your head

Napkin, keep your bad ideas in our discord server .

Please keep your de-constructive opinions in your head

It's napkin we're talking about here, we all do this in the sk1er discord.

can you all not please

It's napkin we're talking about here, we all do this in the sk1er discord.
The fact remains that you're ruining his thread, so cut it out. Let the repo maintainers decide

As a maintainer, first, the original request was valid an well-formatted, according to our template instructions. It described the idea and contained examples in "lombok" style and plain java, and it tried to explain the target audience.
Regarding the feature itself, I have some comments:

Since it would only apply to boolean fields, making it a property of @Setter is a bit strange. Especially since this is not a very common thing to do.
I think that the suggested naming pattern setXxxx() (without parameters) is misleading. I would prefer toggleXxx() or, as suggested in the proposal flipXxx(). @lombok.Toggle or @lombok.Flip would then be a better annotation.
In my personal experience, I haven't encountered the need for such a feature enough to call it "boilerplate". I don't ythink it is worth the added maintenance load to the lombok code base.

I'm rejecting the request because the expected added benefits do not outweigh the added maintenance load.

Alright. Thank you for the feedback.

@SirNapkin1334 yup. Feel free to lord over your friends on that discord server that whilst your idea went down in flames, they were glorious. I'm going through a bunch of feature requests and this one is stated more plainly and clearly than any of em, so, I'll gladly read through more ideas like this :)


Update to latest golang patch.

What this PR does / why we need it:
Update to latest golang patch.
Which issue(s) this PR fixes:
None.
Special notes for your reviewer:
Release note:

Can you also update https://github.com/gardener/aws-custom-route-controller/blob/main/.ci/pipeline_definitions#L21

Can you also update https://github.com/gardener/aws-custom-route-controller/blob/main/.ci/pipeline_definitions#L21

Good catch. Updated as requested.


Problems generating certificates

Hello, I am attempting to use your container, but am running into the following problems:

It does not seem to recognize domains.conf with only one line entry.
It will revoke the certs immediately after under unspecified circumstances.

Logs from fresh start (with account info already added, and domains.conf populated):
2018-05-18 02:26:53,143 CRIT Supervisor running as root (no user in config file)
2018-05-18 02:26:53,143 WARN No file matches via include "/etc/supervisord.d/*"
2018-05-18 02:26:53,207 INFO RPC interface 'supervisor' initialized
2018-05-18 02:26:53,207 INFO supervisord started with pid 23
2018-05-18 02:26:54,219 INFO spawned: 'watch-domains' with pid 26
2018-05-18 02:26:54,231 INFO spawned: 'crond' with pid 27
#### Registering Let's Encrypt account if needed ####
2018-05-18 02:26:55,271 INFO success: watch-domains entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-18 02:26:55,271 INFO success: crond entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Saving debug log to /var/log/letsencrypt/letsencrypt.log
There is an existing account; registration of a duplicate account with this command is currently unsupported.
#### Creating missing certificates if needed (~1min for each) ####
### Revoke and delete certificates if needed ####
### Reloading supervisord configuration ###

The certbot log output for the above:
2018-05-18 02:26:56,159:DEBUG:certbot.main:certbot version: 0.23.0
2018-05-18 02:26:56,163:DEBUG:certbot.main:Arguments: ['-n', '--agree-tos', '-m'<EMAIL_ADDRESS>'--server', 'https://acme-v02.api.letsencrypt.org/directory']
2018-05-18 02:26:56,164:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-05-18 02:26:56,214:DEBUG:certbot.log:Root logging level set at 20
2018-05-18 02:26:56,216:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log

domains.conf:
*.domain.net domain.net

If I put it at the start, and put junk data (literally "junk.") at the end of domains.conf, it will work:
### Reloading supervisord configuration ###
#### Registering Let's Encrypt account if needed ####
Saving debug log to /var/log/letsencrypt/letsencrypt.log
There is an existing account; registration of a duplicate account with this command is currently unsupported.
#### Creating missing certificates if needed (~1min for each) ####
>>> Creating a certificate for domain(s): -d *.domain.net -d domain.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for domain.net
dns-01 challenge for domain.net
Output from authenticator.sh:
Arguments: Namespace(action='create', auth_token=None, auth_username=None, content='{{token}}', delegated=None, domain='domain.net', identifier=None, log_level='DEBUG', name='_acme-challenge.domain.net.', priority=None, provider_name='cloudflare', ttl=None, type='TXT')
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "GET /client/v4/zones?name=domain.net&status=active HTTP/1.1" 200 None
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "POST /client/v4/zones/{{token}}/dns_records HTTP/1.1" 200 None
create_record: True

Output from authenticator.sh:
Arguments: Namespace(action='create', auth_token=None, auth_username=None, content='{{token}}', delegated=None, domain='domain.net', identifier=None, log_level='DEBUG', name='_acme-challenge.domain.net.', priority=None, provider_name='cloudflare', ttl=None, type='TXT')
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "GET /client/v4/zones?name=domain.net&status=active HTTP/1.1" 200 None
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "POST /client/v4/zones/{{token}}/dns_records HTTP/1.1" 200 None
create_record: True

Waiting for verification...
Cleaning up challenges
Output from cleanup.sh:
Arguments: Namespace(action='delete', auth_token=None, auth_username=None, content='{{token}}', delegated=None, domain='domain.net', identifier=None, log_level='DEBUG', name='_acme-challenge.domain.net.', priority=None, provider_name='cloudflare', ttl=None, type='TXT')
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "GET /client/v4/zones?name=domain.net&status=active HTTP/1.1" 200 None
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "GET /client/v4/zones/{{token}}/dns_records?per_page=100&type=TXT&name=_acme-challenge.domain.net&content={{token}} HTTP/1.1" 200 None
list_records: [{'type': 'TXT', 'name': '_acme-challenge.domain.net', 'ttl': 3600, 'content': '{{token}}', 'id': '{{token}}'}]
delete_records: ['{{token}}']
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "DELETE /client/v4/zones/{{token}}/dns_records/{{token}} HTTP/1.1" 200 None
delete_record: True

Output from cleanup.sh:
Arguments: Namespace(action='delete', auth_token=None, auth_username=None, content='{{token}}', delegated=None, domain='domain.net', identifier=None, log_level='DEBUG', name='_acme-challenge.domain.net.', priority=None, provider_name='cloudflare', ttl=None, type='TXT')
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "GET /client/v4/zones?name=domain.net&status=active HTTP/1.1" 200 None
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "GET /client/v4/zones/{{token}}/dns_records?per_page=100&type=TXT&name=_acme-challenge.domain.net&content={{token}} HTTP/1.1" 200 None
list_records: [{'type': 'TXT', 'name': '_acme-challenge.domain.net', 'ttl': 3600, 'content': '{{token}}', 'id': '{{token}}'}]
delete_records: ['5e2e89424cb4e9c3f21c6d08f36cd34a']
Starting new HTTPS connection (1): api.cloudflare.com
https://api.cloudflare.com:443 "DELETE /client/v4/zones/{{token}}/dns_records/{{token}} HTTP/1.1" 200 None
delete_record: True

Running deploy-hook command: deploy-hook.sh
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/domain.net/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/domain.net/privkey.pem
   Your cert will expire on 2018-08-16. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

### Revoke and delete certificates if needed ####
### Reloading supervisord configuration ###

And it does not revoke!
(I also notice that the arguments section in the log under authenticator.sh and cleanup.sh show "None" for cloudflare?  Or is that just to not log the token?)
It looks as if it does not properly handle the last line in the file, or the only line if there's only one line.
Please let me know what debug information I can gather for you.

In addition, while using the workaround above, I receive the following error after it runs its scheduled checks:
Plugins selected: Authenticator manual, Installer None
Cert not yet due for renewal
Keeping the existing certificate

-------------------------------------------------------------------------------
Certificate not yet due for renewal; no action taken.
-------------------------------------------------------------------------------
>>> Watching certificate for main domain domain.net: autocmd config nginx:nginx -s reload executed when certificate is changed.
### Revoke and delete certificates if needed ####
### Reloading supervisord configuration ###
2018-05-18 03:31:53,216 INFO spawned: 'domain.net_autocmd-containers' with pid 75
domain.net_autocmd-containers: added process group
2018-05-18 03:31:54,223 INFO success: domain.net_autocmd-containers entered RUNNING state, process has stayed up for > than 1 second                                                                     s (startsecs)
Launch renew test
Saving debug log to /var/log/letsencrypt/letsencrypt.log
The following error was encountered:
[Errno 5] I/O error
Either run as root, or set --config-dir, --work-dir, and --logs-dir to writeable paths.

Could you please post the parameters you've used to start your docker container? Maybe that helps.

Certainly
docker create --name=letsencrypt --restart always -v letsencrypt:/etc/letsencrypt -v /var/run/docker.sock:/var/run/docker.sock --env<EMAIL_ADDRESS>--env 'LEXICON_PROVIDER=cloudflare' --env<EMAIL_ADDRESS>--env 'LEXICON_CLOUDFLARE_TOKEN={{token}}' adferrand/letsencrypt-dns

Then docker start letsencrypt

OK I will try to reproduce your error tomorrow.

Hello, I think I figured out the problem ! As for some other people, your domains.conf do not end with a newline. Correction is on its way, I think it will correct the two issues, which are related.

Version 2.2.0 released !


More details in progress callback

It would be cool if the progress callback could add more details:

An event for when a command has just started (including an ID of the new command, the target file, and full command line)
An event that fires when a command emits some stdout or stderr (tagged with the ID of the command)
An event for when a command has completed execution (the ID of the command that just finished, including the exit code)
Similar events to the above for "traced" calls
Events for all putNormal, putLoud, putQuiet
Any others?

This could be used to implement an alternative UI for running shake. One idea I have is an interactive ncurses UI that shows you the currently running jobs, with a realtime timer next to each one showing how long it's been running, and the ability to click a job to view the full command line, etc... Also a web UI could be created that streams the events through a websocket and displays a javascript/CSS UI similar to the above description (would be awesome for CI servers)
Of course these UIs would not be built into shake itself, they would be separate libraries that hook into the enhanced progress callback.

I added shakeTrace which gives you the start/stop of traced, and since cmd calls traced you can get info that way. You already have shakeOutput to get messages like putNormal etc. With all that combined, you can write something fancier on top, and I have done so - see https://asciinema.org/a/228101. That is almost entirely outside of the core of Shake (the only integration point is the argument flag).
I'm closing this as all the low hanging fruit is done, and you can do something. If you put together a GUI using what's here, I'd love to see it, and we can see if there is anything in particular that might be worth adding.

That asciinema link doesn't seem to be working :frowning_face:

https://asciinema.org/a/228128 - sorry, I uploaded a better version, but forgot I'd pasted it to you


kali xorg nivdia

image washing out
kali needs
sudo apt-get install dbus-x11 for xiwi to start
using delete backports in xorg target
Installed property nvidia drivers and work with xiwi

crouton doesn't support proprietary nvidia drivers.  Please either move this to your fork or keep the discussion in "discussions"


create schema migration provenance function

Since Project:M36 stores a complete history of the data and metadata (schema) in its transaction graph, a function which extracts all DDL between two points in the graph could be easily generated. In addition, comparing schemas from two branches which shared a common ancestor would be valuable.

This feature is inspired by existing tools which assist in schema migration poorly.
Also add documentation on schema migration tips.


[core] fix our urljoin function for empty urls

RIght now, azure-core is adding an extra slash during url join if you're stub url is jsut query params. Refactor this code to not add a "/" when we don't want it
https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/azure/core/pipeline/transport/_base.py#L124

we decided to not do this


Use uio.h on tizenrt

libtuv-DCO-1.0-Signed-off-by: Daeyeon Jeong<EMAIL_ADDRESS>

LGTM


Corrupt JSON Response Text on Mac (Possible Text Encoding Problem)

I see that there were threads in 2015 tagged "utf-8-issues" but it's unclear whether they were ever resolved. (Apologies in advance, I'm an amateur and don't have a great understanding of text encoding.) I wanted to see if there have been any updates, or anyone has found any solutions for handling characters that get garbled on Macs.
Basically, here's my issue: Used on a PC, my program, which uses VBA-Web to interact with SurveyMonkey's API, works perfectly. However, on Macs, certain characters in JSON responses are getting garbled. For example:
What should be an apostrophe is ,Äô on Mac
An ñ becomes √±
And sometimes, a space after a period becomes ¬†
If anyone has a solution for this it would be a great help. I so appreciate the work you have done putting VBA-Web together!!

Thanks for reporting this issue. Unfortunately, there are persistent UTF-8 issues with Mac support. It'll require a lot of deep work, but it's on the roadmap.

Would the above also apply when you're forced to use a : in the parsed URL. For a specific server I need a :, but when adding it as a parameter, it returns a %3A.
I have only two occurences where this happens, so would it be possible to replace the string with the appropriate characters?


Update tests

Unit tests currently fail; I believe it's because there are two log files being created now, the standard log file and the friendly log file. Verify this, then update tests.

[ ] Assert that one of the files is 'friendly_summary.html'.
[ ] Make a more specific assertion about which log files are found.


New Trick in Person reID

Hi, I am creating this issue to share a new trick in person reID, the arcface loss.
As you mentioned in the README, I re-implemented a tiny repo by following your paper and code before. Recently, I test reID performance using SOTA metric loss in face recognition. Experimental result shows that arcFace loss can converge faster (around 50 epochs vs 100 epochs) and achieve slightly better performance (around 0.7%/0.9% improvement in rank1/mAP) than softmax+triplet loss+center loss.
Looking forward to your response and discussion.

It's a very interesting result because we do some experiments about some face loss, such as cosface, arcface, but the result is a little bit bad. I will check your result first and we can discuss it later.

I do some experiments for arcface and I get the result below

loss
market1501
dukemtmc

cosface
93.6 (83.7)
84.6 (69.8)

arcface
93.6 (84.1)
84.9 (70.7)

Actually, it is lower for the baseline(softmax+triplet), especially on duke

loss
market1501
dukemtmc

baseline
94.6 (86.1)
87.3 (76.0)

Could you shed some light on this situation?

I do some experiments for arcface and I get the result below
loss	market1501	dukemtmc
cosface	93.6 (83.7)	84.6 (69.8)
arcface	93.6 (84.1)	84.9 (70.7)
Actually, it is lower than the baseline(softmax+triplet), especially mAP on duke
loss	market1501	dukemtmc
baseline	94.6 (86.1)	87.3 (76.0)
Could you shed some light on this situation? @lulujianjie

Hi, Xingyu, which data sampler did you use in your experiment? In my experiment, triplet sampler, namely fixed number of identities and images per batch (8x8), achieves better results as I mentioned. The arcface using the default sampler achieves slightly lower performance than the baseline.

I just follow your experiment setting, triplet sampler with 8x8. If I change it to 4x16, it will lead to a poor result.
BTW, I notice you change the bnneck to affine=False.
Anything else I didn't notice?

I also modified the data augmentation method in my repo. But it might not influence the results since the comparison experiments follow the same settings. I will test the arcface using your origin baseline code and report the result later.

I noticed your result is 84.3 for mAP. Actually, it is lower than baseline result 86. Besides, mAP result is very bad on duke.

I noticed your result is 84.3 for mAP. Actually, it is lower than baseline result 86. Besides, mAP result is very bad on duke.

Hi, Xingyu. I tested the arcface using this repo. Here are the experimental results on Market1501.

mAP
Rank1

Baseline (this repo)
85.8
94.2

arcface+triplet sampler
84.3
94.3

------
------
------

Baseline (my repo)
83.4
94.0

arcface+triplet sampler
84.3
94.7

The reason, as you mentioned, maybe that mAP of my repo is lower than this baseline. I will review my code and report it later. Thanks again for your kind discussion.


OpenAPIExtension.decorateOperation() -  never get called

In our project I trying to generate docs for @PreAuthorize to do so I Implemented OpenAPIExtension.decorateOperation()
and registred class like this :
OpenAPIExtensions.getExtensions().add(new SwaggerPreAuthorize());
I see my extension in "chain", but it is never get called.

Can you provide a test scenario? this should be working fine (e.g. this test)


How to scrap a tag of class="" and how to apply condition on selector that scrap meta tag which has property="og:image"

How to scrap a tag of class="" and how to apply condition on selector that scrap meta tag which has property="og:image"
I want to scrap urls having class="" and then want to scrap meta tags having property of og:image
how can i filter my result as above

Duplicate of https://github.com/matthewmueller/x-ray/issues/319


Allow flows to be stopped and started manually

This PR builds on top of #3642
It simplifies some of the api changes in the original PR to make use of the runtime events to get state sent back to the connected editors.
The GET /flows/state endpoint returns an object:
{
   "state": "stop|start|safe"
}

The runtime notification used has id runtime-state and a payload: { state: 'stop|start|safe', deploy: true/false }. The deploy flag tells us if this state change is due to a change in flows. That allows the editor to not show a toast notification for the state change as we'll get a single 'deploy' notification once all the work is done.
It also improves the runtime logging output:
Starting Node-RED in 'stopped' state
    29 Jun 10:30:59 - [info] Active project : Scratch
    29 Jun 10:30:59 - [info] Flows file     : /Users/nol/.node-red/projects/Scratch/flow.json
    29 Jun 10:30:59 - [info] Stopped flows

Deploying whilst stopped
    29 Jun 10:31:35 - [info] Updated flows

Starting a stopped runtime
    29 Jun 10:32:05 - [info] Starting flows
    29 Jun 10:32:05 - [info] Started flows

Deploying whilst started
    29 Jun 10:32:42 - [info] Stopping flows
    29 Jun 10:32:42 - [info] Stopped flows
    29 Jun 10:32:42 - [info] Updated flows   <--- this line is new
    29 Jun 10:32:42 - [info] Starting flows
    29 Jun 10:32:42 - [info] Started flows

Stopping a started runtime
    29 Jun 10:33:32 - [info] Stopping flows
    29 Jun 10:33:32 - [info] Stopped flows

Coverage increased (+0.05%) to 68.335% when pulling b59a3b15f361283d97ddd14bba27fac3841aece1 on pr_3642 into 95f7ea9fc02f8519f58fd0461aff3a21a1ae395e on master.


[google_mobile_ads] Something appear recently in my app interstitial ads - native android bar

Plugin Version
after upgrading to flutter 2.5.1 I found this kind of random behavior on interstitial ads
I used
google_mobile_ads 0.13.5  and google_mobile_ads 0.13.4
i tried it on debug and on release mode but it's also appear in both
its looks a native android widget that came from android studio android templets
##Screen Shot

Hi @mustafa-707, I ran the official sample app from pub and I could not reproduce the issue using google_mobile_ads: 0.13.5,  On which device are you seeing this issue ?  Please share a minimal and complete reproducible code to better address the issue
Thanks

actually i tried the same code with flutter 1.66.2 and flutter 2.2.3 and all good
the problem just happened when I upgrade to flutter 2.5.0 and flutter 2.5.1
my device is Huawei Nova 7i
flutter doctor -v
[√] Flutter (Channel stable, 2.5.1, on Microsoft Windows [Version 10.0.19043.1237], locale en-US)
    • Flutter version 2.5.1 at C:\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision ffb2ecea52 (12 days ago), 2021-09-17 15:26:33 -0400
    • Engine revision b3af521a05
    • Dart version 2.14.2

[√] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at C:\Users\is10vmust\AppData\Local\Android\sdk
    • Platform android-31, build-tools 31.0.0
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[√] Android Studio (version 2020.3)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)

[√] VS Code (version 1.60.2)
    • VS Code at C:\Users\xxxx\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.26.0

[√] Connected device (4 available)
    • JNY LX1 (mobile)                   • AUDUT20221002068 • android-arm64  • Android 10 (API 29)
    • Android SDK built for x86 (mobile) • emulator-5554    • android-x86    • Android 10 (API 29) (emulator)
    • Chrome (web)                       • chrome           • web-javascript • Google Chrome 93.0.4577.82
    • Edge (web)                         • edge             • web-javascript • Microsoft Edge 93.0.961.52

@mustafa-707 does this happen on your specific device or other android device/ emulator too?

this video using the flutter emulator also i tried it in many android devices and the result is same :
https://user-images.githubusercontent.com/74349843/135749928-86246418-12fc-4bf6-9e0a-eab84a8653d8.mp4
also its happen in other type of social login that shows also a random android native bar with the app name , all of that's only notice after moving from flutter 2.3.x to flutter 2.5.x  idk if it caused by the flutter sdk or from another dependencies will try it after removing plugins.

@mustafa-707  does this reproduce on another device or emulator?

No , I tried it on much devices and emulators all them have the same behavior So I guess it's a flutter 2.5.x issue or it's from my project something goes wrong , I will investigate and already wrote an issue on flutter repository

Please share a minimal and complete reproducible code to better address the issue

@mustafa-707  could you please share a sample code or a repository containing minimal code sample? It will help to find the cause of the issue.
Thanks

after much investigation on my project i found the cause of the problem , it was all the time the plugin named :
sentry_flutter
when i removed it it was solved ,
i will let sentry knows that , thank you all guys


Enhance keyword chart view (fix huge legend and provide select top XXX urls)

Here is how "Split View " shows when I select first 20 url's in top 10: https://db.tt/t3Yk6kVh
Maybe you can offer option to only show Chart + Serp pannels like this https://db.tt/1p3FnRhc and remove the other part with url's only, doesn't make sense to have them
Apply same option for "Split View" and "Chart View"

Idea is that nobody wants to see the part with lots of urls like here https://db.tt/hPiQkDS4 .
Just select 20-30 url's and look how annoying it looks, better add only "Chart" + "Serp Table"

Please, please , please add option to select nr of "btn-draw-chart active" from class="btn-draw-chart active" to see instead of clicking on each url. maybe a option like this one :
"Select first ..... url's" to see in chart
Example:
"Select first 5 url's"
"Select first 10 url's"
"Select first 15 url's"
.....etc..

This will help alot. Just think about when you want to see first 30 url's in serp, you need to click on 30 class="btn-draw-chart" just to activate them

I agree.

Any to do date estimation on this feature?


Version
6.x.x
What platforms are you having the problem on?
iOS
System Version
iOS 16/17
On what device are you experiencing the issue?
Real device, Simulator
Architecture
Old architecture
What happened?
{"target":6315,"error":{"localizedRecoverySuggestion":"","code":-11850,"domain":"AVFoundationErrorDomain","localizedFailureReason":"The server is not correctly configured.","localizedDescription":"Operation Stopped"}}
Reproduction
<VideoPlayer source={{ uri: 'https://www.youtube.com/embed/YE7VzlLtp-4?autoplay=1' }} />

Following urls are not video urls, they are web urls. Have a look to react-native-youtube-iframe if you want to use these urls.


Updated Event source to kube-batch.

Signed-off-by: Da K. Ma<EMAIL_ADDRESS>

Hey @k82cn,
Something went wrong with the build.
TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.
View build log
TravisBuddy Request Identifier: 5dbf8f30-c618-11e8-a5a1-47c8df4b6166


[Ionic v4] What Keyboard should we use?

Problem
In v3, ionic-angular had a class Keyboard
This class apparently disappeared in v4, while not listed in the breaking changes
Solutions
There seems to be a Ionic-Native keyboard which provides similar API. But it's a wrapper around ionic-plugin-keyboard, which is deprecated. It suggests to use cordova-plugin-ionic-keyboard instead, but this one has no ionic-native wrapper.
So, in the end, what Keyboard should we use?

@cvaliere it isn't v4 related but rather related to the plugin
my understanding is that https://github.com/ionic-team/cordova-plugin-ionic-keyboard is the supported and actual plugin and that there isn't any native wrapper for it, at least the last time I upgraded my Ionic v3 app I went that way

If I were to guess, I would make the same guess as yours
But it definitely is a breaking change of v4, because the class Keyboard just disappeared
I think the Ionic team should write it in the Breaking Changes and provide an official recommendation to replace it

@cvaliere I'm agree it's confusing, but again, not really v4 related I think, if you upgrade the plugin (switching from the old to the new one) in v3 you would face the same issue
anyhow you are right, maybe the answer is just "there won't be any wrapper for the new plugin." that's it
but I guess the following issue in native is the one you may looking for:
https://github.com/ionic-team/ionic-native/issues/2306

@peterpeterparker I really believe there are two issues:

The plugins are a mess; thanks for pointing out ionic-team/ionic-native#2306
There is a breaking change in Ionic v4, which is that the class Keyboard from the package ionic-angular (absolutely not related to plugins) has disappeared

I'm just saying that this breaking change in Ionic v4 should be written somewhere.

@cvaliere oki doki no worries, let's see what the team would do ;)
sorry didn't want to had noise in the topics, but anyhow your last comment is a good summary!


Add Fedora Dependency to documentation

Describe the bug
Additional dependency needed to build with Fedora 40 cubeb-devel available in the fedora repository
To Reproduce
Try to build with fresh Fedora 40 installation
Expected behavior
Add this to the dependency documentation
Screenshots

Environment

OS: Fedora
Version: 40

Additional context

Just out of curiosity, do you still need this dependency if building from https://github.com/jpochyla/psst/pull/487

I tested it on my laptop running Fedora 40, and it compiles perfectly for me! :P

@SO9010 So we can close this or do we need to add that dependency?

@jacksongoode, we can close it. @rodhfr are you sure you have installed all the needed build dependencies, including GTK? I had a similar issue, but installing all dependencies fixed it.


How can I uninstall radicle?

How can I uninstall radicle?

That depends on how you installed it:

brew uninstall radicle-alpha for mac
apt-get remove radicle for deb
rm ~/.local/bin/radicle* && rm ~/.local/bin/rad && rm ~/.local/bin/rad-daemon-radicle && rm -rf ~/.local/share/radicle  && rm -rf ~/.config/radicle for stack installation with default XDG paths

Let me know if that works!

@NabiKAZ Yes that would correspond to the last bullet-point.

@NabiKAZ closing, but feel free to reopen or comment if you have more questions.


add iPad 2 and iPad Pro to the travis build schemes

@alda-optimizely I added the iPad 2 and iPad Pro to the build schemes


Require leading left bracket in zng values

We were skipping the first char without checking that it was a left bracket, resulting in values like these parsing:
#0:record[foo:string]
0:1;]
0:(1;]

This fixes that.

(I threw in some minor spec tweaks that came up while i was reviewing the spec to verify intended behavior. Didn't seem worthy of a standalone PR but can do so needed.)


libvmi: Promote libvmi from e2e usages to the root pkg/

What this PR does
libvmi has been widely used by source code outside e2e tests. As it serves now both unit tests and e2e tests, it fits to exist in a location that fits both parties.
By promoting libvmi at the root pkg/, we allow both unit and e2e tests to use its services.
While all libvmi builders moved, the factory was left in the e2e test. The factory package has been renamed to libvmifact and used solely in the e2e tests per need.
Before this PR:
After this PR:

Fixes #
Why we need it and why it was done in this way
The following tradeoffs were made:
The following alternatives were considered:
Links to places where the discussion took place:
Special notes for your reviewer

Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

[ ] Design: A design document was considered and is present (link) or not required
[ ] PR: The PR description is expressive enough and will help future contributors
[ ] Code: Write code that humans can understand and Keep it simple
[ ] Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
[ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required
[ ] Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
[ ] Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
[ ] Community: Announcement to kubevirt-dev was considered

Release note

NONE

change: Keep the architecture portion under the e2e tests. As a response to the review.

Thank you @EdDev, great seeing this! Awesome and important job! :tada:

While all libvmi builders moved, the factory was left in the e2e test. The factory package has been renamed to libvmifact and used solely in the e2e tests per need.

I think I get where you're coming from when you say this - since we do tweak these factories according to our e2e tests needs. I have two concerns regarding this:

In another PR I got some pushback regarding using tests/ code in unit tests (which makes a production package import a test package). I think that the factories are very helpful for unit tests, and it would be a shame to duplicate the factory code all over them.
At least IMO the end-goal is that Kubevirt users would be able to use libvmi for their own needs. I think that providing some default factories to serve as a reference could be very valuable for such users that are not familiar yet with libvmi.

WDYT about moving the factories outside of the tests package as well?
We can document that these are just reference factories that we use at our testing and serve as an example. If in the future we'll get to a disagreement over this we can always fallback to the approach presented in this PR and bring factories back to tests package.

Thank you @EdDev, great seeing this! Awesome and important job! 🎉

While all libvmi builders moved, the factory was left in the e2e test. The factory package has been renamed to libvmifact and used solely in the e2e tests per need.

I think I get where you're coming from when you say this - since we do tweak these factories according to our e2e tests needs. I have two concerns regarding this:
1. In [another PR](https://github.com/kubevirt/kubevirt/pull/11494) I got some pushback regarding using `tests/` code in unit tests (which makes a production package import a test package). I think that the factories are very helpful for unit tests, and it would be a shame to duplicate the factory code all over them.

e2e factories should not be needed by unit tests. I hope it is clear that if they are used in unit tests, someone tested something much bigger than an unit.
But if some specific unit tests do need a factory, one should be created for them specifically.

2. At least IMO the end-goal is that Kubevirt users would be able to use libvmi for their own needs. I think that providing some default factories to serve as a reference could be very valuable for such users that are not familiar yet with libvmi.

The current factories are serving only the e2e tests.
There is a need to convince that a factory makes sense to a wider common usage, e.g. for e2e tests, unit tests and potential external projects.
Usually it is easier to convince after you have actual individual factories that can be promoted to the central library/package.
This is exactly what happened with libvmi itself.

WDYT about moving the factories outside of the tests package as well? We can document that these are just reference factories that we use at our testing and serve as an example. If in the future we'll get to a disagreement over this we can always fallback to the approach presented in this PR and bring factories back to tests package.

IMO we should promote only common and well reasoned API/s. To document something we have the README which one can add examples to if it makes sense.

e2e factories should not be needed by unit tests.

Yeah, rethinking about this I guess you're right. There isn't an obvious need to use factories in unit tests.
Alrighty, let's start off this way. We can always circle back to discussing which factories make sense for a broader audience.
Thanks again for this important PR.
/approve

/cc @dankenigsberg

/retest


Klauck/algorithm relaxation

There is still a bug, because (I think) size estimations of indexes are not guaranteed to happen.
Further, I plan to split configurations_by_transformation into separate methods for each transformation and test them separately.

There is still a bug, because (I think) size estimations of indexes are not guaranteed to happen.
I fixed the bug, and some others.


Webserver question

Hi @ricaun,
I have installed & running the last version of your software and it works really well, (I only have some issues wit a NANO that hungs on "BAD jobs" until I restart it by hand, but my question is about the webserver feature.
After installing, I see that the port 80 TCP is opened in the board but I'm unable to access with the browser.
Am I forgetting  any configuration to activate this feature?
Also it could be that this is currently under development so, if it's the case, please forgot any of this message.
Thanks again for your work

Hello @Dark-Manu
If the BAD jobs is consistent like the BAD BAD BAD BAD means you slave have some problem, try to reupload the last version.
Another thing could be the I2C clock too fast, I was testing to add oled display on the same I2C bus and need to set the speed clock to 100000, the old library set the value too high and the slave loses some bytes and receives a wrong job...
You can try the oled tree.

https://github.com/ricaun/DuinoCoinI2C/tree/oled

About the Webserver:
I'm still implement this feature, at the moment the home page is empty by default. The idea is to add some pages with some configuration about the slaves and DuinoCoin stuff.
You can access the page /edit to add some custom page, the user and password is 'admin' and 'admin'.
Use this index.html

https://gist.github.com/ricaun/366088002b08c98bb06873dd1d13f38f

Gonna add some like this:

Wow!, thanks a lot for the info, it's very useful to me.
First, as you suggest, I'll check the speed of the I2C because I also was thinking on add an oled screen to easily check the rig status. Unfortunately, the link you attached, (https://github.com/ricaun/DuinoCoinI2C/tree/oled) takes me to this same project, main's page.
After that I'll mess with the webserver... it looks good.
I'm only a beginner in programming but projects like this help me to improving my skills, thanks again :)
I'll let you know if I have success.

Yes the link sends to a different tree.
https://github.com/ricaun/DuinoCoinI2C/tree/oled
The Esp folder have some new files tô support OLED display.

did some one make index.html to look like more like Arduino IDE serial monitor log ?

where to put the index file ?


Automatic binding Go types with JavaScript (helps with app architecture)

Currently, webview provides all the building blocks to make an awesome GUI app, but it doesn't tell you how to use them. Often it's good to have an un-opinionated library, but in this case it becomes a stumbling block for the new users.
I will follow up with a number of blog posts about best practices I've discovered so far. My biggest pain at the moment is to write JS wrappers for window.external.invoke_() and parse dynamic JSONs in Go to run certain methods in response.
I suggest adding WebView.Register(name string, obj interface{}) method that would register an arbitrary Go struct and expose it under the given name in the JS environment. All public methods will be accessible under their names, and all public fields should be updated automatically, too.
The way I expect it to work is by processing a Go type using reflection and generating some JS code to run invoke_(). Also it should wrap existing ExternalInvokeCallback function into another one that would handle these specific wrapped methods and delegate them to a Go type.
type Counter struct {
	Value int `json:"value"`
}

func (c *Counter) Add(n int) {
	c.Value = c.Value + n
}

func (c *Counter) Reset() {
	c.Value = 0
}

w.Register("counter", &Counter{})

counter.reset();
counter.add(1);
console.log(counter.value);

Done with the bindings in webview-bind branch. Working on some click counter example for Vue.js and/or React.js.

Getting very close. Had to add a well-known registry trick to force IE11 compatibility mode on Windows. Now most of the JS libraries can be normally injected via w.Eval().
Added examples that use Picodom and Vue.js. Both work fine.
But React is still a problem. Works fine on Linux and Mac, but fails to load on Windows (actually, Babel fails to load and it makes the whole React thing pretty much useless).

Ok, Babel works. But React is a huge mess. I ended up replacing it with Preact and keeping the JSX syntax. For the majority of the use cases that should be sufficient, I hope.
You may have a look at the counter-go example. By default it uses picodom.js, which is a featherweight 1kb reactive UI renderer. It's really small and fast, but JS code may look ugly to an unprepared eye.
If you generate and build it with -tags vue - you get a VueJS example. Same Go main code and controller, but different UI layer.
Finally, generating/building it with -tags react runs a Preact example with JSX syntax. You may use all ES6 features in the "frontend" code.
I like that despite using modern JS, none of the examples relies on Node.js or npm. It's all Go.
Also this example shows an interesting way of loading UI code into a webview without running a server. All "frontend" code is injected using w.Eval(), so there are no open HTTP ports, no web server and the app looks a bit more solid.