Find the plugin for PeripheralAXI4

Hi,
I the the plugin for PeripheralAXILite4
plugins += new LsuPeripheralAxiLite4(ioDataWidth = xlen)
Is there any plugins for AXI4? Also, where can I find a list of plugins for naxrisc?
Thank you so much,
Duc

Hi,
There is currently no maintained list of available plugin. If you do "grep -r 'extends Plugin' ."  you should get them all.
For the Axi4 stuff, here is an example to have full AXI interface :
https://github.com/SpinalHDL/NaxRiscv/blob/0fd620074dd486a7bd9accbf783edc96c9ad65fd/src/main/scala/naxriscv/platform/Litex.scala#L27
So you need those 3 plugins and that should be all good.
Regards
Charles

A question, with those AXI4,
plugins += new FetchAxi4(
ramDataWidth = ramDataWidth,
ioDataWidth  = ioDataWidth,
toPeripheral = cmd => toPeripheral(cmd.address)
)
plugins += new DataCacheAxi4(
dataWidth = ramDataWidth
)
I can access the external RAM through AXI4? Do you have any example?
Right now, I can access the external ram through LsuPeripheralAxiLite4 but that is AXIlite, so it will be slow.
Thank you so much,
Duc

Right now, I can access the external ram through LsuPeripheralAxiLite4 but that is AXIlite, so it will be slow.

This is specified by the MMU ioRange parameter
The io region wil gon on the AxiLite, while the non io region will go on the AXI4.
Regards
Charles

Sure, for instance :
https://github.com/SpinalHDL/NaxRiscv/blob/0fd620074dd486a7bd9accbf783edc96c9ad65fd/src/main/scala/naxriscv/Gen.scala#L407
Could be ajusted, currently it mean that all the access in [0x10000000-0x1FFFFFFF] are io region.

Cool,
I will check it. thank you so much.
Duc

hmm that seems ok to me. Can you send me a wave file ?

(and the .asm too)

Hi,
I would like to upload the code
https://www.file.io/uZXH/download/cxLkN0NVE83g
Thank you so much,
Duc

Hi,
I forget the vcd file.
https://file.io/IZX4wujdsPkc
Thanks,
Duc

Hi,
I am still confused. So, how can I make it write AXI to the address 0x80000000
Thank you,
Duc

Hi,
You need to explicitly flush the data cache
there is a custom instruction for this, encoded as 0x0000500F

Hi,
The problem is I dont know which command in rust to complie to make flush the data cache.

hmm, can you put some inline assembly code in rust ?
I don't know rust much.

Hi,
This one is example of inline assembly

Hi,
This one is example of inline assembly and can complie good.
fn delay(cycles: u32) {
for _ in 0..cycles {
unsafe {
asm!("nop");
}
}
}
This is nop code. Do you know the opcode name of 0x0000500F
Thank you so much,
Duc

I have no rust compiler but here is rust code for another riscv cpu.
From that I think you need something like:
asm!(".insn i 0x500F");

Hello,
This worked! Thank you so much.
Duc

@cklarhorst Thanks :D


example code error

example code error: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/join-clause#example
error 1：
Console.WriteLine("GroupJoin3: {0} items in 1 group", totalItems, groupJoinQuery3.Count());
should update：
Console.WriteLine("GroupJoin3: {0} items in {1} group", totalItems, groupJoinQuery3.Count());
or：
Console.WriteLine("GroupJoin3: {0} items in 1 group", totalItems);
error 2：
Console.WriteLine("Group:", prodGrouping.Count());
should update：
Console.WriteLine("Group{0}:", prodGrouping.Count());
or：
Console.WriteLine("Group:");
so the output should update.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7f64bda9-3763-e0a0-fbb9-22f6fd50beca
Version Independent ID: d552325f-7578-efb3-3f6d-927169181391
Content: join clause - C# Reference
Content Source: docs/csharp/language-reference/keywords/join-clause.md
Product: dotnet-csharp
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

A fix to this was done by @imba-tjd in https://github.com/dotnet/samples/pull/1444

closing as duplicate.

A fix to this was done by @imba-tjd in dotnet/samples#1444

Thank you for your reply but it seems that you have solved the issue which I found ,and not the repeat issue.

closing as duplicate.
@kinglionsz This PR was merged this morning. The updates should be live tomorrow, on our regular publishing cycle.

Thank you for your time .


[BUG]: Docker Compose crashes on getting response or opening workspace settings

How are you running AnythingLLM?
Docker (local)
What happened?
Since a few days my docker compose AnythingLLM setup crashes, whenever I ask for an LLM response in a thread or open workspace settings.
The following is shown in the docker logs:
anythingllm-1  | /usr/local/bin/docker-entrypoint.sh: line 7:   164 Illegal instruction     (core dumped) node /app/server/index.js
anythingllm-1 exited with code 0
Docker compose restarts the container, but it keeps happening.
I use groq, but it also happens when connected to a local ollama.
What did I do to break things?
Are there known steps to reproduce?
This happens when I ask for any response in a thread or when I open workspace settings (url/workspace/workspace-name/settings/general-appearance).

What does your docker-compose look like? This seems like some directory or something is being mounted that should not be. Likely some library that should exist but isn't?

This has been reported before and this was the case [https://stackoverflow.com/a/42159385](see this SO comment)

No that seems correct to me. Are you building on and ARM device? I think by default, unless specified, docker compose will build in compatibility mode, but since our dockerfile supports both arch's it should enable autodetection.
What chip arch are you on?

Hello @timothycarambat this is on a Xeon server.

Then that makes sense, Xeon is x86, not arm so it makes sense to only build for amd64

So this is interesting, I copied my (original, simple) docker compose file to another server and AnythingLLM works there.
The difference:
This currently works:
OS: Debian GNU/Linux 10 (buster) x86_64
Host: KVM RHEL-8.6.0 PC (Q35 + ICH9, 2009)
CPU: Intel Xeon Silver 4216 (24) @ 2.099GHz
Memory: 10240MiB / 257696 MiB
Docker Compose version v2.27.0
Docker version 26.1.1, build 4cf5afa

This does not:
OS: Proxmox VE 8.2.2 x86_64
Host: BANFF12 [baremetal]
CPU: Intel Xeon E5-2670 0 (64) @ 2.600GHz
Memory: 92153MiB / 257572 MiB
Docker Compose version v2.27.0
Docker version 26.1.1, build 4cf5afa

While i cant nail down exactly why this occurs i can say that we have had issues reported with Proxmox before, but i dont have that system available to us so i cannot replicate to debug remotely, but i recall is was either permissions or port related

Hi Tim,
I am also experiencing this issue in a VM but using Hyper-V. This showed up when I pulled the :latest docker image two days ago. I don't build the project just use the image. I tried removing the /app/server/storage volume from the docker-compose file to be sure its not my folder giving permission issues and the issue persist with that being freshly populated by the image.
I've turned on debug logging for node, here are the last lines before the error
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/apache-arrow"]
anything-llm  | MODULE 102: Module._load REQUEST ./util/buffer.js parent: /app/server/node_modules/apache-arrow/Arrow.js
anything-llm  | MODULE 102: RELATIVE: requested: ./util/buffer.js from parent.id /app/server/node_modules/apache-arrow/Arrow.js
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/apache-arrow"]
anything-llm  | MODULE 102: Module._load REQUEST ./util/vector.js parent: /app/server/node_modules/apache-arrow/Arrow.js
anything-llm  | MODULE 102: Module._load REQUEST ./util/pretty.js parent: /app/server/node_modules/apache-arrow/Arrow.js
anything-llm  | MODULE 102: RELATIVE: requested: ./util/pretty.js from parent.id /app/server/node_modules/apache-arrow/Arrow.js
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/apache-arrow"]
anything-llm  | MODULE 102: Module._load REQUEST ./visitor/typecomparator.js parent: /app/server/node_modules/apache-arrow/Arrow.js
anything-llm  | MODULE 102: Module._load REQUEST ./Arrow.js parent: /app/server/node_modules/apache-arrow/Arrow.dom.js
anything-llm  | MODULE 102: Module._load REQUEST ./arrow parent: /app/server/node_modules/vectordb/dist/index.js
anything-llm  | MODULE 102: RELATIVE: requested: ./arrow from parent.id /app/server/node_modules/vectordb/dist/index.js
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/vectordb/dist"]
anything-llm  | MODULE 102: load "/app/server/node_modules/vectordb/dist/arrow.js" for module "/app/server/node_modules/vectordb/dist/arrow.js"
anything-llm  | MODULE 102: Module._load REQUEST apache-arrow parent: /app/server/node_modules/vectordb/dist/arrow.js
anything-llm  | MODULE 102: Module._load REQUEST ./remote parent: /app/server/node_modules/vectordb/dist/index.js
anything-llm  | MODULE 102: RELATIVE: requested: ./remote from parent.id /app/server/node_modules/vectordb/dist/index.js
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/vectordb/dist"]
anything-llm  | MODULE 102: load "/app/server/node_modules/vectordb/dist/remote/index.js" for module "/app/server/node_modules/vectordb/dist/remote/index.js"
anything-llm  | MODULE 102: Module._load REQUEST ../index parent: /app/server/node_modules/vectordb/dist/remote/index.js
anything-llm  | MODULE 102: RELATIVE: requested: ../index from parent.id /app/server/node_modules/vectordb/dist/remote/index.js
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/vectordb/dist/remote"]
anything-llm  | MODULE 102: Module._load REQUEST ../query parent: /app/server/node_modules/vectordb/dist/remote/index.js
anything-llm  | MODULE 102: RELATIVE: requested: ../query from parent.id /app/server/node_modules/vectordb/dist/remote/index.js
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/vectordb/dist/remote"]
anything-llm  | MODULE 102: load "/app/server/node_modules/vectordb/dist/query.js" for module "/app/server/node_modules/vectordb/dist/query.js"
anything-llm  | MODULE 102: Module._load REQUEST apache-arrow parent: /app/server/node_modules/vectordb/dist/query.js
anything-llm  | MODULE 102: Module._load REQUEST ../native.js parent: /app/server/node_modules/vectordb/dist/query.js
anything-llm  | MODULE 102: RELATIVE: requested: ../native.js from parent.id /app/server/node_modules/vectordb/dist/query.js
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/vectordb/dist"]
anything-llm  | MODULE 102: load "/app/server/node_modules/vectordb/native.js" for module "/app/server/node_modules/vectordb/native.js"
anything-llm  | MODULE 102: Module._load REQUEST @neon-rs/load parent: /app/server/node_modules/vectordb/native.js
anything-llm  | MODULE 102: looking for "@neon-rs/load" in ["/app/server/node_modules/vectordb/node_modules","/app/server/node_modules","/app/node_modules","/node_modules","/app/.node_modules","/app/.node_libraries","/usr/lib/node"]
anything-llm  | MODULE 102: load<EMAIL_ADDRESS>for module<EMAIL_ADDRESS>anything-llm  | MODULE 102: Module._load REQUEST ./index.node parent: /app/server/node_modules/vectordb/native.js
anything-llm  | MODULE 102: RELATIVE: requested: ./index.node from parent.id /app/server/node_modules/vectordb/native.js
anything-llm  | MODULE 102: looking for ["/app/server/node_modules/vectordb"]
anything-llm  | MODULE 102: Module._load REQUEST @lancedb/vectordb-linux-x64-gnu parent: /app/server/node_modules/vectordb/native.js
anything-llm  | MODULE 102: looking for "@lancedb/vectordb-linux-x64-gnu" in ["/app/server/node_modules/vectordb/node_modules","/app/server/node_modules","/app/node_modules","/node_modules","/app/.node_modules","/app/.node_libraries","/usr/lib/node"]
anything-llm  | MODULE 102: load<EMAIL_ADDRESS>for module<EMAIL_ADDRESS>anything-llm  | /usr/local/bin/docker-entrypoint.sh: line 7:   102 Illegal instruction     (core dumped) node /app/server/index.js
anything-llm exited with code 132

Let me know if there's anything you'd like me to run or test to help figure out the issue
Thanks!

This issue is because the default lancedb vector db does not have support (or can determine) which binary it needs to run lancedb.
<EMAIL_ADDRESS>  version "0.4.11"
  resolved "https://registry.yarnpkg.com/vectordb/-/vectordb-0.4.11.tgz#e5b209fc15fb100d8495c132d0e4cc817dc9065a"
  integrity sha512-hbN1qO08xdhEpoGecWR6UuvfnNFYDgZLvuEDvgq5nFVbA4er5qXTcEyxjnlTzKsVqFGLlPF+91I+JIigneX24w==
  dependencies:
    "@apache-arrow/ts" "^14.0.2"
    "@neon-rs/load" "^0.0.74"
    apache-arrow "^14.0.2"
    axios "^1.4.0"
  optionalDependencies:
    "@lancedb/vectordb-darwin-arm64" "0.4.11"
    "@lancedb/vectordb-darwin-x64" "0.4.11"
    "@lancedb/vectordb-linux-arm64-gnu" "0.4.11"
    "@lancedb/vectordb-linux-x64-gnu" "0.4.11"
    "@lancedb/vectordb-win32-x64-msvc" "0.4.11"

i would imagine linux x64 would be right?

This issue is because the default lancedb vector db does not have support (or can determine) which binary it needs to run lancedb.
That's interesting, so the problem originated in the vectordb. That would explain why the web ui crashed when opening workspace settings.

i would imagine linux x64 would be right?
Yes, I imagine so as well.
The machine that doesn't work (bare metal):
Linux 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64 GNU/Linux
and the KVM that does work:
Linux 4.19.0-26-amd64 #1 SMP Debian 4.19.304-1 (2024-01-09) x86_64 GNU/Linux

This seems like now more of a support issue for https://github.com/lancedb/lancedb
That being said, we are on package 0.4.11 so it is possible could have been fixed since in the SDK. Obviously such a critical piece like the vectordb we have to bump very carefull.

Yes, I think you are right @timothycarambat. Thanks for the help investigating and the great tool!

Hi Tim,
I can confirm that I am on Linux x64 as well and the project is running in a podman container.
Could I suggest tagging the docker images with release versions like the desktop app? Simply so we could roll back in case of issues like this arising. Right now our instance is broken; even with no documents uploaded it crashes because of the default integrated LanceDB. The best option to "fix" this seems to be setting up an alternative vectordb (which is the route I am currently taking, two issues throwing the same error are currently open on the LanceDB repo since oct-nov of 2023 (https://github.com/lancedb/lancedb/issues/592 and https://github.com/lancedb/lancedb/issues/631) so I'm not sure Anything-LLM will start working for us with lancedb anytime soon).
Let me know if there is anything I can do to help. I'd like to echo iFloris and thank you and your community for the amazing open source project that is Anything-LLM!

I have the same issue. Does not work on XEON E5 2660

@computersrmyfriends use the non-native embedder. ONNX cannot run on your CPU, so native model running will not work.

Thank you for the response @timothycarambat . I will try that

I think you meant this ☝ @timothycarambat . I changed it.
But,even after that, once I send a message "hi" to the model on the workspace, it crashes,,,,

Does sending a chat on an empty workspace work? Should not run any command on the embedder or vectordb. Referring to the thread, it looks like an invalid binary from lanceDB if the embedder can be ruled out

Thanks for the super quick response.

The moment i created the workspace, it crashed again.
Then, i sent hi to the new workspace after restarting the container..., it crashed again..

@timothycarambat  Now, I tried chromadb...
It does not crash now. So, the crashing issue was due to the inbuilt Vectordb.
But, it gives me this error:

@timothycarambat  I tried creating another new workspace. With chromadb, it seems to be some different issue. it was giving an error twice and then it crashed.

Also errors from chromadb container if it might help:


(GH-2311) Add LXD Transport

This adds the basic structure for a new LXD transport that will ship
with Bolt. The transport has one configuration option, remote, which
accepts a remote LXD host to connect to. This includes a new VM in the
Vagrantfile that's used for testing. The transport is largely modeled
off of the Docker transport, since they behave relatively similarly by
shelling out to localhost to interact with the containers they are
running.
!feature

Add LXD transport (#2311)
Bolt now includes a new LXD transport to use when connecting with
containers managed with LXD. See
https://puppet.com/docs/bolt/latest/bolt_transports_reference.html for
configuration options.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: lucywyman:x: dontlaughYou have signed the CLA already but the status is still pending? Let us recheck it.

@dontlaugh Alrighty! I got this working both locally and it looks like in CI. I'll try to get this updated with instructions for using the VM for testing as well if people can't test directly on their system, and I also need to add more tests using the remote option. But I think this is getting pretty close!

I haven't quite been able to get this to work with my existing inventory.

inventory.yaml
config:
  transport: ssh
  ssh:
    user: coleman
    private-key: ~/.ssh/id_ed25519
    native-ssh: true
    host-key-check: false

groups:
  - name: cloud
    targets:
      - backfire
      - breakout
      - merc
  - name: lan
    targets:
      - caeser
  - name: gf-containers
    config:
      transport: lxd
      lxd:
        remote: gf
    targets:
      - db
      - faktory
      - gitea
      - matrix
      - nats
        #- puppet
      - vault
      - consul-01
      - consul-02
      - consul-03
  - name: consul
    targets:
      - consul-01
      - consul-02
      - consul-03
    config:
      transport: lxd
      lxd:
        remote: gf
  - name: consul-agents
    targets:
      - faktory
      - gitea
      - matrix
      - nats
        #- puppet
      - vault
    config:
      transport: lxd
      lxd:
        remote: gf

targets:
  # cloud
  - name: backfire
    uri: <IP_ADDRESS>
    config:
      ssh:
        port: 22420
  - name: breakout
    uri: <IP_ADDRESS>
    config:
      ssh:
        port: 22420
  - name: merc
    uri: <IP_ADDRESS>
    config:
      ssh:
        port: 22420
  # lan
  - name: caeser
    uri: caeser.lan
  # gf-containers
  - name: faktory
    uri: faktory
  - name: nats
    uri: nats
  - name: gitea
    uri: gitea
  - name: vault
    uri: vault
    #- name: puppet
    #  uri: puppet
  - name: db
    uri: db
  - name: matrix
    uri: matrix
  - name: consul-01
    uri: consul-01
  - name: consul-02
    uri: consul-02
  - name: consul-03
    uri: consul-03

The output of bolt command run -t gf-containers uptime

output
❯ bolt command run -t gf-containers uptime
/home/coleman/Code/bolt/lib/bolt/logger.rb:198: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/home/coleman/Code/bolt/lib/bolt/logger.rb:207: warning: The called method `log_message' is defined here
Unknown option 'remote' at 'groups.2.config.lxd' at /home/coleman/Code/infra/bolt/inventory.yaml. [ID: unknown_option]
Unknown option 'remote' at 'groups.3.config.lxd' at /home/coleman/Code/infra/bolt/inventory.yaml. [ID: unknown_option]
Unknown option 'remote' at 'groups.4.config.lxd' at /home/coleman/Code/infra/bolt/inventory.yaml. [ID: unknown_option]
The LXD transport is experimental, and may include breaking changes between minor versions. [ID: lxd_experimental]
Started on consul-02...
Started on faktory...
Started on nats...
Started on vault...
Started on db...
Started on gitea...
Started on consul-03...
Started on consul-01...
Started on matrix...
Failed on matrix:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on faktory:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on nats:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on consul-02:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on consul-01:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on vault:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on gitea:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on consul-03:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on db:
  The command failed with exit code 1
  STDERR:
    Error: not found
Failed on 9 targets: db,faktory,gitea,matrix,nats,vault,consul-01,consul-02,consul-03
Ran on 9 targets in 5.76 sec

One thing I might not be clear on is exactly how to fully install Lucy's branch. Here's what I did, after checking out the branch
git fetch lucy
git checkout GH-2311
bundle install --clean --force
bundle binstubs --path=~/bin --force


Fix struct reserved bitwidth spacing in c_template.

Previously, reserved bits would be inserted incorrectly into the struct.

Good one. Thanks!


fix: paste styles shortcut

fix https://github.com/excalidraw/excalidraw/issues/4884
the fix is 2 lines (actionStyles.tsx). The rest are tests.

Looks good.
Probably subject to another PR, but currently copy/paste style for a sticky note will only update the container and not the text within the container.
https://user-images.githubusercontent.com/14358394/157110264-a7096b90-1f71-4a8a-afee-026185813dc0.mp4

Probably subject to another PR, but currently copy/paste style for a sticky note will only update the container and not the text within the container.

Yes, let's fix that in another PR. I have one open from last year that improves style pasting in general.


Crash after upgrade to 3.1.1 and later with android

Bug Information
from plugin verson 3.1.1 my app immediately crashes with unhandled exception
Plugin.Text To Speech.Abstractions.Text To Speech.Speak not found.
Version Number of Plugin: 3.1.1 and later
Device Tested On: Moto4
Simulator Tested On: none
Version of VS: 2017 15.5.7 community
Version of Xamarin: <IP_ADDRESS>0555
Versions of other things you are using:
android 7.0
Steps to reproduce the Behavior
see code snippet
Expected Behavior
using version 3.0.1 is working fine
Actual Behavior
Method is not entered. Excepction will not be caught, app crasshes completly
Code snippet
button with simple method:
        private async void StartAudio(object sender, System.EventArgs e)
        {
            try
            {
                await CrossTextToSpeech.Current.Speak("Test");
            }
            catch (Exception xxx)
            {
                await DisplayAlert("error", xxx.Message, "ok");
            }
        }

Screenshotst

Could you share the crash log from the device? It will help us to troubleshoot the issue

Probably need to completely clean all obj/bin and uninstall/reinstall nugets. Should work fine.


[wip: test doc-builder]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Build the file list and run linters in parallel

Proposed changes

Parallelize building the file list
Parallelize linters
Enable Gitleaks to run against single files because it's parallelized, so there's no reason to treat it in a special way.

Fix #5073
The tasks I parallelized are, by far, the most expensive tasks. We could also parallelize the configuration of linter config files, but that takes a few seconds only.
To implement these changes, I had to refactor things a bit:

Don't use a .gitleaksignore because Gitleaks doesn't provide a way to ignore this file. This was causing issues when running tests.
Don't handle special cases to exclude bad and good tests. Use FILTER_REGEX_EXCLUDE and FILTER_REGEX_INCLUDE instead.
Fix an issue with Checkov not linting files if the configuration file included the list of directories to lint.
Move the logic to run linter tests to a script to reduce duplication in the Makefile, and to enable further error checks. I'll reduce duplication even more in a follow up PR.
Remove the WORKSPACE_PATH variable. Use GITHUB_WORKSPACE only to reduce the things to consider.
Use temporary files to hold the list of files to lint, and return codes so that processes can read them. This is necessary to implement a robust inter-process communication. Environment variables are too fragile for this purpose.
Move linter commands to a dedicated file that processes can source as needed because Bash doesn't currently offer a reliable way to export arrays.
Save output in JSON files and parse them to print it as needed.
Export some functions and variables so that subprocesses can use them.
Move the logic to run "additional installations" to the RunAdditionalInstalls function instead of having them scattered around several places.
Use arrays to store commands instead of an associative array of strings to properly handle escapes and arguments.
To differentiate between different test cases, we now return: 0 if everything was linted fine, 1 if there were errors, 2 if there were errors, but some linters reported success.

Readiness checklist
In order to have this pull request merged, complete the following tasks.
Pull request author tasks

[x] I included all the needed documentation for this change.
[x] I provided the necessary tests.
[x] I squashed all the commits into a single commit.
[x] I followed the Conventional Commit v1.0.0 spec.
[x] I wrote the necessary upgrade instructions in the upgrade guide.
[x] If this pull request is about and existing issue,
I added the Fix #ISSUE_NUMBER label to the description of the pull request.

Super-linter maintainer tasks

[x] Label as breaking if this change breaks compatibility with the previous released version.
[x] Label as either: automation, bug, documentation, enhancement, infrastructure.

/cc @kftsehk :)

@Hanse00 Here's a possible review path:

Review Dockerfile: install parallel
Review .gitignore: ignore terraform state stuff that we fetch before running tflint
Review Makefile, test/run-super-linter-tests.sh: move the logic to run linter tests from makefile to a script, use FILTER_REGEX_EXCLUDE to tell super-linter which test cases to run, instead of having special cases handled in buildFileList when test mode is on.
Checkov configuration files: Use Checkov config files to include the directories to lint so we don't need to account for special cases.
Gitleaks configuration files and workflows/ci.yml: remove .gitleaksignore and use a configuration file to ignore tests when linting the codebase, otherwise "bad" tests will fail validation
Review docs: Gitleaks now behaves as any other linter
Review linterCommands.sh, super_linter.rb, linter.sh (part): move commands from linter.sh to a dedicated file because we need to source this file from worker (we can't pass arrays using env variables)
Review validation.sh and log.sh: minor changes to export needed functions so subprocesses can find them. No need to define error count variables anymore because we use files to hold results and communicate across projects.
Review detectFiles.sh: minor changes, plus move all the "additional installation steps before running linters" in RunAdditionalInstalls. Before this change, it was scattered around several places.

At this point, you should be left with the three main files to review: linter.sh, buildFileList.sh, worker.sh
I'll follow up with another comment, so you can read this in the meantime

The main idea behind the changes in linter.sh, buildFileList.sh, worker.sh is that when there was a loop, we now have a parallel invocation that takes a certain number of arguments.
The main loops to consider are:

buildFileList: Loop over each file in the codebase that we build with git or with find, and categorize each file according to its type. We refactored this loop so that we have multiple processes handling a portion of the list of files to categorize. This let us categorize multiple files in parallel.
linter: Loop over the list of "languages" to lint. For each "language" to lint, we now have a parallel invocation of the LintCodeBase function that we define in worker.sh. This let us lint files for several languages in parallel.
worker: Loop over the files to lint because many linters support linting more than one file for each invocation. This let us avoid paying the startup cost for these linters. In this case, there are a few corner cases to handle when linters aren't able to take more than file to lint at a time.

Finally, I had to refactor the return code logic a bit to handle three cases so we can properly run our test suite:

All good -> return 0
All linters returned errors -> return 1
Some linters returned errors, but some returned ok -> return 2

ℹ JFYI for those who will stumble upon this PR wondering why shfmt started to lint shell files out of the blue since the v6 of super-linter: inter alia this PR removed a historical workaround introduced back in v3.13.0 to skip shfmt if there's no .editorconfig in $GITHUB_WORKSPACE (at that time shfmt probably required .editorconfig 🤔 I couldn't nail that 🤷🏻).
So since the v6 release the shfmt isn't skipped anymore and can lint shell files as it is supposed to and, welp, now we have to align with shell formatting standards (my colleagues are starting to get confused with this new behavior 🤪)
Some refs:

https://github.com/super-linter/super-linter/pull/642#issuecomment-683459600 + https://github.com/super-linter/super-linter/pull/642#issuecomment-683886242
https://github.com/super-linter/super-linter/issues/2068
https://github.com/super-linter/super-linter/blob/v3.13.0/lib/linter.sh#L1239-L1245


rule(macro user_known_k8s_client_container): add node-problem-detecto…

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

If contributing rules or changes to rules, please make sure to also uncomment one of the following line:

/kind rule-update

/kind rule-create

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area engine

/area rules

/area tests

/area proposals

What this PR does / why we need it:
When deploy Node problem detector (0.8.3 or upper) via daemonset,
Docker health checker binary including NPD runs periodically, which causes false positives.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

rule(macro user_known_k8s_client_container): add node-problem-detector pattern to avoid false positive

@DingGGu: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

ce3e708 rule(macro user_known_k8s_client_container): add node-problem-detector pattern to avoid false positive

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign kaizhe
You can assign the PR to them by writing /assign @kaizhe in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

rules/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Welcome @DingGGu! It looks like this is your first PR to falcosecurity/falco 🎉

/release-note-none

/assign @kaizhe

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

b07641a rule(macro user_known_k8s_ns_kube_system_images): add new macro image name inside kube-system namespace

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

LGTM label has been added.  Git tree hash: 5f54a4d50ec0815e9b7d72796d8683a79179eab0

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Kaizhe
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

rules/OWNERS [Kaizhe]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Kaizhe, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

rules/OWNERS [Kaizhe,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/milestone 0.27.0


noUncheckedIndexedAccess and checked bracket notation access

Bug Report
🔎 Search Terms
noUncheckedIndexedAccess bracket access undefined
🕗 Version & Regression Information

I was unable to test this on prior versions because noUncheckedIndexedAccess is available since 4.1

⏯ Playground Link

// @noUncheckedIndexedAccess
const meObj: { [propName: string]: string } = {}
const meKey: keyof typeof meObj = 'meh'
if (meObj[meKey]){
  meObj[meKey].toString() // Object is possibly 'undefined'.
}

Workbench Repro
🙁 Actual behavior
Compile Error: Object is possibly 'undefined'. error
🙂 Expected behavior
Typescript should be able to recognize the existence of the property with a check. This generates a lot of wrong errors in our codebase when the noUncheckedIndexedAccess compiler flag is enabled.

Duplicate of #10530. Type narrowing does not occur for indexed access forms e[k] where k is not a literal.

This one also has similar examples further below https://github.com/microsoft/TypeScript/issues/41612
I'll close this one then.


How to use?

Pls add in documentation how to use
I dont undestand

I wrote a tempoary instruction before writing it properly.
First of all, you can download it from https://github.com/MineEric64/BetterLiveScreen/releases/latest
also you have to download and run it in two or above computers.
how to use

In server, click the button named 'Create Room'
In client, type server's ip next to the button named 'Connect' and click the button named 'Connect'
Server and client are both connected, so you can share the screen when you click the button named 'Go Live'
If you want to watch the user's stream (live), click the button named 'Watch' above the user's profile and you can watch the user's stream (live).

when you have a question about this app (Better Live Screen), you can open the issue and question it.
Thanks!


[DOC][BUG] Default Index Merge Policy is Not Customizable for Cluster Level Settings

What do you want to do?

[x] Request a change to existing documentation
[ ] Add new documentation
[x] Report a technical problem with the documentation
[ ] Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
The current documentation informs users that they can switch OpenSearch merge policies by overriding the cluster-level settings. However, after confirming with @jainankitk, @rishabhmaurya, and @ansjcy  merge policies can only be changed on an index-level basis.
The following section of the documentation is misleading and should be removed. When users follow the documentation and attempt to update the default_index_merge_policy, they encounter a validation error:
indices.time_series_index.default_index_merge_policy (String): This setting allows you to specify the default merge policy for time-series indexes, particularly for those with an @timestamp field, such as data streams. The two available options are tiered (default) and log_byte_size. Using log_byte_size for time-series indexes is recommended for enhancing the performance of range queries with the @timestamp field. To override the merge policy on a per-index basis, you can use the index.merge.policy index setting.

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.
How to recreate error:

Setup an opensource OpenSearch cluster. Ensure that you can curl against it.
Run this command against your cluster endpoint. A validation error is encountered.

curl -XPUT '<cluster_endpoint>/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "indices" : {
    "time_series_index.default_index_merge_policy": "log_byte_size"
  }
}'
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"},"status":400}

Thanks @IanHoang for filing this issue. We should move the cluster setting to static section from dynamic.

Thanks for filing the issue. Do you want to remove the cluster setting or move it? We don't have a section for static cluster-level settings, but can create one. If easier to do than explain, please update here: https://github.com/opensearch-project/documentation-website/edit/main/_install-and-configure/configuring-opensearch/index-settings.md. Thanks!

Synced up with Heather and we can just remove it since it's already in the static section


infantry walk animation broken

After merging harvester logic it seemed like the unit movement animation is broken.
How to reproduce:

construct infantry
move infantry
infantry does not have walking animation

expectation:

infantry should have walking animation

Little investigation points that SHA caea21f70eed07fc71192beb0838d429dbbe4ee7 worked fine. Anything after that is broken.

assignin @arjenvanderende  because he is the bug master... ;-)

Reason: animation is indiscriminate of action. Meaning animating is walk animation, but animating is now also harvesting.
If enabling animation upon movement, a harvester will harvest while moving.
What is required:

infantry should animate while moving
harvester should animate while harvesting

Maybe an abstraction can be made later.


[rc-swift] ConfigDBManager.swift

Implements RC's underlying ConfigDBManager in Swift
Migrates database queue to an actor
Refactors some of the mocks out of the the unit tests
Pod testing works on Xcode 16. Other CI failures look to be transition-related

I commented or resolved all comments except for most of the ones in DatabaseActor which I'll look at in a few hours

Going to merge it here. Thanks for the reviews @ncooke3  and @andrewheard.
One note is that callbacks from the Database APIs are not wrapped in main thread because they're already in a Task, can cause deadlocks and should be replaced by async APIs when the callers are in Swift.

After merging, I rebased rc-swift to the latest main


feat: add runtime changeable auto-play with restartTimer()

Great PR, much appreciated. If you are looking to implement #89 as well that would be great. as the author didn't provide feedback for a month now.

Thanks for addressing the stuff I mentioned, all looks good to me now, do you want to merge or do you still have some stuff you need to do?

It's all ready from my side, thanks. Please merge if happy.
Once this one is merged I'll try to get the other PR updated with these changes.

Cool, thanks for the PR, much appreciated!


fix re-rendered to restart event

Fix re-rendered to restart event, such like change template content via consul etc, response data:
{
    "Type": "Restart Signaled",
    "Time":<PHONE_NUMBER>922288000,
    "Message": "",
    "DisplayMessage": "Template with change_mode restart re-rendered",
    "Details": { },
    "FailsTask": false,
    "RestartReason": "",
    "SetupError": "",
    "DriverError": "",
    "ExitCode": 0,
    "Signal": 0,
    "KillTimeout": 0,
    "KillError": "",
    "KillReason": "",
    "StartDelay": 0,
    "DownloadError": "",
    "ValidationError": "",
    "DiskLimit": 0,
    "FailedSibling": "",
    "VaultError": "",
    "TaskSignalReason": "",
    "TaskSignal": "",
    "DriverMessage": "",
    "GenericSource": ""
}

Sorry can you further clarify?

nomad template has a parameter named change_mode, the job will follow the behavior if the data changed (for example, the data had been updated by valut key, consul KV, nomadVar or nomadServe).

Thanks! That makes sense, perhaps it's better to just change it something like this instead?
if task_event_details.dig("restart_reason")&.match?(/unhealthy/)
  :failure
else
  :success
end

Meaning all other restart events are not considered failures?

You are right, updated!

Thanks!


HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5.

Description of PR
JIRA: HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5.
How was this patch tested?
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6545/1/console in case of problems.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 54s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
shelldocs
0m  0s

Shelldocs was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ branch-3.4 Compile Tests _

+1 :green_heart:
mvninstall
43m  6s

branch-3.4 passed

+1 :green_heart:
compile
0m 21s

branch-3.4 passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
0m 21s

branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
mvnsite
0m 27s

branch-3.4 passed

+1 :green_heart:
javadoc
0m 27s

branch-3.4 passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 23s

branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
shadedclient
32m 11s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 14s

the patch passed

+1 :green_heart:
compile
0m 13s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
0m 13s

the patch passed

+1 :green_heart:
compile
0m 13s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
0m 13s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 17s

the patch passed

+1 :green_heart:
shellcheck
0m  1s

No new issues.

+1 :green_heart:
javadoc
0m 14s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 15s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
shadedclient
32m 32s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 17s

hadoop-project in the patch passed.

+1 :green_heart:
asflicense
0m 35s

The patch does not generate ASF License warnings.

116m 37s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6545/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6545

Optional Tests
dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient xmllint

uname
Linux ed40ad4018bd 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.4 / 1f110c1da42f607b2454a13d68074dc88879874b

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6545/1/testReport/

Max. process+thread count
551 (vs. ulimit of 5500)

modules
C: hadoop-project U: hadoop-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6545/1/console

versions
git=2.25.1 maven=3.6.3 shellcheck=0.7.0

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6545/1/console in case of problems.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 05s

Precommit patch detected.

_ Prechecks _

+1 :green_heart:
dupname
0m 01s

No case conflicting files found.

+0 :ok:
codespell
0m 01s

codespell was not available.

+0 :ok:
detsecrets
0m 01s

detect-secrets was not available.

+0 :ok:
shellcheck
0m 01s

Shellcheck was not available.

+0 :ok:
shelldocs
0m 01s

Shelldocs was not available.

+0 :ok:
xmllint
0m 00s

xmllint was not available.

+1 :green_heart:
@author
0m 01s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m 00s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ branch-3.4 Compile Tests _

+1 :green_heart:
mvninstall
91m 46s

branch-3.4 passed

+1 :green_heart:
compile
4m 32s

branch-3.4 passed

+1 :green_heart:
mvnsite
4m 21s

branch-3.4 passed

+1 :green_heart:
javadoc
4m 25s

branch-3.4 passed

+1 :green_heart:
shadedclient
239m 15s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 04s

the patch passed

+1 :green_heart:
compile
1m 58s

the patch passed

+1 :green_heart:
javac
1m 58s

the patch passed

+1 :green_heart:
blanks
0m 00s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
1m 58s

the patch passed

+1 :green_heart:
javadoc
1m 52s

the patch passed

+1 :green_heart:
shadedclient
150m 41s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
4m 32s

hadoop-project in the patch passed.

-1 :x:
asflicense
5m 23s
/results-asflicense.txt
The patch generated 1 ASF License warnings.

422m 00s

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/6545

Optional Tests
dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient xmllint

uname
MINGW64_NT-10.0-17763 e57f08a91a03 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys

Build tool
maven

Personality
/c/out/precommit/personality/provided.sh

git revision
branch-3.4 / 1f110c1da42f607b2454a13d68074dc88879874b

Default Java
Azul Systems, Inc.-1.8.0_332-b09

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6545/1/testReport/

modules
C: hadoop-project U: hadoop-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6545/1/console

versions
git=2.44.0.windows.1

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


adding apm marks to dispatcher

Codecov Report

Merging #153 (a195407) into master (8d2111c) will increase coverage by 0.32%.
The diff coverage is 76.22%.

:exclamation: Current head a195407 differs from pull request most recent head 756e7b3. Consider uploading reports for the commit 756e7b3 to get more accurate results

@@            Coverage Diff             @@
##           master     #153      +/-   ##
==========================================
+ Coverage   42.30%   42.62%   +0.32%
==========================================
  Files          32       32
  Lines        4482     4507      +25
==========================================
+ Hits         1896     1921      +25
  Misses       2586     2586

Impacted Files
Coverage Δ

assemblyline_core/dispatching/dispatcher.py
75.10% <76.22%> (+0.86%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d2111c...756e7b3. Read the comment docs.


[BUG] Does not run on fkms driver on Bullseye RasPi Lite

Please only report one bug per issue!
Describe the bug
With a fresh install of Raspberry Pi OS Lite (Bullseye 64-bit), when attempting to run with the fkms driver it results in an error (see below).
To Reproduce
Fresh install of Raspberry Pi OS Lite (Bullseye 64-bit)
Run update and upgrade
Install python and git
git clone picamera2
Make sure vc4-kms-v3d is enabled in /boot/config.txt
libcamera-hello should work fine
Attempt to run the preview_drm.py example results in error.
Expected behaviour
A preview window should appear
Console Output, Screenshots
[0:01:53.670866595] [2022]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[0:01:53.697089203] [2034]  WARN RPI raspberrypi.cpp:1258 Mismatch between Unicam and CamHelper for embedded data usage!
[0:01:53.698339062] [2034]  INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media1
[0:01:53.754169076] [2022]  INFO Camera camera.cpp:1035 configuring streams: (0) 640x360-XBGR8888
[0:01:53.754749302] [2034]  INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/picamera2/previews/null_preview.py", line 25, in thread_func
    callback(picam2)
  File "/usr/lib/python3/dist-packages/picamera2/previews/drm_preview.py", line 35, in handle_request
    self.render_drm(picam2, completed_request)
  File "/usr/lib/python3/dist-packages/picamera2/previews/drm_preview.py", line 110, in render_drm
    raise RuntimeError("Failed to reserve DRM plane")
RuntimeError: Failed to reserve DRM plane

Hardware :
Pi 4 8GB
Additional context
It runs fine just by changing to the kms driver, except libcamera-hello then won't bring up a preview window.

Hi, thanks for the question. Perhaps the documentation isn't very clear here, or is assuming too much prior knowledge. The DRM preview can only work when X windows is not running. (You can use Ctrl+Alt+F1 to suspend X windows to try this; use Ctrl+Alt+F7 to return again.) Does that answer the question?

Hi, I was under the impression X windows was not running on Raspberry Pi OS Lite. Is that not the case?

Ok, a little more digging into this and it looks like the initial format is XBGR8888 which is unavailable according to some debug printing I added in. I changed it to BGR888 which was an available mode and then it got stuck at Failed to reserve DRM overlay plane and this is because it tried to create a plane of type ABGR8888, which is also unavailable.
I ran the fbset command to show available video modes and here's what it has available.
pi@raspberrypilite:~ $ fbset -i -v
Linux Frame Buffer Device Configuration Version 2.1 (23/06/1999)
(C) Copyright 1995-1999 by Geert Uytterhoeven

Opening frame buffer device `/dev/fb0'
Using current video mode from `/dev/fb0'

mode "3840x2160"
    geometry 3840 2160 3840 2160 16
    timings 0 0 0 0 0 0 0
    accel true
    rgba 5/11,6/5,5/0,0/0
endmode

Getting further frame buffer information
Frame buffer device information:
    Name        : vc4drmfb
    Address     : 0
    Size        : 16588800
    Type        : PACKED PIXELS
    Visual      : TRUECOLOR
    XPanStep    : 1
    YPanStep    : 1
    YWrapStep   : 0
    LineLength  : 7680
    Accelerator : No

Perhaps the correct video modes were removed from the fkms driver or something. Maybe you have some more ideas. It might be as simple as adding some kind of parameter to the fkms driver to enable more video modes.

Hi, sorry that I misunderstood your question initially. I've run through all the different cases and here's what I've discovered.

If you use the kms (not fkms) driver, everything works - both libcamera-hello and Picamera2. Was this the same for you? We recommend using the kms driver wherever possible as the fkms one exists for legacy OS/application support.

When using fkms, libcamera-hello works but Picamera2 does not (as you reported).

The reason for this is that the fkms driver has incomplete support for RGB-with-alpha formats. I believe it supports ARGB but not ABGR which (by unlucky coincidence) is the format the Picamera2 often defaults to, and also uses all the time for overlays. The former can be overridden (as you've done), but not the latter which is hard-coded.
So in the first instance, obviously I would recommend using kms if you can, and if there's a problem with that we would of course investigate it. Alternatively, you could run your own version of Picamera2. You simply need to clone the repository and list it in your PYTHONPATH environment variable. Then you could edit the overlay format here. I'd be open to PR to make this configurable, though I'd be keen to ensure that all the preview implementations retain identical functionality.

Ah, ok. I'm trying to do something with picamera2 that I was able to with the first picamera, which is to show the output on a TFT display that is only connected through SPI. To do that, it is using FBCP, which I'm not sure how long that will work for, but it's the only thing working at this point. When I tried through the KMS driver, I get the same Failed to Reserve Plane error when no HDMI cable is plugged in and it outputs to HDMI only when it is plugged in.
So, making this a setting that could be overridden would be super helpful. I'm trying to put the instructions on how to do this into a guide for other folks, so having while having it available only in a custom fork would certainly be an option, it's not ideal because I would like to make this setup as easy for other folks as possible. I would be happy to help with testing out changes for this.
By the way, you've done an awesome job with rewriting picamera2 and while it loses some backwards compatibility, it does things in a much better way and I found the code well written that I could follow along for the most part.

Also, I did try and edit it, but stopped after an error occurred a few lines down for the line self.overlay_plane.set_prop("pixel blend mode", 1). I don't recall the exact error message though.

I'll put this on my list to have a look at, though I have a few other things to do first. In the interim, I'd have thought it would be enough just to replace ABGR8888 on that line by ARGB8888 and leave everything else the same...

Oh awesome.

So I think all the changes for this should be in place now. You'll need the latest version of Picamera2 (sudo apt install -y python3-picamera2 which should get you version 0.3.5). You'll also need to do sudo rpi-update and then reboot. The usual warnings apply with rpi-update - it will fetch the most bleeding edge firmware so don't do it with an SD card that has anything on it that you can't afford to lose!

Awesome. I'll try it out when I'm feeling a bit better and am back at my desk.

Ok I tested this without doing rpi-update and verified picamera2 was at 0.3.5 with pip show picamera2 and it seems to be working. Thank you so much.


Enhancing the contributor Section and Structuring

🛠 Fixes Issue
Fixes: #576
👨💻 Description
What does this PR do?
(Enhancing the Contributor Section and Structuring)
This PR enhances the contributor section in the README by:

Updating the contributor graph to use the Vaunt API, providing a visually dynamic representation of all contributors.
Structuring the contributor section for improved readability and visual presentation.
Add responsive alignment to ensure the contributor images display well on mobile and desktop.

These improvements enhance the overall look and feel of the README, making the contributor section more engaging and accessible.
📄 Type of Change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation update (adds or updates related documentation)

📷 Screenshots/GIFs (if any)

✅ Checklist

[x] I am a participant of GSSoC-ext.
[x] I have followed the contribution guidelines of this project.
[x] I made this change on my own.
[ ] I have taken help from some online resources.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my code.
[x] I have added documentation to explain my changes.

Mandatory Tasks

[x] I have self-reviewed the code to ensure quality before submitting it.

🤝 GSSoC Participation

[x] This PR is submitted under the GSSoC program.
[x] I have taken prior approval for this feature/fix.

@YadavAkhileshh  the task has been completed. Kindly review the PR


OOTB support for HF models

From req doc: 2d
P0: Users must be able to serve models from Hugging Face without having to do any additional conversions or configurations
Acceptance criteria:

[ ] Test TGIS image with HF model
[ ] Share instructions

TGIS image is ready to be tested with HF models.

@heyselbi I'll use the ServingRuntime definition from https://github.com/opendatahub-io/caikit-tgis-serving/pull/132/files. Is that correct?

@bdattoma that is correct. Let me know if you run into any issues.

@bdattoma that is correct. Let me know if you run into any issues.

okay thanks. I'll defer testing this since this ticket got removed from 2.5. I'll be back on it after 2.5


Can't parse model from keras2onnx with protocol buffers

I created a model with the following code, and exported it to an onnx file
from tensorflow import keras
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Flatten
import onnx
import keras2onnx

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])

onnx_model = keras2onnx.convert_keras(model, model.name)
onnx.save_model(onnx_model, 'tf1.onnx')

Now, I am trying to load it with my own code in c++, using the following code:
std::string path = 'tf1.onnx'
// Verify that the version of the library that we linked against is
// compatible with the version of the headers we compiled against.
GOOGLE_PROTOBUF_VERIFY_VERSION;
onnx::ModelProto model;

{
    // Read the existing net.
    fstream input(path , ios::in | ios::binary);
    if (!model.ParseFromIstream(&input))
        cerr << "Failed to parse model." << endl;
}

After running the c++ code, I get "Failed to parse model", that means that the parsing fails. This works for several other models parsed from other libraries, like pytorch, mxnet or eddl.
I have tried using onnx.proto and onnx.proto3 from https://github.com/onnx/onnx/tree/master/onnx to generate the headers and the source used for parsing the model, but none will work. Do you have any idea of why does the parsing fail?
Thanks in advance

Can you compare your C++ implementation to this https://github.com/onnx/onnx/blob/e2e12d97af735b8d25ecdbd8cf013c18d5b8d44f/onnx/checker.cc#L775 understand why?

The problem was with some dependencies. Having installed two different versions of protobuf in root and a conda environment created some unseen conflicts that effected this error. Nevertheless after removing both versions and reinstalling protobuf it ran fine.


Pagination example indicating page count

Again, at least two people have asked for this as an example.

:+1:

Since this is just a non-normative example, and it uses an approach that both @tkellen and @steveklabnik endorsed in #384, I'm just gonna merge it. Feel free to revert/amend if necessary.


Added the Fudget, the popular budgeting app. Original iOS version was…

… created using Cordova and has received over 300,000 downloads. Android version was later released. Now here is the Mac version, creating by adapting the original codebase and wrapping in Electron. Thank you Electron team for creating such a great framework!

Thanks, @fruitangdan
I'm not able to access https://bitbucket.org/fruitangdan/fudget -- is it a private repo?

Hi Zeke,
Thanks for your message.
Yes. It is private. Should we remove it from the yml file?
Cheers,
Danny

On 13 Mar 2017, at 19:16, Zeke Sikelianos<EMAIL_ADDRESS>wrote:
Thanks, @fruitangdan
I'm not able to access https://bitbucket.org/fruitangdan/fudget -- is it a private repo?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Should we remove it from the yml file?

Yeah

Thanks. I've removed the repository field.


Track upload/download progress

I really miss this feature, because I'm developing a simple upload application.
I think this would be an interesting feature.
Moved here from a Requests bug

See my answer here: https://github.com/kennethreitz/requests/issues/396#issuecomment-3709921

This is simple to do.
Look at the content-length of the response and download 1/100th of that at a time.

+1 to what @johtso and @kennethreitz said.
@gcastagneti This "recipe" is obviously not very well documented for neither Requests or urllib3. Perhaps you'd like to write a blog post tutorial or doc for how to do this so we can refer others to it?


WordCountVectorizer Memory Issue

Hello,
I noticed during training a model with WordCountVectorizer for category classification and english stemmer with a vocabulary of 3k words and a small MLP model my ram usage jumps to 26 gbs. Do we know what's causing this? in order to get similar ram usage in python i need to make a vocabulary of 60-70k words approximately. Am i doing smth wrong maybe?
Best Regards and thanks for the hard work!

I'll have to take another look at the SciKit Learn Count Vectorizer (I'm assuming this was the one you were using in Python) but at first glance there's a difference in the datastructures used under the hood that has a big effect on memory usage.
https://github.com/scikit-learn/scikit-learn/blob/364c77e04/sklearn/feature_extraction/text.py#L931
Rubix ML Word Count Vectorizer uses PHP arrays under the hood whereas SciKit Learn's Count Vectorizer is using sparse NumPY arrays under the hood. I'm pretty sure int/float scalars in PHP arrays occupy > 128 bits when you account for the 64 bit float/int plus the extra zval "metadata" like reference count, and then another 64 bits to store the index. In contrast, NumPY arrays do not store a separate index, nor any extra metadata, and scalars need not be 64 bit, they can go as low as 8 bits I believe. Combine that with a sparse implementation (0's are not explicitly allocated in memory) you get a huge memory savings with the Python implementation when used to represent word count vectors.

Alright thanks for the answer!


[RFC] Add Option::{todo, unreachable} and Result::{todo, unreachable}

This RFC proposes Result::{todo, unreachable} and Option::{todo, unreachable} functions which work like .unwrap() but imply different semantic reasons for the unwrapping.
.todo() implies that error handling still needs to be done and that unwrapping here is a temporary thing that should be fixed in the future. This is analogous to todo!().
.unreachable() implies that we are unwrapping because we believe the Err or None case cannot occur. This is analogous to unreachable!().
In short, this allows users to preserve semantic information about why they are unwrapping. This proves extremely useful in prototype code, and in distinguishing "unreachable unwraps" and "todo unwraps" from "panic unwraps".
As an example:
// unwrap is still used for the analogous-to-panic!() case
TcpListener::bind(&addr).unwrap();

// we're panicking because error handling is not implemented yet.
// this use case is common in prototype applications.
let int: i32 = input.parse().todo();
let arg2 = std::env::args().nth(2).todo();

// these error states are unreachable.
// this use case is common in static declarations.
NonZeroU32::new(10).unreachable();
Regex::new("^[a-f]{5}$").unreachable();

Rendered

This has a better chance as an ACP (API change proposal) over at https://github.com/rust-lang/libs-team/issues.

As an initial reader, I really like todo.
My concern with unreachable is confusion with https://web.mit.edu/rust-lang_v1.25/arch/amd64_ubuntu1404/share/doc/rust/html/std/intrinsics/fn.unreachable.html (std::intrinsincs::unreachable), which is unsafe, and marks that the code as unreachable, and can lead to UB if the code is reached.

Isn't it called unreachable_unchecked? I notice you linked a docs page from 1.25 which is quite a while ago. (EDIT: actually I see you linked the name of the core intrinsic, rather than the actual function in std. I'm pretty sure there is a pretty strong rule that for public API at least, _unchecked is required for functions with a narrow contract like this.)

Sorry, I should have checked the version -- I remembered seeing, and using the unreachable intrinsinc (although it was experimental), and didn't realise it was renamed when stabilized.

The unreachable! macro indicates a line of code that should never be reached by the flow of execution, but Some(4).unreachable() does get reached and, in fact, does some work (unwraps the Option). Mostly I think expect is adequate here, but otherwise, maybe something more like Some(4).assert_some()?

The unreachable! macro indicates a line of code that should never be reached by the flow of execution, but Some(4).unreachable() does get reached and, in fact, does some work (unwraps the Option). Mostly I think expect is adequate here, but otherwise, maybe something more like Some(4).assert_some()?

 I'm not 100% sure about assert_some/assert_ok because they don't 100% convey the intention of "i'm unwrapping BECAUSE this case cannot happen". In my mind, an assertion is something that could happen but implies the program is in a screwed state (which is analogous to panic!()).
We could combine these a bit to get err_unreachable() and none_unreachable(), which are obvious names that convey intent, but there aren't a lot of function names that look like that.
unreachable_err is sadly unusable because unwrap_err means something completely different.
unwrap_unreachable is also an option.

In general, I think .todo() should be an acceptable name, but .unreachable() could do with some bikeshedding. It looks wrong - it looks like that line should not be reached, but what it's actually saying is that the error is unreachable.

I've updated the RFC to remove Option::unreachable() and Result::unreachable() from consideration.
After reading some comments and thinking over it more, I don't think .unreachable() provides that much benefit to Rust. It can be simulated with .expect("reason for unreachability") quite easily and there are not many benefits for having a dedicated method for it.
There are also many more points of contention in the naming of .unreachable(); it's pretty confusing, not very good, and needlessly distracts from .todo(), which I think is a much stronger improvement.
I'm looking into moving this over into an ACP now. I had no idea that ACPs were a thing; they're not mentioned anywhere in the rfc readme :(

i think you'd also want a todo_err or todo_ok which is the equivalent of unwrap_err

From what I understand, .expect is supposed to imply unreachable. If it is reachable, it should have been handled instead of panicking.
But how is .todo() any better than .expect("TODO")?

But how is .todo() any better than .expect("TODO")?

IDE could mark .todo differently from .expect
(similar to the justification that IDE could mark todo! differently from umimplemented! see https://doc.rust-lang.org/stable/std/macro.todo.html)
Also, clippy could warn .todo separately from .expect

Isn't unwrap… the same, as unreachable?
Like, the whole point of calling unwrap is to say "I'm so confident that this Option is Some that I'm not even going to provide an error message in case it isn't" and while it's unfortunate that the output isn't very similar to unreachable!(), that is what the code means.
I kind of like the idea of todo, but as folks have said, expect("TODO") or expect("FIXME") works just as well, and most IDEs will automatically highlight those strings anyway regardless of context. So… 🤷🏻

Also, clippy could warn .todo separately from .expect

clippy could easily warn .expect("TODO") differently from other generic .expect(...)

Isn't unwrap… the same, as unreachable?

But it's not. Both compile to an explicit panic!(), but both have different messages, and signal different intent in code.
.unwrap()' *could* and often does mean you're confident it won't happen, yes, but it could also mean there's no meaningful behavior so the best thing is just to crash. Or it could mean todo!() as above. But more importantly, "not yet implemented" is a very different message from "Option::unwrap()called onNone`".

Having some functions doing exactly the same thing except for some artificial "semantic differences" seems strange to me, especially when the motivation can already be addressed to some extent by simply comments.

Having some functions doing exactly the same thing except for some artificial "semantic differences" seems strange to me, especially when the motivation can already be addressed to some extent by simply comments.

But how is .todo() any better than .expect("TODO")?

clippy could easily warn .expect("TODO") differently from other generic .expect(...)

It is a bit strange, but I don't think this is without precedence in Rust. All of these arguments could -- too -- be applied onto the todo!() macro.
One can rewrite all panic!()s with panic!("todo") to get a similar result, but that gives you:

a worse error message (worse than "Not yet implemented", cannot be localised either)
is more prone to typos (also, Todo? TODO? ToDo? I've seen all of these in comments).
Can be inconsistent across languages (.expect("する") I've seen する used before in other languages in lieu of todo)
is likely to be forgotten or not done because it's additional work

And the sort of ugly part here is that if the solution for an analogous-to-todo-unwrap is notably more difficult to write than unwrap, I suspect it will ignored frequently, like the situation we're in today.
At the moment, I rarely see .expect("TODO") in work or at home. I don't write it myself because it's more annoying than writing .unwrap().
Although today you can do all of these things:
// this is 27 characters long; no prototype code will ever realistically go through the hassle of writing this
let int: i32 = input.parse().unwrap_or_else(|| todo!());

// This is 19 additional characters and doesn't chain.
// this is useful in Non-Option/Result cases though, where you want to pattern match a single variant.
let Some(arg2) = std::env::args().nth(2) else { todo!() };
// i.e. is not easily possible
let thing = getstate().todo().thing

// And storing important information in comments just never seems to work in practice,
// it falls out of sync with the code and isn't easy to grep for.
// People forget, people don't do it, and there's no guarantee that comment is in english either.

I'll throw out that this could easily be a tiny crate using an extension trait.  Going into STD is a high bar.  If this is so necessary it seems to me that one could make such a crate and prove the popularity first.
That's not a yes or a no, I'm just saying the obvious thing.  I'm sort of meh one way or the other.  However to me if it can't become a popular crate and it isn't something that the compiler itself has to help with, then it doesn't seem to hit the bar to go into std in my opinion.

I'll throw out that this could easily be a tiny crate using an extension trait. Going into STD is a high bar. If this is so necessary it seems to me that one could make such a crate and prove the popularity first.
That's not a yes or a no, I'm just saying the obvious thing. I'm sort of meh one way or the other. However to me if it can't become a popular crate and it isn't something that the compiler itself has to help with, then it doesn't seem to hit the bar to go into std in my opinion.

Sure. I have this functionality in a project at the moment; I can shape it up into a crate.
If this crate catches on with this name -- does that block that name from going into STD? I remember hearing some similar issues with popular Itertools constructs effectively blocking the name from going onto Iterator.

If this crate catches on with this name -- does that block that name from going into STD? I remember hearing some similar issues with popular Itertools constructs effectively blocking the name from going onto Iterator.

The main problem with Itertools is that both are trait methods, and there’s no supertrait shadowing yet, so you’ll get ambiguity errors. An inherent method will take precedent, so unless the extension trait has a different type signature (e.g. like here), it shouldn’t create any issues.

If this crate catches on with this name -- does that block that name from going into STD? I remember hearing some similar issues with popular Itertools constructs effectively blocking the name from going onto Iterator.

The main problem with Itertools is that both are trait methods, and there’s no supertrait shadowing yet, so you’ll get ambiguity errors. An inherent method will take precedent, so unless the extension trait has a different type signature (e.g. like here), it shouldn’t create any issues.

Awesome. I'll publish my crate for this tonight.

This is now published as a crate: https://github.com/zkrising/unwrap_todo (cargo add unwrap_todo).

I think the unreachable variant has merit too. I often find myself writing: .unwrap_or_else(|| unreachable!()). Which is very verbose, and makes the code less readable (especially when the line gets long enough that Clippy formats it in chain form).

I think the unreachable variant has merit too. I often find myself writing: .unwrap_or_else(|| unreachable!()). Which is very verbose, and makes the code less readable (especially when the line gets long enough that Clippy formats it in chain form).

Unwrap todo seems great. I remember when I first learned Rust, I only knew panic!, but then I found todo! and unreachable!. They are not life changing, honestly, but I do use them instead of panic! as I want my code to more clearly express my intention and thinking process.

Going into STD is a high bar. If this is so necessary it seems to me that one could make such a crate and prove the popularity first.

That said, would/will I add a crate just for todo! and/or unreachable!? No.....
I've seen a lot of APIs gone into std because they are just convenient to use or express intention. When I saw them stabilized, I first got surprised, just like you could just do X instead, why need this, but then I am really glad that they was added when I use them.

I can suggest the rules that I use in my code:

Use expect as panic and write a message why this code is actually will not panic
Use expect as unreachable and write a message why this code is actually non unreachable
Use unwrap as todo

In tests and documentation examples the ? should be used.
You can also add this lint to the config:
[lints.clippy]
unwrap-used = "deny"

It would be great to rename unwrap to todo, but unfortunately unwrap is already widely used. However, I think it's worth to reduce usage of unwrap. expect is always more readable and produces clearer error messages. unwrap is ok in rapid prototyping code, this implies its todo semantics in more production code

I cannot agree with this suggestion.
First, there could possibly be an arbitrary large set of possible reasons why an author would use .unwrap(), are we going to take care of every single possible reason? todo() and unreachable() are the only two that the author of this RFC can think of currently, if he comes out with more, are we going to add more methods to Result and Option?
Second, todo!(), unreachable!(), coupled with unwrap_or_else() already cover the scenarios the author of the RFC raised, why add methods to it?
Third, unwrap() has monadic origins, Result and Option are monads, from a type theoretic perspective, there isn't a need to include new methods that behaviour the same as unwrap() that serves no more purpose than what unwrap() already did.
This sounds like the RFC author is desperate to 'contribute' to Rust.

This sounds like the RFC author is desperate to 'contribute' to Rust.

Please don't do this here. This is uncalled for.

This sounds like the RFC author is desperate to 'contribute' to Rust.

Please don't do this here. This is uncalled for.

Apologies, let me withdraw it.

I've been reading all the responses to this and I wanted to point out two things I found sort of interesting:
Temporary panicking in prototypes is common
That is to say, using something for temporary error handling for prototype code isn't a niche thing. A lot of people in those discussions have expressed their usage of something for temporary handling.
A lot of people have chimed in with what they do in their codebases for these semantics. I think that implies heavily to me that there is demand for this semantic, but since the language doesn't provide a standard choice, people assign their own semantics ontop of the existing APIs.
That suggests, to me, that the semantics I'm describing in the OP are at least "real" -- "unwrap todo" is something that people already want to express, and I'm not just suggesting something nobody uses :sweat_smile:.
People have different ways of expressing it
If the above point establishes that "just panic here temporarily" is a real thing people want, then the rest of the discussion goes to show that there's not an agreed way of expressing it.
I've seen, so far:

.unwrap() implies temporary handling, use .expect("reason") for anything deliberate.
.unwrap() is for deliberate cases that need no further explanation, use .expect("todo") to indicate todo-semantics.
.unwrap() implies unreachability, use .expect("todo").
Just use comments above unwraps to attach semantics.
Use .unwrap_or_else(|| todo!()), or let-else, and some others...

There's nothing necessarily wrong with this, but there's a lot of talking-past-eachother in these threads; it seems to me like there's not a consensus on what .unwrap() is even supposed to mean in code. Some people think it should only be used for todo-semantics, some think it should be used for self-evident .expect().
The standard library docs don't seem to take a stance on this either, which likely contributes to this confusion.
I think the discussions (at least, to me) evince .unwrap() being semantically overloaded?
I'm writing a bit too much in this thread now, but I did want to point this out!

Semantics aren't the same as...I don't have a good word, let's go with conventions.  You're suggesting effectively multiple names for the same semantics and that they be so blessed that they belong in stdlib.
I would go so far as to also say that panicking in prototypes isn't great either. You can throw Anyhow or similar at many of these problems and then it's just ? your problems away.  There's an implicit assumption that panicking like that needs to be common.
As I already said it's possible to do this as a crate.  If such a crate became popular I'd be less negative about this.  You're showing that people like to express temporary panicking, but just because people can't agree on .expect("todo") versus whatever else doesn't mean that it's the place of Rust to impose a specific style, nor does it mean that enough people are going to find and adopt it either given that almost a decade of learning material teaches unwrap.  You're implicitly arguing:

Panicking in prototypes is the best way and should be encouraged.
We need a consistent style for it
Your favorite style is the right style for it
This is important enough that having a bunch of functions that do the same thing with different names, one of the basic "don't do this" coding practices, is worth it.
Enough people will find it so that it becomes widely adopted and people won't just keep doing their own thing.

That's a whole lot of points to argue.  You can argue the importance of them, but they're all present in the proposal here.  There's also an implicit assumption that people who hang out on RFC threads on GitHub are representative as well.
I will personally ignore these if they landed but that's because I just don't unwrap like this.  Anyhow and ok_or_else solve the problem quite nicely and don't require a big refactor/cleanup later and the code can move from some form of prototype toy to some form of production by just not panicking way up at the top in main.  If it was my job to make this decision I'd honestly start at requiring you to argue me into why we should encourage this kind of code.

Semantics aren't the same as...I don't have a good word, let's go with conventions. You're suggesting effectively multiple names for the same semantics and that they be so blessed that they belong in stdlib.
I would go so far as to also say that panicking in prototypes isn't great either. You can throw Anyhow or similar at many of these problems and then it's just ? your problems away. There's an implicit assumption that panicking like that needs to be common.
As I already said it's possible to do this as a crate. If such a crate became popular I'd be less negative about this. You're showing that people like to express temporary panicking, but just because people can't agree on .expect("todo") versus whatever else doesn't mean that it's the place of Rust to impose a specific style, nor does it mean that enough people are going to find and adopt it either given that almost a decade of learning material teaches unwrap. You're implicitly arguing:

Panicking in prototypes is the best way and should be encouraged.
We need a consistent style for it
Your favorite style is the right style for it
This is important enough that having a bunch of functions that do the same thing with different names, one of the basic "don't do this" coding practices, is worth it.
Enough people will find it so that it becomes widely adopted and people won't just keep doing their own thing.

That's a whole lot of points to argue. You can argue the importance of them, but they're all present in the proposal here. There's also an implicit assumption that people who hang out on RFC threads on GitHub are representative as well.
I will personally ignore these if they landed but that's because I just don't unwrap like this. Anyhow and ok_or_else solve the problem quite nicely and don't require a big refactor/cleanup later and the code can move from some form of prototype toy to some form of production by just not panicking way up at the top in main. If it was my job to make this decision I'd honestly start at requiring you to argue me into why we should encourage this kind of code.

Agree with you. Is there a standard protocol to close RFC or a way to deem a particular RFC is not worth pursuing? OP could have made his suggestions into a crate and use the empirical evidences (if any) to back up his claim.

Agree with you. Is there a standard protocol to close RFC or a way to deem a particular RFC is not worth pursuing?

This is kind of rude. If you agree with someone, you can just say that (or better yet, just use :+1: and don't say anything).

In my experience of writing academic scientific software, particularly during early development, panicing by unwrap()/expect() in situations where we don't really know how to recover, leads to better quality code than using ?.
If a program panics before finishing, users assume it "did not work", they ignore any partially produced outputs, and they investigate why the panic occurred. This is good, because it means we aren't trusting the possibly incorrect output.
While it's as easy to write '?' and pass the problem up, it's hard to test and get it right. I've seen programs which, (for example), when a hard drive gets full output only partial output but appear to finish successfuly. Or, programs which try, and fail, to recover correctly from unusual error situations, like poisoned mutexes.
If I'm not carefully testing a particular error state, I find it better to 'unwrap' and explode, than believe I'm handling it with ?, but not really bother testing I'm handling it correctly (and it's hard to test things like poisoned mutexes).
Of course, when you are working on a battle-hardened, polished, library intended for use by many people, you add enough tests you can be sure it never panics, and also behaves correctly, but that's a huge amount of work!

@jymchng .expect() exists, when you could just as easily write .unwrap_or_else(|| panic!("")). And, as you have pointed out yourself, todo! is already a macro even though one could just as easily write panic!("todo"). So prior art to extending type-theoretical methods and adding language features for convenience and brevity is there.
Regarding developmental and maintenance burdens, we are discussing whether adding todo() would result in additional maintenance burden to the Rust core team. Looking at OP’s implementation, this is a one-line method calling the panic! macro. This seems quite stable and low-maintenance to me. I put it to you that if the behaviour of the panic! macro changes to such a degree that it breaks this implementation, the Rust community has much larger issues than what is essentially an alias on two types.

@jymchng .expect() exists, when you could just as easily write .unwrap_or_else(|| panic!("")). And, as you have pointed out yourself, todo! is already a macro even though one could just as easily write panic!("todo"). So prior art to extending type-theoretical methods and adding language features for convenience and brevity is there.
Regarding developmental and maintenance burdens, we are discussing whether adding todo() would result in additional maintenance burden to the Rust core team. Looking at OP’s implementation, this is a one-line method calling the panic! macro. This seems quite stable and low-maintenance to me. I put it to you that if the behaviour of the panic! macro changes to such a degree that it breaks this implementation, the Rust community has much larger issues than what is essentially an alias on two types.

Rust is a language of longevity. It is precisely this mentality that got C++ to become this bloated with almost a 100 keywords and thousands of public APIs to maintain. Today, you are thinking that it is only an additional of two one-liner methods - others would think the same, and in no time, the standard library would be bloated with hundreds of similar APIs. panic!() macro is a mechanism to call the compiler-implemented panic workflow, its existence is justified. If we don't have todo!(), I would find this RFC more acceptable. But since we already have this, I still find the reasons to add two new APIs less convincing.

As a libs-api member, I'm not worried about the maintenance burden here. It's definitively a non-issue. Type theoretic concerns are also not going to be factor. (And Result isn't a monad.)

Thirty years ago a C++ maintainer probably thought the same as you did now and it is those who are maintaining it now suffering. The standard library should be very concise and small and the maintainers, in my humble opinion, should be hyper selective and most importantly, prudent, on what gets into the standard library.

Yes, then how do we decide which are the methods that we ought to add and which we should not add?

That is the point of an RFC!
Nobody is proposing a #[todo] attribute or anything like that. The RFC covers exactly what I am proposing and that is .todo().
.unimplemented() and .unreachable() are mentioned, as they draw from the same thing (the existing panic macros). Nothing else is mentioned. Emphatically, there is nothing in this RFC about a #[todo] attribute.

if he comes out with more, are we going to add more methods to Result and Option?

I am not the arbiter on what methods should be added to the language. If I were to propose more things, they would go through the same process.
I don't think it's fair to criticise this RFC based on things it isn't even proposing!

there're two branches in unwrap/expect

(a) unwrap: to move something out of container
(b) panic: maybe dangerous to do that

a good method name should matches both of them.

unwrap isn't a good name, because it matches (a) but not (b).
expect isn't a good name, because it matches (b) but not (a).
todo is a bad name, because it matches neither (a) nor (b). (to do what? who knows?)
unwrap_or_panic is good, because it matches both (a) and (b), but it's too long.
defuse is a better name, because it matches both (a) and (b) and it's short.

So I'd like to propose Result::defuse() Option::defuse().

@liigo However poorly chosen an existing name is, it is very unlikely that a simple renaming of an existing method will be accepted for the standard library.
This RFC is about adding a new method, albeit it being equivalent to an existing method.

I just looked at this again, and I realized string.parse().todo() is actually very ambiguous. It is unclear whether todo refers to handling the Ok or Err variant of the result. Would todo_err and todo_none be better names to show that we are actually treating the negative branch as a todo?

I just came across the blog post Rust’s design goals should be about code. I think however it's actually implemented ("what name"), this RFC fits into the goal of "Clarity of purpose", i.e., making clear that this Option or Result is for now out of concern but needs revision later.

I just looked at this again, and I realized string.parse().todo() is actually very ambiguous. It is unclear whether todo refers to handling the Ok or Err variant of the result. Would todo_err and todo_none be better names to show that we are actually treating the negative branch as a todo?

This a very good point, similar to my comment about the proposed unreachable() method. In terms of clarity it is not clear that todo() handles the Err/None variant. It could suggest that the processing of the Ok/Some variant is still to be done. (Result::unwrap has the same problem but that ship has sailed.)
Expanding the method name to add clarity, such as using todo_err(), chips away at the advantage that todo() is about as easy as unwrap() to type.
I've felt ambivalent about todo() but this makes me lean towards not proceeding. I think it would be better to keep unwrap() to indicate that code is rough and not production-ready (the use case for todo), and work towards better methods for production code that currently uses unwrap. Something like the unwrap-infallible crate or an assert_some() method.

I just looked at this again, and I realized string.parse().todo() is actually very ambiguous. It is unclear whether todo refers to handling the Ok or Err variant of the result. Would todo_err and todo_none be better names to show that we are actually treating the negative branch as a todo?

I'm not sure if .todo on its own is ambiguous; I think .todo() maps clearly onto .unwrap()/.expect(), which themselves don't specify they're for the Positive path.
A really big problem with .todo_err/.todo_none as names is that they are ambiguous depending on how you read them.
If you mentally see .todo as a synonym for unwrap, then .todo_err looks like .unwrap_err, which means "panic on Ok".
Whereas if you read .todo_err as "todo, handle err", then it means "panic on Err".
This alone I think will be the source of a lot of confusion if they were added like this, .todo_err either directly conflicts with .unwrap_err as a name, or directly conflicts with how it reads.
It actually took me a minute to realise what you were proposing was using .todo_err to mean "todo, handle err" rather than akin to .unwrap_err :sweat_smile:.
As for handling the negative paths, I struggle to think of a case where someone would want to write todo for a None path or Err path. We might want to look into them for completeness' sake, but there's some fairly decent naming concerns here.

I'm not sure if .todo on its own is ambiguous; I think .todo() maps clearly onto .unwrap()/.expect(), which themselves don't specify they're for the Positive path.

For Option, unwrap/expect is unambiguously the Some case since there is nothing to unwrap/expect for the None case (ok, you can still argue that we want to expect_none...)
For Result, writing get_data().unwrap() is probably clear that we want to unwrap the data and binary_search().expect("element must exist") actually expects the condition inside the string. But I agree these are ambiguous.
But read_file().todo() is magnitudes more confusing. This sounds like we are trying to implement some logic on the read contents instead of the error. Option/Result generally represents the positive type (especially considering the many type aliases for Result, e.g. io::Result<String> clearly represents "a result of string"), so verbs used on options/results naturally refers to the positive type. Meanwhile saying that "we have a result of bytes, and this result of bytes is TODO" makes no sense.

What about ok_or_todo and/or unwrap_or_todo?

todo!() doesn't exactly mean "panic todo" to me though. For me at least, it means "todo, just let this code compile and do whatever not insane in place of it"

It would be great to have something like this. I often find myself thinking, "Why isn't there a tool, like todo!() for Result and Option, that can remind me to handle it later?" Sure, you can set deny rules, but am I really going to deny unwrap everywhere when I have mutex locks scattered around using unwrap? This feature would be perfect for building quick applications where I can focus on error handling later. It's frustrating to have to worry about it right from the start, especially when I might forget an unwrap I used or end up searching through files for an unwrap I left in while testing.

What's wrong with Option::unwrap_or_else? todo! is a macro, so the signature wouldn't even be clear. Should it be &str or format_args!, latter wouldn't even be more ergonomic. There is even a clippy lint for expect(format!), but I guess former can be inefficient, since it should be only used or todos. What about Option::unimplemented, which I guess would have to be efficient, since it  would be used in application code?
With todo!, unreachable! and unimplemented! there are already 3 similar ways to panic, so why add more?

As a not sophisticated Rust user (I follow many of the weird theoretical discussions about the language, but I rarely engage, as my real use cases are pretty simple) I'm sympathetic to the goal of being able to write very clear and concise code (where the intent is always "self-evident", or almost).
I understand that .unwrap() might not feel as enough given that many people have slightly different understanding of the intent behind its usage and how and when to use it...
But .todo() seems to me like a strange choice of words to convey the specific intent expressed in this RFC. I see how it relates to the todo! macro, but that's the only reason I can understand what .todo() does without having to go into its documentation, I think it shouldn't be like that.
In short, I support the idea of continuing the exploration for better names, even if this leads to a small amount of redundancy on the std lib side, but I also think that RFCs like this one (which one should expect to be controversial) should have some extra work behind before being pushed:

a wider exploration of possible names, and explanations on why ones have been discarded and others chosen
as others have pointed out, maybe publish a crate and let it be used for some time to gather objective empirical knowledge that can be used to support the proposal.

It may be nice if you could shove a bug number or arbitrary string in the arguments.

Rephrasing @leb-kuchen's comment, .unwrap_or_else(|| todo!()) totally satisfies the "IDE could mark .todo differently from .expect" requirement as @lebensterben mentioned above as well. So .todo() only serves the purpose of conciseness, but if we cannot come up with a good, short, clear name, it would be no better than .unwrap_or_else(|| todo!()).

The link in the original post to the crate is incorrect. It should be https://github.com/zkrising/unwrap_todo instead of https://github.com/zkrising/unwrap-todo.


Explorer status is offline

##Expected Behavior
start the explorer with the status online as everything is running
Actual Behavior
I have my api running on port 3000 and it is reachable from my browser, but the Explorer still shows as Status Offline.
yarn start:web is running without any error.
/api/coin is showing online as status
I followed every stepd from https://github.com/bulwark-crypto/bulwark-explorer/issues/77 but that did not help
Steps to Reproduce the Problem
1.Install following the readme
1.yarn start:api
1.yarn start:web
Specifications

Version: Latest
Platform: Ubuntu 16.04
Subsystem:

Fixed with the help from discord. For starting it on localhost the workerport has to be the same as the api port