REPL command returned after creating variation does not create the same image

Describe your environment

GPU: mps
VRAM: 32GB Unified Memory
CPU arch: arm64 M1
OS: macOS
Python: mamba env create -f environment-mac.yaml
Branch: development
Commit: i do not have a line that starts with "Merge", but I am up to date with development

Describe the bug
After generating an image, the REPL returns the command (parameters) which may be used to recreate that same image.
When variation amount is specified, the command returned does not recreate the same image.
To Reproduce

Run python scripts/dream.py
Generate an image using -v e.g. test -s 5 -S 0 -v 0.1, call it "Image 1"
It returns a command e.g. outputs/img-samples/000070.810716037.png: "test -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 810716037 -n 1 -f 0.75 -V 810716037:0.1
Copy that command and paste back into the REPL
A different image is generated, call it "Image 2"

Expected behavior
"Image 1" and "Image 2" are identical.
Additional context
Maybe I am misunderstanding how variations work

This could either be a new bug I introduced during the RFC #266 work (i.e. the prompt string is wrong), or an older bug regarding reproducibility problems in random number generation on M1 systems. Could you compare the behavior against the main branch so that I can distinguish those possibilities?

Same issue on main.
REPL command 1: test -s 5 -S 0 -v 0.1
REPL output 1: [1] outputs/img-samples/000079.1485615218.png: "test" -s5 -W512 -H512 -C7.5 -Ak_lms -V1485615218:0.1 -S0
Result image 1:

REPL command 2: "test" -s5 -W512 -H512 -C7.5 -Ak_lms -V1485615218:0.1 -S0 (same as output 1)
REPL output 2: [2] outputs/img-samples/000080.1485615218.png: "test" -s5 -W512 -H512 -C7.5 -Ak_lms -V1485615218:0.1 -S0 (same as command 2)
Result image 2:

Subsequent runs of the command yield identical images.
Does the expected behavior occur on non-M1 systems?

@mh-dm and @Any-Winter-4079 . Is this a manifestation of the non-reproducibility problem on MPS we were having earlier, or something separate?
I'm pretty sure I've tested reproducibility of variation on Linux and it is OK, but I'll check later today before declaring this to be an M1 specific bug.

@psychedelicious @lstein I did get the same image using
"test" -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 810716037 -n 1 -f 0.75 -V 810716037:0.1
several times

Ccing: @bakkot @blessedcoolant @xraxa
This turns out not to be an M1 specific problem, but a general issue for the variation-generating code on all platforms. If I run this identical command repeatedly I get different images even though I am setting the same seed each time:
dream> test image -S42 -v0.1

The culprit is L104 of ldm/dream/generator/base.pm, indicated by the arrow here:
    def generate_initial_noise(self, seed, width, height):
        initial_noise = None
        if self.variation_amount > 0 or len(self.with_variations) > 0:
            # use fixed initial noise plus random noise per iteration
            seed_everything(seed)
            initial_noise = self.get_noise(width,height)
            for v_seed, v_weight in self.with_variations:
                seed = v_seed
                seed_everything(seed)
                next_noise = self.get_noise(width,height)
                initial_noise = self.slerp(v_weight, initial_noise, next_noise)
            if self.variation_amount > 0:
==>          random.seed() # reset RNG to an actually random state, so we can get a random seed for variations
                seed = random.randrange(0,np.iinfo(np.uint32).max)
            return (seed, initial_noise)
        else:
            return (seed, None)

random.seed() resets the random number generator using the system time. Therefore we will need to implement time travel in order to get back to the state as it was. This is seriously out of scope for this project. I
Commenting out this line of code doesn't interfere with variation generation. Now I get the same series of variations as long as I use a fixed seed. Unfortunately the -S and -V values do not together reproduce the variant image. So there is more work needed to explore this. I will check the main branch to see if anything is different there and will update this comment if so.

Okay, just to comment what goes on in my machine:
"test" -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S 810716037 -n 1 -f 0.75 -V 810716037:0.1 which is what was originally posted produces the same result over and over.
test image -S42 -v0.1 does produce different results. But if you want to get one of the old results, you just need to use the same seed, e.g. record it

get it

and then reuse it

It will generate the same image. The seed could be printed on screen, passed as an argument, added to the image metadata... Many possibilities.
So for example, you generate 1000 images that are variations. Each with a different seed. Want to recreate some of those images? Just reuse the seeds for those specific images. I'm not sure a code that is supposed to generate variations of images, generating every time a different image based on the original is really a bug.

There are two things happening here:

re-running with -v will produce different images
using -S0 will generate a random seed, rather than using 0

The first thing is intentional: if you ask for 5 variations, then another 5 variations, you presumably don't want the same things you got last time. (So @lstein your commenting-out of the random.seed() line here should be reverted.)
The second thing is a bug caused (I expect) by this line - in python, 0 is falsey, so this line will reset the seed. You want to use if seed is not None rather than if seed here.

@bakkot I agree with you.
I think -S should indicate the seed for the original image. For example re-using -S3454 I expect the same image always
"banana slice" -s5 -W512 -H512 -C7.5 -S3454

If now I enter "banana slice" -s5 -W512 -H512 -C7.5 -Ak_lms -v0.1 -S3454 -n3 I expect -S3454 to only indicate that I want to generate random variations based on the image above. So I'll be creating 3 variations based on that image.

And if I enter that command again, I expect 3 new images, because -S3454 only refers to the original banana image (not to the variations -why regenerate the same variations?)

Personally I would prefer "banana slice" -s5 -W512 -H512 -C7.5 -Ak_lms -v0.1 -S3454 -n3 to give repeatable results so I could go back to said results later if desired, or share the prompt. If I wanted more variations I could pass -n6. The downside of this is potentially repeated computation.

@mh-dm You can go back to those results later even when variations are randomized: each variation prints the full command necessary to reproduce that variation.

Personally I would prefer "banana slice" -s5 -W512 -H512 -C7.5 -Ak_lms -v0.1 -S3454 -n3 to give repeatable results so I could go back to said results later if desired, or share the prompt. If I wanted more variations I could pass -n6. The downside of this is potentially repeated computation. (Note: just a personal preference)

But that takes a lot of time. For example if I generate 20 images, then to generate a new one I have to specify -n21 because the first 20 images will be the same again...

@mh-dm You can go back to those results later even when variations are randomized: each variation prints the full command necessary to reproduce that variation.

That seems to not work yet because there is a seed totally random. seed = random.randrange(0,np.iinfo(np.uint32).max) but if we allow to either get this random seed or use an old one, then we can reproduce old variations.

But that takes a lot of time. For example if I generate 20 images, then to generate a new one I have to specify -n21 because the first 20 images will be the same again...

That is a big downside, the question may be how often it happens in practice. If a few variations don't work, I'd personally try changing the variation strength or the sampler or the steps or the prompt or a combination.

Exactly. I think the solution is my comment just above.

I've fixed it insofar as the pretty-printed dream prompt now indicates the correct seed used to generate the variations, regardless of whether it was fixed in the original dream prompt or assigned automatically. I've been using seed values greater than zero, so the -S0 behavior is not an issue (should this be caught as an assertion error, or is it a feature?)
I can revert the call to randomseed() so that -SXXXX -vY.Y never produce the same series of images. However, this behavior is different from what you get without -v, when -nXX produces the same series of images when given a fixed seed.
My current issue is that when specifying the seed and variant produced by the pretty-printed prompt, I'm not getting the expected variant image back.
I suspect that either:

The base seed is not being chosen correctly. Currently it is the seed passed to the image callback for the very first variant in the series.
The seed in -V is not correct. This is the seed that is passed back to the image callback.
The variation amount in -V is not correct, at least when trying to regenerate a single variant image. However, setting this manually to 0.0 doesn't fix the issue.

So at this point I'm going to stop trying to debug the problem and revert the random seed call in get_noise(). I did do some cleanup in dream.py regarding how the pretty-printed prompts are formatted, and I'm committing this to development. I leave it to later (or someone else) to implement @Any-Winter-4079 's proposed solution.

The base seed is not being chosen correctly. Currently it is the seed passed to the image callback for the very first variant in the series.

That's definitely a problem. It used to be the original seed. (Variations have two seeds, which are interpolated between.) So you would never have the same value in -S as -V in the output from a command with -v

I also thought that the variation_amount option (-v) should be set to zero, but this doesn't help.

That is also a problem, independent of the previous thing. -v should not be specified at all in the output from the variation command. -v should be treated more like -n: it is used to determine the parameters for the output images (specifically, it is used to determine -V), rather than being a parameter itself.

I must have an older version because to me
a colorful fire hydrant -n3 -v0.15 produces
[1] outputs/img-samples/001083.944962109.png: "a colorful fire hydrant" -s50 -W512 -H512 -C7.5 -Ak_lms -V944962109:0.15 -SNone
[2] outputs/img-samples/001083.3357559079.png: "a colorful fire hydrant" -s50 -W512 -H512 -C7.5 -Ak_lms -V3357559079:0.15 -SNone
[3] outputs/img-samples/001083.561104318.png: "a colorful fire hydrant" -s50 -W512 -H512 -C7.5 -Ak_lms -V561104318:0.15 -SNone

with no -v and -SNone
Anyway, my fix seems to work on my end, but since there seem to be new changes, it might be a better idea that @bakkot looks at the correct/expected behavior first.

@bakkot has described the expected behavior, my understanding of what should happen was accurate

Using -v > 0 always makes a variation, non-deterministic
Using -V with -v generates a variation based on the seed-weight pairs, non-deterministic
Using -V without -v generates an image based on the seed-weight pairs, determinisitic
The output parameters for result images, if re-run, always produces the same image, at least on that machine. No parameters like iterations or -v should be in that output, because they intentionally do non-deterministic things.

Questions:

Is -S a special case of -V? Is -S 123 === -V 123:1 or something like that? If so, do we need two parameters?
What happens when we specify -S and -V?

Finally - maybe I have missed a revelation while reading through the discussion here that explains this - I have the same issue as originally reported with -S 1 which I assume python considers truth-y:
REPL command 1: test -s 5 -S 1 -v 0.1
REPL output 1: [2] outputs/img-samples/000040.4016874652.png: "test -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S<PHONE_NUMBER> -n 1 -f 0.75 -V<PHONE_NUMBER>:0.1
Image 1:

REPL command 2: test -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S<PHONE_NUMBER> -n 1 -f 0.75 -V<PHONE_NUMBER>:0.1 (same as output 1)
REPL output 2: [2] outputs/img-samples/000040.4016874652.png: "test -s 5 -W 512 -H 512 -C 7.5 -A k_lms -S<PHONE_NUMBER> -n 1 -f 0.75 -V<PHONE_NUMBER>:0.1 (same as REPL output 1 and command 2)
Image 2:

Is -S a special case of -V? Is -S 123 === -V 123:1 or something like that? If so, do we need two parameters?

Not exactly. -S specifies a starting point, and then each -V pair specifies another point and a distance towards that point.
So -S 42 -V 123:0.1  says "start at the noise given by seed 42, then move 10% of the way towards the noise given by seed 123".
As a special case, if you specify :1, that will mean "move all the way towards that point", so -S 456 -V 123:1 will work just like -S 456.
It's probably possible to combine them into a single parameter in some fashion, but it's not really obvious to me that there is a clear way to do that.

What happens when we specify -S and -V?

Hopefully the above answered that question. You are pretty much always going to want to specify -S whenever you specify -V.

If variations have two seeds which are interpolated between, why aren't they both represented in -V as this is a special handling of seeds, where -S indicates the normal handling?

Because the first seed really is special: are starting at that seed, not moving towards it.
If you only have two seeds, yes, you could regard this as being just a weighted average of two seeds, and the order doesn't matter. But once you start getting into variations-of-variations - i.e., when you have more than one value in -V - they're no longer equivalent: starting at point A, moving 50% of the way towards point B, then moving 50% of the way to point C is not the same as starting at point C, moving 50% of the way to point B, then moving 50% of the way to point A.

Finally - maybe I have missed a revelation while reading through the discussion here that explains this - I have the same issue as originally reported with -S 1 which I assume python considers truth-y:

Yeah, there's a separate bug introduced sometime in the last week or two caused by the initial seed getting lost on the way to getting printed. You can tell because the fist line printed in the output has the same seed value in -S and -V, which should never happen.

Thank you for the very clear explanation.
One point I am confused by:

So -S 42 -V 123:0.1 says "start at the noise given by seed 42, then move 10% of the way towards the noise given by seed 123".
As a special case, if you specify :1, that will mean "move all the way towards that point", so -S 456 -V 123:1 will work just like -S 456.

If -S 42 -V 123:0.1 says "start at 42 and go 10% of the way to 123", I would have guessed that -S 42 -V 123:1 says "start at 42 and go all the way to 123" instead of "start at 42 and move all the way back to 42"...

Not to get in the middle of the discussion (I'm reading intently) but what files are involved?

B/c when I try
test -s 5 -S 1 -v 0.1

and ouputs
"test" -s5 -W512 -H512 -C7.5 -Ak_lms -V3422978809:0.1 -S1
which re-entered

If -S 42 -V 123:0.1 says "start at 42 and go 10% of the way to 123", I would have guessed that -S 42 -V 123:1 says "start at 42 and go all the way to 123" instead of "start at 42 and move all the way back to 42"...

Whoops, sorry, I had a typo in my example (now fixed). It does indeed mean "go all the way to 123".

Either I have a previous version of the file that just works

Yeah, I think this broke very recently (and probably only on the development branch), so I'm not surprised if you have an older version which still works correctly. Unfortunately I'm way behind on everything, so I can't tell you exactly which file the issue is in.

The base seed is not being chosen correctly. Currently it is the seed passed to the image callback for the very first variant in the series.

That's definitely a problem. It used to be the original seed. (Variations have two seeds, which are interpolated between.) So you would never have the same value in -S as -V in the output from a command with -v.
That must have gotten broken at some point.

I also thought that the variation_amount option (-v) should be set to zero, but this doesn't help.

That is also a problem, independent of the previous thing. -v should not be specified at all in the output from the variation command. -v should be treated more like -n: it is used to determine the parameters for the output images (specifically, it is used to determine -V), rather than being a parameter itself.
That also must have gotten broken at some point; when I initially introduced this feature -v would not appear in the output at all (which was the correct behavior).

At least some of this has resulted from my refactoring and simplifying how the dream command is reconstructed for display on the screen and in the PNG file. I did not understand the nuances of the variation syntax and so sorry to break your code. I'm just getting back to this conversation and will read through to the end before making any more comments.

Ok, I've read through and understand what needs to be done to fix this. I just have to track and use the very original seed for the -S argument. Thanks for a great discussion, and I really appreciate @bakkot 's lucid explanation of the expected behavior and what -V really means.

Should be working now. Fix committed to development.


<EMAIL_ADDRESS>return "::error::Unable to locate executable file: docker."

I am trying to use the<EMAIL_ADDRESS>action (see workflow and devcontainer below).
I am running it in a local environment using nektos act before run at Github (see log bellow).
However, I got the error "::error::Unable to locate executable file: docker."
Am I missing something?
.github/workflows/hugo.yml
name: deploy-hugo-to-ghpages
on:
  push:
    branches:
      - master
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build at devcontainer
        uses<EMAIL_ADDRESS>        with:
          runCmd: |
            hugo --gc  --minify

.devcontainer.json
// For format details, see https://aka.ms/vscode-remote/devcontainer.json or this file's README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.128.0/containers/hugo
{
	"name": "Hugo",
	"image": "mcr.microsoft.com/devcontainers/base:ubuntu", // Any generic, debian-based image.
	"features": {
		"ghcr.io/devcontainers/features/go:1": {},
		"ghcr.io/devcontainers/features/hugo:1": {
			"version": "0.119.0",
			"extended": true
		},
	},
	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	"forwardPorts": [
		1313
	]
}

gh act -W .github/workflows/hugo.yml
[deploy-hugo-to-ghpages/build] 🚀  Start image=node:16-buster-slim
INFO[0000] Parallel tasks (0) below minimum, setting to 1
[deploy-hugo-to-ghpages/build]   🐳  docker pull image=node:16-buster-slim platform= username= forcePull=true
INFO[0001] Parallel tasks (0) below minimum, setting to 1
[deploy-hugo-to-ghpages/build]   🐳  docker create image=node:16-buster-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[deploy-hugo-to-ghpages/build]   🐳  docker run image=node:16-buster-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"
[deploy-hugo-to-ghpages/build]   ☁  git clone 'https://github.com/devcontainers/ci' # ref=v0.3
[deploy-hugo-to-ghpages/build] ⭐ Run Main Checkout
[deploy-hugo-to-ghpages/build]   🐳  docker cp src=/home/alan/src/websites/alanlivio.github.io/. dst=/home/alan/src/websites/alanlivio.github.io
[deploy-hugo-to-ghpages/build]   ✅  Success - Main Checkout
[deploy-hugo-to-ghpages/build] ⭐ Run Main Build at devcontainer
[deploy-hugo-to-ghpages/build]   🐳  docker cp<EMAIL_ADDRESS><EMAIL_ADDRESS>[deploy-hugo-to-ghpages/build]   🐳  docker exec cmd=[node<EMAIL_ADDRESS>user= workdir=
| Starting...
[deploy-hugo-to-ghpages/build]   ❗  ::error::Unable to locate executable file: docker. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
[deploy-hugo-to-ghpages/build]   ❌  Failure - Main Build at devcontainer
[deploy-hugo-to-ghpages/build] exitcode '1': failure
[deploy-hugo-to-ghpages/build] 🏁  Job failed
Error: Job 'build' failed

I am also facing the same issue here


⚠ Bofrost.be has degraded performance

In 7fd7876, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1127 ms

Resolved: Bofrost.be performance has improved in e906e96 after 10 minutes.


Web: Mute a folder from folder menu

I think it might be useful to be able to mute feeds at the folder level.
For example I have a Java folder and I'd like to mute all feeds within that folder if I'm away for a few weeks. It would be much better if I didn't have to go through all feeds in the folder.

This is possible now — just click on the folder title (or tap on iOS) in Mute Sites and all the feeds within it will mute/unmute.

@nriley sorry I was looking for it in the folder menu popup menu (like you can for feeds). Would it be useful there @samuelclay ?

Yes, I agree it would be useful in the menu. I just added it as a menu item to individual feeds not too long ago.

I'm leaving this only accessible in the Organizer, which allows you to do this. But it's a slow process and the Organizer/Mute Feeds dialog are better for giving users an indication that this will take a little while.


Publish packages to Azure Artifacts and NuGet

Currently, packages are only pushed to MyGet.  Going forward, we should push pre-release packages to MyGet and then the released packages to NuGet, in the same way as we do with the Cake packages.

Cake we ship to Azure Artifacts and not myget, so should probably do the same with Cake.Frosting.

@devlead ah, that is a good point!

Before we push to NuGet.org #89 needs to be done as Cake packages are required to be signed.


[DPE-4836] Add support for yellow status on pre-checks

Currently, pre_upgrade_checks only accept GREEN status before finishing the upgrade of a given unit. That is correct if the user is running the pre-upgrade-checks action.
However, if the cluster is in the middle of the upgrade, we should expect some of the shard replicas to be unassigned until the upgrade ends.
Closes #350

After discussions with @Mehdi-Bendriss we've decided that we only progress with upgrades IF the cluster is green. If, during an upgrade, the cluster got into YELLOW, then we should consider stopping and fixing the issue or aborting the procedure.


Not detecting MIT license on NodeOS

The NodeOS License.txt file is copied from elsewhere. It has a clarification notice at beggining as the original one, but the legal text is the same separated from the copyright, and also the MIT License is crearly specified at the first line, so I think it would be just enough to be detected and showed.
The full text of the license file is:
MIT License

Copyright (c) 2013-2017 Jacob Groundwater, Jesús Leganés-Combarro 'piranna' and
other contributors

This software consists of voluntary contributions made by many individuals. For
exact contribution history, see the revision history available at
https://github.com/NodeOS/NodeOS

The following license applies to all parts of this software:

====

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

A computer has no way of knowing if the added line says "The following license applies to all parts of this software", or "The following license doesn't apply to any part of this software"

Good point...

Because there is unidentified text in the License file that deviates from the expected MIT license, Licensee is properly saying "this isn't the true MIT license", as it requires a human to evaluate the added statements.

So what's your propose here? Maybe put the two paragraphs on the README.md file would be enough? Is there any way to validate the text (online, maybe) previously to commit it?

@piranna Yes, GitHub recommends putting extra details/complexity outside of the license file if possible.

I have removed the extra parts from the LICENSE.txt file and put them on the README.md file as suggested, so this can be closed. Thanks for your help :-)


Share some training results and ask if it is possible to train cars specifically

Overall, the effect of this work is still very good. I used some strange data for testing and also obtained some strange results. I want to share with you here. I also want to ask if this can be done for a certain type, such as training cars, and the model works well when generating cars.

Hi, thanks for trying our codes and demo! I think it is possible to train on a single category for better performance, though we have not testes such a setting.


println don't show in the terminal and how to turn off the ios info(too much) .

Describe the bug

Reproduction
No response
Expected behavior
No response
Full tauri info output
➜  app git:(0.7.0) cargo tauri info

[✔] Environment
    - OS: Mac OS 14.5.0 X64
    ✔ Xcode Command Line Tools: installed
    ✔ rustc: 1.79.0 (129f3b996 2024-06-10)
    ✔ cargo: 1.79.0 (ffa9cf99a 2024-06-03)
    ✔ rustup: 1.27.1 (54dd3d00f 2024-04-24)
    ✔ Rust toolchain: stable-aarch64-apple-darwin (environment override by RUSTUP_TOOLCHAIN)
    - node: 20.14.0
    - pnpm: 9.4.0
    - npm: 10.7.0

[-] Packages
    - tauri [RUST]: 2.0.0-beta.22
    - tauri-build [RUST]: 2.0.0-beta.17
    - wry [RUST]: 0.40.1
    - tao [RUST]: 0.28.1
    - tauri-cli [RUST]: 2.0.0-beta.20
    - @tauri-apps/api [NPM]: 2.0.0-beta.13
    - @tauri-apps/cli [NPM]: 2.0.0-beta.20

[-] App
    - build-type: bundle
    - CSP: unset
    - frontendDist: ../dist
    - devUrl: http://localhost:1420/
    - framework: Vue.js
    - bundler: Vite

[-] iOS
    - Developer Teams: None

Stack trace
No response
Additional context
No response

The device log seems not able to be disabled now because it's hardcoded: https://github.com/tauri-apps/cargo-mobile2/blob/46c0a4209bafa456ce1fca2ea0265c382a1f21bb/src/apple/device/simctl/run.rs#L74-L77
For the println, stdout from the application is somehow suppressed while running. What I did before was use NSLog to let it print to the log. If you want to try NSLog, here's a reference https://github.com/pewsheen/nslog-rs/blob/main/src/lib.rs . But I think there must be a better way to achieve it.

But I think there must be a better way to achieve it.

You can use tauri-plugin-log and the log crate.


anchor keys sync change Anchor.toml [provider] unexpected

when use custom rpc:
[provider]
cluster = "https://rpc.ankr.com/solana_devnet"
wallet = "~/.config/solana/id.json"

and run:

rm target/deploy/*-keypair.json
anchor keys sync

Anchor.toml will be changed to:
[provider]
cluster = ["https://rpc.ankr.com/solana_devnet", "wss://rpc.ankr.com/solana_devnet"]
wallet = "~/.config/solana/id.json"

and we will get this error from anchor keys sync:
Workspace configuration error: Unable to deserialize config: TOML parse error at line 14, column 11
   |
14 | cluster = ["https://rpc.ankr.com/solana_devnet", "wss://rpc.ankr.com/solana_devnet"]
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
invalid type: sequence, expected string or map

Thanks for creating the issue. However, a similar issue exists in https://github.com/coral-xyz/anchor/issues/3222, so I'm closing this one as duplicate to not divert attention.

Just FYI, fixed this in https://github.com/coral-xyz/anchor/pull/3428


Add rule to simplify 1^x to 1

Rule as suggested in #39.
Note that the test could probably be more robust since it just checks that 1^x stays as 1^x.
It probably should check that 1^x is reduced to 1 if x is known to be constant but I didn't see an obvious way to do that in the test suite.

Thanks for the feedback! Updated to address your concerns.

haha fair on the let / const thing - we really should just fix it everywhere so we're not asking for things we don't do ourselves heh
thanks for the quick changes! looks good to me :D

Cool. Thanks for the merge!


Chrome error when items are loaded dynamically using jQuery.

Hi, I am using two select items, where i load countries to first, and cities to the second select list with jQuery based on value in first select item. I am getting it properly on safari browser. but when i opened it in chrome in my mac, it shows as in the screenshot.
I got same bug in firefox also. any fix for this?
Thank you.

Hello there, i got solved the issue, i used id (#city_id) instead of class name(.city_id) for the second select list. but couldn't figure out the actual issue !

Hello there, i got solved the issue, i used id (#city_id) instead of class name(.city_id) for the second select list. but couldn't figure out the actual issue !


Upgrade the repository to yarn v3, pnp enabled

Hello, due to is compatible I greatly suggest to do this update to enable pnp. For information here

Hello, Thank for your request. I added 1 dependency before I saw it. Could you please help to explain how did you do it to resolve conflicts and how pnp going to make project different.

Using pnp let you to not use the node modules folder; instead zipped versions of all the node dependencies are automatically loaded. Now I will add the dependency change...

[ERROR] The image at /var/lib/jenkins/workspace/UpdateWebFrontend/main/blog/2019-03-23-story-2/story-2-4.jpg can't be read correctly. Please ensure it's a valid image.
The "path" argument must be of type string. Received type number (36)

+ yarn run build
Creating an optimized production build...
Failed to compile.

Failed to load config "react-app" to extend from.
Referenced from: /var/lib/jenkins/workspace/UpdateWebFrontend/map/package.json

So... Maybe can I upgrade without enabling pnp?

This type of error usually is due to pnp... Now works?


remove credentialsApi method getCredentialDefinitionsByCsp

getCredentialDefinitionsByCsp method currently returns the customer credentials stored in the credential cache and not the definition.
This method is wrongly named and also we have user story #447 to implement the actual GET credentials method.
So this existing method can be removed.

getCredentialDefinitionsByCsp  is already renamed in #455 . Closing this.


Dynamic interleaving to stream MP4

I have a simple open source Go server for files, httpfolder. I would like to add the ability to stream MP4 files that have the moov box at the end, which confuses some browsers. I know that I could use something like MP4Box to do this.
However, would this package would allow me to do dynamically interleave the moov box without modifying the file on disk? The dynamic equivalent of "mp4box -inter 500 input.mp4"? Thanks for any pointers you can give me!

@biorisk
go-mp4 has only low-level support for parsing/writing.
So, if you are expert in MP4's data structure, you can implement any conversions.
However, in this use case, it is hard and I do not recommend.

@sunfish-shogi, thank you for the feedback. I expected that it would be too hard, but wanted to check.


.github: Pin docker buildx version to v0.9.1 (v2)

GitHub recently rolled out Docker buildx version v0.10.0 on their
builders, which transparently changed the MediaType of docker images to
OCI v1 and added provenance attestations.
Unfortunately, various tools we use in CI like SBOM tooling and docker
manifest inspect do not properly support some aspect of the new image
formats. This resulted in breaking CI, with some messages like this:
level=fatal msg="generating doc: creating SPDX document: generating
SPDX package from image ref quay.io/cilium/docker-plugin-ci:XXX:
generating image package"

This could also lead CI to fail while waiting for image builds to
complete, because the command we use to test whether the image is
available did not support the image types.
This commit attempts to revert buildx back to v0.9.1 to prevent it from
generating the images in a format that other tooling doesn't expect.
Over time we can work on migrating to buildx v0.10, testing various
parts of our CI as we do so.
This is a quick-and-dirty hack to stabilize CI for the short term, then
we can figure out over time how to properly resolve the conflict between
these systems.
Reverts https://github.com/cilium/cilium/pull/23206

Thanks to @chancez for pointing out why the version shouldn't have worked on the previous PR, originally https://github.com/cilium/cilium/pull/23206 was trying to update the docker/build-push-action configuration for provenance (which didn't worK). When I switched to configuring version, I thought it was the same action. However, version is only configured on the docker/setup-buildx-action. So hopefully this actually pins the version this time.

@aanm I was hoping to follow up on that separately once we confirmed that this is switching CI back to generating images in the older / supported format

@aanm https://github.com/cilium/cilium/pull/23221

@joestringer FYI, you can also work around this by doing docker buildx imagetools inspect --raw quay.io/cilium/cilium-ci:10e55d6053762d098cce06cd97abd51afc6faf0f (credit goes to @pippolo84)
Are we ok to backport this to v1.11, CI is broken over there as well.


Interpreted data needs to be placed in a database

The data needs to be read in and placed in a database, storing in each entry the year information for which that data was gathered. This would allow for historical use and for data to be imported in the future.

Fixed in 9f76669


[TASK]: Improve search bar user experience

Describe the task
The search bar got new features, suggestions and autocompletion, but the whole user experience is poor.
Task Items

[x] Use PrimeVue autocomplete component
[x] Integrate using @jamiewaese-uncharted Figma design
[x] Able to select a suggestion via keyboard's arrow keys
[x] Do not block the input typing after 3 characters
[x] Execute a search when suggestions or autocomplete has been selected
[x] #633
[x] Auto complete should hide when there are no suggestions (start to type github.  At gith there are suggestions, and githu has no suggestion but bar remains)

Goal
Improve those features to feel more like a Google search bar.
Screenshots

@YohannParis
"Do not block the input typing after 3 characters" This was broken after recent merge, will fix in an upcoming PR.

"Use PrimeVue autocomplete component"

I looked into using this component and I don't think it's worth it to use it. Less flexibility than InputText and Menu combo and requires refactoring that won't necessarily give any benefit.

We should leverage prime component as much as possible and to implement @jamiewaese-uncharted design.

" Able to select a suggestion via keyboard's arrow keys"

This is already partially implemented. Using the arrow keys highlights different selections on the dropdown menu, and hitting enter will search but due to styling changes they highlight colour is the same as the regular unhighlighted colour, it seems.

This does not work for me on Chrome on Staging nor Production.

"Use PrimeVue autocomplete component"

I looked into using this component and I don't think it's worth it to use it. Less flexibility than InputText and Menu combo and requires refactoring that won't necessarily give any benefit.

We should leverage prime component as much as possible and to implement @jamiewaese-uncharted design.

Of course I may be wrong but I don't see the advantage that the Autocomplete component has in implementing jamie's designs.

" Able to select a suggestion via keyboard's arrow keys"

This is already partially implemented. Using the arrow keys highlights different selections on the dropdown menu, and hitting enter will search but due to styling changes they highlight colour is the same as the regular unhighlighted colour, it seems.

This does not work for me on Chrome on Staging nor Production.

It tried it on chrome on staging now:
https://user-images.githubusercontent.com/16358728/216395716-46c23419-d86d-4ccc-9087-8803e4a0f715.mov
What's missing is just the highlighting (which already comes with the menu component, I think the global styling changes got rid of it) and making the highlighted text appear automatically appear in the searchbar (dunno if this is a requirement, but this is what google's searchbar does. Fyi the autocomplete component doesn't do this either)
Anyway I'm just making these points from the perspective of making our lives easier, saving time, and not doing redundant work.

@shawnyama could you add a video of the new autocomplete?

@shawnyama could you add a video of the new autocomplete?


Terminated due to memory issue

When I try to load another AVPlayer inside the same PanoramaView, it works for few seconds and then the app crashes and the log tells:
"Message from debugger: Terminated due to memory issue"
Is there a way to free the memory?
Thanks

Hi, Im facing an issue
[SceneKit] Error: Scene <StereoVideo.StereoSphericalVideoScene: 0x2837a4380> is modified within a rendering callback of another scene (<StereoVideo.StereoScene: 0x2837a1880>). This is not allowed and may lead to crash.
Do you have any fix for this?


iOS: Should resolve after modal has dismissed

Is there any particular reason that imagePicker.present() is resolved before dismissing?
Complicated modal chaining (ie. showing ImageCropper after imagePicker.present() in modal) is broken because QBImagePicker is not dismissed when resolved.
below example and error is just to represent process
example
// currently in modal

async getCroppedImage() {
  const context = imagepicker.create();
  await context.authorize();
  const selections = await context.present();
  selections.forEach((selection) => {
    selection.getImageAsync(async source => {
      const imageSource = new ImageSource();
      imageSource.setNativeSource(source);
      const cropper = new ImageCropper();
      const cropped = await cropper.show(imageSource); // Error in this line
    });
  });
}

error
Warning: Attempt to present <TOCropViewController: 0x1059b0200> on < QBImagePickerController: 0x105839c00> whose view is not in the window hierarchy!

possible fix
I could fix below error by moving https://github.com/NativeScript/nativescript-imagepicker/blob/0e34f315d360b638bfac2185b500889561dd9088/src/imagepicker.ios.ts#L103
to
        imagePickerController.dismissViewControllerAnimatedCompletion(true, () => {
           this._resolve(assets);

            // FIX: possible memory issue when picking images many times.
            // Not the best solution, but the only one working for now
            // https://github.com/NativeScript/nativescript-imagepicker/issues/222
            setTimeout(utils.GC, 200);
        });

I have this problem now.
Attempt to present <TOCropViewController: 0x105800800> on <UIImagePickerController: 0x105013e00> whose view is not in the window hierarchy!


ros_type_introspection: 0.3.1-0 in 'indigo/distribution.yaml' [bloom]

Increasing version of package(s) in repository ros_type_introspection to 0.3.1-0:

upstream repository: https://github.com/facontidavide/ros_type_introspection.git
release repository: https://github.com/facontidavide/ros_type_introspection-release.git
distro file: indigo/distribution.yaml
bloom version: 0.5.22
previous version for package: 0.3.0-0

ros_type_introspection
* added BSD license
* added an alternative implementation of ShapeShifter

sorry about that, still getting familiar with the process.

No problem, the process is not straight forward when doing it the first time :wink: Now its all "green"...


🚀 [Feature Request] - Title: Add FAQs

Is your feature request related to a problem ? Please describe. ✍
i want too implement FAQs pop up model to bottom down right side, wen user clicks on it faqs pop ups on screen
@anmode please assign me this task
Describe briefly how you would implement this enhancement.
.

Additional context or screenshots
Additional context
Are you contributing for a program?
Other
Code of Conduct

[X] I have read and agree to the Contributing Guidelines of this project.

pls assign this task to me i will add the faq part with some intresting colors


cmd/compile: imperfect error message

What version of Go are you using (go version)?

$ go version
go version go1.20.5 linux/amd64

Does this issue reproduce with the latest release?
Yes
What did you do?

package main

const S = "Go"

type T [S[0]]int // syntax error: type parameters must be named

func main() {}

What did you expect to see?
syntax error: array length must be a constant
or
syntax error: type constraint is incomplete
or
syntax error: either type constraint is incomplete or array length is not a constant
What did you see instead?
syntax error: type parameters must be named

CC @griesemer


RUDY map pre-process

Are your RUDY map of RISC-V dataset pre-processed by smoothen? But the ISPD2015 dataset provided by you are not?
Because the data distribution is very different.

No, they are processed in the same way.
If you are comparing RISC-V with superblue from ISPD2015 after resizing to the same size, they should be very different due to difference in design scale and technology nodes.

yes, it seems that the bin size of RISC-V is 1.5μm×1.5μm and the bin size of ISPD2015 is 3.0μm×3.0μm. After resizing the superblue into 256*256, I tested the pre-trained model from RISC-V on superblue, the performance is not good. So is it reasonable?

Yes, the diversity of our dataset is not yet enough to cover superblue designs, so fine-tuning or retraining is necessary to reach good performance.


Widget configurations

Add a blade component that can take all customization and make it seamlessly work with Laravel Livewire.
Closes #1.
Closes #2.

Tests are passing locally, GA is not liking them. Happy to merge still as I've manually verified too.


Sync eng/common directory with azure-sdk-tools for PR 1106

Sync eng/common directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/1106
See eng/common workflow

/check-enforcer override


Adding "update_environment" after creating the applciation

Hi,
In order to provide a real CD can you guys please add flag that states whether to also update the environment with the new version or not.
Great work!!!
Thanks

Got to check the options of the go aws sdk, but if i find this option we can offer that for sure.

@tboerger have a look - http://docs.aws.amazon.com/sdk-for-go/api/service/elasticbeanstalk.html#type-UpdateEnvironmentInput
Thanks man that would be awesome

I don't really know what happens on the AWS side with this, so you need UpdateEnvironmentInput and not CreateEnvironmentInput ?

Is this closed now?

:+1:


Add standard claims to UserProfile

Firstly, thank-you for taking on maintenance of oidc-client (and migrating to TypeScript)
As per #125, this PR adds extra properties to UserProfile so that standard claims can be accessed by applications.
Since the code already existed to merge these claims into UserProfile (ResponseValidator._mergeClaims()), all this PR does is add the extra properties so that they can be accessed with TypeScript applications.

TSDoc suggestions implemented.
Also added documentation for individual properties.  Information was copied directly from standard.
Let me know if you would like anything else changed.


Authentication: Adding authentication to outbound API calls from Component

Description
For outbound calls from a Component (to another component or to an external API), we want to use Auth 2.0 Client Credentials Flow. The component needs to use its ClientID/Secret to get a token from IDM OAuth 2.0 gateway. It then calls the outbound API with the token. The target of the API call can use the token to understand who is calling and with what Role.
We have several options:

Leave it for every component to implement this process; We would need to somehow share the ClientID/Secret with the component using a Kubernetes secret or secrets vault.
We could provide a set of language-specific libraries to help a component implement this process.
We could use a sidecar to offload the process from the component. This could be either adding additional functionality to the current Istio service mesh component, or developing a new sidecar specifically for this process.

From Mumbai workshop and discussions with @brian-burton and @ferenc-hechler we recommend some additional research around these options. If we could add this into the existing Istio sidecar using an Envoy Filter or similar (see Envoy Filter for OAuth) then this could be the preferred solution. If not, then there were concerns about creating too-many sidecars.
To communicate the ClientID/Secret to the component (or sidecar) this could be shared as a Kubernetes secret generated by the Identity Operator, or by adding to the Secrets Vault.

Started to setup a test scenario for OAuth2 with KeyCloak.
First I installed a component from the testdata:
helm upgrade --install ctk -n components feature-definition-and-test-kit/testData/productcatalog-v1beta4

Looking in KeyCloak I can see that a client ctk-productcatalogmanagement was created:

Client-ID and Client-Secret can be found in the Client-Details > Credentials:

The open-ID endpoints can be found in the Realm-Settings:

The link directs to the .well-known/openid-configuration of the myrealm:

So, now I have the following required information:
Client-ID: ctk-productcatalogmanagement
Client-Secret: wTju...vc1B
Token-Access-Endpoint: https://canvas-keycloak.ihc-dt.cluster-3.de/auth/realms/myrealm/protocol/openid-connect/token

I configured authentication in Postman with these values:

But when trying to retrieve a token an error message is returned:

After some internet research I found, that service accounts have to be enabled in the client settings:

Now Postman is happy :-)

The Payload of the returned JWT:
{
    "exp":<PHONE_NUMBER>,
    "iat":<PHONE_NUMBER>,
    "jti": "194cf4c4-9c1e-4172-9fec-bbab0567b44c",
    "iss": "https://canvas-keycloak.ihc-dt.cluster-3.de/auth/realms/myrealm",
    "aud": "account",
    "sub": "93b2aa50-856d-41ad-b5d0-91012796dbbb",
    "typ": "Bearer",
    "azp": "ctk-productcatalogmanagement",
    "acr": "1",
    "realm_access": {
        "roles": [
            "default-roles-myrealm",
            "offline_access",
            "uma_authorization"
        ]
    },
    "resource_access": {
        "account": {
            "roles": [
                "manage-account",
                "manage-account-links",
                "view-profile"
            ]
        }
    },
    "scope": "profile email",
    "clientHost": "<IP_ADDRESS>",
    "email_verified": false,
    "clientId": "ctk-productcatalogmanagement",
    "preferred_username": "service-account-ctk-productcatalogmanagement",
    "clientAddress": "<IP_ADDRESS>"
}

So, what Postman is doing is to add the JWT as Bearer Authorization to the request (line 2):
GET /service HTTP/1.1
Authorization: Bearer eyJh...q1RA
User-Agent: PostmanRuntime/7.29.2
Accept: */*
Postman-Token: 844143bf-9ee1-43ff-ad02-83c4cc26ff31
Host: canvas-info.ihc-dt.cluster-3.de
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

What we want to achieve is, that an Envoy filter is doing the same, get a token, add it to all outgoing requests and refresh the token after some time.

There is a prototype extension which seems to be doing what we intend to do:
https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/http/injected_credentials/oauth2/v3/oauth2.proto.html

After some internet research I found, that service accounts have to be enabled in the client settings

Do we need to add this to the identity config operator so that this happens automatically?

To be able to retrieve a token I needed this.
Maybe there are other ways for token retrieval and I did it the wrong way.
But If this is the way, then we have to add it to ConfigOperator.

Successfully injected an OAuth2 token for inbound calls through the component Gateway.
(That is not the goal, it should be outbound later)
Envoy has a Secrets-Discovery-Service (SDS). We have to bring the client-secret into the Envoy secrets.
To do this, secrets can be defined from filesystem. And in K8S the Filesystem can be mounted from a ConfigMap or Secret.
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: istio-ingress
  name: oauth2secs-cm
data:
  demo-a-productcatalogmanagement.yaml: |+
    resources:
    - "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
      name: clientsecret
      generic_secret:
        secret:
          # Fill in the client secret your OAuth2 provider assigned to you.
          inline_bytes: "ZGVq...RLTUo"

We are doing this in the istio-ingress namespace as we first try to patch incoming requests.

Now we can define the EnvoyFilter for injecting the secrets:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: envoyfilter-oauth2-test
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
    patch:
      operation: INSERT_BEFORE
      value:
         # https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/credential_injector_filter#oauth2-credential-injector-client-credential-grant
         # (https://github.com/veehaitch/envoy-oauth2-filter-google/blob/main/envoy/envoy.yaml)
         name: envoy.filters.http.credential_injector
         typed_config:
             "@type": type.googleapis.com/envoy.extensions.filters.http.credential_injector.v3.CredentialInjector
             credential:
               name: envoy.http.injected_credentials.oauth2
               typed_config:
                 "@type": type.googleapis.com/envoy.extensions.http.injected_credentials.oauth2.v3.OAuth2
                 token_endpoint:
                   cluster: "outbound|8083||canvas-keycloak.canvas.svc.cluster.local"
                   timeout: 3s
                   uri: "http://canvas-keycloak.canvas.svc.cluster.local/auth/realms/myrealm/protocol/openid-connect/token"
                 client_credentials:
                   client_id: demo-a-productcatalogmanagement
                   client_secret:
                     name: clientsecret
                     sds_config:
                       path_config_source:
                         path: "/config_map/oauth2secs/demo-a-productcatalogmanagement.yaml"
                         watched_directory:
                            path: "/config_map/oauth2secs"

Requests to the echoservice now have an injected Authorization Bearer:

Pasting the authorization token into a JWT parser retrieves the following payload:
{
    "exp":<PHONE_NUMBER>,
    "iat":<PHONE_NUMBER>,
    "jti": "0548dfa9-ace6-49f3-9e00-b58e7a9a90ce",
    "iss": "http://canvas-keycloak.canvas.svc.cluster.local/auth/realms/myrealm",
    "aud": "account",
    "sub": "0ce4c44d-b977-4023-8d95-94e7954a3d09",
    "typ": "Bearer",
    "azp": "demo-a-productcatalogmanagement",
    "acr": "1",
    "realm_access": {
        "roles": [
            "default-roles-myrealm",
            "offline_access",
            "uma_authorization"
        ]
    },
    "resource_access": {
        "account": {
            "roles": [
                "manage-account",
                "manage-account-links",
                "view-profile"
            ]
        }
    },
    "scope": "profile email",
    "clientId": "demo-a-productcatalogmanagement",
    "email_verified": false,
    "clientHost": "<IP_ADDRESS>",
    "preferred_username": "service-account-demo-a-productcatalogmanagement",
    "clientAddress": "<IP_ADDRESS>"
}

Got it now working with a deployed component:
Step1 Patch Sidecar Template
Each istio-proxy sidcar should automatically mount the optional secret envoy-oauth2-secrets. This allows defining secrets based on the mounted secret in the filesystem.
TODO: figure out, how to do this at install time with values.yaml
apiVersion: v1
data:
  config: |-
    # defaultTemplates defines the default template to use for pods that do not explicitly specify a template
    defaultTemplates: [sidecar]
    policy: enabled
    ...
    templates:
      sidecar: |
        {{- define "resources"  }}
          ...
          - name: istio-proxy
           ...
            volumeMounts:
            - name: workload-socket
              ...
[+]         # ----- MOUNTED SECRET -----
[+]         - mountPath: /envoy_secrets/oauth2secs
[+]           name: envoy-oauth2-secrets-vol
          volumes:
          - emptyDir:
            ...
[+]       # ----- SDS VOLUME -----
[+]       - name: envoy-oauth2-secrets-vol
[+]         secret:
[+]           optional: true
[+]           defaultMode: 420
[+]           secretName: envoy-oauth2-secrets

          {{- if .Values.global.imagePullSecrets }}
          imagePullSecrets:
       ...
kind: ConfigMap
metadata:
  ...
  name: istio-sidecar-injector
  namespace: istio-system

Step 2 Create Secret
create a secret which contains the credentials (client_secrets) for all components in the namespace.
envoy-oauth2-secrets/demo-a-productcatalogmanagement.yaml:
resources:
- "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
  name: clientsecret
  generic_secret:
    secret:
      inline_string: "oQH2khKfbC7QPBJTFXDMTGmesH5UpQY0"

(alternative to inline_string is inline_bytes with base64 encoding)
kubectl create secret -n components generic envoy-oauth2-secrets --from-file=envoy-oauth2-secrets --dry-run=client -o yaml | kubectl apply -f -

STEP 3: Create EnvoyFilter for OAuth2 injection
The workload is selected based on labels.
Here we select app=demo-a-productcatalogmanagement
The client_secret is defined using sds_config.path_config_source to the mounted secret.
The watcher monitors updates on the secret with a few seconds delay.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: envoyfilter-oauth2-curlpod
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: demo-a-productcatalogmanagement
  configPatches:
  # https://github.com/istio/istio/issues/47572
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
    patch:
      operation: INSERT_BEFORE
      value:
         # https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/credential_injector_filter#oauth2-credential-injector-client-credential-grant
         # (https://github.com/veehaitch/envoy-oauth2-filter-google/blob/main/envoy/envoy.yaml)
         # name: envoy.filters.http.oauth2
         name: envoy.filters.http.credential_injector
         typed_config:
             "@type": type.googleapis.com/envoy.extensions.filters.http.credential_injector.v3.CredentialInjector
             credential:
               name: envoy.http.injected_credentials.oauth2
               typed_config:
                 "@type": type.googleapis.com/envoy.extensions.http.injected_credentials.oauth2.v3.OAuth2
                 token_endpoint:
                   cluster: "outbound|8083||canvas-keycloak.canvas.svc.cluster.local"
                   timeout: 3s
                   uri: "http://canvas-keycloak.canvas.svc.cluster.local:8083/auth/realms/myrealm/protocol/openid-connect/token"
                 client_credentials:
                   client_id: demo-a-productcatalogmanagement
                   client_secret:
                     name: clientsecret
                     sds_config:
                       path_config_source:
                         path: "/envoy_secrets/oauth2secs/demo-a-productcatalogmanagement.yaml"
                         watched_directory:
                            path: "/envoy_secrets/oauth2secs"

Now we can test this by logging into the prodcatapi-pod:
$ kubectl exec -n components demo-a-prodcatapi-75d6b7577f-n22t6 -it -- /bin/bash

root@demo-a-prodcatapi-75d6b7577f-n22t6:/src# curl http://echoservice.echoservice.svc.cluster.local/ip

log request ip infos
client-ip: <IP_ADDRESS>
header:

Host: echoservice.echoservice.svc.cluster.local
User-Agent: curl/7.64.0
Accept: */*
X-Forwarded-Proto: http
X-Request-Id: f18b8039-d16e-4e05-829c-51544c8a0239
Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ2VkRxRmRIX1dDSTRCYXNMZGhlZWRTNERlbjZXWGtwX25zcVZ6ZExsVUVFIn0.eyJleHAiOjE3MzI2MzYxNzQsImlhdCI6MTczMjYzNTg3NCwianRpIjoiNDIzMjNmNDEtMmYyNy00NTIwLWI5MGItZTNkMTNlYWFiYjViIiwiaXNzIjoiaHR0cDovL2NhbnZhcy1rZXljbG9hay5jYW52YXMuc3ZjLmNsdXN0ZXIubG9jYWw6ODA4My9hdXRoL3JlYWxtcy9teXJlYWxtIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImExOTZjNGYxLTkxYzUtNDk4YS04ZmRjLWU3OTJhMTdjMWExMSIsInR5cCI6IkJlYXJlciIsImF6cCI6ImRlbW8tYS1wcm9kdWN0Y2F0YWxvZ21hbmFnZW1lbnQiLCJhY3IiOiIxIiwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtbXlyZWFsbSIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6InByb2ZpbGUgZW1haWwiLCJjbGllbnRIb3N0IjoiMTAuMTYuMC45OCIsImNsaWVudElkIjoiZGVtby1hLXByb2R1Y3RjYXRhbG9nbWFuYWdlbWVudCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwicHJlZmVycmVkX3VzZXJuYW1lIjoic2VydmljZS1hY2NvdW50LWRlbW8tYS1wcm9kdWN0Y2F0YWxvZ21hbmFnZW1lbnQiLCJjbGllbnRBZGRyZXNzIjoiMTAuMTYuMC45OCJ9.cFUmD4-jIBeuMVjIf8M2rId0EwD7MSopKvub4yQ_ssV_afWpzHHAnx9Ablf_5BlPM-gIsWeyprdSfRaoXacRadrsmXU-Bkf_MabvuCDFMzZyO4KRNM7KKhTcKOxBqfSg0wWlFugLzVefzbMNe36yLo7es2GZnIXU4xBRk3GuKboVuwx-uRJ3vNZX7VQvkmogbMjoN2S4Jaf0XgGGMc7IotYIiOcsqbMWpm1NzQjoDi_KECvVluTeXZDb6YHvda6LlPqes07fFToyO3mXLVgIf6ytInEeeu56DGh3o1NtJNb3zz8gqQ47JqU5vWma_L5IUR9KdFiUcpiKY3YO5b4gRQ
X-Envoy-Attempt-Count: 1
X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/echoservice/sa/default;Hash=cc12ec6e29a6e7c9d2dfcc11199ff037f1446a59d921ef9482a75ea1a0b5afb1;Subject="";URI=spiffe://cluster.local/ns/components/sa/default

ROOT
hello
ip

The authorization bearer was automatically injected.

The HTTP->HTTPS switch can be done using Egress TLS Origination.
An example for one target domain looks like this:
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  namespace: components
  name: echo-free-beeceptor-com
spec:
  hosts:
  - echo.free.beeceptor.com
  ports:
  - number: 80
    name: http-port
    protocol: HTTP
    targetPort: 443
  - number: 443
    name: https-port
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  namespace: components
  name: echo-free-beeceptor-com
spec:
  host: echo.free.beeceptor.com
  trafficPolicy:
    portLevelSettings:
    - port:
        number: 80
      tls:
        mode: SIMPLE # initiates HTTPS when accessing echo.free.beeceptor.com

Testing the same request as above:
root@demo-a-prodcatapi-75d6b7577f-n22t6:/src# curl http://echo.free.beeceptor.com
{
  "method": "GET",
  "protocol": "https",
  "host": "echo.free.beeceptor.com",
  ..."path": "/",
  "ip": "<IP_ADDRESS>:44352",
  "headers": {
    "Host": "echo.free.beeceptor.com",
    ...
    "Authorization": "Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ2VkRxRmRIX1dDSTRCYXNMZGhlZWRTNERlbjZXWGtwX25zcVZ6ZExsVUVFIn0.eyJleHAiOjE3MzI2NjAzODQsImlhdCI6MTczMjY2MDA4NCwianRpIjoiY2VmMDRkNmEtNTA5Yi00MGU0LTlmMzEtZGQ3YjM5ZDZlZmI4IiwiaXNzIjoiaHR0cDovL2NhbnZhcy1rZXljbG9hay5jYW52YXMuc3ZjLmNsdXN0ZXIubG9jYWw6ODA4My9hdXRoL3JlYWxtcy9teXJlYWxtIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImExOTZjNGYxLTkxYzUtNDk4YS04ZmRjLWU3OTJhMTdjMWExMSIsInR5cCI6IkJlYXJlciIsImF6cCI6ImRlbW8tYS1wcm9kdWN0Y2F0YWxvZ21hbmFnZW1lbnQiLCJhY3IiOiIxIiwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtbXlyZWFsbSIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6InByb2ZpbGUgZW1haWwiLCJjbGllbnRIb3N0IjoiMTAuMTYuMC45OCIsImNsaWVudElkIjoiZGVtby1hLXByb2R1Y3RjYXRhbG9nbWFuYWdlbWVudCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwicHJlZmVycmVkX3VzZXJuYW1lIjoic2VydmljZS1hY2NvdW50LWRlbW8tYS1wcm9kdWN0Y2F0YWxvZ21hbmFnZW1lbnQiLCJjbGllbnRBZGRyZXNzIjoiMTAuMTYuMC45OCJ9.RiJpS-rG_wOZvsOGoQz0xMS3jJ8murf-j7fEf3K8QlAjaCOxIvVUGyHTc9LisSVE0-8ZNDN6rb0psQIuzNC4bT7OfG9Lz-H_-GX_S5Go5_pOx2Z09vXtso1etmuNFcmuzkLTFpSWcEIbLNaBqJd9Hklysq23yjLnjKKAa6K5f5J-1FFPmtKLiJHntUN8z8dyQuigG4VAXx0ieRRXLUjyPwA9yhbzljGkzvQUl202lYL218iXYLlC4Rflms3oNYqx0AGwVrj7poRv3j-tmN4yTOC3OhoeN7ZxTv2MlQH1ddO1OEr8pC1HtsVvX3wnGDI_rBi0vcv91GQ_WQBfBoR18g",
    ...
  },
  "parsedQueryParams": {}
}

The important part is   "protocol": "https".
This means the request was sent out with htpps.
For a more advanced solution it might be necessary to setup an egress gateway which then is doing the TLS origination:
https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/

For managing the secrets in Kubernetes there is an alternative to use an SDS server:
https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret
The most important benefit of SDS is to simplify the certificate management.
Without this feature, in k8s deployment, certificates must be

    created as secrets and mounted into the proxy containers.

If certificates are expired, the secrets need to be updated and
the proxy containers need to be re-deployed.
With SDS, a central SDS server will push certificates to all Envoy instances.

Hopefully this can also be used for generic secrets and not only for TLS certs.


Issue

Does this occur consistently?
Repro steps:

Action: azure-dev.commands.cli.provision
Error type: TypeError
Error Message: Cannot read properties of undefined (reading 'configurationFile')
Version: 0.8.4
OS: win32
OS Release: 10.0.26100
Product: Visual Studio Code
Product Version: 1.96.2
Language: en

Call Stack
provision extension.js:2:896098
extension.js:2:896767extension.js:2:896767
extension.js:2:58762extension.js:2:58762
extension.js:2:10404extension.js:2:10404
extension.js:2:9229extension.js:2:9229
_ extension.js:2:8915
callWithTelemetryAndErrorHandling extension.js:2:10311
extension.js:2:58480extension.js:2:58480
extension.js:2:58015extension.js:2:58015
_ extension.js:2:57701
extension.js:2:58291extension.js:2:58291
cw.h extensionHostProcess.js:115:32833
cw.g extensionHostProcess.js:115:31910
cw.executeCommand extensionHostProcess.js:115:31817
Object.executeCommand extensionHostProcess.js:157:37484
extension.bundle.js:2:1769624extension.bundle.js:2:1769624
fulfilled extension.bundle.js:2:1768341

@AlHasSaN0ALi Could you please provide more specific information, such as the complete project code and detailed reproduction steps, so that we can reproduce this issue?

Seems like an auto-created issue with not enough data. I am closing this. Please re-open with more information if this was a real issue


Support dynamic allocation of sys_{mutex,sem}

Is your enhancement proposal related to a problem? Please describe.
The sys_mutex and sys_sem mechanisms, unlike the kernel equivalent (k_mutex and k_sem), do not support run-time dynamic allocation, and must be statically allocated at the build time to work properly.
These mechanisms exist to provide abstraction for the synchronisation primitives such that their performance impact can be minimised (i.e. it internally uses k_mutex and k_sem when CONFIG_USERSPACE=n, and atomic integer and k_futex when CONFIG_USERSPACE=y).
Being unable to dynamically allocate these locks imposes severe restrictions for their practical use and forces applications to use k_mutex and k_sem instead: for instance, any OOP class-like dynamically-instantiable primitives containing synchronisation objects (e.g. dynamically instantiated MQTT client containing a mutex) will have to use k_mutex and k_sem with k_object_alloc instead of using sys_mutex and sys_sem.
This leads to severe performance impact and defeats the purpose of having the sys_mutex and sys_sem abstractions in the first place.
Describe the solution you'd like
Background information
In theory, both sys_mutex and sys_sem should be using k_futex; but, due to the limitations of the current Zephyr kernel implementation, only sys_sem does.
Prior to implementing one of the following solutions, the sys_mutex should be updated to use k_futex first (corresponds to the issue #15138). The following solutions assume that sys_mutex has been updated to use k_futex.
Solution 1 (ideal)

Rework the k_futex mechanism such that struct z_futex_data resides in user-accessible memory location (#25497).
Embed struct z_futex_data into struct sys_mutex and struct sys_sem.

Solution 2 (if we don't make progress on #25497)

Keep kernel-side struct z_futex_data (i.e. assuming #25497 is not implemented).
Make struct z_futex_data kernel object dynamically allocable.
Add a kernel API to link a dynamically allocated kernel object (i.e. z_futex_data) to a dynamically allocated user mode object (i.e. k_futex).

DISCLAIMER: The "solutions" suggested above are meant serve as only general guideline for resolving this issue. Some very in-depth details may not be fully correct.
Additional context
See https://github.com/zephyrproject-rtos/zephyr/pull/36201#discussion_r650628137 for the analysis on the status quo.

cc @dcpleung @andyross @ioannisg @carlescufi

It might be a good idea to fold this in with a lot of other requirements/issues around the synchronization primitives.  It's been mentioned in a few places but really what I think we need to do is condense all the gadgets to a shared backend.  Locks within the  same memory domain should always become atomic_cas operations.  The kernel should take a syscall/API entry only on contention.  Priority inheritance vs. semaphore counting should be a runtime/backend decision (and only on contention).  Spinning (and SMP relaxation!) should probably be part of the same framework, etc...


Version question

Hi
I assembled a genome "recently" with FALCON 0.4.0 - which worked great by the way!
Now we thought that for that particular project FALCON-UNZIP might be very informative and I wanted to give it a try.
I am a little bit confused now regarding the correct version of FALCON-UNZIP. Obviously FALCON is now already at version 0.7 and installing the dependencies for FALCON-UNZIP I realized that everything seems to be switching now to the new BAM standard in smrtpipe3.0.
So I guess my question is, whether I can run FALCON-UNZIP with FALCON 0.4.0, or what version of FALCON-UNZIP I would have to use in order to be compatible.
Another option would be to install FALCON 0.7.0 and the latest FALCON-UNZIP but will this be compatible with the output from FALCON 0.4.0 ?
The genome is > 2Gbp and re-running currently not an option.
Thanks for any clarification
Cheers

@ebioman When I developed the FALCON-UNZIP, I have to use a frozen version of FALCON (v.0.4.*).  However, FALCON-UNZIP only depends on the files / directory structures which I don't think that has been changed up to v.0.7.   It does need the final las files for both 0- and 1- stages.

Thank you very much - clarified things.


Difference in generate paf and parse keypoint with Open pose

Hi, thanks for great job!
May I am not clear, but I find your repo using other way to generate paf with Openpose and different way to parse connections respectively?

Hi, thanks for replying soon.
I know that they are same format but in Openpose the author use greedy and munkres algorithm to group keypoints while your code just using greedy algorithm?

This is because the code follows this work, which optimizes "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields". So no munkres algorithm here.

Oh, I understand. Great! I've read your paper and find "Author reported inference speed of 4.2 fps on 2.8GHz Quad-core CPU and 10 fps on Jetson TX2". Do you use C++ code when testing in Jetson TX2?

These results from some other paper. But it is better to use C++ code for the speed benchmarking.

I got it. Thank you very much!


Change behavior of default version for catalog templates

The defaultVersion field should be the latest version compatible with the current Rancher version. The version field from config.yml should be ignored.

@joshwget whats the current behavior of defaultVersion?

There is no defaultVersion right now in the config.yml. Just version in the config.yml, which really means "the default version compatible with the latest Rancher version".

Closing in favor of #8319 as it should be included with all the file name changes


[BUG] Nil pointer dereference error on restoring the system backup

Describe the bug (🐛 if you encounter this issue)
A system backup which was taken in the v1.4.0-rc1 on restoring stuck in Pending state with nil pointer dereference error.
To Reproduce
Steps to reproduce the behavior:

Install Longhorn v1.4.0-rc1
Create some volumes.
Take the volume backup and system backup.
Create a cluster with Longhorn v1.4.0-rc2.
Restore the system backup.
Restoring stuck in Pending state.

Expected behavior
Restore should complete.
Log or Support bundle
supportbundle_06317708-75d1-48d5-8092-1de9ee200fe7_2022-12-22T20-25-53Z.zip
Environment

Longhorn version: v1.4.0-rc2
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): kubectl

Additional context
If creating system backup in v1.4.0-rc2 and restoring it in the same version doesn't have any problem.

The nil pointer comes from the engineImage.Name in rc-1.
And resolved in rc2 here.
System backup collected the manager image of the current cluster, and use that image for rolling out the backup resources. This means if restoring to v1.4.0-rc1 will contain that bug.

The test is good but it seems not valid enough because the RC1 is still a transient version. Does RC2 work well if restoring from a backup created by RC2? I think yes, because it's a best criteria .
However, this means we need to pay attention to system backup n restore backward compatibility for the system backup created by a previous version.
@khushboo-rancher Let's add this test case to the regression testing if not yet.

The test is good but it seems not valid enough because the RC1 is still a transient version. Does RC2 work well if restoring from a backup created by RC2? I think yes, because it's a best criteria .
However, this means we need to pay attention to system backup n restore backward compatibility for the system backup created by a previous version.
@khushboo-rancher Let's add this test case to the regression testing if not yet.

Wait. We should only allow restoring the system backup to the cluster with the same Longhorn version. @c3y1huang correct?

Wait. We should only allow restoring the system backup to the cluster with the same Longhorn version. @c3y1huang correct?

Yes, see restore path.

@c3y1huang Will the following snapshots and backups list in volume details page be restored?

I don't it will, because the system restore will only leverage volume backup restore, so the behavior will be the same as restoring volume backup. (the restored volume will have no snapshots)

Tested on v1.4.x-head, the following test scenario results in abnormal behavior:
(1) In cluster A, create volume "test-1" and create pv/pvc in default namespace, all from Longhorn UI
(2) Create a pod using this volume:
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-1
spec:
  containers:
    - name: sleep
      image: busybox
      imagePullPolicy: IfNotPresent
      args: ["/bin/sh", "-c", "while true;do date;sleep 5; done"]
      volumeMounts:
        - name: pod-data
          mountPath: /data
  volumes:
    - name: pod-data
      persistentVolumeClaim:
        claimName: test-1

(3) Create backup for this volume

(4) Create system backup
(5) In cluster B, restore this system backup => Volume test-1 can be found restored on Longhorn UI, but it's in faulted state, the job log keeps printing the following messages forvever:
time="2022-12-26T03:48:36Z" level=debug msg="Restoring PersistentVolumeList" backupTargetURL="s3://yang-test-4@us-east-1/" controller=longhorn-system-rollout engineImage="longhornio/longhorn-engine:v1.4.x-head" error="volume is waiting to restore data" systemRestore=test-1
time="2022-12-26T03:48:37Z" level=debug msg="Restoring PersistentVolumeClaimList" backupTargetURL="s3://yang-test-4@us-east-1/" controller=longhorn-system-rollout engineImage="longhornio/longhorn-engine:v1.4.x-head" error="persistentvolume \"test-1\" not found" systemRestore=test-1

(6) delete the restore job and the faulted volume, and retry the restore process again, somehow the restore can be completed without problems.
(7) In cluster B, create a pod like step 2 to use the restored volume test-1 => After the volume is attached and becomes healthy, there is a abnormal snapshot appearing in the snapshots list:

Tested on v1.4.x-head, the following test scenario results in abnormal behavior:
(1) In cluster A, create volume "test-1" and create pv/pvc in default namespace, all from Longhorn UI (2) Create a pod using this volume:
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-1
spec:
  containers:
    - name: sleep
      image: busybox
      imagePullPolicy: IfNotPresent
      args: ["/bin/sh", "-c", "while true;do date;sleep 5; done"]
      volumeMounts:
        - name: pod-data
          mountPath: /data
  volumes:
    - name: pod-data
      persistentVolumeClaim:
        claimName: test-1

(3) Create backup for this volume  (4) Create system backup
(5) In cluster B, restore this system backup => Volume test-1 can be found restored on Longhorn UI, but it's in faulted state, the job log keeps printing the following messages forvever:
time="2022-12-26T03:48:36Z" level=debug msg="Restoring PersistentVolumeList" backupTargetURL="s3://yang-test-4@us-east-1/" controller=longhorn-system-rollout engineImage="longhornio/longhorn-engine:v1.4.x-head" error="volume is waiting to restore data" systemRestore=test-1
time="2022-12-26T03:48:37Z" level=debug msg="Restoring PersistentVolumeClaimList" backupTargetURL="s3://yang-test-4@us-east-1/" controller=longhorn-system-rollout engineImage="longhornio/longhorn-engine:v1.4.x-head" error="persistentvolume \"test-1\" not found" systemRestore=test-1

(6) delete the restore job and the faulted volume, and retry the restore process again, somehow the restore can be completed without problems. (7) In cluster B, create a pod like step 2 to use the restored volume test-1 => After the volume is attached and becomes healthy, there is a abnormal snapshot appearing in the snapshots list:

For the above abnormal behavior, no need to create volume/pv/pvc manually from Longhorn UI, it can be reproduced by dynamic provisioning as following steps:
In cluster A
(1) kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/statefulset.yaml
(2) create backup for a volume from Longhorn UI
(3) create system backup
In cluster B
(4) restore this system backup => the volume has backup will become faulted state
(5) delete the restored volume and restore job, and then retry restore again => volumes can be restored without error
(6) kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/statefulset.yaml => the volume has backup will have a snapshot with invalid date
@c3y1huang Am I missing anything or have any other restrict or prerequisite for restore?

(5) In cluster B, restore this system backup => Volume test-1 can be found restored on Longhorn UI, but it's in faulted state, the job log keeps printing the following messages forvever:
time="2022-12-26T03:48:36Z" level=debug msg="Restoring PersistentVolumeList" backupTargetURL="s3://yang-test-4@us-east-1/" controller=longhorn-system-rollout engineImage="longhornio/longhorn-engine:v1.4.x-head" error="volume is waiting to restore data" systemRestore=test-1
time="2022-12-26T03:48:37Z" level=debug msg="Restoring PersistentVolumeClaimList" backupTargetURL="s3://yang-test-4@us-east-1/" controller=longhorn-system-rollout engineImage="longhornio/longhorn-engine:v1.4.x-head" error="persistentvolume \"test-1\" not found" systemRestore=test-1

Logs from cluster B
2022-12-27T04:04:27.822650926Z time="2022-12-27T04:04:27Z" level=debug msg="Instance handler updated instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf state, old state stopped, new state error"
2022-12-27T04:04:27.822662076Z time="2022-12-27T04:04:27Z" level=warning msg="Instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf crashed on Instance Manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef at ip-10-0-1-163, try to get log"
2022-12-27T04:04:27.932368856Z time="2022-12-27T04:04:27Z" level=warning msg="Instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf crashed on Instance Manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef at ip-10-0-1-163, try to get log"
2022-12-27T04:04:28.049084766Z time="2022-12-27T04:04:28Z" level=warning msg="Replica pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf that not in the engine mode map is marked as failed, current state error, engine name pvc-2938f36e-e080-4272-9483-66ce4397abee-e-d8f78540, active true" accessMode=rwo controller=longhorn-volume currentEngine=pvc-2938f36e-e080-4272-9483-66ce4397abee-e-d8f78540 frontend=blockdev migratable=false node=ip-10-0-1-163 owner=ip-10-0-1-163 state=attaching volume=pvc-2938f36e-e080-4272-9483-66ce4397abee
2022-12-27T04:04:28.171337705Z time="2022-12-27T04:04:28Z" level=warning msg="Try to get requested log for pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf in instance manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef"
2022-12-27T04:04:28.172695476Z time="2022-12-27T04:04:28Z" level=info msg="Deleting instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.174251657Z time="2022-12-27T04:04:28Z" level=warning msg="Instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf crashed on Instance Manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef at ip-10-0-1-163, try to get log"
2022-12-27T04:04:28.174471006Z time="2022-12-27T04:04:28Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf\", UID:\"09449713-25f1-436f-b938-b591d3e85f79\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"2300\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.175421113Z time="2022-12-27T04:04:28Z" level=error msg="Failed to receive log for instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf" error="rpc error: code = NotFound desc = cannot find process pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.175443082Z time="2022-12-27T04:04:28Z" level=warning msg="cannot get crash log for instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf on Instance Manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef at ip-10-0-1-163" error="rpc error: code = NotFound desc = cannot find process pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.282717593Z time="2022-12-27T04:04:28Z" level=info msg="Deleting instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.284004630Z time="2022-12-27T04:04:28Z" level=warning msg="Instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf crashed on Instance Manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef at ip-10-0-1-163, try to get log"
2022-12-27T04:04:28.284417742Z time="2022-12-27T04:04:28Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf\", UID:\"09449713-25f1-436f-b938-b591d3e85f79\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"2305\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.285121515Z time="2022-12-27T04:04:28Z" level=error msg="Failed to receive log for instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf" error="rpc error: code = NotFound desc = cannot find process pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.285135831Z time="2022-12-27T04:04:28Z" level=warning msg="cannot get crash log for instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf on Instance Manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef at ip-10-0-1-163" error="rpc error: code = NotFound desc = cannot find process pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.380823798Z time="2022-12-27T04:04:28Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-2938f36e-e080-4272-9483-66ce4397abee\", UID:\"5318b4f9-351e-479b-9cfb-f23bbbe95443\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"2285\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-2938f36e-e080-4272-9483-66ce4397abee started to use/reuse Longhorn volume pvc-2938f36e-e080-4272-9483-66ce4397abee"
2022-12-27T04:04:28.392566140Z time="2022-12-27T04:04:28Z" level=info msg="Deleting instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.393833673Z time="2022-12-27T04:04:28Z" level=warning msg="Instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf crashed on Instance Manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef at ip-10-0-1-163, try to get log"
2022-12-27T04:04:28.394019456Z time="2022-12-27T04:04:28Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf\", UID:\"09449713-25f1-436f-b938-b591d3e85f79\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"2307\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.395203717Z time="2022-12-27T04:04:28Z" level=error msg="Failed to receive log for instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf" error="rpc error: code = NotFound desc = cannot find process pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.395256562Z time="2022-12-27T04:04:28Z" level=warning msg="cannot get crash log for instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf on Instance Manager instance-manager-r-cb97de6b6f131c858bbc075efdf49eef at ip-10-0-1-163" error="rpc error: code = NotFound desc = cannot find process pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf"
2022-12-27T04:04:28.778365206Z time="2022-12-27T04:04:28Z" level=warning msg="Replica pvc-2938f36e-e080-4272-9483-66ce4397abee-r-2ddda446 that not in the engine mode map is marked as failed, current state error, engine name pvc-2938f36e-e080-4272-9483-66ce4397abee-e-d8f78540, active true" accessMode=rwo controller=longhorn-volume currentEngine=pvc-2938f36e-e080-4272-9483-66ce4397abee-e-d8f78540 frontend=blockdev migratable=false node=ip-10-0-1-163 owner=ip-10-0-1-163 state=attaching volume=pvc-2938f36e-e080-4272-9483-66ce4397abee
2022-12-27T04:04:28.820124594Z time="2022-12-27T04:04:28Z" level=debug msg="Instance handler updated instance pvc-2938f36e-e080-4272-9483-66ce4397abee-r-eaa80eaf state, old state error, new state stopped"
2022-12-27T04:04:29.731408367Z time="2022-12-27T04:04:29Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-2938f36e-e080-4272-9483-66ce4397abee\", UID:\"5318b4f9-351e-479b-9cfb-f23bbbe95443\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"2348\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-2938f36e-e080-4272-9483-66ce4397abee has been detached"
2022-12-27T04:04:57.213448461Z time="2022-12-27T04:04:57Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:57.411356801Z time="2022-12-27T04:04:57Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:57.434589657Z time="2022-12-27T04:04:57Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:57.990938625Z time="2022-12-27T04:04:57Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:57.999917446Z time="2022-12-27T04:04:57Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:58.010601153Z time="2022-12-27T04:04:58Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:58.022101246Z time="2022-12-27T04:04:58Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:58.040252915Z time="2022-12-27T04:04:58Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:58.061237978Z time="2022-12-27T04:04:58Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:04:58.084689929Z time="2022-12-27T04:04:58Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-163] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:05:02.544264665Z time="2022-12-27T04:05:02Z" level=debug msg="CheckEngineImageReadiness: nodes [ip-10-0-1-140] don't have the engine image longhornio/longhorn-engine:v1.4.0-rc2"
2022-12-27T04:05:02.651039867Z time="2022-12-27T04:05:02Z" level=debug msg="Engine Image Controller ip-10-0-1-163 picked up ei-6eab9564 (longhornio/longhorn-engine:v1.4.0-rc2)" controller=longhorn-engine-image engineImage=ei-6eab9564 image="longhornio/longhorn-engine:v1.4.0-rc2" node=ip-10-0-1-163
2022-12-27T04:05:03.239491348Z time="2022-12-27T04:05:03Z" level=info msg="Event(v1.ObjectReference{Kind:\"EngineImage\", Namespace:\"longhorn-system\", Name:\"ei-6eab9564\", UID:\"b7ef12e8-7eae-46f1-84e2-33df519ef34e\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"2495\", FieldPath:\"\"}): type: 'Normal' reason: 'ready' Engine image ei-6eab9564 (longhornio/longhorn-engine:v1.4.0-rc2) is fully deployed on all ready nodes"

The replica failed at executing engine binary before engine image is fully deployed on all ready nodes.

(5) In cluster B, restore this system backup => Volume test-1 can be found restored on Longhorn UI, but it's in faulted state, the job log keeps printing the following messages forvever:
time="2022-12-26T03:48:36Z" level=debug msg="Restoring PersistentVolumeList" backupTargetURL="s3://yang-test-4@us-east-1/" controller=longhorn-system-rollout engineImage="longhornio/longhorn-engine:v1.4.x-head" error="volume is waiting to restore data" systemRestore=test-1
time="2022-12-26T03:48:37Z" level=debug msg="Restoring PersistentVolumeClaimList" backupTargetURL="s3://yang-test-4@us-east-1/" controller=longhorn-system-rollout engineImage="longhornio/longhorn-engine:v1.4.x-head" error="persistentvolume \"test-1\" not found" systemRestore=test-1

Ref: https://github.com/longhorn/longhorn-manager/pull/1658

(6) kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/statefulset.yaml => volume states transfer from detached to healthy, and the volume has backup will have a snapshot with invalid date

This doesn't look related to the system-backup/restore and should be reproducible without the system restore on cluster-B:

Restore a volume from backup.
Create PV/PVC with the same name.
Attach to the workload the same as cluster-A.

@c3y1huang is this issue happening to 1.3? @yangchiu can help check?

@c3y1huang will this happen when restoring the volume in the same cluster?

Yes.

@c3y1huang is this issue happening to 1.3.2 or 1.3.x? @yangchiu can help check?

Yes, this also happens in v1.3.x-head.

Thanks @c3y1huang @yangchiu and @chriscchien for the update. Let's track the invalid createdtime of snapshots of a restored volume at #5153.

Tested on v1.4.x-head (longhorn-manager 9d57b4c). Trying to restore a volume with backing image, data is missing from the restored volume:
In cluster A:
(1) Create a volume with backing image
(2) Create a pod to use this volume:
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-1
spec:
  containers:
    - name: sleep
      image: busybox
      imagePullPolicy: IfNotPresent
      args: ["/bin/sh", "-c", "while true;do date;sleep 5; done"]
      volumeMounts:
        - name: pod-data
          mountPath: /data
  volumes:
    - name: pod-data
      persistentVolumeClaim:
        claimName: test-1

(3) Write some data to the volume:
dd if=/dev/urandom of=/data/test-1 bs=1M count=16

(4) Create system backup
In cluster B:
(5) Manually create backing image
(6) Restore the system backup
(7) The volume with backing image can be restored and in detached state
(8) Create a pod to use this restored volume like in step (2)
(9) Check the volume data by kubectl exec -it test-pod-1 -- /bin/sh, there is no test-1 file created in step (3)

Manually create a volume backup and restore the volume backup, the test-1 file can be found in the restored volume and the content is intact.

Tested on v1.4.x-head (longhorn-manager 9d57b4c). Trying to restore a volume with backing image, data is missing from the restored volume:
In cluster A: (1) Create a volume with backing image (2) Create a pod to use this volume:
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-1
spec:
  containers:
    - name: sleep
      image: busybox
      imagePullPolicy: IfNotPresent
      args: ["/bin/sh", "-c", "while true;do date;sleep 5; done"]
      volumeMounts:
        - name: pod-data
          mountPath: /data
  volumes:
    - name: pod-data
      persistentVolumeClaim:
        claimName: test-1

(3) Write some data to the volume:
dd if=/dev/urandom of=/data/test-1 bs=1M count=16

(4) Create system backup
In cluster B: (5) Manually create backing image (6) Restore the system backup (7) The volume with backing image can be restored and in detached state (8) Create a pod to use this restored volume like in step (2) (9) Check the volume data by kubectl exec -it test-pod-1 -- /bin/sh, there is no test-1 file created in step (3)
* Manually create a volume backup and restore the volume backup, the `test-1` file can be found in the restored volume and the content is intact.

It's expected because volume backup has to be created to restore the data in another cluster.
From @c3y1huang If the system backup volume doesn't contain volume backup then the data will not get restored in cluster-B. Instead it will restore the volume only.

Verified passed on v1.4.x-head (longhorn-manager 9d57b4c). Different kinds of volumes (Created from Longhorn UI, dynamic provisioning, RWO, RWX, volume with backing image, encrypted volume) can all be backup and restored to the new cluster without problem.

Verified passed on v1.4.x-head (longhorn-manager 9d57b4c). Different kinds of volumes (Created from Longhorn UI, dynamic provisioning, RWO, RWX, volume with backing image, encrypted volume) can all be backup and restored to the new cluster without problem.

This is a good test to cover different types of volume creation. cc @longhorn/qa


Rebase mlir-aie on main

This pulls in the changes on the mlir-aie main branch:
cd third_party/mlir-aie
git remote add xilinx git@github.com:Xilinx/mlir-aie.git
git fetch xilinx
git rebase xilinx/main # (has a merge conflict which I resolved)
Motivation: I'd like to start experimenting with the patch https://github.com/Xilinx/mlir-aie/pull/844

I think you need to have the rebase within the mlir/nod-ai repo, otherwise it will break for other users

ALso, @newling I think you want to make another branch called iree-fixes2 rather than rebasing (and hence force pushing) the old one since it will break the flow for old users when there setup looks for the commits that now wont exist.

Ok, I think I understand now. I will

rebase nod.ai/mlir-air:iree_fixes onto Xilinx/mlir-air:main,
make PR in nod.ai/mlir-air (I don't think I have write permission)
come back to this PR in iree-amd-aie.

(and will keep @Abhishek-Varma updated).

That would work, but also you should have write permissions for nod-ai/mlir-air/aie , @stellaraccident could you please add James. Thanks.

nod-ai/mlir-air/aie and nod-ai/mlir-air/air please, @stellaraccident

Was missing the AMD team on mlir-air. Was on the other. Should work now.

As a side note: the addition of type/attribute name fields can be done independent of llvm version bump. If on the old version, the presence of the additional field won't hurt anything, and that has been the only hard API break I've seen this month.

As a side note: the addition of type/attribute name fields can be done independent of llvm version bump. If on the old version, the presence of the additional field won't hurt anything, and that has been the only hard API break I've seen this month.

I think you're talking about this? https://github.com/Xilinx/mlir-aie/pull/809/files

Just btw/fyi I bump aie's llvm dep at least once a week (and often more frequently than even that).

Was missing the AMD team on mlir-air. Was on the other. Should work now.

Thanks!

As a side note: the addition of type/attribute name fields can be done independent of llvm version bump. If on the old version, the presence of the additional field won't hurt anything, and that has been the only hard API break I've seen this month.

Good point, that's what air did with https://github.com/Xilinx/mlir-air/pull/349 -- didn't actually bump llvm.

Closing in favor of https://github.com/nod-ai/iree-amd-aie/pull/47