UNIQUE constraint failed: snapshot_excludes.hash, snapshot_excludes.path

Version: v0.2.9 / e5f58fd65ce4ec14aa493164ba56d59a1da9641a
$ RUST_BACKTRACE=1 cargo run -- -r "sftp:user@example:/path/to/repo" --password-file=/path/to/key
[…]
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.58s
     Running `target/debug/redu -r "sftp:user@example:/path/to/repo" --password-file=/path/to/key`
Logging to "/home/user/.local/share/redu/logs/2024-09-16T22:49:59.596074327+02:00.log"
 (●    ) Getting restic config                                                                                Using cache file "/home/user/.cache/redu/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.db"
 (●    ) Fetching repository snapshot list                                                                                Error: error in db thread

Caused by:
    0: UNIQUE constraint failed: snapshot_excludes.hash, snapshot_excludes.path
    1: Error code 1555: A PRIMARY KEY constraint failed

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.86/src/backtrace.rs:27:14
   1: core::ops::function::FnOnce::call_once
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/core/src/ops/function.rs:250:5
   2: core::result::Result<T,E>::map_err
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/core/src/result.rs:854:27
   3: redu::sync_snapshots::{{closure}}::{{closure}}
             at ./src/main.rs:247:17
   4: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/sys/backtrace.rs:154:18
   5: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/thread/mod.rs:522:17
   6: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/core/src/panic/unwind_safe.rs:272:9
   7: std::panicking::try::do_call
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/panicking.rs:557:40
   8: __rust_try
   9: std::panicking::try
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/panicking.rs:520:19
  10: std::panic::catch_unwind
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/panic.rs:348:14
  11: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/thread/mod.rs:521:30
  12: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/core/src/ops/function.rs:250:5
  13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/alloc/src/boxed.rs:2453:9
  14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/alloc/src/boxed.rs:2453:9
  15: std::sys::pal::unix::thread::Thread::new::thread_start
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/sys/pal/unix/thread.rs:105:17
  16: start_thread
  17: clone3

last lines from the log:
[…]
20:50:49 [INFO] redu::restic: running ""restic" "--repo" "sftp:user@example:/path/to/repo" "--password-file" "/path/to/key" "--json" "ls" "fdf9fdc755120d9ccaf87f3a47d305b88cc340b58bb9996a353cefce7b089f93"" (pid 29182)
20:50:50 [INFO] redu::restic: finished pid 28893
20:50:50 [INFO] redu: snapshot fetched in 25.057822208s (24fb842)
20:50:50 [INFO] redu::restic: running ""restic" "--repo" "sftp:user@example:/path/to/repo" "--password-file" "/path/to/key" "--json" "ls" "f7674983aeff1c65ab56f6d6656d12339e5fa7b5fbb30719fbd08dd41068265a"" (pid 29198)
20:50:50 [INFO] redu: waited 8.291456486s to save snapshot (891462 files)
20:50:50 [ERROR] sqlite: (1555) abort at 12 in [INSERT INTO snapshot_excludes (hash, path) VALUES (?, ?)]: UNIQUE constraint failed: snapshot_excludes.hash, snapshot_excludes.path

Cool project!

Hello! Thank you for the bug report!
It seems that restic might be returning the same path more than once in an excludes list.
I have tested locally creating a snapshot passing the same exclude path twice and indeed restic returns the same directory twice from restic snapshots --json.

If you could run restic snapshots --json | jq '.[] | .excludes | select(.)' and either paste it somewhere or confirm that this is indeed what's happening in your case that would be quite helpful.

That seems plausible, i have around 1000 snapshots, with the oldest from 2016.
With -j 1 i was able to pick the snapshot that redu tripped up on in the logs. That snapshot indeed has duplicate excludes (slightly redacted):
    "excludes": [
      "/home/clonejo/.cache",
      "/home/clonejo/.cargo/bin",
      "/home/clonejo/.cargo/git",
      "/home/clonejo/.cargo/registry",
      "/home/clonejo/.gem",
      "/home/clonejo/.gradle/caches",
      "/home/clonejo/.local/share/Steam",
      "/home/clonejo/.local/share/Steam-ext",
      "/home/clonejo/.local/share/nvim/plugged",
      "/home/clonejo/.m2/repository",
      "/home/clonejo/.mozilla/firefox/*/storage/default/*/cache",
      "/home/clonejo/.npm/_cacache",
      "/home/clonejo/.pyenv",
      "/home/clonejo/.rbenv",
      "/home/clonejo/.rustup/toolchains",
      "/home/clonejo/.vagrant.d",
      "/home/clonejo/.wine/drive_c/windows/Installer",
      "/home/clonejo/Downloads",
      "/home/clonejo/dotfiles/.vim/vundle",
      "/home/clonejo/downloads",
      "/home/clonejo/media",
      "/home/clonejo/tmp",
      "/root/.cache",
      "/var/cache",
      "target/classes",
      "target/debug",
      "target/doc",
      "target/release",
      ".mypy_cache",
      ".venv",
      ".thumbnails",
      "node_modules",
      "target/debug",
      "target/doc",
      "target/release",
      "target/rls",
      "venv"
    ],

Of course it is a commit from 3 years ago :)

PR #77 fixes this if you want to test if it works for you. Otherwise no worries I've tested it locally and will merge and release it later.

Thanks for the fix! I can confirm that redu 0.2.10 does indeed successfully process my restic repo.

Thanks for the fix! I can confirm that redu 0.2.10 does indeed successfully process my restic repo.

Awesome. Thanks!


issue #874 chore: build arm64 as part of release flow

@gdbranco created a ticket to track this https://issues.redhat.com/browse/OCM-2418


add c variant for old str formatting

Its important for example for:
print(datetime.now())
And may be checked in this way:
"%c" % 107 # gives letter 'k'
and this:
"%c" % "k" # gives letter 'k' too

I see, that it fails. Thinking its due to a different text editor settings. But cant check it locally now. Command 'python setup.py develop' fails with message 'unsupported Go version , Grumpy requires at least 1.9...'. But my go version is go1.10.3 linux/amd64.

Yes, I wanted to add a test. I got it how the tests work. I'm running on another system now. Further I will understand, why I had problem in windows linux subsystem (Debian).

Thanks for the test.
I wanted to explore the Windows Linux Subsystem too. Good to know that it messed with your Go versions.
In theory it should work and no Windows port would be needed at all, but lets see...


Provide a mechanism for an extension to determine or set a preferred kernel.

As a corollary to #4363, there should be a way for a VS Code concept of a kernel to map to an existing .ipynb file without requiring an installed kernelspec file.  E.g., consider a situation where a user installs this extension as well as the .NET Interactive extension.  From there they open an existing .ipynb file.  At that point, there is no mechanism for the VS Code version of the .NET Interactive kernel to be auto-selected.
This issue is hopefully more of a discussion about how to add this functionality, and the end result may require changes from VS Code itself.
As a single data point, if a new .ipynb notebook is created, the kernel set to .NET Interactive and it's saved, the notebook's metadata looks like this:
...
"metadata": {
    "language_info": {
        "name": "csharp",
        "nbconvert_exporter": "python"
    },
    "orig_nbformat": 2
}
...

Thanks @brettfo
We have discussed this internally & our plan was to not pre-select the non-python kernels unless we had an exact match for the kernel.
For instance in this example, we might not have a matching kernel & we shoudlnt' ideally pre-select any kernel.
Then the .net extension can nominate its kernel as a preferred kernel using the NotebookKernel.isPreferred property.

Thanks @brettfo
We have discussed this internally & our plan was to not pre-select the non-python kernels unless we had an exact match for the kernel.
For instance in this example, we might not have a matching kernel & we shoudlnt' ideally pre-select any kernel.
Then the .net extension can nominate its kernel as a preferred kernel using the NotebookKernel.isPreferred property.

At our end the fix woudl be in getKernelForLocalRawConnection.
	const kernelSpec = await this.kernelFinder.findKernelSpec(resource, notebookMetadata, cancelToken);
....
....
        if (!kernelSpec && activeInterpreter) {
            await this.installDependenciesIntoInterpreter(activeInterpreter, ignoreDependencyCheck, cancelToken);

            // Return current interpreter.
            return {
                kind: 'startUsingPythonInterpreter',
                interpreter: activeInterpreter
            };
        } else if (kernelSpec) {

To discuss

Should we ever preselect a kernel & just add an exception for .NET interactive?

I think yes, majority of our (Python/Jupyter) users will have our extension.
We can add exceptions for .NET kernels & look for a more generic solution later (if other extensions do not like this pre-selection of kernels by jupyter extension).
If the preselected kernel is a .NET kernel & .NET interactive is installed, then never preselect a .NET kernelspec

Note:
@brettfo VS Code has added the ability for users to define the default kernel provider.
I.e. user can go into the kernel picker and chose .NET kernels as their preferred kernels. This way when .NET extension exposes kernels & has isPreferred set to true, then VS Code will always use those instead of those contributed by Jupyter extension or any other. I.e. the user has the ability to configure their preference.

At our end the fix woudl be in getKernelForLocalRawConnection.
	const kernelSpec = await this.kernelFinder.findKernelSpec(resource, notebookMetadata, cancelToken);
....
....
        if (!kernelSpec && activeInterpreter) {
            await this.installDependenciesIntoInterpreter(activeInterpreter, ignoreDependencyCheck, cancelToken);

            // Return current interpreter.
            return {
                kind: 'startUsingPythonInterpreter',
                interpreter: activeInterpreter
            };
        } else if (kernelSpec) {

To discuss

Should we ever preselect a kernel & just add an exception for .NET interactive?

I think yes, majority of our (Python/Jupyter) users will have our extension.
We can add exceptions for .NET kernels & look for a more generic solution later (if other extensions do not like this pre-selection of kernels by jupyter extension).
If the preselected kernel is a .NET kernel & .NET interactive is installed, then never preselect a .NET kernelspec

Note:
@brettfo VS Code has added the ability for users to define the default kernel provider.
I.e. user can go into the kernel picker and chose .NET kernels as their preferred kernels. This way when .NET extension exposes kernels & has isPreferred set to true, then VS Code will always use those instead of those contributed by Jupyter extension or any other. I.e. the user has the ability to configure their preference.

On a related note, is there a way that our extension can set the .ipynb kernelspec metadata?  I'm thinking of this scenario where I don't have any Jupyter kernels installed locally, but I create a new notebook and want to share it with somebody that's using Jupyter Lab.
I tried setting the notebook document's metadata to add the kernelspec part, but it wasn't serialized to disk; I'm guessing because your extension is explicitly setting that iff there's a Jupyter kernel specified.  Can we have an override for that metadata?
This paired with some hard-coded logic to select our kernel if notebook.metadata.kernelspec.name === '.net-csharp' should get us everything we need.

On a related note, is there a way that our extension can set the .ipynb kernelspec metadata?  I'm thinking of this scenario where I don't have any Jupyter kernels installed locally, but I create a new notebook and want to share it with somebody that's using Jupyter Lab.
I tried setting the notebook document's metadata to add the kernelspec part, but it wasn't serialized to disk; I'm guessing because your extension is explicitly setting that iff there's a Jupyter kernel specified.  Can we have an override for that metadata?
This paired with some hard-coded logic to select our kernel if notebook.metadata.kernelspec.name === '.net-csharp' should get us everything we need.

@brettfo
The metadata is stored in  the document.metadata.custom property. This stores data thats of type Partial<nbformat.INotebookContent> where nbformat is a namespace from the Jupyter package. See here import { nbformat } from '@jupyterlab/coreutils';
You could update the metadata using the VSCode API.
If the object is null, then initialize it with a simple dict thats matches teh type of Partial<nbformat.INotebookContent>.
Once done, that should work.
However I can also see there's a bug at our end, we need to clear the kernel selection when user selects another kenerl (we update the metadata with the last selected Jupyter kernel).
Simple enough fix at our end.

@brettfo
The metadata is stored in  the document.metadata.custom property. This stores data thats of type Partial<nbformat.INotebookContent> where nbformat is a namespace from the Jupyter package. See here import { nbformat } from '@jupyterlab/coreutils';
You could update the metadata using the VSCode API.
If the object is null, then initialize it with a simple dict thats matches teh type of Partial<nbformat.INotebookContent>.
Once done, that should work.
However I can also see there's a bug at our end, we need to clear the kernel selection when user selects another kenerl (we update the metadata with the last selected Jupyter kernel).
Simple enough fix at our end.

@DonJayamanne thanks for the metadata pointer, I'm now able to set kernelspec metadata (and any other metadata) and get it serialized on save.
So now we just need a way for this document metadata to somehow be mappable to our VS Code kernel when there are no Jupyter kernels installed on disk.  E.g., if jupyter kernelspec list returns nothing or if I don't have anything Jupyter-related installed, then how can the document/kernelspec metadata that looks like this:
...
"metadata": {
    "kernelspec": {
        "display_name": ".NET (C#)",
        "name": ".net-csharp"
    }
}
...

know to map to the VS Code concept of a kernel with id === 'dotnet-interactive'?  Should we add "vscode": { "extension_id": "ms-dotnettools.dotnet-interactive-vscode", "kernel_id": "dotnet-interactive" } to the metadata as well? My only issue with that is that Jupyter Lab will only write out the kernelspec part without us developing a custom Jupyter extension purely to write this additional metadata.

@DonJayamanne thanks for the metadata pointer, I'm now able to set kernelspec metadata (and any other metadata) and get it serialized on save.
So now we just need a way for this document metadata to somehow be mappable to our VS Code kernel when there are no Jupyter kernels installed on disk.  E.g., if jupyter kernelspec list returns nothing or if I don't have anything Jupyter-related installed, then how can the document/kernelspec metadata that looks like this:
...
"metadata": {
    "kernelspec": {
        "display_name": ".NET (C#)",
        "name": ".net-csharp"
    }
}
...

know to map to the VS Code concept of a kernel with id === 'dotnet-interactive'?  Should we add "vscode": { "extension_id": "ms-dotnettools.dotnet-interactive-vscode", "kernel_id": "dotnet-interactive" } to the metadata as well? My only issue with that is that Jupyter Lab will only write out the kernelspec part without us developing a custom Jupyter extension purely to write this additional metadata.

only issue with that is that Jupyter Lab will only write out the kernelspec part without us developing a custom Jupyter

In this instance, doesn't this mean the user already has Jupyter kernels installed.

Changing this to point release so this works for the .NET team's release date.

@rchiodo Do we have a specific validation here? Did this work for .NET interactive release? If so I would consider that validation.

For validation: Install both us and .NET interactive. Make sure python and .NET notebooks work.

@rchiodo Not sure if this is on our side our their side. But currently when I do the command to create a new blank .NET Interactive notebook it pops up and briefly shows C# (.NET Interactive) as the cell language. Then immediately, without changing it manually, it changes to Python.  When launching it tries to launch with my current interpreter.

For an existing notebook that was saved to use the .NET kernel, does that still open and use the .NET kernel? I believe that's what this bug was about.
Creating blank notebooks is a separate issue.

That scenario doesn't work right for me either. It actually does open the saved .NET notebook correctly. But I then opened up a previous saved python notebook of mine. It started up a .net kernel when I opened it, and changed all the cell languages to C#. You can see both kernels starting up in the .Net interactive log. VariableTest.ipynb is a python notebook that I use frequently that has a saved kernelspec.

@rchiodo Preference on me pulling this back to backlog versus resolving it (since the saved .NET file does open and use the .NET kernel) and opening a new issue for the issue above?

That last part is the bug then. That's what is supposed to work. We should open a new bug on .NET interactive to fix that.

I think we would then mark this bug as upstream.

Sorry, didn't get around to looking at this until after lunch. I see the same behavior on stable VS Code. Still sounds like from what you said it's upstream either way, so I'll file an issue.


zombie goroutines in client due to missing ClientStreams

Description
Reading a large blob causes hanging goroutines of type grpc.newClientStream in the client.
Detail
It appears that each time we call ReadAt(), it creates a new stream, which is not marked as ClientStreams: true. This, in turn, makes it mark lastSent: true immediately, and never closes the client stream.
Each "ReadAt" chunk is 1MB, so you end up with one lingering goroutine per 1MB of blob size. Pull a 1GB blob, get ~1000 goroutines.
Steps to reproduce the issue:
I am working on a narrow reproducibility case, which I will post here as soon as I have code for it.
Output of containerd --version:
Latest client API
Any other relevant information:
I believe the source of it is here
	Streams: []grpc.StreamDesc{
		{
			StreamName:    "List",
			Handler:       _Content_List_Handler,
			ServerStreams: true,
		},
		{
			StreamName:    "Read",
			Handler:       _Content_Read_Handler,
			ServerStreams: true,
		},
		{
			StreamName:    "Write",
			Handler:       _Content_Write_Handler,
			ServerStreams: true,
			ClientStreams: true,
		},
	},

Note that only the "Write" stream has ClientStreams set to true, while the others are not set (default to false).
When we call contentClient.Read here, we create a new stream, passing the desc of the "Read", which, as stated above, is defaulted to false:
func (c *contentClient) Read(ctx context.Context, in *ReadContentRequest, opts ...grpc.CallOption) (Content_ReadClient, error) {
	stream, err := c.cc.NewStream(ctx, &_Content_serviceDesc.Streams[1], "/containerd.services.content.v1.Content/Read", opts...)
	if err != nil {
		return nil, err
	}
	x := &contentReadClient{stream}
	if err := x.ClientStream.SendMsg(in); err != nil {
		return nil, err
	}
	if err := x.ClientStream.CloseSend(); err != nil {
		return nil, err
	}
	return x, nil
}

The really interesting part is when the above code calls SendMsg. Since ClientStreams is set to false, it immediately sets sentLast = true in grpc, see here:
	if !cs.desc.ClientStreams {
		cs.sentLast = true
	}

Because of this, when we call CloseSend() in our code above, it gets ignored in grpc, see here
func (cs *clientStream) CloseSend() error {
	if cs.sentLast {
		// TODO: return an error and finish the stream instead, due to API misuse?
		return nil
	}

They even have a comment for it above: TODO: return an error and finish the stream instead, due to API misuse?
Unless I misunderstood, we are doing the API misuse they reference?
It is possible I completely misunderstood it, but I see no way how we can read data without creating those lingering zombie goroutines.
As soon as I can get a simple piece of code to reproduce it, I will post here. I can reproduce it, but only as part of a larger library, which will confuse.

OK, here is the simplest way I have to reproduce it:

Load some large blob into containers. I did it by creating an arbitrary file using dd if=/dev/random of=largefile bs=1M count=1024, which gave me a 1GB file.
Run ctr content get
Find the ctr client process using ps
Run kill -ABRT <ctr process>
See hundreds to thousands of zombie goroutines, e.g.:

goroutine 13617 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc.newClientStream.func5(0xc00016ee00, 0xc006466d80, 0x55c5cc22ee20, 0xc006bdff80)
        /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/stream.go:319 +0xd9
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.newClientStream
        /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/stream.go:318 +0x993

goroutine 13673 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc.newClientStream.func5(0xc00016ee00, 0xc0062b1b00, 0x55c5cc22ee20, 0xc005574f60)
        /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/stream.go:319 +0xd9
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.newClientStream
        /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/stream.go:318 +0x993

In my case:
$ sudo ctr -n avitest content get sha256:cf64f69f1034a69ad39685dab4d8b2ee403d2f8860821b2dfec0154ff5bb1b47 > /tmp/content &
$ ps -ef | grep ctr
root     26409 15440  0 09:34 pts/0    00:00:00 sudo ctr -n avitest content get sha256:cf64f69f1034a69ad39685dab4d8b2ee403d2f8860821b2dfec0154ff5bb1b47
root     26410 26409 98 09:34 pts/0    00:00:01 ctr -n avitest content get sha256:cf64f69f1034a69ad39685dab4d8b2ee403d2f8860821b2dfec0154ff5bb1b47
$ sudo kill -ABRT 26410
// tons and tons of goroutines here.

Looks like my analysis isn't complete. Calling CloseSend() is not sufficient according to the comments on NewStream:
To ensure resources are not leaked due to the stream returned, one of the following actions must be performed:

1. Call Close on the ClientConn.
2. Cancel the context provided.
3. Call RecvMsg until a non-nil error is returned. A protobuf-generated
   client-streaming RPC, for instance, might use the helper function
   CloseAndRecv (note that CloseSend does not Recv, therefore is not
   guaranteed to release all resources).
4. Receive a non-nil, non-io.EOF error from Header or SendMsg.

If none of the above happen, a goroutine and a context will be leaked, and grpc will not call the optionally-configured stats handler with a stats.End message.

So calling CloseSend(), while a good idea, will not do it. I just rant a test where I modified the code, rebuilt ctr, and still had the same issue.
It needs to do one of the above 4 things. I am guessing we don't want to close the ClientConn, and we obviously aren't going to rely on getting a true error from Header or SendMsg, which leaves two options:

cancel the context - this would mean a new context for each Read() call. Could be worse.
Keep calling RecvMsg until we get a non-nil error, although I am not sure how that would work.

I just reread the comments on grpc.ClientConn.NewStream():
// NewStream creates a new Stream for the client side. This is typically
// called by generated code. ctx is used for the lifetime of the stream.
//
// To ensure resources are not leaked due to the stream returned, one of the following
// actions must be performed:
//
//      1. Call Close on the ClientConn.
//      2. Cancel the context provided.
//      3. Call RecvMsg until a non-nil error is returned. A protobuf-generated
//         client-streaming RPC, for instance, might use the helper function
//         CloseAndRecv (note that CloseSend does not Recv, therefore is not
//         guaranteed to release all resources).
//      4. Receive a non-nil, non-io.EOF error from Header or SendMsg.
//
// If none of the above happen, a goroutine and a context will be leaked, and grpc
// will not call the optionally-configured stats handler with a stats.End message.

Specifically: ctx is used for the lifetime of the stream.. I am going to open a PR to fix it for Read(), but it appears to be broader than that. We create a new stream in 6 places:

containers/v1.ListStream`
content/v1.Read - subject of this issue
content/v1.List
content/v1.Write
events/v1.Subscribe
snapshots/v1.List

Every one of these assumes a context lives for a while, when NewStream explicitly expects the context to be done/cancelled when the stream is done, or stuff will leak.
I assume these all are auto-generated via make protos, so the issue is not the generated *.pb.go code itself, but the calling code, which should be cancelling the context.
I will get the remoteReaderAt issue in a PR, but someone should figure out where the rest of them are as well.


Support for LangString in Literal Constructor with dataype iri

Literal is created for langString using the Literal Constructor which takes the lexical form and language tag. Should be able to create langString literals using make().

fixed by #63


Inverted Images Exhibit Droopy Eyes

Hi,
I am trying to use the provided E4E model pre-trained on FFHQ dataset to invert CelebA images into latent codes and generated them back. However, I found that all reconstructed face images exhibit droopy eyes, unlike the originals. do you know what is the cause and probably, how to fix it?
Thank you very much!
Examples:

Hi @falloncandra,
I believe the problem is that the input images are not aligned according to the FFHQ alignment process, try running the ffhq alignment for each image before the inversion (see the collab for an exmaple)
Best,
Omer

Hi, I have tried running the FFHQ alignment process as you suggested, and it worked!
Thank you very much :)

Awsome! closing the issue :)


To do: Support pip dependencies

Thanks for this project! I am using Conda to install a mix of PyPI and Conda packages in a private Julia package, so I would love this feature to migrate towards CondaPkg that looks a lot cleaner :) Best,

Yeah it's a brand new project, but pip support is near the top of my priorities. Feel free to watch this issue for updates.

Done, but see #13.

Thanks! :)


Unpin as many dependencies as possible

We currently have almost all dependencies pinned to a specific version, some of which are become deprecated. Un pin and use latest wherever possible

bumps==0.9.0
corner==2.1.0
dynesty==1.0.1
importlib_resources==5.8.0
matplotlib==3.2.2
numpy==1.22.0
periodictable==1.6.1
refl1d==0.8.14
refnx==0.1.19
scipy==1.9.2
tqdm==4.64.0

There are no pinned dependencies  within the dependencies, so this should be okay to just remove and merge


is CropMaxDimensions() supposed to compress Image size from Camera ?

I set CropMaxDimensions(640, 360) in my App
and I ran some Test:
In Device with Android 9:
Image from Gallery with size 16MB can be compress to 760kb
and Image from Camera with size 5.7MB can be compress to 380kb
In Device with Android 10:
Image from Gallery with size 16MB can be compress to 760kb
but Image from Camera with size 3.5MB only compress to 3.4MB
is this possible ?

Hey @deanwinchester4848 welcome to our library community.

When open an issue, please follow our templates to help us to understand and help you.

I believe you are talking about the method setMaxCropResultSize(int, int) can you confirm if this is the case?
Your final statement is is this possible ? don't look like a bug for me, but a question. We have the Q&A Discussion for this here
But about your question. The method setMaxCropResultSize is related to the width and height of the image, not related to the size of it. Any size change is a side effect of this method.
Can you confirm if I understood this right? Thanks

Hey @deanwinchester4848 welcome to our library community.

When open an issue, please follow our templates to help us to understand and help you.

I believe you are talking about the method setMaxCropResultSize(int, int) can you confirm if this is the case?
Your final statement is is this possible ? don't look like a bug for me, but a question. We have the Q&A Discussion for this here
But about your question. The method setMaxCropResultSize is related to the width and height of the image, not related to the size of it. Any size change is a side effect of this method.
Can you confirm if I understood this right? Thanks

Thanks, I get it now.
And is it possible to set the Max Size e.g. 2MB ?

Thanks, I get it now.
And is it possible to set the Max Size e.g. 2MB ?


How do I make a BXL script depend on file contents?

I'm writing a BXL script for linting C++ files with clang-tidy. The script itself is very simple:
def _clang_tidy_impl(ctx: BxlContext):
    actions = ctx.bxl_actions().actions

    # Each target expression expands to a list of targets, so we flatten the
    # input into a flat list of targets.
    targets = []
    for t in ctx.cli_args.targets:
        targets += t

    files = ctx.uquery().inputs(targets)
    cpp_files = []
    for file in files:
        path = file.path
        if path.endswith(".cpp") or path.endswith(".c") or path.endswith(".cc") or path.endswith(".cxx"):
            cpp_files.append(path)

    clang_tidy = ctx.analysis(":clang-tidy").providers()[RunInfo]
    report_file = actions.declare_output("clang_tidy_report.yaml")
    script = actions.write(
        "run_clang_tidy.sh",
        [
            "set -eu",
            cmd_args(
                clang_tidy,
                cpp_files,
                "--export-fixes",
                report_file.as_output(),
                delimiter = " ",
            ),
            # If there are no warnings to report, clang-tidy will not produce an output.
            cmd_args(
                "touch",
                report_file.as_output(),
                delimiter = " ",
            ),
        ],
        with_inputs = True,
        is_executable = True,
    )

    actions.run(
        cmd_args(["/bin/sh", script]),
        category = "clang_tidy",
    )
    ctx.output.ensure(report_file)

# buck2 bxl clang-tidy.bxl:clang_tidy -c build.execution_platforms=some:platform -- --targets demo/...
clang_tidy = bxl_main(
    impl = _clang_tidy_impl,
    cli_args = {
        "targets": cli_args.list(cli_args.target_expr()),
    },
)

(I left out some irrelevant details)
This works well enough as a prototype, except that successes are cached even if I edit a source file and add an error on purpose. I get why this is happening: the build graph doesn't change, so my uquery is cached, so the whole script is cached.
I've tried to add dependencies wherever I could via cmd_args' hidden parameter, but no dice (heh). Buck nags me with a
[2024-08-23T15:34:15.608-04:00] File changed: root//demo/lib.h

but won't let me easily depend on that change.
Is there a way that I can someone add a dependency on the contents of the input files that doesn't involve e.g. reading them via a dynamic action and some hashing?

Main thing: uqueryctx.inputs() gives you a file_set, which if you look in the docs, is just an iterable set of bxl.FileNode. These aren't artifacts, and you're just collecting their paths as strings and writing them into a file, so you can't expect Buck to track them. You need to turn them into source artifacts using ctx.fs.source(node) before you can use them as such.
More things:

Seems that BXL's actions.write does not really support with_inputs = True, even though it has the flag, it doesn't do anything.
BXL also doesn't seem to really implement allow_args = True, even though it has the flag and is documented. The returned list of input artifacts always seems to be empty.
Be aware that buck will print File changed: ... for literally any file that changes, including ones that are not referenced anywhere. Doesn't really mean anything.
This is a bit in the weeds, but generally I wouldn't write the list of files into the script itself. That means buck needs to write a new copy of the script each time the file list changes, i.e. every time you try to clang-tidy a different target. Probably just pass em as args to the script.
And since this is clang-tidy, you may want to do one actions.run per target, so you can configure clang-tidy with include paths from the target in question, for example. Perhaps this is what you have omitted for simplicity. But worth pointing out you probably want ctx.output.ensure_multiple if you're doing that.

I am running a bit of an old Buck2. Maybe those first two BXL issues have been fixed since.
All in all, this works:
def _cat_files_impls(ctx: BxlContext):
    actions = ctx.bxl_actions().actions

    # Each target expression expands to a list of targets, so we flatten the
    # input into a flat list of targets.
    targets = []
    for t in ctx.cli_args.targets:
        targets += t

    # The inputs() returns a file_set, which is a set of bxl.FileNode.
    # We need these as source artifacts.
    files = [ctx.fs.source(file) for file in ctx.uquery().inputs(targets)]

    report_file = actions.declare_output("all.txt")
    script = actions.write(
        "cat_files.sh",
        [
            "OUTPUT=\"$1\"",
            "shift",
            'cat "$@" > "$OUTPUT"',
        ],
        is_executable = True,
    )

    actions.run(
        cmd_args("/bin/sh", script, report_file.as_output(), files),
        category = "cat_files",
    )
    ensured = ctx.output.ensure(report_file)
    ctx.output.print(ensured)

# buck2 bxl cat_files.bxl:cat_files -- --targets thing//:main | xargs cat
cat_files = bxl_main(
    impl = _cat_files_impls,
    cli_args = {
        "targets": cli_args.list(cli_args.target_expr()),
    },
)

; buck2 bxl //thing/cat_files.bxl:cat_files -- --targets //thing:main | xargs cat
...
int main() { return 0; }
void thing() {}

; buck2 bxl //thing/cat_files.bxl:cat_files -- --targets //thing:main | xargs cat
File changed: root//thing/thing.c
...
BXL SUCCEEDED
int main() { return 0; }
void thing() {
        // yeah
}

Main thing: uqueryctx.inputs() gives you a file_set, which if you look in the docs, is just an iterable set of bxl.FileNode. These aren't artifacts, and you're just collecting their paths as strings and writing them into a file, so you can't expect Buck to track them. You need to turn them into source artifacts using ctx.fs.source(node) before you can use them as such.

Fantatic! I understood the root cause but had not found this ctx.fs.source API, thank you so much for pointing it out!
As to your other points:

I have not noticed this so far, all my inputs have always been materialized properly, but perhaps my use cases were just simple enough, or some other action had materialized it before hands.
Out of curiosity, what do you typically use allow_args for? I haven't had a use case for it (or at least I didn't realize that it would have been a convenient feature).
Of course, I just added that to "prove" that it wasn't something wrong with my inotify setup or something similar.
I suppose this answers my question for 2), doesn't it? :) This is a very good point, I hadn't considered that, thanks again for pointing that out.
Agreed, I just wanted to get something simple working first, but I do want to use ensure_multiple both for caching and parallelism.

For (2), it is used all over the prelude, like when putting together rustc argfiles. Those scripts are linked to a specific compile step, and only need rewriting when the dependency lists / rust flags change. That involves writing many flags conditionally, so it's useful for Buck to track which ones were actually used. I think it's just slightly less often useful in BXL, but you might do it for (5). The main benefit of getting dependencies written into the script itself is that you can open that file and see what buck is doing.


refactor: phpcs and composer related updates

Hey there,
I know this looks huge, but as far as I can see, there are no standards in the PHP code. I thought I might add some standards to make the PHP SDK more collaborative for everyone. If you think these changes are too much, I wouldn't mind creating multiple PRs for a smoother transition.

Thanks for taking the time to improve it! We don't mind the size of the PR. This seems to be all internal changes correct? Assuming there are no external changes then i'd consider it a simple patch / cleanup. No need to break it up.

Great job @yigitcukuren, thank you very much! Can you please go through LibExtism.php functions and try to mark as many of them private as possible?

Thank you! I have just updated visibilities to private of all the internal methods.

Thank you very much for your excellent work @yigitcukuren, I moved LibExtism to \Extism\Internal namespace to make it clear that this class is for internal use only. Let me know if you have any concerns or issues with that. Otherwise, I can go on and merge the PR

It is perfectly fine! Please go ahead.


stemcell ci test failed by step disables ipv6 in the kernel

When I running the stemcell CI test and I happened the following error:
/tmp/build/87eb1201/bosh-linux-stemcell-builder/src/github.com/cloudfoundry/stemcell-acceptance-tests/ipv4director/smoke/smoke_test.go:73
------------------------------
[Cmd Runner] 2020/11/15 09:58:53 DEBUG - Running command '/usr/local/bin/bosh -n -d stemcell-acceptance-tests --column=stdout ssh default/0 -r -c sudo netstat -lnp | grep sshd | awk '{ print $4 }''
[Cmd Runner] 2020/11/15 09:59:03 DEBUG - Stdout: <IP_ADDRESS>:22
:::22

[Cmd Runner] 2020/11/15 09:59:03 DEBUG - Stderr:
[Cmd Runner] 2020/11/15 09:59:03 DEBUG - Successful: true (0)
• Failure [9.392 seconds]
Stemcell [It] #141987897: disables ipv6 in the kernel
/tmp/build/87eb1201/bosh-linux-stemcell-builder/src/github.com/cloudfoundry/stemcell-acceptance-tests/ipv4director/smoke/smoke_test.go:92

  Expected
      <[]string | len:2, cap:2>: ["<IP_ADDRESS>:22\t", ":::22"]
  to equal
      <[]string | len:1, cap:1>: ["<IP_ADDRESS>:22"]

Is there anyone can help me?

I had a look on this issue and it seems that ipv6 is still enabled for vms running bionic stemcells on alicloud.
An ipv6 address is shown for the default nic
web/6e626a8c-35d4-467e-b297-26ade1c4467d:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:00:d9:61 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/16 brd <IP_ADDRESS> scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe00:d961/64 scope link
       valid_lft forever preferred_lft forever

and ipv6 is enabled for sshd :
web/6e626a8c-35d4-467e-b297-26ade1c4467d:~# netstat -lnp | grep sshd
tcp        0      0 <IP_ADDRESS>:22              <IP_ADDRESS>:*               LISTEN      3511/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      3511/sshd

Interestingly ipv6 is disabled on the system
web/6e626a8c-35d4-467e-b297-26ade1c4467d:~# cat /proc/sys/net/ipv6/conf/all/disable_ipv6
1

web/6e626a8c-35d4-467e-b297-26ade1c4467d:~# cat /proc/sys/net/ipv6/conf/default/disable_ipv6
1

but this may related to the corresponding settings in cat /etc/sysctl.d/60-bosh-sysctl.conf.
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

Validating the behavior on aws I see that ipv6 is disabled.  The only difference I see right now between alicloud and aws is the additional parameter ipv6.disable=1 in the file /etc/default/grub on aws.
Can someone confirm that ipv6 is disabled by default?
Should the parameter ipv6.disable=1 in the grub configuration set by default for all cloud providers?

I have tried to build a custom alicloud stemcell from the ubuntu bionic branch to test the ipv6 issue but I have problems to get a vm up and running.
Bosh complains with the following error message:
Task 375 | 12:57:52 | Preparing deployment: Preparing deployment (00:00:00)
Task 375 | 12:57:52 | Preparing deployment: Rendering templates (00:00:00)
Task 375 | 12:57:52 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 375 | 12:57:52 | Creating missing vms: dummy/413b7e4f-6863-4370-986e-16fc598f7d4e (0) (00:11:10)
                    L Error: CPI error 'Bosh::Clouds::CloudError' with message 'update i-gw8f5qx4lw605viqzm2p failed and the vm has been deleted.: change i-gw8f5qx4lw605viqzm2p to Running failed: change instance i-gw8f5qx4lw605viqzm2p to Running timeout' in 'create_vm' CPI method (CPI request ID: 'cpi-539718')
Task 375 | 13:09:02 | Error: CPI error 'Bosh::Clouds::CloudError' with message 'update i-gw8f5qx4lw605viqzm2p failed and the vm has been deleted.: change i-gw8f5qx4lw605viqzm2p to Running failed: change instance i-gw8f5qx4lw605viqzm2p to Running timeout' in 'create_vm' CPI method (CPI request ID: 'cpi-539718')

Task 375 Started  Mon Nov 30 12:57:52 UTC 2020
Task 375 Finished Mon Nov 30 13:09:02 UTC 2020
Task 375 Duration 00:11:10
Task 375 error

Updating deployment:
  Expected task '375' to succeed but state is 'error'

Exit code 1

Creating a vm from the uploaded image using the alicloud console does not work too. A message indicated that the resizing of a disk does not work, but I do not found a concrete error log.
I have build the stemcell based on the docker image "ubuntu:bionic" which itself references to the latest tag "ubuntu:bionic-20201119". The docker image introduces a new package "uuid-runtime" which is currently not part of the project.
Do I need to use a specific ubuntu bionic tag to get a working alicloud image?
Do I need to apply additional steps to change the alicloud image which are not defined inside the bosh-linux-stemcell-builder project?

Ipv6 should be disabled by default for all providers. There's even a test for it. I assume you all have seen the unit tests pass already however.
Note, unit tests are currently failing with the latest bionic image. It looks like a few packages were somewhat unexpectedly bumped (or added). They did pass as recently as Nov 23rd so if you'd rather not wait for a fix to be make, you may be able to pin back.
       -_chrony:x:109:
       -ssh:x:110:
       +uuidd:x:109:
       +_chrony:x:110:
       +ssh:x:111:

@mrosecrance thank you for the clarification. Indeed the source code looks fine and the unit tests pass. However on a fresh alicloud vm based on bionic stemcell version 0.6 the parameter ipv6.disable=1 is not set for GRUB_CMDLINE_LINUX.
worker/fe958d03-16a9-4249-afca-66612a8e8bb1:~$ cat /etc/default/grub
GRUB_CMDLINE_LINUX="vconsole.keymap=us net.ifnames=0 biosdevname=0 crashkernel=auto selinux=0 plymouth.enable=0 console=ttyS0,115200n8 earlyprintk=ttyS0 rootdelay=300  audit=1 cgroup_enable=memory swapaccount=1 console=tty0"

Having two blanks between the parameters rootdelay and audit indicates that there is another command removing  ipv6.disable=1 again.
The other problem arise when testing this behavior with a custom stemcell based on bionic version 0.6 fails due problems booting a vm from the custom alicloud image. I wonder if I have to apply additional repacking scripts to the custom stemcell after it was build?

@xiaozhu36 As discussed, can you check that https://bosh.io/jobs/director?source=github.com/cloudfoundry/bosh&version=271.2.0#p%3Ddirector.ipv6_listen is disabled or not set?

@StefanWutz Ok, thanks, I will have a try.

HI @StefanWutz I happened another error when I test again and the error as following:
resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to fetch digest for image 'bosh/main:latest': 429 Too Many Requests
does the image exist?

This error happened in the ci pipeline update-cloud-config stage. Do you have any ideas?

HI. @StefanWutz I have tested again with disable commit: https://github.com/aliyun/bosh-deployment/commit/fdc290f85c43e4183fc41eae28a30052ed375677, but it still failed.

Hi @StefanWutz I think the issue results from the PR: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/commit/b1510daa1f8b181fecc65166746190652b14fa6c
The ubuntu bionic stemcell disable the ipv6 by setting ipv6.disable=1 in the config file /boot/grub/grub.cfg. The file /boot/grub/grub.cfg automatically generated by grub-mkconfig using templates from /etc/grub.d and settings from /etc/default/grub.
Although there is a setting in the for /etc/default/grub: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/commit/b1510daa1f8b181fecc65166746190652b14fa6c#diff-8da63a2e766804496520b738c5d7c28e18411413674cce1761ad1f50b3542d2dR77 and it contains ipv6.disable=1, but it seems like did not take effect in the stemcell, because I logged in the bosh-director and the /etc/default/grub content is
$ cat /etc/default/grub
GRUB_CMDLINE_LINUX="vconsole.keymap=us net.ifnames=0 biosdevname=0 crashkernel=auto selinux=0 plymouth.enable=0 console=ttyS0,115200n8 earlyprintk=ttyS0 rootdelay=300  audit=1 cgroup_enable=memory swapaccount=1 console=tty0"

and we can find there is missing ipv6.disable=1. It also confused me.
As a check, I added the ipv6.disable=1 to the end of GRUB_CMDLINE_LINUX and then run command sudo update-grub and reboot the vm. After that, I found the ipv6 was disabled successfully and the results as following:
$ netstat -lnp | grep sshd | awk '{ print $4 }'
<IP_ADDRESS>:22

So, I think if the stemcell wants to disable ipv6, the grub config should contains ipv6.disable=1 and it also check in the spec test:  https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-bionic/master/bosh-stemcell/spec/support/stemcell_shared_examples.rb#L23 . I think if the test is valid, there will have a error when building the bionic stemcell.
I was planning to build a new bionic image to validate my ideas, but I happened another error:
Step 24/30 : ADD ${OVF_TOOL_INSTALLER} /tmp/ovftool_installer.bundle
ADD failed: stat /var/lib/docker/tmp/docker-builder590657892/VMware-ovftool-4.1.0-2459827-lin.x86_64.bundle: no such file or directory

If you can build a new custom stemcell, you can validate the idea.

isse can be closed is this is fixed within the alicloud cpi see https://github.com/cloudfoundry-incubator/bosh-alicloud-cpi-release/issues/122


former members of congress still in thing

G000569
N000186
S0001179
those three are no longer members but still have contact yaml forms ... probably should yank .
--timball

Yep, probably should be removed, want to submit a PR?

I can

I assume you meant S001179? The one above I was unable to find in the repo and seemed to have an extra 0.

I am working on it.

YAML file lost. I am working on it.


Contact Us -> CRM Airtable integration

📇 User story
As a team members of GC Design System, I want to keep record of the individuals who connect with us via the contact us form and update their progress as a client, so that we don't miss tracking active or prospective users.
✅ Definition of Done / Outcomes
Everyone who connects with us via the Contact Us page is automatically added to the CRM with appropriate tagging.
📜 Acceptance criteria

[x] Consultation with Support on the solution.
[ ] People who submit a contact us are automatically created in Airtable CRM
[ ] Tags are reflected for "active user"; usability research; mailing list, etc.
[ ] Way to prevent duplication of additions in CRM (duplication checks)
[ ] Freshdesk continues to receive tickets re: support questions, demo requests (maybe mailing list and usability research volunteer don't need to go to Freshdesk?)
[ ] Update privacy notice

📝 More info
Currently Support can activate an automation (to add the contact to the CRM with appropriate tag) for anyone who reaches out to us for a demo; mailing list or user research.  They have not done this for contact us generic questions.  They noted if we do it for generic questions; that will be everyone who reaches out to us; thus a better solution would be to create the automation directly from Contact Us website > Airtable (rather than Contact Us website > Freshdesk > trigger automation by Support > Airtable).
🚫 Out of scope
Freshdesk support flow.

Need access to Airtable api

Check with Jimmy P who can give some advice to start integrating with Airtable.

Start with investigation into the Airtable API and take it from there.

Meeting with Jimmy this morning to get some ideas

Rough implementation plan:
CDS Contact Us Form -> Submissions API
In Submissions API:

Send details to Freshdesk API

Create ticket in Freshdesk
Get response, save response ID (ticket ID)

Send details to Airtable API

Get person (check if email address exists)
If person exists

Get all information, store it
Append new information such as tags
Add support ticket number

If person does not exist

Create new person
Add appropriate information such as tags
Add support ticket number

I now have access to the CRM and also created a Service Account for the team that we can use for future integrations. The account is<EMAIL_ADDRESS>


Added Rat in a Maze Problem

Add the hacktoberfest-accepted label please
@GovindCodes @Shanborlang @utkarshkanswal @markanday05 @sushmita04

Since the question is not present in repository, u need to make a folder with question name, add the question.md file with sample input and output and add the ratInMaze.cpp file in the same folder.
you can refer to point 4(make changes locally section) in readme for reference
Hope you will make the changes soon and then generate pull request it will be merged😃

Made changes @YABHAKRITI


Run prettier 0.16.0 on the codebase

Bootstrapped!

👍 Yay! I was waiting to merge in case it conflicted with other PRs but I've merged a bunch and other seem OK so let's merge this.


Site updtes

@nitya
Standardise
Top Level - 30 days of
Sub Levels - Template of course with content items (allowing you to scaffold 30 days of)
Author Guide
Analytics Guide
Structure
Top Level Menu Student Hub - Link to Learn student hub http://aka.ms/learnstudent
Top Level Menu Curricula - Dedicated page with Images http://aka.ms/curricula
Community discussion based on Github discussions

v1 site refresh completed

landing page changes done
curricula pages templated (with existing curricula added)
discussion forum created and linked
roadmaps and playbook section added (content updates in progress)


[MDS][Part 3] Wired all UI components to the data source menu

Description
Building on top of changes in PRs #1003 & #1008, this PR wires all the UI pages to use the selected data source from the Data source menu when performing CRUD operations against the backend.
Detector creation and list pages:
https://github.com/opensearch-project/security-analytics-dashboards-plugin/assets/114732919/ff6e9b4c-4d55-417c-bf7a-509108e7fb2b
Rest of the pages:
https://github.com/opensearch-project/security-analytics-dashboards-plugin/assets/114732919/90229e75-2b0d-44d4-8e89-2c02e915ddc2
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Overall lgtm!
1). Let's ensure CI passes and
2). Let's also ensure if dataSource is invalid like serverless collection, it is handled properly. In that case, we can show the empty findings, alerts, detectors and correlations list.

Overall lgtm! 1). Let's ensure CI passes and 2). Let's also ensure if dataSource is invalid like serverless collection, it is handled properly. In that case, we can show the empty findings, alerts, detectors and correlations list. 3). I believe we are not using any other plugins API is SA, but if we are then let's make sure that integration is working as expected.

I will be working on points 2 and 3 in the next PR where I handle the cases related to unsupported DS selection

Not sure why the cypress test CI keeps getting canceled in github, all tests passed in local run


chore(travis): upgrade go version to latest

What this PR does / why we need it:
Upgrade go version from 1.8 and 1.9 to 1.10.x.
Which issue(s) this PR fixes (optional, close the issue(s) when PR gets merged):
Fixes #202
Special notes for your reviewer:
/cc @ddysher
Release note:

NONE

from 1.8 & 1.9 -> 1.9 & 1.10, but .travis only has 1.10 in this PR?

Upgrade go version from 1.8 and 1.9 to 1.9 and 1.10.

@ddysher

Upgrade go version from 1.8 and 1.9 to 1.10.x.

I have changed it.

So this requires all projects using latest golang, is it too restrict?

No. This version is only used to build and test Nirvana. Users can still build their projects with golang 1.8 and later.

/lgtm


Include USER entry in Dockerfile to pass preflight - RunAsNonRoot test

RunAsNonRoot preflight's test is not passing in multiple container images. Error is the following:
time="2024-08-02T08:42:12Z" level=debug msg="running check" check=RunAsNonRoot
time="2024-08-02T08:42:12Z" level=info msg="detected empty USER. Presumed to be running as root"
time="2024-08-02T08:42:12Z" level=info msg="USER value must be provided and be a non-root value for this check to pass"
time="2024-08-02T08:42:12Z" level=info msg="check completed" check=RunAsNonRoot result=FAILED

Including USER entry in the Dockerfile of the affected images. USER value is the same than used in cnf-app-mac-operator's Dockerfile, which is UBI-based.

I need to revisit the script permissions, because it looks like it's complaining about that:
Checking the loadbalancer pod that didn't run, one of the two containers (the one for testpmd) did not run:
$ oc get pods -n example-cnf
NAME                                                       READY   STATUS             RESTARTS         AGE
cnf-app-mac-operator-controller-manager-65fb868bff-564rk   1/1     Running            0                15m
cnf-app-mac-operator-controller-manager-65fb868bff-86gt8   1/1     Running            0                15m
loadbalancer-5d98fdff59-ltc7d                              1/2     CrashLoopBackOff   11 (2m24s ago)   10m
testpmd-lb-operator-controller-manager-67d57b6895-ffqsj    1/1     Running            0                13m
testpmd-lb-operator-controller-manager-67d57b6895-jf82r    1/1     Running            0                13m
testpmd-operator-controller-manager-65946bdf6c-rlvtp       1/1     Running            0                12m
testpmd-operator-controller-manager-65946bdf6c-z6vb5       1/1     Running            0                12m
trex-operator-controller-manager-685844b6dc-dbw6l          1/1     Running            0                11m
trex-operator-controller-manager-685844b6dc-qvjh2          1/1     Running            0                11m

$ oc get pods -n example-cnf loadbalancer-5d98fdff59-ltc7d -o json
...
        "containerStatuses": [
            {
                "containerID": "cri-o://8b7ff6e94983d2f50b22b0ee3efb6a59c53934b07a5073918dc17189f5e01c5f",
                "image": "quay.io/rh-nfv-int/testpmd-container-app-listener@sha256:dd65ff0cff2f6c3127d04b64c34852c41010121f713453b342be97cf37c72cca",
                "imageID": "quay.io/rh-nfv-int/testpmd-container-app-listener@sha256:dd65ff0cff2f6c3127d04b64c34852c41010121f713453b342be97cf37c72cca",
                "lastState": {
                    "terminated": {
                        "containerID": "cri-o://8b7ff6e94983d2f50b22b0ee3efb6a59c53934b07a5073918dc17189f5e01c5f",
                        "exitCode": 1,
                        "finishedAt": "2024-08-05T09:57:21Z",
                        "message": "+ sleep 60\n+ /usr/local/bin/webserver 8096\nconfigure webserver\ntry to start webserver in port:  8096\nquery received to check startup\nquery received to check readiness\nquery received to check liveness\nquery received to check readiness\nquery received to check liveness\nquery
 received to check readiness\n+ RUN_APP=1\n+ LOG_FILE=/var/log/testpmd/app.log\n+ n=5\n+ [[ 1 == \\0 ]]\nWaiting for log file creation (5)...\n+ '[' 5 -le 0 ']'\n+ echo 'Waiting for log file creation (5)...'\n+ '[' -e /var/log/testpmd/app.log ']'\n+ sleep 2\nquery received to check readiness\nquery received to check
 liveness\n+ n=4\n+ '[' 4 -le 0 ']'\nWaiting for log file creation (4)...\n+ echo 'Waiting for log file creation (4)...'\n+ '[' -e /var/log/testpmd/app.log ']'\n+ sleep 2\n+ n=3\n+ '[' 3 -le 0 ']'\n+ echo 'Waiting for log file creation (3)...'\nWaiting for log file creation (3)...\n+ '[' -e /var/log/testpmd/app.log
']'\n+ sleep 2\nWaiting for log file creation (2)...\n+ n=2\n+ '[' 2 -le 0 ']'\n+ echo 'Waiting for log file creation (2)...'\n+ '[' -e /var/log/testpmd/app.log ']'\n+ sleep 2\nWaiting for log file creation (1)...\n+ n=1\n+ '[' 1 -le 0 ']'\n+ echo 'Waiting for log file creation (1)...'\n+ '[' -e /var/log/testpmd/app
.log ']'\n+ sleep 2\n+ n=0\n+ '[' 0 -le 0 ']'\n+ '[' 0 -le 0 ']'\nERROR: File /var/log/testpmd/app.log not found, exit application\n+ echo 'ERROR: File /var/log/testpmd/app.log not found, exit application'\n+ exit 1\n",
                        "reason": "Error",
                        "startedAt": "2024-08-05T09:56:11Z"
                    }
                },
                "name": "listener",
                "ready": false,
                "restartCount": 5,
                "started": false,
                "state": {
                    "waiting": {
                        "message": "back-off 2m40s restarting failed container=listener pod=loadbalancer-5d98fdff59-ltc7d_example-cnf(16e2b43a-e8c3-4488-87ca-c37f88f50211)",
                        "reason": "CrashLoopBackOff"
                    }
                }
            },
            {
                "containerID": "cri-o://74ad36952bdcb535e57b1b42d99bcf454707b4068dcee5ab13f541354818fb9d",
                "image": "quay.io/rh-nfv-int/testpmd-container-app-testpmd@sha256:61f0d12177db3308e895b38571245a14e7f1cef4bddfce09fec90c14046c98cf",
                "imageID": "quay.io/rh-nfv-int/testpmd-container-app-testpmd@sha256:61f0d12177db3308e895b38571245a14e7f1cef4bddfce09fec90c14046c98cf",
                "lastState": {
                    "terminated": {
                        "containerID": "cri-o://74ad36952bdcb535e57b1b42d99bcf454707b4068dcee5ab13f541354818fb9d",
                        "exitCode": 1,
                        "finishedAt": "2024-08-05T09:52:50Z",
                        "message": "start --eth-peer=0,20:04:0f:f1:89:01 --eth-peer=1,20:04:0f:f1:89:02 --forward-mode=lb --stats-period 1 --cmdline-file /root/testpmd-runtime-cmds.txt'\n+ [[ lb == \\l\\b ]]\n+ CMD='testpmd -l 1-5,41-45  -w 0000:37:02.6  -w 0000:37:03.6  -w 0000:37:0a.6  -w 0000:37:0b.1  --socket-me
m 4096 -n 4 --proc-type auto --file-prefix pg -- --disable-rss --nb-cores=8 --rxq=1 --txq=1 --rxd=1024 --txd=1024 --auto-start --eth-peer=0,20:04:0f:f1:89:01 --eth-peer=1,20:04:0f:f1:89:02 --forward-mode=lb --stats-period 1 --cmdline-file /root/testpmd-runtime-cmds.txt --enable-lb '\n+ CMD='testpmd -l 1-5,41-45  -w
0000:37:02.6  -w 0000:37:03.6  -w 0000:37:0a.6  -w 0000:37:0b.1  --socket-mem 4096 -n 4 --proc-type auto --file-prefix pg -- --disable-rss --nb-cores=8 --rxq=1 --txq=1 --rxd=1024 --txd=1024 --auto-start --eth-peer=0,20:04:0f:f1:89:01 --eth-peer=1,20:04:0f:f1:89:02 --forward-mode=lb --stats-period 1 --cmdline-file /r
oot/testpmd-runtime-cmds.txt --enable-lb   --dut-pci 0000:37:02.6  --dut-pci 0000:37:03.6  '\n+ CMD='testpmd -l 1-5,41-45  -w 0000:37:02.6  -w 0000:37:03.6  -w 0000:37:0a.6  -w 0000:37:0b.1  --socket-mem 4096 -n 4 --proc-type auto --file-prefix pg -- --disable-rss --nb-cores=8 --rxq=1 --txq=1 --rxd=1024 --txd=1024 -
-auto-start --eth-peer=0,20:04:0f:f1:89:01 --eth-peer=1,20:04:0f:f1:89:02 --forward-mode=lb --stats-period 1 --cmdline-file /root/testpmd-runtime-cmds.txt --enable-lb   --dut-pci 0000:37:02.6  --dut-pci 0000:37:03.6   2\u003e\u00261 | tee /var/log/testpmd/app.log'\n+ echo 'testpmd -l 1-5,41-45  -w 0000:37:02.6  -w 0
000:37:03.6  -w 0000:37:0a.6  -w 0000:37:0b.1  --socket-mem 4096 -n 4 --proc-type auto --file-prefix pg -- --disable-rss --nb-cores=8 --rxq=1 --txq=1 --rxd=1024 --txd=1024 --auto-start --eth-peer=0,20:04:0f:f1:89:01 --eth-peer=1,20:04:0f:f1:89:02 --forward-mode=lb --stats-period 1 --cmdline-file /root/testpmd-runtim
e-cmds.txt --enable-lb   --dut-pci 0000:37:02.6  --dut-pci 0000:37:03.6   2\u003e\u00261 | tee /var/log/testpmd/app.log'\n/usr/local/bin/testpmd-wrapper: line 100: /usr/local/bin/testpmd-run: Permission denied\n",
                        "reason": "Error",
                        "startedAt": "2024-08-05T09:52:50Z"
                    }
                },
                "name": "loadbalancer",
                "ready": false,
                "restartCount": 6,
                "started": false,
                "state": {
                    "waiting": {
                        "message": "back-off 5m0s restarting failed container=loadbalancer pod=loadbalancer-5d98fdff59-ltc7d_example-cnf(16e2b43a-e8c3-4488-87ca-c37f88f50211)",
                        "reason": "CrashLoopBackOff"
                    }
                }
            }
        ],
...

I'm nearer to a solution, I've been able to launch testpmd properly, but it's complaining with these log messages:
EAL: Detected 80 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /tmp/dpdk/pg/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: 24000 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
EAL: Error - exiting with code: 1
  Cause: Cannot init EAL: Cannot allocate memory

Checking the docs, as I was suspecting, it's likely due to not using sudo permissions. I'll need to check how to do this for testpmd.

I was able to fix the creation of testpmd and loadbalancer pods, now the problematic one is trexserver... I'll continue next week.

Just for tracking purposes, here's a correct log from trexserver pod:
configure webserver
try to start webserver in port:  8096
INPUT: core_string: 1-3,41-43
INPUT: lb_mac_string: 40:04:0f:f1:89:01,40:04:0f:f1:89:02
pcis = 0000:37:0a.2
pcis = 0000:37:0b.0
get_pci - All PCI addresses captured: 0000:37:0a.2,0000:37:0b.0
get_src_mac - podinfo found
{'name': 'intel-numa0-net3', 'mac': '20:04:0f:f1:89:01', 'namespace': 'example-cnf'}
{'name': 'intel-numa0-net4', 'mac': '20:04:0f:f1:89:02', 'namespace': 'example-cnf'}
get_src_mac - MACs found: 20:04:0f:f1:89:01,20:04:0f:f1:89:02
destination mac obtained from lb_mac_string input - 40:04:0f:f1:89:01,40:04:0f:f1:89:02
[{'version': 2, 'interfaces': ['37:0a.2', '37:0b.0'], 'port_info': [{'dest_mac': '40:04:0f:f1:89:01', 'src_mac': '20:04:0f:f1:89:01'}, {'dest_mac': '40:04:0f:f1:89:02', 'src_mac': '20:04:0f:f1:89:02'}], 'c': 4, 'platform': {'master_thread_id': 1, 'latency_thread_id': 2, 'dual_if': [{'socket': 0, 'threads': [3, 41, 42, 43]}]}}]
2024-08-26T03:19:07+0000 Starting TRex server ...
+ CORE_COUNT=4
+ '[' -z 4 ']'
+ cd /opt/trex/trex-core/scripts
+ n=0
+ '[' 0 -ge 5 ']'
+ ./_t-rex-64 -i --no-hw-flow-stat -c 4
Starting  TRex v2.85 please wait  ...
 set driver name net_i40e_vf
 driver capability  : TCP_UDP_OFFLOAD  TSO
 set dpdk queues mode to MULTI_QUE
 Number of ports found: 2
zmq publisher at: tcp://*:4500
i40evf_execute_vf_cmd(): fail to send cmd 9
i40evf_switch_queue(): fail to switch TX 0 off
i40evf_dev_tx_queue_stop(): Failed to switch TX queue 0 off
i40evf_stop_queues(): Fail to stop queue 0
i40evf_execute_vf_cmd(): fail to send cmd 11
i40evf_add_del_all_mac_addr(): fail to execute command OP_DEL_ETHER_ADDRESS
i40evf_execute_vf_cmd(): fail to send cmd 14
i40evf_config_promisc(): fail to execute command CONFIG_PROMISCUOUS_MODE
i40evf_execute_vf_cmd(): fail to send cmd 14
i40evf_config_promisc(): fail to execute command CONFIG_PROMISCUOUS_MODE
i40evf_reset_vf(): Reset VF NIC failed
i40evf_execute_vf_cmd(): fail to send cmd 9
i40evf_switch_queue(): fail to switch TX 0 off
i40evf_dev_tx_queue_stop(): Failed to switch TX queue 0 off
i40evf_stop_queues(): Fail to stop queue 0
i40evf_execute_vf_cmd(): fail to send cmd 11
i40evf_add_del_all_mac_addr(): fail to execute command OP_DEL_ETHER_ADDRESS
i40evf_execute_vf_cmd(): fail to send cmd 14
i40evf_config_promisc(): fail to execute command CONFIG_PROMISCUOUS_MODE
i40evf_execute_vf_cmd(): fail to send cmd 14
i40evf_config_promisc(): fail to execute command CONFIG_PROMISCUOUS_MODE
i40evf_reset_vf(): Reset VF NIC failed

And here's the one we have right now in this PR:
configure webserver
try to start webserver in port:  8096
sudo: unable to send audit message: Operation not permitted
lspci: Unable to load libkmod resources: error -12
INPUT: core_string: 4-6,44-46
INPUT: lb_mac_string: 40:04:0f:f1:89:01,40:04:0f:f1:89:02
get_src_mac - podinfo found
{'name': 'intel-numa0-net3', 'mac': '20:04:0f:f1:89:01', 'namespace': 'example-cnf'}
{'name': 'intel-numa0-net4', 'mac': '20:04:0f:f1:89:02', 'namespace': 'example-cnf'}
get_src_mac - MACs found: 20:04:0f:f1:89:01,20:04:0f:f1:89:02
destination mac obtained from lb_mac_string input - 40:04:0f:f1:89:01,40:04:0f:f1:89:02
[{'version': 2, 'interfaces': ['37:02.0', '37:02.1'], 'port_info': [{'dest_mac': '40:04:0f:f1:89:01', 'src_mac': '20:04:0f:f1:89:01'}, {'dest_mac': '40:04:0f:f1:89:02', 'src_mac': '20:04:0f:f1:89:02'}], 'c': 4, 'platform': {'master_thread_id': 4, 'latency_thread_id': 5, 'dual_if': [{'socket': 0, 'threads': [6, 44, 45, 46]}]}}]
2024-08-26T12:26:29+0000 Starting TRex server ...
sudo: unable to send audit message: Operation not permitted
+ CORE_COUNT=4
+ '[' -z 4 ']'
+ cd
+ n=0
+ '[' 0 -ge 5 ']'
+ cd /opt/trex/trex-core/scripts
+ ./_t-rex-64 -i --no-hw-flow-stat -c 4
Starting  TRex v2.85 please wait  ...
ERROR in DPDK map
Could not find requested interface 37:02.0
+ n=1
+ sleep 5
+ '[' 1 -ge 5 ']'
+ cd /opt/trex/trex-core/scripts
+ ./_t-rex-64 -i --no-hw-flow-stat -c 4
Starting  TRex v2.85 please wait  ...
ERROR in DPDK map
Could not find requested interface 37:02.0
+ n=2
+ sleep 5
+ '[' 2 -ge 5 ']'
+ cd /opt/trex/trex-core/scripts
+ ./_t-rex-64 -i --no-hw-flow-stat -c 4
Starting  TRex v2.85 please wait  ...
ERROR in DPDK map
Could not find requested interface 37:02.0
+ n=3
+ sleep 5
+ '[' 3 -ge 5 ']'
+ cd /opt/trex/trex-core/scripts
+ ./_t-rex-64 -i --no-hw-flow-stat -c 4
Starting  TRex v2.85 please wait  ...
ERROR in DPDK map
Could not find requested interface 37:02.0
+ n=4
+ sleep 5
+ '[' 4 -ge 5 ']'
+ cd /opt/trex/trex-core/scripts
+ ./_t-rex-64 -i --no-hw-flow-stat -c 4
Starting  TRex v2.85 please wait  ...
ERROR in DPDK map
Could not find requested interface 37:02.0
+ n=5
+ sleep 5
+ '[' 5 -ge 5 ']'
+ echo_ts 'Failed to start TRex Server on 5 retries...'
++ date -u +%Y-%m-%dT%H:%M:%S%z
2024-08-26T12:26:56+0000 Failed to start TRex Server on 5 retries...
+ echo '2024-08-26T12:26:56+0000 Failed to start TRex Server on 5 retries...'

It's like it's not able to retrieve the PCI addresses and use them in TRex script execution, let's see why

from change #88:

RUNNING https://www.distributed-ci.io/jobs/c56d6c46-ef62-45aa-96b7-8f2606ad946c/jobStates

In fact, the example-cnf installation is passing and now the preflight RunAsNonRoot test is passing in all cases. The failure was because one node moved to SchedulingDisabled and affected the MCP, not really related to this change.
I need to run some more tests to confirm the job is stable, but for the moment I'll open the PR for reviews.

Loadbalancer mode worked and I'm currently testing direct mode here: https://www.distributed-ci.io/jobs/dece3576-964e-4cc8-90a2-2ba2d55eda52/jobStates?sort=date, and passed example-cnf installation with a successful result, no differences with the case of using root accounts. Certsuite is also passing and I'm waiting for preflight results that should be good too. So, it's definitely ready for review.

I'm testing now this change with https://github.com/dci-labs/dallas-pipelines/pull/1213 to confirm that verify_tests is working correctly, now that all preflight tests are passing

from change dci-labs/dallas-pipelines#1213:

SUCCESS https://www.distributed-ci.io/jobs/5fe90213-d656-4c19-a678-a1c88f33a969/jobStates

Job is passing, tests_to_verify is working as expected
I'll wait till tomorrow to see if there's a last-minute review, else I'll merge


Bot not loging in. Request you to be in a particular Lat and Long

having this problem. I cant log in!

Uh this is not necrobot github.


$controller failing in angular 1.5

In my unit tests, calling $controller from ngMocks is now throwing a TypeError.
Here is a stack trace:
     TypeError: ($scope,CommentsService,$stateParams,$mdDialog)=>{__cov_hHb9$o2LSNTszpge0wZGsQ.s['3']++;$scope.minimizeComments...<omitted>...;} is not a constructor
      at Object.instantiate (node_modules/angular/angular.js:4619:14)
      at node_modules/angular/angular.js:9870:28
      at node_modules/angular-mocks/angular-mocks.js:2170:12
      at injectController (test/controllers/details/comments-history-details-controller.js:42:3)
      at Context.<anonymous> (test/controllers/details/comments-history-details-controller.js:57:3)

Some other context, pretty standard:
function injectController() {
  $controller('CommentsHistoryDetailsController', {
    $scope: $scope,
    CommentsService: CommentsService,
    $stateParams: $stateParams,
    $mdDialog: $mdDialog
  });
}

My tests were working in 1.4.8, upgrading to 1.5.0 seems to be the cause of this.

Accidental creation - oops!

Thanks for reporting. There must be something else going on here, because otherwise we would have gotten lots of reports. Can you please try to reproduce the problem, starting with this plnkr: http://plnkr.co/edit/mhxA3CoPe9QcWOvInotG?p=preview

Closing due to lack of feedback,

Sorry for the delay in responding. I found the issue, I'm putting it here in case anyone else hits it.
I understand this is not a bug due to the way Angular is implemented (controller as syntax).
The problem was using ES6 arrow functions as controller functions, they do not have a prototype.


Automator: update proxy@master in istio/istio@master

Generated by Automator - 2021-04-21T02:34:27+00:00

@istio-testing: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio
6c0b2e68cc334bb86ee4390815d6269b12bc69ab
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


Add itext core metadata and tests

What does this PR do?
Add itext core metadata and tests
Code sections where the PR accesses files, network, docker or some external service
Checklist before merging

[x] I have considered including reachability metadata directly in the library or the framework (see our contributing document)
[x] I am the original author of all content provided in the pull request, and I did not copy the content from any other source (see the tests section from our contributing document)
[x] For all tests where I am not the sole author, I have added a comment that proves I may publish them under the specified license (see the tests section from our contributing document)
[x] I have properly formatted metadata files (see our formatting guide)
[x] I have added thorough tests (see the tests section from our contributing document)
[x] I have filled all places where my pull request accesses files, network, docker, or any other external service (see the section above)

Hi @olpaw ,
Somewhere in another PR I saw that it's ok to ping you if the PR is not processed for quite some time.
It's been a while since this PR created. Is there a chance you have a look at it? Thanks!

Hi @vitali-pr
Thanks for the reminder. This got lost in the sea of emails. Thanks for the reminder. I will have a look this week.

This is how we use them on java. They are coming with a (separate) font-asian jar. If it's in class path, they are available for a user to load.

So there are no class files in font-asian jar that you could use as typeReachable?
Even if so ... it should be fine to use
      "typeReachable": "com.itextpdf.io.font.Type1Font"

even for the resources in font-asian jar because that jar file only makes sense in combination with the com.itextpdf.io main jar. Thus it's save to assume com.itextpdf.io.font.Type1Font is there if we have font-asian jar on the classpath.

Thus it's save to assume com.itextpdf.io.font.Type1Font is there, if we have font-asian jar on the classpath.

But it would be even better if you know of a class that only gets reachable if one of the resources from font-asian.jar are used. This way an image using com.itextpdf.io would be smaller if none of the resources from font-asian.jar are used by the application that uses com.itextpdf.io.

Thus it's save to assume com.itextpdf.io.font.Type1Font is there

You are right, it will be com.itextpdf.io.font.CidFont then. But the problem is that we don't test these resources using CidFont class. So without a fake CidFont usage in the tests, the tests will just fail in native. Also who knows how users are supposed to use these resources, maybe they'd implement their own FontProgram.
Anyway, I think I found a compromise by just adding com.itextpdf.io.font.CjkResourceLoader type as a condition. I've added a new commit to the review.

But it would be even better if you know of a class that only gets reachable if one of the resources from font-asian.jar are used.

There are no such per cmap classes.

@vitali-pr your PR is now on master. Thanks for your contribution!


Built in warnings to help people migrate to v7?

I wonder if we could add some built in warnings to v7 that would help people migrate.

If a type parameter is a string
If jsPsych.init() is called
Is there a way to warn people who try to use window.jsPsych without messing things up for initJsPsych()?

some errors in user code may be easier to find when window.jsPsych is undefined
@bjoluc what errors do you have in mind? I might be misunderstanding something since I'm new to TypeScript, but if we assume var jsPsych = ... masks the module definition of window.jsPsych, aren't we also assuming that window.jsPsych is already defined?

That's very helpful! Thanks for that example. I agree with you that jsPsych is undefined is more clear, but I think my question is how will an end user even reach the state of calling jsPsych.something() without having already run jsPsych.init()?
To answer my own question, an old experiment of mine from 6.x is probably a good example. In that experiment I construct the timeline which includes calls to jsPsych.timelineVariable and jsPsych.NO_KEYS before the jsPsych.init() call.
With the current set up, the console would log a series of jsPsych is undefined which would be pretty clear. If we defined window.jsPsych.init() then the console would log jsPsych.timelineVariable is not a function etc and then whatever error text we defined for window.jsPsych.init().
For me I think the critical question is how useful can we make the error text and does that justify making the other console warnings less clear? The issue suggests warnings, but in this case it might be worthwhile to throw an uncaught exception. Wrongly calling jsPsych.init() is not only incorrect, but something that needs to be fixed before allowing any further execution of code. going beyond a warning and actually halting execution might make debugging even easier despite the less clear error logs for the other functions.


Error when trying to build a container for development environment

When I run this command to build a container for development environment :

docker-compose -f docker-compose.yml -f docker-compose.dev.yml build

I have this error :

pull access denied for fnagel/docker-typo3-webserver, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: Service 'webserver' failed to build : Build failed

@Oktopuce Did you run docker-compose -f docker-compose.yml before? You need to create the base image in order to use it when creating the dev image.

Try to run this command :

docker-compose -f docker-compose.yml build

and the result is :

INSTALLING REMOTE MODULE mysql
No releases available for package "pecl.php.net/mysql"
install failed
The command '/bin/sh -c install-php-extensions bcmath zip exif gd mysql mysqli pdo pdo_mysql intl soap opcache' returned a non-zero code: 1
ERROR: Service 'webserver' failed to build : Build failed

What about docker-compose build? Should be the same, but you never know.
Might be an issue with the latest commit: 2924532cc247fde164ab9a627dbb326ec5e68d18
You might want to try it when reversed.

Yes it's the same with "docker-compose build", already tried.

Damn. Looks like mysql is outdated. See https://github.com/mlocati/docker-php-extension-installer#supported-php-extensions
Please try with this line in above mentioned line:
RUN install-php-extensions bcmath zip exif gd mysqli pdo pdo_mysql intl soap opcache

Yes it works with this line.


Appstudio update test-component-pac-hktp

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-hktp-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
8 seconds
init

✅ Succeeded
25 seconds
clone-repository

✅ Succeeded
2 minutes
build-container

✅ Succeeded
37 seconds
sanity-inspect-image

✅ Succeeded
39 seconds
deprecated-base-image-check

✅ Succeeded
2 minutes
clamav-scan

✅ Succeeded
5 minutes
clair-scan

✅ Succeeded
49 seconds
sbom-json-check

✅ Succeeded
9 seconds
sanity-label-check

✅ Succeeded
10 seconds
sanity-optional-label-check

✅ Succeeded
7 seconds
show-summary

Pipelines as Code CI/test-component-pac-hktp-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
10 seconds
init

✅ Succeeded
24 seconds
clone-repository

✅ Succeeded
33 seconds
build-container

✅ Succeeded
14 seconds
deprecated-base-image-check

✅ Succeeded
13 seconds
sanity-inspect-image

✅ Succeeded
9 seconds
clair-scan

✅ Succeeded
12 seconds
sbom-json-check

✅ Succeeded
50 seconds
clamav-scan

✅ Succeeded
10 seconds
sanity-optional-label-check

✅ Succeeded
10 seconds
sanity-label-check

✅ Succeeded
8 seconds
show-summary

Pipelines as Code CI/test-component-pac-hktp-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
10 seconds
init

✅ Succeeded
24 seconds
clone-repository

✅ Succeeded
33 seconds
build-container

✅ Succeeded
14 seconds
deprecated-base-image-check

✅ Succeeded
13 seconds
sanity-inspect-image

✅ Succeeded
12 seconds
sbom-json-check

✅ Succeeded
9 seconds
clair-scan

✅ Succeeded
50 seconds
clamav-scan

✅ Succeeded
10 seconds
sanity-optional-label-check

✅ Succeeded
10 seconds
sanity-label-check

✅ Succeeded
8 seconds
show-summary


Some trivial tests on making sim_telarray available for unit tests

I changedit from draft to ready to review because I needed to see the results of the CI.

I am fine with merging. Do you want to add the DB access in a second step?
I am actually pleased by this solution. The tests take 7-10 minutes, was initially worried that it would take much longer.


optimize db connection pool

Now, xcodis use different connection pool for different dbs, this is may be horrible if we want to support 1024 dbs or more.
We may need another better connection pool management. like

every server has a connection pool not db.
use a LIFO cache to store a connection for a db. if no associated db connection, we may use random and then do SELECT.

Hi @siddontang, connection pool per server not db is needed.
Then, do you think SELECT have so many overhead?
I guess its overhead come from the round trip between xcodis and ledisdb, so if we don't wait "ok" response or can send TMPSELECT like command with pipeline together, the overhead would be minimized.

Hi @kouhate , this is the problem and you can see my commit message that I will optimize it later but have not found a better way now. :smile:
For redis, pipeline may be a good choice, but ledisdb does not support it.
For ledisdb, I think may be we can append TMPSELECT db to every command like FROM MASTER.

:+1:
TMPSELECT db may be better.

@kouhate , Now ledisdb supports XUSE command, like below:
XUSE db THEN set a 1
First we will select a db, then do the real command like before. Unlike TMPSELECT, this will change the connection db index.

@siddontang
Maybe this command is mostly used as backend protocol between xcodis and ledisdb, so the syntax is not so important.
This should reduce redundant connection pool :-)

@kouhate
I have optimized and refactored xcodis in develop branch. many changes:

Use goredis to handle RESP, I don't like origin RESP and connection handling, :-), now the code seems simple.
Optimize ledisdb db select using XSELECT db THEN command feature.
Add ledisdb test.

@siddontang
I got the following error when I tried to build latest ledisdb.

make
godep go install -tags 'linenoise leveldb rocksdb lua lmdb' ./...
# github.com/siddontang/ledisdb/server
server/client_resp.go:27: undefined: goredis.RespReader
godep: go exit status 2

This is not solved yet even after I got your goredis library.

go get github.com/siddontang/goredis

What do you think?

run make clean, and rm -rf ./Godeps/_workspace/pkg please.

Oops, works fine.
I didn't know godep enough... Thanks :-)

:smile:
I met the same problem before and make clean had no effect, so I removed Godeps pkg path and it worked.


Accept any FieldValue for terms_set

Change terms_set definition to accept any FieldValue, as the terms query already does, and as the Elasticsearch server implementation already does.
This is not a breaking change at the Typescript level, as FieldValue includes string as one of its alternatives, but the materialization of the change may result in a breaking API change in specific language (e.g. Java) compiled clients.

Looks correct, we probably got this from the documentation, which says that terms is a string array. Going from String to FieldValue breaks the java constructor for this, could we not backport this and leave it for 9? @flobernd @pquentin

Fine with not backporting this for now. For C# it's not a breaking change as we have an implicit conversion operator from string -> FieldValue.

As a side note, I already submitted https://github.com/elastic/elasticsearch/pull/113043 to correct the documentation as well.

correctly validated here, merging


Reuse compression objects

Reusing lz4/gzip reader and writer objects via sync.Pool greatly reduces CPU usage (saves allocations and creates much less garbage).

Yes,I used 1.17.0 and gzip, will  lead to gc use cpu very high.


[untested] Silence notification sound

This patch reduces the priority of the metronome start notification to IMPORTANCE_LOW which should silence the notification on metronome start.  If the notification has IMPORTANCE_DEFAULT by design, please ignore and sorry for the bother.  Caveats: I've written a couple of trivial Android apps, but I'm no expert and this patch is completely untested (not even built).

No, you're completely right - there's no reason for IMPORTANCE_DEFAULT here. I generally don't turn my notification sounds on, so I never would've noticed if this was an issue...
No promises that this'll be released to the Play Store anytime soon - there's a bunch of stuff in develop I need to fix up before the next release - but it'll get there eventually, I hope. Either way, thank you for contributing!

Cool, thanks for a nice app and merging!


[FLINK-24386] Guard JobMaster against exceptions from starting the Op…

…eratorCoordinator
What is the purpose of the change

Improve the stability of the JobMaster when OperatorCoordinator throws an Exception upon start(). Instead of restarting the entire JobMaster, we will directly restart the job instead.

Brief change log

Catch non-fatal Throwables when starting OperatorCoordinator, and fail the job instead of rethrowing the JobMaster main thread.

Verifying this change
This change added tests and can be verified as follows:

Added unit test to verify that an OperatorCoordinator exception triggers a job failure without terminating the running thread.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

Converted to draft because this change needs a lot more testing


Handlbars file checker include html/tidy, which throws error

For a handlebars file, I see the following SyntasticInfo:
Syntastic version: 3.7.0-235 (Vim 800, Darwin, GUI)
Info for filetype: html.handlebars
Global mode: active
Filetype html.handlebars is active
The current file will be checked automatically
Available checkers: handlebars/handlebars html/eslint html/jshint html/tidy html/validator html/w3
Currently enabled checkers: html/tidy handlebars/handlebars

This is causing errors to be thrown for legitimate handlebars syntax. Should html/tidy be disabled for handlebars files?

Please read :h syntastic-composite.

Thanks. I resolved this by doing so
let g:syntastic_filetype_map = { "html.handlebars": "handlebars"}

For some reason, setting that causes the syntax highlighting to be off, and syntax such as {{> partial}} is marked as invalid (the > character is highlighted in red).

Syntastic has nothing to do with syntax highlighting.

Hmm. Yeah, weird why that was the case. When closing the buffer and re-open it, it works fine.


Can't see where emails are being sent when using route with multiple recipients

Hi. First of all thanks for developing postal. Been using it for a few days now and it seems like a great piece of software.
I have a route with a bunch of e-mail address endpoints in it, and when I send a mail to that route it gets forwarded to all those endpoints as expected. But one of the target servers is rejecting my messages and I get a bounce message back every time. In the postal UI I can't find anywhere which address is causing the error because all the messages show the same From and To addresses.
Example:

As you can see one of the messages failed, but I can't see which address is at fault.
Another strange thing is that all the messages show as incoming messages when they are in fact outgoing messages.

If you go into the message in the UI, you should be able to get more detail about that particular message including the error from the receiving server and the email headers that were present.
Incoming means that the messages were received courtesy of some MX records rather than using the SMTP credential so it might be that your setup isn't quite right somehow.

The mail was forwarded through a route, so that makes sense.
But in the message details it only shows that the message was sent to the route address. In the technical details of the bounce I can see that it was sent to a gmail server, but not which address. Even in the email headers it just says To: <route address>, but not the address it was being forwarded to.

I see. The technical information in the UI for the message that you have to expand will include a support ref which should correspond to some log entries so I'd hope you can see the minute detail there?

I was able to find the failing address in the SMTP log. But I really think this should be added to the UI somewhere.

I agree, will mark as feature request


Correct pydoc to show swagger_ui_options param type SwaggerUIOptions

Previously documented as options.ConnexionOptions, which does not exist

"I'm sorry Dave, I'm afraid I can't do that"
(weak joke based on PR # 2001 :)

woohoo, first PR merged :)


iOS exception

When trying to run the Mobile.iOS application in the iOS simulator as-is, I get an exception:
Objective-C exception thrown.  Name: NSInternalInconsistencyException Reason: Application windows are expected to have a root view controller at the end of application launch
Native stack trace:
	0   CoreFoundation                      0x0000000103011f45 __exceptionPreprocess + 165
	1   libobjc.A.dylib                     0x000000010b1b7deb objc_exception_throw + 48
	2   CoreFoundation                      0x0000000103011daa +[NSException raise:format:arguments:] + 106
	3   Foundation                          0x00000001036825ee -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 198
	4   UIKit                               0x00000001059ecf15 -[UIApplication _runWithMainScene:transitionContext:completion:] + 3112
	5   UIKit                               0x00000001059e9ba3 -[UIApplication workspaceDidEndTransaction:] + 188
	6   FrontBoardServices                  0x000000010da2b784 -[FBSSerialQueue _performNext] + 192
	7   FrontBoardServices                  0x000000010da2baf2 -[FBSSerialQueue _performNextFromRunLoopSource] + 45
	8   CoreFoundation                      0x0000000102f3e011 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
	9   CoreFoundation                      0x0000000102f33f3c __CFRunLoopDoSources0 + 556
	10  CoreFoundation                      0x0000000102f333f3 __CFRunLoopRun + 867
	11  CoreFoundation                      0x0000000102f32e08 CFRunLoopRunSpecific + 488
	12  UIKit                               0x00000001059e94f5 -[UIApplication _run] + 402
	13  UIKit                               0x00000001059ee30d UIApplicationMain + 171
	14  ???                                 0x000000011c2e4f9b 0x0 +<PHONE_NUMBER>
	15  ???                                 0x000000011c2e4c85 0x0 +<PHONE_NUMBER>

Any thoughts on what's going wrong? The Mobile.Droid app runs without problems in the simulator by the way.

I don't have my Mac build host handy at the moment but I have a feeling what it could be.
In the Mobile project go to App.xaml.cs and in the constructor after InitializeComponent() add in this line
App.Current.MainPage = new ContentPage();
Let me know if that works, then I will update the project and check-in. Otherwise I will need to wait until tomorrow to get testing on it.

That worked! Thanks :-)

Can App.Current.MainPage = new ContentPage(); be removed? It causes a flicker on launch because of it. How can we update the code so it doesn't have to open this blank page first but the first stack instead?

thanks @michaeldimoudis for your workaround. I will look into it further this week and make an update. I dislike flickers as much as the next person :)


Update to Volar 2.3

Initial checklist

[x] I read the support docs
[x] I read the contributing guide
[x] I agree to follow the code of conduct
[x] I searched issues and couldn’t find anything (or linked relevant results below)
[x] If applicable, I’ve added docs and tests

Description of changes
Update to Volar 2.3.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Please upload report for BASE (main@8ce92a0). Learn more about missing BASE report.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #446   +/-   ##
=======================================
  Coverage        ?   91.51%
=======================================
  Files           ?       10
  Lines           ?     1615
  Branches        ?        0
=======================================
  Hits            ?     1478
  Misses          ?      137
  Partials        ?        0

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


MGDOBR-219 - Enforce SSL Connections from Fleet Manager to Database

Signed-off-by: Mauricio Magnani<EMAIL_ADDRESS>MGDOBR-219
PR owner checklist
Please make sure that your PR meets the following requirements:

[ ] Your code is properly formatted according to this configuration.
Run mvn clean verify -DskipTests so that imports are correctly set.
[ ] Your commit messages are clear and reference the JIRA issue e.g: "[MGDOBR-1] - $clear_explanation_of_what_you_did"
[ ] The layers in the kustomize folder have been updated accordingly.
[ ] All new functionality is tested
[ ] Pull Request title is properly formatted: MGDOBR-XYZ Subject
[ ] Pull Request contains link to the JIRA issue
[ ] Pull Request contains link to any dependent or related Pull Request
[ ] Pull Request contains description of the issue
[ ] Pull Request does not include fixes for issues other than the main ticket

PR reviewer(s) checklist

[ ] Check if code and Github action workflows have been modified and if the modification is safe
[ ] If the modification is safe, add the safe to test label

How to trigger pipelines and use the bots:

Run the end to end pipeline
Annotate the pull request with the label: safe to test. If you want to run the pipeline again, remove and add it again.

Rebase the pull request
Comment with: /rebase.

Deploy BOT
Comment with `/deploy <dev,stable> when the PR has been merged to deploy to a target environment.

@all the certificate will be available as both, environment variables and volume.


Dependency package @types/node-fetch has a new version available

We have identified a dependency on @types/node-fetch (2.5.12). A new version (3.0.3) is available now.

This task should be taken up along with #19231

Version 3.0.3 is deprecated we should stay in the 2.x.x range. @praveenkuttappan is there any way to have the automation tool to only check for new version in the 2.x.x range for this package?

Automation is filing issue based on rush update output. We don't have control at package level.


Different color for line segments

Hi, I am looking to achieve different color for line segments. It would be great feature for Chart.js. Can anybody suggest a workaround. I understand that I can color points with different colors, but can I do the same with line segments?

@harunalikadic the way to do this is to set the line color to a CanvasGradient object that transitions the colours as you want. https://github.com/bbc/chart.bands.js is an example of a library that does something similar

@etimberg thank you, this could be close to a solution, the problem is that my threshold is not fixed, as you can se on a chart mockup above, red and yellow dots are not always at the same vertical point.
At the diagram below, what I am looking to achieve (C) could be achieved with a horizontal gradient (B) but not with a vertical one (A), which, probably, is the default behaviour of the mentioned plugin.
What do you think, can this be achieved in any way with the plugin you are suggesting?

I think the plugin can only do vertical gradients, but it should be relatively straightforward to make your own plugin based on it that does horizontal gradients

Any update ? I should Implement this feature

You should be able to do this now using scriptable options. See this sample: https://github.com/chartjs/Chart.js/blob/master/samples/advanced/line-gradient.html
Note that the sample is for v3, which isn't released yet, so the code might not work with 2.9 exactly. However, this feature is supported in 2.9. We just didn't have the sample available yet. But you should be able to make it work with just some minor tweaks

For reference, sample is here now: https://www.chartjs.org/docs/latest/samples/advanced/linear-gradient.html

Ref for C: https://www.chartjs.org/docs/latest/samples/line/segments.html


Feature: Add Navigation Buttons ("Back" and "Forward")

Currently, the user has to use the browser navigation to move forward and backward between pages recently visited, that could be a bit troublesome. Adding navigation buttons (back and forward) so we can easily move back and forth between pages, especially for lists and user profile
@daccotta If this as a nice addition, I would like to work on it.
Something like:

while it does increase the usability of the site, but it comes at the cost of visual appeal and performance downgrade. most of our users are using the site on phones so they simply use the swipe gestures. So, I don't think we need one more navigation button for the homepage also.

Ok, and yes, thank you for the insights


Emmet is not working on ST3 for Mac book pro

Emmet is not working on ST3 for Mac, I have made several tries per the instructions available in the documentation and the forums and it looks like it can't work.

Me, too.
Have you find out a solution?
@mgharraph


Log zero counters

I found a cheap way to ensure that counters are still logged, even when there are no records. Previously, zero counters would not be logged (having been never incremented, they did not even exist as far as hadoop was concerned).
Actually, not quite all zero counters are logged (see test comments for details), but it should still be a big improvement in terms of diagnosing bugs etc in client code.

👍


RTL support doesn't work

Seems to be displaying the text on the left even when setting TextDirection.ltr

use messageText fix text direction but icon and leftBarIndicator still in left side


Send build log to Slack

Is it possible to send a build log to slack as well in case of error?

Hi @plutov,
I'm not sure that's a great solution as the problem may be sending messages to Slack anyway.