unsafe: add StringData, String, SliceData

Now that reflect.StringHeader and reflect.SliceHeader are officially deprecated, I think it's time to revisit adding function that satisfy the reason people have used these types. AFAICT, the reason for deprecation is that reflect.SliceHeader and reflect.StringHeader are commonly misused. As well, the types have always been documented as unstable and not to be relied upon.
We can see in Github code search that usage of these types is ubiquitous. The most common use cases I've seen are:

converting []byte to string
converting string to []byte
grabbing the Data pointer field for ffi or some other niche use
converting a slice of one type to a slice of another type

The first use case can also commonly be seen as *(*string)(unsafe.Pointer(&mySlice)), which is never actually officially documented anywhere as something that can be relied upon. Under the hood, the shape of a string is less than a slice, so this seems valid per unsafe rule (1), but this is all relying on undocumented behavior. The second use case is commonly seen as *(*[]byte)(unsafe.Pointer(&string)), which is by-default broken because the Cap field can be past the end of a page boundary (example here, in widely used code) -- this violates unsafe rule (1).
Regardless of the thought that people should never rely upon these types, people do, and they do so all over. People also rely on invalid conversions because Go has never made this easy. Part of the discussion on #19367 was about all the ways that people misuse these types today. These conversions are small tricks that can alleviate memory pressure and improve latencies and CPU usage in real programs. The use cases are real, and Go provides just enough unsafe and buggy ways of working around these problems such that now there is a large ecosystem of technically invalid code that just so happens to work.
Rather than consistently saying "don't use this", go veting somewhat, and then ducking all responsibility for buggy programs, Go should provide actual safe(ish) APIs that people can rely on in perpetuity. New functions can live in unsafe and have well documented rules around their use cases, and then Go can finally document what to do when people want this common escape hatch.
Concrete proposal
The following APIs in the unsafe package:
// StringToBytes returns s as a byte slice by performing a non-copying type conversion.
// Slices returned from this function cannot be modified.
func StringToBytes(s string) []byte

// BytesToString returns b as a string by performing a non-copying type conversion.
// The input bytes to this function cannot be modified while any string returned from
// this function is alive.
func BytesToString(b []byte) string

~func DataPointer[T ~string|~[]E, E any](t T) unsafe.Pointer~ eliminating, because realistically a person can just do &slice[0] (although a corresponding analogue does not exist for strings)
I think unsafe.Slice covers the use case for converting between slices of different types, although I'm not 100% sure what the use case of unsafe.Slice is.

Thanks, but I don't see an actual proposal here.  If you want to discuss ideas, please use golang-nuts.  The proposal process should be for a proposal.  That is, suggest some specific functions that we should introduce.  Thanks.

Sure, good point. I've edited my comment with an actual proposal of three new functions. I think unsafe.Slice covers my the fourth common use case I mentioned above, but I'm not 100% sure.

One of the main use cases of unsafe.Slice is to create a slice whose backing array is a memory buffer returned from C code or from a call such as syscall.MMap.  I agree that it can be used to (unsafely) convert from a slice of one type to a slice of a different type.

CC @mdempsky

Maybe this is too ambitious, but being able to create slices with a mmap'd backing store would be really powerful.

from a call such as syscall.MMap

Having mmap'd slices in the standard lib would be a huge win (pun intended). The concrete proposal referring to this only addresses strings -- can we get mmaps as part of that too?

@paulstuart I'm not sure exactly what you are asking, but that sounds like a different issue.  Note that code can already call syscall.Mmap and unsafe.Slice to create a slice backed by mmap'ed memory.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

FWIW, the proposed functions closely parallel the OfString and AsString functions in my unsafeslice package.
That package also provides best-effort mutation detection, since Go string variables are supposed to be immutable and Go programs may generally assume that variables passed as type string are never mutated until they are garbage-collected.

I've seen that package, and the one thing I'd not do is the mutation detection (which you gate behind the unsafe build tag). I'd expect mutation detection to be automatically done when testing / when built with -race, but not to automatically be applied in releases. I'm using the unsafe package, after all :).
I was thinking to link the package in my original writeup, since it is further evidence of the use case.

At the very least, I think it's important for the documentation for the proposed functions to explicitly call out the lifetime issues, especially for BytesToString — a string used as a map key can cause arbitrary memory corruption if it is mutated while the map is still live.

To fully replace reflect.StringHeader and reflect.SliceHeader, let's consider what they permit us to do.  They can be used to read and/or to modify the contents of a string or a slice.
For reading, we can already extract all the elements of a slice, via &s[0], len(s), and cap(s).  We can get the length of a string via len(s), but we can't get the pointer to the data.
For writing, we can create a new slice setting all the elements, via unsafe.Slice followed by a slice expression.  But we can't create a new string.
So to me that suggests, in package unsafe,
// StringData returns a pointer to the bytes of a string.
// The bytes must not be modified; doing so can cause
// the program to crash or behave unpredictably.
func StringData(string) *byte

// String constructs a string value from a pointer and a length.
// The bytes passed to String must not be modified;
// doing so can cause the program to crash or behave unpredictably.
func String(*byte, int) string

These functions should remain meaningful even if we somehow change the representation of slices or strings in the future.
The restrictions on changing the bytes are unfortunate, but the fact is that people are doing these kinds of transformations today.  Omitting these functions from the unsafe package doesn't mean that Go programs won't do them, it just means that they will do in ways that are sometimes even less safe.
It should be feasible to add a dynamic detector for any modifications to these bytes.  This could perhaps be enabled when using the race detector.  This would not be perfect but would detect egregious misuses.
The functions suggested above would then be written as
func StringToBytes(s string) []byte {
    return unsafe.Slice(unsafe.StringData(s), len(s))
}

func BytesToString(b []byte) string {
    return unsafe.String(&b[0], len(b))
}

(To be clear, the reverse is also possible: we can write String and StringData in terms of StringToBytes and BytesToString.)

It does seem like unsafe.String and unsafe.StringData match unsafe.Slice a bit better and are more fundamental operations
than providing StringToBytes and BytesToString as the primitives.
I wonder if we should add unsafe.SliceData as well (code often has to work around len 0 using &s[0]).

I'd like to suggest we reconsider the original proposal from #19367: to add new unsafe.StringHeader and unsafe.SliceHeader types, to handle the remaining advanced use cases that are supported by reflect.{String,Slice}Header but aren't covered by unsafe.Slice.
Concretely, I propose adding:
package unsafe

type StringHeader struct {
    Data *byte
    Len int
}

type SliceHeader[Elem any] struct {
    Data *Elem
    Len, Cap int
}

and allowing conversions between string and unsafe.StringHeader, and also between []Elem and unsafe.SliceHeader[Elem].
Converting an invalid unsafe.{String,Slice}Header (e.g., Len > Cap, or Data==nil and Len>0) into a normal string or slice type should fail, at least in -d=checkptr mode. I'm leaning towards making it a run-time panic (like unsafe.Slice) because the failure conditions are easy to specify/detect, but simply leaving it undefined (like unsafe.Add) seems not unreasonable too.
N.B., my original #19367 proposal also allowed conversions between *string and *unsafe.StringHeader, etc. I think we could still allow that (e.g., particularly to help users transition away from reflect.{String,Slice}Header), but I think it's less error-prone (and marginally better for escape analysis) if we encourage users to construct an unsafe.StringHeader, value convert it to string, and then store that into memory; rather than creating a *unsafe.StringHeader and individually assigning fields in memory.

After thinking about it more, I'm inclined to agree that {Slice,String}{,Data} builtin functions are the way to go. As you say, they support the same functionality. I think that will be easier on tools authors than extending conversion semantics too.

It sounds like we've converged on considering unsafe.StringData, unsafe.String, and unsafe.SliceData.
Does anyone object to those?

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

SliceData doesn't seem to have been completely specified in this thread. I assume its signature would be
SliceData([]ArbitraryType) *ArbitraryType

That is, like unsafe.Sizeof and other functions in the unsafe package it acts as sort of a pseudo-generic function that accepts any type but doesn't take any actual type parameters. And that it returns *T rather than say unsafe.Pointer.
Do I have that right?

Good point.  Your suggestion sounds right, as it roughly corresponds to the opposite of unsafe.Slice.

@mdempsky is this going to go in for 1.20? I've tried to follow the large number of associated PRs but I can't really tell what the current state is.

@cespare As far as I know, they are in.  What's missing is the docs.  I just pinged that CL.
If there is anything else missing, please let me know.  Thanks.

Both the spec and the unsafe package documentation are now done. Closing.

In practice I found out about bytes to string using the following method will result in converted values that are not as expected, to over debug my application I got this

unsafe.String(&bytes[0], byteLen)

I still miss that StringToBytes and BytesToString are not directly provided. As seen here, currently implementation of those functions requires:
import "unsafe"

func String2ByteSlice(str string) []byte {
	if str == "" {
		return nil
	}
	return unsafe.Slice(unsafe.StringData(str), len(str))
}

func ByteSlice2String(bs []byte) string {
	if len(bs) == 0 {
		return ""
	}
	return unsafe.String(unsafe.SliceData(bs), len(bs))
}

Those edge cases with empty strings/slices make current API very verbose as you have to check for those edge cases. Could those two functions be added to standard library?
On a related note, should examples like TextMarshalJSON one in json package use such conversion between strings and bytes? Or is that already automatically detected by the compiler and optimized?

Those edge cases with empty strings/slices make current API very verbose as you have to check for those edge cases.

@mitar I don't think these checks are needed. unsafe.StringData's documentation says:

For an empty string the return value is unspecified, and may be nil.

So it's perfectly safe to feed the returned pointer directly to unsafe.Slice. Same for unsafe.String(unsafe.SliceData(b), len(b)).

@database64128
With the simple one-line way, converting different zero strings might result different slice values:
package main

import "unsafe"

func main() {
	{
		var s = ""
		println(s == "") // true
		var x = unsafe.Slice(unsafe.StringData(s), len(s))
		println(x == nil) // true
	}
	{
		var s = "go"[:0]
		println(s == "") // true
		var x = unsafe.Slice(unsafe.StringData(s), len(s))
		println(x == nil) // false
	}
}

It might satisfy the needs of some cases, but not all.
The situations of converting from zero-length slices to strings are comparatively simpler.
But the verbose way makes sure that the result string will never keep the pointer of the underlying array of blank slices,
package main

import "unsafe"

func ByteSlice2String(bs []byte) string {
	if len(bs) == 0 {
		return ""
	}
	return unsafe.String(unsafe.SliceData(bs), len(bs))
}

func main() {
	var x = make([]byte, 0, 1024)
	var s = unsafe.String(unsafe.SliceData(x), len(x))
	println(s == "") // true
	println(*(*uintptr)(unsafe.Pointer(&s))) // <a non-zero addr>

	var s2 = ByteSlice2String(x)
	println(s2 == "") // true
	println(*(*uintptr)(unsafe.Pointer(&s2))) // 0
}

Why originally proposed functions StringToBytes and BytesToString are not added to unsafe package, so people stop writing various implementations with questionable correctness again and again is beyond my comprehension.

Because StringToBytes is now just unsafe.Slice(unsafe.StringData(s), len(s)) and BytesToString is just unsafe.String(unsafe.SliceData(b), len(b)) which are both very easy and there shouldn't be any implementations with questionable correctness.


DOC: sparse.linalg: Fix output in an example in the lobpcg docstring.

This fixes a failure that I get when I run the refguide check locally.
scipy.sparse.linalg
===================

scipy.sparse.linalg.lobpcg
--------------------------

File "build-install/lib/python3.10/site-packages/scipy/sparse/linalg/_eigen/lobpcg/lobpcg.py", line 260, in lobpcg
Failed example:
    A.toarray()
Expected:
    array([[  1.,   0.,   0., ...,   0.,   0.,   0.],
           [  0.,   2.,   0., ...,   0.,   0.,   0.],
           [  0.,   0.,   3., ...,   0.,   0.,   0.],
           ...,
           [  0.,   0.,   0., ...,  98.,   0.,   0.],
           [  0.,   0.,   0., ...,   0.,  99.,   0.],
           [  0.,   0.,   0., ...,   0.,   0., 100.]])
Got:
    array([[  1,   0,   0, ...,   0,   0,   0],
           [  0,   2,   0, ...,   0,   0,   0],
           [  0,   0,   3, ...,   0,   0,   0],
           ...,
           [  0,   0,   0, ...,  98,   0,   0],
           [  0,   0,   0, ...,   0,  99,   0],
           [  0,   0,   0, ...,   0,   0, 100]])

ERROR: refguide or doctests have errors

(Note that this change does not fix the failure seen recently in https://github.com/scipy/scipy/pull/14563.  @lobpcg has a pull request open that might change the svds docstring.)

Strange that this does not show in https://github.com/scipy/scipy/pull/16320 ? Or it's local only?

@lobpcg, I would have thought it would be failing elsewhere, too, but I haven't looked into why it isn't.  CI might be using a different version than I am of whatever is running the check.  Regardless, the output should be integer, not floating point, because vals is an array of integers.


lsetxattr /dev/mqueue operation not permitted when using docker userns with selinux-enabled

Output of docker version:
Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   e0e49ff
 Built:        Tue Feb 23 06:11:53 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   e0e49ff
 Built:        Tue Feb 23 06:11:53 2016
 OS/Arch:      linux/amd64

Output of docker info:
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.2
Storage Driver: btrfs
 Build Version: Btrfs v3.19.1
 Library Version: 101
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: host bridge null
Kernel Version: 4.1.12-32.el7uek.x86_64
Operating System: Oracle Linux Server 7.2
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 47.16 GiB

Output of /etc/subuid and /etc/subgid:
/etc/subuid
dockremap:100000:65536
/etc/subgid
dockremap:100000:65536

List the steps to reproduce the issue:

Run the docker daemon with --selinux-enabled --userns-remap=default
docker run -it oraclelinux:7.2 bash

Describe the results you received:
Timestamp: 2016-02-29 21:24:14.706357216 -0500 EST
Code: System error

Message: operation not permitted

Frames:

---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40

---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57

---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@240

---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35

---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26

---
5: main
Package: main
File: docker.go@18

---
6: main
Package: runtime
File: proc.go@111

---
7: goexit
Package: runtime
File: asm_amd64.s@1721
docker: Error response from daemon: Cannot start container 2b45ea01fc0f0b1ff5b2caffc5183d1379506a401577db0494f2ed3837cc9947: [9] System error: operation not permitted

Describe the results you expected:
Container should start successfully.
Provide additional info you think is important:
This is happening when we are trying to setFileLabel on /dev/mqueue in mountToRootfs on file vendor/src/github.com/opencontainers/runc/libcontainer/rootfs_linux.go
 97 func mountToRootfs(m *configs.Mount, rootfs, mountLabel string) error {
...
112         case "mqueue":
113                 if err := os.MkdirAll(dest, 0755); err != nil {
114                         return err
115                 }
116                 if err := mountPropagate(m, rootfs, mountLabel); err != nil {
117                         // older kernels do not support labeling of /dev/mqueue
118                         if err := mountPropagate(m, rootfs, ""); err != nil {
119                                 return err
120                         }
121                 }
122                 return label.SetFileLabel(dest, mountLabel)
...

Stracing the docker daemon narrowed it down to the following:
...
9311  lsetxattr("/var/lib/docker/100000.100000/btrfs/subvolumes/bf3a1a7300fbb3bab5ac89f982cdf733e20a1525c4b5005beef93fe203ec6942-init/usr/lib/systemd/system/sysinit.target.wants/dev-mqueue.mount", "security.selinux", "system_u:object_r:svirt_sandbox_file_t:s0:c232,c472", 51, 0) = 0
9621  lsetxattr("/var/lib/docker/100000.100000/btrfs/subvolumes/bf3a1a7300fbb3bab5ac89f982cdf733e20a1525c4b5005beef93fe203ec6942/dev/mqueue", "security.selinux", "system_u:object_r:svirt_sandbox_file_t:s0:c232,c472", 51, 0) = -1 EPERM (Operation not permitted)
...

Narrow it down further by tracing the linux kernel:
we seem to fail at the following function at fs/inode.c:
bool inode_owner_or_capable(const struct inode *inode)
{
        struct user_namespace *ns;

        if (uid_eq(current_fsuid(), inode->i_uid))
                return true;

        ns = current_user_ns();
        if (ns_capable(ns, CAP_FOWNER) && kuid_has_mapping(ns, inode->i_uid))
                return true;
        return false;
}

and kuid_has_mapping returns FALSE that is because inode->i_uid = 0
userns is configured and compiled in the kernel:
# grep CONFIG_USER_NS /boot/config-4.1.12-32.el7uek.x86_64
CONFIG_USER_NS=y

Hoping that maybe someone could provide some pointers here, thanks in advance.

userns is enabled in the kernel
grep CONFIG_USER_NS /boot/config-4.1.12-32.el7uek.x86_64
CONFIG_USER_NS=y

I can reproduce on a fresh rawhide installation

Thanks @runcom, I am not sure that this is really a selinux related policy issue (though I might be wrong), but here is my reason:

Do not see any related AVC denied list even from semodule -DB.
The operation not permitted (EPERM) is returned while we were doing the return label.SetFileLabel(dest, mountLabel) on the mounted /dev/mqueue at  vendor/src/github.com/opencontainers/runc/libcontainer/rootfs_linux.go:122, now looking at the kernel call stack on how we arrived at inode_owner_or_capable():

[<ffffffff812240d4>] inode_owner_or_capable+0xd4/0xe0
[<ffffffff812badc1>] selinux_inode_setxattr+0x81/0x300
[<ffffffff812b3273>] security_inode_setxattr+0x33/0x80
[<ffffffff8122ebc7>] vfs_setxattr+0x77/0xc0
[<ffffffff8122ecad>] ? setxattr+0x9d/0x1c0
[<ffffffff8122ed3e>] setxattr+0x12e/0x1c0
[<ffffffff812184ef>] ? putname+0x6f/0x80
[<ffffffff8121981e>] ? user_path_at_empty+0x6e/0xc0
[<ffffffff8120a848>] ? __sb_start_write+0x58/0x110
[<ffffffff8122ee77>] path_setxattr+0xa7/0xe0
[<ffffffff8122f081>] SyS_lsetxattr+0x11/0x20
[<ffffffff8170cfee>] system_call_fastpath+0x12/0x71

The part inside the inode_owner_or_capable() function that returns FALSE is kuid_has_mapping(ns, inode->i_uid), that is because the inode->i_uid is 0. This inode->i_uid = current_fsuid() is set during the process of mounting mqueue, and it is setting it to 0 at ipc/mqueue.c. IIUIC, now I am just wondering why during the mounting of mqueue the current_fsuid() is not our namespace uid, do we have to take care of that somehow ?

I wonder if /dev/mqueue can not handle user namespace?

Hmm..given the inode owner (0) doesn't have a mapping in the userns (which is correct), could this just be a permissions issue that when mounting /dev/mqueue we need to set ownership to the remapped root? This would be on the runc/libcontainer side of the house, but that can be tested pretty easily

Sure, although how are we handling other tmpfs mounts, they should have the same problem and be handled the same way.

Looking at runc/libcontainer, for tmpfs type, it accepts mounting with context/mountlabel, where as for the mqueue case (at least for this case, it falls back to mount without label and relatel it later - since cannot mount -t mqueue with context).

Right, but is there some way of handling of ownership of the mount point.  IE If I mount tmpfs /dev/shm inside of a container with UserNamespace, who fixes the permissions form 0:0 to DockerRoot:DockerRoot.

OK, digging it a bit further, now not sure if systemd could be in the play as well.
Here I run it on OL6 (sysvinit) system with same exact docker and linux kernel and here is the kernel call stack:
 [<ffffffff81220e5a>] inode_owner_or_capable+0xaa/0x120
 [<ffffffff812b7b13>] selinux_inode_setxattr+0xb3/0x310
 [<ffffffff81210e31>] ? complete_walk+0x51/0xe0
 [<ffffffff812b0a63>] security_inode_setxattr+0x43/0x90
 [<ffffffff8122d12a>] vfs_setxattr+0x9a/0x120
 [<ffffffff8122d28e>] setxattr+0xde/0x220
 [<ffffffff812119df>] ? putname+0x6f/0x80
 [<ffffffff81215973>] ? user_path_at_empty+0x63/0xa0
 [<ffffffff81208d49>] ? __sb_start_write+0x69/0x100
 [<ffffffff81227423>] ? mntput+0x23/0x40
 [<ffffffff8122d53f>] path_setxattr+0x8f/0xb0
 [<ffffffff81204301>] ? filp_close+0x61/0x90
 [<ffffffff8122d571>] SyS_lsetxattr+0x11/0x20
 [<ffffffff816fa26e>] system_call_fastpath+0x12/0x71

Now the important difference here vs the problematic one OL7 (systemd):

inode_owner_or_capable returns TRUE and as expected the inode->i_uid is equal to our namespace uid.
Notice also here there is a mntput in the call stack, IIUIC seems like mntput is trying to mounting it and putting it in our namespace while we do not see it in the OL7 (systemd base).

Now how could this be?

It seems that we cannot also chown after mounting /dev/mqueue, the reason is we will also be getting -EPERM, since inside inode_change_ok() we will again be checking the inode->i_uid i.e uid_eq(current_fsuid(), inode->i_uid), the call stack for this:
 inode_change_ok+0x1a1/0x200
 simple_setattr+0x1b/0x60
 notify_change+0x3ad/0x480
 chown_common.isra.12+0x159/0x270
 SyS_fchownat+0xde/0x130
 ? syscall_trace_leave+0xc6/0x120
 system_call_fastpath+0x12/0x71

Now as raised by @rhatdan earlier with regards to whether /dev/mqueue can handle userns, so I traced the mounting of tmpfs at file mm/shmem.c to see the value of inode->i_uid which is set by current_fsuid(), it returns our remapped id (i.e 100000), however that is not the case for /dev/mqueue, as it returns 0, of course both of this mounting is performed by the same process. Somehow inside the kernel, current_fsuid() is changed to our remapped id for tmpfs but not for /dev/mqueue. I am not sure however if this experiment is suffice to show that /dev/mqueue doesn't support userns.

It's a kernel bug.
During kernel_init(), init_mqueue_fs() will mount a mqueue filesystem with the default namespace 'init_ipc_ns', and this mount is with MS_KERNMOUNT.  Later when docker wants to mount another mqueue, it's using 'current->nsproxy->ipc_ns' as the namespace argument, but this also refers to the default one 'init_ipc_ns', thus sget() just gets the same superblock that's mounted with MS_KERNMOUNT and it's without any selinux label stuff, but docker wants to mount mqueue with its own mount label, so we get "SELinux: mount invalid.  Same superblock...".
This also explains what @thtanaka has found about inode->i_uid.
So docker really should creates a different nsproxy->ipc_ns when trying to mount mqueue.

seems like this has been fixed in the current runc vendored with docker, closing now

Apologies if this is the wrong place to ask, but I am trying to use kubernetes on AWS (via CloudFormation), which uses CoreOS. Currently even the latest alpha release (https://coreos.com/releases/) is running Docker 1.10.3. This issue happens to me when I try to run a service via a ReplicationController and a private docker repository (quay.io).
May 03 22:21:50 <scrubbed> kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
May 03 22:21:50 <scrubbed> kubelet-wrapper[1234]: I0503 22:21:50.629272    1234 provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 03 22:21:50 <scrubbed> kubelet-wrapper[1234]: E0503 22:21:50.945204    1234 pod_workers.go:138] Error syncing pod 6e664d92-117d-11e6-86cd-12595a8629bd, skipping: failed to "StartContainer" for "event-service" with ErrImagePull: "image pull failed for <scrubbed>, this may be because there are no credentials on this request.  details: (Error: Status 403 trying to pull repository <scrubbed>: \"{\\\"error\\\": \\\"Permission Denied\\\"}\")"
May 03 22:21:51 <scrubbed> kernel: IPv6: eth0: IPv6 duplicate address fe80::42:aff:fe02:1208 detected!
May 03 22:21:51 <scrubbed> kubelet-wrapper[1234]: E0503 22:21:51.609254    1234 pod_workers.go:138] Error syncing pod 6e664d92-117d-11e6-86cd-12595a8629bd, skipping: failed to "StartContainer" for "event-service" with ImagePullBackOff: "Back-off pulling image \"<scrubbed>\""

What are my options here? Should I disable --selinux-enabled flag? Is there any way to upgrade to Docker 1.11.0 on CoreOS?

@runcom I'm still seeing this on 1.11.2. Based on the referenced commit/issue, they weren't merged into runc upstream. Is Docker patching this into runc manually?

This should be re-opened as the issue still exists. I have created a PR to fix it at https://github.com/opencontainers/runc/pull/959. The patch is similar to what @thtanaka had proposed earlier. Thanks!

On 17.03.1-ce I am still getting that error.
Issue with enabling selinux-enabled and userns-remap at the same time.

I am seeing this on CentOS 7, 1.12.6. I cannot just tell my users to not use selinux.

Seeing this on Centos 7 with <IP_ADDRESS>-ce with selinux-enabled and userns-remap, any updates on whether this should be fixed??

This looks to depend on a fix in the runtime (Runc), but the fix has not been merged upstream https://github.com/opencontainers/runc/pull/959

So if I may resume and see if I understood correctly
For now, as temporary and easiest reversible solution is to not use --selinux-enabled until this pull is approve ?

userns-remap sets up it's own subdir within /var/lib/docker (forcing you to manually clean the non-remapped one), and forces changes to ownership & file rights which will be broken and likely most containers needing 'em until you do.
SElinux is always an enforce/permissive/disable toggle away; in permissive mode you can easily fix any issues that pop up and other than that the config is pretty much standalone, with decent ops to relabel the whole shebang as the config says it should be labeled.
So my guess would be 1. ;)
Bear in mind however SElinux is not so simple if more serious issues than file labeling arise.

As digging further on that topic/situation
and maybe to help others have the same questions of mine
I would say it's safer to move from a non userns-remap to a userns-remap than from a non SELinux to a SELinux for this reason:
If you run with SELinux disabled, all files created on your system will have no context and no label
than when you will enforce SELinux you will have to at least a restorecon and start to work with SELinux
If you active userns-remap later, you will "only" have to play with the chown command and what goes inside /etc/sub[u|g]id


[HELP]Telegram Sendrequest() not sending keyboard

The following code is not sending the keyboard, just a hello appears.
$keyboard = [
    ['7', '8', '9'],
    ['4', '5', '6'],
    ['1', '2', '3'],
         ['0']
];

$bot->sendRequest("sendMessage", array("text" => 'Hello', 'reply_markup' => array(
  'keyboard' => array($keyboard),
  'one_time_keyboard' => true,
  'resize_keyboard' => true)));

This behaves the same
$keyboard = [
    ['7', '8', '9'],
    ['4', '5', '6'],
    ['1', '2', '3'],
         ['0']
];

$bot->sendRequest("sendMessage", ["text" => 'Hello', 'reply_markup' => [
	  'keyboard' => $keyboard,
	  'one_time_keyboard' => true,
	  'resize_keyboard' => true
	  ]
  ]);

You need to pass json_encode it
$keyboard = [         ['7', '8', '9'],         ['4', '5', '6'],         ['1', '2', '3'],         ['0']     ];     $bot->sendRequest("sendMessage", ["text" => 'Hello', 'reply_markup' => json_encode([         'keyboard' => $keyboard,         'one_time_keyboard' => true,         'resize_keyboard' => true         ])     ]);
And you can use $bot->reply() for that also

Can you give an example of how to use bot reply ?
You assistance is very much appreciated.

Let's say a keyboard string contains a json tring of this format
{"keyboard":[["7","8","9"],["4","5","6"],["1","2","3"],["0"]],"one_time_keyboard":true,"resize_keyboard":true}
which is what I retrieved on json encode.
but when I try
	if(strlen($keyboard)>1)
		$bot->sendRequest("sendMessage", ["text" => $message_output, 'reply_markup' =>$keyboard ]);
	else
		$bot->reply($message_output);

It doesn't work
You implementation using the reply would be a great solution though it would eliminate that loop

It works, I only had to add {} becoz api.ai was removing them from the params, though am waiting from your implementation using the reply.
All I did was
if(strlen($keyboard)>1)
		$bot->sendRequest("sendMessage", ["text" => $message_output, 'reply_markup' => "{".$keyboard."}" ]);
	else
		$bot->reply($message_output);

Its nearly the same $bot->reply('TEXT', $extraParameters);
$keyboard = [
        ['7', '8', '9'],
        ['4', '5', '6'],
        ['1', '2', '3'],
        ['0']
    ];
$bot->reply($message_output,['reply_markup' => json_encode([
        'keyboard' => $keyboard,
        'one_time_keyboard' => true,
        'resize_keyboard' => true
        ])
    ]);

Ok, thanks let me try it.
I made u mistake and put everything in in the $message_output string.

@kenkit , close Issue if u fixed it.

ooh, my bad.
Thanks for reminding me.


Performance issue in the dragover event of the dndLists directive.

Hey, just leaving this here for the author and/or those who care because I spent a little while solving this issue while working with a very large drag and drop list.
Line 339/340 is as follows:
listNode.insertBefore(placeholderNode,
	isFirstHalf ? listItemNode : listItemNode.nextSibling);

Replacing these two lines with
if(isFirstHalf) {
	if(listItemNode.previousSibling != placeholderNode) listNode.insertBefore(placeholderNode, listItemNode);
} else {
	if(listItemNode.nextSibling != placeholderNode) listNode.insertBefore(placeholderNode, listItemNode.nextSibling);
}

dramatically cuts down on the number of DOM layouts and improves the execution time of the dragover event handler.  Without this, it's pretty hobbled at around ~60 dnd list items.
Thanks,
John Downey

amazing job @johntdowney
I had a short list to work with, but with complex moving item. Replacing those lines did the magic. Thanks

Awesome Job! Can Someone add it into the original source?

Glad I could help!  I see this "Close and comment" button, but I'm not sure of the etiquette here regarding whether or not I should close this issue since I opened it, so I'll just hit the comment button instead and assume someone with more authority can close it once the change has been applied.

@marceljuenemann please review the PR, and run the minifier.

Awesome! Thank you so much, that improve a lot than the original.

Fantastic, thanks @johntdowney ! Has a PR been submitted for this fix?

Thank you so much!
Now the drag procedure is much, much more smoothly, especially on a Mac PC with Chrome.
I would like this library to be updated more constantly, the last update is from 2017!

You can improve performance a bit more by throttling:
`
var throttle = true;
element.on('dragover', function(event) {
if (throttle) {
throttle = false;
      event = event.originalEvent || event;
      ...
      ent.addClass("dndDragover");
      event.stopPropagation();
      setTimeout(() => {
        throttle = true;
      }, 100);
    }
    return false;
  });`

I also added some throttling, using lodash.
replacing line 310
element.on('dragover', function(event) {

with
element.on('dragover', _.throttle(dragOver, 50, { leading: true, trailing: false }))
function dragOver(event){

you'll also need to get rid of the extra bracket etc. at the end of the function


Regex 2.01 wrapper

Extension of existing Regex with RegexOptions redesigned.
Plus matching test scripts.
Replacement for "Regex 2.0 wrapper" pull request which can be closed.

Thanks for your submission! We'll soon look into it! Stay tuned :-)

We haven't forgot this one, but we're currently looking into a larger effort to expose a few, much used, .NET libraries. RegEx is part of that effort and there hence is quite some overlap. A little more patience, while we make sure we cover your needs and avoid duplicate functionality.

Due to lack of activity, I'll now close the PR.
However, similar changes can be found in #118

Good afternoon Maria
I'm sorry I was quiet about the regex wrapper, but activity behind it took me off guard - I got a message from you (I mean - "you" as "Microsoft") this project would not be continued and we would get AL built-in regex support instead.
I'm glad somebody else pushed more complete set of regex libraries in, but I can't see any test libraries created for it.
Do you want me to do anything about it? Fix and complete my former set of tests working with new codeunits? Or am I missing something and the tests are there? :-)
Best regards
Jacob

As Maria has gone on a well deserved vacation, I'll try to reply... You're right that we were looking into enhancing the language, but since the timeline is unclear, we've decided to move forward with a wrapper pack as temporary fix. We hope it also covers your needs.
Don't worry about the tests. As these are pure wrappers, there's no need for elaborate testing. We did not introduce any new logic here. So there's no need for you to do anything more, unless you're missing some functionality you'd like us to add :-)


Intersections with common objects result in two schemas under allOf, which cannot be satisfied

I'm not sure if this is a duplicate or a known issue, so apologies if it is!
Zod Schema
const mySchema = z
  .object({
    object: z.object({
      one: z.string(),
    }),
  })
  .and(
    z.object({
      object: z.object({
        two: z.string(),
      }),
    }),
  );

TS Type
type MyType = z.infer<typeof mySchema>;
/*
type MyType = {
    object: {
        one: string;
    };
} & {
    object: {
        two: string;
    };
}
*/

Object
const myObject: MyType = {
    object: {
        one: "foo",
        two: "foo",
    }
}

mySchema.parse(myObject);

With a schema that has a common object, joined with .and or .intersection, you will receive two schemas under allOf which means objects that satisfy Zod & TypeScript won't satisfy the resulting JSON schema.
JSON Schema
{
  "$ref": "#/definitions/mySchema",
  "definitions": {
    "mySchema": {
      "allOf": [
        {
          "type": "object",
          "properties": {
            "object": {
              "type": "object",
              "properties": {
                "one": {
                  "type": "string"
                }
              },
              "required": [
                "one"
              ],
              "additionalProperties": false
            }
          },
          "required": [
            "object"
          ]
        },
        {
          "type": "object",
          "properties": {
            "object": {
              "type": "object",
              "properties": {
                "two": {
                  "type": "string"
                }
              },
              "required": [
                "two"
              ],
              "additionalProperties": false
            }
          },
          "required": [
            "object"
          ]
        }
      ]
    }
  },
  "$schema": "http://json-schema.org/draft-07/schema#"
}

JSON object
{
  "object": {
    "one": "one",
    "two": "two",
  }
}

Validator errors
Message: JSON does not match all schemas from 'allOf'. Invalid schema indexes: 0, 1.
Schema path: #/definitions/mySchema/allOf

  Message: Property 'one' has not been defined and the schema does not allow additional properties.
  Schema path: #/definitions/mySchema/allOf/1/properties/object/additionalProperties

  Message: Property 'two' has not been defined and the schema does not allow additional properties.
  Schema path: #/definitions/mySchema/allOf/0/properties/object/additionalProperties

No problem! But it is :)
https://github.com/StefanTerdell/zod-to-json-schema/issues/68


Dropbox API Showcase

Hi!
I need to use API in order to upload files and place them in Showcase.
Can I do this with the help of the API?
If so, what methods should be used?

You can use the API to upload files to Dropbox itself, e.g., using UploadAsync, but the Dropbox API doesn't offer a way to interact with Showcase. I'll pass this along as a feature request.


Add count-numbers extension

Description
Add the Count Numbers extension.
It has a simple and intuitive interface to count numbers - press Enter to increment the counter.
It is also customizable and supports having multiple counters, as well as changing the value, increment and modulo.
Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

Hi, technically counting down can be configured by setting increment = -1. But I can add a shortcut for it I suppose - what keyboard shortcut do you recommend?

pushed changes


Results page on Stingray is showing unexpected zero results

Hi @bespinosa01 ,
Slack thread for reference.
As we discussed, I am not getting the expected outcome on Stingray.
Details:

I'm using https://stingray-app-8z7io.ondigitalocean.app to do QA
I'm using these test profiles: https://docs.google.com/spreadsheets/d/1Gh86-zH0_nn8rfbCReXLXSL_fLfd8MvQHY_-u0xbBqM/edit#gid=0
Using the data in Row 3, I expect to see a number the same as or similar to column V, or $148.
When I put this data through CO's staging, I get a result that includes SNAP: https://benefits-calculator-staging.herokuapp.com/1d2161c5-a6c4-4533-9b56-b65e2b2bd64c/results/benefits
I also get a non-zero number when using the SNAP screener (https://www.snapscreener.com/)
Stingray displays a 0 result (https://stingray-app-8z7io.ondigitalocean.app/5c48bccd-516a-40c2-b2f5-292e40c3dda4/results/benefits).

Can you look into this? Thank you!

Was a config issue related to FPL limits set on stingray. Was using outdated numbers from 2023 :)


Nicer UPat [run_process_replay]

not ready to review yet, created pr to easily run process replay and also my local versions of ruff and mypy are different for some reason

Seems like a lot of lines, and it looks copy paste from UOp. Is it worth it?

Seems like a lot of lines, and it looks copy paste from UOp. Is it worth it?

I think i'll make it under 15-20
Also it's not really UOps copy paste
In my opinion it's 120% worth it, check out that arange example

down to +17 lines, this is still WIP, a lot of code is quick and hacky still, will make it <10 before ready to review

closed in flavor of #4991


ipxe: use mac instead of net0/mac

All the ipxe, scripts and examples are using net0/mac as mac selector variable, when you have several interfaces and net0 is not the interface being use, the mac sent as selector, could be useless/unknown as selector.
Using just mac variable, the mac of the interface being use is returned, solving the issue.

Yeah, this seems like a better default. Ours align by convention, but its just a convention. We'll test this some more.

This probably requires more consideration, I'll open an issue. OpenStack learned lessons in the past we could apply here. https://bugs.launchpad.net/ironic/+bug/1504482

Another options is use netX but is only available after the boot/dhcp event, if you run config before boot, you can see clearly that is not available. In my case this works in the same way that work mac
These is how looks my boot script:
#!ipxe
set vlan 2001

set idx:int32 0
:loop isset ${net${idx}/mac} || goto loop_done
echo Configuring VLAN ${vlan} on interface net${idx}
vcreate --tag ${vlan} net${idx}
dhcp net${idx}-${vlan} && goto boot || inc idx && goto loop
:loop_done
goto failed

:boot
echo Booting at interface ${netX/ifname}
set server http://${next-server}:8080/
chain ${server}ipxe?uuid=${uuid}&mac=${mac:hexhyp}&domain=${domain}&hostname=${hostname}&serial=${serial} || goto failed

:failed
echo ERROR: Unable to boot at any interface
prompt --key 0x02 --timeout 2000 Press Ctrl-B for the iPXE command line... && shell ||

This remembers me that could be interesting be able to customize the ipxe script being server

Admittedly, the /boot.ipxe endpoint is minimal and optional - it just provides a pretty endpoint for chaining to the /ipxe config endpoint. I'd expect lots of folks are chainloading directly in their own way like you are and that seems perfect.
Yep, its desired to let the ipxe and grub configs be templated and customized per-machine. #92

Rebased and merged as cedeb868f9be0fbd15376c9f6c2a637b85fb97d5, thanks

Rebased and merged as cedeb868f9be0fbd15376c9f6c2a637b85fb97d5, thanks!


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade" workflow

SonarQube Quality Gate Result

Result: :white_check_mark: OK
Triggered by @zxkane on pull_request

Metric
Status
Value
Error Threshold

Reliability rating
:white_check_mark: OK
1
> 1

Security rating
:white_check_mark: OK
1
> 1

Sqale rating
:white_check_mark: OK
1
> 1

Coverage
:white_check_mark: OK
83.70
< 80

Duplicated lines density
:white_check_mark: OK
8.50
> 30

Blocker violations
:white_check_mark: OK
0
> 0

Bugs
:white_check_mark: OK
0
> 0

Code smells
:white_check_mark: OK
9
> 40

Critical violations
:white_check_mark: OK
0
> 0

Major violations
:white_check_mark: OK
0
> 0

Vulnerabilities
:white_check_mark: OK
0
> 0

View on SonarQube
updated: 7/1/2024, 00:56:28 (UTC+0)

SonarQube Code Analytics
Quality Gate passed

Additional information
The following metrics might not affect the Quality Gate status but improving them will improve your project code quality.
Issues
 0 Bugs
 0 Vulnerabilities
 0 Code Smells
Coverage and Duplications
No data Coverage
No data Duplication


Feature Request: read-only transaction using "tx" action

Describe the solution you'd like
I want to send a transaction action, but I can't perform read request transactions on-chain to read a state variable.
I encountered several errors trying to modify it, as it's not supported by default. Using other alternatives like ethers returns a validation error in the frame validator stating that method, params, chainID, etc., are required.
We currently have this: https://onchainkit.xyz/frame/types#frametransactionresponse. I am looking for functionality to read contracts only, but it seems the current method needs to support that.
I would also be happy to hear your thoughts, if there is a hack to this.
Describe alternatives you've considered.
I have tried using ethers and creating a provider to read contracts, which should work fine, but Frame returns an error because I am not using the current way of sending transactions.
Using ethers:
async function getResponse(req: NextRequest): Promise<NextResponse | Response> {
  const provider = new ethers.JsonRpcProvider(OPTIMISM_RPC_URL);
  const contract = new ethers.Contract(
    OPTIMISM_CONTRACT_ADDRESS,
    SendMessageABI,
    provider
  );

  try {
    const value = await contract.value(); // Assuming `value` is the function you want to call

    return NextResponse.json({ value });
  } catch (error) {
    return new NextResponse(`Error reading contract: ${error}`, {
      status: 500,
    });
  }
}

export async function POST(req: NextRequest): Promise<Response> {
  return getResponse(req);
}

The error:
Validation failed: Required property at path /chainId Required property at path /method Required property at path /params Expected string at path /chainId Expected 'eth_sendTransaction' at path /method Expected object at path /params

I have fixed this using getFrameHtmlResponse instead,
Using the following code:
import { getFrameHtmlResponse } from "@coinbase/onchainkit/frame";
import { createPublicClient, getContract, http } from "viem";

async function getResponse(req: NextRequest): Promise<NextResponse | Response> {
  const publicClient = createPublicClient({
    chain: optimism,
    transport: http(),
  });

  const optimismContract = getContract({
    address: OPTIMISM_CONTRACT_ADDRESS,
    abi: ABI,
    client: publicClient,
  });

  const value = await optimismContract.read.value();

  console.log("Value: ", value);

  return new NextResponse(
    getFrameHtmlResponse({
      buttons: [
        {
          label: `Message from Optimism: ${value}`,
        },
      ],
      image: "https://url/read-frame.png",
    })
  );
}


Status of PyO3 support

Tracking issue: https://github.com/PyO3/pyo3/issues/4265
Currently building PyO3 against a free-threaded Python build requires setting UNSAFE_PYO3_BUILD_FREE_THREADED=1. Right now it's not worth doing that unless you're working on free-threaded support in PyO3 since it will segfault or deadlock in many situations.
Myself and @davidhewitt are going to have a standing meeting on Fridays to discuss work on PyO3 free-threaded support and I'm ramping up on the PyO3 codebase.

Closing in favor of the upstream issue.


Cranelift: b{and,or,xor}_not ops not implemented on x86_64

👋  Hey,
It looks like we don't support band_not, bor_not, bxor_not for any integer type (i8/i16/i32,i64,i128).
.clif Test Case
test run
target x86_64

function %a(i32, i32) -> i32 {
block0(v0: i32, v1: i32):
    v2 = band_not v0, v1
    return v2
}
; run: %a(1, 1) == 0

Steps to Reproduce

clif-util test ./the-above.clif

Expected Results
The test to pass
Actual Results
   Running `/home/afonso/git/wasmtime/target/debug/clif-util test ./lmao.clif`
thread 'worker #0' panicked at 'called `Option::unwrap()` on a `None` value', cranelift/codegen/src/isa/x64/lower/isle.rs:506:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 ERROR cranelift_filetests::concurrent > FAIL: panicked in worker #0: called `Option::unwrap()` on a `None` value
FAIL ./lmao.clif: panicked in worker #0: called `Option::unwrap()` on a `None` value
1 tests
Error: 1 failure

Versions and Environment
Cranelift version or commit: main
Operating system: linux
Architecture: x86_64

Can I take this task?

Sure! Thanks for taking it on!

Ok I'm gonna work on this task now


React Konva Transformer above HTML element

Hello, I'm trying to get the React Konva transformer to appear above a HTML element as i'm using it to resize the HTML element.
Here is what my component looks like
<Group>
  <html>
     <button/>
  </html>
</Group
<Transformer
   config
/>

I've tried setting the zIndex of the Html element using divProps however the HTML element is still appearing above the transformer.
Any help is much appreciated.
Thank you :)

That may be not simple to do. By design  component is just placed on top of the canvas with aboslute position.
There are some tricky workarounds to place HTML in between. You can take a look here: https://codesandbox.io/p/sandbox/react-konva-dom-between-konva-objects-dkrig1?file=%2Fsrc%2Findex.js
For now I dont' think it will be part of the library.

Ah, thanks for the response! The demo workaround is not working for me.  Is there a way to change the position property of the transformer as absolute so it is on top of the html elements?

@lavrton I also tried the demo you provided and it doesn't seem to work for me.
Particularly this part (I replaced the refs with my refs)
 setTimeout(() => {
      const child = iframeRef.current.parentElement;
      const parent = stageRef.current.container();
      parent.removeChild(child);
      stageRef.current.content.prepend(child);
    }, 50);

I get "parentElement doesn't exist", "container isn't a function" or " removeChild is not a function" errors back.
Thanks for your help. :)


Recommended Wallets Module

Description
This package includes these wallets: NEAR Wallet, MyNearWallet, Sender, Ledger, WalletConnect.
In this PR we group the setup of the mentioned wallets above.

Created new package @near-wallet-selector/recommended/wallets
Exposed setupRecommendedWallets.
Added whitelist optional property if dApp wants to include only a set of the recommended wallets.

Work Remaining

[ ] Update documentation after finishing implementation.

Closes # (issue)

Checklist:

[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings

Type of change. This type of change is the main reason for the PR.

[ ] FIX - a PR of this type patches a bug.
[ ] FEATURE - a PR of this type introduces a new feature.
[ ] BUILD - a PR of this type introduces build changes.
[ ] CI - a PR of this type introduces CI changes.
[ ] DOCS - a PR of this type introduces DOCS improvement.
[ ] STYLE - a PR of this type introduces style changes.
[ ] REFACTOR - a PR of this type introduces refactoring.
[ ] PERFORMANCE - a PR of this type introduces performance changes.
[ ] TEST - a PR of this type adds more tests.
[ ] CHORE - a PR introduces other changes than the specified above.

Breaking changes

[ ] BREAKING CHANGE - SPECIFY: _______
[ ] NO BREAKING CHANGE - this PR doesn't contain any breaking changes and it's backwards compatible

@kujtimprenkuSQA Agreed with you that using magic strings for this is not optimal; let's go with the approach in #379 and avoid magic strings.


⚠ Bofrost.be has degraded performance

In 2949f2e, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1706 ms

Resolved: Bofrost.be performance has improved in fd38fd5.


Feature/GBI-1185 - Prevent watchers to block app start

https://rsklabs.atlassian.net/browse/GBI-1185


UI for plannable spawn locations

This re-adds the option for users to choose their spawn location, re-tooled to work with the backend code.

Changed base to 4.0. Lets use this as our branch to put 4.0 stuff whilst we get through the initial period of bug fixing with new releases


support non primitive key (fix #5804)

What kind of change does this PR introduce? (check at least one)

[ ] Bugfix
[x] Feature
[ ] Code style update
[ ] Refactor
[ ] Build-related changes
[ ] Other, please describe:

Does this PR introduce a breaking change? (check one)

[ ] Yes
[x] No

If yes, please describe the impact and migration path for existing applications:
The PR fulfills these requirements:

[x] It's submitted to the dev branch for v2.x (or to a previous version branch), not the master branch
[x] When resolving a specific issue, it's referenced in the PR's title (e.g. fix #xxx[,#xxx], where "xxx" is the issue number)
[x] All tests are passing: https://github.com/vuejs/vue/blob/dev/.github/CONTRIBUTING.md#development-setup
[x] New/updated tests are included

If adding a new feature, the PR's description includes:

[ ] A convincing reason for adding this feature (to avoid wasting your time, it's best to open a suggestion issue first and wait for approval before working on it)

Other information:

Thanks for the PR - see https://github.com/vuejs/vue/issues/5804#issuecomment-305950427


better handling of glow installation

Current instalation is broken. The basic steps now are:
git clone https://github.com/charmbracelet/glow.git
cd glow
go build && mv glow $GOPATH/bin

As of 7/18/2021, I got a permission denied error when trying to :GlowInstall the tar file get's downloaded but I needed to run with sudo permissions on it and fallowed the steps in glow.lua:
sudo tar -zxf glow.tar.gz -C "$GOPATH/bin"
[ -f "glow.tar.gz" ] && rm "glow.tar.gz"

hey @Marko-Sanchez what OS are you on?


Does this setup support a mono repo?

I'm submitting a...

[ ] Bug report
[ ] Feature request
[X ] Documentation issue or request

Is this already working or could I easily repurpose this to support deploying apps under "projects" directory and remove the "src" folder.
Cheers

@bjornharvold Hello,
So far I tested it under mono-repo configuration. But definitely, we should go for multiple projects support. Do you have any public project with such configuration?

Hi @maciejtreder. I can create a simple repo and get back to you. Good work on this project :-)

Try this out. I cloned your repo. Added 2 projects. Copied you "src" into both projects. I can't build. Getting an error that "fs" cannot be found.
Anyway, that's a start. Let me know if this is enough to get this working without extra configuration.
https://github.com/bjornharvold/angular-6-monorepo-example


Prefiltering: create Arrays, don't use similar

Since a BSplineInterpolation requires an Array for coefs, it seems sensible to create an Array directly.
This fixes a bug for Images, for which similar creates an Image, but convert(Array, img) might execute a transpose depending on img's storage order. (convert gets called implicilty here.)

Yeah, I guess this was coded under the assumption that similar would actually create an Array (which, I guess, was true once, but can of course never be guaranteed for types outside of Base).
Good catch!


Webpack, include only used components

Hi, is it possible to include only the components needed and not the whole framework?

Theoretically, you should be able to include any component from src. You'll need the pug-loader installed thought.

Ok, thank you :)

Tell me if something does not work as expected
Glad I could help you 😄


Add screensaver command which activates screensaver set in Windows

Additional command as requested in https://lab02research.youtrack.cloud/issue/hassagent-192

Reference for feature cleanup: https://github.com/LAB02-Research/HASS.Agent/issues/220


Null ApplicationId in AWS::Serverless::Application fails with Internal Server Error

Description:

Null ApplicationId in AWS::Serverless::Application fails with unhandled exception
Template:
Resources:
  Application:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: null
        SemanticVersion: 1.0.0

errorMessage=Internal Server Error: Parameter validation failed:
Invalid type for parameter ApplicationId, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>

This should be a really simple bugfix if anyone from the community wants to contribute. The validation check should be added after this check.

Will be released in 1.14.0

Closing this issue as this is released in v1.14.0


Svelte mode

https://github.com/sveltejs/svelte is a "compiler that takes your declarative components and converts them into efficient JavaScript that surgically updates the DOM". The same principle can be applied here, turning declarative Swift code into imperative for better performance. It could also turn Swift into a lower-level language like C++.

@michaeleisel Do you have a use-case in mind?

Using UIKit/SwiftUI will always drop frames here and there, so performance is always a place for improvement. Also, the popularity of svelte suggests that it's helpful for a wide variety of use cases


Quick DLs

Feature request :: Possible to add little links under Gene and Gene Product to quickly copy to clipboard (or open up a FASTA window) the particular sequence? Maybe also allow the user to do that under orthologs... That's something I (and presumably others) do frequently. Would save so many steps!

Currently jbrowse has a download feature that will get the DNA sequence.  When you click on the feature there is a download glyph.  I will have to think about how to get gene product sequences

Added a download button to the protein sequence that will get the fasta from ncbi eutils.  Tried to add same functionality to nucleotide sequence as well but the entrez gene ids seemed to be mapped incorectly for genes in chlamydia.  Maps them to human genes, even though same api call for other bacteria work.  I have submitted this issue to NCBI's technical support.  For now the nucleotide sequence is still available from the jbrowse details view.  Will leave this issue open until the gene api call issue is sorted out

There is now download capabilities for both Protein and nucleotide sequences


FeatureFormView - Improve redraw

Apollo 513

Is that an issue? We'd only want evaluateExpressions() to be called after the user changes the value.

Sorry that's what I meant by "interaction" - interaction by changing the value.

Does it happen in other cases, like when just getting focus?

The only other time expressions are evaluated is when the form view loads in FormViewModel.initialEvaluation()

Taking this out of drafts now. There was some additional handling I needed to add into the footer to account for changes that was previously being handled by the entire form redrawing.
All tests are passing again (using 4160).
There is a crash in core I'm seeing when we call evaluate expressions that happens fairly ofter (but not all the time) when I run the tests:
terminating due to uncaught exception of type boost::wrapexcept<boost::lock_error>: boost: mutex lock failed in pthread_mutex_lock: Invalid argument [generic:22]

I'm not sure yet if it predates these changes.

I'm not sure yet if it predates these changes.

I've verified that it's unrelated to these changes and opened Apollo 519.

There're two other properties on FormViewModel causing full redraws, focusedElement and previouslyFocusedFields. Does redraw on focus change sound like something we should mitigate?

Does redraw on focus change sound like something we should mitigate?
Yes, but I don't think it's a high priority. Enter another, low-priority, issue and we can get to it when we have time.

It won't impact the performance as much as what you fixed in this PR. Thanks!

Yes, but I don't think it's a high priority. Enter another, low-priority, issue and we can get to it when we have time.

Done, Apollo 530


test-backend: Refactor mock.patch to assertLogs.

This PR implements the changes as suggested in issue #15331
Replaces mock.path() for logging with assert.Logs().
Removes mock.patch( ) for logging where it was not happening.
Function do_rest_call() in outgoing_webhook.py has been modified to  return response in order to efficiently catch logs in test_outgoing_webhook.py.
I think all the backend tests have been migrated to use assertLog().
Testing is done using ./tools/test-backend, and CircleCI

Thanks for working on this @m-e-l-u-h-a-n!  I posted a couple comments.  Also, is this intentionally marked as WIP?  It's good to be explicit as to what is in progress when using that.

I marked it as WIP because there are still changes to be done in some files, sorry I forgot to mention that, I would update the comment there.
Also I wanted to ask that only backend tests(contained in zerver/tests/) need to be migrated to use assertLogs as mentioned in the issue #15331  or at other places also(like in analytics/tests/test_views.py)?

@timabbott
I have updated the commit with the suggested changes and also removed WIP label can you please have a look at it.

We should migrate analytics/tests/ as well -- those are backend tests for a different component but the same reasoning for using the better API applies there.

The change in test_realm.py doesn't appear to verify the logging output; can you fix that one and also audit to make sure you didn't miss verifying what was logged in the other places you migrated?
Also, I think this would be most readable as a series of commits where the first is "logging: Remove useless mock.patch when no logging occurs." and then a separate one for the places you're migrating.  I don't think it's essential that you rebase it that way, but it might be a good opportunity to practice the art of splitting a commit with git rebase -i and git checkout -p.

@timabbott I have done the changes and splitted the commits as suggested here is some change sumarry(also included in commit message):

Added return value to do_rest_call() in outgoing_webhook.py, It is used for catching and asserting logs in test_outgoing_webhook_system.py
Logs are not asserted in test_realm.py because it would require users
to be queried using users=User.objects.filter(realm=realm) and the order
of resulting queryset varies for each run.
In test_decorators.py replacement of mock.patch is not done,  not sure
if it's worth to replace it as it's a return value of a function.
Intest_outgoing_webhook_system.py in line 541 I have not asserted the log output which is a very big error traceback as we are already asserting bot_owner_notification.content which verifies exception did occur.

@timabbott I have done the changes and splitted the commits as suggested here is some change sumarry(also included in commit message):

Added return value to do_rest_call() in outgoing_webhook.py, It is used for catching and asserting logs in test_outgoing_webhook_system.py
Logs are not asserted in test_realm.py because it would require users
to be queried using users=User.objects.filter(realm=realm) and the order
of resulting queryset varies for each run.
In test_decorators.py replacement of mock.patch is not done,  not sure
if it's worth to replace it as it's a return value of a function.
Intest_outgoing_webhook_system.py in line 541 I have not asserted the log output which is a very big error traceback as we are already asserting bot_owner_notification.content which verifies exception did occur.

Also some checks are failing due to coverage issues can you please guide me on how to correct it.

I merged the first commit, thanks @m-e-l-u-h-a-n!
The coverage test failure suggests that somehow your changes to test_push_notifications.py have resulted in some dead code in that test.  Use test-backend --coverage test_push_notifications.py to debug.

@timabbott
I have changed the the code a bit in test_push_notification.py where earlier the catching of the log output was not proper and now it works fine. You may have look at it.

Merged as cbfd6464a559217382c0e7eb1bf3fdd0d25acfbf after fixing the use of -> Any as a return type and some work on comments and the commit message.  Thanks @m-e-l-u-h-a-n!
Please study the improved commit message readability and replicate it in your future PRs.
I think there's a bit more work we can do before closing out the original issue; test_sessions.py and test_markdown.py have matches in git grep patch | grep '\(logger\|logging\)', as well as several more.  Want to do another PR with another patch?

Merged as cbfd646 after fixing the use of -> Any as a return type and some work on comments and the commit message. Thanks @m-e-l-u-h-a-n!
Please study the improved commit message readability and replicate it in your future PRs.
I think there's a bit more work we can do before closing out the original issue; test_sessions.py and test_markdown.py have matches in git grep patch | grep '\(logger\|logging\)', as well as several more. Want to do another PR with another patch?

Actually I got busy with some examinations in my college, I would surely try to resolve #15331 as soon as I get time.


Update virtual-wan-global-transit-network-architecture.md

@cherylmc

@skishen525 : Thanks for your contribution! The author, @cherylmc, has been notified to review your proposed change.

#sign-off


remove GKE-exclusive callout for Operator

With EKS support being finalized for the Operator, this removes the callouts that stated that the Operator is only supported on GKE.

This change is


[WIP] vm tracing

Extension of https://github.com/ethereum/py-evm/pull/1515
What was wrong?
Need the ability to produce EVM trace output.
https://github.com/ethereum/go-ethereum/wiki/Tracing:-Introduction
How was it fixed?
Took what was started in #1515 and reworked to be more inline with the architecture of this codebase.
Cute Animal Picture

closing as #1515 now contains these changes.


cargo: be explicit about Rust 2018 compatible versions

This should hopefully will make it clearer when the version
requirements of dependencies can be bumped.
Closes #70, closes #80.

Codecov Report

Merging #82 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #82   +/-   ##
=======================================
  Coverage   92.24%   92.24%
=======================================
  Files           5        5
  Lines         387      387
=======================================
  Hits          357      357
  Misses         30       30

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 558b00c...c73f1e3. Read the comment docs.


error in swagger definition

There is an error in the swagger definition that causes problems in certain languages like java. The return value of api/jobs is not just a an array of schema.Job but it is an object that has the property jobs, which is an array of schema.Job.
Instead of
"200": {
    "description": "Array of matching jobs",
    "schema": {
        "type": "array",
        "items": {
            "$ref": "#/definitions/schema.Job"
        }
    }
},

the swagger file should look like this
"200": {
    "description": "Array of matching jobs",
    "schema": {
        "type": "object",
        "properties": {
            "jobs": {
                "type": "array",
                "items": {
                    "$ref": "#/definitions/schema.Job"
                }
            }
        }
    }
},

Since the swagger file is automatically generated, the fix should be somewhere else.

Thanks for your report!
Can you please test if the recent changes fixes your issues.

I tested version 1fc090965d70ca3e78fabbd557f13f4157499256. Python works fine. Java however has some problems. The class AllOfschemaJobMetaJobState which inherits from the enum SchemaJobState. Enums are not meant to be extended.

I made a small change. The enum was from a swagger comment. In the code there is no inheritance (there is no such thing in Golang) and also no extension of the enum.
Can you try if this fixes your problem.

Ok, I cannot get it to work. The problem is the
"allOf": [
                        {
                            "$ref": "#/definitions/schema.JobState"
                        }
                    ],

part. This creates the dependency and java cannot handle that.

That part is generated by swagger.
Did you try how the swagger.json should be so that it works? I mean you can just edit the swagger file and check if it works.
For example maybe the x-enum-varnames is the problem.
I do not know the specification for the API spec. Maybe there is an alternative to allOf?

I searched around in the internet and did not found a simple solution.
We use this generator: https://github.com/swaggo/swag
I guess the main issue is the allOf .  I think this is a bug in swag . The datatype is correctly detected as enum but it handles it in composable data structures as a collection of objects.
I fear there is no simple solution to this. I can open a ticket at swag and ask what they think of this.

What you can try is to replace all references to jobState with the following json:
        "description": "Final state of job",
        "type": "string",
        "enum": [
            "completed",
            "failed",
            "cancelled",
            "stopped",
            "out_of_memory",
            "timeout"
        ]

And see if this fixes it. In my opinion this is what should be generated.

It seems there is no easy fix.
The missing enum type in golang cannot be easily handled by some code generators.
So I will close this ticket for now.


Travis CI + Badge

I saw there is a Travis CI config but it doesn't seem to be working. I've never really used Travis CI and couldn't figure out if it was a config error or if it just isn't turned on.
Once it's working, it'd be nice to have a badge and include Python 3 in the tests.

OK done!

Awesome! Thanks.


Adjust the way {pipeline,task}run metrics are surfaced.

It's been bugging me that NewController methods for the TaskRun and PipelineRun controllers launch a go routine to harvest metrics, and it occurred to me that there might be a better way.
Borrowing from the CloudEvent client's use of hand-crafted injection logic: https://github.com/tektoncd/pipeline/blob/7297c48d26da98552be4ee3c50d94a130bd8e79d/pkg/reconciler/events/cloudevent/cloudeventclient.go#L29
This change takes a similar approach, creating pkg/{task,pipeline}runmetrics packages, which surface their *Recorder via .Get(ctx) methods.  Rather than the controller spinning off go routine, this piggybacks on informer injection to "Start()" that process after the dependent informers have been started.
/kind cleanup
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[x] Docs included if any changes are user facing
[x] Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes
NONE

/retest
the alpha tests are very flaky 😞

/lgtm

Hmm, the race detector blew up, gonna take a quick look at where

    testing.go:1092: race detected during execution of test
FAIL
FAIL	github.com/tektoncd/pipeline/pkg/reconciler/taskrun	6.991s

I don't see the usual stack trace in the blended -v output, so I'm going to try and repro this locally with -race -count=10 🤞

Nothing at 10, going for 100 while I walk the dog :)

Well all I got was:
race: limit on 8128 simultaneously alive goroutines is exceeded, dying

Let's see if prow can repro it.
/retest

I guess not.


[Task]: Improve message in ValueError raised by Deduplicate PTransform

What needs to happen?
Currently, ValueError raised by Deduplicate PTrasform contains grammatical mistakes.
      raise ValueError(
          'Deduplicate requires at lease provide either'
          'processing_time_duration or event_time_duration.')

Namely:

lease -> least
provide -> providing
no space between either and processing_time_duration

Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

Fixed by #26665 - thanks!


We need to write a program to retry REDCap disconnected files/subjects

urllib3.exceptions.MaxRetryError
 Parsing redcap-import-16244-89.err
 CP00102_daily_activity_and_saliva_sample_collection.csv
Traceback (most recent call last):
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/data/predict1/miniconda3/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x2b9150649cf0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='redcap.partners.org', port=443): Max retries exceeded with url: /redcap/api/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b9150649cf0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/predict1/utility/rpms_to_redcap.py", line 387, in <module>
    r = requests.post('https://redcap.partners.org/redcap/api/', data= fields)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='redcap.partners.org', port=443): Max retries exceeded with url: /redcap/api/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b9150649cf0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

 CP00102_missing_data.csv
Traceback (most recent call last):
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/data/predict1/miniconda3/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/data/predict1/miniconda3/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/data/predict1/miniconda3/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/data/predict1/miniconda3/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/data/predict1/miniconda3/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/data/predict1/miniconda3/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/predict1/utility/rpms_to_redcap.py", line 387, in <module>
    r = requests.post('https://redcap.partners.org/redcap/api/', data= fields)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

 CP00102_family_interview_for_genetic_studies_figs.csv.flat
Traceback (most recent call last):
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/data/predict1/miniconda3/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x2b2fbe1dcc10>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='redcap.partners.org', port=443): Max retries exceeded with url: /redcap/api/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b2fbe1dcc10>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/predict1/utility/rpms_to_redcap.py", line 387, in <module>
    r = requests.post('https://redcap.partners.org/redcap/api/', data= fields)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/data/predict1/miniconda3/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='redcap.partners.org', port=443): Max retries exceeded with url: /redcap/api/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b2fbe1dcc10>: Failed to establish a new connection: [Errno -2] Name or service not known'))

The easiest solution may be to:

grep bsub/ directory for urllib3.exceptions.MaxRetryError string
find the *err files that has it, obtain subject ID
retry all CSV files of that subject if grepping and finding only that CSV file is impossible
This program should run after all subjects have finished

grep -B34 urllib3.exceptions.MaxRetryError /tmp/errors.txt | grep .csv

This gives only csv file names.

cd /data/predict1/utility/bsub/
../parse_redcap_error.py "*err" | grep -B34 urllib3.exceptions.MaxRetryError | grep .csv

This gives only csv file names.

FORCE=1
for form in failed.csv
do
  pushd .
  cd /data/predict1/data_from_nda/Prescient/PHOENIX/PROTECTED/Prescient{site}/raw/{subject}/surveys/
  /data/predict1/utility/rpms_to_redcap.py $form $redcap_dict $API_TOKEN $FORCE
  popd
done

Another idea is catch this error within request and set its hash to zero.

Special characters are making it hard to streamline:

^[[0;31m ME01326_speech_sampling_run_sheet.csv ^[[0m

As an improvement, we removed special characters around $form:
https://github.com/AMP-SCZ/utility/blob/4f55fb8284b626989c8390082d074bb1e186271a/rpms_to_redcap.lsf#L46

We shall actually have to run the whole pipeline for those selected cases. So two ideas:
(i)

set upload=1 for those subjects in date_shift database
and set upload=1 for those forms in {subject}_hashes.csv

(ii)
or, write an rpms_records.txt and rerun the whole RPMS pipeline for upload, clean, down shift

(ii) seems like a big task. So we should just follow (i) and wait for the next run.

We decided to simply retry upload after 180 seconds interval.
Reference:
https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-requests-request
https://requests.readthedocs.io/en/latest/user/advanced/#transport-adapters
https://requests.readthedocs.io/en/latest/api/

If this scheme is successful, we should deploy this on import_records_all.py too.


lambda functions in circuits DSL are binary functions

A very minor issue, but it caused me some time to realize.
In the circuit DSL the boolean binary functions are defined this way:
semantics = {
	'and' : lambda bool1, bool2: bool1 and bool2,
	'or'  : lambda bool1, bool2: bool1 or bool2,
	'xor' : lambda bool1, bool2: bool1^bool2,
	'not' : lambda bool: not bool
}

instead of in this way:
semantics = {
	'and' : lambda bool1: lambda bool2: bool1 and bool2,
	'or'  : lambda bool1: lambda bool2: bool1 or bool2,
	'xor' : lambda bool1: lambda bool2: bool1^bool2,
	'not' : lambda bool: not bool
}

This is actually also mentioned in the readme ("One important point: when defining the semantics of functions, Python must be able to evaluate them one argument at a time"). Being evaluated two arguments at a time causes a 'silenced' error during program evaluation (the program is quietly evaluated to None).

Thanks, approved!


DMT-54 Add semi circle functionality

Related issue

Closes #49

Proposed changes

Add semi-circle option to popout settings
Add functionality to visualize semi-circle and correspondingly adjust the location to center it
Update logic for center-text to properly adjust according to circle type
Refactor center-text into it's own class
Fix labels offsets for semi-circle when placed outside circle

I am wondering if the text is slightly more to the right than being in the center? I might have been staring at it for too long, but when I come back to look at it, it looks a bit off-center.

Other than this very small thing that I do not think anyone will notice, it looks great!

Which text, labels-related or center-text related? To be honest, I can't spot the difference. Also, there is no change between the older versions as far as the alignment on the X-axis. The correlated transformations introduced were only on the Y axis. Also, what are the different branches you are comparing with, because this doesn't exist on dev obviously?! :sweat_smile:

Which text, labels-related or center-text related? To be honest, I can't spot the difference. Also, there is no change between the older versions as far as the alignment on the X-axis. The correlated transformations introduced were only on the Y axis. Also, what are the different branches you are comparing with, because this doesn't exist on dev obviously?! 😅

Oh, I probably should have been a lot more specific! My bad! I am thinking about the center text, and I think I might just have stared at it for too long. It definitively does not exist on dev, it was just a quick change I did in my paint to see if I saw a difference or not, but I think, now that I am looking at it, that it looks good in your version! Ignore this 😆

Which text, labels-related or center-text related? To be honest, I can't spot the difference. Also, there is no change between the older versions as far as the alignment on the X-axis. The correlated transformations introduced were only on the Y axis. Also, what are the different branches you are comparing with, because this doesn't exist on dev obviously?! sweat_smile

Oh, I probably should have been a lot more specific! My bad! I am thinking about the center text, and I think I might just have stared at it for too long. It definitively does not exist on dev, it was just a quick change I did in my paint to see if I saw a difference or not, but I think, now that I am looking at it, that it looks good in your version! Ignore this laughing

Alright then! I thought that you might have compared it with the other branch that is up that has the older implementation. However, those two wouldn't match either, so I was wondering about the comparison between the old and the new version in your image! :stuck_out_tongue:


test: fix test_vm_autoscaling.py which does not work due to userdata improvement

Description
This PR...

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

@weizhouapache a [LL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [LL]: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 6171

@blueorangutan LLtest

@weizhouapache a [LL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[LL]Trillian test result (tid-6730)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43914 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7921-t6730-kvm-centos7.zip

@blueorangutan test

@weizhouapache a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[SF] Trillian test result (tid-7604)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44795 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7921-t7604-kvm-centos7.zip
Smoke tests completed. 111 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_02_trigger_shutdown
Failure
331.85
test_safe_shutdown.py

test_08_upgrade_kubernetes_ha_cluster
Failure
702.53
test_kubernetes_clusters.py


[FEATURE] improve --episode / -ep.

This is a breaking change for UI.
As of right now the episode command option takes values such as 1:3 where 1 is the season and 3 is the episode. Those values were meant to be the other way round. 💀 (episode first then season).
This issue will swap them around and add the ability to just pass episode like so (season will default to 1):
mov-cli golden time --episode 1

wait what... It's already episode first. Am I going crazy? Do I need my pills again?

added in v4.1


Fix more Codacy issues

Some unused variable assignments might be bugs and require a closer look. Therefore I did not "fix" all such Codacy issues.

This pull request fixes 26 Codacy issues.


Handlers for different menu side and custom side menu frame were added.

I have added support for different menu site, and situation when side menu has custom frame (changed by developer)

Hi Rommanski,
Thanks to your commit, your code is so appropriately!


Wrong indentation guides

VSCode Version: 1.39.1
OS Version: Linux x64 5.2.0-3-amd64

As you can see, it highlights the previous matching bracket's indentation guides.

Duplicate of #49342


Update README.md file:

Add Maven configuration section (update Pom.xml).
Add numbering to headers - aligned with the table of contents.

Table of contents links are broken.

Which one? I tested all of them I think, the links are x.x.x.- formatted on purpose.

https://github.com/aerospike/java-object-mapper/tree/readme-maven-configuration

https://github.com/aerospike/java-object-mapper/tree/readme-maven-configuration

You are right, the dots doesn't work in urls only on Intelij readme.md preview.
Thanks, closed.


[core] ServiceRequestError: Bearer token authentication is not permitted for non-TLS protected (non-https) URLs.

Package Name: azure-mgmt-subscription
Package Version: 1.0.0
Operating System: Ubuntu 20.04.2
Python Version: 3.8

Describe the bug
Can't retrieve details from subscription id
To Reproduce
Steps to reproduce the behavior:

Create service principal:

az ad sp create-for-rbac --name "sp_name" --role "Reader" --sdk-auth true

Run this script:

from azure.identity import ClientSecretCredential
from azure.mgmt.subscription import SubscriptionClient

credential = ClientSecretCredential(
    tenant_id,
    client_id,  # Service Principal
    client_secret,
)
subscription_client = SubscriptionClient(credential, subscription_id)

info = subscription_client.subscriptions.get(subscription_id)

Expected behavior
It should successfully return the subscription object
Output
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/mgmt/subscription/operations/_subscriptions_operations.py", line 160, in get
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 211, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send
    response = self.next.send(request)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/mgmt/core/policies/_base.py", line 47, in send
    response = self.next.send(request)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send
    response = self.next.send(request)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send
    response = self.next.send(request)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send
    response = self.next.send(request)
  [Previous line repeated 1 more time]
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/policies/_redirect.py", line 158, in send
    response = self.next.send(request)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py", line 467, in send
    raise err
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py", line 445, in send
    response = self.next.send(request)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py", line 117, in send
    self.on_request(request)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py", line 91, in on_request
    self._enforce_https(request)
  File "/path/to/project/.venv/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py", line 54, in _enforce_https
    raise ServiceRequestError(
azure.core.exceptions.ServiceRequestError: Bearer token authentication is not permitted for non-TLS protected (non-https) URLs.

The signature of SubscriptionClient is SubscriptionClient(credential, base_url=None, **kwargs)  (documented here). When you write SubscriptionClient(credential, subscription_id) you get base_url=subscription_id and the client sends malformed requests. Try subscription_client = SubscriptionClient(credential) instead.

My bad! It works now! Thanks, and sorry for opening an issue about this.


Deploy docs

The documentation is not deployed automatically.

Regarding @sloede's suggestion https://github.com/trixi-framework/TrixiBottomTopography.jl/issues/15#issuecomment-1344098649.

Just to be clear, you should also try to fix the underlying issues why Documenter produced those warnings 😆

According to failed Actions, the problem is that gh-pages does not have permission to be pushed .
Run git push --force origin gh-pages-new:gh-pages
remote: Permission to trixi-framework/TrixiBottomTopography.jl.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/trixi-framework/TrixiBottomTopography.jl/': The requested URL returned error: [4](https://github.com/trixi-framework/TrixiBottomTopography.jl/actions/runs/3636685380/jobs/6161098254#step:4:5)03
Error: Process completed with exit code 128.

The documenter.yml seems not to produce any errors.

Can you still please try to fix the errors as well: https://github.com/trixi-framework/TrixiBottomTopography.jl/actions/runs/3656445536/jobs/6178898305#step:5:12

Ok, sorry I did not notice this error. The latest commit should fix it.

The deployment seems to fail.
┌ Info: Deployment criteria for deploying preview build from GitHub Actions:
│ - ✘ ENV["GITHUB_REPOSITORY"]="trixi-framework/TrixiBottomTopography.jl" occurs in repo="https://trixi-framework.github.io/TrixiBottomTopography.jl"
│ - ✔ ENV["GITHUB_REF"] corresponds to a PR number
│ - ✔ PR originates from the same repository
│ - ✔ `push_preview` keyword argument to deploydocs is `true`
│ - ✔ ENV["GITHUB_ACTOR"] exists and is non-empty
│ - ✔ ENV["DOCUMENTER_KEY"] exists and is non-empty
└ Deploying: ✘

The latest commit hopefully fixes this

It seems to work now.
https://trixi-framework.github.io/TrixiBottomTopography.jl/previews/PR18/
contains all the desired changes.


Update README with instructions to pull image

This is so we can capture instances where user has not already got the image downloaded.

thanks for the fix!