proposal: cmd/vet: warn about variables/values of type reflect.{Slice,String}Header

Here are three ways I commonly see developers misuse reflect.SliceHeader / reflect.StringHeader:
package p

import (
	"reflect"
	"unsafe"
)

// Explicitly allocating a variable of type reflect.SliceHeader.
func a(p *byte, n int) []byte {
	var sh reflect.SliceHeader
	sh.Data = uintptr(unsafe.Pointer(p))
	sh.Len = n
	sh.Cap = n
	return *(*[]byte)(unsafe.Pointer(&sh))
}

// Implicitly allocating a variable of type reflect.SliceHeader.
func b(p *byte, n int) []byte {
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
		Data: uintptr(unsafe.Pointer(p)),
		Len: n,
		Cap: n,
	}))
}

// Use reflect.SliceHeader as a composite literal value.
func c(p *byte, n int) []byte {
	var res []byte
	*(*reflect.SliceHeader)(unsafe.Pointer(&res)) = reflect.SliceHeader{
		Data: uintptr(unsafe.Pointer(p)),
		Len: n,
		Cap: n,
	}
	return res
}

All three of these can lead to memory corruption, as escape analysis analyzes these as "p does not escape" (rather than "p leaks to result"), but none of them are currently diagnosed by either cmd/vet or checkptr.
I propose cmd/vet should look for variables and expressions of type reflect.SliceHeader or reflect.StringHeader (as opposed to *reflect.SliceHeader or *reflect.StringHeader) and warn about them. There should be no false positives for this, and it's consistent with the unsafeptr check that cmd/vet already has.
--
An alternative approach would be to outright disallow reflect.SliceHeader and reflect.StringHeader to be used except as pointed-to types. (Notably, this would be similar to the //go:cgo_incomplete directive suggested in #40507 to prevent misuse of incomplete C struct types.)
I think this would be consistent with Go 1 compat. Go 1 compat is about guaranteeing that programs continue to build and run correctly in the future. I think if we're okay with these programs not running correctly today, we should be okay with them not building either, and I expect users would prefer compilation errors instead of silent memory corruption at runtime.
But perhaps a more friendly approach would be a cmd/vet warning in Go 1.N, and then make it into a compiler error in Go 1.N+1 (and maybe gated by Go language version specified in go.mod).

Unfixed example in the wild: https://github.com/alecthomas/unsafeslice/issues/4
Corresponding static check proposal: https://github.com/dominikh/go-tools/issues/782

Oh yeah, it's super easy to find instances of these mistakes. I found these in under 5 minutes with GitHub search, and only made it to page 5 of search results by time I stopped:

20 examples of `&reflect.SliceHeader{`

https://github.com/asukakenji/go-benchmarks/blob/22b6e1f1e8945cd6457cdcd9a7eb107232aadc59/common/reinterpret/slice.go#L17
https://github.com/kapitan-k/goutilities/blob/bb87f3ebc7d6e02f1e830cdfa8158433a2bca509/unsafe/unsafe.go#L10
https://github.com/hodgesds/offheap/blob/2cbc7fd25acf1c79a0f93d7d24d6512fda335a49/offheap.go#L70
https://github.com/snowwalf/goFeature/blob/2cd77dc1890f287dbbfc607080076c8aab4f1e6d/util.go#L49
https://github.com/gopher-os/gopher-os/blob/1a7aca464ed24473d1a01d1b6a7c42ebabcfc6d0/src/gopheros/kernel/mem_util.go#L18
https://github.com/sriharikapu/goos-e/blob/6559187cea7acf8ec9491d51d4eb1ef1361ac5fc/src/goose/kernal/mem_util.go#L18
https://github.com/ShoichiroKitano/kagemusha/blob/96535bbcd44b3f4d3ef5fd9e91e6c2bf2908a806/kagemusha.go#L30
https://github.com/lukechampine/few/blob/c282d8b842342cde9822744dd21fac7d2a36fcbc/few_test.go#L15
https://github.com/ssttevee/go-quickjs/blob/e069ecd8cf4fbf52f155f0d3b367d43b11ff59c9/internal/util.go#L12
https://github.com/damilarelana/goSystemDesign/blob/1a5360e05e0bf0fdb87ac7962e685a2a8e36d241/_vendor-20200721191145/github.com/etcd-io/bbolt/page.go#L68
https://github.com/theclapp/gio-mirror/blob/0dd77be975432179d6b71499818d4cda990fc110/internal/unsafe/unsafe.go#L15
https://github.com/ldeng7/go-x/blob/bc6b79c40822d5deea7eea959f44ec0b3444b169/monkey/monkey_unix.go#L16
https://github.com/alexbakker/tox4go/blob/f2fe5c787e3e811ae7ffffa8eb4758748ed90b68/cmd/state-lib/lib.go#L61
https://github.com/apparentlymart/go-onig/blob/5774566913b23021e6a2ca6b9d65d262e02a39bb/onig/bindings.go#L173
https://github.com/PieterD/glimmer/blob/f4ac04f74487b5082cb482f1a1d7b9a9513cfbf2/internal/convc/convert.go#L51
https://github.com/soloos/solofs/blob/6b9c058e6c0f3c596bee85fe83296f103521f58c/app/libsolofs/export_solodn.go#L19
https://github.com/neutralinsomniac/gio/blob/0dd77be975432179d6b71499818d4cda990fc110/internal/unsafe/unsafe.go#L15
https://github.com/timob/sindex/blob/c0720f83ad22370f1bfe879d7604230eac576340/unsafe.go#L31
https://github.com/mcronce/gitcrypt/blob/0000000f35e6edd54ddc5a8684bdb650aa4d5e87/pkg/gitcrypt/util.go#L20
https://github.com/lentus/wotscoin/blob/e81589f6fcaf6eba85305b8db2c31ec0a73bb4b5/lib/others/qdb/membind.go#L34

Making it a compiler error will be a hard stop to this. On the other side these code instances will lead to subtle memory corruption bugs that need the right circumstances come together and may not be caught by tests just checking for correct output values of the functioncs involved. They can be hard to diagnose which could justify this.

@martisch, the compiler error gated on go version in the (dependency's) go.mod file solves the friction problem. If it's in your dependency and your dependency says it wants the old version of Go, the code still compiles.
Definitely sounds like we should flag it in cmd/vet. I'm not really sure we need to move on to the compiler error. It would be a very specific thing for a compiler to check. (We don't, for example, flag bad regexp arguments to regexp.MustCompile in the compiler, even though we could plausibly do that in vet.)
I suggest we do just the vet check and not plan on the compiler change.

We don't, for example, flag bad regexp arguments to regexp.MustCompile in the compiler, even though we could plausibly do that in vet.

I think there's a difference that compilers already have to specially handle reflect.SliceHeader and reflect.StringHeader because of the unsafe.Pointer safety rules. E.g., escape analysis, walk, and SSA lowering currently have code specific to using these types.
There's also the difference that misusing regexp.MustCompile still has defined semantics: it panics in a very loud and obvious way, which is clearly user error.
However, misusing reflect.SliceHeader / reflect.StringHeader means silent memory corruption. Not only are these failures harder to track down, I wouldn't be surprised if they're responsible for some of the issues being reported against the Go runtime. E.g., #40397 was reported as a runtime.selectgo crash, which I spent a while looking into; but it's plausible (albeit not yet confirmed) it was actually due to misuse of reflect.SliceHeader in that package: https://github.com/ethereum/go-ethereum/pull/21372/files.

I tried running cmd/vet patched with CL 248192 on the 20 packages I identified above. I wasn't able to immediately compile 6 of them due to missing C dependencies, but vet identifies reflect.SliceHeader misuse in the other 14:

go vet warnings
# github.com/asukakenji/go-benchmarks/common/reinterpret
./slice.go:17:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./slice.go:31:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./slice.go:45:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./slice.go:59:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./slice.go:73:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./slice.go:87:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./slice.go:109:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./slice.go:123:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./slice.go:137:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
# github.com/kapitan-k/goutilities/unsafe
./unsafe.go:10:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./unsafe.go:19:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./unsafe.go:38:37: possible misuse of reflect.SliceHeader
	return *(*[]uint64)(unsafe.Pointer(&reflect.SliceHeader{
	                                   ^
./unsafe.go:47:36: possible misuse of reflect.SliceHeader
	return *(*[]int64)(unsafe.Pointer(&reflect.SliceHeader{
	                                  ^
./unsafe.go:64:37: possible misuse of reflect.SliceHeader
	return *(*[]uint64)(unsafe.Pointer(&reflect.SliceHeader{
	                                   ^
./unsafe.go:73:36: possible misuse of reflect.SliceHeader
	return *(*[]int64)(unsafe.Pointer(&reflect.SliceHeader{
	                                  ^
./unsafe.go:83:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./unsafe.go:93:37: possible misuse of reflect.SliceHeader
	return *(*[]uint64)(unsafe.Pointer(&reflect.SliceHeader{
	                                   ^
./unsafe.go:103:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./unsafe.go:113:36: possible misuse of reflect.SliceHeader
	return *(*[]int64)(unsafe.Pointer(&reflect.SliceHeader{
	                                  ^
./unsafe.go:123:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./unsafe.go:133:37: possible misuse of reflect.SliceHeader
	return *(*[]uint32)(unsafe.Pointer(&reflect.SliceHeader{
	                                   ^
# github.com/gopher-os/gopher-os/src/gopheros/kernel
./mem_util.go:18:38: possible misuse of reflect.SliceHeader
	target := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                    ^
./mem_util.go:37:40: possible misuse of reflect.SliceHeader
	srcSlice := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                      ^
./mem_util.go:42:40: possible misuse of reflect.SliceHeader
	dstSlice := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                      ^
# github.com/sriharikapu/goos-e/src/goose/kernal
./mem_util.go:18:38: possible misuse of reflect.SliceHeader
	target := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                    ^
./mem_util.go:37:40: possible misuse of reflect.SliceHeader
	srcSlice := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                      ^
./mem_util.go:42:40: possible misuse of reflect.SliceHeader
	dstSlice := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                      ^
# github.com/ShoichiroKitano/kagemusha
./kagemusha.go:30:40: possible misuse of reflect.SliceHeader
	original := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                      ^
./kagemusha.go:36:36: possible misuse of reflect.SliceHeader
	page := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                  ^
./kagemusha.go:62:40: possible misuse of reflect.SliceHeader
	original := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                      ^
./kagemusha.go:70:36: possible misuse of reflect.SliceHeader
	page := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                  ^
# github.com/lukechampine/few
./few_test.go:15:34: reflect.SliceHeader composite literal uses unkeyed fields
		_ = *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{uintptr(unsafe.Pointer(&x)), int(unsafe.Sizeof(x)), int(unsafe.Sizeof(x))}))
		                               ^
./few_test.go:24:34: reflect.SliceHeader composite literal uses unkeyed fields
		_ = *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{uintptr(unsafe.Pointer(&x)), int(unsafe.Sizeof(x)), int(unsafe.Sizeof(x))}))
		                               ^
./few_test.go:37:35: reflect.SliceHeader composite literal uses unkeyed fields
			_ = *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{uintptr(unsafe.Pointer(x[j])), int(unsafe.Sizeof(x[j])), int(unsafe.Sizeof(x[j]))}))
			                               ^
./few_test.go:15:33: possible misuse of reflect.SliceHeader
		_ = *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{uintptr(unsafe.Pointer(&x)), int(unsafe.Sizeof(x)), int(unsafe.Sizeof(x))}))
		                              ^
./few_test.go:24:33: possible misuse of reflect.SliceHeader
		_ = *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{uintptr(unsafe.Pointer(&x)), int(unsafe.Sizeof(x)), int(unsafe.Sizeof(x))}))
		                              ^
./few_test.go:37:34: possible misuse of reflect.SliceHeader
			_ = *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{uintptr(unsafe.Pointer(x[j])), int(unsafe.Sizeof(x[j])), int(unsafe.Sizeof(x[j]))}))
			                              ^
# go.etcd.io/bbolt
./freelist.go:107:35: possible misuse of reflect.SliceHeader
	dst := *(*[]pgid)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./freelist.go:297:36: possible misuse of reflect.SliceHeader
		ids := *(*[]pgid)(unsafe.Pointer(&reflect.SliceHeader{
		                                 ^
./node.go:234:34: possible misuse of reflect.SliceHeader
		b := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
		                               ^
./page.go:68:46: possible misuse of reflect.SliceHeader
	return *(*[]leafPageElement)(unsafe.Pointer(&reflect.SliceHeader{
	                                            ^
./page.go:86:48: possible misuse of reflect.SliceHeader
	return *(*[]branchPageElement)(unsafe.Pointer(&reflect.SliceHeader{
	                                              ^
./page.go:95:35: possible misuse of reflect.SliceHeader
	buf := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./page.go:118:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./page.go:135:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./page.go:144:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./tx.go:540:37: possible misuse of reflect.SliceHeader
			buf := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
			                                 ^
./tx.go:579:36: possible misuse of reflect.SliceHeader
		buf := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
		                                 ^
# gioui.org/internal/unsafe
./unsafe.go:15:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./unsafe.go:32:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&sh))
	                                 ^
# github.com/ldeng7/go-x/monkey
./monkey.go:31:40: reflect.SliceHeader composite literal uses unkeyed fields
	tarCode := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{patch.tar, l, l}))
	                                      ^
./monkey_unix.go:16:35: reflect.SliceHeader composite literal uses unkeyed fields
	sl := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{pb, pl, pl}))
	                                 ^
./monkey_unix.go:22:35: reflect.SliceHeader composite literal uses unkeyed fields
	sl := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{p, iLen, iLen}))
	                                 ^
./monkey.go:27:34: possible misuse of unsafe.Pointer
	code := getJumpCode(*(*uintptr)(unsafe.Pointer(rp)))
	                                ^
./monkey.go:31:39: possible misuse of reflect.SliceHeader
	tarCode := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{patch.tar, l, l}))
	                                     ^
./monkey_unix.go:16:34: possible misuse of reflect.SliceHeader
	sl := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{pb, pl, pl}))
	                                ^
./monkey_unix.go:22:34: possible misuse of reflect.SliceHeader
	sl := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{p, iLen, iLen}))
	                                ^
# github.com/PieterD/glimmer/internal/convc
./convert.go:41:35: possible misuse of reflect.SliceHeader
	return *(*string)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./convert.go:51:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./convert.go:95:42: possible misuse of reflect.SliceHeader
	pointers := *(*[]*uint8)(unsafe.Pointer(&reflect.SliceHeader{
	                                        ^
# gioui.org/internal/unsafe
./unsafe.go:15:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./unsafe.go:32:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&sh))
	                                 ^
# github.com/timob/sindex
./list_test.go:71:1: ExampleIterator refers to unknown identifier: Iterator
func ExampleIterator() {
^
./list_test.go:139:1: ExampleStack refers to unknown identifier: Stack
func ExampleStack() {
^
./unsafe.go:31:35: possible misuse of reflect.SliceHeader
	dst := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{Data: data, Len: lenBytes, Cap: lenBytes}))
	                                 ^
./unsafe.go:34:35: possible misuse of reflect.SliceHeader
	src := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{Data: data, Len: lenBytes, Cap: lenBytes}))
	                                 ^
# github.com/mcronce/gitcrypt/pkg/gitcrypt
./util.go:20:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
./util.go:29:35: possible misuse of reflect.SliceHeader
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
	                                 ^
# github.com/lentus/wotscoin/lib/others/qdb
./membind.go:34:35: possible misuse of reflect.SliceHeader
		res = *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{Data:uintptr(v.data), Len:int(v.datlen), Cap:int(v.datlen)}))
		                                ^
./membind.go:65:36: possible misuse of reflect.SliceHeader
		f.Read(*(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{Data:uintptr(v.data), Len:int(v.datlen), Cap:int(v.datlen)})))
		                                 ^

@mdempsky, I see your point about silent memory corruption being the province of the compiler.
But looking at the examples that you found and assuming the compiler gets involved, I wonder if maybe the answer is for the compiler to treat the uintptr field of reflect.SliceHeader and reflect.StringHeader as a pointer as far as type bitmaps. Then this kind of allocation becomes safe and the code becomes correct, instead of forcing everyone to rewrite the code.
There's at least some precedence for this in the way we handle "temporary" uintptrs as unsafe.Pointers in the unsafe.Add-equivalent code today, and also in the way we handle uintptrs in system calls.
We would also want to treat assignment of a plain uintptr to that field as a conversion to unsafe.Pointer in the current unsafe.Pointer vet check (the one that flags things like p := (*T)(unsafe.Pointer(uintptrVariable))).
If there's a way to make existing code correct instead of flagging it being wrong, that might be preferable.
What do you think?

As I understand your counter-proposal, it's to change reflect.{Slice,String}Header such that (1) the Data fields actually denote pointer-holding memory (while remaining declared as type uintptr), and (2) loads/stores of these fields are effectively implicit conversions to/from uintptr (with semantics identical to an explicitly written conversion).
I agree that would better match typical user expectations and make valid a lot of currently invalid user code, which would be great. I have two minor concerns though: one about backwards compatibility, and another about corner cases of language semantics.
--
1. Backwards compatibility. This has the risk of breaking code like:
var h = reflect.SliceHeader{Data: 42}

This use of reflect.SliceHeader is silly and ill-advised, but currently allowed. However, under this proposal, it becomes invalid just like how
var p = unsafe.Pointer(uintptr(42))

has been invalid since Go 1.3.
Now, the documentation for reflect.{Slice,String}Header already warns: "It cannot be used safely or portably and its representation may change in a later release." Also, checkptr would be able to catch this new form of misuse, so the breakage would be obvious and actionable. So I would argue this is an acceptable risk, on par with Go 1.3 breaking p.
2. Language semantics. I'm concerned this muddies type identity for uintptr and implies consequences beyond those intended. E.g., &h.Data now semantically involves an implicit conversion from *unsafe.Pointer to *uintptr. Converting (*T)(&h) where T has the same underlying type as reflect.SliceHeader has similar consequences. There might be weird implications for reflection as well.
I don't immediately see any ways this is likely to bite real-world Go programs, but I'd want to think about this some more.
--
Counter-counter-proposal. If this counter-proposal (i.e., ratifying existing user practice) is a direction we're interested in going in, I'd suggest we consider a slight tweak to it: actually change Data's declared type from uintptr to unsafe.Pointer, and figure out what type-checking exceptions we need to make to continue accepting most (if not all) current Go programs.
As I pointed out above, we are allowed to change the types of these fields (even if it breaks currently valid programs). And if we're going to have to continue special casing loads/stores of the Data field anyway, I think it's preferable to do this once during type checking rather than scattered a bunch of places throughout the back end.
I wouldn't be surprised if simply allowing assignments of uintptr-typed values to Data and vice-versa (i.e., assignments of Data to uintptr-typed variables) was already enough for >99% of currently valid code. At least skimming through Google's internal Code Search for uses of reflect.{Slice,String}Header.Data, I don't see any code this would still break.

@ianlancetaylor raised some discomfort with putting this knowledge in the compiler (as did @mdempsky).
Given that (1) these are used mainly for "interesting" conversions, and (2) we have active discussions on other issues about how to support those kinds of conversions directly, perhaps the right path forward is to finish figuring out what the conversions should look like and then do the vet check. The vet failures can be fixed by either writing the code to only use pointers to these structs or, in many cases, to use whatever the new conversions are and avoid the structs entirely.
We could still consider elevating the vet check to the compiler, under the rationale that the compiler can see that the program may cause memory corruption, but we could start with the vet check. There are of course plenty of other ways to cause memory corruption that the compiler does not reject.

perhaps the right path forward is to finish figuring out what the conversions should look like and then do the vet check.

 I think it would be a disservice to Go users if we know they're routinely misusing reflect.SliceHeader and reflect.StringHeader, and that misuse is easily caught, but we don't help them with that because we're unsure on the best spelling.

I think a vet check is fine, and that's what you asked for in this proposal.  I support that.
I'm just not comfortable with a compiler change.  It doesn't make sense to me.  reflect.StringHeader is just a struct.  I think that having the compiler reject a struct because it is routinely misused is confusing.
Separately, I think it's worth considering why people use reflect.SliceHeader and reflect.StringHeader, and then ensuring that there are alternate mechanisms for all of those uses.  For example, using reflect.SliceHeader to change the type of a slice, or to convert a pointer to a slice, is, I believe, #19367.  Are there any other uses of reflect.SliceHeader that can't be done in other ways?  I can't think of any offhand.
The case of reflect.StringHeader is still open.  As far as I know, given a string, there is no other way to get the pointer to the actual string data.  And as far as I know, there is no way to turn a pointer and a length into a string.  So it seems to me that we should have proposals for ways to do those actions without using reflect.StringHeader.

The special rules about StringHeader and SliceHeader are permission rules.  You are permitted to do a special operation with those structs.  They don't forbid other uses of those structs.
The use in the compiler is only for supporting -d=checkptr, which is a very useful checking operation but doesn't affect code generation.
I think I hear what you are saying, and I agree that there is a problem with these structs in practice, and I agree that a vet check is appropriate.  I just don't see why prohibiting them in the compiler is right.

The use in the compiler is only for supporting -d=checkptr, which is a very useful checking operation but doesn't affect code generation.

No, it does affect code generation. Escape analysis has to know that pointers flow through assignments to Data and code generation has to use write barriers for it.
For example, compile this code with -m -S (i.e., without checkptr), and you'll see different escape analysis for the two and very different assembly:
package p

import (
	"reflect"
	"unsafe"
)

func F(h *reflect.SliceHeader, p unsafe.Pointer) {
	h.Data = uintptr(p)
}

func G(h *mySliceHeader, p unsafe.Pointer) {
	h.Data = uintptr(p)
}

type mySliceHeader struct {
	Data uintptr
	Len, Cap int
}

// Assert that underlying struct types are identical.
var _ = (*reflect.SliceHeader)(&mySliceHeader{})

Ah, OK, thanks.

It sounds like there is consensus here that adding a vet check is OK.
It also sounds like there is no consensus about broader compiler changes.
Since the title of the issue is tracking the vet change, this (the vet change) seems like a likely accept.

No change in consensus, so accepted.

This was done in a78bdc4.

There should be no false positives for this
This check points out an issue in our mmap implementation. It's basically the same as the one in sys/unix, so this change breaks the golang/sys package too, this is the line: https://github.com/golang/sys/blob/0a15ea8d9b02651b828a1b41989a6af25c24cb64/unix/syscall_unix.go#L120. I think this shouldn't be a problem, since the memory pointed to is managed by the user code, not the go runtime.


Networks API naming and authorization.

I think network supported should be treated as a separate plugin.  The current design merges network support nto nameScoping.go, whose focus was container name scoping.  This ireduces our flexibility and is not a very modular approach.  For instance,  it means that another approach to networking would affect the way we do container name scoping.

Why is this network related pull request also making changes to container related commands?

I thought we agreed that the naming convention for commands prefix with the object name followed by action,  e.g. containercreate, containerlist.  But this push is using the opposite, e.g. "deleteNetwork", "inspectNetwork".

Separated network functions from nameScoping and authZ. Changed command names. Swarm integration tests are fail now as 'docker run' without --net is not supported.


Transforms and direct proxying (re-visited)

continuation of #801
I had a look at the new ITransformBuilder changes - while this brings us a step closer, I think this might need a little bit more refactoring...
Since we only use IHttpProxy we call services.AddHttpProxy - we don't have ITransformBuilder registered.
Since all the plumbing is internal, we cannot register ourselves and the extension methods hangs of the builder for the full fledged reverse proxy which requires additional configuration plumbing...
Possible I am missing something of course.

Fair point, the following example code was added to testassets/ReverseProxy.Direct project, but it can't actually resolve the service from DI:
            var transformBuilder = app.ApplicationServices.GetRequiredService<ITransformBuilder>();
            var transformer = transformBuilder.Create(context =>
            {
                context.AddQueryRemoveKey("param1");
                context.AddQueryValue("area", "xx2", false);
                context.AddOriginalHost(false);
            });

System.InvalidOperationException: No service for type 'Yarp.ReverseProxy.Service.ITransformBuilder' has been registered.

I'll move the registration into AddHttpProxy.

thanks!


"Satu Mare" and "Țibeni" are listed in the drop-down menu.

Description:
The villages are listed in the drop-down menu alongside its corresponding county and commune, according to the law
Preconditions:
The website (https://peviitor-ro.github.io/ui.orase/) is up and runing. After every step delete the text you typed from the search bar.

Step 1
Write in the search bar "Satu Mare".

Expected results
The location is listed in the drop-down menu as "Sat Satu Mare SUCEAVA (Satu Mare)".


add test cases for hive-hbase

Hi Rahul,
I added some hive-hbase test cases.  Can you please review.  I got a clean (with some known time out failures) functional pre-commit test.
Thanks,
Krystal

Looks good. Thanks for adding those tests krystal


[BEAM-12769] Fix typo in test class name, CLass -> Class

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @chamikaramj
I renamed the file and the class name inside. Was missed during review of #15343

LGTM. Thanks.

Run Java PreCommit

Run Java PreCommit


updated deletefile

Please fill in this pull request template before submitting
1. This pull request resolves #???
(^Fill in the issue number after #)
2. Description
(Write description of your changes here)
3. Fill in checklist by marking [x]

[ ] I've read the CONTRIBUTING.md
[ ] I was assigned to this issue
[x] My code is formatted using required linting
[ ] I've run my code locally and checked it works

@Shibu-goswami Hi there. This PR does not meet the contribution guidelines so can't be merged unfortunately. Please review the contribution.md and consider submitting another PR, thanks!


Update CDF Command and Control Topic Name

Description
https://zodiacpoolsystems.atlassian.net/browse/NGAC-367
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] Refactor (existing code being refactored)
[ ] This change includes a documentation update

@armanpathan2023 was this a mistake? This looks like a change specific to your company/use case?

Closed until we get feedback.

@ts-amz Yes, this was a mistake specific to our company use case. Apologies for the trouble.


Feature grafana/add resourcequotas persistence initcontainer

@rtluckie
What this PR does / why we need it:
This PR add the feature to config the resourcesquotas in the initcontainer when you enable the option persistence and initChownData
Which issue this PR fixes
#12395
Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[X] DCO signed
[x] Chart Version bumped
[X] Variables are documented in the README.md

/assign @maorfr

Hey @Jdavid7x,
thanks for contributing!
I find it a bit odd to have resources under persistence. do you think there is a more suitable location for these fields?

I think a good solution will be move the initchowndata like you have in stable/prometheus/values.yaml.
initChownData:
  ## If false, data ownership will not be reset at startup
  ## This allows the prometheus-server to be run with an arbitrary user
  ##
  enabled: true

  ## initChownData container name
  ##
  name: init-chown-data

  ## initChownData container image
  ##
  image:
    repository: busybox
    tag: latest
    pullPolicy: IfNotPresent

  ## initChownData resource requests and limits
  ## Ref: http://kubernetes.io/docs/user-guide/compute-resources/
  ##
  resources: {}

But now on grafana is not posible because the initchowndata is a variable and only have true or false

lets try to consult with the author of initChownData.
@dkarnutsch, would you mind taking a look at this PR and share your opinion?
thanks!

I think a good solution will be move the initchowndata like you have in stable/prometheus/values.yaml.

I think this would be the best solution.

Thanks! I am also +1 for that.

@dkarnutsch and @maorfr
I modified the branch with the example of prometheus.
One thing, this will break the compatibility since we created a new group of variables. Are you right with that?

I pushed the last changed discussed.
Regards.

Merged

/ok-to-test
this will probably also require to solve the conflicts in Chart.yaml

/retest

@maorfr
I tried to solve the conflicts but i don't have permissions to apply. Can you show me how to proceed?
thanks.

you need to rebase on top of master, resolve conflicts and force push
or
merge master to your branch, resolve conflicts and push

Done. Thanks for the info.

/lgtm
thanks for contributing!
and well done on the effort :)


V1.3.8 upgrade

Looks like trouble on Linux

Failed: Expected a message of type Akka.Persistence.LoadSnapshotResult, but received {LoadSnapshotFailed<Cause: System.InvalidOperationException: Invalid attempt to read from column ordinal '4'. With CommandBehavior.SequentialAccess, you may only read from column ordinal '5' or greater.
  at Npgsql.BackendMessages.DataRowSequentialMessage+<SeekToColumn>d__2.MoveNext () [0x0005e] in <2ff537c166614a2caa75e138c5a11ce5>:0
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <e29f59307e5e4e33ab73bb6530764ec9>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <e29f59307e5e4e33ab73bb6530764ec9>:0
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <e29f59307e5e4e33ab73bb6530764ec9>:0
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <e29f59307e5e4e33ab73bb6530764ec9>:0
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <e29f59307e5e4e33ab73bb6530764ec9>:0
  at Npgsql.BackendMessages.DataRowMessage+<SeekToColumnStart>d__17.MoveNext () [0x00078] in <2ff537c166614a2caa75e138c5a11ce5>:0

The combination of the changes introduced in v1.3.8 and the way SQL queries are written for this driver appear to have some issues on Linux, specifically. Weird - going to investigate further, but might just set the default for sequential-access to be off and do a release anyway so I can unblock our users.

Hah.... we currently don't run the tests on Windows... That explains that.

Filed an issue related to this: https://github.com/AkkaNetContrib/Akka.Persistence.PostgreSql/issues/53
I just want to get a compiling version of Akka.Persistence.Postgres out there so we can unblock the people on Akka.NET v1.3.8, so I'm going to disable sequential access = on for now.


LaTeX writer produce wrong notice block

In conjunction with the sphinxcontrib-traceables extension Sphinx will produce wrong LaTeX syntax. So far I do not yet assume that it is a bug in the extension code base.
This issue is for your information only. For details, see the Sphinx issue tracker:
sphinx-doc/sphinx/issues/3117

The last few days I've investigate this bug together with some people from the Sphinx project. I was running into two overlapping errors. There is one bug open inside the Sphinx LaTeX writer (see referred issue above), but that was not the reason for the origin bug here with the "traceables" extension.
I've found a mistake that happened while the transition from the old formatter in version 0.1.4 from traceables.py to the new differentiated formatters in display.py. I'll send a PR.

See PR #5


Update Social Media Sharing Text with New URL

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.
The link has been shared with people on social media twice. The second URL is intended to share more information about the experiment on a blog. As we now have a web page with more details about the Music Preferences experiment, let's use the second URL to share more information about the experiment.
Describe the solution you'd like
The second url : https://www.amsterdammusiclab.nl/blog/musical-preferences-china/
The text shared on social media will be as follows if we use the new URL for the second one:
"Let's explore Let's explore musical preferences together!" First URL (i.e., MP experiment URL) . "My favourite Top 3: X X X;" " I know N song out of 64 songs;" " Others' favourite Top 3: Y  X Y." Second URL (i.e., https://www.amsterdammusiclab.nl/blog/musical-preferences-china/)
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Hi @XHuang218, which social media channels do we currently have and who is currently managing our social media channels. Should we assign that person to this issue? I'm afraid I don't have access to those accounts.

Hi @drikusroor, we are using  Sina (a Chinese social media) for people to share their results of the experiment in a text format. The issue I want to address here is related to code update, not the management of the social media platform.  I want to replace the present second URL with a new one by default in the code, which is the webpage of the blog.

This is the data for the social media share feature in the experiment:
{
	"social": {
		"apps": [
			"weibo",
			"share"
		],
		"message": "I explored musical preferences on http://app.amsterdammusiclab.nl//mpst! My top 3 favorite songs: Split, Sun Circle, Letter to Me. I know 3 out of 8 songs. All players' top 3 songs: 偏爱, Try, Without Me",
		"url": "http://app.amsterdammusiclab.nl//mpst",
		"hashtags": [
			"mpst",
			"amsterdammusiclab",
			"citizenscience"
		]
	}
}

For my understanding, you want the url (with a red underline in the screenshot) to be replaced with https://www.amsterdammusiclab.nl/blog/musical-preferences-china/, correct?

Yes, I want the url in the red underline to be replaced with the new one.

I have added the second url to the experiment, both on ACC and APP. It should result into this after clicking the Weibo share button: https://service.weibo.com/share/share.php?url=https%3A%2F%2Fwww.amsterdammusiclab.nl%2Fblog%2Fmusical-preferences-china%2F&title=I explored musical preferences on http%3A%2F%2Fapp.amsterdammusiclab.nl%2F%2Fmpst! My top 3 favorite songs%3A Split%2C Faith in Numbers%2C 歌剧魅影. I know 0 out of 8 songs. All players' top 3 songs%3A 偏爱%2C Try%2C Without Me#_loginLayer_1704377677848


podofo: 0.9.5 -> 0.9.6

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/podofo/versions.
meta.description for podofo is: '"A library to work with the PDF file format"'.

Checks done (click to expand)

built on NixOS
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofobox had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofocolor had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofocountpages had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofocrop had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofoencrypt had a zero exit code or showed the expected version
/nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofogc passed the binary check.
/nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofoimgextract passed the binary check.
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofoimg2pdf had a zero exit code or showed the expected version
/nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofomerge passed the binary check.
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofopages had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofopdfinfo had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofotxt2pdf had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofotxtextract had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofouncompress had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofoimpose had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofoincrementalupdates had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofoxmp had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofosign had a zero exit code or showed the expected version
1 of 18 passed binary check by having a zero exit code.
2 of 18 passed binary check by having the new version present in output.
found 0.9.6 with grep in /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6
directory tree listing: https://gist.github.com/a969fc6ac97d9f12a657019acc639ea9
du listing: https://gist.github.com/1c75690d887f029a0ff81eb405f5fc1b

Test this build with Cachix (click to expand)

One time setup in nixpkgs Git checkout:
cachix use r-ryantm
git remote add r-ryantm https://github.com/r-ryantm/nixpkgs.git

Test this build:
git fetch r-ryantm && git checkout 4994ef2f44b4f19c137a2bcbc8b5e8bc5d694795 && nix-shell --pure -I nixpkgs=. -p podofo

Success on x86_64-linux (full log)
Attempted: podofo
Partial log (click to expand)
shrinking /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofoimg2pdf
shrinking /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofouncompress
shrinking /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofobox
shrinking /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin/podofoencrypt
gzipping man pages under /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/share/man/
strip is /nix/store/90vmpr41dzsx350k5argycaf693hnl1l-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/lib  /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6/bin
patching script interpreter paths in /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6
checking for references to /build in /nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6...
/nix/store/a1a8b9iwmsrbw40m25sg0dam7fwhkzhk-podofo-0.9.6

Success on x86_64-darwin (full log)
Attempted: podofo
Partial log (click to expand)
-- Installing: /nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6/share/man/man1/podofotxt2pdf.1
-- Installing: /nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6/share/man/man1/podofotxtextract.1
-- Installing: /nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6/share/man/man1/podofouncompress.1
-- Installing: /nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6/share/man/man1/podofoxmp.1
post-installation fixup
gzipping man pages under /nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6/share/man/
strip is /nix/store/8axizw5mf6dx7wip65nbkyrmlkhmjhc5-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6/lib  /nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6/bin
patching script interpreter paths in /nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6
/nix/store/vxcjdihsv9xqfnicmzbmkrr4si26ffhy-podofo-0.9.6


Remove Building better services with plain language

On December 17, Building better services with plain language will have its last live session. The e-learning version is not available. This pull request removes the course from:

The pages of the Hub
The site navigation
The homepage featured content section
The homepage Content section

Let's get the homepage fix pushed before this goes out.  https://github.com/cagov/hub.innovation.ca.gov/pull/91

Thanks for doing the fix before this and checking! It was all just in time. Merging now.


Guide/example for creating plugins

Dronecore is the officially supported layer for providing access to PX4, via MAVLink (of course there are other integration points, like the RTSP bridge). We're expecting others to extend the API via plugins, for example, to add support for peripherals that interact with the system via custom MAVLink messages.
We need a simple but realistic example of a plugin and an associated guide that shows how it works/what you need to do to write one.

More than 3 years later this will be covered by mavlink/MAVSDK-docs#219.


python3Packages.apache-airflow: Webserver fails to start with "No module named 'airflow'"

Describe the bug
$ airflow webserver
[2020-04-06 16:51:16,603] {__init__.py:51} INFO - Using executor SequentialExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-04-06 16:51:17,141] {dagbag.py:90} INFO - Filling up the DagBag from /tmp/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: <IP_ADDRESS>:8080
Timeout: 120
Logfiles: - -
=================================================================

Error: No module named 'airflow'
[2020-04-06 16:53:17,370] {cli.py:832} ERROR - No response from gunicorn master within 120 seconds
[2020-04-06 16:53:17,370] {cli.py:833} ERROR - Shutting down webserver

To Reproduce
Steps to reproduce the behavior:

nix-shell -p python3.withPackages (p: [p.apache-airflow])
AIRFLOW_HOME=/tmp/airflow airflow initdb
AIRFLOW_HOME=/tmp/airflow airflow webserver

Expected behavior
A functioning airflow webserver
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
❯ nix-shell -p nix-info --run "nix-info -m"                                                                                ~
 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.30, NixOS, 20.03beta1064.4dc8447c55f (Markhor)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.2`
 - channels(grfn): `"home-manager, nixpkgs-unstable-20.09pre218717.05f0934825c"`
 - channels(root): `"nixos-20.03beta1064.4dc8447c55f, nixos-hardware, nixpkgs-20.09pre218717.05f0934825c"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:
attribute:
  - python3Packages.apache-airflow
# a list of nixos modules affected by the problem
module:

Same issue when installing apache-airflow application:
~/ $ nix-env -i apache-airflow
installing 'apache-airflow-1.10.5'

~/ $ airflow initdb
[2020-04-24 04:25:36,700] {__init__.py:51} INFO - Using executor SequentialExecutor
DB: sqlite:////home/afarkas/airflow/airflow.db
[2020-04-24 04:25:37,588] {db.py:369} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Done.

~/ $ airflow webserver
[2020-04-24 04:25:44,856] {__init__.py:51} INFO - Using executor SequentialExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-04-24 04:25:45,787] {dagbag.py:90} INFO - Filling up the DagBag from /home/afarkas/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: <IP_ADDRESS>:8080
Timeout: 120
Logfiles: - -
=================================================================

Error: No module named 'airflow'

cc @costrouc @ingenieroariel

I was able to replicate:
~/o/nixpkgs> ./result/bin/airflow webserver
[2020-04-24 10:50:09,433] {__init__.py:51} INFO - Using executor SequentialExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-04-24 10:50:09,782] {dagbag.py:90} INFO - Filling up the DagBag from /x/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: <IP_ADDRESS>:8080
Timeout: 120
Logfiles: - -
=================================================================

Error: No module named 'airflow'
~/o/nixpkgs> git log | head -n5
commit c9e02db236a396f31c26efc03e6d0c6ee2edaacb
Merge: a9a38f9ad22 4328c327134
Author: Michael Raskin<EMAIL_ADDRESS>Date:   Fri Apr 24 15:13:30 2020 +0000

This was using:
If you are interested in helping debug
nix-build -A apache-airflow
./result/bin/airflow

For others following, this is what the airflow command being run expands to:
#! /nix/store/kgp3vq8l9yb8mzghbw83kyr3f26yqvsz-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/syqcp8zb9c0vxj2sx6jbv5f9q7yki4jz-python3-3.7.7/bin:/nix/store/l8b7wpwwswfnx8fn86lk
3biliqak8yqq-python3.7-apache-airflow-1.10.5/bin:/nix/store/as7ys868sp1sraw25v8n9qyg69m91ay7-python3.7-ale
mbic-1.3.2/bin:/nix/store/fm5dpmrk6764m3izdl461hi9hsg2babw-python3.7-Mako-1.1.1/bin:/nix/store/qfi1h97flkd
ax6ag0n0aj7545xcivr5g-python3.7-setuptools-45.2.0/bin:/nix/store/0lh7lhyj4j81s8vj4d36xqww78naa17c-python3.
7-dill-<IP_ADDRESS>/bin:/nix/store/6wpjcjrv1w535by8zb7ryhzwzwq532dh-python3.7-Flask-1.1.1/bin:/nix/store/rlw9fv
i7wkx9vssb5gn9wyra189phb6s-python3.7-Babel-2.7.0/bin:/nix/store/rg7xn0k8j4m3anprpgd5cy3miniyjxsi-python3.7
-flask-appbuilder-2.3.0/bin:/nix/store/dp3xjjlnch72jx2kir24dh25fwl8c5y7-python3.7-nose-1.3.7/bin:/nix/stor
e/1vkwlq2808a47dl6j5pw5vz1yvd48dmx-python3.7-coverage-4.5.4/bin:/nix/store/yy5fw2a8jsh1c4kh1klhq7ng783nq6i
s-python3.7-PyJWT-1.7.1/bin:/nix/store/hscnbb4cfqs3mcf2sczh0glyzzmmzryx-python3.7-jsonschema-3.2.0/bin:/ni
x/store/gvk9kdxbc0x8l1g4pz73f4nfcch2vky4-python3.7-flask-swagger-0.2.14/bin:/nix/store/8bfwj384ppfbl55q8m0
xc3abn9l1v82x-python3.7-future-0.18.2/bin:/nix/store/xi2bbrnksf0gnzgrx1r7nx9hpbkrb9sj-python3.7-gunicorn-2
0.0.4/bin:/nix/store/h6gaivxalq87ad9j9qfaivwawwzn91a2-python3.7-json-merge-patch-0.2/bin:/nix/store/qfd6iq
4y6xfqf40w2z0513qy1v28maby-python3.7-Markdown-3.1.1/bin:/nix/store/5nrzmkl70n8xsms2xwb25yzn1mhwvyhk-python
3.7-numpy-1.18.1/bin:/nix/store/n98q8sa05crlxad8bhqsrn7ckzi3kcfl-python3.7-xlrd-1.2.0/bin:/nix/store/ra2nm
j9l26j4nn5cwizca99r88pi1dmx-python3.7-tables-3.6.1/bin:/nix/store/s9kfxlgqf98k6ycw2qvid1637jdcl21a-python3
.7-pbr-5.4.4/bin:/nix/store/j29xzrayzxmm4ifs3vw5qgd922s26v0d-python3.7-Pygments-2.5.2/bin:/nix/store/78mhx
6klcqxkipqaa2hd2nkvhwkh1xch-python3.7-docutils-0.16/bin:/nix/store/wyvzykym54601iry33v05hzp864cbqzi-python
3.7-chardet-3.0.4/bin:/nix/store/nqhiqcvayd6lvfhrflq3zaj81h2j5f30-python3.7-tabulate-0.8.6/bin'${PATH:+':'
}$PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/l8b7wpwwswfnx8fn86lk3biliqak8yqq-python3.7-apache-airflow-1.10.5/bin/.airflow-wra
pped"  "$@"

airflow-wrapped is:
#!/nix/store/syqcp8zb9c0vxj2sx6jbv5f9q7yki4jz-python3-3.7.7/bin/python3.7
# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
import sys;import site;import functools;sys.argv[0] = '/nix/store/l8b7wpwwswfnx8fn86lk3biliqak8yqq-python3.7-apache-airflow-1.10.5/bin/airflow';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/l8b7wpwwswfnx8fn86lk3biliqak8yqq-python3.7-apache-airflow-1.10.5/lib/python3.7/site-packages','/nix/store/as7ys868sp1sraw25v8n9qyg69m91ay7-python3.7-alembic-1.3.2/lib/python3.7/site-packages','/nix/store/fm5dpmrk6764m3izdl461hi9hsg2babw-python3.7-Mako-1.1.1/lib/python3.7/site-packages','/nix/store/hadnldv7ckm66dz37mkifpvxsa7xzgaf-python3.7-MarkupSafe-1.1.1/lib/python3.7/site-packages','/nix/store/j9icidxvz5f7a7lbvk84wyll91lm9fv9-python3.7-SQLAlchemy-1.3.13/lib/python3.7/site-packages','/nix/store/m0d8j69qqpp0bd1w88sh2mnmcyxi46p9-python3.7-python-editor-1.0.4/lib/python3.7/site-packages','/nix/store/g741pd3n2v2zjqwswxv4x0qxr34hrmhd-python3.7-python-dateutil-2.8.1/lib/python3.7/site-packages','/nix/store/8wq3hfsck3k1qfjrib79jv96xp00raxq-python3.7-six-1.14.0/lib/python3.7/site-packages','/nix/store/rqx6b8vm3vixxim2p2a34cyja4d5ck83-python3.7-setuptools_scm-3.4.3/lib/python3.7/site-packages','/nix/store/qfi1h97flkdax6ag0n0aj7545xcivr5g-python3.7-setuptools-45.2.0/lib/python3.7/site-packages','/nix/store/sbiwbcvr6v8rg6m64mn27hp7dj8h5bnq-python3.7-cached-property-1.5.1/lib/python3.7/site-packages','/nix/store/7fab4pp4np2v11jbcaf5n4kfxmdrpiih-python3.7-colorlog-4.1.0/lib/python3.7/site-packages','/nix/store/iwkwaykzygpiif2d533lbx9ycj4prj10-python3.7-configparser-4.0.2/lib/python3.7/site-packages','/nix/store/a19s7kr0p8dyyz0sw5dqwvrz768gzsp8-python3.7-croniter-0.3.31/lib/python3.7/site-packages','/nix/store/0lh7lhyj4j81s8vj4d36xqww78naa17c-python3.7-dill-<IP_ADDRESS>/lib/python3.7/site-packages','/nix/store/6wpjcjrv1w535by8zb7ryhzwzwq532dh-python3.7-Flask-1.1.1/lib/python3.7/site-packages','/nix/store/jvlv78jmhccapbrfx0rmmd8vcb5l07ij-python3.7-itsdangerous-1.1.0/lib/python3.7/site-packages','/nix/store/l66s2rxh9k7m5bhh3nqy2l66gcwyzv66-python3.7-click-7.1.1/lib/python3.7/site-packages','/nix/store/l0x5q8lqvxj5zknnll8cdihh1axw9hmg-python3.7-Werkzeug-0.16.1/lib/python3.7/site-packages','/nix/store/f3nlwjf4y4qascnjjbvm51smfdnjs1a9-python3.7-Jinja2-2.11.1/lib/python3.7/site-packages','/nix/store/8sbm2kh4yxqgd97wp7p5jpww8kb8v5sy-python3.7-flask-admin-1.5.3/lib/python3.7/site-packages','/nix/store/dyp6klkrza12njhck8shc8al2qq4hndz-python3.7-wtforms-2.1/lib/python3.7/site-packages','/nix/store/rlw9fvi7wkx9vssb5gn9wyra189phb6s-python3.7-Babel-2.7.0/lib/python3.7/site-packages','/nix/store/65d9zbq73anghacjfr3zj8i4cnmgi356-python3.7-pytz-2019.3/lib/python3.7/site-packages','/nix/store/rg7xn0k8j4m3anprpgd5cy3miniyjxsi-python3.7-flask-appbuilder-2.3.0/lib/python3.7/site-packages','/nix/store/52rzp54ihhx40lj0qfz3nbaii4h2859f-python3.7-apispec-3.1.1/lib/python3.7/site-packages','/nix/store/z19qczyi5r6xbn0awpa6z8igc944qn4f-python3.7-colorama-0.4.3/lib/python3.7/site-packages','/nix/store/0mzpg09j8jin4zqicrxq548ja5fbki7m-python3.7-Flask-Babel-1.0.0/lib/python3.7/site-packages','/nix/store/6in41is3gm22in0hn3v9iqzy8pf1ap9l-python3.7-speaklater-1.3/lib/python3.7/site-packages','/nix/store/llpvrmdn81w6fbpnq96dl5852ibyyijm-python3.7-Flask-Login-0.4.1/lib/python3.7/site-packages','/nix/store/hx8g9brz0k1j95wa0p45l9ma8z76jl8h-python3.7-blinker-1.4/lib/python3.7/site-packages','/nix/store/92xhx8isx7w1qis89zxsp9w8xmvz2wj0-python3.7-flask-openid-1.2.5/lib/python3.7/site-packages','/nix/store/b6kygg7by37bjln19s7v63lzwvnydrbk-python3.7-python3-openid-3.1.0/lib/python3.7/site-packages','/nix/store/vjxww11cq2cxs8iiccb94pk8qi6s9ysa-python3.7-defusedxml-0.6.0/lib/python3.7/site-packages','/nix/store/nfnqlk5s6fzfjmnvs58aqggf93lqj3ja-python3.7-Flask-SQLAlchemy-2.4.1/lib/python3.7/site-packages','/nix/store/r9svnrrmhkkj48s4pwykpg897p9393rg-python3.7-Flask-WTF-0.14.2/lib/python3.7/site-packages','/nix/store/dp3xjjlnch72jx2kir24dh25fwl8c5y7-python3.7-nose-1.3.7/lib/python3.7/site-packages','/nix/store/1vkwlq2808a47dl6j5pw5vz1yvd48dmx-python3.7-coverage-4.5.4/lib/python3.7/site-packages','/nix/store/dnm31j1vli76wqkg8f3br9g1nqwxlgrx-python3.7-Flask-JWT-Extended-3.24.1/lib/python3.7/site-packages','/nix/store/yy5fw2a8jsh1c4kh1klhq7ng783nq6is-python3.7-PyJWT-1.7.1/lib/python3.7/site-packages','/nix/store/8b4myxbaib4bma3h7pwm323xhc0h4fyh-python3.7-packaging-20.1/lib/python3.7/site-packages','/nix/store/h7hdgmqpqvs1mfjg8vsg1wvdr7fanm5w-python3.7-pyparsing-2.4.6/lib/python3.7/site-packages','/nix/store/nyfxvr4js0ln8l6la0q5aq5w1a9adzfj-python3.7-pycparser-2.19/lib/python3.7/site-packages','/nix/store/rlpw0h2s9rj2kjfq3n5ykjyyqxryb13k-python3.7-cffi-1.14.0/lib/python3.7/site-packages','/nix/store/xq5isqx6syvdxvc6vc8fpv2yryzzf5zv-python3.7-cryptography-2.9/lib/python3.7/site-packages','/nix/store/a914g5ldn70k92j62w46rh0j0qr2q00d-python3.7-ecdsa-0.14.1/lib/python3.7/site-packages','/nix/store/hscnbb4cfqs3mcf2sczh0glyzzmmzryx-python3.7-jsonschema-3.2.0/lib/python3.7/site-packages','/nix/store/7906iy6a5630nxcphs4kpl8nj5vkb3f6-python3.7-attrs-19.3.0/lib/python3.7/site-packages','/nix/store/3mzlm8822rk9gm4w0r6v1pxhmkd6x1w3-python3.7-importlib-metadata-1.5.0/lib/python3.7/site-packages','/nix/store/cgi7bqrgymccrij9rab0jww7rb3s00nl-python3.7-zipp-0.6.0/lib/python3.7/site-packages','/nix/store/76iskzl0qzkg4r6kg7nfc9j31xl3z0mi-python3.7-more-itertools-8.0.2/lib/python3.7/site-packages','/nix/store/ik85nagk24cdml143160zipccnhdkby2-python3.7-pyrsistent-0.15.6/lib/python3.7/site-packages','/nix/store/74xcj81dj67njqak1xzsk22qqf8h0wah-python3.7-marshmallow-3.3.0/lib/python3.7/site-packages','/nix/store/dv983vml5gnpd77dzdaf7i2wpxli1iyc-python3.7-simplejson-3.17.0/lib/python3.7/site-packages','/nix/store/g2lg5hyqhv7ir3bdkphzn06195w98kgj-python3.7-marshmallow-enum-1.5.1/lib/python3.7/site-packages','/nix/store/qdgskpkxykw79w8agj95jmn045lf6jpp-python3.7-marshmallow-sqlalchemy-0.21.0/lib/python3.7/site-packages','/nix/store/wzcrgjw4chwclicmh2v8mrgk48yfjjqf-python3.7-prison-0.1.2/lib/python3.7/site-packages','/nix/store/3ch26ypzx5jqak3r1yparpapc8jiw9c7-python3.7-sqlalchemy-utils-0.36.0/lib/python3.7/site-packages','/nix/store/f90g6z6r5pfg9zqzv9nya2k1mazh5khx-python3.7-PyYAML-5.2/lib/python3.7/site-packages','/nix/store/0gcj6qijjyn2cl79nf3hgvb9370dpz5f-python3.7-flask-bcrypt-0.7.1/lib/python3.7/site-packages','/nix/store/1fyk9f52ri7vbx9rh4gs12rk4zb9f4ai-python3.7-bcrypt-3.1.7/lib/python3.7/site-packages','/nix/store/s7j7b6bfflak3gvbb0zz0l4mcd1lrrlg-python3.7-Flask-Caching-1.8.0/lib/python3.7/site-packages','/nix/store/gvk9kdxbc0x8l1g4pz73f4nfcch2vky4-python3.7-flask-swagger-0.2.14/lib/python3.7/site-packages','/nix/store/mwhjgx4wac7zk0l277k10h15gs6ly8jr-python3.7-funcsigs-1.0.2/lib/python3.7/site-packages','/nix/store/8bfwj384ppfbl55q8m0xc3abn9l1v82x-python3.7-future-0.18.2/lib/python3.7/site-packages','/nix/store/gydk8my4vpggrbccv6sq7y0l5drb7vy8-python3.7-GitPython-3.1.0/lib/python3.7/site-packages','/nix/store/02qp8an6hsr8fw229j5ig7gwjgfh76iq-python3.7-gitdb-4.0.2/lib/python3.7/site-packages','/nix/store/bgsh622mk28992mis9ypiv82lqcc5w4y-python3.7-smmap-3.0.1/lib/python3.7/site-packages','/nix/store/2jd36ff7yhlwh25dwf2dbcrj9xdm2fzc-python3.7-ddt-1.2.2/lib/python3.7/site-packages','/nix/store/xi2bbrnksf0gnzgrx1r7nx9hpbkrb9sj-python3.7-gunicorn-20.0.4/lib/python3.7/site-packages','/nix/store/vj3vavra1bn3vbadkn5jznfqazwcnc3k-python3.7-iso8601-0.1.12/lib/python3.7/site-packages','/nix/store/h6gaivxalq87ad9j9qfaivwawwzn91a2-python3.7-json-merge-patch-0.2/lib/python3.7/site-packages','/nix/store/nsqz3h2dksilni81z72myh8wgk8a4rdk-python3.7-ldap3-2.6.1/lib/python3.7/site-packages','/nix/store/a1b26mhkp9327c7li9i3sj8ngm6vn2k6-python3.7-pyasn1-0.4.8/lib/python3.7/site-packages','/nix/store/9i413lncpk9r0m67vn73myz76c40w0xk-python3.7-lxml-4.5.0/lib/python3.7/site-packages','/nix/store/8d1g09c8j399ji33lidrfq9qm2l76lkw-python3.7-lazy-object-proxy-1.4.3/lib/python3.7/site-packages','/nix/store/qfd6iq4y6xfqf40w2z0513qy1v28maby-python3.7-Markdown-3.1.1/lib/python3.7/site-packages','/nix/store/667sskqrzh71rw92ghcw66s5gbjx2v3h-python3.7-pandas-1.0.3/lib/python3.7/site-packages','/nix/store/q04vqj110zdf4cp2cg3445k52wv2yhp0-python3.7-scipy-1.4.1/lib/python3.7/site-packages','/nix/store/5nrzmkl70n8xsms2xwb25yzn1mhwvyhk-python3.7-numpy-1.18.1/lib/python3.7/site-packages','/nix/store/hdzn80dpwxh1lg4gnghwncgqhh4hmdjg-python3.7-numexpr-2.7.1/lib/python3.7/site-packages','/nix/store/n98q8sa05crlxad8bhqsrn7ckzi3kcfl-python3.7-xlrd-1.2.0/lib/python3.7/site-packages','/nix/store/9f03rc8r63sqqk2rx3v65m8jhxlvdd9a-python3.7-Bottleneck-1.3.1/lib/python3.7/site-packages','/nix/store/jwdappdrfbmmywcjbjljjz1dz4n2zv4q-python3.7-html5lib-1.0.1/lib/python3.7/site-packages','/nix/store/1qxbcdzrgk5i9v6skiwg26sa99v73p3h-python3.7-webencodings-0.5.1/lib/python3.7/site-packages','/nix/store/8njw6wchnrkzfil8752hxdrzmh8zklfa-python3.7-beautifulsoup4-4.8.2/lib/python3.7/site-packages','/nix/store/g04wmc427x01afybw7b6ni95czpakqxc-python3.7-soupsieve-1.9.5/lib/python3.7/site-packages','/nix/store/0bqn6z18h42r9ai35458dx2zdw598vvj-python3.7-openpyxl-3.0.2/lib/python3.7/site-packages','/nix/store/g2h6v8nkm0187zaa94jndphazyr96qvi-python3.7-jdcal-1.4.1/lib/python3.7/site-packages','/nix/store/zp02cxhsbkaxjvh32vxkvwxq350q8mig-python3.7-et_xmlfile-1.0.1/lib/python3.7/site-packages','/nix/store/ra2nmj9l26j4nn5cwizca99r88pi1dmx-python3.7-tables-3.6.1/lib/python3.7/site-packages','/nix/store/hy0mjvgf0pz1fvkxc5vbbpfxsy7h9416-python3.7-mock-3.0.5/lib/python3.7/site-packages','/nix/store/s9kfxlgqf98k6ycw2qvid1637jdcl21a-python3.7-pbr-5.4.4/lib/python3.7/site-packages','/nix/store/329f8rlsvkgi8yipngys1zmzjfkb6g33-python3.7-xlwt-1.3.0/lib/python3.7/site-packages','/nix/store/pyd4j6qp384dccra0gf10lq48xnzj515-python3.7-pendulum-2.0.5/lib/python3.7/site-packages','/nix/store/n69h7nlgv06b0rbyz6yid79162mdngqq-python3.7-pytzdata-2019.3/lib/python3.7/site-packages','/nix/store/7n22ay1an3xzvmcdspfdbkgk6z815hv6-python3.7-psutil-5.7.0/lib/python3.7/site-packages','/nix/store/j29xzrayzxmm4ifs3vw5qgd922s26v0d-python3.7-Pygments-2.5.2/lib/python3.7/site-packages','/nix/store/78mhx6klcqxkipqaa2hd2nkvhwkh1xch-python3.7-docutils-0.16/lib/python3.7/site-packages','/nix/store/hgixccigabr60pdv720p1nsr95a9ly7v-python3.7-python-daemon-2.2.4/lib/python3.7/site-packages','/nix/store/cin2k2zv3wim0fc7kjifwrhmc68wy9h2-python3.7-lockfile-0.12.2/lib/python3.7/site-packages','/nix/store/kmhb70nif21xfg7434rgb82855z9qqjy-python3.7-requests-2.22.0/lib/python3.7/site-packages','/nix/store/n45v09q4wbqwhmlpc2zbqc14a47flbqm-python3.7-urllib3-1.25.8/lib/python3.7/site-packages','/nix/store/l2p2cihvqijiqlhf9v0m8vn3dz9d4js0-python3.7-idna-2.8/lib/python3.7/site-packages','/nix/store/7favzsnm58x787mkdjvsvgdh6plxciys-python3.7-pyOpenSSL-19.1.0/lib/python3.7/site-packages','/nix/store/mrj2g3k2izkbrmdfs8hph3yp59zqbf10-python3.7-certifi-2019.11.28/lib/python3.7/site-packages','/nix/store/617zihgsdbanw4xqcq5hbg6alx7q6mci-python3.7-pysocks-1.7.1/lib/python3.7/site-packages','/nix/store/wyvzykym54601iry33v05hzp864cbqzi-python3.7-chardet-3.0.4/lib/python3.7/site-packages','/nix/store/3ql73q3r2qf0yk80k371s3j8fc835csk-python3.7-setproctitle-1.1.10/lib/python3.7/site-packages','/nix/store/nqhiqcvayd6lvfhrflq3zaj81h2j5f30-python3.7-tabulate-0.8.6/lib/python3.7/site-packages','/nix/store/yvzpw3m5qyy2l93nncf4xs2dg9fcyh2k-python3.7-tenacity-6.0.0/lib/python3.7/site-packages','/nix/store/d4i6cj0p4w89xjjp4378766jb9bzzmsa-python3.7-termcolor-1.1.0/lib/python3.7/site-packages','/nix/store/7va6jz36jkl9mqg3iqq1wqrimm106w1c-python3.7-text-unidecode-1.3/lib/python3.7/site-packages','/nix/store/dg7wzk6k1h1k4sz0v7420nyd62r4chgk-python3.7-thrift-0.13.0/lib/python3.7/site-packages','/nix/store/04a4c3dq5isv7n0xlwjqbj2dr8wvgkrl-python3.7-tzlocal-2.0.0/lib/python3.7/site-packages','/nix/store/1gcbyrwigqy2lidxc414w3rhlgw30952-python3.7-unicodecsv-0.14.1/lib/python3.7/site-packages','/nix/store/zpxark0dsfgxrvvfnb311mp9l48ww64g-python3.7-zope.deprecation-4.4.0/lib/python3.7/site-packages'], site._init_pathinfo());
import os
from airflow import configuration
from airflow.bin.cli import CLIFactory

if __name__ == '__main__':

    if configuration.conf.get("core", "security") == 'kerberos':
        os.environ['KRB5CCNAME'] = configuration.conf.get('kerberos', 'ccache')
        os.environ['KRB5_KTNAME'] = configuration.conf.get('kerberos', 'keytab')

    parser = CLIFactory.get_parser()
    args = parser.parse_args()
    args.func(args)

I modified the command to start a python shell and navigated to result/lib/python37/site-packages the problem seems related to pendulum.
A few questions arise:
#1. Why isn't the original error passed back and we just get something saying it could not be found?
#2. What do we need to add to the checkPhase to make sure the next time we have an error like this we get a build error? python -c "import airflow" ?
x@nixos /n/s/l/l/p/site-packages> bash /x/os/nixpkgs/here
Python 3.7.7 (default, Mar 10 2020, 06:34:06)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import airflow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/l8b7wpwwswfnx8fn86lk3biliqak8yqq-python3.7-apache-airflow-1.10.5/lib/python3.7/site-packages/airflow/__init__.py", line 31, in <module>
    from airflow.utils.log.logging_mixin import LoggingMixin
  File "/nix/store/l8b7wpwwswfnx8fn86lk3biliqak8yqq-python3.7-apache-airflow-1.10.5/lib/python3.7/site-packages/airflow/utils/__init__.py", line 24, in <module>
    from .decorators import apply_defaults as _apply_defaults
  File "/nix/store/l8b7wpwwswfnx8fn86lk3biliqak8yqq-python3.7-apache-airflow-1.10.5/lib/python3.7/site-packages/airflow/utils/decorators.py", line 34, in <module>
    from airflow import settings
  File "/nix/store/l8b7wpwwswfnx8fn86lk3biliqak8yqq-python3.7-apache-airflow-1.10.5/lib/python3.7/site-packages/airflow/settings.py", line 28, in <module>
    import pendulum
ModuleNotFoundError: No module named 'pendulum'

@ingenieroariel buildPythonPackage has a pythonImportsCheck that I think should be usable here

Some additional info:
In case the PATH entry for gunicorn is removed before calling into the wrapper script, the script could be started up without any problems.

fixed Error: No module named 'airflow' by  https://github.com/hardenedlinux/NSM-data-analysis/blob/9d82e231bdf2a4b17ed8e81323fc4f9b8482d054/pkgs/python.nix#L35

nix-build nix-build python.nix

./result/bin/airflow webserver
[2020-05-11 01:04:36,898] {__init__.py:51} INFO - Using executor SequentialExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-05-11 01:04:37,230] {dagbag.py:90} INFO - Filling up the DagBag from /home/gtrun/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: <IP_ADDRESS>:8080
Timeout: 120
Logfiles: - -
=================================================================
[2020-05-11 01:04:37 -0700] [31740] [INFO] Starting gunicorn 19.10.0
[2020-05-11 01:04:37 -0700] [31740] [INFO] Listening at: http://<IP_ADDRESS>:8080 (31740)
[2020-05-11 01:04:37 -0700] [31740] [INFO] Using worker: sync
[2020-05-11 01:04:37 -0700] [31939] [INFO] Booting worker with pid: 31939

nix-shell

with import <nixpkgs> { };
let
  own-airflow = pkgs.callPackage /home/gtrun/project/hardenedlinux-zeek-script/NSM-data-analysis/pkgs/python/apache-airflow {};
in
stdenv.mkDerivation {
  name = "env";
    buildInputs = [
      own-airflow
    ];
}

output

eft]$ airflow webserver
[2020-05-11 01:02:43,051] {__init__.py:51} INFO - Using executor SequentialExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-05-11 01:02:43,367] {dagbag.py:90} INFO - Filling up the DagBag from /home/gtrun/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: <IP_ADDRESS>:8080
Timeout: 120
Logfiles: - -
=================================================================
[2020-05-11 01:02:44 -0700] [5942] [INFO] Starting gunicorn 19.10.0
[2020-05-11 01:02:44 -0700] [5942] [INFO] Listening at: http://<IP_ADDRESS>:8080 (5942)
[2020-05-11 01:02:44 -0700] [5942] [INFO] Using worker: sync
[2020-05-11 01:02:44 -0700] [6216] [INFO] Booting worker with pid: 6216
[2020-05-11 01:02:44,171] {__init__.py:51} INFO - Using executor SequentialExecutor

@GTrunSec I'm reading your airflow package definition, but I can't tell what change there is actually fixing this issue.

@glittershark  https://github.com/NixOS/nixpkgs/pull/87612 PR, check it here.

maybe  you  use the wrong python version.
use command:
which python
see where the python command location ,and use command:
python --version
see which version python you used
it must be the same with you airflow installed(for example，if you install airflow used pip3 install apache-airflow, you must run
your python script use python3 xxx.py
good luck


Disable spell check for non-ascii languages

a patch for resolving #22

Thanks

Hi @gabrielelana, I found that using regex in this case is not a good practice. Since it causes lag when I'm editing a markdown file contains a lot of cjk words. I've just read the help page of 'spell' in vim, and found a better solution.
I'll make a new pull request. Sorry for disturbing.


Kernel restart when running first cell of example_inference_local_cddd.ipynb

Hi, I have created a local CDDD conda environment and am trying to run example_inference_local_cddd.ipynb, however my Jupyter Notebook kernel keeps on restarting when I try to run the first cell within the notebook. In particular, this line of code
from img2mol.inference import *
Unfortunately there's no error code, just this screenshot of the error I get from Jupyter Notebook itself. The same issue repeats after pressing the OK button and running the first cell again.
I believe the CDDD server is also down, as I am getting a connection error that has been reported in Issues before.

There should be error messages displayed in the console prompt where you started the Jupyter Notebook. I close this issue, until you have checked this first.

See comment above.


fix: replace jobq with workerpool lib to fix license issue

ref: longhorn/longhorn#4176
Before we used honestbee/jobq for concurrently getting all volumes from backupstore. We now have to replace it with gammazero/workerpool since honestbee/jobq has a LICENSE issue.
honestbee/jobq provides following features

Create workerpool
Submit a task to workerpool with a timeout setting, and get a TaskTracker as a return value for getting the task results when it is done.

gammazero/workerpool on the other hand,

Has no TaskTracker for tracking the result of the task

=> We have to create a channel to get the results from the task.

Cannot set timeout for the task

=> However, we already have a timeout for util.Execute. Thus, the command will timeout if it can't finish in 1 min(default value).

e2e test passed, run every test 3 times.
https://ci.longhorn.io/job/private/job/longhorn-tests-regression/3508/testReport/

e2e test passed, run every test 3 times. https://ci.longhorn.io/job/private/job/longhorn-tests-regression/3508/testReport/

This is good!


Adds gitlab-ci OP using GQ commitment binding

This PR adds a gitlab-ci OP to OpenPubkey and assorted tests. Testing this feature required adding a number of new testing features and building out ProviderVerifier table based unittests.
GQ Bound PK Tokens
For google and github we bind the CIC (Client Instance Claims) that contains the user's public key to the ID Token by putting a hashed commitment in one of the claims of the ID Token. For Google we put the commitment in the nonce claim, for github we put the commitment in the audience (aud) claim. Gitlab-CI does not provide an nonce claim and does not allow a running job/workflow to specify the aud. For gitlab the value of aud claim can only be specified in the static workflow config. This presents a problem: how to bind the CIC to the ID Token.
We provide a solution that solves this problem for gitlab-ci but also solves this problem more generally. We use the OP's RSA signature on the ID Token to create a GQ signature and then we delete the RSA signature. This GQ signature, functions as proof of knowledge of the RSA signature, enabling third parties to check that the ID Token was really signed by the OP. Additionally as the name suggests a GQ Signature is also a signature and use this signing property to bind the CIC to the ID Token.
More specifically, put the commitment, a.k.a., the hash of the CIC, in the protected header of the GQ signed ID Token and then we GQ sign the payload and the protected header using the header claim "cic". Note that to ensure we don't lose the original protected header of the ID Token, we base64 encode it and set it as the kid (KeyID) in the GQ Signed ID Token's protected header. Below we show an example PK Token that uses a GQ commitment binding:
payload: {
  "iss": "https://token.actions.gitlab.com",
  "aud": "OPENPUBKEY-PKTOKEN:1234",
  ...
}
signatures: [
  {"protected": {"typ": "JWT", "alg": "GQ256", "kid":  Base64(origProtected),
  "cic": SHA3_256(upk=alice-pubkey, alg=ES256, rz=crypto.Rand()),
  "signature": GQSIGN(RSA-sig, (payload, signatures[0].protected))`
  },
  {"protected": {"upk": alice-pubkey, "alg": "EC256", "rz": <rz>},
  "signature": <SIGN(alice-signkey, (payload, signatures[1].protected))>
  },
]

The design is based on the design proposed in https://github.com/openpubkey/openpubkey/issues/100 and should fix https://github.com/openpubkey/openpubkey/issues/26
Security
If configured correctly, this is approach offers the same security as the approach used with github. However GQ bound PK Tokens do introduce a security risk from misconfiguration not present in the github case. The risk is that if a GQ signature is not used and an attacker learns the ID Token, the attacker can compute the GQ signature from the RSA signature and insert their own public key.
We mitigate the risk by:

Requiring that PK Tokens that use the GQ signature to bind the CIC to the ID TOken must have "OPENPUBKEY-PKTOKEN:" prefixed in the audience and that they are not considered valid PK Tokens otherwise. This also prevents attacks in which a gitlab ID Token not intended for use as a PK Token is replayed as a PK Token. Note that this rule does not apply to other types of PK Tokens.
Ensuring that OpenPubkey clients always enforces the deletion of the RSA signature and only generates GQ signature PK Tokens for gitlab. We use memguard to reduce the risk of exposure through process memory.

We argue that this sort of misconfiguration is very unlikely because outside of OpenPubkey, ID Tokens are treated as highly sensitive authentication secrets and within OpenPubkey we specifically prevent this misconfiguration from occurring. Someone must configure their gitlab to put "OPENPUBKEY-PKTOKEN:" in the aud claim, not use the OpenPubkey protocol and then in addition to that they must expose the ID token to an attacker. We can not prevent people from writing their own insecure protocols.
TODOs

[X] Add Gitlab OP
[X] Add support for GQ commitment binding to gq library
[X] Adversarial tests
[X] Gitlab OP unittests
[X] GQ commitment binding VerifierProvider unittests
[X] Gitlab example
[X] Create manual test to check this works on gitlab. Repo set for this purpose here: https://gitlab.com/openpubkey/gl-test
[X] Allow specifying gitlab env variable to read ID Token from
[X] Full PR description of change

Tests
Successful OpenPubkey gitlab-CI job

Gitlab-CI: Manually tested this PR for the commit<PHONE_NUMBER>d11733e99dc3ce859cc0f0efb942bf https://gitlab.com/openpubkey/gl-test/-/jobs/6452884981

Gitlab Actions: Manually tested this PR for the commit<PHONE_NUMBER>d11733e99dc3ce859cc0f0efb942bf https://github.com/openpubkey/gha-test/actions/runs/8384339808/job/22961526436

I have successfully manually tested mfa cosigner on firefox. MFA cosigner on brave is breaking with the error Error during registration: NotAllowedError: The operation either timed out or was not allowed. See: https://www.w3.org/TR/webauthn-2/#sctn-privacy-considerations-client.It error is not related to these changes in this PR because I also tested against main and got the same error. I suspect something in a recent update to brave broke this.

@EthanHeilman Thanks for letting me know. Overall it addresses what I was thinking in #26, I just saw three little things:

It looks like here it's hardcoded to gitlab.com. Did I miss a place that could be specified? Is there a reason this couldn't also be supported with self-hosted installations as long as the host was specified?
It looks like the code and tests are consistent in requiring the ID Token aud prefixed with OPENPUBKEY-PKTOKEN:, but the description above uses OPENPUBKEY-PKTOKEN- (in case anyone comes back here for reference later).
Does the test repo https://gitlab.com/openpubkey/gl-test still exist, and is it possible it's private? I get a 404 when trying to open, which GitLab unfortunately gives for non-existent or no permissions so not sure, but I think a reference to a GitLab CI job with the appropriate settings (e.g. id_token) would be helpful.

@aarongoldenthal-work

It looks like here it's hardcoded to gitlab.com. Did I miss a place that could be specified? Is there a reason this couldn't also be supported with self-hosted installations as long as the host was specified?

I had not thought about that, but that is an excellent point. Created an additional constructor that allows anyone to specify the issuer for GitlabOP
func NewGitlabOp(issuer string, tokenEnvVar string) *GitlabOp {
	op := &GitlabOp{
		issuer:          issuer,
		publicKeyFinder: *discover.DefaultPubkeyFinder(),
		getTokensFunc:   getEnvVar,
		tokenEnvVar:     tokenEnvVar,
	}
	return op
}

It looks like the code and tests are consistent in requiring the ID Token aud prefixed with OPENPUBKEY-PKTOKEN:, but the description above uses OPENPUBKEY-PKTOKEN- (in case anyone comes back here for reference later).

Excellent catch, I'll fix that now. I changed that value so many times. Should be fixed now

Does the test repo https://gitlab.com/openpubkey/gl-test still exist, and is it possible it's private?

I thought I had made it public, but it was private. It is public now
https://gitlab.com/openpubkey/gl-test/
Thanks for all your comments. Let me know if you have anymore. If you end up using gitlab on OpenPubkey please open issuers for any problems or hit me up on email or slack.


Change back to TomlRB gem to resolve breaking attributes

This fixes the issues brought up in #45. The toml gems fixed all of their naming overlap issues and the original toml-rb gem now works again as expected. Based on conversation in #46, this seemed like the best option. I tested if people have done the work-around by wrapping the hash in an array and this still works.

Thanks @codylewandowski - can you squash your changes?  Otherwise this looks good to me. 👍

Nevermind - I forgot github will do it for me now... :)


Ruby faker fixes

Codecov Report

Merging #615 (dfc90f3) into main (3c0f168) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main     #615   +/-   ##
=========================================
  Coverage     92.81%   92.81%
  Complexity     2597     2597
=========================================
  Files           278      278
  Lines          5333     5333
  Branches        582      582
=========================================
  Hits           4950     4950
+ Misses          238      236    -2
- Partials        145      147    +2

Impacted Files
Coverage Δ

...c/main/java/net/datafaker/providers/base/Text.java
77.96% <0.00%> (-0.85%)
:arrow_down:

.../java/net/datafaker/service/FakeValuesService.java
84.72% <0.00%> (+0.19%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Use runMosaicTest for all unit tests

Rename BaseMosaicComposition to MosaicComposition, and MosaicComposition to RenderingMosaicComposition (with some changes)
Use RenderingMosaicComposition in renderMosaic instead of mosaicNodes
Use runMosaicTest in unit tests instead of mosaicNodes
Remove mosaicNodes
Create an internal TestMosaicComposition interface with the implementation of RealTestMosaicComposition
Add unit tests for synchronous renderMosaic

The main number of changes is simply the transfer of tests to runMosaicTest.

[x] CHANGELOG.md's "Unreleased" section has been updated, if applicable.

Just did a little diff minimization. I have some other suggestions/changes but I don't want to block this any further and I'll probably do them myself tonight.


IE11 Testing: Round 1

1. Hero: BG does not scale exactly how we would expect, but at normal window sizes it actually looks ok. It's between sizes where things get funky.
(Try adding this to head... apparently this corrects the CSS background-size: cover)
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
2. Feedback modal: Button slightly overlaps bottom of text input

3. section-panel: Description text just keeps goooooooooooing!

4. Add-on action modal: Is a bit of a mess, but likely caused by the text flow.

5. Account modal: Mild spacing issue

Resolved in https://github.com/StatCan/StatCan.OrchardCore/commit/c2e413cbad86965291f7eee3e9bf37870783fc64.


tsd mocha does not apply to files past index.js

The mocha file for tsd works correctly for the index.js file that is at the root level of the test folder.
Any nested files within the test folder, as well as any files that are siblings to index.js does not hide the green underline that shows "can't find it" or "can't find describe".
Expected that when installing the tsd mocha, that visual studio code should recognize the commands past the initial index.js file within the test directory.

It appears that the December release 0.10.5 has resolved this issue. Now when viewing files outside of index.js (both sibling and nested below it) the green underlines are not rendered.
Nothing in the update seems to point at what resolved this issue.