cmd/go: macOS on arm64 requires codesigning

On the production Apple Silicon machines, Go binaries are killed at start. https://github.com/golang/go/issues/38485#issuecomment-729301832
It looks like all binaries need to be codesigned now, and indeed running codesign -s - on them lets them run correctly.
This stops go test and go run from working, and requires an extra step after go build to get a functional binary.
This also affects the bootstrapped compiler itself.

@cherrymui indeed, binaries produced by the machine's clang are codesigned, in what looks like the same exact way as the output of codesign -s -.
filippo@Filippos-MacBook-Pro tmp % clang hello.c
filippo@Filippos-MacBook-Pro tmp % ./a.out
Hello M1!
filippo@Filippos-MacBook-Pro tmp % codesign -d -v a.out
Executable=/Users/filippo/tmp/a.out
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=510 flags=0x20002(adhoc,linker-signed) hashes=13+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

Actually, codesign outputs don't have the linker-signed flag.

Thanks, @FiloSottile !
Interesting. The C compiler on the DTK doesn't do that...
Could you try one more thing: run clang -v hello.c to see if the C compiler invokes codesign or how it signs the binary? Thanks!

It doesn't look like the linker invocation has anything special, so I assume ld just does that (and @jedisct1 said so on Twitter as well https://twitter.com/jedisct1/status/1328862207715794946).
"/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch arm64 -platform_version macos 11.0.0 11.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o a.out -L/usr/local/lib /var/folders/jh/3ydm4lxd71s2__g_x4hny6r00000gn/T/hello-580ca7.o -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.osx.a

I expect external linking should generate codesigned binaries, but trying `go build -ldflags="-linkmode=external" fails immediately with
# tmp
loadinternal: cannot find runtime/cgo

loadinternal: cannot find runtime/cgo

This is not a failure. It prints a message but it should successfully link the binary.

You're right, I do get the binary. Interestingly, it comes out linker-signed but doesn't run, and codesign doesn't work on it.
filippo@Filippos-MacBook-Pro tmp % codesign -d -v tmp
Executable=/Users/filippo/tmp/tmp
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=7038 flags=0x20002(adhoc,linker-signed) hashes=217+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
filippo@Filippos-MacBook-Pro tmp % ./tmp
zsh: killed     ./tmp
filippo@Filippos-MacBook-Pro tmp % codesign -s - tmp
tmp: the codesign_allocate helper tool cannot be found or used

Thanks!
Interesting... I'll see if it is possible to generate LC_CODE_SIGNATURE in the linker. If that's not possible, maybe shell out codesign.
Any down side for that? If the user wants to sign with a different identity, would that still be possible?

I don't understand this.  What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

Any down side for that?

I guess the main problem would be with cross-compilation, but anyway I guess adhoc signatures only work on the same machine where they are generated.
This would be a major pain for projects distributing binaries. I wonder how Homebrew deals with it, since by default they install pre-built "Bottles".

What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

I don't know, but I can imagine 1) to normalize the execution path such that it always check signatures unconditionally, simplifying it and 2) to block binaries generated on other machines (and not properly signed by a trusted Developer ID) which before was sort of the role of the quarantine attributes.

You're right, I do get the binary. Interestingly, it comes out linker-signed but doesn't run, and codesign doesn't work on it.

My understanding is this is a bug already reported to Apple, see https://github.com/Homebrew/brew/issues/9082#issuecomment-727247739

to block binaries generated on other machines (and not properly signed by a trusted Developer ID) which before was sort of the role of the quarantine attributes.

Looks like binaries aren't required to have a signature linked to a Developer ID if they were signed elsewhere. I was able to apply an ad-hoc signature to a binary on my Intel mac and transfer it to my M1 mac and it ran just fine. Appears like binaries just need a signature, doesn't really seem to matter where it came from.

Homebrew maintainer here:

the codesign_allocate helper tool cannot be found or used

That's indeed a bug in codesign, and Apple is aware

My understanding is this is a bug already reported to Apple, see Homebrew/brew#9082 (comment)

Yes. If this occurs, you need to change that file's inode (so copying it someplace and back will work). Once you've done that, you can sign and it will work. See https://github.com/Homebrew/brew/pull/9102 for details.

I don't think they're close to code signing bottles on ARM yet

We do not distribute bottles yet, because our CI is not yet fully operational, but our codebase is otherwise ready. We apply ad-hoc signatures as part of formula installation: https://github.com/Homebrew/brew/pull/9102

As for go, if Go uses its own linker (or anything other than Apple's ld) then it definitely needs to call codesign -s - on the binaries it produces: executables, shared libraries, etc.

@FiloSottile (and others): thank you for the information! 🙇
Do you know if it is possible to use darwin/amd64 architecture to cross-compile binaries that are able to execute on darwin/arm64, providing that the binaries are explicitly signed?
I am interested in what are the options to distribute binaries for Go projects that work on Apple Silicon without having to actually compile them on Apple Silicon.

@mislav Cross-compiling for darwin/arm64 works fine from darwin/amd64 and even  linux/amd64. Obviously the resulting binaries are not signed.  They will run if signed ad hoc on the same machine.  Binaries ad hoc signed on one machine didn't work on another machine for me.

Binaries ad hoc signed on one machine didn't work on another machine for me.

This comment seems to disagree: https://github.com/golang/go/issues/42684#issuecomment-729346276

I was able to apply an ad-hoc signature to a binary on my Intel mac and transfer it to my M1 mac and it ran just fine.

Ok, I tested it again and now cross-signing does work.  I might've transferred the wrong file earlier.  Whoops.

Codesigning from cmd/link should be possible. https://github.com/isignpy/isign by @neilk does that for iOS.

The Zig project is also working on this here: ziglang/zig#7103 (thanks @komuw).

Thanks for the pointers. I'll look into them.

Binaries ad hoc signed on one machine didn't work on another machine for me

I can confirm that a linker or other ad hoc signature is sufficient to run the binary (or shared library) on any machine.

A quick note from trying to hack together a working toolchain today: shelling out to codesign in the linker works, but any subsequent calls to buildid to update the build ID appears to break the signature (preventing the update produces working binaries). Presumably the signature will need to be done (either magically by the linker with LC_CODE_SIGNATURE or shelling to codesign) as the last step after any changes are made to the object.

@rolandshoemaker yeah, I tried that, too, and come to the same conclusion.
I'm not sure whether signing after stamping buildid will cause any issue for the go command's staleness analysis, though.

 Guess Apple does not want developers using their new hardware? Or forcing them to use the entire operating system as a very fancy ssh terminal.

@marcopeereboom Please be mindful of Gopher values when communicating in the Go issue tracker. Criticism is welcome when it adds new information to the discussion, otherwise it does not help us get to a resolution of this issue.

@kb091412 on Twitter found the official docs, which basically confirm everything we had deduced: https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-universal-apps-release-notes#Code-Signing

New in macOS 11 on Macs with Apple silicon, and starting in macOS Big Sur 11 beta 6, the operating system enforces that any executable must be signed before it’s allowed to run. There isn’t a specific identity requirement for this signature: a simple ad-hoc signature is sufficient. This new behavior doesn’t change the long-established policy that our users and developers can run arbitrary code on their Macs, and is designed to simplify the execution policies on Macs with Apple silicon and enable the system to better detect code modifications. This new policy doesn’t apply to translated x86 binaries running under Rosetta 2, nor does it apply to macOS 11 running on Intel-based platforms.
To reduce the impact on existing development workflows, starting with Xcode 12 beta 4, the toolchain will now automatically sign your executables whenever you build from Xcode, or use command-line utilities such as clang(1) or ld(1). Since this signing mechanism generates signatures directly at link time, and doesn’t cover any resource other than the executable, it is expected to be faster than a traditional codesign(1) invocation. If you use a custom workflow involving tools that modify a binary after linking (e.g. strip or install_name_tool) you might need to manually call codesign(1) as an additional build phase to properly ad-hoc sign your binary. These new signatures are not bound to the specific machine that was used to build the executable, they can be verified on any other system and will be sufficient to comply with the new default code signing requirement on Macs with Apple silicon. However, given that these signatures do not bear any valid identity, binaries signed this way cannot pass through Gatekeeper.

Interesting. Would love to see Go implement this ad-hoc signing directly instead of needing to shell out to codesign, so that cross compiling usable binaries is still possible.  Would also be good to use a static key so we can keep reproducibility.

An open-source ad-hoc signing tool was created for NixOS: https://github.com/thefloweringash/sigtool

Using it requires Apple's codesign_allocate, but that's open-source as well: https://opensource.apple.com/source/cctools/cctools-949.0.1/misc/codesign_allocate.c.auto.html

@cherrymui Oh hm, good point. I think signing after buildid would indeed break the staleness check, since adding the LC_CODE_SIGNATURE section will change the file hash, just as stamping buildid after signing breaks the signature for the same reason.
It seems like perhaps buildid on darwin would need to understand where the LC_CODE_SIGNATURE section is (or would be) in the object in order to create a hash that ignores that section and can be recalculated after the signature is added 😕.

Found a workaround that I'm using for now. The AppleMobileFileIntegrity.kext is what handles checking for code signing, and you can see in the Console that AMFI prohibits running:
AMFI: hook..execve() killing pid 9799: Attempt to execute completely unsigned code (must be at least ad-hoc signed).

However there is an undocumented boot argument called amfi_block_unsigned_code which disables this check.
Enter recovery mode by holding down the MacBook power button when the MacBook is powered off and wait until it says that it's loading the settings. Enter settings and open a terminal:
csrutil enable --without nvram

This will leave all security features intact except disable "NVRAM Protections" and "Boot-arg Restrictions". Reboot back into macOS and set the boot argument:
sudo nvram boot-args="amfi_block_unsigned_code=0"

Now reboot once again and you'll be able to run unsigned applications. Note however that a new message will be presented at boot:
AMFI: You have used the amfi_block_unsigned_code boot-arg. This will be removed in a future release.

This will be useful for developing the Go toolchain on Apple Silicon devices and toggling this behaviour.

Using it requires Apple's codesign_allocate, but that's open-source as well: https://opensource.apple.com/source/cctools/cctools-949.0.1/misc/codesign_allocate.c.auto.html

The good news here is that codesign_allocate is only used to verify the basic MachO layout of the __LINKEDIT hidden sections, and then inject an empty, 0-padded placeholder for the code signature as the very last section, plus a matching LC_CODE_SIGNATURE load command.

@cherrymui Oh hm, good point. I think signing after buildid would indeed break the staleness check, since adding the LC_CODE_SIGNATURE section will change the file hash, just as stamping buildid after signing breaks the signature for the same reason.
It seems like perhaps buildid on darwin would need to understand where the LC_CODE_SIGNATURE section is (or would be) in the object in order to create a hash that ignores that section and can be recalculated after the signature is added 😕.

Do you think the code signature will always be of the same size? What I inferred about codesign is that it precalculates the size needed and then calls codesign_allocate to preallocate the space in the MachO binary before the actual SuperBlob structure is generated and inserted.

Based on sigtool's source, it seems like the size of the signature is variable. I'm not a Go developer, but I'd guess that porting sigtool and codesign_allocate (both of which are open source) would make it pretty easy to add signing support to Go.

@leo60228 oh for sure. I was just wondering if you knew if it was fixed; then porting codesign_allocate functionality would amount to finding the offset of string table, and padding out a fixed size slice at string table offset plus its size. But since the size of the signature can be variable, this has to be taken into account, and I guess that the staleness would indeed break because of it, unless the code signature command and the matching section are ignored from this calculation. Anyhow, we'll be figuring out something similar for Zig, so I'll be more than happy to share any exciting news of success (if any).

Thanks all above for thoughts and pointers! Here is my plan:

Before stamping the buildid, allocate the LC_CODE_SIGNATURE load command with the right offset and size, zero-fill the signature blob. Probably done by the linker.
Stamp the buildid. The buildid would be made to be insensitive to the LC_CODE_SIGNATURE blob. (but can be sensitive to the load command itself, which should not change)
Fill in the signature blob. Since the buildid is insensitive to the signature blob, this shouldn't change the buildid.

I guess this may work. I'll try to code up something...

That's my plan for Zig as well minus the buildid step ;-) I also plan to stretch the structure a little bit in the sense I really don't like the fact that codesign_allocate bullies me into having no gaps between the __LINKEDIT hidden sections. I wonder if this is indeed strictly required for the code signature to be properly generated.

You don't have to pay anyone to code sign things right? It's just about running codesign <path-to-binary> and then sharing it with everyone?

You don't have to pay anyone to code sign things right? It's just about running codesign <path-to-binary> and then sharing it with everyone?

As long as we're talking about adhoc code signing, then yep, that's the case. The full command for this would actually be codesign -s - <path_to_binary>.

As long as we're talking about adhoc code signing

And then the users would be still met with a warning. Like this app is from unidentified developer, you sure you want to run it? or it will be okay to run as if everything is normal?

I should mention that real codesigning can be bundled up several ways without leaning the whole process. https://app.circleci.com/pipelines/github/drud/signing_tools has the tools that we use (in Catalina) for signing our golang binaries. It does require an Apple developer account. And they do break it regularly by requiring you to sign off on contract changes.

As long as we're talking about adhoc code signing

And then the users would be still met with a warning. Like this app is from unidentified developer, you sure you want to run it? or it will be okay to run as if everything is normal?

They will indeed. The GateKeeper is still there. If you want to please, you gotta code sign with an actual, valid developer identity.
A little bit more about adhoc code signing, not sure if this was mentioned here or not, but @andrewrk and myself have exchanged a couple of emails with Apple officials, and they told us that modification of an already signed file in-place will end up in SIGKILL 9 by the kernel even if you regenerate the signature. That's because the kernel apparently caches code signing info for each inode. Interestingly, you probably won't notice this when using codesign tool for instance, as it actually does a full file copy under-the-hood. This probably doesn't hurt Go as much as it does Zig where we perform incremental linking in-place.

That's because the kernel apparently caches code signing info for each inode.

I don't have an M1 to test with, so this might be a dumb question. Does modifying a codesigned executable in place invalidate this cache?

That's because the kernel apparently caches code signing info for each inode.

I don't have an M1 to test with, so this might be a dumb question. Does modifying a codesigned executable in place invalidate this cache?

Yep, it unfortunately does. Then, you either have to copy the entire contents into a new inode and code sign anew (thinking here of codesigning solutions that are part of Go/Zig's toolchains), or use codesign tool which will do that for you.

This is the Apple docs on running an app from an unidentified developer. In Catalina, it's considerably more complicated than previously -- you can't just Ctrl-click the app icon and select "Open" to get a dialog with a run-anyway option.
https://support.apple.com/en-us/HT202491
Has this changed in Big Sur?

A little bit more about adhoc code signing, not sure if this was mentioned here or not, but @andrewrk and myself have exchanged a couple of emails with Apple officials, and they told us that modification of an already signed file in-place will end up in SIGKILL 9 by the kernel even if you regenerate the signature. That's because the kernel apparently caches code signing info for each inode. Interestingly, you probably won't notice this when using codesign tool for instance, as it actually does a full file copy under-the-hood. This probably doesn't hurt Go as much as it does Zig where we perform incremental linking in-place.

My understanding is that the inode caching only gets in the way if you first execute the binary before (re-)codesign.  Does Zig execute a binary before a final codesign step?

Yes, it's part of the edit-compile-run development cycle. Programmer edits code. Compiler updates the binary in place. Programmer runs & debugs the application. Repeat.

Oh no, you're right. I guess I didn't explain myself well. That's the use case I had in mind. In Zig we're working on hot code swapping and incremental linking, so you can run the binary and then recompile just bits of it. Everything is done in place.

Oh no, you're right. I guess I didn't explain myself well. That's the use case I had in mind. In Zig we're working on hot code swapping and incremental linking, so you can run the binary and then recompile just bits of it. Everything is done in place.

Ah that all perfect sense.  Thanks for the reply @andrewrk and @kubkon.  Go performs builds in a $TMPDIR, so I guess each binary being built will have a different inode, even when moved into its final directory.

Yep we will be forced to do something similar with the new code signing requirements. Most likely performing the in-place update within the zig-cache directory, and then copying the binary to the output each time. Sadly Darwin has no copy_file_range syscall.

I hacked up something that does code signing. https://go-review.googlesource.com/c/scratch/+/271866
It just a separate command line program, that signs binary after it builds, not yet integrated to the toolchain. I just wanted to see if the algorithm works.
It seems to make the "codesign" command line tool happy. Could someone with an M1 machine give it a try, see if the kernel likes the binaries signed by this program? Just do
GOARCH=arm64 GOOS=darwin go build hello.go # generate unsigned binary
./codesign hello                           # sign

Thanks!
If this works, I'll do it for real.

@cherrymui looks like the kernel doesn't like it unfortunately
2020-11-19 20:13:58.195388-0800 0xbc230    Default     0x0                  0      0    kernel: CODE SIGNING: cs_invalid_page(0x100a5c000): p=48498[test] final status 0x23020200, denying page sending SIGKILL
2020-11-19 20:13:58.195416-0800 0xbc230    Default     0x0                  0      0    kernel: CODE SIGNING: process 48498[test]: rejecting invalid page at address 0x100a5c000 from offset 0x0 in file "/Users/rolandshoemaker/test" (cs_mtime:1605845619.452643497 == mtime:1605845619.452643497) (signed:1 validated:1 tainted:1 nx:0 wpmapped:1 dirty:0 depth:0)

It would be helpful to have a go tool ... command to sign objects besides Go binaries, for instance shell scripts.
I realize that exists on MacOS, but many folks assemble releases on Linux, etc.

@cherrymui I got the same result as @rolandshoemaker. In case it helps, some details:

$ export PATH="$PATH:$HOME/go116arm64/bin"
$ GOOS=darwin GOARCH=arm64 go build -o h0 hello.go
$ GOOS=darwin GOARCH=arm64 go build -o h1 hello.go && codesign -s - ./h1
$ GOOS=darwin GOARCH=arm64 go build -o h2 hello.go && /tmp/scratch/cherrycodesign ./h2

$ ./h0
zsh: killed     ./h0
$ ./h1
hello
$ ./h2
zsh: killed     ./h2

$ ls -la h*
-rwxr-xr-x  1 dmitri  staff  2042752 19 Nov 23:11 h0
-rwxr-xr-x  1 dmitri  staff  2087152 19 Nov 23:12 h1
-rwxr-xr-x  1 dmitri  staff  2058833 19 Nov 23:23 h2

$ diff -U0 <(codesign -d -vvvv ./h1 2>&1) <(codesign -d -vvvv ./h2 2>&1)
--- /dev/fd/11	2020-11-19 23:26:27.000000000 -0500
+++ /dev/fd/12	2020-11-19 23:26:27.000000000 -0500
@@ -1,2 +1,2 @@
-Executable=/private/var/folders/_0/h0671fcn4rgb5pn9c745dx2h0000gn/T/tmp.lKFaKwXh/h1
-Identifier=h1-962eb7c97571405dab96e632e60472cbd44c02f7
+Executable=/private/var/folders/_0/h0671fcn4rgb5pn9c745dx2h0000gn/T/tmp.lKFaKwXh/h2
+Identifier=./h2
@@ -4 +4 @@
-CodeDirectory v=20100 size=16124 flags=0x2(adhoc) hashes=499+2 location=embedded
+CodeDirectory v=20400 size=16061 flags=0x20002(adhoc,linker-signed) hashes=499+0 location=embedded
@@ -6,6 +6,4 @@
-CandidateCDHash sha1=44d83717b7f5d541d2dd5eee1cebfa5e49717088
-CandidateCDHashFull sha1=44d83717b7f5d541d2dd5eee1cebfa5e49717088
-CandidateCDHash sha256=710421ab654b828da6d1341fa487e4243d162103
-CandidateCDHashFull sha256=710421ab654b828da6d1341fa487e4243d162103b59185092c8759fadad50e61
-Hash choices=sha1,sha256
-CMSDigest=f2ba67a05485afc727cd4a9a2f95d1d04266eb0f7519897eeda415bcd21dae80
+CandidateCDHash sha256=c273a9f6c2be4c0bc4f040f542ea6d2c3b7057c4
+CandidateCDHashFull sha256=c273a9f6c2be4c0bc4f040f542ea6d2c3b7057c4ebb7d0b69167a44b8eeef3ff
+Hash choices=sha256
+CMSDigest=c273a9f6c2be4c0bc4f040f542ea6d2c3b7057c4ebb7d0b69167a44b8eeef3ff
@@ -14 +12 @@
-CDHash=710421ab654b828da6d1341fa487e4243d162103
+CDHash=c273a9f6c2be4c0bc4f040f542ea6d2c3b7057c4
@@ -19 +17 @@
-Internal requirements count=0 size=12
+Internal requirements=none

@cherrymui in case it's useful here is the output of jtool --sig for the same binary signed using codesign -s - test and your tool, looks like codesign does a few things differently when constructing the blobs: https://gist.github.com/rolandshoemaker/edfbde70c60d1469a1ec6e5d49f70f86

@cherrymui, I think it's because your ad-hoc CD lacks a CMS encoded signature (type 0x1000) inside the embedded signature 0xfade0cc0 superblob.  Apple's ad-hoc signature includes one with a zero-length payload.

Could you dump a C binary built by the C toolchain? I know that codesign -s - does more, but it seems the one that ld does what I did (ld64-609), only 1 blob, which I tried to do.
Anyway, if that doesn't work, I'll try to do what codesign does...

Oh interesting, yeah clang produces a very different looking blob that is similar to what you're doing:
Blob at offset: 49424 (530 bytes) is an embedded signature of 530 bytes, and 1 blobs
	Blob 0: Type: 0 @20: Code Directory (510 bytes)
		Version:     20400
		Flags:       adhoc (0x20002) (0x20002)
		CodeLimit:   0xc110
		Identifier:  a.out (0x58)
		Executable Segment: Base 0x00000000 Limit: 0x00000000 Flags: 0x00000000
		CDHash:	     5bf2a837dac0d47e78f3c5ffaac83e6a0053ba15f95865f548701068d8fd82bb (computed)
		# of Hashes: 13 code + 0 special
		Hashes @94 size: 32 Type: SHA-256
			Slot   0 (File page @0x0000):	2a7d47fb7ca6d1942ec4b598da0b48420f6773223e758f890da5bf429f0b019c (OK)
			Slot   1 (File page @0x1000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   2 (File page @0x2000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   3 (File page @0x3000):	d20ec350a5e965e5c23c0382ba34e2e3704386f809bbe35979d2156520b6c904 (OK)
			Slot   4 (File page @0x4000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   5 (File page @0x5000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   6 (File page @0x6000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   7 (File page @0x7000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   8 (File page @0x8000):	7475d6c89d2de31db7ebf77586309b2ea6cf8b157fec1534bce1583f4b5cdc7f (OK)
			Slot   9 (File page @0x9000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot  10 (File page @0xa000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot  11 (File page @0xb000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot  12 (File page @0xc000):	5d7445f5164527896faba9cbb23606bb16e0b16d4e88625101f1c7aa5f992c3b (OK)

Sorry about the 0x1000 blob issue; that isn't required after all.
Comparing the output of your codesign.go tool to mine, on the same executable, the only difference I can see is that your code signature load command data size isn't padded to the 4k page size.

I've modified your codesign.go code to pad sz on line 201 -- and it works. The signed executable runs on the M1.

Ouch... codesign.go isn't fixed by padding the LC_CODE_SIGNATURE to 4k; I've deleted my previous post as a result.
It turned out that by running my adhoc signed version of the same binary, it somehow "enabled" running of the same binary that was processed by codesign.go.  (Perhaps the kernel keeps a cache of validated CD hashes?)  After a machine reboot, the codesign.go signed executable stopped working.
I'll try and dig a bit deeper into what the differences are between my tool and codesign.go, with reboots to make sure this time.

OK, after some digging, I think I found the problem.  You are using version 0x20400 of the CodeDirectory (CD), which has the following fields (page 8):

exec segment base
exec segment limit
exec segment flags

These 64-bit fields need to match the __TEXT segment found in the LC_SEGMENT_64, which you would need to parse from the Mach-O header.  You are setting them all to zero, which doesn't match the __TEXT segment.
Since the adhoc signature is as simple as you can get, and you don't need the features of the most recent CD versions, an easier fix here is to just use an earlier version of the CD (see the diff below).  If you use version 0x20001, you don't need the above fields, so you don't need to look for the __TEXT segment, and then your CD passes validation.
I have verified this after rebooting my M1, to be sure. :)

codesign.go diff
$ diff codesign.go codesign.orig.go
75,89c75,89
<       magic   uint32 // magic number (CSMAGIC_CODEDIRECTORY)
<       length  uint32 // total length of CodeDirectory blob
<       version uint32 // compatibility version
<       //
<       flags         uint32 // setup and mode flags
<       hashOffset    uint32 // offset of hash slot element at index zero
<       identOffset   uint32 // offset of identifier string
<       nSpecialSlots uint32 // number of special hash slots
<       nCodeSlots    uint32 // number of ordinary (code) hash slots
<       codeLimit     uint32 // limit to main image signature range
<       hashSize      uint8  // size of each hash in bytes
<       hashType      uint8  // type of hash (cdHashType* constants)
<       _pad1         uint8  // platform
<       pageSize      uint8  // log2(page size in bytes); 0 => infinite
<       _pad2         uint32 // unused (must be zero)
---
>       magic         uint32     // magic number (CSMAGIC_CODEDIRECTORY)
>       length        uint32     // total length of CodeDirectory blob
>       version       uint32     // compatibility version
>       flags         uint32     // setup and mode flags
>       hashOffset    uint32     // offset of hash slot element at index zero
>       identOffset   uint32     // offset of identifier string
>       nSpecialSlots uint32     // number of special hash slots
>       nCodeSlots    uint32     // number of ordinary (code) hash slots
>       codeLimit     uint32     // limit to main image signature range
>       hashSize      uint8      // size of each hash in bytes
>       hashType      uint8      // type of hash (cdHashType* constants)
>       _pad1         uint8      // unused (must be zero)
>       pageSize      uint8      // log2(page size in bytes); 0 => infinite
>       _pad2         uint32     // unused (must be zero)
>       _pad3         [11]uint32 // 64-bit extension?
107a108,110
>       for _, x := range c._pad3 {
>               out = put32be(out, x)
>       }
130d132
< func put64be(b []byte, x uint64) []byte { binary.BigEndian.PutUint64(b, x); return b[8:] }
262c264
<               version:     0x20001,
---
>               version:     0x20400,

Just a heads up that I've also managed to implement adhoc code signing over in Zig's self-hosted compiler. The binaries generated with the self-hosted now are code signed out of the box, and are not greeted with SIGKILL by the kernel (tested on the DTK). If it's of any help, here's the sources: zig/src/link/MachO/CodeSignature.zig. I'll also be more than happy to help out in any way I can if anyone needs any help with this! Just lemme know ;-)

I don't understand this. What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

Their system has long had a mechanism to identify code by a detached (not embedded in the executable code module) ad-hoc signature. This new development gets rid of the detached aspect by requiring signatures to be explicit and attached, even if ad-hoc. It’s a simplification of their system that eliminates a funny quirk in a corner of code signing that wasn’t well-understood.
Details. Excerpting:

New in macOS 11 on Macs with Apple silicon, and starting in macOS Big Sur 11 beta 6, the operating system enforces that any executable must be signed before it’s allowed to run. There isn’t a specific identity requirement for this signature: a simple ad-hoc signature is sufficient. This new behavior doesn’t change the long-established policy that our users and developers can run arbitrary code on their Macs, and is designed to simplify the execution policies on Macs with Apple silicon and enable the system to better detect code modifications.

Interesting. The C compiler on the DTK doesn't do that...

You should be getting linker-signed code if you’re using the toolchain from Xcode 12.0b4 or newer and targeting arm64, regardless of whether you run it on arm64 (including DTK) or x86_64.

It doesn't look like the linker invocation has anything special, so I assume ld just does that

It does, since Xcode 12.0b4. ld64 has two new options, -adhoc_codesign and -no_adhoc_codesign. These can be used to overcome the defaults, which are to create an ad-hoc linker-signed signature when targeting arm64, and to not do so when targeting x86_64.

Interesting... I'll see if it is possible to generate LC_CODE_SIGNATURE in the linker. If that's not possible, maybe shell out codesign.

You can do this, but codesign --sign doesn’t produce the linker-signed bit, which is somewhat valuable:
In order for this new system to adhere to existing developer expectations, it needs to be possible to codesign --sign linker output, either ad-hoc or, more interestingly, with an actual identity, according to existing workflows. If it’s already signed, codesign --sign refuses to operate on it without --force. On the other hand, if it’s signed but with the linker-signed bit, codesign --sign will happily introduce a new signature, even absent --force. (The wrinkle here is that codesign is an OS tool, not an Xcode tool, so this behavior requires codesign on macOS 11.0. Even though you can use Xcode 12.2 on macOS 10.15, you’ll still be using an older codesign in that case, and it knows nothing about the linker-signed attribute. Here’s how I dealt with that discrepancy in Chrome.)
There are other toolchain tools that modify executable modules and also need to be aware of linker-signed code and treat it differently, such as strip and install_name_tool, although possibly neither of these are very relevant to Go linker output.
It would be best if mac-arm64 Mach-O output produced by any part of Go’s toolchain came with an ad-hoc linker-signed signature by default.
As some of the recent commenters above have discovered, it’s not terribly difficult to generate a usable ad-hoc signature on your own.

Re properly signing Go binaries for distribution, could we point the linker at a certificate? Or must we copy to a Mac and run codesign?

Thanks for the information!

It would be best if mac-arm64 Mach-O output produced by any part of Go’s toolchain came with an ad-hoc linker-signed signature by default.

Yeah, that's the plan.

Thanks all above for trying it out and providing informations, especially @jpap !
Just to be sure, could someone confirm that the updated version of codesign.go on https://go-review.googlesource.com/c/scratch/+/271866 works on M1? Thanks.
It is interesting that ld64 uses version 0x20400 with all later fields as zero and it worked (e.g. https://github.com/golang/go/issues/42684#issuecomment-730874626). I may dig a little more.

It is interesting that ld64 uses version 0x20400 with all later fields as zero and it worked (e.g. #42684 (comment)). I may dig a little more.

That's a known bug in jtool; see the example below.

Details
$ codesign -d -vvvvv hello
Executable=hello
Identifier=hello
Format=Mach-O thin (x86_64)
CodeDirectory v=20400 size=510 flags=0x2(adhoc) hashes=13+0 location=embedded
VersionPlatform=1
VersionMin=720896
VersionSDK=720896
Hash type=sha256 size=32
CandidateCDHash sha256=b6417d252c7f06bd3e7060f41616a965c4f0b8ae
CandidateCDHashFull sha256=b6417d252c7f06bd3e7060f41616a965c4f0b8ae800389a82c5a28a89ba26327
Hash choices=sha256
CMSDigest=b6417d252c7f06bd3e7060f41616a965c4f0b8ae800389a82c5a28a89ba26327
CMSDigestType=2
Executable Segment base=0
Executable Segment limit=16384
Executable Segment flags=0x1
Page size=4096
CDHash=b6417d252c7f06bd3e7060f41616a965c4f0b8ae
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
$
$
$ jtool -v --sig hello
Blob at offset: 49472 (4088 bytes) is an embedded signature of 530 bytes, and 1 blobs
	Blob 0: Type: 0 @20: Code Directory (510 bytes)
		Version:     20400
		Flags:       adhoc (0x2) (0x2)
		CodeLimit:   0xc140
		Identifier:  hello (0x58)
		Executable Segment: Base 0x00000000 Limit: 0x00000000 Flags: 0x00000000
		CDHash:	     b6417d252c7f06bd3e7060f41616a965c4f0b8ae800389a82c5a28a89ba26327 (computed)
		# of Hashes: 13 code + 0 special
		Hashes @94 size: 32 Type: SHA-256
			Slot   0 (File page @0x0000):	73d0ed40a765c0efcbf4b4b1cadb65743305224e6b4c174b2c25e11248ffa524 (OK)
			Slot   1 (File page @0x1000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   2 (File page @0x2000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   3 (File page @0x3000):	e2357f1795721768662726f36224f36900802501bd8d93053f2595b3b99f0c79 (OK)
			Slot   4 (File page @0x4000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   5 (File page @0x5000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   6 (File page @0x6000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   7 (File page @0x7000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot   8 (File page @0x8000):	4586671c3ba8ead84432c824ab30f5995c0f60faa6468212836809a2ede0babb (OK)
			Slot   9 (File page @0x9000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot  10 (File page @0xa000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot  11 (File page @0xb000):	ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7 (OK)
			Slot  12 (File page @0xc000):	41b367d82b13868e6ceeb3fb76e7c1c57cd9a233a4550f4074a323bcffc2c6ec (OK)
$

Re properly signing Go binaries for distribution, could we point the linker at a certificate? Or must we copy to a Mac and run codesign?

The linker-signed signatures are ad-hoc, and aside from that one linker-signed bit, they’re essentially the same as the more traditional non-linker-signed ad-hoc signatures (codesign --sign=-). An ad-hoc signature isn’t really a signature in the strict cryptographic sense, it’s effectively just a digest.
Signing with an actual certificate (or, as Apple says, “identity”) is quite a bit different. This adds a CMS message signing (this time, in the cryptographic sense) the digest as proof of who signed the object.
Certainly, golang has strong crypto support, and it may well be possible to build a suitable signature without too much additional effort, but I think that doing it here as part of the linker output would be misguided. If someone’s going to go to the trouble of writing a drop-in for Apple codesign --sign in golang, I’d certainly hope it would be available as a general-purpose tool usable on a variety of Mach-O files, not just those produced by the Go linker. (Although, once you’re at that point, sure, no reason not to give the linker the ability to do it too. Doesn’t really matter.)
As far as “signing for distribution” goes, that needs to be considered in the context of the larger packaging scheme. Notably, you’ve got a bunch of stuff to sign and notarize, then you’ve got a .pkg to build, and then you’ve got to sign, notarize, and notarization-staple that too. You might find drop-in replacements for Apple tools for some of these steps, but there are an awful lot of Apple tools to replace if you want to totally remove macOS from the official-release pipeline.

Just to be sure, could someone confirm that the updated version of codesign.go on https://go-review.googlesource.com/c/scratch/+/271866 works on M1? Thanks.

@cherrymui I tried again with Patch Set 2, it did not work on a MacBook Air with M1. Let me know if I can provide specific information or remote access to it.

@dmitshur what about PS3? I just updated the CL again.
@jpap I hex-dumped a signature generated by ld64, and those fields are actually 0.
Looking again I think I may need to round up the __LINKEDIT segment's VMSize, which I didn't.

@jpap I hex-dumped a signature generated by ld64, and those fields are actually 0.

That's interesting -- here's what I get when I build a simple program on the M1:
$ cat test.c
#include <stdio.h>

int main() {
        printf("hello world\n");
        return 0;
}

$ clang -arch arm64 -o test test.c
$ file test
test: Mach-O 64-bit executable arm64
$ codesign -d -vvvv test
Executable=/Users/jpap/test
Identifier=test
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=509 flags=0x20002(adhoc,linker-signed) hashes=13+0 location=embedded
VersionPlatform=1
VersionMin=720896
VersionSDK=720896
Hash type=sha256 size=32
CandidateCDHash sha256=f54d00ba0c20977c1a2a14e129ca6038500aff97
CandidateCDHashFull sha256=f54d00ba0c20977c1a2a14e129ca6038500aff971a0ef98f1ab927e2fa3fe272
Hash choices=sha256
CMSDigest=f54d00ba0c20977c1a2a14e129ca6038500aff971a0ef98f1ab927e2fa3fe272
CMSDigestType=2
Executable Segment base=0
Executable Segment limit=16384
Executable Segment flags=0x1
Page size=4096
CDHash=f54d00ba0c20977c1a2a14e129ca6038500aff97
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
$

Interesting. Maybe due to ld64 version? I have ld64-609. It generates the signature but the "Executable Segment" fields are 0. (And ld64-607.2 on another machine doesn't sign at all.)

@dmitshur what about PS3? I just updated the CL again.

@cherrymui Same outcome with PS3.

Interesting. Maybe due to ld64 version? I have ld64-609. It generates the signature but the "Executable Segment" fields are 0. (And ld64-607.2 on another machine doesn't sign at all.)
Apparently it's not hard to set those fields. I could just do that, if that's what matters.

I've got 609.7.

Details
$ ld -version_details
{
	"version": "609.7",
	"architectures": [
		"armv6",
		"armv7",
		"armv7s",
		"arm64",
		"arm64e",
		"arm64_32",
		"i386",
		"x86_64",
		"x86_64h",
		"armv6m",
		"armv7k",
		"armv7m",
		"armv7em"
	],
	"lto": {
		"runtime_api_version": 27,
		"static_api_version": 27,
		"version_string": "LLVM version 12.0.0, (clang-1<IP_ADDRESS>)"
	},
	"tapi": {
		"version": "12.0.0",
		"version_string": "Apple TAPI version 12.0.0 (tapi-1<IP_ADDRESS>)"
	}
}
$

Just to be sure, could someone confirm that the updated version of codesign.go on https://go-review.googlesource.com/c/scratch/+/271866 works on M1? Thanks.

@cherrymui I tried again with Patch Set 2, it did not work on a MacBook Air with M1. Let me know if I can provide specific information or remote access to it that would help you.

It works on my M1 (MacBook Air also)... @dmitshur, can you try a reboot before executing the adhoc signed binary?

 I will try again in a bit.

I tried Cherry’s codesign again at Patch Sets 1 through 3. If I didn’t make a mistake, it seems they all work if I compress the binary and uncompress it, but not before that. The binaries were identical in all 3 cases before and after, so maybe this is related to the inode change?

Interesting... Does it work if you simply cp it? And don't try to run the unsigned version first?
If you build a fresh binary, don't try to execute it, sign it, then execute, does it work?

If you build a fresh binary, don't try to execute it, sign it, then execute, does it work?

In my previous tests, I’ve always done go build, then sign, then execute. I may not have cleaned build cache though.

Does it work if you simply cp it?

 If someone wants to give it a try, that would be great. Thanks!

I may not have cleaned build cache though.

Yeah, you may want to clean the build cache, for now, and delete the output binary before running go build. Because signing doesn't change the buildid, the go command may decide you already have the output binary, which is actually not correct.

Ok. I tried PS 4 with a modified program:

# modify p.go to be a new program, one I've definitely never built before

$ GOOS=darwin GOARCH=arm64 go build -o ps4 p.go
$ /tmp/cherrysign_ps4 ./ps4  # cherrysign_ps4 is x/scratch/cherry/codesign@d99345806828aac41e55a447c38157e0cdea1c6a
$ ./ps4
zsh: killed     ./ps4
$ cp ps4 ps4c
$ ./ps4c
hello, take 4

Thanks for trying out!
Interesting. Maybe we do need to always make a copy... I'll do that in the real thing.

Interesting. Maybe we do need to always make a copy... I'll do that in the real thing.

I'm not convinced you need to make a copy -- so long as you don't run it before signing it.  I really doubt the kernel is reading/caching all files marked executable when they appear on the filesystem and before they're executed. :)
I got tripped up with some kind of kernel caching last night -- which is why I kept rebooting to be sure, and had the go cache -clean and rm -f pre-build steps in my test script.  Fortunately the M1 boots super-fast!  (In hindsight, if we run the different {no-signature, apple codesign, cherry's} tests on different-enough binaries it might've helped...)
@dmitshur, does a reboot clear things up for you?

@jpap Yes, rebooting has an effect. I tried building+signing+execing a novel binary to a non-temporary location, it was killed. After rebooting (and doing nothing else), the same binary started to run okay.
However, when I built another novel binary, signed it, and execed, it was still killed.

I have written a test program to generate a random Go or C "hello X", where X is a random number.  The test program builds the random program, adds an adhoc code signature, and then runs it.  If you want to try it, make sure you edit generate.go to set goBinPath to where you have the darwin/arm64 Go compiler installed.
On the M1:

the C program always runs, except if you skip codesign (as you'd expect).
the Go program only runs when you use Apple's codesign tool.

if you link the Go program with Apple's linker, it still won't run: GOOS=darwin GOARCH=arm64 go build -o test -ldflags '-linkmode=external -extldflags "-arch arm64"' ./program.go
before you run the executable test above, and you sign it with Apple's codesign tool, it runs.
it makes no difference if an empty requirement set, and/or empty CMS signature blob is added to the CD.
it makes no difference if you sleep between signing and execution.
it looks like Apple's codesign tool adds a magic hash value to the filename when it sets the CD identifier.  If you sign using codesign.go with the same magic hash, it runs! (I was careful to generate an unsigned random program that I did not run: signed it using Apple's tool, noted the hash, did not run it, then signed a different copy using codesign.go (CD ver 0x20001) with the hash appended to the filename-based identifier, then ran the latter signed binary and it ran.)
The identifier-suffix hash appears to be tied to the contents of the executable: it looks like a sha1, but isn't a sha1 of the CDHash, the unsigned file contents, the filename, or the full path to the file (including the filename or not).  I didn't see any hints in Apple's open source dump on this one, unfortunately.  A random value didn't work either. :)

I have no idea what's special about the Go vs. C program and why the C program always runs despite not having the hash suffix in the CD identifier.
My guess at this stage is that if you can work out how that hash is computed, you're good to go with a simple version 0x20001 adhoc CD.

The identifier-suffix hash appears to be tied to the contents of the executable: it looks like a sha1, but isn't a sha1 of the CDHash, the unsigned file contents, the filename, or the full path to the file (including the filename or not). I didn't see any hints in Apple's open source dump on this one, unfortunately. A random value didn't work either. :)

If the Mach-O module has an LC_UUID load command, it’s the UUID value from uuid_command::uuid, prefixed with 0x55554944 (big-endian 'UUID') to pad the 16-byte UUID to 20 bytes, the length of a SHA-1 digest.
If there’s no LC_UUID load command, it’s a SHA-1 digest of the Mach-O header and load commands. golang’s linker doesn’t create an LC_UUID load command, so this is what you’d want in the near term.
You’ll find this in 10.15.6 Security-59306.140.5/OSX/libsecurity_codesigning/lib/machorep.cpp Security::CodeSigning::MachORep::identificationFor (line 225).
(LC_UUID can be rather valuable, so perhaps you’d consider adding it, although I agree that’s a distinct concern.)
However, please note that while the extra data appended to the identifier is something that codesign --sign may add, it’s not something that ld64 adds when creating a linker-signed executable! It is not a requirement to add this to an identifier. In fact, codesign --sign doesn’t always add this extra data either. Here’s what it tries, in order:

The --identifier option given to codesign --sign on the command line.
The CFBundleIdentifier in the associated Info.plist.
The module’s name, with a hyphen and the extra data discussed above appended.

ld64 linker-signed output always uses just the module’s name.
Rather than writing the linker-signed signature from a separate tool post-link, does it help to write it directly from the linker? For these limited-purpose linker-signed signatures, this is probably the best way to go.

Even if we solve the link problem the signature is modified by build id or the signature invalidates the build id marking the build as stale:
go tool dist: FAILED: /Users/faye/src/goroot/pkg/tool/darwin_arm64/go_bootstrap list -gcflags=all= -ldflags=all= -f={{if .Stale}}	STALE {{.ImportPath}}: {{.StaleReason}}{{end}} cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys: exit status 1
The invalidation kicks in at Building toolchain 3 (unless you configure for debug)

Even if we solve the link problem the signature is modified by build id or the signature invalidates the build id marking the build as stale:

That’s right. Since Apple’s code signature necessarily covers the golang build ID, the golang build ID can’t also cover the code signature. Unfortunately, this may mean that the way that the golang build ID is calculated needs to change for Mach-O. It should be possible to not invalidate existing build IDs from Mach-O files that don’t have an LC_CODE_SIGNATURE, so if there’s any concern about shifting build IDs, the impact should actually be minimal.
When and how is the golang build ID written to the executable? If it’s maintained by a separate tool, then a possible solution would be for that tool to also recompute and write a new ad-hoc linker-signed signature, but only when dealing with an object that’s already linker-signed.
This is exactly how Apple tools that modify linker output now work (since Xcode 12.2b2): strip and install_name_tool are aware that what they do will invalidate a code signature, so if they’re operating on a linker-signed Mach-O file, they’ll generate a new linker-signed signature. (A new library in the toolchain, libcodedirectory.dylib, is responsible for this. Previously, these tools were release-noted as broken for arm64 and responsible for many a poisoned inode.)
If the golang build ID is written by a separate tool post-link, then it’s probably responsible for what @jpap noted earlier today:

if you link the Go program with Apple's linker, it still won't run: GOOS=darwin GOARCH=arm64 go build -o test -ldflags '-linkmode=external -extldflags "-arch arm64"' ./program.go

When you see something like this, try codesign --verify and codesign --verify --strict=all. I suspect that the signature has been invalidated, perhaps by the introduction of a golang build ID as @fayep says. When you have a code signing SIGKILL, you can also check the system log for more information. At the very least, it’ll indicate which page xnu found offensive.

Do you sign the individual binaries, or do you also/instead need to sign the container?
(I don't think there's any place in the format to sign the container, so I suspect the former.)

Each individual architecture “slice” has its own signature covering only that slice.
You can merge different thin Mach-O objects, already validly signed, into a single fat (universal) file, and it remains validly signed.  It’s a very simple archive format and is trivial to work with without lipo, too.
codesign --sign will also (re-)sign all architectures in a fat file.

If it’s maintained by a separate tool, then a possible solution would be for that tool to also recompute and write a new ad-hoc linker-signed signature

My plan is https://github.com/golang/go/issues/42684#issuecomment-730495710 . Basically what you said, generate the signature after stamping the buildid. Maybe we could do it both before and after stamping the buildid, so plain go tool link works.
I'm currently doing it as a separate tool just to make sure I understand the signing algorithm.

It is not a requirement to add this to an identifier.

Does it affect kernel's caching in some magical way? Or, what actually matters? The inode? When the kernel caches the signature's validity, at load time, or when then file is written (closed)? Does it require the binary to have a valid signature at every point of time? Thanks.

(LC_UUID can be rather valuable, so perhaps you’d consider adding it, although I agree that’s a distinct concern.)

Could you provide some information about LC_UUID, how it is useful and how it is generated? I'm happy to add it to the Go linker as long as I understand it. Thanks.

if you link the Go program with Apple's linker, it still won't run: GOOS=darwin GOARCH=arm64 go build -o test -ldflags '-linkmode=external -extldflags "-arch arm64"' ./program.go

This wouldn't be expected to work, because the Go toolchain modifies external-linker emitted binary in a few ways, stamping the buildid is one, and there are others.

Buildid seems to be written into the binary by cmd/buildid.  Perhaps on Darwin (This is all a platform feature which is disabled in intel Macs, but will be handled just fine by them, right?) it could invoke the signer passing the UUID maybe the LC_UUID if that's what it is for?  And return the same on reading.

tl;dr: The basic ver 0x20001 CD is fine, with no UUID, no CD identifier hash suffix, so long as you copy the file after it is signed.
Thank you @markmentovai for the super-useful and detailed information.  I can confirm that the UUID and CD identifier hash suffix doesn't affect the signature validation.  I've updated my gist to include the UUID generation, in case you want to play with it.
There's something fishy about the way the final executable is run...

Codesign with codesign.go...

Codesign with codesign.uuid.go...
Run...
hello 436445343
$ ./test-436445343
zsh: killed     ./test-436445343
$ cp test-436445343 test-yyy
$ ./test-yyy
hello 436445343
$

$ arch
arm64
$

I don't understand why the binary runs within the generate.go program, but not outside it, unless you copy the file.  Is it because generate.go is running under Rosetta?  It gets even more weird with no signature at all, where the unsigned binary runs when launched form the Rosetta process, but otherwise is killed.
$ go run ./generate.go -work -cstool none
Using gc: /Users/jpap/Development/go/src/github.com/golang/go/bin/go
Executable: test-120056221
Building Go program...
Skipping codesign...
Run...
hello 120056221
$ ./test-120056221
zsh: killed     ./test-120056221
$ codesign -d -vvv test-120056221
test-120056221: code object is not signed at all
$ cp test-120056221 test-aaa
$ ./test-aaa
zsh: killed     ./test-aaa
$ file test-120056221
test-120056221: Mach-O 64-bit executable arm64
$

If you don't run the binary under Rosetta, you still need to make the copy.  So it looks like @dmitshur is right about that... would love to know why!
$ go run ./generate.go -work -run=false
Using gc: /Users/jpap/Development/go/src/github.com/golang/go/bin/go
Executable: test-118293903
Building Go program...
Codesign with codesign.go...
$ ./test-118293903
zsh: killed     ./test-118293903
$ cp test-118293903 test-bbb
$ ./test-bbb
hello 118293903
$

@cherrymui https://github.com/golang/go/issues/42684#issuecomment-730495710:

Stamp the buildid. The buildid would be made to be insensitive to the LC_CODE_SIGNATURE blob. (but can be sensitive to the load command itself, which should not change)

I suggest making it sensitive to the LC_CODE_SIGNATURE load command, but skipping or blanking the dataoff and datasize fields and, of course as you say, the data that they point to. Rationale: try to keep the golang build ID insensitive to the Mach-O file being re-signed. While a simple replacement ad-hoc signature might not alter dataoff or datasize, a more complex re-signing could (if the identifier, requirements, entitlements, etc., change, or even if a different-version signature is used, or if the replacement signature is not ad-hoc and carries a CMS signature.) That said, this may wind up being hopeless anyway if the Mach-O file needs to grow to accommodate the new signature.
@cherrymui https://github.com/golang/go/issues/42684#issuecomment-731616123:

Basically what you said, generate the signature after stamping the buildid. Maybe we could do it both before and after stamping the buildid, so plain go tool link works.

Yes, I’d favor striving to always have a legitimate linker-signed signature in place following any tool execution.

It is not a requirement to add this to an identifier.

Does it affect kernel's caching in some magical way? Or, what actually matters? The inode? When the kernel caches the signature's validity, at load time, or when then file is written (closed)? Does it require the binary to have a valid signature at every point of time? Thanks.

The identifier doesn’t enter xnu’s view of trust at all. There are higher-level user-space operations that do consider the identifier, but even then, it’s just an opaque string. For linker-signed purposes, I suggest setting the identifier to the filename of the Mach-O output file (just the basename, not the whole path), without any extra digest or other appendage.
In xnu, code signing is implemented within the VM system, as part of the unified buffer cache, and is attached at the vnode level. Pages are validated at fault time as they are brought in by the pager. 10.15.6 xnu-6153.141.1/osfmk/vm/vm_fault.c vm_fault_enter is the top of the funnel for most of this. You can trace that through to vm_page_validate_cs_mapped and the actual validation in bsd/kern/ubc_subr.c cs_validate_range.
All of this is to say that if you’re interacting with a file via mmap, you risk getting sniffed out. And golang’s linker does write its output to an mmaped region. There you go!
Is this a horrible bug? Absolutely, I think so! But I also think that it’s not an easy one to fix, because it’s based on a central design flaw. In cases where I’ve found legitimately terrible behaviors, like strip creating broken files that codesign wouldn’t be able to fix, Apple has responded rapidly with tool fixes, but the underlying architecture has mostly remained the same. I don’t love that it’s difficult or impossible to mmap something, even to handle as data, just because the kernel happened to sniff out something that it thought was a signature. Bugs abound in this area, I’m afraid.
So what can you do?

You can wait for your pages to leave the UBC, as @dmitshur found at https://github.com/golang/go/issues/42684#issuecomment-731511653. If you don’t want to wait, purge can help. While that’s great for theory and testing, it’s not great for practice or production…or not requiring something to run as root. (purge is faster than a reboot, though, even on one of these new machines.)
As you’ve discovered, you can put your file contents into a new vnode (via a new inode), as long as you write the file with something like write and not writing to an mmaped region. cp or equivalent is fine for this, but the whole thing feels kind of brutish and I don’t love this solution.
You can avoid writing your file by writing to an mmaped region in the first place. Instead, issue writes to a file descriptor, or similar.
One of my favorites: call msync(…, MS_INVALIDATE) on the mmaped region, asking xnu to throw away what it knows about the vnode. You can even use this to “save” a broken vnode from an entirely different process by opening the file,mmaping it, and then calling msync.

ld64 manages to produce output that runs without getting killed (golang post-link modifications like the build ID notwithstanding), so what’s it doing? The most current source available is Xcode 11.3.1 ld64-530/src/ld/OutputFile.cpp ld::tool:: OutputFile::writeOutputFile, which is just a straight mmap in most common cases. But that’s old, and the new truth is that in Xcode 12.2 ld64-609.7, only on arm64, they’ve actually switched to a write approach in every case.

Could you provide some information about LC_UUID, how it is useful and how it is generated? I'm happy to add it to the Go linker as long as I understand it. Thanks.

This is going off in an unrelated direction, because LC_UUID isn’t really relevant to any of this, other than that it factors in to how codesign --sign creates a default identifier. Briefly: it’s the Mach-O equivalent of ELF’s .note.gnu.build-id, a way to uniquely identify a Mach-O file, which is typically used to correlate it with its out-of-band debug info (.dSYM). In Crashpad (that’s one of my things), I use LC_UUID to tie modules loaded on the client back to the original files and their debug info on the server. I think it would be good for the golang linker to write this load command, but it won’t help here at all, and it’s a totally different issue.
@jpap https://github.com/golang/go/issues/42684#issuecomment-731654290:

tl;dr: The basic ver 0x20001 CD is fine, with no UUID, no CD identifier hash suffix, so long as you copy the file after it is signed.

Sure, but I suggest sticking with 0x20400 for a linker-signed signature. This is what Xcode 12.2 ld64 does, and that should be treated as the baseline linker for mac-arm64. Apple would be within their rights to require this as a minimum on mac-arm64, even if they aren’t doing so now, and even though it would be rude of them to break golang output in existence at the time they’d make such a change, I can’t say that this would be something that they’d catch early in testing, and I know that this is something that’s happened before.
What about the zeroes vs. real execSegBase, execSegLimit, and execSegFlags fields? 10.15 codesign --sign leaves these at zero when targeting macOS, but sets them properly for all of Apple’s other operating systems (iOS, etc.) 10.15.6 Security-59306.140.5/OSX/libsecurity_codesigning/lib/machorep.cpp Security::CodeSigning::MachORep::needsExecSeg This has changed in 11.0 (and in Xcode 12.2 ld64 writing linker-signed signatures), which always set them to valid values, even for macOS. While it’s possible and supported to codesign --sign targeting mac-arm64 from a 10.15 host, linker-signed output for mac-arm64 will never omit valid values for these fields, and it’s incredibly easy to set them properly.
Definitely no LC_UUID is required, and the identifier ought not be considered for any practical purpose, particularly for ad-hoc signatures.

There's something fishy about the way the final executable is run...

Yes, invoking Rosetta is basically an end run around the new code signature enforcement that Apple is trying hard to impose for arm64. In order to maintain maximum x86_64 compatibility, some of the new things that come with arm64 aren’t available in the same way for x86_64. One of these things is the requirement that all code be signed. Since xnu is responsible for enforcement, it’s natural that some of these compatibility concessions were made in xnu. In this case, the signature check at execution time is fired off from within 10.15.6 xnu-6153.141.1/bsd/kern/mach_loader.c parse_machfile and load_code_signature. This is 10.15.6 source, and I haven’t seen 11.0 xnu source yet to confirm exactly how this is hooked up for arm64, but it’s perfectly reasonable that when the parent is running under Rosetta, the initial signature check of the new child executable would be conducted under Rosetta rules, regardless of whether the new executable is arm64 or x86_64. That’d make this a bug, and it’s worth a feedback report or, even better, a product-security contact.
mark@arm-and-hammer zsh% /tmp/no_signature; echo $?
zsh: killed     /tmp/no_signature
137
mark@arm-and-hammer zsh% arch -arch x86_64 sh -c /tmp/no_signature; echo $?
0

Bonus information: this form of code signature enforcement isn’t strictly new, it’s been available on x86_64 for a long time as kill semantics (an option to codesign --sign --options) and is also enabled under the hardened runtime (option runtime). But because you won’t find linker output bearing a signature with that option, you won’t really experience the bug on x86_64. If you’re interested, you can mimic the bug on x86_64, even on macOS 10.15, by signing an executable with codesign --sign=- --options=kill and then having something copy it by writing it to a file-backed mmaped region. You’ll find that you can’t run the copy, the same behavior that we now see on arm64 by default.

If you don't run the binary under Rosetta, you still need to make the copy. So it looks like @dmitshur is right about that... would love to know why!

All answered above!
And I know it’s just for testing, but having been down this road recently: don’t rely on the presence of Rosetta, it’s not part of the base OS and may not be there at all!

Phenomenal -- thank you for all of the detail @markmentovai, I think I will sleep better tonight. ❤

I suggest making it sensitive to the LC_CODE_SIGNATURE load command, but skipping or blanking the dataoff and datasize fields and, of course as you say, the data that they point to. Rationale: try to keep the golang build ID insensitive to the Mach-O file being re-signed. While a simple replacement ad-hoc signature might not alter dataoff or datasize, a more complex re-signing could (if the identifier, requirements, entitlements, etc., change, or even if a different-version signature is used, or if the replacement signature is not ad-hoc and carries a CMS signature.) That said, this may wind up being hopeless anyway if the Mach-O file needs to grow to accommodate the new signature.

The executable will definitely grow when using a CMS encoded signature (which usually also come with a set of requirements and entitlements on iOS), unless you include enough padding up front.  Having a go linker option to request padding could be useful if it means a go build can go faster on a cache hit.  (This would be analogous to the codesign_allocate step that's part of an Xcode build, where you can opt for an over-allocation.)  Eating an extra link on cache miss isn't too bad, but gets worse with the external linker which is a lot slower.  Either way, it's not a dealbreaker.

Basically what you said, generate the signature after stamping the buildid. Maybe we could do it both before and after stamping the buildid, so plain go tool link works.

Yes, I’d favor striving to always have a legitimate linker-signed signature in place following any tool execution.

I'm also in favor of this, so long as there is an opt-out flag.  It saves having a custom toolchain that re-signs the binary with a CMS encoded signature from removing the existing signature first: not a big deal, but it would save me some work. ;)  Apple's linker supports this with the -no_adhoc_codesign flag; it would be nice for the go linker to have a similar option.

The CLs above let the toolchain emit signed binaries. Could someone give it a try and see if it works? If not, does it work after copying the binary? (I wrote them yesterday without seeing @markmentovai 's comment. I'll play with the msync. Thanks :)
The CLs can be checked out by doing
git fetch https://go.googlesource.com/go refs/changes/57/272257/1 && git checkout FETCH_HEAD

and rebuild the toolchain.
Thanks!

@markmentovai thanks very much for the detailed reply!

Yes, I’d favor striving to always have a legitimate linker-signed signature in place following any tool execution.

Yes, this is what it does with the current version of the CLs.

Sure, but I suggest sticking with 0x20400 for a linker-signed signature.

and this as well :)

skipping or blanking the dataoff and datasize fields

Yeah, sounds like a good idea. If I understand it correctly, it probably rarely matters, though. Unless one explicitly does go tool buildid -w after a non-ad-hoc signing, or non-ad-hoc signs a toolchain binary (compiler/linker/assemlber/etc.).

LC_UUID isn’t really relevant to any of this

Yeah. Just asked while you are here :) Thanks.

git fetch https://go.googlesource.com/go refs/changes/57/272257/1 && git checkout FETCH_HEAD

I bootstrapped with this source successfully from x86_64 to arm64 creating signed binaries.
The apps behaved as though they were unsigned.  I checked the signatures and they were reported good, so based on the thread and experience, I copied the toolchain tree to another directory (could also have extracted the tarball) and it then compiled dist.  However dist suffered the same issue of stale inode signature cache.  It did appear successfully signed and copying dist to another location made it work.
Congrats.  We just need to move then copy the file back as part of the process and we’re done.

The binaries work only after copying them.
➜  ~ GOARCH=arm64 gotip download 272257
This will download and execute code from golang.org/cl/272257, continue? [y/n] y
Fetching CL 272257, Patch Set 1...
From https://go.googlesource.com/go
 * branch            refs/changes/57/272257/1 -> FETCH_HEAD
HEAD is now at 376d052 cmd/internal/buildid: update Mach-O code signature when rewriting buildid
Building Go cmd/dist using /usr/local/Cellar/go/1.15.5/libexec. (go1.15.5 darwin/amd64)
Building Go toolchain1 using /usr/local/Cellar/go/1.15.5/libexec.

Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, darwin/amd64.
Building packages and commands for target, darwin/arm64.
---
Installed Go for darwin/arm64 in /Users/filippo/sdk/gotip
Installed commands in /Users/filippo/sdk/gotip/bin
Success. You may now run 'gotip'!

➜  ~ /Users/filippo/sdk/gotip/bin/darwin_arm64/go version
[1]    3317 killed     /Users/filippo/sdk/gotip/bin/darwin_arm64/go version

➜  ~ codesign -vv /Users/filippo/sdk/gotip/bin/darwin_arm64/go
/Users/filippo/sdk/gotip/bin/darwin_arm64/go: valid on disk
/Users/filippo/sdk/gotip/bin/darwin_arm64/go: satisfies its Designated Requirement

➜  ~ codesign -dv /Users/filippo/sdk/gotip/bin/darwin_arm64/go
Executable=/Users/filippo/sdk/gotip/bin/darwin_arm64/go
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=104606 flags=0x20002(adhoc,linker-signed) hashes=3266+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

➜  ~ cp /Users/filippo/sdk/gotip/bin/darwin_arm64/go ~/tmp/go

➜  ~ ~/tmp/go version
go version devel +376d052 Sun Nov 22 01:08:02 2020 -0500 darwin/arm64

The CLs above let the toolchain emit signed binaries. Could someone give it a try and see if it works? If not, does it work after copying the binary? (I wrote them yesterday without seeing @markmentovai 's comment. I'll play with the msync. Thanks :)
The CLs can be checked out by doing
git fetch https://go.googlesource.com/go refs/changes/57/272257/1 && git checkout FETCH_HEAD

and rebuild the toolchain.
Thanks!

They are signed correctly, but only work after copying.

Thanks all for confirming!
Does CL https://go-review.googlesource.com/c/go/+/272258 help? Thanks!
To check out,
git fetch https://go.googlesource.com/go refs/changes/58/272258/1 && git checkout FETCH_HEAD

Winner winner, chicken dinner! Great job @cherrymui and thank you to @markmentovai and everyone who shared insights!
➜  ~ GOARCH=arm64 gotip download 272258
➜  ~ /Users/filippo/sdk/gotip/bin/darwin_arm64/go version
go version devel +7716a2f Sun Nov 22 11:44:49 2020 -0500 darwin/arm64
➜  edwards25519 git:(main) ✗ /Users/filippo/sdk/gotip/bin/darwin_arm64/go test
PASS
ok  	filippo.io/edwards25519	9.822s

Thanks all for confirming!
Does CL https://go-review.googlesource.com/c/go/+/272258 help? Thanks!
To check out,
git fetch https://go.googlesource.com/go refs/changes/58/272258/1 && git checkout FETCH_HEAD

Can also confirm that it works. Brilliant. Excellent work, you just made me an even happier M1 owner :)

Thanks all for trying out and sharing feedbacks! And thank you @markmentovai for the detailed information!

Can confirm it works and passes Go's tests.  Thanks!

You can do that all on the m1 mac by replacing the last line with
arch --x86_64 env GODEBUG=asyncpreemptoff=1 ./bootstrap.bash

Does CL https://go-review.googlesource.com/c/go/+/272258 help? Thanks!

As others confirmed above, that CL indeed makes it possible to do go run hello.go and have it work as usual.
That CL enabled me to try running all.bash, which has passed at least once:
src $ ./all.bash
Building Go cmd/dist using /Users/dmitri/go116r2. (devel +7716a2fbb7 Sun Nov 22 11:44:49 2020 -0500 darwin/arm64)
[...]

I'm seeing frequent runs that fail due to a sporadic "signal: illegal instruction". Filed #42774 that collects some of those failures, if it helps.

I can also verify that CL 272258 works.
Then GODEBUG=asyncpreemptoff=1 GOARCH=arm64 ./make.bash then I replaced $GOROOT/bin/go and $GOROOT/bin/gofmt with the binaries in the $GOROOT/bin/darwin_arm64 directory. (I then re-bootstrapped off of a copy of the built darwin/arm64 GOROOT for good measure).
Things work pretty well!

Following up:
I reported the bug encountered when writing to the executable via mmap (without msync(…, MS_INVALIDATE)) to Apple as FB8914231, with an OpenRadar copy.
Possibly of interest, an unrelated effort this weekend also resulted in the filing of FB8914243, a similar bug about signatures remaining attached to vnodes even when the file contents change.
These are longstanding and difficult-to-fix bugs, but arm64’s requirement that all code must be signed has raised visibility, and its requirement to impose “kill” semantics globally has raised severity, so I re-filed with new information.
I also reported the technique to circumvent the requirement that all arm64 code be signed as a security bug, currently embargoed but eventually a copy will be visible here.

@markmentovai I am wondering whether your FB8914231 is not a different manifestation of the same bug we have seen and reported in the past, in some cases, with Apple's own tools: https://github.com/golang/go/issues/42684#issuecomment-729543666

@fxcoudert: FB8914231 is part of what’s happening in https://github.com/Homebrew/brew/issues/9082#issuecomment-727246564. When you invoke the compiler under Rosetta (arch -arch x86_64), its children, including the linker, will also run under Rosetta if possible.
In https://github.com/golang/go/issues/42684#issuecomment-731704900, I said:

ld64 manages to produce output that runs without getting killed (golang post-link modifications like the build ID notwithstanding), so what’s it doing? The most current source available is Xcode 11.3.1 ld64-530/src/ld/OutputFile.cpp ld::tool::OutputFile::writeOutputFile, which is just a straight mmap in most common cases. But that’s old, and the new truth is that in Xcode 12.2 ld64-609.7, only on arm64, they’ve actually switched to a write approach in every case.

The operative part that I left in italics is significant here. When you run the x86_64 slice of ld64 under Rosetta, it still writes output to an mmaped region when possible, instead of using write always. The UBC doesn’t know or care whether the writes came from arm64 or translated x86_64 code, so the bug manifests as in this bug, https://github.com/golang/go/issues/42684, and FB8914231.
You may also be seeing signs of FB8914243 in there, particularly if, during testing, you’re replacing the contents of Mach-O files by recycling the inode rather than forcing a new one. (Not that there should be anything wrong with doing so.)
More generally, once you have a Mach-O image in a vnode that xnu has decided is “bad”, it’ll suffer from the immediate SIGKILL that we’ve been discussing, and attempts to re-sign it in place with codesign --sign will result in the confusing message the codesign_allocate helper tool cannot be found or used, which is not entirely accurate. There was an earlier bug in this area that we encountered during the beta period, so we didn’t speak publicly of it, but I think it’s fine to share now that the beta period is over, so I brought it into OpenRadar: FB8735191. It’s relevant because it discusses how codesign_allocate works and why codesign --sign reports that confusing message when you’re suffering from FB8914231, FB8914243, or other related bugs. In the original FB8735191 report, innocuous use of strip and install_name_tool was another vector to produce unmappable vnodes, and it was [incompletely] resolved by making those tools replace linker-signed code signatures with new ones, but notably the questionable underlying behavior of vm_fault_enter was left intact, giving rise to this new series of bugs.
If you have more questions not specific to golang, feel free to @ me in another bug report, and I’ll see what else I can answer. I don’t want to fill up golang’s bug tracker with too much unrelated noise.

You can do that all on the m1 mac by replacing the last line with
arch --x86_64 env GODEBUG=asyncpreemptoff=1 GOOS=darwin GOARCH=arm64 ./bootstrap.bash

This doesn't see true.  I'm on an M1 Mac mini, and these commands:
git clone https://go.googlesource.com/go cd go/src arch --x86_64 env GODEBUG=asyncpreemptoff=1 GOOS=darwin GOARCH=arm64 ./bootstrap.bash
give an error:
ERROR: Cannot find /Users/jim/go1.4/bin/go.

You do need an x86_64 darwin golang to cross build for arm64. It is easiest
at this point to get that and then use gotip to build master.
On Thu, Dec 17, 2020 at 12:31 PM jimtut<EMAIL_ADDRESS>wrote:

You can do that all on the m1 mac by replacing the last line with
arch --x86_64 env GODEBUG=asyncpreemptoff=1 GOOS=darwin GOARCH=arm64
./bootstrap.bash
This doesn't see true. I'm on an M1 Mac mini, and these commands:
git clone https://go.googlesource.com/go cd go/src arch --x86_64 env
GODEBUG=asyncpreemptoff=1 GOOS=darwin GOARCH=arm64 ./bootstrap.bash
give an error:
ERROR: Cannot find /Users/jim/go1.4/bin/go.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/42684#issuecomment-747684693, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEKAP2BPXHPN7K3DUYJ2X3SVJTCHANCNFSM4TZJN5QA
.

You can now download a native macOS ARM64 binary release from https://golang.org/dl/#go1.16beta1, as it was released today. Look for go1.16beta1.darwin-arm64.pkg or go1.16beta1.darwin-arm64.tar.gz. Then you can use Go 1.16 Beta 1 itself, or use it to build a newer gotip version.


Archived data from an ozone sensor survey

Hey
Is it possible to recover complete data from ECC ozone sensors from several weeks or several months ago?
Is this data archived on the receiving station terminals or on the anyside?

Should be available in S3

http://sondehub-history.s3-website-us-east-1.amazonaws.com/#date/

If you know the serial number of the radiosonde associated with the launch, then the easiest way to dump the data is using pysondehub: https://github.com/projecthorus/pysondehub?tab=readme-ov-file#downloading-archived-radiosonde-telemetry-data
Note that we only decode the XDATA on the tracker website frontend, so it'll just appear in the telemetry as a hex string.


remove all JS relating files

they are all moved to js-cardano-wasm

@darko-mijic @DominikGuzei something to be aware of, we've moved the js to its own repo, so we can simplify the integration, testing of each parts


Why do multiple ES6 imports compile to multiple requires / define() deps with suffixed names?

I didn't find any existing issue for this, but why does
import { foo } from "a";
foo();
import { bar } from "a";
bar();

compile to
define(["require", "exports", "a", "a"], function (require, exports, a_1, a_2) {
    a_1.foo();
    a_2.bar();
});

instead of
define(["require", "exports", "a"], function (require, exports, a) {
    a.foo();
    a.bar();
});

and similar for cjs output? The latter is how one would write the JS, and does away with the _1 _2 names as well.

Of course in this example they can be combined. My original code has a mix of import-from and export-from.
import * as set from "./utility/set";
export { Set } from "./utility/set";

// Do things with set

I could change that to
import * as set from "./utility/set";
const Set = set.Set;
export { Set };

// Do things with set

but even a single import still has the _1 name in the generated JS, which is what I'm really curious about.

I think we'd be willing to take a pull request on this one. Any implementers should be careful to test the case where one import is elided but the other is not.

@DanielRosenwasser What do you mean exactly? Can you tell me which one is correct?
import { foo } from "a";
foo();
import { bar } from "a";
bar();
import { baz } from "b";
baz();

compiled to
define(["require", "exports", "a", "a"], function (require, exports, a_1, a_2, b) {
    a_1.foo();
    a_2.bar();
    b.baz();
});

Or
define(["require", "exports", "a", "a"], function (require, exports, a, b) {
    a.foo();
    a.bar();
    b.baz();
});

I think he means that if an import is elided (like it is just an interface):
For example a.ts:

export interface Foo {
    foo(): void;
    bar(): void;
}

export function foo() {}

Then in b.ts:
import { Foo } from 'a';
const obj: Foo = {
    foo() { },
    bar() { }
};
import { foo } from 'a';
foo();

Which currently compiles to:
define(["require", "exports", 'a'], function (require, exports, a_1) {
    "use strict";
    var obj = {
        foo: function () { },
        bar: function () { }
    };
    a_1.foo();
});


userguide in README error 404

https://github.com/edemaine/cocreate/blob/master/docs/README.md following link.

Thanks for spotting this!


Disable the NPM audit

This PR disables the NPM audit task enabled previously.
The audit still periodically fails project builds due to service unavailability.
Although the corresonding NPM issue is closed, the problem still persists.
This PR disables the auditNodePackages task so the developers don't have to constantly restart project builds locally and on CI.

@armiol, @dmdashenkov PTAL.

@dmdashenkov let's not merge at the moment. As per our discussion, I'd like to get it addressed properly.

@armiol, agreed.

Closing for now in favor of #103.
@dmitrykuzmin FYI.


Endorse: Learning in Public

if you put an emoji reaction here, your profile pic will show up  on my README as an endorsement!

pls endorse me for the learn in public

Hi,
I agree with learning in public.
also, I like your user profile idea.

I agree with learning in public.


Harmony: Shorthand functions in objects do not add the shorthand flag

Code:
var obj = {
  myfunc() { return null; }
}

myfunc node doesn't have truthy shorthand property.
{ type: 'Property',
  key:
   { type: 'Identifier',
     name: 'myfunc',
     range: [ 16, 22 ],
     loc: { start: [Object], end: [Object] },
     parentNode: [Circular],
     parentCollection: [ undefined ] },
  value:
   { type: 'FunctionExpression',
     id: null,
     params: [],
     defaults: [],
     body: { ... }, ... },
  kind: 'init',
  method: true,
  shorthand: false, // !!! here it is
  computed: false,
  parentNode: ...

Ref https://github.com/jscs-dev/node-jscs/issues/1013

It has the method flag. shorthand must refer to {a} style properties, no?

@michaelficarra yes, probably. Does it standardized? What's the difference between foo : function bar() { } and fooBar() { }?

The former can refer to itself by name. The latter has a home object, can make super method calls, and has a bound this.

@michaelficarra okay. What's about:
var myfunc = function () { return 'asd'; };
var myobj = { myfunc };
console.log(myobj.myfunc()); // asd

Here I have myfunc and it's shorthand but not a method, right? That's very confusing.

Yes. Not confusing to me.

So you totally sure that it shouldn't have shorthand truthy?
Oh. As I see, method flag is only set for { myfunc() { } } notation. So I should use it for methods as well as shorthand for properties. Right? Does method flag is used for something else?

Nope, that's what you're looking for.

@michaelficarra Thanks!


Fixes get_crosslink_committees_at_slot() logic and test

PR Description
Fixes get_crosslink_committees_at_slot() logic and penalizeValidatorDecrementsBadActorAndIncrementsWhistleblower test with accurate mock data.
penalizeValidatorDecrementsBadActorAndIncrementsWhistleblower originally initiated the validator list at size 3. The number of validators required to remove this error is supposed to be at least 64, since split splits the validator list into 64 committees. The issue was that split would return empty lists for the empty committees - and the divide by zero error was due to the first item in the shuffling list in get_crosslink_committees_at_slot being an empty list. Thus, testing with 64 validators should fix the problem, since each committee will be non-empty.

I'll let Cem double check this to make sure the changes in get_genesis_beacon_state are compliant with the spec. I'd just suggest you use java.util.Collections.nCopies for some of the data generation so the code isn't as verbose.

I made the changes so it followed the spec exactly (before it didn't match completely).


Add universal index processing

OLM catalogs are switching from sqlite-based to file-system based
index format (also referenced as FBC). The enhancement is described in
https://github.com/operator-framework/enhancements/blob/master/enhancements/declarative-index-config.md
Additional information is available here:
https://github.com/redhat-openshift-ecosystem/community-operators-prod/discussions/512
The switch of default Red Hat catalogs to FBC is expected to occur in Openshift 4.11.
This commit changes the way catalog processing is
done by pre-caching workload to extract
packages-bundles-channels relations from the
index image. It is switching to a new opm command
(opm render), that works for both sqlite-based catalogs and FBC.
/cc @browsell @irinamihai @imiller0 @rcarrillocruz @danielmellado

/hold
@rcarrillocruz @danielmellado Thanks for your comments!
I'm putting this on hold for now, since I don't want it to land in 4.10
Will address your comments next week, together with one more small issue I found (Need to configure the opm with pull secret in .docker/config.json. Although this is deprecated, some indexes still include opm binaries that don't support podman default location for pull secret)

/unhold

/hold cancel

/meow

/retest

/retest

I don't have much context about the whole precache work, but this change looks ok to me, just a refactor to use a new parameter.

/lgtm

Same as Ricky, lacking context overall but the patch seems self-contained.

msg=Error: failed creating IAM Role (ci-op-xcqpz121-f9fd8-q5zqk-worker-role): LimitExceeded: Cannot exceed quota for RolesPerAccount: 1000
/override ci/prow/integration

/cherrypick release-4.10

/cherrypick release-4.10

/cherrypick release-4.10


Draft: DSD IAM Terraform stack

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.50%. Comparing base (a0096e6) to head (7adcd41).

:exclamation: Current head 7adcd41 differs from pull request most recent head 1f344b9. Consider uploading reports for the commit 1f344b9 to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4221   +/-   ##
=======================================
  Coverage   98.50%   98.50%
=======================================
  Files          41       41
  Lines        3952     3952
=======================================
  Hits         3893     3893
  Misses         59       59

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


stat call for path triggers two requests (one List and one Head) with enabled cache

Mountpoint for Amazon S3 version
mount-s3 1.4.0
AWS Region
us-east-2
Describe the running environment
Running in EC2 on Ubuntu 22.04.3 LTS using instance profile credentials against an S3 Bucket in the same account.
Mountpoint options
mount-s3 mhnap-bucket /media/mount-s3 --read-only --log-directory=/tmp/mount-s3/logs --debug --debug-crt --log-metrics --cache /tmp/mount-s3 --metadata-ttl 1000

What happened?
This issue is similar to https://github.com/awslabs/mountpoint-s3/issues/770 that I previously created, but here I enabled metadata caching.
I have created dir1/dir2/dir3/ directory hierarchy with hundred file{1..100} files inside dir3 and uploaded dir1 to my newly created mhnap-bucket.
After, I mounted this bucket using the mount-s3 command and called stat one time for each one of hundred /media/mount-s3/dir1/dir2/dir3/file{1..100} files.
With metadata caching enabled I see two requests (one List and one Head) for each file and for dir1/, dir1/dir2/ and dir1/dir2/dir3/. From https://github.com/awslabs/mountpoint-s3/issues/770 answer I know why there are requests for dir1/, dir1/dir2/ and dir1/dir2/dir3/. But I'm curious about the number of requests per path.
I would be grateful to know whether it's also expected behavior to see one List and one Head request per each path.
I found https://github.com/awslabs/mountpoint-s3/issues/12 issue that seems similar, but I'm not sure whether it's the same.
Cannot paste logs, so uploaded them here: mountpoint-s3-2024-02-22T12-33-06Z.log
Relevant log output
No response

This seems in line with expected behavior: one pair of requests for each directory in the hierarchy and one pair for each file.
As mentioned in #12, Mountpoint needs to make a pair of requests (Head + List) for each entry in order to establish whether to map it to a file or a directory (see docs here).
By using the --cache and --metadata-ttl options, you are allowing Mountpoint to skip repeated requests for each entry (within the set TTL), so in your test you only see one pair of requests for each dir. If you called stat multiple times on the same files, you should not see any additional requests.


PJAX transitions with Swup.js, how to re-init Elder hydration

Hey there ! Very glad to try Elder, coming from Nuxt / Next / Gridsome / Gatsby…
I find ElderJs to be pleasingly simple while still being uber powerful.
I've some precise questions about the "SPA feel" you can get with AJAX-like transitions.
I'm testing Elder with Swup.js, a PJAX library.
It's working correctly and as expected, instead of one specific drawback for now :
Re-hydration (partial and OR global) does not occur after a route change.
For example, the Clock example stays static.
Swup talks about this classic pitfall about PJAX libs.
They offers some hooks to "re-init" JS functions.
https://swup.js.org/getting-started/reloading-javascript
Some food for thought, somehow related :
https://navillus.dev/blog/astro-plus-swup
My question : how can I trigger programmatically what's Elder is doing on an initial page load for making it JS-interactive ? Thanks a lot for your insights.

@JulianCataldo Welcome! Glad you're enjoying Elder.js.
As answered in Discord, this isn't something that Elder.js offers out of the box, but you should be able to take the contents from the hydrateStack and wrap them with some sort of function where you activate the hydrate process.
My gut says this will take some fiddling but should be doable.


First item in input recombination probability must be zero

A check should be implemented in api.py to check that the first value of the input recombination probability array is zero.

Something like below should be added to check_inputs.
    # Check the recombination probability.
    if isinstance(prob_recombination, (int, float)):
        pass
    elif (
        isinstance(prob_recombination, np.ndarray)
        and len(prob_recombination) == num_sites
    ):
        if prob_recombination[0] != 0:
            err_msg = "First value in the recombination probability array must be zero."
            raise ValueError(err_msg)
    else:
        err_msg = (
            "Recombination probability is not a scalar or an array of expected length."
        )
        raise ValueError(err_msg)


FSET does not have a FGET counterpoint

Its not possible to recover just a field value for a key / id. Meanwhile there is a function that works that way for JSON JGET.
I have found a need for this from time to time, especially when objects contain lots of fields and performance is important.
I can make a PR for this.

I would accept a PR. Are you thinking the command would be like this?
FGET key id

FGET key id field
Yea. Something like that.


[no-reply] merge branch 1.7

Sorry for annoying, I selected wrong upstream. This PR should be ignored.