These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

[Note: no meeting last week due to Go team internal summit]
Active issues

Additional closure inlining (original CL https://go.dev/cl/482356 , since rolled back a couple of times)

Is this worth fixing at this point if we’re going to rewrite the inliner?
[Than]: It’s making the inliner behave in a more rational way and it’s better to make the inliner less weird, even if we’re rewriting the inliner.
[Than]: current blocker is definitely a problem with escape analysis
Matthew and/or Cherry can help look

issue #55160

mayMoreStack instrumentation is added in obj.
perhaps David can run a GOSSAHASH bisection.

Status

PGO 5%

[MichaelP] working through through Uber’s specialization CL. I’ve rewritten most of it at this point. It should apply a lot more now.

Loop variable scoping

[David] Compiler work is done.
David and Russ working on user bisection tooling. Might need some improvements to compiler diagnostic output.
For 1.22: need to key off -lang version
[David]: already works with cross-package inlining (you can turn it on with a -gcflag)

LUCI POC

We made the “go” decision on LUCI at the summit.
[MichaelK]: We’re focusing on pre-submit first.

Improved type inference algorithm

[Robert]: Spec work needs to be done (probably during the freeze). Ian and I have some idea how to describe it. Still some work on implementation: Today we can’t pass a partially instantiated generic function to another function f(g[T]) where g is g[T,U] and U can be inferred, but I don’t think that’s a show-stopper.

Inlining overhaul: Design

Than still working on closures inlining bug; Matthew has been busy with PGO and helping with type inference.

Structured all.bash

[MichaelK]: The sooner we get this, the easier LUCI test sharding/merging will be. We’re already using -json for the subrepos.

ARM64 assembler

[Cherry]: No updates from contributor, things may be stalled

watchflakes improvements

[Austin] Given LUCI, what’s the future of watchflakes?
[Michael]: We should look at the gaps between LUCI Analysis and watchflakes. LUCI Analysis doesn’t integrate at all with GitHub issues, but maybe that doesn’t matter because it has its own UI? We might be able to add GitHub integration.
[Cherry]: There are a few things I still want to improve. E.g., subrepos.

Pinner

[MichaelK]: CL looks good and just needs a lot more commenting. There are a lot of almost races that are okay if you do them in the right way. Seems on track to land for 1.21.

Attendees:
Carlos Amedee
Austin Clements
Cherry Mui
David Chase
Ian Lance Taylor
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2023-06-27
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

Nice improvement to link time and RAM from Cherry’s recently landed changes (approx May 2). Numbers:  Frontend Kubelet, similar numbers seen for google internal applications.
Release items

Review of release-blocking and non-release-blocking Go 1.21 issues.
Review of pending 1.21 blog posts

Type inference improvements
Loop var semantics change experiment
slog
wasip1 target
PGO GA (can be a short announcement)
Backward/forward compatibility

Discussion: should we submit CL 503795 for 1.21, the s==s optimization that helps generic string sort?

No. Target 1.22.

[Robert] spec changes in the works.

LUCI

[Eli]: Heschi feels this isn’t moving along quickly enough. If people have cycles to contribute before EOY, that would help.

ARM64 assembler

[Cherry] They just sent a HUGE CL. So things are happening it looks like.

watchflakes improvements

[Cherry]: Basically done. Just “bugs” at this point.

watchflakes: move to cloud

[Cherry] future of this is unclear with LUCI

Explore MacService integration

[MichaelP]: They’re started on this.
[MichaelK]: We may end up using ChromeLabs Macs.

SwissTable

[MichaelP]: Google3 may want to do some benchmarking on this. Maps are used heavily in google3. This may be of value to us.
So this may be worth investing in.

Planning H2 scratch

Go Core planning 2023

Aim to accept iterator proposal in time for 1.22

[Ian]: We want to get user for range loops in first. If that goes in, we’ll do an updated iterator proposal.

Consider landing x/exp/slices (and maybe maps too?) in stdlib

Done in 1.21.

PGO devirt follow-up
min/max codegen optimization
5% performance improvement

[MichaelK]: Some things just aren’t going to benefit as much from PGO.
[MichaelP]: About half of Sweet benchmarks basically don’t use indirect calls, so devirt doesn’t help those.

I have enjoyed following these minutes over the years. Any chance they'll make a comeback?

2023-10-24
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

Go Team Summit (Google-internal) took place last week; lots of good information exchange
we are now <1 month before freeze (freeze begins Nov. 21)
Likely accepted proposal: spec: add range over int, range over func #61405

Specifically, range-over-int for 1.22, and range-over-func behind GOEXPERIMENT=rangefunc for 1.22.

proposal: testing: add Keep, to force evaluation in benchmarks #61179

On hold pending a prototype implementation in the compiler.
Keith:  Anyone working on the prototype?
MichaelK: it is on hold anyway so probably too late for 1.22. We have plenty of time for 1.23.
Keith: it is possible to just add testing.Keep without any compiler changes. Downside is people may add testing.Keep in too many places.
MichaelK: should it be two proposals? One for adding testing.Keep, one for auto-Keep. ?
Keith(AI): post on the issue to clarify.

Discussion of google-internal applications
MichaelK: still working on tracer.

Close to production ready, one issue with cgo.
goal is to land in 1.22 behind a GOEXPERIMENT, maybe on by default.
Already test against Sweet, works well.

Do we want to look over “os: make use of pidfd for linux #62654”

Michael Pratt will review CLs.
There is a new Linux API to query the process status without race. The CLs are going to use that.

Range-over-func tasks

Change GOEXPERIMENT name to “rangefunc”
Yield checking
Named result details

Matthew: when a return statement updates named result values. There is a straightforward solution and we should do that.

David: is Russ doing this, or is Russ busy?
Cherry: how much of this needs to be in for 1.22?
Matthew: I will handle the named result details.

Q: range-over-int fully implemented?

Matthew: believe it is done, will double check. go/types support is also done. Maybe go/ssa needs support.
Matthew: currently behind GOEXPERIMENT=range. Will make it on by default.

Keith: I have a CL to change GC work buffers from gray to white. Work buffer will mean "to be marked and scanned". The benefit will be save a lookup.

David: will this cause more things in the work buffer?
Keith: yes
MichaelK: Austin tried something like this a while back and it is somewhat a wash?

Attendees:
Cherry Mui
David Chase
Eli Bendersky
Matthew Dempsky
Michael Knyszek
Michael Pratt
Than McIntosh

2023-10-31
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

Happy Halloween
FYI

Likely accepted proposal: unique: new package with unique.Handle #62483
Likely accepted proposal: all: add GORISCV64 environment variable #61476

add GORISCV64=rva20u64,rva22u64,rva23u64, not emitting C instructions until dust settles

Range-over-int and range-over-func

proposal accepted
Change GOEXPERIMENT name to “rangefunc”, bring range-over-int out of experiment (Cherry to look into this)
Yield checking for range-over-func (David to look into this)
Spec change of range-over-int (Robert to look into this)

Status

PGO

Michael Pratt: Prototype for function value devirt. Have it implemented, will send it out. Delayed by LUCI work but going fairly well.
Scalable builds: CL from Uber that adds a new command to do preprocessing for ingestion by the go command. Might not get around to landing integration of that for Go 1.22.
Method devirt CLs are already in, got 0.5% of performance gain.
No numbers yet for function value devirt. The prototype has a bunch of limitations.

Loop scoping

David: Talked to Tim King, working on getting the SSA semantics right in the downstream ssa package. Should that happen before the freeze?
Cherry: Maybe doesn't need to be, but pending proposals for go/types and the new go/versions package.
Robert: go/types proposal expected to be accepted soon. Needs prose in the spec, I'll work on it.

Inlining

Than McIntosh: Working on tuning the score adjustments to try and improve performance numbers.
Ran compilebench over the weekend with the new inliner. The compile time has gotten a good deal worse from new heuristics. Slowdown due to heavy use of ir.StaticValue in new heuristics. Looking at a more efficient replacement.
Cherry Mui: What does ir.StaticValue do?
Than McIntosh: detects cases where you assign a local variable "x := 1", then never re-assign "x". In such cases if the RHS is invariant, you can then substitute in the RHS where you see uses of the var.
Keith: There's probably a flag on variables on whether it has been reassigned. That might help to avoid recomputation.
Than McIntosh: concern for the flag is that it could get stale
Matthew: Main complexity we've had is that if we insert new IR concepts during inlining, we make changes that could affect StaticValue. So we do a failsafe that checks all the conditions again to make sure everything makes sense. It should be possible to track that more strictly to avoid the checks.
Any risk for finishing new inliner before Go 1.22? Are you considering downscoping?
Than McIntosh: Doesn't make sense to enable experiment in current state. Still needs work. Compile times are too high and the runtime improvement is too low. 1-2% improvement. Compile time is up 8-9%. For the compiler's SSA package it's 23% slower. The slowdown is mostly not due to additional inlines, but heuristic computation. Pessimistic that we'll ship anything for Go 1.22.

Matthew Dempsky: If hypothetically StaticValue didn't have any overhead, would we be good to ship? We should be able to speed it up. If that's what's keeping us from landing a 1-2% improvement then maybe we can just fix it.
Than McIntosh: We'll have to check.
Matthew Dempsky: Working on inliner scheduling; should have CLs soon.  Filippo seems to believe it'll have a high impact.

Tracer work

Michael Knyszek: in good shape, last corner case is cgo callbacks. Getting Michael Pratt's review. Blocked on cmd/trace update, Felix from DataDog is working on cmd/trace integration. Michael Knyszek as a backup. On track for on by default in 1.22.

Allocation headers CLs

Michael Knyszek: support arenas now, behind a GOEXPERIMENT. Cherry Mui and Keith Randall are reviewing. A concern is that as the header is at the beginning of the allocation, shifting the allocated pointer by 8 bytes. So now allocation is not 16-byte aligned; unsure whether that is a concern or to what degree.

Attendees:
Carlos Amedee
Cherry Mui
David Chase
Eli Bendersky
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2023-11-07
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

Discussion of Keith's demo on faster type switches and asserts

Keith considering about doing some version of the talk for Gophercon
Than: did you use any other benchmarks?
Keith: no. Other benchmarks don't use much type switches.

FYI

Likely accepted proposal: cmd/compile: create GOARCH=wasm32 #63131

this is adding a new GOARCH, not changing the current GOARCH=wasm
there is a 64-bit wasm VM, probably not widely used
this is proposed by the WASI folks
Matthew: sandboxing 32-bit programs may be easier

Likely accepted proposal: sync/atomic: add Uint64Pair #61236

MichaelK: is this 16-byte aligned? e.g.
"the compiler and toolchain will provide the necessary alignment automatically"
Matthew: pad the type size to at least 32 bytes?
MichaelK: the allocation headers will break it (16-byte aligned for object >= 32 bytes), unless it is noscan
Matthew: need type checker support. As we do more and more aligned types, it may be a good idea to revisit how we do alignments
Cherry: SGTM. also as we may do SIMD in the future
Keith: the stack is not aligned, so need to escape to the heap
MichaelK: we should consider to do the AlignedTo types, soon. Or user would do "_ [0]atomic.Uint64Pair". If there is already a way, may give user a better way to specify it.
Cherry: for only atomics, it may not matter much for stack locals, as long as the instruction doesn't fault

1.22 stuff

Robert: a fun CL 539299, math/big: implement Rat.FloatPrec
not urgent but want to get in for 1.22

Cherry will review

execution tracer work:

MichaelK: everything put up for review is pretty close to landing
new tracer will land pretty soon, but enabling by default is blocked by cmd/trace support. cmd/trace code has no abstraction. Working on it. Need review.
What about a release exception, just in case? The freeze exception would include enabling new tracer by default.
MichaelP: happy to review cmd/trace CLs. Release exception sounds fine, but should be done soon.
MichaelK: Felix has a working implementation but not all features. Will have a good chance done by freeze.
Matthew: freeze exception is up to the release team. My impression is that they've been increasingly strict about this.
Matthew: could the cmd/trace development be done in a branch or a separate repo, so power users could pull it in?
MichaelK: if new tracer is not on by default, it will be a hurdle for people to use it.

Inliner work

Than: CL reviews are a little stalled but will get them reviewed in the next two weeks
Matthew will review

Attendees:
Carlos Amedee
Cherry Mui
David Chase
Eli Bendersky
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2023-11-21

Go 1.22 freeze today!
performance comparison Go 1.20 vs tip+PGO

-4.59% geomean on Sweet
BiogoKrishna +20% regression
[Matthew]: does Sweet run old/new configs interleaved
[Michael]: yes, interleaved (but not randomized)
[Michael]: BiogoKrishna is not a representative benchmarks. built with some nonpreemptible loops, a lot of bit arithmetics and table lookups.  The reason we keep it is it is a good preemption benchmarks. It can be sensitive to code layout.
[Michael]: cool debug trick on the dashboard

Please write release notes #61422
Go 1.22 issues
2024 topics

KSLO cost reduction
Go on future platforms (RAM efficiency. NUMA?)
(maybe) Go-Python interop for AI-powered applications
[David]: is it a good idea to use cgo for Go-Python interop?
[Michael]: no. better with pipe or RPC
[Michael]: excited with RAM efficiency work. Not sure if NUMA is the right target. Austin's analysis shows memory bandwidth is going to be an issue.
User survey: impact vs effort mapping graph of the potential optimizations?
[Michael]: intended to do another session with Alice with user survey for 2024 priority. Austin has a good start for RAM efficiency things
[discussions about project prioritization]
[Michael]: go.dev/fast is under RAM efficiency?
[Eli] documentation work is its own thing. Sounds important, but don't spend too much time. New tracing work is also in this direction.

2023-11-28

Go 1.22 RC1 is scheduled on Dec 12 (2 weeks from now)
Please write release notes #61422

[Keith] there was some discussion previously about revising and/or improving the process for putting together release notes -- has anything happened on this front?
sounds like not for this cycle on the new process front
Loop scoping (David or Russ to write relnotes)
Range-over-int (David to write relnotes)
Range-over-func under GOEXPERIMENT (David and/or Russ to write relnotes)
go/types new APIs (Alias, FileVersion) (Robert Griesemer to write relnotes)
New tracer (Michael Knyszek to write relnotes)
Bump bootstrap to 1.20 (MichaelK)
PGO devirtualization changes (Michael Pratt to write relnotes)
Allocation headers (Michael Knyszek to write notes)
CL 544195 runtime: profile contended lock calls (Michael Pratt or Rhys)
CL 534161 runtime/metrics: add STW stopping and total time metrics (Michael Pratt)
CL 537515 runtime/pprof: include labels for caller of goroutine profile (Michael Knyszek or Michael Pratt)
CL 511475 cmd/link: allow deriving GNU build ID from Go build ID ID (Cherry Mui)
CL 493136 cmd/link: rationalize -s and -w flags (Cherry Mui)

Port specific changes that might require release notes

CL 461697 default to PIE on darwin/amd64 (Cherry Mui)
CL 514907 GOARM=softfloat/hardfloat (Keith Randall)
CL 521790 enable register ABI on Loong64 (David Chase or Loong64 owners)
CL 521778 enable plugin on Loong64
[Than] Windows SEH stack unwind #57302? need to ping CL/issue

Robert Griesemer: writing release notes is time consuming because the format is tedious

Cherry Mui: there was some discussion about moving to markdown
Michael Knyszek: this is actively being worked on
Carlos Amedee: good to give feedback to Jonathan
Matthew Dempsky: similar to api/next, we can do relnotes/next
Carlos Amedee: this is the direction

Go 1.22 issues
Status

discussion of range-over-func efficiency
David Chase: Russ's CL is related
Michael Knyszek: do need freeze exception for anything here?
David Chase: iter package needs a separate GOEXPERIMENT
Keith Randall: iter package should land in x/exp?
Matthew Dempsky: need some coroutine stuff from runtime

Attendees:
Carlos Amedee
Cherry Mui
David Chase
Eli Bendersky
Joedian Reid
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2023-12-12

Schedule

go team fixit week (this week)
go team "quiet" week (next week)

Go 1.22 RC1 is rescheduled to next Tuesday, December 19

Thank you for writing the release notes!

topic: //go:linkname

Than: do we need to start making a more concrete plan for disallowing “pull” style go:linknames of runtime/stdlib symbols?
Than: there are a lot linknames in opensource code. If we plan to change this, maybe we start communicating.
MichaelP:  it is still unclear if we can and should change it. If we don't provide an alternative, I don't think we can just turn it off. If we delete entirely, users may resort to assembly.
Matthew: assembly code is a good point, user can still call it from assembly code. Balancing that we want to stop user poke into runtime, while provide things they want to do
Cherry: we can stop assembly code call them in the linker
Keith: we can find the top users of it and try to fix it somehow. The problem is if we change the name of popular things, we can help them
Than: if we leave things entirely unchanged, it would block future projects (e.g. renaming runtime to internal/runtime)
Cherry: for the rename, we could provide forwarding linknames
MichaelK: we can treat the list of existing linknames as a priority list and provide workarounds

rand (handled in 1.22)
nanotime
memhash

MichaelK: fine to break programs that linkname to stoptheworld
Matthew: maybe a good starting point is to start implement the logic in the linker and a vulncheck-style tool for why they might fail the build, to communicate with users, and get feedback from users which linknames are needed
MichaelP: discourage linknames by providing a compiler flag to turn it back on. That will strongly discourage uses in libraries
Robert: we should have a better idea for why users use them
survey from Matthew, MichaelP shows 700 linknames for packages with 10+ imports

Status

compiler error URLs

Robert: Russ thinks we should be doing this with mnemonics, not numbers. Robert Griesemer will look into the mnemonics. Also need to write the documents. Russ vetoed autogenerated docs. The docs could be more explanatory. Carried over to 2024.

Heap analysis improvements -- viewcore

MichaelK: want to work on it this week for fixit, for viewcode.

PGO devirt

MichaelP: done, with minor issues, not clear we want to address them soon

Traceback iterator: simplify defer

Matthew Dempsky: done. will check with Austin.

Attendees:
Carlos Amedee
Cherry Mui
David Chase
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2023-12-19
No meeting today (Go team "quiet" week)

2024-01-02

Allocation headers results

still evaluating impact of GOEXPERIMENT=allocheaders change on google-internal applications. It is generally positive, but not clear to what degree.

Go developer survey

“The top requests for improving toolchain warnings and errors were to make the messages more comprehensible and actionable”
Some of this is also about runtime errors
mild surprises (at least to attendees):

5% targeting wasm
56% “use” ARM64 (!)

“However, Apple hardware isn’t the only factor driving ARM64 adoption: among respondents who don’t develop on macOS at all, 29% still say they develop for ARM64.”
rumor has it that ARM just laid off their whole Go team in China

“improved expressivity [“creature comforts”] (12%), improved error handling (12%), and improved type safety or reliability (9%). Respondents had a variety of ideas for improving expressivity, with the general trend of this feedback being “Here’s a specific thing I write frequently, and I wish it were easier to express this in Go”. The issues with error handling continue to be complaints about the verbosity of this code today, while feedback about type safety most commonly touched on sum types.”
[MichaelP]: 50% said they use protobufs (mostly via gRPC)

2024 plans

Finishing in flight (done by end of Q1)

LUCI migration
Inlining: enable new heuristics
Inlining: scheduler
PGO: scalable builds
Runtime trace overhaul: cleanup (delete old tracer, old cmd/trace, moving old trace parser behind new API)
Runtime trace overhaul: reader API
go build -json

1.23

Range-over-func GA
/e/ URLs?
Type parameters for type aliases?

[Eli]: I think this is very far along. Ask Robert next week.

sync/atomic And/Or
sync/atomic Uint64Pair?
Cost reduction (RAM efficiency)
Performance analysis
[MichaelK]: CockroachDB is excited about contributing benchmarks to our suite.

administrivia

more of the work formerly done by the release team is being transferred over to the core (C+R) team

Attendees:
Austin Clements
David Chase
Dmitri Shuraylov
Eli Bendersky
Michael Knyszek
Michael Pratt
Than McIntosh

2024-01-23

Tree open! 🎉
Upcoming 2024 plans

MichaelK: Is the iter package going to land in 1.23?
Cherry Mui: My tooling for cold path escapes could probably also be used to estimate up-stack escape analysis (by excluding returned values)
Single bit reference counting

Matthew Dempsky: https://mdempsky.notion.site/Dynamic-escape-analysis-76bbeecd3ac4440c88d0cb2f722aaf75
Michael Knyszek: Unique pointer optimization and freegc
We should brainstorm about this.

People are grumpy we dropped Windows 7 support (also without a good error message)

Dmitri Shuralyov: We dropped support in a major release and updated our installer check
The minor release change may have been that we now get a dynamic linking error instead of something nice.
Michael Pratt: This seems to have come from a security fix.
Michael Pratt: Considering a proposal for Windows 7 as a secondary port. Seems like a reasonable path.
Dmitri Shuralyov: We’ve never done that before. This is WAI and went through the proposal process. Adding back support would have to be a new proposal.

Attendees:
Austin Clements
Cherry Mui
David Chase
Dmitri Shuraylov
Joedian Reid
Keith Randall
Michael Knyszek
Michael Pratt
Robert Griesemer

No C&R meeting today (this is a Google Core "no meetings" quiet week).

2024-02-27

Amazing out-of-tree SwissTable results from Peter Mattis (CockroachDB’s CTO)

TL;DR: Iteration is ~60% faster, gets are ~20% faster (though there are some odd slowdowns with gets of non-existent elements on large maps), puts are ~30–45% faster (~20% faster if they grow the map), deletes are a mixed bag
Austin’s benchmark analysis, benchstat, benchplots
Uses extendible hashing to avoid resize latency of earlier implementation
Missing some optimizations that would be fairly easy to do in a runtime implementation. Mostly because it’s using Go generics so can’t do the type specialization we do.
Doesn’t currently even use SIMD, though again in the runtime it wouldn’t be hard to add that.
The implementation is complex, but looks really clean.
Peter is supportive of getting it into the runtime; it seems clear that we (Go C&R team) should work to make this happen.
Keith Randall and Michael Pratt to work on bringing this in.
Michael Pratt: Expecting a long tail of weird performance regressions for google-internal tests and such. Probably needs to be behind a GOEXPERIMENT.

misc wasm discussions
misc meeting-scheduling discussions

Attendees:
Austin Clements
Carlos Amedee
Cherry Mui
David Chase
Dmitri Shuralyov
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2024-03-12
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

Lengthy discussion on iter.Pull and LockOSThread semantics

Case: Both goroutines call LockOSThread

G1 is on a locked OS thread

G1 calls next, which switches to G2

G2 locks its OS thread

G2 calls yield

Option 1: Step 2 does a thread switch
Option 2: Step 2 “donates” the OS thread. Step 3 moves G2 to a new thread
Option 3: Step 3 panics

Case: yield crosses goroutines

G1 is on a locked OS thread

G1 calls next, which switches to G2

G2 passes yield to another goroutine, G3, and continues executing

G3 calls yield, which switches to G1

If step 2 does a coroutine switch, then it has to “donate” the locked thread to G2. But then in step 4, you have both G1 and G2 that have to run on the same locked thread.
Option 1: Step 2 does a thread switch
Option 2: Step 4 interrupts G2 (which may take arbitrarily long) in order to run G1
Option 3: Step 4 panics because yield was called on another goroutine. (just iter.Pull or iter.Pull and range-over-func).
David Chase: G3 can do a lock OS thread, too.

Austin Clements: I’m not sure that matters for this case.
Michael Pratt: That could be a problem with the first case, where G2 calls lock OS thread, but G1 does not.

Case: next crosses goroutines
Michael Knyszek: Cgo callbacks are a pain because that thread is always locked.
Matthew Dempsky: Deprecate LockOSThread and add RunOnLockedOSThread(f func())

Michael Knyszek: Maybe that’s what we should have done in the first place. A lot of UI libraries have a “take this function and run it on the main thread”.
Cherry Mui: go.dev/issue/64777
Ian Lance Taylor: go.dev/issue/64755
Austin Clements: Even if we deprecate LockOSThread, we have to deal with it forever (though the performance may be less of a concern).
Michael Pratt: Passing yield to a RunOnLockedOSThread goroutine, same trouble

Ian Lance Taylor: What’s the downside of making the switch do a full goroutine switch if the thread is locked?

Austin Clements: Semantically that works out. Performance is an issue. You don’t necessarily control whether you’re locked, like in the cgo callback case.
Ian Lance Taylor: iter.Pull is not going to be that common
David Chase: I agree. We have a mechanism that works: goroutines. It’s unfortunate that many of these iterators could easily give you a pull, but we’re not exposing that.
Keith Randall: The issue I see is that you may use some code that uses iter.Pull internally. It feels like an abstraction break between the OS thread locking (maybe from a cgo callback) and the iter.Pull.

Struct packing/reordering

David Chase: Struct packing/reordering is a mechanical thing humans shouldn’t be doing. If we had been doing this with protobufs, we’d have huge savings. 3% RAM footprint, ~20% GC cache traffic. We may end up doing GopherPen and being able to reorder struct fields eliminates a lot of the RAM footprint expense.
David Chase: I was figuring this would work for all structs, but Russ Cox said it should depend on the module language version. But you can do var x *pkgA.T; (*pkgB.U)(x), if T and U have the same underlying type, and if we laid out T and U differently, it’s not clear how we could implement this cast. If this is rare, maybe we can just break it.

Matthew Dempsky: A “fragile conversion” is a conversion form *T to *U where T and U’s underlying type literals appeared in source files from different Go modules.

David Chase: Ran fragilecast ecosystem analysis, only 27 instances in the 21305 10-or-more imports projects. Classification of conversions.

David Chase: I’ve been trying to make it module-aware to reduce false positives, but having trouble. Presumably in this case there’s shared ownership, and a single go.mod language version.

Ian Lance Taylor: What about the case of passing the address of a struct to C code?

David Chase: I don’t pick that up. I’m working on a proposal for a signal type that says “follow the platform rules.” Cgo would automatically add this.

Robert Griesemer: What about per-file versions?

David Chase: I think this would ignore those and just use the module version.

What does “work” mean? Is there a language change here?

Austin Clements: Old code keeps working, even if it has unsafe assumptions about struct layout.
David Chase: There is a small language change here if we disallow “fragile” casts.

Matthew Dempsky: The language spec only has “conversion”, not “cast”.
Matthew Dempsky: Type aliases have a disadvantage for documentation. E.g., if you alias an internal type, the exported type won’t necessarily have documentation.

Austin Clements: Russ and I have talked about fixing that in go doc.

Robert Griesemer: Is reordering based on the structure of the type? Or profile-based? Could it be done in a way that results in the same reordering for the same underlying types?

David Chase: I can imagine someone would want to try profiling, but my assumption is that we would use a stable algorithm that worked the same way for structurally identical types. Partly because there’s unsafe code out there.

Michael Knyszek: How are type aliases represented in documentation today? Should it copy not just the documentation but the definition?

Austin Clements: I think you would want to copy the definition (though don’t hide that it’s an alias).

Matthew Dempsky: So two struct types that are identical would be reordered in the same way
Keith Randall: Fighting concerns. We don’t want to break everyone’s unsafe code all at once, but we don’t want to disallow cross-module conversions.
Austin Clements: That’s why David did the ecosystem analysis. These conversions are very rare.
Keith Randall: What about plain struct conversions?  Memcpy now, but across language versions, a reorder.

David Chase: My analysis doesn’t look for these right now.
Cherry Mui: There’s at least a way to compile this. The pointer case just can’t be fixed.

Attendees:
Austin Clements
Carlos Amedee
Cherry Mui
David Chase
Ian Lance Taylor
Joedian Reid
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2024-03-19
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

FYI

cmd/compile: add go:wasmexport directive #65199 accepted
proposal: runtime/debug: add SetCrashHeader function #64590 declined
proposal: spec: support int(bool) conversions #64825 declined
runtime: CL 561635 changes relative PCs printed in tracebacks #65761

Still an open question. If we’re going to do structured tracebacks, it should probably be tied to the new debug.SetCrashOutput API, so we need to at least decide on the API this release.

discussion about ways to automate Go best practices

Matthew Dempsky: Tangential to that, Austin and I were discussing doing team screencasts. I can feel stuck with my development workflow from 20 years ago and being familiar with workflows other people are using could make us more productive and realize where things could still be improved.

brainstorming ideas for AI-powered apps useful for Go team itself

Solving our own problems through AI, to get us more familiar with AI applications.
OSS project use cases
Build-time use cases

Austin is working on a Go GC efficiency downselection doc

Attendees:
Austin Clements
Carlos Amedee
Cherry Mui
David Chase
Dmitri Shuralyov
Ian Lance Taylor
Joedian Reid
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2024-03-26
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

upcoming inliner heuristics review during this Thursday’s Go breakout
Project status updates

Range-over-func iter.Pull and LockOSThread

David: We could always use goroutines.
Austin: Russ did have an inbetween version that used goroutines and channels, but with fused channel operations.
Michael Knyszek: That might be an easier fallback path and general channels.
Cherry: Did Russ’ fused channel operations work with locked OS threads?
Austin: I’m not sure.
Cherry: It might still be easier to implement the fallback for that with locked OS threads.

Joedian: Are there things we’re worried about for 1.23?

Range-over-func has a lot of open tasks

David: Checking semantics will be fine. Worried about defer with named results. Performance probably okay. Worried about weird things for debugger integration because it’s not clear we can do exactly what they want.

Cleaning up old tracer

Austin: Does this have to happen in 1.23?
Michael Knyszek: It will break with things like iter.Pull, which aren’t supported in the old tracer.

Structured go build -json doesn’t have to happen for 1.23
SwissTables doesn’t have to happen for 1.23, but it would be good for hitting our 5% target.

Michael Pratt: I’m hoping to start this after returning from vacation.

Attendees:
Austin Clements
Carlos Amedee
Cherry Mui
David Chase
Dmitri Shuralyov
Ian Lance Taylor
Joedian Reid
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Robert Griesemer
Than McIntosh

2024-03-28
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

On Thursday 3/28 we held a design review looking at the new inlining heuristics framework being developed as part of the inlining overhaul effort (this was a separate session from the usual C&R meeting)

Slides from the talk: https://docs.google.com/presentation/d/1Lf3WoRyCNicS1K3NCuVl_VnJFhvew_6nAQF_Wx--F54/edit#slide=id.p

Notes from questions/discussion during the talk

Litmus tests

Inline sort.Search with a (small) function literal
Fast path/slow path inlines the fast path
Singleton calls to non-exported functions
Or specifically closures. Comes up as a way to do “function literals”, and also to give more control over defers (though that will prevent inlining).
Russ gave some examples on the bug
A call site weight of 57 makes no sense. We should be able to significantly lower that.

Parameter heuristics: constant feeds if/switch

Michael Pratt: If you could analyze the effect of propagating a constant, you could potentially significantly reduce the size. It would be nice to capture that in the score.
Than: We don’t currently look at the value of these constants. Tracking that could significantly bloat the export data.

Sweet benchmark results

The GoBuild (non-Link) benchmarks are measuring the impact of doing more work on compile time, so it’s not directly comparable with other benchmarks. That’s fine, but means the overall geomean isn’t meaningful.

Difficulties/obstacles

Michael Pratt: With PGO, we’ve noticed that about half of the performance effect comes from optimizing the runtime, in particular the garbage collector. We’ve never checked closely if that’s from making the GC faster or the application doing less allocation.

Robert: Why don’t we let users say “inline this”?

Ian: GCC shows that people often get this wrong.
Russ: Overfitting to “right now”
Ian: OTOH, GCC experience shows that people usually use “don’t inline” correctly.

Keith: To some extent we train our users to write code for the inliner. Like manual hot/cold splits today. It’s better if users don’t have to jump through hoops to get “the right thing.”
Robert: If people write lots of small functions, you might get the large function in the end. If people write large functions, we can’t do anything. Maybe we should tell people to write smaller functions.
Tim: Do we know how well these heuristics correlate with places of user angst?

Russ: Perhaps replace angst with “places where inlining actually helps”
Than: I think that happens with passing concrete values to interface arguments. People get grumpy when they have to contort their code around that.

Ian: Heuristic that specifically targets removing escapes?

Than: Passing a concrete value to an interface helps, but that’s just one example.
Ian: Also feeds into type switch or type assertion. Maybe “likely” interface type at call site? Also unused parameters, though that helps more with interface method calls (e.g., caller does some work to compute a parameter that isn’t used).

Notes from questions/discussion after the talk

Austin: I would expect the common case to be a constructor function that returns a pointer, but the result doesn’t escape past the call site.
Than: That’s hard to capture in a heuristic that’s not doing full escape analysis.
Austin: Long ago, we talked about doing the data flow before inlining, then using that information to drive inlining, then producing final escape summaries.
Than: That would definitely require an escape analysis expert.
Russ: Seems like the strategy is to think about heuristics and try them and see what happens, which doesn’t feel very directed. Maybe, turn the inliner up really high, see what that gives you, and then figure out why. Hash bisection may be able to tell you this: write a function that says “is etcd still > 10% faster?” and run hash bisection to get the set of places where it really matters. Then ask what patterns get you those. Then worry about false positives that come with those.
Than and others: idea from Russ sounds good

probably need to finish stack slot merging before we do this, or the costs of inlining may be too dominant to seek out the wins.
Than: bisection could be made more difficut due to noisy benchmarks
Than: For escape effects, we could measure the allocation count, which should be pretty stable.

Tim: Range-over-funcs don’t exist in any code, but do you have a sense of how that will interact with these when that does start to exist?

David: It resolves things down to flat code pretty often right now. Though if something prevents inlining, the whole house of cards falls apart.

Austin: “Calls cost 57” is a clear red flag. A call isn’t expensive. Right now this is our backstop against making lots of bad decisions, but it also prevents us from making a lot of good decisions.
Austin: “Don’t inline here” heuristics, if they’re working right, should have almost no effect on performance, just reduce binary size. Should check binary size effects of those.

2024-04-02
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

Discussion about how to help automate / and ease the introduction/rollout for Go "best" practices" that we use in the Go core team

Ensure users are getting the benefit of the core team’s work. (For example, IDE integration is great for build-time things, but we don’t have great answers for deploy/run-time.)

PGO example: How do developers consume profiles today? Can we integrate with those? (Vendors like DataDog?)
How do people coordinate production builds today?

“GitOps”: CICD & triggers, Github Actions

should we partner more closely with other teams? If we did, what could we do better?
If we did partner, then what happens? A single integration only does so much, while building a partner relationship keeps on giving.

Keith: It would be great if this problem didn’t exist. A lot of our performance optimizations just happen when you upgrade. Hopefully the places where we couldn’t just make it automatic are a minority of places.
Matthew: Go could be missing more of an “application level framework.” Right now if you want to build applications with Go, you have to use a lot of external packages. Unless you’re writing a 1970’s CLI. (Ian Lance Taylor: Gophers on Rails.) Opinionated Go way.

Michael Knyszek: Service weaver?
Carlos: go-kit
Austin: I don’t know what’s stopping us.
Matthew: What’s stopping me is that I’ve never built an application in Go and it seems scary and keeps changing. Go simplicity for building applications that work nicely over time.

Michael Knyszek: I think there’s room for improving, e.g., net/http/pprof. Gophers Slack was discussing how everyone exposes basic debugging endpoints. Everyone does it differently because net/http/pprof is a security hazard. Where we did try to provide convenience, we’re now, today, doing a bad job. Pprof should just be there, and secure, and working by default. That impedes PGO integration. Could we integrate into the new “hot thing”, OpenTelemetry?

strings.Compare optimization

Keith: Used for sorting. We have an old “Less” implementation, and strings.Compare builds a 3-way compare on two “Less”es. We’ve pushed back on optimizing strings.Compare, but now it’s the thing to use for sorting because “you should be using the built-in comparisons.”

Ian: Seen “if strings.Compare(a, b) < 0” many times because people don’t realize you can just directly compare strings.

Compare the strings once instead of twice, using internal/bytealg
CL 532195
Austin: Can we do the same for cmp.Compare?

Ian: No because we can’t specialize it for string. (We could type switch on string, but not ~string. And it’s not clear if the compiler would specialize in that case anyway)
Austin: We could teach the compiler about this special case.

Keith: The other option was to look for the three-way comparison code pattern and compile that specially. That would catch both strings.Compare and cmp.Compare. But it’s a tricky optimization.

Austin: It could be pretty specific to the pattern in strings.Compare and cmp.Compare.

Way forward: just land this CL. We can undo it if we ever do the general optimization.

Austin: LLVM MLIR uses “basic block arguments” instead of “phi functions” and it sounds like that makes it easier to do block graph manipulations. Have we considered doing that sort of thing?

Matthew: That’s what I was suggesting with reorganizing the compiler around basic blocks that do tail calls instead of functions. This seems to be the popular SSA-like representation these days.
Austin: It seems like we could do an 80/20 thing where we don’t have to restructure the whole compiler toward a sea of basic blocks. We could still have regular functions, and just make it easier to manipulate the BB graph.
Austin: What’s the implication for, say, BB1 dominates BB2 but isn’t a direct predecessor, and BB2 uses a variable introduced in BB1. Does that mean that variable has to be threaded through the arguments to the intermediate BBs?
Matthew: Example: Thorin IR (see Figure 4 on page 4 for syntax/semantics/typing). BB arguments directly correspond to phi functions and you’re allowed to do “implicit capture”. Results in a very concise semantics.

Attendees:
Austin Clements
Carlos Amedee
Cherry Mui
David Chase
Dmitri Shuralyov
Ian Lance Taylor
Joedian Reid
Keith Randall
Matthew Dempsky
Michael Knyszek
Michael Pratt
Than McIntosh

2024-08-06
These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.

FYI

hash/maphash: add func Comparable #54670 re-likely accepted

Resume public meeting minutes at #43930
Went over H2 plan

Bump bootstrap to 1.22
Bump minimum Linux to 3.17
Switch built-in maps to SwissTables
Range-over-func

Range-over-func: Fix addrtaken with inlining [DONE]
Range-over-func: Fix iter.Pull performance under cgo

PGO: Discriminator support
Type params on aliases

Export data support
Enable GOEXPERIMENT=aliastypeparams by default

Structured all.bash: go build -json
GC efficiency
WASM

Implement go:wasmexport
Integrate structs.HostLayout into go:wasmexport and wasm32

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-08-13

Go 1.23.0 is released today 🎉
#68658 cmd/go: breaking change in 1.23rc2 with version constraints in GOPATH mode

Decided not to include CL 603895 in 1.23.0
Discussions on CL 603895
Robert Griesemer: if the file local version is not correct (e.g. go1.21.4 is invalid), it is silently ignored, which may be surprising to user

#64127
Tim King: vet has a "likely match" build tag check that should catch this
Cherry Mui: is the file not being built by go command at all?
David Chase: is it syntactically invalid? What about //go:build !go1.21.4 ?
Michael Knyszek: is there a recorded talk about version upgrading/downgrading

Russ Cox has a blog post (and this)

Continue discussion on the issue

1.24 cross-team features

parameterized type alias, with tools team. Robert Griesemer and Tim King are already talking with them.

Tim King: would be very nice to get it in before the next x/tools version bump, which is September
It is not yet on by default. Tim King has a checklist https://github.com/golang/go/issues/68778
Keith Randall: the flip is prerequisite for x/tools change?

Tim King: yes, but need tools's export data in first before flipping

Keith Randall: bump the bootstrap version. There is already a CL that wants to use 1.22 feature in the compiler

Dmitri Shuralyov: will get in this or next week

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-08-20

Status

Bootstrap bump

Dmitri Shuralyov: LUCI builders are all updated, done a minimal fix for legacy builders, just submitted first CL that requires new version at tip.

Linux bump

Ian Lance Taylor: The actual version we’ll be requiring has not been finalized yet. Brad pointed out that 3.17 is a problem because there are a lot of old Synology boxes out there. It’ll probably be 3.2.

SwissTables

Michael Pratt: I have a full working implementation with extendible hashing. A bunch of benchmarks are slower vs the existing maps. We’ll almost certainly need special paths like fast32/64. Mostly really small benchmarks, like small maps looking up the same key over and over.
Could we make the new map actually generic?

We don’t have anything that makes up instantiations like these in the compiler.
Problem: runtime.a isn’t always available today (we could fix that)
Problem: what types do we instantiate? Do we pre-instantiate common types in the runtime package?

Michael Pratt: We may want the “GC shape” of these to be defined somewhat differently than the usual.

Turning the equality call from an indirect call into a direct equality check would be much faster.

Type parameters on type aliases

Tim King: Most of the hard bits are out for review. Need to turn on more builders. Need to copy various bits into x/tools.
Austin Clements: Could we ever fully decouple these?

Tim King: It would be wasteful. You could still have the compiler do type checking, but then call out to something that would write the data for vet.
Cherry Mui: The compiler export data has a lot more than what tools need and does a lot of non-type-checking work. If my cache is empty and I run go vet, it does a lot of unnecessary work recompiling everything.
Robert Griesemer: There is a source importer that’s existed for years. The API checker does more or less what Cherry said, but doesn’t cache the results. I don’t know how much x/tools depends on caching repeated work. There’s a lot of work in the unified importer to only import what you need. Not impossible, but quite a lot of work.
Cherry Mui: I’ve heard gopls has its own export data format (not positive). Presumably gopls is quite latency sensitive, so if they can do it, vet probably can, too.
David Chase: Do the tools get the same view of the module import graph as the compiler? (This came up because I’m looking at making go doc look at other packages.)
Tim King: I believe it all comes from “go list”, but it might depend on the tool.
Tim King: We also run vet from “go test”, so spawning a duplicate process to write duplicate type information seems wasteful, but it would definitely simplify everything for the two sides to use their own views of export data.

WASM

Cherry Mui: wasmexport is pretty much done. WASIp1 is in. WASM JS is out for review. May want to add more tests.

David Chase: Found loop alignment anomaly on ARM64. Goes different ways on M2/M3.

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-08-27

cmd/go: breaking change in 1.23rc2 with version constraints in GOPATH mode #68658

Michael Matloob proposed solution: max(1.21, fileVersion) independent of -lang (moduleVersion)
We'll probably do this in 1.24 and 1.23.1

Robert Griesemer: 1.23.1 is scheduled next Wednesday, and Monday is U.S. Labor Day. If we want to do it in 1.23.1, it needs to be in soon.
Currently there is no backport issue.
Dmitri Shuralyov: Is it okay to be 1.23.2?
Cherry Mui: I think it's okay. It's not new in 1.23.0.

Proposal: testing/synctest: new package for testing concurrent code #67434

Michael Knyszek and Michael Pratt will review

Elimination of core types from spec #63940

Robert Griesemer: I have implemented 60-70%. Will see how it goes. Will need a proposal and get accepted. There will not be a huge change to users.
Cherry Mui: Is the concept of Core type just in the spec and internal to the compiler? Is there any public API about core type?
Robert Griesemer: will sync with the tools team

Range-func/iterators performance

David Chase: at least for a particular small loop, if the body includes a 4K boundary, it will run 2x fast on Apple M3, but 2x slow on Apple M2, no difference on GCP ARM64 VM
David Chase: we are mostly happy with the performance (on sensible machines) after deadlocals. There is a case that the inliner could do better.

The inline budget calculation about OCLOSURE
Keith Randall: you want the cost to be proportional to the size of the code to create the closure object

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-09-03

Status

Range-over-func: Inliner improvement on handling closures

David Chase: https://go.dev/cl/609095 reduces call site penalty if callee is param.  Seems to reduce binary size, helps some iter benchmarks, there is one regression in gonum, looking at that.  Another approach decoupling OCLOSURE size from cost of containing function would help, but was not done correctly, so, not yet.
Keith Randall: this really only helps if the callee is a known function

Parameterized type alias

Alan has a proposal to support only 2 recent versions or export data in x/tools #68898
Tim King: there is a backlog of go/types users that are not yet go 1.22. They should update to 1.23 or 1.24 to use parameterized type aliases. We should probably figure out a good way to advertise this to tool authors.

golang-dev, Slack, tools team may have some channels

HostLayout

Keith Randall: Is it coming to 1.24?
David Chase: structs.HostLayout landed in 1.23. Working on adding it to syscall and cgo. Ian thinks we probably need to do something with unkeyed struct literals. Vet didn't flag them, as they are "local".
Cherry Mui: another possibility is to have the compiler special-case cgo structs without adding the field.
Ian's proposal is to rewrite it in-place in the cgo tool.

Bump bootstrap to 1.22
SwissTables maps
Wasmexport
Diagnostics

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-09-10

FYI

Likely accepted proposal: runtime: add AddCleanup and deprecate SetFinalizer #67535
Likely accepted proposal: weak: new package providing weak pointers #67552

[From Austin] Make wasmimport/wasmexport accept pointers to HostLayout structs?

proposal: cmd/compile: relax wasm32 function import signature type constraints #66984 includes this but is pretty complex. Do we want to do this whole thing or just start with allowing pointers to HostLayout structs?
Worried about letting the horse out of the barn with a wasmexport that only supports unsafe.Pointer.
Cherry Mui: The proposal is in a good direction. GOARCH=wasm32 is still going and uncertain. What do we do with GOARCH=wasm? We could allow structs with HostLayout and only non-pointer fields?

Michael Knyszek: Should we backport https://go.dev/cl/610738?

Summary: unique.Make intends to clone almost all strings before storing them in the central map, but erroneously stores the original string as the map key.
Pros:

Users of unique.Make don't need to defensively clone strings.

Cons:

It's not really documented anywhere (yet) that we clone these strings. Ian suggested doing so a while ago and I never got around to it. So, this isn't going to break anybody, and there is a workaround.

Ian Lance Taylor: I think we should backport. Eli Bendersky agrees.

Discussed about runtime/compiler office hours and contributors sync.

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-09-17

FYI

Accepted proposal: runtime: add AddCleanup and deprecate SetFinalizer #67535
Accepted proposal: weak: new package providing weak pointers #67552

Michael Knyszek: Getting it landed is trivial, just renaming the package and a method. Strongly feel getting both land is better. AddCleanup is the only thing that will take work, but not a lot.
Carlos Amedee will work on AddCleanup.

Likely accepted proposal: cmd/compile: relax wasm/wasm32 function import signature type constraints #66984
Likely accepted proposal: crypto/subtle: add WithDataIndependentTiming #66450

Michael Knyszek: From Slack: can we make unique.Make(string(byteSlice)) not allocate?

The byte slice will be cloned as a string if we store it, so semantically, it's safe to do this. But I don't know how difficult it would be to make this work in the compiler.
Austin Clements: Is there a general inter-procedural optimization here? (cmd/compile: consider to add move-like optimization for some string<->[]byte operations #50846)

Cherry Mui: Maybe related to David Chase's work about pure / read-only functions
Keith Randall: For F(string(slice)), can we use the escape information to know the string doesn't outlast the call? Have to make sure it also doesn’t escape to the result.
Keith Randall: Worried about synchronization.

Example: Goroutine 2 modifies slice then sends on a channel, F reads from the channel, then uses its string argument. Before this optimization, we’d copy at the call site and see the old data (racy), but after this optimization, we’d copy after the modification and see the new data.

Keith Randall: Bigger problem: F could have a reference to slice and modify it.
Austin Clements: Needs some guarantee of the uniqueness of that slice.

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-10-08
No meeting due to Go quiet week.
2024-10-15
No meeting due to internal team summit.
2024-10-22

FYI

Go 1.24 Release Freeze is in a month (Nov. 21)
Handy reference to which wasm proposals are in what state and where they’re supported

Michael Knyszek: Austin proposed to use a similar mechanism to track GODEBUG. What values are set for each release.

testing.B.Loop auto-Keep semantics

https://github.com/golang/go/issues/61515#issuecomment-2407808624 mentioned "we would implicitly apply Keep to every function argument and result in the closure passed directly to Loop."
Michael Pratt: We don't want to keep too many or too few things alive unexpectedly.
Cherry Mui: Would like to have a mechanism that covers a large portion (80%?) and not keep alive too much. One can always manually keep things alive, but hard to undo.
Junyang Shao: Current implementation keeps the result of the last assignment or function call alive.
Michael Knyszek: Would it be fine to try something? I don't think it is so critical to get it perfectly right. One can always use b.N loop as a workaround, if they don't like the semantics. I realize there's some pressure to get it right the first time, though. Maybe we could do some ecosystem analysis?
Cherry Mui: I'm thinking about doing some analysis, keep everything alive and see if it deoptimizes things unexpectedly.
Michael Pratt: Would be nice to get it right when we release the feature. There are also some complexity in matching just the loop body. It may be simpler to keep everything alive in the function. Things outside of the loop are not important.
Michael Pratt: We could also apply to b.N loop as well. May cause surprise in people's benchmarks. May slow things down, or fix bugs.
Junyang Shao: Austin’s semantics of keep everything alive are pretty straightforward and clear to users.
Michael Pratt: In favor of clear semantics.
Sounds like we can try keeping all function args and results live and see.
Junyang Shao: Can change the implementation to match b.N loop and see if it slows down anything.

Junyang Shao: Do we plan to add interprocedural SSA passes? There is some issue about propagating discriminators for PGO basic block layout.

Keith Randall: We don't have plan for interprocedural SSA for now. We have ideas, like code size for inlining, register clobber sets.
David Chase: Is the problem with the PGO basic block CL is that we need to profile twice? One for regular PGO (inlining+devirt) and another round for basic block.
Cherry Mui: We should use line number (and perhaps column number) first, and only use discriminators if the line/column numbers are the same.
Michael Pratt: Normal PGO workflow (PGO on prod) would work because it is profiling PGO binaries. It is confusing that they get the optimal result on the second run.

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-10-29

FYI

Public office hour is scheduled on Nov. 4, 1pm PT/4pm ET

Michael Pratt: Enable swissmaps by default?

Michael Pratt: Plan to submit all CLs today or tomorrow, implementation is solid.
Michael Pratt: Sweet geomean is 1-1.5% faster. Some microbenchmarks may be slower, e.g. mapaccess with 9-256 elements. Unclear how this maps to production use case. Very small maps have a special code path. After 256 it gets a lot better.

1.24 status

range-over-func cgo performance

Probably not going to do this for 1.24.

Type parameter for type alias

Done

PGO discriminator support

Michael Pratt: We could submit the discriminator support, but less important to evaluating the actual uses.
David Chase: Hard to use.
Junyang Shao: The propagation of discriminators is lossy.
Cherry Mui: Using basic block ID seems not the right approach.
Michael Pratt: Some issues for column numbers: multiple blocks have the same column number, like some Values are moved to another block.
Keith Randall: Line and column numbers are meant for the debugger, not for optimization. The compiler keeps track and propagate them for the debugger.
David Chase: We could use the initial basic block ID when constructing SSA.

Struct reordering

David Chase: Discussed with Tim King and Ian Lance Taylor, but there are concerns about gating this on module Go version. New plan is to change how we treat anon fields, will do an ecosystem analysis. Vet check not for 1.24.
David Chase: I will try a GOEXPERIMENT for Go 1.24.

range-over-func general performance

David Chase: Some CLs went in that help. Related to closure inlining. Good enough for 1.24. I will keep looking for improvements, but I don't expect to find anything more.

Eliminating core types

Robert Griesemer: Working on a proposal. I don't expect it to happen for 1.24.

runtime.AddCleanup

Carlos Amedee: Working local version. Working on Stop. Need to get a CL going.

maphash.Comparable

Keith Randall: It's done.

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-11-05

GoLab is next week in Florence.
Swissmaps is enabled by default 🎉

Michael Pratt: There will be followup optimizations. Make use of compiler intrinsics.

1.24 status (cont.)

testing.B.Loop

Junyang Shao: Implemented as noinlining in the b.Loop body. https://go.dev/cl/612043

PGO discriminators

Junyang Shao: Using column number as discriminator seems to work pretty well. Plumbing is easier. Will split out and make it submittable.

Diagnostics
Flight recorder

Michael Knyszek: Updated proposal.

GC efficiency: Microarchitectural optimizations

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-11-12

Unable to integrate with the latest race.syso from LLVM tip  #70283

Bug in LLVM, fixed in LLVM already.

Junyang Shao: Update on PGO block layout: simple algorithm improves 0.8% for CockroachDB, 0.2% for others. Gopher-lua is 2% regression.

Keith Randall: Is it putting hot blocks together?
Junyang Shao: Use edge frequency, put the hot successor next.

1.24 issues

Please work on them, or bump to the next milestone

sync/atomic: TestNilDeref flaky failure on windows-386 with runtime fatal error #70288

Start failing yesterday, causing frequent trybot failures
Michael Knyszek: We haven't touched Windows builders in a year. Given that it seems to die without saying anything, maybe it is bad dereference during a signal?
Russ fixed it.

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-11-19

FYI

Go 1.24 freeze is this week (Nov 21 at 11:59 PM Eastern; docs)
Likely accepted proposal: testing/synctest: experimental package for testing concurrent code #69687

This is adding testing/synctest behind a GOEXPERIMENT
Damien has CLs out, Michael +2'd, working on fixing nosplit overflow on OpenBSD
Michael Knyszek: Should we prioritize fixing the stack check?
Cherry Mui: Yes. Will file an issue about it.

Austin landed CLs for go build -json

Followup work on the LUCI side to make use of it.
Dmitri Shuralyov: Need a small change (#70435). Currently the structured build failure is behind a GODEBUG and not yet enabled on our builders.

Updated perf dashboard

Michael Knyszek: Added geomean to the main page. This is comparing with Go 1.23.
Numbers are good, -3% on 16-core amd64, -4% on 88-core amd64, -2% on arm64
Michael Pratt: Showing comparison with older Go releases would be good
Michael Knyszek: Have limited machine resources.
Cherry Mui: Numbers should be composable?
Michael Knyszek: Probably fine. We try to not compose numbers after the fact, as machine changes from time to time.

Junyang Shao: Is it possible to merge PGO block layout?

Michael Knyszek: There were some inconsistencies in the CockroachDB metrics reporting. Fixed by now. Looks like your result is using the updated metrics. The inconsistencies are worth looking into, but the numbers are probably still meaningful.
Michael Pratt: The other thing we need to take into account is that we need to add column numbers to the binary. It increases the binary size by 8%, 5% in pclntab and 3% in DWARF.
Michael Pratt: One idea is to add column numbers only in hot functions. Another idea is to add them only when multiple blocks have the same line.
Junyang Shao: DWARF line table is delta encoded, can we do the same thing?
Michael Pratt: Pclntab is similar. Also, it may not help much for column numbers.
Time feels tight for getting it in for this cycle.

Carlos Amedee: AddCleanup proposal included deprecating SetFinalizer. Ian Lance Taylor mentioned that there is no 1-1 mapping from SetFinalizer to AddCleanup. So we break up deprecating SetFinalizer to a separate issue, aiming for 1.25.

Michael Pratt: Document that we prefer AddCleanup when possible
Cherry Mui: Changing finalizer uses in the standard library to Cleanup?
Carlos Amedee: Yes. Working on the os package.
Michael Knyszek: That would help shake out bugs.
Ian Lance Taylor: 42 calls of SetFinalizer in the standard library.

Tim King: Moving issues on milestones. Some times manually, sometimes by gopherbot.

Gopherbot push issues to the next milestone when the final release goes out.
Dmitri Shuralyov: Would still be helpful to manually update

runtime: improve scaling of lock2 #68578 by Rhys

Michael Knyszek: For contended mutexes, many threads spin and hammer on it. Another issue is integrating runtime.mutex to mutex profile incorrectly. Rhys fixed it by allowing 1 thread to take ownership and spin on it. Has a high impact on microbenchmarks. Performance neutral for most of our benchmarks. Trust the implementation as Rhys model checked it. Enabled by default, with a GOEXPERIMENT for easily turning off.

These compiler and runtime meeting minutes are a snapshot of what was discussed in an internal Google meeting. Notes on agenda items that address Google specific needs are elided. Any comments on this issue will be removed, and discussion about topics raised should be taken to the appropriate mailing list. Additionally, any feedback or suggestions for additions to the notes should be handled there as well.
2024-12-10

FYI

Go 1.24 RC1 is scheduled this Thursday (2024-12-12)

Release blockers

all: increased toolchain binary sizes #70699

Michael Pratt: How much do we care about toolchain binary sizes vs. user binary sizes. For toolchain it looks reasonable to me
Michael Knyszek: The bent benchmarks also get bigger. Small binaries got 5-10% bigger.
Keith Randall: #70670 is not a big contributor
https://perf.golang.org/dashboard/?benchmark=Ajstarks_deck_generate&platform=linux%2Famd64&unit=pclntab-bytes

One jump is due to FIPS

Michael Knyszek: All regressed seem to follow the same pattern.
Ian Lance Taylor: Do we want to have a nofips build tag?
Keith Randall: Would be nice to verify that helloworld with fmt doesn't import fips, math/rand doesn't import fips. Make sure we don't include fips by accident.

Tim King: Could math/rand/v2 import fips?
Verified. They don't include fips.

Tim King: If we care about the distribution size, it increases 8%
Michael Pratt: https://logs.chromium.org/logs/golang/buildbucket/cr-buildbucket/8728996672309418097/+/u/log/0 most increases are small binaries. So it is a fixed-size increase, not a percentage.
Removed release blocker

Dmitri Shuralyov: API audit status?

Carlos Amedee: This blocks RC1.
debug/elf: add SHT_GNU_VERDEF section parsing #63952

Ian Lance Taylor: Probably need to revamp
Dmitri Shuralyov: We can break out this one for after RC1?
Ian Lance Taylor: We can also roll back the change and do it in 1.25

encoding: add AppendText and AppendBinary #62384

Ian Lance Taylor: This is just some comment changes

2024-12-17

FYI

Next two weeks (Dec 23–Jan 3) are Go quiet weeks

Go 1.24 RC1 was released last Friday 🎉🎉🎉
Secondary port breakage

ppc64 and s390x are newly broken due to FIPS
Dmitri Shuralyov: We applied broken port policy to windows/arm, as it's been broken for a long time
For ports that just have a few tests failing, we don't need to declare them broken

Android/iOS builders

Corellium builders are missing for a long time
Run on emulators? Roland made good progress on iOS emulator
Dmitri Shuralyov: There are still more work to do, got deprioritized. Maybe have the interrupt person prioritize this work. This is more than one week of work.
Cherry Mui: Would be good to have the emulator work going.
Cherry Mui: The Android x86 ports are emulators. Can we do the same for arm64?
Dmitri Shuralyov: They need to be port to LUCI. Need to do some work to start the emulator before all.bash. Wasm builders are much simpler.
Carlos Amedee: There is some hope as Chrome testing on those
Dmitri Shuralyov will put info on tracking issues

Happy holidays! See you next year


Use built-in canonify

Hugo has canonifyUrls flag that automatically translate baseUrl or just '/' based on settings

Thanks for your advice and I think non-canonicalization is better based on Canonicalization: https://gohugo.io/extras/urls/:

Benefits of non-canonicalization include being able to have resource inclusion be scheme-relative, so that http vs https can be decided based on how this page was retrieved.

Note: In the May 2014 release of Hugo v0.11, the default value of canonifyURLs was switched from true to false, which we think is the better default and should continue to be the case going forward. So, please verify and adjust your website accordingly if you are upgrading from v0.10 or older versions.


RFC: Add descriptive names to doctests

I wasn't sure if this needed an RFC or not. It's a rather minor feature, however, it is a user-visible change, and since I've already written one, I thought I'd open a PR. Feel free to close it if it doesn't require an RFC.
I did some supporting work on this in https://github.com/rust-lang/rust/pull/78429, which made code block info string parsing only split on , and whitespace, so that an attribute of the form name=foo would be possible, but never followed up with the actual RFC.
This RFC proposes the ability to annotate documentation test code blocks with metadata of the form name=NAME. The given name will be used to identify the documentation test when it is run by the test runner, in addition to the current information of test binary, module, and line number.
For example, this documention test:
```name=arithmetic
assert_eq!(2 + 2, 4);
```

Assuming that it is in crate foo, in file src/lib.rs, in module
bar, on line 37, will produce this output when run:
   Doc-tests foo

running 1 test
test src/lib.rs - bar::arithmetic (line 37) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Rendered

Updated and addressed comments. Thank you for the comments!

Mentioned test filtering
Remove stray note that NAME be a valid identifier
Rename IDENT to NAME

A previous version of this RFC required that doctest names be valid rust identifiers. I think this might be a good idea, but someone wasn't convinced. I think it's probably a reasonable restriction, and means that you can use the identifiers as names for the generated test binaries/functions/etc, so it's something that I'd like feedback on. However, I wanted the test of the RFC to be consistent, so I removed references to that restriction for now.

Thanks for the feedback! I added the requirement that names be valid rust identifiers, and used spaces to separate attributes in the examples instead of commas.

Friendly ping!

cc @GuillaumeGomez
Are commas the preferred way (or only?) way to separate annotations on a code block? I used spaces originally, since they were accepted by the parser, but I'm not sure if that changed.
I think that's the only change here that needs to be made.

Oh damn, completely forgot to stabilize this feature. Gonna send a PR for that.
Commas are the only character to separate annotations currently. To add your feature, I think you just need to check for name in here. Once done and ready, please write a comment here so we review it one last time. 👍

@GuillaumeGomez Nice! Okay, updated the RFC to use commas in the example about separating attribute names. I think this is ready to go.

Considering https://github.com/rust-lang/rust/pull/124577 is about to get stabilized, I'm not sure if name is good enough now (is name supposed to be an attribute or a doctest name? Can't know unless you know very well how it works).
Another thing to note is that you will be able to add attributes to generated codeblocks with this syntax: custom,{name=something}, meaning that in your case, name here would not name the doctest but instead be an attribute generated in the HTML. So better underline this case in the RFC.
In any case, I think renaming name into doctest_name would be better. What do you think?

Good catch, you can see that key-values only only certain values here.

I personally think name is better, just because it's short, and the people who use this feature will likely name every doctest, so they'll be writing out the attribute a bunch of times. Another option would be test=NAME, which is also short.

One thing to note is that the eBNF doesn't currently allow to have key=value outside of curly braces since we followed pandoc syntax. I suppose it's ok since @notriddle approved. ;)

Then name is a valid element attribute, so I'm not sure we should keep using name for this case as it would also modify the generated HTML documentation.

This is a very nice feature!
@rfcbot concern braced-unbraced
As mentioned above, we need to decide if we'd like extend the code block attr grammar to support bare name=IDENT or if we would like to use/extend the existing[^1] pandoc syntax {...,name=IDENT,...}.
@rfcbot concern raw-idents
I guess it's implied by the use of the word Rust identifier but I'd like to raise awareness that this includes raw identifiers (prefix: r#) which is good. I think we should call this out explicitly mention this in the RFC text.
@rfcbot concern hard-errors

Multiple name=IDENT words may not appear in a single code block's info
string.

I'd like to see this clarified. I'm pretty sure we don't want to emit a hard error for duplicate names but merely a warn-by-default rustdoc lint for backward compatibility.
@rfcbot reviewed
[^1] Currently feature-gated under custom_code_classes_in_docs

Thank you for the review!

@rfcbot concern braced-unbraced
As mentioned above, we need to decide if we'd like extend the code block attr grammar to support bare name=IDENT or if we would like to use/extend the existing[^1] pandoc syntax {...,name=IDENT,...}.

I personally would prefer the unbraced name=IDENT syntax. Correct me if I'm wrong, but the pandoc braced syntax is only used for adding classes to the generated HTML, and it's also rather verbose.

@rfcbot concern raw-idents
I guess it's implied by the use of the word Rust identifier but I'd like to raise awareness that this includes raw identifiers (prefix: r#) which is good. I think we should call this out explicitly mention this in the RFC text.

I noted that raw identifiers are allowed.

@rfcbot concern hard-errors

Multiple name=IDENT words may not appear in a single code block's info
string.

I'd like to see this clarified. I'm pretty sure we don't want to emit a hard error for duplicate names but merely a warn-by-default rustdoc lint for backward compatibility.

In the same vein, should name=NOT_AN_IDENT warn-by-default as well, and not be used as a test name?

In the same vein, should name=NOT_AN_IDENT warn-by-default as well, and not be used as a test name?

Yes indeed, I forgot to mention that.

https://github.com/rust-lang/rfcs/pull/3311/commits/7f90cc4781a421d5bc5a26b486c1909a593b8602

@rfcbot resolve raw-idents

should [IDENT in name=IDENT] not be used as a test name?

That's a separate point. And I honestly don't know if it's feasible or worth impl'ing this, namely checking that doctest names are unique within the value namespace of the current module (including other doctest names).

That's a separate point. And I honestly don't know if it's feasible or worth impl'ing this, namely checking that doctest names are unique within the "#[test] namespace" of the current module (including other doctest names). Yeah, maybe we should actually check this and emit a warn-by-default lint on name clashes.

Just updated the RFC. If the idea is that name=IDENT doctests will potentially be instantiated as #[test] fn test() { … } items, then they should both be unique across doctests, but also unique across all items in the current namespace.

This feature was discussed in the previous monthly meeting, and we decided that it needed better motivation before adding it.
https://rust-lang.zulipchat.com/#narrow/channel/393423-t-rustdoc.2Fmeetings/topic/2024-12-09/near/487096843
To estimate how common this need might be, I used Crater to measure how many crates had items with more than five doctests to an item. The answer, as it turns out, is 0.6%.
https://github.com/rust-lang/rust/pull/134530


Adding new targeted action endpoint for Kudos.

Fixes #4420
Trying to merge in progress on Kudos work in smaller increments for easier review :)
@jonuy
CC: @angaither

Instead of merging all your progress into dev, you could maybe send incremental PRs into another feature branch. And then when it's done, do the final big merge from the feature branch into dev.
Could be helpful in case there are scenarios where you wouldn't want to merge an unfinished feature into dev/prod.

Ah, cool idea :) I added the help text to this PR and will merge this one, but will do the separate branch idea moving forward cause that sounds rather useful!


wiki instructions for Option 2 fix of  (3.8.0 ‐ 3.9.3) ParallelCluster Build Image Failing during Installation of Minitar Ruby Gem Dependency aren't quite right.

I tried following the option 2 instructions on the wiki page https://github.com/aws/aws-parallelcluster/wiki/(3.8.0-‐-3.9.3)-ParallelCluster-Build-Image-Failing-during-Installation-of-Minitar-Ruby-Gem-Dependency but it failed.  Here is my result and what I did that seems to work but needs validating by someone more familiar with this code section and then update wiki:
The code snippet for Option 2 shows:
## Replace below Line in InstallCinc Step
/opt/cinc/embedded/bin/gem install --local --no-document berkshelf:{{ BerkshelfVersion }}

## Replace with below line, effectively Pin the Minitar version to 0.12
/opt/cinc/embedded/bin/gem install --no-document minitar:0.12
/opt/cinc/embedded/bin/gem install --local --no-document berkshelf:{{ BerkshelfVersion }}

However when I downloaded the source for parallelcluster 3.9.3, the original line that I'm replacing is missing the --local:
/opt/cinc/embedded/bin/gem install --no-document berkshelf:{{ BerkshelfVersion }}

Nonetheless, I replaced that one line with the exact code provided.
I then activated my pcluster 3.9.3 virtual env and ran the shown
pip install ./cli
My build-image failed.
I built a new virtual environment using the same pip install command after activating the new environment, still build failure.
The error message is of the form as shown on the wiki:
"Workflow Execution ID: 'wf-0988f3c0-6389-4138-bf1a-ceffc6b10876' failed with reason: Document arn:aws:imagebuilder:us-west-2:326469498578:component/parallelclusterimage-694714c0-99ae-11ef-8f75-060511802875/3.9.3/1 failed!"
Which suggested this error is still there - but I know I've got the right  minitar now...
I know nothing about "gem install", but after talking with ChatGPT a bit, it seems adding the --local to the berkshelf install does not make sense.  It wasn't there originally, we haven't downloaded berkshelf locally that I can figure out, so I tried removing the --local for the berkshelf install.  ChatGPT tells me that berkshelf will use the minitar:0.12 we installed locally unless berkshelf specifies something specific that precludes it and since berkshelf isn't local, it will grab it from the proper remote repo.
So I changed the code to be:

              /opt/cinc/embedded/bin/gem install --no-document minitar:0.12
              /opt/cinc/embedded/bin/gem install --no-document berkshelf:{{ BerkshelfVersion }}

did the
pip install ./cli

and now my pcluster build-image actually created an ami and I can boot it standalone.  Now to deploy it.
Was this posted code actually tested?  Did I do this right?
As to moving forward with 3.10.x or 3.11.x - I'm having problems with both of those as well.
3.11.1 issue has been filed as https://github.com/aws/aws-parallelcluster/issues/6529
and
3.10.1 issue might be an issue with aws-eda-slurm-cluster CloudFormation code, but I link it here for your review in case you have seen this as well: https://github.com/aws-samples/aws-eda-slurm-cluster/issues/280

Hi @gwolski
Thank you for catching this documentation mistake.
I have updated the  https://github.com/aws/aws-parallelcluster/wiki/(3.8.0-‐-3.9.3)-ParallelCluster-Build-Image-Failing-during-Installation-of-Minitar-Ruby-Gem-Dependency now.
We did internally test the changes for these versions and they were successful. This was a documentation mistake made w.r.t code from 3.10.0 which is the version in which we added the --local flag for gem installation where we install some gems just before we install Minitar and Berkself.

Will take a look at the other issues.


PolyExponential, Spline and DMD ROMs

Pull Request Description
What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)
Closes #627
This PR adds two new surrogate model (for time-dependent) that are aimed to treat the
time-dependency explicitly:

Dynamic Mode Decomposition
PolyExponential

In addition, it fixes the NDSpline ROM that now can be used ad regressor.
What are the significant changes in functionality due to this change request?

For Change Control Board: Change Request Review
The following review must be completed by an authorized member of the Change Control Board.

[x] 1. Review all computer code.
[x] 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
[x] 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
[x] 4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
[x] 5. If significant functionality is added, there must  be tests added to check this. Tests should cover all possible options.  Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in  XML block, the node <internalParallel> to True.
[x] 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change.  The requirements and the requirements test shall be in sync.
[x] 7. The merge request must reference an issue.  If the issue is closed, the issue close checklist shall be done.
[x] 8. If an analytic test is changed/added is the the analytic documentation updated/added?

Job Test mac on 38e8655 : invalidated by @derekstucki

@mandd I addressed your comments.
In addition, I added the tests for DMD and PolyExponential and added the possibility of the multitarget option

Preparing to merge


Laravel Updater

This pull request includes updates for the recent minor version release of Laravel as well as updates to your package dependencies. Feel free to commit any additional changes to the shift-ci-v7.10.3 branch.
Before merging, you need to:

Checkout the shift-ci-v7.10.3 branch
Review all pull request comments for additional changes
Run composer update
Test your application (no tests?)

:warning: You are running version 5.3.4 of guzzlehttp/guzzle. The latest version is 6.5.3. You should review this dependency to see if you can upgrade to the latest version.

:warning: You are running version 1.0.10 of laravel/tinker. The latest version is 2.4.0. You should review this dependency to see if you can upgrade to the latest version.

:information_source: You are running an older version of Laravel. The latest version is 7.10.3. Run the Laravel 7.x Shift to upgrade to the latest version of Laravel.

:warning: You are running version 3.9.3 of ramsey/uuid. The latest version is 4.0.1. You should review this dependency to see if you can upgrade to the latest version.

:warning: You are running version 0.14.0 of league/fractal. The latest version is 0.19.2. You should review this dependency to see if you can upgrade to the latest version.


Multiclass classification

Hello, thank you for the amazing package. I am using the multi:softmax and multi:softprob. it requires me to input the num_class. it seems to be the parameter is always 1 more the class i have in the training set. therefore, when i use the softprob, it returns 1 more class than i have in my training set, i need some help interpret this "additional" class. thank you

by convention, xgboost index starts from 0, which means the classes index should be in [0,num_class -1]

Thank you. I created a dataset with my Y value starting from 0 to 17. when I use softmax, I need to specify num_class = 19, and the predict output is from 1 to 18. when I use softprob in the same case, i have 19 predict probabilities for each instance. If i use num_class = 18, xgboost always crash. I am wondering how to interpret these 19 probabilities while in my training set there is only 18 class. hope this makes my problem more clear.

I have updated the repo to avoid crash and have better error message, could you try the newest version?  If you still found problem, it would be great if you can post a small piece of code to reproduce the error(can be based on randomly generated data)

any updates? I will close this for now, please re-open if you have updates


No provider of 'aspnetcore-runtime-2.2.0-1' found.

Problem encountered on https://dotnet.microsoft.com/download/linux-package-manager/opensuse/runtime-2.2.0
Operating System: Linux openSUSE Leap - x64
Provide details about the problem you are experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.
I've tried to install the run time on opensuse leap 15 but when I ran the command: zypper update it told me:

Building repository 'packages-microsoft-com-prod' cache ................................................................................................[done]
Loading repository data...
Reading installed packages...
The following package update will NOT be installed:
libgstgl-1_0-0
Nothing to do.

and the zypper install aspnetcore-runtime-2.2.0-1 command told me:

Loading repository data...
Reading installed packages...
'aspnetcore-runtime-2.2.0-1' not found in package names. Trying capabilities.
No provider of 'aspnetcore-runtime-2.2.0-1' found.
Resolving package dependencies...
Nothing to do.

@dagood this seems to be an install issue. Do you know what could be happening here?

Thanks for the report.
The zypper update output doesn't seem like it'd affect the results at first glance, but the dot.net page has a bad zypper install command. The part of the package name that specifies a specific version is incorrect: the last bit of the string should be 2.2-2.2.0-1, not 2.2.0-1. I've filed an internal issue to get that fixed. For now, if you need a specific version, you can make that tweak in your install command.
However, 2.2.0 is way out of date! To get the latest, go to https://dotnet.microsoft.com/download/linux-package-manager/opensuse/runtime-current.
The install command there is simpler since you are grabbing the latest version, and it's correct:
sudo zypper install aspnetcore-runtime-2.2

Thank you for your response. In the meantime I used the Software Management in YaST and thankfully the source is registered there too and it had led me automatically to the latest version.


[wip] feat: context menu

I was starting to create the context menu. However, it seems to be a lot intrusive with the bar below:

What's your opinion?
/cc @alanshaw @fsdiogo @lidel @olizilla @akrych

Is right-click on a file checking the checkbox and toggling the bar?
My intuition for context-menu was to be ad-hoc command for a single file that does not touch checkbox logic.

@lidel yes, it's checking the checkbox

The right-click shouldn't tick the checkbox, that way I think it would work.
On the other hand, if the user ticks the checkbox and then right clicks, well, that's on him.

So, the context menu should must affect one file?
On Windows at least, it affects every selected file.

That's also how Google Drive and Onedrive work. Dropbox doesn't have context menu at all. It has a tree dots option like we have on Invision. Maybe we should stick the Context Menu for those three dots on the left of the icon and not have a right-click context menu. What's your thoughts?

Something like that ^

The Invision designs have the three dots on the left side near the checkbox button but I thought it already had too many actions and places to click on the left side so I decided to put it on the right. Maybe if the tree dots were rotated 90 deg it would also look nice

Good call @hacdias, the three horizontal are pretty familiar and look good!

Also, updated the breadcrumbs to be more similar to IPLD Explorer's.


Add support for GZip compression

This change addresses issue #96.
When evaluating Qdrant performance on test data with medium to large payloads, network overhead became a dominating factor, even with QPS not at crazy-high levels. This should help alleviate this problem, as the payloads are highly compressible.

I didn't see any GitHub workflows run for this but otherwise, feel free to land 🙂

I didn't see any GitHub workflows run for this but otherwise

There should be :thinking:

Not sure why CI didn't trigger either, but all tests pass locally! :green_circle:

I had to do a rebase, so I moved PR here: https://github.com/qdrant/rust-client/pull/99


Method not allowed

We've had Boarding set up and working well for the last 2 months or so, however, all of the sudden it has started throwing Method Not Allowed errors when users attempt to sign up.
See attached.

Hey @jabeler, could you also paste the output from your logs/terminal?

Looks like the endpoint for https://github.com/fastlane/fastlane/blob/master/spaceship/lib/spaceship/test_flight/client.rb#L167 may have changed to POST. Looking into this as well.

Confirmed that the user gets created at the app level, but the failure occurs when adding an existing user to a testing group via spaceship. I'll try swapping to POST and report back.

Changed to POST, but now getting a new error
Creating a new tester<EMAIL_ADDRESS>- Test User
#<Spaceship::Client::UnexpectedResponse: <?xml version="1.0" encoding="UTF-8"?>
<String>There is no route for the path 'user/internalTesters/REDACTED_APP_ID_HERE/'.</String>
>
/gems/gems/fastlane-2.53.1/spaceship/lib/spaceship/client.rb:588:in `parse_response'
/gems/gems/fastlane-2.53.1/spaceship/lib/spaceship/tunes/tunes_client.rb:1094:in `testers_by_app'
/gems/gems/fastlane-2.53.1/spaceship/lib/spaceship/tunes/tester.rb:121:in `all_by_app'
/gems/gems/fastlane-2.53.1/spaceship/lib/spaceship/tunes/tester.rb:129:in `find_by_app'
/boarding/app/services/boarding_service.rb:120:in `find_app_tester'
/boarding/app/services/boarding_service.rb:44:in `add_tester'
/boarding/app/controllers/invite_controller.rb:92:in `create_and_add_tester'
/boarding/app/controllers/invite_controller.rb:75:in `submit'

If you look at the response from the actual web page, you can find a javascript file called manage_users_services.js,  this has a bunch of end points defined (https://itunesconnect.apple.com/itc/js/ng-app/services/manage_users_services.b0586ced5d.js)

Looks like Apple changed up their API 🙁

Trying to find a replacement endpoint for getting testers by appId. I think it's by groupId now:
https://itunesconnect.apple.com/testflight/v2/providers/IDKWHATNUMBERTHISIS/apps/APPID/groups/GROUPID/testers?order=asc&sort=status

yeah @phaseOne I think you're right, that comes from: https://itunesconnect.apple.com/olympus/v1/session
	"provider": {
		"providerId": <id>,
		"name": "<team name>",
		"contentTypes": ["SOFTWARE"]
	},

I think there's a helper somewhere that gets the correct groupId by ITC.apps.tf.labels.internalTesters?

yeah, you can get a list of groups at https://itunesconnect.apple.com/testflight/v2/providers/<team_id>/apps/<app_id>/groups
Then, you can use the group_id here:
https://itunesconnect.apple.com/testflight/v2/providers/<team_id>/apps/<app_id>/groups//testerEmails

Might help us: https://itunesconnect.apple.com/itc/bower_components/tf/dist/tf.tidbits.min.ed87a3f149.js

Working on some fixes for these issues here: https://github.com/fastlane/fastlane/pull/10921


Remove 'Detach' action for attached storage accounts under Quick Access node

Storage Explorer Version: 1.14.0-dev
Build: 20200602.3
Branch: master
Platform/OS: Windows 10/ macOS Mojave
Architecture: ia32/x64
Regression From: Not a regression
Steps to reproduce:

Attach one storage account via 'SAS' or 'Name and Key'.
Add the attached storage account to Quick Access.
Check the context menu of the attached storage account under Quick Access node.

Expect Experience:
No 'Detach' action displays.
Actual Experience:
The action 'Detach' displays well.

Fix merged to main

Fix merged to main