runtime/debug: add SetCrashOutput(file *os.File)

Currently there are difficulties with nicely capturing crash output from Go programs. Crashes print to stderr, however, usually both stdout and stderr are used by programs for other purposes. While it's not wrong to output it to stderr, it mixes two outputs together making later separation more difficult.
Capturing crashes is useful for post-mortem debugging and sending reports. Of course, due to unknown crashed program state using a custom callback is understandably dangerous.
A simpler alternative would be to allow changing file descriptor where to print crashes. In the simplest form it could look like:
package main
import "runtime/debug"

func init(){
	debug.SetCrashOutputFD(3)
}

This would allow passing in a fd for separate file, pipe to another "crash monitor" process or connect to an external server. Of course there's a slight possibility of write taking a long time, when passing in a pipe/connection slowing down the crashing of the program.
With regards to naming it could also be SetCrashFD, SetTracebackFD  or  SetTracebackOutputFD.

I would say, yes, to including other runtime output as well. If there's a need for additional separation it can be always introduced later.
I think the fatal output can be detected when the fd write closes due to the program stopping.

@egonelbre how do you propose to use your API on Windows?
I agree. I wanted this feature to capture crash dumps of a Windows service.
/cc @zx2c4
Alex

I've handled this until now with defer/recover in each go routine, which is really very subpar and doesn't handle all crashes.
Rather than setting an output fd, what about setting an output callback? There'd be certain restrictions on what it could do, obviously, but it could still probably accomplish most things that people want. In my case, it'd just copy the bytes to some mmap'd ring buffer log file.

@alexbrainman it would behave in a similar manner that you can provide a fd that outputs to a file or a pipe. Flushing it to Event Log would require some external "crash reporting" service that reads that file or pipe. This isn't ideal, but would allow better handling than what can be currently done. Ideal scenario would be to write directly to event-log, however that would probably require specialized machinery.
@zx2c4 I believe the callback has been proposed a few times, however I cannot find the exact proposals. I think the fundamental issue is that you don't know really much about the crashed system - what's broken and what's not. e.g. maybe you got a panic during sigprof where some code holds a runtime lock "x", which your code needs and you'll deadlock when calling file.Write. This is all hypothetical of course.

what's not. e.g. maybe you got a panic during sigprof where some code holds a runtime lock "x", which your code needs and you'll deadlock when calling file.Write. This is all hypothetical of course.

Right. If you're using debug.SetCrashCallback, then you are signing up for that kind of fun, and it's up to to write Go code that calls the minimal set of things needed. If the runtime can already do that for spitting to a fd, so can a custom handler function.

Ideal scenario would be to write directly to event-log, however that would probably require specialized machinery.

Writing to event log isn't too bad. It'd support the callback-based approach I mentioned.
(In my case, log writes look like this: https://git.zx2c4.com/wireguard-windows/tree/ringlogger/ringlogger.go#n105 )

Right. If you're using debug.SetCrashCallback, then you are signing up for that kind of fun, and it's up to to write Go code that calls the minimal set of things needed. If the runtime can already do that for spitting to a fd, so can a custom handler function.

This means no heap allocation, no stack splits, no map access (IIRC). Not impossible, but a pretty high bar to set for an API.

This means no heap allocation, no stack splits, no map access (IIRC), particularly if you want this to cover runtime throws. Not impossible, but a pretty high bar to set for an API.

But also not so out of place for something in debug, right? We're offering some way to hook into runtime internals, and with that comes the responsibilities of being in the runtime. But maybe there's a better idea here:
The other way of doing this might be to have an unstable and unexported function runtime.setCrashHook, which can then be used by wrapper packages like:
//go:linkname setCrashHook runtime.setCrashHook

That then represents the lowest level. At a higher level, then the debug package can use that to implement things like debug.SetCrashFD and the Windows service package can implement things like svc.SetCrashEventlog. And then most users use these normal functions that our libraries can provide.
Meanwhile, insane users like me can dip down into runtime.setCrashHook, along with all of its dangers, for hooking it up my custom ringbuffer mmap'd logger thing, knowing full well that if anything goes wrong, this is unsupported and my fault, etc.

The restrictions on an output callback would be severe, as @prattmic says, and failure modes would be unpredictable.  In Go we try to avoid that kind of deeply unsafe operation.
I think it would be better to keep this issue focused on the much safer operation of setting a file descriptor.
That said, I don't understand what would happen if the descriptor is a pipe that nothing is reading from.  Or a file opened on a networked file system that is not responding.  What should do in cases like that?

The restrictions on an output callback would be severe, as @prattmic says, and failure modes would be unpredictable. In Go we try to avoid that kind of deeply unsafe operation.

What do you think of my proposal above of allowing this to be a general callback via go:linkname -- i.e. just for libraries? That's a mere "implementation detail", but it'd go a long way of enabling this to be extensible for the adventurous.

That said, I don't understand what would happen if the descriptor is a pipe that nothing is reading from. Or a file opened on a networked file system that is not responding. What should do in cases like that?

I'm not sure we need to worry about those (beyond documentation, perhaps). stderr could already be any of those kinds of descriptors, as set by the parent process, so the same problems would exist today.

That said, I don't understand what would happen if the descriptor is a pipe that nothing is reading from. Or a file opened on a networked file system that is not responding. What should do in cases like that?

I guess this is the main danger with the proposal. As @prattmic mentioned, somebody could pipe stderr to somewhere that isn't being read. I didn't test it, but I think that would block in the same way.
Using a non-blocking write that drops data when the fd is not responding would be nice, however I suspect that would be difficult to implement.
It'll definitely need examples how to write the listening side.
Other than that, I don't have any further ideas.

This might be still difficult due to the constraints, but if the writing is on a different thread and the previous write hasn't finished in appropriate time, the write can be dropped. The threshold could be configured via GODEBUG or similar.

Should we just have GODEBUG=debugfd=3?
/cc @aclements

@zx2c4 I'm not fond of documenting a callback to use with go:linkname either.  I tend to think that people who operate at that level can customize their Go distribution.

I'd be okay with GODEBUG=debugfd=3. It's not straightforward to implement, though, since we don't distinguish between runtime output and anything else printed by print(ln). It would actually be much easier to implement just for tracebacks because we could use something like the gp.writebuf redirection, just to an FD instead. But maybe that's better anyway?
I agree with @alexbrainman that I'm not sure how this would work on Windows. Internally, we treat any "fd" other than 1 or 2 as a raw Windows handle value, but I don't know if such a thing can be meaningfully passed as an environment variable. Is it possible for a parent process to create a handle and pass it to a child like this?
On the topic of a callback, in addition to the subtle and rather onerous restrictions on the callback, those restrictions can also change with new releases. We don't expose anything remotely like that right now.

I did think about GODEBUG, but I suspect that would be complicated to use in Windows services since you don't have a nice way to figure out the fd prior to starting it. The main process needs to respond to messages (https://github.com/golang/sys/blob/ef89a241ccb347aa16709cf015e91af5a08ddd33/windows/svc/example/service.go#L23).
With debug.SetCrashFD allows the main process can start a "monitor process" and set the fd accordingly, while the main process can handle the "service" responsibilities.

@alexbrainman it would behave in a similar manner that you can provide a fd that outputs to a file or a pipe. Flushing it to Event Log would require some external "crash reporting" service that reads that file or pipe. This isn't ideal, but would allow better handling than what can be currently done. Ideal scenario would be to write directly to event-log, however that would probably require specialized machinery.

@egonelbre I don't understand what you are suggesting.
If I have a Windows service written in Go, how can I redirect its crash dump to a file by using debug.SetCrashOutputFD(3) ? Let's say I want crash dump file to be called c:\a.txt.
Similarly, I don't see how using debug.SetCrashOutputFD(3) would help me write to Event Log.
Maybe we can use file path instead of file descriptor? Like debug.SetCrashOutput("/tmp/a.txt").
Alex

@alexbrainman , for writing to a file, I imagine you would open the file and then pass that handle to debug.SetCrashOutputFD (or whatever). I don't know how the event log works, but I imagine you'd start a second process with a pipe between them and the second process would be responsible for writing to the event log.

debug.SetCrashFD allows the main process to start a "monitor process" and set the fd accordingly, while the main process can handle the "service" responsibilities.

Is there a reason this can't be done the other way around? The first process is the monitor process and opens the FD, then starts the second process with the GODEBUG environment set. Does this not work for Windows services?
(I'm not necessarily opposed to having a runtime/debug function for this, just trying to understand the space of constraints. One nice thing about the environment variable is that it can catch very early panics, like during package init.)

@zx2c4 I'm not fond of documenting a callback to use with go:linkname either. I tend to think that people who operate at that level can customize their Go distribution.

Yea, documenting it is probably the wrong thing to do. But having something in there akin to nanotime would be "very nice"...and would allow me to implement something for Go's Windows service package that uses it to ship things to eventlog automatically.
Anyway, I realize pushing this too hard is an loosing battle, for good reason. But if somehow the implementation happened to be structured in a way that made it coincidentally possible to go:linkname it from elsewhere, I would be very happy.

Is it possible for a parent process to create a handle and pass it to a child like this?

You can make a handle inheritable, and then pass its value as an argument to the new process. The value will remain the same in the new process. Alternatively, DuplicateHandleEx allows the one process to duplicate a handle into another process.

@alexbrainman, so roughly what I'm thinking is the following. However, I might also forget some internal details that could make the following difficult with Windows.
The most basic thing is to send the output to a separate file:
package main

import (
	"os"
	"runtime/debug"
)

func main() {
	f, err := os.Open("service_log.txt")
	if err != nil {
		os.Exit(1)
		return
	}

	debug.SetCrashFD(f.Fd())

	// rest of the service logic
}

The next step would be to create a watchdog sub-process:
package main

import (
	"os"
	"os/exec"
	"runtime/debug"
)

func main() {
	r, w, err := os.Pipe()
	if err != nil {
		os.Exit(1)
		return
	}

	// this could also self-start, but in a different mode
	// but we'll use a separate binary for now
	cmd := exec.Command("watchdog.exe")
	cmd.Stdin = r
	// also set other flags here to ensure that the watchdog
	// doesn't immediately close together with the parent.
	cmd.Start()

	debug.SetCrashFD(w.Fd())

	// rest of the service logic
}

You could also create governors using ExtraFiles:
package main

import (
	"os"
	"os/exec"
	"runtime/debug"
)

func main() {
	r, w, err := os.Pipe()
	if err != nil {
		os.Exit(1)
		return
	}

	cmd := exec.Command("watchdog.exe")
	cmd.ExtraFiles = []*os.File{w}
	cmd.Start()

	// monitor `r`
}

// The other process has `debug.SetCrashFD(3)` in init,
// with some check that `3` actually exists.

For windows, I think (named) pipes could also be used:
f, err := os.Open(`\\.\pipe\randomhexvalue`)
...
debug.SetCrashFD(f.Fd())

The first process is the monitor process and opens the FD, then starts the second process with the GODEBUG environment set. Does this not work for Windows services?

Yes, it could work. The main drawback is that windows services need to respond to control messages. https://pkg.go.dev/golang.org/x/sys/windows/svc#Handler
This would mean that the governor process would need to delegate all those messages to the subprocess, which would introduce an additional complication. Although, there might be a way to make subprocess handle the service messages directly, I haven't seen it.

Avoid using named pipes if you can; they're a security landmine. Instead pass a handle to a child process, as in your other example.
However, I still suspect there's a way that we can log things directly to the eventlog even during a panic, without the need to launch a persistent watchdog process. That's what I was proposing doing with go:linkname above.
But if you're really after something maximally robust, a better bet is to share a file mapping -- CreateFileMapping/MapViewOfFile -- and then have the error handle simply write bytes to an address, and voila, they'll be logged. And if you have a watchdog process monitoring the parent process (via WaitForSingleObject(OpenProcess(...)) or via NotifyServiceStatusChange), it can check this shared memory region after termination.
Alternatively, if you want to forgo the child process entirely, you can actually have a crash reporter process spin up via service triggers if the crashing process is able to raise an ETW event. Check out SERVICE_TRIGGER_TYPE_CUSTOM. That's a bit clumsy though, and requires you to be able to successfully call the ETW api from a crashing process.
Perhaps the more robust way to go about this would be to use the built-in Windows Error Reporting functions. Your program could use WerRegisterMemoryBlock or WerRegisterFile to set up the Go panic output destination, and then WER could collect the info there. Then, Go's panic handler would write to that file, or more reliably, write into that registered memory block. You can then pick this up with the normal default WER crash dump files, or better, have your application use WerRegisterRuntimeExceptionModule prior, which will then load a custom DLL of your choosing to then do something useful with the memory block (containing the Go panic output) registered prior.
In other words, if what you're after with this proposal is more robust Windows crash dumps, I think there's actually a lot of room for improvement all over.

Avoid using named pipes if you can; they're a security landmine. Instead pass a handle to a child process, as in your other example.

Fair enough.

In other words, if what you're after with this proposal is more robust Windows crash dumps, I think there's actually a lot of room for improvement all over.

It was meant as a more general purpose, something that would be useful on unix as well, not just windows.
It would improve the windows service situation. And, I do agree that avoiding subprocess and using the OS api would improve it even more. But, it seems that would be a different proposal.
Although, now thinking of it... maybe, a windows service could use os.Open() set the crash fd and use WerRegisterFile on it. That way it would end up getting the info.

Although, now thinking of it... maybe, a windows service could use os.Open() set the crash fd and use WerRegisterFile on it. That way it would end up getting the info.

(Assuming that the Go process is in a position to jump to WriteFile to actually write to the handle. WerRegisterMemoryBlock or similar is probably the more robust route.)

I suppose a program can always do this by re-execing itself with standard error mapped to some descriptor.  Or you can use a wrapper program to do this.
Is it ever useful to do this other than when starting the program?

Oh, you know if all you're trying to do is dup2 over stderr on Windows, you can already hack the panic and println output fd like this:
	f, _ := os.Create(`C:\helloworld.txt`)
	windows.SetStdHandle(^uint32(11), windows.Handle(f.Fd()))
	panic("oh nose!")

That works reasonably well, even inside of services:

Changing the fd isn't particularly difficult, as you can see there, which is why I keep poking at, "what about running custom logger functions?", which is the much more interesting goal (to me, at least).

The issue is not just about changing the fd where to output, but also cleanly separating crash&debug output from both stdout and stderr.

Changing the fd isn't particularly difficult...

The issue is not just about changing the fd where to output, but also cleanly separating crash&debug output from both stdout and stderr.

In the context of a windows service, why do you care about stdout and stderr? They're essentially not really there. Or are you not actually operating within a windows service?

In the context of a windows service, why do you care about stdout and stderr? They're essentially not really there. Or are you not actually operating within a windows service?

It's about regular linux programs, linux services/servers, windows programs and windows services/servers. Essentially, the problem is not restricted to windows service. See https://github.com/golang/go/issues/42888#issuecomment-736061869 as a good example for non-windows-service.

I just committed this for experimentation for my own stuff: https://git.zx2c4.com/wireguard-windows/commit/?id=6753ac7de518ee2ad58e6a2cd9f367cbf2e34ad6
The general idea is that I added a little hook to time_nofake.go:
var overrideWrite func(fd uintptr, p unsafe.Pointer, n int32) int32

// write must be nosplit on Windows (see write1)
//
//go:nosplit
func write(fd uintptr, p unsafe.Pointer, n int32) int32 {
        if overrideWrite != nil {
                return overrideWrite(fd, noescape(p), n)
        }
        return write1(fd, p, n)
}

Then, in my ringlogger global init function, I set that function pointer using go:linkname:
var Global *Ringlogger

//go:linkname overrideWrite runtime.overrideWrite
var overrideWrite func(fd uintptr, p unsafe.Pointer, n int32) int32

func InitGlobalLogger(tag string) error {
	if Global != nil {
		return nil
	}
	root, err := conf.RootDirectory(true)
	if err != nil {
		return err
	}
	Global, err = NewRinglogger(filepath.Join(root, "log.bin"), tag)
	if err != nil {
		return err
	}
	log.SetOutput(Global)
	log.SetFlags(0)
	overrideWrite = globalWrite
	return nil
}

Finally, that points to globalWrite, which is a super clunky fixed buffer buffering mechanism:
var globalBuffer [maxLogLineLength - 1 - maxTagLength - 3]byte
var globalBufferLocation int

//go:nosplit
func globalWrite(fd uintptr, p unsafe.Pointer, n int32) int32 {
	b := (*[1 << 30]byte)(p)[:n]
	for len(b) > 0 {
		amountAvailable := len(globalBuffer) - globalBufferLocation
		amountToCopy := len(b)
		if amountToCopy > amountAvailable {
			amountToCopy = amountAvailable
		}
		copy(globalBuffer[globalBufferLocation:], b[:amountToCopy])
		b = b[amountToCopy:]
		globalBufferLocation += amountToCopy
		foundNl := false
		for i := globalBufferLocation - amountToCopy; i < globalBufferLocation; i++ {
			if globalBuffer[i] == '\n' {
				foundNl = true
				break
			}
		}
		if foundNl || len(b) > 0 {
			Global.Write(globalBuffer[:globalBufferLocation])
			globalBufferLocation = 0
		}
	}
	return n
}

The invocation there of Global.Write writes into the mmap'd ringbuffer without allocations.
So far in testing, this scheme works well. It's not pretty, but it does work.

As a data point, the gVisor project wants to redirect panic/throw output. It works around this problem by dup'ing over stderr with the FD of the desired crash output location (https://cs.opensource.google/gvisor/gvisor/+/master:runsc/cli/main.go;l=194-198?q=dup3&ss=gvisor%2Fgvisor). stderr is otherwise unused by the program, so this works OK. This is a workable solution, but it would certainly be nicer to be able to change the destination directly.

cc @steeve coz of https://github.com/znly/go/commit/6af49debb0b604c1bfe209d5801f985afb327c72

As a data point, the gVisor project wants to redirect panic/throw output. It works around this problem by dup'ing over stderr with the FD of the desired crash output location (https://cs.opensource.google/gvisor/gvisor/+/master:runsc/cli/main.go;l=194-198?q=dup3&ss=gvisor%2Fgvisor). stderr is otherwise unused by the program, so this works OK. This is a workable solution, but it would certainly be nicer to be able to change the destination directly.

cc @steeve coz of znly@6af49de

Hooking the panic itself like that is a lot cleaner than my trick of hooking the write1 function, and probably allows for capturing more info (such as logging for all go routines).

@alexbrainman , for writing to a file, I imagine you would open the file and then pass that handle to debug.SetCrashOutputFD (or whatever).

Windows uses syscall.Handle to wrote to file, not file descriptors. syscall.Handle values are meaningless outside of their process. For example, stderr syscall.Handle is not 3.

I don't know how the event log works,

See https://godoc.org/golang.org/x/sys/windows/svc/eventlog#Log for details. It uses ReportEvent Windows API.
I am OK if Event Log is not supported for crush dumps. If I could get my crash dumps in a file of my choice, that would be good enough for me.

but I imagine you'd start a second process with a pipe between them and the second process would be responsible for writing to the event log.

True.
But normal services are not structured this way. So you would need to change your program. And, if you program is crashing now, you just want to see the crash rather than start restructuring it into 2 processes.
And one process is simpler to program, then 2 processes communicating with each other. Like @zx2c4 said Windows service process (the first process that Windows runs when it starts your service) receives events from Windows, and it needs to reply to these events. So the processes would have to communicate with each other. And I agree with @zx2c4 that using pipes between processes can be tricky - it needs to be 2 way communication because of events received.
And the "monitor" process can still crash. How do you deal with that?
@egonelbre

The most basic thing is to send the output to a separate file:

Yes, writing to a file sounds fine. But then we should just use debug.SetCrashOutput("/tmp/a.txt").

The next step would be to create a watchdog sub-process:

See my objections above about converting single process into 2 processes.

You could also create governors using ExtraFiles:

Does ExtraFiles work on Windows?

// The other process has debug.SetCrashFD(3) in init,
// with some check that 3 actually exists.

I don't think 3 as syscall.Handle value will work. Do you think you can pass 3 to WriteFile Windows API?

For windows, I think (named) pipes could also be used:

Perhaps. Again. For this to work properly, someone needs to be reading from that pipe. Sounds too complicated for me.

Oh, you know if all you're trying to do is dup2 over stderr on Windows, you can already hack the panic and println output fd like this:
f, _ := os.Create(C:\helloworld.txt)
windows.SetStdHandle(^uint32(11), windows.Handle(f.Fd()))
panic("oh nose!")
That works reasonably well, even inside of services:

This is nice. Looks simple enough. Hopefully I will remember this trick next time I debug crashing service.
Alex

Does ExtraFiles work on Windows?

It does not.

I don't think 3 as syscall.Handle value will work. Do you think you can pass 3 to WriteFile Windows API?

Yes, passing handle doesn't work. The debug.SetCrashFD was an example on platforms that support ExtraFiles and passing fd.

And one process is simpler to program, then 2 processes communicating with each other.

Yes, I agree.
All in all, I do agree that this proposal is not sufficient for making good crash handling for Windows Services.
With regards to using debug.SetCrashOutput("/tmp/a.txt"), this would mean that other platforms that do support ExtraFiles  wouldn't be able to use a fd. (See gvisor example.)

Does ExtraFiles work on Windows?

It does not.

I wonder why not. StartProcess can pass Files just fine. Do you know what's different about ExtraFiles? Seems like that's a bug we should fix.

@zx2c4 there's some more information in https://github.com/golang/go/issues/21085.

All in all, I do agree that this proposal is not sufficient for making good crash handling for Windows Services.

debug.SetCrashFD might be OK as an API. But I worry that Linux developers will write debug.SetCrashFD(3) and this will compile on Windows, and Windows developer would not know why this function does not work.

With regards to using debug.SetCrashOutput("/tmp/a.txt"), this would mean that other platforms that do support ExtraFiles wouldn't be able to use a fd. (See gvisor example.)

Fair enough.
Alex

OK, well to circle back a bit, it sounds like the environment variable is not a great idea, at least not by itself, and that we should be thinking about debug.SetRuntimeFD(uintptr), which will accommodate a Unix fd or a Windows handle.
I believe this would apply to all runtime-generated prints (including things like GC traces) but not to user-generated ones (print and println).
And then perhaps on top of the SetRuntimeFD we might also want to allow GODEBUG=fd=N but that would probably only be useful on Unix.
Do I have that right?

Just noting that if we separate runtime prints from application prints we'll have to decide how to handle runtime.printBacklog.  It currently gets both runtime prints and application prints, but maybe getting just runtime prints is better anyhow.

Do I have that right?

From my point of view yes.
Also noting that Windows Services will need a different design to handle crashes, but that would be a different proposal.

Do I have that right?

From my point of view yes.
Also noting that Windows Services will need a different design to handle crashes, but that would be a different proposal.

Why do we require a different design for Windows Services?
I've proposed above several such things that work in a wide variety of cases, including Windows Services.

Why do we require a different design for Windows Services?
I've proposed above several such things that work in a wide variety of cases, including Windows Services.

I'll try to summarize the discussions, but let me know if I missed some option/opinion:
A) Setting a callback directly, debug.SetPanicCallback(func() { ... })
Given the restrictions that the func needs to adhere to it's a really unsafe operation that's usually not exposed. (https://github.com/golang/go/issues/42888#issuecomment-736625441, https://github.com/golang/go/issues/42888#issuecomment-737338402)
B) Having a standard callback via linkname that can be used to hook into.
While it hides the unsafe API, however it's still something not usable for most users. I agree that writing such func is possible, however, but it's more likely people will get the usage wrong. (https://github.com/golang/go/issues/42888#issuecomment-737437206)
C) Changing std fd using windows.SetStdHandle
This doesn't separate runtime printing from user prints to stderr. It would be probably possible to create os.Stderr that is different from handle 2, basically making it difficult to use. I'm not sure what other restrictions this approach might have, e.g. maybe UWP?

Just to clarify, I agree that callback is more powerful and solves the problems more widely, but due to unsafe nature, it seems a dangerous to expose it for common use. So, I'm not opposed to the idea, but it seems like a different proposal on how to exactly do that.

@egonelbre, your list does not include the option I mentioned above, namely debug.SetRuntimeFD(uintptr). That would separate user prints from the Go-runtime-originated stderr, which it sounds like is what Windows Services needs. So I think debug.SetRuntimeFD does work for the Windows Services case.
Am I missing something? We probably don't want to have to revisit this to do something different for Windows.

@rsc, I think @alexbrainman's comment above explained why debug.SetRuntimeFD is poorly suited to Windows services. My takeaway was that, while it's not impossible, Windows expects a particular process structure for services that makes having a second process just for funneling the FD into the event log awkward.
I'm somewhat loathe to propose this, but: on Android, we have logic in the runtime that sends all prints to the syslog because it has a similar issue where stderr is just thrown away. I hate how complex that logic is for such a fundamental operation, but maybe it would make sense to do something similar for Windows services and have their writes go straight to the event log? IMO, in both cases this should be the OS's job, but it's not doing it.

... your list does not include the option I mentioned above ...

@rsc the list was a response to @zx2c4 and why his proposals didn't seem feasible to me at the moment, and not the list of all proposals in the discussion.

So I think debug.SetRuntimeFD does work for the Windows Services case.

Yes, I agree that it improves the situation. Being able to redirect the crash to a file is a step up.
I mentioned Windows Service solution as insufficient, because the conventional approach is to write to EventLog. Sometimes it's also very useful to include other metadata about the system state during the crash.
Of course, the different needs of Windows Service don't invalidate the usefulness of debug.SetRuntimeFD on Linux and Mac.
Adding a special case for Windows Services to write to EventLog also sounds reasonable. It would be safer than a callback. Though, it would require a way of setting or automatically deciding on an event source name.

I hate how complex that logic is for such a fundamental operation, but maybe it would make sense to do something similar for Windows services and have their writes go straight to the event log? IMO, in both cases this should be the OS's job, but it's not doing it.

Sticking that into the Go runtime itself is not a good idea. EventLog is complicated -- more so than the Android logger functions -- and as such will require configuration. This is the kind of thing best suited to go in x/sys/windows/svc or similar. The x/sys/windows package already uses //go:linkname for necessary faculties exposed by the runtime. Adding one for setting the logger callback function would be very sensible. I'll send a patch for that.

@zx2c4 but what does the x/windows/svc side look like?

@egonelbre, your list does not include the option I mentioned above, namely debug.SetRuntimeFD(uintptr). That would separate user prints from the Go-runtime-originated stderr, which it sounds like is what Windows Services needs. So I think debug.SetRuntimeFD does work for the Windows Services case.
Am I missing something? We probably don't want to have to revisit this to do something different for Windows.

For Windows Service debugging we only need a tool to write stack trace somewhere other than stdout and stderr (simply because Windows Service does not have stdout or stderr). As @zx2c4 showed https://github.com/golang/go/issues/42888#issuecomment-737616766 it is not difficult to redirect stderr into a file. So, I guess, this should work for debuging Windows Service.
debug.SetRuntimeFD(uintptr) will work for other scenarios on Windows. The only problem is how do you propose to document what debug.SetRuntimeFD values will be? On Unix values of 0, 1, 2 refer to stdin, stdout and stderr. On Windows handle values of 0, 1 and 2 are meaningless.

Adding a special case for Windows Services to write to EventLog also sounds reasonable.

EventLog is complicated -- more so than the Android logger functions -- and as such will require configuration.

I agree with both @egonelbre and @zx2c4 EventLog is not simple to use and complicated enough to configure. But, on the other hand, it is convention to write any Windows Service crash into EventLog. So it would be nice if we could provide some tool (perhaps as part of golang.org/x/sys/windows/svc package) to write crash dump to EventLog. Perhaps user can call some function to get that effect in production.

@zx2c4 but what does the x/windows/svc side look like?

Yes @zx2c4, https://golang.org/cl/278792 is just a hook in runtime. How do you propose to use it for Windows Service?
Thank you.
Alex

debug.SetRuntimeFD(uintptr) will work for other scenarios on Windows. The only problem is how do you propose to document what debug.SetRuntimeFD values will be? On Unix values of 0, 1, 2 refer to stdin, stdout and stderr. On Windows handle values of 0, 1 and 2 are meaningless.

There are few possible approaches:

In a similar way to debug.WriteHeap and os.NewFile.

Or in other words, it is a file descriptor on unixy platforms and handle on windows.
i.e. you should get the Fd from *os.File for it to be valid and then all examples show:
debug.SetRuntimeFd(os.Stderr.Fd())

Special case 0-2 for Windows (with additional ExtraFiles once it's supported).

I'm not sure how often Windows gives out those as handles. I didn't find information on any guarantees on the values.
Assuming there aren't any guarantees, then at some point in might fail in a weird way.
Given that there isn't a big benefit over using things like os.Stderr.Fd() it might not be worth the risk.

Change the API to accept *os.File instead of fd.

This would make it easier to use correctly. Currently, runtime/debug already depends on os so it won't introduce a new dependency either. Using TCP connections or other handles would still be possible.
debug.SetRuntimeOutput(os.Stderr)

conn, err := net.DialTCP(...)
connfile, err := conn.File() // I didn't verify whether this is supported on Windows
debug.SetRuntimeOutput(connfile)

I'm not sure what the exact downsides of this approach would be.

I would exclude 2. from the possible approaches. I initially used fd because debug.WriteHeap was also using it, that way the API is more consistent. However, thinking about it, using *os.File would probably be safer.

Change the API to accept *os.File instead of fd.

@egonelbre I prefer your option 3. It makes new API usage obvious. And I could use approach similar to https://github.com/golang/go/issues/42888#issuecomment-737616766 to write Windows Service crash into a file.
Thank you.
Alex

@zx2c4 but what does the x/windows/svc side look like?

Sorry for the delay. I'll submit a CL for that so we can see what that looks like.

@zx2c4 but what does the x/windows/svc side look like?

Sorry for the delay. I'll submit a CL for that so we can see what that looks like.

Status: this issue is waiting on seeing what x/windows/svc will look like. @zx2c4 is going to send a CL.

Status: this issue is waiting on seeing what x/windows/svc will look like. @zx2c4 is going to send a CL.

@rsc - I'm really sorry for the delay! I just pushed a CL for you to see the idea I had in mind.

@rsc - I'm really sorry for the delay! I just pushed a CL for you to see the idea I had in mind.

@zx2c4

I just pushed a CL for you to see the idea I had in mind.

I take it you are talking about
https://go-review.googlesource.com/c/sys/+/283602
Generally this will work. But your change still needs design work.
For example, you write event log event for every line written by print function. If my program have more than a couple of print calls, that would create too many events.
Similarly, when printing crash dump, I expect one event written containing whole crash dump. Not having each line of crash dump recorded as a separate event.
I also expect crash dump reported as windows.EVENTLOG_ERROR_TYPE, not as windows.EVENTLOG_WARNING_TYPE as your CL implements.
Perhaps we should just restrict this facility to dumping crash dump into single event log event. And ignore print function output (prints can be redirected as per your suggestion above https://github.com/golang/go/issues/42888#issuecomment-737616766). This will simplify your implementation.
Your CL 283602 also assumes that runtime hook implemented in CL 278792 exists. But what happens, if developer is using old version of Go? You would have to make CL 283602 functionality hidden, if using old version of Go.
Thank you.
Alex

@zx2c4

I just pushed a CL for you to see the idea I had in mind.

I take it you are talking about
https://go-review.googlesource.com/c/sys/+/283602
Generally this will work. But your change still needs design work.
For example, you write event log event for every line written by print function. If my program have more than a couple of print calls, that would create too many events.
Similarly, when printing crash dump, I expect one event written containing whole crash dump. Not having each line of crash dump recorded as a separate event.
I also expect crash dump reported as windows.EVENTLOG_ERROR_TYPE, not as windows.EVENTLOG_WARNING_TYPE as your CL implements.
Perhaps we should just restrict this facility to dumping crash dump into single event log event. And ignore print function output (prints can be redirected as per your suggestion above https://github.com/golang/go/issues/42888#issuecomment-737616766). This will simplify your implementation.
Your CL 283602 also assumes that runtime hook implemented in CL 278792 exists. But what happens, if developer is using old version of Go? You would have to make CL 283602 functionality hidden, if using old version of Go.
Thank you.
Alex

I also expect crash dump reported as windows.EVENTLOG_ERROR_TYPE, not as windows.EVENTLOG_WARNING_TYPE as your CL implements.

That's fine with me and an easy thing to fix up.

Your CL 283602 also assumes that runtime hook implemented in CL 278792 exists. But what happens, if developer is using old version of Go? You would have to make CL 283602 functionality hidden, if using old version of Go.

I can use buildtags for this so that the API is only available on 1.16. I'll update the CL.

I also expect crash dump reported as windows.EVENTLOG_ERROR_TYPE, not as windows.EVENTLOG_WARNING_TYPE as your CL implements.

That's fine with me and an easy thing to fix up.

Your CL 283602 also assumes that runtime hook implemented in CL 278792 exists. But what happens, if developer is using old version of Go? You would have to make CL 283602 functionality hidden, if using old version of Go.

I can use buildtags for this so that the API is only available on 1.16. I'll update the CL.

Perhaps we should just restrict this facility to dumping crash dump into single event log event. And ignore print function output (prints can be redirected as per your suggestion above #42888 (comment)). This will simplify your implementation.

I think being able to override prints is actually a super useful feature. My suggestion above allows for a file descriptor change, but not for an arbitrary function. Specifically, that precludes event log or a lightweight ring logger, or anything else. Plus it makes the change much less invasive.
The CL is now updated. Right now that's "1.16", but we'd realistically bump that to 1.17.

Perhaps we should just restrict this facility to dumping crash dump into single event log event. And ignore print function output (prints can be redirected as per your suggestion above #42888 (comment)). This will simplify your implementation.

I think being able to override prints is actually a super useful feature. My suggestion above allows for a file descriptor change, but not for an arbitrary function. Specifically, that precludes event log or a lightweight ring logger, or anything else. Plus it makes the change much less invasive.
The CL is now updated. Right now that's "1.16", but we'd realistically bump that to 1.17.

Thinking some more, I guess one way of combining the convenience of being able to overwrite print() with the fact that panics are multiline would be to just refactor the panic printer to simply call print with a single buffer.

Thinking some more, I guess one way of combining the convenience of being able to overwrite print() with the fact that panics are multiline would be to just refactor the panic printer to simply call print with a single buffer.

Given how simple the CL is, I wonder whether the runtime really should just do this itself.
It looks like it just needs to collect a line at a time and pass it to windows.ReportEvent.
That's easy enough, provided we know which event ID to use.
Do different apps just make up their own numbers? How are they chosen?
The callback is tricky because at throw the program may be in a very corrupt state
and not really in a position where it can run arbitrary new Go code.
(Of course, most of the time it is fine.)
We could define some convention or function for setting the eid on windows, or have a special fd like -1 meaning system log.
Just listing ideas.

Given how simple the CL is, I wonder whether the runtime really should just do this itself.
It looks like it just needs to collect a line at a time and pass it to windows.ReportEvent.
That's easy enough, provided we know which event ID to use.
Do different apps just make up their own numbers? How are they chosen?
The callback is tricky because at throw the program may be in a very corrupt state
and not really in a position where it can run arbitrary new Go code.
(Of course, most of the time it is fine.)
We could define some convention or function for setting the eid on windows, or have a special fd like -1 meaning system log.
Just listing ideas.

One more observation is that redirecting the stderr entirely is problematic for the user of the program.
If the program doesn't write to standard error (only to the redirected fd) then it silently exits with a non-zero status and the user is left to wonder what has happened (or thinks it succeeded).

One more observation is that redirecting the stderr entirely is problematic for the user of the program.
If the program doesn't write to standard error (only to the redirected fd) then it silently exits with a non-zero status and the user is left to wonder what has happened (or thinks it succeeded).

@zx2c4 and @alexbrainman, any thoughts on https://github.com/golang/go/issues/42888#issuecomment-763848875 ?

@zx2c4 and @alexbrainman, any thoughts on https://github.com/golang/go/issues/42888#issuecomment-763848875 ?

I don't think it's a good idea to have the runtime call ReportEvent on its own. Most apps do not use the eventlog. Some services use the eventlog, but not all services use the ring buffer. The Go api for the eventlog is not in the Go runtime; it's in a helper auxilary module of x/sys/windows. Finally, my own services do not use the eventlog; in my case, I have a file-backed mmap'd ring buffer for this stuff where I simply copy the bytes.
Hence, I'm learning toward CL278792 as the best general solution, combined with https://github.com/golang/go/issues/42888#issuecomment-763638252 .

I don't think it's a good idea to have the runtime call ReportEvent on its own. Most apps do not use the eventlog. Some services use the eventlog, but not all services use the ring buffer. The Go api for the eventlog is not in the Go runtime; it's in a helper auxilary module of x/sys/windows. Finally, my own services do not use the eventlog; in my case, I have a file-backed mmap'd ring buffer for this stuff where I simply copy the bytes.
Hence, I'm learning toward CL278792 as the best general solution, combined with https://github.com/golang/go/issues/42888#issuecomment-763638252 .

Adding that it's possible to write programs that work both as a service and as a regular command line tool.

Adding that it's possible to write programs that work both as a service and as a regular command line tool.

Given how simple the CL is, I wonder whether the runtime really should just do this itself.
It looks like it just needs to collect a line at a time and pass it to windows.ReportEvent.

Yes we can do this. Or have option to dump stack trace to a file. Perhaps user program can call predefined runtime function to set stack trace destination: windows.ReportEvent or file.

That's easy enough, provided we know which event ID to use.
Do different apps just make up their own numbers? How are they chosen?

If you are talking about dwEventID from
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-reporteventw
it is an index to a list of format strings stored in a DLL or executable. It is difficult to explain how this works. It might be easier if you look to see how golang.org/x/sys/windows/svc/evetlog package implemented. Perhaps it would be too complicated to bring all that complexity into runtime.

The callback is tricky because at throw the program may be in a very corrupt state
and not really in a position where it can run arbitrary new Go code.
(Of course, most of the time it is fine.)

CL 283602 hides all that complexity well enough from the user. But CL 283602 uses //go:linkname trick. Maybe it is good enough.

One more observation is that redirecting the stderr entirely is problematic for the user of the program.
If the program doesn't write to standard error (only to the redirected fd) then it silently exits with a non-zero status and the user is left to wonder what has happened (or thinks it succeeded).

Windows service program does not have stderr. So anything you write to stderr goes nowhere. Similarly you cannot just exit Windows service program - the program have to call particular Windows API before exiting - otherwise Windows will get confused about Windows service state.

Most apps do not use the eventlog.

GUI and command line programs indeed do not write to event log. They display their message to the user.

Some services use the eventlog, but not all services use it.

All Windows service programs I familiar with write its error messages to the event log. If Windows service crashes, I expect crash to be written to event log (with error type) - where else would I be looking for that crash? - Windows does not have /var/log directory. Some Windows service programs write different error or non-error messages to event log (error or info or warning type).

The Go api for the eventlog is not in the Go runtime; it's in a helper auxilary module of x/sys/windows. Finally, my own services do not use the eventlog; in my case, I have a file-backed mmap'd ring buffer for this stuff where I simply copy the bytes.
Hence, I'm learning toward CL278792 as the best general solution, combined with #42888 (comment) .

I agree. For WIndows services we can do something like CL 278792, Perhaps we could provide similar facility to write crash dump to a file.
Alex

Given how simple the CL is, I wonder whether the runtime really should just do this itself.
It looks like it just needs to collect a line at a time and pass it to windows.ReportEvent.

Yes we can do this. Or have option to dump stack trace to a file. Perhaps user program can call predefined runtime function to set stack trace destination: windows.ReportEvent or file.

That's easy enough, provided we know which event ID to use.
Do different apps just make up their own numbers? How are they chosen?

If you are talking about dwEventID from
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-reporteventw
it is an index to a list of format strings stored in a DLL or executable. It is difficult to explain how this works. It might be easier if you look to see how golang.org/x/sys/windows/svc/evetlog package implemented. Perhaps it would be too complicated to bring all that complexity into runtime.

The callback is tricky because at throw the program may be in a very corrupt state
and not really in a position where it can run arbitrary new Go code.
(Of course, most of the time it is fine.)

CL 283602 hides all that complexity well enough from the user. But CL 283602 uses //go:linkname trick. Maybe it is good enough.

One more observation is that redirecting the stderr entirely is problematic for the user of the program.
If the program doesn't write to standard error (only to the redirected fd) then it silently exits with a non-zero status and the user is left to wonder what has happened (or thinks it succeeded).

Windows service program does not have stderr. So anything you write to stderr goes nowhere. Similarly you cannot just exit Windows service program - the program have to call particular Windows API before exiting - otherwise Windows will get confused about Windows service state.

Most apps do not use the eventlog.

GUI and command line programs indeed do not write to event log. They display their message to the user.

Some services use the eventlog, but not all services use it.

All Windows service programs I familiar with write its error messages to the event log. If Windows service crashes, I expect crash to be written to event log (with error type) - where else would I be looking for that crash? - Windows does not have /var/log directory. Some Windows service programs write different error or non-error messages to event log (error or info or warning type).

The Go api for the eventlog is not in the Go runtime; it's in a helper auxilary module of x/sys/windows. Finally, my own services do not use the eventlog; in my case, I have a file-backed mmap'd ring buffer for this stuff where I simply copy the bytes.
Hence, I'm learning toward CL278792 as the best general solution, combined with #42888 (comment) .

I agree. For WIndows services we can do something like CL 278792, Perhaps we could provide similar facility to write crash dump to a file.
Alex

The issue I see here is just how much we let Go programs attempt to do during a crash, in which case the runtime may be in a terrible state and really not capable of much. In particular it may not be capable of running arbitrary Go code.
Writing to a different fd is easy. Calling a reporting DLL call on Windows with a user-supplied dwEventId also seems easy. In both cases it's just one system call. I'm not sure we need to (or can reliably) support arbitrary reporting callbacks such as code to write to an mmap'ed ring-buffer file. (Especially if you can already write to a non-ring-buffer file with the fd redirect.)
@alexbrainman says that it's complicated to explain what the dwEventId is, but we don't need to do that in the runtime. We just need to accept a dwEventId and pass it along to the Windows API call.

Does anyone want to make the case against what I wrote in the previous message?
That is, does anyone want to argue that the combination of being able to redirect to an fd and being able to redirect to the Windows event log with a specific dwEventId is not sufficient to solve this problem?
Thanks.

Yes, I object to this plan. I think it's shortsighted and inflexible.
With regards to changing the fd, this is already possible:

On Unix you can redirect stderr with dup2.
On Windows you can redirect stderr with windows.SetStdHandle(^uint32(11), handle).

Maybe you want to come up with a cross-platform API for setting the fd/handle. That's fine, but kind of besides the point.
With regards to Windows services:

Some services use ReportEvent.
Many services do not use ReportEvent.

There are legitimate reasons to avoid the Windows Event Log machinery. There are also a lot of different ways to use ReportError; there are 5 arguments the user might want to customize.
Other Windows facilities use different ways of reporting error information:

WerRegisterMemoryBlock is sometimes desirable.
Different file formats.

Ultimately I think the place to put all of this complicated / changing / specific functionality is in x/sys/windows. That's where the event log functionality is, for example. A WER subpackage could be added also. The modicum of ways things are done on Windows can be implemented in x/sys/windows/*, where it belongs. It would be a mistake to try to shoehorn something as complex as event log into the Go standard library, especially considering all of the eventlog functionality lives in x/sys/windows/svc/eventlog.
Therefore, the right way to proceed is to allow low-level libraries like x/sys/windows provide this, rather than the Go standard library.
This is not unprecedented either. x/sys/windows already interacts in intimate ways with the Go runtime. Its service initialization code has some tricky assembly. Its dll loader uses go:linkname. Unsafe is widespread and necessary. This is the nature of a low-level library like x/sys/windows.
So, in order to allow x/sys/windows to implement this, all that's required is a simple unexported function hook (CL278792), which will then enable x/sys/windows to integrate intimately with it. Such a hook would not be intended to handle all cases; it wouldn't be user-facing. It would instead be an implementation detail of the Go runtime for x/sys/windows to interact with, just as x/sys/windows already interacts with specific runtime details of the Go runtime.

combination of being able to redirect to an fd...

Yes, I agree, but with the caveat that it should use os.File for the API rather than the handle directly, because of discussion near https://github.com/golang/go/issues/42888#issuecomment-749813659.
func SetRuntimeOutput(f *os.File) error {

redirect to the Windows event log with a specific dwEventId is not sufficient to solve this problem?

Here I share concern with @zx2c4 that there is too much configurability in using EventLog and exposing this from runtime will end up accumulating a lot of those variations into runtime. I agree that x/sys/windows is a better place.
Whether using linkname is the best way to implement this, I'm unsure -- however, I don't see a significantly better option either. Maybe there are ways to make it safer, such that it's guaranteed to not do the "wrong thing". Maybe some static analysis that ensures a func is safe to use as such a link-target. Overall, it does seem like Windows needs a way to integrate with the runtime output on a deeper level.

That is, does anyone want to argue that the combination of being able to redirect to an fd and being able to redirect to the Windows event log with a specific dwEventId is not sufficient to solve this problem?

That would be nice, if we could stop here.
One of the Windows services I work on is written in a way that needs to send errors and crashes to a server on the Internet. This thing is designed as 2 processes tree. Parent process is a Windows service. Parent does nothing but launches child process that does all the work. Parent also records all child output / errors and sends it to the Internet.
So this setup is good enough until you have some problem in the parent process. At this moment we just ignore any crashes in the parent process.
If your proposal is accepted, we should be able to redirect parent process crashes into event log or to a file.
It would be nice if I can send my Windows service crashes to the Internet. But I don't expect runtime code to provide this functionality.

Yes, I agree, but with the caveat that it should use os.File for the API rather than the handle directly, because of discussion near #42888 (comment).

I agree with @egonelbre here.

Here I share concern with @zx2c4 that there is too much configurability in using EventLog and exposing this from runtime will end up accumulating a lot of those variations into runtime. I agree that x/sys/windows is a better place.

I also share this sentiment. Windows service developer will look for this functionality in x/sys/windows. Something like CL 283602 should work in terms of API.
Alex

Unexported hooks are really problematic from an API standpoint, especially if widely used programs adopt them.
That can't possibly be the right approach here.
And hooks generally have the problem that the runtime may not be in a good enough state to run the code in the hook.
It is basically impossible to do anything interesting in a signal handler these days on Unix for the same reasons.
Lots of signal handlers seem to work but actually wedge the program in rare cases.
I don't want to repeat that mistake here.
It seems OK for x/sys/windows to do setup, but we really need the critical "can't do much safely" implementation to be in package runtime, where it can be adjusted and updated as the runtime changes. What is the API that x/sys/windows would need for that?

The simplest thing from an API perspective would be something like:
package runtime

func SetPanicNotifier(notify func(panicText string))

But this sounds hard to actually supply as something general purpose, but maybe that's okay?

For what's is worth, we have implemented it at https://github.com/znly/go/commit/6af49debb0b604c1bfe209d5801f985afb327c72
It works well for us and it's pretty simple:
    import (
        "fmt",
        "runtime"
    )

    func onPanic(err interface{}) {
    	   fmt.Println("panic reason:", err)
    	   dump := make([]byte, 1*1024*1024) // 1 MB buffer
    	   runtime.Stack(dump, true)
    	   fmt.Println(string(dump))
    }

    func main() {
    	   runtime.SetGlobalPanicHandler(onPanic)

    	   // provoke a panic from a SIGNAL
    	   var i *int
    	   *i = 42

    	   println("program exit normally")
    }

@steeve - Does this work for all types of panics?
One difference between yours and my proposal is that I was suggesting that instead of passing an err interface, it should instead pass the string that the panic handler is already assembling to print out to stderr. This way code that runs in the handler can be extremely simple and can avoid needing to call into additional runtime bits.
Implementation-wise, here's how I imagine that working so that it's allocation free:
Currently, the panic handler has a series of print and println calls. In my own code I'm just hooking the println function, line buffering all those calls into a pre-allocated buffer, and then dumping it into my ring buffer on each line. But in order to be more general purpose, it could instead aggregate into one buffer before any calls. So the old code is something like this:
func panicing() {
    print(....)
    print(....)
    print("\n")
    print(....)
    print(...)
}

And the new code would become something like:
var preallocatedPanicBuffer [0x1000]byte
func panicing() {
    buf := preallocatedPanicBuffer[:0]
    buf = append(buf, ...)
    buf = append(buf, ...)
    buf = append(buf, '\n')
    buf = append(buf, ...)
    buf = append(buf, ...)
    buf = append(buf, ...)
    if panicHandlerThing == nil || panicHandlerThing(noescape(buf)) {
        print(buf)
    }
}

The big problem with this approach, aside from the buffers, is that the useful output of all those printlns is spread across a bunch of functions in a bunch of files. There's no information that the Go runtime spits out that I'm comfortable discarding from my log. If it's going to stderr, I want it to go to my service ring buffer. So ultimately I wind up with something like:
package runtime

func SetBuiltinPrintHandler(text string) (alsoSendToNormalStderr bool)

Something like this serves my needs very well, but I'm not sure that this is what @rsc (or others here) were going for.
So, hm.

@steeve - Does this work for all types of panics?

As far as I can tell, most. I may be mistaken though.

One difference between yours and my proposal is that I was suggesting that instead of passing an err interface, it should instead pass the string that the panic handler is already assembling to print out to stderr. This way code that runs in the handler can be extremely simple and can avoid needing to call into additional runtime bits.

Well, to us, running non simple code actually was the whole point: allocations, IO... but most importantly being able to access the stack frames in a structured manner. On the premise that another, inner, panic wouldn't get handled of course. Yes, the runtime may be in a bad state, but given the upside, it was worth it for our use case (mobile deployment).
For instance, this allowed us to call into Sentry directly and let it do its thing. Plus the diff being very small, we felt it was a big upside for a small investment.
We didn't try to upstream it though as we figured this wouldn't get accepted anyhow. @rsc arguments are sound, and kind of confirm what we thought. However, it did plug a very big hole on our end regarding mobile crash reporting.

@zx2c4 SetPanicNotifier is an example of a callback that, as @rsc says, has the problem that the runtime may not be in a good enough state to run the code in the hook.

@steeve @zx2c4 I believe that https://github.com/znly/go/commit/6af49debb0b604c1bfe209d5801f985afb327c72#diff-bbc551fed605c9b53290f903906ab32b2e0e132102081dd1f63575798cb8333cL633 should work for all panics, but only panics. It is effectively a global defer, so I can't think of any panics it wouldn't work on.
However it would certainly not work for throws, which this issue would like to capture as well. If the runtime is in a corrupt state, I'm not sure it would even be able to do the interface conversion to pass an error to the crash function. The contents of onPanic allocating a buffer and calling complex functions like runtime.Stacks and fmt.Println are also examples of the kinds of things that the runtime may not be able to support while throwing.

However it would certainly not work for throws, which this issue would like to capture as well.

I suppose this is a concern for me... My println hook + ringbuffer logger means that I can log from any context anywhere. Any where that println can go, my logger can go.
I think it's going to be difficult to get this same level of reliability in a generic context. At least, given the problems identified thus far, I'm not sure I see too many obvious solutions...

It is effectively a global defer, so I can't think of any panics it wouldn't work on.

Exactly. Indeed, it is limited, but to us there was a whole class of crashes we could now properly inspect.
Although I do think a better way to have a generic way to go about it would be to leverage a native crash reported, such as breakpad or crashpad, but there are some blocking issues for that on ARM64.

At CockroachDB we're happy with this patch:
https://github.com/cockroachdb/go/commit/8d39b0baae0b402e812ed6f6100f690bbe466bf1

At CockroachDB we're happy with this patch:
cockroachdb@8d39b0b

That doesn't really address the issue we're facing here, and is already doable using either of the techniques in the first list of bullets of https://github.com/golang/go/issues/42888#issuecomment-776981595

What is the API that x/sys/windows would need for that?

If you look at globalWrite in
https://go-review.googlesource.com/c/sys/+/283602
you will see that it uses ReportEvent Windows API. The API would require Windows handle. That handle should come from golang.org/x/sys/windows/svc/eventlog.Log.Handle.
The reason why I think this code should live in golang.org/x/sys/windows/svc/eventlog, is because there is more related API that allows users to install and remove event log source. You have to do event log source install dance before you can open the even log. If you try to call golang.org/x/sys/windows/svc/eventlog.Open with random source string without installing it first, it will fail.
You can also see what panic should do just be looking at something like golang.org/x/sys/windows/svc/eventlog.Log.Error.
I hope it helps.
Alex

runtime.SetPanicNotifier(func) is not going to happen (see discussion of signal-safe code above).
It might be OK for x/sys/windows to have its own private hook into the runtime, provided it did not itself export a way to run user-specified code in that callback.
So then the question is: what is the non-callback API that x/sys/windows would provide?

Is https://go-review.googlesource.com/c/go/+/278792/ not what you want?  I'm not quite sure I understand.

I agree with @zx2c4 we can use some go:linkname trick similar to https://go-review.googlesource.com/c/sys/+/283602
Whatever interface we find appropriate. We can document it as private and that it will change without notice, if you are worried about third parties relying on it in the future.
Alex

I think what I was asking about is this from CL 283602:
// InstallAsBuiltinPrint causes prints to Go's builtin print and println, as we as
// output during crashes and panics, to be delivered to this event log using the
// specified id.
func (l *Log) InstallAsBuiltinPrint(eid uint32) {

Is this the API that we believe is the general API callers need?
I don't think we want to divert prints entirely, but we could divert them into the log in addition to printing to standard error.

Is this the API that we believe is the general API callers need?
I don't think we want to divert prints entirely, but we could divert them into the log in addition to printing to standard error.

I would not make this API affect all print and println output. I only use print and println for debugging and I tried using @zx2c4 SetStdHandle trick (https://github.com/golang/go/issues/42888#issuecomment-737616766) and that works very well already.
So I would just add new API to redirect crash dump output. The API might look very similar to CL 283602:
// SetAsCrashOutput makes Go runtime output during crashes and panics
// to be delivered to the event log l using the specified event id eid.
func (l *Log) SetAsCrashOutput(eid uint32) {

Under the covers this function will store Log.Handle somewhere in the runtime. And runtime would have to call ReportEvent Windows API with that handle.
Perhaps we should not save Log.Handle, but copy the handle with DuplicateHandle, and use copy handle in the runtime. This way user program can safely call Log.Close without affecting runtime. And runtime would have to close saved handle when done outputing.
I am not sure if we should allow for Log.SetAsCrashOutput to be called more than once - user program might change its mind about where log should go. Perhaps it is OK to do - we would have to close previously saved handle and store new handle.
I also not sure if SetAsCrashOutput is a good name for this function. So happy to hear better suggestions.
Alex

It sounds like maybe we've converged to what happens on Windows.
I agree with Alex we don't mean to redirect user prints (Go println statements in programs).
We do want to redirect crash output.
Does it matter what we do for other runtime-generated prints, like GODEBUG=gctrace=1 output?
It would be easier if all the runtime output went together, and it doesn't seem too bad for those to hit the system log too.
Thoughts?

Does it matter what we do for other runtime-generated prints, like GODEBUG=gctrace=1 output?

I think redirecting runtime-generated prints would allow external tools that capture that output and visualize/analyze it.
I'm not sure there's a good use for keeping them in stderr... however I could imagine people needing to redirect them elsewhere, however, I'm not seen an immediate need. Such separation could be introduced later, if needed.

Does it matter what we do for other runtime-generated prints, like GODEBUG=gctrace=1 output?

If you are asking about Windows event log, I would not put GODEBUG=gctrace=1 output into Windows event log.
Windows event log is stored and viewed as bunch of text messages. Each message can contain line delimiters and is written by calls to golang.org/x/sys/windows/svc/eventlog.Log.Error, ....Info or ....Warning.
Writing whole crash dump with one single call to golang.org/x/sys/windows/svc/eventlog.Log.Error makes sense. I don't see how you would split GODEBUG=gctrace=1 output. If we split line by line, there would be too many log messages. It would be difficult to view that log. GODEBUG=gctrace=1 output is good for files, not for Windows event log.
Otherwise I don't care were GODEBUG=gctrace=1 output goes. I don't use it myself.
Alex

OK, so it sounds like this is where we are.
We separate out crash output from user-generated and other runtime tracing prints (GODEBUG=gctrace=1).
Crash output is eligible to be copied into a separate system log, but it continues to be printed to standard error as well (otherwise a crashing program would exit silently).
On all systems, debug.SetCrashFD(fd uintptr) sets the current redirection file descriptor / handle.
Additionally, x/sys/windows has a linkname backdoor to implement
// SetCrashEvent makes Go runtime output during crashes and panics
// to be delivered to the event log l using the specified event id eid.
func (l *Log) SetCrashEvent(eid uint32) {

We discussed having GODEBUG=crashfd=3, but on reflection that seems like a dangerous thing to have child processes inherit (especially if the parent doesn't know to preserve fd 3 when starting the child, fd 3 will be some unrelated file at crash time). So let's not do that part.
Any objections to this plan?

Does it matter what we do for other runtime-generated prints, like GODEBUG=gctrace=1 output?
It would be easier if all the runtime output went together, and it doesn't seem too bad for those to hit the system log too.

I absolutely care about all output, not just crashes. So conflating those is totally okay with me and in fact exactly what I want.

Additionally, x/sys/windows has a linkname backdoor to implement

By this, you mean: https://go-review.googlesource.com/c/go/+/278792/ , right? If so, that also sounds good to me.

I wrote:

We separate out crash output from user-generated and other runtime tracing prints (GODEBUG=gctrace=1).
Crash output is eligible to be copied into a separate system log, but it continues to be printed to standard error as well (otherwise a crashing program would exit silently).

@zx2c4 you wrote:

I absolutely care about all output, not just crashes. So conflating those is totally okay with me and in fact exactly what I want.

Those are different. Are you saying you care about user-generated output (println statements)? Or only about non-crash runtime-generated output (GC traces)? I don't think we can reasonably special-case println to the log when fmt.Println doesn't go there. But we could potentially send other runtime-generated output if there was a use case. What's the use case?

OK, so it sounds like the plan is:

add debug.SetCrashFD(fd uintptr). That fd gets crash output only, not user-generated prints, nor runtime-generated tracing prints (like GODEBUG=gctrace=1). Stderr still gets the crash output too.
Add a linkname backdoor for x/sys/windows (*Log).SetCrashEvent(eid uint32).
Do not do GODEBUG=crashfd=3 because of concerns about subprocesses.

Last call for objections.

Instead of debug.SetCrashFD(fd uintptr) it would be debug.SetCrashFile(f *os.File), because of the concerns raised by @alexbrainman in https://github.com/golang/go/issues/42888#issuecomment-749352457.

Like @zx2c4, I'd also be interested in getting the the runtime's crash println statement, as they often provide critical context about a throw, which can't do formatting.
In my view, there are three different types of println calls:

prints immediately prior to a throw. In my view, these are effectively part of the throw, providing context about the crash, which may be critical to understanding the crash. To be honest, I'm not sure why one wouldn't want this, beyond technical complexity.
Runtime prints unrelated to crashing (e.g., gctrace, schedtrace). I personally don't have a preference either way here.
User-generated prints. These probably shouldn't be included, but I don't think I've ever seen one in practice, so if they are sufficiently rare, I don't think it would be awful to include them.

Instead of debug.SetCrashFD(fd uintptr) it would be debug.SetCrashFile(f *os.File), because of the concerns raised by @alexbrainman in #42888 (comment).

It's already fd uintptr specifically to allow a Windows handle.
I don't understand why *os.File would be problematic.
It seems unlikely you would need worry about 0, 1, 2 anyway.
@prattmic, re "prints immediately prior to a throw." I think it would be fine for the runtime to set a flag before those prints that includes them in the throw. That can count as "crash output".
It sounds like generally people are happy with this API.

On further reflection, one important benefit of *os.File would be that it would let the runtime/debug package hold on to the underlying file object and keep it from being garbage collected, which in turn would keep the fd from being closed. Changing to *os.File.

@prattmic, re "prints immediately prior to a throw." I think it would be fine for the runtime to set a flag before those prints that includes them in the throw. That can count as "crash output".

Yup, I totally agree.

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

I'm still catching up on Go things. We've had a lot of discussion here so I was wondering what parts of this are in the "likely accept". Does that include something like https://go-review.googlesource.com/c/go/+/278792/ for the purpose of the eventlog x/sys/windows API?

@zx2c4 I believe that what is accepted is:

add debug.SetCrashFD(file *os.File). That file gets crash output only, not user-generated prints, nor runtime-generated tracing prints (like GODEBUG=gctrace=1). Stderr still gets the crash output too.
Add a linkname backdoor for x/sys/windows (*Log).SetCrashEvent(eid uint32).

Yes, see https://github.com/golang/go/issues/42888#issuecomment-824235777.

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

Okay and the "Add a linkname backdoor for x/sys/windows (*Log).SetCrashEvent(eid uint32)." part would get all println statements, right, while SetCrashFD wouldn't. Correct?

I've updated/rebased https://go-review.googlesource.com/c/go/+/278792 which should be inline with this proposal.

The "Add a linkname backdoor for x/sys/windows (*Log).SetCrashEvent(eid uint32)." part would get all println statements, right, while SetCrashFD wouldn't. Correct?

That is not how I read https://github.com/golang/go/issues/42888#issuecomment-824235777. See add debug.SetCrashFD(fd uintptr). That fd gets crash output only, not user-generated prints, nor runtime-generated tracing prints (like GODEBUG=gctrace=1). ... bit.
Alex

Based on https://github.com/golang/go/issues/42888#issuecomment-824247418 I would assume this is including all println statements. Remember: the whole point of the linkname escape hatch is for when fd-based output isn't possible.

Based on #42888 (comment) I would assume this is including all println statements.

I still disagree with you. But I will let others to confirm.
Alex

@rsc - Could we get https://go-review.googlesource.com/c/go/+/278792 in for 1.17? It'd be a nice first step.

@rsc - following up on this... could you change your -2 to a +2 on https://go-review.googlesource.com/c/go/+/278792 ?

@rsc - following up on this... could you change your -2 to a +2 on https://go-review.googlesource.com/c/go/+/278792 ?

Thanks for the ping @zx2c4, Russ did turn it into a +2 and that change got merged on Nov 4th 2021.

Apparently, there is no plan to export some exported APIs to modify the overrideWrite, but is there other plan for this issue?

What's the status on this? Proposal has been accepted but no patch yet, correct? Is anybody working on that? We have a Windows service executable which once in a while crashes, we suspect concurrent map writes, but without the traces we are pretty much lost how and where this happens. So a way to redirect that to a file would be greatly appreciated!

I intended to submit a patch, but never got around to it due to various reasons. I still had it in my github inbox though. But, if you (or anyone else) wants to start working on it before I sort my things out then that's certainly welcome.

Through telemetry, the gopls project has recently been getting a lot of really helpful crash report information from places where we added explicit calls to bug.Reportf to assert conditions we weren't confident enough to panic on. I suspect we'd get a lot more crash reports yet, if only we could detect unhandled panics and fatals.
The idea of a runtime hook for the "recover of last resort" (like the JVM's Thread.UncaughtExceptionHandler fills me with dread, but this proposal would enable us to develop a watchdog as a child process. The parent would call SetCrashOutput on its end of the pipe feeding the child's stdin, and the child would wait for the death of the parent then parse its input and report the crash to telemetry. Having only a textual traceback, it might be slightly impoverished in what it can report--no access to the pclntab means no function-relative line numbers---though if the parent and child are the same executable, there may be more we can do.

@adonovan I had been watching for this API to become available, though having used it I'm a bit disappointed.
Perhaps I didn't understand the use case: I had wanted my process to be able to catch crash output should one occur.
So I can keep an empty "crash.log" file around, that only gets content when a crash occurs. However, there's no date time data output. How would I  cross reference when the crash occurred?
Also, the way it's implemented I cannot remove said crash file if nothing gets written. (I can on mac/Linux but not on Windows because it thinks the file handle is in use)
See my simple example below
var crashLog string
func main() {
	crashLog, _ = filepath.Abs("crash.log")
	f,err := os.OpenFile(crashLog, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, os.ModePerm)
	if err != nil {
		fmt.Println("Error creating file:", err)
		return
	}
	fmt.Println("Created ", crashLog)
	debug.SetCrashOutput(f, debug.CrashOptions{})
	f.Close()

	//simulate crash
	//panic("ERROR")

	//We will only get here if there's no crash...
	cleanupCrashlog()
}
func cleanupCrashlog() {
	stat, err := os.Stat(crashLog)
	if err == nil {
		if stat.Size() == 0 {
			err := os.Remove("crash.log") //fails on Windows, because of win file handling
			if err != nil {
				fmt.Fprint(os.Stderr, "Failed to remove crash log: %v", err)
			} else {
				fmt.Println("Deleted crash log")
			}
		}
	}
}

Can I suggest we either

write some datetime indicators in the crashoutput AND/OR
make a CrashOptions field that users can specify "removeEmptyFile".
Otherwise Im forced to create a crashlog file for every session (where 99% will be empty)
or use a single crashlog where all crashes go but without any date time indicators.

Or am I just missing something? I have no need for a separate monitor process - I think a process should be able to catch its own crash output.

@adonovan

The file modification time records the time of the last write to the file, so you should be able to infer the crash time from it.
For the last crash only
See my scenario is an application that runs periodically. We want to be able to go back and see if there were any crashes at a later time. Ideally we could look in an application specific directory and just count the number of files to see how many crashes occurred.

you will need to create a file for every process, and (one hopes) the vast majority will be empty.
I really do feel though this feature missed the mark. At least if SetCrashOutput() received an interface instead of os.File I feel like there'd be more opportunity to customize the behaviour.

Also, your cleanupCrashlog function is allowed to call SetCrashOutput(nil), after which it should be possible to remove the log file even on Windows.
There it is! I can make that work - thank you!

Thank you for the responses, I appreciate your time on this.

Empty directory == all good.

Sorry, I should have been clearer. I was referring to a manual, visible inspection of said directory. Which I think you'll agree is still easier to do on a single file (that's required for a running process) vs 1000s of leftover files

There's no way that the runtime can make a dynamic call through an interface while in the middle of a crashing.

I still think this makes a case for a future CrashOptions{} field to remove the crash file if it is empty.
I feel like it will be a common requirement where devs want to indicate "put crash info in log file X that exists iff a crash occurs".
Anyway, just my two cents, I differ to your expertise.
Thanks again.

I was referring to a manual, visible inspection of said directory. Which I think you'll agree is still easier to do on a single file (that's required for a running process) vs 1000s of leftover files

Agreed, but that sounds like a tedious manual job best automated and delegated to a computer. ;-) And there's no getting around the need for each running process to have an empty file.
If the process acted as its own crash monitor (using a pipe---see ExampleSetCrashOutput), it could defer the opening of a file until the monitored program actually crashed.

There's no way that the runtime can make a dynamic call through an interface while in the middle of a crashing.

I still think this makes a case for a future CrashOptions{} field to remove the crash file if it is empty. I feel like it will be a common requirement where devs want to indicate "put crash info in log file X that exists iff a crash occurs". Anyway, just my two cents, I differ to your expertise. Thanks again.

There's no way for the runtime to implement that behavior. It would require the runtime to somehow know that the process was never going to crash. The only code that knows that is your main function just before it calls os.Exit(0), assuming there are no other goroutines running. Even then, many long lived programs that you might want to monitor never voluntarily terminate: healthy web servers are often killed by the termination of their container.


[Issue]: Why mypy check is skipped in merge checks?

Describe the issue
For instance, in PR #2466, mypy check should not be skipped.
https://github.com/microsoft/autogen/actions/runs/8772891100/job/24072334718

Steps to reproduce
No response
Screenshots and logs
No response
Additional Information
No response

+1 on the same question.
Now the mypy check keeps failing on my local machine with the error:

These codes were introduced by the PR #2466

@whiskyboy, I'm encountering the same issue when trying to commit. I ran pre-commit clean but I'm still having the issue. Is there a specific version of openai that I need?
I'm using a VSCode Dev Container, if that helps.

@marklysze is this still happening? A fix went in recently

@marklysze is this still happening? A fix went in recently

Thanks @WaelKarkoub, it seems corrected... fetched, merged, and pre-commit clean - looks okay. Thanks!

@thinkall is it safe to close this issue? The fix was to ignore openai_utils in mypy

its still happening for me.

@jtoy is the error still in openai_utils?


Removed extra comma in ts section

What is the purpose of this pull request? (put an "X" next to item)
[x] Documentation update
[ ] Bug fix
[ ] New feature
[ ] Other, please explain:
What changes did you make? (Give an overview)
Removed extra comma
Which issue (if any) does this pull request address?
None
Is there anything you'd like reviewers to focus on?
No

Thanks for the PR! I already merged https://github.com/standard/standard/pull/1594 even though you sent your PR first (apologies!)


Unable to run report due to Microsoft Admin Portals Application

After configuring the Application Registration and connection the next step is unable to complete in an environment that utilises Microsoft Admin Portals as there isn't a ClientID that the command can interpret to a GUID.
ForEach-Object : Cannot convert argument "item", with value:
"MicrosoftAdminPortals", for "Add" to type "System.Guid": "Cannot convert
value "MicrosoftAdminPortals" to type "System.Guid". Error: "Guid should
contain 32 digits with 4 dashes (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx).""
At E:\Documents\WindowsPowerShell\Modules\AzureADAssessment\2.4.0\AzureADAssess
ment.psm1:143 char:132

... ice365' } | ForEach-Object { [void]$ReferencedIdCache.appId.Add($_) } ...

            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo          : NotSpecified: (:) [ForEach-Object], MethodExcept
ion
FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument,Micr
osoft.PowerShell.Commands.ForEachObjectCommand

one my customer trying this tool, reported same exact issue


Per tenant API keys

Data model for storing API keys for OpenAI and Azure OpenAI

Allow marking as default

UI for managing API keys
Link API key to Experiments

If there is only one API key configured auto-link and don't show the UI element

They should be encrypted?


Feat/tar async

I certify that I own, and have sufficient rights to contribute, all source code and related material intended to be compiled or integrated with the source code for the SharpZipLib open source product (the "Contribution"). My Contribution is licensed under the MIT License.
Relates to #223
I have taken a first pass at making TAR support async streams which will help with streaming to and from network streams. I have also tried to dramatically reduce memory allocations associated with reading from and writing to TAR's.
I haven't added new tests yet or really checked what the coverage is like but if you are happy with how it looks as it, I can do that.

This should make the async versions of Zip[In/Out]putStream much clearer.
Anyway, this mostly looks great, but it does lower the test coverage a bit.

Thanks for the feedback! I will create some more tests to cover the changes

Some extra tests have now been added.


Python: Can't debug/break on breakpoints when using virtual environment / pythonpath

This issue has been moved from Microsoft/vscode/
Original author is

@TitaneBoy

VSCode Version: 1.21.1
OS Version: Windows 7
Python version: 3.6.3

Hi. I'm using virtualenv to create virtual environments for my python projects.
To create a virtual environment with virtualenv, we must go on our current project directory (with terminal/console) and execute the command: virtualenv .  (Of course, virtualenv must be already installed)
This command will create some folders for the virtual env and in the folder "Scripts", we can see the python.exe executable with others files

So in order to use the python of my virtual environment,  I use this in my "settings.json" config:
{   "python.pythonPath": "${workspaceFolder}\\Scripts\\python.exe" }
To debug my project, in my launch.json, I have this:
{
      "name": "Launch Program",
      "type": "python",
      "request": "launch",
      "stopOnEntry": false,
      "pythonPath": "${config:python.pythonPath}",
      "program": "${workspaceFolder}/main.py",
      "cwd": "${workspaceFolder}",
      "env": {},
      "envFile": "${workspaceFolder}/.env",
      "debugOptions": [
        "RedirectOutput"
      ]
    }

But when I am launching the debug, it nevers breaks on any breakpoints, even if I set "True" for the "stopOnEntry" option in launch.json.
Is there something I'm missing or there is here a true issue in vscode ?
Also, I know that in order to "activate" virtual environment with virtualenv, we must launch the activate(.bat) script located in the "Scripts" created folder.
Of course, If I remove the settings.json file, it will breaks on my breakpoints, but it will not use the python from the virtual environment.
Thanks for your help

This is a reply from

@vscodebot

(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

${workspaceFolder} variable not replaced when debugging Python (#43303)

Have you added a breakpoint in the main.py file?
What are the contents of your main.py file?

@DonJayamanne
Yes, I've of course added a breakpoint in the main.py file.

Ok, Please try moving your main.py out of your packages3 directory.
The problem is you are trying to setup your workspace as the python interpreter directory.
Please try that and let me know how that goes:
I.e. your virtual environment should be a sub directory of your workspace. Not sure why you would ever want to setup it up the way you have.

@DonJayamanne Thanks for the suggestion. It looks to work now.. :-)  Didn't know that we must put virtual environment folders in a sub-folder.
Just one more question: How vscode manages or detect the site-package of the virtual environment ? As I said before, on a console, I had to execute the "activate.bat" script in order to be in my virtual environment. Then I could execute my main script with python (python main.py). But with vscode, I'm not sure to understand how to proceed with it. Do I need somehow to execute this script (activate.bat) before the debug session ?
Thank you for your help.

How vscode manages or detect the site-package of the virtual environment ?

We don't need to do this, python does this. The extension doesn't look for any site-packages.

As I said before, on a console, I had to execute the "activate.bat" script in order to be in my virtual

You can use the command Python: Create Terminal (that'll automate the task for you).

Do I need somehow to execute this script (activate.bat) before the debug session ?

No.


Add test for export of QuantReLu and add new test cases for QuantHardTanh export

New test cases for QuantHardTanh:

max_val > 1
scaling_impl_type == ScalingImplType.PARAMETER with learned scale.

Update:

feature/graph_order_util_fct merged
vector implementation of multithreshold function

Hi @Tobi-Alonso -- can you please make a branch, isolate just the related commits for adding the new tests and make a PR for that branch instead? Right now this is trying to merge Lucian's entire dev branch with all its changes  into finn dev.


reference the relevant section of the spec in definitions

This can help implementors who are tracing through the spec to add support for parsing various objects.

@karenetheridge   Thanks for adding these comments.  These are very helpful.


Add Bengali (bn-BD) language

Added Bengali (বাংলা) language data in context of Bangladesh 🇧🇩

Thank you for your contribution @thearyanahmed!


Fix issue #44

Coverage remained the same at 52.944% when pulling 997b1154d393998be15609383ab7e005d629c0eb on issue_44 into ab83a8b75745510d584a892fd3064ce4d747459f on master.


spinel: Added feature for true random number generation

Sometimes the host processor doesn't have a good way to generate high-quality random numbers suitable for cryptographic use. However, the NCP often has such capability. This Spinel feature allows the host processor to fetch high-quality random numbers from a TRNG that may be present on the NCP.

Current coverage is 72.54% (diff: 100%)

Merging #1098 into master will decrease coverage by 0.97%

@@             master      #1098   diff @@
==========================================
  Files           124        128      +4
  Lines         16809      18415   +1606
  Methods        2364       2569    +205
  Messages          0          0
  Branches       2029       2234    +205
==========================================
+ Hits          12357      13359   +1002
- Misses         3842       4365    +523
- Partials        610        691     +81

Powered by Codecov. Last update 12854d6...fd8baca


ScriptDomain::ExecuteTask throws errors in some cases

When in a different thread (websocket server), it will throw "Object reference not set to an instance of an object." when run. (but the task still runs)

ScriptDomain::ExecuteTask only works with managed threads, not with native Win32 threads (a managed threads are handled by the CLR). But either way, you should not create your own threads and execute scripting code inside them, managed or not. It is not supported.

you should not create your own threads and execute scripting code inside them, managed or not.

I wasn't, a library I was using was...
But a workaround I guess is to store stuff you want to call inside of a variable and on tick if those are set then call it from there since it would be on the main (managed) thread. It's really hacky, but it works...


Nothing happens after showing "messages"

Cannot type in the cmd after this either

this is supposed to happen...

This is where I am stuck as well. Anybody have a solution?


(maint) MergeBack Release version v4.2.0

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@70eb9b0). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #401   +/-   ##
=========================================
  Coverage          ?   16.66%
=========================================
  Files             ?        1
  Lines             ?        6
  Branches          ?        0
=========================================
  Hits              ?        1
  Misses            ?        5
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70eb9b0...13626d6. Read the comment docs.


Support bool values in a Map stored as a real bool (not int).

This compliments a pr #178 when the Map is retrieved from source other than SQLite (API response, JSON file, etc.)

No idea what's failing, Codedev is so obscure at hiding details.

@vitusortner
I got another pr waiting that unifies repetitive code and adds an exporter function (like the query mapper, but in the opposite direction: to map an entity to a Map). Could you find few moments to go over this pr?

Complimentary Entity-to-Map exporter has landed: #204. Code adjusted to use it (Querying, Inserting, Deleting).

Anyone wanting to have this feature can use my fork which I'll maintain to always sync with upstream:develop: https://github.com/noordawod/floor/tree/develop-noordawod

This one too was supposed to cut on duplication of field names, I don't understand why it wasn't merged.


Screen recording permission requested even when window animation disabled

Running yabai with the example version of yabairc prompts for "Screen & System Audio Recording" permission.  When that's denied it prints the warning:

command 'window_animation_duration' for domain 'config' requires Screen Recording permissions! ignoring request..

The README.md says:

yabai must be given Screen Recording permission if and only if you want to enable window animations, and will request access when necessary

And the yabairc (linked above) includes:
yabai -m config                                 \
...
    window_animation_duration    0.0            \
    window_animation_easing      ease_out_circ  \

The man page says:

window_animation_duration [<FLOAT_SEL>]
Duration of window frame animation.
If 0.0, the change in dimension is not animated.

So I would have thought that window animation was disabled (by being set to 0.0) and that that would case the permission to not be requested?

I'm guessing that at this point:
        } else if (token_equals(command, COMMAND_CONFIG_ANIMATION_DURATION)) {
            struct token_value value = token_to_value(get_token(&message));
            if (value.type == TOKEN_TYPE_INVALID) {
                fprintf(rsp, "%f\n", g_window_manager.window_animation_duration);
            } else if (value.type == TOKEN_TYPE_FLOAT) {
                if (CGPreflightScreenCaptureAccess()) {

There should be an additional if (value.float_value) around the if (CGPreflightScreenCaptureAccess()) { ... } else { ... } so permissions are checked if and only if value.float_value != 0.0?

That seems to do the trick, opened #2379

Fixed in https://github.com/koekeishiya/yabai/commit/4572d69932a11e157985c7c515d1e094b88707c6 https://github.com/koekeishiya/yabai/commit/14f1220c6a8ec10b0480e5b5877b59585fba8e0c


Poster image not displayed / jumps in.

Bug Description
After video preview completes update, the poster image disappears and is replaced after a couple of seconds with a real poster image from WP.
Expected Behaviour
This should not be soo jumpy and the poster image replacement should happen seamlessly.
Screenshots
https://youtu.be/nEWXcRzzdbo

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria
QA Instructions

CC @obetomuniz

Verified in QA


Added config to skip to sign in screen

Description
MA-2906
Skips to sign in screen when the config flag "SIGN_IN_ONLY_ENABLED" is true.
If you've been tagged for review, please check your corresponding box once you've given the :+1:.

[ ] Code review: @bguertin @miankhalid @1zaman @mdinino

Current coverage is 19.09% (diff: 16.66%)

Merging #845 into master will increase coverage by <.01%

@@             master       #845   diff @@
==========================================
  Files           465        465
  Lines         19756      19767    +11
  Methods           0          0
  Messages          0          0
  Branches       2868       2872     +4
==========================================
+ Hits           3771       3774     +3
- Misses        15542      15549     +7
- Partials        443        444     +1

Powered by Codecov. Last update ef5bbf4...f69252b


Bump Dependencies #2233920491

Bumps dependencies for product-is. Link : https://github.com/wso2/product-is/actions/runs/2233920491

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.jenkins-is-staging seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


No search results displayed

Hello,
When I press the F5 key, the program launches my browser and then I enter a search query like "sharepoint" and I get the exception as attached. I have a couple of documents on my development side and I'm entering a query based on those documents

Please help.

@BamideleAriwodola
Thanks for bringing this up. However, the Issues feature is intended for bugs in this particular sample. For technical help questions, please ask a question on StackOverflow. Be sure and tag it "sharepoint".