syscall: define Windows O_ALLOW_DELETE for use in os.OpenFile

The discussion on #32088 has so far been unable to reach consensus on setting FILE_SHARE_DELETE unconditionally in os.OpenFile. Right now it looks likely to be declined, possibly revisited in a few years.
In the interim, it is more difficult on Windows than on Unix to pass an extra flag to open a file with package syscall. syscall.Open takes fake O_* flags, not actual Windows permission bits. It wraps syscall.CreateFile, but it does a lot of argument preparation before making that call:
// package syscall

func Open(path string, mode int, perm uint32) (fd Handle, err error) {
	if len(path) == 0 {
		return InvalidHandle, ERROR_FILE_NOT_FOUND
	}
	pathp, err := UTF16PtrFromString(path)
	if err != nil {
		return InvalidHandle, err
	}
	var access uint32
	switch mode & (O_RDONLY | O_WRONLY | O_RDWR) {
	case O_RDONLY:
		access = GENERIC_READ
	case O_WRONLY:
		access = GENERIC_WRITE
	case O_RDWR:
		access = GENERIC_READ | GENERIC_WRITE
	}
	if mode&O_CREAT != 0 {
		access |= GENERIC_WRITE
	}
	if mode&O_APPEND != 0 {
		access &^= GENERIC_WRITE
		access |= FILE_APPEND_DATA
	}
	sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE)
	var sa *SecurityAttributes
	if mode&O_CLOEXEC == 0 {
		sa = makeInheritSa()
	}
	var createmode uint32
	switch {
	case mode&(O_CREAT|O_EXCL) == (O_CREAT | O_EXCL):
		createmode = CREATE_NEW
	case mode&(O_CREAT|O_TRUNC) == (O_CREAT | O_TRUNC):
		createmode = CREATE_ALWAYS
	case mode&O_CREAT == O_CREAT:
		createmode = OPEN_ALWAYS
	case mode&O_TRUNC == O_TRUNC:
		createmode = TRUNCATE_EXISTING
	default:
		createmode = OPEN_EXISTING
	}
	h, e := CreateFile(pathp, access, sharemode, sa, createmode, FILE_ATTRIBUTE_NORMAL, 0)
	return h, e
}

A direct call to syscall.CreateFile would have to duplicate all that code.
A direct call to syscall.Open would still lose out on the adjustments made inside os.Open, most importantly the call to fixLongPath:
// package os

func openFile(name string, flag int, perm FileMode) (file *File, err error) {
	r, e := syscall.Open(fixLongPath(name), flag|syscall.O_CLOEXEC, syscallMode(perm))
	if e != nil {
		return nil, e
	}
	return newFile(r, name, "file"), nil
}

If package syscall on Windows defined a bit O_ALLOW_DELETE (maybe 0x100000), then syscall.Open could convert that bit into FILE_SHARE_DELETE. Then it would be easy for calls to either syscall.Open or os.OpenFile to cause the underlying call to use FILE_SHARE_DELETE if they really needed it.
This proposal assumes #32088 is declined. It should be considered withdrawn if #32088 is accepted.

I would call this O_ALLOW_RENAME since that's the most common use of the flag. I'd also define it in package "os" so users don't need to write an open wrapper (with multiple implementations) and remember to always invoke that instead of os.OpenFile().
One may also wish to switch off FILE_SHARE_READ & _WRITE to prevent another caller from inadvertently blocking rename/delete by opening the file.

Whatever the name, it's worth asking whether the new name should be defined on non-Windows systems, presumably with the value 0.  The advantage of doing that would be that people who want to run on all systems could use the flag without relying on build tags.  The disadvantage would be a flag that is odd and ineffective on non-Windows systems; on the other hand, that is already true of O_SYNC, or O_APPEND|O_WRONLY, on Windows systems.

Agreed with @ianlancetaylor, that would make maintenance easier without having to breaking up code to platform-specific implementations

I agree with iant. We should consider to portability of the code but the meenless flag should not be added to UNIX. If we don't add the new flag on UNIX, the code for Windows must be compiled with build constraints. i.e. it is same thing that providing windows.Open() from golang.org/x/sys/windows.

@mattn slightly confused, you say you agree with @ianlancetaylor, but then continue that you don't want to add the new flag on UNIX(/non-Windows)?

@alexbrainman thoughts?

@thaJeztah I am simply considering for a reasonable and nondestructive way to make changes to the standard library. This proposal has a positive effect on Windows users, but it shows meaningless flag(s) to UNIX users.

I think putting this functionality in a separate package works just fine (see https://github.com/golang/go/issues/32088#issuecomment-538703906 for details). Why does it have to be in main Go repo?
If some insist, we could move that package somewhere under golang.ord/x/sys or x/exp.
Alex

Alex, could you restate here, since the other issue is so long, why it is that you don't support this proposal?

@alexbrainman
Because code duplicates the one which is in standard library. I don't think that another hack floating around (with forks and duplicates) is a really good idea. If that code is changed in the future - all forks would have to update? which is never going to happen.
The simple answer is - code maintenance.

@alexbrainman could you restate here, since the other issue is so long, why it is that you don't support this proposal?

Because code duplicates the one which is in standard library.

Sure it does. Whole github.com/alexbrainman/goissue34681 package is 375 lines long including docs and tests. It took me around 1 hour to write - I copied os.Open and os.OpenFile, and then copied whatever was missing. It took me longer to write test. It just Go code, there is very little magic there.

I don't think that another hack floating around (with forks and duplicates) is a really good idea.

I am not proposing forks or duplicates. I propose you use github.com/alexbrainman/goissue34681 package. I am happy to move the package, if people have better suggestions.

If that code is changed in the future - all forks would have to update, which is never going to happen.

If correspondent os package code change in the future, and that change affects github.com/alexbrainman/goissue34681 , then sure someone will have to report bug and fix the code. We should have some tests in github.com/alexbrainman/goissue34681 and run them regularly. Just like another package. But, going by my experience, I don't think there will be many changes like that.

The simple answer is - code maintenance.

What is proposed in here is not maintenance free either.
This change is at the heart of os package. How would new flag interact with all other code? How will new flag work with directories? With symlinks? What about file symlinks vs directory symlinks? What about directory junctions? What about network shares?  Including novices and non-Windows users. Should we recommend that they use the flag or not? When? How would their decision would affect their users? Are they in a position to make that decision?
So far I have seen this flag used to open files so they can be moved or deleted. Therefore I created github.com/alexbrainman/goissue34681 package with just Open and OpenFile functions. These are simple replacement of os.Open and os.OpenFile. All your remaining code should not change. And the package should also work for non-Windows use - it just call os.Open and os.OpenFile for them. I think github.com/alexbrainman/goissue34681 package is easier to use then new os pack age flag.
I think users who need that functionality should try github.com/alexbrainman/goissue34681 package before we even discuss adding new flag to os package.

could you restate here, since the other issue is so long, why it is that you don't support this proposal?

See above paragraph.
Alex

then sure someone will have to report bug and fix the code.

Thats implying that someone is going to support this package forever. If one doesn't support it, then we face the situation with forks\duplicates.

This change is at the heart of os package. How would new flag interact with all other code? How will new flag work with directories? With symlinks? What about file symlinks vs directory symlinks? What about directory junctions? What about network shares?  Including novices and non-Windows users. Should we recommend that they use the flag or not? When? How would their decision would affect their users? Are they in a position to make that decision?

All of this is seems solvable by moving this flag to the syscall package.

I think github.com/alexbrainman/goissue34681 package is easier to use then new os pack age flag.
What about transitive dependencies which return *os.File?

If you are so against the idea of having as in standard, maybe we can move it into golang.org/x/sys/windows. That way we can at least ensure that people will maintain it in the future.

Thats implying that someone is going to support this package forever. If one doesn't support it, then we face the situation with forks\duplicates.

Yes. Someone who uses this package has to support it, if it breaks. It could be you, if it is important to you. But like I said before, it is very little code, and it should not break often. If there are enough users of that package it shouldn't be a problem. Should it?

All of this is seems solvable by moving this flag to the syscall package.

I don't see how this flag can live in syscall package. Can you provide more details?

If you are so against the idea of having as in standard, maybe we can move it into golang.org/x/sys/windows. That way we can at least ensure that people will maintain it in the future.

I am fine moving this package into somewhere under golang.org/x. I am not sure about adding 2 new functions (Open and OpenFile) to golang.org/x/sys/windows. golang.org/x/sys/windows has a lot of code, they will get lost there. And golang.org/x/sys/windows package is only available on Windows. github.com/alexbrainman/goissue34681 is suppose to be straight replacement of os.Open and os.OpenFile - it should work on any OS.
Alex

This is a slippery slope. On Windows, CreateFile and NtCreateFile have a lot of very interesting options and overrides. Probably the existing set of parameters are sensible for the majority of normal files people open. For weird files in special conditions, users will probably want the full control of CreateFile or NtCreateFile, not just the stray O_ALLOW_DELETE exception because a user happened to report that particular need to our bug tracker. In other words, I agree here with @alexbrainman - if we want to provide this functionality, let's do it outside of os.OpenFile and instead stick it in x/sys/windows or somewhere sensible like that, where we can expose all the various insane switches and nobs that Windows offers.
tl;dr: :-1: on proposal.

I think the problem was that currently, syscall.Open() handles flags it knows about (setting access, sa, and createmode based on those) then sets a hard-coded sharemode, and discards any other flag that was passed;  https://github.com/golang/go/blob/0e54d28ff7ff63450356811b675a02e0afe610d6/src/syscall/syscall_windows.go#L291
(thinking out loud) I wonder if it would be an option to have sharemode default to the current options, and append any flag that was not consumed by access, sa, or createmode. This would be similar to how the linux variant passes on whatever flags are passed; https://github.com/golang/go/blob/0e54d28ff7ff63450356811b675a02e0afe610d6/src/syscall/syscall_linux.go#L137-L139
(again, just thinking out loud)

I think the problem was that currently, syscall.Open() handles flags

syscall.Open returns a Handle, just like syscall.CreateFile.
If we're discussing syscall., what about just introducing x/sys/windows.OpenFile or syscall.OpenFile that wraps the relevant arguments of CreateFile, or perhaps better, NtCreateFile, and returns (*File, error). So something like:
func OpenFile(name string, access uint32, mode uint32, create uint32, attrs uint32, templateFile Handle, securityAttributes *SecurityAttributes) (*os.File, error)
Then people who want to do wild&crazy things with the Windows file opening routines can just use this and proceed as usual.

The motivation of this as well as the original proposal was to make it easier for Go software authors to write code that works cross-platform. We need to pass FILE_SHARE_DELETE to make Windows behave more like POSIX with respect to file deletion. There is general agreement that we cannot make this the default without regressing existing behavior, but the desire to have a clean, simple solution persists.
Any solution that requires extra dependencies, _windows.go files, build tags, etc. deviates from the goal of writing common code everywhere. I know from experience that it will be much easier to convince a random developer to add O_ALLOW_DELETE to their calls to os.OpenFile than to convince them to pull in an additional dependency with a custom fork of os.OpenFile, or to add Windows-specific .go files, just to get Windows to behave like everyone else.
I like the idea of putting a Windows-specific open routine in x/sys/windows to easily expose the wealth of CreateFile functionality! It would be great if we didn't have to replicate the complexity of calling syscall.CreateFile everywhere that needs Windows-specific functionality.
But I also think FILE_SHARE_DELETE is a special case worthy of inclusion in os.OpenFile, since it changes Windows behavior to be closer to all the other operating systems Go supports. I can't immediately think of any other CreateFile flags that do that.

Since #32088 has been declined, it seems like we should examine this carefully.
@alexbrainman seems to be saying it's not worth adding to package syscall when it can be done externally. But Alex, it's just a couple lines in syscall (that will be easy to maintain: O_ALLOW_DELETE literally means set FILE_SHARE_DELETE in the system call) versus having to maintain a 300+-line package elsewhere and also expect users to find it. Do you object strongly to adding just these few lines to package syscall?
@zx2c4, I understand your point about the many options, and maybe we should make it easier to get at CreateFile directly, but it does seem that this particular bit ("make Windows more like Unix file systems") is going to be wanted more often than most of the other settings.
Does anyone else object to adding these few lines?

but the desire to have a clean, simple solution persists.
Any solution that requires extra dependencies, _windows.go files, build tags, etc. deviates from the goal of writing common code everywhere. I know from experience that it will be much easier to convince a random developer to add O_ALLOW_DELETE to their calls to os.OpenFile than to convince them to pull in an additional dependency with a custom fork of os.OpenFile, or to add Windows-specific .go files, just to get Windows to behave like everyone else.

I think github.com/alexbrainman/goissue34681 package fits your requirements perfectly. It is oddly named, but we can give it better name. And we can move it into more "supported" place.

But I also think FILE_SHARE_DELETE is a special case worthy of inclusion in os.OpenFile, since it changes Windows behavior to be closer to all the other operating systems Go supports.

But the flag is completely meaningless for majority of Go users.
os.OpenFile already has 8 flags. And some flags can be used in combinations. I think it is too many as is. I struggle to remember which flags do what when I review or write code.
So adding 9-th flag of O_ALLOW_DELETE makes things even more confusing for everyone. If I am Linux developer, should I use O_ALLOW_DELETE in my code and when?
Go is supposed to be simple to use.

But Alex, it's just a couple lines in syscall (that will be easy to maintain: O_ALLOW_DELETE literally means set FILE_SHARE_DELETE in the system call) versus having to maintain a 300+-line package elsewhere

The duplicate code is in main_windows.go, and it is 214 lines long.
And since when the line count is more important than clean API and maintainability?
We already have plenty of duplicate code - just compare os and path/filepath packages. We write tests, and make sure we don't break them.
Adding new os.O_ALLOW_DELETE flag won't be easier to maintain then new package. New flag needs to be documented, educated, made sure it works in unusual scenarios (see https://github.com/golang/go/issues/34681#issuecomment-539413777 ). We would need to spend time fixing bugs and debugging flag usage. I am surprised I have to explain all these points to you.

and also expect users to find it.

I think this flag is only useful to narrow group of users. So, discover-ability is not a problem. I am certain, few people who needed this functionality, they already wrote their own code, just like I did myself.
Alex

Adding new os.O_ALLOW_DELETE flag won't be easier to maintain than new package. New flag needs to be documented, educated, made sure it works in unusual scenarios (see #34681 (comment) ). We would need to spend time fixing bugs and debugging flag usage. I am surprised I have to explain all these points to you.

I'm sorry, but those points are not "sound" at all. All of this points can be used against implementing any sort of new functionality - additions to errors/context/net packages, generics, etc. I also don't see a problem with maintenance - we are not going to get more "windows" developers and contributors, if we are going to treat them as second class citizens.
I also suspect there is a problem in communications, because ATM we are arguing about 3 different things:

Having new flag is os/syscall package.
Having new functionality in /x/sys side package.
Not having it at in Go repos.

I think @alexbrainman is right that a new flag is not a great solution.
I patch my syscall_windows.go as follows, and set syscall.Open_FileShareDelete = true once in apps that need it. This wouldn't land upstream, but if enough folks apply it, that would demonstrate over time that there's no downside to making file_share_delete the default for os.Open/etc().
diff --git a/src/syscall/syscall_windows.go b/src/syscall/syscall_windows.go
index de05840..e1455d5 100644
--- a/src/syscall/syscall_windows.go
+++ b/src/syscall/syscall_windows.go
@@ -245,6 +245,8 @@ func makeInheritSa() *SecurityAttributes {
 	return &sa
 }

+var Open_FileShareDelete = false
+
 func Open(path string, mode int, perm uint32) (fd Handle, err error) {
 	if len(path) == 0 {
 		return InvalidHandle, ERROR_FILE_NOT_FOUND
@@ -270,6 +272,9 @@ func Open(path string, mode int, perm uint32) (fd Handle, err error) {
 		access |= FILE_APPEND_DATA
 	}
 	sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE)
+	if Open_FileShareDelete {
+		sharemode |= FILE_SHARE_DELETE
+	}
 	var sa *SecurityAttributes
 	if mode&O_CLOEXEC == 0 {
 		sa = makeInheritSa()

FWIW, the .NET default appears to be only giving FILE_SHARE_READ, but not write. That generally fits my impression of how most Windows apps appear to work too.

Here's another curious insight from reading .NET code: they now support other operating systems, but of course all their constructors and whatnot take a FileShare share parameter because it was originally for Windows. So now they're trying to figure out what to do with that flag. Here's what they do:
        /// <summary>Initializes a stream for reading or writing a Unix file.</summary>
        /// <param name="mode">How the file should be opened.</param>
        /// <param name="share">What other access to the file should be allowed.  This is currently ignored.</param>
        private void Init(FileMode mode, FileShare share, string originalPath)
        {
            _fileHandle.IsAsync = _useAsyncIO;

            // Lock the file if requested via FileShare.  This is only advisory locking. FileShare.None implies an exclusive
            // lock on the file and all other modes use a shared lock.  While this is not as granular as Windows, not mandatory,
            // and not atomic with file opening, it's better than nothing.
            Interop.Sys.LockOperations lockOperation = (share == FileShare.None) ? Interop.Sys.LockOperations.LOCK_EX : Interop.Sys.LockOperations.LOCK_SH;
            if (Interop.Sys.FLock(_fileHandle, lockOperation | Interop.Sys.LockOperations.LOCK_NB) < 0)
            {
                // The only error we care about is EWOULDBLOCK, which indicates that the file is currently locked by someone
                // else and we would block trying to access it.  Other errors, such as ENOTSUP (locking isn't supported) or
                // EACCES (the file system doesn't allow us to lock), will only hamper FileStream's usage without providing value,
                // given again that this is only advisory / best-effort.

It looks like they try to make the Windows stuff sort of work by taking a shared lock when no locking is requested and an exclusive one when any locking is requested. Wild!

@alexbrainman You are asking every project that wants to use this option to copy several hundred lines of code that honestly I have no idea why the code is the way it is. All the special path handling and all... this even requires copying syscall.Open.
It's not a matter of "can" we do it. Yes we can. I did this exact thing to resolve our issue in moby/moby which you also copied in your repo.
"Should" we require all projects to make copies of this code?
The thing is, there's all the path handling is the os package, but then there's also actually modifying what's in syscall.
I'm all for copying the thing I need vs importing a package full of things I don't need but this is, in my opinion, too complicated to copy.

We should consider any cases carefully since people using syscall/os package are not only you. No easy to rollback code. No easy change the behavior. So we froze syscall package and we are working with golang.org/x repository. Adding Open_FileShareDelete is easy but it is hard to know what will break something before add.
Neither I nor you can confirm that all Go users want the FILE_SHARE_DELETE. If some package set Open_FileShareDelete = true, another package might be broken.
We want to know whether most of use-case can be done with x package.

Neither I nor you can confirm that all Go users want the FILE_SHARE_DELETE

I can confirm that not all users want it, and not all users don't want it. It an option that's added to the toolset, that users can use in their code (but don't have to, just like other options)

If some package set Open_FileShareDelete = true, another package might be broken.

I fail to see how this is a concern here; if a package creates a file, it's up to that package how that file is created. If that file is expected to be used/accessed by other packages, then it's up to the package maintainer to document it as a possible breaking change (and update the package version accordingly). I don't see that as anything different than, say, the package changing permissions on a file from 0440 to 0400

We aren't talking about a global here.

@mattn, you misunderstood. From https://github.com/golang/go/issues/34681#issuecomment-540729522:

I patch my syscall_windows.go as follows, and set syscall.Open_FileShareDelete = true once in apps that need it. This would not land upstream, but if enough folks apply it, that would demonstrate over time that ...

So no published package could use syscall.Open_FileShareDelete.

The demonstrate could be done with x package I think.

There's still some question as to whether this belongs in alexbrainman/, x/, or in std/. But aside from that question, here are some comments on the flag proposal.
If this is to be exposed via extra windows-specific flags, I think we ought to be specific about what they are in the names, and be exhaustive about it. So specifically:

O_WINDOWS_SHARE_NONE
O_WINDOWS_SHARE_READ
O_WINDOWS_SHARE_WRITE
O_WINDOWS_SHARE_DELETE

If any of these 4 are specified, then it overrides our current default of read|write.
It's important that these begin with O_WINDOWS_ so that people are clear that this is Windows-only. We could then mention in the documentation that this has no effect on other platforms but isn't an error either -- a no-op. I'm sure there's some very succinct phrasing that can let this behavior be known, and readers of the code will know what's up when they see O_WINDOWS_. They'll think, "Oh that's some Windows-specific flag that's only effective on Windows."
Separately, we could map existing flags to Windows flags we don't yet support, where a sensible analog exists. For example:

O_DIRECT could wind up setting the FILE_FLAG_NO_BUFFERING flag to CreateFile.

On the other hand, some Windows features don't have an analog elsewhere. For example, it would make sense to add in a direct manner the delete-on-close feature the Chromium people wanted, rather than trying to pretend it's the same as O_TMPFILE, which it's not. That would look like:

O_WINDOWS_DELETE_ON_CLOSE

There are at least six other stdlib invocations of CreateFile() which should also set FILE_SHARE_DELETE or be able to. They're documented in #35150.
A flag specific to os.OpenFile() is an incomplete solution; we can't modify six other APIs to take a flag. And duplicating the code for all those APIs isn't sensible.
Suppose we provide an environment variable to enable/disable f.s.d. use in all CreateFile() calls? I'd argue for a variable that disables it.

We're already talking about the Windows-specific package syscall, so there's no need to say O_WINDOWS_something. It can be O_something.
We can easily add O_SHARE_DELETE now and add the others (O_SHARE_NONE, O_SHARE_READ, O_SHARE_WRITE) if anyone actually needs them. Are there real uses for them, or is this just hypothetical completeness?
Thanks @networkimprov for identifying the other CreateFile calls in #35150. I commented there asking whether we should set FILE_SHARE_DELETE unconditionally in those.

I don't have a clear sense of where people are on this.
Should we add syscall.O_ALLOW_DELETE on Windows,
or should we do nothing and close this?

IMO we need to be able to os.OpenFile(..., syscall.O_ALLOW_DELETE, ...).
I honestly find the duplicated flags in os confusing, particularly for non cross-platform concepts, but do not care if it is added to os or not.

Ideally the constant would exist across all platforms, otherwise build tags will be required for source with POSIX-like functionality that builds on Windows (I realise the functionality isn't identical, but for some purposes it's enough).

I agree with @alexbrainman; from https://github.com/golang/go/issues/34681#issuecomment-540473295:

But the flag is completely meaningless for majority of Go users.
os.OpenFile already has 8 flags. And some flags can be used in combinations. I think it is too many as is. I struggle to remember which flags do what when I review or write code.
So adding 9-th flag of O_ALLOW_DELETE makes things even more confusing for everyone. If I am Linux developer, should I use O_ALLOW_DELETE in my code and when?
Go is supposed to be simple to use.

Patching the local syscall.Open() is a better solution than having to use OpenFile() with a windows-specific flag for any os.Open/Create() situation.
Eventually we'll prove that @rsc's suggestion in #32088 to set FILE_SHARE_DELETE by default in os.Open/Create/OpenFile() isn't harmful. After all, Microsoft requested that, and Rust does it.

It seems to me that there are three camps:

People who want some way to use os.OpenFile but opt into FILE_SHARE_DELETE semantics. They do not want to fork os.OpenFile for this purpose. They want ordinary cross-platform Go code to be able to pass this flag without third-party packages. For them, adding O_ALLOW_DELETE (ideally with it defined to be 0 on non-Windows platforms) is a good proposal.
People who want all opens on Windows to pass FILE_SHARE_DELETE. They don't want to have to change open code. They propose to change behavior for all Go programs and have asked for examples of Go programs that would be affected negatively by such a change. At least one such person wants this to be a runtime opt-in.
People who have no interest in passing FILE_SHARE_DELETE in cross-platform Go code. They don't want any behavior change, and they do not want additional flags to os.OpenFile. They point out that passing this flag can already be achieved by calling syscall.CreateFile, and that passing this flag does not eliminate all cases where an open file cannot be deleted (especially on older Windows versions).

Those in camp 1 would like to see this proposal accepted. Those in camp 2 would like to see #32088 reopened and accepted. Those in camp 3 would like both proposals rejected.
I am against #32088. I have been convinced that such a breaking change could negatively affect existing Go programs, which violates the Go compatibility guarantees. It's unfortunate that Go 1.0 did not always pass FILE_SHARE_DELETE, but here we are. I have always agreed that making it a runtime opt-in would be a mistake.
However, I am in support of this proposal. I do not see why adding an additional O_ flag would be controversial. It seems to me that exposing this capability this will help reduce Windows-specific code in software written in Go, and will make it easier to fix some existing POSIX-oriented Go code to work on Windows. That seems valuable to me.

I believe that addition of FILE_SHARE_DELETE by default will break assumptions that otherwise are true. For example, while my process has an open file handle without FILE_SHARE_DELETE, my program is sure that file does exist under its name, can be os.Stat and etc. But this is no longer true with FILE_SHARE_DELETE.
Also, what happens to file handle created with FILE_SHARE_DELETE if other process does actually attempt to delete or rename file? Is it (file handle) still valid? Does it enter some new states visible through WinAPI (if yes, it can also break some programs)?
I'm obviously from camp 1 and againsts #32088.

It sounds like there is no consensus about this option either, so we can of course keep doing nothing.
I would be more persuaded by the "Microsoft requested that" argument for doing #32088 if the argument was coming from someone working on Windows, or maintaining the Go port to Windows, instead of someone whose bio says "Container+Linux guy at Microsoft". Sorry @jstarks but it makes perfect sense for the Linux guy to want that flag. (Most of us with Unix backgrounds think it makes sense, but the Windows users seem not to.)
Is there any more official statement from Microsoft that new programs on Windows should set this flag unconditionally in their calls?

@rsc I think there is some miscommunication--I agree with you and others that #32088 is the wrong choice for Go at this point. This is a change from my original position. I am advocating for this proposal, which is to add a flag to easily allow developers to easily opt into FILE_SHARE_DELETE in code that relies on this behavior and is intended to be cross platform.
Also regardless I think "Microsoft requested" is a weird way to put things, since I am not the Go representative for Microsoft in any mutually agreed capacity (if such a position can or should even exist). I am an engineering lead on the Windows Base team (kernel, file systems, virtualization). My team maintains and contributes to open source container-related components, hence my interest in Go.
I think you've convinced me to update my bio :).

@jstarks I wonder whether you changed your mind because "a bad solution is better than none" :-)
But there is already a workable solution: patch the local stdlib so that an app can opt-in to f.s.d. There are other cases where GOOS=windows requires local patching:
File inode: https://groups.google.com/d/topic/golang-dev/raE01Fa2Kmo/discussion
os.O_SYNC flag: https://github.com/golang/go/issues/35358
We wouldn't see these problems if Go was even half as widely deployed on Windows as Linux. Ideally someone at Google or Microsoft would be testing a selection of major open source Go projects on Windows, for each Go release.
I adopted Go because it supports Windows and MacOS. It's a productive toolkit, but I'm disappointed by Go's maturity on these platforms. For example, this MacOS data-loss bug went undetected until 2018: #26650

But there is already a workable solution: patch the local stdlib

For what it's worth, even though you've been advocating for patching the standard library quite a lot recently, we generally don't consider that a workable solution to a problem.

For example, this MacOS data-loss bug went undetected until 2018: #26650

If you like that particular example read https://danluu.com/deconstruct-files/. Things are much worse pretty much everywhere.

@jstarks, thanks for the clarification about your role at Microsoft.
@alexbrainman and @mattn, we have someone who works on the kernel team at Microsoft saying that we should probably just set the flag unconditionally. Given this new information do you still think we should decline #32088? It seems like that would be less work all around, for us and for users, than adding O_ALLOW_DELETE.

@rsc could we implement #32088 as an experimental opt-in? And if that's successful, enable it by default and provide an opt-out switch in later releases?

read https://danluu.com/deconstruct-files/. Things are much worse pretty much everywhere.

Everywhere? The article covers Unix, and has no references to Windows or MacOS/Darwin. NTFS does journaling, as does HFS+ since MacOS 10.3. So the PC platforms do pretty well -- aside from the fact that Darwin fsync() is a no-op, which foiled os.File.Sync().
Of course the Go project wouldn't want to recommend DIY patching to users, but it's a cost of doing business for some Go projects, and it's generally safer than copying stdlib code and inventing new APIs. Those are reasons to support it.

@alexbrainman and @mattn, we have someone who works on the kernel team at Microsoft saying that we should probably just set the flag unconditionally. Given this new information do you still think we should decline #32088?

If you are referring to @jstarks , I think he said the opposite - from https://github.com/golang/go/issues/34681#issuecomment-556218563

I agree with you and others that #32088 is the wrong choice for Go at this point. This is a change from my original position. ...

Alex

I'm at a loss for a path forward here. I missed @jstarks's reversal above - thanks for flagging it @alexbrainman. There seem to be three options:

Do nothing, like we have been. Removes will fail if the file is open.
Set the flag always (#32088). Removes will succeed even if the file is open, although right now they apparently don't take effect until the file is closed. That could be (newly) confusing for Windows developers. On the other hand Windows has changed to the Unix behavior in the most recent versions.
Add O_ALLOW_DELETE (this issue) and make users update all their code to add it, including rewriting os.Open calls to os.OpenFile in order to pass in the flag. But even if we do that, there is no way in the API to thread it into the various places that temporarily call CreateFile, such as os.Stat or os.Readlink. And if Go code has any handles, even temporarily, without this flag, then the removes are not going to succeed.

There is not consensus about 2, which is why we closed #32088.
But 3 seems like an incomplete solution at best. It will cause a lot of churn, forcing Windows developers to avoid helpers like os.Open, and still not solve the problem 100%. So it seems like not a good option.
It seems like if we can't come to a consensus on 2, then doing 1 until a better idea presents itself is the best path forward. Historically we've been willing to wait for the right idea (or a better understanding of the problem). It seems like that's where we are now.

@rsc did you see this: "could we implement #32088 as an experimental opt-in? And if that's successful, enable it by default and provide an opt-out switch in later releases?"
That might be the better idea...

3 may be incomplete, but today we can't use os.Open or os.OpenFile already.

@networkimprov What would we learn from an experimental opt-in?  The question is not what will happen with programs aware of the behavior change; it's what will happen with programs that are not aware.  And programs that are not aware will not use the experimental opt-in.

By letting projects opt-in to default f.s.d. we provide a solution that doesn't automatically break anything, and can gather data about any conflicts with third-party programs. The experimental period could be 6, 12, or 18mos.
We can request that all projects intended for use on Windows test the opt-in when convenient.
Lacking that option, projects would likely take Alex's approach of re-inventing os.OpenFile() and only calling that on files that get renamed/removed while open. That can't yield much data re conflicts, and risks missing subsequent fixes in pkg os & syscall.

In my opinion, to request to use experimentals to anyone who does not really understand what should do is bit dangerous since some of developer who try to use the experimental will not revert the changes when the experimental is abandoned.

If the experiment is abandoned, code to enable it would not compile. And the experimental mode could be enabled by a runtime env var (i.e. no code change) for anyone who's only testing it.

To elaborate, you could either run a command
C:\...> set GOPOSIXRENAME=1 & app.exe
Or write code
func init() { os.Setenv("GOPOSIXRENAME", "1") }
to set file_share_delete in calls to os.Create/Open/OpenFile(). This functionality would be labeled experimental, meaning it might be withdrawn in a future release.

@networkimprov you seems to ignore Ian's argument.

The question is not what will happen with programs aware of the behavior change; it's what will happen with programs that are not aware. And programs that are not aware will not use the experimental opt-in.

We all spent a lot of time investigating this change. But people that will be affected by this change have very little knowledge about this matter. They won't bother trying your experiment.

The experimental period could be 6, 12, or 18mos.

I don't believe anything will change in 18 month. FILE_SHARE_DELETE will still be here affecting existing programs. If Microsoft were really interested in changing this behavior, they could have started ignoring FILE_SHARE_DELETE in latest versions of Windows. I don't see that is happening. But then we don't need to change anything in Go.
Also having experiment in Go release is too high price to pay for this feature. We already have github.com/alexbrainman/goissue34681 - perfectly fine solution to your problem.
Alex

I'm still at a loss for a path forward here, as I said last week in https://github.com/golang/go/issues/34681#issuecomment-561777243.
Here's another idea. We could decide to try #32088 by landing the change on day 1 of the Go 1.15 cycle and then wait the 3 months to see if any problems are reported before deciding to keep it. If we decided to keep it at the start of the freeze and then any serious problems arose during the freeze (the following 3 months) we could still back it out. So that would be 6 months in which people could try it, including a beta and release candidate.
If we did that trial period, would that ease some of your concerns, @alexbrainman and @mattn?

Here's another idea. We could decide to try #32088 by landing the change on day 1 of the Go 1.15 cycle and then wait the 3 months to see if any problems are reported before deciding to keep it. ...
If we did that trial period, would that ease some of your concerns, @alexbrainman and @mattn?

Not really. Nothing you said in https://github.com/golang/go/issues/32088#issuecomment-537590826 changed.
Alex

I've tried to summarize all the options offered to date. Please point out anything I've missed.

Default FILE_SHARE_DELETE
pro
- Works for os.Create/Open/OpenFile (no user code changes)
con
- May silently impact 3rd party programs (not yet identified)

Default via experimental switch
pro
- Works for os.Create/Open/OpenFile (no user code changes)
con
- Apps that enable the switch may silently impact 3rd party programs

User-patched syscall.Open
pro
- Works for os.Create/Open/OpenFile (no user code changes)
con
- Must re-patch after upgrade
- Apps that enable the patch may silently impact 3rd party programs

Non-stdlib API which duplicates os.Create/Open/OpenFile
pro
- Supported in x/sys/windows (?)
con
- Unfamiliar API for ordinary file ops (user code changes)
- May miss fixes to stdlib
- App-wide use of new API may silently impact 3rd party programs

New os.OpenFile flag O_ALLOW_DELETE
pro
- Supported in stdlib
con
- Doesn't work for os.Create/Open (user code changes)
- Meaningless on Unix
- Files opened this way may silently impact 3rd party programs

Comments:
As for

New os.OpenFile flag O_ALLOW_DELETE

Doesn't work for os.Create/Open (user code changes)

Not sure what you mean, if we add the O_ALLOW_DELETE (@rsc : "(maybe 0x100000)") to the flags param, we are safe .
Also, the above summary should consider how to handle the switched above on "old" (but most) windows versions that do no support per option, e.g.:
Default FILE_SHARE_DELETE - probably no indication
Default via experimental switch - probably no indication, unless its a "smart switch" that will output a runtime warning...no one reads those, but debugging wise it can help
User-patched syscall.Open -  should probably return an error, otherwise behavior would not be as expected.
Non-stdlib API which duplicates os.Create/Open/OpenFile-  should probably return an error, otherwise behavior would not be as expected.
New os.OpenFile flag O_ALLOW_DELETE-  should probably return an error, otherwise behavior would not be as expected.

Re (5) os.Create/Open don't take a flag argument, so would have to be replaced with os.OpenFile.
All Windows versions supported by Go recognize file_share_delete.

All Windows versions supported by Go recognize file_share_delete.

But they don't do the same thing with it. Older ones simply say they deleted the file but don't actually delete the name (it still shows up in directory listings) until the open handles are closed. Newer ones delete the name eagerly, like Unix, not lazily like the old Windows versions.
Overall it seems like there is still no consensus on what to do here, just like in https://github.com/golang/go/issues/32088#issuecomment-537590826.
Given the lack of consensus, this seems like a likely decline.
Leaving open for a week for final comments.

There's a fundamental stdlib gap here; some issue should be open to track it. I can file a new issue listing the above 5 options which have been considered to date... Thoughts?


NoGo Windows :: After restart changes not activated with Access is denied in log

Have:
LoadModule md_module modules/mod_md.so
ManagedDomain domaind.nl www.domaind.nl
MDCertificateAgreement https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf
Initial it says all fine and ask for restart.
After restart I see  (OS 5)Access is denied and asking again for restart.
It happens both when the Apache server is running with System or Administrator account.
Inititial:
[md:info] [pid 5776:tid 456] AH: mod_md (v0.5.0-git), initializing...
[md:info] [pid 10184:tid 372] AH: mod_md (v0.5.0-git), initializing...
[md:info] [pid 10184:tid 1964] domaind.nl: setup staging
[md:info] [pid 10184:tid 1964] domaind.nl: need certificate
[md:info] [pid 10184:tid 1964] registered new account https://acme-staging.api.letsencrypt.org/acme/reg/2907888
[md:info] [pid 10184:tid 1964] domaind.nl: check Terms-of-Service agreement
[md:info] [pid 10184:tid 1964] domaind.nl: setup new authorization
[md:info] [pid 10184:tid 1964] domaind.nl: setup new challenges
[md:info] [pid 10184:tid 1964] updated authz https://acme-staging.api.letsencrypt.org/acme/authz/jDHZI1F9a7KplK0RKN--ZOF9atEZfhwsM9IhTRgwdOU
[md:info] [pid 10184:tid 3020] [client <IP_ADDRESS>:52256] AH: f972ca5e2bdcc62b9ff607c4a1e8e2bd.64d86ecf28901402270935aa2f61ba94.acme.invalid: is a tls-sni-01 challenge host
[md:info] [pid 10184:tid 1964] updated authz https://acme-staging.api.letsencrypt.org/acme/authz/KRS9-Fu2BVoETVbbmxvf6xGpgNF9xCYejs-7ks2sIdw
[md:info] [pid 10184:tid 1964] domaind.nl: monitoring challenge status
[md:info] [pid 10184:tid 3020] [client <IP_ADDRESS>:52328] AH: 7237dd3a897d38e80ae12788930936f7.c605a08202905d490a609ccdaaba57a3.acme.invalid: is a tls-sni-01 challenge host
[md:info] [pid 10184:tid 1964] domaind.nl: checked all domain authorizations
[md:info] [pid 10184:tid 1964] domaind.nl: creating certificate request
[md:info] [pid 10184:tid 1964] domaind.nl: received certificate
[md:info] [pid 10184:tid 1964] domaind.nl: retrieving certificate chain
[md:notice] [pid 10184:tid 1964] AH: 1 Managed Domain has been setup and changes will be activated on next (graceful) server restart.
After restart:
[md:info] [pid 5776:tid 456] AH: mod_md (v0.5.0-git), initializing...
[md:notice] [pid 10184:tid 1964] AH: 1 Managed Domain has been setup and changes will be activated on next (graceful) server restart.
[md:error] [pid 5776:tid 456] (OS 5)Access is denied.  : rename from D:/servers/apacheS/md/domains/domaind.nl to D:/servers/apacheS/md/archive/domaind.nl.1
[md:error] [pid 5776:tid 456] (OS 5)Access is denied.  : AH: domaind.nl: error loading staged set
[md:error] [pid 10520:tid 364] (OS 183)Cannot create a file when that file already exists.  : creating archive dir: D:/servers/apacheS/md/archive/domaind.nl.1
[md:error] [pid 10520:tid 364] (OS 183)Cannot create a file when that file already exists.  : AH: domaind.nl: error loading staged set
[md:info] [pid 10520:tid 364] AH: mod_md (v0.5.0-git), initializing...
[md:error] [pid 10520:tid 364] (OS 183)Cannot create a file when that file already exists.  : creating archive dir: D:/servers/apacheS/md/archive/domaind.nl.1
[md:error] [pid 10520:tid 364] (OS 183)Cannot create a file when that file already exists.  : AH: domaind.nl: error loading staged set
[md:info] [pid 10520:tid 2044] domaind.nl: all data staged
[md:notice] [pid 10520:tid 2044] AH: 1 Managed Domain has been setup and changes will be activated on next (graceful) server restart.

After ~12 hours I see in the log:
[md:warn] [pid 1864:tid 1964] md(domaind.nl): looks complete, but has unknown expiration date.

When I run a2md, the result is: Unrecognized Win32 error code 1

a2md -vvv -a https://acme-staging.api.letsencrypt.org/directory -d d:\apache24\md  drive domaind.nl

Output:
[src\md_store_fs.c:350 trace2][ok] loading type 1 from d:\apache24\md/domains/domaind.nl/md.json
[src\md_store_fs.c:350 trace2][2(No such file or directory)] loading type 2 from d:\apache24\md/domains/domaind.nl/cert.pem
[src\md_store_fs.c:350 trace2][2(No such file or directory)] loading type 3 from d:\apache24\md/domains/domaind.nl/pkey.pem
[src\md_store_fs.c:350 trace2][2(No such file or directory)] loading type 4 from d:\apache24\md/domains/domaind.nl/chain.pem
[src\md_reg.c:166 debug][ok] md{domaind.nl}: incomplete, without private key
[md_cmd_reg.c:275 info][ok] domaind.nl: incomplete, sign up
[src\md_store_fs.c:350 trace2][ok] loading type 1 from d:\apache24\md/httpd.json
[src\acme\md_acme_drive.c:628 trace1][ok] domaind.nl: init driver
[src\md_store_fs.c:350 trace2][ok] loading type 1 from d:\apache24\md/staging/domaind.nl/md.json
[src\acme\md_acme_drive.c:631 trace1][ok] domaind.nl: checked stage md
[src\md_store_fs.c:350 trace2][ok] loading type 2 from d:\apache24\md/staging/domaind.nl/cert.pem
[src\md_store_fs.c:350 trace2][ok] loading type 3 from d:\apache24\md/staging/domaind.nl/pkey.pem
[src\md_store_fs.c:350 trace2][ok] loading type 4 from d:\apache24\md/staging/domaind.nl/chain.pem
[src\acme\md_acme_drive.c:661 debug][ok] domaind.nl: checked creds
[src\md_reg.c:847 debug][ok] domaind.nl: run staging
[src\acme\md_acme_drive.c:680 debug][ok] domaind.nl: staging started, state=1, can_http=0, can_https=1, challenges='tls-sni-01 http-01'
[src\acme\md_acme_drive.c:689 info][ok] domaind.nl: all data staged
[src\acme\md_acme_drive.c:803 debug][ok] domaind.nl: ACME, staging done
[src\md_reg.c:850 debug][ok] domaind.nl: staging done
[md_cmd_reg.c:278 info][ok] domaind.nl: loading
[src\md_store_fs.c:350 trace2][ok] loading type 1 from d:\apache24\md/domains/domaind.nl/md.json
[src\md_store_fs.c:350 trace2][2(No such file or directory)] loading type 2 from d:\apache24\md/domains/domaind.nl/cert.pem
[src\md_store_fs.c:350 trace2][2(No such file or directory)] loading type 3 from d:\apache24\md/domains/domaind.nl/pkey.pem
[src\md_store_fs.c:350 trace2][2(No such file or directory)] loading type 4 from d:\apache24\md/domains/domaind.nl/chain.pem
[src\md_reg.c:166 debug][ok] md{domaind.nl}: incomplete, without private key
[src\md_store_fs.c:350 trace2][ok] loading type 1 from d:\apache24\md/httpd.json
[src\acme\md_acme_drive.c:628 trace1][ok] domaind.nl: init driver
[src\md_store_fs.c:350 trace2][ok] loading type 1 from d:\apache24\md/staging/domaind.nl/md.json
[src\acme\md_acme_drive.c:631 trace1][ok] domaind.nl: checked stage md
[src\md_store_fs.c:350 trace2][ok] loading type 2 from d:\apache24\md/staging/domaind.nl/cert.pem
[src\md_store_fs.c:350 trace2][ok] loading type 3 from d:\apache24\md/staging/domaind.nl/pkey.pem
[src\md_store_fs.c:350 trace2][ok] loading type 4 from d:\apache24\md/staging/domaind.nl/chain.pem
[src\acme\md_acme_drive.c:661 debug][ok] domaind.nl: checked creds
[src\md_reg.c:915 debug][ok] domaind.nl: run load
[src\acme\md_acme_drive.c:821 debug][ok] domaind.nl: preload start
[src\md_store_fs.c:350 trace2][ok] loading type 1 from d:\apache24\md/staging/domaind.nl/md.json
[src\md_store_fs.c:350 trace2][ok] loading type 2 from d:\apache24\md/staging/domaind.nl/cert.pem
[src\md_store_fs.c:350 trace2][ok] loading type 4 from d:\apache24\md/staging/domaind.nl/chain.pem
[src\md_store_fs.c:350 trace2][ok] loading type 3 from d:\apache24\md/staging/domaind.nl/pkey.pem
[src\md_store_fs.c:350 trace2][ok] loading type 1 from d:\apache24\md/staging/domaind.nl/account.json
[src\md_store_fs.c:350 trace2][ok] loading type 3 from d:\apache24\md/staging/domaind.nl/account.pem
[src\md_store_fs.c:350 trace2][720002(Unrecognized Win32 error code 2)] loading type 1 from d:\apache24\md/staging/domaind.nl/authz.json
[src\md_store_fs.c:503 debug][ok] start remove of md staging/domaind.nl/authz.json
[src\acme\md_acme_drive.c:862 debug][ok] domaind.nl: staged data load, purging tmp space
[src\md_store_fs.c:562 debug][ok] purge tmp/domaind.nl (d:\apache24\md/tmp/domaind.nl)
[src\acme\md_acme_drive.c:879 debug][ok] domaind.nl: saved ACME account ACME-.letsencrypt.org-0003
[src\acme\md_acme_drive.c:913 debug][ok] domaind.nl: ACME, preload done
[src\md_store_fs.c:683 err][720183(Unrecognized Win32 error code 1)] creating archive dir: d:\apache24\md/archive/domaind.nl.1
[src\md_reg.c:937 debug][720183(Unrecognized Win32 error code 1)] domaind.nl: load done
[md_cmd_reg.c:297 info][720183(Unrecognized Win32 error code 1)] domaind.nl: error activating new credentials

How does your md\archive dir look? I assume there already is a sub directory named domaind.nl.1?

yes, and no files in it.

@SteffenAL I just pushed a new version where I changed the archiving handling on Windows.
The downside: I needed to restructure the source tree to become similar to how it will look in Apache subversion. Not sure how much that breaks your build or if it makes it easier.

One source tree is much better, and a2md in modules is no problem for me (more or less it belongs in support). You renamed also some files.
Building 0.6.0  all fine.
Now it configures a certificate, although it is not a valid certificate,  I can browse https with it, you can try https://www.vosadministraties.nl/  The certificate says: Issued by: Fake LE Intermediate X1.
Runs on the https://www.apachelounge.com/ production server which has more (vhost)domains with  valid certificates.
Config:
LoadModule watchdog_module modules/mod_watchdog.so
LoadModule md_module modules/mod_md.so
ManagedDomain vosadministraties.nl www.vosadministraties.nl
And the vhosts:
<VirtualHost *:443>
ServerName ........
ServerAlias ........
.....
SSLEngine on
SSLCertificateFile conf/apachelounge.com-chain.pem
SSLCertificateKeyFile conf/apachelounge.com-key.pem
< /VirtualHost>
...
...
<VirtualHost *:443>
ServerName www.vosadministraties.nl
ServerAlias vosadministraties.nl
.....
SSLEngine on
< /VirtualHost>
Log:
[md:info] [pid 11524:tid 448] AH: vosadministraties.nl: staged set activated
[md:info] [pid 11524:tid 448] AH: mod_md (v0.6.0-git), initializing...
[ssl:info] [pid 11524:tid 448] AH01914: Configuring server www.vosadministraties.nl:443 for SSL protocol
[ssl:info] [pid 11524:tid 448] AH02568: Certificate and private key www.vosadministraties.nl:443:0 configured from D:/servers/apacheS/md/domains/vosadministraties.nl/cert.pem and D:/servers/apacheS/md/domains/vosadministraties.nl/pkey.pem


Do not allow workers to produced extracted payloads identical to the scanned payload.

I have had a few instances where I scanned a payload with a worker plugin, and the worker plugin produced an extracted payload that was identical to the payload being scanned by the worker.
For example, review the following sample:
https://www.virustotal.com/gui/file/b180cf82624994d05f7bcdf221372d5149e6382e52036d5b1487a3e5f3f12144/details
It is a corrupt PE that has a section (.bss) that is equal to the PE itself.
If a worker plugin carved out the PE as an extracted payload, the stoQ deduplication logic would prevent the file from being analyzed over and over, but the stoQ object would contain a circular reference, and would not be able to be serialized into a JSON object.
I feel like it would be a worthwhile update to stoQ to prevent this from happening, a simple check of extracted payloads to see if they match the parent payload and to remove the extracted payload if it is the same.  A warning could be logged.

@brbickel @mgeide Please let me know your thoughts on this...

I created PR #164


chore: remove remaining MoveIn compononts

Description
References

Codecov Report

Merging #515 (1defb06) into main (21978a1) will decrease coverage by 40.32%.
Report is 194 commits behind head on main.
The diff coverage is 37.17%.

@@             Coverage Diff             @@
##             main     #515       +/-   ##
===========================================
- Coverage   83.07%   42.75%   -40.32%
===========================================
  Files         456      290      -166
  Lines       10418     5592     -4826
===========================================
- Hits         8655     2391     -6264
- Misses       1438     3123     +1685
+ Partials      325       78      -247

Files
Coverage Δ

source/Application/Actors/CreateActorHandler.cs
0.00% <ø> (-85.72%)
:arrow_down:

...tion/Configuration/Authentication/Authenticated.cs
0.00% <ø> (-100.00%)
:arrow_down:

...n/Configuration/Authentication/NotAuthenticated.cs
0.00% <ø> (-100.00%)
:arrow_down:

...n/Configuration/Commands/CommandSchedulerFacade.cs
0.00% <ø> (-100.00%)
:arrow_down:

...Configuration/Commands/Commands/InternalCommand.cs
0.00% <ø> (-100.00%)
:arrow_down:

...e/Application/Configuration/Queries/QueryResult.cs
0.00% <ø> (ø)

...lication/Configuration/TimeEvents/ADayHasPassed.cs
0.00% <ø> (-100.00%)
:arrow_down:

...Configuration/TimeEvents/TenSecondsHasHasPassed.cs
0.00% <ø> (-75.00%)
:arrow_down:

...urce/Application/IncomingMessages/MessageHeader.cs
95.45% <100.00%> (-4.55%)
:arrow_down:

...tgoingMessages/Common/OutgoingMessageExtensions.cs
0.00% <ø> (ø)

... and 54 more

... and 418 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


⚠ SkipTheCommericals has degraded performance

In 58c568d, SkipTheCommericals ($STC) experienced degraded performance:

HTTP code: 200
Response time: 2904 ms

Resolved: SkipTheCommericals performance has improved in 352f44f.


Errors running tests on MacOSX

I'm getting errors across the whole test suite when running them locally on my Mac OSX Catalina 10.15.4 machine.
    os.makedirs(path, exist_ok=True)
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py:213: in makedirs
    makedirs(head, exist_ok=exist_ok)
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py:223: in makedirs
    mkdir(name, mode)
E   FileNotFoundError: [Errno 2] No such file or directory: '../../../../var/folders/h3/k5sgzmp17cvbz1mm459fkf200000gn/T/tmp9ixj7r4i/fasta'

Looking around, I think this is something to do with the permissions on the /var/folders/ directories
Is there a way we can specify what temporary to use here?

You should be able to specify the path to the temporary directory via an environment variable. See https://docs.python.org/3/library/tempfile.html#tempfile.mkstemp

Setting TMPDIR worked 👍 . Thanks!


Code and dataset availability

Hi authors,
Thanks for the nice work! Just kindly asking when the code and dataset with split to reproduce your result will be available?
Best,
Jiaqi

Hi Jiaqi,
Thank you for your interest in our work! :slightly_smiling_face: I am currently in the process of cleaning up the code. I plan to release the code and the datasets in the next 2-3 weeks.
Best,
Miltos

Dear Jiaqi,
The code cleanup took longer than expected, but I have now released the source code and the datasets. I am closing this issue, but feel free to create a new one if you have more questions.
Best,
Miltos


Can't place tf.nn.moments() on GPU when first dimension is None

At first this sounds like #2508, but this was closed and apparently fixed and the error message is also different. So it might be more related to #139, and possibly to the comment of @mrry on tf.mean on GPUs. The example below works with cpu device context (or no explicit device placement at all). Is this a missing feature or should one not expect that this is fixable?
Environment info
Operating System:
Ubuntu 15.10
Installed version of CUDA and cuDNN:
CUDA7.5/cuDNN4.0.4
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
-rw-r--r-- 1 root root 189170 Jan  1 23:25 /usr/local/cuda/lib/libcudadevrt.a
lrwxrwxrwx 1 root root     16 Jan  1 23:25 /usr/local/cuda/lib/libcudart.so -> libcudart.so.7.5
lrwxrwxrwx 1 root root     19 Jan  1 23:25 /usr/local/cuda/lib/libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x 1 root root 311596 Jan  1 23:25 /usr/local/cuda/lib/libcudart.so.7.5.18
-rw-r--r-- 1 root root 558020 Jan  1 23:25 /usr/local/cuda/lib/libcudart_static.a

If installed from sources, provide the commit hash: fc91629
Steps to reproduce
import tensorflow as tf
import numpy as np

with tf.device("/gpu:0"):
    x = tf.placeholder(tf.float32, shape=[None, 5])
    mean, var = tf.nn.moments(x, axes=[0])
sess = tf.Session()
sess.run([mean, var], feed_dict={x:np.random.randn(10, 5)})

results in
E tensorflow/core/client/tensor_c_api.cc:485] Cannot assign a device to node 'moments/sufficient_statistics/SparseToDense': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
     [[Node: moments/sufficient_statistics/SparseToDense = SparseToDense[T=DT_BOOL, Tindices=DT_INT32, validate_indices=true, _device="/device:GPU:0"](moments/sufficient_statistics/SparseToDense/sparse_indices, moments/sufficient_statistics/Shape_1, moments/sufficient_statistics/SparseToDense/sparse_values, moments/sufficient_statistics/SparseToDense/default_value)]]

What have you tried?

Setting shape=[10, 5] fixes the above problem (to be expected).
Setting device context to with tf.device("/cpu:0"): works with None shape info, too.

The call stack is as such: tf.nn.moments() calls sufficient_statistics(), which contains this part:
    if x_shape.is_fully_defined():
        <some code>
    else:  # shape needs to be inferred at runtime.
      x_shape = array_ops.shape(x)
      select_axes = sparse_ops.sparse_to_dense(axes, array_ops.shape(x_shape),
                                               True, False)

Since the shape contains None the else branch is taken.  Lastly, the SparseToDense op only has a CPU kernel, not a GPU one.  Hence the failure.
Possible fixes:

Somehow change the tf.nn.moments() code path, so that sparse_to_dense() is avoided.
Or, implement a SparseToDense GPU kernel.

Marking as contribution welcome, since we're not likely to work on this in the near future.

/cc @vincentvanhoucke

Yes, it's a known wrinkle, that would only be properly solved with a dedicated op or a GPU implementation of sparse_to_dense. In practice, there should be no good reason to want to enforce a hard placement of all the ops on GPU though. With soft placement, you may let the graph mostly run on GPU and back off to CPU only for this one op without a GPU equivalent.

Thank you for the replies. Out of curiosity, would the performance hit of going down to cpu and up again be negligible? The tf.nn.moments is used in a BatchNorm layer in this case, so with ResNets, many of this transfers would happen. Or is using None in the batchsize itself having already such a negative effect (e.g. because some optimizations can't be realized offline) that this copying does not add that much overhead?

This op operates on four floats. There should be no impact at all.

@osdf: for now, I think a workaround is to use
    sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True))

which should attempt to place non-GPU-available ops on CPU instead.

Thank you @concretevitamin and @vincentvanhoucke. Do you think this issue should be closed? If it turns out that somebody needs this to be placed on GPU it could be reopened. (I briefly thought of taking a stab at it but a bit time-limited right now). Apart from that it serves as a reference if anybody else encounters it.

Please leave it open. It's likely others will hit the same problem.

Pending https://github.com/tensorflow/tensorflow/pull/6031 should fully address this.

I think the issue is unresolved. I thought it would be better to post here, continuing the discussion.
Take this toy example:
with tf.device('/gpu:2'):
    x = tf.placeholder(tf.float32, shape=[None, 100])
    weight_dense_1 = tf.Variable(tf.zeros([100, 10]))
    dense_1_out = tf.matmul(x, weight_dense_1)
    mean, var = tf.nn.moments(dense_1_out, [0], shift=None, name=None, keep_dims=False)
    coeff = tf.rsqrt(var + 1e-8)
    y = dense_1_out * coeff - mean * coeff
    grad = tf.gradients(y, [weight_dense_1], colocate_gradients_with_ops=True)

This immediately shows up some warnings:
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/Rank with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/range_1/start with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/range_1/delta with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/range_1 with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/ListDiff with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/concat/axis with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/concat with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/Gather with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/Const with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/Prod with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/Gather_1 with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/Const_1 with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.
WARNING:tensorflow:Tried to colocate gradients/moments/sufficient_statistics/count_grad/Prod_1 with an op moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs /device:GPU:2. Ignoring colocation property.

And if I force it to run(by setting allow_soft_placement=False), an error is occured:
InvalidArgumentError: Cannot colocate nodes 'gradients/moments/sufficient_statistics/count_grad/Const_1' and 'gradients/moments/sufficient_statistics/count_grad/Cumprod_1/axis: Cannot merge devices with incompatible types: '/GPU:2' and '/CPU:0'
	 [[Node: gradients/moments/sufficient_statistics/count_grad/Const_1 = Const[_class=["loc:@moments/sufficient_statistics/count"], dtype=DT_INT32, value=Tensor<type: int32 shape: [1] values: 0>, _device="/device:CPU:0"]()]]

When setting the placeholder's batch size explicitly, the warning is gone.
I'm using pip3-installed tensorflow-gpu==1.0.1, with Python 3.5.2.
CUDA 8.0.61, cuDNN 5.1.10 is installed on Ubuntu 16.04.

As the problem seems to be about a whole different things, I'm opening a new issue.


SimpleWriter add_way encountered MemoryError

Hi
I actually use SimpleWriter class in order to persist isolines generated by GDAL contour function.
I use "OsmObject" class like in example
class OsmObject(object):
    def __init__(self, **params):
        for k,v in params.items():
            setattr(self, k, v)

And than for each isoline, add way like below. nodes_ids is the array that contains ids of previously written nodes.
tags = {attr_name: str(int(altitude)), "contour": "elevation", "contour_ext": contour_ext, "id": str(ways_counter), "nodesnm": str(len(nodes_ids))}
pbf_writer.add_way(OsmObject(id=ways_counter, nodes=nodes_ids, tags=tags, version=1))

My problem is when I generate a huge amount of ways (3 739 825 when last error occurs), I've got a memory error, always when writing ways. I made several tests with more available memory. It shows that more memory I have, later the problem occurs. I use Python 3.5 on Ubuntu.

Traceback (most recent call last):
File "generatesrtmosm.py", line 395, in
create_isolines(hgtfile, base_filename, args.i)
File "generatesrtmosm.py", line 82, in create_isolines
add_way(gen_feat, bbox, pbf_writer)
File "generatesrtmosm.py", line 122, in add_way
process_points(final_linestring.GetPoints(), altitude, contour_ext)
File "generatesrtmosm.py", line 146, in process_points
pbf_writer.add_way(OsmObject(id=ways_counter, nodes=nodes_ids, tags=tags, version=1))
MemoryError

In attachement, a line chart that shows memory consumption during my processing.
In this example, I want to generate isolines for Mexico: 260 HGT files to process.
For each HGT file, I call GDAL contour function, then I use the result to add nodes and associated ways.
I close the writer only at the end of the processing.
We can see memory consumption behavior on next chart.

It seems that this behavior change when PBF file size is more than 1 Gb. 1.8 Gb when last error occurs.
Does I misunderstand something? Does the amount of nodes and ways is too huge for pyosmium ?
I would appreciate to have an explanation about my current problem.
Thank you.

Thank you for your help.

I saw an error in my Python code (SimpleWriter instance wasn't marked global in a function).  So I decided to test it again. For now, 112 HGT has been processed on 260 and the behavior seems to be the same. I'm waiting for error.
After, I will use a SimpleWriter with a bigger buffer.
Then, I will test with your change. I hope I will succeed in compiling your code. Actually, I use the automatic install.

Yesterday, I also mimic generation of 5 000 000 ways with 2 000 nodes for each way. No problems.
I will give you more information after having complete tests.

Hi Sarah,
I reproduced the error with the code below :
import osmium as o
import os, psutil

class OsmObject(object):
    def __init__(self, **params):
        for k,v in params.items():
            setattr(self, k, v)

if __name__ == "__main__":
    process = psutil.Process(os.getpid())
    print("MEM at init : ", process.memory_info().rss)

    pbf_writer_name = "/home/xx/Maps/hugefile.osm.pbf"
    if os.path.exists(pbf_writer_name):
        os.remove(pbf_writer_name)

    pbf_writer = o.SimpleWriter(pbf_writer_name)

    # id start as the same position than in my original code
    global_nodes_counter = 10000000
    global_ways_counter = 10000000

    # For Mexico, I have 260 HGT, with a mean of 24 000 ways
    for hgt_counter in range(0, 260):
        for way_counter in range(0, 24000):
            nodes_ids = []
            # 2000 is the upper limit of nodes by way (when managing contour lines created by GDAL)
            for node_counter in range(0, 2000):
                # Add node to output osm.pbf & increment nodes counter
                pbf_writer.add_node(OsmObject(id=global_nodes_counter, location=(1.111111, 43.111111), version=1))
                nodes_ids.append(global_nodes_counter)
                global_nodes_counter += 1

            tags = {"elev": str(int(100)), "contour": "elevation", "contour_ext": "elevation_minor",
                    "id": str(global_ways_counter), "nodesnm": str(len(nodes_ids))}
            pbf_writer.add_way(OsmObject(id=global_ways_counter, nodes=nodes_ids, tags=tags, version=1))
            global_ways_counter += 1

        print("HGT {0} - MEM Usage : {1}".format(hgt_counter + 1, process.memory_info().rss))

    pbf_writer.close()
    print("global_ways_counter = {0} / global_nodes_counter = {1}".format(global_ways_counter, global_nodes_counter))

It's generate the error :
xx@xx:~$ python3.5 /home/xx/PycharmProjects/TestOsmium/osmhugepbfwrite.py
MEM at init :  16928768
HGT 1 - MEM Usage : 237899776
HGT 2 - MEM Usage : 91090944
HGT 3 - MEM Usage :<PHONE_NUMBER>
HGT 4 - MEM Usage : 873340928
HGT 5 - MEM Usage : 728633344
HGT 6 - MEM Usage :<PHONE_NUMBER>
Traceback (most recent call last):
  File "/home/xx/PycharmProjects/TestOsmium/osmhugepbfwrite.py", line 36, in <module>
    pbf_writer.add_way(OsmObject(id=global_ways_counter, nodes=nodes_ids, tags=tags, version=1))
MemoryError
xx@xx:~$

Do you obtain the same result?
Thank you for your help.

Thanks for creating a test case. I've just tried and I get a perfectly bound memory usage:
MEM at init :  16650240
HGT 1 - MEM Usage : 29614080
HGT 2 - MEM Usage : 29614080
HGT 3 - MEM Usage : 29884416
HGT 4 - MEM Usage : 29884416
HGT 5 - MEM Usage : 29884416
HGT 6 - MEM Usage : 29884416
HGT 7 - MEM Usage : 29884416

Is it possible that there was still a trace of the older pyosmium version left? I'm about to release a new version tonight. Can you try with that later?

Hi Sarah,
I really appreciate your help.
I'm not comfortable with Python and Linux and yesterday, I made an error when I built and installed last branch. I'm under Python3 and forgot "3" on command line.
So, for newbies like me, if you are under Python3, you should install Pyosmium like this:
git clone git://github.com/osmcode/pyosmium.git
cd pyosmium/
git clone --quiet --depth 1 https://github.com/osmcode/libosmium.git ../libosmium
sudo python3 setup.py build
sudo python3 setup.py install

For now, on test case, it works fine:
MEM at init :  17039360
HGT 1 - MEM Usage : 30539776
HGT 2 - MEM Usage : 30691328
HGT 3 - MEM Usage : 30691328
HGT 4 - MEM Usage : 30691328
HGT 5 - MEM Usage : 30691328
HGT 6 - MEM Usage : 30691328
HGT 7 - MEM Usage : 30691328
HGT 8 - MEM Usage : 30691328
HGT 9 - MEM Usage : 30691328

Currently, contour lines generation for Mexico is in progress.
I hope that I will be able to give you the good news this evening.
Thank you.

No worries, I'm glad to see the test case works for you now, too. Your installation steps are correct but please don't use sudo for the build step.
In any case, the new release is out and contains the fix, so you should be able to go back to using pip3 install -U osmium.


docs: update 5.seo-meta.md

Add the await keyword before useFetch. useFetch is an asynchronous function that returns a Promise, so you need to use the await keyword to wait for the Promise to resolve before assigning its value to the post variable. This ensures that the data is available before it is used in the useServerSeoMeta function.

🔗 Linked issue

❓ Type of change

[x] 📖 Documentation (updates to the documentation, readme or JSdoc annotations)
[ ] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality like performance)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

📝 Checklist

[ ] I have linked an issue or discussion.
[x] I have updated the documentation accordingly.

On second thought probably don't need to use AWAIT here since it's a computed getter.

Thank you @dosstx
Actually I thought about it as well, but for the useSeoMeta example, I believe useFetch is not the right example.
Will update your PR to improve the example.
See my demo on https://stackblitz.com/edit/github-vlzrtd?file=app.vue

Preview link: https://nuxt.com/docs/getting-started/seo-meta?preview=83ceb1ab7b8ae2614a7d3b70bc1eea07


POL-469 Adding the finops foundation waste sensor

Description
[Describe what this change achieves]
Issues Resolved
[List any existing issues this PR resolves]
Contribution Check List

[ ] All tests pass.
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable
[ ] New functionality has been documented in CHANGELOG.MD

          2 Errors

:no_entry_sign:
Please add a CHANGELOG.md file

:no_entry_sign:
Please add labels to this Pull Request

  Generated by :no_entry_sign: Danger


wired line-break on windows10(ssh to ubuntu)

I tried 4 shell client(cmd, mintty, powershell, cmder(conemu))
and adjust many font(monaco, pmingliu, consolas ...etc)
this problem still happened.
(my mac use iTerm is ok)
here is screenshot, please ignore font error(because that i only use cmder for shell client)
cmd:

powershell:

cmder:

mintty(git bash)

thanks!

Hi @yoyoys! Sorry you're having issues!
I don't have a Windows machine, so I can't easily debug this. That said, there are a number of people who are running P9k in WSL without issues.
Check out this blog post and see if it helps:
http://jessicadeen.com/tech/microsoft/badass-terminal-fcu-wsl-edition-oh-my-zsh-powerlevel9k-tmux-and-more/

Hey @yoyoys  - I'm going to close this one since we haven't heard back from you. Please feel free to re-open if you are still having issues!

@yoyoys  I got the same problem, you should try wsl-terminal which based on mintty and wslbridge


Editting workflows is a nightmare > would be good to

It might be an idea to mention editting a workflow can be very laberous, so users better think about how they set them up. Better would be MS to supply a more advanced workflow editor making for so much less hours spent all over the world editing workflows
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 74f112cc-ba95-a7c5-4d6e-ef8e69cad1b7
Version Independent ID: 7cc13e40-96fa-1b2b-6e46-262f83d6e0ce
Content: Configure a workflow in Dynamics 365 Customer Engagement (on-premises)
Content Source: ce/customerengagement/on-premises/customize/configure-workflow-steps.md
Product: d365ce-op
GitHub Login: @Mattp123
Microsoft Alias: matp

Thanks for your comment, @raoulteeuwenPE. This issue has been assigned to the doc author for review.

@raoulteeuwenPE - Thanks for your feedback. For product-related feedback, visit https://experience.dynamics.com/ideas/.
Closing this issue.


cadvisor: fix supported platforms

Motivation for this change
As far as I can tell, cadvisor is as of now very linux-centric
Things done

[x] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

It's a little confusing because Docker supports macOS. This works by running everything in a Linux VM though. AFAICT advisor runs on the Linux VM in these cases not Mac OS.

yeah, the preferred way of running cadvisor these days is as a container itself, so definitely running in Linux


Update white-hat-sender.md

Grammatical updates + link correction
Description of the change:
Reason for the change:
Link to original source:

Closes #

Hello @joanna-roberts,Thanks again for the PR! It's HACKTOBERFEST! We want to show our appreciation by sending                                              you some special Hacktoberfest swag. If you have not already, could you please fill out                                              this form so we can send it to you? Thanks!                                             Team SendGrid DX


title change

We're so eager to add your sample to this repository as soon as possible, but we need all samples follow the same structure.
To make sure we process it as soon as possible, make sure to follow these instructions.

Q
A

Bug fix?
yes

New feature?
no

New sample?
no

Related issues?
fixes #40  , partially #Y, mentioned in #Z

Solution Details

Q
A

Solution requires premium license
no

Which connectors are used?
e.g.: SharePoint, DataVerse, DataVerse for Teams, Outlook, DocuSign, etc.

Solution uses on-premises connectors?
no

Solution uses custom connectors?
no

What's in this Pull Request?
fixing issue #40

had typo in description, this fix is for issue #40

@aprildunnam good afternoon,
this should resolve wrong link issue #40
Thanks,

Thanks for submitting the PR @ValerasNarbutas.  Apologies for the delay, it's been very busy around here. Thanks for your patience!

Thanks for submitting the PR @ValerasNarbutas. Apologies for the delay, it's been very busy around here. Thanks for your patience!

👋 Thank you for getting back to me @aprildunnam  🤗 No worries about the delay, I totally understand that things can get busy.   🤞


First production set from SynGO

Hmmm... This doesn't look right.  Think I may have messed up GIT workflow.  Deleting this one and trying again.


When the two blocks are combined together, the wrong code is generated

when “event_whenmicrobitbuttonpressed” and "control_forever" are combined together,the code generated is wrong,there is not a entry for the progarm

Block structures starting with other heads will cause similar problems.


.Net: Update Ollama Client in Ollama Connector to use OllamaSharp

Currently, the Ollama connector implements its own client.
Consider replacing the current implementation with OllamaSharp.
Related samples

https://elbruno.com/2024/04/01/📎-extending-semantickernel-using-ollamasharp-for-chat-and-text-completion/
https://www.luisllamas.es/en/how-to-use-llama3-with-ollama-sharp/

nuget OllamaSharp only support net5.0, can not install in project
Package OllamaSharp 1.1.1 is not compatible with netstandard2.0 (.NETStandard,Version=v2.0). Package OllamaSharp 1.1.1 supports: net5.0 (.NETCoreApp,Version=v5.0)

Now that Ollama supports the OpenAI API standard, we may not even need a specialized Ollama connector (since a user will simply be able to use the OpenAI connector, when we pick this issue up, we will validate that this is the case and proceed accordingly

Now that Ollama supports the OpenAI API standard, we may not even need a specialized Ollama connector (since a user will simply be able to use the OpenAI connector, when we pick this issue up, we will validate that this is the case and proceed accordingly

FYI even though Ollama does have that feature, it's currently buggy, so unless the external issues are fixed it's still necessary to use the native Ollama protocol. Longer term there also no reason to assume the OpenAI API spec will cover all possible future Ollama features, so it would be a bit of a gamble to rely on the OpenAI protocol exclusively.

Done


Include PGRAs (postgraduate research assistants) on postdoc page

Looked at Postdoc page on test site and Merle is there


[incubator][VC] test: add pv related test

filter readonly action after run test
test coverage 0->73.8%
Signed-off-by: zhuangqh<EMAIL_ADDRESS>

/lgtm
/approve


Recalbox 8.1

A few days ago Recalbox 8.1 was released, is it compatible with the GPI case 2 script or do I need to wait for the update?

I tried applying the 8.0 patch onto the recalbox 8.1 image and it didn't work, I am stuck at the black screen all the time.
I switched to retropie and it works, confirming the recalbox patch requires an update

The issues is solved, this patch is no longer needed as GPI case 2 is natively supported by recalbox since the 8.1 update


Building and signing transactions is done synchronously

TxBuilderResult.js builds transactions and TransactionSigner.js signs transactions as part of their render methods, which means we can't do anything asynchronous as part of this. We would like to be able to fetch some information to make building transactions easier (#387 for example) but can't because of this constraint.

This is irrelevant. As a general direction, we are decomposing build tx and signing tx in most tools


Could not install

Hi everyone,
I'm running npm install -D paseto.js
First I was getting an error in node-gyp rebuild which brought me to python is missing
I have installed the windows-build-tools on node (npm install --global windows-build-tools)
and now it appears the package is building but it spits out errors?
`
PS C:\Users\Pietro\projects\x> npm install -D paseto.js

<EMAIL_ADDRESS>install C:\Users\Pietro\projects\x\node_modules\paseto.js
node-gyp rebuild

C:\Users\Pietro\projects\x\node_modules\paseto.js>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\....\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
extcrypto.cc
win_delay_load_hook.cc
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(27): error C2660: 'v8::Function::Call': function does not take 3 arguments [C:\Users\Pietro\projects\x\node_modules\paseto.js\build\extcrypto_addon.vcxproj]
c:\users\pietro.node-gyp\12.6.0\include\node\v8.h(4148): note: see declaration of 'v8::Function::Call' (compiling source file ..\extcrypto\extcrypto.cc)
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(35): error C2660: 'v8::Function::Call': function does not take 3 arguments [C:\Users\Pietro\projects\x\node_modules\paseto.js\build\extcrypto_addon.vcxproj]
c:\users\pietro.node-gyp\12.6.0\include\node\v8.h(4148): note: see declaration of 'v8::Function::Call' (compiling source file ..\extcrypto\extcrypto.cc)
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(56): warning C4996: 'v8::String::NewFromUtf8': was declared deprecated [C:\Users\Pietro\projects\GameServices\FitXR
.AdminPortal\node_modules\paseto.js\build\extcrypto_addon.vcxproj]
c:\users\pietro.node-gyp\12.6.0\include\node\v8.h(2890): note: see declaration of 'v8::String::NewFromUtf8'
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(74): warning C4996: 'v8::String::NewFromUtf8': was declared deprecated [C:\Users\Pietro\projects\GameServices\FitXR
.AdminPortal\node_modules\paseto.js\build\extcrypto_addon.vcxproj]
c:\users\pietro.node-gyp\12.6.0\include\node\v8.h(2890): note: see declaration of 'v8::String::NewFromUtf8'
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(92): warning C4996: 'v8::String::NewFromUtf8': was declared deprecated [C:\Users\Pietro\projects\GameServices\FitXR
.AdminPortal\node_modules\paseto.js\build\extcrypto_addon.vcxproj]
c:\users\pietro.node-gyp\12.6.0\include\node\v8.h(2890): note: see declaration of 'v8::String::NewFromUtf8'
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(115): error C2664: 'v8::String::Utf8Value::Utf8Value(const v8::String::Utf8Value &)': cannot convert argument 1 fro
m 'v8::Localv8::String' to 'const v8::String::Utf8Value &' [C:\Users\Pietro\projects\x\node_modules\paseto.js\build\extcrypto_addon.vcxproj]
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(115): note: Reason: cannot convert from 'v8::Localv8::String' to 'const v8::String::Utf8Value'
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(115): note: No user-defined-conversion operator available that can perform this conversion, or the operator canno
t be called
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(130): warning C4996: 'v8::String::NewFromUtf8': was declared deprecated [C:\Users\Pietro\projects\GameServices\FitX
R.AdminPortal\node_modules\paseto.js\build\extcrypto_addon.vcxproj]
c:\users\pietro.node-gyp\12.6.0\include\node\v8.h(2890): note: see declaration of 'v8::String::NewFromUtf8'
c:\users\pietro\projects\x\node_modules\paseto.js\extcrypto\extcrypto.cc(147): warning C4996: 'v8::String::NewFromUtf8': was declared deprecated [C:\Users\Pietro\projects\GameServices\FitX
R.AdminPortal\node_modules\paseto.js\build\extcrypto_addon.vcxproj]
c:\users\pietro.node-gyp\12.6.0\include\node\v8.h(2890): note: see declaration of 'v8::String::NewFromUtf8'
gyp ERR! build error
gyp ERR! stack Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:203:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Windows_NT 10.0.17763
gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Users\Pietro\projects\x\node_modules\paseto.js
gyp ERR! node -v v12.6.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN rollback Rolling back<EMAIL_ADDRESS>failed (this is probably harmless): EPERM: operation not permitted, scandir 'C:\Users\Pietro\projects\x\node_modules\fsevents\node_modules'
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.
npm WARN<EMAIL_ADDRESS>No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!      C:\Users\Pietro\AppData\Roaming\npm-cache_logs\2019-07-11T15_18_06_113Z-debug.log
`

Managed to install using Node 10.
I was previously using Node 12

Thanks, I'll take a look at the behavior on Node 12.

I can confirm that paseto.js cannot be build on Node 12.
I can propose two different fixes :
https://github.com/sloonz/paseto.js/tree/nan : Use https://github.com/nodejs/nan/ for extcrypto. Adds a dependency, but compatible with all nodes versions.
https://github.com/sloonz/paseto.js/tree/remove-ext-crypto : Get rid of extcrypto and do everyting with node crypto module. Unfortunately required APIs are only there from Node 11.

@sloonz - I have a bit of downtime in the coming weeks, and am trying to catch up on everything with this repo I've unfortunately been unable to deal with for a while. I'd happily merge in the nan approach upstream here if you'd like to open a PR. I'd also be very willing to throw you a commit bit on this repo, as you seem to have developed your fork substantially.

Confirming it doesn't install on latest node LTS (12.14.0 as of time of writing) running on macOS Catalina (10.15.2).
error /Users/nfantone/Developer/flybondi/lambda-api-key-token-authorizer/node_modules/paseto.js: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments:
Directory: /Users/nfantone/Developer/flybondi/lambda-api-key-token-authorizer/node_modules/paseto.js
Output:
gyp info it worked if it ends with ok
gyp info using<EMAIL_ADDRESS>gyp info using<EMAIL_ADDRESS>| darwin | x64
gyp info find Python using Python version 2.7.16 found at "/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python"

# ...

5 warnings and 3 errors generated.
make: *** [Release/obj.target/extcrypto_addon/extcrypto/extcrypto.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/Cellar/node@12/12.14.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:210:5)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Darwin 19.2.0
gyp ERR! command<EMAIL_ADDRESS><EMAIL_ADDRESS>"rebuild"
gyp ERR! cwd /project/node_modules/paseto.js

Yep, I'll get Node 12 sorted by the end of the week.

Thanks @sjudson we appreciate that.

Closed by #26

Many thanks @sjudson!


Fix indentation issue in getting-started-k8s guide [skip ci]

Fix indentation issue in getting-started-k8s guide.[skip ci]

build


iPhone platform fixes {resubmit}

Hi all, here's the new pull request that doesn't have unecessary commits in it. I've spent time messing with the rebase functionality in order to get it right.
There is a fix in this pull request to get Godot compiling under Msys2 as it was complaining about having too long of an argument list during linkage.

Can we use #ifdefs to detect the SDK version and add or remove those extern
"C" that are causing trouble? We use SDK 8.something on the build server, I
think those are too old for this code. I don't mind supporting the latest
SDKs, but also I'd like to not have to update the build server for a while.
On 15 November 2015 at 21:17, Aren Villanueva<EMAIL_ADDRESS>wrote:

Hi all, here's the new pull request that doesn't have unecessary commits
in it. I've spent time messing with the rebase functionality in order to
get it right.
There is a fix in this pull request to get Godot compiling under Msys2 as
it was complaining about having too long of an argument list during linkage.
You can view, comment on, or merge this pull request online at:
https://github.com/godotengine/godot/pull/2795
Commit Summary

Compile fixes for mingw msys2.
Optimised the oc_filter_hedge and oc_filter_vedge routines in theora
a little.
iOS compile fixes.
Properly fixes compilation in an msys environment.

File Changes

M drivers/SCsub
https://github.com/godotengine/godot/pull/2795/files#diff-0 (44)
M drivers/gles2/rasterizer_gles2.h
https://github.com/godotengine/godot/pull/2795/files#diff-1 (4)
M drivers/gles2/shader_compiler_gles2.h
https://github.com/godotengine/godot/pull/2795/files#diff-2 (2)
M drivers/theora/decode.c
https://github.com/godotengine/godot/pull/2795/files#diff-3 (46)
M platform/iphone/app_delegate.mm
https://github.com/godotengine/godot/pull/2795/files#diff-4 (2)
M platform/iphone/game_center.mm
https://github.com/godotengine/godot/pull/2795/files#diff-5 (2)
M platform/iphone/gl_view.mm
https://github.com/godotengine/godot/pull/2795/files#diff-6 (10)
M platform/iphone/godot_iphone.cpp
https://github.com/godotengine/godot/pull/2795/files#diff-7 (2)
M platform/iphone/icloud.mm
https://github.com/godotengine/godot/pull/2795/files#diff-8 (2)
M platform/iphone/sem_iphone.cpp
https://github.com/godotengine/godot/pull/2795/files#diff-9 (5)
M platform/iphone/view_controller.mm
https://github.com/godotengine/godot/pull/2795/files#diff-10 (3)

Patch Links:

https://github.com/godotengine/godot/pull/2795.patch
https://github.com/godotengine/godot/pull/2795.diff

—
Reply to this email directly or view it on GitHub
https://github.com/godotengine/godot/pull/2795.

This should do the trick.

Can you put tabs on SCsub instead of spaces? :)
On 15 November 2015 at 22:37, Aren Villanueva<EMAIL_ADDRESS>wrote:

This should do the trick.
—
Reply to this email directly or view it on GitHub
https://github.com/godotengine/godot/pull/2795#issuecomment-156887192.

Details details details! ;)

looks good.. what's the theora stuff for?
On 15 November 2015 at 23:04, Aren Villanueva<EMAIL_ADDRESS>wrote:

Details details details! ;)
—
Reply to this email directly or view it on GitHub
https://github.com/godotengine/godot/pull/2795#issuecomment-156889243.

It's a little speedup (5-10%) when playing theora videos. Definitely more noticeable on less powerful hardware like the ipad mini I'm testing on.
Considering that unless I port/write the arm/neon optimised code it doesn't increase the play-back speed enough to be useful, I'm not going to be working on that part of the code any more.

ok, looks good, thanks!
On 15 November 2015 at 23:37, Aren Villanueva<EMAIL_ADDRESS>wrote:

It's a little speedup (5-10%) when playing theora videos. Definitely more
noticeable on less powerful hardware like the ipad mini I'm testing on.
Considering that unless I port/write the arm/neon optimised code it
doesn't increase the play-back speed enough to be useful, I'm not going to
be working on that part of the code any more.
—
Reply to this email directly or view it on GitHub
https://github.com/godotengine/godot/pull/2795#issuecomment-156891974.


api order

    it 'path order' do
      subject.get('/:id') do
        'id'
      end

      subject.get('/foo') do
        'foo'
      end

      get '/foo'
      expect(last_response.body).to eql 'foo'
    end

 Failure/Error: expect(last_response.body).to eql 'foo'

       expected: "foo"
            got: "id"

This is a bug or not?Or I should always write as placing /:id at last.

Not bug. First definition is asked first. ':/id is like a regular expression. 'foo' satisfies it.

ok, thanks. :)


Fixed extern $.merge type

Corrected the type signature of $.merge to indicate that its arguments are mandatory and its return value is never null/undefined

Since there is an internal change in review to address this, I'll be closing this PR.

No problem, I wasn't sure which source was considered "canonical" (that is, if changes here get ported to internal, or vice versa)

PRs get imported to internal, where they are tested and reviewed.

Does the reverse happen, where internal changes are pushed / PR'd to public?

When an internal change gets submitted, it gets pushed to github during the daily sync.
So the workflow for a normal external PR is as follows:

PR gets imported internally, referencing this PR, and is tested and reviewed.
Internal change of this PR gets submitted
During daily sync to Github, internal change gets committed externally, and PR is closed.

That's very helpful, thank you!
On Thu, May 9, 2019, 12:48 PM EatingW<EMAIL_ADDRESS>wrote:

When an internal change gets submitted, it gets pushed to github during
the daily sync.
So the workflow for a normal external PR is as follows:

PR gets imported internally, referencing this PR, and is tested and
reviewed.
Internal change of this PR gets submitted
During daily sync to Github, internal change gets committed
externally, and PR is closed.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/closure-compiler/pull/3366#issuecomment-491041000,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAN5FLM5N3CIXMJMLUISDSDPUR52HANCNFSM4HLT64CA
.


feat!: syntax does not work

Please have a look at this repo: https://github.com/baumrock/conventionalcommits/commits/main

Am I doing anything wrong?
Any help would be greatly appreciated! Thx in advance

This is because feat! is not considered breaking change in the angular preset I think, only in the conventional commit preset

Thx @TriPSs for the quick reply! So how can I make it use the syntax listed in https://www.conventionalcommits.org/en/v1.0.0/ ? I thought I'm already using that...

I'm sorry but I'm lost @TriPSs
I understand that I need to change the preset, but the question is which value do I have to use? I've searched the repo for "preset" but I could not find any helpful information and also try and error with using conventionalcommits did not work.
Also where can I find more information about other preset is installed on your system?

You can find the different presets here and the value to use is described inside the README of the corresponding preset.

Thx @TriPSs that's a pity. I'll stick with the angular preset then as it's easier to use a slightly different syntax than having to npm install external dependencies on every project where I'm using it.
Thx for your help!

@BernhardBaumrock FYI: Since v5 we now compile presets inside the action and since v5.1.0 the conventionalcommits is also added.

hey @TriPSs thx for the heads up! Awesome! Just tested and this works for me:
name: Releases
on:
  push:
    branches:
      - main

jobs:
  changelog:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3
      - name: conventional Changelog Action
        id: changelog
        uses<EMAIL_ADDRESS>        with:
          preset: "conventionalcommits"
          github-token: ${{ secrets.github_token }}

      - name: create release
        uses: actions/create-release@v1
        if: ${{ steps.changelog.outputs.skipped == 'false' }}
        env:
          GITHUB_TOKEN: ${{ secrets.github_token }}
        with:
          tag_name: ${{ steps.changelog.outputs.tag }}
          release_name: ${{ steps.changelog.outputs.tag }}
          body: ${{ steps.changelog.outputs.clean_changelog }}