ARROW-6315: [Java] Make change to ensure flatbuffer reads are aligned

Implements the IPC message format alignment changes for ARROW-6315.
i. MessageReader can read messages with the old alignment
ii. ArrowWriter could choose produces messages with the new alignment or the old format.

@ursabot build

@tianchen92 it looks like this breaks the ArrowTools test

@tianchen92 it looks like this breaks the ArrowTools test

It's really strange, I believe it works fine with maven build on my local machine, is there any difference with travis?

That is interesting, your change passes on my box as well.  Let me try rerunning CI.

@ursabot build

Thanks for the pr @tianchen92 . I'll have to look at it more closely again, but I think when you deserialize you check for the continuation that is not there because it has already been read.

@BryanCutler Thanks a lot for your feedback, I left some comments above and updated this PR.
Besides, this PR always test fail in travis but works fine in local machine, the error log seems useless, do you have any idea how to find the cause? @emkornfield

org.apache.arrow.tools.EchoServerTest
[ERROR] basicTest(org.apache.arrow.tools.EchoServerTest)  Time elapsed: 0.023 s  <<< ERROR!
java.net.SocketException: Connection reset
at org.apache.arrow.tools.EchoServerTest.testEchoServer(EchoServerTest.java:115)
at org.apache.arrow.tools.EchoServerTest.basicTest(EchoServerTest.java:147)

Looks like I can reproduce the error locally, I'll look into it.

Looks like I can reproduce the error locally, I'll look into it.

Awesome, thanks a lot for your help!

@BryanCutler do you have a sense if as part of this change the old/new writing/reading code paths in Java need to be controllable by an environment variable for use with Spark?  I think we need something like that for Python?

I just found the root cause of the error and can post a patch soon.

@BryanCutler do you have a sense if as part of this change the old/new writing/reading code paths in Java need to be controllable by an environment variable for use with Spark? I think we need something like that for Python?

Maybe, I'm not entirely sure. The case that I think might fail is when a user has pyarrow 0.15.0 and using Spark <= 2.4.4 with Arrow Java 0.14.1. It should work going Java -> Python, but then going Python -> Java, it will write the 8-byte prefix by default and Java will choke. So an env var would help in that case I think.

@tianchen92 #5288 is merged, please rebase when you can and hopefully this will be good to go

@tianchen92 #5288 is merged, please rebase when you can and hopefully this will be good to go

Thanks Bryan, I have rebased and let's see if it could pass :)

Could the reader just remember the state, so if it read the continuation
token then read 8 bytes at the end?
On Fri, Sep 6, 2019, 5:43 AM Eric Erhardt<EMAIL_ADDRESS>wrote:

@eerhardt commented on this pull request.
In java/tools/src/main/java/org/apache/arrow/tools/EchoServer.java
https://github.com/apache/arrow/pull/5229#discussion_r321715682:

@@ -132,7 +132,8 @@ public void run() throws IOException {
}
}
writer.end();

     Preconditions.checkState(reader.bytesRead() == writer.bytesWritten());

     // reader does not read padding bytes.

It looks like the C++ implementation only reads the first 4 bytes:
https://github.com/apache/arrow/blob/ARROW-6313-flatbuffer-alignment/cpp/src/arrow/ipc/message.cc#L327-L345
How else are we supposed to know if we should read 4 or 8 bytes? Maybe for
the new format, an 8-byte EOS token should look like {0xFFFFFFFF,
0x00000000}, so we read the continutation token first, and then know to
read the next 4 bytes, which are then 0 to signal EOS.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow/pull/5229?email_source=notifications&email_token=ABCTA5OJ3HFAYGBBW534Q23QIJF6TA5CNFSM4ISCC3P2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCD5EV6Y#discussion_r321715682,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABCTA5PRFC767UAAMWFTZG3QIJF6TANCNFSM4ISCC3PQ
.

Looks like the previous test failure is passing now. Integration tests have lots of failures with JS and Go, but Java <-> C++ look fine. Are there known issues with JS/Go integration, or maybe they don't have the protocol changes yet?
Also, I think from the discussion here https://github.com/apache/arrow/pull/5229#discussion_r321403799 this is going to be a problem. We could fix this as a followup, since it seems C++ is doing the same thing? What do you guys think?

JS hasn't been checked in.  Go has been checked so we might need to investigate if that is failing.

It seems like all the errors on the integration test are due to JS.

I'm going to merge, lets follow-up on the terminal 8 bytes on the mailing list.

@emkornfield did you merge this yet? The PR still looks open, but I see you made a commit

Closing to avoid confusinon.


New "Run Selection" method breaks on switch statement

Issue Type: Bug
I was working on my script last Friday and the switch statement below was working just fine when using "Run Selection" in VS Code.  The PowerShell below also runs fine when the script is executed in a regular powershell window.  There must have been an upgrade to VS Code over the weekend and now the "Run Selection" seems to be behaving differently and the switch statement below throws an error.
"At line:1 char:28

Switch($addressPrefix){

                       ~

Missing condition in switch statement clause.
+ CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : MissingSwitchConditionExpression
"
Sample.
Switch($addressPrefix){

    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}
    <IP_ADDRESS>/16{$subnetPrefix = "<IP_ADDRESS>/24";$gwSubnetAddressPrefix = "<IP_ADDRESS>/24"}

}

VS Code version: Code 1.43.1 (fe22a9645b44368865c0ba92e2fb881ff1afce94, 2020-03-18T07:01:20.184Z)
OS version: Windows_NT x64 10.0.17763

Please start by filing this issue against the powershell extension

Thanks, I've filed this against the PowerShell Extension.


gradle error

When I downloaded from this site, the following error occurred when compiling the project。
Error:A problem occurred configuring project ':app'.

Could not resolve all dependencies for configuration ':app:_debugApkCopy'.
Could not find com.android.support.constraint:constraint-layout:1.0.2.
Searched in the following locations:
https://jcenter.bintray.com/com/android/support/constraint/constraint-layout/1.0.2/constraint-layout-1.0.2.pom
https://jcenter.bintray.com/com/android/support/constraint/constraint-layout/1.0.2/constraint-layout-1.0.2.jar
Required by:
project :app

Thank you for your solution.


Now works on pre-eip-4895 blocks

This EIP added withdrawals which added the withdrawal field to the zeroTracer payload. The decoder always expected this field to be present, but now just uses an empty withdrawal vec in the case that this is missing.

Makes sense. Lets hold off for now.

@BGluth I don't remember the motivation behind this PR. What should we do about it?

Closing as per offline discussion.


Game crashes when I enter different scaled portals

So I made a portal  that should make the player bigger, but when I enter the portal, Minecraft crashes and gives me this error
The game crashed whilst exception in world tick
Error: java.lang.NoSuchFieldError: BASE
Exit Code: -1
I have:
Pehkui 3.0.0
Fabric API 0.43.1
Immersive Portals 0.33
WorldEdit 7.2.7
Fabric for 1.17.1

Use Pehkui 2.2.1 for now

https://github.com/Virtuoel/Pehkui/issues/218


cmd/go: flags to control changes to go.mod, go.sum

Proposed changes
I propose we add three new flags to go subcommands that deal with modules.

-modfile=go.mod - instead of reading and writing go.mod from the current directory or a parent directory, the go command would read and write the specified file. The original go.mod would still determine the module root directory, so this file could be in any directory (perhaps /tmp). It would be an error to use this flag when no actual go.mod file is present.
-sumfile=go.sum - instead of reading and writing go.sum from the directory containing go.mod, the go command would read and write the specified file. It would be an error to use this flag when no original go.mod file is present.
-g - "global mode" - the go command would behave as if no go.mod file were present.

These flags would be allowed when GO111MODULE is set to on or auto (the current default) and rejected when GO111MODULE is set to off. -modfile and -sumfile both require an actual go.mod to be present, so modules must be enabled when they're used. -g does not require an actual go.mod to be present, and in auto mode, it implies that modules are enabled.
Background
The go command updates go.mod and go.sum after any command that needs to find a module for a package not provided by any module currently in the build list. This ensures reproducibility: if you run the same command twice, it should build (or list or test) the same packages at the same versions, even if new versions have been published since the first invocation.
For example, if you run go build ., and a .go file in the current directory imports example.com/m/pkg which is not provided by any known module, the go command will add a requirement on the latest version of the module example.com/m to go.mod. Future runs of go build . will produce the same result.
While these updates are usually helpful, there are many situations where they're not desirable.
Selected issues
gopls (various issues)
gopls loads information about source files in a workspace using golang.org/x/tools/go/packages, which invokes go list. gopls may also run go list directly. In either case, gopls may trigger changes to go.mod and go.sum. This may be caused by user actions that seem unrelated to building anything, for example, opening a file. go.mod appears to change mysteriously on its own, and users don't realize gopls is triggering it.
It's not usually important that the information gopls loads is reproducible; files it operates on are frequently changing. However, it is important that when it resolves an unknown import path to a module, it doesn't need to do so repeatedly since this can add a lot of latency, especially on slow connections.
gopls could set -modfile and -sumfile to temporary copies of the original go.mod and go.sum. The original go.mod and go.sum would not be modified (until the user explicitly runs a command like go build). Resolved module requirements would stay in the temporary files so they would not need to be resolved again.
#25922 - clarify best practice for tool dependencies
Developers need a way to express module requirements that aren't implied by package imports. This is especially useful for tools invoked by go generate. Authors can add tool requirements to go.mod manually or with go get, but these requirements are erased by go mod tidy.
The current recommendation is to create a tools.go file, tag it with // +build tools, then import main packages of needed tools. tools.go will never be built because of the tag, but go mod tidy will read the imports and preserve the requirements. This feels like a hacky workaround rather than a best practice. It also pushes requirements which may not otherwise be needed on downstream modules.
A better solution would be to keep a separate go.tools.mod file with tool requirements, then point to that with -modfile=go.tools.mod when running commands that require tools.
#26640 - allow go.mod.local to contain replace/exclude lines
This is a feature request to keep some go.mod statements out of source control. It's frequently useful for module authors to check out dependencies and point to them with replace statements for local development and debugging. These statements shouldn't necessarily be exposed to users or other developers on the same project though.
Setting -modfile=go.local.mod and -sumfile=go.local.sum would solve this problem, at least partially. The two files could be copied from the regular go.mod and go.sum files and added to .gitignore. Note however, that these local files are used instead of the regular files, not in addition to, so some synchronization might be required.
#30515 - offer a consistent "global install" command
Developers want to be able to install tools from any directory, regardless of the requirements of the current module. go get tool@version may update the current go.mod, so users need to change to a temporary directory without a go.mod file to run commands like this. Tool authors need to be careful when writing installation instructions because of this.
The -g flag would address this issue. It would tell the go command to run as if it were outside any module. Tool authors could write go get -g tool@latest in their installation instructions: this would install the latest version of the tool, regardless of the current directory.
Note that "missing go.mod" is being reconsidered (#32027), so the actual semantics of -g may change: this issue is just about ignoring the current module.
#33710 - module mode removes concept of global docs
In module mode, go doc example.com/pkg prints documentation for the packages named on the command line at the same version they would be built with. Like go build, go doc may add or update requirements in go.mod. This may be undesirable, especially if you're using the documentation to decide whether you want to depend on a package that is not currently imported.
The -g flag would partially solve this. The current module would be ignored, and "global" documentation would be shown.
Note that go doc does not currently work in "missing go.mod" for packages outside std. #33710 would need to be fixed, but -g would provide a useful way to access that mode.
Other related issues
There are a large number of open issues about unexpected and unwanted go.mod changes. The flags suggested here won't solve all these problems, but they provide useful context.

#26977 - go mod why adds a go.mod line
#29452 - go list has too many (more than zero) side effects
#29869 - 'go list' should not resolve or record modules that are not relevant to the requested output fields
#31372 - 'mod verify' does not respect -mod=readonly
#31999 - x/tools/gopls: support go.mod files
#32027 - rethink "missing go.mod" mode
#33326 - go install: don’t fail when go.mod can’t be updated on a read-only system
#34450 - go mod download support -readonly

cc @bcmills @rsc @stamblerre @ianthehat

-g - "global mode" - the go command would behave as if no go.mod file were present.

Per https://github.com/golang/go/issues/32027#issuecomment-533273609, we'll probably want to make the go command less willing to resolve dependencies when no go.mod file is present.
If we do that, presumably the -g flag will allow the go command to resolve the transitive dependencies of packages listed on the command line. Should -g also allow the go command to resolve missing packages found in import statements?

The original go.mod would still determine the module root directory,

but

These flags would be accepted when GO111MODULE is set to on and rejected when GO111MODULE is set to off.

If GO111MODULE is on but there is no go.mod file above the current working directory, what would we use as the module root?

Per #32027 (comment), we'll probably want to make the go command less willing to resolve dependencies when no go.mod file is present.

If we do that, presumably the -g flag will allow the go command to resolve the transitive dependencies of the modules containing the packages listed on the command line. Should -g also allow the go command to resolve missing dependencies found in the import statements of those packages?

I think -g should just make the go command do whatever it would do without a go.mod file.

If GO111MODULE is on but there is no go.mod file above the current working directory, what would we use as the module root?

I contradicted myself a bit here. Updated the text. -modfile and -sumfile require an actual go.mod to be present in order to set the module root directory.

Regarding the file flags: I think having to specify both -modfile and -sumfile is cumbersome. I can't think of any compelling reason to want to share go.sum with any other module, since go mod tidy will throw away the unnecessary lines anyway. So I would suggest that at a minimum, -sumfile be derived from -modfile if it's not set. A more extreme option would be to specify a single prefix, say -modprefix, and add .mod and .sum to it as needed. That may be too strange, though.
This is also a good precedent to set in case there's ever a third file, since nobody will know to override its location before it exists.

@jayconrod regarding the "gopls (various issues)" motivation. I'm not entirely clear on what the issue here is and hence why -g/-modfile/other is a solution.
Is the problem that people are not used to cmd/go having side effects in module mode?
Because for me, I would expect that anything I do inside my editor, e.g. adding an import for a package whose module is not in my go.mod, could have side effects on the main module.
Could you give a bit more background?

#30515 - offer a consistent "global install" command

While not part of the original issue, the consensus for a while seemed to be that a "global" install should also obey replace directives. At least, this is what @ianthehat said in https://github.com/golang/go/issues/30515#issuecomment-469544933. Has the team's position generally changed to not obeying replace directives at all?
I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go <args>.

Possibly worth reiterating that replace directives come in two forms:

directory-based
non-directory-based

I think the consensus that @mvdan refers to above was on the latter being applied.

Sorry, I should have clarified. There were initially some comments about applying non-directory replace directives, but the consensus seemed that we either apply all of them or none of them - to not add a third build mode. @ianthehat's comment that I linked seemed to lean towards applying all replace directives.

Personally, I'm in favour of applying directory replacements only when the source is in a user-controlled directory (as opposed to in global mode where the source is in the module cache). But I feel strongly that installs in global mode should respect other replacements, FWIW.

@mvdan, @rogpeppe: see https://github.com/golang/go/issues/30515#issuecomment-469742759 and https://github.com/golang/go/issues/30515#issuecomment-470224733.
Furthermore, given that the proposed semantics of the -g flag are to do whatever we would do if outside of a module, the question of whether or how to apply replace directives seems orthogonal (and a bit off-topic). #31173 is probably a more appropriate venue for that discussion.

It would be an error to use this flag when no actual go.mod file is present.

We should probably make that even stronger: the module directive in the replacement go.mod file should specify the same module path as the actual go.mod file. (Otherwise, if we're building packages within the module, we will end up resolving what should be module-local imports by looking for an external module.)

regarding the "gopls (various issues)" motivation. I'm not entirely clear on what the issue here is and hence why -g/-modfile/other is a solution.

Is the problem that people are not used to cmd/go having side effects in module mode?

Because for me, I would expect that anything I do inside my editor, e.g. adding an import for a package whose module is not in my go.mod, could have side effects on the main module.

@myitcv, @ianthehat  and @stamblerre have told me there's no canonical issue for this, but they've had to close a lot of issues as "working as intended", pointing to #29452, and they explain this frequently on Slack.
They mentioned one egregious example where someone in a clean workspace switched branches, then tried to switch back but were unable to because go.mod had uncommitted changes. Their editor (and gopls) was open in the background. It had detected changes in open files, run go list indirectly, and modified go.mod as a result.
-modfile would have made these changes to a temporary go.mod file instead of the go.mod in the main repo. #31999 is about supporting go.mod files in gopls, and while there aren't any details there yet, part of the plan is to provide easy ways to make changes and upgrades. It would be difficult to do that without -modfile.

Regarding the file flags: I think having to specify both -modfile and -sumfile is cumbersome. I can't think of any compelling reason to want to share go.sum with any other module, since go mod tidy will throw away the unnecessary lines anyway. So I would suggest that at a minimum, -sumfile be derived from -modfile if it's not set. A more extreme option would be to specify a single prefix, say -modprefix, and add .mod and .sum to it as needed. That may be too strange, though.

This is also a good precedent to set in case there's ever a third file, since nobody will know to override its location before it exists.

I like the simplicity of just saying what both files should be, but I couldn't actually come up with a scenario where you'd want to set -modfile without -sumfile.
How about this: there's no -sumfile flag. If -modfile is set to M, then the sum file is strings.TrimPrefix(M, ".mod") + ".sum".

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go .

I don't think that solves the same set of problems. -modfile would still use the location of actual go.mod file to set the module root directory, not its argument. -modfile just redirects reads and writes, providing a way to control changes.

While not part of the original issue, the consensus for a while seemed to be that a "global" install should also obey replace directives. At least, this is what @ianthehat said in #30515 (comment). Has the team's position generally changed to not obeying replace directives at all?

After thinking through all the weird edge cases and trying out a bunch of things, I came to the conclusion that applying replace directives is a confusing and dangerous ball of scary, and the only sane thing to do is not to apply them, in fact the only sane thing to do is to never ever check replace directives in to your repository in the first place.
Part of the benefit of this proposal is it allows for a workflow that makes that a much easier goal, by allowing an alternate go.mod that has the replace directives in it but is not used by default.

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go <args>.

That would not allow most of the useful patterns (having multiple .mod files in the same directory that you can choose between, or a cache directory with modified .mod files for a bunch of different modules etc)
It would also mean that you have to specify what happens in a bunch of interesting edge cases (thinks like relative paths, are they from the original module or the alternate root?)
I think that specifying the .sum file would/should be incredibly rare, leaving it as the original one would be fine for the majority of use cases.

@mvdan @rogpeppe
On replacements:

We should apply all of them or none of them. A third mode would cause confusion. (https://github.com/golang/go/issues/30515#issuecomment-469742759).
In order to apply file path replacements, we'd need to check out a whole repo. That's very different from what go get does now (especially when a proxy is in use), and there are many ways it could fail. We also won't get reproducible builds if replacement paths point outside the repository.

Personally, I think the downsides of (2) outweigh the benefits, and it's better to have something very simple like -g.

How about this: there's no -sumfile flag. If -modfile is set to M, then the sum file is strings.TrimPrefix(M, ".mod") + ".sum".

Maybe split the difference? If the original is go.mod and -modfile is M.mod, first check whether M.sum exists: if so, use (and update) it, and otherwise use (and update) the original go.sum.
As far as I can tell, that handles both the tools.go.mod case and the /tmp/some-gopath-dir/go.{mod,sum} case.

Maybe split the difference? If the original is go.mod and -modfile is M.mod, first check whether M.sum exists: if so, use (and update) it, and otherwise use (and update) the original go.sum.

As far as I can tell, that handles both the tools.go.mod case and the /tmp/some-gopath-dir/go.{mod,sum} case.

@bcmils Could work, but it seems a little subtle. If a module only depends on std and has no go.sum file, a naïve tool might copy go.mod to /tmp without creating an empty go.sum file. It would then unexpectedly modify the temp go.mod and the original go.sum.

@jayconrod re https://github.com/golang/go/issues/34506#issuecomment-535040055

-modfile would have made these changes to a temporary go.mod file instead of the go.mod in the main repo

But what about the situations where you do want gopls to have these side effects? i.e. adding an import for a package whose module is not in my go.mod.

I think -g should just make the go command do whatever it would do without a go.mod file.

Without a go.mod file, go get example.com/some/really/old/tool (that is, some tool without its own go.mod file) should probably fail, rather than re-resolving the latest transitive imports of that tool and discarding the result.
On the other hand, I think it is probably reasonable to expect go get -g example.com/some/really/old/tool to succeed, even if it is consistently slow.

But what about the situations where you do want gopls to have these side effects? i.e. adding an import for a package whose module is not in my go.mod.

@myitcv It would be up to gopls and the editor to provide a sensible way to do that. Perhaps it could show a warning that go.mod is incomplete and provide a quick fix to add the missing requirement. Or it could update go.mod when a file is saved if it adds new imports.
There are situations where modifying go.mod is not wanted, and -modfile provides gopls with a knob that controls when those modifications happen.

Without a go.mod file, go get example.com/some/really/old/tool (that is, some tool without its own go.mod file) should probably fail, rather than re-resolving the latest transitive imports of that tool and discarding the result.

On the other hand, I think it is probably reasonable to expect go get -g example.com/some/really/old/tool to succeed, even if it is consistently slow.

@bcmills There's a lot of nuance here. Assuming no go.mod is present, should go get example.com/tool when example.com/tool does have a go.mod file? What if the go.mod file is missing some requirements?
Should -g have this variation in behavior for go get only or for other commands, too?

Assuming no go.mod is present, should go get example.com/tool when example.com/tool does have a go.mod file? What if the go.mod file is missing some requirements?

I'm not sure. Probably:

Without -g and outside of a module:
go get should resolve the latest version of the requested packages, compute the resulting build list, try building the requested packages, and error out if any of the import statements in encounters cannot be resolved using that build list.
That way, the number of latest lookups (and, potentially, fetches) is O(N) with the arguments passed to go get, and on successive runs everything else can be resolved using the local module cache.

Without -g and inside of a module:
go get should resolve the latest version of the requested packages, add them to the main module's build list, and resolve and add any additional dependencies as needed during the build, recording the result in the main module's go.mod file.
That still keeps the number of latest lookups down to O(N) with the arguments in the steady state, since any subsequent go get will use the versions recorded in the go.mod file (which will be in the local cache).

With -g and inside of a module:
The user has explicitly told us not to record the result of version resolution.
If the latest version of the module(s) containing the requested packages does not specify all needed dependencies, we have two possible options:

We could reject the go get request and ask the user to retry within a module. Most likely they'll just set -modfile=$(mktemp) instead of -g and run the command again, in which case we've added no value.

We could resolve the full import graph and throw away the results. But I think in most cases the user won't be running go get -g on the same module repeatedly, so that's probably fine.

With -g and outside of a module:
It's the same situation as inside of a module, but instead of -modfile=$(mktemp), they'll likely do pushd $(mktemp -d); go mod init ugh; go get […]; rm -r .; popd.

Should -g have this variation in behavior for go get only or for other commands, too?

Do we need to support the -g flag at all for other commands? I was assuming it would be a get-specific flag.

It would be up to gopls and the editor to provide a sensible way to do that. Perhaps it could show a warning that go.mod is incomplete and provide a quick fix to add the missing requirement. Or it could update go.mod when a file is saved if it adds new imports.

This creates too much of a disconnect with the other go commands to my mind. By way of analogy, this would be equivalent to go test failing because go.mod is incomplete, and then requiring the user to run some specific command to fix it before again running go test.
Hence @bcmills's comment makes sense to me:

Do we need to support the -g flag at all for other commands? I was assuming it would be a get-specific flag.

@jayconrod will your proposed change still be compatible with the advice given at:
https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
(I'm guessing yes, but just want to make sure).
Some projects have migrated to that approach such as this:
https://github.com/atlassian/smith/blob/master/tools.go
But there are other projects which still do go get in a Makefile inside the module directory to fetch tools.

The proposal has no effect on that approach, but...
 I would argue strongly that it was okay as a temporary measure in go1.11 but it is already bad advice today and would become much worse advice when the features in this proposal are available.

That approach is a horrible hack with all sorts of horrendous caveats and problems.

[citation needed]
#33926 is the only issue I'm aware of describing a concrete problem with the // +build tools approach, and the concrete problem cited there (https://github.com/golang/go/issues/33926#issuecomment-527946737, an incompatibility between gopls and one of its dependencies) was due to a breaking change made in one of the tool's non-semantically-versioned dependencies.

Some notes from a few out-of-band conversations with @bcmills @ianthehat and @myitcv:
-modfile

In evaluating this feature request, it would be helpful to have a better understanding of how gopls would use it. Clearly, there are situations where it should and should not modify go.mod. Some extra logic will be required to manage and synchronize both the actual and temporary go.mod files.

@ianthehat @stamblerre Could you comment here with ideas on when this feature would be used and how that would work in gopls? What would the user experience look like, ideally? Is any of this a v1.0.0 blocker?

It seems like most of the time, we want a separate file that is used in addition to the actual go.mod, not instead of. gopls needs to manage additional requirements. For local replacements, we still want the original set of requirements. For tool dependencies, we want additional requirements.

Having -modfile always augment the actual go.mod seems like it loses flexibility though. What if you actually want something different?
An idea: we could add a statement (say include) that pulls in statements from another go.mod file.
A statement like this would only work in the main go.mod file though. Like replace and exclude, it would be ignored in remote modules. It would add a lot of complication to the proxy protocol and sumdb logic to support include in general.
Maybe it could be done as part of the flag? -modfile=+go.local.mod or -modfile=go.local.mod,go.mod would both mean "read both go.mod and go.local.mod and write changes to go.local.mod".

-g

Note that -modfile and -g are separate features that solve different issues, though they are both related to go.mod modifications. I probably should have filed two issues for these though.
It's hard to evaluate -g because we haven't decided what "missing go.mod" mode should do (#32027).

At the moment, @bcmills and I are thinking that it should be possible to build packages in std and ad-hoc packages comprising .go files listed on the command line that only import packages in std. Any build, run, list, test, doc command that requires resolving an import path to a module would fail outside of a module (I'll post a comment on that issue).
If we go with that solution, there's no point in those commands supporting -g, since they won't usually be able to do anything useful.
So -g may as well be a special flag for go get that is tailored to fix #30515.

We haven't been able to agree on a solution for #30515 though. The question is what to do with replace directives.

replace directives should be used for 1) temporarily forking a module to depend on a fix or feature that hasn't been released upstream, and 2) local development, especially of changes that affect multiple modules. In the context of #30515, only (1) is important.
Technically, we could apply replace directives that replace modules with other modules. However, we cannot safely apply replace directives that replace modules with directories, since paths may point outside the module or outside the current repository. See https://github.com/golang/go/issues/30515#issuecomment-469531958 and the comments below. We don't want to introduce a "third mode" where some replace directives are applied but not others, since this adds complication for module authors (increased test surface) and in the go command (lots of opportunity for bugs).
An idea: go get -g could apply module replace directives in the module providing packages named on the command line. It would reject file replace directives with an error. So replace directives would never be partially applied.
gorelease could warn module authors if they are about to publish a version that includes file replace directives in the main go.mod. File replace directives could be kept in a separate go.local.mod, referenced with -modfile.
This seems like a lot of pieces to fit together, so perhaps initially for 1.14: go get -g rejects all replace directives with an error. We could enable module replace directives later on without breaking anyone.

@myitcv

But what about the situations where you do want gopls to have these side effects? i.e. adding an import for a package whose module is not in my go.mod.

@jayconrod

Some notes from a few out-of-band conversations with @bcmills @ianthehat and @myitcv:
-modfile
* In evaluating this feature request, it would be helpful to have a better understanding of how gopls would use it. Clearly, there are situations where it should and should not modify go.mod. Some extra logic will be required to manage and synchronize both the actual and temporary go.mod files.

  * @ianthehat @stamblerre Could you comment here with ideas on when this feature would be used and how that would work in gopls? What would the user experience look like, ideally? Is any of this a v1.0.0 blocker?

I don't think it's obvious gopls should ever modify my go.mod and go.sum. To me, gopls is an extension of my editor, and my editor should not change any files I didn't ask it to. To me, it's tedious to constantly have to keep an eye out for irrelevant go.mod/go.sum changes before checking in new code.
Of course, if I run go commands from inside my editor (either directly or indirectly through a "run", "build" or similar editor command), then my go.mod/go.sum files are subject to change. But in those cases I'm no longer editing, I'm using the editor as a convenient command line shell.
So in conclusion, I see gopls always using -modfile pointing to its own go.mod file as the right decision.

@eliasnaur so what about when go test modifies your go.{mod,sum} - is that also an error from your perspective? Because in that instance you haven't specifically asked it to modify your go.{mod,sum}, you've simply asked to run some tests. go mod edit is the specific command for editing go.mod. If go test did not operate in this way, having to run an additional command (which would require some manual effort) to add the relevant require directives would be tiresome to my mind because that's almost always what I want to do.
Per @jayconrod's request to @ianthehat and @stamblerre above, I think we need to establish what the workflow here would be from a user's perspective before doing something that to my mind goes against what the go tool is already doing.

Another use case for this is go-fuzz. go-fuzz has to augment the source tree with its own dependencies during compilation, but there is ~zero value to the user in having the module containing those dependencies be present in their go.mod/go.sum. (cc @thepudds)

To offer some input from a heavy gopls user: I'm very happy to let gopls modify my go.mod file as it pleases, since that means I can paste in import paths and have them actually show up, even if I have to do cleanup duty later if go.mod contains something I didn't expect.
If it didn't have this ability, they'd break, and then more things end up breaking when I have to go run go in my terminal and gopls doesn't see that something has changed (leading to some bad state only a reload can fix, which is already far too common). I can see a future where gopls wouldn't have the ability to modify go.mod, but it'd really need to better handle watching for changes, and would be kinda disappointing to go from the workflow of "paste in an import and it just works" back to the old workflow of needing to make sure that everything is there before trying to use it.

@eliasnaur so what about when go test modifies your go.{mod,sum} - is that also an error from your perspective?

No. I meant to cover go test by:

Of course, if I run go commands from inside my editor (either directly or indirectly through a "run", "build" or similar editor command), then my go.mod/go.sum files are subject to change. But in those cases I'm no longer editing, I'm using the editor as a convenient command line shell.

In other words, go commands modify my go.* files, as they should. I'm only talking about gopls and editing in general, which shouldn't.

To offer some input from a heavy gopls user: I'm very happy to let gopls modify my go.mod file as it pleases, since that means I can paste in import paths and have them actually resolve, even if I have to do cleanup duty later if go.mod contains something I didn't expect.

I agree, and that's why I haven't complained about gopls before this proposal. With -modfile, gopls can keep a private go.mod file and operate exactly as if it had modified your go.mod file.

In other words, go commands modify my go.* files, as they should

My point was that for some people go test and other build commands (e.g. go list, go build etc) should not have these side effects: https://github.com/golang/go/issues/29452. i.e. an explicit go mod edit commands should be required before running go test etc.
I'm not arguing for that position, just using it to point out that many people have normalised the fact that cmd/go build commands modify go.{mod,sum} in just the same way that I've normalised gopls modifying go.{mod,sum}.

With -modfile, gopls can keep a private go.mod file and operate exactly as if it had modified your go.mod file.

What would the workflow look like when I do want gopls to modify my go.{mod,sum}?
Having a second go.{mod,sum} introduces a second source of truth; the "real" one is used by cmd/go, the "fake" is used by gopls. How do we ensure they remain in sync? How do we point out conflicts?

In other words, go commands modify my go.* files, as they should

My point was that for some people go test and other build commands (e.g. go list, go build etc) should not have these side effects: #29452. i.e. an explicit go mod edit commands should be required before running go test etc.
I'm not arguing for that position, just using it to point out that many people have normalised the fact that cmd/go build commands modify go.{mod,sum} in just the same way that I've normalised gopls modifying go.{mod,sum}.

 So even with a hypothetical fix to #29452, your go test would still do what you expect.

With -modfile, gopls can keep a private go.mod file and operate exactly as if it had modified your go.mod file.

What would the workflow look like when I do want gopls to modify my go.{mod,sum}?

May I ask when and why you want gopls to modify you go.* files? And for each of the cases you do want its modifications, why aren't your inevitable go test, go run, go install or even go mod tidy good enough checkpoinst for actually changing your go.* files?
The only legitimate case I can see is for adding imports that you're going to keep, but how would gopls know the keepers from your spelling mistakes? I can't count the number of times gopls has added irrelevant dependencies to my go.* files just because I code completed or misspelled a package.

Having a second go.{mod,sum} introduces a second source of truth; the "real" one is used by cmd/go, the "fake" is used by gopls. How do we ensure they remain in sync? How do we point out conflicts?

My go.* files are the source of truth, while gopls' own files are for caching queries and pleasing its underlying go commands. If conflicts occur, its conflicting (or all?) changes to its internal go.* files should be wiped out.

 So even with a hypothetical fix to #29452, your go test would still do what you expect.

It's worth noting that go list is a build command; it takes build flags. go mod edit is I think the only go mod subcommand that is literally concerned with mechanically editing go.mod. All the other subcommands will cause a full resolution of dependencies that might result in changes to go.{mod,sum}.
The reason I bring up the go test example is because in another world we could easily have decided that go test should fail if go.{mod,sum} didn't contain the relevant entries for go test to proceed. In such a scenario, a go mod edit or similar mechanical update would have been required as a separate step before running go test again. As it is, go test does as little as it needs to to go.{mod,sum} for the build configuration under test. This feels natural to me, because it avoids the painful error message "you need to run X before continuing".

May I ask when and why you want gopls to modify you go.* files?

Because I want the type checking and analysis that gopls carries out to proceed in much the same way that go test does without me needing to intervene. Similarly, when gopls gains the power to help with things like code generation or other commands, I don't want to be having to manually intervene to run go mod edit to fix my go.mod or click "accept" on a code suggestion to add something to my go.mod.

The only legitimate case I can see is for adding imports that you're going to keep, but how would gopls know the keepers from your spelling mistakes?

I notice you skilfully sidestepped my question on the workflow of when I do want gopls to update my go.{mod,sum} 😉 I'm guessing however you want to manually "copy" the "right" changes from gopls's copy to the real copy?

I can't count the number of times gopls has added irrelevant dependencies to my go.* files just because I code completed or misspelled a package.

This could happen in any scenario: even with go test for example. go mod tidy is your friend here, and I think Rebecca added support for a formatting of go.mod files (which amounts to a go mod tidy from memory). I don't think manually copying/accepting the "right" changes is necessary here, the tools can do the work for us.

If conflicts occur, its conflicting (or all?) changes to its internal go.* files should be wiped out.

I'm certainly not clear what this conflict resolution algorithm would look like: can you sketch it out ?

 So even with a hypothetical fix to #29452, your go test would still do what you expect.

It's worth noting that go list is a build command; it takes build flags. go mod edit is I think the only go mod subcommand that is literally concerned with mechanically editing go.mod. All the other subcommands will cause a full resolution of dependencies that might result in changes to go.{mod,sum}.

I suppose that's what #29452 is about: go list is a build command, but feels like a query command.
However, in the context of gopls its use of go list is an implementation detail.

May I ask when and why you want gopls to modify you go.* files?

Because I want the type checking and analysis that gopls carries out to proceed in much the same way that go test does without me needing to intervene. Similarly, when gopls gains the power to help with things like code generation or other commands, I don't want to be having to manually intervene to run go mod edit to fix my go.mod or click "accept" on a code suggestion to add something to my go.mod.

That's a great example. I want to consent to go.mod changes because adding a dependency is not something I do lightly. From Rus Cox' Our Software Dependency Problem:

"A package, for this discussion, is code you download from the internet. Adding a package as a dependency outsources the work of developing that code—designing, writing, testing, debugging, and maintaining—to someone else on the internet, someone you often don’t know. By using that code, you are exposing your own program to all the failures and flaws in the dependency. Your program’s execution now literally depends on code downloaded from this stranger on the internet. Presented this way, it sounds incredibly unsafe. Why would anyone do this?"

In other words, completing or auto-generating code for me is ok to do more or less automatically, but or adding or changing my dependencies is not.

I notice you skilfully sidestepped my question on the workflow of when I do want gopls to update my go.{mod,sum} wink I'm guessing however you want to manually "copy" the "right" changes from gopls's copy to the real copy?

Whenever you consent to the changes. Whether that is a go build, go test on the command line, or clicking "run" or "accept" in a IDE. And perhaps there is a setting for "I accept all changes" if you really want to accept whatever gopls gives you during edits.

I can't count the number of times gopls has added irrelevant dependencies to my go.* files just because I code completed or misspelled a package.

This could happen in any scenario: even with go test for example. go mod tidy is your friend here, and I think Rebecca added support for a formatting of go.mod files (which amounts to a go mod tidy from memory). I don't think manually copying/accepting the "right" changes is necessary here, the tools can do the work for us.

Yes, but I consider go test and friends consenting to changes. It's not perfect (#29452) and I would personally have GOFLAGS=-mod=readonly if govim supported it.

If conflicts occur, its conflicting (or all?) changes to its internal go.* files should be wiped out.

I'm certainly not clear what this conflict resolution algorithm would look like: can you sketch it out ?

Keep an internal copy of go.* and use -modfile to point to them. Whenever the original go.* files changes, replace the copy.

I'm curious as to whether #34829 would satisfy the gopls use-case less invasively. How important is it to resolve dependencies in uncommitted code, vs. tolerating (and possibly suppressing) errors for unresolved dependencies?

Contemporaneous notes from a conversation with @ianthehat and @stamblerre:

When you add an import, either explicitly or by completion, gopls should update go.mod or offer to do so. The exact behavior needs evaluation.
If changes to go.mod don't happen automatically, they'd probably be suggested Quick Fixes on imports that aren't provided by any module in the build list.
If changes to go.mod happen automatically, they should happen when the file is saved, probably not as soon as the import is added (right after a completion).
A frequent problem: a user opens a file that transitively imports a package not provided by any module in the build list. gopls (via go list) adds a requirement for a new module. The user hasn't edited anything yet and is surprised that go.mod has changed.
How far could we get if go list -e -mod=readonly worked better (#34829)?

It doesn't let us type check, since we can't load packages that transitively depend on missing packages.
It doesn't let us suggest fixes (updates to go.mod). Everything is just broken.
-modfile would let us add the module requirement to a separate go.mod, then suggest a change with the difference. We could type check everything.

How should -modfile be passed to go list?

golang.org/x/tools/go/packages.Config.BuildFlags.
Drivers are expected to know all the flags for go list, including -modfile. Drivers for older versions of the Go command and other build systems should filter this flag out or find something equivalent.

@eliasnaur

go list is a build command, but feels like a query command.

I can sympathise with that point of view: it's certainly not obvious.

However, in the context of gopls its use of go list is an implementation detail.

This is true but it's a critical implementation detail. Whilst go/packages is the abstraction, go list is the very means by which the concept of modules, packages etc are defined for cmd/go-based build systems.
(That's not to say however that we're bound only by what cmd/go offers today, indeed that's what we're discussing here.)

I would personally have GOFLAGS=-mod=readonly if govim supported it.

I think this is a great suggestion for an option in govim. Indeed I just did a bit of experimentation and it already works if GOFLAGS=-mod=readonly is passed to the environment of gopls by govim: you get no modifications to go.{mod,sum} as a result of using Vim/govim, and changes made via cmd/go are correctly picked up by gopls by virtue of govim simulating file watch changes (whilst we wait for full gopls file watching support). We could add this today with a bit of a hack on the govim side, pending support for a workspace config option in gopls at a later date. I've created https://github.com/govim/govim/issues/555 to track this for govim.

Whenever you consent to the changes. Whether that is a go build, go test on the command line, or clicking "run" or "accept" in a IDE.

I'm not clear how go build or go test is any more "safe" than things happening via gopls/go/packages/go list - they will end up having exactly the same side effects (if we consider them being run at the same point in time) and you're still blind to them unless you check your go.{mod,sum}.

Keep an internal copy of go.* and use -modfile to point to them. Whenever the original go.* files changes, replace the copy.

As you mention above, you'd either look to "accept" the changes by:

explicitly running go build, go test and friends
prompting the user

For option 1, this would mean that there is a possibility the subsequent go build/go test will result in a different changes to the go.{mod,sum} than those made in the -modfile copy. Because none of these commands (with the exception of go get) specifies a version of a dependency. This doesn't seem ideal.
For option 2, presumably you'd prompt for every change that gets made to the -modfile go.mod to "sync" it back to the original if the user accepts? If so, this feels like a very noisy workflow to me, especially when I don't get any such prompts when using go test etc.
Per @jayconrod's request to Rebecca and Ian (https://github.com/golang/go/issues/34506#issuecomment-536024494) and subsequent follow up in https://github.com/golang/go/issues/34506#issuecomment-541165801, I think it's worth looking at the scenarios when a go.{mod,sum} change can occur:

incomplete go.{mod,sum} - i.e. go mod tidy not run
new import added in main module and code action run to run goimports equivalent (this happens on save in govim if so configured)
... others?

And also the use cases that motivated this discussion in the first place:

UC1 - people changing Git branches (as I understand it, this generally causes all sorts of problems)
UC2 - people just browsing projects, i.e. with no intention of making changes
UC3 - people who do not want to have gopls touch go.{mod,sum}
... others?

UC3 is covered above and and in https://github.com/govim/govim/issues/555.
UC1 requires a significantly broader solution to my understanding, because changing files like this "underneath" the editor has all sorts of problems.
UC2 is totally valid to my mind, and building on the suggestion discussed between Jay, Rebecca and Ian, it might make sense (assuming the user is not interested in UC3) to not make any changes to go.{mod,sum} until the first user-initiated change is made to a file in the main module. This however creates a weird UX, because all type check errors that might show would then disappear with the addition, say, of a space to a comment. Alternatively, if it's detected the go.{mod,sum} is not tidy, then the user could be prompted to fix this as the workspace is opened.

Just to note, I'm very much in support of fixing workflow issues like this. My reservations are around a lack of clarity on the UI/UX, use cases that we're looking to fix, whether we will end up actually fixing those use cases, and potential issues with conflicts if we were to use -modfile.

scenarios when a go.{mod,sum} change can occur

As I mentioned above, this happens with go-fuzz. We inject a dependency and execute a build, which then changes go.mod in a way that the user doesn't care about at all.

@josharian - sorry, I totally missed that. My comments were more focussed on use cases from a gopls/editor perspective. Although I'll admit I don't really know whether there is overlap or not with go-fuzz in that respect.

On Fri Oct 11, 2019 at 1:52 PM Paul Jolly wrote:

Keep an internal copy of go.* and use -modfile to point to them. Whenever the original go.* files changes, replace the copy.

For option 2, presumably you'd prompt for every change that gets made to the -modfile go.mod to "sync" it back to the original if the user accepts? If so, this feels like a very noisy workflow to me, especially when I don't get any such prompts when using go test etc.

Do you mean that accepting every change is noisy? I read @jayconrod's comment

-modfile would let us add the module requirement to a separate go.mod, then
suggest a change with the difference. We could type check everything.

as there would be a summary of all the changes which I could accept in bulk
if/when I pleased. If I waited to accept changes, or (more likely) never
accepted them, gopls would hopefully keep working.

Here's another use case for this flag.
 At the moment that I need that Go program to run, the local go.mod file is in a bad state. (I'm running the Go program as part of a script to make the go.mod file functional.) As a result, I can't use go run. If I could do go run -modfile=/sometempfile, then I could temporarily get past the broken go.mod and run the file.
(Yes, there are other ways to accomplish this, like copying the Go program into a temp dir, manufacturing a trivial go.mod for it, building it, and running it. But this flag provides a nice, easy simple workaround.)

A quick update:
I've implemented the -modfile flag in CL 202564. There will be no -sumfile flag. Instead, the name of the go.sum file will be derived from the name of the go.mod file by trimming the .mod suffix (if present) and add .sum, as @heschik suggested.
I'm planning to add a -g flag that would apply to go get. It will cause go get to behave as if it were outside a module. After changes made in #32027, it doesn't really make sense for -g to apply to other commands.
This interpretation of -g means there would be no main module, and replace and exclude directives will be ignored. We've discussed the tradeoffs of doing it this way here and in #30515. I'm starting to think that we should support both approaches. Perhaps it should be possible to pass a -g=main flag. This would restrict command line arguments to one main package (or perhaps packages from one "main" module) and would apply replace and exclude directives from that module. File replace directives would be rejected. WDYT?

@jayconrod just so I'm clear, what decision on replace/exclude directives was taken in https://go-review.googlesource.com/c/go/+/203279?

@myitcv In https://golang.org/cl/203279, -g will mean "act as if there is no main module", and it will apply to go get only.
I alluded to having a -g=main flag in my earlier comment, which would have the semantics you wanted: command line arguments could only come from one module, and that module would be treated as the main module for the purpose of replace and exclude directives.
I think it makes sense to have two separate modes. It doesn't seem like everyone will be happy with either one. However, I don't think we'll be able to get -g=main done before the freeze (5 working days from now). The implementation will be more complicated than -g, some design work is needed, and we're thinking about some changes to replace for 1.15 which might affect this (proposal not ready yet).

However, I don't think we'll be able to get -g=main done before the freeze

From the user's perspective, -g=main is what they'll want most of the time when doing "global installs" of programs. So I think adding -g now, without any replace directives, could be a bad decision in the long-term; the behaviour would be pretty much set in stone.
If we really want "act as if there is no main module" today, how about a simpler -modfile=none?

From the user's perspective, -g=main is what they'll want most of the time when doing "global installs" of programs.

I respectfully disagree. From a user's perspective, what they'll want most of the time is a binary that builds in exactly the same configuration no matter where you build it from — that is, one that does not rely on checked-in replace or exclude directives at all.
When that condition holds, -g and the possible -g=main are equivalent.
That being the case, -g sets the right default by encouraging tool authors to get the changes they rely on merged upstream, or to explicitly fork packages for which they require divergent behavior.
Moreover, -g as drafted is much simpler to reason about: it means “pretend that I am not in a module” — nothing more, and nothing less.

From the user's perspective, -g=main is what they'll want most of the time when doing "global installs" of programs.

@mvdan I'm not convinced this is true long-term. As module adoption increases and go command bugs are fixed, I hope we'll see fewer tools that rely on replace directives as fewer problems need to be worked around. replace directives should rarely be necessary in tagged releases.
-g=main will also cause problems for problems for projects that rely on replace directives with file paths. We cannot support those in downloaded modules. So for example, go get -g=main golang.org/x/tools/gopls would break today because there's a file replace directive for golang.org/x/tools.

If we really want "act as if there is no main module" today, how about a simpler -modfile=none?

So -modfile=none instead of -g? That could work, but -g seems like a simpler command line to me. Based on how we've resolved #32027, -modfile=none would only be meaningful for go get, so I'm not convinced -g should have that spelling.

I think -modfile=none would be a mistake.
-modfile means “find the go.mod file at the root of my module and replace it with this other file.”
-modfile=none would therefore mean “find the go.mod file at the root of my module and replace it with none”. But what does it mean to replace a go.mod file with an empty one? What does that do to the module path and the semantics of the current directory?
-g, in contrast, means “pretend that I am not in a module”, which has a well-defined (if evolving) behavior.

In particular, you could imagine combining -g and -modfile to mean “ignore the module containing the current directory, resolve the versions I'm passing you on the command line, and record your decision in this go.mod file.”.

Somewhat tangential, but I don't think -g=main is a good idea for a flag. Like normal flags flags, the flag should either be boolean or not boolean. Otherwise -g main won't work as expected. (Discussed recently in the context of go test -shuffle over at https://github.com/golang/go/issues/28592#issuecomment-538059551.)

I agree with @mvdan that -g=main is always the functionality I want when installing global tools.  If the upstream author needed to use replace and exclude directives to build a tool and tagged it in a release I can't imagine why anyone would then want to go get -g binary@version and have it ignore those directives.
Yes authors should get changes merged upstream in time for their release to avoid the replace when possible, but why punish the consumers of that binary? I realize their are technical challenges around this (failing on filesystem based replaces seems fine to me) but I really don't understand the use case for the current implementation of the -g flag. I'd much rather the current go get -g fail to build a binary if the version its getting has any replaces or excludes rather then to ignore them. Then I'm not silently using the wrong version of the tool. If it fails, fine then at least I know I can git clone to build it.

Directory-based (file path in the parlance above) replace directives are, as I have previously agreed, a different beast to non-directory replace directives. In a different world they could have been named dirreplace or only allowable in a go.mod.local. Therefore I agree with @jayconrod's position that a directory-based replace directive in a released go.mod could even be considered an error. It means nothing to the end user.
So working on the assumption they are an error, we won't seem them in a released tool's go.mod (this would need to be documented of course).
To my mind therefore applying non-directory replace directives does not introduce significant dissonance. Nor do I think that in the case we're discussing (i.e. -g, or whatever name we give it, as a way of a user installing a "global" tool) it's inconsistent with the cmd/go documentation:

Exclude and replace apply only in the main module's go.mod and are ignored in dependencies.

Because we're not talking about a dependency: it's effectively the main module and could be documented as such.
Nor is applying non-directory replace directives an acceptance that a change will not be upstreamed ever. Indeed the very existence of a replace directive is a very strong indicator that's exactly what the tool author is in the process of doing.
I totally accept that it's simpler to understand replace/exclude directives not being applied, but I don't think that leaves us in a better net position. Here's another perspective: not applying replace  directives forces tool authors towards hard forking. This makes it harder to upstream the fix which might  have exactly the opposite intended effect, i.e. people upstream less.
A replace directive (as overloaded/badly named as it may be) signifies a tool author's intent to try and carry on whilst they simultaneously try and upstream a change. Is replace being abused here? I don't think so - it's still being used for the "temporary" solution.
This being the "reality" in which I think we will find most tools, I therefore agree with @mvdan's comment. Furthermore that having -g and -g=main just creates more confusion from the tool author's/end user's perspective

Replying to a comment from @myitcv on CL 203279.

Just a thought regarding the discussion on -g, -g=main, -modfile, -modcacherw etc.
I haven't exactly remained close to all of these discussions but it seems we're adding lots of new flags here, decisions which are permanent.
In the case of -g and -g=main I'm not clear we've arrived at the "right" answer.
With -modfile, I can see use cases, but we haven't experimented with these in gopls for example before making the change. It might be the experiment fails and we no longer need the flag.
With -modcacherw I'm even further from the discussion/change so I defer on that.
Hence I wonder whether documenting these flags as experimental (or dropping the flags altogether and instead having GOEXPERIMENT_XYZ env vars) would be a more prudent next step?

I'd prefer not to have experimental flags in the Go command as part of a regular release. Despite being labelled experimental, these features have a way of becoming difficult to change or remove simply by being used, discussed, and documented.
If we're moderately confident in the design, I think it's reasonable for these flags to be in the 1.14 beta. If it turns out they have problems, we can remove them before the release.
If we're not confident in the design, we can defer these until a future release. However, we've been discussing variations on -g on and off since March (#30515), and I'd love for us to ship something in 1.14 that most people are happy with.

About -modfile
This seems like the less controversial of the two changes discussed here, so I won't say too much more about it. It's submitted last week as CL 202564. It's important that we verify this solves the problems set out to solve, primarily preventing gopls from unexpectedly editing go.mod files.
@stamblerre, @ianthehat Would you be willing and able to prototype functionality related to -modfile in gopls? It would be great to have some feedback on this flag before the 1.14 release candidate is cut in early January. I don't expect everything can be done by then, but it would be good to know if the basic flow of copying the go.mod file to a temporary directory and monitoring changes there works as intended.
Anyone else interested in using this for tools or managing non-production dependencies, please try it out at tip. Let me know if you run into problems, either here or in a new issue.

About -g
This is the flag that's more useful to everyone on a day-to-day basis, but it's also the one we're having trouble agreeing on. I'll try and recap.
-g sets out to solve two problems. First, we need a way to install binaries globally using go get without touching go.mod in the current directory. Second, we need a way to install binaries while respecting replace and exclude directives in authors' go.mod files. -g as proposed here was meant to solve the first problem, leaving the second until later, but after discussion, it seems like we should agree on a solution to both before proceeding. Note that since we've resolved #32027, -g will be specific to go get, so it's fine if it does something that's not meaningful to go list for example.
replace directives are really the hard part. We can't apply replace directives with file paths on the right side, since the go command operates on module zip files, not repositories, and zips do not include replacement files. We don't want to apply some replace directives but not others, since that adds complication to the go command and to module authors' release processes. So it should be all or nothing: for any go get command, either all replace directives are applied, or all are ignored, or the command fails with an error because some replace directive could not be applied. I think we're all in agreement up to this point.
Currently, go get does not apply replace directives when outside of a module. If we start applying replace directives (rejecting those that we cannot apply), some commands that could be built before will no longer be buildable. So this behavior needs to be opt-in with a new flag. The question is whether that flag is -g. That is, should we try to solve both problems together with one flag?
This is really a question of user intent and module author intent. It's subjective so it's hard to know what the "right" choice is.
Several people in this thread and on #30515 have argued convincingly that replace directives will continue to be an important part of people's workflows, and installing binaries with replace directives applies is the common case we should optimize the CLI for.
I'm starting to agree with this point of view. Short-term forks of upstream modules are not going to disappear. My main concern about this was that the behavior was very different from go get outside a module without -g, but perhaps that's desirable. We can advise module authors not to include file path replace directives in go.mod files at released version; perhaps these can be managed with -modfile or simply removed at tagged commits.
There are a lot of smaller use cases that this version -g would not address though: explicitly ignoring replacements; using newer versions of indirect dependencies; installing multiple binaries from multiple modules. Are these common enough cases to need a separate flag? All of them can be worked around by running go get without -g outside a module (as in 1.13) or by using -modfile.

Thanks for the recap @jayconrod. To clarify, with my earlier comment I didn't mean to say that I have a fully fledged plan for -g. Just that I think adding it now without figuring out what we want to do with replace directives would be a mistake in the long run. Users are going to be confused by having both -g and -g=main, and most will default to just using -g.
I agree that having something in 1.14 would be good though, which is why I was trying to think of less invasive changes like -modfile=none. Or perhaps a standalone "ignore the current module" flag would still be useful in 1.14, but I think it's a mistake to spend the -g name on it.
We have a tools call later today, so I think it's a good opportunity to have a quick chat about it then.

I would like to push back on something this subtle being -g. That takes a one-letter flag for something that we can't even explain.
If we do -modfile=none instead, that I can explain: it is exactly equivalent to having an empty go.mod file that is discarded after the command runs. The only gotcha with this is that the command will be slow, because it can't cache any of its decisions in the go.mod file. This is how GO111MODULE=on used to work without a go.mod file, and it led to even quite astute users incorrectly concluding that "modules are slow". It should be documented that -modfile=none is highly non-recommended if it goes in.
All the discussion about replace directives seems difficult to reconcile to me. What if I do go get -modfile=none A B and A and B both have different go.mod with conflicting replaces? It would be much better not to do replaces at all. That's the rule for dependencies in any event.

If we do -modfile=none instead, that I can explain: it is exactly equivalent to having an empty go.mod file that is discarded after the command runs.

-modfile=none may be pretty subtle, too. It may also be surprising because -modfile works in all build commands. For example, running go build -modfile=none ./foo within a module would build the package in directory ./foo, resolving imports to latest versions of modules, even if those imports refer to packages in the same module.

What if I do go get -modfile=none A B and A and B both have different go.mod with conflicting replaces?

This example seems clearer to me. It would have the same behavior as go get A B today when run outside of a module. It would resolve the latest versions of A and B, then run MVS to find a combined build list, then build packages from that. replace would be ignored.
Only -g would enable replace directives, and -g can't be used together with -modfile. It would require that packages listed on the command line come from only one module, and replace directives would only be applied from that module.

I still believe -g is a mistake. It is overfitting to a handful of use cases, adding complexity without a clear model behind it.
Go has never been about solving every possible use case.
Go is about providing easy-to-understand composable building blocks so that users can construct what they need the vast majority of the time, without our planning it for them.
-g does not seem to me like it fits into that.

It is overfitting to a handful of use cases

From my perspective, -g restores the main purpose for which I use go get, previously removed as part of the migration to modules.
Installing tools globally is not a niche use case. See previous comments here and here.

I still believe -g is a mistake. It is overfitting to a handful of use cases, adding complexity without a clear model behind it.
Go has never been about solving every possible use case.
Go is about providing easy-to-understand composable building blocks so that users can construct what they need the vast majority of the time, without our planning it for them.
-g does not seem to me like it fits into that.

@rsc I agree that -g is not a composable building block. However, I think this is a common use case, and it may be common enough to optimize (overfit) for.
I'm planning to spend some more time researching use cases to understand whether this makes sense. We should quantify how often replace comes up and whether there are other, better solutions.

From my perspective, -g restores the main purpose for which I use go get, previously removed as part of the migration to modules.

@cespare I think we all agree that installing tools globally should be supported. The question is how to do it in a way makes the most sense for the ecosystem.
In GOPATH mode, replace directives are not available, but every project can have a vendor directory. Multiple vendor directories caused a lot of problems that prevented projects from being composed. We want to avoid composability problems with modules as much as we can.

Based on discussion here and in the tools meeting today (notes and recording available later), I don't think we're confident enough in the design of -g to proceed in 1.14. Let's put it on hold for now and proceed with -modfile only.

There is still disagreement on whether replace directives should be applied.

@myitcv, @mvdan, @rogpeppe, @pbx0, and others argue this will be important for many binaries long-term, and it's the behavior that most users want and expect.
@bcmills and @rsc argue that this adds complexity, prevents modules from composing well, and interferes with a lot of other use cases.
@bcmills and @ianthehat point out that it's difficult for authors to test their modules with -g.

@bcmills is working on a proposal enhancing replace directives for 1.15. If there are additional semantics in the future, it's difficult to decide what to do about replace now.
I'm planning to spend some time analyzing open source go.mod files to understand how often replace directives are needed in repositories with executables and what solutions make the most sense for them.
It's also possible that -modfile may make replace directives less relevant in the future. I hope that developers may put development-only requirements and replace directives in an alternate file (go.dev.mod or go.tools.mod) so those won't need to be checked into the main go.mod file.

As a user new to modules I'm having trouble intuiting what the use case is for go getting a package outside a module other then to install global binaries. Can sometime explain this more?
If i'm installing global binaries then I feel it should be an error to build while ignoring replace directives. If this behavior is inconsistent with current go get rules then it seems like the command shouldn't be a flag of go get.
I know go install is off the table, but now that modules is out I'm also having trouble understanding when I would ever use go install anymore. I find the tooling changes the most confusing aspect of modules and I don't find any of the current behavior something I'm tied to since its all fairly different post-gopath.

@pbx0 go get is also used to upgrade and downgrade dependencies within a module. With the -d flag, go get won't build anything at all.
When go get is used within a module to install an executable, as with other build commands, it respects requirements and replacements of the current module. This is useful if you want to install a tool used within your module, and you want to link against a newer version of some dependency or perhaps a replacement where you've fixed a bug or added a feature. Any changed requirements are written to go.mod.
go install serves a similar use case, but it doesn't have the same dependency management capabilities.
When go get is used to install an executable outside a module, as with other build commands (before #32027 was resolved), it does not respect replacements, since there is no main module. This encourages composability of modules, but it's pretty inflexible. (Some amount of inflexibility can be a good thing; we're happy with static types, after all).
There isn't really a good way to install an executable that won't build without its own module replacements, other than to clone its repository and run go install from there. Technically, it wouldn't be that difficult for the go command to apply these. However, there's a long list of similar things we can't support: file path replacements, checked-in vendor directories (possibly with local modifications), git submodules, generated code that's not checked in, ...

Adding my 2 cents here in hope it helps clarify the need for a -g-parameter specifically: we have a project where we require goversioninfo, which is a tool we need to invoke before go build (through go generate), and is not a dependency of the actual module we are building. We currently run
GO111MODULE=off go get https://github.com/josephspurrier/goversioninfo
to "just get" the tool without affecting go.mod. We believe
go get -g https://github.com/josephspurrier/goversioninfo (or similar)
would look a lot cleaner.

@jayconrod Thanks for the info. I guess the question I still have is what are the use cases for running go build and get outside a module in its current form?
I feel strongly that if the go tool ever supports a "install global binary" command that that command will error when it can't respect a replacement rather then build it anyway. It feels like the right action for a command purported to install global binary and anything else would be very surprising IMO. Is this something others feel similarly about given the purpose is to install a global binary at a certain version?
In some sense I'm advocating for more inflexibility, I don't want multiple subtly different actions to install binary commands outside of a module and I don't understand the other use-cases besides install a global binary which seems like it would be a popular one even if limited projects without replacements, submodules, ect. I feel ambivalent to where the line is drawn in respecting replacements as long as its an error when it can't be done.

I guess the question I still have is what are the use cases for running go build and get outside a module in its current form?

@pbx0 In 1.14 (after #32027), go build outside of a module will only be useful for small test programs that only depend on the standard library. go get will still be useful for installing binaries, as it is today (though replace directives are not honored).
Not much to add about go get -g and replace right not beyond what's already been said here and in #30515.

Re-milestoning this for Go1.14 since -modfile is merged and seems likely to ship. I'll close this issue after we've validated -modfile in gopls.
I'd like to direct further discussion on -g or any other "global install" to #30515.

@jayconrod What's the status of this issue?  Thanks.

@ianlancetaylor -modfile is implemented. I'd like some of the gopls folks to evaluate it and post feedback after the beta ships. I expect to either close this issue or re-milestone for 1.15 before the RC.

@rodrigc It looks like this Makefile is installing then running several tools at the latest versions in GOPATH mode.
I think the module-mode equivalent of this (using -modfile) would be to create a tools.mod file requiring the versions of the tools you want to use. You could build the tools with mkdir -p tools && go build -modfile=tools.mod -o tools/ example.com/tool.... Then run using executables in that tools subdirectory.
That would get you predictable versions of tools without interfering with your main go.mod file. It also wouldn't have side effects on the rest of your system since you aren't installing anything.

This seems strange to me. I will maintain a file with the names and constraints of executables I need, which makes sense. But then, I want to just point the tool at that file when I use the command for "use this file to give me the things I enumerate". Instead, it seems like the recommendation is to pass each tool enumerated to a different go install invocation. I apologize if I'm misunderstanding what is meant by example.com/..., but I don't see how it can capture the need of downloading tools from several different repositories. So it seems like I only get value out of maintaining this file if I also maintain a script that parses out each dependency and invokes the tool with a different command than I would normally use.

@jayconrod Could you summarize how to use it pls? Is the following still hold 100%?

Copy go.mod to go.local.mod. Add go.local.mod to .gitignore (or equivalent for your workspace).
Run go env -w GOFLAGS=-modfile=go.local.mod. This tells the go command to use that file by default.
Any any replace and exclude directives or other local edits.
Before submitting and in CI, make sure to test without the local file: go env -u GOFLAGS or just -modfile=. Probably also go mod tidy.

@suntong That's all still correct. -modfile was added in Go 1.14.
Right now, -modfile is mostly used in tools and editors (gopls), not so much directly by users on the command line. We're in the process of putting together a proposal that will make it easier to work on multiple modules at the same time. Stay tuned.


File opening appears to be bugged

Errors are thrown no matter what file you open, properly formatted or not.

Ultimately the saving/loading system will be made obsolete once we complete #57


iOS: Add possibility to add iOS device information in game projects

New iOS devices pop up regularly, but existing games can't always be updated to the latest libGDX version. This PR makes the device list accessible to the game code. You can add new iOS device information via IOSApplicationConfiguration.addIosDevice().

It's not ideal as we discussed, but the enum is too restrictive for when new devices pop up and we aren't quick enough. Think its the best solution for now.


Add enable logging FAQ

Description
Adds "enable logging" FAQ.
Quick checks:

[x] I have followed the Code Guidelines.
[x] There is no other pull request for the same
update/change.
[ ] I have written unit tests.
[x] I have made sure that the PR is of reasonable size and can be easily reviewed.

@lovromazgon that should be it right?


Useful things to move to gfm:image from dplug

The Font class which makes it very easy to draw text on an image.
The loadImage function (takes a slice of ubyte, return an Image!RGBA).

Font => not feasible else gfm:image would depend on gfm:math again.
loadImage => done with tag 2.3.0+


ROCM build fixes

What does this PR do?
As per title!

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


GraphQL error when updating bank transfer details

I tried to use the new 'add wire transfer details' feature, and saw there was already info in there for Open Source Collective. I made some small edits and clicked "save" but got the error GraphQL error: Parent collective with id 17403 not found.

Same thing when I tried to update the details for Foundation: GraphQL error: Parent collective with id 17403 not found

@xdamman @alanna Is that solved?

Yes it appears to be fixed now.


Running out of memory on a large project

Please make sure your issue is not addressed in the FAQ.
Done.
Please include the following information:

[x] The version of infer from infer --version.
Tip of tree: 52b72f4bbe24bd354c569efe61fefbb7c7173692
[x] Your operating system and version, for example "Debian 9", "MacOS High Sierra", whether you are using Docker, etc.
Debian 9
[x] Which command you ran, for example infer -- make.
The following command inside Chromium checkout:

$ gn args out/infer
# Use the following GN flags:
# is_debug = false
# is_component_build = true
$ ninja -C out/infer -t compdb cxx cc > db.json
$ autoninja -C out/infer/
$ cd out/infer
$ infer capture --compilation-database ../../db.json
$ infer analyze
# or
$ infer analyze --no-quandary

[ ] The full output in a paste, for instance a gist.

It's messed up (probably because of the multi-threaded output), but I can try to extract the crash stacktrace.

[ ] If possible, a minimal example to reproduce your problem (for instance, some code where
infer reports incorrectly, together with the way you run infer to reproduce the incorrect
report).

Run the commands above :)
Description
I'm trying to run analysis on Chromium project (C/C++) and the analysis step crashes with OOM even when I disable the quandary analysis (as suggested in #1033). Worth noting that 180GB of RAM and 180GB of Swap isn't enough.
Questions

Is there any way to optimize memory usage? I'm not familiar with OCaml, but perhaps there is a way to tweak the GC to be more aggressive and/or to invoke it explicitly more often?

Is there any way to get an estimate of how much memory one would need to run the analysis for a particular project?

Hi.  First, the capture phase is not affected by the choice of analysers.  So using --no-quandary has no effect.
Second, one way to reduce the memory usage of capture is to reduce parallelism in infer.  I believe -j XX will work with compilation databases too.  How many CPUs does the host have?

Thanks for the quick response. Sorry I've copied a wrong command, capture finished successfully. It's analyze failing.
Regarding the CPUs, I have 36 physical cores / 72 threads.

Infer will, by default, use 72 processes on your host.  You can still use -j to limit that, eg -j 36 as well as turning off analysers you don't need.

Thanks @ngorogiannis, I've tried -j 24 and it still needs 200+ GB RAM. Also the speed is a big concern, in 36 minutes it analyzed only 72 files out out 40,000+. I guess I'm doing something wrong. Is there a way to run security checks only (i.e. overflows, use-after-free)?

The speed will always tank once you start swapping -- in fact I would suggest turning off swap entirely, because then you will OOM faster anyway, and it's highly unlikely anything will finish if you ever get into the swap file.
I would use even lower values for -j and maybe use --biabduction-only to restrict to memory safety issues, or --bufferoverrun-only (though not sure how stable this analysis is).

@ngorogiannis, thanks again! This way it seems to run really well.

Closing, feel free to re-open if necessary.


Funcunit does not work properly under Phantomjs

Hello, is funcunit still active? This is a great tool, but I have a problem when using it under phantomjs.
Here is the sample code in demo.test.html file

	<meta charset="UTF-8">
	<title>Using QUnit Tests</title>

	<link rel="stylesheet" href="/css/qunit-1.17.1.css">

The demo.test.js file is very simple.
module('Some Demo tests', {
  beforeEach: function() {

  // F.open('/');
   console.log('First Demo BeforeEach Called');
    return true;

  },
  afterEach: function() {
    return;
  }
});

test('A simple demo with FuncUnit', function() {
  F('body').visible('it shows the body');
});

It works fine in Chrome 39 but does not work in Phantomjs 1.9.8.
I'm using the grunt-contrib-qunit to start Phantomjs to test this. It always timed out.
If I add F.open('/') in the beforeEach function it works. Please help me find out what causes this problem? Is there any workaround?

Hey there, did you get any further with this? I think I might be experiencing a similar issue (see here).

No, FuncUnit just looks very sexy. but very hard to debug.


zones in GCP terraform nodepool config is not actually optional

Context
https://github.com/2i2c-org/infrastructure/blob/c365d39140351b2572d7f22ea9e408aa1f1341cd/terraform/gcp/variables.tf#L87
We have the above optional config to set a zone on nodepools, which defaults to null if not provided. However, I don't think this is actually optional, as I ran into a lot of errors leaving in undefined in PR https://github.com/2i2c-org/infrastructure/pull/2758, such as:
google_container_node_pool.notebook["neurohackademy"]: Creating...
╷
│ Error: Cannot determine zone: set in this resource, or set provider-level zone.
│
│   with google_container_node_pool.notebook["neurohackademy"],
│   on cluster.tf line 238, in resource "google_container_node_pool" "notebook":
│  238: resource "google_container_node_pool" "notebook" {

I think this is a bug in our terraform config.
Proposal
No response
Updates and actions
No response

I couldn't even avoid the above error by setting a zone: https://github.com/2i2c-org/infrastructure/pull/2758#issuecomment-1620491297

I'm happy to pair on investigating this or will do so myself at I experienced got other issues with google_container_node_pool resources so it seems like either state/resource mismatch or possibly new required fields that may be related

Fixed in https://github.com/2i2c-org/infrastructure/pull/2758#issuecomment-1622684841


Removed alternative image text.

Thanks so much! This looks great!

I'm sorry, I didn't fully review the pull request and there was a slight error so I reverted it. Your code is good except there needs to be a set of brackets like this: [] after the exclamation point at the beginning of the image markup. Sorry for the mistake! It should be a quick fix though, and then make another pull request.


Dragon eyes render above its head

Describe the bug
Dragon's eyes (?) render above its head
To Reproduce
Steps to reproduce the behavior:

Summon dragon from spawn egg

Expected behavior
non floaty eyes :D
Screenshots

System (please complete the following information):

Occultism Version: occultism-1.19.2-1.80.1.jar
OS: Windows 11
Minecraft Version: 1.19.2
Modpack Link and Version, or list of mods: Enigmatica 9

Additional context
Add any other context about the problem here.

Also happening out of the seat

uhhhhh

fixed!
The eyes were double broken, not rendered in the right place, and with the wrong texture haha


customizing object keys for import.meta.glob

Description
When using glob import I'm usually trying to do some manipulation of the path. This is the pattern which appears a lot of the time
const globbedArray = Object.entries(import.meta.glob('./routes/*.js'))
const routes = globbedArray.map(([name, item]) => {
  path = name.split('/').at(-1).split('.')[0]
  return [path, item]
})

which works great but the compiled output will contain those paths which aren't needed.
Suggested solution
const routes = import.meta.glob('./routes/*.js', {
  array: true,
  key: (path) => path.split('/').at(-1).split('.')[0],
})

where array: true just uses the Object.entries of the existing output, and key transforms the key used in the object (duplicate handling shouldn't be too much of a concern as it is up to the dev to write it appropriately).
looking at the current importMetaGlob.ts plugin it looks possible to do. if this proposal is accepted I am willing to submit a pr
Additional context
Apologies if I had missed any earlier discussion that is similar to this, most of the import glob issues that I see have been resolved by the current implementation
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

I'm not sure if this is possible because import.meta.glob options are extracted and parse statically, so the key function would have to be pure and cannot access anything outside. Given this footgun, I think it's better to avoid adding function-style options.

ah that does seem tricky.. I was even thinking of using a templating engine but either way it would be complicated.
now it feels more like it should be configured on node side (in vite.config.ts), potentially as another plugin (similar to the experimental import glob plugin pre-v3)

Yeah it'd have to be another option from vite.config.js but I'm not really sure if this feature is worth a new option seems we like to reduce the number of options. An alternative would be creating a Vite plugin to postprocess the glob paths after Vite transforms the glob, but that's getting tricky and not quite sure if it's worth the effort (and potential perf hit) 🤔

I think it is best left as standalone virtual file plugin that does something like:
import routes from 'virtual:glob?name=routes'
// or
const routes = await import('virtual:glob?name=routes')

and have the configuration options in the plugin.
but thanks for the input! at least this will serve as a warning to others who want to tread this path 😅


sync event doesn't fired on collection when model was destroyed

It will be good to get sync event on collection after model was removed from collection AND synced with server.

Sorry, but the way that the events are designed, the "remove" event has already been fired after the model was destroyed. Once the model is removed from the collection, the collection no longer receives events from that model...
If you want to get the sync event for a destroy, listen on the model instead.

It's not very handy to listen for sync on all models instead of collection...
Little strange that collection synchronization goes without sync event but it's OK. I'll learn to live with destroy/remove.
Thanks!

If you have to listen to a sync event on the model to know when it's destroyed, how do you know when to stop listening? Isn't this going to create memory leaks?


[atat] Handle URC codes properly

Currently all unsolicited response codes (URCs) are being added to the response queue, which might not be the correct way of handling them, as this will force one to dequeue them before getting access to other responses.
Consider adding a handler system inspired by https://github.com/arduino-libraries/MKRGSM/blob/b13ad329093a35ee33687de90639c4561a9d6353/src/Modem.cpp#L335

Fixed by https://github.com/BlackbirdHQ/atat/commit/168a4eff1fab6f7f226539fa990438b28707e831


Plugin for Lapce

I wonder if someone has already considered creating a LSP plugin for https://lapce.dev (a new GUI code editor in Rust)
See here for existing examples: https://github.com/lapce-community

I haven’t heard of one yet

I've started an attempt: https://github.com/mayel/lapce-elixir-ls

@mayel please send a PR with links added to Readme and docs

👍 Will do once the plugin works 🤞


MOSES node type distribution doesn't sum to one

Hello Clément,
When processing the MOSES dataset with MiDi compute_all_statistics code, I noticed that the node_type argument contains incorrect values. Here is what I get for the train dataset :
[0.72200687 0.1364436  0.10383305 0.01433876 0.01637907 0.00546271 0.00153594, 0.0]
while the hard coded values in Moses dataset are
[0.722338, 0.13661, 0.103549, 0.1421803, 0.163655, , 0.005411, 0.00150, 0.0]
On what split did you compute the marginal distributions ? It seems that some values differ by an order of magnitude, and the distribution doesn't sum to 1.
Best,
Antoine

Hello Antoine, these values were computed over the full dataset. The reason is that if an atom type appears in the test set but not in the training set, it will result in a NLL of +infinity because the probability of generating it will be 0. It is the same for the distribution of the number of nodes.


refactor: Handle garbage token

Handle a garbage token where the header is not parseable.

:tada: This PR is included in version 1.0.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket: