Include generated object files in commit history and published packages

This issue has been discussed before, I can't seem to find where though. The goal would be to have the released go.opentelemetry.io/auto package be a self-contained distribution.
Currently, when a project takes a dependency on go.opentelemetry.io/auto the project will not compile:
package main

import "go.opentelemetry.io/auto"

func main() {
	i := auto.NewInstrumentation()
	_ = i.Close()
}

$ go version
go version go1.23.2 linux/amd64

$ cat go.mod
module testing/autoimprt

go 1.23.2

require go.opentelemetry.io/auto v0.16.0-alpha

require (
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
	github.com/cespare/xxhash/v2 v2.3.0 // indirect
	github.com/cilium/ebpf v0.16.0 // indirect
	github.com/go-logr/logr v1.4.2 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/google/uuid v1.6.0 // indirect
	github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
	github.com/hashicorp/go-version v1.7.0 // indirect
	github.com/klauspost/compress v1.17.11 // indirect
	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
	github.com/pkg/errors v0.9.1 // indirect
	github.com/prometheus/client_golang v1.20.4 // indirect
	github.com/prometheus/client_model v0.6.1 // indirect
	github.com/prometheus/common v0.60.0 // indirect
	github.com/prometheus/procfs v0.15.1 // indirect
	go.opentelemetry.io/auto/sdk v0.1.0-alpha // indirect
	go.opentelemetry.io/contrib/bridges/prometheus v0.56.0 // indirect
	go.opentelemetry.io/contrib/exporters/autoexport v0.56.0 // indirect
	go.opentelemetry.io/otel v1.31.0 // indirect
	go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.7.0 // indirect
	go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.7.0 // indirect
	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.31.0 // indirect
	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.31.0 // indirect
	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.31.0 // indirect
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.31.0 // indirect
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.31.0 // indirect
	go.opentelemetry.io/otel/exporters/prometheus v0.53.0 // indirect
	go.opentelemetry.io/otel/exporters/stdout/stdoutlog v0.7.0 // indirect
	go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.31.0 // indirect
	go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.31.0 // indirect
	go.opentelemetry.io/otel/log v0.7.0 // indirect
	go.opentelemetry.io/otel/metric v1.31.0 // indirect
	go.opentelemetry.io/otel/sdk v1.31.0 // indirect
	go.opentelemetry.io/otel/sdk/log v0.7.0 // indirect
	go.opentelemetry.io/otel/sdk/metric v1.31.0 // indirect
	go.opentelemetry.io/otel/trace v1.31.0 // indirect
	go.opentelemetry.io/proto/otlp v1.3.1 // indirect
	golang.org/x/arch v0.11.0 // indirect
	golang.org/x/exp v0.0.0-20230224173230-c95f2b4c22f2 // indirect
	golang.org/x/net v0.30.0 // indirect
	golang.org/x/sys v0.26.0 // indirect
	golang.org/x/text v0.19.0 // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20241007155032-5fefd90f89a9 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20241007155032-5fefd90f89a9 // indirect
	google.golang.org/grpc v1.67.1 // indirect
	google.golang.org/protobuf v1.35.1 // indirect
)

$ go build
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/database/sql/bpf_x86_bpfel.go:165:12: pattern bpf_x86_bpfel.o: no matching files found
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/github.com/segmentio/kafka-go/consumer/bpf_x86_bpfel.go:172:12: pattern bpf_x86_bpfel.o: no matching files found
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/github.com/segmentio/kafka-go/producer/bpf_x86_bpfel.go:167:12: pattern bpf_x86_bpfel.o: no matching files found
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/go.opentelemetry.io/auto/sdk/bpf_x86_bpfel.go:159:12: pattern bpf_x86_bpfel.o: no matching files found
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/go.opentelemetry.io/otel/traceglobal/bpf_x86_bpfel.go:213:12: pattern bpf_x86_bpfel.o: no matching files found
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/google.golang.org/grpc/client/bpf_x86_bpfel.go:170:12: pattern bpf_x86_bpfel.o: no matching files found
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/google.golang.org/grpc/server/bpf_x86_bpfel.go:169:12: pattern bpf_x86_bpfel.o: no matching files found
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/net/http/client/bpf_no_tp_x86_bpfel.go:182:12: pattern bpf_no_tp_x86_bpfel.o: no matching files found
../../../pkg/mod/go.opentelemetry.io/auto@v0.16.0-alpha/internal/pkg/instrumentation/bpf/net/http/server/bpf_x86_bpfel.go:174:12: pattern bpf_x86_bpfel.o: no matching files found

We should ship a Go module/package that is self-contained. It should not require any user to generate code within their package cache.
cc @damemi @RonFed
Proposals
Check in object files to git

Pros:

Code, in development and in release, should always be in a buildable and importable state.
Doesn't require any change to release process

Cons:

Bloats the git history size
Adds non-human readable data to git history
Adds non-reviewable code to PRs (reliance on CI to check)

Run and manage our own Go module proxy
https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1233#issuecomment-2448653025

Pros:

Release code will be importable and directly usable

Cons:

Requires significant additional release processes
Requires OTel to manage a Go proxy server
Adds complication to our vanity URL service (may require development work there)
Adds security attack vectors

Commit the object files into release branches
https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1233#issuecomment-2450723817

Pros:

Released code should be importable and directly usable (verification needed)

Cons:

Might bloat git history size if branches need to remain in git history
Requires additional release processes
Adds non-reviewable code to release PRs (reliance on CI to check)

Post-process Go code to contain the object file
https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1233#issuecomment-2450726853

Pros:

Released code will be importable and directly usable
Does not require additional release processes

Cons:

Extra tooling needed to manage post-processing
Is still adding non-human readable code to repository, it is just hiding it is a human readable file
Adds non-reviewable code to release PRs (reliance on CI to check)

Document dependency generation steps
https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1233#issuecomment-2451197662

Pros:

Minimal changes to what we do now

Cons:

Inconvenient and unexpected to downstream consumers

Use Git LFS
https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1233#issuecomment-2452719367

Pros:

Solves problems with repo bloat
less weird than running make generate commands to build object files

Cons:

Not standard git usage (devs need to install git lfs)

https://github.com/open-telemetry/opentelemetry-go-instrumentation/pull/695
I’m not really in favor of this without a really strong reason due to the
complications that binary files create with Git. I don’t think that is
worth the convenience over (for example) a generator that can be run via
Make, so I see it more as a tooling issue than a library issue.
On Wed, Oct 30, 2024 at 5:03 PM Tyler Yahn @.***> wrote:

This issue has been discussed before, I can't seem to find where though.
The goal would be to have the released go.opentelemetry.io/auto package
be a self-contained distribution.
Currently, when a project takes a dependency on go.opentelemetry.io/auto
the project will not compile:
package main
import "go.opentelemetry.io/auto"
func main() {
i := auto.NewInstrumentation()
_ = i.Close()
}
$ go version
go version go1.23.2 linux/amd64
$ cat go.mod
module testing/autoimprt
go 1.23.2
require go.opentelemetry.io/auto v0.16.0-alpha
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cilium/ebpf v0.16.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
github.com/hashicorp/go-version v1.7.0 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.20.4 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.60.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
go.opentelemetry.io/auto/sdk v0.1.0-alpha // indirect
go.opentelemetry.io/contrib/bridges/prometheus v0.56.0 // indirect
go.opentelemetry.io/contrib/exporters/autoexport v0.56.0 // indirect
go.opentelemetry.io/otel v1.31.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.7.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.7.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.31.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.31.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.31.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.31.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.31.0 // indirect
go.opentelemetry.io/otel/exporters/prometheus v0.53.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdoutlog v0.7.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.31.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.31.0 // indirect
go.opentelemetry.io/otel/log v0.7.0 // indirect
go.opentelemetry.io/otel/metric v1.31.0 // indirect
go.opentelemetry.io/otel/sdk v1.31.0 // indirect
go.opentelemetry.io/otel/sdk/log v0.7.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.31.0 // indirect
go.opentelemetry.io/otel/trace v1.31.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
golang.org/x/arch v0.11.0 // indirect
golang.org/x/exp v0.0.0-20230224173230-c95f2b4c22f2 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/text v0.19.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20241007155032-5fefd90f89a9 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241007155032-5fefd90f89a9 // indirect
google.golang.org/grpc v1.67.1 // indirect
google.golang.org/protobuf v1.35.1 // indirect
)
$ go build
..@./internal/pkg/instrumentation/bpf/database/sql/bpf_x86_bpfel.go:165:12: pattern bpf_x86_bpfel.o: no matching files found
..@./internal/pkg/instrumentation/bpf/github.com/segmentio/kafka-go/consumer/bpf_x86_bpfel.go:172:12: pattern bpf_x86_bpfel.o: no matching files found
..@./internal/pkg/instrumentation/bpf/github.com/segmentio/kafka-go/producer/bpf_x86_bpfel.go:167:12: pattern bpf_x86_bpfel.o: no matching files found
..@./internal/pkg/instrumentation/bpf/go.opentelemetry.io/auto/sdk/bpf_x86_bpfel.go:159:12: pattern bpf_x86_bpfel.o: no matching files found
..@./internal/pkg/instrumentation/bpf/go.opentelemetry.io/otel/traceglobal/bpf_x86_bpfel.go:213:12: pattern bpf_x86_bpfel.o: no matching files found
..@./internal/pkg/instrumentation/bpf/google.golang.org/grpc/client/bpf_x86_bpfel.go:170:12: pattern bpf_x86_bpfel.o: no matching files found
..@./internal/pkg/instrumentation/bpf/google.golang.org/grpc/server/bpf_x86_bpfel.go:169:12: pattern bpf_x86_bpfel.o: no matching files found
..@./internal/pkg/instrumentation/bpf/net/http/client/bpf_no_tp_x86_bpfel.go:182:12: pattern bpf_no_tp_x86_bpfel.o: no matching files found
..@.***/internal/pkg/instrumentation/bpf/net/http/server/bpf_x86_bpfel.go:174:12: pattern bpf_x86_bpfel.o: no matching files found
We should ship a Go module/package that is self-contained. It should not
require any user to generate code within their package cache.
cc @damemi https://github.com/damemi @RonFed https://github.com/RonFed
—
Reply to this email directly, view it on GitHub
https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1233,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOA77OFY2RJYDKU4ZOTTA3Z6FCSTAVCNFSM6AAAAABQ47ZXUKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZDKMZTHAYTCNA
.
You are receiving this because you were mentioned.Message ID:
@.***>

The thing is, you cannot execute a make task when someone downloads the package. What we ship now is a broken package.

I meant a make task to run go generate in the module cache. Which is not
pretty, but I would like us to look into other ways to make this easier
before we commit to committing the object files directly.
The BPF object files themselves aren’t code, they’re really release
artifacts. The code manages and depends on these artifacts to run, but to
me they seem kind of like DLLs in that sense. So, the package isn’t broken,
it’s just missing dependencies by default.
It looks like others have had this question before (
https://github.com/cilium/ebpf/discussions/988 with an interesting
solution). I don’t disagree that we should do something to make this
easier to use, but I’m hesitant to commit them.
On Wed, Oct 30, 2024 at 5:22 PM Tyler Yahn @.***> wrote:

The thing is, you cannot execute a make task when someone downloads the
package. What we ship now is a broken package.
—
Reply to this email directly, view it on GitHub
https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1233#issuecomment-2448408512,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOA77M2YPC3Z4ZAQDTCWK3Z6FEYZAVCNFSM6AAAAABQ47ZXUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGQYDQNJRGI
.
You are receiving this because you were mentioned.Message ID:
<open-telemetry/opentelemetry-go-instrumentation/issues/1233/2448408512@
github.com>

cilium/ebpf#988

This is talking about running a make target within the project generating the eBPF code. It is not talking about providing a release target to others downloading the Go module. That approach is going to be flawed as it will require someone who runs go get  or go mod tidy in their project to then go find our distributed Makefile and run another command. This is going to be especially problematic if the auto package is not even a direct dependency.

The code that we ship already directly depends on the files we do not release:
https://github.com/open-telemetry/opentelemetry-go-instrumentation/blob/77e8de9e0f4e33afb00bacb77917bad4f3056d1f/internal/pkg/instrumentation/bpf/database/sql/bpf_x86_bpfel.go#L163-L166
This seems like something that should be in source code if we are already referencing it.

I’m saying we should release those files, but I don’t think publishing them
in a Git commit is the best way to do that.
Are there other (non eBPF) Go packages that embed binary files and commit
them? I’m wondering if there is any precedent for this pattern outside of
eBPF or if we would just be trading one unusual situation for another.
Within eBPF-Go there isn’t much guidance. I linked that other issue to show
that the main problem (is it idiomatic to commit BPF objects to git) is
still pretty unanswered. Maybe this is a gap in the Cilium/bpf2go projects
themselves.
Again, I’m totally with you on publishing these files but I just want to
pause and see if we can think of a better approach, if possible.
On Wed, Oct 30, 2024 at 6:32 PM Tyler Yahn @.***> wrote:

The code that we ship already directly depends on the files we do not
release:
https://github.com/open-telemetry/opentelemetry-go-instrumentation/blob/77e8de9e0f4e33afb00bacb77917bad4f3056d1f/internal/pkg/instrumentation/bpf/database/sql/bpf_x86_bpfel.go#L163-L166
This seems like something that should be in source code if we are already
referencing it.
—
Reply to this email directly, view it on GitHub
https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1233#issuecomment-2448580563,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOA77LCFKYWCDRFDLAB4ADZ6FM7PAVCNFSM6AAAAABQ47ZXUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGU4DANJWGM
.
You are receiving this because you were mentioned.Message ID:
<open-telemetry/opentelemetry-go-instrumentation/issues/1233/2448580563@
github.com>

Looking at the remove import paths documentation, we need to use some VCS as the protocol to download source code (i.e. commit all the code), or run our own module proxy. Looking at the documentation for module proxies, it seems like we could respond to<EMAIL_ADDRESS>requests with a zip file including all the generated code.
This module proxy will need to be added as a managed service run by OTel and require additional step in our release process for uploading the release zip files.

FWIW, here's an example of another eBPF project checking in their object files: https://github.com/k8spacket/k8spacket/blob/65e5268975cc41a4a1b5daed8a534666a8a4fab5/ebpf/tc/tc_bpfel.o

I'm very supportive of improving this part - I agree with @damemi - committing these files to git should only be done if we can't find another reasonable approach.
In Odigos, we call make generate.
From my experience, besides this being not pretty it also makes development harder since each import of our library results in a build error - which makes testing harder.

Another idea: we could make our releases in release branches. Those branches would contained checked-in *.o files. If I understand the GitHub Go releasing model correctly (a big if), we might be able to delete the branch after the release is tagged and published.
I'm not sure if this is better or just the same with regard to binary data being checked in to git.

We could also post-curate the generate steps with a file manipulator that directly defines the object file binary data in the embedded type. This would be equivalent to what protobuf does. For example: https://github.com/open-telemetry/opentelemetry-proto-go/blob/53975ec09b7a48899d5153f09425f326e642a2dc/otlp/trace/v1/trace.pb.go#L963-L1119

@damemi @RonFed: please update the description with any additional proposals or pros/cons you can think of.

@MrAlias the post-curation idea is really interesting. That seems like the safest compromise and the pattern already exists for protobuf. I think that's my favorite option, and we could even separate the generated files into their own sub packages if we wanted.
Just thinking, you could make a philosophical argument here that while this is broken in the traditional Go sense, it's also not a very traditional Go library. I would absolutely expect almost any other Go package to not behave this way. But anyone who's using this, even transitively, should be aware that it depends on compiled eBPF objects... and so I wonder if the extra build/generate steps (like the make command @RonFed shared) could be an acceptable pattern? At least until some better guidance or alternative emerges?
To formalize that option, I'd propose our docs include a section on "using this library" that clearly includes exactly what commands the user needs as a setup step after they run go get (ideally a copy/paste one liner).

Post-process Go code to contain the object file

This object file representation will only be updated on releases or every time we change the c files?
I did not fully understand the advantages relative to just using the object files - is this not the same thing written differently? (i.e will have the same effect on git?)

Run and manage our own Go module proxy

With this approach, if a user go get our module, the proxy server will provide the .o files as if there were in git?
If so, that sounds like a complete solution to me (which will probably require a lot of work though).
A common question to most of the propsed soultions is whether we want to support imports of non-official releases (with commit hashes)? I think some of the options rely on just updating the compiled code in each official release.

I did not fully understand the advantages relative to just using the object files - is this not the same thing written differently? (i.e will have the same effect on git?)

I don't thing there objectively is a difference. The difference is stylistic. There is precedence with protobuf generating binary data directly held in .go files that is checked into repositories.
If others are okay with this approach, I'm okay as well. It is not my first choice though.

With this approach, if a user go get our module, the proxy server will provide the .o files as if there were in git?
If so, that sounds like a complete solution to me (which will probably require a lot of work though).

Correct. I haven't built out a prototype yet, but all the documentation I have read indicates this will be the behavior. We will upload snapshots of our code (maybe already zipped(?)) to some module proxy server. Those snapshots will be served and they will contain all the *.o files. The git history would remain as it currently is in this approach.

A common question to most of the propsed soultions is whether we want to support imports of non-official releases (with commit hashes)? I think some of the options rely on just updating the compiled code in each official release.

That's a good question. We would have to investigate this further to see what kind of support we could provide.
I know we can provide the standard Go proxy as a fallback to our proxy and it may all users to download directly from git history. The issue would be that those downloads would not have the desired object files.
I expect there is some more engineering we could do to try and solve the issue though.

Go mod proxies we could run BTW:

https://github.com/gomods/athens
https://pkg.go.dev/github.com/goproxy/goproxy
JFrog -- https://gavincampbell.dev/post/golang-proxy-artifactory/

The problem I see with the Go proxy solution is it doesn't support custom Probes, and makes it tougher for us to do a contrib-style repo. I guess third party probes can choose to provide their object files however they'd like, but a consistent approach would be nice

Another suggestion from ebpf slack was to use Git LFS, which from a quick search looks like it should be compatible with Go modules. I don't know if that would help with the code reviews and diffs, but it would solve the repo bloat.
I also like the release branch idea, and I think we should start using release branches anyway for backports once our release model is more stable.

It looks like using Git LFS will mean all modules that take this one as a dependency will need to have it installed and configured in their development environments: https://stackoverflow.com/questions/58280942/import-a-golang-module-with-lfs-objects

It looks like using Git LFS will mean all modules that take this one as a dependency will need to have it installed and configured in their development environments: https://stackoverflow.com/questions/58280942/import-a-golang-module-with-lfs-objects

Yeah, that is the downside. But I think it's still an improvement over the weird go cache generate commands you need otherwise since "all" you have to do is install git lfs.
By the way, when we finish the custom probes API https://github.com/open-telemetry/opentelemetry-go-instrumentation/issues/1105 this should contain the blast radius of this issue to just the probe modules themselves, since at that point the main auto-instrumentation framework won't have an explicit dependency on the probes. The problem still exists, but I think that in conjunction with one of the other solutions is going to be enough

I also had an idea for a proposal we could make to the Cilium libraries to support runtime object storage:
The compiled object bytes aren’t actually a build dependency, they’re only needed at runtime when the Cilium libraries load them into the kernel. The file just needs to be there for the Go compiler to build.
So if we built a similar approach to Git LFS and had bpf2go embed a file that just held a pointer to the actual object, then there wouldn’t be any compile errors and you could support any kind of runtime object storage: local, remote, lfs, etc… just have Cilium read the bytes into memory at runtime. The pointer file format would just have to declare the protocol and location and use that if an actual local file isn’t available.
There’s details to work out in the UX, like for example when developing you would want to be able to build a local file that takes precedence over whatever the pointer file says.
The downside is that you would need a connection to wherever the remote file is. We could solve this by supporting a persistent local cache (ie, download it to /tmp and read from there) or even a runtime flag that points to a local directory where the objects live (for example, each release we just publish the objects as a GitHub artifact, users download the ones they want for the probes they’re using, and save that in their host/image/etc). The tradeoff there is file I/O, but you could cache it in memory for efficiency if it's going to be loading/unloading often.
Wdyt? I'm pretty sure this is feasible but I have to look more into the details

SIG notes:

Each *.o file is ~50KB. Every change would mean another 50KB to the git history.

More updates from the slack thread:

We should make sure we're using llvm-strip to reduce the size of our .o files
Cilium already supports direct loading instead of embedding by calling ebpf.LoadCollectionSpec on the file and spec.LoadAndAssign on the go structs from bpf2go. If so, we could make this change today and immediately fix the IDE/compile time issue


Extend Map Layers Functionality to Study Areas Side Menu

Feature Request: Extend Map Layers Functionality to Study Areas Side Menu
Description:
Our current application includes a layer feature in the map view, allowing users to filter areas by groups, with each group only visible in its assigned layer. However, this functionality is limited to the map view. We've received user feedback requesting the extension of this layer-based filtering to the study areas side menu. This extension would enable users to focus on specific areas within a given layer, reducing the clutter of irrelevant information.

Layer Functionality in Areas Menu:
Extend the existing map layers functionality to the study areas side menu, allowing users to filter and work with areas based on the selected layer.

Selective Visibility:
Areas should only be visible in the menu when their corresponding layer is active, aligning with the current behavior in the map view.

UI Integration:
Add a select component beneath the search input in the areas menu. This component will facilitate layer selection, providing a user-friendly way to switch between different layers.

Sreenshots:

Synchronization with Map View:
Ensure that the layer selection is synchronized between the map view and the areas menu, providing a cohesive user experience.

[To be confirmed] User Preference Memory:
The application should remember the user's last selected layer in the areas menu and retain this selection across sessions.

Performance Considerations:
Implement this feature with attention to performance, ensuring that the additional layer functionality does not negatively impact the application’s responsiveness or speed.

About the "User Preference Memory" paragraph: to implement this, an association table between the list of studies and visited layers would be needed, and events like the following should be managed to update the table:

a layer is visited,
a layer is deleted,
a study is deleted.

This association table could be stored globally within the session and also at the level of the list of favorite studies.
At the back-office level, persisting this information is quite challenging because the list of layers is not specific to each user but rather global to all studies.

About the "User Preference Memory" paragraph: to implement this, an association table between the list of studies and visited layers would be needed, and events like the following should be managed to update the table:

a layer is visited,
a layer is deleted,
a study is deleted.

This association table could be stored globally within the session and also at the level of the list of favorite studies.
At the back-office level, persisting this information is quite challenging because the list of layers is not specific to each user but rather global to the study.

I was thinking about local storage persistence, cause it's only UI-related information (layers does not affect the study configuration).
But finally, I don't think it's a good idea from a user experience point of view. Maybe we can consider this as a future improvement if users asks for it


[transaction] Standalone mode, Transaction coordinator with id 0 not found

Describe the bug
I try to test transaction as the doc described, https://pulsar.incubator.apache.org/docs/en/txn-use/
And it throw such error, "Transaction coordinator with id 0 not found"

My standalone.conf only modify the following conf

transactionCoordinatorEnabled=true
allowAutoTopicCreationType=partitioned
allowAutoTopicCreation=true

I have tried using pulsar-2.9.2 and pulsar-2.10.0, both of them has the same problem
I also have tried add -nfw in standalone,  but the problem still exist
And it seems only occur in standalone, in normal cluster the problem would not occur
To Reproduce
Steps to reproduce the behavior:

change standalone.conf, start standalone
run demo

Desktop (please complete the following information):

OS: mac

For not blocking your test, you can try to create admin.topics().createPartitionedTopic(TopicName.TRANSACTION_COORDINATOR_ASSIGN.toString(), 1).
I will try to work on this issue.

For not blocking your test, you can try to create admin.topics().createPartitionedTopic(TopicName.TRANSACTION_COORDINATOR_ASSIGN.toString(), 1). I will try to work on this issue.

Actually i have noticed the pr-10238 and , and this system topic has been created. So i think maybe this is not the reason

Not reproduced.  Could you share your client demo ?

Ah, what is your client version ? is it matched the broker version ?

Ah, what is your client version ? is it matched the broker version ?

My fault, my maven project depends pulsar-flink-connector which is not pulsars-2.9.2
And after matching client version the problem is solved, thanks for your reply. I will close this issue


socket worker fail on recent fedora (was: distributed amuse fails (also probably buildbot problem) )

it seems connected to C codes (these fail >50% of the time but not always:
from amuse.units import units

from amuse.community.distributed.interface import DistributedAmuse, Pilot

instance = DistributedAmuse()

pilot = Pilot()
pilot.resource_name='local'
pilot.node_count=1
pilot.time= 2|units.hour
pilot.slots_per_node=2
pilot.label='local'
instance.pilots.add_pilot(pilot)
instance.wait_for_pilots()

instance.use_for_all_workers()

print "started..."
from amuse.community.fi.interface import Fi
from amuse.community.bhtree.interface import BHTree
f=Fi(channel_type="distributed")
#f=BHTree(channel_type="distributed")
print f.parameters

this works for Fi, but gets stuck for BHTree (most of the time, not always)
this is on recent system (e.g. strw, buildbot)

it may be the sockets worker...(test_c_sockets_implementation.py and starting sockets workers also fail)

after 556a658 seems ok again..

resolved by 556a658

the rate of the worker failings have gone down, but actually there still seems to be a problem  (this time at worker startup)

cannot reproduce...


Add MetalLB in FRR mode via flags, add webhookvalidation disable option

This PR adds the possibility to enable the MetalLB plugin in FRR mode. This is required for IPv6 Peering sessions.
While writing tests for this mode, i came across an issue with the validatingwebhookconfiguration, which is described here:
https://github.com/metallb/metallb/issues/1597
As the maintainers of MetalLB say themselves, this issue is entirely unpredictable and occurs seemingly at random. This is kind of bad, especially for automated tests. Hence, I added a turn-off flag for the validation webhooks. This can be dangerous, as the manifests wont get sanity checked by metallb then, so i only added the possibility to turn this off via the flags and not via the prompt. This way, we can disable them during tests, while keeping the user from just disabling them without knowing what that actually means.
You can also take a look at this blogpost: https://blog.mowoe.com/building-a-multi-node-ipv6-enabled-bare-metal-kubernetes-cluster-with-microk8s-metallb-longhorn-and-vyos.html
to learn more about why frr-mode is handy.

Thank you for this PR @mowoe. It is useful and it would be a good addition. We need to hold it however for a bit. We try to move away from the argument passing using the addon:arg:arg:arg notation as it is rather confusing. We will have to return on this PR as soon as we refactor the metallb addon.
I appreciate your understanding.

Hi @ktsakalozos, thank you for your reply!
I completely understand this decision and i have to say, i am glad, that you decided to abandon this way of argument passing, as i found it a bit tedious.
You can just get back to me here once you are finished with the transition and i will adjust my pr accordingly.


Help Wanted🙏🏻: External monitor causes an offset of the highlighted element and the cutout

more details

Findings so far (macos):

It does not happen without external monitor
It happens when an external 16:9 monitor is connected

It also happens when two 16:9 monitors are connected

It happens when monitors "extend" screen but also when they "mirror" the built-in screen
It does not happen with a external wide-screen monitor connected
It is NOT about screen size. It also happens on the laptop's original screen, if the above conditons apply

Found a workaround to basically applying object-fit cover to the svg, this way the misinterpreted window dimension which causes the offset, will get get cut of, and make the offset disappear

Closing this in favour of 11072fe, the stackoverflow question will stay open, but the workaround seems to be working fine


How wo get the groundtruth point cloud with normal vector from ShapeNet dataset?

Given the 3D model in the shapenet dataset(.obj files), how can I get the groundtruth point cloud with normal vector in your way to train my own model?
Besides, You said that you transformed it to corresponding coordinates in camera coordinate based on camera parameters from the Rendering Dataset.  I don't understand it clearly.  Can you share me how to preprocess the groundtruth point to prepare my data to train your model? I am not familiar with it indeed. Thanks very much!

@tangjilin see data_generation folder

@nywang16 Thanks very much！

Hi, @nywang16 After checking the https://github.com/nywang16/Pixel2Mesh/blob/master/GenerateData/4_make_auxiliary_dat_file.ipynb. I have the following confusions about the data generation part.
About the original data from shapenetcore.v1, are they all resized to be put in a unit cube, which are of size (-0.5,0.5)^3? And why would you choose ellipsoid with a size of (0.2,0.2,0.4) radius?
And you use your rendering_metacamera.txt annotations to transform 3d object to camera coordinates?

Hi @KnightOfTheMoonlight , our method is not sensitive to initial shape, you can also use sphere. The reason for using the ellipsoid as the initial may be that the average shape of the object is closer to the ellipsoid.

It is kind of sensitive though right? You want the initial mesh's vertices to project "nicely" over the input image so that the sampled image features  which are placed on those vertices properly capture the information in the image. If the range of vertices when projected into images space is too small in either direction information about the object will be lost, and similarly if the range is too large.  I would think any shape with uniformly distributed vertices in the range of (-0.5,0.5)^3 would be fine.

Hi, @EdwardSmith1884 . I don't think the initial shape is very sensitive. According to the performance of graph convolution during training, no matter what the initial shape is, it seems that graph convolution tends to learn the shape completely from scratch. The shape of the three-dimensional mesh output by the first few epochs is basically irrelevant, but is mainly affected by topology. Specifically, our experiments in the ECCV supp are as follows.

I think these all do well because they basically have uniformly distributed vertices in the range of (-0.5,0.5)^3 . I would think if the ellipsoid was very thin in one dimension, or the sphere was 1/10 the scale it would do worse. I guess I mean shape doesn't really matter but scale does, at least I observed worse much worse accuracy when I didn't scale the initial mesh properly.

Hi,
Do you know how to download the dataset and unzip it from the link below?
https://drive.google.com/open?id=131dH36qXCabym1JjSmEpSQZg4dmZVQid


KeyError: 'little'

Traceback (most recent call last):
  File "/Users/this/that/Takeout/Mail/other/mbox2eml.py", line 129, in <module>
    month_int = month_abbr_to_int[line_parts[3]]
KeyError: 'little'

also,
python mbox2eml.py --file SOMETHING.mbox --output .
Estimated amount of emails to create: 10953
Start process? [y|n]

Thank you for posting this issue. I will try to reproduce and fix it.
Which python version are you using?

Python 3.9.12

I fixed some minor errors regarding Python v3.
Would you please pull the changes and try again?


Integrate Vagrant

Automatic setup of a CodeCombat development environment using Vagrant, VirtualBox, and Ubuntu.
Once integrated, I can update the wiki to reflect the new instructions:

vagrant up from the root directory
scripts/vagrant/brunch.sh from the root directory
scripts/vagrant/dev-server.sh from the root directory

Once Brunch indicates that it's done compiling, visit http://localhost:3000.
The Vagrantfile and provisioning scripts were originally developed by dpen2000 and are hosted on GitHub under an MIT license. I am contributing based on the permissions under the MIT license to use, copy, modify, merge, publish, distribute, and sublicense without limitation.

Awesome, thanks @AlanHohn! Can you also extend it to running mongod (maybe with the --dpbath argument if needed) so that the dev database is also running, like the dev server and brunch?
@popey456963 @ssfinney want to test it out?

Sure, give me a day or so
On Fri, Mar 27, 2015 at 12:13 AM Nick Winter<EMAIL_ADDRESS>wrote:

Awesome, thanks @AlanHohn https://github.com/AlanHohn! Can you also
extend it to running mongod (maybe with the --dpbath argument if needed)
so that the dev database is also running, like the dev server and brunch?
@popey456963 https://github.com/popey456963 @ssfinney
https://github.com/ssfinney want to test it out?
—
Reply to this email directly or view it on GitHub
https://github.com/codecombat/codecombat/pull/2582#issuecomment-86810884
.

I'll try setting it up now.

Mongo: I think Ubuntu kicks it off automatically on install. When I vagrant ssh into the VM and type mongo it's running, and the node server seems to find it OK.

Trusty: Absolutely. I'll do that next. I'd also like to make it possible to control the RAM & number of CPUs for the VM from environment variables, so people with better dev machines can devote more power to running it. (Right now default is 2048 MB / 2 CPU.)

I'm still learning the typical development flow, but it seems like people need to download the latest MongoDB dump from the production server pretty frequently. When they do that, are there any extra steps to save off any local user progress that I can automate?

Being a completely naive user here, who has never used vagrant -

Downloaded Vagrant (Windows)
Already have Virtual Box installed
Ran installer for Vagrant
Now have a vagrant command I can use in cmd.exe
Do I have to download the github repo, move to it and then run vagrant up?

Have another, better image, showing why it breaks.

Things that I have learnt:

Use Virtual Box 4.3.12, newer versions do not work.
Run Vagrant Up once to download trusty, run it again to actually start the server up.
Turn vb.gui on (made a pull request about this) while debugging.  Don't turn it on when everything works (probably gonna remove pull request).

Thanks for the feedback.
I tried it using 4.3.22, but on a Mac. That may cause some of the differences. After I make the switch to trusty I will try it out on Mac and Windows. A single vagrant up should do.

Interesting, it might be windows being buggy, it generally is with this sort of thing.


Int128 compiler-rt methods (Int128 literal support part 1)

Includes the compiler-rt methods from #11196.
The modulo and dision methods do not work on windows due to the lack of SIMD support in crystal.

In rust's compiler-builtins, the method __muloti4 needed to be in extern "unadjusted" unstead of extern "C". Is it possible to do this in crystal?

It seems that's a calling convention in Rust, but I don't see it listed in LLVM: https://llvm.org/docs/LangRef.html#calling-conventions
I don't know what they call "unadjusted".
You can put a CallConvetion annotation to functions and methods:
https://crystal-lang.org/reference/syntax_and_semantics/annotations/built_in_annotations.html#callconvention
The values there should be updated.
Unfortunately I can't figure out what's this "unadjusted" in Rust... 🤔

I found something: https://github.com/rust-lang/rust/pull/38482#issuecomment-269074031
Good luck!
(I still think we should drop support for Int128 and UInt128)

The muloti4 specs pass on my local vm (llvm 11.1)
Maybe this got fixed in a version between what the CI is using and 11.1 ?
I don't think it would be worth fixing this if this is the case

Specs fail on win32 with both LLVM 10.0 and 11.1 for me. So it appears to be something different. But don't worry about that.
We've already established that Windows support is a sizable task. That should be worked on on its own, after this PR has landed.

Specs fail on win32 with both LLVM 10.0 and 11.1 for me. So it appears to be something different. But don't worry about that.
We've already established that Windows support is a sizable task. That should be worked on on its own, after this PR has landed.

Just checked again, looks like I forgot to replace the win32_std_spec.cr :smile:
Yeah, it does fail on 11.1.

I think this should be ready to review now

Add now-possible int operations to specs

I think this one should be reverted. We shouldn't use new compiler features in std specs. Otherwise you can't run std specs without compiling a new compiler.

(though maybe that changed, no idea... if CI passes it's fine, maybe someone changed that rule...)

(though maybe that changed, no idea... if CI passes it's fine, maybe someone changed that rule...)

Everything necessary to pass the specs is in the prelude, the compiler source hasn't changed.
Should be fine AFAIK


Add metrics & events

Description
Currently logging is the only source of information made accessible about controller activity, we should also publish Kubernetes events and metrics.
Events should be used to surface significant points of interest for operators deploying and manage resources with the controllers.
Metrics should be used to monitor performance.

[ ] initialise event recorder
[ ] add event func e.g. from kustomizationreconciler
[ ] add unit tests
[ ] publish event when returning happy path, requeuing, or setting condition (true/false)


Imported ptus have inconsistent number of bins (Las-X or ptufile related?)

Dear Christoph,
Many thanks for your work, I'm currently looking at FLIM data and it's really nice to be able to import it in Python rather effortlessly.
I have two questions, which I'm not sure are (fully) related to the ptufile library, or relate more to the technicalities behind Leica's software and microscope. So please don't feel obliged to answer.
I took FLIM data using the Leica Stellaris 8 platform, and exported .ptu files using their Las-X software for further custom processing in Python. I have multiple samples on which I performed measurements. The measurements are always according to the same settings. I imported files with the ptufile library.
Questions:

For different samples, the number of time bins appear to be different. (Even though the frequency was always 40 MHz and other settings also equal.) E.g. these are the shapes of two ptu objects that I imported using the ptufile library.
(1, 512, 512, 4, 266)
(1, 512, 512, 4, 282)
How can this be, why are there 266 bins in one sample and 282 bins in the other?
Importantly, (i) are the bin edges chosen differently and do they cover the same amount of time, or (ii) are there bins (perhaps with no detected photons) "missing" (and can bins with a value of zero thus be padded to the end of the measurement)?

The latter would actually be my guess, ie there might be missing bins for samples where there are "bins" that do not have photon counts, and therefor dropped. In which case I can introduce bins with zero counts to "pad" the data until dimensions are consistent among samples.

This question is definitely more Leica related, but perhaps you happen to know (and I have the feeling it is somewhat related to the above issue).
For most samples, the different channels are all exported to the same file, as reflected above by the fact that the 4th dimension has a length of 4 (I have 4 channels). For some samples, I however get 4 files for the single sample, e.g. sample "H1":
H1_seq1.ptu
H1_seq2.ptu
H1_seq3.ptu
H1_seq4.ptu
When imported with ptufile, these files show the following shapes:
(3, 512, 512, 1, X)
(3, 512, 512, 2, X)
(3, 512, 512, 3, X)
(3, 512, 512, 4, X)
Strikingly, these show:
a) Empty channels. E.g. channel seq4 only has values in channel 4, other channels show only zeroes, seq3 only has values in channel 3, and zeroes otherwise, and so forth.
b) I don't understand why there are now suddenly 3 entries along the first dimension, which also all hold data (ie for each of ptu[0,..], ptu[1,..], ptu[2,..] there's data; similar but slightly different).

I cannot look into ptu files, and my initial guess was that the seqX files represent the four channels I used. If so, the unexpected behavior I observe might also be partially due to ptufile, in case (I'm now hypothesising) data for each of the channels are indeed divided over 4 ptu files by Las-X, but ptufile introduces zeros if it only has information for say channel 3, and padds the other dimensions with zeros (this hypothesis is consistent with the file sizes of the seqX files, as the size of seq4 is not larger than seq3, suggesting the information is not redundant but rather specific for a channel).
Any input regarding these questions would be highly appreciated. Many thanks in advance!
All the best, Martijn

After writing this question and talking to a collegue I realized there might actually be a setting in the ptufile import function, and indeed there is.
From your sourcecode (class PtuFile(PqFile)):
    Parameters:
        file:
            File name or seekable binary stream.
        trimdims:
            Axes to trim. The default is ``'TCH'``:

            - ``'T'``: remove incomplete first or last frame.
            - ``'C'``: remove trailing channels not containing photons.
              Else use record type's default :py:attr:`number_channels_max`.
            - ``'H'``: remove trailing delay-time bins not containing photons.
              Else use record type's default :py:attr:`number_bins_max`.

If I change trimdims to "TC", indeed, all number of bins are equal (and set to 4096).
I still don't understand this though, as previously I was pretty sure my 250-270 bins (which were in fact trimmed) covered most of the 25 ns measurement time window corresponding to 40 MHz. What time window do the 4096 bins cover?

Hi @Jintram,
the PTU files exported from Leica software are special in many ways, some of which we are still trying to make sense of.

For different samples, the number of time bins appear to be different.

This is because ptufile by default returns all data/records/photons found in the file and trims the bins, channel, and time dimensions at the highest indices found. It is surprising to get data beyond what is expected from the frequency, but that is what is encoded in the files. I think this is due to some slight variations/inaccuracies in syncing macro and micro clocks. Or it could be a bug in the Leica software. Anyway, ptufile has many options to control which parts are decoded:
https://github.com/cgohlke/ptufile/blob/ae65544920bdd70d22a16e920ca5641bbeed4a6a/ptufile/ptufile.py#L1834-L1914
In your case you want to set dtime=0, which will return number_bins_in_period bins.

For some samples, I however get 4 files for the single sample, e.g. sample "H1":
When imported with ptufile, these files show the following shapes:

I don't know why Leica exports some measurements to separate files. As for ptufile returning different shapes, the situation is similar to the histogram bins. Ptufile scans the encoded data for the highest channel with photons. It does not determine if some lower channels are empty (same for histogram bins). To work around that, specify the channel to load, for example, channel=3 for H1_seq4.ptu.

I don't understand why there are now suddenly 3 entries along the first dimension

Ptufile detects 3 time frames based on frame and line markers in the encoded data. You can integrate the time frames using the frame parameter, for example, frame=-1 will integrate all frames into one.

What time window do the 4096 bins cover?

This is the maximum number of bins the hardware/firmware can collect. Similar, if you do not trim the channel axis, ptufile will return the maximum number of channels the hardware supports.
Btw, to preview the image and metadata contained in a PTU file, run python -m ptufile filename.ptu
Hope this answer your questions.

Dear Christoph,
Many thanks for this extensive reply, this is hugely helpful to understanding the details of analyzing the .ptu files and using ptufile!
Just to organize my thoughts (and perhaps also useful for other readers rather new to this analysis), let me summarize;
So indeed ptufile trims the number of bins if they are empty. Using your suggested command python -m ptufile filename.ptu (very useful) gives me the important information I need to properly analyze my data. In my case:
frequency: 39010000.0
global_resolution: 2.563445270443476e-08
number_bins_in_period: 264

This tells me that my frequency is actually not precisely 40 MHz, but 39.01 MHz, and that the time period over which photons are captured is 25.6 ns, and that 264 bins are spaced across that period. With potentially tail bins lacking or even additional bins added to the end -- this latter phenomenon being confusing as you point out.
Using my_ptu = PtuFile(base_filepath + '.ptu') and dir(my_ptu) I can also see that these parameters are directly available in Python through
>>> my_ptu.frequency
39010000.0
>>> my_ptu.global_resolution
2.563445270443476e-08
>>> my_ptu.number_bins_in_period
264

which is very convenient.
Anyways my approach will thus be to either zero-pad data to 264 bins and ignore bins >264.
You suggest I can also have ptufile return the appropriate number of bins directly by using dtime=0. Before I've been using the function my_ptu = PtuFile(base_filepath + '.ptu') to read my files (I realize now this initializes an object so it's not a function technically I guess), which doesn't allow for the parameter dtime to be passed. However, if I use my_ptu_img = ptufile.imread(base_filepath + '.ptu', dtime=0), I indeed get back the expected dimensions based on above information.
>>> my_ptu_img.shape
(1, 512, 512, 4, 264)

This makes my life much easier.

I don't know why Leica exports some measurements to separate files. (..) To work around that, specify the channel to load (..)

You can integrate the time frames using the frame parameter, for example, frame=-1 will integrate all frames into one.

This behavior indeed remains puzzling, but many thanks for the suggested workarounds!
(I also realized I indeed have three line repetitions in the measurement, so it makes sense that there are 3 frame in the first dimensions, but it's weird that these then don't show up separately in my other files -- but I can now work around this anyways.)
I will also try to delve more into your code to understand options I have to read in these files (e.g. use imread vs. PtuFile).
Again many thanks for your time and thanks for this tool.

Hi @Jintram,
your summary looks correct.
ptufile.imread() is a convenience wrapper for ptufile.PtuFile().decode_image(). The advantage of using the PtuFile class is that it gives access to all metadata and the decoded records in the file.
Not returning the channels without photons is on the TODO list. There is a metadata tag for that (HWInpChan_Enabled) but one can't rely on it being present.

a) Empty channels. E.g. channel seq4 only has values in channel 4, other channels show only zeroes, seq3 only has values in channel 3, and zeroes otherwise, and so forth.

Ptufile v2024.10.10 also trims leading empty channels.

a) Empty channels. E.g. channel seq4 only has values in channel 4, other channels show only zeroes, seq3 only has values in channel 3, and zeroes otherwise, and so forth.

Ptufile v2024.10.10 also trims leading empty channels.

Thanks for incorporating this!


perf(v2): replace unnecessary json stringify(string) with inline string

Motivation
Rather than doing stringify for simple string like below
const a = `test${JSON.stringify(word)}`;

Better do
const a = `test"${word}"`;

https://jsperf.com/stringifyvsdirect/1
Have you read the Contributing Guidelines on pull requests?
yes
Test Plan
(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your changes work. Bonus points for screenshots and videos!)

[x] Locally everything still ok
[ ] Netlify
[ ] CI pass

previously it wasnt a string. Its an object iirc. Some refactor did previously and sometimes miss this small detail

Seems like regression. Need to revert this. Turns out I put JSON.stringify previously to escape backslash on windows.
Should've put comment lol..

Haha we should have also went to see the commit in the past to see why we added that (isn't always useful, but could be useful). Feel free to revert.

Doing it tonight. I am quite occupied this week with personal stuff (eg: renewing my passport)


cannot import cosipy

I just updated to the most recent version of cosipy and I can't  run import cosipy. This issue is caused by an import in COSILike.py:
from cosipy.coordinates.orientation import Orientation_file
If I comment this file out I can import cosipy.
Also, I am not sure who is working on the coordinates class, but the corresponding init files have not been set up.

The Orientation_file import is fixed in my pull request.

Thanks, Eliza. I'll take a look at your PR.

@eneights, I see @israelmcmc has already been going through your PR. I will wait for those comments to resolve.


Stop redacting sensitive keys & data when blanking

Motivation and Context
When we redact a value, we should be able to find out which value was changed or blanked out, but previously we were replacing ALL the sensitive data with "REDACTED" -> "REDACTED" which told us very little about what actually got changed.
What has changed
Only redact values, not keys.
How to test?
Use the wizarding ways.
Links
Trello: https://trello.com/c/R1sxFecV

 Enhancement: be able to see which PII field was updated - only redact the value (3)


Exclude files / folders from clean task

I'm working on a large site with some extra folders in the root of the website (./public/) with client related stuff. When I build the site with gulp-starter the whole root folder gets cleaned (cleared). Is it possible to set an extra config param to exclude folders / files from the clean task?

I'm certainly in favor of this feature... or instead of deleting the actual config.root.dest folder, perhaps adding config options for the clean task so its possible to negate certain folders...
I will try put something together shortly.

yeah, especially when it wipes out index.php for your CMS :)

@stevieg83 you can get around that by change your root dest to something like assets, so only your assets folder gets wiped out by the clean task.

good shout

We definitely need to address this. There's a couple of routes we could go, and maybe it's a combination of them:

We can add an exclude folders/files option like @johanneslamers suggested
Provide an include folders/files option (inverse of above)
We can also add an option to disable the "clean" step - but that'd create some problems if you're  using file revisioning... new files would just keep piling up.

Yes please! I am working on switching over to Blendid from my homegrown build process, but I want to use a customized HTML task to copy my Craft templates to dist/craft/templates (so I can hard-code the revved asset filenames during build). That means I need to use dist as my dest folder...which means the clean task wipes out Craft every time. I would probably lean towards whitelisting rather than blacklisting, but either one would work for my purposes.

@elivz why do you need to hard-code the filenames? We included a Craft plugin to handle that automatically: https://github.com/vigetlabs/blendid/blob/master/extras/craft/ASSETS-README.md

@elivz

I would probably lean towards whitelisting rather than blacklisting, but either one would work for my purposes.

Good call, I think that's a probably a better approach.

Well, I guess I don't need to. But it's always seemed cleaner to me to handle all the revisioning in the build process and not have the extra plugin dependency. I already modified the rev task to replace the asset urls in my Twig templates (PR coming...), now I just need to find a way to move those template files outside of the webroot without "cleaning" my entire CMS installation every time.


Why is the required macro version of mermaid not specified in the documentation?

I'm just curious why the documentation doesn't call out that a specific version macro is required in order for Mermaid to work?
I was able to make some minor changes and get the variant I have installed to work but I feel like it might be worth mentioning that for best results you should install mermaid version X

We now have another option with mermaid.go so I believe this issue can be closed.


Enabled mypy in pre-commit config

@rabernat I think you merged #386 a bit pre-emptively - I've been running mypy just on grid_ufunc.py, and when I run on the whole code on the master branch I get a couple of typing errors that I don't understand:
xgcm/duck_array_ops.py:29: error: All conditional function variants must have identical signatures
doc/conf.py:244: error: Need type annotation for "latex_elements" (hint: "latex_elements: Dict[<type>, <type>] = ...")
Found 2 errors in 2 files (checked 19 source files)

I copied the actual pre-commit config from xarray, removing some datetime stubs that seemed irrelevant.

[x] Passes pre-commit run --all-files

I fixed the type errors - this should be ready to merge now.


Using polymorphism to simplify object storage in canvas #16

https://github.com/1chooo/uml-editor/issues/16

Hi @alexeyleping
Could you please attach a screenshot or a recording of this change to confirm that the app is still working correctly?

Hi @alexeyleping
Could you please attach a screenshot or a recording of this change to confirm that the app is still working correctly?

I believe #16 and #17 will be included in one PR, which means #21 and #22 will be together in one PR. Please create new PRs that contain your excellent work! Could you write the commit messages in English? I'm sorry, but I cannot understand Russian.

hi @1chooo
the sheet with line objects has been deleted, everything has been moved to the shapes sheet, and a check for class membership has been added

hi @1chooo the sheet with line objects has been deleted, everything has been moved to the shapes sheet, and a check for class membership has been added

So does it mean like we don't have to define isInside() in BasicObject, Group, Line. Because we define all in Shape. If yes, I don't think this is great choice which means if we want to draw more object, we cannot add the feature directly to the new graphic, we have to go back to Shape.

Please remove https://github.com/1chooo/uml-editor/pull/23/commits/34010279e4568b830ff7d86a2fa66a5a2555b2b4 and https://github.com/1chooo/uml-editor/pull/23/commits/cb70ce0e97a3bf9af2f7a5b20c713d3371f9d132 because they are not related to this PRs.


All integer flavors bound to (u)int64_t type and narrowing conversion check

Hi !
I was wondering if all integer flavors are really 'bound' to int64_t type ?
From the header:  sqlpp11/include/sqlpp11/data_types/integral/data_type.h, and of experience with your library ( that is amazing btw :) ) it looks like so.
What is the reason behind this choice ?
I would like to enforce verification on unintentional narrowing conversion (-Wconversion) and having the library returning int64_t (or uint64_t) for every integer types creates a lot of noise. I could put static casts everywhere to avoid that, but I think that when defining a column with a 'shortint' as a type, it's more intuitive to expect that same type in return.
Or maybe I'm doing things wrong from the start :)
What is your point of view on that subject ?
Thanks you very much :)

Hi,
Thanks for the flowers :-)
At the time of writing the library, it seemed like the simplest thing to do and it is kind of hard to change now. The rationale was that whatever you do, whether you select a value straight from a table or select SUM(foo), you can be rather certain that the result will fit into 64 bit with most engines.
In hindsight, this was not the best choice. I think I solved it better in sqlpp17, where you can specify the C++ type per column. This is much more flexible and turns out to actually be simpler to implement.
It would be possible to port the model back to sqlpp11, but I lack the time to do so.
If you are feeling adventurous, you might want to checkout sqlpp17.
Best,
Roland

As said, this is hard to change now.
Closing the issue.


compilation fails on ubuntu

Hello,
I'm trying to use mpi crate in a cargo project, and the compilation seems to fail.
I have gcc, and mpich installed on ubuntu.
From the error message, it seems, the compiler can't find "mpi.h"
Is there any environment variable, or something else required to make this work?

Error Log (after running cargo build)
   Compiling mpi-sys v0.2.2
The following warnings were emitted during compilation:

warning<EMAIL_ADDRESS>In file included from src/rsmpi.c:1:
warning<EMAIL_ADDRESS>src/rsmpi.h:3:10: fatal error: mpi.h: No such file or directory
warning<EMAIL_ADDRESS>    3 | #include "mpi.h"
warning<EMAIL_ADDRESS>      |          ^~~~~~~
warning<EMAIL_ADDRESS>compilation terminated.

error: failed to run custom build command for `mpi-sys v0.2.2`

Caused by:
  process didn't exit successfully: `/home/ares/work/nox/nox/target/debug/build/mpi-sys-7ad4d673f8337353/build-script-build` (exit status: 1)
  --- stdout
  OPT_LEVEL = Some(0)
  TARGET = Some(x86_64-unknown-linux-gnu)
  OUT_DIR = Some(/home/ares/work/nox/nox/target/debug/build/mpi-sys-c8e6400adddd0b78/out)
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(true)
  CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
  HOST = Some(x86_64-unknown-linux-gnu)
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
  CFLAGS_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
  CFLAGS_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:warning=In file included from src/rsmpi.c:1:
  cargo:warning=src/rsmpi.h:3:10: fatal error: mpi.h: No such file or directory
  cargo:warning=    3 | #include "mpi.h"
  cargo:warning=      |          ^~~~~~~
  cargo:warning=compilation terminated.

  --- stderr

  error occurred: Command "mpicc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-o" "/home/ares/work/nox/nox/target/debug/build/mpi-sys-c8e6400adddd0b78/out/2e40c9e35e9506f4-rsmpi.o" "-c" "src/rsmpi.c" with args mpicc did not execute successfully (status code exit status: 1).

Thanks!

I was able to fix it.
Apparently, along with mpich, we also need to install libopenmpi-dev as per stackoverflow
This might be a common issue for other new users for rsmpi too, so maybe documenting this would be helpful for them in future.
Thanks.

If you intended to use MPICH, then you needed libmpich-dev. You're using Open MPI now. These implementations are binary-incompatible. Either is fine, but if you have both installed, make sure to use the correct mpiexec.


Cannot use 'Object' as class name as it is reserved

Cannot use 'Object' as class name as it is reserved

I'm updating the extension so it can be used with Yii2.0.13 + PHP 7.2. I still need to fixe some tests and I'll release a new version soon.

Hi，Cannot use 'Object' as class name as it is reserved，7.1 its end of life and is no longer supported.


Got another bug when run with rabbitmq at version 0.3.6

Hi,
After you resolved the bug the workflow not consume queue and release v0.3.6. I have upgraded and when running I got this bug immediately

the length of list workflow_state_json is 0 so it cannot enter to the condition check is None. So it create loop infinite

@volethanh do you have a snippet of code to reproduce?

@volethanh do you have a snippet of code to reproduce?

Just use the manual orchestration then I got that bug


RDBES code review: RS_Region

https://vocab.ices.dk/?ref=1640
delete and replaced by ecoregion?

For the RDBES core group to provide area references (documentation)
Can ICES Ecoregions be applied instead?

RDBES core group: suggest RCGRegion for CodeType - DM need to clarify if they wanted to use already RCG list or make their own RCGregion - reference to EU map region/workplan master table - NA (North Atlantic) as code not good when working with R

Will need to be discussed with Neil in the new year.

I will ask Colin if this is the best way to handle this NA issue in R or there is an easier work around it commonly used for similar situations.

Just found this:
https://www.fisheries-rcg.eu/
@cekv

We already have this code type: https://vocab.ices.dk/?ref=1503 - We should make them use a subset of that or the ecoregion one (https://vocab.ices.dk/?ref=1414) otherwise maybe it should be free-text

Right! will put this up in the next core group, but I saw we are missing one RCG econ in our vocab, I guess we should update it? @cekv
Let's discuss it in the next rmg

Add RCG Econ to the list.


Query Operation Names + TypeGraphQL Default Pluralization Support

Provides the ability to specify queryOperationNames configuration options to alter the GET_ONE, GET_MANY, GET_MANY_REFERENCE and GET_LIST operations. This is particularily useful when typegraphql-prisma cannot find a plural word.
When typegraphql-prisma cannot pluralize a word, it changes the query functions to not have a pluralized form and instead uses prefixes such as findMany and findUnique in front of those resources. This fix changes the default if we encounter one of those types of words.

hey @mwillbanks thanks for your contribution and sorry for beeing veeery late.


RATIS-1092. Move the group management methods from RaftClient to a new interface

See https://issues.apache.org/jira/browse/RATIS-1092

LGTM, +1

@szetszwo Thanks the patch. I have merged it.


Add SemanticTokens to lsp-types crate

There's a proposal for proper syntax highlighting in the protocol:
https://github.com/microsoft/vscode-languageserver-node/blob/8c8981eb4fb6adec27bf1bb5390a0f8f7df2899e/protocol/src/protocol.sematicTokens.proposed.ts
We should switch our bespoke impl to that API. The first step would be to add corresponding rust structs to the lsp-types crate:
https://github.com/gluon-lang/lsp-types
This issue should actually be raised in that repository, but I am raising it here for visibility :-)
Note that lsp-types already contains an alternative extension (see SemanticHighlightingClientCapability), which is different from SemanticTokensClientCapabilities which is actually proposed for the inclusion into the protocol.
Note that this is a relatively high-priority task, so help would be appreciated. cc @kjeremy, I think you  might already be doing some work here?
cc https://github.com/rust-analyzer/rust-analyzer/issues/604

@matklad I have a branch that adds the types to gluon-lang/lsp-types but haven't finished it yet due to some tight schedules at work. I raised the issue of the alternate protocol here: https://github.com/gluon-lang/lsp-types/issues/133
I believe that the md document for SemanticHighlightingClientCapability is just an older version and that the proposed ts file is the correct one. The same thing happened with call hierarchy.

If I don't get to it first I am happy to mentor someone on adding types to the lsp-types crate.

LSP PR: https://github.com/gluon-lang/lsp-types/pull/137

So I think my plan of attack now is to start with textDocument/semanticTokens and for the predefined types and modifiers.
@matklad I think I'm going to need some guidance on how to go about this in terms of getting the semantic information from the syntax.

@kjeremy that's basically what our highlighting.rs does? I think you don't even need to change that, just to transform it's results to the format expected by the LSP

Oh do you mean syntax_highlighting.rs? I'm never looked at this. I had to turn highlighting off because things got weird.

This looks perfect thanks.

For the bits that are beyond LSP: https://github.com/microsoft/vscode/wiki/Semantic-Highlighting-Overview

This has been done.


Mmtl slicing birds dataset

Not necessary to the improvement of this MeTaL fork. The birds dataset will not be used in this project.


VICTORY HAND emoji doesn't have emoji modifiers

In previous version, VICTORY HAND emoji had emoji modifiers.
What's the reason?

The following emoji modifier seems to disappear.

weight_lifter
golfer
man_in_business_suit_levitating
sleuth_or_spy
raised_hand_with_fingers_splayed
point_up
person_with_ball
v
writing_hand

Do you mean the variation selector U+FE0F? Those are now conforming to the Unicode 10 / Emoji 5 spec. The main codepoint includes a variation selector where appropriate and a non_qualified version without it.

Thank you for your reply.
I mean skin color modifier.
Although I think that the emojis should have skin color,
They don't have it.

good catch - i'm looking into why these are missing

this was related to non-qualified/fully-qualified codepoints. the 9 you noted use fully qualified forms (with -FE0F) when used alone, but drop the modifier when used with skin tones. this wasn't handled correctly when building the map. i'm fixing this in PR #110 which will also include all the missing images and updated sheets.

now fixed on master and in 4.0.1
no downstream libraries have been updated yet


FLEX-4355 ~ Adds message priority to XSDs.

This pull request makes the build fail.

retest this please

This pull request makes the build fail.

retest this please

This pull request makes the build fail.

retest this please

SonarQube analysis reported 1 issue

 1 major

Watch the comments in this conversation to review them.

All is well.

This pull request makes the build fail.

This pull request makes the build fail.

This pull request makes the build fail.

retest this please

This pull request makes the build fail.

This pull request makes the build fail.

All is well.

This pull request makes the build fail.

All is well.

All is well.

This pull request makes the build fail.

All is well.

All is well.

All is well.


Translated MinGW Windows headers - complete set

Windows programming with D quickly finds that the core.sys.windows.windows header is incomplete. Work has been done to port the public domain win32 headers to D:
https://github.com/CS-svnmirror/dsource-bindings-win32
(among other repos) and we sometimes copy/paste definitions too. This brings them all under the druntime banner, in the core.sys.windows namespace - same as we do with Posix and stdc headers.  Now, Windows libraries don't need to face incompatibilities anymore from using the different modules to access the same functionality and D will work for low-level Windows devs out of the box (at least 64 bit, we still have the .lib file issue on 32).
I realize this is enormous (I just manually verified the changes of win32->core.sys.windows is good by doing it in each individual file myself!) but these files are typically unchanged for years. It really isn't any more work than us all maintaining our private bindings anyway. Yes, it is 4 MB of code uncompressed, but does that really matter? I think not.
Besides, this will solve the common complaint that D sucks on Windows because most the API isn't even available out of the box. Totally worth doing.

Note also: http://forum.dlang.org/thread/cqgfyjbcftcjzqpveulx@forum.dlang.org?page=2#post-m875ga:2428a:242:40digitalmars.com

And an OLD OLD bug: https://issues.dlang.org/show_bug.cgi?id=317
Those of us who use D on Windows have wanted this for many years.

Part of this does use explicitly private exports and I believe some of the imports might be expecting to be public, but I'm not sure. (sorry if that doesn't make sense)
src/core/sys/windows/windows.d should probably be the package definition, and src/core/sys/windows/core.d may need to go away, depending on how and where it's used.

On Fri, Jan 02, 2015 at 03:14:05PM -0800, Orvid King wrote:

Part of this does use explicitly private exports and I believe some of the imports might be expecting to be public, but I'm not sure. (sorry if that doesn't make sense)

Yeah, I need to try it on some real world code and see what happens
before we actually pull this in, I just wanted to open the PR ASAP so
folks can start to weigh in early too.
This is potentially the most exciting week in D for me for a while!

@adamdruppe Can this be used for testing? It is a translation of 5th Edition Programming Windows book by Charles Petzold which is quite outdated, but still covers a lot of Win32 stuff that are relevant today.

FYI I'm not sure about the build state of that project with the latest compiler. I haven't touched that codebase in a while.

test fail

@adamdruppe any chance to revive this?

On Wed, Jul 08, 2015 at 11:33:19AM -0700, Dmitry Olshansky wrote:

@adamdruppe any chance to revive this?

Yeah, it is still something I want to do, gotta do a little more
translation though and I've been busy with a lot of other things.
Remind me, i'll try to do it this saturday.

So I think this thing is actually working now (lol i thought it would take 7 minutes to throw these in, I'm now at over 7 hours on it)...
But the tester failed. Can someone take a look at this?
https://auto-tester.puremagic.com/show-run.ghtml?projectid=1&runid=1642588&isPull=true
Do I need to manually copy the files from /src to /import? I kinda assumed that would be done automatically.... or is something else causing that?

I guess yes. Just look into win32/win64 makefile. They are (as usual)  just a ton of of copy-pasty copy commands:
https://github.com/D-Programming-Language/druntime/blob/master/win32.mak#L613
P.S. I really-really wish we had that reggae tool to build D.

ugh, all done in a list too :( this is like 188 files... really easy to manage with *.d but a list like that will make me miserable. Easy to generate a script to make the list but still, come on.

If your Walter-foo is high this might be a good time to convince him of Makefiles being evil.
But that's OT, I  think generating these rules is the way to go to merge this.

@adamdruppe The first merge confilic commit is NOT good, you pull a whole lot of work done on master into a single commit. Please go the pull --rebase route. In fact you can just redo the pull on top of fresh upstream master.
Anyhow pinging to see how it goes.

On Wed, Jul 15, 2015 at 12:30:57PM -0700, Dmitry Olshansky wrote:

@adamdruppe The first merge confilic commit is NOT good, you pull a whole lot of work done on master into a single commit. Please go the pull --rebase route. In fact you can just redo the pull on top of fresh upstream master.

How do I do that?
tbh I don't really know how to use git... I just do the
basics over and over again.

Anyhow pinging to see how it goes.

I still have to fix the makefile then it should be set.

How do I do that?

Okay, If I had to do it I'd :
git remote -v # check to see if this has upstream repo as remote pointing to DLang
git checkout master
git pull --rebase upstream master
git checkout -b new_branch_name

# apply your changes from this pull to the working tree
git add std/ # add all files in std/
git commit
git push

Then anytime you want to update this pull:
git checkout branch_name
git pull --rebase upstream master

OK, I think I got the commit rebase worked out. Last thing should just be fixing the makefile.

@adamdruppe - any progress on makefiles?

Should also add fix Issue 317 to commit message

On Tue, Jul 28, 2015 at 09:10:56PM -0700, Dmitry Olshansky wrote:

@adamdruppe - any progress on makefiles?

I'm on it again now. What a pain!

I added the version comparison code in case you're interested: https://github.com/etcimon/windows-headers/commit/96be3200f2ee05e601ba372911e434c4506da0e9#diff-469c8eb924b62375a9d80614bfc3351eR177

On Sun, Aug 02, 2015 at 07:48:46PM -0700, Etienne Cimon wrote:

I added the version comparison code in case you're interested: https://github.com/etcimon/windows-headers/commit/96be3200f2ee05e601ba372911e434c4506da0e9#diff-469c8eb924b62375a9d80614bfc3351eR177

Would you be able to do that as a pull request against my branch
or have we already diverged too much?

Would you be able to do that as a pull request against my branch or have we already diverged too much?

I based my repository on the original dbindings. You can see a dub.json file in there as well

WTF, it looks like the auto tester is still using the old makefile, but I updated it here.
copy src\core\sys\solaris\sys\priocntl.d  import\core\sys\solaris\sys\priocntl.d
        1 file(s) copied.
copy src\core\sys\windows\com.d  import\core\sys\windows\com.d
        1 file(s) copied.
copy src\core\sys\windows\dbghelp.d  import\core\sys\windows\dbghelp.d
        1 file(s) copied.
copy src\core\sys\windows\dll.d  import\core\sys\windows\dll.d
        1 file(s) copied.
copy src\core\sys\windows\stacktrace.d  import\core\sys\windows\stacktrace.d
        1 file(s) copied.
copy src\core\sys\windows\stat.d  import\core\sys\windows\stat.d
        1 file(s) copied.
copy src\core\sys\windows\threadaux.d  import\core\sys\windows\threadaux.d
        1 file(s) copied.
copy src\core\sys\windows\windows.d  import\core\sys\windows\windows.d
        1 file(s) copied.
copy src\core\sys\windows\winsock2.d  import\core\sys\windows\winsock2.d
        1 file(s) copied.
copy src\etc\linux\memoryerror.d  import\etc\linux\memoryerror.d

but the file now lists like 200 things under the windows section:
https://github.com/adamdruppe/druntime/blob/windows_headers/win32.mak
This is really starting to just annoy me. The make process here is way harder than it needs to be.

This is really starting to just annoy me. The make process here is way harder than it needs to be.

INDEED. At least now we have one more of key people ion favor of damping win32/win64 makefiles ;)

@adamdruppe could it be tabs vs spaces?

Oh, I see the problem - take a look at mak/COPY file and copy-paste (again) the full listing of new files there. It's completely brain-dead build script.

And cybershadow's doc tester uses .d, so files there but not in make can break it too. Ugh! (.d is a pretty sane way to do it, but I wish it was consistent one way or another)
BTW we still need to get the updated .libs in here for linking either way. That's not a PR issue though, that's part of the build process.

UGH THE *.D MEANS THE MERE EXISTENCE OF AN OLD WINSOCK HEADER "BREAKS" IT!

BTW we still need to get the updated .libs in here for linking either way. That's not a PR issue though, that's part of the build process.

Let's leave that to some other day. On x64 we use MS SDK so all is good.
On 32bits we may just add an instructions on how to obtain OMF versions of libs use implib and analoguos tools.

Well, I'm hoping Phobos will compile with this now. I'm trying not to break the build of code using the old way, but there's a few differences in things like capitalization. I'm hacking around them with aliases.

@adamdruppe almost there but not quite (

On Thu, Aug 06, 2015 at 03:37:22PM -0700, Dmitry Olshansky wrote:

@adamdruppe almost there but not quite (

Yeah, I still see it, there's a bunch of little things
still and I have to make it compatible enough so Phobos
still compiles too, blargh.
But I'll probably be on it the weekend again.

@adamdruppe ping

On Sat, Aug 29, 2015 at 12:35:27PM -0700, Dmitry Olshansky wrote:

@adamdruppe ping

Yeah, I still plan to wrap this up, just other things have
been on my todo list recently (out of town the last two weekends,
working on the house tomorrow...)

I'm hoping Phobos will compile now, though there's still a LOT of functions that aren't const correct :( We'll probably have to do a massive second-pass effort to fix that.

@adamdruppe any chance to look into this again?

std\windows\registry.d(540): Error: function core.sys.windows.winreg.RegQueryValueExW (const(void*), const(wchar), uint, uint*, ubyte*, uint*) is not callable using argument types (const(void*), Res, typeof(null), uint*, void*, uint*)
Why isn't the alias this being triggered there :( ugh ugh ugh.

It compiles now! Just some macros not being linked in.......

So, for the linker thing, I'm trying to make these header-only. That is, so they will work in the include path without needing to be compiled or linked. So the C macros are being translated into zero-arg templates (as I come across them....) and structs will have all-zero initializers (again, as I come across them). This means ones with char members have a =0 instead of the default.
I'm pretty sure that will remove the need to reference .init in the object file, since all zeroes makes that null in druntime anyway, which means we don't have to actually compile all these in  (which would get painful when doing various -version switches), saving a little binary runtime space too.
But waiting on the autotester to confirm this doesn't break anything else....

And not necessary to add a list of new files in mak\SRCS?

On Thu, Sep 10, 2015 at 03:26:05PM -0700, Dmitry Atamanov wrote:

And not necessary to add a list of new files in mak\SRCS?

I want to avoid that since then the pragma(lib)'s are liable
to trigger when you don't actually want them (especially since
dmd 32 bit doesn't even include them all).
Arguably, those pragmas should just be removed, but I like having
them there anyway. So trying to get the best of both worlds.
I'm finally to the point where Phobos actually builds, there's
just test failures.... for some reason.

Well, I might have to concede at least the winbase.d on which Phobos depends being added to the sources. Seems that dmd, even if the initializers are all zero, wants to reference the .init if they aren't zero by default. (If they are zero by default, it works fine without it. I think this is actually a minor bug, but meh, I REALLY do want to get this merged some time this year, so i'll work with it for now)
Hopefully, last run through the autotester.

@adamdruppe Which version of MinGW Windows headers translated?
I'am using [mingw-w64] (http://sourceforge.net/projects/mingw-w64) and noticed that many of the headers missing in translated D sources.

On Thu, Sep 10, 2015 at 06:19:03PM -0700, Dmitry Atamanov wrote:

@adamdruppe Which version of MinGW Windows headers translated?

They are actually pretty old. I don't even remember where they
came from at first. I borrowed someone else's D translation and
tried to get it to fit into the druntime pattern.
We've been talking about this for years so I figured it is time
to sit down and actually make it happen (and now, almost a year
later than that, it might finally be getting there...)

AAAAAAAARRRRRRRRGGGGGGGGGHHHHHHHHH all tests passed except win64 with this delicious error:
std\process.d(1014): Error: core.stdc.stdio.SEEK_END at ..\druntime\import\core\stdc\stdio.d(230) conflicts with core.stdc.stdio.SEEK_END at ..\druntime\import\core\sys\windows\mmsystem.d(1019)
A symbol conflicts with ITSELF! !@$##*(ing module bugs! I blame bugzilla 314.

AAAAAAAARRRRRRRRGGGGGGGGGHHHHHHHHH all tests passed except win64 with this delicious error

Too bad. However I seriously doubt that it's bugzilla 314. You can seek around for aliases but I've seen this bug with wrong module names supplied that had nothing to do with 314.

On Thu, Sep 10, 2015 at 09:58:49PM -0700, Dmitry Olshansky wrote:

Too bad. However I seriously doubt that it's bugzilla 314. You can seek around for aliases but I've seen this bug with wrong module names supplied that had nothing to do with 314.

I was able to work around that anyway... now it is only giving a
fail on Win64.... and it is doing so with no error message. Come on.

now it is only giving a fail on Win64.... and it is doing so with no error message. Come on.

If you translate the error code to hex, it's 0xC0000005: an access violation.

On Fri, Sep 11, 2015 at 07:15:36AM -0700, Rainer Schuetze wrote:

If you translate the error code to hex, it's 0xC0000005: an access violation.

aye, I noticed that too... but where? I'll have to set up
a win64 box to run it locally (I only have win32 set up)

aye, I noticed that too... but where? I'll have to set up
a win64 box to run it locally (I only have win32 set up)

I'll try your PR on Win64 and see how it goes.

Thanks. I'm pretty sure there's just a DWORD where there
should be a SIZE_T somewhere and the compiler didn't catch
it at the usage point. I'm also going to look at some point
today - I have win64 available here now, just gotta install
VS and dmd on it.

The win64.mak is unbelievably awful. My dmd install on 64 bit just worked well for my programs but fails miserably to build this. Why does the druntime build have to be this bad? It drives me nuts.

TESTS FINALLY PASS!!!!!

Splendid! Squash it and I'll do the honors

@ all, please see https://github.com/D-Programming-Language/druntime/issues/1402 which is a different approach, based on a recent copy of the headers and hopefully will result in less code breakage. Currently blocked on a DMD regression.

Sample of regressions identified by my test script introduced in this PR:
wintest32.d(62): Error: undefined identifier 'PCCH' in module 'core.sys.windows.windows', did you mean alias 'PCH'?
wintest32.d(64):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of PCCH is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 64, (_error_).stringof, "const(char)*"))
wintest32.d(107): Error: undefined identifier 'LPCTCH' in module 'core.sys.windows.windows', did you mean alias 'LPTCH'?
wintest32.d(109):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of LPCTCH is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 109, (_error_).stringof, "const(char)*"))
wintest32.d(113): Error: undefined identifier 'PCTCH' in module 'core.sys.windows.windows', did you mean alias 'PTCH'?
wintest32.d(115):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of PCTCH is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 115, (_error_).stringof, "const(char)*"))
wintest32.d(331): Error: type of FARPROC is:
extern (Windows) int function() nothrow @nogc
but should be:
extern (Windows) int function() nothrow

wintest32.d(344): Error: undefined identifier 'LPCOLORREF' in module 'core.sys.windows.windows', did you mean alias 'COLORREF'?
wintest32.d(346):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of LPCOLORREF is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 346, (_error_).stringof, "uint*"))
wintest32.d(352): Error: type of DLGPROC is:
extern (Windows) int function(void*, uint, uint, int) nothrow @nogc
but should be:
extern (Windows) int function(void*, uint, uint, int) nothrow

wintest32.d(355): Error: type of TIMERPROC is:
extern (Windows) void function(void*, uint, uint, uint) nothrow @nogc
but should be:
extern (Windows) void function(void*, uint, uint, uint) nothrow

wintest32.d(358): Error: type of GRAYSTRINGPROC is:
extern (Windows) int function(void*, int, int) nothrow @nogc
but should be:
extern (Windows) int function(void*, int, int) nothrow

wintest32.d(361): Error: type of WNDENUMPROC is:
extern (Windows) int function(void*, int) nothrow @nogc
but should be:
extern (Windows) int function(void*, int) nothrow

wintest32.d(364): Error: type of HOOKPROC is:
extern (Windows) int function(int, uint, int) nothrow @nogc
but should be:
extern (Windows) int function(int, uint, int) nothrow

wintest32.d(367): Error: type of SENDASYNCPROC is:
extern (Windows) void function(void*, uint, uint, int) nothrow @nogc
but should be:
extern (Windows) void function(void*, uint, uint, int) nothrow

wintest32.d(370): Error: type of PROPENUMPROCA is:
extern (Windows) int function(void*, const(char)*, void*) nothrow @nogc
but should be:
extern (Windows) int function(void*, const(char)*, void*) nothrow

wintest32.d(373): Error: type of PROPENUMPROCW is:
extern (Windows) int function(void*, const(wchar)*, void*) nothrow @nogc
but should be:
extern (Windows) int function(void*, const(wchar)*, void*) nothrow

wintest32.d(376): Error: type of PROPENUMPROCEXA is:
extern (Windows) int function(void*, char*, void*, uint) nothrow @nogc
but should be:
extern (Windows) int function(void*, char*, void*, uint) nothrow

wintest32.d(379): Error: type of PROPENUMPROCEXW is:
extern (Windows) int function(void*, wchar*, void*, uint) nothrow @nogc
but should be:
extern (Windows) int function(void*, wchar*, void*, uint) nothrow

wintest32.d(382): Error: type of EDITWORDBREAKPROCA is:
extern (Windows) int function(char*, int, int, int) nothrow @nogc
but should be:
extern (Windows) int function(char*, int, int, int) nothrow

wintest32.d(385): Error: type of EDITWORDBREAKPROCW is:
extern (Windows) int function(wchar*, int, int, int) nothrow @nogc
but should be:
extern (Windows) int function(wchar*, int, int, int) nothrow

wintest32.d(388): Error: type of DRAWSTATEPROC is:
extern (Windows) int function(void*, int, uint, int, int) nothrow @nogc
but should be:
extern (Windows) int function(void*, int, uint, int, int) nothrow

wintest32.d(405): Error: type of FAILED is:
bool function(int Status) nothrow @nogc
but should be:
bool function(int) pure nothrow @nogc

wintest32.d(408): Error: type of SUCCEEDED is:
bool function(int Status) nothrow @nogc
but should be:
bool function(int) pure nothrow @nogc

wintest32.d(809): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(810): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(811): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(814): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(815): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(819): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(820):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of LIST_ENTRY is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 820, "LIST_ENTRY", (_error_).stringof))
wintest32.d(821): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(822): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(828): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(834): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(835): Error: undefined identifier '_RTL_CRITICAL_SECTION' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION'?
wintest32.d(838): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(839): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(844): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(850): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(856): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?

There are many more (wintest32.d is 8374 lines long), but DMD stops compiling after 20 errors. These are all fixed in #1402.
I uploaded the generated test programs here: git clone git@gist.github.com:bd4bd187133b03c15e79.git

@DmitryOlshansky let's hold off on a merge until we see how CyberShadow's turns out, he seems to making good speed on it and has a better test case than I do.
I agree with him that some of mine are declared @nogc when they shouldn't be and he found a few other bugs in there too (and I was already aware of some const correct issues, which have been fixed in the ~10 months since I forked the bindings to start this, and cybershadow's includes them). So, while mine finally passes the tests in Phobos, if his pass all tests too it is a better way to go.

@DmitryOlshansky let's hold off on a merge until we see how CyberShadow's turns out, he seems to making good speed on it and has a better test case than I do.

Agreed. Having WinAPI covered by tests is much better;)

Closing this because CyberShadow's superior alternative got in! Yay, finally!


MAINT: Migrate to standard version control procedures

It's a bit weird to me to have recipes/mne_python_*.*/ rather than a single recipes/mne_python/.

Most old recipes very well might not work anymore.
I don't think we'll ever really support multiple versions simultaneously, but even if we do it seems more standard to do this with branches (like maint/1.5 and maint/1.6) than it does with folders.
Having multiple versions ruins our git blame for lines (for example our latest construct.yaml only has changes going back ~3mo even though some lines are effectively much older)

@hoechenberger if you agree with migrating to a more standard version control scheme under a single recipes/mne_python/, we could try to migrate gracefully by merging backward (rename 0.23 to root mne_python, then merge 0.24 into mne_python, then 0.25 into 0.23, etc. in single PRs). That way we at least get to see what was changed in each version release. (I'll get all git blame in terms of authorship but at least it'll be clearer when stuff was changed.)

@larsoner I'm also happy with just declaring the latest recipe "the" recipe and deleting the old ones, to make things easier… in any case, +1 to having only a single recipe.


Context property route is undefined when aot is disabled

What version of Elysia is running?
1.1.23
What platform is your computer?
Darwin 23.6.0 arm64 arm
What steps can reproduce the bug?
You can create a simple server like this:
const app = new Elysia({
  aot: false
})
  .get('/hi', ({ route }) => `Hello World ${route}`)

app.listen(3000)

What is the expected behavior?
The route property in the context should not be undefined even when aot is set to false.
What do you see instead?
When you hit the endpoint, you should see Hello World undefined as the response.
Setting aot to true will return Hello World /hi

maybe this solution will solve your problem click

When aot is enabled the route is also missing at version 1.2.6
const app = new Elysia()
    .use(
        opentelemetry({
            resource: otelConfig.resource,
            spanProcessors: [
                new BatchSpanProcessor(
                    new OTLPTraceExporter({
                        url: env.OTEL_EXPORTER_OTLP_ENDPOINT,
                    }),
                ),
            ],
        }),
    )
    .decorate("logger", logger)
    .use(swagger())
    .use(notificationsApp)
    .get("/health", () => ({ status: "OK" }))
    .listen(3000);


Jonmv/badge adjustments

@freva please review and merge.

Hang on, should make historic bars slightly wider again.

There.