[Bug]: Fix Subscriptions search for the different postgres versions

What happened?
Using more complex search syntax, like the | operator test fail as the database does not return a result. This test also used to be flaky, so it might be caused by a race condition somewhere, however higher versions of postgres do seem to make the results more predictable, indicating a relationship to the database version.
See the result of a matrix test run, where the Python version and Postgres version are manipulated.
https://github.com/workfloworchestrator/orchestrator-core/actions/runs/8555513507/attempts/2?pr=599
Version
alpha
What Postgres version are you seeing the problem on?
Postgres 11, Postgres 12, Postgres 13

After the first initial runs, we can start to see a pattern. The behaviour of to_tsquery seems to be different from postgres 14 onwards. Further more it seems that test_multiple_subscriptions is flaky.

There is a difference in how Postgres parses the input to to_tsquery between versions <14 and >=14.
We should be able to search for a single subscription_id with either of the the following queries:

subscriptionId:99933227-08bb-46f5-960c-6a2ec3ddaab4
subscription_id:99933227-08bb-46f5-960c-6a2ec3ddaab4

This is handled by orchestrator-core by parsing the string and passing the result to to_tsquery like so:
to_tsquery('simple', 'subscription <-> id <-> 99933227-08bb-46f5-960c-6a2ec3ddaab4')
The result of this expression is:
-- Postgres <14
'subscription' <-> 'id' <-> ( '99933227' & '-08' & 'bb-46f5-960c-6a2ec3ddaab4' & 'bb' & '46f5' & '960c' & '6a2ec3ddaab4' )

-- Postgres >=14
'subscription' <-> 'id' <-> ( '99933227' <-> '-08' <-> 'bb-46f5-960c-6a2ec3ddaab4' <-> 'bb' <-> '46f5' <-> '960c' <-> '6a2ec3ddaab4' )

The Postgres<14 result means searching for the tokens subscription followed by id followed by 99933227 AND -08 AND ... . This is nonsensical and will never yield a result.
In contrast, the Postgres>=14 result means all the tokens should be consecutive, which is exactly what we need and gives the correct result.

There is another issue, which is that Postgres tokenizes text using a default parser which can read parts of a UUID as a number.
For example, consider the output of the following query:
SELECT * from ts_debug('simple', replace('0e12d6bb-ef55-4e1e-9b10-5cf1afeef440', '-', '_'));

alias
description
token
dictionaries
dictionary
lexemes

sfloat
Scientific notation
0e12
{simple}
simple
{0e12}

numword
Word, letters and digits
d6bb
{simple}
simple
{d6bb}

blank
Space symbols
_
{}
null
null

numword
Word, letters and digits
ef55
{simple}
simple
{ef55}

blank
Space symbols
_
{}
null
null

sfloat
Scientific notation
4e1
{simple}
simple
{4e1}

asciiword
Word, all ASCII
e
{simple}
simple
{e}

blank
Space symbols
_
{}
null
null

numword
Word, letters and digits
9b10
{simple}
simple
{9b10}

blank
Space symbols
_
{}
null
null

numword
Word, letters and digits
5cf1afeef440
{simple}
simple
{5cf1afeef440}

This is the behavior of the default parser (the only one shipping with Postgres). Adding your own parser entails creating a procedure in C and requires superuser privileges on the database. Not a good option.
This has consequences for querying on a UUID. For Postgres >=14, we can just query for the entire UUID, as the query is parsed in the same way as the tsvector, but in Postgres <14, we would have to do it manually or some records can not be found with a UUID search...
I have implemented a workaround for this in #603, which emits separate db queries to ts_parse and let the db parse parts of the search query so that it matches with the ts_vector (This is the normal behavior in Postgres  >=14)


Issues using dataflow plugin

When I try to import dataflow it could not be found, I opened the fsm example and wanted to try this tool out on it. I am not sure if this was an option I needed to select at some point or file I needed to edit so that this plug in would be available but it doesn't seem to be in the hal_plugins folder or on the GUI plugin display.
When I run the following script:
from hal_plugins import dataflow
res = dataflow.Dataflow.analyze(netlist)
I receive the following error:
ImportError: cannot import name 'dataflow' from 'hal_plugins' (/home/mldobs01/hal/build/lib/hal_plugins/init.py)
screen shot of folder for hal_plugins:

screen shot of GUI plugins:

The dataflow plugin appears to be missing in your install. How did you compile HAL? Did you enable the plugin at build time?

I believe I enabled it but it has been a while since installing it and I can't remember/wasn't planning on using the dataflow plug in at the time, is there an easy way to enable the plugin post install or do I need to uninstall everything and reinstall?

If you did not enable the plugin at build time, it was not compiled and is hence unavailable. I believe you don't necessarily  need to reinstall, but it's easiest and you should not lose any data.
If you want to try the manual route, you will still need to recreate your build config with cmake option -DPL_DATAFLOW=1, as described in the link from my last comment. Then you invoke ninja or make, depending on which build system you're using, to compile. This will produce a lib/hal_plugins/dataflow.so library in your build directory. You might also get away with only compiling that one library with ninja dataflow.so or make dataflow.so.
I'm not sure about how you installed HAL. If you used the default install scripts, your plugins should have gone to /usr/local/lib/hal_plugins/. You can copy the dataflow.so from your build directory here. HAL will pick up any new plugins on launch.

I am going to reinstall and try enabling the plugins this time and am using the Ubuntu 22.04 system. For this I would just run the following code correct?
git clone https://github.com/emsec/hal.git && cd hal to clone the Git repository
./install_dependencies.sh
mkdir build && cd build
cmake -DBUILD_ALL_PLUGINS=1
make to compile HAL


[AF] "Changes will be lost…" dialog pop up even no change made

"Changes will be lost…" dialog pop up again and again if user Click "OK" on it then do other operations like go to Manage tab.
Expect result:
Dialog should only pop up once.
Actual result:
Dialog pop up again when doing other operations even no changes made.
Repro steps:

Open or create a function app.
Create a HttpTriggerC# function.
In Integrate page, edit Request Parameter Name -> Go to Develop tab without saving -> Click OK in poped dialog.
Go to Integrate tab, verify "Cahgne will be lost" dialog pop up again.

can not repro


proposal: cmd/go: add fips140 module selection mechanism

Update, Nov 11 2024: An updated version is at https://github.com/golang/go/issues/70200#issuecomment-2468562595. The change is to stop presenting the choice as anything like a module, and instead to use a GOFIPS140 environment variable, analogous to GOOS, GOARCH, CGO_ENABLED, and so on.

We are working toward getting Go crypto FIPS validated, to be able to remove BoringCrypto. The relevant code is going to be crypto/internal/fips/.... People are going to need to be able to select between a few different crypto/internal/fips trees when using any given Go distribution. We propose to treat the crypto/internal/fips tree as its own pseudo-module, with a new go build flag, -fips140, to choose the version.
In general, any build needs to choose between the actual latest source in $GOROOT/src/crypto/internal/fips and an earlier source snapshot stored in module zip form in $GOROOT/lib/fips140. (Earlier snapshots will have been through more validation processes and may be necessary for certain government-related use.)
The crypto/internal/fips module will use these version numbers:

the version in the actual Go 1.X.Y source tree will be called v1.X.Y.
snapshots taken ahead of Go 1.X.Y will be called v1.X.Y-fips.N for N starting at 1 and incrementing.

Note that the versions order correctly: v1.24.0-fips.1 < v1.24.0. The idea is that toward the end of the 1.24.0 cycle, a snapshot would be taken and sent to a lab for validation, so that by the time 1.24.0 is released, "v1.24.0-fips.1" will be an "in-process" FIPS module, meaning a lab has validated it but NIST has not yet signed off. (When NIST signs off, it becomes a "certified" module.)
The -fips140 flag can take one of the following possible values:

-fips140=off (the default) means to use the source in $GOROOT/src/crypto/internal/fips and set the default GODEBUG for compiled binaries to fips140=off. All other flag settings set the default GODEBUG to fips140=on.
-fips140=latest means to use the source in $GOROOT/src/crypto/internal/fips (and default binaries to fips140=on, which I will stop saying).
-fips140=v1.X.Y-fips.N means to use the snapshot in $GOROOT/lib/fips140/v1.X.Y-fips.N.zip.
-fips140=inprocess means to use the version listed in $GOROOT/lib/fips140/inprocess.txt (it will be v1.X.Y-fips.N for some X, Y, N).
-fips140=certified means to use the version listed in $GOROOT/lib/fips140/certified.txt (it too will be v1.X.Y-fips.N for some X, Y, N).

Note that the definitions of inprocess and certified are specific to the Go toolchain being used, to keep builds reproducible. (That is, NIST issuing a certification for a new version does not change the meaning of -fips140=certified in an older Go release.)
crypto/internal/fips will be like a module in the sense that:

When you run go list, packages in crypto/internal/fips/... will be reported as coming from a module named crypto/internal/fips.
crypto/internal/fips has a set of versions, module zips, and checksums.
Earlier snapshots maintained in $GOROOT/lib/fips140 will use the module zip format.
When using an earlier snapshot, the go command unpacks the module zip into the module cache and then uses that source in the build.
Programs that use go list or x/tools/go/packages will see the module information for crypto/internal/fips and will see source coming from the module cache when using earlier snapshots.
crypto/internal/fips will be listed in the go version -m output with its own version, in all builds using those packages. This will enable tools like govulncheck to report vulnerabilities accurately.

crypto/internal/fips will be special, or unlike a module, in these ways:

The module zips will only ever be loaded from $GOROOT/lib/fips140, never from the module proxy.
The checksums for the zips will only ever be loaded from $GOROOT/lib/fips140/fips140.sum, not from the checksum database.
crypto/internal/fips will not be listed in go.mod or participate in module version selection; the -fips140 build flag is the only way to choose the version.
The name "crypto/internal/fips" does not start with a domain name element, which violates golang.org/x/mod/module.CheckPath ("missing dot in first path element").
In -mod=vendor builds, fips code will come from the fips places ($GOROOT or else an unpacked zip), never from the vendor directory.

The specialness seems unavoidable. We could go all the way and make fips completely different and not look anything like a module, but it is necessary to be able to say what version of fips a given program uses, and modules are the vocabulary we use for talking about versions. So it makes sense to reuse that vocabulary. In general the special cases seem like they will not cause much trouble. In particular, programs that use x/tools/go/packages or go list to find information about packages in a build will see a Dir set to a directory in the module cache when the alternate fips locations are being used. Since they already handle seeing other packages in the module cache, they should keep working without any changes.
The only special case that might cause trouble is if clients of module.CheckPath try to check "crypto/internal/fips". If that turns out to be a problem, we can update module.CheckPath to allow crypto/internal/fips as a special case. In my prototype of this functionality, I skipped the calls to module.CheckPath in the few places where it mattered, avoiding any changes to x/mod.
I have a prototype of this code working. It needs some cleanup and tests, but the necessary changes are quite small:
 src/cmd/go/internal/cfg/cfg.go               |   1 +
 src/cmd/go/internal/load/pkg.go              |   5 +-
 src/cmd/go/internal/modfetch/cache.go        |  11 +-
 src/cmd/go/internal/modfetch/codehost/git.go |   2 +
 src/cmd/go/internal/modfetch/coderepo.go     |  14 +-
 src/cmd/go/internal/modfetch/fetch.go        |   2 +-
 src/cmd/go/internal/modfetch/fips.go         | 239 +++++++++++++++++++++++++++
 src/cmd/go/internal/modfetch/repo.go         |  11 +-
 src/cmd/go/internal/modfetch/sumdb.go        |   4 +
 src/cmd/go/internal/modload/build.go         |  10 +-
 src/cmd/go/internal/modload/import.go        |  10 +-
 src/cmd/go/internal/modload/init.go          |   8 +
 src/cmd/go/internal/modload/load.go          |   2 +-
 src/cmd/go/internal/modload/query.go         |   2 +-
 src/cmd/go/internal/work/build.go            |   7 +-
 15 files changed, 306 insertions(+), 22 deletions(-)

In summary, the proposal is to establish the crypto/internal/fips pseudo-module with the version scheme and properties described above, and to add the -fips140 go build flag. (Note that “build flags” apply not just to go build but to most other go commands, including go install, go test, and go list.)

Related Issues and Documentation

proposal: crypto: mechanism to enable FIPS mode #70123
crypto: obtain a FIPS 140-3 validation #69536
crypto/internal/boring: add dev.boringcrypto README.md text
boringcrypto: Not using FIPS compliant calls for SHA functions (also should we add md5? for easy proof of FIPS mode on) #66513 (closed)
proposal: crypto/md5: deprecated cryptography used for security purposes or in security sensitive codepaths #70043 (closed)

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

-fips140=certified means to use the version listed in $GOROOT/lib/fips140/certified.txt (it too will be v1.X.Y-fips.N for some X, Y, N).

Am I understanding correctly that if v1.24.0-fips.1 is "in-process" when 1.24.0 is released, but certified before 1.24.1 is released, then 1.24.1 will include a backport CL to change $GOROOT/lib/fips140/certified.txt to v1.24.0-fips.1?

How many "earlier snapshots" are kept in $GOROOT/lib/fips140? e.g., will I be able to use v1.24.0-fips.1 with Go 1.30.0?
I wonder mainly due to thoughts of (a) distribution size with keeping many versions and (b) there may be a maintenance burden with keeping compatibility with very old modules, particularly if there are dependencies on internals in std.

What is the use-case for -fips140=inprocess?

I also wonder how a process of adding any new public API to the crypto packages would look like after this change. Before adding any new API we would have to wait for it to be available in N amount of latest fips modules (https://github.com/golang/go/issues/70200#issuecomment-245762325)? Or is the -fips140 flag going to define implicitly a build tag such that it can be detected and conditionally compiled with //go:build.

What is the use-case for -fips140=inprocess? Why someone would need to use it instead of latest/certified. Is there any benefit/requirement for using a in process FIPS module?

AIUI submissions to the FIPS validation queue take a long time to become certified with the current NIST backlog. Some policy regimes (I think notably fedramp) may allow using modules that are in the process of becoming certified since they've had lab scrutiny if not yet final sign off. If a dependent user's requirements allow it then -fips140=inprocess would let you benefit from improvements made since the last certification process ended without blocking on NIST to finish the approval of the new module.

At this point there are no FIPS versions in process or certified.  So presumably -fips140=inprocess and -fips140=certified will fail.
You say that go list or x/tools/go/packages will see module information for crypto/internal/fips.  For go list that will presumably be based on the -fips140 command line flag.  How will x/tools/go/packages determine the FIPS version to use?
I imagine that organizations will want to ensure consistent use of a FIPS version across their builds.  Today they can get boringcrypto with a single consistent environment variable GOEXPERIMENT.  I guess the corresponding environment variable would be GOFLAGS=-fips140=certified.  But how will that be applied to x/tools/go/packages?  Or (perhaps the same question) for Bazel or other build systems?
It sounds like the certification process is going to be based on a particular version of the source code.  What will be the schedule for further certifications?  How will this affect ordinary maintenance of the crypto/internal/fips packages?  Should we do typo fixes for comments?  Should we do routine updates to use new standard library functionality, such as the recent changes from sort.Sort to slices.SortFunc (for cases with special-purpose implementations of sort.Interface)?
When using Go 1.25, will it be possible to use -fips140=v1.24.1-fips.1?  (That is, can 1.25 use FIPS for 1.24.)  When using Go 1.25.3 will be possible to use -fpis140=v1.25.1-fips.1?  (That is, can 1.25.3 use FIPS for 1.25.1.)

How many "earlier snapshots" are kept in $GOROOT/lib/fips140? e.g., will I be able to use v1.24.0-fips.1 with Go 1.30.0?
I wonder mainly due to thoughts of (a) distribution size with keeping many versions and (b) there may be a maintenance burden with keeping compatibility with very old modules, particularly if there are dependencies on internals in std.

The idea is to keep only the latest certified and the latest in-process modules as snapshots. Sometimes we might need to have a third snapshot that is going to become in-process but isn't yet, but hopefully we can time things well enough that won't happen.
Indeed, the contact surface between the standard library and the modules is a burden in both directions: the modules might use internal APIs (although I tried to minimize that) and the standard library uses the module APIs (see below).

I also wonder how a process of adding any new public API to the crypto packages would look like after this change. Before adding any new API we would have to wait for it to be available in N amount of latest fips modules (#70200 (comment))? Or is the -fips140 flag going to define implicitly a build tag such that it can be detected in crypto packages for conditional compilation with //go:build.

We discussed this in abstract and we're happy with new APIs returning an error if -fips140 is not off or latest. Concretely, we haven't discussed how to implement that, but an implicit build tag is an option. As long as we're confident we can make it work, we don't need to decide the specifics until we add new module-dependent APIs.

At this point there are no FIPS versions in process or certified.  So presumably -fips140=inprocess and -fips140=certified will fail.

Indeed.

It sounds like the certification process is going to be based on a particular version of the source code.  What will be the schedule for further certifications?  How will this affect ordinary maintenance of the crypto/internal/fips packages?  Should we do typo fixes for comments?  Should we do routine updates to use new standard library functionality, such as the recent changes from sort.Sort to slices.SortFunc (for cases with special-purpose implementations of sort.Interface)?

We'll maintain the crypto/internal/fips tree as usual, with an eye to not breaking the stdlib-facing API too often (see above). At least every year we'll kick off a revalidation to keep a healthy pipeline of module versions moving to in-process and certified.

When using Go 1.25, will it be possible to use -fips140=v1.24.1-fips.1?  (That is, can 1.25 use FIPS for 1.24.)  When using Go 1.25.3 will be possible to use -fpis140=v1.25.1-fips.1?  (That is, can 1.25.3 use FIPS for 1.25.1.)

Yes. Consider that modules take up to a couple years to reach certification at the moment, so the latest certified for Go 1.26 might still be v1.24.1-fips.1. However, only the versions that are snapshotted in lib/fips140 will be available, of which we'll only keep up to three usually (see above).
(Left out the x/tools/go/packages questions that I might not be the best person to answer.)

We discussed this in abstract and we're happy with new APIs returning an error if -fips140 is not off or latest. Concretely, we haven't discussed how to implement that, but an implicit build tag is an option. As long as we're confident we can make it work, we don't need to decide the specifics until we add new module-dependent APIs.

Ok, not everything can/should return an error, but i assume panic is also fine, for new APIs, to avoid build tag we can also add a version const into the fips module.

Why not double down on the pseudo module idea, and instead of having a flag at all, just use replace directives.
That way you can set it in the go.mod for things that really should build in fips, or in a go.work if you want to build temporarily in fips mode without touching the go.mod
replace crypto/internal/fips => crypto/internal/fips/inprocess

#30241 was an accepted but unimplemented proposal for standard library vendoring, would it make sense to have the fips module work like that?

Inevitably, there will come a day when the latest in-process and certified version will be found to contain a vulnerability. What -fips flag is used to update to the patched version?
In OpenBSD land, we talk of versions 7.6-release (which would correspond to your certified) and 7.6-current (which is approximately your gotip) and 7.6-stable (which is the release version with only critical security and reliability patches). So another way of asking my question is what is your name for "FIPS-stable"?
I got verbal assurance in public from Donna Dodson, back when she was in charge of all this, that NIST intends that even the most compliance-sensitive government clients should run "FIPS-stable" and not wait around for re-certification. It would be good to get this in writing from the current administration, but also plan in advance for version naming.
There ought to be some associated public logs maintained at NIST for recording such patches.

Inevitably, there will come a day when the latest in-process and certified version will be found to contain a vulnerability. What -fips flag is used to update to the patched version?

Good point, we discussed this but forgot to mention it in the proposal. The plan if a vulnerability is found in v1.24.0-fips.1 while it's the certified module is to issue v1.24.0-fips.2 and update the certified pointer in a regular Go patch release, because indeed even the most compliance-sensistive deployment is expected to weight in favor of mitigating a security vulnerability.
We can keep the previous module, too, if nonetheless we get complaints about diverging from the certified source. This can make the number of modules in lib/fips140 grow, but we did a retrospective of the security vulnerabilities in the Go crypto packages of the last five years, and only 2-4 (depending how you count) would have had to be fixed in a FIPS module, and all of them could be mitigated from the standard library. We could also be clever and compress them one against the other, if it becomes necessary.

Are there any values of the -fips140 that would set the GODEBUG to enforce?
The interaction between the -fips140 flag and the GODEBUG seem pretty complicated. Do we actually need the GODEBUG? Is there a significant need for changing that at runtime rather than compile time?

You say that go list or x/tools/go/packages will see module information for crypto/internal/fips. For go list that will presumably be based on the -fips140 command line flag. How will x/tools/go/packages determine the FIPS version to use?

Ping @adonovan to comment on the interaction with go/packages here.

cc: @timothy-king @findleyr for x/tools implications. Seems like most of the cruft is handled by go list:

Programs that use go list or x/tools/go/packages will see the module information for crypto/internal/fips and will see source coming from the module cache when using earlier snapshots.

but there may be tools consequences (likely minor) of these rules:

The name "crypto/internal/fips" does not start with a domain name element, which violates golang.org/x/mod/module.CheckPath ("missing dot in first path element").
In -mod=vendor builds, fips code will come from the fips places ($GOROOT or else an unpacked zip), never from the vendor directory.

Are there any values of the -fips140 that would set the GODEBUG to enforce?

No, I expect enforce to be less popular, and applications can opt-in with the regular GODEBUG mechanisms.

The interaction between the -fips140 flag and the GODEBUG seem pretty complicated. Do we actually need the GODEBUG? Is there a significant need for changing that at runtime rather than compile time?

We've heard from a few folks that would like to make a single build and then choose to operate it in FIPS mode or not at runtime. The benefits of GODEBUG=fips140=off are faster crypto/rand and faster key generation, because FIPS 140-3 has some pretty heavy (and not generally useful) requirements on those. Another nice property of this design is that any Go binary can be operated in FIPS mode by just switching the GODEBUG at runtime, if your compliance regime is ok with "updated" modules.
I think build flag and GODEBUG are pretty orthogonal: the former selects the module version, the latter whether the binary operates in FIPS mode. They only interact in that if you used the -fips140 build flag we assume you want a GODEBUG default of on instead of off.

Are there any values of the -fips140 that would set the GODEBUG to enforce?

No, I expect enforce to be less popular, and applications can opt-in with the regular GODEBUG mechanisms.

The interaction between the -fips140 flag and the GODEBUG seem pretty complicated. Do we actually need the GODEBUG? Is there a significant need for changing that at runtime rather than compile time?

We've heard from a few folks that would like to make a single build and then choose to operate it in FIPS mode or not at runtime. The benefits of GODEBUG=fips140=off are faster crypto/rand and faster key generation, because FIPS 140-3 has some pretty heavy (and not generally useful) requirements on those. Another nice property of this design is that any Go binary can be operated in FIPS mode by just switching the GODEBUG at runtime, if your compliance regime is ok with "updated" modules.

This aspect is import to us at Red Hat for how we build and ship our binaries. Likely, we will still have to carry a patch downstream because we prefer detection to be automatic (e.g. if the host is in FIPS mode, the binary should automatically operate in FIPS mode). However, having the mechanism be chosen at runtime from the initial design will make it easier to implement and carry this specific patch on our end. Additionally, overall it simplifies building and distributing binaries with FIPS requirements such that it won't require 2 builds for each component (FIPS / non-FIPS).

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.

crypto/internal/fips will not be listed in go.mod or participate in module version selection; the -fips140 build flag is the only way to choose the version.

I am not sure I understand the motivation behind this. Why would this pseudo-module be so special compared to other modules? My naive take is that the decision to use fips (and which version) should be checked into a file like go.mod or go.work somewhere. Is there a compliance reason?

@timothy-king Are you suggesting that the possible values of the -fips140 option should instead appear in a go.mod file?  It would still be a special case, because the actual source code lives in the standard Go repo.  That might mean that everything that deals with go.mod files would have to know about it.  Is there a big advantage one way or the other?
@adonovan Maybe this question doesn't make sense, but if something like Bazel uses go/packages uses "go list", how do pass a -fips140 option down to "go list"?

Are you suggesting that the possible values of the -fips140 option should instead appear in a go.mod file?

I am not yet suggesting this. I am trying to understand why the pseudo-module selection is not more like a normal module selection. Why not write require crypto/internal/fips v1.24.0-fips.1 or something similar in a go.mod?
Given the proposed flag, I presume that this cannot work for some reason beyond how it is distributed/where it is fetched from. But I don't understand this reason yet.

I am not yet suggesting this. I am trying to understand why the pseudo-module selection is not more like a normal module selection. Why not write require crypto/internal/fips v1.24.0-fips.1 or something similar in a go.mod?

Because this is not the kind of requirement that makes sense for a single dependency or package to enforce. For example some people need to build Kubernetes with FIPS but most people don't. It doesn't make sense to put this requirement in Kubernetes's go.mod. Even among the people who want Kubernetes with FIPS, the specific version of the FIPS code (inprocess vs certified vs latest) that those people want will differ from context to context (mostly company to company), based on their chosen compliance levels. It's very much a build-time decision not a global dependency decision.

@adonovan Maybe this question doesn't make sense, but if something like Bazel uses go/packages uses "go list", how do pass a -fips140 option down to "go list"?

@ianlancetaylor They would set packages.Config.BuildFlags to include -fips140=whatever, or they could set GOFLAGS=-fips140=whatever, or they could even do go env -w GOFLAGS=-fips140=whatever.

@seankhliao

https://github.com/golang/go/issues/30241 was an accepted but unimplemented proposal for standard library vendoring, would it make sense to have the fips module work like that?

I don't think so. We don't want people to be forced into vendoring just to use FIPS, and vendoring answers the question of "where does the code come from during the build" but not "where does the code come from when being copied into the vendor directory?"

Some of @ianlancetaylor's 5 questions have been answered already, but for completeness I'll answer them all in one message here.

At this point there are no FIPS versions in process or certified. So presumably -fips140=inprocess and -fips140=certified will fail.

Yes, right now. Filippo is hoping to have a snapshot checked by the lab ahead of Go 1.24.0, so that in Go 1.24.0 the -fips140=inprocess flag will work. If not, then it will work in some future patch release once the snapshot does get checked. Similarly, when at some point NIST certifies a snapshot, we can issue a patch release adding a value for certified.

You say that go list or x/tools/go/packages will see module information for crypto/internal/fips. For go list that will presumably be based on the -fips140 command line flag. How will x/tools/go/packages determine the FIPS version to use?

They can use packages.Config.BuildFlags or packages.Config.Env if the tool they are using lets them set those. Otherwise they can set GOFLAGS in their environment before invoking the tool, or they can use go env -w.

I imagine that organizations will want to ensure consistent use of a FIPS version across their builds. Today they can get boringcrypto with a single consistent environment variable GOEXPERIMENT. I guess the corresponding environment variable would be GOFLAGS=-fips140=certified. But how will that be applied to x/tools/go/packages? Or (perhaps the same question) for Bazel or other build systems?

For x/tools/go/packages, because it invokes go list, that environment variable will have the desired effect.
There will need to be additional Bazel work, outside the scope of this proposal, to implement something like the -fips140 flag in Bazel. Specifically, the Bazel Go rules will need to be updated to have some workspace-global config knob and then adjust where the crypto/internal/fips packages come from while building the standard library. At the moment, Filippo does not have any clients who care about Bazel, so I think we can defer that work. (For Blaze inside Google, there will be different work, but it is straightforward.)

It sounds like the certification process is going to be based on a particular version of the source code. What will be the schedule for further certifications? How will this affect ordinary maintenance of the crypto/internal/fips packages? Should we do typo fixes for comments? Should we do routine updates to use new standard library functionality, such as the recent changes from sort.Sort to slices.SortFunc (for cases with special-purpose implementations of sort.Interface)?

Filippo mentioned that he plans to kick off new certifications about once a year. The process for that is to make a new snapshot zip file in $GOROOT/lib/fips140 and then send it off. The certification will only apply to that zip file, not to the actual source tree, so the kinds of changes you listed are always fine to make at any time. Those kinds of cleanups would only affect -fips140=latest builds, which are completely unvalidated except for being derived from an earlier snapshot that has been validated. (That's the goal at least; right now s/has been/will have been/ or something confusing like that.) API changes in that tree are a different matter; see next answer.

When using Go 1.25, will it be possible to use -fips140=v1.24.1-fips.1? (That is, can 1.25 use FIPS for 1.24.) When using Go 1.25.3 will be possible to use -fpis140=v1.25.1-fips.1? (That is, can 1.25.3 use FIPS for 1.25.1.)

When using a particular Go toolchain, say 1.25.0, it will only be possible to use the FIPS module snapshots in that toolchain's $GOROOT/lib/fips140. This limits the amount of combinatorial testing necessary for a given toolchain: we only guarantee that a particular Go toolchain works with the FIPS snapshots it ships with. The plan is in the steady state to always ship with the latest possible certified snapshot and the latest possible in-process snapshot. We may also include a latest certified+patched snapshot and latest in-process+patched snapshot, if there are critical security fixes that should be applied to a given snapshot. (For compliance reasons, some people will be able to use those fixes while others may not.)
Needing to keep some earlier snapshots working limits the amount of API changes that can be made to the API in crypto/internal/fips that is used by the rest of the crypto subtree. But changes can be made with appropriate planning, since we don't have to keep all earlier snapshots working, just the ones we choose to keep working.

@rsc I don't think replace directives allow the ecosystem as a whole to manipulate the setting, that's why I suggested them.
They allow a specific module to change the default setting for it's binaries if you build directly from that module, but it is still trivial for users to override that and it libraries can't do anything to modify users. I think it gives you all the features you really need with very few downsides and no need for any new mechanisms.
It would also allow people to use implementations not shipped in the standard library if they really need.
I don't see yet how picking a different fips implementation is really any different to any other "long term fork" choice that users might want to make when building a binary.

It would also allow people to use implementations not shipped in the standard library if they really need.
I don't see yet how picking a different fips implementation is really any different to any other "long term fork" choice that users might want to make when building a binary.

I see making it easy to switch FIPS modules as an anti-goal, and it's why I am happy that only the ones that ship with the toolchain in lib/fips140 would be available.
The API surface between the standard library and the FIPS module is not something I want to expose, because I want us to be able to evolve it over time (albeit slowly), or even make assumptions based on undocumented properties sometimes.
Exposing it as a user-configurable replace creates an expectation that it can be, well, replaced, no matter how we document it.
Sure, downstreams can also fork the toolchain (and we recommend that in some cases), but by then they know what they're in for, and they have the control to keep up.

Deciding which FIPS compliance level you are building for is in many ways analogous to deciding which operating system or architecture you are building for. A generic Go program like gofmt, say, can be built for Windows or for Linux or for FreeBSD, for x86-64 or for arm64 or for mips64. We don't put that information in a go.mod, nor do we require people building gofmt to write a special go.mod or go.work just to configure their build target.
That analogy of course prompts the question of why to use a command-line flag vs an environment variable like GOFIPS (analogous to GOOS and GOARCH). We could plausibly do that instead, and if we took that approach, we'd probably want it to get "baked in" to the toolchain during make.bash, so that everything built with that toolchain defaulted to that GOFIPS setting. The command-line flag seemed clearer since it avoids more latent configuration state in the environment or baked into a toolchain, but the environment variable is fine too. I don't feel very strongly one way or the other.
I do feel strongly that this doesn't belong anywhere near go.mod or go.work, though. It's not that kind of information.

The motivation for the pseudo-module is to have a "dep" line in 'go version -m' that makes govulncheck work unchanged as far as understanding whether a given vuln in this code applies to a given binary or build. If we drop that 'dep' line, we would instead need to add a new field to debug.BuildInfo (or else a new BuildInfo.Setting) that lists the FIPS code version, and then govulncheck would need to add special logic to look for and understand that version and use it when deciding whether a particular vuln is present. The vuln format itself would also need to add a separate field for the FIPS version, like it has for GOOS and GOARCH, and all tools consuming the vuln format would need to handle that specially. Govulncheck can be updated easily; other scanners will probably just have another source of false positives. We can certainly take that approach.
The changes in the go command are really the same either way; it's just a question of what info is shown to govulncheck and other tools.

Yeah, I understand, but I don't think it really helps.
If we make it look like a module but not be one, then things like govulncheck need to be taught all about it anyway, because it has to understand that it is not a module that obeys normal versioning rules and it can't try to work out how to upgrade to fix etc.
I think we would need to think through some examples of what it would look like to add this fips specific vulnerability information to the vulndb, I am not convinced that it all works out cleanly.

We could plausibly do that instead, and if we took that approach, we'd probably want it to get "baked in" to the toolchain during make.bash, so that everything built with that toolchain defaulted to that GOFIPS140 setting. The command-line flag seemed clearer since it avoids more latent configuration state in the environment or baked into a toolchain, but the environment variable is fine too.

I was going to chime in that in my opinion it feels like what is being proposed is an implicit go generate crypto/internal/fips that selects and unpacks the zip, this creates an std that is FIPS compliant, it then compiles with that std and the GODEBUG set, and then undoes the go generate step.  I think this way of looking at it is compatible with what you are saying about the changes being "baked in" to the toolchain (- the undoing the go generate). This does suggest to me that what needs to be written down is a new field in debug.BuildInfo that std is "v1.X.Y-fips.N" and not the default one for GoVersion "v1.X.Y".
Would we ever want to allow other cases of [temporarily] patching in alternative std package implementation at the command line like this?

@timothy-king

This does suggest to me that what needs to be written down is a new field in debug.BuildInfo that std is "v1.X.Y-fips.N" and not the default one for GoVersion "v1.X.Y".

This is only partly correct, since it mis-versions everything outside of crypto/internal/fips. It will be common with the 'certified' setting to have things like std being v1.25.0 and crypto/internal/fips being v1.24.0-fips.1. Note 25 vs 24. Or even v1.24.3 std with v1.24.0-fips.1 crypto. There really are two different versions to track.

Would we ever want to allow other cases of [temporarily] patching in alternative std package implementation at the command line like this?

Maybe, maybe not, but we're explicitly not designing a general mechanism today. (We started down that path a while back and decided there were so few uses it would not be worth the hassle. Even back then, FIPS was the only use we could think of.)

Spoke to @FiloSottile at GoLab, and we are happy to take @ianthehat's suggestion and make this a setting like GOOS and GOARCH. Specifically, the new proposal is:

There is a new target configuration environment variable, GOFIPS140=off/latest/inprocess/certified/v1.X.Y-fips.N, default off, that has the effects previously contemplated for the -fips140 command-line flag.
Like all the other target configuration variables (for example, GOOS, GOARCH, GOARM, CC, CGO_ENABLED, ...), the GOFIPS140 setting during make.bash is "baked" into the toolchain as the default setting for builds with that toolchain. The default for standard Go toolchains will be "off".
When you run go version -m or fetch a debug.BuildInfo, there will be a new BuildSetting with key GOFIPS140 and value "off" or "devel" or else a version. (That is, instead of "inprocess" you would see the specific version that "inprocess" resolved to. For "latest" you would see the Go version if this is a release, or else "devel" if that's the Go version.)
There is no user-visible API or tooling that would refer to the crypto/internal/fips code as a module.
When you run go list, packages in crypto/internal/fips/... may have a Dir in $GOROOT/src/crypto/internal/fips/... or they may come from directories in the the module cache. Either way, they will have a nil Module field, matching the rest of the standard library.
While crypto/internal/fips does have a set of versions in semver syntax, they are entirely separate from the Go module system.
The files in lib/fips140 will have an implementation-defined format. That format will coincidentally use the module zip and checksum file formats, but that is an implementation detail.

Govulncheck will need changes to handle the GOFIPS140 metadata, much as it already has special cases for GOOS and GOARCH.
The list of module special cases are gone; nothing will ever trip over them.
Separate from the module (non-)presentation, the one new benefit of this approach is the separate environment variable. Build containers can export GOFIPS140=inprocess and be sure that no code will accidentally change that elsewhere in the build because they want to set something unrelated in GOFLAGS. The "baking in" also allows preparing an alternate go command binary that always assumes a specific GOFIPS setting, although I expect that not to be used very often.

Looking at the implementation, if we're not going to use modules, then the best place to hook in will be the old vendor import rewriting. The rule will be that when GOFIPS140=v1.2, then the import crypto/internal/fips/any rewrites internally to crypto/internal/fips/v1.2/any. The sources for that package will be found in $GOMODCACHE/fips140/v1.2 (unpacked there if needed).
This rewriting will mean that the FIPS v1.2 symbols will be in the binary as crypto/internal/fips/v1.2/any.Symbol, while the "latest" symbol crypto/internal/fips/any.Symbol will not be in the binary. So a vulnerability report about code in scope for FIPS will need to list the different FIPS variants that are affected as well. But then govulncheck itself needs no updates at all, because the different versions have different symbol names.

@rsc, what's the documentation plan for this?

I was going to document GOFIPS140 at the top of make.bash and in any other places where GOOS, GOARCH, GOAMD64 are documented (I can't find exactly where GOOS and GOARCH are documented today. Do you remember?)
There should also be a go.dev/doc/fips140 or perhaps go.dev/security/fips140 that gives the bigger picture overview.

GOOS and GOARCH are documented, among other places, at https://pkg.go.dev/runtime from runtime/extern.go.

The proposal committee's main sense is just that this all seems complicated. On the other hand, based on my understanding, I think this is as complicated as necessary to support the needs of FIPS. This is also, to a significant extent, an expert interface. If you have to use FIPS mode, you're going to take the time to figure out what you're doing.

Have all remaining concerns about this proposal been addressed?
The proposal details are in https://github.com/golang/go/issues/70200#issuecomment-2468562595 (and the meanings of the GOFIPS140 values are in https://github.com/golang/go/issues/70200#issue-2635884033). The documentation plan is in https://github.com/golang/go/issues/70200#issuecomment-2474231803.

@ianlancetaylor

If I understand correctly, you are suggesting that setting GOFIPS140=off when building a program means "build with $GOROOT/src/crypto/internal/fips and act as though the program specifies //go:debug fips140=off in the main package.

Yes, and that is the default.

When running make.bash, not setting GOFIPS140 at all is exactly the same as setting GOFIPS140=off.

Yes.

When not running make.bash, not setting GOFIPS140 at all means to use the value used when make.bash was run.

Yes.

If GOFIPS140 was not set when running make.bash, or if GOFIPS140=off was set when running make.bash, and GOFIPS140 is not set when building a program, then we build with $GOROOT/src/crypto/internal/fips and act as though the program specifies //go:debug fips140=off in the main package.

Yes.

At run time in all cases we can set GODEBUG=fips140=on to get whatever version of fips140 was used to build.

Yes. It changes various behaviors, not the cryptographic code itself. It will mean that, as an incomplete list, (1) the init-time fips code+data hashing check runs, (2) crypto/rand.Reader runs the fips randomness generator, which seeds itself from /dev/random but then does more instead of using those bytes directly, and (3) crypto/tls changes its default permitted algorithms to be from the fips allowlist.

@aclements

The proposal committee's main sense is just that this all seems complicated. On the other hand, based on my understanding, I think this is as complicated as necessary to support the needs of FIPS. This is also, to a significant extent, an expert interface. If you have to use FIPS mode, you're going to take the time to figure out what you're doing.

I agree with this, although I think the new approach is less complicated than the module approach; thanks to @ianthehat in particular for that pushing back on that.

I agree, I think reading the whole conversation it all feels very complicated, but the actual final solution is actually only moderately complicated, no more so than it needs to be, and much cleaner than some of the intermediate steps might have you believe. Thanks for taking in to account my point of view, I am happy with where we ended up!

Based on the discussion above, this proposal seems like a likely accept.
The proposal is to add a new GOFIPS140 target configuration environment variable that controls the source used for the crypto/internal/fips package and the default value of the fips140 GODEBUG. It can take the following possible values:

GOFIPS140
crypto/internal/fips source
Default fips140 GODEBUG

off
$GOROOT/src/crypto/internal/fips
off

latest
$GOROOT/src/crypto/internal/fips
on

vX.Y
$GOROOT/lib/fips140/vX.Y.zip
on

inprocess
Use vX.Y listed in $GOROOT/lib/fips140/inprocess.txt
on

certified
Use vX.Y listed in $GOROOT/lib/fips140/certified.txt
on

Note that the definitions of inprocess and certified are specific to the Go toolchain being used, to keep builds reproducible. (That is, NIST issuing a certification for a new version does not change the meaning of GOFIPS140=certified in an older Go release.)
Like all the other target configuration variables (for example, GOOS, GOARCH, GOARM, CC, CGO_ENABLED, ...), the GOFIPS140 setting during make.bash is "baked" into the toolchain as the default setting for builds with that toolchain. If unset, the default for standard Go toolchains will be "off". The baked-in default can be overridden by setting GOFIPS140 to a non-empty value when invoking go build or related commands.
When you run go version -m or fetch a debug.BuildInfo, there will be a new BuildSetting with key “GOFIPS140” and value "off" or "devel" or else a version. (That is, instead of "inprocess" you would see the specific version that "inprocess" resolved to. For "latest" you would see the Go version if this is a release, or else "devel" if that's the Go version.)
There is no user-visible API or tooling that would refer to the crypto/internal/fips code as a module.
When you run go list, packages in crypto/internal/fips/... may have a Dir in $GOROOT/src/crypto/internal/fips/... or they may come from directories in the module cache. Either way, they will have a nil Module field, matching the rest of the standard library.
While crypto/internal/fips does have a set of versions in semver syntax, they are entirely separate from the Go module system.
The files in lib/fips140 will have an implementation-defined format. That format will coincidentally use the module zip and checksum file formats, but that is an implementation detail.
GOFIPS140 will be documented at the top of make.bash, in runtime/extern.go and in any other places where GOOS, GOARCH, GOAMD64, etc are documented. There will also be a go.dev/doc/fips140 (or perhaps go.dev/security/fips140) that gives the bigger picture overview.

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
The proposal is to add a new GOFIPS140 target configuration environment variable that controls the source used for the crypto/internal/fips package and the default value of the fips140 GODEBUG. It can take the following possible values:

GOFIPS140
crypto/internal/fips source
Default fips140 GODEBUG

off
$GOROOT/src/crypto/internal/fips
off

latest
$GOROOT/src/crypto/internal/fips
on

vX.Y
$GOROOT/lib/fips140/vX.Y.zip
on

inprocess
Use vX.Y listed in $GOROOT/lib/fips140/inprocess.txt
on

certified
Use vX.Y listed in $GOROOT/lib/fips140/certified.txt
on

Note that the definitions of inprocess and certified are specific to the Go toolchain being used, to keep builds reproducible. (That is, NIST issuing a certification for a new version does not change the meaning of GOFIPS140=certified in an older Go release.)
Like all the other target configuration variables (for example, GOOS, GOARCH, GOARM, CC, CGO_ENABLED, ...), the GOFIPS140 setting during make.bash is "baked" into the toolchain as the default setting for builds with that toolchain. If unset, the default for standard Go toolchains will be "off". The baked-in default can be overridden by setting GOFIPS140 to a non-empty value when invoking go build or related commands.
When you run go version -m or fetch a debug.BuildInfo, there will be a new BuildSetting with key “GOFIPS140” and value "off" or "devel" or else a version. (That is, instead of "inprocess" you would see the specific version that "inprocess" resolved to. For "latest" you would see the Go version if this is a release, or else "devel" if that's the Go version.)
There is no user-visible API or tooling that would refer to the crypto/internal/fips code as a module.
When you run go list, packages in crypto/internal/fips/... may have a Dir in $GOROOT/src/crypto/internal/fips/... or they may come from directories in the module cache. Either way, they will have a nil Module field, matching the rest of the standard library.
While crypto/internal/fips does have a set of versions in semver syntax, they are entirely separate from the Go module system.
The files in lib/fips140 will have an implementation-defined format. That format will coincidentally use the module zip and checksum file formats, but that is an implementation detail.
GOFIPS140 will be documented at the top of make.bash, in runtime/extern.go and in any other places where GOOS, GOARCH, GOAMD64, etc are documented. There will also be a go.dev/doc/fips140 (or perhaps go.dev/security/fips140) that gives the bigger picture overview.

This is mostly done. The only thing left to do will be freezing the zip file of v1.0 once we reach source freeze with the lab, and populating inprocess.txt once the lab signs off on it. Hopefully these will both happen before Go 1.24.0, but updating inprocess.txt may also slip to a minor version.


Bot doesn't leave channel

When using the 'play' command to listen to a song from YouTube, once the song has finished (or the queue is empty) the bot will just idle in the voice channel and not part.
If another user in a different voice channel requests a song, the bot will not move the channel where the user requested the song from.
If the bot is idling in VoiceChannel1 and someone in VoiceChannel2 attempts to move the bot by using the 'summon' command, the error is shown:
justin: >summon
BOTS: An internal error occured while responding to >summon!
ClientException: Already connected to a voice channel in this server

I just tested this in my production (public) copy, it appears to work (the moving channels). Maybe I fixed it recently during the voice revamp. This is why you update your bot often with the >update command 😉
As for the idling, that's a planned fix and should come tomorrow.

The switching of channels is working with a fresh update of the bot, thanks for that.
Just the bot idling in the channel is still a bug

Any updates on getting the bot to leave the voice channel when the queue is finished?

Yes, it's not immediate but should do it within 5 minutes.

Unfortunately with release v2.2.1 it seems to be indefinite

The releases are terribly outdated. Just use master.

Thanks, I'll migrate to the master branch.


Hang on render

Sometimes Fl Studio can be prevented from successfully finishing the render of a track. Possible reasons for this can include time markers that tell FL Studio to skip or pause, and also missing/demo plugins and samples. In the case of an infinite loop, the chunk number will count upward to a certain point until it reaches the loop, in which case, the number just goes back down and starts again. This result of this is that a small number of files, that should be manageable and finish quickly will take an indefinite amount of time to finish (several hours or until the "abort" action is done).
Possible solutions:

Allow the user to set a time limit on the FL rendering process (i.e. one hour)
Use another library to inspect and modify the .flp files to find and fix possible errors
Monitor the window title of FL Studio to tell if the rendering is being blocked or looped (did it get to "Rendering: X/Y", yet? did the number start over?).
Use robot.js to check for errors, dialogs, etc. that can stall the process

Alright, interesting. I'll notify Image Line about the weird skip marker behavior, as this also happens when rendering manually.
Regarding demo plugins and other warning messages, I think electronjs can handle screen-grabbing (doesn't even need robot.js, so that's a win). A timeout also seems nice.
I didn't notice FL's window title changes. That just sparked some ideas, too!
Extracting the actual .flp files seems a bit too hacky and risky, because Image Line probably doesn't want users to reverse engineer their products.
Thank you for the suggestions. I'll try some things!

So, a timeout for warning messages has been implemented. It's set to 2 minutes by default.
Maybe I'll add some user interface for adjustment in the future.

There now is another timeout for maximum render duration, set to 45 minutes by default.
Also, a small workaround handling looping skip markers.
As soon as I confirm those changes on MacOS, I'll declare this solved.

Nice work! Is there a way, though, to automatically click through the pop-ups so that they close? Pre-processing of the file could also accomplish this (in a fairly reliable way). Parsing/modifying an .flp file does not constitute as "reverse-engineering" a program.  There are already libraries to do this. This is no different than using a library to modify a Word document because it won't open.
Here is a Node.js library for working with .flp’s: https://github.com/b15h0y/flp-parser, and here is a C++/ .Net Library based on the node.js library (but with better documentation) https://github.com/monadgroup/FLParser.
I’ll have to try out these changes sometime, but as of now, I am all rendered up 😉
BTW, if a file gets aborted, canceled, or has some other issue, does this information get stored anywhere? It should.

Hey there, just wanted to say that the skip marker bug has been resolved in FL Studio 20.7.3, as mentioned in its changelog:

Bug fixes:
[...]
9377 Audio export freezes when using skip markers


Fix sub folder in repository missing add file dropdown

In repository sub folder missing add file dropdown menu, Probably broken since #20602

I remember specifically fixing this in that PR, need to double-check whether this fix is correct.

Attachment the issue screenshot

Make lgtm work

Please send backport


SimpleSpannerRepository.findAll(Pageable pageable) not returning the correct size and page

I looked in the code of SimpleSpannerRepository for the findAll(Pageable) method which led me to the SpannerTemplate. queryAll(Class<T> entityClass, Pageable pageable):
@Override
public <T> Page<T> queryAll(Class<T> entityClass, Pageable pageable) {
	return queryAll(entityClass, pageable, null);
}

As you can see this method sends a null as parameter where a SpannerQueryOptions object is expected, and thus when it enters the inner queryAll method it fails to read the size and page to be returned.
The inner queryAll implementation is:
@Override
public <T> List<T> queryAll(Class<T> entityClass, Sort sort, SpannerQueryOptions options) {
	Assert.notNull(sort, "sort must not be null!");

	StringBuilder stringBuilder = new StringBuilder();
	SpannerPersistentEntity<?> persistentEntity = this.mappingContext.getPersistentEntity(entityClass);
	stringBuilder.append("SELECT * FROM " + persistentEntity.tableName() + " ");
	SpannerStatementQueryExecutor.buildOrderBy(persistentEntity, stringBuilder, sort);
	if (options != null) {   <---- THIS IS WHERE THE SIZE AND PAGE ARE LOST BECAUSE THE NULL SENT BEFORE
		if (options.hasLimit()) {
			stringBuilder.append(" LIMIT " + options.getLimit());
		}
		if (options.hasLimit()) {
			stringBuilder.append(" OFFSET " + options.getOffset());
		}
	}
	stringBuilder.append(";");
	return query(entityClass, Statement.of(stringBuilder.toString()), options);
}

This null argument blocks from getting the correct subset of records.
The temporal solution I used was to build a instance of SpannerQueryOptions with the fields limit and offset with the required values and use SpannerTemplate.queryAll(Class<T> entityClass, Pageable pageable, SpannerQueryOptions options) instead.

I just saw that you fixed and merged the bug into master branch, I'm wondering if this fix would be available in the 4th milestone or I simple have to update the snapshot for the dependency in maven? Thanks.


Can I ask how to localize the language？

I can't find the language file

Currently the code is not setup for localization. This would be something that would need to be done before any language localizations are done.

So he can only speak English?

The trainer application is only in English that's correct.


Question related testing TMO using new dataset

Hi, sorry for disturbing you again and again. I wanted to ask you that if I want to test the performance of TMO model on new dataset(For example ultrasound videos) do I need to write new test code for it or I can use the codes of test_davis, test_fbms or test_ytobj? I am really confused about it, because format of ultrasound videos are different from real world videos.
I would be really grateful if you give me some instructions or clues about it.

Please, can I use code of " if mode == 'val'" in trainer.py file to calculate F and J for new dataset? even can I use this code for evaluating TMO on DAVIS 2016?(I know how to evaluate TMO on DAVIS 2016 using its official code, but I want to know what about using code of  if mode == 'val' in trainer.py )

I am really sorry for disturbing you so much, the reason is that I really found TMO interesting so that I want to use it for Ultrasound videos.

In my ultrasound dataset the annotations are normalized by dividing by 255, so I am thinking if I do not change the annotations to original format, will the test_davis work properly while inferring? I mean if I simply put my ultrasound dataset instead of DAVIS(Making the same directories), will the code of preprocessing dataset, the code of evaluating model and the code of if mode == 'val' in trainer.py work properly or I must change the annotations back to original format(The pixel values should be between 0 and 255 instead of being between 0 and 1)
One confusing thing is that, I tested the code of if mode == 'val' in trainer.py for calculating J and F values, the result is a bit different compare to the DAVIS 2016 official code. For example when I check my results on DAVIS 2016 official code, the J values is 85.6, but when I test the same results using code of if mode == 'val' in trainer.py, the J value is 85.5.(The difference is very small).

I will be very happy to hear about this issue from you.

Thank you so much for responding,
You mean I should make the background pixels [0,0,0] and the foreground pixels value [128,0,0] for annotations only, right?

Yes, just make sure that the annotations have the pixel values of either [0,0,0] or [128,0,0].
I recommend you to check the pixel values and modify them if needed.
I cannot tell if it works because I didn't see your data.

How can I check what are the annotations pixel values? even I tried to find what are the annotations pixel values of DAVIS 2016, I could not find a trusted method.

Hi, you can check the pixel values by converting PIL images to tensor format.

I converted the PIL images to numpy array and checked the pixel values, is it correct?


test: add environment check for ERC1155 mintToBatch tests

Problem solved
Short description of the bug fixed or feature added

PR-Codex overview
This PR refactors the test for the mintToBatch function in the ERC1155 Edition module, enhancing its structure and clarity. It modifies the test to run conditionally based on the presence of a secret key and organizes the minting logic more effectively.
Detailed summary

Changed describe to run conditionally with process.env.TW_SECRET_KEY.
Improved organization of the test case for mintToBatch.
Added account to the contract deployment parameters.
Consolidated the minting transaction logic.
Retained the NFT expectation check after minting.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

#5115  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @joaquim-verges and the rest of your teammates on  Graphite

Merge activity

Oct 21, 7:13 PM EDT: A user  merged this pull request with the Graphite merge queue.


Added Binary_tree_is_symmetirc_or_not.cpp

Solution to check whether a given binary tree is symmetric tree or not.

@abhi-shek421 What do I need to fix? Are you talking about the driver code?

@ct-27 Yes add driver code.

Done @abhilashmnair. Please check.

Thank you for your contribution @ct-27


Change twitter.nim owner

The current owner did not respond to my PR to add support for uploading media to twitter for 30 days and I have added a bunch of procs for doing this, his bio also says 'not hosting on github'
I don't really want to step on anyone's toes here but it seems like the default 'twitter' package should at least support requests to the media endpoint.

@dom96 opinion?

Seems reasonable. If @dchem wants it changed back we should allow it though.

Might want to give him a ping on<EMAIL_ADDRESS>

Yeah as I said not wanting to step on toes but the library is missing what I would consider core functionality for something that is called 'twitter' in the repo. I'd quite like to see it even if it's not my code in the repo (have been my forked version for: https://github.com/snus-kin/flickr-image-bot).

Seems reasonable. If @dchem wants it changed back we should allow it though.

Merging. Will revert if necessary (doesn't seem likely).

Apologies for late reply. I have been unfortunately focused on other projects. No objections. Thank you.


docs: useradd issue

What happened?
I'm currently working through https://frankenphp.dev/docs/docker/#running-with-no-capabilities and see the line useradd -D ${USER}; mentioned several times.
Following through the example:
docker run -it --rm dunglas/frankenphp bash
export USER=www-data
useradd -D ${USER}

just prints the commands help.
-D for useradd is print or change default useradd configuration.
-D for adduser is Don't assign a password
From my understanding the intention is to add a user www-data without a password on debian? That would be useradd without specifying -p. But even then
cat /etc/passwd | grep www-data

show the user already exists, so can it be omitted because it serves nothing?
Build Type
Docker (Debian Bookworm)
Worker Mode
Yes
Operating System
GNU/Linux
CPU Architecture
x86_64
PHP configuration
The one installed in the docker image dunglas/frankenphp

Relevant log output
No response

In addition, in the upstream image, the home folder for the user www-data is configured to be /var/www:
cat /etc/passwd | grep www-data
> www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin

but the user has no access to it, as everything is owned by root
ls -ldh /var/www
drwxr-xr-x 3 root root 4.0K Nov 21 17:54 /var/www

Any reason why /app is preferred over `/var/www'?

You're right, www-data already exists in all the official php images. I'd go a step further and change the user id of www-data to 1000 like this image does (so file permissions are correctly mapped when using the image locally).
As for /app vs /var/www, I think /app is a container convention. Before containers, application files usually were just copied to a permanently running server, in other words they were 'variable' and therefore put into the var folder. Containers are usually deployed as a whole and the application files aren't changed afterwards.

I found the -D issue in the docs and made a PR to fix it here: #1243
I think that's helpful regardless of the existing www-data account, because people might want to use other named accounts. Maybe the example name in the docs should be changed though.


Match output of catch2 when listing tests with default reporter

My team uses Clion and while transitioning our libraries to snitch from Catch2 I noticed CLion choking on the tests. Clion has built in Catch2 support, and in the readme snitch claims to support Clion via this.
I modified the snitch main function to write the value of argv to a file, and saw that clion calls the test binary twice:

before running tests with --list-tests --order lex ~[.]
run the tests with -r xml -d yes --order lex

I compared the output with the list tests command between the latest Catch2 and snitch. I copied a representation of the difference at the bottom of this description. But the key differences are:

tests aren't ordered
no header text "Matching test cases:"
no final "132 matching test cases"
test names aren't indented
no tags output for each test

Of these items Clion at least only seems to care about 2, 4, and 5.
Adding the test count at the end doesn't matter to Clion and would be a much larger change. I have two branches in my repo here I attempted it and it was either:

Static variable in the default reported struct. Which isn't very nice.
change default reporter to a struct with a counter variable as member. A large change that I didn't actually get to work.

Catch2
$ unit-tests-catch2.exe --list-tests --order lex ~[.]
Matching test cases:
  Asserts and contracts
      [assert]
  COBS framing
      [cobs][ranges]
  COBS unframing
      [cobs][ranges]
  CRC Algorithm
  GPS <-> UTC Conversions
  block_allocator
      [blocks][memory][pool]
  block_allocator custom alignment
      [blocks][memory][pool]
  inplace_function
      [functional][larid]
  is_scoped_enum
      [c++23][type_traits]
  memory_pool_external
      [memory][pool]
  memory_resource
      [larid][memory_resource]
  ........... many more tests I won't waste space on .......
132 matching test cases

No test listed
$ unit-tests-catch2.exe --list-tests --order lex [.]
Matching test cases:
0 matching test cases

Snitch
$ unit-tests-snitch.exe --list-tests --order lex ~[.]
warning: unknown command line argument '--order'
COBS framing
COBS unframing
basic_netbuf
empty ispanstream
non-empty ispanstream
empty istreambuf_iterator
empty ostreambuf_iterator
memory_resource
block_allocator
block_allocator custom alignment
memory_pool_external

Thanks for reporting this. We can merge this PR without the "X matching test cases", and leave that for a subsequent PR.

change default reporter to a struct with a counter variable as member. A large change that I didn't actually get to work.

This would be the correct approach. Do you want to pursue it, or would you rather I take it from here? If you want to keep at it, feel free to open a draft PR and we can provide some feedback along the way.

This would be the correct approach. Do you want to pursue it, or would you rather I take it from here? If you want to keep at it, feel free to open a draft PR and we can provide some feedback along the way.

I will open a draft PR with what I have. I think it is 95% done and I will point out where I got stuck in the PR.


Xstate with typescript: state.value is always reported as a string even if it can be an object

Description
When using Xstate with React I sometimes find the need to pass properties of state.value when it is an object to child components. There are no problem doing this in javascript with nullchecks, but I can't manage to get Typescript to allow this.
The Typescript compiler gives the error: Property 'X' does not exist on type 'string'.
Expected Result
When typing the state like this:
interface LightStateSchema {
  states: {
    green: {};
    yellow: {};
    red: {
      states: {
        walk: {};
        wait: {};
        stop: {};
      };
    };
  };
}

I think code like this should work:
{state.value.red && <RedComponent redState={state.value.red} />}

Actual Result
Instead typscript gives the error:
Property 'red' does not exist on type 'string'
Reproduction
Here is a Codesandbox which more or less just use the typescript example from: https://xstate.js.org/docs/guides/typescript.html
On line 104 typescript gives the error:
https://codesandbox.io/s/intelligent-water-81rzj?file=/src/App.tsx:2061-2132
Additional context
Xstate 4.9.1
@xstate-react 0.8.1
Not sure if this is a bug in the Xstate typescript implementation, or if I havn't understood how to type the state in Xstate.

I am just curious what is your use case behind passing pretty much internal structures of the machine around. I mean with your interface, only one property will be ever-present and you would have to do checking which one. Doesn't that kinda beat the purpose of the state machine?
interface RedComponentProps {
  redState: {
    walk: {};
    wait: {};
    stop: {};
  };
}

What about passing a whole service instead and then with useService in child component you can easily do current.matches and work with it as intended.
I don't know, just my opinion as it feels you are complicating it for yourself.

The most common usecase I have is when using Styled components.
Lets say I have a statemachine which describes an input (which handles focus/validity/pristine/toched states of the input):
https://xstate.js.org/viz/?gist=a0352e44d41ddcb36cd533a28720ca8b
Then I would like to pass some of these props directly to the styled components so they can set their style accordingly.
For example I might have an label (styled) component, which could reuse the same interface as the validity state:
Label

Instead of having to do something like this:
const invalid = state.matches("validity.valid");

Label

The second example would be really messy if I would like to pass a childstate that could have more than just 2 different values/states.

Two things:

Prefer using state.matches(...), your use-case is what that is for:

const valid = state.matches({ validity: 'valid' });

If you're set on using state.value directly, understand that state.value is typed to string | StateValueMap and coerce the type appropriately:

(state.value as Record<string, string>).validity


use an explicit export list in Haskell

from
https://wiki.haskell.org/Performance/GHC :
3 Modules and separate compilation

In general, splitting code across modules should not make programs less efficient. GHC does quite aggressive cross-module inlining: when you import a function f from another module M, GHC consults the "interface file" M.hi to get f's definition.

For best results, use an explicit export list. If you do, GHC can inline any non-exported functions that are only called once, even if they are very big. Without an explicit export list, GHC must assume that every function is exported, and hence (to avoid code bloat) is more conservative about inlining.

How can we check that a haskell file does NOT have an explicit export list?

I had the impression that a lot of out modules already use export lists ...

Many don't.... some do.
I want all. Unless there is a better reason.


Add stripe key

Fails

:no_entry_sign:

No CHANGELOG added.

:no_entry_sign:

[danger-plugin-jest] Could not read test results. Danger cannot pass or fail the build.

:no_entry_sign:

This PR includes changes to the Emission Pod's native code but does not have a package.json
change for the update to the "native-code-version". If this is fine, add #native_no_changes to your PR message.

  Generated by :no_entry_sign: dangerJS

I think we can close this PR in favor of https://github.com/artsy/emission/pull/1086

I'm closing this PR since I cherry-picked and added the relevant change to https://github.com/artsy/emission/pull/1086/files (commit: https://github.com/artsy/emission/pull/1086/commits/5f45ad60e0e44c0f821db30ab2c3740eb81d8bea).


AddNewAddress

Add new Google Video address starting with a5m. Previous conflicts are due to expired codes I had forked, and now it is solved.

Well, for those differences that seem not to change anything, they surely change nothing..... Perhaps it is because I open those files on Windows, which may affect end of line. Anyway, all I want to do is to add 30 new addresses, and did not mean to do others.

I checkout you changes locally and find that there are unicode characters U+FEFF before these two lines, please create another commit to remove them, and then I will merge your changes into master branch.

Which encoding style should I select, UTF-8 or ASCII?

Which two lines contains Unicode?


More consistent use of throwIf to eval exception conditions

Also note the TODO on the LocalStorageService commit - should we bite the bullet and swap that to use XH.appCode now?


Tunnel destructor prevents subsequent stderr interception

When the Tunnel.close() is called, the ioloop writer and reader instances are closed and oddly this makes subsequent stderr writes not intercepted by the orchestrator.
Eg:
# -*- coding: utf-8 -*-
import time

from chopsticks.tunnel import Local

def func():
    import __bubble__
    __bubble__.debug("Hi there!")
    time.sleep(1)  # Wait stderr to be flushed

for i in range(3):
    print("Call #%s" % i)
    m = Local()  # triggers __del__ in cpython starting 2nd iteration
    m.call(func)

outputs the following:
$ python3 test.py server
Call #0
[localhost] Hi there!
Call #1
Call #2

...no second output...

Please note that the comment
 m = Local()  # triggers __del__ in cpython starting 2nd iteration

comes from a test I originally made outside of a loop.
Anyway you get the idea.


Extending the package model to my own model

Hi there
I can see that I can define a custom model to use in the config file:
    /*
     * The model class to use when converting an incoming email to a message.
     * It must extend the default model class
     */
    'model' => \BeyondCode\Mailbox\InboundEmail::class,

Above is the default model it uses.
I have changed it to:
    /*
     * The model class to use when converting an incoming email to a message.
     * It must extend the default model class
     */
    'model' => Email::class,

Now, I wish to use my own model, because I want to store the e-mails on another table. For this, I have created my own model:
App/Email.php:
namespace App;

use BeyondCode\Mailbox\InboundEmail as InboundEmail;

class Email extends InboundEmail
{
    protected $table = 'emails';
}

However, above gives me below error:
Undefined property: App\Email::$message {"exception":"[object] (ErrorException(code: 0): Undefined property: App\\Email::$message at /Users/Username/Sites/playground/vendor/beyondcode/laravel-mailbox/src/InboundEmail.php:130)

I suspect it is because I don't have the original methods in my own model App/Email.php, that are present in the package model.
Isn't there a way that I can use my own model, by extending the package model, but without copying the content of the package model into my own?

HI @oliverbj ! I try the same. I reply the model to my own model. But by retrieving the model by Nova, i receive the same error as you. Have you find any solution?
Thanks for sharing.


Create TutorialMVVM

MVVM Tutorials have been added and can be found in the "Tutorials/MVVM" folder: https://github.com/ABTSoftware/SciChart.Wpf.Examples/tree/master/Tutorials/MVVM


createResponse on factory creates missing reasonPhrase

PSR17's createResponse
public function createResponse(int $code = 200, string $reasonPhrase = ''): ResponseInterface;

means, that there is never a null value on $reasonPhrase
this means that within the Response implementation, using the factory is always empty, cause the null check https://github.com/Nyholm/psr7/blob/master/src/Response.php#L49.
what is OK, but dangerous is also line https://github.com/Nyholm/psr7/blob/master/src/Response.php#L50 cause its checks against the property and the assignment agains the variable within the method.

I would suggest the following:
This would not only fix my issue, but also the issue given null as reason which a not supported status code, which leads to an interface implementation issue cause reasonPhrase should not return null as far as i unterstand the PSR7.
// If we got no body, defer initialization of the stream until Response::getBody()
if ('' !== $body && null !== $body) {
    $this->stream = Stream::create($body);
}

$this->statusCode = $status;
$this->reasonPhrase = (string) $reason;

$this->setHeaders($headers);

if ('' === $this->reasonPhrase && isset(self::PHRASES[$this->statusCode])) {
    $this->reasonPhrase = self::PHRASES[$this->statusCode];
}

$this->protocol = $version;


Added profiles and import/export support

Changes
The main purpose of this PR is to support multiple profiles in the wallet.
Additional changes were made to support this:

Database

Standardized entity record classes
Added ProfileRecord class
Added database migrations (to create default profile for existing users)

Redux Store

Separated credentials state from wallet state
Standardized entity slice state
Added memoized redux selectors
Added selector factories (for selecting object relationships)

Hooks

Added utility hooks useSelectorFactory, useSelectorFactoryCallback

Screens

Abstracted DetailsScreen to support multiple types of import reports
Abstracted QRScreen for adding credentials and profiles

Components

Added reusable modal variant components (RestoreItemModal/BackupItemModal)

Lib Methods

Updated import/export profile and wallet methods
Added import/export encrypted file support
Updated sharePresentation method to leave presentations unsigned (going to be updated later)
Added custom error type HumanReadableError

PT Stories

As a user I should be able to create another profile in the app
The name of the currently selected profile should show on the Credential Preview
Add profile name to “Accept Credentials” screen
Implement choose profile screen before adding credential
Add support for backing up individual profiles
Backup wallet should backup all profiles
Restore wallet should handle multiple profiles.
Rename "Add New Profile" to "Create New Profile"

See #204


support recent nightly to get better debug info on no_std issues

The following rust-lang PR should give us better debug info for duplicate lang itemissues:
Issue:
https://github.com/rust-lang/rust/issues/60561
PR:
https://github.com/rust-lang/rust/pull/64883
Can we get support for recent nightly including this PR?

Hey @brenzi , I figured out a way to use the latest compiler. Please see the branch "test-2019-10". To use that in your enclave, you can add a patch section with fixed path:
[patch.'https://github.com/baidu/rust-sgx-sdk.git']
sgx_alloc = { path = "../../../ding/rust-sgx-sdk/sgx_alloc" }
sgx_build_helper = { path = "../../../ding/rust-sgx-sdk/sgx_build_helper" }
sgx_cov = { path = "../../../ding/rust-sgx-sdk/sgx_cov" }
sgx_crypto_helper = { path = "../../../ding/rust-sgx-sdk/sgx_crypto_helper" }
sgx_libc = { path = "../../../ding/rust-sgx-sdk/sgx_libc" }
sgx_rand = { path = "../../../ding/rust-sgx-sdk/sgx_rand" }
sgx_rand_derive = { path = "../../../ding/rust-sgx-sdk/sgx_rand_derive" }
sgx_serialize = { path = "../../../ding/rust-sgx-sdk/sgx_serialize" }
sgx_serialize_derive = { path = "../../../ding/rust-sgx-sdk/sgx_serialize_derive" }
sgx_serialize_derive_internals = { path = "../../../ding/rust-sgx-sdk/sgx_serialize_derive_internals" }
sgx_tcrypto = { path = "../../../ding/rust-sgx-sdk/sgx_tcrypto" }
sgx_tcrypto_helper = { path = "../../../ding/rust-sgx-sdk/sgx_tcrypto_helper" }
sgx_tdh = { path = "../../../ding/rust-sgx-sdk/sgx_tdh" }
sgx_tkey_exchange = { path = "../../../ding/rust-sgx-sdk/sgx_tkey_exchange" }
sgx_tprotected_fs = { path = "../../../ding/rust-sgx-sdk/sgx_tprotected_fs" }
sgx_trts = { path = "../../../ding/rust-sgx-sdk/sgx_trts" }
sgx_tse = { path = "../../../ding/rust-sgx-sdk/sgx_tse" }
sgx_tseal = { path = "../../../ding/rust-sgx-sdk/sgx_tseal" }
sgx_tservice = { path = "../../../ding/rust-sgx-sdk/sgx_tservice" }
sgx_tstd = { path = "../../../ding/rust-sgx-sdk/sgx_tstd" }
sgx_tunittest = { path = "../../../ding/rust-sgx-sdk/sgx_tunittest" }
sgx_types = { path = "../../../ding/rust-sgx-sdk/sgx_types" }
sgx_ucrypto = { path = "../../../ding/rust-sgx-sdk/sgx_ucrypto" }
sgx_unwind = { path = "../../../ding/rust-sgx-sdk/sgx_unwind" }
sgx_urts = { path = "../../../ding/rust-sgx-sdk/sgx_urts" }

cheers, I'll have a look - but it will take time. Will this enter the next release?

of course! but I'm not sure if I can do this earlier. it seems that all the forked crates can build well using nightly-2019-10-04.
and I think there's probably another release around Nov. 12th.

This doesn't seem to work for me. If I build
https://github.com/scs/test-no-std/tree/substrate-primitives
with
cargo +nightly-2019-08-01 build

this goes through building sgx_tstd successfully
But with
cargo +nightly-2019-10-04 build

I get errors:
error[E0432]: unresolved import `core::prelude::v1::__rust_unstable_column`
  --> /home/abrenzikofer/.cargo/git/checkouts/rust-sgx-sdk-fc8771c5c45bde9a/166d481/sgx_tstd/src/prelude/v1.rs:52:5
   |
52 |     __rust_unstable_column,
   |     ^^^^^^^^^^^^^^^^^^^^^^ no `__rust_unstable_column` in `prelude::v1`

error[E0432]: unresolved imports `core::prelude::v1::Decodable`, `core::prelude::v1::Encodable`
  --> /home/abrenzikofer/.cargo/git/checkouts/rust-sgx-sdk-fc8771c5c45bde9a/166d481/sgx_tstd/src/prelude/v1.rs:84:5
   |
84 |     Decodable,
   |     ^^^^^^^^^ no `Decodable` in `prelude::v1`
85 |     Default,
86 |     Encodable,
   |     ^^^^^^^^^ no `Encodable` in `prelude::v1`

error[E0432]: unresolved import `alloc_crate::collections::CollectionAllocErr`
  --> /home/abrenzikofer/.cargo/git/checkouts/rust-sgx-sdk-fc8771c5c45bde9a/166d481/sgx_tstd/src/collections/mod.rs:46:9
   |
46 | pub use alloc_crate::collections::CollectionAllocErr;
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `CollectionAllocErr` in `collections`

error: internal compiler error: src/librustc/session/mod.rs:379: can't buffer lints after HIR lowering

thread 'rustc' panicked at 'Box<Any>', src/librustc_errors/lib.rs:915:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.40.0-nightly (032a53a06 2019-10-03) running on x86_64-unknown-linux-gnu

note: compiler flags: -C panic=abort -C debuginfo=2 --crate-type rlib

note: some of the compiler flags provided by cargo are hidden

error: aborting due to 4 previous errors

For more information about this error, try `rustc --explain E0432`.
error: could not compile `sgx_tstd`.


mult loader supports?

Setting a loader directly for the template

new HtmlWebpackPlugin({
  // For details on `!!` see https://webpack.github.io/docs/loaders.html#loader-order
  template: '!!handlebars!src/index.hbs'
})

how solve ? , i want:
  template: '!!handlebars!!html!src/index.hbs'

Works only for loader which can be combined - just use the following syntax:
!!loader1!loader2!src/index.hbs


duplicated and missing organisations when querying /v2/organizations

Command

cf orgs | sed '1,3d' | sort | wc -l
cf orgs | sed '1,3d' | sort -u | wc -l

What occurred, incl. error and/or stack trace
The first command gives back 129 org names, the second 94 and in both cases some orgs are missing.

CF_TRACE=true cf orgs| grep '^GET'
GET /v2/organizations?order-by=name HTTP/1.1
GET /v2/organizations?order-by=name HTTP/1.1
GET /v2/organizations?order-direction=asc&page=2&results-per-page=50 HTTP/1.1
GET /v2/organizations?order-direction=asc&page=3&results-per-page=50 HTTP/1.1

CLI Version
6.22.2+a95e24c-2016-10-27

CC API Endpoint Version
API endpoint: https://endpoint (API version: 2.65.0)

Platform & Shell Details
We are running Ubuntu 14.04 on our dev landscape and stemcell 3312 in our prod system

Any other relevant information
The problem is not related to cf cli because I was able to reproduce it with a self-developed python script using the requests module.
Taking out the parameters order-by=name and order-direction=asc from my script solved the problem though.

@SocalNick @utako This looks like it's likely a CAPI issue. Would you mind helping @paolostivanin to debug his problem?

@paolostivanin this looks like a CLI issue, the second and third page request are missing the order-by=name query parameter.

@SocalNick but I do have the issue with a self developed Python script (I am using requests, I am not calling cf cli)...

@paolostivanin we can't help you with your Python script, but if the CLI is using two different orders across pages, it's not an accurate listing of all orgs.

@SocalNick Normally those links are in the response. Do the next links in the response contain the name query?

@fraenkel good call:
cf curl "/v2/organizations?order-by=name&results-per-page=1"
{
   "total_results": 6,
   "total_pages": 6,
   "prev_url": null,
   "next_url": "/v2/organizations?order-direction=asc&page=2&results-per-page=1",
   "resources": [

@dsabeti I moved the story to our tracker

Great, thanks.


Documentation Sprint

This issue is for items related to the documentation sprint for tags. For example, if you want to claim any tags, or ask questions about documenting a tag, you can do that here.
If you want to see an example .mdx file with some advanced usage, look at
https://github.com/siefkenj/pretext-book/blob/6b2f3fc1079dbd37a189bf614203accb974c8bf2/packages/docs/pages/reference/elements/exercise.mdx?plain=1#L1

Documentation leaderboard:

Jason (0)
Sean (0)
Mitch (0)
David (0)
Oscar (0)
Hammad (0)
Nathan (0)


Improve Garbage Collection using Janitor

As the library grows into a more concrete and highly composite one, there arrive a lot of issues with garbage cleaning and how we handle disconnecting events, destroying objects and instances. As of now, we just don't care about memory leaks! And yes, that is true. We are just hoping, that we don't run into such problems. And this hope dies down as we start using Nature2D for full fledged projects.
Problems that exist:

:Destroy() methods aren't implemented properly.
There is no way to destroy the Engine and its entities as a whole.
There's no way to destroy points properly without messing up other objects.
Destroying GuiObjects of constraints, points and rigid bodies doesn't destroy their objects (classes) subsequently.
Its hard to implement garbage cleaning in a library with such a versatile hierarchy of physics objects, without using a maid class.
For situations like destroying points - we disconnect all events, destroy GuiObjects and its class, its parent constraint and the parent rigid body of the constraint (if any). This is hard to implement with current ways of garbage cleaning.

The library is prone to have memory leaks, but this pull request aims to solve all problems related to memory leaks and garbage cleaning using Janitor. This PR will remain a draft until all changes have been proposed and revised.

Started working this!

good job using the explicit cleanup method, that's always what i recommend because it results in less assuming


Laravel 10

Todo
Create new branch v2 where this should merge to

:tada: This PR is included in version 2.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Package Requested

[X] I would like someone else to build the manifest.
[ ] I would like help so I can submit the manifest.
[X] I have performed a search and couldn't find this package.
[X] I have checked that the installer for this package is not a standalone or portable executable or a .zip file
[ ] I think there is a new version available and I have provided the URL.

Please provide the following information
Publisher: N/A
Publisher Url: https://soundux.rocks/
Package Name: Soundux
Description: A cross-platform soundboard
Package Version: 0.2.7
Installer URL: https://github.com/Soundux/Soundux/releases/download/0.2.7/soundux-0.2.7-windows-setup.exe
Current Package Identifier (for new versions): Soundux.Soundux

which blocks silent installation.

Isn't that the equivalent of a UAC prompt for a device driver?

Hey, thanks for opening this. Winget has been on our radar, and we are interested in maintaining this ourselves.
We are currently in the middle of a major rewrite. After that is mostly done, I can try to get the installation as silent as possible.
Is the device driver installation dialog a hard requirement? Because I don't think we can avoid it.

I would need to look into VB-Cable if there's an option for installing the driver silently.
Otherwise, there's nothing we can do about it because we're dependent on it on Windows and it's proprietary software.

VB-Cable is already shipped with VoiceMeter I think which is winget-pkgs currently. I also read somewhere it has silent install and I can't remember where I read the details about it.

Thanks, maybe I can take inspiration from that

@D3SOX See how I did it for the winbtrfs-np package from Scoop. You would need to programmatically add the driver certificate before installing the driver to avoid the prompt.


Request timeout when uploading large objects with git lfs

Gitea version (or commit ref): 1.12.3 docker
Git version: 2.24.3
Operating system: Docker image built for raspberry pi
Database (use [x]):

[ ] PostgreSQL
[ ] MySQL
[ ] MSSQL
[x] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[x] No
[ ] Not relevant

Description
The issue is very similar to #11906 , but is happening with lfs objects:
[Macaron] 2020-08-24 22:33:42: Started PUT .../info/lfs/objects/81fc15137be1a4264ea33370250f8253528370223dc1e8b175815ae84b1a78df for <IP_ADDRESS>
[Macaron] 2020-08-24 22:34:10: Started POST .../info/lfs/objects/batch for <IP_ADDRESS>
[Macaron] 2020-08-24 22:34:10: Completed POST .../info/lfs/objects/batch 200 OK in 267.707332ms
[Macaron] 2020-08-24 22:34:11: Completed PUT .../info/lfs/objects/81fc15137be1a4264ea33370250f8253528370223dc1e8b175815ae84b1a78df 500 Internal Server Error in 29.202370851s

I think push is also retrying the upload multiple times,
I tried to search the docs for some configuration of a timeout but couldn't find it.
Reproducing online is a bit difficult: I'm trying to upload a few GBs, and the push ends up uploading more than 10 GBs without ending.

500 Internal Server Error indicates server error. Can you check gitea.log file for additional error details

I didn't know there were other logs. Now I see this error:
2020/08/29 01:18:23 ...lfs/content_store.go:71:Put() [E] Whilst putting LFS OID[37e9651bf0a29f3274cefd322d3e4ddd34860657459f446c40ccb6138e1ef477]: Failed to copy to tmpPath: /data/git/lfs/37/e9/651bf0a29f3274cefd322d3e4ddd34860657459f446c40ccb6138e1ef477.tmp Error: unexpected EOF

Just to rule it out, can this be disk space error?

Unfortunately no, that is the first thing I checked. I also tried to go to git/lfs/37/e9 and I created a test file of 2GB without errors.

Are you running with a proxy?

Yes, gitea is behind nginx.


Can i modify the bot to add new selector ?

I just wonder if I can add new selectors such as Zaif.jp, bitflyer.com etc.. as many users in Japan use these exchanges.
I have APIs provided from them.

mind if you'd advise where trading symbols can be found usually ?
i wish to custom codes on my own.

it worked one way or another..


Unsupported operation sqlite3_open_v2

Motivated by Ralf Jung's blog post about the last 2 years of miri development I've tried to run diesels test suit using miri. Especially the sqlite backend does a lot of nasty pointer things, so another layer of checks beside asan and friends would be helpful. I run into the following error message:
error: unsupported operation: can't call foreign function: sqlite3_open_v2
   --> diesel/src/sqlite/connection/raw.rs:47:13
    |
47  |             ffi::sqlite3_open_v2(database_url.as_ptr(), &mut conn_pointer, flags, ptr::null())
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't call foreign function: sqlite3_open_v2
    |
    = help: this is likely not a bug in the program; it indicates that the program performed an operation that the interpreter does not support

    = note: inside `sqlite::connection::raw::RawConnection::establish` at diesel/src/sqlite/connection/raw.rs:47:13
note: inside `<sqlite::connection::SqliteConnection as connection::Connection>::establish` at diesel/src/sqlite/connection/mod.rs:92:30
   --> diesel/src/sqlite/connection/mod.rs:92:30
    |
92  |         let raw_connection = RawConnection::establish(database_url)?;
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `test_helpers::connection` at diesel/src/test_helpers.rs:10:13
   --> diesel/src/test_helpers.rs:10:13
    |
10  |             SqliteConnection::establish(":memory:").unwrap()
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `connection::transaction_manager::test::sqlite_transaction_is_rolled_back_upon_deferred_constraint_failure` at diesel/src/connection/transaction_manager.rs:896:25
   --> diesel/src/connection/transaction_manager.rs:896:25
    |
896 |         let conn = &mut crate::test_helpers::connection();
    |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside closure at diesel/src/connection/transaction_manager.rs:889:5
   --> diesel/src/connection/transaction_manager.rs:889:5
    |
887 |       #[test]
    |       ------- in this procedural macro expansion
888 |       #[cfg(feature = "sqlite")]
889 | /     fn sqlite_transaction_is_rolled_back_upon_deferred_constraint_failure() {
890 | |         use crate::connection::transaction_manager::AnsiTransactionManager;
891 | |         use crate::connection::transaction_manager::TransactionManager;
892 | |         use crate::result::Error;
...   |
923 | |         );
924 | |     }
    | |_____^
    = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

I should note that I kind of expected that. I mainly raise this issue as the blog post explicitly calls for opening issues for unsupported operations.
Environment information:

Ubuntu 22.04
rustc --version:

rustc 1.63.0-nightly (c06728704 2022-05-19)
binary: rustc
commit-hash: c0672870491e84362f76ddecd50fa229f9b06dff
commit-date: 2022-05-19
host: x86_64-unknown-linux-gnu
release: 1.63.0-nightly
LLVM version: 14.0.4

Steps to reproduce:
git clone https://github.com/diesel-rs/diesel
cd diesel
git checkout cbdfee30b206fff0aba62fa482fd837c0edd2cea
cd diesel
cargo miri test --features "sqlite" --no-default-features

I think the only way forward for this would be to change libsqlite3-sys to use #[cfg(miri)] to do a pure-Rust in-memory implementation (beyond compiling the C library to Rust via that c2rust converter, I forget its name).
I mean, we do have https://github.com/rust-lang/miri/issues/11 which would allow these things directly, but considering it has been untouched for years, it may take more years.

Yeah, I should probably have worded this a bit more carefully in the blog post... we're not going to be able to implement Python, PHP, or sqlite in Miri. :/

I guess to resolve this issue properly we should add something to https://github.com/rust-lang/miri/blob/master/CONTRIBUTING.md that says what things are in scope of miri and what are out of scope

👀 https://github.com/maurer/miri/pull/1

Oh wow. oO

Closing as we'll not implement this shim in Miri.