go vet complains about undefined methods, even though go test succeeds

What version of Go are you using (go version)?
1.10.3
$ go version
go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?
Yes, 1.10.3 is the latest release.
What operating system and processor architecture are you using (go env)?
linux, amd64
> go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/prashant/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"

What did you do?
Made go vet complain about an undefined method, even though go test  works fine.
Repro in: https://github.com/prashantv/govet-bug
/home/prashant/go/src/github.com/prashantv/govet-bug
$ ls
registry

/home/prashant/go/src/github.com/prashantv/govet-bug
$ go test ./registry
ok      github.com/prashantv/govet-bug/registry (cached)

/home/prashant/go/src/github.com/prashantv/govet-bug
$ go vet ./registry
# github.com/prashantv/govet-bug/registry_test
registry/registry_test.go:11:3: r.HandleGroup undefined (type *registrytest.Registry has no field or method HandleGroup)

What makes this case tricky is that:
registry_utils_test.go is a test file that exports an unexported method
registry_test.go is trying to call the exported method through an external type that embeds the original Registry.
If I import the registry package, then the error goes away.
What did you expect to see?
go vet to not complain about a missing field/method, since go test passes, and the method should be promoted from registry.Registry to registrytest.Registry.
What did you see instead?
The error as reported earlier,
registry/registry_test.go:11:3: r.HandleGroup undefined (type *registrytest.Registry has no field or method HandleGroup)

In a larger codebase internally, go vet crashed,
Made go vet crash.
# [...]/registry_test
<unknown line number>: internal compiler error: no function definition for [0xc420c60360] FUNC-method(*registry.Registry) func(http.ResponseWriter, *http.Request)

Thanks for reporting the bug.  I can reproduce this with 1.10 but not with 1.11beta3.  So I assume this is fixed in the upcoming 1.11 release.  This isn't something we will fix on the release branch, so closing this issue.

I see this issue for go 1.12.
go version
go version go1.12 darwin/amd64

Anyone know how to fix this, I'm on go1.12.1 darwin/amd64, see the same issue.

given the recent reports starting on March 5th of this being re-introduced in 1.12, this issue should be re-opened, or a new one created.
This is still an on-going problem

Any update on this. This just happened to me.  I am using go1.11.5 darwin/amd64. GOVET fails with undefined: <struct_name> when calling an struct from the same package that is in a different file.

Any update on this. This just happened to me. I am using go1.11.5 darwin/amd64. GOVET fails with undefined: <struct_name> when calling an struct from the same package that is in a different file.
Which version do you recommend to be on?

for me, I had to update separate tooling my organization uses, which itself depends on go. So I am assuming in my case that there was some non-obvious conflict between the go version on my system and the go version my tooling was using.
while it would be a little hard to believe that this is the case for everyone still having this issue, if you are using some separate tooling (rather than calling go vet directly) try updating that tool

This is still happening to me on 1.12.5, created a new issue https://github.com/golang/go/issues/32408

This isn't something we will fix on the release branch, so closing this issue.

why was this closed, till today I can reproduce with go1.12.6, why go vet complain undefined ... something; but it's clearly all well defined, because go test can pass.

Reopening to see if it still occurs.  For people reporting it: does it occur with the original test case?

@tx0c, I can't reproduce this issue with go1.12.6 using the steps as originally posted. If you are seeing an issue with similar symptoms, the underlying cause may or may not be the same. Please file a new issue with concrete steps (and source files) to reproduce it.
$ go1.12.6 get -d github.com/prashantv/govet-bug/registry

$ cd $GOPATH/src/github.com/prashantv/govet-bug

_gopath/src/github.com/prashantv/govet-bug$ go1.12.6 test ./registry
ok      github.com/prashantv/govet-bug/registry 0.020s

_gopath/src/github.com/prashantv/govet-bug$ go1.12.6 vet ./registry

_gopath/src/github.com/prashantv/govet-bug$


symbols-no-unused rule isn't counting usage for symbols only used as overrides

Summary
I'm being told by the "Tidy" assistants I should use symbols, on artboards that are symbols.
Steps to Reproduce
Steps required to reproduce the problem:

Opened comments-annotations.sketch.zip
Ran the "Tidy" assistant
View Results
See:

My only inkling is I may have created these particular symbols by duplicating another and then renaming it. Perhaps it's picking up some history there and erroneously flagging it?
cc @jedrichards @carmenlanza

Looks like the rule is probably neglecting to search through symbols being used as overrides.

QA

Checkout the branch
In the root, yarn and then yarn build
cd assistants/tidy
yarn package-tarball
Test in Sketch, confirm that a symbol only used as a value in an another symbols override doesn't trigger a "Symbol should be used" violation

@jedrichards Tidy doesn't seem to be reporting any violations for me (outside of artboards or unused symbols. Using SHA a4d57877 from fix/count-overrides-as-used-symbols, current Experimental build with the attached .tgz and file.
Archive.zip

the original issue that was reported, which i completely misread, is that if an artboard itself is a symbol, it's not being counted as in use. Artboards as symbols seem to be handled differently by the app in general.

If you create an artboard and convert it to a symbol, it is not added to the Symbols page.
If you add an artboard on the Symbols page directly, convert it to a symbol, and then insert the instance on the page...no violation but it does not appear like other artboards.

@jedrichards Change of direction, just remove the rule per our discussion in Slack https://sketch.slack.com/archives/CK9MW0PL2/p1590080701029300?thread_ts=1590069730.020500&cid=CK9MW0PL2

@smithzoo Ok it's removed from Tidy. Same instructions as above should allow you to confirm.

Verified successful removal of the symbols-unused rule.


Landing page doesn't feature a short description of gush

Wondering what gush might be, I went to its homepage.  However the landing page has absolutely no description whatsoever of what gush could possibly do for me.
Please add at least a short sentence summarizing of why I should bother looking at gush's documentation to find out more.  It would be a waste if people left the site without having a closer look at gush, since it seems very useful.
by @seschwar

added


[IA-2645] Delete apps when a workspace is deleted

Currently, users are having issues because if a workspace is deleted without their apps being deleted the apps are orphaned, accruing cost, and not accessible by the user in any way.
We decided to detect this at workspace deletion time, and not allow workspace deletion if apps would be orphaned. We can auto-delete these apps if they are in 'RUNNING' or 'ERROR' status, but otherwise user intervention is required.
All of these changes are contained in the DeleteWorkspaceModal. Included below are screenshots of the various conditions
Delete with no apps existing (current use case):

Delete with non deletable apps detected (user action required before deletion). Note we remove all subsequent pieces that ask for user input when requesting this specific action from them:

Delete with only auto-deletable apps (no user action required, submitting will cause these apps to be deleted for them):

Delete with only auto-deletable apps with confirmation typed:

The network console when this modal is submitted with 5 auto-deletable apps detected (the requests seem to appear twice, but one of them is the pre-flight options method submitted by fetch. The delete action is submitted only once per app)

Also it looks like the build is failing due to something this recently merged PR was trying to fix: https://github.com/DataBiosphere/terra-ui/pull/2452 if you merge in dev now hopefully the build will work

@jdcanas I'd be happy to pair with you on this, if you'd like, just to work on code style and simplifying logic.

@jdcanas I'd be happy to pair with you on this, if you'd like, just to work on code style and simplifying logic.

I understood and implemented the changes you recommend, let me know if they look good. We can chat if we end up having some back and forth


Replace bintray repository

PR related to task #2136

PackageCloud repository used for rabbitmq since it is recommend by official rabbitmq documentation and looks very solid (they archive packages for 5 years)
Erlang-solution repository used for erlang for Ubuntu/Debian distribution - there is no better replacement, rabbitmq doesn't release erlang packages for Ubuntu/Debian distribution on rabbitmq github repository
RabbitMQ github repository used for erlang package for RedHat/Centos distribution. Well suited to RabbitMQ

/azp run


ZIndex Fix

Description of changes:
Updating zIndex: 1 ➡ zIndex: 1200 to fix any possible position issues with other components
Related issues (if any):
😭
While integrating this fantastic package, I came across a ton of problems trying to backtrack the styling zIndexing for a fixed nav I had while using the lightbox and couldn't for the life of me find any non-clunky, hulk-smash method of fixing blanketCSS.
👍
This setup gives decent leeway for anyone who might want to use this version with any components that utilized position styles that may interfere. Feel free to eviscerate my logic for the pull request; I'd love to know if there are any possible downstream complications with this edit! 🙏
Otherwise Stay

It's an amazing edit and an extremely late response from me because I never saw it! It's already fixed by a just merged PR.
You can get the fix after I publish the latest beta version.


new uWorld and Name address ？？？

new uWorld and Name address ？？？

Please make a pull request, I'm not updating these offsets anymore.
I will merge the request then,

#define UWORLD			 0x3A4C128
#define GNAMES			 0x394EF10

@lmssky  How do you get these addresses  ？？？

#define UWORLD 0x3A4C128
#define GNAMES 0x394EF10
The address is not correct, or has been updated.

我正在使用，它们没问题

Does the other shift change?

Is there any change in the other offsets??

No change

localPlayerState  is 0

the addresses   Is it used on this project  ？

closing, new offsets in #101


Improve ZIP handling in interactive update

[x] Have you signed the Contributor License Agreement?
[x] Are you working against an Issue?

#397

This change makes it so that the zip archive is extracted in interactive update as well and the path to the nested installer is used in other functions. Made the logic of extracting and determining path of the nested installer into a function as it's common between autonomous and interactive updates.
The logic is based off of the fact that if InstallerType is ZIP, there will always be atleast one NestedInstallerFile node in the manifest that contains RelativeFilePath key and the first matching RelativeFilePath from an older manifest is used as the path to the nested installer. Multiple RelativeFilePath nodes under NestedInstallerFiles are only possible if the NestedInstallerType is portable. In this PR, we do not check the InstallerType of all RelativeFilePath nodes for NestedInstallerType: portable. The assumption is that it's very rare that if an older version contained all portable RelativeFilePath objects, then one of them has changed in newer version to be an another InstallerType (e.g. MSI, nullsoft etc). In case of portables as well, only the first matching RelativeFilePath object is passed to determine the InstallerType to keep the logic simple.
Shifting of values from root to installer is needed in cases where the NestedInstallerFiles object is common between all installer nodes and is present in the root of the manifest instead for better readability. One of the many examples of such manifests is junnegun.fzf. Since the interactive update only works by reading the Installers node under the installer manifest, we need a way to read the NestedInstallerFiles in the root of the manifest. I did it by shifting all related values to installer level at the start of the interactive update. Ofcourse this comes at a cost of slightly more verbose manifest, but I think it's okay for now considering the benefit of a successful update. We can make another function ShiftInstallerValuesToRoot to take out common fields to the root level but I think that's beyond the scope of this PR as it's applicable to new, autonomous update commands as well and being tracked in issue https://github.com/microsoft/winget-create/issues/185

Microsoft Reviewers: codeflow:open?pullrequest=https://github.com/microsoft/winget-create/pull/398&drop=dogfoodAlpha

/azp run

Apologies for the delay, since the interactive update needs to be manually tested, I needed to checkout your branch and try it manually in order to review this. Shifting the root properties to the installer node is a great addition and it will help improve the interactive flow.
Once thing I wanted to call out was that I found a bug when trying to update junnegun.fzf interactively. All of the architectures get updated to 'x64' thus making the updated manifest invalid. I created an issue here: https://github.com/microsoft/winget-create/issues/405 but I went ahead and approved this PR since that is not related to the feature you just added. Thanks!


[Experiment] One-way closure parameters

Introduce an experimental mode (behind the flag
experimental-one-way-closure-params) that places one-way
constraints between closure parameter types and references to those
parameters within the body of the closure. The intent here is to
break up constraint systems further, potentially improving type
checking performance and making way for larger closure bodies to be
supported.
This is a source-breaking change when the body of a single-expression
closure is used to determine the parameter types. One obvious example
is when there is no contextual type, e.g.,
let _ = { $0 + 1 }

this type-checks today because 1 becomes Int, which matches the
+ overload with the type (Int, Int) -> Int, determining the
parameter type Int for the closure. Such code would not type-check
with one-way constraints.

@DougGregor This failure is actually want is preventing my PR for BindParam as well because currently it's possible to infer inout from @lvalue used in the body e.g. _ = { $0 = 1 }. I managed to make that one work but there are more complicated examples of the behavior is SwiftPM and in the test suite.

@DougGregor This failure is actually want is preventing my PR for BindParam as well because currently it's possible to infer inout from @lvalue used in the body e.g. _ = { $0 = 1 }. I managed to make that one work but there are more complicated examples of the behavior is SwiftPM and in the test suite.

Ah, tricky, tricky. Maybe we can (temporarily?) patch things up in SwiftDriver and SwiftPM to at least get a complete set of results, but it might be that one-way constraints cannot be used for single-expression closures. Or maybe for closures involving $ variables.

Yeah, if I remember correctly there were some compatibility suite failures with that as well.

This is behind a flag, so I'm going to consider it ready to merge as an experimental thing to make it easier to play with in the future. We'll depend on this notion for the (also experimental) multi-statement closure support.

Here's just the experimental flag, which includes the semantics we're going to need elsewhere. The "one way by default" experiment is over at https://github.com/apple/swift/pull/32222.

@swift-ci please smoke test

@swift-ci please smoke test

@swift-ci smoke test Linux


Migration

Hi,
What column type would you use for the migration?
Thanks for a promising package!
//Tim

You basically have two fields, one for latitude and one for longitude. I would recommend:
            $table->decimal("latitude", 15, 12)->nullable();
            $table->decimal("longitude", 15, 12)->nullable();

Thank you!
You should att these details to the docs.
Thanks for an awesome field!
//Tim


SUGGESTION: Implement a 'DOS_Sleep' function?

Hi!
I recently stumbled across your project, and it works fantastically; I'm working on a project that replicates BIOS-like behavior, but I'm running into an issue where I can't cause my app to "sleep" for 2 or 3 seconds before refreshing the display.
Any ideas how I could go about doing that?

I can add that at some point! In the meantime you should be able to use SDL_Delay, which takes a millisecond value.

If I do that, the application ends up freezing up, so what I ended up doing was something like this,
int tick = 0;

int KdGetTickCount() {
    return tick;
}

int KdDispatch(int delay) {
     int cond = 0;

     if ((tick + 1) == delay) {
          tick = 0, cond = 1;
     }

     return cond;
}

this way, I can call the function on multiple passes and return either 0 or 1, depending on whether I want the function to signal to the rest of the application that I want it to continue.
My use-case is basically a full-screen BIOS-type UI, with a "copyright" splash screen that behaves as if it's a wizard.

Glad you found a solution! Both SDL_Delay and the standard library function 'sleep' do suspend the program. If I add something, it'll be along those lines.
I reread your first message, you're looking to have the program run uninterrupted, but only refresh periodically? Ideally, things would work by not refreshing at all until something changes, like the cursor blinks, or there's user input (causing a selection highlight to change, for example). At some point, I'll be revamping things to get it to work like that.
A couple thoughts for now: If your question is to do with slow performance related to continuous refreshing, definitely avoid using DOS_RenderChar and DOS_RenderString, especially for a full screen's worth of text - they're pretty slow. Use a DOS_Console or 'screen' via DOS_InitScreen. Also, DOS_LimitFrameRate will slow things down a little.

Yes, I'm looking to "delay" the app without calling sleep under the hood.


abseil-cpp: 20200225.2 -> 20200923.3

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/abseil/abseil-cpp/releases.
meta.description for abseil-cpp is: "An open-source collection of C++ code designed to augment the C++ standard library"
meta.homepage for abseil-cpp is: "https://abseil.io/"
meta.changelog for abseil-cpp is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 20200923.3 with grep in /nix/store/lv3jykhqxsl5vfdnq4by0d763rqvwrsa-abseil-cpp-20200923.3

directory tree listing: https://gist.github.com/91aab597729bd486d4575768d1ef885a

du listing: https://gist.github.com/864dbf0dd06e4fba098a553626cd5b70

Rebuild report (if merged into master) (click to expand)

875 total rebuild path(s)

249 package rebuild(s)

240 x86_64-linux rebuild(s)
211 i686-linux rebuild(s)
202 x86_64-darwin rebuild(s)
222 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
abseil-cpp
beancount
bear
deja-dup
duplicity
duply
dvc-with-remotes
fava
gcalcli
google-cloud-cpp
grpc
kmymoney
ledger2beancount
lieer
linuxPackages-libre.sysdig
linuxPackages.sysdig
linuxPackages_4_14.sysdig
linuxPackages_4_19.sysdig
linuxPackages_4_4.sysdig
linuxPackages_4_9.sysdig
linuxPackages_5_10.sysdig
linuxPackages_5_4.sysdig
linuxPackages_5_9.sysdig
linuxPackages_hardened.sysdig
linuxPackages_hardkernel_4_14.sysdig
linuxPackages_hardkernel_latest.sysdig
linuxPackages_latest-libre.sysdig
linuxPackages_latest.sysdig
linuxPackages_latest_hardened.sysdig
linuxPackages_latest_xen_dom0.sysdig
linuxPackages_latest_xen_dom0_hardened.sysdig
linuxPackages_lqx.sysdig
linuxPackages_testing_bcachefs.sysdig
linuxPackages_xen_dom0.sysdig
linuxPackages_xen_dom0_hardened.sysdig
linuxPackages_zen.sysdig
murmur
or-tools
python37Packages.arviz
python37Packages.baselines
python37Packages.beancount
python37Packages.beancount_docverif
python37Packages.cirq
python37Packages.dm-sonnet
python37Packages.edward
python37Packages.goobook
python37Packages.google-api-core
python37Packages.google-cloud-access-context-manager
python37Packages.google-cloud-asset
python37Packages.google-cloud-automl

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/lv3jykhqxsl5vfdnq4by0d763rqvwrsa-abseil-cpp-20200923.3 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A abseil-cpp https://github.com/r-ryantm/nixpkgs/archive/4043bb9636451cb9de386ba816ee85595c287c0f.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/lv3jykhqxsl5vfdnq4by0d763rqvwrsa-abseil-cpp-20200923.3
ls -la /nix/store/lv3jykhqxsl5vfdnq4by0d763rqvwrsa-abseil-cpp-20200923.3/bin

Pre-merge build results
NixPkgs review skipped

Maintainer pings
cc @andersk for testing.

Same story as #100827 and #107846, this will break lots of packages. abseil-cpp is designed to be vendored, not shared. We’re probably going to need to provide multiple versions.

Same story as #100827 and #107846, this will break lots of packages. abseil-cpp is designed to be vendored, not shared. We’re probably going to need to provide multiple versions.

addressed by #112951


Run micro_tutorial get error

I have bulit TorchCraftAI and CherryPi on unbuntu 18.04.
But I got this error when I input this cmd
./build/tutorials/micro/micro_tutorial -vmodule state=-1,micro_tutorial=1 -scenario vu_zl
I 6788/XXXXX 04/24 13:02:39 [micro_tutorial.cpp:246] Scenario: vu_zl
I 6788/XXXXX 04/24 13:02:39 [micro_tutorial.cpp:247] Model: PF
I 6788/XXXXX 04/24 13:02:39 [micro_tutorial.cpp:248] Resume: false
I 6788/XXXXX 04/24 13:02:39 [micro_tutorial.cpp:249] Evaluate: false
I 6788/XXXXX 04/24 13:02:39 [micro_tutorial.cpp:256] resultsJSON: ./metrics-rank-0.json
I 6788/XXXXX 04/24 13:02:39 [micro_tutorial.cpp:257] resultsCheckpoint: ./trainer_latest.bin
I 6788/XXXXX 04/24 13:02:39 [micro_tutorial.cpp:284] Model has 221568 total parameters
I 6788/XXXXX 04/24 13:02:39 [micro_tutorial.cpp:290] Begin training!
F0424 13:02:39.818872  6796 forkserver.cpp:575] You must call ForkServer::startForkServer! Call it as early as possible (in main, after parsing command line flags, but before initializing gloo/mpi/anything else)!
*** Check failure stack trace: ***
F0424 13:02:39.818872  6797 forkserver.cpp:575] You must call ForkServer::startForkServer! Call it as early as possible (in main, after parsing command line flags, but before initializing gloo/mpi/anything else)!F0424 13:02:39.819079  6798 forkserver.cpp:575] You must call ForkServer::startForkServer! Call it as early as possible (in main, after parsing command line flags, but before initializing gloo/mpi/anything else)!F0424 13:02:39.819214  6799 forkserver.cpp:575] You must call ForkServer::startForkServer! Call it as early as possible (in main, after parsing command line flags, but before initializing gloo/mpi/anything else)!F0424 13:02:39.819252  6800 forkserver.cpp:575] You must call ForkServer::startForkServer! Call it as early as possible (in main, after parsing command line flags, but before initializing gloo/mpi/anything else)!

Sorry Jeremy, I just saw this. The tutorial fell a little out of date. Try adding this line prior to cherrypi::init() in microtutorial.cpp: https://github.com/TorchCraft/TorchCraftAI/blob/develop/scripts/botplay/botplay.cpp#L36


Horizontal Lists: Use full width

Before:

After:

As far as I know, hlist's are not used by the auto API generator.as for other sphinx elements, this is specific to only the .. hlist:: directive.


Scalable merge/compaction of big doc values segments.

The question is regarding the scalable merge/compaction of doc values, given the following context:

I have a large sharded index.
Each shard can contain segments of millions of documents.
There are several hundreds fields in the index, and half of them are doc values.

Sometimes, I face issues with merge times when I need to merge or compact a large segment. The problem is that it's a single-threaded operation where a single segment is merged in a single merger thread.
From the codec doc values format of version 9.x, it appears possible to use map-reduce techniques when writing new large doc value segments. This is because all metadata is read before any field data can be read, and all doc value types have offset and length fields in the metadata.
My basic idea is to write each field in parallel to a separate file and then perform a low-level merge of the binary data (just appending bytes to the final file). After that, I can rewrite only the metadata to update the offsets.
As I am still new to Lucene development, could someone please provide some critique of this idea?
p.s. Unfortunately, the same idea is not applicable for the inverted index format due to its complexity.

My basic idea is to write each field in parallel to a separate file and then perform a low-level merge of the binary data (just appending bytes to the final file). After that, I can rewrite only the metadata to update the offsets.

This is an interesting approach, I'd like to explore it more. There has been some discussion on this problem in #9626 , that you might find useful.
I think it makes sense to chip away at this problem with one format type at a time. Let's start with the approach you have in mind for doc-values. If you want to raise a PR for this, I'm happy to iterate with you on it. We can start with a draft PR that demonstrates the idea first, and benchmark it for viability. And eventually refine it to consolidate across the general segment merging framework.
I was playing around with doing this for the postings format some time back. It has been on the shelf for some time, but I guess it's time to dust it off and try again. Hopefully, we can collaborate on some common learnings.

Hi, @rmuir!

are you sure docvalues is really the slow part of your merge. I actually think doing this for terms/postings would be more bang-for-the-buck?

I am not stating that the doc values are the heaviest part of the force merge process. In my case, the rewriting of doc values from the original segment (10 millions docs) took 318 seconds, which is comparable to the time it takes to merge posting lists. The fully parallel writing (w/o a final metadata update) took 23 seconds!

docvalues is a bit harder and trickier: typically docvalues are only a tiny fraction of merge costs, compared to postings (especially merging the terms seems to be very intensive).

there are some real traps here with docvalues, especially string fields (SORTED/SORTED_SET). In order to merge these >?>fields, it has to remap the ordinals which requires an additional datastructure to do. Doing this for many fields at once without >being careful could spike memory (and possibly for little benefit as again these fields are typically much faster to merge than >indexed ones).

Hmm. After examining the codec code in version 9.x, I came to the opposite conclusion. Please correct me if I'm wrong, but it appears that each doc values field data consists of two files: meta and data. Moreover, it seems that each doc value field is written separately and without sharing data between them.
Perhaps I wasn't clear earlier, but what I meant was to write multiple doc values using the original codec, if that's possible. For instance, if I have two fields, I would have four files (two data files and two meta files). Then, I could copy those data files at the byte level, using the something like cat file1 > all_fields; cat file2 >> all_fields. As for the metadata files, I would need to fix the absolute numbers (i.e., the offsets). Writing of data files is parallel operation, updating metadata is a single-threaded.

Check it out here for string fields to see what i mean:
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L662-L663
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java#L849-L850
it would be bad to instantiate a ton of these OrdinalMap instances (one for each field) all at once, rather than sequentially which keeps the memory low.

Yes, I saw that code. However, I thought it is intended for use when merging multiple segments into one, field by field.
May I share a simple example to explain why I think so? Let's say we have the following example:
var logger = LogManager.getLogger("lucene_index");
var infoStream = new Log4j2InfoStream(logger, Level.toLevel("info"));

var out = FSDirectory.open(Path.of("/home/sherman/experiments7/"));
var cfg = new IndexWriterConfig();
cfg.setUseCompoundFile(false);
var indexWriter = new IndexWriter(out, cfg);
for (var i = 0; i < 10000; i++) {
    var doc = new Document();
    doc.add(new SortedSetDocValuesField("field1", new BytesRef("term1_" + i)));
    if (i < 500) {
        doc.add(new SortedSetDocValuesField("field2", new BytesRef("term1_" + i)));
    }
    indexWriter.addDocument(doc);
}
indexWriter.close();

In this example, each field has its own term dictionary, because I see two calls of addTermsDict() method.
I think I need to conduct further investigation to create a prototype that can produce the same files as a result.
Maybe, I missed something.

the rewriting of doc values from the original segment (10 millions docs) took 318 seconds, which is comparable to the time it takes to merge posting lists. The fully parallel writing (w/o a final metadata update) took 23 seconds!

Wow this is amazing speedup!  Can you provide some details?  Where is the prototype implementation, how many cores/threads did you use, etc.
@rmuir's concern about concurrently enumerating multiple OrdinalMaps is a real risk: this data structure can be memory consuming, especially for high cardinality (many unique strings) fields, and, indices where each segment has very different sets of strings for that field.  But maybe if we limited the concurrency of such costly fields in some way this might be OK.

Hi, @mikemccand!
Hi, Mike!
I re-run the tests.
My hardware:
Model name: Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz
96 cores.
The test was quite simple: rewriting all doc values of a specific index segment, which is similar to what we do when we run a compaction process.
So, in this test I had a segment with 8.021.709 documents and the following statistics of doc values fields (yes, we have a lot of doc values fields):
SORTED_NUMERIC=1649
SORTED=1
SORTED_SET=4863
The total size of .dvd file is: 8.3G
The baseline (single thread) took 249 seconds.
The parallel test (32 threads) took 19 seconds.


Update dependency org.apache.lucene:lucene-highlighter to v9

This PR contains the following updates:

Package
Type
Update
Change

org.apache.lucene:lucene-highlighter (source)
compile
major
2.9.3 -> 9.4.0

⚠ Dependency Lookup Warnings ⚠
Warnings were logged while processing this repo. Please check the logs for more information.

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

Renovate Ignore Notification
As this PR has been closed unmerged, Renovate will ignore this upgrade and you will not receive PRs for any future 9.x releases. However, if you upgrade to 9.x manually then Renovate will reenable minor and patch updates automatically.
If this PR was closed by mistake or you changed your mind, you can simply rename this PR and you will soon get a fresh replacement PR opened.


~7 Second (31%) difference between Mix test and mix test --partitions in Ecto Heavy test suites

Environment

Elixir version (elixir -v): 1.11.3
Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): Postgres:12-alpine (docker container)
Ecto version (mix deps): 3.4.6
Database adapter and version (mix deps): 3.4.6
Operating system: Ubuntu 18.04
Original issue: https://github.com/elixir-lang/elixir/issues/10853

Current behavior
I was able to replicate this on this branch: https://github.com/SophisticaSean/elixir-async-testing/tree/testing-performance-issues
I know the tests are contrived but I didn't want to have to write a bunch of application logic to replicate my issue. It has something to do with tests that take a lot of Ecto time. The behavior I see is that on the mix test run it'll sometimes hang on the longer running Ecto heavy tests. No other tests report as complete when it hangs on these Ecto heavy tests. I believe this is why running parallel mix test --partitions runs is faster, they all separately hang on those heavy Ecto tests but since they're running in parallel they don't collectively slow each other down. We tested in this in the original issue with just a :timer.sleep implementation and adding --max-cases fixes the issue. This fix doesn't work for Ecto heavy tests though.
From this behavior I would hazard a guess that there's something wrong in the implementation of Ecto.Sandbox that's bottlenecking the tests.
My expectation would be that mix test would handle this case for me and have several other cpu threads working on other test files in tandem with the main mix test thread. So at any given time we should always have tests running and completing.
Run tests script again (use gdate instead of date if you're on MacOs or Unix):
function run_tests {
  unset pids
  start=$(date +%s.%N);
  for i in {1..$1}; do
    MIX_TEST_PARTITION=${i} mix test --partitions $1 &
    pids[${i}]=$!
  done

  for pid in ${pids[*]}; do
    wait $pid
  done
  dur=$(echo "$(date +%s.%N) - $start" | bc);
  printf "Execution time: %.6f seconds\n" $dur
};

Results

config/test.exs: ecto_loop_count: 1000
config/test.exs: timer_sleep_loop_count: 10
32 Core Server ~2.6 ghz clock speed:

vanilla time mix test --max-cases 384 average over 20 runs: 21.868 seconds
run_tests 6 average over 20 runs: 17.3519 seconds
31% speed difference.

Expected behavior
Vanilla mix test should be fastest, or very close to the hand parallelized version. Instead run_tests 6 is 31% faster. This is also the case for my production test suite on a private github repo with ~750 tests and a lot of tests that interact with Ecto.

@SophisticaSean can you please see what happens when you increase your ecto_loop_count to 5000 or 10000? It takes 6.2 seconds for me to load the test files and I want to rule that partitioning is faster to load code rather than running it.
Alternatively, try this command: mix test --exclude test --exclude doctest and MIX_TEST_PARTITION=... mix test --exclude test --exclude doctest --partitions 6 and see how faster it is.

Ok, I was unable to reproduce the issue on my machine. My guess is that we are hitting a bottleneck somewhere between 8 to 32 cores.

I ran it on my 32 core server and I'm seeing this:
10000 ecto_loop_count test timeout
5000 ecto_loop_count:
time mix test --max-cases 384: 54.130 seconds
run_tests 6: 36.226630 seconds

5000 ecto_loop_count:

This is very interesting. I am sure we are spawning all of those tests concurrently but the tests running longer makes a strong impact on the overall results.
I have a couple more theories for us to test (in isolation):

What happens if you wrap the Ecto test in a Repo.checkout call?

What happens if instead of creating a user, you sleep on the database side of things?

So 3.77 seconds for vanilla mix test and 1.914 seconds for partitioned mix test

To be clear, this is just one of the partitions, which means it load only part of the tests. Anyway, this is good enough to say the issue likely isn't with load times. Thanks!

Results for 5000 Ecto loop count on 32 core machine:
checkout wrap inside loop
  test "complex ecto user test" do
    loop_count = Application.get_env(:async_testing, :ecto_loop_count)
    Enum.each(1..loop_count, fn x ->
      AsyncTesting.Repo.checkout(fn ->
        email = Ecto.UUID.generate()
        name = "Tim#{x}"
        {:ok, user} = User.create(name, email)
        assert user.name == name
        assert user.email == email

        user = User.get_by_email(email)
        assert user.name == name
        assert user.email == email
      end)
    end)
  end

mix test --max-cases 384  58.667 seconds average over 3 runs
run_tests 6 41.653 seconds average over 3 runs

Results for 5000 (:timeout increased to 60_000) Ecto loop count on 32 core machine:
checkout wrap outside loop
  test "complex ecto user test" do
    loop_count = Application.get_env(:async_testing, :ecto_loop_count)

    AsyncTesting.Repo.checkout(fn ->
      Enum.each(1..loop_count, fn x ->
        email = Ecto.UUID.generate()
        name = "Tim#{x}"
        {:ok, user} = User.create(name, email)
        assert user.name == name
        assert user.email == email

        user = User.get_by_email(email)
        assert user.name == name
        assert user.email == email
      end)
    end)
  end

mix test --max-cases 384  47.863 seconds average over 3 runs
run_tests 6 36.11 seconds average over 3 runs

What happens if instead of creating a user, you sleep on the database side of things?
I'm not sure what you mean by this, could you clarify?

@SophisticaSean ok, so the checking out of connections definitely play a role.

I'm not sure what you mean by this, could you clarify?

Instead of User.create, do something like Repo.query!("SLEEP 0.1") or whatever is the SQL syntax for it.
Besides the above, can you please try to see what happens when using the regular pool instead of the ownership pool?

Results for AsyncTesting.Repo.query!("SELECT pg_sleep(#{duration});") with duration of 0.01 and run_count of 5000
test "complex ecto database sleep test" do
  loop_count = Application.get_env(:async_testing, :ecto_user_sleep_loop_count)
  duration = Application.get_env(:async_testing, :ecto_user_sleep_duration)
  Enum.each(1..loop_count, fn _x ->
      assert :ok = User.sleep(duration)
  end)
end

mix test --max-cases 384  59.312 seconds average over 3 runs
run_tests 6 58.866 seconds average over 3 runs

Results for AsyncTesting.Repo.query!("SELECT pg_sleep(#{duration});") with duration of 0.001 and run_count of 5000
same code as above

mix test --max-cases 384  41.736 seconds average over 3 runs
run_tests 6 24.654  seconds average over 3 runs

Fantastic! Can you try with “SELECT 1”? It seems the issue is not the sleep but simply don’t the queries.
and then let’s try without the ownership pool. :)

SELECT 1; results on Sandbox pool
Results for AsyncTesting.Repo.query!("SELECT 1;;") with run_count of 5000
  test "ecto database select_1 test" do
    loop_count = Application.get_env(:async_testing, :ecto_user_select_1_loop_count)
    Enum.each(1..loop_count, fn _x ->
        assert :ok = User.select_one()
    end)
  end

mix test --max-cases 384  39.876 seconds average over 3 runs
run_tests 6 20.462 seconds average over 3 runs

SELECT 1; results on Default pool
Results for AsyncTesting.Repo.query!("SELECT 1;;") with run_count of 5000
  test "ecto database select_1 test" do
    loop_count = Application.get_env(:async_testing, :ecto_user_select_1_loop_count)
    Enum.each(1..loop_count, fn _x ->
        assert :ok = User.select_one()
    end)
  end

mix test --max-cases 384  30.9 seconds average over 3 runs
run_tests 6 18.969 seconds average over 3 runs

Ok, I have one last request. Can you wrap this loop in Repo.checkout(fn -> ... end) and let us know how it goes? Thank you! :heart:

No worries, I'm happy to help. :)
SELECT 1; Repo.checkout wrap results on sandbox pool
Results for AsyncTesting.Repo.query!("SELECT 1;;") with run_count of 5000
  test "ecto database select_1 test" do
    loop_count = Application.get_env(:async_testing, :ecto_user_select_1_loop_count)
    AsyncTesting.Repo.checkout(fn ->
      Enum.each(1..loop_count, fn _x ->
          assert :ok = User.select_one()
      end)
    end)
  end

mix test --max-cases 384  29.94 seconds average over 3 runs
run_tests 6 17.32 seconds average over 3 runs

SELECT 1; Repo.checkout wrap results on default pool
Results for AsyncTesting.Repo.query!("SELECT 1;;") with run_count of 5000
code is same as above

mix test --max-cases 384  28.295 seconds average over 3 runs
run_tests 6 15.795 seconds average over 3 runs

Ok, I have two last requests to test my theory.
First, can you run the same tests above (it can be whatever pool), but making sure that only the test above is executed? Something like this mix test --only test:"test ecto database select_1 test".
Second, instead of using the connection pool, can you please start a Postgrex connection in the test and use it for the queries? It would be something like this:
@config Repo.config()
test "...." do
  {:ok, pid} = Postgrex.start_link(@config)
  enum do
    Postgres.query(pid, "SELECT 1")
  end
end

And then you can try to run tests only with this test name again, so we remove any noise from other tests.

In any case, it is expected for the connection pool to be a bottleneck at some point, as it is a single process. These last tests is going to help confirm how much of it is contributed to by the pool or if there are other factors at play.
How would we solve this bottleneck in practice? If we were seeing this issue in production, I would address it by having n connection pools and picking one at random on every operation. This is actually fairly straight-forward and it can even be solved today by using the dynamic repo functionality.
However, we are talking about tests here and a big part of the issue here is confirmed to be the ownership pool. Once we moved from the ownership pool to the default pool, we got a 10 seconds improvement. But the ownership pool really has to be a global shared resource, with no partitioning, because it is meant to be accessed by any process that you might allow in the system. Its whole job is to share a connection between processes that want to collaborate together and at the moment I cannot think of anything we can do to speed this up - but I will keep throwing brain cycles at it.

SELECT 1; Repo.checkout wrap sandbox pool this test only
Results for AsyncTesting.Repo.query!("SELECT 1;;") with run_count of 5000
  @tag :select_1
  test "ecto database select_1 test" do
    loop_count = Application.get_env(:async_testing, :ecto_user_select_1_loop_count)
    AsyncTesting.Repo.checkout(fn ->
      Enum.each(1..loop_count, fn _x ->
          assert :ok = User.select_one()
      end)
    end)
  end

mix test --only select_1 --max-cases 384  30.223 seconds average over 3 runs
run_tests 6 (with --only select_1 filter) 17.104 seconds average over 3 runs

manual postgrex connection on default pool
  @tag :manual_postgrex
  test "manual postgrex select_1 test" do
    loop_count = Application.get_env(:async_testing, :manual_postgrex_loop_count)
    {:ok, pid} = Postgrex.start_link(@config)
    Enum.each(1..loop_count, fn _x ->
      Postgrex.query!(pid, "SELECT 1;", [])
    end)
  end

mix test --only manual_postgrex --max-cases 384  27.027 seconds average over 3 runs
run_tests 6 (with --only manual_postgrex filter) 14.424 seconds average over 3 runs

So our whole test suite is async: true so I'm wondering if we should work on an option of disabling (or a readme/help/tutorial) to show people who are building async test suites how to reap a possible performance enhancement here. Right now Ecto funnels you through async: false and sandbox friendly testing so I think we need a path for the performance sensitive folks to squeeze out as much test performance as possible.

I'm 100% fine leaving sandbox behind if its causing performance issues. I just need a path to do so.

I think there is a misunderstanding. The sandbox is the reason why you can use async: true because we need a mechanism to share a connection between processes. So Ecto is pushing you towards async: true and sandboxing. Without the sandbox, everything would be async: false and execution would be much, much slower.
To sum up, here are the numbers (with `--max-cases):

With ownership pool and max cases: 40s
With default pool and max cases: 30s
With direct connection and max cases: 27s

I think we can probably do something about 1 -> 2 and make the ownership pool (and testing) faster. I don't see, at the moment, much purpose on working on 2 -> 3.

However, the fact that a direct Postgrex connection still takes roughly the same time as the pool tells me the problem is elsewhere. Which is weird, because we have kind of ruled out the problem being elsewhere. Can we please try sleep once again? Maybe our mistake was using high sleeps. What happens if you do this:
  @tag :sleep_1
  test "sleep 1" do
    loop_count = Application.get_env(:async_testing, :sleep_loop_count)
    Enum.each(1..loop_count, fn _x ->
      Process.sleep(1)
    end)
  end

@SophisticaSean can you please run the tests again with the ownership pool + select 1, but this time using this branch for db_connection.
This should reduce the bottleneck by making all reads go through a ETS table, which should be the majority of hits.

You're right, sorry about that I misunderstood what Sandbox was doing.
Here's the results for the Process.sleep(1) request and 10,000 loop_count:

mix test --only sleep_1 --max-cases 384  24.15 seconds average over 3 runs
run_tests 6 (with --only sleep_1 filter) 23.183 seconds average over 3 runs

I'll work on the results for that other branch.

SELECT 1; Repo.checkout wrap sandbox pool this test only
using this db_connection branch: https://github.com/elixir-ecto/db_connection/pull/238
Results for AsyncTesting.Repo.query!("SELECT 1;") with run_count of 5000
  @tag :select_1
  test "ecto database select_1 test" do
    loop_count = Application.get_env(:async_testing, :ecto_user_select_1_loop_count)
    AsyncTesting.Repo.checkout(fn ->
      Enum.each(1..loop_count, fn _x ->
          assert :ok = User.select_one()
      end)
    end)
  end

mix test --only select_1 --max-cases 384  30.553 seconds average over 3 runs
run_tests 6 (with --only select_1 filter) 17.188 seconds average over 3 runs

@SophisticaSean sweet, we went from 40 to 30 for the ownership tests! Can you please try running that branch in your original repo that started all this and let us know how much it improve in practice?

Here's the results for the Process.sleep(1) request and 10,000 loop_count:

Thank you! The Postgrex.start_link approach should be completely asynchronous... so I don't understand why it is running faster with partitions. It would be nice if we could now reproduce the slow down without Postgrex but it seems sleep is not enough to do it. :(

Results of the test suite this issue originated from on the new db_connection branch:

mix test --max-cases 384 24.617 second average over 3 runs
run_tests 6  17.314 second average over 3 runs

The test suite this is coming from is 762 tests and they're pretty evenly balanced between reads/writes so it looks like the change will help in the long run or for projects with many more ecto tests/ ecto reads. The averages seem to be about the same as when we started :(
I appreciate you taking the time to help me through this. I'm really eager to shave 7 seconds off our test run time if possible by alleviating this issue! :)

Results of the test suite this issue originated from on the new db_connection branch:

So no improvements unfortunately. I don't think the difference is read/writes. Instead I think in our journey to simplification, we made it so the DBConnection ownership pool became the bottleneck, but that isn't the case in your app.

I appreciate you taking the time to help me through this. I'm really eager to shave 7 seconds off our test run time if possible by alleviating this issue! :)

Yeah, unfortunately I just ran out of options. We ruled out the issue being Ecto and DBConnection and I am skeptical the issue will be Postgrex. So we either need to validate it being a Postgrex issue or try to reproduce without it.

So how can I help? The example repo still showcases the very same discrepancy I'm seeing in my private repo. I don't think there should be a ~13 second difference between partition and no partition mix test.

We need to continue boiling it down. We were already able to bring it down to Postgrex, removing Ecto from the equation. So now we need to further change the example so we get closer to the bottleneck.

Any news here on a more minimal case? :) Thanks!

Oh, I had an idea. Can we focus on the Postgrex case, where you pass the options to start it manually without using Ecto at all? Then can we benchmark using a socket vs a hostname? Just the issue is something ip related.

Closing this for now, as it is surely not an Ecto issue. If you decide to take a further look at this, let's move this to Postgrex and continue from there. Thanks!


[TE/JAX] Custom call with FFI - lowering all attributes with bind all

Description
Lowering all attributes with bind all in FusedAttnFwd.
Type of change

[ ] Documentation change (change only to the documentation, either a fix or a new content)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Infra/Build change
[ ] Code refractor

Checklist:

[x] I have read and followed the contributing guidelines
[x] The functionality is complete
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

/te-ci jax

/te-ci jax

/te-ci jax


Add @method annotations to Concurrency facade for IDE support

Added @method annotations to the Concurrency facade to enhance IDE auto-completion and documentation generation. These annotations reflect the static methods available in the ConcurrencyManager.

Added @method static array run(\Closure|array $tasks)
Added @method static DeferredCallback defer(\Closure|array $tasks)

No changes to the functionality or logic of the code. Existing tests have been run and all pass successfully.

I've opened a PR to ensure these get their docblocks auto generated via our facade documenter.
https://github.com/laravel/framework/pull/52750


Changes getByName getter to return 404s if the app (NB: it's permissi…

…on) is not found, as opposed to 403s when the app's permission doesn't exist.
This is necessary because Orca's UpsertApplicationTask first queries for the existence of an application (using the getByName() method) before invoking either the update() or create() endpoints on Front50.
tag: https://github.com/spinnaker/fiat/issues/51
@duftler @jtk54 PTAL

LGTM

LGTM

LGTM2.


Improve app responsiveness, add indicator when the user needs to wait

@linl33 Can I work on this issue?


Import-DbaCsv - ensure the reader is closed in the inner loop

Import-DbaCsv - ensure the reader is closed in the inner loop

Type of Change

[x] Bug fix (non-breaking change, fixes #7162 )
[ ] New feature (non-breaking change, adds functionality, fixes # )
[ ] Breaking change (effects multiple commands or functionality, fixes # )
[x] Ran manual Pester test and has passed (`.\tests\manual.pester.ps1)
[ ] Adding code coverage to existing functionality
[ ] Pester test is included
[ ] If new file reference added for test, has is been added to github.com/sqlcollaborative/appveyor-lab ?
[ ] Nunit test is included
[ ] Documentation
[ ] Build system

Purpose

Ensure that the file reader is closed in the inner loop where the bulk copy is being done.
Approach

Verified using Process Explorer that the file is locked. Added a finally block so that the reader will be closed in all cases.
Commands to test

See the integration tests.

@omicronx90, can you share how you integrated the proposed code changes into your local environment? Also, I recommend you add breakpoints in your script and step through your code that you listed in #7162. You can use Process Explorer to find if there are locks still present on the csv file after the exception occurs. In my testing with the updated code changes the file lock is released.

@omicronx90, can you share how you integrated the proposed code changes into your local environment? Also, I recommend you add breakpoints in your script and step through your code that you listed in #7162. You can use Process Explorer to find if there are locks still present on the csv file after the exception occurs. In my testing with the updated code changes the file lock is released.

I added the code at lines 647 - 666 as below, i have attached screenshot i used with the lines included in C:\Program Files\WindowsPowerShell\Modules\dbatools\1.0.139\functions\Import-DBACsv.ps1

`
#$reader.Close()
#$reader.Dispose()
$completed = $true
} catch {
$completed = $false
                    if ($resultcount -is [int]) {
                        Write-Progress -id 1 -activity "Inserting $resultcount rows" -status "Failed" -Completed
                    }
                    $reader.Close()
             $reader.Dispose()
                    Stop-Function -Continue -Message "Failure" -ErrorRecord $_
 ##############################
                }finally {
 try {
            $reader.Close()
             $reader.Dispose()
          } catch {
      }
	}`

@omicronx90, can you share how you integrated the proposed code changes into your local environment? Also, I recommend you add breakpoints in your script and step through your code that you listed in #7162. You can use Process Explorer to find if there are locks still present on the csv file after the exception occurs. In my testing with the updated code changes the file lock is released.

I added the code at lines 647 - 666 as below, i have attached screenshot i used with the lines included in C:\Program Files\WindowsPowerShell\Modules\dbatools\1.0.139\functions\Import-DBACsv.ps1

`
#$reader.Close()
#$reader.Dispose()
$completed = $true
} catch {
$completed = $false
                    if ($resultcount -is [int]) {
                        Write-Progress -id 1 -activity "Inserting $resultcount rows" -status "Failed" -Completed
                    }
                    $reader.Close()
             $reader.Dispose()
                    Stop-Function -Continue -Message "Failure" -ErrorRecord $_
 ##############################
                }finally {
 try {
            $reader.Close()
             $reader.Dispose()
          } catch {
      }
	}`

Hi Sorry i noticed there were new code updates i pulled the latest file with your additional changes and tested again the problem is still occurring i can see

@omicronx90, can you share how you integrated the proposed code changes into your local environment? Also, I recommend you add breakpoints in your script and step through your code that you listed in #7162. You can use Process Explorer to find if there are locks still present on the csv file after the exception occurs. In my testing with the updated code changes the file lock is released.

I took a copy of the file with the changes you made and copied the file to C:\Program Files\WindowsPowerShell\Modules\dbatools\1.0.139\functions\Import-DbaCsv.ps1

@omicronx90, if a file is updated under C:\Program Files\WindowsPowerShell\Modules\dbatools\1.0.139 it is necessary to do the following:
$dbatools_enabledebug = $true
Import-Module -Name dbatools -RequiredVersion 1.0.139 -Force

This causes the installed module to reload all of the individual command definitions and will allow you to debug with the updated code.
To summarize the steps:

Edit or replace the file at C:\Program Files\WindowsPowerShell\Modules\dbatools\1.0.139\functions\Import-DbaCsv.ps1
Run the two commands listed above to reload the dbatools module using the individual command definitions.
Run your script to verify if the issue is still occurring.

@omicronx90, if a file is updated under C:\Program Files\WindowsPowerShell\Modules\dbatools\1.0.139 it is necessary to do the following:
$dbatools_enabledebug = $true
Import-Module -Name dbatools -RequiredVersion 1.0.139 -Force

This causes the installed module to reload all of the individual command definitions and will allow you to debug with the updated code.
To summarize the steps:

Edit or replace the file at C:\Program Files\WindowsPowerShell\Modules\dbatools\1.0.139\functions\Import-DbaCsv.ps1
Run the two commands listed above to reload the dbatools module using the individual command definitions.
Run your script to verify if the issue is still occurring.

@lancasteradam  , Thanks so much for this.  I have tested again after importing the functions and it seems to be working well so far !

Looks fab to me, thank you so much 🙇


Update dependency to kubernetes 1.18.6

What this PR does / why we need it:
Updated dependency to Kubernetes v1.18.6 and tidied go.sum file with Go 1.14.4.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:

/assign @irfanurrehman @hectorj2f @jimmidyson
Kubernetes just released v1.18.6, let's keep dependency fresh.

/approve

/lgtm


Mainly to add Spectrum Protect Plus installation to GitOps framework for Production Reference Guide

Apart from the GitOps for SPP, additional logic is added to scripts/bootstrap.sh to use set-git-source.sh

Tested the script with AWS gitops project that I have - all working - got an occasional hung that will make gitops not available. Adding a 5 seconds sleep to make the behavior more consistent.


Add logging for tracing

Added, or elevated to info level, logging to indicate when detector/classifier nodes received component data and perform processing.

PTAL @cameron-a-johnson


"make load" step got stuck

I designed my own CFU to accelate FFT.
However, after make prog step was finished successfully, my terminal got stuck somewhere in make load.
make[3]: Leaving directory '/home/limx/CFU-Playground/proj/fft/build'
Running interactively on FPGA Board
make -C /home/limx/CFU-Playground/soc -f /home/limx/CFU-Playground/soc/common_soc.mk load_hook
make[3]: Entering directory '/home/limx/CFU-Playground/soc'
MAKEFLAGS=-j8 /home/limx/CFU-Playground/scripts/pyrun ./common_soc.py --output-dir build/digilent_arty.fft --csr-json build/digilent_arty.fft/csr.json --cpu-cfu  /home/limx/CFU-Playground/proj/fft/cfu.v --uart-baudrate 1843200 --target digilent_arty "--cpu-variant=perf+cfu" --toolchain symbiflow --software-load --software-path /home/limx/CFU-Playground/proj/fft/build/software.bin
INFO:Workflow:Setting sys_clk_freq to 75MHz.
make[3]: Leaving directory '/home/limx/CFU-Playground/soc'
/home/limx/CFU-Playground/soc/bin/litex_term --speed 1843200  --kernel /home/limx/CFU-Playground/proj/fft/build/software.bin /dev/ttyUSB2

It won't continue to load everything on my FPGA.
What I know is that the problem may rise from cfu.v, because if I change this file to another easy and 100% correct one, the problem disappeared.
But I cannot figure out what's wrong with my cfu.v.
module Cfu (
  input               cmd_valid,
  output              cmd_ready,
  input      [9:0]    cmd_payload_function_id,
  input      [31:0]   cmd_payload_inputs_0,
  input      [31:0]   cmd_payload_inputs_1,
  output reg          rsp_valid,
  input               rsp_ready,
  output reg [31:0]   rsp_payload_outputs_0,
  input               reset,
  input               clk
);

  wire [31:0] cfu0;
  wire signed [15:0] degree0, degree1, degree2;
  wire neg;

  assign degree0 = ($signed(cmd_payload_inputs_0[15:0]) < 0)
                 ? (16'd180 - $signed(cmd_payload_inputs_0[15:0]))
                 : $signed(cmd_payload_inputs_0[15:0]);
  assign degree1 = degree0 % 16'd180;
  assign degree2 = (degree1 > 16'd90) ? (16'd180 - degree1) : degree1;
  assign neg     = cmd_payload_inputs_1[0];
  assign cfu0    = (neg << 16) + degree2;

  // Only not ready for a command when we have a response.
  assign cmd_ready = ~rsp_valid;

  always @(posedge clk) begin
    if (reset) begin
      rsp_payload_outputs_0 <= 32'b0;
      rsp_valid <= 1'b0;
    end else if (rsp_valid) begin
      // Waiting to hand off response to CPU.
      rsp_valid <= ~rsp_ready;
    end else if (cmd_valid) begin
      rsp_valid <= 1'b1;

      rsp_payload_outputs_0 <= (cmd_payload_function_id == 10'b0)
          ? cfu0
          : 32'b0;
    end
  end
endmodule

Thanks in advance! （￣︶￣）

I find out that if I comment this line assign cfu0    = (neg << 16) + degree2; or change it to assign cfu0 = 32'b0;, it works. Weird! Why?

Well, I've changed my cfu,v and got this in my terminal.
make[3]: Leaving directory '/home/limx/CFU-Playground/soc'
/home/limx/CFU-Playground/soc/bin/litex_term --speed 1843200  --kernel /home/limx/CFU-Playground/proj/fft/build/software.bin /dev/ttyUSB1

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2022 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Feb  5 2023 22:07:56
 BIOS CRC failed (expected %x, got 0x4e20f513)
 The system will continue, but expect problems.

Then, it will print messy code.

Hi @limingxuan-pku ; hmm, these symptoms point to some part of the circuit not meeting timing and therefore corrupting data.    It looks like you are using Arty A7 board and the SymbiFlow / F4PGA toolchain, am I correct?

Hi @limingxuan-pku ; hmm, these symptoms point to some part of the circuit not meeting timing and therefore corrupting data. It looks like you are using Arty A7 board and the SymbiFlow / F4PGA toolchain, am I correct?
Yes, I use Arty A7 and SymbiFlow, working environment is Virtual Machine Ubuntu 20.04. So could you please teach me how to recognize and avoid such timing issues?

Hi @limingxuan-pku ---
The files generated by SymbiFlow will be in $CFU_PLAYGROUND/soc/build/digilent_arty.<projname>/gateware/.  In there, you can look at report* files, for example report_timing.setup.rpt.
But now that I'm looking at the files, I can't figure out where the requested 75MHz clock (from LiteX) is being passed to the SymbiFlow tools.   Specifically, the .sdc file is empty, and I don't see anything like a create_clock command in the .xdc file.
@kgugala who can I ask for more info here?


destroy is broken again

from any splom mock off https://github.com/plotly/plotly.js/pull/2505 then calling Plotly.purge(gd) (which calls splom.destroy) gives:

Fixed in https://github.com/dy/regl-scatter2d/commit/d60bfa765eb385717e7b1ba5833be44810e44ca9

thanks for fix!


chore: Remove JWK feature

Feature or Problem
In hindsight, the thing that needed to be implemented to make it possible for users of the nkeys to be able to implement this functionality for themselves, all that was needed is the change introduced in #42
This PR removes the jwk feature to better align this with the upstream nkeys module.
Related Issues

Release Information

Consumer Impact

Testing

Unit Test(s)

Acceptance or Integration

Manual Verification

As an aside, I would love to cut a new release (0.4.2?) to make the functionality from #42 available from a crate rather than having to do a git dependency to this repo to pull it in.

would love to get this in as part of #45 🙂


0.2.0

Releasing version 0.2.0.

could you just make a branch within the repo so it will perform the branch build?

gonna need to fix the conflicts

https://ci.appveyor.com/project/jkuester/cucumber-forge-desktop/builds/25593814


Promote IJ 2021.3 to Stable

Checklist

[-] I have filed an issue about this change and discussed potential changes with the maintainers.
[-] I have received the approval from the maintainers to make this change.
[+] This is not a stylistic, refactoring, or cleanup change.

Please note that the maintainers will not be reviewing this change until all checkboxes are ticked. See
the Contributions section in the README for more
details.
Discussion thread for this change
Issue number: #3001
Description of this change
Promotes 2021.3 version as stable. I've built the plugin locally and been using it with IntelliJ 2021.3.1 for couple of days with no issues.

It's good to know that the plugin is working for you without any issues in 2021.3.1. As soon as we integrated PR #3155, we can create a manual release. I'd first push it to the plugin's beta channel and a few days / a week later to the stable channel.

Sounds good. I saw the failing tests only when opened the PR. Can you elaborate on the Beta channel? I don't see it in the IJ settings like I see beta channel for Kotlin plugin for example.

If you're looking for it in 2021.3, you might not have seen the beta channel of the plugin as there wasn't any plugin version for 2021.3 available in the marketplace. I just pushed one. See https://github.com/bazelbuild/intellij/issues/2757#issuecomment-882545832 for how to sign up to the beta channel.


ApiResponses annotation is ignored when placed on an interface

To keep implementation code as short as possible, I try to put all annotations on the rest controller interface. It works with @Api and @ApiOperation but not with @ApiResponses.
Am I missing something (like enabling something in SpringSwaggerConfig) ?
Sample of not working ApiResponses :
@RequestMapping("service")
@Api(value="service", description="Operations on service")
public interface SimpleService {

    @RequestMapping(method = RequestMethod.POST, consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
    @ApiOperation(value="Register a new client", response=RegistrationDTO.class)
    @ApiResponses({
        @ApiResponse(code=201, message="Client successfuly registered"),
        @ApiResponse(code=400, message="Bad request")
    })
    ResponseEntity<RegistrationDTO> register(InformationDTO information);
}

@RestController
public class SimpleServiceImpl implements SimpleService {

    @Override
    public ResponseEntity<RegistrationDTO> register(@Validated @RequestBody InformationDTO information) {
        // (...)
    }
}

Swagger output :
"responseMessages": [
    { "code": 200, "message": null, "responseModel": "RegistrationDTO" },
    { "code": 201, "message": "Created" },
    { "code": 401, "message": "Unauthorized" },
    { "code": 403, "message": "Forbidden" },
    { "code": 404, "message": "Not Found" }
],

Sample of working ApiResponses :
@RequestMapping("service")
@Api(value="service", description="Operations on service")
public interface SimpleService {

    @RequestMapping(method = RequestMethod.POST, consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
    @ApiOperation(value="Register a new client", response=RegistrationDTO.class)
    ResponseEntity<RegistrationDTO> register(InformationDTO information);
}

@RestController
public class SimpleServiceImpl implements SimpleService {

    @Override
    @ApiResponses({
        @ApiResponse(code=201, message="Client successfuly registered"),
        @ApiResponse(code=400, message="Bad request")
    })
    public ResponseEntity<RegistrationDTO> register(@Validated @RequestBody InformationDTO information) {
        // (...)
    }
}

Swagger output :
"responseMessages": [
    { "code": 200,"message": null,"responseModel": "ReporterRegistrationDTO" },
    {"code": 201,"message": "Client successfuly registered","responseModel": "Void"  },
    {"code": 400,"message": "Bad request","responseModel": "Void" },
    {"code": 401,"message": "Unauthorized" },
    {"code": 403,"message": "Forbidden"},
    {"code": 404,"message": "Not Found"}
],

@jlamby any interest in submitted a pull request with the fix you have identified?

Yeah, sure.
I was just not quite sure if it was a bug or the expected behaviour ;)


Boresight calibration for defocused PSFs

Boresight calibration should work with at least 200nm defocused PSFs (V2 to be delivered as part of R3.0.2)

First light defocus could be ~300-500nm. I believe John provided PSFs at this range of defocus, and we should make sure we have a solution for this case.

Just to clarify, by "should work", does that mean we should be measuring the boresight offset to within requirements, even with 500 nm of defocus?

Good point. We haven't written a requirement for first light. I think our performance could be relaxed, since at first light, the observatory WFE doesn't meet its requirement either. We just need to be good enough to subsequently place an isolated star near the center of the FOV for the subsequent CGI focus and alignment activity. To throw out a number: how's 100mas (~5px) 1 sigma? We could probably relax further if needed.

After some further discussions at JPL, we have elected to change our first light plan to include a focus sweep so that DRP doesn't need to develop new functionality. (It'll also have the added benefit of making the subsequent phase retrieval easier) At first light, we will still observe the boresight field, but now we will take multiple sets of images, each with CGI's focus control mirror (FCM) at a different focus value, with focus steps of ~100nm.
The DRP would only need to produce "close enough" results on the best-focused image of this first light set. That said, given the nominal step size, we'd be a maximum of 50nm from best focus, with maybe another (ballpark) 75nm of other aberrations. @manduhmia, I remember you did some performance testing on defocused simulated images. Does DRP meet the requirement for, say, the 100nm defocus simulated image, as a proxy for rms[50nm defocus, 75nm other aberrations]?
I think this means we can ditch the R&D on improved algorithms to handle strongly defocused images (ie: don't have to do the work originally requested in this ticket!). But this dataset will be off-nominal for astrometry (N sets of M images). Does this necessitate manual image sorting and/or a helper script for the first light activity only? We can't know ahead of time which image set will be closest to focus.
A corollary - in order for someone (could be us could be CTC. Thoughts?) to use these datasets to estimate the best focus value for the FCM, they'll need clean L2b data products.  Does the astrometry module save the L2a and L2b products? If not, can that be a new issue for R3.0.2?


dotnet: infrastructure cleanup

Description of changes
This is some infrastructure fixes and improvements pulled out of #190129
@NixOS/dotnet
I've regenerated all the sdk versions/*.nix to verify that there are no changes.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

FYI @GGG-KILLER

Result of nixpkgs-review pr 290351 run on x86_64-linux 1

1 package built:

nuget-to-nix


Add Microsoft.Plugin.Uri.UnitTests to CI + Fix Tests

Summary of the Pull Request
Adds Microsoft.Plugin.Uri.UnitTests to CI
PR Checklist

[x] Applies to #5873
[x] CLA signed. If not, go over here and sign the CLA
[x] Tests added/passed
[ ] Requires documentation to be updated
[x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

Info on Pull Request
What does this include?
Validation Steps Performed

All checks should be valid
[WIP] Manually go to Azure Devops to validate the test was actually ran.

who loves CI?  <- me :)

As a note, tests are fixed but are still not actually ran :(


Don't focus the selection after using Align Transform With View

It made minor adjustments difficult as the camera moved every time Align Transform With View was used. For the record, this was added in 4a218b9862b73b643ade6e8b0f5c205e9fbc9ca4.
This closes #36738.

Thanks!

Cherry-picked for 3.2.2.


More return info on LtiOutcomesHelper methods

For one reason or another, we've been running into errors when using the
LtiOutcomesHelper, and it's
difficult to diagnose the problem when the reason is hidden. This should
expose a bit more info on why a false result is being sent back. I added
a new BasicResult class that can be implicitly cast to a bool. The root
and v1 LisResult classes now implement this BaiscResult and use its
IsValid property. For the message, Exceptions return back its ToString
result, and imsx_StatusInfoType will return the imsx_description.

@ravenclawmage That is really clever. I have wanted more information myself, but could not figure out a way to add the message without breaking interface. Thank you!

Thanks! :)


Update default version of crane from 0.14.0 to 0.19.1

Since I just tried to use crane copy --all-tags with the version installed by the setup-tools action, however, it failed with:
Error: unknown flag: --all-tags

It turns out that feature was added in 0.15.0 in May 2023, but that version is still newer than the version installed by this action.
Whilst I can manually override the version myself, it seems we should update the default version to prevent others from running into similar footguns.
https://github.com/google/go-containerregistry/compare/v0.14.0...v0.19.1

@jkutner @jjbustamante I don't suppose someone could review? :-)

Thank you :-)


Allow user to provide specific initial harmonies

A parameter called initial_harmonies, which is a list of lists (i.e., a list of incipient harmonies), is added to harmony_search(). This allows for the user to provide specific starting points for the search.
The value of initial_harmonies is passed all the way to _initialize() for injection to _harmony_memory. Checks were added to validate the length and number of dimensions of initial_harmonies. Also, some code around initializing _harmony_memory was refactored.
The change was tested in the Anaconda Python 2.7.13 environment.

Hi @gfairchild, Do you think that the error handling for checking the validity of initial_harmonies is appropriate? I am not sure where may be a better place to insert it. Also, this is my very first PR! Please let me know if I am not doing something right. Thanks!

Oh, and one other thought: could you add some examples to the examples folder that demonstrate these new features?

I am new to this PR business... Should I close this, and then create another PR? Thanks!

Also, I am thinking to show the new features in one or two of the examples you already have. What would you suggest? Thanks!

No, you don't need to create a new PR. We just work together on this one PR. I'll go back over your changes hopefully later today.

I have incorporated your comments and add an example to specify initial harmonies. Should the histories be shown as well? Or maybe just print a note saying that the history is also available?

Hi @gfairchild, You are most likely quite busy at work! It would be great to get some feedback from you! Thanks.

I'll finish reviewing this tomorrow. Sorry I've been MIA! Life is crazy right now.
A few things:

I'm wondering if the initial harmonies might be better placed in the objective function. What are your thoughts?
We'll want to update the readme, changelog, and setup.py to indicate the new changes and version bump (let's go to 1.4).

Sorry for not replying sooner! I was travelling.

RE: I'm wondering if the initial harmonies might be better placed in the objective function. What are your thoughts?
Say, if I want to run HS using multiple sets of initial harmonies, then would I need to reinitialize objective function object multiple times as well?
RE: We'll want to update the readme, changelog, and setup.py to indicate the new changes and version bump (let's go to 1.4).
Yes, will do!

This is great! 2 more small suggested changes, and then I'm happy merging this and updating PyPI:

Can you update the version number in setup.py?
Would you mind creating an AUTHORS.md file in the vein of https://github.com/gfairchild/pyxDamerauLevenshtein/blob/master/AUTHORS.md and add yourself as a contributor? I want to make sure you get proper credit.

Thank you!

Thanks, Geoffrey! Done!

Merging now! Also updating PyPI, which should be done very shortly.

This is officially live on PyPI now! Thank you very much!


Refactor response parsers. Close #81.

Before starting #84, I thought I would deal with #81 first to avoid having to refactor after.
I created a behaviour for parsers and moved the response handling there,
making it overridable in case we need to do some special handing for a driver.
What do you think?

@tuvistavie this pull request is great, I think adding tests for Hound.ResponseParser will make it even better.

@vitalis Just added some tests for Hound.ResponseParser!

@tuvistavie :+1: :+1: :+1:


insertOrder报错

Traceback (most recent call last):
File "C:\veighna_studio\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\veighna_studio\lib\threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "C:\veighna_studio\lib\site-packages\vnpy\event\engine.py", line 60, in _run
self._process(event)
File "C:\veighna_studio\lib\site-packages\vnpy\event\engine.py", line 73, in _process
[handler(event) for handler in self._handlers[event.type]]
File "C:\veighna_studio\lib\site-packages\vnpy\event\engine.py", line 73, in <listcomp>
[handler(event) for handler in self._handlers[event.type]]
File "C:\veighna_studio\lib\site-packages\vnpy_spreadtrading\engine.py", line 460, in process_tick_event
algo.update_tick(tick)
File "C:\veighna_studio\lib\site-packages\vnpy_spreadtrading\template.py", line 161, in update_tick
self.on_tick(tick)
File "C:\Github\work\vnpy_shiquan3\spread_trading\algo.py", line 128, in on_tick
self.quote_active_leg(new_quote_price)
File "C:\Github\work\vnpy_shiquan3\spread_trading\algo.py", line 192, in quote_active_leg
self.send_leg_order(
File "C:\Github\work\vnpy_shiquan3\spread_trading\algo.py", line 246, in send_leg_order
self.send_order(leg.vt_symbol, price, abs(leg_volume), Direction.SHORT)
File "C:\veighna_studio\lib\site-packages\vnpy_spreadtrading\template.py", line 273, in send_order
vt_orderids: list = self.algo_engine.send_order(
File "C:\veighna_studio\lib\site-packages\vnpy_spreadtrading\engine.py", line 608, in send_order
vt_orderid: str = self.main_engine.send_order(
File "C:\veighna_studio\lib\site-packages\vnpy_riskmanager\engine.py", line 61, in send_order
return self._send_order(req, gateway_name)
File "C:\veighna_studio\lib\site-packages\vnpy\trader\engine.py", line 188, in send_order
return gateway.send_order(req)
File "C:\veighna_studio\lib\site-packages\vnpy_yuanta\gateway\yuanta_gateway.py", line 127, in send_order
return self.td_api.send_order(req)
File "C:\veighna_studio\lib\site-packages\vnpy_tap\gateway\tap_gateway.

line 801, in send_order
error_id, session, order_id = self.insertOrder(order_req)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 0: invalid continuation byte

已经修复


Add an integration test of our import.meta support.

Will fail until https://github.com/Polymer/tools/pull/307 is merged

(master now has the import.meta stuff for bundler merged in.)


Link to Win32 binaries in program files folder cannot be invoked

A brief description

I created links (ln -s) to some Win32 binaries in program files, but they do not run when invoked.
If I directly invoke the .exe file, it will run as expected; if I create a link to something like notepad.exe, it will also work.

Expected results

The programs should start and run in windows when I type in ./link_name

Actual results (with terminal output if applicable)

No program runs. The command ends in error state. This is indicated by the color change of fish terminal. No error message is shown.

Your Windows build number

14951.rs_prerelease.161014-1700

Steps / All commands required to reproduce the error from a brand new installation

cd ~

############# These won't run ################

ln -s /mnt/c/Program\ Files\ \(x86\)/Microsoft\ VS\ Code/Code.exe vscode.exe
./vscode.exe

# with quotes in filename
ln -s "/mnt/c/Program Files (x86)/Google/Chrome/Application/chrome.exe" chrome.exe
./chrome.exe

# without .exe
ln -s "/mnt/c/Program Files (x86)/Microsoft VS Code/Code.exe" vscode
./vscode

############## All of these works: #############

/mnt/c/Program\ Files\ \(x86\)/Google/Chrome/Application/chrome.exe

/mnt/c/Program\ Files\ \(x86\)/Microsoft\ VS\ Code/Code.exe

# creating link to notepad.exe works
ln -s "/mnt/c/Windows/System32/notepad.exe" notepad
./notepad

Required packages and commands to install

Chrome and VS Code in Windows system.

This could be because you are attempting to launch an application from within VolFs rather than DrvFs
If you look at https://blogs.msdn.microsoft.com/wsl/2016/10/19/windows-and-ubuntu-interoperability/ it mentions

NT binaries can only run out of DrvFs paths (for example: /mnt/c/Windows/System32).

@MartinLenord If I understand VolFs and DrvFs correctly, then I do not think this is the case.
The binaries themselves are mounted both on DrvFs. Both Program Files (x86)/ and Windows/System32/ are in /mnt/c/.
The symbolic link files are stored in the same folder. I have tried several different places, including home directory and /tmp, but the results are the same.
If it is a problem related to file system, then chrome.exe and Code.exe should produce the same result as notepad.exe.
What's really confusing is that neither symbolic link nor program files directory causes the problem. It is when they are combined that the feature stops working.

How about a workaround: try a bash alias (or maybe the corresponding functionality in whatever shell you are using)? I just tried
alias notepad='/mnt/c/Program\ Files\ \(x86\)/Internet\ Explorer/iexplore.exe'
and "notepad" now works to start the windows program.

@MartinLenord - What I mean by running the binary out of VolFs (or LxFs as we're referring to it now) is that the if the executable itself is in VolFs it will not work. For example /home/user/foo.exe.  If you run an exe out of /mnt/c/... with a VolFs current working directory the binary will launch, but the current working directory will not propagate.
@PowerSnail - If I understand you correctly you're only having issues with vscode.exe?  If so, I suspect you're hitting a name collision with vscode.exe.  Can you run the following command?
which vscode.exe

It looks like this was resolved by one of @SvenGroot's symlink changes after Creator's Update.


Improve performance

Context
This PR creates new Item, Entry and Record classes which each know how to parse a string of RSF which represents them. In combination with the changes made to the download of an RSF file when first creating a new RegisterClient, objects of the above classes only need to parse data when they are being accessed by the client, removing the burden of additional memory overhead until necessary.
Through this refactor, we improve the memory performance of creating a new RegisterClient quite substantially (see details on the related Trello card for details).
Changes proposed in this pull request
Restructure the way in which items, entries and records and stored and parsed
Add tests for client library (existing and new changes)
Guidance to review

Version bumped to 0.4.0.


Docs are simply not clear

I have been trying to get the basic (local) use case working for the better part of a day and I don't even have it hooked up to a db, and yet things don't make sense or even work.
I hope I'm wrong but an API should not be more complicated than the 2 or 3 IF / ELSE conditions it needs to wrap around, ugh.
Its not clear how to provide authorization checking on all non /login /logout calls that need to be protected. The docs for serializeUser don't seem to suggest doing anything. The language looks like passport does this for you, but then the code examples seem like its really an API call a user would need to make / create. Where does this fit in with routes?
Where is the API docs for each passport function located? It looks like some methods can take an object param, others a function, this really needs to be explained somewhere other than inspecting code or looking at examples which don't fully explain behavior.
I take issue with this paragraph...
"Each subsequent request will not contain credentials, but rather the unique cookie that identifies the session. In order to support login sessions, Passport will serialize and deserialize user instances to and from the session."
I'm using session and there is no SetCookie header being sent back from the server and I have done the login as suggested by the docs. What is the name of the cookie? More so, this last sentence to me say passport will serialize / deserialize for me, but why are there samples of the API calls right below?
Is req.IsAuthenticated() supposed to do something here? It seems to be important except for 2 reasons.

Its not in the docs
Its not a function (at least in my code, its not a method inside the request object)

How does passport add this method to the request object? seems like its some magically middleware step that forgot to make its way into the docs?

seems like this bit is missing somewhere
var isAuthenticated = function (req, res, next) {
if (req.session.passport && req.session.passport.user) {
return next();
}
else {
res.send(401, {'message': 'Not authorized'});
}
}
...
app.use('/some/path/that/requires/auth', isAuthenticated, require('./server/routes/somerouter').router);
so this seems silly / bad to do a check on the session like that, or rather the passport should be doing this for me right? I mean its creating the session value "user", seems bad I'm guessing at that being there. What am I missing here...

Passport has been terribly confusing for me as well. Why is isAuthenticated() completely undocumented? I'd be happy to contribute documentation myself, but I'm currently having trouble finding where the documentation repo is.

This video really helped me.
https://www.youtube.com/watch?v=twav6O53zIQ
I had a similar problem, after looking at lots of examples I couldn't even get it working at all.  The video covers almost everything you need for a simple LocalStrategy setup using passport-local, the passport docs were good enough to fill in the rest.  Hope that helps

Everyone knows. There is a great course on pluralsight.

req.isAuthenticated() will check whether the user has an authenticated session (i.e. logged in at some point in the past). If it returns false you need to redirect to a route that authenticates the user with passport. In our application we redirect to /login.
If you want to check whether they are authenticated on a request and automatically redirect to /login, I would highly recommend checking out the connect-ensure-login middleware.

I think my issue is req.isAuthenticated is undefined. So perhaps I did not set this all up correctly. I'll take a look at  connect-ensure-login. I just created my own isAuthenticated method and I use that instead which seems to work but also seems hackery. I would like to use the official function.

Oh I see, in that example, isAuthenticated is supposed to be undefined if the user is not authenticated? I see the confusion. Its used as a 1. method and as an 2. undefined type. That is a very difficult assumption to make as I assumed it was always a method that would return true or false

If it is undefined then your middlewares may not be in the correct order.

Great, that helps, I think i figured out my issue. Docs still need to show this in action tho.

Actually the docs are very clear but only when read carefully. To quote the first couple of sentences:

Passport's sole purpose is to authenticate requests, which it does through an extensible set of plugins known as strategies. Passport does not mount routes or assume any particular database schema, which maximizes flexibility and allows application-level decisions to be made by the developer.

Passport is structured to just authentication middleware. You're trying to ask questions that Passport should not be able to answer. If Passport included a default (de)serialisation method for users it would have to make an irrelevant assumption about what users are and how they are structured.
I suggest you look at pre-defined strategies for starting with Passport. For your specific case I'd suggest looking the basic LocalStrategy. The server.js file has plenty of comments to explain exaclty how passport works and is easy to understand. https://github.com/passport/express-4.x-local-example

I understand you. But you're missing my point I think. Again quoting from the first paragraph:

The API is simple: you provide Passport a request to authenticate, and Passport provides hooks for controlling what occurs when authentication succeeds or fails.

isAuthenticated() is not documented as it is not considered a core feature of passport. How it made its way into request.js is unclear to me. But you can see that since 2012 @jaredhanson has taken a stance against implementing session management into passport.
Passport.js only provides a protocol that helps authenticating users. It can manage a session but is not intended to do so. You can always fix the documentation if you disagree with this To do so check: https://github.com/passport/www.passportjs.org.

I just have to +1 the need for more documentation on Passport, particularly thins like serializeUser which are simply not explained fully.  This Stack Overflow question: https://stackoverflow.com/questions/27637609/understanding-passport-serialize-deserialize has 38 upvotes, 10 favorites, and has been viewed 27,273 times.  This Stack Overflow question: https://stackoverflow.com/questions/27637609/understanding-passport-serialize-deserialize has 157 upvotes, 81 favorites, and has been viewed 47,552 times!
When 70k viewers (which admittedly might be "only" 10k actual people) are going to Stack Overflow to because the documentation isn't providing them with the answers, I think that's a sign the docs could use a few more paragraphs of explanation.

Passport has been terribly confusing for me as well. Why is isAuthenticated() completely undocumented? I'd be happy to contribute documentation myself, but I'm currently having trouble finding where the documentation repo is.

Its 2024 and still not documented


GetMySavedTracks returns 20 tracks, regardless

Seemingly, 20 tracks are always returned, even if another number less than 51 is requested. Not a big issue since 20 works just fine, but herebye reported as a discrepancy to the API specification. 50 tracks worked a year ago.

@chjan Are you using the method to which you can pass options (https://github.com/kaaes/spotify-web-api-android/blob/ac9b499a58d05364eb8fe0cf154d1247a6af9581/spotify-api/src/main/java/kaaes/spotify/webapi/android/SpotifyService.java#L1109) ?

Confirmative. I use these options, then with length set to 50. 20 is returned. Maybe something wrong here from my side?
Map<String, Object> options = new HashMap<String, Object>();         {             options.put("offset", offset);             options.put("length", length);         }

You should be using "limit" option, not "length"

@XbakkeyX Thanks for spotting this! Yes, the parameter should be limit, as described on the documentation.
I'll close this ticket since this doesn't seem to be a bug in the library.


node-neigh: Fix node removal and invalid neigh entry due to buggy arping response correlation

Please see commit messages.
For backporters: please ignore the last commit.

test-me-please

test-me-please

test-net-next

test-net-next

test-4.9

test-4.9

test-gke

test-gke

test-gke

test-gke

test-me-please

test-me-please

test-me-please

test-me-please

test-net-next

test-net-next

test-net-next

test-net-next

test-me-please

test-me-please

test-me-please

test-me-please


Fulfill conditions of sale when the apartment's housing company is released from regulation.

Hitas Pull Request
Description
Fulfill conditions of sale when the apartment's housing company is released from regulation.
Pull request checklist
Check the boxes for each DoD item that has been completed:

Testing

[x] Changes have been tested
[x] Automatic tests have been added

Database

[ ] Database migrations will work in the DEV & TEST environments
[ ] initial.json has been updated to work with migrations
[ ] Oracle migration has been updated

Documentation

[ ] Tooltips have been added in the frontend for all new fields
[ ] OpenAPI definitions have been updated
[ ] Test instructions have been written for the customer in the appropriate ticket in Jira
[ ] Terminology page in Confluence has been updated

Test plan

[x] Automated tests

Tickets
This pull request resolves all or part of the following ticket(s): HT-574

Added logic for calling fulfill_conditions_of_sales_for_housing_companies in the housing company update endpoint. Both the housing company edit page and thirty-year regulation released by plot department function should use this endpoint.

Made updating improvements optional, since when the housing company update endpoint is used to manually update regulation status on to released by the plot department, the frontend doesn't need to know the state of improvements at that time.

I'll have to duplicate some code to add a patch method, but it can be done.


Gitlab Issue / Merge Request template.

https://docs.gitlab.com/ee/user/project/description_templates.html

Close as not planned currently.