Add verify scripts for make gen to run during PR

This PR fixes: https://github.com/kubernetes-sigs/descheduler/issues/483
As part of this change, we want to add steps to ensure that make gen has been run and there are no changes being generated during the PR.
For now, I've tested the code in local for the steps which I've added in the Makefile and below is the output for same,
 ->  make verify-gen
./hack/verify-makegen.sh

Diff in the PR has changed
make: *** [Makefile:112: verify-gen] Error 1

Since it's reporting error, which means it's pointing that the generators have been downloaded at the latest in the _output directory in my local.

I signed it

/ok-to-test
/refresh

/check-cla

@damemi also, I've come across a doubt during local execution. When I run make gen in my local I see the below output,
->  make gen
./hack/update-generated-conversions.sh
./hack/update-generated-deep-copies.sh
./hack/update-generated-defaulters.sh

But, when I explicitly run this command L4, I'm getting below error as well,
->  ./conversion-gen "k8s.io/code-generator/cmd/conversion-gen"
F0228 21:55:13.353319    8843 conversion.go:204] Failed loading boilerplate: open /Users/pravaragrawal/Work/go_work/src/k8s.io/code-generator/hack/boilerplate.go.txt: no such file or directory
goroutine 1 [running]:

Is this error something which is to be ignored at this point?

"k8s.io/code-generator/cmd/conversion-gen"

@pravarag the command you linked to is a go build command that is attempting to compile the conversion-gen binary. It looks like you are running it as a command itself, which is trying to run conversion-gen with its own package as a parameter.
So yes, you can ignore this error. In general, trying to run these bash scripts line-by-line may not work very well (due to imports of other init files setting variables etc). You should just be running make gen, or the individual gen targets/scripts

@pravarag you might also be able to take a look at the PR I just opened to add a similar verify script for the table of contents: https://github.com/kubernetes-sigs/descheduler/pull/516
That could help you on these other generators. We could also probably move some of the redundant code to a lib file so it isn't run over and over, but that's not vital.
A problem I found is that the files from _output (ie, our generator binaries) aren't copied to the temp directory with git archive (since they are .gitignore'd). So I had to manually copy that directory like https://github.com/kubernetes-sigs/descheduler/pull/516/files#diff-d257d22e73af313e0d4311fccbdd5054c2affab762a1d2965fd68217f0331963R36
Hope this helps

A problem I found is that the files from _output (ie, our generator binaries) aren't copied to the temp directory with git archive (since they are .gitignore'd). So I had to manually copy that directory like https://github.com/kubernetes-sigs/descheduler/pull/516/files#diff-d257d22e73af313e0d4311fccbdd5054c2affab762a1d2965fd68217f0331963R36

Thanks @damemi , I'll check the changes done in mentioned PR.

@damemi I've made some changes and pushed the latest code taking reference from the script you suggested in previous comments. But, I keep getting bad substitution error on this line and I'm guessing it's something to do with how diff _output is being used. So, one doubt here - since in this case as when we run the generators, I don't see a file other than the ones generated in _output directory which may get changed unlike README.md in for this change. Hence, which two files I must be comparing here to check the diff output and proceed based on that?

@pravarag the script shouldn't be running diff _output, because _output is just the temporary directory where the generator binaries are downloaded to.
If you see this line in verify-vendor.sh, it is calling:
diff -Naupr --ignore-matching-lines='^\s*\"GoVersion\":' go.mod "${_deschedulertmp}/go.mod"

which is essentially
diff go.mod {tmpdir}/go.mod

Because when we update the vendor, we expect changes to go.mod.
If you are running the generators for this PR without any significant changes to the code (for example, modifying the API) then you will not see any generated changes besides.
As an example, if you check out this commit from my fork and run make gen, you'll see that it adds the following to pkg/api/v1alpha1/zz_generated.conversion.go:
diff --git a/pkg/api/v1alpha1/zz_generated.conversion.go b/pkg/api/v1alpha1/zz_generated.conversion.go
index 727b9c776..8f56bf309 100644
--- a/pkg/api/v1alpha1/zz_generated.conversion.go
+++ b/pkg/api/v1alpha1/zz_generated.conversion.go
@@ -151,6 +151,7 @@ func autoConvert_v1alpha1_DeschedulerStrategy_To_api_DeschedulerStrategy(in *Des
        out.Enabled = in.Enabled
        out.Weight = in.Weight
        out.Params = (*api.StrategyParameters)(unsafe.Pointer(in.Params))
+       out.Foo = in.Foo
        return nil
 }

@@ -163,6 +164,7 @@ func autoConvert_api_DeschedulerStrategy_To_v1alpha1_DeschedulerStrategy(in *api
        out.Enabled = in.Enabled
        out.Weight = in.Weight
        out.Params = (*StrategyParameters)(unsafe.Pointer(in.Params))
+       out.Foo = in.Foo
        return nil
 }

So, if you imagine a PR that just has the above commit (and did not run make gen), then your verify script would do a couple things when it runs in CI:

Copy the current entire descheduler code directory to a {tmpdir}
pushd into the {tmpdir} (should be something like _tmp/kube-vendor.ZhSdEN/descheduler if you are running locally)
Run make gen (or each individual script) inside the {tmpdir}
popd out to the real directory
Compare the current directory to the tmpdir with something like diff -Naupr pkg/ {deschedulertmp}/descheduler/pkg/ (any generated changes will just be under pkg/)

So, what we need to do now is play with the calls to the hack scripts within your verify scripts. Specifically, we need to find a way to pass the new "PRJ_PREFIX" (deschedulertmp) to these binaries to fill in those parameters. I'll also play around with this, but if you find a solution before me feel free to post it :)

Hi @damemi thanks again for your review and as suggested, I've tried to find a way around it but I'm not sure whether that's a valid one or not.
So, what I've thought is to add an export PRJ_PREFIX=$_deschedulertmp here after this line and then maybe we might be able to change the actual value inside generator script for conversions to that of _deschedulertmp. And then one we have calculated the diff, we can do an unset PRJ_PREFIX here on this line. So this may sound not-so-best solution around it and I'm still trying to test it in local if that actually works. But since, we might not want to make any modifications in the generator script so might have to try this way 🤔

Hi @pravarag

So, what I've thought is to add an export PRJ_PREFIX=$_deschedulertmp here after this line and then maybe we might be able to change the actual value inside generator script for conversions to that of _deschedulertmp.

This is actually along the lines of how I was thinking we could fix this, so I think we're on the right track. However, I checked out your updated scripts and ran them with a fake change (same as above https://github.com/kubernetes-sigs/descheduler/pull/507#issuecomment-795883719) but it looks like they are still doing the same thing (modifying the root files, and then all scripts are failing with a diff).
It's important that when testing these, we not only test the "passing" case, but also the "failure" case. So when you test it locally, you should try making an API change (like above) and verify that the script fails as expected. In other words, when only conversions should fail, make verify-gen should only print a failure for that script and not modify any files in the root project tree.
I did find that manually copying the generator scripts over and replacing the input-dir with a leading ./ prefix worked (see this commit to see what I mean: https://github.com/damemi/descheduler/commit/bfd9658359ddc2fef758bf07ceda6ff18bdabdfd).
$ make verify-gen
ls: _output/bin/golangci-lint: No such file or directory
./hack/verify-conversions.sh
PRJ_PREFIX=: ,./hack/../_tmp/kube-verify.rtXrac/descheduler
Generated output differs:
diff -Naupr pkg/api/v1alpha1/zz_generated.conversion.go ./hack/../_tmp/kube-verify.rtXrac/descheduler/pkg/api/v1alpha1/zz_generated.conversion.go
--- pkg/api/v1alpha1/zz_generated.conversion.go	2021-03-15 16:17:34.000000000 -0400
+++ ./hack/../_tmp/kube-verify.rtXrac/descheduler/pkg/api/v1alpha1/zz_generated.conversion.go	2021-03-15 16:17:48.000000000 -0400
@@ -125,6 +125,7 @@ func autoConvert_v1alpha1_DeschedulerPol
 	out.EvictLocalStoragePods = (*bool)(unsafe.Pointer(in.EvictLocalStoragePods))
 	out.IgnorePVCPods = (*bool)(unsafe.Pointer(in.IgnorePVCPods))
 	out.MaxNoOfPodsToEvictPerNode = (*int)(unsafe.Pointer(in.MaxNoOfPodsToEvictPerNode))
+	out.Foo = in.Foo
 	return nil
 }

@@ -139,6 +140,7 @@ func autoConvert_api_DeschedulerPolicy_T
 	out.EvictLocalStoragePods = (*bool)(unsafe.Pointer(in.EvictLocalStoragePods))
 	out.IgnorePVCPods = (*bool)(unsafe.Pointer(in.IgnorePVCPods))
 	out.MaxNoOfPodsToEvictPerNode = (*int)(unsafe.Pointer(in.MaxNoOfPodsToEvictPerNode))
+	out.Foo = in.Foo
 	return nil
 }

Generated conversions verify failed.
Generated conversions verified.
./hack/verify-deep-copies.sh
Generated deep-copies verified.
./hack/verify-defaulters.sh
Generated Defaulters verified.

However trying to set PRJ_PREFIX="." like how you are currently doing it did not. This is interesting, and we may want to look at that more.
For now, I will let you decide if you would like to implement these changes as I did in the commit I shared or continue investigating better ways we can effectively call the scripts directly. @ingvagabund and @seanmalloy wdyt?

Hi @damemi thanks for correcting me in above comment and I'll surely test by making a fake API change further. Also, I can continue to investigate a little more in finding a different way if we are not planning to merge this change urgently 🤔 Otherwise, we'll go with how you have suggested by copying the generator scripts manually.

/kind cleanup

Also, I can continue to investigate a little more in finding a different way if we are not planning to merge this change urgently

@pravarag there is no strict timeline for getting this merged. Feel free to investigate further and come up with the best solution.

I attempted to find an example of another k8s project that is doing a similar type of verification. I found the below example from the kind project.
Does the verification:
https://github.com/kubernetes-sigs/kind/blob/64ac2e7f5d47e391432aa13898b78aaa0b98e7a4/hack/make-rules/verify/generated.sh#L56-L74
Does the code generation:
https://github.com/kubernetes-sigs/kind/blob/64ac2e7f5d47e391432aa13898b78aaa0b98e7a4/hack/make-rules/update/generated.sh#L34-L37
Not saying we should do it exactly like this, but seeing another example might help.

@damemi @seanmalloy I tried to look for another way in order to pass ${PRF_PREFIX} value to generators scripts, but couldn't find any. The main problem which I've realized:
The init.sh script which will run at the beginning of running generators inside generator script. And since it'll set PRJ_PREFIX value back to Descheduler root, that's why I think exporting PRJ_PREFIX didnt' work like @damemi mentioned here.
I even tried to check references as suggested by @seanmalloy above pointing to kind project but that also didn't work somehow in our case. Even though they have similar structure but they are calculating REPO_ROOT here before running generators and I feel that could be the difference. So for implementing something like that in our case, it might need a change in generator script or the init.sh script on how we have defined the PRJ_PREFIX currently.
There was one more way I was trying to implement (using sed to modify PRJ_PREFIX value) which is based on this SO example I found here but that also didn't work somehow.
For now, I've pushed the changes in all three verify scripts similar to how @damemi suggested in his last comment by copying generators in verify scripts. Let me know if we are good to go or not 🤞🏾

/test pull-descheduler-verify-master

Let me know if we are good to go or not.

@pravarag looks like one of the CI checks is failing. I can review again once the CI check is passing. Let me know if you need any help getting the CI check to pass.
Thanks!

@damemi @seanmalloy for the above CI check error, I can see that after I've added this change, the verify-defaulters.sh script is throwing below error,
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x12cbce7]

goroutine 1 [running]:
k8s.io/gengo/examples/defaulter-gen/generators.getManualDefaultingFunctions(0xc004b14a10, 0x0, 0xc00ab92ab0)
	/Users/pravaragrawal/Work/go_work/src/github.com/pravarag/descheduler/_tmp/kube-verify.MMMDRZ/descheduler/vendor/k8s.io/gengo/examples/defaulter-gen/generators/defaulter.go:148 +0xa7
k8s.io/gengo/examples/defaulter-gen/generators.Packages(0xc004b14a10, 0xc00011cf00, 0x13719ab, 0x6, 0xc004b14a10)
	/Users/pravaragrawal/Work/go_work/src/github.com/pravarag/descheduler/_tmp/kube-verify.MMMDRZ/descheduler/vendor/k8s.io/gengo/examples/defaulter-gen/generators/defaulter.go:274 +0x75f
k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc00011cf00, 0xc00007d050, 0x13719ab, 0x6, 0x139ef48, 0x0, 0x0)
	/Users/pravaragrawal/Work/go_work/src/github.com/pravarag/descheduler/_tmp/kube-verify.MMMDRZ/descheduler/vendor/k8s.io/gengo/args/args.go:206 +0x1a9
main.main()
	/Users/pravaragrawal/Work/go_work/src/github.com/pravarag/descheduler/_tmp/kube-verify.MMMDRZ/descheduler/vendor/k8s.io/code-generator/cmd/defaulter-gen/main.go:76 +0x385

I'm wondering if it could be due to the passing of PRJ_PREFIX="." instead of previous value is making this happen. The other two scripts did work fine. And I think for the mentioned path which would be something like "./pkg/apis/componentconfig/v1alpha1,./pkg/api/v1alpha1" , the generator code is not able to obtain the required package from the downloaded ones. Please correct me if I'm wrong here in my understanding.

./hack/verify-conversions.sh
Generated output differs:
diff -Naupr pkg/api/v1alpha1/zz_generated.conversion.go ./hack/../_tmp/kube-verify.Tunn7Y/descheduler/pkg/api/v1alpha1/zz_generated.conversion.go
--- pkg/api/v1alpha1/zz_generated.conversion.go	2021-03-31 09:20:26.448978691 +0000
+++ ./hack/../_tmp/kube-verify.Tunn7Y/descheduler/pkg/api/v1alpha1/zz_generated.conversion.go	2021-03-31 09:22:10.661940342 +0000
@@ -123,6 +123,7 @@ func autoConvert_v1alpha1_DeschedulerPol
 	out.Strategies = *(*api.StrategyList)(unsafe.Pointer(&in.Strategies))
 	out.NodeSelector = (*string)(unsafe.Pointer(in.NodeSelector))
 	out.EvictLocalStoragePods = (*bool)(unsafe.Pointer(in.EvictLocalStoragePods))
+	out.EvictSystemCriticalPods = (*bool)(unsafe.Pointer(in.EvictSystemCriticalPods))
 	out.IgnorePVCPods = (*bool)(unsafe.Pointer(in.IgnorePVCPods))
 	out.MaxNoOfPodsToEvictPerNode = (*int)(unsafe.Pointer(in.MaxNoOfPodsToEvictPerNode))
 	return nil
@@ -137,6 +138,7 @@ func autoConvert_api_DeschedulerPolicy_T
 	out.Strategies = *(*StrategyList)(unsafe.Pointer(&in.Strategies))
 	out.NodeSelector = (*string)(unsafe.Pointer(in.NodeSelector))
 	out.EvictLocalStoragePods = (*bool)(unsafe.Pointer(in.EvictLocalStoragePods))
+	out.EvictSystemCriticalPods = (*bool)(unsafe.Pointer(in.EvictSystemCriticalPods))
 	out.IgnorePVCPods = (*bool)(unsafe.Pointer(in.IgnorePVCPods))
 	out.MaxNoOfPodsToEvictPerNode = (*int)(unsafe.Pointer(in.MaxNoOfPodsToEvictPerNode))
 	return nil
Generated conversions verify failed.
Generated conversions verified.
./hack/verify-deep-copies.sh
Generated deep-copies output differs:
diff -Naupr pkg/api/v1alpha1/zz_generated.deepcopy.go ./hack/../_tmp/kube-verify.O6DRe4/descheduler/pkg/api/v1alpha1/zz_generated.deepcopy.go
--- pkg/api/v1alpha1/zz_generated.deepcopy.go	2021-03-31 09:20:26.448978691 +0000
+++ ./hack/../_tmp/kube-verify.O6DRe4/descheduler/pkg/api/v1alpha1/zz_generated.deepcopy.go	2021-03-31 09:22:19.018658887 +0000
@@ -46,6 +46,11 @@ func (in *DeschedulerPolicy) DeepCopyInt
 		*out = new(bool)
 		**out = **in
 	}
+	if in.EvictSystemCriticalPods != nil {
+		in, out := &in.EvictSystemCriticalPods, &out.EvictSystemCriticalPods
+		*out = new(bool)
+		**out = **in
+	}
 	if in.IgnorePVCPods != nil {
 		in, out := &in.IgnorePVCPods, &out.IgnorePVCPods
 		*out = new(bool)
diff -Naupr pkg/api/zz_generated.deepcopy.go ./hack/../_tmp/kube-verify.O6DRe4/descheduler/pkg/api/zz_generated.deepcopy.go
--- pkg/api/zz_generated.deepcopy.go	2021-03-31 09:20:26.448978691 +0000
+++ ./hack/../_tmp/kube-verify.O6DRe4/descheduler/pkg/api/zz_generated.deepcopy.go	2021-03-31 09:22:19.010658199 +0000
@@ -46,6 +46,11 @@ func (in *DeschedulerPolicy) DeepCopyInt
 		*out = new(bool)
 		**out = **in
 	}
+	if in.EvictSystemCriticalPods != nil {
+		in, out := &in.EvictSystemCriticalPods, &out.EvictSystemCriticalPods
+		*out = new(bool)
+		**out = **in
+	}
 	if in.IgnorePVCPods != nil {
 		in, out := &in.IgnorePVCPods, &out.IgnorePVCPods
 		*out = new(bool)
Generated deep-copies verify failed.
Generated deep-copies verified.
./hack/verify-defaulters.sh

It looks like you might just need to run make gen. I think https://github.com/kubernetes-sigs/descheduler/pull/523 made these changes and forgot to run the generator, so now we are catching it in this PR (meaning the test works! Woo!)
You can just add that change in a new commit, since it's separate from the real work here

It looks like you might just need to run make gen. I think #523 made these changes and forgot to run the generator, so now we are catching it in this PR (meaning the test works! Woo!)
You can just add that change in a new commit, since it's separate from the real work here

@pravarag I submitted PR https://github.com/kubernetes-sigs/descheduler/pull/541 to fix the generated code on the master branch. We can do the final review on your PR once then generated code is fixed on master. Thanks!

@pravarag I submitted PR #541 to fix the generated code on the master branch. We can do the final review on your PR once then generated code is fixed on master. Thanks!

Thanks @seanmalloy for the make gen changes and I could see that the PR has been merged. I did a rebase and ran again but it still seems to be failing 😕

Yeah, the issue I pointed out was separate and was fixed by merging the make gen changes. The seg fault is still there (I was hoping it would also be resolved by make gen but it seems it wasn't unfortunately)

Yeah, the issue I pointed out was separate and was fixed by merging the make gen changes. The seg fault is still there (I was hoping it would also be resolved by make gen but it seems it wasn't unfortunately)

I was thinking if it could be coming due to me running a lower version of go (1.15) in my local. But I upgraded it to 1.16.2 and still the same result. I think it's not able to download or may be refer to some packages under _tmp directory while running verify-defaulters.sh with the path=./hack. It's strange that it's coming for default generators only 😕

Yeah, the issue I pointed out was separate and was fixed by merging the make gen changes. The seg fault is still there (I was hoping it would also be resolved by make gen but it seems it wasn't unfortunately)

@damemi I was thinking if it could be coming due to me running a lower version of go (1.15) in my local. But I upgraded it to 1.16.2 and still the same result. I think it's not able to download or may be refer to some packages under _tmp directory while running verify-defaulters.sh with the path=./hack. It's strange that it's coming for default generators only 😕
Update: in my local I tried to build the changes within $GOPATH thinking if that could be the reason pointing to the SIGSEV error but unfortunately no luck there as well. @seanmalloy @damemi could it be related to upcoming 1.21 changes or shall I wait until 1.21 changes are merged if any?

@pravarag I did not look into the seg fault error in detail. It sound like only the verify-defaulters.sh script is causing the seg fault. One simple idea would be to do multiple smaller PRs. You could submit a separate PR to only add the hack/verify-conversions.sh script. Then if that works do another PR to add hack/verify-deep-copies.sh.

@pravarag I did not look into the seg fault error in detail. It sound like only the verify-defaulters.sh script is causing the seg fault. One simple idea would be to do multiple smaller PRs. You could submit a separate PR to only add the hack/verify-conversions.sh script. Then if that works do another PR to add hack/verify-deep-copies.sh.

Thanks @seanmalloy , as you suggested I've shortened this PR to include changes for verify-conversions.sh only. And if this goes well, I'll create two other PRs for verify-deep-copies.sh & verify-defaulters.sh.

Hi @pravarag, thanks for working on this so far. I agree that we can just merge the verify checks for the 2 scripts we know are working and continue to debug why defaulters fails like that. Thanks for sticking with this 🙂
/approve

Thanks @damemi , it was all your help :) and it's great to see my first PR coming towards a closure. Also, this PR only consists changes for verify-conversions so do you want me to include changes for verify-deep-copies as well along with this PR and take verify-defaulters separately? Or like how @seanmalloy mentioned above, three PRs for respective three generators?

three PRs for respective three generators

I think we can do three separate PRs. One for each generator.

So the automated CI checks all work. But I get the below error when running make verify-gen from my laptop. I get the same error on Mac OSX and Linux.
$ make verify-gen
ls: cannot access '_output/bin/golangci-lint': No such file or directory
./hack/verify-conversions.sh
mktemp: failed to create directory via template ‘./hack/../_tmp/kube-verify.XXXXXX’: No such file or directory
make: *** [verify-gen] Error 1

@pravarag were you able to successfully run make verify-gen from your local computer? Am I doing something wrong?

@pravarag were you able to successfully run make verify-gen from your local computer? Am I doing something wrong?

@seanmalloy here's an output from my local,
-bash>descheduler(add-verify-scripts)$
 ->  pwd
/Users/pravaragrawal/Work/go_work/src/github.com/pravarag/descheduler
-bash>descheduler(add-verify-scripts)$
 ->  make verify-gen
./hack/verify-conversions.sh
Generated conversions verified.
-bash>descheduler(add-verify-scripts)$
 ->  ls -latr
total 156
drwxr-xr-x  4 pravaragrawal staff   128 Feb 15 19:57 .github
-rw-r--r--  1 pravaragrawal staff  1591 Feb 15 19:57 CONTRIBUTING.md
-rw-r--r--  1 pravaragrawal staff 11357 Feb 15 19:57 LICENSE
-rw-r--r--  1 pravaragrawal staff   617 Feb 15 19:57 SECURITY_CONTACTS
drwxr-xr-x  3 pravaragrawal staff    96 Feb 15 19:57 charts
-rw-r--r--  1 pravaragrawal staff   886 Feb 15 19:57 cloudbuild.yaml
drwxr-xr-x  3 pravaragrawal staff    96 Feb 15 19:57 cmd
-rw-r--r--  1 pravaragrawal staff   148 Feb 15 19:57 code-of-conduct.md
drwxr-xr-x  6 pravaragrawal staff   192 Feb 15 19:57 examples
drwxr-xr-x  5 pravaragrawal staff   160 Feb 15 19:57 kubernetes
drwxr-xr-x  3 pravaragrawal staff    96 Feb 15 19:58 _output
drwxr-xr-x  6 pravaragrawal staff   192 Mar  8 20:22 ..
-rw-r--r--  1 pravaragrawal staff   200 Mar 14 12:38 .golangci.yml
-rw-r--r--  1 pravaragrawal staff    65 Apr  4 10:16 .gitignore
-rw-r--r--  1 pravaragrawal staff   949 Apr  4 10:16 Dockerfile
-rw-r--r--  1 pravaragrawal staff   777 Apr  4 10:16 Dockerfile.dev
-rw-r--r--  1 pravaragrawal staff   219 Apr  4 10:16 OWNERS
drwxr-xr-x  6 pravaragrawal staff   192 Apr  4 10:16 docs
-rw-r--r--  1 pravaragrawal staff   432 Apr  4 10:16 go.mod
-rw-r--r--  1 pravaragrawal staff 65877 Apr  4 10:16 go.sum
drwxr-xr-x  3 pravaragrawal staff    96 Apr  4 10:16 metrics
drwxr-xr-x  7 pravaragrawal staff   224 Apr  4 10:16 pkg
drwxr-xr-x 12 pravaragrawal staff   384 Apr  4 10:16 vendor
-rw-r--r--  1 pravaragrawal staff  4338 Apr  7 08:58 Makefile
drwxr-xr-x 30 pravaragrawal staff   960 Apr  7 08:58 .
-rw-r--r--  1 pravaragrawal staff 25846 Apr  7 08:58 README.md
drwxr-xr-x 19 pravaragrawal staff   608 Apr  7 08:58 hack
drwxr-xr-x  6 pravaragrawal staff   192 Apr  7 08:58 test
drwxr-xr-x 70 pravaragrawal staff  2240 Apr  7 08:58 _tmp
drwxr-xr-x 16 pravaragrawal staff   512 Apr  7 08:58 .git

I'm not getting the error which you mentioned above with my current clone of the repository. But, I'm not sure if it's not coming since I already have the working-repository setup. I'll try to check by doing a fresh clone as well.

Okay, I was getting the similar error like @seanmalloy mentioned above in my local(OSX) when I did a fresh clone of the fork:
-bash>descheduler((HEAD detached at origin/add-verify-scripts))$
 ->  make verify-gen
ls: cannot access '_output/bin/golangci-lint': No such file or directory
./hack/verify-conversions.sh
mktemp: failed to create directory via template ‘./hack/../_tmp/kube-verify.XXXXXX’: No such file or directory
make: *** [Makefile:124: verify-gen] Error 1

And I found the culprit, it's a missing mkdir command before this line. After adding the same, it works fine without any error,
-bash>descheduler((HEAD detached at origin/add-verify-scripts))$
 ->  make verify-gen
ls: cannot access '_output/bin/golangci-lint': No such file or directory
./hack/verify-conversions.sh
Generated conversions verified.

I'll make changes and push the code to working branch then.

Awesome work @pravarag! Could you please squash this down to 1 commit?
/hold
for squash

Awesome work @pravarag! Could you please squash this down to 1 commit?

Squashed the commits with the latest push @damemi 🙂

Updated the review comments.

/lgtm

Thanks @seanmalloy @damemi  🙂


Fix Node.js 12 support

Latest stable release of Node.js 12 includes updated V8 engine which has finally removed some of the deprecated API that we are using. JavaScript ecosystem has relatively low half-life periods for versions so we need to upgrade.

Use v8::Local instead of v8::Handle

v8::Handle has been deprecated in favor of v8::Local since V8 4.5 and has been removed somewhere after V8 7.0. Node.js 12 (current stable) is upgrading V8 engine to 7.4+ which does not support v8::Handle.
Mass-replace v8::Handle → v8::Local to avoid that issue. It's supported even by the ancient Node versions packaged in older Ubuntu releases.

Use NAN helpers

v8::Object::Set() and v8::FunctionTemplate::GetFunction() have had their API changed between V8 versions. Upgrading Node.js to 12 breaks them completely.
Instead of using this API directly we should be using NAN helpers that encapsulate the changes and provide common API for Maybe types.

I have manually verified these changes with the following Node.js versions:

Node.js v4.2.6 — packaged with Ubuntu 16 LTS (Xenial Xerus), checked by CircleCI
Node.js v8.16 — current maintenance LTS version of Node.js, still somewhat supported
Node.js v10.16 — current active LTS version of Node.js, recommended for current users
Node.js v12.6 — latest stable version of Node.js, in development

Currently CircleCI checks only Node.js v4. This PR does not update the build system so you'll have to trust me that it actually builds, for now.

This PR does not update the build system so you'll have to trust me that it actually builds, for now.

but can we add checks for different node versions to CircleCI later? or check them in BuildBot 🤔

@vixentael,

but can we add checks for different node versions to CircleCI later?

Sure thing. I'll add them, just in a separate PR.


DOWN OR BLOCKED at log

Hi.
Build uashield at Azure WM. I am run script and always see this at log "DOWN OR BLOCKED" Is it normal? How i can be sure that all working good?
https://pastebin.com/raw/yuudXEY1

proxy enabled or not?

proxy enabled or not?

Same for me. Proxy is enabled.

proxy enabled or not?

in cmd command yes. Is this tool needed standalone VPN with ru ip?

same problem

I think its because we all share the same proxies.
which makes us easy to block.
thats why I dont enable it.

we have over 10k active machines. we will never have enough proxy to fully handle this. So attack efficiency is lover. Use VPN and disable proxy if you want more efficiency.


Add sample application and README file

Checking the PR branch in forked repo: https://github.com/JatinSanghvi/keda-external-scaler-azure-cosmos-db (including the README file) might provide a better sense of the changes that were made
Checklist

[x] Commits are signed with Developer Certificate of Origin (DCO)

Fixes

Addresses issue https://github.com/kedacore/keda-external-scaler-azure-cosmos-db/issues/2

Addresses issue https://github.com/kedacore/keda-external-scaler-azure-cosmos-db/issues/3

Addresses issue https://github.com/kedacore/keda-external-scaler-azure-cosmos-db/issues/9 (need review to confirm)

Scales targets based on count of Cosmos DB container's partition-ranges with non-zero pending items. Existing implementation scales according to the estimate of items pending to be processed across all partitions, which is incorrect. Cosmos DB architecture limits the degree of parallelism of "processing new changes to the container" to the "number of logical partition ranges" (More information).  The logical partition ranges is dependent on the physical partitions, which is turn depends on combination of container data storage size and container throughput (More information). For small containers with low throughput provisioned, we expect the scale targets to not scale beyond 1.

Fixes generation of metric names. Existing implementation generates the metric name in method GetMetricSpec but uses metric name from request metadata in method GetMetrics. If metric name is not specified in ScaledObject definition, returning metric name in GetMetricSpec won't make the metric-name part of scaler metadata when GetMetrics is called. On other hand, if the metric name is supplied by user in ScaledObject definition, GetMetricSpec would override the name, and KEDA won't be able to get the metrics for the generated metric-name when calling GetMetrics. New implementation provides consistent behavior across methods. All methods will reuse the metric-name if provided by user, else generate one based on other 'required' properties in ScaledObject.

Bases metric name generation based on lease container properties. The current metric-name generation logic will create same metric name if two different lease containers are created to monitor the same container.

Uses standard names as in Cosmos DB documentation for trigger properties. Connection instead ConnectionString (matches with other KEDA scalers), ContainerId instead of CollectionName and ProcessorName instead of LeaseCollectionPrefix.

Other changes

Adds sample producer and consumer projects with steps in README file to test scaling of target application between 0, 1 and 2 replicas.
Refactors code and updates unit tests.

Tagging @pragnagopa, @TsuyoshiUshio, @divyagandhisethi, @ealsur, @lpapudippu

@ahmelsayed, @tomkerkhove - Can you please review the PR?

@ahmelsayed, @tomkerkhove - Gentle reminder for PR review.

I just got back from holiday and will review asap

@pragnagopa @ealsur Please take a final look

Ha, I have missed the last commit @JatinSanghvi.
Would it be possible to change ghcr.io/kedacore/keda-cosmosdb-scaler back to ghcr.io/kedacore/keda-external-scaler-azure-cosmos-db please?
This is important for our scaler approach where people need to understand that it's an external scaler and follow the convention. THank you!

This is important for our scaler approach where people need to understand that it's an external scaler and follow the convention. THank you!

@JatinSanghvi  - please use same name in mcr as well.

@JatinSanghvi  - please use same name in mcr as well.

Sure @pragnagopa, I was going to use the same name to avoid confusion.


Add an endpoint to get statistics of ORT run

Correct me if I'm wrong, but I didn't find an issue for this, although we've had some discussions about the need of the endpoint.
There is a need to get some statistics of an ORT run. The main consumer of this data in the UI would be the run results overview page .../repositories/${repoId}/runs/${ortRunIndex}, where some graphs would be based on the data.
Gathering the statistical data separately from the different (upcoming) endpoints (such as vulnerabilities #794, issues #405, rule violations #795, packages #940 and dependency graphs #944) is possible but would soon become too time-consuming, as they will be list endpoints, listing all items they provide, while for this purpose we would only need total counts, such as

issue counts for each severity
vulnerability counts "
rule violation counts "
number of projects per package manager

The endpoints listed in the issues above are more suitable to the sub-pages, showing the individual results like vulnerabilities and issues.
Details and provided data from this endpoint is subject to discussion.

Are there any plans from your side to address this, @mnonnenmacher?

It's currently not on our agenda.

It's currently not on our agenda.

@lamppu from our side will probably be looking into this soon.

Resolved by https://github.com/eclipse-apoapsis/ort-server/pull/1410 (auto-close again did not work).


Fix CI

lcobucci/jwt 4.2.0 contains BC, for example: https://github.com/lcobucci/jwt/pull/835/files. Now all attempts to generate JWT with symmetric algorithms and small length keys will fail.


links to examples are broken

in the readme,
Example: [/showcaseBadges/your.SCN.ID](/showcaseBadges/your.SCN.ID)

this links to https://github.com/SAP-samples/sap-community-activity-badges/blob/main/showcaseBadges/your.SCN.ID which is a 404
should this link to https://devrel-tools-prod-scn-badges-srv.cfapps.eu10.hana.ondemand.com/showcaseBadges/your.SCN.ID instead?
is devrel-tools-prod-scn-badges-srv.cfapps.eu10.hana.ondemand.com a server that will keep running?

Actually it's a relative URL on purpose. We use the README.md as the source content of the root of the service when running:
https://devrel-tools-prod-scn-badges-srv.cfapps.eu10.hana.ondemand.com/
It's also designed so if people wanted they could fork, adapt, reuse, and then deploy the service on their own account.  The relative URL in the README would then still work.
https://github.com/SAP-samples/sap-community-activity-badges/blob/main/srv/routes/intro.js#L12

Although the activity cards are something we will detail more in about two weeks. They will go along with another surprise element of Devtoberfest that isn't quite ready yet.

Although I get what you are seeing since we reused the same content on the project readme. I will do some cleanup there this afternoon to be a more holistic README for the whole project.  I missed that on my to-do list. But full details on the whole project will have to wait until the above mentioned surprise. ;)

no worries, just stumbled across it 😄

I am curious since you were looking around in the repo; did you already find the @larshp specific Easter Egg?

I didnt read any of the code/files, but now I will look through it :)

found it, yiiiha


need to update read the configuration key in read the docs

https://readthedocs.org/projects/mondo/

Finished in our call today.


Need preload feature like preload_page_view package #5

Where next index is automatically preload. so for this we dont have to wait for for widget to load
https://pub.dev/packages/preload_page_view

Does implementing this feature solve issue with adding animations during switching between pages?
It would be great to have animated lazy load indexed stack to mimic standard navigation temporary flows or smth similar.

hey @moinkhan780 @dmitry-komar, the purpose of a lazy indexed stack is to not load widgets that are yet to be shown, so I think adding a preload feature would defeat the purpose of the widget imo
I'm happy to hear your thoughts and use cases for this feature!

@marcossevilla, in my Math Quiz Module, I've integrated a webview widget for rendering math expressions using KaTeX. The rendering process takes around 1 second or even longer, which can impact the user experience when navigating through questions and options. This is where I see the need to automatically preload the next index to enhance the overall performance and ensure a smoother quiz-taking experience for users.

hi @moinkhan780 sorry I'm replying too late to this, but I didn't implement the stack like this because it defeats the purpose of having a lazy loaded stack; preloading makes the stack render children that are not yet shown, which is exactly the opposite of the widget's utility.

closing this issue but happy to continue the conversation!


Module path should be "github.com/BBVA/raft-badger", not "github.com/bbva/raft-badger"

Background
Module path is inconsistent with go import path.
GO111MODULE=on, as doc said, import "github.com/BBVA/raft-badger", then get this error:
go: finding module for package github.com/BBVA/raft-badger
go: downloading github.com/BBVA/raft-badger v1.0.1
go: found github.com/BBVA/raft-badger in github.com/BBVA/raft-badger v1.0.1
go: test1 imports
        github.com/BBVA/raft-badger<EMAIL_ADDRESS>parsing go.mod:
        module declares its path as: github.com/bbva/raft-badger
                but was required as: github.com/BBVA/raft-badger

Solution
Fix the module path:

Rename the module path to "github.com/BBVA/raft-badger": https://github.com/BBVA/raft-badger/blob/master/go.mod#L1

module github.com/BBVA/raft-badger
go 1.14
require (
	…
)

Change the doc document to use import "github.com/bbva/raft-badger".

@fernandezvara  @aalda  Could you help me review this issue? Thx :p

I just released a new version fixing the module path. Thanks for the warning.

@aalda  Thank you for your contribution!


[genart] add genart strategy

Added genart strategy for GEN.ART DAO. NFT membership ownership + ERC20 balance.

Hi! There is currently someone working on a strategy to do exactly what you trying to do but in a more generic way. The strategy will allow to define 2 sub strategies as parameters, one strategy will be used to define if someone would have permission to vote, and another define the voting power, read more here: https://github.com/snapshot-labs/snapshot-strategies/pull/238#pullrequestreview-835833088

Hey, honestly we'd like to get it merged asap. Also, we might want to add something in the future (e.g. special handling for our gold members).


Can not enter the edit mode

Traceback (most recent call last):
File "C:/Users/wangyan27/Documents/maya/2020/scripts\dwpicker\designer\editarea.py", line 163, in paintEvent
self.paint(painter)
File "C:/Users/wangyan27/Documents/maya/2020/scripts\dwpicker\designer\editarea.py", line 170, in paint
shape.draw(painter)
AttributeError: Shape instance has no attribute 'draw'
when entering the edit mode(Ctrl + E), this error pops out, I'm using Maya 2020. Anyone could help? Thanks a lot.

My bad, savage refactoring.
This is fixed by latest commit

Awsome~!


pipeline: add build and staging deployments

Linked issue
This can be used to test and demo the project for others. Later a docker build script should be added for a one-liner deployment with Now.

:tada: This PR is included in version 0.2.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Merge v4.8.1 into master

Coverage increased (+0.05%) to 67.824% when pulling aaa9340a86e13f80a2087d387116c07a5f4bad76 on toanzian/v4.8.1-to-master into f8e680076d5e817a94377e8b0b58a6ee8ed2b3e3 on master.


Update meter_billing hookspec

Amazon expects datetime object for timestamp and has a dry_run option that can be passed through for testing.

Did you want to include updated unit tests for the modified functionality?

Yep, this is not timeboxed by demo so put it in WIP to make it a "complete". With tests, comments etc.

Creating a new branch off of main to avoid conflicts.


increased button size

increased size of the random quote button
closes #200

I think the button looks perfect, no need for changes.


Fixing query options

This issue is created follow up #8880.
Background
Currently, Pinot SQL OPTION keyword is a REGEX match that is allow ANYWHERE in the SQL string. This causes SQLi issues.
Backward-Compatibility Issue
#8880 proposed an alternative syntax of using OPTION similar to SQL setStatement (see: https://calcite.apache.org/docs/reference.html)
However, this creates a backward-incompatible change towards the Pinot SQL syntax.

it hasn't been design-reviewed by the community;
it changes the syntax to be a "statement" instead of a "clause" (e.g. https://docs.microsoft.com/en-us/sql/t-sql/queries/option-clause-transact-sql?view=sql-server-ver16)

Thus we propose to hold off the syntax change until we come to a consensus on which syntax to use.
SQLi Resolution

#8905 proposed a temporary solution to only allow OPTION keyword at end of SQL string.
also marked the REGEX syntax as deprecated (although we have no easy way to inform end-user in the query result)

Alternative Syntax Change
Note, for the syntax code segment: <, > is used to quote reserved keywords or reserved operators.
As STATEMENT
Pinot query now only allows a single STATEMENT. So there's no difference setting OPTIONS as "clause" associated with the statement, or associates the OPTION with the entire SQL statement list context.
Potentially acceptable syntax listed below:

Standard SET statement

setStatement:
    <SET> identifier <=> literal <;>

identifier:
    simpleIdentifier | <">quoted.complex.identifier<">

literal:
    stringLiteral | integerLiteral | doubleLiteral | booleanLiteral

PRO: standard SET statement is default supported in Calcite, also supported by major SQL engines:

https://dev.mysql.com/doc/refman/8.0/en/set-statement.html,
https://www.postgresql.org/docs/current/sql-set.html
CON: standard SET statement only allows a single key-value, for multiple key-value options, multiple set statement is needed

Special OPTION keyword

optionStatement:
  <OPTION> <(> optionKVPair [<,> optionKVPair ]* <)> <;>

optionKVPair:
  identifier <=> literal

identifier:
    simpleIdentifier | <">quoted.complex.identifier<">

literal:
    stringLiteral | integerLiteral | doubleLiteral | booleanLiteral

PRO: similar usage of current OPTION clause.
CON: I couldn't find a commonly used SQL system that supports this statement type, not easy for users understand the syntax.
AS CLAUSE
We can also extend Calcite's syntax to support OPTION clause in SELEC statement
the syntax will be similar to:
select:
      <SELECT> [ <ALL> | <DISTINCT> ]
          { * | projectItem [<,> projectItem ]* }
      <FROM> tableExpression
      [ <WHERE> booleanExpression ]
      [ <GROUP BY> { groupItem [<,> groupItem ]* } ]
      [ <HAVING> booleanExpression ]
      [ <OPTIONS> <(> optionKVPair [<,> optionKVPair ]* <)> ]
      <;>

optionKVPair:
  identifier <=> literal

identifier:
    simpleIdentifier | <">quoted.complex.identifier<">

literal:
    stringLiteral | integerLiteral | doubleLiteral | booleanLiteral

PRO: This is almost identical to current Pinot OPTION syntax
CON: for each statement we need to extend and add OPTION syntax (e.g. INSERT, CREATE, DELETE, and other DML/DQL we add in the future); also I think requires us to alter Calcite's core syntax parsing extension template.
Closing Thoughts

Please comment/reply on this issue to share your understanding, and if any of the descriptions I posted is incorrect or imprecise.
if there's any alternative syntax not listed above. please kindly share in the comment as well. I will incorporate in the alternative syntax list.

Vote for using SET for the following reasons:

It is standard Calcite syntax, so very easy to support
It is one of the common ways to set the query options
Presto follows this convention to set query options, so this might simplify the connector (cc @xiangfu0)

any additional comments or suggestions: @xiangfu0 @jackjlli @siddharthteotia @yupeng9
I would love to solicit more feedback before choosing the syntax. Or if there's any additional channel to share this for more feedback please do let me know

I'm also incline to SET, as SET is more commonly adopted by other data platforms, which requires less learning curves for users to leverage.
And I think what Jackie mentioned for SET in Presto is that Presto uses SET to set runtime parameters, which is similar to what Postgres does.

looks like <SET> key = value is the way to go. implementing it soon

yes let me update the PR description


xr.testing.assert_equal does not test for dtype

In #4622 @toddrjen points out that xr.testing.assert_equal does not test for the dtype, only for the value. Therefore the following does not raise an error:
import numpy as np
import xarray as xr
import pandas as pd

xr.testing.assert_equal(
    xr.DataArray(np.array(1, dtype=int)), xr.DataArray(np.array(1, dtype=float))
)
xr.testing.assert_equal(
    xr.DataArray(np.array(1, dtype=int)), xr.DataArray(np.array(1, dtype=object))
)
xr.testing.assert_equal(
    xr.DataArray(np.array("a", dtype=str)), xr.DataArray(np.array("a", dtype=object))
)

This comes back to numpy, i.e. the following is True:
np.array(1, dtype=int) == np.array(1, dtype=float)

Depending on the situation one or the other is desirable or not. Thus, I would suggest to add a check_dtype argument to xr.testing.assert_equal and also to DataArray.equals (and Dataset and Variable and identical). I have not seen such an option in numpy, but pandas has it (e.g. pd.testing.assert_series_equal(left, right, check_dtype=True, ...) . I would not change __eq__.

Thoughts?
What should the default be? We could try True first and see how many failures this creates?
What to do with coords and indexes? pd.testing.assert_series_equal has a check_index_type keyword. Probably we need check_coords_type as well? This makes the whole thing much more complicated... Also  #4543

+1 on adding check_dtype. I think it should be default False for assert_equal.
It could be default True for assert_identical but this would be very backward incompatible.

this has come up in https://github.com/pydata/xarray/pull/3706#issuecomment-583259053 before

My concern with assert_identical is the name.  It implies, to me, that there is no difference at all between the two objects.  It was highly unexpected for me that it didn't do that.  I think at the very least it should be clarified in the documentation that it doesn't do that.
If the default for assert_identical isn't change, I wonder whether a new function might be worthwhile.  I am concerned having to append check_dtype=True for every test would hurt test clarity.  And there is also the problem with
Also, just checking dtype won't be sufficient in all cases.  Consider this:
import numpy as np
import xarray as xr

a = xr.DataArray(np.array(1.0, dtype=np.object))
b = xr.DataArray(np.array(1, dtype=np.object))

xr.testing.assert_identical(a, b)

I think for the purpose of testing being able to make sure the result is exactly what you expect is important.

There seems to be a subtle difference between comparing Dataset and DataArray.
DataArray compares coords:
https://github.com/pydata/xarray/blob/03d8d56c9b6d090f0de2475202368b08435eaeb5/xarray/core/dataarray.py#L2792
while Dataset compares _variables:
https://github.com/pydata/xarray/blob/03d8d56c9b6d090f0de2475202368b08435eaeb5/xarray/core/dataset.py#L1457-L1459
Thus once we compare xarray.core.variable.IndexVariable and once we compare xarray.core.variable.IndexVariable and the comparison takes different code paths...
import xarray as xr
air = xr.tutorial.open_dataset("air_temperature")

type(air._variables["lat"])
# xarray.core.variable.IndexVariable

type(air.air.coords["lat"])
# xarray.core.dataarray.DataArray

I agree with @toddrjen that the words of assert_identical strongly suggest that the dtypes would be checked.
I would suggest we either:

Fail on different dtypes in 0.17
If there are lots of breaks / downstream libraries that may break: add a FutureWarning on different dtypes and change in the future

Happy to help on this!


Add default values to Number and Range fields

Hey, I noticed these two fields didn't allow default values to be set.

Thanks Matt 🙌


AttributeError: 'QuartileDiscretizer' object has no attribute 'undiscretize_idxs'

Lime explanation worked perfectly in python environment, also in local tornado api from postman call,
I created docker image, while running docker image I am getting the following error
[E 190716 09:53:36 Tornado6.0:162] Traceback (most recent call last):
      File "C:\app\Tornado6.0.py", line 179, in _blocking_predict
        explanation = self.explanationCall(explainer, X_exp, model.predict_proba)
      File "C:\app\Tornado6.0.py", line 168, in explanationCall
        exp_data = explainer.explain_instance(instance, predict_fn, num_features=6)
      File "C:\Python\lib\site-packages\lime\lime_tabular.py", line 337, in explain_instance
        data, inverse = self.__data_inverse(data_row, num_samples)
      File "C:\Python\lib\site-packages\lime\lime_tabular.py", line 547, in __data_inverse
        inverse[1:] = self.discretizer.undiscretize(inverse[1:])
      File "C:\Python\lib\site-packages\lime\discretize.py", line 158, in undiscretize
        for x in ret[:, feature]])
      File "C:\Python\lib\site-packages\lime\discretize.py", line 158, in <listcomp>
        for x in ret[:, feature]])
      File "C:\Python\lib\site-packages\lime\discretize.py", line 127, in get_undiscretize_value
        if self.undiscretize_idxs[feature][val] == self.precompute_size:
    AttributeError: 'QuartileDiscretizer' object has no attribute 'undiscretize_idxs'

Dont understand why this docker run should throw this error, may i know how to solve this please?

discretize_continuous=False removed the Error in docker run,
discretize_continuous=True, discretizer='quartile' Throwing error in docker run, in local python env running perfect


Fix false positives for backreference and invalid escape in regexp/no-useless-character-class rule

Pull Request Test Coverage Report for Build 629670959

40 of 43 (93.02%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.01%) to 91.431%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

lib/utils/index.ts
38
41
92.68%

Totals

Change from base Build 627635402:
-0.01%

Covered Lines:
1964

Relevant Lines:
2075

💛  - Coveralls


shooter tuning

This pr is solely to find the optimal control loop gains for the shooter that decrease steady state error for reliable shots

better but not perfect


Add reference to privacy whitepaper

https://bugs.chromium.org/p/chromium/issues/detail?id=850164
What's changed, or what was fixed?

adding reference to privacy whitepaper

Target Live Date: 2018-06-24

[ ] This has been reviewed and approved by (NAME)
[ ] I have run gulp test locally and all tests pass.
[ ] I have added the appropriate type-something label.
[X] I've staged the site and manually verified that my content displays correctly.

CC: @petele

:tada: This has been pushed live to https://developers.google.com/web/


fix(clerk-js): Fix the input box-shadow on iOS 16 devices

Description
wip
Checklist

[ ] npm test runs as expected.
[ ] npm run build runs as expected.
[ ] (If applicable) JSDoc comments have been added or updated for any package exports
[ ] (If applicable) Documentation has been updated

Type of change

[ ] 🐛 Bug fix
[ ] 🌟 New feature
[ ] 🔨 Breaking change
[ ] 📖 Refactoring / dependency upgrade / documentation
[ ] other:

!preview

!preview

Hey @anagstef, your preview is currently deploying and should be available shortly.

Status
Preview
Updated (UTC)

:cook: Deploying...
Inspect
Mar 07, 2024 11:35 AM


Slide child mounted before style applied

Hi,
It seems that the  child is mounted before style (width) is applied to the . Im my usecase I need to get the dimensions of the child, but on mount it's the full width of the parent container, which makes me think mounting takes place before the style.
I've looked at your docs, and the iDangerous docs, but couldn't find any event or way to fix my problem. Do you have any idea?

Sorry for the delay on this. Will look into this

Do you by chance have a reproducible example?


[Question] How to adjust font size

I'd like to build my own copy of pdf but am having trouble changing the size of text and code.

https://github.com/OI-wiki/OI-Wiki-export/blob/master/oi-wiki-export/oi-wiki-export.tex
这是导出的文件的模版，可以尝试自定义一下


esp-idf-v4.0 support

Hello!
How I can build and use this library for esp-idf-v4.0 ?
I try to compile it and have some troubles.

Is there any interest here I too am having compile issues with v4.1

ok I got it working.
I moved the library outside the core directory  ( idf.py complains and fails continually on other projects).
I updated the Makefile adding the appropriate relative path.
then simply used make instead of idf.py
so
make menuconfig
make flash monitor
This works on the spiffs example out of the box

After some investigation I found that one difference here is that cmake is now the default

CMake-based build system, which is the default since ESP-IDF V4.0. ESP-IDF also supports a legacy build system based on GNU Make, which was the default before ESP-IDF V4.0.

so my guess is that when the source code sqlite3.c was created you used make and now you need to use cmake

@Yardie-  @Elegma I just got it working on esp-idf 4.1.1 and pushed my changes.
Thank you for the valuable inputs.

Wicked well done.  I will try it soon. I will also try enabling 8_3 filenames and see if I can get it to work on a virtual fat.
On 14 February 2021 5:25:27 pm AWST, "Arun (Arundale Ramanathan)"<EMAIL_ADDRESS>wrote:

@Yardie-  @Elegma I just got it working on esp-idf 4.1.1 and pushed my
changes.
Thank you for the valuable inputs.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/siara-cc/esp32-idf-sqlite3/issues/6#issuecomment-778751324

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Hi Arun
What was your actual build process please.
Ok I'm on 4.2 stable release
There are some differences but I can't see how it changes the build.
I've tried a couple of things and keep failing.
It's so close I can feel it but I am new to this Cmake stuff and I can't nut it out.
attempt 1
follow instructions  down load both git repositories
fails
attempt 2
download and unzip both to ~/esp
cd ~/esp
cp -r ~/esp/esp32-idf-sqlite3-examples-master/spiffs ~/esp
mkdir ~/esp/spiffs/components
cp -r ~/esp/esp32-idf-sqlite3-master ~/esp/spiffs/components
cd ~/esp/spiffs
added to CMakeLists.txt
set(EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components)
Executing "ninja all"...
[0/1] Re-running CMake...
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32
CMake Error at /home/g/esp/esp-idf/tools/cmake/component.cmake:221 (message):
CMake Error at /home/g/esp/spiffs/main/CMakeLists.txt:4
(idf_build_component):
Unknown CMake command "idf_build_component".

Call Stack (most recent call first):
/home/g/esp/esp-idf/tools/cmake/scripts/component_get_requirements.cmake:60 (include)
/home/g/esp/esp-idf/tools/cmake/scripts/component_get_requirements.cmake:76 (__component_get_requirements)

Thanks

@Yardie- I have been following this, assuming SDK is installed at ~/esp/esp-idf:
cd ~/esp/esp-idf/components
git clone https://github.com/siara-cc/esp32-idf-sqlite3
cd ../examples
git clone https://github.com/siara-cc/esp32-idf-sqlite3-examples
cd esp32-idf-sqlite3-examples/spiffs
idf.py build
make flash
make monitor

After this you should be able to see DBs created and records inserted in SPIFFS.

ok so it still needs to be built with make not cmake
idf.py menuconfig fails but make menuconfig works
idf.py flash fails but make flash works
ok that will do for now thanks heaps


chore: bump up Expo version in crna-kitchen-sink example

Issue: N/A
What I did
Update expo requirement from ^24.0.2 to ^30.0.1 for crna-kitchen-sink example. Also remove crna's react-native-scripts to use expo.
Motivation: I tried to use crna-kitchen-sink, but I got stuck in "Starting packager".
PS: whatwg-fetch was added to fix this issue: "Can't Find Variable Self"
How to test
Is this testable with Jest or Chromatic screenshots? NO
Does this need a new example in the kitchen sink apps? NO
Does this need an update to the documentation? NO
If your answer is yes to any of these, please make sure to include it in your PR.
For maintainers only: Please tag your pull request with at least one of the following:
["cleanup", "BREAKING CHANGE", "feature request", "bug", "documentation", "maintenance", "dependencies", "other"]

Thank you for updating the example :)

@elucaswork can you just confirm that the example is working? I mean try to do a fresh install and see that it still works.

@Gongreg I just double checked with a fresh install. It's working for iOS and Android 👍


Bicep or Terraform?
Bicep
Module Classification?
Resource Module
Module Name
avm/res/insights/scheduled-query-rule
Module Details
This module was migrated from CARML by @krbar without an owner.
Note: when someone volunteers to be the owner of this module, please reach out to @krbar to for changing the ownership of the existing GitHub teams as well.

Module proposal: https://github.com/Azure/Azure-Verified-Modules/issues/376
PR: https://github.com/Azure/bicep-registry-modules/pull/712

Do you want to be the new owner of this module?
No
Module Owner's GitHub Username (handle)
No response
(Optional) Secondary Module Owner's GitHub Username (handle)
No response

Hi.  I'd like to take ownership of this module.  Thank you

Hi @mcarter106,
Thanks for requesting/proposing to be an AVM module owner!
We just want to confirm you agree to the below pages that define what module ownership means:

Team Definitions & RACI
Shared Specification (Bicep & Terraform)
Module Support

Any questions or clarifications needed, let us know!
If you agree, please just reply to this issue with the exact sentence below (as this helps with our automation 👍):
"I CONFIRM I WISH TO OWN THIS AVM MODULE AND UNDERSTAND THE REQUIREMENTS AND DEFINITION OF A MODULE OWNER"
Thanks,
The AVM Core Team
#RR

I CONFIRM I WISH TO OWN THIS AVM MODULE AND UNDERSTAND THE REQUIREMENTS AND DEFINITION OF A MODULE OWNER.

Hi @emichellecarter,
Thanks for confirming that you wish to own this AVM module and understand the related requirements and responsibilities!
We just want to ask you to double check a few important things.
Please use the following values explicitly as provided in the module index page:

You must be the owner of the GitHub teams as outlined here.
Please check that your name has been updated in the module index page (this should happen shortly after you confirmed ownership).

You can now take ownership of this module and start improving it as needed! ✅ Happy coding! 🎉
Any further questions or clarifications needed, let us know!
Thanks,
The AVM Core Team

@emichellecarter, since this module has been migrated by @krbar (Kris Baranek), he's created the required GH teams. Please work with him to take ownership over these teams and let me know in a comment here when it happened.
Please also remove the ORPHANED.md file from the module's folder and its content from the README.md, by following the steps described in point 5 here and let me know in a comment when this is done. Thanks!

Hello, I would like to take an ownership of this module.

@Brunoga-MS, this module is already being handed over to @emichellecarter. If you're interested in taking ownership over another module, I suggest checking this list for modules that need an owner: https://github.com/Azure/Azure-Verified-Modules/issues?q=is%3Aopen+is%3Aissue+label%3A"Needs%3A+Module+Owner+%3Amega%3A"+label%3A"Language%3A+Bicep+%3Amuscle%3A"

@emichellecarter, since this module has been migrated by @krbar (Kris Baranek), he's created the required GH teams. Please work with him to take ownership over these teams and let me know in a comment here when it happened.
Please also remove the ORPHANED.md file from the module's folder and its content from the README.md, by following the steps described in point 5 here and let me know in a comment when this is done. Thanks!

@matebarabas  FYI: the teams are updated, @emichellecarter is now the maintainer of them

Thanks @krbar for closing the loop and handing over the teams (realted issue). The orphaned.md file has also been removed and the readme has been regenerated (PR). I'm closing this issue now.


Update testcontainers-scala-dynalite, ... to 0.40.7

Updates

com.dimafeng:testcontainers-scala-dynalite
com.dimafeng:testcontainers-scala-mockserver
com.dimafeng:testcontainers-scala-scalatest

from 0.39.12 to 0.40.7.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.dimafeng" } ]

labels: test-library-update, early-semver-major, semver-spec-minor

old PR


Finish the new converter of LLVM IR into BIR

Currently, there are two converters of LLVM IR into BIR in llvmir2hll (see source code in src/llvmir2hll/llvm/llvmir2bir_converters):

orig Original converter, which has been in the decompiler since its beginning.
new A complete rewrite of the converter.

The used converter can be set via --backend-llvmir2bir-converter when running retdec-decompiler.sh. Currently, the default one is orig, because the new converter is not finished.
We should finish the converter. It has the following two main issues:

It generates goto targets without labels (e.g. goto xyz; is generated, but there is no xyz label in the code).
It generates more goto statements than the original converter.

A word of caution: Several people have already tried to finish the new converter, but none of them succeeded. It is not an easy task.

The default LLVM IR to BIR converter was changed in e6f47cf6ae6bd3d79793b61364831fca74739bbf. This commit is already in master.


Implement yarn workspaces for local development

probably not needed, the workflow I mention in https://github.com/react-navigation/react-navigation-core/issues/23 is good enough imo


can I use another key for the data list？

can I use another key for the data list? I tried to change the "i" to the "key", but it didn't respond every time.

This just happened to me as well. We wanted to use more verbose property names like width, height, xposition etc. How come the bound value-containing variables need to have the same name as the variables you are setting the values into?

I just want to see if I can use other keys, such as ID, key and so on

No, you have to use those keys


Codeunit 99000854 OnBeforeForecastConsumption

Can we add new event in Codeunit 99000854.
local procedure ForecastConsumption(var DemandInvtProfile: Record "Inventory Profile"; var Item: Record Item; OrderDate: Date; ToDate: Date) UpdatedOrderDate: Date
var
    ForecastEntry: Record "Production Forecast Entry";
    ForecastEntry2: Record "Production Forecast Entry";
    NextForecast: Record "Production Forecast Entry";
    CustomCalendarChange: Array[2] of Record "Customized Calendar Change";
    TotalForecastQty: Decimal;
    ReplenishmentLocation: Code[10];
    ForecastExist: Boolean;
    NextForecastExist: Boolean;
    ReplenishmentLocationFound: Boolean;
    ComponentForecast: Boolean;
    ComponentForecastFrom: Boolean;
    **IsHandled: Boolean;     // New**
begin
    **OnBeforeForecastConsumption(DemandInvtProfile,Item,OrderDate,ToDate,UpdatedOrderDate,IsHandled);   // New**

// NEW +
**[IntegrationEvent(false, false)]
local procedure OnBeforeForecastConsumption(var DemandInvtProfile: Record "Inventory Profile"; var Item: Record Item; OrderDate: Date; ToDate: Date; var UpdatedOrderDate: Date; var IsHandled: Boolean);
begin
end;**
// NEW -

Sorry forgot to add the Is Handled piece.
local procedure ForecastConsumption(var DemandInvtProfile: Record "Inventory Profile"; var Item: Record Item; OrderDate: Date; ToDate: Date) UpdatedOrderDate: Date
var
    ForecastEntry: Record "Production Forecast Entry";
    ForecastEntry2: Record "Production Forecast Entry";
    NextForecast: Record "Production Forecast Entry";
    CustomCalendarChange: Array[2] of Record "Customized Calendar Change";
    TotalForecastQty: Decimal;
    ReplenishmentLocation: Code[10];
    ForecastExist: Boolean;
    NextForecastExist: Boolean;
    ReplenishmentLocationFound: Boolean;
    ComponentForecast: Boolean;
    ComponentForecastFrom: Boolean;
    IsHandled: Boolean;             // NEW
begin
    // NEW+
    OnBeforeForecastConsumption(DemandInvtProfile,Item,OrderDate,ToDate,UpdatedOrderDate,IsHandled);
    if IsHandled then
        exit;
    // NEW-
    UpdatedOrderDate := OrderDate;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


chore: Code cleaneup

requires pr#33
Removed dead code and unused import, primed library for tech debt sweeping. Removed box from perk functions to increase speed and decrease memory.

LGTM!!!!

I was cooking too hard this has been moved to #104


Combined parfor chunking and caching PRs.

This replaces #6025 and #7522.  There was overlap between these two PRs around using the dynamic thread count so rather than delaying the merge I went ahead and combined them.
This combined PR provides an API for selecting a parfor chunk size to deal with load balancing issues and it eliminates all use of static thread counts in generated parfor code.  Thus, parfor code (even with reductions) is now cacheable and if you change the chunksize or thread count after reloading from cache then you will use the new values as they are applied correctly in the code now.
Closes #2556
Closes #3144

NOTE: I think this might also fix this: https://github.com/numba/numba/issues/5890#issuecomment-661848040

@DrTodd13, patch to make the reduction use runtime threadcount is at https://github.com/IntelLabs/numba/pull/70

Can you make a suggestion here? I am out of office attending to my father who is very ill. Is there no way to test in get_thread_id if we are not in a parallel region?

@DrTodd13 I've a couple of ideas about how to do this and will attempt those, I can take care of this and report back here when a solution is found. Will catch up OOB.

@DrTodd13 I've added a few patches in https://github.com/stuartarchibald/numba/commits/pr_7625_cont_1 on top of this PR, the commit messages should hopefully explain what they do.

https://github.com/stuartarchibald/numba/commit/5950626c898224baa22390432e100f823fb7067f
https://github.com/stuartarchibald/numba/commit/166593a9dedf80d49276a43df2e1309b2fab8125
https://github.com/stuartarchibald/numba/commit/b27ffdb1e0c171833ae2b8e7d1ca964fb7ab35d8
https://github.com/stuartarchibald/numba/commit/f6e91835e6d12d8aac173be41de854da70cfd493

They essentially fix a few bugs I found and do a small amount of refactoring. I think the first two are necessary for correctness. If you are happy with them I can make a PR against this branch for you to look at? Thanks!

@stuartarchibald Looks good to me so please create the PR. Thanks!

Thanks for checking @DrTodd13, PR is https://github.com/IntelLabs/numba/pull/71

@DrTodd13 As discussed, this is a test which is producing behaviour that I'm not sure is expected:
from numba import njit, parallel_chunksize, get_parallel_chunksize, prange
import numpy as np

@njit(parallel=True)
def bar():
    acc = 0
    print("Bar entry:", get_parallel_chunksize())
    with parallel_chunksize(30):
        print("Bar ctx entry:", get_parallel_chunksize())
        for i in prange(1):
            acc += i
        print("Bar ctx exit:", get_parallel_chunksize())
    print("Bar exit:", get_parallel_chunksize())
    return acc

@njit(parallel=True)
def foo():
    acc = 0
    print("Foo entry:", get_parallel_chunksize())
    with parallel_chunksize(10):
        print("Foo ctx entry:", get_parallel_chunksize())
        for i in prange(1):
            acc += bar()
        print("Foo ctx exit:", get_parallel_chunksize())
    print("Foo exit:", get_parallel_chunksize())

foo()

which for me gives:
Foo entry: 0
Foo ctx entry: 10
Bar entry: 0
Bar ctx entry: 30
Bar ctx exit: 30
Bar exit: 0
Foo ctx exit: 10
Foo exit: 0

I'm not sure why Bar entry/exit has a value of 0. For comparison, this is what happens under NUMBA_DISABLE_JIT=1:
$ NUMBA_DISABLE_JIT=1 python test_7625_nested_ctx.py
Foo entry: 0
Foo ctx entry: 10
Bar entry: 10
Bar ctx entry: 30
Bar ctx exit: 30
Bar exit: 10
Foo ctx exit: 10
Foo exit: 10

@stuartarchibald
Now that I see your example, this is the expected behavior.  Well, at least it is what I intended so the question remains is will you agree that this should be the expected behavior.  What is going on is that when the parallel executor starts, it resets the chunksize to 0 (do the default simple equal division of work to threads) for each worker.  The reason that I did this is that I was thinking that a custom chunksize for one parallel region is unlikely to be right for a nested parallel region so best to reset to the default behavior.  So, if you don't enter any parallel regions then it makes sense that the chunksizes are maintained in the non-jit (or non-parallel) mode.  Obviously, this goes against your expectations and if I hadn't been the one to implement it then that might have been my expectation as well.  I can easily make it do what you expect but my only reluctance for me to do that is that I think that being the default is unlikely to be best performance wise and it will be easy for programmers to forget they are coming from a chunksize context that may have an effect they don't want.  So, if they give it no thought then my approach may be better.  If they give it a modicum of thought then they might expect the chunksize to carry over into the parallel workers, but the question is if they give it even more thought whether they will want it to carry over by default or not.  Curious as to your opinion.

Capturing the failing test from CI. OSX python 3.9, NumPy 1.21 failed with:

======================================================================
FAIL: test_TestParforNumPy (numba.tests.test_parfors.TestParforsRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/1/s/numba/tests/test_parfors.py", line 91, in test_TestParforNumPy
    self.runner()
  File "/Users/runner/work/1/s/numba/tests/test_parfors.py", line 80, in runner
    self.subprocess_test_runner(test_module=themod,
  File "/Users/runner/work/1/s/numba/tests/support.py", line 581, in subprocess_test_runner
    self.assertEqual(status.returncode, 0, streams)
AssertionError: 1 != 0 :
captured stdout:
captured stderr: /Users/runner/work/1/s/numba/core/typed_passes.py:329: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.

File "numba/tests/test_parfors.py", line 1247:
    def test_0d_array(self):
        def test_impl(n):
        ^

  warnings.warn(errors.NumbaPerformanceWarning(msg,
..F./Users/runner/work/1/s/numba/core/typed_passes.py:329: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.

File "numba/tests/test_parfors.py", line 1014:
    def test_dead_randoms(self):
        def test_impl(n):
        ^

  warnings.warn(errors.NumbaPerformanceWarning(msg,
..................
======================================================================
FAIL: test_argmax (numba.tests.test_parfors.TestParforNumPy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/1/s/numba/tests/test_parfors.py", line 1115, in test_argmax
    self.check(test_impl1, A)
  File "/Users/runner/work/1/s/numba/tests/test_parfors.py", line 282, in check
    self.check_parfors_vs_others(pyfunc, cfunc, cpfunc, *args, **kwargs)
  File "/Users/runner/work/1/s/numba/tests/test_parfors.py", line 260, in check_parfors_vs_others
    np.testing.assert_almost_equal(parfor_output, py_expected, **kwargs)
  File "/Users/runner/miniconda3/envs/azure_ci/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 599, in assert_almost_equal
    raise AssertionError(_build_err_msg())
AssertionError:
Arrays are not almost equal to 7 decimals
 ACTUAL: 4
 DESIRED: 2

----------------------------------------------------------------------
Ran 22 tests in 235.433s

FAILED (failures=1)

i.e. incorrect result from numba.tests.test_parfors.TestParforNumPy.test_argmax
I cannot reproduce this locally.
Looking at the test implementation:
https://github.com/numba/numba/blob/b1be2f12c83c01f57fe34fab9a9d77334f9baa1d/numba/tests/test_parfors.py#L1102-L1133
seems like there's use of np.random.randint and np.random.ranf without a np.seed being set, which means this test is not testing with the same values across runs. I'm not convinced that this test failure is a result of this PR vs. some combination of values from the RNG that trigger an already present bug. Will rerun the test job and see if it passes.

@DrTodd13 As discussed, this is a test which is producing behaviour that I'm not sure is expected:
from numba import njit, parallel_chunksize, get_parallel_chunksize, prange
import numpy as np

@njit(parallel=True)
def bar():
    acc = 0
    print("Bar entry:", get_parallel_chunksize())
    with parallel_chunksize(30):
        print("Bar ctx entry:", get_parallel_chunksize())
        for i in prange(1):
            acc += i
        print("Bar ctx exit:", get_parallel_chunksize())
    print("Bar exit:", get_parallel_chunksize())
    return acc

@njit(parallel=True)
def foo():
    acc = 0
    print("Foo entry:", get_parallel_chunksize())
    with parallel_chunksize(10):
        print("Foo ctx entry:", get_parallel_chunksize())
        for i in prange(1):
            acc += bar()
        print("Foo ctx exit:", get_parallel_chunksize())
    print("Foo exit:", get_parallel_chunksize())

foo()

which for me gives:
Foo entry: 0
Foo ctx entry: 10
Bar entry: 0
Bar ctx entry: 30
Bar ctx exit: 30
Bar exit: 0
Foo ctx exit: 10
Foo exit: 0

I'm not sure why Bar entry/exit has a value of 0. For comparison, this is what happens under NUMBA_DISABLE_JIT=1:
$ NUMBA_DISABLE_JIT=1 python test_7625_nested_ctx.py
Foo entry: 0
Foo ctx entry: 10
Bar entry: 10
Bar ctx entry: 30
Bar ctx exit: 30
Bar exit: 10
Foo ctx exit: 10
Foo exit: 10

@stuartarchibald Now that I see your example, this is the expected behavior. Well, at least it is what I intended so the question remains is will you agree that this should be the expected behavior. What is going on is that when the parallel executor starts, it resets the chunksize to 0 (do the default simple equal division of work to threads) for each worker. The reason that I did this is that I was thinking that a custom chunksize for one parallel region is unlikely to be right for a nested parallel region so best to reset to the default behavior. So, if you don't enter any parallel regions then it makes sense that the chunksizes are maintained in the non-jit (or non-parallel) mode. Obviously, this goes against your expectations and if I hadn't been the one to implement it then that might have been my expectation as well. I can easily make it do what you expect but my only reluctance for me to do that is that I think that being the default is unlikely to be best performance wise and it will be easy for programmers to forget they are coming from a chunksize context that may have an effect they don't want. So, if they give it no thought then my approach may be better. If they give it a modicum of thought then they might expect the chunksize to carry over into the parallel workers, but the question is if they give it even more thought whether they will want it to carry over by default or not. Curious as to your opinion.

@DrTodd13 I think what you describe makes sense in terms of picking defaults that are likely to give reasonable performance in these situations and am happy to leave the implementation behaving in this way. If nested regions would benefit from some specific chunksize then it's relatively easy to just add this in to user code. My only suggestion would be to ensure that this is all explicitly documented (as it does differ to all the other TLS properties, which are transferred into nested regions) and that the deviation between the context manager behaviour when called from Python and compiled code is noted. Thanks.

NOTE: Tested the nesting behaviour and get_thread_id with:
from numba import njit, prange, get_thread_id
import numpy as np

@njit(parallel=True)
def inner(n):
    acc = 0
    for i in prange(n):
        print("inner ", get_thread_id())
        acc += 1
    print("nested serial", get_thread_id())
    for i in prange(n):
        print("inner ", get_thread_id())
        acc += 1

@njit(parallel=True)
def outer(n):
    acc = 0
    for i in prange(n):
        print("outer ", get_thread_id())
        acc += 1
        inner(3)
    print("outer serial", get_thread_id())
    for i in prange(n):
        print("outer ", get_thread_id())
        acc += 1
        inner(3)

outer(3)

and the C code:
#include <omp.h>
#include <stdlib.h>
#include <stdio.h>

double inner(size_t n)
{
    double acc = 0.;

#pragma omp parallel for
    for (size_t i = 0; i < n; i++)
    {
        printf("inner %d\n", omp_get_thread_num());
        acc += 1.;
    }

    printf("nested serial %d\n", omp_get_thread_num());

#pragma omp parallel
    for (size_t i = 0; i < n; i++)
    {
        printf("inner %d\n", omp_get_thread_num());
        acc += 1.;
    }
    return acc;
}

double outer(size_t n)
{
    double acc = 0.;
#pragma omp parallel for
    for (size_t i = 0; i < n; i++)
    {
        printf("outer %d\n", omp_get_thread_num());
        acc += 1.;
        acc += inner(2);
    }

    printf("outer serial %d\n", omp_get_thread_num());

#pragma omp parallel for
    for (size_t i = 0; i < n; i++)
    {
        printf("outer %d\n", omp_get_thread_num());
        acc += 1.;
        acc += inner(2);
    }

    return acc;
}

int main(void){
    omp_set_nested(0x1);
    outer(3);
}

In Python, the reported thread ID when using either the OpenMP or TBB backed does approximately the same as the C code. i.e. the call in the "nested serial" region returns the ID of the thread that is the "main" thread in the nested region.

I added a big block of descriptive text in the documentation about this resetting of chunksize inside parallel regions.

Thanks for the update on the docs @DrTodd13, I'll take a look at it shortly. Also, public CI is failing, looks like it's just some flake8 problems:
numba/tests/doc_examples/test_parallel_chunksize.py:23:17: F841 local variable 'pre_prange_chunksize' is assigned to but never used
numba/tests/doc_examples/test_parallel_chunksize.py:25:21: F841 local variable 'in_prange_chunksize' is assigned to but never used
numba/tests/doc_examples/test_parallel_chunksize.py:25:81: E501 line too long (83 > 80 characters)
numba/tests/doc_examples/test_parallel_chunksize.py:27:17: F841 local variable 'post_prange_chunksize' is assigned to but never used
numba/tests/doc_examples/test_parallel_chunksize.py:27:81: E501 line too long (81 > 80 characters)
numba/np/ufunc/parallel.py:680:81: E501 line too long (108 > 80 characters)

I found an compilation error with a common pattern to use get_thread_id():
import numpy as np
from numba import njit, prange, set_num_threads, get_num_threads
from numba import get_thread_id

@njit(parallel=True)
def foo(arr, data):
    # In this version,
    set_num_threads(data.size)
    for i in prange(arr.size):
        tid = get_thread_id()
        ## Uncomment the print to stop `tid` from apparently
        ## optimized to 0 for all threads
        # print(tid)
        data[tid] += 1
        arr[i] = tid

arr = np.arange(32)
data = np.zeros(4)
foo(arr, data)

print(arr)
print(data)

Without the print(tid), the output is:
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[32.  0.  0.  0.]

as if tid is assumed to be 0 all the time.
With the print(tid), the output is:
[0 0 0 0 0 0 0 0 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0]
[16.  0.  8.  8.]

The previous error occurs on main as well

I found an compilation error with a common pattern using get_thread_id():
import numpy as np
from numba import njit, prange, set_num_threads, get_num_threads
from numba import get_thread_id

@njit(parallel=True)
def foo(arr, data):
    # In this version,
    set_num_threads(data.size)
    for i in prange(arr.size):
        tid = get_thread_id()
        ## Uncomment the print to stop `tid` from apparently
        ## optimized to 0 for all threads
        # print(tid)
        data[tid] += 1
        arr[i] = tid

arr = np.arange(32)
data = np.zeros(4)
foo(arr, data)

print(arr)
print(data)

Without the print(tid), the output is:
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[32.  0.  0.  0.]

as if tid is assumed to be 0 all the time.
With the print(tid), the output is:
[0 0 0 0 0 0 0 0 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0]
[16.  0.  8.  8.]

I'm not sure yet that this is a bug. Using the TBB threading layer I can reproduce the above result, but I cannot reproduce with OpenMP or workqueue. Assuming the TBB threading layer is being used, there's no guarantee that threads will spin up and/or join the task arena for a given parallel region unless there's sufficient work, without the print the amount of work is tiny so it probably "decides" it's not worth adding more threads, but with the print it might just go over some threshold for enough work to add more threads.
The "adding some more threads" can be observed in the output:

[0 0 0 0 0 0 0 0 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0]

There's no thread ID of 1 (assuming 4 cores), it's decided to use thread 0 (main thread) and then add in a couple of others. I also note that making the array arr progressively larger and running without the print leads to more and more threads joining to do work. Does this make sense or are you thinking of something else?

Ok, i agree. Both omp and workqueue backends are doing the right thing. So it is probably just TBB scheduling at work in my original report.

@sklam I fixed some problems with your suggestions in the C file relating to printf format specifiers but now 5 of the tests are failing "before install."  I haven't seen this before so don't know what to suggest but perhaps you know.  I don't think the problems are related to your suggestions or my fixes.

@sklam I fixed some problems with your suggestions in the C file relating to printf format specifiers but now 5 of the tests are failing "before install." I haven't seen this before so don't know what to suggest but perhaps you know. I don't think the problems are related to your suggestions or my fixes.

@DrTodd13 don't think it's this patch, think it's #8152 (fixed by #8151).

/AzurePipelines run

On a further look, whilst this test has some deterministic issues (the RNG seed isn't set) I think this failure on the Azure OSX build is genuine and I can reproduce it locally using this branch and the TBB backend (but not on mainline, or with the OpenMP backend). The error is:
2022-06-14T14:03:06.8684310Z ======================================================================
2022-06-14T14:03:06.8685090Z FAIL: test_argmax (numba.tests.test_parfors.TestParforNumPy)
2022-06-14T14:03:06.8686580Z ----------------------------------------------------------------------
2022-06-14T14:03:06.8687270Z Traceback (most recent call last):
2022-06-14T14:03:06.8687900Z   File "/Users/runner/work/1/s/numba/tests/test_parfors.py", line 1118, in test_argmax
2022-06-14T14:03:06.8688540Z     self.check(test_impl2, A)
2022-06-14T14:03:06.8689140Z   File "/Users/runner/work/1/s/numba/tests/test_parfors.py", line 282, in check
2022-06-14T14:03:06.8689850Z     self.check_parfors_vs_others(pyfunc, cfunc, cpfunc, *args, **kwargs)
2022-06-14T14:03:06.8690580Z   File "/Users/runner/work/1/s/numba/tests/test_parfors.py", line 260, in check_parfors_vs_others
2022-06-14T14:03:06.8691370Z     np.testing.assert_almost_equal(parfor_output, py_expected, **kwargs)
2022-06-14T14:03:06.8692790Z   File "/Users/runner/miniconda3/envs/azure_ci/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 599, in assert_almost_equal
2022-06-14T14:03:06.8693900Z     raise AssertionError(_build_err_msg())
2022-06-14T14:03:06.8694440Z AssertionError:
2022-06-14T14:03:06.8694980Z Arrays are not almost equal to 7 decimals
2022-06-14T14:03:06.8695620Z  ACTUAL: 4
2022-06-14T14:03:06.8696090Z  DESIRED: 2

I used this locally to reproduce (it's a trimmed down version of the failing test in the test suite), it usually hits the breakpoint within 50 iterations:
from numba import njit
import numpy as np

def check(impl, arg):
    njitted = njit(impl)
    pjitted = njit(parallel=True)(impl)
    python = impl

    expected = python(arg.copy())
    got_njitted = njitted(arg.copy())
    got_pjitted = pjitted(arg.copy())

    try:
        np.testing.assert_allclose(expected, got_njitted)
        np.testing.assert_allclose(expected, got_pjitted)
    except Exception as e:
        import pdb; pdb.set_trace()
        pass
    else:
        print("ok")

def test_argmax():
    def test_impl1(A):
        return A.argmax()

    def test_impl2(A):
        return np.argmax(A)

    n = 211
    A = np.array([1., 0., 3., 2., 3.])
    B = np.random.randint(10, size=n).astype(np.int32)
    C = np.random.ranf((n, n))  # test multi-dimensional array
    check(test_impl1, A)
    check(test_impl1, B)
    check(test_impl1, C)
    check(test_impl2, A)
    check(test_impl2, B)
    check(test_impl2, C)

for i in range(1000):
    test_argmax()

output:
(Pdb) l
 13         try:
 14             np.testing.assert_allclose(expected, got_njitted)
 15             np.testing.assert_allclose(expected, got_pjitted)
 16         except Exception as e:
 17             import pdb; pdb.set_trace()
 18  ->         pass
 19         else:
 20             print("ok")
 21
 22     def test_argmax():
 23         def test_impl1(A):
(Pdb) p e
AssertionError('\nNot equal to tolerance rtol=1e-07, atol=0\n\nMismatched elements: 1 / 1 (100%)\nMax absolute difference: 2\nMax relative difference: 0.5\n x: array(2)\n y: array(4)')
(Pdb) p arg
array([1., 0., 3., 2., 3.])

and it looks like it's failing using the inlined array input array([1., 0., 3., 2., 3.]), where indexes 2 and 4 both hold the same value.
I think this will need resolving as either "existing bug" or "bug in this patch" (and fixed) prior to merge.

@DrTodd13 as discussed OOB, the few outstanding items on this PR are:

Set the OpenMP backend parallel for-loop to use dynamic scheduling (else the chunksize won't have as much impact?)
Add a motivating example to the user docs with a really biased work load, e.g. work load increasing with respect to loop index, and offer some basic examples of the use of setting the chunk size and the context manager.
Move the existing detailed documentation on the implementation into the developer docs (suggest into here: https://github.com/numba/numba/blob/158b117234db18c9bf415a4624340c04a375fdf4/docs/source/developer/threading_implementation.rst?plain=1#L1-L3)
Add a test for the issue reported in #7518

Many thanks!

Set the OpenMP backend parallel for-loop to use dynamic scheduling (else the chunksize won't have as much impact?

The default chunksize of 1 with dynamic would work well for parfor use cases but is there going to be too much overhead for other uses of the vectorize/guvectorize path?

Move the existing detailed documentation on the implementation into the developer docs (suggest into here: https://github.com/numba/numba/blob/158b117234db18c9bf415a4624340c04a375fdf4/docs/source/developer/threading_implementation.rst?plain=1#L1-L3
)

Which existing detailed documentation?

Set the OpenMP backend parallel for-loop to use dynamic scheduling (else the chunksize won't have as much impact?

The default chunksize of 1 with dynamic would work well for parfor use cases but is there going to be too much overhead for other uses of the vectorize/guvectorize path?

This is a good point, I think for low intensity kernels the dynamic scheduler would end up causing overhead. What do you think about doing this: the parallel_for function API could have an additional argument int dynamic_scheduling which if set can be used to inform the scheduling at runtime. I think if this were implemented the omp parallel for could have schedule(runtime) with it configured at runtime via omp_set_schedule(dynamic/static) based on the dynamic_scheduling argument? Parfors can set this argument as part of its code generation and the standard guvectorize/vectorize can elect to not set it. I think this will "fix" the issue with OpenMP, and also let the TBB backend specify a more specific partitioning.

Move the existing detailed documentation on the implementation into the developer docs (suggest into here: https://github.com/numba/numba/blob/158b117234db18c9bf415a4624340c04a375fdf4/docs/source/developer/threading_implementation.rst?plain=1#L1-L3
)

Which existing detailed documentation?

The docs here: https://github.com/numba/numba/pull/7625/files#diff-03a635d2d8fbe72934bd351660296e5bb40b60490860eb63b6490db8fbc8c02bR614, these are very detailed and great for those wanting to know about all about the implementation details. For the first item in the user facing docs, was thinking it might be good to provide a more simple introductory overview with a motivating example of a poorly balanced load, and then to reference out to the more detailed documentation, what do you think?

@stuartarchibald Per our discussion, we're holding off on all the changes for dynamic scheduling across all backends.  I moved some of the documentation details about chunk sizing to the developer files.  I added a test of caching of a parfor function with a reduction with different thread numbers but that really needs a good review as I'm not very confident I used the infrastructure in test_caching.py correctly.  I also added a np.nan test in argmin and argmax.

@stuartarchibald Per our discussion, we're holding off on all the changes for dynamic scheduling across all backends. I moved some of the documentation details about chunk sizing to the developer files. I added a test of caching of a parfor function with a reduction with different thread numbers but that really needs a good review as I'm not very confident I used the infrastructure in test_caching.py correctly. I also added a np.nan test in argmin and argmax.

@DrTodd13 Thanks for adding these updates. I'll make sure that I take a good look at the caching tests. It turns out that the public CI build is currently failing, seems like the test:
numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest.test_unbalanced_example

is producing:
numba.core.errors.UnsupportedRewriteError: Failed in nopython mode pipeline (step: convert to parfors)
Overwrite of parallel loop index

File "numba/tests/doc_examples/test_parallel_chunksize.py", line 29:
            def func1():
                <source elided>
                        if cur % 2 == 0:
                            cur //= 2
                            ^

which doesn't look quite right given cur is not the loop index var, but it is an alias of it?

@DrTodd13 thanks for the fixes, I've reviewed the code up to f70175b (changes look fine, thanks), and have responded to your questions in line.

The latest changes looks good to me

@DrTodd13 I've opened https://github.com/IntelLabs/numba/pull/72 to address the duplication of test files, please could you take a look and if you approve merge in, many thanks.

RE the outstanding comment from: https://github.com/numba/numba/pull/7625#discussion_r836577063, PR https://github.com/numba/numba/pull/8186 has 991a965 which removes the proposed unification method on the ExternalFunctionPointer type and the tests at least pass in public CI. Is this sufficient evidence to suggest it is not needed?

gpuci run tests (just running this as there are some changes to numba.core - I'm pretty sure it won't affect anything negatively in CUDA, but just double-checking here)

one unresolved comment: https://github.com/numba/numba/pull/7625/files#r904748398

A windows test failed:

A windows test failed:

======================================================================
FAIL: test_caller (numba.tests.test_parfors_caching.TestParforsCache)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\s\numba\tests\test_parfors_caching.py", line 45, in test_caller
    self.run_test(f, num_funcs=3)
  File "D:\a\1\s\numba\tests\test_parfors_caching.py", line 22, in run_test
    self.assertPreciseEqual(f(ary), f.py_func(ary))
  File "D:\a\1\s\numba\tests\support.py", line 390, in assertPreciseEqual
    self.fail("when comparing %s and %s: %s" % (first, second, failure_msg))
AssertionError: when comparing [0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1] and [0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]: 0.1 != 0.10000000000000003

----------------------------------------------------------------------

It's strange that this should suddenly start failing. It looks like a minor numerical error, probably just from using reductions/accumulation in here:
https://github.com/numba/numba/blob/33e94b0773b88b92fa8f59abfe51c246be2d472a/numba/tests/parfors_cache_usecases.py#L9-L25
I think the "fix" is to use np.testing.assert_allclose in the check_module and run_module methods in https://github.com/numba/numba/pull/7625/files#diff-89333c093ac43075778fe1a5bdd16ed10fd6380e4a76dc5e9e8fdb148596f679

smoketesting at BFID numba_smoketest_cpu_yaml_110

:tada:


Alpha Blending / Transparency

As of writing, the vertex pulling renderer does not support alpha blending for several reasons:

vertex pulling optimizations (3-face index buffer) preclude backface culling
vertex pulling selection of visible faces may be "wrong" when the chosen face is a "back face"

the face will be alpha blended, but it should be culled entirely

For these reasons, alpha blending is currently disabled in the vertex pulling pipeline (using BlendState::REPLACE).
If we are going to support transparency, I think it should happen in its own pipeline, or at least a specialized pipeline. There will need to be some big changes to the index buffering as well as the shader code.

RE #2 , I will still reserve a byte for alpha in any color mode, but it may go unused for a while.


Adds support for buildNeedsWorkspace option in Build Flow Plugin

Honors existing Jenkins behavior of not adding this option, thereby keeping it false, by default.
use buildNeedsWorkspace() to set to true for a build flow job

@daspilker good catch. Comments removed and commits squashed.

Thanks!


watershed only takes shapefile as pour point input

Which seems a bit inconsistent when necessary pre processing tools like snap pour points don't output shapefiles and raster to vector points is not yet ported.

@lucfr You're right that this is inconsistent. Early on, I had made the input of the Watershed tool take either a raster or a vector input. This either/or proved challenging to support, specifically, for the QGIS plugin for WhiteboxTools. Between supporting vector or raster pour point inputs, I think that vector inputs are probably more widely used and so I decided. The thinking for the snap pour points tools was always that I would update them to input/output vector as soon as I add support for writing vector files. In fact, I have just completed the code for writing shapefile geometries (attribute data are still a work in progress). I need to test this functionality by using it in a tool. The SnapPourPoint and JensenSnapPourPoint tools are the perfect candidates. As such, hopefully this inconsistency will be rectified very soon.

Exciting news! Looking forward to the upcoming changes and additions! :)

I've just uploaded the new versions of the SnapPourPoints and JensenSnapPourPoints tools that take vector points as inputs and outputs. So I'm marking this as closed. Please let me know if there is any further issue.


Oxide console commands do not work

The Oxide console commands only work in the in-game chat, not the server console. They do not show with tab-completion either. This may have to do with being allowed to or not, but the server console should be able to regardless.

This seems to have been fixed.


Add rotriever.toml

Coverage remained the same at 99.351% when pulling 2f742a1b2c9288ac7c8675ad6620747904bf0339 on AmaranthineCodices:add-toml into 5500de5c663188309fa117e20b25dfef76c979bd on Roblox:master.


MP Config Map injection behavior restored to original

Resolves #4414
Shoudl behave similar to original before #1721 was applied
Injecting without explicit key will inject all properties, injecting with a key will inject properties where keys have the prefix removed (and only containing keys that are prefixed with the key).
Signed-off-by: Tomas Langer<EMAIL_ADDRESS>

/trigger

Closing this PR, it is broken. Will reopen (cannot build using our pipelines)


Path to static assets are broken

When transpiling, library vendor/foo/bar will be now accessible under vendor/phabel/transpiler71/foo/bar. This is a problem when this library has static assets, such as .js or .css files, which are referenced directly.
For instance, my plugin accesses assets like this:
$cssFile = $mainPluginURL . 'vendor/graphql-by-pop/graphql-clients-for-wp/build/static/css/main.css';

After transpiling, this code must be fixed:
$cssFile = $mainPluginURL . 'vendor/phabel/transpiler71/graphql-by-pop/graphql-clients-for-wp/build/static/css/main.css';

Maybe this is related to #10, as to enable developers to add custom hooks to fix these issues.

I imagined some packages might have issues with this, I guess I'll just have to create a symlink from the original path to the transpiled path.

Now the transpiled packages are copied to the original path, too.
Symlinking is a bit more problematic, since it's broken on Windows, and composer has some issues with it anyway due to the lack of cleanup logic, still have to figure out how to stuff it into composer (somewhat related to #8).