Redeploy reports that it has failed, but if you refresh it, it actually worked

I'm not sure if this is specifically a trial app issue or not, but redeploying is reporting failures to me (and the status in the tree indicates as much) but if I refresh the tree, the Active deploy changes and the site reflects that the changes were redeployed.
Weird.

This was happening because I edited a file directly from my files node.  Probably not something worth worrying about.  A couple of redeploys later, and the issue resolved itself.


ResourceLoader.load_threaded_request throws Parse Error when loading multiple resources

Godot version
4.0.2
System information
Windows 10, AMD Ryzen 7 3700X
Issue description
When using ResourceLoader.load_threaded_request(path) to load mulitple resources simultaneously, some resources are not properly loaded and the error load: res://resources/test - Copy (11).tres:3 - Parse Error: [ext_resource] referenced nonexistent resource at: <path> <C++ Source>  scene/resources/resource_format_text.cpp:490 @ load() is thrown. (when the referenced resource at the path is obviously present).
In godot 4.0.1-stable this error doesn't occur.
Seems related to previous issue [#71726]
Steps to reproduce
start loading a bunch of resources using ResourceLoader.load_threaded_request(resource_path).
In the minimal reproduction project, simply run the main scene.
Minimal reproduction project
ResourceLoaderMRP.zip

Same issue here.

When a scene containing multiple resources, this issue reproduces stablely.
[MRP]
Clone https://github.com/bakneko/godot-template and run in godot-4.0.2.
Check core/class/loader.gd

Same here.

Probably fixed by #74405.


Attempted to (cross-)compile a cgo program for Linux/ARMv5 using clang.
What did you see happen?
$ docker run --rm -it golang:1.22rc2
root@f78e65b632f2:/go# apt update && apt install -y crossbuild-essential-armhf clang
[...snip...]
root@f78e65b632f2:/go# clang --version
Debian clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
root@f78e65b632f2:/go# CC='clang --target=armv5-none-linux-gnu -mfloat-abi=hard --sysroot=/usr/arm-linux-gnueabihf' GOARCH=arm GOARM=5 CGO_ENABLED=1 go build runtime/cgo
# runtime/cgo
gcc_libinit.c:44:8: error: large atomic operation may incur significant performance penalty; the access size (4 bytes) exceeds the max lock-free size (0  bytes) [-Werror,-Watomic-alignment]
gcc_libinit.c:47:6: error: large atomic operation may incur significant performance penalty; the access size (4 bytes) exceeds the max lock-free size (0  bytes) [-Werror,-Watomic-alignment]
gcc_libinit.c:49:10: error: large atomic operation may incur significant performance penalty; the access size (4 bytes) exceeds the max lock-free size (0  bytes) [-Werror,-Watomic-alignment]
gcc_libinit.c:69:9: error: large atomic operation may incur significant performance penalty; the access size (4 bytes) exceeds the max lock-free size (0  bytes) [-Werror,-Watomic-alignment]
gcc_libinit.c:71:3: error: large atomic operation may incur significant performance penalty; the access size (4 bytes) exceeds the max lock-free size (0  bytes) [-Werror,-Watomic-alignment]

What did you expect to see?
The runtime/cgo package should build without error, like when using gcc.
root@f78e65b632f2:/go# CC=arm-linux-gnueabihf-gcc GOARCH=arm GOARM=5 CGO_ENABLED=1 go build runtime/cgo
root@f78e65b632f2:/go# arm-linux-gnueabihf-gcc --version
arm-linux-gnueabihf-gcc (Debian 12.2.0-14) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This seems to have been introduced with CL505455 addition of __atomic_load_n.
This bit exceeds the max lock-free size (0  bytes) looks weird, a size can never be zero, It is probably some sort of compiler bug.
clang-14 --target=armv5-none-linux-gnu -mfloat-abi=hard --sysroot=/usr/arm-linux-gnueabihf test.c
test.c:8:15: warning: large atomic operation may incur significant performance penalty; the access size (4 bytes) exceeds the max lock-free size (0  bytes) [-Watomic-alignment]
        int result = __atomic_load_n(&myVar, __ATOMIC_RELAXED);
                     ^

I think "max size 0" means that the platform does not natively support atomic operations and they have to be emulated. https://bugs.llvm.org/show_bug.cgi?id=38593

Interesting! That seems to be the real reason, it is just poorly worded.
Mauri

I wonder if we can just ignore the warning in runtime/cgo. Since we build with -Werror it ends up failing the build. Depending on the C compiler version -Wno-atomic-alignment might not even be available. The user can also supply this flag manually when it is needed, but given the misleading error message that might not be initially clear, but that is clang to blame.
diff --git a/src/runtime/cgo/cgo.go b/src/runtime/cgo/cgo.go
index 1e3a502918..4bd8e6a53a 100644
--- a/src/runtime/cgo/cgo.go
+++ b/src/runtime/cgo/cgo.go
@@ -29,6 +29,8 @@ package cgo

 #cgo solaris CPPFLAGS: -D_POSIX_PTHREAD_SEMANTICS

+#cgo arm CFLAGS: -Wno-atomic-alignment
+
 */
 import "C"

cc @golang/runtime @golang/arm

In triage, one thing we're wondering is if this error is actually benign and we're OK with unaligned atomics (which IIRC break some atomicity guarantees) or if we need to actually fix this in the code CL 505455 modified.
CC @cuonglm @ianlancetaylor

I don't think the issue here is alignment. I think clang is just warning that ARMv5 doesn't really have atomics, as @corhere mentioned.
In our own atomic code, we call into a kernel-provided stub to help with atomics on ARM <7: https://cs.opensource.google/go/go/+/master:src/runtime/internal/atomic/sys_linux_arm.s;l=79-83.
In the kernel, that is https://elixir.bootlin.com/linux/latest/source/arch/arm/kernel/entry-armv.S#L749 -> https://elixir.bootlin.com/linux/latest/source/arch/arm/include/asm/assembler.h#L374. For ARMv5 that is ultimately a nop, because Linux simply doesn't support SMP for ARMv5 at all.
I don't know what the clang-generated code looks like (if someone could provide the disassembly, that would be helpful), but I suspect it is just warning that it has to call into a helper.
If this is accurate, I suspect ignoring the warning is fine.

Compiler Explorer
a:
  push {r11, lr}
  ldr r0, .LCPI0_0
  mov r1, #0
  bl __atomic_load_4
  pop {r11, lr}
  bx lr

The __atomic_load_4 symbol is defined in GCC's libatomic. (When the -latomic flag is provided, Compiler Explorer will link the binary but won't render the disassembly for some reason.) The libatomic function is presumably implemented in terms of the kernel memory-barrier helper.

If we only see the warnings for runtime/cgo/gcc_libinit.c, then we can add a #pragma line to that file to disable the warning.  Something like
#pragma GCC diagnostic ignored "-Watomic-alignment"

I just confirmed that this works. It is less intrusive than disabling the warning in a #cgo directive. Thanks, Ian.


[BUG] Progress reported on every step, regardless of value of steps_per_print

Describe the bug
After upgrading from 0.7.5 to 0.7.7, deepspeed is reporting progress on every training step, regardless of the value of stepe_per_print in the config.
I see something like
[2022-12-12 17:19:59,974] [INFO] [timer.py:197:stop] 0/1567, RunningAvgSamplesPerSec=13.03086402503502, CurrSamplesPerSec=13.308834034384216, MemAllocated=2.66GB, MaxMemAllocated=19.12GB

at every tick of my progress bar.
It looks like #2553 made some changes to this reporting for reasons I don't quite understand, so maybe this was always the intended behavior. But it makes the tqdm progress bars I'm using to monitor my training pretty useless.
Expected behavior
Previously, that info would only be printed every steps_per_print steps
ds_report output
DeepSpeed C++/CUDA extension op report
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
JIT compiled ops requires ninja
ninja .................. [OKAY]
op name ................ installed .. compatible
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
[WARNING]  sparse_attn requires a torch version >= 1.5 but detected 2.0
[WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
async_io ............... [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
spatial_inference ...... [NO] ....... [OKAY]
DeepSpeed general environment info:
torch install path ............... ['/opt/miniconda3/envs/hf/lib/python3.8/site-packages/torch']
torch version .................... 2.0.0.dev20221210+cu117
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.3
deepspeed install path ........... ['/opt/miniconda3/envs/hf/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.7.7, unknown, unknown
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7
System info (please complete the following information):

OS: Ubuntu 22.04
GPU count and types: 2x 3900
Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
Python version: 3.8.5

@jncasey, apologies for the break. We are addressing with #2603

Same issue. Please release the new version soon :)

@Parskatt, please see #2629

@Parskatt, are you still seeing this issue with master or v0.80? It should have been fixed by #2603. Thanks!

@tjruwase Sorry, I've recently been using pytorch's DDP instead, so I can't check.

@Parskatt, thanks for the response. Glad to hear you are unblocked.
Will close this for now, but feel free to re-open if needed.


Make session cookies secure by default

I noticed that enable :sessions will set cookies with HttpOnly but not Secure. There should be a way of enabling this by default?

Have a look in this post at Stackoverflow
http://stackoverflow.com/questions/12344088/how-to-make-rack-session-cookies-httponly

I already read that. Padrino should make it easier to enable secure sessions. Maybe check if Rack::SSL was added to the middleware stack.

I haven't done it yet but this is in my TODO list.
https://github.com/tobmatth/rack-ssl-enforcer
https://github.com/josh/rack-ssl

In fact, cookie without the secure attribute couldn't be transferred safely even if SSL is available.
So I basically agree with your opinion, but I worried our compatibility.
If we enable the option by default, existing project mixed of http and https will be broken.
So I’d like to suggest to provide the secure attribute as optional feature.

@namusyaka there was precedent where we opted for activating the feature instead of moving using a backward-compatible path: safe buffers.
There is the option of implementing the feature now and writing to log that it will be flipped with the next version if not explicitly set. Also, the encryption state of the connection is known through the request object, so it is possible to warn if a secure cookie goes out over a non-secure connection.

Where is this currently? I just ran into this trying to secure an updated padrino app (and esp now heroku is providing free cert mngmt for paid dynos).
having enable :sessions should provide a secure option easily under the least principle of surprise and would be a great "secure by default" option. More to the point, trying to find documentation on this is old and outdated, which makes one slightly skittish turning on ssl in padrino (in fact, even disabling sessions and putting in rack-ssl-enforcer/rack-ssl tells me my csrf protection needs to also be disabled.). This should be way easier considering how many people are not security experts and really just wanna "turn ssl on" in a transparent way.
I still continue to be floored by padrino though and the work you guys have all done. I am surprised how often i reach for it as a tool compared to rails or django these days.  =]

@wakatara I'm not part of the team anymore, but I think most that's missing is someone having time and sending that PR :).

Hehe... Fair enough. So, what needs to be done precisely, implement the rack-ssl gem solution inside the sessions that padrino uses?
If I understand correctly (and I am by no means an expert on this, so please correct me if I'm wrong since I've just been reading up on it), in order to have this support ssl properly this needs two things?

Set and use session cookies only over secure connections
Mark cookies as Secure so the browser doesn’t transmit them when requesting a non-secure URL

Is that correct? Would this support a minimum for being able to use enable :sessions after this on ssl and having it work? I'd be willing to take a shot at it, but might need a little help since I've never waded into the framework code.


Please update axios

Description
The types returned from AxiosDataFetcher::fetch() from @sitecore-jss/sitecore-jss-nextjs is incompatible with the actual type declared in axios. This happens after updating Axios to the latest version.
Please note that the code runs fine, but won't perform a production build because of a typing error.
JSS returns this type:
export interface AxiosResponse<T = any>  {
  data: T;
  status: number;
  statusText: string;
  headers: any;
  config: AxiosRequestConfig;
  request?: any;
}

Whereas an up-to-date Axios package expects a different, incompatible one:
export interface AxiosResponse<T = any, D = any> {
  data: T;
  status: number;
  statusText: string;
  headers: RawAxiosResponseHeaders | AxiosResponseHeaders;
  config: InternalAxiosRequestConfig<D>;
  request?: any;
}

It bares noting that putting overrides in package.json might work, but doesn't in our case, for some reason. A little bit like this:
{
  "dependencies": {
    "@sitecore-jss/sitecore-jss-nextjs": "^21.0.7",
    "axios": "^1.3.3"
  },
  "overrides": {
    "axios": "$axios"
  }
}

Should work, but doesn't.
Expected behavior
No error, and an up-to-date Axios subdependency in sitecore-jss-nextjs.
Steps To Reproduce

Install the jss-nextjs starter
Update axios to its latest version
Open data-fetcher.ts
Observe the error that VScode or the build process will spew out.

Possible Fix
Please update Axios in @sitecore-jss/sitecore-jss-nextjs at your earliest convenience.
Your Environment

Sitecore Version: 9.3
JSS Version: 21.0.7
Browser Name and version: Firefox 110
Operating System and version (desktop or mobile): Windows 10 desktop
Link to your project (if available): N/A

Screenshots

Hi @thany !
First, thank you for reporting this.
Second, a bit of bad news and some context. Updating axios to latest version unfortunately breaks the react app due to this bug: https://github.com/facebook/create-react-app/issues/11889
Regardless, we'll address the issue one way or another: either update axios after react-scripts is updated (hopefully soon), or replace axios usage in package with fetch, to avoid conflicts.
And last, but not least, there are some workarounds you could use. While not an official fix, I hope they can help.

AxiosDataFetcher is only used in Styleguide-Tracking component. If it's not something your nextjs app uses, you can remove lib/data-fetcher and your app should work fine after.
You could bring AxiosDataFetcher into the app remove the import from base package. Version upgrade requires some changes to it, however, this is what it would look like when using v1.x:
https://github.com/Sitecore/jss/blob/feature/axios-upgrade/packages/sitecore-jss/src/axios-fetcher.ts
This should help avoid version conflicts too.

I have also tried to override the depedency in package.json, from memory I went like this:
"overrides": {
  "axios": "$axios"
}

It worked fine. But strangely this also proves that updating axios can be done without errors, or compiling it with this override in place would undoubtedly reproduce your error. Perhaps because CRA is not a part of a Next.js app, iirc.

Can you please look into this because of: An issue discovered in Axios 0.8.1 through 1.5.1 inadvertently reveals the confidential XSRF-TOKEN stored in cookies by including it in the HTTP header X-XSRF-TOKEN for every request made to any host allowing attackers to view sensitive information.

Newer versions of axios would also address the following Security findings:

https://app.snyk.io/vuln/SNYK-JS-AXIOS-6032459 (https://www.cve.org/CVERecord?id=CVE-2023-45857)
https://app.snyk.io/vuln/SNYK-JS-AXIOS-6124857
https://app.snyk.io/vuln/SNYK-JS-FOLLOWREDIRECTS-6141137 (https://www.cve.org/CVERecord?id=CVE-2023-26159 - once axios updates it's own dependency)

+1 to upgrade axios


Latest intel-CPU not working properly

Hi!
It seems like if you use docker pull dizcza/docker-hashcat, the hashcat throws the following error within the container:
root@901d3870e258:/# hashcat -b
hashcat (v3.6.0) starting in benchmark mode...

clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR

Started: Tue Oct 16 15:11:50 2018
Stopped: Tue Oct 16 15:11:50 2018

After attempting to manually fix the problem, I went back and got the previous intel-cpu base image from Dockerhub and after changing the download URL for hashcat, I got it working fine:
hashcat (v3.6.0) starting in benchmark mode...

OpenCL Platform #1: Intel(R) Corporation
========================================
* Device #1: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz, 246/987 MB allocatable, 1MCU

Hashtype: MD4

Speed.Dev.#1.....:   116.9 MH/s (8.82ms)

Hashtype: MD5
......

I think the problem may be the OpenCL runtime as that's the only thing that's changed.
This was tested on Ubuntu 18.04.1, Ubuntu 16.04.3 and generic AWS linux.

I got it working by taking the Dockerfile from https://hub.docker.com/r/dizcza/docker-hashcat/builds/bunmybfnrni7lwnumkjroqf/ which has all the old dependencies, but changed the following:
RUN cd /hashcat && \
    wget --no-check-certificate https://hashcat.net/files_legacy/${HASHCAT_VERSION}.7z && \
    7zr x ${HASHCAT_VERSION}.7z && \
    rm ${HASHCAT_VERSION}.7z

To this:
RUN cd /hashcat && \
    wget --no-check-certificate https://hashcat.net/files/${HASHCAT_VERSION}.7z && \
    7zr x ${HASHCAT_VERSION}.7z && \
    rm ${HASHCAT_VERSION}.7z

I hope this helps! :)

Yeah. I would like to keep though intel opencl runtime up to date. Besides, links for the previous opencl runtimes that I hardcoded in dockerfile might get lost/removed from official intel site.

I just reverted the broken change in intel-cpu branch. Tested on my own PC. You can test it in a few minutes - once dockerhub builds the latest commit.
If anyone gets it working with the newest opencl runtime and headers, please, let me know.

I just reverted the broken change in intel-cpu branch. Tested on my PC. You can test it in a few minutes.
If anyone gets it working with the newest opencl runtime and headers, please, let me know.


Change url for creating issue

Is your feature request related to a problem? Please describe.
When creating a issue, I would like to open choose issue type
Describe the solution you'd like

// CONTRIBUTING.md
- [new-issue]: https://github.com/nostalgic-css/NES.css/issues/new
+ [new-issue]: https://github.com/nostalgic-css/NES.css/issues/new/choose

Because the issue template is not applied when opening the issue creation page directly.

:tada: This issue has been resolved in version 2.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


move sql driver imports from lib to cmd

This moves all the sql driver imports out of the lib package into cmd.
This is so that compiling apps which import goose don't take forever to compile drivers they don't need.
See also https://bitbucket.org/liamstask/goose/issue/47/is-it-necessary-to-import-sql-drivers-in

@zabawaba99 this might fix the issue you were having earlier this morning.

:rocket: :man:


Figure out CI issues

Since we've added inference tests, the CI almost always times out.
We should figure out why, starting by putting times in tests to see where the tests fail.

Issue opened here: https://github.com/huggingface/api-inference-community/issues/94

Let's close this for now after #212 has been merged.


Evaluation score calculation

Last part of and closes https://github.com/keptn/keptn/issues/6560
Signed-off-by: Klaus Strießnig<EMAIL_ADDRESS>

Preview: https://deploy-preview-1361--keptn.netlify.app/docs/concepts/quality_gates/#quality-gate-result-on-the-keptn-bridge

Very nicely done!


Release 0.11.2

All checks have passed @Licenser

Look at the files section Clippy added an annotation. It seems that warnings as error aren't turned on we should fix that

ok, enabledp edentic clippy, denied warnings and made clippy shut up. :)


Test files (images, XMP, actions) added

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have signed the Adobe Open Source CLA.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

@kmikawa I'll update the links in a separate PR. This one only for adding the images.


Getting "The service is unavailable" while deploying the app using azure:app service

Does this occur consistently?
Repro steps:

Action: postCreateWebApp
Error type: 503
Error Message: The service is unavailable.
Version: 0.17.0
OS: darwin
OS Release: 18.7.0
Product: Visual Studio Code
Product Version: 1.46.0
Language: en

Call Stack
extension.bundle.js:309:399343
retryCallback extension.bundle.js:225:116429
retryCallback extension.bundle.js:157:31051
extension.bundle.js:157:32664
handleRedirect extension.bundle.js:171:7747
extension.bundle.js:157:29836
Request._callback extension.bundle.js:1:584810
Request.r._callback.r.callback.r.callback extension.bundle.js:201:4468
Request.emit events.js:203:13
Request.<anonymous> extension.bundle.js:201:17577
Request.emit events.js:208:15
IncomingMessage.<anonymous> extension.bundle.js:201:16367
Object.onceWrapper events.js:291:20
IncomingMessage.emit events.js:208:15
endReadableNT _stream_readable.js:1168:12
processTicksAndRejections task_queues.js:77:11

Hi @neethumoltp
Thanks for reporting. This error is usually transient and can be resolved by trying again at a different time.
But if it's not, could you provide repro steps for this?


[RIP-10] Add unit test for ConsumerSubCommand

What is the purpose of the change
unit test for ConsumerSubCommand
Brief changelog
unit test for ConsumerSubCommand
Verifying this change
unit test for ConsumerSubCommand
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.2%) to 48.879% when pulling 53b2695ec9a87dde55b9d5cdcb6e885ff032c0ff on crazyowen:develop into 4f8d96c4a753d24dc2c2e8933d5d2908821f922f on apache:develop.

Hi,@crazyowen you should notice your code coverage!

Hi,@crazyowen you should notice your code coverage!

Yes, I have added new test for ConsumerSubCommand.


go/build: reduce redundancy in error messages from Import in module mode

What version of Go are you using (go version)?

$ go version
go version devel +a0894ea5b5 Mon Oct 7 18:50:14 2019 +0000 darwin/amd64
Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jayconrod/Library/Caches/go-build"
GOENV="/Users/jayconrod/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/jayconrod/go"
GOPRIVATE=""
GOPROXY="off"
GOROOT="/Users/jayconrod/Code/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/jayconrod/Code/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jayconrod/Code/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/go-build844872233=/tmp/go-build -gno-record-gcc-switches -fno-common"
What did you do?
$ go mod init m
$ cat >hello.go <<EOF
package main

import (
	"fmt"
	"go/build"
	"os"
)

func main() {
	os.Setenv("GOPROXY", "off")
	_, err := build.Import("example.com/hello", ".", build.FindOnly)
	fmt.Println(err)
}
EOF
$ go run hello.go

What did you expect to see?
cannot find module providing package example.com/hello: module lookup disabled by GOPROXY=off

What did you see instead?
Package name is repeated three times, and some irrelevant implementation details are exposed.
go/build: importGo example.com/hello: exit status 1
can't load package: package example.com/hello: cannot find module providing package example.com/hello: module lookup disabled by GOPROXY=off

I wrote up a related experience report on the general strategy for reducing redundancy. That strategy seemed to work out pretty well for the error messages we tried last cycle.


Teleport - checking funds visual bug

how this can happen
https://github.com/kodadot/nft-gallery/assets/5887929/dd8d3d68-0afb-4f7f-8f09-0d119cb8b3ce

👋


Pinning columns on the right

Fixes #31.

thanks for the pr. i'm going to have to put time aside to go through this and think it through.

pining columns to the right is new functionality i've introduced in v3. you can check out the latest in that branch, or wait for v3 release in the coming days.
by the way, i didn't run this PR, i had a quick look at the code. when i implemented, i got into problems getting it working with floating header and footer, and also grouping (plus the new multi-level grouping). in addition, the columnController has been totally re-factored to be easier to understand and support the multi-level of grouping - you might be interested in taking a look!


NIFI-1822: Allow concurrent execution in ExecuteScript

Added a ScriptEngine pool for use by ExecuteScript, to enable it to run with multiple concurrent tasks.

I also updated MockProcessorContext to allow for a configurable maxConcurrentTasks value, as it was previously hard-coded to 1.

I added some unit tests (not yet complete) and made the changes referenced above in a branch here: https://github.com/apache/nifi/compare/master...alopresto:pr387?expand=1

I have opened a new PR 443 with unit tests, fixes for the above issues, and additional changes.

Thanks much for the corrections/improvements/additions!


Issue When I try to train  Train BiT++ on Colab

I am trying to train your model on Google colab uisng following command:
!python -m torch.distributed.launch --nproc_per_node=1 train_bit.py --config ./configs/bit++_rbi.yaml
But i get following error (most probably regarding some issue related to the GPU available on Colab):
`ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 7212) of binary: /usr/local/bin/python
Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/usr/local/lib/python3.10/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/usr/local/lib/python3.10/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/usr/local/lib/python3.10/site-packages/torch/distributed/run.py", line 752, in run
elastic_launch(
File "/usr/local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train_bit.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time      : 2023-06-19_09:52:32
host      : aea723e3180b
rank      : 0 (local_rank: 0)
exitcode  : 1 (pid: 7212)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================`
As I dont know about the backend settings needed to run this code regarding GPU, can you please guide me how to solve this error

I have similar error when using inference in a linux computer/local gpu. Do you have a solution yet? Thank you!
/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects --local_rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
warnings.warn(
Warning! No module named 'sounddevice'
Warning! No module named 'keras'
local_rank: 0
Traceback (most recent call last):
File "./tools/inference/inference.py", line 59, in
torch.cuda.set_device(local_rank)
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/cuda/init.py", line 314, in set_device
torch._C._cuda_setDevice(device)
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/cuda/init.py", line 217, in _lazy_init
torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 53740) of binary: /home/riselab/anaconda3/envs/BiT/bin/python
Traceback (most recent call last):
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run
elastic_launch(
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/riselab/anaconda3/envs/BiT/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
./tools/inference/inference.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time      : 2023-07-11_07:48:34
host      : riselab
rank      : 0 (local_rank: 0)
exitcode  : 1 (pid: 53740)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Thank you! It turned out to be an driver issue.


juypter kernel "not ready" when starting in carnd-term1 environment

test.ipynb notebook does not run.
kernel is not up.
environment is build and activated succesfully.
running conda locally on windows 10 64bit.

what solved the issue for me was to change the kernel.json file located in
\conda\envs\carnd-term1\share\jupyter\kernels\python3
it should be{
"argv": [
"PATH/conda/envs/carnd-term1/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}
instead of
{
"argv": [
"PATH/conda/envs/carnd-term1/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}


Don't reference versions in admin editions filter

This was added to get around an issue during the Rails 4.1 upgrade that meant
ActiveRecord wasn't able to generate a valid query when trying to eager load the
last_authors on a collection of editions. Unfortunately, this resulted in a
highly inefficient queries that required a lot of writing to disk and was causing
problems with the MySQL server.
To get around the issue whilst we investigate further, the eager-loading of
last_author is being removed. This makes the page n+1 (where n is 50) as it now
loads each last_author with an individual query, but the additional impact to
users is fairly minimal on this already-slow page.
The root problem is that ActiveRecord doesn't seem to be able to eager-load the
last_author association correctly. This association is a has_one that is
through another has_one association (most_recent_version) that also has an
order clause to determine the precise one.

:+1:


Create share commitments using nmt instead of a normal merkle tree

Currently, we are creating the share commitments by taking a normal merkle root over the nmt subtree roots. We should use an nmt over the subtree roots as well.

We thought about doing this for the row and column roots as well. The problem here is that the nodes in a nmt need to be ordered by nid . The nid of a parity share is 0xfffff.... . This means when you erasure code a column, (c1/r2k) will have the nid of 0xfffff.... and (c2/r1) will have a nid of most likely 0. But in the nmt over the column roots (c1/r2k) will be on the left of (c2/r1). This breaks the nmt making it not possible to construct a nmt over the col & row roots as we are currently constructing them.

One question that came up if the nmt is required to be a binary tree. This might not be the case as the nmt is forked from nebulous which does not require a binary tree.

There is an assumption in the NMT algorithm that the tree is perfect (i.e. the leaves are a power of 2).

It's not clear to me what the benefit/rationale of using an NMT is. All shares have the same known namespace, so there's no need to know the min/max NID of the shares. The commitment won't be identical above the subtree roots for blobs that span multiple rows since parity shares aren't accounted for in the PFB commitment.

There is an assumption in the NMT algorithm that the tree is perfect (i.e. the leaves are a power of 2).

Not advocating for anything here but just noting that is sth we introduced and could change in NMT.

yes I think we should do this if we can fit it in after thinking about it. The main benefit would be that other implementations would be able to use a single merkle tree implementation.

There is a consideration that we could improve ZK-rollup computation by not having a tree but only one hash. The question is if there is a situation where we need a Merkle proof of the commitment from the subtree roots. Then we would increase this inclusion proof by having all subtree roots as siblings.

Just noting that there is a size downside to using an NMT over the subtree roots:

Tree
Commitment size

Binary Merkle Tree
32 bytes

Namespaced Merkle Tree
98 bytes

Let's say we live in a world where we get our commitment proofs by PFB Transactions, as proposed in ADR 11. Now that a fraud poof happens in the state machine of rollkit we would normally need a Merkle inclusion proof to the shares in which the fraud exists for compact fraud proofs. The question is whether we could get smaller inclusion proofs of the Merkle Tree commitment rather than the RowRoots. And in some rare cases, this could be true. In most cases, this will not be true. I don't see a merkle inclusion proof to the commitment being useful.


Fix nil pointer dereference error when parsing config map

Changes

:bug: Fix nil pointer dereference error when parsing the cors-policy field config-contour config map

PS: Error found while investigating issues in serving tests:

https://github.com/knative/serving/pull/15204
https://github.com/knative/serving/pull/15231

/kind bug
Release Note

Resolved a nil pointer exception triggered by the cors-policy field being set to an empty value.

/lgtm
/approve

Was there some malformed config in the serving tests?

Was there some malformed config in the serving tests?

in the serving e2e script, I set the cors-policy field to empty to disable the feature, which should result in a parsing error. That's why the subsequent tests fail, since the net-contour pod enters in an error state and doesn't recover. I'm waiting on the test PR to confirm if I'm properly removing the field instead of setting it to empty

/cherry-pick release-1.14


When hitting enter to select a suggestion, the input does not clear.

This is a very specific bug. It occurs when:

focusFirstSuggestion is set to true.
The input is to be cleared when a suggestion is selected.
- i.e. onSuggestionSelected() contains this.setState({ value: '' });.
You hit Enter to select the first suggestion that was automatically focused onto.
The first suggestion must not match exactly the thing that was typed in.

Demo
This pen has two major differences with the basic one:

focusFirstSuggestion is set to true.
A function is passed in into the onSuggestionSelected prop:

  onSuggestionSelected = () => {
    this.setState({ value: '' })
  };

Try it here.

Focus on the input field.
Type c (lowercase), and wait for suggestions to appear.
Press Enter to select the first suggestion.

Observed behaviour:
Note that the c remains, and the input is not cleared.
Expected behaviour:
The input should be cleared, since there is already a focus on the first item of the suggestions. And we have onSuggestionSelected set to clear the input for us. Unfortunately, onChange is being called after onSuggestionSelected when it should not be.
The Problem
When the suggestion is selected by hitting Enter, the onSuggestionSelected function is called, and the input is properly cleared. However, immediately after that, maybeCallOnChange is called.
Line 369 to 376:
this.onSuggestionSelected(event, {
  suggestion: focusedSuggestion,
  suggestionValue: newValue,
  sectionIndex: focusedSectionIndex,
  method: 'enter'
});

this.maybeCallOnChange(event, newValue, 'enter');

This takes us to the definition on line 160 for maybeCallOnChange():
maybeCallOnChange(event, newValue, method) {
  const { value, onChange } = this.props.inputProps;

  if (newValue !== value) {
    onChange(event, { newValue, method });
  }
}

Since the current value was typed in by the user, and the newValue came from the selected suggestion, onChange is called with the newValue. This leads us to our simple onChange function:
onChange = (event, { newValue, method }) => {
  this.setState({ value: newValue });
}

As you can see, it just sets the value to the state and you end up with the new suggestion in the input rather than a cleared box.
The (temporary) Solution
In the onChange handler function that we pass down to the Autosuggest component, we must first check if the method is enter. If it is, then prevent the function from continuing:
onChange = (event, { newValue, method }) => {
  // prevent change if artist was just selected
  if (method === 'enter') return;
  // otherwise, let the change go through
  this.setState({ value: newValue });
}

A Permanent Solution
We need to prevent onChange from being called after onSuggestionSelected when:

focusFirstSuggestion is set to true, and;
The user pressed Enter to trigger onSuggestionSelected.

@adrianmcli What a great example of a properly submitted PR!
Thanks a lot for taking the time to explain the problem and fixing it 👍

This is fixed in v7.0.2.

@marcos-ld thanks for the kind comments! It's been 1.5 years since the PR was merged, and since then I've actually met @moroshko in person at ReactConf too haha. Always nice to receive these kinds of comments 👍
It does kind of make me sad that you had to do this workaround though, I wonder if there's a way to make it less implicit. At the end of the day, it seems that onChange is sometimes getting called when the user doesn't really want to call it. Maybe there should be a separate onSelectionChanged that is called? Hmmm...


FeedBack - Split ReadMe and ChangeLog in two separate files

What do you think of the splitting of readme and changelog in two .md files ?
We can also put the detail of resources in a new file ?
Suggested PR #190

Awesome suggestion! I'm all for it! :smile:
@kwirkykat would this work for your release management?

@luigilink Regarding detail of the resource. Is it how SharePointDsc does it? if so we have a separate issue #182 for that already :D

@kwirkykat or @mbreakey3 Is it okay that we split this into to separate files like SharePointDsc? There won't be any problems when you release?

@johlju No problems splitting out the change log


Avoid gdscript errors.

Gdscript was erroring on the latest.

GDScript: Rework type check with is (GH-73489).
This change involves a slight compatibility breakage for the behavior of the is operator. It now requires a type known as compile time, so it’s not possible to use it to check if two variables have the same type. For this, use typeof() or the new is_instance_of() function.


[branch] add feature/mobx

Do you need a new branch for mobx?
It seems you should have a new feature branch about mobx first, then I can send PR to it.
https://github.com/HaveF/react-universally-1/commit/5b4ffff8abe851ae0db8d71d7a5840e9e4bcec42

You can create a PR against master/next now and change the base branch later on. Its one of the nice features added rather recently here on Github :)

Released :)  feature/mobx
Thanks again!


Not working

Bug

Describe the bug

When starting biscuit I get this error
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Ramit Katyal\git\Biscuit\biscuit\__main__.py", line 17, in <module>
    from biscuit.app import App
ModuleNotFoundError: No module named 'biscuit'

https://github.com/billyeatcookies/Biscuit/assets/96387132/cef4ca01-34ea-4500-b6ce-4f3262e1411a
(i have custom powershell prompt dont mind it)
By the way I already have the libraries installed
System Information

OS: Windows 10
Python Version: 3.11

@arnav56745 sorry for the delay! (uni)
you need to run with -m flag as a module. see the README for running and compiling!

@billyeatcookies  in the previous readme it said to
$ python biscuit

thats why i got this error


Failed to work with mypy --install-types .

Steps to Reproduce

Install mypy by rye install mypy.
Run mypy on some project with command mypy --install-types ..

Expected Result
Missing stub packages are installed.
Actual Result
Installing missing stub packages:
/Users/user/.rye/tools/mypy/bin/python -m pip install types-python-dateutil types-requests

Install? [yN] y

/Users/user/.rye/tools/mypy/bin/python: No module named pip

note: Run mypy again for up-to-date results with installed types

Version Info
rye 0.15.2
commit: 0.15.2 (e2beb3a90 2023-10-04)
platform: macos (x86_64)
self-python<EMAIL_ADDRESS>symlink support: true

Stacktrace
No response

Hi, I'm not sure if I understand correctly. --extra-requirement seems to be options of tools. But shouldn't stub packages are install for individual virtualenvs of different Python version?

If you want to use it in the project's own virtual environment, you should be able to use it via rye add types-python-dateutil types-requests & rye sync (not verified)?

This error happens because pip is not installed into virtualenvs managed by rye. This is very intentional but it does have this unfortunate effect. You can install pip into that virtualenv by adding it as a requirement, but manually managed tools and dependencies will likely be removed on next sync.
It's better to directly add requirements instead of relying on other tools installing stuff.


Selenium 4 Alpha - Webdrivers not auto-updating with require call

Summary
I've come back to do some coding. And webdrivers is pinning chromedriver to v71
Debug Info
Please provide the following information for bug reports:

Webdrivers version:

    webdrivers (4.4.1)
      nokogiri (~> 1.6)
      rubyzip (>= 1.3.0)
      selenium-webdriver (>= 3.0, < 4.0)

Ruby version: 2.4.9 - Issue on all rubies
Operating system / CI Environment: CI for SitePrism. Repo and failure is here: https://travis-ci.com/github/site-prism/site_prism/jobs/343236075
Browser and version: Chrome latest

Expected Behavior
Browser to run
Actual Behavior
Browser error on <Selenium::WebDriver::Error::WebDriverError: no sessionId in returned payload>
Possible issue is that webdrivers is currently permitting 4.0.0.alpha versions. Not sure if it's tested with those. The issue is only with the alpha versions, and selenium 3.142.x seems fine.
Note that 4.0.0.alphaN is < 4.0

@luke-hill That's odd. Can you please set Webdrivers.logger.level = :DEBUG before launching the browser and share the logs here?

Done. I ran it again, there's a heap of debug statements in this travis run

Based on the logs you shared, webdrivers is successfully downloading a compatible chromedriver build version 83.0.4103.x for your Chrome build version 83.0.4103.x:
2020-06-04 07:56:27 DEBUG Webdrivers Browser version: 83.0.4103.97
2020-06-04 07:56:27 DEBUG Webdrivers making System call: ["/usr/bin/google-chrome", "--product-version"]
2020-06-04 07:56:27 DEBUG Webdrivers System call returned: 83.0.4103.97

2020-06-04 07:56:28 DEBUG Webdrivers making System call: ["/home/travis/.webdrivers/chromedriver", "--version"]
2020-06-04 07:56:28 DEBUG Webdrivers System call returned: ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416})

This tells me that what you're experiencing seems to be an issue between Selenium 4 and chromedriver's interaction. I haven't been keeping up with the changes in Selenium 4 to accurately tell you what exactly is going on, but please consider opening a ticket there. Another thing you can try is downgrading to Chrome 81 or 82 and see if it makes a difference.

Any advice you have for now? As this is running through travis and is affecting us releasing a CI upgrade to our gem support. Should we for now not support selenium4 alpha?

cc/ @p0deje could you shed some light alex? Maybe this is something you could help/advise on

I cannot reproduce the problem using 4.0 alpha. This is the test script I used:
requirer 'bundler/inline'

gemfile do
  source 'https://rubygems.org'

  gem 'selenium-webdriver', '4.0.0.alpha5'
  gem 'webdrivers', '4.4.1'
end

require 'selenium-webdriver'
require 'webdrivers'

Selenium::WebDriver.logger.level = :debug
Webdrivers.logger.level = :DEBUG

Selenium::WebDriver.for(:chrome)&.quit

$ ruby test.rb
/Users/p0deje/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/selenium-webdriver-4.0.0.alpha5/lib/selenium/webdriver/common/driver.rb:46: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/p0deje/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/selenium-webdriver-4.0.0.alpha5/lib/selenium/webdriver/common/driver.rb:74: warning: The called method `initialize' is defined here
/Users/p0deje/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/selenium-webdriver-4.0.0.alpha5/lib/selenium/webdriver/common/driver.rb:76: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/p0deje/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/selenium-webdriver-4.0.0.alpha5/lib/selenium/webdriver/common/driver.rb:297: warning: The called method `create_bridge' is defined here
2020-06-23 05:21:20 DEBUG Webdrivers Checking current version
2020-06-23 05:21:20 DEBUG Webdrivers /Users/p0deje/.webdrivers/chromedriver is not already downloaded
2020-06-23 05:21:20 DEBUG Webdrivers making System call: ["//Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "--version"]
2020-06-23 05:21:20 DEBUG Webdrivers System call returned: Google Chrome 83.0.4103.106

2020-06-23 05:21:20 DEBUG Webdrivers Browser version: Google Chrome 83.0.4103.106
2020-06-23 05:21:20 DEBUG Webdrivers Checking current version
2020-06-23 05:21:20 DEBUG Webdrivers /Users/p0deje/.webdrivers/chromedriver is not already downloaded
2020-06-23 05:21:20 DEBUG Webdrivers making System call: ["//Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "--version"]
2020-06-23 05:21:20 DEBUG Webdrivers System call returned: Google Chrome 83.0.4103.106

2020-06-23 05:21:20 DEBUG Webdrivers Browser version: Google Chrome 83.0.4103.106
2020-06-23 05:21:20 DEBUG Webdrivers making System call: ["//Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "--version"]
2020-06-23 05:21:20 DEBUG Webdrivers System call returned: Google Chrome 83.0.4103.106

2020-06-23 05:21:20 DEBUG Webdrivers Browser version: Google Chrome 83.0.4103.106
2020-06-23 05:21:20 DEBUG Webdrivers Making network call to https://chromedriver.storage.googleapis.com/LATEST_RELEASE_83.0.4103
2020-06-23 05:21:20 DEBUG Webdrivers Get response: #<Net::HTTPOK 200 OK readbody=true>
2020-06-23 05:21:20 DEBUG Webdrivers Latest version available: 83.0.4103.39
2020-06-23 05:21:20 DEBUG Webdrivers Deleting /Users/p0deje/.webdrivers/chromedriver.version
2020-06-23 05:21:20 DEBUG Webdrivers making System call: ["//Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "--version"]
2020-06-23 05:21:20 DEBUG Webdrivers System call returned: Google Chrome 83.0.4103.106

2020-06-23 05:21:20 DEBUG Webdrivers Browser version: Google Chrome 83.0.4103.106
2020-06-23 05:21:20 DEBUG Webdrivers Checking current version
2020-06-23 05:21:20 DEBUG Webdrivers /Users/p0deje/.webdrivers/chromedriver is not already downloaded
2020-06-23 05:21:20 DEBUG Webdrivers making System call: ["//Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "--version"]
2020-06-23 05:21:20 DEBUG Webdrivers System call returned: Google Chrome 83.0.4103.106

2020-06-23 05:21:20 DEBUG Webdrivers Browser version: Google Chrome 83.0.4103.106
2020-06-23 05:21:20 DEBUG Webdrivers making System call: ["//Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "--version"]
2020-06-23 05:21:20 DEBUG Webdrivers System call returned: Google Chrome 83.0.4103.106

2020-06-23 05:21:20 DEBUG Webdrivers Browser version: Google Chrome 83.0.4103.106
2020-06-23 05:21:20 DEBUG Webdrivers Making network call to https://chromedriver.storage.googleapis.com/LATEST_RELEASE_83.0.4103
2020-06-23 05:21:20 DEBUG Webdrivers Get response: #<Net::HTTPOK 200 OK readbody=true>
2020-06-23 05:21:20 DEBUG Webdrivers Latest version available: 83.0.4103.39
2020-06-23 05:21:20 DEBUG Webdrivers chromedriver URL: https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_mac64.zip
2020-06-23 05:21:20 DEBUG Webdrivers Making network call to https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_mac64.zip
2020-06-23 05:21:20 DEBUG Webdrivers Get response: #<Net::HTTPOK 200 OK readbody=true>
2020-06-23 05:21:20 DEBUG Webdrivers Successfully downloaded /var/folders/3s/yybq9c5n4pd0z9t4phtp6ml00000gn/T/20200623-73366-16rg2kychromedriver_mac64.zip
2020-06-23 05:21:20 DEBUG Webdrivers Decompressing /var/folders/3s/yybq9c5n4pd0z9t4phtp6ml00000gn/T/20200623-73366-16rg2kychromedriver_mac64.zip
2020-06-23 05:21:21 DEBUG Webdrivers Completed download and processing of /Users/p0deje/.webdrivers/chromedriver
2020-06-23 05:21:21 DEBUG Selenium Executing Process ["/Users/p0deje/.webdrivers/chromedriver", "--port=9515"]
2020-06-23 05:21:21 DEBUG Selenium polling for socket on ["<IP_ADDRESS>", 9515]
2020-06-23 05:21:21 DEBUG Selenium polling for socket on ["<IP_ADDRESS>", 9515]
2020-06-23 05:21:21 DEBUG Selenium polling for socket on ["<IP_ADDRESS>", 9515]
Starting ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
2020-06-23 05:21:21 INFO Selenium -> POST session
2020-06-23 05:21:21 INFO Selenium    >>> http://<IP_ADDRESS>:9515/session | {"capabilities":{"firstMatch":[{"browserName":"chrome"}]}}
2020-06-23 05:21:21 DEBUG Selenium      > {"Accept"=>"application/json", "Content-Type"=>"application/json; charset=UTF-8", "User-Agent"=>"selenium/4.0.0.alpha5 (ruby macosx)", "Content-Length"=>"58"}
2020-06-23 05:21:23 INFO Selenium <- {"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"83.0.4103.106","chrome":{"chromedriverVersion":"83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416})","userDataDir":"/var/folders/3s/yybq9c5n4pd0z9t4phtp6ml00000gn/T/.com.google.Chrome.EDu6fp"},"goog:chromeOptions":{"debuggerAddress":"localhost:59384"},"networkConnectionEnabled":false,"pageLoadStrategy":"normal","platformName":"mac os x","proxy":{},"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:virtualAuthenticators":true},"sessionId":"a2b06bafb9f18cf5a85ef922b2527ff9"}}
2020-06-23 05:21:23 INFO Selenium -> DELETE session/a2b06bafb9f18cf5a85ef922b2527ff9
2020-06-23 05:21:23 INFO Selenium <- {"value":null}

According to logs, the Webdrivers downloaded ChromeDriver and told Selenium to use it correctly.

Ok, I'll try on alpha5 in my repo again. If that fails again for the same reason I'll try do an in-line ruby script like you did. There will be differences as I'm using site_prism (I'll try remove cucumber).
Maybe it's some interop with some other gems, but I'm using a reasonably common stack (Capybara/Cucumber)

Ignore me PEBKAC - The issue is I was creating the payload with w3c set to false.
When setting the selenium webdriver logger to debug I get this
2020-06-23 11:19:30 INFO Selenium <- {"sessionId":"550ba3701972511900474d7d820d4835","status":33,"value":{"message":"session not created: Missing or invalid capabilities\n  (Driver info: chromedriver=83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416}),platform=Linux 4.15.0-106-generic x86_64)"}}


Fix sending telegram cropped message

We need to use unicode string in cropped message.

Is this going to be merged in any time soon?


KDCA

Added KDCA (still have features to add)

Couple things; it looks like UAE, CCA, and AFR airlines have been removed in this commit..?
Also seems to be a lot of unnecessary lines of code in the modified airline files; "alpha: false" is redundant and unneeded, as is changing "aircraft{ }" to "fleet{ default[ ]}}" when an airline only has one fleet type...
It looks like the aircraft in FDX.json have also been duplicated.

@jpeterson1346 Actually, the issue is that you still have old commits hanging around in your fork. When you updated to sync your fork with zlsa/gh-pages, you got everything "up to date", but didn't remove the extra commits that were squashed in the merge... When in doubt, just have a look at your fork and make sure your branch is "even" with the main repository before starting work on your next project. Then, all your work will probably be able to merge automatically, which is much easier on the reviewer (it's literally a single press of a big green button versus a few dozen steps on the command line, and manually resolving each conflict).

^You can see here that this branch is "91 commits ahead", most of those are 'left over', and only about 30 of them are actually related to your work on KDCA. To have the pull request only include those 30 commits that are relevant, you want to make sure you start working with a branch that is even (I keep one called 'clean' that I just keep in sync with zlsa/gh-pages, and make branches off of it for each of my projects). There's a nice guide on youtube about keeping forked repositories in sync with the upstream that I found helpful.
In theory, I may be able to fix this on my end, but there will be a LOT of conflicts to work through, and it would be better if you could find a way to start a new branch that is "even" with the main repository and make your changes there, so that they don't conflict with changes made in other already-merged pull requests.

I was able to extract the changes you wanted to make, and resolve all conflicts. This has now been merged via commit 02a5f70. Finally! (this was a tough one!) :smiley:


re-import support (from both actions and toolkit)

To get it work on 4.20 and later, #4  should be merged too.

This is necessary when working with external text editors, to simplify workflow and reduce inconsistency

When you use this asset commit, you should put
pTextAsset->Modify();
To TextAssetFactory.cpp 107 line:
if (FFileHelper::LoadFileToString(NewText, *ResolvedSourceFilePath))
{
pTextAsset->Text = NewText;
pTextAsset->Modify();
return EReimportResult::Succeeded;
}

@samrrr Thanks, added it.


iOS: Make sure extensions are utilizing database resources correctly

See this tweet https://twitter.com/groue/status/1203893850386976774?s=20 that references this document https://github.com/groue/GRDB.swift/blob/dev/dead10cc/Documentation/AppGroupContainers.md#use-a-database-pool

Hello,
This search for proper use of SQLite databases shared in an App Group container is part of a work-in-progress, which is not completed yet. The document above is part of the draft pull request https://github.com/groue/GRDB.swift/pull/663. This PR is part of a "plan" roughly described in https://github.com/groue/GRDB.swift/issues/643#issuecomment-560014034.
So far, GRDB is able to avoid crashes in applications. In extensions, nothing is quite settled yet.


Add unit tests for Utils.ts file

As part of https://github.com/SasanLabs/VulnerableApp-facade/issues/56 we have introduced a unit testing framework and now we are looking for adding more unit test for the project.
we are missing unit tests for https://github.com/SasanLabs/VulnerableApp-facade/blob/main/facade-app/src/Utilities/Utils.ts file.
Have a look at codecov report for finding what all is missing and cover that as part of this issue.
Codecov report: https://app.codecov.io/gh/SasanLabs/VulnerableApp-facade/tree/main/facade-app/src/Utilities

hi @preetkaran20 I would like to work on this issue. Can you assign it to me?

Hi @amrendranath ,
Sure

closing issue as already handled by @amrendranath in #92


Add conntrack_exporter to exporters

Uses libnetfilter_conntrack for direct access to the Linux kernel's connection table and exports network connections like this:
# HELP conntrack_open_connections How many open connections are there to the remote host?
# TYPE conntrack_open_connections gauge
conntrack_open_connections{host="<IP_ADDRESS>:3306"} 49

This exporter was created because we had trouble pinning down why sometimes connections from an app to redis were timing out, and when we used this exporter to graph the network connections on a redis instance we saw that instead of 3-4 stable connections our apps were creating far more and they were intermittent. Saved the day for us, and it wasn't possible to do this with node_exporter's tcpstat because the latter sums up all connections from everywhere and doesn't break them down by remote host.
This exporter also has almost zero overhead. We ran it in production and there was no noticeable increase in CPU or RAM usage, so we have plans to roll it out on all internal boxes.
Small writeup why this is complementary to node_exporter's tcpstat is here.
This doesn't go against Prometheus's guidelines against label explosion here because as mentioned in this FAQ, it's supposed to be run on an internal, non-public facing server. Therefore the number of metric:label pairs created should be very reasonable.

Thank you for sharing your exporter. This exporter is useful only in highly limited circumstances. On a machine running a web browser or talking out to SaaS services this is likely to have cardinality issues, with high levels of churn. For the use cases listed event logging such as via an iptables LOG target or profiling with tcpdump are likely a better fit.
I'm afraid we're not willing to list this on this page as the risk of unintentionally taking out Prometheus is greater than the potential benefits.

While I understand your decision @brian-brazil, I'd like to respectfully offer counter-points to your stated rationale. Having actually used this in production, and wishing I had something like it in previous projects, I think this fills a pretty important niche among existing exporters.

This exporter is useful only in highly limited circumstances.

Quoting from the exporter's README:
Uses:
* Operations: monitor when a critical link between your microservices is broken.
* Security: monitor which remote hosts your server is talking to.
* Debugging: correlate intermittently misbehaving code with strange connection patterns.

This is just meant to give a taste and obviously not exhaustive, and I think all of these are far from exotic. Apologies that my PR description was confusingingly only highlighting one rare use case, I was just sharing the reason why this exporter was created. Aside from the list above, this exporter can also do other cool things like allow Prometheus to alert when some specific external host attempts to connect, break down node egress by LAN vs WAN in a graph, etc. It's all up to the creativity of the user.

On a machine running a web browser or talking out to SaaS services this is likely to have cardinality issues, with high levels of churn.

Running a web server (exposed to the public Internet) is exactly the kind of usage we advise against in the readme, and we clearly explain why. Not sure what other safety net we can offer the end user without throwing out the baby with the bathwater :)
Talking out to SaaS services is actually a perfect use case for this exporter: usually the number of endpoints connected to is small and constant (one per each SaaS service the app uses) so we're okay regarding cardinality, and it's extremely valuable to have visibility into connection activity for those outside services.

For the use cases listed event logging such as via an iptables LOG target or profiling with tcpdump are likely a better fit.

I agree those would be alternative ways to debug intermittent connection issues, but it's hard to argue with the power of a graph even in this use case:

And of course, as I write above there's many use cases for this, including some that tcpdump and iptables won't help with.
I think the Prometheus team was correct in your decision to make the tcpstat collector disabled by default, and wrote next to it Warning: the current version has potential performance issues in high load situations. There shouldn't be anything in the default installation that allows a typical user to shoot themselves in the foot. However even though it only exports basic TCP state totals, tcpstat was still considered valuable enough to warrant it being included inside the official system exporter. I believe the same logic applies to this exporter, which shows even deeper visibility and is complementary to tcpstat: it's extremely useful in a large variety of use cases, but as a third party exporter it isn't part of a standard Prometheus installation (so it's "disabled by default") and can have a similar ominous warning next to it in the docs:

Warning: has potential performance issues in high load situations that can affect the stability of your Prometheus server; see README for details on how to use this exporter safely.

I can update my PR if you'll consider merging with such a change. If you'd also like me to add more info to the warning in the README as well, I don't mind doing that too, just let me know.
P.S. Thanks for an excellent monitoring platform! :)


Removed //! spare line at end of headers

Some of the headers had a //! at the end but most didn't.  They have all been removed in bitcoin/src/ to make the files consistent

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.191%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
19168

Relevant Lines:
23041

💛  - Coveralls

To get rid of the first patch you likely want to use git rebase -i master then kill the first patch (how you do that will depend on how you have git configured to work with your editor). Then when you only have a single patch on top of master branch you can run git push -f to force push the changes to github.

First patch removed.


Critical bug

No way to work with 2d. z axis based logic
If change quadtree setting on example scene from 64 0 64 to 64 64 0 it will eat all memory and crash unity with ur pc and possible ur mom nokia

Incredibly amusing; thank you for the report!

I also encountered this issue; while trying to solve it, I kept running into "stack overflow." Could you please tell me if there will be updates to this package in the future?

Sure, I can fix it, hopefully in the near future!


TSX  Default Generic Arrow Function Syntax Highlighting

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: Version: 1.78.2
OS Version: Darwin arm64 21.3.0

Related to https://github.com/microsoft/TypeScript/issues/47062
TSC has been updated to successfully parse the below TSX code snippet:
const Component = <T = undefined>(props: PropType<T>) =>
content

however this currently breaks the syntax highlighting in VSCode's typescriptreact parser.
The same can be said for the trailing comma added by prettier (3.0.3):
const Component = <T = undefined,>(props: PropType<T>) =>
content

Some short term work-arounds are available:
Using extends
const Component = <T extends unknown = undefined>(props: PropType<T>) =>
content

Using function syntax
const Component = function<T = undefined>(props: PropType<T>) {
  return
content

}

Despite these workarounds, the original code seems syntactically valid, and more concise, so I believe this to be a bug

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.81.1. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

Upgraded to 1.81.1; Issue persists


postgres schema creation fails on existing columns with custom schema types

Given a schema like the one below and a database that has had CREATE EXTENSION postgis run on it, the auto-migration works perfectly fine. The first time. Running it again on the same database with no changes to the schema results in sql/schema: invalid type "USER-DEFINED" for column "geo".
package schema

import (
	"github.com/facebookincubator/ent"
	"github.com/facebookincubator/ent/dialect"
	"github.com/facebookincubator/ent/schema/field"
)

// Geography holds the schema definition for the Geography entity.
type Geography struct {
	ent.Schema
}

// Fields of the Geography.
func (Geography) Fields() []ent.Field {
	return []ent.Field{
		field.Bytes("geo").
			SchemaType(map[string]string{
				dialect.Postgres: "geography",
			}),
	}
}

Edit: Fixed accidental usage of field.String() instead of field.Bytes(). Doesn't really make a difference for this bug, but this way the type matches what the field actually scans as.

I have the same issue. We added a wrapper to the migrate to run CREATE EXTENSION ltree before create but it gives us sql/schema: invalid type "USER-DEFINED" for column "path" from time to time. The creation of the schema looks like this:
type Team struct {
	ent.Schema
}

// Fields of the Team.
func (Team) Fields() []ent.Field {
	return []ent.Field{
		field.String("name").NotEmpty(),
		// path of the team from the root node.
		// Path needs to follow the ltree format <ident>.<ident>.<ident>
		field.String("path").SchemaType(map[string]string{
			dialect.Postgres: "ltree",
		}),
	}
}

@a8m Do you think this is a problem in my code? Is there anywhere else I can create an extension?

Hey @DeedleFake and @christie-zhao,
The schema migration doesn't work with "user defined types" at the moment - it's WIP and will be added in the near future.

@a8m Do you have any pointers as to why it doesn't work / what could be done to implement it? I'd be happy to implement it, just not quite sure where to begin 😅

This has been partially implemented by #994
Migrations will work so long as the custom types don't change. currently the is covertible logic will disallow conversion between user defined types.
This could be agumented by adding a a check if a dialect specific schema type is present but I am not sure how useful it will be


adding back cudnn 5.1 support

Tested: manually verified via docker.
Closes #1607 .
cc #1343 #1302 #1584

For the record, cudnn 5.1 will only be supported in a best-effort basis so one is strongly encouraged to update to cudnn 6 and above.

The bitonic sort thing is also needed for this one?
I'm adding a Jenkins job to test this.

The bitonic sort is basically needed for windows (which is a separate topic), but yeah, it helps clean up code clutter so I included it here :)

Build running: https://ci.pytorch.org/jenkins/job/caffe2-builds/job/py2-cuda8.0-cudnn5-ubuntu16.04-trigger-test/1/
If it passed and this PR is merged then I'll add it to the default set.

00:01:11.875 --     CUDA version        : 8.0
00:01:11.875 --     CuDNN version       : 5.1.10

@Yangqing There's a couple of test failures, see the test build

Triggered rebuild here
It is not part of the default set of builds yet.

Triggered rebuild here.

cudnn 5 passed here: https://ci.pytorch.org/jenkins/job/caffe2-builds/job/py2-cuda8.0-cudnn5-ubuntu16.04-trigger-test/


How to build binaries

Hi,
not sure if this is an issue or I am missing something, but regarding to the description, there should be a binary file in the downloaded .zip file. But it's missing the bin or the instructions how to build it.
I checked the build.ps1 file but it looks like it just tries to copy some file from not existent ./bin folder.
line 35: cp "bin/Release/pass-winmenu.exe" "$PKGDIR/pass-winmenu.exe"
Any hint how to get this running is appreciated.

Are you trying to install the program or do you want to set up your own development environment?
If you just want to install it, you can grab the release from here, the zip file contains the pass-winmenu.exe binary.

Thanks, I missed the release page and fiddled with the source files ;)


Electron Helper in processes is very confusing for Electron projects

Hi. I'm very confused with processes in Activity Monitor while working on Electron projects.
The "Electron Helper" are usually processes of unbuilt projects, and while debugging performance or related stuff, too confused with this because have both processes of VSCode and currently launched app (launched with prebuilt Electron).
After build, all processes are usually renamed to something like "My App Helper", but VSCode still has dummy names there - please, fix this.

VSCode Version: 1.2.1
OS Version: OS X 10.11.5

Steps to Reproduce:

Launch VSCode
Check Activity Monitor

There is no such problem on Windows:

We could rename the process I guess.

Apparently it's possible.

joaomoreno/gulp-atom-electron#36

I'd suggest using up-to-date electron-packager for builds instead of gulp-atom-electron which I see in the package.json.

00826ad

Cool:


Helm tasks runs kubectl cluster-info for which it may not have sufficient priviledges

Now the helm task will run kubectl cluster-info which will result in a forbidden warning because cluster-info may not be accessible for the given service connection, e.g. Error from server (Forbidden): services is forbidden: User "system:serviceaccount:xxx-sa:xxx cannot list resource "services" in API group "" in the namespace "kube-system"
Originally posted by @vaclavbroz in https://github.com/microsoft/azure-pipelines-tasks/pull/13189#issuecomment-673392164

https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/HelmDeployV0/src/helm.ts#L173

@vaclavbroz Is this affecting the status of the task (success/failure) or is it just a warning?

So far no, but that can be related to the fact that the Helm task apparently redirects stderr to stdout (did not verify). So it's a ticking bomb maybe.

We are sorry for the inconvenience, but this step was added in the task to collect some information which would be used to enhance overall experience for the customers while using these tasks. Thanks for pointing out this issue. I'll get in touch with the right people to decide what needs to be done about it.

Hi
This is happening to us too. It seems the complaining starts this morning (13th Aug). Was there some release?
From code it seems that kubectl cluster-info was executed previously on different place and maybe it never run for us before today.

@vaclav-dvorak This code was not reachable earlier due to a code bug, which was rectified recently. But looks like, its creating some new issues now. (Although that  warning is an expected behavior, because not all the service principals would have required permissions for running that command). If it's not affecting the state of the task, please continue to use the pipelines the way you were using previously while we decide whether any further changes have to be done.
cc: @ammohant @ds

@shigupt202 pipeline is working fine (well except for the warning). Just wanted to add my two cents :)

Discussed this with the team. Since the issue is not affecting any of the customer scenarios and is only appearing in the logs to inform that the service account is not having enough permissions to perform that operation, I think we can continue with it for now.
@vaclavbroz This is a kubectl command, and not a helm command. So, we don't need to worry about it getting written in stderr or stdout. Closing the issue for now. Feel free to reopen if it affects your pipelines in any way.

Discussed this with the team. Since the issue is not affecting any of the customer scenarios and is only appearing in the logs to inform that the service account is not having enough permissions to perform that operation, I think we can continue with it for now.
@vaclavbroz This is a kubectl command, and not a helm command. So, we don't need to worry about it getting written in stderr or stdout. Closing the issue for now. Feel free to reopen if it affects your pipelines in any way.

Fine for me.
Once the kubectl command run within the HelmDeploy task will somehow start putting the warning in stderr the pipeline task HelmDeploy will start being marked as failed. Given the fact you did not discover the issue with logging before releasing this pull-request I guess you do not have any automated integration tests covering this scenario. From that I understand there is a non-null probability you will have an incident later with the root cause described here.
I would suggest running a simple tests prior querying the specific namespace, e.g. with kubectl auth can-i and not executing the command if not authorised.


🚚 Move tests back / ⬆ Upgrade @hover/javascript

Move tests back to ./test to minimize diff against upstream repo
Move some test setup into beforeAll/beforeEach hooks
Upgrade @hover/javascript to 4.17.0 and fix some related linting issues

Codecov Report

Merging #88 into master will decrease coverage by 2.77%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #88      +/-   ##
==========================================
- Coverage   94.44%   91.66%   -2.78%
==========================================
  Files           2        2
  Lines          72       72
  Branches       11       11
==========================================
- Hits           68       66       -2
- Misses          3        6       +3
+ Partials        1        0       -1

Impacted Files
Coverage Δ

lib/index.ts
94.64% <0.00%> (-3.58%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dbac3d3...447cbf2. Read the comment docs.

:tada: This PR is included in version 2.5.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Migrate System.Runtime.WindowsRuntime

Still in draft because there are build failures.
Migrated from 3.1.
The System.Runtime.WindowsRuntime nuget package has this information:

Supports .NET Standard 2.0.
Throws PNSE in .NET 6.0.

When consuming this nuget package in a console app, then printing typeof(System.Runtime.InteropServices.WindowsRuntime.WindowsRuntimeBuffer).Assembly.CodeBase I confirmed that:
When targeting .NET Framework 4.6.2, the installed version is used: file:///C:/windows/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.WindowsRuntime/v4.0_<IP_ADDRESS>__b77a5c561934e089/System.Runtime.WindowsRuntime.dll
When targeting .NET 6.0, PNSE is thrown (pending figuring out if the package assembly is being used):
Unhandled exception. System.PlatformNotSupportedException: Operation is not supported on this platform. (0x80131539)

As discussed offline - we said that https://learn.microsoft.com/en-us/dotnet/core/compatibility/interop/5.0/built-in-support-for-winrt-removed means we don't need to bring this forward.   This package should be deprecated and folks should use CsWinRT.

Built in support for WinRT has been deprecated. For that reason, we have decided this package will be obsoleted and not migrated. https://learn.microsoft.com/en-us/dotnet/core/compatibility/interop/5.0/built-in-support-for-winrt-removed


Something is amiss in handling IASI (perhaps all hyperspectral IR)

As I introduce T-skin as a control variable, I am re-doing some of the exercise I've done before of looking at each of the main instruments increments and how they compare w/ GSI's. IASI (and CrIS-FSR) has always been in the suspicious list. I want to have a record of this here so we can work together to address whatever issue affects hyperspectral IR.
The temperature increment for microwave instruments such as ATMS (clear sky) and GMI (all sky) are quite consistent between GSI and JEDI. The corresponding T-skin increments are also quite consistent - with still a puzzle over Antartica (but not a show stopper - see e.g., https://github.com/JCSDA-internal/fv3-jedi/pull/1257.
For IASI, the story is quite alarmingly different. For starts, GSI does not fully converge its minimization when given up to 100 iteration to crank; but so be it. JEDI, on the other hand, seems to converge in about 40 something iterations, which is suspicious in itself. But the real issue is how different the increments from both systems look (perhaps not a surprise given the convergence differences - chicken and egg).
Interestingly enough, the increments on specific humidity between the two systems are quite reasonable - here is an example at 850 hPa - GSI (top) JEDI (bottom)

Unfortunately, this is what temperature looks like at the same level:

notice T increments are multiplied by a factor of 2 for plotting enhancement purposes; still -diff in increments are larger than increments themselves.
Now here is what T-skin increments looks like:

It looks as though the weight given to whatever induces the sensitivity of Tb to T-skin is overweighted.

Here is another set of diagnostics. These are now calculated as follows:
displayed-inc-diff = aave(abs(jedi(temperature),g)  - aave(abs(gsi(temperature),g)
for three instruments/platforms: GMI-GPM; ATMS-N20 and IASI-Metop-B, respectively:
GMI

ATMS-N20

IASI-MetopB

Another curious result comes from running both GSI and JEDI, for IASI-MetopB, with and without correlated errors.
Now here is what the measure above looks like for the two corresponding GSI experiments (no-Correlations-minus-Correlations):

and here is a similar results from JEDI:

In this case, I tend to think the behavior in JEDI seems to me more reasonable between having and not having correlated R. Something looks a miss (to me) in how GSI is handling these two flavors of Var.
These obviously don't show how different JEDI's results are from GSI's (see above), which indicate JEDI's to be the odd ball. But when it comes to on/off correlated R, the behavior of JEDI is more what I'd expect to see.

I am finding out that this whole IR thing might be due to differences in CRTM coeffs:

when I try running JEDI pointing to GSI coeffs, JEDI die complaining of the following:

CRTM_AerosolCoeff_Load(FAILURE) : Error reading AerosolCoeff file /discover/nobackup/projects/gmao/dadev/rtodling/JEDI/x49/j49rt00/fvInput/gsi/etc/fix_ncep20221018/REL-2.4.0-jcsda/CRTM_Coeffs/Little_Endian/AerosolCoeff.bin
AerosolCoeff_ValidRelease(INFORMATION) : A AerosolCoeff data update is needed. AerosolCoeff release is 3. Valid release is 4.

when I try running GSI with the CRTM coeffs used in JEDI, GSI fails w/ the following error:

CRTM_Init(FAILURE) : Error loading IRwaterCoeff data from /discover/nobackup/projects/gmao/advda/SwellStaticFiles/je
di/crtm_coefficients/2.4.1/Nalli.IRwater.EmisCoeff.bin; Process ID: 4
crtm_interface*init_crtm:  ERROR crtm_init error_status=           3
The funny thing is that months ago I had been able to run JEDI pointing to the GSI-CRTM coeffs ... obviously JEDI evolved and this is no longer possible - I know for a fact that one thing the changed in JEDI-CRTM are the Aerosol-related files ... I need to understand their relationship to "assimilating IR"! Maybe somebody knows?!

Thanks for Matt, I am not able to run GSI w/ exactly the same version of CRTM as that used in JEDI - pointing to the CRTM coeffs that JEDI uses. Here is figure of the increments for Tskin when using GSI as we normally have it in x0049 (top) and GSI running JEDI's version of CRTM (bottom):

minor differences  over ice that would/could be of importance weren't the issues above way more prominent.
And here is the difference in the profile of temperature averaged in the way I showed above for other cases gsi(def) - gsi(jedi(crtm)).

Some differences in the upper atmosphere, but nothing to be too alarmed as the differences we are finding between GSI and JEDI.

Just to give a general feel for the difference in handling antenna vs brightness temperature for some instruments / platforms in GSI, here is a profile fig (similar to calculations above) comparing two all obs exps w/ GSI when Ta is converted to Tb.

From this, it would seem the differences seen in the treatment of, say, ATMS-N20 in JEDI vs GSI would not all be explained by the Ta vs Tb story ... nonetheless, I will do two exps in GSI (ta vs tb) when using only ATMS-N20 for completion.

Thanks, Ricardo.  You may be right, that the signal for MW alone would likely be larger since those obs are affected most by the Ta/Tb diff.

I think it could be explained by Ta and Tb difference. In Ricardo’s figure, the difference is mostly shown above 5mb, those are higher levels affected by ATMS channel 15. ATMS channel 15 is not bias corrected, and all of the other channels are bias corrected so attena correction is partially taken care of in those channels but not in channel 15. That’s why we see differences above 5mb.
From: rtodling @.>
Date: Monday, September 30, 2024 at 2:47 PM
To: GEOS-ESM/swell @.>
Cc: Zhu, Yanqiu (GSFC-6101) @.>, Assign @.>
Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Something is amiss in handling IASI (perhaps all hyperspectral IR) (Issue #426)
CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.
Just to give a general feel for the difference in handling antenna vs brightness temperature for some instruments / platforms in GSI, here is a profile fig (similar to calculations above) comparing two all obs exps w/ GSI when Ta is converted to Tb.
tinc_allobs_tbXta.png (view on web)https://github.com/user-attachments/assets/2b9acff1-9ebb-439d-9c4f-440f6c302d84
From this, it would seem the differences seen in the treatment of, say, ATMS-N20 in JEDI vs GSI would not all be explained by the Ta vs Tb story ... nonetheless, I will do two exps in GSI (ta vs tb) when using only ATMS-N20 for completion.
—
Reply to this email directly, view it on GitHubhttps://github.com/GEOS-ESM/swell/issues/426#issuecomment-2383921223, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARUMEBX25MF2QJFRPVRBKLTZZGMDZAVCNFSM6AAAAABOG6HI4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBTHEZDCMRSGM.
You are receiving this because you were assigned.Message ID: @.***>

I think it could be explained by Ta and Tb difference. In Ricardo’s figure, the difference is mostly shown above 5mb, those are higher levels affected by ATMS channel 15. ATMS channel 15 is not bias corrected, and all of the other channels are bias corrected so attena correction is partially taken care of in those channels but not in channel 15. That’s why we see differences above 5mb. From: rtodling @.> Date: Monday, September 30, 2024 at 2:47 PM To: GEOS-ESM/swell @.> Cc: Zhu, Yanqiu (GSFC-6101) @.>, Assign @.> Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Something is amiss in handling IASI (perhaps all hyperspectral IR) (Issue #426) CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Just to give a general feel for the difference in handling antenna vs brightness temperature for some instruments / platforms in GSI, here is a profile fig (similar to calculations above) comparing two all obs exps w/ GSI when Ta is converted to Tb. tinc_allobs_tbXta.png (view on web)https://github.com/user-attachments/assets/2b9acff1-9ebb-439d-9c4f-440f6c302d84 From this, it would seem the differences seen in the treatment of, say, ATMS-N20 in JEDI vs GSI would not all be explained by the Ta vs Tb story ... nonetheless, I will do two exps in GSI (ta vs tb) when using only ATMS-N20 for completion. — Reply to this email directly, view it on GitHub<#426 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARUMEBX25MF2QJFRPVRBKLTZZGMDZAVCNFSM6AAAAABOG6HI4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBTHEZDCMRSGM. You are receiving this because you were assigned.Message ID: @.***>

@gmao-yzhu I understand the diffs I see I due to Ta vs Tb  ...

Here is the case when only ATMS-N20 is used. It seems the differences are still of the order of when all other observations are being assimilated.

Just want to check – did you use bufr_ncep_atms_ta_remap.yaml or bufr_ncep_atms_remap.yaml?
From: rtodling @.>
Date: Wednesday, October 2, 2024 at 12:41 PM
To: GEOS-ESM/swell @.>
Cc: Zhu, Yanqiu (GSFC-6101) @.>, Mention @.>
Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Something is amiss in handling IASI (perhaps all hyperspectral IR) (Issue #426)
CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.
Here is the case when only ATMS-N20 is used. It seems the differences are still of the order of when all other observations are being assimilated.
Screenshot.2024-10-02.at.12.38.47.PM.png (view on web)https://github.com/user-attachments/assets/0c945213-ec33-4d15-9d64-d2722a2f1016
—
Reply to this email directly, view it on GitHubhttps://github.com/GEOS-ESM/swell/issues/426#issuecomment-2389126883, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARUMEBSDXQEP4I2EUNJMNILZZQO3XAVCNFSM6AAAAABOG6HI4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBZGEZDMOBYGM.
You are receiving this because you were mentioned.Message ID: @.***>

Just want to check – did you use bufr_ncep_atms_ta_remap.yaml or bufr_ncep_atms_remap.yaml?

@gmao-yzhu Not quite sure I understand your question. The tests I have done are all GSI tests; there are no yaml's involved in these!
Also: can you check your replies to these exchanges ... something odd is happening where it seems that every reply from you have some email thing attached to it that makes the msgs nearly unreadable.

I am getting a better picture of what the true issue behind these bad T-skin increments might be.
I think I have nailed it to the https://github.com/JCSDA-internal/ufo/blob/2149dd956ecdf4c8b1924c0d0682a34eb8b1d94d/src/ufo/filters/obsfunctions/ObsErrorFactorSurfJacobianRad.cc#L4
After trying multiple things (including w/ other ObsFunctions), not only did I seem to have nailed the issue to this, as I believe the problem to be in entries in this line:
https://github.com/JCSDA-internal/ufo/blob/2149dd956ecdf4c8b1924c0d0682a34eb8b1d94d/src/ufo/filters/obsfunctions/ObsErrorFactorSurfJacobianRad.cc#L178
and the following line.
As a hack test I multiplied the term quantity in line 180 by factor in the dozens, the increments of T skin start looking more like what I get in GSI.

I understand now why the bias term knob about would not work - it would not be consistent w/ GSI to have bias terms in the IR in the same in MW. But still don't see what else could be wrong.

When adjusting the sensitivity coefficients (for area types) my magnifying them by certain factor this is what the comparison of the vertical increments of jedi vs gsi look like - compare w/ similar plot above.

notice there is a small difference in the x-axis as compared to the earlier figure. This adjustment does make things look better ... and it could benefit from closer attention - but that's the wrong thing to do.
just for the record these are the values used:
obserr_dtempf:
- 10.0
- 30.0
- 40.0
- 20.0
- 40.0
the parameters for the emissivities are left unchanged!

BTW: I am not find in JEDI the "massaging" of the jacobian wrt ozone for wave numbers between 996 and 1170. That is, GSI (in setuprad) zeroes out these jacobians to remove the ozone influence on Tb from this channels. These sensitivities are associated w/ the stratosphere and lower troposphere. We should consider adding this to JED at some point ... however, just for the record, I show here a case when GSI's zeroing out of these Jac is removed - that is, the sensitivities are present. The black covers compares JEDI w/ the modified GSI and the green curve compares the modified GSI w/ the original GSI. The differences resulting from the wiping out of the jacobinans are much smaller than the differences we see between any GSI variation and JEDI.  These results are for AIRS.

Adding to the discussion, I want to revisit the impression given above that results for GMI are considerably better than for other instruments. I don't think this is the case. The thing w/ GMI is that its increments are small to begin with so the differences are "differences" look small. Here is a zoomed version of the GMI vertical differences show about (black), and on top of that adding the JEDI (green) and GSI (yellow) increments - show as global_ave(abs(tinc))

BTW: it is always the case that JEDI's "increment" is alway larger than GSI's ... I am starting to wonder about VarBC ...

It’s not just radiance assimilation that results in large increments in JEDI. The assimilation of observations from rawinsondes behaves similarly when comparing increments from the two systems. Therefore, this is not related to VarBC at all.

Adding to the discussion, I want to revisit the impression given above that results for GMI are considerably better than for other instruments. I don't think this is the case. The thing w/ GMI is that its increments are small to begin with so the differences are "differences" look small. Here is a zoomed version of the GMI vertical differences show about (black), and on top of that adding the JEDI (green) and GSI (yellow) increments - show as global_ave(abs(tinc))

BTW: it is always the case that JEDI's "increment" is alway larger than GSI's ...

Some time has passed since I last updated this and showed any results here.
I want to show the group - and remind myself - where things stand on this T-skin issue for the hyperspectral IR. I am now using a version of JEDI from 19 Nov 2024 - this is combined with a updated version of SWELL (still at works) that updates to the Oct 2024 variable change sprint (issue #457).
Here is a figure comparing GSI vs JEDI Tskin from IASI on Metop-B.

It seems to me what we are dealing w/ here is a magnitude issue; the shape of the increment is certain correct; and we get JEDI increment over all surfaces just as we do from GSI.

Just for the sake of argument I am show here the same picture as above, but now scaling the Tskin increment from JEDI by 0.18 - ie multiplying by a fact to reduce the increment on Tskin:

This is just to make the argument that all this is is some sort of scaling in JEDI.
Something tells me this is associated w/ the nearSSTRetCheck term ... and I know I "played" with this term before ... but I want to do it again.

I agree Ricardo, other than scaling it would be hard to choose between the two...so that's a real clue and perhaps a simpler issue to resolve than if something more insidious was wrong in the UFO, etc.
I realize you scaled this arbitrarily just to make the point, but I'm still curious how you arrived at the value 0.18 to get such a good match.  Did you scale by something like the ratio of the global RMS values between the GSI and (unscaled) JEDI increment?

Sounds good.  Btw, I had a long tag-up with Tom yesterday and I mentioned that we really have to focus on making JEDI more efficient, in terms of memory and speed, now that we're trying to run at scale.  He completely agrees, and this was originally a priority of the AOP this year.  Unfortunately, NOAA pulled the money designated to pay for this as part of the GDAS testbed app, which has now been removed from NOAA's plans for the year. So we'll all need to figure out a way to get JCSDA covered to do this work.

I managed to run successfully now ... but JEDI - as usual - converges earlier than GSI (that rarely ever converges when handling radiance.  JEDI goes up to 58 iterations .. and so, the increment doesn't change in any significant way - no point in showing (visually identical). So the factor 0.18 remains reasonable - again the factor is just for argument sake.

I am starting to suspect the problem is in this function: CloudDetectMinResidualIR

Clearly something is amiss in
ObsErrorFactorSurfJacobianRad.cc
I can multiple this line:
by a factor and make the increment look rather rather acceptable (not shown to avoid confusion).

here is the result of the increment from the change above:

Closing in.  Nice.

The adjusted values I proposed above turn out to be a little too large for CrIS-FSR - for this, I believe a factor of 5 from the original values should suffices. Here is a figure of difference of increment w/ GSI for two cases in JEDI: top using default values (which are in principle common to GSI), and bottom, is diff w/ JEDI using factor of 5 readjusted values.

Ricardo, the new factor clearly provides very close values for the increments for JEDI and GSI, which is great.  But do I understand correctly that you view this as a "cosmetic" fix and that there's still an actual problem with how the emissivity values are communicated between UFO and CRTM?  Since I don't have access to JCSDA-internal, I'm not clear about Ben's and your thinking on this. Thanks.