syscall/js: performance considerations

I was porting some frontend Go code to be compiled to WebAssembly instead of GopherJS, and noticed the performance was noticeably reduced. The Go code in question makes a lot of DOM manipulation calls and queries, so I decided to benchmark the performance of making calls from WebAssembly to the JavaScript APIs via syscall/js.
I found it's approximately 10x slower than native JavaScript.
Results of running a benchmark in Chrome 75.0.3770.80 on macOS 10.14.5:
  131.212518 ms/op - WebAssembly via syscall/js
   61.850000 ms/op - GopherJS via syscall/js
   12.040000 ms/op - GopherJS via github.com/gopherjs/gopherjs/js
   11.320000 ms/op - native JavaScript

Here's the benchmark code I used, written to be self-contained:
Source Code
main.go
package main

import (
	"fmt"
	"runtime"
	"syscall/js"
	"testing"
	"time"

	"honnef.co/go/js/dom/v2"
)

var document = dom.GetWindow().Document().(dom.HTMLDocument)

func main() {
	loaded := make(chan struct{})
	switch readyState := document.ReadyState(); readyState {
	case "loading":
		document.AddEventListener("DOMContentLoaded", false, func(dom.Event) { close(loaded) })
	case "interactive", "complete":
		close(loaded)
	default:
		panic(fmt.Errorf("internal error: unexpected document.ReadyState value: %v", readyState))
	}
	<-loaded

	for i := 0; i < 10000; i++ {
		div := document.CreateElement("div")
		div.SetInnerHTML(fmt.Sprintf("foo bar baz %d", i))
		document.Body().AppendChild(div)
	}

	time.Sleep(time.Second)

	runBench(BenchmarkGoSyscallJS, WasmOrGJS+" via syscall/js")
	if runtime.GOARCH == "js" { // GopherJS-only benchmark.
		runBench(BenchmarkGoGopherJS, "GopherJS via github.com/gopherjs/gopherjs/js")
	}
	runBench(BenchmarkNativeJavaScript, "native JavaScript")

	document.Body().Style().SetProperty("background-color", "lightgreen", "")
}

func runBench(f func(*testing.B), desc string) {
	r := testing.Benchmark(f)
	msPerOp := float64(r.T) * 1e-6 / float64(r.N)
	fmt.Printf("%f ms/op - %s\n", msPerOp, desc)
}

func BenchmarkGoSyscallJS(b *testing.B) {
	var total float64
	for i := 0; i < b.N; i++ {
		total = 0
		divs := js.Global().Get("document").Call("getElementsByTagName", "div")
		for j := 0; j < divs.Length(); j++ {
			total += divs.Index(j).Call("getBoundingClientRect").Get("top").Float()
		}
	}
	_ = total
}

func BenchmarkNativeJavaScript(b *testing.B) {
	js.Global().Set("NativeJavaScript", js.Global().Call("eval", nativeJavaScript))
	b.ResetTimer()
	js.Global().Get("NativeJavaScript").Invoke(b.N)
}

const nativeJavaScript = `(function(N) {
	var i, j, total;
	for (i = 0; i < N; i++) {
		total = 0;
		var divs = document.getElementsByTagName("div");
		for (j = 0; j < divs.length; j++) {
			total += divs[j].getBoundingClientRect().top;
		}
	}
	var _ = total;
})`

wasm.go
// +build wasm

package main

import "testing"

const WasmOrGJS = "WebAssembly"

func BenchmarkGoGopherJS(b *testing.B) {}

gopherjs.go
// +build !wasm

package main

import (
	"testing"

	"github.com/gopherjs/gopherjs/js"
)

const WasmOrGJS = "GopherJS"

func BenchmarkGoGopherJS(b *testing.B) {
	var total float64
	for i := 0; i < b.N; i++ {
		total = 0
		divs := js.Global.Get("document").Call("getElementsByTagName", "div")
		for j := 0; j < divs.Length(); j++ {
			total += divs.Index(j).Call("getBoundingClientRect").Get("top").Float()
		}
	}
	_ = total
}

I know syscall/js is documented as "Its current scope is only to allow tests to run, but not yet to provide a comprehensive API for users", but I wanted to open this issue to discuss the future. Performance is important for Go applications that need to make a lot of calls into the JavaScript world.
What is the current state of syscall/js performance, and are there known opportunities to improve it?
/cc @neelance @cherrymui @hajimehoshi

It would be also good to benchmark with Firefox and see the results.
IIUC, you are just benchmarking DOM manipulation. And since DOM manipulation anyways happens outside wasm, it is just about the price of context jump from wasm land to browser land and back. In that case, I wonder if it is even within the control of syscall/js and not the underlying wasm engine.
Would be also good to benchmark equivalent code using Rust and C and compare the benchmarks. I think that may be a better apples-apples comparison just to compare syscall/js performance with other languages.

As @agnivade said, probably worth trying Firefox. V8 is known to have some performance problems with the Wasm code generated by the Go compiler.

It would be also good to benchmark with Firefox and see the results.

Agreed. I'll do this later and share results.

IIUC, you are just benchmarking DOM manipulation. And since DOM manipulation anyways happens outside wasm, it is just about the price of context jump from wasm land to browser land and back. In that case, I wonder if it is even within the control of syscall/js and not the underlying wasm engine.

Yes. When I said syscall/js, I meant the entire performance cost of jumping from Wasm to browser and back. It's what the user sees when they use the API to interact with the JavaScript world.

Would be also good to benchmark equivalent code using Rust and C and compare the benchmarks. I think that may be a better apples-apples comparison just to compare syscall/js performance with other languages.

Agreed, that would be good and more representative of the actual WebAssembly <-> JS call overhead. Doing that would give us more information. I won't have a chance to do this, but if someone else can, it'd be helpful.

Perhaps it's not worth doing anything substantial here before something like WASI is standardized. @neelance even did a WIP implementation at https://github.com/golang/go/issues/31105.

I've tried the benchmark again with recent development versions of 3 browsers:
Chrome Canary
Version 77.0.3824.0 (Official Build) canary (64-bit)

    114.154496 s/op - WebAssembly via syscall/js
     63.350000 s/op - GopherJS via syscall/js
     11.740000 s/op - GopherJS via github.com/gopherjs/gopherjs/js
     11.360000 s/op - native JavaScript

Firefox Nightly
69.0a1 (2019-06-13) (64-bit)

     94.150003 s/op - WebAssembly via syscall/js
     85.300000 s/op - GopherJS via syscall/js
      7.695000 s/op - GopherJS via github.com/gopherjs/gopherjs/js
      7.405000 s/op - native JavaScript

Safari Technology Preview
Release 85 (Safari 13.0, WebKit 146<IP_ADDRESS>)

     57.249996 s/op - WebAssembly via syscall/js
     42.866666 s/op - GopherJS via syscall/js
      5.536666 s/op - GopherJS via github.com/gopherjs/gopherjs/js
      5.073333 s/op - native JavaScript

The results are pretty consistent across the 3 browsers in that doing lots of DOM queries via WebAssembly was about 10x slower than with pure JavaScript.

Could you share the code to take the benchmark to output the values [s/op]?

Thanks for the tests @dmitshur. I would have thought that after https://hacks.mozilla.org/2018/10/calls-between-javascript-and-webassembly-are-finally-fast-🎉/, the DOM access overhead would have reduced in Firefox. And interesting that Safari is much faster for DOM access than Firefox.
The tests with Rust/C should give us a better idea on what exactly can be improved from Go side. If anybody can post results for that, that'll be great.

@hajimehoshi Sure. I've updated the source code in the original post.

@martisch suggested that I add a "real-world" example that demonstrates the performance hit of webassembly compared to running natively. A good example is the "gophers" demo from Gio (gioui.org). With modules enabled and using Go 1.13 (tip), you can build and run the demo with two commands:
    $ export GO111MODULE=on
    $ go run gioui.org/cmd/gio -target js gioui.org/apps/gophers # for building gophers
    $ go run github.com/shurcooL/goexec 'http.ListenAndServe(":8080", http.FileServer(http.Dir("gophers")))' # for serving gophers on localhost:8080

Then, open a browser and open http://localhost:8080. Press Ctrl-P to enable continuous redrawing and a frame timer in the top right corner. The target frame time is ~16.7ms (60 Hz), but on my macbook pro it almost never hit the target.
Running the example natively,
     $ go run gioui.org/apps/gophers

and pressing Ctrl-P (Cmd-P on macOS) easily hit the 60 Hz target.
In both Chrome and Firefox the builtin profiler is a great way to see what takes up the time. I've attached a screenshot of a single frame from Chrome's "Performance" tab. The frame time is 24ms.

Unfortunately, the function names are all mangled ("wasm-function[]") which makes it much harder to discern which functions take up time.
CC @cherrymui who recently optimized wasm.

Thanks @eliasnaur. I have sent CL 183457 which should alleviate the DOM overhead to some extent. Would you be able to try that and check if it helps at all ? Just a note that the CL only optimizes DOM overhead, so if your app is heavy on computations in the wasm land itself, it might not help very much.
Regarding profiles, yes the Chrome profiler is a great tool. The wasm-function[] is indeed a bother (see bug I filed with the Chrome team). Until then, may I take this chance to suggest you to use wasmbrowsertest ? It was mentioned in @johanbrandhorst's Gophercon talk. Using this you can natively take cpu profiles for wasm just as you would do for amd64. It automagically converts wasm-function to their appropriate names and you can directly analyze the profiles using go tool pprof :slightly_smiling_face: That should give you better insight into your app regarding what's going on and you can see if there is a possibility to optimize hot functions.

Thanks @agnivade, wasmbrowsertest is definitely useful for running benchmark and standalone tests on wasm. However, the full drawing and rendering to a window doesn't lend itself to that model yet.
Fortunately, I figured out how to bring back function names: the gioui.org/cmd/gio command passed -ldflags=-w -s which as a side effect stripped the function names from browser debuggers. I've removed the flags which didn't save much space anyway.
Finally I updated my comment to add the -stats flag that enable profiling without Ctrl-P.

Re: function names, it is a cold cache phenomenon as far as I understood. For the first time, it comes up as wasm-function, and then on all consecutive reloads, the names show up. Although, it is hard to reproduce. See the bug I filed.
Anyways, I see some syscall/js.ValueCall in the profile. So my CL shouldTM be able to help. Feel free to give it a try whenever you have a chance.

I tested with your CL 183457 which seems to help: the frame times are lower and more consistent. This is an example for a 17ms frame (the above profile had frame times above 20ms):

However, the CPU usage still seems too high. According to the profile, almost 10ms of CPU time is spent building the vector shape for the frame timer in the top right corner. The text layout code is definitely CPU heavy and unoptimized, but 10ms seems excessive.
To verify the profile, I cut out the rendering of the statistics label and redid the profile:

Firefox also misses the frame target:

It looks like CPU heavy code is faster in Firefox, whereas DOM calls are slower. Perhaps DOM calls are only slower because Firefox' WebGL implementation is slower.
In summary, it looks like the demo is CPU bound, leading to the claim that Go generates inefficient webassembly.
I'll work on preparing a benchmark that can run in wasmbrowsertest and that skips all rendering/DOM calls.

Great stuff ! I think we are getting somewhere. Yes, the wasm code generation can use some love. I have a couple of CLs which apply some rewrite optimizations which were there in amd64 but absent in wasm, which should go in when the tree opens.
But it would be great if you can prepare a standalone benchmark. That would allow us to compare the generated code with amd64 and see if there are some obvious places for improvement.

I split the UI update from its rendering and added a benchmark. To see the difference, I ran:
    $ go test -bench . -count 8 -run=NONE -cpu 1 gioui.org/apps/gophers > native.bench
    $ GOOS=js GOARCH=wasm go test -exec ~/go/bin/wasmbrowsertest -bench . -count 8 gioui.org/apps/gophers > wasm.bench
    $ benchstat native.bench wasm.bench
    name  old time/op  new time/op   delta
    UI    14.9µs ± 1%  216.5µs ±22%  +1354.01%  (p=0.000 n=7+8)

So more than 10 times slower on wasm compared to native code, at least on my 2014 MBP.

I investigated the profiles and started looking at the GOSSAFUNC output of some hot functions. The amd64 code showed lots of (MUL/DIV)SS. However, the wasm code showed something interesting, there were lots of F32DemoteF64 and F64PromoteF32 in the generated code. For example:
v395 00419 (14) F32Load	"".ctrl1+32(SP)
v395 00420 (14) F64PromoteF32
v395 00421 (14) F64Sub
v395 00422 (14) F64Const	$(0.5)

And in fact, several times, code like this was generated -
v403 00474 (213) I32WrapI64
v403 00475 (213) F32Load	""..autotmp_318-64(SP)
v403 00476 (213) F64PromoteF32
v403 00477 (213) F32DemoteF64
v403 00478 (213) F32Store	$0

This means all 32 bit FP values are being promoted to 64bit, then worked on, and then again demoted to 32 bit before being written back to memory.
A quick look into WasmOps.go revealed that 32 bit FP instructions were missing. And then I understood why. It is because all the FP registers (F0-F15) are treated as 64 bit registers.
Now here is where my speculation begins. Since Go SSA works with only registers, these virtual registers were created to work with SSA. But in the generated code, all references to registers are rewritten to local.(get|set|tee). So theoretically it should be possible to construct another set of 32bit registers and add 32 bit FP instructions which just deal with them, and avoid this 32-64 jump.
@neelance / @cherrymui - Is this analysis correct ? If so, how would you recommend to extend the F0-F15 register set to include 32 bit registers too. I have a local CL where I have already added the 32 bit instructions. Now I just need to fix these local.(get|set|tee) to work with 32 bit values.

The F64PromoteF32+F32DemoteF64 combination should only happen if rounding to 32 bits is actually necessary. In many cases the Go spec allows to use 64 bit precision for float32 values.
Yes, it is possible to add registers for 32 bit floats, but I'm not sure how much this would affect performance, because I guess that CPUs are not faster on 32 bit floats than on 64 bit floats (might be wrong).

The F64PromoteF32+F32DemoteF64 combination should only happen if rounding to 32 bits is actually necessary.

I think you are referring to this
case ssa.OpWasmLoweredRound32F:
		getValue64(s, v.Args[0])
		s.Prog(wasm.AF32DemoteF64)
		s.Prog(wasm.AF64PromoteF32)

I actually found another code path in case ssa.OpWasmF32Store where getValue64 actually generates a F64PromoteF32 and then because of if v.Op == ssa.OpWasmF32Store {, another AF32DemoteF64 gets added. I did not look much deeper into it though.

Yes, it is possible to add registers for 32 bit floats, but I'm not sure how much this would affect performance,

Sure, if there is no perf boost, then there is no use. But I would like to try and check the benchmarks. What is the right way to add 32 bit registers ? Just add F16-F32 ? Or is there another way ?

Is the F64PromoteF32 the one emitted by case ssa.OpLoadReg: of ssaGenValueOnStack? If yes, then this is indeed something we could optimize.

What is the right way to add 32 bit registers ? Just add F16-F32 ? Or is there another way ?

This is not easy to describe in a few words...

@agnivade does your CL contain fixes to 32-bit integral instructions? it also amazed me to see such 32-64-32 int/fp convertions.

I have not sent any CL yet. And no, I have not looked into 32bit integral instructions.

In my experience, a large portion of the time spent in trivial js.Value.*() calls is decoding the UTF-8 encoded strings, such as the member name.
As of May 2021, in cases like function calls, it might be significantly faster to cache the result of this.Get("functionName").Call("bind", this) and then use Invoke instead of Call.


west.yml: Pull in fix for build issues of flexcan driver on imx-rt

Pull in ASSERT/NDEBUG fix similar to what exists on Kinetis for imx-rt.
Signed-off-by: Kumar Gala<EMAIL_ADDRESS>

Fixes nightly failures reported:
https://buildkite.com/zephyr/zephyr-daily/builds/132


Make a couple layout changes on org admin view

create an "Organization Information" sub-heading under "Edit Organizatiaon [Org Name]" heading
move Privacy Policy URL and Privay Policy fields out of "Primary Contact" sub-heading and into newly created "Organization Information" sub-heading

@mgmccarthy did you mean to include the gitignore changes?

@tonysurma, it's from some files I parked in the AllReady directory locally, and Git picked them up. I did not want to commit them, so I added them to GitIgnore (b/c I'd like to keep the files where they are)
I can move the local files and revert the gitignore if you want.


Possible to use conditions to change Task behaviour rather than making the Task Run/Skip ??

I have a situation where I have an Android app setup on VSTS that is running a series of Gradle Tasks, and I want to pass it different commands based on a variable - in this case, branch name.
My application structure has several Product Flavours / Build Variants, and the assembleRelease command takes a llloooonnggg time - so much that the Task is often timing out as it goes over the 60 minute limit! (I know I can update the timeout but that's a bad option...)
Now, I have various of my Flavours/Variants that exist on certain Long-Term-Support branches, so I am thinking about trying to use the condition - using startsWith System.PullRequest.TargetBranch to determine what branch the Pull Request is aimed at, and passing a smaller subset of assembleXYZ commands in the tasks: ... argument.
From what I can tell, conditions normally makes a task/step/job/stage run or skip... so am not sure if my line of thinking is going to work.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3f151218-9a11-0078-e038-f96198a76143
Version Independent ID: 09c4d032-62f3-d97c-79d7-6fbfd89910e9
Content: Conditions - Azure Pipelines
Content Source: docs/pipelines/process/conditions.md
Product: devops
Technology: devops-cicd
GitHub Login: @andyjlewis
Microsoft Alias: alewis

Updated description to be more clear.

Thanks for your question. It looks like you're working through an issue with your scenario or implementation, rather than an issue with the documentation. Here are a few options where you might consider asking your question:

Azure DevOps on Stack Overflow
Azure DevOps Support Bot


Cannot configure the iOS platform

What I did:

npm install react-native-vector-icons --save

cd ios
pod init
vim Podfile
pod update

The result:
Update all pods
Updating local specs repositories
Analyzing dependencies
Fetching podspec for RNVectorIcons from node_modules/react-native-vector-icons
[!] No podspec found for RNVectorIcons in node_modules/react-native-vector-icons

The path is relative, so since your podfile seems to be in the ios folder your path should be ../node_modules/react-...


Deduplicate findings, closed findings getting deleted

Be informative
Using the dedupe function a closed finding, which is the original one, gets closed even though I would expect that it is preserved.
Steps to reproduce
Steps to reproduce the behavior:

Go to system settings and activate deduplication and the deletion of those when there are more than 2
Use the following json as an example and import the scan results as type "Jfrog Xray Unified Scan":
{ "total_rows" : 4,
"rows" : [
{
"cves": [
{
"cve": "CVE-2021-44228",
"cvss_v3_score": 10
}
],
"cvss3_max_score": 10,
"summary": "Apache Log4j2 \u003c=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (\u003e2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to ?true? or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".",
"severity": "Critical",
"severity_source": "CVSS V3 from NVD",
"vulnerable_component": "gav://org.apache.logging.log4j:log4j-core:2.11.1",
"impacted_artifact": "generic://generic/generic-42.jar",
"impact_path": [
"generic://generic/generic-42.jar"
],
"path": "tools-generic",
"fixed_versions": [
"2.15.0"
],
"published": "2021-12-10T13:52:51+01:00",
"artifact_scan_time": "2021-08-05T14:27:03+02:00",
"issue_id": "XRAY-191654",
"package_type": "maven",
"provider": "JFrog",
"description": "Apache Log4j2 \u003c=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (\u003e2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to ?true? or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".",
"references": [
"https://logging.apache.org/log4j/2.x/security.html"
],
"project_keys": []
}
,{
"cves": [
{
"cve": "CVE-2021-44228",
"cvss_v3_score": 10
}
],
"cvss3_max_score": 10,
"summary": "Apache Log4j2 \u003c=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (\u003e2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to ?true? or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".",
"severity": "Critical",
"severity_source": "CVSS V3 from NVD",
"vulnerable_component": "gav://org.apache.logging.log4j:log4j-core:2.11.1",
"impacted_artifact": "generic://generic/generic-42.jar",
"impact_path": [
"generic://generic/generic-42.jar"
],
"path": "tools-generic",
"fixed_versions": [
"2.15.0"
],
"published": "2021-12-10T13:52:51+01:00",
"artifact_scan_time": "2021-08-05T14:27:03+02:00",
"issue_id": "XRAY-191654",
"package_type": "maven",
"provider": "JFrog",
"description": "Apache Log4j2 \u003c=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (\u003e2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to ?true? or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".",
"references": [
"https://logging.apache.org/log4j/2.x/security.html"
],
"project_keys": []
}
,{
"cves": [
{
"cve": "CVE-2021-44228",
"cvss_v3_score": 10
}
],
"cvss3_max_score": 10,
"summary": "Apache Log4j2 \u003c=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (\u003e2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to ?true? or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".",
"severity": "Critical",
"severity_source": "CVSS V3 from NVD",
"vulnerable_component": "gav://org.apache.logging.log4j:log4j-core:2.11.1",
"impacted_artifact": "generic://generic/generic-42.jar",
"impact_path": [
"generic://generic/generic-42.jar"
],
"path": "tools-generic",
"fixed_versions": [
"2.15.0"
],
"published": "2021-12-10T13:52:51+01:00",
"artifact_scan_time": "2021-09-06T13:55:02+02:00",
"issue_id": "XRAY-191654",
"package_type": "maven",
"provider": "JFrog",
"description": "Apache Log4j2 \u003c=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (\u003e2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to ?true? or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".",
"references": [
"https://logging.apache.org/log4j/2.x/security.html"
],
"project_keys": []
}
,{
"cves": [
{
"cve": "CVE-2021-44228",
"cvss_v3_score": 10
}
],
"cvss3_max_score": 10,
"summary": "Apache Log4j2 \u003c=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (\u003e2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to ?true? or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".",
"severity": "Critical",
"severity_source": "CVSS V3 from NVD",
"vulnerable_component": "gav://org.apache.logging.log4j:log4j-core:2.11.1",
"impacted_artifact": "generic://generic/generic-42.jar",
"impact_path": [
"generic://generic/generic-42.jar"
],
"path": "tools-generic",
"fixed_versions": [
"2.15.0"
],
"published": "2021-12-10T13:52:51+01:00",
"artifact_scan_time": "2021-09-06T13:55:02+02:00",
"issue_id": "XRAY-191654",
"package_type": "maven",
"provider": "JFrog",
"description": "Apache Log4j2 \u003c=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (\u003e2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to ?true? or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".",
"references": [
"https://logging.apache.org/log4j/2.x/security.html"
],
"project_keys": []
}
]}
Now wait till the dedupe algorithm marks the original one and finds the duplicates

Go to the finding, which is marked as original and choose close finding and add some notes, e.g. "false positive"

After that the vulnerability should have the state "mitigated" and it should be visible that a note is attached to it

Repeat step 2 and wait till the dedupe was done. The already mitigated finding was deleted now

Expected behavior
As the close findings is a feature that is useful for false positives, it should have the same behavior as for risk acceptance (using risk acceptance for closing the finding will not produce this bug) that the original finding is preserved.
Deployment method (select with an X)

[ ] Docker Compose
[ X ] Kubernetes
[ ] GoDojo

@devGregA could you reproduce this bug?

Does nobody else have this problem?

@devGregA I still think that this is a huge issue and should be checked if reproduceable by others.


Large requests are getting truncated

I am running some service configured like so:
BlazeBuilder.bindHttp( port, "<IP_ADDRESS>" ).withLengthLimits(
      maxRequestLineLen = Integer.MAX_VALUE,
      maxHeadersLen     = Integer.MAX_VALUE
    )

However, when I POST to the service with a large piece of json (~65K bytes), it truncates the json cutting off a large portion of the data. As a result, when I try to parse the json, I get a parse error. Is there any other configuration I can play with to enable my service to handle large json bodies or is there some enforced cap on the size of the json data you can send to an http4s service?

Note: from the gitter chat, this is on 0.15, which is EOL.
If you can provide an example, we can attempt to help you debug.

Please reopen if this is still an issue.


i18n

Is it possible to introduce i18n ourselves while using this library? If so, is it possible to have some pointers on how to achieve it? I would gladly update your documentation afterward with an how-to. Thank you

Hello @giant995 ,
Can you give me a use case ?
The package itself does not have any strings in its own

I'm coding a platform for a client and it's in french.So I wanted to know if we could provide our own translations for the labels of the filter, for example. I hope I'm clear enough in my request.

Just add translations to your django locale file and compile messages:
# translations for Django Slick Reporting
msgid "Week"
msgstr "Woche"


Removed unnecessary inspect call from sql adapter

Hi guys !
Not sure why or how I let this happen, but I discovered today that I forgot a inspect call on sql adapter :hankey:. This PR removes it, and adds a new line to improve the API docs readability.
Truly sorry about this guys ! :cry:

:+1:

@guilhermeFranco Thank you for this good catch. :+1:


Update app dialog disappears after a couple of seconds

The In app update is working but the dialog only shows for a couple of seconds before disappearing, any way I can solve this ?

Hi @Noel-96, Can you check whether there are any error logs in the console

Closing as no response from the author


run bindgen at build time

This will allow linking to the correct .so on either FreeBSD 13 or 14.
Fixes #16

This PR should mostly fix the problem.  However, in its current state it will result in broken docs on docs.rs.  I need a few more days to fix that.

@andoriyu this is ready to review now.  The CI is failing due to #20 .

Force pushed to fix the build on i686-unknown-freebsd and to remove the obsolete bindgen.sh script.


Its not working more

Hi!
Its working like charm till now. At this time its stopped working and providing presence of numbers. Let me know if there any update regarding it.
Thanks

I tried it like 5 minutes ago and everything seems fine. I've use a freshly opened Web Whatsapp and ctrl-c/v'ed the script from 'ui.js' and pasted it to the console.
If it doesn't work anymore then the extension bugged out. I'd like to hear from other people who experience the same problem.

Hi @LoranKloeze !
Thanks for quick response. I have checked with different conditions and found that its work only for number which we have in our contact as well our number must saved in that numbers contact list. Can You please confirm its right or not? I have tried many time and found this.
Thanks

Yes, I see that just in whatsappweb, Store object doesn't update "isOnline" status, if that number doesn't have our number in their contact list. If they have, everything is working. I think they made a security update a couple days ago.

You are correct. But the information about the online status still flows into Web WhatsApp, I can see it coming in through the websocket. I guess WA has changed something about the API. When I have the time, I'll look into it.

It seems that online element is now named "__x_isOnline" instead that "isOnline". I changed it and it seems to work now

I changed line 249 of ui.js with:
if (m.__x_isOnline) {

Hi! @mertxkaan @LoranKloeze I have marked that if we send message to that number, its working. Let me know if anyone can find solution for it.
Thanks

@LoranKloeze Any update in this issue?
Thanks!

I found a way to subscribe to presence updates but the server doesn't react on that subscription. It's still on my list

Hi!
I have tried below code but I think its not worth. There no any changes in presence after subscribe via below codes.
Store.Wap.subscribePresence(919400000000+ '@c.us').then(function(r){ console.log(r)})
Thanks

@LoranKloeze
I think below codes are removing subscription. If we prevent to run it than may be We can run it long time till We refresh.
clearAllPresence: function() {                 this.forEach(function(e) {                     e.reset(),                     e.set({                         stale: !0,                         isSubscribed: !1                     })                 })             },

@LoranKloeze Any updates?

@karopass, you don't get presence updates from numbers that aren't in your contact list. WA has changed this. A few months ago you could get presence updates from all numbers but not anymore

@LoranKloeze Yes I know it but its still not get Presence update if number is in my contact list but have never send message to that contacts. There no any hope about run like before?
Thanks

Nope, I don't think so. I'll remove the references to the online statuses in the code and the readme.

There is a new parameter called stale. The presence are not updated when stale is true. But works fine if stale is false. But manually setting it to false doesn't work.

There is an app for iPhone named NETWA that collects presence data of up to 10 numbers. The app connect to their server and update information. I'm quite sure they use socket flow information of presence considering they have 1 s resolution. It stopped working begin of may. Then they add possibility of only 1 number per user, then 2, 4 and now 10 again. So they have managed how to solve the problem.
And I have tested they don't send any message to numbers you monitor. So, there should be the way to get it.

This means someone has to put in a lot of time to reverse engineer the undocumented API. I'm a little busy with commercial projects. Maybe later this year :)

I'm gonna close this. WhatsApp has fixed the privacy issue we we're using.
This will never work again. I'm sorry.
Please don't open this issue again.
Thanks.


ARROW-13238: [C++][Compute][Dataset] Use an ExecPlan for dataset scans

Replaces the body of AsyncScanner::ScanBatchesAsync with usage of an ExecPlan

I'll test this on EC2 when I get a chance and compare to 4.0/current master.

@ursabot please benchmark lang=Python

(Looks like we need to rebase before the Conbench benchmarks can run.)

@ursabot please benchmark lang=Python

From Conbench seems like performance isn't really affected. I'll manually run those benchmarks with more iterations as well.

This got stuck while I was testing it (I'll try to get a backtrace shortly). But performance indeed seems to be on par with before.

4.0.1:
        "iterations": 10,
        "max": "10.501172",
        "mean": "9.783475",
        "median": "9.783779",
        "min": "9.203580",
        "q1": "9.451842",
        "q3": "10.074056",

5.0.0 master
        "iterations": 10,
        "max": "12.266335",
        "mean": "10.373638",
        "median": "10.223064",
        "min": "9.785621",
        "q1": "10.084575",
        "q3": "10.322508",

Count rows, no filter, 10 iterations:
Min: 0.39461345200015785 s
Median: 0.420534689500073 s
Mean: 0.4404416943999422 s
Max: 0.5733599779996439 s

This PR
        "iterations": 10,
        "max": "11.053488",
        "mean": "10.406635",
        "median": "10.273359",
        "min": "9.980684",
        "q1": "10.145428",
        "q3": "10.701346",

Count rows, no filter, 10 iterations:
Min: 0.3555871419994219 s
Median: 0.38891541300017707 s
Mean: 0.4031996126002014 s
Max: 0.5227700120003647 s

(It got stuck when a filter was applied)

Ok, actually, I'm not sure if it was stuck or just me being inpatient…

Now that I'm running it again it just seems that count_rows with a filter got very slow.
Before:
Min: 24.285114823000185 s
Median: 25.309564675000274 s
Mean: 25.45627587750014 s
Max: 26.564001424999333 s

After:
Min:<PHONE_NUMBER>0700029 s
Median:<PHONE_NUMBER>8250028 s
Mean:<PHONE_NUMBER>6600005 s
Max:<PHONE_NUMBER>839998 s

Test script
import statistics
import time
import pyarrow
import pyarrow.dataset
import pyarrow.fs

fs = pyarrow.fs.S3FileSystem(region="us-east-2")

print("PyArrow:", pyarrow.__version__)

times = []
for _ in range(10):
    start = time.monotonic()
    ds = pyarrow.dataset.dataset([
        "ursa-labs-taxi-data/2009/01/data.parquet",
        "ursa-labs-taxi-data/2009/02/data.parquet",
        "ursa-labs-taxi-data/2009/03/data.parquet",
        "ursa-labs-taxi-data/2009/04/data.parquet",
        ],
        format="parquet",
        partitioning=["year", "month"],
        filesystem=fs,
    )
    print("Rows:", ds.scanner(use_async=True, use_threads=True).count_rows())
    times.append(time.monotonic() - start)

print("No filter")
print("Min:", min(times), "s")
print("Median:", statistics.median(times), "s")
print("Mean:", statistics.mean(times), "s")
print("Max:", max(times), "s")

print("With filter")
times = []
expr = pyarrow.dataset.field("passenger_count") > 1
for _ in range(10):
    start = time.monotonic()
    ds = pyarrow.dataset.dataset([
        "ursa-labs-taxi-data/2009/01/data.parquet",
        "ursa-labs-taxi-data/2009/02/data.parquet",
        "ursa-labs-taxi-data/2009/03/data.parquet",
        "ursa-labs-taxi-data/2009/04/data.parquet",
        ],
        format="parquet",
        partitioning=["year", "month"],
        filesystem=fs,
    )
    print("Rows:", ds.scanner(use_async=True, use_threads=True, filter=expr).count_rows())
    times.append(time.monotonic() - start)
print("Min:", min(times), "s")
print("Median:", statistics.median(times), "s")
print("Mean:", statistics.mean(times), "s")
print("Max:", max(times), "s")

Well that's disappointing. I was actually expecting it to get faster

If we correct the CountRows issue above, CountRows indeed gets much faster!


Optimize middleware default options

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

20 of 20 (100.0%)  changed or added relevant lines in 8 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.004%) to 97.824%

Totals

Change from base Build<PHONE_NUMBER>6:
-0.004%

Covered Lines:
3551

Relevant Lines:
3630

💛  - Coveralls


File issues into OSO Insta-analysis Epic

What is it?
https://github.com/opensource-observer/oso/milestone/46
The intention of this epic to to capture how do we turn a grant exploration into < 1 day.
i.e. Foundation gives us a CSV dump of a grant round. We can import this into oss-directory, run pipeline, show valuable insights automatically in the web app. All in less than a day.
What aspects of an ecosystem report are something we should be able to generate for any collection?
Note: This assumes we've already shipped the OSO terminal MVP and this is follow-on work.
Note: If it's useful, feel free to pull in issue from our OSO terminal backlog

I've reviewed your issues and added one of my own. Will close out now!


zlib 0.6

Given the dependency from cabal-install, which @tibbe has implied in the past will not be getting backported patches for 1.18 anymore, this may need to wait until GHC 7.10.
tasty-0.10.1 depended on by:

zlib-<IP_ADDRESS> (==0.8.*). No maintainer.

tasty-hunit-0.9.2 depended on by:

zlib-<IP_ADDRESS> (==0.8.*). No maintainer.

zlib-<IP_ADDRESS> depended on by:

JuicyPixels-<IP_ADDRESS> (>=<IP_ADDRESS> && < 0.6). Vincent Berthoux<EMAIL_ADDRESS>@Twinside
cabal-install-<IP_ADDRESS> (>=0.5.3 && < 0.6). Michael Snoyman<EMAIL_ADDRESS>@snoyberg. @haskell
pandoc-<IP_ADDRESS> (==0.5.*). No maintainer. @jgm
yesod-bin-1.4.7 (==0.5.*). Michael Snoyman<EMAIL_ADDRESS>@snoyberg. @snoyberg
zlib-lens-<IP_ADDRESS> (>=0.5.4 && < 0.6). Edward Kmett<EMAIL_ADDRESS>@ekmett

Bumped in JuicyPixels-<IP_ADDRESS>

Submitted a revision for zlib-lens.

Looks like zlib's test suite has a restrictive upper bound (pinging @dcoutts):
tasty-hunit-0.9.2 depended on by:

zlib-<IP_ADDRESS> (==0.8.*). No maintainer.

But looks like both cabal-install and io-streams still have upper bounds (the latter just being for the test suites):
zlib-<IP_ADDRESS> depended on by:

cabal-install-<IP_ADDRESS> (>=0.5.3 && < 0.6). Michael Snoyman<EMAIL_ADDRESS>@snoyberg. @haskell
io-streams-<IP_ADDRESS> (==0.5.*). No maintainer. @mightybyte

Pinging @rthomas

Update: no progress here :( but a PR for fixing this in cabal-install was merged.

Update: we've got a new dependent on the old zlib:
tasty-hunit-0.9.2 depended on by:

zlib-<IP_ADDRESS> (==0.8.*). No maintainer.

zlib-<IP_ADDRESS> depended on by:

cabal-install-<IP_ADDRESS> (>=0.5.3 && < 0.6). Michael Snoyman<EMAIL_ADDRESS>@snoyberg. @haskell
io-streams-<IP_ADDRESS> (==0.5.*). No maintainer. @mightybyte
xlsx-<IP_ADDRESS> (==0.5.4.*). Scott Murphy<EMAIL_ADDRESS>@smurphy8. @qrilka

io-streams does not depend on zlib directly.  Also, Gregory Collins is the maintainer of that package, not me.

@gregorycollins would you like to be listed as the stackage maintainer of io-streams? (This just means you'll try to be responsive to requests for fixing build failures and bumping dependency versions.)
io-streams has been pulled into stackage transitively by other packages, so this would just move pings related to io-streams from those dependencies to you directly. snapframework/io-streams/issues/34
Just follow the instructions on http://stackage.org for "package authors".

In version <IP_ADDRESS> of xlsx I've loosened the bounds of zlib dependency

Almost:
tasty-hunit-0.9.2 depended on by:

zlib-<IP_ADDRESS> (==0.8.*). No maintainer.

zlib-<IP_ADDRESS> depended on by:

Agda-<IP_ADDRESS> (>=<IP_ADDRESS> && < 0.6.1). Andrés Sicard-Ramírez<EMAIL_ADDRESS>@asr. @agda
cabal-install-<IP_ADDRESS> (>=0.5.3 && < 0.6). Michael Snoyman<EMAIL_ADDRESS>@snoyberg. @haskell

Agda is not supporting zlib <IP_ADDRESS> due to Issue https://github.com/agda/agda/issues/1518.

Update: no change.

It seem Idris also gets a segmentation fault with zlib <IP_ADDRESS> (see https://github.com/idris-lang/Idris-dev/pull/2433).

Sounds like the Agda & Idris issue w/ zlib <IP_ADDRESS> will be fixed in GHC 7.10.3: https://ghc.haskell.org/trac/ghc/ticket/10904

I wouldn't hold breath on this, as cabal-install with zlib-0.6 will most likely coincide with GHC 8.0

< as cabal-install with zlib-0.6 will most likely coincide with GHC 8.0
Is there some particular reason for it?
I just saw that cabal-install.cabal is using zlib >= 0.5.3 && < 0.7.

Sounds like the Agda & Idris issue w/ zlib <IP_ADDRESS> will be fixed in GHC 7.10.3:
https://ghc.haskell.org/trac/ghc/ticket/10904

Using the ghc-7.10 branch which includes the fix to GHC issue 10904, I couldn't reproduce the Agda issue with zlib 0.6.1.*.

tasty-<IP_ADDRESS> depended on by:

zlib-<IP_ADDRESS> (>=0.8 && < 0.11). No maintainer. @haskell

tasty-hunit-0.9.2 depended on by:

zlib-<IP_ADDRESS> (==0.8.*). No maintainer. @haskell

zlib-<IP_ADDRESS> depended on by:

Agda-<IP_ADDRESS> (>=<IP_ADDRESS> && < 0.6.1). Andrés Sicard-Ramírez<EMAIL_ADDRESS>@asr. @agda
cabal-install-<IP_ADDRESS> (>=0.5.3 && < 0.6). Michael Snoyman<EMAIL_ADDRESS>@snoyberg. @haskell
stack-<IP_ADDRESS> (>=<IP_ADDRESS> && < 0.6). FP Complete<EMAIL_ADDRESS>@snoyberg. @commercialhaskell

I've bumped stack's upper bound for zlib on Hackage

The only thing left here (sans a test-suite) is a cabal-install that allows the newer zlib. That means we have to wait for GHC 8, right?

@bergmark I believe that assessment is correct.

Is a cabal-install backport possible?

We can't upgrade Cabal at all, unfortunately, because then you would not be able to install packages that depend on both, for example ghc-mod. This includes transitive dependencies as well.

Cabal doesn't depend on zlib, only cabal-install

Oh right, thanks. Then I don't know the reason to not upgrade it.

Right, but don't we build cabal-install with stackage?

There will be cabal-install with relaxed upper bounds for zlib and QuickCheck (not sure if cabal tests are run). Very soon :tm:, PR is merged.


drivers: pwm: nrf_sw: use Nordic HAL to configure GPIO

Use Nordic HAL to configure GPIO output. This adds support for P1 GPIOs,
when pin numbers above 31 are used.
Signed-off-by: Marcin Niestroj<EMAIL_ADDRESS>

@anangl Could you help with assigning second reviewer?


Nothing is being generated

Hello I am running in pycharm and the gui pops up but I dont see any text transcription and I get the error:
"You exceeded your current quota, please check your plan and billing details."
but I have the premium account, is this defaulting to GPT-4 or something? I dont see any text populating and checking the settings on the openai site to see my usage nothing is coming up... Do I need to upgrade to 120$/mo plan to use this?

Any help is appreciated thanks

Here is from the openAI account page

Here is a snapshot of my account from the OpenAI website usage stats

Oh it turns out you do have to pay extra or more to use the API for this.... bummer


Minor Optimizations & Code Re-Org

Applied many minor optimizations in parser code
Expanded comments
Clarified the way some methods work by re-organizing their flow

Codecov Report

Merging #51 (6bbc217) into main (a15482b) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main      #51      +/-   ##
==========================================
- Coverage   99.98%   99.98%   -0.01%
==========================================
  Files          48       48
  Lines        6371     6351      -20
  Branches     1355     1145     -210
==========================================
- Hits         6370     6350      -20
  Misses          1        1

Flag
Coverage Δ

unittests
99.98% <100.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

lib/cli_command_parser/command_parameters.py
100.00% <100.00%> (ø)

lib/cli_command_parser/context.py
100.00% <100.00%> (ø)

lib/cli_command_parser/nargs.py
100.00% <100.00%> (ø)

lib/cli_command_parser/parameters/base.py
100.00% <100.00%> (ø)

...ib/cli_command_parser/parameters/option_strings.py
100.00% <100.00%> (ø)

lib/cli_command_parser/parameters/options.py
100.00% <100.00%> (ø)

lib/cli_command_parser/parse_tree.py
100.00% <100.00%> (ø)

lib/cli_command_parser/parser.py
100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes


FAQ is missing the most important question

Maybe it's just me being dumb today, but I think this question deserves an answer there (or right in the README):
What's the main point? / Which problem does this project solve?
The description has this:

An open source kernel extension enabling native macOS HD audio for not officially supported codecs without any filesystem modifications.

I think, this needs to be expanded, it's not clear as it is… at least with a link to some place where these things are explained — what's the difference between native/non-native HD audio and why does it matter, what does it mean for a codec to be not officially supported and what kind of support are we talking about. Without context, these words may mean a lot of very different things.

Hi, are you sure you need that information? If you stumbled on this project across the net, you almost certainly don't need it.
If you check the support topics from the readme, you could get the idea. The list of the supported (hardware) codecs is available on the wiki and the matter of native/non-native audio driver usage is relatively important due to e.g. AppleHDA-specific plugins required for playing DRM-protected movies in iTunes.
However, I have no idea why it bothered you and what could you do with this information :%.


PDF Loader Issue

AttributeError: module 'PyPDF2' has no attribute 'PdfReader'

Traceback
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
File "/Users/serge/Downloads/ClassGPT-main/app/01_❓_Ask.py", line 52, in <module>
    res = query_gpt(chosen_class, chosen_pdf, query)
File "/Users/serge/Downloads/ClassGPT-main/app/utils.py", line 51, in query_gpt
    documents = loader.load_data(pdf_tmp_path)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/llama_index/readers/llamahub_modules/file/pdf/base.py", line 22, in load_data
    pdf = PyPDF2.PdfReader(fp)

It should already be fixed. If the issue persists, please don't hesitate to reopen it


UI calls column types "string" when they have no native arrow js-api implementation

Description
In the UI, columns whose types should be (for example) ZoneId or LocalDateTime get called String types when hovering over the column names. These columns should be called their correct types, with an additional message that mentions the string preview.
Steps to reproduce
Run the following code:
from deephaven import empty_table

t = empty_table(10).update(['tz1 = parseTimeZone("GMT")', 'tz2 = parseTimeZone("America/New_York")'])
t = t.update(["ldate1 = '2022-03-28'", 'ldate2 = parseLocalDate("2075-08-08")'])
t = t.update(["ltime1 = '10:15:30.46'", 'ltime2 = parseLocalTime("12:01:01.4567")'])

Expected results
When I hover my mouse over any of these columns, I expect to see their types displayed as ZoneId, LocalDate, etc.
Actual results
The types get labelled and displayed as strings, but the metadata table confirms that they are actually of the correct type:

Versions

Engine Version: 0.29.0
Java Version: 17
OS: MacOS
Browser: Safari
Docker: NA - built & run with gradle

Suggested change: Move the actual column type into the brackets and change the previewing line to "Previewing as String"

Superceded by tickets on the Core side for this: https://github.com/deephaven/deephaven-core/issues/3358
and https://github.com/deephaven/deephaven-core/issues/2102
Web UI doesn't know when a column is converted to a Preview column at the moment.


IsConnected in async loop is big performance hog

hey there domj, i wondered why using this plugin was eating 30% of my CPU. then i found the async loop to be the issue. it seems it is not needed anyway, is it? the IsConnected works just fine on its own.
cool contrib anyway. was very surprised to see it on nuget when looking for something to get a playstation controller going. Thanks for that!

It works fine but IIRC it is a fairly expensive operation and as such I thought it better be done async. Just not in the AsyncLoop that just runs as fast as it can and fully consumes the CPU thread 😀
Motzi also raised this issue, and solved it temporarily by adding a Sleep into the AsyncLoop.

I will update the nuget with this or maybe a timer triggering the operation. The Controller can also likely be Cached. It's really been ages since I did this 😅
So it works with Playstation controller as well? That's nice.

ok i see. regarding the ps controller you need middleware. there is http://ds4windows.com/ which hooks between with an extra driver and exposes the ps controler to the os. it ends up somehow seen as xbox controller by windows.

Very cool, thanks!

This issue should be fixed now.
Please try it out and open a new issue if there are any problems still, thanks!


Continuation token for subsequent queries does not work

I have been having a very strange porblem. I use a mobile app to access the Document DB via a REST service. I am passing through the continuation token in the http headers. When the mobile client sends the first continuation token to me in a subsequent request, it works just fine, and I send over a new continuation token for the second page of results.
Now, things get interesting. I get zero results for the third page, even though I have a continuation token and I have verified that I have more than 40 records in the request. So, I changed the continuation token manually on the client side (I updated the number and the index in the continuation token) and sent it and in this scenario it works. I compared the continuation token sent by CosmosDB and the continuation token manipulated by me, and there's no difference. What could be the issue?

@bchong95 please look into this.

Maybe related to #376?


Issue with nested pages

The docs build fails when nested pages are used:
pages = [
    "Quick start" => "quick-start.md",
    "Design" => "design.md",
    "Models" => "models.md",
    "API" => "API.md",
    "Tutorials" => [
        "Regression - Boston" => "tutorials/regression-boston.md",
        "Logistic - Titanic" => "tutorials/logistic-titanic.md",
        "Classification - IRIS" => "tutorials/classification-iris.md",
    ]
]

results in following error in CI:
Deploying: ✔
ERROR: LoadError: MethodError: no method matching pagelist2str(::Documenter.Document, ::Pair{String, Any})

Closest candidates are:
  pagelist2str(::Any, ::Pair{String, String})
   @ DocumenterVitepress ~/.julia/packages/DocumenterVitepress/IO55P/src/vitepress_config.jl:75
  pagelist2str(::Any, ::String)
   @ DocumenterVitepress ~/.julia/packages/DocumenterVitepress/IO55P/src/vitepress_config.jl:62
  pagelist2str(::Any, ::Pair{String, <:AbstractVector})
   @ DocumenterVitepress ~/.julia/packages/DocumenterVitepress/IO55P/src/vitepress_config.jl:81

However the following works fine:
pages = [
    "Quick start" => "quick-start.md",
    "Design" => "design.md",
    "Models" => "models.md",
    "API" => "API.md"
]

It looks like this may have been what was being adressed by https://github.com/LuxDL/DocumenterVitepress.jl/pull/35.
However, the above CI example were run with [4710194d] + DocumenterVitepress v0.0.4. Maybe the previous fix wasn't part of the latest registered version?
Thanks for the great work, a very promising upgrade to docs format!

For reference, here is the reference to the make.jl.
Note that when building locally, it fails with the following:
[ Info: RenderDocument: rendering document.
[ Info: DocumenterVitepress: rendering MarkdownVitepress pages.
┌ Warning: DocumenterVitepress: no assets directory found.
└ @ DocumenterVitepress C:\Users\jerem\.julia\packages\DocumenterVitepress\IO55P\src\writer.jl:217

ERROR: LoadError: SystemError: opening file "C:\\Users\\jerem\\OneDrive\\Evovest\\NeuroTreeModels.jl\\docs\\build\\tutorials\\classification-iris.md": No such file or directory

I noticed that the build folder has some of docs src files directly under build while unders are under build/.documenter:

Not sure it's may relate to the package.json: "docs:dev": "vitepress dev build/.documenter"? As mentionned, the build is fine on CI (when nested pages are removed), but locally, the make.jl fails.

Should be fixed in v0.0.6


proposal: trimmed down context object

One of the biggest things we've talked about for M2 is stripping down the core botbuilder package to its essence.  Thought I'd start with proposing a trimmed down context object:
see updated shape proposal https://github.com/Microsoft/botbuilder-js/issues/66#issuecomment-366403012
export interface BotContext {
    /** The Bot object for this context. */
    bot: Bot;

    /** The received activity. */
    request: Activity;

    /** Queue of responses to send to the user. */
    responses: Partial<Activity>[];

    /** If true at least one response has been sent for the current turn of conversation. */
    readonly responded: boolean;

    /** Deletes an existing activity. */
    deleteActivity(id: string): Promise<void>;

    /** Sends a set of activities to the user. */
    postActivity(activities: Partial<Activity>[]): Promise<ConversationResourceResponse[]>;

    /** Flushes the response queue. Calls postActivity() */
    flushResponses(): Promise<ConversationResourceResponse[]>;

    /** Replaces an existing activity. */
    updateActivity(activity: Partial<Activity>): Promise<void>;
}

Let me go through and try to justify each part on the context object...
bot
This is really to provide access to the adapter as we've found you often need access to this. In particular, Middleware for MS Teams needs access to the underlying adapter to get an access token for calling other REST services they need to call.
request
Obviously provides access to the received activity.
responded
This is a flag that indicates whether a reply has been sent for the current turn. This is potentially useful from within middleware to know if you should run logic on the trailing edge of a request. For instance, QnA Maker can look at this flag on the trailing edge of the request and only call the service in a fallback case should no other logic in the bot respond.
responses and flushResponses()
This provides a means of queuing up outbound responses. Bots often end up sending more than one message to the user for the same turn and given that your bots logic might be spread across multiple components it's possible for several components to queue up an outgoing response on the same turn. Since the bots state is written out every time activities are sent it's best to queue up all of the responses so that you can collapse state updates down to a single write per turn.
The flushResponses() method gives you a way of manually flushing the response queue should you need to and it's also the method called when the context is about to be destroyed.  All this method would do is call the new postActivity() method and then delete the flushed responses on completion.
postActivity(), updateActivity(), and deleteActivity()
These are new methods I'm proposing to explicitly perform CRUD operations against the underlying adapter in a way that goes through middleware. We recently realized that while we have a postActivity middleware pipe we're missing delete and update pipes. Currently updates and deletes for existing activities need to happen using the adapter directly which bypasses middleware meaning that updates & deletes won't be logged.
I have another proposal coming which will recommend we add new updateActivity and deleteActivity pipes to middleware.  What's nice about these 3 new methods I'm proposing is that they will a) provide a direct means of posting activities (something we've been asked for) and b) they will align name wise with the names of the middleware pipes that will get run when each method is called.  Should make it clearer what triggers the running of those pipes.

This would address issue #59

I'm a big fan of the overall direction towards simplicity. If I understand correctly, you're essentially proposing to turn BotContext into a CRUD + Flush container.
The only genuine issue I have is that we're getting awfully close to being formal "State Management". By that, I mean the the combination of adding responses, manipulating existing responses, then syncing that state with a remote user. We're walking up to that state management issue via CRUD operations, and not through a more deliberate approach.
In that regard, I really like the Action/Reducer pattern that Redux uses. I also wonder if we should treat sending more akin to replication, where the local state of the bot is replicated to the user wherever they might be.
Now, at the risk of undermining my own thinking, if we implement a strict CRUD layer here in the BotContext, I don't think anything prevents us from enriching it with another layer down the road.
At the Nitpicky level, I have a few minor concerns:

It's not clear, on the Send path, what's Sync and what's Async. That confusion is what leads to the need for FlushResponses. I believe a common pattern is:

Send a Typing Indicator
FlushResponses() to get it out to the user
Assemble the actual response.
Send that response.

That pattern should be semantically simple.

It's not clear that batching is possible beyond "Responses is an Array". The API doesn't offer any hints, and with the lack of clarity around sync/async it's a bit more confusing. Do I batch up "Typing" + "Delay" + "Actual Response" or do I flush them one at a time?

I'm not sure how much I like "responded". I 100% get the need for components to tell "Has anything happened", but some responses - such as "Typing" - would qualify even though they're not doing anything. I think we should consider some more semantically rich options.

I don't like PostActivity as a name. SendActivity seems much more obvious.

//Chris

So to be clear... The update and delete methods I've added are to better model the v3 protocol that the Bot Framework supports. If this looks like a more formal "State Management" that's because the v3 protocol supports this level of CRUD operation.  You'll notice that I'm specifically NOT modeling read because that's not consistently supported across all channels.
With this proposed interface you could send a typing indicator directly using postActivity() which would be the same as adding it to the responses queue and calling flushResponses().
This layer is intended to be a very raw layer not unlike HTTP Request and Response. There's very little that's intuitive about those interfaces either. You basically need to read the documentation to understand the ins and outs of working at this raw of a layer. The idea is that you will add middleware plugins that extend the context object with members that are semantically more meaningful.  There's nothing hear that even tells you what types of activities you can send. You need to add a piece of middleware that adds queueReply() and queueTyping() methods. The point though is that we want that opinion to be replaceable so it needs to layer on top of something more raw.
The addition of the responded property was to eliminate the need for a more explicit handled = true flag to indicate that the request has been handled and shouldn't be processed any further.  Components like QnA Maker that have been added as middleware need the ability to know if they should run their logic or not. The con with the explicit approach of setting a flag to 'true' is that its super easy to forget to do this in your code.  In most cases you can use the delivery of an activity as a good indicator that the request has been handled.  Even the sending of a "typing" indicator is a good indicator that the request is being handled in some way. If we remove the automatic responded flag then we need to add back in an explicit handled = true which nobody, including myself, liked.  This is a very real issue for components like QnA Maker so we can't just punt and say we're not going to solve their problem.
I'm not opposed to renaming postActivity() to sendActivity() but we should then also rename the middleware, bot, and adapter methods to Middleware.sendActivity(), Bot.send(), and Adapter.send(). They should all match for consistency purposes.

There's actually a further simplification that can be made here... I think we can remove the batching aspects of context.responses and context.flushResponses() and replace this with a piece of middleware:
export class BatchOperations implements Middleware {

    public contextCreated(context: BotContext, next: () => Promise<void>): Promise<void> {
        let responses: Partial<Activity>[] = [];
        context.batch = {
            reply: (text: string) => {
                responses.push({ type: 'message', text: text });
            },
            typing: () => {
                responses.push({ type: 'typing' });
            },
            flush: () => {
                const flushed = responses;
                responses = [];
                return context.postActivity(flushed);
            }
        }

        return next()
            .then(() => context.batch.flush());
    }
}

This will actually simplify our internal middleware implementation because we no longer need to explicitly flush() the response queue as part of our pipeline processing and the lifetime of the context object becomes cleanly tied to the lifetime of the contextCreated pipeline.  If you want batching semantics you can mimic the current behavior by simply using the middleware above as the first piece of middleware in your stack.

Updating this proposal to reflect the latest thinking spanning several other discussion threads:
export interface TurnContext {
    /** The Bot object for this context. */
    bot: Bot;

    /** The received activity. */
    request: Activity;

    /** If `true` at least one response has been sent for the current turn of conversation. */
    readonly responded: boolean;

    /** Gets a value previously cached on the context. */
    get<T = any>(key: string): T;

    /** Returns `true` if [set()](#set) has been called for a key. The cached value may be `undefined`. */
    has(key: string): boolean;

    /** Caches a value for the lifetime of the current turn. */
    set(key: string, value: any): this;

    /** Sends a set of activities to the user. */
    sendActivities(activities: Partial<Activity>[]): Promise<ResourceResponse[]>;

    /** Replaces an existing activity. */
    updateActivity(activity: Partial<Activity>): Promise<void>;

    /** Deletes an existing activity. */
    deleteActivity(id: string): Promise<void>;

    /** Registers a handler to be notified of and potentially intercept the sending of activities. */
    onSendActivities(handler: (activities: Partial<Activity>[], next: () => Promise<ResourceResponse[]>) => Promise<ResourceResponse[]>): this;

    /** Registers a handler to be notified of and potentially intercept an activity being updated. */
    onUpdateActivity(handler: (activity: Partial<Activity>, next: () => Promise<void>) => Promise<void>): this;

    /** Registers a handler to be notified of and potentially intercept an activity being deleted. */
    onDeleteActivity(handler: (id: string, next: () => Promise<void>) => Promise<void>): this;
}

Changes of Note

Rename of BotContext to TurnContext.
Added get(), has(), and set() methods that components can use to cache values for the lifetime of the current turn. These methods would follow the basic semantics of a Map. See  issue #56
Added sendActivities(), updateActivity(), and deleteActivity() methods to context object. The Bot Framework protocol supports not just sending activities but updating and deleting them as well. See issue #67
Removed the responses queue and related flushResponses() method. This added complexity to the perceived lifetime of the context object and can easily be added back on using middleware.
Added onSendActivities(), onUpdateActivity(), and onDeleteActivity() methods for registering callbacks that want to be notified of and potentially intercept outbound calls from the bot. This will let us simplify middleware by getting down to a single pipe. See issue #74

One question about the proposed interface is that C# programmers tend to like really well spelled out methods like sendActivities() where from a JS perspective I'd lean towards something shorter like send(). How should we bridge differences like this?

I don't know how to respond to this DCR. It seems like a catch-all summarizing a variety of proposals being discussed in other issues, although you are also discussing some of them here.

I know you don't lie the on handlers. are their other specific line items you have issues with?  I just started work on this last night so now is a great time to comment.
I'll be publishing an m2 branch sometime today and I would expect all of these various changes to just collect there until after the Milan hack is over. I implemented the above changes to the context object last night and so far they feel pretty good implementation wise.  But nothing is set in stone of course.

BTW...  The vast majority of DCR discussions is happening over in the .NET repo. I'm actually lagging behind them for once... lol

If you're trying to mirror Map, then please add a delete(key) function. If you're not, then we can get rid of has(key) because get(key) can return undefined if the key doesn't exist (in fact that's the behavior of Map.prototype.get(key) so I'm not sure why they bother with has(key)) and set(key, undefined) can delete the key. All in all, though, I like the idea of mirroring Map.
In which case, I think I'd put those all under something like context.cache, e.g. context.cache.get(key). That gives a lot more, well, context for what those generically-named functions do. It even lets you make cache an actual Map in JS.
Following that line of thinking, I would say let cache be the common naming between JS and .net, and let cache be the appropriate type for each language. That lets you share naming for the concept but let the actual implementation be idiomatic: Map in JS, and Dictionary (or whatever) in .net.

This work is done.


Fix Issue with Overflowing Trending Coins

suggest you use "overflow: hidden" in the css for the component Trending

Done!


Customer request: Add support for clearing the spatial observations

This feature was requested by a customer in our online forums.
This change adds the ability to request the spatial observations be cleared (removed from the scene and freed).
It also renames SpatialAwarenessObjParent to SpatialAwarenessObserverParent for clarity.
The Spatial Awareness demo scene has been updated to add a sphere that allows for toggling the observers (first click suspends and clears, second click resumes, etc).
Testing performed

Build and deploy to HoloLens (il2cpp and .net backends). Confirm that the meshes clear and resume.
Build and run in editor using an early file observer prototype.

would be great if we could add tests for this: creating the spatial observer, suspending, clearing, resuming

/azp run mrtk_pr

/azp run mrtk_docs

would be great if we could add tests for this: creating the spatial observer, suspending, clearing, resuming

Once i get the file / model observer PR completed, i can add a test for it. :)


fix: Make isWindow work for electron

Updates the isWindow check to also pass for Electron window objects.
Closes #540

Hey @knubie, thanks for opening a PR. There's actually already a PR open that implements the same fix here https://github.com/clauderic/dnd-kit/pull/532. It was opened before yours, so I'll be merging that one.

Great, sorry for the noise!


Question: what to do for many files?

what to do for many files that are often in lib folder?

While this plugin is great step to show developers right way, it is still requires moving every jar to separate folder.
I came here from https://stackoverflow.com/questions/364114/can-i-add-jars-to-maven-2-build-classpath-without-installing-them
P.S. Asked on https://stackoverflow.com/questions/45307633/maven-equivalent-for-gradle-compile-filetreedir-libs-include-jar

This is gross. I think you missed the point, this plugin is a crutch, not the right way to do anything. You have a list of versioned jars, the majority of which are in Central. Just do the work and fix your pom.

You have a list of versioned jars, the majority of which are in Central. Just do the work and fix your pom.

And I have to too many times for old projects.
When creating new project to handle old jars, I would recommend now just use maven-install-plugin
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.company</groupId>
	<artifactId>3rd-parties-jars</artifactId>
    <version>1.0</version>

    <build>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-install-plugin</artifactId>
          <version>2.5.2</version>
          <executions>
            <execution>
              <id>install-ojdbc7</id>
              <phase>install</phase>
              <goals>
                <goal>install-file</goal>
              </goals>
              <configuration>
                <file>ojdbc7-<IP_ADDRESS>.0.jar</file>
                <groupId>com.oracle</groupId>
                <artifactId>ojdbc7</artifactId>
                <version><IP_ADDRESS>.0</version>
                <packaging>jar</packaging>
              </configuration>
            </execution>
            <!--
            <execution>
              <id>install2</id>
              <phase>package</phase>
              <goals>
                <goal>install-file</goal>
              </goals>
              ... etc

            </execution>
            ... other executions
             -->
          </executions>
        </plugin>
      </plugins>
    </build>
  </project>

Then this non-maven-jar-maven-plugin is direction if it is possible to get and compile sources later (when modification is needed.)


fix: Use pipes to insert cert data as a multiline string

This is templating out like this
data:
  tls.cert: |
    -----BEGIN CERTIFICATE-----
    MIIBjTCCATSgAwIBAgIRAKGEhukoBVNKfq+72cQPlUgwCgYIKoZIzj0EAwIwJTEj
    MCEGA1UEAxMacm9vdC5saW5rZXJkLmNsdXN0ZXIubG9jYWwwHhcNMjIwMzE2MTcy
    MDQ4WhcNMzIwMzE2MDUyMDQ4WjAlMSMwIQYDVQQDExpyb290LmxpbmtlcmQuY2x1
    c3Rlci5sb2NhbDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABH4L2yIwWc5+Y60i
    3gp1Crd20fFv/7ROJ2ZxM6LFso2z7uuIP9PrfdU1z3ZJmVZbwSHAIwPkxSVZV2P9
    +L9rDdijRTBDMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEBMB0G
    A1UdDgQWBBRp7oA/OTYSWpXyy/13qkWKZK9JJzAKBggqhkjOPQQDAgNHADBEAiBa
    Fot+ljrjNCUgDk0WIsffnrygkq0e2IkxwD7Nqv4BUQIgV3G6Le7J9Z0rm+KH/uWO
    r3nrYdRewLjMNVk1XZZaR/E=
    -----END CERTIFICATE-----

Which is what we need

:tada: This PR is included in version 1.0.0-alpha.7 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


state.persist is broken in HA 2022.3

https://github.com/home-assistant/core/pull/66068 changed the signature of async_restore_entity_added from taking an entity str to taking a RestoreEntity

It looks like we'll need an "Entity" that extends "RestoreEntity". But we aren't actually using "Entity" at the moment.
I believe this will require us to treat all entities similar to how they were handled in PR #129

Confirm that I am unable to create persistent state vars with the latest versions of everything installed. This is my first day playing with PyScript and I'm glad to see it's not just me!

I haven't tested this with any of my actual pyscripts because I'm not quite ready for a 2022.3 upgrade yet. But if any brave souls want to try PR #332 and let me know if it works for you that'd be great.

Thanks, @dlashua! I just pulled the modified file into my custom_components location, restarted HASS, and boom! The problem is gone, and persisting states appears to work.

@jrowberg glad to hear it. Hopefully, this weekend I'll have some time to fix this properly if @craigbarratt doesn't beat me to it.

Another thanks to @dlashua! I tried the temporary fix, as well. Seems to work :-)

Most of my pyscript services were missing after the update --> they are now back
All persisted states were missing after the update --> they are now back

Note: My persisted states all were on "unknown" after the HASS restart. This is my default value from state.persist. That's why I triggered some state changes manually to get the states up to date again. But I guess I could have just waited some longer for the actual state changes.^^


Security bug in SMF present in Elk as well

I'm referring to:
https://github.com/SimpleMachines/SMF2.1/issues/5534
It's a bug present in 2.0 as well, so quite a long story, nothing new.
Basically: any reply to a moderated topic, posted by someone with permission to approve posts is visible in action=recent and in the search by everybody.
Since it has been disclosed already it's pointless to keep it hidden.
Steps to reproduce:

have post moderation active,
create a new topic with a user that is post-moderated (the topic will be unapproved),
login with an account that has permission to post without moderation (e.g. admin),
reply to the topic,
logout,
go to action=recent and see the message posted by the admin.

Workaround: if you have post-moderation active, instruct your moderators to use the "unapprove" button on each reply done to a moderated topic.
This way all the posts will be unapproved and visible only to the people that have permissions to view unapproved posts.
To verify the checkbox "Approve this post" doesn't seem to be honored if not checked (i.e. if you want to post the message directly in unapproved status you cannot).

I knew about if for a while now noticed long time ago when i would see hidden posts from moderated topics. figured time to post it for the next versions.
keep in mind affects search as well and SSI

TBH, being that technically a security bug (at least IIRC similar things were considered security issues, exposing data that are not supposed to be exposed) probably a security report "a long time ago" would have been better. :slightly_smiling_face:
Anyway, thanks for reporting it!
And not to butt into SMF matters, but probably this should be fixed in 2.0 as well (but maybe a new release is already planned).


Feature/trading capacities

We need the local and remote trading capacities for all active, pending and inactive channels in order to create the trading statuses table for the CLI.

 Trading Status for Brokers


separate sudo and user capabilities

currently the script requires sudo from a nonroot user.
It should separate distinct permissions options. Some use cases:

packages for system, configs for user
no access to root -- packages for user if possible, configs for user regardless, list packages already installed / just installed / unable to install
packages, configs, and manual packages for system; clean config for user?

deploy a virtual machine

NixOS

chroot

Windows Subsystem for Linux

VMware/VirtualBox ISO

deploy SSH keys for git


Bower support

Notes

Helps with #197 (but you'll still have to register your package).
You might want to change the keywords.
normalize.css and Raleway are included in index.html, but aren't part dependencies of the Bower package. You might want to add css/normalize.css to the main field or add normalize.css as a dependency if you want people to always use normalize with Skeleton (you'll still need to remind users to include the CSS file though). Also, you might want to add a note that people can get Raleway from Google Fonts.

No problem.
I recommend adding normalize as a dependency. While it is already in the repository, it's easier for someone to update their version of normalize without having to wait for Skeleton's version to be updated. Besides, Skeleton probably doesn't need any specific version of Normalize, and Bower might let you have optional dependencies.


feat(app): field service groups connected to indexedDb

:tada: This PR is included in version 2.130.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


GGUF

Can you add an additional type like GGUF, the successor to GGML
Thanks

tracking this in #568! will close this issue and defer to there.


[T7][W10-C4]

I have no further comments on the user guide. Try to get the functionality up sooner so that you can use it yourself and figure out how to fine tune UI and command format further.


Create task section

Create it according to figma design

I am working on this!!

@Bhavik-ag  updates?


QOQaCLI add soldout info

add the information about soldout -> if not soldout -> display info about gauge level

closed by #3


Consider moving the OpenIddict YesSql stores to a separate project

https://github.com/OrchardCMS/Orchard2/issues/233#issuecomment-265261317
@jersiovic do you think it would be worth moving the YesSql to an "official" OpenIddict.YesSql package? If so, I'd like to include it in the upcoming beta2 release.

I think YesSQL still is evolving at least I'm not sure collections feature its on its final version (@sebastienros is who can give info related with that). So, if it evolves we will need you maintain it. Maybe it is better you wait till collections feature is on its final version. That's the only con I see.
Related to pros, maybe someone is interested on using Openiddict with YesSQL instead of EF, and by the way this is good way of promoting the use of YesSQL by people not interested in Orchard.

Or it would make sense in order to be able to use something else than YesSql, in case you would only use the SaaS part of ORchard, meaning without CMS modules.

I see, but do we will need to replace current dependency on YesSql per sth more generic moving its interfaces like ISession to an abstractions package non related to YesSql? Or do you mean that those who want to be able to use something else should offer alternate implementations of YesSql interfaces?

So, if it evolves we will need you maintain it.

I'm fine with that.

Maybe it is better you wait till collections feature is on its final version.

@sebastienros do you have an approximate date for that? :smile:

@sebastienros I said that I believe collections in yessql are not on its final version and when it will be ready  it will impact on Orchard OpenId stores because of this issue: https://github.com/OrchardCMS/Orchard2/issues/233 there you say you wanted to change YesSql to use configuration to set which class uses which collection, but right now we need to use a using statement.


Question on reliable UDP if the last message is lost.

Hello,
I have tried to use Aeron and everything looks good. Therefore plan to start to work on production.
However I have one question that relate to reliable UDP.
As I look at the document, it will do a NAK and ask for resend when there is missing message.
But the example is all about when you get at least one message and then can deduce what have been missed.
What if it is the last and only message at one hour interval and then it lost, will Aeron also able to detect this message have been lost and resend it?
I want to see if I need to manually handle this case. I have checked LossDetector class and maybe I have overlook something.
Thanks you very much.

Look on the publication side for heartbeats sent in the absence of data.

thanks you


Add Mafengwo to karmada adopters

Signed-off-by: hansedong<EMAIL_ADDRESS>What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:

Hi Hongshuai @hansedong Thank you very much! That means a lot to the community.
/lgtm
/approve


Tutorials Tests/Checks FAIL (dapr app not ready / timing)

Expected Behavior
Tutorials pass
Tutorials are technically unsupported parts of the Quickstart branch, but I don't like seeing regressions creep in like this and treat it as a P1, non ship blocker, non PR blocker that we should investigate.  It likely points to a real product issue.
Actual Behavior
Tutorials Fail
Here is a good example where the dapr app client tries to call the target dapr app's /neworder api and it fails because it's not yet available, likely because it hasn't started yet.  I can reproduce timing issues like this on my local KinD deployment too.
Failure example
Steps to Reproduce the Problem
Use the link to Action above.
But more importantly, it repros when you do a multi app run of the app on your local machine, e.g. using
dapr run -k -f .

@msfussell
@yaron2

Might need maintainer on dapr-cli to look at this since it feels more like health check issues for multi app run on kubernetes.

@yaron2 you mentioned you may be able to help with this dapr cli, multi app run health check timing?  TY

@paulyuk The failure seems to be transient and more related to why dapr-dev-redis was going into crashloop backof https://github.com/dapr/quickstarts/actions/runs/9702023201/job/26776912327#step:18:82...
Are you able to repro this locally?

Hey @mukundansundar and @yaron2, the issue I reported above repros locally.
the redis crash loop does not repro for me, using KinD
dapr init -k --dev
⌛  Making the jump to hyperspace...
ℹ  Note: To install Dapr using Helm, see here: https://docs.dapr.io/getting-started/install-dapr-kubernetes/#install-with-helm-advanced

ℹ  Container images will be pulled from Docker Hub
✅  Deploying the Dapr control plane with latest version to your cluster...
✅  Deploying the Dapr dashboard with latest version to your cluster...
✅  Deploying the Dapr Redis with latest version to your cluster...
✅  Deploying the Dapr Zipkin with latest version to your cluster...
ℹ  Applying "statestore" component to Kubernetes "default" namespace.
ℹ  Applying "pubsub" component to Kubernetes "default" namespace.
ℹ  Applying "appconfig" zipkin configuration to Kubernetes "default" namespace.
✅  Success! Dapr has been installed to namespace dapr-system. To verify, run `dapr status -k' in your terminal. To get started, go here: https://aka.ms/dapr-getting-started

> kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
dapr-dev-redis-master-0            1/1     Running   0          24m
dapr-dev-redis-replicas-0          1/1     Running   0          24m
dapr-dev-redis-replicas-1          1/1     Running   0          23m
dapr-dev-redis-replicas-2          1/1     Running   0          23m
dapr-dev-zipkin-7d5f8fc8b5-wds69   1/1     Running   0          24m

But the original crash above of Node app not loading with dapr run -k -f . is still happening locally.  Have you tried it?
 pyadmin   hello-kubernetes    master ≢  ?2    dapr init -k --dev
⌛  Making the jump to hyperspace...
ℹ  Note: To install Dapr using Helm, see here: https://docs.dapr.io/getting-started/install-dapr-kubernetes/#install-with-helm-advanced

ℹ  Container images will be pulled from Docker Hub
✅  Deploying the Dapr control plane with latest version to your cluster...
✅  Deploying the Dapr dashboard with latest version to your cluster...
✅  Deploying the Dapr Redis with latest version to your cluster...
✅  Deploying the Dapr Zipkin with latest version to your cluster...
ℹ  Applying "statestore" component to Kubernetes "default" namespace.
ℹ  Applying "pubsub" component to Kubernetes "default" namespace.
ℹ  Applying "appconfig" zipkin configuration to Kubernetes "default" namespace.
✅  Success! Dapr has been installed to namespace dapr-system. To verify, run `dapr status -k' in your terminal. To get started, go here: https://aka.ms/dapr-getting-started
 pyadmin   hello-kubernetes    master ≢  ?2    dapr run -k -f dapr.yaml
ℹ  This is a preview feature and subject to change in future releases.
ℹ  Validating config and starting app "nodeapp"
ℹ  Deploying app "nodeapp" to Kubernetes
ℹ  Deploying service YAML "/home/pyadmin/src/paulyuk/quickstarts/tutorials/hello-kubernetes/node/.dapr/deploy/service.yaml" to Kubernetes
ℹ  Deploying deployment YAML "/home/pyadmin/src/paulyuk/quickstarts/tutorials/hello-kubernetes/node/.dapr/deploy/deployment.yaml" to Kubernetes
⚠  Error deploying pod to Kubernetes. See logs directly from Kubernetes command line.
ℹ  Writing log files to directory : /home/pyadmin/src/paulyuk/quickstarts/tutorials/hello-kubernetes/node/.dapr/logs
ℹ  Validating config and starting app "pythonapp"
ℹ  Deploying app "pythonapp" to Kubernetes
ℹ  Deploying deployment YAML "/home/pyadmin/src/paulyuk/quickstarts/tutorials/hello-kubernetes/python/.dapr/deploy/deployment.yaml" to Kubernetes
ℹ  Streaming logs for containers in pod "pythonapp-5cd765b8f4-zgqlm"
ℹ  Writing log files to directory : /home/pyadmin/src/paulyuk/quickstarts/tutorials/hello-kubernetes/python/.dapr/logs
ℹ  Starting to monitor Kubernetes pods for deletion.
== APP - pythonapp == HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e5c485f50>: Failed to establish a new connection: [Errno 111] Connection refused'))
== APP - pythonapp == HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e5c422150>: Failed to establish a new connection: [Errno 111] Connection refused'))
== APP - pythonapp == HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e5c4282d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
== APP - pythonapp == HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e5c4283d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
== APP - pythonapp == HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e5c4220d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
== APP - pythonapp == HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e5c415cd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
== APP - pythonapp == HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e5c432450>: Failed to establish a new connection: [Errno 111] Connection refused'))
== APP - pythonapp == HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9e5c4385d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
== APP - pythonapp == HTTP 500 => {"errorCode":"ERR_DIRECT_INVOKE","message":"failed to invoke, id: nodeapp, err: failed to invoke target nodeapp after 3 retries. Error: rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp <IP_ADDRESS>:50002
...

Note @greenie-msft and I also tried to put Dapr Resiliency in place, and Orders still got skipped until the NodeApp was ready!
We added ./resources/resiliency.yaml:
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
  name: myresiliency
# similar to subscription and configuration specs, scopes lists the Dapr App IDs that this
# resiliency spec can be used by.
spec:
  # policies is where timeouts, retries and circuit breaker policies are defined.
  # each is given a name so they can be referred to from the targets section in the resiliency spec.
  policies:
    # retries are named templates for retry configurations and are instantiated for life of the operation.
    retries:
      retryInvokeForever:
        policy: constant
        maxInterval: 5s
        maxRetries: -1 # retry indefinitely

  # targets are what named policies are applied to. Dapr supports 3 target types - apps, components and actors
  targets:
    apps:
      nodeapp:
        retry: retryInvokeForever

and modified dapr.yaml common section to apply the resiliency policy crd
version: 1
apps:
  - appDirPath: ./node
    appID: nodeapp
    appPort: 3000
    containerImage: ghcr.io/dapr/samples/hello-k8s-node:latest
    createService: true
  - appDirPath: ./python
    appID: pythonapp
    containerImage: ghcr.io/dapr/samples/hello-k8s-python:latest
common: # optional section for variables shared across apps
  resourcesPath: ./resources # any dapr resources to be shared across apps

This is almost as if this type of exception (Max retries exceeded with url: /neworder ) does not factor into dapr resiliency.  Unless the try catch for the sample is interfering?

The issue here is that the app can't reach it's Dspr instance because the latter isn't up yet. This means the resiliency policy is ineffective as the requests never hit Dspr to begin with

I'm investigating

Thank you.  It put my brain in a loop thinking about how your app's dapr sidecar can check on the other app's sidecar, if it's not their, or you dont know the healthz endpoint you're trying to hit for remote app.

Thanks for the explanation, Yaron. So the retries we're seeing in the logs are coming from python http client?

@yaron2 @msfussell - per our chat the tactical solution in 1.14 for this is will revert back to single dapr run -- ... commands since Multiapp run doesn't know about dependencies and ordering priorities that affect timing.  I filed a separate issue requesting support for that here <https://github.com/dapr/cli/issues/1435 >
I will take ownership of the 1.14 scoped fix so we get tests passing again.  PR on the way.

@paulyuk The failure seems to be transient and more related to why dapr-dev-redis was going into crashloop backof https://github.com/dapr/quickstarts/actions/runs/9702023201/job/26776912327#step:18:82... Are you able to repro this locally?

Hey - I am hitting this issue now, and it is blocking tests with --dev init, but it will not occur locally.  It's only in the GH action runner.

fixed by #1057 and https://github.com/dapr/cli/pull/1437