x/tools/go/ssa: generics support

This is a tracking issue for supporting generics within the x/tools/go/ssa package. This depends heavily on proposals for "go/ast" https://github.com/golang/go/issues/47781 and "go/types" https://github.com/golang/go/issues/47916.
A reasonable goal is to have ssa available at roughly the same time as the release of generics so tool authors can take advantage of it. Earlier so tool authors that build on top of ssa can support generics at the same time as the release would be even better.

There are two relevant contexts we need to consider:

Whole program tools and packages like x/tools/go/callgraph/{...} that will see the original source and all of the possible instantiation sites.
x/tools/go/analysis/passes/buildssa and similar passes that can have access to just the exported type data for imported packages. ATM in ssa, these are primarily used to build stub functions for functions in the imported packages.

My current idea is to "monomorphize" or "stencil" the instances of the generics functions. This would create a copy of the function specialized for each type instantiation.
Cons:

More copies of the function.
Further from the source than an uninstantiated version.
Pros:
Simple for the ssa package to implement.
Expect it to be relatively simple for ssa users to adjust their existing checkers/analysis.

As for whether there will be too many copies of functions with this approach, it is not yet clear. I am somewhat optimistic it will not. For whole program analysis, this is roughly k=1 context-sensitivity in all of instantiation locations (which strikes me as a reasonable thing to try anyways).
For incremental program analysis (like buildssa), ssa needs to at least provide a stub for generic functions defined in other packages and locally instantiated. Providing an empty stub for each instantiation won't be too much blowup. Whether such empty stubs would be sufficient for ssa users is not yet clear.

I think that stencilling has the advantage of preserving the signatures of the ast  whereas dictionaries or gcshape stencilling would have fewer copies and be more aligned with the current compiler implementation.   However, these are questions of implementation of instantiation whilst the API for using ssa with generics is still open.

For incremental program analysis (like buildssa), ssa needs to at least provide a stub for generic functions defined in other packages and locally instantiated. Providing an empty stub for each instantiation won't be too much blowup. Whether such empty stubs would be sufficient for ssa users is not yet clear.

I am also not sure whether empty stubs would be sufficient.  The ssa interpreter comes to mind as one which it seems would require instantiations, the pointer package may be more easily adapted with instantiations (it is based on a notion of type size which would become variable where it is not now).
Are there thoughts about providing an instantiation API to users of SSA vs instantiating ast/types and using that at ssa construction time?   I would be inclined to explore using the ast/types instantiation APIs to provide an instantiation API on ssa.

For incremental program analysis (like buildssa), ssa needs to at least provide a stub for generic functions defined in other packages and locally instantiated. Providing an empty stub for each instantiation won't be too much blowup. Whether such empty stubs would be sufficient for ssa users is not yet clear.

I am also not sure whether empty stubs would be sufficient. The ssa interpreter comes to mind as one which it seems would require code instantiations, the pointer package may be more easily adapted with code instantiations (it is based on a notion of type size which would become variable where it is not now). Maybe "ssa.value instantiations" instead of code instantiations could be an alternative.

Ah wait, the interpreter is whole-program.  Not so sure about the pointer package, it is in one sense whole-program whereas in another it can be applied to libraries.

Not so sure about the pointer package, it is in one sense whole-program whereas in another it can be applied to libraries.

x/tools/go/pointer falls into the whole-program paradigm.
Zooming out a bit. If we have an instantiation "p.F[local]" within a package "q", the only stubs solution for buildssa does means we would never see an instance of "p.F[local]". This would create some holes for analysis: like incremental pointer analysis, "is there a reachable instruction that modifies this field", etc.

the pointer package may be more easily adapted with code instantiations (it is based on a notion of type size which would become variable where it is not now

Non-instantiated generic functions [if we choose to have them] do not need to be considered reachable/real code. So tools could dodge this by skipping them. (Not in love with this, but I think it would be okay?)

Are there thoughts about providing an instantiation API to users of SSA vs instantiating ast/types and using that at ssa construction time?

@scott-cotton I don't think I understand the question well enough to answer. Can you give more details?

Are there thoughts about providing an instantiation API to users of SSA vs instantiating ast/types and using that at ssa construction time?

@scott-cotton I don't think I understand the question well enough to answer. Can you give more details?

The intuition I have is to explore making ssa instantiable for consumers of ssa.  Something perhaps (very roughly) like ssa.Instantiate(v ssa.Value, ty1, ty2,...) which would do the stencilling (or whatever implementation).    Not sure whether you are considering this, but if it is a desirable thing to provide then implementing that may also serve buildssa, whereas if the instantiations are hidden in buildssa, built by instantiating on the ast and then produce the ssa, then providing this seems like it would be harder.  This idea is just an intuition I have, it is not really fleshed out...

Not so sure about the pointer package, it is in one sense whole-program whereas in another it can be applied to libraries.

x/tools/go/pointer falls into the whole-program paradigm.
Zooming out a bit. If we have an instantiation "p.F[local]" within a package "q", the only stubs solution for buildssa does means we would never see an instance of "p.F[local]". This would create some holes for analysis: like incremental pointer analysis, "is there a reachable instruction that modifies this field", etc.

Regarding whole program vs incremental, I think it will help to clarify some things, so we are talking about the same properties:

Is whole program analysing behaviour only reachable from main entry points or does it also include having the whole source but no entry-points?  For example, a package with no mains and all its dependants -- is it whole program or not?
Does incremental mean bottom-up (depended upon first, like printf checker detecting and propagating wrappers) or where units of analysis are piece-wise independent (like prove in the compiler on a loop)?

On point 1), godoc -analysis=pointer works on library code and uses x/tools/go/pointer
while in principal the basis of the analysis is only valid for whole program in terms of reachable code from a given set of entry points.   It is run anyway and provides information.
Yes I agree the buildssa stubs approach would create holes in what you are calling incremental analysis.  I also agree that stubs may be preferable for other tools using ssa from buildssa on non-main programs.   That combination of things may also make it interesting to consider an instantiation api as noted in the previous comment: with such a thing callers could decide what to instantiate and if and when.

Something perhaps (very roughly) like ssa.Instantiate(v ssa.Value, ty1, ty2,...) which would do the stencilling (or whatever implementation).

I think it might be possible to provide this as a feature for *ssa.Function, and *ssa.Type. Arbitrary Values will result in unconnected instructions. I do think it would be helpful to have a clear use case before expanding the API with this though. SSA's API tends not to expose much for users to modify. A convincing case would be something that (1) comes up in practice and (2) is very hard to otherwise support. I think we can wait on this as a feature.

Regarding whole program vs incremental, I think it will help to clarify some things, so we are talking about the same properties:

The distinction I am really interested in is:

All packages.Packages used for building are transitively loaded and built. This is what I was somewhat inaccurately calling wholeprogram.
The current packages.Package is available for 1 package and gcexportdata is available for direct imports (and supports go/types.Importer). This is what I'm referring to as incremental.

godoc -analysis=pointer would fall into the first bucket. buildssa falls into the second bucket.

That combination of things may also make it interesting to consider an instantiation api as noted in the previous comment: with such a thing callers could decide what to instantiate and if and when.

For buildssa, the source[/ast] needed for generation from a direct[/transitive] import is not immediately available. What would we stencil from? It needs to be serialized somewhere in the on disk cache. This is solvable in a variety of ways, but it not currently supported.
A practical implication of this is that we currently expect that in a different package from a generic function, one will only see the stub for an instantiation "math.Max[int]". If "math" did not do this specific instantiation, "math.Max[int]", there will be no instantiated body for "math.Max[int]" at any point available to an analysis.Analyzer. These Analyzers would be expected to summarize "Max" on the Pass looking at "math" and apply this summary/Fact on the instantiation. These are going to be more challenging to write as one would need to deal with type parameters.

To reduce the scope of work, you might want to consider abandoning go/ssa/interp.

TBH go/ssa/interp has proven to be massively useful for testing these changes. After this effort, there might be a 4-5 year stretch of it not being too helpful after this. But for the moment it is too valuable for development to not support it.
As for go/pointer, if we monomorphize generics, this package should "just work". Maybe some minor hiccups? If we 'interpret type parameters' [somehow], that going to require many changes to go/pointer. Similarly if we suggest an instantiation type and monomorphize w.r.t. the suggested type, users will need to understand how to connect the pieces. This would have a big impact on go/pointer. The problems caused by not having an interpretation for uninstantiated type params goes beyond go/pointer though. It'll impact anyone trying to use ssa on generics.

any update here? Since Go 1.18 has been released linting is failing on my side.
ERRO [runner] Panic: buildssa: package "heartbeat" (isInitialPkg: true, needAnalyzeSource: true): t: goroutine 8469 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/golangci/golangci-lint/pkg/golinters/goanalysis.(*action).analyzeSafe.func1()
	github.com/golangci/golangci-lint/pkg/golinters/goanalysis/runner_action.go:101 +0x155
panic({0x1b84e00, 0xc00018bf20})
	runtime/panic.go:838 +0x207
golang.org/x/tools/go/ssa.(*Program).needMethods(0xc009952b60, {0x1e267e8?, 0xc00018bf20?}, 0x0)
	golang.org/x/tools@v0.1.10/go/ssa/methods.go:237 +0x5b1
golang.org/x/tools/go/ssa.(*Program).needMethods(0xc009952b60, {0x1e267c0?, 0xc0033c4738?}, 0x0)
	golang.org/x/tools@v0.1.10/go/ssa/methods.go:233 +0x708
golang.org/x/tools/go/ssa.(*Program).needMethods(0xc009952b60, {0x1e26720?, 0xc003655300?}, 0x0)
	golang.org/x/tools@v0.1.10/go/ssa/methods.go:209 +0x448
golang.org/x/tools/go/ssa.(*Program).needMethodsOf(0xc009952b60, {0x1e26720?, 0xc003655300?})
	golang.org/x/tools@v0.1.10/go/ssa/methods.go:145 +0x70
golang.org/x/tools/go/ssa.(*Package).build(0xc00737cb40)
	golang.org/x/tools@v0.1.10/go/ssa/builder.go:2284 +0x111
sync.(*Once).doSlow(0xc009952b60?, 0xc004120870?)
	sync/once.go:68 +0xc2
sync.(*Once).Do(...)
	sync/once.go:59
golang.org/x/tools/go/ssa.(*Package).Build(...)
	golang.org/x/tools@v0.1.10/go/ssa/builder.go:2272
golang.org/x/tools/go/analysis/passes/buildssa.run(0xc009952a90)
	golang.org/x/tools@v0.1.10/go/analysis/passes/buildssa/buildssa.go:72 +0x2ee
github.com/golangci/golangci-lint/pkg/golinters/goanalysis.(*action).analyze(0xc00290d330)
	github.com/golangci/golangci-lint/pkg/golinters/goanalysis/runner_action.go:187 +0x9c4
github.com/golangci/golangci-lint/pkg/golinters/goanalysis.(*action).analyzeSafe.func2()
	github.com/golangci/golangci-lint/pkg/golinters/goanalysis/runner_action.go:105 +0x1d
github.com/golangci/golangci-lint/pkg/timeutils.(*Stopwatch).TrackStage(0xc001171630, {0x1bee783, 0x8}, 0xc001d62f48)
	github.com/golangci/golangci-lint/pkg/timeutils/stopwatch.go:111 +0x4a
github.com/golangci/golangci-lint/pkg/golinters/goanalysis.(*action).analyzeSafe(0xc001d8da40?)
	github.com/golangci/golangci-lint/pkg/golinters/goanalysis/runner_action.go:104 +0x85
github.com/golangci/golangci-lint/pkg/golinters/goanalysis.(*loadingPackage).analyze.func2(0xc00290d330)
	github.com/golangci/golangci-lint/pkg/golinters/goanalysis/runner_loadingpackage.go:80 +0xb4
created by github.com/golangci/golangci-lint/pkg/golinters/goanalysis.(*loadingPackage).analyze
	github.com/golangci/golangci-lint/pkg/golinters/goanalysis/runner_loadingpackage.go:75 +0x1eb
WARN [runner] Can't run linter goanalysis_metalinter: goanalysis_metalinter: buildssa: package "heartbeat" (isInitialPkg: true, needAnalyzeSource: true): t
ERRO Running error: 1 error occurred:
	* can't run linter goanalysis_metalinter: goanalysis_metalinter: buildssa: package "heartbeat" (isInitialPkg: true, needAnalyzeSource: true): t

make: *** [lint] Error 3

You should upgrade the linter. It is being worked on.

Milestone is set to Go 1.19, will this ever be fixed in Go 1.18.x?

@sonatard the go/ssa package is in x/tools, not the Go standard library.
We are in the process of reviewing @timothy-king's implementation, but it will take some time (most likely ~weeks).

The extraction pipeline of  golang.org/x/text/cmd/gotext is also affected. Sorry for sounding rude, but I think this should have been a release blocker, even if it's not part of the Go repository itself.

We felt that it would take time for people to write generic code where this change mattered.  We felt that people who rely on tools that use the ssa package would have the option of not writing generic code until the tools were updated.  Since this change is not tied to the release cycle, we felt that it would be OK to complete the work even after the release.
Sorry for the trouble.  There was no perfect decision here.

@tux21b, I am unaware of a problem with gotext. I'd be happy to take a look at that but I don't see an open issue for it? A reproducer would be great. Please CC me on any new issue. Thanks!

@tux21b, I am unaware of a problem with gotext. I'd be happy to take a look at that but I don't see an open issue for it? A reproducer would be great. Please CC me on any new issue. Thanks!

@rsc Sure. I have created two issues:
https://github.com/golang/go/issues/52552
https://github.com/golang/go/issues/52553
Many thanks for looking into it :)

Status Update – 2022-05-16
x/tools/go/ssa is expected to not crash on code containing generics. If you encounter a panic within the ssa package or any package in x/tools that uses ssa, please file a new issue on the issue tracker.
The SSA emitted for non-generic functions will remain backwards compatible with the previously emitted SSA. A call from a non-generic function to a generic function will contain a Call to an *ssa.Function that is an instantiation of the generic function with an appropriate signature for the given type arguments. Instantiations do not appear within the x/tools/go/analysis/passes/buildssa.SSA.SrcFuncs field.
For the time being, SSA represents source functions and methods that have type parameters as functions with empty bodies. (This behavior is expected to change in a future update. See remaining work)
The ssa.BuilderMode flag ssa.InstantiateGenerics controls how ssa builds generic function instantiations.

ssa.InstantiateGenerics is off (default):  Instantiations have empty bodies. (Expected to change.)
ssa.InstantiateGenerics is on: calls to generic functions are instantiated by concrete type arguments. These instances will build a specialized body from the original syntax and the provided type arguments. This instantiation happens exhaustively. This results in a complete function body instantiation for every instantiation that is required in the ssa.Program. This may be expensive. This mode is likely to be appropriate for whole program analysis or analysis that requires knowing all runtime types to be correct. For “modular” analysis with x/tools/go/analysis, this will only be able to instantiate functions whose bodies are available (i.e. instantiations of generic functions defined in separate package will have empty bodies during compositional analysis).

Tool developers that use ssa will need to decide which mode works best. Our experience with updating ssa users within x/tools (https://github.com/golang/go/issues/52504) has been that usage of SSA with generics has been backwards compatible once the appropriate ssa.BuilderMode is selected.
Remaining work – 2022-05-16
This issue is not yet resolved. Here is the expected ongoing work for this issue.

We are extending compilation to create a body for generic functions from the AST regardless of the BuilderMode. There will be at least one generic function per source function. Details for how this will be achieved are still being determined. This mode is intended to support analysis from x/tools/go/analysis.
Create bodies for instantiations when ssa.InstantiateGenerics is off.
We will create a proposal for the public interface changes for SSA. This is expected to include:

New Instructions/Values if any [largely dependent on previous steps]
Non-backwards compatible updates to existing Instructions
Exposing new fields for navigating generic functions [e.g. Function._Origin -> Function.Origin ]
Exposing new utility functions for navigating from a source function to instantiations

Determine the long term BuilderMode for x/tools/go/analysis/passes/buildssa.Analyzer.

Thank you everyone for your patience while we roll out updates to this library.

This is currently in the 1.19 milestone.  Anything further for 1.19?  Should the milestone move to 1.20?  Thanks.

Status Update - 2022-07-20
I have a prototype that creates a body for each generic function from the AST regardless of the BuilderMode. This covers points 1 and 2 from https://github.com/golang/go/issues/48525#issuecomment-1128385150. The prototype is able to build the compiler's tests in test/typeparam. The prototype still requires review and more extensive testing, but this is a significant milestone. I expect to make the changes publicly available for review and submit a proposal for the API changes in August.
Thank you for your continued patience.

Status Update - 2022-08-31
We are pleased to share a candidate implementation of generic types and functions in go/ssa. We are currently soliciting feedback, though we expect that it is close to its final state and hope to submit it in September, at which point we will close this issue.
API changes:

For generic functions and methods produced from syntax trees, we build a generic ssa.Function. Type parameters, and types containing them, may now appear as the operands and results of ssa.Instructions. For example, a hypothetical generic min[T] function would contain an instruction corresponding to the binary operation T < T.
Instances of generic functions may be created in one of two forms, determined by the InstantiateGenerics builder mode flag. (The flag has the same behavior as before this update.)

When InstantiateGenerics is disabled (the default), each instantiation is materialized as a thin wrapper that delegates to the generic function. For example, given a hypothetical generic slices.Reverse[T] function, the instance Reverse[int] would be a wrapper that calls Reverse[T], coercing its argument from []int to []T and its result from []T to []int.
When InstantiateGenerics is enabled, each fully instantiated instance is expanded and specialized for its particular types. In this mode, Reverse[int] would contain the complete logic of slice reversal specialized to integers. Partial instantiations are still materialized as wrappers. For example, a generic method TreeMap[K,V].Contains may be partially instantiated to a wrapper TreeMap[string, V].Contains while TreeMap[string, int].Contains is fully expanded and specialized.

The ChangeType instruction can now represent a coercion to or from a parameterized type to an instance of the type.
Const values can now represent zero values of any type, including structs, arrays, and type parameters.
go/analysis/passes/buildssa builds with the default build mode. (ssa.InstantiateGenerics is off.)
ssa.Function has three new methods to help navigate generic functions and instantiations:

TypeParams, which returns the function’s list of type parameters;
TypeArgs, which returns the instantiations of the function’s type parameters;
Origin, which returns the generic function, given one of its instantiations.

go/pointer and go/ssa/interp require ssa.InstantiateGenerics to be enabled, and there are no plans to support these with ssa.InstantiateGenerics off. We are still assessing whether to update the packages in go/callgraph InstantiateGenerics off.
Please let us know if you have any feedback on the interface changes. Feedback on the implementation may be left on Gerrit.
If you are interested in testing this, please patch in https://go-review.git.corp.google.com/c/tools/+/425496. If you encounter bugs, please update this issue.

Congratulations all on the progress, I know this has been an enormous undertaking!
I expect that most interested parties have probably already chimed in, but given that the API has stabilized perhaps now is a good time to promote this issue to a proposal. The proposal committee may choose to simply accept the existing discussion as complete, but we do have a policy that all API changes should have an associated proposal. Having this in the proposal meeting notes will also elevate visibility.

Status update I posted on https://github.com/golang/go/issues/54984#issuecomment-1320369949

There are no known bugs with go/ssa over Google's internal Go code base with BuilderMode(0).


switch queue tie-break to first LGTM time

Fixes #1286

Also fyi @krousey, current build cop.

Changes pushed, PTAL. Would like to get this in asap.

Skipping hack/../hack/verify-all.sh
!!! 'gofmt -s' needs to be run on the following files:
./mungegithub/mungers/submit-queue_test.go
!!! 'golint' problems:
/home/travis/gopath/src/k8s.io/contrib/mungegithub/mungers/mungerutil/time_cache.go:39:1: exported function NewLabelTimeCache should have comment or be unexported
/home/travis/gopath/src/k8s.io/contrib/mungegithub/mungers/mungerutil/time_cache.go:46:1: exported method LabelTimeCache.FirstLabelTime should have comment or be unexported

Fix that and feel free to self LGTM

applying label. Will push when this merges.


Constant folding crashes for values of different types

Constant folding enables the DAPHNE compiler to calculate many operations on constants already at compile-time. The respective code can be found in src/ir/daphneir/DaphneDialect.cpp.
For instance, in the following DaphneDSL script, constant folding evaluates 1 * 1 at compile-time, as can be seen when looking at the IR after parsing (no constant folding yet) and some initial simplifications (including constant folding) (use --explain parsing,parsing_simplified):
print(1 * 1);

IR after parsing:
module {
  func.func @main() {
    %0 = "daphne.constant"() {value = 1 : si64} : () -> si64
    %1 = "daphne.constant"() {value = 1 : si64} : () -> si64
    %2 = "daphne.ewMul"(%0, %1) : (si64, si64) -> si64
    %3 = "daphne.constant"() {value = true} : () -> i1
    %4 = "daphne.constant"() {value = false} : () -> i1
    "daphne.print"(%2, %3, %4) : (si64, i1, i1) -> ()
    "daphne.return"() : () -> ()
  }
}
IR after parsing and some simplifications:
module {
  func.func @main() {
    %0 = "daphne.constant"() {value = 1 : si64} : () -> si64
    %1 = "daphne.constant"() {value = false} : () -> i1
    %2 = "daphne.constant"() {value = true} : () -> i1
    "daphne.print"(%0, %2, %1) : (si64, i1, i1) -> ()
    "daphne.return"() : () -> ()
  }
}
1

However, currently, constant folding requires the operands to have the same type. For instance, the following two DaphneDSL scripts would be perfectly amenable to constant folding, but make the DAPHNE compiler crash with the respective error messages:

print(1 * as.si32(1));

daphne: /daphne/thirdparty/llvm-project/llvm/lib/Support/APInt.cpp:226: llvm::APInt llvm::APInt::operator*(const llvm::APInt &) const: Assertion `BitWidth == RHS.BitWidth && "Bit widths must be the same"' failed.
...
Segmentation fault (core dumped)

print(1.0 * as.f32(1.0));

daphne: /daphne/thirdparty/llvm-project/llvm/lib/Support/APFloat.cpp:1129: llvm::lostFraction llvm::detail::IEEEFloat::multiplySignificand(const llvm::detail::IEEEFloat &, llvm::detail::IEEEFloat): Assertion `semantics == rhs.semantics' failed.
...
Segmentation fault (core dumped)

A possible solution could be to apply type promotion just like the DAPHNE compiler would do for runtime kernels.

Hi, please assign me to this issue.

Hi, please assign me to this issue.

Sure, go ahead :)


When the usage and health analytics will be available again? Thanks

Usage and health analytics are currently unavailable for Smart Home actions.
This is very inconvenient.
Could you provide the date please.
Thanks

Duplicate of #349


Update content with Service Endpoint video

Update content with Service Endpoint video

Hi @KumudD - Could you review this PR? If you approve the update to the landing page, we can move the commit to the private repo for validation on staging. Thanks.

@ranand12 Please can you add the video link within the Secure network traffic card instead. Thanks!

@KumudD - can you take a look? I moved the video link ... thanks.

@Jak-MS Looks good.  Please merge. Thanks!


ERROR Unable to load component jk_bms_ble.number:

Hello, I'm not new to ESP, Arduino and so on but new to ESPhome. So I don`t understand how to work with it. But I got some simple samples working. But not the esp32-ble-example.yaml.
What I did:
Installed HomeAssistance with ESPhome (have it on ioBroker as well) put the RAW esp32-ble-example.yaml in a new ESPhome node, edit some values to my needs and clicked install then got the error "Unable to load component jk_bms_ble.number" I tried it several times and tested around also with api and mqtt, esp-idf and arduino.
So what I did wrong?
# Enable logging
logger:

# Enable Home Assistant API
#api:
#  encryption:
#    key: "xxxxxxxx"

ota:
  password: "ec30....................."

wifi:
  ssid: "xxxxxxx"
  password: "xxxxxxxxx"

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Jk-Bms Fallback Hotspot"
    password: "c38BzekXLFWE"

#captive_portal:

substitutions:
  name: jk-bms
  device_description: "Monitor and control a JK-BMS via bluetooth"
  external_components_source: github://syssi/esphome-jk-bms@main
  mac_address: C8:47:8C:E1:E2:E1
  # Defaults to "JK02". Please use "JK04" if you have some old JK-BMS version (f.e. JK-B2A16S hw 3.0, sw. 3.3.0)
  protocol_version: JK02

esphome:
  name: ${name}
  comment: ${device_description}
  project:
    name: "syssi.esphome-jk-bms"
    version: 1.1.0

esp32:
  board: esp32dev
  framework:
#  The arduino platform crashs
    type: arduino
    version: latest
#    type: esp-idf
#    version: latest

external_components:
  - source: ${external_components_source}
    refresh: 0s

# This feature is only available with frameworks ['arduino'] <= 2022.3.1
# https://github.com/esphome/esphome/pull/2930: Add mqtt for idf >= 2022.4.0
#
mqtt:
  broker: <IP_ADDRESS>
  username: "xxxxxx"
  password: "xxxxxx"
  port: 1893
  id: "mqtt_client"

esp32_ble_tracker:

ble_client:
  - mac_address: ${mac_address}
    id: client0

jk_bms_ble:
  - ble_client_id: client0
    protocol_version: ${protocol_version}
    throttle: 5s
    id: bms0

binary_sensor:
  - platform: jk_bms_ble
    balancing:
      name: "${name} balancing"
    charging:
      name: "${name} charging"
    discharging:
      name: "${name} discharging"

button:
  - platform: jk_bms_ble
    retrieve_settings:
      name: "${name} retrieve settings"
    retrieve_device_info:
      name: "${name} retrieve device info"

number:
  - platform: jk_bms_ble
    jk_bms_ble_id: bms0
    balance_trigger_voltage:
      name: "${name} balance trigger voltage"
    cell_count:
      name: "${name} cell count"
    total_battery_capacity:
      name: "${name} total battery capacity"
    cell_voltage_overvoltage_protection:
      name: "${name} cell voltage overvoltage protection"
    cell_voltage_overvoltage_recovery:
      name: "${name} cell voltage overvoltage recovery"
    cell_voltage_undervoltage_protection:
      name: "${name} cell voltage undervoltage protection"
    cell_voltage_undervoltage_recovery:
      name: "${name} cell voltage undervoltage recovery"
    balance_starting_voltage:
      name: "${name} balance starting voltage"
    voltage_calibration:
      name: "${name} voltage calibration"
    current_calibration:
      name: "${name} current calibration"
    power_off_voltage:
      name: "${name} power off voltage"
    max_balance_current:
      name: "${name} max balance current"
    max_charge_current:
      name: "${name} max charge current"
    max_discharge_current:
      name: "${name} max discharge current"

sensor:
  - platform: jk_bms_ble
    jk_bms_ble_id: bms0
    min_cell_voltage:
      name: "${name} min cell voltage"
    max_cell_voltage:
      name: "${name} max cell voltage"
    min_voltage_cell:
      name: "${name} min voltage cell"
    max_voltage_cell:
      name: "${name} max voltage cell"
    delta_cell_voltage:
      name: "${name} delta cell voltage"
    average_cell_voltage:
      name: "${name} average cell voltage"
    cell_voltage_1:
      name: "${name} cell voltage 1"
    cell_voltage_2:
      name: "${name} cell voltage 2"
    cell_voltage_3:
      name: "${name} cell voltage 3"
    cell_voltage_4:
      name: "${name} cell voltage 4"
    cell_voltage_5:
      name: "${name} cell voltage 5"
    cell_voltage_6:
      name: "${name} cell voltage 6"
    cell_voltage_7:
      name: "${name} cell voltage 7"
    cell_voltage_8:
      name: "${name} cell voltage 8"
    cell_voltage_9:
      name: "${name} cell voltage 9"
    cell_voltage_10:
      name: "${name} cell voltage 10"
    cell_voltage_11:
      name: "${name} cell voltage 11"
    cell_voltage_12:
      name: "${name} cell voltage 12"
    cell_voltage_13:
      name: "${name} cell voltage 13"
    cell_voltage_14:
      name: "${name} cell voltage 14"
    cell_voltage_15:
      name: "${name} cell voltage 15"
    cell_voltage_16:
      name: "${name} cell voltage 16"
    cell_voltage_17:
      name: "${name} cell voltage 17"
    cell_voltage_18:
      name: "${name} cell voltage 18"
    cell_voltage_19:
      name: "${name} cell voltage 19"
    cell_voltage_20:
      name: "${name} cell voltage 20"
    cell_voltage_21:
      name: "${name} cell voltage 21"
    cell_voltage_22:
      name: "${name} cell voltage 22"
    cell_voltage_23:
      name: "${name} cell voltage 23"
    cell_voltage_24:
      name: "${name} cell voltage 24"
    cell_resistance_1:
      name: "${name} cell resistance 1"
    cell_resistance_2:
      name: "${name} cell resistance 2"
    cell_resistance_3:
      name: "${name} cell resistance 3"
    cell_resistance_4:
      name: "${name} cell resistance 4"
    cell_resistance_5:
      name: "${name} cell resistance 5"
    cell_resistance_6:
      name: "${name} cell resistance 6"
    cell_resistance_7:
      name: "${name} cell resistance 7"
    cell_resistance_8:
      name: "${name} cell resistance 8"
    cell_resistance_9:
      name: "${name} cell resistance 9"
    cell_resistance_10:
      name: "${name} cell resistance 10"
    cell_resistance_11:
      name: "${name} cell resistance 11"
    cell_resistance_12:
      name: "${name} cell resistance 12"
    cell_resistance_13:
      name: "${name} cell resistance 13"
    cell_resistance_14:
      name: "${name} cell resistance 14"
    cell_resistance_15:
      name: "${name} cell resistance 15"
    cell_resistance_16:
      name: "${name} cell resistance 16"
    cell_resistance_17:
      name: "${name} cell resistance 17"
    cell_resistance_18:
      name: "${name} cell resistance 18"
    cell_resistance_19:
      name: "${name} cell resistance 19"
    cell_resistance_20:
      name: "${name} cell resistance 20"
    cell_resistance_21:
      name: "${name} cell resistance 21"
    cell_resistance_22:
      name: "${name} cell resistance 22"
    cell_resistance_23:
      name: "${name} cell resistance 23"
    cell_resistance_24:
      name: "${name} cell resistance 24"
    total_voltage:
      name: "${name} total voltage"
    current:
      name: "${name} current"
    power:
      name: "${name} power"
    charging_power:
      name: "${name} charging power"
    discharging_power:
      name: "${name} discharging power"
    temperature_sensor_1:
      name: "${name} temperature sensor 1"
    temperature_sensor_2:
      name: "${name} temperature sensor 2"
    power_tube_temperature:
      name: "${name} power tube temperature"
    state_of_charge:
      name: "${name} state of charge"
    capacity_remaining:
      name: "${name} capacity remaining"
    total_battery_capacity_setting:
      name: "${name} total battery capacity setting"
    charging_cycles:
      name: "${name} charging cycles"
    total_charging_cycle_capacity:
      name: "${name} total charging cycle capacity"
    total_runtime:
      name: "${name} total runtime"
    balancing_current:
      name: "${name} balancing current"
    errors_bitmask:
      name: "${name} errors bitmask"

switch:
  - platform: jk_bms_ble
    charging:
      name: "${name} charging"
    discharging:
      name: "${name} discharging"
    balancer:
      name: "${name} balancer"

  - platform: ble_client
    ble_client_id: client0
    name: "${name} enable bluetooth connection"

text_sensor:
  - platform: jk_bms_ble
    errors:
      name: "${name} errors"
    total_runtime_formatted:
      name: "${name} total runtime formatted"

Which ESPHome version do you use?

[v2021.11.4]

Could you try to update ESPHome to 2022.6? I've made a copy of your configuration above. I've changed
esp32:
  board: esp32dev
  framework:
#  The arduino platform crashs
    type: arduino
    version: latest
#    type: esp-idf
#    version: latest

to
esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: latest

and executed the compile & flash task:
$ esphome run dremeier.yaml
INFO Reading configuration dremeier.yaml...
INFO Cloning https://github.com/syssi/esphome-jk-bms.git@main
INFO Generating C++ source...
INFO Core config or version changed, cleaning build files...
INFO Compiling app...
Processing jk-bms (board: esp32dev; framework: espidf; platform: platformio/espressif32 @ 3.5.0)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
 - framework-espidf 3.40302.0 (4.3.2)
 - tool-cmake 3.16.4
 - tool-ninja 1.7.1
 - toolchain-riscv32-esp 8.4.0+2021r2-patch2
 - toolchain-xtensa-esp32 8.4.0+2021r2-patch2
 - toolchain-xtensa-esp32s2 8.4.0+2021r2-patch2
Reading CMake configuration...
Library Manager: Installing bblanchon/ArduinoJson @ 6.18.5

Unpacking  [------------------------------------]    0%
Unpacking  [------------------------------------]    1%
Unpacking  [------------------------------------]    2%
[...]

INFO Successfully compiled program.
INFO Resolving IP address of jk-bms.local

with trouble.

I it doesn't work out of the box please clean up a build folder by:
esphome clean config.yaml
esphome run config.yaml

If your dashboard doesn't provide a "clean build" button you could also rename the node to create a new build folder.

can´t update the ESPhome dashboard it is fixed in the sources for the iobroker app. But I´ll try to install a comadline ESPhome in a VM. Do you have a hint how to do?

On linux this is pretty easy. In best case python3 and pip3 (the python package manager) is already installed. In worst-case you have to install both first:
apt-get install python3 python3-pip

Now you can use pip3 to install the latest ESPHome version:
pip3 install esphome

In a few weeks/month you can update by:
pip3 install esphome -U

The package manage will install the package into your home directory. The executable / command line tool can be found here:
.local/bin/esphome

Either you write the path always in front of the command:
.local/bin/esphome run config.yaml

Or you have to extend your $PATH to include ".local/bin/" f.e. by extending the .bashrc:
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

If you start a new shell now / open a new terminal window the CLI command esphome should be available without passing the path.

perfect! will do tomorrow, have to go now :-(

Same here. Have a nice workday! ;-)

So, I installed ESPhome on a fresh ubuntu 20.04 and run the the above .yaml with success!
Then I used the esphome-flasher to flash the binary and got the error: "Unexpected error: The firmware binary is invalid (magic byte=FF, should be E9)"
Using 'COM4' as serial port.
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP32
Connecting....

Chip Info:
 - Chip Family: ESP32
 - Chip Model: ESP32-D0WDQ6 (revision 1)
 - Number of Cores: 2
 - Max CPU Frequency: 240MHz
 - Has Bluetooth: YES
 - Has Embedded Flash: NO
 - Has Factory-Calibrated ADC: YES
 - MAC Address: 40:91:51:AB:8C:B8
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
 - Flash Size: 4MB
Unexpected error: The firmware binary is invalid (magic byte=FF, should be E9)

btw:
-in the .yaml, does I have to change the "mac_address: C8:47:8C:E1:E2:E1" to the mac of the BMS?
-can I use any OTA pw?
-I like to use only mqtt, the framework can still be esp-idf? what I´ve to do with the "api:"?

If you are able to passthrough a USB device you could use esphome run config.yaml to flash the ESP using esphome via ttyUSB0 f.e.

Flashing the firmware file using a generic flasher should also be possible. In this case it is important to pick the correct file. Which file did you try exactly?

You have to change the MAC address to the address of your BMS. If you don't know the MAC address yet just let the ESPHome node boot with a wrong MAC address. The node will report all BLE devices at the log. You can lookup the address of your BMS there and flash again using the correct address.

You can use any OTA password. If you set none everybody having access to your local network could re-flash/OTA your node.

Yes. MQTT is supported at ESP-IDF since a few ESPHome releases. The api is just another communication protocol (to talk to a Home Assistant instance). Home Assistant is able to discover and integrate ESPHome nodes with a single click. If you don't use Home Assistant MQTT is the way to go.

Thanks!
I flashed now with a passthrough USB and also with the esphome-flasher successfully. I don't know where was the mistake before.
Wifi and BLE connects fine and mqtt with data comes in. All fine so far.
But, after 8 attempts of reconnecting ble, the connection refused.
maybe you can read more out of the log file?
output_2022-07-10_15-21-51.log

The BMS can handle a single connection simultaneously only. A new connection can be established only if the red status LED is blinking. Closing the Android app (for example) doesn't stop the BLE connection. You have to force close the app to disconnect from the BMS.
If the status LED is blinking and you are unable to connect using the ESP please reboot the ESP first and restart the BMS as second step to identify the origin of the issue.

but the esp is connecting with the BMS for short time and data are showing up in the mqtt broker, just after 8 reconnection the esp is giving up. The BMS app on the iphone is closed and out of range. There is just a second bms with different mac which is connected with another ESP-ble projekt.

I'm on vacation and didn't have the chance to inspect your log yet. I will have a look later the week.

some news:
the esp is now connected to the uart-rs485 port of the BMS. Thats works perfect and stable to the mqtt broker.
Sending data to the api, the most sensor and switch data are missing some are working and after a while the the connection to the api closed, maybe dueto my old version in the esphome app of ioBroker.
For me just the mqtt is working and is fine. But I guess some like to use the api. How can we debug that issue?
here my yaml files
[batterie-1_uart_api.yaml.txt](h
batterie-1_UART_MQTT.yaml.txt
ttps://github.com/syssi/esphome-jk-bms/files/9079478/batterie-1_uart_api.yaml.txt)

I'm on vacation and didn't have the chance to inspect your log yet. I will have a look later the week.

enjoy your vacation! Take your time. I´ll try to investigate further :-)

The UART connection is another protocol and read-only. Please don't expect the same data/measurements/values & features here compared to BLE. I recommend the UART connect to monitor the BMS. To control the switches or change settings a BLE connection is required.

I've inspected your output_2022-07-10_15-21-51.log. I haven't seen any of the exceptions and assertions yet. I will update my ESPHome install in a few weeks and try to reproduce the issue.

The issue is may be related to this one:
https://github.com/esphome/issues/issues/3327
https://github.com/esphome/esphome/pull/3594

the ESP is now up and running with direkt wires. Just reading the data is fine for me.

Alright. Let's close this issue because it's already about different aspects. Feel free to create a new issue if something doesn't work as expected!


#41: Removed FbServiceUnavailable

@rultor merge

@rultor merge

@gilraenburland OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@gilraenburland Done! FYI, the full log is here (took me 2min)


Don't save empty files to zip

Particularly now that we have data and analysis scripts we can expect some of them to be empty. The Gist feature already does this filtering

Looks good to me.


Migrating storage account for IAI binaries to IIOT_TEST subscription.

Changes:

Storing storage account and service connection names as variables.
Using azureiiot storage account in IIOT_TEST subscription.

Run of Azure.Industrial-IoT pipeline that successfully published files to the new storage account can be found here.


API documentation doesn't include port information

I just had someone ask me which port to use to access the API.
This means our documentation is lacking, as that's pretty much the most basic question you can have when trying to use the api.

fixed in #553


When only one side of the gripper contacts an object, it cannot grab it.

Hi,
I use turtlebot3 with Open manipulator-X in gazebo. I use the ros2 version of #57.
I would like to grab a Coke can.
The plugin has been successfully loaded.
However, if only one side of the gripper is contacted first during the closing process, the can goes out of the gripper.
In short, the grasp fails if it cannot make contact from both sides at the same time.
Are there any parameters to improve this?
I am a beginner, so please let me know if there are any possible ways to improve.
I am in the process of trying different things, possibly due to the robot's configuration file.
Thanks.

Sorry.
I was able to grasp it after trying different ways.
The cause was the weight of the object was heavy.
It seems that the weight was such that the gripper could not close.
By making the object lighter, I was able to grasp it.
I will close this issue.
Thank you very much!


Update whats-new.md with changes in Custom Triggers called with authorization header

Adds to our release notes some recent changes related to custom triggers with authorization.
Harness Developer Pull Request
Thanks for helping us make the Developer Hub better. The PR will be looked at
by the maintainers.
What Type of PR is This?

[ ] Issue
[ ] Feature
[x] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[x] Tested Locally
[ ] Optional Screen Shoot.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app


Add Vault's structured encrypt/decrypt APIs (GEA-11388)

Rebased onto latest main to bring in the test fix for events with empty messages.


Replace signal-based queue dumps with console based one

As soon as debug console support is merged, the USR1/USR2 way of dumping the event queue should be unified by replacing it with an additional command on the debug console.

@marc-casperlabs are these 1.5.0 stuff, seems the PR is already merged??

It's not 1.5 stuff and the merged PR is just a prerequisite, sorry!

@Fraser999 is this for 1.5?

No, it's not a blocker @piotr-dziubecki , just a nice-to-have.


Error when i want see likes count

when i want see likes count gave this message:
BadMethodCallException
Call to undefined method App\User::ratings()

@see https://github.com/mohamednagy/Laravel-rating#rating
to rate a model use
$user->rate($postModel, 5);
to get the average ratings use:
$post->ratingsAvg();
to get the total ratings count use:
$post->ratingsCount();


github action to collect and upload past test results

By doing this in a Github action, we can use the automatic auth token (which hopefully has the right permissions) and get the heavy tasks of downloading/parsing json files done incrementally to speed up the process

I probably need to install the requests package in the python environment for this to work


types.ftl.go is using tabs not spaces

Assuming we want to be consistent with the rest of our code with this.
(removed lines have spaces, added lines have tabs)

This should be expected now that we're running go fmt on these types.ftl.go files after we generated them. go ❤ tabs 😂

I'll close this as expected for now, but happy to reopen if I'm missing something here @matt2e 👍

All good!


[AutoRelease] t2-sqlvirtualmachine-2023-06-21-90088(can only be merged by SDK owner)

https://github.com/Azure/sdk-release-request/issues/4213
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=1034
BuildTargetingString
azure-mgmt-sqlvirtualmachine
Skip.CreateApiReview
true

issue link:https://github.com/Azure/sdk-release-request/issues/4213


[xgboost4j-spark] Checkpointing performance issue

I just noticed a weird performance issue when using checkpoints in the Spark wrapper for XGBoost.
Before the first checkpoint, the learning time is constant per tree, but after the first checkpoint, the time needed to learn a new tree is much higher, and it grows with the number of trees.
Here is the parameter map used by XGBoost:
Map(alpha -> 0.0, min_child_weight -> 1.0, sample_type -> uniform, base_score -> 0.5, colsample_bylevel -> 1.0, grow_policy -> depthwise, skip_drop -> 0.0, lambda_bias -> 0.0, silent -> 0, scale_pos_weight -> 1.0, seed -> 0, features_col -> features, num_early_stopping_rounds -> 0, label_col -> label, num_workers -> 5, subsample -> 1.0, lambda -> 1.0, max_depth -> 5, probability_col -> probability, raw_prediction_col -> rawPrediction, tree_limit -> 0, custom_eval -> null, rate_drop -> 0.0, max_bin -> 16, train_test_ratio -> 1.0, use_external_memory -> false, objective -> binary:logistic, eval_metric -> logloss, num_round -> 500, timeout_request_workers -> 1800000, missing -> NaN, checkpoint_path -> viewfs://root/user/XXX/YYYY/checkpoint, tracker_conf -> TrackerConf(0,python), tree_method -> auto, max_delta_step -> 0.0, eta -> 0.3, colsample_bytree -> 1.0, normalize_type -> tree, custom_obj -> null, gamma -> 0.0, sketch_eps -> 0.03, nthread -> 4, prediction_col -> prediction, checkpoint_interval -> 100)
Attached is a small graph, where this issue is shown on the same dataset, with different checkpointing number. Please disregard the discrepancy between the two run, as they might not have been launched with the same multithreading parameters.

A quick profiling (using async-profiler) before and after the checkpoint might give a hint to the problem. Before the checkpoint, I see many calls to tree::CQHistMaker or tree::GlobalProposalHistMaker, but after the checkpoint point, the C++ calls are all to CPUPredictor::PredLoopSpecialize. I do not know whether this means that the method used for learning is not the same after the checkpoint, or that the time needed to evaluate the tree is so long that it the profiler only sees it. I could upload some flamegraphs, but I would need first to make sure they are actually representative of what is going on.

IMO, there were unnecessary reshuffle after each checkpoint. Is there any info around memory utilization?

@cq not true, extra shuffle happens for every 100 iterations in this case..

@chenqin and @CodingCat , do you need any other "benchmark" ?

Hello, has this question raised any interest? we are willing to investigate and contribute and for that wanted to check first that no update / fix existed... Is anyone watching/involved?

@CodingCat Is anyone in your org looking at this issue?

no, please go ahead, we do not have bandwidth to take care of it for now

I have started investigating the issue. And it looks like I found one possible cause of it. The prediction caches are not initialized after the first checkpoint (gbm_.predictor_.cache_). And if the caches have not been initialized they will not be filled. So every iteration all predictions for every previous tree are recomputed. We are losing caches when propagating the booster to the driver. There are two possible solutions:

Reinitialize the caches after starting from checkpoint. It will solve the part of the problem. The cost is that we need to recompute caches on the first iteration after every checkpoint.
Propagate the caches with a booster. I think there is some kind of issue with serialization of C++ object.

@bourbaki thanks for the analysis and I agree with it
regarding how to resolve it, I think those two solutions have their own problems as you said

the first iteration right after the checkpoint would be super slow, and it becomes slower and slower with the growth of trees

the problem is we have to change the serialization format of booster which may hurt the other bindings of xgboost, and furthermore, the serialized booster would be in a huge size if the prediction results are happen to be very large

the ideal fix should be more fundamental...I got some ideas and am about to start my work...but I just noticed you added I am preparing a pull request after I read your comments last time
please let me know your proposal before you submit the code, and I am actually in a hurry to get this fix due to an internal request

@bourbaki if you are dedicated to this, I can assign this to you

@CodingCat Yes. I am dedicated to this. What approach would you suggest?

@bourbaki sorry for the late reply, really busy in these days, my suggestion would be

since the checkpoint is a functionality built in xgboost4j-spark layer, we can just pursue a solution in this level,

the point is to get cache initialized with a booster from checkpoint, I would suggest we

2.1 save the prediction results with Spark
2.2 expose an API to directly call Init Function in Predictor
2.3 when starting training from a checkpoint, we should load back data (maybe have to through HDFS API directly) saved in 2.1, and use a resembled version of PredLoopSpecalize to update the cache with the loaded Dataset
the most important thing we need to be careful is to guarantee the deterministic partitioning

It sounds good. But there's one thing that bothers me with the current design of xgboost4k-spark.
At every checkpoint we collect booster data from executors to a driver to save it. But then to start the next iteration instead of reusing the current state of executors we copy a booster from the driver to each executor. These copies are redundant in the case when there are no failures in the training proces and we just want to get the current state of a booster without interrupting the training process. It would be cool to have something like this:
val boosters1: RDD[Booster] = train(..., iterations=100).cache
val boosterToSave = boosters1.first
boosterToSave.save(...)
val boosters2 = continueTrain(boosters1, iterations=100).cache
....

It sounds good. But there's one thing that bothers me with the current design of xgboost4k-spark.
At every checkpoint we collect booster data from executors to a driver to save it. But then to start the next iteration instead of reusing the current state of executors we copy a booster from the driver to each executor. These copies are redundant in the case when there are no failures in the training proces and we just want to get the current state of a booster without interrupting the training process. It would be cool to have something like this:
val boosters1: RDD[Booster] = train(..., iterations=100).cache
val boosterToSave = boosters1.first
boosterToSave.save(...)
val boosters2 = continueTrain(boosters1, iterations=100).cache
....

In this case when training goes according to the plan we only copy booster to driver and then restart the training from the cached state.

yes, in that way we need some way to pass the message from driver to executor saying "please move on"...leading us to more complexities from the message system for training...
if we want to pursue that approach, we need to extend rabit layer to allow tracker and worker to communicate more than "start/stop/I died/please reborn", etc.

but still we need to save the prediction results to allow the training recover from a checkpoint..which does not reduce the complexity I talked about earlier

@bourbaki are you working on this?

ping @bourbaki

Sorry for the late reply. I have been busy for the last couple of weeks. I started to work on it. I am planning to spend some time on this week.

@bourbaki are you still working on this? if you are too busy, I can take over it

@CodingCat Sorry, I am too busy right now. You can take over it.

Hello @CodingCat. We allocated significant time to fix checkpointing performance issue in XGBoost4J-Spark (it is a blocker for us) in July-August. So I am wandering what is a current status of your work and whether you prepare any pull requests now or I could take this over.
Thank you

I am currently working on it

@CodingCat could you please share the status of your work and confirm the approach you decided to take?
This issue blocks us (Criteo). We allocated resources to fix it. Let us know how we can contribute or how you can hand over if you have higher priorities.

it’s still WIP in my side, and I will share the design in the format of a RFC within this month (actually I have shared some in this thread) and code later ,

update:
basically, I have run the prototype fix with our production data and everything looks fine. Now, it comes to the cost of fix. The fix under test goes to the path that the prediction cache is rebuilt for each checkpoint interval, however, it still slows down first iteration of each interval. However, a more fundamental fix requires API changes/adding, and the added API might be only consumed by jvm package, we need to evaluate whether it is worth that
let me start a new design discussion thread on this

Thank you for the update. I am glad you have a fix and from your description I think I have a similar fix being tested on production dataset too.
I am looking forward for you sharing notes|patches so we can compare our notes.
By the way, my fix is published as a branch in our fork. I can link it here if you are interested.
I did not submit a pull request cause it is not yet ready for it: there are some "unit tests" to show the effect of checkpointing and some not very clean logging.

@CodingCat do you have any updates on your side?
I want to prepare a RFC with my idea for potential fixes. What do you think?

Can this be closed, now that #5082 is merged?


test/e2e: Increase test timeout

It takes the checkup ~ten minutes to reach the Run phase on the dev cluster, so it makes no sense that the e2e timeout is ten minutes. Increasing to one hour.

Its important to note that the actual test duration is only one minute.


Fallback to using the hicolor theme if the selected theme doesn't exist

Forgive the large diff due to formatting.
The only actual change is adding .or_else(|| THEMES.get("hicolor")) after THEMES.get(self.theme).

Hey @Palladinium Can you please run cargo fmt and force push?

I did - I get no changes running cargo fmt on my end.

Strange, can you run rustup update and try again ?

I tried both cargo +stable fmt and cargo +nightly fmt after rustup update, no change. What should I be seeing?
The only other thing I can think of is that I don't have any rustfmt.toml configs, just running the defaults. Do you have any settings on your end that we should commit to the repo?

I just cloned your repo and got no changes from rustfmt. I guess Github diff is broken. Sorry about that.
Merging now :)

I will test you changes an release the new version for Onagre soon. Thanks a lot for the fix !


Update to k8s v1.26.0, container-runtime v0.14.1

Signed-off-by: Waleed Malik<EMAIL_ADDRESS>What this PR does / why we need it:
Update to k8s v1.26.0, container-runtime v0.14.1.
Which issue(s) this PR fixes:

Fixes #
What type of PR is this?

/kind chore
Special notes for your reviewer:
Does this PR introduce a user-facing change? Then add your Release Note here:

NONE

Documentation:

NONE

/lgtm


Use multiple threads for maven-metadata.xml

Otherwise generating this file can be quite slow the first time it is accessed.
The HTTP connections were also not being closed, which was probably the root cause of the resource leak.

/hold
Failure is related

/test jvm-build-service-in-repo-apicurio-e2e


[Fix] Support AMP with DETR-detectors

Motivation
Related to https://github.com/open-mmlab/mmdetection/pull/10018
Currently, mixed precision training is not supported in DETR.
There are two reasons.

Computing areas of boxes are easy to overflow in FP16.
https://github.com/open-mmlab/mmdetection/blob/ecac3a77becc63f23d9f6980b2a36f86acd00a8a/mmdet/structures/bbox/bbox_overlaps.py#L151-L154
The maximum value for FP16 is 65504, which is prevalent in predictions of the early training phase.

  File "/mmdetection/mmdet/models/dense_heads/detr_head.py", line 406, in _get_targets_single
    assign_result = self.assigner.assign(
  File "/mmdetection/mmdet/models/task_modules/assigners/hungarian_assigner.py", line 133, in assign
    matched_row_inds, matched_col_inds = linear_sum_assignment(cost)
  File "/usr/local/lib/python3.8/site-packages/scipy/optimize/_lsap.py", line 86, in linear_sum_assignment
    return _lsap_module.calculate_assignment(cost_matrix, maximize)
ValueError: matrix contains invalid numeric entries

The dtype of targets is generated from predictions, but it should be FP32 like other code.

  File "/mmdetection/mmdet/models/dense_heads/detr_head.py", line 274, in loss_by_feat_single
    cls_reg_targets = self.get_targets(cls_scores_list, bbox_preds_list,
  File "/mmdetection/mmdet/models/dense_heads/detr_head.py", line 360, in get_targets
    neg_inds_list) = multi_apply(self._get_targets_single,
  File "/mmdetection/mmdet/models/utils/misc.py", line 219, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/mmdetection/mmdet/models/dense_heads/detr_head.py", line 437, in _get_targets_single
    bbox_targets[pos_inds] = pos_gt_bboxes_targets
RuntimeError: Index put requires the source and destination dtypes match, got Half for the destination and Float for the source.

Modification

Add an option to normalize bboxes to prevent overflow
Generate targets with the type of ground-truth rather than prediction.

 We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.16 out of 17 committers have signed the CLA.:white_check_mark: freepoet:white_check_mark: minato-ellie:white_check_mark: lovelykite:white_check_mark: wufan-tb:white_check_mark: hhaAndroid:white_check_mark: Czm369:white_check_mark: zwhus:white_check_mark: i-aki-y:white_check_mark: gihanjayatilaka:white_check_mark: keyakiluo:white_check_mark: xin-li-67:white_check_mark: RangeKing:white_check_mark: lucianovk:white_check_mark: MambaWong:white_check_mark: JingweiZhang12:white_check_mark: nijkah:x: yalibianYou have signed the CLA already but the status is still pending? Let us recheck it.

@nijkah We are planning a regression bencharm for next monday, can you please finish it before then? Thanks

Hi, @hhaAndroid . To make the code clean, I made a BaseIoULoss class.
Feel free to correct it if the change is too much.

Hi, @hhaAndroid. I finished all the work. Can I ask you to check it?

@nijkah Perhaps I did not describe it clearly. What I mean is that you can convert the prediction bbox to fp32 directly in the iou loss place https://github.com/open-mmlab/mmdetection/blob/dev-3.x/mmdet/models/task_modules/assigners/match_cost.py#L167, so that the change is minimal

@nijkah Perhaps I did not describe it clearly. What I mean is that you can convert the prediction bbox to fp32 directly in the iou loss place https://github.com/open-mmlab/mmdetection/blob/dev-3.x/mmdet/models/task_modules/assigners/match_cost.py#L167, so that the change is minimal

@hhaAndroid I understand. But, would it be better to support computing IoU in FP16 rather than casting tensors to FP32? It will save computation costs.
Since there is a trade-off between the clearness and the performance, I will follow your decision.
If you think casting it to FP32 is better, I will modify the code in https://github.com/open-mmlab/mmdetection/blob/dev-3.x/mmdet/models/task_modules/assigners/match_cost.py#L167 and https://github.com/open-mmlab/mmdetection/blob/main/mmdet/models/dense_heads/detr_head.py#L320. (Both cost and loss should be modified.)

@nijkah Okay, I understand. If you want to save memory, your method is the most appropriate. what is the basis for setting the norm_value value? If NAN still occurs, is it necessary to increase this parameter further? Is this a hyperparameter? And it is essential to experiment with how much memory is saved.  There seems to be quite a bit of validation work. I feel like it's too late for today

@hhaAndroid . I understand. Then, is it better to modify the current code as casting tensors to FP32? or just postpone this PR with validations?
Using the maximum value of tensors to normalize boxes is possible for general cases. But I thought that normalizing with 100.0 is in the scope of FP16 since the maximum values of boxes are less than 10,000 in most cases. I agree with you it should be verified further.

@nijkah Or this pr temporarily use the safest practice that is to use the fp32 practice, and then separately create the PR to verified

@nijkah To be safe, let's consider merging it in the next release. You can prioritize working on RT-DETR and we'll release a new version after RT-DETR is merged. Thank you!

https://github.com/open-mmlab/mmdetection/pull/10827


User SNBAT send/withdraw cUSD/CELO to their own active Valora address

Frequency: 100%
Repro App version:  Test Flight build Valora app build  v1.0.0 (17) , Android Valora Mainnet build v1.0.0 (16)
Repro on device: iPhone 7+ (13.3.1), Google Pixel Xl (8.1)
Test Account: Backup Key:<PHONE_NUMBER>.
Repro Steps:

Launch the Application and sign in with test account
Tap on the Hamburger menu.
Tap on the CELO option.
Tap on Withdraw CELO section
Copy and paste the own account number.
enter amount and proceed to withdraw CELO

Current Behavior: User is able to WIthdraw CELO from his own account.
Expected Behavior:  User should not be able to withdraw from his own account
Investigation:
Payment received notification is shown when the user proceeds for their own account.
Withdrawal… activity is shown but received activity is not shown.
Attachment: CELO_Withdraw_Own.mp4 (Bug observed at 30 sec)

@MuckT Can you please comment as user is able to withdraw CELO using own account address


Preserve AttributeError in slotted classes with cached_property

Summary
In slotted classes' generated __getattr__(), we try __getattribute__() before __getattr__(), if available, and eventually let AttributeError propagate. This matches better with the behaviour described in Python's documentation Customizing attribute access.
Fix https://github.com/python-attrs/attrs/issues/1230
Pull Request Check List

[x] Do not open pull requests from your main branch – use a separate branch!

There's a ton of footguns waiting if you don't heed this warning. You can still go back to your project, create a branch from your main branch, push it, and open the pull request from the new branch.
This is not a pre-requisite for your your pull request to be accepted, but you have been warned.

[x] Added tests for changed code.
Our CI fails if coverage is not 100%.
[x] New features have been added to our Hypothesis testing strategy.
[x] Changes or additions to public APIs are reflected in our type stubs (files ending in .pyi).

[x] ...and used in the stub test file tests/typing_example.py.
[x] If they've been added to attr/__init__.pyi, they've also been re-imported in attrs/__init__.pyi.

[x] Updated documentation for changed code.

[x] New functions/classes have to be added to docs/api.rst by hand.
[x] Changes to the signature of @attr.s() have to be added by hand too.
[x] Changed/added classes/methods/functions have appropriate versionadded, versionchanged, or deprecated directives.
The next version is the second number in the current release + 1.
The first number represents the current year.
So if the current version on PyPI is 22.2.0, the next version is gonna be 22.3.0.
If the next version is the first in the new year, it'll be 23.1.0.

[x] Documentation in .rst and .md files is written using semantic newlines.
[x] Changes (and possible deprecations) have news fragments in changelog.d.
[x] Consider granting push permissions to the PR branch, so maintainers can fix minor issues themselves without pestering you.

ping @diabolo-dan :)

thanks everyone!


Update machine1 from Ubuntu 22.04 to 24.04

Description
The result of the workflow will be a Ubuntu 24.04 host. Previously this was a 22.04 host.
Why is this needed
Ubuntu 24.04 is the latest LTS release.
Fixes: #
How Has This Been Tested?

Tested with libvirt as the vagrant provider on Ubuntu 22.04.1 (x64).
Tested with virtualbox as the vagrant provider on Ubuntu 22.04.1 (x64).

How are existing users impacted? What migration steps/scripts do we need?

Ubuntu 24.04 has a /boot partition, Ubuntu 22.04 had /boot on the / partition. The kexec portion has been updated to mount the /boot partition. No change is required to run the steps. The change was introduced in the workflow template.

Checklist:
I have:

[X] updated the documentation and/or roadmap (if required)
[ ] added unit or e2e tests
[ ] provided instructions on how to upgrade

Hey @ttwd80 . Thanks for this. Mind squashing all these commits with duplicate comments? Update VAGRANTVBOX.md and Update VAGRANTLVIRT.md


NIFI-3497 ScanAttribute Tagging

Adding tagging to ScanAttribute.

@joetrite it would be good to get a clean master, create a branch, put your changes on it, then submit that PR.  This one has a couple merge commits now and they're probably not needed (rebase would likely have done the trick).  Can you clean up the commit history for this PR?  Definitely appreciate the work you're putting in and don't want to put you through hoops but would like to keep the commit clean.

created new pull request #1564 with single commit. closing this.  thanks @joewitt @alopresto for all the help and advise.


CDPSDX-4030: Added entitlement for new Scalable SDX cluster shape.

Jira: https://jira.cloudera.com/browse/CDPSDX-4030
This is dependent on changes in #14312 so will be blocked until that is merged.

@sxxgrc other PR is merged, please rebase this, resolve conflicts and let this be merged as well into master

@horadla23 Thank you for the update. There is an ongoing conversation on our side again about the entitlement, as soon as that is resolved I will move forward with this.
For reference, currently the shape is only available on 7.2.17 and only available within CB. We will soon merge in the exposure of it in the CDP CLI, but this will also be internal only. Just letting you know in case this eases any current concerns about the shape changes being available.
The main blocked task for this is exposing it in the UI as well.

I am trying to understand why we merge things without a clear picture. There is no reason to merge these commits (the previous one was merged today) if customers can not use it, but they can not use it because we don't know what will be the name and there is no scaling at the moment. The same thing can be achieved for testing via internal SDX API with a custom template.

please make sure this PR is ready for merge, as that's a strict requirement for every new big feature, especially those that are still under dev and test.

@sxxgrc please make sure this PR is ready for merge by the end of the week as we are planning to branch early next week. If this PR is not merged this week, we have to revert https://github.com/hortonworks/cloudbreak/pull/14312 from this branch and you have to resubmit it to 2.72

@sxxgrc can you please make this item your top priority? As I see it shouldn't be that complex to finish it this week since you already have an almost done PR.


[VS Code Insiders] Nodejs and npm not working!

VSCode Version: 1.52.0-insider (user setup)
OS Version: Windows_NT x64 10.0.19042

Steps to Reproduce:

Downloaded VS Code insider
Downloaded Node through this website
Created a .js file
Tried

var msg = 'Hello World';
console.log(msg);

Got an error saying

node: The term 'node' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

Extra note: In CMD when I type node -v it works but in the terminal in vs code insiders when I type node -v it doesn't work. This is the same for npm -v

Does this issue occur when all extensions are disabled?: Yes(note: Code Runner is still there)

Did you sign out or restart your computer after installing Node.js? Environment variables won't change for already running programs (and programs launched by them).

No I haven't, I will restart my comp now.

Did you sign out or restart your computer after installing Node.js? Environment variables won't change for already running programs (and programs launched by them).

Thanks a ton @yume-chan. I restarted my computer jn and it worked. Really, thanks a LOT.
BTW jn means just now


Bugfix/boolean arrays

Fixed issue #30.

Six other tests raise errors for me, but the one failure has been fixed.

But Spark must be working better on Travis than on my laptop. I'd say it's mergeable.


this model can train multi-class polyp segmentation??

I was looking for a multi-class segmentation model and came across what you created. Is this model capable of segmenting multi-class polyps present in colonoscopy images?

ConvFormer can be used to replace the transformer blocks in transformer-based or cnn-transformer hybrid models to improve the related models' performance. The number of segmentation categories had no effect on its use.

ConvFormer can be used to replace the transformer blocks in transformer-based or cnn-transformer hybrid models to improve the related models' performance. The number of segmentation categories had no effect on its use.

Wow, that's good. thanks to your answer!


ftplib.error_perm: 530 Login failed

When I try demo-of-the-trick-commands-on-the-mambo this demo on linux-mint I got
OpenCVAvailable is True
Traceback (most recent call last):
  File "examplefly.py", line 12, in <module>
    mambo = Mambo(None, use_wifi=True)
  File "/home/ihsancemil/.local/lib/python3.5/site-packages/pyparrot/Minidrone.py", line 310, in __init__
    self.groundcam = MamboGroundcam()
  File "/home/ihsancemil/.local/lib/python3.5/site-packages/pyparrot/Minidrone.py", line 237, in __init__
    login = self.ftp.login()
  File "/usr/lib/python3.5/ftplib.py", line 419, in login
    resp = self.sendcmd('PASS ' + passwd)
  File "/usr/lib/python3.5/ftplib.py", line 272, in sendcmd
    return self.getresp()
  File "/usr/lib/python3.5/ftplib.py", line 245, in getresp
    raise error_perm(resp)
ftplib.error_perm: 530 Login failed

I can read camera stream by my self so I know that my wifi connected to the mambo
import cv2
import time

vcap = cv2.VideoCapture("rtsp://<IP_ADDRESS>/media/stream2 ! rtph264depay ! h264parse ! omxh264dec ! appsink max-buffers=1 drop=true")

while(1):
    ret, frame = vcap.read()
    if(type(frame) != type(None)):
    	cv2.imshow('VIDEO', frame)
    	if cv2.waitKey(1) == 27:
        	break

this code is working

This bug was fixed in the latest update.  The issue is that the new mambo firmware broke ftp which is how we use the groundcamera.  Also, I'm glad your opencv code works but it doesn't in general keep the stream working across all OS's. That's why I did a different approach for vision.  But I am glad it works well for you!

Vov that was a fast answer :D I will update pyparrot via git then. Thanks... And yes I switch windows to linux to make that code work

Yes master branch works perfectly


waffle.io Badge

Merge this to receive a badge indicating the number of issues in the ready column on your waffle.io board at https://waffle.io/LEDS/X-data
This was requested by a real person (user LEDS) on waffle.io, we're not trying to spam you.

ok!


Issue with predict_on_batch dimensions

Hi,
I trained a model with inputs input_1 (InputLayer)         (None, 1440, 1440, 3) and batches of size 3 (because of the images dimensions), and I saved it.
When I load the model to test it and ask the same image data generator to generate the same type of batches, I get the following error from the method predict_on_batch: ValueError: Error when checking : expected input_1 to have shape (None, 1440, 1440, 3) but got array with shape (3, 1400, 1400, 3)
Any idea why during the training I can give arrays/batches of dimensions (3, 1400, 1400, 3) but not during the testing?

This can be removed...


Add Neuralangelo

Is your feature request related to a problem? Please describe.
I want to try out Neuralangelo
Describe the solution you'd like
Add the functionality
Describe alternatives you've considered
None
Additional context
https://github.com/NVlabs/neuralangelo

Hi @MohamedAliRashad, I believe the sister project SDFStudio has this implemented since their repo mainly targets SDF research similar to this paper.


at the end of installation, get this error messages

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements-video.txt'

use git clone --recurse-submodules https://github.com/sdbds/magic-animate-for-windows/
to git pull submodules


Time-series UI - Range filter doesn't reset when changing buckets

Context

Steps to Reproduce

Filter the data in the widget
Change the number of buckets

Current Result
The filter doesn't reset properly it ends up filtering the first bucket. You can press play although it seems filtered, and it goes trough all the data.
Expected result
Browser and version
Safari Version 10.1.1 (12603.2.4)

Hey @rubenmoya should we close this issue? The time series widget UI now is different.

Yeah, let's close it :)


Adjusting for height of camera

Hi,
I have mounted my raspberrypi and the software is running great. The only issue is that instead of logging one person walking in and out of the store, it often logs 2-3. I assume that my ceiling is lower than yours was.
I was hoping you could provide some guidance on how to adjust the config.xml file to compensate for the lower ceiling height.
Thanks again.

In general the *BLOBWIDTH settings are the ones to edit. Because people walking shoulder-to-shoulder tend to appear as a single blog, we set the app to detect abnormally wide blobs and count them as 2 people (midblobwidth) or 3 people (maxblobwidth).
However, do also be aware that lighting plays a significant part in this: sometimes shadows can count as a separate person. The only real way around that is to use a depth sensor like a kinect, rather than a plain 2D camera. Or some much fancier algorithms. Either way, we had to mess around with careful placement for a while before finding one that was decently accurate.
Oliver.

@oliverhumpage thanks for this response.
Hoping you could provide a little information on the variables "history", "mogthreshold", "maxdistance", "persistance", "minarea", "maxarea", "contourthreshold" and "threshold". Some of these are mentioned in your blog post but I would really appreciate if you could expand a bit on these.
Thanks again.

Hi @bsamaroo, So:
"history" : is the length of frames the MOG2 holds to adapt to the background. this is filled continuously!
"mogthreshold" : is the threshold to describe the background model
"maxdistance" : is the furthest distance a blob or person can travel in a set number of frames
"persistance" : is how long you want to track the blob before it starts being deleted
"minarea" : is the smallest a blob can be, this is the radius of the blob. Anything below this will not be recognised.
"maxarea": again is the largest a blob can be.
"contourthreshold" : is the threshold of the blob detection, so if the blob is white a value of around 200 is ok .
"threshold" : is the generic threshold of the image


[Creator] Generate single api template when user does not provide version or version set id

Is this support for generating a template that contains multiple versions of an API?


Unexpected name clash

Reported by Anders Larsson in the zulip:
pub fn byteToWord (n: u8) -> u16 {
    n.into()
}

triggers:
[Info ] Generated the partial file (because of errors): ./Mylib.lean
[Error] Name clash detected: the following identifiers are bound to the same name "core.convert.Into":
- trait_decl_id: 0
  Source: '/rustc/65ea825f4021eaf77f1b25139969712d65b435a4/library/core/src/convert/mod.rs', lines 444:0-444:24
- trait_impl_id: 0
  Source: '/rustc/65ea825f4021eaf77f1b25139969712d65b435a4/library/core/src/convert/mod.rs', lines 444:0-444:24
You may want to rename some of your definitions, or report an issue.
[Error] Error when registering the name for id: trait_impl_id: 0:
The chosen name is already in the names set: core.convert.Into
Source: '/rustc/65ea825f4021eaf77f1b25139969712d65b435a4/library/core/src/convert/mod.rs', lines 444:0-444:24

For context, the llbc for that is:
trait core::convert::From<Self, T>
{
    fn from : core::convert::From::from
}

trait core::convert::Into<Self, T>
{
    fn into : core::convert::Into::into
}

fn core::convert::{impl core::convert::Into<U> for T#3}::into<T, U>(@1: T) -> U
where
    // Inherited clauses:
    [@TraitClause0]: core::convert::From<U, T>,

impl<T, U> core::convert::{impl core::convert::Into<U> for T#3}<T, U> : core::convert::Into<T, U>
where
    [@TraitClause0]: core::convert::From<U, T>,
{
    fn into = core::convert::{impl core::convert::Into<U> for T#3}::into
}

fn core::convert::num::{impl core::convert::From<u8> for u16#40}::from(@1: u8) -> u16

impl core::convert::num::{impl core::convert::From<u8> for u16#40} : core::convert::From<u16, u8>
{
    fn from = core::convert::num::{impl core::convert::From<u8> for u16#40}::from
}

fn core::convert::Into::into<Self, T>(@1: Self) -> T

fn test_crate::byteToWord(@1: u8) -> u16
{
    let @0: u16; // return
    let n@1: u8; // arg #1
    let @2: u8; // anonymous local

    @2 := copy (n@1)
    @0 := core::convert::{impl core::convert::Into<U> for T#3}<u8, u16>[core::convert::num::{impl core::convert::From<u8> for u16#40}]::into(move (@2))
    drop @2
    return
}

fn core::convert::From::from<Self, T>(@1: T) -> Self

This imports the very generic impl:
impl<T, U> Into<U> for T
where
    U: From<T>,
{ ... }

My guess is that the name you generate for that is core.convert.Into which is the same as the trait name.

Generally speaking, that's slightly annoying.
But here the fix is simple: we need to add hand-written for those (because they should be part of the standard library), and by doing so get get to choose the name. I'll do that later.

Thanks for investigating btw!

I'm closing this because I solved it for the specific issue above, and I opened a more general issue: https://github.com/AeneasVerif/aeneas/issues/380


Getting an error when adding User and Password on protected service

Getting the following error when adding User and Password on protected service
Running script DataServicePillager2...
ERROR: No JSON object could be decoded
Plunderin' done, in 0:00:00.070000
Completed script DataServicePillager2...
Failed to execute (DataServicePillager2).
Failed at Wed Jul 19 16:35:09 2017 (Elapsed Time: 0.11 seconds)
Not sure if I am missing something. It works when I unprotect the service

It sounds like it is failing to authenticate. Is this a service that you could share some details about for debugging? Where is it hosted?

Updated to fix issue with authentication error on non standard web adaptor name, also tested with AGO.


[BUG] java.nio.channels.ClosedChannelException   &  writeAddress(..) failed

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
I'm experiencing some issues that are hard to reproduce and solve, such as the following exceptions, which occur 1000-2000 times a day in production. The exceptions are very random and difficult to reproduce directly. I need help!

Expected Behavior
No response
Steps To Reproduce
The problem occurs very randomly and is difficult to reproduce directly.
Environment
ShenYu version(s):2.4.3

Debug logs
No response
Anything else?
No response

u have some custom plugin?

Yes, there are several, such as plugins that support full-link grayscale publishing, and plugins that support sse responses.


Feature don't render on the map after applying filter on a layer and navigating to other layers.

https://localgovdev.mapsdevext.arcgis.com/apps/instant/reporter/index.html?appid=1d87c0739da84c63822e7eaf68e7d65b

Go to Poly layer and apply filter. Notice the filtered feature is rendered.
Go to another layer and choose a feature, it doesn't render on the map it but the popup details appear on the panel.
Go back to poly layer, and notice the filter is reset but no feature is rendered even as we choose the feature from the panel.

The acceptable behavior is to render other layer's features when navigating after applying filter on a particular layer. The filter set on the layer shouldn't reset itself as we navigate to other layers.

@skrishn
We are able to reproduce this and will deliver it in next code drop

@skrishn
Cause

We have a validLayers and nonVisibleValidLayers array to keep valid and non-valid layers separately.
On opening a layer we were disabling the visibility of other layers to show only one layer on the map (Enabling visibility when navigating to the layer list).
When the filter is applied we refresh the layer list to update features count.
On refreshing the layer, we were getting a callback where again layers were filtered but other layer visibility is disabled that is why they are getting added in the nonVisibleValidLayers array.
Hence some features don't get rendered on the map after applying the filter.

Solution

While navigating to the layer list update the layer visibility state and then refresh the layer list.

Dev Checklist

Items
Is required
Status
Comments

Required NLS changes?
No

Required Backward compatibility
No

Required 508
No

Need Sanitize?
No

Impact Analysis Report

Impacted Areas
Comments

Have Configuration changes?
No

Have Runtime changes?
Yes

Impact on RTL
No

Impact areas
- layer list   - map

Impact on runtime
Appropriate features should render on the map after applying the filter on a layer and navigating to other layers.


Get another block of IPs from Networks for ROSA

Describe the issue
In AWS SEA Test we have a ROSA cluster with some randomly chosen IP blocks.
Additional context

Not needed yet until we set up ROSA in Prod

Definition of done

Connect with WAM(?) and ask for another block of IPs like we initially did with SEA Prod

There's a meeting Julian is attending Monday for this, we will know more

We've asked them in Rocket Chat #devops-rosa-operations

Warren: We're not ready for this, too much in flux about the design.  The upgrade isn't going well and we need to sort this out.

Waiting until we firm up our networking and until ROSA is ready to go to prod.  Changes to ROSA network possibly coming from Ryan J

still relevant

not relevant


Add Mixed Case

Allow for continuous parents in discrete variables.
Use multilinear logistic regression.
Add flag for conditional linear gaussian or mixed.
First case draw variables as before, second case draw variables random

Consider instead of using flag a ratio


Discussion: Approaches to multi-tenancy with EDVs

On the 13th of May WG call a good discussion on the considerations that surround the different approaches to expressing multi-tenancy in URLs used for communication with an EDV server was had, brought about primarily due to a conversation on how best to achieve discovery see #12. Currently in the specification today the tenancy (essentially the EDV instance abstraction) is expressed through a URL path convention (e.g https://example.com//docs/), another common approach to multi-tenancy is to leverage sub-domains (e.g https://.example.com/docs/). However there are distinct tradeoffs that come with these different approaches, for example when using the subdomain approach EDV instances would be fire-walled from one and other when interacting via a User Agent (browser). Where as when using the path based URL convention, mechanisms like /.well-known endpoints cannot be used as a discovery tool for EDV instances.

Discussed on the 24th June WG call, proposed resolution is for the edv spec to remain agnostic or permissive of both models for multi-tenancy. The spec should include language that documents this position.

Since EDV's use ZCAPs for authorization, multi-tenant systems can be constructed in a variety of different ways.


fix: use path module instead of hardcoded '/'

This PR fixes #535 as current implementation is different for windows or unix systems.

/rtm

@all-contributors please add @Vetsoo for code


Building silx against development version of numpy

When the version of numpy installed contains the git tag (like in the https://anaconda.org/scientific-python-nightly-wheels), silx fails to build due to the version label which cannot be parsed.

I cannot reproduce the issue with the branch adding support for numpy2 (PR #4080).
Did you had the issue with the current tip of the project? Then this should be fixed by #4080

Fixed by PR #4080


IBM + Kdtree DEBUG mode fix

IBM + KDtree error with gfortran debug mode is fixed.
Speed inside the body is added in IBMClass.
Surface integrals are computed more efficiently .

@stecolumbus, any ideas of why so many test cases are crashing now?

All test passed, ready for merge? Did you fix the file ending? I guess we can fix that in a different pull request.


Support multiple nodes and OCP in local cluster

As discussed in https://github.com/kubevirt/ovs-cni/pull/1, it is desirable to test the code with multiple nodes and on different platforms too. In order to accomplish that, we should copy more of KubeVirt cluster helpers.
It is not clear to me yet how to do it, not pile unused and unrelated code from kubevirt here yet.

I will take this one

Resolved in #5


swift 3 Cannot inherit from non-open class 'JTAppleDayCellView' outside of its defining module

I am getting this error and I believe the classes need to marked as open vs. public so they can be used outside of the module where they are defined.

I merged your request.
You can close this request if all is now well.


Allow search queries with @searchable to retrieve more than 1000 items.

Which Category is your question related to?
GraphQL/ @searchable
Amplify CLI Version
4.27.1
What AWS Services are you utilizing?
ElasticSearch Service, Amplify, Appsync
I would like to know if it is, in any way, possible to retrieve over 1000 elements per request from my ElasticSearch instance via the @searchable in amplify, using the query generated by Codegen.
If trying to increase the limit in the GraphQL query over 1000 I receive the error:
"List size cannot exceed 1000"
I have found the following guide on how to do it with the scan request : Here but I couldn't Replicate with the Search query generated by codegen.
to reproduce: create a model with @searchable tag, insert over 1000 elements and try to  execute a query with a limit over 1000.

@celobpm you could do this with a lambda resolver: https://docs.amplify.aws/cli/graphql-transformer/directives#function
However, if you are trying to load that much data, I would recommend pagination which is supported out of the box using the nextToken parameter of the search query.

@edwardfoyle , thanks! i wil try with the lambda resolver!  the nextToken is too slow for a few situations when i need many items returned.


GF Check Fonts

I get this error:
Start
Traceback (most recent call last):
  File "checkfamily.py", line 18, in <module>
    script_path = glob.glob(r'/Users/*/Library/Application Support/Glyphs/Scripts/mf-glyphs-scripts')[0]
IndexError: list index out of range
End

Are you copying and pasting the scripts into the macro window?
The scripts have to be installed into the Glyphsapp script folder.
Your Glyphsapp scripts folder path should be the following:
/Users/your-username/Library/Application Support/Glyphs/Scripts.
Could you paste the location of your Glyphsapp scripts folder please?
glob.glob is like regex for retrieving paths.
Cheers,
Marc

They are properly installed inside the Glyphs script folder,  but inside a "MarckFolei" folder, instead of the default "mf-glyphs-scripts"

Marc, you actually don't have to use glob, you can also do something like here
import sys, os
parentDir = os.path.abspath('..')
if parentDir not in sys.path:
    sys.path.insert(0, parentDir)

The advantage is that you make one less assumption about where the Glyphs/Script directory is located.

@impallari That explains everything ;)
@graphicore there is another issue by doing it your way. I'll write this up tomorrow.

I got my way from the Glyphs forum.

@graphicore yep, worked a charm.
Updated in e511928
closed

Just updated the script from e511928.
Here is the error I get:
Start
Traceback (most recent call last):
  File "checkfamily.py", line 19, in <module>
    from QA import (
ImportError: No module named QA
End

So what dependency should I install for the QA module?

Hey Alexei,
in glyphs, close all fonts. Then type this into the macro window/
import os
print os.path.abspath('..')

What do you get?

I get:
/

Does your version of checkfamily.py have:
script_path = os.path.abspath('..')
if script_path not in sys.path:
    sys.path.append(script_path)
from QA import (
    glyphs,
    fontinfo,
    metrics,
)

ahhh, I found the issue.
missing init.pyc

Ok, I've updated.
Repull the repo and try again please.

Works now. Thank you!

After running fix fonts for GF spec script on https://github.com/alexeiva/YesevaOne/commit/28c4adb11c11c450eca9ac81fd3796ed91c06de9
The QA script produces this error. Before all was okay.
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/vanilla/vanillaBase.py", line 196, in action_
    self.callback(sender)
  File "checkfamily.py", line 77, in buttonCallback
    main(**self.w.__dict__)
TypeError: main() takes no arguments (24 given)

Nevermind. The issue disappeared after restarting Glyphs

I keep getting this error every second time I run check Font.
Any ideas what the cluster is all about?
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/vanilla/vanillaBase.py", line 196, in action_
    self.callback(sender)
  File "checkfamily.py", line 83, in buttonCallback
    main(**self.w.__dict__)
TypeError: main() takes no arguments (28 given)


npm ERR! 404 Not Found: pupperteer@latest

npm install --save pupperteer
npm ERR! code E404
npm ERR! 404 Not Found: pupperteer@latest

npm ERR!      C:\Users\sebas\AppData\Roaming\npm-cache\_logs\2018-08-22T20_17_28_361Z-debug.log

It might not be related but whenever I use pupperteer in my application and deploy it Heroku it keeps crashing.

@Sebastian-Nielsen puppeteer is the correct, isn`t pupperteer


Supporting PostgreSQL

Please review previous closed issues before filling out a new one! Duplicate issues will be closed without comment.
It seems uploading PostgreSQL is not supported yet, in contrary of other DBs such as MySQL. So, it would be great if LeetHub v3 will support PostgreSQL.

Hi @snowphone,
Good finding! I have added PostgreSQL to the supported languages in 2c41f29.
Version v0.0.8 should be available in the store soon :)

Wow... you're so fast! After the release, I'll test it too :)

It works 😀


Review for PyScript 2022.12.1

https://github.com/pyscript/pyscript/releases/tag/2022.12.1

The site is effectively broken:

Wondering if there's a way to first pin it to an older version.