Tensor-to-primitive/detensoring HLO conversion

HLO can only represent tensors, meaning that values that should not be tensors are still wrapped in them and operated on just as if they were real dense data. This is most easily visible with loops and conditionals, where the loop iterator initialization, increment, and condition are all in tensors. This results in host readbacks and a bunch of other extraneous work when really these should just be modeled as simple primitive values (i32/index, etc).
For example, this input loop:
  func @main() -> tensor<i32> attributes {iree.module.export} {
    %cst = constant dense<1> : tensor<i32>
    %cst_0 = constant dense<3> : tensor<i32>
    %0 = iree.do_not_optimize(%cst) : tensor<i32>
    %1 = iree.do_not_optimize(%cst_0) : tensor<i32>
    %2 = "xla_hlo.while"(%0) ( {
    ^bb0(%arg0: tensor<i32>):	// no predecessors
      %3 = "xla_hlo.compare"(%arg0, %1) {comparison_direction = "LT"} : (tensor<i32>, tensor<i32>) -> tensor<i1>
      "xla_hlo.return"(%3) : (tensor<i1>) -> ()
    },  {
    ^bb0(%arg0: tensor<i32>):	// no predecessors
      %3 = xla_hlo.add %arg0, %arg0 : tensor<i32>
      "xla_hlo.return"(%3) : (tensor<i32>) -> ()
    }) : (tensor<i32>) -> tensor<i32>
    return %2 : tensor<i32>
  }

is turned into the following CFG:
func @main() -> tensor<i32> attributes {iree.module.export} {
  %cst = constant dense<1> : tensor<i32>
  %cst_0 = constant dense<3> : tensor<i32>
  %0 = iree.do_not_optimize(%cst) : tensor<i32>
  %1 = iree.do_not_optimize(%cst_0) : tensor<i32>
  br ^bb1(%0 : tensor<i32>)
^bb1(%2: tensor<i32>):	// 2 preds: ^bb0, ^bb2
  %3 = "xla_hlo.compare"(%2, %1) {comparison_direction = "LT"} : (tensor<i32>, tensor<i32>) -> tensor<i1>
  %4 = extract_element %3[] : tensor<i1>
  cond_br %4, ^bb2(%2 : tensor<i32>), ^bb3(%2 : tensor<i32>)
^bb2(%5: tensor<i32>):	// pred: ^bb1
  %6 = xla_hlo.add %5, %5 : tensor<i32>
  br ^bb1(%6 : tensor<i32>)
^bb3(%7: tensor<i32>):	// pred: ^bb1
  return %7 : tensor<i32>
}

Which then after lowering through flow has the condition dispatched to the device and the condition read back (via flow.tensor.load):
func @main() -> tensor<i32> attributes {iree.module.export} {
  %cst = constant dense<1> : tensor<i32>
  %cst_0 = constant dense<3> : tensor<i32>
  %cst_1 = constant dense<1> : vector<3xi32>
  %0 = iree.do_not_optimize(%cst) : tensor<i32>
  %1 = iree.do_not_optimize(%cst_0) : tensor<i32>
  br ^bb1(%0 : tensor<i32>)
^bb1(%2: tensor<i32>):	// 2 preds: ^bb0, ^bb2
  %3 = flow.ex.stream.fragment(%arg0 = %cst_1 : vector<3xi32>, %arg1 = %2 : tensor<i32>, %arg2 = %1 : tensor<i32>) -> tensor<i1> {
    %8 = flow.dispatch @main_ex_dispatch_0::@main_ex_dispatch_0[%arg0 : vector<3xi32>](%arg1, %arg2) : (tensor<i32>, tensor<i32>) -> tensor<i1>
    flow.return %8 : tensor<i1>
  }
  %4 = flow.tensor.load %3 : tensor<i1>
  cond_br %4, ^bb2(%2 : tensor<i32>), ^bb3(%2 : tensor<i32>)
^bb2(%5: tensor<i32>):	// pred: ^bb1
  %6 = flow.ex.stream.fragment(%arg0 = %cst_1 : vector<3xi32>, %arg1 = %5 : tensor<i32>) -> tensor<i32> {
    %8 = flow.dispatch @main_ex_dispatch_1::@main_ex_dispatch_1[%arg0 : vector<3xi32>](%arg1) : (tensor<i32>) -> tensor<i32>
    flow.return %8 : tensor<i32>
  }
  br ^bb1(%6 : tensor<i32>)
^bb3(%7: tensor<i32>):	// pred: ^bb1
  return %7 : tensor<i32>
}

If instead we found these host-only values (even if only scalar tensors to start) we could run the whole loop in the host VM and avoid the readback.
This is also visible in inputs that have dynamic update slices (translated through to flow.tensor.update), where the current loop iterator value is needed to know where to update the tensor (mapping back to which timestep is being processed, etc). These updates need to be recorded into the command buffer on the host which means that we perform a readback to effectively compute an offset and then throw it back to the device.
Other enhancements around indirect dispatch and dynamic flow.tensor.update (#1160) will make some of these cases not so bad when device->host data dependencies really do exist, however if we can remove all of the trivial cases without relying on that we'll have much more readable IR and much lower overhead at runtime.

This is still critically important and will become moreso next quarter. Tagging @silvasean as we've talked about it before in case he knows someone who may be willing to work on this (it'd be useful upstream in MLIR for IREE as well as anything routing through MHLO/TOSA/etc).

Hi @benvanik ,
I have been contributing to MLIR (specially the spv dialect, with a lot of help from @antiagainst 😄 ) for a while (https://reviews.llvm.org/people/commits/14762/). I would love to contribute to IREE as well.
Is this issue still something that needs to be picked up?
I had a quick look and got the gist of what needs to be done. If it's still fine work on that, I will inspect it in more detail.

Yeah! It'd be useful to sketch out the full pipeline of MHLO|TOSA -> linalg-on-tensors -> IREE's flow and see where this may be able to be upstreamed (if at all). Brainstorming a bit:

Since we are migrating to linalg-on-tensors as our input dialect over the next few months it's possible that something could be done there which would let such an analysis/pass happen after MHLO/TOSA/etc such that it can be shared across all of them. This may also help solve a lot of the corner cases and avoid needing to specialize things to any other input dialect - such as handling broadcasting behavior. Instead of having specific patterns for converting mhlo.add to std.addi, for example, we let the existing patterns for linalg do that and instead perform the simplification on linalg.generic/indexed_generic ops (and those may literally be the only two ops we deal with, which is why it would be so nice!).
The major work that then happens could be a parameterized analysis pass that goes through IR and determines which values chains should be detensored. The pass would be looking for which values flowed between other tensor ops vs. those that were extracted out into scalars, and maybe in the future have parameters like "limit detensoring to ops producing a tensor of <= 4/16/32 elements" (so we could pull out vector<4xi32> for example). If a value is determined to be detensored the conversion should be fairly straightforward then and may just be the addition of a helper method in linalg that takes the body of a linalg.generic/indexed_generic and clones it out.
For example:
%A = ...
%B = ...
%tensor = linalg.generic ins(%A, %B : tensor<i32>, tensor<i32>) -> tensor<i32> {
 ^bb0(%a: i32, %b: i32) :
  %c = addi %a, %b: i32
  linalg.yield %c : i32
}
%scalar = extract_element %tensor[] : tensor<i32>

The analysis would look at the extract_element's source %tensor, note that it was a candidate for detensoring (single element, etc), and then clone out the op:
%A = ...
%B = ...
%a = extract_element %A[] : tensor<i32>
%b = extract_element %B[] : tensor<i32>
%scalar = addi %a, %b: i32

The nice thing here is that if that was applied greedily then if whatever was producing %A or %B could be detensored then the next step would do that too all the way up the chain. If the transformation was done cloning then if there were other users of the value that couldn't be detensored the op would still remain for those uses (if %tensor was used by other actual tensor operations/etc).
This may end up just being some canonicalizations on linalg ops looking for this kind of thing - then the greedy application is taken care of by MLIR's canonicalization pass. For example, a pattern on linalg.generic that checked to see if any user of the result was extract_element and if so cloned the body out/updated usage may be the entirety of it! If more control was desired then mlir::applyOpPatternsAndFold would let this happen outside of canonicalization (useful for example if we wanted to use analysis results).

An alternative to all of this is to do something specific to HLO - which has the advantage of being something that would work today in IREE but is more difficult to do as each of the dozens of MHLO ops would need to be supported. You'd still be performing the analysis but then instead of touching linalg ops you'd be mapping things like mhlo.add : tensor<i32> -> std.addi : i32 directly. It could also be hacked together under integrations/tensorflow/compiler/ before seeing how it may be upstreamed into tensorflow's repo.
This may even be useful as an experiment before doing something like the above linalg approach as it'll help identify where the transformation is useful and act as a verification for whether the linalg-based one is performing well. Iterating in the simpler scope here on what the analysis pass looks like before then working to upstream and apply it to linalg may help to get the design in a good state for broader review on MLIR's discourse.
Writing additional MHLO->std patterns in addition to the ones already being written for converting to linalg-on-tensors feels like a lot of work and code to maintain, though, so even if it takes a little time for IREE's linalg-on-tensor support to come online it could still help others looking to use linalg-on-tensors today and then be extremely reusable for all MLIR users in the future (even if they are not using IREE).
Happy to chat more about this on discord or here!

Thanks a lot for the detailed response. This all sounds really exciting and I am eager to contribute.
I am a newbie so please excuse any possible misunderstanding/not-really-getting-it.
From your description, it seems that the more immediate course of action would be to do detensoring exclusively to HLO (the 2nd alternative above). This might not be as general and useful as doing so for linalg on tensors but would serve as a good experiment that can later on guide a similar effort but within linalg on tensors.
From my side, I don't mind working on something specific to HLO that might be reverted later and replaced with a more general/useful implementation. In any case, it will be a wonderful learning experience and I can help with "porting" those optimizations to linalg on tensors later on.
Based on that, my naive understanding is that the first concrete steps would be the following:

Add a pass in integrations/tensorflow/compiler that would search for HLO ops with scalar tensor operands. It would then re-write those ops into std ops that work with scalar types.
For this, I think we can take a similar approach to how canonicalization is handled in tablegen in general. In particular, we might add an ODS flag like: canBeDetensored. And for ops with this field set to true, an <op>::getDetensoringPattern.
In a way, that pass would be quite similar to how the Canonicalizer pass work in MLIR.

Does that make sense to you?

Makes sense! Starting small with simple cases likely to be encountered from normal tensorflow code will help prevent the need for an exhaustive MHLO pattern mapping. For example, using the simple test linked in the description means you only need to support compare and add ops. If you take the tblgen approach like you mention then the initial work may mostly be setting up the new td files and build system goo :)
A larger, more realistic example would be https://github.com/google/iree/blob/473186214a19447932b02bc78d9cf47a332b2dc3/iree%2Ftest%2Fe2e%2Fmodels%2Funidirectional_lstm.mlir#L100 - probably a good goal for initial proof of concept as the source tf.while ends up as that mhlo.while and supporting that case will enable almost anything using tf.while to work.
A larger representative example will also help flush out any issues that may be discovered: I suspect there are cases where we'll need to carry along information through the CFG and getting some concrete examples will help figure out next steps. A particular gotcha will be cases where there's no unidirectional flow (so not host -> device or device -> host but instead device -> host -> device or some other kind of sandwich). Crossing that bridge when we come to it is fine and anything you were able to get together would be really valuable in informing that follow-on work!

Oh nice to see you here, @KareemErgawy-TomTom! :D  Just wanted to say hi and thanks for considering contributing to IREE!

Oh nice to see you here, @KareemErgawy-TomTom! :D  Just wanted to say hi and thanks for considering contributing to IREE!

Update:
Apologies for taking almost a month to provide an update. IREE is a bit overwhelming and it took me sometime to find my way around and get familiar with the code base a bit more.
I spent the past few weeks:

studying the different dialects involved in some of the supported e2e flows,
getting my env setup (I have a macbook as a mian dev machine and it seems the project infrastructure is not there yet to fully support macOS so I revived an old Linux machine of mine 😄),
studying how the passes are setup and playing a bit with the code starting from iree-translate-main.cc changing some stuff to understand the behavior driving the translation process a bit better.

With that, I believe a somewhat good starting point would be:

Add a pass called something like mhlo::convertOpsWithScalarTensorsToStdOpsPass().
The pass would be inserted in buildFlowTransformationPipeline(OpPassManager& passManager) after self-insert-unmeta.

I still need to workout the details of how the pass will be actually implemented. But at least this is how I believe an MHLO-specific solution might look like.
Let me know if you have any comments and hopefully the next update would be much quicker.

Update:
Apologies for taking almost a month to provide an update. IREE is a bit overwhelming and it took me sometime to find my way around and get familiar with the code base a bit more.
I spent the past few weeks:

studying the different dialects involved in some of the supported e2e flows,
getting my env setup (I have a macbook as a mian dev machine and it seems the project infrastructure is not there yet to fully support macOS so I revived an old Linux machine of mine 😄),
studying how the passes are setup and playing a bit with the code starting from iree-translate-main.cc changing some stuff to understand the behavior driving the translation process a bit better.

With that, I believe a somewhat good starting point would be:

Add a pass called something like mhlo::convertOpsWithScalarTensorsToStdOpsPass().
The pass would be inserted in buildFlowTransformationPipeline(OpPassManager& passManager) after self-insert-unmeta.

I still need to workout the details of how the pass will be actually implemented. But at least this is how I believe an MHLO-specific solution might look like.
Let me know if you have any comments and hopefully the next update would be much quicker.

My gut is that there are two parts to this:

The actual conversion of all HLO's to corresponding scalar code (e.g. hlo.add -> std.addi or std.addf)
The policy of which HLO's we should convert.

For 1., we are moving to a world of linalg-on-tensors, so you should make this pass exclusively work on linalg on tensors (not on HLO). One benefit of this is that the payload of a rank-0 linalg-on-tensors op will already contain ops like std.addi/addf, so you won't need to do any conversion there! You will literally just be inlining the linalg op payloads!
For 2., my intuition is an algorithm that first identifies connected components of the use-def graph which consist exclusively of rank-0 tensor code, and then a policy that determines for each connected component whether it makes sense to detensorize. (for the purpose of connected components, a block argument is considered connected to its corresponding successor arg in the predecessor blocks).

My gut is that there are two parts to this:

The actual conversion of all HLO's to corresponding scalar code (e.g. hlo.add -> std.addi or std.addf)
The policy of which HLO's we should convert.

For 1., we are moving to a world of linalg-on-tensors, so you should make this pass exclusively work on linalg on tensors (not on HLO). One benefit of this is that the payload of a rank-0 linalg-on-tensors op will already contain ops like std.addi/addf, so you won't need to do any conversion there! You will literally just be inlining the linalg op payloads!
For 2., my intuition is an algorithm that first identifies connected components of the use-def graph which consist exclusively of rank-0 tensor code, and then a policy that determines for each connected component whether it makes sense to detensorize. (for the purpose of connected components, a block argument is considered connected to its corresponding successor arg in the predecessor blocks).

Thanks @silvasean for the comment. I went through the available docs and talks that I could find to understand the current status of linalg-on-tensors.
I have a few questions:

This might be trivial to you guys, but how would the syntax for a rank-0 linalg-on-tensors op (say linalg.generic) look like? I am trying to find an example that but couldn't find any and tried to play with some of the available examples and change them to work on rank-0 tensor but didn't manage to get any example to compile.
Is there any flow currently implemented anywhere (IREE or anything else) that implements some kind of lowering that passes through linalg-on-tensors? As far as I understood, IREE didn't migrate to use linalg-on-tensors yet. This is not really crucial and I can start prototyping with any snippet that has 0-rank ops, but just curious.
Is linalg-on-tensors the better place (as opposed to HLO) to prototype the detensoring logic at the moment? I mean in terms of maturity and completeness of implementation.
This might be totally dumb question, but if the end goal is to convert linalg (or HLO) ops to std ops whenever possible/beneficial (i.e. as a start when such ops work on 0-rank tensors), what difference does it make if we work with linalg-on-tensors vs linalg-on-buffers? I mean wouldn't an analysis on memrefs also work? I didn't research the answer to question very thoroughly so I might be missing something very obvious here.

Thanks @silvasean for the comment. I went through the available docs and talks that I could find to understand the current status of linalg-on-tensors.
I have a few questions:

This might be trivial to you guys, but how would the syntax for a rank-0 linalg-on-tensors op (say linalg.generic) look like? I am trying to find an example that but couldn't find any and tried to play with some of the available examples and change them to work on rank-0 tensor but didn't manage to get any example to compile.
Is there any flow currently implemented anywhere (IREE or anything else) that implements some kind of lowering that passes through linalg-on-tensors? As far as I understood, IREE didn't migrate to use linalg-on-tensors yet. This is not really crucial and I can start prototyping with any snippet that has 0-rank ops, but just curious.
Is linalg-on-tensors the better place (as opposed to HLO) to prototype the detensoring logic at the moment? I mean in terms of maturity and completeness of implementation.
This might be totally dumb question, but if the end goal is to convert linalg (or HLO) ops to std ops whenever possible/beneficial (i.e. as a start when such ops work on 0-rank tensors), what difference does it make if we work with linalg-on-tensors vs linalg-on-buffers? I mean wouldn't an analysis on memrefs also work? I didn't research the answer to question very thoroughly so I might be missing something very obvious here.

Example:

#map = affine_map<() -> ()>
module  {
  func @tensor(%arg0: tensor<f32>, %arg1: tensor<f32>) attributes {iree.module.export} {
    %0 = linalg.init_tensor [] : tensor<f32>
    %1 = linalg.generic {indexing_maps = [#map, #map, #map], iterator_types = []} ins(%arg0, %arg1 : tensor<f32>, tensor<f32>) outs(%0 : tensor<f32>) {
    ^bb0(%arg2: f32, %arg3: f32, %arg4: f32):  // no predecessors
      %2 = addf %arg2, %arg3 : f32
      linalg.yield %2 : f32
    } -> tensor<f32>
    return
  }
}

@MaheshRavishankar @nicolasvasilache  can give you pointers. There is enough working for you to start this work I think.

For prototyping feel free to do it wherever you want. HLO might be slightly easier if you want to try to slot this into our existing pipelines for now (the bulk of the algorithm will be the same -- converting it to linalg-on-tensors will be a simplification). But we want this pass to work well for TOSA (and other frontends) and the way to do that is with linalg-on-tensors.

At the place in the pipeline that we need to run this, there is only linalg on tensors. The analysis on memrefs would also be much more difficult.

Example:

#map = affine_map<() -> ()>
module  {
  func @tensor(%arg0: tensor<f32>, %arg1: tensor<f32>) attributes {iree.module.export} {
    %0 = linalg.init_tensor [] : tensor<f32>
    %1 = linalg.generic {indexing_maps = [#map, #map, #map], iterator_types = []} ins(%arg0, %arg1 : tensor<f32>, tensor<f32>) outs(%0 : tensor<f32>) {
    ^bb0(%arg2: f32, %arg3: f32, %arg4: f32):  // no predecessors
      %2 = addf %arg2, %arg3 : f32
      linalg.yield %2 : f32
    } -> tensor<f32>
    return
  }
}

@MaheshRavishankar @nicolasvasilache  can give you pointers. There is enough working for you to start this work I think.

For prototyping feel free to do it wherever you want. HLO might be slightly easier if you want to try to slot this into our existing pipelines for now (the bulk of the algorithm will be the same -- converting it to linalg-on-tensors will be a simplification). But we want this pass to work well for TOSA (and other frontends) and the way to do that is with linalg-on-tensors.

At the place in the pipeline that we need to run this, there is only linalg on tensors. The analysis on memrefs would also be much more difficult.

Update:

A few days ago, I reached out to @nicolasvasilache and he also supported the idea that detensoring should be implemented directly on linalg-on-tensors. Nicolas also provided some more details to help get the ball rolling.
I started a WIP review for the first steps here: https://reviews.llvm.org/D96271. I will leave the explanation of what is done so far and how it is implemented to the docs in the review :).

It would be great if you can have a look and provide any feedback that can push this further.

Detensoring is becoming a priority for performance optimization work in IREE, so we're taking a closer look at integrating the upstream work into IREE (and possibly extending it as needed) now.

Sorry for taking sooo much time for integrating detensoring. I just wrote a message to Sean on Discourse a few hours ago that I tried to follow the changes in IREE but since I don't have enough time, I couldn't follow the fast pace of how things are going.
I still would love to give this a try if someone from IREE can help me. I can provide more details about the current detensoring implementation in linalg-on-tensors and patch it whenever needed, and whoever can help me from the IREE team (maybe @ScottTodd) can give pointers on where to start.
If you have the time and energy to collaborate with me on this, it would be great.

I'm just getting up to speed on this now, but here's what I'm trying:

Insert passManager.addNestedPass<FuncOp>(mlir::createLinalgDetensorizePass()); between these passes here: https://github.com/google/iree/blob/1ac7c5b6224538e482c3e20267685ad1584751a5/iree/compiler/Dialect/Flow/Transforms/Passes.cpp#L109-L110
Run iree-translate -iree-input-type=mhlo -iree-mlir-to-vm-bytecode-module -iree-hal-target-backends=dylib-llvm-aot on some input that uses control flow (TF while, TOSA if, etc.), and check the output, like: https://gist.github.com/ScottTodd/c361a5bfc87827df55b52ab90bd779c3

It looks like detensoring is working how we'd want (at least for this case), but we have some IR that needs further folding to lower through the rest of our pipeline. One case from that gist is here:
  %6 = zexti %5 : i1 to i8
  %7 = tensor.from_elements %6 : tensor<1xi8>
  %8 = linalg.tensor_collapse_shape %7 [] : tensor<1xi8> into tensor<i8>
  %9 = flow.tensor.load %8 : tensor<i8>

IR like that is surviving all the way down to something like this:
  ^bb3(%13: i32):  // pred: ^bb1
    %14 = tensor.from_elements %13 : tensor<1xi32>
    %15 = flow.ex.stream.fragment(%14) : (tensor<1xi32>) -> tensor<i32> =
        (%arg2: tensor<1xi32>) -> tensor<i32> {
      %17 = flow.tensor.reshape %arg2 : tensor<1xi32> -> tensor<i32>
      flow.return %17 : tensor<i32>
    }
    %16 = hal.tensor.cast %15 : tensor<i32> -> !hal.buffer_view
    return %16 : !hal.buffer_view
  }

we shouldn't need to go i32 -> 1xi32 -> i32, and that tensor.from_elements should be turned into some flow.* op (or just be folded away)

RE tensor.from_elements - it looks like I added a flow.tensor.splat that we could lower this to, then our flow.tensor.load canonicalization can just look for flow.tensor.splat and pull out the value (better than mixing dialects). flow.tensor.splat doesn't do anything yet (it should be lowered into a hal.command_buffer.fill in flow->hal conversion) but to start if we fold them all away it's fine to leave that TODO.

(I don't know if I like where it's located - this should be under Dialect/Flow/Conversions/ - but https://github.com/google/iree/blob/cd247ca50cfeeb3ff0800aa7190bbeed94b1b70d/iree/compiler/InputConversion/Common/ConvertUpstreamToIREE.cpp is where the tensor.from_elements -> flow.tensor.splat conversion would live)

I can work on these 2 changes that will clean-up the detensored code in the above example:

Create a rewrite pattern from tensor.from_elements to flow.tensor.splat.
Create a canonicalization pattern to extract flow.tensor.splat, flow.tensor.load op pairs to the underlying primitive value.

Unless someone else did that, I can do it quickly.

I did start on those at https://github.com/ScottTodd/iree/commits/detensorize , and I have some IR notes in this gist. Feel free to work on those patterns, starting from my changes or from scratch :)
I can try finding/making a test model that will show the performance benefits from detensoring, since our open source coverage is a bit sparse in that area.

From discussion on Discord, seems like there are some linalg.generic tensorizations missing as well?
https://cdn.discordapp.com/attachments/689906000043573354/867108835838656522/unknown.png

The linalg.tensor_collapse_shape folding would be a good one for upstream.

From discussion on Discord, seems like there are some linalg.generic tensorizations missing as well?
https://cdn.discordapp.com/attachments/689906000043573354/867108835838656522/unknown.png

Here's a more controlled repro of that: https://gist.github.com/ScottTodd/e6b035b658a4b0e05a9336dba1dc8452

@MaheshRavishankar The behavior in the example you posted is due to the specific cost model used be default. Right now, we have 2 models:

PureControlFlowDetectionModel (used by default): which basically looks for detensorable ops involved in conditional branches and from that tries to discover loops and if-conditions. This is why the 2nd linalg.generic in the picture you posted is detensored while the others aren't.
AggressiveDetensoringModel (used only in test code at the moment): which looks for all detensorable ops (regardless of their involvement in loops or conditions and detensors them. I guess this model would have detected the other 2 linalg.generic ops in the example.

I think we can either:

Extend the first model to detensor ops whose results are used in branches.
Add a new model that detensors ops whose result are used in branches and combine it with the first model above.
Both are equivalent approaches. WDYT?

@MaheshRavishankar The behavior in the example you posted is due to the specific cost model used be default. Right now, we have 2 models:

PureControlFlowDetectionModel (used by default): which basically looks for detensorable ops involved in conditional branches and from that tries to discover loops and if-conditions. This is why the 2nd linalg.generic in the picture you posted is detensored while the others aren't.
AggressiveDetensoringModel (used only in test code at the moment): which looks for all detensorable ops (regardless of their involvement in loops or conditions and detensors them. I guess this model would have detected the other 2 linalg.generic ops in the example.

I think we can either:

Extend the first model to detensor ops whose results are used in branches.
Add a new model that detensors ops whose result are used in branches and combine it with the first model above.
Both are equivalent approaches. WDYT?

Its cool that you have these models! That should be easy to control then. I think extending the first one to detensor ops whose results are used in branches makes sense to me right now.

I think extending the first one to detensor ops whose results are used in branches makes sense to me right now.

Agreed! The most impact this will have to start is preventing readbacks on loops, so it'd be great to see that handled.

The program in control_flow_test.py also shows how this will help: https://gist.github.com/ScottTodd/34523eb5850a0884c2f3c2e342ce69e3

Nice. Some further work on 1d-1 element vs 0d should take care of the last
bits.
On Wed, Jul 21, 2021, 11:55 AM Scott Todd @.***> wrote:

The program in control_flow_test.py
https://github.com/google/iree/blob/main/integrations/tensorflow/e2e/control_flow_test.py
also shows how this will help:
https://gist.github.com/ScottTodd/34523eb5850a0884c2f3c2e342ce69e3
[image: image]
https://user-images.githubusercontent.com/4010439/126543843-24a9358e-62e3-430c-bb64-2c2d2837a55e.png
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google/iree/issues/1159#issuecomment-884418198, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AADYVAFAIFKMO67HFWCVKWLTY4J3DANCNFSM4LPTE5PQ
.

Continuing to make progress here. With the refactoring in https://github.com/google/iree/pull/6586 and passes in https://github.com/google/iree/blob/main/iree/compiler/Dialect/Flow/Transforms/Passes.cpp like:
  passManager.addNestedPass<FuncOp>(createFusionOfTensorOpsPass());
  passManager.addNestedPass<FuncOp>(mlir::createCSEPass());
+ // NOTE: must run flow->tensor ops and canonicalization after detensorizing.
+ passManager.addNestedPass<FuncOp>(mlir::createLinalgDetensorizePass());
  passManager.addPass(memref::createResolveShapedTypeResultDimsPass());
  passManager.addNestedPass<FuncOp>(
      IREE::Flow::createConvertToFlowTensorOpsPass(
          /*runBeforeDispatchRegionFormation=*/true));
+ passManager.addNestedPass<FuncOp>(mlir::createCanonicalizerPass());
  passManager.addNestedPass<FuncOp>(
      IREE::Flow::createDispatchLinalgOnTensorsPass());

Test model status:

xla_ops/while.mlir translates successfully
tosa_ops/if.mlir fails (and is not fully detensored, see the cost model discussions above)
collatz.mlir from control_flow_test.py fails

The Collatz model ends up with IR like this (full IR dump here):
// -----// IR Dump After LinalgDetensorize //----- //
...
^bb5(%28: f32):  // pred: ^bb1
  %29 = tensor.from_elements %28 : tensor<1xf32>
  %30 = linalg.tensor_collapse_shape %29 [] : tensor<1xf32> into tensor<f32>
  %31 = hal.tensor.cast %30 : tensor<f32> -> !hal.buffer_view
  return %31 : !hal.buffer_view
}
...
// -----// IR Dump After ConvertToFlowTensorOps //----- //
...
// -----// IR Dump After Canonicalizer //----- //
...
^bb5(%18: f32):  // pred: ^bb1
  %19 = flow.tensor.splat %18 : tensor<f32>
  %20 = hal.tensor.cast %19 : tensor<f32> -> !hal.buffer_view
  return %20 : !hal.buffer_view
}
...
// from flow.tensor.splat -> hal.command_buffer.fill_buffer lowering
error: 'hal.command_buffer.fill_buffer' op operand #4 must be 32-bit signless integer, but got 'f32'

Need some different way to get from the f32 scalar to a hal.buffer_view? Or that shouldn't be detensored? (I can think of a few ways to get around this, but I'm wondering what would make the most sense - @benvanik ?)

Here's some IR from the TOSA if test before/after: https://gist.github.com/ScottTodd/24aea1903bebe41dc74af234ee325b68. I don't yet understand why the %3 = flow.tensor.load %2 : tensor<i8> is not getting turned into a %0 = hal.buffer.load<%buffer_1 : !hal.buffer>[%off] : i8 like it is in the baseline.

For the f32 fill we should insert a bitcast - which I don't think exists upstream or in the VM yet. We can allow the hal op to take any 32-bit value (or anything <=4 bytes) and then when converting hal->vm can insert the cast. I had thought I needed this for something else and added it but maybe in some long-lost branch.

For the f32 fill we should insert a bitcast - which I don't think exists upstream or in the VM yet. We can allow the hal op to take any 32-bit value (or anything <=4 bytes) and then when converting hal->vm can insert the cast. I had thought I needed this for something else and added it but maybe in some long-lost branch.

bitcast could be coming to upstream (standard ops) with @GMNGeoffrey 's https://llvm.discourse.group/t/rfc-introduce-a-bitcast-op/3774 + https://reviews.llvm.org/D105376

Ah maybe that's what I was remembering - needing it, seeing that work getting done, but then it not landing and me getting distracted and forgetting why I needed it :P

I'll plan on implementing that f32 fill via a bitcast. May have some more specific questions as I get into the details :)

tosa_ops/if.mlir fails (and is not fully detensored, see the cost model discussions above)

I am working on patching the cost model to account for that situation.

Here is a patch for this: https://reviews.llvm.org/D107358.

Here is a patch for this: https://reviews.llvm.org/D107358.

Nice! The TOSA if program looks like this with that commit, after detensoring and cleanup:
func @if_true_test(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub} {
  %c10_i32 = constant 10 : i32
  %0 = hal.tensor.cast %arg0 : !hal.buffer_view -> tensor<i1>
  %1 = hal.tensor.cast %arg1 : !hal.buffer_view -> tensor<i32>
  %2 = tensor.extract %0[] : tensor<i1>
  %3 = tensor.extract %1[] : tensor<i32>
  cond_br %2, ^bb1, ^bb2(%3 : i32)
^bb1:  // pred: ^bb0
  %4 = tensor.extract %1[] : tensor<i32>
  %5 = addi %4, %c10_i32 : i32
  br ^bb2(%5 : i32)
^bb2(%6: i32):  // 2 preds: ^bb0, ^bb1
  %7 = flow.tensor.splat %6 : tensor<i32>
  %8 = hal.tensor.cast %7 : tensor<i32> -> !hal.buffer_view
  return %8 : !hal.buffer_view
}

(it still fails to completely compile with similar logs to my comment above, but this does simplify it further :D)

do you have CSE running after that? I'd expect %4 to be deduped with %3

do you have CSE running after that? I'd expect %4 to be deduped with %3

Not immediately, but that does get picked up by a later run: https://github.com/google/iree/blob/c3ea7a879abfa86aa7c0994afba2b19d7354a459/iree/compiler/Dialect/Flow/Transforms/Passes.cpp#L180-L183

good to look at the IR after that point - otherwise like above it looks like there are still tensor ops in the inner loop :)
if you are running this via iree-opt you can build your own pipeline: -pass-pipeline=iree-flow-blah-blah,canonicalize,cse

I'm noticing that after detensoring and using the bitcast op, all of the computation in the collatz model was moved to the VM (IR dumps here), which then requires the -iree-vm-target-extension=f32 feature. There is no clear error message for that when the extension is not enabled, just a segfault here: https://github.com/google/iree/blob/36d3efcb3fb4fda86c4cbb947ebf17c8ef22131f/iree/compiler/Dialect/VM/Conversion/StandardToVM/ConvertStandardToVM.cpp#L156-L157 after https://github.com/google/iree/blob/36d3efcb3fb4fda86c4cbb947ebf17c8ef22131f/iree/compiler/Dialect/VM/Conversion/TypeConverter.cpp#L76-L83
Is that (requiring the f32 extension when detensoring is used) acceptable, or should those ops/types be converted using emulation in the VM?

We should fix it so that emits a proper error :P
We build f32 into the runtime by default so we should probably also enable the extension in the compiler by default. We'll want some modes for doing things like disabling f32 detensoring for devices that don't support it on the host, though I feel like the best route there is to not use models that use floats if your target hardware doesn't support it and you want tiny builds (no soft float).

As @silvasean mentioned, the default mode should have worked here, so most probably a bug from a quick look at the code. Will take a look...

I finally had the time to look into this, sorry I am in the process of changing jobs and moving from Berlin to Munich at the moment.
The issue is in detecting the control flow ops for the if condition. Here is what happens:

Detensoring is kicked off by the condition value of the cond_br %24, ^bb3, ^bb4 op (here).
%23 is defined by what appears from first-glance as a detensorable op.
Based on the current implementation, this detensorable op is added to the work list.
Somewhere down the line when this op is popped up from the work list, its operands are in-turn added to the work list. One of those operands is %22 which results from the reduction calculation.
This is where things breakdown. When detensoring eventually looks at the op defining %22, it discovers that this op is no detensorable and therefore conservatively fails to detensor all the discovered ops so far (here).

I think, it's time to make it a bit more flexible. The examples we encountered so far didn't need such flexibility.  Will look into that and again sorry for the slowness here, life is very busy at the moment.

I have a potential fix for the issue. It is more of removing a self-imposed restriction in the code than a fix. You can find the relevant commit here: https://github.com/ergawy/llvm-project/commit/0dbd1e8c28557be3b4d673120caa1f6c470a6ff3.
With this change, the only remaining linalg.generic op after detensoring in the reduction op as expected (and of course the other regression test pass expect for one that was written specifically to test that detesoring fails with the above mentioned restriction).
You can find the output of the previously failed example here: https://gist.github.com/ergawy/b528f1db884b59ec60d2d7ebac69776f.
@ScottTodd I think it makes sense to test with the above fix since the amount of changed code is tiny and then if things work smoothly, I will open a PR to MLIR. WDYT?

SGTM, thanks @ergawy ! (and no worries about being busy, the help is appreciated!)
I ran a build of IREE with detensoring enabled and that change, and the affected tests still build+pass.
I spent the last few weeks trying to find open source models with interesting flow control and couldn't actually import any of the models that I looked at :/. I think for now we'll have to live with the existing small scale programs for primary correctness + performance coverage.

I started a patch to address the above issue: https://reviews.llvm.org/D109965.

I'm hitting an issue that I think is related here that actually results in a compilation failure (as opposed to performance issues) because IREE doesn't handle i1 tensors well at all: https://github.com/google/iree/issues/3102#issuecomment-963409877

I'm hitting an issue that I think is related here that actually results in a compilation failure (as opposed to performance issues) because IREE doesn't handle i1 tensors well at all: #3102 (comment)

Does -iree-flow-enable-linalg-detensorize help there? It still isn't enabled by default. (There are many things overlapping here and we should try to keep the issues focused...)

It does not, unfortunately
Also, unrelated, the only mention I see of that flag in the codebase is in its definition. It should probably have some tests...

Also, unrelated, the only mention I see of that flag in the codebase is in its definition. It should probably have some tests...

Well yeah :P I spent about three weeks trying to generate/find representative test cases and didn't get very far. Any models I tried to compile hit frontend issues, either in their own frameworks or in TF->MLIR. The MLIR code has unit test coverage in MLIR core and is mostly an optimization in IREE itself. I have been wanting to flip the flag (https://github.com/google/iree/pull/6863) to enable coverage in the few tests (if, while, collatz, etc.) that are affected.

@GMNGeoffrey @ScottTodd Stale P1 item here, please take a look when you get a chance and update or deprioritize!

I think @rsuderman was going to look at getting this work over the finish line.


Unable to use in two different pages

I am planning to use the autocomplete input in two pages, I do not need a map view, I simply need to provide the user with options upon typing. On my signup page, the google places autocomplete works fine, however if I include this in any other page, nothing appears upon typing.
const GooglePlacesInput = (props) => {

    return (

        <GooglePlacesAutocomplete
            placeholder="Select hometown"
            query={{
                key=APIKEY,
                language: "en",
                types: '(cities)'
            }}
            onPress={data => props.selectTown(data)}
        />
    )
}

as I mentioned this works as expected on one file, with no issues. however when including it in any other page I get no results.

@fpolica91 or @conor909
Can you please provide a reproducible example of this by showing more detailed code or better yet, linking to a snack?


Error running annh5ad2rds.R

Hi to the developers,
Thank you very much for the thoughtful .h5ad to Seurat conversion R script.I tried using that on the data I downloaded form the STOMICS database, however I ran into a problem.
Here is the code I ran and the error ( I ran this with R/4.2)
sama0023@arxcss:~/STOMICS$ Rscript annh5ad2rds.R --infile ./Stomics_data/E16.5_E2S3.MOSTA.h5ad --outfile ./Stomics_data/E16.5_E2S3.MOSTA.RDS
Registered S3 method overwritten by 'SeuratDisk':
  method            from
  as.sparse.H5Group Seurat
Attaching SeuratObject

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

[1] "Converting h5ad to h5seurat file."
Warning: Unknown file type: h5ad
Creating h5Seurat file for version <IP_ADDRESS>00
Adding X as data
Adding X as counts
Adding meta.features from var
Adding spatial as cell embeddings for spatial
Warning: Cannot find a reduction named pca (PCs in varm)
Adding annotation_colors to miscellaneous data
Adding layer count as data in assay count
Adding layer count as counts in assay count
[1] "Loading h5seurat file."
Validating h5Seurat file
Initializing Spatial with data
Adding counts for Spatial
Adding feature-level metadata for Spatial
Initializing count with data
Adding counts for count
Adding reduction spatial
Adding cell embeddings for spatial
Adding miscellaneous information for spatial
Adding command information
Adding cell-level metadata
Warning: Invalid name supplied, making object name syntactically valid. New object name is n_genes_by_countslog1p_n_genes_by_countstotal_countslog1p_total_countsannotationRegulon...Acaa1aRegulon...AhrRegulon...Alx1Regulon...Alx4Regulon...Arid3aRegulon...ArntRegulon...Arnt2Regulon...ArxRegulon...Atf2Regulon...Atf3Regulon...Atf4Regulon...Atf6Regulon...Bach2Regulon...Barhl1Regulon...Barhl2Regulon...Bcl6Regulon...Bcl6bRegulon...Bclaf1Regulon...Bhlhe22Regulon...Bhlhe40Regulon...Bhlhe41Regulon...BmycRegulon...Borcs8Regulon...BptfRegulon...Brf1Regulon...CebpaRegulon...CebpbRegulon...CebpdRegulon...CebpgRegulon...CebpzRegulon...Chd1Regulon...CicRegulon...ClockRegulon...Cnot3Regulon...Creb1Regulon...Creb3Regulon...Creb3l1Regulon...Creb3l2Regulon...CrxRegulon...CtcfRegulon...Cux1Regulon...Cux2Regulon...DbpRegulon...Dbx1Regulon...Ddit3Regulon...Ddx4Regulon...Dlx1Regulon...Dlx2Regulon...Dlx3Regulon...Dlx5Regulon...Dlx6Regulon...Dmrt2Regulon...E2f1Regulon...E2f2Regulon...E2f3Regulon...E2f4Regulon...E2f5Regu [... truncated]
Adding miscellaneous information
Adding tool-specific results
Warning message:
Cannot add objects with duplicate keys (offending key: spatial_), setting key to 'spatialla_'
Error in<EMAIL_ADDRESS>, c("x", "y")) :
  undefined columns selected
Calls: unique -> [ -> [.data.frame
Execution halted

Can you please explain how this can be corrected?
Many thanks in advance,
Shani.

I did a but of digging and the error is due to failing of line:
<EMAIL_ADDRESS>c('x', 'y')])
As there are no x and y coordinates in meta.data.
How can I fix that? Is there an external location I could retrieve this information from?
Many thanks,
Shani.

Hi, thanks for your question,  please wait a moment, we will study it.

Thanks for the reply @junhouhuiBGI .
Can I then maybe get access to the corresponding data that is suitable to be processed by this approach? Thanks.

Hi, i also ran the script for converting anndata to seurat object for visium data, but i get a error message: Error in if (!x[[i]]$dims) { : the condition has length > 1. May i know how i solve this issue? Thank you

Hi to the developers,
Thank you very much for the thoughtful .h5ad to Seurat conversion R script.I tried using that on the data I downloaded form the STOMICS database, however I ran into a problem.
Here is the code I ran and the error ( I ran this with R/4.2)
sama0023@arxcss:~/STOMICS$ Rscript annh5ad2rds.R --infile ./Stomics_data/E16.5_E2S3.MOSTA.h5ad --outfile ./Stomics_data/E16.5_E2S3.MOSTA.RDS
Registered S3 method overwritten by 'SeuratDisk':
  method            from
  as.sparse.H5Group Seurat
Attaching SeuratObject

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

[1] "Converting h5ad to h5seurat file."
Warning: Unknown file type: h5ad
Creating h5Seurat file for version <IP_ADDRESS>00
Adding X as data
Adding X as counts
Adding meta.features from var
Adding spatial as cell embeddings for spatial
Warning: Cannot find a reduction named pca (PCs in varm)
Adding annotation_colors to miscellaneous data
Adding layer count as data in assay count
Adding layer count as counts in assay count
[1] "Loading h5seurat file."
Validating h5Seurat file
Initializing Spatial with data
Adding counts for Spatial
Adding feature-level metadata for Spatial
Initializing count with data
Adding counts for count
Adding reduction spatial
Adding cell embeddings for spatial
Adding miscellaneous information for spatial
Adding command information
Adding cell-level metadata
Warning: Invalid name supplied, making object name syntactically valid. New object name is n_genes_by_countslog1p_n_genes_by_countstotal_countslog1p_total_countsannotationRegulon...Acaa1aRegulon...AhrRegulon...Alx1Regulon...Alx4Regulon...Arid3aRegulon...ArntRegulon...Arnt2Regulon...ArxRegulon...Atf2Regulon...Atf3Regulon...Atf4Regulon...Atf6Regulon...Bach2Regulon...Barhl1Regulon...Barhl2Regulon...Bcl6Regulon...Bcl6bRegulon...Bclaf1Regulon...Bhlhe22Regulon...Bhlhe40Regulon...Bhlhe41Regulon...BmycRegulon...Borcs8Regulon...BptfRegulon...Brf1Regulon...CebpaRegulon...CebpbRegulon...CebpdRegulon...CebpgRegulon...CebpzRegulon...Chd1Regulon...CicRegulon...ClockRegulon...Cnot3Regulon...Creb1Regulon...Creb3Regulon...Creb3l1Regulon...Creb3l2Regulon...CrxRegulon...CtcfRegulon...Cux1Regulon...Cux2Regulon...DbpRegulon...Dbx1Regulon...Ddit3Regulon...Ddx4Regulon...Dlx1Regulon...Dlx2Regulon...Dlx3Regulon...Dlx5Regulon...Dlx6Regulon...Dmrt2Regulon...E2f1Regulon...E2f2Regulon...E2f3Regulon...E2f4Regulon...E2f5Regu [... truncated]
Adding miscellaneous information
Adding tool-specific results
Warning message:
Cannot add objects with duplicate keys (offending key: spatial_), setting key to 'spatialla_'
Error in<EMAIL_ADDRESS>, c("x", "y")) :
  undefined columns selected
Calls: unique -> [ -> [.data.frame
Execution halted

Can you please explain how this can be corrected?
Many thanks in advance, Shani.

Hi, I have read your description of annh5ad2rds.R's error running.
You can run these python codes before you run the convert script:
import anndata
import numpy as np
import pandas as pd

ann_data = anndata.read('../demo_data/E16.5_E2S3.MOSTA.h5ad')
arr = np.array(ann_data.obs.index.str.split('_').to_list(), dtype=np.uint32)
ann_data.obs['x'] = pd.Series(arr[:, 0], index=ann_data.obs.index)
ann_data.obs['y'] = pd.Series(arr[:, 1], index=ann_data.obs.index)
ann_data.write_h5ad('../demo_data/E16.5_E2S3.MOSTA.test.h5ad')

May these can help you solve the problem.
Wait for your reply~

Hi, could you show me the data or how to get the data you use if it's unclassified？

Hi, i also ran the script for converting anndata to seurat object for visium data, but i get a error message: Error in if (!x[[i]]$dims) { : the condition has length > 1. May i know how i solve this issue? Thank you

Hi, could you show me the data or how to get the data you use if it's unclassified？

This issue has been closed because it has been inactive for 14 days.
You may reopen this issue if it has been closed in error.
Thx for your report.


seat: Assume there is always a keyboard device

This makes it so that users first keystroke doesn't get 'eaten' This makes cage easier to use, especially when foot is the only client

It seems a bit weird. For example, I don't want my touchscreen-only device to show a keyboard capability.

It seems a bit weird. For example, I don't want my touchscreen-only device to show a keyboard capability.

Maybe I can try something else, if I move the mouse, it doesn't eat the first keystroke, so maybe I need to include that extra call to update_capabilities, but see what is populating &seat->keyboard_groups somehow...

Argh, wait, the more I test, the more I realize this is common to wayfire, and labwc, and I think even sway, not just cage, it's a wlroots issue


Add support for no_std

Second attempt at support no_std, this time without splitting the crate into two. Instead, a use_std feature is used, which is enabled by default.
Fixes #13

Updated with given feedback.

@sfackler I'm happy with this modulo a bit of bikeshedding, and I'll leave it up to you to merge

Also thanks @Amanieu, I'm excited to see more libs start picking up the no_std standard!

Updated again, it should work on 1.0.0 now.

Looks good to me, thanks!

I'm going to take a pass over the docs to make sure everything looks right and then push a release.

Made a couple minor tweaks: #71


Table function runs twice when it is in an Angular-Material mdTab directive

I'm using angular-material, and ngTasty and I cannot figure out why the tasty table is running twice when it is added inside one of the angulars md-tab directive. Something either in material or tasty. Of course my table function is making a rest call which is now firing twice.
http://jsfiddle.net/s46gs95e/

For a reason I don't understand yet, when ngTable was contained inside the Material Tab, the 'params' was detected change after the initialization, and was again calling the xhr.
I fixed by making sure that ngTable will run max one xhr at a time.

Also just release on bower the version 0.5.9 with this fix in it!


Enable Google Assistant TemperatureControlTrait for climate devices without mode

Some zwave climate devices, like the:

Danfoss Devolo Home Control Radiator Thermostat

don't provide an operation mode setting since they don't implement the COMMAND_CLASS_THERMOSTAT_MODE zwave class, see:
https://github.com/home-assistant/home-assistant/blob/2cc626309285dba1ad21f0ffd65017ccc2dfa7d4/homeassistant/components/zwave/discovery_schemas.py#L54
https://github.com/home-assistant/home-assistant/blob/9c52a3ce223c06f2d29b0e61c59d660dd180e5b8/homeassistant/components/climate/zwave.py#L107
This excludes them from providing the TemperatureSettingTrait and will therefor not be available in the google assistant, see:
https://github.com/home-assistant/home-assistant/blob/457708cbdaaf75a9eb5a904f395ef8c7a2793cb9/homeassistant/components/google_assistant/trait.py#L533
For these climate devices, the TemperatureControlTrait can be used to provide querying and setting the temperature from google assistant.

Hi @KarlVogel,
It seems you haven't yet signed a CLA. Please do so here.
Once you do that we will be able to review and accept this pull request.
Thanks!

I'm not really a fan of this as it splits the handling of thermostat devices. That Trait in GA is more intended for things like Ovens.
We should probably hold off on this until home-assistant/architecture#22 has been resolved. That would ensure all climate devices actually follow the same format and we don't have to keep hacking around all the random differences between all of them.

According to google the trait is for:

Trait for devices (other than thermostats) that support controlling temperature, either within or around the device. This includes devices such as ovens and refrigerators.

So I figured it wasn't as hacky as the BrightnessTrait for media players :-)

True, less hacky than that. I do think the problem should be resolved at the DOMAIN level though within HA.

I think that a better fix would be that if a thermostat has no operation mode, we set it to "AUTO" and we limit the modes to AUTO.

I agree with @balloob. Although report the state to GA as heatcool, rather than auto. (GA doesn't seem to display a temp if you use auto).

This PR seems to have gone stale. Closing it.


raise when there is only one type of error present in fetch_offsets

fixes #720

Codecov Report

Merging #723 into master will decrease coverage by 1.77%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #723      +/-   ##
==========================================
- Coverage   84.96%   83.18%   -1.78%
==========================================
  Files          26       26
  Lines        3378     3378
  Branches      539      539
==========================================
- Hits         2870     2810      -60
- Misses        365      423      +58
- Partials      143      145       +2

Impacted Files
Coverage Δ

pykafka/simpleconsumer.py
79.65% <0%> (-3.93%)
:arrow_down:

pykafka/managedbalancedconsumer.py
74.3% <0%> (-16.67%)
:arrow_down:

pykafka/connection.py
71.27% <0%> (-8.52%)
:arrow_down:

pykafka/broker.py
88.96% <0%> (-1.95%)
:arrow_down:

pykafka/rdkafka/producer.py
87.22% <0%> (-1.6%)
:arrow_down:

pykafka/topic.py
84.04% <0%> (-1.07%)
:arrow_down:

pykafka/cluster.py
70.86% <0%> (-0.87%)
:arrow_down:

pykafka/protocol.py
96.66% <0%> (-0.28%)
:arrow_down:

pykafka/handlers.py
97.01% <0%> (ø)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1f85b96...217865b. Read the comment docs.

@iciclespider These seem like valid issues, here's a PR that addresses them https://github.com/Parsely/pykafka/pull/729


FlushAsync only saving 1 row to database

Hi,
I am trying to output multiple rows to the database with the SQL Output binding but each time only one rows gets inserted.
I've got the following code:
    public class StoreMetadata
    {
        [FunctionName("StoreMetadata")]
        public async Task RunAsync(
            [QueueTrigger("extracted-blobs", Connection = "BlobStorage")]string myQueueItem,
            ILogger log,
            [Sql("dbo.Ibans",
            CommandType = System.Data.CommandType.Text,
            ConnectionStringSetting = "SqlConnectionString")] IAsyncCollector<Iban> ibans
)
        {
            log.LogInformation("Store metadata function has been triggered!");

            DetectedMetadataEvent detectedMetadataEvent = null;

            try
            {
                detectedMetadataEvent = JsonSerializer.Deserialize<DetectedMetadataEvent>(myQueueItem);
            }
            catch (Exception ex)
            {
                // TODO: Handle
                log.LogError($"Error while deserializing: {ex.Message}");

            }

            if (detectedMetadataEvent != null)
            {
                if (detectedMetadataEvent.PiiEntities != null)
                {
                    foreach ( CustomPiiEntity entity in detectedMetadataEvent.PiiEntities)
                    {
                        if (entity != null)
                        {
                            if (entity.Category.Equals(PiiEntityCategory.InternationalBankingAccountNumber.ToString())){
                              await ibans.AddAsync(new Iban
                                {
                                    AccountNumber = entity.Text,
                                    SubCategory = entity.SubCategory,
                                    DocumentId = 4004,
                                    ConfidenceScore = entity.ConfidenceScore,

                                });
                            }
                        }
                    }
                }

                   await ibans.FlushAsync();
            }

        }
    }

When  debugging I can see that the variable 'ibans' does have multiple objects so I don't understand why only 1 rows gets inserted into the database, maybe because '_rowLock' is 1?

Hi @lucyzhang929, I am not assigning primary keys within the code since I have a field called ID  as auto increment in my database.

@Ansiem I can confirm this is a bug when dealing with identity columns and I'll be working on a fix.
In the meantime you can workaround this by moving the call to FlushAsync() into the for loop so you call it after every item you add.
foreach ( CustomPiiEntity entity in detectedMetadataEvent.PiiEntities)
                    {
                        if (entity != null)
                        {
                            if (entity.Category.Equals(PiiEntityCategory.InternationalBankingAccountNumber.ToString())){
                              await ibans.AddAsync(new Iban
                                {
                                    AccountNumber = entity.Text,
                                    SubCategory = entity.SubCategory,
                                    DocumentId = 4004,
                                    ConfidenceScore = entity.ConfidenceScore,

                                });
                               await ibans.FlushAsync();
                            }
                        }
                    }

@Ansiem I just published a new version of the extension with the fix for this issue, let us know if you're still seeing any issues after updating!
https://github.com/Azure/azure-functions-sql-extension/releases/tag/v0.1.319-preview

@Charles-Gagnon issue has been resolved in the update, thank you!


download_strategy: Fix the timeout value.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

References https://github.com/Homebrew/brew/pull/14629.


Bring Install.ps1 ERB up to date

Here is a similar install.p1 that has been kept up to date to allow great compatibility - https://github.com/puppetlabs/puppetlabs-chocolatey/blob/master/templates/InstallChocolatey.ps1.erb

[x] has TLS 1.1/1.2 fixes (done with #111)
[ ] Fix PowerShell Output Redirect Bug in PowerShell v2/3
[ ] Allow switching out 7zip for built-in compression to remove the download from the internet
[ ] Fix any other formatting

cc @mwrock @docwhat

Related to reviewing changes for #109 in #111.

Also see #113 as an alternative that uses the official version.

Another alternative would be to just use the official version... drop it in files/ and use cookbook_file.  Then updating it would be as easy as grabbing the latest and dropping it in.


Fix coverage tracking

Codecov Report

Merging #3 into master will decrease coverage by 28.78%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master       #3       +/-   ##
===========================================
- Coverage     100%   71.21%   -28.79%
===========================================
  Files           3       11        +8
  Lines          35      198      +163
===========================================
+ Hits           35      141      +106
- Misses          0       57       +57

Impacted Files
Coverage Δ

spec/lib/graphiti/controller/configuration_spec.rb

spec/lib/graphiti/model_spec.rb

spec/graphiti_spec.rb

lib/graphiti/controller/graphql_request.rb
62.5% <0%> (ø)

lib/graphiti.rb
100% <0%> (ø)

lib/graphiti/router/controller_function.rb
61.9% <0%> (ø)

lib/graphiti/controller.rb
75% <0%> (ø)

lib/graphiti/attribute.rb
79.16% <0%> (ø)

lib/graphiti/router.rb
48.14% <0%> (ø)

lib/graphiti/router/schema_builder.rb
42.1% <0%> (ø)

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d04b11e...d85b4fb. Read the comment docs.


Uncaught AttributeError in aider@dev line 8

Aider version: 0.54.7-dev
An uncaught exception occurred:
Traceback (most recent call last):
  File "aider@dev", line 8, in <module>
    sys.exit(main())
  File "main.py", line 640, in main
    coder.run()
  File "base_coder.py", line 727, in run
    self.run_one(user_message, preproc)
  File "base_coder.py", line 764, in run_one
    message = self.preproc_user_input(user_message)
  File "base_coder.py", line 753, in preproc_user_input
    return self.commands.run(inp)
  File "commands.py", line 204, in run
    return self.do_run(matching_commands[0][1:], rest_inp)
  File "commands.py", line 179, in do_run
    return cmd_method(args)
  File "commands.py", line 1079, in cmd_settings
    settings = format_settings(self.parser, self.args)
  File "format_settings.py", line 13, in format_settings
    show = scrub_sensitive_info(args, parser.format_values())
AttributeError: 'NoneType' object has no attribute 'format_values'

Thanks for trying aider and filing this issue.
The fix is available in the main branch. You can get it by installing the latest version from github:
aider --install-main-branch

# or...

python -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git

If you have a chance to try it, let me know if it works better for you.


[WIP] NER Enhancement

Added code to,

store checkpoints
load checkpoints
batch inference
f1 score

Note and questions:

Using the same MAX parameters.
300d vector
Added F1 score from addons. Should we have this? If yes, I might need to update this to the latest version as I have created F1 PR to addons around last year-end.
Should I add sentence sorting for batch inference?

cc @frreiss

@SSaishruthi I believe the tests need to be updated to reflect the new batch input (list of string)?

@MLnick Updated test and result fields


Update field.html to match latest allauth version

The file /allauth_themes/bootstrap/templates/allauth/elements/field.html looks quite different from current allauth https://codeberg.org/allauth/django-allauth/src/branch/main/allauth/templates/allauth/elements/field.html
For example, it misses https://codeberg.org/allauth/django-allauth/commit/791a1116fdca1368553b3be63aa5e2907371b16f
It would be great to have the templates up-to-date to the current allauth version.
BTW Thanks for your work on these bootstrap themes, it's great!

Wohoo, thanks a lot. Yeah, I found it very nice, and it saved me a lot of time, so thanks again for sharing your proof of concept.


Possible implementations for v1.1

Add blacklist, cleared list, and critical/overdue list
Import google maps or whatever maps
2a: Remove browser, replace with maps location and full info
2b: Minimise info on the cards column
2c. Address validation
Add fields: current amount owed, multiple phone numbers, date last cleared, lifetime owed, total days overdue, times borrowed
3a: Sort by name, sort by amount owed
3b: Update and improve add command
3c: Add command and UI button to borrow and repay
Add password login
Add add tag, filter by tags

Note the abc stuff don't have to be done by the same person doing the main one, nor do they have to be up by 1.1

for 2. I think google maps would mainly be used by the profile tab display (expands when the person card listed in the left column is clicked) which will display all the info of that person.

alright edited


Push latest changes to Cocoapods

When installing version 1.1.0 via Cocoapods, I am not able to get the latest code. I cannot see alwaysShowFloatingLabel on JVFloatLabeledTextField.

version 1.1.1 is now available


Rasa x

Rasa Open Source version
rasa==2.8.0
Rasa SDK version
Rasa SDK Version==2.8.6
Python version
3.8
What operating system are you using?
Linux
What happened?
Getting this error when running rasa x aftyer installation
Command / Request
rasa x

Relevant log output
ImportError: cannot import name 'local' from 'rasax.community'

I am also facing the same issue, does anyone know the solution?

No solution given yet

do you have rasa x installed? (running rasa --version should tell you). Important to note though that we are no longer maintaining Rasa X .

Yes I have rasa x installed

the most recent rasa x version which is compatible with Rasa Open Source 2.8 is rasa x 1.0.4 (1.1 and later are only compatible with Rasa Open Source 3+).

Okay so how do I install rasa x with the specific version

@robertson-oa mhm which way did you use to install rasa x?

@tmbo From the rasa X documentation: pip3 install rasa-x --extra-index-url https://pypi.rasa.com/simple
But I want to be able to install specific version 1.0.4

Right, you should be able to do that using pip3 install rasa-x==1.0.4 --extra-index-url https://pypi.rasa.com/simple

similarly, @SumitVerma11 please either install a version that corresponds to your installed Rasa Open Source version https://rasa.com/docs/rasa-enterprise/changelog/compatibility-matrix/.
TL;DR:
If you are using Rasa Open Source 2.x: pip3 install rasa-x==1.0.4 --extra-index-url https://pypi.rasa.com/simple
If you are using Rasa Open Source 3.x: pip3 install rasa-x==1.1.3 --extra-index-url https://pypi.rasa.com/simple

@tmbo It worked thanks for your feedback


Openssl is rejected by Apple for using non-public API

https://github.com/openssl/openssl/issues/2545

Testing:

https://ci.appveyor.com/project/ingenue/hunter/build/1.0.1958
https://travis-ci.org/ingenue/hunter/builds/254368342?utm_source=github_status&utm_medium=notification

Uploading:

https://ci.appveyor.com/project/ingenue/hunter/build/1.0.1960
https://travis-ci.org/ingenue/hunter/builds/254384024?utm_source=github_status&utm_medium=notification

Released: https://github.com/ruslo/hunter/releases/tag/v0.19.34


Add OptionalAtmospheres from SpaceDock

This pull request was automatically generated by SpaceDock on behalf of SamBelanger, to add OptionalAtmospheres to CKAN.
Mod details:
name = /mod/1525/OptionalAtmospheres
author = SamBelanger
abstract = This mod adds atmospheres to planets and moons which don't have any
license = MIT
Homepage = https://forum.kerbalspaceprogram.com/index.php?/topic/163142-wip-130-optionalatmospheres-oa-v1491/
description = Hello KSP players, here is an interesting mod that adds atmospheres to those planets and moons which don't have atmospheres of the Kerbol system.
README :

Download the mods Kopernicus (1.3.0-4 IMPORTANT !!), ModularFlightIntegrator (It is already integrated in Kopernicus!) and the mods ModuleManager (2.8.0 IMPORTANT !!)

Put the mods files in the GameData folder

Start the game (KSP)

Have fun

And if you have a suggestion or find bugs or errors, contact us<EMAIL_ADDRESS>or make a private message in the forum of KSP or otherwise) Thank you very much!
Please direct questions about this pull request to SamBelanger.

SpaceDock Remake (CLOSE PLZ)

C'mon plz close


equest had invalid authentication credentials

Thank you for making this, I'm trying to implement this in my app but every time when i hit record I'm getting error from google
equest had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project

can you explain how can fix this
thank you

you try api create key credentials new Authenticating to a Cloud API Service

how can solve this issue i just include   in  my index.html file but still getting this issue

Did you edit or this file recorder.js ?


Installing ROS2 in windows 11 pro

Hi There,
Is it possible to install ROS2 with windows 11 Pro?
I am reading the documentation and it is saying that Only Windows 10 is supported.
Also, in the readme file ROS2 humble version link is given but newer version is jazzy.
will jazzy be supported or i should go with humble version?
Thanks in advance.

Hello, you can try to use docker inside windows 11


⚠ Bofrost.es has degraded performance

In 0fa73c8, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1619 ms

Resolved: Bofrost.es performance has improved in 4180648 after 9 minutes.


WAL: Extract error message

Currently wal.c uses pgmoneta_log_message in case of an error.
The error message should be extracted instead and displayed using an ERROR log

Got it. It should be easy as the format is declared in Postgres' documents.


Add getAccount and enhance account filtering

This PR:

Adds and accountFilter: AccountFilter param to getAllAccounts() in order to enable the narrowing-down of cached accounts returned
Adds getAccountByFilter API to replace now marked-for-deprecation getAccountBy APIs
Extends AccountFilter class to optionally include all the properties in AccountInfo except idToken and idTokenClaims so accounts are filterable by all of their searchable properties
Adds new AccountFilter properties to the set of matchers in CacheManager.getAccountsFilteredBy() as well as their respective private matching methods
Adds filtering by loginHint

Codecov Report

Merging #6499 (7bf4625) into dev (81d34b4) will decrease coverage by 4.52%.
Report is 326 commits behind head on dev.
The diff coverage is 20.00%.

Flag
Coverage Δ

msal-angular
96.73% <ø> (+0.22%)
:arrow_up:

msal-browser
85.43% <ø> (-1.04%)
:arrow_down:

msal-common
?

msal-core
?

msal-node
80.75% <ø> (-2.64%)
:arrow_down:

msal-node-extensions
41.76% <20.00%> (-33.88%)
:arrow_down:

msal-react
94.24% <ø> (-0.45%)
:arrow_down:

node-token-validation
?

Files Changed
Coverage Δ

...ions/msal-node-extensions/src/utils/Environment.ts
20.51% <0.00%> (-22.08%)
:arrow_down:

lib/msal-angular/src/constants.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.broadcast.service.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.guard.ts
90.78% <ø> (+0.64%)
:arrow_up:

lib/msal-angular/src/msal.interceptor.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.module.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.navigation.client.ts
93.33% <ø> (+0.47%)
:arrow_up:

lib/msal-angular/src/msal.redirect.component.ts
100.00% <ø> (ø)

lib/msal-angular/src/msal.service.ts
100.00% <ø> (ø)

lib/msal-angular/src/packageMetadata.ts
100.00% <ø> (ø)

... and 60 more

... and 177 files with indirect coverage changes

Hi,
Just a quick question regarding this change. Is there a reason why the interface IPublicClientApplication was not updated with the method definition changes? I don't know if I'm missing something
See example below:
Interface https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/d055ad8cb91961dbc0639861c618f11900485853/lib/msal-browser/src/app/IPublicClientApplication.ts#L49
Implementation https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/d055ad8cb91961dbc0639861c618f11900485853/lib/msal-browser/src/app/PublicClientApplication.ts#L245


Use BATS for tests

As BATS is better maintained and looks nicer than Urchin, I think we should try and see if it behaves better as well.
Its from @sstephenson, which is the creator of rbenv among other things.
Installation of BATS can be done by adding the following to .travis.yml
before_install:
  - sudo add-apt-repository ppa:duggan/bats --yes
  - sudo apt-get update
  - sudo apt-get install bats
script:
  - bats test/bats

Per https://github.com/nvm-sh/nvm/pull/601#issuecomment-219444480, BATS is bash-specific and thus can’t work for our needs.


Bulk deleting clusters should auto-focus input

Setup

Rancher version: 2.7-head
Browser type & version: Firefox

Describe the bug

The confirmation modal for bulk deletion of clusters requires clicking on the input before text can be entered. Typing without focus on the input can cause selections to change in the background.
To Reproduce

Create multiple clusters
Navigate to Cluster Managerment => Clusters
Select multiple cluster
Click Delete

Result
The confirmation modal is rendered without the "Enter {cluster-name} below to confirm:" focused.
Expected Result

"Enter {cluster-name} below to confirm:" focused is focused so that I can start typing to continue.
Screenshots

NOTE: As I start typing without focus on the input (0:07-0:09), selected clusters are changed behind the modal

https://user-images.githubusercontent.com/835961/204644694-22b14730-7ceb-4cff-80ac-d374e6eebd4f.mp4
Additional context

There's something in the epinio-dev branch that either adds this or provides a similar mechanism in promptRemove (@scures can provide further details). That branch needs to merge to master at some point, alternatively it should be done in a similar way

epinio-dev will not be merged within the 2.7 Q2 time frame, so the port needs to happen. it's pretty small


Feature message box extendable action

Feature
In order to give the user the possibility to add metadata to a MessageBoxAction this feature introduces a new action type MessageBoxExtendedAction. This allows the user to give actions e.g. id's and make them directly selectable by a testing framework like wdi5.
Implementation

To ensure downwards compatibility the old Actions can be used as before
To add metadata the old actions can be optionally extended by wrapping them with the new ExtendedAction using composition to enable a loosely coupling
To ensure future metadata extensions the metadata is defined as a generic object
To ensure an intuitive usage for the user polymorphism is implemented and the user can pass an ExtendedAction the same way to the action list as an old Action

Remarks

To ensure esLint doesn't mark the class keyword (used in this PR) as reserved keyword the ecmaVersion got updated from 5 to 6

Thanks for the review @akudev . As discussed I'll continue to work on the PR internally via Gerrit.


KAFKA-17214: Add 3.8.0 version to core and client system tests

This one should be cherry-picked to the 3.9 branch as well.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@cmccabe I cherry picked this to 3.9 as you will need these changes


Uvicorn error on startup + reloader not working properly

Problem description
I am running FastAPI inside docker, and the uvicorn reloader is not working as it should. I could not find an similar issue thus I am reporting it here to see if anybode ran into this.
The API works fine, but without reloader development work involves shutting down and up docker, plus such an error is probably not fit for production. It still detects changes in the code but only after 1 or 2 minutes, but every time it reloads / starts up it throws the following warning (likely related):
api_1  | INFO:     Uvicorn running on http://<IP_ADDRESS>:8000 (Press CTRL+C to quit)
api_1  | INFO:     Started reloader process [8] using statreload
api_1  | INFO:     Started server process [10]
api_1  | 2021-02-22 10:25:55 | uvicorn.error | {/usr/local/lib/python3.7/site-packages/uvicorn/_impl/asyncio.py:62} | INFO :  Started server process [10]
api_1  | INFO:     Waiting for application startup.
api_1  | 2021-02-22 10:25:55 | uvicorn.error | {/usr/local/lib/python3.7/site-packages/uvicorn/lifespan/on.py:26} | INFO :  Waiting for application startup.
api_1  | INFO:     Application startup complete.
api_1  | 2021-02-22 10:25:55 | uvicorn.error | {/usr/local/lib/python3.7/site-packages/uvicorn/lifespan/on.py:38} | INFO :  Application startup complete.
...
api_1  | WARNING:  StatReload detected file change in 'app/utils/colorgram.py'. Reloading...
api_1  | INFO:     Shutting down
api_1  | 2021-02-22 10:30:22 | uvicorn.error | {/usr/local/lib/python3.7/site-packages/uvicorn/_impl/asyncio.py:208} | INFO :  Shutting down
api_1  | INFO:     Waiting for application shutdown.
api_1  | 2021-02-22 10:30:22 | uvicorn.error | {/usr/local/lib/python3.7/site-packages/uvicorn/lifespan/on.py:43} | INFO :  Waiting for application shutdown.
api_1  | INFO:     Application shutdown complete.
api_1  | 2021-02-22 10:30:22 | uvicorn.error | {/usr/local/lib/python3.7/site-packages/uvicorn/lifespan/on.py:46} | INFO :  Application shutdown complete.
api_1  | INFO:     Finished server process [10]
api_1  | 2021-02-22 10:30:22 | uvicorn.error | {/usr/local/lib/python3.7/site-packages/uvicorn/_impl/asyncio.py:75} | INFO :  Finished server process [10]

My entrypoint.sh reads:
bash -c "uvicorn app.main:app --host <IP_ADDRESS> --port 8000 --reload"

Environment

OS: Docker (Linux)
FastAPI Version: 0.62.0
Python version: 3.7.9
Uvicorn: Running uvicorn 0.12.3 with CPython 3.7.9 on Linux

Additional context
Dockerfile
FROM python:3.7

# Set environment varibles
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV CODE_SRC=app

WORKDIR /code/
ENV PATH=$PATH:/code/
ENV PYTHONPATH=$PYTHONPATH:/code/

COPY pyproject.toml poetry.lock ./

RUN set -ex; \
    apt-get update ; \
    apt-get install ffmpeg libsm6 libxext6 -y ; \
    apt-get install jq -y ; \
    apt-get install -y --no-install-recommends git pkg-config ; \
    pip install poetry ; \
    poetry config virtualenvs.create false ; \
    poetry install --no-interaction --no-ansi ; \
    apt-get autoremove -y ; \
    apt-get clean -y ; \
    rm -rf /var/lib/apt/lists/*

COPY . .

ENTRYPOINT [ "./entrypoint.sh"]

Thanks!

It seems as though you’re copying your directory into the Docker container as opposed to mounting the directory from the host.
When you add files from the host into a container via the Dockerfile it’s not working off of the files on host, but rather compressing those files and uploading them permanently to the container. I don’t know the command you’re using to run the container so I’m not 100% certain this is your issue or not though.
In order to mount the directory from the host you need to add it to your Docker run command like this:
docker run -v /code:/home/myuser/code my_image
In the -v argument the directory to the left is the directory inside the container, and the directory to the right is the directory on the host machine. This will map that directory directly into the Docker container, without this the Docker container is unaware of changes to the files on the host machine without a restart.

Hi @Cleptomania thanks for helping out.
I am using docker-compose, where I map the root on the host to /code on the Docker:
version: "3"

services:

  api:
    build:
      context: ./app
    volumes:
      - .:/code/
    env_file:
      - ./.env
    ports:
      - "8000:8000"
    command: api

The command api just calls the entrypoint.sh with that argument (this is to account for different envs etc):

set -e

case "$1" in
  api|uvicorn)
    if [ "$DEVELOPMENT" == "1" ]; then
      echo 'DEVELOPMENT'
      bash -c "uvicorn app.main:app --host <IP_ADDRESS> --port 8000 --reload"
    else
      echo 'NOT ALLOWED'
    fi
  ;;
esac

I believe the mapping is correct, but I might be wrong?
This is my repo structure
FASTAPI
├─── .env
├─── .gitignore
├─── docker-compose.yml
├─── README.md
│
└───app
    ├───main.py
    └───Dockerfile

thanks!

Hi there, I can replicate the problem. Using an older version fixes the issue. (in my case 0.12.3, I didn't try any other)

Hi there, I just solved it by explicitly passing uvicorn the path where my development code lives:
--reload-dir app

so the command becomes
bash -c "uvicorn app.main:app --host <IP_ADDRESS> --port 8000 --reload --reload-dir app"

Hope this helps!

Thanks for the help here everyone! 👏  🙇
Thanks for reporting back and closing the issue @illpack  👍

Sorry for the long delay! 🙈  I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.


Inquiry about Tauri Implementation for Application Packaging

Might I inquire as to whether there are any plans to utilize Tauri for packaging purposes? At present, the Electron version's size and loading speed are rather considerable. As such, I was hoping to ascertain whether Tauri will see implementation for application packaging.

Thanks for raising this up! We will look into this.

I wrapped it locally using tauri and it works fine, but a few export buttons require calling the tauri API to work and require some effort to handle.

@ztjhz If possible, I am willing to study and propose a PR in the next two days.

Sure! We are happy to accept any PR. :)

@ztjhz  Packaged a mini BetterChatGPT in my fork release
Create a new mini branch if possible:laughing:It would be better if I could 水 a PR


MAID-2730: make p2p crypto-generic

I think you forgot to rebase off latest master cause there are 2 commits in this PR.

 Closing this in favor of #65


THRIFT-5633: [build infra] add ubuntu jammy in build infra

add ubuntu jammy in build infra

[x] Did you create an Apache Jira ticket?  (not required for trivial changes)
[x] If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
[x] Did you squash your changes to a single commit?  (not required, but preferred)
[x] Did you do your best to avoid breaking changes?  If one was needed, did you label the Jira ticket with "Breaking-Change"?
[x] If your change does not involve any code, include [skip ci] anywhere in the commit message to free up build resources.

yes i have been trying to resolve these during last few weeks.

the cross-test ones are swift, not sure if we recently merged a swift change broke it, or a new swift version broke it.

i guess due to uuid not properly working in swift, so is golang and rust, for which i tried to add in:

#2700
#2695

for the ruby case i had no clue;

for the java ssl errors, i had the impression that it was due to jdk bug but it was not a case because i had upgraded java to jdk 17
- #2686
and the error persists; because we have java and kotlin build covered in GitHub action so it should be fine

my plan is to gradually cover more languages in GitHub action and then fix the cross test actions; for travis, that will be fixed later if i have more time


MOM6 in GEOS

AGCM.rc.tmpl in coupled mode sets USE_DATASEA: 0 to use a live ocean model (MOM5).
This option has been replaced with OCEAN_MODEL: in GuestOcean_GridComp.F90.  Valid choices for OCEAN_MODEL: are DATASEA, MOM or MOM6.

I am closing this PR and will re-open with 2 separate PR's
The first one will be purely structural, where new code will be added but neither be compiled or exercised.
The second PR will have modifications that will exercise the newly added code, but will be zero-diff with respect to develop.


Set up machine and appropriate CI jobs to run the new playbook testing scripts

This will be putting any work from #842 and #864 into production kicked off via our CI tooling.
Will also require the installation of VM software (likely VirtualBox) onto one of our systems

To keep a log of the work being done :

There's a consistent error when running the playbooks in parallel, in that (it seems) when one playbook fails (due to an external reason), the other ones with stop instantly and say Connection to <IP_ADDRESS> closed by remote host. Hypothesis that this is connected either to vagrant halt or vagrant destroy.
The Windows section of the testScript will find the wrong IP address occasionally, due to the the way I put finding it into the script. As of #990 I believe this should work, however there are now issues of  previous runs leaving files around.
Certain roles in (mainly) the unix playbook timing out : ant-contrib,cmake,ccache,Nvidia-Cuda

The Connection to <IP_ADDRESS> closed by remote host issue wasn't tied to either vagrant halt or vagrant destroy. It seems it is, however, tied to the multiple VMs running at once in some way - which is yet to be determined. A workaround for this has been to reduce the build executors on the Jenkins slave to 1, although this is not ideal as the job now takes ~8 hours to complete.
Another potential fix is to start thetestScript in the Jenkins job to start at different times- This appeared to work in https://ci.adoptopenjdk.net/job/SXA-PlaybookCheckParallel/52/ wherein the failed OSs were due to timeout or unresponsive network issues.
The timeouts didn't occur when running a single playbook at once, either- though this may just be luck.

ayy

@Willsparker so this can be closed?

@karianna I think he's added that image onto the wrong issue as that one is for Dockerfile regular testing - https://ci.adoptopenjdk.net/view/Tooling/job/SXA-DockerfileCheck/
Saying that, with the addition of the retry operations in https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/995 we are probably very close to being able to close this - need a few more test runs and to re-enable the build/test operations within the script to validate not only that the playbooks work but that they work well enough to perform builds and execute tests.
We have the playbook for setting up a machine for this at https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/985 and now have a machine at https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/910 which can be used to run this in production.
Current job is on the tooling tab at https://ci.adoptopenjdk.net/job/SXA-PlaybookCheckParallel/ and will have the SXA- prefix removed (at least) once it's as stable as it can be (which is also true for the dockercheckfile job referenced above!)

We're still hitting the Connection to <IP_ADDRESS> closed by remote host error on a lot of runs of the job.
Just FYI, For windows, the following error message occurs due to whatever is closing the VMs, as this is the error message that is printed when ansible can't connect to the VM. Therefore, it can be considered as Windows equivalent to Connection closed by remote host :
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ConnectionError: HTTPSConnectionPool(host='<VM_IP_ADDRESS>', port=5986): Max retries exceeded with url: /wsman (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff67278add0>: Failed to establish a new connection: [Errno 113] No route to host',))

I've gone down a new road that completely removes the vagrant commands, other than vagrant up, and will start the playbook by doing it much like it has to be done with the Windows VMs- executes ansible-playbook on the machine and then points it at the VM. With #999, vmDestroy.sh is ran before testScript.sh is run so the VMs are still being destroy and cleared off the machine, once a new instance of the job is running, unless --retainVM is not specified- then it will destroy at the end of testScript.sh, however we can't do this on the SXA-PlaybookCheckParallel job, as that may close the other VMs that may still be running.
Initial impressions seem okay- There have been a few run throughs where finishing (succeeded or failed) VMs don't cause the other VMs to close, however this may be a bit of a false positive as the issue seems to be intermittent. If this works-around the connection closed issue, the build and testJDK scripts will have to be called in the same way as the Windows VM too.

https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1013
https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/992 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/999 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1000 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1003 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1007 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1016 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1020 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1022 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1025 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1026 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1029
https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1030 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1034 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1035 https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/1040
Docs: https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/993

I've got to the point where 4 playbooks are stable and consistently are building JDKs successfully . (running testScript.sh -v $OS -u https://github.com/Willsparker/openjdk-infrastructure/tree/git_check ) -  currently debugging the windows Playbook as it seems the win_get_url ansible module has started to corrupt downloads for some reason (#1039).

See : https://ci.adoptopenjdk.net/job/SXA-PlaybookCheckParallel/187/

Further to the above update - the ansible Module itself is currently broken in Ansible 2.9.x. I have rolled back both Jenkins Machines to ansible 2.5.1 and now all the playbooks are stable, and the various linux OSs can build a JDK and test it. Now just to get Windows to build the JDK and test it - though I've ran into a problem where after the playbook has been ran, the shared folder between the VM and the host machine doesn't appear, unless vagrant reload is executed.


Wrap models not machines?

@pat-alt
Congratualations on the launch of this new package 🎉  Great to have the integration with MLJ!
I'm not familiar with conformal prediction, but I nevertheless wonder why this package wraps MLJ machines rather than models. If you wrap models, then you will buy into MLJ's model composition. So, a "conformally wrapped model" will behave like any other model: you can insert in pipeline, can wrap in tuning strategy, and so forth.
New models in MLJ generally implement the "basement level" model API. Machines are a higher level abstraction for: (i) user interaction; and (ii) syntax for building learning networks which are ultimately "exported" as standalone model types.
Here are other examples of model wrapping in MLJ: EnsembleModel (docs), BinaryThresholdPredictor, TunedModel, IteratedModel. What makes things a little complicated is the model hierarchy: the model supertype for the wrapped model depends on the supertype of the atomic model. So for example, we don't just have EnsembleModel we have DeterministicEnsembleModel (for ordinary point predictors) and ProbabilisticEnsembleModel (for probabilistic predictors) but the user only sees a single constructor EnsembleModel; see here. (A longer term goal is to drop the hierarchy in favour of pure trait interface, which will simplify things, but that's a little ways off yet.)
Happy to provide further guidance.
cc @azev77

@ablaom currently MLJ models can give deterministic & probabilistic predictions (GLM predicts an entire distribution).
If methods for conformal predictive inference become more developed, can users automatically combine their favorite (appropriate) deterministic predictor w/ their favorite predictive interval method, to make probabilistic predictions?
For example:
currently  model = LGBMRegressor() gives deterministic predictions
would a user be able to create a new model composing a deterministic model LGBMRegressor w/ their favorite (appropriate) predictive interval method e.g. conformal(type=naive) to create a new model w/ probabilistic predictions?

For example:
currently model = LGBMRegressor() gives deterministic predictions
would a user be able to create a new model composing a deterministic model LGBMRegressor w/ their favorite (appropriate) predictive interval method e.g. conformal(type=naive) to create a new model w/ probabilistic predictions?

Yes, I guess that's the design I am suggesting. So, similar to the way BinaryThresholdPredictor wraps a probabilistic predictor and makes it deterministic.

Hi @ablaom 👋🏽  Thanks very much for this suggestion and sorry for the delayed response (been battling Covid this week while also trying to finish my JuliaCon proceedings submission 😅 ). I will implement this first thing once I turn back to working on this package some time next week 👍🏽

Implemented in #10, but will keep this open for now, because I still want to iron out a few things (some related questions here).

#18 should now be strictly in line with MLJ @ablaom, so I will close this. Will still need to figure out how to handle downstream tasks like evaluate in the future.


Create CODEOWNERS

Adds CODEOWNER file.
@randytqwjp Can you check if you can give me a write permission to the repository from the github configuration?
I lost it when I'm kicked out from the organization, and now I cannot add labels to issues, create commits in the repo etc.

@sanposhiho  sorry for the delay i missed this

@randytqwjp
Can you check if you can add me to the repository collaborator to grant the write permission? I cannot even merge this approved PR 😅

@sanposhiho seems like i cant.. let me merge the PR instead to see if it grants you any permission


A need for "culling"?

When adding a large number of items to the layout, the whole application gets very sluggish.
Here is an example of the standard demo with 200 extra items in the drop-down menu, the response-time lag is very noticable:

I think a system for not calculating stuff and skipping events on objects that are not visible will need to be added.
Culling is the term I heard from 3D graphics, don't know if there is another name for this concept in GUI space.

Interesting, yah that's slower than I would expect too. Are you using release or debug? I'd also expect that ~200 elements should still be fine. It might be a good time for some profiling. :)
Hmmm I also thought the dropdown wouldn't even create the menu item nodes until the dropdown menu opened. It shouldn't be too hard to add a check whether a parent is visible currently.
I was also considering doing a quadtree to check for collisions too and only re-render if the current element changed. That'd also make it way more efficient on CPU too.

I had to add a 100 items in the dropdown before it started lagging, but still it should be able to handle 10s of thousands before slowing down.
Culling non-visible elements should help. I did a basic profiling and it seemed that one of the layout proc's in fidget_dev/openglbackend took up the largest percentage of time. There was also a lot of time sent in the opengl calls draw calls.


Code Cleanup needed

Describe the bug

p_source_key in add_any_resources.sql is unused (corresponding call needs update)
com.ibm.fhir.persistence.jdbc.dao.impl.SetTenantDAO is unused

Additional context
n/a

PR waiting review


Reverse geocoding

Add entry to location's document about the name of the location. Use Google's geocoding services.

Solved around https://github.com/axelpale/tresdb/commit/b042ebb2d0c761587e48a22b0bcaf61964b71326


DOC: link DataArray.expand_dims to Dataset.expand_dims

[ ] Closes #xxxx
[ ] Tests added
[ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ] New functions/methods are listed in api.rst

Help navigation in the docs of going between the DataArray method and the Dataset method.

Thanks @raybellwaves !


KMP-3014: Enhanced error generation

KMP-3014
It's hard to handle errors generated by the gem in the App which uses it because the errors are too common. There is only Zoom::AuthenticationError and Zoom::Error for any other cases.
This PR improves error generation. The error classes added for common http codes.
https://marketplace.zoom.us/docs/api-reference/error-definitions is used for reference.

@rantler Feedback addressed.


Re-connect to last Pebble on the startup

This would make development much easier since I won't have to go to devices and wait for scan completion to test anything. I can take a stab at it if you wish.

Done in ble branch


[Merge] enable p2p networking for Merge milestone

PR Description
Fixed Issue(s)
related to #4504
Documentation

[X] I thought about documentation and added the documentation label to this PR if updates are required.

Changelog

[X] I thought about adding a changelog entry, and added one if I deemed necessary.

I had to lower down my changes\pr ratio :)


MM-18173 Removed getState from Add Channel Member method

Removed getState from Add Channel Member method.
Summary
Removed getState from Add Channel Member method since the Redux store is not looking for it. 3rd param for this method is a postRootID, which is not sent from this request.
Ticket Link
https://mattermost.atlassian.net/browse/MM-18173
Checklist
No changes to the UI or the result of the request.
Device Information
This PR was tested on:
iPhone X 12.4 (Simulator)
iPhone Xs Mac 12.3

/cherry-pick release-1.25