Pack/unpack fusion blocked by collapse_shape/expand_shape

We found there are some un-fused pack/unpack ops in models and I think it is due to the collapse_shape and expand_shape between their fusable producer and consumer ops. For example:
func.func @collapse_pack_and_unpack_expand(%arg0: tensor<4x3072x64xf32>, %arg1: tensor<3072xf32>, %arg2: tensor<32x256x8x1xf32>) -> tensor<256x12xf32> {
  %cst = arith.constant 0.000000e+00 : f32
  %0 = tensor.empty() : tensor<3072x64x4xf32>
  %1 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (d2, d0, d1)>, affine_map<(d0, d1, d2) -> (d0)>, affine_map<(d0, d1, d2) -> (d0, d1, d2)>], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0, %arg1 : tensor<4x3072x64xf32>, tensor<3072xf32>) outs(%0 : tensor<3072x64x4xf32>) {
  ^bb0(%in: f32, %in_0: f32, %out: f32):
    %10 = arith.addf %in, %in_0 : f32
    linalg.yield %10 : f32
  } -> tensor<3072x64x4xf32>
  // Collapse before packing
  %collapsed = tensor.collapse_shape %1 [[0], [1, 2]] : tensor<3072x64x4xf32> into tensor<3072x256xf32>
  %2 = tensor.empty() : tensor<384x256x8x1xf32>
  %pack = tensor.pack %collapsed outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [8, 1] into %2 : tensor<3072x256xf32> -> tensor<384x256x8x1xf32>
  %3 = tensor.empty() : tensor<384x32x8x8xf32>
  %4 = linalg.fill ins(%cst : f32) outs(%3 : tensor<384x32x8x8xf32>) -> tensor<384x32x8x8xf32>
  %5 = linalg.mmt4d ins(%pack, %arg2 : tensor<384x256x8x1xf32>, tensor<32x256x8x1xf32>) outs(%4 : tensor<384x32x8x8xf32>) -> tensor<384x32x8x8xf32>
  %6 = tensor.empty() : tensor<3072x256xf32>
  %unpack = tensor.unpack %5 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [8, 8] into %6 : tensor<384x32x8x8xf32> -> tensor<3072x256xf32>
  // Expand after unpacking
  %expanded = tensor.expand_shape %unpack [[0, 1], [2]] : tensor<3072x256xf32> into tensor<12x256x256xf32>
  %7 = tensor.empty() : tensor<256x12xf32>
  %8 = linalg.fill ins(%cst : f32) outs(%7 : tensor<256x12xf32>) -> tensor<256x12xf32>
  %9 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (d1, d0, d2)>, affine_map<(d0, d1, d2) -> (d0, d1)>], iterator_types = ["parallel", "parallel", "reduction"]} ins(%expanded : tensor<12x256x256xf32>) outs(%8 : tensor<256x12xf32>) {
  ^bb0(%in: f32, %out: f32):
    %10 = arith.addf %out, %in : f32
    linalg.yield %10 : f32
  } -> tensor<256x12xf32>
  return %9 : tensor<256x12xf32>
}

// -----// IR Dump After FormDispatchRegions (iree-flow-form-dispatch-regions) //----- //
util.func public @collapse_pack_and_unpack_expand(%arg0: !hal.buffer_view, %arg1: !hal.buffer_view, %arg2: !hal.buffer_view) -> !hal.buffer_view attributes {iree.abi.stub, iree.reflection = {iree.abi.declaration = "sync func @collapse_pack_and_unpack_expand(%input0: tensor<4x3072x64xf32>, %input1: tensor<3072xf32>, %input2: tensor<32x256x8x1xf32>) -> (%output0: tensor<256x12xf32>)"}} {
  %cst = arith.constant 0.000000e+00 : f32
  %0 = hal.tensor.import %arg0 "input0" : !hal.buffer_view -> tensor<4x3072x64xf32>
  %1 = hal.tensor.import %arg1 "input1" : !hal.buffer_view -> tensor<3072xf32>
  %2 = hal.tensor.import %arg2 "input2" : !hal.buffer_view -> tensor<32x256x8x1xf32>
  %3 = tensor.empty() : tensor<3072x64x4xf32>
  %4 = flow.dispatch.region -> (tensor<3072x64x4xf32>) {
    %16 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (d2, d0, d1)>, affine_map<(d0, d1, d2) -> (d0)>, affine_map<(d0, d1, d2) -> (d0, d1, d2)>], iterator_types = ["parallel", "parallel", "parallel"]} ins(%0, %1 : tensor<4x3072x64xf32>, tensor<3072xf32>) outs(%3 : tensor<3072x64x4xf32>) {
    ^bb0(%in: f32, %in_0: f32, %out: f32):
      %17 = arith.addf %in, %in_0 : f32
      linalg.yield %17 : f32
    } -> tensor<3072x64x4xf32>
    flow.return %16 : tensor<3072x64x4xf32>
  }
  %collapsed = tensor.collapse_shape %4 [[0], [1, 2]] : tensor<3072x64x4xf32> into tensor<3072x256xf32>
  %5 = tensor.empty() : tensor<384x256x8x1xf32>
  %6 = flow.dispatch.region -> (tensor<384x256x8x1xf32>) {
    %pack = tensor.pack %collapsed outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [8, 1] into %5 : tensor<3072x256xf32> -> tensor<384x256x8x1xf32>
    flow.return %pack : tensor<384x256x8x1xf32>
  }
  %7 = tensor.empty() : tensor<384x32x8x8xf32>
  %8 = linalg.fill ins(%cst : f32) outs(%7 : tensor<384x32x8x8xf32>) -> tensor<384x32x8x8xf32>
  %9 = flow.dispatch.region -> (tensor<384x32x8x8xf32>) {
    %16 = linalg.mmt4d ins(%6, %2 : tensor<384x256x8x1xf32>, tensor<32x256x8x1xf32>) outs(%8 : tensor<384x32x8x8xf32>) -> tensor<384x32x8x8xf32>
    flow.return %16 : tensor<384x32x8x8xf32>
  }
  %10 = tensor.empty() : tensor<3072x256xf32>
  %11 = flow.dispatch.region -> (tensor<3072x256xf32>) {
    %unpack = tensor.unpack %9 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [8, 8] into %10 : tensor<384x32x8x8xf32> -> tensor<3072x256xf32>
    flow.return %unpack : tensor<3072x256xf32>
  }
  %expanded = tensor.expand_shape %11 [[0, 1], [2]] : tensor<3072x256xf32> into tensor<12x256x256xf32>
  %12 = tensor.empty() : tensor<256x12xf32>
  %13 = linalg.fill ins(%cst : f32) outs(%12 : tensor<256x12xf32>) -> tensor<256x12xf32>
  %14 = flow.dispatch.region -> (tensor<256x12xf32>) {
    %16 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (d1, d0, d2)>, affine_map<(d0, d1, d2) -> (d0, d1)>], iterator_types = ["parallel", "parallel", "reduction"]} ins(%expanded : tensor<12x256x256xf32>) outs(%13 : tensor<256x12xf32>) {
    ^bb0(%in: f32, %out: f32):
      %17 = arith.addf %out, %in : f32
      linalg.yield %17 : f32
    } -> tensor<256x12xf32>
    flow.return %16 : tensor<256x12xf32>
  }
  %15 = hal.tensor.export %14 "output0" : tensor<256x12xf32> -> !hal.buffer_view
  util.return %15 : !hal.buffer_view
}

I think potentially we can swap tensor.collapse_shape and tensor.pack when the original dims are divisible by the inner tile sizes. For example:
%collapsed = tensor.collapse_shape %1 [[0], [1, 2]] : tensor<3072x64x4xf32> into tensor<3072x256xf32>
%pack = tensor.pack %collapsed outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [8, 1] into %2 : tensor<3072x256xf32> -> tensor<384x256x8x1xf32>

can be rewritten into:
%pack = tensor.pack %1 outer_dims_perm = [0, 1, 2] inner_dims_pos = [0, 2] inner_tiles = [8, 1] into %2 : tensor<3072x64x4xf32> -> tensor<384x64x4x8x1xf32>
%collapsed = tensor.collapse_shape %pack [[0], [1, 2], [3], [4]] : tensor<384x64x4x8x1xf32> into tensor<384x256x8x1xf32>

Then the tensor.pack can be fused with its producer. The same transform can also be done on tensor.unpack.
It's also worth noting that there are some simpler cases with only unit dims collapsed. For example:
#map = affine_map<(d0, d1, d2, d3) -> (d2, d3, d0, d1)>
#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d2, d3)>
module {
  func.func @collapse_unit_dim_and_pack(%arg0: tensor<1x256x1x64xf32>, %arg1: tensor<128x64x8x1xf32>) -> tensor<128x32x8x8xf32> {
    %cst = arith.constant 0.000000e+00 : f32
    %0 = tensor.empty() : tensor<1x64x1x256xf32>
    %1 = linalg.generic {indexing_maps = [#map, #map1], iterator_types = ["parallel", "parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x256x1x64xf32>) outs(%0 : tensor<1x64x1x256xf32>) {
    ^bb0(%in: f32, %out: f32):
      linalg.yield %in : f32
    } -> tensor<1x64x1x256xf32>
    %collapsed = tensor.collapse_shape %1 [[0, 1], [2, 3]] : tensor<1x64x1x256xf32> into tensor<64x256xf32>
    %2 = tensor.empty() : tensor<32x64x8x1xf32>
    %pack = tensor.pack %collapsed outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [8, 1] into %2 : tensor<64x256xf32> -> tensor<32x64x8x1xf32>
    %3 = tensor.empty() : tensor<128x32x8x8xf32>
    %4 = linalg.fill ins(%cst : f32) outs(%3 : tensor<128x32x8x8xf32>) -> tensor<128x32x8x8xf32>
    %5 = linalg.mmt4d ins(%arg1, %pack : tensor<128x64x8x1xf32>, tensor<32x64x8x1xf32>) outs(%4 : tensor<128x32x8x8xf32>) -> tensor<128x32x8x8xf32>
    return %5 : tensor<128x32x8x8xf32>
  }
}

@dcaballe fyi

Hi @hanhanW, I think you have been looking into this issue before. Do you know if some solutions have been proposed before for this case?

@Max191 I think you hit the same issue about propagating pack across reshape ops. Do you have some patterns in flight?

@Max191 I think you hit the same issue about propagating pack across reshape ops. Do you have some patterns in flight?

I haven't run into this issue before, so I don't have any patterns for it, but it seems like a useful pattern to have. Maybe we can add something here: https://github.com/llvm/llvm-project/blob/main/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp

Interesting, I didn't know we already have transforms to bubble pack/unpack ops. So looks like we want to do here is to have BubbleUpPackThroughCollapseShapeOp and PushDownUnPackThroughExpandShapeOp

Interesting, I didn't know we already have transforms to bubble pack/unpack ops. So looks like we want to do here is to have BubbleUpPackThroughCollapseShapeOp and PushDownUnPackThroughExpandShapeOp

Yes, and there could also be cases where we want to propagate pack through expand_shape or unpack through collapse_shape. Perhaps we can just add BubbleUpPackThroughReshape and PushDownUnPackThroughReshape.

I noticed that mlir::linalg::populateDataLayoutPropagationPatterns is only enabled in transform dialect and a specific convolution case. Do we have a plan to enable it globally by default?

Do we have a plan to enable it globally by default?

@qedawkins might know the answer. I think it is not ready yet, so they are put at preprocessing layer.

I just found that I filed https://github.com/openxla/iree/issues/13531 before, but I never get a chance to implement it. Let's use this issue for tracking. @pzread are you going to work on it? I assign the issue to you for now, but feel free to drop if you don't plan to work on it.

There has been a lot of discussion about layout propagation, but the direction we are moving towards is to avoid materializing pack/unpack in GlobalOptimization/Flow. The DataLayoutPropagation patterns are still useful for path finding in the short term, but they are only opt-in in the preprocessing layer right now, e.g. https://github.com/openxla/iree/blob/e612e91f79f5b95cc73fa3276aefad67860d36a6/compiler/src/iree/compiler/Preprocessing/Common/ConvertConvToChannelsLast.cpp#L683

To enable layout propagation in GlobalOptimization, will it be a good idea to add it under a flag? So we can see its impact on the performance first
Also how fast we are moving toward dropping pack/unpack materialization? If it is happening in a few weeks, then I'll probably wait for thatl; otherwise I'm happy to add these patterns and see how it performs.

@Max191 is adding propagation prototype under a flag in https://github.com/openxla/iree/pull/16585
I think we can add the patterns upstream, because they are very useful.

@Max191 is adding propagation prototype under a flag in #16585
I think we can add the patterns upstream, because they are very useful.

I should note that this propagation does not propagate through reshapes yet. Considering reshapes in the full program analysis makes things pretty difficult, so for now I am not adding any propagation through reshape. I think it could make sense to instead add the upstream patterns as a preprocessing step before the analysis. I actually have another PR coming later that will add a preprocessing step to the analysis pass, so maybe we can try adding it there once everything in flight lands.

I think adding the patterns to bubble up/down pack/unpack ops through reshapes could be a good starting point that is kind of horizontal to a more generic data layout propagation. The main goal here is not enable fusion of pack/unpack ops with their immediate producer/consumers. We have similar bubble up/down patterns for other ops (vector.insert/extract and vector.bitcast) with similar transformation enablement purposes. Note that this issue has been a long lasting one: https://github.com/openxla/iree/issues/15242 so having a solution for it in the short term would be awesome now that we understand what is going on!

Tested with https://github.com/openxla/iree/pull/16731 and it shows 16% improvement on Falcon7b (x86).
The major regression on DeepLabV3 and the issue is tracking on https://github.com/openxla/iree/issues/16772.
@hanhanW I saw you start moving materialization away from the global optimization. My current implementation is to swap pack/unpack with reshape at global optimization level. So I'm curious about the timeline on your side and do you think it is worth to land #16731 ?

Tested with #16731 and it shows 16% improvement on Falcon7b (x86).

wow, really nice improvements!

@hanhanW I saw you start moving materialization away from the global optimization. My current implementation is to swap pack/unpack with reshape at global optimization level. So I'm curious about the timeline on your side and do you think it is worth to land #16731 ?

I think it is still worth to push it forward because it shows the optimization opportunities on the early materialization path. We should be able to apply similar optimization in the next evolution of data-tiling. However, I honestly don't have a clear picture about how we propagate encodings through reshapes. My intuition is that we can get such information at stream level or right before converting them to stream. The analysis is much harder because they are formed into dispatches.. The encoding has more information today comparing to the very beginning stage. It has indexing_maps and roles; we might be able to make it work with encodings with some constraints.
Moving the materialization away from global optimization does not mean that the work is a waste. To me, it is more like "we need to consider such chance in the roadmap". If we can't do it in the data-tiling infra, we can move the passes to preprocessing stage. Others still can play with it and demonstrate performance. So yes, the pass is still valuable to me.

Merged https://github.com/openxla/iree/pull/16731


How to configure custom EventLog Name and Source using appsettings.json?

It seems that by default, the created logger object writes to the Application log using the Application event source. I've created a new Event Log that I want all my event to be written to instead so events are easier to find and I can have a longer audit trail history by not having the Application log rotate when it gets full from events from other sources.
How can I do this using appsettings.json as opposed to using the overloaded method (ILoggingBuilder.AddEventLog(EventLogSettings)?
In Program.cs I have:
logging.AddEventLog();
logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));

It'd be nice to be able to do this in appsettings.json (perhaps you can and I'm using the wrong keys here but I can't find a reference on how these map to the Logger objects):
  "Logging": {
    "Log": "myLog",
    "Source": "Account Management Portal",
    "IncludeScopes": false,
    "LogLevel": {
      "Default": "Warning",
      "myLog": "Critical"
    }
  }

I can do this by using the overload for .AddEventLog(EventLogSettings) like so:
logging.AddEventLog(new EventLogSettings() { LogName = "myLog", SourceName = "mySource" });

but the disadvantage of course is configuration flexibility across environments.

We are not currently supporting this.

Provided you have the following appsettings.json:
{
    "Redis": {
      "ConnectionString": "localhost"
    }
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information"
    },
    "EventLog": {
      "LogName": "MyLogName",
      "SourceName": "My App"
    }
  }
}

You can configure EventLog logger like this:
...
.ConfigureLogging((hostingContext, logging) =>
{
	logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
	logging.AddConsole();
	logging.AddEventLog(hostingContext.Configuration.GetSection("Logging:EventLog").Get<EventLogSettings>());
})
...

Hope that helps.


Updated handling of Appfile per #4

Updated handling of Appfile per #4

@yvanvds let me know if this looks ok

Yes, that is what I had to change.

Great!
At some point, some of these steps can be automated, say using a wizard-style approach, maybe even creating a local website to run the wizard... anyway, food for thought...


Custom model for Ensemble

Hi, I was just wondering how to go about using my own custom trained GBM model instead of the default "face-recognition-ensemble-model.txt" file. So I have trained a GBM ensemble model on my own custom dataset and was wondering how to use that in DeepFace.find or Deepface.analyze or any other functions. Thanks in advance.

Ensemble model is totally hard-coded. You cannot pass a custom model to it unfortunately. But I can add this feature in the next release.
Could you explain more about your custom model. What are features e.g.?

Hi,
Thanks for the reply. So i followed this link of yours and instead of using the provided dataset in the "tests/dataset/" directory, I used my own dataset of around 10-15k pairs of positive and negative faces and found representations for the same. After that I trained a LightGBM model and saved the gbm model as "face-recognition-ensemble-model-custom.txt". So now instead of using the default "face-recognition-ensemble-model.txt" file, how can I use this custom txt file of mine

Here is the link I followed : https://sefiks.com/2020/06/03/mastering-face-recognition-with-ensemble-learning/

if your features are same with this tutorial, then you can modify the content of face-recognition-ensemble-model-custom.txt
This file is stored in HOME/.deepface/weights/. For example, exact path is "C:\Users\Sefik.deepface\weights" in my environment. If the name of this file stays same but you can update the content of this file.

Yes the features remain the same, only the dataset changes if i am not wrong. Great, I will try this and see if it works and let you know.

you can delete that file if something is broken, it recovers the base file in this case.

Okay Thank you !!!


Question: How to add "start" parameters?

Hello,
For integration with GitEnterprise and TeamCity I need to specify following SonarQube parameters:
/d:sonar.analysis.mode=
/d:sonar.useWsCache=
/d:sonar.issuesReport.html.enable=
/d:sonar.issuesReport.console.enable=
/d:sonar.language=
/d:sonar.github.pullRequest=
/d:sonar.github.repository=
/d:sonar.github.login=
/d:sonar.github.oauth=
/d:sonar.github.endpoint=
/d:sonar.github.deleteOldComments=

See also https://docs.sonarqube.org/display/PLUG/GitHub+Plugin
Is it correct that currently one need to add new properties into SonarBeginSettings.cs, which will reflect the additional Sonar parameters?
I wonder if it would make sense to allow Cake users to specify arbitrary Sonar parameters using a weakly typed interface (e.g. via a Dictionary<string, string> on SonarBeginSettings )? Would you accept such change?
Oleg

You can specify custom attributes. SonarBeginSettings inherits from
ToolSettings which has a property ArgumentCustomization for that purpose.
On Fri, Apr 13, 2018, 5:06 AM Oleg Grytsynevych<EMAIL_ADDRESS>wrote:

Hello,
For integration with GitEnterprise and TeamCity I need to specify
following SonarQube parameters:
/d:sonar.analysis.mode=
/d:sonar.useWsCache=
/d:sonar.issuesReport.html.enable=
/d:sonar.issuesReport.console.enable=
/d:sonar.language=
/d:sonar.github.pullRequest=
/d:sonar.github.repository=
/d:sonar.github.login=
/d:sonar.github.oauth=
/d:sonar.github.endpoint=
/d:sonar.github.deleteOldComments=
See also https://docs.sonarqube.org/display/PLUG/GitHub+Plugin
Is it correct that currently one need to add new properties into
SonarBeginSettings.cs, which will reflect the additional Sonar
parameters? Is it correct?
I wonder if it would make sense to allow Cake users to specify arbitrary
Sonar parameters using a weakly typed interface (e.g. via a Dictionary<string,
string> on SonarBeginSettings? Would you accept such change?
Oleg
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/AgileArchitect/Cake.Sonar/issues/59, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AA6OHoW8BLHMlZ8M2_ZwpNV38omKuOPXks5toJTQgaJpZM4TTVmo
.

Adding a Dictionary<string, string> makes sense to me, makes it easier when making running the task from CakeBuild

I have posted a PR with "explicit" implementation of the parameters.
I also think the Dictionary<string,string> would simplify life for many people, who use "less popular" SonarQube parameters and plugins. It can be implemented in the next PR.

ArgumentCustomization in ToolSettings does the same thing. Why duplicate that.
https://cakebuild.net/api/Cake.Core.Tooling/ToolSettings/50AAB3A8


Where does taffy db stores data ?

is it in local storage or indexeddb of browser or something else ?

TaffyDB store data in memory - as a Javascript array. And I like any other Javascript array it is purged on page refresh (browser) or restart (node.js). You can use the .store() method to store your TaffyDB in the browser's localStorage though. But most folks push and pull data to the server and use TaffyDB to interact with and manage the data client side.


Update quoter.plugin.js

Disable Contextmenu

Why?

Why?

So you can use it with 305!

Why?

So you can use it with 305!

http://prntscr.com/mypjq9

The effort would be better spent fixing the issue rather than removing the feature

The effort would be better spent fixing the issue rather than removing the feature

Well then you could do it, I do not have the idea of it. Therefore, I have disabled the faulty option. to continue to use the plugin, and shared it.

I will fix it when I have free time, yes.

allright thx for that


move some compile-fail tests to a more appropriate location

These are testing validity invariants, after all.

@bors r+

:hourglass: Testing commit 6e7a8c017abaea2d3dacf21b50d27c33f7e8bfde with merge 8f01d862d8713658dc155a4ce6d010a092fa080d...


Add :strict opt to parse-opts

...that throws an error when an arg is provided that isn't defined in the spec. Defaults to false to maintain current behavior.
This seem reasonable to include in here?

@cap10morgan I would like to challenge this behavior: in Clojure we also don't throw when we provide extra arguments. Why would we do this on the command line?
Strict can mean several things. I almost introduced another strict thing here:
https://github.com/babashka/cli/pull/14/commits/c30a3828292432f50cc0929aae33f832987d4f29

@borkdude Yeah, you mean with a map arg? I was thinking this was kind of analogous to a wrong arity exception (translating from CLI semantics to Clojure fn call semantics). But yeah, the named args are more like a map arg.
I certainly wouldn't want this to be the default behavior, but since we're translating from CLI semantics to Clojure fn call semantics, this struck me as kind of analogous to a wrong arity situation like the above example. And it's just a nicer UX for some CLI apps to error out if you e.g. misspelled an optional arg. That could implemented outside of this library, but it would duplicate a lot of what's happening in here.
I'd be happy to rename it. Something like :strict-opts or :closed-spec perhaps?
I could also imagine implementing it as a separate validate-args fn (or similar) that takes the args vector you received at runtime and the spec and either throws or returns error data.
But either way, if you'd rather not support this directly in the lib that's OK too.

Perhaps an extra argument to parse-args / parse-opts can work: either :closed true or :unknown-opt (fn [..])

That sounds reasonable. I'll play around with it.

Perhaps an extra argument to parse-args / parse-opts can work: either :closed true or :unknown-opt (fn [..])

The :closed true arg to parse-* would be exactly what I have now but renaming :strict to :closed. Just wanted to make sure we were on the same page there. If that sounds good, I think I'd like to start there.

Do you think :closed-spec might be a better name? Seems a little clearer to me.

So, what I was getting at with spec2 is this:
https://github.com/clojure/spec-alpha2/wiki/Differences-from-spec.alpha#closed-spec-checking
The closed argument specifies what should be considered closed in the context of the valid? check.
We could do: :closed true or :closed #{:foo :bar} if you want to pass in a spec that has a large amount of options that can be used in multiple contexts and you just expect these two keys.

I think I prefer the name :closed for this reason btw, future extensibility of the meaning of closed.

We could do: :closed true or :closed #{:foo :bar} if you want to pass in a spec that has a large amount of options that can be used in multiple contexts and you just expect these two keys.

You wouldn't necessarily even need a spec if you provide :closed #{:foo :bar} which could be nice in some use cases.

Indeed!

Going to add some tests next

To be somewhat consistent, I think dispatch for subcommands could also support this, but this could happen in a separate PR.


docs: 1.0.3 changelog

[skip ci]

Could we maybe reword the updates to be more descriptive? Something like the form "Verbed noun, affects you because x. #1234"
I recognize we haven't been doing this so take this advice with a grain of salt if this is super time-sensitive.

What we've been doing is using the changelog as just a simple list of prs that are included in the release. The (forthcoming) release notes will include higher-level, more user-friendly descriptions of what's in the release.

:shipit:


How to prepare the .pkl file for training?

I cleaned up the setup code and wrote the code for preparing pkl file from csv file.
https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/blob/master/src/prepare_dataset.py

Thank you very much

Hi guys! @HLH13297997663  @lyakaap
When I run python3 v2clean.py tuning -d 0,1,2,3 --n-gpu 2, I got error No such file or directory: input/gld_v2train.pkl ```
in make_train_loaders
train = pd.read_pickle(data_root + 'train.pkl')
Could you help me? How to solve it? Thanks a lot!

Hi guys! @HLH13297997663 @lyakaap
When I run python3 v2clean.py tuning -d 0,1,2,3 --n-gpu 2, I got error No such file or directory: input/gld_v2train.pkl ```
in make_train_loaders
train = pd.read_pickle(data_root + 'train.pkl')
Could you help me? How to solve it? Thanks a lot!

Try creat pkl file by https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/blob/master/src/prepare_dataset.py and put it into input.


can you add another template based on QAbstractListModel

hello!
can you add another template based on QAbstractListModel.

Hi! Which one?

This QAbstractListModel

But I think table model with one column is equal list model. QAbstractListModel doesn't add any functionality.


[iam] Add policy autoscaling policy

To be able to stop/start instances on autoscaling group we need to add
these policies:

autoscaling:DescribeAutoScalingGroups
autoscaling:SuspendProcesses
autoscaling:ResumeProcesses

Hello @diodonfrost
This is a PR to fix some missing permissions:
An error occurred (AccessDenied) when calling the SuspendProcesses operation: User: XXXXX is not authorized to perform: autoscaling:SuspendProcesses
...

An error occurred (AccessDenied) when calling the DescribeAutoScalingGroups operation: User: XXXXXXX is not authorized to perform: autoscaling:DescribeAutoScalingGroups
...

An error occurred (AccessDenied) when calling the ResumeProcesses operation: User: XXXX is not authorized to perform: autoscaling:ResumeProcesses [...]
...

Could you take a look please ?
Thank you !

Hello
That strange, I didn't have this issue and these policies are already exist in the code:
https://github.com/diodonfrost/terraform-aws-lambda-scheduler-stop-start/blob/master/main.tf#L51-L53

Hello !

That strange, I didn't have this issue and these policies are already exist in the code [...]

Thank you for your response, my bad, everything is OK.
Sorry for the for the setback !


fix: pass ctx for prefixed commands

What type of pull request is this?

[x] Non-breaking code change
[ ] Breaking code change
[ ] Documentation change/addition
[ ] Tests change

Description
This PR just makes sure Context is passed for prefixed commands that have one or more parameters.
Changes

Pass in ctx for the call_with_binding call at the end of call_callback for PrefixedCommand for parameter handling.

Checklist

[x] I've formatted my code with Black
[x] I've ensured my code works on Python 3.10.x
[x] I've tested my code

Wait, was this just... missing the whole time?

Wait, was this just... missing the whole time?

...I spent 20 minutes trying to find the bug because I didn't realize it was that simple.
(It was in 1.4.X and before though.)


load_test type error

Currently, running dune build in this branch yields
61 |            ~sender:(Core.Address.of_key_hash sender.key_hash)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: The function applied to this argument has type
         source:Crypto.Key_hash.t -> Core__User_operation.t
This argument cannot be applied with label ~sender

This change fixes said error.

Thanks for the fix! I applied the patch and made you co-author against the latest commit in #539


IV-4904 Initial commits of disable monitoring script

This downgrades an instance from Level 3 support (full capabilities) to Level 2 support by disabling monitoring. It can be run against any RightLink 5, 6 or 10 Server and must be run every boot.

LGTM


Zero Touch Provisioning Service Customer API

Description of changes:
Adding an example of a customer account deployment for a serverless API that verifies and registers certificates and creates IoT Things in the customer's account. This can be used by ZTP service providers to bootstrap devices and credentials into customers accounts.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Because I'm not so savvy with Github and merging branches from forks this ended up as a duplicate of https://github.com/aws-samples/iot-reference-architectures/pull/3043


Error in processing of business statements

Hi Jonathan, thanks for the tool, it looked like it did work but then when i reconciled what it had output it had a glitch where it fails if the 3rd column states "Internet Banking" which it does when you make account transfers to other accounts. Other than that it works great, many thanks, Tim.

Thanks for the report Tim. I don't have such a statement, so it's hard to fix without one - if this is causing you some problems and you'd be prepared to, you could email me an example statement at<EMAIL_ADDRESS>


How to create a new record?

this.get('store').createRecord() - does not work

hi @falkon !  That's a question that's answered by the Ember Data documentation, and not related to Ember Infinity.  Please read the docs, and if you're still confused, open up an issue on Stack Overflow


CAMEL-11628: camel-mqtt - Fix in MQTTEndpoint for connection loop bug.

https://issues.apache.org/jira/browse/CAMEL-11628

Thanks for the PR it has been merged. Do you mind closing this?


Tooltip merging with low numbers

Not sure if this is a bug or if i am just misinterpreting how it works, but under roughly the same conditions as #7 the merging doesn't seem to work, here is it set to 30 and will merge only when its just 3 apart: https://jsfiddle.net/Lsm5rxyf/1/

Fixed in 2.0.0.


add boxr vignette

Adds technical vignette about programmatic access to Box at UA

Oooh, nice! So the methods described here are not the methods I use!
I am using a service app with OAuth to work with box programmatically interactively from my computer. You set the token as an environment variable. This doesn't work on servers.
Next week I can draft an additional section describing what I'm doing, if that sounds helpful?

I am using a service app with OAuth to work with box programmatically interactively from my computer. You set the token as an environment variable. This doesn't work on servers.

What servers do you mean?  This is the setup I'm using for Posit Connect, and I'm not sure why it wouldn't work for other servers.  It's maybe a little "weird" to copy paste the contents of the token json file into an env variable, but it should work

But, yes, please do contribute additional methods if you've found another way!

I am using a service app with OAuth to work with box programmatically interactively from my computer. You set the token as an environment variable. This doesn't work on servers.

What servers do you mean? This is the setup I'm using for Posit Connect, and I'm not sure why it wouldn't work for other servers. It's maybe a little "weird" to copy paste the contents of the token json file into an env variable, but it should work

Oh, yeah! I'm sure the way you're describing works. Upon digging, I'm using an interactive app (https://r-box.github.io/boxr/articles/boxr-app-interactive.html), basically to the letter of that tutorial. (Specifically, the method I'm using is to have the BOX_CLIENT_ID and BOX_CLIENT_SECRET set as environment variables. I was not able to get to work on Posit Connect, which is not surprising given the tutorial 😆. I think you described a workaround for that which culminates in the .json file (instead of supplying both the ID and SECRET).
So, for this PR I suggest adding a sentence about interactive apps linking to the documentation, and otherwise keeping this page as it is. I do think it's good to direct future-us to the interactive apps docs, because I don't think I knew about this documentation page - I figured it out independently when I was having trouble with the service app. It's possible the next person won't miss it, but can't hurt to leave the breadcrumbs!

So, for this PR I suggest adding a sentence about interactive apps linking to the documentation, and otherwise keeping this page as it is. I do think it's good to direct future-us to the interactive apps docs, because I don't think I knew about this documentation page - I figured it out independently when I was having trouble with the service app. It's possible the next person won't miss it, but can't hurt to leave the breadcrumbs!

Ok, I can definitely add that.  The interactive apps are only meant for interactive use (i.e. not on a server like Posit Connect), so they're not expected to work, but they are easier to set up I think and useful if you want to do stuff programmatically just on your local computer.


Windows: Fix interface selection by name

When @soffokl cleaned up the reg key stuff in #59, it looks like a mistake was introduced.
There are two sets of registry keys, one where the TAP driver lives (SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}) and one where the interface network config lives (SYSTEM\CurrentControlSet\Control\Network\{4D36E972-E325-11CE-BFC1-08002BE10318}). The second key somehow was removed and replaced with the first, which broke the interface selection.
This PR fixes that.

No problem :-) Please merge my PRs when ready following review—I think there's four now.

@neilalexander Oh I thought you'd be able to merge after my approval. Will merge now!


Support for MotherDuck

Albeit the sql-exapmple.ipynb is working for duckdb, it is failing for motherduck.
To connect to Motherduck:
# ---------------------------------
# MotherDuck (Cloud)
# ---------------------------------
!pip install "ibis-framework[duckdb]"
token = "<add your token>"
connection_uri = f"duckdb://md:{token}@test_db"

To get the token you must first create an account
https://motherduck.com/docs/authenticating-to-motherduck/
The add data to datastore cell fails with:
ProgrammingError: (duckdb.duckdb.InvalidInputException) Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: Invalid Error: RPC 'SETUP_PLAN_FRAGMENTS' failed: Deadline Exceeded (DEADLINE_EXCEEDED, request id: '05515ef3-a7fe-4c9b-afc9-a57289617d16')
[SQL: INSERT INTO images (id, image, _fold) VALUES (?, ?, ?),
....
(Background on this error at: https://sqlalche.me/e/14/f405)

Here is the traceback:
---------------------------------------------------------------------------
InvalidInputException                     Traceback (most recent call last)
File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1910, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1909     if not evt_handled:
-> 1910         self.dialect.do_execute(
   1911             cursor, statement, parameters, context
   1912         )
   1914 if self._has_events or self.engine._has_events:

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/engine/default.py:736, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    735 def do_execute(self, cursor, statement, parameters, context=None):
--> 736     cursor.execute(statement, parameters)

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/duckdb_engine/__init__.py:160, in ConnectionWrapper.execute(self, statement, parameters, context)
    159     else:
--> 160         self.__c.execute(statement, parameters)
    161 except RuntimeError as e:

InvalidInputException: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: Invalid Error: RPC 'SETUP_PLAN_FRAGMENTS' failed: Deadline Exceeded (DEADLINE_EXCEEDED, request id: '05515ef3-a7fe-4c9b-afc9-a57289617d16')

The above exception was the direct cause of the following exception:

ProgrammingError                          Traceback (most recent call last)
Cell In[48], line 2
      1 # Insert data from the 'images_df' DataFrame into the 'images' table
----> 2 _ = db.execute(images.insert(images_df))
      4 # Insert data from the 'captions_df' DataFrame into the 'captions' table
      5 _ = db.execute(captions.insert(captions_df))

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/superduperdb/base/datalayer.py:386, in Datalayer.execute(self, query, *args, **kwargs)
    384     return self.delete(query, *args, **kwargs)
    385 if isinstance(query, Insert):
--> 386     return self.insert(query, *args, **kwargs)
    387 if isinstance(query, Select):
    388     return self.select(query, *args, **kwargs)

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/superduperdb/base/datalayer.py:427, in Datalayer.insert(self, insert, refresh, encoders)
    425     if random.random() < s.CFG.fold_probability:
    426         r['_fold'] = 'valid'  # type: ignore[assignment]
--> 427 inserted_ids = insert.execute(self)
    429 if refresh and self.cdc.running:
    430     raise Exception('cdc cannot be activated and refresh=True')

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/superduperdb/backends/ibis/query.py:846, in IbisInsert.execute(self, db)
    843 encoded_documents = self._encode_documents(table=table)
    844 ids = [r[table.primary_id] for r in encoded_documents]
--> 846 db.databackend.insert(
    847     self.table_or_collection.identifier, raw_documents=encoded_documents
    848 )
    849 return ids

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/superduperdb/backends/ibis/data_backend.py:36, in IbisDataBackend.insert(self, table_name, raw_documents)
     34 def insert(self, table_name, raw_documents):
     35     if not self.in_memory:
---> 36         self.conn.insert(table_name, raw_documents)
     37     else:
     38         self.conn.create_table(table_name, pandas.DataFrame(raw_documents))

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/ibis/backends/base/sql/alchemy/__init__.py:777, in BaseAlchemyBackend.insert(self, table_name, obj, database, overwrite)
    775         if overwrite:
    776             bind.execute(to_table.delete())
--> 777         bind.execute(to_table.insert().values(obj))
    779 else:
    780     raise ValueError(
    781         "No operation is being performed. Either the obj parameter "
    782         "is not a pandas DataFrame or is not a ibis Table."
    783         f"The given obj is of type {type(obj).__name__} ."
    784     )

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1385, in Connection.execute(self, statement, *multiparams, **params)
   1381     util.raise_(
   1382         exc.ObjectNotExecutableError(statement), replace_context=err
   1383     )
   1384 else:
-> 1385     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/sql/elements.py:334, in ClauseElement._execute_on_connection(self, connection, multiparams, params, execution_options, _force)
    330 def _execute_on_connection(
    331     self, connection, multiparams, params, execution_options, _force=False
    332 ):
    333     if _force or self.supports_execution:
--> 334         return connection._execute_clauseelement(
    335             self, multiparams, params, execution_options
    336         )
    337     else:
    338         raise exc.ObjectNotExecutableError(self)

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1577, in Connection._execute_clauseelement(self, elem, multiparams, params, execution_options)
   1565 compiled_cache = execution_options.get(
   1566     "compiled_cache", self.engine._compiled_cache
   1567 )
   1569 compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
   1570     dialect=dialect,
   1571     compiled_cache=compiled_cache,
   (...)
   1575     linting=self.dialect.compiler_linting | compiler.WARN_LINTING,
   1576 )
-> 1577 ret = self._execute_context(
   1578     dialect,
   1579     dialect.execution_ctx_cls._init_compiled,
   1580     compiled_sql,
   1581     distilled_params,
   1582     execution_options,
   1583     compiled_sql,
   1584     distilled_params,
   1585     elem,
   1586     extracted_params,
   1587     cache_hit=cache_hit,
   1588 )
   1589 if has_events:
   1590     self.dispatch.after_execute(
   1591         self,
   1592         elem,
   (...)
   1596         ret,
   1597     )

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1953, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1950             branched.close()
   1952 except BaseException as e:
-> 1953     self._handle_dbapi_exception(
   1954         e, statement, parameters, cursor, context
   1955     )
   1957 return result

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/engine/base.py:2134, in Connection._handle_dbapi_exception(self, e, statement, parameters, cursor, context)
   2132     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
   2133 elif should_wrap:
-> 2134     util.raise_(
   2135         sqlalchemy_exception, with_traceback=exc_info[2], from_=e
   2136     )
   2137 else:
   2138     util.raise_(exc_info[1], with_traceback=exc_info[2])

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/util/compat.py:211, in raise_(***failed resolving arguments***)
    208     exception.__cause__ = replace_context
    210 try:
--> 211     raise exception
    212 finally:
    213     # credit to
    214     # https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/
    215     # as the __traceback__ object creates a cycle
    216     del exception, replace_context, from_, with_traceback

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1910, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1908                 break
   1909     if not evt_handled:
-> 1910         self.dialect.do_execute(
   1911             cursor, statement, parameters, context
   1912         )
   1914 if self._has_events or self.engine._has_events:
   1915     self.dispatch.after_cursor_execute(
   1916         self,
   1917         cursor,
   (...)
   1921         context.executemany,
   1922     )

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/sqlalchemy/engine/default.py:736, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    735 def do_execute(self, cursor, statement, parameters, context=None):
--> 736     cursor.execute(statement, parameters)

File ~/.pyenv/versions/superduper/lib/python3.11/site-packages/duckdb_engine/__init__.py:160, in ConnectionWrapper.execute(self, statement, parameters, context)
    158         self.__c.execute(statement)
    159     else:
--> 160         self.__c.execute(statement, parameters)
    161 except RuntimeError as e:
    162     if e.args[0].startswith("Not implemented Error"):

Following error:
ProgrammingError: (duckdb.duckdb.InvalidInputException) Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
Original error: "FATAL Error: You've encountered an internal MotherDuck error. You can help us diagnose and fix the issue by contacting support via slack or<EMAIL_ADDRESS>and referencing error ID: 20240103a0d."
(Background on this error at: https://sqlalche.me/e/14/f405)

Here the investigations of the MotherDuck team:
We identified the issue and a possible workaround.
Ibis inlines the blob values in the INSERT INTO statement which makes the logical execution plan that has to be shipped between the DuckDB client and motherduck quite large and finally leads to the timeout (4MB is the current maximum size we support, the plan in the example notebook was around 48MB). We will add support for larger plans at some point, but I want to suggest a workaroud.
Instead of inlining the BLOB values in the INSERT INTO statement, you can use DuckDB's zero-copy mechnism to insert values from images_df with a statement like this:
INSERT INTO images (id, image, _fold) SELECT id, image, 'test' FROM images_df
(instead of the INSERT INTO images VALUES(..) that is currently generated within _ = db.execute(images.insert(images_df)) )
Since the insert is happening behind the Ibis abstraction, it might require some special-casing for motherduck, but overall I'd advise to perform inserts in such a way even if we are going to add support for larger plans in the future.


Adding a tag with a library version

Hey!
Your project uses Go Modules, but I didn't find any version of the project in the tags. Can you create a tag to make full use of your library?

Hi @mlaymer, apologies for the delay in getting back to you here!
We just enabled Releases for this repo, starting with https://github.com/domainr/whois/releases/tag/v0.1.0


Add classifiers to add useful version information to JSON API

Coverage remained the same at 96.742% when pulling 3de669a789597705d7fd4693f4a7eff31690eab1 on classifiers into f88f85a07e828c2b7ea2d4c88394ef58fc0492ed on master.


The state of React component testing

TL;DR
Discussion around React component testing that will influence the default way sapling tests component projects. Share what worked & what didn't work when testing React components even if it's outside the realm of react-sapling.
Alright, so
We currently use Karma as our test runner. This was done to load in any arbitrary webpack configuration for a project and have it just work TM be it TypeScript, ClojureScript, or plain ol' ES6. If the component can be built with Webpack, then Karma knows how to handle it.
One goal of sapling is to make testing a little less painful. If another framework or library helps with that goal, then it's something we must consider.
This discussion stems from the component react-flopdown, which is utilizing a different test runner—Jest.
I see some options (not an exhaustive list):

Replace Karma with Jest

Could be an option, but note that some thought will be necessary to get it working with non-ES6 projects (TypeScript, ClojureScript, etc.) It may be super simple, but I haven't looked into Jest + TypeScript setup for several months.

Support Jest and Karma

Freedom for component developers to use what they want at the cost of fragmentation. We already allow components to be written in multiple languages. Do we also want to support multiple test runners?

Learn from Jest & adapt Karma to support things that it can

What things did Jest bring to the table? Where did Karma fall short?

Thoughts? Struggles that you've run into (good and bad) with component testing in the past? Any and all feedback on this is very welcome even if it's short, i.e. "I liked framework X because Y".

So I wrote a few tests with Karma before pluggin in Jest.  While the set up to get jest working was a bit of a struggle, once up and running it's a robust testing suite with plenty of documentation.  It's also been used already by other team members of WAT so there are examples in the wild for us to look at already.  Actually that's mostly how I got this set up, by looking at Kevin's configuration.  I see two options for this:

Have Jest rolled into sapling

This would require no setup on behalf of the component dev, which might be nice, but then it would restrict customization or tweaking a bit.  Also, it's easy enough should someone NOT wish to use whatever testing suite that comes with sapling to simply rip it out of the package.json and run their own script.

Have a guide on sapling of how to set up Jest testing in each individual component repo at the developers discretion.  Again, now that we have flopdown set up the configuration could be copied over fairly easily.  The downside is it still takes some work, even if it's a little, and that adds friction to the idea of being able to quickly build react components.

Final thoughts, at least for now it seems like most of these are leaning toward es6 or typescript.  The unknown is how Jest plays with typescript, but it seems like supporting a broad range of language types is good in theory but maybe not necessary in practice.  Perhaps this is shortsighted but can't we cross that bridge when we get there?

Thanks for the feedback Jordan.

Have Jest rolled into sapling

This would require no setup on behalf of the component dev, which might be nice, but then it would restrict customization or tweaking a bit. Also, it's easy enough should someone NOT wish to use whatever testing suite that comes with sapling to simply rip it out of the package.json and run their own script.

Yep. Totally valid & acceptable for someone to use their own test configuration. I think a no-config default that meets the majority of use cases is valuable be it Karma, Jest, or XYZ.

Leave it up to the component dev

We could just leave a guide on sapling's README on how to set up Jest testing in each individual component repo at the developers discretion. Again, now that we have flopdown set up the configuration could be copied over fairly easily. The downside is it still takes some work, even if it's a little, and that adds friction to the idea of being able to quickly build react components.

I think a default test suite that covers the majority of the component test needs is a goal that we want so long as it's overridable.

Final thoughts, at least for now it seems like most of these are leaning toward es6 or typescript. The unknown is how Jest plays with typescript, but it seems like supporting a broad range of language types is good in theory but maybe not necessary in practice. Perhaps this is shortsighted but can't we cross that bridge when we get there?

With the current state of languages that we would actually use for development this is totally valid. Technically we can force Jest to call out to work with Webpack as necessary and gain all the benefits of it, but yeah. We can totally cross that bridge later.

Jest support is coming to sapling to better support existing components projects that wrote their tests using mocks / etc. and wish to switch to react-sapling. See issue 72 for more information.
Closing this for now.

😂


Added pagination option to include a Previous and Next button

Will default Previous and Next Buttons to False.
In using the list.js component, we needed an option to include "Prev/Next" buttons in our pagination. Attempting to dynamically add them in our implementation were overwritten by the pagination refresh function where it calls paginationList.clear()

resolve conflict


A faster way to verify changes

After making a change and before opening a pull request for it, I'd like to build the changed file and verify it looks right. The contributing guide provides a way to build the docs. There are a lot of docs and this takes some time.
Is there a faster way to do this? Maybe a way to build just an individual file? Typically a change is just a word or 2 in 1 file. Waiting for a build of the entire set of docs (looks like over 50K files) maybe adds time to the process of contributing unnecessarily.

Are you after a capability that the Preview view on Github doesn't give you, but that the DocFx build does? Generally, the GitHub preview mode is good enough for simple edits. It can get confused by markdown sometimes, and its rendering isn't perfect, but it's generally good enough unless you need to test cross-repo links or includes. I'd only bother with the DocFx build if I was creating a lot of files, and wanted to verify that navigation was working.


proposal: Go 2: support lambda expression

Author background

Would you consider yourself a novice, intermediate, or experienced Go programmer?
experienced
What other languages do you have experience with?
Java, JavaScript, Python

Related proposals

Has this idea, or one like it, been proposed before?
Seached, but didn't find similar one
Does this affect error handling?
No
Is this about generics?
Yes. This probably will make go generics more powerful like it is in Java. It will work better with the related issue: https://github.com/golang/go/issues/49085, and probably a better way to sovle the issue https://github.com/golang/go/issues/59101

Proposal

What is the proposed change?
Support lamdba function. I'll display examples to demonstrate why it is so necessary.
For exmaple, we created a struct Optinal and it's related methods (pretty much like the Optional class in Java which is extreamly useful in all kinds of business developments)

package optional

type Optional[T any] struct {
	val *T
	ok  bool
}

func OfNullable[T any](v *T, ok bool) Optional[T] {
	return Optional[T]{v, ok}
}

func OK[T any](v *T) Optional[T] {
	return Optional[T]{v, true}
}

func Nil[T any]() Optional[T] {
	return Optional[T]{}
}

func (o Optional[T]) Get() *T {
	return o.val
}

func (o Optional[T]) GetOrElse(v *T) *T {
	if o.ok {
		return o.val
	} else {
		return v
	}
}

func Map[F, T any](o Optional[F], f func(*F) *T) Optional[T] {
	if !o.ok {
		return Nil[T]()
	}
	return OK(f(o.val))
}

The Map function can't be integrated into Optional is the same reason mentioned in issue https://github.com/golang/go/issues/49085.
The main purpose of Optional here is just like it is in Java: to solve tedious nil judgement. For example, we have the following structs
type QueryStudentResponse struct {
	Student *Student
}

type Student struct {
	Name     string
	BackPack *BackPack
}

type BackPack struct {
	Books *Books
	Phone *Phone
}

type Books struct {
	BookNames []string
}

type Phone struct {
	Number string
}

If I want to get the phone number from the response, in the none generic and non Optional version, I have to do this:
func GetStudentPhoneNumber(resp *QueryStudentResponse) string {
	if resp != nil && resp.Student != nil && resp.Student.BackPack != nil && resp.Student.BackPack.Phone != nil {
		return resp.Student.BackPack.Phone.Number
	}
	return ""
}

And this scenario is so common and it is not very possible to write a common function for each struct.
If we use the Optional above, it looks like this:
func GetStudentPhoneNumber(resp *QueryStudentResponse) string {
	o := OfNullable(resp)
	o1 := Map(o, func(f *QueryStudentResponse) *Student {
		return f.Student
	})
	o2 := Map(o1, func(f *Student) *BackPack {
		return f.BackPack
	})
	o3:= Map(o2, func(f *BackPack) *Phone {
		return f.Phone
	})
	pno := Map(o3, func(f *Phone) *string {
		return &f.Number
	})
	return pno.GetOrElse("")
}

The types actually can be refrenced just like we do it in value declaration with :=. So we can remove the arguments and return values' parameters and of course the "return" statement, just like in Java. if we support lambda, it looks like this:
func GetStudentPhoneNumber(resp *QueryStudentResponse) string {
	o := OfNullable(resp)
	o1 := Map(o, f -> f.Student)
	o2 := Map(o1, f -> f.BackPack)
	o3 := Map(o2, f -> f.Phone)
	pno := Map(o3, f -> &f.Number)
	return pno.GetOrElse("")
}

If we solve the issue mentioned before (https://github.com/golang/go/issues/49085), it's final version will be like this:
func GetStudentPhoneNumber(resp *QueryStudentResponse) string {
	return OfNullable(resp).
		Map(f -> f.Student).
		Map(f -> f.BackPack).
		Map(f -> f.Phone).
		Map(f -> &f.Number).
		pno.GetOrElse("")
}

Finally all the same like the Optional in Java. And this will also make create things like Stream in Java possible too. These two works together with generics will make it much more convenient to dispose all kinds of tedious while inevitable tasks. Meanwhile, I think this lambda is not big change since it is just a syntactic sugar, so I think this proposal should put in Go 1 instead Go 2 (I kept the orignal generated title lol).

Who does this proposal help, and why?
Please refer to the above part
What would change in the language spec?
anonymous function declaration
Is this change backward compatible?

It is backward compatible.
Before
Please refer to the above part
After
Please refer to the above part

Orthogonality: how does this change interact or overlap with existing features?
Is the goal of this change a performance improvement?
No. Only impact the compile phase

Costs

Would this change make Go easier or harder to learn, and why?
No big difference. It is just a easier way to express the same idea
What is the cost of this proposal? (Every language change has a cost).
Syntatic sugar, not a big cost in my view.
How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected?
I think only gofmt will be affected.
What is the compile time cost?
Yes this will cost more compilation time.
What is the run time cost?
No big difference.
Can you describe a possible implementation?
As mentioned above.
Do you have a prototype? (This is not required.)
As mentioned above.

Duplicate of #21498

Got it. The reason I refered it as a better example is that I found the existing issue discussed that the example displayed is not very convincing, and I think this one would be more convincing. Anyway I have joined the discussion there and refered this example. Thanks!


Can't move cursor by arrow keys while renaming

The arrow keys were bound to expand or collapse items by default and it looks not to be considered to skip the listener while renaming. Can we add a condition !tree.isRenaming here?
  useKey(
    'arrowright',
    e => {
    },
    isActiveTree && !dnd.isProgrammaticallyDragging && !tree.isRenaming
  );

  useKey(
    'arrowleft',
    e => {
    },
    isActiveTree && !dnd.isProgrammaticallyDragging && !tree.isRenaming
  );

Fixed in 1.1.4


copy/paste interrupted by log panel redraws

I have some trouble copy/pasting from process-compose (PC) 0.88 log panel with kitty on wayland.
I tried to unfollow the log but even then, when I start selecting text, PC seems to redraw the screen which breaks my selection.
Not redrawing the log panel should fix this.

I am not sure they are the same, I use the terminal copy mode that should be independant of the application, and I can copy/paste with process-compose, just the selection gets interrupted by kitty because of the terminal redraws I suppose (not 100% sure though) ?

Unfortunately, I had to disable the selection mode in Wayland sessions.
You can try it by unsetting the WAYLAND_DISPLAY env variable.


Update for TSTool 14.6.0 release

Need to update the documentation for the TSTool 14.6.0 release.

I updated the documentation for the release.  Close this issue.


Add Mel's info

Mel, you added 5000 + files somehow.

What, lol. Bundle mistake somehow? I'll try to fix it tomorrow, thanks for pointing it out


🎛  New Control panel : Create boilerplate and establish tech structure

Background
The ticket is to establish the basic technique structure of New Control panel
Scope

Repo:  data-platform-control-panel
Web framework
Features

Login flow by using Authentication Platform (Auth0 + AAD)
Basic User & Permission management

Database storage

User

UI
auto-testing
Choose where the cpanel is deployed
Deployment related stuff

Value / Purpose
Get the "empty" app ready for main feature implementation
User Types
No response
Proposal
Based on the existing Control panel with

Python django framework
Login flow
PostgresSQL
UI (minimum)

Definition of Done

[x] Repo
[ ] Runnable code with simple endpoints
[ ] Deployed on chosen platform
[ ] Runnable auto-tesitng

See attached link for the User Research Catalogue user journey flow for your information - https://miro.com/app/board/uXjVNQBMdwU=/

update:  we had a chat yesterday about how the app will be deployed,  a few tickets have been raised as follow-up,  so I will remove the deployment from this ticket


LAMMPS return nan for potential energy, tot energy, pressure, etc

I have a Flare++ generated LAMMPS potential with the following header:
DATE: Thu Sep 15 08:15:18 2022 CONTRIBUTOR: Martin Siron
2
chebyshev
2 15 3 1730730
quadratic
7.00 7.00 7.00 7.00
 -3.7081202824163029e-01 -7.5121626026607249e+00  2.0285553143110349e+00 -2.5931799006379350e+00 -6.7339221940738980e-01
 -4.1086598196837585e+00  1.3867461307437310e+00 -1.3400453595520432e+00 -6.9342900823929199e-01  2.4547893054698768e+00
 -3.2365345306921256e-01  1.0697579523633678e+00 -1.0009252464915903e+00  5.5103596436798528e+00 -2.1893434014683351e+00
  2.1787352297543379e+00 -1.1271605000247291e+00  2.9668920730647503e+00 -2.6111849900449071e+00  1.2615246000666112e+00
 -2.9663719425271040e-01 -9.3284008733906543e-01 -5.8615111890141236e-01 -7.4564621228041972e-02  1.4580132256288607e+00
 -1.4538317320828966e+00  2.7242482436344915e+00 -2.8533072739625709e-01  2.8989992114976531e+00  8.1221376518358834e-01
  4.3954511805509924e+00 -1.3443434289366962e-01  2.6440157329586711e+00  1.3811351812136543e+00  2.5036757080927146e+00
 -1.4827050791523391e+00  6.5440934762730762e-01 -1.6827116757154350e+00 -1.5447686540050540e+00 -3.7877171837065617e+00

I have previously compiled LAMMPS with Flare++ and successfully ran MD calculation with potential with the following header:
DATE: Tue Sep  6 13:37:36 2022 CONTRIBUTOR: Martin Siron
2
chebyshev
2 12 3 720600
quadratic
3.70 3.70 3.70 3.70
 -6.8142479285372162e-02 -1.7471659356174324e-01  8.9928444575977290e-02  9.5382460755559270e-02 -1.0573593939254869e-01
 -1.2702611633853572e-01  6.1525379261817802e-02  8.4021357499511784e-02 -2.8557872564111516e-02 -1.0541056145460515e-02
 -7.8123686721719710e-03  4.8391611702859487e-02  5.8877126903733215e-02  1.0994480866102428e-01 -7.9615695655338978e-02
 -1.0038966687721338e-02  1.1600338166083181e-01  1.6778484081738967e-01 -1.1450035780444216e-01 -7.5884153313299924e-02

I cannot get LAMMPS to run properly with the new potential -- it does not seem to be able to calculate the energies:
LAMMPS (29 Sep 2021 - Update 3)
KOKKOS mode is enabled (src/KOKKOS/kokkos.cpp:97)
  will use up to 1 GPU(s) per node
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
  In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
  For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
  For unit testing set OMP_PROC_BIND=false
  using 1 OpenMP thread(s) per MPI task
Reading data file ...
  orthogonal box = (0.0000000 0.0000000 0.0000000) to (11.408130 11.408130 19.495504)
  1 by 1 by 1 MPI processor grid
  reading atoms ...
  215 atoms
  read_data CPU = 0.002 seconds
Replicating atoms ...
  orthogonal box = (0.0000000 0.0000000 0.0000000) to (34.224390 34.224390 58.486512)
  1 by 1 by 1 MPI processor grid
  5805 atoms
  replicate CPU = 0.003 seconds
Reading potential file TiO2.txt with DATE: Thu
FLARE will use up to 90.00 GB of device memory, controlled by MAXMEM environment variable
Neighbor list info ...
  update every 1 steps, delay 10 steps, check yes
  max neighbors/atom: 2000, page size: 100000
  master list distance cutoff = 9
  ghost atom cutoff = 9
  binsize = 9, bins = 4 4 7
  1 neighbor lists, perpetual/occasional/extra = 1 0 0
  (1) pair flare/kk, perpetual
      attributes: full, newton on, kokkos_device
      pair build: full/bin/kk/device
      stencil: full/bin/3d
      bin: kk/device
Setting up Verlet run ...
  Unit style    : metal
  Current step  : 0
  Time step     : 0.005
Per MPI rank memory allocation (min/avg/max) = 4.539 | 4.539 | 4.539 Mbytes
Step PotEng KinEng TotEng Temp Press Density
       0         -nan    750.22588         -nan         1000         -nan    3.7529171
     100            0         -nan         -nan         -nan         -nan    3.7529171
     200            0         -nan         -nan         -nan         -nan    3.7529171

I have a system with a V100 and 100GB of memory.
My input file is as so:
units           metal
boundary        p p p

atom_style      atomic

read_data       data.meam

replicate 3 3 3

newton on
pair_style      flare
pair_coeff      * * flare.txt

timestep 0.005
velocity all create 1000.0 454883 mom yes rot yes dist gaussian

thermo_style custom step pe ke etotal temp press density
thermo 100

fix 2 all nvt temp 1000.00 2400.00 0.1

dump           1 all atom 10000 dump.meam

run             10000000

I am unsure how to troubleshoot.

Hi @mhsiron
So you have the same code/executable, but the old potential runs well while the new potential does not? Have you tried running with the CPU code? Just to make sure whether it is a problem of Kokkos

Hi @YuuuXie thanks for your reply. It does appear to be related to Kokkos -- running without newton/kokkos appears to output computed energies.
What might make the first potential work with kokkos that does not work with the second potential?
Thanks for your help!

Hi Martin,
This is puzzling. So with the same executable, it works with one potential file, but not the other?
What command do you use to run LAMMPS?

Exactly! I am using:
lmp -k on g 1 -sf kk -pk kokkos newton on neigh full -in in.script
Happy to provide the potentials as well (attached).
The one that gives me the problem is the one where I used 7A as the cutoff -- my only guess is it could have to do with memory? The 7A must be larger to process. But that's just my guess...
TiO2_3A.zip
TiO2_7A

That command looks good.
Do you happen to have your input structure as well? Does it have any weird features like isolated atoms?
Note that we are in the process of merging this FLARE++ code into the main FLARE repository, it is currently on the development branch: https://github.com/mir-group/flare/tree/development
There's a chance there are some bugfixes in there that were done after migrating from flare_pp, so you could try to use the files in https://github.com/mir-group/flare/tree/development/lammps_plugins

Here's the input structure. It does not have any isolated atoms.
I will work on building with the development branch to see if that removes this error!
data.zip

I am unable to compile LAMMPS with the development branch of Flare:
3 errors detected in the compilation of "/lammps/src/compute_flare_std_atom.cpp".
CMakeFiles/lammps.dir/build.make:789: recipe for target 'CMakeFiles/lammps.dir/lammps/src/compute_flare_std_atom.cpp.o' failed
gmake[2]: *** [CMakeFiles/lammps.dir/ammps/l/lammps/src/compute_flare_std_atom.cpp.o] Error 1

I see now that your output contains
FLARE will use up to 90.00 GB of device memory, controlled by MAXMEM environment variable

which indicates that you've set the environment variable MAXMEM=90. Does your GPU actually have 90 GB of memory?
As for the build errors on the development branch, you may have to first uninstall the old patch from this directory, ./uninstall.sh /path/to/lammps from lammps_plugins/, then ./install.sh /path/to/lammps, since there are some changes to our CMake patch (the development branch should run quite a bit faster because it uses KokkosKernels/cuBLAS).

Hi @anjohan even with the MAXMEM=10, I get the following output:
LAMMPS (29 Sep 2021 - Update 3)
KOKKOS mode is enabled (src/KOKKOS/kokkos.cpp:97)
  will use up to 1 GPU(s) per node
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
  In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
  For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
  For unit testing set OMP_PROC_BIND=false
  using 1 OpenMP thread(s) per MPI task
Reading data file ...
  orthogonal box = (0.0000000 0.0000000 0.0000000) to (11.408130 11.408130 19.495504)
  1 by 1 by 1 MPI processor grid
  reading atoms ...
  215 atoms
  read_data CPU = 0.004 seconds
Reading potential file TiO2.txt with DATE: Thu
FLARE will use up to 12.00 GB of device memory, controlled by MAXMEM environment variable
Neighbor list info ...
  update every 1 steps, delay 10 steps, check yes
  max neighbors/atom: 2000, page size: 100000
  master list distance cutoff = 9
  ghost atom cutoff = 9
  binsize = 9, bins = 2 2 3
  1 neighbor lists, perpetual/occasional/extra = 1 0 0
  (1) pair flare/kk, perpetual
      attributes: full, newton on, kokkos_device
      pair build: full/bin/kk/device
      stencil: full/bin/3d
      bin: kk/device
Setting up Verlet run ...
  Unit style    : metal
  Current step  : 0
  Time step     : 0.005
Per MPI rank memory allocation (min/avg/max) = 2.219 | 2.219 | 2.219 Mbytes
Step PotEng KinEng TotEng Temp Press Density
       0         -nan    27.661671         -nan         1000         -nan    3.7529171
     100            0         -nan         -nan         -nan         -nan    3.7529171
     200            0         -nan         -nan         -nan         -nan    3.7529171
     300            0         -nan         -nan         -nan         -nan    3.7529171

It seems Flare ignores MAXMEM<12?
As for compiling this was on a fresh LAMMPS download, but for 29Sep2021 update 3. I just tried it on 17Feb2022 and it appears to fail at the same error:
3 errors detected in the compilation of "lammps/l2/lammps/src/compute_flare_std_atom.cpp".
CMakeFiles/lammps.dir/build.make:789: recipe for target 'CMakeFiles/lammps.dir/lammps/l2/lammps/src/compute_flare_std_atom.cpp.o' failed
gmake[2]: *** [CMakeFiles/lammps.dir/lammps/l2/lammps/src/compute_flare_std_atom.cpp.o] Error 1
CMakeFiles/Makefile2:769: recipe for target 'CMakeFiles/lammps.dir/all' failed
gmake[1]: *** [CMakeFiles/lammps.dir/all] Error 2
Makefile:135: recipe for target 'all' failed
gmake: *** [all] Error 2

Here is the full error output in VERBOSE=1 mode:
[ 18%] Building CXX object CMakeFiles/lammps.dir/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp.o
/nfs/site/disks/msironml/lammps/l2/lammps/lib/kokkos/bin/nvcc_wrapper -DFFT_KISS -DKOKKOS_DEPENDENCE -DLAMMPS_GZIP -DLAMMPS_JPEG -DLAMMPS_MEMALIGN=64 -DLAMMPS_OMP_COMPAT=4 -DLAMMPS_PNG -DLAMMPS_SMALLBIG -DLMP_KOKKOS -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX -D_MPICC_H -I/nfs/site/disks/msironml/lammps/l2/lammps/src -I/nfs/site/disks/msironml/lammps/l2/lammps/src/KSPACE -I/nfs/site/disks/msironml/lammps/l2/lammps/src/MACHDYN -I/nfs/site/disks/msironml/lammps/l2/lammps/src/MANYBODY -I/nfs/site/disks/msironml/lammps/l2/lammps/src/MOLECULE -I/nfs/site/disks/msironml/lammps/l2/lammps/src/RIGID -I/nfs/site/disks/msironml/lammps/l2/lammps/lib/kokkos/core/src -I/nfs/site/disks/msironml/lammps/l2/lammps/lib/kokkos/containers/src -I/nfs/site/disks/msironml/lammps/l2/lammps/lib/kokkos/algorithms/src -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/lib/kokkos -I/nfs/site/disks/msironml/lammps/l2/lammps/src/KOKKOS -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/styles -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/lib/kokkos/core/src -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/lib/kokkos/containers/src -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/lib/kokkos/algorithms/src -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-build/src -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/impl -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-build/src/impl -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/impl/tpls -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/blas -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/blas/impl -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/sparse -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/sparse/impl -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/graph -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/graph/impl -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/batched -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/batched/dense -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/batched/dense/impl -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/batched/sparse -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/batched/sparse/impl -I/nfs/site/disks/msironml/lammps/l2/lammps/build1/_deps/kokkoskernels-src/src/common -isystem /nfs/site/itools/em64t_SLES12SP5/pkgs/ics/2019.4/compilers_and_libraries_2019.4.243/linux/mpi/intel64/include -isystem /nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build -O2 -g -DNDEBUG -fPIC -fopenmp -expt-extended-lambda -Wext-lambda-captures-this -arch=sm_70 -Xcompiler -std=c++14 -MD -MT CMakeFiles/lammps.dir/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp.o -MF CMakeFiles/lammps.dir/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp.o.d -o CMakeFiles/lammps.dir/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp.o -c /nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp
/nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build/Eigen/src/Core/DenseStorage.h(223): warning: __host__ annotation is ignored on a function("DenseStorage") that is explicitly defaulted on its first declaration

/nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build/Eigen/src/Core/DenseStorage.h(223): warning: __device__ annotation is ignored on a function("DenseStorage") that is explicitly defaulted on its first declaration

/nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build/Eigen/src/Core/DenseStorage.h(233): warning: __host__ annotation is ignored on a function("operator=") that is explicitly defaulted on its first declaration

/nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build/Eigen/src/Core/DenseStorage.h(233): warning: __device__ annotation is ignored on a function("operator=") that is explicitly defaulted on its first declaration

/nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build/Eigen/src/Core/DenseStorage.h(248): warning: __host__ annotation is ignored on a function("DenseStorage") that is explicitly defaulted on its first declaration

/nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build/Eigen/src/Core/DenseStorage.h(248): warning: __device__ annotation is ignored on a function("DenseStorage") that is explicitly defaulted on its first declaration

/nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build/Eigen/src/Core/DenseStorage.h(249): warning: __host__ annotation is ignored on a function("operator=") that is explicitly defaulted on its first declaration

/nfs/site/disks/msironml/lammps/l2/lammps/build1/Eigen3_build-prefix/src/Eigen3_build/Eigen/src/Core/DenseStorage.h(249): warning: __device__ annotation is ignored on a function("operator=") that is explicitly defaulted on its first declaration

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(84): error: class "LAMMPS_NS::Neighbor" has no member "add_request"

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(84): error: namespace "LAMMPS_NS::NeighConst" has no member "REQ_FULL"

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(84): error: namespace "LAMMPS_NS::NeighConst" has no member "REQ_OCCASIONAL"

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(129): warning: variable "beta_init" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(129): warning: variable "beta_counter" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(130): warning: variable "B2_val_1" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(130): warning: variable "B2_val_2" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(106): warning: variable "nall" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(107): warning: variable "newton_pair" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(435): warning: variable "n_size" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(437): warning: variable "beta_val" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(471): warning: variable "tmp" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(471): warning: variable "nwords" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(571): warning: variable "radial_string" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(571): warning: variable "cutoff_string" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(572): warning: variable "radial_string_length" was declared but never referenced

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(572): warning: variable "cutoff_string_length" was declared but never referenced

3 errors detected in the compilation of "/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp".
CMakeFiles/lammps.dir/build.make:789: recipe for target 'CMakeFiles/lammps.dir/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp.o' failed
make[2]: *** [CMakeFiles/lammps.dir/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp.o] Error 1
make[2]: Leaving directory '/nfs/site/disks/msironml/lammps/l2/lammps/build1'
CMakeFiles/Makefile2:769: recipe for target 'CMakeFiles/lammps.dir/all' failed
make[1]: *** [CMakeFiles/lammps.dir/all] Error 2
make[1]: Leaving directory '/nfs/site/disks/msironml/lammps/l2/lammps/build1'
Makefile:135: recipe for target 'all' failed
make: *** [all] Error 2

On LAMMPS stable_23Jun2022, it appears to not have this issue, so far. Will update if compile finishes.
Perhaps a new version of LAMMPS has the following:
/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(84): error: class "LAMMPS_NS::Neighbor" has no member "add_request"

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(84): error: namespace "LAMMPS_NS::NeighConst" has no member "REQ_FULL"

/nfs/site/disks/msironml/lammps/l2/lammps/src/compute_flare_std_atom.cpp(84): error: namespace "LAMMPS_NS::NeighConst" has no member "REQ_OCCASIONAL"

Confirmed that LAMMPS stable_23June2022 compiled with development branch of Flare LAMMPS plugin with Kokkos. With the same CMAKE compiler tags I used it did not work with 17Feb2022 or 29Sep2021 Update 3.
In terms of my original problem, the "7A" flare PP potential, the Flare development branch did not fix this:
LAMMPS (23 Jun 2022)
KOKKOS mode is enabled (src/KOKKOS/kokkos.cpp:105)
  will use up to 1 GPU(s) per node
  using 1 OpenMP thread(s) per MPI task
Reading data file ...
  orthogonal box = (0 0 0) to (11.40813 11.40813 19.495504)
  1 by 1 by 1 MPI processor grid
  reading atoms ...
  215 atoms
  read_data CPU = 0.002 seconds
Reading potential file TiO2.txt with DATE: Thu
FLARE will use up to 12.00 GB of device memory, controlled by MAXMEM environment variable
Neighbor list info ...
  update every 1 steps, delay 10 steps, check yes
  max neighbors/atom: 2000, page size: 100000
  master list distance cutoff = 9
  ghost atom cutoff = 9
  binsize = 9, bins = 2 2 3
  1 neighbor lists, perpetual/occasional/extra = 1 0 0
  (1) pair flare/kk, perpetual
      attributes: full, newton on, kokkos_device
      pair build: full/bin/kk/device
      stencil: full/bin/3d
      bin: kk/device
Setting up Verlet run ...
  Unit style    : metal
  Current step  : 0
  Time step     : 0.005
Per MPI rank memory allocation (min/avg/max) = 2.219 | 2.219 | 2.219 Mbytes
   Step         PotEng         KinEng         TotEng          Temp          Press         Density
         0  -nan            27.661671     -nan            1000          -nan            3.7529171
       100   0             -nan           -nan           -nan           -nan            3.7529171
       200   0             -nan           -nan           -nan           -nan            3.7529171

@mhsiron Have you tried running it with Kokkos in either Serial or OpenMP mode?
Also: Your 7 Å potential link has expired, so I'm not able to download it.

I'm not too familiar with Kokkos -- would you mind letting me know the command for Serial or OpenMP mode?
From this is my understanding that I would have to compile for each mode correct?
Here's the 7A potential again!

Hi @mhsiron ,
Sorry that I didn't get around to following up on this.

Yes, you can compile it with either -DKokkos_ENABLE_OPENMP=ON (for OpenMP) or nothing (for Serial), then replace the g 1 with t 1 (1 thread or Serial) or t 4 (4 threads).

Your potential link expired again.

Hi @anjohan ,
No worries -- completely understand! Lets try round 3 with the potential link haha! Here it is (looks like this service is 30 days): link.
I will attempt the recompile -- I am working with my cluster to upgrade our CUDA to be on par with later PyTorch versions first to see if this may fix some issues. Will update soon.

I used the following compile cmd:
cmake -C ../cmake/presets/kokkos-serial.cmake -C ../cmake/presets/kokkos-openmp.cmake ../cmake -DPKG_KOKKOS=ON  -DKokkos_ENABLE_OPENMP=ON -DBUILD_OMP=yes
This was with GCC 12.1, CMake 3.25

I am still waiting on the cluster to upgrade CUDA to either 11.3 or 11.6 to see if this resolves the problem for Kokkos with CUDA support.

After upgrading to CUDA 11.6, and recompiling, I am still reporting the same error with Kokkos+GPU. Only Kokkos+OMP seems to work for 7A potential. 4A potential has no issue on either compilations.

Hi @mhsiron ,
Thank you for investigating further!
I finally got around to running this on my own setup, and it does indeed produce nan (both energies and forces) for me as well with the large cutoff. I'm looking into it.

Hm, it appears that when the cutoff (and thus the number of neighbors) grows too large, CUDA simply doesn't launch the radial and spherical harmonics basis calculation, giving no warning whatsoever.
For now, you can get it to run by replacing the second max_neighs with something like std::min(max_neighs, 32) on line 276 of pair_flare_kokkos.cpp.


K8s Pod/namespace information in exported fields

I would like to know if the dcgm -exporter can be configured in a way that k8s pod/namespace infos of the pod currently using a GPU can be exported along with the utilization metrics.
Let me describe quickly my motivation (maybe there is another solution to achieve my goal):
I would like to track which pods in a K8s cluster are using what amount of GPU resources. This could then be used for reporting/showback to different teams, capacity planning and also optimization suggestions. In a large cluster with lots of GPUs this is challenging at times

You should be using something like kube-state-metrics for that https://github.com/kubernetes/kube-state-metrics
It exposes pod request/limit as well as node capacity/allocatable information.

I know what you mean but as far as I can see the metrics from kube state metrics and the ones from the GPU monitoring are not connected to each other. So I know what pods/processes run on a node and also what GPUs are on that node and how well/badly they were used but I don't know which pod it was.
Please let me know if I misunderstood you or if I'm not aware of something obvious to enable this use case for my workloads.

I know what you mean but as far as I can see the metrics from kube state metrics and the ones from the GPU monitoring are not connected to each other. So I know what pods/processes run on a node and also what GPUs are on that node and how well/badly they were used but I don't know which pod it was.
Please let me know if I misunderstood you or if I'm not aware of something obvious to enable this use case for my workloads.

i have the same exact problem, so far I found that https://developer.nvidia.com/blog/monitoring-gpus-in-kubernetes-with-dcgm/  Nvidia GPU monitoring looks like it's able to pull in the metrics with gpu/pod assignment. So far I can only see it from the screenshots on this page : https://docs.nvidia.com/datacenter/cloud-native/kubernetes/dcgme2e.html#gpu-telemetry
you can see that the first Prometheus screenshot has all the needed metrics. Question is, how do I implement it? from my understanding, this feature is included in the latest version of gpu-monitoring-tools, so it looks like, in my case, it's just as simple as loading a new image onto my GPU cluster, and update Kubernetes to at least version 1.12. Hopefully someone is able to provide more details.

turns out dcgm-exporter can do that already, it is just disabled by default if you install it via gpu-operator. You can eighter install dcgm-exporter via helm because the chart is setting up everything correctly or you modify your daemonset manually.
Make sure you have the env variable DCGM_EXPORTER_KUBERNETES=true volumeMount to /var/lib/kubelet/pod-resources in your container spec.
containers:
      ...
        env:
        - name: DCGM_EXPORTER_KUBERNETES
          value: "true"
        ...
        volumeMounts:
        - mountPath: /var/lib/kubelet/pod-resources
          name: pod-gpu-resources
          readOnly: true

I'm going to close this issue and open one in the gpu-operator repo.


Domain join does not work?

Hi!
I have a question rather than a problem this time.
I implemented this on AZURE (pipeline and release is in place), and created two servers (DSCDC01 and DSCFile01) for testing.
The automation account kicks in and applies most of the config, but fails on the "Computer" part.

The domain controller is configured, and a domain admin account install was created to facilitate the joining.

Is there anything I missed?
Thanks,
F.
PS: Otherwise great job, really appreciate the effort put into this

TimeCreated  : 9/29/2022 12:25:01 PM
ProviderName : Microsoft-Windows-DSC
Id           : 4508
Message      : Job {B4D36778-3FF1-11ED-A810-0022489A36E6} :
               Attempting to send the status report using Report Manager WebDownloadManager. AgentId is 07809499-3FF1-11ED-A810-0022489A36E6.

TimeCreated  : 9/29/2022 12:25:01 PM
ProviderName : Microsoft-Windows-DSC
Id           : 4252
Message      : Job {B4D36778-3FF1-11ED-A810-0022489A36E6} :
               MIResult: 1
               Error Message: Computer 'DSCFile01' failed to join domain 'contoso.com' from its current workgroup 'WORKGROUP' with following error message: The system cannot find the file specified.
               Message ID: FailToJoinDomainFromWorkgroup,Microsoft.PowerShell.Commands.AddComputerCommand
               Error Category: 14
               Error Code: 14
               Error Type: MI

TimeCreated  : 9/29/2022 12:25:01 PM
ProviderName : Microsoft-Windows-DSC
Id           : 4103
Message      : Job {B4D36778-3FF1-11ED-A810-0022489A36E6} :
               This event indicates that a non-terminating error was thrown when DSCEngine was executing Set-TargetResource on DSC_Computer DSC resource. FullyQualifiedErrorId is
               FailToJoinDomainFromWorkgroup,Microsoft.PowerShell.Commands.AddComputerCommand. Error Message is Computer 'DSCFile01' failed to join domain 'contoso.com' from its current workgroup 'WORKGROUP' with following error
               message: The system cannot find the file specified..

TimeCreated  : 9/29/2022 12:24:53 PM
ProviderName : Microsoft-Windows-DSC
Id           : 4332
Message      : Job {B4D36778-3FF1-11ED-A810-0022489A36E6} :
                Resource execution sequence :: [File]file_Z__DoesNotWork::[FilesAndFolders]FilesAndFolders, [File]file_C__Test_Dev_Environment::[FilesAndFolders]FilesAndFolders,
               [File]file_C__Test_Frankfurt::[FilesAndFolders]FilesAndFolders, [File]file_C__Test::[FilesAndFolders]FilesAndFolders, [File]file_C__Test_Test1File1.txt::[FilesAndFolders]FilesAndFolders,
               [File]file_C__Test_Test1File2.txt::[FilesAndFolders]FilesAndFolders, [File]file_C__GpoBackup::[FilesAndFolders]FilesAndFolders,
               [xRegistry]HKEY_LOCAL_MACHINE_SYSTEM_CurrentControlSet_Services_Netlogon_Parameters__DBFlag::[RegistryValues]RegistryValues,
               [xRegistry]HKEY_LOCAL_MACHINE_SYSTEM_CurrentControlSet_Services_W32Time_Parameters__NtpServer::[RegistryValues]RegistryValues, [WindowsFeature]File-Services::[WindowsFeatures]WindowsFeatures,
               [WindowsFeature]Telnet-Client::[WindowsFeatures]WindowsFeatures, [xWindowsFeature]DisableSmbV1::[SecurityBase]SecurityBase, [PowerShellExecutionPolicy]ExecutionPolicyAllSigned::[SecurityBase]SecurityBase,
               [UserRightsAssignment]DenyLogonLocallyForAdministrator::[SecurityBase]SecurityBase, [UserRightsAssignment]AllowLogonLocally::[SecurityBase]SecurityBase,
               [Computer]ComputerDSCFile01::[ComputerSettings]ComputerSettings, [TimeZone]TimeZone::[ComputerSettings]ComputerSettings, [NetIPInterface]DisableDhcp_DscWorkshop 0::[NetworkIpConfiguration]NetworkIpConfiguration,
               [IPAddress]NetworkIp_DscWorkshop 0::[NetworkIpConfiguration]NetworkIpConfiguration, [DefaultGatewayAddress]DefaultGateway_DscWorkshop 0::[NetworkIpConfiguration]NetworkIpConfiguration,
               [DnsServerAddress]DnsServers_DscWorkshop 0::[NetworkIpConfiguration]NetworkIpConfiguration, [WinsSetting]LmhostsLookup_DscWorkshop 0::[NetworkIpConfiguration]NetworkIpConfiguration,
               [WindowsEventLog]System::[WindowsEventLogs]WindowsEventLogs, [WindowsEventLog]Application::[WindowsEventLogs]WindowsEventLogs, [WindowsEventLog]Security::[WindowsEventLogs]WindowsEventLogs,
               [xRegistry]DscLcmController_MaintenanceWindowMode::[DscLcmController]DscLcmController, [xRegistry]DscLcmController_MonitorInterval::[DscLcmController]DscLcmController,
               [xRegistry]DscLcmController_AutoCorrectInterval::[DscLcmController]DscLcmController, [xRegistry]DscLcmController_AutoCorrectIntervalOverride::[DscLcmController]DscLcmController,
               [xRegistry]DscLcmController_RefreshInterval::[DscLcmController]DscLcmController, [xRegistry]DscLcmController_RefreshIntervalOverride::[DscLcmController]DscLcmController,
               [xRegistry]DscLcmController_ControllerInterval::[DscLcmController]DscLcmController, [xRegistry]DscLcmController_MaintenanceWindowOverride::[DscLcmController]DscLcmController,
               [xRegistry]DscLcmController_WriteTranscripts::[DscLcmController]DscLcmController, [xRegistry]DscLcmController_MaxLcmRuntime::[DscLcmController]DscLcmController,
               [xRegistry]DscLcmController_LogHistoryTimeSpan::[DscLcmController]DscLcmController, [xRegistry]DscLcmController_SendDscTaggingData::[DscLcmController]DscLcmController,
               [File]DscLcmControllerScript::[DscLcmController]DscLcmController, [ScheduledTask]DscControllerTask::[DscLcmController]DscLcmController,
               [Script]MaintenanceWindowsCheck::[DscLcmMaintenanceWindows]DscLcmMaintenanceWindows, [xRegistry]StartTime_Always::[DscLcmMaintenanceWindows]DscLcmMaintenanceWindows,
               [xRegistry]Timespan_Always::[DscLcmMaintenanceWindows]DscLcmMaintenanceWindows, [xRegistry]DayOfWeek_Always::[DscLcmMaintenanceWindows]DscLcmMaintenanceWindows,
               [xRegistry]On_Always::[DscLcmMaintenanceWindows]DscLcmMaintenanceWindows, [xRegistry]DscVersion::[DscTagging]DscTagging, [xRegistry]DscEnvironment::[DscTagging]DscTagging,
               [xRegistry]DscBuildDate::[DscTagging]DscTagging, [xRegistry]DscBuildNumber::[DscTagging]DscTagging, [xRegistry]DscModules::[DscTagging]DscTagging,
               [JeaRoleCapabilities]ReadDiagnosticRole::[DscDiagnostic]DscDiagnostic, [JeaSessionConfiguration]DscEndpoint::[DscDiagnostic]DscDiagnostic.

This seems to be the log from event view

This has more to do with the DSC resource that handles the domain join than with the DscWorkshop.
How do the parameters look like that you pass on to the resource? Can you provide the section of the RSOP files? Does a domain join work when using the UI or the cmdlet Add-Computer?

Please keep this open - I will come back to this - I think I have some notion on what went wrong; to be fair I am a bit struggling with how to use the certificate for encryption, once I got that sorted I test this.

Did it work?


rasa run: remove default model path

Proposed changes:

Remove default value of argument model for rasa run so that it is actually possible to fetch a model from a server.

fixes https://github.com/RasaHQ/rasa/issues/3931
Status (please check what you already did):

[x] made PR ready for code review
[ ] added some tests for the functionality
[x] updated the documentation
[ ] updated the changelog
[x] reformat files using black (please check Readme for instructions)

So in a quick test, i opened up a bot, ran rasa run —connector cmdline, and got
No model found. Train a model before running the server using `rasa train`.

after training and running the same command, i got the same error (on master my bot loads). Does this officially mean I must pass -m models now? If so, we need a better error and additions to the docs, etc

Yes, if you want to use a local model, you need to pass -m models. Will update the error message.

This is kind of a bummer since people have gotten used to rasa run just doing its magic — is there any other workaround, by e.g. putting the default in later after checking for a model from a server? Not sure how that fetching code works.

The problem is that we always first check if there is a model path given, if that is the case, we assume the user wants to load a local model. Only if there is no model path, e.g. None, we load it from the server or remote storage. So, if we now have a default model path, we never reach the point where we are loading from a server.
But I can take another look, maybe I can find another solution. Will ping you once I have a solution.

Only if there is no model path, e.g. None, we load it from the server or remote storage.

and then if this fails to return a model, or they have no remote storage configured, can we then try to load it from local? i.e. we would switch the order in which we run these?

Updated the PR. Changed the loading order.

oops, we definitely should have re-named the PR after updating its functionality 😅

Mhm, but rasa run with no model in models/ would still fail right? Cause we have the check in cli/run whether a model is present.


Map char ranges for case-insensitivity at parse time.

Fix #75.

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @huonw (or someone else) soon.
If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.
Please see CONTRIBUTING.md for more information.

Included in #78.


Snippets integration

While it was quite handy to be able to add a button for the "Snippets: Available" command, it would be even nicer if one could add a button for a particular snippet...

I'm not sure how one would do that UI, but I've added it as an enhancement.

I like the idea.
Instead of simply showing the existing commands when clicking the three bars button, you could show a menu. In this menu I can select either "Add a new snippet" or "Select command".
You may also offer a setting "show all snippets in one drop down" which will then do exactly this instead of showing a single button for each.
Another idea to add snippets would be a snippets section in the settings.


Add a option to draw the back of the shirt.

This adds an isBack option to the specs, and changes the neckline and hides the badges if that is set.

This is really cool @ewanmellor 😎  thanks for this ⚽ 🥅 💥