pkgs.dockertools.buildLayeredImage: customisable layering strategy

Allow customisation of the algorithm used to convert nix references graph (created from docker image contents) to docker layers.
A collection of building blocks (python functions) is provided, which can be assembled into a processing pipeline by specifying a list of operations (and their initial arguments) via a nix lang list.
A graph of nix references is first converted into a python igraph.Graph object (with each vertex representing a nix path), and then fed into the user defined pipeline. Each stage in the pipeline represents a function call, with initial arguments specified by in nix, and the last argument being the result of the previous stage in the pipeline (or the initial Graph object).
Each step of the pipeline is expected to produce a data structure consisting of arbitrarily nested lists/dicts with Graph objects (representing docker layers) at it's leafs. The result of the last stage in the pipeline is recursively flattened (with each dict converted into list of values), until a flat list of Graphs remains. This is then output as a json array of arrays (each Graph converted into an array of paths).
This functionality is made available via new layeringPipeline argument to the streamLayeredImage/buildLayeredImage functions. The default value of the argument has been chosen to to preserve current layering. behaviour.
Functions available to use in the pipeline:

popularity_contest: sorts paths by popularity, with each path put in it's own layer. This, when followed by limit_layers, results in layers identical to current implementation (implementation copied and adapted from closure-graph.py).
subcomponent_out: splits given nix references graph into 2 graphs (preserving edges within each sub-graph):

main: includes specified paths and all of their dependencies (including transitive)
rest: all remaining paths

subcomponent_in: similar to the above but main output includes dependants of the provided paths (instead of dependencies)
split_paths: implements idea described by @roberth, by removing edges targeting given paths, and splitting resulting graph into 3:

main: specified paths + their dependencies - common paths (see below)
common: paths which can be reached from both the specified paths and any of the remaining roots of the input graph
'rest': paths which cannot be reached from the specified paths

limit_layers: limits the number of layers to desired amount by combining excess layers from the end of the list into one
reverse: reverses given list of layers. This, if inserted between popularity_contest and limit_layers, achieves the same result as layeredStrategies.popularityWeightedTop from this PR.
split_every: splits given graph into layers with given number of paths each (apart from the last layer, which might contain less paths).
remove_paths: removes vertices corresponding to given paths from a Graph (most likely not a good idea to use this apart for removing roots)
over: maps given function over a value of a given key in a dict. Similar to over from haskell lenses, or ramda. Useful for applying further transformations to selected sub-graph in the result of subcomponent_out/in or split_paths.
map: map given function over all elements of a list

Some images to demonstrates behaviour of subcomponent_out/in and split_paths.
Given the following input graph, and paths argument set to ["A"]:

Result of subcomponent_out:

With main output consisting of the green vertices and rest of red.
Result of subcomponent_in:

Colours: as above.
Result of split_paths:

main - green
common - purple
rest - red.
Note the edge removed from the graph above. The common sub-graph is calculated by taking an intersection of the sub-graph formed by vertices which can be reached from A and one which can be reached from Root. If A was the root, then both common and rest would be empty.
Example of a custom layeringPipeline
Used to package an application written in python, optimising for sharing layers between rebuilds of the same image (but also with a layer containing python runtime with all it's dependencies, which can be shared between images of different application built on top of the same version of python).
The pipeline: https://github.com/adrian-gierakowski/nix-docker-custom-layering-strategy-example/blob/6eec668c4ef19b09926d8b53b7003575a115273e/default.nix#L42-L108
The resulting docker layers: https://github.com/adrian-gierakowski/nix-docker-custom-layering-strategy-example/blob/119c15a98a16b9b85d676d9ea86ce231d7306d55/store-layers.json
TODOs:

[ ] write proper documentation
[ ] add integration tests
[ ] better error messages when user defined pipeline is invalid. We can detect 'syntactic' errors, invalid function names and number/types of arguments. Currently there is none of this, user will simply get a cryptic error throw by python if something doesn't line up.
[ ] consider if names of subcomponent_out/in and split_paths could be improved. The former are based on corresponding method of igraph.Graph class and the latter is the first thing that came to mind when I started working on it.
[ ] split_every: allow customising to walk the graph in depth-first or breath-first mode (currently it's simply iterates the vertices in the order in which they are stored in the igraph.Graph object, which is a bit arbitrary and probably not very useful).
[ ] remove code duplication between existing references-by-popularity and the new flatten-references-graph (probably by implementing the first in terms of the latter).

Motivation for this change
Current algorithm for grouping nix paths into layers is optimised for a specific use case (sharing layers between unrelated images), but it pessimisms others. For example the reuse of layers between consecutive rebuilds of images in which amount of nix paths exceeds maxLayers. This significantly slows down development of applications with large number of dependencies (where each dep is a separate nix path).
The need for customising the layering algorithm has been highlighted in a couple of issues:
https://github.com/NixOS/nixpkgs/pull/116446
https://github.com/NixOS/nixpkgs/issues/48462
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

a little "DSL" of Nix functions in front of it

Do you mean something like this:
let
  layeringPipeline = with pkgs.dockerTools.layeringDSL; compile [
    (subcomponent_out [pkgs.python3])
    (over "rest" (pipe [
      (split_paths [ankisyncd])

Yes, something like this aligns more with what people expect from a nix file.

Perhaps we can also identify some usage patterns and exploit those to make it more intuitive.

Not sure anymore. Maybe try to limit the nesting instead?
At least for the splitting operations, the user specifies a path. This means that over calls aren't necessary to find the layer where the operation should be applied.
For popularity_contest we'll still need something like over, but I don't think that's a problem.
It also noticed that because of over, operations are applied to isolated parts of the graph, rather than the whole graph. I've tried running split_paths twice in the pipeline, without over, but that did not work.
I would expect this
nix-build --expr 'with import ./. {}; dockerMakeLayers { baseJson = writeText "json" "{}"; contentsList = [ hello figlet cowsay ]; pipeline = [ ["split_paths" [ hello ] ]   ["split_paths" [ figlet ] ] ]; }'
to work, but it ran into a
type error
Traceback (most recent call last):
  File "/nix/store/3g9j2hw15zj4xdk45b3hzcy5i27inq57-flatten-references-graph-0.1.0/bin/.flatten_references_graph-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/3g9j2hw15zj4xdk45b3hzcy5i27inq57-flatten-references-graph-0.1.0/lib/python3.8/site-packages/flatten_references_graph/__main__.py", line 44, in main
    print(main_impl(file_path))
  File "/nix/store/3g9j2hw15zj4xdk45b3hzcy5i27inq57-flatten-references-graph-0.1.0/lib/python3.8/site-packages/flatten_references_graph/__main__.py", line 24, in main_impl
    result = flatten_references_graph(
  File "/nix/store/3g9j2hw15zj4xdk45b3hzcy5i27inq57-flatten-references-graph-0.1.0/lib/python3.8/site-packages/flatten_references_graph/flatten_references_graph.py", line 36, in flatten_references_graph
    return create_list_of_lists_of_strings(pipe(
  File "/nix/store/b6wvrfxrl3cbqi8vdld63mah9d11p067-python3.8-toolz-0.11.1/lib/python3.8/site-packages/toolz/functoolz.py", line 303, in __call__
    return self._partial(*args, **kwargs)
  File "/nix/store/3g9j2hw15zj4xdk45b3hzcy5i27inq57-flatten-references-graph-0.1.0/lib/python3.8/site-packages/flatten_references_graph/pipe.py", line 74, in pipe
    return tlz.pipe(
  File "/nix/store/b6wvrfxrl3cbqi8vdld63mah9d11p067-python3.8-toolz-0.11.1/lib/python3.8/site-packages/toolz/functoolz.py", line 627, in pipe
    data = func(data)
  File "/nix/store/b6wvrfxrl3cbqi8vdld63mah9d11p067-python3.8-toolz-0.11.1/lib/python3.8/site-packages/toolz/functoolz.py", line 303, in __call__
    return self._partial(*args, **kwargs)
  File "/nix/store/3g9j2hw15zj4xdk45b3hzcy5i27inq57-flatten-references-graph-0.1.0/lib/python3.8/site-packages/flatten_references_graph/split_paths.py", line 99, in split_paths
    split_path_indices = list(unnest_iterable(map(
  File "/nix/store/b6wvrfxrl3cbqi8vdld63mah9d11p067-python3.8-toolz-0.11.1/lib/python3.8/site-packages/toolz/functoolz.py", line 303, in __call__
    return self._partial(*args, **kwargs)
  File "/nix/store/3g9j2hw15zj4xdk45b3hzcy5i27inq57-flatten-references-graph-0.1.0/lib/python3.8/site-packages/flatten_references_graph/split_paths.py", line 78, in split_path_spec_to_indices
    vertex = find_vertex_by_name_or_none(graph)(split_path_spec)
  File "/nix/store/b6wvrfxrl3cbqi8vdld63mah9d11p067-python3.8-toolz-0.11.1/lib/python3.8/site-packages/toolz/functoolz.py", line 303, in __call__
    return self._partial(*args, **kwargs)
  File "/nix/store/3g9j2hw15zj4xdk45b3hzcy5i27inq57-flatten-references-graph-0.1.0/lib/python3.8/site-packages/flatten_references_graph/lib.py", line 100, in find_vertex_by_name_or_none
    return graph.vs.find(name)
AttributeError: 'dict' object has no attribute 'vs'

Let's simplify to a working example:
dockerMakeLayers { baseJson = writeText "json" "{}"; contentsList = [ hello figlet ]; pipeline = [ ["split_paths" [ hello ] ]    ]; }

This outputs a set of layers:
[ [hello]
  [libdn2 libunistring glibc]
  [figlet bash]
]

That is as expected; main, common and rest.
If I specify another split_paths in the pipeline, I would expect it to run for each layer and produce the following (with empty layers still in place to show how the result came to be (via split_paths and then a concatMap)
[ [hello] [] []
  [] [glibc] [libdn2 libunistring]
  [figlet bash] [] []
]

Here, split_paths has an effect on both [libdn2 libunistring glibc] and [figlet bash].
Whether my expectation is optimal is debatable. Perhaps it's better to apply it only to components that contain split_paths arguments, because it results in fewer layers. Otoh, splitting all layers has the benefit of commutativity, which could be really important when paths are in hash order (ie nondeterministic for practical purposes).
Could you try to make it possible, at least for the top-level pipeline, to allow operations to work on all components simultaneously? It seems that at least the splitting functions will behave like identities for components that are entirely unrelated to the argument path(s).

splitting the python code into its own repo

I have no preference. Happy to do whatever you (and other maintainers) decide.

I vote to keep it all here.

I copied over the code but haven't touched the original package since I wasn't sure if we should keep it around (within nixpkgs only dockerTools depended on it).

I'd consider the old script an internal implementation detail. If someone used it without dockerTools, they're better off forking it than using a wrapper that will bit rot.

@roberth
apologies for the delay
Regarding:
nix-build --expr 'with import ./. {}; dockerMakeLayers { baseJson = writeText "json" "{}"; contentsList = [ hello figlet cowsay ]; pipeline = [ ["split_paths" [ hello ] ] ["split_paths" [ figlet ] ] ]; }'

I believe what you are trying to achieve could be done with:
pipeline = [
  ["split_paths" [ hello ] ]
  ["flatten"]
  ["map" ["split_paths" [ figlet ] ] ]
]

We could make a version of pipe which would give the same result without user having to add flatten and map manually. Not sure if that should be the default for the top level pipeline. I actually had it working this way initially, but as you notice it can make it difficult to apply operations like popularity_contest to only one specific layer. In some cases the layers where not ordered in the way I'd like them to (which matters if you want to apply limit_layers to the final result). Anyway adding the auto-flattening version of pipe should be straightforward. Hope to be able to do it later this week.

I'm still a bit concerned about the need for this amount of configuration. The capability is probably great for power users who will be able to fine tune their images and it's very useful for experimentation. I just don't want people to have to become power users.
Perhaps I have given up too soon on optimization (in the math sense) when thinking about the split_paths approach. I've mentioned before that it optimizes for minimizing changes between updates of the same image as opposed to popularity_contest which optimizes for cross-image sharing. I didn't make the extra step to formalize that though. There's no reason it can't be done though, so a fully automatic split_paths may still be possible.
Having a good algorithm without configuration should alleviate any DSL issues.
A fully automatic split_paths could split layers one at a time, selecting the optimal split in each iteration. If we find a good definition of optimal we don't need any configuration at all. If it does not work great in some cases, we can let the user adjust the metric instead of making the whole thing manual.
At a high level, for every split we're looking to maximize the expected saved cost of transmission.
maxpath ∈ nodes(graph)  P(change in path) * cost_of_changegraph,layers(path))
where P(change in path) is the probability that path changes between versions of the image
and cost_of_change is the cost of transmitting the layers that need to be transmitted when path changes.
This seems to be an accurate representation of what we're trying to achieve.
To use it, we need to define the two functions. cost_of_change is the most tractable one. The sum of the size of the changed layers is a good approximation. I don't think we need to account for per-layer overhead, partly because I don't expect the algorithm to produce small layers.
cost_of_changegraph,layers(path) = size(intersect(split_atgraph(layers, path), layers))
Here, split_at performs the split according to split_paths and gives all the dependents of _path_ a new identity, so the intersect function can do its job. This is why we need graph as well as layers.
So far, this seems to be accurate. P(change in path) is by far the most unpredictable one. One approximation is to assume that each byte in the closure is equally likely to change. Compared to an equal probability for all paths, this seems reasonable because big paths have more entropy. It will also have the effect of putting the small config files near the root together in the "rest" layer, not wasting layers. It also doesn't favor putting small dependencies in layers. However, because they're deeper down, those will be grouped according to the needs of the bigger paths, as a consequence of how split_paths works.

We could make a version of pipe which would give the same result without user having to add flatten and map manually.

While writing the above, I've assumed the split_paths to apply to all layers, like in the "list transformer" (as in Haskell) pipe you suggest; not just the layer containing path. I'm not sure if we need the commutativity of this variation though, because we apply the iterations of it in a deterministic order that isn't too arbitrary either. Perhaps splitting less per split_paths operation is good because it leaves more layers to be decided by the optimization step instead. I can't judge which is better at this point.

it can make it difficult to apply operations like popularity_contest to only one specific layer. In some cases the layers where not ordered in the way I'd like them to (which matters if you want to apply limit_layers to the final result).

The necessity of this seems to ultimately stem from the desire to use as many layers as possible without having too much configuration. Automatic iterated split_paths is also automatic and can just stop at the configured maximum. I hope we don't need to combine multiple automatic algorithms.
Just a side note, if you could generate a graphviz graph in a passthru attribute, that would be very useful when trying to understand real-world images. Clusters seem like a perfect match for layers.

@roberth

I'm still a bit concerned about the need for this amount of configuration. The capability is probably great for power users who will be able to fine tune their images and it's very useful for experimentation. I just don't want people to have to become power users.

Yes, I believe in most cases uses should be able to select one of predefined algorithms/presets based on their particular needs. Your proposal sounds interesting, however I believe this could be addressed in a separate PR. I think the capabilities implemented in this PR are still useful so it would be great if we could get this merged while we try to figure out the "fully automatic" algorithm which optimises for image rebuilds.

We've just had a call. The plan is to work on an automatic algorithm before working on the DSL and documentation, because those may be affected by this work.
I don't expect many changes to dockerTools in the short term, although Adrian has experienced some merge conflicts during the past month. Last month has been busier than usual; possibly because of the release. If more than usual PRs show up again, I will reconsider this. (And I'll keep this PR in mind when other PRs come in)

sorry, I have been busy with a new project, I might be able to get back to it in about a month or so. Alternatively I could set up a bounty and contribute some $ to it if anyone would be interested in taking this up

@adrian-gierakowski Thanks for the update. It'd be great if anyone could make progress on this.
This pr brings good changes, such as tests and a foundation for building smarter layering strategies. It would even fix a bug, #140908, so I think the priority should be to get this merged. The strategies themselves can be marked as experimental with a message during evaluation, to manage expectations and invite collaboration.
Regarding the bounty, that may be quite helpful. It'd be most effective if you could help with the merge-readiness part. That way the bounty can be used for a more specific task that doesn't require as much background.
I should probably also plug https://opencollective.com/nix-deployments which is a new fund for maintenance and development of Nix-based deployment tech. You can donate for specific projects, such as the NixOps 2 release, or "Anything around Containers", where the ideas are open for discussion.
That said, a bounty works differently and might better fit your needs. Specifically, it is more specifically scoped and not paid ahead. Do whatever you're comfortable with; it's highly appreciated regardless.

@roberth I could probably spend 1-2 hours on this next week to complete any work you think necessary to get this merged. This would obviously not be enough to implement the fully automatic strategy. I could probably work on the latter in about a month or so, and submit it as a separate PR, together with documentation etc. In the meantime I’ll look into setting up a bounty in case someone else is keen to do it earlier.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/2

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/2

@rihardsk thanks for the feedback! I defiantly intend to push the ball forward on this PR, however my work situation changed soon after I submitted this PR and I haven't had much spare time atm to continue working on it.

I know that this PR might be obsolete and there now seem to be some other PRs tackling the same problem

I'm not aware of any other PRs which provide a better solution to the problem this PR is trying to address so I hope we can get this PR merged eventually.

EDIT: it seems that the error is caused by "popularity_contest" strategy having a "layer_limit" set that is higher than the actual number of dependencies in the subgraph.

Nice, I'll try to replicate this in tests and see if can catch it early to provide better feedback to the user.
@roberth you wrote above that:

It'd be most effective if you could help with the merge-readiness part.

what would have to be done (apart from rebasing and resolving conflicts) for this PR to be considered merge ready?

@adrian-gierakowski i'm definitely enjoying the flexibility your solution provides. I've only had a quick glance on those other docker image layering PRs so i wasn't sure if anything there attempts to provide a general solution.
One slight annoyance I've had with this PR is that the layering strategy that I've created has ended up being deeply nested, here's a sample from my project:
let
  # Helper functions for layering strategy creation
  popularityHelper = overWhat: layerLimit: [
    "over"
    overWhat
    [
      "pipe"
      [
        ["popularity_contest"]
        ["reverse"]
        ["limit_layers" layerLimit]
      ]
    ]
  ];
  # Note, there seems to be a bug, where the layer creation code fails with
  #   TypeError: reduce() of empty sequence with no initial value
  # that seems to be caused by the popularity_contest strategy having a layerLimit
  # that is higher than the actual number of dependencies in the dependency
  # subgraph. To work around this, try decreasing the layerLimitForRest argument.
  splitFromMain = splitWhat: layerLimitForRest:
    let mainProcessingStrategy =
      if layerLimitForRest > 0
        then
          [
            ["subcomponent_in" splitWhat]
            (popularityHelper "rest" layerLimitForRest)
          ]
        else
          [
            ["subcomponent_in" splitWhat]
          ]
      ;
    in [
      "over"
      "main"
      [
        "pipe"
        mainProcessingStrategy
    ]
  ];

  layeringPipeline = [
    ["subcomponent_out" [models]]
    [
      "over"
      "rest"
      [
        "pipe"
        [
          ["split_paths" [cudatoolkit]]
          (splitFromMain [cudatoolkit] 10)
          (popularityHelper "common" 30)
          [
            "over"
            "rest"
            [
              "pipe"
              [
                ["split_paths" [gateway]]
                (splitFromMain [gateway] 1)
                # (popularityHelper "common" 30)  # cannot use this because of TypeError: reduce() of empty sequence with no initial value
                [
                  "over"
                  "rest"
                  [
                    "pipe"
                    [
                      ["split_paths" [segmenter]]
                      (splitFromMain [segmenter] 1)
                      # (popularityHelper "common" 30)  # cannot use this because of KeyError: 'common'
                      [
                        "over"
                        "rest"
                        [
                          "pipe"
                          [
                            ["split_paths" [translator]]
                            (splitFromMain [translator] 4)
                            # (popularityHelper "common" 30)
                            [
                              "over"
                              "rest"
                              [
                                "pipe"
                                [
                                  ["split_paths" [marian-server]]
                                  (splitFromMain [marian-server] 2)
                                  # (popularityHelper "common" 30)
                                  [
                                    "over"
                                    "rest"
                                    [
                                      "pipe"
                                      [
                                        ["split_paths" [pkgs.python38Packages.supervisor]]
                                        # (splitFromMain [pkgs.python38Packages.supervisor] 10)
                                        # (popularityHelper "common" 30)
                                        [
                                          "over"
                                          "rest"
                                          [
                                            "pipe"
                                            [
                                              ["subcomponent_out" [gateway_conf translator_conf supervisord_conf]]
                                              # (popularityHelper "rest" 5)
                                            ]
                                          ]
                                        ]
                                      ]
                                    ]
                                  ]
                                ]
                              ]
                            ]
                          ]
                        ]
                      ]
                    ]
                  ]
                ]
              ]
            ]
          ]
        ]
      ]
    ]
  ];
in {} # ...

Maybe I'm using it wrong, but it would be cool if I could avoid all the nesting here and instead have something that looks more linear and somehow automatically pipes the "rest" part to the next step (avoiding "over").

I think what this needs is

rebase
a test case reproducing rihardsk's problem, assuming it also affects the default pipeline
some expectation management regarding the dsl. It's in development and is very likely to change. Worst case, users can delete their pipelines, so that's not terrible, but it is when they don't expect to have to.
merge

@rihardsk it seems that you're most knowledgeable about the issue. Do you think you can write a test case?

@SuperSandro2000 which tool did you use to format pkgs/build-support/docker/default.nix in this commit? I hope to make merge conflicts more manageable by running the same tool on the branch of this PR before doing rebasing it on top of master. Thanks!

@roberth
finally managed to find some time to take care of this

I think what this needs is

rebase

done, fixed a couple of bugs along the way which came up after running nix-build nixos/tests/docker-tools.nix

a test case reproducing rihardsk's problem, assuming it also affects the default pipeline

this was fixed by adding initialiser to following reduce call:
https://github.com/NixOS/nixpkgs/blob/c2f1a7d9c8e6efb033bd0cbd0c646a0ba2afde55/pkgs/build-support/flatten-references-graph/src/flatten_references_graph/lib.py#L263-L264
test case exercising this code path added here

some expectation management regarding the dsl. It's in development and is very likely to change. Worst case, users can delete their pipelines, so that's not terrible, but it is when they don't expect to have to.

is comment next the function argument enough?
https://github.com/NixOS/nixpkgs/blob/c2f1a7d9c8e6efb033bd0cbd0c646a0ba2afde55/pkgs/build-support/docker/default.nix#L840-L841

merge

your call :)

@utdemir

Instead of using heterogeneous collections and objects we don't know their type of and having to write generic functions that do type inspection like as_list or flatten, we can try to get objects to have a certain shape.

Could you please provide some concrete examples? as_list's purpose is to coerce a value into a list, and where it's used the shape that is needed is a list. I am not sure how "shape" I should replace a list with if the list is the correct data structure for the task at hand

managed to find some time to take care of this

Awesome.
I hope to merge this soon, so that we have a foundation for further improvements. There's a slight risk that something might break, but this is a leap forward because it replaces ad hoc jq queries by readable and testable code.
Reviews by others are very much appreciated. Review checklist

[ ]  Nix code (solve IFD issue mentioned in review comment, otherwise seems to look good)
[ ]  Python idioms/packaging: help wanted
[ ]  Review the graph code (nice to have because it's experimental anyway. Can be deprioritized IMO)

@roberth I have now fixed the IFD. Not sure about to do about the other 2 points. I still need to fix the merge conflicts but it would be great to get some indication if any other major changes would be required before the PR can be merged. I don't really have a lot of time to do a major refactor so if that is required (motivated purely by coding style) then I will probably close this PR. Thanks!

@roberth I've resolved merge conflicts locally but unfortunately it seems like nix-build nixos/tests/docker-tools.nix is currently failing on master
they pass for ca072c08a254 and 38c1400f67f6af73821d5be82f0ddab548e707e2 also seems to be file (not finished running yet, but the build error below has not manifested)
tar: ./proc/sys/vm/drop_caches: Cannot open: Permission denied
tar: ./proc/sys/vm/mmap_rnd_bits: Cannot open: Permission denied
tar: ./proc/sys/vm/mmap_rnd_compat_bits: Cannot open: Permission denied
tar: ./proc/sys/vm/stat_refresh: Cannot open: Permission denied
tar: ./proc/sysrq-trigger: Cannot open: Permission denied
tar: ./proc/timer_list: Cannot open: Permission denied
tar: ./proc/tty/driver: Cannot open: Permission denied
tar: ./proc/vmallocinfo: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors
building '/nix/store/b92hkdcgf7wqclv4xmp56gn80mlsiaq9-runtime-deps.drv'...
[    3.200305] reboot: Power down
error: builder for '/nix/store/avh0ka80ri60cf7rvcx72z3zzdjwfj3z-etc-customisation-layer.drv' failed with exit code 2;
       last 10 log lines:
       > tar: ./proc/sys/vm/compact_memory: Cannot open: Permission denied
       > tar: ./proc/sys/vm/drop_caches: Cannot open: Permission denied
       > tar: ./proc/sys/vm/mmap_rnd_bits: Cannot open: Permission denied
       > tar: ./proc/sys/vm/mmap_rnd_compat_bits: Cannot open: Permission denied
       > tar: ./proc/sys/vm/stat_refresh: Cannot open: Permission denied
       > tar: ./proc/sysrq-trigger: Cannot open: Permission denied
       > tar: ./proc/timer_list: Cannot open: Permission denied
       > tar: ./proc/tty/driver: Cannot open: Permission denied
       > tar: ./proc/vmallocinfo: Cannot open: Permission denied
       > tar: Exiting with failure status due to previous errors
       For full logs, run 'nix log /nix/store/avh0ka80ri60cf7rvcx72z3zzdjwfj3z-etc-customisation-layer.drv'.
error: builder for '/nix/store/j0rijnn2diczp3v8y09qx9v9paalwv08-image-via-fake-chroot-customisation-layer.drv' failed with exit code 2;
       last 10 log lines:
       > tar: ./proc/sys/vm/compact_memory: Cannot open: Permission denied
       > tar: ./proc/sys/vm/drop_caches: Cannot open: Permission denied
       > tar: ./proc/sys/vm/mmap_rnd_bits: Cannot open: Permission denied
       > tar: ./proc/sys/vm/mmap_rnd_compat_bits: Cannot open: Permission denied
       > tar: ./proc/sys/vm/stat_refresh: Cannot open: Permission denied
       > tar: ./proc/sysrq-trigger: Cannot open: Permission denied
       > tar: ./proc/timer_list: Cannot open: Permission denied
       > tar: ./proc/tty/driver: Cannot open: Permission denied
       > tar: ./proc/vmallocinfo: Cannot open: Permission denied
       > tar: Exiting with failure status due to previous errors
       For full logs, run 'nix log /nix/store/j0rijnn2diczp3v8y09qx9v9paalwv08-image-via-fake-chroot-customisation-layer.drv'.
error: 1 dependencies of derivation '/nix/store/26jj8z6mixsy9g8jwsb4n0napccj9219-etc-conf.json.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8yv8xk31f0lkjc4xrlr55sah40rwn9zc-image-via-fake-chroot-conf.json.drv' failed to build
error: 1 dependencies of derivation '/nix/store/gnwpgasm59qy8ril01ippnd14ncsz8ms-stream-etc.drv' failed to build
error: 1 dependencies of derivation '/nix/store/z39kshy20znn5ihxslwlwcqyk5ca13fw-nixos-test-driver-docker-tools.drv' failed to build
error: 1 dependencies of derivation '/nix/store/an4f5jvrkmp71xmpdr5xvaq1h5kzlawi-vm-test-run-docker-tools.drv' failed to build

I checked

reverting ca072c08a2543b4a7a107ebbfbb03ab23426f6ed doesn't help as it brings back the error it fixed

adding --exclude=proc to tar invocation seems to fix it:
https://github.com/NixOS/nixpkgs/blob/8efd5d1e283604f75a808a20e6cde0ef313d07d4/pkgs/build-support/docker/default.nix#L921-L925

conflicts resolved and tests are passing

man I really need this. Is this in a usable state?
also @adrian-gierakowski
the main thing I want is just to have my root derivation always automatically be put in its own layer. Is there a simple way to specify that with your changes?

Yes it’s usable.
You can separate any derivation into its own layer by using split_paths or subcomponent_out followed by subcomponent_in like it’s done in the example above for ankisyncd (see the linked repo for resulting layers).

This really deserves a merge!
I've been using it for over half a year with no issues.

This really deserves a merge!
I've been using it for over half a year with no issues.

Thanks! I’d love to have it merged it as well. @roberth what do you think?

It's become difficult for me to commit to maintaining much new code at this time, but you've shown responsibility, so if you'd be interested in co-maintaining dockerTools, that would make it feasible for me (or us) to accept these changes sooner. Would you be interested in doing that?

It's become difficult for me to commit to maintaining much new code at this time, but you've shown responsibility, so if you'd be interested in co-maintaining dockerTools, that would make it feasible for me (or us) to accept these changes sooner. Would you be interested in doing that?

Absolutely!

I've squashed and rebased the changes onto master, also fixed some typos in https://github.com/elohmeier/nixpkgs/commit/806147b85d7fe316b2aa8ee0b12d99147ccbc9f6. The layeringPipeline functionality works really well, thanks @adrian-gierakowski. Would be great to have this in nixpkgs at some point.

I've squashed and rebased the changes onto master, also fixed some typos in elohmeier@806147b. The layeringPipeline functionality works really well, thanks @adrian-gierakowski. Would be great to have this in nixpkgs at some point.

thanks @elohmeier, that's awesom
just reviewed the diff and reset my branch onto yours

@roberth I've now fixed the pkgs/by-name CI failure (see https://github.com/NixOS/nixpkgs/pull/122608/commits/138a6730a316cf899aa47a20d1b06d58d0fc2c36) so I believe this should be ready to merge (can do a final squash if necessary). Not sure about the solution in the last commits for providing dev shell for package development but it's working for me (not as easy as running nix-shell in package dir, but I don't think it's possible due to import restriction)

@roberth sorry to bother you but would be great not to let this PR rot again. Thanks!

Yes please! Like I said, I've been using this in production for ~6 months

can we please stop just letting this branch languish?

@teto @tomberek I see you merged 2 most recent PRs affecting dockerTools. Since @roberth seems busy, would you be able to have a look? Thanks!

It really sucks to be ignored like this.

sry but I have limited interest in docker (the current infra covers my cases) and this clearly exceeds my skill.
On a quick glance, this PR (+7,235 −65) seems to serve niche scenarii and I would think it's best if it lives outside nixpkgs. Even for you, so you remain in control of this complex code.

@teto thank you for your reply!

sry but I have limited interest in docker (the current infra covers my cases) and this clearly exceeds my skill. On a quick glance, this PR (+7,235 −65) seems to serve niche scenarii and I would think it's best if it lives outside nixpkgs. Even for you, so you remain in control of this complex code.

I appreciate that what the PR brings might not be of interest to you, or many casual users of containers, but for anyone deploying containers in production, with packages which have large dependency trees (often the case in dynamic languages like JS, python or ruby), the way packages are organised into layers by streamLayeredImage is a non-starter: I would not be able to convince anyone why has learned how to craft their Dockerfile's to minimise re-builds and speed up deploys to switch to building containers with nixpkgs' dockerTools. There is clearly a demand for improved layering strategies as evidenced by the tickets linked in the PR description, comments by others in the PR who are already using this solutions, or existence of projects like https://github.com/nlewo/nix2container.
Besides, as mentioned by @roberth, the PR brings other desirable chanegs:

This pr brings good changes, such as tests and a foundation for building smarter layering strategies. It would even fix a bug, https://github.com/NixOS/nixpkgs/issues/140908, so I think the priority should be to get this merged.

Regarding size of the PR: at least half of it are tests (which should aid with maintenance), and a big chunk is a copy of existing references-by-popularity package, which, as suggested by @roberth, would be removed (see end of this comment)
Regarding complexity, and maintenance, I would argue that instead asking me to take my code elswere, it would be more beneficial for the community to have me added as a co-maintainer of dockerTools (as suggested by @roberth) which would increase teams capacity to review other related PRs (by now I know the code inside out).
If I were to move this code to another repo, I'd still need to make some changes to dockerTools in order to allow me to inject layering code, since I wouldn't want to duplicate any of the functionality already present in dockerTools.

as evidence that this PR is addressing a real need of dockerTools users, I've collected 20 user-names of people who reacted with :+1: or :tada: on the initial post and recent comments asking to get this merged (here and here)
arijoon
avakhrenev
bhipple
cdepillabout
colonelpanic8
countablecloud
elohmeier
fullykubed
jkachmar
lbpdt
loewymalkov
misuzu
phaer
rimmington
roberth
Smona
the-sun-will-rise-tomorrow
tomberek
yajo
Zaunei

I'm looking and would want to merge this, but it's more than I can quickly look at. I do want something like this, but I can't claim to understand the PR yet. @adrian-gierakowski would it be possible to do a synchronous review this weekend?

I'm looking and would want to merge this, but it's more than I can quickly look at. I do want something like this, but I can't claim to understand the PR yet. @adrian-gierakowski would it be possible to do a synchronous review this weekend?

@tomberek sure, let me know when would work best for you. I'm in GMT. Thanks!

@tomberek just in case you’re able to find some time for it this weekend: I’m available 6:30-9pm tonight and tomorrow. Thanks

Just rebased on master, fixing merge conflicts, applying nixfmt to new nix files, and squashing all commits into one.
I've backed up previous state of this branch on at https://github.com/adrian-gierakowski/nixpkgs/tree/ag/docker-customisable-layering-strategy-before-rebase-and-nixfmt-2024-11-09

also ran nix-build nixos/tests/docker-tools.nix locally and tests passed  ✅

@roberth @tomberek Please merge this or an articulate a reason why it can't be merged. It has been more than 3 years. I have been using this to build all the containers at my small business for almost as year now with no issues!

@adrian-gierakowski @colonelpanic8 In principle I'm in favor of merging this as soon as 24.11 is branched off, which is planned for Thursday.
Originally I wanted to do more for this PR in terms of review and perhaps improving it further, but that has turned out unrealistic, so I am content that this is at the very least an improvement in terms of maintainability and ability to evolve this solution.
Thank you @adrian-gierakowski for implementing this (and persevering, oh my), @colonelpanic8 for testing, and @tomberek for offering to help with review.

Thanks @roberth!
Would you be willing to nominate me, or support my self-nomination to be added as docketTools maintainer and commiter here? I’d be happy to iterate on this to make further improvements and eventually stabilise the interface. Thanks!

I will, but if anyone would only look at the numbers, they may disagree. I can trust you because we've talked directly and you've shown professionalism etc, but that won't apply to others, so it's probably best to wait a bit. Until then, ping me so I can check a couple of things and merge for you. These won't necessarily be in-depth reviews, or very large diffs, so that's two reasons why it will be much much quicker, especially compared to this one.

@adrian-gierakowski i messaged you in Matrix. Any other good way to get in touch?

@tomberek I don't see your message, but just sent you one on matrix as well (found you on #users:nixos.org)

addressed all recent comments
ran tests locally and they still pass

How has this gone so far? Any issues?

As someone who has been waiting for this functionality for a long time, I just tried this out -- The pipeline syntax could definitely use more documentation and examples but now that I've wrapped my head around it, it works very well.
A few snippets for common use cases:
Flatten python3 and all its dependencies into a single layer as a common base layer, handle other paths like previously
[
    ["subcomponent_out" [pkgs.python3]]
    ["over" "rest" ["pipe" [
        ["popularity_contest"]
        ["limit_layers" 100]
    ]]]
]

Force the often-changing custom application code into its own layer
[
    ["subcomponent_in" [my_app]]
    ["over" "rest" ["pipe" [
        ["popularity_contest"]
        ["limit_layers" 100]
    ]]]
]

Combining the two
[
    ["subcomponent_in" [my_app]]
    ["over" "rest" ["pipe" [
        ["subcomponent_out" [pkgs.python3]]
        ["over" "rest" ["pipe" [
            ["popularity_contest"]
            ["limit_layers" 100]
        ]]]
    ]]]
]

The ["over" "rest" ["pipe" [ and resulting rightward drift is a bit awkward. But in any case, it does what I need it to do, and thanks for finally getting this landed!

I was already using it on an older forked version of nixpkgs, but I have now moved to using the version that is in unstable and I'm (pretty unsurprisingly) having no issues!


Unable to build scheme "SpeechToTextV1" using Carthage

The command I'm using is:
carthage build swift-sdk --platform iOS --no-use-binaries --cache-builds
The Swift version I'm using is 5.1
*** Building scheme "SpeechToTextV1" in WatsonDeveloperCloud.xcodeproj
Build Failed
Task failed with exit code 1:
/usr/bin/xcrun dwarfdump --uuid /Users/kevin/Library/Caches/org.carthage.CarthageKit/DerivedData/11.3.1_11C504/swift-sdk/v3.3.0/Build/Intermediates.noindex/ArchiveIntermediates/SpeechToTextV1/IntermediateBuildFilesPath/UninstalledProducts/iphoneos/SpeechToTextV1.framework/SpeechToTextV1
This usually indicates that project itself failed to compile. Please check the xcodebuild log for more details: /var/folders/mq/c7bgc49s5f5dwhsyl56rt0900000gn/T/carthage-xcodebuild.KKgTMt.log
carthage-xcodebuild.KKgTMt.log

@xezero thanks for filing this. I'll take a look at trying to replicate this.

Thank you! It also seems like some of the other schemas don’t finish building with Carthage either, like TextToSpeech.

@xezero what version of Carthage are you using?

@repjarms 0.34.0

So it looks like the x86 framework is what’s failing. The frameworks for arm architectures build fine.

@repjarms Would it be possible to add SPM support for SpeechToTextV1 (the Package.swift seems to be missing a few targets like this one)?
That would indirectly resolve this issue for me as well if I can simply compile via SPM :)

Is this still an issue?


[cgroups2] Introduces API to read the peak memory usage.

The cgroups2 control file memory.peak contains the maximum recorded memory usage for a cgroup and its descendants since creation.
Here we introduce cgroups2::memory::peak_usage to read this maximum.

memory.peak doesn’t always exist. It was introduced in May 2022 and was backported into RHEL 9.3 but our CentOS 9 distribution doesn’t use it.
src: https://stackoverflow.com/questions/66291245/is-it-possible-to-implement-max-usage-in-bytes-for-cgroup-v2

Was intended to be used when there is an OOM event to report the maximum memory usage. Opted to use the value of memory.current at the time when the OOM takes place instead, because memory.peak does not exist on all hosts that support cgroups v2.


Some improvement ideas

A quick list of improvements that could be nice:

[ ] Set all images to the same size, preferably to $COLUMNS if available.
[ ] Link to gify if we like a particular one. I don't know if it's possible.
[ ] Display npm summary when install has finished.
[ ] Display some sort of progress interleaved with gifs in some way.

I appreciate your time & effort, but these improvements go way beyond the scope of this project, which is kind of just a joke tool :)

I understand :)
Thing is, I'm installing a sh**load of stuff lately. So for me it's not a toy, but a true relief! :laughing:
More seriously I find the idea of sindre awesome, and your project too. It deserves to be a first class toy.
I think normalizing image sizes and displaying the end summary would make this perfect.
Would you be interested in some PRs?

Absolutely, PRs are always welcome :)


[WIP] Refactor the application to use React.useReducer

To Do

[x] Create Player interface.
[x] Create PlayerList interface.
[x] Create ReducerActionTypes enum.
[ ] Create state object literal.
[ ] Create useReducerAndDispatchMethods hook.

Player interface
interface Player {
  id: string;
  name: string;
  color: string;
}

PlayerList interface
interface PlayerList {
  id: string;
  title: string;
  players: Player[];
}

ReducerActionTypes enum
This enum references all of the available action types for the reducer.
enum ReducerActionTypes {
  UPDATE_SORTABLE_PLAYER_LIST,
  UPDATE_GAME_SCORE,
}

state structure
Here is the proposed data structure to the application's state:
const state = {
  /**
   * References the sortable player lists.
   */
  playerLists: [
    {
      id: "player-list-innocent",
      title: "Innocent",
      players: [],
    },
    {
      id: "player-list-unknown",
      title: "Unknown",
      players: [
        // ...
        { id: 1, color: "red", name: "" },
      ],
    },
  ],

  /**
   * References the current game scores.
   */
  gameScores: {
    innocent: { wins: 0, losses: 0 },
    imposter: { wins: 0, losses: 0 },
  },
};

useReducerAndDispatchMethods hook
Each dispatch action will adhere to the following schema:
const createNewDispatchMethod = () => ({
  type: ReducerActionTypes.YOUR_DISPATCH_TYPE, // Must be a unique identifier
  payload: ..., // Can be any data type
});

import { useReducer } from "react";

const initialState = { ... };
const createReducer = (state, action) => {
  switch (action.type) {
    case ReducerActionTypes.UPDATE_SORTABLE_PLAYER_LIST:
      // Do some work and return the new state

    default:
      return state;
  }
};

const updatePlayerList = (newPlayerList, playerListId) => ({
  type: ReducerActionTypes.UPDATE_SORTABLE_PLAYER_LIST, // Here I'm using an `enum`
  payload: {
    id: playerListId,
    players: newPlayerList,
  },
});

const useReducerAndDispatchMethods = () => {
  const [state, dispatch] = useReducer(createReducer, initialState);

  return {
    state,
    dispatchMethods: {
      updatePlayerList: (...args) => dispatch(updatePlayerList(...args)),
    },
  };
};

Kedyn and I talked about useReducer. Depending on the direction we're going with the app, seems like there's some talk to implement Redux

@Kedyn, are you down to implement Redux in the application?

@Kedyn is implementing redux


DefaultInput

Added multiple components, namely: "DefaultInput", "DefaultSelect", "DefaultCheckbox", to have a streamlined way of creating input fields

IMPORTANT!! Before merging this pull request, please first check out #52


Feat: allow marking audits as [non-]blocking at use site

Until now, audits could be marked as blocking (default) or non-blocking only at their definition. Some users reasonably found this counter-intuitive, since the audit's logic and whether it blocks execution or not are orthogonal concepts.
This PR adds support for marking an audit as blocking or non-blocking at its use-site, i.e. in a model that references it. The syntax to do that is demonstrated below:
MODEL (
  name model_name,
  kind FULL,
  audits (
    some_audit(blocking := false)
  )
);

...

FYI @plaflamme

Can we please update documentation

Can we please update documentation

Done, thanks for pointing it out 👍


Mutate (and likely steps too) code isn't shown in Hamilton UI

Current behavior
Running mutate decorator (and likely step/pipe) with the Hamilton Tracker shows the correct DAG structure, but no code is attached.
Screenshots

Steps to replicate behavior
from hamilton.function_modifiers import mutate
import pandas as pd

def transformed_data(raw_data: pd.DataFrame) -> pd.DataFrame:
    return ... # do your regular stuff here

# turns initial_data into initial_data + 1
@mutate(transformed_data)
def _normalize_columns(df: pd.DataFrame) -> pd.DataFrame:
    for column in df.columns:
        df[column] = (df[column]-df[column].min())/(df[column].max() - df[column].min())
    return df

@mutate(transformed_data, outlier_threshold=10)
def _remove_outliers(df: pd.DataFrame, outlier_threshold: float) -> pd.DataFrame:
    return df[df < outlier_threshold]

driver:
from hamilton_sdk import adapters
from hamilton import driver

tracker = adapters.HamiltonTracker(
   project_id=...,  # modify this as needed
   username="...",
   dag_name="mutate_example",
   tags={"environment": "DEV", "team": "MY_TEAM", "version": "mutate"},

)
dr = (
  driver.Builder()
    .with_config({})
    .with_modules(mutate_example)
    .with_adapters(tracker)
    .build()
)

Library & System Information
Latest python, SDK, & hamilton UI versions
Expected behavior
That the code shows up.
Additional context
Guess - we're not attaching the source code appropriately or referencing it correctly for the UI to show it.

The problem here is the originating functions... @mutate isn't attached to the function and we don't collect the references. We may want to:

Add to originating functions
Create a auxiliary_functions variable to store the attached ones

I just checked, pipe_output / step has the same behavior (not surprised)!

The problem here is the originating functions... @mutate isn't attached to the function and we don't collect the references. We may want to:

Add to originating functions
Create a auxiliary_functions variable to store the attached ones

@elijahbenizzy It would make sense to store the reference in Applicable. Then using step / apply_to / mutate (which has apply_to under the hood for a function pointer) signals that the user is manually adding nodes via pipe_* instead of Hamilton default behaviour of connecting functions via names / types.

After some thought I agree with 2., because we can use the same function in multiple "pipes" or "mutates", but it makes sense to really show the code only once.

The problem here is the originating functions... @mutate isn't attached to the function and we don't collect the references. We may want to:

Add to originating functions
Create a auxiliary_functions variable to store the attached ones

After some thought I agree with 2., because we can use the same function in multiple "pipes" or "mutates", but it makes sense to really show the code only once.

The added benefit of this is that we can have some notion of "helper" functions -- store more than just the specific functions that we use. This could enable us to crawl the modules for functions using static analysis, etc... To make the UI more valuable.


Windows 11 Hotkeys did not recognized

Hello,
i don't know what i'm doing wrong, but the Hotkeys sometimes did not work since i use Windows 11.
sometimes they work, and after a restart, they don't.
i have to change the hotkey to an other binding and then, sometimes, they will work again.

I've been using the tool (personally) with W11 for months now, no issues. Before that, since W11 was development release, people were using it without any problem on the hotkeys front.
Would you mind telling me if you are installing or using the portable version, also which hotkeys are you using?

i use AltGr + \ as my hotkey. thats sometimes does't work anymore.
AltGr + F10 is working. as far as i can tell.
i use Autohotkey with a a different keyboardlayout: https://eurkey.steffen.bruentjen.eu/?lang=de
There is on the "" key the "¬" symbol but the settings window show it as an other hotkey:

maybe thats the Problem?

That would be the problem, AltGr is not a key per se but a weird combination of keys meaning: "left Control plus right Alt", sometimes "left Control plus left Alt" and even just Alt (but that one also depends on the application).
It will prove almost impossible to add it into this UI. Not just because space constrains, but because I'll need a keyboard that has the key (I don't own one), and use all the different layouts that make sense of the key (as not all of them work the same). As I recall in a Spanish ISO keyboard I had, it was just Alt if I used European layout but with Latin layout it was Ctrl+Alt. Lastly, not all applications respond equally to the key, so it will still be impossible to get it 100% working.
On top of that, you are combining AltGr with the Numpad keys (that have a different meaning depending on if the NumLock is toggled or not).
My advice would be to not use that key, given how erratic its behavior is. If still want to use it, try manually editing the configuration file:
[HOTKEYS]
default  = <^>!\
username = <^>!Numpad9
password = <^>!Numpad8
totp     = <^>!Numpad7

Links to the key names and the meaning of the symbols is inside the configuratin file.
Hope that helps!

Nice, thank you!
The Config file was the correct hint!
While i  experimenting with different Hotkeys, i rebind F13 - F24 to a second Layer on my Keyboard and configured this keys in the config file.
The GUI didn't recognized this funktion keys but thats okay for me.

Glad you got not just your issue solved but also improved upon.
The level of customization one achieves with AHK is so underwhelming... thus, a one-fits-all solution is never achievable, one can only think of so many scenarios :P


@types/mocha has broken compilation

I am now getting these compile errors when compiling my mocha-typescript tests
../node_modules/@types/mocha/index.d.ts(456,11): error TS2420: Class 'Suite' incorrectly implements interface 'ISuite'. Type 'Suite' provides no match for the signature '(target: TFunction): void | TFunction'.
../node_modules/@types/mocha/index.d.ts(499,11): error TS2420: Class 'Test' incorrectly implements interface 'ITest'. Type 'Test' provides no match for the signature '(target: Object, propertyKey: string | symbol, descriptor: TypedPropertyDescriptor): void | TypedPropertyDescriptor'.

@silkentrance I don't think so.
I will release a patch later today or during the weekend.
Btw, I added you (I think) as contributor on npm so you should be able to push releases to npm.

@pana-cc I will have to look into that thing with npm push/publish as I haven't done that before as a contributor.


Incorrect type checking callback

/* @flow */

type FuncType = () => void;

function foo(callback: FuncType): void {
  callback('asadsa');
}

foo(() => {
});

resulting message: No errors!
https://flowtype.org/try/#0PQKgBAAgZgNg9gdzCYAoVAXAngBwKZgBiArgHYDGAKrgQLxgAUAlGLQHxgBucAlgCYBudFDLkMPOKTBQ4cBuQCGMGACMF5ANYAuIqOr4mO7vzABvVGDCLlazQwDkCgM4K+L+0yEBfYbIbNWDnMvTyA

It's by design https://github.com/facebook/flow/issues/2155

it is weak :(

Closing to keep discussion in the existing issue.


vocab file not found

Hi, i've cloned tensorflow and your repos and when i try to run this, i get this error. Apparently the word_counts.txt file doesn't has problems, but don't know what's going on.
(tensorflow) MacBook-Pro-de-Carlos-2:im2txt Carlos$ sudo bazel-bin/im2txt/run_inference \ --checkpoint_path=${CHECKPOINT_PATH} \ --vocab_file=word_counts.txt \ --input_files=${IMAGE_FILE}
INFO:tensorflow:Building model.
CRITICAL:tensorflow:Vocab file  not found.
INFO:tensorflow:Initializing vocabulary from file:
Traceback (most recent call last):
  File "/Users/Carlos/code/tensorflow/models/research/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module>
    tf.app.run()
  File "/Users/Carlos/code/tensorflow/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "/Users/Carlos/code/tensorflow/models/research/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 55, in main
    vocab = vocabulary.Vocabulary(FLAGS.vocab_file)
  File "/Users/Carlos/code/tensorflow/models/research/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/vocabulary.py", line 48, in __init__
    reverse_vocab = list(f.readlines())
  File "/Users/Carlos/code/tensorflow/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 183, in readlines
    self._preread_check()
  File "/Users/Carlos/code/tensorflow/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 80, in _preread_check
    compat.as_bytes(self.__name), 1024 * 512, status)
  File "/Users/Carlos/code/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: ; No such file or directory

I gave permissions to word_counts.txt file and now i'm getting the problem described here, but still don't know how to make it run:
https://github.com/KranthiGV/Pretrained-Show-and-Tell-model/issues/8


[FEATURE] adding MPS GRUCell for efficiency

Describe the bug
This is related to the #1499 , the GRUCell is not implemented and the GRU version is also not optimized on MPS compare to pytorch and tensorflow variant.

I don't think this is related to #1499 but we definitely need to speed up our RNN implementations.


Option to change sort order of entries

I'd like to be able to sort entries most-recent first, but I can imagine there may be other preferred orders, so an option to allow sorting would be great!
Thanks for Upmin Admin!

It's dead Jim.


Seedcracker 1.18.2 issue

I donwload seedcrackeX-2.12 for 1.18.2 but I wonder why everytime I join server or I create new world it keep Saving Wold then
crashing

have the same problem

Same bro.

@19MisterX98 pls help

See #102

i tried that but i could not get it to work. i have the same error as in that one but the fix did not work


4.2.3 applying restrictive permissions to non *.log files?

Question
Should restrictive permissions (g-w,o-rwx) be applied to non "*.log" files, such as utmp, wtmp and btmp, and lastlog within rule 4.2.3?
Currently rule 4.2.3 is using pattern "*.log". Do you anticipate any issue with applying those permissions to them as well? Maybe worth mentioning, those files will not belong to the root group.
Environment:

Ansible Version: 2.13.7
Host Python Version: /
Ansible Server Python Version: 3.8.10
Additional Details:

hi @boris-stojnev
Thank you for raising this issue, This has now been incorporated and released.
I will close this issue, please feel free to reopen or raise an new issue based on the new release.
thanks
uk-bolly


Travis builds

This builds binaries and posts them to CloudFiles.
For instance, here's the binary for this branch.
The encrypted secret will need to be changed for the main repo too.

You can merge this whenever it's ready.


feat(designer): Make Designer compilable for other platforms than web

At the moment the Designer can only run on the web platform because of the html and js libraries used for the test platform. If we add alternative implementations for this, the Designer could also be run with native dart on Windows/Linux/Mac systems. I found two options that could be used for a native dart implementation of the test screen to embedd the App.

The webview_flutter package could help to embed the App website. However, we probably lose instant preview feature, because there is probably no communication possible between the Designer and the webview.

We might be able to use MaterialApp() inside the Designer to embed the App. This will probably also need changes in the App to support communication between the two apps. We need to test whether and how big the impact on performance is.

https://docs.flutter.dev/platform-integration/web/embedding-flutter-web


BridgeCrew compliance updates

Potential problem when deleting a deployed bundle with CMK:
│ Error: cannot purge key "localdevcosmosmongo0015" because vault "Vault: (Name \"localdevcosmosmongo0015\" / Resource Group \"local-dev-cosmosmongo-0015\")" has purge protection enabled

Continuing testing.


Fireman Burns Meme

Input
{
  "template_id": "fireman-burns-meme",
  "lady_says": "I hate this project.",
  "font_size": 30
}

Template

Please wait, we've just started cooking your meme 👨🍳. *Click the 'Actions' tab to see the kitchen 🍳 *

Wow such an interesting project.

Wow such an interesting project.


Need to update Docker Hub page for 0.7.5

https://hub.docker.com/r/cribl/scope
The Overview tab contains 4 references to version 0.7.1.
The first one, being an example, is arguably okay, but all the rest should definitely be updated to 0.7.5 (and maybe the example too).
There's also a typo where it says:
`:0.7.1` is the official tag for this release.

We should delete the leading colon.
The Tags tab will of course need a 0.7.5 tag added when we release, but that goes without saying.

The leading colon is now gone.


✨ Mobile - User settings

Cc: @matt-goldman @tkapa

Hi team
Pain

The new V3 flyout design contains a Settings button that we currently have no use for so is hidden.
Suggested Solution

We should consider the types of settings we should allow the user to modify and implement a settings page where this can be managed.

Figure: Mockup by Ken
Tasks

[ ] Investigate options
[ ] Implement settings page
[ ] Add a new Settings page
[ ] Move the Delete Profile functionality into the Settings page
[ ] Update the Delete Profile button in the Flyout to open the Settings page

Thanks!

Done - a Settings page has been implemented.

Figure: Simplified flyout with new Settings option

Figure: New Settings page


"Message for vendor" not being displayed on Discussion

v2.0.8 (already seen on older versions)

On some stores, "Message(s) for vendor" is/are not being shown on Discussion tab on Order Details, as expected. I don't know if they arrived or not. I've seen this problem on some stores, but not every store.

If you close the app, then open it again and look at the same order, do the messages appear?
When the message was sent, were both the seller and buyer online? Or only the buyer?

No, they don't appear after restarting.
Buyer was online. Seller I don't know, but I believe they were offline.
I see this behavior on the following stores:
QmSbxq9EA3y3ye4EV8hpzCdN9KHS7wGK7wRV5h9fWrwdLe
QmTJrqtMZXvUts7Ta5Wa9cBDcRTfwmPQ8eYxx85ET9XSA9

This is working correctly, it just takes a while for the server to return the message.
The delay is being tracked as a server issue here:
https://github.com/OpenBazaar/openbazaar-go/issues/669


:bug: Solve the problem of scroll bar blocking pop-up box

Fix https://github.com/halo-dev/halo/issues/2201
What type of PR is this?
/kind bug
Does this PR introduce a user-facing change?
none

修改之后测试图

Nice work!
不过建议对当前这个 PR 留下一个 Release Note，可以按照以下格式修改 PR 描述：
#### Does this PR introduce a user-facing change?

<!--
如果当前 Pull Request 的修改不会造成用户侧的任何变更，在 `release-note` 代码块儿中填写 `NONE`。
否则请填写用户侧能够理解的 Release Note。如果当前 Pull Request 包含破坏性更新（Break Change），
Release Note 需要以 `action required` 开头。
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->

```release-note

可参考：<https://raw.githubusercontent.com/halo-dev/halo/master/.github/pull_request_template.md>。

/lgtm

/retitle fix: the problem of adding link pop-up box showing scroll bar


What's the usage of face parsing result?

You have generated face parsing images during preprocessing.

Ok! It's different from the paper.


[CP] A roll of dart_style which can format empty nullable record types ()?

Commit(s) to merge
7c83e8ad4a67cb4cb3d628d59c6df237cae3ee73
Target
stable
Prepared changelist for beta/stable
https://dart-review.googlesource.com/c/sdk/+/306911
Issue Description
The formatter has a bug where a nullable type record with no fields (i.e. ()?) discards the ?. The formatter has an internal self-check that it only changes whitespace which fails and the result is that the formatter exits with an error message and doesn't format the file at all.
What is the fix
The fix is a trivial one-line change that preserves the ? if there is one. There are no other changes in this dart_style roll.
Why cherry-pick
I don't expect nullable empty record types to be very common, but they are valid and meaningful, and without this fix, users are completely unable to format any part of a file that happens to contain ()?.
Risk
low
Issue link(s)
https://github.com/dart-lang/dart_style/issues/1224
Extra Info
No response

The current CP CL causes build breakage and can't be landed.

Thanks, @itsjustkevin!


Add c++20 kernel

The purpose of this PR is to add a c++20 kernel.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 51.60%. Comparing base (d203fdf) to head (a962fb7).

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #77   +/-   ##
=======================================
  Coverage   51.60%   51.60%
=======================================
  Files          15       15
  Lines         593      593
=======================================
  Hits          306      306
  Misses        287      287


Allow string interpolation in property keys

Closes #267

Codecov Report

Merging #566 into master will decrease coverage by 0.15%.
The diff coverage is 98.71%.

@@            Coverage Diff             @@
##           master     #566      +/-   ##
==========================================
- Coverage   93.91%   93.75%   -0.16%
==========================================
  Files         230      230
  Lines        8281     8295      +14
==========================================
  Hits         7777     7777
- Misses        504      518      +14

Flag
Coverage Δ

#dotnet
93.75% <98.71%> (-0.16%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/Bicep.Core/Syntax/SyntaxBase.cs
74.07% <ø> (+1.66%)
:arrow_up:

src/Bicep.Core/TypeSystem/TypeValidator.cs
95.03% <95.23%> (-0.46%)
:arrow_down:

src/Bicep.Core/Diagnostics/DiagnosticBuilder.cs
100.00% <100.00%> (ø)

src/Bicep.Core/Emit/ExpressionConverter.cs
95.18% <100.00%> (+0.08%)
:arrow_up:

src/Bicep.Core/Emit/ExpressionEmitter.cs
98.59% <100.00%> (+0.13%)
:arrow_up:

src/Bicep.Core/Emit/TemplateWriter.cs
99.04% <100.00%> (ø)

src/Bicep.Core/Parser/ParseDiagnosticsVisitor.cs
100.00% <100.00%> (ø)

src/Bicep.Core/Parser/Parser.cs
94.52% <100.00%> (+0.23%)
:arrow_up:

src/Bicep.Core/Syntax/ObjectPropertySyntax.cs
95.65% <100.00%> (-4.35%)
:arrow_down:

src/Bicep.Core/Syntax/ObjectSyntaxExtensions.cs
100.00% <100.00%> (ø)

... and 16 more


kubectl port-forward does not close the local connection when the server closes the connection

I am trying out kubectl port-forward and I think it does not close the client-side connection properly when the server closes the connection.
I realize this example is probably deeply unhelpful unless you have a zookeeper image lying around, but here's what I observed...
When I ssh in to the minion and target the pod IP, I send the "status" message (ruok) and the server sends the reply (imok) and closes the connection.
justinsb@kubernetes-minion-nyy1:~$ echo ruok | nc <IP_ADDRESS> 2181
imok
justinsb@kubernetes-minion-nyy1:~$

But when I try to do this on the client:
 kubectl port-forward -p zookeeper-rplo3 2181

and then
echo ruok | nc <IP_ADDRESS> 2181
imok
<hangs>

I hope to publish this zookeeper image early this week, if you can't find another test-image where the server closes the connection.

@ncdc

We haven't had issues using curl with port-forward to retrieve web pages from e.g. nginx. Is there something different about how Zookeeper and/or nc handle closing connections?

Ok, I've been able to reproduce. I'm working on a fix now.

@liggitt


Microsoft .NET Core 3.1.300

Microsoft Reviewers: Open in CodeFlow

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: alefranz sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Codecov migration to marketplace app

Hi, Tom from Codecov here.
We noticed that you are using Codecov with fairly high frequency, and we’re so excited to see that! However, because you are not using our app, you may have experienced issues with uploading reports or viewing coverage information. This is due to rate-limiting issues from GitHub.
In order to prevent any future outages, we ask that you move over to our GitHub app integration.
The process is extremely simple and shouldn’t require more than a few clicks, and you should not expect any downtime. By moving to our app, you will no longer need an admin or separate account to manage the relationship with GitHub as the team bot.
Let me know if you have any questions, or if I can help at all with this process.

Hi @thomasrockhu, thanks for the headsup!  I enabled the app, but now codecov has added itself to the CI and keeps failing builds from a lack of coverage.  I don't recall enabling this feature, and I don't see an obvious way to disable it.  Could you point me in the right direction?  :pray:

Hi @iamdefinitelyahuman taking a look now!

Hi @iamdefinitelyahuman just to get some clarity here. Are your CI builds failing or are you seeing failed status checks on your PRs? If it's failed CI builds, could you link me to one?

It's a failed status check. Here's an example: https://github.com/iamdefinitelyahuman/brownie/pull/564
There are 2 new CI jobs, codecov/patch and codecov/project that are both failing because the coverage diff is too low.  While I appreciate having this information, I'd rather not have a drop in coverage cause that a PR or commit to be labelled as failing.

Got it, the product team took a quick look and the most likely reason is that the original setup was not correct, and by switching to the app, it self-corrected.
If you would prefer to not have this happen, I can open a PR to add a codecov.yml file in your repo. It would contain
coverage:
  status:
    project:
      informational: true
    patch:
      informational: true

Let me know if you prefer to do this

that would be perfect :+1:

Here you go @iamdefinitelyahuman https://github.com/iamdefinitelyahuman/brownie/pull/565

Still getting failed status checks...  reopening this issue for now as a reminder to deal with it.  @thomasrockhu if you have any insight it's much appreciated! But you've already gone above and beyond here so no worries if not, I'm sure the documentation will reveal a solution :)

Hi @iamdefinitelyahuman, just wanted to check in here and see what I could do to help! Did switching to target: 0 work for you?

Yep, solved it :)  Thanks for checking!


Not able to open solution/project in visual studio 2015 Enterprise

Not able to open solution/project in visual studio 2015 Enterprise.
Is there any C# version is also available.

@anjumrizwi Just try to reconfigure and submit Pull Request with your enhancements to open it in VS2015.
If you convert it to C#, there will be a  C# version.

@anjumrizwi any luck getting the project to open with VS 2015?

I can open it in VS 2015.  Depending on where the module is stored, you may have to edit the references to get everything wired up.
Does the project not open at all?  Do you get an error when trying to open it?  Or have you got it to work?


[DO NOT MERGE] SEO edits for top scoring industry tests

Edited title and description for length and keywords.
@DaniHalfin
Edits from PR 2155.

Left some comments @jdmartinez36 and @DaniHalfin

@DaniHalfin is this ready to merge?

@tuco-ulf It doesn't look like the changes I commented about were made.

Addressed separately


Add APR API

Goal:
Get listed on Yield aggregators like DefiLlama and showcase our APR prior to starting Liquidity Incentive campaign
E.g.: https://defillama.com/yields
Todo:
Reuse existing logic to calculate APR and export it
Required:
Highest possible APR per asset (Fee + Farm)
Optionally:
Low+High Fee APR & APY per asset
Low+High Farm APR & APY per asset

const farmClient = new FarmClient(api);
farmClient.getFarmApr(id, 'omnipool')


Update bio_ecg.py typo

Found a typo in a comment and wanted to fix.

@rubinovitz thanks a lot :)


OLS-246: Use llamaindex only as retriever

Description

Use llamaindex only for retrieving index and matching data based on query.
Use langchain for LLM call.
Integrate with token truncation.
Some refactoring.

Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library

Related Tickets & Documents

Related Issue # OLS-246
Closes #

Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] PR has passed all pre-merge test jobs.
[ ] If it is a core feature, I have added thorough tests.

Testing

Please provide detailed steps to perform tests related to this code change.
How were the fix/results from this change verified? Please provide relevant screenshots or results.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (bd4265b) 96.09% compared to head (29157eb) 95.15%.
Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #431      +/-   ##
==========================================
- Coverage   96.09%   95.15%   -0.95%
==========================================
  Files          32       34       +2
  Lines        1075     1093      +18
==========================================
+ Hits         1033     1040       +7
- Misses         42       53      +11

Files
Coverage Δ

ols/app/endpoints/ols.py
100.00% <100.00%> (ø)

ols/src/prompts/__init__.py
100.00% <100.00%> (ø)

ols/src/prompts/prompts.py
100.00% <100.00%> (ø)

ols/src/query_helpers/docs_summarizer.py
93.82% <100.00%> (-0.02%)
:arrow_down:

ols/utils/token_handler.py
100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

follow up items on this:

using a different prompt template when we are not injecting any rag documents (https://github.com/openshift/lightspeed-service/pull/431/files#r1494851545)
potentially pivoting to langchain from llama index if we adopt FAISS for our embedding storage format
we may want to move the rag functions out to their own module rather than putting them in "docs_summarizer"

/approve
/lgtm


Does rewrite works on Windows?

I am using Windows 7 and local-web-server 1.2.7.  When I do
ws

The site is running ok.
When I do:
ws --rewrite '/yeah/* -> http://localhost:9000/$1'

I get
The filename, directory name, or volume label syntax is incorrect.

I tried changing the path many times but still got the same error.

on Windows you need to use double-quotes.. I've no idea why, I don't know the platform very well.
$ ws --rewrite "/yeah/* -> http://localhost:9000/$1"

@75lb Thanks!


Enable codegen for exception filters by default

I was reading an article about C# 6.0 last week which included wording along the lines of "VB.NET and F# have already had exception filters for years now" and remembered I'd looked at the codegen implementation for pattern guards on F# try/with expressions a while back and was surprised to find it used catch and the rethrow instruction instead of a real filter handler.
I took another look at the codegen and the logic is already in place to generate bona-fide exception filters; they can even be enabled in the release versions of the F# compiler via the --generate-filter-blocks flag. A comment in fscopts.fs says the logic was disabled by default due to Mono bugs related to exception filters.
Would it be reasonable to re-enable this logic by default? I don't know whether the Mono bugs were ever fixed, but exception filters are going to become much more widespread now thanks to C# 6.0 -- so if the bugs haven't yet been fixed, it seems like there will be new motivation to do so.
The reason we may want to re-enable exception filters by default -- filters better preserve stack traces and improve debugging experience [1].
Some interesting links with more info about exception filters:

[1] https://roslyn.codeplex.com/discussions/541301
[2] http://gregbee.ch/blog/why-doesnt-c-sharp-have-exception-filters
[3] https://smellegantcode.wordpress.com/2009/02/26/fatal-exceptions-and-why-vb-net-has-a-purpose/
[4] http://blogs.msdn.com/b/dotnet/archive/2009/08/25/the-good-and-the-bad-of-exception-filters.aspx
[5] https://web.archive.org/web/20091218171254/http://blogs.msdn.com/clrteam/archive/2009/02/05/catch-rethrow-and-filters-why-you-should-care.aspx

Please include codegen tests covering the new behavior, and update any that relied on old behavior.
From a quick grep I can see that there are existing tests which use the removed --generate-filter-blocks switch, those will clearly need updating.

@latkin Ok, I've updated the tests which check for the presence of the old flag, or explicitly rely on the old behavior. I still need to run the full test suite to look for any tests which implicitly rely on the old behavior.

I've added a codegen test to verify the IL produced when the --exnfilters+ switch is specified (this PR makes it the default).
Are there any other changes that need to be made for this?

@latkin - could you kick of a full Windows test run with this change?
I don't know of any specific reason not to do this.  However

it would be good to re-check the exception-handling-debugging experience we get with this, both with "First chance" excepitons and "Just my code" on/off in Visual Studio

We need some kind of validation on Mono.  That may be easiest by also sending this as a trial PR to the fsharp4 branch http://github.com/fsharp/fsharp where we do AppVeyor Linux testing.  That branch contains most of the fsharp4 work from here so the commits should slot in pretty easily.

Validating now.

There are still 25 tests failing (across codegen and compiler options) in fsharpqa.
Looks like it is a broad range of list/seq comprehension tests and query tests that have been affected - the generated code for the backing IEnumerables is changed to rely on filters now.

Thanks @latkin. I'll try to fix those failing tests soon so we can get this merged.

I pushed these changes to fsharp/fsharp as Don suggested, and the build passed successfully on Mono+Linux (x86). See fsharp/fsharp#396 for details.

I'm running the fsharpqa tests locally and can see that some of them are failing; however, the test infrastructure deletes the generated files so I can't use them to create new baselines. I'm keeping the existing tests/baselines and modifying them to use the --exnfilters- flag when compiling.
Is there an easy way of creating baseline (*.bsl) files for the codegen tests? I imagine it involves compiling each test script twice (netfx20 and netfx40) then running the result through ildasm -- I'm just wondering if you have a script or other tool to automate the process.

I think the fsharpqa CodeGen tests should be fixed now.

Thanks for updating the codegen tests.  After looking into it a bit more, I'm fairly certain we cannot just turn this on by default.
The main issue is that exception filters are not simply a built-in catch+test+rethrow optimization, they actually have different semantics which result in different order of execution.  F# exception pattern guards have until now represented the runtime semantics and operation order of catch+test+rethrow. We can't just flip a switch and cause existing F# code to behave differently like this.
e.g. consider this code
let cond () = printfn "condition"; true

try
  try
    printfn "throw"
    failwith "boom"
  finally
    printfn "finally"
with
   e when cond() -> printfn "caught"

Behavior until now is as you would probably expect:
throw
finally
condition
caught

With filters enabled, the output is (with extra * added):
throw
condition
finally
condition*
caught

[The * line is interesting - I assume it's a bug in the codegen, but maybe it's some attempt to work around the changed order of operations.]
condition changing places to execute before finally is actually correct and expected, if the guard clause is codegened as a filter. The blog post linked in [4] above explains this pretty well - the CLR executes filter blocks all the way up the stack until one of them returns true, before any finally blocks are executed.  This is in contrast to checks inside of catch blocks, which always execute after any enclosed finally blocks.
It's even worse if a finally block has a side effect or itself throws a different type of exception - the filter which originally said it would process the exception might no longer be valid at all.
I don't think it's acceptable to make a behavior change like this without distinct syntax or opt-in from the user.

Lincoln is quite correct about this.  The exception filter change is certainly observable to applications and so a change in behavior would be considered breaking.  If we want to add filters I believe a specific exception filter syntax would be the way to go.
@dsyme do you have any thoughts on this.

Thanks for the fantastic analysis everyone.
Given that this is indeed a breaking change (and one that gives a very different semantics), I would be happy with the feature as an opt-in feature, enabled by a command-line option, but which is off by default.  (I could well imagine enabling this feature in many libraries and production systems due to the better crash dumps and other debugging stack traces)
I don't know if the feature can be reconfigured in time to make the cut for F# 4.0 - I will leave it to @KevinRansom  and @latkin to decide
don

This is already present as an opt-in feature, via the --generate-filter-blocks flag.  The flag is marked as deprecated, so you get a warning, but it works.  The warning can be disabled with --nowarn:75.
I think that's probably where it should stay for now.  Per my examples above, it seems there are still implementation bugs, so it doesn't seem prudent to un-deprecate this yet.

Closing this out as we can't take the change as proposed.  Fixes for the filter codegen implementation or modifications to how the flag is handled can be handled in dedicated PRs/issues.

[The * line is interesting - I assume it's a bug in the codegen, but maybe it's some attempt to work around the changed order of operations.]

Indeed, the generated code re-does the pattern matching in the catch block.  I imagine that was easiest to implement.  If we wanted to avoid that, the filter would have to remember which pattern matched and the values of all the variables bound by the pattern, and the catch block would have to execute the corresponding expression based on the saved information.


Vector values

Just like Olympics, Vector values would be cool and feel less empty

Duplicate of #29
Not following feature request template


Problem: github issues are not really searchable anymore

Description
Currently it's impossible to search anything in issues anymore since people embed their full configs in dropdown instead of attaching. For example, I'm trying to search for an issue now and it doesn't get filtered at all https://github.com/hyprwm/Hyprland/issues?q="workspaces"+"fade"+"animation"+is%3Aissue. No way I'm going through 300 open issues to check duplicates

Well, even if the rules were changed and people started attaching their configs instead of embedding them, the damage is already far too gone to fix the problem.


Optimize TreeSet methods and remove unused RedBlackTree methods

Benchmarks show, that RedBlackTree.difference() is slower than subsequently calling RedBlackTree.remove().
TreeSet.difference(TreeSet) bench(100) took 0.964 sec.
TreeSet.removeAll(TreeSet) bench(100) took 0.798 sec.
TreeSet.difference(TreeSet) bench(200) took 1.913 sec.
TreeSet.removeAll(TreeSet) bench(200) took 1.691 sec.
TreeSet.difference(TreeSet) bench(300) took 2.775 sec.
TreeSet.removeAll(TreeSet) bench(300) took 2.486 sec.
TreeSet.difference(TreeSet) bench(400) took 3.877 sec.
TreeSet.removeAll(TreeSet) bench(400) took 3.249 sec.
TreeSet.difference(TreeSet) bench(500) took 4.706 sec.
TreeSet.removeAll(TreeSet) bench(500) took 4.092 sec.
TreeSet.difference(TreeSet) bench(600) took 5.885 sec.
TreeSet.removeAll(TreeSet) bench(600) took 4.954 sec.
TreeSet.difference(TreeSet) bench(700) took 6.802 sec.
TreeSet.removeAll(TreeSet) bench(700) took 5.724 sec.
TreeSet.difference(TreeSet) bench(800) took 7.856 sec.
TreeSet.removeAll(TreeSet) bench(800) took 6.417 sec.
TreeSet.difference(TreeSet) bench(900) took 8.724 sec.
TreeSet.removeAll(TreeSet) bench(900) took 7.351 sec.

We should also implement benchmarks for union() and intersection().
Because difference() is slower, we will remove it from RedBlackTree or change the implementation and delegate to remove(). Analog for union() if it is slower than add(). Intersection has to check contains() before adding() (or removing()) an element (depends on the direction).

Pre-caching random generated testees leads to:
TreeSet.difference(TreeSet) bench(1000) took 3.217 sec.
TreeSet.removeAll(TreeSet) bench(1000) took 1.712 sec.
TreeSet.difference(TreeSet) bench(900) took 2.967 sec.
TreeSet.removeAll(TreeSet) bench(900) took 1.437 sec.
TreeSet.difference(TreeSet) bench(800) took 2.643 sec.
TreeSet.removeAll(TreeSet) bench(800) took 1.294 sec.
TreeSet.difference(TreeSet) bench(700) took 2.21 sec.
TreeSet.removeAll(TreeSet) bench(700) took 1.075 sec.
TreeSet.difference(TreeSet) bench(600) took 1.819 sec.
TreeSet.removeAll(TreeSet) bench(600) took 0.902 sec.
TreeSet.difference(TreeSet) bench(500) took 1.579 sec.
TreeSet.removeAll(TreeSet) bench(500) took 0.762 sec.
TreeSet.difference(TreeSet) bench(400) took 1.233 sec.
TreeSet.removeAll(TreeSet) bench(400) took 0.628 sec.
TreeSet.difference(TreeSet) bench(300) took 0.892 sec.
TreeSet.removeAll(TreeSet) bench(300) took 0.476 sec.
TreeSet.difference(TreeSet) bench(200) took 0.574 sec.
TreeSet.removeAll(TreeSet) bench(200) took 0.337 sec.
TreeSet.difference(TreeSet) bench(100) took 0.327 sec.
TreeSet.removeAll(TreeSet) bench(100) took 0.15 sec.

After some optimizations we clearly see, that RedBlackTree.difference() is really fast compared to removeAll(). No need to remove it.
TreeSet.diff(TreeSet) bench(1000) took 0.448 sec.
TreeSet.removeAll(TreeSet) bench(1000) took 1.405 sec.
TreeSet.diff(TreeSet) bench(900) took 0.151 sec.
TreeSet.removeAll(TreeSet) bench(900) took 1.266 sec.
TreeSet.diff(TreeSet) bench(800) took 0.143 sec.
TreeSet.removeAll(TreeSet) bench(800) took 1.228 sec.
TreeSet.diff(TreeSet) bench(700) took 0.118 sec.
TreeSet.removeAll(TreeSet) bench(700) took 1.007 sec.
TreeSet.diff(TreeSet) bench(600) took 0.097 sec.
TreeSet.removeAll(TreeSet) bench(600) took 0.843 sec.
TreeSet.diff(TreeSet) bench(500) took 0.078 sec.
TreeSet.removeAll(TreeSet) bench(500) took 0.698 sec.
TreeSet.diff(TreeSet) bench(400) took 0.065 sec.
TreeSet.removeAll(TreeSet) bench(400) took 0.577 sec.
TreeSet.diff(TreeSet) bench(300) took 0.047 sec.
TreeSet.removeAll(TreeSet) bench(300) took 0.445 sec.
TreeSet.diff(TreeSet) bench(200) took 0.032 sec.
TreeSet.removeAll(TreeSet) bench(200) took 0.28 sec.
TreeSet.diff(TreeSet) bench(100) took 0.018 sec.
TreeSet.removeAll(TreeSet) bench(100) took 0.131 sec.

Same for intersect()...
TreeSet.intersect(TreeSet) bench(1000) took 0.284 sec.
TreeSet.retainAll(TreeSet) bench(1000) took 3.108 sec.
TreeSet.intersect(TreeSet) bench(900) took 0.2 sec.
TreeSet.retainAll(TreeSet) bench(900) took 2.698 sec.
TreeSet.intersect(TreeSet) bench(800) took 0.177 sec.
TreeSet.retainAll(TreeSet) bench(800) took 2.629 sec.
TreeSet.intersect(TreeSet) bench(700) took 0.159 sec.
TreeSet.retainAll(TreeSet) bench(700) took 2.227 sec.
TreeSet.intersect(TreeSet) bench(600) took 0.158 sec.
TreeSet.retainAll(TreeSet) bench(600) took 1.895 sec.
TreeSet.intersect(TreeSet) bench(500) took 0.124 sec.
TreeSet.retainAll(TreeSet) bench(500) took 1.592 sec.
TreeSet.intersect(TreeSet) bench(400) took 0.094 sec.
TreeSet.retainAll(TreeSet) bench(400) took 1.279 sec.
TreeSet.intersect(TreeSet) bench(300) took 0.076 sec.
TreeSet.retainAll(TreeSet) bench(300) took 0.92 sec.
TreeSet.intersect(TreeSet) bench(200) took 0.052 sec.
TreeSet.retainAll(TreeSet) bench(200) took 0.602 sec.
TreeSet.intersect(TreeSet) bench(100) took 0.025 sec.
TreeSet.retainAll(TreeSet) bench(100) took 0.332 sec.

And union()...
TreeSet.union(TreeSet) bench(1000) took 0.375 sec.
TreeSet.addAll(TreeSet) bench(1000) took 1.706 sec.
TreeSet.union(TreeSet) bench(900) took 0.237 sec.
TreeSet.addAll(TreeSet) bench(900) took 1.472 sec.
TreeSet.union(TreeSet) bench(800) took 0.215 sec.
TreeSet.addAll(TreeSet) bench(800) took 1.379 sec.
TreeSet.union(TreeSet) bench(700) took 0.184 sec.
TreeSet.addAll(TreeSet) bench(700) took 1.134 sec.
TreeSet.union(TreeSet) bench(600) took 0.166 sec.
TreeSet.addAll(TreeSet) bench(600) took 1.014 sec.
TreeSet.union(TreeSet) bench(500) took 0.134 sec.
TreeSet.addAll(TreeSet) bench(500) took 0.815 sec.
TreeSet.union(TreeSet) bench(400) took 0.111 sec.
TreeSet.addAll(TreeSet) bench(400) took 0.654 sec.
TreeSet.union(TreeSet) bench(300) took 0.081 sec.
TreeSet.addAll(TreeSet) bench(300) took 0.507 sec.
TreeSet.union(TreeSet) bench(200) took 0.051 sec.
TreeSet.addAll(TreeSet) bench(200) took 0.305 sec.
TreeSet.union(TreeSet) bench(100) took 0.029 sec.
TreeSet.addAll(TreeSet) bench(100) took 0.174 sec.

Super-cool! This issue can be obviously closed.


Generated CorePlugin has mixed tabs and spaces

The auto-generated CorePlugin.cs file by the Duality Editor contains mixed tabs and spaces.
I use Productivity Power Tools, which pops up a prompt asking me to Tabify and Untabify everytime I open it up.
I propose modifying the auto-generated file to either use tabs or spaces exclusively, instead of interchangeably.

Fixed. Thanks!


Best way to deploy/test LLM models on TensorRT-LLM for production

Hello,
I am using a fine tuned open source LLM and it works great in the Docker after following the instructions to build TensorRT-LLM.
However, after building the wheel install package I am not able to install it on Ubuntu VM, fails at TensorRT installation step.
Can someone please help with the following:

What is the recommended way to deploy using TensorRT-LLM?
If its using Docker/container are there any instructions available to build custom image(with custom model serving logic) on top the one provided by TensorRT-LLM?

Thanks.

Hi @juney-nvidia ,
I think we need to rebuild the Docker image if we need any more python packages, custom inference code and rebuild it. I was using incorrect command to re-build it.
Is there a way to run it on Ubuntu 22.04.3 LTS?
Thanks for your response!
Thanks.

@amir1m Do you still have the problem? If not, we will close it soon.


In lookup_int, PyLong_Check comes before PyBool_Check and appears to catch Booleans as an error

This was an incorrect observation.


(FSM)Simplify test suite generation

right now (Sep/2015), FSM/JCUnit chooses states not only events as factors. But, except for the first one, states should always be  determined by the previous state and an input symbol (event) given to the FSM. As discussed in [Introduction to Combinatorial Testing]4, in this situation we can exclude them during test suite generation from factors and after the process finishes we can append states determined by previous states and input to each test  case. By this optimization, we should be able to improve FSM/JCUnit's performance.

This feature is useful but all the functionalities of JCUnit can work without this. Pushing this ticket to future.

Closing this issue since FSM support will be removed in 0.10.x.


Staff status uses default (short) API token expiry

Me and @genghisken made groups for the live server but left the VRA and ourselves (and you @jackleland)  as staff.
Then we realised that the staff tokens didn't have access: because they use the default expiry which is 1 day.
Would if be possible to make staff status have a very long expiry (e.g. 30 years) so that status trumps all others?
Or do we think that's not a good idea?

A couple of options:

We could make a 'staff group' which has very long token expiry time and add the staff members to it. This is available with the current state of the repo without modifications
I can change the authentication check to not check expiry time for staff members, shouldn't take long but will be a PR

This of course does mean that if a staff token gets compromised then the owner of that key will always be able to access the api with write permissions, which obviously isn't great security-wise

Isn't that true if all the things we give a stupid long life to the token?
On Wed, 4 Dec 2024, 17:20 Jack Leland, @.***> wrote:

This of course does mean that if a staff token gets compromised then the
owner of that key will always be able to access the api with write
permissions, which obviously isn't great security-wise
—
Reply to this email directly, view it on GitHub
https://github.com/genghisken/psat-server-web/issues/38#issuecomment-2518075921,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADXN74DWMGQHWLV4MEA5MU32D42UJAVCNFSM6AAAAABTAO5T4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJYGA3TKOJSGE
.
You are receiving this because you authored the thread.Message ID:
@.***>

Maybe we just live with the fact that staff members don't automatically get access to the API & add them to the 30 year expiry group that we just created. I'm happy with that situation.


Move GCPPubSub source to "contrib" in preparation from removing it from core.

Proposed Changes

Move the gcppubsub controller to a separate statefulset.
Create separate RBAC rules for the gcppubsub controller.

We should do the same with the AWS source (at least).
Release Note

The GCPPubSub source has been moved to a separate controller and set of release YAML files.

I'll let @Harwayne lgtm this, but minor note: a change to https://github.com/knative/eventing-sources/blob/master/hack/release.sh#L25 might also be required.

/lgtm
/hold
Holding for the two comments, which are optional, just want to make sure you see them first.
Note that this will also miss some of the mark down files that point at specific paths. E.g. contrib/gcppubsub/samples/README.md in which every ko and kubectl path is incorrect.

Updated, PTAL.

/lgtm