Refactor, improve, and occasionally modify copy_files handling

Closes #1742.
Motivation
In short, the copy_files handling was in desperate need of a behind-the-scenes overhaul. There are several reasons for this.

As noted in #1742, you can't do Path() on a Future, which means any recipes doing Path() (or string concatenation of paths) in a Flow are going to crash when dispatched over a workflow engine.
There are several file handling functions that do redundant things or are handled in existing dependencies: copy_decompress_files, copy_decompress_tree, copy_decompress_files_from_dir.
Because of Point 2, not all the functions handle things in the same way. For instance, only some have glob support right now, which is confusing.
It is often unclear to the user how/when a tree structure is copied or not. If you specify a folder, the contents are copied. If you use a dictionary format, it can be retained depending on the value. This is a bit complex. Additionally, the former behavior is not consistent with how users use Linux. For instance, cp -r on a directory doesn't unpack the directory --- it copies the directory as-is, along with all of its contents. We should strive to be consistent with Linux behavior.
There are many instances where the type hints are simply inaccurate. For instance, in the Espresso recipes, the prev_dir: str | Path positional argument is often getting dict entries thrown into, as a result of pw_copy_files and such (see the test suite).

Because of this mess, things were in desperate need of a revamp, which this PR addresses.
User-Facing Changes
In most cases, there are no breaking changes to users and is mostly an internal refactor. However, there are two exceptions:

When a directory is passed in to copy_files, it is copied as-is (i.e. the tree structure is retained). This is identical to the behavior of cp -r. If a user wants to copy the contents of a directory, they should use a glob pattern (i.e. *).
The Espresso recipes' prev_dir: str | Path positional arguments have been updated to reflect Point 1 above. The prev_dir positional arguments have been updated to copy_files: dict[SourceDirectory, Filenames]. If you want to copy all the file from a given directory but not the directory itself, you can do copy_files={"mydir": "*"} or, for convenience, even copy_files="mydir/*" or copy_files=["mydir/*"] (although this is intentionally not advertised in the type hint). There is likely a way to have no breaking change here (i.e. by auto-unpacking for the recipes where that makes sense), but this will have to be handled in a follow-up PR because it got incredibly complicated very quickly.

Behind-the-Scenes
The copy_decompress_tree and copy_decompress_files_from_dir functions are removed. All of those features, including glob support, are now handled in the newly updated copy_decompress_files function.
This is how the copy_files are translated in quacc.runners.prep.calc_setup:
if copy_files:
    if isinstance(copy_files, (str, Path)):
        copy_files = {Path(copy_files).parent: Path(copy_files).name}
    elif isinstance(copy_files, list):
        copy_files = {Path(file).parent: Path(file).name for file in copy_files}
    for source_directory, filenames in copy_files.items():
        copy_decompress_files(source_directory, filenames, tmpdir)

The docstring for the updated copy_decompress_files is illustrative of the various uses:
Filenames = str | Path | list[str | Path]
SourceDirectory = str | Path

def copy_decompress_files(
    source_directory: SourceDirectory,
    filenames: Filenames,
    destination_directory: str | Path,
) -> None:
    """
    Copy and decompress `filenames` from the `source_directory` to the `destination`
    directory.

    For example, to copy the files `CHGCAR` and `WAVECAR` from the `source_directory` to
    the `destination` directory, use the following:

    ```python
    copy_decompress_files(
        source_directory="/path/to/source",
        filenames=["CHGCAR", "WAVECAR"],
        destination="/path/to/destination",
    )
    ```

    This function also supports glob patterns for any of the entries within `filenames`.

    For example, to copy and decompress all files in the `source_directory` with the
    extension `.gz` to the `destination` directory, use the following:

    ```python
    copy_decompress_files(
        source_directory="/path/to/source",
        filenames=["*.gz"],
        destination="/path/to/destination",
    )
    ```

    If a directory is specified in `filenames`, that directory and its contents will be
    copied and decompressed to the `destination` directory.

    For example, to recursively copy the entire directory `prior_run` and decompress its
    files from the `source_directory` to the `destination` directory, use the following:

    ```python
    copy_decompress_files(
        source_directory="/path/to/source",
        filenames=["prior_run"],
        destination="/path/to/destination",
    )
    ```

    Sometimes, you may want to copy a directory but only keep some of the files for the
    sake of saving space. In other words, you want to retain the tree structure of the
    files with respect to some parent directory. To do this, you can specify the
    tree to retain in the `filenames` argument. For example, to copy and decompress the
    files `prior_run/CHGCAR` and `prior_run/WAVECAR` from the `source_directory` to the
    `destination` directory while the tree structure, use the following:

    ```python
    copy_decompress_files(
        source_directory="/path/to/source",
        filenames=["prior_run/CHGCAR", "prior_run/WAVECAR"],
        destination="/path/to/destination",
    )
    ```

Users have full flexibility now, and everything is internally consistent.

@Nekkrad, @tomdemeyere: Thank you very much for your patience with the PR reviews while I was out of town and then trying to handle this absolute nightmare of a refactor.
I am tagging you in this PR because, as things are implemented right now, there is a slight but notable breaking change to some of the Espresso recipes. The easiest way for to summarize this would be to check out the "Files changed" tab and go to the tests for the Espresso recipes. For example, here you will notice that
pp_results = post_processing_job(prev_dir=results["dir_name"])

was changed to
pp_results = post_processing_job({results["dir_name"]: "*"})

Why the change? The long answer is see my description in the PR summary. The short answer is as follows:
To streamline things, when a user passes in a directory to the copy_files keyword argument used in the various recipes, the directory will be copied as-is. This is in contrast to the old behavior where everything got unpacked. If you want all the files in a directory but not the directory itself the way to do that would be to use the * glob pattern. The following three methods do exactly the same thing:
pp_results = post_processing_job({results["dir_name"]: "*"})
pp_results = post_processing_job(f"{results['dir_name']}/*")
pp_results = post_processing_job(Path(results["dir_name"], "*"))

In general, however, I am recommending that users take the dictionary approach because if results is a Future (e.g. when dispatched via a workflow engine), you can get a crash as noted in #1742.
I recognize that this would be a breaking change, and changes are annoying. I generally try to avoid them whenever possible. However, I have been unable to come up with a solution that would be clear and retain old functionality in this scenario.
I also understand that, given the huge size of this PR, my intentions may not be clear. I am happy to hop on a brief call if that would be helpful. In any case, while this PR is critical, I would like to open this up for discussion before I go ahead and merge.

@Andrew-S-Rosen I will be looking at this probably later today or tomorrow. Feel free to merge and if there are breaking changes me or Tom will fix them

Thanks, @Nekkrad! Much appreciated. I'll likely go ahead and merge this in the evening here so that it can unblock a few other PRs (including one of yours). I'll make any changes to your PR to ensure everything passes after I do --- no need for you to do anything there. In any case, I wanted to communicate this minor but notable change in terms of how files are handled.

@tomdemeyere No worries. Try not to open GitHub too much on your time off. 😉
Much of this refactoring were due to nice feature additions you made, so thank you! I will go ahead and merge this shortly, but when you return, don't hesitate to reach out if there are any issues/concerns.


Add Sileo Depictions

Hey man, I think I'm done adding Sileo depictions for you. After you accept this, check it out! If you need help doing anything, don't hesitate to let me know!
Thanks!

Uhh, hang on. I forgot to modify Packages

There we go. You should be ready to go. If you wanted to check it out, add mac-user669.github.io/cydia to your Sileo and LMK what to change. Editing the description and changelog should be pretty self-explanatory.

There are a few things that I want to change but I won't bother you with those since I can do them myself for sure. Like you said, self explanatory. This'll for sure do for a nice basis though. TYVM!

Thanks!


Sometimes opened Tabs don't keep position after active Tab with Multi-Account Containers

MiddleMouse clicking a link that's set to "Always open in" with Multi-Account Containers the opened Tab sometimes doesn't keep index
Depends on https://github.com/mozilla/multi-account-containers/pull/1095 and https://github.com/mozilla/multi-account-containers/pull/1066

Fixed with version 0.58. Requires MAC version 6.0.0


python312Packages.tencentcloud-sdk-python: 3.0.1232 -> 3.0.1233

Diff: https://github.com/TencentCloud/tencentcloud-sdk-python/compare/refs/tags/3.0.1232...3.0.1233
Changelog: https://github.com/TencentCloud/tencentcloud-sdk-python/blob/3.0.1233/CHANGELOG.md
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 342649 run on x86_64-linux 1

4 packages built:

python311Packages.tencentcloud-sdk-python
python311Packages.tencentcloud-sdk-python.dist
python312Packages.tencentcloud-sdk-python
python312Packages.tencentcloud-sdk-python.dist


More intuitive interface for batch computation

Based on feedback from my students, the current interface based on inheritance is not intuitive.
How about adding a new keyword parameter batchedfunc to optimize! and crossinterpolate and so on?
This is similar to scipy optimizer interface, where we can pass a function computing derivative, with jac.
@rittermarc
What do you think?

Is your proposal to just add a boolean batchedfunc, in which case the function will receive sets of indices instead of being evaluated on each element?
I agree that the current interface is not very intuitive, but I'm not sure how much of an improvement this would be.

If your function f object supports both:
crossinterpolate2(Float64, f, ....; batchedfunc=f, ....)

If you have two distinct functions:
crossinterpolate2(Float64, f, ....; batchedfunc=f2, ....)


Use local time for macro expansion

This might be more of a feature request rather than a bug?
When the current weather macro gets expanded, it seems like it does not use local time for expansion. For example, my local time (Toronto time) is already past midnight (09-18 Wednesday), but when I expand my macro, the today gets expanded to Tuesday still.

Similarly, the status bar weather also shows weather for 09-17 Tuesday:

In plugin configuration I set weather fetch interval to be 5 minutes, and it seems like even with 5-min update interval it is not updating to the new day, so I guess it has to do with how vc-weather determines "today" (maybe by UTC time? Im not sure)
Obsidian v1.6.7
Plugin v0.1.4

Yes I need to clarify that in the readme. The time you are seeing is the time of the last update to the weather and depending on how often the stations update in your area it can be off from the current time by quite a lot. Here in Edmonton it can be off by well over a half hour at times. I will add macros for the current time as well as that would make more sense to insert that into your notes. I'll post back here as soon as that is done.
Thanks, William

Sorry for the delay, I wanted to complete the new weather alerts feature before creating a new release and that took a littrle longer than expected.
Available in v0.2.0
The new current Date and Time macros are...

Macro
Description
Replaced With Example

%year1-now%
Expands to the current year long
2024

%year2-now%
Expands to current year short
24

%month1-now%
Expands to current month
1 - 12

%month2-now%
Expands to current month
01 - 12

%month3-now%
Expands to current month
Jan

%month4-now%
Expands to current month
January

%date1-now%
Expands to current month
1 - 31

%date2-now%
Expands to current month
01 - 31

%dow1-now%
Expands to current day of the week
Sun

%dow2-now%
Expands to current day of the week
Sunday

%hours24-now%
Current 24 hours
00 to 23, 1:00am = 13

%hours12-now%
Current 12 hours
12 hours format

%mins-now%
Current minutes
00 - 59

%secs-now%
Current seconds
00 - 59

%ampm1-now%
AM or PM for 12 hour time
AM or PM

%ampm2-now%
am or pm for 12 hour time
am or pm

I also updated a couple of the default templates to take advantage of these. If you are using the default ones, I suggest resetting them. You can do that by opening the data.json settings file for this plugin and deleteing all the template key/value lines then open the plugins settings in Obsidian and make any change (Toggle statusbar off/on for example) and Obsidian will re-create all the missing settings.

can confirm this works! Thanks William ❤


Closing the app takes 3–10s on MacBook Pro with M1 Pro

This is not really a problem but just an odd observation.

There is a fix for this coming in the next version, NNW 6.1.


Wrong video source in gstreamer

If I read the documentation correctly the gstreamer.py video source here should be video0 to target the Coral camera?

@FranciZ Yes, the dev board would use /dev/video0, but that's handled here. That's default parameter which is only set if no arguments is being passed into it


Fix/luaproto encode hooks

Summary
Transcoding between GRPC and JSON requires some types to be handled exceptionally.
Hooks provided by lua-protobuf are insufficient as they:

cannot hook native types (required for bytes)
are not executed recursively (required for Any)

Further, gateway must read custom JSON names from proto definitions. Conversion must also allow to keep camel_case names, enum names (instead of numbers) and to emit also JSON with default values.
Full changelog

added file plugins/grpc-gateway/protojson.lua handling proto->JSON formatting
updated rockspecs to include this new file

In tools/grpc.lua:

refactored to be more readable
add_path and parse_file methods are now exposed instead of for_each_method
extra hook on every field of every message included (to extract json_name option value)
path( "spec/fixtures/grpc" ) excluded from internally added paths as they are only intended for testing and might contain files conflicting with user code. Mixing testing code with production code is bad practice anyway.

in plugins/grpc-gateway/deco.lua

removed code from ancient times where include_imports directive was false.
adapted to new tools/grpc.lua
decoding/encoding of messages is now done by using protojson.lua

in plugins/grpc-web

adapted to new tools/grpc.lua
no functional changes have been made
schema extended to allow also additional files (useful for google.protobuf.Any conversions)
schema allows to add also additional import paths (useful for libraries)

Also fixed bug when shorter binding path matched longer one (e.g. /test matched also /test/something/else).
Issue reference
Fix #7469
Todo: more test coverage

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: git-torrent:x: Pavol OstertagPavol Ostertag seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

I am sorry for not providing tests, but my setup is only AKS cluster with our app. I can only patch files and watch the output in the console. Not much for coding/debugging.
Please feel free to update the code to be more efficient or more idiomatic.
What is missing:

snake_case is still not converted to lowerCamelCase - I considered using __newindex metamethod, but this would not handle custom Json names so I probably stuck to current recursive traversal
default values are not patched - current algorithm traverses metatable and not descriptor so it does not know about default values at all
handling of missing types for any type conversion. I had to use explicit p:loadfile() call what cannot be generalized. I considered using p.unknown_type hook, but script does not know the file name for the type. So maybe some type->file mapping?

[ ] if there is a missing type, what possible mechanisms can I use to load them on the fly? Or up front? Maybe in configuration, schema.lua?

We appreciate your contribution. This is a large PR and we need time to review it.

We appreciate your contribution. This is a large PR and we need time to review it.

Thank you.
All the tests are working except spec/02-integration/07-sdk/03-cluster_spec.lua which fails for both, cassandra and postgres. I hope it is not caused by my changes (plugin is tested daily in our environment). I have no idea how to fix these.

All the tests are working except spec/02-integration/07-sdk/03-cluster_spec.lua which fails for both, cassandra and postgres. I hope it is not caused by my changes (plugin is tested daily in our environment). I have no idea how to fix these.

I'm trying to investigate the issue.

All the tests are working except spec/02-integration/07-sdk/03-cluster_spec.lua which fails for both, cassandra and postgres. I hope it is not caused by my changes (plugin is tested daily in our environment). I have no idea how to fix these.

I'm trying to investigate the issue.

The failing test runs slow in my env and passed. Guess we need to have a longer timeout.

Please rebase the PR. We are dropping the wRPC protocol, which should bring more freedom to how we support gRPC.

Sorry. This PR takes really long to review and new changes seem to collide with it. Could you solve conflicts?

There are two major issues that stand in our way:

my lack of time (every conflict resolution takes me lot of time as I need to recall what to do and where, my daytime job has nothing to do with Lua)
my inability to make unit-tests better than they are right now in reasonable time

I can resolve conflicts this week (maybe today if lucky), but the unit-tests require substantially more time to learn and master.

@git-torrent As this is a very large PR and hard to maintain, and we see no hope for it to be merged soon, I'm thinking about taking over this PR myself if you don't mind. Of course, you will be given credit. What do you think?
My first step would be to break this PR into 3 or more PRs so it's easier to review.

@git-torrent As this is a very large PR and hard to maintain, and we see no hope for it to be merged soon, I'm thinking about taking over this PR myself if you don't mind. Of course, you will be given credit. What do you think? My first step would be to break this PR into 3 or more PRs so it's easier to review.

I appreciate your offer. With this PR having more badges (labels) than old military officer, I think your care would help.
Just one thing please: Names of the variables/functions are optimized for me from the future. Of course, feel free to optimize the code, but until merged, I will be still in charge of keeping our (company) fleet of Kongs up to date, which require this gRPC<->REST operating. Keeping the variable names (when possible) would make this upgrade&patch process easier.
If I can be of any further help, please write me.
Thank you :)

Closing this due to lack of activity. Please re-open if needed.

@hbagdi > Closing this due to lack of activity. Please re-open if needed.
How can you close XXL PR where lack of inactivity is clearly an effect of being XXL.  I do not expect to be paid, I expect Kong to pay little interest in community's work. This way I would use Traefik next time :(

@git-torrent Sorry for the confusion. The closing is done in a fixed process without much look at the PR and has not much intention on it.
I'm reopening this PR but sadly I'm occupied by other issues and cannot find time to work on this recently.
I'll convert this into a draft PR so it won't show up in the list and get closed mistakenly.

@git-torrent Sorry for the confusion. The closing is done in a process without looking at what the PR is doing and has not much intention on it. I'm reopening this PR but sadly I'm occupied by other issues and cannot find time to work on this recently. Please understand that we need to prioritize our work for different urgency. I'll convert this into a draft PR so it won't show up in the list and get closed mistakenly.

@StarlightIbuki I am afraid I need your help with unit tests. I have to rebase this branch from time to time to prepare patch for our deployments. It is useful to have unit-tests done, but I cannot do that. Can you please point me to some exemplary solution of what you wanted? I tried to look at existing test-cases, but mine are still failing. Thank you

@git-torrent Hi. Glad to know you're still interested in maintaining the RP. When I said taking over, I mean I will help to rework the PR, write proper tests for it and etc. It's fine to not fix the test or rebase to keep maintaining the PR.
The good news is that I have planned this PR's work. I will hand on it after the 3.4 release if no unexpected urgent work gets in the way.
If you still want to write the unit test, please tell me the purpose of the test (which part is tested? What do we want to test?) and the problem you're facing.

@git-torrent Hi. I'm gradually starting the rework of this PR.
First thing is to make a full list of the improvement this PR wants to do.
Please correct me/add to the list if I made a mistake or miss something.

Wrap the pb library so we can isolate the hack to it;
Better handling of pb messages:
Make the empty table/array distinctable. This is already done: #10790;
More careful handling of numbers, say, int64, NaN, etc, and boolean/byte types;
Name conversions (camel/snake case);
Support of .google.protobuf.*
Allow different configs to coexist for different instances of isolated pb library, including decode_default_message use_proto_names, local enum_as_name, emit_defaults
Make decoding/encoding behavior configurable for the grpc-gateway plugin;
Endpoint sort;
Fix path searching;
additional_protos;
PUT method;

@git-torrent Hi. I'm gradually starting the rework of this PR. First thing is to make a full list of the improvement this PR wants to do. Please correct me/add to the list if I made a mistake or miss something.

Wrap the pb library so we can isolate the hack to it;
Better handling of pb messages:
Make the empty table/array distinctable. This is already done:  fix(*): gRPC and cJSON's behavior to empty array #10790;
More careful handling of numbers, say, int64, NaN, etc, and boolean/byte types;
Name conversions (camel/snake case);
Support of .google.protobuf.*
Allow different configs to coexist for different instances of isolated pb library, including decode_default_message use_proto_names, local enum_as_name, emit_defaults
Make decoding/encoding behavior configurable for the grpc-gateway plugin;
Endpoint sort;
Fix path searching;
additional_protos;
PUT method;

Thank you. I am not sure if I can map your steps to this PR completely. I miss a support for json_name option in proto files to handle overrides.
Also, I am not sure if you want to preserve proto<->json transcoding as a standalone code. It is a standalone library in C++, C# and Go, so I guess it is better for maintenance. For pb.(encode_)hook to work properly, it must be able to handle scalar types (bytes in particular) and also be recursive (to decode Any type).
I would also like to prepare unit-tests to be sure none of the steps breaks existing (PR) code. I still struggle to run tests in different configurations.
What do you think?

I miss a support for json_name option in proto files to handle overrides.

Edited.

Also, I am not sure if you want to preserve proto<->json transcoding as a standalone code. It is a standalone library in C++, C# and Go, so I guess it is better for maintenance.

Sure. Better maintenance is also a target of this task.

For pb.(encode_)hook to work properly, it must be able to handle scalar types (bytes in particular) and also be recursive (to decode Any type).

It's included as a part of #4. I'm not very sure what the "recursive mean". Could you elaborate?

I would also like to prepare unit-tests to be sure none of the steps breaks existing (PR) code. I still struggle to run tests in different configurations.

I will add tests for each of those changes.

I'm not very sure what the "recursive mean". Could you elaborate?

Message serialized into google.protobuf.Any can contain another google.protobuf.Any attribute and so on. To transcode them the hook must treat Any as new message instance and start over. Moreover, the hook must know the @type in advance. This is handled by additional_protos or some other kind of DescriptorPools from C++ or python protobuf implementations.

And I'm inviting you to review PRs, ...

Gladly, thank you :)

Message serialized into google.protobuf.Any can contain another google.protobuf.Any attribute and so on. To transcode them the hook must treat Any as new message instance and start over.

Yeah, I tried this proto that given below in Lua and Python.
message Phone {
   optional string name        = 1;
   optional int64  phonenumber = 2;
   optional Person item        = 3;
}
message Person {
   optional string name     = 1;
   optional int32  age      = 2;
   optional string address  = 3;
   repeated Phone  contacts = 4;
}

In python, the reference of a Person object can not be assigned to Phone's item.
But in Lua, you can assigned the Person object (such as named person) to Phone's item, and it encountered a core dump when I was using lua-protobuf to encode message.
So I think may be we shouldn't to build a ref cycle. Type can refer to itself, but data cannot. Maybe we can try to imagine how we can implement the encode function in this situation. Encoding one message object A, and it contains another message object B, and B also need the A, encoder will never obtain the complete message data. So the core dump happened because stack is full.

Hello, how is this going?

@git-torrent I underestimated the amount of work and am sorry for failing to keep the promise. 3.5 is filled with other features and for personal reasons, I was off for nearly 2 weeks. This improvement cannot be fit in a gap and has to be planned. I'm trying to make this a part of 3.6.

@git-torrent I underestimated the amount of work and am sorry for failing to keep the promise. 3.5 is filled with other features and for personal reasons, I was off for nearly 2 weeks. This improvement cannot be fit in a gap and has to be planned. I'm trying to make this a part of 3.6.

Can I be of any help? I would still gladly contribute to unit-tests (if I know how to do them :) )

@git-torrent I underestimated the amount of work and am sorry for failing to keep the promise. 3.5 is filled with other features and for personal reasons, I was off for nearly 2 weeks. This improvement cannot be fit in a gap and has to be planned. I'm trying to make this a part of 3.6.

Can I be of any help? I would still gladly contribute to unit-tests (if I know how to do them :) )

We want to rework the PR and before the implementation is written we don't know how to unit test them. The best help we could expect from you should be a list of the improvements that we want in detail like the list we made before. To be honest, I don't fully understand the PR's intention, and the list I made is not very in-depth.

We want to rework the PR and before the implementation is written we don't know how to unit test them. The best help we could expect from you should be a list of the improvements that we want in detail like the list we made before. To be honest, I don't fully understand the PR's intention, and the list I made is not very in-depth.

Google provides automated GRPC-REST translation in their own API gateway solution. Kong provides its own, but lacks many of the translation features required by Google. The goal of this PR is to bring google-compliant translation into Kong.
For this purpose google offers several libraries (go, c++, C#, python), but no lua-one. This library is protojson.lua in PR.
The rest is to use protojson library in grpc-gateway plugin.
Does this bring more light into this PR?

@git-torrent 3.5 is delayed and we have planned so many things for 3.6. I see little hope that we can fit this improvement into the foreseeable future... Sorry again for this message.
I reviewed this fix and have some personal views. The behavior may be easier to do as a separate functionality instead of a part of Kong. It could be a set of functions or a module (to_JSON, from_JSON?) of lua-protobuf, or an extension to it (not necessarily an official part of the lib).
That way it should also be easier to unit test (and you don't need to follow the Kong's conversion to write the tests).

@git-torrent 3.5 is delayed and we have planned so many things for 3.6. I see little hope that we can fit this improvement into the foreseeable future... In fact, we have many other possible improvements in the queue. Sorry again for this message. I reviewed this fix and have some personal views. IMO, your effort is worth a dedicated repository. It may be easier to do as a separate functionality instead of a part of Kong. It could be a set of functions or a module (to_JSON, from_JSON?) of lua-protobuf, or an extension to it (not necessarily an official part of the lib). That way it should also be easier to unit test (and you don't need to follow the Kong's conversion to write the tests). Then from Kong, the PR should be much smaller and easier to review.

Oh, that could be. Is there any guide to write modules or extension to lua-protobuf? Or some example on the web (google did not help me much :) ). I am not the C guy, so I have to learn that first.

And in my very personal POV, there's no need to write such a library in C given LuaJIT emits quite efficient binary and cooperates with C types well.
If you still want to PR to lua-protobuf directly, you could start with Lua C APIs: https://www.lua.org/manual/5.1/manual.html#lua_Alloc

@git-torrent 3.5 is delayed and we have planned so many things for 3.6. I see little hope that we can fit this improvement into the foreseeable future... In fact, we have many other possible improvements in the queue. Sorry again for this message. I reviewed this fix and have some personal views. IMO, your effort is worth a dedicated repository. It may be easier to do as a separate functionality instead of a part of Kong. It could be a set of functions or a module (to_JSON, from_JSON?) of lua-protobuf, or an extension to it (not necessarily an official part of the lib). That way it should also be easier to unit test (and you don't need to follow the Kong's conversion to write the tests). Then from Kong, the PR should be much smaller and easier to review.

Oh, that could be. Is there any guide to write modules or extension to lua-protobuf? Or some example on the web (google did not help me much :) ). I am not the C guy, so I have to learn that first.

By module, I mean "Lua module". There's no such mechanism of lua-protobuf to support plugins. To begin with, I guess you could request some features from the library maintainer, like custom handlers for build-in types or even direct fixes to those build-in types' behavior (with an option). You could write your module in Lua to handle the proto file importing, and if the features you requested do not make it into the library, you could wrap over the original library, or make some hooks/monkey patches on it.

Ok, let's write it in Lua. Do you suggest I should agree with author of lua-protobuf to add here https://github.com/starwing/lua-protobuf a protojson.lua file that does the job for us (and required protoc.lua?).
Btw, I've already reported a missing support for hooking data-types and it was rejected as too laborious. I would not expect anything more :( (it seems nobody wants to be compatible with google).

Do you suggest I should agree with author of lua-protobuf to add here starwing/lua-protobuf a protojson.lua file that does the job for us (and required protoc.lua?).

That would be ideal. If they do not like the idea, you could try make it a library which imports lua-protobuf.

Btw, I've already reported a missing support for hooking data-types and it was rejected as too laborious. I would not expect anything more :( (it seems nobody wants to be compatible with google).

Lua is not a mainstream user of the protobuf and gRPC and people seem to tolerate those glitches in their use. After all, with some workarounds, they can still use it.

Hi, any chance this PR will solve the issue where int64 types are transcoded to JSON like floats with lost digits? According to the specs these should be transcoded as strings.

Hi, any chance this PR will solve the issue where int64 types are transcoded to JSON like floats with lost digits? According to the specs these should be transcoded as strings.

yes, this PR solves this by enabling int64_as_string option https://github.com/starwing/lua-protobuf?tab=readme-ov-file#options

Great news! Any chance this PR will be merged soon? Without this fix, the gRPC-gateway plugin is not really much usable for anyone who uses 64-bit integers like identifiers...
I think I will try to rebase this PR against the latest master and test if it works as expected...

FYI I rebased that branch onto the current master and it builds fine, unfortunately, the int64 types don't really transcode well - they are now wrapped as strings but still processed as floats.
1718291265706240595 becomes "1.7182912657062e+18"
but almost there!

Made it working if anyone is interested, as long you don't mind that in JSON the integers are prefixed with #:
diff --git a/kong/tools/protojson.lua b/kong/tools/protojson.lua
--- a/kong/tools/protojson.lua	(revision 2ea36c87bb42c0a2555f8c9f791baf9654de2c97)
+++ b/kong/tools/protojson.lua	(date<PHONE_NUMBER>881)
@@ -243,13 +243,12 @@
   elseif ( f_type == "bytes" ) then
     return ngx.encode_base64( f )
   elseif ( is_int[ f_type ] ) then -- parse numeric value
-    if ( type(f) ~= "number" or f ~= f or f ~= math.floor( f ) ) then -- possibly remove
-      error( ("invalid integer value at %s, got: %s"):format(path, f), 0 )
-    end
-
     if is_int64[ f_type ] then
       return tostring( f )
     else
+      if ( type(f) ~= "number" or f ~= f or f ~= math.floor( f ) ) then -- possibly remove
+        error( ("invalid integer value at %s, got: %s"):format(path, f), 0 )
+      end
       return f
     end
   elseif ( is_decimal[ f_type ] ) then
@@ -585,7 +584,7 @@
     pb.option( "enum_as_value" )
   end

-  pb.option( "int64_as_number" )
+  pb.option( "int64_as_string" )

   return message_to_json( pb.decode( msg_type, msg ), msg_type, path )
 end

Unfortunately, there is another problem in the opposite direction (JSON to proto for instance when transcoding a POST body) - 64-bit integers are transcoded wrong without warning!
An example:<PHONE_NUMBER>706240595 (or "1718291265706240595" passed as a string, does not matter) in JSON is transcoded to<PHONE_NUMBER>706240512` at the proto side (note the last two digits) - there is apparently some precision loss during the transcoding process...
This is especially dangerous if this is an ID of something because these IDs get silently modified without any warning. Troubleshooting this could be quite a nightmare.
Any idea where to look to fix this one?

Thank you for your input. I am leaving for a short vacation, let me fix it upon arrival.

Any ETA for the fix?

Related issues/PRs to the problem I have reported above:
https://github.com/mpx/lua-cjson/issues/37
https://github.com/mpx/lua-cjson/pull/36

A good news is that we are working on https://github.com/Kong/lua-resty-simdjson, which may solve the issues this PR aims to solve


Istio mesh expansion setup script causes dhclient race which loses resolver settings

Running 0.2.12.
I set up a VM in my mesh.  dnsmasq was correctly added to the resolv.conf.
A few hours after, dhclient rewrote resolv.conf so that it was no longer using dnsmasq.
$ cat /etc/resolv.conf
domain c.kube-pw.internal
search c.kube-pw.internal. google.internal.
nameserver <IP_ADDRESS>

prepend domain-name-servers <IP_ADDRESS>; is still at the end of my dhclient.conf.
$ sudo dhclient -v -1

$ cat /etc/resolv.conf
domain c.kube-pw.internal
search c.kube-pw.internal. google.internal.
nameserver <IP_ADDRESS>
nameserver <IP_ADDRESS>

The Istio VM setup script does this:
  # Update DHCP - if needed
  grep "^prepend domain-name-servers <IP_ADDRESS>;" /etc/dhcp/dhclient.conf > /dev/null
  if [[ $? != 0 ]]; then
    echo 'prepend domain-name-servers <IP_ADDRESS>;' >> /etc/dhcp/dhclient.conf
    # TODO: find a better way to re-trigger dhclient
    dhclient -v -1
  fi
}

This looks to start a second dhclient, rather than change the behaviour of the running one:
root       580  0.0  0.0  20468  1984 ?        Ss   14:51   0:00 /sbin/dhclient -4 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
root      1314  0.0  0.0  20348  1096 ?        Ss   15:18   0:00 dhclient -v -1

When the second one renews, everything is OK.  When the first one renews, the system loses its dnsmasq resolver, and then can't resolve Istio components any more.
Nov 20 22:13:59 localhost dhclient[1586]: DHCPREQUEST of <IP_ADDRESS> on eth0 to <IP_ADDRESS> port 67
Nov 20 22:13:59 localhost dhclient[1586]: DHCPACK of <IP_ADDRESS> from <IP_ADDRESS>
Nov 20 22:13:59 localhost dnsmasq[1516]: reading /etc/resolv.conf
Nov 20 22:13:59 localhost dnsmasq[1516]: using nameserver <IP_ADDRESS>#53 for domain svc.cluster.local
Nov 20 22:13:59 localhost dnsmasq[1516]: ignoring nameserver <IP_ADDRESS> - local interface
Nov 20 22:13:59 localhost dnsmasq[1516]: using nameserver <IP_ADDRESS>#53
Nov 20 22:13:59 localhost dhclient[1586]: bound to <IP_ADDRESS> -- renewal in 41454 seconds.

Nov 21 00:28:55 localhost dhclient[569]: DHCPREQUEST of <IP_ADDRESS> on eth0 to <IP_ADDRESS> port 67
Nov 21 00:28:55 localhost dhclient[569]: DHCPACK of <IP_ADDRESS> from <IP_ADDRESS>
Nov 21 00:28:55 localhost dnsmasq[1516]: reading /etc/resolv.conf
Nov 21 00:28:55 localhost dnsmasq[1516]: using nameserver <IP_ADDRESS>#53 for domain svc.cluster.local
Nov 21 00:28:55 localhost dnsmasq[1516]: using nameserver <IP_ADDRESS>#53
Nov 21 00:28:55 localhost dhclient[569]: bound to <IP_ADDRESS> -- renewal in 42134 seconds.
Nov 21 00:40:18 localhost node_agent[1670]: I1121 00:40:18.071485    1670 nodeagent.go:112] Sending CSR (retrial #0) ...
Nov 21 00:40:18 localhost node_agent[1670]: 2017/11/21 00:40:18 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: lookup istio-ca on <IP_ADDRESS>:53: no such host"; Reconnecting to {
istio-ca:8060 <nil>}

You should be able to restart dhclient with:
systemctl restart networking

help wanted is tagged already...


How to train my data on Bert ranking ?

Please correct me if I am wrong, instead TF_IDF we can use Bert for classification like for the insuranceQA V1 ? (configs.rankings.ranking_insurance_interact model )
If yes, how can we train it on csv file ?

You can do it in the same way as with other ranking models. Please, see the documentation https://deeppavlov.readthedocs.io/en/master/features/models/neural_ranking.html#training-and-inference-on-your-own-data. That is to train on your own csv file you need to provide train, dev, test files in the appropriate file format and use the siamese_reader component as a reader and the siamese_iterator as an iterator in your configuration file.


Add ability to hide horizontal rule in dds-content-group-horizontal element

The problem

We need to be able hide horizontal rule in the dds-content-group-horizontal element.
The solution
The solution would be an attribute we can pass in the element to hide the horizontal rule. Similar to how no-border is used in the dds-cta-block element.
Application/website
AEM (Marketing)
Business priority
Medium Priority = upcoming release but is not pressing
What time frame would this ideally be needed by (if applicable)
No response
Examples
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate issues

@EAVilleda can you please provide a use case or scenario of where you're hoping to use this in context?

@photodow Item Horizontal Group is one of many components that should support hiding the Horizontal Rule on any page in AEM. https://jsw.ibm.com/browse/ADCMS-1748

Ah, ok I think I see now. There are a few horizontal rules in this component, but if I understand correctly you're asking for the horizontal rule at the bottom of the component that's apart of the content block. correct? It appears we are incorrectly hard coding it here to true.
Here are the functional specs for the border used in Content block horizontal, and Content block.
@EAVilleda let me know if I misunderstood. I'm going to reclassify this as a bug for now, and make sure the description is clear with the correct component.

That's correct, I am asking about the last horizontal rule, at the bottom of the component.


Test for estimators

This need to be improved.
For example, compare the predict of y with the average of y.

Need to do some tests for ordinal methods (pull request #57)


[WIP] fix to not reflect all the files at once

If you have a lot of files, EncryptedFileManager tries to reflect all the files at once, which will likely cause you to hit your open file limit (default on ubuntu is 1024), and will also open up a lot of connections to the reflector server.
This uses DeferredSemaphore, see: https://patrickmn.com/software/using-twisteds-deferred-semaphore/ so that only 10 files are reflected at a time.

          1 Error

:no_entry_sign:
Please include a CHANGELOG entry.

          1 Message

:book:
See http://keepachangelog.com/en/0.3.0/ for details on good changelog guidelines

  Generated by :no_entry_sign: Danger


Beta 2021

Describe the bug
A clear and concise description of what the bug is and what you expected to happen.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See not error

Screenshots
If applicable, add screenshots to help explain your problem.
Device (please complete the following information):

iOS Version:
iOS Device:
unc0ver Version:

Place an "x" between the brackets if true:

[ ] this is a bug others will be able to reproduce
[ ] this issue is present with all tweaks uninstalled(except for default packages) or disabled
[ ] this issue is present after a rootfs restore
[ ] this issue is present on the latest version of unc0ver

Logs
If applicable, add logs or error messages here.

stop the spam plz

stop the spam plz


ModuleNotFoundError

I tried to run code from the Minecraft branch and got the following errors:
ModuleNotFoundError: No module named 'agentverse.tasks.minecraft_2players_painting'
ModuleNotFoundError: No module named 'voyager.env'

The following four commands do not work in Ubuntu:
pyobjc-core==9.2
pyobjc-framework-ApplicationServices==9.2
pyobjc-framework-Cocoa==9.2
pyobjc-framework-Quartz==9.2

See #121. Regarding the dependencies, our testing has been limited to macOS environments. It may be feasible to exclude these problematic dependencies and proceed with running the environment. Not all of these dependencies are essential for running the environment.


Can't open file .obj with blender or meshlab

I am using the NeRF-SLAM source code from this Github link: https://github.com/jrpowers/NeRF-SLAM, and when training, I am using the GUI of instant-ngp. The issue is that when I export the .obj file, I am unable to open it with Blender or meshlab. @ToniRV @jrpowers

Probably it‘s because all vertices are inf in the .obj file. If it's the case, you can try modify nerf_fusion.py line 70 and line71 (offset & render_aabb) to some value instead of inf.

Probably it‘s because all vertices are inf in the .obj file. If it's the case, you can try modify nerf_fusion.py line 70 and line71 (offset & render_aabb) to some value instead of inf.

Thank you so much, that is exactly the issue I've been encountering. I have a question, though: are there any rules for replacing 'inf' values? If possible, could you give me an example?


⚠ API - Lens has degraded performance

In 35941df, API - Lens ($LENS_API_URL) experienced degraded performance:

HTTP code: 503
Response time: 579 ms

Resolved: API - Lens performance has improved in 1f8c7e1 after 8 minutes.


Add nodeSelector attribute to linux-based container manifests

Is your feature request related to a problem? Please describe.
It is always frustrating when I am hosting a mixed-pool cluster containing both Windows and Linux nodes, and a helm install (such as the keptn quickstart) does not target appropriate nodes. This results in a broken deployment as the scheduler allocates linux containers to windows node pools, causing the quickstart installer to time out/hang indefinitely.
Describe the solution you'd like
Please consider adding a nodeSelector targeting the appropriate os (kubernetes.io/os: linux) to the manifest files and pod specs for keptn containers.
Describe alternatives you've considered
It is possible to run helm template to extract the helm chart and then manually patch using kustomize, but this is both time consuming and fragile when the underlying helm template(s) are updated. It is also possible to add a taint to either the linux or windows node pools but this is a very heavyweight solution to a relatively simple problem of scheduling pods based on os requirements.
Additional context
N/A

Sounds good.
Two questions:

Is this a default which is always available/set on all Kubernetes nodes on all distributions (GKE, K3s, OpenShift)? On my (rather old) GKE nodes I can see it is there:Labels:
                beta.kubernetes.io/arch=amd64
                beta.kubernetes.io/instance-type=e2-standard-4
                beta.kubernetes.io/os=linux
                cloud.google.com/gke-nodepool=default-pool
                cloud.google.com/gke-os-distribution=ubuntu
                failure-domain.beta.kubernetes.io/region=us-central1
                failure-domain.beta.kubernetes.io/zone=us-central1-c
                kubernetes.io/arch=amd64
                kubernetes.io/hostname=gke-***
                kubernetes.io/os=linux

Would it also make sense to go one step further and specify the desired architecture (e.g., kubernetes.io/arch=amd64)?

Sounds good.
Two questions:

Is this a default which is always available/set on all Kubernetes nodes on all distributions (GKE, K3s, OpenShift)? On my (rather old) GKE nodes I can see it is there:Labels:
                beta.kubernetes.io/arch=amd64
                beta.kubernetes.io/instance-type=e2-standard-4
                beta.kubernetes.io/os=linux
                cloud.google.com/gke-nodepool=default-pool
                cloud.google.com/gke-os-distribution=ubuntu
                failure-domain.beta.kubernetes.io/region=us-central1
                failure-domain.beta.kubernetes.io/zone=us-central1-c
                kubernetes.io/arch=amd64
                kubernetes.io/hostname=gke-***
                kubernetes.io/os=linux

Would it also make sense to go one step further and specify the desired architecture (e.g., kubernetes.io/arch=amd64)?

@christian-kreuzberger-dtx according to the Kubernetes well known labels page this should be assigned by the kubelet across all kubernetes flavours. I can't vouch for it on every cloud provider however as I have only made use of it personally on Azure and AWS.

This is my opinion only, but specifiying the desired architecture would only be necessary if you found that the container was incompatible with other architecture types. This may not hold true if the provider can emulate or otherwise handle the architecture differences. For my specific use case the architecture is not an issue but I can't speak for others.

@christian-kreuzberger-dtx according to the Kubernetes well known labels page this should be assigned by the kubelet across all kubernetes flavours. I can't vouch for it on every cloud provider however as I have only made use of it personally on Azure and AWS.

This is my opinion only, but specifiying the desired architecture would only be necessary if you found that the container was incompatible with other architecture types. This may not hold true if the provider can emulate or otherwise handle the architecture differences. For my specific use case the architecture is not an issue but I can't speak for others.

Thanks!
ad 2.

specifiying the desired architecture would only be necessary if you found that the container was incompatible with other architecture types

We only build our containers for amd64, and AFAIK they won't run on other architectures (e.g., ARM on a Raspberry PI, to name a prominent one).

I would expect it is relatively uncommon to have several node pools of a common operating system but differing architecture however!

I thought so too. Though this was also the first time of hearing about a clusters spanning across multiple OS :D

Thanks!
ad 2.

specifiying the desired architecture would only be necessary if you found that the container was incompatible with other architecture types

We only build our containers for amd64, and AFAIK they won't run on other architectures (e.g., ARM on a Raspberry PI, to name a prominent one).

I would expect it is relatively uncommon to have several node pools of a common operating system but differing architecture however!

I thought so too. Though this was also the first time of hearing about a clusters spanning across multiple OS :D

Hi!
I would like to work in this issue. Please assign it to me if it's still available.


babeld: 1.9.1 → 1.9.2

Motivation for this change
Dear all,

Babeld-1.9.2 is available from

  https://www.irif.fr/~jch/software/files/babeld-1.9.2.tar.gz
  https://www.irif.fr/~jch/software/files/babeld-1.9.2.tar.gz.asc

For more information about the Babel routing protocol, please see

  https://www.irif.fr/~jch/software/babel/

This is a bug fix release.  It fixes two bugs where IPv4 prefixes could be
represented incorrectly, with a range of confusing symptoms ; many thanks
to Faban Bläse for diagnosing the issue.  In addition, it fixes incorrect
parsing of unknown address encodings, thanks to Théo Bastian for the fix.

21 April 2020: babeld-1.9.2

  * Fixed two issues that could cause IPv4 routes to be represented
    incorrectly, with a range of confusing symptoms.  Thanks to
    Fabian Bläse.
  * Fixed incorrect parsing of TLVs with an unknown Address Encoding.
    Thanks to Théophile Bastian.
  * Fixed access to mis-aligned data structure.  Thanks to Antonin Décimo.

-- Juliusz Chroboczek

_______________________________________________
Babel-users mailing list
<EMAIL_ADDRESS>https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/babel-users

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@fpletz

@GrahamcOfBorg build babeld

@GrahamcOfBorg test babeld

This built for me, and as far as i could tell, babeld ran for me.


BIGTOP-4246: Refactor APIs for Cluster

The command for cluster will be adjusted in future PRs

+1 lgtm

It looks good overall, but can you show a screenshot of your run?

Hi, this is only API changes, here is the screenshot of testing results on Postman

Thanks @MacChen01 @wuchunfu


Registering Service is rather slow

We are currently evaluating different ZeroConfiguration Libraries and functionality-wise jmdns works rather nicely. But a big caveat is that the Service registering takes around 7 - 8 seconds, whereas with other libraries it seems to be well under a second. The culprit seems to be in the waitForAnnounced calls inside the registerService methods. Now I do understand that for a truly "save registering", you have to check if the service was announced in between, but why does it take so much longer than listening to existing Services with ServiceListener?
Any hints and explanations would be appreciated, especially in comparison to other mDNS/DNS-SD libraries out there :-)
We are using jmdns version 3.5.2 installed with maven.

Im seeing this too on Android. Registering a service takes a fair while. Did you end up finding a solution to this @schulz-m ?

@mustafasezgin Unfortunately we still haven't had time to look into this, will definitely keep it updated once we could!

I experimented a bit with this.
It seems the main wait (~5s) happens when we do:
this.waitForAnnounced(DNSConstants.SERVICE_INFO_TIMEOUT);
Here it waits until the jmsDNS instance itself is announced.  I suspect that we somehow miss the first announce on start and then must wait until DNS announces itself again.
The other wait is after we register our service (~2S):
info.waitForAnnounced(DNSConstants.SERVICE_INFO_TIMEOUT);
Not sure why this takes so long.
I found two ways to make jmdns faster.
The first is to remove the two waits in JmDNSImpl.registerService. The disadvantage is of course that we are not sure the service is announced but for me that would not be a problem. I am not sure though if this affects other parts of the system.
The second way I found is to remove only
this.waitForAnnounced(DNSConstants.SERVICE_INFO_TIMEOUT);
and put it into create.  This also speeds up the process a lot.
So my question is: Can we safely remove the waits? What purpose do they serve?

Both timeouts mentioned seem to be way too high. RFC - Section 8.1 describes the probing step to be complete (when no conflicts arise) after about 750ms.

@ViToni Is this really the feature we are looking at? The spec says that it should probe before it announces address records by itself, but the code seems to check whether it can find announcements for its own services after registering them. Or do I misinterpret the code?

@kaikreuzer I haven't checked the code itself. Just thought how registering a service should work according to the spec:
For a service to be registered I would expect these prerequisites to be sucessfully fullfilled

Probing
Announcing (after succesful Probing) => regsitration completed

The probing step takes about 0-250ms + 3x 250ms (so at most 1000ms) to be succesful but could fail earlier. When an mDNS service is registered dynamically (the mDNS service is already up for some time) the random offset fof 0-250ms might be omitted so it would be even faster.
As said: didn't check the code but the timeouts seem too high and waiting for its own service doesn't seems correct either (but this is a different story).


[feat]: Enable email notifications

Prerequisites

[X] I have searched for existing issues that already suggest this feature, without success.

Describe the Feature Request
Setup email in settings menu:

[x] add email form
[ ] save will automatically send test email
[ ] Button to resend test email
[x] When email is added, we need a success message notifying that an email will be sent, and direct user to go to email spam folder and mark it is not spam so further emails can be sent.

Describe the Use Case
Ability to view notifications from the Embassy when not visiting in the UI.
Describe Preferred Solution
No response
Describe Alternatives
No response
Anything else?
No response

Due to email being a centralized pile a shit, instead of sending email from an SMTP on EOS, users can insert external SMTP credentials (SES, Mailgun, etc) in EOS settings, and Embassy will use that.

Add option to dubscribe to start9 email delivery

Enable Notification options:
[On Success, On Failure] as checkboxes so that both can be selected to enable an "always" notify option
Summary/Detail (that would include some sort of log file of detail)
Enable Notification Method:

Start9 Delivery
Custom SMTP
a) Mail Server
b) Mail Server Port
c) SSL/TLS Checkbox
d) Email From
e) Email To
f) Subject Line (Allow variables like Device Name)
g) Enable Authentication (BASIC Auth, MS Graph API Integration, OAuth2)

Enable Output Options - TXT based email, HTML based email, PDF Attachment with 1 of the 2 prior methods.
Email Delivery when using the SMTP server of the recipient Mailbox is local (on the same box) is private and never leaves the server. When coupled with SSL, there is a high level of privacy. O365 and Gmail have ways to send unauthenticated email (when using a custom domain name) so long as the recipient address is at the same domain.


Issues with building Azure-DCAP-Client on Windows

The referenced power-shell script get-prereqs.ps1 does not get executed by default and I ran the following scripts from an elevated power-shell script.
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine
Unblock-File .\get-prereqs.ps1
Also got the following error with Visual Studio Build:
Severity    Code    Description    Project    File    Line    Suppression StateError
MSB3030    Could not copy the file "C:\Packages\Azure-DCAP-Client\src\Windows\curl-ca-bundle.crt" because it was not found.    dcap_provider    C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets    4551

Trouble-shooted further by specifying the command in an elevated power-shell window
Invoke-WebRequest -OutFile curl-ca-bundle.crt -Uri https://curl.haxx.se/ca/cacert.pem
Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.
At line:1 char:1

Invoke-WebRequest -OutFile curl-ca-bundle.crt -Uri https://curl.haxx. ...

+ CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
Fixed the problem by making power-shell use TLS1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Great if the README.md file can be updated to indicate the extra steps to take the build to work.

cc @CodeMonkeyLeet


Fix badge in readme

See https://github.com/badges/shields/issues/8671. I've also gone ahead and changed the URL to the new format GitHub uses for linking to workflows.

thanks @svenluijten !


[INFO]

Environment
Please provide as many details as you can:

Hosting type

[] Form.io
[x ] Local deployment

Version:

Formio.js version: 4.4.4
Frontend framework: Angular
Browser: Chrome
Browser version: e 80.0.3987.122 (

Steps to Reproduce

I don't submit to formio, i have (submit) event in wich i manage the submission. If i have an error in the procedure, the front-end shows the message "Submission complete". How can i modify this behavior
This is my procedure
``
onSubmit(submission: any) {
let codiceStato: number;
/*
proprietà state contiene info su operazione
draft: ho premiuto save
submitted: ho premuto submitted
*/
this.data = submission['data'];
if (submission['state'] === StatiForm.codiceStatoDRAFTFormio) {
codiceStato = StatiForm.codiceStatoBOZZA;
}
if (submission['state'] === StatiForm.codiceStatoSUBMITTEDFormio) {
codiceStato = StatiForm.codiceStatoINVIATO;
}
try {
 if (Costanti.TIPO_STRUTTURA_WIZARD.localeCompare(this.tipoStruttura) === 0) {
   this.currentStep = this.formio.formio.page;
 }
   console.log('SUBMISSION ' + JSON.stringify(submission.data) + '**********');
  // this is the post data
   this.sottomettiForm({"data": submission.data}, this.idModulo, this.idIstanza, codiceStato, this.currentStep);

} catch (e) {
this.msgErr = "Errore preparazione dati, istanza non salvata";
}
}
``

We're currently addressing a backlog of GitHub issues, and as part of this effort, some inactive issues may be marked as closed. This isn't a dismissal, but a step toward more efficient tracking.
If you feel the issue is still relevant, please re-open and we'll ensure it gets the attention it deserves. Your understanding is appreciated as we work to enhance our open-source responsiveness.


Fix  this._component is null

related to #10635

This is not an actual fix since we don't know it became null in the first place. We need to figure out why it's null and fix that.

hi @satya164 ,  this is the exception http://crashes.to/s/280c411449d

@jerson yes, but why does it happen

This doesn't fix the fact that this._component is null. Please fix the root cause instead of covering up errors.

I meet the mistake is this._component is undefined; what problem is this?

@jerson there's some comments that haven't been addressed yet, do you still plan to ship this PR?

This doesn't fix the fact that this._component is null. Please fix the root cause instead of covering up errors.

@jerson there's some comments that haven't been addressed yet, do you still plan to ship this PR?

Closing since the last comment is a question and this PR has been inactive in a while. If you plan to continue working on this please send a new PR that fixes the root cause.


GetFullyQualifiedNameTest.ValidateLinuxGetFullyQualifiedDomainName test failed

Steps to reproduce

pushd src/libpsl-native
cmake -DCMAKE_BUILD_TYPE=Debug .
make -j
make test

Expected behavior
test success
Actual behavior
test failed

[  FAILED  ] GetFullyQualifiedNameTest.ValidateLinuxGetFullyQualifiedDomainName

Environment data
Mac OS X 10.11.4

> $PSVersionTable

Reason
In ValidateLinuxGetFullyQualifiedDomainName,
GetFullyQualifiedName() function of my machine has returned first uppercase phrase (eg: Xxx-Yyy-Zzz).
But expected host name was lowercase phrase(eg: xxx-yyy-zzz).
Therefore, this test has been failed.
Proposal resolution
Do I convert the result for GetFullyQualifiedName() function into lowercase string ?

For the resolution, I'm really not sure. This test has been really iffy. The code is here, and the function it's testing is here.
There seems to be much disagreement between getaddrinfo()->ai_canonname and gethostname(); which, to be fair, is probably to be expected (nodename and canonnname are not necessarily identical).
I think the main problem is here, where the library first attempts to get the hostname, and if it looks like an FQDN, uses it; otherwise it attempts to get the cannonname. This logic was implemented in 2d442eb73279a24638e63c4a82344de5717e5a70.
@palladia do you remember why this was done? It's causing problems :smile:

If there is no effect of doing this evaluation as case insensitive, there's no harm in calling ToLower on the string. However, there looks to be a case insensitive assert, so I would suggest ASSERT_STRCASEEQ rather than doing the conversion

@ritalin can you verify if this still reproduces? I've modified the test (and I'm getting sorely tempted to delete it, there are just so many edge cases).

In v6.0.0-alpha.10, all test is green, and  build process has been successful.
Thanks.


Dedent

Per https://github.com/sveltejs/kit/pull/9270#issuecomment-1450855548, this adds a dedent utility for the places where we need to generate code from template literals. It allows our code to be much more readable, and makes the generated code more reliably readable as well.
Need to use it for the virtual modules declared inline, and figure out whether to use it in render.js (I think the answer is probably 'yes', but want to make sure we can do so without introducing any performance overhead)
Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

[x] It's really useful if your PR references an issue where it is discussed ahead of time. In many cases, features are absent for a reason. For large changes, please create an RFC: https://github.com/sveltejs/rfcs
[ ] This message body should clearly illustrate what problems it solves.
[ ] Ideally, include a test that fails without this PR but passes with it.

Tests

[ ] Run the tests with pnpm test and lint the project with pnpm lint and pnpm check

Changesets

[ ] If your PR makes a change that should be noted in one or more packages' changelogs, generate a changeset by running pnpm changeset and following the prompts. Changesets that add features should be minor and those that fix bugs should be patch. Please prefix changeset messages with feat:, fix:, or chore:.

I think the render.js stuff should probably happen as part of a separate refactor, if at all


Add non deterministic as failed cause for workflow task

What changed?
Add new enum value to WorkflowTaskFailedCause for non deterministic error.

Why?
SDK could report non deterministic error properly, and server side could emit metrics accordingly.

How did you test it?
eye_balls

Potential risks
No

@yiminc I'm trying to find the issue/pr where we were talking about this but I couldn't - but we should really consider separating the causes that SDKs can produce from the causes server can produce. As it stands it's very confusing what can come from where.
So, now we have a value for this which is better than nothing, but we should try to address that problem as well long run, which might be hard to do in a backwards compat way.

Ah, here: https://github.com/temporalio/temporal/pull/1726


Support a friendlier format

Editing JSON for the slides is not optimal. Instead, we can consider:

Changing the format to something else (e.g. TOML)
Using proportions of the screen instead of raw coordinates
Having a slide editor TUI of some sort :D

Tell me what you think :)

Hi !
What about creating a converter, Markdown to JSON?
to create simple slides


Implement basic C++ idioms

[Who]   As a software developer using GroundUpDB
[What]  I'd like to #include files as I do with other projects
[Value] To make learning how to use GroundUpDB in C++ easier
[Who]   As a software developer using GroundUpDB
[What]  I need to scope my imports to namespaces
[Value] In order to avoid collisions in class and type names
[Who]   As a software developer using GroundUpDB
[What]  I want to use an updated DB library without recompiling
[Value] To minimise disruption to my application's users
[Who]   As the GroundUpDB product owner
[What]  I want to be able to accept new component implementations without reworking my code
[Value] To minimise the cost of integrating new contributions, and make adding contributions easy

For inspiration: https://google.github.io/styleguide/cppguide.html


Use cross-spawn when sh is false for better Windows spawning support

This became a much deeper issue than I had first realized! https://github.com/nodejs/node-v0.x-archive/issues/2318 has a lot of information related to the problem I was having.
Specifically, my home folder on Windows has a space in it. FILE_ACTIVE would provide the full path to the file I was building. But node's escaping isn't ideal when it tries to pass that to the Windows cmd executable.
Based on the numerous references to cross-spawn in the issue above, I ended up implementing this by using cross-spawn but only when sh is false. Using it when sh is true broke a ton of use cases in the spec and I'm sure out in the wild that were implemented like this:
{   "cmd": "echo Hello World!",   "sh": true }

Fixed Travis in https://github.com/johnogle222/atom-build/pull/1

This looks awesome. Thank you so much for taking the time to delve into this! I wanna give this a quick test, which I'll most likely be able to do tomorrow then I'm happy to merge it and make release.
Hooray for better cross-platform support :cake: !

Thank you so much for this! :+1:


Added search feature

Hi @TakangNina , this feature is really impressive. I am grateful for the work you've put into this. But there's a small bug. When we select a tag then the appropriate results are shown. That works fine. But when we select "all" after that, then a blank screen is shown.
https://github.com/user-attachments/assets/5530edbb-122a-4a56-8f6d-b9954b4f1634
If you can resolve that then that would be really great!

Not a problem I will work to resolve it
Thank you, you rock🔥

Fixed that can you check please?

Done,
Thanks! also the navbar feature is great!


[kor] downfall achievements

dont do it now we haven't finished adding new ones or changing existing ones

dont do it now we haven't finished adding new ones or changing existing ones

!!! OK~


Move scroll_behavior into widgets

Previously this code was in the animation layer, which didn't make much
sense because scrolling is a widget concern.

@collinjackson @chinmaygarde

Lgtm


YARN error Sprockets::FileNotFound in Welcome#index

couldn't find file 'yarn_components/raven-js/dist/raven' with type 'application/javascript'
Checked in these paths:
/Users/indianic/Documents/GitHub/peatio-1/app/assets/images
/Users/indianic/Documents/GitHub/peatio-1/app/assets/javascripts
/Users/indianic/Documents/GitHub/peatio-1/app/assets/stylesheets
/Users/indianic/Documents/GitHub/peatio-1/vendor/assets/javascripts
/Users/indianic/Documents/GitHub/peatio-1/vendor/assets/stylesheets
/Users/indianic/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/clipboard-rails-1.7.1/vendor/assets/javascripts
/Users/indianic/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/clipboard-rails-1.7.1/vendor/assets/stylesheets
/Users/indianic/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/grape-swagger-ui-2.2.8/vendor/assets/fonts
/Users/indianic/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/grape-swagger-ui-2.2.8/vendor/assets/images
/Users/indianic/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/grape-swagger-ui-2.2.8/vendor/assets/javascripts
/Users/indianic/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/grape-swagger-ui-2.2.8/vendor/assets/stylesheets

Hello @samirpatelindianic,
I can help you get Peatio running. I also offer development services.
You can always reach me at<EMAIL_ADDRESS>Best,
Oskar

Fixed. needs documentation update.

@samirpatelindianic Could you explain what you did to fix that?
I'm having the same issue.

Somehow I again had same issue today.
Earlier I remember I installed rails and update gems and it worked fine..
If you find solution please message me and i’ll Do the same if I come
across a solution
On Tue, 6 Feb 2018 at 9:47 PM, mauvafo<EMAIL_ADDRESS>wrote:

@samirpatelindianic https://github.com/samirpatelindianic Could you
explain what you did to fix that?
I'm having the same issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/peatio/peatio/issues/707#issuecomment-363475374, or mute
the thread
https://github.com/notifications/unsubscribe-auth/APlugmJdBmrqiIOtMXjrJL4K7sOCChqZks5tSHsjgaJpZM4R4C7p
.

--
Thanks,
Samir Patel
Australia : +61 2 8317 1138
USA : +1<PHONE_NUMBER> ext 722
Whatsapp : +91<PHONE_NUMBER>
Skype : Indianic.samirp
www.indianic.com

I removed all the yarns I had:
npm uninstall yarn -g
gem uninstall yarn
Then, I got it using apt-get:
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install yarn
After that, you should see this:
yarn --version
1.3.2
Finally, I executed the same command executed by the dockerfile:
bundle exec rake tmp:create yarn:install assets:precompile


[ADAM-812] fix for javadoc errors on JDK8, fixes #812

Fixes #812

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/890/
Test PASSed.

Thanks, Michael!


Monitor does not work

Hello! Thanks for the integration of stampC3 into platformIO!
I have a stampC3 and stampC3U and flashing works perfectly with this two, but Serial Monitor does not.. Here is the output from the terminal:
--- Terminal on COM14 | 115200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
Calculated: 86b6856493d23fe741a1623c3d149a38bdf91ca0101173d85f25c3fdb6d605ee
Expected: e7bbd322b12b609ed444a0547cebdc54bbcf46aefed1e04449d9827791f3e8ab
Attempting to boot anyway...
entry 0x403ce000
at this point nothing happens.. I use the example with changing the led color by pushing the button (which works)

hello,
To enable Serial Monitor in platform.io you need to add these in the .ini
build_flags =
    -DARDUINO_USB_CDC_ON_BOOT=1         ; this and
    -DARDUINO_USB_MODE=1                ; this to enable "USB CDC" option

Thanks for the solution @nicolasdb


[table setup] Make SUM/MIN/MAX/... checkbox create the metric instantenously

I am using sqlite to extract information with caravel.
The database have two columns time_estimated and time_spent with numeric types, in List Table Column I checked Sum, Min and Max.
When I want make a bar chart, I want to sum the time of columns, but in Metrics only appears COUNT(*) and INTEGER columns, time_spent and time_estimated are lost.
I made a trick with date columns, because are stored as integer with timestamp format, a datetime(date_create,'unixepoch') it works, I tried to make the same with NUMERIC, I used CAST(time_spent as integer) but in caravel in List Table Column the column type is UNKNOWN and in Metrics stills lost.
How can I sum NUMERIC columns?

Which version of caravel?

I know it's not intuitive, but after checking the sum checkbox you need to hit save for the related metrics to be created.

Hello we are using Caravel 0.9.0

I will check the save button.

It works!
May be can be usefull to put the Save button in all Tab.
Thanks a lot!


Milestone 1 Feedback

Congrats on finishing Milestone 1! We can see that you put some thought into this project. Below we list some specific feedback you can use to improve your project.
We provide tick boxes for you to use in the future as you address these concerns to improve the final grade of your project.
If anything is unclear, please feel free to ask questions in this issue thread.
Group 17

[ ] There are useless titles for this milestone submission (ex. Usage) -2

[ ] Missing Function 2. Not all the team members have been assigned to an issue - mechanics -2

[ ] Usage documentation could be improved for clarity (i.e., it is not explicitly clear to the user how to use the project, or some of the wording is confusing, for example, I’m not sure what output extcolors gives?) -2

All and all I think this group did a great job of creating issues to make sure everyone was assigned a function. The project structure was done well. Each function’s docstrings were also done well. Lastly, the branches were also done well in terms of naming them useful names, as well as not having redundant branches. As a side note remember to have every member commit in the GitHub repo equally.

Hello @andytai7, thanks for your feedback. We have a couple of questions about the specific feedback for understanding since we need to address these concerns in the future.

The Usage title was left in as we have not made the functions yet. We were going to add the usage once the functions were written. Were we supposed to get rid of it?
The function 2 issue was created and assigned to @ataciuk. It was also assigned to Milestone 1 and was on the project board, but it was not closed. Maybe that is why you thought it was missing?
extcolors is a package that we were importing for our project. I don't think we need to write documentation for that?

Thank you!

Yes, there should only be items on the github repo that is needed for that milestone. Anything extra is not necssary.

Function 2's issue was created 17 minutes ago.

Oh i see, you are correct for this, marks changed +2

Okay noted, we will ensure to follow this in future.
The screenshot shows that it was closed 17 minutes ago, and not created. If you open the issue it was created 4 days ago.
Thanks!

Function issues for that milestone should be closed before milestone submission. Does that make sense?

Got it, thanks

Thank you and good luck on Milestone 2.


Iterate over rows of a Model and with pydantic objects

It would be nice to be able to iterate over a Model.
Something along those lines:
class DataFrame(pl.DataFrame, Generic[ModelType]):
    ...
    def iter_models(self, validate: bool = True) -> list[ModelType]:
        if validate:
            self.model.validate(self)

        pydantic_objs = []
        for r in df_model.rows(named=True):
            pydantic_objs.append(self.model.model_construct(**r))

        return pydantic_objs

If accepted, I am willing to take this one

Hello! Could you expand on how you would use this? I note that we already have Model.iter_models and pt.DataFrame.iter_models, but I think you knew that, and I'm failing to spot what behaviour you desire.


Add Machine Learning Tutorials

https://github.com/ujjwalkarn/Machine-Learning-Tutorials

Thanks Ujjwal!


Add CODEOWNERS

See https://github.com/Senzing/template-repository/issues/19

Pull requests:

https://github.com/Senzing/template-repository/pull/20


DataArray attributes

Description
When coding index functions using the daily_downsampler make sure to use 'keep_attrs=True' or to create appropriate attrs (minimally suggest : units, long_name, standard_name) as an ordered dictionary
Example:
from xclim.utils import daily_downsampler as dds
grouper = dds(da, freq=freq)
output = grouper.max(dim='time',keep_attrs=True)
In cases where the index calculation results in a change of units (or both units and name attrs) they should be modified (as an ordered dictonary) before returning the dataarray.
E.g. Calculation of growing season length from temperature data would change all attributes (units, long_name, standard_name)

I suggest to use with_attrs for this (see indices.py).

Ok. @huard If I am understanding correctly 'with_attrs' will assign the listed attributes in the decorator to the output xr.DataArray.
However, the keep_attrs=True (in the resample / grouby) is potentially still of interest for cases when we wish to retain some or  all of the orginal attrs...  For example in some cases units won't change but standard names etc will

@tlogan2000 Correct.
That's my understanding as well.

Donc si on change les attributs dans le calcul d'un indice, ce changement se fera écrasé par @with_attrs c'est ça?

Ok c'est beau, Travis a répondu à ma question.


update to support inherited resources 1.3.0

Please, forgive me if I made some mistake because I never used github and git. I followed the tutorial of https://github.com/gregbell/active_admin/blob/master/CONTRIBUTING.md to help me to contribute to active admin project . I ran all tests and all tests passed.

I can not understand your argument. I had read the issue 808 before I've created this fork. What motivated me to create this fork was the fact that inherited resources 1.3.0 provides support to mass assignment roles which is a new feature of Rails 3.1. I modified active admin to support this changing and I ran all tests before make a pull request. I did anything wrong?

Greg mentions there are reloading issues which are breaking ActiveAdmin.  Maybe that is old information.  It is possible that the app has problems in development mode which the tests are not picking up.  I am not sure, I do not know the details unfortunately.

I don't know if my fork presents some problem in development mode but I'm using it as replacement for the active admin gem in a project. I'm using this fork for a week and I did not find any problem yet.

I can confirm that specs are passing with this change on Ruby 1.9.3-p0.
+1 for this PR. I think the new mass assignment feature in Inherited Resources 1.3 is a must for almost any project. I have no idea how things worked before :)

Merged in to master. Thanks @fabiormoura!


[AIRFLOW-2747] Explicit re-schedule of sensors

Make sure you have checked all steps below.
JIRA

[X] My PR addresses the following Airflow JIRA issues and references them in the PR title.

https://issues.apache.org/jira/browse/AIRFLOW-2747

Description

[X] Here are some details about my PR, including screenshots of any UI changes:

Please see Jira for detailed description of the proposal and screenshots

Tests

[X] My PR adds the following unit tests

test_ready_to_reschedule_dep.py
New tests in test_base_sensor.py

Commits

[X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[X] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

Codecov Report

Merging #3596 into master will increase coverage by 0.08%.
The diff coverage is 88.57%.

@@            Coverage Diff             @@
##           master    #3596      +/-   ##
==========================================
+ Coverage   76.85%   76.94%   +0.08%
==========================================
  Files         204      205       +1
  Lines       15522    15618      +96
==========================================
+ Hits        11930    12017      +87
- Misses       3592     3601       +9

Impacted Files
Coverage Δ

airflow/exceptions.py
100% <100%> (ø)
:arrow_up:

airflow/sensors/base_sensor_operator.py
97.61% <100%> (+0.95%)
:arrow_up:

airflow/ti_deps/deps/ready_to_reschedule.py
100% <100%> (ø)

airflow/models.py
88.91% <100%> (+0.18%)
:arrow_up:

airflow/ti_deps/dep_context.py
100% <100%> (ø)
:arrow_up:

airflow/www_rbac/views.py
71.33% <40%> (-0.24%)
:arrow_down:

airflow/www/views.py
68.77% <40%> (-0.18%)
:arrow_down:

airflow/ti_deps/deps/base_ti_dep.py
90.69% <0%> (+2.32%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa17b5b...80e9c47. Read the comment docs.

@seelmann Can you base onto master?

Maybe we could even make the rescheduling default behaviour for Airflow 2.0, and get rid of the blocking tasks. That would also simplify the code/logic.

Rebased. 66 tests failed, seems all AWS (S3, Redshift, Dynamo) related, is this a know problem?

@seelmann I think this might have to do with a new release of Boto3: https://pypi.org/project/boto3/#history

@Fokko I rebased again and made following chages:

Instead of using a reschedule boolean flag I switched to mode property with possible values poke (default) and reschedule to allow other modes in future (e.g. in_scheduler as mentioned by Maxime).
Changed the Gantt view based on feedback from Pedro, see screenshots in the Jira issue.
Changed the CSS class in Gantt for the NONE state to display as white instead of black, streamlined to graph and tree view.
Added more tests and updated documentation

From my PoV it's ready to be merged.

I will try to spend some time on the pr tomorrow.

@seelmann , I put some small comments / question on the pr. Great work!

Codecov Report

Merging #3596 into master will increase coverage by 0.02%.
The diff coverage is 82.4%.

@@            Coverage Diff             @@
##           master    #3596      +/-   ##
==========================================
+ Coverage   77.52%   77.55%   +0.02%
==========================================
  Files         198      199       +1
  Lines       15842    15963     +121
==========================================
+ Hits        12282    12380      +98
- Misses       3560     3583      +23

Impacted Files
Coverage Δ

airflow/models.py
89.01% <100%> (+0.2%)
:arrow_up:

airflow/sensors/base_sensor_operator.py
97.87% <100%> (+1.2%)
:arrow_up:

airflow/exceptions.py
100% <100%> (ø)
:arrow_up:

airflow/ti_deps/dep_context.py
100% <100%> (ø)
:arrow_up:

airflow/ti_deps/deps/ready_to_reschedule.py
100% <100%> (ø)

airflow/www_rbac/views.py
72.04% <35.29%> (-0.59%)
:arrow_down:

airflow/www/views.py
68.85% <35.29%> (-0.46%)
:arrow_down:

airflow/bin/cli.py
64.74% <0%> (ø)
:arrow_up:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e5eee8...cdd8f89. Read the comment docs.

@feng-tao I replied and resolved all your questions and comments, PTAL. Currently I added additional commits on top, when all is fine I'll squash and rebase everything.

Will take a look tonight.

Thanks @seelmann . LGTM.

@mistercrunch PTAL, you can shut down some idling (or should I say poking) machines now ;)

Will this work equally well on sensors as well as regular tasks?

To make it work for regular operators I think the only thing to do is to override the dep function in your operator and add the ReadyToRescheduleDep, just like in https://github.com/apache/incubator-airflow/pull/3596/files#diff-81901d44cd480538f1f35b641b9ede0cR126. But I didn't test that yet.

I think this feature needs some work

the tasks that are in "reschedule" just show up as "None" in the tree view:

This makes it hard-to-impossible to know what is going on and they probably need to show up as a new state, or just make them show up as running, queued or something? Showing as None isn't helpful. (I even knew what was going on and was still confused!

The scheduler (when using SequentialExecutor, but that isn't relevant) logs this task as Success!
[2018-12-15 18:59:13,635] {jobs.py:1100} INFO - 1 tasks up for execution:
     <TaskInstance: hello_world.wait 2018-12-15 18:50:00+00:00 [scheduled]>
[2018-12-15 18:59:13,649] {jobs.py:1135} INFO - Figuring out tasks to run in Pool(name=None) with 128 open slots and 1 task instances in queue
[2018-12-15 18:59:13,656] {jobs.py:1171} INFO - DAG hello_world has 0/16 running and queued tasks
[2018-12-15 18:59:13,656] {jobs.py:1209} INFO - Setting the follow tasks to queued state:
     <TaskInstance: hello_world.wait 2018-12-15 18:50:00+00:00 [scheduled]>
[2018-12-15 18:59:13,698] {jobs.py:1293} INFO - Setting the following 1 tasks to queued state:
     <TaskInstance: hello_world.wait 2018-12-15 18:50:00+00:00 [queued]>
[2018-12-15 18:59:13,699] {jobs.py:1335} INFO - Sending ('hello_world', 'wait', datetime.datetime(2018, 12, 15, 18, 50, tzinfo=<Timezone [UTC]>), 1) to executor with priority 2 and queue default
[2018-12-15 18:59:13,701] {base_executor.py:56} INFO - Adding to queue: airflow run hello_world wait 2018-12-15T18:50:00+00:00 --local -sd /Users/ash/airflow/dags/foo.py
[2018-12-15 18:59:13,742] {sequential_executor.py:45} INFO - Executing command: airflow run hello_world wait 2018-12-15T18:50:00+00:00 --local -sd /Users/ash/airflow/dags/foo.py
[2018-12-15 18:59:15,558] {__init__.py:51} INFO - Using executor SequentialExecutor
[2018-12-15 18:59:15,755] {models.py:273} INFO - Filling up the DagBag from /Users/ash/airflow/dags/foo.py
[2018-12-15 18:59:15,833] {cli.py:530} INFO - Running <TaskInstance: hello_world.wait 2018-12-15T18:50:00+00:00 [queued]> on host themisto.localdomain
[2018-12-15 18:59:21,427] {jobs.py:1439} INFO - Executor reports hello_world.wait execution_date=2018-12-15 18:50:00+00:00 as success for try_number 1

The dag I was testing this with as:

from airflow import DAG
import datetime
import airflow
from airflow.operators.dummy_operator import DummyOperator
from airflow.sensors.time_sensor import TimeSensor
start = (
airflow.utils.dates.days_ago(2)
)
dag2 = DAG('hello_world',
schedule_interval='*/5 * * * *',
start_date=start,
catchup=False)
with dag2:
(
TimeSensor(
task_id='wait',
target_time=datetime.time(20),
mode='reschedule',
) >>
DummyOperator(task_id='dummy')
)

@seelmann Are you able to work on fixing these two issues?

@ashb I can work on those during the holidays.
Regarding 1 (the "None" state): I agree it's not optimal that there is not indication about what's going on. Do you want to have a new dedicated state (in state.py)? Or is is just about the visualization in the tree view (and other views make also sense IMHO)? Adding a new state was discussed but decided against. Changing the visualization in views should be possible, in Gantt view it's already done.
Regarding 2 (the success log): What do you expect should be logged instead of success? If I look into sequential_executor.py it always returns sets success or failed, depending if the command execution was successful or not. Should we make all executors aware of the reschedule state?
PS: We run 1.10.0 with this patch successfully in production since November :)

@seelmann Thanks, that would be ace! This feature is very nice.
I'm mostly concerned with the visualisation on the Graph, Tree, and Task Instance Detail pages. Gannt isn't one I look at very often, but looking at it now I don't see anything indicating retries? (That or I cherry-picked it on to 1-10-test wrong) Could you give some screenshots?
As for the log? Hmmm, There are only a few executors so updating them to be aware of it and log not log something incorrect shouldn't be too much work?

Screenshots are attached at the Jira.
The Gantt view should show a white bar when it's rescheduled in thus in "None" state, see https://issues.apache.org/jira/browse/AIRFLOW-2747#comment-16616842
In a previous version all rescheduled executions were visible, but that ended up in too many small bars: https://issues.apache.org/jira/browse/AIRFLOW-2747#comment-16541539

Another thing that should be improved: The start date, end date, and duration in the tooltip when hovering over that task only shows the values of the last run, should be from first to last.

@ashb @seelmann To simplify things, I'm all in for setting the re-schedule method as the default scheduling method for sensors, and get rid of the blocking sensors. This will also enable the use of sensors on the SequentialExecutor (apart from the reschedule state). But more important from my perspective, it will greatly simplify the logic since we don't have to maintain two branches of the sensor execution (legacy, and re-scheduling). Curious what you guys think.

@Fokko If the intention is to cherry-pick into 1.10.2 I'd not make it the default yet. It would change the behaviour of sensors, e.g. short poke intervals below 5 seconds are no longer possible and likely increase scheduler load. For 2.0 I think it's ok, let's just add a note to UPDATING.md.

I fully agree, @seelmann. My suggestion was to make it default for Airflow 2.0

I created several sub-tasks in https://issues.apache.org/jira/browse/AIRFLOW-2747

Thanks @seelmann


Fixed VCPKG_LOAD_VCVARS_ENV option to be used to disable vcvars loading

Fixes https://github.com/microsoft/vcpkg/issues/40480.
When setting VCPKG_LOAD_VCVARS_ENV=OFF in a triplet, vcpkg/CMake complains that it can't find a compiler.

Without this fix, currently one need
set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE ${VCPKG_ROOT_DIR}/scripts/toolchains/windows.cmake)

in a Windows triplet to disable vcvars loading.