[SPARK-24748][SS] Support for reporting custom metrics via StreamingQuery Progress

What changes were proposed in this pull request?
Currently the Structured Streaming sources and sinks does not have a way to report custom metrics. Providing an option to report custom metrics and making it available via Streaming Query progress can enable sources and sinks to report custom progress information (E.g. the lag metrics for Kafka source).
Similar metrics can be reported for Sinks as well, but would like to get initial feedback before proceeding further.
How was this patch tested?
New and existing unit tests.
Please review http://spark.apache.org/contributing.html before opening a pull request.

@tdas @jose-torres  @HeartSaVioR

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Test build #92659 has started for PR 21721 at commit b7b2c3b.

Test build #92659 has finished for PR 21721 at commit b7b2c3b.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92659/
Test FAILed.

Merged build finished. Test FAILed.

I think that SS'metrics should be report to Spark UI. Then user can lean application operations!

Though I haven't take a look yet, I would like to see this feature (mentioned from https://github.com/apache/spark/pull/21622#issuecomment-399677099) and happy to see this being implemented!
While I love the feature, I agree with @jose-torres that it is going to be a new public API (part of Datasource V2) so worth to discuss regarding the API itself before having specific implementation.

Test build #92694 has started for PR 21721 at commit 9db1a95.

@jose-torres @HeartSaVioR , Addressed the initial comments. Will add Writer support for custom metrics and add MemorySink as an example. I am ok to move out Kafka custom metrics into a separate PR but the lag metrics is valuable IMO.

Looks fine to me with a MemorySink example. I don't think a formal discussion is super necessary - the major advantage of the mixin model is to let us add things like this without impacting the broader API.

Test build #92870 has started for PR 21721 at commit 47d802b.

Test build #92870 has finished for PR 21721 at commit 47d802b.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92870/
Test FAILed.

Test build #92874 has started for PR 21721 at commit bca054f.

@jose-torres I have removed the Kafka lag metrics out of this PR and added writer metrics and the number of rows in the memory sink  as an example.

Test build #92874 has finished for PR 21721 at commit bca054f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92874/
Test FAILed.

Test build #92876 has started for PR 21721 at commit 5e732cb.

Test build #92876 has finished for PR 21721 at commit 5e732cb.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92876/
Test FAILed.

@jose-torres, addressed initial comments.
@tdas, can you also take a look when possible ?

Test build #93793 has finished for PR 21721 at commit 04c5b6f.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93793/
Test FAILed.

@HeartSaVioR thanks for taking time to review. Addressed the comments, can you take a look again?
Regarding the mixin interface, would like to take feedback from others.
@jose-torres @tdas @zsxwing could you take a look at the patch and also comment on https://github.com/apache/spark/pull/21721#discussion_r206241038 ?

Test build #93813 has started for PR 21721 at commit ef65d51.

Test build #93813 has finished for PR 21721 at commit ef65d51.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93813/
Test FAILed.

@HyukjinKwon , have addressed the comments and modified SourceProgress and SinkProgress to take String instead of JValue so that this can be easily used from Java. Regarding the default value in the ctor, I am not sure if its an issue because the object is mostly read only and would be an issue only if the user tries to construct it from Java. I have added overloaded ctors anyways. Please take a look.

Test build #93907 has started for PR 21721 at commit 3e5d9d8.

Test build #93907 has finished for PR 21721 at commit 3e5d9d8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93907/
Test FAILed.

Looks fine otherwise to me too if the test passes

Test build #94008 has started for PR 21721 at commit 1775c2a.

Test build #94008 has finished for PR 21721 at commit 1775c2a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94008/
Test FAILed.

retest this please

Test build #94038 has started for PR 21721 at commit 1775c2a.

The tests keeps failing and looks unrelated. @HyukjinKwon Let me know if you think theres something I should look into.

Test build #94038 has finished for PR 21721 at commit 1775c2a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94038/
Test FAILed.

retest this please

Yup, looks the resent Kafka upgrade has an issue.

Test build #94097 has started for PR 21721 at commit 1775c2a.

Test build #94097 has finished for PR 21721 at commit 1775c2a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94097/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #94121 has started for PR 21721 at commit 1775c2a.

Test build #94121 has finished for PR 21721 at commit 1775c2a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94121/
Test FAILed.

retest this please

Test build #94147 has started for PR 21721 at commit 1775c2a.

Test build #94147 has finished for PR 21721 at commit 1775c2a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94147/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #94200 has started for PR 21721 at commit 1775c2a.

Test build #94200 has finished for PR 21721 at commit 1775c2a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94200/
Test FAILed.

Test build #94320 has started for PR 21721 at commit 01e8451.

@HyukjinKwon , the master code changed and I had to rebase and fix issues. Can you take it forward ? There seems to be unrelated test failures in Kafka 0.10 integration suite.

Test build #94320 has finished for PR 21721 at commit 01e8451.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94320/
Test PASSed.

Merged to master.

Since I'm continuously working on data source v2 API, this gets my attention. Do we have a story for the metrics in data source v2 streaming API? It's weird to me that we add public APIs that only work for micro-batch.
For streaming API, the abstraction is that, we have a logical scan for a streaming source in a query(to keep query specific states like offsets), and a physical scan to do the actual job, for each micro-batch or for the entire continuous query(if needsReconfigure is false).
Where does the metrics fit in with the abstraction? It's ok that it only works for micro-batch now, but we must have a clear plan that we can and will make it work for continuous.
cc @tdas @zsxwing @rdblue

Also ping @rxin @marmbrus . Another related PR is https://github.com/apache/spark/pull/21819.

For continuous queries, the progress could still be reported by posting QueryProgressEvent to the listener for each epoch (instead of micro-batch). The StreamingQueryProgress also could mostly be the same.
I am not clear on how the continuous query would get the metric updates. We may need some mechanism to post metric updates while the query continues to run. Right now the SQL metrics relies on accumulator and the accumulators might not be updated unless the task completes.

It’s better to not release such APIs without thinking about how to support continuous queries, since it may need to change APIs, which should be avoided if possible. I propose to revert this PR. It would be great that there is a design doc for streaming source metrics APIs to discuss how to support all modes before committing.

@zsxwing @gatorsmile , this PR does not add new APIs as such. It builds on the existing StreamingQueryProgress and adds custom metrics to it. StreamingQueryProgress as such is not reported for continuous mode. When its reported this would be part of that. Are you proposing to not report StreamingQueryProgress at all for micro-batch until things are figured out for continuous mode.? Otherwise I don't see how adding this would break things.
cc @jose-torres

The CustomMetrics are traits which can be mixed in if necessary. (see https://github.com/apache/spark/pull/21721#issuecomment-403878383) and does not affect any other API as such. When query progress is supported for continuous mode, changes can be made if necessary. I can further investigate on how to report query progress for continuous mode.

@arunmahadevan yeah, it's better to figure out the solution for continuous mode as well. As you mentioned, the current SQL metrics are not updated unless the task completes, so we may need to add new APIs to support reporting metrics for continuous mode. It would be great that there will be a consistent API for all modes. Let's step back and think about it.

@zsxwing @gatorsmile @cloud-fan
As you all know, IMHO, the thing is that continuous mode doesn't support StreamingQueryProgress for now. We allowed these incomplete state because continuous mode is in experiment with filing follow-up issue SPARK-23887, and the issue has been open for 3 months. If we would like to take it seriously, SPARK-23887 is better to be marked as a "Blocker" and we should deal with the issue sooner than later. Otherwise it doesn't sound good if continuous mode blocks others.

Unfortunate thing is that continuous mode allows different epochs between partitions, hence query progress just doesn't fit so hard to address SPARK-23887 for now. My 2 cents is continuous mode also should sync epoch in query so all partitions deal with same epoch. When we deal with aggregation it should be essential.

There are many unknowns to be figured out for continuous mode. Though the way to capture the metrics would be different for continuous execution, the interface of whats reported is not expected to change. Given that we already report progress for micro-batch and as a user of spark the changes in the patch are quite useful to report custom metrics for what works right now and since it does not impact other parts of DataSourceV2 apis (only the sources that wants to report custom metrics would add the traits) IMO, we can keep this and continue to investigate in a time-bound manner of how to capture metrics for continuous mode.

Can we remove and replace this to another one when we are clear on how to deal with continuous mode?
I merged this after having sufficient talks with @HeartSaVioR. The JIRA (https://issues.apache.org/jira/browse/SPARK-23887) was by @jose-torres who left a sign-off here.
BTW, I hope those discussions should better be made in those PRs or JIRAs first. Is this just because some concerns were found later? it looked just good and many positive feedbacks but just looks held off for no reasons from my point of view.

@HyukjinKwon yes we can mark it unstable. Like I mentioned multiple times in previous comments the traits added here like CustomMetrics, SupportsCustomReaderMetrics etc have nothing specific to micro batch or continuous mode and un-affected when we finally start reporting progress for continuous mode. The way to collect and report metrics in continuous mode needs to be figured out and I think should be discussed in respective JIRAs.

Note that, data source v2 API is not stable yet and we may even change the abstraction of the APIs. The design of custom metrics may affect the design of the streaming source APIs.
I had a hard time to figure out the life cycle of custom metrics. It seems like its life cycle should be bound to an epoch, but unfortunately we don't have such an interface in continuous streaming to represent an epoch. Is it possible that we may end up with 2 sets of custom metrics APIs for micro-batch and continuous? The documentation added in this PR is not clear about this.

I actually thought those all of them are part of DataSource V2. Why are we fine with changing those interfaces but not okay with this one and we consider reverting it?
Other things should be clarified if there are some concerns, yea of course. In this case, switching it to Unstable looks alleviating the concerns listed here enough.

Stuff like this merits api discussions. Not just implementation changes ...

@rxin its for streaming sources and sinks as explained in the doc
It had to be shared between classes in reader.streaming and writer.streaming, so was added in the parent package (similar to other streaming specific classes that exists here like StreamingWriteSupportProvider.java
MicroBatchReadSupportProvider.java)
we could move all of it to a streaming package.

@arunmahadevan, feel free to pick up the commits in my PR in your followup if they have to be changed. I will close mine.

I created a follow up PR to move CustomMetrics (and a few other streaming specific interfaces in that package) to 'streaming' and mark the interfaces as Unstable here - https://github.com/apache/spark/pull/22299

My 2 cents, the root reason is the lifecycle of reporting query progress is tied to finishTrigger and we read updated metrics from executed plan which continuous mode doesn't have both finishTrigger as well as finished plan to be executed.
I'm not aware of how/when updated information of nodes of physical plan are transmitted from executor to the driver, but we should avoid using executed plan as a source to read information, and find alternative to be compatible between micro-batch and continuous mode. It doesn't apply only metrics but also watermarks.
I'm not sure it is viable, but It could be via RPC or whatever once we can aggregate the information from driver. Then each operators can send information on driver directly and driver can aggregate them and utilize once a batch or an epoch is finished.

It seems like its life cycle should be bound to an epoch, but unfortunately we don't have such an interface in continuous streaming to represent an epoch. Is it possible that we may end up with 2 sets of custom metrics APIs for micro-batch and continuous?

@cloud-fan we could still report progress at the end of each epoch (e.g. here and via the EpochCordinator). There need not be separate interfaces for the progress or the custom metrics, just the mechanisms could be different.

I skimmed about how AccumulatorV2 works, and looks like the values in a task are reported along with CompletionEvent which is triggered when a task ends. Then in continuous mode driver even doesn't have updated metrics. It should not couple with lifecycle of task.

Can someone write a design doc for the metrics support? I think this is an important feature for data source v2 and we need to be careful here. The design doc should explain how custom metrics fit in the abstraction of data source v2 API, how the metrics API would look like for batch, micro-batch and continuous (I feel metrics is also important for batch sources), and how the sources report metrics physically (via task complete event? via heartbeat? via RPC?).
@rxin just sent an email to the dev list about the data source v2 API abstraction, it would be great if you guys can kick it and talk about the metrics support.
It's very likely that the custom metrics API would be replaced by something totally different after we finish the design. I don't think we should rush into something that works but not well designed.

So .. @cloud-fan, and @rxin, how about this:

Mark this as Unstable for now - which means we likely happen to change this if we face a design issue.
Write a design doc to cover up continuous and batch and find out better way (or check if the current way works).

I will take a look at this tomorrow, since I’m already looking at data
source apis myself. Can provide opinion after another look on whether we
should keep it unstable or revert.
On Fri, Aug 31, 2018 at 12:07 AM Hyukjin Kwon<EMAIL_ADDRESS>wrote:

So .. @cloud-fan https://github.com/cloud-fan, and @rxin
https://github.com/rxin, how about this:

Mark this as Unstable for now - which means we likely happen to
change this if we face a design issue.
Write a design doc to cover up continuous and batch and find out
better way (or check if the current way works).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/21721#issuecomment-417572882, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AATvPIgkFv4_bAQyopGaxBUfF6Zt6Dvvks5uWODMgaJpZM4VExgx
.

Thank you @rxin for your time and efforts.

If batch query also leverages AccumulatorV2 for metrics, IMHO it might not need to redesign metrics API start from scratch. For batch and micro-batch the metrics API work without any concerns (it is getting requests for improvement though), and for continuous mode the metrics just don't work because task never finishes.
The change in metrics affects both query status as well as SQL tab in UI. I haven't concerned too deeply with metrics on continuous mode so not sure about current state of UI and the ideal shape of UI, so will spend time to play with. My 2 cents, once we have existing metrics work well, we could find out some ways to let current metrics work well with continuous mode, to not break other things as well.
One thing I would like to ask to ourselves is, would we treat epoch id as batch id? For checkpointing we already did it, and in some streaming framework they represent stream between epochs as logical batch which makes sense to me. If we deal with watermark we are likely to update watermark per epoch, as well as dealing with state, and if my understanding is correct epoch id looks like just an alias of batch id.

I spent more hours to take a look at how SQL UI can update the metrics information before task ends, and now I guess I may understand what was the concern from @cloud-fan here.
This is different from how we allow custom metrics in StateStore. Every SQL metrics even custom metrics in StateStore are accumulators, which are taken care of executor heartbeat (Honestly I didn't notice it. My bad) and UI updates these information. Custom metrics in StateStore is only updated when state operation is going to be finished for each partition, but they are exposed to SQL UI anyway and gets updated dynamically in the UI (I meant the values can be updated even for running batch).
With StreamingQueryProgress, we are also exposing information which are only calculated when they're needed, and now it is when finishTrigger is called, so mostly batch ends. Custom metrics in this patch placed here: they're additional information for StreamingQueryProgress, hence intentional to be updated per batch. They're not actually SQL metrics, but its name would lead someone thinking why it doesn't follow SQL metrics. Maybe the name matters?
So there're two desires to add custom information:

metrics to be updated in every heartbeat: they will be exposed to SQL UI, and also can be collected and added to StreamingQueryProgress like custom metrics in StateStore.
information to be updated in each batch: they will be exposed to only StreamingQueryProgress.

And the target of the patch is latter.
But we know 2 is only applied to micro-batch, and current StreamingQueryProgress is not suitable for continuous mode because of these reasons: 1. Unless we stop processing or snapshot metrics once epoch ends, metrics can't be correct for specific epoch. 2. Showing the information for latest epoch (which all partitions finished) no longer represents the most recent. 3. Some metrics are expected to be reset per batch, whereas it doesn't happen in continuous mode. If we reset metrics per epoch, metrics in SQL tab in UI will be really looking odd (because it just shows current state of metrics, not bound to epoch).
So IMHO it's likely that StreamingQueryProgress will not be available for continuous mode even afterwards (not only for custom metrics), and we may want to rely on running SQL metrics. That's actually what other streaming frameworks are providing metrics as of now, but they are also showing these metrics as aggregated values in time window or even time-series. Spark doesn't need to have such feature for batch and micro-batch, but in continuous mode, without that feature these SQL metrics will be really hard to see after long run (say 1 month). That's the hard thing when we want to make modes being transparent: the requirements of metrics for batch/micro-batch and continuous mode are just different, and metrics may not be only issue.

Given the uncertainty about how this works across batch, streaming, and CP, and given we are still flushing out the main APIs, I think we should revert this, and revisit when the main APIs are done.
In general for API design, it is best to flush out the big skeletons first, and then work on filling the gaps. Think about building a house. You build the frame, and put the stud in, the walls, and then do the final finish. You don't start by putting plumbing fixtures in one room when you are still moving the main plumbing lines.

BTW I think this is probably SPIP-worthy. At the very least we should write a design doc on this, similar to the other docs for dsv2 sub-components. We should really think about whether it'd be possible to unify the three modes (batch, microbatch streaming, CP).

FYI, I submitted #22334 to revert #21819 and #21721.

Is there a JIRA which closure would indicate DataSource API V2 is done and this PR would then go futher? Is that SPARK-25531 or SPARK-22386?
I am asking as my PR #22143 depends on this PR.


postgresql missing in list

Missing postgresql in list.
https://github.com/Senzing/docker-init-container/blob/cb78f007f2bfe5b37a24937d8ba13f8bf0842390/init-container.py#L1272
Probably better to construct logic as elif structure as seen at https://github.com/Senzing/docker-init-container/blob/cb78f007f2bfe5b37a24937d8ba13f8bf0842390/init-container.py#L563-L585

Pull Requests:

https://github.com/Senzing/docker-init-container/pull/49


bin stubs not created for a certain repo

pnpm version: 0.31.0
Code to reproduce the issue:
git clone http://github.com/mapbox/mapbox-gl-js
cd mapbox-gl-js
pnpm install
ls node_modules/.bin

Expected behavior:
A list of bin stubs from dependencies/devDependencies.
Actual behavior:
ls: node_modules/.bin: No such file or directory

It works fine in other repos I tested, but for some reason fails for this one. On a CI server (Circle), it's even stranger — some stubs are created and some others (like eslint) are not.
Additional information:

node -v prints: v6.5.0
OS X El Capitan

cc @rstacruz

can verify it

can verify it

related #96 and reasoning in #58

not sure what to do

@iamstarkov yeah, the difference here is that it doesn't create bin stubs for direct dependencies (not deep ones).

I have noticed this issue as well today. I think there is an uncatched promise error during installation... I am looking into it now

no, I don't see not catched promise errors. Seems like it was broken in v0.31.0. v0.30.0 creates the binstubs

this commit caused the issue https://github.com/rstacruz/pnpm/commit/1c92146e424164afdec57f9d483ebe69470fcf90

@zkochan but why? linkBins just moved from one place to another

I really don't know, but when I revert that commit, it works...

When I return that line, it works https://github.com/rstacruz/pnpm/commit/1c92146e424164afdec57f9d483ebe69470fcf90#diff-f6618e1cc731d58106a806b7679a7616L106

I've found the issue.... Not in the commit I was thinking... reverting that commit was just fixing one of the issues caused by tarball dependencies not working. The fix is easy, I think I will create a PR soon

Fixed in v0.31.2

@zkochan thank you very much for such a quick fix!

нема за що :-)
you're welcome


gem install bundler

http://bundler.io/

Bundler provides a consistent environment for Ruby projects by tracking and installing the exact gems and versions that are needed.
Bundler is an exit from dependency hell, and ensures that the gems you need are present in development, staging, and production. Starting work on a project is as simple as bundle install.


Issue #4 - Paper Files Location & AiC

Philippe Michon Jun 21st at 8:38 AM
4. For AiC, we need to manage paper files location. We could manage it with a complete CIDOC CRM path something like: "We have some documents about this person and they are located there".  It's quite a lot more nodes for not that much semantic. So we seem to prefer a simpler approach like: e21->p1->e33->p2->e55 (Location of the paper files). I know that Stephen modeled the complex path so we could vet with the team what they prefer.
Stephen Hart  2 months ago
I've modeled it in the pur CIDOC cru, with the path:
E39Actor->P70iisDocumentedBy->E31Document->E22Man-madeObject->P54hasCurrentPermanentLocation->E53Place
But I'm not convinced by the P1 is identified by, as the files concerning the actor is not an identifier.
The property P62 Depicts has domain E24 Physical man-made thing and range E1 Entity, but it is only for images, not texts.
Stephen Hart  2 months ago
Therefore I am not sure what's the easiest solution. A simple P3 has note can not be tagged with an E55 Type to identify it as the file location, so it's not a good solution either
Philippe Michon  2 months ago
I made a mistake in my first reply I meant e21->p67i->e33->p2->e55 (file location). But I think Stephen's proposal is not so complicated and valid :slightly_smiling_face:

This have been implemented in the v1.5 of the TM


Implemented selectable armor divisor in the apply damage dialog

@mjeffw we talked about this a loooong time ago. Appears to be working, except the 100 and infinite levels of armor divisor break the layout.

Umm, damagecalculator.js line 24 defines
// -1 means 'Ignores DR'
const armorDivisorSteps = [-1, 100, 5, 3, 2, 1]

This is the division progression the existing DR hardening handling code uses. So that's where -1 as infinite comes from. I also noticed that divisor 10 is missing, but I didn't want to do anything except to introduce this new feature initially. If you're otherwise fine with this PR I'll gladly extend this to include armor divisor 10.


libmesode: Add a patch from upstream to fix SSL verification

Motivation for this change
SSL certificate verification in profanity (libmesode consumer) does not work because NixOS ships CA certificates in a file. The upstream profanity-im/libmesode@532ed1e9d3e71e5bea0752e03dbacd4139d750d1 fixes this.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

What does one need to get this merged ? Just curious, as I'm maintaining an overlay in my local configuration for this. :)
Thanks!


Update Netcentric private documentation

Once repo is public and package published in NPM update the Netcentric documentation to reflect the new links to the published package
https://projects.netcentric.biz/wiki/display/FRONTEND/Stylelint+Recommended+Configuration

Done


Adding sourcemaps for coverage reports

html coverage reports currently show the compiled es5
Would be nice to either:

Add sourcemaps to istanbul reports
Switch to nyc and compile on the fly

(2 is maybe undesirable if you want the assurance of testing against the actual dist code. fwiw though, something like this patch does the trick.)

If you'd like to submit a PR switching us to nyc, I think that'd be great.

https://github.com/airbnb/hypernova/pull/63


Question: Where to investigate for the "Dead Night" task given by Sidor?

I am posting this here as there's no ModDB page for this.
For the Dead Night task obtained from Sidor, I ran across roughly the entire Darkscape map at night (10 PM - 3 AM) and still can't figure out what/where to look for. Could you please help me with this?
P.S. I absolutely LOVE these new tasks! House of Horrors is the crown jewel of this addon.

Hey there
You're not the only person that had issues with it
Here's where the task script fires (cursor position)

I didn't want to add a quest marker there to not make it too obvious and keep the element of surprise
I might need to think of a way to attract the player to this place somehow... maybe add some noises or something

Hey there You're not the only person that had issues with it Here's where the task script fires (cursor position)
I didn't want to add a quest marker there to not make it too obvious and keep the element of surprise I might need to think of a way to attract the player to this place somehow... maybe add some noises or something

Thanks a lot! Yeah, I think adding noises is a good idea. House of Horrors executes this perfectly.


Add durable logging support

When sending in batches asynchronously, it's possible some events may be lost if the process were to fail before the batch was sent. Adding durable logging support would persist the log events to disk (or other reliable offline/shared store) so that lost events could be resubmitted when the process recovers.

Example of sink with durable logging https://github.com/serilog/serilog-sinks-loggly/blob/dev/src/Serilog.Sinks.Loggly/Sinks/Loggly/DurableLogglySink.cs


ADR: Host Packages

What this PR does / why we need it:
Proposal

I fully approve this 👍
But I'm also concerned how this will go in practice. Mainly airgap installations, where they will not have mirrors. I'm a little bit afraid that this will break those installs, and the end-customer will look at us to resolve. But I could also be wrong on this. On the other hand: If your environment is airgapped, I do expect the customer be at that enterprise level to do have a mirror in place 🤞

I don't imagine any airgap installations will be able to install the OS without a package repository accessible by the server.
I think I would take this further still - surely we do not expect to support the OS packages themselves even for packages like docker and containerd. The supported versions for the customer's platform should be in the repo mirror already.  If the package and it's deps are in a supported repo (e.g. not Universe if it's Ubuntu) then we don't need to supply the package at all, merely make sure that the supported combinations are available for the target OS.  That would narrow down the list of packages we absolutely need to include in the bundle.

Packages like docker and kubernetes are available via 3rd party repositories
https://docs.docker.com/engine/install/ubuntu/
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl


Traning and testing losses return 'nan' during training

Hi all,
First of all thank you for your amazing job on ES-RNN pytorch implementation.
I have been able to run the M4 scripts, but I encountered troubles trying to run ESRNN architecture on an another data source.
I am trying to apply this model on the M5 competition. The input is the following tables:

I define the model as following (taking some parameters from the initial config dict) and use previously defined tables to train it:

But I observe that the network does not seem to learn, because of 'nan' returned during training. Is there something I missed ?

Hi,
Thanks for using our implementation. For what I can see in the data there are multiple zeros in the target variable. The model will return nan losses if the target variable of any serie has zero in the first ds (since the model normalizes the series by previous values). Can you try adding 1 (or any scalar) to all the target variable for all ds? You can then substract the scalar to the predictions.

Hi !
Thanks a lot, it works !

Hi there,
I am running into a similar problem. While I do get a value for training loss and testing loss, I get a nan for my evaluation metrics: sMAPE, MASE and OWA, when I am fitting the model.

Infered frequency: W-SUN
=============== Training ESRNN  ===============
========= Epoch 0 finished =========
Training time: 12.04543
Training loss (50 prc): 0.49441
Testing loss  (50 prc): 0.02670
OWA: nan
SMAPE: nan
MASE: nan

I have checked and there are no zeroes in the first ds. What could be causing the problem?


The getter 'scrollExtent' was called on null.

widget层级关系:
NestedScrollView ---> .Container .--> TabBarView --> SmartRefresher
出现切换到当前的tab时滑不动(只能上下滑动一点距离,超过屏幕的条目没办法滑出来)
错误日志:
flutter: ══╡ EXCEPTION CAUGHT BY GESTURE ╞═══════════════════════════════════════════════════════════════════
flutter: The following NoSuchMethodError was thrown while handling a gesture:
flutter: The getter 'scrollExtent' was called on null.
flutter: Receiver: null
flutter: Tried calling: scrollExtent
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #0      Object.noSuchMethod (dart:core-patch/object_patch.dart:50:5)
flutter: #1      RefreshPhysics.applyBoundaryConditions (package:pull_to_refresh/src/internals/refresh_physics.dart:216:43)
flutter: #2      ScrollPosition.applyBoundaryConditions (package:flutter/src/widgets/scroll_position.dart:373:35)
flutter: #3      ScrollPosition.setPixels (package:flutter/src/widgets/scroll_position.dart:202:33)
flutter: #4      ScrollPositionWithSingleContext.setPixels (package:flutter/src/widgets/scroll_position_with_single_context.dart:83:18)
flutter: #5      ScrollPositionWithSingleContext.applyUserOffset (package:flutter/src/widgets/scroll_position_with_single_context.dart:126:5)
flutter: #6      ScrollDragController.update (package:flutter/src/widgets/scroll_activity.dart:372:14)
flutter: #7      ScrollableState._handleDragUpdate (package:flutter/src/widgets/scrollable.dart:498:12)
flutter: #8      DragGestureRecognizer._checkUpdate. (package:flutter/src/gestures/monodrag.dart:383:46)
flutter: #9      GestureRecognizer.invokeCallback (package:flutter/src/gestures/recognizer.dart:182:24)
flutter: #10     DragGestureRecognizer._checkUpdate (package:flutter/src/gestures/monodrag.dart:383:7)
flutter: #11     DragGestureRecognizer.handleEvent (package:flutter/src/gestures/monodrag.dart:253:9)
flutter: #12     PointerRouter._dispatch (package:flutter/src/gestures/pointer_router.dart:75:13)
flutter: #13     PointerRouter.route (package:flutter/src/gestures/pointer_router.dart:102:11)
flutter: #14     _WidgetsFlutterBinding&BindingBase&GestureBinding.handleEvent (package:flutter/src/gestures/binding.dart:218:19)
flutter: #15     _WidgetsFlutterBinding&BindingBase&GestureBinding.dispatchEvent (package:flutter/src/gestures/binding.dart:198:22)
flutter: #16     _WidgetsFlutterBinding&BindingBase&GestureBinding._handlePointerEvent (package:flutter/src/gestures/binding.dart:156:7)
flutter: #17     _WidgetsFlutterBinding&BindingBase&GestureBinding._flushPointerEventQueue (package:flutter/src/gestures/binding.dart:102:7)
flutter: #18     _WidgetsFlutterBinding&BindingBase&GestureBinding._handlePointerDataPacket (package:flutter/src/gestures/binding.dart:86:7)
flutter: #22     _invoke1 (dart:ui/hooks.dart:250:10)
flutter: #23     _dispatchPointerDataPacket (dart:ui/hooks.dart:159:5)
flutter: (elided 3 frames from package dart:async)
flutter:
flutter: Handler: "onUpdate"
flutter: Recognizer:
flutter:   VerticalDragGestureRecognizer#3ba64
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════

widget层级关系:
NestedScrollView ---> .Container .--> TabBarView --> SmartRefresher
出现切换到当前的tab时滑不动(只能上下滑动一点距离,超过屏幕的条目没办法滑出来)
错误日志:
flutter: ══╡ EXCEPTION CAUGHT BY GESTURE ╞═══════════════════════════════════════════════════════════════════
flutter: The following NoSuchMethodError was thrown while handling a gesture:
flutter: The getter 'scrollExtent' was called on null.
flutter: Receiver: null
flutter: Tried calling: scrollExtent
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #0 Object.noSuchMethod (dart:core-patch/object_patch.dart:50:5)
flutter: #1 RefreshPhysics.applyBoundaryConditions (package:pull_to_refresh/src/internals/refresh_physics.dart:216:43)
flutter: #2 ScrollPosition.applyBoundaryConditions (package:flutter/src/widgets/scroll_position.dart:373:35)
flutter: #3 ScrollPosition.setPixels (package:flutter/src/widgets/scroll_position.dart:202:33)
flutter: #4 ScrollPositionWithSingleContext.setPixels (package:flutter/src/widgets/scroll_position_with_single_context.dart:83:18)
flutter: #5 ScrollPositionWithSingleContext.applyUserOffset (package:flutter/src/widgets/scroll_position_with_single_context.dart:126:5)
flutter: #6 ScrollDragController.update (package:flutter/src/widgets/scroll_activity.dart:372:14)
flutter: #7 ScrollableState._handleDragUpdate (package:flutter/src/widgets/scrollable.dart:498:12)
flutter: #8 DragGestureRecognizer._checkUpdate. (package:flutter/src/gestures/monodrag.dart:383:46)
flutter: #9 GestureRecognizer.invokeCallback (package:flutter/src/gestures/recognizer.dart:182:24)
flutter: #10 DragGestureRecognizer._checkUpdate (package:flutter/src/gestures/monodrag.dart:383:7)
flutter: #11 DragGestureRecognizer.handleEvent (package:flutter/src/gestures/monodrag.dart:253:9)
flutter: #12 PointerRouter._dispatch (package:flutter/src/gestures/pointer_router.dart:75:13)
flutter: #13 PointerRouter.route (package:flutter/src/gestures/pointer_router.dart:102:11)
flutter: #14 _WidgetsFlutterBinding&BindingBase&GestureBinding.handleEvent (package:flutter/src/gestures/binding.dart:218:19)
flutter: #15 _WidgetsFlutterBinding&BindingBase&GestureBinding.dispatchEvent (package:flutter/src/gestures/binding.dart:198:22)
flutter: #16 _WidgetsFlutterBinding&BindingBase&GestureBinding._handlePointerEvent (package:flutter/src/gestures/binding.dart:156:7)
flutter: #17 _WidgetsFlutterBinding&BindingBase&GestureBinding._flushPointerEventQueue (package:flutter/src/gestures/binding.dart:102:7)
flutter: #18 _WidgetsFlutterBinding&BindingBase&GestureBinding._handlePointerDataPacket (package:flutter/src/gestures/binding.dart:86:7)
flutter: #22 _invoke1 (dart:ui/hooks.dart:250:10)
flutter: #23 _dispatchPointerDataPacket (dart:ui/hooks.dart:159:5)
flutter: (elided 3 frames from package dart:async)
flutter:
flutter: Handler: "onUpdate"
flutter: Recognizer:
flutter: VerticalDragGestureRecognizer#3ba64
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════

这个bug是升级到 1.5.3 之后出现的,之前版本不存在

问题重现不了,起码附个样例代码把

Got no reply long time. Close this issue


Patch 1

Added an administrator prompt for meeting summary/department minutes, and department meeting agenda setting.

@microsoft-github-policy-service agree


Documentation update?

Hi! I was trying to use the tltorch.factorized_layers.FactorizedLinear, but it seems like the parameters listed on the documentation page http://tensorly.org/torch/dev/modules/generated/tltorch.factorized_layers.FactorizedLinear.html#tltorch.factorized_layers.FactorizedLinear are different from the parameters the model actually takes in now, and I'm not sure how the old parameters are mapped to the new parameter. In particular, I wonder how we can specify the shape of the tensor with the current model parameters in_tensorized_features and out_tensorized_features, and it would be great if you could update the documentation to match the current definition of the model.
Thank you very much!

Thanks, I've pushed an update. Let me know if anything is unclear.


Initialize in parent does not cause child component update

I have a component with a child in a slot with input, the parent runs (slow) initialize which gets some data (intended to be the input to the child component), however, the child does not get updated with this data. Maybe I am using some feature wrong or maybe it's a bug? I am on halogen/v5.0.0-rc.4.
How to reproduce: I made https://github.com/OndrejSlamecka/reproduce-halogen-bug based on the components example and attempting to make the reproduction obvious with this commit. You can build the contents with
spago bundle-app -t dist/example.js and open in a browser with file:///.../dist/index.html.
I then added a spy function in the next commit, showing "render slot", "render button", (wait for load), "render slot" but then no "render button" as I would hope.

Initialize runs after the component has been rendered for the first time, so the child component will first receive "not loaded".
A changing input value in a component is dealt with by a receiver. In here you have added the plumbing for the value to be set on the child in init, but there's no receiver hooked up, so that's why it's not being updated after the 1s delay.

I'd also recommend that you H.fork the async action in the Container initializer, as until the initializer for a component completes all other queries, actions, and messages are queued - meaning it will appear on the page, but if the user tries to interact with it, nothing will happen during that async period, and then once it completes all the actions that occurred in the mean time will be triggered at once.

Take a look at this example for child components that receive values from the parent.

Neat, thank you! I went with the naive way of converting my Halogen 4 code to 5 and hit the wall here.


add browserify support and prep for publishing on npm

added browser property to package to be compatible with browserify.  Also added .npmignore in hopes that you publish to NPM as well, but if not I guess I can continue to point to your github in my package.

Thanks @ruzz311


Front-end toolchain within the VM

Hey Jeff. I'm a huge fan of DrupalVM for local development. I'm also responsible for my team's development infrastructure, for which we're using a fork of DrupalVM. We're also developing with Sass and compiling it with Compass, as well as enabling LiveReload with the guard-livereload ruby gem. The workflow is great on our current OSX infrastructure.
However, I'm also starting some work with a developer who only has access to a Windows machine. I love that DrupalVM is still an option for him, but I'm also thinking about how to solve the front-end pipeline on windows. I've been able to get the toolchain working on Windows in the past, but it was slow and a bit buggy. Would you consider accepting a PR that would move the Sass/Compass/LiveReload tooling within the VM? I'm thinking that may be more flexible and performant, not to mention useful for the wider community. If you are willing to consider it, I'd love some suggestions for a pattern to look at for the implementation. For example, would you think of it as an "extra" for the installed_extras section? Or an additional configuration option?
Thanks for your consideration, and even more for all the awesome work!

You can do most of this at the moment already.
To install ruby gems (guard-livereload etc) you can add it to the ruby_install_gems:
ruby_install_gems:
  - sass
  - guard-livereload

For NPM pacakges you can use nodejs_npm_global_packages
nodejs_npm_global_packages:
  - name: phantomjs

To enable port forwarding (I think you need this for livereload no?) you can create a Vagrantfile.local next to the regular Vagrantfile with the following:
config.vm.network "forwarded_port",
  guest: 35729,
  host: 35729,
  protocol: tcp

Yes, and to follow up on what @oxyc said, I don't want to add in a particular front end tool chain, but leave it up to the user/team to choose what they want and install the appropriate gems/npm modules.
Some teams use Ruby, some Node, and within both ecosystems there are dozens if not hundreds of different combos of preferred tools for Sass, optimization, compilation, live reload, browser preview, etc. for different themes and theme systems, and I don't want to pigeonhole Drupal VM into one over all the others.

Great, thanks! I'll give @oxyc's suggestions a try.


Trouble importing character

Describe your bug here.
When trying to import a character in the character editor, an error appears reading: "Error: No Animation Found".
I do have the accompanying .xml file along with the sritesheet, and even still it simply will not import into the editor. No character or anything visible. (I am using the engine through Wine with linux, I'd doubt that'd affect anything though).

Command Prompt/Terminal logs (if existing)
Nope, nothing
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
Yes, placing the character spritesheet and .xml (mods/ModTemplate/My-Mod/images/characters)

same issue here, older builds work fine, but anything not made in adobe wont work for some reason in this newer version

Try going under the animations tab. If you either have the xml open in another window, or already know what animations go to the ones that'll be used in the json, try typing the names in for each animation.

Try going under the animations tab. If you either have the xml open in another window, or already know what animations go to the ones that'll be used in the json, try typing the names in for each animation.

From there would I need to use the character editor?

Try going under the animations tab. If you either have the xml open in another window, or already know what animations go to the ones that'll be used in the json, try typing the names in for each animation.

I configured all of the animations, disregarding the lack of the character in the editor. I added the .json's name to the character list, and when attempting to use the chart editor, there was still no character in sight. Just a transparent Boyfriend as the opponent.


CB-16668 Reenabling consumption service build

turning back cinsumption-service build to why it fails
local build works
./gradlew :cloud-consumption:publishToMavenLocal works
since we lost the logs this is the only way to find out why the build failed on releng portal

Going to merge it since integration-tests do not validate anything related to this change.


Session validation

Changes require db rebuild.
Summary of changes:

removed JS generation of session strings
added sessions to db and updated scripts
retrieve sessions in update/add routes and added templating to render them in place of the removed JS
added session validation

Session rework was in support of session validation. Will close https://github.com/world-class/compass/issues/47
There is also an extra commit to get rid of some require statement that inexplicably made it's way into my code and I didn't notice until after I committed the changes.

Sessions don't load on firefox. @BlairCurrey


SiChuan Top IT Voctional institute

add SiChuan Top IT Voctional institute

@zhengdebin Pull request merged. Thank you!


Feature: Add octal support to Change IP Format.

I couldn't see any tests for the existing functionality so I've added some that ensure each of the four input and output formats is tested once.
The logic for octal and decimal is very similar so I've pulled some of it out into a separate method to save duplicating all the bit-fiddling.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


v0.12.4 - Appointment button function added.

Appointment button function, using cookie-created home page and patient profile page.


Service mode

service mode with coil and switch test

Coverage decreased (-0.4%) to 91.706% when pulling 17642f74eba1f9c080c5e8040254c1d01f8728f0 on service_mode into 575be7d55164794ca0e3df68f17e4d61ccf6462a on dev.

Coverage decreased (-0.4%) to 91.706% when pulling 17642f74eba1f9c080c5e8040254c1d01f8728f0 on service_mode into 575be7d55164794ca0e3df68f17e4d61ccf6462a on dev.

Coverage decreased (-0.3%) to 91.781% when pulling 17642f74eba1f9c080c5e8040254c1d01f8728f0 on service_mode into 575be7d55164794ca0e3df68f17e4d61ccf6462a on dev.


make sure we've got analytics set up

set up & tested on all main pages

assuming that it'll be copied across now :) cc @lorrainechu3n


#358 Glossary project

Fixes #358
Reminders

See the README for more details on how to work with the Rancher docs.

Verify if changes pertain to other versions of Rancher. If they do, finalize the edits on one version of the page, then apply the edits to the other versions.

If the pull request is dependent on an upcoming release, make sure to target the release branch instead of main.

Description

Adds a glossary of Rancher-related terms.
Since we were struggling to add a plugin that would provide prettier Markdown syntax for definition lists, we're using HTML tags. The tags are necessary for Algolia to index the terms and make them searchable.
I also included some CSS styling.

List of terms to define

[x] ☰
[x] ⋮
[x] Apps
[x] Apps & Marketplace
[x] Catalogs (section)
[x] Catalogs (Helm chart repos)
[ ] Cloud credentials
[ ] Community (Rancher edition) -> discuss with Cam
[ ] Deployment
[ ] Downstream cluster
[ ] Extensions
[ ] External auth
[ ] Hosted cluster
[ ] Imported
[ ] Integrations
[ ] Hosted cluster
[ ] Hosted pools
[ ] K3s
[ ] Local auth
[ ] Local cluster
[ ] Machine pool
[ ] Managed cluster
[ ] Neuvector Prime
[ ] Node template
[ ] Prime
[ ] Project
[ ] Project resource quotas
[ ] PSA
[ ] PSA Config Template
[ ] PSP
[ ] Rancher chart
[ ] Rancher CLI
[ ] Rancher Enterprise
[ ] Rancher Kubernetes API (RK-API)
[ ] Rancher Kubernetes Engine
[ ] Rancher Prime
[ ] Rancher server
[ ] Registered cluster
[ ] RKE1
[ ] RKE2
[ ] RKE Government
[ ] Roles
[ ] UI
[ ] Upstream cluster
[ ] Users
[ ] Workload

Screenshot as of Feb 15 (latest):

Comments

We (@martyav & @sunilarjun) looked into adding a plugin that would provide native Markdown syntax for definition lists. We landed on this page , which has instructions on adding remark-deflist (repo) to your Docusaurus instance, but the instructions didn't seem to work for our config. I tried remark-definition-list (repo), but that didn't work either. While looking for a solution, I ran across this comment  on an issue  complaining that the instructions for adding Remark plugins needed improvement. It seems that the plugins need a remark-rehype handler to be configured. There was a PR  to add this functionality to the Docusaurus config file, but I don't think it's available yet on Docusaurus 2. We might need to upgrade to Docusaurus 3. In the meantime, the HTML tags are perfectly functional.

For the purposes of dividing up the word on definitions --
@LucasSaintarbor, could you do Managed Clusters through Rancher Enterprise, and @sunilarjun could you take RK-API through Upstream clusters?
Roughly,
Me - A to M
Lucas - M to Rancher E
Sunil - Rancher F to Z

Some definitions are still in progress but wanted to post the list here as it is nearing completion:
Rancher Kubernetes API (RK-API)
Versions: Rancher v2.8 and later
Users can utilize standard API requests with respect to available Rancher resources through the RK-API.
Rancher Kubernetes Engine (RKE)
A CNCF-certified Kubernetes distribution that runs entirely within Docker containers. There are two main versions of RKE available. The 1.x software line is referred to as RKE1. RKE2 or RKE Government is a Kubernetes distribution that focuses on security and compliance for U.S. Federal Government entities.
Related Terms: RKE1, the 1.x software line of RKE.
Rancher Prime
Versions: Rancher v2.8 and later
Rancher Prime is a new edition of the commercial, enterprise offering built on the the same source code. Installation assets are hosted on a trusted registry owned and managed by Rancher with additional value coming in from security assurances, extended lifecycles, access to focused architectures and Kubernetes advisories.
Related terms: Rancher Enterprise
Rancher Server
Versions: All versions
Rancher Server is a Kubernetes management tool to deploy and run clusters anywhere and on any provider. Rancher includes all the software and downstream cluster components used to manage the entire Rancher deployment.
Related terms: Rancher
Registered Cluster
RKE1
RKE1 is a certified Kubernetes distribution and CLI/library which creates and manages a Kubernetes cluster.
Related terms: RKE (Rancher Kubernetes Engine)
RKE2
RKE2 is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
Related Terms: RKE Government
Roles
Within Rancher, roles determine what actions a user can make within a cluster or project.
UI
Upstream Cluster
An upstream cluster is the local Kubernetes cluster that your Rancher installation is provisioned on.
Users
Workload
Workloads are objects that set deployment rules for pods. Based on these rules, Kubernetes performs the deployment and updates the workload with the current state of the application. Workloads let you define the rules for application scheduling, scaling, and upgrade.

Since https://github.com/rancher/rancher-docs/pull/1305 is merged, https://github.com/rancher/rancher-docs/pull/1332 is now open, and we've ported some of the comments from this PR to the parent issue, I'm closing this PR. Closing the PR helps clean up the repo as we have a number of very long-lived PRs currently open.

Since https://github.com/rancher/rancher-docs/pull/1305 is merged, https://github.com/rancher/rancher-docs/pull/1332 is now open, and we've ported some of the comments from this PR to the parent issue, I'm closing this PR. Closing the PR helps clean up the repo as we have a number of very long-lived PRs currently open.


Avoid usage of std::fs::rename as it leads to unexpected failures

Is your refactoring request related to a problem? Please describe.

Usage of std::fs::rename leads to unexpected operations failing if the source and destination paths are on different partitions. This kind of error manifests as a "Invalid cross-device link" error (OS 18) on linux devices.
The documentation for std::fs::rename alludes to this error. Below is a snippet of the error section:

This function will return an error in the following situations, but is not limited to just these cases:

from does not exist.
The user lacks permissions to view contents.
from and to are on separate filesystems.

Currently there are 3 files which use this logic, or use a function which uses this logic.

[ ] crates/common/tedge_utils/src/file.rs#L118
[ ] crates/common/tedge_utils/src/fs.rs#L26
[ ] crates/extensions/c8y_config_manager/src/upload.rs#L301

Describe the solution you'd like

Create a utility method which is capable of moving files across different filesystem mount points (e.g. across different partitions).
An example of this utility can be found in the rustup utils
Describe alternatives you've considered

Additional context

This use of rename must definitely be fixed: crates/extensions/c8y_config_manager/src/upload.rs#L301 as this is to move a configuration uploaded by a child device to the appropriate location. This use must be fixed using the move_file function provided by tedge_utils.
The later function (move_file) has to be fixed as currently using rename. As suggested by @reubenmiller, one can take inspiration from rustup copy_and_delete function. We still have to find the best way to switch from a failing rename to copy_and_delete. This should be done only because the file systems are not the same.
Warning The two uses of rename in tedge_utils::fs must be kept unchanged. These are to implement atomic writes with atomically_write_file_sync and atomically_write_file_async. For the rename to be atomic it must a rename and not a copy & delete. However the parameters provided to these functions are error prone. The first argument must be removed and computed after the target file path (using set_extension("tmp")).

Verified (as the test is not so simple as it relies on having multiple partitions etc.)


please choose a path that is not inside “desktop” “downloads” or “documents”

Describe the bug
i downloaded the genshin launcher and everything but whenever i start up the launcher it says im choosing an invalid path but there’s no other place to put the file, idk what to do. also i watched a tutorial and the guy in the video had a tab under the location thing and i dont have that.
Screenshots

 - macOS:  13.6.6 Ventura

Device: iMac Retina 5k 27nch 2017
CPU: 3.5 GHz Quad-Core Intel Core i5
Dedicated GPU (if any): Radeon Pro 575 4 GB

is ur launcher in /Applications?
Make sure it is in /Applications and not /Applications Support

is ur launcher in /Applications?
Make sure it is in /Applications and not /Applications Support

it is in applications, yes

Put it in ur $Home

Put it in ur $Home, Seems like Yaagl didn't get the required permissions to put stuff on those paths, can fix that by giving it full disk access.

oh my god it worked tysm 😭

Put it in ur $Home, Seems like Yaagl didn't get the required permissions to put stuff on those paths, can fix that by giving it full disk access.

Hello, how do you do this?


sample program echo_server response abort message

my env

container base image is ubuntu 20.04 and my host PC also ubuntu 20.04
default sample program return abort message
I didn't edit any codes.
server
/usrsctp/programs# ./echo_server

client
/usrsctp/programs# ./client <IP_ADDRESS> 7

tcp_dump result
10:20:20.166425 IP <IP_ADDRESS>.56702 > <IP_ADDRESS>.7: sctp (1) [INIT] [init tag:<PHONE_NUMBER>] [rwnd: 131072] [OS: 10] [MIS: 2048] [init TSN:<PHONE_NUMBER>]

10:20:20.166584 IP <IP_ADDRESS>.7 > <IP_ADDRESS>.56702: sctp (1) [ABORT]

10:20:20.166764 IP <IP_ADDRESS>.7 > <IP_ADDRESS>.56702: sctp (1) [INIT ACK] [init tag:<PHONE_NUMBER>] [rwnd: 131072] [OS: 10] [MIS: 2048] [init TSN:<PHONE_NUMBER>]

10:20:20.166787 IP <IP_ADDRESS>.56702 > <IP_ADDRESS>.7: sctp (1) [ABORT]

How to fix it??
server
/usrsctp/programs# ./echo_server 11111 22222

client
/usrsctp/programs# ./client <IP_ADDRESS> 7 0 22222 11111

this is working.
what is diff??
And echo_server cant' connect with socat
# socat - sctp:<IP_ADDRESS>:7

./echo_server

3 questions

how to connect client and echo_server without udp_encap
diff with and without udp_encap
how to connect echo_server with socat

Do you have sctp compiled into your kernel?
User SCTP is in user space, and if the kernel has a SCTP stack in it, your user
space stack, which is listening will not get the packets and instead the kernel will..
And of course it will respond with an ABORT.
You should validate your kernel configuration does not include SCTP
R

On Dec 14, 2022, at 8:56 PM, sewon @.***> wrote:
my env
https://www.google.com/url?q=https://user-images.githubusercontent.com/12094135/207753932-e60f4a51-1629-48c2-a5fb-bf7d185cc195.JPG&source=gmail-imap&ust=1671674190000000&usg=AOvVaw3Cz53KsZZBuC-RK_dbqVK1
container base image is ubuntu 20.04 and my host PC also ubuntu 20.04
default sample program return abort message
I didn't edit any codes.
server
/usrsctp/programs# ./echo_server
client
/usrsctp/programs# ./client <IP_ADDRESS> 7
tcp_dump result
10:20:20.166425 IP <IP_ADDRESS>.56702 > <IP_ADDRESS>.7: sctp (1) [INIT] [init tag:<PHONE_NUMBER>] [rwnd: 131072] [OS: 10] [MIS: 2048] [init TSN:<PHONE_NUMBER>]
10:20:20.166584 IP <IP_ADDRESS>.7 > <IP_ADDRESS>.56702: sctp (1) [ABORT]
10:20:20.166764 IP <IP_ADDRESS>.7 > <IP_ADDRESS>.56702: sctp (1) [INIT ACK] [init tag:<PHONE_NUMBER>] [rwnd: 131072] [OS: 10] [MIS: 2048] [init TSN:<PHONE_NUMBER>]
10:20:20.166787 IP <IP_ADDRESS>.56702 > <IP_ADDRESS>.7: sctp (1) [ABORT]
How to fix it??
—
Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/sctplab/usrsctp/issues/670&source=gmail-imap&ust=1671674190000000&usg=AOvVaw3wUE2Ajn-5R5la4sv4LTv5, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AAQZ4GX5IFCVZTEAAHYZ2VTWNJ3EZANCNFSM6AAAAAAS7F57QI&source=gmail-imap&ust=1671674190000000&usg=AOvVaw2EwSF7HWuduPZ-9Qr74shd.
You are receiving this because you are subscribed to this thread.

Randall Stewart
@.***

@stewrtrs
thx for reply.
then, when should I exclude sctp kernel module? when build or runtime or both??


Stale message improvement

Finishes up #102 by improving stale message and adding an action to login.
Had to include the error template.
Screenshot

cool!

@hkeeler - I had thought about that too. I think it's ok, at least for now. It's good to give the person some sort of action to take. We could reword it all a bit to just say "Go to the home page" and the button to just say "Home" ... or something else entirely.
Any thoughts @keileyo @designlanguage?

Also, it looks like the text that landed has diverged from what was in the issue. I think the "expired" language is more accurate, since this primarily happens if a user doesn't click their link in the timeout window.

@wpears - I agree. Updated in e9adb41 and we should be ok here.


regarding the multi-node elastic using docker compose

I tried the docker-compose file to setup  multi-node containers. I i bring up the containers. How will access the kibana interface. I tried with the http:///80 , but the kibana does not work.

my ip address points to digital ocean instance. how will access the same from my local web browser ?
containers are up and running for me as well

my ip address points to digital ocean instance. how will access the same from my local web browser ?
containers are up and running for me as well

@blacktop , do you have an email id. I have another requirement on the same and i can email you

@blacktop , do you have an email id. I have another requirement on the same and i can email you

It should be working now (as soon as the new version of blacktop/elasticsearch finishes building on the DockerHUB.
For building a multi-node cluster on DigitalOcean you can do this - https://www.digitalocean.com/community/tutorials/how-to-set-up-a-production-elasticsearch-cluster-on-ubuntu-14-04
The important thing to notice is that the nodes need to think they are all on the same private network (and not be exposed to the public internet) although X-Pack will secure that for you, but you might need SSL.
Or you could set up a docker-swarm and run it that way.
Best of luck.

It should be working now (as soon as the new version of blacktop/elasticsearch finishes building on the DockerHUB.
For building a multi-node cluster on DigitalOcean you can do this - https://www.digitalocean.com/community/tutorials/how-to-set-up-a-production-elasticsearch-cluster-on-ubuntu-14-04
The important thing to notice is that the nodes need to think they are all on the same private network (and not be exposed to the public internet) although X-Pack will secure that for you, but you might need SSL.
Or you could set up a docker-swarm and run it that way.
Best of luck.

Hi, I will try the same. when i meant multi-node it was multi-container micro service arch.
I have a private cloud with openstack and spinned up VM's using the same and executed your docker-compose file for elastic. in this case i tried use the Open stack VM ip:address:80, should we use any nginx reverse proxy to port 80 in public network ? for Kibana . I am guessing
then below should work ?
With docker-engine : $(docker inspect -f '{{ .NetworkSettings.IPAddress }}' elstack)

I have an project idea regarding the containerization technology since you have loads of experience in Docker/DevOps can we discuss about the same ?

@mkscala I don't believe I have time to help with that.
I have sooo many projects I am trying to work on now that I can barely keep up with.

ok that's fine, if you are very busy then it is ok.
It is an e-learning startup think about it and let me know the concept is Lab as as Service.
Or you can help on some doubts or you can support me as a Contributor.


Add support for Windows

I use both Windows and Fedora Linux, and while the plugin installs without any issues on Fedora, it had to be adapted to work on Windows.
This pull request:

Bumpsmlua to version to 0.9, since version 0.6.2 (which is the one used in Cargo.lock) doesn't compile on Windows
Removes mlua's vendored feature, since it clashes with module (a compile-time error is generated), and without module the code doesn't compile.
Updates lua-json5's code so that it compiles using mlua 0.9 (ToLua has been renamed to IntoLua in newer versions of mlua, so the code had to be fixed)
Adds an install.ps1 script to install the plugin on Windows, and updates the README to reflect this change, which emulates what install.sh does on linux and macos.

I'm currently using the forked plugin myself:
https://github.com/Crax97/dotfiles/blob/c213d0574775ba361f080adf4afa33177f4261e7/nvim/lua/plugins.lua#L14-L18
And, using my setup, it installed fine both on Fedora 38 and Windows build 22621

Seems good to me. Sorry for the late response, and thank you for your contribution! 😃


Class "Backpack\CRUD\app\Console\Commands\Process" not found on fresh installation

Bug report
What I did:
cd my_fresh_laravel project
composer require backpack/crud
php artisan backpack:install
What I expected to happen:
backpack would be installed smoothly
What happened:
encountered the following error:
Class "Backpack\CRUD\app\Console\Commands\Process" not found
What I've already tried to fix it:
composer dump-autoload
Backpack, Laravel, PHP, DB version:
PHP VERSION:
PHP 8.1.2 (cli) (built: Aug  8 2022 07:28:23) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.2, Copyright (c) Zend Technologies
with Zend OPcache v8.1.2, Copyright (c), by Zend Technologies
LARAVEL VERSION:
v9.36.4@15ce569fd93124e8e2257c24e3ed85b9ef9951d6
BACKPACK VERSION:
5.4.3@f1c257e10b8228e48916a7b99b325224e7415910

UPDATE
i checked the folder where the file supposedly existed and it is not there.


New function of corslite now you can retrieve data and set synchronus call or ajax call and make post request not just get

New function of corslite now you can retrieve data and set synchronus call or ajax call and make post request not just get.

Thanks for the contribution, but per the README, we're intentionally limiting corslite to GET requests because that's the subset that's supported by IE 8's XDomainRequest. Corslite is intentionally compatibility focused; If you would like more modern AJAX functionality, you're best off with a different library.


MediaFusion Prowlarr Indexer Not Appearing in Sonarr for Searches

I added the MediaFusion indexer to Prowlarr and synced it with both Radarr and Sonarr. The indexer appears and works as expected in Radarr under the "Indexers" page. However, it does not show up in Sonarr's "Indexers" page or function for searches.
Steps to Reproduce:

Add the MediaFusion indexer in Prowlarr.
Sync the indexer with Radarr and Sonarr.
Verify functionality in Radarr (works as expected).
Check Sonarr (indexer missing and searches fail).

Expected Behavior:
The MediaFusion indexer should appear and work in Sonarr, similar to its behaviour in Radarr.
Actual Behaviour:
The indexer is missing from Sonarr, and it does not function for searches.
Please let me know if you need additional details or logs to troubleshoot this issue.

Hi, Im not a *Arr user, The mediafusion prowlarr template was created by a contributor.  So ill leave this to any other contributor to answer.


Eth consensus updates

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.

Project coverage is 36.23%. Comparing base (5ba8b66) to head (9e6d473).

Files with missing lines
Patch %
Lines

src/app/migrations.rs
0.00%
3 Missing :warning:

src/app.rs
0.00%
2 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #322      +/-   ##
===========================================
+ Coverage    35.89%   36.23%   +0.34%
===========================================
  Files           23       23
  Lines         9995     9901      -94
===========================================
  Hits          3588     3588
+ Misses        6407     6313      -94

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Strange behavior of NavigationAgent2D

Tested versions
Godot 4.3 dev6 and Godot 4.2.2
System information
Godot v4.3.dev6 - Windows 10.0.19045 - Vulkan (Forward+) - dedicated Radeon RX 560 Series (Advanced Micro Devices, Inc.; 31.0.14001.45012) - Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz (4 Threads)
Issue description
NavigationAgent2D sometimes gets stuck at intersections of NavigationRegion2D baked mesh
All parameters are default
https://github.com/godotengine/godot/assets/130399274/4c2eece8-d6ce-486f-8bfe-55c212508595
Steps to reproduce
Create a huge navigation mesh, and use NavigationAgent2D to avoid obstacles
Minimal reproduction project (MRP)
nav.zip

I also want to clarify that I changed various properties of the agent itself, then returned everything to default, after which I began to notice this problem

In this case it is a matter of adjusting the path desired distance and target desired distance to the agent's movement speed.
The agent in that test project moves at a speed of 550.0.
Using the default distances of  20 and 10 px there is no way at that speed that the agent does not regularly over- or undershoots the path positions. That is also what it gets stuck, it never is able to get to a position where it can increase its internal path index because the physics step movement distances are too large.
This is mentioned in the agent documentation under common problems as a setup issues here https://docs.godotengine.org/en/latest/tutorials/navigation/navigation_using_navigationagents.html#pathfollowing-common-problems

In this case it is a matter of adjusting the path desired distance and target desired distance to the agent's movement speed.
The agent in that test project moves at a speed of 550.0.
Using the default distances of 20 and 10 px there is no way at that speed that the agent does not regularly over- or undershoots the path positions. This is also what gets the agent stuck. The agent is never able to get to a position where it can increase its internal path index because the physics step movement distances are too large.
This is mentioned in the agent documentation under common problems as a setup issues here https://docs.godotengine.org/en/latest/tutorials/navigation/navigation_using_navigationagents.html#pathfollowing-common-problems
I rember there was an old Godot 3 PR #62682 from lawnjelly that tried to mitigate this by doing path segment checks but it was never ported to Godot 4 development branch and then closed. It would still be viable but by now the NavigationAgent logic has changed so much that the old PR would likely need a full rework.

I knew you would mention the motion speed option (forgot to remove it), I hope you set the speed to default and tested my project because this option didn't change anything, the PROBLEM is there, with all the default options.
(I don’t understand why immediately remove this problem from the bugs list, apparently it will drown in a pile of old problems...)

You are right and it also happens on a few edges with reduced speed.
Likely the issue is amplified by some of the very, very, very long navigation mesh polygon edges. The agent uses Geometry2D edge segment checks internally and those naturally run into precision issues with very long segements. As soon as the edge length is reduced to a more manageable length the issue disappears for me using the defaults.

How then can you increase your movement speed? I set the speed to 30, the problem is still there :'(

I went back to the original state of your test project and the only thing that I changed was the size of the navigation mesh to reduce the edge length.
The issues only happens on navigation mesh polygons that are extremely large.
This tells us that this is at its core a precision issue with the geometry functions to find the closest position on a triangle. The navigation map needs to find the closest positions on the navigation mesh surfaces. It finds the closest polygon and then the closest position to the agent on the triangles of the polygon. When precision becomes an issue this will give positions that are more and more different with more and more larger gaps every time the agent moves a little and a new path is requested.
I now realized that your script updates the agent target position every single physics step with the mouse position.
When I remove that part or do the update with a timer on a longer interval (e.g. 0.5 sec) the precision issue is still visible with the path positions having larger than usual point gaps but the agent no longer gets stuck.
That very frequent path updates can cause problems is also mentioned in the common problems of the agent documentation. The reason why it is such a problem ( apart from performance) is that because the navigation map needs to find the closest positions on the navigation mesh surfaces this might give positions that are either behind or in front of the agent every time. Since an agent is just a point in pathfinding, it has no orientation, when the position is one update in front and one update behind the agent direction the agent gets stucked bouncing between the two constantly changing start positions.

@smix8 Got it, thanks for the detailed explanation.
Is this a problem with the Recast library?
So big 2D worlds with pathfinding are not about Godot? Is there really no way to get around this problem?

@smix8 I came up with an interesting method that could solve my problem, it is a slightly different pathfinding method, and its accuracy is not always perfect, but this pathfinding is used in Game Maker, and for most cases it is ideal, it would be great if it was added as additional option for pathfinding.
I attached NavigationRegion2D to my character who should look for a path, now NavigationRegion2D is a descendant of CharacterBody2D, we draw a region polygon around the character, the accuracy of the path depends on this region, and if after some time we constantly baked this region in a new way, it could bypass obstacles. Dynamic baking is certainly bad for performance, but this region is not large, only around the character, which would not greatly impact performance.
https://github.com/godotengine/godot/assets/130399274/b86cfc49-0eab-4e38-8fe4-b55c62a5cac7
And if NavigationRegion2D had a new Node mode in the SourceGeometryMode parameter, in it it would be possible to select a 2D node from the scene, which would store all the static objects that would participate in baking the region, then such a method could be used!
Such a pathfinding would look something like this:
https://github.com/godotengine/godot/assets/130399274/11b215cf-8f9e-4563-86fa-a94bacb313cd
such problems can occur if the region around the character is too small:
https://github.com/godotengine/godot/assets/130399274/9231dca4-f60d-4676-bfc3-0262586fabec

Your idea is in general heading in the right direction as you only want navigation mesh within a reasonable distance around your active actors when dealing with large worlds. Your proposed solution might even be a working solution for some projects.
Naturally I am part of the doomsayer-squad that sees all the technical constrains that exist in the current navigation system that may cause problems for this so bear with me.
You said you added 100 agents and performance was nearly the same which I believe.
The performance issues do not directly correlate with the agent count and more with the navigation map and navigation mesh complexity. You can have ten thousands of agents if your entire navigation map only has 1-2 polygons to search through. Things are very different in performance when you have ten thousands polygons to search through just for 1-2 agents.
On a static navigation map with the navigation mesh not changing things will stay performance stable but as soon as you change region or navigation mesh the navigation map needs to update its navigation mesh polygon soup and all the edge connections. Those navigation map updates get increasingly costly the larger and more complex the combined navigation mesh gets.
While this is a general performanc issue with navigation meshes, by moving a NavigationRegion2D node with your agent like proposed it triggers such a costly navigation map update on every single frame. On top every single agent connected to that map will query a new navigation path on the map change. Because your moved region is still freeform the entire geometry parse and rebake of the navigation mesh is also far more costly compared to a simple grid-based navigation that just flips a cell state. I think you can see why this proposed approach will ballon into a performance issues quickly with the current system.
In general it is always better to not move regions and instead toggle static region chunks around the active actors.

Keeping the region chunks in place avoids a good part of the heavy recalculations costs every frame that would happen when a region is moved. Also when regions stay fixed they can fit into a large world grid and this way can also more efficiently be rebaked for runtime changes. If regions are moved every frame arbitrary with agents all those things require very costly recalculations. That is why nearly all large open world games are done on static navigation mesh chunk systems.

Closing as there is a proposal open to track the feature part of this discussion and there is no real actionable issue left here.


Naming z-index helpfully

If you've ever used z-index you know that it doesn't take long before you've got z-index 1, 2, 5, 10, 99, 101 and 999999.
One great thing about atomic CSS is that you can use it to limit the number of variations in margins, font sizes, shades of grey etc. Tailwind by default has a sensible z-index scale but I can't help but feel that there's a better naming convention. Of course, these would mostly be on a per-site basis.
For a site I'm working on now I'd probably want to name them something like...
content
content_floating
header
modal
I understand that Tailwind makes customising this easy. What I'm wondering is, does anyone have a mental model that helps them position their layers sensibly?

bootstrap has a pretty sensible default for this, based on a ton of research on their part:
$zindex-dropdown:          1000 !default;
$zindex-sticky:            1020 !default;
$zindex-fixed:             1030 !default;
$zindex-modal-backdrop:    1040 !default;
$zindex-modal:             1050 !default;
$zindex-popover:           1060 !default;
$zindex-tooltip:           1070 !default;

Don't necessarily agree with their z-index defaults, (1-7 or 10-70 should do; if you're always using the z-index variables, that is), but that should be a good start…

For bootstrap it makes more sense to have meaningful names as they have a bunch of components, where those can be applicable.
In tailwind you only have general utilities and I think it's best if those names are defined for the given project.
Good thing that you can just rename them in your project's config.


Add ResponseTimeout Property and Add to Requests to VepWrapper

I am making a PR now to this fork ... I think it would be good if we do not send the requestTimeout parameter when the value set is zero. By not sending it, we keep the genome-nexus server code compatible with Ensembl and our standalone ec2 instance deployment of the vep service for annotation.genomenexus.org. We would be able to deploy these updates to the current annotation.genomenexus.org server without having to first update VEP by not setting a requestTimeout in that deployment.


Feature: provide better social links for profiles

Type of feature
🍕 Feature
Current behavior
Currently the project focuses on LinkedIn and Twitter as social links for user profiles. Twitter no longer exists (now called X) and many open source contributors prefer other platforms such as fediverse alternatives.
Suggested solution
Provide the ability to insert alternative social profile links into the Open Sauced profile.
Additional context
GitHub itself expanded the list of options for profile links to enable developers to no longer be limited to X.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Contributing Docs

[X] I agree to follow this project's Contribution Docs

Can you add a screenshot of the current behavior?

Thanks for opening the issue @andypiper! @dev-phantom, they're mentioning supporting more social links like GitHub, e.g. here's my GitHub profile, and here's my OpenSauced profile.

Thanks for opening the issue @andypiper! @dev-phantom, they're mentioning supporting more social links like GitHub, e.g. here's my GitHub profile, and here's my OpenSauced profile.

Oh nice
Thanks for the clarity


Add support for watch on MacOS systems

Fix #75

usually, people name "darwin" the apple binary

For the record, folks have reported that this PR fixes watcher on macos (both intel & m1). Ready for review & merge :slightly_smiling_face:

(I just moved the root watcher directory into tools/watcher)


Pre VM function renames

Some function renames to change "container" to "instance".
Add support for instance type to some functions.

Testsuite passed

Looks good to me!


WinAppDriver fails to find any element on Share Windows Store App Picker if using AccessibilityID but Inspect.exe sees it

WinAppDriver does not find elements by accessibility_id in a Windows Store Share dialog.
STEPS:
I have an application that lets me share log files

When I click on a button Share dialog pops up giving me windows store apps I can use to share my log file such as Mail, OneNote, etc
Any element having AutomationId, including Mail Windows Store App can not by found by AutomationId but can be found by Name (see attached image)
However, Inspect.exe app sees both Name and AutomationId (as per attached image)

Below is code sample using Name that works and code sample using AccessibilityID that fails.
CODE THAT FINDS ELEMENT USING NAME
When /^I see external element "([^\"]*)"$/ do |name|
		sleep(2.0)
		desktop_session_setup()
		item = $DesktopSession.find_elements(:name, name)[0]
		desktop_session_end()

		unless item != nil
			fail("External element with #{name} not shown.")
		end
end

CODE THAT FAILS TO FIND ELEMENT USING ACCESSIBILITYID
When /^I see external "([^\"]*)" by id$/ do |accessibility_id|
    sleep(2.0)

    desktop_session_setup()
    item = $DesktopSession.find_elements(:accessibility_id, accessibility_id)[0]
    desktop_session_end()

    unless item.displayed?
      fail("Item with id #{accessibility_id} not found.")
    end
end

Hi @db-007,
Thanks for the detail annotated screenshots that provide clear context of the issue. Would you be able to provide the log from WinAppDriver.exe for both find elements commands above? We would like to see the JSON wire protocol command generated and its payload from your script. E.g.
==========================================
POST /session/D4348D8A-BD5C-41F0-A216-9B0EB0557378/elements HTTP/1.1
Accept: application/json, image/png
Content-Length: 53
Content-Type: application/json;charset=utf-8
Host: <IP_ADDRESS>:4723

{"using":"accessibility id","value":"AlarmPivotItem"}
HTTP/1.1 200 OK
Content-Length: 100
Content-Type: application/json

{"sessionId":"D4348D8A-BD5C-41F0-A216-9B0EB0557378","status":0,"value":[{"ELEMENT":"42.71938.3.3"}]}

@timotiusmargo how do I get you the log you need?

Typically, WinAppDriver.exe can run as a standalone program running as a console application logging all requests and responses in your machine under test. If you run other client on top such as Appium, the logs are normally embedded within the client log.

Sorry, but I am unable to find any logs

I am facing similar issue
POST /session/DD0375E5-70D8-4EC4-8D92-908FF9FA88CB/element HTTP/1.1
Accept: application/json, image/png
Content-Length: 31
Content-Type: application/json;charset=utf-8
Host: <IP_ADDRESS>:4723
{"using":"name","value":"ANPS"}
HTTP/1.1 404 Not Found
Content-Length: 139
Content-Type: application/json
{"status":7,"value":{"error":"no such element","message":"An element could not be located on the page using the given search parameters."}}
==========================================
DELETE /session/DD0375E5-70D8-4EC4-8D92-908FF9FA88CB HTTP/1.1
Accept: application/json, image/png
Host: <IP_ADDRESS>:4723
HTTP/1.1 200 OK
Content-Length: 12
Content-Type: application/json
{"status":0}

I seem to have a similar issue. I use inspect to find the AutomationId and then use FindElementByAccessibilityID() to find the element. In my case this fails when the app is a self build C# GUI. It is not just one element. I commented out the element it could not find and ran the tests to see if it could find the next, but that did not work either.
The code I use in C# to create the test:
// Launch the app
AppiumOptions options = new AppiumOptions();
options.AddAdditionalCapability("app", "C:\\path\\to\\app.exe");
options.AddAdditionalCapability("appArguments", "\"some-argument\"");
try
{
	this.appSession = new WindowsDriver<WindowsElement>(new Uri("http://<IP_ADDRESS>:4723"), options);
}
catch (ArgumentNullException error)
{
	Console.WriteLine(error.Message);
}
catch (UriFormatException error)
{
	Console.WriteLine(error.Message);
}

// ResizeWindow
var maximizeButton = this.appSession.FindElementByAccessibilityId("Maximize-Restore");
maximizeButton.Click();

The console logs from WinAppDriver:
==========================================
POST /session HTTP/1.1
Accept: application/json, image/png
Connection: Keep-Alive
Content-Length: 249
Content-Type: application/json;charset=utf-8
Host: <IP_ADDRESS>:4723
User-Agent: selenium/3.141.0 (.net windows)

{"desiredCapabilities":{"app":"C:\\path\\to\\app.exe","appArguments":"\"some-argument\"","platformName":"Windows"},"capabilities":{"firstMatch":[{"platformName":"Windows"}]}}
HTTP/1.1 200 OK
Content-Length: 238
Content-Type: application/json

==========================================
POST /session/DEFA5321-0FB7-4992-A47F-1D0A0610412B/element HTTP/1.1
Accept: application/json, image/png
Content-Length: 55
Content-Type: application/json;charset=utf-8
Host: <IP_ADDRESS>:4723
User-Agent: selenium/3.141.0 (.net windows)

{"using":"accessibility id","value":"Maximize-Restore"}
HTTP/1.1 404 Not Found
Content-Length: 139
Content-Type: application/json

{"status":7,"value":{"error":"no such element","message":"An element could not be located on the page using the given search parameters."}}

I installed V1.1.1 for the WinAppDriver and the issue seems to be resolved.


Cost for handling linear friction-cone constraints

This task refers to writing the c++ class of: https://github.com/loco-3d/crocoddyl/blob/devel/unittest/python/crocoddyl/cost.py#L627
And it also includes to add the Python bindings + unittest code.
Note that this task requires to be solved first #295.

solver in #657


Streaming response handling

Hi!
A handful of endpoints support response streaming, but the hackney/finch stuff is so deeply nested that it's not fully clear how to actually use those.
Are there any examples or references what to do to properly handle streaming HTTP responses?

@philss Any chance you have some input on this? I unfortunately do not use Elixir in any production code so am a bit limited on being able to assist on this question.

I managed to get it working, it's using normal streaming functionality in clients like tesla

@dvcrn  ❤ Any chance you can make a PR? ❤

I'm not sure what the right format for a PR like this would be. There are a lot of aws endpoints that support streaming of data, and multiple clients are supported. For my stuff I only used finch and the streaming support that's baked into that, but it would be different for hackney 🤔
Maybe a note in the readme?

That would work :-) Anything to avoid the dreaded DenverCoder9 scenario and would help anyone else along :-)
If there's anything I can pick up from your README PR and integrate it natively within aws-elixir we can take it as a secondary step 👍

@dvcrn Can you give me a hint on how you did it?
I enabled the finch http client:
    %AWS.Client{
      access_key_id: key.access_key_id,
      secret_access_key: key.secret_access_key,
      region: "eu-central-1",
      http_client: {AWS.HTTPClient.Finch, [finch_name: MyApp.Finch]}
    }

But when I send a request with this (I am using the PollyAPI) I dont get a stream response like this:
{:ok,
 %{
   "Body" => <<73, 68, 51, 4, 0, 0, 0, 0, 0, 35, 84, 83, 83, 69, 0, 0, 0, 15, 0,
     0, 3, 76, 97, 118, 102, 53, 56, 46, 55, 54, 46, 49, 48, 48, 0, 0, 0, 0, 0,
     0, 0, 0, 0, 0, 0, 255, 243, ...>>
 },
 %{
   body: <<73, 68, 51, 4, 0, 0, 0, 0, 0, 35, 84, 83, 83, 69, 0, 0, 0, 15, 0, 0,
     3, 76, 97, 118, 102, 53, 56, 46, 55, 54, 46, 49, 48, 48, 0, 0, 0, 0, 0, 0,
     0, 0, 0, 0, 0, 255, ...>>,
   headers: [
     {"x-amzn-requestid", "c8c11524-fb8d-40b6-aedb-7ff555ab2b6b"},
     {"x-amzn-requestcharacters", "5"},
     {"content-type", "audio/mpeg"},
     {"transfer-encoding", "chunked"},
     {"date", "Tue, 05 Mar 2024 11:31:39 GMT"}
   ],
   status_code: 200
 }}

That is not a streaming response (am I wrong?) also the transfer encoding is chunked.
Can you give me a hint on what I have to do differently?

@RudolfVonKrugstein Possibly this: https://github.com/philss/aws-s3-stream-download-poc helps you along the way until @dvcrn (hopefully) gets back to you with more details.

Adding to v1 to try and improve some documentation around this type of operation

@RudolfVonKrugstein @onno-vos-dev sorry about that, I didn't receive email notifications for some reason...
Here's my hacked together implementation: https://github.com/dvcrn/chatgpt-ui/blob/main/lib/chatgpt/anthropic.ex#L97-L218
Basically you have to

Enable streaming support with a new finch client because this repo here doesn't support streaming yet (https://github.com/dvcrn/chatgpt-ui/blob/main/lib/chatgpt/streaming_finch.ex)
Add a callback handler to :streamfx opt (that's just how I called it, has to match what you use in StreamingFinch)
Inside the callback function you need to decode the aws streaming response which looks like this: https://docs.aws.amazon.com/transcribe/latest/dg/streaming-setting-up.html#streaming-event-stream

Prelude tells you how many bytes are for the entire thing and for headers
Next 4 bytes are CRC
next n bytes (however many the prelude says there are) are header content, followed by actual payload
payload size is x = (total size - header size - 4x4), so we know that after headers, the next x bytes are payload

Then just decode the json, get the "bytes" key and base64 decode that
decoded_payload =
          payload
          |> Jason.decode!()
          |> Map.get("bytes")
          |> Base.decode64!()
          |> Jason.decode!()

Hope that helps!

Oh has this been added?


fix(source-convert): add support for api version in source convert

Description
Update source convert to support api versions. This allows for artifacts built with different api version to be deployable
Checklist
All items have to be completed before a PR is merged

[x] Adhere to Contribution Guidelines
[X] Updates to Decision Records considered?
[X] Updates to documentation at DX@Scale Guide considered?
[x] Tested changes?
[x] Unit Tests new and existing passing locally?

Codecov Report

Merging #884 (dd91644) into develop (7fa9259) will increase coverage by 1.27%.
The diff coverage is 59.75%.

:exclamation: Current head dd91644 differs from pull request most recent head 794201a. Consider uploading reports for the commit 794201a to get more accurate results

@@             Coverage Diff             @@
##           develop     #884      +/-   ##
===========================================
+ Coverage    64.34%   65.62%   +1.27%
===========================================
  Files           49       48       -1
  Lines         1593     1533      -60
  Branches       355      336      -19
===========================================
- Hits          1025     1006      -19
+ Misses         568      527      -41

Impacted Files
Coverage Δ

...ages/core/src/artifacts/ArtifactFilePathFetcher.ts
34.04% <0.00%> (+3.56%)
:arrow_up:

...org/packageQuery/InstalledPackagesQueryExecutor.ts
40.00% <ø> (ø)

packages/core/src/package/SFPPackage.ts
84.93% <ø> (ø)

packages/core/src/org/SFPOrg.ts
56.06% <56.06%> (ø)

packages/core/src/utils/ObjectCRUDHelper.ts
85.71% <85.71%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3d3f12...794201a. Read the comment docs.


[Bug] [ProcessInstance] API queryProcessInstancesByTriggerCode not working

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

[ERROR] 2023-07-12 20:23:13.453 +0800 org.apache.dolphinscheduler.api.exceptions.ApiExceptionHandler:[52]
org.springframework.jdbc.BadSqlGrammarException:
### Error querying database.  Cause: org.h2.jdbc.JdbcSQLSyntaxErrorException: Ambiguous column name "update_time"; SQL statement:
select

        b.id
        , b.name, b.process_definition_version, b.process_definition_code, b.project_code, b.state, b.recovery, b.start_time, b.end_time, b.run_times,host,
        command_type, b.command_param, b.task_depend_type, b.max_try_times, b.failure_strategy, b.warning_type,
        warning_group_id, b.schedule_time, b.command_start_time, b.global_params, b.flag,
        update_time, b.is_sub_process, b.executor_id, b.history_cmd,
        process_instance_priority, b.worker_group,environment_code, b.timeout, b.tenant_code, b.var_pool,
        dry_run, b.test_flag, b.next_process_instance_id, b.restart_time, b.state_history

        from t_ds_trigger_relation a
        join   t_ds_process_instance b on a.job_id  = b.id
        where a.trigger_type = 0 and a.trigger_code = ? [90059-210]

What you expected to happen

queryProcessInstancesByTriggerCode should work properly.

How to reproduce

Call DS queryProcessInstancesByTriggerCode open api.

Anything else
No response
Version
dev
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Should fix it before the release of 3.2.0. Otherwise, it will block users who do not use UI.


Ook to Ook!

Ook to Ook! because according to esolangs that's what its called.

@Richienb What? I’m not adding a language

Oh right.