Various security issues

Motivation for this change
I ran into a couple of issues while I was working on some related tooling.. Here is a collection of a few of those that I could motivate myself into addressing in some way..
The most important/interesting is probably the SDDM change.
I did run NixOS tests for those packages that have them.
We have to figure out how and what we want to backport to 18.03 & 18.09.
cc @Mic92 since we talked about the changes already
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[X] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[X] Fits CONTRIBUTING.md.

Success on x86_64-darwin (full log)
Attempted: taglib, yara
The following builds were skipped because they don't evaluate on x86_64-darwin: batik, kiwix, sddm
Partial log (click to expand)
-- Installing: /nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1/lib/libtag_c.0.0.0.dylib
-- Installing: /nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1/lib/libtag_c.0.dylib
-- Installing: /nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1/lib/libtag_c.dylib
-- Installing: /nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1/include/taglib/tag_c.h
-- Installing: /nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1/lib/pkgconfig/taglib_c.pc
post-installation fixup
strip is /nix/store/g5r4apl0za012ffs6ladinwa5w0m1l3k-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1/lib  /nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1/bin
patching script interpreter paths in /nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1
/nix/store/cqk88hjbjx2xg8vp7bkmc7ldxh5ah1ds-taglib-1.11.1/bin/taglib-config: interpreter directive changed from "/bin/sh" to "/nix/store/n9hba031gjky8hpjgx9fnlaxhidyzxbz-bash-4.4-p23/bin/sh"

Success on x86_64-linux (full log)
Attempted: sddm, taglib, yara
The following builds were skipped because they don't evaluate on x86_64-linux: batik, kiwix
Partial log (click to expand)
shrinking /nix/store/5hf502dvxv04z8lnlzbqr616vy4p5hm6-yara-3.8.1/bin/yara
shrinking /nix/store/5hf502dvxv04z8lnlzbqr616vy4p5hm6-yara-3.8.1/lib/libyara.so.3.8.1
gzipping man pages under /nix/store/5hf502dvxv04z8lnlzbqr616vy4p5hm6-yara-3.8.1/share/man/
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/5hf502dvxv04z8lnlzbqr616vy4p5hm6-yara-3.8.1/lib  /nix/store/5hf502dvxv04z8lnlzbqr616vy4p5hm6-yara-3.8.1/bin
patching script interpreter paths in /nix/store/5hf502dvxv04z8lnlzbqr616vy4p5hm6-yara-3.8.1
checking for references to /build in /nix/store/5hf502dvxv04z8lnlzbqr616vy4p5hm6-yara-3.8.1...
/nix/store/k0q25mfpd8kqn9mdanb48wg667k264d6-sddm-0.18.0
/nix/store/5lp8y3135jy4fm80897r9faiy5bqcml0-taglib-1.11.1
/nix/store/5hf502dvxv04z8lnlzbqr616vy4p5hm6-yara-3.8.1

Success on aarch64-linux (full log)
Attempted: sddm, taglib, yara
The following builds were skipped because they don't evaluate on aarch64-linux: batik, kiwix
Partial log (click to expand)
stripping (with command strip and flags -S) in /nix/store/59w9rx2yqs8wlhqamby4rx7cd1jxhc05-sddm-0.18.0/lib  /nix/store/59w9rx2yqs8wlhqamby4rx7cd1jxhc05-sddm-0.18.0/libexec  /nix/store/59w9rx2yqs8wlhqamby4rx7cd1jxhc05-sddm-0.18.0/bin
patching script interpreter paths in /nix/store/59w9rx2yqs8wlhqamby4rx7cd1jxhc05-sddm-0.18.0
/nix/store/59w9rx2yqs8wlhqamby4rx7cd1jxhc05-sddm-0.18.0/share/sddm/scripts/wayland-session: interpreter directive changed from "/bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
/nix/store/59w9rx2yqs8wlhqamby4rx7cd1jxhc05-sddm-0.18.0/share/sddm/scripts/Xsession: interpreter directive changed from " /bin/sh" to "/nix/store/dsyc1z7ck08ga7l0b1jcxx35wj69qcii-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/59w9rx2yqs8wlhqamby4rx7cd1jxhc05-sddm-0.18.0...
postPatchMkspecs
postPatchMkspecs
/nix/store/59w9rx2yqs8wlhqamby4rx7cd1jxhc05-sddm-0.18.0
/nix/store/l6kbpaql9kq8axfdbm03bdylwx9drl71-taglib-1.11.1
/nix/store/15ss7z3pj1h80z45wwmsv37b32pc05ld-yara-3.8.1

@bkchr oh, I must have missed those. I did those commits a few weeks ago and asked mic92 to test it once more to not be blind for my own issues...
@GrahamcOfBorg test plasma5

Timed out, unknown build status on x86_64-linux (full log)
Attempted: tests.plasma5
Partial log (click to expand)
building '/nix/store/sb02f4x1cnzxpv5yi82zws0h4y3xhg6g-xsession-wrapper.drv'...
building '/nix/store/3hb4bjvqx1sfrfbvzqxbvwdlya7v329h-sddm.conf.drv'...
cannot build derivation '/nix/store/37j5hbm63idrkq04y8siy7l9mg5i217w-etc.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/h3vk19z8pikk2ixjn6xr1g73zwwc39kj-nixos-system-machine-19.03.git.07db5f1.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/50djkzr9kx05hbjygdnnbwgibz4l5nh1-closure-info.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/d7ayn91z6p433razm2bnhyn0rs6v6wsn-run-nixos-vm.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/x4hk7zh7vr4j0arwxcm8sv0am6dsbg6s-nixos-vm.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/qraf9dglhxm1qkvjprid7lpkd73lnchq-nixos-test-driver-plasma5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/mw088c40hqy7bmh8r20cz82l2kncb0d2-vm-test-run-plasma5.drv': 1 dependencies couldn't be built
error: build of '/nix/store/mw088c40hqy7bmh8r20cz82l2kncb0d2-vm-test-run-plasma5.drv' failed

Timed out, unknown build status on aarch64-linux (full log)
Attempted: tests.plasma5
Partial log (click to expand)
machine: performing optical character recognition
machine: sending monitor command: screendump /build/ocrin.ppm
machine: performing optical character recognition
machine: sending monitor command: screendump /build/ocrin.ppm
machine: performing optical character recognition
machine: sending monitor command: screendump /build/ocrin.ppm
machine: performing optical character recognition
machine: sending monitor command: screendump /build/ocrin.ppm
building of '/nix/store/0iydfszs8nsfyy149hgrkfq6srfy16cl-vm-test-run-plasma5.drv' timed out after 3600 seconds
error: build of '/nix/store/0iydfszs8nsfyy149hgrkfq6srfy16cl-vm-test-run-plasma5.drv' failed


Add Serverless Workflow Specification v0.5 definition models

Many thanks for submitting your Pull Request :heart:!
What this PR does / why we need it:
Special notes for reviewers:
Adds the Serverless Workflow Specification v0.5 definition models
Additional information (if needed):
Coming soon:

YAML/JSON Parsers
Validation

@Neuroglia thanks! this is really cool.
Just one question, is there a way to update the README.md so that .NET noobs like myself can try it out? Thanks!

@jplane can you also please take a look?

@tsurdilo I will update the Readme tomorrow morning, along with the addition of the parsers ;)

Sorry for the delay guys, I've got a lot on my plate and I thought it would be a great idea to add fluent builders before checking in, which is taking far more time than I originally thought. I will commit my changes ASAP and will keep you posted ;)

@Neuroglia np at all, take your time and make sure you get rest too! Looked at the updated README - wow, really cool !

@tsurdilo Working right now on signing the commit. Then IMHO opining, everything's ready to go.
Only part missing so far is schema and integrity validation, but they'll be coming very soon.

@Neuroglia all your commits must be signed. You can squash all of them in one, sign, and force push.

@ricardozanini I a Git noob, sorry guys! How can I sign them off all at once?

@Neuroglia don't worry about it :) We can just merge.
For next ones try git commit -s -m "...."
Thank you!


[Feature Request]: Prevent deletion of running Runs

Feature description
Our KFP runs create Ray clusters, so we use ExitHandler to remove the clusters. With setting erminateStatus: CancelledRunFinally in the relevant DataSciencePipelinesApplication, we can guarantee that the ExitHandler will be executed if a Run is stopped.
However, when a running Run is deleted, the ExitHandler is not executed, and it leaves a lot of orphan resources.
If users, before deleting a Run, should stop it we would not have this issue.
Describe alternatives you've considered
Guarantee to execute ExitHandler when a Run is deleted.
Anything else?
No response

/transfer data-science-pipelines


New Error  - 302 Moved / CAPTCHA

Hello again!
Now, this was working a treat about an hour ago, but now I'm getting errors because google is no longer returning results, it's returning a 302 (if I log the raw output I get an html page, so the JSON is failing to parse and then the module is erroring - should check that it's valid json maybe?).
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE>

302 Moved

The document has moved
<A HREF="https://ipv4.google.com/sorry/index?continue=https://www.google.com/trends/api/explore%3Fhl%3Den-US%26req%3D%257B%2522comparisonItem%2522%253A%255B%257B%2522keyword%2522%253A%2522Women%27s%2520march%2522%252C%2522hl%2522%253A%2522en-US%2522%252C%2522endTime%2522%253A%25222017-02-15T10%253A39%253A43.469Z%2522%252C%2522startTime%2522%253A%25222004-01-01T00%253A00%253A00.000Z%2522%252C%2522time%2522%253A%25222004-01-1%25202017-02-15%2522%257D%255D%252C%2522cat%2522%253A0%257D%26tz%3D300&hl=en-US&q=EgRSmJyrGO_fkMUFIhkA8aeDS4wL63ml_8QWbc51FLsxqzmXnxSDMgFj">here.

This links over to a CAPTCHA like this;

Looks like we're going to have to implement the authentication and cookie sending afterall!

Ah, was hoping we wouldn't need to. Ok, thanks!

On Feb 15, 2017, at 05:43, Joel<EMAIL_ADDRESS>wrote:
Hello again!
Now, this was working a treat about an hour ago, but now I'm getting errors because google is no longer returning results, it's returning a 302 (if I log the raw output I get an html page so the JSON is failing to parse and then the module is erroring - should check that it's valid json maybe?).

302 Moved
302 Moved
The document has moved
here.

This links over to a CAPTCHA like this;
Looks like we're going to have to implement the authentication and cookie sending afterall!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@Dayjo Do you know roughly how many requests you made before getting the captcha?

@pat310 I couldn't give you an exact number but it was only around 10. However, my workmate's screensaver is the google trends screensaver which doesn't auth, so even that got the CAPTCHA which was highly amusing. Not sure if that had done X number of requests that had pushed it over the limit or what.

@pat310 Can I help with the cookie / login implementation? I noticed your branch 49addcookie but not sure where you've got to with it / what's next etc.

I think the best option to resolve this issue is simply to handle the error.
If we can make it detect whether it's valid JSON and not throw a fatal error at least that way any script running this won't completely fail when it tries to decode the html.
Happy to give this a go.

@Dayjo That's a good idea, just need to put a try/catch around JSON.parse

This is fixed by https://github.com/pat310/google-trends-api/pull/58 and the errors should be caught and made available in the .catch()


Log warning on non-public modules

As per #675. Will take suggestions for the log message if anyone has a better one.

@FiniteReality pointed out that this is useless because Assembly.ExportedTypes already filters to public types. ¯\_(ツ)_/¯


Fix DnD for multiday events to not cut end date

Do you want to request a feature or report a bug?
Bug / Question
What's the current behavior?
Currently it's not possible to DnD multiday event (in case of showMultiDayTimes=true). When dragging starts, end time is limited to end of current day by this line of code https://github.com/intljusticemission/react-big-calendar/blob/475674c6167391bd6212c1247fa29c1aea4c1b00/src/utils/TimeSlots.js#L129
@jquense question, is this just some safety done before DnD was introduced, or does it have it's meaning?
If this line is ignored, DnD seems to work for both one day and multiday events. End time is correctly calculated. Only small issue is, that only first day is visualized while dragging, but that I can live without.
What's the expected behavior?
To be able to DnD multiday event (in case of showMultiDayTimes=true) correctly. So the start and end time are not limited to the current day.

@xtursky This seems like it would be an easy fix if your solution above works. Mind submitting a PR?

@arecvlohe thanks for info. No problem to create a PR.
As I asked above, not sure if this is needed when DnD is not used.
So I wasn't sure if to change this generally or just for DnD addon. I'll prepare PR, test it and we will see.

@xtursky hi, any updates on the fix? Thanks!

@felgeekpe yeap, there is already PR waiting for review, just didn't reference it correctly. Thanks for asking.


Add new section classes to the text editor

Add the new section classes to the editor menu that already hosts the "secret" section class.

Added in 1.11


"name" in manifest is wrong; Separate out App Name away from "name"

The "name" is being output with spaces which is incorrect, whereas I still want to be name the application with spaces, which is a different thing. Please separate out, this is both a bug (manifest spec isn't followed) as well as an enhancement ("name" manifest should be disconnected from real application name used on .exe / .app)
See: https://github.com/nwjs/nw.js/wiki/Manifest-format#name
Before Web2Executable my workflow of a manual build was as follows:
"name" in manifest would be like: "client_brand_some-project"
set the application name in the filesystem to: "Brand - Some Project"
I would also apply the same to the Window Title: "Brand - Some Project"

Hey Martin,
Thank you for the feature request/bug report :)
You are absolutely right. I seemed to have missed that little section in the manifest about the name being lowercase with no spaces. Right now, I'm working on a redesign of the W2E GUI and will include this change in the new version. This will mainly update the aesthetics and provide a cleaner UI.
Cheers!

This has been fixed and will be in the next build.


Document extension less file packing

Related: https://github.com/NuGet/Home/issues/7601

Related:master-nkolev92-extensionless


Add aliases

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Edit Work in Small Batches

Description
A few minor edits to improve readability in the article.

There is an "ERROR render of "section" failed" error in the hugo command of the Run npm test step. There appear to be several upstream "execute of template failed" errors. I assume these errors are not related to the change in this PR (only to a single Markdown file). I am not familiar with Hugo, but I might take a look later today.


add source entry for new package conditional_substitution

Please Add This Package to be indexed in the rosdistro.
This package provides a single utility function for choosing substitutions in a launch file depending on a condition.
The source is here: https://github.com/ottojo/ros_conditional_substitution
Checks

[x] All packages have a declared license in the package.xml
[x] This repository has a LICENSE file
[x] This package is expected to build on the submitted rosdistro

New package review checklist
You can copy-paste the below into your review comment when reviewing a new package addition into rosdistro.

[x] At least one of the following must be present

[x] Top level license file: https://github.com/ottojo/ros_conditional_substitution/blob/43819308a242fd3cfce9b28acf04fb885723cf4f/LICENSE
[ ] Per package license files:

[x] License is OSI-approved: MIT
[x] License correctly listed in package.xmls
[x] Public source repo: https://github.com/ottojo/ros_conditional_substitution
[x] Source repository contains ROS packages (though https://github.com/ottojo/ros_conditional_substitution/blob/43819308a242fd3cfce9b28acf04fb885723cf4f/setup.py#L18 could be updated)
[ ] Each package meets REP-144 naming conventions: I'm not sure about this one.  The name here is really generic, so I think violates Global Guidelines bullet 1 and possibly bullet 4.  Also, can you please explain how this differs from https://github.com/ros2/launch/pull/734 ?

It does not differ at all from https://github.com/ros2/launch/pull/734, which is why i opened https://github.com/ros2/launch/pull/776 and converted this back to draft. I only opened this because i didn't know about the implementation in launch, which so far is only available in rolling i think, and which i only discovered just after opening this :upside_down_face:

All right.  In that case, I'm going to go ahead and close this since we can get the same functionality from that PR.


ImGui Begin/End Issue

I recently started porting my p100 super (not only) paste to the new update base and when I tried to debug my code I couldn't because of Begin/BeginChild and End/EndChild mismatch recoverable error. I didn't find any missing End/EndChild pairs in the menu. Note: the menu and the imgui widgets code has been modified
edit: my guess is that the debug window is causing the issue

it doesn't relies to base itself anyhow, probably you put ImGui::Begin into 'if' branch, when ImGui::End must be executed always
so instead of
if (ImGui::Begin(...))
{
    ...
    ImGui::End()
}

it should be
ImGui::Begin(...)
{
    ...
    ImGui::End()
}

@rollraw no. commented out all of my menu code including the T::Render etc. and the issue still persists while debugging.

the issue "solved itself" only when I commented out the whole main_menu region inside the MENU::MainWindow procedure. which isn't actually solving the issue at all, because I cannot access the menu

Note: the menu and the imgui widgets code has been modified

this is the reason. some of changes you did causing this sort of issues.


Update ex14_22_TEST.cpp

The original main() only calls the constructor Sales_data(const std::string &s) but not the assignment operator.

Fix the issue #454


Merging domains is broken if out_of_scope intent is defined by user / make default actions configurable

Rasa version: master
Issue: The out_of_scope intent is added to the list of default intents and hence its omitted when the domain is fetched as a dict because of this line
While merging multiple domains, the domains are fetched as dicts which would not contain out_of_scope intent even if the user has defined it in their domain.
@wochinge mentioned that out_of_scope intent was added to default intents because we need it for the TwoStageFallbackAction in order to know whether the user denied the suggestion. Currently there is no way to configure default actions and so the only way is to make certain things default. We could make the deny intent configurable, something like -
forms:
- two_stage_fallback:
    deny_intent: "out_of_scope"

👉 Slack thread

@melindaloubser1 @erohmensing What do you think of the suggestion for the config? Do you have other ideas?

If it was configurable, would that solve the problem with a user adding out_of_scope to their intent list?

@melindaloubser1 It would if because of that we can remove out_of_scope from the list of default intents. I am not sure if that would bring a different side effect. 😅

@m-vdb @ricwo It would be awesome if one of you could have a look at this as I think it's relevant for the RC. There is also an out of GitHub Slack thread about this 😬

I think I need to complicate this further 🙈
When we configure the action_two_stage_fallback below the forms: key in the domain, then we are configuring something as form which actually isn't a form but a loop. That's not great from a UX perspective 😬

Can we just check whether action_two_stage_fallback is listed as a key under forms, and if that's the case, we don't skip it in this line? https://github.com/RasaHQ/rasa/blob/master/rasa/shared/core/domain.py#L839


Priority Doesn't Override FIFO Execution

Conductor Version: v2.25.5
Dynomite Version: dynomite-v0.6.16
issue:
I just submitted three jobs sequentially with varying priority (1,99,50) and they appear to run in the order submitted not in order of priority. I am only running one worker and starting it after all jobs are submitted
curl -X POST --header 'Content-Type: application/json' --header 'Accept: text/plain' -d '{}' 'http://conductor-api:9138/api/workflow/priority-workflow?priority=1'
curl -X POST --header 'Content-Type: application/json' --header 'Accept: text/plain' -d '{}' 'http://conductor-api:9138/api/workflow/priority-workflow?priority=99'
curl -X POST --header 'Content-Type: application/json' --header 'Accept: text/plain' -d '{}' 'http://conductor-api:9138/api/workflow/priority-workflow?priority=50'

The priority shows correctly in the ui so it seems to be set correctly on the workflow.

@neokyle1 The priority is set when a message is added to task queues, and there's a chance the messages are polled fast enough to not enforce priority. Can you try with large number of executions please?

@kishorebanala I created 30 tasks and the result was the same. Also, I created a bash script to help reproduce. It does the following sequentially.

creates a simple task def
creates a simple workflow def
create 10 workflow instances with priority of 50
create 10 workflow instances with priority of 99
create 10 workflow instances with priority of 1
wait 3 seconds
starts a worker loop (poll then update the simple task) and prints the task data (priority, submission order, id)

Here is the log output
updated task defs
updated workflow defs
creating 10 workflow instances with priority of 50
created workflow instance datetime: Tue Apr  7 21:56:18 UTC 2020 index: 1 priority: 50 id: 5a7d4215-6556-4182-8562-3181fca80c22
created workflow instance datetime: Tue Apr  7 21:56:18 UTC 2020 index: 2 priority: 50 id: e5cc5b7d-e07e-49d8-b449-d7d7320e7b90
created workflow instance datetime: Tue Apr  7 21:56:18 UTC 2020 index: 3 priority: 50 id: c80fb5e6-3db7-4b80-b5b5-18c9f0e937bc
created workflow instance datetime: Tue Apr  7 21:56:19 UTC 2020 index: 4 priority: 50 id: 34520391-117e-4fe0-ae61-3d49dc242086
created workflow instance datetime: Tue Apr  7 21:56:19 UTC 2020 index: 5 priority: 50 id: b603502b-4ae8-4167-a23c-913f4fea6ba5
created workflow instance datetime: Tue Apr  7 21:56:19 UTC 2020 index: 6 priority: 50 id: d87890b5-c7f3-4c56-9df6-7dc04378b0b4
created workflow instance datetime: Tue Apr  7 21:56:19 UTC 2020 index: 7 priority: 50 id: 2b506ac3-783f-4636-bc09-7cbb603fd55a
created workflow instance datetime: Tue Apr  7 21:56:20 UTC 2020 index: 8 priority: 50 id: ab9aca87-089d-4e52-9bb6-89bcc090c364
created workflow instance datetime: Tue Apr  7 21:56:20 UTC 2020 index: 9 priority: 50 id: 0b8ae087-d757-4f12-b994-d0627c06fdcf
created workflow instance datetime: Tue Apr  7 21:56:20 UTC 2020 index: 10 priority: 50 id: 1c727808-64b9-49d5-b97d-ec717bfc8b03
creating 10 workflow instances with priority of 99
created workflow instance datetime: Tue Apr  7 21:56:21 UTC 2020 index: 11 priority: 99 id: dfe872fe-5790-4f61-b114-7202aa9de02e
created workflow instance datetime: Tue Apr  7 21:56:21 UTC 2020 index: 12 priority: 99 id: f154ace9-a0b6-4a6d-830a-773f111eef69
created workflow instance datetime: Tue Apr  7 21:56:21 UTC 2020 index: 13 priority: 99 id: 0428e7df-6695-4aab-9497-d20bef362296
created workflow instance datetime: Tue Apr  7 21:56:22 UTC 2020 index: 14 priority: 99 id: de6b3f8a-764b-4e86-b248-bfcdff474a5f
created workflow instance datetime: Tue Apr  7 21:56:22 UTC 2020 index: 15 priority: 99 id: 6414aa1d-42c4-448b-a336-7c8715c28051
created workflow instance datetime: Tue Apr  7 21:56:22 UTC 2020 index: 16 priority: 99 id: d6e2ac4f-552c-498f-99f2-1cb84645c21a
created workflow instance datetime: Tue Apr  7 21:56:23 UTC 2020 index: 17 priority: 99 id: 377141b3-f085-4fe4-9f37-6fcd6ec9ae45
created workflow instance datetime: Tue Apr  7 21:56:23 UTC 2020 index: 18 priority: 99 id: da580b3e-73d0-4f8b-8bc1-d73f16906e53
created workflow instance datetime: Tue Apr  7 21:56:23 UTC 2020 index: 19 priority: 99 id: 0c30a94b-9ce2-4616-b325-11ccec8bb0bf
created workflow instance datetime: Tue Apr  7 21:56:23 UTC 2020 index: 20 priority: 99 id: fcedcf68-d91c-42fa-9068-567906ce4092
creating 10 workflow instances with priority of 1
created workflow instance datetime: Tue Apr  7 21:56:24 UTC 2020 index: 21 priority: 1 id: d6ee28d6-1ffb-4267-b42e-0c2a456bfca8
created workflow instance datetime: Tue Apr  7 21:56:24 UTC 2020 index: 22 priority: 1 id: 89bb2ab3-f1bf-4e51-985c-21f36b436193
created workflow instance datetime: Tue Apr  7 21:56:24 UTC 2020 index: 23 priority: 1 id: aa7bc790-835f-4fdc-8f86-ee5161c3f342
created workflow instance datetime: Tue Apr  7 21:56:25 UTC 2020 index: 24 priority: 1 id: cbe51ce2-3e83-4838-9620-45301e74eaa1
created workflow instance datetime: Tue Apr  7 21:56:25 UTC 2020 index: 25 priority: 1 id: 8c32c0dc-5342-4403-ad5e-630a0ddff93d
created workflow instance datetime: Tue Apr  7 21:56:25 UTC 2020 index: 26 priority: 1 id: 797e7007-6d8e-42c7-b882-99dbc984943d
created workflow instance datetime: Tue Apr  7 21:56:25 UTC 2020 index: 27 priority: 1 id: 8c5f7d87-9879-4190-aa4c-5d06e9caaf66
created workflow instance datetime: Tue Apr  7 21:56:26 UTC 2020 index: 28 priority: 1 id: 9d804b8c-afd4-4d1a-aab0-df0001c2d3a0
created workflow instance datetime: Tue Apr  7 21:56:26 UTC 2020 index: 29 priority: 1 id: 60801bdd-ae35-4575-b7ce-a8e09ed9c590
created workflow instance datetime: Tue Apr  7 21:56:26 UTC 2020 index: 30 priority: 1 id: 588aca8d-f5b1-4724-97b2-858be377829a
updated task datetime: Tue Apr  7 21:56:30 UTC 2020 task_id: 5d1e9699-962b-43bb-8fee-a592f24d2c23 workflow_instance_id: 5a7d4215-6556-4182-8562-3181fca80c22 workflow_priority: 50 workflow_index: 1
updated task datetime: Tue Apr  7 21:56:32 UTC 2020 task_id: 22e78d03-feb3-46ae-8266-a818916e48cf workflow_instance_id: e5cc5b7d-e07e-49d8-b449-d7d7320e7b90 workflow_priority: 50 workflow_index: 2
updated task datetime: Tue Apr  7 21:56:33 UTC 2020 task_id: 2c59c4bf-4e89-4607-873b-6c72552668cd workflow_instance_id: c80fb5e6-3db7-4b80-b5b5-18c9f0e937bc workflow_priority: 50 workflow_index: 3
updated task datetime: Tue Apr  7 21:56:35 UTC 2020 task_id: 8183bad6-dedf-4896-b6ae-c0dd741d34e0 workflow_instance_id: 34520391-117e-4fe0-ae61-3d49dc242086 workflow_priority: 50 workflow_index: 4
updated task datetime: Tue Apr  7 21:56:37 UTC 2020 task_id: 0400098c-edca-43ce-8e94-51750e907f30 workflow_instance_id: b603502b-4ae8-4167-a23c-913f4fea6ba5 workflow_priority: 50 workflow_index: 5
updated task datetime: Tue Apr  7 21:56:39 UTC 2020 task_id: b7d0fc3a-54f6-4b0e-b216-fd52d93d9cda workflow_instance_id: d87890b5-c7f3-4c56-9df6-7dc04378b0b4 workflow_priority: 50 workflow_index: 6
updated task datetime: Tue Apr  7 21:56:41 UTC 2020 task_id: 0b6525e3-23f8-4b1c-820b-b7611807f606 workflow_instance_id: 2b506ac3-783f-4636-bc09-7cbb603fd55a workflow_priority: 50 workflow_index: 7
updated task datetime: Tue Apr  7 21:56:42 UTC 2020 task_id: 5ecf768a-9120-4593-9fb7-eda20fc5a4b9 workflow_instance_id: ab9aca87-089d-4e52-9bb6-89bcc090c364 workflow_priority: 50 workflow_index: 8
updated task datetime: Tue Apr  7 21:56:44 UTC 2020 task_id: 9834ba92-4206-4f4f-a8a8-614d42533cc5 workflow_instance_id: 0b8ae087-d757-4f12-b994-d0627c06fdcf workflow_priority: 50 workflow_index: 9
updated task datetime: Tue Apr  7 21:56:46 UTC 2020 task_id: d45b7a2c-87b2-4fbe-902d-5c5f4340849d workflow_instance_id: 1c727808-64b9-49d5-b97d-ec717bfc8b03 workflow_priority: 50 workflow_index: 10
updated task datetime: Tue Apr  7 21:56:48 UTC 2020 task_id: 6a157260-3c54-494e-aaae-d1180955a17e workflow_instance_id: dfe872fe-5790-4f61-b114-7202aa9de02e workflow_priority: 99 workflow_index: 11
updated task datetime: Tue Apr  7 21:56:49 UTC 2020 task_id: c3d359c6-0b99-493f-9d18-d9dec29113c8 workflow_instance_id: f154ace9-a0b6-4a6d-830a-773f111eef69 workflow_priority: 99 workflow_index: 12
updated task datetime: Tue Apr  7 21:56:51 UTC 2020 task_id: 3b430a71-f33a-40c8-af56-188fe2406588 workflow_instance_id: 0428e7df-6695-4aab-9497-d20bef362296 workflow_priority: 99 workflow_index: 13
updated task datetime: Tue Apr  7 21:56:53 UTC 2020 task_id: 336cbb6b-0ce1-4a07-aabc-753047b08e7f workflow_instance_id: de6b3f8a-764b-4e86-b248-bfcdff474a5f workflow_priority: 99 workflow_index: 14
updated task datetime: Tue Apr  7 21:56:55 UTC 2020 task_id: 4a9d003b-f689-413c-910e-297254c2f744 workflow_instance_id: 6414aa1d-42c4-448b-a336-7c8715c28051 workflow_priority: 99 workflow_index: 15
updated task datetime: Tue Apr  7 21:56:57 UTC 2020 task_id: f8aedcf4-74b9-4a5b-8982-4f0bfdf3a66e workflow_instance_id: d6e2ac4f-552c-498f-99f2-1cb84645c21a workflow_priority: 99 workflow_index: 16
updated task datetime: Tue Apr  7 21:56:58 UTC 2020 task_id: 78505fd8-8823-46c0-b1ab-683c26027e98 workflow_instance_id: 377141b3-f085-4fe4-9f37-6fcd6ec9ae45 workflow_priority: 99 workflow_index: 17
updated task datetime: Tue Apr  7 21:57:00 UTC 2020 task_id: 4919baa8-9333-461f-b28d-91a85388cb45 workflow_instance_id: da580b3e-73d0-4f8b-8bc1-d73f16906e53 workflow_priority: 99 workflow_index: 18
updated task datetime: Tue Apr  7 21:57:02 UTC 2020 task_id: 9e699e6d-7f20-442c-ac91-a03ef25c9173 workflow_instance_id: 0c30a94b-9ce2-4616-b325-11ccec8bb0bf workflow_priority: 99 workflow_index: 19
updated task datetime: Tue Apr  7 21:57:03 UTC 2020 task_id: f20abe4a-e9aa-4a2c-8ca6-e1412c98a5b3 workflow_instance_id: fcedcf68-d91c-42fa-9068-567906ce4092 workflow_priority: 99 workflow_index: 20
updated task datetime: Tue Apr  7 21:57:05 UTC 2020 task_id: 6be29f6f-ee00-4805-9471-c6ee3539d2bc workflow_instance_id: d6ee28d6-1ffb-4267-b42e-0c2a456bfca8 workflow_priority: 1 workflow_index: 21
updated task datetime: Tue Apr  7 21:57:07 UTC 2020 task_id: 87d2558a-ba2e-4164-b252-bd65373665c6 workflow_instance_id: 89bb2ab3-f1bf-4e51-985c-21f36b436193 workflow_priority: 1 workflow_index: 22
updated task datetime: Tue Apr  7 21:57:09 UTC 2020 task_id: a5db1312-9eb2-49f4-b025-0f776543e173 workflow_instance_id: aa7bc790-835f-4fdc-8f86-ee5161c3f342 workflow_priority: 1 workflow_index: 23
updated task datetime: Tue Apr  7 21:57:10 UTC 2020 task_id: 32a72735-42c9-4d90-ac01-08b269575490 workflow_instance_id: cbe51ce2-3e83-4838-9620-45301e74eaa1 workflow_priority: 1 workflow_index: 24
updated task datetime: Tue Apr  7 21:57:12 UTC 2020 task_id: 8a9c42f9-f992-4d89-aa1e-a0f6f7c90f81 workflow_instance_id: 8c32c0dc-5342-4403-ad5e-630a0ddff93d workflow_priority: 1 workflow_index: 25
updated task datetime: Tue Apr  7 21:57:14 UTC 2020 task_id: 03ea92d9-fb4c-4aef-a1f3-8b1446c72479 workflow_instance_id: 797e7007-6d8e-42c7-b882-99dbc984943d workflow_priority: 1 workflow_index: 26
updated task datetime: Tue Apr  7 21:57:15 UTC 2020 task_id: 8c5e8a5b-457c-413f-91cd-9252afda1270 workflow_instance_id: 8c5f7d87-9879-4190-aa4c-5d06e9caaf66 workflow_priority: 1 workflow_index: 27
updated task datetime: Tue Apr  7 21:57:17 UTC 2020 task_id: 1b458d6c-8eb4-4e76-b4bd-fc30f831e7aa workflow_instance_id: 9d804b8c-afd4-4d1a-aab0-df0001c2d3a0 workflow_priority: 1 workflow_index: 28
updated task datetime: Tue Apr  7 21:57:19 UTC 2020 task_id: 320eb5e0-7667-4bfc-8f8a-5ad44c421b00 workflow_instance_id: 60801bdd-ae35-4575-b7ce-a8e09ed9c590 workflow_priority: 1 workflow_index: 29
updated task datetime: Tue Apr  7 21:57:21 UTC 2020 task_id: 5a524410-cb7f-4a44-a83c-ad0eae783f39 workflow_instance_id: 588aca8d-f5b1-4724-97b2-858be377829a workflow_priority: 1 workflow_index: 30
no tasks are available datetime: Tue Apr  7 21:57:22 UTC 2020
no tasks are available datetime: Tue Apr  7 21:57:24 UTC 2020
no tasks are available datetime: Tue Apr  7 21:57:25 UTC 2020
no tasks are available datetime: Tue Apr  7 21:57:27 UTC 2020

Here is the script (uses the jq command for json manipulation):
#!/bin/bash
[ "${1:-}" ] || { echo >&2 "conductor url (scheme, hostname and port) must be passed as ARG 1"; exit 1; }
conductor_api_url="${1}"
type jq >/dev/null 2>&1 || { echo >&2 "jq command must be installed, see https://stedolan.github.io/jq/download/"; exit 1; }

task_defs='[{"name": "example_task","description": "used as a placeholder task","ownerEmail"<EMAIL_ADDRESS>task_def_response=$(curl --fail -sS -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --data "${task_defs}" "${conductor_api_url}/api/metadata/taskdefs" 2>&1 ) || { echo >&2 "could not upadate task defs: ${task_def_response}"; exit 1; } && echo >&2 "updated task defs"
workflow_defs='[{"description": "test priority","name": "priority_workflow","outputParameters": {},"ownerEmail": "user@example.com","schemaVersion": 2,"version": 1,"tasks": [{"name": "example_task","taskReferenceName": "example_task","inputParameters": {"index": "${workflow.input.index}"},"type": "SIMPLE"}]}]'
workflow_def_response=$(curl --fail -sS -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' --data "${workflow_defs}" "${conductor_api_url}/api/metadata/workflow" 2>&1 ) || { echo >&2 "could not upadate workflow defs: ${workflow_def_response}"; exit 1; } && echo >&2 "updated workflow defs"

index=0
priority="50"
echo >&2 "creating 10 workflow instances with priority of ${priority}"
for i in {1..10}; do
	index=$((index+1))
	result="$( curl --fail -sS -X POST --header 'Content-Type: application/json' --header 'Accept: text/plain' -d "{\"index\": ${index}, \"priority\": ${priority}}" "${conductor_api_url}/api/workflow/priority_workflow?priority=${priority}" )" || { echo >&2 "could not start workflow"; exit 1; } && echo >&2 "created workflow instance datetime: $(date -u) index: ${index} priority: ${priority} id: ${result}"
done
priority="99"
echo >&2 "creating 10 workflow instances with priority of ${priority}"
for i in {1..10}; do
	index=$((index+1))
	result="$( curl --fail -sS -X POST --header 'Content-Type: application/json' --header 'Accept: text/plain' -d "{\"index\": ${index}, \"priority\": ${priority}}" "${conductor_api_url}/api/workflow/priority_workflow?priority=${priority}" )" || { echo >&2 "could not start workflow"; exit 1; } && echo >&2 "created workflow instance datetime: $(date -u) index: ${index} priority: ${priority} id: ${result}"
done
priority="1"
echo >&2 "creating 10 workflow instances with priority of ${priority}"
for i in {1..10}; do
	index=$((index+1))
	result="$( curl --fail -sS -X POST --header 'Content-Type: application/json' --header 'Accept: text/plain' -d "{\"index\": ${index}, \"priority\": ${priority}}" "${conductor_api_url}/api/workflow/priority_workflow?priority=${priority}" )" || { echo >&2 "could not start workflow"; exit 1; } && echo >&2 "created workflow instance datetime: $(date -u) index: ${index} priority: ${priority} id: ${result}"
done
sleep 3
while true; do
	poll_response=$( curl --fail -sS --header 'Content-Type: application/json' --header 'Accept: application/json' "${conductor_api_url}/api/tasks/poll/example_task" 2>&1 ) || { echo >&2 "could not poll task: ${poll_response}"; exit 1; } && \
	if [ -z "${poll_response}" ]; then
	      echo >&2 "no tasks are available datetime: $(date -u)"
	else
		# echo "${poll_response}" | jq
		task_id="$(echo "${poll_response}" | jq -er '.taskId')" || { echo >&2 "could extract .taskId"; exit 1; } && \
		workflow_instance_id="$(echo "${poll_response}" | jq -er '.workflowInstanceId')" || { echo >&2 "could extract .workflowInstanceId"; exit 1; } && \
		workflow_priority="$(echo "${poll_response}" | jq -er '.workflowPriority')" || { echo >&2 "could extract .workflowPriority"; exit 1; } && \
		workflow_index="$(echo "${poll_response}" | jq -er '.inputData.index')" || { echo >&2 "could extract .inputData.index"; exit 1; } && \
		success_payload="$(echo "${poll_response}" | jq -e '.status = "COMPLETED"')" || { echo >&2 "could not update task"; exit 1; } && \
		update_response="$( curl --fail -sS -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d "${success_payload}" "${conductor_api_url}/api/tasks"  2>&1 )" || { echo >&2 "could not update task: ${update_response}"; exit 1; } && \
		echo "updated task datetime: $(date -u) task_id: ${task_id} workflow_instance_id: ${workflow_instance_id} workflow_priority: ${workflow_priority} workflow_index: ${workflow_index}"
	fi
	sleep 1
done

Yes, I meet the same issue. The priority doesn't work as expected!

@apanicker-nflx @kishorebanala This is a critical issue. Could you help to take a look at this issue when you get a chance? Thanks in advance.

I think this is still an issue. I will re-test once 3.0 is released...

What persistence layer and queue implementation are being used?

@kishorebanala I tested this with conductor 3.0.2 and it is still broken (every time). The best I can tell, dyno-queues is using the ZRANGEBYSCORE redis function to pop messages off the queue. After queuing some tasks using the above script, I checked the result of that command manually:
❯ redis-cli ZRANGEBYSCORE conductor_queues.test.QUEUE.example_task.c -inf +inf WITHSCORES

 1) "a87fd0ea-d346-4a13-925e-03a701edc3fc"
 2) "1620078645956.5"
 3) "438ab46d-3f58-44c9-8196-6f7c9829707f"
 4) "1620078646166.5"
 5) "5aeeb322-133e-4c44-970e-33f29d021e4c"
 6) "1620078646372.5"
 7) "7a493726-665d-4a45-8bba-b016b99bf6bd"
 8) "1620078646571.5"
 9) "8a40a161-05f8-4b2c-9cf5-d2ea305726ca"
10) "1620078646775.5"
11) "0c49b69a-d818-4033-bd94-4b08d68c1546"
12) "1620078646972.5"
13) "55ccb2c5-2aa8-4a2c-8400-b2e56f4f6354"
14) "1620078647185.5"
15) "aa7425f4-3a1b-4b0b-b2f0-5ca90402d623"
16) "1620078647389.5"
17) "370c43c3-9b84-42df-80a7-587bc59e7f46"
18) "1620078647612.5"
19) "29d2a1a5-2fb3-4a44-bb59-76fb3b38e558"
20) "1620078647815.5"
21) "edbd8129-dd5a-4ec0-bb42-d5d941ccf137"
22) "1620078648012.99"
23) "bcb8cf9a-1fd6-43af-b19d-2eb8b78c21f3"
24) "1620078648225.99"
25) "d1de6492-80be-4762-a6a6-31d9aae3bbd9"
26) "1620078648436.99"
27) "56b8bed8-b0c1-4125-a4e7-bd0d21d4efe4"
28) "1620078648636.99"
29) "f2d38d32-ffe0-4121-80bf-023b7eef2640"
30) "1620078648832.99"
31) "51be056f-07e6-4130-a297-e8c2bda77740"
32) "1620078649035.99"
33) "f232ce57-127b-4cd4-8d6a-c505f99d2924"
34) "1620078649232.99"
35) "c74d4226-afe7-4171-8894-c8c6215b29bd"
36) "1620078649431.99"
37) "b5b5360b-d036-4563-b138-3853dfb83829"
38) "1620078649635.99"
39) "f2b316fb-23b1-4c04-8541-815e93ad5d6c"
40) "1620078649836.99"
41) "31c5b088-401a-4242-97bc-0957b242940c"
42) "1620078650034.01"
43) "57a4e1ee-9f65-4173-965d-32313a243217"
44) "1620078650292.01"
45) "001a1a16-93e7-46cb-82eb-5859fbbee993"
46) "1620078650535.01"
47) "82c3c7fa-e909-406c-9968-a5224f9704c3"
48) "1620078650731.01"
49) "d08d79ee-522b-41e9-9d98-21415aa847c4"
50) "1620078650952.01"
51) "3424b8fc-f24e-4ad3-8ae0-682df6ee227d"
52) "1620078651152.01"
53) "a33ff9f4-8fe6-40ca-b12f-0341827bf31e"
54) "1620078651352.01"
55) "f701c64f-34f0-467d-a988-a05c91293523"
56) "1620078651570.01"
57) "c989b899-f775-4bf2-8f7b-fa0f06640f6c"
58) "1620078651774.01"
59) "ab9dab03-55ec-4534-b3c9-1d01befe766f"
60) "1620078651970.01"

It looks like the SCORE value is comprised of the task's creation time as an epoch millisecond plus its timeout millisecond value plus the priority value as its decimal. It seems to me that the priority would only ever have an affect on the return order (and thus dequeue order) if multiple tasks' millisecond value were exactly the same, which is extremely unlikely.
I can't really tell if this is the intended design of dyno-queues' priority feature or not.

tried priority in V3.7.3 with mysql persistence, this issue still exists. Any suggestion? @kishorebanala


Revert "Update fsouza/go-dockerclient to the last version."

Reverts remind101/empire#614

Did it break something?

@phobologic I'm using this branch to test with my own ECS cluster. So far I had one stall emp run (the very first one), I can't reproduce now, it's working everytime with a few different commands.

:+1:

We're never merged this so I'm assuming the update is ok? :)
Ok to close?

Yeah we can close.


[TechDocs] Sync entityRef and path updates

Hey, I just made a Pull Request!
Update the path at the same time the entityRef is updated to prevent the path update from happening before the entityRef is updated and we end up dispatching an unnecessary API request to the old entity name with the new path which results in 404 error.
:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[ ] All your commits have a Signed-off-by line in the message. (more info)

Thanks @camilaibs!  I attempted to reproduce the type of error we were seeing internally in this repo, but had trouble doing so in isolation.  While this seems like a reasonable clean up, I'm hesitant to merge it in if it's not strictly necessary (especially given the small API addition, as Emma mentioned).
Whaddya say if I close this for now, but we can re-evaluate if this starts to more actively cause issues?

Thanks @camilaibs! I attempted to reproduce the type of error we were seeing internally in this repo, but had trouble doing so in isolation. While this seems like a reasonable clean up, I'm hesitant to merge it in if it's not strictly necessary (especially given the small API addition, as Emma mentioned).
Whaddya say if I close this for now, but we can re-evaluate if this starts to more actively cause issues?

Closing 🙌🏻


Initial setup

@Mottie Thank you very much for this work. I couldn't almost have done it better! Great in which detail you've looked at the style of my repositories. 👍
While we've discussed about the system requirements before, they're not complete yet (at least from my point of view). I'm quite sure that we'll go ahead with the JSON database, but I'd like to finalize the system requirements for this way before. Currently, there's no specific mention of how the JSON database build should be set up in detail, due to the fact that this way wasn't clearly defined yet.
Therefore I've moved this initial commit into a new branch so that we can finalize the specifications first – as I expect that the master branch should always be stable. So let's continue discussing in the concept issue.
About the licensing question, I think MIT is essential for usage here. But for the code that generates the mapping itself, I think MIT isn't required. But because I'm not absolutely sure how to realize a dual-licensed project from a legal point of view, we can let it as it is.
Wouldn't it make sense to provide the sources in the metadata object instead of comments? When they would be entered by users manually without providing sources we could fill in "provided by users" or something similiar.
There's one thing you could do in the meantime; please update the version in the package.json to 0.1.0 by --amend your previous commit – I think it will take some time until we really have a stable version.

Another update.

A settings.js file was added to the src folder.

This might not be the ideal location, but for initial testing it makes changing some settings easier.
"unicode-format" set to "\\u{value}" - double backslash is needed during the build process to prevent errors. The JSON in the dist folder will have a single backslash.
"output-format" set to 4 - Format of the JSON in the dist folder. The value is used in the JSON.strinify() space argument.
"current" - contains all the lower case characters to process. These values are only used to process diacritic data (non-diacritic characters are ignored). Ideally this should be set to all the diacritics contained within the language files, but we don't have a full list yet 😉.

The equivalents are now automatically generated. I noticed you starred the unorm repository, but I found that ES6 normalize() does the same thing. And since it's only needed during the build process, we probably don't have to worry about supporting older browsers.

I think that covers the changes I made.

Thank you! Awesome that you've found a way to generate equivalents automatically. :tada:
I'd love to dig into the build in detail, but I'm missing some documentation. Could you please update the system requirements in the issue and the spec branch to reflect the current state?
Especially:

Meaning behind html-entities.js file at resources
Meaning behind settings.js, especially "current" and "unicode-format" properties. What a task they fulfill and how they are used in the build
That diacritics.json is a collection of all languages and unicode characters will be encoded
Equivalents are generated automatically and not part of the database source files anymore

Basically I think we'll need to separate the spec into two sections: dist and src. That way we can better document the file structure.
I've a few questions about the way of implementation:

What's the reason for writing "\u00df" into the property "base" and not "ß" in the source file? Or maybe make it optional if it's the same? I see that it's already excluded in the SysReq
Why not list "current" in settings.js as an array? This way we could make the list vertically rather than horizontally (easier to read from my point of view)
What should be done if the "list" and "sources" sources are the same? I'd vote to just use a simple array (no object) in all cases
I've seen you're including many equivalents – not just diaresis – and I agree that it isn't wrong to generate them. But, from a JS point of view we only need the equivalents and not entities (because they are converted in the DOM), so we'd need to have a way to ignore unnecessary equivalents. Maybe all equivalents starting with "%" or "&"? What do you think? Would then be a part of the server side component.
As the only Grunt specific tasks are to strip JSON comments and to clean/move files, I'd vote to refactor the build to a normal Node.js build. This would make it

cleaner
possible to add JSDOC comments
possible to use one or multiple ES6 build class(es)
possible to refactor long functions (makeEquivalents, buildDis and maybe decodeUnicode) into smaller parts
...

When I understand the build in detail, especially the meaning behind "current" (when the documentation is updated), I can say more about Ideally this should be set to all the diacritics contained within the language files, but we don't have a full list yet. From my point of view this would mean that we have a redundant database. If this is true(?), I'd vote to generate them from the source files automatically as I'm quite sure this will cause much additional work as users will not enter them (or not enter them correctly)

Could you please update the system requirements in the issue and the spec branch to reflect the current state?

Sure! I just wanted to get something in place as for me concepts/specs are great, but I like to see examples 😁

Meaning behind html-entities.js file at resources

As I stated before, someone might use this resource to parse HTML and since ü can be represented as ü as well as a digit and hex value, it would be best to include the entity as well. The html-entities.js file was created rather quickly by copy & pasting the rendered page at https://dev.w3.org/html5/html-author/charref, I then used my editor to add the formatting. I remaining process I used is documented within the file.

Meaning behind settings.js, especially "current" and "unicode-format" properties. What a task they fulfill and how they are used in the build

I created the settings.js file as a temporary method to pass the "current" list of diacritics to process into the script. I say temporary because I was only using it to build the example, and I wanted an easy way to modify the list without needing to edit the grunt file.
Eventually we'll need to refine the process. I still don't know if we should provide everything to the user or build the data per request.
As for the "unicode-format", I found that different languages/situations use a different format. At the top of the file you'll see:

C/C++/Javascript: \\u{value}   (use double backslash or errors occur)

Python:           u"\u{value}"
Wikipedia         U+{value}

So I suspect there might be some other formats we haven't considered.

That diacritics.json is a collection of all languages and unicode characters will be encoded

Yep, as I said though, I wasn't sure if we're providing already processed information or if it is generated on request.

Equivalents are generated automatically and not part of the database source files anymore

I don't think I'm done yet. Not all equivalents combinations have been covered, so we may still need to add an "extra" equivalents file. For example, if you look at this page, the second example in the table is a "q" with a combining diacritic dot both above and below. The order can be switched, so q + ◌̇ + ◌̣ is visibly equivalent to q + ◌̣ + ◌̇. This extra file might need to cover these cases. And if you think it's uncommon, just look at the Vietnamese alphabet.

What's the reason for writing "\u00df" into the property "base" and not "ß" in the source file?

No particular reason. I've just gotten use to always escaping unicode characters... my OCD made me do it!

Why not list "current" in settings.js as an array?

We could. I just don't think typing out an array is as efficient as making a string and using .split(). And as I said, the file was only meant to be temporary until we figure out more stuff about the library. I can definitely split it up to make it easier to read.

What should be done if the "list" and "sources" sources are the same?

Did you mean "list" and "mapping"? Yeah, I'd like to keep a simple array.

from a JS point of view we only need the equivalents and not entities... so we'd need to have a way to ignore unnecessary equivalents

What if the equivalents was an object?
"\u00fc": {
    "mapping": {
        "base": "u",
        "decompose": "ue"
    },
    "equivalents": {
        "unicode": "\u00FC",
        "html_decimal": "ü",
        "html_hex": "ü",
        "encoded_uri": "%C3%BC",
        "html_entity": "ü",
        "unicode_combinations": [
            "u\u0308"
        ]
    }
}

I'd vote to refactor the build to a normal Node.js build

I'm not 100% sure what you mean. I'm all for refactoring! Also, I'd need to learn how to make "proper" comments for JSDOCs.

From my point of view this would mean that we have a redundant database.

I was envisioning the user picking the languages they needed along with the unicode format, if they needed it. The library would then generate a file, or files with their settings they use... the "current" setting list would actually come from the src/{lang}/{lang}.json file of the languages they picked.
I guess I should have explained my thoughts better, but I get distracted easy 😉.

Sure! I just wanted to get something in place as for me concepts/specs are great, but I like to see examples

Building code while defining the spec helps finding difficulties and makes it easier to get an overview. But this also means more effort and it's up to you if you'd like to spend time on something that might need to get refactored again. I have a small time window, therefore I personally prefer to finish the spec in theory first.

As I stated before, someone might use this resource to parse HTML

So based on this file you're generating the non-diaresis (combining diacritics) equivalents?

to pass the "current" list of diacritics to process into the script

If I understand your build script correctly, you're  generating the equivalents for each character inside this "current" list. Then you're iterating over the language files and set the equivalents property to one of the generated equivalents properties. This means, you're generating the whole Latin table, even they aren't used? From my point of view that would need to get refactored to just generate equivalents on demand, so the equivalents will be generated in the loop itself and not passed as a parameter.
Am I right with what I'm thinking? Do you agree?

As for the "unicode-format", I found that different languages/situations use a different format

We'll need one format that can be generated to each output format (JS, Wikipedia, Python, ....) on request. But, this should be done by the server side component, as users won't build the database. I'd vote to just generate the {value} part instead of e.g. U+{value}. This way we could add the prefix (in this case U+) in the server side component.

Yep, as I said though, I wasn't sure if we're providing already processed information or if it is generated on request.

We're generating a file that can be converted to each output format and to each filter/order. This means that we just need the diacritics.json file in my opinion, as the language files are already a filtered output (by language).

I don't think I'm done yet. Not all equivalents combinations have been covered

What's your solution suggestion?

No particular reason. I've just gotten use to always escaping unicode characters... my OCD made me do it!

I think we can just make it optional in this case. What do you think? Btw: What is OCD? :bowtie:

Did you mean "list" and "mapping"? Yeah, I'd like to keep a simple array.

Yes, that's what I meant. But I meant to keep a simple array "sources": [], so there wouldn't be an object with "list" and "mapping" due to the fact that they may be the same. What do you think?

What if the equivalents was an object?

Great idea! I'd vote to use this approach, as this makes it more understandable and filterable.

I'm not 100% sure what you mean. I'm all for refactoring! Also, I'd need to learn how to make "proper" comments for JSDOCs.

What I mean is to dispense on Grunt. The only Grunt tasks the current implementation uses is to strip json comments and delete files. We won't need Grunt for our build, as all other functionalities are implemented independently. Therefore I'd refactor the current build to a simple "build.js" file, where we can improve it with above named advantages. You can then call it using $ node build.js or using a npm shortcut defined in scripts in package.json, e.g. $ npm build.

I was envisioning the user picking the languages they needed along with the unicode format

Yeah, but not in this repository, as stated above. Users won't use the data provided here. It's just the database the server side component will use to provide the generated mapping information by specified parameters. This repository should be the place to generate a common diacritics.json file based on user created diacritic mapping information.
I hope it's clear now. I think that's the reason why I still don't understand the manual creation of current.

I personally prefer to finish the spec in theory first.

Okay, I'll spend some more time working on the spec. I'll update it with what we've discussed so far.

you're generating the non-diaresis (combining diacritics) equivalents?

LOL, oops, yeah I guess we'd need to include that too.

you're generating the whole Latin table, even they aren't used?

Yes, for now. We only have German set up as an example. Generating the entire Latin table will minimize our initial work, otherwise we would need to update the "current" list after each language was added. Since we plan to build the data on demand, this will get changed.

we could add the prefix (in this case U+) in the server side component.

Ok, that sounds good.

Not all equivalents combinations have been covered... What's your solution suggestion?

I think it may need to be manually coded. If you remember from my initial attempt, the "ä" can also be represented by the "Cyrillic small letter a with diaeresis (U+04D3)". I don't know of any place that lists visual equivalents in this way.

make {escaping unicode characters} optional in this case. What do you think? Btw: What is OCD?

We can make it optional as long was we emphasize to the user that the data much be loaded as UTF8.
OCD is an abbreviation for Obsessive Compulsive Disorder. I don't actually have it, but I have some traits 😉.

What if the equivalents was an object?... Great idea!

Ok, I'll start updating the spec

We won't need Grunt for our build

Ok, I haven't used npm as a build script before, but I just read up on it and it doesn't seem too bad. I'll see what I can do.

It's just the database the server side component will use to provide the generated mapping information by specified parameters

So, if we're using the data generated with this library on the server-side, won't we need to generate the entire Latin table? Now I'm confused again LOL. I'll look over the spec in more detail tomorrow, when my brain is working better.

Okay, I'll spend some more time working on the spec. I'll update it with what we've discussed so far.

Great!

LOL, oops, yeah I guess we'd need to include that too.

Absolutely!

Ok, that sounds good.

Okay, I'll spend some more time working on the spec. I'll update it with what we've discussed so far.

Great!

LOL, oops, yeah I guess we'd need to include that too.

Absolutely!

Ok, that sounds good.

Great.

I think it may need to be manually coded. If you remember from my initial attempt, the "ä" can also be represented by the "Cyrillic small letter a with diaeresis (U+04D3)". I don't know of any place that lists visual equivalents in this way.

I thought this is what you're doing here?

We can make it optional as long was we emphasize to the user that the data much be loaded as UTF8.

Sorry, we've misunderstood in that case. I didn't meant to make encoding optional, I meant to make "base" optional as it's the same in case of ß. What do you think?

OCD is an abbreviation for Obsessive Compulsive Disorder. I don't actually have it, but I have some traits :wink:

Oh yeah, the first thing what I was thinking when I saw this was the Kevin Spacey at Scrubs :laughing:

Ok, I haven't used npm as a build script before, but I just read up on it and it doesn't seem too bad. I'll see what I can do.

Don't worry, you're not on your own. I'll convert the build to a Node.js one to make thing easier.

So, if we're using the data generated with this library on the server-side, won't we need to generate the entire Latin table? Now I'm confused again LOL. I'll look over the spec in more detail tomorrow, when my brain is working better.

Okay, I'll spend some more time working on the spec. I'll update it with what we've discussed so far.

Great!

LOL, oops, yeah I guess we'd need to include that too.

Absolutely!

Ok, that sounds good.

Great.

I think it may need to be manually coded. If you remember from my initial attempt, the "ä" can also be represented by the "Cyrillic small letter a with diaeresis (U+04D3)". I don't know of any place that lists visual equivalents in this way.

I thought this is what you're doing here?

We can make it optional as long was we emphasize to the user that the data much be loaded as UTF8.

Sorry, we've misunderstood each other in that case. I didn't meant to make encoding optional, I meant to make "base" optional as it's the same in case of ß. What do you think?

OCD is an abbreviation for Obsessive Compulsive Disorder. I don't actually have it, but I have some traits :wink:

Oh yeah, the first thing what I was thinking when I saw this was Kevin Spacey at Scrubs :laughing:

Ok, I haven't used npm as a build script before, but I just read up on it and it doesn't seem too bad. I'll see what I can do.

Don't worry, you're not on your own. I'll convert the build to a Node.js one to make things easier when the spec is done.

So, if we're using the data generated with this library on the server-side, won't we need to generate the entire Latin table? Now I'm confused again LOL. I'll look over the spec in more detail tomorrow, when my brain is working better.

Let me clarify the situation. In this repository we're going to collect diacritic mapping information with the help of native speakers. For contributors it should just be possible to create their json/js file in this repository directly on GitHub and send the PR. They shouldn't need to do any command line things in order to submit their mapping information. That simple setup will (hopefully) lead to active contributions.
This means that we can't require users to run e.g. $ grunt dist at the and, since this would require to clone, install dependencies and run things. What we'll do is implementing a build bot that will run our build on Travis CI and commits changes directly to a dist branch in this repository. Therefore once you merge something or you commit something yourself the dist branch will be updated automatically. Some people already doing this to update their gh-pages branch when something changes in the master branch (e.g. this script). Since we won't versioning the source files it's allright to always have the latest files in this branch.
So, now that we've clarified how users contribute and how files are generated the last point to clarify is what actually should be generated. As the filtering, sorting and all other customization of the output is part of the server-side component we just need to generate one json file that contains all data in a common structure. The server-side component can then just determine what a user needs based on GET parameters in the URL, fetch the json from this repository (dist branch), modify the json (sort, filter, etc.) and output it as JSON. Every setting a user can choose (e.g. the unicode prefix) would then be a parameter for the server-side component. But not part of our build in this repository.
Clarified?

we could add the prefix (in this case U+) in the server side component... Ok, that sounds good.

Actually, for example when we normalize ö into o\u0308 we will need to return an array instead of that combination: ["o", "0308"]; but then the user can't tell if the second value is really hex or decimal. Maybe we can just return \u0308 and use a simple string replace method (server side) if the user want any other format?

I don't know of any place that lists visual equivalents in this way... I thought this is what you're doing here?

As much as I understand it now, the normalize function only converts a character with a diacritic into a letter plus a combining diacritic and vice versa. It doesn't know what characters are visual equivalents. For example, I just tried this code:
// Cyrillic small letter o with diaeresis
'\u04e7'.normalize("NFD");

The result was \u043E\u0308 which is a "Cyrillic small letter o" plus a combining diaeresis, which happens to be another visual equivalent... ugh LOL.
But, I think we can apply the same normalize functions to each visual equivalent that a user provides, then generate those alternatives. So the user provides \u04e7, and we use the normalization code to include \u043E\u0308 as another equivalent.

Don't worry, you're not on your own

:fireworks:

Clarified?

Ok, I think I pretty much understand the goal here.
I've just updated the spec branch! Whew.

use a simple string replace method (server side) if the user want any other format?

I agree with that!

It doesn't know what characters are visual equivalents

Summarized wen can generate diacritics into a letter plus a combining diacritic and vice versa, but we can't generate all visual equivalents?  😞 I'm afraid I have to say that I lost overview about how we can determine all visual equivalents and why we have so much combinations in your example with the "ö" character.
I'm curious how you specified that in the spec.

I've just updated the spec branch! Whew.

I'm looking forward to see it. If you haven't received my mail you'll have to check your spam folder and set the IP of my vServer to your whitelist. Gmail is driving me crazy...

@Mottie May I ask you to take a look at the recent changes? And maybe adjust the build to generate the missing equivalents.extras? I'm done for today.

Ok, no problem!

So from working on this I noticed a problem. The "extras" entry should be an array of objects in case there is more than one normalization form. The result will look like this:
"ü": {
    "mapping": {
        "base": "u",
        "decompose": "ue"
    },
    "equivalents": {
        "unicode": "\\u00FC",
        "html_decimal": "ü",
        "html_hex": "ü",
        "encoded_uri": "%C3%BC",
        "html_entity": "ü",
        "extras": [
            {
                "unicode": "u\\u0308",
                "html_decimal": "ü",
                "html_hex": "ü",
                "encoded_uri": "u%CC%88"
            }
        ]
    }
}

Maybe it would be better to change the structure of the "equivalents" to be an array... something like this:
"ü": {
    "mapping": {
        "base": "u",
        "decompose": "ue"
    },
    "equivalents": [
        {
            "unicode": "\\u00FC",
            "html_decimal": "ü",
            "html_hex": "ü",
            "encoded_uri": "%C3%BC",
            "html_entity": "ü",
        }, {
            "unicode": "u\\u0308",
            "html_decimal": "ü",
            "html_hex": "ü",
            "encoded_uri": "u%CC%88"
        }
    ]
}

Or would you prefer to keep the "extras" separate as in the first example?

@Mottie Sorry, but I don't understand what you mean with "more than one normalization form"? 😟

If you look at figure 3 from this page, you'll see that the Angstrom symbol has two normalization forms:

So we will need entries for both. That's why I think an array would be better.

I agree to create an equivalents object for each normalization form containing above named formats (e.g. encoded_uri). But don't we need to add the normalized character without any encoding too?

don't we need to add the normalized character without any encoding too?

Sure, that would be an easy addition.
I'll update the spec.

Ok, I've updated the build. Let me know if I need to fix anything.

@Mottie Fantastic! I've just implement the deployment task. Once we merge something into master the dist branch will be automatically updated.


Separated out tests in a different GHA workflow

Description
Separated out test, integTest and yamRestTest in a different GHA workflow
Issues Resolved
Part of https://github.com/opensearch-project/opensearch-ai-flow-framework/issues/73
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Closing this PR, as the plugin is quite lightweight now and doesn't take much time to process. This PR has some good discussion for the CI. We can always revisit this later later.


Jsmi374

I have added the analysis for my second city to the MD file.
@nquarder @eculler

Nice work @jsmi374


LWRP Support?

Im wondering if its working on LWRP, or any plans?

No, it only supports the legacy pipeline. I strongly recommend using VFX Graph for LWRP/URP/HDRP: https://github.com/keijiro/Smrvfx


Timelapse script process

I was curious about the image renaming process in the timelapse.sh script, where it renames each image from 0001.jpg to xxxx.jpg. Is this necessary for the video processing to sequence the images correctly?
My concern is that this removes the timestamps from the image filename (and updates the file creation/modified date), so it can make it more difficult to go find an image from a certain time of night.
I was thinking of saving an image from the same time each night (say midnight), and then after a year, compile those images into a video to show the precession of the stars relative to the earth.

Hey!
I like the idea of making a video using one image per night.
The reason why images are renamed to a sequence is because avconv/ffmpeg required a filename with a number between 0 and 9999 in order to generate a video output.
I would suggest making a copy of the image captured around midnight before the timelapse starts and store it in a ‘precession’ directory.
That way you only need to keep one image per night and that won’t fill up the SD card.

Why not make softlinks in timelapse.sh instead of copies of the images. That saves on disk space.

Well that’s a pretty good idea.
1 Make soft links on all images in the directory
2 Run avconv
3 Remove soft links
4 Find image close to midnight and store it somewhere else because archives are only stored for a certain number of nights (set in config.sh)

Ah, that makes sense. I'll try to figure out the softlinks scripting and image finding as I have time. It might take me a while as I'm not a Linux expert. ;)

I pushed a change inspired by cbassa's pull request to address the issue. #4 in the list would need to be treated separately.

Thanks Thomas! Still working on #4 slowly in my free time.

This is what I have so far:
find /home/pi/allsky/images/20190817/ -maxdepth 1 -type f -newermt "2019-08-18 00:00:00" \! -newermt "2019-08-18 00:00:45"  | sort -tr |head -1 | xargs -I {} cp -n {} /home/pi/allsky/Precession/
It finds files in a specific directory, and copies the first file between the two times over to the Precession directory. I think my next step is to make it compatible with a script. It might also be made simpler by using the -newermt -1 or something similar to find things made in the last 24 hours, which is when this script would most logically be running. Anyway, I'll get to that later.

Hi, great piece of work!
I'd like to bring the topic of soft links together with using an external / mounted file system for storing the images: I am using a mounted folder from my NAS, but CIFS apparently doesn't support symbolic links. So I changed the directory for the soft links to a temporary directory (/home/pi/allsky/temp), have all symlinks created in this directory and after creating the video and saving it to the NAS, the local temp folder is deleted.
So this is "my" timelapse.sh:
`#!/bin/bash
source /home/pi/allsky/config.sh
source /home/pi/allsky/scripts/filename.sh
cd /home/pi/allsky/
RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m' # No Color
if [ $# -lt 1 ]
then
echo -en "${RED}You need to pass a day argument\n"
echo -en "    ex: timelapse.sh 20180119${NC}\n"
exit 3
fi
echo -en "* ${GREEN}Creating symlinks to generate timelapse${NC}\n"
mkdir /home/pi/allsky/temp/sequence/
find images, make symlinks sequentially and start avconv to build mp4; upload mp4 and move directory
find "/home/pi/allsky/images/$1" -name ".$EXTENSION" -size 0 -delete
ls -rt /home/pi/allsky/images/$1/.$EXTENSION |
gawk 'BEGIN{ a=1 }{ printf "ln -sv %s /home/pi/allsky/temp/sequence/%04d.'$EXTENSION'\n", $0, a++ }' |
bash
ffmpeg -y -f image2 -r 25 -i temp/sequence/%04d.$EXTENSION -vcodec libx264 -b:v 2000k -pix_fmt yuv420p images/$1/allsky-$1.mp4
if [ "$UPLOAD_VIDEO" = true ] ; then
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$MP4DIR" -e "set net:max-retries 1; put images/$1/allsky-$1.mp4; bye"
fi
echo -en "* ${GREEN}Deleting sequence${NC}\n"
rm -rf /home/pi/allsky/temp/sequence
echo -en "* ${GREEN}Timelapse was created${NC}\n"`

I finally got around to finishing the script for grabbing the midnight photos every night:
It searches for the first photo taken between midnight and 45 seconds after midnight, and then copies it to a Precession folder in the images directory. On my own installation, I added this to the EndOfNight.sh script. Feel free to include if you like it.
`#!/bin/bash
source /home/pi/allsky/config.sh
source /home/pi/allsky/scripts/filename.sh
cd  /home/pi/allsky/scripts
LAST_NIGHT=$(date -d '24 hours ago' +'%Y%m%d')
MIDNT=$(date -d '12am today' +'%c')
MIDNT2=$(date -d '12am today+45 seconds' +'%c')
THIS_DAY=$(date -d 'today' +'%Y%m%d')
Copy image just after midnight to Precession folder
if [[ $PRECESSION == "true" ]]; then
echo -e "Finding first image after Midnight for precession\n"
mkdir -p /home/pi/allsky/images/Precession/
find /home/pi/allsky/images/$LAST_NIGHT/ -maxdepth 1 -type f -newermt "$MIDNT" ! -newermt "$MIDNT2" | sort -tr |	head -1 | xargs -I {} cp -n -p {} /home/pi/allsky/images/Precession/$THIS_DAY.jpg
if [[ $UPLOAD_PRECESSION == "true" ]] ; then
OUTPUT="/home/pi/allsky/images/Precession/$THIS_DAY.jpg"
#"find /home/pi/allsky/images/Precession/ -maxdepth 1 -type f -newermt "$MIDNT""
lftp "$PROTOCOL"://"$USER":"$PASSWORD"@"$HOST":"$PRECESSION_DIR"
-e "set net:max-retries 1; put $OUTPUT; bye"
fi
echo -e "\n"
fi`

This issue appears to have been resolved (thanks Brian).  If you think it might still be relevant, please test it with the newest version and submit a new issue if needed.  Thanks


Kiwix.KiwixJS version <IP_ADDRESS>

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.1 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

No problem installing the package here:

I can't reproduce this install failure locally. I have already tried closing and re-opening this PR.

@ItzLevvie Can you flag this for a dev to look at? It's failing validation, but it installs fine from the manifest locally with no errors. I've tried closing and re-opening this PR three times now.

@wingetbot run

@ItzLevvie Thanks for taking a look. As it's still failing, and it is signalling some kind of certificate error, I'll try rebuilding the package, in case there was some issue I didn't detect during signing. What is confusing is that it installs fine on my machine, with no certificate complaint.


Stored XSS security bug

Steps to reproduce:
Use the chrome console to remove the clientside  URL check:

Enter malicous "javascript:" URL

Send link around to phish people using defacements or CSRF to other sites.

Reagrding a fix, that would obvisouly require a backend. Referncing this. https://github.com/bauripalash/fossurl/issues/2#issuecomment-440883946
Ill say to just focus on the re code.
Good luck

I disagree with the parent comment, it would be pretty simple to fix this on the client-side.
Instead of arbitrarily redirecting to the given URL, you could validate that the url starts with http:// or https://, and not redirect if it doesn't match.

@AntonyGarand In the new version I'm using regular expression to validate URLs before creating new short urls

@AntonyGarand In the new version I'm using regular expression to validate URLs before creating new short urls

He means to validate before redirecting instead of when creating the link, That's true, I haven't thought of that.
Thanks for the input @AntonyGarand

@p410n3 @AntonyGarand I'm not sure about efficiency but I added regex to verify the URL before redirecting https://github.com/bauripalash/fossurl/commit/5537e32384876e10c48edda603cc436876d78be8


fix validation of FIIntermediaryFI tag

FedWireMessage contained a bug that would require BeneficiaryIntermediaryFI to be present even if FIIntermediaryFI was not. This PR updates isFIIntermediaryFIValid() to match the spec for tag {6200} (FIIntermediaryFI):

If present, {4000}, {4100} and {4200} are required.

PS I've started on a unit test to verify the fix based on the payload provided in the issue. I had to leave for an appointment, but I'll get back to it afterwards.￼

Codecov Report

Merging #95 into master will decrease coverage by 1.94%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #95      +/-   ##
==========================================
- Coverage   88.55%   86.60%   -1.95%
==========================================
  Files          72       72
  Lines        5792     4852     -940
==========================================
- Hits         5129     4202     -927
+ Misses        413      400      -13
  Partials      250      250


Cleanup: BrokerResponeJSON related cleanup.

This is one (of many coming) PR's to eliminate BrokerResponseJSON.

Replaced DefaultReduceService tests with BrokerReduceServiceTests.
Migrated the following tests to use BrokerResponseNative instead of
BrokerResponseJson:

QueriesSentinelTest
QueryExceptionTest
AggregatinoGroupByForMultiValueTest
PlanMakerTest

Similar PR's expected to migrate all the tests from json version to
native version of broker response.

Current coverage is 54.56% (diff: 100%)

No coverage report found for master at 99de48d.

Powered by Codecov. Last update 99de48d...077df20

LGTM otherwise. For double comparison, please try to use Assert.assertEquals(actual, expected, delta) for readability.


Failing build

@ampolloreno and @vtomole
Please fix the failing build.  Looks like it was induced by your PR

Just reran the build, and everything passed. Closing.


feat: check server capabilities usin the preferred resources endpoint

Description

Using the preferred resources endpoit allows us to test for specific API
groups that contain a specific resource kind, in this case we can test
for which API groups that exist and support Ingress.
ci: remove git from the docker build context
Pass the versio data via build arguments instead of passing the entire
git database. This cuts the build context in half.
ci: update multi-arch build flow
Use the build process from faas-netes to simplify the multi-arch build
process.
ci: allow pushing to fork ghcr repos
ci: push preview images for PRs
ci: build multi-arch in paraller for speed
Motivation and Context

[ ] I have raised an issue to propose this change (required)

Closes #51
Closes #44
Improve #50
How Has This Been Tested?

Create the files cluster.yaml and test.yaml
# cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
  - containerPort: 31112 # this is the NodePort created by the helm chart
    hostPort: 8080 # this is your port on localhost
    protocol: TCP

# test.yaml
apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: nodeinfo
  namespace: openfaas
spec:
  domain: "nodeinfo.myfaas.club"
  function: "nodeinfo"
  ingressType: "nginx"
  path: "/v1/profiles/(.*)"

Then, deploy a 1.19 cluster using
kind create cluster --image kindest/node:v1.19.11 --config=cluster.yaml

and install the test operator
$ kind create cluster --config=cluster.yaml
$ arkade install openfaas \
	--basic-auth=false \
	--clusterrole \
	--ingress-operator \
	--set ingressOperator.image=ghcr.io/lucasroesler/ingress-operator:pr-1

$ faas-cli store deploy nodeinfo
$ kubectl apply -f test.yaml
$ kubectl get ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME       CLASS    HOSTS                  ADDRESS   PORTS   AGE
nodeinfo   <none>   nodeinfo.myfaas.club             80      5s

$ kubectl -n openfaas logs deploy/ingress-operator
I0829 09:49:55.758358       1 main.go:54] Starting FunctionIngress controller version: latest-dev commit: 56f035ce56c466f8b3b73cafd55dd735af933e58
W0829 09:49:55.758563       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0829 09:49:55.770796       1 main.go:94] cluster supports ingress in: extensions/v1beta1, networking.k8s.io/v1
I0829 09:49:55.771156       1 controller.go:70] Setting up event handlers
I0829 09:49:55.771239       1 core.go:72] Waiting for informer caches to sync
I0829 09:49:55.872037       1 core.go:77] Starting workers
I0829 09:49:55.872053       1 core.go:83] Started workers
I0829 09:49:55.872087       1 controller.go:103] FunctionIngress name: nodeinfo
I0829 09:49:55.872098       1 controller.go:112] fni.Spec.UseTLS() false
I0829 09:49:55.872102       1 controller.go:113] createIngress false

Specifically take note of this log line:
I0829 09:49:55.770796       1 main.go:94] cluster supports ingress in: extensions/v1beta1, networking.k8s.io/v1

Deploy a 1.22 cluster using
kind create cluster --image kindest/node:v1.22.1 --config=cluster.yaml

Re-run the tests
$ kind create cluster --config=cluster.yaml
$ arkade install openfaas \
	--basic-auth=false \
	--clusterrole \
	--ingress-operator \
	--set ingressOperator.image=ghcr.io/lucasroesler/ingress-operator:pr-1

$ faas-cli store deploy nodeinfo
$ kubectl apply -f test.yaml
$ kubectl -n openfaas get ing
NAME       CLASS    HOSTS                  ADDRESS   PORTS   AGE
nodeinfo   <none>   nodeinfo.myfaas.club             80      5s

The logs now show only one supported API
$ kubectl -n openfaas logs deploy/ingress-operator
I0829 10:07:16.357689       1 main.go:54] Starting FunctionIngress controller version: latest-dev commit: 56f035ce56c466f8b3b73cafd55dd735af933e58
W0829 10:07:16.357836       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0829 10:07:16.366955       1 main.go:94] cluster supports ingress in: networking.k8s.io/v1
I0829 10:07:16.367052       1 controller.go:70] Setting up event handlers
I0829 10:07:16.367089       1 core.go:72] Waiting for informer caches to sync
I0829 10:07:16.467864       1 core.go:77] Starting workers
I0829 10:07:16.467917       1 core.go:83] Started workers
I0829 10:07:43.764285       1 controller.go:103] FunctionIngress name: nodeinfo
I0829 10:07:43.764332       1 controller.go:112] fni.Spec.UseTLS() false
I0829 10:07:43.764338       1 controller.go:113] createIngress true

Note: I0829 10:07:16.366955       1 main.go:94] cluster supports ingress in: networking.k8s.io/v1
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Impact to existing users

None
Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I've read the CONTRIBUTION guide
[x] I have signed-off my commits with git commit -s
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

I loved the testing instructions on this PR. I think you win the award for the best submitter of PRs in the openfaas org.


Roll Packages from b69f54ee0358 to a6821898bd5a (3 revisions)

https://github.com/flutter/packages/compare/b69f54ee0358...a6821898bd5a
2023-11-09<EMAIL_ADDRESS>Roll Flutter from 4b4a1feb46d1 to f66215027e14 (17 revisions) (flutter/packages#5361)
2023-11-09<EMAIL_ADDRESS>[go_router] Retain query parameter during refresh and first redirect (flutter/packages#5276)
2023-11-09<EMAIL_ADDRESS>Use specific version of mac_toolchain (flutter/packages#5356)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages-flutter-autoroll
Please CC flutter-ecosystem<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Health check logging levels based on error

As a result of comment from @aozarov from Issue #231, we should alter the logging levels based on If the Health Check results in 200, it should be a DEBUG/FINE level event, and all other results should be a higher level.
I favor WARN for non-200 results.

obsoleted by https://github.com/GoogleCloudPlatform/jetty-runtime/issues/4


Fix issue of not showing errors of Settings.toml

Purpose

When there is a syntax error in Settings.toml file causes bad sad error without an indication of what caused it.

Fixes https://github.com/ballerina-platform/ballerina-lang/issues/20579
Approach

Throwing a BLangCompilerException with the message invalid Settings.toml due to <REASON> if Settings.toml file has any syntax error.

Samples
Settings.toml
[central]
acesstoken="2833b8f6-c230-36db-a21e-f0"
2223,,,,

ballerina build -a
error: invalid Settings.toml due to Key is not followed by an equals sign on line 3: 2223,,,,

Remarks

List any other known issues, related PRs, TODO items, or any other notes related to the PR.

Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[x] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #20750 into master will decrease coverage by 0.62%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #20750      +/-   ##
==========================================
- Coverage   15.21%   14.59%   -0.63%
==========================================
  Files          50       51       +1
  Lines        1341     1398      +57
  Branches      208      214       +6
==========================================
  Hits          204      204
- Misses       1121     1178      +57
  Partials       16       16

Impacted Files
Coverage Δ

tool-plugins/vscode/src/extension.ts
0% <0%> (ø)
:arrow_up:

...ugins/vscode/src/syntax-highlighter/highlighter.ts
0% <0%> (ø)
:arrow_up:

...plugins/vscode/src/syntax-highlighter/activator.ts
0% <0%> (ø)
:arrow_up:

...plugins/vscode/src/syntax-highlighter/scopeTree.ts
0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b577044...dd58501. Read the comment docs.


Write test cases for topProjects() in projectResolver.test.ts

@divine-comedian deprecated, closing.


Gracefully handle unknown yaml keys

Low priority but would be nice to get better error messages/validation when a key is mistyped.
Example config
lables:
  - name: bug
    color: CC0000

Accidentally mis-spelling labels as lables will cause a Plugin is not a contructor error.
2019-02-14T18:29:24.357037+00:00 app[web.1]: 18:29:24.356Z ERROR event: Plugin is not a constructor (id=734cd71c-3086-11e9-8c64-6facabd86ce3)
2019-02-14T18:29:24.357071+00:00 app[web.1]:   TypeError: Plugin is not a constructor
2019-02-14T18:29:24.357079+00:00 app[web.1]: at Promise.all.Object.entries.map (/app/node_modules/probot-settings/lib/settings.js:24:14)
2019-02-14T18:29:24.357082+00:00 app[web.1]: at Array.map (<anonymous>)
2019-02-14T18:29:24.357084+00:00 app[web.1]: at Settings.update (/app/node_modules/probot-settings/lib/settings.js:19:52)
2019-02-14T18:29:24.357086+00:00 app[web.1]: at github.repos.getContent.then.res (/app/node_modules/probot-settings/lib/settings.js:8:49)
2019-02-14T18:29:24.357089+00:00 app[web.1]: at process._tickCallback (internal/process/next_tick.js:68:7)

It took me digging into the code to see what the issue was. Definitely user error but it would be helpful to handle this specific case with a better log message, e.g. key 'foo' is not supported or whatnot.

I just hit this issue also, can confirm a decent error message would be nice. Perhaps a validation step for the entire file schema before attempting to deserialize it?


Remove getmesh update from readme

The update command doesn't exist anymore (was it removed by accident or was that planned?)

removed as planned (when renaming thingy happened), considering the maintenance cost!


Add final to methods to prepare for future inline

Since https://github.com/apache/incubator-pekko/pull/305 is basically ready and just waiting for an upstream fix to Scala 2.13, this PR brings in some changes from that PR in preparation for the RC so that we don't potentially break users when we do add the inline.
The changes are quite trivial, i.e. it involves add final to some methods inside of ByteIterator. If you don't do this and enable the inliner then the inliner complains with the following
[error] /Users/mdedetrich/github/incubator-pekko/actor/src/main/scala-2.12/org/apache/pekko/util/ByteIterator.scala:227:12: org/apache/pekko/util/ByteIterator$MultiByteArrayIterator::clear()V is annotated @inline but could not be inlined:
[error] The method is not final and may be overridden.
[error]       clear()

While this will prevent users from overriding the methods, it doesn't make sense to override this method because the implementation relies on this implementation to work. Furthermore other similar methods are marked as final so I believe missing this was an oversight like due to @inline currently not doing anything.

Let me merge this PR, there are a few more methods that I need to verify/add

Yes but I am quite sure this is final, I am going through all Pekko modules just to see that these methods are not manually overriden

So I just updated the PR to remove the final from remote/src/main/scala/org/apache/pekko/remote/artery/Association.scala. Note that Association is marked as private and internal ABI so we can also add it later but the @inline that is already there is like a mistake (its not possible to inline because compareAndSwapObject is from java SDK). Its like that methods calling swapState need to be inlined but again that is something we can do later since this is all private internal pekko stuff/
The ByteString methods here are more critical
Will merge PR once tests pass.

Merging, test failure is from link validator which is unrelated.