2357 add junctions to suburban widgets

The previous PR was closed by mistake. Opening a new one.

@ziv17 Following the correspondence here you can find a road segment table (As to 03-2022) here.
I waiting for a new updated of that table, hope to get it soon. For now you can make sure to create a csv and adjust our parser code AND make sure loading works as expected. Let me know if you find additional errors as the one you found and I'll send the new segments table ASAP

Also @ziv17 can we create tests that use GPS data of junctions (we have this data from the CBS) and use segments GPS and maybe even data from google maps Geocoding to find errors?

Codecov Report
Attention: 14 lines in your changes are missing coverage. Please review.

Comparison is base (76abf30) 52.96% compared to head (f422db9) 53.35%.

Files
Patch %
Lines

anyway/parsers/cbs/executor.py
57.14%
12 Missing :warning:

anyway/models.py
87.50%
1 Missing :warning:

anyway/widgets/widget_utils.py
97.77%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #2476      +/-   ##
==========================================
+ Coverage   52.96%   53.35%   +0.39%
==========================================
  Files         117      118       +1
  Lines        9635     9744     +109
==========================================
+ Hits         5103     5199      +96
- Misses       4532     4545      +13

Flag
Coverage Δ

unittests
53.35% <88.23%> (+0.39%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hi @atalyaalon ,
I am not sure how to handle discrepancies like the following:
Accidents in suburban junction 4193 (road=1, מחלף דניאל) report it is on km 22 (see query below)
Road segment 10054 starts at km: 22.9
I did not find the road segment that ends there.
query:
select road1, non_urban_intersection , km, km_accurate, accident_timestamp from markers_hebrew where non_urban_intersection=4193 order by accident_timestamp

Hi @atalyaalon ,
I am not sure how to handle discrepancies like the following:
Accidents in suburban junction 4193 (road=1, מחלף דניאל) report it is on km 22 (see query below)
Road segment 10054 starts at km: 22.9
I did not find the road segment that ends there.
query:
select road1, non_urban_intersection , km, km_accurate, accident_timestamp from markers_hebrew where non_urban_intersection=4193 order by accident_timestamp

Hi @atalyaalon
In essence, this PR adds OR suburban_junction in ( <juncltion list> ) to widget queries that has the segment_id field in the filter. It is finally ready.

@ziv17 , this pr looks great! Awesome work!
I want to merge ASAP.
A few questions right before merging:

I assume once merged we should load all of the CBS Data from past years to make sure the RoadJunctionKM tables are filled?
Should we add non_urban_intersection index to relevant tables that use get_query (say markers_hebrew) to make sure cache fill runs in a reasonable time?
Besides get_query, do we have other calls in our code of infographics widgets we need to adjust? (We can open separate issues for them and merge this one)
TODOs in code:

TODO of implementation of top_road_segments_accidents_per_km_widget - perhaps you can open a separate issue?
TODO of remove road_segment_name if road_segment_id exists: I saw you have a draft pr. I assume it would be the next pr we'll merge?

1. I assume once merged we should load all of the CBS Data from past years to make sure the RoadJunctionKM tables are filled?

Yes, correct

2. Should we add non_urban_intersection index to relevant tables that use get_query (say markers_hebrew) to make sure cache fill runs in a reasonable time?

Yes

3. Besides get_query, do we have other calls in our code of infographics widgets we need to adjust? (We can open separate issues for them and merge this one)

If there are widgets that do not use get_query(), then yes. I can take it as a task if you want.

4. TODOs in code:

* TODO of implementation of top_road_segments_accidents_per_km_widget - perhaps you can open a separate issue?

Improve implementation: db filter of string not equal and included in list #2507

* TODO of remove road_segment_name if road_segment_id exists: I saw you have a draft pr. I assume it would be the next pr we'll merge?

Remove road_segment_name from query if road_segment_id is present. [#2503]

Hi @atalyaalon ,
I incorporated the latest commit in dev, and the tests fail on
def parse_html_walla(item_rss, html_soup):
    # For some reason there's html here
    description = BeautifulSoup(item_rss["summary"], features="lxml").text

    author = html_soup.find("div", class_="author").get_text().strip()
    return author, description

in line     author = html_soup.find("div", class_="author").get_text().strip()
The find returns None, and the get_text() fails.
(This code is 2.5 years old). This test does not provide much value, as it only tests that the real data that returns in not empty.
What do you think?

Hi @atalyaalon , I incorporated the latest commit in dev, and the tests fail on
def parse_html_walla(item_rss, html_soup):
    # For some reason there's html here
    description = BeautifulSoup(item_rss["summary"], features="lxml").text

    author = html_soup.find("div", class_="author").get_text().strip()
    return author, description

in line  author = html_soup.find("div", class_="author").get_text().strip() The find returns None, and the get_text() fails. (This code is 2.5 years old). This test does not provide much value, as it only tests that the real data that returns in not empty. What do you think?

@ziv17 In that case you can remove the test

Hi @atalyaalon ,

1. I assume once merged we should load all of the CBS Data from past years to make sure the RoadJunctionKM tables are filled?

Yes, correct

Great

Should we add non_urban_intersection index to relevant tables that use get_query (say markers_hebrew) to make sure cache fill runs in a reasonable time?

Yes

Great, can you create a list of the tables and create new migration / add to current DB migration in this pr?
I want to make sure it's added in this PR

Besides get_query, do we have other calls in our code of infographics widgets we need to adjust? (We can open separate issues for them and merge this one)

If there are widgets that do not use get_query(), then yes. I can take it as a task if you want.

Yes, can you create a list of these widgets in an issue and we'll priorities? (This won't impact current PR, we won't wait)

TODOs in code:

TODO of implementation of top_road_segments_accidents_per_km_widget - perhaps you can open a separate issue?

Improve implementation: db filter of string not equal and included in list #2507

* TODO of remove road_segment_name if road_segment_id exists: I saw you have a draft pr. I assume it would be the next pr we'll merge?

Remove road_segment_name from query if road_segment_id is present. [#2503]

Yes, can you create a list of these widgets in an issue and we'll priorities? (This won't impact current PR, we won't wait)

Opened 2508

can you create a list of the tables and create new migration / add to current DB migration in this pr?
I want to make sure it's added in this PR

Will do

Hi @atalyaalon ,
in anyway/widgets/road_segment_widgets/accident_type_vehicle_type_road_comparison_widget.py there is a query with road (1 or 2), but no road_segment. Should all junctions on these roads be added too?

Below widgets do not use get_query(), and need to be handled specifically.
anyway/widgets/widget_utils.py, get_involved_counts)
anyway/widgets/road_segment_widgets/killed_and_injured_count_per_age_group_widget_utils.py
anyway/widgets/all_locations_widgets/seriously_injured_killed_in_bicycles_scooter_widget.py
anyway/widgets/road_segment_widgets/front_to_side_accidents_by_severity.py
anyway/widgets/road_segment_widgets/killed_and_injured_count_per_age_group_stacked_widget.py
anyway/widgets/road_segment_widgets/killed_and_injured_count_per_age_group_widget.py
anyway/widgets/road_segment_widgets/road2_plus1_widget.py: use road_segment_id instead of _name.

Below widgets do not use get_query(), and need to be handled specifically.
anyway/widgets/widget_utils.py, get_involved_counts)
anyway/widgets/road_segment_widgets/killed_and_injured_count_per_age_group_widget_utils.py
anyway/widgets/all_locations_widgets/seriously_injured_killed_in_bicycles_scooter_widget.py
anyway/widgets/road_segment_widgets/front_to_side_accidents_by_severity.py
anyway/widgets/road_segment_widgets/killed_and_injured_count_per_age_group_stacked_widget.py
anyway/widgets/road_segment_widgets/killed_and_injured_count_per_age_group_widget.py
anyway/widgets/road_segment_widgets/road2_plus1_widget.py: use road_segment_id instead of _name.

Tables with road_segment_id:
AccidentMarkerView - markers_hebrew - Index exists
InvolvedMarkerView - involved_markers_hebrew - Index exists
VehicleMarkerView - vehicles_markers_hebrew - Need to add

Hi @atalyaalon
I added the junctions to widgets that I missed, and added non_urban_intersection index to markers_hebrew, involved_markers_hebrew, and vehicles_markers_hebrew

Hi @atalyaalon This PR is ready for your review

Note: No need to resolve conflicts, I resolved them here: https://github.com/data-for-change/anyway/pull/2527
I'll go over this pr during this week, perform some tests and hopefully we'll merge it in the coming days, well done Ziv.

Closes #2546

Closes #2546


Preserve routing metadata

Per a discussion on the mailing list, it can be useful to introspect the URI template that is associated with a given resource. One way to implement this would be to check for an attribute and fill it in as part of the add_route logic. For example:
class Thing(object):
    def __init__(self):
        self.uri_templates = []

    def on_get(self, req, resp):
        pass

# ...

def add_route(self, uri_template, resource):

    # ...

    if hasattr(resource, 'uri_templates'):
        resource.uri_templates.append(uri_template)

    # ...

We could also do this:
    if hasattr(resource, 'uri_templates'):
        resource.uri_templates.append(uri_template)
    else:
        resource.uri_templates = [uri_template]

But that would break in the case that Thing defined __slots__.
Another option would be to store the URI template in the routing tree and then set an attribute on each Request instance:
def __call__(self, env, start_response):

    # ...

    responder, params, resource, uri_template = self._get_responder(req)

    req = self._request_type(env, options=self.req_options,
                             uri_template=uri_template)

    # ...

Either way, this would provide a way for middleware and hooks to key off the URI template to perform additional actions.

Currently the line:
responder, params, resource, uri_template = self._get_responder(req)

has been moved to inside the try-catch. what about something like this:
responder, params, resource, uri_template = self._get_responder(req)
setattr(req, 'uri_template', uri_template)

Thinking about this, it may be helpful to do both. Tooling may wish to enumerate the resources, examining each for the routes associated with it. And people have asked for a way to know what route was matched for a particular request (see also #581).
We should try to do this in a non-breaking way first, if it isn't too ugly.


I'd like to (try to) extend this beyond YouTube, Twitch and others

Hi Gabriel. I've been absolutely loving this. Keeping a Twitch stream in the corner while doing homework is so great. Above anything: thank you.
However, it doesn't work quite the same for Hulu; the initial display doesn't include the entire picture and needs to be rescaled/enlarged to show the whole frame. I'd like to change that because I love Hulu shows like "It's Always Sunny in Philadelphia" etc.
If possible, could you give me a quick overview of how it all works? (I know how to code, just a simple architecture explanation would be nice) And where to start when trying to add built-in functionality for other streaming services? Any insight would be appreciated, but if it's not worth your time I understand. Again, thank you for this.

Above anything: thank you.

Cool. You're welcome.

However, it doesn't work quite the same for Hulu

Do you have a screenshot? I tried to sign up for Hulu, but it's not available in my country and VPNs/proxies are also blocked.

If possible, could you give me a quick overview of how it all works?

Are you using Chrome 70 with the native player? If so, it's just:
document.getElementsByTagName('video')[0].requestPictureInPicture(); // and forget everything else...

[...] add built-in functionality for other streaming services

We have no control over the native player. We can't change width, height, position, video quality, controls, etc. But if you're talking about the popup player, then I can explain it better.


Upgrade gwt-module.dtd from 2.4.0 to 2.7.0 (and use public XSD instea…

…d of SVN tag xsd)

master: 55df36f0ebdc4a7fde8a1c7e9c0e17d190eadb6a


convert engine build to kotlin

migrate engine build to kotlin.

@soloturn are you planning to take this up again?

am currently out of ideas how to properly include jmh without dragging dependencies somewhere else.

running out of ideas how to fix this i asked in the gradle forums:
https://discuss.gradle.org/t/exclude-jmh-files-in-resulting-jar-jmh-plugin-bug-or-our-bug/48450


How do i use dynamic path created by applyParams in sitemap

@manishachaudhary1234 can you describe more?
The example is an example of how to use the dynamic parameters in the routes

Solved. Thankyou


added Unlift derivation for reader-like effects with subcontexts

@Odomontois you said "Что-то боюсь его имплиситом делать"

Yeah. This function looks really good, but the amount of type parameters, that need to be inferred through the implicit search makes me think it would be rarely found.
Moreover, it could create conflicts if some ZIO implicits are reexported through context companions. That claim should be investigated though.
So while I've found this very helpful, I'd like to make this def subContextUnlif non-implicit at least for a while.
@klimovsv what do you think?

I am okay with that. I think we should compare this derived instance with same instances for Env and ZIO. May be we could remove  particular instances in favor of this derivation.

@klimovsv could you please make this def non-implicit, so I could merge now


Feature/13, 14 - Add theme to meta-diagram, fix the example to use the same.

Closes #13 #14

I like @ddelpiano 's suggestion to merge the theme props. Besides that I'm good

@afonsobspinto yea i added the change for that

@zsinnema could you please merge this one.


doc: update dockerTools.exportImage content and use doc conventions

Description of changes
This PR updates the structure of dockerTools.exportImage to follow doc conventions and adds a lot more content and examples to make it easier to understand what exportImage does.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2024-02-01-documentation-team-meeting-notes-107/39146/1


I can't understand the fix for the build failing because of resource authorization

The instructions dictate:

Navigate to the pipeline in Azure Pipelines.
Switch the pipeline's default branch to the branch that includes the new service connection reference. For example, if you've made a change in a branch called features/add-resource, then switch the pipeline's default branch to features/add-resource.
Save the pipeline.
Revert back to the prior default branch and save the pipeline again.

I am trying to fix a build in my PR that is refusing to build configured with yaml because of resource authorization.
I am struggling to work out what you mean by "Navigate to the pipeline in Azure Pipelines".  I can't see anything on the website that states that it is a pipeline.
I'm not in release management, this is for a PR build.  Please can you help instruct me on how I can achieve this work around?

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ee4ec9d0-e0d5-4fb4-7c3e-b84abfa290c2
Version Independent ID: 3e2b80d9-30e5-0c48-49f0-4fcdfedf5eee
Content: Resources - Azure Pipelines
Content Source: docs/pipelines/process/resources.md
Product: devops
GitHub Login: @vtbassmatt
Microsoft Alias: alewis

https://dev.azure.com/<your_organization>/<your_project>/_build will give you the list of build pipelines. From there you should be able to find the pipeline that's giving you trouble and follow the instructions.
Thanks for the doc feedback, too. For a GitHub user, the instructions aren't clear enough since you may never have logged into your Azure DevOps account.

@vtbassmatt Thanks for replying so fast.
When I follow your instructions I "edit" the build that is giving me problems, but I can't work out what step 2 means.  Does this mean, change the branch that triggers the build?
I've found out how to complete the workaround.
In the documentation, I propose the following change:

Navigate to the Build definition of the build that is failing.
Click Edit.  You should see the YAML script that defines the build.
Click the button with the three vertical dots, and select Triggers.
Navigate to the YAML tab (far left)
Highlight the Get Sources within the Pipeline
Switch the pipeline's default branch to the branch that includes the new service connection reference. For example, if you've made a change in a branch called features/add-resource, then switch the pipeline's default branch to features/add-resource.
Save the pipeline.
Revert back to the prior default branch and save the pipeline again.

I've expanded out the first point into 5 points to make it clearer

Thanks. The user interface is undergoing some flux right now, and not all customers see the exact same steps. That's why the message is a little more generic than what you wrote.
Soon, this won't be a problem at all, as we'll address the underlying authorization differently.


Not an Issue

Hey every one, just wanted to ask.
Is there some reason to name a function as variable in JavaScript?
(I mean using Dep two times. Just was crawling in Vue.js and found this)
var Dep = function Dep () {
    this.id = uid++;
    this.subs = [];
  };

Just say yes/no, don't bother to explain, I'll search it by myself.
Thanks!

Anonymous vs non Anonymous functions, stack traces and variable hoisting. Looking these up should help you

yes


Don't format string already formatted.

If args is null or empty, the message should not be formatted.

Hi @magol,
Thank you for the contribution, very appreciative!


Remove usage of NAD ready condition

NetworkAttachmentsReady condition explains whether the requested NAD resource exists, but is also used to explain whether the attachments are present in pod deployed. Because of this the messages we have for NAD conditions contains "creating" which does not match fully the implementation in Dataplne Node (it just checks existence).
This drops NAD condition used in Dataplane node, and squash the check into input peady condition check. This also resolves the problems with NAD conditions (It is not initialized. Also it is not set to True when but only set to False when something goes wrong).

recheck

/test dataplane-operator-build-deploy-kuttl

recheck


why set uv = f0 = 0 in norm_interp_f0?

https://github.com/MoonInTheRiver/DiffSinger/blob/3d050f76aefb766d004fddcf52e9307affefd3c4/utils/pitch_utils.py#L45-L51
Hey guys I found you set  uv = f0 = 0   at line 50, what's the intension behind this? Is that a typo or I miss something really important?

sorry I've missed another "="


Remove gtfs_Static dependency, unless it is ready to use

#458
#457
#456  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


clasp create --rootDir ./dist

Expected Behavior
You should be able to create specify rootDir. rootDir is specified the local directory in which clasp will store your project files. Also rootDir is store in .clasp.json.
clasp create --rootDir ./dist

.clasp.json
{
    "scriptId": "<SCRIPT_ID>",
    "rootDir": "./dist"
}

Actual Behavior
After clasp create, You must move pulled files and edit .clasp.json.
clasp create --rootDir ./dist
mkdir ./dist
mv appsscritp.json ./dist/

Valid issue, wouldn't be hard to add.


Add reflection to nodes

[This is a continuation of this discussion of describe]
In addition to the regular handler, each node implementation should export a reflection function that describes its inputs and outputs.
This is useful in e.g.

a visual editor, which can understand nodes in this way, especially if the inputs or outputs depend on the configuration of the node (or even incoming data!)
self describing APIs, e.g. when exposed as cloud function
as a way to express errors in a graph and iterate towards resolving those (here, missing inputs should trigger an $error output (see google/labs-prototypes#129) that is the same as calling reflection with the current inputs and wired schemas, see below)

It takes three parameters, all optional:

inputs, which are inputs as if they were passed to the node
input schemas, the currently expected schema of these and other inputs
output schemas, the currently expected output schemas

The output of the reflection function contains:

input schemas, including indicating which are required – note that this can depend on the provided inputs! (see below)
output schemas
how complete the response is, which is one of

calling with the inputs is valid
supplying all inputs marked as required will be sufficient, assuming they match the schema
the inputs are not complete yet and will depend on supplying at least some of [list of missing inputs]

For nodes that have fixed inputs and outputs this boils down to schema validation and we should supply conveniences to generate reflection automatically, ideally from Typescript schemas that are used to implement the node itself.
When input or output schemas are supplied to the call, besides those actually selecting different behaviors in some cases (see below), reflection should verify that inputs conforming to those schemas are valid and/or that outputs will conform to the supplied schema. The return value should only cover the part of the schema that is actually used. This allows the input  data to contain more data / only a subset of the data from the output to be used. But it also allows a higher level validator to make sure that it's fine to ignore the other parts, which is not always the case (e.g. ignoring the second part of Restaurant { name: string, closedPermanently: boolean } might be bad). LLMs might be reasonably good at detecting such cases, FWIW.
Examples of more complex scenarios:

promptTemplate: The actual inputs depend on the {{ variables }} defined in the template. So you can imagine this sequence of calls:

empty call --> input schema describing template as required (and the output schema for prompt, which will be the same for all subsequent calls)
providing template that contains {{ foo }} --> input schema describing template and foo and indicating that supplying a string for foo will be sufficient
(optional:) calling with template and foo --> same input and output schema, but indicating that this would be a valid call

jsonata: The actual outputs depend on the query:

empty call --> input schema describing expression, json and raw + union of possible outputs (TODO: since raw has a default value, technically we should assume result unless raw = true will be provided)
calling with inputs for expression and setting raw to true --> ideally replying with both the input and output schema defined by the expression.
...

runJavascript: Function implies expected parameters
passthrough: Supplying either input or output schema implies the other
etc.

To be discussed: Clearly some of these require advanced parsing of the workload, which might be highly non-trivial to implement. Should we instead call an LLM to provide a reasonable guess? They are pretty good at understanding e.g. code or jsonata queries, but will not always be correct. And it would seem odd to hardcode a specific LLM into nodes. So maybe instead of saying "I don't know", a node can reply with a prompt to an LLM that would answer the question. And maybe we add a way to call reflection again with the answer from the LLM, so that it can turn that into a proper response for us (or even ask another question).
This can also be called in other orders, especially if $role is provided (see google/breadboard-ai#56):

promptTemplate: This time the template is the missing piece:

Call with foo, bar and other given inputs --> input schema describing template with a description that says that the template can contain {{foo}} and {{bar}} (and same output as above)
Given $role and the new description ask an LLM to write a template, call again with that template --> if valid, respond as such, otherwise provide feedback such as missing inputs, e.g. if the LLM hallucinated other template variables

runJavascript, jsonata, etc. could all work the same way
We could even start with just $role and go with hallucinated inputs and then suggest nodes that provide those inputs. This would be a way to bootstrap graph generation from a few key nodes, such as a template.

Victory 🚢  Let's track any remaining bits separately.


Feature | Use OSX password prompt

Instead of asking for password with the custom "Sudo password" prompt
use the osx password prompt by attempting to run a "sudo-required"
command.
This will prompt as follows on OSX, even is there's a typo in the password, we
have retry capabilities by using the native password prompt feature.

We are already using the system prompt, simply with a custom --prompt to make it clearer what password is being requested, since the user did not type sudo themselves. In fact commandWithSudoNoPrompt does the exact same thing as commandWithSudo except the latter avoids sudo if root, and prints a warning if sudo is not found.


Set internalApiPath from header if running in seamless mode

Following from this discussion on the community forum this PR sets the internal api path if running in seamless mode.
Left to do:

[x] Decide on appropriate header to set (currently x-internal-host)
[x] Add documentation

@instification It's awesome that you're tackling this issue.
There was a problem, at some point with cross-request pollution. Imagine that Volto's nodejs server is handling all requests, and because of nodejs's async nature (as far as I understand it, and I'm not sure of this) you may get execution of code "out of sync" between two different separate requests. If the config object is a global, and is mutated but shared between the requests, you'll get issues.
https://stackoverflow.com/questions/35649998/is-it-possible-to-have-thread-local-variables-in-node

@instification @tiberiuichim today I can use the variables RAZZLE_INTERNAL_API_PATH and RAZZLE_API_PATH, so that Plone requests made by the Volto server use RAZZLE_INTERNAL_API_PATH and requests made by the browser use RAZZLE_API_PATH?

@instification @tiberiuichim today I can use the variables RAZZLE_INTERNAL_API_PATH and RAZZLE_API_PATH, so that Plone requests made by the Volto server use RAZZLE_INTERNAL_API_PATH and requests made by the browser use RAZZLE_API_PATH?

@wesleybl yes that's correct as I understand it :)

@wesleybl RAZZLE_INTERNAL_API_PATH is intended to be used in containerised environments. Specially the ones that the network topology prevents an easy communication between containers private/public interfaces (k8s, etc...) For the rest, it is fine that your Volto SSR server access to the public backend API URL.

@instification It does look good, but I think we should be a bit cautious about this one.
Specially I'm concerned about Express and make sure that it does work under heavy load conditions. We had problems in the past with the fact that NodeJS does not have the "threadlocals" concept, thus it's difficult to have proper requests isolation.
It would be great if you lead this and test it in your servers, having good first hand feedback will help, for sure. If that adds some load tests it would be amazing.
Let's talk during the conference about the details.
Thanks for looking into this!

@sneridagh I have already started to look at isolating it following @tiberiuichim's advice to move away from using the global config object. I made some progress, certainly for api requests it is possible to look up per request whether there is a header  set (for SSR requests) or whether the window.env.apiPath is set. I will try to get a proof of concept pushed here before the conference.
I don't think there are too many other places where apiPath is being read, but I am not 100% sure how simple it will be to access the req or window objects in those scenarios.
Presumably if we get the thread isolation issue resolved via not relying on the config object, then the load equation would be similar to if we were running with fixed api paths via env vars. But I think having some benchmarks and comparisons will be important.
Look forward to discussing more at the conference!

@tiberiuichim I had a go at implementing the hook/hoc helpers. It seems to be working well for me locally. Just want to check this is the right way of doing it before proceeding. I will probably need to look at the test failures next to see what the changes are breaking.
I think we discussed keeping the change to the settings singleton for backwards compatibility?

actually there's still something wrong in development/seamless mode. It should work via the devProxyToApiPath but it is not correctly detecting it in the calculateApiPath function. I need to debug it a bit more.
But I would be grateful for your opinion on how I have implemented the hooks/hocs.


Navigation bar elements flickering

Describe the bug
Some navigation bar elements(UIBarButtonItems and UINavigationitems with custom views) flicker when navigation bar gets hidden interactively.

To Reproduce
The issues manifests quite easily, for example when setting up an item with custom image view:
    let imageView = UIImageView(image: UIImage(named: "AppIcon"))
    navigationItem.rightBarButtonItem = UIBarButtonItem(customView: imageView)

or event just an empty view
    let titleView = UIView(frame: CGRect(x: 0, y: 0, width: 200, height: 30))
    titleView.backgroundColor = .white
    navigationItem.titleView = titleView

Here's a sample project that demonstrates this:
Sample.zip
Expected behavior
Would be nice if it wouldn't flicker.

Related issues - #325, #257.


Does it support vertical scrolling

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

我没看到


feat(lifecycle-operator): support linked spans in KeptnAppVersion

Fixes: #2577

awesome work, just had some minor comments on naming in the code, but other than that it seems to work as expected:


Not working on Scaleway servers

Installation is successful but the traffic is not routed trough the server.
More info here : https://community.scaleway.com/t/i-dont-have-a-working-vpn/1790

The thread doesn't actually have any information at all, more like disinformation.
You can also see that it's working for others, so you could probably consider that this issue could be in your end and not a problem with the script or Scaleway.
Still, I have just created one Scaleway instance to check for you, and it works. I wanted to check on the ARM servers too, but they are likely out of capacity and wouldn't let me get one so I just tested with the Avotons.

It could be that you just didn't set up the NAT correctly.
You need to specify the internal IP address on the first installation step (which is there by default) and then specify the public one when asked later.

You need to specify the internal IP address on the first installation step (which is there by default) and then specify the public one when asked later.

Later ? It only asks one time for the IP adress.
So I have to keep it this way ?
First I need to know the IPv4 address of the network interface you want OpenVPN
listening to.
IP address: <IP_ADDRESS>

Try to actually READ what the script is asking you for:

First I need to know the IPv4 address of the network interface you want OpenVPN
listening to.

Do a ifconfig and check what's the freaking IP address assigned to the network interface (eth0).
If you specify the right IP you are asked for, the script will later detect the NAT and ask you for the public IP and will work perfectly.
I now see you haven't even bothered to read the script at all and look at how it works. I would consider this reasonable except for the fact you are actually maintaining a fork of this script which you apparently don't even understand.
So please, refrain from opening any issue in the future.

I am not familiar with NAT, sorry.
So when I see :

First I need to know the IPv4 address of the network interface you want OpenVPN
listening to.

Why would I put a private IP ??
Now I understand, it explains a lot ^^
The noob apologizes

Why would I put a private IP ??

Because it is the actual IP address assigned to the interface, which is what the script is asking about.

Yep that's right, no public IP on ifconfig ^^

(I'd just like to point out that I wasn't the only one confused with Scaleway) https://github.com/Nyr/openvpn-install/issues/112

The right IP address is detected by default and user needs to manually break the configuration for it to fail.
Not my problem if people doesn't read what they are asked about.

I set the both correct IPs but it is still not routing internet traffic. I used a x64 machine with ubuntu 16.04 from scaleway. Any updates about scaleway relative problems ?

The right IP address is detected by default and user needs to manually break the configuration for it to fail.

I have run the script more than once, changing absolutely nothing, using an Ubuntu 17.04 droplet. Now, the IP detected by the script matches the inet section in eth0. Is that the correct configuration?

Running the iptables commands  in https://github.com/Nyr/openvpn-install/issues/92 results in an empty output.

@BrainBlasted you are using an EOL distribution and as such, is not supported. But yes, the instructions on this thread still stand.


Regent: Leaking index and field spaces

Regent tasks are currently leaking index and field spaces when they exit. The compiler should emit code to clean them up.

Is there a work around for this issue in the short term? The runtime warnings are really annoying.

Th runtime warnings should all be off by default now unless you explicitly enable them.

I want to make a note that many of the Regent tests are also leaking layout constraints.

Aren't our layout constraints all static? How exactly are we supposed to clean them up? Do we need a cleanup think to clean up the results of our registration thunk?

It's a consequence of field spaces not being destroyed; field spaces are holding references to layout constraints, so until they are destroyed, those references are still alive.

I think we fixed this a long time ago?

Do you want to try turning on the runtime checks and confirming that? 😉

We've been through several rounds of that and have fixed various issues along the way. I don't think we run legion_gc.py in CI (as far as I know) but I believe we've cleared several large applications of leaks and am not aware of any outstanding ones...

I didn't mean legion_gc.py. I meant running with the -lg:leaks flag which will give you a warning if you leak any resources that you created. If you convert warnings to errors you'll also get hard crashes in the CI if you are leaking anything.

If you convert warnings to errors

How do you do this?

Build with -DLEGION_WARNINGS_FATAL

I can't actually run the Regent test suite in this mode because it causes warnings like the following to turn into errors as well:
$ ./regent.py tests/regent/run_pass/call_task_polymorphic1.rg
[0 - 7ff84f868100]    0.000151 {4}{threads}: reservation ('utility proc 1d00000000000000') cannot be satisfied
[0 - 70000b5c6000]    0.103747 {4}{runtime}: [warning 1071] LEGION WARNING: Region requirement 0 of operation t1 (UID 3, provenance: tests/regent/run_pass/call_task_polymorphic1.rg:91) in parent task main (UID 2) is using uninitialized data for field(s) 102,103 of logical region (1,1,1) (from file /Users/elliott/Programming/Legion/legion/runtime/legion/legion_ops.cc:1838)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1071

What I want is to make only the warnings from -lg:leaks into errors.
Anyway, there is a change in https://gitlab.com/StanfordLegion/legion/-/merge_requests/1257 which cleans up the leaks I am currently aware of (and have had the patience to manually check, since I cannot rely on an automated test suite to confirm).

Actually it's worse than that because I have to build with -DLEGION_WARNINGS_FATAL to get the list of tests that might be hitting leak errors, then recompile without -DLEGION_WARNINGS_FATAL so that I can actually run them to completion to see if they actually leak (because if hit some other warning-turned-into-an-error first, I can't know if the code would have leaked if it actually ran to completion, because execution is terminated prematurely).
So I think I cannot validate the current situation at all except through extensive manual effort, which I am not willing to do right now.

You can manually turn the warnings in this function into errors to get the behavior you want:
https://gitlab.com/StanfordLegion/legion/-/blob/master/runtime/legion/legion_context.cc?ref_type=heads#L6334-6523
I don't think it's a common enough use case to do this generally for everyone.

Just for posterity, the issue I am seeing now (even without leak checks or hard errors) is that index spaces are being deleted too eagerly. A sample reproducer and some explanatory comments are available here:
https://gitlab.com/StanfordLegion/legion/-/merge_requests/1257#note_1901183719

I've removed the index space requirement checks in the inordercommit branch since they don't do anything anymore there. That branch should make it into the master branch later today.

I think that https://gitlab.com/StanfordLegion/legion/-/merge_requests/1257 is ready to go now, just waiting for users to confirm that it doesn't break anything.

Good to go from my side if you are happy with it. The inordercommit branch has merged.

There are two major caveats currently remaining:
Caveat 1: Some Programs Rely on Leaks
For example, S3D has code that looks like:
task make_partition(...)
  var part = ...
  var part_color = c.legion_index_partition_get_color(..., __raw(part))
  return part_color
end

The compiler has a very simple escape analysis that would detect if part were returned directly. But because we're getting the __raw value and passing that into a C API call, and then returning the color, I see basically no reasonable way for the compiler to determine that there's a dataflow here.
I see two possible solutions:

Assume that __raw always escapes. If you ever use __raw for any purpose, we'll leak the object.
Add a separate __leak operator to explicitly leak objects so that we do not collect the partition at the end of the task. This means that some user programs (like S3D) will need to be modified to account for this change.

Caveat 2: Tracing does not Support Deletion
This can get really subtle because Regent has to create implicit index spaces in some cases. Here are some examples of code that creates index spaces implicitly:
for i = 0, N do
  c.legion_begin_trace(...)
  __demand(__index_launch)
  for j = 0, M do
    ...
  end
  c.legion_end_trace(...)
end

Regent has to create an index space [0, M) for the j loop in order to do the index launch. Previously, we simply leaked all such index spaces. Now we collect them once the launch is issued; but this means that we now cannot trace the loop. This has to be fixed by creating an index space explicitly:
var M_space = ispace(int1d, M)
for i = 0, N do
  c.legion_begin_trace(...)
  __demand(__index_launch)
  for j in M_space do
    ...
  end
  c.legion_end_trace(...)
end

This is obnoxious but at least reasonably straightforward. But here's a case where the reason why the index space gets created is pretty non-obvious:
for i = 0, N do
  c.legion_begin_trace(...)
  some_task_launch(...)
  c.legion_end_trace(...)
end

Do you see the index space? No?
It's on the very first line for i = 0, N. The reason is that when Regent launches some_task_launch, it tries to be helpful and assumes that [0, N) is the sharding space for the launch. This is maybe not the best choice in this case, but it makes more sense in cases where we have inner loops that cannot be turned into index spaces: at least Regent can associate a sharding space and point to each individual task and more or less get the right distribution.
I don't see a way to fix this without some heavy-duty LICM in the compiler. And worse, because the user can call C API methods directly for the tracing, we don't even necessarily know that the code is going to be traced.
This could also be fixed by the runtime supporting index space deletion inside traces, but I don't know if that is anywhere on the horizon.

This could also be fixed by the runtime supporting index space deletion inside traces, but I don't know if that is anywhere on the horizon.

I'm not going to support deletion inside traces because those traces are immediately non-replayable forever because you can't delete the same index space (or any resource really) twice.

I don't see a way to fix this without some heavy-duty LICM in the compiler.

Why don't you just move all deletions of implicit index spaces to the very end of a task body? I realize that might create a sub-optimal lifetime for some resources (especially in long-running tasks), but it would be sound.

I guess if your doing it via scopes, then in the case of this code:
for i = 0, N do
  c.legion_begin_trace(...)
  __demand(__index_launch)
  for j = 0, M do
    ...
  end
  c.legion_end_trace(...)
end

why isn't the "end of the scope" after the last statement, e.g. the end trace call? So the deletion is done outside of the trace.

The thing to remember is that these are just demonstrative cases and actual code can get arbitrarily more complicated. For example, here's one variation that I had forgotten about in my original comment:
var M : int
while ... do
  c.legion_begin_trace(...)
  M = 0
  for i = 0, N do
    M += 1 -- or any other arbitrary computation
    __demand(__index_launch)
    for j = 0, M do
      ...
    end
  end
  c.legion_end_trace(...)
end

M is now a loop-carried dependency. (Or we can model this as any other arbitrary computation.) This means there is no opportunity to do LICM; we are simply stuck creating it inline.
Additionally there can be any level of nesting between the traced loop and our index launch. Or even function calls. Not every Regent application has the entirety of its main loop exposed in this way, there can be additional levels of indirection through __demand(__local) task calls.
Why is it not possible to defer the index space deletion to the end of the trace? That also seems like a sound implementation. If you support index space creation it seems like the inverse operation should also be supported.

LICM is probably want we really want here for optimization purposes, but absent that, this feels like the right semantics.
If you're worried about the performance of back-to-back trace replays, we can actually safely add deletion operations to the list of special operations we allow between traces without invalidating preconditions since by definition deletions can't mess with preconditions for a trace since that would be use-after-free. I can put that optimization in so that even if you have those implicit deletions between traces you'll still get the fast back-to-back trace replays.

Why is it not possible to defer the index space deletion to the end of the trace? That also seems like a sound implementation. If you support index space creation it seems like the inverse operation should also be supported.

We don't support index space creation inside of traces either. It happens to work in the case of immediate index space creations because they don't produce operations that are captured by the trace. If you tried to do a deferred index space creation (e.g. from a future) that will result in an error.

So the only way I see to implement this at the moment is to modify the C API to buffer index space deletions and apply them all after the legion_runtime_end_trace call. Everything else is insufficient because of (a) local task launches, (b) across compilation units.
The problem with doing it at the C API layer is that if the user uses a C++ library that does tracing, I can't capture that. But I suppose this is enough of a corner case that maybe it's fine.

So the only way I see to implement this at the moment is to modify the C API to buffer index space deletions and apply them all after the legion_runtime_end_trace call. Everything else is insufficient because of (a) local task launches, (b) across compilation units.

Can you explain why that is? Why can't Regent just look at the syntax of the body of the loop and say I'm going to put the deletion statements after all the statements in the scope? It shouldn't even need to care whether the statements in the scope are Regent statements or some black-box external C statements.

Fundamentally, tracing uses dynamic scope, while loop bodies are lexical scope. That means you can end up in situations like:
__demand(__local)
task rk_stage(N : int)
  __demand(__index_launch)
  for i = 0, N do -- implicit index space created here
    ...
  end
  ...
end

And then use it later like:
for t = 0, T do
  c.legion_runtime_begin_trace()
  for rk = 0, 4 do
    rk_stage(N)
  end
  ... -- other stuff
  c.legion_runtime_end_trace()
end

In theory these tasks could even be in different compilation units and thus not visible to the compiler at the same time. And again, this is just a demonstrative example, and real codes can apply arbitrary additional levels of loops, scopes, task calls (as long as they are local), etc.
There is in general no meaningful way for Regent to lift the implicit index spaces out far enough such that they would be outside of the enclosing trace calls.

Yet another reason for automatic tracing.

I guess the other thing I'm confused about is why you need to make an implicit index space. We still support "immediate" domains for index space launches:
https://gitlab.com/StanfordLegion/legion/-/blob/master/runtime/legion.h?ref_type=heads#L1675
You shouldn't even need to make an implicit index space in these cases.

The only reason to make an implicit index space is if you're doing LICM and you're going to reuse the index space over and over.

The sharding space for single task launches must always be a full index space:
https://gitlab.com/StanfordLegion/legion/-/blob/master/runtime/legion.h?ref_type=heads#L1574
I need to look at the index task launch case further; I do not immediately see where in the compiler we're actually making index spaces. The code that I do see appears to make only domains, as you suggest. (And the sharding space in that case is unset, so it inherits from the launch domain.)

Ok, I see it now. Regent was lifting += into a task and issuing a single task launch for that task, which it then subsequently decided to assign a sharding space. The sharding space goes through the above API which forces a conversion into a full index space.

I implemented 1 level of LICM for the sharding spaces associated with loops. They are now generated before the loop, preserved through the entire loop body, and then cleanup is deferred after the loop finishes. This has been effective at least in the cases I've tested here for getting the subsequent deletions out of traces. In principle, a sufficiently deeply nested loop (especially with functions) would defeat this, but it's been good enough for all our application so far.
Looking at my use case 1 from https://github.com/StanfordLegion/legion/issues/606#issuecomment-2115922544, it appears that this is a non-issue because I simply don't delete index partitions 🤣. So that will need to be dealt with at some point, but I think not yet. If we recursively delete from the root index space then it will not be an issue for leaks at program termination, though it could potentially contribute to memory usage growth during application execution. However, as the title of this issue indicates, this is mainly about index and field spaces, so I think I will defer this.

because I simply don't delete index partitions 🤣. So that will need to be dealt with at some point, but I think not yet. If we recursively delete from the root index space then it will not be an issue for leaks at program termination, though it could potentially contribute to memory usage growth during application execution.

Right, this will only apply to use cases where you are making lots of index partitions over the lifetime of the application. If you're making just a few and reusing them then it won't be an issue, and when you delete the root index space Legion will just clean everything up.

Re: my point about LICM here https://github.com/StanfordLegion/legion/issues/606#issuecomment-2197683702. Unfortunately this is not sufficient. S3D has deeper loop nesting which (unsurprisingly) causes my simple hack to fail. So now we're back to these options:

Disable sharding of single tasks entirely (perhaps via a flag). I'm exploring this with S3D to confirm it works, but it's strictly speaking a breaking change as some applications will need this flag now to function correctly.
Implement real LICM. However there is a danger even this will be defeated by local task launches, if we do not implement an interprocedural analysis. (I don't really like this option in either variant. Even the single-task optimization is more than I really want to bite off right now and it can still fail in obscure ways when it doesn't work.)
Modify Legion to allow either:

Assigning immediate index spaces as the sharding spaces for single task launches.
Permit deleting index spaces during traces.

Try to find a clever way to distinguish between loops where single task launches require sharding and ones where that's not required. For example, a loop that contains both index launches and single task launches is likely to be an outer loop, and not simply a loop where the user has forgotten to annotate __demand(__index_launch) and didn't realize it wasn't being applied. Therefore we could reasonably shut off sharding in that case because it likely wouldn't have helped anyway, and isn't likely to be catastrophic to performance. However, we are still making assumptions about the structure of user programs (e.g., that users do not mix index launches and single task launches in the same outer loop where the user would expect the single task launches to get sharded).


Fix Endpoint URI for KafkProducer

When doing a route from Kafka topic to Kafka topic I noticed that functionality like:
from("kafka:localhost:9092?topic=from&zookeeperHost=localhost&groupId=mytest&zookeeperPort=2181&autoOffsetReset=smallest).to("kafka:localhost:9092?topic=to&zookeeperHost=localhost&groupId=mytest&zookeeperPort=2181);

did not work as the Exchange header of kafka.TOPIC was being used instead of the UriParam in the KafkaEndpoint for the producer.

Closing as this PR should be off of the 2.15.3 tag.


Issue with "SAML 2.0" in @site/docs/reference/integrations/saml-20.md

Describe the problem

Page: SAML 2.0
Source: https://github.com/demisto/content-docs/blob/master/content-repo/extra-docs/integrations/saml_2.md

Screenshots

Environment

OS: [e.g. Windows]
Browser: [e.g. chrome, safari, firefox..]
Browser Version:

Suggested fix

test


categories & sub categories for vulns

There is the possibility to add a category to a vulnerability.
When looking in the vulnerability template list it would be nice to be able to filter them by category.

[ ] Category (and other columns) filterable/sortable on /vulnerabilities
[ ] Category (and other columns) filterable/sortable on /vulnerabilities/templates

Having sub-categories allow more flexibility.
Example:

Web

Server-Side Injection
Security Misconfiguration
Broken Authentication and Session Management
etc.

Infrastructure

Vulnerable service
Vulnerable protocol
Vulnerable host
Bad architecture design
etc.

Mobile

Android
iOS
App related
API related
Storage related
etc.

[ ] Hierarchical categories / ability to have sub-categories

Some teams will create custom categories, other will use some OWASP reference for web, some will use CWE, some will use a national ref., some will sue bugcrowd VRT, etc.
Also actually the category list can't be configured.

It would be nice that the admin or super admin could be able to create a custom category list.

[x] Categories are customizable

unstale

Seems to work flawlessly


LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20221002155335272 to main

LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20221002155335272 to main with localized lcls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 42.425%

Totals

Change from base Build<PHONE_NUMBER>:
0.01%

Covered Lines:
29301

Relevant Lines:
64269

💛  - Coveralls


[backport] CI: Allow /test jenkins to trigger Jenkins only

Backport of #7126 by @kmaehashi

/test jenkins

Worked as expected. Updated the usage:
https://github.com/cupy/cupy/wiki/Project-Governance#pull-request-ci-guide


Fix build issue: Use partial instead of template

Only way to build successfully when running hugo server
Thought this may resolve the build issue. maybe?

Looks like it did.

My Hugo version is:
Hugo Static Site Generator v0.46/extended darwin/amd64 BuildDate: unknown

@dantudor Thank you for that. Really appreciated


Moved away from deprecated thread::sleep_ms

Refactored to use thread::sleep and time::Duration and updated link in
README.md

This is awesome, thank you so much!!!!! :tada: The one little change I made was removing the .README.md.swp file and adding *.swp to the .gitignore. Also thank you for updating the readme link!!!

I can't believe that I added in my .swp file.  So embarrassing!   Thanks for your work in putting these exercises together.  I ran through all of them and found them to be very helpful!

No worries, it happens :) I'm glad you liked the exercises!!! :heart: :heart: :heart:


Issue 88 Mark a pending payment transaction as succeeded or failed

Related issue: https://github.com/killbill/killbill-client-ruby/issues/88
Updates:
Mark a pending payment transaction as succeeded or failed

@reshmabidikar Please help me review this PR. Thanks


chore: fix dark mode text color

Describe the changes you have made in this PR
Fixes unused classes
Type of change
(Remove other not matching type)

chore

How has this been tested?

Checklist

[x] Self-review of changed code
[x] Manual testing
[x] Added automated tests where applicable
[x] Update Docs & Guides
For UI-related changes
[x] Darkmode
[x] Responsive layout

from where those colors are referenced?

Those are used in PopIcons (see classes passed to Arrow Up and Down icons)

@im-adithya In the PR description you mention "Fixes unused classes", however it seems you just adjusted the colors. Also the new colors do not seem to match the figma designs?

I actually meant dark:color-green-400 dark:color-orange-400 which don't mean anything and are unused. I took the dark classes from above as I didn't find emerald and amber colors for dark mode (see popicon colors in dark mode here and here) as we would want to match the amount color to the arrow inside the icon


Error deploying from successful build

Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name: Release
Environment

Server - Azure Pipelines
Account Name: Chris Canal
Team Project Name: Storm-Ideas
build definition name: Build and Publish images
Build Numbers:
20190521.8
20190521.9
20190522.1
20190522.2

Agent - Hosted or Private:
Host (Hosted VS2017)

Issue Description
Creating release from a successful build fails to deploy.

Task logs
Build 20190522.2-BuildAndPublish.zip
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs
Deploy Logs:

Exception Message: Exception: Microsoft.TeamFoundation.DistributedTask.Orchestration.Server.Artifacts.InvalidRequestException (type ReleaseManagementHandledException)
Exception Stack Trace:    at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseOrchestrator.FailOrchestrationImplementation(Int32 releaseId, Int32 releaseStepId, Nullable1 releaseDeployPhaseId, Action3 rejectStep, Exception ex) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseOrchestrator.cs:line 243
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseOrchestrator.FailOrchestration(Int32 releaseId, Int32 releaseStepId, Nullable1 releaseDeployPhaseId, Exception ex) in D:\v2.0\P1\_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseOrchestrator.cs:line 232    at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseOrchestrator.InvokeRunner(Release release, ReleaseEnvironmentStep deployStep, ReleaseEnvironment releaseEnvironment, DeployPhaseSnapshot snapshotToProcess, Func5 runDeployPhase, Action4 acceptStep, Action3 rejectStep, Action4 handleException) in D:\v2.0\P1\_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseOrchestrator.cs:line 493    at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseOrchestrator.ProcessDeployPhasesImplementation(Release release, ReleaseEnvironmentStep deployStep, Func5 runDeployPhase, Action4 acceptStep, Action3 rejectStep, Action`4 handleException) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseOrchestrator.cs:line 122
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseOrchestrator.ProcessDeployPhases(Release release, ReleaseEnvironmentStep deployStep) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseOrchestrator.cs:line 70
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.OrchestratorServiceProcessorV2.ExecuteDeployStep(Release release, ReleaseEnvironmentStep releaseEnvironmentStep) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\OrchestratorServiceProcessorV2.cs:line 1149
Inner Exception Details:
Exception Message: VS402925: Build ID '20190522.2' is invalid. Make sure that the build ID is valid and try again. (type InvalidRequestException)
Exception Stack Trace:    at Microsoft.VisualStudio.Services.ReleaseManagement.Artifact.Extensions.Build.BuildArtifact.TryParseBuildId(InputValue versionData, Boolean shouldThrow) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\ArtifactType\Build\BuildArtifact.cs:line 3302
at Microsoft.VisualStudio.Services.ReleaseManagement.Artifact.Extensions.Build.BuildArtifact.GetArtifactConfigurationVariables(IVssRequestContext context, ArtifactSource artifactSource) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\ArtifactType\Build\BuildArtifact.cs:line 2401
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseRunner.GetArtifactVariables(Release release, IVssRequestContext requestContext) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseRunner.cs:line 378
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseRunner.GetMergedReleaseVariables(Release release, ReleaseEnvironment releaseEnvironment, ReleaseEnvironmentSnapshotDelta deploymentDelta, Boolean includeArtifactVariables) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseRunner.cs:line 581
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseRunner.GetAutomationEngineInput(Release release, ReleaseEnvironment releaseEnvironment, ReleaseEnvironmentStep step, Int32 t
Exception: Microsoft.TeamFoundation.DistributedTask.Orchestration.Server.Artifacts.InvalidRequestException

Agent Log:

##[Error 1]
Exception Message: VS402925: Build ID '20190522.2' is invalid. Make sure that the build ID is valid and try again. (type InvalidRequestException)
Exception Stack Trace:    at Microsoft.VisualStudio.Services.ReleaseManagement.Artifact.Extensions.Build.BuildArtifact.TryParseBuildId(InputValue versionData, Boolean shouldThrow) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\ArtifactType\Build\BuildArtifact.cs:line 3302
at Microsoft.VisualStudio.Services.ReleaseManagement.Artifact.Extensions.Build.BuildArtifact.GetArtifactConfigurationVariables(IVssRequestContext context, ArtifactSource artifactSource) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\ArtifactType\Build\BuildArtifact.cs:line 2401
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseRunner.GetArtifactVariables(Release release, IVssRequestContext requestContext) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseRunner.cs:line 378
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseRunner.GetMergedReleaseVariables(Release release, ReleaseEnvironment releaseEnvironment, ReleaseEnvironmentSnapshotDelta deploymentDelta, Boolean includeArtifactVariables) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseRunner.cs:line 581
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseRunner.GetAutomationEngineInput(Release release, ReleaseEnvironment releaseEnvironment, ReleaseEnvironmentStep step, Int32 trialNumber, DeployPhaseSnapshot snapshotToProcess, ReleaseEnvironmentSnapshotDelta deploymentDelta) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseRunner.cs:line 286
at Microsoft.VisualStudio.Services.ReleaseManagement.Server.Processors.DeployPhaseRunner.Run(Release release, ReleaseEnvironment releaseEnvironment, ReleaseEnvironmentStep deployStep, DeployPhaseSnapshot snapshotToProcess) in D:\v2.0\P1_work\13\s\ReleaseManagement\Service\ReleaseManagement2\Server\Processors\DeployPhaseRunner.cs:line 190

@chris-stormideas how are you creating releases from successful build. Is it manual create release dialog and typing in "20190521.8" into combo box? "20190521.8" is a build number but not build ID. You cannot create releases from "build number" but you create them using "build ID - which is an integer".
Recently we have made an enhancement to fail create release API request if the build ID is invalid. That will block you from creating releases if invalid build ID is passed.
You can find build ID of your build from URL parameters. For example: https://dev.azure.com/OrgName/ProjectName/_build/results?buildId=9636341&view=results, 9636341 is the build ID.

Hi @Lovakumar,
Nothing manual was performed, all initiated via the UI and the combo box values being auto selected.

@chris-stormideas we have made couple of fixes in this area. Can you please check your scenario and let us know if it still repro-ing for you.

@chris-stormideas we have made couple of fixes in this area. I am closing this issue for now. If it is still repro-ing for you, please reply back with repro details and we will re-activate this issue.


Investigate if CVE-2023-34396 affects struts1

Thanks to this project for keeping legacy Struts 1.x solutions secured.  It is great to see.
I recently learned of CVE-2023-34396 which was published in June 2023, and it says it affects Struts 2.x and 6.x, but the affected versions list all versions, including 1.x.  See: https://www.cve.org/CVERecord?id=CVE-2023-34396
In looking at this, I believe it may also affect Struts 1.x, particularly in how multipart form uploads are handled.  I think the CVE primarily affects CommonsMultipartRequestHandler:
https://github.com/weblegacy/struts1/blob/main/core/src/main/java/org/apache/struts/upload/CommonsMultipartRequestHandler.java
Here's how the CVE was fixed in Struts 2.5.31, as shown here: https://github.com/apache/struts/compare/STRUTS_2_5_30...STRUTS_2_5_31?diff=split&w=#diff-9c690161b1ba4ba15ccf8b3991857785b66c99dd2a87a95543b18e411deeb17f
Thoughts?

Hello @nwbarrett,
thank you very much for your report about the CVE and your analysis and the hint for the solution! I will schedule it for the next release. I hope to make the change in the next 2 to 3 weeks.
Regards
Stefan

Hi @nwbarrett,
the CVE-2023-34396 is fixed in the current main branch: 3b4bfd2e735d88cad6caf0ac2940c9f5704841d0.
Greetings
Stefan

Hi @nwbarrett,
this issue is released with version 1.5.0-RC2.
Greetings
Stefan


Add dependencies to setup.py file

Currently no requirements are set in setup.py

Hey @lubianat even though this is fixed, it isn't up on PyPI yet, so you still need to bump the version and upload it

Sure, thanks. I've been bumping versions manually, do you have a workflow
for that?
Em ter., 24 de jan. de 2023 10:19, João Vitor @.***>
escreveu:

Hey @lubianat https://github.com/lubianat even though this is fixed, it
isn't up on PyPI yet, so you still need to bump the version and upload it
—
Reply to this email directly, view it on GitHub
https://github.com/lubianat/wdcuration/issues/7#issuecomment-1401937700,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB4NC74EHUORUZVU6KEE3I3WT7JEZANCNFSM6AAAAAAT3MUFCA
.
You are receiving this because you were mentioned.Message ID:
@.***>


Ordered list inside stat block not displaying correctly.

The first line in an ordered list inside stat block not displaying correctly. I am seeing the number, with the line starting on the next line below. The line below the ordered list is indented when it shouldn't be, Also, when changing this to a standard state block, the page layout get all weird.
Additional Details
http://homebrewery.naturalcrit.com/share/rJXFUU9aYW
or
Brew code to reproduce : Click to expand

Spectator
Medium aberration, lawful neutral

Armor Class 14 (natural armor)
Hit Points 39 (6d8 + 12)
Speed 0 ft., fly 30 ft.

STR
DEX
CON
WIS
INT
CHA

8 (-1)
14 (+2)
14 (+2)
14 (+2)
13 (+1)
11 (+0)

Skills Perception +6
Condition Immunities prone
Senses darkvision 120 ft., passive Perception 16
Languages Deep Speech, Undercommon
Challenge 3 (700 XP)

Hover. The spectator hovers as long as it is alive.
Telepathy. The spectator can communicate telepathically with any creature within 100 feet of it that can understand a language.

Actions
Bite. Melee Weapon Attack: +1 to hit, reach 5 ft., one target. Hit: 2 (1d4 - 1) piercing damage.
Eye Rays. The spectator uses two of the following eye rays. It can use each ray only once per turn. Each ray targets a creature the spectator can see within 90 feet of it.

Confusion Ray. The target must succeed on a DC 13 Wisdom saving throw, or it can't take reactions until the end of its next turn. On its turn, the target can't move, and it uses its action to make a melee or ranged attack against a randomly determined creature within range. If the target can't attack, it does nothing on its turn.

Paralyzing Ray. The target must succeed on a DC 13 Constitution saving throw or be paralyzed for 1 m>nute. The target can repeat the saving throw at the end of each of its turns, ending the effect early on a success.

Fear Ray. The target must succeed on a DC 13 Wisdom saving throw or be frightened for 1 minute. The target can repeat the saving throw at the end of each of its turns, with disadvantage if the spectator is visible to the target, ending the effect early on a success.

Wounding Ray. The target must make a DC 13 Constitution saving throw, taking 16 (3d10) necrotic damage on a failed save, or half as much damage on a successful one.

Create Food and Water. The spectator creates enough food and water to sustain itself for 24 hours.
Reactions
Spell Reflection. If the spectator makes a successful saving throw against a spell, or a spell misses it, the spectator can choose another creature within 30 feet of it that it can see. The spell affects the chosen creature instead of the spectator.

You can work around the problem by removing the blank lines after every list item: http://homebrewery.naturalcrit.com/share/SyW_Id3AKZ. (I'm not sure what exactly goes wrong in the style sheet with those lines there, but given the workaround is easy it seems best to just use it.)

huh. I could have sworn that I had tried that. Thanks.
So any idea how to get rid of the indent on Create Food and Water?

There's a style rule that adds a 1em indent to .phb ol+p. I assume that's necessary in certain cases, but you can customise your existing <style> block by adding    .phb hr+blockquote ol+p { text-indent: 0px; } -- hopefully that won't break anything else.

Sorry, I am just now getting back to this. That worked thanks.


Add test skeleton test_snapshot_cr

longhorn/longhorn#6298
https://github.com/longhorn/longhorn/issues/6312

@mergify backport v1.5.x


Automatic observation in VDOM

I've found my Cycle code has quite a bit of boilerplate in view code, around combining and mapping multiple observables. Simplistically, it ends up with a chunk of code at the start of each view function - a source of merge conflicts. Or less simplistically but more performantly, one can further entangle mapping with VDOM to recreate less VDOM unnecessarily. These things are much discussed in #204.
Reading about CALMM's Karet:
https://github.com/calmm-js/karet
...  I wonder if something similar is workable with Cycle? Some better VDOM driver that understands embedded observables, and "does the right thing". Offhand it seems that notion could fit reasonably in Cycle's architecture. If such a thing exists, I'd appreciate a pointer. Or tips on how to achieve it.
(I put this in as an issue rather than on Gitter, so that the next person searching can find it down the road.)

Hi Kyle. We had this type of feature long ago in Cycle.js, we used to call it "transposition" and recently we've decided to drop it in Cycle Unified: https://cycle.js.org/releases.html#cycle-unified-breaking-changes
We used to be quite excited about it, but it turns out it looks and behaves a bit too magically for the average beginner, and turned out to be a source of confusion even though it was neat. Overall the need for a combine prior to vdom creation is not that big of a hassle. Additionally, removing transposition simplified the implementation too, specially when supporting multiple stream libraries.
@TylorS can comment more if he feels like adding something.

@staltz Ah, thanks, that reasoning sounds right on, particularly about ease of understanding for beginners. Is such a thing possibly as an option add-on, though? Either as an extension/replacement cycle/http, or a function to pass vdom through on the way to cycle/http?
I've found that I can't resist the simplicity of just xs.combine()ing at the top - when more complexity could avoid needlessly recreating an entire component's VDOM, when only a small portion could have changed.

@kylecordes https://github.com/motorcyclejs/motorcyclejs/issues/35

when more complexity could avoid needlessly recreating an entire component's VDOM, when only a small portion could have changed.

In the rare occasions that I use React, I'm very aware of this issue, because creating a deep tree of ReactElements (equivalent to VNodes) is expensive and that shows clearly in performance. However, I've surprised myself with snabbdom, that creating thousands of VNodes didn't affect framerate. It does affect GC, though, but not that visibly to the user. In those cases, you can use a snabbdom thunk to avoid recreating a portion of the tree.

@staltz Ah, interesting. I was aware that React's VDOM implementation is a bit heavy, but not aware the difference was so stark. I will assume the straightforward single xs.combine() at the top of each view is performant enough until measurement shows otherwise.
@typeetfunc Thanks... that will take some reading and thinking.
(I admit I am a bit fuzzy on how Cycle and Motorcycle fit together - I know the answer used to be "like Cycle, but with Most and Snabbdom", but now Cycle can use Most and does use Snabb. Wish there was a wiki page or whatever with an up-to-date explanation.)

@staltz What do you think about some kind of Ahead-of-Time Compilation to convert vtree's written with transposition, to a non-transposed syntax (Am I make sense)?

@geovanisouza92 Speaking of AOT, I am very (very) familiar with Angular, which has a slick AOT implementation for its template syntax. It has lots of advantages, and as tooling support improves, few disadvantages. Vue.js is somewhat similar in this way, offering build-time compilation of templates.
But.
There is one key limitation of systems which AOT: they are inherently less dynamic than systems which don't AOT. With an AOT system, you can't use the same tools programming at runtime, to make new "components", as are used during development. This makes some ideas, like generic, runtime generation of complex UIs, less achievable.
Currently, Cycle is 100% dynamic. I'm hesitant to recommend moving away from that. I'm thinking the automatic-observation idea (modulo @staltz 's point about ease of understanding for beginners) would still allow complete dynamism.

I think the biggest problem with transposition for Cycle.js is that it makes it hard to visualize in dataflow, and we're likely going to focus more and more on Cycle.js as a framework where you code through visualizable dataflow graphs. A combine of multiple vdom streams is flat in the dataflow chart and is easily visualizable either in paper sketches or DevTool and beyond. How do you sketch transposition as a dataflow graph? It's hard because transposition is both "combineLatest" and "flatMap" intertwined in the vdom tree. Flatmap itself is already hard to visualize in a dataflow graph.
Don't get me wrong, I understand the nice side of transposition, I came up with it independently, I thought it was my own idea, but it's so simple it could be co-invented elsewhere. Transposition is also similar to the react mental model where you embed an dynamic instance of a component inside another component.
The verdict is that this is a feature that, despite being cool, doesn't fit well with the guiding principles in Cycle.js: explicitness (or "traceability"), visualizable code, readability over writability.

This comment isn’t particularly relevant to the thread.
@kylecordes, with Motorcycle, we use mostly-dom, a derivative of snabbdom with some rather large breaking changes and design differences. "Drivers" are no longer special in Motorcycle, instead it’s possible to design the effectful side of your application using just functions as one would in the main function. Motorcycle, because of the different run type signature, requires a browser that supports ES2015’s Proxy.
If you’d like to know more, head over to the Motorcycle chat room

Indeed, we are stretching the connection to original topic. But... speaking of VDOM, perhaps the thing I like least about Cycle, is that cycle-dom appears to be mostly untyped and offers no compile-time or in-IDE assistance about the structure of h() properties. mostly-dom looks pretty compelling, therefore. Still, I suspect I'm better off not dividing attention between cycle and motorcycle (yet?).

That's just a matter of doing it, I believe most of those typings could be done in Cycle DOM. PRs welcomed. :) Sometimes with the cross-pollination between Cycle and Motorcycle it's hard to keep up-to-date on either of the sides.

I want to second @kylecordes in

I've found my Cycle code has quite a bit of boilerplate in view code, ...

My uneasiness feeling comes from using .select(). I haven't selected anything from dom in years. Hence, it feels like an unnecessary chore.
Cycle as general tool, vs more narrow task of html-based UI (setting conceptual context)
It is true, that cycle grew as a UI building tool. UI provides a good ground for testing the tool, and showing its awesomeness. But, with visualization of data flow, cycle becomes a much more general tool.
Therefore, we may look at creating html-based UI as one more task. And this task may have different solutions that are built and work on cycle's.
HTML flow, as it is done with existing approach: driver, isolate, ...
It seems to be a common pattern:

Select elements from dom, to assemble actual sources, which event should push our component.
Wire business logic circuit starting with sources, all way to things that either displayed in dom, or regulate structure of dom.
Consume things, generated by logic, turn it into html-respresenting structure, sinking it downstream.

Of cause, separation between items 2 and 3 depends on task, at hand. When it is something like beautiful matrix multiplication example by @staltz , there is no clear boundary. But when it is a case of boring form, or form-like UI, or UI that doesn't transform much, has tons of truly html things, separation between 2 and 3 is usually very clear indeed.
The boring case is the majority of what is written. Even the name, business logic, gives it away.
The reason I haven't been doing any dom selection for years now, is that boilerplate of 1 and 3 is done by template system.
I also had, once, 2.5 year ago a place with really nice dynamic UI component, which was on a brink of insanity, when written in angular's template. Cycle's raw creation of elements would've simplified that problem for me.
Template vs raw dom dump is a false dilemma for current cycle
It may seem that react-like style of assembling dom at the bottom of data flow and use of templates like vue or angularjs are incompatible, forcing devs to choose either-or (mind you, this polarization is a great marketing tool :smiling_imp: )
I'd argue that vue/angular-like template system for components can be done on top of cycle, with use of dom driver directly still available as a row, more flexible tool, for places that cannot be served by much simpler templating.
The unexpected awesomeness here will be that, when template system is written cycle, at runtime it creates boilerplate circuitry around business logic circuitry, and mechanics of template becomes observable in cycle devtool, giving total introspection into an opinionated template tool. Template tools have to be opinionated, because they have to implement items 1 and 3. The problem with opinions is that they can be surprising to user, when user tries to catch a bug. Observability in cycle devtool will totally eliminate such surprise.
Templated components lib
Of cause, templates should be a library. This has no place in cycle's core.
Of cause, at least initially, template should be simple: places for named sources, places that use memory streams (or signals, or pullables, can be updated, when cycle gets these). Couple of directives like vue's v-if and v-for is enough for simple components, which is the best place to use template approach. Filters have no place in templates. Filters are things inside event/data pipeline/stream.
Template system takes care of item 1 and 3, thus, wrapping business cycle item 2. So, given main function businessLogic(sources) => sinks, creation of template-d component may look like
TemplatedComponent(template: string, businessLogic, sources) => sinks

Of cause, templated system should have a registry of known components, so that html tag <my-component> is mapped to registered component creating function.
Of cause, since templated system is tasked with insertion/removal of html, it should have its own opinionated routing.
A particular template system library may have its own specific hooks for devtools. Click on source of clicks, and devtool shows html element that produces clicks, and a place in a template. There is a room to go wild.
No matter how template system is done, it becomes transparent in devtool. No other framework has this magic!
With such lib, cycle has a great chance to go mainstream, I feel.

Let's put counter example into template-d form.
Original is
function main(sources) {
  const decrement$ = sources.DOM
    .select('.decrement').events('click').mapTo(-1);

  const increment$ = sources.DOM
    .select('.increment').events('click').mapTo(+1);

  const action$ = xs.merge(decrement$, increment$);
  const count$ = action$.fold((x, y) => x + y, 0);

  const vdom$ = count$.map(count =>
    div([
      button('.decrement', 'Decrement'),
      button('.increment', 'Increment'),
      p('Counter: ' + count)
    ])
  );
  return { DOM: vdom$ };
}

Template for this can be

  <button @click="decrement$">Decrement</button>
  <button @click="increment$">Increment</button>

Counter: {count$}

and business logic function is
function counter(sources) {
  const decrement$ = sources.decrement$.mapTo(-1);

  const increment$ = sources.increment$.mapTo(+1);

  const action$ = xs.merge(decrement$, increment$);
  const count$ = action$.fold((x, y) => x + y, 0);

  return { count$ };
}

Somehow, I feel that counter becomes simpler, cause there is no dom there.
Template figures itself that @click are places that produce events, hence, these are sources for logic, while other places consume events, and need streams (memory streams?) from logic part. If output stream is missing, template can fail fast with precise error.

Note how use of templates shields from use of ids. And classes in such html will be for css styling, not for locating from js.

This template idea sounds like a much larger thing, to result in a much more different framework to use, than the bit of view boilerplate reduction I have been thinking of - Automatic fine-grained observation.  I'm not sure I fully understand yet how to get optimal results without it.  I wrote a bit of code which suffers from retreating too much VDOM for each change, and I partially (only somewhat effectively) took care of the resulting overhead by using a snabbdom thunk.
I'm not sure whether the template idea would \ preserve what I think is one of the most distringuishing aspects of Cycle: that there is no extra framework-provided layer or magic glue between one component and another. Still, if someone made such a template add on for Cycle it would be quite interesting to try out and compare.

@kylecordes I am with you on

if someone made such a template add on for Cycle it would be quite interesting to try out and compare.

It seems that there can room for experimentation (fun), and it will give more options.
On

I'm not sure whether the template idea would \ preserve what I think is one of the most distringuishing aspects of Cycle: that there is no extra framework-provided layer or magic glue between one component and another.

Template system should be adding things only between your logic circuit and dom on entry and on exit. All other sources/sinks should be just let to pass through wrapping function, resulting in zero additions between your logical components.
devtool will show you what template system adds, so that you can immediately judge if library tastes good to oneself.


Allow explicit config of paths

You currently require that LND's macaroon and cert are in a specific path(s). In BTCPay, we have them elsewhere which makes it impossible to integrate helipad (https://github.com/btcpayserver/btcpayserver-docker/pull/567). I opened a PR with the required changes but I don't have a rust IDE and don't know its syntax to finish the PR. https://github.com/Podcastindex-org/helipad/pull/8
Ideally, LND_ADMINMACAROON and LND_TLSCERT environment variables would take precedence if set, and helipad would attempt them first.

Ah, this is a duplicate of #5

I’m fixing this right now. The new behavior will be that env variables take precedence over everything, then a config file will be read that specifies the various paths wanted, then finally a set of sane fallbacks will be used. These changes will be in the next release.


Runtime Error

Thanks for your error report and we appreciate it a lot.
Checklist

I have searched related issues but cannot get the expected help.
The bug has not been fixed in the latest version.

Describe the bug
A clear and concise description of what the bug is.
Reproduction

What command or script did you run?
python -u ./tools/train.py configs/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k.py

Did you make any modifications on the code or config? Did you understand what you have modified?
use mmcv==1.3.9

What dataset did you use?
ADEChallengeData2016
Environment

Please run python mmseg/utils/collect_env.py to collect necessary environment infomation and paste it here.
sys.platform: linux
Python: 3.8.2 (default, Mar 26 2020, 15:53:00) [GCC 7.3.0]
CUDA available: True
GPU 0: GeForce RTX 3090
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.2.r11.2/compiler.29618528_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.9.0
PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) oneAPI Math Kernel Library Version 2021.2-Product Build 20210312 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.1
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
CuDNN 8.0.5
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.10.0
OpenCV: 4.5.3
MMCV: 1.3.9
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMSegmentation: 0.11.0+
2021-08-13 16:43:11,382 - mmseg - INFO - Distributed training: False
2021-08-13 16:43:11,711 - mmseg - INFO - Config:
norm_cfg = dict(type='SyncBN', requires_grad=True)

You may add addition that may be helpful for locating the problem, such as

How you installed PyTorch [e.g., pip, conda, source]
Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Error traceback
If applicable, paste the error trackback here.
Traceback (most recent call last):
  File "/home/wyh/Codes/SwinTransformer/tools/train.py", line 163, in <module>
    main()
  File "/home/wyh/Codes/SwinTransformer/tools/train.py", line 152, in main
    train_segmentor(
  File "/home/wyh/Codes/SwinTransformer/mmseg/apis/train.py", line 116, in train_segmentor
    runner.run(data_loaders, cfg.workflow)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 133, in run
    iter_runner(iter_loaders[i], **kwargs)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 60, in train
    outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/wyh/Codes/SwinTransformer/mmseg/models/segmentors/base.py", line 152, in train_step
    losses = self(**data_batch)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func
    return old_func(*args, **kwargs)
  File "/home/wyh/Codes/SwinTransformer/mmseg/models/segmentors/base.py", line 122, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/wyh/Codes/SwinTransformer/mmseg/models/segmentors/encoder_decoder.py", line 157, in forward_train
    loss_decode = self._decode_head_forward_train(x, img_metas,
  File "/home/wyh/Codes/SwinTransformer/mmseg/models/segmentors/encoder_decoder.py", line 100, in _decode_head_forward_train
    loss_decode = self.decode_head.forward_train(x, img_metas,
  File "/home/wyh/Codes/SwinTransformer/mmseg/models/decode_heads/decode_head.py", line 186, in forward_train
    seg_logits = self.forward(inputs)
  File "/home/wyh/Codes/SwinTransformer/mmseg/models/decode_heads/uper_head.py", line 92, in forward
    laterals = [
  File "/home/wyh/Codes/SwinTransformer/mmseg/models/decode_heads/uper_head.py", line 93, in <listcomp>
    lateral_conv(inputs[i])
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/mmcv/cnn/bricks/conv_module.py", line 200, in forward
    x = self.norm(x)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 731, in forward
    world_size = torch.distributed.get_world_size(process_group)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 748, in get_world_size
    return _get_group_size(group)
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 274, in _get_group_size
    default_pg = _get_default_group()
  File "/home/wyh/anaconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 358, in _get_default_group
    raise RuntimeError("Default process group has not been initialized, "
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

Same error with single-gpu, but multi-gpu training works.

Hi, it seems that this is caused by SyncBN. For training with single gpu, you may use BN instead. Just modify norm_cfg like this:
norm_cfg = dict(type='BN', requires_grad=True)


How to compute the OA and mIoU on the Semantickitti dataset

How do you compute the OA and mIoU on the Semantickitti dataset? You don't seem to have published the code for it.

@longmalongma Hi, when you submit to official semanticKITTI benchmark, it will generate output log for OA and mIoU.  Also, if you want to compute OA and mIoU for validation set, you can use official semanticKITTI api here just by following codes:
$ ./evaluate_semantics.py --dataset /path/to/kitti/dataset/ --predictions /path/to/method_predictions --split train/valid/test # depending of desired split to evaluate

By running test_semantic_kitti_grid.py, it will automatically generate predicted labels for evaluation.

@longmalongma Hi, when you submit to official semanticKITTI benchmark, it will generate output log for OA and mIoU. Also, if you want to compute OA and mIoU for validation set, you can use official semanticKITTI api here just by following codes:
$ ./evaluate_semantics.py --dataset /path/to/kitti/dataset/ --predictions /path/to/method_predictions --split train/valid/test # depending of desired split to evaluate

By running test_semantic_kitti_grid.py, it will automatically generate predicted labels for evaluation.

Thanks for your reply, I see it!

@longmalongma Hi, when you submit to official semanticKITTI benchmark, it will generate output log for OA and mIoU. Also, if you want to compute OA and mIoU for validation set, you can use official semanticKITTI api here just by following codes:
$ ./evaluate_semantics.py --dataset /path/to/kitti/dataset/ --predictions /path/to/method_predictions --split train/valid/test # depending of desired split to evaluate

By running test_semantic_kitti_grid.py, it will automatically generate predicted labels for evaluation.

Thanks for your reply, I see it!

Can it compute OA and mIoU for test set???

@longmalongma You need to submit on official benchmark of semanticKITTI

Thanks for your reply, can you tell me which files ｓhould I submit? only .label files from sequences 08,11 to 21? Can you give me the web site to refer to the predicted results？　Ｓorry, I am first use senmantickitti daatset.

Hi, @longmalongma , sequence 08 only for validation set. If you want to submit to official semantickitti benchmark here, you need to firstly generate sequence 11~21. Then zip them together and submit to the single scan competitions.

Hi, @longmalongma , sequence 08 only for validation set. If you want to submit to official semantickitti benchmark here, you need to firstly generate sequence 11~21. Then zip them together and submit to the single scan competitions.

Thanks, it only need .label files?