[CHECKLIST]: Geo Processor Code Review

General Description
Geo Processor container code review, refactor, and processor evaluation.  Initial review of processor code and fixes provided an opportunity to optimize, document, and define a standard Python coding format.  Pylint and Black Python packages are used to implement a code formatting standard with Numpy docstring formatting to document classes, method, and modules.  Optimization of the code has helped reduce redundant code, better documentation and better readability.  Documentation can be generated from the code instead of double documenting.
TODO

[x] Refactoring Code
[x] Implement Pylint and Black with Numpy as the docstring format
[x] Processor Review and bug fixes

@jeffsuperglide
Reviewing branch: https://github.com/USACE/cumulus-api/tree/refactor/complete-overhaul-snodas

[ ] HRRR appears to be fixed.  Resulting DSS looks good.
[ ] Need DB Migration to set temporal_duration back to 3600 (back to PER-CUM in DSS)
[ ] SNODAS Interpolated needs date fix - showing one day behind primary product

Spot checking products.
Develop vs Stable - matches

[ ] NDGD RTMA AIRTEMP 1hr
[ ] NDGD RTMA PRECIP 1hr

@jeffsuperglide Do we need to check anything else before closing this?


希望可以把微信和企业微信整合到一个docker里面

希望可以把微信和企业微信整合到一个docker里面，不然一个镜像4个G 太大了

Actually, those two images are sharing the same base image, which means that they are sharing your storage already. :)


YouTube Comments

Hello, I notice in #17 it's stated that getting comments is not part of the InnerTube API. I'm not sure if things have changed or if I am misunderstanding what constitutes as part of the InnerTube API, but by doing the following I have managed to get the comments:

Send a next request to https://www.youtube.com/youtubei/v1/next?key={key} with the specified video ID in the data.
Extract the continuation token. There's a default, a "Top" sort, and a "New" sort. I've only tried the default.
Sending a second next request without specifying the video ID, but instead specifying the continuation in the data block.
This should return the first 20 or so comments in a very ugly nested way.

Something I've yet to figure out is how to get a highlighted comment to appear at the top of the json list. If you click on a YouTube comment's date, it will open a link with a "&lc=" param that has the comment's ID. And in the comments it will appear at the top as "Highlighted".
If I use the continuation token for the second request from the dev tools inspector when loading the highlighted comment link in the browser then the second next request properly returns the highlighted comment at the top of the json list.
However, if I try using the continuation retrieved from the first next request programmatically then it always returns the comments without the highlighted comment at the top, so it can be assumed the highlighted comment is tied to the continuation token which seems to be generated outside of the scope of the next endpoint, unless I've simply not found the correct way yet.

Hi, apologies for the late reply, I'll take a look into this now

I've been able to reproduce the ability to list the first n comments (either "top" or "newest").
Here's the (admittedly lashed together) script I used:
from innertube import InnerTube

ENGAGEMENT_SECTION_COMMENTS = "engagement-panel-comments-section"
C0MMENTS_TOP = "Top comments"
COMMENTS_NEWEST = "Newest first"

def parse_text(text):
    return "".join(run["text"] for run in text["runs"])

def extract_engagement_panels(next_data):
    engagement_panels = {}
    raw_engagement_panels = next_data.get("engagementPanels", [])

    for raw_engagement_panel in raw_engagement_panels:
        engagement_panel = raw_engagement_panel.get(
            "engagementPanelSectionListRenderer", {}
        )
        target_id = engagement_panel.get("targetId")

        engagement_panels[target_id] = engagement_panel

    return engagement_panels

def parse_sort_filter_sub_menu(menu):
    menu_items = menu["sortFilterSubMenuRenderer"]["subMenuItems"]

    return {menu_item["title"]: menu_item for menu_item in menu_items}

def extract_comments(next_continuation_data):
    return [
        continuation_item["commentThreadRenderer"]
        for continuation_item in next_continuation_data["onResponseReceivedEndpoints"][
            1
        ]["reloadContinuationItemsCommand"]["continuationItems"][:-1]
    ]

# YouTube Web CLient
client = InnerTube("WEB", "2.20240105.01.00")

# ShortCircuit - Dell just DESTROYED the Surface Pro! - Dell XPS 13 2-in-1
video = client.next("BV1O7RR-VoA")

engagement_panels = extract_engagement_panels(video)
comments = engagement_panels[ENGAGEMENT_SECTION_COMMENTS]
comments_header = comments["header"]["engagementPanelTitleHeaderRenderer"]
comments_title = parse_text(comments_header["title"])
comments_context = parse_text(comments_header["contextualInfo"])
comments_menu_items = parse_sort_filter_sub_menu(comments_header["menu"])
comments_top = comments_menu_items[C0MMENTS_TOP]
comments_top_continuation = comments_top["serviceEndpoint"]["continuationCommand"][
    "token"
]

print(f"{comments_title} ({comments_context})...")
print()

comments_continuation = client.next(continuation=comments_top_continuation)

comments = extract_comments(comments_continuation)

for comment in comments:
    comment_renderer = comment["comment"]["commentRenderer"]

    comment_author = comment_renderer["authorText"]["simpleText"]
    comment_content = parse_text(comment_renderer["contentText"])

    print(f"[{comment_author}]")
    print(comment_content)
    print()

(I'll add this to the examples/ directory in case it helps anyone else)
I'll have a fiddle with highlighting a comment now in case I can figure out what's going on there

It looks like highlighting a comment sends off a request to the /next endpoint with some params and the videoId. I'll see if I can whip up a quick PoC for this now

I think I've figured out what was happening with highlighting a comment not working. The continuation tokens for "top" and "newest" you can extract from engagementPanels aren't influenced by the params passed to the /next endpoint, however the continuation token for the comment-item-section does change.
The below example ignores the engagementPanels entirely and instead uses the continuation token for the comments item section:
from innertube import InnerTube

# YouTube Web CLient
CLIENT = InnerTube("WEB", "2.20240105.01.00")

def parse_text(text):
    return "".join(run["text"] for run in text["runs"])

def flatten(items):
    flat_items = {}

    for item in items:
        key = next(iter(item))
        val = item[key]

        flat_items.setdefault(key, []).append(val)

    return flat_items

def flatten_item_sections(item_sections):
    return {
        item_section["sectionIdentifier"]: item_section
        for item_section in item_sections
    }

def extract_comments(next_continuation_data):
    return [
        continuation_item["commentThreadRenderer"]
        for continuation_item in next_continuation_data["onResponseReceivedEndpoints"][
            1
        ]["reloadContinuationItemsCommand"]["continuationItems"][:-1]
    ]

def extract_comments_continuation_token(next_data):
    contents = flatten(
        next_data["contents"]["twoColumnWatchNextResults"]["results"]["results"][
            "contents"
        ]
    )
    item_sections = flatten_item_sections(contents["itemSectionRenderer"])
    comment_item_section_content = item_sections["comment-item-section"]["contents"][0]
    comments_continuation_token = comment_item_section_content[
        "continuationItemRenderer"
    ]["continuationEndpoint"]["continuationCommand"]["token"]

    return comments_continuation_token

def get_comments(video_id, params=None):
    video = CLIENT.next(video_id, params=params)

    continuation_token = extract_comments_continuation_token(video)

    comments_continuation = CLIENT.next(continuation=continuation_token)

    return extract_comments(comments_continuation)

def print_comment(comment):
    comment_renderer = comment["comment"]["commentRenderer"]

    comment_author = comment_renderer["authorText"]["simpleText"]
    comment_content = parse_text(comment_renderer["contentText"])

    print(f"[{comment_author}]")
    print(comment_content)
    print()

video_id = "BV1O7RR-VoA"

# Get comments for a given video
comments = get_comments(video_id)

# Select a comment to highlight (in this case the 3rd one)
comment = comments[2]

# Print the comment we're going to highlight
print("### Highlighting Comment: ###")
print()
print_comment(comment)
print("---------------------")
print()

# Extract the 'params' to highlight this comment
params = comment["comment"]["commentRenderer"]["publishedTimeText"]["runs"][0][
    "navigationEndpoint"
]["watchEndpoint"]["params"]

# Get comments, but highlighting the selected comment
highlighted_comments = get_comments(video_id, params=params)

print("### Comments: ###")
print()

for comment in highlighted_comments:
    print_comment(comment)

$ python app.py
### Highlighting Comment: ###

[@alphacompton]
The built in mic on the 2-1 is exceptional and the camera is excellent from your video sample. Look like a better buy especially if it's cheaper than the Surface pro.

---------------------

### Comments: ###

[@alphacompton]
The built in mic on the 2-1 is exceptional and the camera is excellent from your video sample. Look like a better buy especially if it's cheaper than the Surface pro.

[@ouilsen2]
As a Surface Pro user I have one observation....

...

Hope that helps!
Please let me know if you have any further questions, or if this answers your query
Best, Tom

Hi, thanks for the detailed reply.
The idea behind the highlighting was to store a reference (such as the comment ID) to it in a database and come back to it later. One such use case would be a system that checks for the existence of a membership badge on a user's message monthly. That's why it would have been ideal to have a way to programmatically jump straight to the comment in 1 request like in the browser (on the initial lookup, not just subsequent ones).
Unfortunately from your response it seems "highlighting" a comment internally is done with the comment's watchEndpoint params, so the initial request for the comment will require scraping them all until the target comment is found by checking for the comment ID, and then storing the params instead of the comment ID for future immediate lookup.
Would this work, or do you suspect the params of comments change often?
Thanks again.

Hi @EtorixDev, apologies for the late turn around on a reply to your last comment. I believe the params field contains base-64 encoded protobuf data (potentially also url-encoded). You should be able to decode the contents of the param using a tool such as https://protobuf-decoder.netlify.app/. It is possible that the protobuf structure contains the comment ID, and that all other fields are static. If this is the case, you should be able to generate the correct params value knowing only the comment ID.
Unfortunately I went to test this using the examples/list-video-comments-highlighted.py example script I wrote a while back and it seems YouTube has changed their comments API around again. If I get some spare time I'll give the API another poke, however I hope this comment has at least given you a bit of a steer :slightly_smiling_face:

Hi,
I was wondering if you had any information regarding the Comments Preview widget and how the comments could be obtained externally - it seems to cycle between specific kinds of comments rather than ones sorted by relevance.
(not my screenshot, but the comment preview is there)


Emit a warning if the base URL leads to an empty page

We met with a potential user today who initially configured their ADO pipeline with a url parameter that was slightly off, resulting in the scanner seeing a blank page. The scan appeared successful, but wasn't really scanning anything meaningful.
It would be nice for the action to detect this case and warn about the probable misconfiguration. Maybe something along the lines of "empty body" and/or "4xx/5xx response code" to trigger a warning. It should probably either only apply to the base URL, be suppressable, or both.

PM to determine next steps for UI changes

We will bundle this into feature work: ADO-Improvements -1

I'm not certain we want to pursue this.  If the webserver returns a success code the page is scanned. The scanner package does check the status codes and throws if the page doesn't exist. In the case of the user mentioned above, I assume the webserver was serving a page it just happened to be empty? That seems like an edge case that on its own is probably good for the end-user to fix.
The accessibility-insights-scan package doesn't currently have infrastructure for detecting a blank page.  If the base URL or a linked page is served successfully with a 200 status code but is blank it will return two failure instances from axe-core:
2 failure instances
* (1) document-title:  Ensures each HTML document contains a non-empty <title> element
* (1) html-has-lang:  Ensures every HTML document has a lang attribute

There is also no way of doing this without modifying the service repo to handle this case.  If we were to pursue this further, we would likely want to make it optional with a flag that can be passed into the crawler. We would also have to figure out how to structure the output properly to handle this scenario, as the crawler is expected to return axe results.
This is the last open issue on the ADO-improvements-1 feature. I talked to Rob and he said he was okay with either closing this issue as-is if everyone agrees. Otherwise the plan will be to push it to a later feature.

Closed after discussion with the managers.


fix(sdg): fix multiindex misalignment

Fix for https://github.com/owid/etl/issues/437

@larsyencken sorry, did it in a rush. Post-mortem description

Something that would have been good in the description is any change to the actual downstream data after this.

There's no downstream data... yet.

So I can't say I fully understand what's going on.

Might have been cached files or something like that. No idea. I haven't investigated since as I mentioned, it has no downstream data.


Exists command to Edge always returning 0?

Exists command to Edge always returning 0?

Fixed

This issue was tagged on the latest release but exists command still returns only zero.

I'm experimenting with the same issue.
Some debugging information:

v0.1.11: works fine
v0.1.21: broken (it always returns 0)
v0.1.22: broken (it always returns 0)

we are working on it


tegra210_xusb_firmware needs to be available during early boot

As mentioned in #12, the xusb firmware needs to be available during early kernel initialization, particularly if the rootfs is on a USB device.
Possible approaches to resolve this:

Build the firmware into the kernel image via CONFIG_EXTRA_FIRMWARE.  This might not be great from a licensing perspective.
For non-USB boots, use systemd-udevd instead of eudev.  systemd-udevd includes the firmware loading helper, which was removed from eudev.  Patches for eudev to restore that functionality could also be added.
Use an initrd that at least has the needed firmware.  This is the approach taken by L4T.

Using a minimal initrd with the firmware in it solves the USB problem, but appears to cause problems with later firmware loads (after the real rootfs has been mounted).  So I suspect we'd have to go all-in with a more-complete initrd for that approach.  Otherwise, one still needs a udev firmware helper.
For now, I'll add in eudev patches to make the firmware helper work, then revisit after some further investigation.

@madisongh I did a clean build last night and tried this out this morning and eth0 comes up fine. Thanks for the patch.

@madisongh noticing today that I need to restart networking after boot in order to get eth0 up
/etc/init.d/networking restart

Any errors in the boot log about it?   Race between the USB coming up and the networking initscript, perhaps?

I see the firmware get loaded
root@jetson-tx1:~# dmesg | grep firmware
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    7.691201] udevd[177]: writing '/lib/firmware/tegra21x_xusb_firmware' (124416) to '/sys/devices/platform/tegra-xhci/firmware/tegra21x_xusb_firmware/data'
[    7.712337] tegra-xhci tegra-xhci: Firmware File: tegra21x_xusb_firmware (124416 Bytes)

It looks to me like the networking initscript is being called too early?
 6.352108] ipip: IPv4 over IPv4 tunneling driver
[    6.357232] ip_tables: (C) 2000-2006 Netfilter Core Team
[    6.362706] arp_tables: (C) 2002 David S. Miller
[    6.367352] TCP: cubic registered
[    6.370663] Initializing XFRM netlink socket
[    6.375075] NET: Registered protocol family 10
[    6.380359] mip6: Mobile IPv6
[    6.383377] ip6_tables: (C) 2000-2006 Netfilter Core Team
...

Maybe massage the order of the init scripts?

As another clue it seems that on a fresh install eth0 always comes up. But, upon a reboot eth0 is missing. I'm trying to think of persistent elements that would cause this behavior. I tried disabling the udev-cache but that didn't do the trick.

Hmm.  I haven't seen this on reboots in my testing with the sato image, but there could be a race where eth0 isn't showing up in time for the networking initscript to see it.  Which image are you running?

it's my own based on core-image-minimal with a bit more debugging/network stuff in there so I bet its some issue with something delaying things in an out of order way. When I get a chance I'll try a more standard image.

I have seen this issue when it takes a long time for the USB controller to probe all of the attached USB peripherals - in my case, this was a custom board with a custom USB device attached that the controller could see but could not complete all of the protocol handshaking.  I expect that using some udev rules to trigger bringing the interface up when it becomes visible would help in this situation.

Having the initrd in place to load the xusb controller's firmware during early boot (which is also done in the stock L4T Ubuntu image) takes care of this for typical use cases.


Environment Build Changes

Bit of tidying up - changes to req.txt, setup.py and the creation of what hopefully becomes a one line install script for people who want to be on the bleeding edge.
Also contains activate.ps1 for people using python venvs within vs code - means all you gotta do is type activate.ps1 into the console and it activates the venv.

I don't think we should be using sudo with pip for this installer. It might be good to create a virtual environment with a symlink to a script that "activates" the venv and launches LedFx.
For the install:
python3 -m venv ~/ledfx-venv
export PATH=$PATH:$HOME/ledfx-venv/bin
...install ledfx here

Script to launch LedFx:
#!/usr/bin/env bash
export PATH=$PATH:$HOME/ledfx-venv/bin
ledfx

Or, install in the user environment:
...
pip install --user -r requirements.txt
python3 ~/ledfx-workdir/LedFx/setup.py build
python3 ~/ledfx-workdir/LedFx/setup.py install

Ready for merge - installs for the user who calls install.sh.
Install instructions are simply;
curl -sSL https://install.ledfx.app | bash

This also changes the default bind address to <IP_ADDRESS> - fixes docker issues and fixes issues on Raspberry Pi's (if binding to <IP_ADDRESS> it won't listen on external adaptors)

I just did a fresh install on a pi zero and needed to change the following to work:
sudo apt-get install -y gcc \
        git \
        libatlas3-base \
        libavformat58 \
        portaudio19-dev \
        pulseaudio \
        python3-pip

It would be great to get a few more tests on this install script


Phragmén Validator Election

Replaces the naive validator selection in the staking srml, as explained in #1772.

The old approach was: Validators are chosen merely by sorting their staked values. If anyone nominated, then it would have been automatically assigned to its first vote. This PR entirely removes this logic. The new algorithm requires nominators, otherwise, previous generation validators are chosen naively.

TODOs:

[x] Complete test cases: leftovers of https://github.com/paritytech/substrate/pull/1782 + a few ones dedicated to the new logic. Special attention should be paid to have test-cases that demonstrate the aforementioned goals of the module.

[x] (OPTIONAL) Improve readability and castings between BalanceOf<T> and other numeric types if possible.

Come up to a conclusion about some edge cases @gavofyork  @shawntabrizi :

[x] Should a validator candidate with no vote be considered?

Current status: NO. validators are filtered by having approval_stake >= 0 before the first round of the election, where approval_stake is the sum of stakes coming from votes.
Is a vote with stake == 0 completely null and useless? if yes, then the above point is correct. If no, then the above point is already incorrect since we are currently treating a validator with no votes and a validator with tons of zero-votes the same => both are filtered out. Correct.

[x] What's the expected behavior when the number of validator candidates (after being filtered by approval_stake) is less than ValidatorCount<T>?

Current status: All of them are elected without any computation. Correct.

[x] The same condition as above + the number of possible candidates is less than ValidatorCount<T> and MinimumValidatorCount<T>?

Current status: Nothing special. This should most likely change and some emergency condition be triggered (e.g. Emit an event to other modules and stop processing blocks?). Correct: Keep the previous set and do nothing further.

@bkchr I've pushed a stable code that addresses some but not all of your comments if you want to already have a look.

I've added one final test to check the internal scores of the algorithm and they seem to be in sync with the same config executed in python (e.g. this and this). I believe that based on the info available (reference + discussion) this PR is good to go.

@bkchr Thanks for the new review. All fixed here.
https://github.com/paritytech/substrate/pull/1951


Graphs showing old data greyed out

Showing the data from previous lap in the graph for tyre pressures but faded out.
Could be used as a reference to see if something is working as intended or not.

no 😂
Tried and it's a mess in the graph xD


Fixed the virtual machine hostname that causes the virtual machine to…

@Todo after merge:

Create a new runtime

#123

We should not crop the vm item_hash. If the source of this bug cannot be resolved, we should then use another representation of the item_hash such as it's base32 encoding.

Tested with this code =>
#!/bin/bash

python3.9 -m http.server 8080


Add support for filtering stack traces in the log4j2 implementation

This is similar to what %xEx{filters(...)} does.

:grey_exclamation: Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-07T13:49:00.144+0000

Duration: 8 min 20 sec

:robot: GitHub comments
To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@felixbarny / @eyalkoren Since version 1.29.0 of the Java agent was released recently, could you please spare some time to provide feedback on this PR?

ping

What are some of the use cases for wanting to filter out a stack frame?
Using regex seems like it can be quite heavy on performance. Would matching via String#contains be good enough? If not, maybe something like the WildcardMatcher from the Java agent that uses String#regionMatches?

My use case is to filter out (most of) the framework stack frames, which can be very noisy on an application server and usually do not add valuable information.
e.g. I would like to compress
com.example.SomeException: Some Message
	at com.example.Bar.foo(Bar.java:321)
	at com.example.Foo.bar(Foo.java:123)
	at com.example.Foo$Proxy$_$$_WeldClientProxy.bar(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:151)
	at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
	at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:338)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
	at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
	at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1551)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1551)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1551)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1551)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
	at java.lang.Thread.run(Thread.java:750)

into something like
com.example.SomeException: Some Message
	at com.example.Bar.foo(Bar.java:321)
	at com.example.Foo.bar(Foo.java:123)
	... suppressed x lines
	at java.lang.Thread.run(Thread.java:750)

This should increase readability (especially when there are also nested exceptions) and reduce storage costs.
I am aware that in some (rare) case we might loose valuable stack frames, but overall I think its worth the downside.

I will have a look into String.contains

Oops, didn't see this PR out there.  I just submitted something similar except that it uses log4j2's PatternFormatter to render the Throwable and should support all of the patterns that log4j2 supports.  I'll test to ensure that it also works with filters.
https://github.com/elastic/ecs-logging-java/pull/177

@tobiasstadler please take a look at #177 - would you be able to use this approach for your needs?
Since your implementation is log4j2-specific as well, it seems like a good approach for utilizing inherent log4j2 stuff with some more options.
I didn't review both yet, but in general, same overhead that would be applied through using the same capabilities in log4j2 is acceptable and we need to see if and how stack-as-array is supported.
Please let me know what you think. Thanks!

@eyalkoren My solution uses contains while the log4j implementation uses startsWith. So #177 will only solve my problem partially.

@tobiasstadler

My solution uses contains while the log4j implementation uses startsWith. So https://github.com/elastic/ecs-logging-java/pull/177 will only solve my problem partially.

You could roll your own LogEventPatternConverter and have it filter out the frames in that manner and use that in the exception pattern.  It is more involved but it gives you total control over the format.  That's honestly how I'm likely to use that PR if it's merged as I am interested in getting close to the logstash ShortenedThrowableConverter and log4j2 doesn't offer most of those features and I think trying to build that into EcsJsonSerializer will be ... involved.
It's a bit of a shame that the throwable converters that ship with log4j2 aren't really designed to be extended, that would have made it pretty trivial to build a custom one that just tweaks the behavior of filtering out stack frames.

@eyalkoren My solution uses contains while the log4j implementation uses startsWith. So https://github.com/elastic/ecs-logging-java/pull/177 will only solve my problem partially.

Yes, I am aware. Yours is more performant. This is why I emphasized that in general, same overhead that would be applied through using the same capabilities in log4j2 is acceptable.

You could roll your own LogEventPatternConverter and have it filter out the frames in that manner and use that in the exception pattern. It is more involved but it gives you total control over the format. That's honestly how I'm likely to use that PR if it's merged

Could that be a solution that satisfies all? What if you take the more efficient approach from this PR and use it within a custom LogEventPatternConverter, which is added to the ecs-logging lib as well (with a custom pattern) and use the log4j-native configuration approach from #167 ?
Does this make sense?
@tobiasstadler just an example where providing the maximum flexibility is preferable- assume that we use the filtering approach from this PR only and it solves what it needs to perfectly. However, for another user the most important thing is to put a hard limit on the number of stack trace elements they want to get logged. The general-purpose approach allows that with the builtin converter even though we didn't think of that.

I just want to "exclude" stack frames from stack traces that match some list of patterns. I am fine with anything providing support for this.

@tobiasstadler could you port your desired implementation into a LogEventPatternConverter and find the best approach to load it on demand, as org.apache.logging.log4j.core.config.plugins.Plugin or otherwise, so it would work well with #177 ?

I'm not convinced that we should add custom implementations for LogEventPatternConverter in ecs-logging-java. Wouldn't it be enough to be able to use custom LogEventPatternConverter implementations that live in the application's code base?

Wouldn't it be enough to be able to use custom LogEventPatternConverter implementations that live in the application's code base?

It would, but if others can benefit from it, why not have it here? After all, it is very relevant to log manipulation that is probably common for ECS users. Are you worried about the maintenance burden? That indeed has overhead, but if we make it pluggable, it provides a nice encapsulation and I think it could be a nice example for other users of how it can be done.
What about placing it in a separate module dedicated for plugins?

I just think adding utilities is not in scope of this project and could be a slippery slope. I agree it's useful but not specific to ECS. Maybe upstream log4j2 would be a better home for that LogEventPatternConverter.

@tobiasstadler we decided to go with the generic approach of #177 because it is extensible very flexibly, as demonstrated through CustomExceptionPatternConverter and the corresponding EcsLayoutWithExceptionPatternTest.
With that, you can apply any custom formatting, like the one in this PR. Would that work well for your use case?

I will have a look


[OCPCLOUD-1570] Add initial end user documentation

In the interest of making the reviews easier, I'm splitting the work from this card into several PRs. This PR contains the first sections which contains an overview/summary of the CPMS, a quick look at some of the features, a start of a glossary and some discussion about how the rolling update and on delete strategies work.
I have a list of other things to include in follow up PRs which I'll push once this stuff is reviewed:

[ ] More detail on failure domains

[ ] How do the failure domains relate to cloud provider concepts
[ ] How to configure failure domains on each provider

[ ] How does this fit in with the rest of Machine API

[ ] Differences between CPMS and regular MachineSets
[ ] Why do we have failure domains
[ ] Why do we have rolling update strategies

[ ] Troubleshooting guide

[ ] Examples of when the operator will go degraded, and why
[ ] Corner cases

[ ] FAQs

[ ] Connection to Cluster API
[ ] How are providers integrated

/lgtm

/approve
Spoke with the other reviewers, ok to merge, will follow up if there's any additional feedback


178-dev-derivation-agent-python-example

@markushofmeister Can you copy/adapt this example for your next agent and see if there's anything unclear/missing?

When I try to run the dockerised tests, I get the following error message suggesting that there is a path issue for the Blazegraph password.
ERROR: for blazegraph  Cannot create container for service blazegraph: invalid mount config for type "bind": bind source path does not exist: /tests/dummy_services_secrets/blazegraph_passwd.txt

I was developing/running everything in WSL2 and it was working fine. However, I can confirm that I ran into the same issue when running in Windows. It seems when running in Windows, the credential files were added executable permission when copying .tests folder into the docker container. However, I have tried to remove the permission manually but it's still not working... As the recommended way of developing docker containers on Windows, according to The World Avatar wiki, is through WSL2, I have put up a big notice in the README and pointed developers to PropertyValueEstimationAgent as an example if one wants to develop in Windows.

When I try to run the dockerised tests, I get the following error message suggesting that there is a path issue for the Blazegraph password.
ERROR: for blazegraph  Cannot create container for service blazegraph: invalid mount config for type "bind": bind source path does not exist: /tests/dummy_services_secrets/blazegraph_passwd.txt

I was developing/running everything in WSL2 and it was working fine. However, I can confirm that I ran into the same issue when running in Windows. It seems when running in Windows, the credential files were added executable permission when copying .tests folder into the docker container. However, I have tried to remove the permission manually but it's still not working... As the recommended way of developing docker containers on Windows, according to The World Avatar wiki, is through WSL2, I have put up a big notice in the README and pointed developers to PropertyValueEstimationAgent as an example if one wants to develop in Windows.

I guess that's good enough for now and we can always improve later. Those platform dependent permission issues are sometimes tricky to solve.


add initial feedback pop up

This just adds the initial feedback back pop on large screens to help make it easier to provide feedback.

The feedback currently appears once 50% of the document has been scrolled.
A thumbs up will close the feedback and reappear after the revisibility timeout.
A thumbs down will lead to asking if they want to file an issue.

The content definitely needs to be reworded. Suggestions would be greatly appreciated.

[ ] Increase the timeout between revisibility from 5s to whatever is deemed appropriate.
[ ] Add the feedback forms to cloud/bartholomew.
[ ] Tweak the gh issue tempalte.

I cannot trigger this in a local clone. Any instructions on what I need to do?

It was changed to pop up after scrolling 50% of the page.

I cannot trigger this in a local clone. Any instructions on what I need to do?

a) I also cannot trigger this on certain pages in a local clone. For example Spin CLI Reference (no matter what I do; scroll up and down from top to bottom etc.)
b) On pages where I can get to trigger it works. However, on those pages the Table of Contents (ToC) hyperlinks cease to be hyperlinks (the show the headings of the page but clicking on the ToC headings does not take the user to the heading position)

The implementation is supposed to trigger only on spin pages and it is just a prototype at this point in time just to get some feedback on it. The overlap of the ToC is a known issue that will be fixed.

Incorporated some of the feed back

The ToC works as expected now
The feedback form pops up on every template now except the common.
There is exponential backoff between every successive "Still finding" prompt.
There is a close button that closes the feedback on the particular page and does not open it again until page reload.

Wow, that was quick. Let me go and take another look.

I may have missed something from initial ideation to implementation while being out :-) So bear over with me.
The initial need we've identified, which initiated this feature, was to get a quality indicator for docs. In order to make that frictionless for users, we wanted to focus on a single indicator, which was whether users felt served by the documentation, in terms of getting answers to a particular question they came to the docs to get answers to. I'd like if we can stick to that, and not make this a substitute for general feedback.
Stating the one question "Did you find answer to the question?" (Or a better english phrase) I'd like us to stick to. That'll help give us a ratio of respondents feeling served, to those who don't. For those who don't feel served by the docs, the follow-up question would be "What was the question?", in order to capture what we missed to either explain, or didn't make discoverable.
I feel that's a low stake ask on users, and it's hopefully good at capturing the readers mindset as requirements to improvements.


react-scripts start Create React App requires dependency

Is this a bug report?
Yes
Did you try recovering your dependencies?
Yes
Environment
Environment Info:

  System:
    OS: macOS 10.14.2
    CPU: x64 Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  Binaries:
    Node: 9.11.2 - ~/.nvm/versions/node/v9.11.2/bin/node
    npm: 5.6.0 - ~/.nvm/versions/node/v9.11.2/bin/npm
  Browsers:
    Chrome: 72.0.3626.121
    Safari: 12.0.2
  npmPackages:
    react: ^16.8.4 => 16.8.4
    react-dom: ^16.8.4 => 16.8.4
    react-scripts: 2.1.8 => 2.1.8
  npmGlobalPackages:
    create-react-app: Not Found

Steps to Reproduce

Ran npx create-react-app examples --typescript
Ran npm run start
Got an error.

Expected Behavior
This should work without me having to do anything off of a fresh install.
Actual Behavior
Got the error:
><EMAIL_ADDRESS>start /Users/ryanwilson/work/athlinks-component-library/examples
> react-scripts start

There might be a problem with the project dependency tree.
It is likely not a bug in Create React App, but something you need to fix locally.

The react-scripts package provided by Create React App requires a dependency:

  "jest": "23.6.0"

Don't try to install it manually: your package manager does it automatically.
However, a different version of jest was detected higher up in the tree:

  /Users/ryanwilson/work/athlinks-component-library/node_modules/jest (version: 23.4.2)

Manually installing incompatible versions is known to cause hard-to-debug issues.

If you would prefer to ignore this check, add SKIP_PREFLIGHT_CHECK=true to an .env file in your project.
That will permanently disable this message but you might encounter other issues.

To fix the dependency tree, try following the steps below in the exact order:

  1. Delete package-lock.json (not package.json!) and/or yarn.lock in your project folder.
  2. Delete node_modules in your project folder.
  3. Remove "jest" from dependencies and/or devDependencies in the package.json file in your project folder.
  4. Run npm install or yarn, depending on the package manager you use.

In most cases, this should be enough to fix the problem.
If this has not helped, there are a few other things you can try:

  5. If you used npm, install yarn (http://yarnpkg.com/) and repeat the above steps with it instead.
     This may help because npm has known issues with package hoisting which may get resolved in future versions.

  6. Check if /Users/ryanwilson/work/athlinks-component-library/node_modules/jest is outside your project directory.
     For example, you might have accidentally installed something in your home folder.

  7. Try running npm ls jest in your project folder.
     This will tell you which other package (apart from the expected react-scripts) installed jest.

If nothing else helps, add SKIP_PREFLIGHT_CHECK=true to an .env file in your project.
That would permanently disable this preflight check in case you want to proceed anyway.

P.S. We know this message is long but please read the steps above :-) We hope you find them helpful!

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>start: `react-scripts start`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!      /Users/ryanwilson/.npm/_logs/2019-03-18T20_56_51_898Z-debug.log

Extra Notes:
I ran this inside of an npm project already because I need to create a react project bundled with the project so that people can easily test out the components written in the main project. I can't have the react web app stuff in my main project because it needs to remain as small as possible and only include the components when pulled in by npm i. Somehow this is finding the jest in an entire directory up from the project, it should only look in the local directory. When running npm ls jest I get:
<EMAIL_ADDRESS>/Users/ryanwilson/work/component-library/examples
└─┬<EMAIL_ADDRESS>  └──<EMAIL_ADDRESS>
It only find the correct one so I'm not sure why it is failing to start.
I've also used many libraries where they have a second project inside their library to let you test out the library very easily by just cloning it. Doing it this way allows the npm library to remain small while also avoiding having to maintain two separate projects for a single library. Also really helps with developing.
Going to have to use SKIP_PREFLIGHT_CHECK=true to bypass this error for now, but it would be nice if this was officially supported out of the box.
Please correct me if I'm wrong but doesn't npm by default only checks the local and global node_modules so there should be no way it can actually get one directory up to try to use those node_modules. I've done this same thing about a year ago and haven't had any problems with node_modules stepping on each other but I could be wrong about that.

Please correct me if I'm wrong but doesn't npm by default only checks the local and global node_modules so there should be no way it can actually get one directory up to try to use those node_modules. I've done this same thing about a year ago and haven't had any problems with node_modules stepping on each other but I could be wrong about that.

Node does recursively look through parent node_modules. I think that the error message explains the "why" pretty well, and SKIP_PREFLIGHT_CHECK=true is your way around it. I'm closing this as a duplicate of #6627!


fix: paginator generator nullability bug caused by documents

Issue #
The paginator generator is not code-generating nullable literals for item paginators of lists that have values that are nullable. It will only code-generate nullability if the list is marked as sparse and not check if the list has values that are nullable.
Lists are not supposed to contain nullable items unless marked as sparse, so this is non-modeled behavior that we were not looking for but is necessary because documents are always nullable in smithy kotlin.
Description of changes
The paginator generator now checks if the collection is made of nullable symbols and code generates nullability
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Yeah, this was my mistake I marked as ready for review before adding the tests!


Train on more than a single image

Hi,
I was wondering if more than one image could be used when generating the training set for LodeSTAR.
Thank you very much!
L

I think this might be what you're looking for ?

That's right! We recommend using a few crops for training where possible, though there are diminishing returns the more you add.

Yes! In that case, I would call training_images rather than training_set in
generator = dt.models.lodestar.LodeSTARGenerator(     training_set,     3,      (group, equivariance),      batch_size=8,      min_data_size=200,      max_data_size=201 )
?
Thanks a lot!

An example would be
training_images = [crop1, crop2, crop3, ...]
random_sampler = dt.Value(lambda: random.choice(training_images))
generator = dt.models.lodestar.LodeSTARGenerator(random_sampler, 3, ...)


add git URL format to json + developers docs

copying the content from faq.md to json.md and developers.md

landed on master, thanks!


Django test handling

Since Django tests start a new transaction and never commit it, any on_commit hooks will never get a chance to run in test mode. We've worked around this by writing a test utility that simply calls all registered on_commit hooks, but sometimes this isn't enough.
What do you think about the idea of running on_commit hooks when a SAVEPOINT is committed, but only if we are in test mode? I know there are probably some false positives, but it would simplify testing quite a bit for us.
I'd love to discuss this idea more.

Thanks for the thoughtful report.
I'm pretty hesitant to introduce any changes to the production code that are test-aware. In general, Django doesn't even really have a concept of "test mode" that you can hook into to check whether tests are currently being run, and I think that's a good thing -- the tests should be responsible for setting up the conditions for their success, it shouldn't be the responsibility of production code to be testing-aware.
And specifically the idea of running hooks on savepoint commit is unsafe, I think. Some savepoints might be "would have been a transaction commit if we weren't within a test transaction," but not all. Some savepoints would always have been savepoints, because they're nested in another transaction even under production usage. In these cases your proposal would result in a significant change in code semantics when under test, which is not good.
The documentation already discusses this issue, and suggests using TransactionTestCase if you need to test the results of an on_commit hook. Since you need to use TransactionTestCase to test any other transaction behavior in Django, and on_commit is "transaction behavior," this seemed like a reasonable limitation to me, and so far in my projects it's been adequate.
If that's not adequate, I'd be inclined to go the same direction you did, and write a utility you can call from a test which runs any pending on_commit hooks; this shouldn't be too difficult to write.
Since transaction-hooks has been merged into Django 1.9, I don't really want to add any more features to it, but we could see how the other Django core devs feel about adding a utility like that to Django 1.10. I don't have strong feelings either way about it.

Oh, I didn't realize it was in 1.9. I'll keep an eye on that and see if there's anything we can do to alleviate some of our testing pain.

Ok. Closing this issue for now, then.


 This situation can be simplified by converting Criterion into a builder struct, this lets us combine a few methods that generate all the previous outcomes. So far I got something like this working:
Criterion::default()
    .measurement_time(Duration::from_secs(3))
    .inputs(vec![1024, 32 * 1024, 1024 * 1024])
    .function(Function("alloc", |b, i| {}))
    .bench()
/* Output
Benchmarking alloc with input 1024
Benchmarking alloc with input 32768
Benchmarking alloc with input 1048576
*/
    .inputs(vec![7, 11, 13])
    .functions(vec![Function("par_fib", |b, i| {}), Function("seq_fib", |b, _| {})])
    .bench()
/* Output
Benchmarking par_fib with input 7
Benchmarking par_fib with input 11
Benchmarking par_fib with input 13
Benchmarking seq_fib with input 7
Benchmarking seq_fib with input 11
Benchmarking seq_fib with input 13
*/
    .function(Function("no_input", |b, &()| {}))
    //^ FIXME second argument of the closure is unnecessary
    .bench()
/* Output
Benchmarking no_input
*/
    .functions(vec![Function("A", |b, &()| {}), Function("B", |b, &()| {})])
    .bench();
/* Output
Benchmarking A
Benchmarking B
*/

A minimal PoC has been implemented in the criterion-builder branch. The PoC contains just the builder, no benchmarking is actually perform when you call bench.
I wonder if we can do something similar to the Bencher struct which has also grown quite a few methods for the different timing loops.
cc @faern

This has (more or less) been implemented now with the API improvements in 0.2.0, so I'm going to close this.


[STRAPI-based Page]: AI-generated summaries Documentation

Issue Name
AI-generated summaries
Issue Description
AI-generated summaries can help facilitate browsing of records which have exhaustive descriptions. We have generated such summaries for ClinEpiDB and are investigating different ways for displaying the summaries for a record. To support the display we have drafted documentation for the knowledge center to describe the purpose and value of the AI-generated summaries
Related issues:

https://github.com/NIAID-Data-Ecosystem/nde-portal/issues/251
https://github.com/NIAID-Data-Ecosystem/nde-crawlers/issues/159
https://github.com/NIAID-Data-Ecosystem/nde-portal/issues/252

Issue Discussion
This issue has been discussed at the biweekly meeting dated 2024.10.15
Mockup URL
https://docs.google.com/document/d/1_H6ema9JDMvwEeTWiCCAh-P5k-ak2qajIYlA-wohuAM/edit?tab=t.0
Related WBS task
https://github.com/NIAID-Data-Ecosystem/nde-roadmap/issues/45
For internal use only. Assignee, please select the status of this issue

[ ] Not yet started
[ ] In progress
[ ] Blocked
[ ] Will not address

Status Description
No response
STRAPI page status check list

[ ] This page/documentation/change has yet to be discussed between NIAID, Scripps, Leidos
[ ] This page/documentation/change has been discussed/reported between NIAID, Scripps, Leidos
[ ] This page/documentation/change has been mocked up
[ ] This page/documentation/change has been implemented on Staging
[ ] This page/documentation/change has been approved for Production
[ ] This page/documentation/change has been implemented on Production

This documentation is now on Staging.
The Ask: We seek approval to move it to production (Pending the approval and integration of AI-generated summaries)

Hi @gtsueng and @EmilyHaag - I have updated this status to 'ongoing discussion' to provide time to assess and respond to feedback from ClinEpiDB.


Doris deploy problem and create table filed issue.

hi all,
I deployed a doris cluster with one fe and three be, version 0.12.
fe and be 10002 deployed on one server,and be 10003,10004 deploy on the other two servers;
show backends can't get the disk infomations from 10003,10004.

I found the follow warning in log/be.WARING.log.xxxx, and have no idea

W0313 13:48:47.399816  4857 task_worker_pool.cpp:1186] finish report olap table state failed. status:-1, master host:<IP_ADDRESS>, port:9020
W0313 13:48:48.404908  4855 utils.cpp:101] fail to get master client from cache. host=<IP_ADDRESS>, port=9020, code=7
W0313 13:48:48.404987  4855 task_worker_pool.cpp:1061] finish report task failed. status:-1, master host:<IP_ADDRESS>port:9020
W0313 13:48:58.405503  4855 utils.cpp:101] fail to get master client from cache. host=<IP_ADDRESS>, port=9020, code=7
W0313 13:48:58.405591  4855 task_worker_pool.cpp:1061] finish report task failed. status:-1, master host:<IP_ADDRESS>port:9020

And when i tried to create table , wrong again

CREATE TABLE example_db.table_hash     (     k1 TINYINT,     k2 DECIMAL(10, 2) DEFAULT "10.5",     v1 CHAR(10) REPLACE,     v2 INT SUM     )     ENGINE=olap     AGGREGATE KEY(k1, k2)     COMMENT "my first doris table"     DISTRIBUTED BY HASH(k1) BUCKETS 32     PROPERTIES ("storage_type"="column");

ERROR 1064 (HY000): errCode = 2, detailMessage = Failed to create partition[table_hash]. Timeout. Unfinished mark: 10003=10178, 10003=10242, 10003=10182

I found the follow warning in fe/log/fe.warn.log

2021-03-13 13:46:19,985 WARN 91 [ReportHandler.deleteFromBackend():664] delete tablet[10234 -<PHONE_NUMBER>] from backend[10002] because not found in meta
2021-03-13 13:46:19,985 WARN 91 [ReportHandler.deleteFromBackend():651] failed add to meta. tablet[10298], backend[10002]. errCode = 2, detailMessage = db[-1] does not exist
2021-03-13 13:46:19,985 WARN 91 [ReportHandler.deleteFromBackend():664] delete tablet[10298 -<PHONE_NUMBER>] from backend[10002] because not found in meta
2021-03-13 13:46:19,985 WARN 91 [ReportHandler.deleteFromBackend():651] failed add to meta. tablet[10238], backend[10002]. errCode = 2, detailMessage = db[-1] does not exist
2021-03-13 13:46:19,985 WARN 91 [ReportHandler.deleteFromBackend():664] delete tablet[10238 -<PHONE_NUMBER>] from backend[10002] because not found in meta2021-03-13 13:46:19,985 WARN 91 [ReportHandler.deleteFromBackend():651] failed add to meta. tablet[10302], backend[10002]. errCode = 2, detailMessage = db[-1] does not exist
2021-03-13 13:46:19,985 WARN 91 [ReportHandler.deleteFromBackend():664] delete tablet[10302 -<PHONE_NUMBER>] from backend[10002] because not found in meta

This problem has been solved.
The problem caused by network block.
We need ensure that be can access fe 9030.
some suggestion:

check the firewall status, or reset the firewall with "sudo iptables -F"
check the Listening port with "netstat -lnpt"
if you are using cloud server, check the server secure setting.


Mailbox Field Bug

I'm submitting a Bug report.

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  <!-- Please search GitHub for a similar issue or PR before submitting -->
[ ] Feature request
[ ] Documentation issue or request
[ ] Other: <!-- Please describe: -->

Current behavior

Live Server automatically changes the page structure when it encounters a mailbox field.

Expected behavior

Rendering the page correctly.

Environment
Browser:

- [x] Chrome (desktop) version 66.0.3359.181 (Official Build) (64-bit)
- [ ] Firefox version XX
- [ ] Safari (desktop) version XX
- [ ] IE version XX
- [ ] Edge version XX

For Tooling issues:

- Live Server: 5.1.1 <!-- Check which version is installed -->
- Platform:  Mac <!-- Mac, Linux, Windows -->
- Visual Studio Code: 1.24.1<!-- `code --version` -->

Others
If you want to reproduce the bug shown above, here is my blog project link.

I don't think it is live server's issue. This is the issue of any extension that you've installed in chrome.

Workaround:
Open the server in Guest or incognito window .

Anyway, I've tried on my laptop. So issue so far.

It works correctly after I closed all my Chrome extensions. Now I'm trying to find out which one does the matter.
Thanks for your reply! Live Server is awesome! I would rate it five stars!


dataexplorer: pin jdk17

Description of changes
Since the default jdk changed to jdk19, DataExplorer fails at startup:
$ ./result/bin/DataExplorer
java.lang.NullPointerException: Cannot read field "width" because the return value of "gde.config.Settings.getWindow()" is null
	at gde.ui.DataExplorer.execute(DataExplorer.java:406)
	at gde.GDE.main(GDE.java:544)

Therefore, pin to jdk17, which is also the latest recommended version by upstream:

The only requirement is the installation of a Java run-time of version
8 to Java 17 for the 32 bit versions and Java 11 to 17 for the 64 bit
versions, such as IcedTea or open JDK.

https://www.nongnu.org/dataexplorer/index.html#Introduction
Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 208646 run on x86_64-linux 1

1 package built:

dataexplorer

The program runs and works fine with my battery charger.


Training

Excellent Work! Can you tell me how much computing resources are spent on training the model you are using now?

Hi! So far we mostly use off-the-shelf modules which do not require further training. You can refer to the ControlNet and IP-Adapter paper to get an idea of the training costs.


vopd v_dual_and_b32 first argument problem

v_dual_add_f32 v17, v22, v26 :: v_dual_and_b32 v16, v21, v25 - ok
v_dual_and_b32 v17, v22, v26 :: v_dual_and_b32 v16, v21, v25 - error: invalid instruction

Unfortunately the set of X opcodes is not the same as the set of Y opcodes.  The assembler is correct to complain.


Error: Starting Kyuubi by client yaooqinn.kyuubi.service.ServiceException: Unable to create Kyuubi namespace /kyuubiserver on ZooKeeper

Describe the bug
when i set the keytab and principal in the spark-defaults-conf.xml and start the kyuubi, the error happened.So i discard the  kerberos auth and delete the keytab and pricipal in the spark-defaults-conf.xml and start the kyuubi server, the message in log display when i make the zk.ha true, kyuubi will use kerberos to auth and the error is there is no spark.yarn.keytab.As a  result,how do i connect to zeekeeper without kerberos? My zookeeper is no auth.
To Reproduce
Steps to reproduce the behavior:

Configurations
frist i export environment:
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_HOME=/opt/spark/spark-2.1.3-bin-hadoop2.6
second i editor spark-deaults.conf file to add these:
spark.yarn.principal<EMAIL_ADDRESS>spark.yarn.keytab ****.keytab
Environments
spark:spark-2.3-bin-hadoop2.7
linux:centos7
java:1.8
kyuubi:master
hadoop: CDH-5.13.3
zookeeper:3.4.5+cdh5.13.3
Operations
/opt/spark/kyuubi-0.7.0-bin-spark-2.1.3/bin/start-kyuubi.sh
--master yarn
--conf spark.kyuubi.ha.enabled=true
--conf spark.kyuubi.ha.zk.quorum=<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>
--conf spark.kyuubi.ha.zk.client.port=2181
--conf spark.kyuubi.authentication=KERBEROS
--conf spark.driver.memory=20g
--conf spark.hadoop.fs.hdfs.impl.disable.cache=true
--conf spark.executor.heartbeatInterval=30s
See error
19/07/25 14:56:02 INFO state.ConnectionStateManager: State change: CONNECTED
19/07/25 14:56:02 ERROR client.ZooKeeperSaslClient: An error:
(java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in
Kerberos database (7) - UNKNOWN_SERVER)]) occurred when evaluating Zookeeper Quorum
Member's received SASL token. This may be caused by Java's being unable to resolve the
Zookeeper Quorum Member's hostname correctly. You may want to try to adding '-
Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS environment.
Zookeeper Client will go to AUTH_FAILED state.
19/07/25 14:56:02 ERROR zookeeper.ClientCnxn: SASL authentication with Zookeeper Quorum
member failed: javax.security.sasl.SaslException: An error:
(java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in
Kerberos database (7) - UNKNOWN_SERVER)]) occurred when evaluating Zookeeper Quorum
Member's received SASL token. This may be caused by Java's being unable to resolve the
Zookeeper Quorum Member's hostname correctly. You may want to try to adding '-
Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS environment.
Zookeeper Client will go to AUTH_FAILED state.
19/07/25 14:56:02 ERROR curator.ConnectionState: Authentication failed
Error: Starting Kyuubi by client yaooqinn.kyuubi.service.ServiceException: Unable to create
Kyuubi namespace /kyuubiserver on ZooKeeper
Run with --help for usage help or --verbose for debug output
19/07/25 14:56:02 INFO server.KyuubiServer: Shutting down KyuubiServer
19/07/25 14:56:02 INFO util.ShutdownHookManager: Shutdown hook called
Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

This might not be problem of kyuubi or zk. It is the keytab that you use may have issues. try google or may see this https://stackoverflow.com/questions/13850252/cannot-get-kerberos-service-ticket-krbexception-server-not-found-in-kerberos-d/15987751

Thank you for you great answer, there is a little doubt about it, i try to make the ha is false and server can login successfully with the keytab i supply.Only when i make the ha is true,the error happened and i also try to make the auth is none and ha is true, there is also the same error.So i want to make sure is really the problem of keytab?

This might not be problem of kyuubi or zk. It is the keytab that you use may have issues. try google or may see this https://stackoverflow.com/questions/13850252/cannot-get-kerberos-service-ticket-krbexception-server-not-found-in-kerberos-d/15987751

In addition, my zookeeper is not configured kerberos auth.I wonder if that's going to be a problem.

we are not reaching zk yet, may be only kdc part. I am not sure whether a non-kerberized zk will reject anyone who has a kerberos ticket or not. Currently I guess the problem is that the keytab you specified is not able to auth via kdc. Of course, as a workaround, skipping this part might work.

OK，thank you very much!

The problem has been solved.The problem is the keytab that kyuubi server use to connect zk. It seems that the kyuubi server will use kerberos auth to connect zk.And the zkCli's default keytab is something like zookeeper/******.When i configure the zkCli and add the correct keytab in kdc,the problem is solved.


Controlling widget labels

Table widgets have no label, okay I'll put my tables in their own groups and use the group labels. For consistency I'll do this for all my output widgets...
So now for widgets that have labels I have the Group label and the widget label. I don't need both, so in the widget appearance tab I choose label option 'Hide' ... wtf?  The widget label in the editor disappears white the label text remains on the dashboard!  This is the opposite of what I expect and want and does not seem very useful to me.
I can remove the label text from the widget... this removes the text from the display - but blank space for the label text remains, and in the editor I now have an 'undefined chart' Ugly.
In comparison, when editing ui-group nodes, the 'Display Group Name' option does control if the group name appears on the dashboard or not
I think we should be able to control if a widget label appears on the dashboard. I think it should always appear in the editor.

in the widget appearance tab I choose label option 'Hide'

That is, I  think, an error in node-red.  That should say Name, not Label.  It applies to all nodes. The Appearance tab is for how the node appears in the editor.


Play a .ts file using mux.js

We've been trying to play a .ts file with mux.js and it seems dot working.
Although if we create a manifest file (m3u8) which direct to a ts file Shaka manages to play it.
We use  at the top of the html page.
and init the player with muxjs: window.muxjs = muxjs;
I guess it's kind of a flag to use mux.js when getting HLS and can be used also when getiing a .ts file.

So is the content playing or not?  You said "not working", but also "manages to play it".  What is the problem here?  If there is an error, what is the error?  What browser are you using?  Can you provide a URL link to the manifest? If the browser supports native TS content (e.g. Safari or Edge), then we'll play it natively instead of repackaging it.

@joeyparrish Yes, exactly.

We don't use mux.js when playing src= (direct playback of single file).  This is because we just give the URL to the browser and tell it "play this".  Since we have no control over the network or the playback, we can't transform it as we play.  We also may not be able to download the file and transform it all at once since it may be massive and we can't fit it in memory.  So we only support playing src= when the browser supports playing it directly.

@talmelamed, does this answer your question?  Sorry we aren't able to help with this.


Update to rayon 0.8

Let's move to rayon 0.8 before we release a new version.

Hibitset needs rayon 0.8 then, too.
Crossbeam could be bumped to 0.3.

Seriously?


Adding classifier tutorial

Showcases examples of how to run unifrac KNN and RandomForests with phylo transform

Could skbio.FeatureTable.read, and the table API (rather than pandas) be used?On Jun 24, 2024, at 11:37, Jamie Morton @.***> wrote:Showcases examples of how to run unifrac KNN and RandomForests with phylo transform
You can view, comment on, or merge this pull request online at:
  https://github.com/scikit-bio/scikit-bio-tutorials/pull/7
Commit Summary
1553932  adding classifier tutorial
File Changes  (1 file)
M
07-outcome-prediction/classification.ipynb
(217)

Patch Links:
https://github.com/scikit-bio/scikit-bio-tutorials/pull/7.patch
https://github.com/scikit-bio/scikit-bio-tutorials/pull/7.diff
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

Yes? I think we should

See my PR #8  on how to read data files using skbio.Table and skbio.SampleMetadata.

Let's merge for now. @mortonjt and I will expand later.


Mujoco 2.0.0 with dm_control

Hello,
I am working with another repository that needs Mujoco version 2.0.0 installed for it to work (as well as dm_control). I downloaded Mujoco 2.0.0 as well as the license from this website and put it in the .mujoco folder. I tested out mujoco out and it worked as expected.
However, when I install dm_control it automatically installs mujoco 2.2.0. How can I download Mujoco 2.0.0 with the dm_control library? I did install mujoco-py version <IP_ADDRESS> but for some reason, mujoco 2.0.0 isn't being detected.  Any help would be appreciated.

If you need MuJoCo 2.0.0, the easiest way is to pip install dm_control==0.0.364896371 since that's the last version on PyPI that supports that version of MuJoCo.
I can't help with mujoco-py I'm afraid, but the same comment goes there too: you need to figure out what the latest version of mujoco-py that supports MuJoCo 2.0.0.
Any reason why you have to use 2.0.0?

I got it to work. I didn't have to run any pip command to install mujoco. I just downloaded Mujoco 2.0.0 from the Roboti website and added a license key and it ended up working. Thanks for the help!


Port timedscopegen tool

// Copyright (c) Microsoft Corporation. All rights reserved.

please check if this is needed. we can leave it for another PR.

Refers to: src/CodeGenerators/TimedScopeGen/TimedScopeClassTemplate.cs:1 in a8e11fa. [](commit_id = a8e11fa08ac6d3eae9c351c6c1a9997bb1ae3b59, deletion_comment = False)


(573) SharePoint link

Changes
Add SharePoint link to Project
We know that users collate documents in a SharePoint folder as part of the prepare journey, and that users of this service will need to update/add to them.
Add a SharePoint link property to the Project model.
Fix typo of memoized instance variable
Restrict project routes
We don't use all seven default routes, only create the ones we do use.
Collect SharePoint link on Project creation
We want to collect the SharePoint link when the user creates a project.
Add some basic validation to check that the SharePoint link is a valid URL with the https scheme. Only run validation on create, so that previous projects do not become invalid.
Show SharePoint link on Project Details
Show SharePoint link on Project Summary
Checklist

[x] Attach this pull request to the appropriate card in Trello.
[x] Update the CHANGELOG.md if needed.

 Collect a link to the project SharePoint folder at handover

@mec This is ready for re-review when you are ready. Moving the call to sanitize into a helper silenced the Brakeman warning which is nice (don't know why it wasn't silenced when called directly in the view 🤷 )

@jkempster34 I've also just noticed that a proper share link from SharePoint has a different host to one that is copy and pasted out of the URL bar - I expect this will cause issues for users, but I think we find out?
url: https://educationgovuk-my.sharepoint.com/personal/meyric_rawlings_education_gov_uk/_layouts/15/onedrive.aspx?id=%2Fsites%2FServiceDeliveryDirectorate%2FShared Documents%2FA2C and Interventions%2FComplete conversions%2C transfers and changes&listurl=https%3A%2F%2Feducationgovuk.sharepoint.com%2Fsites%2FServiceDeliveryDirectorate%2FShared Documents&remoteItem={"mp"%3A{"webAbsoluteUrl"%3A"https%3A%2F%2Feducationgovuk-my.sharepoint.com%2Fpersonal%2Fmeyric_rawlings_education_gov_uk"%2C"listFullUrl"%3A"https%3A%2F%2Feducationgovuk-my.sharepoint.com%2Fpersonal%2Fmeyric_rawlings_education_gov_uk%2FDocuments"%2C"rootFolder"%3A"%2Fpersonal%2Fmeyric_rawlings_education_gov_uk%2FDocuments%2FComplete conversions%2C transfers and changes"}%2C"rsi"%3A{"listFullUrl"%3A"https%3A%2F%2Feducationgovuk.sharepoint.com%2Fsites%2FServiceDeliveryDirectorate%2FShared Documents"%2C"rootFolder"%3A"%2Fsites%2FServiceDeliveryDirectorate%2FShared Documents%2FA2C and Interventions%2FComplete conversions%2C transfers and changes"%2C"webAbsoluteUrl"%3A"https%3A%2F%2Feducationgovuk.sharepoint.com%2Fsites%2FServiceDeliveryDirectorate"}%7
proper share link: https://educationgovuk.sharepoint.com/:f:/s/ServiceDeliveryDirectorate/EltrV1yBDlxDh1MMDH35AuYBfDAygCzvTuvDy1B3a_vq9Q?e=NOGcIG


Keep empty lines when formatting nodes

This PR inspects empty lines between nodes to keep them when it makes sense.
This change applies to:

ArrayExpression
ObjectExpression
TSInterfaceBody
TSTypeLiteral
TSEnumDeclaration

Sweet! This is one of the things I was missing that made big objects like in descriptions.ts pretty janky.


Forecasting the future

Hi,
First of all thanks for putting this ARIMA summary online.
Second, I have a question. How would someone make predictions on the future (test set) if observed data are still unknown (NaN). Here is a sample code I tried https://github.com/kthouz/Misc/blob/master/arima_tuto.ipynb

I have the sales input data for 2 years say for 2014-2015 from month 04(April)-12(Dec) and need to forecast the value for the year 2016.
In this case,i have only small amount of data say less than 50 rows.
For this i am getting the error "ValueError: maxlag should be < nobs" ,so i browsed and found that i need to set the lesser maxlag value,but have no clue where to set it.
Already tried setting it in fit(maxlag=intValue,ic='bic') but it is not getting assigned with the new value and again the same error.
Where i need to and how i should set it actually?
Please help.
Looking for your response.
This is the code i tried:
def test_final_output():
df = pd.read_csv('../../data/salesoneproduct.csv', index_col=0)
nobs=len(df)
df.index.name=None
df.reset_index(inplace=True)
start = datetime.datetime.strptime("2014-04-01", "%Y-%m-%d")
date_list = [start + relativedelta(months=x) for x in range(0,48)]
df['index'] =date_list
df.set_index(['index'], inplace=True)
df.index.name=None
df.columns= ['riders']
df['riders'] = df.riders.apply(lambda x: int(x)*100)
mod = sm.tsa.statespace.SARIMAX(df.riders, trend='n', order=(0,1,0), seasonal_order=(1,1,1,12)
results = mod.fit(ic='bic',maxlag=10)
start = datetime.datetime.strptime("2019-04-01", "%Y-%m-%d")
date_list = [start + relativedelta(months=x) for x in range(0,12)]
future = pd.DataFrame(index=date_list, columns= df.columns)
df = pd.concat([df, future])
df['forecast'] = results.predict(start = 48, end = 59, dynamic= True)
df[['riders', 'forecast']].ix[-24:].plot(figsize=(12, 8))
print (df.forecast/100)
please provide a solution thanks in advance


for multicopter landings make sure that the copter moves

to the landing waypoint first before the descending phase starts

@LorenzMeier @AndreasAntener This makes sure that the copter first moves to the landing waypoint before it starts to descend. For the approach it will used the altitude specified in the landing waypoint.
I'm not very sure if this is the correct altitude. What is the meaning of the landing waypoint altitude?

From a user standpoint i would agree that the altitude in a land command would mean 'go here at this altitude and then land'. 0 altitude could be interpreted as 'keep current altitude'

@sanderux Thanks for your feedback! Makes sense, I can add the logic for the zero case.

@sanderux @tumbili We should be careful with special cases that are not clearly documented/specified. It could as well be that 0 is a valid relative landing WP altitude if the target landing point is below home. Because the user always specifies the altitude for a normal WP I would just keep that behaviour consistent as Roman described.

@AndreasAntener Yes, good argument.

Yes, true. QGC (and most GCS i think) maintain the altitude of the last WP when planning a mission so it should not be needed.

Tested and working. But before the yaw setpoint changed back to what it had at home for the landing. We can probably take this because we need to have a look at yawing during missions anyway.


Update annotated model

Related to https://binnenland.atlassian.net/browse/GN-5108
Updates the annotated template generation to align with the OSLO standard (linked to MOW v2 https://github.com/lblod/app-mow-registry/tree/version/2.0.0)
What's change

Update html decoration with the new model
Add Mocha test to simplify testing
Libraries update
Refactor to improve the readability

I agree. Thanks for the review! :pray:


Link not working

This might be client-side for me, or better yet just me being stupid cuz I'm not too familiar with this site, but when trying to use the link to the website app (http://dons20.github.io/sre), it says the page doesn't exist or rather, "There isn't a GitHub Pages site here." I ended up just installing the css zip file myself in which it worked fine then (looks great and performs stupendously btw).

Hi, thanks for bringing my attention to that, I could've sworn I updated the link.
Here is the new link (no longer using Github Pages): Steam Review Editor v1
Let me know if you have any issues accessing that one.

(looks great and performs stupendously btw)

Thank you very much, glad you like it!

Wow that was fast! Thanks for the quick fix!


Add Foxy support

PR description
This PR attempts to make domain_bridge a Foxy-compatible package - based off of the galactic branch.
The following changes have been introduced:

Re-add the GenericPublisher/GenericSubscription classes to this package's source tree. https://github.com/ros2/domain_bridge/pull/60/commits/0c05dab08241af7e4c2695a600e323b1eefd1166
Cherry-pick some of the compress/decompress mode logic before #24 was rebased on #30. https://github.com/ros2/domain_bridge/pull/60/commits/0c05dab08241af7e4c2695a600e323b1eefd1166
Re-add rosidl_target_interfaces in the CMakeLists.txt file. https://github.com/ros2/domain_bridge/pull/60/commits/0c05dab08241af7e4c2695a600e323b1eefd1166
Update the rmw_connextdds test dependency to read rmw_connext_cpp; this package is named differently in Foxy. https://github.com/ros2/domain_bridge/pull/60/commits/0c05dab08241af7e4c2695a600e323b1eefd1166
Update CI to target Foxy. https://github.com/ros2/domain_bridge/pull/60/commits/cce4273e91f729a98f799ab9aa973626fd8a8a73
Use explicit rclcpp::Duration(int64_t ns) constructor over rclcpp::Duration::from_nanoseconds. https://github.com/ros2/domain_bridge/pull/60/commits/9f3b66adcaf892c334330152bf3c174bc3c6d0a1
Obtain domain ID from rcl_node_options_t. https://github.com/ros2/domain_bridge/pull/60/commits/925a5e1798f21c53103e87d4f96f09adc1c9f940
Use rmw types for QoS options class. https://github.com/ros2/domain_bridge/pull/60/commits/699ab3a76559f75a6d8ee30bd40d6951fe32aa2a
Define a function to convert from rmw_time_t to rclcpp::Duration. https://github.com/ros2/domain_bridge/pull/60/commits/1562d57950f6bf67a15bc1c42734008783d89a19
Remove unused "unreachable logic" code. https://github.com/ros2/domain_bridge/pull/60/commits/cfb36cf38a2f9a5f26708323736430c81cc0cbdd
Update "Galactic" references to read "Foxy" as appropriate. https://github.com/ros2/domain_bridge/pull/60/commits/e614643ec8a3ff3ec4858504c21e1b4e080f64be
Replace set_domain_id functionality by doing the same through the rcl layer. https://github.com/ros2/domain_bridge/pull/60/commits/3272a4800db648d4351d287ce5ec85acf5e72701, https://github.com/ros2/domain_bridge/pull/60/commits/549654aaf6047bb28a474d7ffccdbebf0a9886d5, https://github.com/ros2/domain_bridge/pull/60/commits/eb58092084f524447d3c4424bf692ad0b75f1866

TODOs
I'll just list the things that are missing in Foxy for now:

Either backport the node option generating hooks added to rclcpp_components::ComponentManager in https://github.com/ros2/rclcpp/pull/1702 to Foxy - which is a difficult ask because this is most likely an ABI breaking change, or "revert" https://github.com/ros2/domain_bridge/pull/32, but that may have performance implications because (quoting @rebecca-butler) "This would allow the bridge and all other nodes to be run in the same process, ideally reducing latency."  My bad, I started from the main branch instead of galactic.
[x] rclcpp::*Policy enum classes and the respective getters.
These can be accessed through rmw/types.h but I suspect the PR these were introduced in (https://github.com/ros2/rclcpp/pull/1467) can be backported.

This is doable without backporting https://github.com/ros2/rclcpp/pull/1467 too, just kind of a slog.
https://github.com/ros2/domain_bridge/pull/60/commits/699ab3a76559f75a6d8ee30bd40d6951fe32aa2a.
https://github.com/ros2/domain_bridge/pull/60/commits/1562d57950f6bf67a15bc1c42734008783d89a19.

[x] Domain ID getters/setters.
Technically this functionality is exposed in rcl_node_options_t, so it's just a deficiency in the rclcpp API (in Foxy). In case rclcpp::InitOptions::set_domain_id cannot be backported to Foxy, the domain_id can be obtained through get_rcl_node_options (minus some mutex concerns..). Not sure how to set the domain_id though. Refer to https://github.com/ros2/rclcpp/issues/910 and the PRs that fell out of it.

[x] rclcpp::Context::get_domain_id.

https://github.com/ros2/domain_bridge/pull/60/commits/925a5e1798f21c53103e87d4f96f09adc1c9f940.

[x] rclcpp::InitOptions::set_domain_id.

This can not be replaced using get_rcl_node_options since that is a const function. Not sure how to compensate for this API without backporting at least https://github.com/ros2/rcl/pull/678, and ideally also https://github.com/ros2/rclcpp/pull/1165 - although this one is most likely not an ABI-compatible patch.
My attempt with https://github.com/ros2/domain_bridge/pull/60/commits/3272a4800db648d4351d287ce5ec85acf5e72701, https://github.com/ros2/domain_bridge/pull/60/commits/549654aaf6047bb28a474d7ffccdbebf0a9886d5, https://github.com/ros2/domain_bridge/pull/60/commits/eb58092084f524447d3c4424bf692ad0b75f1866 fails spectacularly, since (of course) modifying a const object through const_cast is undefined behavior.

[x] rclcpp::Duration::from_nanoseconds.
Use the Duration(rcl_duration_value_t nanoseconds) constructor instead.

https://github.com/ros2/domain_bridge/pull/60/commits/9f3b66adcaf892c334330152bf3c174bc3c6d0a1.

[x] Remove stray comments, debug code, etc.
[x] Document utility functions and other enums/structs (gen pub-sub) introduced in this PR.

Closes https://github.com/ros2/domain_bridge/issues/59.
Signed-off-by: Abrar Rahman Protyasha<EMAIL_ADDRESS>

Hi. I just tried to test this branch with with ROS foxy, but i encountered an error when a node begins to publish some data. I used the example_bridge_config.yaml file as mentioned in the readme file and published data in the chatter topic. the error is:
terminate called after throwing an instance of 'std::runtime_error'
  what():  unexpected callback being called

this is my example_bridge_config.yaml file:
name: my_bridge
from_domain: 0
to_domain: 3
topics:
  chatter:
    type: example_interfaces/msg/String

am i missing something alongway doing it or it's known issue?

@antonyramsy this branch was still WIP and I didn't have enough bandwidth to get back to it for a while now.
I remember seeing these errors when I was testing this branch too, and the problem was that in Foxy, rcl offers no such API to set domain IDs -- check https://github.com/ros2/domain_bridge/pull/60#discussion_r742555634 for what I'd already (unsuccessfully) tried. I just never had time to fully verify that my approach in https://github.com/ros2/domain_bridge/pull/60/commits/efa97ae33fed98103a5030cf70e605dc7295ea1a works.
I might have some time to revive this PR soon-ish, but if you need this branch to work I think the way to go would be to run the tests in this package and debug the errors from there. As it is, I think your configuration is correct but the branch is not ready to land.

@aprotyas  Thanks a lot for the response. I will consider using this package along with other approaches i can take, and will inform if any progress in the development has taken place.

@jacobperron as of ff41c31, the end-to-end tests pass in this branch. I'm still struggling against the default lifespan and deadline values in QOS profiles.
From https://github.com/ros2/rmw/pull/301#pullrequestreview-595709837, it looks like in Foxy, the default zero values for the lifespan and deadline means each RMW uses its own default value, which is default to test against compared to the "max" duration enforced post-Foxy (i.e. rclcpp::Duration(std::numeric_limits<int64_t>::max())). Is there a way around this?

@aprotyas  I've tested this branch. first i got this error:
CMake Deprecation Warning at /opt/ros/humble/share/rosidl_cmake/cmake/rosidl_target_interfaces.cmake:32 (message):
  Use rosidl_get_typesupport_target() and target_link_libraries() instead of
  rosidl_target_interfaces()

and changed rosidl_target_interfaces() with rosidl_get_typesupport_target(). after that another error appeared which looks like the message is not being created:
Starting >>> domain_bridge
--- stderr: domain_bridge
~/domain_bridge_ws/src/src/domain_bridge/domain_bridge.cpp:43:10: fatal error: domain_bridge/msg/compressed_msg.hpp: No such file or directory
   43 | #include "domain_bridge/msg/compressed_msg.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

full text log:
log.txt

@aprotyas Hi, sorry for delay. I've tested the updated code both on foxy and humble. in normal condition(sending some data in one domain and receiving it in another one) both if them are ok. but i realized that bidirectional bridging in ROS foxy is not working as i expect (maybe i'm doing it in the wrong way) but in humble it works fine(by using the same YAML config file in different versions of ROS). here is the config i'm using to test bidirectional bridge:
name: my_bridge
from_domain: 2
to_domain: 3
topics:
  chatter:
    type: example_interfaces/msg/String
    remap: chitter
    bidirectional: True

with this config i was able to publish some date on topic chatter from domain 2 and receive it in domain 3 under chitter topic. and vice versa(publishing in domain 3 in topic chatter and receiving it in domain 2 in topic chitter ) in ROS humble.
but doing the same in ROS foxy, although you are able to send data from domain 2 to 3, but you wont be able to see published data from domain 3 in topic chatter in domain 2 in topic chitter (the topic chitter does not show up in ros2 topic list command result.)


branch 0.3 web build fail

ERROR in ./app/containers/Dashboard/components/DashboardItem.tsx
Module not found: Error: Can't resolve '../../../components/DownloadCsv' in '/soft/davinci/webapp/app/containers/Dashboard/components'
@ ./app/containers/Dashboard/components/DashboardItem.tsx 78:0-58 885:81-92 929:31-42
@ ./share/containers/Dashboard/index.tsx
@ ./share/routes.ts
@ ./share/app.tsx
@ multi ./node_modules/react-app-polyfill/ie11.js ./share/app.tsx
ERROR in ./app/containers/Widget/components/Workbench/OperatingPanel.tsx
Module not found: Error: Can't resolve '../ComputedConfigForm' in '/soft/davinci/webapp/app/containers/Widget/components/Workbench'
@ ./app/containers/Widget/components/Workbench/OperatingPanel.tsx 81:0-55 2430:29-47
@ ./app/containers/Widget/components/Workbench/index.tsx
@ ./app/routes.ts
@ ./app/app.tsx
@ multi ./node_modules/react-app-polyfill/ie11.js ./app/app.tsx
ERROR in ./app/containers/Dashboard/components/DashboardItem.tsx
Module not found: Error: Can't resolve './DashboardItemMask' in '/soft/davinci/webapp/app/containers/Dashboard/components'
@ ./app/containers/Dashboard/components/DashboardItem.tsx 76:0-52 1061:38-55
@ ./share/containers/Dashboard/index.tsx
@ ./share/routes.ts
@ ./share/app.tsx
@ multi ./node_modules/react-app-polyfill/ie11.js ./share/app.tsx
ERROR in ./app/containers/Widget/components/Workbench/OperatingPanel.tsx
Module not found: Error: Can't resolve './Dropbox' in '/soft/davinci/webapp/app/containers/Widget/components/Workbench'
@ ./app/containers/Widget/components/Workbench/OperatingPanel.tsx 74:0-32 1987:35-42
@ ./app/containers/Widget/components/Workbench/index.tsx
@ ./app/routes.ts
@ ./app/app.tsx
@ multi ./node_modules/react-app-polyfill/ie11.js ./app/app.tsx
ERROR in ./app/containers/Widget/components/Workbench/index.tsx
Module not found: Error: Can't resolve 'containers/Dashboard/components/DashboardItemMask' in '/soft/davinci/webapp/app/containers/Widget/components/Workbench'
@ ./app/containers/Widget/components/Workbench/index.tsx 77:0-82 685:37-54 686:35-52
@ ./app/routes.ts
@ ./app/app.tsx
@ multi ./node_modules/react-app-polyfill/ie11.js ./app/app.tsx
Child html-webpack-plugin for "index.html":
1 asset
Entrypoint undefined = index.html
Child html-webpack-plugin for "share.html":
1 asset
Entrypoint undefined = share.html

Please provide your build environment information.

ubuntu 16.04
node  v10.15.3
npm 6.4.1
yarn 1.16.0

It is caused by the case sensitivity of the file name in linux OS system, we will fix it in time.

fixed: #1025

#1035 add webpack case sensitive plugin to check this problem when build on no-case-sensitive file name OS.


Conformer generation for a certain molecule makes one less conformer than requested

Describe the bug
We've encountered a situation where, when using EmbedMultipleConfs on a particular bicyclopentane-containing molecule, we sometimes get one less conformer than requested.
To Reproduce
>>> from rdkit import Chem
>>> from rdkit.Chem import AllChem
>>> mol = Chem.MolFromSmiles('[H][O][C](=[O])[N]([C@]12[C]([H])([H])[C@]([N]([H])[C](=[O])[C]3=[C]([H])[N]([C]([H])([H])[C]([H])([H])[H])[c]4[c]([H])[c]([H])[c]([H])[c]([C]([H])([H])[c]5[c]([H])[c]([H])[c]([C]([F])([F])[F])[c]([H])[c]5[H])[c]43)([C@@]1([H])[C]#[N])[C@@]2([H])[C]([H])([H])[c]1[c]([H])[c]([H])[c]([H])[c]([H])[c]1[H])[S](=[O])(=[O])[c]1[c]([H])[c]([H])[c]([H])[c]([H])[c]1[H]', sanitize=False)
>>> AllChem.EmbedMultipleConfs(mol, numConfs=1, randomSeed=1, pruneRmsThresh=0.25)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
0
>>> AllChem.EmbedMultipleConfs(mol, numConfs=2, randomSeed=1, pruneRmsThresh=0.25)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
1
>>> AllChem.EmbedMultipleConfs(mol, numConfs=3, randomSeed=1, pruneRmsThresh=0.25)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
2

Expected behavior
If the method can produce one conformer when two are requested, then it would make sense that it could produce that one when only one is requested (instead it produces 0 conformers when 1 is requested).
Screenshots
The conformers that are produced look reasonable. However the carboxylic acid group seems a little distorted (the C=O bond length is just 1 A)

Configuration (please complete the following information):

RDKit version: rdkit                     2021.03.1        py38hba24be9_0    conda-forge
OS: Mac OS 10.14.3
Python version (if relevant): python                    3.8.6           h624753d_5_cpython    conda-forge
Are you using conda? Yes
If you are using conda, which channel did you install the rdkit from? conda-forge

Do you see the same result if you don't use a nonzero pruneRmsThresh value?  The RMS pruning looks to be a postprocessing step.

Hi @jasondbiggs, thanks for the tip. I've dug a bit deeper today and do see the same result if I remove the pruneRmsThresh value.
>>> AllChem.EmbedMultipleConfs(mol, numConfs=1, randomSeed=1)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
0

I've tried a few other things and found:

Picking a different randomSeed can generate a conformer for the numConfs=1 case

>>> AllChem.EmbedMultipleConfs(mol, numConfs=1, randomSeed=2)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
1

But for numConfs=2, it's not until randomSeed=7 that we get a second conformer:

>>> AllChem.EmbedMultipleConfs(mol, numConfs=2, randomSeed=1)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
1
>>> AllChem.EmbedMultipleConfs(mol, numConfs=2, randomSeed=2)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
1
>>> AllChem.EmbedMultipleConfs(mol, numConfs=2, randomSeed=3)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
1
>>> AllChem.EmbedMultipleConfs(mol, numConfs=2, randomSeed=4)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
1
>>> AllChem.EmbedMultipleConfs(mol, numConfs=2, randomSeed=5)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
1
>>> AllChem.EmbedMultipleConfs(mol, numConfs=2, randomSeed=6)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
0
>>> AllChem.EmbedMultipleConfs(mol, numConfs=2, randomSeed=7)
<rdkit.rdBase._vecti object at 0x1333cf990>
>>> mol.GetNumConformers()
2

I agree that the problem is probably somewhere after conformer generation. I poked around a bit in the relevant code but nothing popped out at me as being the obvious cause.
Maybe there's some distorted geometry that's being created, which causes getMolSelfMatches or _isConfFarFromRest to silently discard some conformers.

This isn't connected to RMS pruning.
That's a pretty highly constrained system - the bicyclopentane group where four of the atoms have specified chirality makes things difficult for the conformer generator - and the code is having problems generating conformers for it.
In cases like this there are two things you can try:

Switch to using random coordinate embedding (useRandomCoords=True). This can (but does not always) allow things to work better for highly constrained molecules.
Increase maxAttempts. This defaults to zero which means that it internally gets set to 10*numAtoms. The code tries this number of times to generate each conformer. You can make this value as large as you want... more tries means a longer wait (but a greater chance of success).

Thanks, @greglandrum - useRandomCoords=True seems like the best solution here, and now that I look I see you've recommended it before. Thanks for the tip!


N16: Learn Wgpu

I didn't really have an image I wanted to use as all the changes have been mostly code based.

I'll get that fixed. For future reference, is there a way to run the CI locally?

Merged, thanks for the PR!


Add next class functionality

Change the return value of nextClass() when the user asks next class at the last class of the day.

Added with and without next day follow-up - [f82dbd3a8fc1542dc4d58efb28503f9bb21e6015]


[LPIF] Stream ID port width is too narrow

Version 1.1 of the LPIF specification has a width of 8 bits for pl_stream and lp_stream ports. Current RTL implements a width of 3 bits.
Observed in 9-21-2021 RTL

Fixed. Updated the RTL. Also fixed pl_tmstmp_stream.

This should be fixed in 0.8.6 drop.

Port widths are now correct. The parameter widths defined in lpif_ctl.sv are still 3 bits though. Observed in RTL v0.8.6


package Ray 2.35.0 and add ray-adag

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

This incorporates changes from #172 for aiohttp

Now, back to py312...


Support iOS 8.0

Add support for iOS 8.0

@cjndubisi, I didn't realize that Alamofire and Moya had started to support iOS 8 again. (https://github.com/Moya/Moya/pull/787) Thanks for your work! I'll merge this PR and make a new release when the test succeed.

@cjndubisi Thanks for contribution! 0.4.0 has been released :tada: