support message passing and keeping native WebView in memory when react component is unmounted

PR Status
Ready for review. We'd like this PR to get merged if it gets approved.
Overview of Changes
By default, moving a React WebView component in the view hierarchy creates a new native WebView and refreshes the iframe. In Discord's mobile React Native apps which use react-native-webview, we need the ability to move the WebView without refreshing the iframe, and this PR adds support for that on iOS and Android by reusing the native WebView instance. We also need to be able to pass messages between the WebView and the react-native app without a react tag (e.g. when the React component for the WebView is not mounted), and this PR adds new APIs that support that, too. To test these changes, we added a "Portals" button to the RN-webview sample app. We've also been testing these changes in the Discord app.
High level react-native-webview API changes
This PR updates react-native-webview with the following new APIs

the webViewKey react prop- a key for accessing specific WebView instances. When this is defined, the native WebView stays in memory until explicitly released.
releaseWebView - a function that explicitly releases native WebViews given a WebView key.
the messagingWithWebViewKeyEnabled react prop - enables message passing without a react tag (i.e. without needing the react component to be mounted)
injectJavaScriptWithWebViewKey - supports passing messages / injecting javascript to the native WebView given a WebView key even if the react component for the native WebView is not mounted anywhere
addOnMessageListenerWithWebViewKey - When the webViewKey is defined, onMessage and similar callbacks like onLoadStarted don't work. This method adds a listener for messages from the WebView given a WebView key, and this handles messages even when the react component is unmounted.

How This Works
iOS Changes:
We have a singleton map where the keys are strings, and the values are WKWebViews. When React Native asks for a new view, we create a new RNCWebView instance. RNCWebView is a parent of WKWebView, and when the props say to reuse the native WebView instance, RNCWebView looks for the existing WKWebView instance from the map before it decides whether it needs to create a new WKWebView instance. Additionally, when we move a WKWebView between two RNCWebView parents, we need to remove the reference to the old parent that was held through the observer. We also have a map of web view keys  to RNCWebViews, so we can remove the correct observer from the WKWebView. If we don't remove the observer when we change the WKWebView's parent, the app will crash with an EXC_BAD_ACCESS error.
Android Changes:
We renamed the old RNCWebView, which extended the Android WebView to InternalWebView. Then we were able to  wrap InternalWebView with a new view, which we named RNCWebView, which is a FrameLayout. Now that InternalWebView is wrapped by the RNCWebView view group, the Android architecture can match the iOS architecture where the RNCWebView has a one-to-one relationship with the react component, and the inner / native WebView does not have a one-to-one relationship with the react component. Similar to the iOS architecture, we also have singleton maps that hold references to RNCWebViews and InternalWebViews for WebView keys.
Before and After
Here's the behavior where the iframe refreshes when webViewKey undefined.
https://user-images.githubusercontent.com/3821698/165408230-849f75bc-771d-4658-8171-5b64fb1bf277.mov
Here's the behavior where the iframe maintains its state as the react WebView moves in the view hierarchy when webViewKey is defined.
https://user-images.githubusercontent.com/3821698/166070390-78b6e6be-701b-4e03-9bd2-b5c9c57cda17.mov
Testing messaging passing without a react tag
See https://github.com/discord/react-native-webview/pull/6 and https://github.com/discord/react-native-webview/pull/12 for screen recordings and explanations on how we tested the message passing between the WebView and the react-native app without a react tag.

In https://github.com/react-native-webview/react-native-webview/pull/2469/commits/03d597c5008c1e132bc9c91ef249c46c2b382fa5 , I added the changes from https://github.com/discord/react-native-webview/pull/4 (and changes from some of the commits on https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 that came after that PR ). While integrating this with the Discord iOS app, I realized that I needed the ability to release the WebView with a web-view-key via a static method when the React WebView class component wasn't available.
https://user-images.githubusercontent.com/3821698/166984290-98c15af3-716d-4844-a127-868e9a54e7a4.mp4

https://github.com/react-native-webview/react-native-webview/pull/2469/commits/3a8d9c7bd076798d50ea60b484fabc972e4b246a pulls in a change from https://github.com/discord/react-native-webview/pull/5 that fixes an EXC_BAD_ACCESS crash that I was running into when navigating away from the WebView, backgrounding the app, foregrounding the app, and navigating back to the WebView.

Overall I really like this approach, at work we have been developing this feature set as well. The API is very similar and we have it fully working on Android. Hopefully we can work together to get this feature set stable and in the open source codebase here.

@mattapperson , that's exciting! Do you have this working on iOS also, or do you only have it working on Android so far? Do you have the code for your changes publicly viewable (e.g. in a fork of the repo)? If not, are you willing to share that e.g. via a PR into this repo or into https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 ? I'm curious about the similarities and differences between your current approach versus my iOS changes in this PR and my proposed Android changes in the PR description

Nothing is public yet for this for our effort, but I hope to very soon.

@mattapperson , sounds good! Feel free to tag me when you all are ready to make that work public. I'm interested in collaborating to get the feature set stable and merged into this open-source / upstream react-native-webview code

@TheAlmightyBob , thanks for the feedback!
@Titozzz and @jamonholmgren , do you have any thoughts on the proposal in the PR description?

Fyi I'm in a major rewrite of the lib for the new architecture, so you'll probably have a lot of conflicts to handle.

Fyi I'm in a major rewrite of the lib for the new architecture

@Titozzz , in the new architecture, on Android, is there a plan to make the Android WebView get wrapped by an RNCWebView similar to what's currently on iOS? I'm wondering, since I would've had to do that anyway to make the functionality in this PR work for Android

I have implemented this feature for Android by referring to this PR and https://github.com/discord/react-native-webview/pull/2 (These helped us a lot). I've forked react-native-webview/react-native-webview to a private repo on my company's organization and added diffs like this: https://github.com/cou929/react-native-webview/pull/1
(Note: This https://github.com/cou929/react-native-webview/pull/1 branch is just for sharing, so this is not the actual code I internally use in our production. Also, in this branch, we only implemented the requirements we needed, which might be not sufficient for common use cases.)
I hope this PR is going to be merged into the master eventually due to the maintenance cost of our forked branch. But it looks like it hasn't been updated in months. Do you have any plans for this PR? Is there anything I can do to help move forward?

@cou929 , that's great that you got this working! Coincidentally, @nealmanaktola (my team member / coworker) got this working on Android in Discord's fork of react-native-webview last week in https://github.com/discord/react-native-webview/pull/11 and https://github.com/discord/react-native-webview/pull/12 .
@TheAlmightyBob , if we were to get this PR updated with both iOS and Android changes, and after this PR gets reviewed, would you be open to merging this PR? @Titozzz mentioned a new architecture for react-native-webview. Would we need to wait for those changes to complete before merging this PR?

Do you have any plans for this PR? Is there anything I can do to help move forward?

I'll get back to you on that after I talk with my team and after @TheAlmightyBob responds to the question above. I'd like this PR, eventually with both iOS and Android changes, to get merged into react-native-webview/react-native-webview.

if we were to get this PR updated with both iOS and Android changes, and after this PR gets reviewed, would you be open to merging this PR?

@TheAlmightyBob, would you prefer a PR per platform (one for iOS and one for Android) or a single PR with both iOS and Android changes?

Do you have any plans for this PR? Is there anything I can do to help move forward?
I'll get back to you on that after I talk with my team and after TheAlmightyBob responds to the question above.

@cou929 , for now, you can use Discord's fork of react-native-webview if you'd like to use these features on both iOS and Android https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 . Later, depending on the response to https://github.com/react-native-webview/react-native-webview/pull/2469#issuecomment-1238404004 , we can hopefully get these iOS and Android changes merged into react-native-webview/react-native-webview.

You can check my work for new arch the branch is published already and 95% of features are good

would you be open to merging this PR?

Yes, but...

@Titozzz mentioned a new architecture for react-native-webview. Would we need to wait for those changes to complete before merging this PR?

...I haven't been involved with this work and don't know the scope/timeline. I guess I'd encourage checking out his branch as he mentions above?

would you prefer a PR per platform (one for iOS and one for Android) or a single PR with both iOS and Android changes?

Although I expect it'll be very large, I think I'd lean towards a single PR. But if there's a logical way to organize the commits for reviewing incrementally, that might help.

@Titozzz , do you have a rough estimate on when https://github.com/react-native-webview/react-native-webview/commits/new-arch will merge to master?
@nealmanaktola and I will check out that branch and see if there are any major conflicts with out changes. Depending on when that new-arch branch is expected to merge, @nealmanaktola and I could either make our PR against master or against new-arch. @TheAlmightyBob , do you have any preferences on which branch we should make the PR against? If we were to make the PR against master, would you prefer us updating everything on this PR, or would making a completely new PR be easier?

Although I expect it'll be very large, I think I'd lean towards a single PR. But if there's a logical way to organize the commits for reviewing incrementally, that might help.

Sounds good! @nealmanaktola and I will initially plan on creating a single PR for both iOS and Android.

I need to understand what to do with new iOS lifecycle on Fabric, it's actually more about deciding than code left

Depending on when that new-arch branch is expected to merge, @nealmanaktola and I could either make our PR against master or against new-arch. @TheAlmightyBob , do you have any preferences on which branch we should make the PR against?

Not a strong one. I'm not sure how "stable" the new-arch branch is right now, whether you would have trouble rebasing on top of that. I also don't know how realistic the possibility is of this PR getting merged first.

If we were to make the PR against master, would you prefer us updating everything on this PR, or would making a completely new PR be easier?

Er, did you mean to say if you were to make it against new-arch? It's already against master...?
In general I think it depends on the nature/scope of the changes. If you're just pushing new commits, then updating this PR seems to make sense since there's already been some discussion here. If you're basically redoing everything (which might be essentially what happens if you target new-arch?), or other such drastic changes, then a new PR might make sense.

@donaldchen

for now, you can use Discord's fork of react-native-webview if you'd like to use these features on both iOS and Android https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 . Later, depending on the response to #2469 (comment) , @nealmanaktola or I can take care of updating this PR, so we can hopefully get the iOS / Android changes merged into react-native-webview/react-native-webview.

That’s amazing. Thank you so much!

Er, did you mean to say if you were to make it against new-arch? It's already against master...?
In general I think it depends on the nature/scope of the changes. If you're just pushing new commits, then updating this PR seems to make sense since there's already been some discussion here. If you're basically redoing everything (which might be essentially what happens if you target new-arch?), or other such drastic changes, then a new PR might make sense.

@TheAlmightyBob , I was asking if it'd be easier to create a brand new PR against master, but since we're not going to redo everything in this PR, @nealmanaktola and I will tentatively plan on updating this PR (still against master) with our new Android changes and some small adjustments on iOS. We can revisit creating a PR against new-arch some time in the future - perhaps after the new-arch and its timeline for merging is more stable

@Titozzz  and @TheAlmightyBob , @nealmanaktola  and I have updated the PR title, description, and commits with our updated iOS and Android changes. This PR is ready for a full review, and we're hoping to merge this to master after we've addressed any requested changes you might have. The PR description has a "High level react-native-webview API changes" section, and if the high level API looks good to you, @nealmanaktola or I can update this PR with simple docs for these new APIs under https://github.com/react-native-webview/react-native-webview/tree/master/docs

Whoops, I didn't mean to remove anyone as a reviewer, and I can't seem to explicitly add anyone as a reviewer. @Titozzz , we'd like your review as well if you're available. Thanks!

We also need to be able to pass messages between the WebView and the react-native app without a react tag (e.g. when the React component for the WebView is not mounted)

When the webViewKey is defined, onMessage and similar callbacks like onLoadStarted don't work

Could you explain a bit more about this scenario?

Could you explain a bit more about this scenario?

@TheAlmightyBob
To clarify, the way it's currently built is that any message event gets published to the EventEmitter when a webViewKey is set, so instead of messages being sent to the regular bound onMessage fn, it gets sent via the EventEmitter
like here: https://github.com/react-native-webview/react-native-webview/pull/2469/files#diff-e9700631fc9fa5600d2e1ee70eb2d28aa2a1fbc695b85f99981972d3904ca485R1816
We added addOnMessageListenerWithWebViewKey because with these changes, it's now possible for messages to be dropped when listening to messages via the onMessage prop fn.  Message drops happen when the react component is unmounted but the WebView instance is still alive.
Now with respect to onLoadStarted and other callbacks, those callbacks will not trigger when the react component is unmounted. Since those callbacks rely on the viewId/viewTag, they will not trigger when the react component is unmounted.
When the WebView react component is mounted, these other callbacks do trigger.

@nealmanaktola Thanks for the explanation.

When the WebView react component is mounted, these other callbacks do trigger.

Does that mean that this line from the description is slightly inaccurate, and it should just say that the callbacks don't work when the component is unmounted?

When the webViewKey is defined, onMessage and similar callbacks like onLoadStarted don't work

I'm still wrapping my head around things, but I'm kind of curious if it could make sense to have a new kind of "proxy" object instead of these separate ___WithWebViewKey functions.... like, something that could sort of mimic a webViewRef (not necessarily the full API, but in a more familiar way)...?
I haven't fully thought this through, but I'm vaguely imagining being able to do something like
const webViewProxy = getWebViewProxy('key', { onMessage: this.handleMessage, onLoadStart: this.handleLoadStart });
...
webViewProxy.injectJavascript('...');
...
webViewProxy.release();

I'm a little concerned about the src folder ballooning with tons of little files for each of these methods/events and it becoming difficult to keep them straight with their "normal" counterparts... I at least appreciate the problem you're solving now, but trying to imagine how best to keep this manageable going forward...

Does that mean that this line from the description is slightly inaccurate, and it should just say that the callbacks don't work when the component is unmounted?

Yeah that line from the description is slightly inaccurate. We'll update that. Good call. onMessage does not work at all (even when the component is mounted or unmounted) when WebView has a webViewKey. However, the other callbacks do indeed work when the component is mounted.

I'm still wrapping my head around things, but I'm kind of curious if it could make sense to have a new kind of "proxy" object instead of these separate ___WithWebViewKey functions.... like, something that could sort of mimic a webViewRef (not necessarily the full API, but in a more familiar way)...?

Agreed, I think right now the withWebViewKey paradigm doesn't really scale that well. A single proxy object does make sense to me and provides some familiarity as with a webViewRef as you noted. In terms of implementation, I am thinking that WebViewProxy would be a wrapper object over these functions that we've already defined.
I have to think a bit more on how this would actually work though.

@nealmanaktola

onMessage does not work at all (even when the component is mounted or unmounted) when WebView has a webViewKey

could it? i.e., could it still trigger the regular onMessage if that exists? Is there a technical reason, or is it more just that you want to steer people to use addOnMessageListenerWithWebViewKey if they're using webViewKey?

@TheAlmightyBob it definitely could. It's more that we wanted to steer people to use addOnMessageListenerWithWebViewKey if they're using a webViewKey

Hi guys, awesome work, I’m waiting it for a long time. Do you think if I could use this implementation to create a custom navigation where each link opens in a new screen with a single webview instance ?

Do you think there's a way to move the webview instance back ?
https://user-images.githubusercontent.com/2651240/191662096-0f1ee169-0f27-44ef-a9fc-d2f8f6b237d5.mov

@lazaronixon , to clarify, are you asking if the changes in this PR will allow you to use the same native WebView instance in different screens within a react-navigation Navigator both when opening new links and when going back? If so, then I think that should be possible with these changes.
For opening new links, when the newest link / screen mounts its React WebView component with a given WebView key, it'll reuse the native WebView. For going back, currently the previous screens will only reuse the native WebView if their React WebView components remount. With the current API, you might be able to use https://reactnavigation.org/docs/use-focus-effect/ and remount the React WebView component when the screen refocuses.
@nealmanaktola  and I have also discussed the possibility of adding a rebind method for using the shared native WebView with a React WebView component that is already mounted (without needing to remount the react component). That API isn't ready in this PR though

@TheAlmightyBob
Would you like to see the WebViewProxy changes described in https://github.com/react-native-webview/react-native-webview/pull/2469#issuecomment-1247357261 in this PR? Trying to determine if this is a suggested improvement that could happen in a separate PR or if this is a blocking requested change for this PR.

@nealmanaktola I still need to get through the rest of this PR, but my main concern with making the Proxy a separate PR is that it might be a breaking change for anybody who starts using this.
Otherwise, or if you think the functions you've already defined would need to stay anyway, I wonder if they could be moved to a subfolder? Possibly exported as a module?
Just pondering how to avoid filling up the src folder with lots of little files for each of these functions, as well as perhaps aiding discoverability...

I still need to get through the rest of this PR, but my main concern with making the Proxy a separate PR is that it might be a breaking change for anybody who starts using this.

That's a good point. I think given that some other callbacks won't be triggered when the view is unmounted. I think it might be cleaner to introduce this concept now.
I'll think about this and update the PR accordingly.
Thank you for the detailed review!

@TheAlmightyBob Thanks the additional passes, I haven't had time the past week to look through your additional feedback. I'll spend some time this week addressing it :)

@donaldchen @nealmanaktola A heads-up that, since there hasn't been recent activity on this PR, we're going to prioritize getting the new architecture support in first. Still fans of this idea though!

@TheAlmightyBob makes sense - haven't had time recently to make these changes, but will get to it.

Hey bot, let's keep this one open

@TheAlmightyBob , I addressed some of the initial feedback this week. Here's some of the high level feedback I addressed in the below commits.

Added the WebViewProxy to encapsulate all the ***WithWebViewKey methods
Renamed the Android container to RNCWebViewContainer and renamed the Android WebView to RNCWebView for backward compatibility (particularly around the customization use case)
Removed the maps referencing the outer WebView containers. (We can now get the parent or superview of the internal WebView).

I hope to address the other comments at another time. For now, are you available to review the commits above, and can you clarify if there's any other high level feedback that I'd need to address in this PR before this PR can get merged? Thanks!

.

Hey @donaldchen , sorry I completely missed that there'd been an update here until that bot brought my attention to it (I guess it's good for something after all...)
Have you by chance looked at https://github.com/react-native-webview/react-native-webview/pull/2874 at all? That introduces an RNCWebViewWrapper on Android that subclasses FrameLayout, like your RNCWebViewContainer. It has different reasons for doing so, of course, but hopefully the approaches are compatible?

Hey @donaldchen , the last batch of changes look good to me, my comments are mostly small things.
The two biggest "high-level" concerns on my mind (neither of which I think we will be news to you) are:
The new architecture. The big refactor has been completed/released, so this will need to be made compatible (e.g. most of the Android implementation has been converted to Kotlin). I don't expect any major issues/incompatibilities, but not 100% certain
Documentation/commenting. This definitely introduces complexity, and we want to make it as easy as possible both to use and to maintain

Hey @TheAlmightyBob , thanks for the feedback! That makes sense. I'd like to update this PR to work with the new architecture, and I'll plan on adding more documentation, too. My availability is still TBD, but I'm hoping I can come back to this in one to two months. We'll tag you when this is ready for another review. Thanks!

@donaldchen I have tried to use your branch, however the webview shows a blank page. It does not load for me.
i have added
<WebView
        webViewKey="web1"
        messagingWithWebViewKeyEnabled={true}

        ref={webviewEl}

My code also postMessages
 const js = `
window.WebViewBridge = {
  onMessage: (message) => {
    const { type, data } = JSON.parse(message || '{}');
    document.dispatchEvent(new CustomEvent(type, { detail: data }));
  }
}

@DavidGOrtega Does it work if you use it "normally" (without the webviewkey stuff)?
I haven't used the branch with the latest changes, but a previous version was working for me.

@DavidGOrtega , I haven't been available to test this branch recently, but I hope to test more when I get around to working on making this work with the new react native architecture https://github.com/react-native-webview/react-native-webview/pull/2469#issuecomment-1572325629 .
Like @TheAlmightyBob suggested, you could try an older commit such as https://github.com/react-native-webview/react-native-webview/pull/2469/commits/413cae346d84b4be0e3f3e93a1c4d5aa2586d9d5 . That one was before my last batch of changes in March.
Also, what are you passing in for the source prop? If you pass in what the Portals.tsx example uses, does it work?
https://github.com/discord/react-native-webview/blob/237796aced1f986a5f8c33d5be295c5fbf74e728/example/examples/Portals.tsx#L160
https://github.com/discord/react-native-webview/blob/237796aced1f986a5f8c33d5be295c5fbf74e728/example/examples/Portals.tsx#L82-L103

My code also post messages
const js = `
window.WebViewBridge = {
onMessage: (message) => {
const { type, data } = JSON.parse(message || '{}');
document.dispatchEvent(new CustomEvent(type, { detail: data }));
}
}
any hint?

Can you share more context on where this code is running? Is this code in a react-native-desktop app using a WebView, or is this code that runs inside the WebView?
You can view https://github.com/discord/react-native-webview/blob/237796aced1f986a5f8c33d5be295c5fbf74e728/example/examples/Portals.tsx#L215 for an example of posting messages to the WebView and https://github.com/discord/react-native-webview/blob/237796aced1f986a5f8c33d5be295c5fbf74e728/example/examples/Portals.tsx#L112 for an example of listening to messages sent from the WebView back to the react-native app that hosts the WebView

Hi!

@DavidGOrtega Does it work if you use it "normally" (without the webviewkey stuff)?

Yes

If you pass in what the Portals.tsx example uses, does it work?

Let me try it.
The issue is more puzzling than I thought. Sometime loads the page, sometimes does not. For sure what is happening also is that If it loads and the device get a locked screen if I go back the blank screen is there.
Have I determined how the white screen appears at the beginning? Sort of.
My app requests lots of permissions. Using this webview if it does not ask for all the permissions... it works, if it takes longer to reach the point where the UI should display it does not.
This is tested in android 11 within a real device

Small update from my end: I had begun updating this to support the new arch in https://github.com/discord/react-native-webview/pull/27 , but I haven't had time to get those changes fully working yet. I hope to revisit those changes again some time in the future

Hello 👋, this PR has been opened for more than 2 months with no activity on it. If you think this is a mistake please comment and ping a maintainer to get this merged ASAP!

@TheAlmightyBob , I'd like to keep this PR open if possible. I'm still hoping to get back to this in the future, but I don't have a specific timeline yet

can this please be merged

Hello 👋, this PR has been opened for more than 2 months with no activity on it. If you think this is a mistake please comment and ping a maintainer to get this merged ASAP!

@TheAlmightyBob , I'd like to keep this PR open if possible. I'm still hoping to get back to this in the future, but I don't have a specific timeline yet

i tried this PR, it crashes if transporting into a modal or another screen

i tried this PR, it crashes if keying it into a modal, or another screen, then unmounting that screen

@Darren120 , this branch https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 has a more stable version of this code. For the branch in this PR (https://github.com/discord/react-native-webview/tree/11.18.1-discord-2), I haven't had time to resolve all the errors and issues yet when making it work with the new react-native architecture

Thank you for your work. I will try that branch.
but on the previous branch i have a question
is it necessary to use global context and pass the webview memo around? Or can I just mount a new webview with an existing webview key?
I am trying to reuse webview in a new screen/ modal. but if i exit the screen where the original webview is initialized, it crashes the app.

i tried this PR, it crashes if keying it into a modal, or another screen, then unmounting that screen

@Darren120 , this branch https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 has a more stable version of this code. For the branch in this PR (https://github.com/discord/react-native-webview/tree/11.18.1-discord-2), I haven't had time to resolve all the errors and issues yet when making it work with the new react-native architecture

hm i am using that version, but it seems like it keeps crashing when unmounting.
Screen A => Screen B (initilize web view) => Screen C (shared webview key)
Screen B to C navigation is no problem, once i navigate to screen A, it crashes

i tried this PR, it crashes if keying it into a modal, or another screen, then unmounting that screen

@Darren120 , this branch https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 has a more stable version of this code. For the branch in this PR (https://github.com/discord/react-native-webview/tree/11.18.1-discord-2), I haven't had time to resolve all the errors and issues yet when making it work with the new react-native architecture

UPDATE:
haha sorry about that, i forgot to prebuild my expo while jumping around branches. no more crashes! excellent work! i hope we can get this merged!

i tried this PR, it crashes if keying it into a modal, or another screen, then unmounting that screen

@Darren120 , this branch https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 has a more stable version of this code. For the branch in this PR (https://github.com/discord/react-native-webview/tree/11.18.1-discord-2), I haven't had time to resolve all the errors and issues yet when making it work with the new react-native architecture

hello, i am wonder what happened to
injectJavaScriptWithWebViewKey? is there an alternative to this in the stable version?

i tried this PR, it crashes if keying it into a modal, or another screen, then unmounting that screen

@Darren120 , this branch https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 has a more stable version of this code. For the branch in this PR (https://github.com/discord/react-native-webview/tree/11.18.1-discord-2), I haven't had time to resolve all the errors and issues yet when making it work with the new react-native architecture

An issue I am encountering is cookies are either not being shared if another webview sets them, or its being resetted
I think it broke sharedCookiesEnabled

i tried this PR, it crashes if keying it into a modal, or another screen, then unmounting that screen

@Darren120 , this branch https://github.com/discord/react-native-webview/commits/11.18.1-discord-1 has a more stable version of this code. For the branch in this PR (https://github.com/discord/react-native-webview/tree/11.18.1-discord-2), I haven't had time to resolve all the errors and issues yet when making it work with the new react-native architecture

further debugging, seems like the cookies are not being synced if a web view is activity being viewed while portaled.
for example, i am showing the web view in another screen by portaling it there, but after a while my auth cookies are not synced. if i stay on the original screen, the cookie remains synced, and navigating to the portaled webview works fine. so i guess while webview is portaled and activity being viewed, it is not syncing cookies

FYI: I still like the idea behind that PR, I'm thinking of rewriting webview around new arch with this capability


Linux WebKitGTK Memory Consumption

Webview looks to be a very good framework looking at the binary size,
which is very tiny compared to the 115mb Electron Hello World,
but there is another major problem, memory.
I'm no benchmarking expert, but this is RAM usage in different UI frameworks:
(Debian 10 + MATE, 8GB, i7 3770, GT 1030)

GTK: ~30mb
Webview Framework: 200-300mb, including WebKit being loaded into memory
Electron Quick Start: 50-130mb

(These may be inaccurate, tell me if you have different results)
Comparing Electron performance to Webview performance,
they are about the same, as they both load a web rendering engine into memory.
And with WebKit being 300mb to install on Linux,
it can be seen to be even heavier than Electron if you don't already have it installed.
There may be some usable alternatives to WebKit, but I think optimizations can be made,
even if changing dependencies would not be possible.
For some people, they may be using this for the HTML/CSS rendering, and don't need JS.
At that point, using a UI library would be practical, but I think having an option to disable
JS and use WebKit's click event binding feature would not be a bad idea, assuming it
is possible within WebKitGTK.
(I don't think this is an issue with Windows/Android, as their web rendering software
is built in and ready to go.)
Here are some raw benchmarks for WebKitGTK on my old laptop.
http://petabyte.heb12.com/filedump/benchmark

Accidentally posted to early

Can you specify what you used to produce the benchmarks?
So far I've been compiling the old c/mshtml v0.1.1 branch on Ubuntu and win10. I could produce some comparisons, for ubuntu 18.04 and 20.04, and possibly mswin via cygwin or WSL

I used top | grep WebKit to just see WebKit results.

Drafting a longer reply to some of the things you mentioned. But I did some searching on how to clarify memory usage metrics in *nix today; not much clarity but .. I definitely think this track of thinking about metrics has long-term value .. and that since unix/linux are closeset to c/cpp (vs mac/win), and c/cpp is the core code of this webview/webview project, that linux and webkitgtk make sense as the first systems/workflows to document and measure
Anyhow, some data:
(OS ver + Destop Environment, RAM, Disk size/used, Machine)
(Ubuntu 18.04 + i3, 8GB, 331G/299G, Lenovo x240)
As I understand the manpage for ps, these should show the immediate usage for virtual and pyhsical memory of a webview app and the underlying webkit/network process [vsz is in kibibytes, rsz/rss is in kilobytes, so numbers here should be megabytes]
$ echo "$(ps -o vsz,command ax | grep app_0709 | head -1 | cut -d ' ' -f 1) / 1024" | bc
100396
$ echo "$(ps -o rsz,command ax | grep app_0709 | head -1 | cut -d ' ' -f 1) / 1000" | bc
57
$ echo "$(ps -o vsz,command ax | grep WebKit | head -1 | cut -d ' ' -f 1) / 1024" | bc
84115
$ echo "$(ps -o rsz,command ax | grep WebKit | head -1 | cut -d ' ' -f 1) / 1000" | bc
89

and with kibi/kilo -bytes
$ ps aux | head -1; ps aux| grep -i webkit|grep -v grep; ps aux|grep app_07 |grep -v grep
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ob         451  0.0  1.1 86134104 88592 pts/5  SLl  07:48   0:08 /usr/lib/x86_64-linux-gnu/webkit2gtk-4.0/WebKitWebProcess 7 15
ob         452  0.0  0.6 103327148 51208 pts/5 SLl  07:48   0:00 /usr/lib/x86_64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess 8 15
ob         444  0.0  0.8 102838512 65172 pts/5 Sl   07:48   0:02 app_docix/app_0709-1000

Output of top command corresponding to your results. [not clear what VIRT means here, whereas 'vsz' above is ~100MB? and I don't even have 82GB of disk space]
$ top |head -7|tail -1
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
$ top | grep app_
  444 ob        20   0 98.074g  65480  54564 S   0.3  0.8   0:03.05 app_0709-1000
$ top | grep -i webkit
  451 ob        20   0 82.144g  89304  71452 S   4.6  1.1   0:09.71 WebKitWebProces
  452 ob        20   0 98.540g  51196  37072 S   0.3  0.7   0:00.20 WebKitNetworkPr

My google-fu is failing so far on better clarification, but some other approaches are discussed at this page: https://elinux.org/Runtime_Memory_Measurement

I also had a massive number in VIRT, I'm not sure where it got that from, but on my old 2005 Thinkpad I got ~250mb, so I thought that seemed about right.

So my ideas for fixing this:

Use NetscapeGTK? (couldn't install it from Debian repos)
Another good alternative is described in another project by this organization owner: https://github.com/zserge/lorca

@pufflegamerz - FYI  zserge is the one behind this repo

I don't think, however, that there is any desire from the community here to move to netscapeGTK or chromium embedded.

Well, looking at the current usage, I would not use this for an actual Linux-based production project.

I'm highly interested in this topic and comparisons with other possibilities for Linux webview deployment. It might take me a few days to follow up, but @pufflegamerz has brought up numerous points here that seem worth attention

I can't think of any widely available alternative for Linux atm and Lorca is taking a completely different approach to what webview is trying to accomplish.
If we have a better and widely available alternative to webkitGTK it is surely reasonable to talk about it, but I don't think, that the memory usage of webkitGTK is in scope of our project here.

@geaz, I don't mean to step on toes or divert energies; just wanted to follow through this thought while I'm starting to think more closely about the project
My takeaways and a bit of detail:

WebKitGTK has memory management issues
For this project, limiting core *nix support to default Ubuntu LTS seems like a central point and consistent with the overall aim for simplicity
Production use of a WebView-based application for memory sensitive cases on Linux might be possible with another engine, which could be an interesting experiment, and possibly a future supplemental/extension backend

Similar to @pufflegamerz's tests with webview-x cpp code and the webkitGTK sample code, I get outsized memory usage results with the v0.1.1 c code, and also with the tiny webkitGTK-based surf browser. Also I get intermittent streams of this warning when running webview-c or surf from command-line:
Memory pressure relief: Total: res = 16084992/16089088/4096, res+swap = 11452416/11452416/0

So, consistent with #228, it seems safe to say that webkitGTK is the main culprit and not webview's implementation.
The current situation works for my purposes, but it might make sense to someone to write alternate implementations of the webview api. Possibly for:

https://github.com/qt/qtwebkit - funny that WebKit came from KHTML, but then QT's dependencies..
HTML-only backends like dillo or netsurf - I like the notion above of non-js and other minimized use-cases
Closed-source and probably absurdly expensive but focused on embedded scenarios: https://www.ekioh.com/flow-browser/

I did some browser tests:
NetSurf - Fast, minimal, JS support, hard to install, same support level as IE8 probably.
Dillo - Tiny, lightweight, minimal html/css support
QtWebKit - Couldn't test, I don't know Qt or C++ enough. May be easier on memory, and may have better memory management.

There were a few that I avoided, such as
https://ultralig.ht/
and maybe 1 other (I can't remember it's name),
because they were proprietary.

I'm with @geaz & @nothingismagick here. The fact that webview doesn't have any outside requirements other than GtkWebKit which has widespread use and tends to already be installed on most machines in my experience is what makes this library so awesome. I'd say if there's memory issues in GtkWebKit these should be raised upstream.

I do want to maybe get in contact with a WebKitGTK developer and ask how memory is managed there, or somebody else can do it if they already know one.

Though old, this may be applicable: https://bugs.webkit.org/show_bug.cgi?id=17148

@abeMedia
I do think it was a good idea to research some other possible web rendering engines, and see how WebKitGTK compares against them though.
Looking through the WebKitGTK documentation again and seeing what memory optimizations could be added here
would maybe be a good idea, and that is a dead end, I can close this and move it to a WebKitGTK bug tracker. (I'm not entirely sure if this is a WebKitGTK or a WebKit issue)

Seeing that an alternative to WebKitGTK is unwanted, I will close this.

@petabyt , I'm reviving the discussion, since we now have an officially maintained "lightweight" version of WebKit, the WPE. More info here https://github.com/webview/webview/issues/638


[process] Server process has been terminated，The error has been consistently reported in VSCode.

✓ Running on http://localhost:8080 (0.2s)
The Dart VM service is listening on http://<IP_ADDRESS>:8181/IDGi_8oYxNc=/
The Dart DevTools debugger and profiler is available at: http://<IP_ADDRESS>:8181/IDGi_8oYxNc=/devtools?uri=ws://<IP_ADDRESS>:8181/IDGi_8oYxNc=/ws
[hotreload] Hot reload is enabled.
Press either R or r to reload
Unhandled exception:
SocketException: Failed to create server socket (OS Error: Failed to start accept), address = ::, port = 8080
#0      _NativeSocket.bind (dart:io-patch/socket_patch.dart:1032:7)

#1      _RawServerSocket.bind. (dart:io-patch/socket_patch.dart:1812:15)

#2      _ServerSocket.bind. (dart:io-patch/socket_patch.dart:2097:15)

#3      _HttpServer.bind. (dart:_http/http_impl.dart:3186:27)

#4      serve (package:shelf/shelf_io.dart:60:16)

#5      withHotreload. (package:shelf_hotreload/src/with_hotreload.dart:81:21)

#6      withHotreload (package:shelf_hotreload/src/with_hotreload.dart:102:3)

[process] Server process has been terminated
StdinException: Error setting terminal line mode, OS Error: 句柄无效。
, errno = 6
PS D:\DartProj\my_project> dart_frog dev
✓ Running on http://localhost:8080 (0.2s)
The Dart VM service is listening on http://<IP_ADDRESS>:8181/F3K2U8HRi0Q=/
The Dart DevTools debugger and profiler is available at: http://<IP_ADDRESS>:8181/F3K2U8HRi0Q=/devtools?uri=ws://<IP_ADDRESS>:8181/F3K2U8HRi0Q=/ws
[hotreload] Hot reload is enabled.
Press either R or r to reload
Unhandled exception:
SocketException: Failed to create server socket (OS Error: Failed to start accept), address = ::, port = 8080
#0      _NativeSocket.bind (dart:io-patch/socket_patch.dart:1032:7)

#1      _RawServerSocket.bind. (dart:io-patch/socket_patch.dart:1812:15)

#2      _ServerSocket.bind. (dart:io-patch/socket_patch.dart:2097:15)

#3      _HttpServer.bind. (dart:_http/http_impl.dart:3186:27)

#4      serve (package:shelf/shelf_io.dart:60:16)

#5      withHotreload. (package:shelf_hotreload/src/with_hotreload.dart:81:21)

#6      withHotreload (package:shelf_hotreload/src/with_hotreload.dart:102:3)

[process] Server process has been terminated
StdinException: Error setting terminal line mode, OS Error: 句柄无效。
, errno = 6
PS D:\DartProj\my_project> ^C
PS D:\DartProj\my_project> ^C
PS D:\DartProj\my_project> ^C
PS D:\DartProj\my_project> ^C
PS D:\DartProj\my_project> dart_frog dev
✓ Running on http://localhost:8080 (0.3s)
The Dart VM service is listening on http://<IP_ADDRESS>:8181/AhhHKro5uoo=/
The Dart DevTools debugger and profiler is available at: http://<IP_ADDRESS>:8181/AhhHKro5uoo=/devtools?uri=ws://<IP_ADDRESS>:8181/AhhHKro5uoo=/ws
[hotreload] Hot reload is enabled.
Press either R or r to reload
Unhandled exception:
SocketException: Failed to create server socket (OS Error: Failed to start accept), address = ::, port = 8080
#0      _NativeSocket.bind (dart:io-patch/socket_patch.dart:1032:7)

#1      _RawServerSocket.bind. (dart:io-patch/socket_patch.dart:1812:15)

#2      _ServerSocket.bind. (dart:io-patch/socket_patch.dart:2097:15)

#3      _HttpServer.bind. (dart:_http/http_impl.dart:3186:27)

#4      serve (package:shelf/shelf_io.dart:60:16)

#5      withHotreload. (package:shelf_hotreload/src/with_hotreload.dart:81:21)

#6      withHotreload (package:shelf_hotreload/src/with_hotreload.dart:102:3)

[process] Server process has been terminated
StdinException: Error setting terminal line mode, OS Error: 句柄无效。
, errno = 6

HI @Chenlin2020-center thank you for opening this issues 💙 !
Do you mind outlining the steps we need to to follow to reproduce this issue?

HI @Chenlin2020-center thank you for opening this issues 💙 !嗨，感谢您打开本期💙！
Do you mind outlining the steps we need to to follow to reproduce this issue?您是否介意概述重现此问题所需遵循的步骤？

Thank you, the issue has been resolved. It was a localhost problem. Localhost was not mapped to <IP_ADDRESS>.


More robust link-updating

Fixes #170

Moves links to reference lists to make them easier to change.
Makes links relative to branch you are on - doesn't hard code to main
Updates find/replace to use reference lists
Makes sure that including file sections also includes references
Testing/fixing of references/links working

This is bc some docs like readme, CLA, contributors and contributing exist
in base GitHub folders rather than docs...but get read in and used in docs
so we don't have to duplicate info.
I agree it isn't ideal. I did do templating with the GitHub repo...
On Thu, Nov 9, 2023, 6:23 AM Joey Reid @.***> wrote:

@.**** commented on this pull request.
I like the reference lists, but I don't understand the need for
replace_links_in_markdown, if these are all relative links, and they are,
shouldn't it just work? Why does CLA.md need to be replaced with the full
path to the repo and branch instead of just letting GH resolve the relative
link?
In main.py
https://github.com/TIDES-transit/TIDES/pull/181#discussion_r1388043285:

@@ -10,18 +10,80 @@

log = logging.getLogger("mkdocs")
-FIND_REPLACE = {  # original relative to /docs : redirect target

"<CONTRIBUTING.md>": "Contributing Section",
"(CODE_OF_CONDUCT.md)": "(development/#tides-code-of-conduct)",
"(CLA.md)": "(development/#tides-contributor-license-agreement)",
"CONTRIBUTING.md)": "development/)",
"": "LICENSE",
"contributors.md)": "development/#contributors)",
"architecture.md)": "architecture)",
"tables.md": "tables",
"https://tides-transit.github.io/TIDES/governance/":"governance.md"
+BRANCH_NAME_KEYWORD = "{branch_name}"
+GITHUB_REPO = f"http://github.com/TIDES-transit/TIDES/tree/{BRANCH_NAME_KEYWORD}"

+# targets for these link keys will be updated upon doc build
+UPDATE_LINKS = {
What happens if you define a new link in a markdown document, but don't
add it to this dict? If I understand correctly, to update docs you'd have
to update this code as well? I don't love that, maybe we could move the
templating into the markdown, so that link definitions go from
[CLA]:CLA.md to [CLA]:{GITHUB_REPO}/CLA.md? Or am I misunderstanding?
In main.py
https://github.com/TIDES-transit/TIDES/pull/181#discussion_r1388055028:

"architecture.md)": "architecture)",

"tables.md": "tables",
"https://tides-transit.github.io/TIDES/governance/":"governance.md"
+BRANCH_NAME_KEYWORD = "{branch_name}"
+GITHUB_REPO = f"http://github.com/TIDES-transit/TIDES/tree/{BRANCH_NAME_KEYWORD}"

+# targets for these link keys will be updated upon doc build
+UPDATE_LINKS = {

"[architecture]":"./architecture",
"[table schemas]":"./tables",
"[contributors]":"./development.md#contributors",
"[contributors.md]":"./development.md#contributors",
"[contributing]": "./development",
"[code of conduct]": "development#code_of_conduct",
"[CLA]":f"{GITHUB_REPO}/CLA.md",
"[license]": f"{GITHUB_REPO}/LICENSE)",

is there supposed to be a ) in this format string?
In main.py
https://github.com/TIDES-transit/TIDES/pull/181#discussion_r1388074584:

@@ -10,18 +10,80 @@

log = logging.getLogger("mkdocs")
-FIND_REPLACE = {  # original relative to /docs : redirect target

"<CONTRIBUTING.md>": "Contributing Section",
"(CODE_OF_CONDUCT.md)": "(development/#tides-code-of-conduct)",
"(CLA.md)": "(development/#tides-contributor-license-agreement)",
"CONTRIBUTING.md)": "development/)",
"": "LICENSE",
"contributors.md)": "development/#contributors)",
"architecture.md)": "architecture)",
"tables.md": "tables",
"https://tides-transit.github.io/TIDES/governance/":"governance.md"
+BRANCH_NAME_KEYWORD = "{branch_name}"
+GITHUB_REPO = f"http://github.com/TIDES-transit/TIDES/tree/{BRANCH_NAME_KEYWORD}"

Why do we need GITHUB_REPO? Shouldn't ./LICENSE work instead of
https://github.com/TIDES-transit/TIDES/tree/main/LICENSE? Or does that
not work?
—
Reply to this email directly, view it on GitHub
https://github.com/TIDES-transit/TIDES/pull/181#pullrequestreview-1722606861,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGCBI4DAGLE2OVZ4HN4DDDYDTRNJAVCNFSM6AAAAAA7DR24I6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTOMRSGYYDMOBWGE
.
You are receiving this because you were assigned.Message ID:
@.***>


Update license copyright

Description
Fixes the wrong copyright sentence.

@CyonAlexRDX Didn't even noticed that you created a PR #629. I am closing this.


fix: avoid blocking the send queue when recovering controller from missed Send Data callback fails

fixes: https://github.com/zwave-js/node-zwave-js/issues/6469

@zwave-js-bot automerge


Update README.md, UserGuide.md, DeveloperGuide.md

Closes #20, closes #103

Closes #104
This PR does the following to DG:

Class diagram for Person class is updated
addalt and edit features are added under implementation
Prefix summary is added

Noticed issue on markbind style errors, created #107 to address this issue


Flutter Web

Question, this package works at Flutter Web?

No it doesn't. But the forked version is probably going to get flutter web support, it's nbrg_pdf_viewer_flutter

Thanks!

You're welcome! ✌🏻😃


Automated cherry pick of #6628

Cherry pick of #6628 on release-5.29.
/cc  @jufab

Will try to auto merge this PR once all tests and checks are passing. This might take up to an hour.

Trying to auto merge this PR.

Pull Request successfully merged
SHA: 447ee3dba2cda4b539cecb2c8c36b1483008617f


ARIMA order issue - d should always be less than 1

For ARIMA model it doesn't accept d greater than 0
This code:
from verticapy.machine_learning.vertica.tsa import ARIMA, AR, ARMA, MA
model = ARIMA(name='arima',order = (2,1,2))
model.drop()
model.fit(input_relation=vdf,ts = 'Time',y='Value', test_relation=vdf_test)
pred = model.predict(vdf=vdf_test,ts='Time', y='Value')

gives this error:

This is because you have the wrong Vertica version. ARIMA is available since 23.4


Transformation during indexing

Hi, great tool!
I've taken the liberty of adding the capability to provide a transformation function for changing the data during reindexing on-the-fly. I believe it addresses issue #54 and #83 in a simple to use way and provides a lot of power to the users (I added it because I needed to change some values during migrating an index due to app changes).
WDYT? Your comments and remarks are most welcome of course.
If you believe it's worthy of adding to the tool, then there can be additional interesting stuff to add afterwards:

Pass metadata to function (src/target info)
Allow batch conversion as well
Allow user to provide a full-blown module as well (so it can include external dependencies and maintain some simple state)
Allow user to transform one record to either 0 (removal) or more than one (split) records
More flexible error handling during transformation (ignore up to X, X%, etc.)

Harel

Coverage increased (+1.8%) to 46.316% when pulling 555337115b91723d5cf777c7f210207f930075c8 on harelba:data-transformation into 7bb66fa4c9244967f4f14d39eb9369c646b9ce4c on mallocator:master.

by the way - I wanted to ask about the Cluster vs NoCluster difference in initializing the workers - Do you think it would be better to just call worker.initialize from the NoCluster implementation instead of partially re-implementing the worker initialization code in the NoCluster class?
The way it's done today forced me to expose the initialize_transform class in the worker (and in tests) instead of encapsulating it in the worker as a hidden part of the impelmentation.
Harel

Hi Harel,
Thanks for contributing. I appreciate you taking the time to add more functionality to the tool. The same question as before still applies though: What can this feature do that you can't do via the normal query API, especially when considering the scripting API?
It seems though that this is a popular feature and even if you can do it via the query API it might just make sense to have this on top of it. I will be adding some comments/questions to the code you might help shed some light on.
Thanks!

As to the best way to implement the worker:
Yes exposing something internal is not great, but the worker implementations do have a parent-child relationship so I think that should be okay. In any case I would want to try and avoid duplicate code... you know, the whole DRY KISS thing.

Hi,
Thanks for taking the time to review my solution. I'll reference the specific points separately, but I do want to try and explain the higher-level reasons behind this kind of feature in my view (sorry for the long reply..).
I believe that the main underlying reason is the fact that the tool is being used (by me and I guess by other people) as a migration tool and not just an export tool. This means that it can be used to convert some index/type from version X to version X+1. In many cases, this means that application requirements/code (which writes the data) has changed and there are several options:

Deleting the entire dataset since it no longer matches the application logic (in order to prevent inconsistencies etc.). Obviously, not a good solution in most cases :)
Having the application code contain logic for both the old and new version. Extremely complicates app code, very error prone, the data in ES does not mean anything anymore without the app itself, and sometimes it's not a viable solution (depending on the scenario).
Reindexing the data to another index (*_v2), applying the relevant changes to the data during reindexing, and moving the app to work with _v2 afterwards. This leaves the data model very clean, the app remains clean in terms of code, very minor room for errors (since the data in the new index can be checked before moving the app to work with it). It also keeps the data much more "reusable", since other apps can access the data directly for new use-cases, and not only through the original app (as in option 2).

I think that the function-based solution allows to implement option 3 above very easily.
Regarding the comparison between the query API solution and the function-based solution:

Using the query API for this means that the transformation cannot be validated and tested before actually running on the cluster. The function-based solution can be easily tested separately.
The function-based solution provides much more expressiveness and power than the scripting API. I'll give two specific examples which I've used:

One of the fields in each doc was a complex hash based on several other fields. The hash calculation has been changed and is now based on an additional field. The hash calculation itself is very complex and required using a library which calculates sha256 encoding.
Converting a value of a field to another value, depending on a mix of several other values (This reflected a change in the app, due to a bug in the original code). Using function-based if() statements made this a very easy task.

The query API is very complicated to use relative to a simple javascript function
Performance - Using the scripting API means that the transformation will happen inside the cluster itself, impacting its performance. The function-based means that the transformation happens in an external process/machine and not in the ES cluster, preventing unneeded load and GC on the cluster. Script expressions in ES are specifically considered heavy operations.
Separation of concerns - Using the query API means mixing between choosing which documents need to be migrated/exported (a "source" concern) and how to modify the documents (a "target" concern). The function-based solution separates the two (it kind of adds another optional concern of "transform" between the "source" and "target" concerns).
Not ES-specific - Using the query API for this type of changes means that it won't work for source drivers other than ES. The function-based solution can be used in all source/target driver combinations. This is actually a specific incarnation of the separation of concerns point above.

Hope that this answers your original question. I'll be happy to provide any more information if needed.
Harel

Alright, I was initially against it, but you have me convinced.
Thanks for the patch!

Thanks a lot! That's awesome!
Please reference me on any issues being open related to this feature, and I'll be happy to comment and fix them if needed.
Harel


$env:MSI_SECRET

Great write-up. I’ve got the request posting to the queue and the queue trigger working. The managed identity is giving me a problem, first time working with it. Is there a post on how to set it up including creating the secret etc? I set the MSI_SECRET in application settings to a password, but cannot tell if that line in the profile.ps1 is being executed. I added the Connect-AzAccount -Identity in different places and get varying errors about sockets and Connect-AzAccount not being found.
I'll try using Get-Module to see if the Az module is being installed. Is there a sanity check to see it that actually happened? Through Kudu?

Useful link. Got the example up and running.
https://docs.microsoft.com/en-us/azure/app-service/overview-managed-identity


Updatifier support

In principle, sounds good. However, it's use is currently limited, and it would be nice to be able to have beta channels.
How do we use it? I guess we might need to choose a tagging format before we use it.

We can use it when we have a release system. Are we intending on using GitHub for releases or were we going to use something else, such as Jenkins?

I have a Jenkins server for CI builds (but may switch to TeamCity, dunno yet), but I was thinking GitHub releases as the actual releases.

@dualspiral Sounds good, in that case we need to agree on a tagging system and then we are good to use Updatifier.

@dualspiral @hsyyid do note that the tag shouldnt have v in it if we want jitpack to work properly. Learnt that it doesnt work properly when a release tag doesnt start with a number.
This can be seen at https://jitpack.io/#hsyyid/EssentialCmds/v7.6d where 0.4 is the last successful auto deployed version, since after that version hsyyid switched to using vx.xc format for release tag.

I don't think we will do this, we've seen little work on Updatifier, and I think we'll be able to integrate with Ore eventually.


TLS Required?

FTP works great, no issues really besides it is a little slow (unexpectedly compared to SMB) even when doing FTP within a LAN.
(It seems Random File Access is slow for FTP). But I am okay with this.
This is not an issue per se, so even if it was deemed as no fix, it is okay. It is just to bring this to your attention.
I have TrueNAS server, and setup FTP service. By default in TrueNAS FTP settings, there is a setting called
TLS Policy, by default, this settings is set to On.

And when it is set to On or anything except Off and Auth. Cifs Document Provider fails to list any files and directories.
This is not a major issue, I personally set it to Auth so the requirement for TLS encryption happens during authentication only.
But I thought that might be cool if it is also enforced while doing data transfer too.
Here is more information about TLS Required command: http://www.proftpd.org/docs/contrib/mod_tls.html#TLSRequired
This setting according to proftpd should be Off by default. But somehow TrueNAS has it On.
If I set it to On, I used FileZilla on my PC and it is able to see the files and directories, File Manager + android app is able to see them too.
I want to thank you @wa2c for implementing FTP/FTPS support. What a wonderful developer 🙂
This is wonderful, now I can access my NAS even when I am not home.
Heck, I can play PS2 and PSP games on the go too 👍🏻

I can't find a way to fix this problem in code.
But I found that by changing the following check on in TrueNAS, the files are visible
Services > FTP > General Options > ADVANCED OPTIONS > TLS No Session Reuse Required

However,  this settings help reads as follows...

Setting this option reduces the security of the connection, so only use it if the client does not understand reused SSL sessions.

It seems to SSL/TLS session reusing is not supported now.
[NET-408] problem connecting to ProFTPD with FTPES - ASF JIRA

@RebelliousX
Please hold off on this problem until Apache Commons supports session reuse.

@wa2c Thank you very much for your work, I really appreciate it. TLS No Session Reuse fixes the problem for now. I am okay with this.


Escaping removes original cursor highlight in Gvim

In Gvim, I used ESC to leave the multiple cursors mode, which it did. The ruler shows that the cursor was able to move around. However, the cursor was not highlighted in Gvim. I can reset the cursor by refreshing the file (:e). I tested this issue in terminal vim with no problem whatsoever.

Duplicate of #201

Unable to reproduce. Closing issue.


Code error | example_stiffness

Dear Dashdotrobot,
I came across your code and love the work. Currently I'm trying to fully understand the code, which is quite a challenge. I have completed by BSc. in aerospace engineering and an currently doing a Msc. I'm quite proficient in Matlab but Python is quite new for me. The examples are helpful but I can't get example_stiffness.py to run properly. I get an error:
Traceback (most recent call last):
File "\example_stiffness.py", line 46, in
fem = bc.BicycleWheelFEM(g, r_sec, s_sec)
File "C:\bikewheelfem.py", line 705, in init
offset = geom.lace_offset[e]
IndexError: index 0 is out of bounds for axis 0 with size 0
Could you indicate to as what the issue might be?
Thank you for sharing the code!
Best regards,
Matthijs

Hi Matthew,
Thanks for your reply. I indeed ran the code as-is, and encountered the
problem. I will try it again later, thank you for fixing the error!
Currently I am doing an internship with a company that designs and sells
composite road race wheels. I was interested to learn a bit more on the
effect of certain spoke patterns.
Furthermore, I had the idea of building a code in which I could include a
model of a composite rim and vary spoke count/stiffness/pattern to see what
would give the optimal stiff wheel. I am currently trying to set up an
input file for Abaqus, which I later want to convert to an easy adjustable
input which can easily test different configurations.
Quite a challenge though. But it is nice to see you have actually succeeded
in generating the code!  Did you verify it against any actual wheels?
Best regards,
Matthijs
2016-07-22 20:28 GMT+02:00 Matthew Ford<EMAIL_ADDRESS>
Closed #4 https://github.com/dashdotrobot/bike-wheel-calc/issues/4.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dashdotrobot/bike-wheel-calc/issues/4#event-732109145,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATlZ2cLHb2JQlS2ZcV5ICy2WbkEUo928ks5qYQvmgaJpZM4JPk7I
.


docs(badge): add docs for badge component

:tada: This PR is included in version 0.2.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


fix: changeLocation error occurred when I use Vue and React in QianKun

issue
[Bug] changeLocation error occurred when I use Vue and React in QianKun (a micro frontend frame)

I use QianKun as the micro front-end framework. The main app is based on Vue3 and<EMAIL_ADDRESS>and one of the sub apps is based on React16. The React sub app is loaded when I navigate to the route path "/mfe-react-center/page2". And then i will click a button, by doing so, the app will navigate to another page of the React sub app, and the route path is "/mfe-react-center/page3". Meanwhile, the vue-router of the main app catched the popstate event, and the popStateHandler will get the parameter state, but the data structure of the parameter state does not meet the requirements of vue-router because the popstate event is triggerred by the React sub app. finally, historyState.value will get an incorrect value.
{
  key: "furyvw",
  state: undefined
}

An error occurred when I try to navigate to another page by clicking the menus of the main app. Instead of the expected http://localhost:3000/mfe-vue-center/page4, we get http://localhost:3000undefined.
Uncaught (in promise) DOMException: Failed to execute 'replace' on 'Location': 'http://localhost:3000undefined' is not a valid URL.

I found the releated code of vue-router：
// source code of push method of vue-router
const currentState = assign(
  {},
  // use current history state to gracefully handle a wrong call to
  // history.replaceState
  // https://github.com/vuejs/vue-router-next/issues/366
  historyState.value,
  history.state,
  {
    forward: to,
    scroll: computeScrollPosition(),
  }
);
changeLocation(currentState.current, currentState, true);
const state = assign({}, buildState(currentLocation.value, to, null), { position: currentState.position + 1 }, data);
changeLocation(to, state, false);

As mentioned above, historyState.value refers to an incorrect value. When I navigate to another page through the menus of the main app, the push method of vue-router is executed, and the currentState gets a value without the property current, and then changeLocation is executed and get a value http://localhost:3000undefined. Obviously, the navigation will be failed.
Therefore, the core of the problem is that the route navigation within the React sub app is monitored by the popStateHandler of vue-router of the main app, but the state passed does not match the data structure expected by vue-router.

Although I could solve this problem by changing all the navigation scenarios of the React sub app to notify the main app and let the main app control the navigation, the change will be very invasive. And I think a lot of people will have the same problem when using micro front end, so I want to solve this problem through PR!

PR
For the problem mentioned in the issue, I analyzed some code execution process of vue-router and got the solution.
Let's analyze the process of initialization and route navigation of vue-router to see how state changes.
Step 1: Initialization analysis
First, when the web application first loads, history.state has no value.

In history mode, the method createWebHistory will be called, and then the method useHistoryStateNavigation will be called. During the execution of method useHistoryStateNavigation, currentLocation and historyState will be initialized。
let currentLocation = {
  value: createCurrentLocation(base, location),
};
let historyState = { value: history.state };

Then we will check the value of historyState.value, and since there is no initial history.state value, we will perform a changeLocation method in replace mode, During the execution of method changeLocation, historyState.value = state will be executed, so historyState has a value.
So the value of historyState.value is going to look something like this:
{
  back: null,
  current: "/mfe-vue-center/page1"
  forward: null,
  position: 30,
  replaced: true,
  scroll: false
}

Step 2: Click the main app menu and navigate by vue-router
The push method will be called for navigation, and the value of parameter to is "/mfe-react-center/page2". During the execution of method push, currentState will be computed, and the value is described below:
{
  back: null,
  current: "/mfe-vue-center/page1",
  forward: "/mfe-react-center/page2",
  position: 30,
  replaced: true,
  scroll: {left: 0, top: 0}
}

Then the first changeLocation method will be executed in replace mode. after the execution, currentState and historyState.value will get the same value.
changeLocation(currentState.current, currentState, true);
const state = assign({}, buildState(currentLocation.value, to, null), { position: currentState.position + 1 }, data);
changeLocation(to, state, false);

vue-router then calculates a new state by buildState. The logic behind buildState is to change the value of back and current. back receives the value of currentLocation.value and current receives the value of the parameter to passed through the push method.
the new value of state is described below:
{
  back: "/mfe-vue-center/page1",
  current: "/mfe-react-center/page2",
  forward: null,
  position: 31,
  replaced: false,
  scroll: null
}

Then a method changeLocation(to, state, false) is performed, and the value of historyState.value will be the same as the value of the new state.

The reason why changeLocation are performed twice is that the first time is mainly to correct the current history.state value to ensure the correctness of the forward property and facilitate subsequent forward operations. The second time is when the real pushState is performed based on the new state (back and current are updated on the second time). I have to say, this is a great design!
Finally, update the value of  currentLocation by currentLocation.value = to.
Step 3: Navigate within the React sub app

The react-router push method is triggered by clicking the button inside the React sub app, hoping to jump to the path "/mfe-react-center/page3". And popStateHandler of vue-router in the main app is then triggered, and the value of the parameter state is described below.
{  key: "paf114",  state: undefined }

historyState.value records the value of state.
Step 4: Click the main app menu and navigate to another page by vue-router
Then I will click the main app menu and navigate to another page. push method will be executed, and the value of currentState will be like this:
{  forward: "/mfe-vue-center/page4",  key: "paf114",  scroll: {left: 0, top: 0},  state: undefined }

Because the currentState.current is undefined, performing the first changeLocation will throw an error. We will get a url with a value "http://localhost:3000undefined", and location.assign will be executed, this will cause the browser to jump to the Error page.
Uncaught (in promise) DOMException: Failed to execute 'replace' on 'Location': 'http://localhost:3000undefined' is not a valid URL.

Therefore, the solution is to verify the validity of the state. If the state is not valid, the valid state should be calculated by other reliable values.
We can hack like this：
if (!state.current) {
  state = assign({}, historyState.value, {
    back: historyState.value.current,
    current: to,
    position: historyState.value.position + 1,
  })
}

The idea is to update the value of back and current attributes of state based on the value of historyState.value.current and to.

Thanks for the detailed description. Ideally, the history.state, especially the properties used by Vue Router, should be preserved while navigating and this should be coordinated by the micro frontend solution:

And popStateHandler of vue-router in the main app is then triggered,

The microfrontend solution should deactivate vue router listener if the vue app is not active as it doesn't need to be ran.
When running a Vue app, the router assumes that it is the one modifying the URL with pushState() and that the state exists and is valid. It only aims to correct the state when the application starts because the router should be the only place all navigation go through.
It's a good idea to make the code more resiliant to invalid state but it should do the strict minimum (e.g. only setting current and maybe forward)

Thanks for the detailed description. Ideally, the history.state, especially the properties used by Vue Router, should be preserved while navigating and this should be coordinated by the micro frontend solution:

And popStateHandler of vue-router in the main app is then triggered,

The microfrontend solution should deactivate vue router listener if the vue app is not active as it doesn't need to be ran.
When running a Vue app, the router assumes that it is the one modifying the URL with pushState() and that the state exists and is valid. It only aims to correct the state when the application starts because the router should be the only place all navigation go through.
It's a good idea to make the code more resiliant to invalid state but it should do the strict minimum (e.g. only setting current and maybe forward)

thank you for your reply. But what I mean is that the main app is always active in the whole web application lifecycles. and the sub apps are parts of the web application.

I will then provide a codesandbox demo for reproduction.

I gave this more thought and I don't think this fix is enough because it will still trigger the navigation guards with invalid and unrecognized routes.
Basically, the router should not listen to popstate events coming from outside of the Vue app, and that's something that needs to be adapted in the micro frontend library. For example, they could have a custom History implementation that handles that case.
If anything else is missing in vue router code and it's reasonable to add it, we can add it.


[Edge Computing] Taking advantage of GPU to increase performance in exaflop cloud computing to instantiate big data services in an optimal manner.

.### Topic Suggestion
[Edge Computing] Taking advantage of GPU to increase performance in exaflop cloud computing to instantiate big data services in an optimal manner.
Proposed title of article
Taking advantage of GPU to increase performance in exaflop cloud computing to instantiate big data services in an optimal manner.
Proposed article introduction
As the Edge Computing world evolve in terms of new technology, every company, governmental entities, information data centers and even in agricultural revolution considers data as a new and extremely useful phenomenon for their existence. Modern supercomputers and clouds have proved  they can successfully work with petabyte data sets, however, giving a try to  exabyte size datasets  had tend to be a bit difficulty to some users, since high performance and high bandwidth is required to transfer and process such huge volumes of data over the network.
Previously big data were mostly established purposely  for storing the data and applying some basic analytics modules. Today and in future, as the practice is evolving, there is a need to adopt more advanced newly emerging software and systems for analyzing commercial and increase operational performance to coincide the user needs.
This article provides steps on how to fully utilize the high performance of Big Data services offered by cloud computing technology taking into consideration the use of GPU instead of CPU to control the number of resources and data links as its known as a rule of the thumb that Cloud computing absorb massive data into distributive storage and ensures data reliability by redundant storage.
Key takeaways
-Brief understanding of cloud computing and Big Data.
-Reduce cost and maximize performance
-Dive into GPU
Article quality
This article will be unique in such that cloud scalability and cloud elasticity as a factor in cloud computing and in particular regarding big data management systems  will be fully covered as the current systems posted hardly handle data peaks automatically. Most of the time, cloud scalability is triggered manually rather than automatically and the state-of-the-art of automatic scalable systems shows that most algorithms are reactive or proactive and frequently explore scalability from the perspective of better performance. However, a proper scalable system as proposed in this article would allow both manual and automatic reactive and proactive scalability based on several dimensions such as security, workload rebalance  and redundancy for availability.
References
N/A

I note that one sentence takes the whole paragraph.

There are some significant readability challenges in your description that raise concerns over how much review/editing would be required from our peer review team. We don't provide editing services and recommend that students leverage online editing services (there are both free and paid services that you can explore), or try engaging with the community or other peers in your network to have them review prior to submitting a revised proposal. Thank you.


Upgrade the docker version to 23.0.6

Description
The docker version of the GitHub action runner is 23.0.6.
Instead of downgrading the engine version I want to give a shot to upgrade the client version.
Related issue and scope

[x] I opened an issue to propose and discuss this change (#5433)

My changes affect the following components

[ ] API
[ ] Controller
[ ] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[ ] Scheduler
[x] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[ ] Tests
[ ] Deployment
[ ] CLI
[ ] General tooling
[ ] Documentation

Types of changes

[ ] Bug fix (generally a non-breaking change which closes an issue).
[x] Enhancement or new feature (adds new functionality).
[ ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[ ] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

Codecov Report

Merging #5436 (34be7c3) into master (9371d62) will decrease coverage by 0.12%.
The diff coverage is 100.00%.

:exclamation: Current head 34be7c3 differs from pull request most recent head 612ade0. Consider uploading reports for the commit 612ade0 to get more accurate results

@@            Coverage Diff             @@
##           master    #5436      +/-   ##
==========================================
- Coverage   76.69%   76.57%   -0.12%
==========================================
  Files         241      241
  Lines       14632    14632
  Branches      616      616
==========================================
- Hits        11222    11205      -17
- Misses       3410     3427      +17

Files Changed
Coverage Δ

...sk/core/containerpool/docker/DockerContainer.scala
98.79% <ø> (ø)

...enwhisk/core/containerpool/docker/RuncClient.scala
100.00% <100.00%> (ø)

... and 6 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Since upgrading the docker version could impact many downstream, I think it would be better to separate the change to upgrade docker and the change to maximize the build space for multi-runtime tests.

Once this PR is merged, I would rebase this PR.

It turns out that this change is not directly related to the failure of multi-runtime tests.
So it's not that urgent now but it is still worth discussing.

LGTM this is very long overdue, are there any blockers to merging this. I believe this should only affect docker client right anyways?

I have been thinking of running a performance test with this version.
And last time we upgraded the docker version, some downstream like IBM needed some time to prepare for taking the change in.
I thought that a similar thing might happen.

This PR has opened for a while, I would merge this.


WSL Shutdown problem

Version
Microsoft Windows [Version 10.0.19043.1288]
WSL Version

[X] WSL 2
[ ] WSL 1

Kernel Version
5.10.16
Distro Version
Kali Linux 2021.3
Other Software
No Docker.
Repro Steps
This is a duplicate of the issue 6878, as I am facing the same problem.
First I installed it with Win-Kex, but I reinstalled in order to check if Kex was doing it, but it did not solve the problem.

Run WSL
Open Task Manager
Observe vmmem is running
Shutdown WSL (wsl --shutdown) or (wsl --terminate kali-linux)
VMMEM shutdown for a while then run again after a few seconds
check wsl (wsl -l -v)
WSL is running again.

The only distro this happens to is Kali. Ubuntu works normally.
I tried clearing my folder history as suggested.
Expected Behavior
WSL to shutdown and Vmmem to not be in the process list.
Actual Behavior
Vmmem restarting with Kali Linux running after shutting it down.
Diagnostic Logs
This is a wsl.etl file, but i was not allowed to upload it as such.
wsl.txt

I have the exact same issue and it's really annoying, I don't have docker either and I tried clearing the cache. Also this is happening for me with a Debian distro.

Same problem here too, but removing kex from kali using the purge command and a Windows restart  "fixed" it in my case. I noticed that when kex was still installed, after closing kali, in Windows 11 once in a while seemed that the focus of open windows in the desktop changed. Maybe kex does not close correctly and then it tries to access some file in the kali fs and so it causes kali to start again?

Ok I also have the same problem but I never installed Kex and someone suggested I enable debugShell so once I did, I noticed other than it restarting itself after running wsl --shutdown it also restarts itself after switching to different applications I also disabled my av software while testing and av software wasn't the cause
Also, the restart doesn't happen when you quickly switch between applications but when like when you stay on one app for a few mins and then switch to another one

@Iskers @KiKoS0 @MRColorR Do any of you guys happen to have ArmouryCrate installed?
cause I asked about this on superuser.com and with the help of this answer https://superuser.com/a/1746987/229729 found out that ArmouryCrate.UserSessionHelper.exe process for some reason is trying to create a file at this path \\wsl.localhost\kali-linux\usr\lib\win-kex\win-kex

In my case I didn't have ArmouryCrate installed. But I agree that other similar apps may have the same behavior.

Yeah, you might be able to find out which process does it like searching for strings in the memory for wsl.localhost maybe

This problem occurs only in new versions of wsl.
On default version https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi
it shutdowns perfectly

faced the same issue with wsl ubuntu.
any fix or preferred workaround for this?


Api Optimize

Add task: support multiple tasks
add get_output_data api
task.hosts: support add common pamameters to all hosts

Already support with 2.0.0-beta, closed


Better API document about status/header in  Response

most time we need to set both header/status and  content,  i spend much time to findout how to set a json response with content-type header  --  use the "with()", please document it out.
server.request(by("foo")).response(header("content-type", "application/json"));

the real sample should has the json content here.
PS. now we has the json shortcut. But i still need to set bot status code(500) and error  reason text.

Then what about response(seq("status 500 with error message", "status 200 with result))?

server.response(seq(and(status(500), with(text("Error Message"))), and(status(200), with(text("Result")))));


Show alert/warning when host is out of date

Once we introduce a new version of the functions host with breaking changes, existing function apps have to be manually upgraded. The UI needs to show some prompt so that users are at least aware that the new version exists and how to upgrade to it.

Assigning to @mattchenderson for initial UI design. We need to find the right balance of making it visible enough, but without it getting in the way if they want to stick to old version.
Also, it should have a button that actually fixes it up. We can ask users to set some obscure app setting to some weird tilde value. Note that all current users will see this, so we need to make it good. It's not an edge case.

Included in 4/26 mocks

428f5b43e4ca88daf8b1058d095a9bad150f957c


T-Pose when sprinting backwards or sideways while locked on to an enemy

This is more than likely related to Movement Behavior Overhaul, I have yet to perform any real diagnosis of this issue. Will be worked on for the next version!
After some diag, I've further identified when exactly this issue occurs. Working further on fixing it.
The problem can be fixed quickly by jumping in the meantime. It does not occur very often.

Movement Behavior Overhaul has been removed to fix this issue. Most/all of it's features are implemented by other mods anyways, which I was none the wiser to really.


Put rancher-metadata to default section of answers.json

So standalone containers can also resolve it

LGTM


improve efficiency of sample swap detection

A couple ideas:

count a subset of k-mers
rolling k-mers

Or throw out k-mers altogether and switch to somalier.


Assertion error on multi image (batch) testing

Thanks for your error report and we appreciate it a lot.
Checklist

I have searched related issues but cannot get the expected help.
The bug has not been fixed in the latest version. (Using MMDetection 2.4.0)

Describe the bug
The following error is displayed when evaluating retinanet on batch-size of 2 per gpu. Works as expected with batch-size 1.
assert features.dim() == 4 and features.size(0) == 1
AssertionError
File : mmcv/lib/python3.7/site-packages/mmcv/ops/masked_conv.py", line 31, in forward
Reproduction

What command or script did you run?

CUDA_VISIBLE_DEVICES=0,1,2,3 tools/dist_test.sh my_retinanet.py \ epoch_1.pth 4 --eval bbox

Did you make any modifications on the code or config? Did you understand what you have modified?
Yes. I added the following line after line 148 in tools/test.py
samples_per_gpu = 2
What dataset did you use?
Custom MSCOCO Dataset.
Environment
Python 3.7 with Pytorch 1.4.0 GPU
Please run python mmdet/utils/collect_env.py to collect necessary environment information and paste it here.
sys.platform: linux
Python: 3.7.0b3 (default, Mar 30 2018, 04:35:22) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda-10.1
NVCC: Cuda compilation tools, release 10.1, V10.1.243
GPU 0,1,2,3: GeForce GTX 1080 Ti
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.4.0
PyTorch compiling details: PyTorch built with:

GCC 7.3
Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CUDA Runtime 10.1
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.3
Magma 2.5.1
Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.5.0
OpenCV: 4.4.0
MMCV: 1.1.1
MMDetection: 2.4.0+11b1ef8
MMDetection Compiler: GCC 7.3
MMDetection CUDA Compiler: 10.1

You may add addition that may be helpful for locating the problem, such as

How you installed PyTorch [e.g., pip, conda, source]
Using pip
Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Error traceback
If applicable, paste the error trackback here.
Traceback (most recent call last):
  File "../tools/test.py", line 206, in <module>
    main()
  File "../tools/test.py", line 186, in main
    args.gpu_collect)
  File "/home/uttaran/Small_Object_Detection/mmcv/mmdetection/mmdet/apis/test.py", line 96, in multi_gpu_test
    result = model(return_loss=False, rescale=True, **data)
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/uttaran/Small_Object_Detection/mmcv/mmdetection/mmdet/core/fp16/decorators.py", line 51, in new_func
    return old_func(*args, **kwargs)
  File "/home/uttaran/Small_Object_Detection/mmcv/mmdetection/mmdet/models/detectors/base.py", line 170, in forward
    return self.forward_test(img, img_metas, **kwargs)
  File "/home/uttaran/Small_Object_Detection/mmcv/mmdetection/mmdet/models/detectors/base.py", line 147, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "/home/uttaran/Small_Object_Detection/mmcv/mmdetection/mmdet/models/detectors/single_stage.py", line 112, in simple_test
    outs = self.bbox_head(x)
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/uttaran/Small_Object_Detection/mmcv/mmdetection/mmdet/models/dense_heads/guided_anchor_head.py", line 243, in forward
    return multi_apply(self.forward_single, feats)
  File "/home/uttaran/Small_Object_Detection/mmcv/mmdetection/mmdet/core/utils/misc.py", line 54, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/home/uttaran/Small_Object_Detection/mmcv/mmdetection/mmdet/models/dense_heads/ga_retina_head.py", line 107, in forward_single
    cls_score = self.retina_cls(cls_feat, mask)
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/mmcv/ops/masked_conv.py", line 110, in forward
    self.padding)
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/mmcv/ops/masked_conv.py", line 31, in forward
    assert features.dim() == 4 and features.size(0) == 1
AssertionError
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/uttaran/Small_Object_Detection/mmcv/bin/python', '-u', '../tools/test.py', '--local_rank=3', 'retinanet.py', 'epoch_1.pth', '--launcher', 'pytorch', '--eval', 'bbox']' returned non-zero exit status 1.

Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

  File "/home/uttaran/Small_Object_Detection/mmcv/lib/python3.7/site-packages/mmcv/ops/masked_conv.py", line 31, in forward
    assert features.dim() == 4 and features.size(0) == 1

Just as the error suggests, masked_conv doesn't support the case when batch size > 1.


How do i discard a event in a eventHandlder

disruptor.handleEventsWith(a).then(b).then(c);
private void b(CommandEvent event, long sequence, boolean endOfBatch) {
if(xxx){
discard this event
i dont want  c to hand this event
}
}

I think this question was already answered in a previous issue. The feature is not supported by the Disruptor API but you can easily set event.Discarded = true; in b and ignore discarded events in c.

This is the correct approach.


fix: make preview use correct buffer

I wonder then, what is the use of the preview buffer? Is it just when inccommand=split?

Yes... With the change I made though, it won't use the preview buffer even when inccommand=split (since the preview function returns 1 and not 2)

Oh okay, I get it after re-reading the docs, thanks then !


[DROOLS-1135] fix race conditions in OTN and sink propagator intialization

Merged by https://github.com/droolsjbpm/drools/compare/89cc437...c94053d


remote executor (Linux) ends up with macOS binary from select

When running the code in https://github.com/NoRedInk/noredink-ui/pull/1394, running buck2 build //:docs.json fails on the remote executor because it ends up with a Mach-O format executable instead of a Linux one. I'm not sure how this could be happening! The relevant code for downloading the file is in prelude-nri/elm/BUCK, and looks like this:
ELM_COMPILER_URL = select({
    "config//os:linux": "https://github.com/elm/compiler/releases/download/0.19.1/binary-for-linux-64-bit.gz",
    "config//os:macos": "https://github.com/elm/compiler/releases/download/0.19.1/binary-for-mac-64-bit.gz",
})

ELM_COMPILER_SHA256 = select({
    "config//os:linux": "e44af52bb27f725a973478e589d990a6428e115fe1bb14f03833134d6c0f155c",
    "config//os:macos": "05289f0e3d4f30033487c05e689964c3bb17c0c48012510dbef1df43868545d1",
})

http_file(
    name = "elm_compiler_archive",
    urls = [ELM_COMPILER_URL],
    sha256 = ELM_COMPILER_SHA256,
)

genrule(
    name = "elm_compiler_binary",
    cmd = "gzip --decompress --stdout --keep $(location :elm_compiler_archive) > $OUT && chmod +x $OUT",
    out = "elm",
    executable = True,
    remote = True,
    visibility = ["PUBLIC"],
)

If I copy that code to my root BUCK and run it with the following target, the right binary is downloaded and I can get the version in a text file just fine.
genrule(
    name = "elm_version",
    out = "version.txt",
    cmd = "$(exe :elm_compiler_binary) --version > $OUT",
    remote = True,
)

At @krallin's request, I'm opening a new issue for this rather than continuing discussion in #249 including the output of buck2 audit configurations and buck2 audit execution-platform-resolution //:docs.json. Here they are:
$ buck2 audit configurations
prelude//platforms:default#6553ba90a552c2c0:
  prelude//cpu/constraints:x86_64                             (prelude//cpu/constraints:cpu)
  prelude//os/constraints:macos                               (prelude//os/constraints:os)
toolchains//:platforms#d532bb5d6580e1e1:
  prelude//cpu/constraints:x86_64                             (prelude//cpu/constraints:cpu)
  prelude//os/constraints:linux                               (prelude//os/constraints:os)

$ buck2 audit execution-platform-resolution //:docs.json
root//:docs.json (prelude//platforms:default#6553ba90a552c2c0):
  Execution platform: toolchains//:platforms
    Execution platform configuration: toolchains//:platforms#d532bb5d6580e1e1
    Execution deps:
    Toolchain deps:
      toolchains//:elm (prelude//platforms:default#6553ba90a552c2c0) (toolchains//:platforms#d532bb5d6580e1e1)
      toolchains//:python (prelude//platforms:default#6553ba90a552c2c0) (toolchains//:platforms#d532bb5d6580e1e1)

and just for comparison's sake, here's the one that works:
$ buck2 audit execution-platform-resolution //:elm_version
root//:elm_version (prelude//platforms:default#6553ba90a552c2c0):
  Execution platform: toolchains//:platforms
    Execution platform configuration: toolchains//:platforms#d532bb5d6580e1e1
    Execution deps:
      prelude//os_lookup/targets:os_lookup (toolchains//:platforms#d532bb5d6580e1e1)
      root//:elm_compiler_binary (toolchains//:platforms#d532bb5d6580e1e1)
    Toolchain deps:
      toolchains//:genrule (prelude//platforms:default#6553ba90a552c2c0) (toolchains//:platforms#d532bb5d6580e1e1)

Nevermind that comment, it's just the diff view that had me confused

This:
elm_toolchain = rule(
    impl = _elm_toolchain_impl,
    attrs = {
        "elm": attrs.dep(
            providers = [RunInfo],
            default = "prelude-nri//elm:elm_compiler_binary",
        ),
        "elm_format": attrs.dep(
            providers = [RunInfo],
            default = "prelude-nri//elm:elm_format_binary",
        ),
        "_isolated_compile": attrs.dep(default = "prelude-nri//elm:isolated_compile.py"),
    },
    is_toolchain_rule = True,
)

Should use attrs.exec_dep, not attrs.dep. Does that fix it?

hey! It does! Fantastic!
I guess exec_dep carries the target platform with it? Is that the difference? Is that why the docs (which I've just consulted) say that it is what you should use if you plan to execute stuff from it?

I think this should answer your questions: https://buck2.build/docs/rule_authors/configurations/
The TL;DR is that exec deps will participate in choosing the exec platforms, and will be configured for the exec platform (Linux in your case), whereas deps are configured for the target (Mac in your case)


Suggestion: Set default tool for certain ADV events

When entering talk scene, set tool to Menu tool for UI interaction
When leaving talk scene, set tool to School tool for movement
Potentially others

Could be done inside the interpretters.