Error when updating to v0.71.3 from 0.70.7

Description
I'm upgrading a very old 0.64 codebase originally created with the typescript template to latest 0.71.3.
To start I've upgraded just to 0.68 because the Android build didn't succeed anymore in 2023 with react-native 0.64: everything went smoothly and I've got a working 0.68 Android build.
At this point I've tried to upgrade to 0.71.3 but it failed, so I've gone through each and every major version up to 0.70.7. I didn't try to build the application with react-native 0.70 yet (this isn't the focus of this bug report), but npx react-native upgrade went smoothly up to 0.70.7.
Unfortunately as soon as I try to upgrade to 0.71.3 I get the following:
$ npx react-native upgrade 0.71.3
warn Package react-native-vector-icons contains invalid configuration: "dependency.assets" is not allowed. Please verify it's properly linked using "react-native config" command and contact the package maintainers about this.
info Fetching diff between v0.70.7 and v0.71.3...
info Applying diff...
warn Excluding files that exist in the template, but not in your project:
  - App.js
  - __tests__/App-test.js
error Automatically applying diff failed. We did our best to automatically upgrade as many files as possible
error Patch failed to apply for unknown reason. Please fall back to manual way of upgrading
info You may find these resources helpful:
• Release notes: https://github.com/facebook/react-native/releases/tag/v0.71.3
• Manual Upgrade Helper: https://react-native-community.github.io/upgrade-helper/?from=0.70.7&to=0.71.3
• Git diff: https://raw.githubusercontent.com/react-native-community/rn-diff-purge/diffs/diffs/0.70.7..0.71.3.diff
error Upgrade failed. Please see the messages above for details.
info Run CLI with --verbose flag for more details.

package.json has already been updated with npx @rnx-kit/align-deps --requirements<EMAIL_ADDRESS>--write as I did with previous updates.
Verbose:
$ npx react-native upgrade 0.71.3 --verbose
warn Package react-native-vector-icons contains invalid configuration: "dependency.assets" is not allowed. Please verify it's properly linked using "react-native config" command and contact the package maintainers about this.
info Fetching diff between v0.70.7 and v0.71.3...
info Applying diff...
warn Excluding files that exist in the template, but not in your project:
  - App.js
  - __tests__/App-test.js
debug "git apply" failed. Error output:
Falling back to direct application...
Falling back to direct application...
Applied patch to '.gitignore' with conflicts.
Applied patch to '.node-version' cleanly.
Applied patch to '.ruby-version' cleanly.
error: App.js: does not exist in index
Applied patch to 'Gemfile' cleanly.
error: __tests__/App-test.js: does not exist in index
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Applied patch to 'android/build.gradle' cleanly.
Applied patch to 'android/gradle.properties' cleanly.
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Applied patch to 'ios/client/AppDelegate.h' cleanly.
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Performing three-way merge...
Applied patch to 'tsconfig.json' with conflicts.

error Automatically applying diff failed. We did our best to automatically upgrade as many files as possible
error Patch failed to apply for unknown reason. Please fall back to manual way of upgrading
info You may find these resources helpful:
• Release notes: https://github.com/facebook/react-native/releases/tag/v0.71.3
• Manual Upgrade Helper: https://react-native-community.github.io/upgrade-helper/?from=0.70.7&to=0.71.3
• Git diff: https://raw.githubusercontent.com/react-native-community/rn-diff-purge/diffs/diffs/0.70.7..0.71.3.diff
error Upgrade failed. Please see the messages above for details

My theory is that this is due to the 0.71 Typescript by default migration. Somehow since my project already used Typescript it fails. Just a theory though.
P.S.
Please also ignore the fact that I'm on ppc64le. I've set up a separate LD_PREFIX for x86_64 binaries and I'm dynamically translating the x86_64 Android SDK (which works like a charm) but this bug report is just about the upgrade process and not the Android build.
React Native Version
0.71.3
Output of npx react-native info
warn Package react-native-vector-icons contains invalid configuration: "dependency.assets" is not allowed. Please verify it's properly linked using "react-native config" command and contact the package maintainers about this.
info Fetching system and libraries information...
System:
    OS: Linux 6.2 Gentoo Linux
    CPU: (32) ppc64 POWER9, altivec supported
    Memory: 48.87 GB / 62.37 GB
    Shell: 5.1.16 - /bin/bash
  Binaries:
    Node: 18.14.2 - ~/.nvm/versions/node/v18.14.2/bin/node
    Yarn: 1.22.19 - /usr/bin/yarn
    npm: 9.5.0 - ~/.nvm/versions/node/v18.14.2/bin/npm
    Watchman: 2<IP_ADDRESS> - /usr/bin/watchman
  SDKs:
    Android SDK:
      API Levels: 29, 31, 32, 33, 33
      Build Tools: 29.0.2, 29.0.3, 30.0.2, 30.0.3, 31.0.0, 33.0.2
      Android NDK: Not Found
  IDEs:
    Android Studio: Not Found
  Languages:
    Java: 11.0.18 - /usr/bin/javac
  npmPackages:
    @react-native-community/cli: Not Found
    react: 18.2.0 => 18.1.0
    react-native: 0.71.3 => 0.70.7
  npmGlobalPackages:
    *react-native*: Not Found

Steps to reproduce
I'd say try to create a 0.64 react-native app with the typescript template and try to upgrade it to 0.71, but I didn't try that yet.
Snack, code example, screenshot, or link to a repository
Unfortunately I cannot share the app I'm currently trying to upgrade. If I will manage to pinpoint the issue to something specific I'll include a repro. In the meantime I'm opening this to get some feedback about the possible causes.

You will have to update your project manually to use TS, if you wish.

My project did already use Typescript, that's why I thought it might have been the issue.

What is the problem of this issue though? Is the app not building or are you just reporting npx react-native upgrade not working well?

npx react-native upgrade doesn't work at all: it silently fails without creating any merge conflict.

I have a repro:

Create a react-native 0.70 app using the typescript template: npx react-native init client --template<EMAIL_ADDRESS>Try to upgrade it: npx react-native upgrade 0.71.3
See error

info Fetching diff between v0.70.6 and v0.71.3...
info Applying diff...
warn Excluding files that exist in the template, but not in your project:
  - .flowconfig
  - App.js
  - __tests__/App-test.js
error Excluding files that failed to apply the diff:
  - android/app/src/main/java/com/client/newarchitecture/modules/MainApplicationTurboModuleManagerDelegate.java
  - android/app/src/main/jni/CMakeLists.txt
Please make sure to check the actual changes after the upgrade command is finished.
You can find them in our Upgrade Helper web app: https://react-native-community.github.io/upgrade-helper/?from=0.70.6&to=0.71.3
error Automatically applying diff failed. We did our best to automatically upgrade as many files as possible
error Patch failed to apply for unknown reason. Please fall back to manual way of upgrading
info You may find these resources helpful:
• Release notes: https://github.com/facebook/react-native/releases/tag/v0.71.3
• Manual Upgrade Helper: https://react-native-community.github.io/upgrade-helper/?from=0.70.6&to=0.71.3
• Git diff: https://raw.githubusercontent.com/react-native-community/rn-diff-purge/diffs/diffs/0.70.6..0.71.3.diff
error Upgrade failed. Please see the messages above for details.
info Run CLI with --verbose flag for more details.

Verbose:
$ npx react-native upgrade 0.71.3 --verbose
info Fetching diff between v0.70.6 and v0.71.3...
info Applying diff...
warn Excluding files that exist in the template, but not in your project:
  - .flowconfig
  - App.js
  - __tests__/App-test.js
error Excluding files that failed to apply the diff:
  - android/app/src/main/java/com/client/newarchitecture/modules/MainApplicationTurboModuleManagerDelegate.java
  - android/app/src/main/jni/CMakeLists.txt
Please make sure to check the actual changes after the upgrade command is finished.
You can find them in our Upgrade Helper web app: https://react-native-community.github.io/upgrade-helper/?from=0.70.6&to=0.71.3
debug "git apply" failed. Error output:
Falling back to direct application...
Applied patch to '.gitignore' cleanly.
Applied patch to '.node-version' cleanly.
Applied patch to '.ruby-version' cleanly.
error: App.js: does not exist in index
Applied patch to 'Gemfile' cleanly.
error: __tests__/App-test.js: does not exist in index
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Applied patch to 'android/build.gradle' cleanly.
Applied patch to 'android/gradle.properties' cleanly.
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Applied patch to 'ios/client/AppDelegate.h' cleanly.
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Performing three-way merge...
Applied patch to 'tsconfig.json' with conflicts.

error Automatically applying diff failed. We did our best to automatically upgrade as many files as possible
error Patch failed to apply for unknown reason. Please fall back to manual way of upgrading
info You may find these resources helpful:
• Release notes: https://github.com/facebook/react-native/releases/tag/v0.71.3
• Manual Upgrade Helper: https://react-native-community.github.io/upgrade-helper/?from=0.70.6&to=0.71.3
• Git diff: https://raw.githubusercontent.com/react-native-community/rn-diff-purge/diffs/diffs/0.70.6..0.71.3.diff
error Upgrade failed. Please see the messages above for details

I have a repro:

Create a react-native 0.70 app using the typescript template: npx react-native init client --template<EMAIL_ADDRESS>Try to upgrade it: npx react-native upgrade 0.71.3
See error

info Fetching diff between v0.70.6 and v0.71.3...
info Applying diff...
warn Excluding files that exist in the template, but not in your project:
  - .flowconfig
  - App.js
  - __tests__/App-test.js
error Excluding files that failed to apply the diff:
  - android/app/src/main/java/com/client/newarchitecture/modules/MainApplicationTurboModuleManagerDelegate.java
  - android/app/src/main/jni/CMakeLists.txt
Please make sure to check the actual changes after the upgrade command is finished.
You can find them in our Upgrade Helper web app: https://react-native-community.github.io/upgrade-helper/?from=0.70.6&to=0.71.3
error Automatically applying diff failed. We did our best to automatically upgrade as many files as possible
error Patch failed to apply for unknown reason. Please fall back to manual way of upgrading
info You may find these resources helpful:
• Release notes: https://github.com/facebook/react-native/releases/tag/v0.71.3
• Manual Upgrade Helper: https://react-native-community.github.io/upgrade-helper/?from=0.70.6&to=0.71.3
• Git diff: https://raw.githubusercontent.com/react-native-community/rn-diff-purge/diffs/diffs/0.70.6..0.71.3.diff
error Upgrade failed. Please see the messages above for details.
info Run CLI with --verbose flag for more details.

Verbose:
$ npx react-native upgrade 0.71.3 --verbose
info Fetching diff between v0.70.6 and v0.71.3...
info Applying diff...
warn Excluding files that exist in the template, but not in your project:
  - .flowconfig
  - App.js
  - __tests__/App-test.js
error Excluding files that failed to apply the diff:
  - android/app/src/main/java/com/client/newarchitecture/modules/MainApplicationTurboModuleManagerDelegate.java
  - android/app/src/main/jni/CMakeLists.txt
Please make sure to check the actual changes after the upgrade command is finished.
You can find them in our Upgrade Helper web app: https://react-native-community.github.io/upgrade-helper/?from=0.70.6&to=0.71.3
debug "git apply" failed. Error output:
Falling back to direct application...
Applied patch to '.gitignore' cleanly.
Applied patch to '.node-version' cleanly.
Applied patch to '.ruby-version' cleanly.
error: App.js: does not exist in index
Applied patch to 'Gemfile' cleanly.
error: __tests__/App-test.js: does not exist in index
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Falling back to direct application...
Applied patch to 'android/build.gradle' cleanly.
Applied patch to 'android/gradle.properties' cleanly.
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Applied patch to 'ios/client/AppDelegate.h' cleanly.
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
error: repository lacks the necessary blob to perform 3-way merge.
Falling back to direct application...
Performing three-way merge...
Applied patch to 'tsconfig.json' with conflicts.

error Automatically applying diff failed. We did our best to automatically upgrade as many files as possible
error Patch failed to apply for unknown reason. Please fall back to manual way of upgrading
info You may find these resources helpful:
• Release notes: https://github.com/facebook/react-native/releases/tag/v0.71.3
• Manual Upgrade Helper: https://react-native-community.github.io/upgrade-helper/?from=0.70.6&to=0.71.3
• Git diff: https://raw.githubusercontent.com/react-native-community/rn-diff-purge/diffs/diffs/0.70.6..0.71.3.diff
error Upgrade failed. Please see the messages above for details

What is the problem of this issue though?

As it seems you're reporting a failure with react-native upgrade, you should open this issue against the CLI:
https://github.com/react-native-community/cli/issues
Moreover, upgrading from an external template to the default template is not supported (that's also the reason why it failed for you).

Closing in favor of https://github.com/react-native-community/cli/issues/1866

Hi @cortinico ,
I am getting below error after my react native version moved from 0.64 to 0.70 in my current project.
New Version
0.70.13
Old Version
0.64.2
Can anyone help me with this issue?
error: Error: Unable to resolve module ./app.json from D:\VR-Setup\UI-Projects\newproject\index.js:
None of these files exist:
app.json(.native|.android.jsx|.native.jsx|.jsx|.android.js|.native.js|.js)
app.json\index(.native|.android.jsx|.native.jsx|.jsx|.android.js|.native.js|.js)
5 | import {AppRegistry} from 'react-native';
6 | import App from './App';
7 | import {name as appName} from './app.json';
| ^
8 | import 'react-native-gesture-handler';
9 | AppRegistry.registerComponent(appName, () => App);
10 |
at ModuleResolver.resolveDependency (D:\VR-Setup\UI-Projects\newproject\node_modules\metro\src\node-haste\DependencyGraph\ModuleResolution.js:152:15)
at DependencyGraph.resolveDependency (D:\VR-Setup\UI-Projects\newproject\node_modules\metro\src\node-haste\DependencyGraph.js:264:43)
at Object.resolve (D:\VR-Setup\UI-Projects\newproject\node_modules\metro\src\lib\transformHelpers.js:170:21)
at resolveDependencies (D:\VR-Setup\UI-Projects\newproject\node_modules\metro\src\DeltaBundler\graphOperations.js:466:33)
at processModule (D:\VR-Setup\UI-Projects\newproject\node_modules\metro\src\DeltaBundler\graphOperations.js:232:31) at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async traverseDependenciesForSingleFile (D:\VR-Setup\UI-Projects\newproject\node_modules\metro\src\DeltaBundler\graphOperations.js:221:3)
at async Promise.all (index 0)
at async initialTraverseDependencies (D:\VR-Setup\UI-Projects\newproject\node_modules\metro\src\DeltaBundler\graphOperations.js:204:3)


Add code documentation

These commits add doc stubs to all the packages. This partly addresses #173.

Coverage remained the same at 52.24% when pulling 861e1b9910a0108eebc7bb67363b7929b7628feb on add-src-doc into 985af03738cb5a3ca12088830f5fcefdeb40848c on master.

Ugh, I don't like the comments by coveralls, so I've disabled that. The new test coverage for pull requests can still be seen in the ci-status.

I don't like the comments by coveralls

Is it possible to have comments only when coverage decreases (by x%), but not if it stays the same?

No, it's not. Just on or off. We can however configure that the coverall test fails either when the coverage falls below a threshhold or decreases by an amount.


Rename PreloadedSpriteSingle to LoadedSprite

Fixes: #375

Changes Unknown when pulling acaa5ddfb13566b47b13bf225e57d1d144812b31 on loaded-sprite into ** on master**.

Changes Unknown when pulling acaa5ddfb13566b47b13bf225e57d1d144812b31 on loaded-sprite into ** on master**.

Blocked by https://github.com/golang/go/issues/16268.


Html string sanitizer for display

There used to be a Sanitizer to remove unwanted Html fragments for display in a view.
Is it still there under some other name / will it be supported again in the future ? If not, do you have a good solution to share ?
Thanks

MVC will fully HTML-encode a string that is being rendered. We don't have any plans to support sanitizing HTML fragments.
cc @blowdart


Compressed Plots

Getting an error about the file size of new plots. Any chance you'll update this to work with compressed plots?

As a quick fix, you can update the src on your local copy to decrease the minimum size of the plot:
https://github.com/maxbanton/chia-plot-mover/blob/master/src/mover.py#L15
Change 108 to the size of your compressed plots. Haven't tested myself, but I assume this will resolve your issue.
(will be playing with compressed plots in a few days myself).


Implement Correction formula for PM2.5 available on local server

In the next firmware version we will implement a corrected PM2.5 value that will be a new property on the local-server.
Documentation is already updated.
New property is called:
pm02Compensated
I would suggest, Home Assistant checks if that new property is available and then uses that for the standard PM2.5. If not it should fall back to pm02 as currently implemented.

Is this already implemented?


Would be nice to be able to not show warnings from packages/

Maybe this is already possible?  Normally I want to see warnings in packages/ but right now I'm working around INFO warnings in intl.dart: https://github.com/dart-lang/intl/pull/76

This is a pretty low priority issue for me.  Just figured I should file.

Cool. We had one other report of this. We shouldn't be showing warnings from the packages/ dir. What repo is this happening in for you?

a local project which Atom created for me. I can push it to github, but right now it's not even a git repo.

I'm guessing these warnings don't show up until after you've navigated to the file once?

http://filebin.ca/2DVK5lM6wSvT/grid.tgz

Not sure.  I had more than one project open in the same atom workspace.  The other project I had open used this library, this one didn't.  However, Sky now depends on intl as of yesterday.

@brianwilkerson @stereotype441
It looks like pub is creating dart files in .pub/deps/debug/intl/.

Closing in favor of https://github.com/dart-lang/sdk/issues/24212; thanks for the report.


"Copy from stream" in "Create stream" only unfolds the first time

If you can cancel the stream creation and start the flow again, the unfold doesn't unfold:
https://user-images.githubusercontent.com/28347/125699924-fe616f2a-9638-4431-8f7b-e108d2b32c80.mov

Should be a minor fix. Looking into it!


2d moist bubble

This is a draft of a setup with moisture and precipitation inside ClimaAtmos. This is related to #214
Items on our list:

[x] initial condition
[x] pressure computed using Thermodynamics.PhaseEquil_rhoeq
[x] double check the 2D vs 3D kinetic energy computations
[x] rhs! for rhoq_tot, rhoe_tot and rho
[ ] add precipitation variables a later PR
[ ] think about caching a later PR
[x] bring back regression test in the test script
[ ] bring back validation test (long run with video outputs) For that it would be great to add the diagnosed q_liq and q_ice to Ya

Co-authored-by: Anna Jaruga<EMAIL_ADDRESS>

check out #228


Fix mishandling Karun in Kavee Matra Checker

What does this changes
referring to this issue: #791
This changes Karun(sound silence) handler algorithm with another function
What was wrong
referring to this issue: #791
the reason is khavee matra checker only check LAST character AKA. ตัวสะกด
for example 'มนตร์' will return กด because the karun handled text by old cleaner is มนต instead of มน
How this fixes it
This new function check if there's Karun is it possible to silence the last two characters/vowel
if so we delete unnecessary sound form for example
พักตร์ will be พัก since it is possible to silence ต to ร
Fixes #...
Your checklist for this pull request
🚨Please review the guidelines for contributing to this repository.

[ ] Passed code styles and structures
[ ] Passed code linting checks and unit test

Coverage: 92.053% (+0.01%) from 92.043% when pulling dd4f0a117cd303f8bb57a268f75369530f402b88 on HRNPH:dev into 462a83ea5d7676d4c2c20164d1182a636a925cc5 on PyThaiNLP:dev.


ZeRO-3 Slowdown

Fix significant zero-3 slowdown compared to zero-2, based on #1069.
Two key parts of this PR:

Fix hook registration problem that created off-by-one tracing issue
Move tracing book-keeping to parameter partition logic.

Hi @tjruwase ,
I am recently merging your modifications with the customized allgather I introduced in another PR #1188 .
I placed the code in my personal repo at here:  zhen/zero3_debug
Basically, I replaced the torch allgather with my own allgather. https://github.com/zarzen/DeepSpeed/blob/zhen/zero3_debug/deepspeed/runtime/zero/partition_parameters.py#L677_L683
I can see performance improvements, especially for the backward part.
While after analyzing the timeline I found two issues. let me bring the timeline at here:

Two issues:

the prefetching only partially overlaps with computation.
It would be nice to see the next-next module also prefetched. (As denoted in green circle)
there is a pretty large gap between the computation of current module and next-fetch-module operation
Which I denoted in red box.

For the first issue, I think the it is mostly due to the condition check in the prefetch_next_sub_modules:
def prefetch_next_sub_modules(self,...
Do you have idea the timeline gap of second issue?
And for the first issue, do you have some suggestions for launching the next-next prefetching earlier?

the prefetching only partially overlaps with computation.
It would be nice to see the next-next module also prefetched. (As denoted in green circle)

I think we to consider that prefetching increases memory consumption, at the extreme we lose the benefit of parameter partitioning. This is my guess for why we restrict next-next module prefetching. Of course this depends on the model size and on the exposed latency of parameter gather on the critical path. Perhaps @samyam can chime in.

2. there is a pretty large gap between the computation of current module and next-fetch-module operation
Which I denoted in red box.

Nothing comes to mind right away for why this is happening. Is this timeline showing for the entire forward or for individual module computation?

the prefetching only partially overlaps with computation.
It would be nice to see the next-next module also prefetched. (As denoted in green circle)

I think we should consider that prefetching increases memory consumption, at the extreme we lose the benefit of parameter partitioning. This is my guess for why we restrict next-next module prefetching. Of course this depends on the model size and on the exposed latency of parameter gather on the critical path. Perhaps @samyam can chime in.

agree, memory consumption is definitely an important consideration. while I saw there is a inflight size checking for param gathering. that param suppose to limit the memory usage. but for now the size of a module is smaller than the stage3_prefetch_bucket_size is not going to work.

there is a pretty large gap between the computation of current module and next-fetch-module operation
Which I denoted in red box.

Nothing comes to mind right away for why this is happening. Is this timeline showing for the entire forward or for individual module computation?

this timeline is for two consecutive modules, not the entire timeline for forward.
I can share the profiling file if you would like to look into details.

Hi @tjruwase  @samyam
After detailed analysis, I found the source of small cudaMemcpy operations.
Every time the release_sub_module is invoked, the _partition_param function will be called, which contains a memory operation at here:
https://github.com/microsoft/DeepSpeed/blob/3c4a94924ec1474afa2b78a9ccfcfb5690dc4f49/deepspeed/runtime/zero/partition_parameters.py#L729
I would recommend to replace it with the following:
param.data = torch.empty(1, dtype=self.dtype, device=param.device)

Thus, it can avoid the memory movement, besides, the element_wise_kernel won't be launched for setting the memory. It would reduce the forward time from ~180ms to ~150ms in my case.
For the pipelining issue, I found the prefetch_next_sub_modules takes about ~500us, in my setup.
I found even though the param gathering operation is launched in async way. The other parts take significant amount of time.
I would like to place the prefetch_next_sub_modules function call into a separate thread. But I am not sure whether it is going to break exist deisgn. Could you comment a little bit?
Here is the current time:

You can see the computation kernels of next module is delayed by the prefetch_next_sub_modules function call.

I would recommend to replace it with the following:
param.data = torch.empty(1, dtype=self.dtype, device=param.device)

Please go ahead with this change. Thanks!

For the pipelining issue, I found the prefetch_next_sub_modules takes about ~500us, in my setup.
I found even though the param gathering operation is launched in async way. The other parts take significant amount of time.
I would like to place the prefetch_next_sub_modules function call into a separate thread. But I am not sure whether it is going to break exist deisgn. Could you comment a little bit?

Can you clarify if you mean separate stream or python thread? Regardless, it seems to me that making prefetch_next_sub_modules more asynchronous should be fine. We just need to be careful to properly coordinate with the next fetch_sub_module. We can also think of unit tests for the corner cases as I don't think our existing unit tests do this.

For the pipelining issue, I found the prefetch_next_sub_modules takes about ~500us, in my setup.
I found even though the param gathering operation is launched in async way. The other parts take significant amount of time.
I would like to place the prefetch_next_sub_modules function call into a separate thread. But I am not sure whether it is going to break exist deisgn. Could you comment a little bit?

Can you clarify if you mean separate stream or python thread? Regardless, it seems to me that making prefetch_next_sub_modules more asynchronous should be fine. We just need to be careful to properly coordinate with the next fetch_sub_module. We can also think of unit tests for the corner cases as I don't think our existing unit tests do this.
@samyam what do you think?

A separate python threads. because I didn't see any computation or communication on CUDA is required for release_sub_module. Even there are memory operations, they should probably happen on the default cuda stream, maintained by PyTorch.

Which way do you think is most convienient for the change? (since it is a very small modification, so I didn't plan for a PR at this moment. )

I think these changes can wait until your PR is ready because they won't give much benefits without your PR. What do you think?

Which way do you think is most convienient for the change? (since it is a very small modification, so I didn't plan for a PR at this moment. )

I think these changes can wait until your PR is ready because they won't give much benefits without your PR. What do you think?

Sure. we can leave it as it is for now, and it is just a one line change. Thanks!

Hi, is this PR and #1188 still under the plan for merging to the master?

@zarzen, apologies for the delay on this. The merge was delayed because of our latest release a few weeks ago. We will now focus on reviewing and merging both PRs soon. Thanks for your patience.

@zarzen, my teammates have reviewed this PR and it looks good. The next step is to rerun correctness and perf tests, especially since significant code was merged into the master branch for our last release. Can you also run perf tests on your side to check that there are no regressions. I will be doing the same. Although this is a bit of work, but hopefully it would make merging #1188 easier.

@zarzen, my teammates have reviewed this PR and it looks good. The next step is to rerun correctness and perf tests, especially since significant code was merged into the master branch for our last release. Can you also run perf tests on your side to check that there are no regressions. I will be doing the same. Although this is a bit of work, but hopefully it would make merging #1188 easier.

sure, I have seen you merged the master into my branch, Thanks!
I will start the regression tests ASAP, possibly complete before Friday.

Subsumed by #1453


SyntaxError while integrated with nuxt js 3

Current behavior
It shows the error:
  SyntaxError: The requested module '@psf/bitcoincashjs-lib' does not provide an export named 'TransactionBuilder'
  at ModuleJob._instantiate (node:internal/modules/esm/module_job:131:21)
  at async ModuleJob.run (node:internal/modules/esm/module_job:213:5)
  at async ModuleLoader.import (node:internal/modules/esm/loader:308:24)
  at async ViteNodeRunner.interopedImport (node_modules/.pnpm/vite-node@1.4.0/node_modules/vite-node/dist/client.mjs:383:28)
  at async ViteNodeRunner.directRequest (node_modules/.pnpm/vite-node@1.4.0/node_modules/vite-node/dist/client.mjs:253:24)
  at async ViteNodeRunner.cachedRequest (node_modules/.pnpm/vite-node@1.4.0/node_modules/vite-node/dist/client.mjs:189:14)
  at async ViteNodeRunner.dependencyRequest (node_modules/.pnpm/vite-node@1.4.0/node_modules/vite-node/dist/client.mjs:233:12)
  at async node_modules/.pnpm/@swapkit+sdk@1.0.2_@babel+core@7.24.3_@polkadot+util-crypto@12.6.2_fastestsmallesttextencoder_wvhcu6wkucfh3vpswcidayh6xy/node_modules/@swapkit/sdk/dist/index.js:1:31
  at async ViteNodeRunner.runModule (node_modules/.pnpm/vite-node@1.4.0/node_modules/vite-node/dist/client.mjs:362:5)
  at async ViteNodeRunner.directRequest (node_modules/.pnpm/vite-node@1.4.0/node_modules/vite-node/dist/client.mjs:346:5)

Expected behavior
should not have error
Functionality

[ ] Swap
[ ] Liquidity
[ ] Savers
[ ] Loan
[ ] Wallet
[ ] Transactions
[ ] Api
[X] Other

Packages

[X] @swapkit/sdk
[ ] @swapkit/core
[ ] @swapkit/api
[ ] @swapkit/helpers
[ ] @swapkit/tokens
[ ] @swapkit/toolbox-cosmos
[ ] @swapkit/toolbox-evm
[ ] @swapkit/toolbox-utxo
[ ] @swapkit/wallet-evm-extensions
[ ] @swapkit/wallet-keepkey
[ ] @swapkit/wallet-keplr
[ ] @swapkit/wallet-keystore
[ ] @swapkit/wallet-ledger
[ ] @swapkit/wallet-okx
[ ] @swapkit/wallet-trezor
[ ] @swapkit/wallet-wc
[ ] @swapkit/wallet-xdefi

Details
Package versions:
# Paste the output of `<npm|yarn|pnpm|bun pm> ls --depth=0 | grep @swapkit` here
pnpm ls --depth=0 |grep @swapkit
@swapkit/sdk 1.0.2

@HelloRWA workaround for that would be to create .d.ts file, add:
declare module "@psf/bitcoincashjs-lib";

and include it in your tsconfig.json for now as @psf/bitcoincashjs-lib is outside of our possibility to fix and we are right now looking for some solution to replace.


[Improvement] Add graphql command help text

Description:
We need to add the graphql command help text (cli-help/ballerina-graphql.help file) to https://github.com/ballerina-platform/ballerina-lang/tree/master/cli/ballerina-cli/src/main/resources/

Closing this issue since the PR is already merged


Update config.yaml.example

Add some information about alerts. I ran into an issue trying to figure out how to add more information to the alert. I'm not sure this is the right place to put this but something like this would've helped me out.

I recommend adding this to examples/*.yaml. This file may get visibility, but it doesn't actually do anything in config.yaml. I am also planning on either highlighting this better in the documentation or changing it to be default behavior.


check-migration-names

We can add check_migration_names from @adamchainz
Blog post:
https://adamj.eu/tech/2020/02/24/how-to-disallow-auto-named-django-migrations/#a-custom-system-check

There's no need - Django 4.0+ never generates migration names like 'auto_...' since https://code.djangoproject.com/ticket/31516 . I need to update that post with a banner.

Thank you for your information.


Windows ConPTY does not allow mouse interactions in the integrated terminal

Does this issue occur when all extensions are disabled?: Yes

VS Code Version:

Version: 1.71.1 (user setup)
Commit: e7f30e38c5a4efafeec8ad52861eb772a9ee4dfb

OS Version:

OS: Microsoft Windows 11 Home
Version: 10.0.22000 Build 22000

Steps to Reproduce:

Open any CLI tool that supports mouse events in the VS Code integrated terminal.
Try interacting with the terminal (=> not working)
Disable ConPTY
Close and reopen the terminal session
Try to interact with the CLI again (=> working)

Hello, I am aware that this issue was posted before, but it was closed years ago, and it still persists.
VS Code uses ConPTY by default in the integrated terminal. ConPTY does not allow mouse interactions, yet. This means, that you have to disable terminal.integrated.windowsEnableConpty, to be able to interact with CLIs via the mouse.
But if I am not mistaken, the Windows Terminal ships its own ConPTY? According to this comment, Windows Terminal supports mouse interactions (with ConPTY) since v1.9: https://github.com/microsoft/terminal/issues/376#issuecomment-849756080
Would it be possible to integrate that into VS Code as well?

+@dhowett as an fyi since I'll be on leave before I get to the root of this.
I'd reckon it should be in 22000. I forget if there's something wacky we're doing to get mouse mode enabled by default (we might be)

Due to the timing of when 10.0.22000 went into feature freeze, mouse mode unfortunately did not make the cut. It should be available in 22621.

@DHowett I'm running now Windows 11 10.0.22621.1992 and the mouse input is still not working. Do you know the Windows 11 version that will get this feature?


Extend scenario metadata

This PR adds additional information to the metadata. This allows for constrained and unconstrained arguments to be included in the metadata.
I am considering adding additional serialization support.

Something odd is happening with the _build directory now. I wonder if it has to do with using yaml instead of pickle.


redirect to home page after silent refresh

Hello, I did silent refreshing regarding your documentation.
When I try to call
     this         .oauthService         .silentRefresh()         .then((info) => console.warn('refresh ok', info))         .catch((err) => console.error('refresh error', err));
Then I got message in console:
OAuthSuccessEvent info: null type: "silently_refreshed"
And then app is redirect to main page.
How to prevent this redirecting? Thank you.

I did investigation and now i know that redirection is envoked via
  parent.postMessage(location.hash, location.origin);

Hi @artemukolov did you find any solution for this? As in my application is also getting redirected to main(Home) page after silent refresh is performed.

@shriharry not yet.

Thanks for this. Can you compare you example with the demo application? There is effect does not occur. What's different there?

We have the same problem. I think the main difference in our project is that we use HashStrategy for routing. Is there a way to redirect to url before silently refreshed if silent refresh is setup automically?

@ndevtag Have you tried this approach from the docs about HashStrategy yet?

this ist not the case i've explained here. The case i've explained belongs to the silent refresh, not the initial login. Or what do you mean?

Apologies, my suggestion then might've created more confusion to this issue, instead of being helpful.
 Either way @manfredsteyer's earlier remark probably holds: try to compare with the demo app and/or create a minimal repro that shows the issue, so others can further assist. Without a way to reproduce the issue it's really hard to help.

I think i've written that the main difference is the location strategy where the problem only occures on HashStartegy

Hi, i´m running into the same problem. Is there any solution or workaround on this problem?
This also occurs in the sample-app if you turn on hash-strategy on call silentRefresh from another route as "/home".
To reproduce it in sample-app, enable hash-strategy in app.routes.ts and:
eg. add button to flight-edit.component.ts:

Flight Edit!

Hier könnte auch der Datensatz mit der Id {{id}} stehen!

<button class="btn btn-default" (click)="flightService.testSilentRefresh()">Test silent refresh</button>

and add to flight.service.ts:
testSilentRefresh() {
    this.oauthService.oidc = true;
    this.oauthService
      .silentRefresh()
      .then(info => console.debug('refresh ok', info))
      .catch(err => console.error('refresh error', err));
  }

Is there any solution yet? I am having the same issue.

I haven´t found an workaround yet because i don´t have an idea what causes this issue.
Still hoping for @manfredsteyer to find an solution or workaround.

I'm seeing this and I'm not using hashing strategy. Have yet to step through the code, but have tried setting "clearHashAfterLogin" to false, but that made no difference.

@silamon Thank you for pointing that out. Though we also missed out on this.
Setting up both leads to a working system.

Not sure if this relates to this issue directly but posting here in case someone may find it helpful. If you're doing client authorization in your back-end, be sure to add the silent-refresh.html url endpoint to the list of allowed redirect uris in your database

We have the same Problems. We ran into redirect in Iframe on Windows with Chrome. Other Constellations are working great. Chrome on Mac works. Windows and Firefox works!

@manfredsteyer  Any updates on the issue?
I facing the same issue, I am using the hash location strategy, getting redirected to the home page after silent refresh. Totally stuck.

Just to chip in - I have 2 very similar apps with same oauth setup and this issue occurs in only one of these.
I have a feeling that this might be related to a custom baseHref setting in angular.json's build script configuration (as this should be the only difference between my apps).

Posting here in case someone may find it helpful since I ran into a similar issue. If you're doing client authorization in your back-end, be sure to add the silent-refresh.html url endpoint to the list of allowed redirect uris in your database

If that was the case, then the silent refresh would fail (instead of working fine and redirecting to the homepage).
Also, my issue might be different, cause it's not related to HashStrategy - I'm not using it in my apps.

@mmanista I am using implicit flow and I already added the silent-refresh.html in allowed urls, and already mentioned it in angular.json, but no success. Redirecting back to home page.

Seems like this is solved with version 9.1 which lands soon.
If you clone this repos, the sample application demonstrates this using the newest version. To reproduce it, do the following:

In flags.ts, set useHash to true
Login with implicit flow
Assure yourself, that you've been logged in
Switch to the route 'Book  a Flight'
Scroll down and press 'Refresh Token'
In the console, you see, that the refresh took happen. Also, you are not redirected to the home route

Here is some additional info about using hash-based routing:
https://manfredsteyer.github.io/angular-oauth2-oidc/docs/additional-documentation/routing-with-the-hashstrategy.html
Feel free to reopen this issue, if I'm missing sth here.

@manfredsteyer  thanks issue resolved in the latest version.

@manfredsteyer although the new update did resolved my issue, but it lead to a totally different issue after updating to 9.2.0. the prod build doesn't gives any error. but running the prod build is not working at all. I am just stuck to a blank screen. The same issue is not reproducible in normal build.

Hello, I did silent refreshing regarding your documentation.
When I try to call
 this .oauthService .silentRefresh() .then((info) => console.warn('refresh ok', info)) .catch((err) => console.error('refresh error', err));
Then I got message in console:
OAuthSuccessEvent info: null type: "silently_refreshed"
And then app is redirect to main page.
How to prevent this redirecting? Thank you.

I am facing the same issue, did you find out any solution ?

Hello, I did silent refreshing regarding your documentation.
When I try to call
 this .oauthService .silentRefresh() .then((info) => console.warn('refresh ok', info)) .catch((err) => console.error('refresh error', err));
Then I got message in console:
OAuthSuccessEvent info: null type: "silently_refreshed"
And then app is redirect to main page.
How to prevent this redirecting? Thank you.

I am facing the same issue, did you find out any solution ?

I had same issue because I have navigation inside token listener,  I use it to redirect to previous page after logging and every silent refresh oauthService.state is empty and then it redirects to main page. I fixed it with simple check of oauthService.state
this.oauthService.events
        .pipe(
          filter((e: any) => e.type === 'token_received'))
        .subscribe(({ type }) => {
          **if (!this.oauthService.state) { return; }**
          this.router.navigateByUrl(decodeURIComponent(this.oauthService.state));
        });


Is it possible to not render all of the http content on server side route?

I'm using this as an API endpoint. I receive a webhook and I don't need to render my whole app just to tell the server that everything is "ok".
Should I look for another package (any suggestions) or is this the right package for this?
Thanks

Yeah. This is what you should look at.

Ok, thanks. I've already got a Picker route in place, but I can see that Picker responds with much more than what I need for a simple POST endpoint. It appears to be trying to render the whole app.

Thanks, I found it in the docs.
Looks like I just needed to add this  -> return req.method == "POST"; after my res.end();


Legalizes Auth

It's better to add some md5 + hash + guid + idk generator to UA or just copy it from wizards launcher

An API call that gets the relevant UAgent is preferable. This approach still signals that its a custom launcher.


Adds DisplayDesc inequality comparison operators

Closes #472

Do we want a strong ordering here, or would a partial ordering make more sense?
I was thinking:

a <= b implies all component values compare <=

a < b implies a <= b and a != b (at least one component <, others <=)

elements are incomparable if (a.x < b.x) && (a.y > b.y) for some fields x and y

You make a good point, however std::strong_ordering and std::partial_ordering are C++20 features that are unavailable and will have to be manually implemented for now. :(
The underlying implementations differ on ordering. Neither define what happens if the width is the same as the height on Resolution ordering.
Ref: https://wiki.libsdl.org/SDL_GetClosestDisplayMode

The modes are scanned with size being first priority, format being second priority, and finally checking the refresh rate. If all the available modes are too small, then NULL is returned.

As usual, SDL's documentation is a single sentence while DirectX is an entire page.
Ref: https://docs.microsoft.com/en-us/windows/win32/api/dxgi1_2/nf-dxgi1_2-idxgioutput1-findclosestmatchingmode1

FindClosestMatchingMode1 resolves similarly ranked members of display modes (that is, all specified, or all unspecified, and so on) in the following order:

ScanlineOrdering
Scaling
Format
Resolution
RefreshRate

When FindClosestMatchingMode1 determines the closest value for a particular member, it uses previously matched members to filter the display mode list choices, and ignores other members. For example, when FindClosestMatchingMode1 matches Resolution, it already filtered the display mode list by a certain ScanlineOrdering, Scaling, and Format, while it ignores RefreshRate. This ordering doesn't define the absolute ordering for every usage scenario of FindClosestMatchingMode1, because the application can choose some values initially, which effectively changes the order of resolving members.

FindClosestMatchingMode1 matches members of the display mode one at a time, generally in a specified order.

If a member is unspecified, FindClosestMatchingMode1 gravitates toward the values for the desktop related to this output. If this output is not part of the desktop, FindClosestMatchingMode1 uses the default desktop output to find values. If an application uses a fully unspecified display mode, FindClosestMatchingMode1 typically returns a display mode that matches the desktop settings for this output.
Because unspecified members are lower priority than specified members, FindClosestMatchingMode1 resolves unspecified members later than specified members.

The you for the documentation links/quotes.
Since the method is a new virtual method, you could document its behaviour as implementation defined. Leave it to the SDL/DirectX implementation how to order the matching.
Or, you could develop your own matching algorithm, so behaviour is consistent across implementations, in which case the ordering chosen here could dictate how matching works.

In terms of strong order or partial ordering, those concepts have been in C++ for many many years, and references to that could be found in various STL algorithms from early on. They weren't yet formalized they way they have been for C++20 though.
At any rate, I don't believe a partial ordering would be any more work that what you've already implemented. It's done by defining each of the individual comparison operators.

After reading the descriptions of strong ordering vs partial ordering more closely. I think strong ordering applies best.
If the width and height are equal between two resolutions, the tie-breaker is the refresh-rate in most cases. Given that, partial ordering is not a good fit because it claims that any two description comparisons can return false for all non-compound comparisons. This can't happen if we are only checking for width, height, and refresh rate. By leaving out the formats and directx's scanline ordering and scaling the comparisons are between values that all implementations have.
Other than deciding on ordering, was there any specific changes to be made?

I think that's where the partial ordering actually shines.
If resolutions match, and only the refresh rates are different, then the two are comparable, with the ordering set by refresh rate.
DisplayDesc{x, y, 50} < DisplayDesc{x, y, 60}

What would make two modes incomparable is if one had a larger width, while the other had a larger height. There is no obvious ordering in such a case.
DisplayDesc{1280, 720, 60} ? DisplayDesc{1024, 768, 60}

Another possible way a display mode might be incomparable, is if one has lower resolution, but higher refresh rate.
DisplayDesc{1280, 720, 60} ? DisplayDesc{1920, 1024, 50}

My only real concern here is if this is the right code. No concerns about the specifics of how it was written.

What would make two modes incomparable is if one had a larger width, while the other had a larger height. There is no obvious ordering in such a case.
DisplayDesc{1280, 720, 60} ? DisplayDesc{1024, 768, 60}

Another possible way a display mode might be incomparable, is if one has lower resolution, but higher refresh rate.
DisplayDesc{1280, 720, 60} ? DisplayDesc{1920, 1024, 50}

In the implementations of "Give me all available Display Description" functions that I've seen, they are ordered by width, then height, then refresh rate. In the case of 1280x720x60 vs 1024x768x60 The 1280x720x60  would be listed first and in the case of 1280x720x60 vs 1920x1024x50 the 1920x1024x50 would be listed first.

Should we close this, or did we want to extract the code for some kind of lexicographical comparison function?

Yeah, go ahead and close it. If I need a separate type of comparison I can always use a lambda.


Delete GCP000339 (submission id: 627a1e1728d70900018e0c2f) and change GCST

We need to delete this submission as the user has clubbed sumstats for different traits under one study.
The sumstat is now split and submitted under PMID: 36681271 (6614fa7cb73c740001cd3cd1). The paper mentions GCSTs from GCP000339.
After deleting the submission, the 2 GCSTs from 6614fa7cb73c740001cd3cd1 need to be changed
GCST90399399 needs to be changed to GCST90102551
GCST90399400 needs to be changed to GCST90102552

@Santhi1901 done please validate

The GCSTs have been changed correctly


Permalinks for wheels from old pre-releases

An unintended consequence of fixing:

https://github.com/rerun-io/rerun/issues/1354

is that if users are working with a pre-release for some use-case, once they know it's working for them they might want to save that link to use it again in the future.
By clearing the old artifacts, we currently break those links. It would be nice to have a way to correctly permalink to one of these old builds without cluttering up the releases pages.

@jleibs We have stable links on gh releases and on respective package managers. Not sure what else you're looking for?


Call API button issue

I am having trouble getting the "Call API" button to work after login. You need to download the server seed and start it to call this API is the error I keep getting, and looking at the network traffic log, it seem that I am getting a 404 on an OPTIONS request. The examples for the backend does not handle the initial OPTIONS request that the browser makes. I've tried this with the Nodejs, Golang, and Ruby seed examples and none of them seems to work.

To have the app work with the API, I had to set the Access-Control-Allow-Origin and Access-Control-Allow-Headers response headers in the backend for the secured endpoint. In the case of the golang example:
func SecuredPingHandler(w http.ResponseWriter, r *http.Request) {
	allowHeaders := r.Header.Get("Access-Control-Request-Headers")
	w.Header().Set("Access-Control-Allow-Origin", "*")
	// Reflect `Access-Control-Request-Headers` header back
	w.Header().Set("Access-Control-Allow-Headers", allowHeaders)
	respondJson("All good. You only get this message if you're authenticated", w)
}


Multiline placeholder input support implemented

This pull request contains changes which allow chat input bar to have a multi line long text placeholders

Codecov Report

Merging #640 into master will decrease coverage by 0.02%.
The diff coverage is 95.65%.

@@            Coverage Diff            @@
##           master    #640      +/-   ##
=========================================
- Coverage   61.22%   61.2%   -0.03%
=========================================
  Files          92      92
  Lines        4965    4977      +12
=========================================
+ Hits         3040    3046       +6
- Misses       1925    1931       +6

Impacted Files
Coverage Δ

...ttoAdditions/Source/Input/ExpandableTextView.swift
71.22% <95.65%> (+2.71%)
:arrow_up:

...Chat Items/TextMessages/Views/TextBubbleView.swift
83.16% <0%> (-3.07%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be10289...0b17378. Read the comment docs.


Fixed navigation bar not being entirely hidden on Android

Description
In full-screen mode on Android, the navigation bar was not entirely hidden which I tested on multiple devices. After changing a flag, the navigation bar started to behave as expected.
Also refer to the official sample about immersive mode.
Tasks

[ ] Tested on Linux
[ ] Tested on Windows
[ ] Tested on macOS
[ ] Tested on iOS
[X] Tested on Android

How to test this PR?
#include <SFML/Graphics.hpp>

using std::string;
using sf::Font;
using sf::Sprite;
using sf::Texture;

int main(int argc, char *argv[])
{
    sf::VideoMode screen(sf::VideoMode::getDesktopMode());

    sf::RenderWindow window(screen, "", sf::Style::Fullscreen);
    window.setFramerateLimit(30);

    sf::Texture texture;
    if(!texture.loadFromFile("image.png"))
        return EXIT_FAILURE;

    sf::Sprite image(texture);
    image.setPosition(screen.width / 2, screen.height / 2);
    image.setOrigin(texture.getSize().x/2, texture.getSize().y/2);

    sf::Font font;
    if (!font.loadFromFile("sansation.ttf"))
        return EXIT_FAILURE;

    sf::Text text("Tap anywhere to move the logo.", font, 64);
    text.setFillColor(sf::Color::Blue);
    text.setPosition(10, 10);

    // Loading canary.wav fails for me for now; haven't had time to test why

    /*sf::Music music;
    if(!music.openFromFile("canary.wav"))
        return EXIT_FAILURE;
    music.play();*/

    sf::View view = window.getDefaultView();

    sf::Color background = sf::Color::Black;

    // We shouldn't try drawing to the screen while in background
    // so we'll have to track that. You can do minor background
    // work, but keep battery life in mind.
    bool active = true;

    while (window.isOpen())
    {
        sf::Event event;

        while (active ? window.pollEvent(event) : window.waitEvent(event))
        {
            switch (event.type)
            {
                case sf::Event::Closed:
                    window.close();
                    break;
                case sf::Event::KeyPressed:
                    if (event.key.code == sf::Keyboard::Escape)
                        window.close();
                    break;
                case sf::Event::Resized:
                    view.setSize(event.size.width, event.size.height);
                    view.setCenter(event.size.width/2, event.size.height/2);
                    window.setView(view);
                    break;
                case sf::Event::LostFocus:
                    background = sf::Color::White;
                    break;
                case sf::Event::GainedFocus:
                    background = sf::Color::Black;
                    break;

                    // On Android MouseLeft/MouseEntered are (for now) triggered,
                    // whenever the app loses or gains focus.
                case sf::Event::MouseLeft:
                    active = false;
                    break;
                case sf::Event::MouseEntered:
                    active = true;
                    break;
                case sf::Event::TouchBegan:
                    if (event.touch.finger == 0)
                    {
                        image.setPosition(event.touch.x, event.touch.y);
                    }
                    break;
                default:
                    break;
            }
        }

        if (active)
        {
            window.clear(background);
            window.draw(image);
            window.draw(text);
            window.display();
        }
        else {
            sf::sleep(sf::milliseconds(100));
        }
    }
}

Just realized we had a similar PR open #1538 but the author didn't provide an updated one. Looking at the two changes, what's the difference between yours and the other one: https://github.com/SFML/SFML/pull/1538/files#diff-a9064007c5b52fa9f173fdd52d3f0dc3R177 ?

@eXpl0it3r Apart from the other files changed in that PR, mine removes setting the other flag (SYSTEM_UI_FLAG_LOW_PROFILE) which doesn't have any effect after setting SYSTEM_UI_FLAG_HIDE_NAVIGATION and is therefore unnecessary.

SFMLBuildMaster: Build this please.


Requesting ability to download only the audio stream of a Periscope

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2019.02.08. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[ x] I've verified and I assure that I'm running youtube-dl 2019.02.08

Before submitting an issue make sure you have:

[ x] At least skimmed through the README, most notably the FAQ and BUGS sections
[ x] Searched the bugtracker for similar issues including closed ones
[ x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[ ] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ x] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

Description of your issue, suggested solution and other information
I'm requesting the ability to download just the audio stream of a periscope if possible.  Currently only only one format without a code listed, so there are no options.

No such format exists.


feat: text to speech

Description
Convert text to speech using Bard API
Motivation and Context
Using Bard text to speech is useful for self hosted assistant
How Has This Been Tested?
Tested manually with different accounts
Usage
from bardapi import Bard

bard = Bard(token= '...' )
audio = bard.speech('Hello, I am bard! How can I assist you?', lang = 'en-US') # bytes audio
with open('bard_response.mp3', 'wb') as f:
    f.write(audio)

Now you can open the file and play it...

Doesn't works

@Jishan76
What do you mean it doesn't work?
You got error? the audio not playing?

I tested it, and it worked successfully. Where can I find more information about the "lang" argument?
I would appreciate it if you could provide additional information about the options available for "lang".
I will accept this pull request and take some time to think and make modifications to the structure. Thank you.
Thank you for your contribution!

@dsdanielpark
From yesterday, bard API updated, and supports 40+ different languages including speaking.
List of languages:
https://support.google.com/bard/answer/13575153?hl=en
The language parameter should be one from the supported languages in format of

English

en-US

Hebrew

iw-IL

not sure where we can find list of this format codes, or which languages supported except for these

Thank you for your contribution.
Due to other projects, I couldn't properly keep up with the updates for Bard. I will make sure to conduct some tests with the terms listed on the checklist!


New categorical judgments---SUBTYPE?

@jonsterling @cangiuli It seems difficult to support subtyping without a new judgment form. The tricky case is
a : A, f : (-> A B) >> ($ f a) in B'

where B is a subtype of B'. The current rule will infer that a type of f is (-> A B), and then insist on the equality between B and B'. Even worse, when this goal is generated, the "B" is just a meta-variable in that goal. It seems there is no easy way to support subtyping without a dedicated judgment holding the unresolved subtyping relation.

I think it's a reasonable idea... I wonder if we can somehow generate the equality and subtyping goals from a single spec (not necessary but maybe interesting).

By the way, we could define SUBTYPE (ty0, ty1) as TRUE (-> [x : Ty0] (= Ty1 x x)).

@favonia indeed; another way to define the subtype relation (which they use in Nuprl) is MEMBER (lam [x] x) (-> ty0 ty1).

@jonsterling Hmm... any reason to use a longer definition?

@favonia I'm not sure; I have a feeling that these are basically equivalent.

This was added as an atomic judgment now.


Allow the checkstyle output format

I couldn't get this building without changing to the @community packages first, so that's included too.
Other than that, it's a simple addition on the php7-dom package.

This should fix #4

I've added tests in the phan versions that have checkstyle output.

Merged as 8dc6e64efe0cd2c3598ccb86fb30fcf29d5a7a2c.


Modified to view in responsive

Added class to logo, modified style to make the page responsive...

Thanks


Update README.md

Pull Request Description
What issue does this change request address?
test ci
What are the significant changes in functionality due to this change request?

For Change Control Board: Change Request Review
The following review must be completed by an authorized member of the Change Control Board.

[ ] 1. Review all computer code.
[ ] 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
[ ] 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
[ ] 4. Automated Tests should pass.
[ ] 5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options.  Multiple short tests are preferred over one large tes.
[ ] 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change.  The requirements and the requirements test shall be in sync.
[ ] 7. The merge request must reference an issue.  If the issue is closed, the issue close checklist shall be done.
[ ] 8. If an analytic test is changed/added, the the analytic documentation must be updated/added.
[ ] 9. If any test used as a basis for documentation examples have been changed, the associated documentation must be reviewed and assured the text matches the example.

Job CentOS 8 on 59d1e70 : invalidated by @alfoa

Job CentOS 8 on 59d1e70 : invalidated by @alfoa

Job CentOS 8 on 59d1e70 : invalidated by @alfoa

restarted after merging crow dir check change

Job CentOS 8 on 59d1e70 : invalidated by @alfoa

same error, more interesting results:
Making ...
./build_raven: line 106: cd: /home/civet/civet/build_42/crow: No such file or directory

Job CentOS 8 on 59d1e70 : invalidated by @joshua-cogliati-inl
test civet


Concerns on default KFSERVING_VERSION in python SDK

/kind bug
Curious what's the considerations in our python SDK https://github.com/kubeflow/kfserving/blob/master/python/kfserving/kfserving/constants/constants.py#L23 that we default KFSERVING_VERSION to v1alpha2 instead of v1beta1? The mismatch between the default value and the cluster KFServing version can create pitfalls sometimes.

@Peilun-Li this constant is only used in e2e tests looks, in the KFS client API we are defaulting to v1beta1 https://github.com/kubeflow/kfserving/blob/42007c532286a1e43893ef2be03b15e104bfd7a4/python/kfserving/kfserving/api/kf_serving_client.py#L310


Add resource identifiers for linking

Thanks for your contribution! Please replace this text with a description of what this PR is changing or adding and why, list any relevant issues, and review the contribution guidelines below.

[ ] I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

See our contributor guide for general expectations for PRs.
Larger or significant changes should be discussed in an issue before creating a PR.
Contributions to our repos should follow the Dart style guide and use dart format.
Most changes should add an entry to the changelog and may need to rev the pubspec package version.
Changes to packages require corresponding tests.

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

coverage: 90.857% (-0.5%) from 91.317%
when pulling 49914d778a4dc9c0307cd6f0f333a2463274aa77 on addResourceIdentifiersForLinking
into 95d226d290d4d5c583c4e45b04048442f244c656 on main.


Accessing Function Expressions in Material Functions

Is there a way of accessing and renaming material parameters in material functions?

Btw. the example of accessing material properties doesn't work any more
parent_material = material_instance.Parent
for expression in parent_material.Expressions:
parameter_name = expression.ParameterName
parameter_group = expression.Group
LogPython: Error: 'unreal_engine.UObject' object has no attribute 'ParameterName'

You can always <uobject>.properties() to list what things you can change. You can also do get_property(<name>) and set_property(<name>, <value>).


[Button] Simplify .clear

This PR refactors the .clear button class to leverage the button-hollow-style() mixin.
Closes #9961, which references @rafibomb's #9739.

@rafibomb can you take a look at this please?

I like this way better - good move @andycochran! I'll test and merge


DUB regression: describe fails if no D compiler found.

$ dub describe
Failed to invoke the compiler dmd to determine the build platform: '"dmd"' is not recognized as an internal or external command,
operable program or batch file.

This happens if no D compiler is found on the path. (Despite the error message, doesn't have to be dmd, any D compiler would make it work)
Used to work at least in DUB 0.9.21

This is a consequence of adding the new targets field to the dub describe output (targeted at build tools). Maybe we should just make that opt-out, and/or fail gracefully if no compiler was found.

This issue has become slightly more annoying now that DMD has been ported to D, and has a horrible startup performance: more than 3 seconds to do something as simple as dmd --version, on my Windows 32-bit machine. Because dub describe also calls DMD, the describe operation also becomes super slow (annoying for my DDT test suite).
The workaround I got is to not use DMD, but LDC instead, by making sure that DUB picks up the LDC compiler first. This works if I put LDC in the PATH after DMD's entry. (Odd, I thought the first PATH entry would take priority for DUB?)


Increase timeout for pulling large images

This is an addition to #631. Added 60 seconds, because of regular time-outs on bigger files.
Related to #630
Signed-off-by: Jeroen Knoops<EMAIL_ADDRESS>

Thanks for the PR @JeroenKnoops -- Can you fix your commit message so it passes the linting? (<72 characters for the body)

Thanks for the PR @JeroenKnoops -- Can you fix your commit message so it passes the linting? (<72 characters for the body)

fixed.. sorry! :)


Use HTTP codes to determine MessagingClientErrorCode

https://github.com/invertase/dart_firebase_admin/issues/22
@rrousselGit looking for some feedback on this. If y'all agree on the direction I will add tests.

I cut down on the amount of MessagingClientErrorCode. Felt like some were duplicated.
Added int? httpStatusCode to MessagingClientErrorCode.
Added more docs from Firebase docs.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Could a contributor please provide some feedback on this? I've been using this is a production environment successfully for weeks now.

@lesnitsky @Ehesp  I'd love to get feedback on this and come to a conclusion. Thanks!

Closed in favour of https://github.com/invertase/dart_firebase_admin/pull/43
Thanks for this though! It helped a lot to understand the problem :)


e2e-test: Add script to update server image

Add a script to redeploy the server in the kind cluster. It is useful to quickly test new changes without tearing down the cluster and bringing it up again.
Signed-off-by: Andrea Panattoni<EMAIL_ADDRESS>

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 50.946%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
862

Relevant Lines:
1692

💛  - Coveralls


Postgres: add ColumnExpression BigSerial and SmallSerial

There's a column expression for Serial https://github.com/JasperFx/weasel/blob/17951b8a57e968dff15c3c07ecfcaad2a8b49255/src/Weasel.Postgresql/Tables/Table.cs#L391, but as Postgres supports more serial types there should be expression for these too -- at least for bigserial.
I can submit PR if the additon is approved.

More info why it's needed...
With Marten a column can be created with
myTable.AddColumn("id", "bigserial").AsPrimaryKey();

// ...
await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync();

the the table gets created as expected, with Postgres creating the column with bigint and setting up the sequence.
But in the next run Marten (actually: Weasel) tries to update the column-definition from bigint to bigserial which results in an exception:
alter table public.XYZ alter column position type bigserial;

Unhandled exception. Marten.Exceptions.MartenSchemaException: DDL Execution for 'All Configured Changes' Failed!

alter table public.XYZ alter column position type bigserial;

 ---> Npgsql.PostgresException (0x80004005): 42704: type "bigserial" does not exist
   at Npgsql.Internal.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|213_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken)
   at Weasel.Postgresql.PostgresqlMigrator.executeDelta(SchemaMigration migration, DbConnection conn, AutoCreate autoCreate, IMigrationLogger logger)
  Exception data:
    Severity: ERROR
    SqlState: 42704
    MessageText: type "bigserial" does not exist
    File: parse_type.c
    Line: 270
    Routine: typenameType

So my naive assumption is that when BigSerial column-expression is supported, that use-case should work.

Sounds right. Are you open to contributing to adding that?

Yep, I'll create a PR.

Thank you :heart:


vulkaninfo: Fix window\surface lifetime

Looks like there are some merge conflicts.
Also, remember to clang-format your code.

Also, remember to clang-format your code.

DAMMIT, not again!
Yea the offending commit seems to do something similar to this. It does not seem to manage window lifetime though, which I assumed would be the problem; does it actually work to fix #136 (I am still waiting on confirmation for this PR too)? I may be biased but my commit feels cleaner managing the lifetime on the same scope.

Clang-format gets me a lot too.  In fact, I have a piece of paper taped to my monitor to remind me.
It won't offended me to revert the commit.  Your implementation looks cleaner.  I am a bit curious why you are using an array instead of a linked list to handle multiple surfaces.

I am not a natural C programmer, so I welcome convention notes.
In C++ I would use std::vector. So I do the same here the closest I can approximate it...

Not sure what the CI complains about now. Seems glitched. Would you PTAL?

@jeremyk-lunarg  It would be good to run some sample output thru JSON validation.  I can show you how to do that if needed.
Unfortunately, I've not found an automatable tool that we could put in CI; the validators I've found are all websites into which you paste text and press a button.  If anyone has recommendations for automatable tools, I'd welcome feedback.


Change width in the home page?

I'm just starting playing around with the theme, which is super cool, actually!
I have managed to increase the width of the pages, but I can't figure out how to increase (say, to 80%) the home page. Is there an easy way to do so?
Thanks
Gianluca

You would need to use custom CSS (refer to Academic's documentation website). Then you can try something similar to:
.container {
    width: 80% !important;
}

Great --- thanks! I'm fiddling with the theme and it's very interesting. Not sure I found what I like, but your answer is very helpful. Many thanks!

@gcushen I added
.container {
    width: 80% !important;
}

inside ./assets/scss/custom.scss and serve the website with hugo serve but I don't see any difference.
but it work when I add this in my custom.scss
it is the right way ??
@media (min-width: 576px) {.container-sm, .container {max-width: 80%; }}
@media (min-width: 768px) {.container-md, .container-sm, .container {max-width: 80%;}}
@media (min-width: 992px) {.container-lg, .container-md, .container-sm, .container {max-width: 80%;}}
@media (min-width: 1200px) {.container-xl, .container-lg, .container-md, .container-sm, .container {max-width: 80%;}}

also if I try
.container {max-width: 80% !important;} the navbar don't align with the content


(PE-25560) Add a status event to resource callback

After this change, when resources are sent a callback they will also get a status event.  This event is similar to the sync status event but without any property details, since there is no property associated with a callback on a resource.

CLA signed by all contributors.

Closing this to rename branch/associate with PUP-9339.


Convert C++ mutation classes to polymorphic value types

Note:

There remain some parts of these that are unimplemented; filling those out will will come separately.
Callers still use e.g. std::unique_ptr<Mutation>, mostly tests. This is not harmful because this code isn't yet used in any production capacity. I'll address it separately.

The general pattern here was laid down in #3529, and all the general commentary about how this works still applies. My take is that this approach is about as close as we're going to get to having polymorphic C++ "just work":

You can add these things to collections and operator== does the right thing.
You can accept a parameter of const Mutation& and subclasses are accepted
You can accept a parameter of Mutation and subclasses are also accepted without slicing
Subclass types can be recovered with a casting constructor, similar to the way we cast on the other platforms.
Far less e.g. std::shared_ptr<const Mutation> littering every parameter and variable declaration we have.

The cost is:

Base classes have to duplicate their external interface: once as virtual methods on the protected inner class and again as non-virtual methods that delegate
Subclasses have to implement a few extra constructors
Subclasses are mostly implemented in the Rep rather than where you might logically think they go using conventional polymorphism

Note the diffbase on #3551.


Fastest implementation of an elementwise gradient?

Not sure if this is the place to post this, so let me know if I should be putting this elsewhere.
I'm trying to implement gradient-enhanced kriging with JAX. Ideally, the user should be able to define a kernel and JAX will be able to automatically find its gradient/hessian and generate the appropriate covariance matrix. Here's what I've tried so far:
Kernel Implementation
import jax.numpy as np
from jax import jit, jacrev, jacfwd, vmap
from abc import ABC, abstractmethod

class Kernel(ABC):
    def __init__(self, name: str, debug=True):
        self.name = name

    def __call__(self, x1: np.ndarray, x2: np.ndarray):
        raise NotImplementedError()

    @abstractmethod
    def forward(self, x1: np.ndarray, x2: np.ndarray, thetas: np.ndarray):
        raise NotImplementedError()

class RBF(Kernel):
    def __init__(self, length_scale=1.0):
        self.length_scale = length_scale
        super(RBF, self).__init__('RBF')

    def __call__(self, x1: np.ndarray, x2: np.ndarray):
        return self.forward(x1, x2, np.array([self.length_scale]))

    def forward(self, x1: np.ndarray, x2: np.ndarray, thetas: np.ndarray):
        assert thetas.shape == (1,)
        length_scale = thetas[0]

        # d = np.linalg.norm(
        #     np.expand_dims(x1, 0) - np.expand_dims(x2, 1),
        #     axis=-1
        # ) / length_scale
        # return np.exp(-0.5 * (d * d))
        d = (np.expand_dims(x1, 0) - np.expand_dims(x2, 1)) / length_scale
        d = d * d
        d = np.sum(d, axis=-1)
        return np.exp(-0.5 * d)

class RBFGrad(RBF):
    def __init__(self, length_scale=1.0):
        super(RBFGrad, self).__init__(length_scale)
        self.dkdx1 = jit(jacfwd(super(RBFGrad, self).forward, argnums=0))
        self.dkdx2 = jit(jacfwd(super(RBFGrad, self).forward, argnums=1))
        self.dk2dx1dx2 = jit(jacfwd(jacrev(super(RBFGrad, self).forward, argnums=0), argnums=1))

    def forward(self, x1: np.ndarray, x2: np.ndarray, thetas: np.ndarray):
        K = super().forward(x1, x2, thetas)
        dx2 = self.dkdx2(x1, x2, thetas).sum(-2)
        upper = np.concatenate([dx2[:, :, i] for i in range(dx2.shape[-1])], axis=1)
        dx1 = self.dkdx1(x1, x2, thetas).sum(-2)
        left = np.concatenate([dx1[:, :, i] for i in range(dx1.shape[-1])], axis=0)
        dx1dx2 = self.dk2dx1dx2(x1, x2, thetas).sum(2).sum(-2)
        dx2_concatenated = np.concatenate([
            dx1dx2[:, :, :, i]
            for i in range(dx1dx2.shape[-1])
        ], axis=1)
        hess = np.concatenate([
            dx2_concatenated[:, :, i]
            for i in range(dx2_concatenated.shape[-1])
        ], axis=0)

        # form the overall covariance matrix
        # [
        #     [K,       dK/dx2      ],
        #     [dK/dx1,  dK^2/dx1dx2 ]
        # ]
        return np.concatenate([
            np.concatenate([K, upper], axis=1),
            np.concatenate([left, hess], axis=1)],
            axis=0
        )

Test
import pytest
from gpgrad.kernels import *
import jax.numpy as np
import numpy as onp
import jax

def test_rbf():
    a = np.linspace(0, 10).reshape(-1, 1)
    k = RBF(1.0)
    result = k(a, a)
    assert result.shape == (len(a), len(a))
    assert np.allclose(np.diag(result), 1.0)
    assert np.allclose(k(a, a), k(a.reshape(-1, 1), a.reshape(-1, 1)))

def test_rbfgrad():
    a = np.linspace(0, 10).reshape(-1, 1)
    b = np.linspace(0, 10).reshape(-1, 1)

    k = RBFGrad(1.0)
    result = k(a, b)
    assert result.shape == (2*len(a), 2*len(a))
    # diagonals of kernel matrix should be 1.0
    assert np.allclose(np.diag(result[:len(a), :len(a)]), 1.0)
    # diagonals of derivative matrices wrt. x1 or x2 should be 0.0
    assert np.allclose(np.diag(result[:len(a), len(a):]), 0.0)
    assert np.allclose(np.diag(result[len(a):, :len(a)]), 0.0)
    # diagonals of hessian wrt x1 and x2 should be 1.0
    assert np.allclose(np.diag(result[len(a):, len(a):]), 1.0)

I think this gets me what I want, though I'm admittedly a little confused by the resulting shapes of the derivatives (dx1 and dx2 in RBFGrad). For inputs of shape (N, d), dx1 and dx2 shapes are (N, N, N, d). I think this is because JAX is trying to take the derivative of each row of x1/x2 wrt each element of the NxN matrix of RBF.forward(). This results in lots of 0.0's in the derivatives (see dx1 or dx2 in RBFGrad.forward() before .sum(-2)). This suggests O(N) operations that simply evaluate to 0.0. This is exacerbated when I evaluate the second derivative matrix, making the code quite slow to run for a modest number of rows for x.
On my laptop, test_rbf() runs in about 600ms without JIT compiling. test_rbfgrad() runs in about 15s even with JIT compiling on the derivative functions. The main bottleneck there is the second derivative function.
Is there a faster way to implement this in JAX, perhaps by eliminating the numerous operations that result in 0's in the derivatives?

If you don't want to calculate a bunch of off-diagonal terms, you might try using vmap on top of jacfwd or jacrev.

Thanks for the question! I agree with what @shoyer said.
To say a bit more, if f: R^n -> R^m then we expect its Jacobian to be an m x n matrix. The function being differentiated here, namely RBF.forward with respect to its first argument, takes an input of shape (50, 1) and produces an output of shape (50, 50), which is why we expect to see a Jacobian of shape (50, 50, 50, 1). (If we want to think in flattened terms, we'd say that the input has dimension n=50 and the output has dimension m=2500.)
The trouble here is we just want to see the data axes of size 50 as batch axes. That is, we really want to think of the kernel as taking two vectors and outputting a scalar. We only carry along the batch axis in the implementation for vectorization efficiency.
But with vmap, we don't need to batch by hand, and as a result it can be easier to express the functions and derivatives we want. Concretely, take a look at the forward_ and dkdx2_ methods below (note the underscores on the end), and the assertion:
class RBF(Kernel):
    # ...

    def forward_(self, x1: np.ndarray, x2: np.ndarray, thetas: np.ndarray):
        assert thetas.shape == (1,)
        length_scale = thetas[0]
        dist_sq = np.vdot(x1, x1) + np.vdot(x2, x2) - 2 * np.vdot(x1, x2)
        return np.exp(-0.5 * dist_sq / length_scale**2)

class RBFGrad(RBF):
    def __init__(self, length_scale=1.0):
        super(RBFGrad, self).__init__(length_scale)
        self.dkdx1 = jit(jacfwd(super(RBFGrad, self).forward, argnums=0))
        self.dkdx2 = jit(jacfwd(super(RBFGrad, self).forward, argnums=1))
        self.dk2dx1dx2 = jit(jacfwd(jacrev(super(RBFGrad, self).forward, argnums=0), argnums=1))

        self.dkdx2_ = jit(vmap(vmap(grad(super().forward_, argnums=1), (0, None, None)), (None, 0, None)))

    def forward(self, x1: np.ndarray, x2: np.ndarray, thetas: np.ndarray):
        K = super().forward(x1, x2, thetas)
        dx2 = self.dkdx2(x1, x2, thetas).sum(-2)
        dx2_ = self.dkdx2_(x1, x2, thetas)
        assert np.allclose(dx2, dx2_)
        # ...

I rewrote the forward_ method so that it more obviously applies to single vectors at a time, and also to use the polarization identity ||u - v||^2 = ||u||^2 + ||v||^2 - 2 u \cdot v which gives us more matrix multiplies and I think is often faster (though one should benchmark against computing ||u - v||^2 more directly as before). I also added the dkdx2_ method, which uses vmap to do all the batching, and compared against the old calculation (it may be faster to use jvp than grad but this was convenient). That second bit, with the vmaps, is the main thing I wanted to illustrate, as it was basically @shoyer's advice.
One other piece of advice would be to put jit on RBFGrad.forward, because the more code XLA can see the more it can optimize things.
WDYT?

There may be a nicer way to write the nested vmap using the vectorize wrapper.

One more thought: to get the second derivatives, you just need to write grad(grad(...)) inside the double-vmap. Getting rid of batch dimensions makes things so much easier!

It might be worth noting that this vmap + auto-diff strategy is actually rather fundamental to JAX. Even jacfwd and jacrev themselves are mere thin wrappers over vmap + jvp/vjp: https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html#Composing-VJPs,-JVPs,-and-vmap

Oh vmap, of course! Thanks @shoyer and @mattjj, I'll check in again if I run into more trouble.

This works great, and is much faster than my original implementation! Though the second derivative will need jacrev(grad(...)) as grad returns a vector.


Also sign InRelease for Jessie & Stretch (not just Xenial)

InRelease is only signed for Xenial due the logic on line 106.
This change causes it to also be signed for Debian Jessie & Stretch.
Its not the most elegant fix, An array of release names that require InRelease signing should probably be set somewhere and if release_array.include? "self.codename"  used.
I also had to regenerate the signing key to sign with SHA-256 by adding these lines to ~/.gnupg/gpg.conf
After this my apt updates worked with 0 errors.
personal-digest-preferences SHA256
cert-digest-algo SHA256

@russmac - Thanks for this. Thanks too for your hint on gpg conf. FWIW I didn't need to regenerate keys, changing those settings seemed to be enough to resolve my issues with a Stretch repo?!

@JedMeister Thanks for pointing that out, I think I had two seperate issues. My key was a SHA type considered sufficiently weak enough that it did need to be redone for apt-secure to consider it acceptable, It also wasn't being used to sign. Yours was probably one it accepted.

@krobertson Any comments on this? It resolves a significant issue.

Shouldn't this an option from the command line in addition of the automatic guessing ?
I'm running a repo not tied to a specific distribution (as it only provide an executable), and I would like to provide the InRelease too.


HTTP chunks instead of WebSockets

We decided to use HTTP chunks instead of WebSockets but I think we have a problem. [This] (https://groups.google.com/forum/#!msg/golang-nuts/nIaN09QmE3k/ohmbp4uhv6QJ) states that we cannot flush our http.ResponseWriter and due to that we cannot get the progress in the xhr request from the client.
I'd suggest sending back the list of tests that are pending in the POST /results/submission request. Every test.Call(...) will be a separate go routine that will add the results to memcache once they are finished. We can pool every 10 seconds to get the results and the pending tests from there. WDYT?

I will give it a try and make the App Engine implementation a http.Flusher. If that does not work, your proposal is our last resort.

Closing as we're now using a WebSocket proxy and this works reasonably well.


Example links in documentation of latest release point to current master implementation

This example on the master branch seems a little outdated. Only by reading through other issues in this repo I was able to find this new example.
I'd like to suggest that at least the new example should get referenced in the newest documentation. https://docs.rs/argmin/0.4.7/argmin/solver/simulatedannealing/struct.SimulatedAnnealing.html Currently, it links to the old, outdated one.

Actually the example on the master branch is the newer one, but if you're using 0.4.7 then you will need the "older one".
Thanks for reporting this though, because the documentation on docs.rs is wrongly linking to the example on the master branch, which is not helpful at all. I will fix this hopefully soon.

but if you're using 0.4.7 then you will need the "older one".

I'm not sure, what you mean. Isn't 0.4.7 the newest version? Or is the one on the master branch already the example for 0.4.8 or 0.5.0?
Anyway, thank you for working on fixing the issue! :)

The master branch is currently ahead of 0.4.7 (which is the newest released version). In the master branch you will also find changes which aren't released yet. In this case for instance #139, which allows one to choose the random number generator used in SA.

I tried to fix this, but none of the options I could come up with was satisfactory:

Link to the latest release in Github. While it is possible to link to the tarball of the latest release, it is not possible to directly link to the code of the latest release.
Alternatively one could link to the tag of the latest release, which is possible; however, this would require manually changing the links in the documentation prior to every release. Both 1. and 2. would fix the problem for the documentation on docs.rs, but would break for the one on argmin-rs.github.io.
Add examples to the documentation. It is possible to document examples (via doc = true in Cargo.toml); however, this doesn't work for examples which require dev-dependencies, because those are not compiled for some reason. Even if it would work, the examples would be listed on the left bar, right underneath the library name, potentially leading to confusion. Some issues/PRs that might be of relevance:

https://github.com/rust-lang/docs.rs/issues/132
https://github.com/rust-lang/docs.rs/issues/238
https://github.com/rust-lang/rfcs/pull/3123
https://github.com/rust-lang/cargo/issues/3475

I would prefer it if it was possible to add the examples to the documentation directly. Since this is not possible right now I'm not sure how to properly address this right now. I might just remove the individual links in the documentation and have two dedicated links to the examples on the main page of the documentation.

It's unfortunate that there's no optimal solution atm. Thank you for doing your best though!


[SUPPORT] 100% update on 10mn keys in single partition slow

I have a single 700MB file containing 10mn rows (all unique keys, key is single column, single partition for all rows).

Create brand new table
Using spark datasource on the 700MB to write in COW insert mode, all rows to single partition takes 7.3mins (i thought could be faster but its acceptable). 88 parquet files end up on S3 table path after this succeeds (total size 288MB)
Repeat the exact same spark-submit (but with COW upsert mode) from point 2 above to single partition with exact same 700MB file (takes 5hrs to finish)

--total-executor-cores 16 --driver-memory 2G --executor-memory 16G --executor-cores 2
dynamic allocation of executors is on, shuffle service is on
spark 2.4.6
hudi 0.5.3
"spark.sql.shuffle.partitions" is "16"
"hoodie.insert.shuffle.parallelism" is "16"
"hoodie.upsert.shuffle.parallelism" is "16"
private static final String DEFAULT_INLINE_COMPACT = "true";
private static final String DEFAULT_CLEANER_FILE_VERSIONS_RETAINED = "1";
private static final String DEFAULT_CLEANER_COMMITS_RETAINED = "1";
private static final String DEFAULT_MAX_COMMITS_TO_KEEP = "3";
private static final String DEFAULT_MIN_COMMITS_TO_KEEP = "2";
private static final String DEFAULT_EMBEDDED_TIMELINE_SERVER_ENABLED = "false";
private static final String DEFAULT_FAIL_ON_TIMELINE_ARCHIVING_ENABLED = "false";
rest of configs are default

log shows INFO  org.apache.hudi.index.bloom.HoodieBloomIndex - InputParallelism: ${16}, IndexParallelism: ${0}
shuffle size seems to be extremely high! any idea how to speed this up? how long does it take you to do 100% update? ie run same 10mn/700MB file twice on new table
UPDATE:
most time spent:

countByKey at WorkloadProfile.java:67
2020/07/15 10:08:24
4.8 h

Details for Stage 13 (Attempt 2)
Total Time Across All Tasks: 5.8 h
Locality Level Summary: Process local: 12
Shuffle Read: 15.7 GB / 602006097
Shuffle Write: 153.4 MB / 6839204
Shuffle Spill (Memory): 287.5 GB
Shuffle Spill (Disk): 10.6 GB
As a test i repeated same steps on a different table using hoodie 0.4.6 in spark local mode and it only took 173mins

Can you try setting the config "hoodie.bloom.index.bucketized.checking" to false and try. Kindly report back with the observation.

It looks like this is the same setting in 0.4.6. Can you try checking other configs related to index lookup between 0.4.6 and 0.5.3 to see if any defaults are changes. I will check back in a day or so.

Also wondering if we are indeed comparing apples to apples. Are both the runs (0.4.6 vs 0.5.3) happening in the same setup ?

Actually was using 0.4.6-SNAPSHOT before bucketized.checking code landed. I changed hoodie.bloom.index.bucketized.checking to false on hudi 0.5.3 and time down to 107mins :) I will now try hudi 0.5.3 in local mode

@bvaradar this image shows tasks for flatMapToPair at HoodieBloomIndex.java:308 stage, how can the number of records/shuffle size be so much larger than both my input file and existing table?

@tooptoop4 : Can you provide us the spark DAGs with times (Job, Stage and Task level) between 0.5.3 (with bucketized bloom index on) and 0.5.3 (with bucketized bloom index off). We need to see why you are seeing such a massive performance difference.
Regarding your question, Please take a look at the comment in https://github.com/apache/hudi/blob/master/hudi-client/src/main/java/org/apache/hudi/index/bloom/HoodieBloomIndex.java#L249
This is basically an exploded RDD of record-Key with files to be compared.
Thanks,
Balaji.V

@tooptoop4 : Pinging again to see if you can get us the information between the 2 runs w/o bucketized bloom index

@bvaradar i noticed "There is insufficient memory for the Java Runtime Environment to continue." error so i reduced SPARK_WORKER_MEMORY (ie leave more room for OS memory). Now the timings I get are: 43mins for hoodie.bloom.index.bucketized.checking = false. 59 mins for hoodie.bloom.index.bucketized.checking = true.
hoodie.bloom.index.bucketized.checking = false

hoodie.bloom.index.bucketized.checking = true


ARTEMIS-4906 CentOS 7 is EOL so remove Docker file

A grep for "centos" shows some remaining references to this (or older Java 11 or generic variants), e.g in the script for preparing the images, and some tests.
Seems like they should be going at the same time.

@gemmellr, those issues should be resolved now. Thanks for the review!


Document aliasing rules for resources

Fixes #314

This change is

r? @Xaeroxe

Thanks for the review!
bors r+


bump hspec-discover

I am not opposed to having all of the cabal patches in one PR. :P

Yeah, I know but I don't create tasks that way.  I've just been letting it build up due to getting open source permissions too.  It should be nearly over.


[deliver] Add old Release notes to new version

How can i copy the old release notes to the new version. Because some of of the release notes are everytime the same like please rate the app, ... and then only the changes will change but this i will do manually at itunesconnect.
Any idea how to do this?
Thanks :)

Hey @appcelerate, there currently is not a way to do this, but if you would like to submit a PR with this feature, that would be amazing! I am going to close this issue for now and mark it as a feature request and please feel free to open it if you have some time to bite off that work! 🚀