BUG: GPU Python Wrapping Fails on Linux

Description

Building ITK with ITK_USE_GPU=ON and ITK_WRAP_PYTHON=ON on Linux results in a build failure.
Steps to Reproduce

Configure ITK with ITK_USE_GPU=ON, ITK_WRAP_PYTHON=ON. ITK_USE_SYSTEM_FFTW was also turned ON to use cuFFTW. Then, build ITK.
Expected behavior

The build succeeds and Python wrappings are generated.
Actual behavior

[1/38] Generating .pyi files for Python wrapper interface
FAILED: Wrapping/CMakeFiles/itk-stub-files Wrapping/Generators/Python/itk/_proxies.pyi Wrapping/Generators/Python/itk/__init__.pyi /home/tom/builds/ITK-base-dev/Wrapping/CMakeFiles/itk-stub-files /home/tom/builds/ITK-base-dev/Wrapping/Generators/Python/itk/_proxies.pyi /home/tom/builds/ITK-base-dev/Wrapping/Generators/Python/itk/__init__.pyi
cd /home/tom/builds/ITK-base-dev/Wrapping && /home/tom/anaconda3/bin/python3.8 /home/tom/repos/ITK/Wrapping/Generators/Python/itk/pyi_generator.py --pyi_dir /home/tom/builds/ITK-base-dev/Wrapping/Generators/Python/itk --pkl_dir /home/tom/builds/ITK-base-dev/Wrapping/Generators/Python/itk-pkl --index_list_file /home/tom/builds/ITK-base-dev/Wrapping/GlobalIdxFilesList.txt
Traceback (most recent call last):
  File "/home/tom/repos/ITK/Wrapping/Generators/Python/itk/pyi_generator.py", line 609, in <module>
    class_name = unpack(class_files, options.pyi_dir)
  File "/home/tom/repos/ITK/Wrapping/Generators/Python/itk/pyi_generator.py", line 391, in unpack
    base = merge(class_definitions)
  File "/home/tom/repos/ITK/Wrapping/Generators/Python/itk/pyi_generator.py", line 450, in merge
    raise AssertionError(
AssertionError: The basic values of two associated classes do not match.
Item 1: class `itkCastImageFilter::CastImageFilter` has_new_method: `True` typed: `True` is_abstract: `False` is_enum: `False` has_superclass: `True`Item 2: class `itkGPUCastImageFilter::CastImageFilter` has_new_method: `True` typed: `True` is_abstract: `False` is_enum: `False` has_superclass: `False`
[14/38] Linking CXX shared module Wrapping/Generators/Python/itk/_ITKMetricsv4Python.so
ninja: build stopped: subcommand failed.

Reproducibility

100%
Versions

ITK 5.3.0
Environment

Ubuntu 20.04
CMake 3.21.3
Ninja 1.10.0
Additional Information

Likely due to some POINTER vs POINTER_WITH_SUPERCLASS mismatch for GPUCastImageFilter's wrapping.
Found in attempt to reproduce related https://github.com/InsightSoftwareConsortium/ITK/issues/3744

I can confirm this, building on Mac too, with ITK_USE_GPU=ON and ITK_WRAP_PYTHON=ON:
cd /opt/local/var/macports/build/_Users_nilason_ports_graphics_InsightToolkit-devel/py311-InsightToolkit-devel/work/build/Wrapping && /opt/local/bin/bin/python3.11 /opt/local/var/macports/build/_Users_nilason_ports_graphics_InsightToolkit-devel/py311-InsightToolkit-devel/work/InsightToolkit-5.3.0/Wrapping/Generators/Python/itk/pyi_generator.py --pyi_dir /opt/local/var/macports/build/_Users_nilason_ports_graphics_InsightToolkit-devel/py311-InsightToolkit-devel/work/build/Wrapping/Generators/Python/itk --pkl_dir /opt/local/var/macports/build/_Users_nilason_ports_graphics_InsightToolkit-devel/py311-InsightToolkit-devel/work/build/Wrapping/Generators/Python/itk-pkl --index_list_file /opt/local/var/macports/build/_Users_nilason_ports_graphics_InsightToolkit-devel/py311-InsightToolkit-devel/work/build/Wrapping/GlobalIdxFilesList.txt
Traceback (most recent call last):
  File "/opt/local/var/macports/build/_Users_nilason_ports_graphics_InsightToolkit-devel/py311-InsightToolkit-devel/work/InsightToolkit-5.3.0/Wrapping/Generators/Python/itk/pyi_generator.py", line 609, in <module>
    class_name = unpack(class_files, options.pyi_dir)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/var/macports/build/_Users_nilason_ports_graphics_InsightToolkit-devel/py311-InsightToolkit-devel/work/InsightToolkit-5.3.0/Wrapping/Generators/Python/itk/pyi_generator.py", line 391, in unpack
    base = merge(class_definitions)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/var/macports/build/_Users_nilason_ports_graphics_InsightToolkit-devel/py311-InsightToolkit-devel/work/InsightToolkit-5.3.0/Wrapping/Generators/Python/itk/pyi_generator.py", line 450, in merge
    raise AssertionError(
AssertionError: The basic values of two associated classes do not match.
Item 1: class `itkCastImageFilter::CastImageFilter` has_new_method: `True` typed: `True` is_abstract: `False` is_enum: `False` has_superclass: `True`Item 2: class `itkGPUCastImageFilter::CastImageFilter` has_new_method: `True` typed: `True` is_abstract: `False` is_enum: `False` has_superclass: `False`
make[2]: *** [Wrapping/CMakeFiles/itk-stub-files] Error 1

Switching to ITK_USE_GPU=OFF, makes it build successfully

I can confirm this bug too from the latest sources.


DX Improvement: recover from failed DB

Hello maintainers.
I followed the guide in development/getting-started.rst but had some issues.
I did not have enough resources allocated, and followed the section:
make initdb is slow or appears to make no progress
without success.
This patch fixes three issues in recovery from a failed DB setup:

if xz has decompressed a file already, it fails to work on a second attempt, and stops make initdb
make clean does not remove files previously created by make initdb
The documentation states to run make serve and then kick off make initdb. This will not work if make serve has gotten far enough to open a web session. I changed the documentation to something that worked for me.

Please accept these changes.

Thanks @matt-land!


[$250] FAB - Nothing happens when Start chat is clicked after creating a room offline

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.65-0
Reproducible in staging?: Y
Reproducible in production?: Y
Found when executing: https://github.com/Expensify/Expensify/issues/302054
Logs: https://stackoverflow.com/c/expensify/questions/4856
Issue reported by: Applause-Internal team
Action Performed:

Log in with a new account
Create a workspace
Go offline
Create a new room
Navigate to the LHN
Navigate to FAB - Start chat

Expected Result:
Start chat modal should open.
Actual Result:
Nothing happens when Start chat is clicked after creating a room offline. On iOS and Android native, you can see it opening for a short time but it disappears after. On mWeb Safari, the page shakes after the tap.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/115492554/e4b393d6-5868-489c-ad9d-6e7e5ea296a4

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~0114d6fec2c1cc3911
Upwork Job ID:<PHONE_NUMBER>872132096
Last Price Increase: 2024-04-29

Proposal
Please re-state the problem that we are trying to solve in this issue.
When we open start chat again after creating a room while offline, nothing happen.
What is the root cause of that problem?
When we create a new room, it will optimistically set the loading to the new room form
https://github.com/Expensify/App/blob/08b7ff41656a34373761fb2bd505cc0dfa64be7b/src/libs/actions/Report.ts#L1955-L1957
Because we are offline, the loading state is always true. When we reopen the page again, it will immediately close because of the below effect.
https://github.com/Expensify/App/blob/08b7ff41656a34373761fb2bd505cc0dfa64be7b/src/pages/workspace/WorkspaceNewRoomPage.tsx#L146-L152
The (isOffline && isLoading) condition is true. When the page is mounted, we clear the loading state (set to false)
https://github.com/Expensify/App/blob/08b7ff41656a34373761fb2bd505cc0dfa64be7b/src/pages/workspace/WorkspaceNewRoomPage.tsx#L128-L130
and this condition (wasLoading && !isLoading) is also becomes true.
What changes do you think we should make in order to solve the problem?

The useEffect is added (in https://github.com/Expensify/App/pull/31655) so it will only close the page when the request is successful, but we added back the code to close it immediately in Report.addPolicyReport, so it's really weird, I think it's a bad merge, so we can remove the dismiss code here.
https://github.com/Expensify/App/blob/08b7ff41656a34373761fb2bd505cc0dfa64be7b/src/libs/actions/Report.ts#L2012-L2013
To fix this issue, we can set initWithStoredValues: false to the new room form state. This way, the data will always be empty initially. We don't want the previous state to affect the new page.
formState: {
    key: ONYXKEYS.FORMS.NEW_ROOM_FORM,
    initWithStoredValues: false,
},

Then, we can remove this effect
https://github.com/Expensify/App/blob/08b7ff41656a34373761fb2bd505cc0dfa64be7b/src/pages/workspace/WorkspaceNewRoomPage.tsx#L128-L130
Or trigger it when it's unmounted instead.
NOTE: there is currently a bug in Onyx where initWithStoredValues doesn't work. To fix it we should initialize the onyx key only if initWithStoredValues is true.
-if (
+if (mapping.initWithStoredValues &&
    ((value !== undefined
        && !Onyx.hasPendingMergeForKey(key))
    || mapping.allowStaleData)
) {

@mollfpr there's a proposal above for you to take a look at

@bernhardoj Is there any conversation regarding the bug of initWithStoredValues? It should be critical if the property doesn't work.

@mollfpr I'm not aware of any conversation of that

The @bernhardoj proposal looks good to me! For the issue on initWithStoredValues, I think we can decide after the PR is created.
🎀 👀 🎀 C+ reviewed!

For the issue on initWithStoredValues, I think we can decide after the PR is created.

Sorry, I don't follow what this means. Mind explaining for me?

I notice that initWithStoredValues is only works for the first time the page rendered.
https://github.com/Expensify/App/assets/25520267/15d30dbe-7296-4ee0-b70a-c875732004cd
Here's in the video, after the refresh and then opening the new room page, you can see in the console log it will show undefined which is expected because I set initWithStoredValues to false. But the second time I opened the page, it didn't show undefined but instead the value stored.
I'm not sure if this is expected and what causing the issue, but @bernhardoj may be another option is to clear the formState after it unmounts can be the option to dodge the issue?

may be another option is to clear the formState after it unmounts can be the option to dodge the issue?

It won't work if the user closes the app/tab. We need to fix the onyx first before creating the App PR. We can create a local state of loading with an initial value of false, but I think it's better to fix the onyx instead of avoiding it.

@bernhardoj Could you create a discussion of the onyx issue? We can confirm if it is an issue before fixing it.

I think the property name is self-explanatory and currently it doesn't work as the name and comment suggests,

but here is the discussion.

Ok understood. So my apologies, can you summarize for me the changes we'll be making in Onyx and in App?

@mollfpr so, based on the discussion, it's confirmed the current behavior of initWithStoredValues is wrong. To further prove that it's a bug, we can run a unit test. In onyx repo itself, there is already a unit test for withOnyx + initWithStoredValues, but the test case is currently wrong.

You can see the key name that is set to Onyx ('test_key') and is connected to the component ('test') is different. That's why the test will always pass. You can update the 'test_key' to 'test' and see it will fails.
@puneetlath the plan will be,

Update the onyx unit test case
Apply the onyx initWithStoredValues bug fix in here

Update the onyx in the App and apply the App fix

the same issue actually happens in useOnyx hook too, but I'm not confident to fix the new hook. initWithStoredValues currently only works with Onyx.connect

@bernhardoj The plan looks good to me!

Thanks for laying that all out @bernhardoj. Sounds great.

The Onyx PR is here
cc: @mollfpr @puneetlath

Onyx PR is merged. The current onyx version is 2.0.32 but we can't update it yet because 2.0.33 version has a regression, so we need to wait until the regression is fixed.

@bernhardoj We have this bump https://github.com/Expensify/App/pull/42057, are the changes included in the 2.0.40?

Our Onyx PR is in 2.0.39, so yes it will be included in 2.0.40. We can hold for that then.

Do we have an issue for the Onyx bump?

I think Onyx has been bumped past this version. What's left to do now?

What's left is the App PR. Here is the App PR.
cc: @mollfpr

@mollfpr bump on the checklist so we can pay.

@puneetlath I think we still need to wait a bit based on my comment here

@puneetlath The Onyx PR is merged. I think we can have QA retest this.

Bug fixed and not reproducible
https://github.com/Expensify/App/assets/43996225/6fc4ce3e-f106-4fa8-83b5-1168f9bca6bf

Ok, so we're ready for the checklist now yeah @bernhardoj @mollfpr? So that we can pay?

Yes @mollfpr

[@mollfpr] The PR that introduced the bug has been identified. Link to the PR:
[@mollfpr] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:

No offending PR.

[@mollfpr] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
[@mollfpr] Determine if we should create a regression test for this bug.

The regression step should be enough.

[@mollfpr] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.

Go offline
Create a new room from FAB
Open the new room page again from FAB
Verify the new room page is shown
👍 or 👎

@mollfpr - $250 C+

$250 approved for @mollfpr

@muttmuure what is left to do here? Can we close?

I think the payment for me is still pending.

@bernhardoj paid $250 for C


[ADAM-743] exclude conflicting jackson dependencies

Fixes #743

What are the conflicting jackson dependencies used for in utils?
Nit: can you prefix the commit message with [ADAM-774]?

HEAD contains versions 2.1.x and 2.3.x
$ mvn dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building ADAM_2.10: CLI 0.17.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ adam-cli_2.10 ---
[INFO] org.bdgenomics.adam:adam-cli_2.10:jar:0.17.1-SNAPSHOT
...
[INFO] +- org.apache.spark:spark-core_2.10:jar:1.2.0:provided
...
[INFO] |  +- org.json4s:json4s-jackson_2.10:jar:3.2.10:provided
[INFO] |  |  +- org.json4s:json4s-core_2.10:jar:3.2.10:provided
[INFO] |  |  |  +- org.json4s:json4s-ast_2.10:jar:3.2.10:provided
[INFO] |  |  |  \- org.scala-lang:scalap:jar:2.10.0:provided
[INFO] |  |  |     \- org.scala-lang:scala-compiler:jar:2.10.0:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:provided
...
[INFO] +- org.bdgenomics.utils:utils-io_2.10:jar:0.2.2:compile
[INFO] |  +- org.bdgenomics.utils:utils-misc_2.10:jar:0.2.2:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.3.2:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.3.1:compile
[INFO] |  \- com.amazonaws:aws-java-sdk:jar:1.7.5:provided
[INFO] |     +- com.fasterxml.jackson.core:jackson-core:jar:2.1.1:provided
[INFO] |     +- com.fasterxml.jackson.core:jackson-annotations:jar:2.1.1:provided
...

vs. 2.3.x for this pull request
$ mvn dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building ADAM_2.10: CLI 0.17.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ adam-cli_2.10 ---
[INFO] org.bdgenomics.adam:adam-cli_2.10:jar:0.17.1-SNAPSHOT
...
[INFO] +- org.apache.spark:spark-core_2.10:jar:1.2.0:provided
...
[INFO] |  +- org.json4s:json4s-jackson_2.10:jar:3.2.10:provided
[INFO] |  |  +- org.json4s:json4s-core_2.10:jar:3.2.10:provided
[INFO] |  |  |  +- org.json4s:json4s-ast_2.10:jar:3.2.10:provided
[INFO] |  |  |  \- org.scala-lang:scalap:jar:2.10.0:provided
[INFO] |  |  |     \- org.scala-lang:scala-compiler:jar:2.10.0:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:provided
[INFO] |  |     +- com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:provided
[INFO] |  |     \- com.fasterxml.jackson.core:jackson-core:jar:2.3.1:provided
[INFO] +- org.bdgenomics.utils:utils-io_2.10:jar:0.2.2:compile
[INFO] |  +- org.bdgenomics.utils:utils-misc_2.10:jar:0.2.2:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.3.2:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.3.1:compile
[INFO] |  \- com.amazonaws:aws-java-sdk:jar:1.7.5:provided
...

Ah. I was more going for, what code in utils has the jackson dependency? I think we may be able to fix that in utils by removing or upgrading the Jackson dep; it may be old?

From what I can see there isn't a direct dependency.  It appears to be a transitive dependency where old versions are pulled in by aws-java-sdk in the module utils-io.
The class in the stack trace is part of Apache Spark
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/sink/MetricsServlet.scala

Ah, OK! SGTM.

Merged! Thanks @heuermh!


[SPARK-30265][INFRA] Do not change R version when releasing preview versions

What changes were proposed in this pull request?
This PR makes it do not change R version when releasing preview versions.
Why are the changes needed?
Failed to make Spark binary distribution:
++ . /opt/spark-rm/output/spark-3.0.0-preview2-bin-hadoop2.7/R/find-r.sh
+++ '[' -z /usr/bin ']'
++ /usr/bin/Rscript -e ' if("devtools" %in% rownames(installed.packages())) { library(devtools); devtools::document(pkg="./pkg", roclets=c("rd")) }'
Loading required package: usethis
Updating SparkR documentation
First time using roxygen2. Upgrading automatically...
Loading SparkR
Invalid DESCRIPTION:
Malformed package version.

See section 'The DESCRIPTION file' in the 'Writing R Extensions'
manual.

Error: invalid version specification '3.0.0-preview2'
In addition: Warning message:
roxygen2 requires Encoding: UTF-8
Execution halted
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.0.0-preview2:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 18.619 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 13.652 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  5.673 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  2.081 s]
[INFO] Spark Project Networking ........................... SUCCESS [  3.509 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  0.993 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  7.556 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  5.522 s]
[INFO] Spark Project Core ................................. FAILURE [01:06 min]
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project Graph API ............................ SKIPPED
[INFO] Spark Project Cypher ............................... SKIPPED
[INFO] Spark Project Graph ................................ SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:04 min
[INFO] Finished at: 2019-12-16T08:02:45Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (sparkr-pkg) on project spark-core_2.12: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :spark-core_2.12

Does this PR introduce any user-facing change?
No.
How was this patch tested?
manual test.

Test build #115381 has started for PR 26904 at commit adf9977.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20188/
Test PASSed.

Test build #115385 has started for PR 26904 at commit 9aa5e45.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20191/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115381/
Test PASSed.

Test build #115385 has finished for PR 26904 at commit 9aa5e45.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115385/
Test PASSed.

Merged to master


Add name to all column selector labels in reporter

If you had a joined datasource and the column names were the same then you couldn't know from which datasource the column was. I extended the selector labels so that now you can see its name that has the information about the datasource it belongs to

Tested and confirming that this now looks good. The only part left now is the CL entry.

Changed

What: The column selectors in reporter(prefilter, label/primary/secondary column) now all show the label and name of the column, they can also now be searched except for the presort.

Why: To make it easier to differentiate between the different fields that have the same label, for example after using a Join.

How: Replaced the default bootstrap select with the vue-select component. That provides a more feature rich select implementation.


[Question] log messages:  lock is held by.... failed to acquire lease...

I am running longhorn 1.0.0 in 3-node Kubernetes cluster and I see the following kind of error messages on 2 of 3 nodes:
│ csi-provisioner I0703 20:54:54.665760       1 leaderelection.go:350] lock is held by csi-provisioner-74557755-hfgj9 and has not yet expired                                  │
│ csi-provisioner I0703 20:54:54.665804       1 leaderelection.go:246] failed to acquire lease longhorn-system/driver-longhorn-io

│ csi-attacher I0703 20:55:39.039876       1 leaderelection.go:326] lock is held by csi-attacher-5cc849c8dd-hcfgs and has not yet expired                                      │
│ csi-attacher I0703 20:55:39.039948       1 leaderelection.go:222] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io

│ csi-resizer I0703 20:56:20.289761       1 leaderelection.go:326] lock is held by csi-resizer-686bd4b6d7-qttkl and has not yet expired                                        │
│ csi-resizer I0703 20:56:20.295327       1 leaderelection.go:222] failed to acquire lease longhorn-system/external-resizer-driver-longhorn-io

But longhorn is working and everything seems to be fine. Should I worry about these messages?
Is there a way how I can foce to restart all longhorn services on  one node?
Thanks for help
===
Ralph

It seems my issue is related to #1202  ...?

Yeah, it's not concerning. The leader election message from Kubernetes library is a bit misleading, to be frank.

ok, thanks for explaining this.


Add context to log messages

Because the custom logger adapter had to be removed, user context should be added again. This can be done more elegantly using raven's capabilities, see http://raven.readthedocs.org/en/latest/usage.html#adding-context.

Note that this could include almost anything, e.g. a version tag of sipa or something similiar. In this case, more is better!


Update DISCLAIMER-WIP

The extra word does not appear necessary. In US and, to the best of my knowledge, UK English it should not be there.


Wood puzzle background should tile

The wooden puzzle background (also visible when launching the game) should tile, so that there aren't black bars on the left and right sides.

The wood puzzle background now tiles. The black bars on the left and right sides are no longer visible.


cherry picks

Cherry-picking a handful of performance and bug fixes.
to: @graceguo-supercat @michellethomas @timifasubaa @williaster

Codecov Report

Merging #58 into airbnb-master will increase coverage by 2.51%.
The diff coverage is 66.1%.

@@                Coverage Diff                @@
##           airbnb-master      #58      +/-   ##
=================================================
+ Coverage          60.96%   63.48%   +2.51%
=================================================
  Files                260      261       +1
  Lines              19818    19794      -24
  Branches            1996     1992       -4
=================================================
+ Hits               12082    12566     +484
+ Misses              7727     7219     -508
  Partials               9        9

Impacted Files
Coverage Δ

superset/assets/src/visualizations/markup.js
26.66% <ø> (+26.66%)
:arrow_up:

superset/assets/src/visualizations/time_table.jsx
16.81% <0%> (+16.81%)
:arrow_up:

...et/assets/src/dashboard/components/SliceHeader.jsx
31.14% <0%> (ø)
:arrow_up:

superset/viz.py
81.47% <100%> (ø)
:arrow_up:

superset/views/core.py
74.7% <100%> (+0.03%)
:arrow_up:

superset/assets/src/visualizations/index.js
100% <100%> (ø)

...src/explore/components/controls/MetricsControl.jsx
79.86% <100%> (+1.24%)
:arrow_up:

...ts/src/visualizations/deckgl/layers/screengrid.jsx
38.18% <100%> (+38.18%)
:arrow_up:

...ssets/src/visualizations/deckgl/layers/scatter.jsx
27.83% <100%> (+27.83%)
:arrow_up:

superset/assets/src/components/Loading.jsx
87.5% <100%> (ø)
:arrow_up:

... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9071513...473d225. Read the comment docs.

Could you leave a comment with the commits or pr numbers so we can keep track next time we merge?


Don't unnecessarily calculate SHA256 over identity labels

The methods generating identity labels' SHA256 show up high in certain
CPU profiles when generating Hubble flows. It turns out, the SHA256
value isn't actually used anywhere in the code anymore.
Thus drop the SHA256 calcuation and related fields/methods to speed up
identity lookup, e.g. when generating Hubble flows.
See individual commit messages for details.
Speed up identity lookup in Hubble and L7 proxy by no longer calculating SHA256 over labels.

/test-1.23-net-next

@joestringer I don't think we use this field in hubble-ui

Thanks for confirming @geakstr.
Given that this information isn't used outside Cilium and tests are passing (or hitting known flakes), I'm marking this PR as ready for review.

/test-1.23-net-next

GKE test failed due to #20013 (this PR was created before the PR fixing it was merged). All other tests passed and reviews are in. Marked as ready to merge.

Marked the PR to be backported to 1.11 given that this leads to a considerable performance improvement and PR #18783 removing last use of the functions removed by this PR was backported to 1.11 as well.


Discuss: Support modify_exceptions with TypeError

Context:
I am modifying our other project (jupysql) to support it's exception class (ValueError and TypeError) with custom error message. However, only ValueError can be applied with modify_exceptions decorator to wrap the exception case.
I propose to support TypeError as well, which is much more generic.
Current implementation in def modify_exceptions:
...
        except ValueError as e:
...

Proposed implementation:
...
        except (ValueError, TypeError) as e:
...

@edublancas Let me know what you think, since we need this change to unblock the https://github.com/ploomber/jupysql/issues/30
Meanwhile, I can think about how to add the if statement to avoid the duplicated community link in this PR as well...

@tonykploomber it's better to do this discussion over a PR since it's a small fix.
We should support both types and I think the if (which I mentioned yesterday is a good fix). Let me review it.

Also, you should add to the PRs closes: # issue number so we don't have to manually do it.

@tonykploomber also, I didn't notice it but you forgot to add changelog entries. I've added it so we can release but make sure it's part of the PR next time.

Thanks Ido!


Connect to vcenter through Proxy

Hi,
I have to go through a proxy to connect to my vcenter.
The pyvim.connect don't seems to handle that.
Is it possible to connect with pyvim.SmartStubAdapter ? Don't really know if it does the same thing.

Closing for now. We can reopen in the future if people still want this.

Hello guys. This would really be usefull. I thought just setting https_proxy env var could be enough but doesnt seem so ...
Any suggestion ?


Fix exception when X-XHR-Referer isn't a invalid URI

Fix: client sends invalid X-XHR-Referer → XDomainBlocker raises URI::InvalidURIError
Get rid of 'same_origin?': warning: URI.escape is obsolete

@arthurnn @reed @dhh

@reed @kaspth Thanks for the comments. Updated.

Thanks.  If you find a way to get rid of the obsolete warning without losing the functionality, I'd be happy to merge it.


fix: error uploading patterns to server

Signed-off-by: Abhishek-kumar09<EMAIL_ADDRESS>Description
Upload handler was not working as expected and throwing error while uploading
This PR fixes https://github.com/meshery/meshery/issues/4371
Notes for Reviewers
Signed commits

[x] Yes, I signed my commits.

Codecov Report

Merging #4374 (6ea1575) into master (b055507) will decrease coverage by 7.71%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4374      +/-   ##
==========================================
- Coverage   31.85%   24.13%   -7.72%
==========================================
  Files          59       59
  Lines        5054     5054
==========================================
- Hits         1610     1220     -390
- Misses       2985     3497     +512
+ Partials      459      337     -122

Flag
Coverage Δ

gointegrationtests
?

unittests
24.13% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

mesheryctl/internal/cli/root/system/status.go
0.00% <0.00%> (-62.83%)
:arrow_down:

mesheryctl/internal/cli/root/system/start.go
0.99% <0.00%> (-53.98%)
:arrow_down:

mesheryctl/internal/cli/root/system/restart.go
2.50% <0.00%> (-47.50%)
:arrow_down:

mesheryctl/internal/cli/root/system/stop.go
0.85% <0.00%> (-36.76%)
:arrow_down:

mesheryctl/internal/cli/root/system/update.go
1.05% <0.00%> (-34.74%)
:arrow_down:

mesheryctl/internal/cli/root/system/check.go
1.39% <0.00%> (-22.38%)
:arrow_down:

mesheryctl/internal/cli/root/system/logs.go
3.29% <0.00%> (-20.88%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba2fb54...6ea1575. Read the comment docs.

https://user-images.githubusercontent.com/48255244/136661337-fe93b7a2-f44e-49a0-9e20-37ee5cbc0783.mp4

@utkarsh-pro, please review


PATCH method errors on requestSignature

I figured out that when calling requestSignature the HTTP method PATCH does not seem to be supported as it results in a Signature Error.
Example:
const auth = PowerAuthAuthentication.password("1234");
const signature = await powerAuth.requestSignature(auth, "PATCH", "/api/endpoint", "{ \"myJson\": \"body\" }")

The above throws an error, but switching it out for "PUT" or "POST" works perfectly fine. Note: I tested this on the ios simulator.
Luckily this is not a blocking issue for us right now, but still wanted to let you know

Hi @BerniWittmann, thank you for the report - this is definitely something that can be improved. The HTTP methods are actually restricted in the mobile stack core, the RN wrapper only propagates the error, see: https://github.com/wultra/powerauth-mobile-sdk/blob/2702829bbfc9451ee308c91c46357f10ed7c5959/src/PowerAuth/PublicTypes.cpp#L125
We never got a hard request to go beyond the limited GET, POST, PUT, or DELETE methods (for some reason, we also support HEAD), but if this was a blocking at some point, feel free to escalate the issue and we will have a look with priority.

Note: I will transfer this issue to https://github.com/wultra/powerauth-mobile-sdk


[FEATURE] Integration with Cloudwatch cross-account observability

Is there an existing issue for this?

[X] I have searched the existing issues

Feature description
Ref: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Cross-Account-Cross-Region.html
AWS has supported cloudwatch cross-account cross-region observability feature where you can set up Monitoring accounts to centralize cloudwatch metrics from the Source accounts. This feature allows us to do metrics federation in certain accounts, which lead me to think what would happen if we set up the cloudwatch exporter to pull metrics from those "Monitoring account"? Would it actually be able to pull all the federated metics?
Then I did a test and it turned out that the exporter can only retrieve metrics from the account itself.
For example, I have two accounts A and B where A is the monitoring account and B is the source account. With a setup to connect account B to A, I can get B's CW metrics in account A. But then if I set up the exporter to pull the metrics from account A, all the metrics I can retrieve will have a label {account_id = "{account_A_id}"} which means it automatically filters out account B's metrics.
Upon reading the code, it seems that this account_id label is set when filtering out metrics and its value is derived from the IAM role that the exporter is using.
I am wondering if we have any plan to support fetching federated metrics in the future?
What might the configuration look like?
No response
Anything else?
No response

I found that this introduces additional settings in SDKs usage: IncludeLinkedAccounts in listing metrics and AccountId in getting metric data.


Adding my profile page while fixing a bug in the vigenere cypher project

Welcome Aboard!
Can't wait to see more contributions!


Memory optimization in async tp-linear

This PR introduces the memory optimization methods implemented in #203, but now allows them to be used in the async comm regime. This PR also includes commits related to fixing the row-parallel tp-linear in #172, so both of those PRs should be merged first to make reviewing this one easier. This PR also includes two modes: recomputing or not the all_gather. Recomputing is more-memory efficient, but slightly slower.
Here is a table that summarizes my observations on a tp4 llama8b model on A100 GPUs:

Method
Average throughput (tok/sec/gpu)
Max memory reserved (GB)

Baseline (current main implementation)
3415
72.6

Sync no-recompute
3525 (+3%)
68.5 (-6%)

Sync recompute
3427
61.9 (-15%)

Async no-recompute
3587 (+5%)
68.8 (-5%)

Async recompute
3526 (+3%)
60.9 (-16%)

These changes should prove to be very useful for more efficient training. I recommend using the async-recompute setting, but using async-norecompute might make more sense for the extra throughput when memory is not a concern. In addition, as dp and pp increase and optimizer states and parameters become more sharded, the memory savings this PR brings should only increase as these affect memory activations. Very useful for scaling to larger models.
I attach the wandb logs for llama8b (top) experiments and a tiny 152M model (bottom) to study the effects on smaller models.

(blue = sync baseline, green = sync no-recompute, purple = sync recompute, yellow = async baseline, gray = async recompute, red = async no-recompute).

lgtm as well


After removing RegisterMap() application doesn't work properly

Base on rule that application must works without any maps, chatter should be fixed.
Thanks @ubbeK for good eye - https://github.com/StarcounterSamples/Chatter/issues/17

Test image app on new develop, without mapping - works as a charm.


M3-1309 attach file new ticket

Purpose
Add attachments during ticket creation. Note that this involves creating the ticket first, then adding an attachment as these can't currently be combined.
Notes

Refactor existing ticket attachment logic into components for reuse in the support ticket drawer
Adds recompose as a dependency (technically already used as a sub-dependency for some other import)

Add

User can now add an attachment when creating a support ticket

Coverage increased (+0.3%) to 31.114% when pulling 0a5a9d76c60cac8f1c8f36e9584ff790952ee516 on Jskobos:M3-1309-attach-file-new-ticket into 53b699dee064e78b3e9e824c3e7cb99c2731442d on linode:develop.

Tested happy path and sad path, works great. There's a few nit picks, but nothing significant. Well done, as always, Jared.


Add support for regexp-based redirections.

Afraid of getting this merged considering that the same redirection is magnitudes more expensive if using regexes. It does seem that some cases warrant it though!

Missing redirecting on a parameter that isn't used on the target (for example, /blog/(foo|bar) to /blog/baz)


Discussion: Can probes exist without belonging to a check

The probes (experimental) currently need to belong to a check to be run. What is users need probes that don't fit into a check? Do we create a new check? Do we put all new probes into their own check? In this case, shall we "hide" these checks from the --checks option?
I think this will eventually be a case-by-case discussion. But it would be good to have a story.
I'm currently thinking that if there's not obvious check for a new probe, we could have a fallback "special" check that contains all the probes and not exposed the check within the --checks option. This would also allow us to migrate probes to part of a public check once we think that probe(s) should be "promoted" to a public check (based on probe popularity, etc).

Action importer.
On Wed, Sep 6, 2023, 09:45 github-actions[bot] @.***>
wrote:

Stale issue message
—
Reply to this email directly, view it on GitHub
https://github.com/ossf/scorecard/issues/3235#issuecomment-1707526175,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZQI54W7ADKPZMMRXJOBLQTXY7ITPANCNFSM6AAAAAAZYXEMXU
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

D

This exists to some extent already. For the Binary-Artifact conversion (#3508), we ended up making two probes:

freeOfAnyBinaryArtifacts
freeOfUnverifiedBinaryArtifacts

Only freeOfUnverifiedBinaryArtifacts is hooked up, the other is disconnected from the Binary-Artifact check.
A similar situation will happen with some of the optional Pinned-Dependencies enhanced probes:

https://github.com/ossf/scorecard/issues/2132#issuecomment-1871598574
#2733 (this one may fit under Dangerous-Workflows too?)

I think the bigger question will be how raw data collection will work in these cases, as sometimes they use the same data (like freeOfAnyBinaryArtifacts above), but the two Pinned-Dependencies will need to fetch more data.

Also linking #3840, which has me thinking about this again.
There is a CLI argument, --probes, which will run some probes. This would work for people that just want the data, but wouldn't handle arbitrary scoring decisions. Something like blank-imports/custom checks may be better suited for that (#3095)
How to make it work
The initial implementation of --probes required setting up extra maps, and has fallen out of date. Ideally, this is setup in a way that keeps itself updated.
There's at least two piece of info we need:

which raw data fields the probes need (right now, assumes 1:1 mapping)
https://github.com/ossf/scorecard/blob/fb3edd9d63f0c84858d88a65c47205676e330489/probes/entries.go#L195-L200
which runner func the probe uses
https://github.com/ossf/scorecard/blob/fb3edd9d63f0c84858d88a65c47205676e330489/probes/entries.go#L158-L164

We can always use the same strategy from checks, which is an init() block which calls registerCheck:
https://github.com/ossf/scorecard/blob/fb3edd9d63f0c84858d88a65c47205676e330489/checks/all_checks.go#L57-L58
If we had a registerProbe func, we could presumably have all this data registration taken care of by the probes in the various impl.go files:
func init() {
	// may not be able to use constants from checks package due to circular imports
	requiredRawData := []string{
		"Maintained",
		"SAST",
	}
	supportedRequestTypes := []probes.RequestType{
		probes.CommitBased,
	}
	if err := registerProbe(Probe /* name */ , Run /* func */ , requiredRawData, supportedRequestTypes); err != nil {
		// this should never happen
		panic(err)
	}
}

This would also let us register supported request types (which relates to #3718).

So I played around with this a little bit in a branch: https://github.com/spencerschrock/scorecard/tree/register-probes
the init approach would still require blank imports somewhere, as the probe packages which aren't hooked up to the evaluation code aren't loaded.
https://github.com/ossf/scorecard/blob/c940d9bcf025a2de1b73fd0e058b0a823881a3d9/probes/entries.go#L24-L25

Partially completed via #3876, but #3718 may prevent this one from being closed.


fix: Ternary filter indicator

Fixes #3922

Hey @danharrin. Just commenting here as was using a TernaryFilter today and no indicators are being rendered at all. The filter itself works fine but no feedback displayed. Seems to be fine in 2.15.49 before this commit however?

Hey, it should now inherit the behaviour of the SelectFilter, so it should indicate when Yes or No is selected

Because of the state of the filter, which could very well be a boolean, it never reaches return ["{$this->getIndicator()}: {$label}"];. I think what happens is (thoughts in comments)...
        $this->indicateUsing(function (array $state): array {
            if (blank($state['value'] ?? null)) { // if false, will return empty array
                return [];
            }

            $label = $this->getOptions()[$state['value']] ?? null; //no options as its a TernaryFilter so no label

            if (blank($label)) { //...so returns an empty array
                return [];
            }

            return ["{$this->getIndicator()}: {$label}"]; //never reached
        });

@titantwentyone #4040 should fix this


Draggables should use position as their defaultPosition

Hi, first of all, love this library.
In this example
Setting the value of x and y
<script>
	import {draggable} from 'svelte-drag'

	let x = 100;
	let y = 0;
</script>

Should intuitively set the box at x=100 on first render. It does not. I see that defaultPosition achieves this, but I think intuitively, the x and y I pass in should be the initial position, as these can be props exposed to its parent.

Good catch, will fix this

Fixed https://github.com/PuruVJ/svelte-drag/releases/tag/2.4.1

Thank you!


Should plugin have access to canvas?

Should a plugin have access to the canvas so that it can add additional visual aspects to the canvas? eg a plugin might draw a bar graph next to a node to indicate the concentration of the node.

Instead of drawing directly to canvas, I think the better approach is to use the CanvasElement base class, which I've used to represent nodes, reactions, etc. They have event handlers (for mouse clicking, painting, etc.) and have necessary functions to act as sort of canvas "widgets".
What is currently left to be done is to expose the CanvasElement class to the user and perhaps provide some examples in the documentation.


workspace/symbol: inconsistency between documentation and implementation

The documentation says the attributes of WorkspaceSymbolParams are searchScope and searchKind, but the client must send search_scope and search_kind instead.
rust-analyzer version: rust-analyzer 17afa2e77 2022-01-24 stable
rustc version: rustc 1.58.1 (db9d1b20b 2022-01-20)

Looks like the struct def is missing a #[serde(rename_all = "camelCase")]
https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/rust-analyzer/src/lsp_ext.rs#L498-L512


implementing the auto hide or show ToolBar while Reading the blog

Implementing the auto hide or show ToolBar while Reading the blog Detail by scrolling.But there is a problem that the window has a splash screen when the ToolBar is hiding. I want to  fix this problem in next pull Request.

I will test and merge the pull request at this weekend.
Thanks a lot.


Only recognises 1st few seconds

Is it due to mel.n_len3000 that is the max of a single inference? If you feed some of the longers samples that whisper.cpp uses I presume its the  mel.n_len3000 max as know they are much longer.
rock@rock-5b:~/openai-whisper/minimal_build$ ./minimal ../models/whisper.tflite ../samples/hp0.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 2 seconds

[_SOT_][_NOT_] Henry F. Phillips from Wikipedia, the free encyclopedia at en.wicopedia.org.

Anyway brilliant to see it running on tflite as running on debian with a rk3588, only have the 8gb model, so zram & a 32gb swap on a nvme finally got me there :)
I was wondering as you have split Whisper up into several models do any of the model quantise to full int8?
I have also been playing with ArmNN which could also use the MaliG610
Also there is a 3 core 6 Tops NPU (3x 3 Tops)
ArmNN think it needs full int8 quant, not really sure about RKNPU2 https://github.com/rockchip-linux/rknn-toolkit2 as haven't really played with it.
But wondering if there are some that could and models could be partitioned (they already are) but maybe ones that don't quant could run cpu and rest use either gpu/npu?

In order to run for longer than 30 seconds of audio, we must adjust the application code for each inference run.
I was able to produce only a hybrid whisper.tflite model(activations are in float32 and weights are in quant int8).
work in progress to generate int8 quant tflite model

Yeah its no prob as was just running longer samples to get a better benchmark the inference time given doesn't have much resolution.
I was just interested how tflite benches against ggml of whisper.cpp
To be honest I wish the window of whisper was 10 sec as likely it would reduce load much but be much more better aligned for command sentences.
Transcription ASR & command ASR prob have slightly different profiles and requirements as Linux could do with a specification so that a wav or stream can feed any plugged in model and just pass on any associated metdata with its output.
It would be really good to see Whisper as a standalone Linux service as well as the great work you and bjnortier seem to be doing on IOS/Android.
I was also looking at the models directory where encoder and decoder seem to be 2 different models as that could be great to run across multiple devices such as CPU/GPU/NPU.
The RK3588 npu is a 3 core 2 Tops and got me thinking like the coral edge that also has a 2 core 2 Tops version or simply add another m.2 or usb npu as Tops is not the greatest metric but 1 watt for 2 Tops seems common and far more efficient than CPU or GPU.
So the more the model can be partitioned to be faster and more efficient the better and hence thinking npu efficiency wise seem best and why asking about int8 quant models.
Also thinking faster than realtime is a better option than streaming as the model wants that 10/30sec context for accuracy so trying to get the small model faster than realtime, likely would still fit in 4-6 tops maybe?

@StuartIanNaylor
I plan to create encoder and decoder TFLite models. I will keep this issue open until I have successfully generated the models.

@nyadla-sys that would open up the possibility of the larger models on relatively modest hardware if they can be then imported into frameworks such as ArmmNN/RKNNToolkit2 then likely speech processing can reside on CPU whilst encoder & decoder could be split by mobile GPU & NPU or combinations.
The WER actually drops off a bit of a cliff for the Tiny & Base models where the Small model only starts to approach the levels that has won Whisper so much acclaim and its actually the Medium model where Whisper ASR gains SOTA levels.
It could be interesting especially with a RK3588 as the MaliG610 is approx equivalent to the CPU in terms of ML but doesn't have the threading limitations of TFLite whilst the NPU is 2Tops x3 cores which I think 2Tops is roughly about what either the CPU/GPU could achieve alone,
Prob for the 1st time I might pay the Apple premium and try and pick up a M1 mini 16gb after Xmas 2nd user as the ML performance per watt on those things is just insane.

@StuartIanNaylor I managed to generate encoder/decoder tflite models ,please refer the below notebook for more details.
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/whisper_encoder_decoder_tflite.ipynb

Thats weird as the tokeniser is a verbatum copy & paste but is giving no output
python encoder-decoder.py

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
== Input details ==
name: serving_default_x.1:0
shape: [   1   80 3000]
type: <class 'numpy.float32'>

DUMP INPUT
{'name': 'serving_default_x.1:0', 'index': 0, 'shape': array([   1,   80, 3000], dtype=int32), 'shape_signature': array([   1,   80, 3000], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}

== Output details ==
name: PartitionedCall:0
shape: [   1 1500  384]
type: <class 'numpy.float32'>

DUMP OUTPUT
{'name': 'PartitionedCall:0', 'index': 557, 'shape': array([   1, 1500,  384], dtype=int32), 'shape_signature': array([   1, 1500,  384], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}

Inference time= 1.456505537033081
(1, 1500, 384)
[[[ 0.1696541   0.07308465  0.04009005 ...  0.05437365  0.14940399
    0.13367367]
  [-0.3666988   1.0912716   0.04315966 ...  0.24082077  0.16366142
    0.7891257 ]
  [ 0.07376133  1.5414851   0.41237825 ... -0.60459065 -0.8289902
    0.71447384]
  ...
  [ 0.7533766  -1.8113835   0.34083468 ... -0.24959154 -0.39260525
    0.47391105]
  [-0.03803208 -0.5879506   0.14105844 ...  0.60848457  0.19444972
    0.35870117]
  [ 0.1888677  -0.16534436 -1.418708   ...  0.077178   -0.5052388
   -0.16890407]]]
whisper-decoder-language-hybrid.tflite

(1, 4)
(1, 1500, 384)
400
370
452
7177
6280
1029
406
437
428
1941
393
360
337
291
11
1029
437
291
393
360
337
428
1941
13
50257
Inference time= 5.223773241043091

I just downloaded the models locally and would say working but the tokeniser is not but the code is identical
import whisper
import torch
import tensorflow as tf
import numpy as np
import argparse
import os
import warnings
import tqdm
from whisper.audio import load_audio, log_mel_spectrogram,pad_or_trim,N_FRAMES, SAMPLE_RATE
from transformers import AutoTokenizer
import time
tflite_model_path = 'whisper-encoder-hybrid.tflite'

def representative_dataset_random():
    for _ in range(100):
      data = np.random.rand(1, 80, 3000)
      yield [data.astype(np.float32)]

def representative_dataset():
    for _ in range(1):#Change this to 100 and provide 100 different audio files from known dataset
      mel_from_file = log_mel_spectrogram('jfk.flac')
      segment = pad_or_trim(mel_from_file, N_FRAMES)
      segment = tf.expand_dims(segment, 0)
      print(segment.shape)
      yield [segment]

tflite_model_path = 'whisper-encoder-hybrid.tflite'

# Load the TFLite model and allocate tensors
interpreter_enc = tf.lite.Interpreter(model_path=tflite_model_path)
interpreter_enc.allocate_tensors()

print("== Input details ==")
print("name:", interpreter_enc.get_input_details()[0]['name'])
print("shape:", interpreter_enc.get_input_details()[0]['shape'])
print("type:", interpreter_enc.get_input_details()[0]['dtype'])

print("\nDUMP INPUT")
print(interpreter_enc.get_input_details()[0])

print("\n== Output details ==")
print("name:", interpreter_enc.get_output_details()[0]['name'])
print("shape:", interpreter_enc.get_output_details()[0]['shape'])
print("type:", interpreter_enc.get_output_details()[0]['dtype'])

print("\nDUMP OUTPUT")
print(interpreter_enc.get_output_details()[0])

# Get input and output tensors
input_details = interpreter_enc.get_input_details()
output_details = interpreter_enc.get_output_details()
output_tensor = interpreter_enc.get_output_details()[0]['index']

# Test the model with random data
input_shape = input_details[0]['shape']
mel_from_file = log_mel_spectrogram('jfk.flac')
input_tensor = pad_or_trim(mel_from_file, N_FRAMES)
input_tensor = tf.expand_dims(input_tensor, 0)

audio = whisper.load_audio('jfk.flac')
audio = whisper.pad_or_trim(audio)
mel = whisper.log_mel_spectrogram(audio)
mel = np.expand_dims(mel,0)
start_time = time.time()
#input_tensor = np.array(input_tensor-128, dtype=np.int8)
interpreter_enc.set_tensor(input_details[0]['index'], mel)

interpreter_enc.invoke()
print("Whisper Encoder Inference executed successfully\n")
print("Inference time=", time.time() - start_time)
encoder_output_data = interpreter_enc.get_tensor(output_tensor)
print(encoder_output_data.shape)
print(encoder_output_data)
#np.savetxt("encoder_output.txt", encoder_output_data.reshape((3,-1)), fmt="%s", header=str(encoder_output_data.shape))

tflite_model_path='whisper-decoder-language-hybrid.tflite'
print(tflite_model_path)

# Load the TFLite model and allocate tensors
interpreter = tf.lite.Interpreter(model_path=tflite_model_path)
interpreter.allocate_tensors()

decoder_input_ids = torch.tensor([50258, 50266, 50358, 50363])
decoder_input_ids = tf.expand_dims(decoder_input_ids, 0)
print(decoder_input_ids.shape)
print(encoder_output_data.shape)

input_tensor_1 = interpreter.get_input_details()[0]['index']
interpreter.set_tensor(input_tensor_1, encoder_output_data)

input_tensor_2 = interpreter.get_input_details()[1]['index']
interpreter.resize_tensor_input(input_tensor_2, decoder_input_ids.shape)
# Allocate memory for input and output tensors
interpreter.allocate_tensors()
interpreter.set_tensor(input_tensor_2, decoder_input_ids)
output_tensor = interpreter.get_output_details()[0]['index']
start_tokens = [50258, 50266, 50358, 50363] #<|startoftranscript|><|ja|><|translate|><|notimestamps|>
tokens = start_tokens
start_time = time.time()
while(True):
    interpreter.invoke()
    output_data = interpreter.get_tensor(output_tensor)
    cleaned = np.argmax(output_data, axis=-1)
    last_token = cleaned[0,-1]
    print(last_token)
    tokens.append(last_token)
    new_value = tf.constant([last_token], dtype=tf.int64)
    new_value = tf.reshape(new_value, (1,1))
    decoder_input_ids = tf.concat([decoder_input_ids, new_value], axis=1)
    input_tensor_2 = interpreter.get_input_details()[1]['index']
    interpreter.resize_tensor_input(input_tensor_2, decoder_input_ids.shape)
    # Allocate memory for input and output tensors
    interpreter.allocate_tensors()
    interpreter.set_tensor(input_tensor_2, decoder_input_ids)
    if last_token == 50257:
      break

print("Inference time=", time.time() - start_time)
model_id = "openai/whisper-tiny"
tokenizer = AutoTokenizer.from_pretrained(model_id)
skip_special_tokens=True
tokenizer.batch_decode(np.expand_dims(tokens, axis=0), skip_special_tokens=skip_special_tokens)[0]

could you please change from decoder_input_ids = torch.tensor([50258, 50266, 50358, 50363]) #<|startoftranscript|><|ja|><|translate|><|notimestamps|>
to
decoder_input_ids = torch.tensor([50258, 50259, 50359, 50363]) #<|startoftranscript|><|en|><|transcribe|><|notimestamps|>

No the same, no output from the tokeniser or message.
Just running locally with the same code from colab.

//make sure you have done below two steps
'''
pip install optimum[onnxruntime] transformers git+https://github.com/openai/whisper.git
from optimum.onnxruntime import ORTModelForSpeechSeq2Seq
from transformers import (
set_seed,
AutoProcessor
)
from pathlib import Path
import os
SEED = 42
Export vanilla & optimized onnx model
def export_vanilla_optimized_onnx(model_checkpoint):
set_seed(SEED)
processor = AutoProcessor.from_pretrained(model_checkpoint)
# Vanilla
model = ORTModelForSpeechSeq2Seq.from_pretrained(model_checkpoint, from_transformers=True, use_cache=True)
onnx_path = Path(os.path.join("exported_onnx_models/", model_checkpoint))
model.save_pretrained(onnx_path)
processor.save_pretrained(onnx_path)

export_vanilla_optimized_onnx('openai/whisper-tiny')
'''

and also provide full path to model_id = "openai/whisper-tiny"

Please share your python script and I can try on my linux machine and update you

Its the above but I am missing the full path as just have 'openai/whisper-tiny' but no model in that path

Pls make sure you have downloaded openai folder  something like above in your machine

Colab is very slow at times :)
I will let you know


Fixes for building 3.2 on macOS

Description
This is an integration branch including previously unreviewed fixes from @cyrush for osxfixup for python 2/3 and previously reviewed fixes from @markcmiller86 for CGNS (#5685), HDF5 (#5679) and Silo (#5663). So, the only really new and unreviewed stuff here is osxfixup
All these changes were to get build_visit working on macOS.
Resolves #5594
Resolves #4516
After merging, I think we should close #5592 without merging it, correct?
How Has This Been Tested?
Both @cyrush and @markcmiller86 have run build_visit to completion and confirmed we get a working .dmg and .tar.gz files. We are still working out the code signing and notarizing steps. However, those will involve changes only to the masonry opts file.
@aowen87 32rel-mcm86-28apr21-bv-fixes-macos is a good branch to try testing on your own macOS machine if you have a moment.
Checklist:

[x] I have followed the style guidelines of this project.~~
[x] I have performed a self-review of my own code.~~
[x] I have commented my code where applicable.~~
[x] I have updated the release notes.~~
[ ] I have made corresponding changes to the documentation.
[ ] I have added debugging support to my changes.
[ ] I have added tests that prove my fix is effective or that my feature works.
[ ] I have confirmed new and existing unit tests pass locally with my changes.
[ ] I have added any new baselines to the repo.
[x] I have NOT made any changes to protocol or public interfaces in an RC branch.~~
[x] I have assigned reviewers (see VisIt's PR procedures for more information).~~

https://github.com/visit-dav/visit/search?q=sexe
Note: I am not suggesting we change all of these as part of this PR.

@cyrush I wasn't thinking when I assigned reviewers here. I have now removed you as a reviewer and add @ARSanderson.
The only code really needing further review here is the osxfixup. Its sort of defeating the purpose if review your own changes 😄

Looks good from me @markcmiller86. Ping me again when the release notes and docs are done and I can review those too.

@rusu24edward relnotes now updated. Doc changes should have been N/A. I updated comment to reflect that. Awaiting approval now.

@rusu24edward relnotes now updated. Doc changes should have been N/A. I updated comment to reflect that. Awaiting approval now.

There aren't developer docs for this?

There aren't developer docs for this?

Don't think so. There are docs already regarding masonry. This update is simply fixing bugs.

@markcmiller86 I'll give it a shot. This will also be my first attempt at using masonry, so we'll see how this goes!

Should I just run the standard build_visit with this branch?

Sure. That is good test. Running masonry does require you have certs defined though we should probably make it possible to run masonry without requiring that step


Missing correction when fixing with new kibana-plugin

Forgot to correct the URL when clicking the github icon below

https://github.com/Karql/elastalert-kibana-plugin/blob/master/public/components/main/main.tsx#L74
current
'https://github.com/bitsensor/elastalert-kibana-plugin',

fix
'https://github.com/Karql/elastalert-kibana-plugin',

Thanks 👍


isNil()

you seem to be able to get rid of the
if err != nil {
  panic("processor error")
}

by doing a function like this according to : check nil interface
func isNil(i interface{}) {
  if i != nil {
     panic("processor error")
  }
}

then just call it like :
isNil(err)
But i dont know how to make it so you can access that function everywhere 🤣 so if this is right it should be a simple pull request for someone :)

Ugh, why isn't this a Go builtin!


:sparkles: Documentation (Please)

Is your feature request related to a problem? Please describe.
A common user won't click on any of the classes because they ain't documented.
Describe the solution you'd like
Document EVERYTHING per class.
Describe alternatives you've considered
Actually reading/understanding the inner demo(s) per class(es).
Additional context
Nope, N/a

Resolve with: https://github.com/JumperBot/Java-SomeUtils/commit/3a2a9fd48e6df5eb9ad1f860f57796eb0b932331 and https://github.com/JumperBot/Java-SomeUtils/commit/2ae6a791fc7c462a15dc024fd2c268decb430586


Is the getLock method thread-safe?

I would like to know if the following method is thread-safe and if it is then there is no need to create a new instance on every request?
org.redisson.api.RedissonClient.getLock(...)

If you can reuse instances you don't need to recreate them every time a request comes in!
    @Override
    public RLock getLock(String name) {
        return new RedissonLock(connectionManager.getCommandExecutor(), name);
    }

It's fully thread-safe.


Customizable browse filtering shortcut

Why/User Benefit/User Problem
Nearly every time I browse mangadex, I filter by:

JP + KR original language (not CN since there's A LOT (see this reply for #671 ) and I'm mostly not interested)
sort by latest uploaded (won't go into the limitations of the API that doesn't allow language filtering)
I don't even bother with filtering out tags I'm not interested in, since that'd be too much to do

Doing this every single time is bothersome.
What/Requirements
Either a way to customize the default filtering (something like what #673 is asking for, but more complete), or a way to "save" filter settings and have a shortcut handy.

As of 2.11.0 you can set the filters how you want in the browse screen then save it, and set it as a default for every search


Package Add: Adds 4ngel2769.MessyOrganizer, fixes version

Checklist for Pull Requests

[x] Have you signed the Contributor License Agreement?
[x] Is there a linked Issue?

Manifests

[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

hello, @vikingnope , the following line https://github.com/4ngel2769/messy-file-organizer/releases/download/v1.0.0-beta.1/MessyFileOrganizer-setup-x64.exe must be changed to https://github.com/4ngel2769/messy-file-organizer/releases/download/v1.0.0/MessyFileOrganizer-setup-x64.exe
Sorry for the confusion

@stephengillie, this is a tray based app

Automatic Validation ended with:
(Automated response - build 883.)

Automatic Validation ended with:
(Automated response - build 883.)


Broken routes.yml file?

Maybe I'm reading this wrong, but all the members/ links refer to nonexistent views.

Yep, Liebling doesn't support Members currently (only members newsletters) 😕
I added that file while working on the Newsletter. Probably I will add support for that but not anytime soon.


2680 fix option set import/export i18n

Fix option set import preferred_locale and also export.
Import wasn't using the ones from configatron and export was just using the name for the option, not one
of it's translations. Now we are getting the translation for the first language preferred.

Just a few comments and looks good to merge.

@hooverlunch Please check if i did set the preferred_locale in the right way on the configatron.


merge package template from Kronos-Tools/npm-package-template

.travis.yml

chore(travis): merge from template .travis.yml

Codecov Report

Merging #166 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #166   +/-   ##
=======================================
  Coverage   61.53%   61.53%
=======================================
  Files           1        1
  Lines          13       13
=======================================
  Hits            8        8
  Misses          5        5

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d04df5...731a1d9. Read the comment docs.

Coverage remained the same at 60.0% when pulling 731a1d9946207e6de250bba3e415339727d48a17 on template-sync-1 into 5d04df5ffc3a7f40160581fc919a880b24d1210d on master.


servicenetworking release

https://github.com/Azure/sdk-release-request/issues/4136

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-arm-servicenetworking


Fixed file name Manifestmanifest to ApplicationManifest

@suneetnangia : Thanks for your contribution! The author, @rwike77, has been notified to review your proposed change.

@suneetnangia   Thank you for finding this typo and correcting it.  The update looks good to me.  #sign-off


Kafka utilities are exiting with a timed out waiting message

1) Install and run lenses Box: Done
https://docs.lenses.io/dev/lenses-box/index.html#development-environment
http://localhost:3030 is running fine, I can log in and see the admin panel
2) Run the bash shell command in the terminal: Done
sudo docker run --rm -it --net=host landoop/fast-data-dev bash
3)Create a new topic with CLI : Error occurs
root@fast-data-dev / $ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic my-topic
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
	at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
	at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
	at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
	at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
	at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)

None of the Kafka command utilities is working
What did I miss?

Hi SinaiNday, could you please provide the command that you use to start Lenses Box?
Is the Lenses Box container also running in the host network?

Hi SinaiNday, could you please provide the command that you use to start Lenses Box?
Is the Lenses Box container also running in the host network?

docker run -e ADV_HOST=<IP_ADDRESS> -e EULA="https://dl.lenses.io/d/?id=MY_PRIVATE_ID_SENT_BY_EMAIL" --rm -p  3030:3030 -p 9092:9092 lensesio/box

Since the ports 3030 and 9092 are running after launching lenses box, I ran this command
docker run --rm -it  -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9581:9581 -p 9582:9582 -p 9583:9583 -p 9584:9584 -e ADV_HOST=<IP_ADDRESS> landoop/fast-data-dev:latest
and then this error popped up
root@fast-data-dev / $ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic my-topic
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
[2020-03-20 04:13:17,925] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
 (kafka.admin.TopicCommand$)

I have a single broker running right now in the admin panel

@ulfox  I just solved it


Fix CI

Changes:

Put type constraint at the end to satisfy the compiler
Add runtime-benchmarks rust features where they are missing
Use new OnRuntimeUpgrade hooks

bot rebase

bot merge


SciHib dataset download

I’m updating the Wikipedia articles on Open access and Sci Hub and trying to graph the historical download rates via Sci Hub (as per this article). Would you be able to send me/deposit the raw data for that graph? Do you know if there are any additional datasets that could be added to extend before 2015 or after 2018?

Thanks @TS404 for your interest and your contributions to Wikipedia!
Regarding Sci-Hub download logs, I am not aware of any logs before 2015 or after 2018. All three log files I am aware of are tracked in this repository:

scihub-logs_2015-09_2016-02.tsv.xz which contains the data from https://doi.org/10.5061/dryad.q447c combined into a single file.
2017.statistics.tab.xz contains the data from this tweet.
2017.statistics.1016.j.tab.xz contains the data from this tweet and has extra downloads missing from 2017.statistics.tab.xz.

The data used for the "Downloads per day on Sci-Hub for months with access logs" figure was calculated in this notebook and exported to monthly-log-summaries.tsv. Note that it doesn't contain the extra downloads from 2017.statistics.1016.j.tab.xz (see https://github.com/greenelab/scihub/issues/56).
In a notebook in another repository I have done some additional analysis of the combined data with missing 2017 downloads. If you are interested in downloads per month per article, see this file.
Would it be helpful if I updated monthly-log-summaries.tsv and the visualization to include the missing 2017 DOIs? Especially if this updated figure could make it into the Wikipedia article!

@TS404 in https://github.com/greenelab/scihub/commit/afcb27d47ef1323f4499b3d02939814717cf6afd, I updated our processing of the 2017 logs to include the missing 10.1016/j DOIs. See the updated data in monthly-log-summaries.tsv and the updated visualization in scihub-log-downloads.svg.
Hopefully these updated monthly download counts will be useful for your Wikipedia edits. Also feel free to use any figures from this repository or study. They are all openly licensed.

That's excellent, thank you. It's a pity that there's not more data out there .I've contacted sci-hub through twitter (the only contact route I've been able to find) so we'll see if you're happy to release another batch.
I'll definitely use this dataset for now though. I'll probably re-plot from the data, just for font/size consistency with the other figs but I'll link to this repo as the data source.
You're also welcome to suggest any edits to the Wikipedia article (or directly edit) if you've any suggestions!

Cool! Feel free to link to your Wikipedia changeset here and close this issue when it's complete. I do Wikipedia edits from time to time but have never done something this major... although I should haha.

Figure now uploaded to Wikimedia commons and integrated into the Wikipedia article!

FYI Looks like you've plotted number of unique articles downloaded per day rather than the number of articles downloaded:

See all three possible visualizations in 02.summary-viz.ipynb. I'd probably recommend downloads per day because it is invariant based on the number of days in the month. But number of unique articles downloaded per day will be lower for months with longer log days due to the same articles being requested.
The downloads per day data should look like:

This is calculated as n_downloads / n_log_days.

That's excellent, thank you. It's a pity that there's not more data out there .I've contacted sci-hub through twitter (the only contact route I've been able to find) so we'll see if you're happy to release another batch.

The more log data we have, the easier it is to deanonymize SH users. I'm hoping Alexandra doesn't doesn't release more usage data, or that she anonymizes it better in the future. Just my 2 cents.

@stevemclaugh good point. Omitting the following information from the logs could help prevent user doxxing:

timestamps with hour-minute-second resolution. Timestamps to the calendar day are sufficient for most research.
IP address identifiers that can create visitor profiles with all the articles a given user has downloaded.

Those would be my to recommendations to Sci-Hub if they release more logs. Of course, I am also interested in the logs for research purposes, but yes there is a flip side of the coin.

Thank you! I've now corrected the diagram (I'd misinterpreted the data headings).


It worked the first time, but it never worked after it

Did you delete any modmail channels manually? If so, that would likely cause this issue. Always close the channels with !close instead.
To fix this, delete the db/threads.json file in the bot's folder, then restart the bot (this resets all threads, so be sure to close any pending ones first).

Closed due to inactivity


Max  seq length

System Info
Thanks team for great work,
As I see in the report the max ctx length is 16k right ? But the config max pos embed is 4k, and when I modify to 8k the result and memory just work within 4k. Am I missing sth ?, thanks.
  "num_mem_blocks": 2,
  "max_position_embeddings": 4096,
  "lora_rank": 128,
  "mamba_headdim": 64,
  "use_mem_rope": false,
  "rope_theta": 10000,
  "model_type": "zamba2",
  "num_attention_heads": 32,
  "num_hidden_layers": 54,
  "num_key_value_heads": 32,
  "num_logits_to_keep": 1,
  "num_query_groups": 32,
  "pad_token_id": 0,
  "rms_norm_eps": 1e-05,
  "sliding_window": null,
  "state_size": 64,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.43.0.dev0",
  "use_cache": true,
  "use_mamba_kernels": true,
  "use_shared_block_lora": true,
  "se_shared_attention_lora": false,
  "vocab_size": 32000
}

Who can help?
No response
Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction
example code
Expected behavior
16k context length

They added the parameter use_long_context=True (see HF page) but it's not working better:

I have erroneous generations when I generate an answer from a long context :

Thank you! Zamba2 looks like really good. :)


Notebook Dataset Performance: Correct typos and add figure

Example notebook data performance contains many typing errors/ grammatical mistakes etc. Importantly, output figures are not displayed on the webpage. This PR fixes these issues.
Changes proposed in this pull request:

Display output figures.
Correct typing errors.
Fix the deprecation warning related to "The module qcodes.dataset.database is deprecated".
Have short (no repetition) and long(100 repetition)  experiment the same 'sizes' of data so that comparison is easier.

@WilliamHPNielsen
@jenshnielsen
@Dominik-Vogel

The figures in the html output at https://qcodes.github.io/Qcodes/examples/DataSet/Dataset Performance.html are generated in CI. I suspect that the space in the filename may also be a problem that causes missing figures.

Could you also rewrite the last few cells to use get_parameter_data?

I suggest that we delete everything from
This is probably more useful as a numpy array. Here we use squeze to get rid of any singleton dimensions.

and down. We do not want to promote get_data_by_id anymore

The figures in the html output at https://qcodes.github.io/Qcodes/examples/DataSet/Dataset Performance.html are generated in CI. I suspect that the space in the filename may also be a problem that causes missing figures.

@jenshnielsen Changing the name to have no spaces did not help. But making figures static (by removing %matplotlib) did help in getting the figures in the HTML.  I suspect we have to add some more extensions to Sphinx.
Some suggest the extensions in the discussion below.
https://stackoverflow.com/questions/30397384/embedding-matplotlib-plots-in-sphinx-how-to-show-documentation.

Ok thats a shame. We are using a different extension to build the notebooks to rst https://github.com/spatialaudio/nbsphinx I am not sure these things are compatible.
I think its fine to change the notebook from %matplotlib notebook to %matplotlib inline which will give static figures.


Refactor the build-one.sh script

Switch to npm ci, which will not update the package-lock.json file
Move the call to webpack into build-one.sh

Looks great! Would there be any additional value in making npm build:test still work, as a trivial wrapper around build-one.sh? I don't know if that's a common npm idiom for local development or something.

Possibly! I've been only using the build-one.sh script directly, but we can always add the script entry back in to the package.json later :+1:


Awards vocabularies: Invalid pagination parameters err.

Package version (if known): 9.1
Describe the bug
After adding more than 10500 custom awards, when I click on the next page > in the awards modal or search for awards it works fine,  but if I click the last page number or >>  I get this error in the console with the message Oops! Something went wrong while fetching results.
Steps to Reproduce

add > 10000 awards
Go to upload page  open add awards modal
click on >> or last result
See error Oops! Something went wrong while fetching results.
check the console:

Expected behavior
paginate to the last result
Screenshots (if applicable)
[2022-09-01 12:53:01,231] DEBUG in errors: A resource error handler caught the following exception:
Traceback (most recent call last):
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/resources.py", line 65, in view
    return view_meth()
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/content_negotiation.py", line 116, in inner_content_negotiation
    return f(*args, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/parsers/decorators.py", line 51, in inner
    return f(self, *args, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/parsers/decorators.py", line 51, in inner
    return f(self, *args, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/responses.py", line 39, in inner
    res = f(*args, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/invenio_records_resources/resources/records/resource.py", line 77, in search
    hits = self.service.search(
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/invenio_records_resources/services/records/service.py", line 223, in search
    search = self._search("search", identity, params, es_preference, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/invenio_records_resources/services/records/service.py", line 198, in _search
    search = self.search_request(
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/invenio_records_resources/services/records/service.py", line 173, in search_request
    search = interpreter_cls(search_opts).apply(identity, search, params)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/invenio_records_resources/services/records/params/pagination.py", line 35, in apply
    raise QuerystringValidationError("Invalid pagination parameters.")
invenio_records_resources.services.errors.QuerystringValidationError: Invalid pagination parameters.
<IP_ADDRESS> - - [01/Sep/2022 12:53:01] "GET /api/awards?q=&sort=bestmatch&page=10486&size=5 HTTP/1.1" 400 -

Additional context
I tried changing the max number to  10500 with no success
if change it to big number like 100000, then I will get different err:
[2022-09-01 13:00:34,964] DEBUG in errors: A resource error handler caught the following exception:
Traceback (most recent call last):
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/resources.py", line 65, in view
    return view_meth()
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/content_negotiation.py", line 116, in inner_content_negotiation
    return f(*args, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/parsers/decorators.py", line 51, in inner
    return f(self, *args, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/parsers/decorators.py", line 51, in inner
    return f(self, *args, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/responses.py", line 39, in inner
    res = f(*args, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/invenio_records_resources/resources/records/resource.py", line 77, in search
    hits = self.service.search(
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/invenio_records_resources/services/records/service.py", line 224, in search
    search_result = search.execute()
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/elasticsearch_dsl/search.py", line 715, in execute
    self, es.search(index=self._index, body=self.to_dict(), **self._params)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/elasticsearch/client/__init__.py", line 1670, in search
    return self.transport.perform_request(
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/elasticsearch/transport.py", line 415, in perform_request
    raise e
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/elasticsearch/transport.py", line 381, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 330, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'Result window is too large, from + size must be less than or equal to: [10000] but was [52430]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.')
/home/samk13/.virtualenvs/test_voc_venv/lib/python3.9/site-packages/flask_resources/errors.py:97: DeprecationWarning: 'JSONEncoder' is deprecated and will be removed in Flask 2.3. Use 'Flask.json' to provide an alternate JSON implementation instead.
  return json.dumps(body, cls=JSONEncoder)

<IP_ADDRESS> - - [01/Sep/2022 13:00:34] "GET /api/awards?q=&sort=bestmatch&page=10486&size=5 HTTP/1.1" 500 -

(tranfering back to records-resources, even if the issue happened with awards it would happen also with any other type or records).
Exception is raised here due to the fact that default_max_results is passed to the pagination (see here) and it's value is 10000, then the query is out of range.
Increasing the default would just postpone the problem. We need to make the pagination widget aware of this maximum.

The UI should behave properly and provide a valid error message and you should be able to click on >> to go to the last possible page - i.e e.g. 9950-9999. So probably there's an "off by one"  error somewhere here.
That said, we won't be able to paginate beyond the configured max number of results due to how search is working. Basically the further back you get in a search results, the more memory your search engine needs to allocate on each node to hold the full sorted list in memory. And do that on every single HTTP request you execute to paginate the results.
Instead a scan REST API endpoint should be provided (the service layer already has some sort of support for it), but the REST API would need some sort of resumption token that it uses to keep a the context around. The OAI-PMH server for instance is already doing this job, and OAI-PMH is a way to harvest all records.

I didn't encounter the issue on V12RC2, so I suppose it's been fixed.


gradle error

Hello I'm getting this error when i try to gradle my code
i'm using: compile 'com.github.satyan:sugar:1.4'
Error:Execution failed for task ':app:transformClassesWithDexForDebug'.

com.android.build.api.transform.TransformException: com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command 'C:\Program Files\Java\jdk1.7.0_79\bin\java.exe'' finished with non-zero exit value 2

any suggestion??

I think this is a MultiDex error. You might have crossed the 65k reference limit.
Refer here.
Also checkout the stackoverflow post  for solution.

Thanks @PadaliaRushabh  i followed the post and it works !!!!

you are welcome!


Spring Configuration Client requests multiple values from Configuration Server

Describe the bug
Libraries:

Spring Boot: 3.1.2
Spring Cloud: 2022.0.4
Spring Cloud GCP: 4.7.2

I have a Spring Boot application that runs inside a K8s cluster. The deployment.yaml configures the environment like:
env:
  - name: SPRING_PROFILES_ACTIVE
    value: development

The application correctly picks up the environment name when booting up and runs with 2 profiles: kubernetes and development everything is great!
Except when I add Spring Cloud Configuration. I see the application making 2 calls, one for a default profile and one for the correct development/kubernetes one
Correctly finds this profile:
Fetching config from server at : http://configuration-service:8080
Located environment: name=dog-service, profiles=[kubernetes,development], label=null, version=null, state=null

Then fails with:
Application run failed
org.springframework.cloud.config.client.ConfigClientFailFastException: Could not locate PropertySource and the resource is not optional, failing: None of labels [] found

Back in the configuration service I can see 2 calls made:
dog-service/development/application.yaml <-- Correctly found
dog-service/default/application.yaml <-- will fail as there is no such configuration on the repo
Random guess: does the app fire a request to the configuration server before getting its active profile set by the environment?

I am not entirely clear on exactly how everything is cofigured...
but your assumption is right.
Spring Boot has always loaded configuration without and active profiles to discover potentially activated profiles first, and then goes back and loads configuration with all the activated profiles.  Prior to Spring Cloud 2022.0.4 Spring Cloud Config only ever loaded configuration after all active profiles were discovered, the problem was you couldn't activate profiles from configuration from the config server.  Now there will be a request to the config server with no active profiles first, and then a second one with all the active profiles.
I'm a little unclear how your app is configured so this first request is failing, but the second request succeeded...

Well, the problem is that my custom configuration server throws an exception as it can't find anything for the "default" profile (It was only expecting "development" or "production").
I guess the problem is that my custom Configuration Server needs to not throw an exception when it receives a call for a file it doesn't have. I guess I could return an empty org.springframework.cloud.config.environment.Environment?
Would this cause Spring to use the "default" profile and not the "development" one? Or does it prefer the activated profile ig there is one?

OK this makes more sense.
If you look at how the config server responds when a file does not exist it does not throw an exception it returns a response, but with no property sources
{
"name": "application",
"profiles": [
"default"
],
"label": null,
"version": "22542bc51277718c381187b5b8dca549f170819f",
"state": null,
"propertySources": []
}

Makes sense, I hope Spring prefers the active profile properties and not the default empty ones 👍 Thanks for the help!


add initial DO pipeline files

This PR addresses issue #1. I've initialized the targets pipeline for this project, which currently downloads and filters DO data from the WQP and NWIS, and outputs some summary information and figures about data coverage.
Any feedback or questions about the filtering steps or the DO data are welcome! Let's DO it!

Looks good, @lekoenig. Thanks for getting this rolling! I have a few detailed, in-line questions in comments.  Am I right that you are just getting everything (as in you are not aggregating temporally at all?). Also, for the DV, are you just getting mean? If we also want min and max, what does that entail and do you have any ideas of how to best format those different "target variables" (min, mean, max)?

Thanks, @jsadler2! A goal of this PR is to initiate these kinds of discussions about what input data we want to target to suit our modeling needs, and what we want that data to look like, so I appreciate your input!
To start, this PR aims to retrieve observations of [DO] from NWIS continuous sensors (preferred, where available), NWIS daily summaries (for NWIS sites that do not report instantaneous data; for now, just grab the mean), and discrete/snapshot samples available from the Water Quality Portal. So there's no temporal aggregation, etc. yet since here I was focused on getting an inventory of the number of sites and observation-days.
We can definitely summarize and output the NWIS data as daily min/mean/max for all sites where that information is reported or can be estimated from the high-frequency time series. I envision replacing the multiple targets for continuous, daily, and wqp data with one target that contains min/mean/max [DO] as available across all sites and days.
I'd appreciate further input on the discrete samples. These samples have much greater spatial coverage but aren't always sampled repeatedly, so they represent a single snapshot measurement on the day the sample was taken. Can these data still be useful for modeling? Perhaps even in static model development to inform relevant features? Or to inform a model of mean daily [DO], which might itself contain information relevant to expected min/max? Pinging @galengorski here, too.

I'd appreciate further input on the discrete samples. These samples have much greater spatial coverage but aren't always sampled repeatedly, so they represent a single snapshot measurement on the day the sample was taken. Can these data still be useful for modeling? Perhaps even in static model development to inform relevant features?

I think it's a good idea to pull the discrete samples if it's not much more work. It's still an unknown how useful these would be to the model, but I'm guess these could be pretty useful for informing the spatial distribution (assuming there is some temporal overlap)

I agree that the discrete data might have use later on in the modeling effort. I can imagine aggregating discrete samples in similar areas to fine tune a model to make predictions in an area where don't have great coverage from the high resolution data. I also think that from a network design perspective a common question might be, "Where should we invest in installing a high frequency sensor, and where can we get away with making discrete measurements?" I wonder if we might be able to shine a light on that by strategically using the discrete data with the model.

I was looking at this some more today. Are the diagonal white stripes the weekends?

I was looking at this some more today. Are the diagonal white stripes the weekends?

I think so! Reminds me of cellular automata.

I suggest we can go ahead and merge this PR, which represents the initial site and data inventory. The pipeline currently builds a table called "DB_DO_sitelist.csv" which represents the site list we need for matching point observations to stream segments (issue #16).
The next step is to output the combined DO data that can be used for modeling:

I envision replacing the multiple targets for continuous, daily, and wqp [discrete] data with one target that contains min/mean/max [DO] as available across all sites and days.

One other thing I think we should resolve before this merge and that is the inclusion of the DRB.WQdata.rds file in the repo history. Even though you removed it in a later commit, because it was earlier committed and pushed, it is part of the repo history and will therefore always be downloaded whenever the repo is cloned. I think the easiest way to remove it from the repo history is though the BFG tool (https://rtyley.github.io/bfg-repo-cleaner/). To run the BFG tool, you need Java installed and then it's pretty straightforward following the documentation. I can do that on my end if you like, @lekoenig - just let me know.


Upload container also in gate

it makes sense to upload container image also in gate (or honestly only
in gate), but if people will misuse access they might merge PR bypassing
zuul, so post will at least kick afterwards.
Signed-off-by: Artem Goncharov<EMAIL_ADDRESS>

regate

regate


Added cl_half support for test_printf

According to issue description (printf section):
https://github.com/KhronosGroup/OpenCL-CTS/issues/142

Merging as discussed in the March 28th teleconference.