[SPARK-24266][K8S][3.0] Restart the watcher when we receive a version changed from k8s

What changes were proposed in this pull request?
This is a straight application of #28423 onto branch-3.0
Restart the watcher when it failed with a HTTP_GONE code from the kubernetes api. Which means a resource version has changed.
For more relevant information see here: https://github.com/fabric8io/kubernetes-client/issues/1075
Does this PR introduce any user-facing change?
No
How was this patch tested?
This was tested in #28423 by running spark-submit to a k8s cluster.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

cc @holdenk

Test build #127914 has started for PR 29533 at commit 8ad475b.

Test build #127914 has finished for PR 29533 at commit 8ad475b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127914/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32540/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32540/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32540/
Test FAILed.

We also see this problem with Spark 2.4.X so I would very much like to see this landed for both versions.

Ok sounds like something to keep in mind.
Jenkins retest this please.

This comment exists merely to link this PR with PR #29496 intended for Spark 2.4 branch

Any review update, @holdenk ?

Jenkins retest this please.

I haven’t had a chance to look, but given it’s a backport if you have don’t feel like you need to wait for me.

Test build #128023 has started for PR 29533 at commit 8ad475b.

Test build #128023 has finished for PR 29533 at commit 8ad475b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128023/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32650/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32650/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32650/
Test FAILed.

Is there some way to run that expanded test information for this single failed R integration test as described in the message?
- Run SparkR on simple dataframe.R example *** FAILED ***
  The code passed to eventually never returned normally. Attempted 70 times over 2.0003406448333334 minutes. Last failure message: false was not true. (KubernetesSuite.scala:315)
Run completed in 12 minutes, 36 seconds.
Total number of tests run: 19
Suites: completed 2, aborted 0
Tests: succeeded 18, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.0.2-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  3.625 s]
[INFO] Spark Project Tags ................................. SUCCESS [  8.191 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  3.628 s]
[INFO] Spark Project Networking ........................... SUCCESS [  5.584 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  2.920 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 10.042 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  3.774 s]
[INFO] Spark Project Core ................................. SUCCESS [02:20 min]
[INFO] Spark Project Kubernetes Integration Tests ......... FAILURE [15:49 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  18:48 min
[INFO] Finished at: 2020-08-29T17:24:33-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:2.0.0:test (integration-test) on project spark-kubernetes-integration-tests_2.12: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :spark-kubernetes-integration-tests_2.12

Test build #128053 has started for PR 29533 at commit 745ee6b.

Test build #128053 has finished for PR 29533 at commit 745ee6b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128053/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32679/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32679/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32679/
Test FAILed.

Test build #128054 has started for PR 29533 at commit 8ad475b.

Test build #128054 has finished for PR 29533 at commit 8ad475b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128054/
Test PASSed.

Test build #128055 has started for PR 29533 at commit 6449efa.

Test build #128055 has finished for PR 29533 at commit 6449efa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128055/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32680/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32680/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32680/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32681/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32681/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32681/
Test FAILed.

Does the integration test flakiness described by @holdenk in SPARK-32354 [1] apply to this build?
[1] https://issues.apache.org/jira/browse/SPARK-32354

Huh did someone re-enable the R tests?
Anyways Jenkins retest this please.

I didn't see any change for reenabling. I guess it's disabled by Jenkins environment issue and is recovered due to some reason.

BTW, cc @zhengruifeng for 3.0.1 RC3 since the K8s IT might be broken on RC3.

Retest this please.

Test build #128175 has started for PR 29533 at commit 6449efa.

Test build #128175 has finished for PR 29533 at commit 6449efa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128175/
Test PASSed.

@dongjoon-hyun  seems like it's not a blocker in the jira.

Yes. The issue itself is not a blocker. I pinged you because of the K8s integration fails consistently on SparkR test.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32800/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32800/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32800/
Test FAILed.

So is this is ready to merge to 3.0?  I guess we just missed 3.0.1.

Jenkins retest this please

Test build #129051 has started for PR 29533 at commit 6449efa.

Test build #129051 has finished for PR 29533 at commit 6449efa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129051/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33672/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33672/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33672/
Test FAILed.

It seems that SparkR test fail.
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Run SparkR on simple dataframe.R example *** FAILED ***

Jenkins retest this please.

Test build #129084 has started for PR 29533 at commit 6449efa.

Test build #129084 has finished for PR 29533 at commit 6449efa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129084/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33705/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33705/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33705/
Test FAILed.

Test build #129113 has started for PR 29533 at commit 64c69a9.

Test build #129113 has finished for PR 29533 at commit 64c69a9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129113/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33734/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33734/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33734/
Test FAILed.

Test build #129119 has started for PR 29533 at commit 0e556ff.

Test build #129119 has finished for PR 29533 at commit 0e556ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129119/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33738/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33738/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33738/
Test FAILed.

Regarding did someone re-enable the R tests? @holdenk should they be disabled on the branch-3.0 branch?

I think they should be disabled, the Jenkins install isn't configured for testing the R version in 3.0. Go ahead and comment it out.

Looking more carefully at history and #22145 it appears that simply rebasing should fix this.

Test build #129196 has started for PR 29533 at commit c882891.

Test build #129196 has finished for PR 29533 at commit c882891.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129196/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33811/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33811/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33811/
Test FAILed.

Well, rebase didn't do it.
@shaneknapp Could this problem be similar to what you saw in #22145  ?

@holdenk On reflection, I don't think this patch set should both turn off SparkR k8s testing and implement this fix.  They should probably be separate PRs if it is desirable to turn it off so that it is easily reverted.  Is it possible to merge this even with the presence of the integration test failure?

I would prefer to instead disable the R tests first. You can of course submit that in another PR and ping me to review that one.

Test build #129428 has started for PR 29533 at commit 0605745.

Test build #129428 has finished for PR 29533 at commit 0605745.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129428/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34034/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34034/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34034/
Test PASSed.

@holdenk  Ok, integration tests pass.
It was just a two word deletion...

@dongjoon-hyun Is this ok to merge?

@jkleckner I tried this patch in production but it does not seem to work.
Distribution:
git clone https://github.com/apache/spark.git
git checkout branch-3.0
git fetch origin pull/29533/head:backport-SPARK-24266-to-branch-3.0
git checkout backport-SPARK-24266-to-branch-3.0
git checkout -b rebased-backport-SPARK-24266-to-branch-3.0
git rebase branch-3.0
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g"
./dev/make-distribution.sh --name spark-3.0-24266 --tgz -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes

Image creation:
./bin/docker-image-tool.sh -r my-registry -t my-tag -n -u 0 -b java_image_tag=11-jre-slim build
./bin/docker-image-tool.sh -r my-registry -t my-tag push

When I execute my long-running spark application I get
...
20/10/22 17:30:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
io.fabric8.kubernetes.client.KubernetesClientException: too old resource version:<PHONE_NUMBER><PHONE_NUMBER>)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:259)
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

I'm not sure if I made a mistake or there's a problem in the patch. Thanks

I tried this patch in production but it does not seem to work.
...
I'm not sure if I made a mistake or there's a problem in the patch. Thanks

Your rebase of the patch looks correct when I tried it out.
Unfortunately, there must be something in the patches to k8s code after 2.4 and before the fix #28423 was merged.
My interest in landing this patch has been to unblock #29496 due to the backporting policies for Spark 2.4 which is what we use and I don't have a setup to test this for the 3.0 branch.
@redsk if you would like to look into that, it would be helpful.
The candidate patch sets that are in the k8s can be viewed with something like this:
git log --oneline  branch-3.0..0432379f9923768a767566e9ac5a4021cfe8d052 --  resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s
0432379f99 [SPARK-24266][K8S] Restart the watcher when we receive a version changed from k8s
c8f3bd861d [SPARK-31696][K8S] Support driver service annotation in K8S
85dad37f69 [SPARK-31601][K8S] Fix spark.kubernetes.executor.podNamePrefix to work
b8ccd75524 [SPARK-29905][K8S] Improve pod lifecycle manager behavior with dynamic allocation
7699f765f5 [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
ed06d98044 [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
1254c88034 [SPARK-31118][K8S][DOC] Add version information to the configuration of K8S
d273a2bb0f [SPARK-20628][CORE][K8S] Start to improve Spark decommissioning & preemption support
f9f06eee98 [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
86fdb818bf [SPARK-30715][K8S] Bump fabric8 to 4.7.1

@stijndehaes can you guide as to which of these might affect this backport?

@redsk @jkleckner The error line you are seeing comes from the class ExecutorPodsWatchSnapshotSource this is somewhere else. I thought there was another mechanism for the executors to work with connection problems.
This also looks like driver logs? The fix here is in the spark-submit application, you should watch the logs of that i.s.o. the driver. Can you tell me if these were driver logs or spark-submit logs?

Hi, @jkleckner and all.
Is there any updates on this PR?

@dongjoon-hyun  Since I don't have a test environment to try out or observe the problem mentioned by @redsk it will have to be taken up by someone like @redsk.
I am fully on a critical path for the next many weeks and the back port to 2.4 is working fine for us so I can't spend any more time on this now.
And as mentioned by @stijndehaes , this could very well be a separate issue from the fix that 24266 addresses.
I did spend some time last week and thought the patch sets that upgrade fabric8 which had fixes that might plausibly explain the problem seen by @redsk  .
I could revert the one line patch that you wanted to revert that was requested by @holdenk but I feel caught in the middle and would prefer that you both agree.

I reverted the SparkR per your instruction and also rebased to branch-3.0.

Test build #130558 has started for PR 29533 at commit 9015c82.

Thanks, @jkleckner . For the SparkR part, we can ignore it in this PR if it fails again. And, disabling should be handled by a separate PR if we do it in branch-3.0.

I reverted the SparkR per your instruction and also rebased to branch-3.0.

Test build #130558 has finished for PR 29533 at commit 9015c82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130558/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35158/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35158/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35158/
Test FAILed.

@redsk Could you file a new JIRA for your observation, please?

@dongjoon-hyun  Thank you for the merge and hanging in there.  Hopefully @shockdm will revive the 2.4 branch fix kicked off in #29496

@dongjoon-hyun I've created SPARK-33349 as requested. Thanks

Thank you so much, @redsk !


Suspected VRAM Memory Leak when reloading Config

Already reported ? *

[X] I have searched the existing open and closed issues.

Regression?
No
System Info and Version
 System/Version info
Hyprland, built from branch  at commit 0f594732b063a90d44df8c5d402d658f27471dfe  (props: bump version to 0.43.0).
Date: Sun Sep 8 16:48:21 2024
Tag: v0.43.0, commits: 5196

flags: (if any)

System Information:
System name: Linux
Node name: terminator
Release: 6.10.7-1-cachyos-cachyos-lto
Version: #1 SMP PREEMPT_DYNAMIC Mon, 02 Sep 2024 09:06:28 +0000

GPU information:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] [10de:2482] (rev a1) (prog-if 00 [VGA controller])
NVRM version: NVIDIA UNIX x86_64 Kernel Module  560.35.03  Fri Aug 16 21:39:15 UTC 2024

os-release: NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

plugins:

======Config-Start======

Config File: /home/luke/.config/hypr/hyprland.conf: Read Succeeded
#
# Please note not all available settings / options are set here.
# For a full list, see the wiki
#

$monitor1 = DP-3
$monitor2 = HDMI-A-1

# Configure Monitors
monitor=$monitor1,2560x1440@165,1080x0,1,bitdepth,10, vrr, 1
monitor=$monitor2,1920x1080@75,0x0,1,bitdepth,10, transform, 3

# Workspaces
workspace=1,monitor:$monitor1,default:true
workspace=2,monitor:$monitor1
workspace=3,monitor:$monitor1
workspace=4,monitor:$monitor1
workspace=5,monitor:$monitor1
workspace=6,monitor:$monitor2,default:true
workspace=7,monitor:$monitor2
workspace=8,monitor:$monitor2
workspace=9,monitor:$monitor2
workspace=10,monitor:$monitor2
workspace=1337,monitor:$monitor1

# Start Shit
exec-once = waybar
exec-once = hyprpaper
exec-once=dbus-update-activation-environment --systemd WAYLAND_DISPLAY XDG_CURRENT_DESKTOP
exec-once=bluetoothctl connect AC:80:0A:9C:18:45 # connect Sony WH-1000XM4 headphones via Bluetooth

# Source a file (multi-file configs)
# source = ~/.config/hypr/myColors.conf

# Set programs that you use
$terminal = kitty
$fileManager = thunar
$menu = wofi --show drun

# Some default env vars.
env = XCURSOR_SIZE,24
env = QT_QPA_PLATFORMTHEME,qt5ct # change to qt6ct if you have that

# For all categories, see https://wiki.hyprland.org/Configuring/Variables/
input {
    kb_layout = us
    kb_variant =
    kb_model =
    kb_options =
    kb_rules =

    follow_mouse = 1

    touchpad {
        natural_scroll = no
    }

    accel_profile = flat

    sensitivity = 0 # -1.0 to 1.0, 0 means no modification.
}

general {
    gaps_in = 5
    gaps_out = 10
    border_size = 2
    col.active_border = rgba(89b4fa80) rgba(a6e3a180) 45deg
    col.inactive_border = rgba(59595980)

    # Every other layout is bad
    layout = dwindle

    # Please see https://wiki.hyprland.org/Configuring/Tearing/ before you turn this on (Why tf would you turn this on)
    allow_tearing = true
}

decoration {
    # See https://wiki.hyprland.org/Configuring/Variables/ for more

    rounding = 8

    blur {
        enabled = true
        size = 3
        passes = 1
    }

    drop_shadow = yes
    shadow_range = 10
    shadow_render_power = 3
    shadow_offset = 1 1
    col.shadow = rgba(1a1a1a80)
}

animations {
    enabled = yes

    # Some default animations, see https://wiki.hyprland.org/Configuring/Animations/ for more

    bezier = myBezier, 0.05, 0.9, 0.1, 1.05

    animation = windows, 1, 7, myBezier
    animation = windowsOut, 1, 7, default, popin 80%
    animation = border, 1, 10, default
    animation = borderangle, 1, 8, default
    animation = fade, 1, 7, default
    animation = workspaces, 1, 6, default
}

dwindle {
    # See https://wiki.hyprland.org/Configuring/Dwindle-Layout/ for more
    pseudotile = yes # master switch for pseudotiling. Enabling is bound to mainMod + P in the keybinds section below
    preserve_split = yes # you probably want this
}

gestures {
    # See https://wiki.hyprland.org/Configuring/Variables/ for more
    workspace_swipe = off
}

misc {
    # See https://wiki.hyprland.org/Configuring/Variables/ for more
    force_default_wallpaper = 0 # Set to 0 or 1 to disable the anime mascot wallpapers
    vfr = true # Variable Framerate, when nothing is happening on screen lower display frames.
}

# Example windowrule v1
# windowrule = float, ^(kitty)$
# Example windowrule v2
# windowrulev2 = float,class:^(kitty)$,title:^(kitty)$
# See https://wiki.hyprland.org/Configuring/Window-Rules/ for more
windowrulev2 = suppressevent maximize, class:.* # You'll probably like this.
windowrulev2 = float,class:^(pavucontrol)$
windowrulev2 = float,class:^(nwg-look)$
windowrulev2 = float,class:^(waypaper)$
windowrulev2 = float,class:^(openrgb)$
windowrulev2 = float,class:^(rice-settings)$

# Game Mode Related Window Rules
windowrulev2 = immediate, class:^steam_app_\d+$
windowrulev2 = workspace 1337,class:^steam_app_\d+$
workspace = 1337, border:false, rounding:false

windowrulev2 = immediate, class:^steam_proton$
windowrulev2 = workspace 1337,class:^steam_proton$
workspace = 1337, border:false, rounding:false

# See https://wiki.hyprland.org/Configuring/Keywords/ for more
$mainMod = SUPER

# Rice Settings (Developed by me)
bind = SUPER, grave, exec, rice-settings

# Multimedia Keys
bindle = , XF86AudioRaiseVolume, exec, wpctl set-volume -l 1.2 @DEFAULT_AUDIO_SINK@ 5%+
bindle = , XF86AudioLowerVolume, exec, wpctl set-volume @DEFAULT_AUDIO_SINK@ 5%-
bindl = , XF86AudioMute, exec, wpctl set-mute @DEFAULT_AUDIO_SINK@ toggle
bindl = , XF86AudioMicMute, exec, pamixer --default-source -t
bind = , XF86AudioPrev, exec, playerctl previous
bind = , XF86AudioPlay, exec, playerctl play-pause
bind = , XF86AudioPause, exec, playerctl pause
bind = , XF86AudioNext, exec, playerctl next

bind = $mainMod, RETURN, exec, $terminal
bind = $mainMod SHIFT, RETURN, exec, [float] $terminal
bind = $mainMod, Q, killactive,
bind = $mainMod SHIFT, Q, exit,
bind = $mainMod, E, exec, $fileManager
bind = $mainMod SHIFT, E, exec, [float] $fileManager
bind = $mainMod, TAB, togglefloating,
bind = $mainMod, SPACE, exec, $menu
bind = $mainMod, P, pseudo, # dwindle
bind = $mainMod, V, togglesplit, # dwindle
bind = $mainMod, M, fullscreen, 1
bind = $mainMod, F, fullscreen, 0
bind = $mainMod, B, exec, firefox

# screenshot
bind = $mainMod SHIFT, S, exec, grimshot savecopy area - | wl-copy

# Lock Screen
bind = $mainMod, L, exec, hyprlock

# Game mode bind (temporary)
bind = $mainMod SHIFT, G, exec, /home/luke/.config/apexos/scripts/gamemode-fullscreen-optimisations

# Enable Tearing
bind = $mainMod SHIFT, T, exec, /home/luke/hyprtearing

# Lock Window Focus
bind = CONTROLSHIFT, L, exec, /home/luke/.config/apexos/scripts/lock-primary-monitor

# Move window
bind = SUPER SHIFT, h, movewindow, l
bind = SUPER SHIFT, l, movewindow, r
bind = SUPER SHIFT, k, movewindow, u
bind = SUPER SHIFT, j, movewindow, d

# Resize window
bind = SUPER ALT, h, resizeactive, -160 0
bind = SUPER ALT, l, resizeactive, 160 0
bind = SUPER ALT, k, resizeactive, 0 -160
bind = SUPER ALT, j, resizeactive, 0 160

# Switch workspaces with mainMod + [0-9]
bind = $mainMod, 1, workspace, 1
bind = $mainMod, 2, workspace, 2
bind = $mainMod, 3, workspace, 3
bind = $mainMod, 4, workspace, 4
bind = $mainMod, 5, workspace, 5
bind = $mainMod, 6, workspace, 6
bind = $mainMod, 7, workspace, 7
bind = $mainMod, 8, workspace, 8
bind = $mainMod, 9, workspace, 9
bind = $mainMod, 0, workspace, 10

# Move active window to a workspace with mainMod + SHIFT + [0-9]
bind = $mainMod SHIFT, 1, movetoworkspace, 1
bind = $mainMod SHIFT, 2, movetoworkspace, 2
bind = $mainMod SHIFT, 3, movetoworkspace, 3
bind = $mainMod SHIFT, 4, movetoworkspace, 4
bind = $mainMod SHIFT, 5, movetoworkspace, 5
bind = $mainMod SHIFT, 6, movetoworkspace, 6
bind = $mainMod SHIFT, 7, movetoworkspace, 7
bind = $mainMod SHIFT, 8, movetoworkspace, 8
bind = $mainMod SHIFT, 9, movetoworkspace, 9
bind = $mainMod SHIFT, 0, movetoworkspace, 10

# Example special workspace (scratchpad)
bind = $mainMod, O, togglespecialworkspace, magic
bind = $mainMod SHIFT, O, movetoworkspace, special:magic

# Scroll through existing workspaces with mainMod + scroll
bind = $mainMod, mouse_down, workspace, e+1
bind = $mainMod, mouse_up, workspace, e-1

# Move/resize windows with mainMod + LMB/RMB and dragging
bindm = $mainMod, mouse:272, movewindow
bindm = $mainMod, mouse:273, resizewindow

======Config-End========

Description
It seems as every time I reload the Hyprland config, commandline or not, the VRAM slowly creeps up, until it reaches about 800-1100MB of VRAM then stops going up no matter how many times or how I reload the config.
I am using hyprctl reload, however I have attempted to use another reload command with the same result.
I have noticed there was a similar issue a while ago, https://github.com/hyprwm/Hyprland/issues/809
This is pretty much exactly my problem.
How to reproduce

Reload the Hyprland config in any way, hyprctl reload
View VRAM usage using nvidia-smi

Crash reports, logs, images, videos

Upon restarting/logging out of the Hyprland session, the GPU memory is perfectly free again only using ~220MiB.

Are you sure it's also present in 0.39.1? If not, can you bisect?

I have an error whilst running make all, here is my output:
`❯ git clone --recursive https://github.com/hyprwm/Hyprland
Cloning into 'Hyprland'...
remote: Enumerating objects: 37807, done.
remote: Counting objects: 100% (3599/3599), done.
remote: Compressing objects: 100% (788/788), done.
remote: Total 37807 (delta 3118), reused 3100 (delta 2810), pack-reused 34208 (from 1)
Receiving objects: 100% (37807/37807), 68.43 MiB | 14.06 MiB/s, done.
Resolving deltas: 100% (29572/29572), done.
Submodule 'subprojects/hyprland-protocols' (https://github.com/hyprwm/hyprland-protocols) registered for path 'subprojects/hyprland-protocols'
Submodule 'subprojects/tracy' (https://github.com/wolfpld/tracy) registered for path 'subprojects/tracy'
Submodule 'subprojects/udis86' (https://github.com/canihavesomecoffee/udis86) registered for path 'subprojects/udis86'
Cloning into '/home/luke/hyprland_patch/Hyprland/subprojects/hyprland-protocols'...
remote: Enumerating objects: 136, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 136 (delta 31), reused 30 (delta 26), pack-reused 88 (from 1)
Receiving objects: 100% (136/136), 30.74 KiB | 507.00 KiB/s, done.
Resolving deltas: 100% (64/64), done.
Cloning into '/home/luke/hyprland_patch/Hyprland/subprojects/tracy'...
remote: Enumerating objects: 40684, done.
remote: Counting objects: 100% (647/647), done.
remote: Compressing objects: 100% (356/356), done.
remote: Total 40684 (delta 377), reused 527 (delta 290), pack-reused 40037 (from 1)
Receiving objects: 100% (40684/40684), 29.28 MiB | 13.14 MiB/s, done.
Resolving deltas: 100% (29073/29073), done.
Cloning into '/home/luke/hyprland_patch/Hyprland/subprojects/udis86'...
remote: Enumerating objects: 2477, done.
remote: Counting objects: 100% (63/63), done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 2477 (delta 34), reused 46 (delta 20), pack-reused 2414 (from 1)
Receiving objects: 100% (2477/2477), 1.81 MiB | 4.33 MiB/s, done.
Resolving deltas: 100% (1503/1503), done.
Submodule path 'subprojects/hyprland-protocols': checked out 'e06482e0e611130cd1929f75e8c1cf679e57d161'
Submodule path 'subprojects/tracy': checked out '37aff70dfa50cf6307b3fee6074d627dc2929143'
Submodule path 'subprojects/udis86': checked out '5336633af70f3917760a6d441ff02d93477b0c86'
❯ cd Hyprland
❯ git bisect start
status: waiting for both good and bad commits
❯ git bisect good v0.39.1
status: waiting for bad commit, 1 good commit known
❯ git bisect bad v0.43.0
Bisecting: 367 revisions left to test after this (roughly 9 steps)
[2f278dc883f3e4e84fb1d18154132f8f4efe42cf] egl: fixup format modifier lookups with implicit modifiers
❯ make all
make clear
make[1]: Entering directory '/home/luke/hyprland_patch/Hyprland'
rm -rf build
rm -f ./protocols/.h ./protocols/.c ./protocols/.cpp ./protocols/.hpp
rm -rf ./subprojects/wlroots-hyprland/build
make[1]: Leaving directory '/home/luke/hyprland_patch/Hyprland'
make release
make[1]: Entering directory '/home/luke/hyprland_patch/Hyprland'
cmake --no-warn-unused-cli -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_INSTALL_PREFIX:STRING=/usr/local -S . -B ./build -G Ninja
Not searching for unused variables given on the command line.
CMake Warning (dev) at /usr/share/cmake/Modules/GNUInstallDirs.cmake:253 (message):
Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
target architecture is known.  Please enable at least one language before
including GNUInstallDirs.
Call Stack (most recent call first):
CMakeLists.txt:5 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.
-- The C compiler identification is GNU 14.2.1
-- The CXX compiler identification is GNU 14.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Gathering git info
-- Found Python3: /usr/bin/python3.12 (found version "3.12.5") found components: Interpreter
Building itab.c/itab.h...
-- Setting up wlroots
-- Found PkgConfig: /usr/bin/pkg-config (found version "2.1.1")
-- Found WaylandScanner at /usr/bin/wayland-scanner
-- Found wayland-protocols at //usr/share/wayland-protocols
-- Configuring Hyprland in Release with CMake
-- Checking deps...
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found OpenGL: /usr/lib/libOpenGL.so  found components: GLES3
-- Checking for modules 'xkbcommon;uuid;wayland-server;wayland-client;wayland-cursor;wayland-protocols;cairo;pango;pangocairo;pixman-1;libdrm;libinput;hwdata;libseat;libdisplay-info;libliftoff;libudev;gbm;hyprlang>=0.3.2;hyprcursor>=0.1.7;hyprutils>=0.1.1'
--   Found xkbcommon, version 1.7.0
--   Found uuid, version 2.40.2
--   Found wayland-server, version 1.23.0
--   Found wayland-client, version 1.23.0
--   Found wayland-cursor, version 1.23.0
--   Found wayland-protocols, version 1.37
--   Found cairo, version 1.18.2
--   Found pango, version 1.54.0
--   Found pangocairo, version 1.54.0
--   Found pixman-1, version 0.43.4
--   Found libdrm, version 2.4.123
--   Found libinput, version 1.26.2
--   Found hwdata, version 0.387
--   Found libseat, version 0.8.0
--   Found libdisplay-info, version 0.2.0
--   Found libliftoff, version 0.5.0
--   Found libudev, version 256
--   Found gbm, version 24.2.2-arch1.1
--   Found hyprlang, version 0.5.2
--   Found hyprcursor, version 0.1.9
--   Found hyprutils, version 0.2.1
-- Looking for execinfo.h
-- Looking for execinfo.h - found
-- Configuration supports execinfo
-- Looking for backtrace in execinfo
-- Looking for backtrace in execinfo - not found
-- Looking for sys/timerfd.h
-- Looking for sys/timerfd.h - found
-- Checking for module 'epoll-shim'
--   Package 'epoll-shim', required by 'virtual:world', not found
-- XWAYLAND Enabled (NO_XWAYLAND not defined) checking deps...
-- Checking for modules 'xcb;xwayland;xcb-util;xcb-render;xcb-xfixes;xcb-icccm;xcb-composite;xcb-res;xcb-ewmh;xcb-errors'
--   Found xcb, version 1.17.0
--   Found xwayland, version 24.1.2
--   Found xcb-util, version 0.4.1
--   Found xcb-render, version 1.17.0
--   Found xcb-xfixes, version 1.17.0
--   Found xcb-icccm, version 0.4.2
--   Found xcb-composite, version 1.17.0
--   Found xcb-res, version 1.17.0
--   Found xcb-ewmh, version 0.4.2
--   Found xcb-errors, version 1.0.1
-- SYSTEMD support is requested (NO_SYSTEMD not defined)...
-- Setting precompiled headers
-- Setting link libraries
-- Checking for module 'hyprutils>=0.1.1'
--   Found hyprutils, version 0.2.1
-- Checking for modules 'tomlplusplus;hyprutils>=0.1.1'
--   Found tomlplusplus, version 3.4.0
--   Found hyprutils, version 0.2.1
-- Configuring done (1.4s)
-- Generating done (0.0s)
-- Build files have been written to: /home/luke/hyprland_patch/Hyprland/build
cmake --build ./build --config Release --target all
[0/2] Re-checking globbed directories...
[50/252] Performing configure step for 'wlroots-hyprland'
FAILED: /home/luke/hyprland_patch/Hyprland/subprojects/wlroots-hyprland/src/wlroots-hyprland-stamp/wlroots-hyprland-configure
cd /home/luke/hyprland_patch/Hyprland/subprojects/wlroots-hyprland && meson setup --reconfigure build --buildtype=release -Dwerror=false -Dxwayland=enabled -Dexamples=false -Drenderers=gles2 -Dbackends=drm,libinput -Db_sanitize=none && /usr/bin/cmake -E touch /home/luke/hyprland_patch/Hyprland/subprojects/wlroots-hyprland/src/wlroots-hyprland-stamp/wlroots-hyprland-configure
ERROR: Neither source directory 'build' nor build directory None contain a build file meson.build.
[57/252] Building CXX object hyprctl/CMakeFiles/hyprctl.dir/main.cpp.o
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:15: release] Error 1
make[1]: Leaving directory '/home/luke/hyprland_patch/Hyprland'
make: *** [Makefile:34: all] Error 2`

And the issue is not present in 0.39.1, what would be the suspect in that case?

no clue, gotta bisect.

cmake --build ./build --config Release --target all
[0/2] Re-checking globbed directories...
[50/252] Performing configure step for 'wlroots-hyprland'
FAILED: /home/luke/hyprland_patch/Hyprland/subprojects/wlroots-hyprland/src/wlroots-hyprland-stamp/wlroots-hyprland-configure
cd /home/luke/hyprland_patch/Hyprland/subprojects/wlroots-hyprland && meson setup --reconfigure build --buildtype=release -Dwerror=false -Dxwayland=enabled -Dexamples=false -Drenderers=gles2 -Dbackends=drm,libinput -Db_sanitize=none && /usr/bin/cmake -E touch /home/luke/hyprland_patch/Hyprland/subprojects/wlroots-hyprland/src/wlroots-hyprland-stamp/wlroots-hyprland-configure

ERROR: Neither source directory 'build' nor build directory None contain a build file meson.build.
[57/252] Building CXX object hyprctl/CMakeFiles/hyprctl.dir/main.cpp.o
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:15: release] Error 1
make[1]: Leaving directory '/home/luke/hyprland_patch/Hyprland'
make: *** [Makefile:34: all] Error 2`

I have done bisect, but upon running make all to compile it, this is the error I receive.

Worked after git bisect skip
❯ git bisect skip
Bisecting: 367 revisions left to test after this (roughly 9 steps)
warning: unable to rmdir 'subprojects/wlroots-hyprland': Directory not empty
[c8873b958dd9330c364339ac4ab58e32b27d82b4] internal: fix fullscreen typos (#7134)

It seems as the problem falls within this commit: renderer: fixup nvidia driver version checks

I can confirm the bad commit is renderer: fixup nvidia driver version checks.

After cloning from the main branch, with the latest updates, just changing static int  driverMajor = 0; to int  driverMajor = 0; in render/Renderer.cpp, the problem is completely gone. No clue why a static int vs normal int would cause this problem to occur.

why not just force disable explicit_sync_kms instead of that (wrong) patch?
IMO seems like an nvidia issue

I suppose I can do that, what does my patch do that makes it wrong? How can changing the int var type disable explicit sync? Genuinely asking. Not surprised if it is an Nvidia issue lol considering many people are having similar issues

you made it non static

Just running nvidia-smi in a tty with no display server running shows an increase in vram usage every time it is run, this is not a Hyprland issue but a wider driver leak with anything that uses KMS.


[ New Extension ] Youtube to MP3 or MP4 download

Is your feature request related to a problem? Please describe.
I often download youtube videos , mainly mp3 files for music for offline playing but the problem is Webstore don't provide any extension for it , so I created my own and I would like to add it to this project.
There are many website that allow you to download from youtube as MP3 or MP4 but the problem is that I have to do too much manual work , so I created this extension where when I click the extension button , it will copy present youtube or music.youtube link and paste in to another website which will be used to download the mp3/mp4 file and then we will switch back to the youtube website , whether it is youtube.com or music.youtube.com
What problem is this feature trying to solve?
Removing repetitive task of copying and pasting links to download mp3 or mp4 file from youtube using already available websites.
GSSoC'24

Hello @Sulagna-Dutta-Roy ,
Thank you for considering the addition of a new extension to address the current issues with existing ones in the repository. Based on the situation where current extensions for downloading YouTube MP4 files are non-functional and there's no available extension for downloading MP3 files, I agree that such a contribution would be significant.
Introducing a new working extension to fill this gap not only enhances the functionality of the repository but also meets a clear user demand. Therefore, I believe this contribution meets the criteria for at least a level 2 rating. It addresses a specific need, improves the overall user experience, and adds value to the repository.
Please let me know if you can update the level to level2, I look forward to your feedback.
Best regards,
Mohit


Cherry-pick 11 patches from main to 2.3.x

Description of changes
Cherry-picks the following 11 PRs from main to 2.3.x:

#190
#201
#200
#206
#198
#210
#188
#212
#213
#214
#218

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

#92 was merged to this branch in #270, so we're good after merging this


BigQuery Snippets: Use autodetected location

Use autodetected location.

In the example provided, it is written that The location must be specified, but the example works even without setting location for datasets located in US and asia-northeast1. Whether specifying location for an upload job is an absolute requirement or not is affecting an application I'm working on. Can you provide details?

See b/116968956

Can you add a snippet such as https://github.com/googleapis/google-cloud-python/pull/8422 that does show how to override this?

@tswast I can't add a new snippet. That's not in scope of the sample I'm trying to standardize, fixing the attached bug.
I'm updating this snippet in a few languages.
I can change this PR to help fix the bug in some other way, but would like explicit instructions on how to do that.

Per the bug, if you aren't adding a new snippet (location autodetection is a new feature, after all), please add an explanation comment as we did in Java. https://github.com/googleapis/google-cloud-java/pull/5475

Like the Java snippet, I added a note that the location is inferred here.


kontena-cli tries to interpolate docker-compose variables as stack variables

Steps to reproduce
Create two files in a folder:
# kontena.yml
stack: demo
version: 1.0.0

services:
  node:
    extends:
      file: docker-compose.yml
      service: node

# docker-compose.yml
version: '2.1'
services:
  node:
    image: node:alpine
    environment:
      - ENV_VAR=$ENV_VAR

then run:
$ ENV_VAR=whatever
$ kontena stack validate
 [error] RuntimeError : Undeclared variable 'ENV_VAR' in c:/repos/kontenabug/docker-compose.yml:5 --       - ENV_VAR=$ENV_VAR
         See C:/Users/charl/.kontena/kontena.log or run the command again with environment DEBUG=true set to see the full exception

Expected result:

the stack should be validated

Actual result:

the stack is not validated. The kontena-cli throws an error about ENV_VAR being undeclared. This should be the case, since ENV_VAR is not a kontena variable, but a "regular" docker-compose variable.

Note: Adding a kontena stack variable definition in the docker-compose.yml file allows to validate the stack. This create a non-valid docker-compose file, but it makes me think that the docker-compose is treated as a kontena stack file because of the extend.
# docker-compose.yml
version: '2.1'

+variables:
+  myvar:
+    type: string
+    default: 'test'
+    from:
+      env: ENV_VAR
+    to:
+      env: ENV_VAR

services:
  node:
    image: node:alpine
    environment:
      - ENV_VAR=$ENV_VAR

This looks like a regression from #2707 at Kontena::Cli::Stacks::YAML::Reader#_external_file. It used to pass the extending stack variables to the Reader for the new stack, but it doesn't do that anymore.

Should we implement the variables thing from depends to extends? Then it would become:
stack: demo
version: 1.0.0
variables:
  env_var:
    type: string
    from:
      prompt: Enter env var

services:
  node:
    extends:
      file: docker-compose.yml
      service: node
      variables:
        ENV_VAR: ${env_var}

or do we want to allow undeclared env-variables in compose?

IMHO extends should behave like it's part of the stack yml. No need to re-declare variables.

So you should do:
stack: demo
version: 1.0.0
variables:
  ENV_VAR:
    type: string
    from:
      prompt: Enter env var

if your compose yaml has $ENV_VAR?

I think the specific example here would also fail on Kontena 1.3 without any variables: ... section in the top-level kontena.yml?
The interpolation in the extended docker-compose.yaml is fixed in #2950, if you explicitly define the variable in the extending kontena.yml. I think that's enough to fix the 1.4 regression?


[ENG-2336] Send all additional DOI data over datacite

Purpose
Currently we are only sending a small portion of our node's metadata to datacite, this fix will send all the available metadata.
Changes

modifies formatting for metadata being sent to datacite
add management command to sync pre-this-PR metadata

QA Notes
Please make verification statements inspired by your code and what your code touches.

Verify at https://doi.test.datacite.org/repositories/demo.cos/dois that DOIs are being minted when a project or draft reg is registered. ⚠ You will need appropriate credentials to do this. ⚠
Verify Registration process is timely and error free.

Documentation
🐞 fix, no docs.
Side Effects
None that I know of.
Ticket
https://openscience.atlassian.net/browse/ENG-2336

A couple of comments. Looks good though!

LGTM!


feature/mx-1632 structured file support

PR Context

So far mex drop only allows JSON file upload.
This establishes compatibility with most common structured file formats.

Added

support for more structured file formats

And one more: could you document the allowed file types in swagger as well? That makes using the API easier.


GPU device placement issues

I think most of the issues encountered by users result from device strings processing and mismatch. I'm trying a bunch of error-prone combinations and trying to see how can those mismatch be catched and corrected.
For now I only tried very small changes.
2 (known) issues remain:

prior device mismatch
embedding net with custom density estimator mismatch

But both items do not expose any device attribute for now so maybe more invasive changes could be needed

looks good, thanks!
Regarding the prior device: there is a function for checking the prior device, but it is not used currently:
https://github.com/mackelab/sbi/blob/bc4d43bf60ec714790ef8baa4b08cc6e5b821e45/sbi/utils/torchutils.py#L42-L51
We could use this function in sbi/inference/base.py to check the prior (if it is not None).
Regarding the embedding and density estimator: this happens outside of the SBI loop, no? E.g., the user would use get_nn_models function to build their density estimator with embedding net, right? Here one quick fix would be to check the embedding net for its device, and then
a) throw an error that it should be on the CPU for now, or
b) throw a warning and move it to CPU to then compose it with the density estimator on the CPU.
I would tend to b). And you?
@michaeldeistler @jan-matthis what do you think?

Codecov Report

Merging #610 (bc4d43b) into main (b8551d0) will decrease coverage by 0.32%.
The diff coverage is 89.47%.

:exclamation: Current head bc4d43b differs from pull request most recent head b4b4063. Consider uploading reports for the commit b4b4063 to get more accurate results

@@            Coverage Diff             @@
##             main     #610      +/-   ##
==========================================
- Coverage   66.77%   66.45%   -0.33%
==========================================
  Files          67       66       -1
  Lines        4199     4158      -41
==========================================
- Hits         2804     2763      -41
  Misses       1395     1395

Flag
Coverage Δ

unittests
66.45% <89.47%> (-0.33%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

sbi/analysis/__init__.py
100.00% <ø> (ø)

sbi/utils/conditional_density_utils.py
55.74% <ø> (ø)

sbi/utils/sbiutils.py
76.32% <86.04%> (+2.54%)
:arrow_up:

sbi/analysis/conditional_density.py
51.51% <100.00%> (-4.05%)
:arrow_down:

sbi/inference/posteriors/base_posterior.py
77.27% <100.00%> (-2.18%)
:arrow_down:

sbi/inference/posteriors/direct_posterior.py
98.11% <100.00%> (-0.23%)
:arrow_down:

sbi/inference/posteriors/mcmc_posterior.py
73.57% <100.00%> (-0.19%)
:arrow_down:

sbi/inference/posteriors/rejection_posterior.py
37.50% <100.00%> (-8.16%)
:arrow_down:

sbi/inference/posteriors/vi_posterior.py
66.66% <100.00%> (-15.95%)
:arrow_down:

sbi/samplers/rejection/rejection.py
41.25% <100.00%> (-0.73%)
:arrow_down:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8551d0...b4b4063. Read the comment docs.

I'm having trouble understanding which cases are going wrong when.
Is the issue that get_posterior_nn() (etc) can not handle the embedding_net being on the GPU?

Yes, that's one of the problems. We designed sbi such that the user would just say device="cuda" and we take care of moving everything to the device. However, it happens that users have their priors, data, or embeddings on the device already, and then posterior_nn() would compose a Flow with mixed devices and things break.

Hey ! Sorry I had a bunch of stuff to work on, I'll work on the aforementioned issues some more tonight. @janfb I personally like APIs that are not error compliant but indicate clearly what went wrong -so option a-. But sbi is generally built around the "warn and correct" concept so I'll try to implement b

@michaeldeistler I can show you the cases that go wrong in screenshare like I did with @janfb at some point if you want?

I've done a couple functions that check what seems to be the most common error cases. I'm not sure I have propagated those functions everywhere needed though. For the embedding_net I managed to implement a "warn and correct" behavior, but not for the prior check because I feel like I would have needed to "wrap" a potentially custom prior into another prior that replicates the sampling and then moves the data accordingly. Tell me what you think ?

Arf I realize I redefined a prior checking function. I'll remove it.

@LouisRouillard I think quickest for now would be to just do a single commit with the black formatting changes. Then I'll quickly have a look :)

Ok so I adaanced on the tests. I think I have good coverage over the few functions I implemented. I expanded the integration test test_train_with_different_data_and_training_device and added a new integration test test_embedding_nets_integration_training_device that tackles the use case that was failing for me. It checks that everything runs even with wrong device assignments and that the users is warned when data or modules are moved automatically.

I sadly have conflicts, I'll try to rebase over main and pray it's not too bad ^^'

Watch out for sbi/utils/plot.py -- the file got deleted along the way

Great! Is this ready for review?

Ah just saw discord...I'll review it now

All test in tests/inference_on_device_test.py and pytest tests/user_input_checks_test.py including slow and gpu run locally !


node EBADENGINE

We get this error:
EBADENGINE Unsupported engine {
EBADENGINE   package<EMAIL_ADDRESS>EBADENGINE   required: { node: '>=16 <18' },
EBADENGINE   current: { node: 'v18.12.1', npm: '8.15.1' }
EBADENGINE }

Does ORT use the installed node version? What if it isn't installed? I think it should not use whatever is installed, but the required version. I think ORT should use https://github.com/tj/n which is what node developers use to use different node versions.
This is specially important for multi-repo systems using different versions of node.

Does ORT use the installed node version?

Yes, ORT uses the node version in PATH (the same is true for all external tools ORT depends on). But depending on how you run ORT that may mean different things. If you run ORT from the Docker image, the node version (currently 18.14.2) from the Docker image will be used, not the one from the host.

What if it isn't installed?

Analysis will fail completely. ORT will not try to bootstrap node or any other tool.

I think ORT should use https://github.com/tj/n which is what node developers use to use different node versions.

The combination of bootstrapping n plus running n auto or n engine indeed looks interesting.

This is specially important for multi-repo systems using different versions of node.

Which means the bootstrapping would need to happen in beforeResolution, not once in the Docker image.


Folders

Type: Bug
I make certain changes when I check out to another branch (mono repo), where the folders and structure have changed, and when I check out to the develop branch, folders stay there.
VS Code version: Code 1.75.0 (e2816fe719a4026ffa1ee0189dc89bdfdbafb164, 2023-02-01T15:23:45.584Z)
OS version: Windows_NT x64 10.0.22000
Modes:
Sandboxed: No
Remote OS version: Linux x64 <IP_ADDRESS>-microsoft-standard-WSL2

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscoreces:30445986
pythondataviewer:30285071
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vsclangdc:30486549
c4g48928:30535728
dsvsc012:30540252
azure-dev_surveyonecf:30548226
pyindex848:30577860
nodejswelcome1cf:30587006
2e4cg342:30602488
pyind779cf:30657577
f6dab269:30613381
pythonsymbol12cf:30657549

Additionally, when I click on a component, all of the node modules open instead of just the ones that are inside the component.

It doesn't recognize changes, its bad.

@GIT-MDW, it would be extremely helpful if you would be able to create a recording (https://gifcap.dev) to illustrate the issue. Based on the initial post, it seems that you are using WSL. If that is the case, are you storing your files in the Windows file system or the Linux file system?

ctrl + space doesn't show files such as .css may be there are another  types that i don't know

ctrl + shift + E doesn't show correct folder place

Sorry I cant because its private

@GIT-MDW, could you please try to reproduce the problem in a new repository? Thanks!

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


Remove duplicate GetLatestEpoch

Remove a duplicate, and inconsitent call to getLatestLogRoot.
Previously, GetLatestEpoch had a flow like this:

GetLatestLogRoot
Calculate LatestMapRevision
GetLatestLogRoot (which could be different!)
GetMapRevision

Codecov Report

Merging #990 into master will increase coverage by 0.14%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #990      +/-   ##
==========================================
+ Coverage   49.77%   49.92%   +0.14%
==========================================
  Files          29       29
  Lines        2033     2029       -4
==========================================
+ Hits         1012     1013       +1
+ Misses        840      835       -5
  Partials      181      181

Impacted Files
Coverage Δ

core/keyserver/epochs.go
37.83% <0%> (+0.99%)
:arrow_up:

core/client/get_and_verify.go
28.94% <0%> (ø)
:arrow_up:

core/client/client.go
30% <0%> (+0.62%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8e8705...f7c27a5. Read the comment docs.

PTAL


Allow to use Excalidraw to draw over a Map Background (google ou any other option).

Hello, 👋
1st. I love Excalidraw as a designer I like simple and efficient tools. For me, only one tool is missing.
The vector (draw) tools. (Like Sketch, Glyphs or Illustrator.)
I am working on an open source side project.
For this project I would like to use Excalidraw as a drawing tools on a map (Google Map or any other map).
I would like to be able to set and get the coordinate of some object relative to the map (to do calculations). Area, distance... ft, miles, and more.
How is the best way to "try" do this ?

Do a fork and to do this very specific feature on my side?
Extend Excalidraw in any way ? Plugins...
Try to add this feature in the Excalidraw and pull a request ?

Any advice is welcome.
How to add some user data value to the JSON Excalidraw format, for example ?
Thank you again for sharing this project.

Have you tried our recently release package?
https://github.com/excalidraw/excalidraw/tree/master/src/packages/excalidraw

Have you tried our recently release package?
https://github.com/excalidraw/excalidraw/tree/master/src/packages/excalidraw

No, not yet, but I will. That blow my mind.
Nice and smart.
Thank you.

No, not yet, but I will. That blow my mind.
Nice and smart.
Thank you.

How'd this go? Is it easy to georectify excalidraw drawings onto the map? I'm thinking of trying this myself.


Update schedule.yml

FYI, you should be setting your secrets in your project's settings not directly in the schedule.yml file. Setting them here gives anyone access to both your GH auth token and Wakatime tokens, potentially exposing unwanted details of your system and edited files.


feat(dal): restore edges on component upgrade

When a component is upgraded to a different schema variant, we need to recreate all the edges (if possible). This attempts to do that, and also updates the component diagram summary row to have the new sockets.

bors r+


⚠ Jenkins Plugins Artifactory has degraded performance

In 4e275be, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1157 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 05d4369 after 23 minutes.


Is 'eponymous' a fork?

Is 'eponymous' a fork?
https://github.com/eponymousjs/eponymous

It certainly seems to be. I got in touch with the author and he's updated the license file to clarify the heritage of the code.
We're not involved in the fork but I'm hopeful that some good improvements will come from it. We'll be watching for changes to incorporate back into angularjs-server.


Upgrade to Spring Cloud Finchley

Finchley is compatible with Spring Boot 2.
See https://spring.io/blog/2018/03/23/spring-cloud-finchley-m9-has-been-released

I've created a feature/Finchley branch: https://github.com/arey/spring-petclinic-microservices/tree/feature/Finchley
Datasource initialization failed. There is a circular dependency:

The dependencies of some of the beans in the application context form a cycle:

   entityManagerFactory defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]
      ↓
   entityManagerFactoryBuilder defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]
      ↓
   jpaVendorAdapter defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]
┌─────┐
|  scopedTarget.dataSource defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]
↑     ↓
|  org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker

Thinking the issue could be related to https://github.com/spring-projects/spring-boot/issues/12374 I've tested Spring Boot 2.0.1, but it does not fixed it.

Meege into master


esp_http_client: Add SOURCE method for NTRIP (IDFGH-2029)

Now that 742d4689dd728cb3bd5f06747b04b7d8930b7d02 has been merged, the NTRIP v1.0 protocol is almost entirely supported using the http client, which greatly simplifies its implementation.
In order to allow data upload to an NTRIP caster, the custom SOURCE method is required. The existing SUBSCRIBE/UNSUBSCRIBE/etc are already non-standard methods.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@nebkat Is there any publicly available server instance to validate working of NTRIP with ESP-IDF


[Native-Animated] Support for Animated.interpolate

This change adds native animated support for Animated.interpolate
Animated.interpolate allows for defining nodes that outputs an interpolated value of their input node based on the interpolation node configuration. For now native animated implementation only supports a linear interpolation for a given input and output ranges (ranges can consists of multiple segments). Native interpolation node is compatible with the JS implementation with the exception that not all attributes that can be used in JS are supported. Before we migrate interpolation node from JS->native we verify that only supported props are used.
Test Plan
Run JS tests: npm test Libraries/Animated/src/__tests__/AnimatedNative-test.js
Run java tests: buck test ReactAndroid/src/test/java/com/facebook/react/animated

@facebook-github-bot shipit


Update Field.php

Please check if the PR fulfills these requirements

[ ] The commit message follows our guidelines
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

What is the current behavior? (You can also link to an open issue here)

What is the new behavior (if this is a feature change)?

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

Other information:

Fixes a weird issue where the $model->structure getter suddenly stops working. Should be faster anyways to call the getStructureAttribute method directly. Not sure why it breaks, but this fixes it


create a proper release for Maven central repo

Latest version (1.1.3) released to Maven Central on 7/18/2017:
http://search.maven.org/#search|ga|1|owasp-security-logging


OAK-10465 : backported changes to 1.22 branch

I created a separate PR because the changes are a bit different for the 1.22 branch.
Maven scope is compile for 1.22 whereas it is provided for the trunk.

Ah, I missed that.

@rishabhdaim - could you please update the fixVersion info in the ticket?

@rishabhdaim - could you please update the fixVersion info in the ticket?

done.


ADMIN PR: Upgrade Ondatra version to head

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 73.191%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1608

Relevant Lines:
2197

💛  - Coveralls


Add Bearer Auth Helper Method In Connections

This adds one more helper method to create a Bearer AuthZ connection across the connection files. Easy low-risk change.

@xuzhang3 @apudovkin-ms
Low risk change. Please review

ping @nechvatalp

@nechvatalp can you please review this? It's an easy low risk change to add a helper function


GitHub Actions: build.yml: add platform parameter

This parameter allows us to build RT flavor of EVE and publish it as well as NVIDIA, Pollux and IMX flavours of EVE for ARM

As per request by @rouming and @rene

@uncleDecart for some reason the PR doesn't run the build workflow with these changes (hence get-run-id fails)

@rouming I added also platform to buildondemand action which we can invoke manually to try it out

Try to approve, but I didn't see the build run from Roman's approval so it looks like the issue is still present.

Yes, the build still does not produce output hence get run id fails.

Okay, we will have to change eve artefact name. On it...

Try again

And it failed with the same issue again:
Error: Unable to download artifact(s): Artifact not found for name: eve-kvm-amd64
@uncleDecart  I will politely ignore this PR until I get a positive confirmation that this issue has been fixed.

@eriknordmark tests started to work, failures seems not related to changes in GitHub Actions. Image is pulled and tests are ran on the PR image.

@eriknordmark tests started to work, failures seems not related to changes in GitHub Actions. Image is pulled and tests are ran on the PR image.

Thanks. I'll merge this.

Merge conflict with this PR https://github.com/lf-edge/eve/pull/4041. Eden should be bumped to 0.9.9 first and then this PR should be updated


eval(repr(op)) fails for cirq.SingleQubitCliffordGate.

Description of the issue
See title.
How to reproduce the issue
>>> z_clifford = cirq.SingleQubitCliffordGate.from_pauli(cirq.Z).on(cirq.LineQubit(2))
>>> repr(z_clifford)
cirq.SingleQubitCliffordGate(X:-X, Y:-Y, Z:+Z).on(cirq.LineQubit(2))
>>> eval(repr(z_clifford))
Traceback (most recent call last):

  File "/usr/local/google/home/tanujkhattar/anaconda3/envs/cirq/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3457, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-14-99e3d364586d>", line 1, in <module>
    eval(repr(z_clifford))

  File "<string>", line 1
    cirq.SingleQubitCliffordGate(X:-X, Y:-Y, Z:+Z).on(cirq.LineQubit(2))
                                  ^
SyntaxError: invalid syntax

Cirq version
0.16dev

Taking this.


Write more tests

Title of change
Write more tests
Checklist

[x] Unit tests written
[ ] Manually tested
[x] New dependencies were added with yarn
[x] Prettier & ESLint were run

Deployed to: http://staging.quran.com:32903

Deployed to: http://staging.quran.com:32905

I love how prettier makes tests easier to read.

Deployed to: http://staging.quran.com:32910

Deployed to: http://staging.quran.com:32911


Default single resolver no longer supports slug

Regarding: #2112 single resolver documentId undefined -> now returns null
I would like to ask about this breaking change. In my code I rarely use documentId to query for a single document. Most of the time I use slug as documented here:
http://docs.vulcanjs.org/resolvers.html#Collection-Resolvers:
single: used to return a single document matching an _id or slug
Querying by slug using the single resolver is no longer supported. Is that really what was intended by this change? If so, why?
Additionally, the single resolver could be passed a selector to query for a single document by any field that was designated "selectable" and "unique" in the schema. This seems to be a legitimate use case when you have a secondary key in your collection. This is now also deprecated.
This will result in tons of refactoring in my code - after just completing tons of refactoring when we standardized graphql CRUD operations. Before I embark on that I just want to make sure that this was intended.

Sorry I completely missed that, I thought that #2112 only changed the behavior of the single resolver when nothing was passed at all (no documentId, no _id, no slug, etc.). I didn't realized it broke fetching documents by slug, which is not great since people (well, me at least) currently rely on it.
I reverted some of the changes in the PR for now until we can find a better way of doing things to both adopt the new behavior described by @eric-burel while also preserving backwards compatibility.

@eric-burel You can accomplish what you want, I believe, by iterating through the own enumerable properties of the selector object, and making sure that each one has a non-empty value - otherwise return { result: null }.
If you are ambitious, you could also verify that each of the keys in selector are defined in SimpleSchema as "selectable" and "unique".

Yep definitely. Unicity actually can't really be checked statically, you would need a concept of multiple fields primary key instead. (you can have 2 fields that are not unique at all, but the combination of the two is unique, I don't think you can express this in SimpleSchema/MongoDB). You can just check that fields are selectable, and if there is only one field, that it is unique.
We can also throw an error if 2 documents are found, imo that's an error case of the single resolver.
The thing is that syntax needs thinking as those features are massively used and thus can be changed afterward. So in any case we will have to create a POC and validate it together, I was too eager to merge.
I'll refactor the DataTable a bit I think (to make it easier to replace its style without duplicating logic, as for the forms), it will be the occasion to think about terms/resolvers/hocs API and propose ideas.


Index customizer broken in 1.6.x?

Had to adjust my docker images today for building stuff, and I noticed that my index customizer script (originally devised in #1174 - now listed in the README) was failing because of not finding uppercase_mb:
undefined method `uppercase_mb' for #<Asciidoctor::PDF::IndexCatalog:0x0000557242ea0f60>
Did you mean?  uppercase_pcdata
  Use --trace for backtrace

From here, it looks like the upcase method is probably what we want to use? Looks like that's a standard Ruby string method, so probably wasn't being used before due to not being in the oldest supported ruby?
https://my.diffend.io/gems/asciidoctor-pdf/1.5.4/1.6.0#d2h-121859
if so, the README should be updated.

You're correct that uppercase_mb was a patch to provide proper Unicode support for the String#upcase method. Any custom code should be using APIs provided by Ruby, not private methods in Asciidoctor PDF. Since this method was always private, I don't feel there is a needed to mention it in the README. However, I would consider adding a mention of it to the CHANGELOG file. The commit you are looking for is 7e9e0bbe9f4ddad41b38725c607588c65618ebae. If you'd like this to be mentioned in the CHANGELOG, feel free to submit a PR to add it.

Thanks for the quick response! #1977 fixes the reference in the README to uppercase_mb. (It was already there) No idea if anyone else hit this, I didn't realize it was a private method, but it might be worth putting in the changelog. I'll add that to my PR.

Ah. I now see what you are referring to. That was definitely a mistake to use uppercase_mb in that example as that is not a method that should have been exposed.

PR merged. Thanks!


Ability to delete cards outside of editorial workflow

Projects that do not utilize the editorial workflow have no way of deleting posts through the interface. We want to provide a way to do this with minimal impact to the current interface, so we'll simply provide the delete button when viewing the post directly, on the Entry page.
Requirements

Add a "Delete Entry" button to the entry page, on the left side of the footer bar, opposite the "Save" and "Cancel" buttons.
The button can be styled to match the Cancel button - text with no background or border.

would love to help with this O_O, any direction?

@juliankrispel I actually have this working already for the non-editorial workflow - I'm currently trying to figure out a confusing issue with the editorial workflow that's causing the UI to get out of sync with the repo. I'd appreciate a second pair of eyes to try to find the source of the bug, though - if you're still interested, let me know here or on Gitter and I'll push my branch and explain the issue in more detail.

@Benaiah I am interested/would like to help!

This will be a nice feature. @Benaiah will this have control flag, something like deletable:true/false flag?

This will be a nice feature. @Benaiah will this have control flag, something like deletable:true/false flag?

@Benaiah it doesn't look like you really meant to close this in #485?

@tech4him1 why not? That PR enabled deletion in the simple workflow, which is the subject of this issue ("Ability to delete cards outside of editorial workflow").

Oops, I'm sorry, I thought it was for both because you said in #485 that "This does not close #291 completely".

@tech4him1 my bad, that's a mistake in #485.

Has this ability added? I receive this message when trying to delete entry - https://prnt.sc/haq723
What this error could be?

Netlify CMS version 0.7.3
this is from console:
http://prntscr.com/haq909

@aleksand88 This issue has been closed as deletion was added. For help resolving your issue, please reach out on our Gitter channel: https://gitter.im/netlify/NetlifyCMS.


[Feature Request] More sound playing options

Currently sounds can only be played through the included player in rx.audio
However this requires that either the sounds be located locally on the app machine (which is fine if you have a small number of mp3's you might be working with) or linked via some http url from elsewhere (which is not entirely super secure)
In case you have your sounds stored in some database somewhere and send it over via some urllib request there is currently no way to directly handle it. (write it to disk first and play it from there? Feels redundant).
Also the available props don't have access to more fine grained control of the sound. Such as setting the cursor back to the beginning or somewhere in the middle. You have to engage in dirty tricks with the playing and loop props. Assuming you're not using exclusively the visual part of the audio component.

Can you use a data uri? How big of audio files are you dealing with?
There is also an example on this page https://reflex.dev/docs/api-reference/browser_javascript/ for playing an audio file using the JavaScript APIs directly via rx.call_script

They're small. A few seconds each.
But I have tens of thousands. Easily several GB. And my usecase would prefer they start playing as quickly as possible once a button is pressed.
I was hoping I could avoid learning javascript but thanks. I'll check out that example too

If you have audio data stored in your database or something, you can set the audio URL to a base64 data uri it will play it. To restart from the beginning, clear the URL and reset it again.
Here is some sample code (I'm loading from disk here, but the owl_sound_effect could just be bytes from anywhere).
import base64
from pathlib import Path

import reflex as rx

owl_sound_effect = Path("assets/owl.mp3").read_bytes()

def convert_to_data_url(data, mime_type: str = "audio/mpeg"):
    base64_data = base64.b64encode(data).decode("utf-8")
    return f"data:{mime_type};base64,{base64_data}"

class State(rx.State):
    audio_clip: str = ""
    playing: bool = False

    def play_clip(self):
        self.audio_clip = ""
        yield
        self.audio_clip = convert_to_data_url(owl_sound_effect)
        self.playing = True

def index() -> rx.Component:
    return rx.fragment(
        rx.color_mode_button(rx.color_mode_icon(), float="right"),
        rx.vstack(
            rx.heading("Sound effect!"),
            rx.button("Play", on_click=State.play_clip),
            rx.audio(url=State.audio_clip, playing=State.playing, width="100%"),
        ),
    )

app = rx.App()
app.add_page(index)
app.compile()

Oh you can play base64 directly. That definitely looks like a useful workaround.
I'll give it a try tomorrow

Does not seem to be any actionable work remaining so considering this as fixed. Please reopen with specific requests if any remain.


(MicroPython support) ImportError: no module named 'typing' pi pico

I am trying to run adafruits HX8357 display with the raspberry pico, running micropython using Blinka.
when I import displayio I get the error above

Is displayio even compatible with Micropython ? It has seemingly never been tested there, has it ? It's not an issue to use typing without guards in a library that is not supported on MP. This should probably be an issue in https://github.com/adafruit/Adafruit_Blinka_Displayio

I don't know why I didn't even think about that sooner. Yeah, this library will not run on MicroPython. It requires the Pillow library and I'm fairly sure you need a single board computer for that.

Is it fair to close this then?  Or should there be any action item?

I don't know why I didn't even think about that sooner. Yeah, this library will not run on MicroPython. It requires the Pillow library and I'm fairly sure you need a single board computer for that.

Displayio is part of CircuitPython, yes? So even if I use CircuitPython on my Raspberry Pi Pico it wouldn't work?

Display IO relies on C python libraries, particularly pillow, in order to function. It isn't designed to run on the Pico.

Displayio is part of CircuitPython, yes? So even if I use CircuitPython on my Raspberry Pi Pico it wouldn't work?

To clarify, you can use displayio if you run Circuitpython instead of Micropython, as it is a builtin library of CP. It's a display and graphics library that needs low level access and optimizations. I don't know what Micropython has to offer as graphics libraries, but I believe it has LVGL bindings that can be compiled in (maybe not by default), which you could use independently of whatever blinka libraries you need.

I think now I get it. Blinka is really created for single board computers, and not microcontrollers, right?
So, running python with Blinka, displayio would work because of the python libraries mentioned like pillow.
Blinka also works in Micropython on a microcontroller but I guess only very limited then. is this correct?

@Nicchessla Blinka runs fine on MicroPython. It was really this library that was created for SBCs.

Closing as this is pretty much written for linux and not micropython. It would be better to better to either use MicroPython's RGB Display library or go with CircuitPython instead.


Fix faulty logic in call method of Runner class, fixes #370

Change res = cb(cb_name) and res to res = cb(cb_name) or res
so that __call__ method does not always return False but returns
True if at least one of the callbacks returns True.

Thanks! There's some redundant stuff in this diff - could you please run the things mentioned here and re-submit, checking that the only thing in the diff is the actual change you made?: https://docs.fast.ai/dev/develop.html#things-to-run-after-git-clone

@fg91 bump!

Stripped the notebook.
Ps.: Sorry for the late reaction, started a new position which kept me busy.

Many thanks

The same mistake (res = cb(cb_name) and res should be res = cb(cb_name) or res) exists also in notebook 9b in which Runner is converted to Learner. Would you like me to prepare a pr?
One additional question:
in Learner.fit we do:
for epoch in range(epochs):
    self.do_begin_epoch(epoch)
    if not self('begin_epoch'): self.all_batches()

However, do_begin_epoch already returned return self('begin_epoch') so we call all callbacks twice which is not necessary, correct?
Shell I change this to
if not self.do_begin_epoch(epoch): self.all_batches()

in the pr?

@fg91 perhaps you could try it and see if it works? If so, a PR would be welcome.


[manual-sync] Address CVE-2024-24786

Manual sync of:

https://github.com/open-cluster-management-io/governance-policy-framework-addon/pull/124

Closes #312

/cherry-pick release-2.10

/cherry-pick release-2.9

/cherry-pick release-2.8


population statistics: show degree selection only if that is relevant for study programme

eg for Tietojenkäsittelytieteen kandi there is no need to show degree selection but tietojenkäsittelytieteen koulutusohjelma the degree should be selectable

Same as #218


Benchmark suite for Turtle.Pattern

Implement one.

There's now a simple criterion-based benchmark in the bench branch.


Contrast isn't great in find results

Testing #145629
We draw the find match selection on top of the text which reduces contrast pretty badly
Terminal:

Editor:

Should we use a border instead of a solid rectangle to workaround this? It would look different from the editor but at least it wouldn't introduce contrast problems
This is what it looks like with these style overrides:
.xterm-decoration.xterm-find-result-decoration {
    border: 1px solid rgba(234, 92, 0, 1) !important;
    box-sizing: border-box;
    background: transparent !important;
}

The white background on selection is due to "terminal.selectionBackground": "#ffffff"
We should support border styles anyway to cover the high contrast case:

I like the idea of a border instead of rectangle. It's pretty hard to see otherwise

I'd like to work on this if no one else is

@martin2469 we'll be working on this in the next couple of days as it blocks the release

The border colour seems good for dark themes, but it's a bit hard to see for light ones.


Fix broken original zcash tests

There are several zebra original tests broken due to some changes we made for komodo network:

MAX_MONEY changed
interest calculation added
address prefix changed
min difficulty changed
target spacing for testnet changed
zcash network upgrades activation heights changed.
in total around 20 tests are broken when cargo test is executed.
We need to fix those tests.

For discussion:
I am thinking if it is possible to ensure that the original zcash tests work without our fixes, so we could always pull their latest changes with tests working.
As stated above zcash tests are broken due to network parameters changed for the kmd network.
So I think maybe we could leave their network parameters intact and create new networks for kmd main and testnet with our specific parameters so we always will add komodo rules only for our networks. This would make the original zcash rules working for their network parameters and zcash tests should work as well.
(Ofc there some difficulties with that, f.e. having different MAX_MONEY for both networks - this would require changes in core level like add the network param to the Amount object or provide access from Amount to the current network in another way)

some tests were fixed for komodo
the rest were disabled
(PR #2)