Sporadic scala.MatchError inside Task.timeoutTo

Sometimes Task.timeout fails with scala.MatchError with the following stacktrace.
scala.MatchError: null
	at monix.eval.Task.$anonfun$timeoutTo$1(Task.scala:2208)
	at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:147)
	at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:108)
	at monix.eval.internal.TaskRestartCallback$$anon$1.run(TaskRestartCallback.scala:128)
	at monix.execution.internal.Trampoline.monix$execution$internal$Trampoline$$immediateLoop(Trampoline.scala:66)
	at monix.execution.internal.Trampoline.startLoop(Trampoline.scala:32)
	at monix.execution.schedulers.TrampolineExecutionContext$JVMOptimalTrampoline.startLoop(TrampolineExecutionContext.scala:146)
	at monix.execution.internal.Trampoline.execute(Trampoline.scala:39)
	at monix.execution.schedulers.TrampolineExecutionContext.execute(TrampolineExecutionContext.scala:65)
	at monix.execution.schedulers.BatchingScheduler.execute(BatchingScheduler.scala:50)
	at monix.execution.schedulers.BatchingScheduler.execute$(BatchingScheduler.scala:47)
	at monix.execution.schedulers.ExecutorScheduler.execute(ExecutorScheduler.scala:34)
	at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:72)
	at monix.execution.Callback$Base.run(Callback.scala:244)
	at monix.eval.internal.TaskConnection$TrampolinedWithConn.run(TaskConnection.scala:227)
	at monix.execution.internal.Trampoline.monix$execution$internal$Trampoline$$immediateLoop(Trampoline.scala:66)
	at monix.execution.internal.Trampoline$ResumeRun$1.run(Trampoline.scala:51)
	at monix.execution.internal.forkJoin.AdaptedForkJoinTask.exec(AdaptedForkJoinTask.scala:27)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

At first glance, the only place in code where MatchError could happen in timeoutTo is if Task.race produces null.
I can't reproduce this case manually, as it happens rarely under some load on unknown conditions.
Environment:
Java:
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

Scala:
2.12.8
Monix libraries in classpath:
io.monix.monix_2.12-3.0.0-RC2
io.monix.monix-catnap_2.12-3.0.0-RC2
io.monix.monix-eval_2.12-3.0.0-RC2
io.monix.monix-execution_2.12-3.0.0-RC2
io.monix.monix-java_2.12-3.0.0-RC2
io.monix.monix-kafka-1x_2.12-1.0.0-RC2
io.monix.monix-reactive_2.12-3.0.0-RC2
io.monix.monix-tail_2.12-3.0.0-RC2

Thanks for the report, are you able to see if it also happens in 3.0.0-RC3? (monix-kafka Rc4)

I think I would have to update my production environment to 3.0.0-RC3 version which I am a bit afraid of for now :) Sadly, couldn't reproduce on test environment, maybe not enough load

Hi @Z1kkurat,
This could be a contract violation of Callback in Task.create / Task.async.
Are you by any chance calling calback.onSuccess multiple times, concurrently?

@Avasil this issue with the null value comes from TaskRestartCallback. We'll need to investigate.
@Z1kkura can you give us some minimal code we could use in testing? Any pointers would be appreciated.

@alexandru I'm afraid the only thing this code does is creating AsyncSubject and waiting for firstL with timeout, not much of a logic.
val subject = AsyncSubject.apply[Unit]()
subject.firstL.timeout(timeout)

@Z1kkurat this helps a lot, thanks. We'll investigate.

A test that does this at high load shouldn't be too hard to build.

Hi @Z1kkurat,
I've forked 3.0.0-RC2 and made some relatively small changes on this branch, in order to identify the issue: https://github.com/monix/monix/pull/917
I've pushed a new test version with it: 3.0.0-RC2-cb74787
(synchronizing to Maven Central now)
I have a suspicion that this is a contract violation, but I couldn't reproduce your issue exactly as it is. Can you help us by trying out this intermediate version?
What this version does is to protect the Callback in Task.create. In case there's a contract violation and that callback gets called twice, then an IllegalAccessException will get thrown exactly where the violation happens. And Task.create itself should catch this exception and report it via the reporter in your used Scheduler.

I will try to use this version in the next few weeks, thanks!

@Z1kkurat any updates on this issue?
Can you try testing the version that I've prepared in that ticket? (#917)

seeing this as well
scala.MatchError: null
	at monix.tail.internal.IterantFromReactivePublisher$IterantSubscriber.finish(IterantFromReactivePublisher.scala:119)
	at monix.tail.internal.IterantFromReactivePublisher$IterantSubscriber.onComplete(IterantFromReactivePublisher.scala:152)
	at monix.reactive.internal.rstreams.ReactiveSubscriberAsMonixSubscriber.$anonfun$onComplete$1(ReactiveSubscriberAsMonixSubscriber.scala:91)
	at monix.execution.Ack$AckExtensions$.syncOnContinue$extension(Ack.scala:117)
	at monix.reactive.internal.rstreams.ReactiveSubscriberAsMonixSubscriber.onComplete(ReactiveSubscriberAsMonixSubscriber.scala:91)
	at monix.reactive.observers.SafeSubscriber.$anonfun$onComplete$1(SafeSubscriber.scala:67)
	at monix.execution.Ack$AckExtensions$.syncOnContinue$extension(Ack.scala:117)
	at monix.reactive.observers.SafeSubscriber.onComplete(SafeSubscriber.scala:63)
	at monix.reactive.internal.operators.DoOnErrorOperator$$anon$1.onComplete(DoOnErrorOperator.scala:34)
	at monix.reactive.internal.operators.UpstreamTimeoutObservable$$anon$1.onComplete(UpstreamTimeoutObservable.scala:128)
	at monix.reactive.internal.rstreams.AsyncSubscriberAsReactiveSubscriber$$anon$1.onComplete(SubscriberAsReactiveSubscriber.scala:153)
	at monix.reactive.observers.buffers.AbstractSimpleBufferedSubscriber$$anon$1.signalComplete(SimpleBufferedSubscriber.scala:156)
	at monix.reactive.observers.buffers.AbstractSimpleBufferedSubscriber$$anon$1.fastLoop(SimpleBufferedSubscriber.scala:236)
	at monix.reactive.observers.buffers.AbstractSimpleBufferedSubscriber$$anon$1.run(SimpleBufferedSubscriber.scala:129)
	at monix.execution.internal.InterceptRunnable.run(InterceptRunnable.scala:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

We'll need further investigation then.
@arcolife what version are you using?

@alexandru Note that it's most likely a different issue in IterantFromReactivePublisher

@Avasil @alexandru  I created a separate issue for what @arcolife reported https://github.com/monix/monix/issues/1180
Thanks


Unsupported JetPack revision 35.3.1 aborting script

Required Info

Camera Model
D455

Operating System & Version
Linux L4T (Ubuntu 20.04)

Kernel Version (Linux Only)
5.10.104-tegra

Platform
NVIDIA Jetson

Language
python

Segment
Robot

Issue Description
Trying to install librealsense from source to solve pyrealsense2 issues. I accidentally deleted the rules file before so installing librealsense from scratch again. When trying to apply patched kernel module for Jetson L4T, I get the error "Unsupported JetPack revision 35.3.1 aborting script"

Based on Issue https://github.com/IntelRealSense/librealsense/issues/11391#issue-1569854831, I need to match the supported JetPack version. Looking into the patch files, supported versions shown below.

Highest supported patch is 35.1 while I have 35.3.1.
Need suggestions on how to downgrade since there have multiple issues which say downgrading is not possible. Any help would be appreciated.

Hi @jidnyesha-patil  You should be able to downgrade by flashing the JetPack using the Nvidia SDK Manager tool, as described at the link below.
https://forums.developer.nvidia.com/t/downgrade-jetpack/169373/3
Alternatively, you could build librealsense from source in RSUSB backend mode by including the flag -DFORCE_RSUSB_BACKEND=TRUE in your CMake build instruction.  An RSUSB build of librealsense bypasses the kernel.

Got it! I am trying the RSUSB backend mode first since I must get SDK manager to allow flashing JetPack.I am currently using the pre-flash SD card that came with the kit. Will keep this thread updated.
Thank you @MartyG-RealSense

@jidnyesha-patil Try to append "35.3.1" to "35.1", like: "35.1 | 35.3.1")

The RSUSB mode installation worked!

That's great to hear that RSUSB was successful for you, @jidnyesha-patil - thanks very much for the update!


Image method that ensures rendition matches both specified dimensions and maintains aspect ratio of source image

There should be an image resizing method similar to max which produces a resulting image that exactly matches the given dimensions.
For example (copied from docs), a portrait image of width 1000, height 2000, treated with the max dimensions 1000x500 (landscape) would result in the image shrunk so the height was 500 pixels and the width 250.  However, the resulting output would be 1000x500 with the 500x250 image centered horizontally and vertically (in case the dimensions required upscaling which cannot happen).
There should optionally be an argument for the background color when the image does not fill the entire canvas.  Transparent images should be allowed to have a transparent output canvas.

I'm intrigued what specific situation you've experienced, where this suggestion would help. Could you elaborate?

Related: #1694

Like the comment mentioned above this is typically useful when the image being produced is passed to a third party or used in a format where one does not directly control the display characteristics.  If the image is simply being rendered as html output for viewing on the website it is simple to use a centering div.  But in certain situations it is preferable to return an image of the exact requested dimensions so that positioning can be controlled


Simplify mutex counter, replacing map with int

The map added complexity for no benefit. We only need an int to illustrate the mutex here

@googlebot I signed it!


Please don't add fontawesome inline

Please, remove the fontawesome inside the component that destroys any other use of fontawesome pro.
https://use.fontawesome.com/releases/v5.2.0/css/all.css should not be placed inline in any case...

Hi @ekandreas , sorry for the inconvenience, for that font awesome, we will remove in next release.
again thank you for your feedback.

This issue has been solved in v0.2.4.

@coroo I think that as a result of removing font awesome, the refresh button, and possibly other buttons, now show a blank icon unless you manually add font awesome. Yet it seems that the installation documentation does not mention that you must manually add font awesome


Please exetend it to a full blown package manager for a linux distribution

@org-tekeli-borisp please see https://github.com/astral-sh/uv/issues/9452


Setting the VMI-Under-Test CPUs to be four

This PR is changing the amount of CPUs requested to be four.
Doing this allows the checkup to run the Oslat test on a full core (in this checkup - guest CPUs 2-3).
By doing this Oslat is optimized to not have neighboring CPU noise.

@orelmisan let's hold this PR until we have a cluster with the HCO alignCPUs enabler

passed on CNV4.15 cluster:
make e2e-test
mkdir -p /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache
podman run --rm \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup:/go/src/github.com/kiagnose/kubevirt-realtime-checkup:Z \
	-v /home/ralavi/go/src/github.com/kiagnose/kubevirt-realtime-checkup/_go-cache:/root/.cache/go-build:Z \
	-v /home/ralavi/.kube/sno03-cnvqe2-rdu2:/root/.kube:Z,ro \
	--workdir /go/src/github.com/kiagnose/kubevirt-realtime-checkup \
	-e KUBECONFIG=/root/.kube/kubeconfig \
	-e TEST_NAMESPACE=realtime-checkup-1 \
	-e TEST_CHECKUP_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup:devel \
	-e VM_UNDER_TEST_CONTAINER_DISK_IMAGE=quay.io/ramlavi/kubevirt-realtime-checkup-vm:latest \
	docker.io/library/golang:1.20.12 \
	go test -v ./tests/... -test.v -test.timeout=1h -ginkgo.v -ginkgo.timeout=1h
=== RUN   TestTests
Running Suite: Tests Suite - /go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests
========================================================================================
Random Seed:<PHONE_NUMBER>

Will run 1 of 1 specs
------------------------------
[BeforeSuite]
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/tests_suite_test.go:39
[BeforeSuite] PASSED [0.002 seconds]
------------------------------
Checkup execution should complete successfully
/go/src/github.com/kiagnose/kubevirt-realtime-checkup/tests/checkup_test.go:80
• [784.826 seconds]
------------------------------

Ran 1 of 1 Specs in 784.828 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestTests (784.83s)
PASS
ok  	github.com/kiagnose/kubevirt-realtime-checkup

logs:
$ oc logs -f job/realtime-checkup
2024/01/01 12:54:28 kubevirt-realtime-checkup starting...
2024/01/01 12:54:28 Using the following config:
2024/01/01 12:54:28 	"vmUnderTestTargetNodeName": ""
2024/01/01 12:54:28 	"vmUnderTestContainerDiskImage": "quay.io/ramlavi/kubevirt-realtime-checkup-vm:latest"
2024/01/01 12:54:28 	"oslatDuration": "10m0s"
2024/01/01 12:54:28 	"oslatLatencyThresholdMicroSeconds": "45µs"
2024/01/01 12:54:28 Waiting for VMI "realtime-checkup-1/realtime-vmi-under-test-xp2nr" to boot...
2024/01/01 12:56:23 VMI "realtime-checkup-1/realtime-vmi-under-test-xp2nr" had successfully booted
2024/01/01 12:56:23 Login to VMI under test...
2024/01/01 12:57:13 Running Oslat test on VMI under test for 10m0s...
2024/01/01 13:07:17 Oslat test completed:
taskset -c 2-3 oslat --cpu-list 2-3 --rtprio 1 --duration 10m0s --workload memmove --workload-mem 4K
oslat V 2.60
Total runtime: 		600 seconds
Thread priority: 	SCHED_FIFO:1
CPU list: 		2-3
CPU for main thread: 	0
Workload: 		memmove
Workload mem: 		4 (KiB)
Preheat cores: 		2

Pre-heat for 1 seconds...
Test starts...
Test completed.

        Core:	 2 3
Counter Freq:	 2096 2096 (Mhz)
    001 (us):	 0 0
    002 (us):<PHONE_NUMBER><PHONE_NUMBER>
    003 (us):	 1 0
    004 (us):	 136 137
    005 (us):	 49 49
    006 (us):	 5 4
    007 (us):	 17 18
    008 (us):	 16 16
    009 (us):	 4 4
    010 (us):	 4 3
    011 (us):	 1 1
    012 (us):	 0 0
    013 (us):	 0 0
    014 (us):	 0 0
    015 (us):	 0 0
    016 (us):	 0 0
    017 (us):	 0 0
    018 (us):	 0 0
    019 (us):	 0 0
    020 (us):	 0 0
    021 (us):	 0 0
    022 (us):	 0 0
    023 (us):	 0 0
    024 (us):	 0 0
    025 (us):	 0 0
    026 (us):	 0 0
    027 (us):	 0 0
    028 (us):	 0 0
    029 (us):	 0 0
    030 (us):	 0 0
    031 (us):	 0 0
    032 (us):	 0 0 (including overflows)
     Minimum:	 1 1 (us)
     Average:	 2.000 2.000 (us)
     Maximum:	 10 10 (us)
     Max-Min:	 9 9 (us)
    Duration:	 599.736 599.736 (sec)

[root@realtime-vmi-under-test-xp2nr cloud-user]#
2024/01/01 13:07:17 Max Oslat Latency measured: 10µs
2024/01/01 13:07:17 Trying to delete VMI: "realtime-checkup-1/realtime-vmi-under-test-xp2nr"
2024/01/01 13:07:17 Waiting for VMI "realtime-checkup-1/realtime-vmi-under-test-xp2nr" to be deleted...
2024/01/01 13:07:23 VMI "realtime-checkup-1/realtime-vmi-under-test-xp2nr" was deleted successfully

@orelmisan let's hold this PR until we have a cluster with the HCO alignCPUs enabler

now ready for review

We need to consider masking CPUs 2 and 3 to be realtime, while the other two are not using spec.domain.cpu.realtime.mask.
https://kubevirt.io/user-guide/virtual_machines/numa/#running-real-time-workloads

Change: Set the guest SMT to disabled.

We need to consider masking CPUs 2 and 3 to be realtime, while the other two are not using spec.domain.cpu.realtime.mask. https://kubevirt.io/user-guide/virtual_machines/numa/#running-real-time-workloads

I am not against, but let's consult with @matosatti , and add it in a separate PR.

We need to consider masking CPUs 2 and 3 to be realtime, while the other two are not using spec.domain.cpu.realtime.mask. https://kubevirt.io/user-guide/virtual_machines/numa/#running-real-time-workloads

I am not against, but let's consult with @matosatti , and add it in a separate PR.

I don't think this is necessary: only setting all vCPUs as realtime is supported officially (because there are known problems by not doing that).


[doc bug]: Title shows .NET 5 when .NET 6 documentation is selected

Article URL
https://learn.microsoft.com/en-us/dotnet/desktop/wpf/?view=netdesktop-6.0
Technology
WPF (Windows Presentation Foundation)
Version

.NET 6

Description
Main title shows .NET 5 when .NET 6 version of the documentation is selected.

In case this is something that needs a fix, I offer to work on this.

Thank you for opening this. I'm going to have to add .NET 7 to the system soon and I'll go through and look at all the places where it says .NET 5/6 and adjust them. Thanks again.


LV2 State implementation

original at https://github.com/atsushieno/android-audio-plugin-framework/issues/33
The corresponding AAP API exists (for simple byte array based API, not detailed like Atom), but it is not implemented.

Now lack of this feature causes crashes on JUCE AudioPluginHost when we try to set "bypass" on LV2 plugins.

Actually it does not seem to always happen. Need more investigation.

It does try to save state, so any plugin that crashes there (e.g. assert(false)) would crash there.

We have partial jalv port for State part now, but it is disabled due to unexpected confusion between ordinal workers and state workers. Once we figured it out, we still need the actual serialization as well.

No idea what was causing the blocker, but state worker does not seem to block non-state worker on sfizz anymore, so the state worker is enabled again.

Basic state serialization and deserialization are implemented in 7de29ed. It still needs support for non-float types.


Beta

Added the ID switch to Set/New-AutotaskAPIResource
AutotaskAPI.zip
prior to changes :

after changes :

Should probably be the changed to ParentID to make the use case clear.

Closing. Manually merging these changes.


🐞 - Stackblitz throws NullInjectorError

Which @taiga-ui/* package(s) are the source of the bug?
Don't known / other
Please provide a link to a minimal reproduction of the bug
https://taiga-ui.dev/next/components/button#appearance
Is this issue blocking you?
Non-Blocking
Description
Many stackblitz examples throws this error
Error: R3InjectorError(AppModule)[InjectionToken
[TUI_IS_MOBILE]: Mobile browser detection ->
InjectionToken An abstraction over window.navigator.userAgent object ->
InjectionToken An abstraction over window.navigator.userAgent object ->
InjectionToken An abstraction over window.navigator.userAgent object]:

NullInjectorError: No provider for InjectionToken An abstraction over window.navigator.userAgent object!

This error is not reproducible for https://taiga-ui.dev/components/button#appearance (v3.15.0), but can be reproduced in next-version https://taiga-ui.dev/next/components/button#appearance (future v3.16.0).
Probably, it can be somehow connected with this changes
https://github.com/Tinkoff/taiga-ui/pull/3364
Angular version
12
Taiga UI version
3.16.0
Which browsers have you used?

[X] Chrome
[ ] Firefox
[ ] Safari
[ ] Edge

Which operating systems have you used?

[X] macOS
[ ] Windows
[ ] Linux
[ ] iOS
[ ] Android

I am not sure but I think that problem will be fixed after release.
Possible explanation:
next-version of demo opens stackblitz with @ng-web-apis/common: "2.1.0", but published @taiga-ui/cdk contains @ng-web-apis/common: "2.0.1". They will be synced in the next release.

fixed by 3.16

@nsbarsukov doesn't work in the next version


[STAL-2369] feat: add javascript.ImportsPackage stub

What problem are you trying to solve?
Currently, we do not have a way to easily check if a JavaScript file imports a certain package. This is problematic because we cannot write rules that only target files that import React. It is also difficult to implement in individual rules as JavaScript has many different ways of importing modules and/or items, so it'd be really helpful if we had some way to just check if some module is imported without relying on our users to write complicated queries to check for every possible import style.
What is your solution?
Add a stub function javascript.importsPackage that allows the user to check if a package is imported or not. While the groundwork has been laid to allow users to get really powerful information about what packages are imported and be creative about how they use it, we do not expose all of this functionality yet as the ddsa.javascript API has not been finalized. As such, for now, we only expose a simple stub function to check for a package being imported or not, with the nitty-gritty details like what imports a file contains and if they're aliases or modules or not being unexposed in the API. The relevant classes and methods that contain this logic are marked as private.
The logic is quite similar to the already-implemented Go and Terraform FileContexts.
Tests cover the various ways a user may import JavaScript packages, since we have to account for ES6 modules, CommonJS requires, and dynamic imports, along with the various ways we can destructure these imports.
Alternatives considered
What the reviewer should know
The PackageImport data structure represents a JavaScript import. An import in JavaScript can either bring in the whole module or just a subset of the module, which are available directly in the file instead of having to be prefixed with the module name to be accessed. To elaborate on what I mean here, consider the following two imports:
import 'foo'; // brings the foo module into the file
// ...
foo.method();
console.log(foo.variable);

vs.
import { method, variable } from 'foo'; // brings in a subset of the module foo and brings them in directly
// ...
method();
console.log(variable);

In the first example, we consider foo to be the imported "name", with no alias or no module it's being imported from. However, the second example brings in method and variable as the imported "name", and is imported from foo.
The logic for figuring out whether a package is imported or not is as follows:
If we import a module directly, that is the module we check against, i.e., import.name. Otherwise, if we bring in a subset, we consider where it is imported from to be the module we check against, since we are bringing items from that module into scope, albeit a subset of it.
Hence, the logic for detecting if a module is imported or not looks like the following, in pseudocode:
function importsPackage(packageName) {
  for (const _import of this.imports) {
    if (_import.isAModule) {
      return _import.name === packageName;
    } else {
      return _import.importedFrom === packageName;
    }
}

squashing & merging


Preserve a user-defined content-length for HEAD request

Motivation:
A Content-Length for chunked encoding should be null. When HEAD method is set, a response should not contain any content. Since there is no content in the response body, it causes the response handler to update Content-Length to 0. #4509
Modifications:

Allow explicitly setting a null content-length via HttpHeaderBuilder.contentLength(-1).

If -1 is set to .contentLength(), the value isn't added to the actual headers. It is used as a mark to indicate a content-length should not be set depending on the payload.
HttpHeaders.isContentLengthSet() will return true if -1 is set.

Set HttpHeaders.contentLength() to -1 if headers is decoded from the transport and the content-length is empty.

It makes isContentLengthSet() return true and then AggregatedHttp{Request,Response} keep the original content-length for null content.

Set endOfStream to true where a request or response is created with an empty body. This could optimize the performance for an empty body response because it can make a stream close early without writing an empty chunk.

Result:

You no longer see 0 Content-Length when a response is streamed for a HEAD request.
Fixes #4509

What do you think about leaving contentLength() property as it is and introducing a new property contentLengthUnknown()? i.e.

Revert the semantic changes in contentLength() and remove isContentLengthSet().
Add a new boolean property contentLengthUnknown

setter.contentLengthUnknown(boolean)
getter.isContentLengthUnknown()

If a user calls setter.contentLength(..), contentLengthUnknown becomes false.
If a user calls setter.contentLengthUnknown(true), contentLength is unset.

Sounds good. I like the idea and the better naming.

Time to take some dusts off, @ikhoon  😉

Sorry for the delay. 😅 I will get back to this next week.

Can I review this? 🤔

Yes, please. This PR is ready to review.

🚀 🚀 🚀


feat(search): update search to DocSearch v3

👋🏻 Popper team!
This PR integrates the new version of DocSearch, that better reflects what we think documentation search should be.
Description:
Reducing users' efforts to get them started integrating your product is key to adoption. We created DocSearch in 2015 as a community effort to solve this problem.
We've had time to try different approaches since then and would like to propose you a new documentation search experience.
This new experience has been well received and is now live on the Docusaurus v2 integration: facebook/docusaurus#2815.
Preview:
.

Mobile
Desktop

Changes:
You can read about all the new features in the Docusaurus PR , but here's a summary:

We replaced the dropdown list for a modal
You can open the search modal with Ctrl + K, ⌘ + K or /
No hard refresh (we leverage the Gatsby navigation API in this integration)
The experience works on mobile and is close to a native mobile experience
Recent searches
Favorite searches
Selection search
Search suggestions
Offline detection
The bundle size decreased and we now provide React components which can be lazy loaded.

What's next
This new DocSearch version has been used in production for a couple of weeks now, on the DocSearch website and on a few Docusaurus v2 integrations.
This version is marked as alpha for now because the underlying libraries that we use (e.g., autocomplete-core) are still being worked on at @algolia to make sure that it handles more cases than DocSearch and usual autocomplete search experiences. We plan to go stable in the coming weeks but no major changes will happen to DocSearch.
Let us know what you think!

Thanks! Sorry I didn't realize I wasn't following this repo 😳


Value cannot be null

What You Are Seeing?
An exception was thrown while invoking the constructor 'Void .ctor(IFileSystem, ICakeEnvironment, INuGetContentResolver, ICakeLog, ICakeConfiguration)' on type 'NuGetPackageInstaller'. ---> System.ArgumentNullException: Value cannot be null.
Parameter name: path1
What is Expected?
Expected to get it working
What version of Cake are you using?
0.35.0
Are you running on a 32 or 64 bit system?
x64
What environment are you running on?  Windows? Linux? Mac?
NAME="Linux Mint"
VERSION="19.1 (Tessa)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 19.1"
VERSION_ID="19.1"
How Did You Get This To Happen? (Steps to Reproduce)
I'm using bootstap sh script with sample cake.build file in the root
Output Log
./build.sh --verbosity=Diagnostic
Feeds used:
  /home/disposer/.nuget/packages/
  https://api.nuget.org/v3/index.json

All packages listed in /home/disposer/work/dfin/venue-cake/tools/packages.config are already installed.
Module directory does not exist.
NuGet Config not specified. Will use NuGet default mechanism for resolving it.
Error: Autofac.Core.DependencyResolutionException: An exception was thrown while activating Cake.Commands.BuildCommand -> Cake.Core.Scripting.ScriptRunner -> Cake.Core.Scripting.Analysis.ScriptAnalyzer -> λ:Cake.Core.Scripting.Processors.Loading.ILoadDirectiveProvider[] -> Cake.NuGet.NuGetLoadDirectiveProvider -> Cake.NuGet.Install.NuGetPackageInstaller. ---> Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor 'Void .ctor(IFileSystem, ICakeEnvironment, INuGetContentResolver, ICakeLog, ICakeConfiguration)' on type 'NuGetPackageInstaller'. ---> System.ArgumentNullException: Value cannot be null.
Parameter name: path1
  at System.IO.Path.Combine (System.String path1, System.String path2) [0x00003] in <65984520577646ec9044386ec4a7b3dd>:0
  at NuGet.Common.NuGetEnvironment.GetFolderPath (NuGet.Common.NuGetFolderPath folder) [0x00058] in <c9b6d8ba44d840a594d9bae9ddd53803>:0
  at NuGet.Common.NuGetEnvironment.GetFolderPath (NuGet.Common.NuGetFolderPath folder) [0x00064] in <c9b6d8ba44d840a594d9bae9ddd53803>:0
  at NuGet.Configuration.XPlatMachineWideSetting..ctor () [0x0000c] in <d9bf5a3137394fefa4038f4faad72388>:0
  at Cake.NuGet.Install.NuGetPackageInstaller..ctor (Cake.Core.IO.IFileSystem fileSystem, Cake.Core.ICakeEnvironment environment, Cake.NuGet.INuGetContentResolver contentResolver, Cake.Core.Diagnostics.ICakeLog log, Cake.Core.Configuration.ICakeConfiguration config) [0x00109] in <cba3a8dab4d847a1a02801159438f04a>:0
  at (wrapper dynamic-method) System.Object.lambda_method(System.Runtime.CompilerServices.Closure,object[])
  at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate () [0x0008f] in <2fad6ed004784647b9555cb639266f73>:0
   --- End of inner exception stack trace ---
  at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate () [0x00113] in <2fad6ed004784647b9555cb639266f73>:0
  at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance (Autofac.IComponentContext context, System.Collections.Generic.IEnumerable`1[T] parameters) [0x000b2] in <2fad6ed004784647b9555cb639266f73>:0
  at Autofac.Core.Resolving.InstanceLookup.Activate (System.Collections.Generic.IEnumerable`1[T] parameters, System.Object& decoratorTarget) [0x0002c] in <2fad6ed004784647b9555cb639266f73>:0
   --- End of inner exception stack trace ---
  at Autofac.Core.Resolving.InstanceLookup.Activate (System.Collections.Generic.IEnumerable`1[T] parameters, System.Object& decoratorTarget) [0x00071] in <2fad6ed004784647b9555cb639266f73>:0
  at Autofac.Core.Resolving.InstanceLookup.Execute () [0x0007f] in <2fad6ed004784647b9555cb639266f73>:0
  at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance (Autofac.Core.ISharingLifetimeScope currentOperationScope, Autofac.Core.IComponentRegistration registration, System.Collections.Generic.IEnumerable`1[T] parameters) [0x00071] in <2fad6ed004784647b9555cb639266f73>:0
  at Autofac.Core.Resolving.ResolveOperation.ResolveComponent (Autofac.Core.IComponentRegistration registration, System.Collections.Generic.IEnumerable`1[T] parameters) [0x00000] in <2fad6ed004784647b9555cb639266f73>:0
  at Autofac.Core.Resolving.ResolveOperation.Execute (Autofac.Core.IComponentRegistration registration, System.Collections.Generic.IEnumerable`1[T] parameters) [0x00016] in <2fad6ed004784647b9555cb639266f73>:0
  at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent (Autofac.Core.IComponentRegistration registration, System.Collections.Generic.IEnumerable`1[T] parameters) [0x00041] in <2fad6ed004784647b9555cb639266f73>:0
  at (wrapper dynamic-method) System.Object.lambda_method(System.Runtime.CompilerServices.Closure)
  at Cake.Commands.CommandFactory.CreateBuildCommand () [0x00000] in <a105507eb0df4ed3b55d00ab6c3a93e0>:0
  at Cake.CakeApplication.CreateCommand (Cake.CakeOptions options) [0x000ab] in <a105507eb0df4ed3b55d00ab6c3a93e0>:0
  at Cake.CakeApplication.Run (Cake.CakeOptions options) [0x0000e] in <a105507eb0df4ed3b55d00ab6c3a93e0>:0
  at Cake.Program.Main () [0x000d1] in <a105507eb0df4ed3b55d00ab6c3a93e0>:0

Not sure if it's the same issue exactly, but I get the same error message when I try to use the external NuGet client after previously having used the in-process one (either by changing cake.config or by using the --nuget_useinprocessclient flag).
I can solve the issue by fully deleting the tools folder and then re-running .\build.ps1.

(Moving to GitHub Discussions/Q&A)


Error, any Idea about this Failor

python pimpd.py
Traceback (most recent call last):
File "pimpd.py", line 37, in
mpd_client = reconnectingclient.ReconnectingClient()
File "/home/pi/pimpd/pimpd/reconnectingclient.py", line 11, in init
super(ReconnectingClient, self).init(use_unicode=True)
TypeError: init() got an unexpected keyword argument 'use_unicode'
Regards Peter

Hi
please double check that you've got python-mpd2, not python-mpd installed.

Thank You !!!!
Regards Peter


Pretty sure dhcpcd isn't necessary if you're using systemd-networkd

Yep.. it's replaced by systemd-resolved


feature-request: add ability to override email address for resend

Use Case:

Sometimes a submission will come to the wrong form. In this case it would nice to manually specify an email and override the original admin recipient.

Possible UI/UX:

[x] User selects all submissions they want to resend
[x] The user presses the Resend button (or perhaps clicks a dropdown to choose "Resend to Manual Email"
[x] A modal window opens where the user can see the email address of the original email
[x] The user can override or specify a new email address
[x] The user sends and the recipient receives the email

Thanks again :)

@mi-ke-dev I have just released a new 1.1.0 version. This feature was already included in the roadmap. :)


[N6] Wrong value in AddressSearch after refresh

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Create NewDot account, verify it, log in
Create Workspace
Do the add Bank account step (first step) of VBA flow (use this SO)
You should be in Company Step
Type something in the Company Address field, choose an address result
Type something in another field
Refresh the page

Expected Result:
The Company Address field should have the value we input before refresh
Actual Result:
The Company Address doesn't have the right value:
https://user-images.githubusercontent.com/87341702/137230266-e6c60a2c-34f4-47b6-9e45-7b50b1b91a72.mov
Workaround:
The user can input the address again
Platform:

Where is this issue occurring?

Web
iOS
Android
Desktop App
Mobile Web

Version Number:
Reproducible in staging?:
Reproducible in production?:
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
Expensify/Expensify Issue URL:
Issue reported by:
Slack conversation:
View all open jobs on GitHub

Ah yeah, I hit the undefined, undefined bit too. Noted that here: https://expensify.slack.com/archives/C020EPP9B9A/p1634207884072800?thread_ts=1634147967.452400&cid=C020EPP9B9A
Are you taking this one on, @aldo-expensify?

I'm not able to reproduce this.

jk it happens with 88 Kearney Street

Unfortunately the fix I tried did not work. Going to keep digging

In this case, I think we'll just want the premise field to become addressStreet here.

Well, it actually seems to work if we add types: 'address' in the query object
Do you want me to submit a PR?

If you input 88 Kearny Street you get the full address result and not the premise with missing data.

In the example of 88 Kearny Street, it looked like the premise had all the necessary data. But that might not work consistently.
Also, heads up! I encountered some queries where Google Places would return a result, but then spit back an invalid address. For example, if you input 25220 Quail Ridge Road, Escondido, CA 97027 then Google Places will take that as a valid input, but the place returned in the callback is just Quail Ridge Road, Escondido, CA 97027 (missing street number). I think this maybe means that there's a development planned there but that address doesn't exist yet. Let's make sure we cover that case as well.

If you input 88 Kearny Street you get the full address result and not the premise with missing data.

In the example of 88 Kearny Street, it looked like the premise had all the necessary data. But that might not work consistently.
Also, heads up! I encountered some queries where Google Places would return a result, but then spit back an invalid address. For example, if you input 25220 Quail Ridge Road, Escondido, CA 97027 then Google Places will take that as a valid input, but the place returned in the callback is just Quail Ridge Road, Escondido, CA 97027 (missing street number). I think this maybe means that there's a development planned there but that address doesn't exist yet. Let's make sure we cover that case as well.

😨  will test that too! thanks!

Yeah, so it looks like the type: address doesn't cover that case alone. Is it safe to assume that any valid address street must start with a number?

you are right, it is missing the number :(

If so, we should throw an error when the end result does not begin with a number?
I guess that is an option, I'll have a look again to see if there is a way we could filter incomplete results

Is it safe to assume that any valid address street must start with a number?

I think this is probably a safe assumption 99.99% of the time, so if we find counter-examples then we can handle them when they pop up?

I think you are probably right there

2. street_number

That sounds good to me, but I would suggest doing (2) and (3) as filters in our backend so we don't show those results to the user. I think it is better experience than showing results that eventually will lead to validation errors. What do you think about that?

It might not be a perfect solution, but it will be an improvement from what we have currently. We can address issues with other addresses as they come up. I think we should maybe also add some unit test coverage of the saveLocationDetails function to help maintain it in the future.

What is the estimate on the PR being ready for review? Anything I can do to help?

Hmm, I have to update the "Steps / QA" section. I can do that now. About the tests that @roryabraham suggested, I'm having some doubts about it.
To make it easier to test, I would remove the function saveLocationDetails from AddressSearch and make it receive a callback onChangeText as a parameter. This function is very related to the AddressSearch component and probably won't be used anywhere else, so I'm a bit hesitant about moving it to a separate file just for testing purposes. Another option is to define it in the AddressSearch component file and export it from there, but I haven't seen another case where we export something else besides the component.


examples/dialog.md - Add missing close button to sample html

The modal dialog sample is missing close button, which exists in the actual demo script on this page.

Good catch, thanks!


Material You?

Hey,
i was wondering if the App has Material You? And if not, could you add that? Because it is just great and would make it a lot better to look at!

Yes. Why do you think it doesn't?


Add a transfer function shader widget

Adds a new shader widget directive transferFunction. Transfer functions can be used to map specific data values to color and opacity values via a set of control points. In between control points, the values are linearly interpolated.
A preview of the underlying lookup texture is shown in the UI panel for a transfer function.
Points are added to the panel via a click, their color is changed by setting a new color in the picker, and then right-clicking the point. Points are removed via a modifier and double-click on the point.

Hey @chrisj, for sure, no worries!
From the link you sent, I've created a transfer function that mimics an invlerp so you can compare here
More generally, you can add points by clicking, change their color by right clicking after selecting a color in the color picker, and remove points by shift double clicking for something like this
At the moment, the transfer function does not show any indication of the underlying data distribution, which can make creating them a little tricky. Let me know if you have questions, happy to help


Add pre-commit hook for hlint

hlint is a Haskell linting and code suggestion tool. Output looks something like this:
$ hlint --cross cross1.hs cross2.hs print.hs
print.hs:1:8: Error: Use print
Found:
  putStrLn $ show "Hello"
Why not:
  print "Hello"

cross1.hs:6:9: Warning: Reduce duplication
Found:
  a = 1
  b = 2
  c = 3

Why not:
  Combine with cross2.hs:5:9

2 suggestions

@sds How would you suggest we parse and display this kind of output? Should we only consider the first line of each suggestion, ignoring the "Found" and "Why not" stanzas?

How would you suggest we parse and display this kind of output? Should we only consider the first line of each suggestion, ignoring the "Found" and "Why not" stanzas?

I think that would be a great place to start. If it turns out that you need more information, you can always add it.

Makes sense to me. The user could always just run hlint manually to get the verbose output.


(BKR-1551) Updates for Beaker 4

Beaker 4's subcommands significantly change how hypervisor methods are called during test runs. Instead of performing the full run with one command, the workflow now looks more like this:

Run beaker init --hosts=<hosts-file> - this creates a .beaker folder that contains subcommand_options.yaml. This subcommand options file preserves state over the course of subsequent subcommands
Run beaker provision. This creates the SUTs requested in the hosts file, but does not run any tests.
Run beaker exec <tests-or-suites> to run the tests on the SUTs. Repeat if needed.

Previously, beaker-docker used host objects to store several types of ruby objects for easy access, including a logger, the docker image, and the docker container. https://github.com/puppetlabs/beaker/pull/1557 will stop beaker from writing any :logger objects to saved yaml files entirely, and this PR addresses the other issues:

First commit: instantiate a logger to replace the one that used to come from the host object
Second commit: stop persisting host['docker_container'], and instead persist host[docker_container_id] -- use this ID to find the right container via the Docker API instead of expecting a container object to be available already
Third commit: similarly, stop persisting host['docker_image'] and persist a host['docker_image_id'] instead

There are still more ways to improve beaker-docker's behavior when using Beaker subcommands, and I plan to make more PRs to beaker and this repo, but the changes in this PR are allowing me to use beaker-docker with Beaker 4.


Add a way to specify certain commands to not be registered to Brigadier

On platforms like Bukkit and Paper where Brigadier is optional we should have a way to specify certain commands to not be registered to Brigadier.
On platforms like Velocity where we always register with Brig, maybe we should implement a way to register these commands as greedy string commands (Bukkit style) internally.
edit:
A more robust and platform-agnostic approach may be to use command meta to indicate a node is no-brig, and then have that node and every node below it registered as a greedy string to brigadier.

On platforms like Velocity where we always register with Brig, maybe we should implement a way to register these commands as greedy string commands (Bukkit style) internally.

wouldn't it be better to do this for all cases where brigadier is used then? we could do it in the cloud brigadier manager in that case

That probably makes sense -- whatever metadata is used could be recognized by the brigadier manager.


Create Primary Rounded button blue

One need to make this component using HTML and Tailwind CSS. I would suggest to use Tailwind Playgrounds to make things faster and quicker.
Here is a reference to the component.

To view other details, Please check figma design
Color to be used:- Normal : bg-blue-600 & on hover : bg-blue-700
After building the component please raise a PR with a screenshot of the component and add the component in Button/src/Primary/text-button-rounded.html by replacing the existing code and rename the file to text-button-rounded-blue.html.
So, In short you need to do:-

Make the Component
Replace the code of Button/src/Primary/text-button-rounded.html with your code
Rename text-button-rounded.html to text-button-rounded-blue.html

If you need to use any icon please use it from Hero Icons
Good luck.

i want to work on this issue

I would like to work on it.


Grid subset

@SorooshMani-NOAA @BahramKhazaei-NOAA
Start the conversation of subset issues.
Scripts to l subset the hi-resolution mesh and attach it to the low-resolution one

@yunfangsun how would you like to proceed with this? Should we set up a meeting or do you have specific questions to address here?

@SorooshMani-NOAA @BahramKhazaei-NOAA
Hi Soroosh,
Could you please let me know your Ocsmesh env address on Hera, so I can play with your env on Hera, since Bahram is not available today, I would like to know if we could have a small meeting including Bahram next Monday or Tuesday to guide us through the process of it?
Thank you!
Best,
Yunfang

Hi @yunfangsun, sure the latest environment that I used on Hera was /scratch2/STI/coastal/noscrub/Soroosh.Mani/apps/miniconda3/envs/sim2, it might not have some of the latest developments in OCSMesh, but it does have subsetting related functionality.
Just for future reference, let me describe a little bit about the subsetting and what I mean by it in this context:
Subsetting usually means to extract a part of existing mesh only, but in the context we're talking about (especially using ocsmesh) it means to extract part of an existing fine mesh and attach it to an existing coarser mesh. This results in a mesh that is "optimized" for our "area of interest".
ocsmesh implements this functionality in the subset_n_combine script, which can be used through ocsmesh's CLI or the script file can be copied from the package (at ocsmesh/cli/subset_n_combine.py) and modified as desired and used.
To use it from CLI:
usage: ocsmesh scripts subset_n_combine [-h] [--rasters RASTERS [RASTERS ...]] [-o DIRECTORY] [-a] [-c CRS] [-d CUTOFF] [-s WIND_SPEED] [-n N_LAYERS] [--adv-upstream-area-max AREA]
                                        [--adv-rel-island-area-min AREA]
                                        fine_mesh coarse_mesh region_of_interset

positional arguments:
  fine_mesh             Path to the fine mesh
  coarse_mesh           Path to the coarse mesh
  region_of_interset    Path to the shape file for region of interest or storm track

optional arguments:
  -h, --help            show this help message and exit
  --rasters RASTERS [RASTERS ...]
                        List of rasters used for calculating depth cutoff
  -o DIRECTORY, --out DIRECTORY
                        Output directory
  -a, --outall          Flag to indicate all intermediate shapes to be written to disk
  -c CRS, --crs CRS     Input mesh CRS and output CRS [experimental]
  -d CUTOFF, --cutoff CUTOFF
                        Cut-off depth (positive up) for calculating region subsetted from fine mesh
  -s WIND_SPEED, --isotach-speed WIND_SPEED
                        Isotach (34, 50 or 64) to use if region of interset is storm track
  -n N_LAYERS, --adv-buffer-n-layers N_LAYERS
                        Number of buffer layers to mesh between fine and coarse mesh
  --adv-upstream-area-max AREA
                        Advanced option: maximum segment area for calculating upstream
  --adv-rel-island-area-min AREA
                        Advanced option: minimum segment area for dropping islands during shape cleanup

This script assumes there are 2 existing meshes, one finer than the other and these are to be combined based on the shape stored at region_of_interset shapefile. The region of interest is further cut off by depth specified by -d or --cutoff. If a wind speed is provided (using -s or --isotach-speed) the input shape metadata is searched for the shape with specified windspeed. This is assumed to be the 34, 50 or 64 knot wind speed provided for hurricanes as the original usecase of this script was for hurricane meshing.

With all the inputs, both the meshes are clipped by the input shape such that the meshes complement each other, then a buffer zone is cut out of the coarse mesh. In the last step this buffer zone is remeshed to provide a more or less seamless transition between the fine and the coarse mesh.

Soroosh, here are the sms mesh files Joseph sent to me:

I thought that new16.map is the mesh file to load, however, it gives me the following error:

@BahramKhazaei-NOAA, I'm actually not fully familiar with all the files SMS has. I know that OCSMesh supports reading 2dm files from SMS and nothing else. Can you save the mesh as 2dm using SMS? Or you can ask Joseph to provide the 2dm file to you instead.
I think the mesh should actually be in the h5 file in your case. Maybe it's a good idea to add a xmdf reader to ocsmesh too. Can you send (by email) the h5 file to me? If it looks easy I can add a reader for it

I sent you the files. I'm trying to see if I can convert to 2dm format. My SMS software needs a new license.

OK, great. In that case I'll just work on adding the reader capability later when I'm done with other issues/tasks.

Hi @SorooshMani-NOAA
I am using the environment /contrib/Yunfang.Sun/conda3/envs/simulation/ under noaa-nos-none-ca-hsofs-c.
There is no problem to load ocsmesh
from copy import deepcopy
from pathlib import Path
import requests
import tempfile
from datetime import datetime, timedelta
import zipfile
import fiona
import geopandas as gpd
from cartopy.feature import NaturalEarthFeature
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from pyproj import CRS
from shapely import geometry, ops
import xarray as xr
from rasterio.plot import show as rioshow
import ocsmesh
import pyschism
import pyschism.driver
There is no problem to use ocsmesh.Mesh.open etc.
However, when I try to use subset:
ocsmesh scripts subset_n_combine -h
It provides the following error:
Cell In[2], line 1
ocsmesh scripts subset_n_combine -h
^
SyntaxError: invalid syntax
Could you please give me any suggestions to fix it?
Thank you!
Yunfang

@yunfangsun, it seems that you're trying to use the command line entrypoint from inside a Python session. You need to run this command inside your bash, not in Python. When you install ocsmesh a new command is added to your environment (i.e. ocsmesh) and you can use it just like ls or pip to access meshing scripts, for example by calling ocsmesh scripts subset_n_combine -h inside your bash

@SorooshMani-NOAA
Thank you! I see, now I can use it in the bash.
Yunfang

Hi @SorooshMani-NOAA ,
When I am using ocsmesh scripts subset_n_combine --out new -s 34 -n 3 newatlant.2dm final_mesh.2dm file1.txt, I received the following error message:
/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/cli.py:24: UserWarning: Scripts CLI is used for experimental new features and is subject to change.
warnings.warn(
2023-05-15:15:05:51,894 ERROR    [collection.py:234] `file1.txt' not recognized as a supported file format.
Traceback (most recent call last):
File "fiona/ogrext.pyx", line 136, in fiona.ogrext.gdal_open_vector
File "fiona/_err.pyx", line 291, in fiona._err.exc_wrap_pointer
fiona._err.CPLE_OpenFailedError: 'file1.txt' not recognized as a supported file format.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/contrib/Yunfang.Sun/conda3/envs/simulation/bin/ocsmesh", line 8, in
sys.exit(main())
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/main.py", line 168, in main
OCSMesh(parser.parse_args(), ocsmesh_cli).main()
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/main.py", line 64, in main
self._cli.execute(self._args)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/cli.py", line 31, in execute
self._script_dict[args.scripts_cmd].run(args)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/subset_n_combine.py", line 112, in run
self._main(
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/subset_n_combine.py", line 532, in _main
poly_isotach = self._get_region_bounded_by_isotach(track_file, wind_speed)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/subset_n_combine.py", line 138, in _get_region_bounded_by_isotach
gdf_region_of_interset = gpd.read_file(track_file)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/geopandas/io/file.py", line 259, in _read_file
return _read_file_fiona(
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/geopandas/io/file.py", line 303, in _read_file_fiona
with reader(path_or_bytes, **kwargs) as features:
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/fiona/env.py", line 457, in wrapper
return f(*args, **kwds)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/fiona/init.py", line 335, in open
colxn = Collection(
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/fiona/collection.py", line 234, in init
self.session.start(self, **kwargs)
File "fiona/ogrext.pyx", line 587, in fiona.ogrext.Session.start
File "fiona/ogrext.pyx", line 143, in fiona.ogrext.gdal_open_vector
fiona.errors.DriverError: 'file1.txt' not recognized as a supported file format.
Could you please give me any suggestions?
Thank you!
Yunfang

Hi @SorooshMani-NOAA
with `ocsmesh scripts subset_n_combine --rasters /lustre/all/sun/data/data/gebco_2022_n90.0_s0.0_w-90.0_e0.0.tif --out new -s 34 -n 3 newatlant.2dm final_mesh.2dm file34'
There is no mesh generated, and It gives the following error message:
Traceback (most recent call last):
File "/contrib/Yunfang.Sun/conda3/envs/simulation/bin/ocsmesh", line 8, in
sys.exit(main())
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/main.py", line 168, in main
OCSMesh(parser.parse_args(), ocsmesh_cli).main()
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/main.py", line 64, in main
self._cli.execute(self._args)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/cli.py", line 31, in execute
self._script_dict[args.scripts_cmd].run(args)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/subset_n_combine.py", line 112, in run
self._main(
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/subset_n_combine.py", line 622, in _main
utils.finalize_mesh(jig_combined_mesh)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/utils.py", line 367, in finalize_mesh
boundary_polys = get_mesh_polygons(mesh)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/utils.py", line 156, in get_mesh_polygons
lines = get_boundary_segments(target_mesh)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/utils.py", line 129, in get_boundary_segments
raise ValueError(
ValueError: Mesh boundary crosses itself! Folded element(s)!
Could you give me any suggestions?

@yunfangsun this usually means one of the input meshes is not valid. Can you send me the location of the mesh, I can take a look and see what's going on

@SorooshMani-NOAA Another subsetting question: I have a grid that I clipped with OCSMesh and I need to write in gr3 format. I use the follwoing:
m_final.write('alaska_final_0_360_v2.gr3', format='gr3', overwrite=True)

and get this error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 1
----> 1 m_final.write('alaska_final_0_360_v2.gr3', format='gr3', overwrite=True)

File /scratch2/STI/coastal/noscrub/Soroosh.Mani/apps/miniconda3/envs/sim2/lib/python3.9/site-packages/ocsmesh/mesh/mesh.py:164, in EuclideanMesh.write(self, path, overwrite, format)
    161     savevtk(str(path), self.msh_t)
    163 else:
--> 164     raise ValueError(f'Unhandled format {format}.')

ValueError: Unhandled format gr3.

@SorooshMani-NOAA I think I was running the command wrong. So, the format is the grid format before write?
m_final.write('alaska_final_0_360_v2.gr3', format='2dm', overwrite=True)

@BahramKhazaei-NOAA maybe it's just a bug on ocsmesh side. Use grd as your format argument instead of gr3 and see if it works. 2dm is the format that is readable by QGIS. The extension in the path (I think) does not matter in this case, the format argument is what specifies the writing format.

@SorooshMani-NOAA
I have tried to set n layers from 9 to 40, but the layer number changing is not working for this problem.
The error message is still as follows:
Traceback (most recent call last):
File "/contrib/Yunfang.Sun/conda3/envs/simulation/bin/ocsmesh", line 8, in
sys.exit(main())
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/main.py", line 168, in main
OCSMesh(parser.parse_args(), ocsmesh_cli).main()
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/main.py", line 64, in main
self._cli.execute(self._args)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/cli.py", line 31, in execute
self._script_dict[args.scripts_cmd].run(args)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/subset_n_combine.py", line 112, in run
self._main(
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/cli/subset_n_combine.py", line 622, in _main
utils.finalize_mesh(jig_combined_mesh)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/utils.py", line 367, in finalize_mesh
boundary_polys = get_mesh_polygons(mesh)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/utils.py", line 156, in get_mesh_polygons
lines = get_boundary_segments(target_mesh)
File "/contrib/Yunfang.Sun/conda3/envs/simulation/lib/python3.9/site-packages/ocsmesh/utils.py", line 129, in get_boundary_segments
raise ValueError(
ValueError: Mesh boundary crosses itself! Folded element(s)!

I have tried to set n layers from 9 to 40, but the layer number changing is not working for this problem.

Wow! That's really high, it should have worked if this was the issue. Give me some time to see if I can come up with a better answer!

The issue is the finalize_mesh function. This function removes "pinched nodes" and "element islands" in a for loop. After the first round of clean up, when new set of pinched nodes are removed the error appears. This seems to be due to a shift in index vs id, but I'm not sure how. This is a snapshot of how mesh looks like when it becomes invalid during the finalization:

I need to spend some more time to say what is exactly happening under the hood. I'm also worried that this might be due shapely to version change.

@SorooshMani-NOAA I subset/clip from the larger mesh (hollow) based on the boundaries of the smaller mesh (black). However, the resulting mesh does not have any boundary nodes. This is written at the end of the resulting gr3 file:
0 ! total number of ocean boundaries
0 ! total number of ocean boundary nodes
0  ! total number of non-ocean boundaries
0 ! Total number of non-ocean boundary nodes

while in a normal gr3 file information regarding boundary nodes are written on the ending lines.

The resulted mesh should have open boundaries on semi-circular boundaries at N and S, and land boundaries on E and W sides.
I use the following cmd for clipping:
clipped_msht = utils.clip_mesh_by_shape(m_coarse.msh_t, shape=mesh_shape_fine, fit_inside=False)
m_final = Mesh(clipped_msht)

@yunfangsun please install ocsmesh version 1.3.1 (should be uploaded to PyPI in ~15min from now) or latest from main branch. I fixed the mesh cleanup issue and tested subsetting using your shape and mesh inputs. Please let me know if you run into any more issues. Thanks!

@SorooshMani-NOAA
Thank you! I have used the new ocsmesh to generate a subset mesh.

No problem. I'm glad that it works now!

Thanks Soroosh. So based on the set_open/island functions definition above, they cannot be used to update one node's boundary type? For example, in case I know the correct boundary type of a node I change that using the node ID (rather than a polygon).

they cannot be used to update one node's boundary type?

No currently OCSMesh does not support setting type for specific nodes. The closest thing to that you can do is to create shapely.geometry.Point object with the location of node of interest and buffer it by a small amount and use that resultant polygon and call set_open/set_land

The subset_mesh.boundaries.set_open(box(70, 170, 77, 176), merge=True) does not change anything in the boundary nodes.

Sorry I just noticed I mistakenly swapped lats and lons in the box definition! I will correct my previous comment now. Tested and it works:

Note that the box is a rough estimate, you can give it a more exact shape so that it exactly matches the boundary you're expecting.

@SorooshMani-NOAA Worked for me too. I tried everything on that cmd except changing the latlons. Thanks a lot for your help. This should help me fix boundary nodes.

From @yunfangsun: The subsetted mesh has the following issue during run:

: ABORT:  Elem        12141  and      2401065  have opposite orientation


sample use case: Type object with particular constructor signature

Annotating 'Type object with particular constructor signature' requires intersection.
Suppose we have a
# returns cached instance of SomeClass or creates a new instance
def get_cached_or_create(klass):
  if _is_cached(klass):
    return _get_cached(klass)
  else:
    print(f"creating instance for {klass}")
    instance = klass()
    _set_cached(klass, instance)
    return instance

Now, how should we type get_cached_or_create?
Naive typing would be to accept type[T]:
T = TypeVar('T')

def get_cached_or_create(klass: type[T]) -> T:
  ...

However, it does not express the fact T's constructor must be callable without argument.
Above signature suggests get_cached_or_create can be called with type object with any constructor signature.
The desired typing of get_cached_or_create in fact is:
T = TypeVar('T')

def get_cached_or_create(klass: type[T] & Callable[[], T]) -> T:
  ...

Hi!

it does not express the fact T's constructor must be callable without argument.

I think it's better to bind the type to a TypeVar rather than using Intersection for this.
For example...
from typing import Protocol, TypeVar

class HasConstructorNoArgsRequired(Protocol):
    def __init__(self) -> None: ...

T = TypeVar('T', bound=HasConstructorNoArgsRequired)

def spam(a: type[T]) -> T: ...

class Foo:
    def __init__(self) -> None: ...

class Bar:
    def __init__(self, a1: int) -> None: ...

spam(Foo)  # Valid
spam(Bar)  # Invalid, Pylance(reportGeneralTypeIssues)

type[T] & Callable[[], T] can be seen as expressing constraints on the constructor of the T class, but it also appears to express complex metaclass behavior.

Specifying __init__ signature is not enough because the class can override __new__ or its metaclass's __call__ (I think mypy has limitations with them, but pyright handles them alright).

the class can override __new__

I think we can do this to specify __new__ in Protocol.

or its metaclass's __call__

This would certainly be difficult to express with type annotations.
However, it's challenging to express or constrain metaclass behavior through static typing.
Therefore, I think specifying __init__ or __new__ signature in a Protocol is sufficient.

mypy has limitations with them

Certainly, I tried this on the mypy playground, it didn't result in an error.
Maybe this should be reported to the mypy developers.

it's challenging to express or constrain metaclass behavior through static typing.

Well, that's my point exactly. It will get much easier with intersections.

I understand. Indeed, this approach is necessary to cover those cases.
Providing examples of what is valid and invalid would make it more clear.
from collections.abc import Callable
from typing import Self, TypeVar

T = TypeVar('T')

class A:
    def __init__(self) -> None: ...

class B:
    def __new__(cls) -> Self: ...

class C:
    def __init__(self, num: int) -> None: ...

class D:
    def __new__(cls, num: int) -> Self: ...

class _Meta1(type):
    def __new__(cls, name, bases, attributes) -> Self: ...
    def __call__(self) -> 'E': ...

class E(metaclass=_Meta1): ...

class _Meta2(type):
    def __new__(cls, name, bases, attributes) -> Self: ...
    def __call__(self, num: int) -> 'F': ...

class F(metaclass=_Meta2): ...

def f1(klass: type[T] & Callable[[], T]) -> T: ...

f1(A)  # Valid
f1(B)  # Valid
f1(C)  # Invalid
f1(D)  # Invalid
f1(E)  # Valid
f1(F)  # Invalid  # Well, is this available for any static type checkers?

def f2(klass: type[T] & Callable[..., T]) -> T: ...

f2(A)  # Valid
f2(B)  # Valid
f2(C)  # Valid
f2(D)  # Valid
f2(E)  # Valid
f2(F)  # Valid

def f3(klass: type[T]) -> T: ...

f3(A)  # Valid
f3(B)  # Valid
f3(C)  # Valid
f3(D)  # Valid
f3(E)  # Valid
f3(F)  # Valid

Is my example OK?

Looks good. Exactly what I was thinking.
Thank you for clarifying!

Why do you need an intersection here? Isn't it sufficient to use Callable[[], T] as the input type? In your example, you've intersected this type with type[T], but it's not clear to me what value that provides from a typing standpoint.

Isn't it sufficient to use Callable[[], T] as the input type?

Probably I have over simplified the example.
My actual case required an object with following behaviours at the least:

is callable with no argument and returns its instance
has __name__ attribute (used for error messages)
Hashable (used as cache key) to identify T
can be passed to isinstance (for an assertion, which is not strictly necessary)
can be passed to issubclass (once, there was a plan to return an instance of subclass if already cached)

The gist is, though, that the doc comment for the param states "class object with void constructor", and there is no type annotation that exactly matches that.

Keep in mind that prior decisions from type checkers not to enforce something for "pragmatic" reasons is shining a light on why this might be needed. Unlike many of the prior ones, I find the intersection-based solution if intersections are added sufficient to resolve this.

So, continuing from example given in https://github.com/CarliJoy/intersection_examples/issues/26#issuecomment-1712777735:
class C:
    def __init__(self, num: int) -> None: ...

class CSub(C):
    # in practice, signature of __init__ method may be incompatible with parent class
    def __init__(self) -> None: ...

v1: type[C] = C  # Valid
v2: type[C] & Callable[..., C] = C  # Valid
v3: type[C] & Callable[[], C] = C  # Invalid
v4: type[C] & Callable[[int], C] = C  # Valid

v1_sub: type[C] = CSub  # Valid
v2_sub: type[C] & Callable[..., C] = CSub  # Valid
v3_sub: type[C] & Callable[[], C] = CSub  # Valid
v4_sub: type[C] & Callable[[int], C] = CSub  # Invalid


Don't remove the lychee config file

This fixes the removal of unncessary files, which currently removes too much, as it removes the config file. But also the lychee-bin and lychee-lib files:

$ touch lychee-bin; git status; rm -f lychee*!(lychee-bin|lychee-lib); git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add ..." to include in what will be committed)
lychee-bin
nothing added to commit but untracked files present (use "git add" to track)
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
(use "git add/rm ..." to update what will be committed)
(use "git restore ..." to discard changes in working directory)
deleted:    lychee.toml
no changes added to commit (use "git add" and/or "git commit -a")

Here, we can see that not only the lychee-bin file has been removed, but the lychee.toml file has been removed too (which prevents the action from running properly).
Now, with my new command:

touch lychee-bin; touch lychee-bar; git status; rm -f lychee!(-bin|-lib|*.toml); git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add ..." to include in what will be committed)
lychee-bin
lychee-bar
nothing added to commit but untracked files present (use "git add" to track)
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add ..." to include in what will be committed)
lychee-bin
nothing added to commit but untracked files present (use "git add" to track)

Note: I tried running this as rm -f lychee!(lychee-bin|lychee-lib|*.toml), but for some reason, the bin and lib files were still being deleted.

Thanks for the fix!


Upgrade open-interpreter to 0.1.7

open-interpreter 0.1.7 uses generator methods in respond method to replace recursion to avoid infinite recursion bug.

LGTM


[BUG] Longhorn Snapshots are not deleted after expired Backups (Velero)

Describe the bug (🐛 if you encounter this issue)
We are using Velero to create backups from the Kubernetes manifests and the persistent volumes (in our example we backup Harbor).
If we create a backup, Velero saves the K8s manifests to a Object Storage (MinIO) and creates snapshots resources to trigger Longhorn backups with the velero-plugin-for-csi. Longhorn writes the backups to another MinIO bucket.
If we delete a Velero backup or the backup is expired, the snapshot (snapshots.longhorn.io) are not deleted:

We are using Velero v1.9.4 with EnableCSI feature and the following plugins:

velero/velero-plugin-for-csi:v0.4.0
velero/velero-plugin-for-aws:v1.6.0

We have the same issue in Velero v1.11.0 with EnableCSI feature and the following plugins:

velero/velero-plugin-for-csi:v0.5.0
velero/velero-plugin-for-aws:v1.6.0

To Reproduce
Steps to reproduce the behavior:

Install the newest version of Velero and Rancher-Longhorn
In Longhorn configre a S3 Backup Target (we are usng MinIO for this)
Enable CSI Snapshot Support for Longhorn.
Create a backup (for example with the Schedule below): velero backup create --from-schedule harbor-daily-0200
Delete the backup velero backup delete <BACKUPNAME>
The snapshot (snapshots.longhorn.io) is not deleted.

Expected behavior
The snapshot is deleted.
Environment

Longhorn version: 102.2.0+up1.4.1
Velero version:
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Rancher-Longhorn Helm Chart
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE2, v1.25.7+rke2r1

Number of management node in the cluster: 1x
Number of worker node in the cluster: 3x

Node config

OS type and version: Ubuntu

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): VMs on Proxmox
Number of Longhorn volumes in the cluster: 17

Additional context
Velero Backup Schedule for Harbor
---
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: harbor-daily-0200
  namespace: velero #Must be the namespace of the Velero server
spec:
  schedule: 0 0 * * *
  template:
    includedNamespaces:
    - 'harbor'
    includedResources:
    - '*'
    snapshotVolumes: true
    storageLocation: minio
    volumeSnapshotLocations:
      - longhorn
    ttl: 168h0m0s #7 Days retention
    defaultVolumesToRestic: false
    hooks:
      resources:
        - name: postgresql
          includedNamespaces:
          - 'harbor'
          includedResources:
          - pods
          excludedResources: []
          labelSelector:
            matchLabels:
              statefulset.kubernetes.io/pod-name: harbor-database-0
          pre:
            - exec:
                container: database
                command:
                  - /bin/bash
                  - -c
                  - "psql -U postgres -c \"CHECKPOINT\";"
                onError: Fail
                timeout: 30s

VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: longhorn
  namespace: longhorn-system
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: driver.longhorn.io
deletionPolicy: Delete

VolumeSnapshotClass
In our second cluster, with Velero v1.11.0 installed, we created the following resource (but same issue here):
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: longhorn
  namespace: longhorn-system
  labels:
    velero.io/csi-volumesnapshot-class: 'true'
driver: driver.longhorn.io
deletionPolicy: Delete
parameters:
  type: bak

VolumeSnapshotLocation
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: longhorn
  namespace: velero
spec:
  provider: longhorn.io/longhorn

I have also opened an issue in velero: https://github.com/vmware-tanzu/velero/issues/6179

Maybe this issue is related to:

https://github.com/vmware-tanzu/velero/issues/4383
https://github.com/vmware-tanzu/velero/issues/3465
https://github.com/longhorn/longhorn/issues/2029

Would you mind providing the support bundle which includes the logs of CSI sidecars? Maybe it contains any clues about CSI external-snapshotter was triggered by Velero deleting backups.

@weizhe0422 here the support bundle. Thanks for any help.
Info

Start Backup: 2023-05-01 09:12:59 +0200 CEST
Complete Backup: 2023-05-01 09:14:29 +0200 CEST
Delete Backup: 2023-05-01 09:17:34 +0200 CEST

supportbundle_a3236774-99ca-4ab5-a2a5-74c925273bb4_2023-05-01T07-20-00Z.zip

Related to https://github.com/longhorn/longhorn/discussions/5797 ?

@tcoupin thanks but this is not a solution because if I use --snapshot-volumes=false then velero does not trigger a backup for the persistent volumes. So Velero only backups the manifests/YAML's.

I am also facing the same issue with v1.9.1
deleting velero backup is cleaning most of the resources related to corresponding backups like backups.longhorn.io, volumesnapshotcontent, volumesnapshot but snapshot.longhorn.io is still present in the system.
And backup started failing when number of snapshot objects increased> ~250.
Sharing both longhorn support bundle as well as snapshot controller logs
longhorn-support-bundle_8afefff1-085e-4f4e-97ae-3f0a518555ab_2023-05-09T05-19-35Z.zip
snapshot-controller.log

I do not use --snapshot-volumes=false but I add a cronjob who deletes the snapshot.longhorn.io refered by backups.longhorn.io.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: longhorn-system
  name: snapshot-cleaner
rules:
- apiGroups:
  - longhorn.io
  resources:
  - backups
  verbs:
  - 'list'
- apiGroups:
  - longhorn.io
  resources:
  - snapshots
  verbs:
  - 'list'
  - 'delete'
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: snapshot-cleaner
  namespace: longhorn-system
subjects:
- kind: ServiceAccount
  name: sa-snapshot-cleaner
  namespace: longhorn-system
roleRef:
  kind: Role
  name: snapshot-cleaner
  apiGroup: ""
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-snapshot-cleaner
  namespace: longhorn-system
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: snapshot-cleaner
  namespace: longhorn-system
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      activeDeadlineSeconds: 30
      template:
        metadata:
          creationTimestamp: null
        spec:
          affinity: {}
          containers:
            - args:
                - '-c'
                - >-
                  cat <(kubectl get backups.longhorn.io -n longhorn-system -o
                  custom-columns=SNAPSHOT:.spec.snapshotName | grep
                  '^snapshot-'|sort|uniq) <(kubectl get snapshot.longhorn.io -n
                  longhorn-system -o custom-columns=SNAPSHOT:.metadata.name |
                  grep '^snapshot-'|sort|uniq)|sort|uniq -c|awk '$1==2 {print
                  $2}'|grep -v '^\n$'|xargs kubectl delete snapshot.longhorn.io
                  -n longhorn-system --wait=false
              command:
                - /bin/bash
              image: bitnami/kubectl:latest
              imagePullPolicy: Always
              name: snapshot-cleaner
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: OnFailure
          schedulerName: default-scheduler
          securityContext: {}
          serviceAccount: sa-snapshot-cleaner
          serviceAccountName: sa-snapshot-cleaner
          terminationGracePeriodSeconds: 30
  schedule: '*/5 * * * *'
  successfulJobsHistoryLimit: 3
  suspend: false

When I create a backup via the Longhorn GUI and then delete this backup, no snapshots remain (everything works as expected).

I tried to reproduce the issue again by creating a VolumeSnapshot resource and afterwarts delete that and the issue occurs again.
VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: longhorn
  namespace: longhorn-system
  labels:
    velero.io/csi-volumesnapshot-class: 'true'
driver: driver.longhorn.io
deletionPolicy: Delete
parameters:
  type: bak

VolumeSnapshot
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: new-snapshot-test
  namespace: harbor
spec:
  volumeSnapshotClassName: longhorn
  source:
    persistentVolumeClaimName: harbor-jobservice

Remove the VolumeSnapshot: kubectl delete volumesnapshot new-snapshot-test -n harbor

Why does Longhorn remain the snapshot?
Why does it work in the GUI but not without it?

I still found something interesting in the snapshot-controller logs:

Today I upgrade Longhorn from v1.4.1 to v1.4.2 and the issue still occurs. 😔😔

Today I noticed that I have deployed a snapshot controller like described in this documentation.
Although I already had an snapshot controller "rke2-snapshot-controller" on my cluster. I am not sure if this is comes with a rancher update or something. Anyway I removed my snapshot-controller and test the issue again.
VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: longhorn
  namespace: longhorn-system
  labels:
    velero.io/csi-volumesnapshot-class: 'true'
driver: driver.longhorn.io
deletionPolicy: Delete
parameters:
  type: bak

VolumeSnapshot
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: new-snapshot-test
  namespace: harbor
spec:
  volumeSnapshotClassName: longhorn
  source:
    persistentVolumeClaimName: harbor-jobservice

Remove the VolumeSnapshot: kubectl delete volumesnapshot new-snapshot-test -n harbor

The same issue occurs but there is some interessting logs:

After I created the VolumeSnapshot the following logs are written:

@vineetsingh5 do you found a solution for this issue?

@weizhe0422 do you found something interessting in the support bundle?

I'm having almost the same setup and versions and the same issue!
One interesting log line found on longhorn-csi-plugin:
longhorn-csi-plugin-5k8lg longhorn-csi-plugin time="2023-10-06T08:12:20Z" level=info msg="DeleteSnapshot: req: {\"snapshot_id\":\"bak://pvc-c57da450-ce82-44c8-ac83-0a039634a334/backup-04db0d0fe4ef49f1\"}" longhorn-csi-plugin-5k8lg longhorn-csi-plugin time="2023-10-06T08:12:20Z" level=info msg="DeleteSnapshot: rsp: {}" csi-snapshotter-5d899fdcfc-xv627 csi-snapshotter E1006 08:12:20.143392       1 snapshot_controller_base.go:265] could not sync content "snapcontent-55c4399b-1dec-4cf2-b9bd-a4eff27f315e": snapshot controller failed to update snapcontent-55c4399b-1dec-4cf2-b9bd-a4eff27f315e on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-55c4399b-1dec-4cf2-b9bd-a4eff27f315e": StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshotcontents/snapcontent-55c4399b-1dec-4cf2-b9bd-a4eff27f315e, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d4fa9b1d-416e-4df5-ad74-d3ac6bec3b66, UID in object meta:

I update to Velero v1.12.2, using the velero-plugin-for-csi v0.6.2 & velero-plugin-for-aws v1.8.2, but still the same issue.

Delete the backup velero backup delete
The snapshot (snapshots.longhorn.io) is not deleted.

I believe what Velero triggered is to delete the corresponding Longhorn backup (out-of-cluster) instead of the Longhorn snapshot (in-cluster, immutable COW layers for volume).
In the current design, deleting a backup is independent of deleting the corresponding snapshot generated by that backup. What is your expectation here? You can check https://longhorn.io/docs/1.5.3/snapshots-and-backups/scheduling-backups-and-snapshots/ works for you but this is our built-in mechanism nothing related to Velero.
@mantissahz please follow up.

@innobead thanks for your reply.
What I want: I have a Velero schedule that creates/triggers backup of my persistent volumes with a retention period of e.g. 7 days. After this retention period 7 days Velero deletes these backups, but the corresponding snapshots are not deleted and consumes disk space that I don't want. As a workaround, I have a recurring job that deletes these snapshots (retain 7), but there are two disadvantages.

I'm using up disk space for snapshots I don't want and that are stored in my object store already.
For example, if I trigger 3 manual backups with Velero, the recurring job doesn't delete the snapshots based on the creation timestamp like Velero does. This means that I lose backup data that is older than 4 days.

@R-Studio @innobead
I think this issue can be simplified to completely exclude Velero.
At the core the issue here is that Longhorn does not delete snapshots or backups when the backing CSI VolumeSnapshot resource is deleted.
As a user of Longhorn that is interfacing with CSI and not native Longhorn resources, I expect the state of Longhorn resources to reflect the state of my CSI resources.

If I create a CSI VolumeSnapshot I expect Longhorn to create a snapshot/backup/bi. This works!
If I delete a CSI VolumeSnapshot I expect Longhorn to delete the backing snapshot/backup/bi that it created. This doesn't work.

Therefore I think it's fair to state that Longhorn is currently only providing a partial implementation of the CSI interface/spec.
Velero is just using this common CSI interface as it is intended to be used and expecting it to have the desired effect. This is not a Velero issue.

Perhaps this should be opened as a new issue with a smaller scope (CSI spec conformance).

Thanks for the valuable info.
We will improve this, as it's quite important for space efficiency.

VolumeSnapshot type: snap
Both creation and deletion works

Create a CSI VolumeSnapshot => Longhorn creates a Snapshot
Delete the CSI VolumeSnapshot => Longhron deletes the Snapshot

Note that, Longhorn won't delete the latest snapshot which is just behind the volume-head and will only marks it as removed
Reference: https://longhorn.io/docs/1.6.2/concepts/#243-deleting-snapshots
VolumeSnapshot type: bak
Both creation and deletion works

Create a CSI VolumeSnapshot => Longhorn creates a Backup
Delete the CSI VolumeSnapshot => Longhron deletes the Backup

As mentioned by @innobead  above and @PhanLe1010  in another thread(https://github.com/vmware-tanzu/velero/issues/6179),
Longhorn first creates the Snapshot(Longhorn snapshot) and then creates the Backup(longhorn Backup) based on that snapshot.
After backup creation, the snpashot is no longer binding with the backup.
Thus, when users delete the CSI VolumeSnapshot, Longhron only deletes the backup(longhorn backup) but not snapshot(Longhorn Snapshot).
That is why we can see the snapshots(longhorn snapshot) remaining in this issue. And it is expected because a Longhorn snapshot can be corresponding to multiple Longhorn backups
There are two ways to auto delete these snapshots.

Setup a snapshot-delete recurring job to periodically delete the Longhron Snapshot.

In the following v1.7.0 release, we have new setting auto-cleanup-when-delete-backup which can auto clean up the snapshot when the backup is deleted. Reference: https://github.com/longhorn/longhorn-manager/pull/2783

VolumeSnapshot type: bi
Both creation and deletion works

Create a CSI VolumeSnapshot => Longhorn creates a Backingimage
Delete the CSI VolumeSnapshot => Longhron deletes the BackingImage

cc @R-Studio @julienvincent the csi volumesnapshot creation and deletion are implemented in Longhorn. I think we just have some confusion about Snapshot and Backup in Longhorn.
cc @larssb , yes, since Longhorn Backup and Longhorn Snapshot is not binding together after creation, you have to delete the Longhorn Snapshot manually or using a snapshot-delete recurring job

@ChanYiLin thanks for the great summary. The new setting auto-cleanup-when-delete-backup sounds really helpful and we will test it after Rancher releases the Helm chart for v1.7.0 (and close this issue if it works).

I am going to close the issue for now.
Feel free to open the issues if there is any new comments


[BUG]

PLEASE READ THE CONTRIBUTING GUIDELINES BEFORE YOU SUBMIT
Describe the bug
Attempting to edit an ITEM with the following condition causes the app to crash: ITEM has a picture attached to it from one's preexisting pictures, and the picture was deleted after the ITEM was created.
To Reproduce
Steps to reproduce the behavior:

Click on 'CREATE'
Fill in the fields
Click on 'SELECT PHOTO' at the bottom
Select the picture
Save item by clicking on 'SAVE ITEM'
Delete the chosen picture from phone
Return to HOT STUFF app and click on 'ITEMS'
Click to edit the item whose picture was deleted
App crashes

Expected behavior
Expected to be able to edit the item regardless of the picture being deleted or not.
Device (please complete the following information):

Device: Google Pixel 6
Android Version 14
Version 34

Additional context
Another condition was tested:
Choose 'TAKE PHOTO' as opposed to 'SELECT PHOTO'.
The ITEM can be edited despite the original photo being deleted. The app did not crash.
As for the picture that was originally attached to the ITEM, it was not there, there's just black space (for what it's worth, dark theme enabled in app).

Great find, I should have a hot fix ready for later this afternoon.

I tested again, along with going back to the items that were causing the app to crash, and everything is working fine.
I like the quick message telling the user the picture is missing due to location change. It could deter users to assume that the app deleted their picture.
The message disappears a bit quick though. Just my 2 cents.
I would say the bug is fixed, thank you

Fantastic! I'll make a note to lengthen that message time out. Appreciate the report.


Admin login

how to setup a admin role , couldnt find anything related to it

There is currently very little reason to create an Admin role. Its available in principle, but haven't added it to the normal flow because its kinda useless. Let me explain:

Creating an account essentially creates an environment where you can work in isolation to other things. Its great if you have multiple people that want to use it, for example.
And admin role is mostly about having full access to the database according to the Backend. Meaning: if you ask the backend for an entity id owned by another user as an Admin, you will receive it. The idea with this was to be able to do cross-user data analysis at some point, to enable cost tracking and shared API keys.

Now, unless you have several users and want to be able to track their usage, its not really all that useful to create an Admin user at the moment. I literally haven't done it myself yet =) But its coming!
With all that said, its an issue that this noise (useless admin role) exists, and that's my fault. Thanks for the question!


Unable to create automation from Tapo T110 Contact Sensor

Describe The Bug
There is no option in HomeKit to create an automation based on the state of the contact sensor. When the accessory is selected for automation, only the power state on / off is available for selection.
I think this might be fixed if the contact sensor is created as a separate accessory instead of a characteristic of the same accessory (that already has a power state)
Expected behavior
Able to use the contact sensor state for HomeKit automations
Logs

Plugin Config
    {
        "addresses": [
            "<IP_ADDRESS>"
        ],
        "name": "TPLink Tapo Platform",
        "email"<EMAIL_ADDRESS>        "password": "<REDACTED>",
        "platform": "HomebridgeTPLinkTapo",
        "_bridge": {
            "username": "11:22:33:44:55:66",
            "port": 12345
        }
    }

Plugin Version
v4.0.0
Homebridge Version
v1.6.1
Node.js Version
v18.17.0
NPM Version
v9.6.7
Operating System
Raspbian GNU/Linux 11

Hey, there is possible, i do have automation when i open the door at night to turn on the lights, you need to select to create an automation when a sensors detects something

Yes, that’s the category I selected. And when I select the plug, here is the screen I get:

As you can see from the above, the home app won’t let me choose the contact sensor state, it only allows me to choose the power state of the plug.

I see, so you want to automate the contact sensor from the plug itself, in that case the ticket tile is wrong as you specified T110

Ah, my bad. I meant P110. I’m not sure how I can fix the title. Should I close this and file a new ticket?

Ah my bad. I’ve fixed the title. I meant to say P110

No need, i will update it


HTTP: Fix the handle function example

When trying out the new (amazing) HTTP interface of Deno, I found a mistake in the documentation:
The example of the handle function does not work, as URL provides the path via the pathname attribute, not path.
MDN docs on the topic
Thanks for your amazing work on the manual 🙇

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Errata

Description
"Iterative Generaiton"  should be "Iterative Generation"
(Optional:) Please add any files, screenshots, or other information here.
No response
(Required) What is this issue most closely related to? Select one.
Choose One
Internal issue ID
95570007-93cc-4b57-8d28-b5ee7ae083e0
Paper URL
https://arxiv.org/html/2401.10034v3
Browser
Firefox/131.0
Device Type
Desktop

This report matches the PDF.
Please contact the authors directly for content corrections to the article, this repository only tracks issues with the HTML format.


Reflecting Sybase ASE tables into MS SQL (Azure DB) - Syntax error in DDL

Hi,
Reposting this here, originally created a discussion at https://github.com/sqlalchemy/sqlalchemy/discussions/9243
I'm attempting to load metadata from Sybase ASE, and then run create_all() on an SQL Server connection, however the generated DDL is causing syntax errors.
It fails on the first column, because create default zero_dflt as 0 or a variation of it, based on the datatype, is being appended to each line.
Full column line for example: colname SMALLINT NOT NULL DEFAULT create default zero_dflt as 0,
Here is a gist containing the relevant code, the full DDL, and the full error message: https://gist.github.com/ZachMassia/6bae47330d21e9da490178ea6c8f4f46
In the meantime I'm attempting to move things over with SSMA, but it would be fun to have it all in one place, in Python.

It is not uncommon for people to try migrating table definitions from one DBMS to another using a technique similar to the one you describe. Unfortunately, it is also fairly common for such attempts to fail due to differences in the underlying implementations of the different DBMSs.
If you can use a proper migration tool then that would be the best approach.

Hi Gord,
Thanks for the reply.
I've managed to get the tables moved over with SSMA (https://learn.microsoft.com/en-us/sql/ssma/sybase/migrating-sybase-ase-databases-to-sql-server-azure-sql-db-sybasetosql?view=sql-server-ver16)
Would you recommend using the MS tools for the recurring ETL as well?

Would you recommend using the MS tools for the recurring ETL as well?

I have no experience with them, but they seem to be built specifically for your use case and appear to be pretty well-documented
https://learn.microsoft.com/en-us/sql/ssma/sybase/migrating-sybase-ase-data-into-sql-server-azure-sql-db-sybasetosql?view=sql-server-ver16
so if I was in your situation I would be inclined to give them a try them first.

Sounds good. Thanks!


tbow + dragon arrows projectile swap bug

Rune arrows work as expected:

Dragon arrows do not show any swap:

so... I can't afford to shoot dragon arrows (dark bow shoots 2, different thing). can you show me you shooting them in-game? rsn "Laiduf".

nvm someone else got me these values, I just forgot to add them.

added https://github.com/runelite/plugin-hub/pull/4247 to be released in 3.2.1

should be fixed in 3.2.1


Add package name to deprecations

As suggested by @nicolas-grekas in https://github.com/doctrine/DoctrineBundle/pull/951#discussion_r272850188 and @greg0ire in https://github.com/doctrine/mongodb-odm/pull/2005#discussion_r275119872, we should make sure to add the package name to all deprecations messages. Currently, some simply say "will be removed in 2.0", others use some variation of "ODM 2.0". To allow users to quickly pinpoint where those deprecations are coming from, we should always use the full package name, like "in doctrine/mongodb-odm 2.0".

I should have quoted my source for this idea: a discussion with @nicolas-grekas on the train to the EUC hackathon :)

It's a great idea. The way we have many deprecation notices set up right now, people might not realise what package is causing it without looking at the trace. They shouldn't have to do that.


Notarize plugin to prevent malware flag in macOS Catalina

Came here to report a similar issue to #3 and #9, but wanted to add that it would be wonderful if the plugin could be notarized by Apple, so it can prevent errors like this.

I agree; it is planned but not considered an emergency as there is a workaround (mentioned in issue #3).

No worries. Thanks for replying!


Throws error while creating/editing record with json fieldtype

I have a model which has a column named data with json datatype (the migration is included below).
Viewing the record works fine. Edit record screen also opens without any error.
After editing, on submit, it thros the error. On visiting new record path (i.e. http://localhost:9999/admin/action/new) it thorws the same error.
Error stack trace
Ruby version: ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
Rails version: Rails 5.1.4
Database: psql (PostgreSQL) 9.5.13
class CreateActions < ActiveRecord::Migration[5.1]
  def change
    create_table :actions do |t|
      t.references :activity, foreign_key: true
      t.references :user, foreign_key: true
      t.json :data

      t.timestamps
    end
  end
end

Dup of #2329.