[BUG] RKE1 Clusters taking hours to come back up after a migration via backup/restore operator

Rancher Server Setup

Rancher version: v2.6-head Commit ID: 240554b
Installation option (Docker install/Helm Chart): Helm

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE1

Proxy/Cert Details: byo-valid

Information about the Cluster

Kubernetes version: v1.24.10
Cluster Type (Local/Downstream): Downstream

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): EKS

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom): Admin

Describe the bug
When doing a migration of Rancher from one instance to another, the RKE1 clusters are taking significantly longer to come up than they have previously. Normally it would take around 5-10 minutes after the restore for the RKE1 cluster to be back up, but now it's taking hours to come up.
It does seem to vary as I've seen it come back up anywhere from 30 minutes after restore, to over 5 hours (probably longer than that, but 5 hours was the last check I did before going to bed)
To Reproduce

Create a v2.6-head Rancher instance (this will repro on any version it seems, but this version has taken the longest so far)
Go into the local cluster > Storage > Secrets
Create a new opaque secret with your S3 credentials (where the backup file will be stored)
Go to local > Apps > Rancher Backups
Install Rancher Backups with either v2.1.4 or v2.1.5-rc2 (if you've enabled pre-release versions on the Rancher instance)
Create a downstream EKS RKE1 cluster with k8s v1.24.10
Create a downstream EKS RKE2 cluster with k8s v1.24.10+rke2r1
Once the clusters are provisioned, go to local > Rancher Backups > Backups and create a backup
Begin the migration process (noted here)
Once migrated, wait for the clusters to come back up

Result
It takes a very long time for the RKE1 cluster to come back up
Expected Result
It takes 10 minutes or less for the RKE1 cluster to come back up
Screenshots
Cluster Management Page - Five Hours and Cluster Is Not Up

Additional context
Quite possibly related to https://github.com/rancher/rancher/issues/40080 but making this ticket in case it's a separate issue

Currently waiting on @nickwsuse to provide an environment where this has occurred so I can debug it. Otherwise, when I free up, I can attempt to reproduce it on my own.

/forwardport 2023-Q2-v2.7x

@nickwsuse (cc @ronhorton @sowmyav27 ), moved this one to "To Test" to see if it's in fact reproducible - I don't believe it was reproduced lately and could be related to testing steps where the same backup is reused for multiple restores.

I've not encountered this issue at all in the last three weeks or so. I think it's safe to say it's no longer an issue so I'll close the ticket.


Adaptive bitrate streaming transformation not working.

final transformedUrl = cloudinary
.video('DEV_ASSETS/20230518_102910')
.transformation(
Transformation().transcode(Transcode.streamingProfile("auto")))
.toString();
transcode is not defined inside Transformation. and the same goes for Transcode.  It's working fine in the case of images and video.

Hey there, can you remove the toString() and try it again?

@ravivr-dev So that doesn't look flutter related but rather there is no predefined streaming profile called auto. When you upload a video we automatically generate a URL with sp_auto transformation per https://cloudinary.com/documentation/adaptive_bitrate_streaming#automatic_streaming_profile_selection and the chunks get generated on-the-fly.
If you want to use a pre-defined profile to pre-generate a strem then please pick one from https://cloudinary.com/documentation/adaptive_bitrate_streaming#predefined_streaming_profiles
let me know if you have any other questions.

@ravivr-dev it looks like we need to add support for auto profile so what you can do now is pass raw transformations via the generic() action group per https://cloudinary.com/documentation/flutter_media_transformations#passing_transformations_as_strings
here, you would use the URL format of the transformations instead of their full name so in this case it would be something like generic('sp_auto')
let me know if that works?

Closing the issue, feel free to use the VideoEdit class for ABR


question: does it maintained?

I forked the modification of ricardovcorrea,
because this library seems to be no longer maintained?
add features:

[x] TS support by .d.ts
[x] longPress event support
[x] fix the delay when close the Modal
[x] function components and hooks
[x] useNativeDriver trouble fixed
[x] use TouchableOpacity
[x] giving the X close button more space from the borders
[x] ...

Maybe we should maintain it, this is indeed a great library.
Thx~

Link to the fork?

Link to the fork?

here 😄


Spectators are being kicked

In 1.19.3 there might be some kind of spectator bug where players are getting kicked for spectating recieving the following error in console ReadTimeoutException : null and there is not much to find on Google about this.

Just giving myself the 50% credit I deserve 🤣


New attempt: map to NOMAD basesections

EDIT: this is the branch that we want to work on going forward
Made a new branch so that we don't loose too much on the other branch.

NXobject does not inherit any BaseSection (and thus does not have normalization)
We don't create any new classes for the normalization any more, but only add normalize functions (using super(self.__class__, self)). This should work since we now don't run into any recursion problems anymore.
All non NX-object NeXus classes inherit from one of the existing base sections once again

Closed in favor of #427


feat: add PyFuture to await Python awaitables

Relates to #1632
Draft based on #3610

@wyfo I'm sorry for the painfully slow review here.
Given that allow_threads is currently under intense scrutiny and we haven't yet decided on a concrete solution, #3610 risks getting stuck for a bit while we figure out a way forward for that API.
Does this patch rely on #3610 getting merged, or is it possible to rebase this on main? That would allow us to move forward with some of these remaining async PRs.

There is one thing bothering me with this implementation: the name PyFuture. In fact, there is Future object in Python, and it doesn't match the underlying type of PyFuture.
PyFuture is the result of calling __await__, so it's an iterator, but that's all we know, and I don't know any naming convention for this object – PEP 492 doesn't give it a name.
I've chosen PyFuture because it sounds like Rust Future, but that's maybe not a good reason. If you have some name suggestion, I'm interested.

The error in CI (other than non_local_definition lint) is due to the bug mentioned in #4055.

As written above, the name PyFuture was not a good name (the Python object doesn't even have a name, see this discussion), so I dropped it here to reuse it in #4057  where it's more suited.
Instead, I chose the more explicit name await_in_coroutine, making it easier to document, and for the user to understand that it must be used in coroutine context.


Buyme Coffe link in README is not working

I'm really grateful for this massive helper! Keep on doing good work and fix the link please, so you can enjoy some coffee too!

@danvick , the URL you currently have "https://www.buymeacoffee.com/wb5M9y2Sz" shows a page not found.
You may want to update it, here is the help page to do that https://help.buymeacoffee.com/en/articles/5885219-how-to-change-your-username-page-url
Great work on Flutter Form Builder, thanks

Thanks for letting me know. I haven't visited the site in a while and it seems things. Some new requirements were added and needed updating - that's why my link was down.
I'll be updating the link as I merge the latest PRs and release a new version of the package(s)


Download for MacOs its the wrong file

Its downloading an AMD64 exec file

The download is Darwin-amd64 which is the correct OS-ARCH.


Chat Folder renaming broken on Firefox

Bug Report
Installation Method
Docker
Environment

Open WebUI Version: v0.3.33

Operating System: NixOS unstable

Browser (if applicable): Librewolf 131.0.3

Confirmation:

[x] I have read and followed all the instructions provided in the README.md.
[x] I am on the latest version of both Open WebUI and Ollama.
[ ] I have included the browser console logs.
[ ] I have included the Docker container logs.
[x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:
Chat folder name changes
Actual Behavior:
Chat folder name remains the same
Description
Bug Summary:
Chat folder renaming is broken on FIrefox
Reproduction Details
Steps to Reproduce:
Open open-webui on firefox. Double click to rename or use the menu. Press enter after renaming.
Logs and Screenshots
Browser Console Logs:
N/A. There is the suprious "Edit" console log when double clicking (that should probably get cleaned up) but no logs related to the failure nor any related network traffic.
Docker Container Logs:
N/A
Screenshots/Screen Recordings (if applicable):
N/A
Note
If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!

Duplicate #6391

That's a discussion item. This is a bug report of an issue that repros on firefox. That discussion just contains information confirming that it is a firefox related problem.
Please leave active bugs open so that people can find them.

@PretzelVector We need your help!

I don't understand but not my project. You do you.


chore(): [release-1.3] Update OWNERS - add kim-tsao

Description

Which issue(s) does this PR fix or relate to

Fixes #issue_number

PR acceptance criteria

[ ] Tests
[ ] Documentation
[ ] If the bundle manifests have been updated, make sure to review the rhdh-operator.clusterserviceversion.yaml file accordingly

How to test changes / Special notes to the reviewer

/assign @kim-tsao

/assign @nickboldt


Minor documentation improvement

refs #85
Please provide enough information so that others can review your pull request:
Improves documentation about, "how to use pre-commit" locally. Also informs that users with permission can create PR directly against this repository.

Once #84 is merged conflicts will be solved.


build, test: support go tip at travis

This fixes travis testing fail when we test etcd with Go tip
(https://travis-ci.org/coreos/etcd/jobs/94755325). Note that
test at Go tip won't fail the whole test, only returning warning
messages.

Will see if it still breaks in travis.

Failed again. https://travis-ci.org/coreos/etcd/jobs/94761563#L264
Will fix. Something related to -race flag.

trying with VM. Not container in travis.

Even in my local machine this fails:
./godev test -timeout 3m -cover   github.com/coreos/etcd/client github.com/coreos/etcd/discovery github.com/coreos/etcd/error github.com/coreos/etcd/etcdctl/command github.com/coreos/etcd/etcdmain github.com/coreos/etcd/etcdserver github.com/coreos/etcd/etcdserver/auth github.com/coreos/etcd/etcdserver/etcdhttp github.com/coreos/etcd/etcdserver/etcdhttp/httptypes github.com/coreos/etcd/pkg/fileutil github.com/coreos/etcd/pkg/flags github.com/coreos/etcd/pkg/idutil github.com/coreos/etcd/pkg/ioutil github.com/coreos/etcd/pkg/netutil github.com/coreos/etcd/pkg/osutil github.com/coreos/etcd/pkg/pbutil github.com/coreos/etcd/pkg/types github.com/coreos/etcd/pkg/transport github.com/coreos/etcd/pkg/wait github.com/coreos/etcd/proxy github.com/coreos/etcd/raft github.com/coreos/etcd/snap github.com/coreos/etcd/storage github.com/coreos/etcd/storage/backend github.com/coreos/etcd/store github.com/coreos/etcd/version github.com/coreos/etcd/wal --race -cpu 1,2,4

But this succeeds:
./godev test -timeout 3m -cover --race -cpu 1,2,4  github.com/coreos/etcd/client github.com/coreos/etcd/discovery github.com/coreos/etcd/error github.com/coreos/etcd/etcdctl/command github.com/coreos/etcd/etcdmain github.com/coreos/etcd/etcdserver github.com/coreos/etcd/etcdserver/auth github.com/coreos/etcd/etcdserver/etcdhttp github.com/coreos/etcd/etcdserver/etcdhttp/httptypes github.com/coreos/etcd/pkg/fileutil github.com/coreos/etcd/pkg/flags github.com/coreos/etcd/pkg/idutil github.com/coreos/etcd/pkg/ioutil github.com/coreos/etcd/pkg/netutil github.com/coreos/etcd/pkg/osutil github.com/coreos/etcd/pkg/pbutil github.com/coreos/etcd/pkg/types github.com/coreos/etcd/pkg/transport github.com/coreos/etcd/pkg/wait github.com/coreos/etcd/proxy github.com/coreos/etcd/raft github.com/coreos/etcd/snap github.com/coreos/etcd/storage github.com/coreos/etcd/storage/backend github.com/coreos/etcd/store github.com/coreos/etcd/version github.com/coreos/etcd/wal

I posted a question at golang-nuts (https://groups.google.com/forum/#!topic/golang-nuts/MT4_KDSHLNc). We still want to pass the test even with this issue. I will investigate more.

./godev test -timeout 3m -cover github.com/coreos/etcd/client -race -cpu 1,2,4; also fails. :(

playing with Travis CL, to make sure it works. Please don't mind me making some noise in build history.

https://golang.org/pkg/flag/ says
Flag parsing stops just before the first non-flag argument ("-" is a non-flag argument) or after the terminator "--".

That's why integration test was not running, I think.

Changes:

Position flags at the beginning, arguments at the end reference https://golang.org/pkg/flag
integration test was not running, now running https://travis-ci.org/coreos/etcd/jobs/94804195#L300
Test with go tip does not fail but only prints out warning message, because go master branch might not be stable but we still want to see if it breaks https://travis-ci.org/coreos/etcd/jobs/94804195#L594

Please take a look again.
/cc @xiang90 @jonboulle

integration test was not running, now running

We do not want to run integration test on travis. We only run it on semaphoreci . Travis is too slow.

@gyuho The other two ideas sound good to me. However the bash code looks messy. @jonboulle probably can give you some suggestion.

@xiang90 got it. I will disable integration test. Yeah my bash is bad :(

Just commented out integration test.

@xiang90 Just deleted whole lines on integration test. Thanks!

@gyuho OK. I found a simpler solution for you. Check https://docs.travis-ci.com/user/customizing-the-build/#Skipping-a-build and search Rows that are Allowed to Fail . Then we can remove all bash hack for tip.

@xiang90 Awesome. I will change using that. Thanks,

Closing this in favor of https://github.com/coreos/etcd/pull/3957.


Merge release-2.30 into main

DO NOT SQUASH

Tests failing for react, I will create another branch out of release branch and open another PR with synced main

https://github.com/prometheus/prometheus/pull/9457


Links bug on iPhone 3.x

I've noticed today that links on JQTouch have a little bug.
http://jqtouch.com/preview/demos/main/#home
When I click on "User Interface" for example, the page loads, then a screen of the previous page seems to appears for a very short time, then disappear.
I've noticed this bug only on iphone with OS 3.x ( no problem on iOS4 ).

Should be fixed-


Requests

just wanted to make the library and all loaded when home is loaded
Better thumbnails hd if possible

Hide Chinese shows
not necessarily needed but auto open on downloads when no wifi

Hi, the scheduled Anime episode is not showing pls help


token pre-sale strategy / financial model

As a product owner
to launch a succesful ICO
I need to prepare a financial model and a token pre-sale strategy.

not a concern of the open-source library


add a maintainer?

seems like there's a bit of a backlog with the PRs. Can we get another person with commit prove to help review and merge them?
Ideally, if we could have a GitHub org for this project, then a maintainer other than @clvv would be able to add new maintainers. This has worked out great for http://github.com/sous-chefs!

I'd be willing to help a bit with the maintenance.

I sent @clvv an email. Let's see if he responds.
If nothing comes from that, perhaps @PythonNut or @josephholsten could set up an org independently?

It's kind of hard to tell what's going on because there's been so many forks. I think infokiller/fasd may be the one that has diverged the most from this version.

@marlonrichert, @clvv has been inactive since 2016 (hope he's still alive), so probably you'll never get a response back from him.
As for the GitHub org, I'm not sure if it makes sense to create a whole org just for one simple repo. Maybe it's better to just create a new fork and add collaborators to it.

It's kind of hard to tell what's going on because there's been so many forks. I think infokiller/fasd may be the one that has diverged the most from this version.

@infokiller, would you be willing to make your fork the "official" one or set up a new GitHub org for fasd?

@clvv has been inactive since 2016 (hope he's still alive), so probably you'll never get a response back from him.

He's alive and well and published a cryptography paper in 2019: https://wdai.us

I'm happy to make my fork official or set up an org. My fork is indeed maintained, and I made some improvements, like bugfixes, shellcheck compliance, new options, better debugging (this was a big paint point), and other stuff I probably don't remember. I also have code in my dotfiles that integrates it with fzf, which I believe is better than the built in matching it had (faster, more accurate, more flexible, more maintainable, etc.), and I can add it to this repo.
That said, there's cons in me doing this:

I only tested my previous changes in bash 4+ and zsh. I probably need to clean up the code to handle some bash 4+ features (the original code is supposed to be POSIX compliant). I'm willing to do it if it's not much work, but if it will make maintenance much harder I will probably lose interest in the project.
I don't use MacOS, so I can't test that configuration.

@infokiller I'm using fzf-tab plugin, so it replaces the default zsh TAB behaviour to always use fzf. Do you think your fasd fork will present incompatibility with it?

I'm willing to do it if it's not much work, but if it will make maintenance much harder I will probably lose interest in the project.

I think that alone already presents a good argument for making it an org.

@FreddieOliveira no it shouldn't cause any issues: it only defines functions that integrate them, and completion will be opt-in
But also note that the fzf changes are not in the fork right now (they're in my dotfiles repo).

Ok, I've grown tired of waiting, so I created an org @shell-fishes and added you all to it as owners. It's the worst possible name I could come up with, so please consider coming up with a less terrible name.
If you would like to set the scope for the org as wider than just maintaining fasd, that would be cool too. I have created two orgs like this in the past: @sous-chefs & @terra-farm. They have patterns about codes of conduct &c that we could adopt, or not. Discussion about the org can go in https://github.com/shell-fishes/meta/issues

OK, nice initiative! I accepted the invitation. Now we should decide which version/fork of fasd will be our initial base.

It's good to see continued interest on this project. I apologize for ghosting this project for so long.
I really have no intention of making any further changes to fasd. It is by all intents and purposes a completed project. I personally view it (and use it) similar to things like cd, grep or sed. You don't want to find out your grep is behaving differently because of some new update.
I have always been using my local copy of fasd, and that is probably what I would recommend for anyone using fasd. In retrospect, I should not have submitted this to any package managers. And at the current stage (I know there are bugs, and probably some terrible vulnerabilities), it really shouldn't be offered on any package manager and should only be used at users' own discretion.
That being said, I think the best path forward is to have community-based forks. I will update the README to indicate the inactive status of this repository, as well as giving pointers to forks.

@clvv I think the biggest question is: do you mind us using the name fasd, or should we change it for our fork?
By your analogy to cd, grep or sed, it sounds like you're supporting differing and incompatible implementations that share the same command name. But I know we all want to respect your work and don't want to confuse users.

@infokiller could you hit up @whjvenyl and see if we could get him/her involved as well?
Been using this fork for ages, and can see you've pulled some changes from there as well.

Sure they're welcome to join and I'll compare the forks

There's continued discussion in this issue: https://github.com/shell-fishes/meta/issues/2

@infokiller @laur89 @benmccann  sorry for the late reply. 🙈 i haven't updated those configs in ages and in the meantime kinda drifted more into the field of doing ux related work.

@whjvenyl I'm afraid I don't understand your response as you don't seem to have posted anything in either of these issues until now and I don't know what configs you're talking about

@infokiller I'm a bit confused that the install instructions on your repo point to @whjvenyl's fork

@benmccann I probably just merged a commit/PR that did that change. I think I'm the only user of my fork so I didn't bother much about docs, and as I said in https://github.com/shell-fishes/meta/issues/2 I think other alternatives may be better for the future


modify others text for better reading

This is actually a question.
Perhaps Earliest/Latest/End shouldn't use String() for a time.Time. It gets noisy when you have a timezone (because of Monotonic):

Perhaps we can use time.RFC3339Nano?

Or some middleground, without the T:

Or without millis/nanos:

Or without millis/nanos and aligned:

Or with millis/nanos, aligned, without monotonic:

And (I know I'm talking mostly about date but...) with everything aligned:

We can also remove the timezone. The user probably knows the timezone they are in? (unless they are running in a server...)

@brenol Thanks for the suggestion, this is actually the problem that I've put off in the long-term.
The pattern without millis/nanos looks good. And I also think every field should be aligned.

It would be great if you work on it and open a PR ;)


Recursive compontent that emits cannot be created

Version
3.0.7
Reproduction link
https://codesandbox.io/s/recursive-emit-bug-10073
Steps to reproduce
See codesandbox for illustration of the problem

Create a recursive component that has an 'emits: ["event"]'
Emit a custom event $emit("event", depth) from recursive component
Add v-bind="$attrs" to every recursive inclusion from recursive component to enable emitting from any level
Finally use the recursive component with an @event="..." handler

What is expected?
The @event="..." is called for any "depth"
What is actually happening?
Only the first (top level) of the recursive component is handled

I spent some hours debugging this issue. According to documents we should add emits: ["<name>"] to our components. However, by doing so, we capture the event listener and effectively remove it from $attrs at the first "level".
The system behaves according to specifications, but it feels wrong to have to leave out emits: ["<name>"] when you build an emitting recursive component. At least for me it was very counter-intuitive.
Sadly I do not have a good solution as I understand why you capture any defined props and emits from the $attrs. However, maybe I am just not doing it right, would love to hear how we should create an emitting recursive component.
P.S. I found this problem when I was creating a file explorer component using a single recursive component.

See https://codesandbox.io/s/recursive-emit-bug-forked-weqfy?file=/src/components/RecursiveEvent.vue

Thanks for the information! I thought that solution (re-emitting event @event="(depth) => $emit('event', depth)") was suboptimal because of call stack depth and performance reasons. For posterity: I was wrong on that front https://webkit.org/blog/6240/ecmascript-6-proper-tail-calls-in-webkit/
TL;DR; If you use ES6 and have strict mode on, Tail Call Optimization is used. Essentially it changes a returning function call at the end of a function into a jump.

Yeah, I did also aware of the problem. I think I should reopen this issue for further discussion.

Hi,
so as summary there is two way for recursive nesting component:
1. Emits from Template and not have defineEmits on the script
Template of recursive nested component
Where the event is generated:
@click.stop="$emit('start', element)"
Where the children that is nested:
v-bind="$attrs"
Template of grand parent
@start='dragStart'
2. Emits normally from script but have a recieve/sender of the event in each node
Template  of recursive nested component
Where the event is generated:
@click.stop="handleClick($event, element.node)"
Where the children that is nested
@start="(node) => $emit('start', node)"
Script
const emit = defineEmits(['start'])
function handleClick(event:any,node:Node) { emit('start', {event, node}); }
Template of grand parent
@start='dragStart'
is this correct?
can you give me a suggest of which is the best way?


PVC Stuck In Attaching

I have multiple PVCs that will not get past attaching. Sometimes, after waiting hours, it will attach, other times it will never attach. If I scale back, the PVC reports detached (eventually). Then scaling back up, I get back into the same place.
The current support bundle has a pod named sonarr-4k trying to attach, similar issues on multiple other PVCs.
Edit: Forgot to attach support bundle
longhorn-support-bundle_0975d9e2-4f2c-4f48-bf4f-29c3beb40b39_2020-12-15T20-13-45Z.zip

Version: v1.0.2

The root cause is the deadlock issue #1599 we fixed in the upcoming release v1.1.0
The log of longhorn manager longhorn-manager-tvxx6:
2020-12-14T17:59:05.123529455-08:00 E1215 01:59:05.122945       1 engine_controller.go:209] fail to sync engine for longhorn-system/config-sonarr-4k-e-cad7f07d: failed to get process config-sonarr-4k-e-cad7f07d: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2020-12-14T17:59:05.124031655-08:00 time="2020-12-15T01:59:05Z" level=warning msg="Dropping Longhorn engine longhorn-system/config-sonarr-4k-e-cad7f07d out of the queue: fail to sync engine for longhorn-system/config-sonarr-4k-e-cad7f07d: failed to get process config-sonarr-4k-e-cad7f07d: rpc error: code = DeadlineExceeded desc = context deadline exceeded"

The log of instance engine manager instance-manager-e-70acdbd1:
2020-12-14T17:55:58.756105573-08:00 time="2020-12-15T01:55:58Z" level=info msg="Storing process logs at path: /var/log/instances"
2020-12-14T17:55:58.756133944-08:00 [longhorn-instance-manager] time="2020-12-15T01:55:58Z" level=info msg="Instance Manager listening to <IP_ADDRESS>:8500"
2020-12-14T17:55:58.762144787-08:00 tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
2020-12-14T17:55:58.762159556-08:00 tgtd: work_timer_start(146) use timer_fd based scheduler
2020-12-14T17:55:58.762164122-08:00 tgtd: bs_init(387) use signalfd notification
2020-12-14T17:56:00.367878956-08:00 [longhorn-instance-manager] time="2020-12-15T01:56:00Z" level=debug msg="started new process manager update watch"
2020-12-14T17:56:04.989463529-08:00 [longhorn-instance-manager] time="2020-12-15T01:56:04Z" level=info msg="Process Manager: prepare to create process config-sonarr-4k-e-cad7f07d"
2020-12-14T17:56:04.989545445-08:00 [longhorn-instance-manager] time="2020-12-15T01:56:04Z" level=info msg="Process Manager: created process config-sonarr-4k-e-cad7f07d"
2020-12-14T18:00:07.397061548-08:00 [longhorn-instance-manager] time="2020-12-15T02:00:07Z" level=info msg="Process Manager: prepare to create process config-plex-auto-collection-e-196d8c85"

(Get stuck due to the deadlock issue)

Workaround:

Directly killing the pod instance-manager-e-70acdbd1. It's better to upgrade to v1.1.0 before applying the workaround so that you won't hit this bug again.
Warning: This workaround will lead to all volumes attached to the node will get reattached. Please make sure the workloads on the node is not using the volumes.

@shuo-wu I'm also experiencing this issue even after upgrading to v.1.1.100.
These are a few notes of what I've been observing:

Every time I reboot a node, there are always a small number of nodes that get in some kind of phantom stage:

some may appear as detached to Longhorn (as per UI) but can't be reattached again because kubernetes complains that they are already attached to some node. It's similar to https://github.com/longhorn/longhorn/issues/1370, yet they never recover and the errors are in a loop nonstop
the pvc stays in "Attaching" mode permanently like reported in this issue

I don't know if the former are two separate issues, but I've just spent some time trying to get to the bottom of this. I've managed to reproduce the second problem, namely, pvc stuck in Attaching state.  I guess this behaviour is similar to restarting a cluster node. In your comment you mention Please make sure the workloads on the node is not using the volumes.. However, this is not possible in the scenario I'm trying to debug: when I have to reboot a node, kubernetes upgrades, etc.

I've managed a workflow to get out of this:

first I try to kill the pod attached to the pvc. Some of the failed pvc would get reattached fine
however, this doesn't always work. As an extreme measure I have to delete the workload entirely, wait for the volume to become detached, and then recreate the workload again

I'm not sure which logs would be useful for this, but happy to try anything as this has caught me many times and it's not nice

@cortopy Thank you for reporting!

For the 1st issue, volume cannot be reattached, how do you trigger it? Does the volume become detached automatically? Issue #1370 typically happens when a volume is attached but the related pod is started on another node. If the volume is detached but Multi-Attach error still gets triggered, I guess somehow the VolumeAttachment CR is not removed and the Longhorn volume detachment is not issued by Kubernetes. You can check it by kubectl -n <namespace> get volumeattachment. The workaround is to stop the related pod, check and make sure the VolumeAttachment is deleted, and then restart the pod.

For the 2nd issue, it seems that you can always reproduce it. Can you describe the stop in details (including your environment)? And which instance manager pods will you delete?

Since you mentioned both issues may be related to node reboot. Let me explain what will happen ideally after node reboot(node is down then back immediately). This may help you understand the root of the issues:
1. When the node is rebooted, Longhorn will mark the node unavailable immediately and delete the instance manager pods on the node. Then the volume on the node will become state unknown. Since the node is down, the pod deletion will get stuck in state Terminating.
2. Kubernetes won't do anything (pod eviction) here since the node will be back soon.
3. When the node is back, Longhorn and Kubernetes will detect the nod available simultaneously. Then instance manager pods cam be restarted successfully. The Longhorn volume will be detached then reattached immediately. Other pods on the node will become completed if there is no special config. If the workload of the volume is a pod of a deployment/statefulset, the pod will be restarted along with the volume reattachment.
Since the issues are complicated, you can directly generate a support bundle and send it to<EMAIL_ADDRESS>To help pinpoint the issues, it's better to provide the problematic volume name and the date the issue happened.

thank you @shuo-wu for looking into this!!
I'm not sure I have a reproducible series of steps, but I'll try to explain as best as I can.
Last week I had two big events that disturbed my k8s cluster:

Bump of kubernetes version
One particular node was flapping between readiness and non-readiness, mainly for two reasons: network problems and https://github.com/rancher/rancher/issues/31793

In regards to 1, I wouldn't know how to reproduce it as I can't downgrade the version. But I'll pay attention next time for sure
As for 2, the closest I can get to reproduce the errors is by either:

killing all the pods in the longhorn namespace. This is probably similar to a major disaster where the whole cluster goes down at once, but it helped me to repeat the process
I sometimes got similar errors by killing an instance manager pod, but killing all of them at once would bring the issues more often

Because I don't know how to reproduce for sure, this is likely to go into some kind of chaos testing paradigm.
In any case, I've just tried once again now with my cluster stable for a few days since the bad events. I have two volumes in permanent "Attaching" state. In these two cases it's a RWX volume. I can see the issue is the pods for the volumes not starting properly. They log waiting with nfs server start
I've also observed similar behaviour for RWO volumes, although I couldn't reproduce this time

Have you killed the Longhorn workloads again since 2 volumes getting stuck in "Attaching" state? If No, the related logs may be still there. Then you can generate the support bundle via Longhorn UI.
Previously we encountered similar issues triggered by RWX volumes. It seems the problem is not solved. We plan to refactor this part in this or next release then re-test this case.
https://github.com/longhorn/longhorn/issues/2015
BTW, is this related to the root cause found in #2629? @joshimoo

@shuo-wu I don't have the full context for this issue, but might be related to the volume not getting attached to the share-manager node, since user mention the below about RWX volumes.
In any case, I've just tried once again now with my cluster stable for a few days since the bad events. I have two volumes in permanent "Attaching" state. In these two cases it's a RWX volume. I can see the issue is the pods for the volumes not starting properly. They log waiting with nfs server start

This issue might also be related https://github.com/longhorn/longhorn/issues/2659

also getting stuck in attaching ... a statefulset pod in my case

Having this issue after an upgrade 1.4.0 -> 1.5.2 -> 1.6.2  -> 1.7.0.
At first I could scale down the workloads using the PVC's and then the stuck volumes would get "detached" so I could "do stuff". By now it's completely stuck in the "attaching" state with a spinning wheel wether or not the using workloads are there or not.
What I'm seeing in the csi-attacher logs is:
✗ k logs pod/csi-attacher-7987c8f7b9-pgls2  -n longhorn-system | grep pvc-cc294cfb-253d-4cbd-b84a-b4230db2b988
I0904 15:55:43.022777       1 csi_handler.go:243] "Error processing" VolumeAttachment="csi-7df1d7dbb3959b1c3dd7d32ad8722dac2e27ee46765e0dfa0e4b97a141255f1b" err="failed to attach: rpc error: code = DeadlineExceeded desc = volume pvc-cc294cfb-253d-4cbd-b84a-b4230db2b988 failed to attach to node pi4 with attachmentID csi-7df1d7dbb3959b1c3dd7d32ad8722dac2e27ee46765e0dfa0e4b97a141255f1b"

The volume replicase are at two other nodes, pi1, pi2:

while the workload is scheduled on pi4.


Gradle sync 报错

Gradle sync failed: Could not find any matches for im.fir:sdk:latest.integration as no versions of im.fir:sdk are available.
Required by:
android:app:unspecified
Consult IDE log for more details (Help | Show Log)

Sorry but I didn't meet this problem. You can try deleting this line and build again.


Provide json format for response when the server throw BadMessageException

This will make the response nicer (in json), instead of an xml response with input stack trace.

@AndrewJSchofield I'm deploying this to a test cluster and will include the response for a REST request, would that be acceptable?

@AndrewJSchofield I'm deploying this to a test cluster and will include the response for a REST request, would that be acceptable?

I do see that the effect of this might be to make kafka-rest behave better, so a test in that repo which shows the benefit of this work is acceptable.

@AndrewJSchofield thanks for pointing out about kafka-rest, I'm testing this with kafka-rest atm, and it turns out the error is raised at lower level than application, so using registerExceptionMappers doesn't work.
Will update this PR and raise another PR for kafka-rest once I figure out the correct error handler.

@AndrewJSchofield this PR should be ready to review now, I have added tests to validate that it should work (remove stack trace from error response).
Could you please have a look? thanks


add DispatchHandlerInterceptor

Codecov Report

Merging #4348 into master will decrease coverage by 0.01%.
The diff coverage is 5.4%.

@@            Coverage Diff             @@
##           master    #4348      +/-   ##
==========================================
- Coverage   41.02%   41.01%   -0.02%
==========================================
  Files        2417     2418       +1
  Lines       74649    74678      +29
  Branches    10117    10118       +1
==========================================
+ Hits        30625    30626       +1
- Misses      41206    41235      +29
+ Partials     2818     2817       -1

Impacted Files
Coverage Δ

...ercorp/pinpoint/flink/process/TBaseFlatMapper.java
58.33% <0%> (ø)
:arrow_up:

...collector/receiver/DispatchHandlerFactoryBean.java
0% <0%> (ø)
:arrow_up:

...ctor/receiver/EmptyDispatchHandlerInterceptor.java
0% <0%> (ø)

...avercorp/pinpoint/thrift/io/FlinkTBaseLocator.java
0% <0%> (ø)
:arrow_up:

...nt/collector/receiver/DelegateDispatchHandler.java
0% <0%> (ø)
:arrow_up:

...corp/pinpoint/io/request/DefaultServerRequest.java
44.44% <100%> (+6.94%)
:arrow_up:

...vercorp/pinpoint/web/vo/ApplicationAgentsList.java
72.3% <0%> (-3.08%)
:arrow_down:

...m/navercorp/pinpoint/test/OrderedSpanRecorder.java
72.11% <0%> (-1.93%)
:arrow_down:

...point/rpc/client/DefaultPinpointClientHandler.java
70.38% <0%> (-1.28%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf90dff...1d9d2d0. Read the comment docs.


fans that are Present may have no attribute "Functional"

On water cooled systems,  fan 1 is usually not installed.  When polling this fan, we see it has an attribute of "Present"=True.  However it does not have an attribute of "Functional".   This is WAD.
Need to update the Robot Framework fan test code that derermines the number of installed fans in the system to allow for this.  Do not count a fan unless it is both "Present" and "Functional" .

https://gerrit.openbmc-project.xyz/#/c/openbmc/openbmc-test-automation/+/12592/


Disallow to set the state to signup from the console

Hello !
I know that I can disactivate the "signUp" link but the user can still go to the console and type in the console  :
AccountsTemplates.setState('signUp');

Is there a way for fix this problem ?

I finally used :
AccountsTemplates.setState = function() {};

Well, with AccountsTemplates.configure({forbidClientAccountCreation: true}); you should be able to achieve what you're asking for...
But you're right: there's no filter on the setState method checking the forbidClientAccountCreation configuration options.
I'd say the following should work:
AT.prototype.setState = function(state, callback) {
  check(state, String);

  if (!this._isValidState(state) || (AccountsTemplates.options.forbidClientAccountCreation && state === 'signUp')) {
    throw new Meteor.Error(500, "Internal server error", "accounts-templates-core package got an invalid state value!");
  }

  this.state.form.set("state", state);
  if (!this.avoidClearError) {
    this.clearState();
  }
  this.avoidClearError = false;

  if (_.isFunction(callback)) {
    callback();
  }
};


Release with new Features?

Can you Release a new NUget package?

@MrHinsh Done https://www.nuget.org/packages/TfsUrlParser#versions-body-tab
Sorry for let you waiting and thank you for your contribution


apps/speed.c: generate evp_cipher keys implicitly

First patch enables 'implicit' EVP_CIPHER key generation via EVP_CIPHER_CTX_rand_key. I tested this with an engine which has its own key generation routine.
Second patch is to stop most of the valgrind complaining with speed tool.

Merged. Thanks.


Enhancement/profile view

This PR makes a general improvement to the profile view now called settings, with a restructuring of the UI into different components for better readability and scalability.
Merge once all methods are functional.

Summary by Sourcery
Refactor the profile view into a settings view, introducing new components for editing profile, changing password, and connecting a wallet. Update navigation and routing to reflect these changes.
New Features:

Introduce a new settings section with components for editing profile, changing password, and connecting a wallet.

Enhancements:

Refactor the profile view into a settings view with a more modular component structure for improved readability and scalability.

@sourcery-ai review


Token holders data

Draft PR awaiting feedback.
Probably need to store a bigger list of known accounts/programs. Not sure if there's a great source for that somewhere.
Should we format with percentages?
Any other stats on the top holders?
Should the limit be configurable?
Closes: #26

@slimeonmyhead possibly add percentage of total token holdings (if significant, >= .1%?)
i.e. 70,000,000 (1%)

solscan has address tagging/labeling, but not seeing that data in an API format. We could create an indexed table and fill it from various sources (scraping, manual imports)

Updated with owned % as the rank


Inbox zero emoji is not visible

Bug Report Dump (Auto-generated)

Version 1.17.0<PHONE_NUMBER>)
Device: iPhone X (iOS 11.2.1)
TestFlight: true

NVM. Emoji came after i restart the app


ENH: Add FluoroSimulator module

The "Virtual Cath Lab" module simulates fluoroscopy using a single-plane or bi-plane C-arm.
Motion for each component of the C-arm can be controlled using the module. As the C-arm is moved, the simulated image will be updated in real-time. This image is generated using volume rendering on a CT image using custom transfer functions.
Users can also simulate the addition of contrast agent in the image by filling in segmented regions of the with high intensity values.

This is probably a good place for a discussion as well before this is integrated. In the current publications we refer to the module as the "Virtual Cath Lab" module, instead of the "FluoroSimulator" module.
Do we want to rename the module to "Virtual Cath Lab"?

@Sunderlandkyl I've invited you to this repository with write access so that you can commit.
I like the Virtual Cath Lab module name. It better reflects that the module's scope is not just fluoro image simulation.

Please also add documentation (minimum add a few sentences about it to the main readme file but preferably also add a dedicated page with a few paragraphs of text and fewsl screenshots and reference to the paper.

@Sunderlandkyl @lassoan
I will miss the meeting today (pulled clinical), but catch up tomorrow.
We can decide if we are going to wait to release with the manuscript or just put it out there.
From the Cardiac Center and other funding agency perspectives (return on investment) it is important to publish and have something to show for the work we do which is why I try to follow the "publish and release" approach.  This is still far more than most do (publish and release nothing!), but happy to discuss in this case.
I do think it would be good to have this integrated so that it is easier to interface with other projects, but I do want to get credit for all the work we did.
Thank you.

@mattjolley Sorry, I thought that we had decided at the meeting last week to go ahead and make it public.
I'm OK with holding off for now if we want to wait for the publication.

@Sunderlandkyl
You are fine.  We just have a bunch of projects that have not been written up and this is one may to help motivate that.  But if we feel the benefits for releasing now outweigh that, we can release.  We can discuss next week.


Ipopt always finished with status Invalid_Option

Hello,
I ran the simulation launch, and there was a such warning:
started core service [/rosout]
process[controller-2]: started with pid [12511]
process[visualization-3]: started with pid [12512]
process[simulator-4]: started with pid [12513]
process[state_estimator-5]: started with pid [12514]
Python: 14.400000 m , 1441
Python: 14.400000 m , 1441
Starting LMPC node.

And the car does not move.
The program was executed with default parameters. Is there something wrong with the code or the parameter values? There are so many parameters that I don't know how to configure them to have a good result.
Anyone can give me an idea? Many thanks.

I am facing this issue. Have you been able to fix it?

I solved this issue by installing the HSL library. Follow the steps in the Ipopt installation guide. Don't forget to also follow the following steps concerning the setup.


[WIP] Force-terminate zombie instances

Goes with https://github.com/chanzuckerberg/idseq-infra/pull/227
TODO:

remove precondition: do terminate command regardless (will be no-op if unnecessary)
test if terminate command works if instance has finished terminating
shuffle instance order randomly in terminate command

Closing because the need for this change has not reoccurred and the change has the potential to lead to new problems down the line unless we invest in further investigation.


Models are showing for admins but not users

Installation Method
Installed using Docker
Environment

Open WebUI Version: v0.4.8
Ollama (if applicable): v0.5.1
Operating System: macOS Sonoma 14.1
Browser (if applicable): Firefox Developer 134.0b8

Confirmation:

[x] I have read and followed all the instructions provided in the README.md.
[x] I am on the latest version of both Open WebUI and Ollama.
[x] I have included the browser console logs.
[x] I have included the Docker container logs.
[x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:
Models should be visible to non-admin users as per the settings in the Admin Panel.
Actual Behavior:
No models are visible to non-admin users, even though all models are enabled in the Admin Panel settings.
Description
Bug Summary:
Models are visible for admin users but not for regular users. Previously, there was a "whitelist" feature to manage model visibility, but this feature appears to have been removed or is missing. As a result, non-admin users cannot access any models despite them being enabled.
Reproduction Details
Steps to Reproduce:

Log in as an admin and navigate to Admin Panel -> Settings -> Models.
Confirm that all models are enabled.
Log in as a non-admin user.
Check for available models in the user interface.

Observed Result: No models are displayed for non-admin users.
Logs and Screenshots
Browser Console Logs:
No apparent errors.
console-export-2024-12-10_13-50-34.txt
Docker Container Logs:
docker_logs.txt
Screenshots/Screen Recordings (if applicable):
For a regular user, no models found.

Additional Information
It seems the model management feature ("whitelist") has been removed or altered in the current version. This issue started occurring after updating to Open WebUI v0.4.8 and Ollama v0.5.1.

Go to Admin Panel -> Settings -> Models.
Edit each model's visibility to public.  (Just because it's green and enabled does NOT mean it's public.  You have to edit it and select Public from the dropdown.)
Scroll down and click "Save & Update".

It would be nice if the UI/UX were more intuitive here.  And maybe a toggle just like "enabled", so we don't have to go in and edit each model in order for it to be visible to our users.


no NPM package is problematic

When I was reading the docs, (https://hiukim.github.io/mind-ar-js-doc/face-tracking-quick-start/webpage), I noticed that integration is done using HTML <script> tags and that there is no NPM package.
The reason why I'm opening up an issue is because most web devs do not integrate using raw HTML <script>. This makes integrating this repo very difficult. In most modern js frameworks, you can't (or it's difficult to) put things in the top level HTML that requires javascript interactivity like this. This is because js frameworks such as React take over the scripting that would otherwise occur on the top level HTML file.
In my opinion, for this library to be viable for most js frameworks, there would need to an NPM library that gives the developer the ability to import the library and use it in their files.

I think the author would agree with you, except that he has had an NPM package for quite a while. Here is the link to the installation page - https://hiukim.github.io/mind-ar-js-doc/installation
The examples may show HTML examples, but that may just be the easier example to demo without knowing which framework you're building for.
Package : > npm i mind-ar --


feat(backup): multiple backup stores support

Support multiple backup stores to create, delete and update them.
Remove the default backup target instead using Default flag in BackupTarget CR spec.
Adding a flag ReadOnly in BackupTarget CR spec prevent backups from being created and storing on this remote backup target.
And provide HTTP endpoints to create, delete, update and list secrets.
Ref: longhorn/longhorn#5411

Is this ready for review?

Is this ready for review?

@innobead Yes, but LEP https://github.com/longhorn/longhorn/pull/6630 is still in review.

Hi @mantissahz , I have reviewed the PR, PTAL, thanks!

BTW, do we have upgrade plan for the new backup CRD changes
I think we need to modify the backup CRDs when user upgrades from previous version to this version
so the CRDs will have fields like backupTargetName,..., etc.
cc @mantissahz

BTW, do we have upgrade plan for the new backup CRD changes

@ChanYiLin, I have added the upgrade path parts for multiple backup targets but it still has one problem I need to fix. I will have an update soon.
Thanks for your review.

I think if we can make sure system backup and csi backup can support specifying backupTarget instead of using default
Then we can remove the whole default logic and prevent race condition of updating status
cc @innobead @derekbit @james-munson

@mantissahz ready for review?

@mantissahz ready for review?

@derekbit, Yes, I think it is ready for review.
And I will start to modify the test cases.

@mantissahz Need to make sure the routes of the backup resource creation without any problem, e.g.

Longhorn cluster and add an empty backup store
Longhorn cluster add non-empty backup store
DR volume

Please have a complete test plan for the feature.

@coderabbitai pause

There is a volume whose spec.backupTargetName is set to target-001. If an user removes the backup target target-001 and add target-002. Who will update volume.spec.backupTargetName?

Users need to update the spec.backupTargetName by themself.

For the system backup and restore, should we include none-default BackupTarget CRs?

Yes, I think we should include BackupTarget CRs.

Some tasks need to be examined and done:

[ ] When deleting a backup target assigned to the volumes, the field Spec.BackupTargetName and label backup-target of the volumes must be modified to the default.
[ ] The corresponding backup targets of DR volumes can not be deleted and URLs can not be modified.
[ ] System Backup test plans for saving and restoring the backup targets.
[ ] Clean up the backup target URL before deleting it while uninstalling Longhorn.

What will happen after a valid URL of a backup target to an invalud URL? Will the backup volumes and backups be cleaned up? Any difference before and after applying the feature.

The backup volume and backups will not be cleaned up if the backup target URL is modified from valid one to invalid one in current implementation and this PR because getInforFromBackupStore failed will directly return back and skip the synchronization for backup volumes, backup backing images and system backups.

What will happen after a valid URL of a backup target to an empty string? Will the backup volumes and backups be cleaned up? Any difference before and after applying the feature.

The same as the current implementation, in this PR the backup volumes, backups, backup backing images, and system backups will be cleaned up when the backup target URL is empty.

For a newly created volume, the v.spec.backupTargetName is always empty, so users need to change it later. We should support a parameter backupTargetName in the storageClass, so the v.spec.backupTargetName can be from the storageClass.

If the field v.spec.backupTargetName is not assigned during creation, it will be the default backup target name when mutating,
but YES we can introduce backupTargetName in the parameters of the StorageClass

but YES we can introduce backupTargetName in the parameters of the StorageClass

Then, need to implement it in the PR.

Hi @mantissahz , as we discussed, remember to update below settings title with default prefix

backup target -> Default backup target
Backup Target Credential Secret -> Default Backup Target Credential Secret
Backupstore Poll Interval -> Default Backupstore Poll Interval

cc @derekbit

as we discussed, remember to update below settings title with default prefix or update in setting description.

backup target -> Default backup target
Backup Target Credential Secret -> Default Backup Target Credential Secret
Backupstore Poll Interval -> Default Backupstore Poll Interval

@a110605  Modified. Could you take a look? Thanks

@a110605 Modified. Could you take a look? Thanks

Looks good. 👍

Need to update the descriptions of the two settings. They only serve for default backup store.
https://github.com/longhorn/longhorn-manager/blob/master/types/setting.go#L361-L377

Need to update the descriptions of the two settings. They only serve for default backup store.
https://github.com/longhorn/longhorn-manager/blob/master/types/setting.go#L361-L377

Have a new commit e6a5b4e to address it.


Angular Version 17

Hey, I'm on Angular 17 and the Fishbone is not working.
Do you plan any updates for future angular versions?
Thanks in advance!

There are plans to fix it and get it working. Or you can submit a PR too. Am just a little held up at the moment. So it will be a week at least before I can update. A few others have requested this too, btw :-)

please check same issue for me


tests, network: Unify TCP connectivity tests

What this PR does
Before this PR:
Currently there are 4 tests that verifies TCP connectivity in following ways:

pod to VM on the same node, over nodes network
pod to VM on different same node, over nodes network
pod to VM on different same node (over pod network)
pod to VM on different same node (over pod-network)

After this PR:
Unify (3) & (4) to single test, by running two pods, one on the same node as the VM and another on different node, and perform the connectivity check in the same test.
This PR change tests that perform connectivity check cross nodes to not skip silently but fail loudly, with a informative message.

Fixes #
Why we need it and why it was done in this way
The following tradeoffs were made:
The following alternatives were considered:
Links to places where the discussion took place:
Special notes for your reviewer
New package is introduced requirements to accommodate various checks to make sure the env meets a tests requirements.

Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

[ ] Design: A design document was considered and is present (link) or not required
[ ] PR: The PR description is expressive enough and will help future contributors
[ ] Code: Write code that humans can understand and Keep it simple
[ ] Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
[ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required
[ ] Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
[ ] Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
[ ] Community: Announcement to kubevirt-dev was considered

Release note

NONE

/test pull-kubevirt-e2e-k8s-1.31-sig-network

/uncc @kbidarkar @RamLavi

/test pull-kubevirt-e2e-k8s-1.31-sig-network

Changes: resolve  https://github.com/kubevirt/kubevirt/pull/13174#discussion_r1838500838 https://github.com/kubevirt/kubevirt/pull/13174#discussion_r1838494740 https://github.com/kubevirt/kubevirt/pull/13174#discussion_r1838488557

Changes: Removed the fist commit, existing skip remain as is.


Fix for test errors due to event_key non-uniqueness

Description & motivation
There is no known method of generating unique surrogate keys for events (without using window functions) given how GA4 writes duplicate events. Using window functions within stg_ga4__events would negate the date partition set on base_ga4__events and is undesirable.
This PR moves the window functions downstream and ensures that models that rely on event_keys being unique use a "pick 1" strategy when confronted with scenarios where there are multiple duplicate event keys to join.
This image demonstrates how including a row_number() window function causes BQ to bill for all records in the underlying base table (68MB of records in this case)

Checklist

[x ] I have verified that these changes work locally
[na ] I have updated the README.md (if applicable)
[x ] I have added tests & descriptions to my models (and macros if applicable)
[x ] I have run dbt test and python -m pytest . to validate exists tests

Good catch, I didn't think that through.
I finally have some clients on GA4 that I can test with. Looking at a relatively large site, there are plenty of examples of duplicate events.
One example: I have 13 click events with identical timestamp, event name, etc. Only thing that's different is engagement_time_msec. But even if I add that into the surrogate key, I get back 14k events that fail the uniqueness test.
So I'm at a loss. We could run a distinct on all non-nested fields to dedupe events as a service to the end-user. I have a slight concern that we could lose distinct information contained in the nested fields, though.
Suggestions?

@willbryant @dgitis I've dug into this quite a bit and have the following changes proposed in this PR. Description has been updated to reflect my thinking. Any concerns?
This PR fixes all defined unique tests, btw.

We discussed this months ago and rejected it, but maybe we look again at
deduplicating events with the same timestamp or even just page views. We'd
lose a tiny fraction of event tracking accuracy from legitimate duplicates,
gain accuracy from accidental duplicates, and solve a bunch of problems.
[image: Damon Gudaitis, Principal at Caret Juice Marketing]
Damon Gudaitis
Principal, Caret Juice Marketing
www.caretjuice.com
e: @.*** | p:<PHONE_NUMBER>
On Sat, Oct 22, 2022 at 10:46 AM Adam Ribaudo @.***>
wrote:

Good catch, I didn't think that through.
I finally have some clients on GA4 that I can test with. Looking at a
relatively large site, there are plenty of examples of duplicate events.
One example: I have 13 click events with identical timestamp, event name,
etc. Only thing that's different is engagement_time_msec. But even if I
add that into the surrogate key, I get back 14k events that fail the
uniqueness test.
So I'm at a loss. We could run a distinct on all non-nested fields to
dedupe events as a service to the end-user. I have a slight concern that we
could lose distinct information contained in the nested fields, though.
Suggestions?
—
Reply to this email directly, view it on GitHub
https://github.com/Velir/dbt-ga4/pull/74#issuecomment-1287867929, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ADM6FXGUKXM4H6QT3RJDDFDWEQR5VANCNFSM6AAAAAARJGNSA4
.
You are receiving this because your review was requested.Message ID:
@.***>

@dgitis The only way I can think to remove duplicates would be to group by all non-nested fields (everything except event_params and user_properties. But then you lose the 2 nest fields.
Can you think of another way to remove duplicates that doesn't involve window functions?

A suggestion came in from #measure Slack around including to_json_string(event_params) in the surrogate key and it worked for the 2 websites that I tested. Can anyone else test with their data?
Run the project and then test with dbt test -m stg_ga4__events

Would it be possible to do the de-duping on the way into the base events table, rather than the staging table reads?
This would have the advantage that, thanks to the incremental build process, we are only ever dealing with one source date table at once, so the partition pruning issue becomes moot.

@willbryant if you have a suggestion on how to handle de-duping on the way to the base events table, I'd love to hear it. I've given it some thought and have come up short. Were you able to confirm whether the event_keys are now unique?
The use case for partition pruning is being able to create incremental tables on top of stg_ga4__events. For example, in the future we'd like the session mart to be incremental which means it will have to filter on event_date_dt to find the latest session_start events. Just wanted to make sure that was clear.

I think my datasets must be much smaller than you guys. If I include the event params in the grouping criteria, then I don't see any duplicate events from my web stream, at least in the few source table dates I've spot-checked. I do see them very regularly from Firebase events though, which is interesting.

@adamribaudo-velir I think I probably just don't understand the issue, I haven't tried it. Is there something stopping us from doing a group by/distinct on the load from the raw events tables to the base table?
I thought it was the event params/user properties type arrays, but you've sort of solved that with the to_json_stringify solution.
What else have I missed 😂

I think there's nothing left to understand because it's solved now that it's clear that event_params can be used to dedupe :) The only reason this is occurring in the stg model and not base is due to DBT conventions stating that base models shouldn't modify the raw data aside from renaming and light conversions.

Ah, I see. I thought the current guidance was that sources shouldn't be messing with the data, not base tables.
In any case, even if it needs a rename and/or an extra layer, would materializing and de-duping "on the way in" instead of "on the way out" help us?
(But, raised #80 with an existing issue that might be relevant in the decision.)

I ran this revision against 43 GB of data from a base partition. Unfortunately, this revision didn't pass.

I'll try to track down what events didn't pass tomorrow.

For reference, that site has an ad_impression event that fires a lot. If it's the lone cause then I'd approve this PR as being better than the status-quo. Failing 202 times in this volume of data is pretty darn close to zero.

The other big array-of-struct field is user_properties, so it would be interesting to see if that field varies for those near-duplicates.

I'm thinking that if we want to consider deduplicating events and making
events unique at the same time, we might want to drop events with duplicate
event_params arrays. I'm fine work possible edge cases like mine (pending
confirmation) requiring extra implementation effort.
On Mon, Oct 24, 2022, 10:40 p.m. Will Bryant @.***>
wrote:

The other big array-of-struct field is user_properties, so it would be
interesting to see if that field varies for those near-duplicates.
—
Reply to this email directly, view it on GitHub
https://github.com/Velir/dbt-ga4/pull/74#issuecomment-1290014222, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ADM6FXEDIFQBQJFTF7OYRKLWE5XEPANCNFSM6AAAAAARJGNSA4
.
You are receiving this because you were mentioned.Message ID:
@.***>

There's a discussion about this in the #measure Slack instance and this was very interesting:
"There’s nothing in those individual events that determine a sequence number or a timestamp – there’s no deterministic order either. Essentially, hits that happen within a batch are completely devoid of any information as to when those events actually happened AND what their relative order is. That’s why sending a timestamp as a custom event parameter with all hits is very useful if you want to"
So if you want distinct events, you need to update your GA4 implementation to include a sequence or random int or original timestamp. It can't be solved entirely in BQ.

@willbryant's idea to perform this between the source data and the base table has a lot of merit.
When we deliberately set the partitions that we are querying, window functions only operate on those partitions. Querying, yesterday's data with where event_date_dt in (current_date, date_sub(current_date interval 1 day)) will force the window function to only scan yesterday's and today's partitions.
This could solve a lot of problems that we are having. We could move all window functions here, like the session_first_last models and add session_first_page_view, session_last_page_view etc... to each event, do the row_number calculation and not cause failures on the unique test for event_key.
We could fix session_start in some way by generating a session_start event where it is missing or selecting the earliest event with referrer info populated and use that for first_event calculations possibly adding something like session_first_event_key.
Basically, we process all of our window functions between the two partitioned tables and the stg_ga4__events table retains its fluid sizing which is convenient for ad-hoc queries against staging tables.

Maybe we could add a client_side_timestamp_field variable. Users working on large sites, can use this variable to map any custom timestamp, as discussed in #measure Slack, and, if this variable is set, we add it to the event_key.
I like this solution a bit better than the status quo.

Apologies if this is an obvious point but re uniqueness, I only personally had an issue when the defined test started failing. That is, no-one was complaining about non-unique events from GA4, which is never 100% perfect data anyway; just that the DBT pipeline failed.
What I'm getting at here is that if we generate event_key I would expect it to be unique - ie. I would expect the de-duping to match up with the event_key generation, whatever that is.
I don't personally have any strong opinion on what should go into event_key, as clearly I am a smaller user than you guys and don't run into the same level of issues.

Re session_start, I'm definitely keen on generating this based on first events rather than session_start events. (As I commented in #69, I think this needs to be from any event, not just page-view events, in order to cover mobile/Firebase.)
I like your direction in https://github.com/Velir/dbt-ga4/pull/74#issuecomment-1290997489. My comments around the intra-day headaches in #80 may not be a blocker - Google explicitly documents the intraday data as incomplete and non-final in a number of respects, so I think people will be OK with it being not cleaned up the same way.
Support for an optional client-side de-duping parameter makes sense to me. One thought, perhaps rather than hard-coding this sort of thing, we should be allowing people to configure the variables that go into event_key, and then we can just document how you would go about utilizing an extra custom param to do extra uniqueness control. This may give others the flexibility solve different issues too.

The dbt build command that is used by default in dbt cloud runs the tests just after each model is built while other threads build and test other models. When a test fails, dbt skips any child models that rely on the model that failed the test. Because the stg_ga4__events table is a requirement for every other table, we need our solution, whatever it is, to actually work.
I'm also okay with removing the test and settling with what we have now.

For session_start, if we generate a session_start event, we will also need to deduplicate as missing and duplicate session_start events are both problems. If we go with replacing session_start entirely, we might want to consider using page_view or screen_view (and make this our default everywhere we have page_view to support mobile) and possibly add a variable to add a list of valid session events to support measurement protocol "sessions".
I had intended the de-duping parameter to be configurable.

What I had in mind was not just configuring that one parameter, but configuring the entire list, eg.
{{ dbt_utils.surrogate_key(var('event_key_parameters', ['session_key', 'event_name', 'event_timestamp', etc])) }} AS event_key

(with whatever the current thinking is on the default list of event_key fields should be)
I'm using surrogate_key here which seems to be the standard way of doing hashing now, but mainly because it takes care of array arguments for us.

That looks good to me. When we document it, either in readme.md or in a comment, we could recommend adding a timestamp in the client to ensure uniqueness as our intended use of this variable.

If the goal is to provide unique event_keys for the majority of users who have a small-medium implementation, this PR does that. For the remaining users with large implementations, they can update their GTM implementation to include a unique value in the event params and this PR will automatically incorporate that into the event_key without the need for an additional client_side_timestamp_field variable. So I'm not following why the surrogate key for events would need to be configurable. Why would anyone need fewer fields in the surrogate key? Why would anyone need more event params in the surrogate key if we already include all event params?

If there's a suggestion to 'dedupe on the way in' I'm definitely open to it, I just need to see some code to fully understand the implementation. Can someone open a PR (even a draft one) with those changes? I'm just not following what that would look like.

This discussion is getting pretty long, I'm hoping that someone can provide an alternative PR so we can have a more informed discussion about the merits.

Ah yeah duh, good point re event params.
Testing out this branch at b3dd3d277f0ba6df9e1b9f5f6ee531e81743e8e4 against just my small non-prod dataset first, I get:
20:25:44  Failure in test unique_dim_ga4__sessions_session_key (models/marts/core/core.yml)
20:25:44    Got 19 results, configured to fail if != 0
20:25:44
20:25:44    compiled SQL at target/compiled/ga4/models/marts/core/core.yml/unique_dim_ga4__sessions_session_key.sql

With that commit the session test passes, and all tests pass on my non-prod dataset. On my prod dataset (~104 GB for base_ga4__events), I get:
21:01:17  Failure in test unique_stg_ga4__events_event_key (models/staging/ga4/stg_ga4.yml)
21:01:17    Got 58258 results, configured to fail if != 0
21:01:17
21:01:17    compiled SQL at target/compiled/ga4/models/staging/ga4/stg_ga4.yml/unique_stg_ga4__events_event_key.sql

I pulled the full rows for an event_key that had 3 rows, and the json dump was exactly the same for all 3. So they were true duplicates. I then had a look at the event names of all the non-unique rows:

Looks like those are all firebase events.

Thanks for checking! I updated the readme with our notes.

Unfortunately in this case the events are generated by Firebase itself, so injecting those parameters isn't available as a solution.
I have non-unique events with the current main branch, so if you wanna go ahead and merge this PR in the meantime, no reason to let my feedback delay you. But personally the #66 issue is unfortunately still not resolved (and perhaps needs to remain so until we can try another de-duping approach).

shoot, I see what you mean. That reminds me that there are some automatic gtag.js events that could produce the same situation as well.
Ok yea, I'd like to merge this PR but leave #66 open and maybe summarize what we've learned so far in that issue.


chore: promote h52 to version 0.0.19

chore: promote h52 to version 0.0.19
this commit will trigger a pipeline to generate the actual kubernetes resources to perform the promotion which will create a second commit on this Pull Request before it can merge

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign q1323829945
You can assign the PR to them by writing /assign @q1323829945 in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


MQTT Event

Hi,
i need to check the result of my backup but i don't receive the mqtt event and when i add the sensor the status is unavailable.
if i use Mqtt Explorer to check the db, i don't see google backup.
The rest works with no problem 👍
how can i check ?

Hi,
i need to check the result of my backup but i don't receive the mqtt event and when i add the sensor the status is unavailable.
if i use Mqtt Explorer to check the db, i don't see google backup.
The rest works with no problem 👍
how can i check ?

Sorry it took me so long to reply. What does your HA configuration for MQTT sensor look like? Here's what I recommend in the readme for my addon:
`  - platform: mqtt
name: "Google Backup Results"
state_topic: "googlebackup/result"
json_attributes_topic: "googlebackup/result"

platform: mqtt
name: "Google Adhoc Backup Results"
state_topic: "googlebackup/adhocresult"
json_attributes_topic: "googlebackup/adhocresult"`

Also, I've not used MQTT Explorer myself, but I've used MQTT Lens for debugging. I've noticed that I need to have MQTT Lens up & running and currently connected and subscribed to the topic in order for events to show up AS THEY HAPPEN. I can't bring up MQTT Lens AFTER running a backup and expect to see the results.

The MQTT event works great. It is not marked as "Retained", so you don't see it in your mqtt db if you check for it AFTER. Open your mqtt explorer program BEFORE, run a dobackup and when it finishes you will see the mqtt entry.


Fatal error when run any yii  console command.

When I run yii migrate/history on console terminal. The following error shown.
Exception 'yii\base\UnknownPropertyException' with message 'Setting unknown property: dektrium\rbac\RbacConsoleModule::admins'
I look in RbacConsoleModule.php, it did not have admins property. So I modified method bootstrap($app) in Bootstrap.php
            // if dektrium/user extension is installed, copy admin list from there
            if ($this->checkUserModuleInstalled($app) && ($app instanceof WebApplication)) {
                $app->getModule('rbac')->admins = $app->getModule('user')->admins;
            }

Then it works again.
My environments is on PHP 5.6 on Windows, Yii 2.0.5, yii2-rbac : dev-master 6998d19.

Fixed!


Build Python with pymalloc

See #11

Thanks for this. I'm pretty sure the --without-pymalloc is a bit of cargo-culling left over from a previous incarnation. I don't think there's any reason it needs to be preserved, though.


Fix bug in chunking code for distributed controlled gates

A couple of assert conditions were not properly formulated and this made the chunking code fail.
In addition, when exactly 2 MPI processes were used, the chunking code was deactivated. Now this limitation has been removed.
PS: using the new pull-request flow that merge the modifications to "development" first.

It makes sense to me those changes. I merge it.


Create separate date setting for list pages

I prefer a longer format for the date (e.g. March 12, 2018) shown on individual posts, but that format doesn't look great on listing pages.
Tags and archives should use a date format with a consistent width so they don't look like this:

Another option might be to just hard-code the date format on lists,  but that may be too restrictive.


Typo "@@ERROR"→"@@ERROR"

https://docs.microsoft.com/en-us/sql/t-sql/functions/error-transact-sql?view=sql-server-ver15
#PingMSFTDocs

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Merging per recent similar approval. Thanks @hyoshioka0128.