az synapse pipeline update error

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Related command
<!az synapse pipeline update --workspace-name wrkname5001 --name MypipelineToSynapse --file @"C:\Users\commom....\PipelineSettings.json"
Command group 'synapse' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
The command failed with an unexpected error. Here is the traceback:
'properties'
Traceback (most recent call last):
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/synapse/manual/operations/artifacts.py", line 82, in create_or_update_pipeline
KeyError: 'properties'>
Describe the bug
<!-Unable to update the pipeline parameter from the input JSON. JSON file consist of the parameters generated from az synapse pipeline show command>
To Reproduce
< just execute below  command with inputs in JSON file.
az synapse pipeline update --workspace-name wrkname5001 --name MypipelineToSynapse --file @"C:\Users\commom....\PipelineSettings.json">
Expected behavior

Environment summary
<!azure-cli                         2.37.0
core                              2.37.0
telemetry                          1.0.6
Dependencies:
msal                            1.18.0b1
azure-mgmt-resource             21.1.0b1>
Additional context

route to CXP team

@ManojLava Thank you for reaching out to us, we are looking into it.

@ManojLava I was unable to reproduce this issue, can you please share more details? az synapse pipeline update --workspace-name samplewspace --name Issue22839 --file @"pipeline.json"
This is the response body I used as json file https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-or-update-pipeline.
Can you retry this please and let me know what happens?
Here are my environment details.
azure-cli                         2.37.0
core                              2.37.0
telemetry                          1.0.6
Dependencies:
msal                            1.18.0b1
azure-mgmt-resource             21.1.0b1

Hi Team,
Thanks for your reply. My issue is still not resolved. My guess is that my JSON file format is not correct. Please guide how to get correct format JSON file from existing Pipeline.

@ManojLava You can find the sample response JSON Body here and tweak according to your need https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-or-update-pipeline#request-body

Hi Team,
I trying to execute below command to JSON response for a specific file as per details share in above link
PUT {endpoint}/pipelines/{pipelineName}?api-version=2020-12-01
Below is the steps I have execute to make this api call
Piece of my code to get access token and moving it to headers variable
$Credential = Get-Credential
Connect-AzAccount -Credential $Credential
$token = (Get-AzAccessToken).Token
$headers = @{ Authorization = "Bearer $token" }
Assigning the uri variable with Azure details and Pipeline name
$uri = "https://web.azuresynapse.net/subscriptions/$SubscriptionID/"
$uri += "resourceGroups/$ResourceGroup/providers/Microsoft.Synapse/"
$uri += "workspaces/$workspaceName/"
$uri += "pipelines/MyPipeline?api-version=2020-12-01"
$response = Invoke-RestMethod -Method Put -ContentType "application/json" -Uri $uri -Headers $headers
I'm getting below error stating invalidate operation.
CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
what is the wrong with my api call? I'm not able to debug this issue.

@ManojLava we could see 404 response which is expected because we do not have that synapse resource. Hope you are calling it from powershell. Also this forum is for azure cli issues exclusively.
I will mark this issue for closure now.

can anyone please confirm whether this line is a warning or an error response?


Performance improvement tips?

Question
I've been using intero on emacs and in my case, I'm experiencing very laggy performance when typing. I ran the profiler and I've attached the report which states quite a lot of CPU hit is caused by intero-network-call-sentinel
Can something be done to improve the performance on that side?
This is a fairly standard stack project with about 120 modules; with resolver:
resolver: lts-12.26

+ timer-event-handler                                           24422  25%
+ ...                                                           22121  23%
- intero-network-call-sentinel                                  17427  18%
 - delete-process                                               15742  16%
  - intero-network-call-sentinel                                 2070   2%
   - delete-process                                              2026   2%
    - intero-network-call-sentinel                                527   0%
     - delete-process                                             520   0%
      - redisplay_internal (C function)                            24   0%
       - eval                                                      20   0%
        - if                                                       11   0%
         - replace-regexp-in-string                                11   0%
            abbreviate-file-name                                    6   0%
        - let*                                                      9   0%
         - list                                                     4   0%
          - powerline-raw                                           2   0%
           - if                                                     2   0%
            - progn                                                 2   0%
             + let*                                                 2   0%
          + funcall                                                 1   0%
         + append                                                   2   0%
         + concat                                                   1   0%
       + #<compiled 0x3d0541>                                       4   0%
     + #<compiled 0x12135ad>                                        6   0%
    + redisplay_internal (C function)                             188   0%
   + #<compiled 0x12135ad>                                         42   0%
  + redisplay_internal (C function)                              1038   1%
 + #<compiled 0x12135ad>                                         1662   1%

I can provide further details if necessary.

Interesting. So the network calls are a performance problem? They're used basically to make async read-only calls to intero.
Relatedly, I will probably be dropping support for that in the next intero version, because the network package dropped its Network module which I was using.
So probably I can go back to the drawing board on making async calls.

Related #599

@chrisdone it seems like it yes. I've repeated the profiling after multiple starts of emacs and on various projects of differing sizes. It seems to me that this is the bottleneck.
If there's anything I can help with (including contributions, if you can point me to sources I can read to understand the inner workings of Intero),  I'll gladly do so. I use Intero for my daily workflow, and I'd like to contribute.


Constraint changer - skills tab

While this PR adds a whole new tab for "Skills" and tries to make them fully parametrizable, unfortunatelly, due to certain limitations, even when the MethodHandle reflection-way of using 'reflection without utilizing the java.lang.reflection.* stuff' (which is blocked by the game's SecurityManager) is utilized to strip some public static final members of their final modifier - the fields are still not changable.
At this point, it's less about my potentially wrong use of MethodHandles and a bit more about the SecurityManager just not letting me change them; which sortof makes the whole "lets make all skills' attributes parametrizable" idea somewhat moot.
BUT - as long as the header dividers in LunaSetting's CSV mention that "Changing these values currently does nothing" and actually invoking reflection that tries to change them methods are commented out, it can still function as a mod... that half-works and half doesn't work.
Out of all the implemented skills, (currently) only the Field Repairs skill has 6 unchangable parameters. The number will surely rise as more skills are covered though, but as long as the header/divider mentions that they don't work - it should be fine i think.
Of course, further refactoring and breaking the method into updateWorkingParts() and //updateNonWorkingParts() would be greatly appreciated where the WorkingParts thing will actually perform work that I know works fine, and the commented-out NonWorkingParts will perform work that I know doesn't work fine - namely the reflection thing I've spent time developing.

Well, the whole problem with public static final members is that their values are inlined in the API, so even when stripped of their final modified and changed, the inlined value might remain.
As it is, should just give up on trying to change the final fields and move on.


It works only for one textbox. Other textboxes in the app that use this behaviour are ignore

I have actually reproduced this in a simple test app. If it's useful I can make a PR with the test app.

Go ahead and branch it, then add your test app.

Just as a quickfix (i have discovered no side effects but maybe there are some):

remove the handling of _isSubscribed in OnAutoCompleteItemsSource:

if (e.NewValue == null) {    tb.TextChanged -= onTextChanged;    tb.PreviewKeyDown -= onKeyDown; } else {    //New source.  Add the callbacks    tb.TextChanged += onTextChanged;    tb.PreviewKeyDown += onKeyDown; }


Colorpicker

It would be awesome if someone could created (or already has created) a color picker in the MahApps style. Including some color converters Getting and Settings a Color or brush. Currently i'm using one from Xceed WPF Toolkit which is nice but it stands out that it's not the same theme as the rest of the application.

Hi @TripleNico ,
I have one in the pipeline. Right now I am waiting for the feedback of @punker76 . See #3710
A preview is also available, but you need to build the dll on your own: https://github.com/timunie/TimsWpfControls
Happy coding
Tim

Ah, that's really awesome! I will follow this and hopefully it get's implemented. Until then i will stick with our current version but it's exited to know that this control will be added in the feature! Keep up the good work!


thermald: 1.7.2 -> 1.8

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/thermald/versions.
meta.description for thermald is: '"Thermal Daemon"'.
Compare changes on GitHub

Checks done (click to expand)

built on NixOS
/nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8/bin/thermald passed the binary check.
1 of 1 passed binary check by having a zero exit code.
1 of 1 passed binary check by having the new version present in output.
found 1.8 with grep in /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8
directory tree listing: https://gist.github.com/e55d618c9e2127165563f344a483a182
du listing: https://gist.github.com/2e8d1fcf0b180f3ba38e123380966e17

Instructions to test this update (click to expand)

One-time optional setup to skip building using Cachix:

Install cachix from https://cachix.org/
Use r-ryantm's cache:cachix use r-ryantm

Test this update by entering a nix shell, seeing what is inside the
result, and if applicable, running some binaries:
nix-shell --pure -I nixpkgs=https://github.com/r-ryantm/nixpkgs/archive/c064995331adbc19f7abacd32f505022d080ac26.tar.gz -p thermald
ls -la /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8

cc @abbradar for testing.

No attempt on aarch64-linux (full log)
The following builds were skipped because they don't evaluate on aarch64-linux: thermald
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: thermald
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: thermald
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8
shrinking /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8/sbin/thermald
gzipping man pages under /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8/share/man/
strip is /nix/store/dxf1m7dhc4qb655bdljc1fsd74v1nag3-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8/sbin
patching script interpreter paths in /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8
checking for references to /build in /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8...
moving /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8/sbin/* to /nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8/bin
/nix/store/mzbll3hfqqmil9zfs8552kyjplhpx88n-thermald-1.8


feat: add getVideoIdFromUrl to get id

apply issues/1137.

Thanks for your PR. On second thought, I don't think it's this library's business to do URL parsing. Users of the library should be responsible for handling this.
We don't want to have to maintain logic that parses URLs.


Sequence Delimitation Item not present in the element list

When using the method parseDicom on DICOM containing a SQ element with undefined length I have the feeling it's not correct to not return the SequenceDelimitationItem as an item of said element
In this example : x00080096 (ReferringPhysicianIdentificationSequence) is a Sequence of undefined length
It is followed by a SequenceDelimitationItem, but it's nowhere to be found.
I think it should be part of the items list of the current tag like the ItemDelimitationItem ("xfffee00d") is for Item ("xfffee000")

Is the current behaviour the expected behaviour or do you confirm this is bug ?
Thank you for your work

I used your example : https://rawgit.com/cornerstonejs/dicomParser/master/examples/dumpWithDataDictionary/index.html
with google chrome and a break point juste after the dataSet = dicomParser.parseDicom(byteArray, options); in the index.html to produce the screen shot

It does look like it used to be that way a long time ago (here are the changes that introduced the change). @chafey any thoughts on this one?

This sounds like a bug - the intent of the design is to expose all of the
detail of the underlying bitstream and SequenceDelimitationItem should be
present as one of the items in a sequence of undefined length.  I seem to
recall some other bugs related to undefined length sequence parsing - it
would love to see someone review this in depth, add tests and refactor if
necessary
On Fri, Oct 1, 2021 at 9:23 AM Bryan Cool @.***> wrote:

It does look like it used to be that way a long time ago (here
https://github.com/cornerstonejs/dicomParser/commit/f63b40858c110f671ae0867a10ac2ce7624f1b82
are the changes that introduced the change). @chafey
https://github.com/chafey any thoughts on this one?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cornerstonejs/dicomParser/issues/143#issuecomment-932275373,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJVXWVZJIN3WDTDENVG74DUEW75PANCNFSM4MNFTS5A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Note that this will be a breaking change, as it will change the reported length of the SQ element.


Tabs: mobile support

Feature request
Overview
It's a great idea that your components can be downloaded separately, but they are not optimized for mobile platforms and do not support theming through design tokens.
I tried adding more tabs than in your example, and they didn't collapse into a dropdown menu or scroll if they didn't fit within the screen width.
Examples in other libraries
I don't understand Ant's philosophy and their attitude towards users, but at least their tabs are adaptive.
http://ant.design/components/tabs/

I tried adding more tabs than in your example, and they didn't collapse into a dropdown menu or scroll if they didn't fit within the screen width.

as radix is unstyled this decision is completely up to you


[BFO-312] Add SA specific tags to SARIF

Closes #
Reason for Proposed Changes
Proposed Changes

I submit this contribution under the Apache-2.0 license.

/merge


[mieb] update script for final re-run

Checklist

[x] Run tests locally to make sure nothing is broken using make test.
[x] Run the formatter to format the code using make lint.

re-run script for datasets that needed to be fixed (flickr, r-oxford, r-paris, blink's 4 settings). We are changing r-oxford and r-paris to use the MultiChoice evaluator instead of the retrieval evaluator to match its original setting in the paper, which will be merged later in #1572

Just a comment that the test error is perhaps due to some unexpected version mismatch, also seen in #1572. Am merging now and will revisit after rerun.


Support Go project for reading current version

Dependencies

Feature Request
Read current version in a Go project

Hey, @philippart-s
I would love to work on this issue can you give some brief info about this issue.

hi @sshiv5768 ok !

@sshiv5768 did you still interested to work on this issue ?

Hi @philippart-s I see no one is currently assigned to the project and would like to start work on it. Can I be assigned to the issue?

Hi, @slim-patchy.
Yes you can, I assign the issue to you.
Thanks 😉

Thanks @philippart-s I will follow up with a PR soon

I am currently working out how best to put some code to parse the very long output of the command go list -m -versions, is the directory src/main/java/io/jenkins/plugins/conventionalcommits/utils a good place for it? For a background the output for the hey repo can look something like the following:
go list -m: loading module retractions for<EMAIL_ADDRESS>verifying go.mod<EMAIL_ADDRESS>initializing sumdb.Client: open /opt/homebrew/bin/go/pkg/sumdb/sum.golang.org/latest: not a directory

So some regex may be warranted for such a task...

yes if your code is GO oriented (or just used for a Go project) put it in a class GoProjectType

Sure, no problem! Have just completed a first draft of the code for Go support, on to the testing stage next. Should be done over the next few days.

Just submitted PR #156 to follow-up on this issue.

I'll review it as soon as possible


Add Close or Cancel button in Make Distribution dialog

The Make Distribution dialog does not have a Close or Cancel button.

Aye, modal dialogs on Mac have no buttons in the window titlebar.

I don't think there is supposed to be a "Close" or "Cancel" button. The "Ok" button does nothing but closing the dialog.

I've never used this dialog though, I don't even understand what it's used for.

Me neither :) just seemed weird with no cancel button. No biggie.


Allow multiple etcd host

Change Description
This patch allow set multiple etcd servers. like an etcd cluster in docker swarm/kubernetes.
Steps to Test
./lnd-debug \
   --db.backend=etcd \
   --db.etcd.host=etcd1:2379,etcd2:2379 \

Pull Request Checklist
Testing

[ ] Your PR passes all CI checks.
[ ] Tests covering the positive and negative (error paths) are included.
[ ] Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

[ ] The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
[ ] Commits follow the Ideal Git Commit Structure.
[ ] Any new logging statements use an appropriate subsystem and logging level.
[ ] Any new lncli commands have appropriate tags in the comments for the rpc in the proto file.
[ ]  There is a change description in the release notes, or [skip ci] in the commit message for small changes.

📝 Please see our  Contribution Guidelines for further guidance.

Signed the commit. i dont know what to write to release note.

Can be something simple, like "Allow multiple etcd hosts to be specified in db.etcd.host."
The important thing is that you link to this PR in the release notes, which makes that specific check in the CI green.

@bitbandi, remember to re-request review from reviewers when ready

@bitbandi, remember to re-request review from reviewers when ready

@bitbandi, remember to re-request review from reviewers when ready

@bitbandi, remember to re-request review from reviewers when ready

@bitbandi, remember to re-request review from reviewers when ready

conflict resolved


Synchronizing hot fixes with dev

Codecov Report

Merging #285 (629bc38) into develop (7920849) will increase coverage by 1.55%.
The diff coverage is 38.19%.


applying changes for problem refered on:

https://github.com/logstash-plugins/logstash-output-nagios/issues/3
According to error:
NameError: undefined local variable or method `message' for  ....
and tilde delimeter problem. Seems I dont use timestamps in the nsca message I changed the tilde agianst colon (:) as delimeter.

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then  say 'jenkins, test it'.

please jenkins, test this.

Merged manually, including other changed in https://github.com/logstash-plugins/logstash-output-nagios_nsca/commit/fd0b85152f4b1f6925c206da074b65bcfbeca4e4


Repo sync for protected CLA branch

The pull request is created from master637344916729430603 to master to fix git push error for protected CLA branch

@cmcclister : Thanks for your contribution! The author(s) have been notified to review your proposed change.


last_result() is broken

Hi,
I'm using last_result and have found that the generated URL is screwed up.
It works if I change the following code:
request({method: 'GET', url: build_url(last_result_url + API, jobname)}, function(error, response, body) {

to the following:
request({method: 'GET', url: last_result_url + API + jobname}, function(error, response, body) {

The reason why is because the call to build_url fails when it util.format, as there is no %s in the string, the host is copied into the string a second time. It seems that the last_result_url already has the host information in it.
In my case the host is different to the value returned in the last_result. Both work, just appears to be a config problem with our jenkins due to network migrations.
I would do a pull request, but I'm not 100% clear on your logic for using build_url.

+1

yup the url is broken :(


Can't change date format

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
I can't change the date format in the left column. I tried to use localized format but there is no behaviour
Expected behaviour
I would like to change date format into DD MM YYYY
Steps To Reproduce
No response
Mode
Event (default)
Card Version
9.6.1
Home Assistant Version
2024.2.3
Configuration
No response
Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

I'm not the OP, but might have the same failed expectation.
Using the following configuration:
- type: custom:atomic-calendar-revive
  entities: calendar.family
  dateFormat: "dddd, MMMM D, YYYY h:mm A"

  # the following options are just for simplification
  disableLocationLink: true
  showRelativeTime: false
  showDatePerEvent: true
  showLocation: false

I get this calendar event:

Instead of 02 (the day number), I expected it to display "Saturday, March 02, 2024 8:00 AM".
Maybe this is a documentation issue. The description of dateFormat is just "Custom date format", but doesn't specify which date can be formatted. I expected it to be the date in the left column (which I am able to modify using f.e. showWeekDay, but it doesn't render the way I want it to!), but maybe that's not correct.

I found a working example of dateFormat in #1334:
- type: custom:atomic-calendar-revive
  entities: calendar.family
  dateFormat: "dddd, MMMM D, YYYY h:mm A"
  defaultMode: Calendar
  showDate: true

So I guess that dateFormat only applies to defaultMode: Calendar, not defaultMode: Events. The documentation does not properly reflect this, in my opinion.
Again, I'm not OP, so I might have just hijacked their issue. You shouldn't close the ticket until they confirm that they were also expecting dateFormat to work for defaultMode: Events.

@hheimbuerger @Luana-98 the documentation probably needs some clarification here, dateFromat is for the card date, not the event dates.

its not currently possible to change the date format of the events themselves.


Add router DI definition to Bootstrap

Added the router DI definition to the bootstrap:

RouteStack -> instantiator becomes TreeRouteStack::factory
TreeRouteStack -> instantiator becomes TreeRouteStack::factory

Basically, moving this to the skeleton app led to a FAQ situation, and
if we need a FAQ for common setup, we need to code such that the FAQ
is no longer necessary. Additionally, you can override the definition
at any point within your own configuration, as app/module
configuration is applied after the explicit configuration in the
bootstrap.

Had to merge this, as I discovered other tests were failing, and other PRs as well, due to the configuration having been moved.


[Drawer] Handle closing of drawer on navigation Pop (Browser Back Button)

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Summary 💡
Currently the Drawer component can be closed by clicking outside the drawer, or handled programatically by calling a handleClose function after execution of any list tiles inside it.
But the browser back button, or if in phones, the back gesture (Swipe right, press the android back button) just pops from the history and goes back to the previous page, or closes the website if already in root route.
It would be amazing to have this handling to be embedded inside the drawer component, for PWA implementations
Examples 🌈
Expected Behaviour:
IF Drawer is open, it closes and overrides the pop navigation on a browser back button, android back button or an iOS swipe right gesture.
Motivation 🔦
I am developing a PWA using MUI, and it has been an amazing experience, but the users intuitively hit the back button to close the drawer, which leads them to exit my website, or in PWA case, exit the application if they are on the root route.
Would love to just close the drawer as expected, and make them retain on the website / PWA

Why can't you achieve this if the component is in controlled mode? You can control the open prop on any event required.

@utkarshgupta2504 you can't directly prevent the browser back navigation, but you can listen to the "popstate" event and then tell it to navigate forward on mobile and make it close the drawer. It's not ideal (especially if you store state in the URL which causes the page to behave differently) but for most cases it should get the job done and is better than having users exit your PWA. Here is a full example of an implementation I use which does the trick:
// Drawer.tsx
import { Drawer, useMediaQuery } from '@mui/material'
import { useTheme } from '@mui/material/styles'
import { useCallback, useEffect, useState } from 'react'

export const useIsMobile = () => {
  const theme = useTheme()
  return useMediaQuery(theme.breakpoints.down('sm'))
}

const Component = () => {
  const isMobile = useIsMobile()
  const [isOpen, setIsOpen] = useState(false)

  const closeDrawer = useCallback(() => {
    setIsOpen(false)
  }, [])

  const onPopState = useCallback(() => {
    if (isMobile && isOpen) {
      window.history.forward()
    }
    closeDrawer()
  }, [isMobile, isOpen, closeDrawer])

  useEffect(() => {
    window.addEventListener('popstate', onPopState)
    return () => {
      window.removeEventListener('popstate', onPopState)
    }
  }, [onPopState])

  return (
    <Drawer open={isOpen} onClose={closeDrawer}>
      {/* ... */}
    </Drawer>
  )
}

export default Component


Add resolveInternal to swagger2openapi options

Description

Adding resolveInternal option when using swagger2openapi to convert Swagger 2.0 to oapi3.
Motivation and Context

Some $ref doesn't seem to get resolved without it.
https://github.com/PaloAltoNetworks/docusaurus-openapi-docs/issues/372
How Has This Been Tested?

After adding this option i ran the gen-commands and all $refs seemed to have been resolved. The generated docs looked as expected.
Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist

[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes if appropriate.
[x] All new and existing tests passed.

Hi @gustavste, thanks for the PR. Can you run yarn format --fix to address the linter issues?


I cannot reproduce the results of the baseline because its context_relevancy is too high.

I cannot reproduce the results of the fiqa baseline in this notebook:
https://github.com/explodinggradients/ragas/blob/main/experiments/baselines/fiqa/dataset-exploration-and-baseline.ipynb
At the end of this notebook, is shows the score:
{'NLI_score': 0.8655555555555556, 'answer_relevancy': 0.8737666666666667, 'context_ relevancy': 0.8181444444444443, 'ragas_score': 0.8517704492684051}
But when I test, the score I get is:
{
      "context_ relevancy": 0.10368744449698047,
      "faithfulness": 1.0,
      "answer_relevancy": 0.9286177818722253,
      "context_recall": 0.6370370370370371,
      "harmfulness": 0.0,
      "ragas_score": 0.300955397960847,
}

I noticed that my context_relevancy is very low. I know that in the latest pr, the prompt used to test context_relevancy has been modified, but I am not running with this latest version. But I don’t think it’s caused by this, because this jupyter notebook seems to be running with an old version.
In addition to the context_relevancy value, there seem to be some gaps in the other values. So I'm wondering, what could be causing this?

Hi @zc277584121 , the results for the quickstart notebook were generated using an older version of fiqa dataset which contained very few rows.  So if you run it now on the entire dataset the scores will be different.

Thank you for your answer.
So can you update it and consider it as a standard baseline? This way other RAG frameworks can easily compare with it.

I saw that it was using the first 30 rows. My results were also generated using the first 30 rows, not the entire data set.
@shahules786 , You mean, it uses an older version of the dataset and it has less than 30 rows?

In the example, it selects the first 3 rows of the dataset fiqa_eval["baseline"].select(range(3)), not 30. I can run and update the notebook with the newer version. Would that help?

Thank you very much for your answer. I suggest you provide a stable and reproducible baseline so that other RAG frameworks can be compared with it. 😊

Sure. Could you tell me what you're trying to achieve? Is this issue a blocker for your use case?
I'm asking because since we are updating the code to improve metrics, there can be changes in the values of scores dedicated in the quickstart.

@shahules786 , This will not block our use. Even if the reproduced results are different from those in the notebook, I will currently use the results of my local run instead of those in the notebook. I was originally worried that there was something wrong with my reproduction process, but now it seems there is no problem.

@zc277584121 Thank you.


[Bug] Import Cookie Error on Hyperion/Miyoushe

Description
The tool always returns an error code -100 when I try to import cookies from my browser.

Reproduction Steps

Go to Miyoushe on my browser and Login
Open Dev Tools -> Console -> type document.cookie and copy all data
Open Starward -> Set CN Server on Star Rail and go to Hyperion Toolbox
Click Login -> Input Cookie

Expected Behavior
No response
Screenshots
No response
Starward Version
0.9.0-preview.1
Windows Version
19044.3086
Log
No response
Additional Context
No response

Hi, I'm sorry to see you have encountered some difficulties.
However, Hyperion has strict account risk control mechanisms. So I cannot provide any other assistance except to suggest you log in again.


Wrong message when get new Title

when someone levels up and get a new title everyone on the server see the message.skillup.nobroadcast message. This is confusing becaurese evryone thinks he has leveld up.

4ed48da227376ad6eb74525cdfe24b5d093f9de0


Failed to discover test case if import external Class and use the class attributes before case

Environment data

VS Code version: 1.32.3
Extension version (available under the Extensions sidebar): python 2019.3.6352 (2 April 2019)
OS and version: windows 7
Python version (& distribution if applicable, e.g. Anaconda): 3.6
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
Relevant/affected Python packages and their versions: pytest

Expected behaviour
should find the test case , and show the button Run Test

Actual behaviour
if import class , and use the class attribute, find test case failed

Steps to reproduce:

in same folder, create data.py and init.py

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-

class Http():
    headers = {
        "content-type": "application/json"
    }
    headers_patch = {
        "content-type": "application/json-patch+json",
    }

class Template():
    nfInstance_m = {
        "nfInstanceID": "4947a69a-f61b-4bc1-b9da-47c9c5d14b60",
        "nfType": "AMF",
        "nfStatus": "REGISTERED"
    }
    nfInstance_m = {
        "nfInstanceID": "4947a69a-f61b-4bc1-b9da-47c9c5d14b61",
        "nfType": "AMF2",
        "nfStatus": "REGISTERED"
    }

in test file, import data and class, then add class attributes into list

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-

from assertpy import assert_that

from data import Http, Template

# Init Data
nfInstance_list = [Template.nfInstance_m,Template.nfInstance_moc]

class Test_instances():
    def test_nfinstance_init(self):
        assert_that(1).is_equal_to(1)

try to trigger and find test case, but failed, no button Run Test

run this case by pytest, it can pass

Logs
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
=================================== ERRORS ====================================
______ ERROR collecting nrf-test-wgl/1.R15.1.0/nf-management/test_xx.py _______
nrf-test-wgl\1.R15.1.0\nf-management\test_xx.py:49: in <module>
    nfInstance_list = [Template.nfInstance_m,Template.nfInstance_moc]
E   AttributeError: type object 'Template' has no attribute 'nfInstance_moc'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!

if add class into test file, can find case

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-

from assertpy import assert_that

# from data import Http, Template

class Http():
    headers = {
        "content-type": "application/json"
    }
    headers_patch = {
        "content-type": "application/json-patch+json",
    }

class Template():
    nfInstance_m = {
        "nfInstanceID": "4947a69a-f61b-4bc1-b9da-47c9c5d14b60",
        "nfType": "AMF",
        "nfStatus": "REGISTERED"
    }
    nfInstance_moc = {
        "nfInstanceID": "4947a69a-f61b-4bc1-b9da-47c9c5d14b61",
        "nfType": "AMF2",
        "nfStatus": "REGISTERED"
    }

# Init Data
nfInstance_list = [Template.nfInstance_m,Template.nfInstance_moc]

class Test_instances():
    def test_nfinstance_init(self):
        assert_that(1).is_equal_to(1)

Sorry i am not able to reproduce the error. I have 3 files inside of folder directory, data.py, test_.py, and init.py` (which is empty). Can you please confirm if this is what you have as well?

@karrtikr
Sorry,when I met this issue in my project, there are multiple folders and files, when file this issue I just add simple test file to reproduce  it in same folder.
Today I reproduce  this issue, find the root cause is same name file(data.py) have be import in other test files.  if the new test file import
this is my reproduction:

the test_1.py import data.py in same folder
the test_2.py import data.py in same folder,  and the data.py have different data,this is my purpose
due to this issue, the test_2.py will use data.py in nf-discovery directory, not the data.py under same directory.

This seems like a separate issue.

Today I reproduce this issue, find the root cause is same name file(data.py) have be import in other test files. if the new test file import
this is my reproduction:

the test_1.py import data.py in same folder
the test_2.py import data.py in same folder, and the data.py have different data,this is my purpose
due to this issue, the test_2.py will use data.py in nf-discovery directory, not the data.py under same directory.

As far as this issue, this is supposed to happen. PYTHONPATH variable in .env file specifies where to start searching for modules. If data.py is found someplace else, it just uses that.
You could just use different names for each data.py file.

@karrtikr
Sorry,when I met this issue in my project, there are multiple folders and files, when file this issue I just add simple test file to reproduce it in same folder.

And about this bug, I am not able to reproduce it on my end. Can you please be more specific to the folder structure and file contents?

Hello! I have reviewed this issue and have tested a similar scenario successfully on our testing rewrite. Therefore I am going to close this issue but please comment or open a new issue if you are still seeing a problem when you try this yourself!
To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.
Thanks!


Display tours from the database on the home page

Implement read operation for all tours in the database.

Tours can be loaded from the database


Azure.Storage.Blobs UploadAsync raises System.ObjectDisposedException: Cannot access a disposed object.

When I try to upload different files, I get the following error:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.ConnectStream.EndWrite(IAsyncResult asyncResult)
   at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_1(Stream stream, IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.Stream.<CopyToAsyncInternal>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.RequestContent.StreamContent.<WriteToAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RequestActivityPolicy.<ProcessNextAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RequestActivityPolicy.<ProcessAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.BlobRestClient.BlockBlob.<UploadAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<UploadInternal>d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<>c__DisplayClass53_0.<<GetPartitionedUploaderBehaviors>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.PartitionedUploader`2.<UploadInternal>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<UploadAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<UploadAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

While the error is sporadic, in some cases it always occurs for the same file. I couldn't figure out the charecteristics of the specific file, but it is not the file size or file name. It happens with small files (less than 1 MB) or larger files (300 MB).
This is the code I use to upload the file:
var serviceUri = new Uri(BlobConnectionString);
var blobServiceClient = new BlobServiceClient(serviceUri);
var container = blobServiceClient.GetBlobContainerClient(ContainerName);
var blobClient = container.GetBlockBlobClient(fileName);

using (var fileStream = File.OpenRead(fileName))
{
	var blobUploadOptions = new BlobUploadOptions
	{
		TransferOptions = new StorageTransferOptions
		{
			InitialTransferSize = BlockSize,
			MaximumConcurrency = Environment.ProcessorCount * 2
		}
	};
	await blobClient.UploadAsync(fileStream, blobUploadOptions);
}

Environment:
Azure.Core 1.7.0
Azure.Storage.Blobs 12.7.0
Azure.Storage.Common 12.6.0
Target Framework: .NET 4.8
Windows 10

IDE and version : Visual Studio 2019 Enterprise - 16.8.2

Thank you for your feedback.  Tagging and routing to the team best able to assist.

One of our customers has the same error when he tries to upload a file with UploadAsync.
Environment:
Azure.Identity 1.2.3
Azure.Security.KeyVault.Keys 4.1.0
Azure.Storage.Blobs 12.7.0
Target Framework: .NET 4.7.2
Target Platform:  x64

This seems to be a multi-threaded issue. I call multiple instances of my class from multiple threads. When I call sequentially, it seems to be working just fine. There might be a shared state within the framework (that SSL object?) that get disposed by different threads.

@jaschrep-msft, could you take a look?

This is the code my tool uses to upload a file:
string tenantId = "<sp tenant>";
string clientId = "<sp appId>";
string clientSecret = "<sp secret>";
string connectionString = "";
ClientSecretCredential cred = new ClientSecretCredential(tenantId, clientId, clientSecret);
var vaultUri = new Uri("https://abc.vault.azure.net/");
KeyClient keyClient = new KeyClient(vaultUri, cred);
KeyVaultKey rasKey = await keyClient.GetKeyAsync("blobKey");
IKeyEncryptionKey key = new CryptographyClient(rasKey.Id, cred);
IKeyEncryptionKeyResolver keyResolver = new KeyResolver(cred);
ClientSideEncryptionOptions encryptionOptions = new ClientSideEncryptionOptions(ClientSideEncryptionVersion.V1_0)
{
	KeyEncryptionKey = key,
	KeyResolver = keyResolver,
	// string the storage client will use when calling IKeyEncryptionKey.WrapKey()
	KeyWrapAlgorithm = "RSA1_5"
};

BlobClientOptions options = new SpecializedBlobClientOptions() { ClientSideEncryption = encryptionOptions };
BlobClient blob = new BlobServiceClient(connectionString, options).GetBlobContainerClient("test").GetBlobClient("test.txt");
using (FileStream file = File.OpenRead(@"c:\temp\test.txt"))
{
	await blob.UploadAsync(file);
}

I'm currently migrating some code from the Storage SDK v9 to v12. During testing I've been seeing this issue for both blobs and queues. Here's an example stack trace from QueueClient.SendMessageAsync in case it helps. Feel free to contact me internally if you want more information, as we hit this fairly regularly
   at System.Net.ConnectStream.EndWrite(IAsyncResult asyncResult)
   at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_1(Stream stream, IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.RequestContent.ArrayContent.<WriteToAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RequestActivityPolicy.<ProcessNextAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RequestActivityPolicy.<ProcessAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.LoggingPolicy.<ProcessAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.LoggingPolicy.<ProcessAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Queues.QueueRestClient.Messages.<EnqueueAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Queues.QueueClient.<SendMessageInternal>d__83.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Queues.QueueClient.<SendMessageAsync>d__81.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

@mikamins It's cut off from the top of your stack trace, but I assume your exception also reads System.ObjectDisposedException: Cannot access a disposed object.  Object name: 'SslStream'.? That's good to note, as the Queues package doesn't actually use Streams, only text that is passed down to Azure.Core (with the exception of QueueRestClient.EnqueueAsync_CreateMessage(), which is generated code.
I notice @coding-jesus, the only person I see an Azure.Core version from, is using 1.7.0, even though their versions of Azure.Storage.Blobs and Azure.Storage.Common only require Core version 1.6.0. Core 1.7.0 looks as if it came out the exact day this issue was first reported. I'd be curious to know if downgrading the version of Azure.Core to 1.6.0 eliminates this problem.

@mikamins It's cut off from the top of your stack trace, but I assume your exception also reads System.ObjectDisposedException: Cannot access a disposed object.  Object name: 'SslStream'.? That's good to note, as the Queues package doesn't actually use Streams, only text that is passed down to Azure.Core (with the exception of QueueRestClient.EnqueueAsync_CreateMessage(), which is generated code.
I notice @coding-jesus, the only person I see an Azure.Core version from, is using 1.7.0, even though their versions of Azure.Storage.Blobs and Azure.Storage.Common only require Core version 1.6.0. Core 1.7.0 looks as if it came out the exact day this issue was first reported. I'd be curious to know if downgrading the version of Azure.Core to 1.6.0 eliminates this problem.

Yes, the Exception.Message in the queue scenario is Cannot access a disposed object.\r\nObject name: 'SslStream'.  I am using Azure.Core 1.6.0 so it seems that downgrading would not fix the issue
<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Azure.Core" version="1.6.0" targetFramework="net472" />
  <package id="Azure.Storage.Common" version="12.6.0" targetFramework="net472" />
  <package id="Azure.Storage.Queues" version="12.5.0" targetFramework="net472" />
  <package id="Microsoft.Bcl.AsyncInterfaces" version="1.0.0" targetFramework="net472" />
  <package id="System.Buffers" version="4.5.0" targetFramework="net472" />
  <package id="System.Diagnostics.DiagnosticSource" version="4.6.0" targetFramework="net472" />
  <package id="System.Memory" version="4.5.3" targetFramework="net472" />
  <package id="System.Memory.Data" version="1.0.0" targetFramework="net472" />
  <package id="System.Numerics.Vectors" version="4.5.0" targetFramework="net472" />
  <package id="System.Runtime.CompilerServices.Unsafe" version="4.6.0" targetFramework="net472" />
  <package id="System.Text.Encodings.Web" version="4.6.0" targetFramework="net472" />
  <package id="System.Text.Json" version="4.6.0" targetFramework="net472" />
  <package id="System.Threading.Tasks.Extensions" version="4.5.2" targetFramework="net472" />
  <package id="System.ValueTuple" version="4.5.0" targetFramework="net472" />
</packages>

Yes, the Exception.Message in the queue scenario is Cannot access a disposed object.\r\nObject name: 'SslStream'.  I am using Azure.Core 1.6.0 so it seems that downgrading would not fix the issue
<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Azure.Core" version="1.6.0" targetFramework="net472" />
  <package id="Azure.Storage.Common" version="12.6.0" targetFramework="net472" />
  <package id="Azure.Storage.Queues" version="12.5.0" targetFramework="net472" />
  <package id="Microsoft.Bcl.AsyncInterfaces" version="1.0.0" targetFramework="net472" />
  <package id="System.Buffers" version="4.5.0" targetFramework="net472" />
  <package id="System.Diagnostics.DiagnosticSource" version="4.6.0" targetFramework="net472" />
  <package id="System.Memory" version="4.5.3" targetFramework="net472" />
  <package id="System.Memory.Data" version="1.0.0" targetFramework="net472" />
  <package id="System.Numerics.Vectors" version="4.5.0" targetFramework="net472" />
  <package id="System.Runtime.CompilerServices.Unsafe" version="4.6.0" targetFramework="net472" />
  <package id="System.Text.Encodings.Web" version="4.6.0" targetFramework="net472" />
  <package id="System.Text.Json" version="4.6.0" targetFramework="net472" />
  <package id="System.Threading.Tasks.Extensions" version="4.5.2" targetFramework="net472" />
  <package id="System.ValueTuple" version="4.5.0" targetFramework="net472" />
</packages>

@tg-msft this line in the generated client is the only line in Queues for the code path @mikamins follows that touches using or Dispose(). The original reported issue in blobs has one more Dispose() call, but it is on a separately implemented buffer stream that does not pass the call down to any other stream.
The method I linked looks like it handles the request body correctly, and code samples from users don't appear to have any stream mismanagement. I don't think this is an issue in storage's generated code. I wonder if this is in Core's handling of the web stack.

@tg-msft this line in the generated client is the only line in Queues for the code path @mikamins follows that touches using or Dispose(). The original reported issue in blobs has one more Dispose() call, but it is on a separately implemented buffer stream that does not pass the call down to any other stream.
The method I linked looks like it handles the request body correctly, and code samples from users don't appear to have any stream mismanagement. I don't think this is an issue in storage's generated code. I wonder if this is in Core's handling of the web stack.

@coding-jesus would you be able to give more details about your multithreaded setup?  Or are you separately constructing a BlobServiceClient each time and working your way down to a BlockBlobClient from there?

@coding-jesus would you be able to give more details about your multithreaded setup?  Or are you separately constructing a BlobServiceClient each time and working your way down to a BlockBlobClient from there?

@jaschrep-msft I tried both and got the same results. I used a shared client object and initialized a new one for every call, and both resulted in the same error. I create multiple background jobs using Hangfire, and each is responsible for uploading a single file to Azure Blob Storage.

@jaschrep-msft I tried both and got the same results. I used a shared client object and initialized a new one for every call, and both resulted in the same error. I create multiple background jobs using Hangfire, and each is responsible for uploading a single file to Azure Blob Storage.

+@pakrym if this is only showing up with HttpWebRequestTransport in case he has any ideas what's going on

+@pakrym if this is only showing up with HttpWebRequestTransport in case he has any ideas what's going on

I'll take a look. Wonder if it's a side-effect of some of our timeout logic.

I'll take a look. Wonder if it's a side-effect of some of our timeout logic.

This issue would be fixed in the next Azure.Core release in the beginning of February. You can use steps from https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/core/Azure.Core/CHANGELOG.md#added-2 as a temporary work-around.

This issue would be fixed in the next Azure.Core release in the beginning of February. You can use steps from https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/core/Azure.Core/CHANGELOG.md#added-2 as a temporary work-around.

I've been hitting this issue when trying to download blobs in parallel.  Does this fix potentially also address downloads?

I don't think so. Can you please create a new issue with more details of your problem?


feat(clerk-js): Retheme org invite members form

Description
Add new design on org's invite member form.
Checklist

[ ] npm test runs as expected.
[ ] npm run build runs as expected.
[ ] (If applicable) JSDoc comments have been added or updated for any package exports
[ ] (If applicable) Documentation has been updated

Type of change

[ ] 🐛 Bug fix
[x] 🌟 New feature
[ ] 🔨 Breaking change
[ ] 📖 Refactoring / dependency upgrade / documentation
[ ] other:

Packages affected

[ ] @clerk/backend
[ ] @clerk/chrome-extension
[x] @clerk/clerk-js
[ ] @clerk/clerk-expo
[ ] @clerk/fastify
[ ] gatsby-plugin-clerk
[ ] @clerk/localizations
[ ] @clerk/nextjs
[ ] @clerk/clerk-react
[ ] @clerk/remix
[ ] @clerk/clerk-sdk-node
[ ] @clerk/shared
[ ] @clerk/themes
[ ] @clerk/types
[ ] build/tooling/chore

!preview

Hey @octoper, your preview is currently deploying and should be available shortly.

Status
Preview
Updated (UTC)

:cook: Deploying...
Inspect
Dec 19, 2023 10:29 PM

@panteliselef Thank you for reporting this. This is actually a bug that exists in the main branch and will be fixed right after we merge this PR. This PR is unrelated with the issue you mention.


How to best achieve window cycling and symmetric flipping ?

Hello, thank you very much for Yabai, it is truly an amazing tool! I recently started trying to implement the following two behaviours:
Cycling window layout within one space

and flipping the window layout symmetrically

I am uncertain if there is a straightforward method to achieve these 2 behaviours.

You'll want to send the following commands
yabai -m space --rotate 90
yabai -m space --mirror y-axis
yabai -m space --mirror x-axis

I have these in my skhd configuration
# rotate tree
ctrl + alt + cmd - r : yabai -m space --rotate 90

# mirror tree y-axis
ctrl + alt + cmd - y : yabai -m space --mirror y-axis

# mirror tree x-axis
ctrl + alt + cmd - x : yabai -m space --mirror x-axis

Thanks for the pointers @eramdam! I seemed to have overlooked the "spaces" section while trying to solve this only with "window" commands. For the symmetric flipping yabai -m space --mirror y-axis is exactly what I was looking for. However, the cycling behaviour is not the same as "rotate". As the former keeps the same layout and cycles the applications whereas the latter cycles the layout itself. I hope this figure illustrates my point:

In your specific case of 3 windows, you can use the following command to perform a cycling as you descibe, regardless of which window is currently focused. Note that this does not scale to layouts with more/less windows without modifications.
# cycle clockwise
yabai -m window first --swap next && yabai -m window first --swap last

# cycle counter-clockwise
yabai -m window first --swap last && yabai -m window first --swap next

Thank you very much @koekeishiya! The command works perfectly for my use case.

The following bash scripts should cycle windows in the given direction regardless of how many windows are in the layout.
cycle_clockwise.sh:
#!/bin/bash

win=$(yabai -m query --windows --window last | jq '.id')

while : ; do
    yabai -m window $win --swap prev &> /dev/null
    if [[ $? -eq 1 ]]; then
        break
    fi
done

cycle_counterclockwise.sh
#!/bin/bash

win=$(yabai -m query --windows --window first | jq '.id')

while : ; do
    yabai -m window $win --swap next &> /dev/null
    if [[ $? -eq 1 ]]; then
        break
    fi
done

If you would like to define the sort order different from the natural sort order of the bsp, you could use this script.
swaprotate_space_windows.sh
#!/bin/bash
[[ "$1" == "--reverse" ]] || reverse="| reverse "
yabai -m query --windows --space  |
jq -re '
map(select(.minimized != 1))
| sort_by(.display, .frame.x, .frame.y, ."stack-index", .id)
'"$reverse"'
| first as $first
| map( $first.id, .id )
| .[]' |
tail -n +3 |
xargs -n2 sh -c 'yabai -m window $1 --swap $2' sh
You can run this with --reverse to reverse your sort order and of course you can fiddle with the jq filters to get your desired order.

Hi,
I tried these two in my skhdrc and restarted yabai but it didn't work. Am I doing something incorrectly? No other shortcuts assigned to these keys
# cycle_clockwise cycle through windows
fn - c : /Users/username/.config/skhd/CustomScripts/cycle_clockwise.sh

# cycle clockwise
fn - s : yabai -m window first --swap next && yabai -m window first --swap last

thank you


City animation[GSSOC'23]

Title and Issue number

City animation
Issue #4252
Close #4252

Checklist:

[X] I have mentioned the issue number in my Pull Request.
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have created a helpful and easy to understand README.md

@MohitGupta121 @TusharKesarwani @pranjay-poddar please take a review of my PR & merge my PR.

@MohitGupta121 @TusharKesarwani @pranjay-poddar please take a review of my PR & merge my PR.


Debezium benchmark comparisons (to Kafka)

Sequin vs Debezium Benchmark Proposal
Objective
Compare the performance and resource utilization of Sequin against Debezium for streaming changes from PostgreSQL, focusing on end-to-end latency and throughput under various write patterns.
Infrastructure Setup
Source Database

AWS RDS PostgreSQL 15 (db.r6g.xlarge)
4 vCPUs, 32 GB RAM
General Purpose SSD (gp3)

CDC Processors
Two identical EC2 instances (c6g.xlarge):

4 vCPUs, 8 GB RAM
Ubuntu 22.04 LTS
Deployed via ECS

Instance 1:

Debezium Server (latest version)
Configured with Kafka Connect
Connected to AWS MSK (Kafka)

Instance 2:

Sequin (latest version)
Configured with identical resources
Connected to AWS MSK (Kafka)

Destination

AWS MSK (Kafka) cluster
Same configuration for both setups
Single topic per setup

Test Methodology
Data Generation

Create test table with mixed data types:

CREATE TABLE benchmark_records (
    id SERIAL PRIMARY KEY,
    string_field VARCHAR(100),
    numeric_field DECIMAL(10,2),
    timestamp_field TIMESTAMP,
    json_field JSONB
);

Test Patterns:

Bulk inserts (10K, 100K, 1M records)
Mixed workload (70% inserts, 20% updates, 10% deletes)
Rapid single-row updates (1K updates/sec to same row)

Metrics to Capture

End-to-end Latency

Time from DB commit to Kafka message delivery
P50, P95, P99 latencies

Throughput

Messages processed per second
Maximum sustainable throughput before degradation

Resource Utilization

CPU usage
Memory usage
Network I/O

Benchmark Process

Setup Phase

Deploy and configure both systems
Verify basic connectivity and processing
Warm up systems with small test loads

Test Execution

Run each test pattern sequentially
Allow system cool-down between tests
Capture metrics continuously
Run each test minimum 3 times

Monitoring

Use CloudWatch for resource metrics
Custom instrumentation for latency measurements
Log all error conditions and anomalies

import psycopg2
import random
import json
import time
import threading
from datetime import datetime
from concurrent.futures import ThreadPoolExecutor
import argparse
import statistics
from dataclasses import dataclass
from typing import List

@dataclass
class ThroughputStats:
    window_size: float  # seconds
    operations: List[float]  # operations per window

    @property
    def mean_throughput(self) -> float:
        return statistics.mean(self.operations) if self.operations else 0

    @property
    def p95_throughput(self) -> float:
        if not self.operations:
            return 0
        sorted_ops = sorted(self.operations)
        idx = int(len(sorted_ops) * 0.95)
        return sorted_ops[idx]

class LoadGenerator:
    def __init__(self, dsn: str, target_throughput: int):
        """
        Args:
            dsn: Postgres connection string
            target_throughput: Target operations per second
        """
        self.dsn = dsn
        self.target_throughput = target_throughput
        self.stats = ThroughputStats(window_size=1.0, operations=[])
        self.stop_event = threading.Event()

        # Create connection pool
        self.conn = psycopg2.connect(dsn)
        self.conn.autocommit = True

    def _generate_record(self) -> tuple:
        """Generate a single test record"""
        return (
            f"string-{random.randint(1, 1000000)}",
            random.uniform(1, 1000),
            datetime.now(),
            json.dumps({
                "key1": random.randint(1, 100),
                "key2": "value" + str(random.randint(1, 100)),
                "key3": random.choice([True, False])
            })
        )

    def _perform_insert(self, cur) -> None:
        """Perform single insert operation"""
        record = self._generate_record()
        cur.execute("""
            INSERT INTO benchmark_records
            (string_field, numeric_field, timestamp_field, json_field)
            VALUES (%s, %s, %s, %s)
            RETURNING id
        """, record)
        return cur.fetchone()[0]

    def _perform_update(self, cur, id_to_update: int) -> None:
        """Perform single update operation"""
        record = self._generate_record()
        cur.execute("""
            UPDATE benchmark_records
            SET string_field = %s,
                numeric_field = %s,
                timestamp_field = %s,
                json_field = %s
            WHERE id = %s
        """, (*record, id_to_update))

    def _perform_delete(self, cur, id_to_delete: int) -> None:
        """Perform single delete operation"""
        cur.execute("DELETE FROM benchmark_records WHERE id = %s", (id_to_delete,))

    def _operation_worker(self) -> None:
        """Worker thread that performs operations at the target rate"""
        with self.conn.cursor() as cur:
            while not self.stop_event.is_set():
                start_time = time.time()

                # Determine operation type based on distribution
                op_type = random.choices(
                    ['insert', 'update', 'delete'],
                    weights=[70, 20, 10]
                )[0]

                try:
                    if op_type == 'insert':
                        self._perform_insert(cur)
                    elif op_type == 'update':
                        # Get random existing ID
                        cur.execute("SELECT id FROM benchmark_records ORDER BY RANDOM() LIMIT 1")
                        result = cur.fetchone()
                        if result:
                            self._perform_update(cur, result[0])
                    else:  # delete
                        cur.execute("SELECT id FROM benchmark_records ORDER BY RANDOM() LIMIT 1")
                        result = cur.fetchone()
                        if result:
                            self._perform_delete(cur, result[0])

                except Exception as e:
                    print(f"Error performing {op_type}: {e}")
                    continue

                # Calculate sleep time to maintain target throughput
                elapsed = time.time() - start_time
                sleep_time = max(0, (1.0 / self.target_throughput) - elapsed)
                if sleep_time > 0:
                    time.sleep(sleep_time)

    def _stats_worker(self) -> None:
        """Worker thread that collects throughput statistics"""
        window_start = time.time()
        operations_in_window = 0

        while not self.stop_event.is_set():
            time.sleep(0.1)  # Check every 100ms

            current_time = time.time()
            if current_time - window_start >= self.stats.window_size:
                self.stats.operations.append(operations_in_window / self.stats.window_size)
                operations_in_window = 0
                window_start = current_time
            else:
                operations_in_window += 1

    def run(self, duration_seconds: int) -> ThroughputStats:
        """
        Run the load generator for specified duration

        Args:
            duration_seconds: How long to run the load generator

        Returns:
            ThroughputStats object with collected metrics
        """
        # Start worker threads
        workers = []
        with ThreadPoolExecutor(max_workers=self.target_throughput) as executor:
            # Start operation workers
            for _ in range(min(self.target_throughput, 20)):  # Cap at 20 threads
                workers.append(executor.submit(self._operation_worker))

            # Start stats collector
            workers.append(executor.submit(self._stats_worker))

            # Run for specified duration
            time.sleep(duration_seconds)
            self.stop_event.set()

        return self.stats

def main():
    parser = argparse.ArgumentParser(description='Generate mixed workload for Postgres')
    parser.add_argument('--dsn', required=True, help='Postgres connection string')
    parser.add_argument('--throughput', type=int, default=100, help='Target operations per second')
    parser.add_argument('--duration', type=int, default=60, help='Duration in seconds')
    args = parser.parse_args()

    # Initialize and run load generator
    generator = LoadGenerator(args.dsn, args.throughput)
    stats = generator.run(args.duration)

    # Print results
    print(f"\nLoad Generator Results:")
    print(f"Target throughput: {args.throughput} ops/sec")
    print(f"Actual mean throughput: {stats.mean_throughput:.2f} ops/sec")
    print(f"P95 throughput: {stats.p95_throughput:.2f} ops/sec")

if __name__ == "__main__":
    main()

Reading: https://debezium.io/blog/2020/02/25/lessons-learned-running-debezium-with-postgresql-on-rds/

We were observing a few data-types with base64 encoded data. As documented, that’s the default for NUMERIC columns, but can be difficult to handle for consumers. To convert to a format that’s easier to parse at the expense of some accuracy loss, we configured the data type specific properties as documented. Specifically, set decimal.handling.mode to string to receive NUMERIC, DECIMAL and equivalent types as a string (eg. "3.14") and set hstore.handling.mode to json to receive HSTORE columns as a JSON string.

Wondering if we need to provide these out of the box transforms for numerics and hstore columns?

We had not enabled heartbeats in Debezium. Heartbeats are needed to control the WAL disk space consumption in the following cases:

There are many updates in a monitored database but only a minuscule amount relates to the monitored table(s) and/or schema(s). We handled this case by enabling heartbeats by setting heartbeat.interval.ms.

The PostgreSQL instance contains multiple databases where the monitored database is low-traffic in comparison to the others. Since the WAL is shared by all databases in an instance it keeps accumulating data which cannot be removed until Debezium reads it. But since the high traffic databases are not monitored Debezium is not able to communicate to the database that the WAL files can be removed to reclaim disk space. To solve this scenario we triggered "heartbeat" events by periodically updating a single row in a table created in the monitored database using the below query:
CREATE TABLE IF NOT EXISTS heartbeat (id SERIAL PRIMARY KEY, ts TIMESTAMP WITH TIME ZONE);
INSERT INTO heartbeat (id, ts) VALUES (1, NOW()) ON CONFLICT(id) DO UPDATE SET ts=EXCLUDED.ts;

Since this is a common use-case that comes up when using Debezium with PostgreSQL, an issue has been created to track this at DBZ-1815.

this second issue requires you to touch records in your database periodically just for debezium to ack the replication slow

How do deletes with CASCADE DELETE statements work for Sequin?

I have a repo working end to end. Initial results:Debezium:
(venv) ➜  sequin-debezium-benchmark python consumer_stats.py
Stats saved - Avg Latency: 277.55ms, P50: 278.00ms, P95: 318.00ms, P99: 322.00ms, Throughput: 2000.00 msgs/sec
Stats saved - Avg Latency: 385.66ms, P50: 389.00ms, P95: 402.00ms, P99: 406.00ms, Throughput: 2000.00 msgs/sec
Stats saved - Avg Latency: 429.98ms, P50: 433.00ms, P95: 460.00ms, P99: 464.00ms, Throughput: 2000.00 msgs/sec
Stats saved - Avg Latency: 458.78ms, P50: 461.00ms, P95: 474.00ms, P99: 478.01ms, Throughput: 0.20 msgs/sec
Stats saved - Avg Latency: 309.92ms, P50: 384.00ms, P95: 472.00ms, P99: 506.00ms, Throughput: 1295.20 msgs/sec
Stats saved - Avg Latency: 236.37ms, P50: 252.00ms, P95: 446.00ms, P99: 510.00ms, Throughput: 1260.20 msgs/sec
Stats saved - Avg Latency: 237.80ms, P50: 248.00ms, P95: 499.00ms, P99: 510.00ms, Throughput: 1120.20 msgs/sec
Stats saved - Avg Latency: 248.23ms, P50: 250.50ms, P95: 496.00ms, P99: 509.00ms, Throughput: 1190.20 msgs/sec
Stats saved - Avg Latency: 252.93ms, P50: 254.00ms, P95: 496.00ms, P99: 507.00ms, Throughput: 1190.20 msgs/sec

Sequin:
(venv) ➜  sequin-debezium-benchmark python consumer_stats.py
Stats saved - Avg Latency: 2184.79ms, P50: 2216.00ms, P95: 2991.00ms, P99: 3033.00ms, Throughput: 772.60 msgs/sec
Stats saved - Avg Latency: 3852.34ms, P50: 3732.00ms, P95: 7073.00ms, P99: 7307.01ms, Throughput: 1245.40 msgs/sec
Stats saved - Avg Latency: 7461.13ms, P50: 7521.00ms, P95: 10410.00ms, P99: 10653.00ms, Throughput: 776.60 msgs/sec
Stats saved - Avg Latency: 11531.11ms, P50: 12287.00ms, P95: 14495.00ms, P99: 14786.00ms, Throughput: 1040.40 msgs/sec
Stats saved - Avg Latency: 16320.75ms, P50: 16722.00ms, P95: 19622.00ms, P99: 19786.01ms, Throughput: 1218.60 msgs/sec
Stats saved - Avg Latency: 20059.33ms, P50: 19622.00ms, P95: 23748.00ms, P99: 24003.00ms, Throughput: 897.00 msgs/sec
Stats saved - Avg Latency: 24313.69ms, P50: 24012.00ms, P95: 27819.05ms, P99: 28060.03ms, Throughput: 1045.00 msgs/sec
Stats saved - Avg Latency: 28923.93ms, P50: 28963.00ms, P95: 31692.05ms, P99: 32025.01ms, Throughput: 1289.20 msgs/sec
Stats saved - Avg Latency: 32709.37ms, P50: 32677.50ms, P95: 35587.05ms, P99: 35786.04ms, Throughput: 1205.60 msgs/sec
Stats saved - Avg Latency: 36489.87ms, P50: 36559.00ms, P95: 39240.35ms, P99: 39557.01ms, Throughput: 1224.60 msgs/sec
Stats saved - Avg Latency: 40004.52ms, P50: 40056.00ms, P95: 42456.00ms, P99: 42702.00ms, Throughput: 1136.00 msgs/sec
Stats saved - Avg Latency: 44187.44ms, P50: 44247.00ms, P95: 47032.00ms, P99: 47226.00ms, Throughput: 1296.60 msgs/sec
kStats saved - Avg Latency: 48820.14ms, P50: 48961.00ms, P95: 50559.05ms, P99: 50660.00ms, Throughput: 1624.80 msgs/sec
Stats saved - Avg Latency: 51636.76ms, P50: 51213.00ms, P95: 53939.00ms, P99: 54149.00ms, Throughput: 894.60 msgs/sec
Stats saved - Avg Latency: 54689.10ms, P50: 55243.00ms, P95: 56919.00ms, P99: 57095.00ms, Throughput: 1067.40 msgs/sec
Stats saved - Avg Latency: 59627.28ms, P50: 59652.50ms, P95: 61853.00ms, P99: 62058.01ms, Throughput: 1573.60 msgs/sec
Stats saved - Avg Latency: 62828.84ms, P50: 62504.00ms, P95: 65701.05ms, P99: 65908.01ms, Throughput: 1246.80 msgs/sec
Stats saved - Avg Latency: 65780.61ms, P50: 65811.00ms, P95: 68553.05ms, P99: 68837.06ms, Throughput: 896.40 msgs/sec
Stats saved - Avg Latency: 70889.58ms, P50: 70952.00ms, P95: 72641.15ms, P99: 72868.04ms, Throughput: 1239.20 msgs/sec
Stats saved - Avg Latency: 74012.58ms, P50: 74207.00ms, P95: 77735.00ms, P99: 77961.00ms, Throughput: 1072.40 msgs/sec
Stats saved - Avg Latency: 78788.98ms, P50: 78469.00ms, P95: 80900.00ms, P99: 81015.00ms, Throughput: 1503.20 msgs/sec

Current major issue is that Sequin does not include a timestamp for when the message was delivered to Kafka. Debezium does.So for Sequin we use the timestamp of when the message was consumed from Kafka instead.So calculating the latency for debezium is more accurate for delivery.

We can add to the metadata?
On Mon, Dec 02, 2024 at 3:31 PM, Carter Pedersen < @.*** > wrote:

I have a repo working end to end. Initial results:
Debezium:
(venv) ➜  sequin-debezium-benchmark python consumer_stats. py (
http://consumer_stats.py/ )
Stats saved - Avg Latency: 277.55ms, P50:
278.00ms, P95: 318.00ms, P99: 322.00ms, Throughput: 2000.00 msgs/sec
Stats
saved - Avg Latency: 385.66ms, P50: 389.00ms, P95: 402.00ms, P99:
406.00ms, Throughput: 2000.00 msgs/sec
Stats saved - Avg Latency:
429.98ms, P50: 433.00ms, P95: 460.00ms, P99: 464.00ms, Throughput: 2000.00
msgs/sec
Stats saved - Avg Latency: 458.78ms, P50: 461.00ms, P95:
474.00ms, P99: 478.01ms, Throughput: 0.20 msgs/sec
Stats saved - Avg
Latency: 309.92ms, P50: 384.00ms, P95: 472.00ms, P99: 506.00ms,
Throughput: 1295.20 msgs/sec
Stats saved - Avg Latency: 236.37ms, P50:
252.00ms, P95: 446.00ms, P99: 510.00ms, Throughput: 1260.20 msgs/sec
Stats
saved - Avg Latency: 237.80ms, P50: 248.00ms, P95: 499.00ms, P99:
510.00ms, Throughput: 1120.20 msgs/sec
Stats saved - Avg Latency:
248.23ms, P50: 250.50ms, P95: 496.00ms, P99: 509.00ms, Throughput: 1190.20
msgs/sec
Stats saved - Avg Latency: 252.93ms, P50: 254.00ms, P95:
496.00ms, P99: 507.00ms, Throughput: 1190.20 msgs/sec
Sequin:
(venv) ➜  sequin-debezium-benchmark python consumer_stats. py (
http://consumer_stats.py/ )
Stats saved - Avg Latency: 2184.79ms, P50:
2216.00ms, P95: 2991.00ms, P99: 3033.00ms, Throughput: 772.60 msgs/sec
Stats saved - Avg Latency: 3852.34ms, P50: 3732.00ms, P95: 7073.00ms, P99:
7307.01ms, Throughput: 1245.40 msgs/sec
Stats saved - Avg Latency:
7461.13ms, P50: 7521.00ms, P95: 10410.00ms, P99: 10653.00ms, Throughput:
776.60 msgs/sec
Stats saved - Avg Latency: 11531.11ms, P50: 12287.00ms,
P95: 14495.00ms, P99: 14786.00ms, Throughput: 1040.40 msgs/sec
Stats saved

Avg Latency: 16320.75ms, P50: 16722.00ms, P95: 19622.00ms, P99:
19786.01ms, Throughput: 1218.60 msgs/sec
Stats saved - Avg Latency:
20059.33ms, P50: 19622.00ms, P95: 23748.00ms, P99: 24003.00ms, Throughput:
897.00 msgs/sec
Stats saved - Avg Latency: 24313.69ms, P50: 24012.00ms,
P95: 27819.05ms, P99: 28060.03ms, Throughput: 1045.00 msgs/sec
Stats saved
Avg Latency: 28923.93ms, P50: 28963.00ms, P95: 31692.05ms, P99:
32025.01ms, Throughput: 1289.20 msgs/sec
Stats saved - Avg Latency:
32709.37ms, P50: 32677.50ms, P95: 35587.05ms, P99: 35786.04ms, Throughput:
1205.60 msgs/sec
Stats saved - Avg Latency: 36489.87ms, P50: 36559.00ms,
P95: 39240.35ms, P99: 39557.01ms, Throughput: 1224.60 msgs/sec
Stats saved
Avg Latency: 40004.52ms, P50: 40056.00ms, P95: 42456.00ms, P99:
42702.00ms, Throughput: 1136.00 msgs/sec
Stats saved - Avg Latency:
44187.44ms, P50: 44247.00ms, P95: 47032.00ms, P99: 47226.00ms, Throughput:
1296.60 msgs/sec
kStats saved - Avg Latency: 48820.14ms, P50: 48961.00ms,
P95: 50559.05ms, P99: 50660.00ms, Throughput: 1624.80 msgs/sec
Stats saved
Avg Latency: 51636.76ms, P50: 51213.00ms, P95: 53939.00ms, P99:
54149.00ms, Throughput: 894.60 msgs/sec
Stats saved - Avg Latency:
54689.10ms, P50: 55243.00ms, P95: 56919.00ms, P99: 57095.00ms, Throughput:
1067.40 msgs/sec
Stats saved - Avg Latency: 59627.28ms, P50: 59652.50ms,
P95: 61853.00ms, P99: 62058.01ms, Throughput: 1573.60 msgs/sec
Stats saved
Avg Latency: 62828.84ms, P50: 62504.00ms, P95: 65701.05ms, P99:
65908.01ms, Throughput: 1246.80 msgs/sec
Stats saved - Avg Latency:
65780.61ms, P50: 65811.00ms, P95: 68553.05ms, P99: 68837.06ms, Throughput:
896.40 msgs/sec
Stats saved - Avg Latency: 70889.58ms, P50: 70952.00ms,
P95: 72641.15ms, P99: 72868.04ms, Throughput: 1239.20 msgs/sec
Stats saved
Avg Latency: 74012.58ms, P50: 74207.00ms, P95: 77735.00ms, P99:
77961.00ms, Throughput: 1072.40 msgs/sec
Stats saved - Avg Latency:
78788.98ms, P50: 78469.00ms, P95: 80900.00ms, P99: 81015.00ms, Throughput:
1503.20 msgs/sec

Current major issue is that Sequin does not include a timestamp for when
the message was delivered to Kafka. Debezium does.
So for Sequin we use the timestamp of when the message was consumed from
Kafka instead.
So calculating the latency for debezium is more accurate for delivery.
—
Reply to this email directly, view it on GitHub (
https://github.com/sequinstream/sequin/issues/651#issuecomment-2513192594
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AAHCKNODEZLXIZL53RMECL32DTUVHAVCNFSM6AAAAABSIFJMNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJTGE4TENJZGQ
).
You are receiving this because you authored the thread. Message ID: <sequinstream/sequin/issues/651/2513192594
@ github. com>

Debezium:
(venv) ➜  sequin-debezium-benchmark git:(main) ✗ python consumer_stats.py
Stats saved - Avg Latency: 240.28ms, P50: 233.00ms, P95: 474.00ms, P99: 497.00ms, Throughput: 2445.60 msgs/sec
Stats saved - Avg Latency: 234.09ms, P50: 188.00ms, P95: 439.00ms, P99: 461.00ms, Throughput: 2467.64 msgs/sec
Stats saved - Avg Latency: 218.15ms, P50: 165.00ms, P95: 438.00ms, P99: 448.00ms, Throughput: 2456.37 msgs/sec
Stats saved - Avg Latency: 232.10ms, P50: 244.00ms, P95: 447.00ms, P99: 502.00ms, Throughput: 2463.45 msgs/sec
Stats saved - Avg Latency: 242.79ms, P50: 263.00ms, P95: 460.00ms, P99: 495.00ms, Throughput: 2455.26 msgs/sec
Stats saved - Avg Latency: 241.78ms, P50: 237.00ms, P95: 471.00ms, P99: 496.00ms, Throughput: 2195.72 msgs/sec

Sequin:
(venv) ➜  sequin-debezium-benchmark git:(main) ✗ python consumer_stats.py
Stats saved - Avg Latency: 162.51ms, P50: 162.00ms, P95: 232.00ms, P99: 249.01ms, Throughput: 2892.83 msgs/sec, Avg Replication Latency: 54.62ms, Avg Delivery Latency: 109.01ms
Stats saved - Avg Latency: 148.43ms, P50: 148.00ms, P95: 218.00ms, P99: 250.00ms, Throughput: 2877.37 msgs/sec, Avg Replication Latency: 56.12ms, Avg Delivery Latency: 100.54ms
Stats saved - Avg Latency: 148.39ms, P50: 149.00ms, P95: 225.00ms, P99: 250.00ms, Throughput: 2830.50 msgs/sec, Avg Replication Latency: 54.89ms, Avg Delivery Latency: 95.84ms
Stats saved - Avg Latency: 150.34ms, P50: 149.00ms, P95: 224.00ms, P99: 249.00ms, Throughput: 2862.78 msgs/sec, Avg Replication Latency: 55.67ms, Avg Delivery Latency: 90.94ms
Stats saved - Avg Latency: 151.84ms, P50: 152.00ms, P95: 215.00ms, P99: 237.00ms, Throughput: 2749.33 msgs/sec, Avg Replication Latency: 55.12ms, Avg Delivery Latency: 90.47ms
Stats saved - Avg Latency: 376.25ms, P50: 370.00ms, P95: 535.00ms, P99: 585.00ms, Throughput: 2585.12 msgs/sec, Avg Replication Latency: 52.80ms, Avg Delivery Latency: 226.00ms

much improved measurements, in fact we are beating Debezium 🙌 I added some timestamps to consumer event metadata and track the replication and delivery latency in addition to total latency stats in the kafka consumer.I did notice, however, that CPU usage during debezium testing was ~15% and during Sequin testing was ~70%. so further investigation is needed. this might be postgres so this is best investigated in the cloud setup.

We will implement TableReaders which is in progress in SQN-3471
Then return to ConsumerBroker architecture, related to this memo:
Dec 17, 2024: ConsumerBroker
This will improve write performance up to ~8k msg/sec change data capture, potentially higher throughput. Additionally this will improve CDC latency to sub 100 ms (depending on latency between Sequin <> source Postgres database and Sequin <> destination (Kafka, Redis, etc.). See:

https://github.com/sequinstream/sequin/pull/763
https://discord.com/channels/1294098062049804329/1319000739237855287

This should also improve throughput of TableReaderServer operations.


test: fix env secrets for bot users

Already covered by this PR: https://github.com/SAP-samples/cf-routing-samples/pull/31