sys/log: Add persistent log index

It may be useful to keep log index even if after factory reset.
On startup:
A. (As usual) Firmware scans all persistent logs to determine the next available index.
B. Firmware reads the persistent_index value from flash, if available.
C. The next_index RAM variable is set to the greater of these two values.
On each log write:

Compare the next_index variable to persistent_index.
If the difference is less than 500, schedule an event to write a new persistent_index to flash.
The new persistent_index is equal to: next_index + 1000.

Note 1: the numbers 500 and 1000 were chosen arbitrarily.
Note 2: In step 2, the new persistent_index is written asynchronously because the logging task might not tolerate the time required to write to flash (e.g., ble_ll task).

I will take a closer look at the code, but for now I just have a high level comment.
Now that we support separate log indices per log (!MYNEWT_VAL(LOG_GLOBAL_INDEX)), I think much of the motivation for this feature is gone.  Prior to the separate index feature, there was a fear that the same index could be used by two different persistent logs.  For example:

Persistent log writes entry with index=1.
Console log writes entry with index=2.
Device reboots.
Persistent log writes entry with index=2.

If each log can have its own running index, this scenario can't happen with persistent logs.
Non-persistent logs (console, cbmem) continue to reuse indices, even if the separate index feature is enabled.  Do we care about this?  Ideally there would be no reuse by any logs whatsoever, but I wonder if reuse among transient logs is a big enough issue to warrant this feature.
If we don't care about reuse among transient logs, then all we care about is remembering the most-recent index when logs get erased (factory reset).  In this case, this feature can be simplified quite a bit:

No need to schedule an index-persist-event at write-time.  Just persist an index upon factory reset.
No need to persist a seperate index per log.  Just persist the greatest index+1 to a single config setting.  Or just write a dummy entry to each persistent log after the erase.

On the other hand, if we want to prevent reuse among transient logs, I do think we need the more complicated implementation.
I don't know the answer.  I'm just asking because it seems a shame to add such a messy feature for something that has never been an issue in practice (by messy I mean the feature itself, not the implementation :)).


How to setup React-Native 0.15.0 with Jest 0.8.0

My app and tests were running fine w/ older versions of RN (0.14.2) & Jest (0.5.4) and babel-core. I am trying to upgrade...
Is there a working example of testing React-Native 0.15.0 JSX components  using Jest 0.8.0?
I followed the Babel Jest instructions from here: (http://babeljs.io/docs/setup/#jest)[http://babeljs.io/docs/setup/#jest]
When I run a JSX test : NODE_ENV=test jest "src/components/__tests__/FormButton-test.js"
I get "Unexpected reserved word"

Unexpected reserved word

Using Jest CLI v0.8.0, jasmine1
 FAIL  src/components/__tests__/FormButton-test.js
● Runtime Error
SyntaxError: /Users/barton/projects/MFV/MFV-mobile/src/components/__tests__/FormButton-test.js: /Users/barton/projects/MFV/MFV-mobile/src/components/FormButton.js: /Users/barton/projects/MFV/MFV-mobile/node_modules/apsl-react-native-button/Button.js: Unexpected reserved word
npm ERR! Test failed.  See above for more details.

which word is reserved?
When I try to compile in iOS I get:

Unknown option .babelrec.presets

[BABEL] /Users/barton/projects/MFV/MFV-mobile/node_modules/react-native/packager/react-packager/src/Resolver/polyfills/prelude_dev.js: Unknown option: /Users/barton/projects/MFV/MFV-mobile/.babelrc.presets

Also, when I run my "non JSX" tests I have to add ".default" on the require:

test snippet fails:

jest.dontMock('../authReducer');
var  authReducer = require('../authReducer');
console.log('authReducer',authReducer);

console.log

Using Jest CLI v0.8.0, jasmine1
Running 1 test suite...authReducer { default: [Function: authReducer] }
 FAIL  src/reducers/auth/__tests__/authReducer-test.js (2.544s)
● authReducer › SESSION-TOKEN-REQUEST › it starts fetching
  - TypeError: authReducer is not a function
        at Spec.<anonymous> (src/reducers/auth/__tests__/authReducer-test.js:33:18)
1 test failed, 14 tests passed (15 total in 1 test suite, run time 3.144s)
npm ERR! Test failed.  See above for more details.

But this works:

jest.dontMock('../authReducer');
var  authReducer = require('../authReducer').default;
console.log('authReducer',authReducer);

output

~/projects/MFV/MFV-mobile (feature/actions)$ npm test src/reducers/auth/__tests__/authReducer-test.js

><EMAIL_ADDRESS>test /Users/barton/projects/MFV/MFV-mobile
> NODE_ENV=test jest "src/reducers/auth/__tests__/authReducer-test.js"

Using Jest CLI v0.8.0, jasmine1
Running 1 test suite...
        authReducer function authReducer() {var state = arguments.length <= 0 || arguments[0] === undefined ? initialState : arguments[0];var action = arguments[1];
  if (!(state instanceof InitialState)) return initialState.mergeDeep(state);
  switch (action.type) {
  ....removed for brevity
  return state;}
 PASS  src/reducers/auth/__tests__/authReducer-test.js (2.595s)
15 tests passed (15 total in 1 test suite, run time 3.229s)

Here is my

.babelrc

{
  "presets": [
    "es2015",
    "react"
  ],
  "plugins": [
    "transform-class-properties",
    "transform-decorators"
  ]
}

Here is my

package.json:

{
  "name": "MFV-mobile",
  "version": "0.0.1",
  "private": true,
  "jest": {
    "scriptPreprocessor": "./node_modules/babel-jest",
    "setupEnvScriptFile": "./node_modules/react-native/jestSupport/env.js",
    "testPathIgnorePatterns": [
      "/node_modules/"
    ],
    "moduleFileExtensions": [
      "js",
      "json",
      "es6"
    ],
    "testFileExtensions": [
      "es6",
      "js"
    ],
    "unmockedModulePathPatterns": [
      "promise",
      "source-map",
      "key-mirror",
      "immutable",
      "fetch",
      "react",
      "redux",
      "redux-thunk",
      "src/"
    ]
  },
  "scripts": {
    "test": "NODE_ENV=test jest",
    "start": "react-native start",
    "test:watch": "npm run test -- --watch",
    "test-chrome": "node-debug --preload false --nodejs --harmony ./node_modules/.bin/jest --runInBand"
  },
  "dependencies": {
    "apsl-react-native-button": "git@github.com:bartonhammond/react-native-button.git",
    "immutable": "^3.7.5",
    "key-mirror": "^1.0.1",
    "react-native": "^0.15.0",
    "react-native-gifted-spinner": "0.0.3",
    "react-native-simple-store": "^0.1.0",
    "react-native-simpledialog-android": "^1.0.2",
    "react-native-tab-navigator": "^0.2.11",
    "react-native-vector-icons": "^0.8.5",
    "react-redux": "^3.1.0",
    "redux": "^3.0.4",
    "redux-thunk": "^1.0.0",
    "regenerator": "^0.8.42",
    "tcomb-form-native": "^0.3.0",
    "underscore": "^1.8.3",
    "validate.js": "^0.9.0"
  },
  "devDependencies": {
    "babel-jest": "*",
    "babel-plugin-transform-class-properties": "^6.3.0",
    "babel-plugin-transform-decorators": "^6.1.18",
    "babel-preset-es2015": "^6.1.18",
    "babel-preset-react": "^6.1.18",
    "jest-cli": "^0.8.0",
    "react": "^0.14.3",
    "react-addons-test-utils": "^0.14.3"
  }
}

Similar problem on 0.16:
Using Jest CLI v0.8.1, jasmine1
 FAIL  App/Models/__tests__/HealthCheck-test.js (0.111s)
● HealthCheck › toFirebase › it serialize the instance data to an object
  - TypeError: _HealthCheck2.default is not a function
        at Spec.<anonymous> (App/Models/__tests__/HealthCheck-test.js:9:14)
1 test failed, 0 tests passed (1 total in 1 test suite, run time 0.775s)

Are you also on Babel 6?

No - I backed off to 0.15.0 till the smoke clears...

Did you resolve your issue? It seems like the starter kit now has 0.15 in it?

I'm looking at this again now.  I am attempting to use RN  0.17
Having problems but not clear what the issue is.
Followed Setup from http://facebook.github.io/jest/docs/tutorial-react.html#content
package.json:
{
  "name": "snowflake",
  "version": "0.0.6",
  "private": true,
  "jest": {
    "scriptPreprocessor": "./node_modules/babel-jest",
    "setupEnvScriptFile": "./node_modules/react-native/jestSupport/env.js",
    "testPathIgnorePatterns": [
      "node_modules/"
    ],
    "testFileExtensions": [
      "js"
    ],
    "unmockedModulePathPatterns": [
      "promise",
      "source-map",
      "./node_modules/key-mirror",
      "./node_modules/immutable",
      "fetch",
      "./node_modules/react",
      "./node_modules/redux",
      "./node_modules/redux-thunk",
      "fbjs"
    ],
    "verbose": true,
    "testPathDirs": [
      "src/"
    ],
    "collectCoverage": false
  },
  "scripts": {
    "test": "jest",
    "start": "react-native start",
    "test:watch": "npm run test -- --watch",
    "test-chrome": "node-debug --preload false --nodejs --harmony jest --runInBand",
    "docs": "./node_modules/docker/docker -w -i src/ -o ../snowflake-pages "
  },
  "dependencies": {
    "apsl-react-native-button"<EMAIL_ADDRESS>    "immutable": "^3.7.5",
    "key-mirror": "^1.0.1",
    "react-native": "^0.17.0",
    "react-native-gifted-spinner": "0.0.3",
    "react-native-simple-store": "^0.1.0",
    "react-native-simpledialog-android": "^1.0.2",
    "react-native-tab-navigator": "^0.2.11",
    "react-native-vector-icons": "^0.8.5",
    "react-redux": "^3.1.0",
    "redux": "^3.0.4",
    "redux-thunk": "^1.0.0",
    "regenerator": "^0.8.42",
    "tcomb-form-native": "^0.3.0",
    "underscore": "^1.8.3",
    "validate.js": "^0.9.0"
  },
  "devDependencies": {
    "babel-core": "~5.8.34",
    "babel-jest": "^6.0.1",
    "docker": "^0.2.14",
    "istanbul": "gotwarlost/istanbul#source-map",
    "jest-cli": "^0.8.2",
    "react": "^0.14.3",
    "react-addons-test-utils": "^0.14.3"
  }
}

~/projects/MFV/snowflake (feature/upgrade)$ npm test
><EMAIL_ADDRESS>test /Users/barton/projects/MFV/snowflake
> jest
Using Jest CLI v0.8.2, jasmine1
 FAIL  src/components/__tests__/ErrorAlert-test.js
 FAIL  src/components/__tests__/FormButton-test.js
 FAIL  src/reducers/profile/__tests__/profileActions-test.js
 FAIL  src/reducers/auth/__tests__/authReducer-test.js
 FAIL  src/reducers/auth/__tests__/authActions-test.js
 FAIL  src/components/__tests__/LoginForm-test.js
 FAIL  src/components/__tests__/Header-test.js
 FAIL  src/components/__tests__/ItemCheckbox-test.js
 FAIL  src/reducers/profile/__tests__/profileReducer-test.js
npm ERR! Test failed.  See above for more details.

Repeating from babel-jest issue...
Within debugger, stepping through the following require statement
/**
 * ## Under test
 * Object under test
 */
jest.dontMock('../ErrorAlert');
var ErrorAlert = require('../ErrorAlert'); //<<<<

I see it blow up in
node_modules/jest-cli/src/environments/JSDOMEnvironment.js

at  line 36:
  runSourceText(sourceText, filename) {
    return this.global.eval(sourceText + '\n//# sourceURL=' + filename); // <- line 36
  }

The file eval'd is https://github.com/bartonhammond/snowflake/blob/master/src/components/ErrorAlert.js

Whittling down the class under test to be  a empty class, if import SimpleAlert is commented out the class evaluates in JSDOMEnviornment
import SimpleAlert from 'react-native-simpledialog-android'; // <- causes error

var ErrorAlert = class ErrorAlertClass{
 ///empty for debugging
};
module.exports = ErrorAlert;

Note, SimpleAlert is mocked:
https://github.com/bartonhammond/snowflake/blob/master/src/mocks/react-native-simpledialog-android.js
'use strict';
var SimpleAlert = {};
SimpleAlert.alert = jest.genMockFunction();
module.exports = SimpleAlert;

I have my 58 non-JSX tests working now.  My JSX tests fail in node_modules/jest-cli/src/environments/JSDOMEnvironment.js  doing the eval.
This is working for non-JSX:
package.json requirements
  "jest": {
    "scriptPreprocessor": "./node_modules/babel-jest",
    "setupEnvScriptFile": "./node_modules/react-native/jestSupport/env.js",
   ....
 "scripts": {
    "test": "rm -rf ./node_modules/jest-cli/.haste_cache && jest ",
    "start": "react-native start",
    "test:watch": "npm run test -- --watch",
    "test-chrome": "node-debug --preload false --nodejs --harmony jest --runInBand",
  ...
 "devDependencies": {
    "babel-jest": "^6.0.1",
    "babel-plugin-syntax-async-functions": "^6.3.13",
    "babel-plugin-transform-async-to-generator": "^6.3.13",
    "babel-plugin-transform-regenerator": "^6.3.18",
    "babel-preset-es2015": "^6.3.13",
    "babel-preset-react": "^6.3.13",
    "istanbul": "gotwarlost/istanbul#source-map",
    "jest-cli": "^0.8.2",
    "react-addons-test-utils": "^0.14.3"
  }

with ~/.babelrc:
{
  "presets": ["react", "es2015"],
  "plugins": [
    "syntax-async-functions",
    "transform-regenerator",
    "transform-async-to-generator"
  ]
}

Then in code, for any module that is exported "default" you have to now reference the import as:
require('../../lib/constants').default;

For reference https://github.com/bartonhammond/snowflake/tree/feature/upgrade

I mostly faced issues with modules that import other modules.
Ex: Module A instantiate B, the import B from ... will fail because of the babel translation mangling the import.

@masom what was the error message?   How did you address it?  Are you doing JSX Jest tests?
My error with JSX Jest is: SyntaxError: Unexpected token { form the eval in JSDOMEnvironment.js.   I wrote out all the files that get eval'd and can't find a single one that has invalid }.

I tried eval-ing the file causing the error and  found the line causing the error.  I guess it's another Babel setup thing? - but destructuring is part of ES2015 and I've included that as "devDependancy" and also in my .babelrc
var React = require('react-native');
var {  //<------ unexpected {
  Text
} = React;

What am I missing?

@bartonhammond I was having issues with destructuring in a React Native project I'm working in the other day and the suggestion of using babel-preset-react-native seemed to fix the destructuring problem for me (#700).
FWIW

this should work now.


More fixes

This change is

Current dependencies on/for this PR:

main

PR #3744

PR #3745   👈

This comment was auto-generated by Graphite.


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_d2llsvlf2q/clone/recipe/meta.yaml:15: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_d2llsvlf2q/clone/recipe/meta.yaml:25: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_d2llsvlf2q/clone/recipe/meta.yaml:43: missing_dev_url: The recipe is missing a dev_url
ERROR: /tmp/abs_d2llsvlf2q/clone/recipe/meta.yaml:43: missing_license_family: The recipe is missing the about/license_family key.
WARNING: /tmp/abs_d2llsvlf2q/clone/recipe/meta.yaml:43: missing_description: The recipe is missing a description
ERROR: /tmp/abs_d2llsvlf2q/clone/recipe/meta.yaml:43: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found


The instructions for adding the trial subscription are incorrect (at least for educational organizations)

The directions for adding the trial subscription were incorrect for my tenant (an educational organization). Instead of finding a trial license when searching for Teams Premium, I instead had to go to the details of the paid Teams Premium for Faculty license and then there was a "Start free trial" link next to the Buy button.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 89daab79-5c8a-4693-2218-50027e1249b2
Version Independent ID: 89daab79-5c8a-4693-2218-50027e1249b2
Content: Roll out a Teams Premium trial - Microsoft Teams
Content Source: Teams/teams-premium-trial.md
Service: msteams
GitHub Login: @MikePlumleyMSFT
Microsoft Alias: mikeplum

Hello @Eric-SeattleU
I was able to find Teams Premium for Faculty Trial without the need to first go to the  paid version.
Please let me know if I misunderstood.
Thanks for your feedback!

We appreciate any feedback that improves the content in the Microsoft docs but we haven't had any response from you so we are going to close this issue.
Please feel free to re-open this issue if there is a specific area of the docs that we can improve or make better. Thank you.


gnutls_handshake() failed: The server name sent was not recognized for https://pmc-geofence.trafficmanager.net/debian/9/prod/pool/

Describe the issue
https://pmc-geofence.trafficmanager.net/debian/9/prod/pool/ appears to be down
#6 3.894   gnutls_handshake() failed: The server name sent was not recognized
12:09:01   #6 3.900 Fetched 1519 kB in 1s (1469 kB/s)
12:09:01   #6 3.901 E: Failed to fetch https://pmc-geofence.trafficmanager.net/debian/9/prod/pool/main/m/mssql-tools/mssql-tools_<IP_ADDRESS>-1_amd64.deb?geofence=true  gnutls_handshake() failed: The server name sent was not recognized
12:09:01   #6 3.901 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
When did the issue occur?
September 11 10:30 AM
If applicable, what package did you attempt to install, and from which repo?
mssql-tools/
Repo: https://packages.microsoft.com/debian/9/prod/pool/main/m/mssql-tools/
Version: <IP_ADDRESS>.1
Steps to Reproduce
Just browse in the repo
https://packages.microsoft.com/debian/9/prod/pool/main/m/mssql-tools/
Actual Result
The webpage at https://pmc-geofence.trafficmanager.net/debian/9/prod/pool/main/m/mssql-tools/mssql-tools_<IP_ADDRESS>-1_amd64.deb?geofence=true might be temporarily down or it may have moved permanently to a new web address.
Expected Result
download the package
Screenshots

Additional context

Going to close this in favor of https://github.com/microsoft/linux-package-repositories/issues/170.


Suggest: Optimise Input Parameters if we can get generic type information in test/demo/HttpDemoUtil.java

Is there a way to get generic type information that bypasses Java Type Erasure ?
I tried many different ways to get generic type information , but i failed.
Expect someone to help .

I tried many different ways to get generic type information , but i failed.
Expect someone to help .

It feels so hard

I don't know

hhhhhh


timeline selection persists on dataset change

Haha looks funny ;)

This bug is obsolete, however this one is still here:

Done with last PR.


Remove extraneous semicolon

refs idaholab/moose#26878

Job VTB Documentation, step Sync Docs on 86e7358 wanted to post the following:
View the site here
This comment will be updated on new commits.


[roles-0.1.0] Asset Tooltip z-index

z-index issue with tooltip and Select Asset dropdown
Originally posted by @nicolaus-sherrill in https://github.com/decentdao/decent-interface/issues/2165#issuecomment-2245998282

Closed via https://github.com/decentdao/decent-interface/pull/2216


Version 3.0.0 Major Release

TODO: https://github.com/remarkjs/remark/tree/master/packages/remark-cli

:tada: This PR is included in version 3.0.0 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


Improve UnorderedListWebView so it can be used with the ListLabelingC…

…ommand.

Coverage increased (+0.007%) to 98.229% when pulling 994f99ced5b9f4c75cf56a76e908633af3d516b8 on list_labeling into 812d804defb23b568afc28c2ba568abba25fe54c on master.


Building shared libraries

It would be nice to be able to build a .so with a single make shared or something similar.

Manual steps (for Linux) are ready:
k4rtik@DevDebian:~/Projects/cpslib$ gcc -shared -fpic -o libpslib.so pslib_linux.c common.c
k4rtik@DevDebian:~/Projects/cpslib$ gcc -o driver driver.c -L. -lpslib
k4rtik@DevDebian:~/Projects/cpslib$ export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
k4rtik@DevDebian:~/Projects/cpslib$ ./driver | head
 Disk usage
total:<PHONE_NUMBER>d
used:<PHONE_NUMBER>d
free:<PHONE_NUMBER>d
percent: 25.535572

 Physical Disk partitions
Partitions : 1
/dev/sda1 / ext4 rw,relatime,errors=remount-ro,data=ordered

k4rtik@DevDebian:~/Projects/cpslib$

@nibrahim would you like to verify where this suffices for #22? I will have to figure out how to integrate these steps portably into our build system.

More than enough! Great work. :)
--
Cordially,
Noufal
http://nibrahim.net.in

If you are using these steps, you may also want to include -Wall -Werror while building the so.


Close connections when unloading

As mentioned in #35 - plugin should close telnet and websocket ports when un/reloading.

Should be better behaving now with commit 4ea1b97

Is this fixed?

Should be fixed


Make {{ devices }} list optional

Define it as empty list by default, remove unneccessary 'default([])' checks
Signed-off-by: Eduard Egorov<EMAIL_ADDRESS>

Thanks for this PR, can you also do the same for raw_journal_devices? Thanks!

Sure, I will. Thanks!

@eduardegorov any updates?

@leseb , sorry! If it is still actual, please pull.
Note: this variable is also defined in roles/ceph-common/defaults/main.yml:
##################
# Temporary Vars #
##################
# NOTE(SamYaple): These vars are set here to they are defined before use. They
# should be removed after a refactor has properly seperated all the checks into
# the appropriate roles.

journal_collocation: False
raw_multi_journal: False
osd_directory: False
bluestore: False
dmcrypt_journal_collocation: False
dmcrypt_dedicated_journal: False
raw_journal_devices: {}                       <--- HERE

osd_auto_discovery: False

The problem is I don't know what is the difference between {} and [] but looks like it's the same.. Let me know if any updates are needed.

Hey {} is a dictionary where [] is an array

test this please

One last thing, please update roles/ceph-common/defaults/main.yml and run generate_group_vars_sample.sh. Thanks!

Fixed, thank you, Sebastien! I had already updated roles/ceph-common/defaults/main.yml and run generate_group_vars_sample.sh in the 3rd commit - am I missing something there?

Hum I'm just not sure why length > 1 didn't work and == 0 seems to work.

test this please

test this please

test this please

test this please


[BUG] Service Bus reconnection not behaving as expected

Library name and version
Azure.Messaging.ServiceBus (7.6.0)
Describe the bug
We have noticed we sometimes get exceptions from Service Bus that look something like the following
 ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:<queue-name-here>$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:<queue-name-here>, Timestamp:2022-05-28T07:21:30 Reference:3bc9b3fb-f97a-4f20-8106-68e709da3b08, TrackingId:1685d478-88e8-4bfd-b2a6-7093948b2e31_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:21:30 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

I did some light searching and couldn't seem to find any information on the most meaningful parts of the error. Once we see this error it seems the client gets into a state where it will no longer send any messages, and we usually have to restart the service. Even if we leave it for hours, it never seems to reconnect.
The documentation for Service Bus seems to suggest that reconnection should be handled automatically by the library so we were unsure what the recommended way of handling this occurrence is, or how to programmatically spot it to know when to try something more forceful.
Expected behavior
After server failure, client library should reconnect
Actual behavior
After server failure, client library does not reconnect
Reproduction Steps
Unfortunately I do not have any. The problem seems intermittent and caused by something server side.
Environment

.NET Core 3.1
Docker container (mcr.microsoft.com/dotnet/aspnet:3.1)
Running in AKS

If any more detailed information is required I am happy to provide it but I don't know what would be useful.

Thank you for your feedback.  Tagging and routing to the team member best able to assist.

@JoshLove-msft
After the initial exception we have our own retries which end up with the same exception. Eventually this will get deadlettered and we have retried it much later and you just end up with the same error.

Do the exceptions have different timestamps in the message? I don't think we cache this exception anywhere so getting the exception again should mean its coming from the service. Are you able to include the relevant exceptions? Also, it would be helpful if you could include SDK logs if those are enabled (see https://docs.microsoft.com/en-us/dotnet/azure/sdk/logging).

Can you also provide us with the namespace name and entity name so that the service team can take a look? If you would prefer not to leave it in the issue, you can submit this information in a support ticket.

@JoshLove-msft apologies for the late reply, had bank holidays here in UK.

Do the exceptions have different timestamps in the message?

Yes. Exceptions are as follows:
sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:21:30 Reference:3bc9b3fb-f97a-4f20-8106-68e709da3b08, TrackingId:1685d478-88e8-4bfd-b2a6-7093948b2e31_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:21:30 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:21:16 Reference:9d59e7ff-4845-48d3-82a3-2ecd2accccb1, TrackingId:6877b759-ca72-4358-8de1-82cd2a678f59_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:21:16 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:21:05 Reference:8840c316-faf9-4571-b7aa-5bdba6db8a9e, TrackingId:0ab9142b-5400-4af2-ae57-34ed8a3c71b1_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:21:05 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:20:53 Reference:0000f043-075a-4edf-894c-87f3187b7051, TrackingId:b414fa31-ce93-44fb-8595-53b094681d16_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:20:53 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:20:41 Reference:26600d69-efab-49dc-9dfc-c00371f383c5, TrackingId:648585c0-05c2-45b0-9084-fc996ae3feae_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:20:41 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:20:30 Reference:1ae91609-39f4-4c68-8156-da3d2748e875, TrackingId:d003c669-30b7-45e5-bb6c-f6f98d6f9e51_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:20:30 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:20:19 Reference:3e44f915-e213-4d6f-8ef4-b04214475aa4, TrackingId:53c3737e-f320-40f8-9a3d-8a94f3ebe1bb_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:20:19 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:20:07 Reference:956aa39e-f2d7-4fb6-86dc-f63baf10bd90, TrackingId:f8a9001c-0f7b-45f6-abdf-ed69dcf1ad02_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:20:07 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:19:56 Reference:21007ede-d648-4d8d-a2ac-d4175bc4f28d, TrackingId:000f2271-1497-4f47-b9e1-c99761f94916_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:19:56 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:19:45 Reference:eb82c850-ef02-4657-9e8a-a6028b8ec3e8, TrackingId:8058639c-530c-4914-90c9-9669770dd2ad_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:19:45 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:19:33 Reference:cab04456-aaf3-4082-b8ad-c9f0e34eb93a, TrackingId:86ad8d32-a49c-443d-be7d-a497ba325331_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:19:33 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:17:05 Reference:7d15384a-b113-4234-a186-f732cb91abd3, TrackingId:5ee509b2-b122-4eec-8764-a6badf073994_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:17:05 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:16:54 Reference:79a86c69-9b1e-4533-9eaf-9f52182ea481, TrackingId:fbbb367a-2ad5-4d5a-82cc-17fc21074f3f_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:16:54 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:16:43 Reference:0ffc0680-6122-4e8f-bc7e-e4e162551fba, TrackingId:b8a11356-ee3f-45d2-b43f-db65b4f7fb46_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:16:43 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:16:31 Reference:f3fbb70a-53fe-4a9f-9393-7e0d92488017, TrackingId:f215f279-f04b-4ea9-9663-c67c50891f8c_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:16:31 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:16:20 Reference:13da6a52-4d0e-4918-806b-16e3779e62e7, TrackingId:9d958c32-f594-4973-989f-1722c9630fdf_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:16:20 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:16:08 Reference:7ea9cd6b-2d93-430d-89ac-2ea6a46a26a5, TrackingId:809810c4-7c60-4840-b05f-b30355aa9760_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:16:08 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:15:57 Reference:c6097eb3-404d-4499-b04c-79e7b0030221, TrackingId:abf0b849-0e21-4439-9251-3745ce818c1c_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:15:57 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:15:46 Reference:4e93ba39-d7e2-4e29-9738-b31ff8bfdfaa, TrackingId:9a711298-dc50-432f-bb29-7e8f037f6ef1_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:15:46 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:15:34 Reference:40ea6d19-7c89-4d86-9476-5a68747c8c53, TrackingId:881b4424-b8a1-4b1e-b332-7dd9b97fbba0_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:15:34 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:15:23 Reference:9deda66a-acef-4b4b-9ebb-5b6ba48f544a, TrackingId:5591922b-2b7b-440f-b438-0483ea257b73_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:15:23 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

sandbox.matching.commands-30104287-5dcf-497e-8534-fbbdbff028d4: ScheduleMessageAsync Exception: Azure.Messaging.ServiceBus.ServiceBusException: The link 'G23:RR:213866011:637468380794500000:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands$management:103182:sender' is force detached. Code: ServerError. Details: AmqpControlProtocolClient.Fault. TrackingId:4dbabb40-1cc1-4165-84d9-208b5d2968ef_B35, SystemTracker:wk-sbx01-westeurope-servicebus:Queue:sandbox.matching.commands, Timestamp:2022-05-28T07:15:12 Reference:7a357238-0844-46a7-92c9-ebd29b865fc2, TrackingId:c42daacf-ad51-4049-8778-59df99835b40_G23, SystemTracker:NoSystemTracker, Timestamp:2022-05-28T07:15:12 (GeneralError)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessageInternalAsync(IReadOnlyList`1 messages, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<ScheduleMessagesAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.ScheduleMessagesAsync(IReadOnlyList`1 messages, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusSender.ScheduleMessagesAsync(IEnumerable`1 messages, DateTimeOffset scheduledEnqueueTime, CancellationToken cancellationToken).

I'll have to see about the logging, is there any way I can narrow it to just the necessary logs? We have a lot of services and with this being somewhat transient I'd need to roll it out in many places.
Namespace is wk-sbx01-westeurope-servicebus and the queue name was sandbox.matching.commands in the examples I gave.

Here is the information I got from the service team:

We were migrating data between databased around that time. As a result, the entity had some downtime. If it lasted more than few minutes, ask the customer to file a support incident. We will investigate and provide root cause.

@JoshLove-msft Whilst the migration makes complete sense on why it lost connection, why does it require restarting the pod for it to reconnect? That's the main issue.
I'd need to find concrete examples but we had one service (thankfully not production) that after the initial connection dropping spent days unable to send messages until we restarted the pod.

@JoshLove-msft Whilst the migration makes complete sense on why it lost connection, why does it require restarting the pod for it to reconnect? That's the main issue.
I'd need to find concrete examples but we had one service (thankfully not production) that after the initial connection dropping spent days unable to send messages until we restarted the pod.

This is what the service team will be able to help dig into if you file a support incident. Based on the logs that you shared, the client is trying to reconnect but it repeatedly gets an error from the service. If their investigation indicates that the client is doing something wrong when reconnecting, we can certainly fix that.

@JoshLove-msft Just want to add we've realised something, it's not fully disconnected for sends at least. It seems it only deadletters scheduled messages, all other messages on the queue works fine. We've got a service currently in this state but we can't leave it like it is forever as it has customer impacts.

It seems it only deadletter scheduled messages, all other messages on the queue work fine. We've got a service currently in this state but we can't leave it like it is forever as it has customer impacts.

Can you clarify how you are determining this? I thought the issue was that messages are not being scheduled in the first place. You are saying you are getting the exception above and then the messages are ending up in the deadletter queue? In any case, if you can file a support ticket, the service team will be able to dig into the service logs to determine what is going wrong.

@JoshLove-msft
Sorry I wasn't clear, we have somewhat chained commands where one command or event will cause another which is what was deadlettering, not the command we were trying to send.
We determined it was only scheduled sends because we have other parts of the system that worked absolutely fine, but this particular one (doing a scheduled send) was throwing an exception when it was attempted, even when retrying an hour after the original exception.
This was in a different service to the exceptions that I posted in this thread, but we had not noticed that those also were exceptions for scheduled sends.
To be clear, after restarting the service, the sends started working again. There is something particularly different to the way we do those sends, they all work off the same client which is what was so weird.
As I put in my previous message I have already raised a support ticket.