more info on PLAYWRIGHT_LAUNCH_OPTIONS

Hey,
Thanks for creating this package. its really great.
I'm facing an issue with memory, because of using playwright all the memory of my machine is getting filled and i couldn't ssh into the VM. So, I was testing few things, like changing the browsers etc.
I couldn't find where I can find documentation on PLAYWRIGHT_LAUNCH_OPTIONS? I want to use firefox instead of chromium.
Thanks in advance

https://github.com/scrapy-plugins/scrapy-playwright#supported-settings points to https://playwright.dev/python/docs/api/class-browsertype#browser-type-launch


Add icons to the system

We often need to manage custom icons in Pharo.
Maybe we could have a page on icons management and explain how to add our own icons.
One way to do it is to do:
ThemeIcons compile:  'myLogoContents
	^ ''' , (Base64MimeConverter mimeEncode: 'logo.png' asFileReference binaryReadStream) contents , '''' classified: 'Material-Design-Lite-Core'.

ThemeIcons compile: 'myLogoIcon
	^ icons
			at: #''myLogoIcon''
			ifAbsentPut:[ Form fromBinaryStream: self myLogoContents base64Decoded asByteArray readStream ]' classified: 'Material-Design-Lite-Core'.

Smalltalk ui icons myLogoIcon

Favor usage of #binaryReadStreamDo: to ensure the resources are closed.


Utilities and spark ci gh actions

Change Logs
Describe context and summary for this change. Highlight if any code was copied.
Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

if its ready for review, please add the jira link.


Ability to add (append) tick suffix to graph axis

As a Publisher, I want to be able to add a suffix to the graph axis so my numbers are represented correctly.
This issue is reported here https://github.com/datahq/datahub-qa/issues/133
Acceptance criteria

[x] I can have tick suffixes in simple graphs
[x] documented in http://datahub.io/docs/features/views

Tasks

[x] refactor datapackage-render-js library to handle suffix property in the simple spec
[x] test out and rebuild showcase-js
[x] add docs
[x] deploy

FIXED, see https://datahub.io/anuveyatsu/simple-views and its views property https://pkgstore.datahub.io/90998f7f90e086bd5fc7c9075dfda43b/simple-views/3/datapackage.json


Remove Multidex usages

Since the min SDK is 26, it is no longer necessary to use the Multidex library.
See the following for more info: https://developer.android.com/build/multidex#mdex-on-l

@arpallares any chance to have a look at this PR?

LGTM, but I think @SurlySlacker can review this.

@SurlySlacker did you have a chance to have a look at this PR?


Bank account verification: Users were very uncomfortable with being asked to enter this information. They didn't understand why they needed to give the information for a parking permit.

Impact: Users may abandon Verify at this point.
Evidence: “Don’t see why my bank details are needed for a parking permit.”
“Go back to Verify to check options as not clear on bank account.”
“Better explanation needed – transitory information.”
“Haven’t been asked about bank account – maybe I missed it – would open a new window to check.”
“Don’t need to give you bank account, what I’ve given you is good enough.”
“Expect not to see bank account or an explanation as to why asking.”
“Bureaucratic nonsense to ask for this.”
“Would like to know why they need it.”
“Hacking worry.”
“Bank account – I don’t like that.”
“What was the alternative?”
“I don’t like giving my bank account details.”
“There is a padlock but I’m nervous.”
“Absolutely would not give bank account details. No idea who has access, where it’s going to be sent or used – just would not. I don’t do online banking because it’s not secure.”

See also #602 #588


AttributeError: module 'pynumaflow.sourcer.proto.source_pb2_grpc' has no attribute 'add_sourceServicer_to_server'

In the async source server has a typo at line 184.  The add_sourceServicer_to_server function should read add_sourceServicer_to_server (upper case S in source).

Message from the maintainers:
Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

This looks like a pretty straightforward "fix", but I'm more accustomed to using GitLab, where I can create an MR (merge request) directly from an issue.  Would I need to fork the project and then create a PR?  How do I tie the PR back to this issue?


Remove default tags from global level

Removing code for generating global tags object (https://github.com/iheartradio/play-swagger/issues/70 ).
Also added an extra test to make sure that a "routes" tag is not added when a default routes file is used.
cc: @kailuowang

I had to a couple of dependencies in order to run tests. I'm not sure how the existing CI is passing. I could use some pointers on how to investigate this.
cc: @kailuowang @Javakky-pxv

https://github.com/JodaOrg/joda-time

https://github.com/google/error-prone

@kailuowang
Can I ask you to review this PR as I do not have sufficient knowledge of this PR?

@kailuowang Can I ask you to review this PR as I do not have sufficient knowledge of this PR?

Sorry, I pretty much forgot about the code as well. I think the expected behavior is that the code generates a tag for each route from the path name of the route file in which the route is defined. That means that routes defined in the root route file will have a pretty useless tag of the root file path. It will be ideal to remove this tag for the root file but retain the tags for routes in referenced routes files.

@kailuowang  I raised another pr a few years back and lost track of it. It got auto-closed due to no activity. You can check it for reference.
https://github.com/iheartradio/play-swagger/pull/342

#70

@kailuowang
I am in favor of this change.
It is a BREAKING CHANGE, but I am not sure if this is a milestone for ver 1.0.0, and I am considering releasing it as 0.14.0 and advertising it in the CHANGE LOG.
By the way, do you have a roadmap for ver 1.0.0? Or is it already substantially 1.0.0?

Imo, it should've been 1.0 by now. I don't have a roadmap. I stopped using
swagger or play long time ago, and thus not a stakeholder of this lib. I
don't think I have anything to say in regards to it's features or roadmap.
One thing you might want to consider is a roadmap to scala 3. A big portion
of reflection based code might needs to be rewritten.
On Tue, Mar 7, 2023, 12:19 AM Kazuma Iemura @.***>
wrote:

@kailuowang https://github.com/kailuowang
I am in favor of this change.
It is a BREAKING CHANGE, but I am not sure if this is a milestone for ver
1.0.0, and I am considering releasing it as 0.14.0 and advertising it in
the CHANGE LOG.
By the way, do you have a roadmap for ver 1.0.0? Or is it already
substantially 1.0.0?
—
Reply to this email directly, view it on GitHub
https://github.com/iheartradio/play-swagger/pull/525#issuecomment-1457554628,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAUKOMH3P3SWG7UJGP45X3W23AO7ANCNFSM6AAAAAATNSKTJY
.
You are receiving this because you were mentioned.Message ID:
@.***>

@kchaitanya1195 @kailuowang
Sorry for the delay.
I will use this change as an opportunity to change the major version to 1.0.0.
I will update the README soon, but it will likely stay the same until I get around to it ......
Support for Scala 3 is an important change, but I will take the plunge and move up to 2.0.0 if there are major changes at that time.


Fixed missing TaskInfoStatus and TaskInfoType handling in ObjectExtensions

Pull Request
Related issue
Fixes #509
What does this PR do?

Adds 4 lines of code to include TaskInfoStatus and TaskInfoType in the respective function.

PR checklist
Please check if your PR fulfills the following requirements:

[x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
[x] Have you read the contributing guidelines?
[x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

Tests and formatted ran successfully

Requesting the reviewers to have a look.

Thanks for your contribution
bors merge


Day303

Resolves #616.

Pull Request Test Coverage Report for Build 1155

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 1145:
0.0%

Covered Lines:
5

Relevant Lines:
5

💛  - Coveralls

Pull Request Test Coverage Report for Build 1147

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 1145:
0.0%

Covered Lines:
5

Relevant Lines:
5

💛  - Coveralls


added pointInfo

Coverage increased (+0.9%) to 23.497% when pulling 901b30041dd683a0d1bf70f709aa45267d0f4c40 on rasterApi into 766a4735ca21b845f0eda2446bcc8f70c14b3bb6 on master.


[Perl] Bugfixes and sync with python symbolic code

Fixes for ImageIter
Convolutional RNN
Improved Visualization
PearsonCorrelation metric
Fixed tests.

@piiswrong could you please merge this pull, it has passed all tests and the code has no conflicts.


fix issue with docker volume on windows

Proposed Changes
Fixes an issue on Windows where the volumes are not being mounted correctly when running a pim.
Type of Change
What kind of change to packageless is this?

[x] Bug Fix
[ ] Feature
[ ] Documentation
[ ] Repository Enhancement
[ ] Testing

Checklist
Before the Pull Request can be considered for merging, the following Checklist should have the corresponding fields completed:

[x] All tests have passed locally after running go test ./...
[ ] I have added tests that prove my fix or feature works as expected
[x] I have added any necessary documentation for my fix or feature

Comments
Any further information you want to provide can go here.
No tests were added as they are not needed.

Codecov Report

Merging #72 (461b5cf) into main (054d360) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #72   +/-   ##
=======================================
  Coverage   88.82%   88.82%
=======================================
  Files           7        7
  Lines         725      725
=======================================
  Hits          644      644
  Misses         54       54
  Partials       27       27

Flag
Coverage Δ

subcommands-coverage
88.82% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 054d360...461b5cf. Read the comment docs.


[suggestion] how about using docker for querydsl test environment?

In the current querydsl test environment, vagrant is used.
This environment is good, but it requires a number of pre-requisites. (VirtualBox, Puppet, Vagrant..)
If we provide various databases as docker compose, wouldn't it be simpler to build a test environment?
for example,
version: '2'
services:
  mysql:
    image: mysql:5.5
    environment:
      MYSQL_USER: querydsl
      MYSQL_PASSWORD: querydsl
      MYSQL_DATABASE: querydsl
      MYSQL_ROOT_PASSWORD: querydsl
    volumes:
      - /var/lib/mysql
    ports:
      - "3306:3306"
  postgre-sql:
    image: postgres:9.1
    ports:
      - "5433:5432"
    environment:
      POSTGRES_USER: querydsl
      POSTGRES_PASSWORD: querydsl
      POSTGRES_DATABASE: querydsl
      POSTGRES_ROOT_PASSWORD: querydsl
    volumes:
      - /var/lib/postgresql/data
.
.
.
(Other necessary database environments...)

It's possible. Usually, I just test against the database of my preference (which runs in Docker) and rely on CI for all various databases that are supported.

I usually just run the docker-compose file we use in CI locally.

Ah here it is: https://github.com/querydsl/querydsl/blob/master/travis/docker-compose.yml
I guess we could mention it in the README

Oh, docker-compose already exists! :)
I'll change it to an issue that mentions the docker-compose environment in the Readme.

I’ve been meaning to update the readme and move the docker-compose.yml file to the root of the project. Thanks for the reminder.


amdgpu: black screen after resume from suspend

Describe the bug
i suspended the computer yesterday and resumed today by hitting ENTER. the computer is on, but i only see a black screen, not even a cursor
Steps To Reproduce
Steps to reproduce the behavior:

suspend
resume
black screen

Expected behavior
have image on screen
Screenshots
imagine an all black screenshot
Additional context
Similar to previous issues:

https://github.com/NixOS/nixpkgs/issues/170429
https://github.com/NixOS/nixpkgs/issues/176915
https://github.com/NixOS/nixpkgs/issues/185212

amdgpu crashed the kernel
Mar 29 03:48:43 gaming kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=1931602, emitted seq=1931604
Mar 29 03:48:43 gaming kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
Mar 29 03:48:43 gaming kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset begin!
Mar 29 03:48:43 gaming kernel: amdgpu 0000:03:00.0: amdgpu: Failed to disallow df cstate
Mar 29 03:48:44 gaming kernel: [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
Mar 29 03:48:47 gaming kernel: amdgpu 0000:03:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000036 SMN_C2PMSG_82:0x00000000
Mar 29 03:48:47 gaming kernel: amdgpu 0000:03:00.0: amdgpu: Failed to disable smu features.
Mar 29 03:48:47 gaming kernel: amdgpu 0000:03:00.0: amdgpu: Fail to disable dpm features!
Mar 29 03:48:47 gaming kernel: [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <smu> failed -62
Mar 29 03:48:48 gaming kernel: [drm] psp gfx command DESTROY_TMR(0x7) failed and response status is (0x80000306)
Mar 29 03:48:48 gaming kernel: amdgpu 0000:03:00.0: amdgpu: MODE1 reset
Mar 29 03:48:48 gaming kernel: amdgpu 0000:03:00.0: amdgpu: GPU mode1 reset
Mar 29 03:48:48 gaming kernel: amdgpu 0000:03:00.0: amdgpu: GPU smu mode1 reset
Mar 29 03:48:49 gaming systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Mar 29 03:48:51 gaming kernel: amdgpu 0000:03:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000036 SMN_C2PMSG_82:0x00000000
Mar 29 03:48:51 gaming kernel: amdgpu 0000:03:00.0: amdgpu: GPU mode1 reset failed
Mar 29 03:48:51 gaming kernel: amdgpu 0000:03:00.0: amdgpu: ASIC reset failed with error, -62 for drm dev, 0000:03:00.0
Mar 29 03:48:58 gaming .gsd-power-wrap[2018]: Error setting property 'PowerSaveMode' on interface org.gnome.Mutter.DisplayConfig: Timeout was reached (g-io-error-quark, 24)
Mar 29 03:49:01 gaming kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset succeeded, trying to resume
Mar 29 03:49:01 gaming kernel: [drm] PCIE GART of 512M enabled (table at 0x0000008000900000).
Mar 29 03:49:01 gaming kernel: [drm] VRAM is lost due to GPU reset!
Mar 29 03:49:01 gaming kernel: [drm] PSP is resuming...
Mar 29 03:49:02 gaming kernel: [drm] failed to load ucode SMC(0x31)
Mar 29 03:49:02 gaming kernel: [drm] psp gfx command LOAD_IP_FW(0x6) failed and response status is (0x80000306)
Mar 29 03:49:02 gaming kernel: [drm] reserve 0xa00000 from 0x81fd000000 for PSP TMR
Mar 29 03:49:03 gaming kernel: amdgpu 0000:03:00.0: amdgpu: RAS: optional ras ta ucode is not available
Mar 29 03:49:03 gaming kernel: amdgpu 0000:03:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
Mar 29 03:49:03 gaming kernel: amdgpu 0000:03:00.0: amdgpu: SMU is resuming...
Mar 29 03:49:03 gaming kernel: amdgpu 0000:03:00.0: amdgpu: smu driver if version = 0x0000000f, smu fw if version = 0x00000013, smu fw program = 0, version = 0x003b2900 (59.41.0)
Mar 29 03:49:03 gaming kernel: amdgpu 0000:03:00.0: amdgpu: SMU driver if version not matched
Mar 29 03:49:06 gaming kernel: amdgpu 0000:03:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000036 SMN_C2PMSG_82:0x00000000
Mar 29 03:49:09 gaming kernel: amdgpu 0000:03:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000036 SMN_C2PMSG_82:0x00000000
Mar 29 03:49:09 gaming kernel: amdgpu 0000:03:00.0: amdgpu: Failed to SetDriverDramAddr!
Mar 29 03:49:09 gaming kernel: amdgpu 0000:03:00.0: amdgpu: Failed to setup smc hw!
Mar 29 03:49:09 gaming kernel: [drm:amdgpu_device_ip_resume_phase2 [amdgpu]] *ERROR* resume of IP block <smu> failed -62
Mar 29 03:49:09 gaming kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset(1) failed
Mar 29 03:49:09 gaming kernel: [drm] Skip scheduling IBs!
Mar 29 03:49:09 gaming kernel: [drm] Skip scheduling IBs!
Mar 29 03:49:09 gaming kernel: [drm] Skip scheduling IBs!
...
Mar 29 03:49:09 gaming kernel: [drm] Skip scheduling IBs!
Mar 29 03:49:09 gaming kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset end with ret = -62
Mar 29 03:49:09 gaming kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* GPU Recovery Failed: -62
Mar 29 03:49:09 gaming kernel: snd_hda_intel 0000:03:00.1: Refused to change power state from D0 to D3hot
Mar 29 03:49:19 gaming kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=1931604, emitted seq=1931606
Mar 29 03:49:19 gaming kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
Mar 29 03:49:19 gaming kernel: amdgpu 0000:03:00.0: amdgpu: GPU reset begin!
Mar 29 03:49:19 gaming kernel: amdgpu 0000:03:00.0: amdgpu: Failed to disallow df cstate
Mar 29 03:52:24 gaming kernel: INFO: task X:cs0:1456 blocked for more than 122 seconds.
Mar 29 03:52:24 gaming kernel:       Not tainted 6.2.0 #1-NixOS
Mar 29 03:52:24 gaming kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 03:52:24 gaming kernel: task:X:cs0           state:D stack:0     pid:1456  ppid:1431   flags:0x00004002
Mar 29 03:52:24 gaming kernel: Call Trace:
Mar 29 03:52:24 gaming kernel:  <TASK>
Mar 29 03:52:24 gaming kernel:  __schedule+0x30d/0x1270
Mar 29 03:52:24 gaming kernel:  schedule+0x61/0xe0
Mar 29 03:52:24 gaming kernel:  schedule_timeout+0x123/0x160
Mar 29 03:52:24 gaming kernel:  ? _raw_spin_unlock_irqrestore+0x27/0x50
Mar 29 03:52:24 gaming kernel:  ? dma_fence_add_callback+0x6a/0xe0
Mar 29 03:52:24 gaming kernel:  dma_fence_wait_any_timeout+0x204/0x260
Mar 29 03:52:24 gaming kernel:  amdgpu_sa_bo_new+0x464/0x530 [amdgpu]
Mar 29 03:52:24 gaming kernel:  ? preempt_count_add+0x74/0xa0
Mar 29 03:52:24 gaming kernel:  amdgpu_ib_get+0x43/0x90 [amdgpu]
Mar 29 03:52:24 gaming kernel:  ? drm_sched_fence_alloc+0x1e/0x40 [gpu_sched]
Mar 29 03:52:24 gaming kernel:  amdgpu_job_alloc_with_ib+0x72/0xb0 [amdgpu]
Mar 29 03:52:24 gaming kernel:  amdgpu_vm_sdma_update+0x30b/0x400 [amdgpu]
Mar 29 03:52:24 gaming kernel:  amdgpu_vm_ptes_update+0x2b0/0x800 [amdgpu]
Mar 29 03:52:24 gaming kernel:  amdgpu_vm_update_range+0x21e/0x750 [amdgpu]
Mar 29 03:52:24 gaming kernel:  amdgpu_vm_bo_update+0x2b0/0x570 [amdgpu]
Mar 29 03:52:24 gaming kernel:  amdgpu_vm_handle_moved+0x10c/0x120 [amdgpu]
Mar 29 03:52:24 gaming kernel:  amdgpu_cs_ioctl+0x130c/0x2090 [amdgpu]
Mar 29 03:52:24 gaming kernel:  ? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
Mar 29 03:52:24 gaming kernel:  drm_ioctl_kernel+0xb6/0x140 [drm]
Mar 29 03:52:24 gaming kernel:  drm_ioctl+0x22a/0x3e0 [drm]
Mar 29 03:52:24 gaming kernel:  ? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
Mar 29 03:52:24 gaming kernel:  ? ioctl_has_perm.constprop.0.isra.0+0xbd/0x120
Mar 29 03:52:24 gaming kernel:  amdgpu_drm_ioctl+0x4d/0x80 [amdgpu]
Mar 29 03:52:24 gaming kernel:  __x64_sys_ioctl+0x8b/0xc0
Mar 29 03:52:24 gaming kernel:  do_syscall_64+0x3c/0x90
Mar 29 03:52:24 gaming kernel:  entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 29 03:52:24 gaming kernel: RIP: 0033:0x7f9d2b304321
Mar 29 03:52:24 gaming kernel: RSP: 002b:00007f9d217fe7d0 EFLAGS: 00000246 ORIG_RAX:<PHONE_NUMBER>000010
Mar 29 03:52:24 gaming kernel: RAX: ffffffffffffffda RBX: 00007f9d217fe890 RCX: 00007f9d2b304321
Mar 29 03:52:24 gaming kernel: RDX: 00007f9d217fe890 RSI: 00000000c0186444 RDI:<PHONE_NUMBER>000013
Mar 29 03:52:24 gaming kernel: RBP: 00000000c0186444 R08: 00007f9d217fe9e0 R09: 00007f9d217fe988
Mar 29 03:52:24 gaming kernel: R10:<PHONE_NUMBER>918890 R11:<PHONE_NUMBER>000246 R12: 00007f9d217fe9b0
Mar 29 03:52:24 gaming kernel: R13:<PHONE_NUMBER>000013 R14:<PHONE_NUMBER>000003 R15: 00007f9d217fe988
Mar 29 03:52:24 gaming kernel:  </TASK>
Mar 29 03:52:24 gaming kernel: INFO: task X:3921621 blocked for more than 122 seconds.
Mar 29 03:52:24 gaming kernel:       Not tainted 6.2.0 #1-NixOS
Mar 29 03:52:24 gaming kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 03:52:24 gaming kernel: task:X               state:D stack:0     pid:3921621 ppid:1431   flags:0x00000002
Mar 29 03:52:24 gaming kernel: Call Trace:
Mar 29 03:52:24 gaming kernel:  <TASK>
Mar 29 03:52:24 gaming kernel:  __schedule+0x30d/0x1270
Mar 29 03:52:24 gaming kernel:  schedule+0x61/0xe0
Mar 29 03:52:24 gaming kernel:  schedule_preempt_disabled+0x18/0x30
Mar 29 03:52:24 gaming kernel:  __mutex_lock.constprop.0+0x38d/0x700
Mar 29 03:52:24 gaming kernel:  amdgpu_dm_atomic_commit_tail+0x5f9/0x2c30 [amdgpu]
Mar 29 03:52:24 gaming kernel:  ? __sbitmap_get_word+0x24/0x70
Mar 29 03:52:24 gaming kernel:  ? sbitmap_get+0x97/0x1d0
Mar 29 03:52:24 gaming kernel:  ? blk_mq_do_dispatch_sched+0xa1/0x3b0
Mar 29 03:52:24 gaming kernel:  ? _raw_spin_trylock+0x17/0x60
Mar 29 03:52:24 gaming kernel:  ? _raw_spin_unlock+0x19/0x40
Mar 29 03:52:24 gaming kernel:  ? get_page_from_freelist+0x1436/0x1580
Mar 29 03:52:24 gaming kernel:  ? pre_validate_dsc+0x75/0x430 [amdgpu]
Mar 29 03:52:24 gaming kernel:  ? drm_atomic_helper_check_planes+0x156/0x230 [drm_kms_helper]
Mar 29 03:52:24 gaming kernel:  ? amdgpu_dm_atomic_check+0x6d3/0x1180 [amdgpu]
Mar 29 03:52:24 gaming kernel:  ? preempt_count_add+0x74/0xa0
Mar 29 03:52:24 gaming kernel:  ? _raw_spin_lock_irq+0x1d/0x50
Mar 29 03:52:24 gaming kernel:  ? _raw_spin_unlock_irq+0x1f/0x40
Mar 29 03:52:24 gaming kernel:  ? __wait_for_common+0x1a7/0x1e0
Mar 29 03:52:24 gaming kernel:  ? __pfx_schedule_timeout+0x10/0x10
Mar 29 03:52:24 gaming kernel:  ? preempt_count_add+0x74/0xa0
Mar 29 03:52:24 gaming kernel:  ? _raw_spin_lock_irqsave+0x28/0x60
Mar 29 03:52:24 gaming kernel:  ? complete_all+0x24/0x90
Mar 29 03:52:24 gaming kernel:  commit_tail+0x91/0x130 [drm_kms_helper]
Mar 29 03:52:24 gaming kernel:  drm_atomic_helper_commit+0x11f/0x150 [drm_kms_helper]
Mar 29 03:52:24 gaming kernel:  drm_atomic_commit+0x97/0xd0 [drm]
Mar 29 03:52:24 gaming kernel:  ? __pfx___drm_printfn_info+0x10/0x10 [drm]
Mar 29 03:52:24 gaming kernel:  drm_mode_gamma_set_ioctl+0x399/0x530 [drm]
Mar 29 03:52:24 gaming kernel:  ? __pfx_drm_mode_gamma_set_ioctl+0x10/0x10 [drm]
Mar 29 03:52:24 gaming kernel:  drm_ioctl_kernel+0xb6/0x140 [drm]
Mar 29 03:52:24 gaming kernel:  drm_ioctl+0x22a/0x3e0 [drm]
Mar 29 03:52:24 gaming kernel:  ? __pfx_drm_mode_gamma_set_ioctl+0x10/0x10 [drm]
Mar 29 03:52:24 gaming kernel:  ? ioctl_has_perm.constprop.0.isra.0+0xbd/0x120
Mar 29 03:52:24 gaming kernel:  amdgpu_drm_ioctl+0x4d/0x80 [amdgpu]
Mar 29 03:52:24 gaming kernel:  __x64_sys_ioctl+0x8b/0xc0
Mar 29 03:52:24 gaming kernel:  do_syscall_64+0x3c/0x90
Mar 29 03:52:24 gaming kernel:  entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 29 03:52:24 gaming kernel: RIP: 0033:0x7fcd3a504321
Mar 29 03:52:24 gaming kernel: RSP: 002b:00007fffec233be0 EFLAGS: 00000246 ORIG_RAX:<PHONE_NUMBER>000010
Mar 29 03:52:24 gaming kernel: RAX: ffffffffffffffda RBX: 00007fffec233c70 RCX: 00007fcd3a504321
Mar 29 03:52:24 gaming kernel: RDX: 00007fffec233c70 RSI: 00000000c02064a5 RDI:<PHONE_NUMBER>000010
Mar 29 03:52:24 gaming kernel: RBP: 00000000c02064a5 R08:<PHONE_NUMBER>9d31f0 R09:<PHONE_NUMBER>9d33f0
Mar 29 03:52:24 gaming kernel: R10:<PHONE_NUMBER>000042 R11:<PHONE_NUMBER>000246 R12:<PHONE_NUMBER>9d2b30
Mar 29 03:52:24 gaming kernel: R13:<PHONE_NUMBER>000010 R14:<PHONE_NUMBER>9ce570 R15:<PHONE_NUMBER>000000
Mar 29 03:52:24 gaming kernel:  </TASK>
Mar 29 03:52:24 gaming kernel: INFO: task kworker/u32:26:3921699 blocked for more than 122 seconds.
Mar 29 03:52:24 gaming kernel:       Not tainted 6.2.0 #1-NixOS
Mar 29 03:52:24 gaming kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 03:52:24 gaming kernel: task:kworker/u32:26  state:D stack:0     pid:3921699 ppid:2      flags:0x00004000
Mar 29 03:52:24 gaming kernel: Workqueue: amdgpu-reset-dev drm_sched_job_timedout [gpu_sched]
Mar 29 03:52:24 gaming kernel: Call Trace:
Mar 29 03:52:24 gaming kernel:  <TASK>
Mar 29 03:52:24 gaming kernel:  __schedule+0x30d/0x1270
Mar 29 03:52:24 gaming kernel:  ? __wake_up_klogd.part.0+0x56/0x80
Mar 29 03:52:24 gaming kernel:  ? dev_vprintk_emit+0x175/0x19d
Mar 29 03:52:24 gaming kernel:  schedule+0x61/0xe0
Mar 29 03:52:24 gaming kernel:  schedule_preempt_disabled+0x18/0x30
Mar 29 03:52:24 gaming kernel:  __mutex_lock.constprop.0+0x38d/0x700
Mar 29 03:52:24 gaming kernel:  dm_suspend+0xcc/0x1e0 [amdgpu]
Mar 29 03:52:25 gaming kernel:  amdgpu_device_ip_suspend_phase1+0x72/0xe0 [amdgpu]
Mar 29 03:52:25 gaming kernel:  amdgpu_device_ip_suspend+0x20/0x70 [amdgpu]
Mar 29 03:52:25 gaming kernel:  amdgpu_device_pre_asic_reset+0xd5/0x290 [amdgpu]
Mar 29 03:52:25 gaming kernel:  amdgpu_device_gpu_recover.cold+0x5f8/0xb4a [amdgpu]
Mar 29 03:52:25 gaming kernel:  amdgpu_job_timedout+0x18a/0x1c0 [amdgpu]
Mar 29 03:52:25 gaming kernel:  ? _raw_spin_unlock+0x19/0x40
Mar 29 03:52:25 gaming kernel:  drm_sched_job_timedout+0x77/0x110 [gpu_sched]
Mar 29 03:52:25 gaming kernel:  process_one_work+0x1e2/0x3b0
Mar 29 03:52:25 gaming kernel:  worker_thread+0x54/0x3a0
Mar 29 03:52:25 gaming kernel:  ? __pfx_worker_thread+0x10/0x10
Mar 29 03:52:25 gaming kernel:  kthread+0xe9/0x110
Mar 29 03:52:25 gaming kernel:  ? __pfx_kthread+0x10/0x10
Mar 29 03:52:25 gaming kernel:  ret_from_fork+0x29/0x50
Mar 29 03:52:25 gaming kernel:  </TASK>
Mar 29 03:54:27 gaming kernel: INFO: task X:cs0:1456 blocked for more than 245 seconds.
Mar 29 03:54:27 gaming kernel:       Not tainted 6.2.0 #1-NixOS
Mar 29 03:54:27 gaming kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 03:54:27 gaming kernel: task:X:cs0           state:D stack:0     pid:1456  ppid:1431   flags:0x00004002
Mar 29 03:54:27 gaming kernel: Call Trace:
Mar 29 03:54:27 gaming kernel:  <TASK>
Mar 29 03:54:27 gaming kernel:  __schedule+0x30d/0x1270
Mar 29 03:54:27 gaming kernel:  schedule+0x61/0xe0
Mar 29 03:54:27 gaming kernel:  schedule_timeout+0x123/0x160
Mar 29 03:54:27 gaming kernel:  ? _raw_spin_unlock_irqrestore+0x27/0x50
Mar 29 03:54:27 gaming kernel:  ? dma_fence_add_callback+0x6a/0xe0
Mar 29 03:54:27 gaming kernel:  dma_fence_wait_any_timeout+0x204/0x260
Mar 29 03:54:27 gaming kernel:  amdgpu_sa_bo_new+0x464/0x530 [amdgpu]
Mar 29 03:54:27 gaming kernel:  ? preempt_count_add+0x74/0xa0
Mar 29 03:54:27 gaming kernel:  amdgpu_ib_get+0x43/0x90 [amdgpu]
Mar 29 03:54:27 gaming kernel:  ? drm_sched_fence_alloc+0x1e/0x40 [gpu_sched]
Mar 29 03:54:27 gaming kernel:  amdgpu_job_alloc_with_ib+0x72/0xb0 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_vm_sdma_update+0x30b/0x400 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_vm_ptes_update+0x2b0/0x800 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_vm_update_range+0x21e/0x750 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_vm_bo_update+0x2b0/0x570 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_vm_handle_moved+0x10c/0x120 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_cs_ioctl+0x130c/0x2090 [amdgpu]
Mar 29 03:54:27 gaming kernel:  ? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
Mar 29 03:54:27 gaming kernel:  drm_ioctl_kernel+0xb6/0x140 [drm]
Mar 29 03:54:27 gaming kernel:  drm_ioctl+0x22a/0x3e0 [drm]
Mar 29 03:54:27 gaming kernel:  ? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
Mar 29 03:54:27 gaming kernel:  ? ioctl_has_perm.constprop.0.isra.0+0xbd/0x120
Mar 29 03:54:27 gaming kernel:  amdgpu_drm_ioctl+0x4d/0x80 [amdgpu]
Mar 29 03:54:27 gaming kernel:  __x64_sys_ioctl+0x8b/0xc0
Mar 29 03:54:27 gaming kernel:  do_syscall_64+0x3c/0x90
Mar 29 03:54:27 gaming kernel:  entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 29 03:54:27 gaming kernel: RIP: 0033:0x7f9d2b304321
Mar 29 03:54:27 gaming kernel: RSP: 002b:00007f9d217fe7d0 EFLAGS: 00000246 ORIG_RAX:<PHONE_NUMBER>000010
Mar 29 03:54:27 gaming kernel: RAX: ffffffffffffffda RBX: 00007f9d217fe890 RCX: 00007f9d2b304321
Mar 29 03:54:27 gaming kernel: RDX: 00007f9d217fe890 RSI: 00000000c0186444 RDI:<PHONE_NUMBER>000013
Mar 29 03:54:27 gaming kernel: RBP: 00000000c0186444 R08: 00007f9d217fe9e0 R09: 00007f9d217fe988
Mar 29 03:54:27 gaming kernel: R10:<PHONE_NUMBER>918890 R11:<PHONE_NUMBER>000246 R12: 00007f9d217fe9b0
Mar 29 03:54:27 gaming kernel: R13:<PHONE_NUMBER>000013 R14:<PHONE_NUMBER>000003 R15: 00007f9d217fe988
Mar 29 03:54:27 gaming kernel:  </TASK>
Mar 29 03:54:27 gaming kernel: INFO: task X:3921621 blocked for more than 245 seconds.
Mar 29 03:54:27 gaming kernel:       Not tainted 6.2.0 #1-NixOS
Mar 29 03:54:27 gaming kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 03:54:27 gaming kernel: task:X               state:D stack:0     pid:3921621 ppid:1431   flags:0x00000002
Mar 29 03:54:27 gaming kernel: Call Trace:
Mar 29 03:54:27 gaming kernel:  <TASK>
Mar 29 03:54:27 gaming kernel:  __schedule+0x30d/0x1270
Mar 29 03:54:27 gaming kernel:  schedule+0x61/0xe0
Mar 29 03:54:27 gaming kernel:  schedule_preempt_disabled+0x18/0x30
Mar 29 03:54:27 gaming kernel:  __mutex_lock.constprop.0+0x38d/0x700
Mar 29 03:54:27 gaming kernel:  amdgpu_dm_atomic_commit_tail+0x5f9/0x2c30 [amdgpu]
Mar 29 03:54:27 gaming kernel:  ? __sbitmap_get_word+0x24/0x70
Mar 29 03:54:27 gaming kernel:  ? sbitmap_get+0x97/0x1d0
Mar 29 03:54:27 gaming kernel:  ? blk_mq_do_dispatch_sched+0xa1/0x3b0
Mar 29 03:54:27 gaming kernel:  ? _raw_spin_trylock+0x17/0x60
Mar 29 03:54:27 gaming kernel:  ? _raw_spin_unlock+0x19/0x40
Mar 29 03:54:27 gaming kernel:  ? get_page_from_freelist+0x1436/0x1580
Mar 29 03:54:27 gaming kernel:  ? pre_validate_dsc+0x75/0x430 [amdgpu]
Mar 29 03:54:27 gaming kernel:  ? drm_atomic_helper_check_planes+0x156/0x230 [drm_kms_helper]
Mar 29 03:54:27 gaming kernel:  ? amdgpu_dm_atomic_check+0x6d3/0x1180 [amdgpu]
Mar 29 03:54:27 gaming kernel:  ? preempt_count_add+0x74/0xa0
Mar 29 03:54:27 gaming kernel:  ? _raw_spin_lock_irq+0x1d/0x50
Mar 29 03:54:27 gaming kernel:  ? _raw_spin_unlock_irq+0x1f/0x40
Mar 29 03:54:27 gaming kernel:  ? __wait_for_common+0x1a7/0x1e0
Mar 29 03:54:27 gaming kernel:  ? __pfx_schedule_timeout+0x10/0x10
Mar 29 03:54:27 gaming kernel:  ? preempt_count_add+0x74/0xa0
Mar 29 03:54:27 gaming kernel:  ? _raw_spin_lock_irqsave+0x28/0x60
Mar 29 03:54:27 gaming kernel:  ? complete_all+0x24/0x90
Mar 29 03:54:27 gaming kernel:  commit_tail+0x91/0x130 [drm_kms_helper]
Mar 29 03:54:27 gaming kernel:  drm_atomic_helper_commit+0x11f/0x150 [drm_kms_helper]
Mar 29 03:54:27 gaming kernel:  drm_atomic_commit+0x97/0xd0 [drm]
Mar 29 03:54:27 gaming kernel:  ? __pfx___drm_printfn_info+0x10/0x10 [drm]
Mar 29 03:54:27 gaming kernel:  drm_mode_gamma_set_ioctl+0x399/0x530 [drm]
Mar 29 03:54:27 gaming kernel:  ? __pfx_drm_mode_gamma_set_ioctl+0x10/0x10 [drm]
Mar 29 03:54:27 gaming kernel:  drm_ioctl_kernel+0xb6/0x140 [drm]
Mar 29 03:54:27 gaming kernel:  drm_ioctl+0x22a/0x3e0 [drm]
Mar 29 03:54:27 gaming kernel:  ? __pfx_drm_mode_gamma_set_ioctl+0x10/0x10 [drm]
Mar 29 03:54:27 gaming kernel:  ? ioctl_has_perm.constprop.0.isra.0+0xbd/0x120
Mar 29 03:54:27 gaming kernel:  amdgpu_drm_ioctl+0x4d/0x80 [amdgpu]
Mar 29 03:54:27 gaming kernel:  __x64_sys_ioctl+0x8b/0xc0
Mar 29 03:54:27 gaming kernel:  do_syscall_64+0x3c/0x90
Mar 29 03:54:27 gaming kernel:  entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 29 03:54:27 gaming kernel: RIP: 0033:0x7fcd3a504321
Mar 29 03:54:27 gaming kernel: RSP: 002b:00007fffec233be0 EFLAGS: 00000246 ORIG_RAX:<PHONE_NUMBER>000010
Mar 29 03:54:27 gaming kernel: RAX: ffffffffffffffda RBX: 00007fffec233c70 RCX: 00007fcd3a504321
Mar 29 03:54:27 gaming kernel: RDX: 00007fffec233c70 RSI: 00000000c02064a5 RDI:<PHONE_NUMBER>000010
Mar 29 03:54:27 gaming kernel: RBP: 00000000c02064a5 R08:<PHONE_NUMBER>9d31f0 R09:<PHONE_NUMBER>9d33f0
Mar 29 03:54:27 gaming kernel: R10:<PHONE_NUMBER>000042 R11:<PHONE_NUMBER>000246 R12:<PHONE_NUMBER>9d2b30
Mar 29 03:54:27 gaming kernel: R13:<PHONE_NUMBER>000010 R14:<PHONE_NUMBER>9ce570 R15:<PHONE_NUMBER>000000
Mar 29 03:54:27 gaming kernel:  </TASK>
Mar 29 03:54:27 gaming kernel: INFO: task kworker/u32:26:3921699 blocked for more than 245 seconds.
Mar 29 03:54:27 gaming kernel:       Not tainted 6.2.0 #1-NixOS
Mar 29 03:54:27 gaming kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 03:54:27 gaming kernel: task:kworker/u32:26  state:D stack:0     pid:3921699 ppid:2      flags:0x00004000
Mar 29 03:54:27 gaming kernel: Workqueue: amdgpu-reset-dev drm_sched_job_timedout [gpu_sched]
Mar 29 03:54:27 gaming kernel: Call Trace:
Mar 29 03:54:27 gaming kernel:  <TASK>
Mar 29 03:54:27 gaming kernel:  __schedule+0x30d/0x1270
Mar 29 03:54:27 gaming kernel:  ? __wake_up_klogd.part.0+0x56/0x80
Mar 29 03:54:27 gaming kernel:  ? dev_vprintk_emit+0x175/0x19d
Mar 29 03:54:27 gaming kernel:  schedule+0x61/0xe0
Mar 29 03:54:27 gaming kernel:  schedule_preempt_disabled+0x18/0x30
Mar 29 03:54:27 gaming kernel:  __mutex_lock.constprop.0+0x38d/0x700
Mar 29 03:54:27 gaming kernel:  dm_suspend+0xcc/0x1e0 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_device_ip_suspend_phase1+0x72/0xe0 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_device_ip_suspend+0x20/0x70 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_device_pre_asic_reset+0xd5/0x290 [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_device_gpu_recover.cold+0x5f8/0xb4a [amdgpu]
Mar 29 03:54:27 gaming kernel:  amdgpu_job_timedout+0x18a/0x1c0 [amdgpu]
Mar 29 03:54:27 gaming kernel:  ? _raw_spin_unlock+0x19/0x40
Mar 29 03:54:27 gaming kernel:  drm_sched_job_timedout+0x77/0x110 [gpu_sched]
Mar 29 03:54:27 gaming kernel:  process_one_work+0x1e2/0x3b0
Mar 29 03:54:27 gaming kernel:  worker_thread+0x54/0x3a0
Mar 29 03:54:27 gaming kernel:  ? __pfx_worker_thread+0x10/0x10
Mar 29 03:54:27 gaming kernel:  kthread+0xe9/0x110
Mar 29 03:54:27 gaming kernel:  ? __pfx_kthread+0x10/0x10
Mar 29 03:54:27 gaming kernel:  ret_from_fork+0x29/0x50
Mar 29 03:54:27 gaming kernel:  </TASK>

full system log: amdgpu_crash.txt.zip
probably something upstream (amdgpu) has to fix
Notify maintainers

Metadata

system: "x86_64-linux"
host os: Linux 6.2.6, NixOS, 22.11 (Raccoon), 22.11.3196.cd34d6ed7ba
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.11.1
channels(root): "home-manager-22.11.tar.gz, nixos-22.11, nixos-hardware, nixos-unstable"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

I am running 6.2.6 now, but before the reboot it was 6.2.0 (NixOS 22.11.2999.a7cc81913bb).
Could be already fixed in that version. I watch it.

Had a look at the log, that's a known issue. You have too little free RAM to suspend as suspending means evacuating all VRAM into system memory after the kernel allocator is already in NOIO mode thus cannot swap further. I worked around this with https://git.dolansoft.org/lorenz/memreserver which gets executed before sleep and forces the kernel to keep emough free memory around for supend to succeed. Adjust the amount of memory to your card VRAM +1GiB.
I should make this a NixOS module at some point.

The crash is due to the fact that amdgpu aborts the suspend, which leads to the kernel attempting s2idle which is not properly supported on this platform with an AMD GPU leading to the SMU failing.

that's a known issue. You have too little free RAM to suspend as suspending means evacuating all VRAM into system memory

i have 32 GB RAM, 32 GB SWAP and 8 GB VRAM. i see how that can be an issue in this context when RAM and VRAM are filled
i had reported a similar issue before and there where multiple fixes
https://gitlab.freedesktop.org/drm/amd/-/issues/2223
https://github.com/torvalds/linux/commit/8d4de331f1b24a22d18e3c6116aa25228cf54854 (in 6.1)
https://github.com/systemd/systemd/issues/25151 is still open

AMD did fix that just attempting to go to S3 doesn't result in a GPU reset after TTM fails to evacuate the GPU VRAM, instead it aborts the S3 suspend attempt. But it still isn't able to suspend under memory pressure with a GPU with external VRAM because of some annoying design limitations on the PM subsystem (namely that there are no subsystem constraints and no phases, this has also bitten me on the storage/SCSI side). These limitations mean that it cannot perform writeback or swapping at the point where the VRAM eviction happens. Thus even with a lot of swap or nominally free memory being used as cache you end up with this issue. I'm running 64GiB RAM/96GiB swap and still had the same problem.
systemd just adds fuel to the fire by then attempting to go into s2idle which on AMD does a bunch of work which can result in issues on systems which aren't expected to go into s2idle, but on paper this behavior is acceptable. Crashing/hanging the GPU SMU by doing things not supported on the platform to it is technically on AMD.
I've been running my workaround for more than two years and never had any issues again. It essentially just installs a sleep hook which runs before the kernel actually suspends devices which allocates a bit more memory than the GPU has VRAM, forces the kernel to actually back it with real RAM (by locking it and writing zeroes to it) and then terminates. If the kernel cannot find real free RAM to back this allocation, it has to clear caches, do writebacks or swap out memory pages here while the system is still under normal operation. Then it terminates, leaving behind a large amount of truly free RAM which the kernel can then immediately use to evacuate VRAM into.
I'm thinking of spending some time to make this nice, i.e. a proper module which detects all GPUs with onboard VRAM (it's not needed for most APUs/notebook GPUs which share RAM), sums the amount, adds a fudge factor and then performs this without the need to manually configure anything.

I have an ancient desktop with an AMD dGPU (8GB VRAM) and a laptop with a ryzen 2 CPU (512M VRAM). I have experienced the black screen problem a lot on the laptop but only once or twice on the desktop.
I tried your memreserver with this package def - very hopeful that it does the trick because it’s annoying AF with the laptop.
┌────
│ { lib
│ , stdenv
│ , fetchFromGitLab
│ , gigaBytes ? 9
│ }:
│
│ stdenv.mkDerivation rec {
│   pname = "memreserver";
│   version = "<IP_ADDRESS>00414";
│
│   src = fetchFromGitLab {
│     domain = "git.dolansoft.org";
│     owner = "lorenz";
│     repo = pname;
│     rev = "094963f0a90a6b059240ecc6fff9aeb8213e64cc";
│     hash = "sha256-wLHnOR+lgWFy0IdbQBKKA6HcMLejZHpfScNT9KDfSlw=";
│   };
│
│   postPatch = ''
│     substituteInPlace Makefile
│       --replace /usr/local $out
│       --replace /etc $out/lib
│
│     substituteInPlace main.c
│       --replace 'amount = 5' 'amount = ${toString gigaBytes}'
│       --replace ' 5G' ' ${toString gigaBytes}G'
│
│     substituteInPlace memreserver.service
│       --replace /usr/local $out
│       --replace ' 5G' ' ${toString gigaBytes}G'
│   '';
│
│   preInstall = ''
│     mkdir -p $out/{bin,lib/systemd/system}
│   '';
│
│   meta = with lib; {
│     description = "Reserve memory for AMDGPU VRAM";
│   };
│ }
└────

@peterhoeg AFAIK all Ryzen 2000-series mobile CPUs have an integrated GPU. Unless you also have a separate dedicated GPU on your notebook your issues have a different cause and will not be fixed by memreserver. Integrated GPUs do not have dedicated VRAM but rely on a slice of RAM shared with the CPU which is kept in self-refresh during S3 so it does not need to be evacuated, thus the problem cannot occur there.

So I've done some work on https://git.dolansoft.org/lorenz/memreserver, it now uses libdrm to dynamically determine the amount of RAM to be reserved as well as skipping the process if no GPU is found which requires this. It still only works for AMD GPUs as these are the only ones I've personally experienced the problem on, but I own no Intel dGPUs which are probably also affected.
Please test this improved version and report any issues. If it works out well, I need to rename it to something better (open to suggestions, I haven't found anything good yet) and make a NixOS module for it. Maybe we could even enable it automatically if amdgpu is in initrd.kernelModules or initrd.availableKernelModules, otherwise people need to know to turn the right knob to not get suspend failures.

I have a similar issue, which might be related:
My laptop with Ryzen 6800H & RX 6850M XT fails to suspend.
After I run systemctl suspend or sudo systemctl suspend, my laptop screen goes blank after 0.5 seconds, then it wakes up automatically after 5 seconds & then it goes blank again after 5 seconds. But it still doesn't enter suspend (the power button & other LEDs remain solid instead of blinking).
Curiously, sudo pm-suspend works without any issues. Even more curiously, PM_DEBUG=true sudo pm-suspend has the same behavior as systemctl suspend.

Here is the dmesg output for `systemctl suspend`
[ 3531.806232] PM: suspend entry (deep)
[ 3531.822353] Filesystems sync: 0.016 seconds
[ 3532.087298] Freezing user space processes
[ 3532.088759] Freezing user space processes completed (elapsed 0.001 seconds)
[ 3532.088762] OOM killer disabled.
[ 3532.088763] Freezing remaining freezable tasks
[ 3532.089739] Freezing remaining freezable tasks completed (elapsed 0.000 seconds)
[ 3532.089755] printk: Suspending console(s) (use no_console_suspend to debug)
[ 3532.108526] queueing ieee80211 work while going to suspend
[ 3532.109646] amdgpu 0000:03:00.0: amdgpu: Failed to disallow df cstate
[ 3532.109668] ------------[ cut here ]------------
[ 3532.109669] WARNING: CPU: 14 PID: 33555 at drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:2489 dm_suspend+0x1a6/0x1c0 [amdgpu]
[ 3532.110186] Modules linked in: ccm af_packet uinput cmac algif_hash algif_skcipher af_alg bnep amdgpu msr mt7921e mt7921_common mt76_connac_lib snd_hda_codec_realtek mt76 snd_hda_codec_generic mousedev mac80211 ledtrig_audio snd_hda_codec_hdmi iommu_v2 gpu_sched snd_hda_intel drm_buddy drm_ttm_helper snd_intel_dspcfg ttm btusb snd_intel_sdw_acpi edac_mce_amd btrtl snd_hda_codec drm_display_helper edac_core intel_rapl_msr btbcm intel_rapl_common btintel crc32_pclmul btmtk polyval_clmulni hid_multitouch drm_kms_helper r8169 snd_hda_core polyval_generic gf128mul nls_iso8859_1 ghash_clmulni_intel wmi_bmof bluetooth cfg80211 agpgart sha512_ssse3 ideapad_laptop realtek snd_hwdep sha512_generic sparse_keymap nls_cp437 snd_pcm aesni_intel i2c_algo_bit ucsi_acpi wdat_wdt syscopyarea platform_profile typec_ucsi sp5100_tco vfat ecdh_generic crypto_simd mdio_devres fat snd_timer rfkill cryptd snd ecc typec rapl sysfillrect watchdog tpm_crb video k10temp i2c_piix4 libphy libaes cdc_acm sysimgblt
[ 3532.110260]  soundcore libarc4 thermal roles battery wmi i2c_hid_acpi ip6_tables tpm_tis i2c_hid tpm_tis_core joydev xt_conntrack tiny_power_button nf_conntrack i2c_designware_platform acpi_cpufreq evdev i2c_designware_core acpi_tad nf_defrag_ipv6 input_leds button ac led_class nf_defrag_ipv4 mac_hid serio_raw ip6t_rpfilter ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel libcrc32c nfnetlink ctr loop cpufreq_ondemand tun tap macvlan bridge stp llc kvm_amd ccp kvm drm irqbypass fuse backlight deflate i2c_core efi_pstore configfs efivarfs tpm rng_core dmi_sysfs ip_tables x_tables autofs4 hid_generic ext4 usbhid hid crc32c_generic crc16 mbcache jbd2 xhci_pci xhci_pci_renesas xhci_hcd nvme thunderbolt nvme_core atkbd usbcore libps2 vivaldi_fmap t10_pi crc32c_intel crc64_rocksoft crc64 crc_t10dif i8042 usb_common crct10dif_generic crct10dif_pclmul crct10dif_common rtc_cmos serio dm_mod dax
[ 3532.110334] CPU: 14 PID: 33555 Comm: kworker/u32:16 Tainted: G        W          6.2.10 #1-NixOS
[ 3532.110338] Hardware name: LENOVO 82UH/LNVNB161216, BIOS K9CN34WW 07/22/2022
[ 3532.110341] Workqueue: events_unbound async_run_entry_fn
[ 3532.110349] RIP: 0010:dm_suspend+0x1a6/0x1c0 [amdgpu]
[ 3532.110561] Code: 4c 89 e7 e8 fc eb 1f 00 48 89 ef e8 34 db 00 00 4c 89 f7 e8 0c bb ff ff e9 f0 fe ff ff 4c 89 e6 4c 89 ef e8 ac 15 20 00 eb d6 <0f> 0b e9 9b fe ff ff e8 9e 25 23 e6 66 66 2e 0f 1f 84 00 00 00 00
[ 3532.110563] RSP: 0018:ffffa8c94a847d10 EFLAGS: 00010286
[ 3532.110565] RAX:<PHONE_NUMBER>000000 RBX: ffff8b198d218980 RCX:<PHONE_NUMBER>000000
[ 3532.110566] RDX:<PHONE_NUMBER>000000 RSI: ffffffffa816606a RDI: ffff8b198d200000
[ 3532.110567] RBP: ffff8b198d200000 R08: ffffffffa8857120 R09: 00000000a91f06f6
[ 3532.110567] R10: ffffffffffffffff R11: ffffffffa890d0d8 R12: ffff8b198d200000
[ 3532.110568] R13:<PHONE_NUMBER>000001 R14: ffff8b198d2169f8 R15: ffff8b198e5b7808
[ 3532.110569] FS:<PHONE_NUMBER>000000(0000) GS:ffff8b1cae980000(0000) knlGS:0000000000000000
[ 3532.110570] CS:  0010 DS: 0000 ES: 0000 CR0:<PHONE_NUMBER>050033
[ 3532.110571] CR2: 00007f666628d81c CR3: 000000019a410000 CR4:<PHONE_NUMBER>750ee0
[ 3532.110572] PKRU: 55555554
[ 3532.110572] Call Trace:
[ 3532.110575]  <TASK>
[ 3532.110580]  amdgpu_device_ip_suspend_phase1+0x71/0xe0 [amdgpu]
[ 3532.110710]  amdgpu_device_suspend+0xc7/0x180 [amdgpu]
[ 3532.110838]  pci_pm_suspend+0x77/0x160
[ 3532.110842]  ? __pfx_pci_pm_suspend+0x10/0x10
[ 3532.110844]  dpm_run_callback+0x8c/0x1e0
[ 3532.110847]  __device_suspend+0xf1/0x4f0
[ 3532.110849]  async_suspend+0x1e/0x70
[ 3532.110851]  async_run_entry_fn+0x34/0x130
[ 3532.110853]  process_one_work+0x1c8/0x3c0
[ 3532.110857]  worker_thread+0x51/0x390
[ 3532.110859]  ? __pfx_worker_thread+0x10/0x10
[ 3532.110861]  kthread+0xed/0x120
[ 3532.110863]  ? __pfx_kthread+0x10/0x10
[ 3532.110865]  ret_from_fork+0x2c/0x50
[ 3532.110869]  </TASK>
[ 3532.110870] ---[ end trace<PHONE_NUMBER>000000 ]---
[ 3532.405191] amdgpu 0000:03:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
[ 3532.405338] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* KGQ disable failed
[ 3538.036639] amdgpu 0000:03:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000000 SMN_C2PMSG_82:0x00000000
[ 3538.036642] amdgpu 0000:03:00.0: amdgpu: Failed to disable smu features.
[ 3538.036645] amdgpu 0000:03:00.0: amdgpu: Fail to disable dpm features!
[ 3538.036646] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <smu> failed -62
[ 3540.036736] [drm] psp gfx command UNKNOWN CMD(0xFFFFFFFF) failed and response status is (0xFFFFFFFF)
[ 3540.036740] [drm:psp_suspend [amdgpu]] *ERROR* Failed to terminate tmr
[ 3540.036903] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <psp> failed -22
[ 3540.039748] ACPI: EC: interrupt blocked
[ 3540.068622] amdgpu 0000:03:00.0: amdgpu: MODE1 reset
[ 3540.068631] amdgpu 0000:03:00.0: amdgpu: GPU mode1 reset
[ 3540.165412] nvme 0000:06:00.0: VC buffer not found in pci_save_vc_state
[ 3540.166554] amdgpu 0000:03:00.0: amdgpu: GPU psp mode1 reset
[ 3540.421912] [drm] psp is not working correctly before mode1 reset!
[ 3540.421913] amdgpu 0000:03:00.0: amdgpu: GPU mode1 reset failed
[ 3540.421963] amdgpu 0000:03:00.0: PM: pci_pm_suspend_noirq(): amdgpu_pmops_suspend_noirq+0x0/0x40 [amdgpu] returns -22
[ 3540.422095] amdgpu 0000:03:00.0: PM: dpm_run_callback(): pci_pm_suspend_noirq+0x0/0x2a0 returns -22
[ 3540.422100] amdgpu 0000:03:00.0: PM: failed to suspend async: error -22
[ 3540.422630] ACPI: EC: interrupt unblocked
[ 3540.647408] PM: noirq suspend of devices failed
[ 3540.649004] [drm] PCIE GART of 512M enabled (table at 0x00000082FEB00000).
[ 3540.649029] [drm] PSP is resuming...
[ 3540.649207] [drm] PCIE GART of 1024M enabled (table at 0x000000F41FC00000).
[ 3540.649238] [drm] PSP is resuming...
[ 3540.666039] nvme nvme0: Shutdown timeout set to 10 seconds
[ 3540.669318] nvme nvme0: 16/0/0 default/read/poll queues
[ 3540.671128] [drm] reserve 0xa00000 from 0xf41e000000 for PSP TMR
[ 3541.441333] amdgpu 0000:36:00.0: amdgpu: RAS: optional ras ta ucode is not available
[ 3541.451720] amdgpu 0000:36:00.0: amdgpu: RAP: optional rap ta ucode is not available
[ 3541.451722] amdgpu 0000:36:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
[ 3541.451724] amdgpu 0000:36:00.0: amdgpu: SMU is resuming...
[ 3541.452093] amdgpu 0000:36:00.0: amdgpu: SMU is resumed successfully!
[ 3541.453532] [drm] DMUB hardware initialized: version=0x0400002E
[ 3541.659457] [drm] kiq ring mec 2 pipe 1 q 0
[ 3541.663593] [drm] VCN decode and encode initialized successfully(under DPG Mode).
[ 3541.664169] [drm] JPEG decode initialized successfully.
[ 3541.664181] amdgpu 0000:36:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[ 3541.664183] amdgpu 0000:36:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ 3541.664185] amdgpu 0000:36:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ 3541.664185] amdgpu 0000:36:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
[ 3541.664186] amdgpu 0000:36:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
[ 3541.664187] amdgpu 0000:36:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
[ 3541.664188] amdgpu 0000:36:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
[ 3541.664188] amdgpu 0000:36:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
[ 3541.664189] amdgpu 0000:36:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
[ 3541.664190] amdgpu 0000:36:00.0: amdgpu: ring kiq_2.1.0 uses VM inv eng 11 on hub 0
[ 3541.664191] amdgpu 0000:36:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[ 3541.664192] amdgpu 0000:36:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 1
[ 3541.664193] amdgpu 0000:36:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 1
[ 3541.664193] amdgpu 0000:36:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 1
[ 3541.664194] amdgpu 0000:36:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 1
[ 3549.278279] [drm:psp_v11_0_memory_training [amdgpu]] *ERROR* send training msg failed.
[ 3549.278474] [drm:psp_resume [amdgpu]] *ERROR* Failed to process memory training!
[ 3549.278630] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -62
[ 3549.278765] amdgpu 0000:03:00.0: amdgpu: amdgpu_device_ip_resume failed (-62).
[ 3549.278766] amdgpu 0000:03:00.0: PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -62
[ 3549.278772] amdgpu 0000:03:00.0: PM: failed to resume async: error -62
[ 3549.281863] OOM killer enabled.
[ 3549.281865] Restarting tasks ... done.
[ 3549.287585] random: crng reseeded on system resumption
[ 3549.600078] PM: suspend exit
[ 3549.600179] PM: suspend entry (s2idle)
[ 3549.617504] Filesystems sync: 0.017 seconds
[ 3549.619985] Freezing user space processes
[ 3549.621478] Freezing user space processes completed (elapsed 0.001 seconds)
[ 3549.621480] OOM killer disabled.
[ 3549.621481] Freezing remaining freezable tasks
[ 3549.622600] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[ 3549.622606] printk: Suspending console(s) (use no_console_suspend to debug)
[ 3549.641515] queueing ieee80211 work while going to suspend
[ 3549.642135] amdgpu 0000:03:00.0: amdgpu: Failed to disallow df cstate
[ 3549.642146] ------------[ cut here ]------------
[ 3549.642147] WARNING: CPU: 6 PID: 33609 at drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:2489 dm_suspend+0x1a6/0x1c0 [amdgpu]
[ 3549.642384] Modules linked in: ccm af_packet uinput cmac algif_hash algif_skcipher af_alg bnep amdgpu msr mt7921e mt7921_common mt76_connac_lib snd_hda_codec_realtek mt76 snd_hda_codec_generic mousedev mac80211 ledtrig_audio snd_hda_codec_hdmi iommu_v2 gpu_sched snd_hda_intel drm_buddy drm_ttm_helper snd_intel_dspcfg ttm btusb snd_intel_sdw_acpi edac_mce_amd btrtl snd_hda_codec drm_display_helper edac_core intel_rapl_msr btbcm intel_rapl_common btintel crc32_pclmul btmtk polyval_clmulni hid_multitouch drm_kms_helper r8169 snd_hda_core polyval_generic gf128mul nls_iso8859_1 ghash_clmulni_intel wmi_bmof bluetooth cfg80211 agpgart sha512_ssse3 ideapad_laptop realtek snd_hwdep sha512_generic sparse_keymap nls_cp437 snd_pcm aesni_intel i2c_algo_bit ucsi_acpi wdat_wdt syscopyarea platform_profile typec_ucsi sp5100_tco vfat ecdh_generic crypto_simd mdio_devres fat snd_timer rfkill cryptd snd ecc typec rapl sysfillrect watchdog tpm_crb video k10temp i2c_piix4 libphy libaes cdc_acm sysimgblt
[ 3549.642426]  soundcore libarc4 thermal roles battery wmi i2c_hid_acpi ip6_tables tpm_tis i2c_hid tpm_tis_core joydev xt_conntrack tiny_power_button nf_conntrack i2c_designware_platform acpi_cpufreq evdev i2c_designware_core acpi_tad nf_defrag_ipv6 input_leds button ac led_class nf_defrag_ipv4 mac_hid serio_raw ip6t_rpfilter ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel libcrc32c nfnetlink ctr loop cpufreq_ondemand tun tap macvlan bridge stp llc kvm_amd ccp kvm drm irqbypass fuse backlight deflate i2c_core efi_pstore configfs efivarfs tpm rng_core dmi_sysfs ip_tables x_tables autofs4 hid_generic ext4 usbhid hid crc32c_generic crc16 mbcache jbd2 xhci_pci xhci_pci_renesas xhci_hcd nvme thunderbolt nvme_core atkbd usbcore libps2 vivaldi_fmap t10_pi crc32c_intel crc64_rocksoft crc64 crc_t10dif i8042 usb_common crct10dif_generic crct10dif_pclmul crct10dif_common rtc_cmos serio dm_mod dax
[ 3549.642476] CPU: 6 PID: 33609 Comm: kworker/u32:70 Tainted: G        W          6.2.10 #1-NixOS
[ 3549.642478] Hardware name: LENOVO 82UH/LNVNB161216, BIOS K9CN34WW 07/22/2022
[ 3549.642480] Workqueue: events_unbound async_run_entry_fn
[ 3549.642484] RIP: 0010:dm_suspend+0x1a6/0x1c0 [amdgpu]
[ 3549.642677] Code: 4c 89 e7 e8 fc eb 1f 00 48 89 ef e8 34 db 00 00 4c 89 f7 e8 0c bb ff ff e9 f0 fe ff ff 4c 89 e6 4c 89 ef e8 ac 15 20 00 eb d6 <0f> 0b e9 9b fe ff ff e8 9e 25 23 e6 66 66 2e 0f 1f 84 00 00 00 00
[ 3549.642679] RSP: 0018:ffffa8c9433f7d10 EFLAGS: 00010282
[ 3549.642680] RAX:<PHONE_NUMBER>000000 RBX: ffff8b198d218980 RCX:<PHONE_NUMBER>000000
[ 3549.642681] RDX:<PHONE_NUMBER>000000 RSI: ffffffffa816606a RDI: ffff8b198d200000
[ 3549.642682] RBP: ffff8b198d200000 R08: ffffffffa8857120 R09: 00000000a91f2b96
[ 3549.642683] R10: ffffffffffffffff R11: ffffffffa890dbe8 R12: ffff8b198d200000
[ 3549.642684] R13:<PHONE_NUMBER>000001 R14: ffff8b198d2169f8 R15: ffff8b198e5b70e8
[ 3549.642685] FS:<PHONE_NUMBER>000000(0000) GS:ffff8b1cae780000(0000) knlGS:0000000000000000
[ 3549.642686] CS:  0010 DS: 0000 ES: 0000 CR0:<PHONE_NUMBER>050033
[ 3549.642686] CR2: 00007f666627edd4 CR3: 00000001e086c000 CR4:<PHONE_NUMBER>750ee0
[ 3549.642687] PKRU: 55555554
[ 3549.642688] Call Trace:
[ 3549.642691]  <TASK>
[ 3549.642695]  amdgpu_device_ip_suspend_phase1+0x71/0xe0 [amdgpu]
[ 3549.642828]  amdgpu_device_suspend+0xc7/0x180 [amdgpu]
[ 3549.642951]  pci_pm_suspend+0x77/0x160
[ 3549.642954]  ? __pfx_pci_pm_suspend+0x10/0x10
[ 3549.642956]  dpm_run_callback+0x8c/0x1e0
[ 3549.642959]  __device_suspend+0xf1/0x4f0
[ 3549.642962]  async_suspend+0x1e/0x70
[ 3549.642964]  async_run_entry_fn+0x34/0x130
[ 3549.642966]  process_one_work+0x1c8/0x3c0
[ 3549.642970]  worker_thread+0x51/0x390
[ 3549.642972]  ? __pfx_worker_thread+0x10/0x10
[ 3549.642974]  kthread+0xed/0x120
[ 3549.642976]  ? __pfx_kthread+0x10/0x10
[ 3549.642977]  ret_from_fork+0x2c/0x50
[ 3549.642981]  </TASK>
[ 3549.642982] ---[ end trace<PHONE_NUMBER>000000 ]---
[ 3549.939679] amdgpu 0000:03:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
[ 3549.939825] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* KGQ disable failed
[ 3555.600745] amdgpu 0000:03:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000000 SMN_C2PMSG_82:0x00000000
[ 3555.600747] amdgpu 0000:03:00.0: amdgpu: Failed to disable smu features.
[ 3555.600750] amdgpu 0000:03:00.0: amdgpu: Fail to disable dpm features!
[ 3555.600751] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <smu> failed -62
[ 3557.600740] [drm] psp gfx command UNKNOWN CMD(0xFFFFFFFF) failed and response status is (0xFFFFFFFF)
[ 3557.600743] [drm:psp_suspend [amdgpu]] *ERROR* Failed to terminate tmr
[ 3557.601080] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <psp> failed -22
[ 3557.604179] ACPI: EC: interrupt blocked
[ 3575.436015] ACPI: EC: interrupt unblocked
[ 3576.337058] [drm] PCIE GART of 1024M enabled (table at 0x000000F41FC00000).
[ 3576.337082] [drm] PSP is resuming...
[ 3576.343656] [drm] PCIE GART of 512M enabled (table at 0x00000082FEB00000).
[ 3576.343677] [drm] PSP is resuming...
[ 3576.359216] [drm] reserve 0xa00000 from 0xf41e000000 for PSP TMR
[ 3577.132397] amdgpu 0000:36:00.0: amdgpu: RAS: optional ras ta ucode is not available
[ 3577.142704] amdgpu 0000:36:00.0: amdgpu: RAP: optional rap ta ucode is not available
[ 3577.142706] amdgpu 0000:36:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
[ 3577.142708] amdgpu 0000:36:00.0: amdgpu: SMU is resuming...
[ 3577.143699] amdgpu 0000:36:00.0: amdgpu: SMU is resumed successfully!
[ 3577.145070] [drm] DMUB hardware initialized: version=0x0400002E
[ 3577.352164] [drm] kiq ring mec 2 pipe 1 q 0
[ 3577.356467] [drm] VCN decode and encode initialized successfully(under DPG Mode).
[ 3577.357056] [drm] JPEG decode initialized successfully.
[ 3577.357064] amdgpu 0000:36:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[ 3577.357066] amdgpu 0000:36:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ 3577.357067] amdgpu 0000:36:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ 3577.357068] amdgpu 0000:36:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
[ 3577.357068] amdgpu 0000:36:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
[ 3577.357069] amdgpu 0000:36:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
[ 3577.357070] amdgpu 0000:36:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
[ 3577.357070] amdgpu 0000:36:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
[ 3577.357071] amdgpu 0000:36:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
[ 3577.357072] amdgpu 0000:36:00.0: amdgpu: ring kiq_2.1.0 uses VM inv eng 11 on hub 0
[ 3577.357073] amdgpu 0000:36:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[ 3577.357073] amdgpu 0000:36:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 1
[ 3577.357074] amdgpu 0000:36:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 1
[ 3577.357075] amdgpu 0000:36:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 1
[ 3577.357076] amdgpu 0000:36:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 1
[ 3579.943317] [drm:psp_hw_start [amdgpu]] *ERROR* PSP load kdb failed!
[ 3579.943490] [drm:psp_resume [amdgpu]] *ERROR* PSP resume failed
[ 3579.943640] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -62
[ 3579.943783] amdgpu 0000:03:00.0: amdgpu: amdgpu_device_ip_resume failed (-62).
[ 3579.943784] amdgpu 0000:03:00.0: PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -62
[ 3579.943795] amdgpu 0000:03:00.0: PM: failed to resume async: error -62
[ 3579.947575] OOM killer enabled.
[ 3579.947579] Restarting tasks ... done.
[ 3579.950001] random: crng reseeded on system resumption
[ 3580.262595] PM: suspend exit

Here is the dmesg output for `sudo pm-suspend`
4858.351615]  </TASK>
[ 4858.351615] ---[ end trace<PHONE_NUMBER>000000 ]---
[ 4858.581783] amdgpu 0000:03:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
[ 4858.581956] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* KGQ disable failed
[ 4858.803973] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* failed to halt cp gfx
[ 4858.804214] amdgpu 0000:03:00.0: amdgpu: SMU: response:0xFFFFFFFF for index:7 param:0x00000000 message:DisableAllSmuFeatures?
[ 4858.804217] amdgpu 0000:03:00.0: amdgpu: Failed to disable smu features.
[ 4858.804220] amdgpu 0000:03:00.0: amdgpu: Fail to disable dpm features!
[ 4858.804220] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <smu> failed -121
[ 4858.804366] amdgpu 0000:03:00.0: amdgpu: SMU: response:0xFFFFFFFF for index:46 param:0x00000000 message:PrepareMp1ForUnload?
[ 4858.804367] amdgpu 0000:03:00.0: amdgpu: [PrepareMp1] Failed!
[ 4858.804368] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* SMC failed to set mp1 state 2, -121
[ 4866.069225] [drm] PSP is resuming...
[ 4866.300264] [drm:psp_hw_start [amdgpu]] *ERROR* PSP create ring failed!
[ 4866.300404] [drm:psp_resume [amdgpu]] *ERROR* PSP resume failed
[ 4866.300503] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -62
[ 4866.300592] amdgpu 0000:03:00.0: amdgpu: amdgpu_device_ip_resume failed (-62).
[ 4909.124991] PM: suspend entry (deep)
[ 4909.126770] Filesystems sync: 0.001 seconds
[ 4909.393629] Freezing user space processes
[ 4909.427025] Freezing user space processes completed (elapsed 0.033 seconds)
[ 4909.427038] OOM killer disabled.
[ 4909.427040] Freezing remaining freezable tasks
[ 4909.428163] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[ 4909.428241] printk: Suspending console(s) (use no_console_suspend to debug)
[ 4909.428713] wlp4s0: deauthenticating from 30:cc:21:e3:6c:62 by local choice (Reason: 3=DEAUTH_LEAVING)
[ 4909.657909] r8169 0000:05:00.0 enp5s0: Link is Down
[ 4910.038318] amdgpu 0000:03:00.0: amdgpu: Failed to disallow df cstate
[ 4910.038329] ------------[ cut here ]------------
[ 4910.038330] WARNING: CPU: 12 PID: 49361 at drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:2489 dm_suspend+0x1a6/0x1c0 [amdgpu]
[ 4910.038554] Modules linked in: snd_seq_dummy snd_seq snd_seq_device ccm af_packet uinput cmac algif_hash algif_skcipher af_alg bnep amdgpu msr mt7921e mt7921_common mt76_connac_lib snd_hda_codec_realtek mt76 snd_hda_codec_generic mousedev mac80211 ledtrig_audio snd_hda_codec_hdmi iommu_v2 gpu_sched snd_hda_intel drm_buddy drm_ttm_helper snd_intel_dspcfg ttm btusb snd_intel_sdw_acpi edac_mce_amd btrtl snd_hda_codec drm_display_helper edac_core intel_rapl_msr btbcm intel_rapl_common btintel crc32_pclmul btmtk polyval_clmulni hid_multitouch drm_kms_helper r8169 snd_hda_core polyval_generic gf128mul nls_iso8859_1 ghash_clmulni_intel wmi_bmof bluetooth cfg80211 agpgart sha512_ssse3 ideapad_laptop realtek snd_hwdep sha512_generic sparse_keymap nls_cp437 snd_pcm aesni_intel i2c_algo_bit ucsi_acpi wdat_wdt syscopyarea platform_profile typec_ucsi sp5100_tco vfat ecdh_generic crypto_simd mdio_devres fat snd_timer rfkill cryptd snd ecc typec rapl sysfillrect watchdog tpm_crb video k10temp
[ 4910.038592]  i2c_piix4 libphy libaes cdc_acm sysimgblt soundcore libarc4 thermal roles battery wmi i2c_hid_acpi ip6_tables tpm_tis i2c_hid tpm_tis_core joydev xt_conntrack tiny_power_button nf_conntrack i2c_designware_platform acpi_cpufreq evdev i2c_designware_core acpi_tad nf_defrag_ipv6 input_leds button ac led_class nf_defrag_ipv4 mac_hid serio_raw ip6t_rpfilter ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables sch_fq_codel libcrc32c nfnetlink ctr loop cpufreq_ondemand tun tap macvlan bridge stp llc kvm_amd ccp kvm drm irqbypass fuse backlight deflate i2c_core efi_pstore configfs efivarfs tpm rng_core dmi_sysfs ip_tables x_tables autofs4 hid_generic ext4 usbhid hid crc32c_generic crc16 mbcache jbd2 xhci_pci xhci_pci_renesas xhci_hcd nvme thunderbolt nvme_core atkbd usbcore libps2 vivaldi_fmap t10_pi crc32c_intel crc64_rocksoft crc64 crc_t10dif i8042 usb_common crct10dif_generic crct10dif_pclmul crct10dif_common rtc_cmos serio dm_mod dax
[ 4910.038636] CPU: 12 PID: 49361 Comm: kworker/u32:31 Tainted: G        W          6.2.10 #1-NixOS
[ 4910.038638] Hardware name: LENOVO 82UH/LNVNB161216, BIOS K9CN34WW 07/22/2022
[ 4910.038640] Workqueue: events_unbound async_run_entry_fn
[ 4910.038644] RIP: 0010:dm_suspend+0x1a6/0x1c0 [amdgpu]
[ 4910.038844] Code: 4c 89 e7 e8 fc eb 1f 00 48 89 ef e8 34 db 00 00 4c 89 f7 e8 0c bb ff ff e9 f0 fe ff ff 4c 89 e6 4c 89 ef e8 ac 15 20 00 eb d6 <0f> 0b e9 9b fe ff ff e8 9e 25 23 e6 66 66 2e 0f 1f 84 00 00 00 00
[ 4910.038845] RSP: 0018:ffffa8c94b3dfd10 EFLAGS: 00010282
[ 4910.038847] RAX:<PHONE_NUMBER>000000 RBX: ffff8b198d218980 RCX:<PHONE_NUMBER>000000
[ 4910.038848] RDX:<PHONE_NUMBER>000000 RSI: ffffffffa816606a RDI: ffff8b198d200000
[ 4910.038848] RBP: ffff8b198d200000 R08: ffffffffa8857120 R09: 00000000a91c4786
[ 4910.038849] R10: ffffffffffffffff R11: ffffffffa8913138 R12: ffff8b198d200000
[ 4910.038850] R13:<PHONE_NUMBER>000001 R14: ffff8b198d2169f8 R15: ffff8b198bdd00e8
[ 4910.038851] FS:<PHONE_NUMBER>000000(0000) GS:ffff8b1cae900000(0000) knlGS:0000000000000000
[ 4910.038852] CS:  0010 DS: 0000 ES: 0000 CR0:<PHONE_NUMBER>050033
[ 4910.038853] CR2: 00007f666627238c CR3: 000000019a410000 CR4:<PHONE_NUMBER>750ee0
[ 4910.038854] PKRU: 55555554
[ 4910.038854] Call Trace:
[ 4910.038856]  <TASK>
[ 4910.038861]  amdgpu_device_ip_suspend_phase1+0x71/0xe0 [amdgpu]
[ 4910.038991]  amdgpu_device_suspend+0xc7/0x180 [amdgpu]
[ 4910.039120]  pci_pm_suspend+0x77/0x160
[ 4910.039124]  ? __pfx_pci_pm_suspend+0x10/0x10
[ 4910.039125]  dpm_run_callback+0x8c/0x1e0
[ 4910.039129]  __device_suspend+0xf1/0x4f0
[ 4910.039131]  async_suspend+0x1e/0x70
[ 4910.039132]  async_run_entry_fn+0x34/0x130
[ 4910.039135]  process_one_work+0x1c8/0x3c0
[ 4910.039138]  worker_thread+0x51/0x390
[ 4910.039141]  ? __pfx_worker_thread+0x10/0x10
[ 4910.039143]  kthread+0xed/0x120
[ 4910.039145]  ? __pfx_kthread+0x10/0x10
[ 4910.039147]  ret_from_fork+0x2c/0x50
[ 4910.039151]  </TASK>
[ 4910.039151] ---[ end trace<PHONE_NUMBER>000000 ]---
[ 4910.354499] amdgpu 0000:03:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
[ 4910.354636] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* KGQ disable failed
[ 4910.628795] [drm:gfx_v10_0_hw_fini [amdgpu]] *ERROR* failed to halt cp gfx
[ 4910.629023] amdgpu 0000:03:00.0: amdgpu: SMU: response:0xFFFFFFFF for index:7 param:0x00000000 message:DisableAllSmuFeatures?
[ 4910.629026] amdgpu 0000:03:00.0: amdgpu: Failed to disable smu features.
[ 4910.629029] amdgpu 0000:03:00.0: amdgpu: Fail to disable dpm features!
[ 4910.629030] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <smu> failed -121
[ 4910.629267] [drm:psp_ring_cmd_submit [amdgpu]] *ERROR* ring_buffer_start = 000000004f562829; ring_buffer_end = 00000000ce6a8326; write_frame =<PHONE_NUMBER>103cdd
[ 4910.629412] [drm:psp_ring_cmd_submit [amdgpu]] *ERROR* write_frame is pointing to address out of bounds
[ 4910.629553] [drm:psp_suspend [amdgpu]] *ERROR* Failed to terminate tmr
[ 4910.629693] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <psp> failed -22
[ 4911.502922] ACPI: EC: interrupt blocked
[ 4911.628687] nvme 0000:06:00.0: VC buffer not found in pci_save_vc_state
[ 4918.043441] ACPI: PM: Preparing to enter system sleep state S3
[ 4918.150834] ACPI: EC: event blocked
[ 4918.150837] ACPI: EC: EC stopped
[ 4918.150838] ACPI: PM: Saving platform NVS memory
[ 4918.152724] Disabling non-boot CPUs ...
[ 4918.155456] smpboot: CPU 1 is now offline
[ 4918.158258] smpboot: CPU 2 is now offline
[ 4918.160781] smpboot: CPU 3 is now offline
[ 4918.162921] smpboot: CPU 4 is now offline
[ 4918.165054] smpboot: CPU 5 is now offline
[ 4918.167262] smpboot: CPU 6 is now offline
[ 4918.169317] smpboot: CPU 7 is now offline
[ 4918.171267] smpboot: CPU 8 is now offline
[ 4918.173377] smpboot: CPU 9 is now offline
[ 4918.175415] smpboot: CPU 10 is now offline
[ 4918.177343] smpboot: CPU 11 is now offline
[ 4918.179173] smpboot: CPU 12 is now offline
[ 4918.180938] smpboot: CPU 13 is now offline
[ 4918.182786] smpboot: CPU 14 is now offline
[ 4918.183156] Spectre V2 : Update user space SMT mitigation: STIBP off
[ 4918.184414] smpboot: CPU 15 is now offline
[ 4918.185152] ACPI: PM: Low-level resume complete
[ 4918.185180] ACPI: EC: EC started
[ 4918.185181] ACPI: PM: Restoring platform NVS memory
[ 4918.186189] AMD-Vi: Virtual APIC enabled
[ 4918.186293] AMD-Vi: Virtual APIC enabled
[ 4918.186323] LVT offset 0 assigned for vector 0x400
[ 4918.186824] Enabling non-boot CPUs ...
[ 4918.186860] x86: Booting SMP configuration:
[ 4918.186861] smpboot: Booting Node 0 Processor 1 APIC 0x1
[ 4918.189459] ACPI: \_SB_.PLTF.C001: Found 3 idle states
[ 4918.189688] Spectre V2 : Update user space SMT mitigation: STIBP always-on
[ 4918.189695] CPU1 is up
[ 4918.189717] smpboot: Booting Node 0 Processor 2 APIC 0x2
[ 4918.192034] ACPI: \_SB_.PLTF.C002: Found 3 idle states
[ 4918.192261] CPU2 is up
[ 4918.192277] smpboot: Booting Node 0 Processor 3 APIC 0x3
[ 4918.194641] ACPI: \_SB_.PLTF.C003: Found 3 idle states
[ 4918.195010] CPU3 is up
[ 4918.195025] smpboot: Booting Node 0 Processor 4 APIC 0x4
[ 4918.197823] ACPI: \_SB_.PLTF.C004: Found 3 idle states
[ 4918.198133] CPU4 is up
[ 4918.198147] smpboot: Booting Node 0 Processor 5 APIC 0x5
[ 4918.200430] ACPI: \_SB_.PLTF.C005: Found 3 idle states
[ 4918.200880] CPU5 is up
[ 4918.200897] smpboot: Booting Node 0 Processor 6 APIC 0x6
[ 4918.203245] ACPI: \_SB_.PLTF.C006: Found 3 idle states
[ 4918.203618] CPU6 is up
[ 4918.203631] smpboot: Booting Node 0 Processor 7 APIC 0x7
[ 4918.205931] ACPI: \_SB_.PLTF.C007: Found 3 idle states
[ 4918.206480] CPU7 is up
[ 4918.206492] smpboot: Booting Node 0 Processor 8 APIC 0x8
[ 4918.208849] ACPI: \_SB_.PLTF.C008: Found 3 idle states
[ 4918.209310] CPU8 is up
[ 4918.209324] smpboot: Booting Node 0 Processor 9 APIC 0x9
[ 4918.211627] ACPI: \_SB_.PLTF.C009: Found 3 idle states
[ 4918.212275] CPU9 is up
[ 4918.212288] smpboot: Booting Node 0 Processor 10 APIC 0xa
[ 4918.214647] ACPI: \_SB_.PLTF.C00A: Found 3 idle states
[ 4918.215171] CPU10 is up
[ 4918.215186] smpboot: Booting Node 0 Processor 11 APIC 0xb
[ 4918.217487] ACPI: \_SB_.PLTF.C00B: Found 3 idle states
[ 4918.218262] CPU11 is up
[ 4918.218275] smpboot: Booting Node 0 Processor 12 APIC 0xc
[ 4918.220641] ACPI: \_SB_.PLTF.C00C: Found 3 idle states
[ 4918.221244] CPU12 is up
[ 4918.221257] smpboot: Booting Node 0 Processor 13 APIC 0xd
[ 4918.223568] ACPI: \_SB_.PLTF.C00D: Found 3 idle states
[ 4918.224434] CPU13 is up
[ 4918.224447] smpboot: Booting Node 0 Processor 14 APIC 0xe
[ 4918.226829] ACPI: \_SB_.PLTF.C00E: Found 3 idle states
[ 4918.227526] CPU14 is up
[ 4918.227539] smpboot: Booting Node 0 Processor 15 APIC 0xf
[ 4918.229872] ACPI: \_SB_.PLTF.C00F: Found 3 idle states
[ 4918.230820] CPU15 is up
[ 4918.233478] ACPI: PM: Waking up from system sleep state S3
[ 4918.235838] ACPI: EC: interrupt unblocked
[ 4918.568715] ACPI: EC: event unblocked
[ 4918.569313] [drm] PCIE GART of 1024M enabled (table at 0x000000F41FC00000).
[ 4918.569336] [drm] PSP is resuming...
[ 4918.572722] [drm] PCIE GART of 512M enabled (table at 0x00000082FEB00000).
[ 4918.572744] [drm] PSP is resuming...
[ 4918.586203] nvme nvme0: Shutdown timeout set to 10 seconds
[ 4918.589687] nvme nvme0: 16/0/0 default/read/poll queues
[ 4918.591528] [drm] reserve 0xa00000 from 0xf41e000000 for PSP TMR
[ 4918.736378] r8169 0000:05:00.0 enp5s0: Link is Down
[ 4918.813928] [drm:psp_hw_start [amdgpu]] *ERROR* PSP create ring failed!
[ 4918.814095] [drm:psp_resume [amdgpu]] *ERROR* PSP resume failed
[ 4918.814234] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -62
[ 4918.814356] amdgpu 0000:03:00.0: amdgpu: amdgpu_device_ip_resume failed (-62).
[ 4918.814358] amdgpu 0000:03:00.0: PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -62
[ 4918.814364] amdgpu 0000:03:00.0: PM: failed to resume async: error -62
[ 4918.841629] amdgpu 0000:36:00.0: amdgpu: RAS: optional ras ta ucode is not available
[ 4918.852220] amdgpu 0000:36:00.0: amdgpu: RAP: optional rap ta ucode is not available
[ 4918.852222] amdgpu 0000:36:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
[ 4918.852225] amdgpu 0000:36:00.0: amdgpu: SMU is resuming...
[ 4918.855407] amdgpu 0000:36:00.0: amdgpu: SMU is resumed successfully!
[ 4918.857102] [drm] DMUB hardware initialized: version=0x0400002E
[ 4918.919248] snd_hda_intel 0000:03:00.1: CORB reset timeout#2, CORBRP = 65535
[ 4919.723574] [drm] kiq ring mec 2 pipe 1 q 0
[ 4919.727297] [drm] VCN decode and encode initialized successfully(under DPG Mode).
[ 4919.727803] [drm] JPEG decode initialized successfully.
[ 4919.727813] amdgpu 0000:36:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[ 4919.727816] amdgpu 0000:36:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ 4919.727818] amdgpu 0000:36:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ 4919.727819] amdgpu 0000:36:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
[ 4919.727820] amdgpu 0000:36:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
[ 4919.727821] amdgpu 0000:36:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
[ 4919.727821] amdgpu 0000:36:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
[ 4919.727822] amdgpu 0000:36:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
[ 4919.727823] amdgpu 0000:36:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
[ 4919.727824] amdgpu 0000:36:00.0: amdgpu: ring kiq_2.1.0 uses VM inv eng 11 on hub 0
[ 4919.727825] amdgpu 0000:36:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[ 4919.727825] amdgpu 0000:36:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 1
[ 4919.727826] amdgpu 0000:36:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 1
[ 4919.727827] amdgpu 0000:36:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 1
[ 4919.727828] amdgpu 0000:36:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 1
[ 4919.739549] OOM killer enabled.
[ 4919.739552] Restarting tasks ... done.
[ 4919.741941] random: crng reseeded on system resumption
[ 4919.890245] PM: suspend exit

@utkarshgupta137 That's an unrelated kernel issue, please report it to https://gitlab.freedesktop.org/drm/amd

I'm inclined to close this issue since there is nothing we can really do about upstream issues. If it turns out @lorenz' workaround is effective at mitigating this issue, you can create a feature request for implementing it.

I have a similar issue, which might be related: My laptop with Ryzen 6800H & RX 6850M XT fails to suspend. After I run systemctl suspend or sudo systemctl suspend, my laptop screen goes blank after 0.5 seconds, then it wakes up automatically after 5 seconds & then it goes blank again after 5 seconds. But it still doesn't enter suspend (the power button & other LEDs remain solid instead of blinking). Curiously, sudo pm-suspend works without any issues. Even more curiously, PM_DEBUG=true sudo pm-suspend has the same behavior as systemctl suspend.
Here is the dmesg output for systemctl suspend
Here is the dmesg output for sudo pm-suspend

My issue was related to deep sleep in /sys/power/mem_sleep. My laptop didn't have deep sleep option by default, but I was able to enable it using UMAF. Disabling it again solved the problem.

I had the issue again today on NixOS 22.11.4479.d4a9ff82fc1 with Linux 6.3.5 and created a bugreport upstream: https://gitlab.freedesktop.org/drm/amd/-/issues/2635

Still happens with Linux 6.7.6. New upstream issue: https://gitlab.freedesktop.org/drm/amd/-/issues/3208
I configured the memreserver and see how well it works.

@lorenz Thank you for your workaround, it's working very well for me so far!

@infinisil I'm glad it works well for you too, I should probably take some time to clean up/finish https://github.com/NixOS/nixpkgs/pull/225819 so it can be used out-of-the-box :)


[BUG] Incorrect broadcast results caused by mismatch dtypes of losses when using 1bit adam and 0/1 Adam with PP TP

Describe the bug
Incorrect broadcast results caused by different dtypes of losses across stages of pipeline parallel, when using 1bit adam and 0/1 Adam with PP TP.
To watch
To watch the dtype of losses when broadcasting across stages, I log before and after the dist.broadcast(tensor=losses, src=self.global_rank, group=self.mpu.get_pipe_parallel_group()) in _aggregate_total_loss, deepspeed/runtime/pipe/engine.py.
To Reproduce
Execute UT TestOneBitAdamFP16Pipeline / TestZeroOneAdamFP16Pipeline / TestOneBitLambFP16Pipeline in tests/unit/runtime/half_precision/onebit/test_onebit.py.
Here is the code with comments that show where I log:
    def _aggregate_total_loss(self):
        # Scale loss, average among DP ranks, and bcast loss to the rest of my DP group
        if self.is_last_stage():
            loss = self._scale_loss_by_gas(self.total_loss)
            self.dp_group_loss = loss.clone().detach()

            agg_loss = self.dp_group_loss.clone().detach()
            if self.is_data_parallel:
                dist.all_reduce(agg_loss, group=self.mpu.get_data_parallel_group())
                agg_loss /= self.dp_world_size

            assert self.global_rank in self.grid.pp_group

            losses = torch.stack([self.dp_group_loss, agg_loss])
            if self.is_pipe_parallel:
                # log before broadcast and print losses
                dist.broadcast(tensor=losses, src=self.global_rank, group=self.mpu.get_pipe_parallel_group())
                # log after broadcast and print losses
        else:
            # Get loss from last stage
            src_rank = self.grid.stage_to_global(self.num_stages - 1)
            assert src_rank in self.grid.pp_group
            losses = torch.Tensor([0., 0.]).to(self.device)
            # log before broadcast and print losses
            dist.broadcast(tensor=losses, src=src_rank, group=self.grid.get_pipe_parallel_group())
            # log after broadcast and print losses
            self.dp_group_loss = losses[0].clone().detach()
            agg_loss = losses[1].clone().detach()

        return agg_loss

Execute Result
[2023-11-23 09:34:02,196] [INFO] [engine.py:587:_aggregate_total_loss] [Rank 2] before dist.broadcast(is_last_stage) (tensor([2.2441, 2.2461], device='cuda:2', dtype=torch.float16), torch.float16), src_rank=2
[2023-11-23 09:34:02,196] [INFO] [engine.py:587:_aggregate_total_loss] [Rank 3] before dist.broadcast(is_last_stage) (tensor([2.2500, 2.2461], device='cuda:3', dtype=torch.float16), torch.float16), src_rank=3

As you can see, pipeline engine is trying to broadcast losses in rank2 at the last_stage of pipeline parallel, to other stage of rank 0, since rank 2 and rank 0 are in the same process group, and the same problem exists between rank 3 and rank 1. Noted that losses in last stage is fp16.
[2023-11-23 09:34:03,088] [INFO] [engine.py:604:_aggregate_total_loss] [Rank 1] after dist.broadcast(other stage) (tensor([3.9727, 0.0000], device='cuda:1'), torch.float32)
[2023-11-23 09:34:03,121] [INFO] [engine.py:604:_aggregate_total_loss] [Rank 0] after dist.broadcast(other stage) (tensor([3.9727, 0.0000], device='cuda:0'), torch.float32)

But after broadcast, you can see the losses in rank 0 at the other stage is not identical to those in rank 2, and the same problem exists between rank 1 and rank 3. Noted that losses in other stage is fp32.

ps. you can see the full log in Additional context.

Expected Result
The losses in rank 0 at the other stage should be identical to those in rank 2, and the same logic applys between rank 1 and rank 3.
reason
The loss tensor in other stage is initialized by losses = torch.Tensor([0., 0.]).to(self.device), and its dtype = torch.float32 if the device is GPU or NPU.
The loss tensor in last stage is valued by losses = torch.stack([self.dp_group_loss, agg_loss]), and its dtype depend on the loss_fn of current model. For example, the loss_fn of ut uses loss_fn=nn.CrossEntropyLoss() so that the dtype is torch.float16, and by contrast, when training gpt_neox,  the dtype of loss tensor is torch.float32, which is consistent with the dtype of output and mask. (You may change it by turning on fp16_lm_cross_entropy.)
In a nutshell, the dtype of loss tensor in last stage is varied from different loss_fn definitions and some other logics, and if the dtype of loss tensor in last stage is different from that in other stage, the broadcast problem would happen.
Solution
In the last stage, change the dtype of loss tensor into float32, see pr. #4727
The test result shows as follows:
before dist.broadcast
[2023-11-24 14:06:04,709] [INFO] [engine.py:590:_aggregate_total_loss] [Rank 2] before dist.broadcast(is_last_stage) (tensor([2.3203, 2.3203], device='cuda:2'), torch.float32, device(type='cuda', index=2)), src_rank=2 (1, 2)
[2023-11-24 14:06:04,710] [INFO] [engine.py:590:_aggregate_total_loss] [Rank 3] before dist.broadcast(is_last_stage) (tensor([2.3203, 2.3203], device='cuda:3'), torch.float32, device(type='cuda', index=3)), src_rank=3 (1, 2)

after dist.broadcast
[2023-11-24 14:06:05,016] [INFO] [engine.py:608:_aggregate_total_loss] [Rank 1] after dist.broadcast(other stage) (tensor([2.3203, 2.3203], device='cuda:1'), torch.float32)
[2023-11-24 14:06:05,043] [INFO] [engine.py:608:_aggregate_total_loss] [Rank 0] after dist.broadcast(other stage) (tensor([2.3203, 2.3203], device='cuda:0'), torch.float32)

ps. you can see the full log in Additional context.

Solution possible risk
No risk. For performance, since the shape of loss tensor is torch.Size([2]) with an initial value torch.tensor([0., 0.]).
As for accuracy, change loss tensor info fp32 will not harm its accuracy.
System info

Python version 3.8
torch 2.1.0
x8 A100s

Additional context
1.both losses are fp32
[2023-11-24 14:06:04,692] [INFO] [engine.py:1219:_exec_optimizer_step] [Rank 3] _exec_optimizer_step AFTER mem_status
[2023-11-24 14:06:04,692] [INFO] [engine.py:1219:_exec_optimizer_step] [Rank 1] _exec_optimizer_step AFTER mem_status
[2023-11-24 14:06:04,693] [INFO] [engine.py:599:_aggregate_total_loss] [Rank 1] after self.grid.stage_to_global
[2023-11-24 14:06:04,701] [INFO] [engine.py:605:_aggregate_total_loss] [Rank 0] before dist.broadcast(other stage) (tensor([0., 0.], device='cuda:0'), torch.float32, device(type='cuda', index=0)), src_rank=2 (0, 2)
[2023-11-24 14:06:04,701] [INFO] [torch.py:198:broadcast] [Rank 0] torch Backend before broadcast
[2023-11-24 14:06:04,701] [INFO] [engine.py:605:_aggregate_total_loss] [Rank 1] before dist.broadcast(other stage) (tensor([0., 0.], device='cuda:1'), torch.float32, device(type='cuda', index=1)), src_rank=3 (0, 2)
[2023-11-24 14:06:04,705] [INFO] [engine.py:569:_aggregate_total_loss] [Rank 2] self.total_loss = prescaled_loss = (tensor(2.3203, device='cuda:2', dtype=torch.float16), torch.float16, device(type='cuda', index=2))
[2023-11-24 14:06:04,705] [INFO] [engine.py:569:_aggregate_total_loss] [Rank 3] self.total_loss = prescaled_loss = (tensor(2.3203, device='cuda:3', dtype=torch.float16), torch.float16, device(type='cuda', index=3))
[2023-11-24 14:06:04,706] [INFO] [engine.py:573:_aggregate_total_loss] [Rank 2] self.dp_group_loss = loss.clone().detach() = (tensor(2.3203, device='cuda:2', dtype=torch.float16), torch.float16, device(type='cuda', index=2))
[2023-11-24 14:06:04,706] [INFO] [engine.py:573:_aggregate_total_loss] [Rank 3] self.dp_group_loss = loss.clone().detach() = (tensor(2.3203, device='cuda:3', dtype=torch.float16), torch.float16, device(type='cuda', index=3))
[2023-11-24 14:06:04,707] [INFO] [engine.py:584:_aggregate_total_loss] [Rank 2] after dist.all_reduce, agg_loss = (tensor(2.3203, device='cuda:2', dtype=torch.float16), torch.float16, device(type='cuda', index=2))
[2023-11-24 14:06:04,708] [INFO] [engine.py:584:_aggregate_total_loss] [Rank 3] after dist.all_reduce, agg_loss = (tensor(2.3203, device='cuda:3', dtype=torch.float16), torch.float16, device(type='cuda', index=3))
[2023-11-24 14:06:04,709] [INFO] [engine.py:590:_aggregate_total_loss] [Rank 2] before dist.broadcast(is_last_stage) (tensor([2.3203, 2.3203], device='cuda:2'), torch.float32, device(type='cuda', index=2)), src_rank=2 (1, 2)
[2023-11-24 14:06:04,710] [INFO] [engine.py:590:_aggregate_total_loss] [Rank 3] before dist.broadcast(is_last_stage) (tensor([2.3203, 2.3203], device='cuda:3'), torch.float32, device(type='cuda', index=3)), src_rank=3 (1, 2)
[2023-11-24 14:06:05,015] [INFO] [engine.py:593:_aggregate_total_loss] [Rank 3] after dist.broadcast(is_last_stage) (tensor([2.3203, 2.3203], device='cuda:3'), torch.float32)
[2023-11-24 14:06:05,016] [INFO] [engine.py:608:_aggregate_total_loss] [Rank 1] after dist.broadcast(other stage) (tensor([2.3203, 2.3203], device='cuda:1'), torch.float32)
[2023-11-24 14:06:05,042] [INFO] [engine.py:593:_aggregate_total_loss] [Rank 2] after dist.broadcast(is_last_stage) (tensor([2.3203, 2.3203], device='cuda:2'), torch.float32)
[2023-11-24 14:06:05,043] [INFO] [engine.py:608:_aggregate_total_loss] [Rank 0] after dist.broadcast(other stage) (tensor([2.3203, 2.3203], device='cuda:0'), torch.float32)
[2023-11-24 14:06:05,044] [INFO] [engine.py:612:_aggregate_total_loss] [Rank 0] after losses[1].clone().detach() 2.3203125
[2023-11-24 14:06:05,044] [INFO] [engine.py:614:_aggregate_total_loss] [Rank 0] exit _aggregate_total_loss
[2023-11-24 14:06:05,044] [INFO] [engine.py:395:train_batch] [Rank 0] Monitoring summary_events
[2023-11-24 14:06:05,044] [INFO] [engine.py:410:train_batch] [Rank 0] exit pipe/engine train_batch
[2023-11-24 14:06:05,044] [INFO] [alexnet_model.py:145:train_cifar] [Rank 0] after engine train_batch() 2.3203125
STEP=0 LOSS=2.3203125
[2023-11-24 14:06:05,044] [INFO] [alexnet_model.py:143:train_cifar] [Rank 0] before engine train_batch()
[2023-11-24 14:06:05,045] [INFO] [engine.py:328:train_batch] [Rank 0] enter pipe/engine train_batch
[2023-11-24 14:06:05,045] [INFO] [engine.py:1404:_exec_schedule] [Rank 0] _exec_schedule enter
[2023-11-24 14:06:05,045] [INFO] [engine.py:833:_exec_load_micro_batch] [Rank 0] _exec_load_micro_batch enter
[2023-11-24 14:06:05,047] [INFO] [engine.py:879:_exec_load_micro_batch] [Rank 0] _exec_load_micro_batch exit
[2023-11-24 14:06:05,048] [INFO] [engine.py:1000:_exec_send_activations] [Rank 0] _exec_send_activations enter
[2023-11-24 14:06:05,049] [INFO] [engine.py:1036:_exec_send_activations] [Rank 0] _exec_send_activations exit
[2023-11-24 14:06:05,049] [INFO] [engine.py:1144:_exec_recv_grads] [Rank 0] _exec_recv_grads enter
[2023-11-24 14:06:05,049] [INFO] [engine.py:1201:_exec_recv_grads] [Rank 0] _exec_recv_grads exit
[2023-11-24 14:06:05,049] [INFO] [engine.py:756:_exec_backward_pass] [Rank 0] _exec_backward_pass enter
[2023-11-24 14:06:05,052] [INFO] [engine.py:830:_exec_backward_pass] [Rank 0] _exec_backward_pass exit
[2023-11-24 14:06:05,052] [INFO] [engine.py:264:_exec_reduce_tied_grads] [Rank 0] _exec_reduce_tied_grads enter
[2023-11-24 14:06:05,052] [INFO] [engine.py:281:_exec_reduce_tied_grads] [Rank 0] _exec_reduce_tied_grads enter
[2023-11-24 14:06:05,052] [INFO] [engine.py:284:_exec_reduce_grads] [Rank 0] _exec_reduce_grads enter
[2023-11-24 14:06:05,052] [INFO] [engine.py:293:_exec_reduce_grads] [Rank 0] _exec_reduce_grads exit
[2023-11-24 14:06:05,053] [INFO] [engine.py:1207:_exec_optimizer_step] [Rank 0] _exec_optimizer_step
[2023-11-24 14:06:05,053] [INFO] [engine.py:1211:_exec_optimizer_step] [Rank 0] _exec_optimizer_step BEFORE STEP and mem status
[2023-11-24 14:06:05,053] [INFO] [engine.py:2046:_take_model_step] [Rank 0] _take_model_step
[2023-11-24 14:06:05,053] [INFO] [engine.py:2057:_take_model_step] [Rank 0] _take_model_step OnebitAdam (
Parameter Group 0
    betas: [0.9, 0.999]
    bias_correction: True
    eps: 1e-08
    lr: 1e-05
    max_grad_norm: 0.0
    weight_decay: 3e-07
)

2.Original problem log, which last stage is fp16 and other stage is fp32
[2023-11-23 09:34:02,195] [INFO] [engine.py:1215:_exec_optimizer_step] [Rank 3] _exec_optimizer_step AFTER mem_status
[2023-11-23 09:34:02,195] [INFO] [engine.py:1215:_exec_optimizer_step] [Rank 1] _exec_optimizer_step AFTER mem_status
[2023-11-23 09:34:02,195] [INFO] [engine.py:595:_aggregate_total_loss] [Rank 1] after self.grid.stage_to_global 3
[2023-11-23 09:34:02,195] [INFO] [engine.py:579:_aggregate_total_loss] [Rank 2] after dist.all_reduce 2.24609375
[2023-11-23 09:34:02,195] [INFO] [engine.py:579:_aggregate_total_loss] [Rank 3] after dist.all_reduce 2.24609375
[2023-11-23 09:34:02,195] [INFO] [engine.py:602:_aggregate_total_loss] [Rank 0] before dist.broadcast(other stage) (tensor([0., 0.], device='cuda:0'), torch.float32), src_rank=2
[2023-11-23 09:34:02,195] [INFO] [torch.py:198:broadcast] [Rank 0] torch Backend before broadcast
[2023-11-23 09:34:02,196] [INFO] [engine.py:602:_aggregate_total_loss] [Rank 1] before dist.broadcast(other stage) (tensor([0., 0.], device='cuda:1'), torch.float32), src_rank=3
[2023-11-23 09:34:02,196] [INFO] [engine.py:587:_aggregate_total_loss] [Rank 2] before dist.broadcast(is_last_stage) (tensor([2.2441, 2.2461], device='cuda:2', dtype=torch.float16), torch.float16), src_rank=2
[2023-11-23 09:34:02,196] [INFO] [engine.py:587:_aggregate_total_loss] [Rank 3] before dist.broadcast(is_last_stage) (tensor([2.2500, 2.2461], device='cuda:3', dtype=torch.float16), torch.float16), src_rank=3
[2023-11-23 09:34:03,087] [INFO] [engine.py:589:_aggregate_total_loss] [Rank 3] after dist.broadcast(is_last_stage) (tensor([2.2500, 2.2461], device='cuda:3', dtype=torch.float16), torch.float16)
[2023-11-23 09:34:03,088] [INFO] [engine.py:604:_aggregate_total_loss] [Rank 1] after dist.broadcast(other stage) (tensor([3.9727, 0.0000], device='cuda:1'), torch.float32)
[2023-11-23 09:34:03,121] [INFO] [engine.py:604:_aggregate_total_loss] [Rank 0] after dist.broadcast(other stage) (tensor([3.9727, 0.0000], device='cuda:0'), torch.float32)
[2023-11-23 09:34:03,122] [INFO] [engine.py:608:_aggregate_total_loss] [Rank 0] after losses[1].clone().detach() 0.0
[2023-11-23 09:34:03,122] [INFO] [engine.py:610:_aggregate_total_loss] [Rank 0] exit _aggregate_total_loss
[2023-11-23 09:34:03,122] [INFO] [engine.py:589:_aggregate_total_loss] [Rank 2] after dist.broadcast(is_last_stage) (tensor([2.2441, 2.2461], device='cuda:2', dtype=torch.float16), torch.float16)
[2023-11-23 09:34:03,122] [INFO] [engine.py:395:train_batch] [Rank 0] Monitoring summary_events
[2023-11-23 09:34:03,122] [INFO] [engine.py:410:train_batch] [Rank 0] exit pipe/engine train_batch
[2023-11-23 09:34:03,122] [INFO] [alexnet_model.py:145:train_cifar] [Rank 0] after engine train_batch() 0.0
STEP=0 LOSS=0.0
[2023-11-23 09:34:03,123] [INFO] [alexnet_model.py:143:train_cifar] [Rank 0] before engine train_batch()
[2023-11-23 09:34:03,123] [INFO] [engine.py:328:train_batch] [Rank 0] enter pipe/engine train_batch
[2023-11-23 09:34:03,123] [INFO] [engine.py:1400:_exec_schedule] [Rank 0] _exec_schedule enter
[2023-11-23 09:34:03,123] [INFO] [engine.py:829:_exec_load_micro_batch] [Rank 0] _exec_load_micro_batch enter
[2023-11-23 09:34:03,124] [INFO] [engine.py:875:_exec_load_micro_batch] [Rank 0] _exec_load_micro_batch exit
[2023-11-23 09:34:03,126] [INFO] [engine.py:996:_exec_send_activations] [Rank 0] _exec_send_activations enter
[2023-11-23 09:34:03,126] [INFO] [engine.py:1032:_exec_send_activations] [Rank 0] _exec_send_activations exit
[2023-11-23 09:34:03,126] [INFO] [engine.py:1140:_exec_recv_grads] [Rank 0] _exec_recv_grads enter
[2023-11-23 09:34:03,126] [INFO] [engine.py:1197:_exec_recv_grads] [Rank 0] _exec_recv_grads exit
[2023-11-23 09:34:03,127] [INFO] [engine.py:752:_exec_backward_pass] [Rank 0] _exec_backward_pass enter
[2023-11-23 09:34:03,129] [INFO] [engine.py:826:_exec_backward_pass] [Rank 0] _exec_backward_pass exit
[2023-11-23 09:34:03,129] [INFO] [engine.py:264:_exec_reduce_tied_grads] [Rank 0] _exec_reduce_tied_grads enter
[2023-11-23 09:34:03,129] [INFO] [engine.py:281:_exec_reduce_tied_grads] [Rank 0] _exec_reduce_tied_grads enter
[2023-11-23 09:34:03,129] [INFO] [engine.py:284:_exec_reduce_grads] [Rank 0] _exec_reduce_grads enter
[2023-11-23 09:34:03,130] [INFO] [engine.py:293:_exec_reduce_grads] [Rank 0] _exec_reduce_grads exit
[2023-11-23 09:34:03,130] [INFO] [engine.py:1203:_exec_optimizer_step] [Rank 0] _exec_optimizer_step
[2023-11-23 09:34:03,130] [INFO] [engine.py:1207:_exec_optimizer_step] [Rank 0] _exec_optimizer_step BEFORE STEP and mem status
[2023-11-23 09:34:03,130] [INFO] [engine.py:2042:_take_model_step] [Rank 0] _take_model_step
[2023-11-23 09:34:03,130] [INFO] [engine.py:2052:_take_model_step] [Rank 0] _take_model_step OnebitAdam (
Parameter Group 0
    betas: [0.9, 0.999]
    bias_correction: True
    eps: 1e-08
    lr: 1e-05
    max_grad_norm: 0.0
    weight_decay: 3e-07
)

The loss tensor in last stage is valued by losses = torch.stack([self.dp_group_loss, agg_loss]), and its dtype depend on the loss_fn of current model. For example, the loss_fn of ut uses loss_fn=nn.CrossEntropyLoss() so that the dtype is torch.float16, and by contrast, when training gpt_neox, the dtype of loss tensor is torch.float32, which is consistent with the dtype of output and mask. (You may change it by turning on fp16_lm_cross_entropy.)

@RUAN-ZX, thanks for investigating and creating a PR for this tricky problem. However, I don't think hardcoding to fp32 is the best choice because the performance overhead may not always be appropriate. In the snippet above, you describe the various conditions that determine the appropriate dtype. Is it possible to use this information to programmatically configure the dtype of the loss?

@RUAN-ZX, thanks for investigating and creating a PR for this tricky problem. However, I don't think hardcoding to fp32 is the best choice because the performance overhead may not always be appropriate. In the snippet above, you describe the various conditions that determine the appropriate dtype. Is it possible to use this information to programmatically configure the dtype of the loss?

@tjruwase, thanks for your suggestion. Well, there are many things to affect the dtype of losses, such as configurations, code logic, loss_fn. So that's no general logic to determine the appropriate dtype, since such logic varies from model to model.
And we have considered, if we can leverage the dtype of the first process/rank in the last stage and set it as a dtype standard, so that the rest processes/ranks could follow it. However, that becomes a problem of communication across processes, how to broadcast the standard with lowest performance overhead? We don't know which process will be the first process to set the standard, and hence more than one broadcast has to be done (or one synchronization has to be done), and that cause performance problems.
More importantly, since the loss tensor is a shape of torch.Size([2]), the cost of executing .float() can be ignored. So we don't need to complicate the solution. I know, hardcoding is not that cool and neat, I have thought it over and had several discussions with others, but just can't work out a better solution. If you have better idea, please share it :) I'd love to try it.

@RUAN-ZX, thanks for the explanation. To be clear, my performance concern is not the cost of the typecast, but rather fp32 communication vs bf16/fp16 communication. Or am I misunderstanding that this change will double the communication volume?

@RUAN-ZX, thanks for the explanation. To be clear, my performance concern is not the cost of the typecast, but rather fp32 communication vs bf16/fp16 communication. Or am I misunderstanding that this change will double the communication volume?

@tjruwase, yes you are right about the communication volume. What I think is the size of the loss tensor is fixed and it is so small that we can also ignore its communication volume. Even if we double it, it can't be a performance problem at all, since it is constant level of complexity. Or, if we try to broadcast the dtype info as I mentioned before, that extra broadcast would absolutely become a problem. I just don't know if there exists a better third way to solve it :)

@tjruwase Wait, maybe I can get the dtype through self.communication_data_type, which is commonly used in ZeRO logics such as stage_1_and_2.py and stage_3.py. Before communications, they always transform the dtypes of their tensors into self.communication_data_type.
I have tested the UT and trained neox and they have both worked well!

@RUAN-ZX, thanks for the analysis of the options. Now, I agree with you that we should keep your current solution since the loss tensor is fixed size. For such a size, the latency would be dominant factor not the bandwidth or volume. Does that make sense?

@RUAN-ZX, thanks for the analysis of the options. Now, I agree with you that we should keep your current solution since the loss tensor is fixed size. For such a size, the latency would be dominant factor of communication, not the bandwidth or volume. Does that make sense?

@tjruwase, yes it does make sense, that's why extra communication should be definitely avoid. And current solution that change loss tensor into fp32 doesn't cause latency problem.
Something interesting is the initialization method of loss tensor in other stage is losses = torch.Tensor([0., 0.]).to(self.device), which means it is designed to be communicated in fp32 precision, and this issue raises because the original author doesn't anticipate that the loss tensor in last stage can be fp16. So basically we just need to add the dtype change.

Great. Thanks for the fix!

Interesting bug reported. However, I wonder if similary bug exixts when activations broadcast. I am training my network via pipeline parallel, and find the dtype of the 1st stage output is float32, while the dtype of the 2nd stage input is float16, which lead to incorrect value after broadcast.
I try to solve this issue by hacking into pipe/engine and force the input of the 2nd stage to be float32. Now the training seems work. In fact, I am not an expert of GPU communication, thus not sure about the real reason.