Make PushQueue test less flaky

The previous test depended on ordering of goroutines which is not
reliable. It would consistently fail when running repeatedly. This
change refactors the test to provide more coverage while not relying on
any undefined ordering.
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: howardjohn
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [howardjohn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@howardjohn: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-new-install-k8s-presubmit-tests-master
1d891bc8337dca6ddf0f2ebc74d0bbaaa336d193
link
/test integ-new-install-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/lgtm

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-new-install-k8s-presubmit-tests-master
1d891bc8337dca6ddf0f2ebc74d0bbaaa336d193
link
/test integ-new-install-k8s-presubmit-tests-master

istio-racetest-master
1d891bc8337dca6ddf0f2ebc74d0bbaaa336d193
link
/test istio-racetest-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

istio-racetest-master
1d891bc8337dca6ddf0f2ebc74d0bbaaa336d193
link
/test istio-racetest-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

istio-racetest-master
1d891bc8337dca6ddf0f2ebc74d0bbaaa336d193
link
/test istio-racetest-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


Vector cannot be imported on the pip install version

Hi,
I installed it using pip. The Vector cannot be imported:
In [20]: from deepx.nn import Vector
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-20-26bcfa1b5977> in <module>()
----> 1 from deepx.nn import Vector

ImportError: cannot import name 'Vector'

perhaps due to this line:
https://github.com/sharadmv/deepx/blob/master/deepx/nn/__init__.py#L2
Is there any reason you commented it out?

@sharadmv I updated it but there are still issues with other elements such as Image. For example, try this example,
It still complains about Image, for example:
from deepx.nn.primitives import Image

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-19-f2be1c741bdd> in <module>()
----> 1 from deepx.nn.primitives import Image

~/anaconda2/envs/tensorflow/lib/python3.6/site-packages/deepx/nn/primitives.py in <module>()
----> 1 from ..core import Data
      2 from ..core import Shape
      3
      4 def Scalar(**kwargs):
      5     return Data(Shape(()), **kwargs)

ImportError: cannot import name 'Data'

In [ ]:

Ah right, that example no longer works. Refer to the new README.md for the new way of getting conv nets working


Update x11 keypress upper code Fixed #243

Please provide Issues links to:

Issues: #243

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Register my domain

Requirements

[x] You have completed your website.
[x] The website is reachable.
[x] You have put a full stop (.) at the end of each of the CNAME, MX and NS records.
[x] The CNAME record doesn't contain https:// or /.
[x] There is sufficient information at the owner field.

Description

Link to Website

CLA Assistant Lite bot: Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I hereby sign the CLA

You can retrigger this bot by commenting recheck in this Pull Request

Please sign the CLA

I have read the CLA Document and I hereby sign the CLA


Project Root
core
Package Name
swarmauri_core
Resource Kind
DecAgentAPI
Component Name
IDecAgentAPI
Generate Content?
yes
Feature Description
API for decentralized agents interaction
Motivation
API for decentralized agents interaction
Potential Solution
No response

nice


AMD bootc - amdgpu kernel modules build process

This PR implements a custom build process for out-of-tree amdgpu modules, instead of using dkms.

The same version of the out-of-tree source code is downloaded, and a script is executed to build the modules.
After the kernel modules are built, they are packaged into an RPM and installed in the final amd-bootc image as a package.

@Okoyl what is going on with this PR?


Fix bug and update arguments in GridCells

This PR did:

Fix the bug when initializing GridCells with "random_gridscales": False.
Make non-random phase_offsets for GridCells.
The offsets separate into two parts. One is organized, and the other is random.
(e.g. if n=101, then 100 cells have organized phase offsets, and the last 1 cell's phase offset is random.)

Thank you for this very clear PR. I am happy to merge the random bug fix PR. I will also happily merge the new function you have added to set phase offsets non-randomly however please could you add a doctoring to this function so users know exactly what is happening when it is called. Once this small change is made I will merge!
Also (side note) I still can't see the figure you tried to send me. Could you send it directly to my e-mail address at<EMAIL_ADDRESS>

Done! The docstring of GridCell is updated.

Ah, sorry, I meant could you add a docstring to set_phase_offsets just so people know what this function does and how

Oh, I see!
I add the docstring and hope it is clear enough.

Don't know if you still use this but the API for GridCells changes slightly in 1.7.0 (for the better!).  See 1.7.0 release notes. "random_gridscale" is no longer accepted as an arg but it should be only a very minor change. Sorry for the inconvenience.

I just noticed this email...
Thanks for reminding me of this change!
Very happy to see RatInABox becoming better and better.
I also attended your talk several weeks ago, it was excellent. And
congratulation again for your great eLife paper!
Best,
Hsuan-Pei
Tom George @.***> 於 2023年6月7日 週三 上午3:32寫道：

Don't know if you still use this but the API for GridCells changes
slightly in 1.7.0 (for the better!). See 1.7.0 release notes.
"random_gridscale" is no longer accepted as an arg but it should be only a
very minor change. Sorry for the inconvenience.
—
Reply to this email directly, view it on GitHub
https://github.com/TomGeorge1234/RatInABox/pull/9#issuecomment-1579335297,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGJCWH44KMX7NYTXWD567JTXJ6AVTANCNFSM6AAAAAASQODAZ4
.
You are receiving this because you authored the thread.Message ID:
@.***>


Redundant div in table cell

Bug Report

Package versions:
devexteme version: 20.2.7
devextreme-react version: 20.2.7

Steps to reproduce:

Create table with cellRender.
Bug is reproducing on demo page: https://js.devexpress.com/Demos/WidgetsGallery/Demo/DataGrid/CellCustomization/React/Light/
Current behavior:

There is redundant div.
Expected behavior:

There is no redundant div.
Screenshots:

Note: There is no div in angular and in previous (20.2.6) react version.


[BUG]

I added the Azure SDK for iOS at the XCode Version 15.4 (15F31d), through menu File > Add Package Dependencies... and informed the Github URL.
After selecting the basic options and added my project as a target I got the error saying: "the package manifest at '/Package.swift' cannot be accessed (/Package.swift doesn't exist in file system) in https://github.com/Azure/azure-sdk-for-ios
".
I continued anyway, but the SDK is not working. Could you please share some thoughts?
Thanks in advance!
Rodrigo Gularte

Hi @RodrigoGularte
Unfortunately we do not support Swift Package Manager for our SDK because of limitations that we have.
See the documentation section for detailed information QuickStart: Add 1:1 video calling to your app
.


[Help] Problem installing onnxruntime-training for iOS example

I am trying to run on_device_training/mobile/ios on an M1 Mac. I could not install onnxruntime-training.
Inspecting the verbose output of Pip, it seems there exists no .whl for ARM. I've tried Python 3.11 and 3.10.
Also, onnxruntime-training==1.16.0 specified in requirements.txt is not released, yet. Is this an internal version?
Alternatively, @vraspar, could you maybe compress the artifacts and upload them here so I can try out the iOS example?
Thanks!

Related #158.

We will be releasing the python package for mac in ort 1.16 release (which should happen very soon).
In the meantime, you could consider getting the nightly python package using
pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-training-cpu

Thank you for the link and instruction. However, it seems that the website requires authentication:
$ pip3 install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-training-cpu
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/
Collecting onnxruntime-training-cpu
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/onnxruntime-training-cpu/1.16.dev20230828001/onnxruntime_training_cpu-1.16.0.dev20230828001-cp311-cp311-macosx_11_0_arm64.whl (8.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 1.1 MB/s eta 0:00:00
User for aiinfra.pkgs.visualstudio.com:

After I just pressed ⏎, Pip went ahead and attempted to install some older versions.
WARNING: 401 Error, Credentials not correct for https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/cerberus/
INFO: pip is looking at multiple versions of onnxruntime-training-cpu to determine which version is compatible with other requirements. This could take a while.
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/onnxruntime-training-cpu/1.16.dev20230825001/onnxruntime_training_cpu-1.16.0.dev20230825001-cp311-cp311-macosx_11_0_arm64.whl (8.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 1.8 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/onnxruntime-training-cpu/1.16.dev20230824001/onnxruntime_training_cpu-1.16.0.dev20230824001-cp311-cp311-macosx_11_0_arm64.whl (8.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 1.3 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/onnxruntime-training-cpu/1.16.dev20230822001/onnxruntime_training_cpu-1.16.0.dev20230822001-cp311-cp311-macosx_11_0_arm64.whl (7.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.9/7.9 MB 1.8 MB/s eta 0:00:00
ERROR: Cannot install onnxruntime-training-cpu==1.16.0.dev20230822001, onnxruntime-training-cpu==1.16.0.dev20230824001, onnxruntime-training-cpu==1.16.0.dev20230825001 and onnxruntime-training-cpu==1.16.0.dev20230828001 because these package versions have conflicting dependencies.

The conflict is caused by:
    onnxruntime-training-cpu 1.16.0.dev20230828001 depends on cerberus
    onnxruntime-training-cpu 1.16.0.dev20230825001 depends on cerberus
    onnxruntime-training-cpu 1.16.0.dev20230824001 depends on cerberus
    onnxruntime-training-cpu 1.16.0.dev20230822001 depends on cerberus

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Thank you for your help.

I solved the above issue by manually installing from the .whl file.

After installing onnxruntime-training-cpu, I cannot import onnxruntime any more.
$ python3 -m onnxruntime
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/__init__.py", line 53, in <module>
    from onnxruntime.capi import onnxruntime_validation
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 145, in <module>
    has_ortmodule, package_name, version, cuda_version = validate_build_package_info()
                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 140, in validate_build_package_info
    raise import_ortmodule_exception
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 70, in validate_build_package_info
    from onnxruntime.training.ortmodule import ORTModule  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/__init__.py", line 21, in <module>
    raise RuntimeError("ORTModule is not supported on this platform.")
RuntimeError: ORTModule is not supported on this platform.

After I reinstalled onnxruntime, I still have errors.
$ pip3 install --force-reinstall onnxruntime
$ python3 artifacts_gen.py
/opt/homebrew/lib/python3.11/site-packages/transformers/configuration_utils.py:380: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
  warnings.warn(
/opt/homebrew/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:595: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/opt/homebrew/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:634: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
  File "/Users/sichanghe/AFileFolder/microsoft--onnxruntime-training-examples/on_device_training/mobile/ios/artifacts_gen.py", line 18, in <module>
    import onnxruntime.training.onnxblock as onnxblock
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/__init__.py", line 6, in <module>
    from onnxruntime.capi._pybind_state import (
ImportError: cannot import name 'PropagateCastOpsStrategy' from 'onnxruntime.capi._pybind_state' (/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/_pybind_state.py)

And, if I force reinstall onnxruntime-training-cpu, I am back to the first problem.
$ pip3 install --force-reinstall onnxruntime_training_cpu-1.16.0.dev20230824001-cp311-cp311-macosx_11_0_arm64.whl onnxruntime-training-cpu
$ python3 artifacts_gen.py
/opt/homebrew/lib/python3.11/site-packages/transformers/configuration_utils.py:380: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
  warnings.warn(
/opt/homebrew/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:595: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/opt/homebrew/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:634: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
  File "/Users/sichanghe/AFileFolder/microsoft--onnxruntime-training-examples/on_device_training/mobile/ios/artifacts_gen.py", line 18, in <module>
    import onnxruntime.training.onnxblock as onnxblock
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/__init__.py", line 53, in <module>
    from onnxruntime.capi import onnxruntime_validation
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 145, in <module>
    has_ortmodule, package_name, version, cuda_version = validate_build_package_info()
                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 140, in validate_build_package_info
    raise import_ortmodule_exception
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 70, in validate_build_package_info
    from onnxruntime.training.ortmodule import ORTModule  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/__init__.py", line 21, in <module>
    raise RuntimeError("ORTModule is not supported on this platform.")
RuntimeError: ORTModule is not supported on this platform.

After installing onnxruntime-training-cpu, I cannot import onnxruntime any more.

This seems to be a problem. I will investigate.

After I reinstalled onnxruntime, I still have errors.

onnxruntime-training-cpu and onnxruntime cannot both be installed together. I would recommend uninstall onnxruntime-training-cpu before installing onnxruntime.

And, if I force reinstall onnxruntime-training-cpu, I am back to the first problem.

Will investigate. In the mean time, if you're looking for the training utilities in onnxruntime, you could try to (not sure if it will work).
import onnxruntime.training.api as orttraining
from onnxruntime.training import artifacts

For this issue:
$ pip3 install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-training-cpu
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/
Collecting onnxruntime-training-cpu
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/onnxruntime-training-cpu/1.16.dev20230828001/onnxruntime_training_cpu-1.16.0.dev20230828001-cp311-cp311-macosx_11_0_arm64.whl (8.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 1.1 MB/s eta 0:00:00
User for aiinfra.pkgs.visualstudio.com:

Please first install all dependencies from pip. And then install onnxruntime-training-cpu from the link I shared.
pip install cerberus flatbuffers h5py numpy onnx packaging protobuf sympy setuptools
pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-training-cpu

@baijumeswani, thanks for the updated instructions.
Unfortunately, I still cannot successfully run the generation script.
I first uninstalled onnxruntime.
$ pip3 uninstall onnxruntime
Found existing installation: onnxruntime 1.15.1
Uninstalling onnxruntime-1.15.1:
  Would remove:
    /opt/homebrew/bin/onnxruntime_test
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime-1.15.1.dist-info/*
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/*
  Would not remove (might be manually added):
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/build_and_package_info.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/checkpointing_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/ort_trainer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/pt_patch.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/training/training_session.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/quantization/matmul_weight4_quantizer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/Checkpoint.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/FloatProperty.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/IntProperty.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/ModuleState.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/OptimizerGroup.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/ParameterOptimizerState.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/PropertyBag.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/StringProperty.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/_checkpoint_storage.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/amp/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/amp/loss_scaler.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/api/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/api/checkpoint_state.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/api/lr_scheduler.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/api/module.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/api/optimizer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/artifacts.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/checkpoint.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/experimental/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/experimental/exporter.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/experimental/gradient_graph/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/experimental/gradient_graph/_gradient_graph_tools.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/model_desc_validation.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/_graph_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/_training_graph_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/blocks.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/checkpoint_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/loss/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/loss/loss.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/model_accessor.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/onnxblock.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/optim/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/onnxblock/optim/optim.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/_apex_amp_modifier.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/_ds_modifier.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/_megatron_modifier.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/_modifier.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/_modifier_registry.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/_multi_tensor_apply.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/config.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/fp16_optimizer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/fused_adam.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/optim/lr_scheduler.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_cache.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_codegen.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_common.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_decompose.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_ir.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_lowering.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_op_config.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_sorted_graph.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_sympy_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/kernel/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/kernel/_mm.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/kernel/_slice_scel.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ort_triton/triton_op_executor.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_custom_autograd_function.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_custom_autograd_function_exporter.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_custom_autograd_function_runner.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_custom_gradient_registry.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_custom_op_symbolic_registry.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_execution_agent.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_fallback.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_fallback_exceptions.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_gradient_accumulation_manager.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_graph_execution_interface.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_graph_execution_manager.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_graph_execution_manager_factory.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_inference_manager.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_io.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_logger.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_onnx_models.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_runtime_inspector.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_torch_module_factory.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_torch_module_interface.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_torch_module_ort.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_torch_module_pytorch.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_training_manager.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/experimental/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/experimental/hierarchical_ortmodule/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/experimental/hierarchical_ortmodule/_hierarchical_ortmodule.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/experimental/json_config/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/experimental/json_config/_load_config_from_json.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/graph_transformer_registry.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/options.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/ortmodule.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cpu/aten_op_executor/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cpu/aten_op_executor/aten_op_executor.cc
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cpu/aten_op_executor/setup.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cpu/torch_interop_utils/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cpu/torch_interop_utils/setup.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cpu/torch_interop_utils/torch_interop_utils.cc
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/fused_ops/fused_ops_frontend.cpp
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/fused_ops/multi_tensor_adam.cu
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/fused_ops/multi_tensor_apply.cuh
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/fused_ops/multi_tensor_axpby_kernel.cu
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/fused_ops/multi_tensor_l2norm_kernel.cu
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/fused_ops/multi_tensor_scale_kernel.cu
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/fused_ops/setup.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/fused_ops/type_shim.h
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/torch_gpu_allocator/setup.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/cuda/torch_gpu_allocator/torch_gpu_allocator.cc
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/torch_cpp_extensions/install.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/orttrainer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/orttrainer_options.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/postprocess.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/torchdynamo/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/torchdynamo/ort_backend.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/torchdynamo/register_backend.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/data/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/data/sampler.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/hooks/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/hooks/_statistics_subscriber.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/hooks/_subscriber_base.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/hooks/_subscriber_manager.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/hooks/merge_activation_summary.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/utils/torch_io_helper.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/bart/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/llama/__init__.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/llama/benchmark.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/llama/benchmark_all.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/llama/convert_to_onnx.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/llama/llama_inputs.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/llama/llama_parity.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/llama/quant_kv_dataloader.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/models.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/onnxruntime_cuda_txt2img.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/onnxruntime_tensorrt_txt2img.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/ort_optimizer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/ort_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/whisper/benchmark.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/whisper/benchmark_all.py
Proceed (Y/n)? Y
  Successfully uninstalled onnxruntime-1.15.1

Then, I first installed the dependencies and onnexruntime-training-cpu.
$ pip3 install cerberus flatbuffers h5py numpy onnx packaging protobuf sympy setuptools
Requirement already satisfied: cerberus in /opt/homebrew/lib/python3.11/site-packages (1.3.5)
Requirement already satisfied: flatbuffers in /opt/homebrew/lib/python3.11/site-packages (23.5.26)
Requirement already satisfied: h5py in /opt/homebrew/lib/python3.11/site-packages (3.9.0)
Requirement already satisfied: numpy in /opt/homebrew/lib/python3.11/site-packages (1.25.2)
Requirement already satisfied: onnx in /opt/homebrew/lib/python3.11/site-packages (1.14.1)
Requirement already satisfied: packaging in /opt/homebrew/lib/python3.11/site-packages (23.1)
Requirement already satisfied: protobuf in /opt/homebrew/lib/python3.11/site-packages (3.20.3)
Requirement already satisfied: sympy in /opt/homebrew/lib/python3.11/site-packages (1.12)
Requirement already satisfied: setuptools in /opt/homebrew/lib/python3.11/site-packages (68.1.2)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /opt/homebrew/lib/python3.11/site-packages (from onnx) (4.7.1)
Requirement already satisfied: mpmath>=0.19 in /opt/homebrew/lib/python3.11/site-packages (from sympy) (1.3.0)
$ pip3 install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-training-cpu
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/
Requirement already satisfied: onnxruntime-training-cpu in /opt/homebrew/lib/python3.11/site-packages (1.16.0.dev20230824001)
Requirement already satisfied: cerberus in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.3.5)
Requirement already satisfied: flatbuffers in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (23.5.26)
Requirement already satisfied: h5py in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (3.9.0)
Requirement already satisfied: numpy>=1.16.6 in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.25.2)
Requirement already satisfied: onnx in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.14.1)
Requirement already satisfied: packaging in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (23.1)
Requirement already satisfied: protobuf in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (3.20.3)
Requirement already satisfied: sympy in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.12)
Requirement already satisfied: setuptools>=41.4.0 in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (68.1.2)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /opt/homebrew/lib/python3.11/site-packages (from onnx->onnxruntime-training-cpu) (4.7.1)
Requirement already satisfied: mpmath>=0.19 in /opt/homebrew/lib/python3.11/site-packages (from sympy->onnxruntime-training-cpu) (1.3.0)

Pip skipped the installation, so I uninstalled onnexruntime-training-cpu and reinstalled it.
$ pip3 uninstall onnxruntime-training-cpu
Found existing installation: onnxruntime-training-cpu 1.16.0.dev20230824001
Uninstalling onnxruntime-training-cpu-1.16.0.dev20230824001:
  Would remove:
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/build_and_package_info.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/checkpointing_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/ort_trainer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/pt_patch.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/training/training_session.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/quantization/matmul_weight4_quantizer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/Checkpoint.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/FloatProperty.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/IntProperty.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/ModuleState.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/OptimizerGroup.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/ParameterOptimizerState.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/PropertyBag.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/tools/ort_format_model/ort_flatbuffers_py/fbs/StringProperty.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/*
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/bart/*
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/llama/*
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/models.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/onnxruntime_cuda_txt2img.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/onnxruntime_tensorrt_txt2img.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/ort_optimizer.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/stable_diffusion/ort_utils.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/whisper/benchmark.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime/transformers/models/whisper/benchmark_all.py
    /opt/homebrew/lib/python3.11/site-packages/onnxruntime_training_cpu-1.16.0.dev20230824001.dist-info/*
Proceed (Y/n)? Y
  Successfully uninstalled onnxruntime-training-cpu-1.16.0.dev20230824001
$ pip3 install cerberus flatbuffers h5py numpy onnx packaging protobuf sympy setuptools
Requirement already satisfied: cerberus in /opt/homebrew/lib/python3.11/site-packages (1.3.5)
Requirement already satisfied: flatbuffers in /opt/homebrew/lib/python3.11/site-packages (23.5.26)
Requirement already satisfied: h5py in /opt/homebrew/lib/python3.11/site-packages (3.9.0)
Requirement already satisfied: numpy in /opt/homebrew/lib/python3.11/site-packages (1.25.2)
Requirement already satisfied: onnx in /opt/homebrew/lib/python3.11/site-packages (1.14.1)
Requirement already satisfied: packaging in /opt/homebrew/lib/python3.11/site-packages (23.1)
Requirement already satisfied: protobuf in /opt/homebrew/lib/python3.11/site-packages (3.20.3)
Requirement already satisfied: sympy in /opt/homebrew/lib/python3.11/site-packages (1.12)
Requirement already satisfied: setuptools in /opt/homebrew/lib/python3.11/site-packages (68.1.2)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /opt/homebrew/lib/python3.11/site-packages (from onnx) (4.7.1)
Requirement already satisfied: mpmath>=0.19 in /opt/homebrew/lib/python3.11/site-packages (from sympy) (1.3.0)
$ pip3 install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-training-cpu
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/
Collecting onnxruntime-training-cpu
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/onnxruntime-training-cpu/1.16.dev20230830001/onnxruntime_training_cpu-1.16.0.dev20230830001-cp311-cp311-macosx_11_0_arm64.whl (8.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 73.3 kB/s eta 0:00:00
Requirement already satisfied: cerberus in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.3.5)
Requirement already satisfied: flatbuffers in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (23.5.26)
Requirement already satisfied: h5py in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (3.9.0)
Requirement already satisfied: numpy>=1.16.6 in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.25.2)
Requirement already satisfied: onnx in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.14.1)
Requirement already satisfied: packaging in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (23.1)
Requirement already satisfied: protobuf in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (3.20.3)
Requirement already satisfied: sympy in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.12)
Requirement already satisfied: setuptools>=41.4.0 in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (68.1.2)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /opt/homebrew/lib/python3.11/site-packages (from onnx->onnxruntime-training-cpu) (4.7.1)
Requirement already satisfied: mpmath>=0.19 in /opt/homebrew/lib/python3.11/site-packages (from sympy->onnxruntime-training-cpu) (1.3.0)
Installing collected packages: onnxruntime-training-cpu
Successfully installed onnxruntime-training-cpu-1.16.0.dev20230830001

But, when I ran the generation script again, it still failed with the exact same problem as before.
$ python3 artifacts_gen.py
/opt/homebrew/lib/python3.11/site-packages/transformers/configuration_utils.py:380: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
  warnings.warn(
/opt/homebrew/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:595: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/opt/homebrew/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:634: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
  File "/Users/sichanghe/AFileFolder/microsoft--onnxruntime-training-examples/on_device_training/mobile/ios/artifacts_gen.py", line 18, in <module>
    import onnxruntime.training.onnxblock as onnxblock
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/__init__.py", line 53, in <module>
    from onnxruntime.capi import onnxruntime_validation
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 145, in <module>
    has_ortmodule, package_name, version, cuda_version = validate_build_package_info()
                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 140, in validate_build_package_info
    raise import_ortmodule_exception
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_validation.py", line 70, in validate_build_package_info
    from onnxruntime.training.ortmodule import ORTModule  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/onnxruntime/training/ortmodule/__init__.py", line 21, in <module>
    raise RuntimeError("ORTModule is not supported on this platform.")
RuntimeError: ORTModule is not supported on this platform.

For your information, @baijumeswani.

I updated the code in https://github.com/microsoft/onnxruntime/pull/17380.
Could you uninstall onnxruntime and onnxruntime-training-cpu and try to install the latest onnxruntime-training-cpu nightly using
pip install cerberus flatbuffers h5py numpy onnx packaging protobuf sympy setuptools
pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-training-cpu

It works. I successfully ran artifact_gen.py.
Full command output.
$ pip3 install cerberus flatbuffers h5py numpy onnx packaging protobuf sympy setuptools
Requirement already satisfied: cerberus in /opt/homebrew/lib/python3.11/site-packages (1.3.5)
Requirement already satisfied: flatbuffers in /opt/homebrew/lib/python3.11/site-packages (23.5.26)
Requirement already satisfied: h5py in /opt/homebrew/lib/python3.11/site-packages (3.9.0)
Requirement already satisfied: numpy in /opt/homebrew/lib/python3.11/site-packages (1.25.2)
Requirement already satisfied: onnx in /opt/homebrew/lib/python3.11/site-packages (1.14.1)
Requirement already satisfied: packaging in /opt/homebrew/lib/python3.11/site-packages (23.1)
Requirement already satisfied: protobuf in /opt/homebrew/lib/python3.11/site-packages (3.20.3)
Requirement already satisfied: sympy in /opt/homebrew/lib/python3.11/site-packages (1.12)
Requirement already satisfied: setuptools in /opt/homebrew/lib/python3.11/site-packages (68.1.2)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /opt/homebrew/lib/python3.11/site-packages (from onnx) (4.7.1)
Requirement already satisfied: mpmath>=0.19 in /opt/homebrew/lib/python3.11/site-packages (from sympy) (1.3.0)
$ pip3 install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-training-cpu
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/
Collecting onnxruntime-training-cpu
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/onnxruntime-training-cpu/1.16.dev20230904001/onnxruntime_training_cpu-1.16.0.dev20230904001-cp311-cp311-macosx_11_0_arm64.whl (8.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 1.8 MB/s eta 0:00:00
Requirement already satisfied: cerberus in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.3.5)
Requirement already satisfied: flatbuffers in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (23.5.26)
Requirement already satisfied: h5py in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (3.9.0)
Requirement already satisfied: numpy>=1.16.6 in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.25.2)
Requirement already satisfied: onnx in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.14.1)
Requirement already satisfied: packaging in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (23.1)
Requirement already satisfied: protobuf in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (3.20.3)
Requirement already satisfied: sympy in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (1.12)
Requirement already satisfied: setuptools>=41.4.0 in /opt/homebrew/lib/python3.11/site-packages (from onnxruntime-training-cpu) (68.1.2)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /opt/homebrew/lib/python3.11/site-packages (from onnx->onnxruntime-training-cpu) (4.7.1)
Requirement already satisfied: mpmath>=0.19 in /opt/homebrew/lib/python3.11/site-packages (from sympy->onnxruntime-training-cpu) (1.3.0)
Installing collected packages: onnxruntime-training-cpu
Successfully installed onnxruntime-training-cpu-1.16.0.dev20230904001
$ python3 artifacts_gen.py
/opt/homebrew/lib/python3.11/site-packages/transformers/configuration_utils.py:380: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
  warnings.warn(
/opt/homebrew/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:595: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/opt/homebrew/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:634: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

2023-09-06 12:50:08,600 root [INFO] - Custom loss block provided: CustomCELoss
2023-09-06 12:50:08,611 root [DEBUG] - Building training block _TrainingBlock
2023-09-06 12:50:08,611 root [DEBUG] - Building block: CustomCELoss
2023-09-06 12:50:08,611 root [DEBUG] - Building block: CrossEntropyLoss
2023-09-06 12:50:09,480 root [DEBUG] - Building gradient graph for training block _TrainingBlock
2023-09-06 12:50:09.613202 [I:onnxruntime:Default, constant_sharing.cc:256 ApplyImpl] Total shared scalar initializer count: 499
2023-09-06 12:50:09.620483 [I:onnxruntime:Default, graph.cc:3556 CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_1_0_1_5'. It is no longer used by any node.
2023-09-06 12:50:09.620505 [I:onnxruntime:Default, graph.cc:3556 CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_1_0_1_0'. It is no longer used by any node.
2023-09-06 12:50:09.626150 [I:onnxruntime:Default, graph.cc:3556 CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_1_0_1_4'. It is no longer used by any node.
2023-09-06 12:50:09.626160 [I:onnxruntime:Default, graph.cc:3556 CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_1_0_1_1'. It is no longer used by any node.
2023-09-06 12:50:09.626164 [I:onnxruntime:Default, graph.cc:3556 CleanUnusedInitializersAndNodeArgs] Removing initializer 'ortshared_1_0_1_2'. It is no longer used by any node.
2023-09-06 12:50:09,664 root [DEBUG] - The loss output is onnx::loss::2. The gradient graph will be built starting from onnx::loss::2_grad.
2023-09-06 12:50:09,720 root [DEBUG] - Adding gradient accumulation nodes for training block _TrainingBlock
2023-09-06 12:50:09,799 root [INFO] - Saved training model to MyVoice/artifacts/training_model.onnx
2023-09-06 12:50:09,838 root [INFO] - Saved eval model to MyVoice/artifacts/eval_model.onnx
2023-09-06 12:50:10,709 root [INFO] - Saved checkpoint to MyVoice/artifacts/checkpoint
2023-09-06 12:50:10,709 root [INFO] - Optimizer enum provided: AdamW
2023-09-06 12:50:10,709 root [DEBUG] - Building forward block AdamW
2023-09-06 12:50:10,710 root [DEBUG] - Building block: AdamWOptimizer
2023-09-06 12:50:10,711 root [INFO] - Saved optimizer model to MyVoice/artifacts/optimizer_model.onnx

Thank you, @baijumeswani! I will keep you updated with how trying out the app goes for me.


Run the test entropy src end reqs in continuous mode

Run the test entropy src end reqs  in continuous mode and configure the EDN to force a CSRNG trigger a couple of times.

Tagging @moidx for visibility

With continuous mode you mean the Auto Request Mode, right? If so, code for that will be added to solve #7424.

do we feel like this test is already accomplished by entropy_src_edn_req_test? I feel like it is.

I opened this issue motivated by this comment from @moidx.
@moidx can we close this issue?

I think this issue can be closed now.
nit: We should change the name of entropy_testutils_boot_mode_init as it sets fips_mode = true, which means that we are running in continuous mode and not boot mode.

i have a PR where i'm cleaning up entropy_testutils.
And yeah, the boot mode terminology is a bit weird...but boot mode has meaning in both edn and entropy_src, so it's a little annoying. Right now entropy_src is in fips mode, but edn is not.

@tjaychen - unless you feel otherwise, from lowRISC side we believe this issue to be closed.
Could you add the related PR for entropy_testutils and close out if nothing else is to be done. Otherwise please comment and let @doreis-lowrisc what is still needed. Thanks!

let's keep it open for a a bit more.
There are still a fair number of tests that directly use boot_mode instead of auto_mode.
We can't quite make the switch yet until #14571 is merged, otherwise those tests will likely just hang.

@vogelpi do you want to close this issue?
I know your latest PR of changing RND to URND also came with switching from boot to auto.  So I think that satisfies the spirit of this issue.

Yes, let's close this issue. It's pretty cool that we have are now using auto/continuous mode more widely. Thanks for the effort you've put into this @tjaychen and @andreaskurth !


Utilize Admin plugin's slug function

https://github.com/getgrav/grav-plugin-admin/issues/1186

thanks for your contribution


Modules: Module to be removed does not exist

When you "Remove all BAC modules" from the user database by click on the button in the ribbon and one or more modules that should be removed, do not exist, you get an error message.
This message should be avoided.

Released with version <IP_ADDRESS>


UVM report mock not getting flushed with new tests

I found this in my old notes, without any context. Have to understand what I meant by it and whether it's still relevant.


[3.12] Segfault on TorchInductor test happening in triton

Repro command: gdb --args python -m pytest test/inductor/test_torchinductor.py::CpuTests::test_multi_gpu_recompile_on_index_cpu test/inductor/test_torchinductor.py::GPUTests::test_mixed_mm_cuda (may have to run on a debug Python 3.12 build)
gdb backtrace:
0x00007ff8b4c5b51f in mlir::detail::OperandStorage::OperandStorage(mlir::Operation*, mlir::OpOperand*, mlir::ValueRange) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
(gdb) bt
#0  0x00007ff8b4c5b51f in mlir::detail::OperandStorage::OperandStorage(mlir::Operation*, mlir::OpOperand*, mlir::ValueRange) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#1  0x00007ff8b4c4e867 in mlir::Operation::create(mlir::Location, mlir::OperationName, mlir::TypeRange, mlir::ValueRange, mlir::DictionaryAttr, mlir::OpaqueProperties, mlir::BlockRange, unsigned int) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#2  0x00007ff8b4c4e087 in mlir::Operation::create(mlir::Location, mlir::OperationName, mlir::TypeRange, mlir::ValueRange, mlir::NamedAttrList&&, mlir::OpaqueProperties, mlir::BlockRange, mlir::RegionRange) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#3  0x00007ff8b4c4df04 in mlir::Operation::create(mlir::OperationState const&) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#4  0x00007ff8b4b9a170 in mlir::OpBuilder::create(mlir::OperationState const&) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#5  0x00007ff8b48910c5 in mlir::OpBuilder::create<mlir::LLVM::InsertElementOp, mlir::Value&, mlir::LLVM::ExtractElementOp, mlir::Value> (location=..., this=0x7ff842dfbea0) at /home/williamwen/.triton/llvm/llvm-ed4e505c-ubuntu-x64/include/mlir/IR/Value.h:615
#6  MMA16816SmemLoader::loadX4 (this=this@entry=0x7ff842dfaea0, batch=batch@entry=0, mat0=0, mat1=0, ptrs=..., matTy=<EMAIL_ADDRESS>shemTy=...) at /data/users/williamwen/triton/third_party/nvidia/lib/TritonNVIDIAGPUToLLVM/ConvertLayoutOpToLLVM/SharedToDotOperandMMAv2.cpp:418
#7  0x00007ff8b489223a in operator() (__closure=0x7ff75c0c3980, batch=0, a=0, b=0) at /home/williamwen/.triton/llvm/llvm-ed4e505c-ubuntu-x64/include/llvm/ADT/SmallVector.h:91
#8  0x00007ff8b488f5b3 in std::function<void (int, int, int)>::operator()(int, int, int) const (__args#2=0, __args#1=0, __args#0=0, this=0x7ff842dfb140) at /usr/include/c++/11/bits/std_function.h:586
#9  loadArg (rewriter=..., loc=<EMAIL_ADDRESS><EMAIL_ADDRESS>encoding=<EMAIL_ADDRESS>smemObj=..., typeConverter=typeConverter@entry=0x7ff842dfc9b0, thread=..., isA=true)
    at /data/users/williamwen/triton/third_party/nvidia/lib/TritonNVIDIAGPUToLLVM/ConvertLayoutOpToLLVM/SharedToDotOperandMMAv2.cpp:647
#10 0x00007ff8b488fb15 in SharedToDotOperandMMAv2::convertLayout (opIdx=0, rewriter=<EMAIL_ADDRESS>tensor=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, encoding=..., smemObj=..., typeConverter=0x7ff842dfc9b0, thread=...)
    at /data/users/williamwen/triton/third_party/nvidia/lib/TritonNVIDIAGPUToLLVM/ConvertLayoutOpToLLVM/SharedToDotOperandMMAv2.cpp:779
#11 0x00007ff8b489e3a7 in (anonymous namespace)::LocalLoadOpConversion::lowerSharedToDotOperandMMA (this=0x7ff842dfb6d0, isOuter=false, dotOperandLayout=..., mmaLayout=..., rewriter=..., typeConverter=0x7ff842dfc9b0, adaptor=..., op=...)
    at /data/users/williamwen/triton/third_party/nvidia/lib/TritonNVIDIAGPUToLLVM/ConvertLayoutOpToLLVM.cpp:89
#12 (anonymous namespace)::LocalLoadOpConversion::lowerSharedToDotOperand (rewriter=..., typeConverter=0x7ff842dfc9b0, adaptor=..., op=..., this=0x7ff842dfb6d0) at /data/users/williamwen/triton/third_party/nvidia/lib/TritonNVIDIAGPUToLLVM/ConvertLayoutOpToLLVM.cpp:131
#13 (anonymous namespace)::LocalLoadOpConversion::matchAndRewrite (this=this@entry=0x7ff75c0c3230, <EMAIL_ADDRESS>adaptor=..., rewriter=...) at /data/users/williamwen/triton/third_party/nvidia/lib/TritonNVIDIAGPUToLLVM/ConvertLayoutOpToLLVM.cpp:64
#14 0x00007ff8b482986e in mlir::ConvertOpToLLVMPattern<mlir::triton::gpu::LocalLoadOp>::matchAndRewrite (this=0x7ff75c0c3230, op=0x7ff75c0a8cb0, operands=..., rewriter=...) at /home/williamwen/.triton/llvm/llvm-ed4e505c-ubuntu-x64/include/mlir/IR/OpDefinition.h:1716
#15 0x00007ff8b64295d0 in mlir::ConversionPattern::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&) const () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#16 0x00007ff8b6471a3b in mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>)::$_2::operator()() const () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#17 0x00007ff8b646e55f in mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>) ()
   from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#18 0x00007ff8b642a4d9 in (anonymous namespace)::OperationLegalizer::legalize(mlir::Operation*, mlir::ConversionPatternRewriter&) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#19 0x00007ff8b6429674 in mlir::OperationConverter::convert(mlir::ConversionPatternRewriter&, mlir::Operation*) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#20 0x00007ff8b642a8d1 in mlir::OperationConverter::convertOperations(llvm::ArrayRef<mlir::Operation*>) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#21 0x00007ff8b643238b in mlir::applyPartialConversion(mlir::Operation*, mlir::ConversionTarget const&, mlir::FrozenRewritePatternSet const&, mlir::ConversionConfig) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#22 0x00007ff8b48ef47a in (anonymous namespace)::ConvertTritonGPUToLLVM::runOnOperation (this=0x7ff75c0038a0) at /home/williamwen/.triton/llvm/llvm-ed4e505c-ubuntu-x64/include/llvm/ADT/ArrayRef.h:70
#23 0x00007ff8b4c99966 in mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#24 0x00007ff8b4c9a2b1 in mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) ()
   from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#25 0x00007ff8b4c9ca7b in mlir::PassManager::run(mlir::Operation*) () from /data/users/williamwen/triton/python/triton/_C/libtriton.so
#26 0x00007ff8b49b2695 in operator() (self=..., mod=..., __closure=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /home/williamwen/.triton/llvm/llvm-ed4e505c-ubuntu-x64/include/mlir/IR/OpDefinition.h:108
#27 0x00007ff8b49be0d8 in pybind11::detail::argument_loader<mlir::PassManager&, mlir::ModuleOp&>::call_impl<void, init_triton_ir(pybind11::module&&)::<lambda(mlir::PassManager&, mlir::ModuleOp&)>&, 0, 1, pybind11::detail::void_type> (f=..., this=0x7ff842dfd840)
    at /home/williamwen/.triton/pybind11/pybind11-2.11.1/include/pybind11/detail/../detail/type_caster_base.h:1136
#28 pybind11::detail::argument_loader<mlir::PassManager&, mlir::ModuleOp&>::call<void, pybind11::detail::void_type, init_triton_ir(pybind11::module&&)::<lambda(mlir::PassManager&, mlir::ModuleOp&)>&> (f=..., this=0x7ff842dfd840)
    at /home/williamwen/.triton/pybind11/pybind11-2.11.1/include/pybind11/detail/../cast.h:1454
#29 operator() (__closure=0x0, __closure@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, call=..., call@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
    at /home/williamwen/.triton/pybind11/pybind11-2.11.1/include/pybind11/pybind11.h:254
#30 _FUN () at /home/williamwen/.triton/pybind11/pybind11-2.11.1/include/pybind11/pybind11.h:224
#31 0x00007ff8b499fb5f in pybind11::cpp_function::dispatcher (self=<optimized out>, args_in=args_in@entry=0x7fff70e386e0, kwargs_in=kwargs_in@entry=0x0) at /home/williamwen/.triton/pybind11/pybind11-2.11.1/include/pybind11/pybind11.h:946
--Type <RET> for more, q to quit, c to continue without paging--
#32 0x00000000005259c1 in cfunction_call (func=func@entry=0x7fffb2f3f710, args=args@entry=0x7fff70e386e0, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:537
#33 0x00000000004c1327 in _PyObject_MakeTpCall (tstate=tstate@entry=0xf2a0e5d0, callable=callable@entry=0x7fffb2f3f710, args=args@entry=0x7fffea8c0a00, nargs=<optimized out>, keywords=keywords@entry=0x0) at Objects/call.c:240
#34 0x00000000004c1708 in _PyObject_VectorcallTstate (tstate=0xf2a0e5d0, callable=0x7fffb2f3f710, args=0x7fffea8c0a00, nargsf=<optimized out>, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#35 0x00000000004c1730 in PyObject_Vectorcall (callable=callable@entry=0x7fffb2f3f710, args=args@entry=0x7fffea8c0a00, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:325
#36 0x000000000060cf62 in _PyEval_EvalFrameDefault (tstate=0xf2a0e5d0, frame=0x7fffea8c0948, throwflag=0) at Python/bytecodes.c:2706
#37 0x000000000061421d in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0xf2a0e5d0) at ./Include/internal/pycore_ceval.h:91
#38 _PyEval_Vector (tstate=0xf2a0e5d0, func=0x7fffb031c710, locals=locals@entry=0x0, args=0x7fffea8c0928, argcount=4, kwnames=0x0) at Python/ceval.c:1683
#39 0x00000000004c1136 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:419
#40 0x00000000004c1633 in _PyObject_VectorcallTstate (tstate=0xf2a0e5d0, callable=0x7fffb031c710, args=0x7fffea8c0928, nargsf=9223372036854775812, kwnames=0x0) at ./Include/internal/pycore_call.h:92
#41 0x00000000004c1730 in PyObject_Vectorcall (callable=callable@entry=0x7fffb031c710, args=args@entry=0x7fffea8c0928, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:325
#42 0x000000000060cf62 in _PyEval_EvalFrameDefault (tstate=0xf2a0e5d0, frame=0x7fffea8c08b0, throwflag=0) at Python/bytecodes.c:2706
#43 0x000000000061421d in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0xf2a0e5d0) at ./Include/internal/pycore_ceval.h:91
...

Segfault output:
Fatal Python error: Segmentation fault

Thread 0x00007ff837e00640 (most recent call first):
  <no Python frame>

Thread 0x00007ff83aa00640 (most recent call first):
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/collections/__init__.py", line 447 in _make
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/tokenize.py", line 538 in _generate_tokens_from_c_tokenizer
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/inspect.py", line 1241 in getblock
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/inspect.py", line 1274 in getsourcelines
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 84 in _get_fn_file_line
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 1288 in ast_to_ttir
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 113 in make_ir
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 276 in compile
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 397 in _precompile_config
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 203 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/autotune_process.py", line 643 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 781 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1031 in precompile_with_captured_stdout
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1035 in <lambda>
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1010 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ff83b400640 (most recent call first):
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/ast.py", line 52 in parse
  File "/data/users/williamwen/triton/python/triton/runtime/jit.py", line 794 in parse
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 1294 in ast_to_ttir
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 113 in make_ir
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 276 in compile
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 397 in _precompile_config
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 203 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/autotune_process.py", line 643 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 781 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1031 in precompile_with_captured_stdout
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1035 in <lambda>
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1010 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ff83be00640 (most recent call first):
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 1198 in visit
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/ast.py", line 417 in generic_visit
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 1163 in visit_Expr
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/ast.py", line 407 in visit
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 1201 in visit
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 351 in visit_compound_statement
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 443 in visit_FunctionDef
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/ast.py", line 407 in visit
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 1201 in visit
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/ast.py", line 415 in generic_visit
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 359 in visit_Module
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/ast.py", line 407 in visit
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 1201 in visit
  File "/data/users/williamwen/triton/python/triton/compiler/code_generator.py", line 1294 in ast_to_ttir
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 113 in make_ir
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 276 in compile
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 397 in _precompile_config
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 203 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/autotune_process.py", line 643 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 781 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1031 in precompile_with_captured_stdout
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1035 in <lambda>
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1010 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ff83ea00640 (most recent call first):
  File "/data/users/williamwen/triton/python/triton/runtime/cache.py", line 123 in put
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 284 in compile
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 397 in _precompile_config
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 203 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/autotune_process.py", line 643 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 781 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1031 in precompile_with_captured_stdout
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1035 in <lambda>
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1010 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ff83f400640 (most recent call first):
  File "/data/users/williamwen/triton/python/triton/runtime/cache.py", line 122 in put
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 284 in compile
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 397 in _precompile_config
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 203 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/autotune_process.py", line 643 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 781 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1031 in precompile_with_captured_stdout
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1035 in <lambda>
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1010 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ff83fe00640 (most recent call first):
  File "/data/users/williamwen/triton/python/triton/runtime/cache.py", line 122 in put
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 284 in compile
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 397 in _precompile_config
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 203 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/autotune_process.py", line 643 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 781 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1031 in precompile_with_captured_stdout
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1035 in <lambda>
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1010 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x00007ff842e00640 (most recent call first):
  File "/data/users/williamwen/triton/python/triton/backends/nvidia/compiler.py", line 212 in make_llir
  File "/data/users/williamwen/triton/python/triton/backends/nvidia/compiler.py", line 302 in <lambda>
  File "/data/users/williamwen/triton/python/triton/compiler/compiler.py", line 282 in compile
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 397 in _precompile_config
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/triton_heuristics.py", line 203 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/autotune_process.py", line 643 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 781 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1031 in precompile_with_captured_stdout
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1035 in <lambda>
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1010 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007ffff7ed6740 (most recent call first):
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 355 in wait
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 655 in wait
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/threading.py", line 997 in start
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 202 in _adjust_thread_count
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/thread.py", line 179 in submit
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/_base.py", line 608 in map
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1034 in precompile
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1107 in __call__
  File "/data/users/williamwen/pytorch/torch/_inductor/select_algorithm.py", line 1405 in autotune_select_algorithm
  File "/data/users/williamwen/pytorch/torch/_inductor/kernel/mm.py", line 398 in tuned_mixed_mm
  File "/data/users/williamwen/pytorch/torch/_inductor/fx_passes/post_grad.py", line 312 in mixed_mm
  File "/data/users/williamwen/pytorch/torch/_inductor/graph.py", line 923 in call_function
  File "/data/users/williamwen/pytorch/torch/fx/interpreter.py", line 202 in run_node
  File "/data/users/williamwen/pytorch/torch/_inductor/graph.py", line 1199 in run_node
  File "/data/users/williamwen/pytorch/torch/fx/interpreter.py", line 145 in run
  File "/data/users/williamwen/pytorch/torch/_inductor/graph.py", line 730 in run
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273 in time_wrapper
  File "/data/users/williamwen/pytorch/torch/_inductor/compile_fx.py", line 784 in fx_codegen_and_compile
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81 in inner
  File "/data/users/williamwen/pytorch/torch/_inductor/codecache.py", line 987 in load
  File "/data/users/williamwen/pytorch/torch/_inductor/compile_fx.py", line 498 in compile_fx_inner
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273 in time_wrapper
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81 in inner
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81 in inner
  File "/data/users/williamwen/pytorch/torch/_inductor/debug.py", line 304 in inner
  File "/data/users/williamwen/pytorch/torch/_dynamo/repro/after_aot.py", line 83 in debug_wrapper
  File "/data/users/williamwen/pytorch/torch/_inductor/compile_fx.py", line 1382 in fw_compiler_base
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273 in time_wrapper
  File "/data/users/williamwen/pytorch/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 169 in aot_dispatch_base
  File "/data/users/williamwen/pytorch/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 672 in aot_wrapper_synthetic_base
  File "/data/users/williamwen/pytorch/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 470 in aot_wrapper_dedupe
  File "/data/users/williamwen/pytorch/torch/_functorch/aot_autograd.py", line 685 in create_aot_dispatcher_function
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273 in time_wrapper
  File "/data/users/williamwen/pytorch/torch/_functorch/aot_autograd.py", line 958 in aot_module_simplified
  File "/data/users/williamwen/pytorch/torch/_dynamo/backends/common.py", line 65 in compiler_fn
  File "/data/users/williamwen/pytorch/torch/_inductor/compile_fx.py", line 1478 in compile_fx
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81 in inner
  File "/data/users/williamwen/pytorch/test/inductor/test_torchinductor.py", line 434 in compile_fx_wrapper
  File "/data/users/williamwen/pytorch/torch/_dynamo/repro/after_dynamo.py", line 127 in debug_wrapper
  File "/data/users/williamwen/pytorch/torch/_dynamo/repro/after_dynamo.py", line 127 in debug_wrapper
  File "/data/users/williamwen/pytorch/torch/_dynamo/output_graph.py", line 1368 in call_user_compiler
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273 in time_wrapper
  File "/data/users/williamwen/pytorch/torch/_dynamo/output_graph.py", line 1296 in compile_and_call_fx_graph
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81 in inner
  File "/data/users/williamwen/pytorch/torch/_dynamo/output_graph.py", line 1079 in compile_subgraph
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 2409 in _return
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 2424 in RETURN_VALUE
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 799 in step
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 884 in run
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 2234 in run
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 517 in transform
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 172 in _fn
  File "/data/users/williamwen/pytorch/torch/_dynamo/bytecode_transformation.py", line 1167 in transform_code_object
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 570 in compile_inner
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273 in time_wrapper
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 703 in _compile
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81 in inner
  File "/data/users/williamwen/pytorch/torch/_utils_internal.py", line 70 in wrapper_function
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 410 in _convert_frame_assert
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 981 in catch_errors
  File "/data/users/williamwen/pytorch/torch/_dynamo/eval_frame.py", line 420 in _fn
  File "/data/users/williamwen/pytorch/test/inductor/test_torchinductor.py", line 442 in check_model
  File "/data/users/williamwen/pytorch/test/inductor/test_torchinductor.py", line 595 in check_model_gpu
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81 in inner
  File "/data/users/williamwen/pytorch/test/inductor/test_torchinductor.py", line 2507 in test_mixed_mm
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81 in inner
  File "/data/users/williamwen/pytorch/test/inductor/test_torchinductor.py", line 9843 in new_test
  File "/data/users/williamwen/pytorch/torch/testing/_internal/common_utils.py", line 2763 in wrapper
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/unittest/case.py", line 589 in _callTestMethod
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/unittest/case.py", line 634 in run
  File "/data/users/williamwen/pytorch/torch/testing/_internal/common_utils.py", line 2862 in _run_custom
  File "/data/users/williamwen/pytorch/torch/testing/_internal/common_utils.py", line 2890 in run
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/unittest/case.py", line 690 in __call__
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/unittest.py", line 343 in runtest
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda>
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/_pytest/config/__init__.py", line 206 in console_main
  File "/data/users/williamwen/py312-debug-env/lib/python3.12/site-packages/pytest/__main__.py", line 7 in <module>
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/runpy.py", line 88 in _run_code
  ...

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, markupsafe._speedups, psutil._psutil_linux, psutil._psutil_posix, cuda_utils (total: 24)

Thread 68 "pt_main_thread" received signal SIGSEGV, Segmentation fault.

Notes:
Running the tests on CPU does not segfault. Running test_mixed_mm_cuda alone passes (most of the time). I also see a segfault on the test_mixed_mm2_cuda test (also preceded by test_multi_gpu_recompile_on_index_cpu).

Sharing in the IR usually makes easier for people to repro/debug, you can get it by setting export MLIR_ENABLE_DUMP=1

https://gist.github.com/williamwen42/7d269e24b6f3cdb85acdef3690cc9a07

Hi, any updates on this? This should be fixed ideally ahead of the PyTorch 2.4 release as we will package in a triton binary.

This may be related to https://github.com/triton-lang/triton/issues/2853.
I've isolated a pytorch minimal repro: https://gist.github.com/williamwen42/b8e8bb1c70c87525430525cecd6fd85e and a triton minimal repro: https://gist.github.com/williamwen42/e75ffd8389aa4b908ffcbc00cdad5790.
This is happening beyond Python 3.12


issue when loading 0.14

Just to confirm, is there a 0.14 version of the utils?
When attempting to load the scenario errors are thrown up see below;

I'll tag the last V0.14 (to be used with the V0.14 scenario code).


Unsubscribe scenarios - documented list request

Do you have a documented set of scenarios yet where the un-subscribe will be triggered? This is critical to IRIS in the design of our UX and in understanding the scope of our interactions with HMRC.

Hi P-Tracy,
As this is a non-technical question, I have forwarded this to SDST for you. They will be in touch with you via e-mail.
Thanks,
Jacob


Lowering ONNXMatMulInteger to Krnl dialect

This patch adds the lowering of ONNXMatMulInteger to Krnl dialect.
The lowering of KrnlMatMul to affine now supports integers for SIMD code. This patch also adds unsigned integer support for add and mul in MathBuilder.
This patch enables bertsquad-12-int8 in Jenkins since it now passes the test.

Jenkins Linux s390x Build #10672 [push] Lowering ONNXMatMulInteg... started at 01:54

Jenkins Linux amd64 Build #10662 [push] Lowering ONNXMatMulInteg... started at 00:54

Jenkins Linux ppc64le Build #9720 [push] Lowering ONNXMatMulInteg... started at 01:57

Jenkins Linux amd64 Build #10662 [push] Lowering ONNXMatMulInteg... passed after 1 hr 27 min

Jenkins Linux ppc64le Build #9720 [push] Lowering ONNXMatMulInteg... passed after 1 hr 52 min