2024-10-27T20:09:46.8495634Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8495848Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8497606Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8497759Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8499499Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8499646Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8501668Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8501798Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8503476Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8503622Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8505539Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8505748Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8507491Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8507663Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8509439Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8509651Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8511406Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8511569Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8513392Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8513791Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8515529Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8515667Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8517378Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8517543Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8519308Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8519463Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8521183Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8521369Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8523118Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8523284Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8524999Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8525148Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8526891Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8527104Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8529102Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8529371Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8531407Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8531609Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8533419Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8533678Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8535801Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8536070Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8537852Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8538050Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8539827Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8539984Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8541730Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8541951Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8543958Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8544162Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8547104Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8547500Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8549800Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8550004Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8551891Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8552064Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8553811Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8553979Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8555688Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8555839Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8557659Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8557896Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8559715Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8559934Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8561701Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8561854Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8563666Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8564049Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8566081Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8566265Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8568120Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8568297Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8570040Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8570179Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8571921Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8572106Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8617719Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8618328Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8620211Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8620476Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8622206Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8622408Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8624219Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8624380Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8626187Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8626384Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8628106Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8628250Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8635470Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8638373Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 898
2024-10-27T20:09:46.8644039Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8646713Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8652343Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8652817Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8654641Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8654950Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8656747Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8656988Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8658883Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8659129Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8660866Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8661027Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8662876Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8663042Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8664890Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8665110Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8667261Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8670293Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8673754Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8673963Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8677321Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8677599Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8681012Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8681198Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8684610Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8684806Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8688131Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8688314Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8690725Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8690947Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8692742Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8692901Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8694610Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8694805Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8696515Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8696654Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8698406Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8698537Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8700242Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8700376Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8702042Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8702192Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8703887Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8704065Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8705768Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8705959Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8707664Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8707798Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8709498Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8709633Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8711386Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8711537Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8713249Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8713396Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8715157Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8715299Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8716966Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8717124Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8718820Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8718994Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8720697Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8720846Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8722543Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8722734Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8724431Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8724560Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8726224Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8726355Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8728030Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8728170Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8729861Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8730192Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8731928Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8732086Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8733975Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8734151Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8735820Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8736001Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8737717Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8737850Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8739515Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8739650Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8741929Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8742181Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8743964Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8744099Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8745766Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8745923Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8747592Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8747724Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8749535Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8749730Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8751440Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8751575Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8753269Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8753471Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8755246Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8755394Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8767224Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8767642Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 898
2024-10-27T20:09:46.8771996Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8774780Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8776694Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8776884Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8778821Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8779039Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8780761Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8780896Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8782611Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8782888Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8784731Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8800251Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8802378Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8802527Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8804452Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8804583Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8806370Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8806479Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8808391Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8808580Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8810782Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8810948Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8813303Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8813470Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8815817Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8816021Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8818045Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8818263Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8820460Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8820612Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8822522Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8822650Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8824865Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8825085Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8827137Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8827286Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8829466Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8829613Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8831406Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8831532Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8833230Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8833319Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8835013Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8835099Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8836769Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8836860Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8838744Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8838875Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8840732Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8840855Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8842524Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8842616Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8844336Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8844478Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8846260Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8846373Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8848080Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8848181Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8850068Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8850192Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8852453Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8852797Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8855146Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8855316Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 8000
2024-10-27T20:09:46.8857212Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:09:46.8857338Z: TdsParserStateObjectManaged.ReleasePacket | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2, Packet DataLeft 7105
2024-10-27T20:10:16.7564980Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:10:16.7584665Z: SqlError.ctor | ERR | Info Number -2, Error State 0, Error Class 11, Error Message 'Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.', Procedure '', Line Number 0, Batch Index -1
2024-10-27T20:10:16.7592891Z: TdsParser.ProcessPendingAck | INFO | Connection Object Id 4, State Obj Id 1, Processing Attention.
2024-10-27T20:10:16.7602042Z: TdsParserStateObjectManaged.ReadSyncOverAsync | Info | State Object Id 1, Session Id a1c1dcc0-26d1-4561-82c0-dd83721009e2
2024-10-27T20:10:16.7602377Z: SqlError.ctor | ERR | Info Number -2, Error State 0, Error Class 11, Error Message 'Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.', Procedure '', Line Number 0, Batch Index -1
2024-10-27T20:10:16.7606074Z: <sc.SqlInternalConnectionTds.BreakConnection|RES|CPOOL> 4, Breaking connection.
2024-10-27T20:10:16.7673652Z: SqlDataReader.Dispose | ERR | Error Message: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding., Stack Trace:    at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlConnection.cs:line 2132
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlInternalConnection.cs:line 774
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParser.cs:line 1452
   at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose) in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 883
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParserStateObject.netcore.cs:line 391
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 2152
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 2045
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 1058
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value) in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 1295
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParser.cs:line 2072
   at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParser.cs:line 1967
   at Microsoft.Data.SqlClient.TdsParser.ProcessAttention(TdsParserStateObject stateObj) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParser.cs:line 8160
   at Microsoft.Data.SqlClient.TdsParser.ProcessPendingAck(TdsParserStateObject stateObj) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParser.cs:line 382
   at Microsoft.Data.SqlClient.TdsParserStateObject.ResetCancelAndProcessAttention() in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParserStateObject.netcore.cs:line 179
   at Microsoft.Data.SqlClient.TdsParserStateObject.CloseSession() in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 747
   at Microsoft.Data.SqlClient.SqlDataReader.TryCloseInternal(Boolean closeReader) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 1062
   at Microsoft.Data.SqlClient.SqlDataReader.Close() in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 936
   at Microsoft.Data.SqlClient.SqlDataReader.Dispose(Boolean disposing) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 851
2024-10-27T20:10:16.7674423Z: CachedAsyncState.ResetAsyncState | API | ObjectId null, Client Connection Id null, AsyncCommandInProgress=null

I think I understand the problem. New test, plug this in into the testharness i've uploaded a couple of times and when you run it you should find that when you run it it never completes, which is what we're seeing here. The only reason we get a failure is the cancellation exception.
        public static PacketData CombinePackets(List<PacketData> packets)
        {
	        int totalLength = SumPacketLengths(packets);
	        byte[] buffer = new byte[totalLength];
	        int offset = 0;
	        for (int index = 0; index < packets.Count; index++)
	        {
		        PacketData packet = packets[index];
		        Array.Copy(packet.Array, packet.Start, buffer, offset, packet.Length);
	        }

	        return new PacketData(buffer,0,totalLength);
        }

        [Fact]
        public static void NewTest2()
        {
	        int dataSize = 300;
	        List<PacketData> expected = new List<PacketData>
	        {
                CreatePacket(120,5),
                CreatePacket(61,6),
                CreatePacket(21,7)
	        };
	        List<PacketData> input = new List<PacketData>
	        {
		        CombinePackets(expected)
	        };

	        var stateObject = new TdsParserStateObject(input, dataSize, isAsync: false);

	        var output = MultiplexPacketList(false, dataSize, input);

	        ComparePacketLists(dataSize, expected, output);
	}

Ok. I understand the problem now. Lets see if I can explain it. I'm specifically using the DataReaderCancellationTest.CancellationTokenIsRespected_ReadAsync but this issue could occur at any point and depends on the exact splitting of packets. I'm hoping it's the same problem in multiple places and that it's not fixed.
All packets in a message sequence are required to be of the packet size defined by the connection apart from the last packet in a message sequence. When a lot of packet splitting is happening we can end up with long sequences of reconstructed packets and at the end of a sequence this means that the last buffer that we receive from the network can contain the trailing part of packet [-2] the complete short final packet [-1] and an additional attention ack packet [-0] which importantly is not part of the original message sequence, it's the start (and end) of a new message.
In the case where had a partial packet and then the next buffer from the network contains the parts of packets outlines above the code was being too strict about handling the contents of the partial packet that was stored in memory. It was using an IsCompleted check to see if the partial packet contained exact the amount of data required to complete the packet which started at 0 in the partial packet buffer. This didn't work where there was more data than a single packet due to the additional packet from the new message sequence and this caused the contents of the partial packet to be ignored and a network read to be attempted.
We had all the data but some additional data I wasn't expecting meant I couldn't use it and instead of using the partial packet which contained the EOM flag I tried to get more data from the network and there wasn't never going to be any. Parser stalled. Eventual timeout.
I think the latest push fixes it. It also cleans up the code a bit, adds tests, excludes some infra methods from code coverage. spelling fixes.

I do currently get a NullReferenceException, same as the pipeline tests
https://sqlclientdrivers.visualstudio.com/public/_build/results?buildId=100536&view=logs&j=d4241037-f2bd-5f17-abd6-2a6a026e10c3&t=69de19b5-ed13-5270-4be9-3a854e5c42e3&l=267

Thanks @Wraith2 - your explanation makes sense. I can see the cause of the NullReferenceException - in TdsParserStateObject.ReadSniSyncOverAsync, PartialPacketContainsCompletePacket initially returns false (because no partial packet is present, because nothing's been read from the network).
As a result of this returning false, no packets are read from the network, readPacket is left uninitialized and passed down to the underlying SNI - either managed or native. This materialises as a NRE in the managed SNI and an AccessViolationException in the native SNI.
I've made a spot fix on L2089 in my local branch, setting it as so:
- bool readFromNetwork = PartialPacketContainsCompletePacket();
+ bool readFromNetwork = _partialPacket == null || PartialPacketContainsCompletePacket();

This lets the indicated test pass, but I need to check to see how the multiplexer uses PartialPacketContainsCompletePacket to see if both references to it truly need to be identical and if the code path through ReadSniSyncOverAsync makes sense for readPacket when readFromNetwork is false. I'll do that this evening.

Thanks @Wraith2 - your explanation makes sense. I can see the cause of the NullReferenceException - in TdsParserStateObject.ReadSniSyncOverAsync, PartialPacketContainsCompletePacket initially returns false (because no partial packet is present, because nothing's been read from the network).
As a result of this returning false, no packets are read from the network, readPacket is left uninitialized and passed down to the underlying SNI - either managed or native. This materialises as a NRE in the managed SNI and an AccessViolationException in the native SNI.
I've made a spot fix on L2089 in my local branch, setting it as so:
- bool readFromNetwork = PartialPacketContainsCompletePacket();
+ bool readFromNetwork = _partialPacket == null || PartialPacketContainsCompletePacket();

This lets the indicated test pass, but I need to check to see how the multiplexer uses PartialPacketContainsCompletePacket to see if both references to it truly need to be identical and if the code path through ReadSniSyncOverAsync makes sense for readPacket when readFromNetwork is false. I'll do that this evening.

This did not resolve the NullReferenceException for me on the CancelledCancellationTokenIsRespected_ReadAsync test

This did not resolve the NullReferenceException for me on the CancelledCancellationTokenIsRespected_ReadAsync test

Maybe https://github.com/dotnet/SqlClient/blob/2e924313b539f9c0df94560bd8dd4c6f1e9f956d/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs#L2089 simply needs to be
bool readFromNetwork = !PartialPacketContainsCompletePacket();
(notice the !)

bool readFromNetwork = !PartialPacketContainsCompletePacket();

With his, I was able to do 100 passes of CancelledCancellationTokenIsRespected_ReadAsync successfully!
Will try the others as well now.

Unfortunately, the CancelledCancellationTokenIsRespected_ReadAsync still hangs for me.
It needed 11 passes on the "default" settings to hang, but I can reproduce it every time if I set the timebomb to these values:
 private async Task TimeBombAsync(SqlCommand command)
  {
      await SleepAsync(2990, 3010);
      command.Cancel();
  }

MichealZ, is there any chance you could reset the state of the SqlClient repo? on the machine I've been using? I tried getting it up to date with the current repo when i pushed previous changes but it somehow ended up identifying changes in >100 files from netfx, not sure what went wrong...
Sure, give me a few minutes. I always have this problem as well, not sure why :)

Sure, give me a few minutes. I always have this problem as well, not sure why :)

Done

Changes pushed, lets see what the CI thinks now. This includes your https://github.com/Wraith2/SqlClient/pull/5 PR @edwardneal
I'll try and get to the multithreaded failure after work. I think I should start by investigating the same thing that showed me the partial packet problem, being stuck in a network read with a non-null _partialPacket, in theory that should be very rare.
Thanks @MichelZ !

Just adding more data. I still get intermittent failures in DataReaderCancellationTest.CancellationTokenIsRespected_ReadAsync. But I also get consistent failures for SqlNotificationTest.Test_SingleDependency_AllDefaults_SqlAuth

I've just tested Test_SingleDependency_AllDefaults_SqlAuth and it's running and succeeding in a loop for me. Can you give it a test on the latest code when you get a chance please.

I've just identified the problem with the AsyncCancelledConnectionsTest.CancelAsyncConnections test and I don't think it's anything to do with this PR. I enhanced the tracker dictionary that I added to the test to help me debug into it and added the connection. With the very exact timing given in https://github.com/dotnet/SqlClient/pull/2714#issuecomment-2443580722 I can reproduce the issue and investigate the connections' state.
Here's what I found:

The connection timeout when running the test suite is infinite. I've hit problems with that before where I've forgotten to start the sql server and the tests don't fail they just hang indefinitely (and confusingly). I believe that we've hit another manifestation of https://github.com/dotnet/SqlClient/issues/422 where there just aren't enough resources to do everything and stuff stops working, one of the thing we've seen in 422 is that things that should succeed or error just hang. Combine that with the fact that it's linux networking and it can only be reproduced on low processor count machines and you've got enough suggestive correlation to convince me. I still cannot reproduce the failure at all on any machine I own even in ubuntu limited to 2 processors.
Can anyone else think of any reason this can't be the case? Can anyone think of a way to validate it? I can't get from the DbInternalConnection instance in the debugger to the TdsInternalConnection to inspect any of the TdsStateParserObject state to confirm that the multiplexer related state it contains is consistent. However since we're looking at login code here if there were a problem with it then it would be quite likely to be hit in every other test that requires a connection, which is everything in the manual testing project.
I've attached the updated test file with the tracker and time changed so you can try to replicate and inspect.
AsyncCancelledConnectionsTest.zip

Building on that, I've added a couple more tracking points: AsyncCancelledConnectionsTest.zip
I think the connection was closed because it was reset already.
What I can see from the added tracking is that it either hangs on ReadAsync() or NextResultAsync()

From the image about we can see that it has done ReadAsync() 52 times successfully.
It does not advance to "read_finished" and it does not track an exception in "exception_read" or "unexpected_exception"
All items that remain in the tracker feature the same pattern.

I've gathered a set of memory dumps and I'm fairly certain you're correct - it's thread pool exhaustion. I've left process 405155 running on the testbed @MichelZ, along with the various dotnet-dump/dotnet-gcdump/etc. utilities. I have a heap dump which includes the TdsParserStateObject instances, and I'm trying to load the PDBs so that I can interpret it.
In my hanging instance, I can see 8 thread pool threads. One is in use by XUnit, another is in use by the thread waiting on Parallel.ForEach; in both cases, these are waiting on the other six. I think Parallel.ForEach has started four instances, (or at least, I can see four TdsParser/TdsParserStateObjectManaged/SNITCPHandle instances) so there's not a lot of room for manoeuvre if we hit a code path which runs async-over-sync, and it looks like this is what happened - it the remaining six seem to have deadlocked on one another.
I haven't yet tested with a wider thread pool, but I'll do that after work.

For what it's worth, I can still reproduce it when setting MinThreads to 32, altough less Tasks seem to hang (i.e. I get 3-8 during the default MinThreads, and 1-2 with 32 MinThreads)
I can see that a callback is registered here (code executed in test):
https://github.com/dotnet/SqlClient/blob/69dc7d415d60ab1533ad6e87df565cc60bac2596/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs#L5579
But the Continuation is never executed for the hanging tasks here:
https://github.com/dotnet/SqlClient/blob/69dc7d415d60ab1533ad6e87df565cc60bac2596/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs#L5390
Maybe that helps getting to the bottom of it?
(I've added simple Console.WriteLine's just before these calls)
It also seems that most calls get short-circuited and completed here, without needing a continuation:
https://github.com/dotnet/SqlClient/blob/69dc7d415d60ab1533ad6e87df565cc60bac2596/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs#L5573
e.g. in my latest test (with 32 MinThreads) I have 3 Register Continuation, and 1 CompleteAsyncCallback, aka 2 hanging.

Your traces concerned me, they indicated that the connection was somehow open which would invalidate my theory.
So i added a state change handler to the connections and dumped the connection state into the tracker on the pre and post debug lines. The output is confusing. I'll reorder it but leave it otherwise unchanged.
Key = "handler", Value = {System.Data.StateChangeEventHandler}
Key = "connection", Value = {InnerConnection = {Microsoft.Data.ProviderBase.DbConnectionClosedNeverOpened}}
Key = "doone.preopen", Value = ({10/30/2024 2:25:27 PM}, Closed)
Key = "(Closed, Open)", Value = (Closed, Open, "   at System.Environment.get_StackTrace()\n   at Microsoft.Data.SqlClient.ManualTesting.Tests.AsyncCancelledConnectionsTest.<>c__DisplayClass8_0.<DoManyAsync>b__0(Object sender, StateChangeEventArgs e) in /home/sqlclient/SqlClient/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs:line 104\n   at Microsoft.Data.SqlClient.SqlConnection.OnStateChange(StateChangeEventArgs stateChange) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlConnection.cs:line 1146\n   at Microsoft.Data.SqlClient.SqlConnection.SetInnerConnectionEvent(DbConnectionInternal to) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlConnectionHelper.cs:line 261\n   at Microsoft.Data.SqlClient.SqlConnectionFactory.SetInnerConnectionEvent(DbConnection owningObject, DbConnectionInternal to) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlConnectionFactory.cs:line 275\n   at Microsoft.Data.ProviderBase.DbConnectionClosedConnecting.TryOp...")
Key = "doone.postopen", Value = ({10/30/2024 2:25:28 PM}, Closed)

So the connection is initially closed at the preopen point, then transitions from closed to open (the (Closed, Open) tuple key) then doesn't transition state again but in the postopen debug point it's now in the closed state again. And no exception was handled.

I have no idea if I'm on the wrong track or not, but I have added the following simple code after this line here:
https://github.com/dotnet/SqlClient/blob/69dc7d415d60ab1533ad6e87df565cc60bac2596/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs#L5621
 var timeout = DateTime.UtcNow.AddMinutes(1);
                while (!testTask.IsCompleted){

                    {
                        Console.WriteLine("Task still waiting.");
                        Thread.Sleep(1000);

                        if (timeout < DateTime.UtcNow){
                            Console.WriteLine("Timeout. Cancelling.");
                            completionSource.TrySetCanceled();
                        }
                    }
                }

Which aborts the Async operation after 1 minute.
This allows the test to complete.
It also means that the TaskCompletionSource here does never complete/fire, it's set to _stateObj._networkPacketTaskSource.
Does that mean we're waiting on the network in this instance for some reason and are not receiving any more data?
Maybe another problem with attention?
Or I could be completely off of course and it has nothing to do with that

Yes, that means it waiting on network data. A way to catch that read being started in the debugger would be idea.

Just adding more data. I still get intermittent failures in DataReaderCancellationTest.CancellationTokenIsRespected_ReadAsync. But I also get consistent failures for SqlNotificationTest.Test_SingleDependency_AllDefaults_SqlAuth

I've just tested Test_SingleDependency_AllDefaults_SqlAuth and it's running and succeeding in a loop for me. Can you give it a test on the latest code when you get a chance please.

I tested Test_SingleDependency_AllDefaults_SqlAuth against main in my environment and it fails there. So that one isn't an issue with this PR. It's something about my setup. You can ignore it.

So what I can see from a packet capture is that for "hanging" tasks, the packets are:
SQL batch
Response (with 0xFD10 aka DONE & DONE_COUNT)
Attention
Response (with 0xFD20 aka DONE & DONE_ATTN)
You can find the full capture here of all tasks:
wireshark.zip
In this case there were 3 hanging Tasks, 3, 50 and 51.
The packets are "marked" by modifying the SELECT statement slightly to include the parent id:
builder.AppendLine($"SELECT name, '{parent}' FROM sys.tables");
I also have a full memory dump of this exchange if someone's interested.
I can see on the TdsParserStateObjectManaged that _attentionSent = true and _snapshotStatus = 0 (so not received?)
The _partialPacket._buffer is:
0x04 0x01 0x00 0x15 0x00 0x35 0x01 0x00 0xFD 0x20 0x00 0xFD 0x00 0x00 [...]
indicating the 0xFD20 DONE DONE_ATTN arrived.

So it's stuck on a network read with a non-null partial packet?

So it's stuck on a network read with a non-null partial packet?

I'm not sure. I can't see any related stacks in the Parallel Stacks view that would suggest it's waiting on the network.
Additionally, _readingCount is 0, which seems to suggest it's not on a network-waiting path?

Here's the memory dump

I can't see any tasks at all when I break into the stuck test with the debugger. It's all very complicated and very confusing.

After cleaning up the debugging i've found that all the stuck connections have an _pendingCallbacks==2 which means 1 from the ctor and 1 from an async read and that there is a complete packet stuck in the _partialPacket. Now to work out how we got there...

Pull and try the latest please.
Run: 100
Test run for /home/sqlclient/SqlClient/artifacts/Project/bin/Unix/Debug.AnyCPU.ManualTests/net8.0/ManualTests.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.5.5+28b46ff510 (64-bit .NET 8.0.10)
[xUnit.net 00:00:00.10]   Discovering: ManualTests (method display = ClassAndMethod, method display options = None)
App Context switch Switch.Microsoft.Data.SqlClient.UseManagedNetworkingOnWindows enabled on Unix <IP_ADDRESS>
[xUnit.net 00:00:00.52] ManualTests: Non-serializable data ('System.Object[]') found for 'Microsoft.Data.SqlClient.ManualTesting.Tests.CertificateTestWithTdsServer.BeginWindowsConnectionTest'; falling back to single test case.
[xUnit.net 00:00:00.52] ManualTests: Non-serializable data ('System.Object[]') found for 'Microsoft.Data.SqlClient.ManualTesting.Tests.CertificateTestWithTdsServer.BeginLinuxConnectionTest'; falling back to single test case.
[xUnit.net 00:00:00.69]   Discovered:  ManualTests (found 953 test cases)
[xUnit.net 00:00:00.70]   Starting:    ManualTests (parallel test collections = on [2 threads], stop on fail = off)
[xUnit.net 00:00:44.50]   Finished:    ManualTests
  Passed Microsoft.Data.SqlClient.ManualTesting.Tests.AsyncCancelledConnectionsTest.CancelAsyncConnections [43 s]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 44.9758 Seconds

This has definitely improved things.
However, I now have this exception:

{[unexpected_exception, Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException: Method Debug.Fail failed with '_pendingCallbacks values is invalid after decrementing: -1
', and was translated to Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException to avoid terminating the process hosting the test.
   at Microsoft.Data.SqlClient.TdsParserStateObject.DecrementPendingCallbacks(Boolean release) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParserStateObject.netcore.cs:line 249
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadAsyncCallback(IntPtr key, PacketHandle packet, UInt32 error) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParserStateObject.netcore.cs:line 495
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSni(TaskCompletionSource`1 completion) in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 2465
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 2058
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 1058
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value) in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 1295
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParser.cs:line 2072
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 3341
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 3630
   at Microsoft.Data.SqlClient.SqlDataReader.ReadAsyncExecute(Task task, Object state) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 5032
   at Microsoft.Data.SqlClient.SqlDataReader.InvokeAsyncCall[T](SqlDataReaderBaseAsyncCallContext`1 context) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 5581
   at Microsoft.Data.SqlClient.SqlDataReader.ReadAsync(CancellationToken cancellationToken) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 5013
   at System.Data.Common.DbDataReader.ReadAsync()
   at Microsoft.Data.SqlClient.ManualTesting.Tests.AsyncCancelledConnectionsTest.RunCommand(SqlConnection connection, ConcurrentDictionary`2 tracker, String commandText, Boolean poison, Int32 parent) in /home/sqlclient/SqlClient/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs:line 250
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.UnwrapPromise`1.TrySetFromTask(Task task, Boolean lookForOce)
   at System.Threading.Tasks.Task.RunOrQueueCompletionAction(ITaskCompletionAction completionAction, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
]}

So this hang seems to be cause by the Debug build from the Assert of _pendingCallbacks here:
https://github.com/dotnet/SqlClient/blob/1f57d73c9e34c3840d6c0964d5b4f54575686450/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParserStateObject.netcore.cs#L249
This causes this call to fail:
https://github.com/dotnet/SqlClient/blob/1f57d73c9e34c3840d6c0964d5b4f54575686450/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs#L2443
Which means that the _readingCount is not decremented here:
https://github.com/dotnet/SqlClient/blob/1f57d73c9e34c3840d6c0964d5b4f54575686450/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs#L2453
My fix for this is to use another try/finally block to encapsulate everything in the outer finally block - right after the Interlocked.Increment here:
https://github.com/dotnet/SqlClient/blob/1f57d73c9e34c3840d6c0964d5b4f54575686450/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs#L2428

               try
                { }
                finally
                {
                    Interlocked.Increment(ref _readingCount);

                    try {
                        handle = SessionHandle;

                        readFromNetwork = !PartialPacketContainsCompletePacket();
                        if (readFromNetwork)
                        {
                            if (!handle.IsNull)
                            {
                                IncrementPendingCallbacks();

                                readPacket = ReadAsync(handle, out error);

                                if (!(TdsEnums.SNI_SUCCESS == error || TdsEnums.SNI_SUCCESS_IO_PENDING == error))
                                {
                                    DecrementPendingCallbacks(false); // Failure - we won't receive callback!
                                }
                            }
                        }
                        else
                        {
                            readPacket = default;
                            error = TdsEnums.SNI_SUCCESS;
                        }
                    }
                    finally
                    {
                        Interlocked.Decrement(ref _readingCount);
                    }
                }

While this makes the test work, I wonder if that DecrementPendingCallbacks assert of _pendingCallbacks values is invalid after decrementing: -1 is a problem. In the test the Exception:
https://github.com/dotnet/SqlClient/blob/1f57d73c9e34c3840d6c0964d5b4f54575686450/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs#L238
gets ignored here, because it's a poison = true run:
https://github.com/dotnet/SqlClient/blob/1f57d73c9e34c3840d6c0964d5b4f54575686450/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs#L154
is this the correct way of doing it?

How did we end up with an unbalanced pendingCount? all the locations I changed I attempted to maintain the invariants like that by changing as little as possible. Even with a read from the partialPacket the count should be incremented and decremented correctly.

Also note the comment in the function:
        internal int DecrementPendingCallbacks(bool release)
        {
            int remaining = Interlocked.Decrement(ref _pendingCallbacks);
            SqlClientEventSource.Log.TryAdvancedTraceEvent("TdsParserStateObject.DecrementPendingCallbacks | ADV | State Object Id {0}, after decrementing _pendingCallbacks: {1}", _objectID, _pendingCallbacks);
            FreeGcHandle(remaining, release);
            // NOTE: TdsParserSessionPool may call DecrementPendingCallbacks on a TdsParserStateObject which is already disposed
            // This is not dangerous (since the stateObj is no longer in use), but we need to add a workaround in the assert for it
            Debug.Assert((remaining == -1 && SessionHandle.IsNull) || (0 <= remaining && remaining < 3), $"_pendingCallbacks values is invalid after decrementing: {remaining}");
            return remaining;
        }

Tests in this repo are not normally run in Debug mode so this could have existed and been happening for a long time and we wouldn't see it. I've got the test running through your script in a loop with this PR changes and i'll see if it gets to 100. If it does then we'll see what people think.

I've also managed to get a NullReferenceException here:
https://github.com/dotnet/SqlClient/blob/1f57d73c9e34c3840d6c0964d5b4f54575686450/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParserStateObjectManaged.cs#L38
with this Stack trace:
at Microsoft.Data.SqlClient.SNI.TdsParserStateObjectManaged.CheckPacket(PacketHandle packet, TaskCompletionSource`1 source) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParserStateObjectManaged.cs:line 38
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadAsyncCallback(IntPtr key, PacketHandle packet, UInt32 error) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParserStateObject.netcore.cs:line 495
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSni(TaskCompletionSource`1 completion) in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 2469
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 2058
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 1058
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value) in /_/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.cs:line 1295
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/TdsParser.cs:line 2072
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 3341
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 3630
   at Microsoft.Data.SqlClient.SqlDataReader.ReadAsyncExecute(Task task, Object state) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 5032
   at Microsoft.Data.SqlClient.SqlDataReader.InvokeAsyncCall[T](SqlDataReaderBaseAsyncCallContext`1 context) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 5581
--- End of stack trace from previous location ---
   at Microsoft.Data.SqlClient.ManualTesting.Tests.AsyncCancelledConnectionsTest.RunCommand(SqlConnection connection, ConcurrentDictionary`2 tracker, String commandText, Boolean poison, Int32 parent) in /home/sqlclient/SqlClient/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs:line 256

This again should only affect Debug builds as far as I can see, but probably still worth fixing?

Another interesting one on an Assert:
Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException: Method Debug.Fail failed with 'Calling command from an object that isn't this reader's command
 ', and was translated to Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException to avoid terminating the process hosting the test.
    at Microsoft.Data.SqlClient.SqlDataReader.Cancel(SqlCommand command) in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlDataReader.cs:line 752
    at Microsoft.Data.SqlClient.SqlCommand.Cancel() in /_/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlCommand.cs:line 1077
    at Microsoft.Data.SqlClient.ManualTesting.Tests.AsyncCancelledConnectionsTest.TimeBombAsync(SqlCommand command) in /home/sqlclient/SqlClient/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs:line 292
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
    at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
    at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
    at Microsoft.Data.SqlClient.ManualTesting.Tests.AsyncCancelledConnectionsTest.SleepAsync(Int32 minMs, Int32 maxMs) in /home/sqlclient/SqlClient/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs:line 303
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
    at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
    at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
    at System.Threading.Tasks.Task.TrySetResult()
    at System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut()
    at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
    at System.Threading.ThreadPoolWorkQueue.Dispatch()
    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Interesting... but I'm going to leave that well alone because as far as I know it's nothing to do with me and it won't cause any problem in production.
I got through 100 iterations in debug mode of the single test that was previous failing. Everything is fixed as far as I know. Other people confirming that would be excellent.

Interesting... but I'm going to leave that well alone because as far as I know it's nothing to do with me and it won't cause any problem in production. I got through 100 iterations in debug mode of the single test that was previous failing. Everything is fixed as far as I know. Other people confirming that would be excellent.

Are you still using the 2990/3010 timings?
And I think I've got additional exception logging added, as some of them are swallowed here - that's probably why you don't see them:
https://github.com/dotnet/SqlClient/blob/00c242602c26c0432c990960a839768e677ca948/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs#L154

I'm not using your timings at the moment I'm on a clean pull of this PR. I did run it last night through 100 iterations with your timings and it succeeded without exceptions.

I have run the test in Release mode and it succeeded the 100 iterations

On the bright side, I've not seen any more test failures with the latest code. However, attempting to stress the async cancel path, I modified CancellationTokenIsRespected_ReadAsync:
        public static async Task CancellationTokenIsRespected_ReadAsync()
        {
            const string longRunningQuery = @"
with TenRows as (select Value from (values (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) as TenRows (Value)),
    ThousandRows as (select A.Value as A, B.Value as B, C.Value as C from TenRows as A, TenRows as B, TenRows as C)
select *
from ThousandRows as A, ThousandRows as B, ThousandRows as C;";

            for (int i = 1; i <= 1000; i++)
            {
                using (var connection = new SqlConnection(DataTestUtility.TCPConnectionString))
                {
                    await connection.OpenAsync();

                    Stopwatch stopwatch = Stopwatch.StartNew();
                    try
                    {
                        using (var source = new CancellationTokenSource())
                        {
                            source.CancelAfter(i);
                            using (var command = new SqlCommand(longRunningQuery, connection))
                            using (var reader = await command.ExecuteReaderAsync(source.Token))
                            {
                                while (await reader.ReadAsync(source.Token))
                                {
                                }
                            }
                        }
                        Assert.True(false, "Cancellation did not trigger exception.");
                    }
                    catch (InvalidOperationException ioe)
                    {
                        Assert.Contains("Operation cancelled by user", ioe.Message);
                    }
                    catch (OperationCanceledException oce)
                    {
                        Assert.Contains("A task was canceled", oce.Message);
                    }
                    catch (SqlException se)
                    {
                        Assert.Contains("Operation cancelled by user", se.Message);
                    }

                    Assert.True(stopwatch.ElapsedMilliseconds < 10000, "Cancellation did not trigger on time.");
                }
            }
        }

I've been running multiple times in debug mode and it sometimes fails. Success takes about 9 minutes for me. But it's failed twice for me now.

Also, it this not rather unfairly moving the goals? The original report that there was a failure in that test was sound and the only problem was that I couldn't reproduce it, given a reproduction I've managed to track it down and eliminate it. Searching for every more stressful situations to show problems in debug mode seem unfair since it's never going to run in debug mode in tests or for users.

Yeah, I think we‘re way better off with this PR merged. Additional issues should probably be handled as normal bugs against main?

I think we can only merge when we're sure it's stable.
I also think that running in modes that are not realistic and for extreme extended periods of time to generate errors and then waiting for me to track them down is stressful and frustrating. If you're going to do that then I need help from the MS side to produce a repeatable reproduction that I can spend fruitful time debugging and fixing.
Though the time this has been open I've not seen any evidence of code review from the maintainers and that's absolutely required. Fuzzing through increasingly stressful test runs is not sufficient evidence of correctness. Has anyone read through the changes to see if they're logically correct?

How many iterations does it take to generate a failure? and is there a way to replicate it without running for 9 minutes? Also, it this not rather unfairly moving the goals? The original report that there was a failure in that test was sound and the only problem was that I couldn't reproduce it, given a reproduction I've managed to track it down and eliminate it. Searching for every more stressful situations to show problems in debug mode seem unfair since it's never going to run in debug mode in tests or for users.

It fails about every other run for me, maybe more, usually in less than a minute. In Release mode, a failure looks like an InvalidOperationException with a message of Internal connection fatal error. and doesn't provide any helpful details on what the problem is.
 Microsoft.Data.SqlClient.ManualTesting.Tests.DataReaderCancellationTest.CancellationTokenIsRespected_ReadAsync
   Source: DataReaderCancellationTest.cs line 22
   Duration: 625 ms

  Message:
    Assert.Contains() Failure
    Not found: Operation cancelled by user
    In value:  Internal connection fatal error.

  Stack Trace:
    DataReaderCancellationTest.CancellationTokenIsRespected_ReadAsync() line 52
    --- End of stack trace from previous location ---

I understand this might feel like moving the goalposts, but I'm trying to target the scenarios testing this has exposed as weak points in the flow (cancellations). The existing code has had a lot of time out in the wild to work out bugs. I'm also not saying this new test is a blocker. However, if we can show that the test always passes on main but fails here, I'd much rather catch these difficult bugs now rather than try and grok the code again in the future when it will be even harder to identify.
I don't want to downplay the amount of effort that has gone in here by @Wraith2 and now @MichelZ in helping test. It's really awesome to see this so close and I truly appreciate the effort that has been put in. But this is a critical path that we need to get right. We've had security issues in the past where the result stream gets messed up and wrong results go to the wrong command. I don't know if bugs here can result in that, but the risk feels real to me.

I think we can only merge when we're sure it's stable.

I agree.

I also think that running in modes that are not realistic and for extreme extended periods of time to generate errors and then waiting for me to track them down is stressful and frustrating. If you're going to do that then I need help from the MS side to produce a repeatable reproduction that I can spend fruitful time debugging and fixing.

I also agree there. I'm not going to hold this up if the repro is too difficult. I'm trying to narrow it down but also share findings sooner than later to make sure the PR doesn't just sit.

Though the time this has been open I've not seen any evidence of code review from the maintainers and that's absolutely required. Fuzzing through increasingly stressful test runs is not sufficient evidence of correctness. Has anyone read through the changes to see if they're logically correct?

I've read through the changes. They look good to me, however, the sheer complexity of them means bugs will easily slip past a reading. I've encouraged others to keep an eye on this one but they are likely waiting for the bugs to be found before focusing on it too much. It was only yesterday that the existing tests started passing consistently.
I'm going to run the modified test against main to see if it passes consistently there.
Here's another failure (debug, Windows, net6, native SNI). Do these details help? How can I help narrow it down, if not?

  Stack Trace:
Debug.Fail(String message)
TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) line 2066
TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) line 1951
TdsParser.ProcessAttention(TdsParserStateObject stateObj) line 8109
TdsParser.ProcessPendingAck(TdsParserStateObject stateObj) line 365
TdsParserStateObject.ResetCancelAndProcessAttention() line 179
TdsParserStateObject.CloseSession() line 747
SqlDataReader.TryCloseInternal(Boolean closeReader) line 1005
SqlDataReader.Close() line 933
SqlDataReader.Dispose(Boolean disposing) line 848
DataReaderCancellationTest.CancellationTokenIsRespected_ReadAsync() line 45
ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
Task.RunContinuations(Object continuationObject)
Task`1.TrySetResult(TResult result)
TaskCompletionSource`1.TrySetResult(TResult result)
SqlDataReader.CompleteAsyncCall[T](Task`1 task, SqlDataReaderBaseAsyncCallContext`1 context) line 5662
SqlDataReaderBaseAsyncCallContext`1.CompleteAsyncCallCallback(Task`1 task, Object state) line 5361
ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
ThreadPoolWorkQueue.Dispatch()
WorkerThread.WorkerThreadStart()
Thread.StartCallback()

Do these details help? How can I help narrow it down, if not?

In each case where something goes wrong we need to know how we got there. In those dumps you can see that a large part of the packet data is filled with FF, that indicates that the particular buffer Is likely part of a multiple packets that got stuck together then the multiplexer pulled them apart filling the unused space with f's to help debugging.
The problem is that knowing how we got to any particular place is complicated and the more test code you add the more you disrupt timings which change the observed behaviour. Ideally we'd be able to dump all packets at on the connection we're interested in as they leave the network layer for the tds layer but when we're running 100 connections this is both disruptive and very hard to implement. Each thread is using 350+ packets in just this short test.
Given a sequence of 8K buffers we can, in theory, just run them through the test harness and see if the multiplexer deconstructs and reconstructs them successfully. Unfortunately there is another class of problem which the test harness can't cover, errors caused by internal behaviour of the real library in consuming packets. The test harness runs packets. The real library uses the data inside the packets to run.
 The 04 00 and 04 01are the indicators i'm looking for, they've followed by the length. So it looks like the current packet header is04 01 00 40 00 38 F8 00` meaning 0x40 bytes of data, which is 64, so 56 bytes of payload meaning that
`04 01 00 40 00 38 F8 00 00
unused buffer:
00 00 06 00 00 00 02 00 00 00 0A 00 00 00 05 00
00 00 01 00 00 00 08 00 00 00 E4 0B 00 00 00 DF
00 00 00 00 FE 04 03 00 00 00 FD 02 00 C1 00 00
00 00 00 00 00 00 00```
is one complete packet, and the buffer also contains another packet header `04 01 00 15 00 38 01 00` 0x15 length is 21, -8 means 13 bytes of payload which is
```04 01 00 15 00 38 01 00 FD
20 00 FD 00 00 00 00 00 00 00 00 00```

after this it's all the F's which shows that we've got the right bits of data because all the non-right bits were blanked.

So the data is correctly accounted for but the current packet buffer should have been split out into two packets. The data inside the packet we're currently reading is the thing that triggered the IsValidTdsToken failure but it doesn't look like the failure is caused by packet reconstruction logic because the packets contents and lengths add up correctly.

How do we fix it? I need a reproduction I can run quickly to the point of failure multiple times so I can add in debugging statements and breakpoints to poke at it.

In this specific case I'd add something to console print, wait for attached debugger, break, whatever works and try to identify how a partial packet from PartialPacket with ContainsCompletePacket=true and DataLength>RequiredLength ended up being used in a call to SetBuffer because it shouldn't have been without passing through the multiplexer.

I've rebased to main to fix the conflicts. Locally the net6 tests won't build but I'm hoping the CI has no problems with them. I've also added a try-catch around pending read increment/decrement which @MichelZ has found causing him problems during testing, other places in the code use a try catch pattern with this variable so it feels natural to add it in the location I have.
I ran tests on windows with native SNI over the weekend and didn't enounter any test failures that trace back to partial packets. I had a small number of test failures where the cancellation had not been sent by the time the test completed (_attentionSending and _attentionSent were false). Those are a timing issue not to do with the change I've made.
@cheenamalhotra and @saurabh500 can you review please?
Given the timescales for the 6.0 release being fixed and it being an important LTS milestone I no longer think that this change should be part of that release. I think that the 6.0 should go out without this change ( and the required following change to enable fast async string) and that the versions after that should include these changes.
Opinions?

On the bright side, I've not seen any more test failures with the latest code. However, attempting to stress the async cancel path, I modified CancellationTokenIsRespected_ReadAsync:
        public static async Task CancellationTokenIsRespected_ReadAsync()
        {
            const string longRunningQuery = @"
with TenRows as (select Value from (values (1), (2), (3), (4), (5), (6), (7), (8), (9), (10)) as TenRows (Value)),
    ThousandRows as (select A.Value as A, B.Value as B, C.Value as C from TenRows as A, TenRows as B, TenRows as C)
select *
from ThousandRows as A, ThousandRows as B, ThousandRows as C;";

            for (int i = 1; i <= 1000; i++)
            {
                using (var connection = new SqlConnection(DataTestUtility.TCPConnectionString))
                {
                    await connection.OpenAsync();

                    Stopwatch stopwatch = Stopwatch.StartNew();
                    try
                    {
                        using (var source = new CancellationTokenSource())
                        {
                            source.CancelAfter(i);
                            using (var command = new SqlCommand(longRunningQuery, connection))
                            using (var reader = await command.ExecuteReaderAsync(source.Token))
                            {
                                while (await reader.ReadAsync(source.Token))
                                {
                                }
                            }
                        }
                        Assert.True(false, "Cancellation did not trigger exception.");
                    }
                    catch (InvalidOperationException ioe)
                    {
                        Assert.Contains("Operation cancelled by user", ioe.Message);
                    }
                    catch (OperationCanceledException oce)
                    {
                        Assert.Contains("A task was canceled", oce.Message);
                    }
                    catch (SqlException se)
                    {
                        Assert.Contains("Operation cancelled by user", se.Message);
                    }

                    Assert.True(stopwatch.ElapsedMilliseconds < 10000, "Cancellation did not trigger on time.");
                }
            }
        }

I've run this quite a few times now on multiple machines, and while I get some cancellations did not trigger on time, I have nerver seen another failure from this.
I also count the timing issues and do not abort the test, so tests with timing issues ran to completion

I've run this a few more times on my systems and the results look good to me. I'm tempted to say put it in 6.0 because I really want the async perf improvement in there, but that's probably not wise. 😄

It depends on the time remaining. The next PR is substantially easier and smaller. I thought release date was going to coincide with netconf but that's gone already.

According to the 6.0.0 Milestone it's currently planned for Dec 9th

Dates got pushed out a bit. Milestones have the current plans.

Just checking where this is up to. I see we've got a several new reviewers. If you want a teams call to talk through what I've done just let me know.

Hi @Wraith2, thanks for the detailed writeups and design notes. I really appreciate your thorough approach to this. I'm working through this PR to add my review and I'd love to connect on teams soon to check my understanding. My microsoft alias is the same as my github alias.
Also, apologies if I missed some details on this somewhere, but you had a point in the original PR that partial packets were possible due to an "underlying incorrect assumption in the networking handling code". Can you share any additional details on this? It would help my understanding of how this change interacts with the networking code. Thanks!

Email sent.
After spending more time in debugging for this PR I think that the problem was an oversight caused by not understanding the original code completely.
The original sync code that is the core of the library must be able to handle partial packets because if it couldn't then sync code would never work on linux because the occurrence of partial packets on linux is commonplace. This means that the sync code path doesn't need to check that the input buffer contains a complete packet because it doesn't care, it can handle incomplete packets.
The async code was added a long time after the original sync code. The async code causes long delays when executing the replay mechanism used to support multi-packet payloads. Those delays mean that it's virtually impossible to observe partial packets while async. The async code takes entire input buffers and adds them to the packet list that it uses. It assumes that those input buffers are complete. This was a reasonable assumption given the observable data. Unfortunately I think it was wrong.
If you run the code fast enough in async multi-packet situations you can encounter a network read where packet-size of data was requested but less than packet-size of data was available in the os buffer even though the complete packet will be packet-size in length. The read will return partial-first-size of data and the async code path does not check that partial-first-size against the size of the full packet defined in the packet header. This packet will be added to the async buffer list.
The next data will not start with a tds header because the next data received will be the remainder of the partial packet. The async playback mechanism does not handle this case and will eventually attempt to replay the packet and interpret the first 8 bytes of the packet as a tds header, this will result in incorrect things happening and (hopefully) a crash.
I believe the current code is incapable of running fast enough to encounter the problem. When you speed up async multi-packet reads it occurs often enough to spoil test runs but it's not entirely deterministic and the more debugging logic you add to try and catch it the slower execution becomes and the less it happens.
When it happens you get a packet buffer list that contains data like the below. This is one of the very few times I managed to catch it under the debugger. The first 8 bytes must contain the packet header and you can see that the final packet does not.

Thank you, that makes sense. I wanted to make sure there wasn't some additional low-level networking behavior we were working around. Email received; I'll reach out shortly.

The need for multiplexing is clear given the current code structure, and the multiplexing logic itself is well contained. The tests give me confidence that we're multiplexing correctly.

Excellent. The tests along with all the new issues that occurred with the linux CI runs are the best indicator that we're doing to get until it arrives with end users.

I have less confidence in the overall orchestration of network calls and that the multiplexing logic is invoked correctly. There are too many different locations and situations in which the logic is invoked, I can't really reason about it. At the end of the day, we may just have to trust our existing code coverage.

The multiplexing was invoked incorrectly. More precisely I'd missed a couple of places that pending reads could terminate and those have now been added. The effort required to untangle the internal layering is very high, the risk is very high and there are no immediate benefits to consumers. We've simplified and transformed some parts of the codebase over the years this repository has existed but the core parts are hard to change safely.
In general large refactorings are unlikely to be helpful in this current codebase when the time could be spent on SqlClientX bringing it to feature parity.

Code Structure
The big thing that makes this challenging to reason about is that we're looping in two places (one for network reads, one for packet multiplexing), 4 or 5 layers apart and coordinating between them with state. The loops are also inverted from what you would expect (network read outside, multiplex inside). There's too much indirection at the moment. I really want us to move toward tighter loops at the same layer.

There's also the smux layer. The multiplexer sits above that so I don't have to deal with untangling the smux header and the tds header.
Using less indirection by dropping the unmanaged sni implementation entirely is a long term goal. It will allow us to collapse some layers of indirection. Unfortunately we can't do that until the performance of the managed sni is roughly equivalent to the managed version. The last time I saw numbers mentioned it was ~10 throughput behind. It also has problems that we can't solve like https://github.com/dotnet/SqlClient/issues/422 which tracks back to the runtime.

Performance numbers
It would be exciting to compile some performance numbers to show the impact of this change. Do you have any tests in mind that you used while implementing? I'll also think about anything I can add on.

I haven't run any performance numbers for this PR in isolation. In general I split my work into two categories; Perf and Fixes. For perf work numbers are the main driving force so I'll explain what I'm measuring and why. For fixes correctness is the goal and unless I have reason to believe that it will negatively affect performance and even then I'll make changes that slow things down because I consider correctness more important than speed. I consider this (and the previous part-1) a Fix pr for correctness.
For this PR there will be some level of performance drop. I expect it to be small because on the happy path there should be few to no new allocations with some constant cpu required to evaluate the new path. On unhappy paths where we require reconstructions we will observe the query that encounters the fragmentation slow down when it hits the fragmented buffers and it will then continue seeing fragments until the query ends. The next query on the same connection will start with unfragmented packets again and will see the typical speed.
If your workload is fixed length type heavy you should see little to no change. The small additional packet receive time will be hidden by the work required to get the values out of the packet. If your workload contains large multi-buffer values than this will slow you down more noticeably but part-3 of this series of PR's will linearize multi-buffer fetches so you're probably doing to see an overall speed increase.