MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch

Which service(blob, file, queue) does this issue concern?
Blob
Which version of the SDK was used? Please provide the output of pip freeze.
azure-storage-blob 2.1.0
What problem was encountered?get_blob_to_bytes(container, missing_filename, validate_content=True).content is NOT reporting any MD5 mismatch issue while Azure Databricks filesystem API (%fs head OR with open(file) as f:  print (f.read()) reporting MD5 mismatch error and read fails.
Have you found a mitigation/solution?
No
Note: for table service, please post the issue here instead: https://github.com/Azure/azure-cosmosdb-python.

Hi @pchinta, thanks for reaching out.
Could you please provide a bit more details so that we can try to repro this issue? How big was the file? Have you validated the stored MD5 on your own?

Blaize

Hi,
The blob is stored on a storage account owned by Blaize (copied on the thread). Blob size is 4MB and it is outlook mail. Since its PII info we do not have copy of the blob, and so did not validate MD5 of the blob. Attached is the error reported for the blob in scope of the issue by databricks file system API for the same file.
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
@Blaize Berry Blaize.Berry<EMAIL_ADDRESS>Could you please help us understand on MD5 computation to setup repro by Azure Storage SDK team?
Best Regards,
Purna Chandra Rao Chinta
Support Escalation Engineer
Microsoft Azure Rapid Response Team
Email : pchinta<EMAIL_ADDRESS>Working hours: 0900 – 2000 Central Time (Wed – Sat)
If you need immediate attention for any of the ongoing case while I am out of shift, please send a separate email with case ID to arrbackup<EMAIL_ADDRESS>A Duty Manager will take action and loop in an available ARR engineer to help you.  If you have any feedback of your Support Experience, please feel free to contact my manager – Kirk Beller at kbeller<EMAIL_ADDRESS>or +1<PHONE_NUMBER>.
[Microsoft Logo]
From: Ze Qian Zhang<EMAIL_ADDRESS>Sent: Wednesday, August 14, 2019 5:46 PM
To: Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi @pchintahttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpchinta&data=02|01|pchinta%40microsoft.com|6e72cd58a18d4f7db57008d721093a46|72f988bf86f141af91ab2d7cd011db47|1|0|637014195828947831&sdata=wAKis%2BQVe%2BWIbk0dYLHiXYUtxcW%2FwfLjSK9dP5wGBM4%3D&reserved=0, thanks for reaching out.
Could you please provide a bit more details so that we can try to repro this issue? How big was the file? Have you validated the stored MD5 on your own?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fissues%2F627%3Femail_source%3Dnotifications%26email_token%3DAM3JFIHLMJHRZFNUS2XWYWTQESDLZA5CNFSM4ILZHVI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KKTTA%23issuecomment-521447884&data=02|01|pchinta%40microsoft.com|6e72cd58a18d4f7db57008d721093a46|72f988bf86f141af91ab2d7cd011db47|1|0|637014195828947831&sdata=az6aXvMPX4uG9VqFwhvZpMDcATF21LGfrHehXe0ADXI%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM3JFIEJ5QEJDVLATFWGRS3QESDLZANCNFSM4ILZHVIQ&data=02|01|pchinta%40microsoft.com|6e72cd58a18d4f7db57008d721093a46|72f988bf86f141af91ab2d7cd011db47|1|0|637014195828957825&sdata=T6WUgzH%2FjPkfVe2NyZiXfsfY6s3Pdt%2BPTQG49GZxhlc%3D&reserved=0.
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:737)
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:855)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at com.databricks.backend.daemon.dbutils.FSUtils$$anonfun$head$1.apply(DBUtilsCore.scala:200)
at com.databricks.backend.daemon.dbutils.FSUtils$$anonfun$head$1.apply(DBUtilsCore.scala:190)
at com.databricks.backend.daemon.dbutils.FSUtils$.com$databricks$backend$daemon$dbutils$FSUtils$$withFsSafetyCheck(DBUtilsCore.scala:81)
at com.databricks.backend.daemon.dbutils.FSUtils$.head(DBUtilsCore.scala:190)
at com.databricks.dbutils_v1.impl.DbfsUtilsImpl.head(DbfsUtilsImpl.scala:53)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw$$iw$$iw$$iw$$iw.(command-940836796107301:1)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw$$iw$$iw$$iw.(command-940836796107301:44)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw$$iw$$iw.(command-940836796107301:46)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw$$iw.(command-940836796107301:48)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw.(command-940836796107301:50)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw.(command-940836796107301:52)
at line75fe9781da5b4e23ae11050a735562e331.$read.(command-940836796107301:54)
at line75fe9781da5b4e23ae11050a735562e331.$read$.(command-940836796107301:58)
at line75fe9781da5b4e23ae11050a735562e331.$read$.(command-940836796107301)
at line75fe9781da5b4e23ae11050a735562e331.$eval$.$print$lzycompute(:7)
at line75fe9781da5b4e23ae11050a735562e331.$eval$.$print(:6)
at line75fe9781da5b4e23ae11050a735562e331.$eval.$print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:215)
at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply$mcV$sp(ScalaDriverLocal.scala:197)
at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:197)
at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:197)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:679)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:632)
at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:197)
at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$8.apply(DriverLocal.scala:368)
at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$8.apply(DriverLocal.scala:345)
at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:233)
at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:48)
at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:271)
at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:48)
at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:345)
at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:644)
at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:644)
at scala.util.Try$.apply(Try.scala:192)
at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:639)
at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:485)
at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:597)
at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:390)
at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:855)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at com.databricks.backend.daemon.dbutils.FSUtils$$anonfun$head$1.apply(DBUtilsCore.scala:200)
at com.databricks.backend.daemon.dbutils.FSUtils$$anonfun$head$1.apply(DBUtilsCore.scala:190)
at com.databricks.backend.daemon.dbutils.FSUtils$.com$databricks$backend$daemon$dbutils$FSUtils$$withFsSafetyCheck(DBUtilsCore.scala:81)
at com.databricks.backend.daemon.dbutils.FSUtils$.head(DBUtilsCore.scala:190)
at com.databricks.dbutils_v1.impl.DbfsUtilsImpl.head(DbfsUtilsImpl.scala:53)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw$$iw$$iw$$iw$$iw.(command-940836796107301:1)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw$$iw$$iw$$iw.(command-940836796107301:44)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw$$iw$$iw.(command-940836796107301:46)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw$$iw.(command-940836796107301:48)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw$$iw.(command-940836796107301:50)
at line75fe9781da5b4e23ae11050a735562e331.$read$$iw.(command-940836796107301:52)
at line75fe9781da5b4e23ae11050a735562e331.$read.(command-940836796107301:54)
at line75fe9781da5b4e23ae11050a735562e331.$read$.(command-940836796107301:58)
at line75fe9781da5b4e23ae11050a735562e331.$read$.(command-940836796107301)
at line75fe9781da5b4e23ae11050a735562e331.$eval$.$print$lzycompute(:7)
at line75fe9781da5b4e23ae11050a735562e331.$eval$.$print(:6)
at line75fe9781da5b4e23ae11050a735562e331.$eval.$print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:215)
at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply$mcV$sp(ScalaDriverLocal.scala:197)
at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:197)
at com.databricks.backend.daemon.driver.ScalaDriverLocal$$anonfun$repl$1.apply(ScalaDriverLocal.scala:197)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:679)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:632)
at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:197)
at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$8.apply(DriverLocal.scala:368)
at com.databricks.backend.daemon.driver.DriverLocal$$anonfun$execute$8.apply(DriverLocal.scala:345)
at com.databricks.logging.UsageLogging$$anonfun$withAttributionContext$1.apply(UsageLogging.scala:238)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at com.databricks.logging.UsageLogging$class.withAttributionContext(UsageLogging.scala:233)
at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:48)
at com.databricks.logging.UsageLogging$class.withAttributionTags(UsageLogging.scala:271)
at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:48)
at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:345)
at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:644)
at com.databricks.backend.daemon.driver.DriverWrapper$$anonfun$tryExecutingCommand$2.apply(DriverWrapper.scala:644)
at scala.util.Try$.apply(Try.scala:192)
at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:639)
at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:485)
at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:597)
at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:390)
at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
at java.lang.Thread.run(Thread.java:748)

We are not explicitly computing the MD5 of the blob. I believe that functionality is provided out of the box via the Python SDK when the validate_content flag is set to True. I believe that there is some subtlety to the computation in that an MD5 is only evaluated for the first 4 MB with that flag set to true for performance purposes (see https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/baseblobservice.py#L1923
), but the particular blob that we’re encountering hash mismatch errors with in Databricks mounted storage is less than 1 MB in size so this shouldn’t be a factor. I can’t comment with 100% certainty on how Databricks computes the MD5 with their mounted storage setup, but from the logs I’ve seen it looks like they’re using the Java SDK for Azure Storage under the hood.
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Date: Wednesday, August 14, 2019 at 6:00 PM
To: Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Blaize Berry<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Microsoft - Alex Kaps<EMAIL_ADDRESS>Subject: EXT: RE: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)

Blaize

Hi,
The blob is stored on a storage account owned by Blaize (copied on the thread). Blob size is 4MB and it is outlook mail. Since its PII info we do not have copy of the blob, and so did not validate MD5 of the blob. Attached is the error reported for the blob in scope of the issue by databricks file system API for the same file.
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
@Blaize Berry Blaize.Berry<EMAIL_ADDRESS>Could you please help us understand on MD5 computation to setup repro by Azure Storage SDK team?
Best Regards,
Purna Chandra Rao Chinta
Support Escalation Engineer
Microsoft Azure Rapid Response Team
Email : pchinta<EMAIL_ADDRESS>Working hours: 0900 – 2000 Central Time (Wed – Sat)
If you need immediate attention for any of the ongoing case while I am out of shift, please send a separate email with case ID to arrbackup<EMAIL_ADDRESS>A Duty Manager will take action and loop in an available ARR engineer to help you.  If you have any feedback of your Support Experience, please feel free to contact my manager – Kirk Beller at kbeller<EMAIL_ADDRESS>or +1<PHONE_NUMBER>.
[Microsoft Logo]
From: Ze Qian Zhang<EMAIL_ADDRESS>Sent: Wednesday, August 14, 2019 5:46 PM
To: Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi @pchintahttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpchinta&data=02|01|pchinta%40microsoft.com|6e72cd58a18d4f7db57008d721093a46|72f988bf86f141af91ab2d7cd011db47|1|0|637014195828947831&sdata=wAKis%2BQVe%2BWIbk0dYLHiXYUtxcW%2FwfLjSK9dP5wGBM4%3D&reserved=0, thanks for reaching out.
Could you please provide a bit more details so that we can try to repro this issue? How big was the file? Have you validated the stored MD5 on your own?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fissues%2F627%3Femail_source%3Dnotifications%26email_token%3DAM3JFIHLMJHRZFNUS2XWYWTQESDLZA5CNFSM4ILZHVI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KKTTA%23issuecomment-521447884&data=02|01|pchinta%40microsoft.com|6e72cd58a18d4f7db57008d721093a46|72f988bf86f141af91ab2d7cd011db47|1|0|637014195828947831&sdata=az6aXvMPX4uG9VqFwhvZpMDcATF21LGfrHehXe0ADXI%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM3JFIEJ5QEJDVLATFWGRS3QESDLZANCNFSM4ILZHVIQ&data=02|01|pchinta%40microsoft.com|6e72cd58a18d4f7db57008d721093a46|72f988bf86f141af91ab2d7cd011db47|1|0|637014195828957825&sdata=T6WUgzH%2FjPkfVe2NyZiXfsfY6s3Pdt%2BPTQG49GZxhlc%3D&reserved=0.

Hi,
We can compute MD5 of the blob in storage account and see if MD5 being updated on blob is correct..
https://github.com/giventocode/azure-blob-md5
For windows:

Install GO (as on link).
Create a folder for md5 and open cmdprompt (as admin), browse to the folder and run below commands:

go get github.com/giventocode/azure-blob-md5
go build -o bmd5.exe github.com/giventocode/azure-blob-md5

Create environment variables as ACCOUNT_NAME and ACCOUNT_KEY and provide values inline to storage account in scope of issue.
Open another prompt (as admin) as environment variables created above will not be available at cmd prompt at step 2 above.
Browse to “bmd5.exe” (at path on step 2) and run below command:

bmd5 -b blob -c container

Validate the MD5 generated with

MD5 reported on error (one of them)

MD5 on blob (content-md5 on azure portal).
If both match, then there is issue with md5 computed by databricks file system API.

Best Regards,
Purna Chandra Rao Chinta
From: Blaize Berry<EMAIL_ADDRESS>Sent: Thursday, August 15, 2019 12:32 AM
To: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Alex Kaps<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
We are not explicitly computing the MD5 of the blob. I believe that functionality is provided out of the box via the Python SDK when the validate_content flag is set to True. I believe that there is some subtlety to the computation in that an MD5 is only evaluated for the first 4 MB with that flag set to true for performance purposes (see https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/baseblobservice.py#L1923https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fblob%2Fmaster%2Fazure-storage-blob%2Fazure%2Fstorage%2Fblob%2Fbaseblobservice.py%23L1923&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579426298&sdata=YBreKhiI43sdsMDxS%2FsU7xs1PkpXW3ec3zJbgeNF5s8%3D&reserved=0
), but the particular blob that we’re encountering hash mismatch errors with in Databricks mounted storage is less than 1 MB in size so this shouldn’t be a factor. I can’t comment with 100% certainty on how Databricks computes the MD5 with their mounted storage setup, but from the logs I’ve seen it looks like they’re using the Java SDK for Azure Storage under the hood.
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Purna Chandra Rao Chinta <pchinta<EMAIL_ADDRESS>Date: Wednesday, August 14, 2019 at 6:00 PM
To: Azure/azure-storage-python <reply<EMAIL_ADDRESS>Azure/azure-storage-python <azure-storage-python<EMAIL_ADDRESS>Blaize Berry <Blaize.Berry<EMAIL_ADDRESS>Cc: Mention <mention<EMAIL_ADDRESS>Microsoft - Alex Kaps <Alexander.Kaps<EMAIL_ADDRESS>Subject: EXT: RE: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)

Blaize

Hi,
The blob is stored on a storage account owned by Blaize (copied on the thread). Blob size is 4MB and it is outlook mail. Since its PII info we do not have copy of the blob, and so did not validate MD5 of the blob. Attached is the error reported for the blob in scope of the issue by databricks file system API for the same file.
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
@Blaize Berry Blaize.Berry<EMAIL_ADDRESS>Could you please help us understand on MD5 computation to setup repro by Azure Storage SDK team?
Best Regards,
Purna Chandra Rao Chinta
Support Escalation Engineer
Microsoft Azure Rapid Response Team
Email : pchinta<EMAIL_ADDRESS>Working hours: 0900 – 2000 Central Time (Wed – Sat)
If you need immediate attention for any of the ongoing case while I am out of shift, please send a separate email with case ID to arrbackup<EMAIL_ADDRESS>A Duty Manager will take action and loop in an available ARR engineer to help you.  If you have any feedback of your Support Experience, please feel free to contact my manager – Kirk Beller at kbeller<EMAIL_ADDRESS>or +1<PHONE_NUMBER>.
[Microsoft Logo]
From: Ze Qian Zhang <notifications<EMAIL_ADDRESS>Sent: Wednesday, August 14, 2019 5:46 PM
To: Azure/azure-storage-python <azure-storage-python<EMAIL_ADDRESS>Cc: Purna Chandra Rao Chinta <pchinta<EMAIL_ADDRESS>Mention <mention<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi @pchintahttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpchinta&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579426298&sdata=aycTaj60TMgqx4QlUZy7syClXoqsD4IN0s1uimXco%2FI%3D&reserved=0, thanks for reaching out.
Could you please provide a bit more details so that we can try to repro this issue? How big was the file? Have you validated the stored MD5 on your own?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fissues%2F627%3Femail_source%3Dnotifications%26email_token%3DAM3JFIHLMJHRZFNUS2XWYWTQESDLZA5CNFSM4ILZHVI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KKTTA%23issuecomment-521447884&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579436293&sdata=%2BJO12gj6nurUQFeXeN%2B1l4oxjv%2Ftxh2a4JVTGBc4N9M%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM3JFIEJ5QEJDVLATFWGRS3QESDLZANCNFSM4ILZHVIQ&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579446294&sdata=gvvVdf%2B8NlRhoEYktjq%2F%2FKuxec5u%2BAho81CPZvObYUE%3D&reserved=0.

Hi Purna,
I have computed the MD5 of the blob in the storage account and got c1b80a33c85d2b1cb092d92fc817324d, which is consistent with the ContentMD5 blob property.
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Date: Thursday, August 15, 2019 at 7:25 AM
To: Blaize Berry<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Microsoft - Alex Kaps<EMAIL_ADDRESS>Subject: EXT: RE: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi,
We can compute MD5 of the blob in storage account and see if MD5 being updated on blob is correct..
https://github.com/giventocode/azure-blob-md5
For windows:

Install GO (as on link).
Create a folder for md5 and open cmdprompt (as admin), browse to the folder and run below commands:

go get github.com/giventocode/azure-blob-md5
go build -o bmd5.exe github.com/giventocode/azure-blob-md5

Create environment variables as ACCOUNT_NAME and ACCOUNT_KEY and provide values inline to storage account in scope of issue.
Open another prompt (as admin) as environment variables created above will not be available at cmd prompt at step 2 above.
Browse to “bmd5.exe” (at path on step 2) and run below command:

bmd5 -b blob -c container

Validate the MD5 generated with

MD5 reported on error (one of them)

MD5 on blob (content-md5 on azure portal).
If both match, then there is issue with md5 computed by databricks file system API.

Best Regards,
Purna Chandra Rao Chinta
From: Blaize Berry<EMAIL_ADDRESS>Sent: Thursday, August 15, 2019 12:32 AM
To: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Alex Kaps<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
We are not explicitly computing the MD5 of the blob. I believe that functionality is provided out of the box via the Python SDK when the validate_content flag is set to True. I believe that there is some subtlety to the computation in that an MD5 is only evaluated for the first 4 MB with that flag set to true for performance purposes (see https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/baseblobservice.py#L1923https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fblob%2Fmaster%2Fazure-storage-blob%2Fazure%2Fstorage%2Fblob%2Fbaseblobservice.py%23L1923&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579426298&sdata=YBreKhiI43sdsMDxS%2FsU7xs1PkpXW3ec3zJbgeNF5s8%3D&reserved=0
), but the particular blob that we’re encountering hash mismatch errors with in Databricks mounted storage is less than 1 MB in size so this shouldn’t be a factor. I can’t comment with 100% certainty on how Databricks computes the MD5 with their mounted storage setup, but from the logs I’ve seen it looks like they’re using the Java SDK for Azure Storage under the hood.
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Purna Chandra Rao Chinta <pchinta<EMAIL_ADDRESS>Date: Wednesday, August 14, 2019 at 6:00 PM
To: Azure/azure-storage-python <reply<EMAIL_ADDRESS>Azure/azure-storage-python <azure-storage-python<EMAIL_ADDRESS>Blaize Berry <Blaize.Berry<EMAIL_ADDRESS>Cc: Mention <mention<EMAIL_ADDRESS>Microsoft - Alex Kaps <Alexander.Kaps<EMAIL_ADDRESS>Subject: EXT: RE: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)

Blaize

Hi,
The blob is stored on a storage account owned by Blaize (copied on the thread). Blob size is 4MB and it is outlook mail. Since its PII info we do not have copy of the blob, and so did not validate MD5 of the blob. Attached is the error reported for the blob in scope of the issue by databricks file system API for the same file.
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
@Blaize Berry Blaize.Berry<EMAIL_ADDRESS>Could you please help us understand on MD5 computation to setup repro by Azure Storage SDK team?
Best Regards,
Purna Chandra Rao Chinta
Support Escalation Engineer
Microsoft Azure Rapid Response Team
Email : pchinta<EMAIL_ADDRESS>Working hours: 0900 – 2000 Central Time (Wed – Sat)
If you need immediate attention for any of the ongoing case while I am out of shift, please send a separate email with case ID to arrbackup<EMAIL_ADDRESS>A Duty Manager will take action and loop in an available ARR engineer to help you.  If you have any feedback of your Support Experience, please feel free to contact my manager – Kirk Beller at kbeller<EMAIL_ADDRESS>or +1<PHONE_NUMBER>.
[Microsoft Logo]
From: Ze Qian Zhang <notifications<EMAIL_ADDRESS>Sent: Wednesday, August 14, 2019 5:46 PM
To: Azure/azure-storage-python <azure-storage-python<EMAIL_ADDRESS>Cc: Purna Chandra Rao Chinta <pchinta<EMAIL_ADDRESS>Mention <mention<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi @pchintahttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpchinta&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579426298&sdata=aycTaj60TMgqx4QlUZy7syClXoqsD4IN0s1uimXco%2FI%3D&reserved=0, thanks for reaching out.
Could you please provide a bit more details so that we can try to repro this issue? How big was the file? Have you validated the stored MD5 on your own?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fissues%2F627%3Femail_source%3Dnotifications%26email_token%3DAM3JFIHLMJHRZFNUS2XWYWTQESDLZA5CNFSM4ILZHVI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KKTTA%23issuecomment-521447884&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579436293&sdata=%2BJO12gj6nurUQFeXeN%2B1l4oxjv%2Ftxh2a4JVTGBc4N9M%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM3JFIEJ5QEJDVLATFWGRS3QESDLZANCNFSM4ILZHVIQ&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579446294&sdata=gvvVdf%2B8NlRhoEYktjq%2F%2FKuxec5u%2BAho81CPZvObYUE%3D&reserved=0.

Also, when I try to retrieve the blob with the Databricks fs utilities I get the error below. Please note the checksum values that I’ve bolded in the stack trace. As you said, it appears that the Databricks FSUtils is computing the checksum incorrectly:
ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.head.
: java.io.IOException
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:737)
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:855)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at com.databricks.backend.daemon.dbutils.FSUtils$$anonfun$head$1.apply(DBUtilsCore.scala:200)
at com.databricks.backend.daemon.dbutils.FSUtils$$anonfun$head$1.apply(DBUtilsCore.scala:190)
at com.databricks.backend.daemon.dbutils.FSUtils$.com$databricks$backend$daemon$dbutils$FSUtils$$withFsSafetyCheck(DBUtilsCore.scala:81)
at com.databricks.backend.daemon.dbutils.FSUtils$.head(DBUtilsCore.scala:190)
at com.databricks.backend.daemon.dbutils.FSUtils.head(DBUtilsCore.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:295)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
... 27 more
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Blaize Berry<EMAIL_ADDRESS>Date: Thursday, August 15, 2019 at 9:35 AM
To: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Microsoft - Alex Kaps<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi Purna,
I have computed the MD5 of the blob in the storage account and got c1b80a33c85d2b1cb092d92fc817324d, which is consistent with the ContentMD5 blob property.
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Date: Thursday, August 15, 2019 at 7:25 AM
To: Blaize Berry<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Microsoft - Alex Kaps<EMAIL_ADDRESS>Subject: EXT: RE: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi,
We can compute MD5 of the blob in storage account and see if MD5 being updated on blob is correct..
https://github.com/giventocode/azure-blob-md5
For windows:

Install GO (as on link).
Create a folder for md5 and open cmdprompt (as admin), browse to the folder and run below commands:

go get github.com/giventocode/azure-blob-md5
go build -o bmd5.exe github.com/giventocode/azure-blob-md5

Create environment variables as ACCOUNT_NAME and ACCOUNT_KEY and provide values inline to storage account in scope of issue.
Open another prompt (as admin) as environment variables created above will not be available at cmd prompt at step 2 above.
Browse to “bmd5.exe” (at path on step 2) and run below command:

bmd5 -b blob -c container

Validate the MD5 generated with

MD5 reported on error (one of them)

MD5 on blob (content-md5 on azure portal).
If both match, then there is issue with md5 computed by databricks file system API.

Best Regards,
Purna Chandra Rao Chinta
From: Blaize Berry<EMAIL_ADDRESS>Sent: Thursday, August 15, 2019 12:32 AM
To: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Alex Kaps<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
We are not explicitly computing the MD5 of the blob. I believe that functionality is provided out of the box via the Python SDK when the validate_content flag is set to True. I believe that there is some subtlety to the computation in that an MD5 is only evaluated for the first 4 MB with that flag set to true for performance purposes (see https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/baseblobservice.py#L1923https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fblob%2Fmaster%2Fazure-storage-blob%2Fazure%2Fstorage%2Fblob%2Fbaseblobservice.py%23L1923&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579426298&sdata=YBreKhiI43sdsMDxS%2FsU7xs1PkpXW3ec3zJbgeNF5s8%3D&reserved=0
), but the particular blob that we’re encountering hash mismatch errors with in Databricks mounted storage is less than 1 MB in size so this shouldn’t be a factor. I can’t comment with 100% certainty on how Databricks computes the MD5 with their mounted storage setup, but from the logs I’ve seen it looks like they’re using the Java SDK for Azure Storage under the hood.
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Purna Chandra Rao Chinta <pchinta<EMAIL_ADDRESS>Date: Wednesday, August 14, 2019 at 6:00 PM
To: Azure/azure-storage-python <reply<EMAIL_ADDRESS>Azure/azure-storage-python <azure-storage-python<EMAIL_ADDRESS>Blaize Berry <Blaize.Berry<EMAIL_ADDRESS>Cc: Mention <mention<EMAIL_ADDRESS>Microsoft - Alex Kaps <Alexander.Kaps<EMAIL_ADDRESS>Subject: EXT: RE: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)

Blaize

Hi,
The blob is stored on a storage account owned by Blaize (copied on the thread). Blob size is 4MB and it is outlook mail. Since its PII info we do not have copy of the blob, and so did not validate MD5 of the blob. Attached is the error reported for the blob in scope of the issue by databricks file system API for the same file.
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
@Blaize Berry Blaize.Berry<EMAIL_ADDRESS>Could you please help us understand on MD5 computation to setup repro by Azure Storage SDK team?
Best Regards,
Purna Chandra Rao Chinta
Support Escalation Engineer
Microsoft Azure Rapid Response Team
Email : pchinta<EMAIL_ADDRESS>Working hours: 0900 – 2000 Central Time (Wed – Sat)
If you need immediate attention for any of the ongoing case while I am out of shift, please send a separate email with case ID to arrbackup<EMAIL_ADDRESS>A Duty Manager will take action and loop in an available ARR engineer to help you.  If you have any feedback of your Support Experience, please feel free to contact my manager – Kirk Beller at kbeller<EMAIL_ADDRESS>or +1<PHONE_NUMBER>.
[Microsoft Logo]
From: Ze Qian Zhang <notifications<EMAIL_ADDRESS>Sent: Wednesday, August 14, 2019 5:46 PM
To: Azure/azure-storage-python <azure-storage-python<EMAIL_ADDRESS>Cc: Purna Chandra Rao Chinta <pchinta<EMAIL_ADDRESS>Mention <mention<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi @pchintahttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpchinta&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579426298&sdata=aycTaj60TMgqx4QlUZy7syClXoqsD4IN0s1uimXco%2FI%3D&reserved=0, thanks for reaching out.
Could you please provide a bit more details so that we can try to repro this issue? How big was the file? Have you validated the stored MD5 on your own?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fissues%2F627%3Femail_source%3Dnotifications%26email_token%3DAM3JFIHLMJHRZFNUS2XWYWTQESDLZA5CNFSM4ILZHVI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KKTTA%23issuecomment-521447884&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579436293&sdata=%2BJO12gj6nurUQFeXeN%2B1l4oxjv%2Ftxh2a4JVTGBc4N9M%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM3JFIEJ5QEJDVLATFWGRS3QESDLZANCNFSM4ILZHVIQ&data=02|01|pchinta%40microsoft.com|fdd36421135f4572c73208d72141fa20|72f988bf86f141af91ab2d7cd011db47|1|0|637014439579446294&sdata=gvvVdf%2B8NlRhoEYktjq%2F%2FKuxec5u%2BAho81CPZvObYUE%3D&reserved=0.

Hi Blaize,
As we understood from the call, content-MD5 is being computed manually by your application and is the reason we were seeing different MD5 value.
As agreed we will close this thread. Thank you for your help.
Best Regards,
Purna Chandra Rao Chinta
From: Blaize Berry<EMAIL_ADDRESS>Sent: Thursday, August 15, 2019 9:44 AM
To: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Alex Kaps<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Also, when I try to retrieve the blob with the Databricks fs utilities I get the error below. Please note the checksum values that I’ve bolded in the stack trace. As you said, it appears that the Databricks FSUtils is computing the checksum incorrectly:
ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.head.
: java.io.IOException
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:737)
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:855)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at com.databricks.backend.daemon.dbutils.FSUtils$$anonfun$head$1.apply(DBUtilsCore.scala:200)
at com.databricks.backend.daemon.dbutils.FSUtils$$anonfun$head$1.apply(DBUtilsCore.scala:190)
at com.databricks.backend.daemon.dbutils.FSUtils$.com$databricks$backend$daemon$dbutils$FSUtils$$withFsSafetyCheck(DBUtilsCore.scala:81)
at com.databricks.backend.daemon.dbutils.FSUtils$.head(DBUtilsCore.scala:190)
at com.databricks.backend.daemon.dbutils.FSUtils.head(DBUtilsCore.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:295)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
... 27 more
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Blaize Berry<EMAIL_ADDRESS>Date: Thursday, August 15, 2019 at 9:35 AM
To: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Microsoft - Alex Kaps<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi Purna,
I have computed the MD5 of the blob in the storage account and got c1b80a33c85d2b1cb092d92fc817324d, which is consistent with the ContentMD5 blob property.
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Date: Thursday, August 15, 2019 at 7:25 AM
To: Blaize Berry<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Microsoft - Alex Kaps<EMAIL_ADDRESS>Subject: EXT: RE: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi,
We can compute MD5 of the blob in storage account and see if MD5 being updated on blob is correct..
https://github.com/giventocode/azure-blob-md5https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgiventocode%2Fazure-blob-md5&data=02|01|pchinta%40microsoft.com|25dcf8ca614347b6739008d7218f0a97|72f988bf86f141af91ab2d7cd011db47|1|0|637014770570283669&sdata=1kKyzLlU3a5FXf9XDYc0yYQRSg800RpLVGvnQuoUCjM%3D&reserved=0
For windows:

Install GO (as on link).
Create a folder for md5 and open cmdprompt (as admin), browse to the folder and run below commands:

go get github.com/giventocode/azure-blob-md5
go build -o bmd5.exe github.com/giventocode/azure-blob-md5

Create environment variables as ACCOUNT_NAME and ACCOUNT_KEY and provide values inline to storage account in scope of issue.
Open another prompt (as admin) as environment variables created above will not be available at cmd prompt at step 2 above.
Browse to “bmd5.exe” (at path on step 2) and run below command:

bmd5 -b blob -c container

Validate the MD5 generated with

MD5 reported on error (one of them)

MD5 on blob (content-md5 on azure portal).
If both match, then there is issue with md5 computed by databricks file system API.

Best Regards,
Purna Chandra Rao Chinta
From: Blaize Berry<EMAIL_ADDRESS>Sent: Thursday, August 15, 2019 12:32 AM
To: Purna Chandra Rao Chinta<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Azure/azure-storage-python<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Alex Kaps<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
We are not explicitly computing the MD5 of the blob. I believe that functionality is provided out of the box via the Python SDK when the validate_content flag is set to True. I believe that there is some subtlety to the computation in that an MD5 is only evaluated for the first 4 MB with that flag set to true for performance purposes (see https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/baseblobservice.py#L1923https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fblob%2Fmaster%2Fazure-storage-blob%2Fazure%2Fstorage%2Fblob%2Fbaseblobservice.py%23L1923&data=02|01|pchinta%40microsoft.com|25dcf8ca614347b6739008d7218f0a97|72f988bf86f141af91ab2d7cd011db47|1|0|637014770570293669&sdata=I1ALBsr%2FcOvZ2bc%2FNCiHlJWp8l8b%2FrOgZKlUjYuFh5E%3D&reserved=0
), but the particular blob that we’re encountering hash mismatch errors with in Databricks mounted storage is less than 1 MB in size so this shouldn’t be a factor. I can’t comment with 100% certainty on how Databricks computes the MD5 with their mounted storage setup, but from the logs I’ve seen it looks like they’re using the Java SDK for Azure Storage under the hood.
Best,
Blaize Berry
Staff Software Engineer (Machine Learning) – Austin, TX
M:<PHONE_NUMBER>
Blaize.Berry<EMAIL_ADDRESS>[/Users/b0b00ci/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_301523036]
From: Purna Chandra Rao Chinta <pchinta<EMAIL_ADDRESS>Date: Wednesday, August 14, 2019 at 6:00 PM
To: Azure/azure-storage-python <reply<EMAIL_ADDRESS>Azure/azure-storage-python <azure-storage-python<EMAIL_ADDRESS>Blaize Berry <Blaize.Berry<EMAIL_ADDRESS>Cc: Mention <mention<EMAIL_ADDRESS>Microsoft - Alex Kaps <Alexander.Kaps<EMAIL_ADDRESS>Subject: EXT: RE: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)

Blaize

Hi,
The blob is stored on a storage account owned by Blaize (copied on the thread). Blob size is 4MB and it is outlook mail. Since its PII info we do not have copy of the blob, and so did not validate MD5 of the blob. Attached is the error reported for the blob in scope of the issue by databricks file system API for the same file.
Caused by: com.microsoft.azure.storage.StorageException: Blob data corrupted (integrity check failed), Expected value is c1b80a33c85d2b1cb092d92fc817324d, retrieved wbgKM8hdKxywktkvyBcyTQ==
at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:466)
at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
@Blaize Berry Blaize.Berry<EMAIL_ADDRESS>Could you please help us understand on MD5 computation to setup repro by Azure Storage SDK team?
Best Regards,
Purna Chandra Rao Chinta
Support Escalation Engineer
Microsoft Azure Rapid Response Team
Email : pchinta<EMAIL_ADDRESS>Working hours: 0900 – 2000 Central Time (Wed – Sat)
If you need immediate attention for any of the ongoing case while I am out of shift, please send a separate email with case ID to arrbackup<EMAIL_ADDRESS>A Duty Manager will take action and loop in an available ARR engineer to help you.  If you have any feedback of your Support Experience, please feel free to contact my manager – Kirk Beller at kbeller<EMAIL_ADDRESS>or +1<PHONE_NUMBER>.
[Microsoft Logo]
From: Ze Qian Zhang <notifications<EMAIL_ADDRESS>Sent: Wednesday, August 14, 2019 5:46 PM
To: Azure/azure-storage-python <azure-storage-python<EMAIL_ADDRESS>Cc: Purna Chandra Rao Chinta <pchinta<EMAIL_ADDRESS>Mention <mention<EMAIL_ADDRESS>Subject: Re: [Azure/azure-storage-python] MD5 validation using Storage SDK is not happening while databricks filesystem API reports MD5 mismatch (#627)
Hi @pchintahttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpchinta&data=02|01|pchinta%40microsoft.com|25dcf8ca614347b6739008d7218f0a97|72f988bf86f141af91ab2d7cd011db47|1|0|637014770570293669&sdata=o3o4j5aw8zfBuGd8v%2F3DE5LZmy1JU3gOFk%2BH6IcbkF4%3D&reserved=0, thanks for reaching out.
Could you please provide a bit more details so that we can try to repro this issue? How big was the file? Have you validated the stored MD5 on your own?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-python%2Fissues%2F627%3Femail_source%3Dnotifications%26email_token%3DAM3JFIHLMJHRZFNUS2XWYWTQESDLZA5CNFSM4ILZHVI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KKTTA%23issuecomment-521447884&data=02|01|pchinta%40microsoft.com|25dcf8ca614347b6739008d7218f0a97|72f988bf86f141af91ab2d7cd011db47|1|0|637014770570303661&sdata=GFQmfmIDZCLwv5Kd5jFIBTFyI%2FgOIS94oMJYszj5OLk%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM3JFIEJ5QEJDVLATFWGRS3QESDLZANCNFSM4ILZHVIQ&data=02|01|pchinta%40microsoft.com|25dcf8ca614347b6739008d7218f0a97|72f988bf86f141af91ab2d7cd011db47|1|0|637014770570303661&sdata=0neopASTKK6ohL0p1e%2F%2BZmgIMWCHphrhyvzmdkhUFDA%3D&reserved=0.

@pchinta I see the issue was resolved.
To clarify, validate_content verifies the individual chunks when downloading a large blob.
On a side note, please avoid exposing cx's contact information on Github.


Fix an issue with ELFExternalSymbolResolver

The ELFExternalSymbolResolver is used to resolve symbols to external libraries.
This PR fixes an issue where it fails to resolve symbols that don't directly live in the <EXTERNAL> namespace, but are instead nested under a class namespace within <EXTERNAL> .
Tested:
Built Ghidra and ran the FixupELFExternalSymbolsScript.java script. It successfully moved symbols (and the class they were nested under) from the unknown  imports bucket to the correct external library.

Out of curiosity, and so I understand correctly, the symbols were nested like the following?
<EXTERNAL>::std::string...

Here's an example that I encountered, as shown in the symbol table window:

Running FixupELFExternalSymbolsScript with this PR gives us:

And I'm now able to click through to the function definitions in LibQt5Core.so

Is there a process to get the change merged after approval?

Is there a process to get the change merged after approval?

It still needs to be approved by a collaborator before it can be merged. I'm not sure what the internal review is like if minor changes even go through it. Things have a tendency to sit a while.

Ah I see, thanks for the clarification Andrew!

Any updates on this?

Would this potentially fix a Low-level Error: Unable to resolve type: QDebug error I'm seeing when trying to decompile some functions in an app (recognized as ARM:LE:32:v8) linked against libQt5Core.so? There doesn't seem to be workaround?

Would this potentially fix a Low-level Error: Unable to resolve type: QDebug error I'm seeing when trying to decompile some functions in an app (recognized as ARM:LE:32:v8) linked against libQt5Core.so? There doesn't seem to be workaround?

Nope. You just need to create a structure named QDebug and it should pick it up. You might need to move it to the Demangler category.

Fixed with dcc3043?

Fixed with dcc3043?

Trying a qt5 binary with external qt5 libs, I'm not able to cause any failure to move nested libraries.
GP-4145 didn't explicitly address any issues with nested namespaces under EXTERNAL, so I'm guessing it was fixed before GP-4145.
If someone is seeing issues where a namespace nested under EXTERNAL is causing a problem for the External Symbol Resolver analyzer, please include some detailed info about the state of the symbols that are failing to be fixed... ie. is it mangled in the main binary vs. de-mangled in the library, viceversa, etc.
I'm going to close this PR.


Provide convenience method for loading language library

(Please correct me if anything of the following is wrong)
If I understand it correctly, for all parser implementations there is always a tree_sitter_<lang> function, and it always has the same signature.
Currently jtreesitter only provides a Language(MemorySegment) constructor, so you have to generate boilerplate code which looks up the tree_sitter_<lang> function and invokes it (as done in the test code). This can be an obstacle for new users of jtreesitter because they either have to be a bit familiar with java.lang.foreign, or blindly copy code they don't understand.
It would be useful if Language provided a convenience method for this, for example:
public static Language loadLanguage(SymbolLookup parserLibrary, String languageName)

The user could then easily use SymbolLookup#libraryLookup to load the library and then use that Language#loadLanguage method.
If you want I can try to create a proof-of-concept PR for this.

The plan is to eventually integrate those bindings into the parsers (see tree-sitter/tree-sitter-java#182).

But that is specifically for tree-sitter-java, right? That would certainly be useful, but I was thinking of a more general solution for all parsers, e.g. Python, JSON, ... since they all have a tree_sitter_<lang> function with the same signature (?).

The CLI will generate bindings for all parsers like it does for other languages.

Ah, I think I misunderstood you. Is the plan to generate Java bindings for all parsers, and the tree-sitter-java one was just an example? That would be great then!
But would it make sense nonetheless to add a generic loadLanguage method here, for cases where a repository does not include a bindings/java/.../TreeSitter<lang>.java yet?
I was thinking of something like this:
public final class Language {
    /**
     * Loads a language using the given symbol lookup for the native library.
     * For example:
     * {@snippet lang=java :
     * Path pathToLibrary = Path.of("libtree-sitter-python.so");
     * SymbolLookup libraryLookup = SymbolLookup.libraryLookup(pathToLibrary, Arena.ofAuto());
     * Language language = Language.loadLanguage(libraryLookup, "python");
     * }
     *
     * @throws IllegalArgumentException If the Tree-sitter language function cannot be found using the symbol lookup
     */
    public static Language loadLanguage(SymbolLookup symbolLookup, String languageName) throws IllegalArgumentException {
        String functionName = "tree_sitter_" + languageName;
        MemorySegment functionAddress = symbolLookup.find(functionName)
            .orElseThrow(() -> new IllegalArgumentException("Language function '%s' not found".formatted(functionName)));

        var voidPtr = ValueLayout.ADDRESS.withTargetLayout(MemoryLayout.sequenceLayout(Long.MAX_VALUE, ValueLayout.JAVA_BYTE));
        var funcDesc = FunctionDescriptor.of(voidPtr);
        var function = Linker.nativeLinker().downcallHandle(functionAddress, funcDesc);
        MemorySegment languagePointer;
        try {
            languagePointer = ((MemorySegment) function.invokeExact()).asReadOnly();
        } catch (Throwable t) {
            throw new RuntimeException("Failed to call language function", t);
        }

        return new Language(languagePointer);
    }

    /**
     * Creates a new instance from the given language pointer.
     *
     * <p>Normally you don't have to obtain the language pointer yourself. Instead, you can either use the
     * generated Java bindings for a parser, for example:
     * {@snippet lang=java :
     * var pointer = TreeSitterPython.language();
     * Language language = new Language(pointer);
     * }
     * Or you can use {@link #loadLanguage(SymbolLookup, String)} to obtain a {@code Language} instance.
     *
     * @implNote It is up to the caller to ensure that the pointer is valid.
     *
     * @throws IllegalArgumentException If the language version is incompatible.
     */
    public Language(MemorySegment address) {
        // ...
    }

    // ...
}

But would it make sense nonetheless to add a generic loadLanguage method here

Only until the bindings are autogenerated, at which point it'll be deprecated.


Rename after to on_success

We decided to do this some time ago. The change is backwards-compatible, so should not break anything just yet.

@pyromaniac but how then we will migrate to on_success without deprecation notice? Are we going to support both names?

Consider it a soft deprecation :) We will just replace everything at once at a PR after the gem update. I was too lazy to add a deprecation :(


docs: Update README.md

remove unnecessary line from document

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 90.61%. Comparing base (531c4c0) to head (a3c7c24).
Report is 1 commits behind head on master.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3002   +/-   ##
=======================================
  Coverage   90.61%   90.61%
=======================================
  Files          74       74
  Lines        2962     2962
  Branches      469      469
=======================================
  Hits         2684     2684
  Misses        278      278

Flag
Coverage Δ

Linux
90.61% <ø> (ø)

Windows
?

macOS
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


⚠ ClamAV has degraded performance

In 13a1023, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 147 ms

Resolved: ClamAV performance has improved in 2161f78 after 5 minutes.


is saml auth supported?

?

not sure what you mean by this...
I have SAML based auth setup on our Jenkins instance - to to lint our Jenkinsfiles we just use a token to auth with the server


Profile switching does not work in development version 0.12

In the current dev version 0.12 profile changing does not work because the web interface uses wrong rest api commands. The complete host part with port is missing.

Thanks for the heads up, I think I've fixed this already in UI repo itself, but I haven't merged the UI into 0.12 recently (it's been undergoing quite a few changes). I'll get this sorted soon.

This should be fixed in eec09c4 - Can you give it a test for me please?

It works :)


Fixed Automatic Translation of variable names

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] All the files I changed are in the same world language (for example: only English changes, or only Chinese changes, etc.)
[x] My changes do not use shortened URLs or affiliate links.

Closes #XXXXX

Thank you for opening this pull request. This is a standard message notifying you that we have reviewed your pull request and have decided not to merge it. We would welcome future pull requests from you. Thank you and happy coding.


Changed linter to ruff.

Signed-off-by: Pavel Kirilin<EMAIL_ADDRESS>

Codecov Report

Merging #210 (62c8ae1) into develop (e66f3aa) will increase coverage by 5.29%.
Report is 125 commits behind head on develop.
The diff coverage is 72.49%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@             Coverage Diff             @@
##           develop     #210      +/-   ##
===========================================
+ Coverage    67.62%   72.91%   +5.29%
===========================================
  Files           37       48      +11
  Lines          942     1514     +572
===========================================
+ Hits           637     1104     +467
- Misses         305      410     +105

Files
Coverage Δ

taskiq/__init__.py
100.00% <100.00%> (ø)

taskiq/abc/formatter.py
100.00% <100.00%> (ø)

taskiq/abc/middleware.py
100.00% <ø> (ø)

taskiq/abc/result_backend.py
100.00% <ø> (ø)

taskiq/abc/schedule_source.py
100.00% <100.00%> (ø)

taskiq/acks.py
100.00% <100.00%> (ø)

taskiq/api/__init__.py
100.00% <100.00%> (ø)

taskiq/cli/common_args.py
100.00% <100.00%> (ø)

taskiq/cli/watcher.py
0.00% <ø> (ø)

taskiq/cli/worker/log_collector.py
100.00% <ø> (ø)

... and 34 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Fixes Page isGeneratedCollection function

I'm not 100% sure on this one, but I know that the function is definitely broken as there is no $this->vObj->ptID.
Working on getting stuff working with E_NOTICE turned on so prepare for the barrage of PRs....

Thanks


Defraggler with df.exe

Based on this article defraggler should be run using df.exe, not defraggler.exe. I can make a PR to correct this, but I wanted to open it up to discussion in case anyone else has more experience with this tool and maybe I'm missing something.
I have tested Tron's syntax and all it does is open the Defraggler interface, but replacing defraggler.exe with df.exe and including the same command line switches seems to work just fine.

I think (I'll double-check) the bundled defraggler.exe is actually just a renamed df.exe, to make it clear to users what the .exe is for.

If you're using the portable zip download from the Piriform site this would include a separate df.exe (and df64.exe). This also kind of brings up the point that it might be a good idea to have Tron detect 64-bit architecture and use the appropriate tools when available.

I'm not sure a 64-bit Defraggler binary would make any real-world difference. I believe I purposefully left it out since it doesn't grant any performance increase. Generally I try to only include 64-bit binaries if there might be some functional reason

Verified defraggler.exe is indeed a renamed df.exe. Closing.


Can't scroll in settings page "pop-up" menus

Hi, I for some reason the last couple releases can't scroll under the various pop-up setting pages. For example if I go to Settings -> Providers -> Number of hours in hourly forecast, it opens with 72 selected and the top one shows 66, but I can't swipe up higher to select 60 for some reason. The same thing occurs on any one of these "pop-up" settings pages. Days in forecast, Number of minutes in minutely forecast, Open-Meteo preferred model, etc..
I'm on Android 15 with GrapheneOS, not sure if anyone else is seeing the same thing.
On these types of menus:

Scrolling is fixed in release 144, works in all the "pop over" menus now. Thanks :)


[BUG]:  Header padding

Description
Padding in header causing name to collapse
Reproduction Steps
Visible once user is logged in
Expected Behaviour

Actual Behaviour

Operating System / Platform
Windows
(optional) What browsers are you seeing the problem on?
No response
Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Fix with (https://github.com/webdevcody/code-racer/pull/88)


Subsampling tooltip

Fixes issue #37.
This PR adds a standard help icon to the top of the layout of the cross- and auto-correlograms. When it is clicked, a modal popup is displayed which describes subsampling strategy.
Text for popup should be generated from CorrelogramSubsamplingInfo.md in the src/python/sortingview/gui/extensions/correlograms/ directory (along with the rest of the correlograms components).
Also changed, the Cross-Correlograms view had been using a cut-and-pasted version of the useLocalUnitIds hook; I replaced this with a reference to the library version.

Writeup looks perfect to me.


JupyterLab 4 PyPi release

I saw the pull request got merged a while back, is the extension ready for a pypi release? I have not been able to get a local install to complete.

It looks like the releaser is setup, I will try to carry out a 4.0 release.

I did not release the package yet because there is an error with the icon next to the breadcrumb that is badly positioned and not accessible for click; see skipped integration test: https://github.com/jupyterlab-contrib/jupyterlab-favorites/blob/a07b08c2cadeccdac6947fa0416afdb0c82c4927/ui-tests/tests/jupyterlab_favorites.spec.ts#L48

How badly does this affect functionality? It might be worth releasing a version with that bug if folder names still appear at the top and are clickable.

Version 3.2.0 has been released with support for Lab 4: https://pypi.org/project/jupyterlab-favorites/

Huge thanks @fcollonval, such an improvement to my workflow (and sanity) to have this back!


support auth

Signed-off-by: Michael Kalantar<EMAIL_ADDRESS>Fixes https://github.com/iter8-tools/iter8/issues/1100

I've verified that I can access IKS with a rebuilt version of iter8.
$ iter8 k getAPP     ID    COMPLETED FAILED TASKS TASKS_COMPLETED
default ingsl true      false  7     7


Training on custom data

How do I train on custom data? can you please share the script to create npy files?

Hi, plz look up that code!

Create dataset using augmentation function
https://github.com/junhwanjang/face_landmark_dnn/blob/master/augmentation/test_create_dataset.py

Train the model using train code
https://github.com/junhwanjang/face_landmark_dnn/blob/master/modeling/train_mobilenets.py

@junhwanjang
how to use test_augment.py? i get error that no txt file, i tried even pts file. my input dataset has images and pts files.
https://github.com/junhwanjang/face_landmark_dnn/blob/master/augmentation/test_augment.py


Set ms.author to Rainer Sigwald

@v-alje
Reassign to the dev lead, who agreed to be the point of contact, given that the docs come from code comments.


Biorand does not have permission

BioRand version 3.0.1
Seed R720-JW0Y-XQHHH0ZL31000009K49KZZZZZZZZZZZZZZZ for RE2
No matter if I run it as administrator or move the folder to somewhere else like the desktop, I always have denied permission...

Folder was write only, issued fixed, sorry.


Move "slack" package from openwhisk to openwhisk-catalog

@houshengbo
Uploading OpenWhisk-Repo-Proposal-2016-06-14 - Open Catalog only.pptx…
Issue #4 is a prereq.
Move sample "slack" in current repo./directory:

/openwhisk-openwhisk/catalog/slack

to the "openwhisk-catalog" repo. under directory structure:

/openwhisk-catalog/packages/slack

including action files

post.js

Update install script (in /openwhisk-catalog/packages directory, after Issue #4 is completed) :

installSlack.sh

Refer to openwhisk: https://github.com/openwhisk/openwhisk/issues/666
Refer to blue: https://github.ibm.com/BlueMix-Fabric/bluewhisk/issues/2040


TrackBy Demo plunker

I'm submitting a bug report...
I was going through the ngFor directive documentation The Live Demo shown over this link does not provide enough illustration.
Expected behavior : Demo is expected to have all examples for the directive

Angular version: 2.0.X

Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

Language: [all | TypeScript X.X | ES6/7 | ES5]

The live demos of the API are going to be revamped soon. On the other hand, API issues goes to angular/angular so please, open an issue there so they know about this :)


Show a full-width graph in visualize

A large application topology causes the nodes that represent some of the components to end up outside the image canvas.
This edit shows a full-width graph in the Visualize tab. This will put the table after the graph instead of aside, allowing to display a larger and more relaxed topology to improve the readability.
Note that this feature doesn't fix the bug: a very large topology will still causes the nodes to end up outside the canvas, even at full with. A possible fix could be to take the size passed to the createGraph function as max width and drow nodes inside that space. Then, by setting the svg width equal to the width used by the elements that it contains, you can obtain a smaller canvas with the table aside when possible, and a full-width canvas otherwise.

Could you please commit the corresponding update also to the master branch?

Could you please commit the corresponding update also to the master branch?


Fix: evaluating absolute nodeset expressions with an attribute context node

Fixes part of enketo/enketo-core#917, for absolute nodesets. There is also one test confirming that the evaluator properly handles relative nodesets, but recomputation of those expressions will need to be fixed in enketo-core.

I'll let this sit for a day to see if anyone else thinks of any risk and if not we can squash and merge.
@jkuester you may find the ability to bind to attributes interesting.

Is it worth a comment somewhere saying that this is to address a bug in Chrome/Safari and matches the behavior Firefox already has? Or do we consider the Github history sufficient?

Is it worth a comment somewhere saying that this is to address a bug in Chrome/Safari and matches the behavior Firefox already has? Or do we consider the Github history sufficient?

I think a comment would be appreciated.


Pressing up on a field errors

Uncaught Error: This should not happen

It looks like this is a bug introduced when removing jQuery.


Properly set size of shadow atlas quadrant when subdivision is 8 or higher.

Fixes: https://github.com/godotengine/godot/issues/89312
I still can't figure out why this seemingly regressed between dev1 and dev2. Both of these problems have been present for a long time.
Problem 1: Shadow atlas quadrant allocation
Each atlas stores shadows for up to subdivision * subdivision lights (omnilights count for 2). When we set the quadrant subdivisions we reserve enough space in the shadows array for that many lights. We use the length of this shadows array to validate how many shadows can go in each quadrant.
When we set the size of the atlas, we have to clear the quadrants and then resize them. But there we were reserving enough space for 1 << subdivision lights. This is not always the same as subdivision * subdivision! For the last quadrant, this led the system to think that it could store 256 shadows, when it really only had space for 64. So the 65th shadow would try to render outside the framebuffer and cause the error.
Oddly enough I caught this bug in the OpenGL renderer when implementing shadows. I remember at the time noticing it, but not doing anything in the RD renderers because it wasn't causing any issues and I didn't have time to investigate why.
Problem 2: No validation for render pass rect size
draw_list_begin is supposed to validate the size of the render pass rect. But it was missing a pair of parentheses, so instead of checking if the rect was fully inside the framebuffer. It checked if the rect began outside the framebuffer on the X-axis, but otherwise fit within the framebuffer. This bug has been present for a long time.
Not every device crashed/froze despite us allowing values that should not have been allowed as different drivers handle breaking spec differently.

If those two issues pre-exist the apparent regression, should we consider cherry-picking this to 4.2?

Thanks!

If those two issues pre-exist the apparent regression, should we consider cherry-picking this to 4.2?

I need to make a separate PR as this one won't cherrypick cleanly.


Romove lag when closing editors from the "Open Editors" panel.

In the "Open Editors" panel there is a noticeable lag when you click to close editor. It is not a serious issue but in my opinion it is off putting and I think it should be improved.

What I explained is more of a feature request, not a bug. When you press the X in the Open Editors there is a lag when closing the file.

Is it only slow from Open Editors, and not elsewhere?

It is not slow per se, more like it is not smooth.

I don't see this. Please include a screen recording

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


Bump Ansible Galaxy metadata

Ansible Galaxy metadata should be updated to reflect the role's support for new releases of Fedora and Debian.

Resolved by !8

Resolved by #8


custom query string variable?

wondering if there is an option to replace the rel= with a string of one's choosing, via the options.
thanks

I would like to have that option also. Is it possible?


Add release workflow on GitHub actions

This workflow automates the creation of a new release on GitHub. The workflow is triggered when a new tag is created. Then, it builds a new version of the plugin and generates a new release uploading the built.

Looks good to me and works fine.
The only minor comment is that the GitHub release title and body is automatically generated with the title and body of the last commit.

Looks good to me and works fine.
The only minor comment is that the GitHub release title and body is automatically generated with the title and body of the last commit.

Yes. I know. My thought was that this workflow will run when a new tag is created so that tag will point at a commit that says something meaningful like Release 0.1.0 or similar. I think for now this will be enough because this plugin my a temporary solution as there are some parts that can be submitted to OpenSearch project.

LGTM

I'm opening a new PR in favor of this. The new PR is more generic and it's been already tested in other plugins such as kbn_radar.


Source of file path originating an alert

For each alert that is generated, there is a full path to the file the alert originates from. When I access the Code Scanning REST API, that information is not in there.
Where does it come from and is it accessible by users ?

We're extracting the information about file locations associated with the alert from the results of the CodeQL analysis.
For the moment, this information is not included in the API responses, but we are planning on introducing it very soon.


[DOCS 📄] Compact hacktoberfest banner for README.md

Description

Right now the current hacktoberfest logo banner is very large and consuming a lot of space.
Suggest a new compact logo that is similar to this original one in this README.md.

Screenshots
No response
Additional information
No response

Hey I wwould like to work on this issue can this issue be assigned to me.
Thanks in advance for assigning me 😁

Will this image be fine in the README ?

Do you have any other images to choice from? One which only has the hacktoberfest written with the logo and 000010 below, like a Badge basically. Without the background.
I'm sure I saw something similar in another repository

Do you have any other images to choice from? One which only has the hacktoberfest written with the logo and 000010 below, like a Badge basically. Without the background.
I'm sure I saw something similar in another repository

Now I got your point,
Like just a header logo right?
Would a smaller logo be fine ?

You can show your Suggession

Here's what you're expecting for :

Yes this

Add this one

Is this issue not accepted in Hacktoberfest ?

Accepted


Debug info on contract errors

This PR adds some debug info output in case error is encountered during contract execution:
running 1 test
panicked at 'Simulating stack height limit', smart_contracts/contracts/test/do-panic/src/main.rs:9:5
Wasm execution error: TrapCode::Unreachable
current instance stack height = 19
test test::panic::panic_in_contract_should_yield_trap_unreachable ... ok

Closes #4002
Closes #4003
Things to be considered
Change introduced in this commit causes some performance tests to fail:

host_function_metrics_has_acceptable_gas_cost -> used<PHONE_NUMBER>00 gas; it should use no more than<PHONE_NUMBER>71 gas
host_function_metrics_has_acceptable_size -> host-function-metrics became 116188 bytes long; up to 102447 bytes long would be acceptable.
faucet_costs -> EXPECTED_FAUCET_INSTALL_COST is<PHONE_NUMBER>0 instead of<PHONE_NUMBER>0

Theoretically, the panic print happens only if test-support feature is enabled, so it shouldn't affect any contracts in production - only these under development will be affected. Therefore, we may consider disabling this feature in the above performance tests.
However, we need to be sure that this change will not increase the gas cost in uncontrolled way. If there is a risk, we should remove the print and look for other solution, possibly as a part of ticket #4104.

I would think this could target dev?

After discussion with @EdHastingsCasperLabs, the following approach is proposed:

decouple the performance regression check from tests (long term)
-- currently, our tests may fail not because the system is broken, but because the performance changed; these tests will even fail on performance improvement
-- our target is for the tests to fail if we broke EE, have some missing FFI, etc. and for the performance to be monitored on CI (there should be a gas cost/file size baseline, and CI should inform us when calculated cost/size of the PR varies too much)
remove the need to use print (especially in panic handler) by improving the error reporting (long term)
-- having the conditionally compiled print might be confusing (for example, people may try to chase an error in production, they will enable test-support, which will result in an error in different place, because of wasm structure change)
-- we shall be able to provide more detailed reason of failure (aka, more fine grained error than just Trap::Unreachable) so that users can reason about what happened
split the host-function-metric contract in two (orthogonal)
-- one part should just call different FFIs (as in the top of the file)
-- second part should exercise the extern with long names

Temporary solution:

make the tests passing by increasing the allowed limits
remove the "performance improved" asserts
ship (and improve later, as described above)

@RitaMAllenCA
I'll update the PR, before merging we should maybe reconsider if we want to ship the temporary solution, because once given it might get adopted by developers and there could be a pushback if we are about to remove it sometime in the future.

Closed in favor of https://github.com/casper-network/casper-node/pull/4229


Use 'fabric-api' mod ID in fabric.mod.json

Users may be confused when they see that it depends on fabric, not knowing that it's referring to the Fabric API.

As of Minecraft 1.19.2, Fabric API's mod ID has changed from fabric to fabric-api. When backporting from 1.19.2 to earlier versions, you must alter the depends section of your fabric.mod.json to expect fabric.

From https://fabricmc.net/wiki/tutorial:setup#mod_setup

You should also add the Fabric API dependency for releases on CF and MR

Fixed with commit https://github.com/Theoness1/EatingAnimation/commit/fdd45081f89bf591ef7a4f88278424099128595a. Thanks!


Can't update my SysNand

Hi, I create my own issues instead of posting in another one.
I'm currently with a N3DS with Sys 9.0.0-20E written and i'm stuck at Section IV of the guide. I can't update my system. I tried to restore my Nand that I got from the first part of the tutorial, but nothing. Can't update, the N3DS just reboot after saying that it didn't work (the internet connexion works)
What am I doing wrong ?
Thank you :)

duplicate of https://github.com/Plailect/Guide/issues/716


Change all Event routes/urls

Currently, when you are on the View, Edit, or track_volunteers the url is eventsList/<eventID>/______. We should change the first part of this from eventsList/ to event/.

 Change all Event routes/urls #942

 Change all Event routes/urls #942


Refactor tests

Sorry that it turned out so big and really hard to review now. Was not intended :D In case of uncertainty, let me know, we can also go over together.

Switch from testing.postgresql to pytest-postgresql
Use pytest fixtures for creating the database and getting the AsyncClient (http client)
Use async tests (necessary now)
Place all tests in app/tests
utils/models.py provides some reusable functions to create data in the database
Testdata is placed in tests/testdata and contains functions to get the data and to validate
Contains some fix commits. Not clean, but I stumbled across them while writing tests, and it is easier to justify them when directly covered by tests and working
GitHub actions already uses the new tests in this PR

Currently covered from tests (Sonar Cloud code coverage integration planned as a follow-up):

All CRUD operations
Most API endpoints (only the multi get currently missing), even codegen and project upload
CodeGeneration
PandasParser with Excel, CSV and JSON

conftest.py contains the fixtures for Pytest, which are injected in the parameters of the test functions. The scope of the database is currently per test, so each test gets a whole new database. A cross-test database would be possible too, if necessary.
pytestmark = pytest.mark.asyncio at the top of some files marks all tests as @pytest.mark.asyncio (async tests) automatically.

I addressed some of your comments.

API: Delete tests now checks, whether the object was really deleted in db
API: Dedicated tests for 404 when requesting not existing data. Moved this out of the delete test, as it was not correct here
API: Checks in create tests, if object is also created in db
CRUD: Check, whether objects are created in the database. Also compared to the returned object
Random data: Now uses the seed set as pytest parameter


is it safe?

I'v read in the documents that i should i inject a js script in my project.
does this wrapper inject it directly as this could cause safety concerns like cross-site injection...etc.

No , its linked to Pakistan

Fly, you fools


Blockquote's are not shown

Hi,
I'm just starting with Hugo and the Ananke theme. Thanks first of all for the efforts put into the creation of the theme!
I noticed that using blockquote (> in markdown), results in HTML code . The quote is not shown in browsers.
A similar issue suggests that the html tags should result in

mm...also new at commenting at github :) another try:
I notice that when using blockquote in markdown, the resulting html is <blockquote> </blockquote>
While a similar issue suggests it should be <blockquote class="blockquote">

Thanks, @Money-Samurai Some of the issues may not be related to blockquote styling, but I did add some styling for them.

Thanks budparr! the update works, i cloned the latest version and quotes are nicely shown now in the built site

(with a line to the left of the quote)


FFMpeg error?

Trying to run manual.py, I keep getting unexplained errors. I've updated the script, ffmpeg, everything that needs to be installed according to the instructions, still get an undescribed error 1. Even went through and threw UTF-8s all over the place, still can't figure out what is wrong.
Error converting file, FFMPEG error.
Traceback (most recent call last):
File "/scripts/mkvtomp4.py", line 570, in convert
for timecode in conv:
File "/scripts/converter/init.py", line 243, in convert
timeout=timeout, preopts=preopts, postopts=postopts):
File "/scripts/converter/ffmpeg.py", line 553, in convert
total_output, pid=p.pid)
FFMpegConvertError: <FFMpegConvertError error="Exited with code 1", pid=45761, cmd="/usr/local/bin/ffmpeg -fix_sub_duration -i /Data/Videos/TV Shows/Whose Line Is It Anyway/Season 11/Whose Line Is It Anyway! - s11e13 - Gina Rodriguez.mkv -vcodec copy -map 0:0 -c:a:0 copy -map 0:1 -metadata:s:a:0 language=und -f mp4 -threads auto -y /Data/Videos/TV Shows/Whose Line Is It Anyway/Season 11/Whose Line Is It Anyway! - s11e13 - Gina Rodriguez.mp4">

I still get that error. FFMPEG still says the same thing. Reinstalled ffmpeg, not sure what else to do.
Also, is there a better way to update other than redownloading the zip and overwriting everything and resetting it all up?
[mp4 @ 0x808c51a20] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x808c51a20] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to '/Data/Videos/TV Shows/Whose Line Is It Anyway/Season 11/Whose Line Is It Anyway! - s11e13 - Gina Rodriguez.mp4':
Metadata:
encoder         : Lavf56.40.101
Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 29.97 tbr, 16k tbn, 1k tbc (default)
Stream #0:1(und): Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo (default)
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x808c51a20] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @ 0x808c51a20] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
frame=    2 fps=0.0 q=-1.0 Lsize=       5kB time=00:00:00.04 bitrate= 843.2kbits/s
video:4kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 17.329819%
Conversion failed!

I still get that error. FFMPEG still says the same thing. Reinstalled ffmpeg, not sure what else to do.
Also, is there a better way to update other than redownloading the zip and overwriting everything and resetting it all up?
[mp4 @ 0x808c51a20] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x808c51a20] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to '/Data/Videos/TV Shows/Whose Line Is It Anyway/Season 11/Whose Line Is It Anyway! - s11e13 - Gina Rodriguez.mp4':
Metadata:
encoder         : Lavf56.40.101
Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 29.97 tbr, 16k tbn, 1k tbc (default)
Stream #0:1(und): Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo (default)
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x808c51a20] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @ 0x808c51a20] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
frame=    2 fps=0.0 q=-1.0 Lsize=       5kB time=00:00:00.04 bitrate= 843.2kbits/s
video:4kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 17.329819%
Conversion failed!

Can you upload the offending file somewhere and email me the link?
Sent from my iPhone

On Dec 31, 2015, at 20:55, LandMineHare<EMAIL_ADDRESS>wrote:
I still get that error. FFMPEG still says the same thing. Reinstalled ffmpeg, not sure what else to do.
Also, is there a better way to update other than redownloading the zip and overwriting everything and resetting it all up?
[mp4 @ 0x808c51a20] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x808c51a20] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to '/Data/Videos/TV Shows/Whose Line Is It Anyway/Season 11/Whose Line Is It Anyway! - s11e13 - Gina Rodriguez.mp4':
Metadata:
encoder : Lavf56.40.101
Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 29.97 tbr, 16k tbn, 1k tbc (default)
Stream #0:1(und): Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo (default)
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x808c51a20] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @ 0x808c51a20] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
frame= 2 fps=0.0 q=-1.0 Lsize= 5kB time=00:00:00.04 bitrate= 843.2kbits/s
video:4kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 17.329819%
Conversion failed!
—
Reply to this email directly or view it on GitHub.

Can you upload the offending file somewhere and email me the link?
Sent from my iPhone

On Dec 31, 2015, at 20:55, LandMineHare<EMAIL_ADDRESS>wrote:
I still get that error. FFMPEG still says the same thing. Reinstalled ffmpeg, not sure what else to do.
Also, is there a better way to update other than redownloading the zip and overwriting everything and resetting it all up?
[mp4 @ 0x808c51a20] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x808c51a20] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to '/Data/Videos/TV Shows/Whose Line Is It Anyway/Season 11/Whose Line Is It Anyway! - s11e13 - Gina Rodriguez.mp4':
Metadata:
encoder : Lavf56.40.101
Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 29.97 tbr, 16k tbn, 1k tbc (default)
Stream #0:1(und): Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo (default)
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x808c51a20] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @ 0x808c51a20] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
frame= 2 fps=0.0 q=-1.0 Lsize= 5kB time=00:00:00.04 bitrate= 843.2kbits/s
video:4kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 17.329819%
Conversion failed!
—
Reply to this email directly or view it on GitHub.

So i have not had any problems since updating. sorry should have replied earlier but i wanted to let it run for a few days and test.
One problem i do have it that it does not UnRAR or Unzip files. I will open a separate ticket.

And I haven’t been able to get a personal/private email to MDH about where I could/should upload the offending file. Still working on figuring that part out.

On Jan 14, 2016, at 05:05, traxinv<EMAIL_ADDRESS>wrote:
So i have not had any problems since updating. sorry should have replied earlier but i wanted to let it run for a few days and test.
One problem i do have it that it does not UnRAR or Unzip files. I will open a separate ticket.
—
Reply to this email directly or view it on GitHub https://github.com/mdhiggins/sickbeard_mp4_automator/issues/387#issuecomment-171638753.