Hash does not match id during prune

Output of restic version
I typically use:
restic 0.8.3
compiled with go1.10 on linux/amd64

but I also tried with:
restic 0.9.2 compiled with go1.10.3 on linux/amd64
How did you run restic exactly?
restic -r <repo_path> forget --keep-last 5 --prune
What backend/server/service did you use to store the repository?
This may probably be very relevant: the source of the data is on a USB hard disk, which is mounted always in the same mount point. The repository is instead a SMB share mounted on a specific constant path via cifs. SMB share is stored on a ext4 filesystem.
I've seen this failure reported other times, but I created a new issue anyway to explain this particular situation.
Expected behavior
Prune should succeed.
Actual behavior
Backing up succeeds. Then, I run the above command, which should remove exactly one snapshot. Prune fails returning exit code 1. Running prune once again instead seems to work and report 5 available snapshots. The error reported by the first prune command is:
enter password for repository:
repository 2b913e7d opened successfully, password is correct
Applying Policy: keep the last 5 snapshots snapshots
snapshots for ([...]):

keep 5 snapshots:
ID        Date                 Host                 Tags        Directory
----------------------------------------------------------------------
[...]
----------------------------------------------------------------------
5 snapshots

remove 1 snapshots:
ID        Date                 Host                 Tags        Directory
----------------------------------------------------------------------
[...]
----------------------------------------------------------------------
1 snapshots

1 snapshots have been removed, running prune
counting files in repo
building new index for repo
[8:32] 100.00%  30451 / 30451 packs
incomplete pack file (will be removed):<PHONE_NUMBER>c07ffa260e5369a3a1d14c750a8e58ef4e1a88d69339717233874b
incomplete pack file (will be removed): 052ae43adbf9501b7acee6ff25ace7a5f57ebc254428c5c4d5e350390f0161e4
incomplete pack file (will be removed): 06ffe477f19fc9f0857897ce98170951400694d155f4efcc0d93cca29dcf0153
incomplete pack file (will be removed): 08af490a1f8debbca69808314b24bc61f45be5e12c084ecd304518b8875a2bde
incomplete pack file (will be removed): 098c7c1c13f5070c7265aeb16aa2f2e3c10fd53147114e2d5dc5434ea29dd9ba
incomplete pack file (will be removed): 0d0db3810b0be0fbc3582521c34b4cf5729418c23853e47f08021717a30967a7
incomplete pack file (will be removed): 11c7e8b95ed3eff46b2819bc3608a0bbe2afa658df1e5a5a94745f76cfba0304
incomplete pack file (will be removed): 11ef82a4fe5f62fc37ad432fb9ffc1611fa1fbc060c28b06160b8f7b0e05d095
incomplete pack file (will be removed): 1281ca112f53a1a1944b10cf258c3fabd36209b394f837a6afea5922c4c41697
incomplete pack file (will be removed): 1c5afa76417bcbc932499b000ab4fbc5c1fabf604d006883c454e6fdad9b3836
incomplete pack file (will be removed): 1c6606767eb76b658e554d8189a468a1db418df1dc448905428b2b087703932c
incomplete pack file (will be removed): 28a3b8b4c8d466fa4998d34412bee15ba6e05e4623a0fee0f39823411a1eca12
incomplete pack file (will be removed): 2bae1122447e8de3001327596b3af9983414307856d71da12db1ede6b871166f
incomplete pack file (will be removed): 3182fea28f3095ed9db5c3e41ff5cd34856c881f0f797004ff7ce3c66705525a
incomplete pack file (will be removed): 31f069689a117e800522e5568ff3df49883a85d9d1a29da3fb66c758a7caf10d
incomplete pack file (will be removed): 3279f70d1f19b400df47acc858c9967ba1845fc84f85e6a0e880ea19dea4c3d4
incomplete pack file (will be removed): 362358d67c70d287b35f149245f7d17faa274c86286779397b6c0a50d958d3e2
incomplete pack file (will be removed): 3aa916f478c9b53c59d2e9a4b23f81acf3dc2612c43612f14e2c6c7edee6f5d5
incomplete pack file (will be removed): 3ac13a9630f020b88a00d43211141126a0f83aa11bcd9cfa74acd2c30c5eb7d0
incomplete pack file (will be removed): 3f67a59f7fb7f86ebb3baea412fef3690dc3422d743430f411c2d89275d7436c
incomplete pack file (will be removed): 3fc8f32d65c08e9d94954bd25faa886eeffb782a53ac66f4e606509539fd5786
incomplete pack file (will be removed): 4db38e486f34070801bf13e71d468c8927c66bff4fb55f53fa1c48ce605b6cf9
incomplete pack file (will be removed): 4f1380d6ce36a11eb8158ebfa331f9fbb657fc57d3e683b738cf31bc55ef87da
incomplete pack file (will be removed): 4ff210a088d4a93ef8dadfbd5548ad243988b084cfbe4f320d1eb5f7826890cb
incomplete pack file (will be removed): 5853d3e8d5bebb3ffd4157ca6c9bdd62302486d14b09df2f60027422484fe483
incomplete pack file (will be removed): 64cbd81bcb3c0a3839400bf51c256f5ef48cf55ebdc947c9542a2157e6dbcb45
incomplete pack file (will be removed): 67080490cf05f4f4c2b2a79a329f7ae12d42803199a1a218a06329d96eeb8f48
incomplete pack file (will be removed): 701b3d47ed15afd38b4a54f4dfef8dac39adfd7fa5c220b4a27fc08f663ffba6
incomplete pack file (will be removed): 744f4a5f609cc8e0d32d6db97d41e442f64a3b4f375385f07dd9d28ebcb2733a
incomplete pack file (will be removed): 78fc31910055722c4f30d4dc98a8a102b62717169b4017463af808a64e5a271b
incomplete pack file (will be removed): 7ce924782de235bd51417e2f4b7cda9ba769d4c059eea12a016482ed8a18a7ee
incomplete pack file (will be removed): 7fefd7b9bdf8b271087996b781a1555aab0b7f87e4776e15657c2cff8c8abf20
incomplete pack file (will be removed): 80f7cae4759579cc2d24404bdc4d1ed1734c13957b825b228080d436b1f6d799
incomplete pack file (will be removed): 84d6087c5930c270f52f1f328aae66b9a1f5c37d8e950dbf6552957fb7ecce5b
incomplete pack file (will be removed): 869c67459a7d996706896a1d1a4c2ae31be1a40a4daa9d422f1e0befa3d1dcea
incomplete pack file (will be removed): 8aa1cb77f0e799835ff635b1fbb15c67c6ced46da705df7e13d89577d4f5f441
incomplete pack file (will be removed): 97b80c967d5649794deb32cc6915c1b14e537e0d1b8004a198860bf7fd5dd941
incomplete pack file (will be removed): 9a83cd05dfdcd5cfdeb32c52aa629b53e0f66d1cacbb9c9a36c89c75a7b0367a
incomplete pack file (will be removed): 9b34ae7836789b440715ee337721d56a0e4f874319b4a9f9d7bef76553cdc4b5
incomplete pack file (will be removed): 9f2a40348559b9d8e9b8323aeb39752e92beb3df06b243c7ab7ef9cdc1d27c69
incomplete pack file (will be removed): 9f715e1df6d580914abcd1fb565bca83ec862128c71f1a0b5a0ca8e1497a86df
incomplete pack file (will be removed): ac312cb7d3163de46f4a1a17ee18d3584426f08d2cc77b763baa08dcba27eea6
incomplete pack file (will be removed): ada80341e6fac08e2b4cba6c5d3ccdf4b6d55337598ae327f67e3a08a1dc167d
incomplete pack file (will be removed): b473443d7609fc9680d298ce9b498e0bf8155d96cc36b0a3d1cdfda007629856
incomplete pack file (will be removed): b4939cbed564a8e32c25d94eb83d3b95cbcfd1d7b3ff70efe991f6fdf4c9e33b
incomplete pack file (will be removed): b604e614ed8a4cd19aba44feb1f9608151a81c02eb1beef8484a2f56db3d2b6b
incomplete pack file (will be removed): b64921a9827653f311c38f77622188fe8aa59b28f3083fd12b1884a62ff235a7
incomplete pack file (will be removed): ba6c08b1bb7c9f717ce6b4dcd27d45e22fd6b8dd7cd9ba324af1ee051e72f2bd
incomplete pack file (will be removed): be1e7a1489b837859b23f6b28ff4de93f894d4cb01651c633d05487836e5d8eb
incomplete pack file (will be removed): be7d36159084ba965e2773ea331496a453df42d4cf5db940d925e13ea8ac5809
incomplete pack file (will be removed): c1290a18a7cb37b071749629d3780153c5e750e461d8f9c823dcf975b9817a45
incomplete pack file (will be removed): c65e2b45b9d5e2e7bf4413f67ac9b4ba2fab16ce799656ccd025ebc013dcae77
incomplete pack file (will be removed): c6d6eb2b00e8fad066abaa9f3495ad772e3beb74b5152775d524bf49a861fc58
incomplete pack file (will be removed): c73c3cb9a2794ef6a8960c530d515c1a250f11b4bc5d90a8230a24ce67a5e551
incomplete pack file (will be removed): c988a36ddd335b5b8e4b76ba63235632f0015ce8c8f70fcc85f8db1e7f1b185d
incomplete pack file (will be removed): daaed644c6cfb078f6211df1c5ef121fd62eb1b0f81a6d542b0388d5bfa928cf
incomplete pack file (will be removed): dd3d09a8062d80aa2c97759005f9729a88391af54f214c0895caadb34aeb571e
incomplete pack file (will be removed): e5952a97b5da1234beb15bcec2e8b31231834328b44ab2ee7cbcfa90c98f90d4
incomplete pack file (will be removed): e8c6d9c2794945503574853b5d36291d92503d6a259b51823b38f714fc178a4e
incomplete pack file (will be removed): e8da886b00b552c6bb95e9656b69366c568f778136f3d4ce49cbc0978ca13156
incomplete pack file (will be removed): e96b280db2ded87eeddfde558a72d1756e01dc430e1706a1a540586b71606623
incomplete pack file (will be removed): f00ceebe50cd5062abb52b4b73f19f95cbeed29e674b04ec3dce24251888635e
incomplete pack file (will be removed): fa1649e4c66455325e9e320bf06fbfd2e038a70d49346ab28c87b21cf36bc14e
incomplete pack file (will be removed): ff96a7f6ed64b19412c5bd69503eec2ad76cd2a82bb1d16dc7ec1ab1ffee45c3
repository contains 30387 packs (117851 blobs) with 147.154 GiB
processed 117851 blobs: 24540 duplicate blobs, 29.393 GiB duplicate
load all snapshots
find data that is still in use for 5 snapshots
[0:00] 100.00%  5 / 5 snapshots
found 93121 of 117851 data blobs still in use, removing 24730 blobs
will remove 64 invalid files
will delete 67 packs and rewrite 8017 packs, this frees 29.468 GiB
hash does not match id: want ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352, got 2ac3712c12e1b363c94fd6637fa8c09e8cffbc6d0bfc13699d83b8121e45c085
github.com/restic/restic/internal/repository.Repack
        src/github.com/restic/restic/internal/repository/repack.go:35
main.pruneRepository
        src/github.com/restic/restic/cmd/restic/cmd_prune.go:278
main.runForget
        src/github.com/restic/restic/cmd/restic/cmd_forget.go:240
main.glob..func10
        src/github.com/restic/restic/cmd/restic/cmd_forget.go:24
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).execute
        src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:762
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:852
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).Execute
        src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:800
main.main
        src/github.com/restic/restic/cmd/restic/main.go:86
runtime.main
        /usr/local/go/src/runtime/proc.go:198
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2361

Steps to reproduce the behavior
Backup once. Prune with the above command. Prune fails. Second prune command succeeds.
This seems to be always reproducible.
Do you have any idea what may have caused this?
Unfortunately no idea.
Do you have an idea how to solve the issue?
I tried to rebuild the index. It succeeds, but the the same behaviour can be reproduced: backup, prune, failure. I also removed restic cache from the home directory, nothing changed.
I created this repo for development purposes, I'm not much interested in the data it contains. I can freely experiment on it to fix this issue if someone is interested. I cannot provide the repo itself as it contains a private data set.
Did restic help you or made you happy in any way?
Of course. Seems to be a good backup solution, but still needs a few critical features.

I believe this is the same issue as this one: https://github.com/restic/restic/issues/1596

Actually I believe this is the feature-issue tracking this: https://github.com/restic/restic/issues/828

What do you mean by "hardware issue"? I have many services (git repos, storage services etc...) on the same filesystem, and nothing has ever shown any sign of corruption. I also have other types of backups created via SMB on the same share, I've never noticed broken data. With restic instead, I just created the repo and I've never been able to prune.
I'll do some other tests to see what happens.

Are all the incomplete pack files resulting from a missing/failed fflush()/fsync() after writing?

I've never noticed broken data. With restic instead, I just created the repo and I've never been able to prune.

Right. It's not necessarily a hardware issue. I was just very loosely quoting what has been discussed in previous issues. The reason to suspect hardware issues is because the error indicates that data has been somehow been corrupted/modified in the repo without restic knowing about it.
But judging from the number of issues on this theres probably also something else going on as well.

I'm running other tests, but yes, according to the reports, there seem to be a lot of suspicious hardware faults... I'll run checks on the disk and I'll create other restic repositories to see if I can reproduce this situation.

Is it possible this can be caused by interrupted operations? Is each operation treated like a transaction and is it possible to always interrupt safely? For instance, can I send SIGINT during a backup without breaking anything?

running the strace might help learn what goes on in terms of syscalls.
writes can be interrupted by SIGINT, but generally that should be fine.
my guess is that it's some interaction with the cifs/samba layer that is not operating as it should; whether that is restic's fault or not is hard to know without more data.
ping @fd0, have you seen this before?

Thanks for the report, I'm sorry for the late reply.
Incomplete pack files are usually the result of restic being aborted (e.g. by pressing ^C) during a run of restic backup. It will then abort and not finalize the file, so it is too short to be valid, which is then detected early on.
This case (in contrast to the others) is a bit odd because CIFS is involved, which may or may not be the cause.
I have several ideas how we can try to drill down here. The core issue is that restic wrote data with the SHA256 hash of ad4b41a807[...], but when reading the data back it got data with the hash 2ac3712c12e[...]:
hash does not match id: want ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352, got 2ac3712c12e1b363c94fd6637fa8c09e8cffbc6d0bfc13699d83b8121e45c085

So, we need to find out what's going on and where a (potential) bug in restic may be located.
I'd like you to do the following

Run sha256sum on the file, report back the hash that is returned
Paste the output of ls -al of the file
Run the following commands to extract the information from the index files about the file in question:

restic list index | while read idxid; do \
         restic cat index $idxid | jq '.packs | map(select(.id | contains ("ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352")))'; \
done

Thanks!

Hello,
thank you for your feedback! What I understand of your explanation is that this is the expect behavior when the user sends SIGINT to the restic process. Is this correct? Cause if this is true, then there may be no need to investigate: it is absolutely likely I sent SIGINT to the process more than once, I thought that was allowed.
If this is the case I have a couple of questions: it seems my repo is now in a very bad state, I cannot prune anymore. Isn't this something restic should avoid with something like a transaction? Second question: if SIGINT is expected to somehow break the repo, what is the proper way to interrupt restic gracefully?
If the above is instead not true and you want to investigate further, I can try to provide the info, but I'm sorry, I need to ask which file you're referring to when you say "the file". What file should I run sha256sum and ls -al on?
Thank you!

@carlonluca You're looking for the pack file referenced in the error message:
hash does not match id: want ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352, got 2ac3712c12e1b363c94fd6637fa8c09e8cffbc6d0bfc13699d83b8121e45c085

So in your case the file should be in your repo path:
ls -la your_restic_repo/data/ad/ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352

You can checksum it by cat-ing the file using restic:
restic cat pack ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352 | sha256sum

What I understand of your explanation is that this is the expect behavior when the user sends SIGINT to the restic process. Is this correct?

When restic is interrupted, it is normal to have incomplete files in the repo, and restic is able to take care of it gracefully. So we can ignore all those errors here, the data in incomplete pack files is not in use anyway.
@askielboe is right, I meant the file restic complained about that the hash is wrong, that's another (much more relevant and interesting) case here, which is not handled in a good way at all. I'd like to achieve two things:

Find out what's going on, why the hash does not match
Make restic handle this situation much more gracefully, probably there's a lot of data to be rescued and not the whole file is lost, I have plenty of ideas how to do that, but none of that is implemented yet ;)

The first point is much more important: Up to now we've only seen cases where a hardware error was the most likely cause, but I' agree: with the number of reports there may be something else going on, e.g. a bug in restic.
So, It'd be very helpful if you could run some more tests for me once you've completed the ones I laid out above! That may be enough to find the bug (if there is one...)
Since using restic cat is slightly different to running sha256sum /path/to/file (as @askielboe suggested), I'd like you to run both, so we can check if there's a difference.
Are you able to compile Go code? That'd be very helpful, because the next thing I'm going to do is write some code which examines the broken file...
Thanks for your time!

Thank you for explaining the point. This is the info you asked me:
sha256sum ./data/ad/ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352
2ac3712c12e1b363c94fd6637fa8c09e8cffbc6d0bfc13699d83b8121e45c085  ./data/ad/ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352

ls -al ./data/ad/ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352
-rwxr--r--+ 1 luca luca 7723795 set  2 11:56 ./data/ad/ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352

restic -r [...] cat pack ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352 | sha256sum
enter password for repository:
Warning: hash of data does not match ID, want
  ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352
got:
  2ac3712c12e1b363c94fd6637fa8c09e8cffbc6d0bfc13699d83b8121e45c085
2ac3712c12e1b363c94fd6637fa8c09e8cffbc6d0bfc13699d83b8121e45c085  -

I'm not sure if I did something wrong but this is the output of the script:
./test.sh
enter password for repository:
Fatal: wrong password or no key found

I'm not a go programmer, but it is interesting to learn something new, I'll try to build.

No the password is correct. The script is not working for some reason, not sure why... Each command by it self works.
However it seems that setting the password with the env variable makes the script work. Here is the output:
[]
[]
[]
[]
[]
[
  {
    "id": "ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352",
    "blobs": [
      {
        "id": "7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799",
        "type": "data",
        "offset": 0,
        "length": 1557750
      },
      {
        "id": "3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a",
        "type": "data",
        "offset": 1557750,
        "length": 152089
      },
      {
        "id": "d1cc4b006abc4b6cc6452e265cd571ea831424cd365ba12f640c7b7fd14d3c3e",
        "type": "data",
        "offset": 1709839,
        "length": 6013809
      }
    ]
  }
]
[]
[]
[]
[]
[]
[]

I tried and I had no problems building restic. I can test what you need.

@carlonluca @askielboe I've pushed a new branch debug-1999 with extra code to find out what's going on here.
Please pull the repo, then run git checkout debug-1999 and build restic with go run build.go -tags debug. Then you should have the restic debug examine command, it can be used like this (I've manually inserted an error into the file):
$ ./restic debug examine 8d83cae01cae47ec5dcf03dc2f2156ab40d8f65cc0cfa57971ed432fc1d0f608
repository ac7fd880 opened successfully, password is correct
examine 8d83cae01cae47ec5dcf03dc2f2156ab40d8f65cc0cfa57971ed432fc1d0f608
  file size is 1689865
    index 09ed6903:
      data blob e95f651fd16f8e8f6da57a890a442bb658902d20c02493f497673b0a79747268, offset 0     , raw length 1049556
      data blob 6161e68de39f8aa02a790d09750c7d79ef0c8b39c1ec978daad3332b9625558a, offset 1049556, raw length 640199
      file sizes match
      loading blob e95f651fd16f8e8f6da57a890a442bb658902d20c02493f497673b0a79747268 at 0 (length 1049556)
error decrypting blob: ciphertext verification failed
      loading blob 6161e68de39f8aa02a790d09750c7d79ef0c8b39c1ec978daad3332b9625558a at 1049556 (length 640199)
         successfully decrypted blob (length 640167), hash is 6161e68de39f8aa02a790d09750c7d79ef0c8b39c1ec978daad3332b9625558a
         IDs match
    index bf0435db does not contain the file
      data blob e95f651fd16f8e8f6da57a890a442bb658902d20c02493f497673b0a79747268, offset 0     , raw length 1049556
      data blob 6161e68de39f8aa02a790d09750c7d79ef0c8b39c1ec978daad3332b9625558a, offset 1049556, raw length 640199
      file sizes match

Please paste the output here! :slight_smile:

Thanks for looking into this!
I copied the broken pack back to the repository, but it's not in any of the indexes.
This is the output from debug examine:
~/w/r/restic ❯❯❯ ./restic debug examine 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc
debug enabled
repository 328abcf7 opened successfully, password is correct
examine 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc
  file size is 4708120
    index 6afbde31 does not contain the file
    index b1b40b1b does not contain the file
    index aab2b68a does not contain the file
    index b0b632e0 does not contain the file
    index c902db49 does not contain the file
    index 26cf689d does not contain the file
    index c56fe371 does not contain the file
    index d1d8cf3f does not contain the file
    index dcca476a does not contain the file
    index bdc2cf14 does not contain the file
    index 50d364cf does not contain the file
    index 7c554086 does not contain the file
    index e5768a4f does not contain the file
    index 4042abc2 does not contain the file
    index 067858ff does not contain the file
    index 2148915d does not contain the file
    index 386a0957 does not contain the file
    index e2791dab does not contain the file
    index e9d2fe28 does not contain the file
    index 7133e7e0 does not contain the file
    index 58f4700b does not contain the file
    index 9b16b39d does not contain the file
    index 9fee2d8d does not contain the file
    index cdbb1050 does not contain the file
    index 7522a602 does not contain the file
    index 9c6b5c9e does not contain the file
    index d2ccb0c3 does not contain the file
    index 9e54849d does not contain the file
    index 20a3b0f3 does not contain the file
    index 2c28ae45 does not contain the file
    index 8d54689b does not contain the file
    index 02a5b13b does not contain the file
    index f4ed8930 does not contain the file
    index 9aa75043 does not contain the file
    index c5b5c2a8 does not contain the file
    index 4da847de does not contain the file
    index 106baa52 does not contain the file
    index e3930f63 does not contain the file
    index 2cc7fcde does not contain the file
    index 35b4bb5d does not contain the file
    index facc9d6b does not contain the file
    index 34013b37 does not contain the file
    index 6dae7b29 does not contain the file
    index 92daac5a does not contain the file
    index f7ee7d4c does not contain the file
    index 5a902404 does not contain the file
    index 7ae30956 does not contain the file
    index 472a5bda does not contain the file
    index 2c620708 does not contain the file
    index 02d1b55e does not contain the file
    index 4fe483da does not contain the file
    index 0053d123 does not contain the file
    index 77874a21 does not contain the file
    index 98372aa3 does not contain the file
    index f2dcc150 does not contain the file
    index df8a2018 does not contain the file
    index e68a33ff does not contain the file
    index 3e00765c does not contain the file
    index 7fd88f1a does not contain the file
    index 92ac0e5d does not contain the file
    index 064b50ac does not contain the file
    index 6058589e does not contain the file
    index 42be9c17 does not contain the file
    index db400996 does not contain the file
    index fbb8ed98 does not contain the file
    index 021ef985 does not contain the file
    index 6ac4d448 does not contain the file
    index 2b5de6af does not contain the file
    index 5bf6c63a does not contain the file
    index 04406ae3 does not contain the file
    index b56ffa8a does not contain the file
    index dc463765 does not contain the file
    index e1d827c7 does not contain the file
    index 4b9699e3 does not contain the file
    index d5e893e2 does not contain the file
    index 1e984d73 does not contain the file
    index 4cf7817e does not contain the file
    index e51b74db does not contain the file
    index ed03799e does not contain the file
    index 0a0f91ee does not contain the file
    index 56f6dfd7 does not contain the file
    index d54d59c5 does not contain the file
    index 728dec39 does not contain the file
    index 2d74e6eb does not contain the file
    index a18390a9 does not contain the file
    index 1a98813d does not contain the file
    index 869b9c1b does not contain the file
    index 596b821a does not contain the file
    index 69511245 does not contain the file
    index efa90468 does not contain the file
    index 5f5402be does not contain the file
    index f588b8b9 does not contain the file
    index 9e71b904 does not contain the file
    index 02b7f247 does not contain the file
    index 8f4329a8 does not contain the file
    index 04609414 does not contain the file
    index 0b69c9fb does not contain the file
    index 4c1c8d4c does not contain the file
    index 45a8e325 does not contain the file
    index e40d44bb does not contain the file
    index 810fcf78 does not contain the file
    index 0e0678e1 does not contain the file
    index 68f97743 does not contain the file
    index 0ac9672f does not contain the file
    index c64fcd69 does not contain the file
    index c40fc8f1 does not contain the file
    index ed0ff53e does not contain the file
    index e00906a7 does not contain the file
    index 61b10704 does not contain the file
    index 43c9038f does not contain the file
    index e3d9be62 does not contain the file
    index d8bae113 does not contain the file
    index c63f1134 does not contain the file
    index e3a394fa does not contain the file
      data blob 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497, offset 0     , raw length 2701558
      data blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af, offset 2701558, raw length 914512
      data blob f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06, offset 3616070, raw length 528188
      data blob 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed, offset 4144258, raw length 563678
      file sizes match

This is the output for me:
./restic -r [...] debug examine ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352
debug enabled
enter password for repository:
repository 2b913e7d opened successfully, password is correct
examine ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352
  file size is 7723795
    index 47442452 does not contain the file
    index 54469107 does not contain the file
    index 3286998b does not contain the file
    index 6c559685 does not contain the file
    index 7fdc6666 does not contain the file
    index 93f9faf5:
      data blob 7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799, offset 0     , raw length 1557750
      data blob 3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a, offset 1557750, raw length 152089
      data blob d1cc4b006abc4b6cc6452e265cd571ea831424cd365ba12f640c7b7fd14d3c3e, offset 1709839, raw length 6013809
      file sizes match
      loading blob 7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799 at 0 (length 1557750)
         successfully decrypted blob (length 1557718), hash is 7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799
         IDs match
      loading blob 3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a at 1557750 (length 152089)
         successfully decrypted blob (length 152057), hash is 3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a
         IDs match
      loading blob d1cc4b006abc4b6cc6452e265cd571ea831424cd365ba12f640c7b7fd14d3c3e at 1709839 (length 6013809)
error decrypting blob: ciphertext verification failed
    index bcc388f7 does not contain the file
    index 94c0aaf9 does not contain the file
    index ab5a06e6 does not contain the file
    index a11ac17c does not contain the file
    index dd2d999b does not contain the file
    index fe3d54b5 does not contain the file
      data blob 7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799, offset 0     , raw length 1557750
      data blob 3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a, offset 1557750, raw length 152089
      data blob d1cc4b006abc4b6cc6452e265cd571ea831424cd365ba12f640c7b7fd14d3c3e, offset 1709839, raw length 6013809
      file sizes match

Ok, cool, thank you very much! I've pushed some more code, can you please update the branch, build restic and try again?
@askielboe I forgot that the pack isn't in any index for your repo, so I added code to try to read the blobs regardless.
In January, a user on IRC reported that he dug into a file with an invalid hash in a restic repo, found out which blob from which file was affected, then went on to re-encrypt the same file again with the same nonce to get the same ciphertext. In the end, a single bit in the blob was modified.
I've implemented trying to repair the blob by single bit flips, you can try this for yourself by running restic debug examine --try-repair <id>, let's see if this helps. Depending on your machine it should not take more than a few hours, and it's worth a shot (you'll even get a nice progress display). :)

Here is the output using --try-repair:
~/w/r/restic ❯❯❯ ./restic debug examine --try-repair 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc                                                                                debug-1999
debug enabled
repository 328abcf7 opened successfully, password is correct
examine 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc
  file size is 4708120
  wanted hash 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc, got 7a34c5089b7ec56803e1c0ed0ff21e446e31c25251809e571ecf2db07f2ac6c8
  ========================================
  looking for info in the indexes
  ========================================
  inspect the pack itself
      data blob 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497, offset 0     , raw length 2701558
      data blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af, offset 2701558, raw length 914512
      data blob f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06, offset 3616070, raw length 528188
      data blob 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed, offset 4144258, raw length 563678
      file sizes match
      loading blob 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497 at 0 (length 2701558)
         successfully decrypted blob (length 2701526), hash is 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497, ID matches
      loading blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af at 2701558 (length 914512)
error decrypting blob: ciphertext verification failed
        trying to repair blob with single bit flip
         spinning up 8 worker functions
913127 byte of 914512 done (0.01%), 2426.90 guesses per second, ETA 0ssss
        blob could not be repaired by single bit flip
      loading blob f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06 at 3616070 (length 528188)
         successfully decrypted blob (length 528156), hash is f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06, ID matches
      loading blob 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed at 4144258 (length 563678)
         successfully decrypted blob (length 563646), hash is 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed, ID matches

Ah, yeah, you have more CPU cores and your CPU is much faster than mine, I only get about ~250 guesses per second :)
Ok, so for you it's not a single bit-flip, hm. I've added another commit which adds a byte-repair mode (just plain brute force, you have a fast CPU so you can try it:
$ ./restic debug examine --try-repair --repair-byte 0bdacdfd923e2b3977701bfdd6dc8945d93bc15c0874a0202e56c4270cea6303

I updated the branch and now I got:
./restic -r [...] debug examine ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352
debug enabled
enter password for repository:
repository 2b913e7d opened successfully, password is correct
examine ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352
file size is 7723795
wanted hash ad4b41a807fef9716a60f92f63334d3b28fd5a8a744a43069455fcd42f1db352, got 2ac3712c12e1b363c94fd6637fa8c09e8cffbc6d0bfc13699d83b8121e45c085
looking for info in the indexes
index 93f9faf5:
data blob 7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799, offset 0     , raw length 1557750
data blob 3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a, offset 1557750, raw length 152089
data blob d1cc4b006abc4b6cc6452e265cd571ea831424cd365ba12f640c7b7fd14d3c3e, offset 1709839, raw length 6013809
file sizes match
loading blob 7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799 at 0 (length 1557750)
successfully decrypted blob (length 1557718), hash is 7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799, ID matches
loading blob 3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a at 1557750 (length 152089)
successfully decrypted blob (length 152057), hash is 3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a, ID matches
loading blob d1cc4b006abc4b6cc6452e265cd571ea831424cd365ba12f640c7b7fd14d3c3e at 1709839 (length 6013809)
error decrypting blob: ciphertext verification failed
inspect the pack itself
data blob 7d24a223f5943d7b6aeb98e548f7b98c7dbad03878a7e8ee6f101a89fd78f799, offset 0     , raw length 1557750
data blob 3d24f1a976e8efaf042f6fa1aeb6f6d748b9a649de7f845dfe5bf148d450fb0a, offset 1557750, raw length 152089
data blob d1cc4b006abc4b6cc6452e265cd571ea831424cd365ba12f640c7b7fd14d3c3e, offset 1709839, raw length 6013809
file sizes match
tried also to fix but the procedure is taking a considerable amount of time. It reports more than 5 hours and the ETA is increasing. After many minutes I'm still at 0.00%, processing 300 bytes per second. I'll let you know if I'll finish the procedure.

@fd0 is it useful for you if I complete the fix procedure? It seems it is really taking long.

No luck..
~/w/r/restic ❯❯❯ ./restic debug examine --try-repair --repair-byte 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc                                                                         debug-1999
debug enabled
repository 328abcf7 opened successfully, password is correct
examine 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc
  file size is 4708120
  wanted hash 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc, got 7a34c5089b7ec56803e1c0ed0ff21e446e31c25251809e571ecf2db07f2ac6c8
  ========================================
  looking for info in the indexes
  ========================================
  inspect the pack itself
      data blob 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497, offset 0     , raw length 2701558
      data blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af, offset 2701558, raw length 914512
      data blob f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06, offset 3616070, raw length 528188
      data blob 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed, offset 4144258, raw length 563678
      file sizes match
      loading blob 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497 at 0 (length 2701558)
         successfully decrypted blob (length 2701526), hash is 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497, ID matches
      loading blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af at 2701558 (length 914512)
error decrypting blob: ciphertext verification failed
        trying to repair blob by finding a broken byte
         spinning up 8 worker functions
914472 byte of 914512 done (0.01%), 66 byte per second, ETA 0sssssss
        blob could not be repaired by single bit flip
      loading blob f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06 at 3616070 (length 528188)
         successfully decrypted blob (length 528156), hash is f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06, ID matches
      loading blob 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed at 4144258 (length 563678)
         successfully decrypted blob (length 563646), hash is 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed, ID matches

@fd0 is it useful for you if I complete the fix procedure? It seems it is really taking long.

Ah, yeah, it depends a lot on how large the broken blob is (and on your CPU, obviously). You don't need to run it, it was just a quick test.
So, here comes my analysis: In general, the following holds:

During backup, restic wrote a bunch of blobs into a file, appended an encrypted header and the length of the header, then saved the file to the backend with the SHA256 hash as the file name
Reading back the data, it has a different SHA256 hash
Each blob as well as the header is encrypted and signed, so we can test each blob (and the header) for modification separately by checking the signature

From the test code's output we know:

restic can successfully read the header length and the header and decrypt it, so the header length and the header are both intact
From the contents of the header we know how long the blobs in the file are, so we can read them one by one and check the signature
Most blobs are intact, exactly one of them (for both repos discussed in this issue) does not verify, so it probably has been modified at some point somewhere
The data before and after the bad blob is unmodified

In summary, for both repos, a part of a file was modified. The following observations assume that the modification has happened accidentally (i.e. not on purpose by an attacker).
The modification may have happened within restic or outside of it. What I've seen so far makes makes me think it's very unlikely that the modification of the data has happened within restic. The reasoning is as follows:

During backup, restic reads a blob from a file, hashes it, and if it is not yet stored in the repo, encrypts it and writes it to a temporary file which later is saved to the repo. If the modification happened before encryption, the signature would match, so it must have happened after encryption.
For the name the file is saved as in the repo, restic calculates the SHA256 hash while it writes data to the file. If the modification has happened after encryption of the blob, but before it is written to the file (and the SHA256 hash is updated), the SHA256 hash would match. This is not the case, the hash is wrong, so it must have happened after writing it to the file.
Therefore, I think the modification must have happened after restic was almost done with the file. When restic considers a file to be "full enough", it'll append the header, the header length, reads back the whole file and saves it to the repo. The modification could have happened after writing the data to the file, before (or during) saving it to the repo. At this point, restic is only streaming data from a file to a backend (e.g. over HTTP(s) or to another file), the program logic is rather simple.

I'm at a loss what happened here, and I'm sorry, but what I see from the result of the analysis makes me think it's a fault outside of restic, i.e. harddisk, memory, or a network error. If it were a bug in restic (which is entirely possible!), then I would have expected it to appear much more often for the huge sets of data restic processes for various users (~1TB of data, with at most 600KB modified for @askielboe)...
For @carlonluca (repo mounted via CIFS from a server, stored on ext4 there), the modification may have been caused by the following issues:

RAM on the source machine
RAM on the CIFS server
Storage device on the CIFS server
Network error between source machine and server (I'm not sure if data transferred via CIFS is authenticated, maybe we can rule this one out)

For @askielboe (repo accessed via sftp, stored on a NAS):

RAM on the source machine
RAM on the NAS
Storage device on the NAS
We can rule out a network error here because sftp is used through SSH, which authenticates data.

I'm not sure what to do next here. Ideally, I'd love to get the results of memtest86 for all machines involved... but that'll be time-consuming and (in case of @askielboe's NAS) maybe not even possible...
We can also try to examine the damage a bit more closely, by decrypting the broken blob while disregarding the signature. If it's something from a file format that is readable without the context (or we can even find the source file on your machines), we would be able to see how many bytes have been modified and in which way exactly. I'm not sure if that helps.
So, what do you think? Is there anything I missed, or any conclusions I got wrong?

Ah, I've got something we can try: run the following on all Linux-based machines involved, then try again:
# echo 3 > /proc/sys/vm/drop_caches

This will clear all file system caches from memory, so the data needs to be re-read from disk. Then run sha256sum on the broken file in question, best would be on the machine which stores the file, or even better: run the test code again (without any --try-* options).

Thanks for the detailed analysis @fd0. Your reasoning makes sense, and I agree that it's difficult to imagine how the error could be with restic. I guess the only thing restic can do then is to attempt to recover from these kinds of errors. But I also realize that corrupted data in the repo can be an indication of a serious issue (hardware failure), and not something we would want to take lightly.
Maybe a first step could be to improve the error message a bit, given what we've discovered so far. Currently, at least for casual users, it may not be clear to them what to do about such errors, and when they actually mean for the state of their repos. We could maybe point users to a place in the documentation where they can read about possible issues / solutions regarding bad hashes / corrupted data.
If @carlonluca has a way to reproduce this more or less consistently it would be interesting to dig more into that. I will try to run a memtest on the NAS and on my laptop when I get a chance.
Is there any way to know how much of the blob has changed? If we could find the unencrypted data we could look at the modification pattern as an indication of the source, perhaps? You mentioned that someone on IRC had done something along those lines, but it's not clear to me how to do that.

Is there any way to know how much of the blob has changed? If we could find the unencrypted data we could look at the modification pattern as an indication of the source, perhaps?

I could push some code to do that...
First, you can instruct restic to print which files reference the blobs, like this:
$ restic find --blob 2e752c192106e989c40cd937d204fc48b4f1622f99d8a806331f182a2c9930e7
Found blob 2e752c192106e989c40cd937d204fc48b4f1622f99d8a806331f182a2c9930e7
 ... in file /integration_helpers_test.go
     (tree c47c2a9c9d540f497950d33f269e008f0eb5b5e1c24d99874f9ff9bbc8d28526)
 ... in snapshot 934f7ca4 (2018-10-03 10:39:17)

Then print the nodes from the tree to get a bit more info:
$ restic cat blob c47c2a9c9d540f497950d33f269e008f0eb5b5e1c24d99874f9ff9bbc8d28526 | \
  jq '.[] | map(select(.content != null)) | map(select(.content[] | contains("2e752c192106e989c40cd937d204fc48b4f1622f99d8a806331f182a2c9930e7")))'

If the file consists of a single blob then check if the file is still available (make a copy please), so we can compare the content later.

I've pushed code which will disregard the signature and save the raw data from the blob into a file. Can you please both pull, compile the code, and run the examine again? Thanks!
$ ./restic debug examine 0015018f22308125e301b207b73e534b9db9177781f0dc0c9c51ba187de47f03
[...]
      loading blob 1aa5d02ca4b856a563281a044c6ceaa3eb7ee77c82cb7282070964412590e82f at 0 (length 35324)
error decrypting blob: ciphertext verification failed
decrypt of blob 1aa5d02c stored at 1aa5d02ca4b856a563281a044c6ceaa3eb7ee77c82cb7282070964412590e82f.bin
[...]

What I can say is that I tried once again to run restic on the same dataset (actually it has a few more files) and the same happened. Pruning results in an error.

Oh, interesting! Can you paste the output of the examine command? Do you still have the exact source files so we can examine what was modified?

Running restic find with the blob from the broken pack reveals that the blob is part of a RAW image 2016-05-04/DSC_0779.NEF:
$ ./restic find --blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af
debug enabled
repository 328abcf7 opened successfully, password is correct
Found blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af
 ... in file /Lightroom CC/5d1a5b8944db40c998d8b48ed1baaf37/originals/2016/2016-05-04/DSC_0779.NEF
     (tree 24abcca60455c7407b71d60ba00a1feb254ba4803ff27e26c77e65118155d70c)
 ... in snapshot fbfa0cf6 (2018-03-24 10:36:03)
( ... finds it in many more snapshots ... )

Tree nodes:
$ ./restic cat blob 24abcca60455c7407b71d60ba00a1feb254ba4803ff27e26c77e65118155d70c | \
jq '.[] | map(select(.content != null)) | map(select(.content[] | contains("6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af")))'
debug enabled
[
  {
    "name": "DSC_0779.NEF",
    "type": "file",
    "mode": 384,
    "mtime": "2016-05-04T12:49:25+02:00",
    "atime": "2016-05-04T12:49:25+02:00",
    "ctime": "2017-12-24T16:11:56+01:00",
    "uid": 501,
    "gid": 20,
    "user": "askielboe",
    "group": "staff",
    "inode": 7153124,
    "device_id": 16777224,
    "size": 28697640,
    "links": 1,
    "content": [
      "f9b309bfba9576a22e40d1b550311a3c493d40ec1402cff267e34d298b0ce807",
      "21aea92ddfcf279a2e4dcda6f12e58894d6d28e2c2ff3237b74bde6f55f46e7c",
      "a216484024ea38600a02975f84d5751a8159fdce2721eb797ff7f5e8622ed2a9",
      "eb07db10a600eb9c992dfde45838b48d17a11e93479492f44df54a7f5e8b76ab",
      "6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af",
      "6fc18fd5cba5492d38b112de2c9b96a8d355b66229f2cdfa6e90998931128437",
      "5059c72ab73d69cc286df9ef3516c7e3befcf1aafd88274d2a19869e2f0df8d1",
      "66da2fcb6cfab50cde664d7827c2166287608f4ec8f5af17c4111147691914fd",
      "bce2cb66abf4d70fae85a9b1394944de20b643fd60990d65402441a84894744b",
      "f35d898c783f23c4f5b61c8ddfbee05639b4e7957ff174b2495dab7740c91b22",
      "e07a9946c870afba76db13ac586a6e5951cab661c69009dd981204c1124eec25",
      "6e2dda5ab24c75b823ae9a0b6e51e592a307bd6b76e6c78dba87f812edbafc2a",
      "50a300debd1a0df47641d1398b74ae2aef03e3fbc67c317e96ed21b28511d627",
      "a9112c57c2c8ce898c417f140d6a158ff2455e2188d7ba769e83e5be0e81691e",
      "21726b812aaafbd772a40a35a771917925bd426b4662f3da1847fae49b3d1434",
      "501e988bf02fbd4d13b74e5af529289c29f52522501a6df5ace0f0857daf056a",
      "fa4fb1d55b8abf7726ecf291e04c9d36f4802f9de26dd02980773986e1fbce9f",
      "7a51c778780e82297beb5e1a3bfafa6508cb3978fb542762b3521f68887dd937",
      "dc0750ca030e7a7555267f085ff2e2bec2e4b29cd1399c9581c61e337b7dd463"
    ]
  }
]

restic succeeds in restoring the file:
./restic restore fbfa0cf6 -i "/Lightroom CC/5d1a5b8944db40c998d8b48ed1baaf37/originals/2016/2016-05-04/DSC_0779.NEF" -t ~/restic/restore/
debug enabled
repository 328abcf7 opened successfully, password is correct
restoring <Snapshot fbfa0cf6 of [/Volumes/BACKUP/Lightroom CC] at 2018-03-24 10:36:03.32108098 +0100 CET by<EMAIL_ADDRESS>to /Users/askielboe/restic/restore/

And the file seems to be same same as the one stored in Lightroom:
$ ls -l *.NEF
-rw-r--r--  1 askielboe  staff  28697640 Oct  3 15:26 DSC_0779_lightroom.NEF
-rw-------  1 askielboe  staff  28697640 May  4  2016 DSC_0779_restored.NEF
$ gsha256sum DSC_0779_lightroom.NEF
584cdd8fab7e52da09ffa5960a1b945951c817dc267840e5fa71ad363bcdfa60  DSC_0779_lightroom.NEF
$ gsha256sum DSC_0779_restored.NEF
584cdd8fab7e52da09ffa5960a1b945951c817dc267840e5fa71ad363bcdfa60  DSC_0779_restored.NEF

Running restic examine with your new commit dumps the blob to a file:
$ ./restic debug examine 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc
debug enabled
repository 328abcf7 opened successfully, password is correct
examine 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc
  file size is 4708120
  wanted hash 28d5bcd6e514e766feb70052c9b212069b47e09fe2d295f5fd27bce8d6999bdc, got 7a34c5089b7ec56803e1c0ed0ff21e446e31c25251809e571ecf2db07f2ac6c8
  ========================================
  looking for info in the indexes
  ========================================
  inspect the pack itself
      data blob 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497, offset 0     , raw length 2701558
      data blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af, offset 2701558, raw length 914512
      data blob f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06, offset 3616070, raw length 528188
      data blob 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed, offset 4144258, raw length 563678
      file sizes match
      loading blob 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497 at 0 (length 2701558)
         successfully decrypted blob (length 2701526), hash is 01db218adbcb31df98a7dc2e7a71a691ace67b74cff17b8918afc776f6ab3497, ID matches
      loading blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af at 2701558 (length 914512)
error decrypting blob: ciphertext verification failed
decrypt of blob 6a94ca24 stored at 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af.bin
      loading blob f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06 at 3616070 (length 528188)
         successfully decrypted blob (length 528156), hash is f4634cd188251beeb39f190582c8f041aa7a031f4101b5f5438e27ed9ef7eb06, ID matches
      loading blob 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed at 4144258 (length 563678)
         successfully decrypted blob (length 563646), hash is 973f9f1744ac0d4ac93899ab2b9eb28d2c0b058c723792f02cd04c3a4b9c66ed, ID matches

Ah, great! After you removed the broken file from the repo, the next backup introduced the missing blobs again (that's why you were able to successfully restore the file). This makes it easy to extract the original blob:
$ restic cat blob 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af > 6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af.orig

Can you make a binary diff, e.g. by using xxd on the two files and then running diff -au? Or can you maybe upload both files somewhere (if that's okay for you)? I'd like to see what the structure of the differences is.
We can by the way only do that because we're using the counter mode for AES for encryption, so that a flipped bit in the ciphertext corresponds to exactly the same flipped bit in the plaintext (which is also why it is so important to check the authenticity of data before attempting to decrypt it).

Ah, cool that restic recovers from this after deleting the pack. :) I was a actually bit puzzled that it just recovered the .NEF with no errors.
Here are the two blobs for comparison. The diff is quite large, around 2000 lines in the hexdump, so I've just uploaded the raw blobs in a .zip archive for you to examine.
The original blob (.orig) as retrieved from restic cat blob, and the one recovered from the corrupted pack using restic debug examine (.bin):
6a94ca24a7dc65bb338d7028533401bfbf3f486da28f877979027737ab59f9af.zip

Hm, odd: The differences start at offset 0x646fa, the blob is unmodified again starting at offset 0x6c6fa, which means 32768 bytes are modified. That's exactly 32KiB. The start offset is not a multiple of 32KiB though. XORing the bytes does not show any obvious pattern.
I'm thinking aloud here: For a memory (RAM) error that's a rather surprising result, it could hint at storage problems. Maybe the file system on the NAS, or the underlying storage medium, uses 32KiB blocks? Even then, I would have expected the start offset to be a multiple of 32KiB...
Another thought: The usual transfer chunk size for data over sftp is also 32KiB...
It'd be great to have the same data from @carlonluca so we can compare it.

Interesting!
I'm running Synology Hybrid Raid with four disks, one disk fault tolerance. This is the filesystem:
ash-4.3# tune2fs -l /dev/vg2/volume_1
tune2fs 1.42.6 (21-Sep-2012)
Filesystem volume name:   1.41.12-2657
Last mounted on:          /volume1
Filesystem UUID:          ebc76139-f15e-4dc0-bd95-342c9eef17c5
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              609583104
Block count:<PHONE_NUMBER>
Reserved block count:     92560
Free blocks:              155081863
Free inodes:              608313936
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      442
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sat Nov 24 19:32:52 2012
Last mount time:          Thu Jul 12 15:42:27 2018
Last write time:          Thu Jul 12 15:42:27 2018
Mount count:              124
Maximum mount count:      23
Last checked:             Sat Nov 24 19:32:52 2012
Check interval:           15552000 (6 months)
Next check after:         Thu May 23 20:32:52 2013
Lifetime writes:          24 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      f5e67d0a-bc57-4269-bee4-699086013fa3
Journal backup:           inode blocks

Ok, first thing you asked me was to run debug examine on the new repo. This is what I see when it fails:
found 101183 of 107436 data blobs still in use, removing 6253 blobs
will remove 102 invalid files
will delete 102 packs and rewrite 2455 packs, this frees 8.554 GiB
hash does not match id: want 7d26eb4258f31e39230b05e7da9f248a2fa5843eb0978cd7e027f8dca09e31d5, got 5e078b5cadc511673b6f2216c6f82b2bb5440f9ad050432959f36273010c42e6
github.com/restic/restic/internal/repository.Repack
        /build/restic-4SDbaH/restic-0.8.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/repository/repack.go:35
main.pruneRepository
        /build/restic-4SDbaH/restic-0.8.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/cmd_prune.go:278
main.runForget
        /build/restic-4SDbaH/restic-0.8.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/cmd_forget.go:235
main.glob..func9
        /build/restic-4SDbaH/restic-0.8.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/cmd_forget.go:24
github.com/spf13/cobra.(*Command).execute
        /build/restic-4SDbaH/restic-0.8.3+ds/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:698
github.com/spf13/cobra.(*Command).ExecuteC
        /build/restic-4SDbaH/restic-0.8.3+ds/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:783
github.com/spf13/cobra.(*Command).Execute
        /build/restic-4SDbaH/restic-0.8.3+ds/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:736
main.main
        /build/restic-4SDbaH/restic-0.8.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/main.go:69
runtime.main
        /usr/lib/go-1.10/src/runtime/proc.go:198
runtime.goexit
        /usr/lib/go-1.10/src/runtime/asm_amd64.s:2361

Debug examine reports:
repository 4415f7da opened successfully, password is correct
examine 7d26eb4258f31e39230b05e7da9f248a2fa5843eb0978cd7e027f8dca09e31d5
  file size is 5375650
  wanted hash 7d26eb4258f31e39230b05e7da9f248a2fa5843eb0978cd7e027f8dca09e31d5, got 5e078b5cadc511673b6f2216c6f82b2bb5440f9ad050432959f36273010c42e6
  ========================================
  looking for info in the indexes
  ========================================
  inspect the pack itself
      data blob dddf268db95b2fa4213b85067acb28e89b7cce9d6baf400502172a1a77d241d3, offset 0     , raw length 286200
      data blob 4edd7992407336bdcd1a6635866581968f819050b741afc7cc9f5f379c0a652c, offset 286200, raw length 727992
      data blob 13057762ef010f2852492879764e02a8a2e0df25ff1819f726026f4476934b33, offset 1014192, raw length 1517212
      data blob 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3, offset 2531404, raw length 2844062
      file sizes match
      loading blob dddf268db95b2fa4213b85067acb28e89b7cce9d6baf400502172a1a77d241d3 at 0 (length 286200)
         successfully decrypted blob (length 286168), hash is dddf268db95b2fa4213b85067acb28e89b7cce9d6baf400502172a1a77d241d3, ID matches
      loading blob 4edd7992407336bdcd1a6635866581968f819050b741afc7cc9f5f379c0a652c at 286200 (length 727992)
         successfully decrypted blob (length 727960), hash is 4edd7992407336bdcd1a6635866581968f819050b741afc7cc9f5f379c0a652c, ID matches
      loading blob 13057762ef010f2852492879764e02a8a2e0df25ff1819f726026f4476934b33 at 1014192 (length 1517212)
         successfully decrypted blob (length 1517180), hash is 13057762ef010f2852492879764e02a8a2e0df25ff1819f726026f4476934b33, ID matches
      loading blob 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3 at 2531404 (length 2844062)
error decrypting blob: ciphertext verification failed

so I searched the blob:
./restic -r [...] find --blob 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3
debug enabled
enter password for repository:
repository 4415f7da opened successfully, password is correct
Found blob 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3
 ... in file [...]/20161126_230830_high.mkv
     (tree 7dfc4bf202ada0460c8a23a2997489feab0f71d872f716174447f4d81a444078)
 ... in snapshot 3ef1b37b (2018-10-02 23:27:22)
Found blob 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3
 ... in file [...]/20161126_230830_high.mkv
     (tree 7dfc4bf202ada0460c8a23a2997489feab0f71d872f716174447f4d81a444078)
 ... in snapshot a27909fe (2018-10-02 22:31:39)
Found blob 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3
 ... in file [...]/20161126_230830_high.mkv
     (tree 7dfc4bf202ada0460c8a23a2997489feab0f71d872f716174447f4d81a444078)
 ... in snapshot d714794a (2018-10-03 21:25:25)

I then tried to restore the file:
./restic -r [...] d714794a -i "[...]/20161126_230830_high.mkv" -t .
debug enabled
enter password for repository:
repository 4415f7da opened successfully, password is correct
restoring <Snapshot d714794a of [...] at 2018-10-03 21:25:25.800516269 +0200 CEST by [...]

and the file was restored properly. I then checked the md5 of the restored file with the original one and I got a perfect match. What did I do wrong?

@carlonluca To get a copy of the original and the modified blobs for comparison I did the following:
Start by pulling the latest commit from the debug-1999 branch, recompile and run restic examine again you should get a dump of the modified blob.
It should let you know that it saved the modified blob with a .bin extension.
$ ./restic debug examine 7d26eb4258f31e39230b05e7da9f248a2fa5843eb0978cd7e027f8dca09e31d5
(...)
loading blob 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3 at 2531404 (length 2844062)
error decrypting blob: ciphertext verification failed
decrypt of blob 04cab192 stored at 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3.bin

Since you're able to restore the file perhaps you still have the original blob. You can recover the original using restic cat:
$ restic cat blob 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3 > 04cab192e213693bda9df9179529aa6a468c3e3d0631faf2254ec17623cd00e3.orig

Now you can compare the two blobs as described in @fd0's comment above: https://github.com/restic/restic/issues/1999#issuecomment-426721653

I then checked the md5 of the restored file with the original one and I got a perfect match. What did I do wrong?

Apparently, the  blob in question is present in the repo more than once, and the one selected by restic restore was unmodified. Which is great, so we can do the same as with @askielboe's repo, and then see what has been modified :)

Hope I did it properly. This is the diff: https://app.box.com/s/xvubjn37y4xpn25fbrnyqqrnd7klqomt. The repo contains private data so I'll leave it here until you download it and then I'll remove it.

Got it, you can remove the file.

Thank you very much!
Quick analysis:

the broken part of the blob starts at offset 0x135fa4 (1269668) in the blob, the first good byte is at 0x2b5fb0 (2842544) which means 1572876 bytes changed. That's not even a multiple of 16.
within the file, the start is 0x39fff0 (3801072) and the first good byte is at 0x51fffc (5373948)

I still haven't got a clue what's going on. It also does not look like a memory (RAM) failure, but rather a storage failure. Odd.

To me these tests, along with your reasoning @fd0 seem to suggest that this is most likely a hardware issue, or at least an issue outside of restic.
What are your thoughts on handling these cases in restic? Are there any meaningful paths that we could take to improve the user experience when repo data gets corrupted?
A few thoughts/ideas:

Documentation: We could extend the error message somewhat to inform the user that they should check the consistency of their repository. Judging from the number of issues on this, there is a bit confusion about what a wrong object hash means. I'm not entirely sure if we've covered all cases of this, and can say anything definitive about it yet, but it seems that at least some of it could be hardware related - and warrant some action by the user to protect their backups.

Active recovery: Given that I managed to repair my repository by removing the broken pack, and let restic take care of the rest, would it be a viable option to let restic try to repair the repo by re-uploading bad blobs to the repo if the data is available? At least it would feel safer to let restic manipulate the repository, rather than having to modify the objects manually.

More tests: Run more tests to figure out what's going on before we take action. If we had some way of reliable reproduction that would be helpful, but since it's likely a hardware failure I guess that's difficult.

In any case it would still be nice to have a way to recover from corrupted blobs using restic. I probably don't currently know enough about the internals of restic to decide what the best approach is, but I'd like to help out if possible.

What are your thoughts on handling these cases in restic? Are there any meaningful paths that we could take to improve the user experience when repo data gets corrupted?

Oh yes, absolutely! restic needs to handle such cases a lot better. When the hash does not match for a file, this is just an indication that something is wrong, it should be clearly printed as such. Restic can then try to load and decrypt the individual blobs. For the blobs which could be successfully verified, restic can just use them. The others (which do not verify) should be discarded, that's what we have the signature for. After all, they could have been modified by an attacker, and should be treated as such. This allows at least to salvage the unmodified parts of the file.
Then, restic prune should proceed, not including any broken blobs in the repository. This makes restic check complain that data is missing, but most of the time the original data is still there (like in your case) and the repository will just repair itself. Or users discard and forget the snapshots which reference the missing data.
We should definitely do 1) and 2), 3) is hard to do because I don't have any idea what's going on. It'd be great to have some feedback should you ever find out what's going on with your hardware!

The discussion about how to handle these cases flows into a forum thread I made a while ago with an idea about restic offering a suggestion about what to do next, in case of an error: https://forum.restic.net/t/deterministic-output-of-restic-check-to-know-what-to-do-next/865?u=matt
The example there is for check, but I could see it happening for errors on any other command, too. I would be interested in implementing this if someone wanted to give me a pointer as to the best, centralized place to handle errors in this way.

I wanted to post here and let you know that I've encountered this same issue.
$ restic version
restic 0.9.3 compiled with go1.11.1 on linux/amd64

And here is the output from the crash I experienced:
$ restic --verbose prune
repository 1234abcd opened successfully, password is correct
counting files in repo
building new index for repo
[32:26:19] 88.26%  6785000 / 7687417 packs
repository contains 6785000 packs (85358468 blobs) with 31.357 TiB
processed 85358468 blobs: 2230223 duplicate blobs, 1.748 TiB duplicate
load all snapshots
find data that is still in use for 276 snapshots
[3:31:18] 100.00%  276 / 276 snapshots
found 43986057 of 85358468 data blobs still in use, removing 41372411 blobs
will remove 0 invalid files
will delete 1678011 packs and rewrite 777903 packs, this frees 9.305 TiB
hash does not match id: want 47ac6c02178760171f58fef41d43f53d16b2af4c0b281386b1cdb1f1a781415c, got f7107ae08f0656aae76d52e89fb1402c2ad4e820c08bf0059120addac7774c24
github.com/restic/restic/internal/repository.Repack
        /restic/internal/repository/repack.go:35
main.pruneRepository
        /restic/cmd/restic/cmd_prune.go:278
main.runPrune
        /restic/cmd/restic/cmd_prune.go:85
main.glob..func18
        /restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
        /restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        /restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        /restic/vendor/github.com/spf13/cobra/command.go:800
main.main
        /restic/cmd/restic/main.go:86
runtime.main
        /usr/local/go/src/runtime/proc.go:201
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333

I'm using AWS S3 to store my repository, so I don't expect that my storage system is corrupting this data. I've experienced restic getting killed due to excessive memory usage many times. I think it is much more likely that this corruption happened during an OOM kill event.
If there is any further information I can provide, please ask.

I'm using AWS S3 to store my repository, so I don't expect that my storage system is corrupting this data.

Agreed, that's at least not likely. What's more likely is this case is either a bug we haven't found, or memory issues on the machine which ran restic...

I think it is much more likely that this corruption happened during an OOM kill event.

Hm, that's an interesting hypothesis. It would help if you could check out the branch debug-1999, run go run build.go, and then run the analysis code I described above:
$ restic debug examine 47ac6c02178760171f58fef41d43f53d16b2af4c0b281386b1cdb1f1a781415c

Based on the output we can maybe see what's going on here.

It's a bug, the error returned by List() is ignored here: https://github.com/restic/restic/blob/d10754e2b4974bc6e69e71b23b472f82c0aa92fe/internal/list/list.go#L64
I'll open a PR shortly which fixes this issue.

Thank you so much for the help and quick fix!
I have built the debug-1999 branch, and used it to run restic debug examine ... as described:
$ ./restic version
debug enabled
restic 0.9.3 (v0.9.3-46-g2d0433d55ab1) compiled with go1.9.4 on linux/amd64
$ ./restic debug examine 47ac6c02178760171f58fef41d43f53d16b2af4c0b281386b1cdb1f1a781415c
debug enabled
repository 1234abcd opened successfully, password is correct
body not drained, 0 bytes not read, error: unexpected EOF
load <index/43e56f5149>: invalid data returned, ignoring
body not drained, 0 bytes not read, error: unexpected EOF
load <index/43ddc92f40>: invalid data returned, ignoring
Load(<index/43eb88b1d3>, 0, 0) returned error, retrying after 576.185522ms: Copy: The difference between the request time and the current time is too large.
body not drained, 0 bytes not read, error: unexpected EOF
load <index/c2120ea44c>: invalid data returned, ignoring
body not drained, 0 bytes not read, error: unexpected EOF
load <index/c246aef04e>: invalid data returned, ignoring
body not drained, 0 bytes not read, error: unexpected EOF
load <index/c23c61cdf1>: invalid data returned, ignoring
examine 47ac6c02178760171f58fef41d43f53d16b2af4c0b281386b1cdb1f1a781415c
  file size is 4434961
  hash for file content matches
  ========================================
  looking for info in the indexes
    index f2895a5f:
      data blob bae2a10a0c27870a12f367417652626268617404d48c47e9e545496cc440d0e3, offset 0     , raw length 990083
      data blob 2d26e47cd610d9a9de50ab9fe807451806bda361a5451580b0e929349bda77f3, offset 990083, raw length 328654
      data blob 9ce6e4563d39aa3de72177f4b9a96901d8c116e4d7ed105f4c97a0b74ef9309f, offset 1318737, raw length 2098058
      data blob 41b8f984e5282752d7d35f4ca547a271864ee2c3f9843fef4b15cda768197f2d, offset 3416795, raw length 1017982
      file sizes match
      loading blob bae2a10a0c27870a12f367417652626268617404d48c47e9e545496cc440d0e3 at 0 (length 990083)
         successfully decrypted blob (length 990051), hash is bae2a10a0c27870a12f367417652626268617404d48c47e9e545496cc440d0e3, ID matches
      loading blob 2d26e47cd610d9a9de50ab9fe807451806bda361a5451580b0e929349bda77f3 at 990083 (length 328654)
         successfully decrypted blob (length 328622), hash is 2d26e47cd610d9a9de50ab9fe807451806bda361a5451580b0e929349bda77f3, ID matches
      loading blob 9ce6e4563d39aa3de72177f4b9a96901d8c116e4d7ed105f4c97a0b74ef9309f at 1318737 (length 2098058)
         successfully decrypted blob (length 2098026), hash is 9ce6e4563d39aa3de72177f4b9a96901d8c116e4d7ed105f4c97a0b74ef9309f, ID matches
      loading blob 41b8f984e5282752d7d35f4ca547a271864ee2c3f9843fef4b15cda768197f2d at 3416795 (length 1017982)
         successfully decrypted blob (length 1017950), hash is 41b8f984e5282752d7d35f4ca547a271864ee2c3f9843fef4b15cda768197f2d, ID matches
  ========================================
  inspect the pack itself
      data blob bae2a10a0c27870a12f367417652626268617404d48c47e9e545496cc440d0e3, offset 0     , raw length 990083
      data blob 2d26e47cd610d9a9de50ab9fe807451806bda361a5451580b0e929349bda77f3, offset 990083, raw length 328654
      data blob 9ce6e4563d39aa3de72177f4b9a96901d8c116e4d7ed105f4c97a0b74ef9309f, offset 1318737, raw length 2098058
      data blob 41b8f984e5282752d7d35f4ca547a271864ee2c3f9843fef4b15cda768197f2d, offset 3416795, raw length 1017982
      file sizes match
$ echo $?
0

I'm not sure how to interpret this output, but at the very least, it returned a successful return code. (It took 30-45 minutes of run time to give me this output.)
Second thing: I have built the master branch (revision 9310cd0cd67) which contains PR #2066. I have also merged in the debug-1999 branch into this. I am re-running the restic --verbose prune with this build, to see if I can reproduce the crash in my original report.
Thanks again! Please let me know if there is any additional information I can provide.

Hm, interesting. It did not detect anything wrong with the file in the repo.
Would you mind cancelling the run of prune and running this command first:
$ restic cat pack 47ac6c02178760171f58fef41d43f53d16b2af4c0b281386b1cdb1f1a781415c | sha25sum

Let's see if we get the data back from the repo that restic thought it wrote...

Oh, it also says:
  hash for file content matches

This means that this time around, restic was able to read the correct data from S3. Hm. I can think of several different explanations:

S3 returned the wrong data
Something in between restic and S3 changed data, like an enterprise "malware protection gateway" which unwraps TLS (do you have something like that?)
Not enough data was returned or the connection was reset, and restic (or the library we're using for s3 access) did not handle that well

Hi,
Unfortunately, I'm experiencing the same problem. Have been for a while. I have a 7 day rolling backup going on and for the past few weeks, I've been unable to prune. I'm running my backups on a repo that is stored on a ZFS filesystem.
I've tried with mainline 0.9.3 release and also the debug-1999 branch.
My pruning gets to about 90% then fails with the hash doesn't match id error.
% restic cat pack 3af14a9de69c133d3b1087fed6c90e74c06489725eb1c051db396ce2f7196786 | sha256sum
Warning: hash of data does not match ID, want
  3af14a9de69c133d3b1087fed6c90e74c06489725eb1c051db396ce2f7196786
got:
  a5363b0f1cbf4a00321864d4fdb440a954162d10dd809ac24b57501f7a43beb5
a5363b0f1cbf4a00321864d4fdb440a954162d10dd809ac24b57501f7a43beb5  -

% ./restic debug examine 3af14a9de69c133d3b1087fed6c90e74c06489725eb1c051db396ce2f7196786
debug enabled
repository ab35c951 opened successfully, password is correct
examine 3af14a9de69c133d3b1087fed6c90e74c06489725eb1c051db396ce2f7196786
  file size is 4242688
  wanted hash 3af14a9de69c133d3b1087fed6c90e74c06489725eb1c051db396ce2f7196786, got a5363b0f1cbf4a00321864d4fdb440a954162d10dd809ac24b57501f7a43beb5
  ========================================
  looking for info in the indexes
    index 78ad2314:
      data blob 0625bf489d3bad6a245cdfd053bfd48118caa694a6f7b7c2a5222ae52a26a919, offset 0     , raw length 1192491
      data blob 3bc76c62366728edf64d2fa677558bd397fe544807d4e55595f6816783a46b54, offset 1192491, raw length 550655
      data blob 24d6d44ee609ebfec07bab9f2f771402131df1b2a4f882beb3b26c2baecd6c93, offset 1743146, raw length 541698
      data blob a080fe7c83f1e5a5744a4bc3b4214f917d6ac084d30541359382d9724467b387, offset 2284844, raw length 1957660
      file sizes match
      loading blob 0625bf489d3bad6a245cdfd053bfd48118caa694a6f7b7c2a5222ae52a26a919 at 0 (length 1192491)
error decrypting blob: ciphertext verification failed
decrypt of blob 0625bf48 stored at 0625bf489d3bad6a245cdfd053bfd48118caa694a6f7b7c2a5222ae52a26a919.bin
      loading blob 3bc76c62366728edf64d2fa677558bd397fe544807d4e55595f6816783a46b54 at 1192491 (length 550655)
         successfully decrypted blob (length 550623), hash is 3bc76c62366728edf64d2fa677558bd397fe544807d4e55595f6816783a46b54, ID matches
      loading blob 24d6d44ee609ebfec07bab9f2f771402131df1b2a4f882beb3b26c2baecd6c93 at 1743146 (length 541698)
         successfully decrypted blob (length 541666), hash is 24d6d44ee609ebfec07bab9f2f771402131df1b2a4f882beb3b26c2baecd6c93, ID matches
      loading blob a080fe7c83f1e5a5744a4bc3b4214f917d6ac084d30541359382d9724467b387 at 2284844 (length 1957660)
         successfully decrypted blob (length 1957628), hash is a080fe7c83f1e5a5744a4bc3b4214f917d6ac084d30541359382d9724467b387, ID matches
  ========================================
  inspect the pack itself
      data blob 0625bf489d3bad6a245cdfd053bfd48118caa694a6f7b7c2a5222ae52a26a919, offset 0     , raw length 1192491
      data blob 3bc76c62366728edf64d2fa677558bd397fe544807d4e55595f6816783a46b54, offset 1192491, raw length 550655
      data blob 24d6d44ee609ebfec07bab9f2f771402131df1b2a4f882beb3b26c2baecd6c93, offset 1743146, raw length 541698
      data blob a080fe7c83f1e5a5744a4bc3b4214f917d6ac084d30541359382d9724467b387, offset 2284844, raw length 1957660
      file sizes match

@dharrigan interesting, thanks for the data point! Since it's stored on ZFS I assume that the FS and the storage layer below is not at fault. Can you maybe run memtest on the machine which runs restic (and does the encryption)?

Hi,
Thanks for the update. I'll fire up memtest this evening before going home and hopefully have some results tomorrow - for as we know, it can take a bit of time :-)
-=david=-

@dharrigan there's something else you can do: as far as I know zfs has some internal checking/scrubbing process, can you try that so we can definitely eliminate file system or storage corruption?

Hi,
I perform a scrub every 15 days, and as you can see, the latest scrub was
successful:
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 0h0m with 0 errors on Fri Feb  1 01:00:08 2019

-=david=-
I prefer encrypted and signed messages.
Fingerprint: 110A F423 3647 54E2 880F ADAD 1C52 85BF B20A 22F9
No trees were harmed in the sending of this message, however, a number of
electrons were inconvenienced.
This email is sent without prejudice.
On Thu, 21 Feb 2019 at 18:49, Alexander Neumann<EMAIL_ADDRESS>wrote:

@dharrigan https://github.com/dharrigan there's something else you can
do: as far as I know zfs has some internal checking/scrubbing process, can
you try that so we can definitely eliminate file system or storage
corruption?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/restic/restic/issues/1999#issuecomment-466119266, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAaEjitrGEJX-dvTN-h01g5hLBkcUalgks5vPuohgaJpZM4WpF03
.

Just did another scrub, no issues
# zpool status
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 0h2m with 0 errors on Thu Feb 21 19:02:43 2019

-=david=-
I prefer encrypted and signed messages.
Fingerprint: 110A F423 3647 54E2 880F ADAD 1C52 85BF B20A 22F9
No trees were harmed in the sending of this message, however, a number of
electrons were inconvenienced.
This email is sent without prejudice.
On Thu, 21 Feb 2019 at 18:59, David Harrigan<EMAIL_ADDRESS>wrote:

Hi,
I perform a scrub every 15 days, and as you can see, the latest scrub was
successful:
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 0h0m with 0 errors on Fri Feb  1 01:00:08 2019

-=david=-
I prefer encrypted and signed messages.
Fingerprint: 110A F423 3647 54E2 880F ADAD 1C52 85BF B20A 22F9
No trees were harmed in the sending of this message, however, a number of
electrons were inconvenienced.
This email is sent without prejudice.
On Thu, 21 Feb 2019 at 18:49, Alexander Neumann<EMAIL_ADDRESS>wrote:

@dharrigan https://github.com/dharrigan there's something else you can
do: as far as I know zfs has some internal checking/scrubbing process, can
you try that so we can definitely eliminate file system or storage
corruption?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/restic/restic/issues/1999#issuecomment-466119266,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAaEjitrGEJX-dvTN-h01g5hLBkcUalgks5vPuohgaJpZM4WpF03
.

Hi,
Okay, so I changed the memory in my machine, ran memtest and ensured zero errors. I'm unfortunately, still experiencing the same problem. Note, the hashes have changed as I got a bit fed up and wiped all but one of the snapshots.
repository ab35c951 opened successfully, password is correct
counting files in repo
building new index for repo
[22:36] 100.00%  148697 / 148697 packs
repository contains 148697 packs (6526671 blobs) with 665.735 GiB
processed 6526671 blobs: 5081238 duplicate blobs, 449.560 GiB duplicate
load all snapshots
find data that is still in use for 1 snapshots
[0:26] 100.00%  1 / 1 snapshots
found 863576 of 6526671 data blobs still in use, removing 5663095 blobs
will remove 0 invalid files
will delete 13204 packs and rewrite 122251 packs, this frees 649.052 GiB
hash does not match id: want 3af14a9de69c133d3b1087fed6c90e74c06489725eb1c051db396ce2f7196786, got a5363b0f1cbf4a00321864d4fdb440a954162d10dd809ac24b57501f7a43beb5
github.com/restic/restic/internal/repository.Repack
	/restic/internal/repository/repack.go:34
main.pruneRepository
	/restic/cmd/restic/cmd_prune.go:278
main.runPrune
	/restic/cmd/restic/cmd_prune.go:85
main.glob..func18
	/restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
	/restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
	/restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
	/restic/vendor/github.com/spf13/cobra/command.go:800
main.main
	/restic/cmd/restic/main.go:86
runtime.main
	/usr/local/go/src/runtime/proc.go:201
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1333

I am also getting this:
restic prune repository da2081b5 opened successfully, password is correct counting files in repo building new index for repo [1:12:54] 100.00%  115128 / 115128 packs incomplete pack file (will be removed): 029bbacd7c0fff2cd42a7442caa92dfc0cc66df2d269998f521cdc7bef27276d incomplete pack file (will be removed): 05d0c51e47b5c5a041ddea89bf097833b09da3f5b91c432a3c7f78259b6acf50 incomplete pack file (will be removed): 09bb387822cb274605939595a9e45311f816623db291223e19d4d1df14feaf1c incomplete pack file (will be removed): 0b0ad1912bf3b885b6a3a30ac9595f678e573383a4df921183b3936a80af1413 incomplete pack file (will be removed): 0c86f0e72a5e57a07b1ef92a63097c613768833dd41d75302bfd644fc42b6198 incomplete pack file (will be removed): 0cc2e8b1232584f705047fa039f31c00be2cb9755b08964ee2cad29958c1f2ed incomplete pack file (will be removed): 0d64cc67d7cfa4d9e9b224b1f543004a621562b39df3380d2533d352bc3bc7f1 incomplete pack file (will be removed): 111c917d00a4b5a4537e614d3068ec30210fed33d55e74b7fddabcdb72deecc2 incomplete pack file (will be removed): 11cdbb52432a5664fe86d673d38c243f86cdcebb2a450dd905d0e6c795b08f18 incomplete pack file (will be removed): 12b0f67da1f99b15baf00278d6a550c11c43f0acbfd94e2233aba21c0e025d17 incomplete pack file (will be removed): 12fe361209ac6909ef4ad46cb44baa500c51a8805d150bac544ee2ea67542695 incomplete pack file (will be removed): 131931b950527e578d9887fe01f68cdf896236375185508a7f5f8f8d9d26ac2c incomplete pack file (will be removed): 14e99b2b5dcf8e103c24bf1c385df9bffd54a9c18386951db7e9fc8f917a128c incomplete pack file (will be removed): 16ecf72415c35dbb4d73837f72b93f55d47d63031678cd2b3a4cf19d8b268ff4 incomplete pack file (will be removed): 2567d0209077cfec4619c9fbbdeaad71d4a47edfd04dddd1aa0d86b508dbd6bf incomplete pack file (will be removed): 25c3d2b89cc860268b76d5c60f4c7c6382f24a8e4bee16d591182bed456c84d7 incomplete pack file (will be removed): 2e056ee985edae065fe79ad93f490de33331770b4632af35804a187d9891e1d2 incomplete pack file (will be removed): 30f0ca193b0cb0e50bafe9a6841159a35765304233f2806cb51f9380fe01a60c incomplete pack file (will be removed): 31338b2b3a59cc0c592aebd1412b26eeb81d072dbdf0051d41f4c0db01ada9b9 incomplete pack file (will be removed): 34473e6836c18600c98ad6d432422b45a0701d48d67e73e37a64ef1a7ea0a8c3 incomplete pack file (will be removed): 36719ea58d39076b807af8935f0f22b5444e3cd3e0e06df50b522e8caae2c431 incomplete pack file (will be removed): 36d01a86d1f213c6edf40ede5da4657f043f2a9d41242b227e3ec39b00d8d81a incomplete pack file (will be removed): 375b30f87666ac4413533520bf86e1a7ab7d1b0198a40e4209adb777748fe731 incomplete pack file (will be removed): 3815790c5bd607714f7ec2b40d901825bfd89c6cf83b404462f1db38c014eaac incomplete pack file (will be removed): 381c4ba84fec2535409b1f8dde7f0e8b687c8cb84eb5f7beed426f624a53dc56 incomplete pack file (will be removed): 3890d392e9acd74b88d683e610c9c1fea39cde3493a7a0074a7bd063b1c79b25 incomplete pack file (will be removed): 3d84b6af81f09a6a807711af94209045672a19f2b2fdcb0cca138f03463bd9d2 incomplete pack file (will be removed): 3dc60bd4b5f2491ff1dd962d5916a91cf00bf7853ef17a74ce979c51bef99109 incomplete pack file (will be removed): 3ed56a5715f38faece78de71f52ffc7534cc7036ace10213a0597d6644b9e109 incomplete pack file (will be removed): 417404af54befd23635b4fab29bedc2eec6cf73a4b9ca05fc2360743419d2860 incomplete pack file (will be removed): 41a74edbc62aecd6ce67c2e940a79332894d284113bf549b4f7f1f6747a0ebc5 incomplete pack file (will be removed): 46c35c2635604158eb44b0f4a842b2bce705fb9356051a6b4563ded54c4b9d9b incomplete pack file (will be removed): 4eec32695a417b85f1dbb232638f9dcfd4f3ea04b91a66ed3b6a4e5f4f673b95 incomplete pack file (will be removed): 5224be80d6c6a5916a0b05dcd8e56adbbd42a632e8f3a3cefc7ca84a941d394d incomplete pack file (will be removed): 541c3b8a536421fdf47354ccf44e5c42d195f72f38ccaeead5689bad84e239c8 incomplete pack file (will be removed): 59cdc4171372722f73e0655465c9fbad6ded03fe4f27890a3086db61fef6d487 incomplete pack file (will be removed): 5a2108af106a725a4fb65c60a17d982555863ae93ce67c3f484523ea7783ed3d incomplete pack file (will be removed): 5f904e258f5ba6d77e7e86c15bf1b2d13efc7424f1b4b413bbdd7f054c65c362 incomplete pack file (will be removed): 6304d8bd754c02afd7907c088b4d9adf6c964794ecbc68be6c16cc004bda42ed incomplete pack file (will be removed): 63e8c5aae875bc3e999cd4fed5dfe9829297cbd26ad75bba7acdddcc6a7bf5ce incomplete pack file (will be removed): 67a05bbf2433017a8e34936833a74ed64a62b9c4864a67609ec4448a8ed6cc53 incomplete pack file (will be removed): 698cefafbe975a201728fd1959031a39d0280ccfe34e67cc90528e3fd45f1aa4 incomplete pack file (will be removed): 6ad81487b76e98c7948709639e6c2c070c475d195fb9132d79decf40f568488a incomplete pack file (will be removed): 6f37b250e92bf52f54a9708929599980c12d20567815f4dc1d1314460b084d30 incomplete pack file (will be removed): 71111854d066d9f35249d89f19be1f4a69877993aee4f3c28d09c5739153951f incomplete pack file (will be removed): 713b4400489791690a8eb0e0d131093e8b896e203e52d00c73181fba2ec39417 incomplete pack file (will be removed): 72109bfbac6a220b3949fa05d46f4d36cbe751d383ff6c9d89eb3941ccaf4adb incomplete pack file (will be removed): 76cda699c1580a2a3514ae89d2f7411c34da56f39e0cceca175860eeffd78bde incomplete pack file (will be removed): 78486aed1a6d0b2f1db329e3f5fa9a13f9ee2043243cf0ff1d70ec056484f4c0 incomplete pack file (will be removed): 789fc09283978594ba06e02f7d0cd640fdec9cc96ac7a1337dfe20b6fbad9725 incomplete pack file (will be removed): 7bc1d898bcfcffbbb589a3082cb4e10abe5edd76fd3d571daa4b4cbbca2c00ad incomplete pack file (will be removed): 805e466710f69a9b305921e48800224dec554528303b187e455b85423eb3c4bd incomplete pack file (will be removed): 82cec18cb57cdedc2b46d18a8eac83f39df84314e6aa9b71327f5dec84114a08 incomplete pack file (will be removed): 8459f2453ad3131697572a9b11836138b26bc23bc43ef522269d3b986173f475 incomplete pack file (will be removed): 86190d264a939d0f7a9899c83c8a2306a0d2c0e83803cc57361c90b02420e5e5 incomplete pack file (will be removed): 886c697ea210840a0624681c21fb0a2d29293e05e93559bbd3c8f9a42d4d18ff incomplete pack file (will be removed): 8a3560e5a2f6dcae8ca0125060a68ec11167da8b54ff5978ad09d7ffe9313e0b incomplete pack file (will be removed): 8cf6d352e40658001f9d6bf3d5fb75675f14f2825a7b8c002a3b958b04c2124c incomplete pack file (will be removed): 975a69eb97c5d135435d1b8019859b3df38be2a3c521051b813f35d1e7b73f6e incomplete pack file (will be removed): 9c174d223a834afb4925f3e389403d5697fa18e5bdebd4b24c4e03919c949cdd incomplete pack file (will be removed): 9e37702ff92319481fac6e95955a4da9246e28745bf85556efc8780ae36c33c9 incomplete pack file (will be removed): 9f4ef6a6304fc67ad4cbf544a733972bc2eef8ef56eacb110457b214e9a14677 incomplete pack file (will be removed): a3fb101ccd4533811cb2ca112993b9f3addfc3c7ed349dd19759e4cf357646a4 incomplete pack file (will be removed): a7b4107175f24445ed94e47bc93a2a24e00dd444aa56ac2343a852e820eafddd incomplete pack file (will be removed): a96c7bb7a2f423fcac88e5c36e0daf34337477a36f7168134469adcd0803a6fb incomplete pack file (will be removed): a9d1d00c34113b3bd79f118db2db9d0a7a7dbe9095a4d62de50d007d27f2a5da incomplete pack file (will be removed): a9d3a297f49ca3f32d301364dba1dfe0ef2ddeb2f3c86c6864d7cb1a15b04c8d incomplete pack file (will be removed): ad3bd2f5c1aae6058a32f0426253682a78aa8440b42b9af21c29b9f8c33fc1b1 incomplete pack file (will be removed): aea10ad0a163d55545edeec12e1c98321deffc4f99e57f1226458e4469b8d9c4 incomplete pack file (will be removed): aff4022647096208c6bbc2cc4befdfea895b53f570b9bf2ea93c83b257cd108e incomplete pack file (will be removed): b46ff951fb1dc256eb0fccacd10f6f69ab66eda47ac950ac5d9a82a0ccdb3547 incomplete pack file (will be removed): b477e4dcbd04e49566d69489ff4061d2aaf0d334744ee55e75dba28f91235d13 incomplete pack file (will be removed): b65792a41149f4d3e949f981891e32340881315eb175839ceae0dc92f5cdd0cb incomplete pack file (will be removed): b8060c1008d2aaaeae5a5027827640c6aef700291de02dde9d066b15ef718dc2 incomplete pack file (will be removed): bdbb22919d0171dbe252b15991447eb5f009e848eec38aba9fac8d123f7c5232 incomplete pack file (will be removed): bea3239af13f2d90872e61c39b4c6ea23304b8c80d4753686486e82129757e8b incomplete pack file (will be removed): c00c82a66126e687aa7c2e0cc34f786eaf9d57f89f3c9dad442d4844c2df1b45 incomplete pack file (will be removed): c050bebc36c2fd53ed80bd549bb53d73a6a5638451d018374a1c71255acf6f97 incomplete pack file (will be removed): c3d64052dabccedeae502980cdbab6c12b76a6596a726b9b03e18d70883c7960 incomplete pack file (will be removed): c868e8a7d63de269039097b86bff0b753db33c26ff51eb242c868bcc7223fafa incomplete pack file (will be removed): cc2ea735e9582bc18a4f2da6cd938cdce908d7ca0b5b20c72f9046d4dcae6b19 incomplete pack file (will be removed): cdede16d9f140ffb408b017888b197a78154bc146d1e39e35c6085677141c44b incomplete pack file (will be removed): d3d49b55275ee753bf18d399c2a6d8a3b0f47ecbe0e550dbc11afd0efe72bc7d incomplete pack file (will be removed): d45a938eb9060b07cfa1cd2d16a1600a79f600148e076b4230f4f18c1ecd9507 incomplete pack file (will be removed): d45e47b2eefe2d8c3038d7ea46642d1c44b508947665cbddc20ac9a1c700f3f5 incomplete pack file (will be removed): d7ec45ce20ea4939b329cd417f728e1b10308603bf6d324c96d75c427b331555 incomplete pack file (will be removed): d904a7df8d69fe7497d44ed13080428f53c02b567445a4eaea29f069f3a45737 incomplete pack file (will be removed): d997e4fa0ab82a37fd18284c485a2ba5002b093f207487f6ed7aed8c2521483d incomplete pack file (will be removed): da32c9575f7679c6dd687dad5f2cff200339a38dfc8ac5347f494bc67b4371eb incomplete pack file (will be removed): da5d4e53047cd55143f27f40aa4b14ac780c44318963b8320c4f7fa31b976366 incomplete pack file (will be removed): e39078b7208398de034c73a61ea099de9c2cd2fc464ec8daa9c38ff78e4e17a5 incomplete pack file (will be removed): e40b21bdd5e1453fbd7f1a19fe151759fedaff1aa196abfca3bc03166ec647b7 incomplete pack file (will be removed): e44a4284f4813d2eb5802e4ce77602011b8e54642270635a511bbffd9f3ef974 incomplete pack file (will be removed): e44fae91853988b23cbe9c7b5c1137eaf9b161edba1c9308a6c657d478db94ad incomplete pack file (will be removed): ea10fb981f253c2f226e058921e87bf080098a62f9a83c1c53b9e4a0b45cad4a incomplete pack file (will be removed): ea93cef1b9fd631d3358889be47b9c330503e1f06c85da67c36bcd2869e57a9e incomplete pack file (will be removed): ee098c5d2c55a8176eefe74945600c2fac1d8a1ee8c2c7f53303be97b144186b incomplete pack file (will be removed): f06c2ae5277c691ebaf0094fde7f9643ee4a9cc6c0faf32b3abd758fd07c3013 incomplete pack file (will be removed): f09fe8a3b80bc7ef8015a779870665f760ebf0ee15997306596fa5b629ac1841 incomplete pack file (will be removed): f5988f5d60cac08ed5c28b0dc21f18525cc728705f548364c49eef3d88ef5a70 incomplete pack file (will be removed): f64ffcb7e138502f15f46be46ce79b980317aefb42cf796a0ab9250e6efcdbf7 incomplete pack file (will be removed): f7502035a0b1594c37a6c7e0d7559c3c62e314ed990ad31974e061abc75fa10e incomplete pack file (will be removed): f9ffa71cd1ef2bf687c3f5110339ea98f989acad60f01640c12a7820fdda6184 incomplete pack file (will be removed): fafcb28d8a5f0b50c1daa26664c78988a9e6124a97de5984e6f95f25d95beb72 incomplete pack file (will be removed): fcbb8fe47d57b19a83b7c8ca2603c9927a9734c9887300e9d14f340f5154d7d0 repository contains 115023 packs (2075648 blobs) with 550.713 GiB processed 2075648 blobs: 14938 duplicate blobs, 2.430 GiB duplicate load all snapshots find data that is still in use for 611 snapshots [9:49] 100.00%  611 / 611 snapshots found 2051299 of 2075648 data blobs still in use, removing 24349 blobs will remove 105 invalid files will delete 406 packs and rewrite 1326 packs, this frees 5.713 GiB hash does not match id: want f5aaf682ad9b89ed6a632aac0765204ee3c6815fd62b8f5abac963e65603cb68, got e2a1de6225d63265d579ee466215321db6f33f6122dbfad902424b6c52e5e655 github.com/restic/restic/internal/repository.Repack 	/restic/internal/repository/repack.go:34 main.pruneRepository 	/restic/cmd/restic/cmd_prune.go:278 main.runPrune 	/restic/cmd/restic/cmd_prune.go:85 main.glob..func18 	/restic/cmd/restic/cmd_prune.go:25 github.com/spf13/cobra.(*Command).execute 	/restic/vendor/github.com/spf13/cobra/command.go:762 github.com/spf13/cobra.(*Command).ExecuteC 	/restic/vendor/github.com/spf13/cobra/command.go:852 github.com/spf13/cobra.(*Command).Execute 	/restic/vendor/github.com/spf13/cobra/command.go:800 main.main 	/restic/cmd/restic/main.go:86 runtime.main 	/usr/local/go/src/runtime/proc.go:200 runtime.goexit 	/usr/local/go/src/runtime/asm_amd64.s:1337  brianr  ~  restic version restic 0.9.5 compiled with go1.12.4 on linux/amd64  brianr  ~ 

@brianread108 Can you please run sha256sum on the file data/f5/f5aaf682ad9b89ed6a632aac0765204ee3c6815fd62b8f5abac963e65603cb68?

 brianr  …  fedora-backup  data  f5  sha256sum f5aaf682ad9b89ed6a632aac0765204ee3c6815fd62b8f5abac963e65603cb68  e2a1de6225d63265d579ee466215321db6f33f6122dbfad902424b6c52e5e655  f5aaf682ad9b89ed6a632aac0765204ee3c6815fd62b8f5abac963e65603cb68  brianr  …  fedora-backup  data  f5 

The data was being written to a mounted CIFS drive on my Centos Server.
I think it is possible that I ran an unlock and  prune while it is was still backing up!  Not the best idea I guess.
I've moved the repo onto Dropbox now (using rclone) so I'll see if it happens again.

Hi all,
I recently encountered a similar error. The steps and tools listed earlier in this ticket were instrumental in my being able to track down the source of the problem (which was hardware), so thanks to @fd0 and everyone else for documenting all of this.
As a user, it would be fantastic if the "restic debug xxxx" commands were merged into a mainline build. While I didn't have much trouble creating a branch build to get at the tools (thanks also for putting together the Docker-based builder!), I can imagine that more casual users might not want to go that far.
My symptoms were identical to what was posted earlier in the ticket: the sha256 of the pack file did not match the filename, and the "restic debug examine" command was able to show me that one of the internal blobs was corrupted but dump out the corrupted blob data anyway. By re-adding the same file to the repo, I was able to generate a non-corrupt copy of the same blob, and then by passing each good/bad blob through xxd and comparing as suggested above, I was able to see that exactly one bit was flipped between good and bad.
This was the smoking-gun information I needed in terms of tracking it down to a likely memory error (vs. network vs. disk, in a fairly complicated setup with a bunch of moving parts), so a few swapped memory modules later, I sorted out the source of the problem and got everything working without further errors. I wouldn't have been able to diagnose this very easily without the branch-1999 tools above.
I am a recent refugee from Borg, so I also was pleasantly surprised to see that this one-bit error only mangled one bit of my plaintext in Restic, whereas one of my comparable Borg archives seems to have gotten completely hosed (you can find a ticket or two floating around along the lines of "single byte error causes entire Borg archive to be unreadable"). Although I like Borg's compression, I like retaining the ability to access to my data even better. Restic's resilience in this regard could make a good marketing point.
Interestingly, although I was unable to make the memory fail on the machine using multiple rounds of Memtest86 or even by running mprime for half a day, asking Restic to ingest 60 Gb of data was pretty consistent in producing a bit error or two within five minutes. Although maybe you don't necessarily want to market Restic as a stress-tester...

Hi,
As a follow up, this has been my observation too. In my earlier post I was getting the same hashing errors. I eventually swapped out the entire set of DIMMs on my mobo for a newly bought set, ran memtest (a few times - that took a while!) with no errors reported (my previous DIMMs were showing some memory errors).
After a few initial successes with restic (backed up onto a ZFS raidz2 configuration - so that includes checksumming), I started to get the odd hashing error as before.
Unfortunately, this has made me abandon restic as a backup solution for now. Perhaps I have to invest in ECC memory (and everything that entails) in order to gain confidence again.

@dharrigan Note that memory is just one place that bit flips could be occurring. The NIC is another possible culprit. See here for example.

Hi @smlx
Thank you for that helpful link. However, in my case, this is all local to my machine - no network traffic. What you say could be applicable to others perhaps? Agreed, good to bear in mind!

I also had a hash does not match id issue with prune this week; a debug build with --try-repair succeeds, reporting there was a bit flipped:
debug enabled
repository e113796b opened successfully, password is correct
examine f58c821357bb556890c324bc2e84f72464d3f2f2576762f858059f6a917cb782
  file size is 4392684
  wanted hash f58c821357bb556890c324bc2e84f72464d3f2f2576762f858059f6a917cb782, got 77d7cc5b9232c3e05a1410f3ef767c561b31148b39ca4b6f5a41cc4a8cc703d8
  ========================================
  looking for info in the indexes
    index 6906d6c8:
      data blob d29725dfb8d426f37616c405ae50c0395822d8730803477cfb9da2e9da98957a, offset 0     , raw length 534745
      data blob 0b13521afc8d216dc600c240ae62fcf42cb5fa1bc210b3e8f7650bcd1c1072a6, offset 534745, raw length 524320
      data blob ba5a4cdf979a2a96c72838d412850abc8f320e2a273562b812394f928e245fa4, offset 1059065, raw length 532837
      data blob 7c8c46df2cb1cd55278a43bc66ffc0e613667322001ef63165570aff1e1ccc1c, offset 1591902, raw length 553139
      data blob 36f649014f6796a03934f6433153366b6a8ec769fca7f12a4b991fb9f9fe49df, offset 2145041, raw length 531989
      data blob c37791de5000d9aa3f2bf0eaf843c54ec4af5f2822d174d78008e09c9c769f99, offset 2677030, raw length 539279
      data blob ac65e39848689d900120105ef1f962971814bf4b6218d7eeedbaee6925b829f6, offset 3216309, raw length 567348
      data blob 7e1d7c3401d5a0a6b87c4a92e7f930e52f8a3ff1fc9f26fdd1cf0ef816957227, offset 3783657, raw length 608695
      file sizes match
      loading blob d29725dfb8d426f37616c405ae50c0395822d8730803477cfb9da2e9da98957a at 0 (length 534745)
         successfully decrypted blob (length 534713), hash is d29725dfb8d426f37616c405ae50c0395822d8730803477cfb9da2e9da98957a, ID matches
      loading blob 0b13521afc8d216dc600c240ae62fcf42cb5fa1bc210b3e8f7650bcd1c1072a6 at 534745 (length 524320)
         successfully decrypted blob (length 524288), hash is 0b13521afc8d216dc600c240ae62fcf42cb5fa1bc210b3e8f7650bcd1c1072a6, ID matches
      loading blob ba5a4cdf979a2a96c72838d412850abc8f320e2a273562b812394f928e245fa4 at 1059065 (length 532837)
         successfully decrypted blob (length 532805), hash is ba5a4cdf979a2a96c72838d412850abc8f320e2a273562b812394f928e245fa4, ID matches
      loading blob 7c8c46df2cb1cd55278a43bc66ffc0e613667322001ef63165570aff1e1ccc1c at 1591902 (length 553139)
         successfully decrypted blob (length 553107), hash is 7c8c46df2cb1cd55278a43bc66ffc0e613667322001ef63165570aff1e1ccc1c, ID matches
      loading blob 36f649014f6796a03934f6433153366b6a8ec769fca7f12a4b991fb9f9fe49df at 2145041 (length 531989)
         successfully decrypted blob (length 531957), hash is 36f649014f6796a03934f6433153366b6a8ec769fca7f12a4b991fb9f9fe49df, ID matches
      loading blob c37791de5000d9aa3f2bf0eaf843c54ec4af5f2822d174d78008e09c9c769f99 at 2677030 (length 539279)
         successfully decrypted blob (length 539247), hash is c37791de5000d9aa3f2bf0eaf843c54ec4af5f2822d174d78008e09c9c769f99, ID matches
      loading blob ac65e39848689d900120105ef1f962971814bf4b6218d7eeedbaee6925b829f6 at 3216309 (length 567348)
error decrypting blob: ciphertext verification failed
        trying to repair blob with single bit flip
         spinning up 2 worker functions
538565 byte of 567348 done (0.01%), 435 byte per second, ETA 1m6sss
        blob could be repaired by flipping bit 3 in byte 538769
        hash is ac65e39848689d900120105ef1f962971814bf4b6218d7eeedbaee6925b829f6
     done after 20m39.199734992s
decrypt of blob ac65e398 stored at ac65e39848689d900120105ef1f962971814bf4b6218d7eeedbaee6925b829f6.bin
      loading blob 7e1d7c3401d5a0a6b87c4a92e7f930e52f8a3ff1fc9f26fdd1cf0ef816957227 at 3783657 (length 608695)
         successfully decrypted blob (length 608663), hash is 7e1d7c3401d5a0a6b87c4a92e7f930e52f8a3ff1fc9f26fdd1cf0ef816957227, ID matches
  ========================================
  inspect the pack itself
      data blob d29725dfb8d426f37616c405ae50c0395822d8730803477cfb9da2e9da98957a, offset 0     , raw length 534745
      data blob 0b13521afc8d216dc600c240ae62fcf42cb5fa1bc210b3e8f7650bcd1c1072a6, offset 534745, raw length 524320
      data blob ba5a4cdf979a2a96c72838d412850abc8f320e2a273562b812394f928e245fa4, offset 1059065, raw length 532837
      data blob 7c8c46df2cb1cd55278a43bc66ffc0e613667322001ef63165570aff1e1ccc1c, offset 1591902, raw length 553139
      data blob 36f649014f6796a03934f6433153366b6a8ec769fca7f12a4b991fb9f9fe49df, offset 2145041, raw length 531989
      data blob c37791de5000d9aa3f2bf0eaf843c54ec4af5f2822d174d78008e09c9c769f99, offset 2677030, raw length 539279
      data blob ac65e39848689d900120105ef1f962971814bf4b6218d7eeedbaee6925b829f6, offset 3216309, raw length 567348
      data blob 7e1d7c3401d5a0a6b87c4a92e7f930e52f8a3ff1fc9f26fdd1cf0ef816957227, offset 3783657, raw length 608695
      file sizes match

I'm a bit confused on what do do next; I had suspected that the issue was resolved, but if I run prune (or cat blob ...) it crashes trying to decrypt the same ac65e398 blob.
$ restic -r ... cat blob ac65e39848689d900120105ef1f962971814bf4b6218d7eeedbaee6925b829f6 > ac65e39848689d900120105ef1f962971814bf4b6218d7eeedbaee6925b829f6.orig
debug enabled
enter password for repository:
decrypting blob ac65e39848689d900120105ef1f962971814bf4b6218d7eeedbaee6925b829f6 failed: ciphertext verification failed
github.com/restic/restic/internal/repository.(*Repository).loadBlob
        internal/repository/repository.go:166
github.com/restic/restic/internal/repository.(*Repository).LoadBlob
        internal/repository/repository.go:627
main.runCat
        cmd/restic/cmd_cat.go:175
main.glob..func4
        cmd/restic/cmd_cat.go:24
github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:800
main.main
        cmd/restic/main.go:86
runtime.main
        /usr/lib/go/src/runtime/proc.go:203
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1357

I thought perhaps there were multiple bits flipped in the same blob, but I've run examine --try-repair twice now, and it reports successfully flipping the same bit at the same byte, so I wonder if maybe I've misunderstood some of the instructions. Do I need to manually remove this file from the data folder, or perhaps manually overwrite it with the repaired .bin file?
I also just wanted to say thanks for such a great team around restic. This thread is full of strong communication and helpful attitudes. Thank you to @fd0 and the rest of the restic team!

A more helpful error message would be nice.
I got this "hash does not match id" error from "restic forget" on two consecutive nightly backup runs. sha256sum computed the same hash as Restic did. After a reboot the next backup ran without error, and now the problematic file is gone, presumably pruned. So in my case it seems that Restic was not at fault, and the file on disk was fine, but the disk cache was corrupted somehow, despite ECC.
Although the problem was easily solved once I understood it, it took a considerable effort to get to that point. I'm a software engineer, I know what a hash is, I recognize a stack trace when I see one, and I still had to serch for the error message and read this and other discussions. A non-programmer would be completely lost.
Given that this error apparently happens at times for reasons outside of Restic's control, it would help if Restic would explain the situation and suggest some possible remedies instead of giving an error message that only Restic developers can understand. Maybe point to a troubleshooting guide more concise than this long thread?

I agree that something should be done about error messages. Either improving them by providing helpful pointers along with them (which might however be a bit too verbose, unless put at the bottom of the output), and/or a page in the documentation that sums up the different messages there can be, and what one is supposed to do about them.

A little over a year later and I'm in the same situation again. This time I've moved to ZFS and replaced the RAM; either way, I get the crash with prune, and I'm left with a blob that has a single bit flipped and looks like it can be successfully repaired with debug examine --try-repair.
But as above I still don't know what to do with the resulting repaired-abcd.bin, where abcd is the hash.
I tried finding in the repo and can't find a matching file with that hash.
Currently running restic find --blob abcd, but not really sure what to do with the results.
Is there someone that would be so kind as to walk me through what to do with repaired-abcd.bin (and damaged-abcd.bin for that matter)? Pointing me to an instructive issue or documentation page would be adequate! Thanks in advance.

@n8henrie The debug-1999 branch has gained the functionality to extract all intact parts of the damaged pack file, in the meantime. With that it's actually possible to recover (well, actually replace) a damaged pack file. As I haven't posted usage instructions here, here's a complete walk-through:

Build the latest debug-1999 branch (I've updated it a few minutes ago) using go build -tags debug ./cmd/debug
Run restic debug examine --extract-pack <id-of-broken-pack-file> to extract all blobs in the damaged pack file into the current folder (-> it's best to use a temporary folder to reduce the chaos). The packfile must still be located in the proper place inside the repository, but it is not necessary for it to be contained in the repository index (i.e., there's no need to run rebuild-index, please do not run that command right now) The option --try-repair can be used to try to repair a single bit flip, and --repair-byte tries to repair a full byte at once (but is also much slower). The examine command won't modify the repository, so it's safe to run even though it's experimental code.
From here on you can use a release version of restic if you feel safer with that.
Move the damaged pack file out of the repository, create a backup of the index and run restic rebuild-index (which should work fairly when using the current master branch).
Then run restic backup . in the folder which contains the extracted blobs. This let's restic recover the blobs which were previously contained in the damaged pack file.
Now with a bit of luck restic check --read-data should no longer report errors or only a few ones.
If the (still) missing blobs are part of a file of which you still have the original, then run restic backup <original-file>, which will recover the missing blobs. (With restic before 0.10.0 you have to add the --force flag).

@MichaelEischer thanks very much.
Since it seems that --try-repair restores a blob, I was trying to figure out some way to just delete that blob from the repo, or otherwise to backup --force to overwrite the existing blob.
Per your instructions I was able to get the entire repaired pack with --extract-pack, manually mv the existing pack elsewhere, rebuild the index, and then back up to restore the (repaired) pack.
Unfortunately I had to repeat the process a few times, and even after the prune eventually succeeded without errors, I'm now partway into a multi-day check --read-data and can see multiple Pack ID does not match warnings already.
I'm really hoping I can trace these packs back to before I replaced my ram and went to ZFS!
Thanks again for your help, I really appreciate your time and thorough instructions.

So I just ran into this as well and wanted to report my environment and scenario.
I'm backing up both a MacOS machine and a ZFS linux box (w/ ECC RAM) to the same B2 bucket (US region). I encountered the error while running prune on the ZFS linux box. Prior to encountering the error I did an unlock, forget, and check, which completed successfully. Backups were completed using prior versions of restic and go.
This is the ZFS linux box:
$ sudo restic self-update
writing restic to /usr/bin/restic
find latest release of restic at GitHub
latest version is 0.11.0
download SHA256SUMS
download SHA256SUMS.asc
GPG signature verification succeeded
download restic_0.11.0_linux_amd64.bz2
downloaded restic_0.11.0_linux_amd64.bz2
saved 19128320 bytes in /usr/bin/restic
successfully updated restic to version 0.11.0

restic version
restic 0.11.0 compiled with go1.15.3 on linux/amd64

$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name forget b12f3f1e 57aa5805 5c4d0673
enter password for repository:
repository 01618c22 opened successfully, password is correct
unable to create lock in backend: repository is already locked by PID 1820837 on pluto by restic (UID 1003, GID 1003)
lock was created at 2020-12-03 22:55:41 (76h41m49.81627843s ago)
storage ID 001bf1a8
the `unlock` command can be used to remove stale locks

$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name unlock
enter password for repository:
repository 01618c22 opened successfully, password is correct
successfully removed locks

$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name forget b12f3f1e 57aa5805 5c4d0673
enter password for repository:
repository 01618c22 opened successfully, password is correct
[0:02] 100.00%  3 / 3 files deleted

$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name check --read-data-subset=1/100
using temporary cache in /tmp/restic-check-cache-467689704
enter password for repository:
wrong password or no key found. Try again
enter password for repository:
repository 01618c22 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-467689704
create exclusive lock for repository
load indexes
pack e3824730 contained in several indexes: {d89d435c f6976318}
This is non-critical, you can run `restic rebuild-index' to correct this
check all packs
pack 502d145f: not referenced in any index
... (many other packs) ...
pack 59803289: not referenced in any index
225 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
read group #1 of 4757 data packs (out of total 405521 packs in 100 groups)
[30:05] 100.00%  4757 / 4757 packs
no errors were found

$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name stats
[sudo] password for craig:
enter password for repository:
repository 01618c22 opened successfully, password is correct
scanning...
Stats in restore-size mode:
Snapshots processed:   19
   Total File Count:   15508830
         Total Size:   2.786 TiB

$ sudo -u restic --preserve-env=B2_ACCOUNT_ID,B2_ACCOUNT_KEY restic -r b2:bucket-name prune
[sudo] password for craig:
enter password for repository:
repository 01618c22 opened successfully, password is correct
counting files in repo
building new index for repo
[13:20:18] 100.00%  405746 / 405746 packs
repository contains 405746 packs (6223816 blobs) with 2.023 TiB
processed 6223816 blobs: 259070 duplicate blobs, 95.416 GiB duplicate
load all snapshots
find data that is still in use for 19 snapshots
[4:09] 100.00%  19 / 19 snapshots
found 5885670 of 6223816 data blobs still in use, removing 338146 blobs
will remove 0 invalid files
will delete 4105 packs and rewrite 47581 packs, this frees 116.876 GiB
[22:53:25] 57.66%  27435 / 47581 packs rewritten
hash does not match id: want 5c0665243f13205107c7bbcd5089e8b36b7144f837692836a70c4fa7b1cb9f9e, got cb1ffeb35fcfb99d42ec9220259f4f4819e3c8df34b6dffda98344feb3552eb2
github.com/restic/restic/internal/repository.Repack
        /restic/internal/repository/repack.go:37
main.pruneRepository
        /restic/cmd/restic/cmd_prune.go:242
main.runPrune
        /restic/cmd/restic/cmd_prune.go:62
main.glob..func19
        /restic/cmd/restic/cmd_prune.go:27
github.com/spf13/cobra.(*Command).execute
        /home/build/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
        /home/build/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
        /home/build/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
        /restic/cmd/restic/main.go:98
runtime.main
        /usr/local/go/src/runtime/proc.go:204
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1374

Next... attempting remediation with the debug build. Note I tried running go build -tags debug ./cmd/debug and got build ./cmd/debug: cannot find module for path ./cmd/debug.
Prune had been running for about 24 hours before it panic'd for me, and so it's working its way through a second invocation. Does it make any sense to add a mid-level retry to try to distinguish between permanent and transient bit errors?
I'm hesitant to think this is hardware. While old, the machine has a decent Supermicro motherboard and ECC RAM. ZFS scrubs have never discovered any bit errors, and there have been no recent hardware changes. The server has a decent UPS that should kick in for low-voltage conditions. There may be multiple root causes that include hardware, but others in this thread have reported running mprime and memtest without issue, and I think those are going to be harder stressors than restic.
Just trying to look for a common thread here, we have Verizon Fios and use a Ubiquiti EdgeRouter X running software v1.10.7. The NIC is an Intel 82579LM.
$ uname -a
Linux pluto 5.3.1-arch1-1-ARCH #1 SMP PREEMPT Sat Sep 21 11:33:49 UTC 2019 x86_64 GNU/Linux

$ ethtool -i enp0s25
driver: e1000e
version: 3.2.6-k
firmware-version: 0.13-4
expansion-rom-version:
bus-info: 0000:00:19.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

$ lspci | grep -i ethernet
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 05)
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

Thank you everyone for the great tool and support!

@cyounkins I've fixed the build instructions. Thanks for finding the typo.
restic currently doesn't retry downloads when a bit-flip might have occurred. As the file seems to be intact on the second run, the cause is probably some transient error somewhere between the B2 server and your server's CPU. AFAIK the connection to B2 uses HTTPS which should catch any bit-flips during transmission.

Update: A second invocation of prune with no changes to the repository completed successfully, so it appears to have been a transient error when reading the repository on the linux box.
However, I did get 1 additional files were found in the repo, which likely contain duplicate data. immediately after running prune, which seems weird. More details in #3161.
Possible sources of corruption in my situation:

B2 sent the wrong data due to corruption on their side
HTTPS layer processing
Restic processing
If swap was used (unlikely), full disk IO path (not on ZFS)
CPU, RAM

AFAIK the connection to B2 uses HTTPS which should catch any bit-flips during transmission.

That looks to be correct. I now agree the source of corruption is unlikely to be the NIC or intermediate network devices.
One interesting thing I saw on the Arch Wiki on stress testing was that when power-saving features are enabled, corruption may be seen with medium-power applications but not high-power applications. The CPU may be more stable running at full throttle during a stress test than during normal operating conditions.

While I'm not able to offline the machine to run memtest86, I was able to run other tools and found no hardware issues.

mprime blend torture test for 1 hour
linpack using 50% RAM for 30 mins
stressapptest -s 300 -M 20000 (20/32GB test for 5 mins)
stressapptest -s 300 -M 20000 -C (20/32GB CPU-stressful test for 5 mins)
stressapptest -s 300 -M 20000 -f sat1 -f sat2 (stresses disk IO path used for swap)

The kernel logged no hardware errors. While there's still the chance there was something going on with power at the time of the issue, these tests suggest the problem more likely lies in software or the B2 service.

I wanted to follow up on this thread to point others to a potential cause that affected me - a kernel bug affecting some builds before versions 5.3.15, 5.4.2, and 5.5 that can cause CPU register corruption. More details are available in my thread on the forum.

I've been running into this problem as well, which is why I found the issue. In my case, the cause likely is an interrupted backup run on the repo.
To make restic more resilient against such corruption issues, would it be possible to optionally store extra parity information for each repository file in order to enable reconstruction via erasure codes, e.g. using Reed Solomon ?
E.g. using this module:

https://github.com/klauspost/reedsolomon

Many thanks for a great tool!

I've been running into this problem as well, which is why I found the issue. In my case, the cause likely is an interrupted backup run on the repo.

@malemburg The only reason I can imagine how an interrupted backup would be able create contents where the hash does not match the ID is a truncated file in the repository. Here, an erasure code would not help. (There is also #256 tracking that request). Can you give more details about your case? If you get "hash does not match ID" errors I don't think this is due to abborted backups.

The interrupted backup is the only possible interpretation I have from reading the ticket. I know that this happened in that particular repo which had the hash error. Whether this resulted in a truncated file on the server, I cannot say, only that the file SHA256 hash did indeed not match the file name.
The erasure code would help with such cases, since Restic would be able to use recovery information from the parity shards to rebuild the broken file. Minio, for example, works this way: it uses RS and adds 50% parity information to the stored data, making it very robust against such corruption.
Thanks for pointing me to the already open ticket for RS support. I was not aware.

I've encountered the same error with a newly started repo for the notebook in question. As in the OPs situation, I am too using SMB for mounting the restic repo on the Windows notebook. The SMB share is coming from a Linux server using Samba 4.11. Perhaps the problem only occurs with such mounts ?!
This time around, I'm creating a copy of the repo for further inspection.
Some more information: It is likely that the backup runs on the notebook were terminated by the Windows 10 scheduler, since they took too long. This and the connectivity problems with the restic mount on the notebook could explain truncation of files.
How are such files written by restic ? I would assume that the data is first written to a temporary file, which is then renamed to final file name upon completion. If that's the case, it's difficult to understand why truncation due to forced shutdown or network issues could happen after finalization.
The server cannot be causing this, since it's being used by several other machines for backups as well and those don't show any errors.
Anything I could do to help debug this ? Thanks.

@malemburg To better help us debug the issue, please provide as much information as in the new issue template. Include the version of restic, the exact invocation, execution log, and error message. Try to take some investigative steps referenced in this thread like debug examine or manually hashing the file.
The most common cause of Hash does not match id during prune seems to be hardware issues. Run Prime95 blend torture test and memtest if you can.

@malemburg , @cyounkins I've moved the problem analysis into a new issue #3435. It's better to create a new issue for prune issues first to avoid interleaving multiple discussions and then later on merge the ones with the same root cause. Most of the time it is not obvious at first, what the real problem is.

By now a whole bunch of commands exist to repair broken snapshots: https://restic.readthedocs.io/en/stable/077_troubleshooting.html . Restic now also includes several layers of internal integrity checks to catch data corruption even before data is stored in the repository, for example https://github.com/restic/restic/issues/4529 and the upload hashes. Handling of bitflips during decoding/decryption will be improved by https://github.com/restic/restic/pull/4800 . So I guess we can finally put this issue to rest :wave: .
Adding redundancy to heal data corruption during storage is already tracked at https://github.com/restic/restic/issues/256 .