On Nov 10, 2012, at 4:55 AM, Jim Klimov <
jimklimov <at> cos.ru> wrote:
> On 2012-11-10 04:55, Garrett D'Amore wrote:
>> unlikely that any SHA-256 collision will be found in my life time,
>> unless there is a deeper flaw found in the SHA algorithm itself. I'm
>
> Then again, I found and George Wilson (I believe) fixed a bug
> in ZFS itself which caused problems with unverified writes on
> a deduped pool. So even if the hash collisions are not likely
> mathematically, there are situations which look the same and
> dedup verification is not something "too paranoid"- even if
> only to catch ZFS bugs.
>
> To recap the "anecdotal reference", my userdata got corrupted
> somehow so that some blocks from several files could not be
> recovered by raidz2 over 6 disks (I still have little rigid
> idea what went wrong, and I think I have some of those corrupted
> files left dormant on my old box so I could try to research
> their sectors manually some day).
>
> While the block-pointer tree walk (like scrub or normal file
> I/O) did find the presence of an error - a block did not match
> its checksum - the entry in DDT for this checksum existed and
> was not invalidated because of scrub's findings.
>
> Further unverified writes into the pool just incremented the DDT
> counter and new files (restored from backup) remained corrupted.
>
> Writes with verification did find that the on-disk blocks differ
> (they still did not care that one of them actually mismatches
> the checksum) and decreased the DDT counter for "bad block" and
> allocated a new unique one for valid data. I believe that for
> such collided entries as recovery blocks with valid data, new
> normal unique blocks are allocated and never deduped.
>
> Ultimately, there was a panic (the cause of which was what George
> fixed in
https://www.illumos.org/issues/2649 ) because ZFS did
> not do proper housekeeping for such unique blocks, and they were
> still regarded as deduped - but not entered into the DDT. So the
> further writes into the "dedup=verify" file that tried to release
> the unique block from DDT (and which was not there) panicked.
>
> My other related reported bugs on this subject were
>
https://www.illumos.org/issues/1981
>
https://www.illumos.org/issues/2024
>
> My fix for not-panicking due to already having metadata corruption
> of this type, and just releasing the "non-DDT" deduped blocks when
> asked at a risk of "leaks" in the block tree was in the end the
> essence of my 2024, and ranks between cosmetic fix and a usability
> one (the storage system no longer panics in this case). I don't
> think it was integrated - or, after the discussions, if it should
> be - but the patch is on the issue tracker for those in need.
>
> The corrupted DDT entries were never addressed, to my knowledge,
> so it is still possible to have DDT point to unreadable garbage.
> Most of these "corrupted" blocks on my system were flushed when
> the DDT counter went to 0, after all copies of the file (block)
> were "recovered" from backup or other sources, or deleted.
>
>
>
> -------------------------------------------
> illumos-zfs
> Archives:
https://www.listbox.com/member/archive/182191/=now
So we're more than 245 million yottabytes of data before we have a probability of a SHA-256 collision that approaches a probability worth worrying about. To put that more clearly, a *yottabyte* is about a septillion bytes, or a trillion terabytes. Think about that -- nearly a quarter of a billion trillion terabyte drives would be needed to hold this much data. We're talking about truly vast amounts of data before we have to worry about a collision. In all likelihood, I now believe it is unlikely that any SHA-256 collision will be found in my life time, unless there is a deeper flaw found in the SHA algorithm itself. I'm much more worried personally about other natural causes extinction level events ending all life on earth than I am about a SHA-256 collision corrupting my data. (Not that I'm particularly worried about such events.)
RSS Feed