gmane.linux.kernel.device-mapper.dm-crypt

Massive Failure

Due to a combination of hardware failure, ignorance, and stupidity I have
managed to really make a mess of a 7 drive mdadm (raid5) -> lvm -> luks ->
ext3 setup.  So here is the basic setup:

1.  sudo badblocks -c 16384 -s -w -t random -v /dev/sd (on all 7 drives)
2.  sudo mdadm --create --verbose /dev/md0 --level=5 --chunk=256 --force
--raid-devices=7  /dev/sd[a,b,c,d,e,f,g]
3.  pvcreate /dev/md0
4.  vgcreate lg /dev/md0 -s 256M
5.  lvcreate -l22356 -nlv lg
6.  cryptsetup --verify-passphrase --verbose --hash=sha256
--cipher=aes-cbc-essiv:sha256 --key-size=128 luksFormat /dev/lg/lv
7.  cryptsetup luksOpen /dev/lg/lv encrypted
8.  mkfs.ext3 -j -m 3 -O dir_index,filetype,sparse_super,large_file
/dev/mapper/encrypted

So long story short, the hardrives were swapped around and one point which
cause blkid.tab to go crazy and remap the drives to different letters.  All
but one of the mdadm superblocks were erased, but I believe the array was
clean before this happened.  By using basically the same command first used
to create the array mdadm detected that it was already a raid array and
recreated it - unfortunately it thought that the last drive (sdh at this
time) was dirty, and thus began rebuilding it.  Unfortunately commands 3-7
were also reissued, which initially made me believe that the luks header was
destroyed (according to this thread:
http://osdir.com/ml/linux.kernel.device-mapper.dm-crypt/2005-12/msg00045.html).
After some time with dd and a hex editor I have found three luks headers on
the drives - two of which are the new header; however it is my hope that the
other one is the original luks header.

(Continue reading)

headers

Roscoe | 7 Aug 10:50

Re: Massive Failure

Can you use losetup on the header file then cryptsetup luksDump /dev/loopX?

On Thu, Aug 7, 2008 at 6:07 PM, Clayton Shepard <cws@...> wrote:
> Due to a combination of hardware failure, ignorance, and stupidity I have
> managed to really make a mess of a 7 drive mdadm (raid5) -> lvm -> luks ->
> ext3 setup.  So here is the basic setup:
>
> 1.  sudo badblocks -c 16384 -s -w -t random -v /dev/sd (on all 7 drives)
> 2.  sudo mdadm --create --verbose /dev/md0 --level=5 --chunk=256 --force
> --raid-devices=7  /dev/sd[a,b,c,d,e,f,g]
> 3.  pvcreate /dev/md0
> 4.  vgcreate lg /dev/md0 -s 256M
> 5.  lvcreate -l22356 -nlv lg
> 6.  cryptsetup --verify-passphrase --verbose --hash=sha256
> --cipher=aes-cbc-essiv:sha256 --key-size=128 luksFormat /dev/lg/lv
> 7.  cryptsetup luksOpen /dev/lg/lv encrypted
> 8.  mkfs.ext3 -j -m 3 -O dir_index,filetype,sparse_super,large_file
> /dev/mapper/encrypted
>
>
> So long story short, the hardrives were swapped around and one point which
> cause blkid.tab to go crazy and remap the drives to different letters.  All
> but one of the mdadm superblocks were erased, but I believe the array was
> clean before this happened.  By using basically the same command first used
> to create the array mdadm detected that it was already a raid array and
> recreated it - unfortunately it thought that the last drive (sdh at this
> time) was dirty, and thus began rebuilding it.  Unfortunately commands 3-7
> were also reissued, which initially made me believe that the luks header was
> destroyed (according to this thread:
> http://osdir.com/ml/linux.kernel.device-mapper.dm-crypt/2005-12/msg00045.html).

(Continue reading)

headers

Clayton Shepard | 7 Aug 11:02

Re: Massive Failure

Well its not exactly a "header file" I just dumped the first 4mb of each
drive with dd.  Using a hex-editor with search capabalites I was able to
find the luks magic ('L','U','K','S',xBA,xBE) on the same spot on three of
the dumps.  All of them were clearly luks headers and two of them
corresponded exactly to running luksDump on the new /dev/lg/lv (ie the salt,
keys, etc all matched).

Due to the MDADM and LVM headers preceding the luksHeader I doubt that a
straight dump would work.  It is, however, fairly easy to extract all of the
same info straight from the hex editor.  Would there be any point to running
a luksDump, or are you just trying to get the keys and such?

headers

Clayton Shepard | 19 Aug 05:49

Re: Massive Failure

Any ideas on how to check to see if I actually have the correct header?

On Thu, Aug 7, 2008 at 3:37 AM, Clayton Shepard <cws@...> wrote:

> Due to a combination of hardware failure, ignorance, and stupidity I have
> managed to really make a mess of a 7 drive mdadm (raid5) -> lvm -> luks ->
> ext3 setup.  So here is the basic setup:
>
> 1.  sudo badblocks -c 16384 -s -w -t random -v /dev/sd (on all 7 drives)
> 2.  sudo mdadm --create --verbose /dev/md0 --level=5 --chunk=256 --force
> --raid-devices=7  /dev/sd[a,b,c,d,e,f,g]
> 3.  pvcreate /dev/md0
> 4.  vgcreate lg /dev/md0 -s 256M
> 5.  lvcreate -l22356 -nlv lg
> 6.  cryptsetup --verify-passphrase --verbose --hash=sha256
> --cipher=aes-cbc-essiv:sha256 --key-size=128 luksFormat /dev/lg/lv
> 7.  cryptsetup luksOpen /dev/lg/lv encrypted
> 8.  mkfs.ext3 -j -m 3 -O dir_index,filetype,sparse_super,large_file
> /dev/mapper/encrypted
>
>
> So long story short, the hardrives were swapped around and one point which
> cause blkid.tab to go crazy and remap the drives to different letters.  All
> but one of the mdadm superblocks were erased, but I believe the array was
> clean before this happened.  By using basically the same command first used
> to create the array mdadm detected that it was already a raid array and
> recreated it - unfortunately it thought that the last drive (sdh at this
> time) was dirty, and thus began rebuilding it.  Unfortunately commands 3-7
> were also reissued, which initially made me believe that the luks header was
> destroyed (according to this thread:

(Continue reading)