An open list for general Illumos discussion

Jorgen Lundman | 26 Jul 2012 03:04

Resuming ZFS Send


Hello list,

So, trying to move a 6Tb ZFS file system, always dies at 1.25Tb (every time 
too) made me wish there was a way to resume a previously failed "zfs send".

I see there was some discussion about "zfs send --resume" in some 
presentations by Delphix / Nexgenta.

Obviously, it wont help me today, but will it eventually because available 
as a feature? Just a dream, or some work done on it?

Lund

--

-- 
Jorgen Lundman       | <lundman <at> lundman.net>
Unix Administrator   | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo    | +81 (0)90-5578-8500          (cell)
Japan                | +81 (0)3 -3375-1767          (home)

-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175548-9b54c406
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175548&id_secret=21175548-1d01cae1
Powered by Listbox: http://www.listbox.com

headers

Sašo Kiselkov | 26 Jul 2012 10:03

Re: Resuming ZFS Send

On 07/26/2012 03:04 AM, Jorgen Lundman wrote:
> Hello list,

Hello poster, list here  (just joking)

> So, trying to move a 6Tb ZFS file system, always dies at 1.25Tb (every
> time too) made me wish there was a way to resume a previously failed
> "zfs send".

How does it die? Does the system panic? If so, you should be able to
extract a crash dump using the procedure below and post the output here:
http://wiki.illumos.org/display/illumos/How+To+Report+Problems

Cheers,
--
Saso

-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175548-9b54c406
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175548&id_secret=21175548-1d01cae1
Powered by Listbox: http://www.listbox.com

headers

Jorgen Lundman | 26 Jul 2012 12:55

Re: Resuming ZFS Send

Sašo Kiselkov wrote:

> How does it die? Does the system panic? If so, you should be able to
> extract a crash dump using the procedure below and post the output here:
> http://wiki.illumos.org/display/illumos/How+To+Report+Problems

Nah, nothing so major. The "zfs" process seems to just die, to the point that 
"nc" does not even notice it is gone. This exact situation is even on Solaris 
10, so not specific to this mailing-list. (So I wasn't after support directly)

But with 48+ hours transfers, there will always be reasons you might lose a 
connection, or network, or server reboot, or tripping over a cable. So it was 
more a question about the feature, which is definitely needed. Once I have the 
base fs over, I can do incremental transfers, sure.

Lund

-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175548-9b54c406
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175548&id_secret=21175548-1d01cae1
Powered by Listbox: http://www.listbox.com

headers

Sriram Narayanan | 26 Jul 2012 18:16

Re: Resuming ZFS Send

On Thu, Jul 26, 2012 at 4:25 PM, Jorgen Lundman <lundman <at> lundman.net> wrote:
> Sašo Kiselkov wrote:
>
>> How does it die? Does the system panic? If so, you should be able to
>> extract a crash dump using the procedure below and post the output here:
>> http://wiki.illumos.org/display/illumos/How+To+Report+Problems
>
>
> Nah, nothing so major. The "zfs" process seems to just die, to the point
> that "nc" does not even notice it is gone. This exact situation is even on
> Solaris 10, so not specific to this mailing-list. (So I wasn't after support
> directly)
>
> But with 48+ hours transfers, there will always be reasons you might lose a
> connection, or network, or server reboot, or tripping over a cable. So it
> was more a question about the feature, which is definitely needed. Once I
> have the base fs over, I can do incremental transfers, sure.

I wonder if you could use rsync in some way, perhaps by piping the
transfer in case the source server doesn't have enough space for a
local zfs dump.

>
> Lund
>
>
>
>
>
>

(Continue reading)

headers

Sriram Narayanan | 26 Jul 2012 18:19

Re: Resuming ZFS Send

On Thu, Jul 26, 2012 at 9:46 PM, Sriram Narayanan <sriram <at> belenix.org> wrote:
> On Thu, Jul 26, 2012 at 4:25 PM, Jorgen Lundman <lundman <at> lundman.net> wrote:
>> Sašo Kiselkov wrote:
>>
>>> How does it die? Does the system panic? If so, you should be able to
>>> extract a crash dump using the procedure below and post the output here:
>>> http://wiki.illumos.org/display/illumos/How+To+Report+Problems
>>
>>
>> Nah, nothing so major. The "zfs" process seems to just die, to the point
>> that "nc" does not even notice it is gone. This exact situation is even on
>> Solaris 10, so not specific to this mailing-list. (So I wasn't after support
>> directly)
>>
>> But with 48+ hours transfers, there will always be reasons you might lose a
>> connection, or network, or server reboot, or tripping over a cable. So it
>> was more a question about the feature, which is definitely needed. Once I
>> have the base fs over, I can do incremental transfers, sure.
>
> I wonder if you could use rsync in some way, perhaps by piping the
> transfer in case the source server doesn't have enough space for a
> local zfs dump.

I just wanted to clarify - I understand how rsync works and what it's
for. I'm only wondering myself how to resume a transfer of a known
file (e.g. a zfs snapshot) in case dumping the said snapshot as a
regular local file is not an option.

>
>>

(Continue reading)

headers

Jorgen Lundman | 27 Jul 2012 01:41

Re: Resuming ZFS Send

>
> I wonder if you could use rsync in some way, perhaps by piping the
> transfer in case the source server doesn't have enough space for a
> local zfs dump.
>

If I had room to do a "zfs send > phat_file" I would do that, and then 
rsync it over. But alas, we are moving this storage to a bigger storage due 
to the lack of room.

Rsync can do squat when it comes to "zfs send | rsync" since there is no 
seek on the pipe.

If "zfs send" had a mmaped device entry, OR, say like that of zvol 
/dev/zvol/dsk/ entry, we'd have something sexy :)

Lund

--

-- 
Jorgen Lundman       | <lundman <at> lundman.net>
Unix Administrator   | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo    | +81 (0)90-5578-8500          (cell)
Japan                | +81 (0)3 -3375-1767          (home)

-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175548-9b54c406
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175548&id_secret=21175548-1d01cae1
Powered by Listbox: http://www.listbox.com

(Continue reading)

headers

Richard Elling | 26 Jul 2012 19:03

Re: Resuming ZFS Send

On Jul 25, 2012, at 6:04 PM, Jorgen Lundman wrote:

Hello list,

So, trying to move a 6Tb ZFS file system, always dies at 1.25Tb (every time too) made me wish there was a way to resume a previously failed "zfs send".

Root cause can be difficult to track on these failures, unless you have

good visibility into both ends. I've seen everything from failure to decouple

from controlling tty to cron jobs that restart services every day at 5:15AM

(you know who you are)

I see there was some discussion about "zfs send --resume" in some presentations by Delphix / Nexgenta.

Delphix has been contributing enhancements to ZFS and has discussed some

automated recovery for send/receive. These changes are not yet putback into

the illumos-gate.

I am not aware of any recent contributions from Nexenta to the community wrt ZFS.

Nexenta did replace the auto-sync service transport in NexentaStor 3.1.2 (and

removed the nc option :-(

, but that is just stuff between the pipes.

IMHO, "auto-sync" should be called "auto-send" or something other than "sync"

because it is not synchronous, it is ZFS send/receive.

Obviously, it wont help me today, but will it eventually because available as a feature? Just a dream, or some work done on it?

-- richard

ZFS Performance and Training

Richard.Elling <at> RichardElling.com
+1-760-896-4422

illumos-discuss | Archives | Modify Your Subscription

headers

Marcelo Leal | 26 Jul 2012 19:53

Re: Resuming ZFS Send

Hi,

A common problem I have experienced (not identified the cause), in an incremental send/receive, was that the previous snapshot on the destination was not "zero" sized.

I don't know why the previous snap was retaining some data (a few "K"s), if the live filesystem did not changed (believe me ;-)

Maybe was a failure on the previous send that was not well handled and finished the process not consistently (I have some ideas about fragmentation and ZFS holding "garbage" after actually removing snapshots and even entire datasets). Anyway...

Well, to fix this specific case, you just need to do a "rollback" for the previous snapshot on the destination.

The problem is that the incremental send does not complain on the begining of the process, but just after a random time.

If you have a big incremental stream to send, can be really annoying.

For full send/receive, I do not remember to experience any problems that were related to ZFS. All the problems to fully receive the stream were transport related.

For sure a "resume" operation on send/receive process is a must for a Zettabyte Filesystem. As well as people willing to (and having time to) help on the implementation.... ;-)

Hope that helps between here and there.

Leal

-------=- pOSix rules -=-------
[ http://www.eall.com.br/blog ]

I got 99 problems but a "bit" ain't one.

This and other cool T-Shirts here:
[ http://www.cafepress.com/eallonlinestore ]

2012/7/26 Richard Elling <richard.elling <at> richardelling.com>

On Jul 25, 2012, at 6:04 PM, Jorgen Lundman wrote:

Hello list,

So, trying to move a 6Tb ZFS file system, always dies at 1.25Tb (every time too) made me wish there was a way to resume a previously failed "zfs send".

Root cause can be difficult to track on these failures, unless you have

good visibility into both ends. I've seen everything from failure to decouple

from controlling tty to cron jobs that restart services every day at 5:15AM

(you know who you are)

I see there was some discussion about "zfs send --resume" in some presentations by Delphix / Nexgenta.

Delphix has been contributing enhancements to ZFS and has discussed some

automated recovery for send/receive. These changes are not yet putback into

the illumos-gate.

I am not aware of any recent contributions from Nexenta to the community wrt ZFS.

Nexenta did replace the auto-sync service transport in NexentaStor 3.1.2 (and

removed the nc option , but that is just stuff between the pipes.

IMHO, "auto-sync" should be called "auto-send" or something other than "sync"

because it is not synchronous, it is ZFS send/receive.

Obviously, it wont help me today, but will it eventually because available as a feature? Just a dream, or some work done on it?

-- richard

--

ZFS Performance and Training

Richard.Elling <at> RichardElling.com
+1-760-896-4422

illumos-discuss | Archives | Modify Your Subscription

illumos-discuss | Archives | Modify Your Subscription

headers

Matthew Ahrens | 26 Jul 2012 21:04

Re: Resuming ZFS Send

On Thu, Jul 26, 2012 at 10:53 AM, Marcelo Leal <sunos.x86 <at> gmail.com> wrote:

Hi,

A common problem I have experienced (not identified the cause), in an incremental send/receive, was that the previous snapshot on the destination was not "zero" sized.
I don't know why the previous snap was retaining some data (a few "K"s), if the live filesystem did not changed (believe me .

The filesystem did change, perhaps due to atime modifications. "zfs diff" can tell you what changed. You can prevent changes with "zfs set readonly=on <filesystem>".

Well, to fix this specific case, you just need to do a "rollback" for the previous snapshot on the destination.

The problem is that the incremental send does not complain on the begining of the process, but just after a random time.

If you have a big incremental stream to send, can be really annoying.

Rather than explicitly rolling back, use "zfs receive -F". This will ensure that the receive does not fail even if you modify the filesystem in the middle of the receive. As the manpage says:

-F

Force a rollback of the file system to the most

recent snapshot before performing the receive opera-

tion. If receiving an incremental replication stream

(for example, one generated by zfs send -R -[iI]),

destroy snapshots and file systems that do not exist

on the sending side.

--matt

illumos-discuss | Archives | Modify Your Subscription

headers

Marcelo Leal | 26 Jul 2012 22:32

Re: Resuming ZFS Send

Hi,

>> Rather than explicitly rolling back, use "zfs receive -F".

But sometimes you don't want "all" the consequences of "-F" (e.g: my case), and the rollback will fix the problem and will not have side effects.

Leal

[ http://www.eall.com.br/blog ]

2012/7/26 Matthew Ahrens <mahrens <at> delphix.com>

On Thu, Jul 26, 2012 at 10:53 AM, Marcelo Leal <sunos.x86 <at> gmail.com> wrote:

Hi,

A common problem I have experienced (not identified the cause), in an incremental send/receive, was that the previous snapshot on the destination was not "zero" sized.
I don't know why the previous snap was retaining some data (a few "K"s), if the live filesystem did not changed (believe me .

The filesystem did change, perhaps due to atime modifications. "zfs diff" can tell you what changed. You can prevent changes with "zfs set readonly=on <filesystem>".

Well, to fix this specific case, you just need to do a "rollback" for the previous snapshot on the destination.

The problem is that the incremental send does not complain on the begining of the process, but just after a random time.

If you have a big incremental stream to send, can be really annoying.

Rather than explicitly rolling back, use "zfs receive -F". This will ensure that the receive does not fail even if you modify the filesystem in the middle of the receive. As the manpage says:

-F

Force a rollback of the file system to the most

recent snapshot before performing the receive opera-

tion. If receiving an incremental replication stream

(for example, one generated by zfs send -R -[iI]),

destroy snapshots and file systems that do not exist

on the sending side.

--matt

illumos-discuss | Archives | Modify Your Subscription

illumos-discuss | Archives | Modify Your Subscription

headers

Jesus Cea | 2 Aug 2012 01:18

Re: Resuming ZFS Send


On 26/07/12 19:53, Marcelo Leal wrote:
> A common problem I have experienced (not identified the cause), in
> an incremental send/receive, was that the previous snapshot on the 
> destination was not "zero" sized.

If your dataset updates "atime", any read of it will update atime of
files and directories.

--

-- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea <at> jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea <at> jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

headers

Marcelo Leal | 2 Aug 2012 20:54

Re: Resuming ZFS Send

2012/8/1 Jesus Cea <jcea <at> jcea.es>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 26/07/12 19:53, Marcelo Leal wrote:
>> A common problem I have experienced (not identified the cause), in
>> an incremental send/receive, was that the previous snapshot on the
>> destination was not "zero" sized.
>
> If your dataset updates "atime", any read of it will update atime of
> files and directories.

 But that is not the case. All production datasets have this
configuration "off".
 No way to go production with ZFS (email) and atime on. 

 Leal

>
> - --
> Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
> jcea <at> jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
> jabber / xmpp:jcea <at> jabber.org         _/_/    _/_/          _/_/_/_/_/
> .                              _/_/  _/_/    _/_/          _/_/  _/_/
> "Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQCVAwUBUBm5X5lgi5GaxT1NAQKx6AQAggKjm57+YsdPSLZrjiJMm/O4VkoZWQZ8
> zG65nIDCCLgqMpqO7bDPLe+X1lXJxulPdx0DMNQdPNGzQ9BQnn8CnAUgbyMOSLut
> GlM3sSPYUX3+k92dQPY3SHqoTvSEhUPA3PZ+TTHjBQVD/S+/hSYmLtXSa7x5A3tn
> QeFVPGAy5nQ=
> =EOcs
> -----END PGP SIGNATURE-----
>
>
> -------------------------------------------
> illumos-discuss
> Archives: https://www.listbox.com/member/archive/182180/=now
> RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175783-cbc6ae69
> Modify Your Subscription: https://www.listbox.com/member/?&
> Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175548-9b54c406
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175548&id_secret=21175548-1d01cae1
Powered by Listbox: http://www.listbox.com

headers

Matthew Ahrens | 26 Jul 2012 21:07

Re: Resuming ZFS Send

On Wed, Jul 25, 2012 at 6:04 PM, Jorgen Lundman <lundman <at> lundman.net> wrote:

Hello list,

So, trying to move a 6Tb ZFS file system, always dies at 1.25Tb (every time too) made me wish there was a way to resume a previously failed "zfs send".

As others have mentioned, this could be a bug in "zfs receive", and resumable send wouldn't help in that case.

I see there was some discussion about "zfs send --resume" in some presentations by Delphix / Nexgenta.

Obviously, it wont help me today, but will it eventually because available as a feature? Just a dream, or some work done on it?

We (Delphix) are still working on resumable send. We'll be sure to post when it's ready to integrate to Illumos!

--matt

illumos-discuss | Archives | Modify Your Subscription

headers

Jorgen Lundman | 27 Jul 2012 01:42

Re: Resuming ZFS Send

> We (Delphix) are still working on resumable send.  We'll be sure to post
> when it's ready to integrate to Illumos!
>

Fabulous, Quarter 3 goals having 100T to 300T storage, so I'm game to 
"beta" test :)

But seriously, thanks for the continued work.

Lund

--

-- 
Jorgen Lundman       | <lundman <at> lundman.net>
Unix Administrator   | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo    | +81 (0)90-5578-8500          (cell)
Japan                | +81 (0)3 -3375-1767          (home)

-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175548-9b54c406
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175548&id_secret=21175548-1d01cae1
Powered by Listbox: http://www.listbox.com