Discussions forum for users of the Unison file synchronization tool

Russel Winder | 10 Dec 2009 14:04

Re: Unicode support status?

Benjamin,

On Thu, 2009-12-10 at 07:40 -0500, Benjamin Pierce wrote:
>   
> I'm not sure how many people are using the unicode support actively, 
> but in other respects the developer version seems very stable. 
> Anybody have any opinions on how well unicode is working now?
> 
> - Benjamin

I am not completely up to date from trunk, I'll update and build again
next week.  Ubuntu <-> Ubuntu has never been a problem using UTF-8
encoded Unicode file and directory names.  It used to work and it
continues to work.  On the other hand this doesn't need special support
in Unison and I am not sure I am using it.  So this isn't a real data
point.

Ubuntu <-> Mac OS X seems to be fine, stable and works as required.  My
Mac OS X filestore is case insensitive which is standard I believe.  The
problems I do have with transfer to Mac OS X are to do with number of
open files; basically Mac OS X does not have the capability of
concurrently opening enough files for my task -- Ubuntu has no problem
with basically the same transfer.

--

-- 
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel <at> russel.org.uk
Concertant LLP        t: +44 20 7585 2200, +44 20 7193 9203

(Continue reading)

headers

Benjamin Pierce | 10 Dec 2009 14:36

Re: Unicode support status?

> Ubuntu <-> Mac OS X seems to be fine, stable and works as required.   
> My
> Mac OS X filestore is case insensitive which is standard I believe.   
> The
> problems I do have with transfer to Mac OS X are to do with number of
> open files; basically Mac OS X does not have the capability of
> concurrently opening enough files for my task -- Ubuntu has no problem
> with basically the same transfer.

I haven't encountered this issue.  What goes wrong?

     - B

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/unison-users/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/unison-users/join
    (Yahoo! ID required)

<*> To change settings via email:
    unison-users-digest <at> yahoogroups.com 
    unison-users-fullfeatured <at> yahoogroups.com

(Continue reading)

headers

Russel Winder | 10 Dec 2009 15:33

Re: Unicode support status?

On Thu, 2009-12-10 at 08:36 -0500, Benjamin Pierce wrote:
> > Ubuntu <-> Mac OS X seems to be fine, stable and works as required.   
> > My
> > Mac OS X filestore is case insensitive which is standard I believe.   
> > The
> > problems I do have with transfer to Mac OS X are to do with number of
> > open files; basically Mac OS X does not have the capability of
> > concurrently opening enough files for my task -- Ubuntu has no problem
> > with basically the same transfer.
> 
> I haven't encountered this issue.  What goes wrong?

I ask for a sync between Ubuntu server (the source) and Mac OS X client
(the destination) and there are large numbers of files to commit --
sometime 20,000+.  Some directories can have 1000+ files, e.g. Fortress
Subversion checkout.  This seems to trigger an inability to do a sync.
Too many open files is the error message.  By trimming the commit
manually to avoid the directories with >500 files and keeping the total
number of files to be processed <5,000 I get a sync.  But this can take
5 or 6 rounds of work, by which time I am getting grumpy (more grumpy
than usual 

There are no problems with syncs over twice this size where Ubuntu is
the client (destination) for the sync.

I am currently away from my MacBook just now so I cannot prepare a
proper error report.  If you point me at the bug reporting system, I am
almost certain I can prepare a more complete report.

If this problem has an easy solution, it would help me tremendously.

(Continue reading)

headers

Benjamin Pierce | 21 Dec 2009 16:18

Re: Unicode support status?

> I ask for a sync between Ubuntu server (the source) and Mac OS X  
> client
> (the destination) and there are large numbers of files to commit --
> sometime 20,000+.  Some directories can have 1000+ files, e.g.  
> Fortress
> Subversion checkout.  This seems to trigger an inability to do a sync.
> Too many open files is the error message.  By trimming the commit
> manually to avoid the directories with >500 files and keeping the  
> total
> number of files to be processed <5,000 I get a sync.  But this can  
> take
> 5 or 6 rounds of work, by which time I am getting grumpy (more grumpy
> than usual 
>
> There are no problems with syncs over twice this size where Ubuntu is
> the client (destination) for the sync.

Are you saying that the problem is sensitive to which side is running  
as client and which as server, even if the flow of all the transfers  
is the same direction in both cases?

> I am currently away from my MacBook just now so I cannot prepare a
> proper error report.  If you point me at the bug reporting system, I  
> am
> almost certain I can prepare a more complete report.

The bug reporting system is this mailing list.

> If this problem has an easy solution, it would help me tremendously.

(Continue reading)

headers

Jerome Vouillon | 5 Jan 2010 14:59

Re: Unicode support status?

On Mon, Dec 21, 2009 at 10:18:04AM -0500, Benjamin Pierce wrote:
> > I ask for a sync between Ubuntu server (the source) and Mac OS X
> > client (the destination) and there are large numbers of files to
> > commit -- sometime 20,000+.  Some directories can have 1000+
> > files, e.g.  Fortress Subversion checkout.  This seems to trigger
> > an inability to do a sync.  Too many open files is the error
> > message.  By trimming the commit manually to avoid the directories
> > with >500 files and keeping the total number of files to be
> > processed <5,000 I get a sync.  But this can take 5 or 6 rounds of
> > work, by which time I am getting grumpy (more grumpy than usual
> > 

This should be fixed.  Thanks a lot for the bug report!

> One simple thing to try is setting maxthreads to 1.  I'd be interested  
> to see whether this helps.  (I'm having trouble imagining what could  
> be resulting in many files getting opened at the same time.)

Well, maxthreads is set to 1000 in the developers' version of
Unison...   This works great when transferring a lot of small files.
And this does not consume too much ressources, as file contents are
transferred one at a time.

The issue was with the rsync compression : the file descriptor for the
previous version of the target file was kept open between
preprocessing and actual file transfer.  And there can be a lot of
files at this stage...

-- Jerome

(Continue reading)

headers

Benjamin Pierce | 5 Jan 2010 15:54

Re: Unicode support status?

> The issue was with the rsync compression : the file descriptor for the
> previous version of the target file was kept open between
> preprocessing and actual file transfer.  And there can be a lot of
> files at this stage...

Thanks for figuring this out!

    - Benjamin

>
> -- Jerome
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/unison-users/

<*> Your email settings:
    Individual Email | Traditional

(Continue reading)

headers

Urbaniak, Sebastian | 5 Jan 2010 21:21

RE: Unicode support status?

xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">

I noticed a lot of talk about UNICODE lately .... so just out of curiosity I took a quick walk around my file archives and to my horror, in what I though were synched file sets I discovered entire subfolders with non ASCII letters in them, like this one (Dzem) [the z has a dot over it], unpropagated.

Is this something the Unicode=True directive will correct on Win32<->Win32 setups in the upcoming version ?

> -----Original Message-----

> From: unison-users <at> yahoogroups.com [mailto:unison-users <at> yahoogroups.com] On Behalf Of Benjamin Pierce

> Sent: January 5, 2010 09:54

> To: Jerome Vouillon

> Cc: Russel Winder; unison-users <at> yahoogroups.com

> Subject: Re: [unison-users] Unicode support status?

> > The issue was with the rsync compression : the file descriptor for the

> > previous version of the target file was kept open between

> > preprocessing and actual file transfer. And there can be a lot of

> > files at this stage...

> Thanks for figuring this out!

> - Benjamin

> >

> > -- Jerome

> >

__._,_.___

Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___

headers

Benjamin Pierce | 5 Jan 2010 21:35

Re: Unicode support status?

> Is this something the   Unicode=True  directive will correct  on    
> Win32<->Win32 setups    in the upcoming version ?

That's the hope.  I guess it's probably time to make a beta-release of  
the unicode-aware version so that we can shake out any remaining  
problems...

     - B

>
>
> S
>
> > -----Original Message-----
> > From: unison-users <at> yahoogroups.com [mailto:unison-users <at> yahoogroups.com 
> ] On Behalf Of Benjamin Pierce
> > Sent: January 5, 2010 09:54
> > To: Jerome Vouillon
> > Cc: Russel Winder; unison-users <at> yahoogroups.com
> > Subject: Re: [unison-users] Unicode support status?
> >
> > > The issue was with the rsync compression : the file descriptor  
> for the
> > > previous version of the target file was kept open between
> > > preprocessing and actual file transfer.  And there can be a lot of
> > > files at this stage...
> >
> > Thanks for figuring this out!
> >
> >     - Benjamin
> >
> >
> > >
> > > -- Jerome
> > >
> > >
>
>
>
> 

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/unison-users/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/unison-users/join
    (Yahoo! ID required)

<*> To change settings via email:
    unison-users-digest <at> yahoogroups.com 
    unison-users-fullfeatured <at> yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    unison-users-unsubscribe <at> yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

headers

Harakiri | 6 Jan 2010 13:17

Re: Unicode support status?

--- On Tue, 1/5/10, Jerome Vouillon <Jerome.Vouillon <at> pps.jussieu.fr> wrote:

From: Jerome Vouillon <Jerome.Vouillon <at> pps.jussieu.fr>
Subject: Re: [unison-users] Unicode support status?
To: "Russel Winder" <russel.winder <at> concertant.com>, "Benjamin Pierce" <bcpierce <at> cis.upenn.edu>
Cc: unison-users <at> yahoogroups.com
Date: Tuesday, January 5, 2010, 8:59 AM

On Mon, Dec 21, 2009 at 10:18:04AM -0500, Benjamin Pierce wrote:
> > I ask for a sync between Ubuntu server (the source) and Mac OS X
> > client (the destination) and there are large numbers of files to
> > commit -- sometime 20,000+. Some directories can have 1000+
> > files, e.g. Fortress Subversion checkout. This seems to trigger
> > an inability to do a sync. Too many open files is the error
> > message. By trimming the commit manually to avoid the directories
> > with >500 files and keeping the total number of files to be
> > processed <5,000 I get a sync. But this can take 5 or 6 rounds of
> > work, by which time I am getting grumpy (more grumpy than usual
> >

>This should be fixed. Thanks a lot for the bug report!

Sorry - does this mean that every unison version before was unable to sync 20.000+ files at once? (i.e. when they have all been changed). If this is indeed the case, the debian lenny maintainers should be informed to issue a fix.

Thanks!

__._,_.___

__,_._,___

headers

Benjamin Pierce | 6 Jan 2010 15:05

Re: Unicode support status?

> > > I ask for a sync between Ubuntu server (the source) and Mac OS X
> > > client (the destination) and there are large numbers of files to
> > > commit -- sometime 20,000+. Some directories can have 1000+
> > > files, e.g. Fortress Subversion checkout. This seems to trigger
> > > an inability to do a sync. Too many open files is the error
> > > message. By trimming the commit manually to avoid the directories
> > > with >500 files and keeping the total number of files to be
> > > processed <5,000 I get a sync. But this can take 5 or 6 rounds of
> > > work, by which time I am getting grumpy (more grumpy than usual
> > > 
>
>   >This should be fixed. Thanks a lot for the bug report!
>
> Sorry - does this mean that every unison version before was unable  
> to sync 20.000+ files at once? (i.e. when they have all been  
> changed). If this is indeed the case, the debian lenny maintainers  
> should be informed to issue a fix.
>

No: the bug was introduced with the support for the rsync program as  
an external transfer helper, so the only released versions it affected  
were 2.31 and 2.32.

     - B

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/unison-users/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/unison-users/join
    (Yahoo! ID required)

<*> To change settings via email:
    unison-users-digest <at> yahoogroups.com 
    unison-users-fullfeatured <at> yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    unison-users-unsubscribe <at> yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

headers

Jerome Vouillon | 6 Jan 2010 15:27

Re: Unicode support status?

On Wed, Jan 06, 2010 at 09:05:23AM -0500, Benjamin Pierce wrote:
> >> > I ask for a sync between Ubuntu server (the source) and Mac OS X
> >> > client (the destination) and there are large numbers of files to
> >> > commit -- sometime 20,000+. Some directories can have 1000+
> >> > files, e.g. Fortress Subversion checkout. This seems to trigger
> >> > an inability to do a sync. Too many open files is the error
> >> > message. By trimming the commit manually to avoid the directories
> >> > with >500 files and keeping the total number of files to be
> >> > processed <5,000 I get a sync. But this can take 5 or 6 rounds of
> >> > work, by which time I am getting grumpy (more grumpy than usual
> >> > 
> >
> >  >This should be fixed. Thanks a lot for the bug report!
> >
> >Sorry - does this mean that every unison version before was unable
> >to sync 20.000+ files at once? (i.e. when they have all been
> >changed). If this is indeed the case, the debian lenny maintainers
> >should be informed to issue a fix.
> 
> No: the bug was introduced with the support for the rsync program as
> an external transfer helper, so the only released versions it
> affected were 2.31 and 2.32.

Well, no.  The bug was only in the developers' version of Unison, not
in any offical release.  (I introduced it well after 2.32.)

-- Jerome

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/unison-users/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/unison-users/join
    (Yahoo! ID required)

<*> To change settings via email:
    unison-users-digest <at> yahoogroups.com 
    unison-users-fullfeatured <at> yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    unison-users-unsubscribe <at> yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

headers

Benjamin Pierce | 6 Jan 2010 15:37

Re: Unicode support status?

> No: the bug was introduced with the support for the rsync program as
>> an external transfer helper, so the only released versions it
>> affected were 2.31 and 2.32.
>
> Well, no.  The bug was only in the developers' version of Unison, not
> in any offical release.  (I introduced it well after 2.32.)

Ah -- I assumed *I'd* introduced it! 

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/unison-users/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/unison-users/join
    (Yahoo! ID required)

<*> To change settings via email:
    unison-users-digest <at> yahoogroups.com 
    unison-users-fullfeatured <at> yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    unison-users-unsubscribe <at> yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/