gmane.comp.shells.zsh.user

Aaron Davies | 7 Aug 08:53

Multi-Minute Startup?

It takes zsh multiple minutes (around 3) to start up on a new server I
just got access to. I have no custom .z* files at this point, so all
that's running should be the /etc files, of which only /etc/zshenv and
/etc/zshrc are actually present. Any ideas what's likely to be slowing
me down so much?
--

-- 
Aaron Davies
aaron.davies <at> gmail.com

headers

Benjamin R. Haskell | 7 Aug 09:00

Re: Multi-Minute Startup?

On Thu, 7 Aug 2008, Aaron Davies wrote:

> It takes zsh multiple minutes (around 3) to start up on a new server I 
> just got access to. I have no custom .z* files at this point, so all 
> that's running should be the /etc files, of which only /etc/zshenv and 
> /etc/zshrc are actually present. Any ideas what's likely to be slowing 
> me down so much?

What O/S and architecture? (Not sure if it's relevant, but maybe Zsh 
doesn't run so fast on a 286.)

Try running zsh -x -l
-l makes it a login shell
-x is to trace commands as they're executed

Maybe that will reveal a particular command that's taking a long time to 
execute.

Best,
Ben

headers

Aaron Davies | 7 Aug 09:20

Re: Multi-Minute Startup?

On Thu, Aug 7, 2008 at 3:00 PM, Benjamin R. Haskell <zsh <at> benizi.com> wrote:
> On Thu, 7 Aug 2008, Aaron Davies wrote:
>
>> It takes zsh multiple minutes (around 3) to start up on a new server I
>> just got access to. I have no custom .z* files at this point, so all that's
>> running should be the /etc files, of which only /etc/zshenv and /etc/zshrc
>> are actually present. Any ideas what's likely to be slowing me down so much?
>
> What O/S and architecture? (Not sure if it's relevant, but maybe Zsh doesn't
> run so fast on a 286.)

linux on some fairly new enterprise server hardware. i don't think
it's the machine's fault. :)

> Try running zsh -x -l
> -l makes it a login shell
> -x is to trace commands as they're executed
>
> Maybe that will reveal a particular command that's taking a long time to
> execute.

ah, looks like it's part of compaudit

+compaudit:84> IFS=: +compaudit:84> read GROUP _i_pw _i_gid GROUPMEM

is it trying to do something like read all users/groups from the
authentication system for completion? that might explain it, as our
directory server is 10,000 miles away. if that's the problem, is there
a way to get rid of it without having write access to the /etc/z*
files?

(Continue reading)

headers

Benjamin R. Haskell | 7 Aug 09:33

Re: Multi-Minute Startup?

On Thu, 7 Aug 2008, Aaron Davies wrote:
> On Thu, Aug 7, 2008 at 3:00 PM, Benjamin R. Haskell <zsh <at> benizi.com> wrote:
>> On Thu, 7 Aug 2008, Aaron Davies wrote:
>>
>>> It takes zsh multiple minutes (around 3) to start up on a new server I 
>>> just got access to. I have no custom .z* files at this point, so all 
>>> that's running should be the /etc files, of which only /etc/zshenv and 
>>> /etc/zshrc are actually present. Any ideas what's likely to be slowing 
>>> me down so much?
>>
>> What O/S and architecture? (Not sure if it's relevant, but maybe Zsh 
>> doesn't run so fast on a 286.)
>
> linux on some fairly new enterprise server hardware. i don't think it's 
> the machine's fault. :)
>

Had ta ask... 

>> Try running zsh -x -l
>> -l makes it a login shell
>> -x is to trace commands as they're executed
>>
>> Maybe that will reveal a particular command that's taking a long time 
>> to execute.
>
> ah, looks like it's part of compaudit
>
> +compaudit:84> IFS=: +compaudit:84> read GROUP _i_pw _i_gid GROUPMEM
>

(Continue reading)

headers

Aaron Davies | 7 Aug 10:52

Re: Multi-Minute Startup?

On Thu, Aug 7, 2008 at 3:33 PM, Benjamin R. Haskell <zsh <at> benizi.com> wrote:
> Something similar came up just under a month ago on this list:
>
> ==
> Eric D. Friedman wrote:
> Subject: new-style completion and large numbers of users
>
> I'm using zsh 4.2.0 on a machine with a very large number of users. When I
> first log on and do a completion of any kind, it triggers a cache-load of
> all of the user names, which is quite time consuming. I'd like to disable
> that behavior but can't see how to do it.  The archives have some ideas for
> old style completion but I didn't see anything for the new system.
> ==
>
> ==
> Angel Olivera replied:
>
> You can make zsh complete just the usernames you will be more likely to use
> with
>
> zstyle ':completion:*' users user1 user2
>
> Not specifying any users will effectively disable login name completions.
> See zshcompsys(1) for a description of this style.
> ==
>
> I forget the order of Zsh's startup files, but maybe you can slip a similar
> zstyle command into something that will run before your slow compaudit
> command that will prevent it from fetching all the groups.

(Continue reading)

headers

Benjamin R. Haskell | 7 Aug 11:28

Re: Multi-Minute Startup?

On Thu, 7 Aug 2008, Aaron Davies wrote:
> On Thu, Aug 7, 2008 at 3:33 PM, Benjamin R. Haskell <zsh <at> benizi.com> wrote:
>> Something similar came up just under a month ago on this list:
>>
>> ==
>> Eric D. Friedman wrote:
>> Subject: new-style completion and large numbers of users
>>
>> I'm using zsh 4.2.0 on a machine with a very large number of users. When I
>> first log on and do a completion of any kind, it triggers a cache-load of
>> all of the user names, which is quite time consuming. I'd like to disable
>> [...]
>
> [...]
>
> Looking at compaudit itself, it looks like it's still running "getent 
> group".

Looking at compaudit more closely leads me to think something might be 
awry with your environment variables (either LOGNAME or EGID). On my 
system (Gentoo w/Zsh 4.3.4), the parts of compaudit germane to getent are:

===============================
[[ -x /usr/bin/getent ]] || getent() {
   if [[ $2 = <-> ]]; then
     grep ":$2:[^:]*$" /etc/$1
   else
     grep "^$2:" /etc/$1
   fi
}

(Continue reading)

headers

Aaron Davies | 8 Aug 03:32

Re: Multi-Minute Startup?

On Thu, Aug 7, 2008 at 5:28 PM, Benjamin R. Haskell <zsh <at> benizi.com> wrote:

> On Thu, 7 Aug 2008, Aaron Davies wrote:
>
>> Looking at compaudit itself, it looks like it's still running "getent
>> group".
>
> Looking at compaudit more closely leads me to think something might be awry
> with your environment variables (either LOGNAME or EGID). On my system
> (Gentoo w/Zsh 4.3.4), the parts of compaudit germane to getent are:
>
>
> ===============================
> [[ -x /usr/bin/getent ]] || getent() {
>  if [[ $2 = <-> ]]; then
>    grep ":$2:[^:]*$" /etc/$1
>  else
>    grep "^$2:" /etc/$1
>  fi
> }
>
> # ... trimmed
>
> # RedHat Linux "per-user groups" check.  This is tricky, because it's very
> # difficult to tell whether the sysadmin has put someone else into your
> # "private" group (e.g., via the default group field in /etc/passwd, or
> # by NFS group sharing with an untrustworthy machine).  So we must assume
> # that this has not happened, and pick the best group.
>
> local GROUP GROUPMEM _i_pw _i_gid _i_ulwdirs

(Continue reading)

headers

Dan Nelson | 8 Aug 04:58

Re: Multi-Minute Startup?

In the last episode (Aug 08), Aaron Davies said:
> FWIW, "getent group" returns 773 lines and takes about 3.5 minutes to
> run.

If it takes that long to return only 773 lines, you should probably
turn on nscd, or if it's already on, crank up the cache size and TTL.

--

-- 
	Dan Nelson
	dnelson <at> allantgroup.com

headers

Aaron Davies | 8 Aug 08:40

Re: Multi-Minute Startup?

On Fri, Aug 8, 2008 at 10:58 AM, Dan Nelson <dnelson <at> allantgroup.com> wrote:
> In the last episode (Aug 08), Aaron Davies said:
>> FWIW, "getent group" returns 773 lines and takes about 3.5 minutes to
>> run.
>
> If it takes that long to return only 773 lines, you should probably
> turn on nscd, or if it's already on, crank up the cache size and TTL.

hmm, i don't see any sign that ncsd's even installed, let alone running.
--

-- 
Aaron Davies
aaron.davies <at> gmail.com

headers

Benjamin R. Haskell | 8 Aug 18:03

Re: Multi-Minute Startup?

On Fri, 8 Aug 2008, Aaron Davies wrote:

> On Fri, Aug 8, 2008 at 10:58 AM, Dan Nelson <dnelson <at> allantgroup.com> wrote:
>> In the last episode (Aug 08), Aaron Davies said:
>>> FWIW, "getent group" returns 773 lines and takes about 3.5 minutes to
>>> run.
>>
>> If it takes that long to return only 773 lines, you should probably
>> turn on nscd, or if it's already on, crank up the cache size and TTL.
>
> hmm, i don't see any sign that ncsd's even installed, let alone running.
>

Maybe just a typo in your email, but it's:

nscd == Name Service Caching Daemon

not ncsd (== National Cyber Security Division [part of Homeland 
Security!])

headers

Aaron Davies | 11 Aug 11:48

Re: Multi-Minute Startup?

On Sat, Aug 9, 2008 at 12:03 AM, Benjamin R. Haskell <zsh <at> benizi.com> wrote:
> On Fri, 8 Aug 2008, Aaron Davies wrote:
>
>> On Fri, Aug 8, 2008 at 10:58 AM, Dan Nelson <dnelson <at> allantgroup.com>
>> wrote:
>>>
>>> In the last episode (Aug 08), Aaron Davies said:
>>>>
>>>> FWIW, "getent group" returns 773 lines and takes about 3.5 minutes to
>>>> run.
>>>
>>> If it takes that long to return only 773 lines, you should probably
>>> turn on nscd, or if it's already on, crank up the cache size and TTL.
>>
>> hmm, i don't see any sign that ncsd's even installed, let alone running.
>>
>
> Maybe just a typo in your email, but it's:
>
> nscd == Name Service Caching Daemon
>
> not ncsd (== National Cyber Security Division [part of Homeland Security!])

d'oh

ok, it is running, and it looks like it should be doing something.

> egrep passwd\|group /etc/nscd.conf
# Currently supported cache names (services): passwd, group, hosts
        enable-cache            passwd          yes

(Continue reading)

headers

Dan Nelson | 11 Aug 17:29

Re: Multi-Minute Startup?

In the last episode (Aug 11), Aaron Davies said:
> On Sat, Aug 9, 2008 at 12:03 AM, Benjamin R. Haskell wrote:
> > On Fri, 8 Aug 2008, Aaron Davies wrote:
> >> On Fri, Aug 8, 2008 at 10:58 AM, Dan Nelson wrote:
> >>> In the last episode (Aug 08), Aaron Davies said:
> >>>>
> >>>> FWIW, "getent group" returns 773 lines and takes about 3.5
> >>>> minutes to run.
> >>>
> >>> If it takes that long to return only 773 lines, you should
> >>> probably turn on nscd, or if it's already on, crank up the cache
> >>> size and TTL.
> 
> ok, it is running, and it looks like it should be doing something.
> 
> > egrep passwd\|group /etc/nscd.conf
> # Currently supported cache names (services): passwd, group, hosts
>         enable-cache            passwd          yes
>         positive-time-to-live   passwd          600
>         negative-time-to-live   passwd          20
>         suggested-size          passwd          211
>         check-files             passwd          yes
>         persistent              passwd          yes
>         shared                  passwd          yes
>         enable-cache            group           yes
>         positive-time-to-live   group           3600
>         negative-time-to-live   group           60
>         suggested-size          group           211
>         check-files             group           yes
>         persistent              group           yes

(Continue reading)

headers

Richard Hartmann | 14 Aug 15:08

Re: Multi-Minute Startup?

On Mon, Aug 11, 2008 at 17:29, Dan Nelson <dnelson <at> allantgroup.com> wrote:

> Raise suggested-size to 1601 (a prime number larger than your current
> list size with some room to grow).

Using a prime hints at an interesting reason. What is it?

Richard

headers

Dan Nelson | 14 Aug 16:42

Re: Multi-Minute Startup?

In the last episode (Aug 14), Richard Hartmann said:
> On Mon, Aug 11, 2008 at 17:29, Dan Nelson <dnelson <at> allantgroup.com> wrote:
> 
> > Raise suggested-size to 1601 (a prime number larger than your current
> > list size with some room to grow).
> 
> Using a prime hints at an interesting reason. What is it?

I just copied the recommendation out of the nscd.conf manpage:

       suggested-size service prime-number

              This  option sets the size of the hash that is used to
              store the cache entries. As this is a hash, it should be
              reasonably larger than the maximum number of entries that
              is expected to be cached simultaneously and should be a
              prime number. It defaults to a size of 211 entries.

--

-- 
	Dan Nelson
	dnelson <at> allantgroup.com

headers

Phil Pennock | 14 Aug 20:37

Re: Multi-Minute Startup?

On 2008-08-14 at 15:08 +0200, Richard Hartmann wrote:
> On Mon, Aug 11, 2008 at 17:29, Dan Nelson <dnelson <at> allantgroup.com> wrote:
> > Raise suggested-size to 1601 (a prime number larger than your current
> > list size with some room to grow).
> 
> Using a prime hints at an interesting reason. What is it?

General hashing algorithm theory.  I forget the math (if I ever knew
it); also, there's a lot of heuristics in there ("that seems to work,
let's go with that").

You tend to not be hashing completely random data and in the absence of
distribution information about the input data, bucketing into a prime
number of slots tends to cause the least pain.

You'll find prime number requirements for many hash bucketing schemes.

-Phil, who should have made more effort to stay awake that lecture

headers

Eric D. Friedman | 14 Aug 21:02

Re: Multi-Minute Startup?

Using a prime number to size a hashtable is a way of getting a good distribution across buckets in the table if one is using division (modulo) by the table size to identify the bucket to use for some key. You can get a refresher in Knuth v3, p. 515-516.

Another, faster approach that you will also see a lot of these days (the java hashtables use this one as of a few releases back), is to set hashtable sizes to one-less-than-some-power-of two. You then assign keys to buckets by doing a bitwise and of the key and that size. This saves the time involved in doing division (still very expensive, even on modern processors) and also removes the need to find a new prime number every time the table grows.

On Aug 14, 2008, at 11:37 AM, Phil Pennock wrote:

On 2008-08-14 at 15:08 +0200, Richard Hartmann wrote:
On Mon, Aug 11, 2008 at 17:29, Dan Nelson <dnelson <at> allantgroup.com> wrote:

Raise suggested-size to 1601 (a prime number larger than your current

list size with some room to grow).

Using a prime hints at an interesting reason. What is it?

General hashing algorithm theory. I forget the math (if I ever knew
it); also, there's a lot of heuristics in there ("that seems to work,
let's go with that").

You tend to not be hashing completely random data and in the absence of
distribution information about the input data, bucketing into a prime
number of slots tends to cause the least pain.

You'll find prime number requirements for many hash bucketing schemes.

-Phil, who should have made more effort to stay awake that lecture

headers

Aaron Davies | 8 Aug 08:33

Re: Multi-Minute Startup?

On Fri, Aug 8, 2008 at 9:32 AM, Aaron Davies <aaron.davies <at> gmail.com> wrote:
> On Thu, Aug 7, 2008 at 5:28 PM, Benjamin R. Haskell <zsh <at> benizi.com> wrote:
>
>> On Thu, 7 Aug 2008, Aaron Davies wrote:
>>
>>> Looking at compaudit itself, it looks like it's still running "getent
>>> group".
>>
>> Looking at compaudit more closely leads me to think something might be awry
>> with your environment variables (either LOGNAME or EGID). On my system
>> (Gentoo w/Zsh 4.3.4), the parts of compaudit germane to getent are:
>>
>>
>> ===============================
>> [[ -x /usr/bin/getent ]] || getent() {
>>  if [[ $2 = <-> ]]; then
>>    grep ":$2:[^:]*$" /etc/$1
>>  else
>>    grep "^$2:" /etc/$1
>>  fi
>> }
>>
>> # ... trimmed
>>
>> # RedHat Linux "per-user groups" check.  This is tricky, because it's very
>> # difficult to tell whether the sysadmin has put someone else into your
>> # "private" group (e.g., via the default group field in /etc/passwd, or
>> # by NFS group sharing with an untrustworthy machine).  So we must assume
>> # that this has not happened, and pick the best group.
>>
>> local GROUP GROUPMEM _i_pw _i_gid _i_ulwdirs
>> if ((UID == EUID )); then
>>  getent group $LOGNAME | IFS=: read GROUP _i_pw _i_gid GROUPMEM
>> else
>>  getent group $EGID | IFS=: read GROUP _i_pw _i_gid GROUPMEM
>> fi
>> ===============================
>>
>> Is getent installed in /usr/bin on that machine? If not, maybe it's doing a
>> grep over a large, generated file? (Three minutes seems excessive, though.)
>
> getent is definitely present, and anyway, /etc/passwd and /etc/group
> are tiny stub files on this box--no more than 50 lines each.
>
>> If getent is there, maybe LOGNAME or EGID (whichever path is appropriate) is
>> unset. So, instead of (w/ a username of 'aaron'):
>
> LOGNAME appears to be set correctly (adavies).
>
>> getent group aaron | IFS=: read GROUP _i_pw _i_gid GROUPMEM
>> (finding a specific group's members, which should be fast, even over a long
>> distance -- otherwise many common operations would be annoyingly slow)
>>
>> it becomes:
>>
>> getent group | IFS=: read GROUP _i_pw _i_gid GROUPMEM
>> (which should be slow, since you're reading all 10,000 groups)
>>
>>
>> Can you find the specific invocation of getent that's slow for you?
>
> The code in my compaudit is
>
> ===============================
> # RedHat Linux "per-user groups" check.  This is tricky, because it's very
> # difficult to tell whether the sysadmin has put someone else into your
> # "private" group (e.g., via the default group field in /etc/passwd, or
> # by NFS group sharing with an untrustworthy machine).  So we must assume
> # that this has not happened, and pick the best group.
>
> local GROUP GROUPMEM _i_pw _i_gid _i_ulwdirs
> while IFS=: read GROUP _i_pw _i_gid GROUPMEM; do
>  if (( UID == EUID )); then
>    [[ $GROUP == $LOGNAME ]] && break
>  else
>    (( _i_gid == EGID )) && break       # Somewhat arbitrary
>  fi
> done <<(getent group)
> ===============================
>
> which looks to me like it *will* get all groups.
>
> FWIW, "getent group" returns 773 lines and takes about 3.5 minutes to run.

i should add that this zsh reports itself as "zsh 4.2.0
(x86_64-suse-linux)", which looks seriously archaic. I don't see any
sign (from a quick scan of YaST and of novell's website) that anything
more recent is available pre-packaged for SuSE enterprise server
9--anyone know different, or will I have to try to convince the admins
to let me install from source?
--

-- 
Aaron Davies
aaron.davies <at> gmail.com

headers

Eric D. Friedman | 8 Aug 20:16

Re: Multi-Minute Startup?

On Aug 7, 2008, at 2:28 AM, Benjamin R. Haskell wrote:

On Thu, Aug 7, 2008 at 3:33 PM, Benjamin R. Haskell <zsh <at> benizi.com> wrote:

Something similar came up just under a month ago on this list:

==

Eric D. Friedman wrote:

Subject: new-style completion and large numbers of users

I'm using zsh 4.2.0 on a machine with a very large number of users. When I

first log on and do a completion of any kind, it triggers a cache-load of

all of the user names, which is quite time consuming. I'd like to disable

[...]

I was the original poster with that long completion pause and none of the suggested solutions actually worked out. Following the advice in this thread, I ran a shell with -l -x on and captured the relavant bits. The big pause occurs at the last line in the following. Any ideas as to what else I could try? My goal is to not have the completion system show me every user in my company's home dir when I do the first cd<TAB>; I'd never use that completion and it takes far too long to get it anyway.

+_setup:69> zstyle -s :completion::complete:cd::users force-list val

+_description:20> name=expl

+_description:22> zstyle -s :completion::complete:cd::users format format

+_description:23> zstyle -s :completion::complete:cd::descriptions format format

+_description:25> zstyle -s :completion::complete:cd::users hidden hidden

+_description:30> zstyle -s :completion::complete:cd::users matcher match

+_description:32> [[ -n '' ]]

+_description:36> zstyle -s :completion::complete:cd::users sort sort

+_description:37> zstyle -s :completion::complete:cd:: sort sort

+_description:46> [[ -z '' ]]

+_description:47> zstyle -a :completion::complete:cd::users ignored-patterns _comp_ignore

+_description:50> zstyle -s :completion::complete:cd::users ignore-line hidden

+_description:61> (( 1 ))

+_description:61> opts=( -F _comp_ignore )

+_description:66> tag=users

+_description:68> shift 2

+_description:69> [[ -z user ]]

+_description:71> [[ -n '' ]]

+_description:75> [[ -n '' ]]

+_description:82> [[ -n '' ]]

+_description:85> set -A expl -F _comp_ignore -J -default-

+_description:89> (( 0 ))

+_description:90> zstyle -a :completion::complete:cd::users fake match

+_description:100> return 0

+_all_labels:39> compadd -F _comp_ignore -J -default- -qS/ -k userdirs

headers

Phil Pennock | 10 Aug 15:03

Re: Multi-Minute Startup?

On 2008-08-08 at 11:16 -0700, Eric D. Friedman wrote:
> I was the original poster with that long completion pause and none of  
> the suggested solutions actually worked out.  Following the advice in  
> this thread, I ran a shell with -l -x on and captured the relavant  
> bits.

Working in an NFS/LDAP/Kerberos environment with a largish number of
users (>10k), I periodically have to debug "what's gone wrong now?".

In ~/.zshenv I set:
----------------------------8< cut here >8------------------------------
[[ -n $TRACE_ZSH_TIME ]] && PS4='+[%D{%M:%S}]%N:%i> '
if [[ -n $TRACE_ZSH ]]; then
        [[ -n "$TRACE_ZSH_FILE" ]] && exec 2> "${TRACE_ZSH_FILE}"
        setopt xtrace
fi
----------------------------8< cut here >8------------------------------

% TRACE_ZSH_TIME=t TRACE_ZSH=t TRACE_ZSH_FILE=$HOME/ztrace.out zsh

This is often informative.

>        The big pause occurs at the last line in the following.  Any  
> ideas as to what else I could try? My goal is to not have the  
> completion system show me every user in my company's home dir when I  
> do the first cd<TAB>; I'd never use that completion and it takes far  
> too long to get it anyway.
[...]
> +_all_labels:39> compadd -F _comp_ignore -J -default- -qS/ -k userdirs

Various things I tried were never sufficient, as userdirs would always
get populated somehow.  Until I cheated.

The short version: ensure that 'userdirs' is a local variable inside the
completion system, populated with the values you want.  I do this by
adding to $_comp_setup -- making sure that none of the rest of the
things that hit LDAP cause slow-downs have led to what I have below.

I keep the users I want to tab-complete in a file, ~/.userdirs and then
hack things so that only that is used.  My code uses zfilter_comments()
which is a function I frequently use to pull in files for data and skip
comments, etc.  Comments in files are handy, reading those files
efficiently in zsh also handy.

The main things are to avoid the automatic population of userdirs when
it's empty and to ensure that things like ssh, scp, etc don't hit LDAP
either.  I have ~/.sshusers contain things like my own usercode, "root"
and various production accounts.  ~/.sshhosts contains hostnames.

Note that I have "users whose home-directories I care about" and "users
I ssh to the accounts of" and these are very different; the former
includes colleagues and the latter includes role accounts.  Because of
this split, we have the loop setting completion of users to $sshusers
for only some commands.  Hrm, wonder why 'users' is used, not
'accounts'?

Note that using "zstyle -e" and "reply=($array)" lets me modify the
array at run-time and have it immediately affect completion.

"is-at-least" is one of the Misc/ functions provided, I autoload it
before this point.

I make sure that I split up the completion dumps by host since different
hosts have different versions of zsh, etc etc, so I don't want that
continually getting trashed within NFS.

Here's hoping that I'm not missing anything relevant when looking over
my config now.  Believe it or not, this is simplified.

----------------------------8< cut here >8------------------------------
function zfilter_comments {
        local f infile="$1"
        while read f; do
                [[ -n ${f%%[$' \t']*\#*} && ${f#[#;]} == $f ]] || continue 
                print -r -- ${f%%[$' \t']*\#*} 
        done < "$infile"
}

typeset -a sshhosts sshusers
[[ -f ~/.sshhosts ]] && sshhosts=( $(zfilter_comments ~/.sshhosts) )
[[ -f ~/.sshusers ]] && sshusers=( $(zfilter_comments ~/.sshusers) )

local c
for c in ssh rsync sftp scp slogin
do
        [[ ${#sshhosts} -gt 0 ]] && \
          zstyle -e ':completion:*:'"$c"':*' hosts 'reply=($sshhosts)'
        [[ ${#sshusers} -gt 0 ]] && \
          zstyle -e ':completion:*:'"$c"':*' users 'reply=($sshusers)'
done
unset c

zstyle -e ':completion:*' my-accounts 'reply=($sshusers)'
zstyle ':completion:*' other-accounts ''
zstyle -e ':completion:*' accounts 'reply=($sshusers)'

function reset_userdirs {
# There is a zsh internal map, userdirs, exposed by zsh/parameter;
# it's read-only though.
	[[ -f ~/.userdirs ]] || return
	local _u
	local -a _ud
	_ud=( $(zfilter_comments ~/.userdirs) )
	for _u in $_ud; do hash -d $_u="/home/$_u"; done
}

if is-at-least 4.2.0; then autoload -Uz compinit ; else autoload -U compinit ; fi
[[ -d "${ZDOTDIR:-$HOME}/.zcompdumps" ]] || mkdir -m 0700 -p "${ZDOTDIR:-$HOME}/.zcompdumps"
compinit -u -d "${ZDOTDIR:-$HOME}/.zcompdumps/${HOST%%.*}"

# Hack to turn off userdirs completion by overriding userdirs.
# This overrides userdirs as a local variable inside the completion
# system, not touching the global variable.  We don't need to load
# zsh/parameter to achieve this.
_comp_setup+=$'\ntypeset -a userdirs\nreset_userdirs'
reset_userdirs
----------------------------8< cut here >8------------------------------

-Phil

headers

Eric D. Friedman | 11 Aug 04:10

Re: Multi-Minute Startup?

Many, many thanks for sharing this -- you've saved me countless hours  
of (probably failing at) figuring this out.  I especially appreciated  
the "fishing lesson" that came with the fish, i.e. the TRACE_ZSH_* tips.

I do wish zsh would make it a little less cumbersome for all of us  
though -- you and I can't be the only people with these environmental  
issues.

Eric

On Aug 10, 2008, at 6:03 AM, Phil Pennock wrote:

> On 2008-08-08 at 11:16 -0700, Eric D. Friedman wrote:
>> I was the original poster with that long completion pause and none of
>> the suggested solutions actually worked out.  Following the advice in
>> this thread, I ran a shell with -l -x on and captured the relavant
>> bits.
>
> Working in an NFS/LDAP/Kerberos environment with a largish number of
> users (>10k), I periodically have to debug "what's gone wrong now?".
>
> In ~/.zshenv I set:
> ----------------------------8< cut here  
> >8------------------------------
> [[ -n $TRACE_ZSH_TIME ]] && PS4='+[%D{%M:%S}]%N:%i> '
> if [[ -n $TRACE_ZSH ]]; then
>        [[ -n "$TRACE_ZSH_FILE" ]] && exec 2> "${TRACE_ZSH_FILE}"
>        setopt xtrace
> fi
> ----------------------------8< cut here  
> >8------------------------------
>
> % TRACE_ZSH_TIME=t TRACE_ZSH=t TRACE_ZSH_FILE=$HOME/ztrace.out zsh
>
> This is often informative.
>
>>       The big pause occurs at the last line in the following.  Any
>> ideas as to what else I could try? My goal is to not have the
>> completion system show me every user in my company's home dir when I
>> do the first cd<TAB>; I'd never use that completion and it takes far
>> too long to get it anyway.
> [...]
>> +_all_labels:39> compadd -F _comp_ignore -J -default- -qS/ -k  
>> userdirs
>
> Various things I tried were never sufficient, as userdirs would always
> get populated somehow.  Until I cheated.
>
> The short version: ensure that 'userdirs' is a local variable inside  
> the
> completion system, populated with the values you want.  I do this by
> adding to $_comp_setup -- making sure that none of the rest of the
> things that hit LDAP cause slow-downs have led to what I have below.
>
> I keep the users I want to tab-complete in a file, ~/.userdirs and  
> then
> hack things so that only that is used.  My code uses  
> zfilter_comments()
> which is a function I frequently use to pull in files for data and  
> skip
> comments, etc.  Comments in files are handy, reading those files
> efficiently in zsh also handy.
>
> The main things are to avoid the automatic population of userdirs when
> it's empty and to ensure that things like ssh, scp, etc don't hit LDAP
> either.  I have ~/.sshusers contain things like my own usercode,  
> "root"
> and various production accounts.  ~/.sshhosts contains hostnames.
>
> Note that I have "users whose home-directories I care about" and  
> "users
> I ssh to the accounts of" and these are very different; the former
> includes colleagues and the latter includes role accounts.  Because of
> this split, we have the loop setting completion of users to $sshusers
> for only some commands.  Hrm, wonder why 'users' is used, not
> 'accounts'?
>
> Note that using "zstyle -e" and "reply=($array)" lets me modify the
> array at run-time and have it immediately affect completion.
>
> "is-at-least" is one of the Misc/ functions provided, I autoload it
> before this point.
>
> I make sure that I split up the completion dumps by host since  
> different
> hosts have different versions of zsh, etc etc, so I don't want that
> continually getting trashed within NFS.
>
> Here's hoping that I'm not missing anything relevant when looking over
> my config now.  Believe it or not, this is simplified.
>
> ----------------------------8< cut here  
> >8------------------------------
> function zfilter_comments {
>        local f infile="$1"
>        while read f; do
>                [[ -n ${f%%[$' \t']*\#*} && ${f#[#;]} == $f ]] ||  
> continue
>                print -r -- ${f%%[$' \t']*\#*}
>        done < "$infile"
> }
>
> typeset -a sshhosts sshusers
> [[ -f ~/.sshhosts ]] && sshhosts=( $(zfilter_comments ~/.sshhosts) )
> [[ -f ~/.sshusers ]] && sshusers=( $(zfilter_comments ~/.sshusers) )
>
> local c
> for c in ssh rsync sftp scp slogin
> do
>        [[ ${#sshhosts} -gt 0 ]] && \
>          zstyle -e ':completion:*:'"$c"':*' hosts 'reply=($sshhosts)'
>        [[ ${#sshusers} -gt 0 ]] && \
>          zstyle -e ':completion:*:'"$c"':*' users 'reply=($sshusers)'
> done
> unset c
>
> zstyle -e ':completion:*' my-accounts 'reply=($sshusers)'
> zstyle ':completion:*' other-accounts ''
> zstyle -e ':completion:*' accounts 'reply=($sshusers)'
>
> function reset_userdirs {
> # There is a zsh internal map, userdirs, exposed by zsh/parameter;
> # it's read-only though.
> 	[[ -f ~/.userdirs ]] || return
> 	local _u
> 	local -a _ud
> 	_ud=( $(zfilter_comments ~/.userdirs) )
> 	for _u in $_ud; do hash -d $_u="/home/$_u"; done
> }
>
> if is-at-least 4.2.0; then autoload -Uz compinit ; else autoload -U  
> compinit ; fi
> [[ -d "${ZDOTDIR:-$HOME}/.zcompdumps" ]] || mkdir -m 0700 -p "$ 
> {ZDOTDIR:-$HOME}/.zcompdumps"
> compinit -u -d "${ZDOTDIR:-$HOME}/.zcompdumps/${HOST%%.*}"
>
> # Hack to turn off userdirs completion by overriding userdirs.
> # This overrides userdirs as a local variable inside the completion
> # system, not touching the global variable.  We don't need to load
> # zsh/parameter to achieve this.
> _comp_setup+=$'\ntypeset -a userdirs\nreset_userdirs'
> reset_userdirs
> ----------------------------8< cut here  
> >8------------------------------
>
> -Phil

headers

Bart Schaefer | 7 Aug 16:56

Re: Multi-Minute Startup?

On Aug 7,  4:52pm, Aaron Davies wrote:
}
} so presumably ~/.zshenv would work. however, putting
} 
} zstyle ':completion:*' users
} zstyle ':completion:*' groups
} 
} in ~/.zshenv doesn't seem to help.

You have to supply the name of at least one user and one group.  Try
something like this:

zstyle ':completion:*' users $LOGNAME
zstyle ':completion:*' groups $(groups)

headers

Bart Schaefer | 7 Aug 09:32

Re: Multi-Minute Startup?

On Aug 7,  2:57pm, Aaron Davies wrote:
}
} It takes zsh multiple minutes (around 3) to start up on a new server I
} just got access to.

I seem to recall this coming up at Panix or somewhere like that, before.
In that case I believe the problem was that it was populating a table of
all the user names on the system.

} only /etc/zshenv and /etc/zshrc are actually present.

So what's *in* those files?  And is "zsh -f" faster?  What about
"zsh -o noglobalrcs" ?

headers

Aaron Davies | 7 Aug 10:33

Re: Multi-Minute Startup?

On Thu, Aug 7, 2008 at 3:32 PM, Bart Schaefer <schaefer <at> brasslantern.com> wrote:
> On Aug 7,  2:57pm, Aaron Davies wrote:
> }
> } It takes zsh multiple minutes (around 3) to start up on a new server I
> } just got access to.
>
> I seem to recall this coming up at Panix or somewhere like that, before.
> In that case I believe the problem was that it was populating a table of
> all the user names on the system.
>
> } only /etc/zshenv and /etc/zshrc are actually present.
>
> So what's *in* those files?  And is "zsh -f" faster?  What about
> "zsh -o noglobalrcs" ?

zshenv just loads /etc/profile

zshrc seems to be mostly setting options, binding editing keys, and
initializing the completion system

zsh -f and zsh -o noglobalrcs both start instantaneously
--

-- 
Aaron Davies
aaron.davies <at> gmail.com

headers

Tomasz Pala | 7 Aug 10:57

Re: Multi-Minute Startup?

On Thu, Aug 07, 2008 at 16:33:21 +0800, Aaron Davies wrote:

> zshenv just loads /etc/profile

It's certainly wrong and should be done in /etc/zprofile.
Please read "5.1 Startup/Shutdown Files".

> zshrc seems to be mostly setting options, binding editing keys, and
> initializing the completion system

zshrc file is used for _interactive_ login, so it contains functions,
aliases and terminal options (like key bindings, shell options and PS*,
which in fact are shell functions).
zprofile file is used for _login_ shell (e.g. xterm -ls, zsh -l or -zsh)
and so it contains environment settings.

--

-- 
Tomasz Pala <gotar <at> pld-linux.org>

headers

Aaron Davies | 7 Aug 11:20

Re: Multi-Minute Startup?

On Thu, Aug 7, 2008 at 4:57 PM, Tomasz Pala <gotar <at> polanet.pl> wrote:
> On Thu, Aug 07, 2008 at 16:33:21 +0800, Aaron Davies wrote:
>
>> zshenv just loads /etc/profile
>
> It's certainly wrong and should be done in /etc/zprofile.
> Please read "5.1 Startup/Shutdown Files".

fair enough. i'm just reporting on the way it's currently set up, i
didn't write these. this is a suse enterprise 9.3 box, anyone know if
this is a standard suse config?

>> zshrc seems to be mostly setting options, binding editing keys, and
>> initializing the completion system
>
> zshrc file is used for _interactive_ login, so it contains functions,
> aliases and terminal options (like key bindings, shell options and PS*,
> which in fact are shell functions).
> zprofile file is used for _login_ shell (e.g. xterm -ls, zsh -l or -zsh)
> and so it contains environment settings.

so as far as segregation of z* functions, this one is OK?
--

-- 
Aaron Davies
aaron.davies <at> gmail.com