ELCIN HAKTANIR | 10 Oct 17:24
Favicon

Re: slapadd newbie


Sorry not 100,000,000 just 100,000


ELCIN HAKTANIR/OKSIJEN/TR/VODAFONE

10/10/2008 06:22 PM

To
ldap-63aXycvo3TyHXe+LvDLADg@public.gmane.org
cc
Subject
Re: [ldap] Re: slapadd newbieLink




100,000,000 subscribers is:2.1G total.

bash-2.05# du -c -h *.bdb

157M    dn2id.bdb
2.0G    id2entry.bdb
2.0M    objectClass.bdb
2.1G    total

again my Configuration information about my Environment:
---------------------------------------------------------------------
openldap-2.4.10-sol9-sparc-local.gz  installed on a System with Configuration:  

bash-2.05# prtdiag
System Configuration:  Sun Microsystems  sun4u Sun Fire 280R (2 X UltraSPARC-III+)
System clock frequency: 150 MHz
Memory size: 2048 Megabytes

========================= CPUs ===============================================

          Run    E$    CPU     CPU
Brd  CPU  MHz    MB   Impl.    Mask
---  ---  ----  ----  -------  ----
 A    0    900   8.0  US-III+  2.3
 B    1    900   8.0  US-III+  2.3

========================= Memory Configuration ===============================

           Logical  Logical  Logical
      MC   Bank     Bank     Bank         DIMM    Interleave  Interleaved
 Brd  ID   num      size     Status       Size    Factor      with
----  ---  ----     ------   -----------  ------  ----------  -----------
 CA    0     0      1024MB   no_status     512MB     2-way        0
 CA    0     2      1024MB   no_status     512MB     2-way        0




Gavin Henry <ghenry <at> suretecsystems.com>
Sent by: bounce-ldap-5624112 <at> listserver.itd.umich.edu

10/10/2008 06:14 PM

Please respond to
Gavin Henry <ghenry-0iySFhgulYrkQYj/0HfcvtBPR1lH4CV8@public.gmane.org>

To
Quanah Gibson-Mount <quanah <at> zimbra.com>
cc
ELCIN HAKTANIR <elcin.haktanir <at> vodafone.com>, ldap-63aXycvo3TyHXe+LvDLADg@public.gmane.org
Subject
[ldap] Re: slapadd newbie






----- "Quanah Gibson-Mount" <quanah-zAQalKWTt5vQT0dZR+AlfA@public.gmane.org> wrote:

> --On Friday, October 10, 2008 5:39 PM +0300 ELCIN HAKTANIR
> <elcin.haktanir-ANTagKRnAhdWk0Htik3J/w@public.gmane.org> wrote:
>
> >
> > Question1:
> > ---------------
> > Is it rational that slapadd took 31 minutes for 100,000
> entries(23Kbyte
> > per subscriber i guess) without index.?
> > I think it is so slow.isn't it?
> > What have i done wrong then?Could you please help to reduce this
> time ?
>
> Verify your DB_CONFIG file.  Set the tool-threads value in slapd.conf,
>
> assuming OpenLDAP 2.3 or later.  Don't use debug flags with slapadd.
> Also
> the disk speed is going to have an impact.

Also see the -q flag:

-q     enable quick (fewer integrity checks) mode.  Does fewer consistency checks on the input data, and no consistency checks when writing the database.  Improves the load time but if any errors or interruptions occur the resulting database will be unusable.


--
Kind Regards,

Gavin Henry.

T +44 (0) 1224 279484
M +44 (0) 7930 323266
F +44 (0) 1224 824887
E ghenry-0iySFhgulYrkQYj/0HfcvtBPR1lH4CV8@public.gmane.org

Open Source. Open Solutions(tm).

http://www.suretecsystems.com/

Suretec Systems is a limited company registered in Scotland. Registered
number: SC258005. Registered office: 13 Whiteley Well Place, Inverurie,
Aberdeenshire, AB51 4FP.












--------------------------------------

Bu elektronik posta ve onunla iletilen bütün dosyalar gizlidir sadece yukarıda isimleri belirtilen kişiler arasında özel haberleşme amacını taşımaktadır. Size yanlışlıkla ulaşmıssa bu elektonik postanın içeriğini açıklamanız , kopyalamanız, yönlendirmeniz ve kullanmanız kesinlikle yasaktır. Lütfen mesajı geri gönderiniz ve sisteminizden siliniz. Vodafone Teknoloji Hizmetleri A.Ş. bu mesajın içeriği ile ilgili olarak hiç bir hukuksal sorumluluğu kabul etmez.

This electonic mail and any files transmitted with it are intended for the private use of the persons named above. If you received this message in error, forwarding, copying or use of any of the information is strictly prohibited. Please immediately notify the sender and delete it from your system. Vodafone Teknoloji Hizmetleri A.S. does not accept legal responsibility for the contents of this message.
--------------------------------------













--------------------------------------

Bu elektronik posta ve onunla iletilen bütün dosyalar gizlidir sadece yukarıda isimleri belirtilen kişiler arasında özel haberleşme amacını taşımaktadır. Size yanlışlıkla ulaşmıssa bu elektonik postanın içeriğini açıklamanız , kopyalamanız, yönlendirmeniz ve kullanmanız kesinlikle yasaktır. Lütfen mesajı geri gönderiniz ve sisteminizden siliniz. Vodafone Teknoloji Hizmetleri A.Ş. bu mesajın içeriği ile ilgili olarak hiç bir hukuksal sorumluluğu kabul etmez.

This electonic mail and any files transmitted with it are intended for the private use of the persons named above. If you received this message in error, forwarding, copying or use of any of the information is strictly prohibited. Please immediately notify the sender and delete it from your system. Vodafone Teknoloji Hizmetleri A.S. does not accept legal responsibility for the contents of this message.
--------------------------------------


Quanah Gibson-Mount | 10 Oct 17:26

Re: slapadd newbie

--On Friday, October 10, 2008 6:24 PM +0300 ELCIN HAKTANIR 
<elcin.haktanir@...> wrote:

>
> Sorry not 100,000,000 just 100,000

Don't top post, please.

So if 100,000 users is 2.1 GB, and you only have 2GB of memory on your 
server, then your server is vastly underpowered for what you want to do. 
Also, your DB_CONFIG file is not tuned correctly for your needs, but given 
the lack of memory on your server, you can't tune it sufficiently to meet 
your needs.

The DB_CONFIG cachesize at a minimum needs to be the size of dn2id.bdb + 
id2entry.bdb, which already exceeds your available memory.

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration

ELCIN HAKTANIR | 13 Oct 10:35
Favicon

Re: slapadd newbie


Hi i am still suffering from slow slapadd for a 100,000 subscriber.It is 27 minutes.
this is not what i am expecting .

I have changed the test environment and also the openLDAP release now i have installed
symas-openldap-silver-2.4.11.0.sun4u.pkg
to a System Configuration:  
Sun Microsystems  sun4v SPARC Enterprise T5220
Memory size: 32640 Megabytes


Is it rational that slapadd took 26 minutes for 100,000 entries(23Kbyte per subscriber i guess) without index.?
I think it is so slow.isn't it?
What have i done wrong then?Could you please help to reduce this time ?
And how can i learn what db_stat exactly mean to me while doing slapadd?



Configuration information about my Environment:
---------------------------------------------------------------------
symas-openldap-silver-2.4.11.0.sun4u.pkg installed on a System with
Configuration:  Sun Microsystems  sun4v SPARC Enterprise T5220
Memory size: 32640 Megabytes

My slapadd command:
-------------------------------------
/opt/symas/bin/sparcv9/slapadd -l /opt/symas/etc/openldap/ldifs/subscribersPart100.ldif  -f  /opt/symas/etc/openldap/slapd.conf -b o=sdftest -q

my DB_CONFIG file is:
---------------------------------------
set_cachesize        10        0        0
set_flags       DB_LOG_AUTOREMOVE
set_flags       DB_TXN_NOSYNC
set_lg_max        10485760
set_lg_bsize        2097152
set_lg_dir        /opt/symas/etc/openldap/transactionlog

my slapd.conf file is:
---------------------------------
tool-threads 2
access to dn="" by * read
access to *
        by self write
        by users read
        by anonymous auth

database        bdb
suffix "o=sdftest"
rootdn "cn=sdf,o=sdftest"
rootpw admin234
index        default                eq
index        objectClass
index        cn
directory        /var/symas/openldap-data/
checkpoint        256000        60


one of the db_stat result

root <at> typhoon:/# /opt/symas/bin/sparcv9/db_stat -h /var/symas/openldap-data/ -m
2GB 25MB        Total cache size
1       Number of caches
1       Maximum number of caches
2GB 25MB        Pool individual cache size
0       Maximum memory-mapped file size
0       Maximum open file descriptors
0       Maximum sequential buffer writes
0       Sleep after writing maximum sequential buffers
0       Requested pages mapped into the process' address space
91M     Requested pages found in the cache (99%)
294     Requested pages not found in the cache
307399  Pages created in the cache
294     Pages read into the cache
307935  Pages written from the cache to the backing file
31233   Clean pages forced from the cache
33201   Dirty pages forced from the cache
103701  Dirty pages written by trickle-sync thread
243256  Current total page count
243256  Current clean page count
0       Current dirty page count
262147  Number of hash buckets used for page location
90M     Total number of times hash chains searched for a page (90995964)
2       The longest hash chain searched for a page
120M    Total number of hash chain entries checked for page (120028164)
31      The number of hash bucket locks that required waiting (0%)
4       The maximum number of times any hash bucket lock was waited for (0%)
9       The number of region locks that required waiting (0%)
0       The number of buffers frozen
0       The number of buffers thawed
0       The number of frozen buffers freed
307713  The number of page allocations
257629  The number of hash buckets examined during allocations
60205   The maximum number of hash buckets examined for an allocation
64434   The number of pages examined during allocations
5844    The max number of pages examined for an allocation
21      Threads waited on page I/O
Pool File: id2entry.bdb
16384   Page size
0       Requested pages mapped into the process' address space
2999318 Requested pages found in the cache (99%)
3       Requested pages not found in the cache
136414  Pages created in the cache
3       Pages read into the cache
136659  Pages written from the cache to the backing file
Pool File: dn2id.bdb
4096    Page size
0       Requested pages mapped into the process' address space
53M     Requested pages found in the cache (99%)
2       Requested pages not found in the cache
168604  Pages created in the cache
2       Pages read into the cache
168607  Pages written from the cache to the backing file
Pool File: objectClass.bdb
4096    Page size
0       Requested pages mapped into the process' address space
25M     Requested pages found in the cache (99%)
2       Requested pages not found in the cache
694     Pages created in the cache
2       Pages read into the cache
695     Pages written from the cache to the backing file
Pool File: cn.bdb
4096    Page size
0       Requested pages mapped into the process' address space
9191165 Requested pages found in the cache (99%)
287     Requested pages not found in the cache
1687    Pages created in the cache
287     Pages read into the cache
1974    Pages written from the cache to the backing file









--------------------------------------

Bu elektronik posta ve onunla iletilen bütün dosyalar gizlidir sadece yukarıda isimleri belirtilen kişiler arasında özel haberleşme amacını taşımaktadır. Size yanlışlıkla ulaşmıssa bu elektonik postanın içeriğini açıklamanız , kopyalamanız, yönlendirmeniz ve kullanmanız kesinlikle yasaktır. Lütfen mesajı geri gönderiniz ve sisteminizden siliniz. Vodafone Teknoloji Hizmetleri A.Ş. bu mesajın içeriği ile ilgili olarak hiç bir hukuksal sorumluluğu kabul etmez.

This electonic mail and any files transmitted with it are intended for the private use of the persons named above. If you received this message in error, forwarding, copying or use of any of the information is strictly prohibited. Please immediately notify the sender and delete it from your system. Vodafone Teknoloji Hizmetleri A.S. does not accept legal responsibility for the contents of this message.
--------------------------------------


Quanah Gibson-Mount | 13 Oct 18:42

Re: slapadd newbie

--On Monday, October 13, 2008 11:35 AM +0300 ELCIN HAKTANIR 
<elcin.haktanir@...> wrote:

This:

> my DB_CONFIG file is:
> ---------------------------------------
> set_cachesize        10        0        0

and this:

> root <at> typhoon:/# /opt/symas/bin/sparcv9/db_stat -h
> /var/symas/openldap-data/ -m
> 2GB 25MB        Total cache size

don't match.  I suggest you fix your DB_CONFIG to do what you expect it to.

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration


Gmane