Re: how much space is needed for htdig index?

If you have the cache on a separate partition you can try smart blocksize it.
This would be more proper to the overall small files.
You'd need to backup the cache temporarily somewhere to reformat the drive though.

Shifting from blocksize 4096 to 1024 on ext3 with journal reduced the used disk space by more than 20%.
Output of df -h (size - used- free - used%):

Blocksize 4096:

/dev/hda12	1.9G		1005M	873M	54%	 /var/cache/wwwoffle (ext3)

Blocksize 1024: 

/dev/hda12	1.9G		739M	1.2G		40%  /var/cache/wwwoffle (ext3)