Journalled File System general discussion

Alessio Focardi | 8 May 2012 10:08

1 billion very small files and JFS

Hi,

I need some help in designing a storage structure for 1 billion of small files (<256 Bytes), and I was
wondering how JFS will fit in this scenario.

JFS seems to be the ideal match because it's able to do block suballocation, but I'm not sure if this is
function available in the linux port. 

So the very big question is: is JFS for linux able to suballocate blocks? 

Also, what suggestion would you gave me to better optimize the fs to waste less space as possible (slack),
even sacrificing some performance?

Keep in mind that I never worked with jfs - I just read some documentation and browsed this mailing list - so
forgive me if my question is silly

Tnx for any suggestion!

Alessio Focardi
------------------

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

headers

Dave Kleikamp | 8 May 2012 15:20

Re: 1 billion very small files and JFS

On 05/08/2012 03:08 AM, Alessio Focardi wrote:
> Hi,
> 
> I need some help in designing a storage structure for 1 billion of small files (<256 Bytes), and I was
wondering how JFS will fit in this scenario.
> 
> JFS seems to be the ideal match because it's able to do block suballocation, but I'm not sure if this is
function available in the linux port.

No, the Linux JFS implementation only supports a 4KB block size.

> So the very big question is: is JFS for linux able to suballocate blocks? 

Sorry, no.

> Also, what suggestion would you gave me to better optimize the fs to waste less space as possible (slack),
even sacrificing some performance?

You are probably better off with ext4 using a 1KB block size. Reiserfs
used to boast about working very well with lots of small files. It also
supports a 1KB block size.

> Keep in mind that I never worked with jfs - I just read some documentation and browsed this mailing list - so
forgive me if my question is silly

Not silly, but I'm not sure jfs is the best solution for you.

> Tnx for any suggestion!
> 
>

(Continue reading)

headers

Bhaskar, K.S | 8 May 2012 15:30

Re: 1 billion very small files and JFS

As the manager of a key-value database (http://fis-gtm.com & 
http://sourceforge.net/projects/fis-gtm) and a long-time jfs user, as I 
read Alessio's e-mail even before Dave's response, I felt that if you 
want a billion small files, a key-value datastore residing in a file 
system but wrapped with an appropriate userland API might fit the 
application need better than a file system.

Asseio, if you are interested in exploring the idea, please write to me 
offlist, since the discussion is likely to be off-topic for a jfs mailing 
list.  If there is enough general interest (write to me off-list or reply 
to this thread), I will set up a public mailing list at Source Forge.

Regards
-- Bhaskar

On 05/08/2012 09:20 AM, Dave Kleikamp wrote:
> On 05/08/2012 03:08 AM, Alessio Focardi wrote:
>> Hi,
>>
>> I need some help in designing a storage structure for 1 billion of small files (<256 Bytes), and I was
wondering how JFS will fit in this scenario.
>>
>> JFS seems to be the ideal match because it's able to do block suballocation, but I'm not sure if this is
function available in the linux port.
>
> No, the Linux JFS implementation only supports a 4KB block size.
>
>> So the very big question is: is JFS for linux able to suballocate blocks?
>
> Sorry, no.

(Continue reading)