Open Knowledge Foundation discussion list

Re: Distributed Storage: Suggestions?

On Wed, May 6, 2009 at 6:42 AM, Rufus Pollock <rufus.pollock <at> okfn.org> wrote:
> 2009/5/1 Lukasz Szybalski <szybalski <at> gmail.com>:
>> On Fri, May 1, 2009 at 11:22 AM, Rufus Pollock <rufus.pollock <at> okfn.org> wrote:
>>> 2009/4/28 Lukasz Szybalski <szybalski <at> gmail.com>:
> [...]
>
>> 1. sector is for highly "read" intensive apps, optimized for extremely
>> fast read.
>
> OK.  I feel a feature-grid coming on :)  For us, I guess that is not
> ultra-high priority.
>
>> 2. kfs(kosmos) seems to be similar to google file system, where you
>> have main server (metaserver), and run chunk servers (how much space
>> you have available) , data gets divided into 64mb and send to chunk
>> servers. You always have 3 copies of the chunk.  The filesystem (at
>> metaserver) seems like one big file server. (+python bindings)
>
> OK. Useful info.

What is the structure of the tahoe fs? how is it different from kfs?

Thanks,
Lucas

>
> At the moment Allmydata-Tahoe is looking pretty promising. We've
> started a basic grid with info here:
>
> <http://grid.okfn.org/>

(Continue reading)

headers

Rufus Pollock | 7 May 17:13

Re: Distributed Storage: Suggestions?

2009/5/7 Lukasz Szybalski <szybalski <at> gmail.com>:
> On Wed, May 6, 2009 at 6:42 AM, Rufus Pollock <rufus.pollock <at> okfn.org> wrote:
>> 2009/5/1 Lukasz Szybalski <szybalski <at> gmail.com>:
[...]
>>> 2. kfs(kosmos) seems to be similar to google file system, where you
>>> have main server (metaserver), and run chunk servers (how much space
>>> you have available) , data gets divided into 64mb and send to chunk
>>> servers. You always have 3 copies of the chunk.  The filesystem (at
>>> metaserver) seems like one big file server. (+python bindings)
>>
>> OK. Useful info.
>
> What is the structure of the tahoe fs? how is it different from kfs?

I haven't looked at kfs in detail but:

  * Seems to be "cluster-oriented" and GFS-like (it says it is built
on ideas from GFS and can be dropped in to replaces HadoopFS).
  * It is built in C++ with some high-level bindings for python.
  * v0.3

Detailed overview of Tahoe can be found in
<http://allmydata.org/source/tahoe/trunk/docs/architecture.txt>

  * Very much about distributed *file storage*.
  * Focused storing and nodes are expected to highly distributed (so
no cluster/cloud computing here).
  * Data is encrypted (so you can put private data on it)
  * Almost entirely in python
  * v1.4

(Continue reading)