gmane.comp.lang.erlang.general

Bob Calco | 2 Jul 15:43

Call for Contributions: Mnesia best practices

Everyone:

I'm looking for thoughts from fellow Erlangers about database design &
implementation in Mnesia. With a heavy SQL background I, like many
relatively new Erlang folks I'm sure, have a tendency to think in terms of
the capabilities of traditional RDBMSs, and to try to normalize every data
model with which I come into contact. 

I'm also used to letting administrators deal with most database operations
and issues, whereas with Mnesia the programmer has basically supreme control
(both to do some really cool and flexible things, and to do some really dumb
things).

So the question is generic, not intended to solve a specific problem. I have
some ideas and will contribute separately. 

The question is: What is the best advice you could give a data architect
about designing and implementing a database in Mnesia from scratch? Examples
of the kinds of issues I'd like to see folks address:

* How to create an optimal data model for performance (vs. reporting,
comparing the SQL way to the Mnesia way). This question is really about
normalization in Mnesia vs. SQL, and tricks like storing whole records in
table fields.

* How to partition data between subsystems, without losing the illusion
they're all one big happy system.

* How to handle complex clustering and failover scenarios.

(Continue reading)

headers

Christian S | 2 Jul 17:58

Re: Call for Contributions: Mnesia best practices

Now, this is not exactly mnesia, but google application engine's
datastore which is based off google bigtable. HOWever, they do present
some problems, their causes, and strategies for handling scalability
when you have a distributed database..

http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine

The other google io presentatiosn over at
http://sites.google.com/site/io/ on datastore are also worth to see.

Also a comment:

> * How to partition data between subsystems, without losing the illusion
> they're all one big happy system.

They are not one big happy system. The illusion must be forgotten and
reality must be faced.
Things like: Stop doing joins. Instead begin to duplicate data, so it
is available directly on first access.
Or: Send the code to execute where the data is, instead of sending the
data to the machine that has the code.

Look at the hoops they go through to implement efficient
statistics-counters in the video.  ACID properties are a costly
luxury, now you have to start conserve your use of it, find when
almost or eventual consistency is enough and use that fact.

Yes, database programming just got trickier, but if your
write-transactions takes 10ms and must wait in a single line, then you
can only do 100 of them per second. If that is orders of magnitude

(Continue reading)

headers

Taavi Talvik | 3 Jul 01:19

Re: Call for Contributions: Mnesia best practices

On Jul 2, 2008, at 4:46 PM, Bob Calco wrote:
> I'm looking for thoughts from fellow Erlangers about database design &
> implementation in Mnesia. With a heavy SQL background I, like many
> relatively new Erlang folks I'm sure, have a tendency to think in  
> terms of
> the capabilities of traditional RDBMSs, and to try to normalize  
> every data
> model with which I come into contact.

uninformed comments..

> The question is: What is the best advice you could give a data  
> architect
> about designing and implementing a database in Mnesia from scratch?  
> Examples
> of the kinds of issues I'd like to see folks address:

First of all.

Mnesia is not actually fully fledged relational database.
It is simple key-value thingy with some query capabilities thrown in.

On the other hand it is really well distributed.

> * How to create an optimal data model for performance (vs. reporting,
> comparing the SQL way to the Mnesia way). This question is really  
> about
> normalization in Mnesia vs. SQL, and tricks like storing whole  
> records in
> table fields.

(Continue reading)

headers

Scott Lystig Fritchie | 23 Aug 02:20

Re: Call for Contributions: Mnesia best practices

There's nothing like replying to a message that's well over a month
old.

Bob Calco <bobcalco <at> tampabay.rr.com> wrote:

bc> I'm looking for thoughts from fellow Erlangers about database design
bc> & implementation in Mnesia.
bc> [...]
bc> That kind of thing. I want to put together something of a master
bc> knowledge base on this big subject that the community can use both
bc> to promote Erlang and to promote Erlang's "best practices"
bc> implementation in the important area of serving data to
bc> applications.

Bob, I think that such a thing would be welcome.  I'd contribute from
time to time, though I confess I've less time for Open Source work than
I prefer.  Have you considered a place to host it?  I'm a fan of
Wiki-style documentation, and TrapExit(*) provides a wiki for
documenting exactly this sort of stuff.

-Scott

(*) http://www.trapexit.org and more specifically,
http://www.trapexit.org/Erlang_Wiki