If you want to scale to the petabyte level, or the billion requests a day, you need to be: Distributed. The data has to be distributed across multiple machines.

Joinless. No joins, and no referential integrity, at least at the data store level.

De-Normalized. No one said this explicily, but I presume there is a lot of de-normalization going on if you are avoiding joins.

Transcationless. No transactions. Those constraints represent something fundamentally different from a relational database.

The Mnesia DBMS runs in the same adress space as the application owning the data, yet the application cannot destroy the contents of the data base. This provides for both fast accesses and efficient fault tolerance, normally conflicting requirements. The implementation is based on features in the Erlang programming language, in which Mnesia is embedded.

Fast realtime key/value lookup. Complicated non realtime queries mainly for operation and maintenance. Distributed data due to distributed applications. High fault tolerance. Dynamic re configuration. Complex objects.

Mnesia is also interesting due to its tight coupling to the programming language Erlang, thus almost turning Erlang into a database programming language. This has many benefits, the foremost being that the impedance mismatch between data format used by the DBMS and data format used by the programming language which is being used to manipulate the data, completely disappears.

X = #person{

name = klacke,

data = {male, 36, 971191},

married_to = eva,

children = [marten, maja, klara]

}.

Complex values . Mnesia supports the ability to handle complex values naturally and efficiently (i.e. via a single lookup operation). "Organizing the telecommunications data in third (or even first) normal form is usually not possible." Storing complex values in columns means that no joins need to be made.

. Mnesia supports the ability to handle complex values naturally and efficiently (i.e. via a single lookup operation). Storing complex values in columns means that no joins need to be made. Data format and address space . The DBMS runs in the same process space as the application. This allows a lookup to return a pointer to the object without the need to marshall the object to or from different data formats, or access it over the wire. The paper covers a common criticism of this approach where an application crash can corrupt the database by explaining that in Erlang an application can not crash in a way that would impact the DBMS. "Erlang processes have the efficiency advantage of running in the same address space but they do not have the possibility to explicitly read or write each others memory."

. The DBMS runs in the same process space as the application. This allows a lookup to return a pointer to the object without the need to marshall the object to or from different data formats, or access it over the wire. The paper covers a common criticism of this approach where an application crash can corrupt the database by explaining that in Erlang an application can not crash in a way that would impact the DBMS. Fault Tolerance . Tables can be replicated to several compute nodes. Write operations apply to all replicas within the context of a transaction, with the ability to update replicas that are not available through updates upon recovery. "This mechanism makes it possible to design systems where several geographically distinct systems cooperate to provide a continuously running nonstop system."

. Tables can be replicated to several compute nodes. Write operations apply to all replicas within the context of a transaction, with the ability to update replicas that are not available through updates upon recovery. Distribution and location transparency . Provide the ability for the application developer to access tables transparently - i.e. regardless of if it is remote or local, or a replica. However, there also exists a means to determine the location of the table and have code execution occur close to the data in the case where performance is critical.

. Provide the ability for the application developer to access tables transparently - i.e. regardless of if it is remote or local, or a replica. However, there also exists a means to determine the location of the table and have code execution occur close to the data in the case where performance is critical. Transactions and ACID . Mnesia supports ACID (Atomicity, Consistency, Isolation and Durability), but also offers the ability to perform in memory only operations on tables (at the expense of durability).

. Mnesia supports ACID (Atomicity, Consistency, Isolation and Durability), but also offers the ability to perform in memory only operations on tables (at the expense of durability). The ability to bypass the transaction manager . The concept of a "dirty interface" is introduced as a light weight locking mechanism that avoids the overhead of using the transaction manager. This capability is useful for performance critical operations, such as performing a record lookup. "These dirty operations are true realtime DBMS operations: they take the same predictable amount of time regardless of the size of the database."

. The concept of a "dirty interface" is introduced as a light weight locking mechanism that avoids the overhead of using the transaction manager. This capability is useful for performance critical operations, such as performing a record lookup. Queries . Queries are expressed using a so-called "list comprehension syntax". For example, to find the names of people with more than X children the expression is formulated as: query [P.name || P < table(person), length(P.children) > X] end .

. Queries are expressed using a so-called "list comprehension syntax". For example, to find the names of people with more than X children the expression is formulated as: . Schema alteration . The Erlang language itself supports the ability to change code of executing processing without stopping the process. This allows the Mnesia database to alter schema dynamically. "Since Mnesia is intended for nonstop applications, all system activities such as performing a backup, changing the schema, dumping tables to secondary storage and copying replicas have to be performed in the background while still allowing the applications to access and modify tables as usual."

The Mnesia system is currently being used to build real products in Ericsson today, thus it is no longer a mere prototype system, it has matured enough to be labeled a product.

I think that increased data volumes will impact day to day programing work far more than multicore will. A constant theme in the work I've done in the last few years has been dealing with larger and larger datasets. What Joe Gregorio calls "Megadata" (but now wishes he didn't). Large data sets are no longer esoteric concerns for a few big companies, but are becoming commonplace.

The big volumes mean you need to be able to write data and not care where it went. And you need keyed lookup for reads built on top of the FS, not in the RDBMS (on the basis that an RDBMS with no joins, constraints or triggers is an indexed filesystem). That will end looking looking something like hadoop, mogilefs or S3 - a data parallel architecture.