Spark is a fast, Hadoop-compatible cluster computing system that lets you write programs using standard Scala functional operations (map, filter, reduce, etc) and automatically distributes them over a cluster. It supports in-memory computation to significantly speed up multi-stage applications, and can be used directly from the Scala console to query large datasets interactively. Check out http://spark-project.org for more details and http://spark-project.org/examples.html for some sample code.

The 0.5 release brings several new features, including:
- New operators: parallel sorting, fold, aggregate, and a new sampling operator.
- Performance improvements: faster shuffle implementation and better cache management.
- Debugging improvements: Spark now reports remote exceptions at the master and filters duplicates.
- Support for running on Apache Mesos 0.9.
- Improved EC2 launch scripts.

For more details, or to download the release, visit http://www.spark-project.org/release-0.5.0.html. You can also keep up with Spark development on GitHub: https://github.com/mesos/spark.

Matei & the Spark team