Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop,MPI, Hypertable, Spark (a new framework for low-latency interactive and iterative jobs), and other applications. Mesos is open source in the Apache Incubator.
You can use Mesos to:
- Run Hadoop, MPI, Spark, and other frameworks on a dynamically shared pool of nodes.
- Run multiple instances of Hadoop on the same cluster to isolate production and experimental jobs, or even multiple versions of Hadoop.
- Run long-lived services (e.g. Hypertable and HBase) on the same nodes as batch applications and share resources between them.
- Build new cluster computing frameworks without reinventing low-level facilities for farming out tasks, and have them coexist with existing ones.
- Fault-tolerant replicated master using ZooKeeper.
- Scalability to 10,000s of nodes using fast, event-driven C++ implementation.
- Isolation between tasks with Linux Containers.
- Multi-resource scheduling (memory and CPU aware).
- Efficient application-controlled scheduling mechanism (resource offers) that lets frameworks achieve their own placement goals (e.g. data locality).
- Java, Python and C++ APIs for developing new parallel applications.
- Web UI for viewing cluster state.
Please, see the following tutorials which explain Apache Mesos in detail.