XUtils

AutoMQ

AutoMQ is a cloud-native, serverless reinvented Kafka that is easily scalable, manage-less and cost-effective.

Java
https://github.com/AutoMQ/automq-for-kafka

AutoMQ: A Cloud-Native fork of Kafka by separating storage to S3 and EBS

GitHub release (with filter) Twitter URL

🔶 Why AutoMQ

  • Cost effective: The first true cloud-native streaming storage system, designed for optimal cost and efficiency on the cloud. Refer to this report to see how we cut Apache Kafka billing by 90% on the cloud.
  • High Reliability: Leverage cloud-shared storage services(EBS and S3) to achieve zero RPO, RTO in seconds and 99.999999999% durability.
  • Serverless:
    • Auto Scaling: Monitor cluster metrics and automatically scale in/out to align with your workload, enabling a pay-as-you-go model.
    • Scaling in seconds: The computing layer (broker) is stateless and can scale in/out within seconds, making AutoMQ a truly serverless solution.
    • Infinite scalable: Utilize cloud object storage as the primary storage solution, eliminating concerns about storage capacity.
  • Manage-less: The built-in auto-balancer component automatically schedules partitions and network traffic between brokers, eliminating manual partition reassignment.
  • High performance:
    • Low latency: Accelerate writing with high-performance EBS as WAL, achieving single-digit millisecond latency.
    • High throughput: Leverage pre-fetching, batch processing, and parallel technologies to maximize the capabilities of cloud object storage. > Refer to the AutoMQ Performance White Paper to see how we achieve this.
  • A superior alternative to Apache Kafka: 100% compatible with Apache Kafka and does not lose any key features, but cheaper and better.

✨Architecture

image

AutoMQ adopts a Shared-Storage architecture, replacing the storage layer of Apache Kafka with a shared streaming storage library called S3Stream in a storage-compute separation manner, making the Broker completely stateless.

Compared to the classic Kafka Shared-Nothing or Tiered-Storage architectures, AutoMQ’s computing layer (Broker) is truly stateless, enabling features such as Auto-Scaling, Self-Balancing, and Partition Reassignment in Seconds that significantly reduce costs and improve efficiency.

⛄ Get started with AutoMQ

Deploy Locally on a Single Host

curl https://download.automq.com/community_edition/standalone_deployment/install_run.sh | bash

The easiest way to run AutoMQ. You can experience features like Partition Reassignment in Seconds and Continuous Self-Balancing in your local machine. Learn more

There are more deployment options available:

💬 Community

You can join the following groups or channels to discuss or ask questions about AutoMQ:

Articles

  • coming soon...