Scaling Kafka to Support PayPal’s Data Growth | PayPal Technology Blog

Scaling tech to extreme scales is usually an interesting story. The levels that PayPal has scaled Kafka are impressive!

Today, our Kafka fleet consists of over 1,500 brokers that host over 20,000 topics and close to 2,000 Mirror Maker nodes which are used to mirror the data among the clusters, offering 99.99% availability for our Kafka clusters. During the 2022 Retail Friday, Kafka traffic volume peaked at about 1.3 trillion messages per day! At present, we have 85+ Kafka clusters, and every holiday season we flex up our Kafka infrastructure to handle the traffic surge. The Kafka platform continues to seamlessly scale to support this traffic growth without any impact to our business.

Detailed article describing how they have achieved this level of scale and resiliency, including their own custom libraries they built to interface with Kafka.