Balancing Replicas Across Racks

The rack awareness feature spreads replicas of the same partition across different racks. This extends the guarantees Kafka provides for broker-failure to cover rack-failure, limiting the risk of data loss should all the brokers on a rack fail at once. The feature can also be applied to other broker groupings such as availability zones in EC2.

You can specify that a broker belongs to a particular rack by adding a property to the broker config: When a topic is created, modified or replicas are redistributed, the rack constraint will be honoured, ensuring replicas span as many racks as they can (a partition will span min(#racks, replication-factor) different racks).

The algorithm used to assign replicas to brokers ensures that the number of leaders per broker will be constant, regardless of how brokers are distributed across racks. This ensures balanced throughput.

However if racks are assigned different numbers of brokers, the assignment of replicas will not be even. Racks with fewer brokers will get more replicas, meaning they will use more storage and put more resources into replication. Hence it is sensible to configure an equal number of brokers per rack. 

来源： http://kafka.apache.org/documentation/#operations   

相关参数：

==================================================================================

一、测试方案

1、节点： 172.28.1.114  3        broker.rack=yanjiao

                 172.28.1.88    4        broker.rack=yanjiao

                 172.28.1.87    5        broker.rack=ucloud

                 172.28.1.89    6        broker.rack=ucloud

2、以 topic zs_invoice测试

     {"topics": [{"topic": "zs_invoice"}],

  "version":1

    }

3、生成分配策略：

 bin/kafka-reassign-partitions.sh --zookeeper 172.28.1.89:2181 --topics-to-move-json-file topics-to-move.json --broker-list "3,4,5,6" --generate

注意：此时生成分配策略时候，会先依次对broker.id进行排序（机架感知配置生效）ex 3,5,4,6  ,再此基础上在进行分配策略；确保partions副本分布在不同的机架位；

bin/kafka-reassign-partitions.sh --zookeeper 172.28.1.89:2181 --reassignment-json-file expand-cluster-reassignment.json --execute -throttle 60000000

5、确认完成并移除限速

bin/kafka-reassign-partitions.sh --zookeeper 172.28.1.89:2181 --reassignment-json-file expand-cluster-reassignment.json --verify