關于Kafka的其他一些內容，堆積情況，retention，auto.offset.reset

producer是線程安全的，多線程分享共用一個producer比用多個 producer整體上要快

如果你想了解大數據的學習路線，想學習大數據知識以及需要免費的學習資料可以加群：784789432.歡迎你的加入。每天下午三點開直播分享基礎知識，晚上20:00都會開直播給大家分享大數據項目實戰。

可以命令行查看某consumer group所有consumer的offset，和落后的多少，也就是說可以查看Kafka中數據的堆積情況，以下來自官方文檔

Sometimes it's useful to see the position of your consumers. We have a tool that will show the position of all consumers in a consumer group as well as how far behind the end of the log they are. To run this tool on a consumer group named my-group consuming a topic named my-topic would look like this:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group

Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
my-topic 0 2 4 2 consumer-1-029af89c-873c-4751-a720-cefd41a669d6 /127.0.0.1 consumer-1
my-topic 1 2 3 1 consumer-1-029af89c-873c-4751-a720-cefd41a669d6 /127.0.0.1 consumer-1
my-topic 2 2 3 1 consumer-2-42c1abd4-e3b2-425d-a8bb-e1ea49b29bb2 /127.0.0.1 consumer-2

This tool also works with ZooKeeper-based consumers:

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group my-group

Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
my-topic 0 2 4 2 my-group_consumer-1
my-topic 1 2 3 1 my-group_consumer-1
my-topic 2 2 3 1 my-group_consumer-2

retention

Kafka保留消費后的數據，但也不是永遠保留，默認7天后，會自動刪除數據。當然我們可以設置保留（retention）幾天，由broker config （也就是server.properties）中的log.retention.ms、log.retention.minutes、log.rentention.hours設置，優先級遞增。默認log.rentention.hours=168。

還有一種retention的設置方法，就是log.retention.bytes，也是server.properties中設置，定義了一個partition存儲的最大大小。兩種方法一種滿足就會刪除。

kafka刪除是按照segment來刪，一次只能整個刪掉一個或多個segment。

另外也可以設置每個topic的retention情況，具體看官方文檔http://kafka.apache.org/documentation/

See http://kafka.apache.org/documentation/#brokerconfigs for the full list of log.retention./log.roll./log.segment.* configs

auto.offset.reset

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):
earliest: automatically reset the offset to the earliest offset
latest: automatically reset the offset to the latest offset
none: throw exception to the consumer if no previous offset is found for the consumer's group
anything else: throw exception to the consumer.

default: latest

每一個topic記錄offset，offset屬于不同的group，一個group只有一個offset（每個partition）