最近写完storm实时统计pv.uv.ip的项目后，前端服务器用flume进行收集，逐步把所有的服务器都增加上。增加的差不多时。Kafka报了如下类似错误：
4709 [main] WARN kafka.producer.async.DefaultEventHandler - Produce request with correlation id 214 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException 4869 [main] WARN kafka.producer.async.DefaultEventHandler - Produce request with correlation id 217 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException 5035 [main] WARN kafka.producer.async.DefaultEventHandler - Produce request with correlation id 220 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException 5198 [main] WARN kafka.producer.async.DefaultEventHandler - Produce request with correlation id 223 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException 5305 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to send requests for topics datasift with correlation ids in [213,224] kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(Unknown Source) at kafka.producer.Producer.send(Unknown Source) at kafka.javaapi.producer.Producer.send(Unknown Source)
查了各种资料，发现是由于Kafka走的默认配置。发送的消息超过了最大字节导致的。修改如下参数即可：
replica.fetch.max.bytes - this will allow
for the replicas in the brokers to send messages within the cluster and
make sure the messages are replicated correctly. If this is too small,
then the message will never be replicated, and therefore, the consumer
will never see the message because the message will never be committed
(fully replicated).
message.max.bytes - this is the largest size of the message that can be received by the broker from a producer.
修改完成之后一切正常。
本文出自 “Xlows” 博客，请务必保留此出处http://xlows.blog.51cto.com/5380484/1633488
原文地址：http://xlows.blog.51cto.com/5380484/1633488
