码迷,mamicode.com
首页 > 其他好文 > 详细

storm trident 如何标记一个batch被处理——coordinator spout

时间:2018-05-27 10:43:15      阅读:158      评论:0      收藏:0      [点我收藏+]

标签:its   stream   this   update   https   class   off   top   abs   

Splitting a stream has no effect on the batch. If you join the stream back together, then yes, it will be the same batch.

Tuples are passed between partitions in the order they‘re emitted (repartitioning happens on groupBy, partitioning operations, and global aggregations). These are the same semantics you get from Storm.

State updates are ordered among batches.

Each batch has both a "txid" and an "attempt id". The attempt id is a random long. This ensures that Storm can distinguish between multiple attempts for the same batch.

Batches are controlled by a single coordinator thread (which is a regular Storm spout) that determines when batches get processed and when they get committed (commits are when state updates happen).

The coordinator also ensures the ordering.

The coordinator abstraction is actually quite elegant. It builds upon the primitives that the tuple tree/acking framework provides to implement a relatively sophisticated distributed coordination algorithm.

Also, is there any way to turn off acking in Trident? Not tagging tuples with message IDs and setting ackers to 0 don‘t seem to work (the latter causes a stack overflow).

No, you can‘t. Acking isn‘t really expensive in Trident as long as your batches are of non-trivial size.

https://groups.google.com/forum/#!topic/storm-user/AUajG72kxmo

storm trident 如何标记一个batch被处理——coordinator spout

标签:its   stream   this   update   https   class   off   top   abs   

原文地址:https://www.cnblogs.com/brainstorm/p/9095079.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!