码迷,mamicode.com
首页 > Web开发 > 详细

spark源码阅读之network(3)

时间:2015-11-22 13:59:06      阅读:164      评论:0      收藏:0      [点我收藏+]

标签:

TransportContext用来创建TransportServer和TransportclientFactory,同时使用TransportChannelHandler用来配置channel的pipelines,TransportClient提供了两种传输协议,一个是数据层(fetch chunk),一个是控制层(rpc)。rpc的处理需要用户提供一个RpcHandler来处理,它负责建立一个用于传输的流, 使用zero-copy以块的形式进行数据传输。TransportServer和TransportClientFactory为每个channel都创建了一个TransportChannelHandler,每个TransportChannelHandler都包含一个TransportClient,这样服务端可以使用该client向客户端发送消息。
该类有两个主要方法一个是创建TransportChannelHandler一个是给channel配置处理器。
  1. private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
  2. TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
  3. TransportClient client = new TransportClient(channel, responseHandler);
  4. TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
  5. rpcHandler);
  6. return new TransportChannelHandler(client, responseHandler, requestHandler,
  7. conf.connectionTimeoutMs(), closeIdleConnections);
  8. }
这个可以看到TransportResponseHandler需要一个Channel,TransportClient需要channel和TransportResponseHandler,TransportRequestHandler需要channel, TransportClient和RpcHandler. TransportChannelHandler需要client,requestHandler,responseHandler. 这里发送channel,client被使用了多次。transportclient的channel可以从responseHandler中获取。这里挺乱的。
  1. public TransportChannelHandler initializePipeline(
  2. SocketChannel channel,
  3. RpcHandler channelRpcHandler) {
  4. try {
  5. TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
  6. channel.pipeline()
  7. .addLast("encoder", encoder)
  8. .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
  9. .addLast("decoder", decoder)
  10. .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
  11. // NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
  12. // would require more logic to guarantee if this were not part of the same event loop.
  13. .addLast("handler", channelHandler);
  14. return channelHandler;
  15. } catch (RuntimeException e) {
  16. logger.error("Error while initializing Netty pipeline", e);
  17. throw e;
  18. }
  19. }
用来给channel配置channelHandler.第一个是处理出通道的处理器,后面是处理进通道的处理器。

下面看看TransportServer。构建一个服务端。
  1. private void init(String hostToBind, int portToBind) {
  2. IOMode ioMode = IOMode.valueOf(conf.ioMode());
  3. EventLoopGroup bossGroup =
  4. NettyUtils.createEventLoop(ioMode, conf.serverThreads(), "shuffle-server");
  5. EventLoopGroup workerGroup = bossGroup;
  6. PooledByteBufAllocator allocator = NettyUtils.createPooledByteBufAllocator(
  7. conf.preferDirectBufs(), true /* allowCache */, conf.serverThreads());
  8. bootstrap = new ServerBootstrap()
  9. .group(bossGroup, workerGroup)
  10. .channel(NettyUtils.getServerChannelClass(ioMode))
  11. .option(ChannelOption.ALLOCATOR, allocator)
  12. .childOption(ChannelOption.ALLOCATOR, allocator);
  13. if (conf.backLog() > 0) {
  14. bootstrap.option(ChannelOption.SO_BACKLOG, conf.backLog());
  15. }
  16. if (conf.receiveBuf() > 0) {
  17. bootstrap.childOption(ChannelOption.SO_RCVBUF, conf.receiveBuf());
  18. }
  19. if (conf.sendBuf() > 0) {
  20. bootstrap.childOption(ChannelOption.SO_SNDBUF, conf.sendBuf());
  21. }
  22. bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
  23. @Override
  24. protected void initChannel(SocketChannel ch) throws Exception {
  25. RpcHandler rpcHandler = appRpcHandler;
  26. for (TransportServerBootstrap bootstrap : bootstraps) {
  27. rpcHandler = bootstrap.doBootstrap(ch, rpcHandler);
  28. }
  29. context.initializePipeline(ch, rpcHandler);
  30. }
  31. });
  32. InetSocketAddress address = hostToBind == null ?
  33. new InetSocketAddress(portToBind): new InetSocketAddress(hostToBind, portToBind);
  34. channelFuture = bootstrap.bind(address);
  35. channelFuture.syncUninterruptibly();
  36. port = ((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
  37. logger.debug("Shuffle server started on port :" + port);
  38. }
这块是netty中构建一个服务器的流程。配置的缓存生成器是内存池分配器。IO使用的是NIO(EPOLL不兼容windows),相关的配置参数看TransportConf

整个spark的network部分的common模块看完了。其余部分有时间在研究。







spark源码阅读之network(3)

标签:

原文地址:http://www.cnblogs.com/gaoxing/p/4985665.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!