Reconfiguration of Apache Storm for InfiniBand Communications


KIPS Transactions on Software and Data Engineering, Vol. 7, No. 8, pp. 297-306, Aug. 2018
10.3745/KTSDE.2018.7.8.297,   PDF Download:
Keywords: Apache Storm, InfiniBand, IPoIB, RDMA, Netty, RJ-Netty
Abstract

In this paper, we address how to apply Apache Storm, a distributed stream processing framework, to InfiniBand, a high performance communication device. An easy way to run Storm on InfiniBand is to simply use IPoIP (IP over InfiniBand). However, this method causes a serious CPU load on the node, which is caused by frequent context switches and buffer copies. To solve this problem, we propose a new communication method using InfiniBand's Remote Direct Memory Access (RDMA) function in Storm. First, we design and implement RJ-Netty (RDMA/JXIO Netty), a new framework that replaces Netty, the legacy framework, to exploit RDMA functionality. Second, we reimplement the related classes so that Storm can use both existing Netty and new RJ-Netty. Third, we extend the JXIO server functionality so as to support multi-threading to maximize the performance of RJ-Netty. Experimental results show that the proposed RJ-Netty significantly reduces CPU load while improving message throughput compared to IPoIB as well as Ethernet. This paper is the first attempt to run Apache Storm on InfiniBand, and we believe that it is an excellent research result that improves the performance of Storm by using InfiniBand RDMA.


Statistics
Show / Hide Statistics

Statistics (Cumulative Counts from September 1st, 2017)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.


Cite this article
[IEEE Style]
S. Yang, S. Son, Y. Moon, "Reconfiguration of Apache Storm for InfiniBand Communications," KIPS Transactions on Software and Data Engineering, vol. 7, no. 8, pp. 297-306, 2018. DOI: 10.3745/KTSDE.2018.7.8.297.

[ACM Style]
Seokwoo Yang, Siwoon Son, and Yang-Sae Moon. 2018. Reconfiguration of Apache Storm for InfiniBand Communications. KIPS Transactions on Software and Data Engineering, 7, 8, (2018), 297-306. DOI: 10.3745/KTSDE.2018.7.8.297.