A study of efficiency improvement of structured overlay network for scalable big data management

Kimihiro Mizutani(1461202)


Big data is extremely large amount of valuable data (e.g., sensor data, life log, and etc.), and it is generally managed by cloud (i.e., data center). As the volume/variety of big data and its demand increase, the data center must upgrade and expand its scale. As the results, the data center management becomes significantly complex and the network operation is entirely beyond human control. Therefore, the automatic data center management will become essential technique to sustain a large scale data center network stably. To realize the automatic data center management, we apply the structured overlay techniques to the management. The structured overlay is good at automatic servers and data management, but its performance does not suitable for big data management. For examples, conventional structured overlays take (1) a number of queries, (2)a large amount of data migration for load balancing, and (3) longer mean time to repair (MTTR) for big data management as the scale of data center expands. These problemscause a lot of both capitalization and operation costs for data center developers regardless of realizing the automatic management. This dissertation introduces the novel three efficiencies improvement methods for conventional structured overlay for solving the (1)-(3) problems. To reduce the query traffic, we propose a novel bulk query transfer scheme. For data migration traffic reduction, we propose the optimal load balancing policy based on kernel estimation scheme. Finally, we use testament protocol for improving MTTR. Using these schemes, the structured overlay realizes automatic data center management with significant cost-less benefit.