ECE Seminar: 1PM Friday July 6th, HWLL313: Optimisation of MapReduce parameters using Genetic Algorithms, presented by Ali Kaheel

In this paper, all issues related to Hadoop MapReduce parameter settings are addressed. Some significant parameters of Hadoop MapReduce are tuned using a novel intelligent technique based on both Genetic programming and Genetic Algorithm to optimise the performance of Hadoop MapReduce job. In Hadoop framework, there are more than 150 configurations parameters. Setting this number of parameters manually is a difficult aim and consumes long time. As a result, these algorithms are used to search for the optimum values of parameter settings. Experiments have been carried out on two typical applications of Hadoop that includes both Word Count Application and Tera Sort application. The results showed that our proposed technique improves MapReduce jobs performance for 20 GB on Word Count Application by 69.63% and 30.31% in comparison with default and Gunther work, respectively. Whilst on Tera sort application, the performance of Hadoop MapReduce is improved by 73.39% and 55.93% compared with default and Gunther work, consecutively.

Presented by

Key learning outcomes