Published Paper Details:

Hadoop is a widely used system for managing enormous volumes of data. Hadoop cluster performance is significantly impacted by the effectiveness of job rescheduling algorithms. However, Hadoop's current scheduling methods are unable to properly account for variations in node efficiency across heterogeneous clusters. Issues like uneven work distribution and underutilization of resources may develop as a result. We suggest a novel strategy termed the Versatile Dynamic Data Placement Algorithm (VDDP), created especially for heterogeneous Hadoop clusters, to tackle these difficulties. Our method makes use of the cluster heartbeat mechanism to gather crucial data, such as node CPUs and memories. This data permits a thorough evaluation of each node's actual load capacity. The VDDP algorithm improves the overall efficiency and resource utilisation of Hadoop clusters in the presence of heterogeneity by dynamically adjusting and balancing the distribution of information across the cluster based on the condition of each node. Our approach meets the needs of heterogeneous clusters where nodes have different processing capacities, in contrast to the current Hadoop implementation, which assumes homogenous computing nodes. Our proposed system effectively handles diverse data types in Hadoop clusters, thereby improving efficiency and performance in large-scale data collection and analysis. Furthermore, we introduce a method to calculate the remaining rate of hardware resources on each node, prioritizing tasks on nodes with higher resource availability based on the task type. Additionally, we incorporate data compression techniques for Input-Output intensive tasks to minimize disk function and expedite task execution. Our VDDP approach outperforms existing scheduling algorithms, according to experimental data, especially in heterogeneous Hadoop clusters, where it considerably improves scheduling efficiency and task execution speed. The VDDP algorithm showcases task-specific enhancements in execution time, delivering superior performance for various task types.