Published Paper Details:

The rapid development of information technology over the last decade means that data appears in a wide range of sensor data, tweets, photos, raw data and unstructured data formats. With such an overwhelming flood of information, current data management systems cannot scale to this enormous quantity of raw, unstructured data -- Big Data, today. We show the basic concepts and designs of big data tools, algorithms and techniques in the present study. We compare the classical data mining algorithms with the Big Data algorithms by using Hadoop / Map Reduce as the core scalable algorithm implementation of Big Data. We implemented the K-means and A-priori algorithms on a 5-node Hadoop cluster with Hadoop / Map Reduce. We use MongoDB as an example to explore NoSQL databases for semi-structured, massive data scaling. Finally, we show the performance of these two algorithms between HDFS (Hadoop Distributed File System) and MongoDB data storage.