Published Paper Details:

Big data contains large data sets that are so voluminous and complex that traditional data processing applications are inadequate to deal with them. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating. The first five important dimensions to big data are characterized as: Volume, Variety, Velocity, veracity and volatile. Big data is generated from multiple sources like social networking sites, sensors, pdf documents. As the big data generated from multiple sources, it is heterogeneous in nature. Big data includes structured, unstructured and semi-structured data. For analyzing the structured data we have Traditional Data Base Management (TDBM). Traditional data bases are inadequate to maintain unstructured and semi-structured. To manage semi-structured data NoSql (Structured Query Language) data bases can be used. Before performing analytics the data should be captured from multiple resources, which is one of the major challenge. Data Scraping is a technique that extracts desired data from big data sources. Big data can be abstracted or scraped by using JSOUP (Third party libraries in Java), web crawlers. The scraped data can automatically loaded to hive engine with the help of flume tool. So in this project we are going to capture big data from various big data sources and then perform analytics on the data which have been scraped from multiple sources.