Apache Pig Big Data Analysis

Big Data Analysis Using Apache Pig

Pig was developed by Yahoo. It is an engine built on the top of MapReduce which will convert PigLatin Script into MapReduce code. Thus Pig has a component known as Pig Engine that accepts the Pig Latin scripts as input and converts those scripts into MapReduce jobs. It is using PigLatin scripting language for the operations like ETL (Extract, Transform and Load). Thus the definition of Apache Pig would be Pig is a high level scripting language that is used with Apache Hadoop Ecosystem. Also anyone can say it is a tool/platform which is used to analyze larger sets of data representing them as data flows.

The main motive behind developing Pig was to reduce the time required for development through its multi query data flows language.

The Key features of Apache Pig are as following

- It has procedural data flow language i.e. PigLatin

- It is mainly used for programming

- It can handles all kinds of data e.g. Structured as well as Unstructured

- By using Pig’s multi-query approach anyone can operate many operation together in a       single flow, reducing the time of multiple times data scanned thus by using this we need to write 1/20th of code and required 1/16th time of development.

- It’s providing Rich Set of operators for filter, join, sort etc.

- It’s providing complex data types e.g. tuples, bags, and maps

- It is generally used by the researcher and programmer

- It operates on the client side of any cluster

- It does not have a dedicated metadata database and schema or data types will be defined in the script itself.

- Through User Defined Functions (UDF) facility in Pig, anyone can execute many languages code like Ruby, Python and Java. In other words in UDF we can use Java, Python and other language code and can execute them by using Pig Script.

Installing Apache Pig

Here I am assuming Hadoop is up and running onto the machine.

Download Apache Pig-0.15.0 from Apache Pig Download Link

Go to the specified location where you want to have Pig installable, then unzip the Apache Pig zipped folder.

$ tar -xzf pig-0.15.0.tar.gz
$ mv pig-0.15.0 pig

Apache Pig Configuration

Setup Environment Variable

Set Environment Variables by editing bashrc file using Edit ~/.bashrc file and append following lines into that and save.

export PIG_HOME = /home/training/Pig

export PATH  = PATH:/home/training/pig/bin

export PIG_CLASSPATH = $HADOOP_HOME/conf

Then execute the below command to source ~/.bashrc

$ source ~/.bashrc

Verifying the Installation

Verify the installation of Apache Pig by typing the pig command. If the installation is successful, you will get the grunt shell of Apache Pig as shown below.

Apache Pig Execution Modes

Local Mode

In local mode, there is no need of Hadoop or HDFS. It will require all file and execution from local system. This mode is generally used for testing purpose.

$ pig -x local

MapReduce Mode

MapReduce mode is where we load or process the data that exists in the Hadoop File System (HDFS) using Apache Pig. In this mode, whenever we execute the Pig Latin statements to process the data, a MapReduce job is invoked in the back-end to perform a particular operation on the data that exists in the HDFS.

$ pig -x mapreduce

Load/Read and Store Data Using Apache Pig 

Apache Pig works on top of Hadoop. Be ensure that Hadoop is up and running. Then start Pig in MapReduce mode.

$ pig -x mapreduce

Suppose we are having Person data as following

PersonDetails

Bill Gates,CEO,Microsoft Iqubal,CEO,SZIAS Steve Jobs,CEO,Apple

Create Directory and Load PersonDetails Data into the HDFS

Navigate to PersonDetails file directory

$ hadoop fs -mkdir /user/training/PIGDATA

$ hadoop fs -mkdir /user/training/PIGDATA/PIG_UDF_DATA

$ hadoop fs -copyFromLocal PersonDetails /user/training/PIGDATA/PIG_UDF_DATA

$ pig -x mapreduce

Processing Data using Apache Pig

grunt> PERSONDETAILS= LOAD '/user/training/PIGDATA/PIG_UDF_DATA/PersonDetails' Using PigStorage(',') AS (name:chararray, designation:chararray, company:chararray);

Reading Data using DUMP command

grunt> dump PERSONDETAILS;

(Bill Gates,CEO,Microsoft)

(Iqubal,CEO,SZIAS)

(Steve Jobs,CEO,Apple)

Storing Data using STORE command

grunt> STORE PERSONDETAILS INTO 'store_pig_latin';

Counters:

Total records written : 3

Total bytes written : 63

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:

job_201601290600_0012

2016-02-02 07:51:23,723 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

Verify by checking the file

grunt> cat store_pig_latin;

Hope you have enjoyed the article.
Author : Iqubal Mustafa Kaki, Technical Specialist

Want to connect with me
If you want to connect with me, please connect through my email - iqubal.kaki@gmail.com