Apache Pig - Maximum Average Salary Analytics

Here I am having large dummy data-set of employee. I will explain you how anybody can do the maximum average salary analysis of this data-set by using Apache Pig.

Salary Data Analytics

Please download the dummy data-set from Employee Dummy Data

Here I am assuming Apache Hadoop and Pig is up running onto your machine.

Create Directory and Load employee_dataset Data into the HDFS

Navigate to dummy data file directory

$ hadoop fs -mkdir /user/training/PIGDATA

$ hadoop fs -mkdir /user/training/PIGDATA/PIG_UDF_DATA

$ hadoop fs -copyFromLocal employee_dataset /user/training/PIGDATA/PIG_UDF_DATA

$ pig -x mapreduce

Processing Data using Apache Pig

Step: 1 - Load dataset with column names and datatypes

grunt> employeeData = LOAD '/user/training/PIGDATA/PIG_UDF_DATA/employee_dataset'  using  PigStorage(',') AS (emp_id:int, emp_name:chararray, job_title:chararray, dept_id:int, salary:float);

Step: 2 - Group records by department

grunt> group_by_dept = GROUP employeeData BY dept_id;

Step: 3 - Calculate average salary by department

grunt> average_sal = FOREACH group_by_dept GENERATE group, AVG(employeeData.salary) AS avgsalary;

Step: 4 - Sort salary in decreasing order and select the top 1 to get the max salary

grunt> sorted_avg_sal = ORDER average_sal BY avgsalary desc;

grunt> avg_max_sal_analytic = LIMIT sorted_avg_sal 1;

Step: 5 - Store results to HDFS

grunt> STORE avg_max_sal_analytic INTO '/home/training/Desktop/output/pig/topdepartment';

Output :

Input(s):

Successfully read 10293 records (535750 bytes) from: "/user/training/PIGDATA/PIG_UDF_DATA/employee_dataset"

Output(s):

Successfully stored 1 records (23 bytes) in: "/home/training/Desktop/output/pig/topdepartment"

Counters:

Total records written : 1

Total bytes written : 23

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:

job_201601290600_0017   ->      job_201601290600_0018,

job_201601290600_0018   ->      job_201601290600_0019,

job_201601290600_0019

2016-02-04 05:09:41,199 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 2 time(s).

2016-02-04 05:09:41,199 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

Step: 6 - Check the output

grunt> cat /home/training/Desktop/output/pig/topdepartment

Output:

77      9.99999986991104E14

Hope you have enjoyed the article.

Author: Iqubal Mustafa Kaki, Technical Specialist

Want to connect with me
If you want to connect with me, please connect through my email - iqubal.kaki@gmail.com

Apache Pig - Stock Market Analytics

Here I am having large dummy data-set of Stock Exchange. We will do the maximum closing price analysis of stock’s shares by using Apache Pig. Please follow the below step for achieving the target.

Please download the dummy data-set from Stock Market Dummy Data

Here I am assuming Apache Hadoop and Pig is up running onto your machine.

Create Directory and Load stock_dummy_data Data into the HDFS

Navigate to dummy data file directory

$ hadoop fs -mkdir /user/training/PIGDATA

$ hadoop fs -mkdir /user/training/PIGDATA/PIG_UDF_DATA

$ hadoop fs -copyFromLocal stock_dummy_data /user/training/PIGDATA/PIG_UDF_DATA

$ pig -x mapreduce

Processing Data using Apache Pig

Step: 1 - Load dataset with column names and datatypes

grunt> stock_records= LOAD '/user/training/PIGDATA/PIG_UDF_DATA/stock_dummy_data' USING PigStorage(',') as (exchange:chararray, symbol:chararray, date:datetime, open:float, high:float, low:float, close:float,volume:int, adj_close:float);

Step: 2 - Group records by symbol

grunt> group_by_symbol = GROUP stock_records BY symbol;

Step: 3 - Calculate the maximum closing price

grunt> max_closing_price = FOREACH group_by_symbol GENERATE group, MAX(stock_records.close) as maxclose;

Step: 4 - Store output by using STORE command

grunt> STORE max_closing_price INTO '/home/training/Desktop/output/pig/stocks' USING PigStorage(',');

Output :

Input(s):

Successfully read 540 records (33176 bytes) from: "/user/training/PIGDATA/PIG_UDF_DATA/stock_dummy_data"

Output(s):

Successfully stored 1 records (12 bytes) in: "/home/training/Desktop/output/pig/stocks"

Counters:

Total records written : 1

Total bytes written : 12

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:

job_201601290600_0015

2016-02-04 02:30:22,583 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

Step: 5 - Check the output

grunt> cat /home/training/Desktop/output/pig/stocks;

Output:

SZIAS, 35.67

Hope you have enjoyed the article.

Author : Iqubal Mustafa Kaki, Technical Specialist

Want to connect with me
If you want to connect with me, please connect through my email - iqubal.kaki@gmail.com

Apache Pig Big Data Analysis

Big Data Analysis Using Apache Pig

Pig was developed by Yahoo. It is an engine built on the top of MapReduce which will convert PigLatin Script into MapReduce code. Thus Pig has a component known as Pig Engine that accepts the Pig Latin scripts as input and converts those scripts into MapReduce jobs. It is using PigLatin scripting language for the operations like ETL (Extract, Transform and Load). Thus the definition of Apache Pig would be Pig is a high level scripting language that is used with Apache Hadoop Ecosystem. Also anyone can say it is a tool/platform which is used to analyze larger sets of data representing them as data flows.

The main motive behind developing Pig was to reduce the time required for development through its multi query data flows language.

The Key features of Apache Pig are as following

- It has procedural data flow language i.e. PigLatin

- It is mainly used for programming

- It can handles all kinds of data e.g. Structured as well as Unstructured

- By using Pig’s multi-query approach anyone can operate many operation together in a       single flow, reducing the time of multiple times data scanned thus by using this we need to write 1/20th of code and required 1/16th time of development.

- It’s providing Rich Set of operators for filter, join, sort etc.

- It’s providing complex data types e.g. tuples, bags, and maps

- It is generally used by the researcher and programmer

- It operates on the client side of any cluster

- It does not have a dedicated metadata database and schema or data types will be defined in the script itself.

- Through User Defined Functions (UDF) facility in Pig, anyone can execute many languages code like Ruby, Python and Java. In other words in UDF we can use Java, Python and other language code and can execute them by using Pig Script.

Installing Apache Pig

Here I am assuming Hadoop is up and running onto the machine.

Download Apache Pig-0.15.0 from Apache Pig Download Link

Go to the specified location where you want to have Pig installable, then unzip the Apache Pig zipped folder.

$ tar -xzf pig-0.15.0.tar.gz
$ mv pig-0.15.0 pig

Apache Pig Configuration

Setup Environment Variable

Set Environment Variables by editing bashrc file using Edit ~/.bashrc file and append following lines into that and save.

export PIG_HOME = /home/training/Pig

export PATH  = PATH:/home/training/pig/bin

export PIG_CLASSPATH = $HADOOP_HOME/conf

Then execute the below command to source ~/.bashrc

$ source ~/.bashrc

Verifying the Installation

Verify the installation of Apache Pig by typing the pig command. If the installation is successful, you will get the grunt shell of Apache Pig as shown below.

Apache Pig Execution Modes

Local Mode

In local mode, there is no need of Hadoop or HDFS. It will require all file and execution from local system. This mode is generally used for testing purpose.

$ pig -x local

MapReduce Mode

MapReduce mode is where we load or process the data that exists in the Hadoop File System (HDFS) using Apache Pig. In this mode, whenever we execute the Pig Latin statements to process the data, a MapReduce job is invoked in the back-end to perform a particular operation on the data that exists in the HDFS.

$ pig -x mapreduce

Load/Read and Store Data Using Apache Pig 

Apache Pig works on top of Hadoop. Be ensure that Hadoop is up and running. Then start Pig in MapReduce mode.

$ pig -x mapreduce

Suppose we are having Person data as following

PersonDetails

Bill Gates,CEO,Microsoft Iqubal,CEO,SZIAS Steve Jobs,CEO,Apple

Create Directory and Load PersonDetails Data into the HDFS

Navigate to PersonDetails file directory

$ hadoop fs -mkdir /user/training/PIGDATA

$ hadoop fs -mkdir /user/training/PIGDATA/PIG_UDF_DATA

$ hadoop fs -copyFromLocal PersonDetails /user/training/PIGDATA/PIG_UDF_DATA

$ pig -x mapreduce

Processing Data using Apache Pig

grunt> PERSONDETAILS= LOAD '/user/training/PIGDATA/PIG_UDF_DATA/PersonDetails' Using PigStorage(',') AS (name:chararray, designation:chararray, company:chararray);

Reading Data using DUMP command

grunt> dump PERSONDETAILS;

(Bill Gates,CEO,Microsoft)

(Iqubal,CEO,SZIAS)

(Steve Jobs,CEO,Apple)

Storing Data using STORE command

grunt> STORE PERSONDETAILS INTO 'store_pig_latin';

Counters:

Total records written : 3

Total bytes written : 63

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:

job_201601290600_0012

2016-02-02 07:51:23,723 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

Verify by checking the file

grunt> cat store_pig_latin;

Hope you have enjoyed the article.
Author : Iqubal Mustafa Kaki, Technical Specialist

Want to connect with me
If you want to connect with me, please connect through my email - iqubal.kaki@gmail.com