Apache Pig - Maximum Average Salary Analytics

Here I am having large dummy data-set of employee. I will explain you how anybody can do the maximum average salary analysis of this data-set by using Apache Pig.

Salary Data Analytics

Please download the dummy data-set from Employee Dummy Data

Here I am assuming Apache Hadoop and Pig is up running onto your machine.

Create Directory and Load employee_dataset Data into the HDFS

Navigate to dummy data file directory

$ hadoop fs -mkdir /user/training/PIGDATA

$ hadoop fs -mkdir /user/training/PIGDATA/PIG_UDF_DATA

$ hadoop fs -copyFromLocal employee_dataset /user/training/PIGDATA/PIG_UDF_DATA

$ pig -x mapreduce

Processing Data using Apache Pig

Step: 1 - Load dataset with column names and datatypes

grunt> employeeData = LOAD '/user/training/PIGDATA/PIG_UDF_DATA/employee_dataset'  using  PigStorage(',') AS (emp_id:int, emp_name:chararray, job_title:chararray, dept_id:int, salary:float);

Step: 2 - Group records by department

grunt> group_by_dept = GROUP employeeData BY dept_id;

Step: 3 - Calculate average salary by department

grunt> average_sal = FOREACH group_by_dept GENERATE group, AVG(employeeData.salary) AS avgsalary;

Step: 4 - Sort salary in decreasing order and select the top 1 to get the max salary

grunt> sorted_avg_sal = ORDER average_sal BY avgsalary desc;

grunt> avg_max_sal_analytic = LIMIT sorted_avg_sal 1;

Step: 5 - Store results to HDFS

grunt> STORE avg_max_sal_analytic INTO '/home/training/Desktop/output/pig/topdepartment';

Output :

Input(s):

Successfully read 10293 records (535750 bytes) from: "/user/training/PIGDATA/PIG_UDF_DATA/employee_dataset"

Output(s):

Successfully stored 1 records (23 bytes) in: "/home/training/Desktop/output/pig/topdepartment"

Counters:

Total records written : 1

Total bytes written : 23

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:

job_201601290600_0017   ->      job_201601290600_0018,

job_201601290600_0018   ->      job_201601290600_0019,

job_201601290600_0019

2016-02-04 05:09:41,199 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 2 time(s).

2016-02-04 05:09:41,199 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

Step: 6 - Check the output

grunt> cat /home/training/Desktop/output/pig/topdepartment

Output:

77      9.99999986991104E14

Hope you have enjoyed the article.

Author: Iqubal Mustafa Kaki, Technical Specialist

Want to connect with me
If you want to connect with me, please connect through my email - iqubal.kaki@gmail.com

Apache Pig - Stock Market Analytics

Here I am having large dummy data-set of Stock Exchange. We will do the maximum closing price analysis of stock’s shares by using Apache Pig. Please follow the below step for achieving the target.

Please download the dummy data-set from Stock Market Dummy Data

Here I am assuming Apache Hadoop and Pig is up running onto your machine.

Create Directory and Load stock_dummy_data Data into the HDFS

Navigate to dummy data file directory

$ hadoop fs -mkdir /user/training/PIGDATA

$ hadoop fs -mkdir /user/training/PIGDATA/PIG_UDF_DATA

$ hadoop fs -copyFromLocal stock_dummy_data /user/training/PIGDATA/PIG_UDF_DATA

$ pig -x mapreduce

Processing Data using Apache Pig

Step: 1 - Load dataset with column names and datatypes

grunt> stock_records= LOAD '/user/training/PIGDATA/PIG_UDF_DATA/stock_dummy_data' USING PigStorage(',') as (exchange:chararray, symbol:chararray, date:datetime, open:float, high:float, low:float, close:float,volume:int, adj_close:float);

Step: 2 - Group records by symbol

grunt> group_by_symbol = GROUP stock_records BY symbol;

Step: 3 - Calculate the maximum closing price

grunt> max_closing_price = FOREACH group_by_symbol GENERATE group, MAX(stock_records.close) as maxclose;

Step: 4 - Store output by using STORE command

grunt> STORE max_closing_price INTO '/home/training/Desktop/output/pig/stocks' USING PigStorage(',');

Output :

Input(s):

Successfully read 540 records (33176 bytes) from: "/user/training/PIGDATA/PIG_UDF_DATA/stock_dummy_data"

Output(s):

Successfully stored 1 records (12 bytes) in: "/home/training/Desktop/output/pig/stocks"

Counters:

Total records written : 1

Total bytes written : 12

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:

job_201601290600_0015

2016-02-04 02:30:22,583 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

Step: 5 - Check the output

grunt> cat /home/training/Desktop/output/pig/stocks;

Output:

SZIAS, 35.67

Hope you have enjoyed the article.

Author : Iqubal Mustafa Kaki, Technical Specialist

Want to connect with me
If you want to connect with me, please connect through my email - iqubal.kaki@gmail.com

Writing Apache Pig UDF’s using Java

Playing with Snake – Writing Apache Pig UDF’s using Java

          

Apache Pig is having capability to execute Java, Python, or Ruby code inside Pig Script as UDF - thus you can use them to load, aggregate, or do sophisticated data analysis. Here I will explain you how to write Apache Pig UDF’s (User Defined Functions) using Java.  Be ensuring you have installed Eclipse and Apache Maven onto your machine.

You can download below explained project from the link IMUApachePigUDF_POC or from GitHub download link

Create a Maven Project for writing Apache Pig UDF by following below steps

File > New > Maven Project > Check Create a simple project

Here you need to add pig-0.15.0 dependency into the POM.XML 

<build> <sourceDirectory>src</sourceDirectory> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <version>3.3</version> <configuration> <source>1.7</source> <target>1.7</target> </configuration> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>org.apache.pig</groupId> <artifactId>pig</artifactId> <version>0.15.0</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>0.20.2</version> </dependency> </dependencies>

Write Java UDF Class 

For writing Apache Pig UDF using Java. We need to implement EvalFunc interface to the class and should override exec method. Here, we are returning the uppercase of the given column in the below explained UDF example.

UpperCaseAttribute Java Class

import java.io.IOException;  import org.apache.pig.EvalFunc;  import org.apache.pig.data.Tuple;  public class UpperCaseAttribute extends EvalFunc<String>{     public String exec(Tuple input) throws IOException {          if (input == null || input.size() == 0)             return null;             String str = (String)input.get(0);             return str.toUpperCase();      }  }

Export class file as JAR

Right click > Export > JAR file > Next

Step 1: Registering the Jar file

Apache Pig works on top of Hadoop. Be ensure that Hadoop is up and running. Then start Pig in MapReduce mode.

$ pig -x mapreduce

First step would be to register the exported JAR by executing below commands. Here I am assuming JAR is at the /home/training/Desktop location

grunt> REGISTER '/home/training/Desktop/UpperCaseAttribute_UDF.jar'

Step 2: Defining Alias

After registering the UDF we need to define an alias by using Define operator.

grunt> DEFINE UpperCaseAttribute UpperCaseAttribute();  

Step 3: Load Data into Apache Pig For Using UDF

Suppose we are having Person data as following

PersonDetails

Bill Gates,CEO,Microsoft Iqubal,CEO,SZIAS Steve Jobs,CEO,Apple

Create Directory and Load PersonDetails Data into the HDFS

Navigate to PersonDetails file directory

$ hadoop fs -mkdir /user/training/PIGDATA

$ hadoop fs -mkdir /user/training/PIGDATA/PIG_UDF_DATA

$ hadoop fs -copyFromLocal PersonDetails /user/training/PIGDATA/PIG_UDF_DATA

$ pig -x mapreduce

Processing Data using Apache Pig

grunt> PERSONDETAILS= LOAD '/user/training/PIGDATA/PIG_UDF_DATA/PersonDetails' Using PigStorage(',') AS (name:chararray, designation:chararray, company:chararray);

Let us use our created UDF to convert the names of the Person in to upper case.

grunt> PersonNameUpperCase = FOREACH PERSONDETAILS GENERATE UpperCaseAttribute(name);

grunt> dump PersonNameUpperCase;

2016-02-03 07:07:25,512 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

2016-02-03 07:07:25,516 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1

2016-02-03 07:07:25,516 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1

(BILL GATES)

(IQUBAL)

(STEVE JOBS)

Hope you have enjoyed the article.
Author : Iqubal Mustafa Kaki, Technical Specialist

Want to connect with me
If you want to connect with me, please connect through my email - iqubal.kaki@gmail.com