Apache Pig – How to run multiple Pig scripts

unit test java pig script

As a developer, sometimes you have the need to run multiple pig scripts to finish a specific task. Logically, you can have a single Pig script perform all the business rules or tasks given to you, but sometimes it is better to compartmentalize each Pig Script to do a specific task.

Running multiple Pig scripts is useful when you have a series of Pig Scripts that need to be ran one after another. You can also schedule an Oozie job to run the following script below. Running multiple Pig scripts is useful when running daily ETL jobs. That way you only schedule one Pig job instead.

 

Run multiple Pig scripts (one by one)

You will need to create a Pig script and call the exec command. These Pig scripts need to be located on your local file system. For illustration purposes, we will call this Pig script PigScriptRunner.pig.

This will run the following 3 scripts:  

 

  1. PigScript1.pig
  2. PigScript2.pig
  3. PigScript3.pig

//PigScriptRunner.pig

Let emphasize that these Pig scripts will not run simultaneously. It will be one after another. Also, the third Pig Hadoop Script includes a param that you can include in case your Pig Script needs one.