Oozie
Installation in Ubuntu
Step 1 :- To install the Oozie server package on an Ubuntu and other Debian system:
$ sudo apt-get install oozie
Step 2 :- To install the Oozie client package on an Ubuntu and other
Debian system:
$ sudo apt-get install oozie-client
Step 3 :- Configuring which Hadoop Version to Use
To use MRv1(without SSL) :
alternatives --set oozie-tomcat-conf /etc/oozie/tomcat-conf.http.mr1
Step 4 :- Edit /etc/oozie/conf/oozie-env.sh file and make the entry
export CATALINA_BASE=/var/lib/oozie/tomcat-deployment
Step 5 :- Start the Oozie server
$ sudo service oozie start
Step 6 :- Accessing the Oozie Server with the Oozie Client
The Oozie client is a command-line utility that interacts with the Oozie server via the Oozie web-services API.
Use the /usr/bin/oozie script to run the Oozie client.
For example, if you want to invoke the client on the same machine where the Oozie server is running:
$ oozie admin -oozie http://localhost:11000/oozie -status
System mode: NORMAL
To make it convenient to use this utility, set the environment variable OOZIE_URL to point to the URL of the Oozie server. Then you can skip the -oozie option.
For example, if you want to invoke the client on the same machine where the Oozie server is running, set the OOZIE_URL to http://localhost:11000/oozie.
$ export OOZIE_URL=http://localhost:11000/oozie
$ oozie admin -version
Oozie server build version: 4.0.0-cdh5.0.0
Step 7 :- Confiduring MySQL for Oozie
Step 1: Create the Oozie database and Oozie MySQL user.
For example, using the MySQL mysql command-line tool:
$ mysql -u root -p
Enter password: ******
mysql> create database oozie;
Query OK, 1 row affected (0.03 sec)
mysql> grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';
Query OK, 0 rows affected (0.03 sec)
mysql> grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';
Query OK, 0 rows affected (0.03 sec)
mysql> exit
Bye
Step 2: Configure Oozie to use MySQL.
Edit
properties in the oozie-site.xml file as follows:
...
<property>
<name>oozie.service.JPAService.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:mysql://localhost:3306/oozie</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.username</name>
<value>oozie</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.password</name>
<value>oozie</value>
</property>
...
Step 3 : Creating the Oozie DatabaseSchema
$ sudo -u oozie /usr/lib/oozie/bin/ooziedb.sh create -sqlfile oozie-create.sql
or
$ sudo -u oozie /usr/lib/oozie/bin/ooziedb.sh create -run
Step 4 : Enabling the Oozie Web Console
To enable
Oozie's web console, you must download and add the ExtJS library to the Oozie
server. If you have not
already done this, proceed as
follows.
Step 4.1: Download the Library
Download
the ExtJS version 2.2 library from http://archive.cloudera.com/gplextras/misc/ext-2.2.zip and place it a convenient location.
Step 4.2: Install the Library
Extract
the ext-2.2.zip file into /var/lib/oozie.
$ cd
Downloads/
$ sudo cp
-avr ext-2.2 /var/lib/oozie/
Step 5 : Installing the Oozie Shared Library in Hadoop HDFS
The Oozie
installation bundles the Oozie shared library, which contains all of the
necessary JARs to enable workflow jobs to run streaming, DistCp, Pig, Hive, and
Sqoop actions.
The Oozie
installation bundles two shared libraries, one for MRv1 and one for YARN. Make
sure you install the right one for the MapReduce version you are using:
·
The shared library file for MRv1 is oozie-sharelib-mr1.tar.gz.
·
The shared library file for YARN is oozie-sharelib-yarn.tar.gz.
sudo -u oozie oozie admin -shareliblist -oozie
http://localhost:11000/ooziesudo service oozie restart
To install the Oozie shared library in Hadoop HDFS in the oozie
user home directory
$ sudo -u
hdfs hadoop fs -mkdir /user/oozie
$ sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie
$ sudo oozie-setup sharelib create -fs hdfs://localhost:8020 -locallib /usr/lib/oozie/oozie-sharelib-mr1
add the below line to oozie-site.xml to recognize the shared lib functionality
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/etc/hadoop/conf</value>
<description>
Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
the relevant Hadoop *-site.xml files. If the path is relative is looked within
the Oozie configuration directory; though the path can be absolute (i.e. to point
to Hadoop client conf/ directories in the local filesystem.
</description>
</property>
$ sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie
$ sudo oozie-setup sharelib create -fs hdfs://localhost:8020 -locallib /usr/lib/oozie/oozie-sharelib-mr1
add the below line to oozie-site.xml to recognize the shared lib functionality
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/etc/hadoop/conf</value>
<description>
Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
the relevant Hadoop *-site.xml files. If the path is relative is looked within
the Oozie configuration directory; though the path can be absolute (i.e. to point
to Hadoop client conf/ directories in the local filesystem.
</description>
</property>
Configuring Support for Oozie Uber JARs
An uber JAR is a JAR that contains other JARs with
dependencies in a lib/ folder inside the JAR. You can configure the cluster to handle
uber JARs properly for the MapReduce action (as long as it does not include any
streaming or pipes) by setting the following property in the oozie-site.xmlfile:
...
<property>
<name>oozie.action.mapreduce.uber.jar.enable</name>
<value>true</value>
...
When this
property is set, users can use the oozie.mapreduce.uber.jar configuration property in their MapReduce workflows to notify
Oozie that the specified JAR file is an uber JAR.
Configuring Oozie to Run against a Federated Cluster
To run
Oozie against a federated HDFS cluster using ViewFS, configure the oozie.service.HadoopAccessorService.supported.filesystems
property in oozie-site.xml as follows:
<property>
<name>oozie.service.HadoopAccessorService.supported.filesystems</name>
<value>hdfs,viewfs</value>
</property>
Trouble shootingsudo cp mysql-connector-java-5.1.35-bin.jar /var/lib/oozie/
