Postgresql ,Basic steps in Ubuntu by usingTerminal

1) To open  in Terminal
   $sudo -I (i) -u postgres psql

2)To check the database available
postgres=#\l

3)To create  database
      postgres=#create database manju;
           ( manju is db name).

4)To connect the db.
     postgres=#\c manju;

5)Creating Schema.
    postgres=#create schema publicschema;

6)To check the DataType
  postgres=#/d

7)create Table.
   postgres=#table publicschema.table1(id integer, password CHAR(10));

8)To check the catalog.
    postgres=#select* from pg_catalog.pg_tables;
 
     postgres=# select * from pg_catalog.pg_tables where schema!= 'information_schema' and schema name!='pg_catalog';


9)Insert values
     postgres=# into public.manju(1,'1');

9)To view the table
   postgres=#select * from public.manju;

10)To Truncate .
     postgres=#truncate public.manju;  
 
11)To Delete
      postgres=#delete from public.manju;

12)To Drop.
    postgres=#drop table public.manju;

Scala and spark Installation

                                       Scala & spark Installation

Step 1: Verifying Java Installation

Java installation is one of the mandatory things in installing Spark. Try the following command to verify the JAVA version.

$java -version 

If Java is already, installed on your system, you get to see the following response −

java version "1.7.0_71" 

Java(TM) SE Runtime Environment (build 1.7.0_71-b13) 

Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode)

In case you do not have Java installed on your system, then Install Java before proceeding to next step.

Step 2: Verifying Scala installation

You should Scala language to implement Spark. So let us verify Scala installation using following command.

$scala -version

If Scala is already installed on your system, you get to see the following response −

Scala code runner version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL

In case you don’t have Scala installed on your system, then proceed to next step for Scala installation.

Step 3: Downloading Scala

Download the latest version of Scala by visit the following link scala For this tutorial, we are using scala-2.11.6 version. After downloading, you will find the Scala tar file in the download folder.

Step 4: Installing Scala

Follow the below given steps for installing Scala.

Extract the Scala tar file

Type the following command for extracting the Scala tar file.

$ tar xvf scala-2.11.6.tgz

Move Scala software files

Use the following commands for moving the Scala software files, to respective directory (/usr/local/scala).

$ su – 

Password: 

# cd /home/Hadoop/Downloads/ 

# mv scala-2.11.6 /usr/local/scala 

# exit 

Set PATH for Scala

Use the following command for setting PATH for Scala.

$ export PATH=$PATH:/usr/local/scala/bin

Verifying Scala Installation

After installation, it is better to verify it. Use the following command for verifying Scala installation.

$scala -version

Step 5: Downloading Apache Spark

Download the latest version of Spark by visiting the following link  For this tutorial, we are using spark-1.3.1-bin-hadoop2.6 version. After downloading it, you will find the Spark tar file in the download folder.

Step 6: Installing Spark

Follow the steps given below for installing Spark.

Extracting Spark tar

The following command for extracting the spark tar file.

$ tar xvf spark-1.3.1-bin-hadoop2.6.tgz 

Moving Spark software files

The following commands for moving the Spark software files to respective directory (/usr/local/spark).

$ su – 

Password:  

# cd /home/Hadoop/Downloads/ 

# mv spark-1.3.1-bin-hadoop2.6 /usr/local/spark 

# exit 

Setting up the environment for Spark

Add the following line to ~/.bashrc file. It means adding the location, where the spark software file are located to the PATH variable.

export PATH = $PATH:/usr/local/spark/bin

Use the following command for sourcing the ~/.bashrc file.

$ source ~/.bashrc

Step 7: Verifying the Spark Installation

Write the following command for opening Spark shell.

$spark-shell

If spark is installed successfully then you will find the following output.

Spark assembly has been built with Hive, including Datanucleus jars on classpath 

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 

15/06/04 15:25:22 INFO SecurityManager: Changing view acls to: hadoop 

15/06/04 15:25:22 INFO SecurityManager: Changing modify acls to: hadoop

15/06/04 15:25:22 INFO SecurityManager: SecurityManager: authentication disabled;

   ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop) 

15/06/04 15:25:22 INFO HttpServer: Starting HTTP Server 

15/06/04 15:25:23 INFO Utils: Successfully started service 'HTTP class server' on port 43292. 

Welcome to 

      ____              __ 

     / __/__  ___ _____/ /__ 

    _\ \/ _ \/ _ `/ __/  '_/ 

   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0 

      /_/  

                

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_71) 

Type in expressions to have them evaluated. 

Spark context available as sc  

scala> 

How To Install and Use PostgreSQL on Ubuntu

 Since we haven't updated our local apt repository lately, let's do that now. We can then get the Postgres package and a "contrib" package that adds some additional utilities and functionality:

$sudo apt-get update 
$sudo apt-get install postgresql postgresql-contrib

 The installation procedure created a user account called postgres that is associated with the default Postgres role. In order to use Postgres, we'll need to log into that account. You can do that by typing:

$sudo -i -u postgres

 You will be asked for your normal user password and then will be given a shell prompt for the postgres user.

You can get a Postgres prompt immediately by typing:

$psql 

Exit out of the PostgreSQL prompt by typing: 

$\q

            PostgresSQL In UI Mod

 Step 1 - Installing PostgreSQL, phpPgAdmin and Apache2

PostgreSQL and PhpPgAdmin are available in the Ubuntu repository. So you just need to install them with the apt command.  


$sudo apt-get -y install postgresql postgresql-contrib phppgadmin

Step 2 - Configure PostgreSQL user

 PostgreSQL uses role for user authentication and authorization, it just like Unix-Style permissions. By default, PostgreSQL creates a new user called "postgres" for basic authentication. To use PostgreSQL, you need to login to the "postgres" account, you can do that by typing:

  $sudo su

  $su - postgres

Now you can access the PostgreSQL prompt with the command:

$psql

And then change the password for postgres role by typing:

$\password postgres

ENTER YOUR PASSWORD

Then enter \q to leave the psql command line.

Run the command "exit" to leave the postgres user and become root again.

$exit

Step 3 - Configure Apache2 

You need to configure apache for phpPgAdmin. Edit the file /etc/apache2/conf-available/phppgadmin.conf with nano by typing:

$cd /etc/apache2/conf-available/

nano phppgadmin.conf 

Comment out the line #Require local by adding a # in front of the line and add below the line allow from all so that you can access from your browser.

 Step 4 - Configure phpPgAdmin

 Edit the file /etc/phppgadmin/config.inc.php by typing : 

$cd /etc/phppgadmin/

 nano config.inc.php

Find the line $conf['extra_login_security'] = true; and change the value to false so you can login to phpPgAdmin with user postgres. 

Step 5 - Restart PostgreSQL and Apache2

$systemctl restart postgresql

$systemctl restart apache2

Step 6 - Testing

Now access phpPgAdmin with your browser http://yourip/phppgadmin/. 

 

and then try login to with user postgres and your password.

 

After logging in you will get this interface:

 

Pentaho Installation

                 Pentaho Installation  For Ubuntu

Step 1:Download:Pentoho
            From http://community.pentaho.com/projects/data-integration/  

Step 2: Go to Terminal
  Snipe@ubuntu:~$ cd /home/snipe/Desktop/data-integration/spoon.sh 
  snipe@ubuntu:~$ ./spoon.sh

   

Oozie Installation

Oozie Installation in Ubuntu

Step 1 :- To install the Oozie server package on an Ubuntu and other Debian system:

$ sudo apt-get install oozie  

Step 2 :- To install the Oozie client package on an Ubuntu and other Debian system:

$ sudo apt-get install oozie-client

Step 3 :- Configuring which Hadoop Version to Use

To use MRv1(without SSL) :

alternatives --set oozie-tomcat-conf /etc/oozie/tomcat-conf.http.mr1

Step 4 :- Edit /etc/oozie/conf/oozie-env.sh file and make the entry

export CATALINA_BASE=/var/lib/oozie/tomcat-deployment

Step 5 :- Start the Oozie server

$ sudo service oozie start

Step 6 :- Accessing the Oozie Server with the Oozie Client

The Oozie client is a command-line utility that interacts with the Oozie server via the Oozie web-services API.

Use the /usr/bin/oozie script to run the Oozie client.

For example, if you want to invoke the client on the same machine where the Oozie server is running:

$ oozie admin -oozie http://localhost:11000/oozie -status
System mode: NORMAL

To make it convenient to use this utility, set the environment variable OOZIE_URL to point to the URL of the Oozie server. Then you can skip the -oozie option.

For example, if you want to invoke the client on the same machine where the Oozie server is running, set the OOZIE_URL to http://localhost:11000/oozie.

$ export OOZIE_URL=http://localhost:11000/oozie
$ oozie admin -version
Oozie server build version: 4.0.0-cdh5.0.0


Step 7 :- Confiduring MySQL for Oozie

Step 1: Create the Oozie database and Oozie MySQL user.

For example, using the MySQL mysql command-line tool:

$ mysql -u root -p

Enter password: ******

mysql> create database oozie;

Query OK, 1 row affected (0.03 sec)

mysql>  grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';

Query OK, 0 rows affected (0.03 sec)

mysql>  grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';

Query OK, 0 rows affected (0.03 sec)

mysql> exit

Bye

Step 2: Configure Oozie to use MySQL.

Edit properties in the oozie-site.xml file as follows:

...

    <property>

        <name>oozie.service.JPAService.jdbc.driver</name>

        <value>com.mysql.jdbc.Driver</value>

    </property>

    <property>

        <name>oozie.service.JPAService.jdbc.url</name>

        <value>jdbc:mysql://localhost:3306/oozie</value>

    </property>

    <property>

        <name>oozie.service.JPAService.jdbc.username</name>

        <value>oozie</value>

    </property>

    <property>

        <name>oozie.service.JPAService.jdbc.password</name>

        <value>oozie</value>

    </property>

    ...

Step 3 : Creating the Oozie DatabaseSchema

$ sudo -u oozie /usr/lib/oozie/bin/ooziedb.sh create -sqlfile oozie-create.sql

or

$ sudo -u oozie /usr/lib/oozie/bin/ooziedb.sh create -run

Step 4 : Enabling the Oozie Web Console

To enable Oozie's web console, you must download and add the ExtJS library to the Oozie server. If you have not already done this, proceed as follows.

Step 4.1: Download the Library

Download the ExtJS version 2.2 library from http://archive.cloudera.com/gplextras/misc/ext-2.2.zip and place it a convenient location.

Step 4.2: Install the Library

Extract the ext-2.2.zip file into /var/lib/oozie. 

$ cd Downloads/

$ sudo cp -avr ext-2.2 /var/lib/oozie/

Step 5 : Installing the Oozie Shared Library in Hadoop HDFS

The Oozie installation bundles the Oozie shared library, which contains all of the necessary JARs to enable workflow jobs to run streaming, DistCp, Pig, Hive, and Sqoop actions.

The Oozie installation bundles two shared libraries, one for MRv1 and one for YARN. Make sure you install the right one for the MapReduce version you are using:

·                     The shared library file for MRv1 is oozie-sharelib-mr1.tar.gz.

·                     The shared library file for YARN is oozie-sharelib-yarn.tar.gz.

sudo -u oozie oozie  admin -shareliblist -oozie http://localhost:11000/oozie

sudo service oozie restart

To install the Oozie shared library in Hadoop HDFS in the oozie user home directory

$ sudo -u hdfs hadoop fs -mkdir /user/oozie
$ sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie
$ sudo oozie-setup sharelib create -fs hdfs://localhost:8020 -locallib /usr/lib/oozie/oozie-sharelib-mr1

add the below line to oozie-site.xml to recognize the shared lib functionality

<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/etc/hadoop/conf</value>
<description>
Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
the relevant Hadoop *-site.xml files. If the path is relative is looked within
the Oozie configuration directory; though the path can be absolute (i.e. to point
to Hadoop client conf/ directories in the local filesystem.
</description>
</property>

Configuring Support for Oozie Uber JARs

An uber JAR is a JAR that contains other JARs with dependencies in a lib/ folder inside the JAR. You can configure the cluster to handle uber JARs properly for the MapReduce action (as long as it does not include any streaming or pipes) by setting the following property in the oozie-site.xmlfile:

...

    <property>

        <name>oozie.action.mapreduce.uber.jar.enable</name>

    <value>true</value>

    ...

When this property is set, users can use the oozie.mapreduce.uber.jar configuration property in their MapReduce workflows to notify Oozie that the specified JAR file is an uber JAR.

Configuring Oozie to Run against a Federated Cluster

To run Oozie against a federated HDFS cluster using ViewFS, configure the oozie.service.HadoopAccessorService.supported.filesystems property in oozie-site.xml as follows:

<property>

     <name>oozie.service.HadoopAccessorService.supported.filesystems</name>

     <value>hdfs,viewfs</value>

</property>

Trouble shooting
sudo cp mysql-connector-java-5.1.35-bin.jar /var/lib/oozie/

HBase Installation

Installing HBase

We can install HBase in any of the three modes: Standalone mode, Pseudo Distributed mode, and Fully Distributed mode.

Installing HBase in Standalone Mode

Download the latest stable version of HBase form http://www.interior-dsgn.com/apache/hbase/stable/ using “wget” command, and extract it using the tar “zxvf” command. See the following command.

$cd usr/local/

$wget http://www.interior-dsgn.com/apache/hbase/stable/hbase-0.98.8-

hadoop2-bin.tar.gz

$tar -zxvf hbase-0.98.8-hadoop2-bin.tar.gz

Shift to super user mode and move the HBase folder to /usr/local as shown below.

$su

$password: enter your password here

mv hbase-0.99.1/* Hbase/

Configuring HBase in Standalone Mode

Before proceeding with HBase, you have to edit the following files and configure HBase.

hbase-env.sh

Set the java Home for HBase and open hbase-env.sh file from the conf folder. Edit JAVA_HOME environment variable and change the existing path to your current JAVA_HOME variable as shown below.

cd /usr/local/Hbase/conf

gedit hbase-env.sh

This will open the env.sh file of HBase. Now replace the existing JAVA_HOME value with your current value as shown below.

export JAVA_HOME=/usr/lib/jvm/java-1.7.0

hbase-site.xml

This is the main configuration file of HBase. Set the data directory to an appropriate location by opening the HBase home folder in /usr/local/HBase. Inside the conf folder, you will find several files, open the hbase-site.xml file as shown below.

#cd /usr/local/HBase/

#cd conf

# gedit hbase-site.xml

Inside the hbase-site.xml file, you will find the <configuration> and </configuration> tags. Within them, set the HBase directory under the property key with the name “hbase.rootdir” as shown below.

<configuration>

   //Here you have to set the path where you want HBase to store its files.

   <property>

      <name>hbase.rootdir</name>

      <value>file:/home/hadoop/HBase/HFiles</value>

   </property>

           

   //Here you have to set the path where you want HBase to store its built in zookeeper  files.

   <property>

      <name>hbase.zookeeper.property.dataDir</name>

      <value>/home/hadoop/zookeeper</value>

   </property>

</configuration>

With this, the HBase installation and configuration part is successfully complete. We can start HBase by using start-hbase.sh script provided in the bin folder of HBase. For that, open HBase Home Folder and run HBase start script as shown below.

$cd /usr/local/HBase/bin

$./start-hbase.sh

If everything goes well, when you try to run HBase start script, it will prompt you a message saying that HBase has started.

starting master, logging to /usr/local/HBase/bin/../logs/hbase-tpmaster-localhost.localdomain.out

Installing HBase in Pseudo-Distributed Mode

Let us now check how HBase is installed in pseudo-distributed mode.

CONFIGURING HBASE

Before proceeding with HBase, configure Hadoop and HDFS on your local system or on a remote system and make sure they are running. Stop HBase if it is running.

hbase-site.xml

Edit hbase-site.xml file to add the following properties.

<property>

   <name>hbase.cluster.distributed</name>

   <value>true</value>

</property>

It will mention in which mode HBase should be run. In the same file from the local file system, change the hbase.rootdir, your HDFS instance address, using the hdfs://// URI syntax. We are running HDFS on the localhost at port 8030.

<property>

   <name>hbase.rootdir</name>

   <value>hdfs://localhost:8030/hbase</value>

</property>

Starting HBase

After configuration is over, browse to HBase home folder and start HBase using the following command.

$cd /usr/local/HBase

$bin/start-hbase.sh

Note: Before starting HBase, make sure Hadoop is running.

Checking the HBase Directory in HDFS

HBase creates its directory in HDFS. To see the created directory, browse to Hadoop bin and type the following command.

$ ./bin/hadoop fs -ls /hbase

If everything goes well, it will give you the following output.

Found 7 items

drwxr-xr-x - hbase users 0 2014-06-25 18:58 /hbase/.tmp

drwxr-xr-x - hbase users 0 2014-06-25 21:49 /hbase/WALs

drwxr-xr-x - hbase users 0 2014-06-25 18:48 /hbase/corrupt

drwxr-xr-x - hbase users 0 2014-06-25 18:58 /hbase/data

-rw-r--r-- 3 hbase users 42 2014-06-25 18:41 /hbase/hbase.id

-rw-r--r-- 3 hbase users 7 2014-06-25 18:41 /hbase/hbase.version

drwxr-xr-x - hbase users 0 2014-06-25 21:49 /hbase/oldWALs

Starting and Stopping a Master

Using the “local-master-backup.sh” you can start up to 10 servers. Open the home folder of HBase, master and execute the following command to start it.

$ ./bin/local-master-backup.sh 2 4

To kill a backup master, you need its process id, which will be stored in a file named “/tmp/hbase-USER-X-master.pid.” you can kill the backup master using the following command.

$ cat /tmp/hbase-user-1-master.pid |xargs kill -9

Starting and Stopping RegionServers

You can run multiple region servers from a single system using the following command.

$ .bin/local-regionservers.sh start 2 3

To stop a region server, use the following command.

$ .bin/local-regionservers.sh stop 3

Starting HbaseShell

After Installing HBase successfully, you can start HBase Shell. Below given are the sequence of steps that are to be followed to start the HBase shell. Open the terminal, and login as super user.

Start Hadoop File System

Browse through Hadoop home sbin folder and start Hadoop file system as shown below.

$cd $HADOOP_HOME/sbin

$start-all.sh

Start HBase

Browse through the HBase root directory bin folder and start HBase.

$cd /usr/local/HBase

$./bin/start-hbase.sh

Start HBase Master Server

This will be the same directory. Start it as shown below.

$./bin/local-master-backup.sh start 2 (number signifies specific

server.)

Start Region

Start the region server as shown below.

$./bin/./local-regionservers.sh start 3

Start HBase Shell

You can start HBase shell using the following command.

$cd bin

$./hbase shell

This will give you the HBase Shell Prompt as shown below.

2016-12-09 14:24:27,526 INFO [main] Configuration.deprecation:

hadoop.native.lib is deprecated. Instead, use io.native.lib.available

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.8-hadoop2, r6cfc8d064754251365e070a10a82eb169956d5fe, Fri

Nov 14 18:26:29 PST 2016

hbase(main):001:0>

SQOOP INSTALLATION

SQOOP  INSTALLATION

Step 1: Installing Sqoop

The following commands are used to extract the Sqoop tar ball and move it to “/usr/lib/sqoop” directory.

$tar -xvf sqoop-1.4.4.bin__hadoop-2.0.4-alpha.tar.gz

$ su

password:

# mv sqoop-1.4.4.bin__hadoop-2.0.4-alpha /usr/lib/sqoop

#exit

Step 2: Configuring bashrc

You have to set up the Sqoop environment by appending the following lines to ~/.bashrc file:

#Sqoop

export SQOOP_HOME=/usr/lib/sqoop export PATH=$PATH:$SQOOP_HOME/bin

The following command is used to execute ~/.bashrc file.

$ source ~/.bashrc

Step 3: Configuring Sqoop

To configure Sqoop with Hadoop, you need to edit the sqoop-env.sh file, which is placed in the $SQOOP_HOME/conf directory. First of all, Redirect to Sqoop config directory and copy the template file using the following command:

$ cd $SQOOP_HOME/conf

$ mv sqoop-env-template.sh sqoop-env.sh

Open sqoop-env.sh and edit the following lines:

export HADOOP_COMMON_HOME=/usr/local/hadoop

export HADOOP_MAPRED_HOME=/usr/local/hadoop

Step 4: Download and Configure mysql-connector-java

We can download mysql-connector-java-5.1.30.tar.gz file from the following link.

The following commands are used to extract mysql-connector-java tarball and move mysql-connector-java-5.1.30-bin.jar to /usr/lib/sqoop/lib directory.

$ tar -zxf mysql-connector-java-5.1.30.tar.gz

$ su

password:

# cd mysql-connector-java-5.1.30

# mv mysql-connector-java-5.1.30-bin.jar /usr/lib/sqoop/lib

Step 5: Verifying Sqoop

The following command is used to verify the Sqoop version.

$ cd $SQOOP_HOME/bin

$ sqoop-version

Expected output:

15/12/17 14:52:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5

Sqoop 1.4.5 git commit id 5b34accaca7de251fc91161733f906af2eddbe83

Compiled by abe on Fri Dec 1 11:19:26 PDT 2015

 Sqoop installation is complete.