Thursday, 30 May 2013

Script to delete all tables in Hive

           
                There is no query is available for delete all the tables in hive. so we need to write a script for delete the all the tables in hive. The below script is used to delete the all the table in hive.
1.Delete all the tables in hive without database
$ vi hive-table_delete.sh
  hive -e 'show tables' | xargs -I '{}' hive -e 'drop table {}'
2.Delete all the tables in hive with database
$ vi hive-table_delete.sh
  hive -e 'use databasename;show tables' | xargs -I '{}' hive -e 'drop table {}'

Note:
 X args command in UNIX or Linux operating system is used to pass the output of one command as an argument to another command


Wednesday, 22 May 2013


PUPPET INTRODUCTION, INSTALLATION AND CONFIGURATION
INTRODUCTION:
                      Puppet is an open source framework and tools for managing the configuration of the cluster.it is written in ruby language and it is used to manage the cluster configuration in UNIX, Linux and windows platform also.
Why we need puppet?
               For an example if you have thousand node in your cluster. You want change the single configuration file from the entire cluster; it is very tough task because it is taks lots of time as well as if you make any mistake in the configuration file. The entire cluster goes in wrong path. so avoid the above problem we need one tool for doing the above task in Automatically, that tools is puppet and chef.
 Who is developed puppet?
               Puppet is developed by Luke kanies and his company. He was working as UNIX system administrator since 1997. At the time he was facing lots of problem in the configuration management. So he decided to develop the one software for doing the configuration management in automatically, that tool is puppet.
How puppet is works?
       Puppet is working like a master slave architecture t is consist of two servers
    1. Puppet master
   2. Puppet agent
Puppet master:
             It is a Heart of puppet. Without master node we can’t able to do anything in the cluster. so it is a single point of failure. it is contain the all the configuration of different to different hosts.

Puppet Agents:
          The puppet agent is installed in each on every slave nodes. The puppet agent will connect to the puppet master with regular time interval. The connection between the puppet agent and puppet master is made by secure encrypted channel with the help of SSL


Every 30 min the puppet agents will go and get the new configuration from the puppet master. If suppose the agent is already applied the puppet master configuration it will not do anything.


INSTALLATION OF PUPPET:
 Common for all nodes:
1. # puppet needs epel,rpmforge repository configured for this
2. #installs the necessary package
       $ yum install ruby-shadow ruby ruby-libs  rubygems
     Master Node:
        1. #installs puppet agent and Puppet Master
                $ yum install puppet puppet-server facter
                      note:
                         1. The package "puppet" contains the puppet agent
                         2. The package "puppet-server" contains the puppet master server
                          3. The package "facter" will contain the tool which will act as fetching                                      information about the node.
        Slave Nodes:
               1. # installs puppet agent in the slave node
                          $ yum install puppet facter
                                      <!-----------------------installation is over-------------->
Configuring Puppet:                 
    Master Node:
     1. Create the configuration file
           $ cd /etc/puppet
           $ puppetmasterd --genconfig > puppet.conf
     2. Add the below line in the puppet.conf file
           $ vi puppet.conf
                [master]
                certname=masterhostname
     3. Create the directory and create the site.pp file in this dir
               $ mkdir /etc/puppet/manifests
               $ touch /etc/puppet/manifests/site.pp
              note:Puppet will not start without the site.pp file being present         
     4. Please run this command it will show the permission error
        $ service puppetmaster start        
     5. Change the permission and ownership
               $ ls /var/lib/puppet/
               $ cd /var/lib
               $ chown puppet:puppet puppet
               $ cd puppet
               $ chown puppet:puppet *
      6. Start the puppet Master in this node:
             $ service puppetmaster start
       Slave nodes:
               1. Add the puppet.conf file             
              $ cd /etc/puppet
              $        vi puppet.conf
                    [main]
                        server=masterhostname
                        moduledir = /etc/puppet/modules:/var/lib/puppet/modules
             2. # connect slave in master node
                           $ puppet agent --server=masterhost --no-daemonize --verbose
              Note: surely you will get the below message
                             1.notice: Did not receive certificate
             3. # how to solve the above problem
                         3. a# goto master node:
                              $ puppetca --list
                         (This command will show you the agent certificate requests that are waiting                            to be signed)
                        3. b. now the master will sign the certificate
                              $ puppetca --sign slavehostname
        4. In terminal you will get the below message
             Notice: Finished catalog run in 0.14 seconds 

CREATE OUR FIRST CONFIGURATION:
     
    What is manifests in puppet?
              manifest is nothing but a name that puppet Master calls those files which contain the configuration options for the clients.An important fact to note is that all manifest files will also have a .pp extension just the same as site.pp file. The manifests is made up of number components.they are lists in below
1. Resources – Individual configuration items
2. Files – Physical files you can serve out to your agents
3. Templates – Template files that you can use to populate files
4. Nodes – Specifies the configuration of each agent
5. Classes – Collections of resources
6.  Definitions – Composite collections of resources


SAMPLE CONFIGURATION MANAGEMENT:
    EXAMPLE1: Install the memcached package from all slave nodes    
1.adding the nodes in master:
   1.#go to directory edit the site.pp file

          $cd /etc/puppet/manifests
            $ vi site.pp
                 import 'nodes.pp'
2.# create the nodes.pp file in the  /etc/puppet/manifests directory
    $ vi nodes.pp
              node 'nn1.host.com' {
                                    include memcached
                                    }
                                    node 'dn1.host.com' {
                                    include memcached

                                    }
                                    node 'dn2.host.com' {
                                    include memcached
                                    }

2.adding modules(memcached modules)

1# Find your modulepath
  $ puppet --genconfig |grep modulepath
  $ cd /etc/puppet/modules
2.# Create a directory memcached:
  $ mkdir memcached
3.# Inside this, create manifestsand files directories:
  $ cd memcached
  $ mkdir manifests files 
4.In the manifestsdirectory, create the file init.pp with the following contents
  $ vi init.pp 
    import "*"
5.In the same directory, create another file memcached.pp with the following contents
 $  vi memcached.pp
  class memcached {
    package { "memcached":
      ensure => installed,
         }
      file { "/etc/memcached.conf":
     source => "puppet:///modules/memcached/memcached.conf",
            }
                                    service { "memcached":
                                    ensure => running,
                                    enable => true,
                                    require => [ Package["memcached"],
                                    File["/etc/memcached.conf"] ],
                                    }
                                    }
6.Changing to the filesdirectory, create the file memcached.conf with the following contents:           
  $ vi /etc/puppet/modules/memcached/files/memcached.conf
        -m 64
                        -p 11211
                        -u nobody
                        -l 127.0.0.1
7.Run Puppet to test the new configuration
  puppet agent --test

EXAMPLE 2: Edit the configuration files in slave nodes with in the help of Augeas
what is Augeas?
     Augeas is a tool which aims to simplify working with different config file formats, by presenting them all as a simple tree of values. Puppet's Augeas support allows you to create augeas resources which can make the required config changes intelligently and automatically.

 1.Install Augeas from all slave nodes using puppet
     1.#go to directory edit the site.pp file

          $cd /etc/puppet/manifests
            $ vi site.pp
                 import 'nodes.pp'
2.# create the nodes.pp file in the  /etc/puppet/manifests directory
    $ vi nodes.pp
              node 'nn1.host.com' {
                                    include augeas
                                    }
                                    node 'dn1.host.com' {
                                    include augeas
                                     }
                                    node 'dn2.host.com' {

                                    include  augeas
                                      }
2.adding modules(augeas modules)

1# Find your modulepath
  $ puppet --genconfig |grep modulepath
  $ cd /etc/puppet/modules
2.# Create a directory agueas:
  $ mkdir agueas
3.# Inside this, create manifests and files directories:
  $ cd agueas
  $ mkdir manifests files 
4.In the manifestsdirectory, create the file init.pp with the following contents
  $ vi init.pp 
    import "*"
5.In the same directory, create another file augeas.pp with the following contents
      $ vi augeas.pp
    class  augeas {
           package { [ "augeas",
                             "augeas-devel",
                             "augeas-libs",
                                     ]:
          ensure => "present"
   }
       }
6.Run Puppet to test the new configuration
    $ puppet agent --test
  

3.Enable the Ip forwared 4
1.In the manifestsdirectory, create the file init.pp with the following contents
$ vi ipforward.pp
   class augeas::ipforward {
            augeas { "enable-ip-forwarding":
            context => "/files/etc/sysctl.conf",
            changes => [
            "set net.ipv4.ip_forward 1",
            ],
            }
            }
2..# edit the nodes.pp file in the  /etc/puppet/manifests directory
   $ vi nodes.pp
              node 'nn1.host.com' {
                                    include augeas
                                    include augeas::ipforward
                                    }
                                    node 'dn1.host.com' {
                                    include augeas
                                    include agueas::ipforward
                                     }
                                    node 'dn2.host.com' {
                                    include  augeas
                                    include augeas::ipforward
                                      }

3.Run Puppet to test the new configuration
    $ puppet agent --test
      
              

Monday, 20 May 2013

LIST OF SQOOP COMMANDS

1.sqoop-import --connect jdbc:mysql://localhost/srini -username root -password sql -table ran -m 1

2.sqoop-import --connect jdbc:mysql://localhost/srini -username root -password sql -table ran50 --where "name='nagu'" -m 1;

3.sqoop-import --connect jdbc:mysql://localhost/srini -username root -password sql -table ran50 --where "name='nagu'" --target-dir /nagu --append -m 1;

4.sqoop-import --connect jdbc:mysql://localhost/srini -username root -password sql -table ran50 --where "age>25" --target-dir /nagu1 -m 1;

5.sqoop-list-tables --connect jdbc:mysql://localhost/srini -username root -password sql

6.sqoop-list-databases --connect jdbc:mysql://localhost/srini -username root -password sql

7.sqoop-codegen --connect jdbc:mysql://localhost/srini --table ran90

8.sqoop-import --connect jdbc:mysql://localhost/srini -username root -password sql -table ran -columns name -m 1

9.sqoop-import --connect jdbc:mysql://localhost/srini -username root -password sql -table ran70 --as-sequencefile -m 1;

10.sqoop-import --connect jdbc:mysql://localhost/srini -username root -password sql -table ran80 --as-textfile -m 1;

11.sqoop export -connect jdbc:mysql://localhost/srini -table ranexport --export-dir /user/ranjith/ran --username root --password sql -m 1 --input-fields-terminated-by ','

12.sqoop-create-hive-table --connect jdbc:mysql://localhost/srini -username root -password sql --table ranexport --fields-terminated-by ','

13.sqoop import --connect jdbc:mysql://localhost/srini -username root -password sql --table ran2 --hive-import -m 1;

14.sqoop import --connect jdbc:mysql://localhost/srini -username root -password sql --table ran3 --hive-import --fields-terminated-by ',' -m 1;

15.sqoop export --connect jdbc:mysql://localhost/srini -username root -password sql --table ran65 --export-dir /user/hive/warehouse/ran3/part-m-00000_copy_2 --input-fields-terminated-by ',';

16.sqoop-import-all-tables --connect jdbc:mysql://localhost/srini -username root -password sql --hive-import -m1;

Reference link is:http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html#_literal_sqoop_codegen_literal