Monday, 5 June 2017

Restore Single Table in Mysql

Steps to Restore Single table from MYSQL Dump:


Suppose we have multiple table as below:

mysql> show tables;
+-----------------------+
 | Tables_in_test  |
+-----------------------+
 | mytable1          |
 | mytable2          |
 | mytable3          |
+-----------------------+
3 rows in set (0.00 sec)

To dump from table we can use below command
$ mysqldump -u username -p -h XXX.XXX.XXX.XXX test > mytestdump.sql

Search for the string pattern table structure as below
$ grep -n 'Table structure' mytestdump.sql

120:-- Table structure for table `mytable1`
304:-- Table structure for table `mytable2`
810:-- Table structure for table `mytable3`

Once we get the starting line of each table we can use sed command to create a file for individual table as below
$ sed -n '304,810 p' mytestdump.sql > mytable2.sql

Now with the below command we can restore individual table into database
$ mysql -u username -p -h XXX.XXX.XXX.XXX DATA-BASE-NAME < mytable2.sql

Sunday, 20 December 2015

Hadoop Commands

Important Commands



To check the hadoop disk detail we need to execute the below command
hadoop dfsadmin –report

To check the list of the nodes that are in the cluster you can run this script:  ./listnodes.sh
It is located in /opt/ibm/biginsights/bin

To create a directory in hdfs through command line
hadoop fs –mkdir filename

To copy file in hdfs directly from command prompt

hadoop fs –copyFromLocal /tmp/filname hdfs://binode1:/9000/folderpath

List all the blocks of file in the filesystem
hadoop fsck / -files -blocks

Copy from local
hadoop fs -copyFromLocal input/docs/quangle.txt /user/tom/quangle.txt

Command for CMX compression
Create  jaql script with extention *.jaql and use the below command
setOptions ( {conf : {"mapred.output.compress" : true, "mapred.output.compression.codec" : "com.ibm.biginsights.compress.CmxCodec" }});

read(lines("/user/biadmin/Activity/Activity.log"))-> write(lines("HDFSActivity/Activity/CMX"));
hdfsShell("-getmerge HDFSActivity/Activity/CMX    /opt/mydata/Activity/Activity.cmx");

hdfsShell("-copyFromLocal /opt/mydata/Activity/Activity.cmx   /user/biadmin/Activity/ActivityCompressedCMX");

Friday, 26 September 2014

Automated IBM Infosphere Biginsights Installation


After creating a small hadoop cluster through IBM Infosphere BigInsights, tried to automate all the steps to create a master node. No matter whatever be the latest build of BigInsights we have.

Hope it help others to do the same.

Supported OS: Centos 6X, Fedora, RHEL

#======================================================================
# Automated script for biginsight installation
#======================================================================
#INSTRUCTION:
# 1.Please follow the below commented manual steps for ssh key based authentication before executing the script
#  2.Put the tar file and the script in the same location
#  3.Pass the name of the tar file as command line argument
#Example:
# $./automatedbiginsights.sh biginsights.enterprise-linux64_b20141813_1321.tar.gz
#======================================================================

# Follow the below extra steps manually
## Generate the keybased login, for these ssh key based authentication should be enabled in the system ( login as a root user)
           ssh-keygen –t rsa            (Press Enter for default)
## Press enter again for default passphrase
           cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys    (login as a root)
           ssh localhost     (Enter yes)
           exit

## Login as biadmin
          su biadmin

          ssh-keygen –t rsa            (Press Enter for default )

# Press enter again for default passphrase

          cat /home/biadmin/.ssh/id_rsa.pub >> /home/biadmin/.ssh/authorized_keys

          ssh localhost     (Enter yes)
          Password:        

          exit        (connection should be closed)

# # Login as root again
          su root
          Password:

          cd  /home/biadmin/.ssh
          ls –l

          chmod 640 authorized_keys                      ( Change the mode of the file)

# # Login as biadmin
          su biadmin

          ssh localhost                     (password prompt should not be there)

          exit

# # Login as root again and edit the sudoers file
          su root

          vi /etc/sudoers

##Search the below line in sudoers file and Disable it
          #Defaults requiretty  
             
## Search for “wheel” and add the below line
                 biadmin               ALL=(ALL)             NOPASSWD:ALL        
         :wq!

#======================================================================
#Copy the below line of code in a script automatedbiginsights.sh and execute it as per the direction mentioned above
#======================================================================

echo "BigInsights Installation begins..........................."

sudo chkconfig sshd on
sudo yum install expect
sudo yum install gawk

filename="biginsights"   #File name biginsights.enterprise-linux64_b20141813_1321 shall be renamed

mkdir -p /home/${USER}/ibm
echo "biadmin/BigData Directory has been created......"

echo " File extraction begin.............."

tarfile=$1
fname=`echo $1 |cut -d'.' -f1`

tar -xzf $1 --transform 's/'${fname}'/'${filename}'/' && mv "${filename}" /home/${USER}/ibm

returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "File has been extracted successfully....."
else
echo "Something is wrong. Please check manually....."
exit 1
fi


bigInsightpath="/home/${USER}/ibm/biginsights"
var1="enable"
var2="all"

#cd ${biginsightpath}/installer/bin
echo " Execution of enableOrDisablePrechecker.sh script has started......."
sudo sh ${biginsightpath}/installer/bin/enableOrDisablePrechecker.sh $var1 $var2

returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "enableOrDisablePrechecker.sh script successfully....."
else
echo "Something is wrong. Please check manually....."
exit 2
fi


sudo sestatus
sudo setenforce         0

echo "SELINUX service will be disabled now, Modification of /etc/selinux/config begins........"
sudo sed -e 's/SELINUX=enable/SELINUX=disabled/g' /etc/selinux/config > /etc/selinux/temp.txt  && sudo mv /etc/selinux/temp.txt /etc/selinux/config

returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "/etc/selinux/config has been modified successfully....."
else
echo "Something is wrong. Please check manually....."
exit 3
fi


sudo rpm –ivH ntp
sudo chkconfig ntpd  --list
sudo service ntpd status      
sudo service ntpd start
sudo service ntpd date


echo "Changing Ulimit value......"
sudo Ulimit –n             32768
sudo Ulimit –u             60000

returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "Operation completed successfully....."
else
echo "Something is wrong. Please check manually....."
exit 4
fi

echo "/etc/security/limits.conf file modification begins........."

sudo
echo "biadmin               hard      nofile   65536
 biadmin               soft      nofile   65536
 biadmin               hard      nproc    65536
 biadmin               soft      nproc    65536 " >> /etc/security/limits.conf


returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "Operation completed successfully....."
else
echo "Something is wrong. Please check manually....."
exit 5
fi


echo "/etc/sysctl.conf file modification begins........."

sudo
echo "kernel.pid_max = 4194303
 net.ipv4.ip_local_port_range = 1024      64000
 net.ipv6.conf.all.disable_ipv6 = 1 " >> /etc/sysctl.conf

returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "Operation completed successfully....."
else
echo "Something is wrong. Please check manually....."
exit 6
fi


echo "/etc/sysconfig/network file modification begins........."

sudo
echo "NETWORKING_IPV6=no
 IPV6INIT=no " >> /etc/sysconfig/network

returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "Operation completed successfully....."
else
echo "Something is wrong. Please check manually....."
exit 7
fi


sudo chown –R ${USER}:${USER} /tmp/${filename}

sudo chkconfig ntpd on

username=${USER}
entrprise="ENTERPRISE"

#cd ${biginsightpath}/installer/hdm/bin

echo "bi-prechecker.sh  script execution begins........."
sudo sh ${biginsightpath}/installer/hdm/bin/bi-prechecker.sh          -u     $username       -m          $entrprise

returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "Operation completed successfully....."
else
echo "Something is wrong. Please check manually....."
exit 8
fi


echo "start.sh script execution begins........."

#cd ${biginsightpath}
sudo sh ${biginsightpath}/start.sh

returnvalue=$?

if [ $returnvalue -eq 0 ]; then
echo "Operation completed successfully....."
else
echo "Something is wrong. Please check manually....."
exit 9
fi

exit
#======================================================================
#                                                     END
#======================================================================   

Ref:
http://www-01.ibm.com/support/knowledgecenter/SSPT3X_2.1.2/com.ibm.swg.im.infosphere.biginsights.install.doc/Install.pdf
http://www-01.ibm.com/support/knowledgecenter/SSPT3X_2.1.1/com.ibm.swg.im.infosphere.biginsights.install.doc/Install.pdf

         
   
Feel free to ask any query.... :)