Sunday 20 December 2015

Hadoop Commands

Important Commands



To check the hadoop disk detail we need to execute the below command
hadoop dfsadmin –report

To check the list of the nodes that are in the cluster you can run this script:  ./listnodes.sh
It is located in /opt/ibm/biginsights/bin

To create a directory in hdfs through command line
hadoop fs –mkdir filename

To copy file in hdfs directly from command prompt

hadoop fs –copyFromLocal /tmp/filname hdfs://binode1:/9000/folderpath

List all the blocks of file in the filesystem
hadoop fsck / -files -blocks

Copy from local
hadoop fs -copyFromLocal input/docs/quangle.txt /user/tom/quangle.txt

Command for CMX compression
Create  jaql script with extention *.jaql and use the below command
setOptions ( {conf : {"mapred.output.compress" : true, "mapred.output.compression.codec" : "com.ibm.biginsights.compress.CmxCodec" }});

read(lines("/user/biadmin/Activity/Activity.log"))-> write(lines("HDFSActivity/Activity/CMX"));
hdfsShell("-getmerge HDFSActivity/Activity/CMX    /opt/mydata/Activity/Activity.cmx");

hdfsShell("-copyFromLocal /opt/mydata/Activity/Activity.cmx   /user/biadmin/Activity/ActivityCompressedCMX");