Hadoop Commands

Important Commands

To check the hadoop disk detail we need to execute the below command

hadoop dfsadmin –report

To check the list of the nodes that are in the cluster you can run this script:  ./listnodes.sh

It is located in /opt/ibm/biginsights/bin

To create a directory in hdfs through command line

hadoop fs –mkdir filename

To copy file in hdfs directly from command prompt

hadoop fs –copyFromLocal /tmp/filname hdfs://binode1:/9000/folderpath

List all the blocks of file in the filesystem

hadoop fsck / -files -blocks

Copy from local

hadoop fs -copyFromLocal input/docs/quangle.txt /user/tom/quangle.txt

Command for CMX compression
Create  jaql script with extention *.jaql and use the below command
setOptions ( {conf : {"mapred.output.compress" : true, "mapred.output.compression.codec" : "com.ibm.biginsights.compress.CmxCodec" }});

read(lines("/user/biadmin/Activity/Activity.log"))-> write(lines("HDFSActivity/Activity/CMX"));
hdfsShell("-getmerge HDFSActivity/Activity/CMX    /opt/mydata/Activity/Activity.cmx");

hdfsShell("-copyFromLocal /opt/mydata/Activity/Activity.cmx   /user/biadmin/Activity/ActivityCompressedCMX");