25 Amazing things you can do in a linux command terminal

LINUX

Linux commandline is absolutely amazing loved by almost all system administrators. Here you will discover powerful and impossible things you can do, just using a command line terminal.

1. Searching files

Searching for certain files are inevitable for admins.  

find /home -type f -name "*.conf"

and to use locate, which is the easiest

sudo updatedb
locate mickey.cpp

Search file contents using grep

$ cat /etc/ hosts | grep -i  "localhost"

To list files that are greater than 500mb, run this. V

find /home/ -type f -size +500M -exec ls -lh {} \;

2. Zip and Unzip

You can easily compress and uncompress .tar.gz and .bz2. Please note that .bz2 takes long time as it is high compression using “j” flag

tar cvfj example.tar.bz2 /home/example (create .bz2 using j flag)
 tar cvzf example.tar.gz  /home/images (create .tar.gz using z flag)
 tar -xzf example.tar.gz -C /home/justin (extract .tar.gz using x and z flag)
 tar -xjf example.bz2 -C /home/melissa (extract .bz2 using x and j flag)

3. Cleaning up files

You can clean up file contents using sed, tr and awk commands in a command line (faster than writing a script).

This tr command will clean up all alpha, numeric and special chars from input file

tr -c "A-Za-z0-9@-._ \n" " " < input.txt > output.txt

This will translate uppercase to lowercase

tr '[:upper:]' '[:lower:]' < rawfiltered.txt > data.txt

Sed command to substitute a word or string in a file.

sed 's/unix/linux/g' myfile.txt

and this will remove blank lines in a file.

sed '/^$/d' input.txt > output.txt

Also you can use sort to sort contents in a file, which also can be merged with sed

sort data.txt > output.txt

You can find more sed examples here: http://www.folkstalk.com/2012/01/sed-command-in-unix-examples.html

4. Security

Check connections from same IPs on Port 80 (http) for multiple ddos attacks

netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1
 netstat -apn | grep ":80"

and also check for http processes

netstat -n | grep :80 |wc -l

5. Hard drive errors

You can look for problems with hard drive using smartctl

$ smartctl -q errorsonly -H -l selftest -l error /dev/sda
 Warning: ATA error count 17 inconsistent with error log pointer 1
 ATA Error Count: 17 (device log contains only the most recent five errors)
 Error 17 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)
 Error 16 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)
 Error 15 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)
 Error 14 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)
 Error 13 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)

You can also use iostat to monitor I/O performance for local hard drive and remote disks

# iostat
 Linux 2.6.18-308.1.1.el5 (server1.example.com)     02/24/2016
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 2.72    0.05    1.11    0.22    0.00   95.89

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              10.71       390.59       208.68  301731339  161206205
sda1              0.00         0.27         0.00     210670          4
sda2              4.81       112.90        94.88   87211079   73297681
sda3              2.23        67.60        37.70   52217183   29124792
sda4              0.00         0.00         0.00        134          0
sda5              1.02        29.13        42.69   22504042   32978328
sda6              0.04         5.43         0.00    4195402       1200
sda7              2.45       175.25        29.83  135379849   23041280
sda8              0.16         0.02         3.58      12282    2762920

6. Cpanel backup and restore

You can pack and restore cpanel account files in a fly using these handy commands in command terminal

/scripts/pkgacct username
 /scripts/restoracct username

7. Remote File Transfer

You can transfer files remotely using SCP from one host to another

scp /home/file.tar.gz user@host.example.com:21Feb16/

8. Server Loads

Check users load, http processes, CPU usage and server load using these commands.

$ uptime
 19:07  up 29 mins, 2 users, load averages: 0.48 0.59 0.55
$ w
18:27  up  2:08, 2 users, load averages: 1.25 1.53 1.44
 USER     TTY      FROM              LOGIN@  IDLE WHAT
 prabhubalakrishnan console  -                16:20    2:06 -
 prabhubalakrishnan s000     -                17:21       - w
$ iostat
disk0       cpu     load average
 KB/t tps  MB/s  us sy id   1m   5m   15m
 51.45  12  0.60   4  1 94  1.06 1.42 1.40

Check for running processes to see if any process takes too much memory or cpu load.

$ ps aux | more

9. Free Disk Space

First, check disk space using this command

# df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/sda5             9.7G  533M  8.7G   6% /
 /dev/sda8             996M   34M  911M   4% /tmp
 /dev/sda7             194G  4.2G  180G   3% /home
 /dev/sda3             9.7G  2.1G  7.2G  23% /var
 /dev/sda2             9.7G  6.0G  3.2G  66% /usr
 /dev/sda1              99M   17M   77M  18% /boot
 tmpfs                1014M     0 1014M   0% /dev/shm

then search for files greater than 100 mb using this command

find /var/www/ -size +100000000c | more

10. DNS Settings

You can check dns settings for your server for correct IP address (A and AAA ipv6).

dig example.com
 host -t  A example.com (ipv4)
 host -t   AAAA example.com (ipv6)

11. Tracing Network Route

Traceroute is an extremely useful tool to fix network path problems.  It shows the transmission path across networks for different continents.

$ traceroute example.com
traceroute to example.com (93.184.216.34), 64 hops max, 52 byte packets
 1  192.168.0.1 (192.168.0.1)  1.267 ms  1.601 ms  1.009 ms
 2  10.23.240.1 (10.23.240.1)  7.628 ms  15.619 ms  11.351 ms
 3  113.193.5.169 (113.193.5.169)  9.953 ms  19.261 ms  8.973 ms
 4  59.163.111.81.static-chennai.vsnl.net.in (59.163.111.81)  9.017 ms  13.421 ms  9.451 ms
 5  172.25.81.134 (172.25.81.134)  56.100 ms  59.540 ms
 172.31.29.245 (172.31.29.245)  52.476 ms
 6  ix-0-100.tcore1.mlv-mumbai.as6453.net (180.87.38.5)  69.474 ms  67.030 ms  47.591 ms
 7  if-9-5.tcore1.wyn-marseille.as6453.net (80.231.217.17)  157.941 ms  153.891 ms  159.319 ms
 8  if-8-1600.tcore1.pye-paris.as6453.net (80.231.217.6)  164.418 ms  170.447 ms  159.127 ms
 9  if-2-2.tcore1.pvu-paris.as6453.net (80.231.154.17)  183.550 ms  158.083 ms  166.330 ms
 10  ae-7.r04.parsfr01.fr.bb.gin.ntt.net (129.250.8.1)  165.362 ms  171.770 ms  167.023 ms
 11  ae-8.r23.londen03.uk.bb.gin.ntt.net (129.250.6.206)  162.218 ms  164.905 ms  165.397 ms
 12  ae-0.r22.londen03.uk.bb.gin.ntt.net (129.250.4.85)  168.716 ms  176.386 ms  173.211 ms
 13  ae-5.r24.nycmny01.us.bb.gin.ntt.net (129.250.2.18)  262.749 ms  305.414 ms  307.083 ms
 14  ae-1.r07.nycmny01.us.bb.gin.ntt.net (129.250.3.181)  307.158 ms  306.288 ms  307.215 ms
 15  ae-0.edgecast.nycmny01.us.bb.gin.ntt.net (129.250.196.174)  308.608 ms  302.816 ms  307.224 ms
 16  93.184.216.34 (93.184.216.34)  306.972 ms  243.375 ms  261.559 ms

12. Bootable drive

You can create a usb bootable drive from iso with single terminal command

$ dd if=/home/user/Downloads/ubuntu.iso of=/dev/sdb1 bs=512M; sync

This creates /dev/sdb1 (USB drive) into a debian bootable.

13. MySQL database

You can interact and manage mysql database directly from the the commandline, cant be easier than this. You dont need phpmyadmin.

mysql -u root -p
create database mydatabase;
create user 'myuser'@'localhost' identified by 'password';
grant all previleges on * . * to 'newuser'@'localhost';
Flush previleges

This will be helpful if you install any CMS and want to create a database in a hurry without any additional software.

14. Github

To download local copy the git project just copy the git url and clone it.

$ git clone https://github.com/example/project.git

You can also upload git files to your project easily with few commands in your command line. No need to download any client. Just clone the directory, edit the files or add and commit to master.

$ git clone https://github.com/example/example.git
$ cd example
$ git add .
$ git commit -m "Final commit"
$ git push origin master

You can find more information https://help.github.com/articles/adding-a-file-to-a-repository-from-the-command-line/

Thats it!

15. Mass Deletion

Removing all files and sub directories  is very easy with R (recursive)  and f (force) flag with the rm command

rm -Rf /home/temp *.tar.gz

Please use this command carefully as it will delete all files and subfolders recursively without any warning. You can undelete and recover all files in the partition using the extundelete utility for ext3/ext4 file systems

$ extundelete /dev/sda4 --restore-all

This will restore all files in partition.

16. Install Development tools

Before you start to compile any new software or packages, be sure to prepare your server with all the latest GNU c/c++ compiler and development libraries. An useful one liner, atleast works well in amazon linux.

yum groupinstall "Development Tools"

17. Command History

Very often you want to take a look at the command history. Just this one simple command will show you the terminal command history

$ history
    3   sed -f <(sed 's/.*/s|\\b&\\b||g/' stopwords.txt) rawfiltered.txt > data.txt
    4   sed -f -i  <(sed 's/.*/s|\\b&\\b||g/' stopwords.txt) rawfiltered.txt > data.txt
    5   sed -fi  <(sed 's/.*/s|\\b&\\b||g/' stopwords.txt) rawfiltered.txt > data.txt
    6   sed -i  <(sed 's/.*/s|\\b&\\b||g/' stopwords.txt) rawfiltered.txt > data.txt
    7  tr -s '[:blank:]' '\n' < rawfiltered.txt | fgrep -vwf stopwords.txt > data.txt 
    8  sed -e -f rawfiltered.txt stopwords.txt > data.txt
    9  sed -ef rawfiltered.txt stopwords.txt > data.txt

18. Dropbox

You can install dropbox from the linux command line and start the daemon

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -
 ~/.dropbox-dist/dropboxd

19. Screenshot server

You can take screenshots of a webpage using a headless Xvfb server running. Although you cant see, the script will capture the screenshot and will create a PNG locally

./screenshot.sh urls.txt

Screenshots using Firefox and Xvfb in Amazon EC2

20. Image Conversion

You can resize, convert images to various formats using imagemagick’s convert utility. Other features include blur, threshold, morphology and tonnes of things to do with this tool.

convert rose.jpg -resize 50% rose.png
convert dragon.gif    -resize 64x64  resize_dragon.gif

You can also convert image to pdf

convert -density 300 -trim example.pdf -quality 100 pdf.jpg

Convert bunch of images to PDF

 convert *.JPG  output.pdf

Refer to imagemagick for more examples http://www.imagemagick.org/script/index.php

21. Cron Jobs

You can schedule run certain scripts daily or weekly or on certain days using CRON. If you have high traffic server, you can schedule backups automatically when there during low traffic days such as sunday.

Edit the cron file and place the command

$crontab -e

The format is

a b c d e /home/backup.sh
  • a: Minute (0-59)
  • b: Hours (0-23)
  • c: Day (0-31)
  • d: Month (0-12)
  • e: Day of the week(0-7 starts Sunday as 0)
  • * denotes every day

This script will run daily only at 12.00 AM start of day

 0 24 * * * /scripts/shell.py 

This script runs every sunday early morning 5 PM

 0 5 * * 0 /scripts/backups.php 

22. Convert Doc to PDF

You can convert document files to pdf using this wv utility

sudo apt-get install wv tetex-extra ghostscript  
wvPDF test.doc test.pdf

or using cups pdf utility

sudo apt-get install cups-pdf
 oowriter -pt pdf your_word_file.doc

23. Youtube videos

You can download youtube videos in a commandline using youtube-dl utility

yum install youtube-dl
youtube-dl <url>

You can download just audio mp3 without the need for ffmpeg

youtube-dl --extract-audio --audio-format mp3 <video URL>

If you have ffmpeg installed, you can convert that flv (downloaded from youtube) to mp3

ffmpeg -i example.flv -acodec libmp3lame mytune.mp3

24. USB drive

Just plug your memory stick to your usb and copy files. This is what you have to do. First you have to mount the disk. To find out the device name (we assume it is /dev/sdb1)

fdisk -l
 mkdir /media/usb
 mount /dev/sdb1 /media/usb
 cp /home/*.tar.gz /media/usb

Formatting the usb drive is also easy

sudo mkfs.msdos /dev/sdb1

Thats it.

25. Hard drive Partitions

You can create partitions in your storage drive using the fdisk command. To view existing partitions, use this command

fdisk -l /dev/sda

To start partitioning the disk, use fdisk and will prompt for a command. n to add new partition, w to write. After partition is created you can format to ext4 file system.

fdisk /dev/sda

Command (m for help): m

Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

You can refer to http://tldp.org/HOWTO/Partition/fdisk_partitioning.html

26. Simple Firewall

You can setup simple IP tables firewall in command line. We open web traffic on port 80 (http), 443 (https), SSH (port 22) and close all other ports

iptables -F (to flush existing rules)
iptables -A INPUT -p tcp -m tcp --dport 25 -j ACCEPT 
iptables -A INPUT -p tcp -m tcp --dport 465 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

and see the rules

$ iptables -L

You can save rules to

Keep in mind that when you restart the server, these rules are gone. You can save Ip tables rule to /etc/sysconfig/iptables

# service iptables stop
 # service iptables start
 # service iptables restart

Warning dont lock yourself out!