Linux commandline is absolutely amazing loved by almost all system administrators. Here you will discover powerful and impossible things you can do, just using a command line terminal.
1. Searching files
Searching for certain files are inevitable for admins. find /home -type f -name "*.conf"
and to use locate, which is the easiest
sudo updatedb locate mickey.cpp
Search file contents using grep
$ cat /etc/ hosts | grep -i "localhost"
To list files that are greater than 500mb, run this. V
find /home/ -type f -size +500M -exec ls -lh {} \;2. Zip and Unzip
You can easily compress and uncompress .tar.gz and .bz2. Please note that .bz2 takes long time as it is high compression using “j” flag
tar cvfj example.tar.bz2 /home/example (create .bz2 using j flag)
tar cvzf example.tar.gz /home/images (create .tar.gz using z flag)
tar -xzf example.tar.gz -C /home/justin (extract .tar.gz using x and z flag)
tar -xjf example.bz2 -C /home/melissa (extract .bz2 using x and j flag)3. Cleaning up files
You can clean up file contents using sed, tr and awk commands in a command line (faster than writing a script).
This tr command will clean up all alpha, numeric and special chars from input file
tr -c "A-Za-z0-9@-._ \n" " " < input.txt > output.txtThis will translate uppercase to lowercase
tr '[:upper:]' '[:lower:]' < rawfiltered.txt > data.txtSed command to substitute a word or string in a file.
sed 's/unix/linux/g' myfile.txt
and this will remove blank lines in a file.
sed '/^$/d' input.txt > output.txt
Also you can use sort to sort contents in a file, which also can be merged with sed
sort data.txt > output.txt
You can find more sed examples here: http://www.folkstalk.com/2012/01/sed-command-in-unix-examples.html
4. Security
Check connections from same IPs on Port 80 (http) for multiple ddos attacks
netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1
netstat -apn | grep ":80"and also check for http processes
netstat -n | grep :80 |wc -l
5. Hard drive errors
You can look for problems with hard drive using smartctl
$ smartctl -q errorsonly -H -l selftest -l error /dev/sda
Warning: ATA error count 17 inconsistent with error log pointer 1
ATA Error Count: 17 (device log contains only the most recent five errors)
Error 17 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)
Error 16 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)
Error 15 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)
Error 14 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)
Error 13 occurred at disk power-on lifetime: 47355 hours (1973 days + 3 hours)You can also use iostat to monitor I/O performance for local hard drive and remote disks
# iostat
Linux 2.6.18-308.1.1.el5 (server1.example.com) 02/24/2016
avg-cpu: %user %nice %system %iowait %steal %idle
2.72 0.05 1.11 0.22 0.00 95.89Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 10.71 390.59 208.68 301731339 161206205
sda1 0.00 0.27 0.00 210670 4
sda2 4.81 112.90 94.88 87211079 73297681
sda3 2.23 67.60 37.70 52217183 29124792
sda4 0.00 0.00 0.00 134 0
sda5 1.02 29.13 42.69 22504042 32978328
sda6 0.04 5.43 0.00 4195402 1200
sda7 2.45 175.25 29.83 135379849 23041280
sda8 0.16 0.02 3.58 12282 2762920
6. Cpanel backup and restore
You can pack and restore cpanel account files in a fly using these handy commands in command terminal
/scripts/pkgacct username
/scripts/restoracct username7. Remote File Transfer
You can transfer files remotely using SCP from one host to another
scp /home/file.tar.gz user@host.example.com:21Feb16/
8. Server Loads
Check users load, http processes, CPU usage and server load using these commands.
$ uptime
19:07 up 29 mins, 2 users, load averages: 0.48 0.59 0.55$ w
18:27 up 2:08, 2 users, load averages: 1.25 1.53 1.44
USER TTY FROM LOGIN@ IDLE WHAT
prabhubalakrishnan console - 16:20 2:06 -
prabhubalakrishnan s000 - 17:21 - w$ iostat
disk0 cpu load average
KB/t tps MB/s us sy id 1m 5m 15m
51.45 12 0.60 4 1 94 1.06 1.42 1.40Check for running processes to see if any process takes too much memory or cpu load.
$ ps aux | more
9. Free Disk Space
First, check disk space using this command
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 9.7G 533M 8.7G 6% /
/dev/sda8 996M 34M 911M 4% /tmp
/dev/sda7 194G 4.2G 180G 3% /home
/dev/sda3 9.7G 2.1G 7.2G 23% /var
/dev/sda2 9.7G 6.0G 3.2G 66% /usr
/dev/sda1 99M 17M 77M 18% /boot
tmpfs 1014M 0 1014M 0% /dev/shmthen search for files greater than 100 mb using this command
find /var/www/ -size +100000000c | more10. DNS Settings
You can check dns settings for your server for correct IP address (A and AAA ipv6).
dig example.com
host -t A example.com (ipv4)
host -t AAAA example.com (ipv6)11. Tracing Network Route
Traceroute is an extremely useful tool to fix network path problems. It shows the transmission path across networks for different continents.
$ traceroute example.com
traceroute to example.com (93.184.216.34), 64 hops max, 52 byte packets
1 192.168.0.1 (192.168.0.1) 1.267 ms 1.601 ms 1.009 ms
2 10.23.240.1 (10.23.240.1) 7.628 ms 15.619 ms 11.351 ms
3 113.193.5.169 (113.193.5.169) 9.953 ms 19.261 ms 8.973 ms
4 59.163.111.81.static-chennai.vsnl.net.in (59.163.111.81) 9.017 ms 13.421 ms 9.451 ms
5 172.25.81.134 (172.25.81.134) 56.100 ms 59.540 ms
172.31.29.245 (172.31.29.245) 52.476 ms
6 ix-0-100.tcore1.mlv-mumbai.as6453.net (180.87.38.5) 69.474 ms 67.030 ms 47.591 ms
7 if-9-5.tcore1.wyn-marseille.as6453.net (80.231.217.17) 157.941 ms 153.891 ms 159.319 ms
8 if-8-1600.tcore1.pye-paris.as6453.net (80.231.217.6) 164.418 ms 170.447 ms 159.127 ms
9 if-2-2.tcore1.pvu-paris.as6453.net (80.231.154.17) 183.550 ms 158.083 ms 166.330 ms
10 ae-7.r04.parsfr01.fr.bb.gin.ntt.net (129.250.8.1) 165.362 ms 171.770 ms 167.023 ms
11 ae-8.r23.londen03.uk.bb.gin.ntt.net (129.250.6.206) 162.218 ms 164.905 ms 165.397 ms
12 ae-0.r22.londen03.uk.bb.gin.ntt.net (129.250.4.85) 168.716 ms 176.386 ms 173.211 ms
13 ae-5.r24.nycmny01.us.bb.gin.ntt.net (129.250.2.18) 262.749 ms 305.414 ms 307.083 ms
14 ae-1.r07.nycmny01.us.bb.gin.ntt.net (129.250.3.181) 307.158 ms 306.288 ms 307.215 ms
15 ae-0.edgecast.nycmny01.us.bb.gin.ntt.net (129.250.196.174) 308.608 ms 302.816 ms 307.224 ms
16 93.184.216.34 (93.184.216.34) 306.972 ms 243.375 ms 261.559 ms12. Bootable drive
You can create a usb bootable drive from iso with single terminal command
$ dd if=/home/user/Downloads/ubuntu.iso of=/dev/sdb1 bs=512M; syncThis creates /dev/sdb1 (USB drive) into a debian bootable.
13. MySQL database
You can interact and manage mysql database directly from the the commandline, cant be easier than this. You dont need phpmyadmin.
mysql -u root -p
create database mydatabase;
create user 'myuser'@'localhost' identified by 'password';
grant all previleges on * . * to 'newuser'@'localhost';
Flush previleges
This will be helpful if you install any CMS and want to create a database in a hurry without any additional software.
14. Github
To download local copy the git project just copy the git url and clone it.
$ git clone https://github.com/example/project.git
You can also upload git files to your project easily with few commands in your command line. No need to download any client. Just clone the directory, edit the files or add and commit to master.
$ git clone https://github.com/example/example.git
$ cd example
$ git add .
$ git commit -m "Final commit"
$ git push origin masterYou can find more information https://help.github.com/articles/adding-a-file-to-a-repository-from-the-command-line/
Thats it!
15. Mass Deletion
Removing all files and sub directories is very easy with R (recursive) and f (force) flag with the rm command
rm -Rf /home/temp *.tar.gz
Please use this command carefully as it will delete all files and subfolders recursively without any warning. You can undelete and recover all files in the partition using the extundelete utility for ext3/ext4 file systems
$ extundelete /dev/sda4 --restore-all
This will restore all files in partition.
16. Install Development tools
Before you start to compile any new software or packages, be sure to prepare your server with all the latest GNU c/c++ compiler and development libraries. An useful one liner, atleast works well in amazon linux.
yum groupinstall "Development Tools"
17. Command History
Very often you want to take a look at the command history. Just this one simple command will show you the terminal command history
$ history
18. Dropbox
You can install dropbox from the linux command line and start the daemon
cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -
~/.dropbox-dist/dropboxd19. Screenshot server
You can take screenshots of a webpage using a headless Xvfb server running. Although you cant see, the script will capture the screenshot and will create a PNG locally
./screenshot.sh urls.txt
Screenshots using Firefox and Xvfb in Amazon EC2
20. Image Conversion
You can resize, convert images to various formats using imagemagick’s convert utility. Other features include blur, threshold, morphology and tonnes of things to do with this tool.
convert rose.jpg -resize 50% rose.png
convert dragon.gif -resize 64x64 resize_dragon.gifYou can also convert image to pdf
convert -density 300 -trim example.pdf -quality 100 pdf.jpgConvert bunch of images to PDF
convert *.JPG output.pdf
Refer to imagemagick for more examples http://www.imagemagick.org/script/index.php
21. Cron Jobs
You can schedule run certain scripts daily or weekly or on certain days using CRON. If you have high traffic server, you can schedule backups automatically when there during low traffic days such as sunday.
Edit the cron file and place the command
$crontab -e
The format is
a b c d e /home/backup.sh
- a: Minute (0-59)
- b: Hours (0-23)
- c: Day (0-31)
- d: Month (0-12)
- e: Day of the week(0-7 starts Sunday as 0)
- * denotes every day
This script will run daily only at 12.00 AM start of day
0 24 * * * /scripts/shell.py This script runs every sunday early morning 5 PM
0 5 * * 0 /scripts/backups.php 22. Convert Doc to PDF
You can convert document files to pdf using this wv utility
sudo apt-get install wv tetex-extra ghostscript
wvPDF test.doc test.pdfor using cups pdf utility
sudo apt-get install cups-pdf oowriter -pt pdf your_word_file.doc
23. Youtube videos
You can download youtube videos in a commandline using youtube-dl utility
yum install youtube-dl youtube-dl <url>
You can download just audio mp3 without the need for ffmpeg
youtube-dl --extract-audio --audio-format mp3 <video URL>
If you have ffmpeg installed, you can convert that flv (downloaded from youtube) to mp3
ffmpeg -i example.flv -acodec libmp3lame mytune.mp3
24. USB drive
Just plug your memory stick to your usb and copy files. This is what you have to do. First you have to mount the disk. To find out the device name (we assume it is /dev/sdb1)
fdisk -l mkdir /media/usb mount /dev/sdb1 /media/usb cp /home/*.tar.gz /media/usb
Formatting the usb drive is also easy
sudo mkfs.msdos /dev/sdb1
Thats it.
25. Hard drive Partitions
You can create partitions in your storage drive using the fdisk command. To view existing partitions, use this command
fdisk -l /dev/sda
To start partitioning the disk, use fdisk and will prompt for a command. n to add new partition, w to write. After partition is created you can format to ext4 file system.
fdisk /dev/sdaCommand (m for help): m
Command action a toggle a bootable flag b edit bsd disklabel c toggle the dos compatibility flag d delete a partition l list known partition types m print this menu n add a new partition o create a new empty DOS partition table p print the partition table q quit without saving changes s create a new empty Sun disklabel t change a partition's system id u change display/entry units v verify the partition table w write table to disk and exit x extra functionality (experts only)
You can refer to http://tldp.org/HOWTO/Partition/fdisk_partitioning.html
26. Simple Firewall
You can setup simple IP tables firewall in command line. We open web traffic on port 80 (http), 443 (https), SSH (port 22) and close all other ports
iptables -F (to flush existing rules)
iptables -A INPUT -p tcp -m tcp --dport 25 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 465 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPTand see the rules
$ iptables -L
You can save rules to
Keep in mind that when you restart the server, these rules are gone. You can save Ip tables rule to /etc/sysconfig/iptables
# service iptables stop
# service iptables start
# service iptables restartWarning dont lock yourself out!
