10+ Things to do when your server is down!

Is your website down? Are you in panic? Seeing a server down or terrible slowness, is a webmasters nightmare. I have often freaked out in such situations because with high traffic site, you could lose lots of clients, and money, if your site is  slow or site down.

serverissues

Very often, when you press the panic button, even experienced server administrators would lose it and it is very important to keep calm and you should always keep a handbook, clearly explaining what to analyze and how to fix. I never used such handbooks, which is why i have decided to write this article, clearly mentioning what to look for and what to do in diagnosing the server issue. The issues could be high server load, php crashes, security attacks, mysql slowing down.

1. Local ISP Network Problems

Over 80% of site issues  caused by problems with local ISP. Congestion or maintenance tasks done by your ISP provider can cause intermittent network problems which can cause your website (hosted elsewhere) in other parts of the world may be unreachable. Try reaching your site through proxy such as (www.hidemyass.com) or try some other ISP provider to access your site from your computer.  Try using an alternate browser or clear your cache.

If you are absolutely sure that your server is having problems, proceed with the following.

2. DNS Issues

Try to ping your server ip address and your domain and see if it is reachable. If your IP is reachable, but the domain name, the problem could be well with the DNS server. Check your bind logs, in case your run your own dns server.

ping example.com
ping 8.8.4.4

3. Firewall

Are you sure that your current IP address is not blocked by the firewall installed by you. If thats the case, you will be totally locked out and you will not be able to access cpanel, ssh, http, mail and http. Sometimes the ISPs assign dynamic IPs and it could be possible that you have blocked one of these IPs using cpanel htaccess.

4. Analyze the Logs

First things first. The logs hold critical information what happened with your server , what caused a failure. It shows you the way to fix your problem.

If you are running cpanel, be sure to analyze these logs. You might want to look at apache errors, server messages and other log files

/etc/httpd/logs/error_log (apache log)
/var/log/messages
/var/log/secure
/etc/httpd/logs/suexec_log
/etc/httpd/logs/suphp_log
/usr/local/cpanel/logs/error_log (cpanel log)
/var/log/named.log (bind dns server)
/var/log/mysqld.log (if you have enabled mysql log)
/var/log/faillog

5. High Server Load

Check for high server load and it is likely that your server might be getting lots of battering from huge traffic.

type w in your command interface to see a quick view of server load.

w

If your server is reeling under heavy load, things could dramatically change and your visitors will get timed out errors or site could be massively slow.  Look for the problem.

6. Check Connections to Port 80

Look for any suspicious processes or IPs connected to port 80 (http). Use this command and see if you could spot something.

netstat -apn | grep “:80”

7. Analyze Top Process

Run the top and look for server load. You should try to keep server load always below 1.0 and anything about 5.0 is ok at peak times and hitting above 10.0 frequently, definitely need server upgrade.  Keep in mind that cpanel updates or performing backups in the background, will cause a spike in server load.

> top

8. Check CPU & Memory Usage

You must analyze the daily process log which can be found in WHM of cpanel. This will really help you narrowing your problem  specific to your domain. You can pickout which script/process is consuming cpu or uses more memory. Use daily process log under your WHM interface.

9. Check Processes

Look for any suspicious or high cpu consuming processes running in your server and kill it. You may want to use WHM like background killer and process manager.

10. Slow MySQL Queries

Over 50% of the problems are caused by mysql. If you experience slow loading / high cpu usage by mysql, its high time you should optimize mysql for slow queries. Its very easy to do that. If you have mysql slow queries enabled writing to a log file, look into it. In most cases, there would a log file like this /var/log/mysql.log. Any poorly written sql queries can consume large cpu and memory resources, slowing the server down on massive scale.

Everything is so much dependent on mysql and if your scripts poorly written or mysql not optimized, the cpu usage and memory consumption would be very high.

You may want to check into these files:

/var/log/mysqld.log (mysql log file, if available)
/etc/my.cnf (mysql configuration file)

Tips to Speed up MySQL

1. Create Indexes to your database, as it will speed up GROUP by and DELETE operations. If you dont create indexes, with your primary key, delete operations over 1000s of records will be massively slow.

2. Optimize and repair tables, which have overheads. This can be done from phpmyadmin.

3. Optimize mysql settings using mysqltuner. There is a huge difference in site performance with optimized mysql server and non-optimized mysql server. Its very easy to optimize mysql. All you have to use is mysqltuner perl script. It will suggest you what lines to add in my.cnf (mysql configuration file). here is how you do it.

wget http://mysqltuner.pl/mysqltuner.pl
perl mysqltuner.pl

11. PHP crashes

If php is causing the crash on the server, chances are the memory is set too low.  Image manipulation libraries  consume memory for image processing.

Edit the php.ini file

memory_limit = 128M

or

you can set this inside a php file.

<?php 
ini_set('memory_limit', '256M');
?>

12. Attacks on Server

You dont want your server vulnerable to attackers. check the log files

/var/log/messages
/var/log/secure

for any brute attacks or failed attempts.

You must run CSF firewall, block all necessary unwanted ports and deny IPs doing  brute attacks. To configure csf under cpanel is very easy. see this tutorial

(i) Check for brute force attacks on port 21 (ftp), port 22 (ssh) and attacks can be thwarted if you change the standard ports to other ports

(ii) Check for recaptcha attacks in your /etc/httpd/logs/error_log  file and block all those suspicious IPs using your firewall.

(iii) Check for SYN flooding

netstat -n -p|grep SYN_REC | wc -l

Any numbers which are very high above 200 are likely to be attack.

(iv) Dos attacks

if  your server is hit by Dos attacks, your server might be inaccessible for long periods of time.

Run this command: It will show you how many ip’s connect to your server. If there are large no of ips connected (over 500) to server its a clear indication of dos attack. of course you can block minor attacks with a firewall.

netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n

Be sure to restart apache, and mysql after you have made any changes. Use mod security to defend against minor dos attacks, but if the attacks never subside, you might have to contact your hosting provider for the hardware firewall.

13. Apache Requests per Second

Go to Cpanel > Apache stats, to see how many connections are server per hour or minute. You might also find some interesting information on apache cpu usage, connections, GET and POST requests per domain and working php scripts.

apstats

14. Setup Website Monitoring

24×7 Monitoring your site is always a good idea, just in case your site goes down again. Use freewebmonitoring.com to track your site.

15. Check Server Hardware

RAM, or disk problems can cause your site crash frequently. Be sure to ask your host to test all the server hardware for any hardware problems.

16. Htaccess Issues

If your site throws up “Internal server error”, surely your .htaccess file is the main source of problem. An incorrect directive  there or it could be wrongly written mod_rewrite rule. Try to comment out the incorrect directives and see if your sites comes back online.

Last but not the least.

17. Hire somebody to help

If you really need somebody or a professional to help you to fix your server issues, hire Platinumservermanagement.com which costs something like $9.95/mon