Seeing a site down or terrible slowness, is a webmasters nightmare. I have often freaked out in such situations because with high traffic site, you could lose lots of clients, and money, if your site is not properly accessible. Taking very good care of websites on weekly basis is much recommended.
Very often, when you press the panic button, even experienced server administrators would lose it and it is very important to keep calm and you should always keep a handbook, clearly explaining what to analyze and how to fix. I never used such handbooks, which is why i have decided to write this article, clearly mentioning what to look for and what to do in diagnosing the server issue. The issues could be high server load, php crashes, security attacks, mysql slowing down.
1. Analyze the Logs
First things first. The logs hold critical information what happened with your server , what caused a failure. It shows you the way to fix your problem.
If you are running cpanel, be sure to analyze these logs. You might want to look at apache errors, server messages and other log files
/etc/httpd/logs/error_log (apache log)
/var/log/messages
/var/log/secure
/etc/httpd/logs/suexec_log
/etc/httpd/logs/suphp_log
/usr/local/cpanel/logs/error_log (cpanel log)
/var/log/named.log (bind dns server)
/var/log/mysqld.log (if you have enabled mysql log)
/var/log/faillog
2. Analyze Server Load, High CPU Usage and Memory
If your server is reeling under heavy load, things could dramatically change and your visitors will get timed out errors or site could be massively slow. Look for the problem. Everything is so much dependent on mysql and if your scripts poorly written or mysql not optimized, the cpu usage and memory consumption would be very high.
Run the Top
Run the top and look for server load. You should try to keep server load always below 1.0 and anything about 5.0 is ok at peak times and hitting above 10.0 frequently, definitely need server upgrade. Keep in mind that cpanel updates or performing backups in the background, will cause a spike in server load.
Daily Process Log
You must analyze the daily process log which can be found in WHM of cpanel. This will really help you narrowing your problem specific to your domain. You can pickout which script/process is consuming cpu or uses more memory.
Background Killer & Process Manager
You may want to use the other tools in WHM like background killer and process manager.
3. Analyze MySQL server
Over 50% of the problems are caused by mysql. If you exprerience slow loading / high cpu usuage by mysql, its high time you should optimize mysql for slow queries. Its very easy to do that. If you have mysql slow queries enabled writing to a log file, look into it. In most cases, there would a log file like this /var/log/mysql.log. Any poorly written sql queries can consume large cpu and memory resources, slowing the server down on massive scale.
You may want to check into these files:
/var/log/mysqld.log (mysql log file, if available)
/etc/my.cnf (mysql configuration file)
Tips to Speed up MysQL
1. Create Indexes to your database, as it will speed up GROUP by and DELETE operations. If you dont create indexes, with your primary key, delete operations over 1000s of records will be massively slow.
2. Optimize and repair tables, which have overheads. This can be done from phpmyadmin.
3. Optimize mysql settings using mysqltuner. There is a huge difference in site performance with optimized mysql server and non-optimized mysql server. Its very easy to optimize mysql. All you have to use is mysqltuner perl script. It will suggest you what lines to add in my.cnf (mysql configuration file). here is how you do it.
wget http://mysqltuner.pl/mysqltuner.pl
perl mysqltuner.pl
4. Security Checks
You dont want your server vulnerable to attackers. check the log files /var/log/messages and /var/log/secure for any brute attacks or failed attempts. Too much of it would also slow down the server. You might want to run CSF firewal, block all necessary unwanted ports and deny IPs doing brute attacks. To configure csf under cpanel is very easy. see this tutorial
Be sure to run the vulnerability checker tools available in WHM panel.
Detecting DoS attacks
Sometimes you never if you are server is hit by Dos attacks. If that happens, your server might be inaccessible for long periods of time. Here is an easy way to find out whether you are attacked.
Run this command: It will show you how many ip’s connect to your server. If there are large no of ips connected to server its a clear indication of dos attack. of course you can block minor attacks with a firewall.
netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n
netstat -n -p|grep SYN_REC | wc -l (check for SYN flooding, high numbers is an attack)
If you have carefully looked and analyzed the above, it will give you a good indication of the source of the problem and you can fix it. Be sure to restart apache, and mysql after you have made any changes.
