CPU usage going through the roof

[doneritehosting]

Hey peter,

Well I actually just started tracking my cpu usage with your post and I found On one day it spiked above 5% .. As mentioned earlier in this post, I looked at my traffic stats and saw alot of IPs just hitting comments.post. I searched the ips on google and they were related to comment spam so I banned the ips in the control panel... cpu usage dropped to 2.5% which I thought still quite high for the amount of traffic I receive. I went to php config, disabled zend then enabled APC and memcache. My cpu usage has been below 1% every since..

[PeterAchutha]

Hi doneritehosting,

Thanks for your suggestions. I will see if I can find out if any IP addresses are hammering away at my site. Unfortunately, I never had my access logs and error logs turned on. I turned them on a few days ago.

I tried, hopefully I understood it correctly, what clivejo had tried. Yesterday or day before, I copied the original directory to a new directory and changed back the name of the directory.

Febrauary 2013 figures
Day CPU Time Execution Time CPU Usage Average memory Processes
1 2832.2s 37159.93s 8.20% 4.63 MB 2210
2 2211.9s 26269.64s 6.40% 4.59 MB 1526
3 2142.95s 25570.27s 6.20% 4.57 MB 1552
4 2917.88s 28262.85s 8.44% 4.60 MB 1529
5 3022.93s 36775.54s 8.75% 4.64 MB 2050
6 2130.23s 29099.25s 6.16% 4.60 MB 1696
7 3592.17s 35132.59s 10.39% 4.60 MB 1901

Day Number of visits Pages Hits Bandwidth
01 Feb 2013 796 3,114 5,209 123.96 MB
02 Feb 2013 789 2,537 4,421 117.15 MB
03 Feb 2013 757 2,370 4,565 128.92 MB
04 Feb 2013 750 2,413 4,258 116.64 MB
05 Feb 2013 898 3,254 5,454 138.63 MB
06 Feb 2013 898 3,114 5,366 149.56 MB
07 Feb 2013 776 2,851 4,567 112.38 MB

But my CPU usage figures are extremely high. Yesterday it hit 10.39%.


I checked back the database access figures and I got the shock of my life. This may be a different problem.

My WordPress site figures for January 2013:-
Day db Total Hourly average
20 170076 170076 7087
21 4,328,541 4,328,541 >180356
22 64685 64685 2695

Day CPU Time Execution Time CPU Usage Average memory Processes
20 957.16s 29716.70s 2.77% 4.60 MB 1572
21 789.12s 22656.81s 2.28% 4.61 MB 1308
22 1078.74s 30269.57s 3.12% 4.60 MB 1659

Day Number of visits Pages Hits Bandwidth
20 Jan 2013 672 2,932 5,711 172.86 MB
21 Jan 2013 514 2,179 4,368 130.89 MB
22 Jan 2013 734 3,033 6,052 184.14 MB



My e-commerce site is even worse (I added the commas)
Day db Total Hourly average
20 466856 483528 20147
21 28,143,987 28,144,597 >1172692
22 325454 325909 13580

Day CPU Time Execution Time CPU Usage Average memory Processes
20 1165.24s 24471.82s 3.37% 4.56 MB 1180
21 1019.75s 21968.05s 2.95% 4.62 MB 1001
22 1303.48s 28569.90s 3.77% 4.55 MB 1226

Day Number of visits Pages Hits Bandwidth
20 Jan 2013 459 2,437 20,908 215.35 MB
21 Jan 2013 428 1,857 18,093 201.92 MB
22 Jan 2013 588 3,017 28,571 309.67 MB


28,143,987 queries is like 1 every 3 milliseconds. I asked, but tech support will not feed back to the administrators these figures. They insist that there is nothing wrong with their figures.

Now I have installed a login lock for my WordPress site to see if that brings the figures down. Is it possible to make a db query every 3 milliseconds? If they are using some form of brute force hacking to login, my guess is that probably 30 PC's are used for the hack.

Thanks - Peter.

[vlasi47gr]

What donerite experienced was the same thing that I experienced. I too banned certain IP's that were hitting me hard and that solved my problem.

[clivejo]

Is it possible to make a db query every 3 milliseconds? If they are using some form of brute force hacking to login, my guess is that probably 30 PC's are used for the hack.

Thanks - Peter.

Its a bot, they are programmed to do this, by people with nothing better to do only cause grief.

Static pure html are easier to serve than a dynamic page which needs hundreds of db queries to build it. That's why we use caches to help speed up the site. I would now identify the bot/script IP's in your access logs and block the IP's. Another method is to use a WP plug-in that will recognise a bot hammering your login page and automatically block it via the .htaccess file. For example 2 wrong attempts and your IP gets blocked. Better still deny access to it totally except for requests from your own IP.

[doneritehosting]

clivejo is right... in all methods... I chose to research the ips same as vlasi and block them from the control panel... that way they are blocked before a page is served or a plugin is used inside wordpress but... its time consuming. The plugin is neat, set and forget and if i run into this problem again plugin it is

[PeterAchutha]

Thanks vlasi47gr, clivejo, doneritehosting

For your comments. I have plenty to tell. Sorry for the delay in responding as I had to wait the 24 hours to get a complete day's statistics.

I had installed the WordPress plugin Limit Login Attempts which blocks you from loging in for 20 minutes after 4 attempts. I might change this plugin. And below is the CPU usage and visitor counts:-

Day Number of visits Pages Hits Bandwidth
01 Feb 2013 796 3,114 5,209 123.96 MB
02 Feb 2013 789 2,537 4,421 117.15 MB
03 Feb 2013 757 2,370 4,565 128.92 MB
04 Feb 2013 750 2,413 4,258 116.64 MB
05 Feb 2013 898 3,254 5,454 138.63 MB
06 Feb 2013 898 3,114 5,366 149.56 MB
07 Feb 2013 776 2,851 4,567 112.38 MB
08 Feb 2013 740 2,667 4,483 111.09 MB
09 Feb 2013 784 3,656 6,046 155.28 MB

And here is the CPU usage for the same days:-

February 2013
Day CPU Time Execution Time CPU Usage Average memory Processes
1 2832.2s 37159.93s 8.20% 4.63 MB 2210
2 2211.9s 26269.64s 6.40% 4.59 MB 1526
3 2142.95s 25570.27s 6.20% 4.57 MB 1552
4 2917.88s 28262.85s 8.44% 4.60 MB 1529
5 3022.93s 36775.54s 8.75% 4.64 MB 2050
6 2130.23s 29099.25s 6.16% 4.60 MB 1696
7 3592.17s 35132.59s 10.39% 4.60 MB 1901
8 1742.9s 31059.24s 5.04% 4.58 MB 1610
9 1776.64s 29313.14s 5.14% 4.58 MB 1743


As you can see the CPU usage dropped to 5% from 8%-11% range so there is some success here but not good enough. Today I upgraded my W3 Total cache to the latest version and activate db caching. I was hoping to get some results in the next two days.

If you looked at the database queries :-

February 2013
>>

Day db Total Hourly average
1 83367 83367 3474
2 63005 63005 2625
3 53806 53806 2242
4 68315 68315 2846
5 69128 69128 2880
6 62358 62358 2598
7 155113 155113 6463
8 83184 83184 3466
9 92584 92584 3858
10 20042 20042 835 <-- reporting error - not divided by correct # of hours but by 24.

So far the database queries have not dropped that much and I am looking at caching the database. At the moment I cannot log into my WordPress control panel as I got the following error message (I can view my site no problem)

--- error message ----
Can’t select database

We were able to connect to the database server (which means your username and password is okay) but not able to select the ***X database.

Are you sure it exists?
Does the user ***X have permission to use the ***X database?
On some systems the name of your database is prefixed with your username, so it would be like username_***X. Could that be the problem?

If you don't know how to set up a database you should contact your host. If all else fails you may find help at the WordPress Support Forums.

---- end of error message ----

I will try again later. Google reported server errors in end December 2012 and Jan 11 2013.

While I am on the db stats you will notice that I marked the last line as reporting error because 835 queries per hour results in 20,040 queries per 24 hours. In actual fact the figure 20042 was for about a quarter day (6 hour block) so the hourly average should have been around 3340 queries per hour. You can check your own MySQL Stats and see if you see the same reporting error. Please do feed back to the Administrators about this reporting error - I can't as tech support won't allow me.

I don't think I have solved the bulk of the CPU usage problems but it looks like my problems are compounded with hacking. Last year I was seeing hackers from USA (NY & Florida), Germany, Moldova and China.

When I checked the error logs I saw this:-

[Sun Feb 10 03:28:51 2013] [error] [client 91.207.6.6] mod_fcgid: stderr: WordPress database error Lost connection to MySQL server during query for query SELECT option_value FROM wp_db_options WHERE option_name = 'limit_login_allowed_lockouts' LIMIT 1 made by require('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), do_action('plugins_loaded'), call_user_func_array, limit_login_setup, limit_login_setup_options, limit_login_get_option, get_option, W3_Db-&gt;query, W3_DbCache-&gt;query, W3_DbCallUnderlying-&gt;query, W3_Db-&gt;query, W3_DbProcessor-&gt;query,

This hacker,91.207.6.6, is from the Ukraine.

So I went back to my Awstats page and I could see a few more countries that had potential hackers.

Visitors domains/countries (Top 10) - Full list for 1st to 9th February 2013

Domains/Countries Pages Hits Bandwidth
United States us 7,271 14,489 498.31 MB
Unknown ip 6,663 9,342 161.11 MB
Malaysia my 2,522 4,963 112.18 MB
China cn 2,327 2,487 11.87 MB
Canada ca 2,202 3,644 96.60 MB
Ukraine ua 1,441 1,458 6.84 MB
Great Britain gb 1,050 2,503 88.07 MB
France fr 386 407 801.77 KB
Australia au 368 825 34.77 MB
Russian Fed ru 297 324 3.25 MB
Others 2569 5975 190.16 MB

From the bandwidth figures you can observe that visitors form China and Ukraine are just hackers. Since my webpages have many photos the Bandwidth / pages figure should be around 40k to 70k or higher. But China and Ukraine figures are too low to be webpage article visitors - they are going for the WordPress Log In page which is basically an empty page. From these figures you will notice that at least 14% of the page traffic is hacking traffic. I have always estimated that 25% of the internet traffic is hacking traffic.

On a side note, I do not know why Awstat has a bunch of Unknow IP's as I checked with an IP locator website and all those I tested could be located, ie, Malaysia, Ireland, ....

I think I have fixed these hackers in some sense. I changed all my passwords from 12 digit to 22 digit passwords. A 12 digit password using Numbers and Lower case alphabets would create a max combination of 95,428,956,661,682,200,000,000,000,000 and that would take 3,026,032,364,969,630,000 years to hack using brute force techniques every 3 millisecond. A 22 digit password using Symbols, Numbers, lower and upper case alphabets would result in 24,447,866,538,280,100,000,000,000,000,000,000,000 ,000,000,000,000,000,000,000,000,000,000,000,000,0 00,000,000,000,000,000,000,000,000,000,000,000,000 ,000 combinations which would take 775,236,762,375,699,000,000,000,000,000,000,000,00 0,000,000,000,000,000,000,000,000,000,000,000,000, 000,000,000,000,000,000,000,000,000,000 years to check all combinations. ... Let the hackers read this ...

Now I am waiting for the database error to disappear so that I can log back into my WordPress control panel.

Thanks - Peter

[PeterAchutha]

Hi Guys,

I found some quirks in the system and decided to get to the bottom of this once and for all. Like everyone has been advising me, I decided to 'delete' the website and start a fresh - to ensure no faulty or hacked scripts.

I didn't really delete anything just created a new folder and created a new database and made a clean installation of WordPress 3.3.1 then upgraded to 3.5.1. Only installed, for security reasons, plugins:-

1. Bottom of every post
2. Configure SMTP
3. Fast Secure Contact Form
4. Limit Login Attempts
5. SI CAPTCHA Anti-Spam
6. WP Security Scan

Exported all my pages through WP Export and imported them to the new site. (Even with PHP 5.3 I could never get PHPMyAdmin to import what it exports.)

The CPU usage shot up during the hours 12-18 to 40.42% and my guess is that yesterday I had repeated this procedure about 4 times - delete and recreate the entire site, as I was trying to find the best method to do this as sometimes the import of pages and post was not complete. I found some sub-directories in the WP /upload directory missing.

My original database was 65.37MB and the new database is 4.99MB. There is no caching at all to modify the results. I will monitor for the next 2 days and check the CPU usage and # of db records accessed.

Regards,
Peter

[PeterAchutha]

Hi guys,

I have narrowed it down to a few possibilities - for the high CPU usage. Please refer to the data below:-

E-commerce Site - Russian Software
« August 2012 »
Day CPU Time Execution Time CPU Usage Average memory Processes
14 312.77s 6837.33s 0.91% 4.51 MB 316
15 325.27s 6159.87s 0.94% 4.51 MB 266
16 448.98s 8014.96s 1.30% 4.52 MB 339
17 266.97s 5603.40s 0.77% 4.51 MB 234
18 316.45s 5980.18s 0.92% 4.51 MB 265
19 245.91s 5702.69s 0.71% 4.51 MB 256
20 387.22s 7093.74s 1.12% 4.51 MB 289
21 299.46s 6593.94s 0.87% 4.51 MB 294
22 227.62s 6183.49s 0.66% 4.51 MB 241
23 676.15s 13540.23s 1.96% 4.51 MB 547
24 1847.96s 42375.25s 5.35% 4.55 MB 1834 <- jump in CPU usage
25 2839.88s 94352.10s 8.22% 4.52 MB 3682
26 3152.3s 86292.15s 9.12% 4.51 MB 3248
27 1906.44s 71060.89s 5.52% 4.53 MB 2726
28 864.66s 27420.54s 2.50% 4.51 MB 1344
29 1219.57s 28630.46s 3.53% 4.50 MB 1175
30 1350.55s 25926.26s 3.91% 4.50 MB 1125
31 1272.41s 25410.42s 3.68% 4.51 MB 1099







WordPress Site
« August 2012 »
Day CPU Time Execution Time CPU Usage Average memory Processes
15 80.01s 2279.52s 0.23% 4.61 MB 139
16 90.01s 3657.40s 0.26% 4.60 MB 175
17 56.02s 2463.92s 0.16% 4.60 MB 134
18 60.39s 2594.99s 0.17% 4.60 MB 162
19 66.1s 3312.35s 0.19% 4.60 MB 185
20 57.09s 2851.86s 0.17% 4.60 MB 179
21 63.38s 2523.19s 0.18% 4.60 MB 156
22 47.81s 2484.61s 0.14% 4.60 MB 138
23 90.49s 4484.37s 0.26% 4.60 MB 250
24 493.43s 19179.17s 1.43% 4.61 MB 1047 <- jump in CPU usage
25 363.53s 16870.02s 1.05% 4.60 MB 943
26 314.71s 15687.40s 0.91% 4.60 MB 940
27 375.22s 21288.20s 1.09% 4.60 MB 1185
28 329.16s 15447.01s 0.95% 4.60 MB 878
29 383.43s 17604.46s 1.11% 4.60 MB 1013
30 423.78s 17355.11s 1.23% 4.59 MB 1082
31 725.06s 23851.61s 2.10% 4.64 MB 1458






WordPress site without plugins and with and without W3 Total Cache + APC
« February 2013 »
Day CPU Time Execution Time CPU Usage Average memory Processes
15 738.52s 18720.31s 2.14% 4.77 MB 1185 <-- New Database & WP 'no' plugins
16 1018.42s 22170.20s 2.95% 4.64 MB 1129
17 750.88s 17386.16s 2.17% 4.77 MB 1025
18 751.16s 17214.87s 2.17% 4.53 MB 1023
19 778.39s 18916.46s 2.25% 4.65 MB 1084
20 2609.05s 25229.26s 7.55% 4.65 MB 1245 <-- edited a post
21 902.42s 20732.50s 2.61% 4.64 MB 1145
22 834.49s 20265.76s 2.41% 4.64 MB 1106
23 852.46s 23645.83s 2.47% 4.65 MB 1220
24 780.51s 20767.71s 2.26% 4.63 MB 1211
25 794.93s 18940.66s 2.30% 4.66 MB 1039 <- added W3 Total Cache
26 2063.42s 35364.16s 5.97% 4.69 MB 2091
27 1316.39s 23579.58s 3.81% 4.64 MB 1562
28 0s 0s 0.00% 0 MB 0


1. You will notice that both sites, running very different software, had a jump in CPU usage on 24th August 2012. This will tend to show that there is something common to both. I, personally, don't think both sites were hacked on the same day. Also the Russians are pretty confident that their e-commerce cart cannot be hacked and both sites are located on different server hosting accounts. But there is a WP installation as a subdomain in the e-commerce site that is not 'public' knowledge.

2. I had done a security check for malware, with http://sitecheck.sucuri.net/scanner/ on all my sites and nothing suspicious was detected.

3. By the 15th February 2013 I had a completely clean installation of WP in a completely new directory. I had removed all non-security plugins but the CPU usage never dropped below 2%.

4. On the 20th of February I edited one post and the CPU usage shot up to 7%.

5. On the 25th / 26th February 2013 I added back all other plugins and installed W3 Total Cache (recommended by WordPress) and in addition activated APC (Advance PHP Cache) which is used by W3 Total Cache for caching opcodes, some posts, ... but database is cached with disk caching. The CPU usage figures did not drop.

6. Going through all the historical data I found that the jump in CPU usage is caused by a jump in usage of two 'tasks' / processes on the 24th August 2012. They are
a. PHP CGI
b. Perl

I have requested from tech support some way of checking which files / post were calling these two tasks but they said they keep no records.


From the above I can only suspect:-

1. My server CPU was down graded to a slower CPU without my knowledge (?)
2. The administrators had changed PHP CGI and Perly modules
3. Apache server operating system has been hacked
4. All caches have been disabled by the Administration

Do you guys have any thoughts on this. I am still running more experiments (removed some plugins) to trace the origin of the high CPU usage. If you have any suggestions that I could try, I will definitely consider them (within reason).


Thanks,
Peter

[clivejo]

Have you been keeping the access logs for the site too?

[PeterAchutha]

Hi clivejo,

Thanks for your reply. No I have not been keep access logs. I did download them once or twice (did the same with the error logs).

- Peter

[PeterAchutha]

Sorry guys, I had to go dark for a while. I had to be invisible as I was not sure if someone was monitoring my sites and emails and blocking all my work.

When I upgraded my plan to include cron jobs I noticed that my site CPU usage dropped below 2% for a day after that it came back to high figures. It looked like somebody was deliberately undoing what I had done every day.

Clivejo, when you asked me whether I had the access and error logs, I had not as I didn't not know what a cron job was and how to implement it. I found out later that basically it was a background task. That's what we called them back in the days of paper tape and punch cards ... I must be ancient in computer terms.

Once I figured that out I got a cron job working to copy out the access and error logs every midnight. And that was when I found out that there were 24 to 29 cron jobs by WordPress every hour!

By the way, I think that this is a security isssue as when I set up my plans I did not want any cron jobs - didn't think they were important. So my access and error logs did not show any cron jobs in the reports/logs. Once I activated cron jobs in my plan the cron jobs showed up in the access and error logs. Looks like cron jobs can run without being reported. The reason I had not allowed any cron jobs in my original plan was so that no one could hack the site and install a cron job but it looks like they just don't get reported. This I think is a security issue which can result in misleading analysis of the situation.


I have had my emails monitored and any real estate deals I confirmed through emails would go sour. So I began to suspect an inside job trying to destroy my work. It has happened before. I am not allowed to apply for patents in Malaysia and I am not allowed to open bank accounts. All this occurred after I began helping our prime ministers - quietly in the back ground. I think there are many higher ups out to destroy my income so that I have to come and ask them for help or funding. That is the way they can take control of any technology I invent. Sounds paranoid and ridiculous right? Go check it out. They take 7 years to respond to my patent application after circulating it. I have had emails and phones calls monitored and intercepted. That is the reason I began to suspect that these higher ups and infiltrated RSP and are trying to destroy my work here. Don't be surprised as tech support have been very obnoxious to me lately. In fact I am think of looking for alternative hosting services.

It is because of the higher ups I have located all my sites outside of Malaysia but the only way they can now intercept my work is through my ISP. What is worse, my ISP cut my line on March 12th without explanation. This has occurred to me before. I am back on line with a new ISP.

Like, doneright said, I found scrappers, eg easau.com attacking my sites. I blocked easau.com and the next day my cpu usage jumped to 10% for a six hour period. I have since blocked all China IP addresses but I think they are coming through proxies. When I blocked 1400 ranges of China IP addresses almost all the cron jobs disappeared.

I have a strong suspicion that there is an inside job at work. Not the tech support but some script. I am still working on it. There was so much research to be done that I got lost in the research. I am gradually coming back to the real problem at hand - the CPU usage.


I am really upset with tech support and I suspect they have no liking for us paying clients. Now I found that my site has been blocked. I cannot upload new files to update my software development and testing.

Best Regards,
Peter Achutha

[clivejo]

Once I figured that out I got a cron job working to copy out the access and error logs every midnight. And that was when I found out that there were 24 to 29 cron jobs by WordPress every hour!

By the way, I think that this is a security isssue as when I set up my plans I did not want any cron jobs - didn't think they were important. So my access and error logs did not show any cron jobs in the reports/logs. Once I activated cron jobs in my plan the cron jobs showed up in the access and error logs. Looks like cron jobs can run without being reported. The reason I had not allowed any cron jobs in my original plan was so that no one could hack the site and install a cron job but it looks like they just don't get reported. This I think is a security issue which can result in misleading analysis of the situation.

I think your are getting a little confused here. The CRON jobs I was referring to are UNIX CRON jobs i.e.

*/10 * * * * /usr/bin/somedirectory/somecommand

We use these to do automated tasked, like taking a backup of the access logs every night. The cron jobs you are seeing in your access logs are part of Word press internal job system. Every-time a user visits a Word press site the internal system can do jobs, ie email you updates, or activate time related content. You will see the server call itself ( wp-cron.php ) This is part of how Word press works and nothing to do with your host!

[clivejo]

Regarding your government spying on you, this is not something RSP can prevent. Emails are transferred in plain text and can be easily intercepted by anyone between you and RSP servers. Email is NOT a secure medium and should never be used as such.

We all have abusive bots and botnets hacking at our sites. This is part of hosting a website on the internet. Even this forum is subject to spam bots and hacking attempts, but thats not RSP's fault.

The biggest attack in history was recently directed at Spamhaus, a project which aims to stop spammers being able to spam!! Just by trying to stop them, you automatically become a target in their war.

[yav0r]

On the other hand Spamhouse are a bunch of blackmailing twats much like the USA BBB a.k.a. better business bureau that will hold you network ransom until you pay them not to block you.

The idea of moderating spam is fine by me, but they are plain Nazis if you ask me.

Anyway they deserved that DDoS long time!

Just my 2 cents and completely off topic

[clivejo]

* clivejo bites his tongue and tries to stay on topic

Blocking IP's is a no-win game. They will just use another one of the millions out there.

OK Ill try and explain this. Every-time someone visits your site your Word Press installation calls its own internal cron system (wp-cron.php). If you get lots of visitors, which I think is what’s happening in your case, this actually generates huge amounts of processing for no good reason! You can disable this 'feature' by adding the following line to your wp-config.php file

define('DISABLE_WP_CRON', 'true');

You should add it after the following line

define('DB_USER', 'your_user');

This should stop all those cron calls and bring down your CPU usage. However, you may find that certain tasks Wordpress usually does automatically are now not working. In this case you would create a CRON job on the server (ie your hosting account) to execute the wp-cron.php script every few hours.

This internal Cron job feature allows Word Press to be installed on cheaper web-hosts who dont allow users to have a UNIX based CRON job. It does usually do a good job, but in high traffic sites becomes a resource hog.