CPU usage going through the roof

[PeterAchutha]

Thanks wjleong,

1. I didn't want to sign up for the Akismet API key that is why I am not using Akisment.

4. I might remove Google Analytics Dashboard as I am not really 'using' it, will do some more checking. It has these tiny charts that shows how many visitors I had for each article when I use the WP control panel to view all posts. That was very helpful.

The others, well I have not checked out Jetpack and may need to see how it works for others and a review. As for the 3 caches W3 Total cache, PHP Cache and Widget Cache, as I have put in a lot of work analyzing their performance for speed of page download to the visitors PC, I won't replace them with Jetpack just yet. Most of my articles have 7 to 14 photos 448x336 pixels per article so caching them on the server side did help a lot.

As for 1-click Retweet/Share/Like that was the only plugin, at that time when I was looking, that had a nice row of buttons for the top or bottom of the page. Many have floating buttons that keep irritating me. I only wanted buttons at the bottom of the page so that after a visitor has read the article they could decide if they wanted to 'like' or 'share'.

But that does not explain why on January 11th 2012 when there was a flood in Chicago the CPU usage went to 15%. They had blocked a whole range of IP addresses, not just mine, on and off. The servers could not be accessed most of the day, which means that the CPU's were idling, which means the CPU usage should have dropped to almost ZERO %.

I used to have high CPU usage on another site - different hosting account, but the third party software developers are changing their code to reduce the CPU usage.

Some data:-
Awstats
Day Number of visits Pages Hits Bandwidth
1-Nov-12 978 3,818 7,208 183.80 MB
2-Nov-12 972 3,253 6,493 172.91 MB
3-Nov-12 854 3,170 6,331 167.64 MB
4-Nov-12 964 3,960 7,695 213.11 MB
5-Nov-12 956 3,778 7,777 158.86 MB
6-Nov-12 1,022 5,138 11,288 169.77 MB
7-Nov-12 985 4,849 7,182 140.03 MB
8-Nov-12 947 3,773 6,587 158.01 MB
9-Nov-12 1,038 3,852 6,431 143.28 MB
1-Feb-13 796 3,114 5,209 123.96 MB
2-Feb-13 789 2,537 4,421 117.15 MB

CPU Data
Day CPU Time (s) Execution Time (s) CPU Usage Average memory Processes
1 1,168.29 40,964.88 3.38% 4.58 MB 2359
2 1,075.33 37,470.76 3.11% 4.60 MB 2191
3 971.84 35,145.59 2.81% 4.60 MB 1986
4 1,084.36 39,425.82 3.14% 4.60 MB 2247
5 1,341.44 44,232.04 3.88% 4.63 MB 2616
6 693.72 24,212.21 2.01% 4.61 MB 1443
7 1,874.62 47,205.26 5.42% 4.61 MB 2688
8 1,239.65 40,259.80 3.59% 4.60 MB 2301
9 1,208.61 39,445.58 3.50% 4.60 MB 2297
1 2,832.20 37,159.93 8.20% 4.63 MB 2210
2 2,211.90 26,269.64 6.40% 4.59 MB 1526

Analysis of above data:-
Day Test Execution Test CPU Test Processes Processes/CPU sec
1 2.02
2 37,692.54 1074.965059 38,047.50 2.04
3 33,856.22 971.600418 33,964.82 2.04
4 39,273.01 1085.970676 39,764.42 2.07
5 48,717.26 1339.909809 45,900.29 1.95
6 22,914.02 694.9212371 24,398.64 2.08
7 65,288.65 1870.62806 45,102.16 1.43
8 31,266.95 1241.676347 40,408.97 1.86
9 39,250.50 1208.572423 40,189.81 1.90
1 92,415.36 2831.600571 37,951.56 0.78
2 29,002.87 2210.497561 25,658.85 0.69

Test execution is calculating how many seconds for the day based upon previous day execution time and CPU % and todays CPU %. The figures are way off. This means that Execution time is not related to CPU usage.

Test CPU shows projecting todays CPU time (s) based upon yesterdays CPU seconds and %. The projections are very close ie there is a correlation.

Test Processes is projecting todays Execution time based on yesterdays execution time and number of processes. The figures are very close which means that execution time is related to the number of processes running.

Porcesses/CPU second is showing how many processes run per CPU second used. This clearly shows that in November 2012 the server was able to run 2 processes per CPU second but by February 2013 the figure has dropped to 0.78 processes per CPU second. The only the only thing I have done to WORDPRESS was to upgrade it over the months since March 2012. All other plugins had been installed before August 2012. In February or end Jan I had upgraded WP to ver 3.5.1 and SEOPressor to v5 and I implemented browser caching in my .htaccess file

## EXPIRES CACHING ##
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/jpg "access 1 day"
ExpiresByType image/jpeg "access 1 day"
ExpiresByType image/gif "access 1 day"
ExpiresByType image/png "access 1 day"
ExpiresByType text/css "access 1 day"
ExpiresByType application/pdf "access 1 days"
ExpiresByType text/x-javascript "access 1 day"
ExpiresDefault "access 1 day"
</IfModule>
## EXPIRES CACHING ##

Browse caching is very usefull and powerful technique if your images are repeated across all your pages. It saves a lot of back and forth communications between server and client and reduce content download so pages appear almost instantly on the client side. But one drawback of browser caching is that if your pages are being modified often the visitors will not see the updates until the cache is cleared. That is why I set browser caching to 1 day. And you can see the improvement as in February 2013 I put the browser caching code in .htaccess file. I did want to try W3 Total cache browser caching as I am trying to learn about browser caching. But you can see the performance difference to my server as the bandwidth has dropped from about 140 - 180 to 118 MB. So clearly less data is being processed and sent out by the server so my CPU usage should drop some more.

Sorry, I am late for church - have to go.

[wjleong]

Peter,

As I told you, try off the cache.

If your traffic isn't huge enough, then cache will only kill you, but not save you.

Somemore Jatpack cache your image to wordpress.com, it perform far better than you cache on server side, as it first will generate load from your server.

Jetpack itself has stat, so you can check everything there, so why use so many plugins to increase the cpu load ?

I don't understood why you hate akismet so much.

share/retweet whatever, Jetpack has it, and exactly same as what you're using it right now!

Advice I gave, try or not depends on you.

Never try never know, please stop complaining high cpu load anymore if you keep all of your current plugins.

Or, seek help at wordpress.org forum.

Good luck.

[PeterAchutha]

Hi wjleong,

You missed the whole point of my concern. The plugins I have are not the issue - almost all of them have been running since March 2012 - no problems. Caches are always good especially if there is sufficient hits to a unmodified page. Other than that its dormant. The CPU (in single processors systems) still has to do the work and process and serve the pages out.

You seem to want to push Jetpack but you do not give any performance reviews for Jetpack or comparisons with other WordPress caches. But the most important point you missed was that when the systems are idle the CPU usage should be low, independent of what plugins are installed.

My concerns were not about my tested out plugins but the reported CPU usage as we are charged for high CPU usage.

If you look closely at the Jan 11 2013 figures you will see that the bandwidth is only 40.17 MB which means the server was shut down most of the day - considering There Was A Flood In Chicago, were my server is located. So a CPU usage of 15.33% is an error in how the reports were calculated. And at the end of the day, if this error is genuine, then all CPU usage figures can be suspect. But you seem to be avoiding the issue.

If someone else more knowledgeable can help me sort this out please do.

[doneritehosting]

I see your point

Traffic

10 Jan 2013 547 3,042 6,005 115.78 MB
11 Jan 2013 184 915 1,568 40.17 MB
12 Jan 2013 529 2,570 4,996 134.82 MB
13 Jan 2013 664 3,002 6,072 176.30 MB


CPU

10 722.97s 20343.72s 2.09% 4.60 MB 1106
11 5298.38s 181411.20s 15.33% 4.60 MB 9454
12 551.8s 20474.61s 1.60% 4.59 MB 1134
13 679.46s 24343.42s 1.97% 4.59 MB 1321

As your server was unreachable due to the flood could it be possible that you are using plugins that pull data from other sources and those plugins were hammering the server trying to get the data that was expected but couldn't? Those plugins would endless loop until they could receive the data expected right?

Its just a theory lol..

[wjleong]

I see your point

Traffic

10 Jan 2013 547 3,042 6,005 115.78 MB
11 Jan 2013 184 915 1,568 40.17 MB
12 Jan 2013 529 2,570 4,996 134.82 MB
13 Jan 2013 664 3,002 6,072 176.30 MB


CPU

10 722.97s 20343.72s 2.09% 4.60 MB 1106
11 5298.38s 181411.20s 15.33% 4.60 MB 9454
12 551.8s 20474.61s 1.60% 4.59 MB 1134
13 679.46s 24343.42s 1.97% 4.59 MB 1321

As your server was unreachable due to the flood could it be possible that you are using plugins that pull data from other sources and those plugins were hammering the server trying to get the data that was expected but couldn't? Those plugins would endless loop until they could receive the data expected right?

Its just a theory lol..

no no no, you did not see the point, he confidence that nothing is related to the plugins. so forget about the plugins.

[wjleong]

Hi wjleong,

You missed the whole point of my concern. The plugins I have are not the issue - almost all of them have been running since March 2012 - no problems. Caches are always good especially if there is sufficient hits to a unmodified page. Other than that its dormant. The CPU (in single processors systems) still has to do the work and process and serve the pages out.

You seem to want to push Jetpack but you do not give any performance reviews for Jetpack or comparisons with other WordPress caches. But the most important point you missed was that when the systems are idle the CPU usage should be low, independent of what plugins are installed.

My concerns were not about my tested out plugins but the reported CPU usage as we are charged for high CPU usage.

If you look closely at the Jan 11 2013 figures you will see that the bandwidth is only 40.17 MB which means the server was shut down most of the day - considering There Was A Flood In Chicago, were my server is located. So a CPU usage of 15.33% is an error in how the reports were calculated. And at the end of the day, if this error is genuine, then all CPU usage figures can be suspect. But you seem to be avoiding the issue.

If someone else more knowledgeable can help me sort this out please do.

I did not push jetpack.

Load can be came from many places as well as imcompatible with the plugins to latest wordpress version.

That's why I always said try to deactivate the plugins and observe.

Well, I'll stop comment in this thread, as I'm not knowledgeable enough to help you.

Good luck to you.

[PeterAchutha]

Thanks doneritehosting for your reply.

I don't think I have any plugins that pull data from other sources.

Feedjit and Flag Counter (both not plugins and both are free versions) do pull data from their own servers but most of the time both flag counter and Feedjit don't count accurately - just approx, because they 'abandon' broken connections and nothing gets displayed. Feedjit is javascript so does not run from the server.

Flag Counter code is

<a href="http://s09.flagcounter.com/more/nkB"><img src="http://s09.flagcounter.com/count/nkB/bg=FFFFFF/ ... /pageviews=1/flags=1/" alt="Free counters!" border="0" height="1000" width="190" ></a>

some part of src="... has been deleted to fir here.


As for the plugins, I have never requested, approved, installed, used any API keys. I have for some reason or other never used API keys. I don't have a Google API key.

As far as I know SEOPressor only runs in the WP control panel when drafting out new articles.

The plugin 1-click Retweet/Share/Like is executed in an iframe to show up all the Google+ and Facebook buttons.

<iframe style="height:25px !important; border:0px .... src="http://www.linksalpha.com/social?blog=The+Bread+Diaries&link=http%3A%2F%2Fba chutha.com%2Fhow-to-make ... &referer=http%3A%2F%2Fbachutha.com%2F">
</iframe>

a lot of the code deleted out to fit here. Can I assume that this is not run from the server but from the client PC?

That is it, the limit of my knowledge. Is there anything else I should check?

Thanks,
Peter

[clivejo]

But that does not explain why on January 11th 2012 when there was a flood in Chicago the CPU usage went to 15%. They had blocked a whole range of IP addresses, not just mine, on and off. The servers could not be accessed most of the day, which means that the CPU's were idling, which means the CPU usage should have dropped to almost ZERO %.

Not necessarily true. The CPU can be doing other things, not just serving out pages. It could be stuck on an Input/Output (I/O) loop, maybe trying to sort out cached files after an attack. The Word Press application is great, but its very complex. It has a lot of 'built in' features which if malfunction could easily generate CPU load doing other tasks. Hackers know the vulnerabilities in the code and by exploiting these they can bring a server to its knees just by requesting a single URL !! That CPU time might be being use to spew out spam emails, or being used to flood another site.

The fact is that you have nowhere near the traffic to be generating that much load on the CPU. To be honest, if I were you I would backup your data and delete everything in the www folder, start again with a new, up-to-date install.

[PeterAchutha]

Thanks clivejo for your comments,

Personally, I don't think my WP site was hacked as the tech support guys said there was a flood in Chicago and the were disabling a whole range of IP addresses. I can sort of agree with this as both site hosted on different hosting accounts http://bachutha.com and http://www.dpawhs.com went down almost together during the same period.

Day Number of visits Pages Hits Bandwidth
11 Jan 2013 1 5 8 105.12 KB


My other site http://www.drpetersnews.com (also on a separate hosting account) went down but not as severely.

Day Number of visits Pages Hits Bandwidth
08 Jan 2013 654 2,662 26,830 288.74 MB
09 Jan 2013 512 2,540 22,881 263.75 MB
10 Jan 2013 550 2,523 20,650 250.06 MB
11 Jan 2013 179 952 7,420 98.03 MB
12 Jan 2013 502 2,200 19,623 233.84 MB
13 Jan 2013 446 2,241 19,878 227.76 MB
14 Jan 2013 584 2,952 25,665 279.40 MB
15 Jan 2013 610 2,791 25,256 283.56 MB

Day CPU Time Execution Time CPU Usage Average memory Processes
8 751.56s 19020.09s 2.17% 4.51 MB 810
9 579.5s 16354.89s 1.68% 4.51 MB 686
10 627.28s 17978.68s 1.82% 4.51 MB 753
11 1690.72s 50032.49s 4.89% 4.48 MB 2262
12 503.85s 16911.17s 1.46% 4.50 MB 722
13 659.14s 18666.79s 1.91% 4.50 MB 816
14 920.23s 23643.91s 2.66% 4.54 MB 1036
15 944.57s 20118.19s 2.73% 4.51 MB 779

As you can observe the CPU figures are quite high for the 11th Jan 2013 even though figures for 10th and 12th are extremely low. This is runs an e-commerce shopping cart hence it appears that the high CPU figures is not WordPress related.

Thanks for your suggestions, is there anything else I could check?

Thanks,
Peter

[doneritehosting]

I'm at a loss then.. like I said it was a theory. I had to ban alot of comment spambot IPs because they were sending my cpu usage over plan allowance.. You have 2 constants in that senerio 1- the server was up and running 2 - No network access in or out. If incoming requests from spambots can drive cpu usage through the roof then outgoing requests from plugins can too especially if one or two are looping because it cannot reach the server its trying to contact.. Your cpu usage dropped back to 1.6 - 1.9% cpu usage after network service was restored on the next 2 days of usage.. are you maintaining those figures in a round about way?

Jetpack is a good plugin but also it could be a example of a plugin that could cause what I am suggesting.. You have to login to use the functions so I assume it polls wordpress occasionally to keep you logged in. Several of the social features in jetpack you have to create accounts with the social sites then login through jetpack to use them..

I'm just guessin but it does fit the situation... The only real way to know would be to chat with one of the server techs about what happened..

As an afterthought enable outgoing connections (if you don't have it enabled) and see what works and what does not work..

[clivejo]

I'm not saying it was hacked, but something was going on that day!

A flood is an attack, it might not have been directed at your domain/account but its still a planned and executed attack by a hacker/hacking team, with some sort of motive.

Your first post asked if it was a reporting error. In my honest opinion, I dont think it is. As I explained, all the work done in regards your account, lets call this a CPU task, can be running across multiple physical CPU cores simultaneously. The figure your seeing is the total of how much time all the CPU's have been involved processing/working on that task.

On shared hosting we share resources, this is great as it keeps the costs down. Unfortunately, we also share the headaches of what other people do with their accounts. ie a spammer gets our shared IP blacklisted and we all suffer because of it.

If your concerned about the actual usage of CPU maybe you should get a dedicated or virtual private server. This way you will have a guaranteed and fixed CPU and can monitor its usage.

[doneritehosting]

That is definately the solution there clivejo, good one too.. but how can his site be hacked or ddos attack when there was no network access to the server? I'm curious myself. To be on the safe side I would scan my puter then change my username and passwords, then monitor the site pretty close for a couple of weeks to see if cpu usage spiked like that again... if it did I would be getting a vps for sure then.

Its a weird situation... Maybe its just a coincidence, There was a network outage and maybe the server had a serious burp at the same time..

[clivejo]

There would have been network access to the servers. The log says there was 40.17 MB transferred on the day in question.

A flood or Distributed Denial of Service (DDOS) is like driving in a snow storm. The wipers can clear the snow up to a certain point. Once it gets too much it overwhelms the wipers and you cant see out!

In the case of a web-server, it tries to keep up with processing the requests but eventually ties itself in knots trying, with the weakest link failing. The server is working as hard as it can to keep up but genuine requests get lost in the flood. To a visitor it looks like your web-server isn’t responding and has gone offline.

These type of attacks dont actually transfer much or any data! They exploit weaknesses in the TCP/IP protocol, for example the SYN flood attack. This prevents a server from servicing other users. It is accomplished by not sending the final acknowledgement to the server's SYN-ACK response (Synchronize-acknowledge) in the handshaking sequence, which causes the server to keep signalling until it eventually times out. The source address from the client is faked and probably doesn’t even exist!. SYN flood attacks can either overload the server or cause it to crash.

Read more about it here - http://en.wikipedia.org/wiki/SYN_flood

[doneritehosting]

Thanks for the info and the link clive, reading up on it now among other things. lol. Yea I saw the 40mb of transfer that day but I assumed it was before or after the server IP was blocked as stated earlier. Like I said its a weird circumstance

[PeterAchutha]

Hi doneritehosting, clivejo,

Thanks for your messages. I thought about them last night and then, this morning, I went to do some digging.

I know, clivejo, that you recommended that I do a clean installation of WordPress, and I have recommended the same to others, at http://wordpress.org/support/, who have had their WP sites infected. This is because it is sometimes much easier to erase everything and restart clean, especially when it is a small site with few visitors. It saves a lot of debugging and tracing and you will never know if you had cleaned out an infected site properly if you try disinfecting a site.

In this case I didn’t want to do that as I have quite a number of visitors to the site so I tried Sucuri SiteCheck. I checked for viruses on all 5 of my sites with http://sitecheck.sucuri.net/scanner/ and they came out clean.

Then I went back to historical records.

I went back and looked at the figures at http://www.dpawhs.com

Day Number of visits Pages Hits Bandwidth
09 Jan 2013 35 276 291 3.35 MB
10 Jan 2013 17 105 143 1.51 MB
11 Jan 2013 1 5 8 105.12 KB
12 Jan 2013 4 4 9 165.30 KB
13 Jan 2013 12 31 92 775.87 KB
14 Jan 2013 13 31 55 1.10 MB

Day CPU Time Execution Time CPU Usage Average memory Processes
9 7.54s 3760.59s 0.02% 4.48 MB 272
10 3.69s 1634.04s 0.01% 4.61 MB 154
11 0.57s 130.74s 0.00% 4.33 MB 26
12 0.48s 91.07s 0.00% 3.82 MB 19
13 1.33s 358.53s 0.00% 4.72 MB 78
14 0.5s 321.86s 0.00% 4.48 MB 19


These figures are sensible compared to my e-commerce site and my WP site. The figures show that the CPU usage figures actually dropped during the Chicago Flood. Oh, by the way, I too thought it was ‘water’ flood and not a ‘virus’ flood and checked for floods in Chicago on the internet, as I am in Malaysia. It’s true there was a ‘water’ flood in Chicago about the same time.

And it puzzled me that the other two sites had higher CPU usages on the 11th January 2013. The only thing that differentiates this site, dpawhs.com, from the others, drpetersnews.com and bachutha.com, is that this site, dpawhs.com, does not use a database.

Then I remembered that on 27August 2012 I received the first of a few warning emails “Your hosting account drpetersnews has been warned because of high CPU usage - it has reached the CPU limit allowed by your hosting plan. Your account has used: 8.22 % CPU average for the last 1 days. Note that your account will be automatically blocked if you use more CPU load than your plan is allowed.”

I contacted tech support about this as I had ‘abandoned’ both my blog sites for 6 weeks – had not written anything and had not even looked at the sites.

Day CPU Time Execution Time CPU Usage Average memory Processes
22 227.62s 6183.49s 0.66% 4.51 MB 241
23 676.15s 13540.23s 1.96% 4.51 MB 547
24 1847.96s 42375.25s 5.35% 4.55 MB 1834
25 2839.88s 94352.10s 8.22% 4.52 MB 3682
26 3152.3s 86292.15s 9.12% 4.51 MB 3248
27 1906.44s 71060.89s 5.52% 4.53 MB 2726
28 864.66s 27420.54s 2.50% 4.51 MB 1344
29 1219.57s 28630.46s 3.53% 4.50 MB 1175

If you looked at the visitor counts

Day Number of visits Pages Hits Bandwidth
21 Aug 2012 577 2,958 28,785 337.63 MB
22 Aug 2012 600 2,980 31,076 339.50 MB
23 Aug 2012 637 3,433 39,804 427.11 MB
24 Aug 2012 645 3,651 37,190 405.39 MB
25 Aug 2012 613 2,825 32,268 346.48 MB
26 Aug 2012 542 2,601 27,787 306.64 MB
27 Aug 2012 714 2,589 20,428 232.73 MB
28 Aug 2012 604 2,251 21,749 244.13 MB
29 Aug 2012 605 2,842 32,247 380.28 MB

The bandwidth does not relate to the CPU usage.

There was very little change in the visitor counts to show up as high CPU usage. I was also having database server errors which was explained by tech support as “Regarding the MySQL issues we are experiencing for the last few days, our administrators are aware of them and are working on an upgrade of the MySQL server. If you are still having any troubles loading the site now, you will have to put the site back online, so that we can check what may be causing the Error 500 page.” (Support Ticket 1251800). This is just a small portion of the conversation I had with tech support but it shows they were upgrading the MySQL servers after which all CPU usage figures went up.

Considering that when no database is used the CPU usage figures are realistic and when a database is used the CPU usage figure shoot through the roof, after August 2012. I would suspect that there is an error in the reporting of the CPU usage figures.

Why do I suspect it is a reporting error? I have found at least two other errors. The first being the Account Usage: Storage figures on the Hepsia Control Panel was not correct and I had reported this and they fixed it within a few days. The second was that the Awstats figures would disappear – set to zero, after a week or two every month. They had fixed this too.

Please let me know if others have seen a jump in CPU usage after August 2012 for sites with and without databases. If you guys can confirm that you are seeing the same situation that will be helpful.