Server :: WordPress Blogging Server Running Slow For What Reason?

I have a wordpress blogging server up and running and i've also got nagios monitoring the speed of webpage download etc.The thing is a couple of weeks ago nagios alerted me that the blog was returning pages really slow loading, when i went to the blog homepage for me also it was very slow. After about 30mins of http connections some finally loading and some not nagios stopped reporting issues, but thats not the end of the story, the graphing of speed i've got set-up on nagios shows quite clearly that ever since that big slow down the pages take avg of extra 2-3 seconds to load. However nothing drastic has changed and the datasize of the page hasn't really changed at all (also monitored).

During that weird period I carried out checks on the server itself like top, free -m, netstat (looking for maybe DOS attack number of connections), looked at mysql see if that was running slow and what processes it was running, checked on number of http processes see if they had ramped up, checked on php and web server errors see if they had increased some what as well. None of these things turned up anything noticeable to be causing such slow blog response.Now its still that average amount high and i'm lost at why this could be? Its niggling at me that something may have got in but i've taken several security steps to try and lock down the wordpress install etc.