Ubuntu :: Get Unresponsive Mouse Clicking/scrolling Ever?

I dont know why this is happening to me. Maybe it's because I have so many things running, but its mainly just browsers and stuff that hardly require cpu usage. Sometimes (and its getting worse now) when i try to highlight the text on a web page, it wont work, the mouse pointer will be the arrow instead of the "I" like symbol which signifies the possibility for text highlighting. Then after a little while it will become the "I" symbol so I can highlight the text. It will switch on and off like that and make it really hard for me to do things.

Also, sometimes I cannot scroll down, it wont work when i hold the side bar thing and scroll it down, but after a few seconds it may work. This happens on my browser as well as amarok so far. I hope someone has experienced something like this because its getting frustrating.

UPDATE: Happens on KMess too. I cant type in the convo box sometimes. So I have to minimize and maximize it in order for it to work. Also if I have a browser up in the background while a Kmess convo box is infront, I cant click to the browser when its right there, even though I can click on it in the taskbar, if you know what i mean.