So I was reading this article in slashdot and was playing with the server. I pretty much have done everything they have on that article except for using eaccelerator or apc.
So I decided to give eaccelerator a try, it seems to be easy to install and found a lot of stuff online about it. It wasn't on the default ubuntu repositories so I had to manually install it.
That one is the one I followed to install it on my ubuntu dapper box. http://2bits.com/articles/installing-eaccelerator-0-9-5-1-on-ubuntu-feisty-7-04.html (to get phpize install php-dev from the deb sources)
Seems to work great and I am loving it! Hope you guys see a difference on the server performance now. I observed that galleries when cached load up in about 0.01-0.14 seconds.