Software :: VoIP Setup With Asterisk Appliance IP04?

I am encountering a strange problem on my VOIP setup Basically, I have a asterisk appliance IP04. I have setup all the extensions and everything. I use a Linksys PAP2T as an ATA remotely. Now, my problem is the ATA sometimes is okay can call SIP and PSTN but sometimes I just can't hear anything. I thought it was my ISP blocking the VOIP packets but I have tried both the SIP softphone and IAX2 softphone on my PC. For IAX2, it works perfectly however in the SIP, I can hear the other end but they cannot hear me.

These are the ports I have opened on my router
1.) UDP 5060 - SIP Port
2.) UDP 10000 - 20000 - RTP Port
3.) UDP 4569 - IAX2

Do I need to open both TCP/UDP for these ports or UDP should be enough? These are the test cases:

1.) Using my WiFi Connection and a analog phone connected to ATA --> Sometimes working sometimes not and sometimes you can call SIP but the other end cannot hear you
2.) Using IAX2 in WiFi connection --> This one works perfectly
3.) Using a mobile phone connected to WiFi Network --> The same...but you can call and go out on PSTN but the other end cannot hear you
4.) Using a mobile phone connected via 3G --> Works perfectly but as expected it is quite slow and voice quality is awful

I want to use SIP rather than IAX2 because it is widely used and since my ATA doesn't support IAX2. Are there other ports I need to open or configure?