My Trixbox seems to be hacked..

The preventive measures you specified are good and mostly keep hackers aways but sometimes a clever hacker can get them bypassed and enter the system. A group of VoIP engineers working at a US based voip service provider, Axvoice, were engaged in same kind of situation. Then after research a utility called fail2ban came to rescue. It's work method is simple, it inspects your log files and blocks the specific ip which is originating such registration request for the time period you specify. It has easy to understand and change regex style for filtering logs like shown below:

failregex = NOTICE.* .*: Registration from '.*' failed for '' - Wrong password
NOTICE.* .*: Registration from '.*' failed for '' - No matching peer found
NOTICE.* .*: Registration from '.*' failed for '' - Username/auth name mismatch
NOTICE.* .*: Registration from '.*' failed for '' - Device does not match ACL
NOTICE.* .*: Registration from '.*' failed for '' - Peer is not supposed to register
NOTICE.* failed to authenticate as '.*'$
NOTICE.* .*: No registration for peer '.*' \(from \)
NOTICE.* .*: Host failed MD5 authentication for '.*' (.*)
NOTICE.* .*: Failed to authenticate user .*@.*

If you're going to expose 5060 to external traffic, you need to be selective about it rather than allowing "everyone".

You probably have something like this:

iptables -A INPUT -i eth0 -p udp -m udp --dport 5060 -j ACCEPT

What you want to do is add the specific IP addresses:

iptables -A INPUT -i eth0 -p udp -m udp --dport 5060 -s xxx.xxx.xxx.xxx -j ACCEPT

For each remote sip phone, you would add one of those lines with xxx.xxx.xxx.xxx being replaced with the static IP address of the remote user.

If they are getting a DHCP address from their ISP, you can google up the IP ranges associated with that ISP and open them as a range, expressing it as a subnet/mask (xxx.xxx.xxx.xxx/mm).

Of course doing so exposes your Trixbox to that entire ISP, which increases your exposure.

Another, and better option, is to force a company policy of requiring VPN hardware at each site, such as the D-Link DIR-330, whereas your remote users plug that into their firewall/router from their ISP, then connect into that. You'll need a VPN server on your side, but then your users can connect security and safely to the PRIVATE side of your LAN, access the Trixbox, and thusly, you do not have to publically expose your Trixbox ever again.