Regarding Trixbox Trojan

(insert picture of you beating a dead horse)

Theres 4 or 5 fanboys making smart comments on any thread addressing this issue. It's pretty sad how little they know about security.. or maybe it's the level of maturity?

dm

it can be turned off, the concern of this issue has been more than voiced already, a solution has been made, if you dont support this project, and the only reason you are here is to promote the negativity of it, you are best saying nothing at all.

after you contribute something useful, then you can say you have a vested interest in this, and you are genuinely concerned.

seriously if it promotes the development of this software which you have paid nothing for, I could care less. I highly doubt that this legitimate company is interested in all your SIP passwords, the information is encrypted, and I have met these developers, they are very smart, they know how to make sure something is secure.

you people have taken this little smudge, and just about blotted out the good of this product.

Couldn't have said it better myself.

its heartwarming that you people have publicized this on the security forums, because now it really could be a potential hole, so far I am still looking for someone who has been harmed by it, and what "crucial" info they lost.

i hope you know how to edit crontab, otherwise youre stuck with that gaping security hole until you stop using trixbox.

die horse die

Here is the dilemma as I see it, I agree that if a sentiment that I share has already been made known and maybe a response to fix the issue has been made to my satisfaction, there is no reason to 'me too' it with a response (Let the horse die). On the other hand, this forum has shown me, Fonality will not take serious a single post for it's greater value unless it is backed by a horde of me too'ers. Too often nothing is done because only one person has publicly complained or asked about something. So where does the balance lie??

Except that in this case, many people have already complained and Fonality has:

1. Acknowledged the error
2. Apologized
3. Announced that they would be taking corrective action in accordance with community demands / wishes.

How those three things translate into the OP's statement that "kerryg doesn't understand this..." baffles me. Believe me, I am not one of Kerry's fanboys, but you guys have made your point and it is valid -- now, let's see what comes of their corrections.

How much do you want me to talk about it. I am FULLY aware, PAINFULLY aware of exactly what every single issue with this is. I am choosing not to respond to every single post. I have stated our position and we are working on a top priority to fix it to everyone's satisfaction. We can even use the tool itself to update itself to remove the security ramifications but we have at LEAST two full days of work to fix it, post the fix for community review, and then implement it. The people who are continually flaming us is not going to make us move any faster as we are going as fast as we can. And I do mean everyone from myself and Andrew, to Chris Lyman to every senior management person so there is complete knowledge within the entire company and nobody here can claim they don't know.

No, it isn't cool and we realize that more than anyone, therefor any new code that has ANY privacy or security implications will be made available for community review prior to us pushing it live.

"any new code that has ANY privacy or security implications will be made available for community review"

Since no one here actually knows what arbitrary code has already been running prior to being informed, perhaps making the past code(s), at least, available for community review would allay some concerns as well. Given the groupthink displayed though, I wonder if community review will be enough.

kerry, are there any plans to publish a privacy policy for trixbox CE regarding the information that is collected, how it is used and what users can do in regards to their data? This would probably be a good idea for all of the private data that Fonality collects, including details needed to register for the forums, etc. I'm sure you've covered this in your many posts this weekend, but as you've found with your blog, keeping everything in one place helps people find it.

I hate to sound like I am repeating myself over and over without saying anything new but we are working on a statement and changes to the proceses and code. I am going to continue to make any responses I have short until I have formalized all of the information.

I would suggest that since these systems may be handling interstate or foreign communications that some form of legal review might be advisable before any such script is run on a client's machine, particularly without prior advice.

"It is available for the most part, just go look in cron,and track the file down and read it."

I haven't installed trixbox for any client for which I have privacy concerns since they instituted the registration process and ignored my requests for a privacy policy. Again, I would suggest Fonality make this prior code, not resulting cron logs, available so anyone who is concerned might review them. This is an open source project after all.

"and I couldnt complain if someone GAVE me a new front door for FREE and kept the key, if I was a mistrusting paranoid blah blah blah that knew for sure this great service was surely for a much greater evil, I have the know how to change the lock out for a new one."

Legally, yes you could. And if you installed those front-or back doors for clients, so might they. Some of my clients pay me to be paranoid.

I've been trying to be helpful in this, while at the same time critical of your kind of groupthink that led to this problem in the first place. There could be much more to this than a simple PR disaster.

There have been multiple references to arbitrary code downloaded from a URL nightly and then executed. If so, this code would not exist on a client machine in any lasting manner, or might be replaced. If this isn't the case, then dispelling this now would be a great idea.

Then a historical record of the actual past code in addition to any new code could and should be made available for review. At this point, this is only a suggestion that might help satisfy anyone with concerns over what has already happened prior to today and without their knowledge.

heres the deal, whatever code that they have, (if they have) pushed out has not caused any damage to anyones system, until now no hackers have had much exposure to the fact that it even exists, it has been said that they have not yet implemented their statistics collecting, trust them as you will, either way its all been pretty harmless up until now, and the info on your phone system is only as secure as you make it, once its plugged into the internet, that can be considered a potential security liability (so says my Tadiran competition), but look at the info there, what is someone going to do - find out who you have been calling - is anyone going to die because of that - which probably isnt the case anyways.

I am not saying it was ok, or that I agree with anything that Fonality has done, they made a mistake, have admitted to it, and are taking steps to make sure its fixed asap, people are acting like they have lost critical top secret files that are crucial to their companies existance, they have not drained anyones bank account via ssh, some people here have brought on World War 9 over a potential problem, not a real problem. China has nuclear missiles - should we bomb them today? This is not progress, its destruction.

Let them do their job, and audit as need when they are done, but dont be nazis over it, very counter productive, and some people need actual work done to fix real problems that are affecting systems.

All hyperbole aside from the peanut gallery, all I am suggesting is a record of any past code, which you and I both know may no longer be available on anyone's system. I would think this would be simple enough, and if done pro-actively it might allay many concerns.

There is no change to the heartbeat code that has existed for over a year, only the new code was added in the latest builds.

well said Kodak, except that 50% of it is from the same people which is just forum spam now.

There is always an alternative. They should be able to fix the pending issues. Lets wait and see. Be happy

Kerry, does the Fonality PBX code do this same sort of BOT like phoning home? I am trying to understand why anyone would design a system like this from a clean sheet of paper when better tools were available, and the only one I could think of was that it was a port of what an existing system did (e.g.) fonality.

Does anyone here know if Fonality's proprietary PBX does this too?

Thx
mike

Dobbs, my comment wasnt directed towards you, there are a couple of other members that are rampantly flaming, I apologize if it seemed directed towards you, your concerns definitely are valid.

I dont represent Kerry or trixbox on this, just annoyed at some peoples persistence to burn the house they are staying in, after the fact has been stated that all their requests will be satisfied.

I have said when we would issue the first statement, we did so. I said when we would issue the next statement, I am writing it now, I said when we would make any changes, and thats tomorrow. I am on track with what I said two days ago.

I am going to finish what I am writing up before doing anything else.

It is a common attitude protecting in all manners the server like those working in a n tier environment despite of applications running inside the box... If you are doing business with Trixbox you should already know (and more particularly in open source software) that it's "a must" to set basic firewall rules only for those services like trunks or others... Imagine a sourceforge employer modifying the isos that kerryg is uploading in order to get access to your shinning phone systems someday! As simple as that! This is not an excuse but more an advise from someone who used to work in small and large deployment environments which in most cases the rule is: the server must always be protected from the outsiders, insiders and the software itself. Actually, in my current company there are thousands of servers running all sort of software/applications and we actually don't care about trojans or exploits since most only communicate at our demand. Should you consider evaluating security manners of the software/applications when you already know that the source is more or less trustfully?! Maybe. Should you care about what the server is doing and where is going during day or night and permit only the necessary in and out "talking's". Yes, you should.

I have said this over and over, I am FAR more concerned about people who are not concerned about security. Security should ALWAYS be important and even though we have taken great steps to ensure the security of our data it is absolutely imparitive that people only question what we do, but that they check to make sure we are not doing bad things. We would never intentionally do something bad, but despite the best of intentions things do slip through the cracks sometimes. As an IT professional you should always question those who are providing code to you. There is nothing about this situation that I take any offense to except people who are posting completely inaccurate information about the security of the data and the level of vulnerability it actually is. I am NOT saying you shouldn't question us and expect a full and complete disclosure of what is going on and it is all forthcoming. I am being VERY careful to spell it all out in as much detail as we are comfortable providing (meaning I am not going to tell you the exact network architecture and encryption algoriths which would completely undermine the security of the system).

yay. double post.

@jahyde:

Do you own some Fonality stock or something? You keep insisting there's nothing wrong, and perhaps in your "wizened" experience, editing crontab to fix the problem is more than enough, but in the real world, things like that tend to get not done unless there's some publicity about things like, say, remote execution of arbitrary code.

It's either that or the whole concept of things such as DNS poisoning and ARP spoofing are so far out of your league you resort to sticking your head in the sand like a good little ostrich.

You tell me. So far all you do is whine about dead horses and how there's nothing wrong, how about actually argueing the point -why- according to you nothing is wrong. Convince me that I'm wrong and you are right.

Dear trixbox CE community,

In an attempt to “Communicate Openly” (a mantra we preach religiously here at Fonality, despite our occasional lapses), I am going to explain the different methods by which trixbox CE systems communicate back to Fonality. Before you read this entire thing, please note that near the bottom of this document you will see that we are going to make a number of changes to “Heartbeat V3.0” as a result of all of your feedback. Thank you for being candid with us, and please accept this document as our sincere desire to remain open with you:

Heartbeat V1.0
Around 18 months ago, with the release of trixbox CE 1.0, we realized that we had a basic problem – we had no idea how many people were actually using trixbox CE. Sure, we knew our download volume…but, nothing else. So, without some basic sense of the viability of this project, we weren’t able to understand its impact – hence we weren’t able to appropriately budget the necessary financial resources to fund/improve trixbox CE. As such, we designed a basic heartbeat system. This system was discussed on the forums prior to launch in a thread started by Andrew Gillis. For the sake of 100% open communication, here is a re-hash of what Heartbeat V1.0 does:

When you use the trixbox CE dashboard, the system sends a generated unique identifier (GUID) back to Fonality. This GUID is generated upon the first connect. This GUID serves the purpose of informing Fonality that a trixbox CE system is actually “in use”. This system only heartbeats when the dashboard is actually used. Here is a complete list of what Fonality learns from this heartbeat:

1. A machine with a GUID has been used.
2. The timestamp of when it was used.
3. The IP address the GUID came from.

Heartbeat V2.0
There was a problem with Heartbeat V1.0. Basically there was no “who” attached to it. This means that we had no way of relating servers to people. As such we could not communicate back with folks in order to give them discounts (heartbeat discount club), give them paid support, let them know about urgent updates to their system, etc. So, around 11 months ago, with the launch of trixbox CE V2.0, we introduced a voluntary registration system, that for the sake of this discussion we shall call “Heartbeat V2.0”. Here is how it works:

When you first use the trixbox CE dashboard and you go to the Admin Panel a pop-up “registration window” appears. It asks you if you want to register. If you do register, Fonality will know exactly what data you chose to send us in your registration process. A byproduct of knowing this is that we could trace “you” to “your GUID”. Hence, we can know “who” is heartbeating. However, if you chose not to register, then Heartbeat V2.0 does not impact you in any way.

Heartbeat V3.0
Recently, Fonality has been trying to grow the CE development team (engineers+QA) as well as learn more about what types of hardware we need to build better interoperability with (there is just too much SIP hardware out there to test/QA everything). As such, we went to some of the vendors in the space and asked them to financially support trixbox CE because we assumed (based on reading the forums) that their products were being used by the trixbox CE community. Their answer, quite predictably, was “sure we will help out, if you can prove our products are in use.” Clearly, neither Heartbeat V1.0 or 2.0 ever dealt with the “what”. Therefore we had no way to answer the “what” question.

This meant that these vendors would never contribute toward our investment in CE, reducing the total dollars spent on CE development. The thing is, these vendors wanted to give us money to drive CE…but they wouldn’t do it unless we could prove CE was delivering business to them.

Thus was born Heartbeat V3.0. Below is an FAQ about Heartbeat V3.0:

When did Heartbeat V3.0 launch?
About three weeks ago.

Who got it?
Anyone who has installed 2.2.10 or later in the GA branch or 2.3.0.10 or later in the beta branch. Or, anyone who has done an “update” to their system in the past three weeks via the web-based package manager or directly via yum from the CLI.

Why didn’t I know about it?
Because we are idiots. More on that later.

What V3.0 does
V3.0 is really a “hardware audit tool” and it essentially tells us a bit about the hardware configuration on your server. Here is an exact list of everything it tells us:

1. IP Phone types and count
2. Useragent details (firmware version & MAC address)
3. OS version
4. RPMs installed
5. Info about cards, such as PSTN interface cards
6. Motherboard details such as manufacturer
7. Asterisk version
8. trixbox CE version
9. Registration Key, if the system is registered (see Heartbeat V2.0 above)

How does V3.0 do what it does?
Your trixbox CE sends an encrypted (unique) message to Fonality once every 24 hours. This message contains the exact data we have described above. It does not contain any user data (phone numbers, user names, email addresses) or usage data (who you called, what you did on the system, configuration, mod, etc.). The code is all open source and human readable on your trixbox CE server.

Um, can you give me a deeper technical explanation?
Ok, here goes. This communication is done via a Perl script which sends encrypted communication back to Fonality. Of course, like most implementations, the first connection to get the key is not encrypted. Once the key is established, the connection becomes secure. This Perl code is available on your server, in human-readable format, should you wish to peruse it. It can be found in /var/adm/bin/registry.pl

What we have learned and what changes we are making

1. Communicate Openly
We never meant to *not* disclose Heartbeat V3.0 three weeks ago. Honestly, we are a growing company, and as companies grow sometimes communication starts to breakdown, especially when we are working like demons to get trixbox CE 2.4 out the door. This is a classic case of communication break-down and nothing more Machiavellian. So, going forward, we are going to make a stronger commitment than ever to being transparent about everything we do, so that the right foot doesn’t trip over the left.

2. Inform new trixbox CE users up-front
The next version of trixbox CE will inform users *during install* about the Heartbeat system, and tell them how to disable it (in a simple manner from the dashboard, with no Linux, Asterisk, or CLI experience required.) This will be completed by December 21, 2007.

3. We are removing even our ability to ever modify this script
Fonality had (note the past tense) reserved the right to change this script at our discretion (to update it in case it is not efficient, make it more accurate, improve its security, make it stop checking in, etc.) The problem with this approach, as pointed out by a number of our community, is that a hacker (highly unlikely) or a malicious employee (only 1 at Fonality has the ability) could…well…be malicious. We took a very serious approach to the security of this solution. I won’t go into all the details, but suffice to say that, besides the aforementioned encryption, the actual server that could make changes is *not* on the Internet, or even available to most Fonality employees. It is proxied through a series of hops and protocols and highly protected at our data center. However, we have to be candid and say that any system that accepts open commands from another system can be a threat. As such, we are going to be removing (yes, removing) even *our* ability to modify this script. The only thing we will have the capacity to do to this script, upon check-in, is to tell it to *stop* running it. We think it is ethical that we retain at least this much control.

In closing, we would like to first apologize for upsetting anyone. Our goals are to fund trixbox CE to give it long term viability, and we got ahead of ourselves in not being transparent enough. Secondly, we would like to again thank you each (with a special shout to Lars) for being so candid with us. Hopefully, we have reacted quickly and ethically to your concerns.

While I am the one posting this, this statement was prepared in conjunction with Andrew Gillis, and Chris Lyman as well as our engineering staff to ensure accuracy and company policy.

I will be posting tomorrow as these fixes begin to roll out to ensure everyone can review and comment on them as they progress.

Chris Lyman
Fonality CEO

Kerrry Garrison
trixbox Community Director

Andrew Gillis
trixbox Founder

seriously, the taunting and bashing is getting more than old, get the above done, and get on to real development like finalizing 2.4 is something that should be higher on the list.

How about you wait until December 21, 2007 when the new version is ready?

As a matter of fact I have :) If you want to read the full deal, here's a link:

http://www.superunknown.org/pivot/entry.php?id=15

Put it there since this forum doesn't seem to like code blocks and overly long lines all that much. Keep in mind I wrote that up before reading kerryg's post below about the fixes.

MadCat, very well done. I would highly recommend posting this to the VOIPSEC mailing list at http://voipsa.org/mailman/listinfo/voipsec_voipsa.org

Whilst all this seems to have been a bit of a mess, it looks like yet another proof of the adage “there is no security in obscurity”.

The level of openness within this product has allowed the community to discover a theoretical problem, raise awareness about the problem and get people moving toward an imminent resolution that the community will verify.

I am sure that many proprietary phone systems do similar or worse “phone home” activities, without the openness or even willingness to discuss what it does, what coding and encryption is in place to protect and secure these processes.

Bitch to a traditional pbx supplier about something like this and you would in all likleyhood get a very glib and terse reply, - “it’s a feature, its secure, trust us” without any way of actually knowing that they actually know what they are doing.

Once the dust has settled, I think we will look back on this issue as a good example of Trixboxs’ responsiveness to a flawed piece of code.