Domino security breach
by Volker Weber
A very large Notes customer apparently has had a security breach. This is not yet public, but is being discussed in back rooms. Here is what I have learned:
One of the things that happened is that a user has copied the directory. The directory has subsequently been exported to a text file, all of the password hashes were extracted and tested against a dictionary. About 10% of the passwords turned out to be trivial. What should the customer have done to prevent this from happening?
One of the things I know is that IBM has introduced a More Secure Internet Password Format in Domino 4.6:
The customer obviously has not used this setting. If he did, then the password hashes would be much harder to test.
What else could have been done to prevent somebody from getting at this information in the first place?
Comments
I'm not quite sure, but it is really necessary to assign normal users the right to copy/replicate the Domino Directory? I don't think so. And if this right is omitted, at least normal users could not download the whole directory.
There is an article on developerWorks, that describes the usage of the extended acl to prevent the access to the HTTPPassword field:
http://www-1.ibm.com/support/docview.wss?uid=swg21244808
Thomas
One possible thing that could have been done is to check for weak passwords on a regular basis. I have a script I run on my servers that goes through all the passwords in the NAB and checks them against a dictionary. If there are any simple dictionary based passwords it sends an alert email so the issue can be fixed.
Now, either the administrator of a system can do this or any normal user can do this.
Dan
I'm not sure that I trust xACL enough yet, but that is one way.
Apart from that, you can make it somewhat harder to do, you can make it harder to do without detection, and you can (probably) make it only possible for someone with some programming skills... but you can't prevent it. The data is there. You don't have to replicate it or copy it. You can write code to get it.
The same user who copied the directory also did the "cracking"? Maybe they should look into how they treat their employees...
Anyway, why doesn't IBM enable this option by default, if it really was that good? One other thing: can't you allow access to the hashes to one uber-administrator account only?
Without knowing how long this customer's Domino Directory has been in production, we don't know whether the default value for the security setting is of consequence in this particular case. It was not the default value for a long time because setting it to the stronger hash would break any very old Domino servers in the network. I'm not sure if current versions have made the strong hash the default, but I do recall discussing it with IBMers who said that they probably should do it.
I think there are different issues/questions nicely inter winded:
How secure is Domino out of the (new) box?
How secure is an upgraded Domino?
How well can you lock it down?
How easy is the lock-down if you know/don't know what you are doing?
How easy is it to assess the security?
IMHO #3 is excellent, #1 is so-so, #2 depends and #3/#4 need improvement. Of course there is always consultants who can do that assessment, who are usually only are funded after a breach.
@Dan: that script would make a nice SNTT!
:-) stw
I'm not sure these kind of article helps as much as you probably intended, and I suspect it will be misused by the FUD machine... Ammo for the illiterate !
Domino can and will be as secure as configured, and SHOULD be more secure by default AND the upgrade should force the more secure passwords AND people should RTFM too AND many more things...
Having said that, if you allow me a constructive thought: In my view, suggestions for improvement of security defaults, belong into the area of responsible private communications to the vendor, and somebody like you does not need to make a big issue out of a misconfigured and mismanaged notes/domino installation.
It's not worthy news to say somebody had a security problem because of NOT following what are industry wide accepted best practices.
Usual best personal regards,
George Chiesa
It's not worthy news to say somebody had a security problem because of NOT following what are industry wide accepted best practices.
George, I respectfully disagree. It is actually BIG news if that somebody depends on the trust people place in its security practices.
What I am trying to find out here is what the "industry wide accepted best practices" are. Using the More Secure Internet Password Format is such a practice I assume. Now I am trying to find out what the rest are.
Once this incident becomes public, there will be neither uncertainty nor doubt. At which point it becomes interesting to differentiate between a product fault or a procedures fault. If the capabilities exist to prevent this breach, it become a procedures fault.
(This is mostly directed to George) I think the general standard of what should be private between a person observing a security weakness and a vendor and what can reasonably be made public depends on whether there is a fix available. If there is no fix available, it is reasonable and desirable to communicate privately. If a fix is easily available, especially without any requirement to upgrade, as it is in this case, it is very reasonable to make a public stink, although it is also good to point out when the issue is a configuration issue and point out what could be done to remediate. Volker has complied nicely with all of this, reporting a problem that has an easy fix and publicly reminding people that there are reasons to configure your systems properly. I think it is an excellent reminder and warning. I also think it is a good question to discuss, as there may be alternative ways to at least make it more difficult to access this data.
I totally agree with Ben. Just think of how many Domino admins might stop by here, read this, check their own config and realize that it could have happened to them as well. And looking at it the other way: How has this done any harm to anyone? Ammo for the illiterater? Well, I guess this ammo can fairly easily be voided and if not, then the illiterate wouldn't care if they had ammo or not, they'd simply make their decisions anyway.
Well, in an enterprise environment, don't use HTTP passwords at all. Authenticate against a LDAP server via a DA instead.
In Notes/Domino 7 use a custom password policy.
If you want to enable HTTP on Domino - take a course.
If you are responsible for Domino security in your domain - take a course too.
Ok, I see. I respect free speech and your opinion.
Let me further elaborate to clarify why I think we should not SPIN a trivial incident, no matter how good news the victim name was...
In my view, the "incident" was a combination of:
a) not best practices, actually obsolete pre 4.6 practices
b) lack of awareness/education (in the good sense)
c) old default values (see next!!!)
d) domino upgrade process NOT automatically enabling more secure passwords (even if you said yes to design changes of nab when you uodated, it's not retroactive, it does not do a computewithform for each existing document in the nab, as you all know, ain't that trivial).
My observation: people are feeding suggestions like this to IBM in private, and I suggest similar suggestions should be sent to the lotus security team in private not blogged because they can be ammo. I'm absolutely confident that many of these suggestions ARE being implemented. As some of you know, I invest a substantial amount of (unpaid) time (and expenses) to document stuff (not just redbook, but also many communications to ibm in private, and postings in private forums, and meetings, etc etc etc) so respectfully, I believe I AM entitled to suggest to always contact lotus security team - in private - before making any public comment about so called incidents. We all know how FUDded the general public is...
G
With all due respect to both George and Volker:
George, I agree with Ben regarding the standard for public disclosure, and I believe Volker's post falls on the correct side of that standard.
And Volker, the headline of your post proves that you have an excellent sense of how to draw attention to your material, but George is right about the fact that it could be misconstrued -- accidentally or deliberately. Since you clearly know that the breach was avoidable by using an available feature that is nearly 10 years old, perhaps a more appropriate headline would have been "Domino Security Goof". But you're the journalist, not me ;-) And if Domino 8 betas are in fact shipping with the weaker setting as the default, which I don't know because I haven't set up a domain from scratch with ND8, then drawing a little extra attention to the subject now probably does more good than bad.
Thank you, Richard.
George, you can safely assume that we have tried to communicate with the customer first. So far he does not speak with us. And yes, Lotus knows.
I like to learn something new every day and therefore welcome every feedback, positive and negative. But I am not exactly a beginner in this space. Eight years ago we reported a fix to the templates which from then on defaulted to "no access". Lotus made this change after we found countless server on the internet, which were easily accessible. At this time we did not disclose some major security breaches we found. Think Army, nab, server.id, default manager ... Nick was the CTO at that time and can attest that we have always tried to communicate the issues privately first.
We are not reckless. But curious. Let's establish what we would regard as good practices regarding passwords in the directory.
I know of this fact for a long time, I even discussed it with Lotus in Orlando 2 years ago. I even created an optimized version of the hash algorithm, which is ~10 times faster than the one in Notes, so I can test around 200000 passwords per second. I did some password analysis for a few customers, usually I get 30% of all HTTP-passwords in less than an hour, often including passwords from CEOs and even admins.
The secure password option is not much better. The computation takes only double the time, because 2 hashes must be computed.
The xACL isn't really secure (despite the problems still existing), because with some tools (e.g. NotesPeek) the password hash is still visible.
There are some more interesting things possible if the HTTP password is identical to the ID-File password. Then you can use the ID without knowing the password at all.
As I said I told Lotus, but it is hard to fix. And I promised not to disclose any details.
The only real workaround is: use good passwords, really good passwords.
Andreas, how does the More Secure Internet Password work? Hash the name, add the password, hash again?
I think this is one of those symptomatic issues that have people talking about how Domino sucks. Because IBM is big on backward compatibility, they allow admins to make the choice to use the old weak passwords. Is it the fault of the software that allows an administrator to do dumb things? Probably not. Is it just a Domino problem? Absolutely not, but I think that issues like this hurt Domino more than IIS or Sharepoint specifically because it's a dumb, easily preventable admin issue rather than a flaw with the product.
It's very easy to be ignorant about things like this in Domino if you don't have expert administration and/or aren't totally sure what's out there in your environment (since the post mentions that it's a very large environment, you're looking at potential for both of these issues).
That said, if I recall correctly, in R7, Domino Domain Monitoring should pop the weak httppassword setting under Security Best Practices.
@volker:
Hash the password, append a random number and hash the result again (as far as I remember, maybe the other way round, but only to hash function calls).
This way identical passwords seldom hash to the same value and a precomputed dictionary attack is impossible.
Unfortunately, to validate a password, the random number must be known, so it is stored more or less in plain text in the final hash. So you can start a dictionary attack because you have everything to compute a hash with the same random value.
I was going to ask about the random number.
You are right about the precompiled attack. I know a few people who recognize the hashes for password, passwort and geheim. :-)
My understanding of the "More Secure" option is that it is a "salted hash", which means that a random string (the "salt") is added to the password, and the combined string is hashed. (I think in the case of Domino that it is actually done such that the password is hashed, then combined with the salt, and then hashed again, but I'm not sure of that. In any case, this only adds a little to the strength that you gain from the salt in the first place.)
The salt must be stored somewhere so that it can be combined with candidate password to test for a match against the hash. Obviously, this means that the More Secure hash is actually not more secure for anyone who can retrieve the correct salt value. The security comes from the fact that the salt is not easily retrievable. I'm not sure where the salt is stored by Domino, and I'm not sure if it is encrypted -- but that would be my guess.
This all means that a dictionary attack against the salted password must combine every word in the dictionary with every possible salt value. With a large enough salt, this makes the effective dictionary size very large.
Security is hard to get right. Thats a fact. At least Lotus chose an established algorithm (there is no known flaw in there), rather than a homegrown. One problem is the visibility of the hash to other users, another problem is that the same algorithm is used in different places, so it is possible to stuff the hash into the Notes authentication process (ID-File password). For this reason I strongly recommend not to use the same password for your ID-File and your HTTP-Account.
The first problem could be solved by storing the hashes in a place, where only the server has access, as under UNIX.
@Richard:
The salt is simply stored in the final hash and is easily recovered. As far as I remember it's a 16bit salt. You don't have to try all salt values for a given password, you have the salt available. So computation time is only twice as much as with the simple hash.
@Andreas: If that's true... wow! That's all I can say. Wow.
It´s not a problem that the salt is easily accessible - it is just there so that the same password does not get the same final hash value every time. This was pre-4.6 behaviour, and made it possible to easily compare a (maybe pre-hashed) list of passwords to *all* HTTP password hashes in names.nsf.
Now with the "salted password", you can still do a dictionary attack - but the difference is you have to actually compute the hashes for every try (with the according salt value). This makes a *huge* difference.
Every password to my knowledge vulnerable to a brute force or dictionary attack - so it´s simply the same story with (salted) HTTP passwords than with everywhere else: Use strong passwords, and you are fine.
Every password to my knowledge vulnerable to a brute force or dictionary attack
Of course it is. However, you need a resource to test against. It is most convenient if you have the hashes. That is why Unix hides the them from the users. Domino does not.
Katherine Holden (security development manager) posted a link to this technote in the Lotus Partner Forum discussion on this topic
Configuring xACLs to protect Internet Password fields in the Domino directory
I'm a bit surprised that people shy away from the xACLs, personally. I've had nothing but success with them; they're extremely flexible and incredibly powerful. The "more secure password" option is good, but as several people here have pointed out it doesn't prevent someone from getting at the data and brute-forcing it. xACLs, done correctly, DO prevent people from getting at the data.
You do have to plan out your implementation carefully, but in my book anything that touches the NAB ought to be planned out carefully anyway.
The other thing to note, as that article Ed suggested does, is that you can't back up and restore the xACL, so you really have to document it. But again, that really ought to be a standard Best Practice anyway.
How many companies out there perform a regular security audit? Particularly on Domino?
A major company came into an organization that I worked for, and didn't have the tools, or the knowledge to perform a simple audit, much less a real serious one.
I guess there's no real reason to close the doors until after you lose a few cows.
@Eric: That is the point - "if xACL done correctly". That is not that easy. I do a xACL based hosting for notes clients, and I found that there are quite many issues with it. I could work my way around it for most of parts, but that was not always easy. (One example - it does not work for Mail-In databases. Another - there are some issues if you have a country bit of a hierarchical name, like most people outside of the US have i.e "CN=Gregory Engels/O=Kompurity/C=DE" - thats why if you try to set up a server with xACL enabled, the setup does not let you register such a name)
@Andreas: with xACL correctly set up, you wouldnt see the passwort hash with notespeek. In a hosted environment I even can you set up the way, that you would only see your own person document and nothing more. (and even not all fields in there)
@Vowe:
Suggestion: amend first line and we're ll happy.
A very large Notes customer - who had not applied best practices known for 10 years - apparently has had a security breach
George, again you seem to know more than I do. I have no idea what the customer did or did not do. I only know the outcome and I have speculated that they may not have used the More Secure Internet Password option. During the course of this discussion I have learned however, that this must not necessarily have been the case.
@Gregory: I'm not Eric, but I take your point. xACLs are definitely complex.
Which does raise one question: knowing that the password field can be a major security vulnerability (especially if synchronized with other systems), and knowing that xACLs can solve this if properly configured, and knowing that properly configuring them is something many admins struggle with, why doesn't IBM provide a brain-dead simple interface to allow admins to protect the password field? Why isn't it available as a potential default, as a "more secure" setting?
I suspect there are practical, technical issues that could make this difficult to implement, but difficult != impossible. Maybe someone with good connections within IBM (*cough*Volker?*cough*Ed?*cough*) could raise the question, particularly in light of this company's issue?
HTTP passwords in the directory were an afterthought. Notes was never designed for username/password authentication. The implementation is severely flawed, as the data is not hidden from the user. xACLs is yet another interesting afterthought but the issue is larger. The directory (formerly names and address book) just serves too many purposes. And there are many many things in there, that a normal user should never see.
As to why hiding the password isn't easier to solve, it probably is in IBM's best interest to make it easier. IGS runs several Domino infrastructures for clients. And you can safely assume that their chance of failure is not necessarily smaller than their customers'.
Finally, IBM is very well aware of this incident. In more ways than one.
@Gregory:You can see the password with NotesPeek, you simply don't see it where you expect it. The field in the person document is hidden even from NotesPeek, that's true otherwise xACL would be useless. But: you can still see the password hash in the summary buffer of the document.
"Every password to my knowledge vulnerable to a brute force or dictionary attack."
That's true. The problem is that simple passwords are easily recovered. Choosing a 8 character random password makes this attack too complex. The interesting thing in Notes is, that you have access to a large number of password hashes, a lot of them being simple (company name, dictionary word, dictionary word with some number appended ...). If you look for a specific password, it can be hard, but if you look for passwords in general, it's almost certain successful. Having hundreds of password, you can look for the best account (CEO, Management, Admins). Often you find one of them amongst your "hacked" list.
