Wednesday, December 31, 2014

KISS with a bit of luck

I have a habit of digging up old code around the new year, taking a look at it and posting it on github. Part of me thinks that I'm doing a bad favour to myself and anyone else that hits that code and decides to uses it. On the other hand looking at your old code is a nice way to see if one have improved in any way over the years. I do stamp warnings all over the place when I do that, so I hope the collateral damage isn't that large.

This year I dug up an old Perl script I wrote with a friend of mine almost 9 years ago. This is just an estimation based on the file system timestamps on the server. The oldest files I found were dated back to February / May 2006. That's the deployment time - it was probably written a while back before that.

Usually I don't blog when I do that but this time the code is interesting to me in ways that I'll unfold later on.


KewlAdmin was born because a friend of ours offered a shell service for his friends, he was getting tired of manually defining vhosts for people so asked us to come up with a solution. The requirements set was pretty simple:

  1. Authenticate users
  2. Allow them to view their current disk quota (based on account class)
  3. Allow to define/remove vhosts

There were solutions at that time to handle these problem but most software required escalated privileges in order to work.

To get this out of the way. The code is in Perl because at that time it was our primary languages and basically we wrote everything in Perl back then :)

Why is this code so interesting to me? Mainly four big reasons.

1. It doesn't store hashed & salted passwords. In fact it doesn't store any passwords at all.
2. It ties in nicely witch existing utilities in *nix that we didn't even think about when we wrote the code.
3. It doesn't require root/escalated privileges to do the work
4. It was actually in use for at least a couple of years - I made sure with the admin that the code is decommissioned before posting the code itself and this blog post.

Don't be fooled. This code is far from perfect. I'll get to the embarrassing bits as soon as we go over those 4 points.

No salt, no hash

I had a shell on this host and so did my friend. We both didn't want to register yet another account in a web application in order to view our quota usage or to set a new vhost for a quick project. With that in mind how can we authenticate existing shell users without requiring more privileges to the script itself?

The solution we came up with seems non-standard and I didn't see it implemented in other software so far. We decided to rely upon ssh and existing credentials. Using the Perl Expect module we opened up a real ssh session to the host itself from our CGI script and passed the users credentials to ssh.
## connect via ssh
$Expect::Log_Stdout = 0;
$ssh = Expect->new("ssh $login\@localhost");
#$ssh->debug(1);
$ssh->expect(2,qq{$login\@localhost's password:});
$ssh->send($passwd . "\n");
unless(scalar $ssh->expect(2,'$') || $ssh->expect(2,"Changing password for")) {#$ssh->expect(2,"Old password:")) {
return 0; #unable to login
}
else {
return $ssh; # logged in
}

This choice also allowed one more feature in the panel. Changing passwords.
Typical password forms want an old password and the new password repeated twice.

If we have the old password we can always authenticate the user with ssh and issue the passwd command changing the password using the existing *nix tool.

sub change_pass {
my ($ip,$login,$old_pass,$new_pass,$repeat_pass) = @_;

return 0 if $new_pass ne $repeat_pass;

my $ssh = login($login,$old_pass,$ip);

unless(check_shell($login)) {
$ssh->send("passwd\n");
}

$ssh->send("$old_pass\n") if (scalar $ssh->expect(2,"Old password:"));
$ssh->send("$new_pass\n") if (scalar $ssh->expect(2,"New password:"));
$ssh->send("$repeat_pass\n") if (scalar $ssh->expect(2,"Re-enter new password:"));



if($ssh->expect(2,"Password changed.")) {
$ssh->close;
return 1;
}

return -1;
}

KISS by accident

I think we never realized how much power this solution gave to the system administrator - which is a good thing.

OpenSSH is a beast when it comes to functionality. Everything is logged, the crypto is properly implemented (something we wouldn't be able to come close too back then and possibly even now). Since we were basically logging in our users via ssh the system configuration applied to KewlAdmin. Rate limiting, monitoring brute force attacks - everything was available with the basic tools that our sysadmin friend was used to.

We did implement a login attempt throttle that banned accounts for an hour after a defined number of attempts based on the source IP. I think that was a good decision back then since OpenSSH would have no way to know the real source IP for the incoming traffic. This also had the benefit of not impacting users shell access just because someone is pounding on the web interface.

The way we obtained the quota limits was also interesting because it shelled out via sudo. Properly configured sudo is great and also logs each invocation. It was called by the script itself as user nobody - because we didn't keep the ssh session alive after logging in the user. Again, this is nice because it gave power to a person who new the tools without forcing him to understand our code.

Most of all. If someone gained access to the database itself. Including write access. No useful information would be leaked out and changing passwords to existing accounts would remain impossible. Without hashes in the DB there's nothing to crack and nothing to alter.

One more thing. If for some reason our admin friend revoked a shell account. He didn't have to remember to do the same in the web interface.

Privilege Separation

Using OpenSSH as our authentication mechanism allowed us to skip escalated privileges while authenticating users.

Using sudo to check the quota's allowed us to skip re-authenticating users all the time.

The remaining bit was actually altering apache configuration with the potentially changed vhosts.

This requires altering a system configuration file and restarting a process. We didn't want the nobody user to be able to do this.

The solution was again simple. We had a second script - kewlup.pl. This script was run with escalated privileges on a cron schedule and it's only task was to read the new configuration from the database, rewriting the vhosts file and restarting apache.

With those three pieces in-line. Bulk of the code was run as user nobody, with no special privileges and all privileged actions were logged & controlled by standard utilities well understood by system administrators.

Living Dead


The most surprising thing is that this script was actually alive and in use.
We never got a 'bug report' against it from any of the users or the admin himself. Though now I see that he did tweak some stuff here & there (like translating error messages from English to Polish, or changing vhosts when he switched domains).

Here are some stats grabbed from the database this script used:

  • registered 228 config updates (ka_config.updates) this denotes the number of times the kewlup.pl script runs that resulted in vhost changes.
  • ka_domains has 139 entries from 105 users.
  • 907 entries in ka_users (unique username + ip login attempts)
  • 417 unique usernames trying to log in
  • 524 failed login attempts

There are only 114 users on the server itself, based on the number of unique users and some usernames I saw the site was subject to brute force attacks. The one that stands out the most is for the 'java' account:

|  59 | X.Y.Z.N     | java                           |        13 |  1153683487|

There's of course no account on the server with that username.

The ugly

Now the ugly parts. The most prominent offender is the way we decided to generate the session key.

sub genkey {
my $key ='';
my @char = ('a' .. 'z','A' .. 'Z', 0 .. 9);
$key  .= $char[rand@char] while length $key != 32;
return $key;
}

Please never do this. Nowadays I would use an UUID generator making sure it's cryptographically secure before touching it.

Though genkey is awful - the way it's used is even uglier.

sub create_cookie {
 # ...
$sth = $dbh->prepare("SELECT id FROM ka_keys WHERE cookie = ?");

##
# generate until unique key
while (1) {
$key = genkey();
$sth->execute($key);
last unless $sth->rows;
}
 # ...
}

We were hitting the DB to check on each call to make sure that the token we generated is unique in the DB. With a proper UUID you should never do that. Just generate the token and handle a potential error case on insert/update with lot's of logging.

Fortunately even with a predicable session key the only thing a hijacked session could do was to alter/add/remove a vhost or view the current quota for an account. That's still far more than I'm comfortable with now but at least there's no obvious way (to me) that an attacker could gain shell access this way.

Second thing I'm not proud of is picking MySQL. This one is a bit personal as MySQL is used a lot by a lot of people but I just don't trust it any data any more. When this code was written, I knew nothing about databases except trivial basics. After 7 years with Oracle & around 3 years with PostgreSQL I'm not touching MySQL with a 10 foot pole.

The last bit that made me uneasy is actually the shell outs to the commands.

Let me list them here:

        # ....
if ($_login =~ /^([A-Za-z][-_A-Za-z0-9]{0,29})$/) {
$login = $1;
        # ....

my $uinfo = `finger -mp $login`;
# ....
my $space_used = `sudo du -sm ~$login | awk '{print \$1}'`;
# ....

Yes, $login was untained before usage but I'm still uneasy about calling any shell command with string interpolation.

Final thoughts

The code is a mess. It did stood the test of time and I'm somewhat happy that it did. Reading Perl after 9 years wasn't as bad as I expected but my inexperience back then really stands out.

  • There is no routing in the code, just a giant if/elsif/else branch for action decision
  • Nowadays I avoid regular expressions, back then I used them far too often
  • Too much stuff is hardcoded (db credentials, vhosts, account classes)
  • Plus all the things mentioned before in this post
I think the saving grace of this piece of code was privilege separation and the use of OpenSSH as the authentication mechanism. Without it I'm sure our friendly shell hosting provider would be brought down by the mere existence of this script.

Standing on the shoulder of giants really fits the bill here. I literally didn't look at this code for 9 years, while ssh remained maintained and updated. With each OpenSSH update our script benefited. I doubt that back then we would even think about timing attacks, encryption algorithms, side channel attacks or even just keeping the system accounts in sync with the web interface database. If we didn't went the path we did this script would probably store a salted md5 hash of user credentials and ran with escalated privileges to do it's job.

With a bit of luck, it was kept simple and served some users for a couple of years.