I see a lot of misinformation floating around about the OpenSSL Heartbleed bug. In case you’ve been living under a rock, OpenSSL versions 1.0.1 through 1.0.1f are vulnerable to a condition where a particular feature will leak the contents of memory. This is bad, because memory often contains things like the private half of public-key cryptographic exchanges (which should always stay private), protected information, parts of your email, instant messenger conversations, and other information such as logins and passwords for things like web applications.
This problem is bad, but freaking out about it, and talking out of our duffs about it, adds to the problem.
1. Not all versions of OpenSSL are vulnerable. Only fairly recent ones, and given the way enterprises patch you might be just fine. Verify the problem before you start scheduling remediations.
2. Heartbleed doesn’t leak all system memory. It only leaks information from the affected process, like a web server running with a flawed version of OpenSSL. A modern operating system prevents one process from accessing another’s memory space. The big problem is for things like IMAP servers and web applications that process authentication data, where that authentication information will be present in the memory space of the web server. That’s why this is bad, but it doesn’t automatically mean that your SSH-based logins to a host are compromised, or anything stored on a vulnerable server.
Of course, it’s always a good idea to change your passwords on a regular basis.
3. People are focusing on web servers being vulnerable, but many services can be, including your email servers (imapd, sendmail, etc.), databases (MySQL), snmpd, etc. and some of these services have sensitive authentication information, too. There’s lots of email that I wouldn’t want others to gain access to, like password reset tokens, what my wife calls me, etc.
4. A good way, under Linux, to see what’s running and using the crypto libraries is the lsof command:
$ sudo lsof | egrep "libssl|libcrypto" | cut -f 1 -d " " | sort | uniq cupsd dovecot dsmc httpd imap-logi java mysqld named nmbd ntpd sendmail smbd snmpd snmptrapd spamd squid ssh sshd sudo tuned vsftpd
This does not list things that aren’t running that depend on the OpenSSL libraries. For that you might try mashing up find with ldd, mixing in -perm and -type a bit.
5. Just because you patched doesn’t mean that the applications using those libraries are safe. Applications load a copy of the library into memory when they start, so you replacing the files on disk means almost nothing unless you restart the applications, too. In my item #3 all of those processes have a copy of libcrypto or libssl, and all would need to restart to load the fixed version.
Furthermore, some OSes, like AIX, maintain a shared library cache, so it’s not even enough to replace it on disk. In AIX’s case you need to run /usr/sbin/slibclean as well to purge the flawed library from the cache and reread it from disk.
In most cases so far I have chosen to reboot the OSes rather than try to find and restart everything. Nuke it from orbit, it’s the only way to be sure.
6. Patching the system libraries is one thing, but many applications deliver libraries as part of their installations. You should probably use a command like find to search for them:
$ sudo find / -name libssl\*; sudo find / -name libcrypto\* /opt/tivoli/tsm/client/ba/bin/libssl.so.0.9.8 /opt/tivoli/tsm/client/api/bin64/libssl.so.0.9.8 /home/plankers/pfs/openssl-1.0.1e/libssl.a /home/plankers/pfs/openssl-1.0.1e/libssl.pc /usr/lib/libssl.so.10 /usr/lib/libssl.so.1.0.1e /usr/lib64/libssl.so.10 /usr/lib64/libssl3.so /usr/lib64/libssl.so /usr/lib64/pkgconfig/libssl.pc /usr/lib64/libssl.so.1.0.1e /opt/tivoli/tsm/client/ba/bin/libcrypto.so.0.9.8 /opt/tivoli/tsm/client/api/bin64/libcrypto.so.0.9.8 /home/plankers/pfs/openssl-1.0.1e/libcrypto.a /home/plankers/pfs/openssl-1.0.1e/libcrypto.pc /usr/lib/libcrypto.so.1.0.1e /usr/lib/libcrypto.so.10 /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.10 /usr/lib64/libcrypto.so /usr/lib64/pkgconfig/libcrypto.pc
In this example you can see that the Tivoli Storage Manager client has its own copy of OpenSSL, version 0.9.8, which isn’t vulnerable. I’ve got a vulnerable copy of OpenSSL 1.0.1e in my home directory from when I was messing around with Perfect Forward Secrecy. The rest looks like OpenSSL 1.0.1e but I know that it’s a patched copy from Red Hat. I will delete the vulnerable copy so there is no chance something could link against it.
7. If you were running a vulnerable web server you should change your SSL keys, because the whole point is that nobody knows your private keys. If someone knows your private keys they’ll be able to decrypt your traffic, NSA-style, or conduct a man-in-the-middle attack where they insert themselves between your server and a client and pretend to be you. This is all bad. Change your keys. Remember that lots of things have SSL keys, like OpenSSH, mail servers, etc. In the case of sshd on Red Hat you can “sudo rm /etc/ssh/ssh_*key*” and restart the service, which will regenerate them.
8. Remember that you have, or should have, defense in depth. Is there a firewall between the world and your SSH server? What is the risk that someone permitted through that hole in the firewall has compromised things? That will inform your response to this. Incident response is all about risk, and choosing the higher-risk problems to solve first is often a good strategy. In my case, the only people allowed in via SSH are people with sudo rights anyhow, so I’m focusing on the web and email servers.
And, while you’re messing with all your SSL certs, step up your SSL security in general. A great testing tool I use is the Qualys SSL Labs Server Test, and they link to best practices from the results page.
Did you like this article? Please give me a +1 back at the source: 8 Practical Notes about Heartbleed (CVE-2014-0160)
This post was written by Bob Plankers for The Lone Sysadmin – Virtualization, System Administration, and Technology. Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License and copyrighted © 2005-2013. All rights reserved.