Openssl segfault bug and building Curl with new openssl libraries

Recently, I met a bug in the openssl library which results in segfault when sending HTTPS requests using the Curl library (which uses openssl).
All the methods of this post have been tested on Ubuntu 12.04.

When using GDB to backtrace the segfault, the segfault looks like:

(gdb) bt
#0  0x00007fbaf69a54cb in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#1  0xca62c1d6ca62c1d6 in ?? ()
#2  0xca62c1d6ca62c1d6 in ?? ()
#3  0xca62c1d6ca62c1d6 in ?? ()
#4  0xca62c1d6ca62c1d6 in ?? ()
#5  0xca62c1d6ca62c1d6 in ?? ()
#6  0xca62c1d6ca62c1d6 in ?? ()
#7  0xca62c1d6ca62c1d6 in ?? ()
#8  0xca62c1d6ca62c1d6 in ?? ()
#9  0x00007fbaf6d10935 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#10 0x00007fba3c12cb70 in ?? ()
#11 0x000000000000000a in ?? ()
#12 0x00007fbaf69a1900 in SHA1_Update () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#13 0x00007fbaf6a23def in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#14 0x00007fbaf69d75e5 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#15 0x00007fbaf69d73c8 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#16 0x00007fbaf69edf9b in EC_KEY_generate_key () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#17 0x00007fbaf6d2f2a4 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#18 0x00007fbaf6d30c03 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#19 0x00007fbaf6d3a373 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#20 0x000000000049a015 in ossl_connect_common ()
#21 0x000000000046a428 in Curl_ssl_connect_nonblocking ()
#22 0x000000000046ee9e in https_connecting ()
#23 0x00000000004671ee in multi_runsingle ()
#24 0x0000000000467cd5 in curl_multi_perform ()
#25 0x0000000000462e9e in curl_easy_perform ()

If you have a look at the dependent libraries used by curl, you could find the ssl libraries it is using:

$ ldd ./curl
    linux-vdso.so.1 =>  (0x00007fffd6039000)
    libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f48cf957000)
    libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f48cf6f9000)
    libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f48cf31d000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f48cf106000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f48ceefe000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f48ceb3f000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f48ce93b000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f48ce71e000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f48cfb9d000)

After googling for a while, I found this should be due to a bug in the openssl library which has been fixed since 1.0.1c, as mentioned on Reference (1). The openssl versions are 1.0.1, 1.0.1a, 1.0.1b, etc. But Ubuntu 12.04 is using some buggy version of openssl 1.0.1 by default.

The next question is just how to build Curl with new versions of openssl without the segfault bug. Again after googling, I found it is not that easy as expected. Many people created hacking ways to do this. I end up finding an easy way to do this:

# install openssl, as the default openssl on Ubuntu 12.04 is buggy about https connections, which could result in segfault
git clone https://github.com/openssl/openssl.git
cd openssl
git checkout OpenSSL_1_0_2g./config --prefix=$LOCAL_DIR no-shared
make -j
make install

# install curl
git clone https://github.com/curl/curl.git
cd curl
git checkout curl-7_48_0

autoreconf -iv
CPPFLAGS="-I$LOCAL_DIR/include" \
LDFLAGS="-L$LOCAL_DIR/lib" \
LIBS="-ldl" \
./configure --disable-shared --prefix=$LOCAL_DIR --without-ldap-lib --without-librtmp --with-ssl
make -j 
make install

After installing curl, the new curl binary executable file contains the new libssl inside statically (OpenSSL/1.0.2g):

$ ldd curl
    linux-vdso.so.1 =>  (0x00007fff8f9fe000)
    libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f5fdae42000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5fdac2b000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5fdaa22000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5fda81e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5fda460000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5fda242000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5fdb080000)
$ ./curl --version
curl 7.48.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.48.0-DEV OpenSSL/1.0.2g zlib/1.2.3.4 libidn/1.23
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

Important notes for developers: when you build your own program against the curl library (libcurl.a), you may find you end up having a binary executable file which still requires the buggy openssl library like:

    libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f0338f94000)
    libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f0338bb8000)

but no worries, I found that this may just be due to the curl library would force you link your program with these buggy openssl libraries by default if you use curl-config to generate your g++ arguments:

$ ./curl-config --libs
-L$LOCAL_DIR/lib -lcurl -lidn -lssl -lcrypto -lssl -lcrypto -lz -lrt -ldl

even if you follow the methods mentioned in this post. However, your program won't really use the buggy openssl libraries on your Ubuntu 12.04 (I have tested my program, and the segfault does not happen any more).









References
(1) https://bugs.launchpad.net/ubuntu/+source/s3cmd/+bug/973741

(2) https://curl.haxx.se/mail/lib-2014-12/0053.html

(3) https://github.com/openssl/openssl/tree/OpenSSL_1_0_2g

(4) https://github.com/curl/curl/tree/curl-7_48_0