Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenSSL 3.0 performance issue: SSLContext.set_default_verify_paths / load_verify_locations about 5x slower #95031

Open
fcfangcc opened this issue Jul 20, 2022 · 7 comments
Open

OpenSSL 3.0 performance issue: SSLContext.set_default_verify_paths / load_verify_locations about 5x slower #95031

fcfangcc opened this issue Jul 20, 2022 · 7 comments
Labels
3.12 performance Performance or resource usage type-bug An unexpected behavior, bug, or error

Comments

@fcfangcc
Copy link

Bug report
Example code in ubuntu20.04(openssl1.1) is much faster than ubuntu22.04(openssl3.x)
Not just speed, CPU occupancy ubuntu22.04(openssl3.x) is many times of ubuntu20.04(openssl1.1)
I'm not sure whether it's OpenSSL or Python adaptation problem

import socket
import ssl
import time

import certifi
hostname = 'www.python.org'  # any support https hostname
times = 100
pem_where = certifi.where()
context = ssl.create_default_context()
verify_total_time = 0

for i in range(times):
    with socket.create_connection((hostname, 443)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            verify_start_time = time.time()
            context.load_verify_locations(pem_where)
            verify_total_time += time.time() - verify_start_time
            ssock.version()
            
print(f"total {verify_total_time:.4f}, avg {verify_total_time/times:.4f}")

in my environment with docker:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 3321 root      20   0  304140  81148  12792 S  42.0   0.8   0:29.69 ipython    (ubuntu22.04)
 3850 root      20   0  203348  52632  11576 S  16.7   0.5   0:06.34 ipython    (ubuntu20.04)

total 5.8634, avg 0.0586  (ubuntu22.04)
total 0.6753, avg 0.0068  (ubuntu20.04)

Your environment

  • CPython versions tested on: 3.10.5
  • Operating system and architecture: ubuntu20.04(openssl1.1) and ubuntu22.04(openssl3.0.2), build from source
  • certifi==2022.6.15
@fcfangcc fcfangcc added the type-bug An unexpected behavior, bug, or error label Jul 20, 2022
@tiran
Copy link
Member

tiran commented Jul 20, 2022

It is a problem in OpenSSL 3.0. Python upstream does not support OpenSSL 3.0 for good reasons. It has performance and backwards compatibility issue. On my system load_verify_locations is about 5 times slower when using system certificates.

3.12.0a0 (heads/main-dirty:88e4eeba25d, Jul 20 2022, 08:22:18) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
OpenSSL 3.0.5 5 Jul 2022
100 loops of 'load_verify_locations' in 3.965sec

3.12.0a0 (heads/main-dirty:88e4eeba25d, Jul 20 2022, 08:22:18) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
OpenSSL 1.1.1n  15 Mar 2022
100 loops of 'load_verify_locations' in 0.871sec

By the way you should not combine ssl.create_default_context() with certifi. A default context already loads the system cert store.

@tiran
Copy link
Member

tiran commented Jul 20, 2022 

import ssl
import sys
import time

LOOPS = 100

print(sys.version)
print(ssl.OPENSSL_VERSION)

ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)

start = time.monotonic()
for i in range(LOOPS):
    ctx.load_verify_locations('/etc/pki/tls/cert.pem')
dur = time.monotonic() - start
print(f"{LOOPS} loops of 'load_verify_locations' in {dur:0.3f}sec")

@fcfangcc
Copy link
Author

Thansk.Example is separate from requests,httpx......
It seems that using ubuntu22 with Python is not a good idea, it`s default openssl3.

@tiran
Copy link
Member

tiran commented Jul 20, 2022

I recommend that you raise a bug with OpenSSL. Their SSL_CTX_set_default_verify_paths and SSL_CTX_load_verify_locations functions are much slower in 3.0 than in 1.1.1.

@tiran
Copy link
Member

tiran commented Jul 20, 2022

According to "perf', OpenSSL 3.0 is spending a lot of time in pthread_rwlock lock/unlock followed by sa_doall, getrn, and several libcrypto string functions (ossl_lh_strcasehash, ossl_tolower, OPENSSL_strcasecmp, OPENSSL_sk_value).

@arhadthedev
Copy link
Member

a lot of time in pthread_rwlock lock/unlock

Fortunately, the issue is known (so no need to report it once more): openssl/openssl#16791 initial report and openssl/openssl#18814 pointing to a root issue.

@tiran tiran changed the title ssl module(load_verify_locations) with openssl3 there are performance problems OpenSSL 3.0 performance issue: SSLContext.set_default_verify_paths / load_verify_locations about 5x slower Jul 20, 2022
@tiran tiran added the performance Performance or resource usage label Jul 20, 2022
@iritkatriel
Copy link
Member

Should we close this as a third party issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.12 performance Performance or resource usage type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tiran @iritkatriel @arhadthedev @fcfangcc