Bash script to plot openssl speed results.
openssl speed
results (easily)On Debian/Ubuntu:
sudo apt install gnuplot git openssl make gcc gcc-mingw-w64-x86-64 cmake ninja autoconf
openssl
is for using the openssl command in the PATH.make gcc
are for makingopenssl
commands from the source code.gcc-mingw-w64-x86-64
is for making openssl.exe with MinGW.cmake ninja
are for buildingoqsprovider
andliboqs
.autoconf
is for makingconfigure
fromconfigure.ac
forLibreSSL
git repo.
On macOS
Command Line Tools by entering on a terminal a command it provides, such as
gcc
Install Homebrew, then
brew install gnuplot coreutils mingw-w64 cmake ninja autoconf
coreutils
is needed to userealpath
command.mingw-w64
is for buildingopenssl.exe
withMinGW
.cmake
andninja
are for buildingoqsprovider
,liboqs
and so on.autoconf
is for makingconfigure
fromconfigure.ac
forLibreSSL
git repo.
If Zsh causes a problem, try changing to Bash
chsh -s /bin/bash
Download scripts:
git clone https://github.com/KazKobara/plot_openssl_speed.git
cd plot_openssl_speed
Help and usage:
./plot_openssl_speed_all.sh -h
./plot_openssl_speed.sh -h
Flowcharts of these scripts.
To compare with PQC data on Web:
Install Node.js and run:
cd ./data_from_web/
npm install --save puppeteer
npx @puppeteer/browsers install chrome@stable
cd ..
then, run the following scripts.
openssl speed
with openssl command in PATH./plot_openssl_speed_all.sh -s 1
- The option ‘
-s 1
’ is to set the measuring time to 1 second to speed up and grab the rough trend. Remove it for accurate measurements.- The following graphs are obtained without ‘
-s 1
’.- The script ignores ‘
-s 1
’ against LibreSSL since itsopenssl speed
does not support-seconds
option and causes an error at least at 2.8.3.
The measurement results (graph files *.png
and their data files *.dat
) are stored in the directories displayed at the end of the output message as follows:
Results are in:
./tmp/default_openssl_1.1.1f/graphs/
For WSL (Windows Subsystem for Linux),
/home/
directory ofUbuntu-20.04
is accessible using File Explore on Windows OS with the following address:
\\wsl$\Ubuntu-20.04\home\
Example of graph list (openssl 1.1.1f in PATH):
Example of graph list (LibreSSL 2.8.3 in PATH):
The following command graphs the speed of openssl command compiled from the source code taged as openssl-3.0.7
, and openssl.exe command cross-compiled by MinGW (x86_64-w64-mingw32-gcc):
./plot_openssl_speed_all.sh -s 1 openssl-3.0.7 openssl-3.0.7-mingw
- By adding
-mingw
after the tag-name, openssl.exe is cross-compiled by Mingw-w64, and then the results are added on WSL. The other computational environment requires Windows binary executable environment.- openssl-3.0.5, shown as an example below, includes vulnerabilities. So use a fixed or latest OpenSSL (or its alternative).
Example of graph list (openssl-3.0.5 from source):
If the openssl command is with liboqs and oqs-provider, plot_openssl_speed_all.sh
v1.0.0 and newer automatically measure and depict the speed of post-quantum algorithms, too. It also provides comparison graphs of sizes and processing cycles published on [[pq-sig-zoo]] and [[ebats]].
Give the command argument openssl-type
as follows,
to measure and depict the speed of openssl with oqs-provider and liboqs tagged by openssl-3.3.1
, 0.6.1
, 0.10.1
, respectively.
./plot_openssl_speed_all.sh -s 1 openssl-3.3.1-oqsprovider0.6.1-liboqs0.10.1
For their master/main branches:
./plot_openssl_speed_all.sh -s 1 master-oqsprovidermain-liboqsmain
As of v1.0.0,
plot_openssl_speed_all.sh
does not acceptopenssl-type
combiningliboqs<tag>-oqsprovider<tag>
with-mingw
.
The processing speed may vary depending on the environment. The above and the following graphs show the results in the following computational environment.
Be careful not to use broken or insufficient-security-level algorithms even if they are faster than the other.
The followings graphs show the processing speeds of PQC’s available at OpenSSL 3.4.0-alpha1 with oqs-provider 0.6.1 and liboqs 0.11.0-rc1.
Digital signature:
KEM (Key Encapsulation Mechanism):
with_webdata.sh depicts ciphertext/signature sizes vs. public-key/signature-verification-key sizes as scatter graphs by collecting the necessary size data from [[pq-sig-zoo]] and [[ebats]] and then combining them with other provided data.
Comparison among 128-(classical-)bit security algorithms:
Comparison among 192-(classical-)bit security algorithms:
Comparison among 256-(classical-)bit security algorithms:
One operation of ECDH includes one scalar multiplication to the generator/base-point and one scalar multiplication to a random point.
Comparison among 128-(classical-)bit security level algorithms:
Comparison among 192-(classical-)bit security level algorithms:
Comparison among 256-(classical-)bit security level algorithms:
Theoretically, the larger the size, the slower the processing speed, but in practice, some counterexamples exist.
The first examples show the former cases.
The parameter a
in the line graphs denotes the reduction rate of the processing speed when the size becomes twice where the size is the bit length of the underlying finite field or ring.
RSA:
ECDH (NIST curve over an extension field of Z2):
ECDH (Brainpool r1 over a prime field):
The following figures show the counter examples.
ECDSA/ECDH (NIST curve over a prime field, OpenSSL 3.0.5 source code built and run on Ubuntu 20.04):
For OpenSSL,
256-bit is by far faster than the smaller sizes 192-bit and 224-bit.
- It does not mean that 256-bit is exceptional in theory, but the assembly implementation has tuned it up, since adding
-UECP_NISTZ256_ASM
to./config
will remove this advantage.- The processing speed of 384-bit and 521-bit may also be improved in the future depending on the necessity.
ECDSA/ECDH(NIST curve over a prime field, OpenSSL 3.3.2 of Homebrew on macOS 14.6):
ECDSA (NIST curve over a prime field, LibreSSL 2.8.3 shipped with macOS 12.4):
For LibreSSL,
the signing speeds are slower than their verification ones (from 2.8.0 to at least 3.9.2).
- In general, signing speeds are faster than their verification speeds for ECDSA and DSA, and libressl had held this propertiy up to 2.7.4 as shown in the following figure.
- 2.8.0 release note says “Added a blinding value when generating DSA and ECDSA signatures, in order to reduce the possibility of a side-channel attack leaking the private key.”
ECDSA(NIST curve over a prime field, LibreSSL 2.7.4 source code built and run on macOS 15.0):
API difference:
When the script invokes the crypt-algorithm with a low-level API, it appends -no-evp
to the crypt-algorithm name. No difference between APIs.
Low-level APIs are deprecated at OpenSSL 3.
Comparison among truncated versions:
sha512-224
, sha512-256
, and sha384
are truncated versions of sha512
, so they show almost the same performance.sha224
is a truncated version of sha256
, so they show almost the same performance, too.Comparison between sha256
and sha512
:
sha256
and sha512
. Hence, the figure indicates that one sha256
compression function is faster than that of sha512
.sha512
is faster than sha256
since the number of sha512
compression-function executions is around half of sha256
where the input bit length of sha256
and sha512
compression functions are 512-bits and 1024-bits, respectively.SHA-3:
sha3-*
.SHA-2
, SHA-3
(sha3-*
) are slower due to the larger security margin [[kec17]].In theory:
AES-*-GCM
and AES-*-CCM
must be slower than the same key-length AES-*-CTR
since they are AES-*-CTR
with their integrity check.AES-*-CCM
must also be slower than AES-*-CBC
, which is shown in the upper left corner in the above graph list, since the integrity check of AES-*-CCM
uses the similar algorithm to AES-*-CBC
.AES-128-*
must be around 1.4 times faster than AES-256-*
since the number of their rounds are 10 and 14, respectively.Counter example:
aes-(128|256)-gcm
and aes-(128|256)-gcm
is by far faster than the others for large-size inputs as follows.LibreSSL 2.8.3 (shipped with macOS 12.4):
LibreSSL 3.3.6 (shipped with macOS 14.6):
LibreSSL 3.9.2 (from the source code):
As shown on the right side of aes128-cbc.png
, aes-128-cbc-no-evp
(128-bit key AES with the legacy mode of operation, CBC) called by way of the low-level API is slower in OpenSSL 1.1.1 than the high-level API and OpenSSL 3.0.5.
- LibreSSL (at least up to 3.9.2) shows the sililar results as OpenSSL 1.
- Low-level APIs are deprecated at OpenSSL 3.
Edit the crypt-algorithm names and the PNG file names in the functions plot_graph_asymmetric()
and plot_graph_symmetric()
in
./plot_openssl_speed_all.sh
whereas the former function is for asymmetric-key cryptosystems and the latter is for symmetric-key cryptosystems.
For example, the follows line saves a graph of measurement results of all the supported eddsa
and ecdsa
digital signatures in ed_ecdsa.png
, its plot data in ed_ecdsa.dat
, and their measurement logs in eddsa.log
and ecdsa.log
, respectively:
${PLOT_SCRIPT} -o "./${GRA_DIR}/ed_ecdsa.png" eddsa ecdsa
As you can see from the above, plot_openssl_speed_all.sh
is a wrapper of ${PLOT_SCRIPT}
(plot_openssl_speed.sh
), and you can directly run the openssl command in the PATH as follows:
./plot_openssl_speed.sh -o "./tmp/default_openssl_1.1.1f/graphs/ed_ecdsa.png" eddsa ecdsa
where:
- ‘_1.1.1f’ to be changed to the version of the openssl command in PATH.
- cf.
./plot_openssl_speed.sh -h
for the usage.
One can specify the openssl command, which is not in the PATH, with -p
option as follows:
./plot_openssl_speed.sh -p "./tmp/openssl-3.0.7/apps/openssl" -o "./tmp/openssl-3.0.7/graphs/ed_ecdsa.png" eddsa ecdsa
If it encounters the following errors:
error while loading shared libraries:
symbol lookup error:
add the path to the shared library to LD_LIBRARY_PATH
(DYLD_LIBRARY_PATH
for macOS) as follows:
(export LD_LIBRARY_PATH=./tmp/openssl-3.0.7${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}; ./plot_openssl_speed.sh -p "./tmp/openssl-3.0.7/apps/openssl" -o "./tmp/openssl-3.0.7/graphs/ed_ecdsa.png" eddsa ecdsa)
The command ldd
(or otool -L
on macOS) shows a list of used shared libraries.
$ ldd ./tmp/openssl-3.0.7/apps/openssl
libssl.so.3 => not found
libcrypto.so.3 => not found
To run the openssl command under a working folder ${TMP}/<openssl-type>
where <openssl-type>
includes <oqsprovider_type>
:
Run the following script once right under the ${TMP}/<openssl-type>
folder:
../../utils/set_oqsprovider.sh
Then, run ../../plot_openssl_speed.sh -p openssl/apps/openssl
in the same folder as follows:
../../plot_openssl_speed.sh -p openssl/apps/openssl -s 1 -o mldsa44_and_mlkem512.png mldsa44 mlkem512
The above scripts also save data, corresponding to the graphs, in files that replaced .png with .dat in the PNG file names. You can create a new data file by combining the contents of them or by editing them.
You can plot the graph of the edited data file by specifying no crypt-algorithms to the argument (without running openssl speed
):
./plot_openssl_speed.sh -d "data_file_to_graph" -o "output_graph_file.png"
The data file name is given by any of the following ways:
-d
option-d
option:
-o "output_graph_file.png"
is given, output_graph_file.dat
is used as the data file.graph.dat
is used as the data file regarding that the default file graph.png
is given to -o
.You can find the default file names in the ‘Usage’ shown by:
plot_openssl_speed.sh -h
If
data_file_to_graph
is different fromoutput_graph_file.dat
, which is the file name replaced.png
with.dat
inoutput_graph_file.png
, thedata_file_to_graph
is copied to theoutput_graph_file.dat
so that anyone can know thatoutput_graph_file.dat
is the data file ofoutput_graph_file.png
.
openssl speed
outputs some of the crypt-algorithm names in multiple columns, the data file generated by plot_openssl_speed.sh
automatically aligns them only in the first column.
plot_openssl_speed.sh
ignores and skips consecutive different TABLE_TYPE crypt-algorithms.
For symmetric-key cryptographies, hash functions, HMACs.
Example:
# type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-ccm 202973.97k 588256.58k 1065011.71k 1314283.52k 1346633.73k 1381728.26k
hmac(sha512) 23408.12k 90165.99k 249721.98k 538953.37k 756375.73k 782985.02k
sha256 30840.78k 88357.72k 199311.27k 292801.60k 334301.56k 319321.27k
This TABLE_TYPE is available in v1.0.0 and newer for digital signatures.
Fill keygen/s
columns of no data with 0
.
In data-plot mode,
plot_openssl_speed.sh
with-l TABLE_TYPE
option (available in v1.0.0 or newer) accepts the right most consecutive blank columns while without the option it guesses the TABLE_TYPE with the number of columns.
Example:
# asymmetric_algorithm sign/s verify/s keygen/s
ecdsa(nistp256) 42359.0 15555.0 0
EdDSA(Ed25519) 29607.0 9474.0 0
rsa3072 553.0 28153.0 0
rsa4096 268.0 16543.0 0
mldsa44 14595.0 40081.0 30223.2
This TABLE_TYPE is available in v1.0.0 or newer for DH key exchanges and
symmetric-key cryptographies except digital signatures.
Fill the columns of no data with 0
.
In data-plot mode,
plot_openssl_speed.sh
with-l TABLE_TYPE
option (available in v1.0.0 or newer) accepts the right most consecutive blank columns while without the option it guesses the TABLE_TYPE with the number of columns.
Example:
# asymmetric_algorithm dec/s enc/s keygen/s dh/s
rsa3072 593.0 26836.0 0 0
rsa4096 254.0 15503.0 0 0
ecdh(nistp256) 0 0 0 19963.0
ecdh(X25519) 0 0 0 30287.0
mlkem512 114039.0 100366.0 77138.0 0
This TABLE_TYPE is available in v1.0.0 and newer for comparison among “sig_ver_keygen” and “dec_enc_keygen_dh” algorithms.
Example:
# asymmetric_algorithm sign/s verify/s keygen/s
mldsa44 14595.0 40081.0 30223.2
mldsa65 8686.0 22795.0 19578.0
mldsa87 7104.0 14383.0 12445.5
#
# asymmetric_algorithm dec/s enc/s keygen/s dh/s
mlkem512 114039.0 100366.0 77138.0 0
mlkem768 73753.0 71901.0 52051.5 0
mlkem1024 49803.0 52377.8 43771.7 0
For v0.0.0 and digital signatures. It graphs the values in the fourth and fifth columns.
Plotting processing times has the following drawbacks: It is hard to distinguish the fastest algorithm when the comparison set includes slow algorithms. Even worse, small processing times are sometimes quantized to 0 or the minimal unit.
Example:
# sign verify sign/s verify/s
rsa4096 0.003922s 0.000061s 255.0 16471.0
dsa2048 0.000296s 0.000219s 3383.0 4557.0
ecdsa(nistp256) 0.0000s 0.0001s 43201.0 15221.0
EdDSA(Ed25519) 0.0000s 0.0001s 24010.0 8805.0
For v0.0.0 and Diffie-Hellman key exchange. It graphs the values in the third column.
Example:
# op op/s
ffdh4096 0.0129s 77.8
ecdh(nistp256) 0.0000s 20643.0
./data_from_web/with_webdata.sh -d <filename>.dat
command reads this data file where:
Example:
# x min 25% 50% 75% max (k|s|v|d|e):name(source) parameter
# x dummy 25% 50% 75% dummy (k|s|v|d|e):'25/50/75% are given' parameter
# x dummy dummy mean dummy dummy (k|s|v|d|e):'only the mean is given' parameter
1 0 76090 86116 134869 134869 s:dilithium2aes(ebats-ryzen7) 2
2 0 42770 42987 43140 43140 v:dilithium2aes(ebats-ryzen7) 2
3 0 33094 33541 33924 33924 k:dilithium2aes(ebats-ryzen7) 2
4 0 74124 74124 74124 74124 s:mldsa44(liboqs0.10.1) 44
5 0 31584.7 31584.7 31584.7 31584.7 v:mldsa44(liboqs0.10.1) 44
6 0 35220.3 35220.3 35220.3 35220.3 k:mldsa44(liboqs0.10.1) 44
7 0 333013 333013 333013 333013 s:ML-DSA-44(pq-sig-zoo) 44
8 0 118412 118412 118412 118412 v:ML-DSA-44(pq-sig-zoo) 44
Each column shows:
x-axis
to plot the data in the line.min
, 25%
, 50%/mean
, 75%
, max
values, respectively.25%
, 50%
, 75%
values. Set them there and put dummy values to min and max.50%/mean
and put dummy values in the other columns.k:
, s:
, v:
, e:
, d:
in the name represent the operation type of keygen, sign, verification, encryption/encapsulation, decryption/decapsulation, respectively.()
represents the data source.with_webdata.sh
does not use it as of v1.0.0. So, you may put a dummy value (but not blank) here.$ awk '/^PRETTY/ {print substr($0,14,length($0)-14)}' /etc/os-release
"Ubuntu 20.04.4 LTS"
$ uname -srm
Linux 5.10.102.1-microsoft-standard-WSL2 x86_64
$ awk '$1$2 == "modelname" {$1="";$2="";$3=""; print substr($0,4); exit;}' /proc/cpuinfo
Intel(R) Core(TM) i7-10810U CPU @ 1.10GHz
Version and configurations of the openssl command in the PATH:
$ openssl version -a
OpenSSL 1.1.1f 31 Mar 2020
built on: Mon Jul 4 11:24:28 2022 UTC
platform: debian-amd64
options: bn(64,64) rc4(16x,int) des(int) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-51ig8V/openssl-1.1.1f=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
Version and configurations of openssl-3.0.5:
$ (export LD_LIBRARY_PATH=./tmp/openssl-3.0.5${LD_LIBRARY_PATH:+:$LD_LIBRARY
_PATH}; ./tmp/openssl-3.0.5/apps/openssl version -a )
OpenSSL 3.0.5 5 Jul 2022 (Library: OpenSSL 3.0.5 5 Jul 2022)
built on: Wed Jul 13 10:43:30 2022 UTC
platform: linux-x86_64
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -fstack-protector-strong -fstack-clash-protection -fcf-protection -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
$ gnuplot -V
gnuplot 5.2 patchlevel 8
$ uname -srm
Darwin 21.5.0 x86_64
$ sysctl machdep.cpu.brand_string
machdep.cpu.brand_string: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
$ openssl version -a
LibreSSL 2.8.3
options: bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
$ gnuplot -V
gnuplot 5.4 patchlevel 3
Either add the folder of libssp-0.dll
to the Windows environment PATH, or run the following commands on a WSL Debian/Ubuntu terminal:
sudo apt install gcc-mingw-w64-x86-64
bash
export MINGW_GCC_VER=$(/usr/bin/x86_64-w64-mingw32-gcc-posix --version | awk '/x86_64-w64-mingw32-gcc-posix/ {print substr($3,1,index($3,"-")-1)}')
cp -p "/usr/lib/gcc/x86_64-w64-mingw32/${MINGW_GCC_VER}-posix/libssp-0.dll" .
exit
Change the options and/or crypt-algorithms given to openssl speed
.
Some versions of openssl commands do not support them.
Check if your security software displays a message that blocks the execution. If so, unblock it and run the same script again.
[kec17]: https://keccak.team/2017/is_sha3_slow.html (TeamKeccak “Is SHA-3 slow?”) [pq-sig-zoo]: https://pqshield.github.io/nist-sigs-zoo/ “Post-Quantum signatures zoo” [ebats]: https://bench.cr.yp.to/ebats.html “eBATS: ECRYPT Benchmarking of Asymmetric Systems”