Decryption-Decompression of AES Protected ZIP ...

Decryption-Decompression of AES Protected ZIP Files on GPUs Tan Nhat Duong High Performance Computing Center Hanoi University of Technology Hanoi, Vietnam [email protected] Duc Huu Nguyen Department of Information Systems Hanoi University of Technology Hanoi, Vietnam [email protected]

Phong Hong Pham Department of Information Systems Hanoi University of Technology Hanoi, Vietnam [email protected]

Thuy Thanh Nguyen Department of Information Systems Hanoi University of Technology Hanoi, Vietnam [email protected]

Abstract—AES is a strong encryption system, so decryptiondecompression of AES encrypted ZIP files requires very large computing power and techniques of reducing the password space. This makes implementations of techniques on common computing system not practical. In [1], we reduced the original very large password search space to a much smaller one which surely containing the correct password. Based on reduced set of passwords, in this paper, we parallel decryption, decompression and plain text recognition for encrypted ZIP files by using CUDA computing technology on graphics cards GeForce GTX295 of NVIDIA, to find out the correct password. The experimental results have shown that the speed of decrypting, decompressing, recognizing plain text and finding out the original password increases about from 45 to 180 times (depends on the number of GPUs) compared to sequential execution on the Intel Core 2 Quad Q8400 2.66 GHz. These results have demonstrated the potential applicability of GPUs in this cryptanalysis field.

Dung Duc Phan High Performance Computing Center Hanoi University of Technology Hanoi, Vietnam [email protected]

exhaustive attack is almost impossible. AES [4], [5] Advanced Encryption Standard, is a block cipher. AES works with 128-bit data blocks (4x4 bytes) with the key length is 128, 192 or 256 bits. AES can easily be implemented at high speed by software or hardware and does not require much more memory. AES with the key size of 128-bit/256-bit, there are up to 2128/2256 cases in the key space for testing to find out the original password. Directly attacking on the key space is not an optimal solution for such strong encryption systems.

Keywords-cuda; gpu; decryption; decompression; password; zip.

I.

INTRODUCTION

The technique of protecting a ZIP file includes stages: compressing the original document and then encrypting the compressed document by a password. The decryption process completely reverses. For the convenience, an encryption key is generated from the sender's password by a hash function. This key is used to encrypt the document. After that, the password is transferred to the recipient via a secure channel and used to generate the same key to decrypt the document. The process of encrypting and decrypting an encrypted ZIP file is shown in Figure 1. Our research tries to recover the content of an encrypted ZIP file without knowing its protection password. In fact, for weak encryption systems such as RC4 or DES, cryptanalysis can be conducted by an exhaustive attack on the whole key space in an acceptable time. The results of [2], [3] have proved this method. For strong encryption systems such as AES which is commonly used in new versions of WinZip (9.0 or higher), an

Figure 1. ZIP file encryption and decryption process

Meanwhile, attacking based on dictionary [6] is a type of attacking on the password space instead of the key space. However, it won’t be useful if there is not any method of reducing the large password space. We do not directly attack on AES key space. Instead, we found that the encryption key of a protected ZIP file is generated from a user password by a hash function which is published in the ZIP file specification [7]. So our strategy is to attack on password space. Although password space is also large but attacking on password space is much more feasible than attacking on AES key space, since dictionary attack methods can be applied. The major obstacle of this cryptanalysis method is that the computational complexity of the hash function is quite high. This is even more difficult, because password salting techniques implemented in new versions of ZIP tools prevent us from using pre-computed attack methods.

To overcome these difficulties, in [1] we first employ the recent password structure analysis method of Weir, Aggarwal, Medeiros, Glodek [8] to reduce the size of the password search space and take advantage of friendly usage features of compression tools which allow to detect incorrect passwords rapidly based on a two-byte password verification value (PVV) stored in the header of the ZIP file. We used the extremely great computing power of modern multi-core GPUs for implementing the complex hash function to concurrently verify passwords from the password search space based on the value PVV and to generate AES encryption keys for all possiblycorrect passwords (ones apart from incorrect passwords), we call them candidate passwords. To find out the original password of the ZIP file from the set of candidate passwords, we have to perform reverse processes – decryption, decompression and plain text recognition on the encrypted ZIP file for all elements of the set of candidate passwords. In this paper, we completely perform the password recovery for encrypted ZIP archives by implementing the processes: decryption, decompression and plain text recognition in parallel on the set of candidate passwords, using graphics cards GPUs. However, since specifications of these techniques are independent and their implementations are closed, so it is very difficult to implement these techniques and then connecting their implementations into a continuous process on GPUs in our method. In the rest of this paper, we briefly introduce GPU and the CUDA technology for general-purpose applications and describe in details our parallel implementation of decryption– decompression, plaintext recognition process for the encrypted ZIP file to find out the correct password from the set of candidate passwords, together with experimental results. II.

CUDA AND GPGPU

In recent years, computing power of graphics processors (GPU) has increased significantly compared to CPU. Until June 2008, NVIDIA's GPU GT200 generation has reached the threshold of 933GFLOPS, more than 10 times over dual-core processor the Intel Xeon 3.2 GHz at the same time. Figure 2 shows a massive increase in computing power of the NVIDIA graphics processors compared to Intel processors. This superiority in performance does not imply the superiority in technology. GPU and CPU are developed in two different directions: while CPU technology speeds up a single task, GPU technology tries to increase the number of tasks that can be performed in parallel. Thus, while the number of cores in common CPUs has not reached 8 cores yet, the number of cores in single GPU has reached 240 and also promises to continue to increase to 500 cores in 2010. As a penalty for the computing power, GPUs lose the flexibility of processing cores. Currently, all processing cores on one single GPU can only execute a single piece of code at a time, so GPU is only suitable for data parallel problems, in which the same program code will be executed in parallel for several different data sets. Fortunately, most problems that require large computing power can be converted to a type of data parallelism. Beside the effort of improving GPU computing power, GPU manufacturers are also interested in providing better

application development environments for common developers to easily program on GPUs. NVIDIA CUDA [9] is a good example of such effort. With CUDA, programmers can exploit GPU computing power for not only graphics processing applications but also general-purpose applications. This technology is one of the important factors for the opening of the recent GP-GPU (General-Purpose computation on Graphics Processing Units) era. The followings are some key features of the programming language supported by CUDA (called CUDA language): 

CUDA language is an extension of C language, so familiar to most developers. CUDA code is divided into two parts: one executed on CPU and the other executed on GPU. The part executed on GPU, also known as parallel kernel, when called, can be executed in parallel on thousands of execution threads. Each thread has a unique identifier used to determine its task.

Figure 2. NVIDIA GPU-Intel CPU performance comparison





CUDA allows programmers to define an arbitrary number of parallel threads, but to avoid the dependence on hardware, threads are divided into blocks with the number not exceeding 768(GT200 generation). This allows a programmer to design his parallel program effectively without caring about the hardware capability.

Memory is hierarchically organized for effective usage. - Main memory: the memory area for CPU code. Only this code can access and modify information here. - Global memory: the memory area that all GPU threads can access to it. Programmers can move data from main memory to global memory by using functions from a CUDA basic library. This memory is often used to store inputs and outputs for parallel threads on GPUs. - Shared memory: the memory area that only threads in one block can access. This memory is integrated on-chip; therefore, the speed of accessing data on it is much higher than on global memory. This memory is often used to store temporary shared data among threads in a block to speed up the process of memory usage.

- Local memory: the memory area allocated to local variables of each thread and one GPU thread cannot access to those from others. With the ability to perform data parallelism on such a lot of threads, GPU is an appropriate choice to the problem of ZIP file cryptanalysis, where each thread takes one password from the password search space to verify and decrypt, decompress, recognize plain text for the encrypted ZIP file if it is a candidate password. The next section of the paper explains in details our GPU-based parallel algorithm using CUDA for protected ZIP files. III.

PARALLELING DECRYPTION-DECOMPRESSION OF AES ENCRYPTED ZIP FILES ON GPUS

As introduced above, our approach in ZIP file cryptanalysis is to attack on the password space, instead of directly attacking on the AES key space. The whole password recovery algorithm is divided into three main steps: 1. Construct the password search space. This process can be performed by some methods such as: generating exhaustively passwords by the password length and a set of input characters, or applying the password structure analysis algorithm from [8] for reducing the size of password search space. 2. Exploit the computing power of GPUs for accelerating the preliminary password checking process. The result of this process is a set of possibly-correct passwords (called the set of candidate passwords) whose size is much smaller than the size of password search space obtained in the step one. 3. Implement decryption, compression and plaintext recognition techniques for verifying each candidate password on GPUs to find out one correct password of the ZIP file. In [1], we solved step one, step two of the algorithm and obtained the set of candidate passwords. Here, we present our parallel algorithm in step three - decryption, compression and plaintext recognition for the set of candidate passwords on GPUs. A. Strategy According to the specification of the ZIP file [7], the process of checking one password consists of the following steps: 1. Generate an AES key from a given password by using the hash function described in the specification - PBKDF2(pw, salt, dkLen) where pw is the given password to check, salt is a random value stored in the compressed file, and dkLen is the AES key size. The function PBKDF2 has a large computational complexity. In fact, this function performs the HMAC-SHA1 algorithm for 1,000 times, thus preventing attacks from common computing systems. The random

value salt is used to prevent from pre-computed attacks. 2.

Decrypt and decompress the encrypted ZIP file using the obtained AES key in the previous step. This algorithm also generates a checking value which will be compared to a MAC (message authentication code) value stored in the archive to decide whether the password is correct or not.

With such the specification, for each of passwords in the password space if we fully implement these steps, the time needed to check the entire password space is very large because decrypting and decompressing the entire content of the encrypted ZIP file is extremely expensive. Instead, in step two we only apply decryption and decompression techniques for a small part of the ZIP file, and then use a plaintext recognition technique proposed by us to quickly check the validation of the AES key for identifying whether that password is the original password or not. As mentioned above, another strategy we have considered in [1] is to take advantage of friendly usage features of compression tools. For example, WinZip allows detecting incorrect passwords rapidly by storing a two-byte password verification value (PVV) in the header of the ZIP file. This value will be compared to a part of the output of the function PBKDF2 (this output value is called TestPVV) for quickly rejecting most incorrect passwords. If a password is accepted, it is not yet guaranteed that the password is correct. However, the number of passwords which can be accepted is significantly smaller than the size of initial password space. We call them candidate passwords. Since the execution of the hash function PBKDF2 takes the main work load of the checking process and it is implemented for all passwords, so our strategy is to implement this hash function on GPUs to effectively check passwords in the given password space in parallel. Assuming that the input password space includes n passwords. In theory, n passwords can be checked - to confirm whether each password is a candidate password or not – at the same time, by calling p (p = n) corresponding to GPU threads. However, this number of threads p is limited by hardware resources, usually p is much smaller than n. Therefore, to check n passwords, we need to sequentially call (n/p) times, each time a batch of p passwords is fed to be checked by p threads in parallel, thus, the password search space should be divided into the corresponding batches. Figure 3 describes such inspection of passwords.

Figure 3. Checking passwords in parallel

Naive idea is implementing the process of finding out the original password (decryption, decompression and plaintext

recognition) after ending the step of obtaining the set of all candidate passwords. However, because the size of the original password space is often enormous together with the high computational complexity of the hash function , so the time for reducing it to the set of candidate passwords is also large. On the other hand, the number of candidate passwords after completing of checking a batch of passwords in parallel is very small, so it is better than to implement the techniques of finding out the original password together with the process of checking candidate password within the same executing thread. This method has an important advantage that if the correct password belongs to one of batches which is early checked, then the program will be immediately stopped without checking the remaining passwords. This significantly reduces the time for recovering password. If any password is a candidate, then the thread of generating that password will use the AES key (generated together with the value TestPVV by the function PBKDF2) for decrypting. As mentioned above, decrypting the entire encrypted ZIP document is not efficient, so we only decrypt a small part of the document, whose size is about a few KBs, then decompress and use the plain text recognition technique based on files headers to determine the original correct password. Next sections will introduce decryption algorithm, decompression algorithm, plaintext recognition technique and our parallelism of these algorithms on GPUs to recover password.

value is encrypted by using the AES block encryption algorithm (the function F in the figure) and the output is X-i blocks. And then blocks M-i, X-i are performed XOR together to create cipher text blocks C-i . Here CTR-i, M-i and X-i have the same size BLOCK_SIZE (16 bytes). Because the XOR operator is the reverse operator of itself, so the decryption process is completely similar to the encryption process. With the AES key, we encrypt the value counter and then performing the XOR operator between the encrypted counter value and the ciphertext to obtain the plaintext. That is the idea of the decryption process. If a thread checks any password as a candidate, it will continue performing the process of decrypting data which read from the ZIP file (encrypted-compressed data or cipher text) by using the function Aes_ctr constructed by us. This function takes the AES key as the input. Data which read from the encrypted ZIP document has the length CHUNK about a few KBs. This function encrypts an increasing counter by the AES algorithm, and then does the XOR operation on this result and data for decrypting. The output of this function is a bytessequence data_compressed or plaintext with the length CHUNK, which is decrypted. The decryption function Aes_ctr is constructed by us as follows: len = 0; /* the size of the data block in AES (16 bytes) */ pos = BLOCK_SIZE; counter = 0; while (len < CHUNK) { /* process each block BLOCK_SIZE */ if (pos == BLOCK_SIZE) { /* increase the counter */ counter++; /* encrypt the counter by AES */ enc_buf = aes_encrypt(AesKey, counter); pos = 0; } /* perform the XOR operation on buffer and encrypted data*/ data_compressed[len++]= data_compressed_encrypted[len++]^encr_buf[pos++];

B. Decryption Algorithm, Decompression Algorithm and Plaintext Recognition Technique 1) Decryption Algorithm WinZip encrypts documents after compressing and only applies encryption for the content of the document without combining with any other data. Documents are encrypted in bytes, using the AES algorithm operating in “CTR” mode, which means the length of compressed data and the length of encrypted-compressed data are the same. Although data is encrypted in bytes, while processing, it is executed in blocks with the size of 16 bytes according to the specification [5].

} Figure 5. Pseudo code of the decryption algorithm

2) Decompression Algorithm

Figure 4. AES CTR mode encryption

Figure 4 illustrates the CTR mode of the AES algorithm where CTR is the counter, M is the message (or plaintext). The counter is initialized with the value 0; the increasing counter

Compressed data which is obtained by applying the above decryption algorithm for the encrypted ZIP document need to be decompressed. This section presents the popular decompression algorithm of WinZip. The output of this algorithm is decompressed-decrypted data data . As default, WinZip chooses the information lossless compression method - Deflate while compressing data. The data format and the compression algorithm are given in the description of Deflate [10] and can be summarized as follows: Data after being compressed includes a sequence of blocks; each of them is compressed using a combination of the LZ77 algorithm and the Huffman code. The LZ77 algorithm uses a sliding window including two parts: search buffer and lookahead buffer to find repeated strings in the input data. Search buffer is used as a dictionary, containing processed data, and look-ahead buffer contains unencrypted data. The second time occurrence of a string will be replaced by a pointer with the

form (distance, length), pointing to the previous repeated string. Figure 6 shows an example of compressing data using LZ77, with the input data as “abracadabra” and the output compressed data in the form “abr(3,1)c(2,1)d(7,4)d”. In the representation of the deflate format, with the distance limit of 32Kbytes and the length limit of 258 bytes, if a string is not repeated in the previous 32 Kbytes, it is considered as a literal byte string. Each type of values (literals, lengths, and distances) is then encrypted by the Huffman code – an algorithm constructing an optimal set of binary code for an alphabet code, based on the occurrence frequency table of the encryption-needed characters. The three sets of alphabet codes used for compressed data are the set of length (3…258), the set of distance (1...32768) and the set of literal (0...255). In fact, the set of length and the set of literal are combined into an unique set of alphabet code: lit/length (0...285). Here, the values of (0…255) represent literals, the value of 256 represents end-of-block, the values from 257 to 285 can be combined with additional bits to represent length values completely. After that, each set of alphabet codes in compressed data is represented with Huffman code: using a Huffman tree for the set of literal/length codes and another one for the set of distance codes. These two Huffman trees are stored at the beginning of each block and right before the compressed data of that block, and they are independent between blocks.



Blocks are compressed by dynamic Huffman code, first by LZ77, then by Huffman code. These two trees are generated and stored with data.

In all cases, the decompression algorithm has the following form: do read block header from input stream; if stored with no compression { skip any remaining bits in current partially processed byte; read LEN and NLEN; copy LEN bytes of data to output; } else if compressed with dynamic Huffman codes read representation of code trees; loop (until the end of block code recognized) decode literal/length value from input stream; if value < 256 copy value (literal byte) to output stream; else if value = end of block (256) break from loop; else (value = 257..285) decode distance from input stream; move backwards distance bytes in the output stream, and copy length bytes from this position to the output stream;. end loop while not last block Figure 7. Pseudo code of the decompression algorithm

We have deployed the decompression function inflate (data_compressed, len) provided in the Zlib [11] library and implement it on GPUs. This function decompresses each block of compressed data data_compressed obtained by the decryption algorithm with the length len. The output of this algorithm is decompressed-decrypted data data which is used for the next technique – plain text recognition to identify the correct password. 3) Plain Text Recognition by Header

Figure 6. An example of compressing data by LZ77

Each block of the compressed data belongs to one of the three following formats; this has been defined based on the header bits at the beginning of each block:  Non-compressed blocks From the second byte, blocks have the following forms: 2 byte: LEN - the number of data bytes in the blocks. 2 byte: NLEN - the one’s complement of LEN. LEN byte: non-compressed data.  Blocks are compressed by static Huffman code, first by LZ77 then by Huffman code. The two Huffman trees for two alphabets are fixed and defined in the description of deflate. There is no need to store them in blocks.

In this paper, we propose the idea of recognizing plain text which is very simple and applied in some common formats such as PDF, DOC. These formats have defined headers that can be collected to represent its document format (e.g. PDF headers always have the form “%PDF-1.x” with x is the version number 0, 1, …, 7). On the other hand, in the header of the ZIP file, there is also the filename field which allows us to identify the format of the original document. When obtaining the original format, we will compare the header of data which is decrypted, decompressed – data with the corresponding header of that format. If they are the same, we can conclude that data belongs to the original document and the generated password by the corresponding thread is the correct password. C. Implementing Algorithms in Parallel on GPUs Based on descriptions of the algorithms described above, we perform parallelism for the algorithms on GPUs to find out the correct password. During the process of executing of a thread, the following functions are performed: GenPW – generate a password based on the id of the thread in blocks, PBKDF2 – the hash function. If the password is a candidate, the executing thread will call the functions decryption, decompression and plain text recognition to decide whether

the password is correct or not. The following is pseudo code of the parallel algorithm: /*******************ASSUMPTION********************** l - the number of password batches is sequentially executed or the number of calling the kernel in parallel; p - the number of threads executed in parallel; salt - a random value stored in the compressed file; dkLen – the AES key size; id – the identifier of the thread; ***************************************************/ for j = 0 to l – 1 { for id = 0 to p-1 in parallel { /* gerenate a password based on the id of the thread */ guess_password = GenPW(id); (TestPVV, AesKey) = PBKDF2(guess_password ,salt, dkLen); if (TestPVV == PVV) { /* decrypt the encrypted compressed data with the length CHUNK, read from the compressed document using the AES key */ data_compressed = Aes_ctr(AesKey, data_compressed_encrypted, CHUNK); /* decompress the decrypted compressed data */ data = inflate (data_compressed, CHUNK); /* recognize plain text based on the extension of the original document and some headers of common formats */ boolean ok = 0; switch (extension) { case PDF: if (header(data) == headerPDF) ok = 1; break; case DOC: if (header(data) == headerDOC) ok = 1; break; case TXT: if (data has character text) ok = 1; break; } if (ok) { // the correct password markPassword(guess_password); exit 0; // end the program } } } Figure 8. Pseudo code of implementing algorithms in parallel on GPUs

After generating one password – guess_password, the thread executes the function PBKDF2 to generate a value AesKey and a value TestPVV. This value TestPVV will be compared with the password verification value (PVV) stored in the header of the ZIP file to verify. If it is a candidate password, the thread continues calling the function Aes_ctr using the value AesKey to decrypt. The output of the decryption function Aes_ctr is compressed data which decrypted - data_compressed. Using this compressed data, the thread continues calling the function inflate to decompress, returning data which is decrypted, decompressed. Based on this data, the plain text recognition is performed, using headers of some common formats, to verify if it is the correct password. When the original password is found out, all the processes are immediately ended.

IV.

EXPERIMENTATION RESULTS

The algorithms of verifying candidate passwords, decryption, decompression and plain text recognition on GPUs are implemented and experimented at High Performance Computing Center, Hanoi University of Technology. The experimentation environment includes one PC equipped with the following items: - The Intel Core 2 Quad Q8400 2.66 GHz. - 8 GB RAM - Two dual graphics cards NVIDIA GeForce GTX 295 (total of 4 GPUs) - OS: Cent OS 5.3. We evaluated the performance of this algorithm implementation by two aspects:  The number of keys AesKey generated (equal to the number of candidate passwords which is generated).  The speed of recovering the original password of encrypted ZIP files (after performing the algorithms decryption, decompression and plain text recognition in parallel) on GPUs. Here we applied the exhaustive approach to demonstrate superior computing power of GPGPU technology in the problem of recovering ZIP file password. This experiment uses the set of upper, lower and numeric characters S = {a-z, A-Z, 0-9} and the optimal number of threads can run in parallel p = 32,768. The result showed that the speed of generating AES keys on GPU increases from 45 to 180 times compared to that on the sequential CPU-based program (approximately 45 times on single GPU, and 180 times on four GPUs). Table 1 demonstrates this result. TABLE I.

COMPARISON OF SPEED OF GENERATING KEYS ON CPU/GPUS

The number of CPU/GPUs 1CPU 1GPU 2GPUs 4GPUs

The number of keys /s 35 1,536 3,072 6,144

Combining with implementations of decryption, decompression and plain text recognition on GPUs, we can see that the time for recovering password significantly increases when performing on GPUs, as shown in table 2. V. CONCLUSIONS In this paper, we successfully implemented decryption, decompression and plain text recognition for encrypted ZIP files in parallel on GPUs. These techniques are performed for all candidate passwords until finding out the correct password. With the original purpose is recovering password for ZIP files, the experimental results have completed the phases of the problem of recovering password for encrypted ZIP files. All the results show that the speed of recovering password that uses the parallel algorithms has given higher performance (45 times for one GPU system and 180 times for 4GPUs system) in comparison to the sequential CPU-based program, along with increasing the chance of recovering password in an acceptable time by analyzing the password structure to form the quality

password space. These results have demonstrated the potential applicability of GPUs in cryptanalysis field. Generally, we want to recover password with more different compression and encryption methods for the phase decryption, decompression and plain text recognition. In this paper, we have only applied the encryption method AES -128 and the compression method deflate for encrypted ZIP files. These two methods are the default options of WinZip when compressing and encrypting a document. In fact, WinZip supports many compression and encryption methods. In addition, we will also consider implementing the proposed algorithms on GPU cluster to exploit the power of such computing system. Finally, the solution proposed in this paper can be customized to apply in cryptanalysis problems on other kinds of protected files such as DOC, PDF. REFERENCES [1]

[2]

E. F. Foundation. Cracking DES: Secrets of Encryption Research, Wiretap Politics and Chip Design. O’Reilly & Associates, Inc, 1998. [3] A. Klein. Attacks on the rc4 stream cipher. Des. Codes Cryptography, 48(3):269–286, 2008. [4] F. I. P. S. P. 197. Advanced encryption standard (aes), 2001. [5] Aes encrytion info http://www.winzip.com/aes_info.htm [6] A. Narayanan and V. Shmatikov, “Fast dictionary attacks on passwords using time space tradeoff”, in CCS05: Proceedings of the 12th ACM conference on Computer and communications security, pages 364–372, New York, NY, USA, 2005. ACM. [7] PKWARE. Zip file format specification, 2007. [8] M. Weir, S. Aggarwal, B. d. Medeiros, and B. Glodek, “Password cracking using probabilistic context-free grammars”, In SP09: Proceedings of the 2009 30th IEEE Symposium on Security and Privacy, pages 391–405, Washington, DC, USA, 2009. IEEE Computer Society. [9] NVIDIA. http://www.nvidia.com/object/cuda_home_new.html [10] RFC1951 - DEFLATE Compressed Data Format Specification. [11] Zlib library http://www.zlib.net.

P. Dung, D. Tan, P. Phong, N. Duc, and N. Thuy, “Applying cuda computing technology in the problem of recovering zip file password. In FAIR09”, Proceedings of the 4th National Symposium of Fundamental and Applied Information Technology Research, 2009 (in Vietnamese). TABLE II. The length of passwords 1 2 3 4

COMPARISON OF TIME FOR R ECOVERING PASSWORD ON CPU/GPUS The number of passwords 62 3,906 242,234 15,018,570

1CPU 4s 8m 2.25h 5.25d

1GPU 23s 47s 4m55s 3h5m

2GPUs 23s 47s 2m30s 1h32m

4GPUs 23s 47s 1m34s 45m48s