One of the fundamental aspects of software development is the transformation of data from ... domain, many of which are
Document number: DXXXX Date: 2016-10-07 Reply-to: Guy Davidson,
[email protected] Reply-to: Paul Kitchin,
[email protected] Audience: Library Evolution (LEWG), Game dev and low latency (SG14)
A proposal to add a transcoding library to the standard library 0. Contents 1. 2. 3. 4. 5.
Introduction Impact on the standard Motivation Design Future work
1. Introduction This proposal introduces support for transcoding. Support for compression and encryption are conspicuously absent from the standard library, and these are both types of transcoding. The author seeks feedback on the approach presented before committing to a thorough treatment.
2. Impact on the standard This proposal is a pure library extension. It does not require changes to any standard classes, functions or headers.
3. Motivation One of the fundamental aspects of software development is the transformation of data from one representation to another. This is even embodied in the standard library through the function std::transform, which performs a supplied unary or binary operation on each element of a range and sends the result to an output iterator. This is an isomorphic transformation. Compression, on the other hand, works in a different way. It does not isomorphically transform data, it in fact turns a range of data into a smaller range of data. Decompression turns a range of data into a larger range of data. Encryption can be isomorphic, but this is not a necessary feature of encryption: the encryptor may wish to perform compression along the way, or add redundant misdirector data. std::transform is therefore not sufficient for the job of this branch of transcoding.
4. Design There are many compression and encryption algorithms, some of which are in the public domain, many of which are patented, commercial property. It is not controversial to suggest that the standard should only include implementations of algorithms that are in the public domain. However, any implementation of a transcoding library should enable third party providers to support a simple interface.
It would be trivial to simply add support for specific algorithms by specifying functions with signatures such as template< class ExecutionPolicy, class InputIt, class OutputIt > OutputIt compress_lzw( ExecutionPolicy&& policy, InputIt first, InputIt last, OutputIt dest, struct lzw_parameter const& p ); Here, the example function compresses a range using the LZW algorithm, controlled by the parameters passed in p, and writes it to dest. However, this lacks flexibility. All that is required is a customisation of the output iterator to write the data after compressing it. Consider an output iterator adaptor based on Unix c ompress : template class lzw_compressor : public OutIt { public: lzw_compressor(OutIt&& dest_, int bits); lzw_operator& operator*() { return *this; } lzw_operator& operator=(const char&); // compress here private: Allocator allocator; int bits; }; The compression work is done in the assignment operator; it can be implemented in external library functions or in the class itself. In this example the implementation exists in the class itself, and is likely to require allocation to store consecutive writes before compressing and sending them to the parent output iterator. This can be used thus: using compressor = sg14::lzw_compressor; std::copy(std::begin(buffer), std::end(buffer), compressor(std::ostream_iterator(file), 16)); Any transcoder can adapt an output iterator, not just a compressor. An encryptor based on RSA encryption might look like this: template class rsa_encryptor : public OutIt { public:
rsa_encryptor(OutIt&& dest_, int m_, int n_); rsa_encryptor& operator*() { return *this; } rsa_encryptor& operator=(const char&); // encrypt here private: int m; int n; }; This can be used thus: using encryptor = sg14::rsa_encryptor; std::copy(std::begin(buffer), std::end(buffer), encryptor(std::ostream_iterator(file), 37, 91)); Transcoding adaptors can be composed. If the user wishes to compress and encrypt a buffer, the code would look like this: using encryptor = sg14::rsa_encryptor; using compressor = sg14::lzw_compressor; auto file = std::copy(std::begin(buffer), std::end(buffer), compressor( encryptor(std::ostream_iterator(file), 37, 91), 16));
5. Future work If this approach is supported, future work will consist of collecting together a selection of encryption/decryption and compression/decompression transcoders that are in the public domain or that can be licensed appropriately for inclusion. A call will be put out to the development community for suggestions for such transcoders. This collection can then be implemented as output iterator adaptors and presented to the committee for approval prior to generating the correct wording.
