Improving Implementation of Code Generators: A Regular-Expression Approach Maria Consuelo Franky Departamento de Ingeniería de Sistemas Pontificia Universidad Javeriana Bogotá, Colombia
[email protected]
Abstract—Code generators are important tools in software development, to automate repetitive coding tasks, facilitate portability, abstract implementation details, and reduce development costs. However, as the complexity of code generators grow, they tend to be harder to maintain, especially when there is a large amount of templates involved. This paper proposes an approach for code generation based on regular expression substitution. Instead of using templates for code generation, this approach transforms existing source code, using regular expression substitution, to implement the required functionality. We are currently applying this technique to strengthen the code generation framework of Heinsohn Business Technology, as part of the "Lion" project co-financed by Colciencias. Our experience shows that, although this approach has a steeper learning curve, it facilitates capitalization of the experience of software development organizations, selecting successfully modules that are taken as a reference source for code generation in future projects. Keywords- Code Generation; Computer Aided Software Engineering (CASE); Frameworks; Web Applications Development
I.
INTRODUCTION
A code generator is “a software tool that accepts as input the requirements or design for a computer program and produces source code that implements the requirements or design” [33]. Code generators are very useful tools to reduce the effort to develop a software system. For instance, in the context of the Model-Driven Architecture (MDA) [37], code generation has a very important role to automatically implement the models created by designers. Similarly, Domain-Specific Modeling (DSM) [34] uses code generation to implement the solution to the domain model. Overall, a code generation tool, combined with appropriate modeling artifacts, can automate repetitive tasks, facilitate portability, abstract implementation details, and reduce development costs, among other benefits [35] [36]. A very common approach to implement code generators is the use of templates. A template describes a way to generate a
This article is part of the Project "Soporte al desarrollo de aplicaciones empresariales mediante frameworks de generación", executed by the SIDRe research group of the Pontificia Universidad Javeriana and Heinsohn Business Technology, co-financed by Colciencias.
Jaime A. Pavlich-Mariscal Departamento de Ingeniería de Sistemas Pontificia Universidad Javeriana Bogotá, Colombia
[email protected] Departamento de Ingeniería de Sistemas y Computación Universidad Católica del Norte Antofagasta, Chile
[email protected]
piece of code from a set of input data, often in the form of models [38]. Template languages can be used to specify the structure of templates and include mechanisms to reference elements from the input data, to perform code selection, and iterative expansion [38]. Templates offer a degree of flexibility in code generation, since one can substitute templates to generate code for different platforms or architectures. However, as the complexity of a code generator grows, more templates are required to be maintained. Moreover, template debugging can be difficult and error prone, since one must first generate code from those templates, execute and debug that code, and then propagate the corrections back to the template. Overall, the more templates a code generator has, the harder is its evolution and maintenance. This paper proposes and analyzes a technique to generate code based on regular expression substitutions. Figure 1 describes the overall approach. Boxes represent assets (source code, configuration files etc.), rounded boxes represent executables, and arrows represent flow of information. The assumption is that there is a reference source code component, i.e., source code that has been successfully used and tested in previous projects. Developers want to include a similar functionality in a current project. A component parameterization consists of a set of regular expression substitutions that parameterize the source code module for its reutilization in other projects. For instance, one can specify that class names must include an arbitrary prefix, before generating the module for the current project. A projectspecific component configuration specifies the concrete values for the parameters denoted by the component parameterization. For instance, the project-specific module configuration can include the specific prefix for the class names for the current project. The code generator takes as input the reference source code from the previous project, uses the regular expressions specified in the component parameterization to find all of the relevant places in the code, and substitutes those places with
the information of the project-specific module configuration. The result is a new source code module that can be directly incorporated into the current project. The code generator can also modify the source code of the current project to better integrate the desired module. Reference Source Code (from a previous project)
Source Code (of the current project)
Component Parameterization
Project‐specific component configuration
Code Generation Framework based in Regular Expressions Modified source code (of the current project) Reference Source Code integrated into current project
Figure 1 Overview of the code generation approach based on regular expression substitutions.
The authors successfully applied this approach to integrate a security module into web applications. Instead of using templates, the source code of the security module is directly reused into a new project. [29,30]. In this same line of work, the authors are executing a project called "Soporte al desarrollo de aplicaciones empresariales mediante frameworks de generación" (“Enterprise application development support through generation frameworks”). This project, co-financed by Colciencias, is a joint effort between the SIDRe research group [49] of the Pontificia Universidad Javeriana and Heinsohn Business Technology (HBT), a software development company [31]. The goal is to apply the proposed approach to strengthen and extend the code generation framework of HBT. The remainder of this paper describes the proposed approach and discusses its value. Section II explains the basic concepts required to understand the approach. Section III explains the main characteristics of template-based and regularexpression-based code generation. Section IV describes a case study that illustrates the usage of regular-expressions for code generation. Section V analyzes the advantages and disadvantages of both approaches, based on the results of the case study. Section VI discusses related work. Section VII concludes. II.
BACKGROUND
Automatic software construction is a technology that has been present since several decades ago and it is considered an important tool for abstracting programming language details [48]. Some of the first Computer Aided Software Engineering (CASE) tools date back to the decade of 1980. CASE tools
assisted engineers to specify the software design and to generate part of the code of the application. These tools, focused mainly in developing software for mainframes, lose popularity with the advent of Internet, the diversity of user interfaces, and the complexity of multi-layered systems [1]. The idea of automatic software generation has regained strength during the last years, particularly for enterprise applications. The development of these applications, which include support for distributed processing across the Internet and multi-layered architectures, require the use of frameworks with the following characteristics [2]:
Ability to capitalize good practices, by using generators that reuse those practices in new projects.
Fast application development and reduced learning time for developers, so that software products can reach the market as quick as possible.
Support for standard software patterns and widespread software architectures, such as .NET and Java EE
Automatic or semi-automatic generation of nonfunctional concerns of the application (e.g. security, auditing, etc.), to let developers focus in the construction of the business-related functional concerns of the application.
In this context, a framework is a set of reusable services and components, organized in an extensible structure, to simplify application development. There are two types of frameworks: infrastructure frameworks, and code generation frameworks. Infrastructure frameworks are software libraries to implement common application concerns, usually associated to non-functional requirements, such as security, persistence, message queuing, etc. [3]. To adequately utilize infrastructure frameworks, developers must follow specific design patterns, such as those described by Alur et al. [11]. These patterns describe best practices to utilize a particular infrastructure framework. To reduce the learning curve of patterns and infrastructure frameworks, software development organizations construct code generation frameworks. Code generation frameworks are used to generate the skeleton of a project and also to progressively generate additional functionality of new modules and use-case implementation (incorporating infrastructure frameworks). In a software application, the automatic generation of its skeleton contributes to reduce the total time and cost of a project and provides firm basis for the development of the remaining components. In addition to automating part of the development process, code generation frameworks can assist source code standardization by incorporating the best development practices within the organization. Since developers do not need to learn all of the details of the underlying infrastructure frameworks, learning and training time can be reduced. All of the above can also leverage the use of agile methods to develop software [4].
Currently, there are well-known standards for infrastructure frameworks [12], but there are no standards for code generation frameworks (except if based on MDE-MDA [34] [36]). However there are several proprietary and open source approaches for code generation [5], [6], [7], [8], [9], and [10]. Software development companies can adapt the above solutions for their specific way to develop applications. III.
CODE GENERATION FRAMEWORKS
A code generation framework comprises multiple individual code generators that create source code with specific functionality. Code generators can be implemented in multiple ways: using template languages, using regular expression substitutions (the proposed approach), and as part of a modeldriven approach. The remainder of this section describes these techniques. A. Template-based Code Generation Template-based code generation approach relies on the use of templates, which denote the way to transform the input data of the generator into textual files. Some widespread template languages are Velocity [16], Jelly [17], FTL [18], Acceleo [39], JET [40], Xpand [41], and MOFScript [42]. In general, a template language has the following characteristics:
It includes text that will be included verbatim in the generated code.
During the processing of a template file there is a context with variables that store the input data to the code generator. Those variables can be referenced in the template file; the template generator substitutes those references with the value of the corresponding variables in the generated code.
A template file can also have conditional and loop statements that will write specific text in the generated code (either once or iteratively), based on logic conditions. A template file can include macros, which are substitution functions that facilitate reuse of template portions.
For instance, the following template uses a language called Velocity [16]. #foreach (${name} in ${list}) import com.${project}.${name};
#end This code contains variable references and a loop to iteratively write a text portion. If the following values are used as input to this template:
project = acme list = Reports,Security The resulting file will be the following: import com.acme.Reports;
import com.acme.Security; The generated code corresponds to the beginning of a Java file that imports two classes. Macros can be used to facilitate reuse of template portions. For instance consider the following macro: #macro (my_macro) section for ${Ejb} ejb --> ${Ejb}_ejb.jar acme acme Application When the template is executed by the code generator, my_macro is invoked, yielding the following code (assuming that the value of ${Ejb} is “Security”) : acme acme Application Security_ejb.jar The text in bold represents the text generated by the macro. A software development company can take advantage of the experience gained from previous projects to make a code generation framework. A previous project that uses good design and programming practices can be used as a base. Source code files of that project can be modified to convert them to templates. Those templates parameterize project information, such as component names, use case names, project and company name, etc.
The above process can be repeated for several modules, until the organization has a large set of templates for diverse functions: skeleton generation, CRUD operations generation, use case implementation generation (of various types), component generation, etc. A code generation framework comprises a set of interrelated code generators, such as the above ones. More specifically, the life cycle of a template-based framework is the following: 1.
From a previous successful project, source code files implementing the desired functionality are selected.
2.
Those source files are used as a reference to create the templates and macros, parameterizing all of those strings in the source files that need to be generalized: project name, package name, class names, use case names, hard-coded access control roles, etc. If it is required to add any functionality to the generated code that was not present in the reference source code, it must be added directly to the templates.
3.
Code generators are created based on those templates, to generate source code with a certain structure. The first code generator should create the skeleton of the new project. The remaining generators should be designed to incrementally add functionality into the generated skeleton.
4.
The generated code can be modified by developers to add new functionality. Unless the generator uses a template language that allows incremental generation [39] [40] [41], the generator cannot be used again to generate the same code, since the changes made by developers can be lost.
When developers find better ways to implement portions of a system, they must implement new versions of the corresponding templates. Developers must repeat the entire template creation process (Steps 1-3). The new version of the generator will be available for new projects, but it may be incompatible with previous projects, unless it is adequately designed for incremental generation. B. Code generation based on regular expression substitutions A regular expression is a pattern that denotes a set of strings of characters. A regular expression has the following main components [19] [20]:
Specific characters that a string must contain.
Character classes that denote characters that a string may contain at a specific position, e.g., numbers, letters, etc.
Quantifiers that indicate the presence or absence of certain characters inside a string. For instance the ‘+’ symbol after a character class indicates that said character class must be present one or more times in the string. The ‘^’ symbol preceding a
character indicates that said character class must not be present in the string. For example, the following regular expression represents all of the strings that denote a class declaration in Java: public[\s]+class[\s]+[^\s]+ \s denotes space character. [\s]+ denotes that one or more spaces can separate the word ‘public’ from ‘class’. [^\s]+ denotes a sequence of one or more characters that are not spaces, which in this expression is used to represent class names. Regular expression tools can detect strings that comply with a given regular expression and can also transform those strings. Examples of such tools are the java.util.regex library of Java [21], and the replaceregexp command in ANT [22]. An example of the latter is the following code that transforms a set of property declarations, inserting the string “New” at the beginning of each property name. For instance, the string prop=value would be transformed into Newprop=value:
