XParse: A Language for Parsing Text to XML - Semantic Scholar

XParse: A Language for Parsing Text to XML James Cheney Cornell University Ithaca, NY 14853

ABSTRACT This paper presents a domain-specific language, XParse, that attempts to combine the power of tools like lex and yacc, which generate efficient parsers from declarative specifications, with the convenience, safety, and usability of textor XML-processing languages such as Perl or XSLT. XParse is a standalone language which provides lex-style regular expression matching and yacc-style LALR(1) parsing. Existing parsing tools such as lex and yacc can be difficult to learn, are usually highly language-dependent, and often vary across languages and platforms. Parsers are therefore difficult to create and reuse, so are usually developed on a per-application basis rather than a per-language basis. Unlike traditional, language-dependent parser tools, the semantic actions in XParse denote XML fragments rather than uninterpreted source code. This design facilitates independent typechecking and analysis of XParse programs, without first translating to some other programming language as is the norm for existing parsing tools. Furthermore, since XML already enjoys wide support, XParse programs can be reused in many environments without modification. We present several applications of XParse, including providing more user-friendly concrete syntax for existing XML dialects like XSLT, parsing existing languages such as Java, and parsing XParse programs themselves. These applications show that XParse (in concert with other XML tools) can be used to help develop useful language tools quickly and conveniently.

1.

INTRODUCTION

Scripting languages such as Perl make it easy to accomplish complex tasks using simple yet powerful text processing techniques such as regular expression pattern matching. XML standards for stylesheets, transformations and queries (including XSLT [4] and XQuery [6]) and scripting or general-purpose languages (including XDuce [11] and CDuce [2]) bring this combination of convenience and power to the world of more richly structured XML data. However, for a variety of reasons (inertia among the most im-

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright 2002 ACM X-XXXXX-XX-X/XX/XX ...$5.00.

portant), many forms of interesting data are still stored and maintained in a human-readable form rather than XML. Examples include most programming languages, TEX markup, and legacy HTML. Consequently, tasks which seem straightforward in theory (such as writing a script to search for or rename all occurrences of a given variable) are impractical because of the effort required to parse such legacy formats. This is not because parsing is inherently difficult, but because existing parsing tools are difficult to learn and use. To make matters worse, finished parsers are usually tailored to a particular task and programming environment, so reusing them often requires considerable effort. We believe that there is a need for text-to-XML parsing tools that combine the power of standard parsing tools with the convenience of scripting languages. We have developed a language called XParse that addresses this need. XParse combines the regular expression matching and LALR(1) parsing techniques of traditional AT&T-style lex and yacc tools for C and other languages with the convenience and flexibility of scripting languages. Like existing tools, XParse programs are declarative specifications of lexical analyzers and parsers. However, the semantic actions of XParse specifications are XML fragments indicating how to interpret the input as tokens or parse trees, rather than being arbitrary, uninterpreted code in some general-purpose language as in traditional lex and yacc. This has several positive consequences: 1. XParse specifications require no “boilerplate” coding to implement abstract syntax tree data structures and can be combined with other XML- or text-processing tools without requiring low-level programming. 2. XParse is a stand-alone language, so specifications can be statically analyzed, typechecked on their own, without first being translated to some target language. 3. Similarly, since XParse programs have meaning independent of any other languages, they can be interpreted, compiled to executable programs, or even compiled to libraries that produce XML data using a standard interface like SAX or DOM. XParse can be used to develop portable and reusable standalone parsers from existing text formats to XML quickly, facilitating rapid development of programming tools using XParse in concert with other XML tools. Existing XML standards such as XSLT and XML Schema are presented as XML dialects, which are very machine-friendly but not as convenient for programmers to read and write. XParse can

be used to parse programmer-friendly concrete syntaxes for such languages to their XML equivalents. Thus, XParse illustrates both how programming research and development can benefit from XML technologies and how XML standards can be enriched with more usable interfaces using established programming language tools. In this paper we present a design for XParse, based in part on insights gained from a simpler prototype implementation. This design is a work in progress. The rest of this paper is structured as follows. In Section 2 we give an overview of the standard lex and yacc style tools. Section 3 describes the XParse language design. Section 4 describes the prototype implementation of XParse, and Section 5 discusses several sample applications, including XParse parsers for XSLT stylesheets, Java source files, and XParse specifications themselves. Section 7 discusses related and future work and concludes.

2.

BACKGROUND: LEX AND YACC

In this paper, we assume familiarity with XML but not with lex and yacc. In this section, we give an overview of the design and implementation of these tools. For good indepth coverage of lex and yacc, see [15]. We discuss this background for several reasons: first, to make this article accessible to readers not intimately familiar with such tools; second, to introduce the underlying scanning and parsing techniques upon which XParse is based; and finally, to support our position that such tools are difficult to learn and use, in part because of their software architecture rather than the (admitted) inherent difficulty of parsing. A lexical analyzer (or scanner ) splits a text stream into tokens. For example, a simple lexical analyzer for English text might tokenize it into words (that is, contiguous sequences of letters) and punctuation, ignoring whitespace (i.e., spaces, tabs, and newlines.) Lexical analyzer generators like lex convert declarative specifications of lexical analyzers to efficient programs. The specifications relate regular expressions to semantic actions describing how the text is tokenized. In lex, some rules for tokenizing a simple form of English are: [a-zA-Z]+ "," "."

{ yylval = yytext; return WORD; } { return COMMA; } { return PERIOD; }

Semantic actions, enclosed in braces, are uninterpreted C code to be executed when the rule is applied. When used in conjunction with yacc, the value returned by a semantic action indicates a token value for the matched string. However, semantic actions can include arbitrary code and need not return. The yylval variable can be used to associate a semantic value with a token (here, the string value of a WORD token). Ambiguities among collections of rules are resolved by giving priority to the longest match and then to the earliest-declared rule. lex translates these rules into nondeterministic finite automata. These NDFAs are converted to deterministic automata using standard algorithms [10]. The program constructed by lex simulates the deterministic automaton until it reaches an accepting state, then executes the associated semantic action. Pure regular expression matching is not powerful enough to handle some languages. In particular, tokenization is not always uniform throughout a language; for example, many languages include string literals (i.e., strings delimited by

quotes) within which the usual tokenization rules do not apply. lex’s start states make it possible to recognize such constructs. Each rule can be associated with one or more start states, and the rule applies only when the lexer is in one of the listed states. The action BEGIN transitions to another state. Also, some lex variants maintain a stack of start states, with semantic actions PUSH and POP. In addition, lex specifications can define and maintain arbitrary additional state; common examples include counters for line numbering and delimiter nesting and symbol tables for correctly parsing C-style typedef identifiers. A parser is a program that reads a sequence of tokens and attempts to build an abstract syntax tree according to the rules of a context-free grammar. Parser generators like yacc transform declarative descriptions of such grammars to efficient parsers. The rules used in yacc grammars are production rules of the form S : α1 . . . αn , where S is a nonterminal symbol and the αi s are either tokens or nonterminal symbols. Such rules are associated with semantic actions which specify how to construct a semantic value for S, given semantic values for the αi s. In C’s yacc notation, some rules for parsing simple arithmetic expressions might be: exp : | | |

INT exp PLUS exp exp TIMES exp LPAREN exp RPAREN

{ { { {

$$ $$ $$ $$

= = = =

mkInt($1); } mkPlus($1,$3); } mkTimes($1,$3); } $1; }

As in lex, the semantic actions are arbitrary C code. The symbol $$ denotes the return value of the action, and symbols $1, $2, etc. denote the semantic values (if any) of the first, second, etc. symbols on the right side of the production rule. Parsers generated by yacc use a shift-reduce parsing algorithm. A shift-reduce parser maintains a stack whose entries are tokens that have been read or nonterminals that have already been reduced. At every step, the parser decides whether to shift a new token from the input and push it onto the stack, or to reduce the top few symbols on the stack according to a matching grammar rule. Shift-reduce parsing can involve a lot of nondeterminism: at each step we need to decide whether to shift or reduce and (if the latter) which rule to reduce. To make these choices, yacc parsers use a finite-state control (with a state stack) to decide when to shift or reduce. Each automaton state is labeled with a collection of partial rules that may eventually match the current stack. The automaton’s input is the symbol consisting of the next unshifted symbol in the input, and the transition for each token indicates whether to shift a new token onto the stack (simultaneously transitioning to a new state), or reduce the current top of the stack using a grammar rule. The technique used by yacc to construct this automaton is called LALR(1). When nondeterministic behavior is still possible, yacc produces warnings called shift-reduce or reduce-reduce conflicts, depending on whether the ambiguity is whether to shift or reduce, or whether to reduce one of two distinct rules. yacc produces a text file describing the automaton and any conflicts on request. Simplistic grammars such as the above expression grammar are usually ambiguous in inessential ways, resulting in shift-reduce conflicts. For example, the expression 1 + 2 ∗ 3 could be parsed as (1+2)∗3 or 1+(2∗3) (the usual convention is to assume the latter form). Expressions such as 1 + 2 + 3

are also ambiguous, because both (1 + 2) + 3 and 1 + (2 + 3) are possible parses. Even though we might consider these parses to be semantically equivalent, they are not equivalent as far as yacc is concerned. This is not always undesirable, because not all operations are associative1 . There are two standard ways to disambiguate such grammars. First, the grammar rules can be rewritten to be more precise: exp

: | times exp : | atomic exp : |

exp PLUS times exp times exp times exp TIMES atomic exp atomic exp INT LPAREN exp RPAREN

Implicitly, these rules make both + and * left-associative (that is, 1 + 2 + 3 parses to (1 + 2) + 3), and they give higher precedence to * than +. This solution makes a grammar considerably more complicated and can be tricky to implement correctly, and it contradicts the spirit of declarative programming, where the ideal is to write as clear a description as possible rather than specialize the rules to a particular model of computation. Unfortunately, sometimes this is the only solution. The second standard solution is to employ precedence directives that specify the associativity and relative precedences of the conflicting operators. The following directives suffice to disambiguate the original ambiguous grammar: %left PLUS %left TIMES These directives state that both PLUS and TIMES are left associative, and the order of the declarations is used to establish their precedence order. The effect of these directives on the expression grammar is identical to the above rewritten grammar. Precedence directives work well for binary operations and a few other common cases, such as dangling-else conflicts. lex and yacc were designed to work together (although they can also be used separately). lex produces a C source file that implements a function int yylex(); which reads the next token from its input and returns a token value (implemented in C using #defined integer constants). Other communication between the lexer and the rest of the program is provided by global variables; for example, yylval contains the semantic value associated with a token, if any. yacc produces C source files that expect a yylex() function of the above type to be provided by the program with which it is compiled, and provides a function int yyparse(); which parses the input, using yylex() to fetch tokens from it.

3.

THE XPARSE LANGUAGE

XParse combines two rather different sublanguages, one for tokenizing and one for parsing. Historically, lex and 1 For example, arithmetic operations in C are not guaranteed to be associative.

%regexp var = [a-z]+ %regexp digits = [0-9]+ %regexp ws = [\ \t] %token INT VAR : string %token PLUS TIMES MINUS DIV LPAREN RPAREN %state INITIAL : token %state COMMENT : token\token %left PLUS MINUS %left TIMES DIV %type Exp = (plus|times|minus|div|int|var) %element doc : Exp %element plus times minus div : (Exp,Exp) %element int var : EMPTY %attlist int (value:string) %attlist var (name:string) %nonterm exp : Exp %start start : doc %counter cdepth Figure 1: Complete XParse Expression Parser I: Declarations yacc were developed as separate tools. Traditional yaccstyle tools produce an interface file that declares the token symbols in the target language, and corresponding lexstyle tools must import this interface in order for the resulting code to make sense and be correct. Since lexers and parsers are usually designed in tandem, this design forces the programmer to switch between two files frequently, and can result in hard-to-find bugs due to inconsistencies between them. We see few reasons to keep this design. Instead, XParse programs combine declarations, lexer rules, and grammar rules into one file. XParse programs consist of three sections: 1. A declarations section, describing the tokens and nonterminals and their types, along with declarations of state names and variables, regular expression abbreviations, token precedences/associativities and starting nonterminal. 2. A lexer section, consisting of a collection of rules associating regular expressions with actions that construct a token sequence. 3. A parser section, consisting of a collection of rules associating grammar productions with semantic actions that construct an XML syntax tree. This makes each parser a self-contained specification that is more convenient to write and maintain and easier to check for errors than two separate files are. As in traditional lex/yacc tools, these sections are separated by lines beginning with %%. Figures 1–3 show a complete XParse program, which implements a simple parser for arithmetic expressions with nested comments and variables. Unlike traditional parser specifications, XParse programs can be either interpreted directly or compiled to runnable code. A session with an XParse interpreter might go as follows:

%% Parser start : exp ;

%% Lexer { {digits} {var} "+" "*" "-" "/" "(" ")" "(*" "\n" {ws}+

{ { { { { { { { {

token([$$]) } token([$$]) } token() } token() } token() } token(