The Challenges of Constraint-Based Test Generation Vitaly Lagoon Cadence Design Systems, USA
[email protected]
Abstract We describe the challenges underlying the design of a constraintbased test generator for functional verification of hardware. We show how these challenges are addressed by IntelliGenTM – a constraint-based test generator by Cadence Design Systems. Categories and Subject Descriptors B.5.3 [Register-TransferLevel Implementation]: Reliability and Testing – Test Generation General Terms
Design, Verification
Keywords functional verification, constraint-based test generation, constraint debugging Microchips can fail for variety of reasons ranging from highlevel design flaws to manufacturing defects. However, logic and functional “bugs” are indisputably the most common cause of chip failures. Semiconductor industry practitioners estimate (e.g., [2]) that over 70% of re-spins are caused by functional flaws in hardware designs. Considering extremely high costs of re-spins and catastrophic costs of recalls, it is not surprising that semiconductor companies invest a large (and growing) share of R&D and financial resources into functional pre-silicone verification of their hardware designs. It has long been recognized that manual development of tests for hardware designs in C or other traditional programming languages is inadequate. Due to vast number of states and execution paths and limited productivity of manual coding, this approach can never verify a state of the art design with an acceptable level of confidence. Thus, the effort and resources invested in verification are generally directed towards more automation, higher levels of abstraction and more sophisticated verification algorithms. Current methods of automatic validation applied by the industry can be generally subdivided into simulation-based functionality testing and formal verification. Simulation-based testing consists in running the design in a simulated environment and checking its behavior on as many automatically produced “interesting” tests as possible. Formal verification aims to automatically prove that the design correctly implements a formal specification i.e., satisfies certain properties. Due to capacity problems, applications of formal verification are usually limited to validation of individual design components, while (inherently incomplete) simulation remains the dominant technique in validating system-wide correctness of designs.
Copyright is held by the author/owner(s). PDPP’11, July 20–22, 2011, Odense, Denmark. ACM 978-1-4503-0776-5/11/07.
A simulation-based verification environment for a deviceunder-test (DUT) normally includes a test generator, a simulator and a checker. The three components are responsible respectively for generating meaningful tests, simulating the work of the device, and checking the results of the simulation for correctness. Additional modules based on coverage metrics are often used to track the progress of validation. In this talk we address the challenges underlying the design of a tool for automatic generation of tests from constraint-based models. Such models provided by verification engineers define formats of test data, control signals, and capture the architectural dependencies and test requirements for the design. We present IntelliGenTM by Cadence Design Systems which is a part of the company’s SpecmanTM solution for automation of hardware verification. The central component of IntelliGen is a mechanism that generates randomized tests based on a constraint-based specification. We demonstrate that a constraint-based test generator needs to answer to a different set of requirements compared to solvers for classic CP tasks, such as resource allocation. To name a few: (1) The test generator must be designed for repeatedly solving the same constraint problem, sometimes millions of times, possibly with respect to changing external inputs. (2) The output of the system should satisfy certain distribution requirements for the produced tests to target different aspects of the design behavior. (3) Bugs found during simulation must be reproducible and for that reason the system must be random-stable i.e., the same model should always generate the same series of tests for the same starting seed, even if used on different platforms or embedded in different contexts. Additional challenges are associated with the need to adequately model class hierarchies and inheritance, dynamic arrays, data types including arbitrary-precision integers, strings and pointers. The tool needs to support various kinds of constraints such as arithmetic, logic, bitwise, as well as global constraints on arrays and special pragmas for controlling distributions. We show how all of these needs are addressed by Specman through its modeling capabilities [3] and constraint solving techniques of IntelliGen [4, 6]. Finally, we argue that a constraint debugger is a “must have” feature of a constraint-based test generator. Due to immenseness of real-life specifications, automatic tools are crucial to the engineers’ ability to analyze and fix problems in test environments. Similarly to other kinds of software, constraint-based specification may contain “bugs” which lead to unexpected behavior such as (a) the specification is infeasible i.e., no solution can be found, (b) solving is too slow or runs out of time, (c) unexpected values are generated in the solutions or (d) some expected and targeted values never occur in the solutions. We present the basic concepts of GenDebugger [1, 5], a constraint debugger integrated in IntelliGen and show how it makes debugging of constraint models a manageable task.
References [1] G. Alexandron, V. Lagoon, R. Naveh, and A. Rich. Gendebugger: An explanation-based constraint debugger. In Third workshop on Techniques foR Implementing Constraint programming Systems (TRICS), 2010. [2] T. Fitzpatrick. Building modular, reusable, transaction-level testbenches in system verilog. 2006 MAPLD International Conference, Sept. 2006. tutorial. [3] Y. Hollander, M. Morley, and A. Noy. The e language: A fresh separation of concerns. In 38th International Conference on Technology of Object-Oriented Languages and Systems, pages 41–50. IEEE Computer Society, 2001. ISBN 0-7695-1095-7.
[4] V. Lagoon and G. Baruch. Method for providing bitwise constraints for test generation. Patent US 6,918,076, Nov. 2009. [5] A. Rich, G. Alexandron, and R. Naveh. An explanation-based constraint debugger. In Proceedings of the 5th international Haifa Verification Conference, pages 52–56. Springer-Verlag, 2009. ISBN 978-3642-19236-4. [6] S. Uziel, A. Noy, V. Lagoon, Y. Kinderman, and A. Gal. System and method for test generation with dynamic constraints using static analysis and multidomain constraint reduction. Patent US 7,870,523, Jan. 2011.
