Mar 27, 2016 - redirects stdout to a string buffer. After. sizeOfBuffer bytes printed, the test is aborted with a FATAL
Virtual-C IDE by Dieter R. Pawelczak E-Mail:
[email protected] Web: https://sites.google.com/site/virtualcide/
Test Suite Compiler (TSC) INTRODUCTION TO TSC Version 1.7 2016-03-27
TSC 1
DISCLAIMER This software is provided "as is". In no event shall the authors or contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption). The applications TSC, SimpleC and Virtual-C IDE (VIDE) are non-commercial products. The author cannot provide support of any kind.
LICENSE and LICENSE INFO The TSC (Test Suite Compiler) is part of the Virtual-C IDE written by Dieter R. Pawelczak. It uses portions of the integrated Simple-C Compiler also written by Dieter R. Pawelczak. Syntactically it is adapted from the Google C++ Testing Framework1 but not using any portions of its code. Conceptually based on the well-known xUnit frameworks and its test dialogs2. Portions of this manual are taken from the paper “A new Testing Framework for C-Programming Exercises and Online-Assessments”, by Dieter Pawelczak, Andrea Baumann, and David Schmudde, published at FECS’15 in Las Vegas, Nevada, USA, July 27-30 2015. Many thanks Andrea and David for your contribution! The Virtual-C IDE (VIDE) serves as a platform for teaching the C-programming language and for learning how to write programs in C. It is freeware for private, non-commercial use. For more information see “Virtual-C IDE - USER MANUAL”. Virtual-C IDE © 2016 by Dieter R. Pawelczak
1
Zhanyong Wan, et al. Google C++ Testing Framework – googletest: https://code.google.com/p/googletest/.
2
K. Beck, Test Driven Development: By Example. Addison-Wesley, 2002.
TSC 2
Introduction The Virtual-C IDE (VIDE) is a programming environment especially for learning and teaching the C programming language. It provides next to debugging capabilities visualizations for the data and control flow. C-programs run within a virtual machine (VM), which allows running platform independent ISO C programs. Access to the VM during execution enables a large set of program analyses. Programming exercises are therefore available for either offline testing of students’ programming code or with integration into an online assessment system. Crucial for the success of programming exercises is the feedback given to the students: it should be easy to understand, transparent, logical and target oriented. The proposed method is testing based on the xUnit test framework: Providing per unit a test suite with a hierarchical set of test cases and tests. Ideally each test checks an individual characteristic of the function under test, so that a student can easily follow the testing results. A test suite (TS) is a single test file and consists of test cases (TC) based on one or more tests. To those of you who are familiar with googletest, the differences (despite the programming language) compared to the Google C++ Testing Framework are: • • • • • • • • • • • •
Random test data via the macros ARGR & RAND. Simplified verification of output parameters with the ARG macro. Predefined tests: function and reference function tests, performance tests, and I/O tests No test fixture macro TEST_F. Instead, test fixtures are provides in the test prologue. Pseudo variables $out, $result, etc. FUT macro for simple error reporting. ASSERT_RANGE macro and ASSERT_EPSILON macro. Test and test case names can be string literals Heap and data segment re-initialization per test case for full application tests, i.e. execution of main(). Automatic prototyping for functions under test. GUI based selection/ de-selection of test cases Dynamic re-linking of C functions.
In general, open a test suite (file extension .tsc) will automatically view the available test cases in the test dialog; compare Fig. 1. In order to edit a test suite, press the edit button3. The test dialog has its on run button. A test suite is always run on the current active file in the editor (even if a project is defined in the build options). On run: first the C module under test (MUT) is compiled, information about the source code is stored in the static source information database SSID. source second, the test suite compiler (TSC) runs and already performs static tests on (MUT) and last, the output of the TSC is linked together with the C module and the MOPVM library extentsions. The later provide access to thevirtual machine during execution. These initial steps are called the static tests. This step might already produce errors and prevent further testing. The resulting executable represents the actual test: during its run, it invokes functions of the C module under test, checks assertions and generates the test report. Test cases summarize the results of the underlying tests in a traffic light scheme (red : fatal errors, yellow: errors and green : no errors). Each test logs ist results in the report. A progress bar and a numerical display of the passes/ fails summarize the overall results. The user can deselect test cases to focus on sepecific test cases. 3
During online assessments editing is not allowed.
Fig. 1: The test dialog
TSC 3
Assertions vs. expectations and warnings In accordance with the Google C++ Testing Framework the test framework distinguishes between assertions and expectations as expressed by the macros ASSERT_* and EXPECT_*. An assertion must be met and contradiction leads to an immediate failure of a test. An expectation might not be fulfilled. This will lead to a failure of the test, but its execution is continued. A typical example for expectation vs. assertion is a function modifying a pointer passed as parameter. It is wrong, if the student does not test for a NULL pointer; still the functional part might be implemented correctly for valid pointers. In case the programming assignment does not rely on the NULL pointer test, this test could use EXPECT_*, whereas the proper functionality is tested by assertions. The behavior of assertions and expectations can be expressed with the macros FATAL() and ERROR() respectively, to print a corresponding message in the report. Additionally, a test can always print warnings with the macro WARN(). Assertion
Expectation
Verifies
ASSERT_TRUE(condition) ASSERT_FALSE(condition) ASSERT_EQ(val1, val2) ASSERT_NE(val1, val2) ASSERT_LT(val1, val2) ASSERT_LE(val1, val2) ASSERT_GT(val1, val2) ASSERT_GE(val1, val2) ASSERT_STREQ(val1, val2)
EXPECT_TRUE(condition) EXPECT_FALSE(condition) EXPECT_EQ(val1, val2) EXPECT_NE(val1, val2) EXPECT_LT(val1, val2) EXPECT_LE(val1, val2) EXPECT_GT(val1, val2) EXPECT_GE(val1, val2) EXPECT_STREQ(val1, val2)
ASSERT_STRNE (val1, val2)
EXPECT_STRNE (val1, val2)
ASSERT_STRCASEEQ(val1, val2)
EXPECT_STRCASEEQ(val1, val2)
ASSERT_STRCASENE (val1, val2)
EXPECT_STRCASENE (val1, val2)
ASSERT_RANGE(val, val1, val2)
EXPECT_RANGE(val, val1, val2)
ASSERT_EPSILON(val, val1, val2)
EXPECT_EPSILON(val, val1, val2)
condition is true (value not 0) condition is false (value is 0) val1 == val2 val1 != val2 val1 < val2 val1 val2 val1 >= val2 Strings with equal content, as like strcmp(val1, val2) == 0 or both values NULL Strings contents differ, as like strcmp(val1, val2) == 0, or one value NULL Strings with equal content (case insensitive) or both NULL Strings contents differ (case insensitive) or one value NULL Checks on int/ floating point values val, if val1
