Purify - Daily 빌드 및 테스트과의 연계

출처 : http://www.ibm.com/developerworks/rational/library/08/0513_gupta-gaurav/index.html


Purify is a very useful tool throughout the software development life cycle.

• Developers can use it to ensure that the new code that they have written is not going to inadvertently cause any memory corruption errors or leaks. 
• Test engineers can use it to catch memory errors during functional verification and system integration testing. 
• And field and support engineers can use it to diagnose memory issues encountered after the software has been deployed. 

Because the cost of detecting and fixing a defect is least during early phases of the software development life cycle, it is best to catch and fix as many issues as possible during development and testing phases. You can achieve that ideal by methodical and systematic use of Purify throughout the software development life cycle. The best way to accomplish this is by automating the use of Purify and integrating it into your software development and testing process.

Automation of a tool eliminates the overhead and makes it effortless to use, which in turn reduces the resistance to its adoption as part of the process. Thus, automation is the key in streamlining the process. For example, you can integrate Purify with your unit or smoke test suite that developers must run before checking in any code changes and require them to fix any new memory errors reported by Purify. In this way, an error is caught as soon as it is introduced and fixed easily, because the code changes are still fresh in the developer's mind. Similarly, you can integrate Purify with your functional and system verification test suite, which you might be running nightly or weekly. Testers can analyze and file defect reports for memory errors reported by Purify. This ensures that new memory errors are caught within a day or a week from the time that they were introduced, which is much better than catching them after releasing the software.

You can learn about using Purify and integrating it into your build makefiles in the article: Navigating C in a leaky boat? Try Purify. If you are already familiar with Purify, you can skip or skim that article. In this article, you will first learn how to change your build and test environment to incorporate Purify into it and about conversions symbols that you can use with Purify options to automate using Purify. Then you will see an example where all of these capabilities are exploited to automate reporting a summary of Purify errors on a Web page.

Incorporating Purify into your build and test environment

The first step in integrating Purify into your software development and testing process is to modify your build and test system. The build system builds an application and the test system runs the application with a test suite (shown in blue in Figure 1). Typically, the process of building the application and running the test suite is automated and scheduled as nightly or weekly jobs.

You need to modify your build system to build a Purify'd application along with the normal application. Normally, you will be building the application without having any debug information (release bits). For building a Purify'd application, it is advisable (although not required) to build the application with debug information (debug bits), and then purify it.

You also need to modify your test system to run the test suite with the Purify'd application, in addition to running it with the normal application. These additional build and test steps are shown in green in Figure 1. After making these changes, add building the Purify'd application and running it with your test suite to your automated nightly or weekly jobs. Later in this article, you will learn about various ways of controlling and automating the actions to be taken when Purify detects memory errors or leaks.

Figure 1. Modifications in the build and the test systems



Using conversion symbols

Purify provides various conversion symbols that you can use to specify values for various options, such as -view-file and -log-file (these options send Purify output to a Purify view file and to an ASCII log file, respectively). Purify replaces these symbols by meaningful expansions and computes a unique file name for saving data. For example, you can put the program name and process ID in the name of log file:

$ purify -log-file=./purifyerrors_%v_%p.plog cc -o progname foo.c bar.c

This command will create an instrumented executable named progname. If you run this, and the process ID for that run is 1234, all Purify errors will be logged in a file named purifyerrors_progname_1234.plog. In the log file name, Purify expands %v to the program executable name and the %p to the process ID.

Adding operations after a Purify run

Automating what happens before running your instrumented application is easy, because you have all of the controls. You can control and automate what happens after running the application by exploiting various Purify features that let you add custom post-processing tasks. Purify enables you to run a script after exiting the instrumented application. You can use this to report a summary of all of the errors reported after the instrumented program exits. To do this, you use the Purify -run-at-exit runtime option.

For example, if your instrumented application is test.pure, you can use this option as follows to print a summary of errors found:

$ setenv PURIFYOPTIONS '-run-at-exit="if %z ; then \ echo \"%v : %e errors, %l bytes leaked.\" ; fi"'

The string that follows the -run-at-exit option is executed by the shell after the program exits. Conversion symbol substitutions are made, such as turning %z into false if there were no Purify errors or leaks during the run. Because of that, the if statement in this example says: "Execute the 'echo' only if there were errors." The echo command, in turn, uses more substitution strings to report how many errors there were. Upon exiting the program, Purify sends a message similar to one:

$ test.pure test.pure : 2 errors, 10 bytes leaked

This is a simple example. However, you can put complex processing into a script, or even in a program, and pass various conversion symbols as arguments to the script or program. For example:

$ setenv PURIFYOPTIONS '-run-at-exit="postprocess.csh %v %z %e %l "'

Table 1 and Table 2 show more details of the substitution strings for conversion symbols.

Table 1. Conversion symbols that can be used with Purify options

Character	Converts to
%v	Program executable name, lowercase V (name of the instrumented executable that you are running)
%V	Full path name of the program, uppercase V (/ replaced by _)
%p	Process ID (pid or PID)

Table 2. Conversion symbols that can be used in Exit-command (-run-at-exit)

Character	Converts to
%z	String value true or false, indicating whether any call chains for errors or leaks were printed (use it to have your exit script act conditionally when Purify finds something of interest to you)
%x	Program's exit status (0 if the program did not call an exit)
%e	Number of distinct access errors printed (displayed)
%E	Total number of errors printed
%l	Number of bytes of memory leaked (lowercase L)
%L	Number of bytes of memory potentially leaked (uppercase L)

Using program exit status


You have already learned how to run your scripts at the end of instrumented program run. Purify also gives you some information through its exit status. By default, Purify does not modify the normal exit status of your program. However, you can choose to have your program exit with a special exit status if Purify finds any access errors or memory leaks. This is a convenient way to flag failing runs in test suites. Use the -exit-status=yes option to enable Purify to insert flags that indicate types of runtime errors. If there are unsuppressed Purify errors, the status code is computed by doing bit-wise OR of the following values, depending upon the type of memory error present:

• 0x40: Memory access errors
• 0x20: Memory leaks
• 0x10: Potential memory leaks

Alternatively, you can replace the calls to exit(status) in your code and the return statement in main() function with a call to the purify_exit(status) function. (See Resources for the article on Purify Application Programming Interface functions.) If you are concerned only about the memory access errors, you can either turn off leak detection at exit by using the -leaks-at-exit=no option, or you can suppress memory leak and potential leak messages. You can also ignore the appropriate bits of exit status. However, the program summary message in the Purify report always shows your original exit status before any other Purify result status bits are OR'ed into it.
Listing 1 is an example that exploits the -exit-status option and uses the exit status to determine whether any errors were found in the program.

Listing 1. Exit status option example

$ cat prog.c #include <stdio.h> int main() { int i,j; i = j+1; /* UMR: Reading un-initialized variable j */ return 0; } $ purify -exit-status=yes cc -g prog.c -o prog.pure $ prog.pure $ echo $? 64

The exit value is not 0 (zero), as returned in function main. It is 64, which is 0x40 in hexadecimal. That is because Purify detects an Uninitialized Memory Read (UMR) memory access error in the program. This option can be easily incorporated in a script that checks the exit status after running the Purify'd executable and takes appropriate actions upon finding errors, such as filing a defect report with the test program or noting the result in the Purify log.

If you want your instrumented application to exit upon detecting the first error, you can use the -exit-on-error option. When you use that option, the program exits the moment that Purify encounters an error (errors that are hidden by using the suppress and kill directives do not count).


Mailing Purify results and assisting analysis

Purify has a -mail-to-user option that you can use to automate reporting of daily or weekly Purify results. When you use this option, Purify will e-mail the error report to the specified addresses of testers and developers, and they can verify the results when they receive the e-mail. For example, suppose that you purify your program this way:

$ purify -mail-to-user=yourid cc -g prog.c -o prog.pure

When you run the prog.pure executable thereafter, the Purify report will be sent automatically to yourid email address.

Sometimes, while analyzing the errors, it is useful to look at not only the function names but also other details, such as the complete path of the file where it is located or the PC values. You can enable Purify to display such information by using these options:

• -show-pc shows you the full PC value
• -show-pc-offset shows you the pc-offset from the start of the function
• -show-directory shows you the directory listing where the file containing the function exists (requires program build with debugging)

Example

In this section, you will see an example that uses most of the options that you learned about in this article. Listing 2 shows the GNUMakefile that contains modified build and test systems (see Downloads to get the source code used in this article). If the application name is memerrors, a new target is added for building a Purify'd application named memerrors.pure. Similarly, a new target is added to run tests with the Purify'd application.

Purify runtime options are set before running the test. For the -log-file option, a unique log file name is created, using conversion symbols and the date command (the file name includes program name, process ID, date, and time). The -run-at-exit option is used to indicate that, after program exits, the addsummary.sh script should be run, along with the arguments specified through conversion symbols (namely, the log file name, whether any error call chain was printed, exit status, count of memory errors found, size of memory leaks, and potential memory leaks). Since the -exit-status=yes option is not used, Purify will not overwrite the exit status and retain the original exit status of the program.

Listing 2. GNUMakefile with modified build and test systems

# Name of Logfile using Purify conversion symbols and date command DATEANDTIME := `date +%Y_%b_%d_%H_%M_%S` LOGFILENAME := %v_pid%p_$(DATEANDTIME).plog # Script to run when Purify'ed program exits PURIFYEXITSCRIPT:= \"addsummary.sh $(DATEANDTIME) $(LOGFILENAME) %z %x %e %l %L\" # Purify Options PURIFYOPTIONS := -log-file=$(LOGFILENAME) -run-at-exit=$(PURIFYEXITSCRIPT) # Targets and Rules all: runtest runpurifytest # Clean clean: $(RM) memerrors memerrors.pure # Build Application memerrors: memerrors.c $(CC) -o $@ $? # Build Purify'ed application memerrors.pure: memerrors.c purify $(CC) -g -o $@ $? # Run Test Suite runtest: memerrors ./memerrors # Run Test Suite with Purify'ed application runpurifytest: memerrors.pure echo Starting test at $(DATEANDTIME) ..... env PURIFYOPTIONS="$(PURIFYOPTIONS)" ./memerrors.pure # End of GNUMakefile

The addsummary.sh script shown in Listing 3 creates an HTML report. It maintains a list file that has one HTML table row for each Purify run so far, in reverse chronological order. When the script is executed upon exiting Purify, it creates a new list file that contains an HTML table row for the latest run, appends the file with previous rows, and replaces the old list file with the new list file. Then it generates an HTML file by an wrapping HTML header and footer around the list file.

Listing 3.Content of the addsummary.sh shell script

#!/bin/sh DATEANDTIME=$1 LOGFILENAME=$2 LOGFULLNAME=`pwd`/$LOGFILENAME ERRORFOUND=$3 EXITSTATUS=$4 ERRORCOUNT=$5 LEAKSIZE=$6 PLEAKSIZE=$7 PURIFYREPORT="purify_reports" REPORTLIST="$PURIFYREPORT.list" REPORTNEWLIST="$REPORTLIST.new" REPORTHTML="$PURIFYREPORT.html" # Start echo Processing $LOGFILENAME created at $DATEANDTIME # Create report list file if it does not exist touch $REPORTLIST # Create a row for the latest Purify run echo "<tr>" >> $REPORTNEWLIST echo "<td>$DATEANDTIME</td>" >> $REPORTNEWLIST if ($ERRORFOUND == "true"); then echo "<td>FAILED</td>" >> $REPORTNEWLIST else echo "<td>Pass</td>" >> $REPORTNEWLIST fi echo "<td>$EXITSTATUS</td>" >> $REPORTNEWLIST echo "<td>$ERRORCOUNT</td>" >> $REPORTNEWLIST echo "<td>$LEAKSIZE bytes</td>" >> $REPORTNEWLIST echo "<td>$PLEAKSIZE bytes</td>" >> $REPORTNEWLIST echo "<td><a href=\"$LOGFULLNAME\">$LOGFILENAME</a></td>" >> $REPORTNEWLIST echo "</tr>\n" >> $REPORTNEWLIST # Add this row at the beginning of the table cat $REPORTLIST >> $REPORTNEWLIST mv $REPORTNEWLIST $REPORTLIST # Create HTML page # Header echo "<html>" > $REPORTHTML echo "<body>" >> $REPORTHTML echo "<table border=1>" >> $REPORTHTML echo "<caption>Purify Test Summary</caption>" >> $REPORTHTML echo "<tr>" >> $REPORTHTML echo "<th>Date & Time</th><th>Result</th><th>Exit Status</th>" >> $REPORTHTML echo "<th>Errors</th><th>Leaks</th><th>Potential Leaks</th>" >> $REPORTHTML echo "<th>Log File</th>" >> $REPORTHTML echo "</tr>\n" >> $REPORTHTML # Add rows for Purify results cat $REPORTLIST >> $REPORTHTML # Footer echo "</table>" >> $REPORTHTML echo "</body>" >> $REPORTHTML echo "</html>" >> $REPORTHTML # Done echo "Successfully updated $REPORTHTML" # End of addsummary.sh

Figure 2 shows the HTML page generated after three runs of the test suite. Each run is represented by a row, and each row has a hyperlink to the Purify log file. Each successive run of the test suite will add a new row at the beginning of the table.

Figure 2. Purify Test Summary report in the browser


Summary

As this article explains, you get maximum benefits when you use Purify regularly and systematically. You now know how to incorporate Purify into your software development and testing process and to automate its use with the help of conversions symbols and options.
Although the example used in this article is simple, it demonstrates how easy it is to integrate Purify into your build and test environment and the value of automating Purify usage. Think about the simplicity of checking the Purify test results summary on a Web page that gets updated automatically every time your test suite is executed. All existing log files are also accessible through the same Web page. The example here is intentionally simple, just to show you the possibilities. You can create a quite sophisticated system that compares results and sends e-mail notifications with precise details upon finding any additional memory errors and leaks. You can fix them as soon as they are introduced. With this knowledge, you are ready to reap the maximum benefits of Rational Purify.