Test Bench Control Part Five

Unified Reporting

Previous posts have detailed the integration of Python into the simulation to allow greater interaction between a script and the simulation, now we also import the logging API into Python. We reuse the same process as for the VPI, SWIG the API and then thinly wrap in Python.
Our motivation here is to make the test process more robust by using the same reporting mechanisms so that we are unlikely as possible to miss an ERROR and thus report a failing test as passing. Traditionally the standard error and standard output of the invoking simulation script will be concatenated and grep'ed (or some regular expression parsing) for ERROR. This is error prone with multiple streams being interleaved, carriage returns and other noise in the resulting stream.
We also loose a great deal of useful information when a logging item is serialized to text. Sometimes a regular expression is used to extract some of that information (e.g. file name or line number) but this is error prone for the fore mentioned reasons. When a format string like "%s error, file %s, line %d" is used we are losing the structure of the information (the error string itself, file name and line number) that we once had and then need to kludge a mechanism to get it back later. What if this information was always kept with the log message so we could later analyse this data? If we had other output streams in addition to text that had richer semantics we could store this data and mine it to produce e.g. triage tables to aid regression debugging.

Messaging API

Most teams seem to have a generic messaging/logging library for their test benches. I have seem them written in both Verilog and C++. This example is written in C++, has some fixed severity levels with programmable verbosity, accounting and callbacks. Some macros are available that provide a convenient interface to record some attributes (filename, line) automatically when used.

This API is also available in Verilog (as above).

 `EXM_INFORMATION("python input set to %s", sim_ctrl_python_filename_r);

We also SWIG this API and make it available in Python thus

  import message

  # adjust output verbosity by filtering echo level
  message.message.instance.verbosity(0)
  # adjust echo control of particular severity
  message.control[message.NOTE].echo = 0
  # as above using artefact of wrapper
  message.control.DEBUG.echo = 1
  # adjust number of messages allowed prior to exit
  # [here set to unlimited]
  message.control.FATAL.threshold = -1

  # emit a warning with format
  message.warning('a warning %(c)d', c=666)
  # emit a string message
  message.note('a note')
  message.success('should be success')
  message.internal('whoops')

It's a bit more complicated this time as it allows us to add python callbacks (more of that follows), but the result of this is output like

  (     WARNING 14:51:17) a warning 666
  (     SUCCESS 14:51:17) should be success
  (    INTERNAL 14:51:17) whoops
  (       FATAL 14:51:17) Too many INTERNAL
  ( INFORMATION 14:51:17)        FATAL : 1
  ( INFORMATION 14:51:17)     INTERNAL : 1
  ( INFORMATION 14:51:17)        ERROR : 0
  ( INFORMATION 14:51:17)      WARNING : 1
  ( INFORMATION 14:51:17)      SUCCESS : 1
  ( INFORMATION 14:51:17)         NOTE : 1
  ( INFORMATION 14:51:17)  INFORMATION : 7
  ( INFORMATION 14:51:17)        DEBUG : 1
  ( INFORMATION 14:51:17)    INT_DEBUG : 0

But we have control over what is displayed and how in Python, and we can use Python mechanisms to control what happens for all sources of messages in the same manner as if using the C++ API.

Message Callbacks

The messaging library has an architecture based on callbacks. We can register an arbitrary number of callbacks on two events

1. Message emission - when a message is logged
2. Termination - when the messaging library is closed as the program exits.

The default display to the standard output is the default installed callback for the emission event, along with an accounting function that keeps track of the number of messages and their severity. The summary table (as above) function is added by default for the termination event. They are managed as hashed lists so can be deleted or replaced with new functions, and new arbitrary functions can also be added.

These callback lists are also available from Python, where we can add callbacks that run in the embedded Python interpreter (see next post regarding test result databases). The example contains no demonstration of using this library from Verilog. It should be possible to provide some access via the DPI with perhaps an exported function, but an elegant solution may be hindered by the lack of a mechanism to pass a task via a reference in System Verilog.

Putting together with VPI

When I first came to write this Verilator did not have a rich enough implementation of vpi_put_value and vpi_get_value, so I added them and the patch has now been folded into the mainline Verilator code. The patch did include a test coded in C++, but I expanded it further in Python and it is available here. It does violate the "do not code test bench logic in Python" rule, but is an interesting example as uses Python's int(), bin(), oct() and hex() as a reference model - whilst using the verilog and message libraries described above.
The test is also created using a simple framework that registers the two separate prologue and epilogue code fragments. Note that the epilogue will only run if the end of simulation is reached. Any error causing the test to exit prematurely will execute the fatal method instead. This is the body of the test_pass.py test

  # Copyright (c) 2012, 2013 Rich Porter - see LICENSE for further details

  import message
  import test

  ############################################################################

  class thistest(test.test) :
    name='test mdb pass'
    def prologue(self) :
      message.message.verbosity(message.INT_DEBUG)
      message.warning('a warning %(c)d', c=666)
      message.note('a note')
    def epilogue(self) :
      message.success('should be success')
    def fatal(self) :
      'Should not be executed'
      message.fatal('Should not be executed')

  ############################################################################

  testing = thistest()

There are a number of other tests in this directory that excercise the VPI/verilog library more. Please peruse at your leisure.

Conclusion

It is possible to fold a level of test scripting into a Verilog test bench simulation that provides easy access to the simulation hierarchy whilst providing a consistent logging interface. This removes any requirement for regular expression parsing of simulation output to determine pass/fail status.
Next we will look at storing messages in a database for later retrieval and analysis.