Test Bench Control Part Eight

Summary

Putting it all together

The github repository contains a complete example that is available to pull, fork, examine and run. You can also peruse the source code online.
There are several prerequisites. You will need either verilator or icarus verilog installed (preferably both), as well as the boost development libraries (format and function at least).

Incomplete Example

This is an example - an experiment really. No specification or architecture as I advocated, just a first, exploratory, system. Amongst the missing features would be

• Show older tests. A big omission as it currently stands. All tests are resident in the database but the UI doesn't give access currently. Need to add navigation buttons to see older log entries.
• Allow filtering of all tests. Need to add UI function to filter by user/block.
• More mining (history of test failures, what/who causes breaks, source repository version, growth of test suite versus ...)
• Filtering messages by ident, scope, filename. To separate out messages from a particular source e.g. a monitor. This might be particularly useful to view the output from two monitors with interesting interaction.
• Arbitrary Attributes on messages (retaining rich semantics of original message context) to allow python style string formatting i.e. %(attribute_name)s. Also allows arbitrary data to be stored that can be easily retrieved when used in conjunction with a message identifier. I have used this elsewhere to successfully store functional coverage information of hierarchy, dimensions and enumerations.
• Functional Coverage. Hopefully I will post more on this in future.
• Accounting for tests expected to fail (test_fail.py, I always include a test that just passes and one that just fails in any regression set to test mechanisms and test driven development deems that these be the first two to write!)
• Build in profiling and performance regression testing infrastructure.
• Recoding database commit code in C++ for speed.
• Extensions using e.g. MySQL and MongoDB.
• Examination of other uses including integrating synthesis and P&R to record results as part of a regression. (Yes I think running synthesis and P&R is a useful addition to function regression testing).
• Block level dashboard - regression,  coverage, synthesis and layout view of block summarised in one place showing trends and current status.

I hope someone, somewhere has found these posts useful.