Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tests] add cover demo #926

Merged
merged 7 commits into from
Jan 7, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 52 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ $ nix develop -c t1-helper run -i <top-name> -c <config-name> -e <emulator-type>
wheres
- `<config-name>` is the configuration name
- `<top-name>` is one of the `t1emu`, `t1rocketemu`
- `<emulator-type>` is one of the `verilator-emu`, `verilator-emu-trace`, `vcs-emu`, `vcs-emu-trace`
- `<emulator-type>` is one of the `verilator-emu`, `verilator-emu-trace`, `vcs-emu`, `vcs-emu-trace`, `vcs-emu-cover`
- `<case-name>` is the name of a testcase, you can resolve runnable test cases by command: `t1-helper listCases -c <config-name> <regexp>`

For example:
Expand Down Expand Up @@ -206,6 +206,12 @@ $ nix develop -c t1-helper check
The `t1-helper check` subcommand will read RTL event produced in `run` stage,
so make sure you `run` a test before `check`.

To get the coverage report, use the `vcs-emu-cover` emulator type:

```console
$ nix develop -c t1-helper run -i t1emu -c blastoise -e vcs-emu-cover mlir.hello
```

#### Export RTL Properties

```shell
Expand Down Expand Up @@ -328,6 +334,51 @@ $ nix build .#t1.<config-name>.<top-name>.cases.intrinsic.matmul -L
$ ls -al ./result
```

#### Developing Coverage

To develop coverage, use the following steps:

1. Write the coverpoint description file at the same level as the test case source code.
2. Update the `default.nix` file to parse the coverpoint description file.

For example, to develop coverage for the `mlir.hello` test case:

tests/mlir/hello/hello.json:
```json
{
"assert": [
sequencer marked this conversation as resolved.
Show resolved Hide resolved
{
"name": "vmv_v_i",
"description": "single instruction vmv.v.i"
Comment on lines +351 to +352
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you align the RTL coverpoint(from LTL) to this name?

Copy link
Contributor Author

@Clo91eaf Clo91eaf Jan 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are actually two rules here:

  1. When Chisel lowers to sv, it will replace . in the label of LTL in the Chisel code with _.
  2. VCS enables the assertions based on wildcard matching of the name in the full design:
    If a full hierarchical name is provided, such as A.C, vcs will determine the matching label. If only the last name is provided, such as C, vcs will still match the same label A.C. It is worth noting that if If there is another name, such as B.C, then vcs will match both A.C and B.C. And Some wildcard operation are also allowed, refer to the Coverage Technology Reference Manual of VCS
    So the 'vmv_v_i' here actually corresponds to the full hierachy name 'TestBench.dut.verification.vmv_v_i'

}
],
"tree": [],
"module": []
}
```

tests/mlir/default.nix:
```shell
if [ -f ${caseName}.json ]; then
${jq}/bin/jq -r '[.assert[] | "+assert " + .name] + [.tree[] | "+tree " + .name] + [.module[] | "+module " + .name] | .[]' \
${caseName}.json > $pname.cover
else
echo "-assert *" > $pname.cover
fi
```

Then, you can run the test building script to check if the coverage is generated correctly:

```shell
nix build .#t1.blastoise.t1emu.cases.mlir.hello -L
```

Use the `vcs-emu-cover` emulator type to run the test case and generate the coverage report:

```shell
nix develop -c t1-helper run -i t1emu -c blastoise -e vcs-emu-cover mlir.hello
```

### Bump Dependencies
Bump nixpkgs:
```shell
Expand Down
2 changes: 2 additions & 0 deletions nix/t1/conversion/sv-to-vcs-simulator.nix
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ stdenv.mkDerivation rec {
"assert"
"-cm_dir"
"./cm"
"-assert"
"enable_hier"
]
++ lib.optionals (!enableCover) [
"-assert"
Expand Down
2 changes: 2 additions & 0 deletions nix/t1/run/run-vcs-emu.nix
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ stdenvNoCC.mkDerivation (rec {
++ lib.optionals emulator.enableCover [
"-cm"
"assert"
"-assert"
"hier=${testCase}/${testCase.pname}.cover"
]
++ lib.optionals emulator.enableTrace [
"+t1_wave_path=${testCase.pname}.fsdb"
Expand Down
38 changes: 31 additions & 7 deletions script/emu/src/Main.scala
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,13 @@ object Main:
attr
) ++ extraArgs
os.proc(args).call().out.trim()

def resolveTestElfPath(
def resolveTestPath(
ip: String,
config: String,
caseName: String,
forceX86: Boolean = false
): os.Path =
): os.Path =
val casePath = os.Path(caseName, os.pwd)
if (os.exists(casePath)) then return casePath

Expand All @@ -61,11 +61,35 @@ object Main:
val nixStorePath = resolveNixPath(
s"${caseAttrRoot}t1.${config}.${ip}.cases.${caseName}"
)
val elfFilePath = os.Path(nixStorePath) / "bin" / s"${caseName}.elf"
val filePath = os.Path(nixStorePath)

filePath
end resolveTestPath

def resolveTestElfPath(
ip: String,
config: String,
caseName: String,
forceX86: Boolean = false
): os.Path =
val testPath = resolveTestPath(ip, config, caseName, forceX86)
val elfFilePath = testPath / "bin" / s"${caseName}.elf"

elfFilePath
end resolveTestElfPath

def resolveTestCoverPath(
ip: String,
config: String,
caseName: String,
forceX86: Boolean = false
): os.Path =
val testPath = resolveTestPath(ip, config, caseName, forceX86)
val coverFilePath = testPath / s"${caseName}.cover"

coverFilePath
end resolveTestCoverPath

def resolveTestBenchPath(
ip: String,
config: String,
Expand Down Expand Up @@ -203,8 +227,8 @@ object Main:
s"Using config=${BOLD}${finalConfig.get}${RESET} emulator=${BOLD}${finalEmuType.get}${RESET} case=${BOLD}$caseName${RESET}"
)

val caseElfPath =
resolveTestElfPath(finalIp.get, finalConfig.get, caseName, forceX86)
val caseElfPath = resolveTestElfPath(finalIp.get, finalConfig.get, caseName, forceX86)
val caseCoverPath = resolveTestCoverPath(finalIp.get, finalConfig.get, caseName, forceX86)
val outputPath = prepareOutputDir(outDir.getOrElse("t1-sim-result"))
val emulator = resolveTestBenchPath(finalIp.get, finalConfig.get, finalEmuType.get)

Expand All @@ -216,7 +240,7 @@ object Main:
)
++ optionals(timeout.isDefined, Seq(s"+t1_timeout=${timeout.getOrElse("unreachable")}"))
++ optionals(isTrace, Seq(s"+t1_wave_path=${outputPath / "wave.fsdb"}"))
++ optionals(isCover, Seq(s"-cm assert"))
++ optionals(isCover, Seq("-cm", "assert", "-assert", s"hier=${caseCoverPath}"))
++ optionals(!leftOverArguments.isEmpty, leftOverArguments)

if dryRun.value then return
Expand Down
Loading