Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabled code coverage measurement #280

Draft
wants to merge 2 commits into
base: oe_port
Choose a base branch
from
Draft

Enabled code coverage measurement #280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dc0d890
Enabled code coverage measurement
jxyang May 20, 2020
2f157ae
Address reiew comments and update commit sha to sgx-lkl-musl
jxyang May 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .azure-pipelines/scripts/measure_code_cov.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it makes sense that there are new scripts which duplicate the non-coverage test logic. Why not extend the existing scripts and add an extra environment variable to enable it?

Copy link
Contributor Author

@jxyang jxyang May 26, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are key differences between this script and the test runner script.

  • This script is conditioned on the fact that sgx-lkl was built with CODE_COVERAGE=1
  • We ideally want to include samples in the code coverage measurement
  • For each test/sample, we run run-hw and run-sw back to back, and extract the collective coverage data from the image.
  • The objective is to measure code coverage. So the coverage data for failed test are included in the final aggregated code coverage metrics.

If you compare this script and test_runner, also measure_one_cov with run_test, there aren't many duplicated code. I prefer to keep them separate for easier maintenance.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining. I'm trying to think how this fits into CI. Given that debug/release build modes have slightly different code paths it may make sense to capture code coverage in both modes and merge it. Or we just do it for one mode if the differences are not worth measuring. Either way, it would mean adding at least one new build job with CODE_COVERAGE=1 and corresponding test/coverage job(s).
You're saying that your script runs both hw and sw. This is a bit unfortunate because it prevents running sw on non-SGX machines and in parallel. Would it be possible to run them separately, dump the coverage files and merge them in a follow-up step?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's hard to merge coverage data from two VMs. It requires a global vm for aggregation. We will leave the work of integrating into pipeline in another PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's actually not that hard to do, you can specify dependencies between jobs and use build artifacts to upload/download job data.


if [ -z $SGXLKL_ROOT ]; then
echo "ERROR: 'SGXLKL_ROOT' is undefined. Please export SGXLKL_ROOT=<SGX-LKL-OE> source code repository"
exit 1
fi

# TODO: add samples to code coverage measurement.
# For now, only measure anything under 'tests' except for LTP.
test_folder_name=$SGXLKL_ROOT/tests
test_folder_identifier="Makefile"
test_exception_list="ltp"

file_list=( $(sudo find $test_folder_name -name $test_folder_identifier | grep -v "$test_exception_list") )

total_tests=${#file_list[@]}
counter=0

rm -f $SGXLKL_ROOT/total_cov.info

for file in ${file_list[@]};
do
counter=$(($counter + 1))
folder=$(dirname $file)
echo "$counter/$total_tests: Measuring code coverage in $folder"
cd $folder
$SGXLKL_ROOT/.azure-pipelines/scripts/measure_one_cov.sh
make clean
done

echo "Done! All coverage data are aggregated to $SGXLKL_ROOT/total_cov.info"
57 changes: 57 additions & 0 deletions .azure-pipelines/scripts/measure_one_cov.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/bin/bash

if [ -z $SGXLKL_ROOT ]; then
echo "ERROR: 'SGXLKL_ROOT' is undefined. Please export SGXLKL_ROOT=<SGX-LKL-OE> source code repository"
exit 1
fi

if [ ! -f "Makefile" ]; then
echo "ERROR: ${0} can only be invoked from a directory that contains Makefile"
exit 1
fi

# Get the timeout from the test module
DEFAULT_TIMEOUT=300
timeout=$(make gettimeout 2> /dev/null)
[[ $? -ne 0 ]] && timeout=$DEFAULT_TIMEOUT
echo "Execution timeout: $timeout"

timeout --kill-after=$(($timeout + 15)) $timeout make run-hw
timeout --kill-after=$(($timeout + 15)) $timeout make run-sw

if ls *.img 1> /dev/null 2>&1; then
mkdir img
sudo umount img
sudo mount -o loop *.img img
sudo rm -rf $SGXLKL_ROOT/cov
mkdir $SGXLKL_ROOT/cov

# Gather all necessary files for lcov
cp -r $SGXLKL_ROOT/src/* $SGXLKL_ROOT/cov
sudo cp -r img$SGXLKL_ROOT/build_musl $SGXLKL_ROOT/cov
sudo cp -r img$SGXLKL_ROOT/sgx-lkl-musl $SGXLKL_ROOT/cov
sudo cp -r $SGXLKL_ROOT/build_musl $SGXLKL_ROOT/cov
sudo cp -r $SGXLKL_ROOT/sgx-lkl-musl $SGXLKL_ROOT/cov

echo "Creating $SGXLKL_ROOT/cov.info"
sudo lcov -d $SGXLKL_ROOT/cov -c -o $SGXLKL_ROOT/cov.info

# Accumulate the coverage data with data from other tests
if [ ! -f "$SGXLKL_ROOT/total_cov.info" ]; then
echo "Copy the code coverage for the 1st run"
sudo mv $SGXLKL_ROOT/cov.info $SGXLKL_ROOT/total_cov.info
else
echo "Aggregating code coverage to $SGXLKL_ROOT/total_cov.info..."
sudo lcov -a $SGXLKL_ROOT/total_cov.info -a $SGXLKL_ROOT/cov.info -o $SGXLKL_ROOT/total_cov.info
fi

# clean up
echo "Cleaning up..."
sudo umount img
rm -rf img
else
echo "ERROR: disk image is not created"
exit 1
fi

exit 0
7 changes: 7 additions & 0 deletions config.mak
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ LKL_DEBUG ?= false
# Select libc version (currently only musl libc is supported)
LIBC ?= musl

# Measure code coverage
CODE_COVERAGE ?= false

SGXLKL_ROOT ?= $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
$(info $$SGXLKL_ROOT = [${SGXLKL_ROOT}])

Expand Down Expand Up @@ -131,6 +134,10 @@ else

endif

ifeq ($(CODE_COVERAGE),true)
SGXLKL_CFLAGS_ENCLAVE_EXTRA += -fprofile-arcs -ftest-coverage
jxyang marked this conversation as resolved.
Show resolved Hide resolved
endif

ifeq ($(DEBUG),true)
CMAKE_BUILD_TYPE=Debug
else
Expand Down
2 changes: 1 addition & 1 deletion sgx-lkl-musl
Open in desktop
Submodule sgx-lkl-musl updated 7 files
+5 −1 Makefile
+1 −0 ldso/dynlink.c
+ − libgcov_musl.a
+1 −0 src/env/__init_tls.c
+1 −6 src/internal/syscall.h
+1 −0 src/misc/syscall.c
+44 −17 src/thread/x86_64/clone.s
2 changes: 1 addition & 1 deletion src/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ CROSS_COMPILE =
RANLIB = $(CROSS_COMPILE)ranlib

CFLAGS_MAIN = -I$(OE_SDK_INCLUDES)
CFLAGS_ENCLAVE = -I$(OE_SDK_INCLUDES) -I${OE_SDK_INCLUDES}/openenclave/3rdparty -fPIE
CFLAGS_ENCLAVE = ${SGXLKL_CFLAGS_ENCLAVE_EXTRA} -I$(OE_SDK_INCLUDES) -I${OE_SDK_INCLUDES}/openenclave/3rdparty -fPIE
LINK_MAIN =

GIT_VERSION = "$(shell git describe --dirty --always --tags || echo unknown)"
Expand Down
2 changes: 1 addition & 1 deletion tests/basic/abort/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ PROG=abort
PROG_C=abort.c

DISK_IMAGE=sgxlkl-abort.img
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/clock/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include ../../common.mk

PROG=clock
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/exit/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ TEST3=raise-test
SRCS=$(wildcard *.c)

DISK_IMAGE=sgxlkl-exit-test.img
IMAGE_SIZE=5M
IMAGE_SIZE=50M

SGXLKL_ENV=SGXLKL_VERBOSE=1 SGXLKL_KERNEL_VERBOSE=1 SGXLKL_TRACE_SIGNAL=1

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/global_vars_test/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ include ../../common.mk
PROG=global_vars_test
PROG_SRC=$(PROG).c
PROG_ARGS=-a -b -c hello
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/helloworld/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include ../../common.mk

PROG=helloworld
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/illegal_instructions/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include ../../common.mk

PROG=illegal_instructions-test
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/nonroot_halt_test/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include ../../common.mk

PROG=nonroot_halt_test
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/pthread_join/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include ../../common.mk

PROG=pthread_join-test
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/signal/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include ../../common.mk

PROG=signal
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/sleep/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include ../../common.mk

PROG=sleep-test
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/basic/stat/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include ../../common.mk

PROG=stat
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down
2 changes: 1 addition & 1 deletion tests/virtio/ping_test/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ SHELL := /bin/bash

PROG=dummy_server
PROG_SRC=$(PROG).c
IMAGE_SIZE=5M
IMAGE_SIZE=50M

EXECUTION_TIMEOUT=60

Expand Down