Don't run tests under xvfb-run unnecessarily (#185)

We seem to be using xvfb-run in the `test-with-pypi.yml` GitHub Action,
and it's not clear why. This PR removes it.

.github/workflows/test-with-pypi.yml

@@ -24,14 +24,7 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Install package under test
         run: python -m pip install .
-      - name: Run tests (Ubuntu)
-        if: matrix.os == 'ubuntu-latest'
-        run: |
-          mkdir testdir
-          cd testdir
-          xvfb-run -a python -X faulthandler -m unittest discover -v scimath
-      - name: Run tests (not Ubuntu)
-        if: matrix.os != 'ubuntu-latest'
+      - name: Run tests
         run: |
           mkdir testdir
           cd testdir