Convert compliance tests to yaml and clean up tests (#69)

* Convert compliance tests to yaml and clean up tests

* Add "comment" field to function tests

* Added instructions for running tests.

* Renamed grammar compliance schema file.

* Clean up error output

* Handle non-exact error output

* Clean up more failing tests

Co-authored-by: springcomp <[email protected]>

.github/workflows/test_validate.yaml

@@ -18,7 +18,7 @@ jobs:
     steps:
       - uses: actions/checkout@v2
       - id: set-matrix
-        run: echo "::set-output name=matrix::$(ls grammar/*.json | jq -R -s -c 'split("\n")[:-1]')"
+        run: echo "::set-output name=matrix::$(ls grammar/*.yml | jq -R -s -c 'split("\n")[:-1]')"
   validate-tests:
     needs: list-tests
     runs-on: ubuntu-latest
@@ -33,7 +33,7 @@ jobs:
           cache-dependency-path: .github/workflows/validate_requirements.txt
       - run: pip install -r .github/workflows/validate_requirements.txt
       - name: Validate grammar tests
-        run: "check-jsonschema --schemafile test_schema.yml ${{ matrix.manifest }}"
+        run: "check-jsonschema --schemafile grammar_schema.yml ${{ matrix.manifest }}"
 
   list-functions:
     runs-on: ubuntu-latest

README.md

@@ -1,4 +1,4 @@
-# JMESPath Enhancement Proposals
+# JMESPath Specification
 
 Any changes to the JMESPath specification
 (https://jmespath.site/specification.html) must have a JEP (JMESPath Enhancement
@@ -50,3 +50,15 @@ these tenets gives your proposal a higher likelihood of being accepted:
   constructs that are difficult to implement in another language.
 * JMESPath strives to have one way to do something.
 * Features are driven from real world use cases.
+
+# JMESPath Compliance Tests
+
+This repo contains a suite of JMESPath compliance tests. JMESPath's implementations can use these tests in order to verify their
+implementation adheres to the JMESPath spec.
+
+## Test Organization
+
+`grammar/*.yml` contains tests for general grammar functionality. These documents must validate against the `grammar_schema.yml`.
+
+`functions/*.yml` contains a description of each function accompanied by a suite of tests/examples. These documents must validate against the `function_schema.yml`.
+

bin/README.md

@@ -0,0 +1,29 @@
+# JMESPath Compliance Tests
+
+This repo contains a suite of JMESPath compliance tests. JMESPath's implementations can use these tests in order to verify their
+implementation adheres to the JMESPath spec.
+
+## Test Organization
+
+`grammar/*.yml` contains tests for general grammar functionality. These documents must validate against the `grammar_schema.yml`.
+
+`functions/*.yml` contains a description of each function accompanied by a suite of tests/examples. These documents must validate against the `function_schema.yml`.
+
+## Running Tests
+
+This repository include a Python `jp-compliance` executable test runner.
+
+```
+jp-compliance <executable> <test yaml> [<test name>..]
+```
+
+This will run the JMESPath compliance tests against a JMESPath executable.
+The executable must accept the query as the only argument, and the input data on stdin.
+The result must be printed to stdout as JSON.
+Errors must be printed to stderr and match expected values.
+
+If your executable requires additional arguments, wrap it in an executable script.
+
+The test YAML must validate against function_schema.yml or test_schema.yml.
+
+Additionally, test names can be supplied to only execute matching tests