Skip to content

Commit

Permalink
Gh-35: Refactor tests to use tox and pytest (#36)
Browse files Browse the repository at this point in the history 
* Refactor test suite to use tox and pytest, and have separate directories for unit and integration tests. Update READMEs, and fishbowl scripts and jinja templates. Refactor fishbowl to generate import strings.

* Copyright and licence
  • Loading branch information
@GCHQDeveloper314
GCHQDeveloper314 authored Jan 5, 2024
1 parent e78aac8 commit a94ccae
Show file tree
Hide file tree
Showing 51 changed files with 3,431 additions and 2,767 deletions.
12 changes: 7 additions & 5 deletions .github/workflows/continuous-integration.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,14 @@ jobs:
curl -o jetty-runner.jar https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.52.v20230823/jetty-runner-9.4.52.v20230823.jar
curl -o road-traffic-rest.war https://repo1.maven.org/maven2/uk/gov/gchq/gaffer/road-traffic-rest/$gafferVersion/road-traffic-rest-$gafferVersion.war
java \
-Dgaffer.storeProperties=src/test/road-traffic-example/store.properties \
-Dgaffer.graph.config=src/test/road-traffic-example/graphConfig.json \
-Dgaffer.schemas=src/test/road-traffic-example/schema \
-DroadTraffic.dataLoader.dataPath=src/test/road-traffic-example/roadTrafficSampleData.csv \
-Dgaffer.storeProperties=tests/road-traffic-example/store.properties \
-Dgaffer.graph.config=tests/road-traffic-example/graphConfig.json \
-Dgaffer.schemas=tests/road-traffic-example/schema \
-DroadTraffic.dataLoader.dataPath=tests/road-traffic-example/roadTrafficSampleData.csv \
-jar jetty-runner.jar --path rest road-traffic-rest.war &
sleep 1m
- name: Run Python tests
run: python3 -m unittest discover -s src
run: |
python3 -m pip install ".[dev]"
python3 -m tox
2 changes: 1 addition & 1 deletion .github/workflows/update-gaffer-version.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
run: |
newVersion=${{ github.event.inputs.version }}
curl -o spring-rest.jar https://repo.maven.apache.org/maven2/uk/gov/gchq/gaffer/spring-rest/$newVersion/spring-rest-$newVersion-exec.jar
java -Dgaffer.schemas=src/test/road-traffic-example/schema -Dgaffer.storeProperties=src/test/road-traffic-example/federatedStore.properties -Dgaffer.graph.config=src/test/road-traffic-example/federatedGraphConfig.json -jar spring-rest.jar &
java -Dgaffer.schemas=tests/road-traffic-example/schema -Dgaffer.storeProperties=tests/road-traffic-example/federatedStore.properties -Dgaffer.graph.config=tests/road-traffic-example/federatedGraphConfig.json -jar spring-rest.jar &
sleep 1m
python src/generate.py
rm spring-rest.jar
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ celerybeat.pid

# Environments
.env
.venv
.venv*
.venv-python-example
env/
venv/
Expand Down
240 changes: 127 additions & 113 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,113 +1,127 @@
<img align="right" width="200" height="auto" src="https://github.com/gchq/Gaffer/raw/develop/logos/logo.png">

# Gaffer Python Client

![ci](https://github.com/gchq/gafferpy/actions/workflows/continuous-integration.yaml/badge.svg)
[<img src="https://img.shields.io/badge/docs-passing-success.svg?logo=readthedocs">](https://gchq.github.io/gafferpy/)

## Features

- Persistently connect to a Gaffer rest api to run operations
- Connect using PKI certificates and SSL
- Generate Python client code for custom Operations, Predicates, Binary Operators and Functions
- Turn existing json queries into Python objects

## Installation

Gafferpy requires Python 3.6+. We don't currently release gafferpy on pypi, but you can install it over ssh with:

```bash
pip install git+https://github.com/gchq/gafferpy.git
```

Or if you have the source code locally and want any changes you make reflected in your installation, you can run:

```bash
pip install -e .
```

## Quick Start

The python shell connects to a running Gaffer REST API.
You can start the Gaffer road-traffic-demo rest server from the Gaffer repository, using the command:

```bash
mvn clean install -pl :road-traffic-demo -Proad-traffic-demo,quick
```

```python
# Import the client library and connector
from gafferpy import gaffer as g
from gafferpy import gaffer_connector

# Instantiate a connector
gc = gaffer_connector.GafferConnector("http://localhost:8080/rest/latest")

# You can use the connector to perform get requests
schema = gc.execute_get(g.GetSchema())

# And also run operations
elements = gc.execute_operation(
operation=g.GetAllElements()
)

# Multiple operations
elements = gc.execute_operations(
operations=[
g.GetAllElements(),
g.Limit(result_limit=3)
]
)

# And operation chains
elements = gc.execute_operation_chain(
operation_chain=g.OperationChain(
operations=[
g.GetAllElements(),
g.Limit(
truncate=True,
result_limit=3
)
]
)
)
```

See [operation examples](https://gchq.github.io/gaffer-doc/v1docs/getting-started/operations/contents) for more examples of operations in python.

## Coding Style
Please ensure that your coding style is consistent with the rest of the Gaffer project. Guides on the coding style for Gaffer can be found [here](https://gchq.github.io/gaffer-doc/latest/ways-of-working/#coding-style)

## Testing

```bash
# To run all of the tests, first deploy the road traffic example
# This needs to be done from the Gaffer java repository
mvn clean install -pl :road-traffic-demo -Proad-traffic-demo,quick

# Then run
python -m unittest discover
```

## Building the documentation

To build the docs locally, assuming you have Python installed, install the docs dependencies:
```bash
cd docs
pip install -r requirements.txt
```

Then use the sphinx Makefile:
```bash
make html
```

## License

Copyright 2016-2022 Crown Copyright

Licensed under the Apache License, Version 2.0 \(the "License"\); you may not use this file except in compliance with the License. You may obtain a copy of the License at

[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
<img align="right" width="200" height="auto" src="https://github.com/gchq/Gaffer/raw/develop/logos/logo.png">

# Gaffer Python Client

![ci](https://github.com/gchq/gafferpy/actions/workflows/continuous-integration.yaml/badge.svg)
[<img src="https://img.shields.io/badge/docs-passing-success.svg?logo=readthedocs">](https://gchq.github.io/gafferpy/)

## Features

- Persistently connect to a Gaffer rest api to run operations
- Connect using PKI certificates and SSL
- Generate Python client code for custom Operations, Predicates, Binary Operators and Functions
- Turn existing json queries into Python objects

## Installation

`gafferpy` requires Python 3.6+. We do not currently release `gafferpy` on PyPI, but you can install it over ssh with:

```bash
pip install git+https://github.com/gchq/gafferpy.git
```

Or if you have the source code locally and want any changes you make reflected in your installation, you can run:

```bash
pip install -e .
```

## Quick Start

The Python shell connects to a running Gaffer REST API.
You can start the Gaffer [`road-traffic-demo`](https://github.com/gchq/Gaffer/blob/master/example/road-traffic/README.md) REST server from the Gaffer repository, using the command:

```bash
mvn clean install -pl :road-traffic-demo -Proad-traffic-demo,quick
```
To connect to the running Gaffer API from Python (more information on the Python shell can be found [here](https://gchq.github.io/gaffer-doc/latest/user-guide/apis/python-api/)):
```python
# Import the client library and connector
from gafferpy import gaffer as g
from gafferpy import gaffer_connector

# Instantiate a connector
gc = gaffer_connector.GafferConnector("http://localhost:8080/rest/latest")
```
Then perform requests against it:
```python
# You can use the connector to perform GET requests
schema = gc.execute_get(g.GetSchema())

# And also run operations
elements = gc.execute_operation(
operation=g.GetAllElements()
)

# Multiple operations
elements = gc.execute_operations(
operations=[
g.GetAllElements(),
g.Limit(result_limit=3)
]
)

# And operation chains
elements = gc.execute_operation_chain(
operation_chain=g.OperationChain(
operations=[
g.GetAllElements(),
g.Limit(
truncate=True,
result_limit=3
)
]
)
)
```

See [Operations Guide](https://gchq.github.io/gaffer-doc/latest/reference/operations-guide/operations) for more examples of operations in Python.

## Developer Guide

### Coding Style
Please ensure that your coding style is consistent with the rest of the Gaffer project. Guides on the coding style for `gafferpy` and Gaffer can be found [here](https://gchq.github.io/gaffer-doc/latest/development-guide/ways-of-working).

### Testing

The `gafferpy` tests are implemented using `tox` and `pytest`.
To run the tests, install the `gafferpy` 'dev' extra:
```bash
pip install -e ".[dev]" # the quotes ensure compatibilty with zsh
```
This will install extra development dependecies for running tests and building documentation.

`gafferpy` has both unit tests and integration tests - the integration tests use the Road Traffic Example to test the Python API. If this is not running, the integration tests are skipped. It is advisable that the integration tests are run prior to any code commits to ensure they do not fail due to any code changes.

*For help starting the Road Traffic Example, see the [Quick Start](#quick-start) section above.*

To run the tests, execute the below from the root directory of `gafferpy`:
```bash
tox
```
By default, `tox` will and try run the tests in multiple test envs (different Python versions) - if they do not exist then they are skipped.
To run the test for a specifc test env e.g. Python3.9, run:
```bash
tox -e py39
```

### Building the documentation

To build the docs locally, assuming you have Make and Python installed, and `gafferpy` installed with the 'dev' extra:
```bash
cd docs
make html
```

### Generating the Python API
`gafferpy` has the ability to regenerate the Python API based upon the Gaffer REST API that a `GafferConnector` object is pointing at - a more detailed description and examples of how to do this can be found [here](./src/fishbowl/README.md).

## License

Copyright 2016-2024 Crown Copyright

Licensed under the Apache License, Version 2.0 \(the "License"\); you may not use this file except in compliance with the License. You may obtain a copy of the License at

[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
23 changes: 15 additions & 8 deletions setup.py
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
import re
from codecs import open
from os import path
from pathlib import Path
from setuptools import setup, find_packages

###############################################################################

name = "gafferpy"
packages = find_packages(where="src")
meta_path = path.join("src", "gafferpy", "__init__.py")
meta_path = Path("src", "gafferpy", "__init__.py")
keywords = ["class", "attribute", "boilerplate"]
classifiers = [
"Development Status :: 4 - Beta",
Expand All @@ -24,20 +24,27 @@
python_requires = ">3.6"
install_requires = []
extras_require = {
"requests": ["requests>=2.4.0"]
"requests": ["requests>=2.4.0"],
"dev": [
"tox",
"pytest",
"requests>=2.4.0",
"sphinx~=7.2.6",
"sphinx-rtd-theme~=1.3.0"
]
}

###############################################################################

here = path.abspath(path.dirname(__file__))
here = Path(__file__).parent


def read(*parts):
def read(parts):
"""
Build an absolute path from *parts* and and return the contents of the
resulting file. Assume UTF-8 encoding.
"""
with open(path.join(here, *parts), "rb", "utf-8") as f:
with open(here / parts, "rb", "utf-8") as f:
return f.read()


Expand All @@ -54,14 +61,14 @@ def find_meta(meta):
)
if meta_match:
return meta_match.group(1)
raise RuntimeError("Unable to find __{meta}__ string.".format(meta=meta))
raise RuntimeError(f"Unable to find __{meta}__ string.")


version = find_meta("version")
uri = find_meta("uri")

# Get the long description from the README.md file
with open(path.join(here, "README.md"), encoding="utf-8") as f:
with open(here / "README.md", encoding="utf-8") as f:
long = f.read()

setup(
Expand Down
Loading

0 comments on commit a94ccae

Please sign in to comment.