update and fix doc builds, DEVELOPMENT_NOTES, and update to latest vi…

…ral-core

.travis.yml

@@ -45,6 +45,7 @@ jobs:
         - if [ -f "$CACHE_DIR/old-docker-tag.txt" ]; then OLD_DOCKER_TAG=$(cat $CACHE_DIR/old-docker-tag.txt); else OLD_DOCKER_TAG=$DOCKER_REPO_PROD; fi; echo "old docker tag = $OLD_DOCKER_TAG"
         - _CACHE_FROM="--cache-from $OLD_DOCKER_TAG"; docker pull $OLD_DOCKER_TAG || _CACHE_FROM=""
         - git describe --tags --always | tee VERSION
+        - if [ -n "$TRAVIS_TAG" ]; then echo "Release $TRAVIS_TAG"; elif [ -n "$TRAVIS_PULL_REQUEST_BRANCH" ]; then echo "LABEL quay.expires-after=10w" >> Dockerfile; elif [[ "$TRAVIS_BRANCH" != "master" ]]; then echo "LABEL quay.expires-after=10w" >> Dockerfile; fi
         - docker build -t local/build-container:build $_CACHE_FROM .
         - travis/deploy-docker.sh
       before_cache:
@@ -67,11 +68,16 @@ jobs:
         - pip install coveralls==1.1
         - coveralls
 
-    - language: python
-      stage:    build
-      env:      TRAVIS_JOB=build_docs
-      install:  travis/install-pip-docs.sh
-      script:   travis/build-docs.sh
-      ## note: this build_docs job does not actually produce the output on readthedocs
-      ## readthedocs does its own build trigger. this travis job exists simply to alert us
-      ## of build failures of the docs because otherwise we would never know.
+    - language: generic
+      stage:    test
+      env:
+        - TRAVIS_JOB=test_docs
+        ## note: this test_docs job does not actually produce the output on readthedocs
+        ## readthedocs does its own build trigger. this travis job exists simply to alert us
+        ## of build failures of the docs because otherwise we would never know.
+      install:
+        - DOCKER_TAG=`travis/list-docker-tags.sh | tail -1`
+        - echo "pulling $DOCKER_TAG"
+        - docker pull $DOCKER_TAG
+      script:
+        - docker run --entrypoint /bin/bash -v `pwd`:/opt/viral-ngs/viral-assemble $DOCKER_TAG -c 'set -e; cd /opt/viral-ngs/viral-assemble; travis/install-pip-docs.sh; cd docs; make html'

DEVELOPMENT_NOTES.md

@@ -8,12 +8,18 @@ means that developers will need docker working within their dev environment--and
 not much else (other than git, and a text/code editor). The code base is also
 modularized and layered. In order to work on code changes, you will:
 
-1. check out the git code repository for this module on your local host machine (`git clone https://github.com/broadinstitute/viral-assemble.git`) and edit with your favorite code/text editor
-1. docker `pull` and `run` the image `FROM` which this is built, while mounting your local git checkout into the container (`docker run -it --rm -v `pwd`/viral-assemble:/opt/viral-ngs/viral-assemble quay.io/broadinstitute/viral-core`)
-1. install this module's dependencies inside the container (within container: `/opt/viral-ngs/viral-assemble/docker/install-dev-layer.sh`)
-1. (optional) snapshot this docker image locally if you want to continue using it and skip the above steps in the future (`docker commit <image hash> local/viral-assemble-dev`)
-1. test code and execution interactively within the container (`cd /opt/viral-ngs/viral-assemble; pytest -rsxX -n auto test/unit`)
-1. push changes back to github (from your host machine) for automated CI testing & builds, using standard, collaborative github code review processes
+1. Clone the git code repository for this module on your local host machine (`git clone https://github.com/broadinstitute/viral-assemble.git`), create and checkout a branch for your development work, and edit with your favorite code/text editor (standard git flow).
+1. Run the docker image on your machine while mounting your local git checkout into the container to test live edits. There are two ways to do this:
+	1. If you aren't making (significant) changes to the tool dependencies in `requirements-conda.txt`, just use the latest docker image of viral-assemble (or the specific tag you want to start from) and test your code changes in there. Do the following:
+		1. docker `pull` and `run` the image `FROM` which this is built, while mounting your local git checkout into the container (`docker run -it --rm -v `pwd`/viral-assemble:/opt/viral-ngs/viral-assemble quay.io/broadinstitute/viral-core`)
+		1. If you made any changes to `requirements-conda.txt`, you'll need to update them within your container. Run `/opt/viral-ngs/source/docker/install-conda-dependencies.sh /opt/viral-ngs/viral-assemble/requirements-conda.txt`). Skip this step if you're only testing non-conda changes on your branch. If this took any significant amount of time, you can snapshot the resulting container (see instructions below).
+	1. If you are making significant changes to the conda tool dependencies, it might be best to start from viral-core and layer on from there. Do the following:
+		1. docker `pull` and `run` the image `FROM` which this is built, while mounting your local git checkout into the container (`docker run -it --rm -v `pwd`/viral-assemble:/opt/viral-ngs/viral-assemble quay.io/broadinstitute/viral-core`)
+		1. Install this module's dependencies inside the container (within container: `/opt/viral-ngs/viral-assemble/docker/install-dev-layer.sh`). This make take a couple minutes.
+		1. (optional) Snapshot this docker image locally if you want to continue using it and skip the install-dev-layer step in the future (`docker commit <image hash> local/viral-assemble-dev`, where the image hash comes from `docker container ls`).
+	1. Either way, whenever you exit the container's command line, it will delete itself. Code changes will always persist in your local checkout (because they were on your host machine), but other OS changes, conda changes, and temp files start fresh again the next time you `docker run` the image (from either the original image, or a local snapshot you made with `docker commit`).
+1. Test code and execution interactively within the container (`cd /opt/viral-ngs/viral-assemble; pytest -rsxX -n auto test/unit`, or whatever you'd like to do).
+1. Push changes back to github (from your host machine) for automated CI testing & builds, using standard, collaborative github code review processes. Note that only your host machine (not the container) should have your github credentials, although both are working on the same working copy of files.
 
 ### Machinery under the hood
 

Dockerfile

@@ -1,4 +1,4 @@
-FROM quay.io/broadinstitute/viral-core:2.0.11
+FROM quay.io/broadinstitute/viral-core:2.0.14
 
 LABEL maintainer "[email protected]"
 

README.md

@@ -7,8 +7,8 @@
 [![Code Health](https://landscape.io/github/broadinstitute/viral-ngs/master/landscape.svg?style=flat)](https://landscape.io/github/broadinstitute/viral-ngs)
 -->
 
-viral-assembly
-=========
+viral-assemble
+==============
 
 A set of scripts and tools for the assembly of viral genomes from NGS data.
 

docs/conf.py

@@ -26,7 +26,7 @@
 MOCK_MODULES = ['scipy', 'pysam', 'Bio', 'Bio.AlignIO', 'Bio.Alphabet',
                 'Bio.Alphabet.IUPAC', 'Bio.SeqIO', 'Bio.Data.IUPACData',
                 'Bio.Seq', 'Bio.SeqRecord', 'pybedtools', 'pybedtools.BedTool',
-                'arrow']
+                'arrow', 'zstd', 'bz2']
 for mod_name in MOCK_MODULES:
     sys.modules[mod_name] = mock.Mock()
 
@@ -44,6 +44,15 @@ def _git_version():
 
 __version__ = _git_version()
 
+
+# -- Obtain upstream viral-core module: this is super hacky and not pinned to any version
+def _get_viral_core():
+    cmd = ['git', 'clone', '--depth=1', 'https://github.com/broadinstitute/viral-core.git']
+    subprocess.check_call(cmd)
+    sys.path.insert(0, os.path.dirname(os.path.abspath('viral-core')))
+_get_viral_core()
+
+
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.

docs/requirements.txt

@@ -2,5 +2,5 @@ Sphinx==1.4.4
 sphinx-argparse==0.1.15
 sphinx_rtd_theme==0.1.9
 matplotlib==2.2.4
-PyYAML==5.1
+#PyYAML==5.1
 mock>=2.0.0