Python3 adaptation

.gitignore

@@ -1,10 +1,12 @@
+bd2k-extras/
 pimmuno.py
 pimmuno_2.py
 *.pyc
 /src/*.egg-info/
 develop_data/
 venv/
 .cache/
+jobStore/
 test-report.xml
 __pycache__
 *.DONE

MANUAL.md

@@ -27,87 +27,59 @@ ProTECT is implemented in the [Toil](https://github.com/BD2KGenomics/toil.git) f
 runs the workflow described in [protect/Flowchart.txt](
 https://github.com/BD2KGenomics/protect/blob/master/Flowchart.txt).
 
+**This manual is a quick adaptation for an adaptation of ProTECT to py3** 
+
 
 # Installation
 
 ProTECT requires Toil and we recommend installing ProTECT and its requirements in a
 [virtualenv](http://docs.python-guide.org/en/latest/dev/virtualenvs/).
 
-ProTECT also requires [s3am](https://github.com/BD2KGenomics/s3am.git) version 2.0.1 to download and
+~ProTECT also requires [s3am](https://github.com/BD2KGenomics/s3am.git) version 2.0.1 to download and
 upload files from S3. We recommend installing s3am in its own virtualenv using the directions in
 the s3am manual, then putting the s3am binary on your $PATH.  ProTECT will NOT attempt to install
-s3am during installation.
+s3am during installation.~ 
+
+currently WIP. for now, **only references to local files will work**. anything that requires access to s3am (s3 buckets) will **fail**. 
 
-ProTECT uses pkg_resources from setuptools to verify versions of tools during install. As of setuptools
+~ProTECT uses pkg_resources from setuptools to verify versions of tools during install. As of setuptools
 39.0.1, some modules were moved to the packaging module. If your machine has setuptools >=39.0.1, you
-will need the packaging module.
+will need the packaging module.~
 
 Lastly, ProTECT uses [docker](https://www.docker.com/) to run the various sub-tools in a
 reproducible, platform independent manner. ProTECT will NOT attempt to install docker during
 installation.
 
-### Method 1 - Using PIP (recommended)
-
-First create a virtualenv at your desired location (Here we create it in the folder ~/venvs)
-
-    virtualenv ~/venvs/protect
-
-Activate the virtualenv
-
-    source ~/venvs/protect/bin/activate
-
-NOTE: Installation was tested using pip 7.1.2 and 8.1.1. We have seen issues with the installation
-of pyYAML with lower versions of pip and recommend upgrading pip before installing ProTECT.
-
-    pip install --upgrade pip
-
-Install Toil
-
-    pip install toil[aws]==3.5.2
-
-Install packaging (required if setuptools>=39.0.1)
-
-    pip install packaging
-
-Install ProTECT and all dependencies in the virtualenv
-
-    pip install protect
-
+~Method 1 - Using PIP (recommended)~
 ### Method 2 - Installing from Source
 
 This will install ProTECT in an editable mode.
 
 Obtain the source from Github
 
-    git clone https://www.github.com/BD2KGenomics/protect.git
+    git clone https://www.github.com/Dranion/protect.git
 
 Create and activate a virtualenv in the project folder (Important since the Makefile checks for
 this and will fail if it detects that you are not in a virtual environment)
 
     cd protect
-    virtualenv venv
+    virtualenv --python=python3 venv
     source venv/bin/activate
 
 Install Toil and pytest
 
     make prepare
 
-Install packaging (required if setuptools>=39.0.1)
+Install the python3 conversion of bd2k and s3am. *s3am is untested as I am running locally* 
 
-    pip install packaging
+    make special_install
 
 Install ProTECT
 
     make develop
 
-## Method 3 - Using Docker
+~Method 3 - Using Docker~
 
-Dockerized versions of ProTECT releases can be found at https://quay.io/organization/ucsc_cgl. These
-Docker containers run the ProTECT pipeline in single machine mode. The only difference between the
-Docker and Python versions of the pipeline is that the Docker container takes the config options,
-described below, as command line arguments as opposed to a config file. Running the container
-without any arguments will list all the available options. Also, currently the dockerized version of
-ProTECT only supports local file export.
 
 # Running ProTECT
 
@@ -173,7 +145,7 @@ in the pipeline, and the information on the input samples. Elements before a `:`
 dictionary read into ProTECT and should **NOT** be modified (Barring the patient ID key in the
 patients dictionary). Only values to the right of the `:` should be edited.
 
-Every required reference file is provided in the AWS bucket `cgl-pipeline-inputs` under the folder
+Every required reference file is provided in the AWS bucket `protect-data` under the folder
 `protect/hg19_references` or `protect/hg38_references`. The `README` file in the same location
 describes in detail how each file was generated. To use a file located in an s3 bucket, replace
 `/path/to` in the following descriptions with `s3://<databucket>/<folder_in_bucket>`.
@@ -547,7 +519,7 @@ purposes:
     12: g/f/jobO4yiE4        return self.run(fileStore)
     13: g/f/jobO4yiE4      File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/job.py", line 1406, in run
     14: g/f/jobO4yiE4        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
-    15: g/f/jobO4yiE4      File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python2.7/site-packages/protect/binding_prediction/common.py", line 566, in merge_mhc_peptide_calls
+    15: g/f/jobO4yiE4      File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python3/site-packages/protect/binding_prediction/common.py", line 566, in merge_mhc_peptide_calls
     16: g/f/jobO4yiE4        raise RuntimeError('No peptides available for ranking')
     17: g/f/jobO4yiE4    RuntimeError: No peptides available for ranking
     18: g/f/jobO4yiE4    ERROR:toil.worker:Exiting the worker because of a failed job on host sjcb10st7
@@ -581,9 +553,9 @@ do not store logs from tools (see BD2KGenomics/protect#275). The error looks sim
     Z/O/job1uH92D        return self.run(fileStore)
     Z/O/job1uH92D      File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/job.py", line 1406, in run
     Z/O/job1uH92D        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
-    Z/O/job1uH92D      File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python2.7/site-packages/protect/mutation_calling/radia.py", line 238, in run_filter_radia
+    Z/O/job1uH92D      File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python3/site-packages/protect/mutation_calling/radia.py", line 238, in run_filter_radia
     Z/O/job1uH92D        tool_version=radia_options['version'])
-    Z/O/job1uH92D      File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python2.7/site-packages/protect/common.py", line 138, in docker_call
+    Z/O/job1uH92D      File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python3/site-packages/protect/common.py", line 138, in docker_call
     Z/O/job1uH92D        'for command \"%s\"' % ' '.join(call),)
     Z/O/job1uH92D    RuntimeError: docker command returned a non-zero exit status (1)for command "docker run --rm=true -v /scratch/bio/ucsc/toil-681c097c-61da-4687-b734-c5051f0aa19f/tmped2fnu/f041f939-5c0d-40be-a884-68635e929d09:/data --log-driver=none aarjunrao/filterradia:bcda721fc1f9c28d8b9224c2f95c440759cd3a03 TCGA-CH-5788 17 /data/radia.vcf /data /home/radia/scripts -d /data/radia_dbsnp -r /data/radia_retrogenes -p /data/radia_pseudogenes -c /data/radia_cosmic -t /data/radia_gencode --noSnpEff --noBlacklist --noTargets --noRnaBlacklist -f /data/hg38.fa --log=INFO -g /data/radia_filtered_chr17_radia.log"
     Z/O/job1uH92D    ERROR:toil.worker:Exiting the worker because of a failed job on host sjcb10st1

Makefile

@@ -45,17 +45,22 @@ help:
 	@echo "$$help"
 
 
-python=python2.7
-pip=pip2.7
+python=python
+pip=pip
 tests=src/protect/test/unit
 extras=
-
 green=\033[0;32m
 normal=\033[0m
 red=\033[0;31m
 
+# WIP 
+special_install: check_venv
+	git clone https://github.com/Dranion/bd2k-extras.git
+	make -C bd2k-extras/bd2k-python-lib develop
+	make -C bd2k-extras/s3am develop
+
 prepare: check_venv
-	@$(pip) install toil==3.8.0 pytest==2.8.3
+	@$(pip) install toil pytest  
 
 develop: check_venv
 	$(pip) install -e .$(extras)
@@ -107,11 +112,10 @@ clean_pypi:
 
 clean: clean_develop clean_sdist clean_pypi
 
-
 check_venv:
-	@$(python) -c 'import sys; sys.exit( int( not hasattr(sys, "real_prefix") ) )' \
-		|| ( echo "$(red)A virtualenv must be active.$(normal)" ; false )
-
+	@$(python) -c 'import sys; sys.exit( int( not (hasattr(sys, "real_prefix") or ( hasattr(sys, "base_prefix") and sys.base_prefix != sys.prefix ) ) ) )' \
+		|| [ ! -z "${VIRTUAL_ENV}" ] \
+		|| ( echo "$(red)A virtualenv must be active.$(normal)\n" ; false )
 
 check_clean_working_copy:
 	@echo "$(green)Checking if your working copy is clean ...$(normal)"

README.md

@@ -1,7 +1,10 @@
-[![Stories in Ready](https://badge.waffle.io/BD2KGenomics/protect.png?label=ready&title=Ready)](https://waffle.io/BD2KGenomics/protect)
 # ProTECT
 ### **Pr**ediction **o**f **T**-Cell **E**pitopes for **C**ancer **T**herapy
 
+Adapation of ProTECT to use python 3.8 instead of 2.7. Currently have tested a complete run using fastq files from [HCC1395 WGS Exome RNA Seq Data](https://github.com/genome/gms/wiki/HCC1395-WGS-Exome-RNA-Seq-Data), with identical results in both version of python. 
+
+Adaptation done using 2to3 and manual bug testing. Manual changes recorded [at changes.md](https://github.com/Dranion/protect/blob/master/changes.md). Since s3am is python2, **currently is local only**, however an untested python3 version of s3am exists [here](https://github.com/Dranion/bd2k-extras/tree/main). Continuing to the original README: 
+
 This repo contains the Python libraries for the Precision Immunology Pipeline developed at UCSC.
 
     src/protect/pipeline/ProTECT.py             - The python script for running the pipeline.
@@ -20,6 +23,6 @@ All docker images used in this pipeline are available at
 
 
 To learn how the pipeline can be run on a sample, head over to the [ProTECT Manual](
-https://github.com/BD2KGenomics/protect/blob/master/MANUAL.md)
+https://github.com/Dranion/protect/blob/master/MANUAL.md)
 
 ProTECT is currently in its infancy and is under continuous development.  We would appreciate users sharing the level 3 data produced by ProTECT with us such that we can better train our predictive models.