Skip to content

Commit

Permalink
Updates and configuration changes; advancing version number to 1.5.0.
Browse files Browse the repository at this point in the history 
This fairly large commit includes the following changes:

- Corrections and updates to README, DEPLOY, and LICENSE docs.
- Update-to-current of Python interpreter version and source dependencies.
- Addition of "--batch_delay" CLI switch and associated code.
- Addition of "--sync_max" CLI switch and associated code.
- Change to syntax of "hsl-base" LDAP query syntax.
- Update of Docker base image to Alpine Linux 3.10.3.

This commit will be tagged as the v1.5.0 release.
  • Loading branch information
Mike Simpson committed Oct 24, 2019
1 parent 6d15608 commit b5d1a9c
Show file tree
Hide file tree
Showing 10 changed files with 137 additions and 112 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
/SCRATCH.md
/SCRATCH.md~
/.gradle
/.project
/.pydevproject
Expand Down
135 changes: 69 additions & 66 deletions DEPLOY.md
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Deployment Notes

Here are the steps that Mike Simpson went through to deploy the
production instance of the patron groups software in August 2019.
production instance of the patron groups software in October 2019.

## Prerequisites

* An IAM account with administrative access in the Library's main
AWS account.
* An existing VPC in that account.

* An existing public subnet on that VPC.
Expand All @@ -23,36 +23,36 @@ at the "Oregon (US-West-2)" AWS region:
section of the EC2 module, created the "PatronETL" security group
to limit access to the host that would be created in the following
step.
* (DONE PREVIOUSLY, NO NEED TO REPEAT) In the "Key Pairs" section of
the EC2 module, created the "PatronETL" key pair and downloaded
and saved a local copy.

* In the "Instances" section of the EC2 module, selected "Launch
Instance", and used the following parameters:
AMI Details: RancherOS - HVM
Instance Type: t2.small
Security Groups: PatronETL
Network: (default VPC)
Subnet: (first subnet on default VPC)
Storage: (left at default 8 GB general purpose)
and selected "Launch", choosing to use the existing "PatronETL"
AMI Details: RancherOS - HVM
Instance Type: t2.small
Security Groups: PatronETL
Network: (default VPC)
Subnet: (first subnet on default VPC)
Storage: (left at default 8 GB general purpose)
and selected "Launch", choosing to use the existing "PatronETL"
key pair when prompted.
* In the EC2 module dashboard, changed the name of the newly-created
EC2 instance to "Patron ETL", and waited for it to finish
initializing.
* Tested connecting to the new instance via SSH, e.g.:

% ssh -i [path to saved key pair file] rancher@[hostname from EC2 dashboard]
$ sudo ros os version
v1.1.0
$ exit
$ sudo ros os version
v1.1.0
$ exit

### Preparation of RancherOS instance for use.

Expand All @@ -61,52 +61,52 @@ In a terminal window on my local workstation:
* Logged into the "Patron ETL" instance and performed upgrade:

% ssh -i [path to saved key pair file] rancher@[hostname from EC2 dashboard]
$ sudo ros os upgrade
[followed prompts to upgrade to latest version]
[upgrade finishes with reboot, which terminates SSH session]
[upgrade finishes with reboot, which terminates SSH session]
% ssh -i [path to saved key pair file] rancher@[hostname from EC2 dashboard]
$ sudo ros os version
v1.5.3
$ exit
v1.5.4
$ exit
* Switched to the "centos" console in RancherOS and added build dependencies:

% ssh -i [path to saved key pair file] rancher@[hostname from EC2 dashboard]
$ sudo ros console enable centos
$ sudo reboot
[SSH session terminated]
$ sudo reboot
[SSH session terminated]

% ssh -i [path to saved key pair file] rancher@[hostname from EC2 dashboard]
$ sudo ros console list
disabled alpine
current centos
disabled debian
disabled default
disabled fedora
disabled ubuntu
$ sudo yum install epel-release
$ sudo yum install git python36 python36-pip
$ git --version
git version 1.8.3.1
$ python3 --version
Python 3.6.8
$ pip3 --version
pip 8.1.2 from /usr/lib/python3.6/site-packages (python 3.6)
disabled alpine
current centos
disabled debian
disabled default
disabled fedora
disabled ubuntu
$ sudo yum install epel-release
$ sudo yum install git python3 python3-pip
$ git --version
git version 1.8.3.1
$ python3 --version
Python 3.6.8
$ pip3 --version
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)
$ exit
* Note that RancherOS already includes Docker services as part of
the base installation.

Expand All @@ -115,22 +115,25 @@ In a terminal window on my local workstation:
* Logged into the "Patron ETL" instance and built patron groups service:

% ssh -i [path to saved key pair file] rancher@[hostname from EC2 dashboard]
$ git clone https://github.com/ualibraries/patron-groups.git ual-patron-groups
$ cd ual-patron-groups
$ git checkout v1.4.6
$ cd src/main/python
$ cd ual-patron-groups
$ git checkout v1.5.0
$ cd src/main/python
$ sudo pip3 install --trusted-host pypi.python.org -r requirements.txt
$ python3 setup.py sdist
$ cd ../../..
$ cd src/main/docker
$ cp ../python/dist/petal-1.4.6.tar.gz .
$ docker build -t pgrps:1.4.6 .
$ cd ../../..
$ cd src/main/docker
$ cp ../python/dist/petal-1.5.0.tar.gz .
$ docker build -t pgrps:1.5.0 .
$ cd ../../..

$ export LDAP_PASSWD=[ldap password]
$ export GROUPER_PASSWD=[grouper password]
$ export SLACK_WEBHOOK=[slack webhook]
$ docker run -e "LDAP_PASSWD=${LDAP_PASSWD}" -e "GROUPER_PASSWD=${GROUPER_PASSWD}" -e "SLACK_WEBHOOK=${SLACK_WEBHOOK}" --rm -d pgrps:1.4.6
$ export GROUPER_PASSWD=[grouper password]
$ export SLACK_WEBHOOK=[slack webhook]
$ docker run -e "LDAP_PASSWD=${LDAP_PASSWD}" \
-e "GROUPER_PASSWD=${GROUPER_PASSWD}" \
-e "SLACK_WEBHOOK=${SLACK_WEBHOOK}" \
--rm -d pgrps:1.5.0
2 changes: 1 addition & 1 deletion LICENSE.txt
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Copyright (c) 2017 Arizona Board of Regents
Copyright (c) 2019 Arizona Board of Regents
All rights reserved.

Redistribution and use in source and binary forms, with or without
Expand Down
20 changes: 10 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ environment using something like [PyEnv][pyenv] and

[ added shim lines to shell startup files as instructed by installation output ]
% pyenv install 3.7.2
% pyenv virtualenv 3.7.2 ual-patron-groups
% pyenv install 3.7.4
% pyenv virtualenv 3.7.4 ual-patron-groups
% pyenv global ual-patron-groups

% which python
Expand All @@ -56,7 +56,7 @@ environment using something like [PyEnv][pyenv] and
/Users/mgsimpson/.pyenv/versions/ual-patron-groups/bin/python

% python --version
Python 3.7.2
Python 3.7.4

### Docker

Expand All @@ -70,7 +70,7 @@ instructions for your OS on the Docker website, e.g. on macOS:
/usr/local/bin/docker
% docker --version
Docker version 18.09.1, build 4c52b90
Docker version 19.03.4, build 9013bf5

### Source

Expand All @@ -95,7 +95,7 @@ prerequisites, and then build a source distribution of the module:
% python setup.py sdist

% ls dist
petal-1.4.6.tar.gz
petal-1.5.0.tar.gz
% cd ../../..

Expand All @@ -105,12 +105,12 @@ Change to the Docker source directory, copy the distribution file over
from the Python side, and build the Docker container image:

% cd src/main/docker
% cp ../python/dist/petal-1.4.6.tar.gz .
% docker build -t pgrps:1.4.6 .
% cp ../python/dist/petal-1.5.0.tar.gz .
% docker build -t pgrps:1.5.0 .

% docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
pgrps 1.4.6 16cfafa9d0f1 12 seconds ago 80.1MB
pgrps 1.5.0 16cfafa9d0f1 12 seconds ago 80.1MB
% cd ../../..

Expand All @@ -136,7 +136,7 @@ hand, instantiate using an interactive shell:
% docker run -e "LDAP_PASSWD=${LDAP_PASSWD}" \
-e "GROUPER_PASSWD=${GROUPER_PASSWD}" \
-e "SLACK_WEBHOOK=${SLACK_WEBHOOK}" \
--rm -it pgrps:1.4.6 /bin/bash
--rm -it pgrps:1.5.0 /bin/bash

# which petl
/usr/bin/petl
Expand All @@ -159,7 +159,7 @@ in the background:
% docker run -e "LDAP_PASSWD=${LDAP_PASSWD}" \
-e "GROUPER_PASSWD=${GROUPER_PASSWD}" \
-e "SLACK_WEBHOOK=${SLACK_WEBHOOK}" \
--rm -d pgrps:1.4.6
--rm -d pgrps:1.5.0

The log output from the "run_petl" invocation should once again wind
up in the Slack channel associated with the "SLACK_WEBHOOK"
Expand Down
16 changes: 5 additions & 11 deletions src/main/docker/Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -1,30 +1,24 @@
FROM alpine:3.8
FROM alpine:3.10.3

ENV PACKAGES="dumb-init musl bash python3 python3-dev ca-certificates tzdata" \
ENV PACKAGES="dumb-init bash python3 python3-dev ca-certificates tzdata" \
LDAP_PASSWD="***supply at runtime***" \
GROUPER_PASSWD="***supply at runtime***" \
SLACK_WEBHOOK="***supply at runtime***"

ADD petal-1.4.6.tar.gz /var/tmp
ADD petal-1.5.0.tar.gz /var/tmp
ADD run_petl /etc/periodic/daily

RUN echo \
#
# set installed script as executable
&& chmod +x /etc/periodic/daily/run_petl \
#
# add edge repositories
&& echo "@edge-testing http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories \
&& echo "@edge-community http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories \
&& echo "@edge-main http://dl-cdn.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories \
#
# add packages
&& apk add --no-cache $PACKAGES \
#
# add symlinks
&& if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi \
&& if [[ ! -e /usr/bin/python-config ]]; then ln -sf /usr/bin/python3-config /usr/bin/python-config; fi \
&& if [[ ! -e /usr/bin/idle ]]; then ln -sf /usr/bin/idle3 /usr/bin/idle; fi \
&& if [[ ! -e /usr/bin/pydoc ]]; then ln -sf /usr/bin/pydoc3 /usr/bin/pydoc; fi \
&& if [[ ! -e /usr/bin/pip ]]; then ln -sf /usr/bin/pip3 /usr/bin/pip; fi \
#
Expand All @@ -33,8 +27,8 @@ RUN echo \
&& echo "US/Arizona" > /etc/timezone \
#
# install petal
&& pip install --trusted-host pypi.python.org -r /var/tmp/petal-1.4.6/requirements.txt \
&& pip install /var/tmp/petal-1.4.6
&& pip install --trusted-host pypi.python.org -r /var/tmp/petal-1.5.0/requirements.txt \
&& pip install /var/tmp/petal-1.5.0

ENTRYPOINT [ "/usr/bin/dumb-init", "--" ]
CMD [ "/usr/sbin/crond", "-f" ]
34 changes: 21 additions & 13 deletions src/main/python/config/petl.ini
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ grouper_passwd = ***override***
grouper_stem = arizona.edu:dept:LBRY:pgrps
batch_size = 100
batch_timeout = 60
batch_delay = 0
sync_max = 1000

#
# Per-group configurations.
Expand All @@ -42,8 +44,8 @@ ldap_query =
(|
(employeeType=0)(employeeType=4)(employeeType=5)(employeeType=7)(employeeType=9)
(employeeType=B)(employeeType=E)(employeeType=F)(employeeType=G)(employeeType=H)
(employeeType=L)(employeeType=M)(employeeType=O)(employeeType=Q)(employeeType=R)
(employeeType=V)(employeeType=W)(employeeType=X)
(employeeType=L)(employeeType=M)(employeeType=O)(employeeType=Q)(employeeType=R)
(employeeType=V)(employeeType=W)(employeeType=X)
)
(|
(employeeStatus=A)(employeeStatus=L)(employeeStatus=P)(employeeStatus=W)
Expand Down Expand Up @@ -115,8 +117,8 @@ ldap_query =
(|
(employeeType=0)(employeeType=4)(employeeType=5)(employeeType=7)(employeeType=9)
(employeeType=B)(employeeType=E)(employeeType=F)(employeeType=G)(employeeType=H)
(employeeType=L)(employeeType=M)(employeeType=O)(employeeType=Q)(employeeType=R)
(employeeType=V)(employeeType=W)(employeeType=X)
(employeeType=L)(employeeType=M)(employeeType=O)(employeeType=Q)(employeeType=R)
(employeeType=V)(employeeType=W)(employeeType=X)
)
(|
(employeeStatus=A)(employeeStatus=L)(employeeStatus=P)(employeeStatus=W)
Expand Down Expand Up @@ -205,15 +207,21 @@ ldap_query =
(eduPersonAffiliation=student)
(isMemberOf=arizona.edu:course:*:*:*:*:*:*:learner)
(|
(studentCPP=*:*:*:*:ABBSDLP*:*)(studentCPP=*:*:*:*:ABSPSM*:*)(studentCPP=*:*:*:*:ACPCRTG*:*)(studentCPP=*:*:*:*:ACRCRTG*:*)(studentCPP=*:*:*:*:AGACNCRTG*:*)
(studentCPP=*:*:*:*:BIOS*:*)(studentCPP=*:*:*:*:BSCRTG*:*)(studentCPP=*:*:*:*:CBA*:*)(studentCPP=*:*:*:*:CBIO*:*)(studentCPP=*:*:*:*:CLTRSCI*:*)
(studentCPP=*:*:*:*:CMM*:*)(studentCPP=*:*:*:*:EHLMS*:*)(studentCPP=*:*:*:*:EHLPHD*:*)(studentCPP=*:*:*:*:EMS*:*)(studentCPP=*:*:*:*:EPI*:*)
(studentCPP=*:*:*:*:GENE*:*)(studentCPP=*:*:*:*:GERO*:*)(studentCPP=*:*:*:*:GHDCRTG*:*)(studentCPP=*:*:*:*:GNCLMS*:*)(studentCPP=*:*:*:*:HADCRTG*:*)
(studentCPP=*:*:*:*:HBHP*:*)(studentCPP=*:*:*:*:HLTH*:*)(studentCPP=*:*:*:*:IMMU*:*)(studentCPP=*:*:*:*:MBIM*:*)(studentCPP=*:*:*:*:MED*:*)
(studentCPP=*:*:*:*:MEPH*:*)(studentCPP=*:*:*:*:MMPHD*:*)(studentCPP=*:*:*:*:NRSC*:*)(studentCPP=*:*:*:*:NURS*:*)(studentCPP=*:*:*:*:PCOL*:*)
(studentCPP=*:*:*:*:PHCL*:*)(studentCPP=*:*:*:*:PHL*:*)(studentCPP=*:*:*:*:PHMY*:*)(studentCPP=*:*:*:*:PHSC*:*)(studentCPP=*:*:*:*:PNP*:*)
(studentCPP=*:*:*:*:PRHL*:*)(studentCPP=*:*:*:*:PRNUND*:*)(studentCPP=*:*:*:*:PRPHND*:*)(studentCPP=*:*:*:*:PRPHSCND*:*)(studentCPP=*:*:*:*:PRPND*:*)
(studentCPP=*:*:*:*:PSHSCRTG*:*)(studentCPP=*:*:*:*:PSIO*:*)(studentCPP=*:*:*:*:PSMS*:*)(studentCPP=*:*:*:*:PSPHD*:*)
(studentCPP=*:*:*:*:ABBSDLP*:*:*:*:*:*)(studentCPP=*:*:*:*:ABSPSM*:*:*:*:*:*)(studentCPP=*:*:*:*:ACPCRTG*:*:*:*:*:*)
(studentCPP=*:*:*:*:ACRCRTG*:*:*:*:*:*)(studentCPP=*:*:*:*:AGACNCRTG*:*:*:*:*:*)(studentCPP=*:*:*:*:BIOS*:*:*:*:*:*)
(studentCPP=*:*:*:*:BSCRTG*:*:*:*:*:*)(studentCPP=*:*:*:*:CBA*:*:*:*:*:*)(studentCPP=*:*:*:*:CBIO*:*:*:*:*:*)
(studentCPP=*:*:*:*:CLTRSCI*:*:*:*:*:*)(studentCPP=*:*:*:*:CMM*:*:*:*:*:*)(studentCPP=*:*:*:*:EHLMS*:*:*:*:*:*)
(studentCPP=*:*:*:*:EHLPHD*:*:*:*:*:*)(studentCPP=*:*:*:*:EMS*:*:*:*:*:*)(studentCPP=*:*:*:*:EPI*:*:*:*:*:*)
(studentCPP=*:*:*:*:GENE*:*:*:*:*:*)(studentCPP=*:*:*:*:GERO*:*:*:*:*:*)(studentCPP=*:*:*:*:GHDCRTG*:*:*:*:*:*)
(studentCPP=*:*:*:*:GNCLMS*:*:*:*:*:*)(studentCPP=*:*:*:*:HADCRTG*:*:*:*:*:*)(studentCPP=*:*:*:*:HBHP*:*:*:*:*:*)
(studentCPP=*:*:*:*:HLTH*:*:*:*:*:*)(studentCPP=*:*:*:*:IMMU*:*:*:*:*:*)(studentCPP=*:*:*:*:MBIM*:*:*:*:*:*)
(studentCPP=*:*:*:*:MED*:*:*:*:*:*)(studentCPP=*:*:*:*:MEPH*:*:*:*:*:*)(studentCPP=*:*:*:*:MMPHD*:*:*:*:*:*)
(studentCPP=*:*:*:*:NRSC*:*:*:*:*:*)(studentCPP=*:*:*:*:NURS*:*:*:*:*:*)(studentCPP=*:*:*:*:PCOL*:*:*:*:*:*)
(studentCPP=*:*:*:*:PHCL*:*:*:*:*:*)(studentCPP=*:*:*:*:PHL*:*:*:*:*:*)(studentCPP=*:*:*:*:PHMY*:*:*:*:*:*)
(studentCPP=*:*:*:*:PHSC*:*:*:*:*:*)(studentCPP=*:*:*:*:PNP*:*:*:*:*:*)(studentCPP=*:*:*:*:PRHL*:*:*:*:*:*)
(studentCPP=*:*:*:*:PRNUND*:*:*:*:*:*)(studentCPP=*:*:*:*:PRPHND*:*:*:*:*:*)(studentCPP=*:*:*:*:PRPHSCND*:*:*:*:*:*)
(studentCPP=*:*:*:*:PRPND*:*:*:*:*:*)(studentCPP=*:*:*:*:PSHSCRTG*:*:*:*:*:*)(studentCPP=*:*:*:*:PSIO*:*:*:*:*:*)
(studentCPP=*:*:*:*:PSMS*:*:*:*:*:*)(studentCPP=*:*:*:*:PSPHD*:*:*:*:*:*)
)
)
(&
Expand Down
6 changes: 3 additions & 3 deletions src/main/python/requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
certifi==2018.11.29
ldap3==2.5.2
requests==2.21.0
certifi==2019.9.11
ldap3==2.6.1
requests==2.22.0
slack_log_handler==0.3.0
Loading

0 comments on commit b5d1a9c

Please sign in to comment.