WIP: Build master with habitat #453

themightychris · 2017-07-28T12:13:21Z

On any Linux machine or Mac w/ Docker installed:

First-time setup

# install habitat
curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh | sudo bash

# initialize habitat configuration
hab cli setup

# clone habitat branch of DistrictBuilder
git clone -b habitat [email protected]:themightychris/DistrictBuilder.git

# change into repository
cd DistrictBuilder

Enter studio and build package

# erase any existing studio to start completely fresh
hab studio rm

# enter habitat studio (a cleanroom for building with only explicit dependencies)
hab studio enter

# execute build. Type "return" to continue when debugger pauses at attach points
build

Setup app within studio after a successful build or install

# activate virtualenv within built package
source "`hab pkg path $HAB_ORIGIN/DistrictBuilder`/bin/activate"

# create sample config file
cp /src/docs/config.dist.xml /src/docs/config.xml

# setup app
cd "`hab pkg path $HAB_ORIGIN/DistrictBuilder`/django/publicmapping"
python setup.py /src/docs/config.{xsd,xml}

themightychris · 2017-07-28T16:43:06Z

pip install -r requirements.txt is currently failing, I've made some progress in ae9d561but I'm not sure to what extent it's app configuration issues to fix at the python/django layer or environmental issues to be fixed at the habitat level ye

mcdisc · 2017-07-28T16:59:18Z

We are aware that technical debt accrued since last revision of DistrictBuilder about 4 years ago is causing failures. We have submitted a funding proposal to clear the technical debt and update the installation documentation. Hopefully these issues will be cleared up before the end of 2017. ============ Dr. Michael P. McDonald Associate Professor, University of Florida 352-273-2371 www.electproject.org @ElectProject

…

On Fri, Jul 28, 2017 at 12:43 PM, Chris Alfano ***@***.***> wrote: pip install -r requirements.txt is currently failing, I've made some progress in ae9d561but I'm not sure to what extent it's app configuration issues to fix at the python/django layer or environmental issues to be fixed at the habitat level ye — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <PublicMapping/districtbuilder#453 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBMSZf4It24ys7c0BhI2DcH8yzDLL8Bks5sSg-bgaJpZM4Omh7y> .

themightychris · 2017-07-28T17:29:29Z

Thanks @mcdisc ! I'm aware of the efforts on that front, I was hoping to help get the old version bare-bones deployable on current systems. I'm pretty familiar with tools to provide an exact environment to the application, even with old/unsupported dependencies being needed. I'm not too familiar with the python/django stack though so I'm pretty ignorant of how big the tech debt mountain may be within the app.

If anyone can help me get past pip install -r requirements completing though I'm more confident about being able to handle the rest of the process. My hope is this work might inspire some developers to play around as a contingency to funding coming through in a timely manner, and maybe it can provide a head start if/when it does.

mcdisc · 2017-07-28T17:41:44Z

We pounded on this last summer, and we have an incomplete install document with workarounds that got us part way through the installation until we ran into errors that appear to require re-engineering of the DistrictBuilder code. Unfortunately, I am on vacation and do not have access to that document. I think Micah Altman last looked at this and may be able to share. ============ Dr. Michael P. McDonald Associate Professor, University of Florida 352-273-2371 www.electproject.org @ElectProject

…

On Fri, Jul 28, 2017 at 1:29 PM, Chris Alfano ***@***.***> wrote: Thanks @mcdisc <https://github.com/mcdisc> ! I'm aware of the efforts on that front, I was hoping to help get the old version bare-bones deployable on current systems. I'm pretty familiar with tools to provide an exact environment to the application, even with old/unsupported dependencies being needed. I'm not too familiar with the python/django stack though so I'm pretty ignorant of how big the tech debt mountain may be within the app. If anyone can help me get past pip install -r requirements completing though I'm more confident about being able to handle the rest of the process. My hope is this work might inspire some developers to play around as a contingency to funding coming through in a timely manner, and maybe it can provide a head start if/when it does. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <PublicMapping/districtbuilder#453 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBMSQI2uJ7jryTn4CKMfGH6polwsbBQks5sShp6gaJpZM4Omh7y> .

themightychris · 2017-07-28T22:19:12Z

Progress! pip install -r requirements.txt completes successfully now.

I've added a line to the main description that activates the created virtualenv post-build now. More stuff should probably still get done in the build phase, including copying whatever parts of this repo are needed at runtime into $pkg_prefix somewhere, but this gets us to a point where setup can continue I think.

themightychris · 2017-07-28T22:25:02Z

Currently, python setup.py ../../docs/config.{xsd,xml} fails with:

ImportError: No module named lxml.etree

Running pip install lxml seems to fix that, does that need to be added to requirements.txt? Then the next failure is:

ImportError: Could not import settings 'publicmapping.settings' (Is it on sys.path?): cannot import name _uuid_generate_random

themightychris · 2017-07-29T04:32:34Z

The settings issue got sorted out by bumping celery to a newer version, working on a timezone issue that's probably habitat's doing now

hectcastro · 2017-08-01T17:21:31Z

Hey Chris,

We sincerely appreciate your efforts here. It is likely that a fair bit of your work will carry over into the technical debt reduction work referenced above. We're a bit resource constrained right now, but would like to lend a hand where we can.

With regard to Habitat, we're a bit unfamiliar with it as an organization. From an ongoing maintenance perspective, it is likely that we'd gravitate toward a more vanilla Docker based packaging approach that builds on top of our generic Django container image (at least for the Django/Celery bits). The other supporting services would use official container images (Redis, PostgreSQL, etc.) locally, but then Amazon Web Services equivalents (ElastiCache, RDS) when deployed to a staging/production environment.

Not sure how that overlaps with your efforts here, but I just wanted to put that point out there for discussion as early as possible.

themightychris · 2017-08-03T14:23:18Z

@hectcastro thanks for the note, I'll keep pecking away at this here and there and appreciate any materials you have I might get some clues from. DistrictBuilder is a project I believe is really important, and habitat is a bit of an academic interest for me right now so I want to see if I can bring them together and show what they can do, even if ultimately your team keeps your current direction (totally understandable!)

Habitat is a new project from the Chef team, and it's essentially package and application lifecycle management reimagined for the container age. It's well-supported and has been aggressively developed. What's great about it is that it completely uncouples the application from the host environment and deployment topology. Traditional docker tools give you containers that are still anchored to specific operating system distributions, leaving you with a time bomb to defuse as distributions turn over. An application packaged with habitat can be exported to a docker container image, or run on a VM or bare metal just the same... and in every case, on every operating system, for all time the entire stack of code down to the ELF interpreter will be the exactly same until the application gets rebuilt or its plan gets changed. It provides very precise control of the entire dependency chain and gives you both community support for maintaining updates to dependencies and a seamless path to publish arbitrary updates/builds of deps if you need to--no more mixing in random software sources when the official distro builds are missing a bug fix or version bump you need. All build processes as well get automated in an application-centric way that runs in a clean-room guaranteed to be consistent across development environments

I'd encourage you to give the commands I listed in the PR description a try and see what it gets you, it will take just a couple minutes

themightychris · 2017-08-03T14:29:05Z

FYI I switched to working on a branch based on the v2.0 tag in an attempt to find more sanity: https://github.com/themightychris/DistrictBuilder/tree/habitat-v2.0

msarahan · 2017-08-25T02:22:47Z

@themightychris advised me on the codeforphilly slack channel on getting this setup. He's missing a tiny bit in the setup instructions up top. Before hab studio enter, you should run hab cli setup and enter the profile name as codeforphilly, and also allow it to create the signing key. The rest doesn't matter. With that, the next error is

Traceback (most recent call last):
  File "setup.py", line 180, in <module>
    main()
  File "setup.py", line 138, in main
    management.call_command('syncdb', verbosity=options.verbosity, interactive=False)
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/core/management/__init__.py", line 134, in call_command
    klass = load_command_class(app_name, name)
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/core/management/__init__.py", line 69, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 8, in <module>
    from django.core.management.sql import custom_sql_for_model, emit_post_sync_signal
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/core/management/sql.py", line 6, in <module>
    from django.db import models
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/db/__init__.py", line 40, in <module>
    backend = load_backend(connection.settings_dict['ENGINE'])
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/db/__init__.py", line 34, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/db/utils.py", line 92, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/db/utils.py", line 24, in load_backend
    return import_module('.base', backend_name)
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 5, in <module>
    from django.contrib.gis.db.backends.postgis.operations import PostGISOperations
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/operations.py", line 8, in <module>
    from django.contrib.gis.geometry.backend import Geometry
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/contrib/gis/geometry/backend/__init__.py", line 11, in <module>
    module = import_module(geom_backend)
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/geos/__init__.py", line 4, in <module>
    import geos.server
  File "/hab/pkgs/codeforphilly/DistrictBuilder/2.0+114#ca42696/20170825012557/lib/python2.7/site-packages/geos/server.py", line 3
    from geos.print import print_map
                  ^
SyntaxError: invalid syntax

This doesn't make sense to me - the imports appear to be valid syntax, aside from the nasty from geos.kml import * It may be the module name conflicting with the built-in print statement, but surely that would have presented itself quite a while back.

Anyway, I will work this weekend to debug further.

themightychris · 2017-08-25T15:41:37Z

@msarahan that's where my efforts went off the rails. IIRC that syntax seems to require python 3, and every version of geos uses it, but half the rest of the app melts if you switch from python2 to python3. So, I don't know how it ever worked, but I also have never built a python app and learned most of what I know about PIP trying to make this work

msarahan · 2017-08-28T01:42:58Z

It does not seem that conda will play as nicely as I had hoped with habitat. Our installer fails to run because of the way that habitat does glibc. It may be possible to patchelf it, but it's not going to be pretty.

Our installer is basically a bash script with a binary blob, consisting of many conda packages (each a .tar.bz2 with a particular folder structure). These appear to be getting extracted successfully. However, after they are extracted, the installer tries to run the python executable that it has just extracted to do some more stuff. That's where we bump into the glibc issue. So, we'd need to sort of intercept the installer mid-way to patchelf python. We'd probably also need to end up patchelf'ing any of the compiled libraries that conda provides. I think that's not worth the headache.

@themightychris sorry this was a wash. In looking at your requirements.txt file, it seems fragile to update all of the dependencies to the latest version. You may have more luck pinning to older versions for the time being, until DistrictBuilder can be refactored and modernized as necessary to use current versions.

themightychris · 2017-08-28T20:34:16Z

@hectcastro @mcdisc does anyone know what the last commit of DistrictBuilder that is running online is? I've gotten past environmental issues I think and just gotten down to the list of python deps in master and the v2 tag just being unworkable. It's hard to see how either of those commits could ever get running

mcdisc · 2017-08-29T03:00:23Z

The most recent build to my knowledge is an AWS AMI for Mexico circa 2013. Micah Altman has a copy of the AMI that was deployed to enable university students at ITAM to draw federal districts for the state of Mexico. ============ Dr. Michael P. McDonald Associate Professor, University of Florida 352-273-2371 www.electproject.org @ElectProject

…

On Mon, Aug 28, 2017 at 4:34 PM, Chris Alfano ***@***.***> wrote: @hectcastro <https://github.com/hectcastro> @mcdisc <https://github.com/mcdisc> does anyone know what the last commit of DistrictBuilder that is running online is? I've gotten past environmental issues I think and just gotten down to the list of python deps in master and the v2 tag just being unworkable. It's hard to see how either of those commits could ever get running — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <PublicMapping/districtbuilder#453 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBMSdeYmXB-1JVkyUNth4nJFQlBxj-yks5scyRJgaJpZM4Omh7y> .

hectcastro · 2017-08-29T11:57:37Z

Unfortunately, it is unclear what version was most recently used from our perspective (most people involved with those efforts have moved on). I started with master and created a branch to try and get the application up via Docker/Docker Compose. Progress on that is in the development-env branch.

Currently, the last RUN command is failing because psycopg2 cannot connect to the templated database endpoint. Next steps are probably to determine how to override that value so that it can try to connect to the commented out PostgreSQL configuration in docker-compose.yml (after links are established).

Hopefully that gives you a better path forward. I'll try to pick this up again when I have more time.

themightychris · 2017-09-05T17:35:56Z

I wonder if this (new?) tool might help, just saw some talk of it in the habitat #python channel: https://github.com/ofek/hatch

themightychris added 3 commits July 28, 2017 08:03

WIP: Add habitat plan

4192397

WIP: Attempt to get through pip install

ae9d561

Move git to build deps and use HEAD instead of configured branch

a9b4dc2

themightychris changed the title ~~Build with habitat~~ WIP: Build with habitat Jul 28, 2017

themightychris added 13 commits July 28, 2017 14:23

Use all cacert env vars instead of disabling verification

d28b5f1

Use newer builds from python2 origin instead of core

0c55ef5

Move direct package URL to requirements.txt

eca15c3

Add env for habitat studio

f30e693

Add habitat results/ to .gitignore

81b6b3d

Add required build headers

ac38374

Remove extra attach

64cd274

Bump some pip versions

0284b4c

Use virtualenv and move pip install to install phase

62092f6

Switch back to core/python2

2bed154

Order build_deps before deps

feb0797

Deactivate strip phase

a0d135b

Use upgraded pip and virtualenv

41afba0

themightychris added 4 commits July 28, 2017 18:28

Add lxml to requirements.txt

2aa2317

Copy django tree into $pkg_prefix

ac2fb9b

Bump celery to 3.1.25

d83ca8c

Bump Django to latest 1.4.x patch release 22

e75652d

themightychris added 2 commits July 29, 2017 01:05

Use tzdata package and patch zoneinfo path into Django

2dc0346

Add geos and psycopg2 to requirements.txt

ca42696

themightychris changed the title ~~WIP: Build with habitat~~ WIP: Build master with habitat Jul 29, 2017

themightychris mentioned this pull request Aug 15, 2017

Docker deployment? #454

Closed

hectcastro mentioned this pull request Aug 30, 2017

Is this project alive? Problems with elderly installation instructions. #456

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Build master with habitat #453

WIP: Build master with habitat #453

themightychris commented Jul 28, 2017 •

edited

Loading

themightychris commented Jul 28, 2017

mcdisc commented Jul 28, 2017 via email

themightychris commented Jul 28, 2017

mcdisc commented Jul 28, 2017 via email

themightychris commented Jul 28, 2017

themightychris commented Jul 28, 2017

themightychris commented Jul 29, 2017

hectcastro commented Aug 1, 2017

themightychris commented Aug 3, 2017 •

edited

Loading

themightychris commented Aug 3, 2017

msarahan commented Aug 25, 2017

themightychris commented Aug 25, 2017

msarahan commented Aug 28, 2017

themightychris commented Aug 28, 2017

mcdisc commented Aug 29, 2017 via email

hectcastro commented Aug 29, 2017

themightychris commented Sep 5, 2017

WIP: Build master with habitat #453

Are you sure you want to change the base?

WIP: Build master with habitat #453

Conversation

themightychris commented Jul 28, 2017 • edited Loading

First-time setup

Enter studio and build package

Setup app within studio after a successful build or install

themightychris commented Jul 28, 2017

mcdisc commented Jul 28, 2017 via email

themightychris commented Jul 28, 2017

mcdisc commented Jul 28, 2017 via email

themightychris commented Jul 28, 2017

themightychris commented Jul 28, 2017

themightychris commented Jul 29, 2017

hectcastro commented Aug 1, 2017

themightychris commented Aug 3, 2017 • edited Loading

themightychris commented Aug 3, 2017

msarahan commented Aug 25, 2017

themightychris commented Aug 25, 2017

msarahan commented Aug 28, 2017

themightychris commented Aug 28, 2017

mcdisc commented Aug 29, 2017 via email

hectcastro commented Aug 29, 2017

themightychris commented Sep 5, 2017

themightychris commented Jul 28, 2017 •

edited

Loading

themightychris commented Aug 3, 2017 •

edited

Loading