Prost! Development

This page describes how to work with Prost! in a development environment.

Setup

The setup will walk you through retrieving the code, setting up the git flow tools, and doing a sample commit.

Grab the Prost! codebase from GitHub

# Choose a repo and grab it:
# public repo via HTTPS
git clone https://github.com/uoregon-postlethwait/prost.git
# public repo via SSH
git clone git@github.com:uoregon-postlethwait/prost.git
# development repo via HTTPS
git clone https://github.com/uoregon-postlethwait/prost-dev.git
# development repo via SSH
git clone git@github.com:uoregon-postlethwait/prost-dev.git

# Enter the development directory.
cd prost

# Checkout the branch you want to work on.
git checkout develop

Optional: Configure local Prost! repo for public and development repos

Some Prost! development occurs on a private development GitHub repository (e.g. quick/dirty custom features for collaborators that aren’t fit for public distribution).

Setup your local git repo to be able to push/pull from both the public GitHub Prost! repo and the private Github Prost! development repo like so. Your initial .git/config should look like this:

[remote "origin"]
     url = https://github.com/uoregon-postlethwait/prost-dev
     fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
     remote = origin
     merge = refs/heads/master

Change it to look like this:

[remote "private"]
     url = https://github.com/uoregon-postlethwait/prost-dev
     fetch = +refs/heads/*:refs/remotes/origin/*
[remote "public"]
     url = https://github.com/uoregon-postlethwait/prost
     fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
     remote = private
     merge = refs/heads/master

Setup Git Flow (AVH) Tools

We follow the git flow model of development. For background see: Why Aren’t You Using Git Flow?.

We use the AVH forked version of the git flow tools because the original git flow tools are no longer maintained, and AVH seems like the best replacement out there at the moment (it’s maintained, and adds a few new features). See https://github.com/petervanderdoes/gitflow.

We use a mildly forked version of git flow hooks provided by https://github.com/jaspernbrouwer/git-flow-hooks, which provides automatic version bumping for releases and hotfixes (and a few other small but nice features). Our forked/patched version (see https://github.com/jasonsydes/git-flow-hooks) just adds the tiny feature of also updating prost/_version.py in addition to the standard updating of the VERSION file.

To setup the git flow environment:

  • Install git flow (the AVH version): https://github.com/petervanderdoes/gitflow/wiki. Use the defaults for all, except for versiontag, for which you should specify ‘v’:

    Version tag prefix? [] v
    
  • Install our (very mildly) patched version of git-flow-hooks:

    # cd to your checkout of the prost repository
    cd /path/to/prost-repo
    
    # cd into the .git directory
    cd .git
    
    # Rename the hooks directory to something else.  The default git hooks
    # directory just has some examples in it.  Safe to rename, or even delete.
    mv hooks hooks.orig
    
    # Clone our patched version of the git flow hooks repo into hooks.
    git clone https://github.com/jasonsydes/git-flow-hooks hooks
    
    # One small change: by default, releases should be "PATCH" level by
    # default (e.g. v0.7.20 -> v0.7.21) just like hotfixes.  To do a "MINOR"
    # "MAJOR" release, you can specify it manually.  See the section on
    # "Releases" below for more info.
    #
    # To enable this, copy and rename the following file to the .git/ dir:
    cp hooks/modules/git-flow-hooks-config.sh.dist git-flow-hooks-config.sh
    
    # cd back to the repo:
    cd ..
    

HotFix

Hotfixes are tiny fixes to an existing release. The hotfix may optionally be pushed to the public repo as well. Do the following:

# Start a hotfix
git flow hotfix start
# Make your changes to the code (i.e. implement the hotfix).
vim thecode.py
# Add and commit the code.
git add thecode.py
git ci -m 'Made these changes...'
# Finish the hotfix
git flow hotfix finish -m
# Push everything to private repository
git push private --mirror
# Optional: If this is to be a public release, push just the master repo and
#    the tag just created to the public repo.
git push public master
git push public v0.7.17

Release

TODO: Add instructions on performing a release with git flow.

Feature

TODO: Add instructions on working on large new features with git flow.

Publish on PyPI

Once you’ve published github, you need to publish to PyPI as well (so that people can do pip install prost). Setup the following ~/.pypirc:

bash
[distutils] # this tells distutils what package indexes you can push to
index-servers =
    pypi
    pypitest

[pypi]
repository: https://pypi.python.org/pypi
username: jasonsydes
password: the_password_for_our_pypi_account

[pypitest]
repository: https://testpypi.python.org/pypi
username: jasonsydes
password: the_password_for_our_pypi_account

Once you’ve got that, you can upload to PyPI:

# Optional: Upload to the test PyPI server (regularly cleaned out).
python setup.py sdist upload -r pypitest
# Confirm it worked at this URL: https://testpypi.python.org/pypi/prost

# Upload to the PyPI server (confirm at
# https://pypi.python.org/pypi/prost).
python setup.py sdist upload -r pypi

Running Prost! from a specific tag or commit

For performing test-data-runs (i.e. for running Prost! on a full dataset where the goal is producing analyzable results, as opposed to developing and then running Prost! on small test datasets), you may not want to run Prost! from a branch, because that branch might change on you. Instead you may wish to run Prost! directly from a specific commit or tag, which is guaranteed not to change. To do so:

# Checkout a specific tag
git checkout tag_name

# Checkout a specific commit
git checkout cc92245

And then simply note the tag_name or commit SHA-1 in your lab notebook for that particular Prost! run.

Note that checking out a specific tag or commit will result in the following omimous warning:

∴ git checkout cc92245
Note: checking out 'cc92245'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at cc92245... added development environment documentation

That’s ok. As long as you only plan on running Prost! and not making any changes, you have nothing to worry about. If you do wish to make code changes, then just follow the directions above.

When you’re finished, just do:

# Switch back to a working branch
git checkout branch_name

# Get recent changes
git pull

Working with Prost! Documentation

Prost! documentation is quickly accessible locally under the doc/ directory.

To build the docs:

cd prost/doc
make html

# Alternatively
cd prost
./build.sh

To read the docs:

Currently, the docs aren’t perfectly intertwined. Eventually, we’ll have them all linked to one another. For now, you can access the individual pages. Just look for the *.html pages. For example, to view this document locally on OS X:

cd prost/doc
open _build/html/development.html

However, see also Using sphinx-autobuild to develop the docs: below for an easier method.

To edit the docs:

The docs are written in Sphinx’s reStructuredText. See http://sphinx-doc.org/rest.html for a nice tutorial. The docs try to follow the Google Python Style Guide, and the parsing of this style is provided by the Napoleon Sphinx plugin. This page of Example Google Style Python Docstrings is particular helpful.

Simply find the *.rst document under the doc/ directory that you wish to edit, edit it, rebuild the docs, and view the result locally in your browser.

For a new document, I usually start by copying an existing *.rst document and modify it.

Using sphinx-autobuild to develop the docs:

Sphinx Autobuild is a nice tool that runs in the background, automatically builds the docs when you save a change, and serves the docs on a local server to your web browser at http://127.0.0.1:8000.

To install:

pip install sphinx-autobuild

To use:

# Easy
cd docs
make livehtml

# If 'Easy' doesnt' work:
cd /path/to/toplevel/repo
sphinx-autobuild docs docs/_build/html/

To view, point your browser to http://127.0.0.1:8000.