Merge pull request #125 from jtyberg/compose

Add example for deploying docker-stacks using Docker Compose

.gitignore

@@ -58,3 +58,6 @@ docs/_build/
 
 # PyBuilder
 target/
+
+# Mac OS X
+.DS_Store

examples/docker-compose/README.md

+This example demonstrate how to deploy docker-stack notebook containers to any Docker Machine-controlled host using Docker Compose.
+
+## Prerequisites
+
+* [Docker Engine](https://docs.docker.com/engine/) 1.10.0+
+* [Docker Machine](https://docs.docker.com/machine/) 0.6.0+
+* [Docker Compose](https://docs.docker.com/compose/) 1.6.0+
+
+See the [installation instructions](https://docs.docker.com/engine/installation/) for your environment.
+
+## Quickstart
+
+Build and run a `jupyter/minimal-notebook` container on a VirtualBox VM on local desktop.
+
+```
+# create a Docker Machine-controlled VirtualBox VM
+bin/vbox.sh mymachine
+
+# activate the docker machine
+eval "$(docker-machine env mymachine)"
+
+# build the notebook image on the machine
+notebook/build.sh
+
+# bring up the notebook container
+notebook/up.sh
+```
+
+To stop and remove the container:
+
+```
+notebook/down.sh
+```
+
+
+## FAQ
+
+### How do I specify which docker-stack notebook image to deploy?
+
+You can customize the docker-stack notebook image to deploy by modifying the `notebook/Dockerfile`.  For example, you can build and deploy a `jupyter/all-spark-notebook` by modifying the Dockerfile like so:
+
+```
+FROM jupyter/all-spark-notebook:55d5ca6be183
+...
+```
+
+Once you modify the Dockerfile, don't forget to rebuild the image.
+
+```
+# activate the docker machine
+eval "$(docker-machine env mymachine)"
+
+notebook/build.sh
+```
+
+### Can I run multiple notebook containers on the same VM?
+
+Yes. Set environment variables to specify unique names and ports when running the `up.sh` command.
+
+```
+NAME=my-notebook PORT=9000 notebook/up.sh
+NAME=your-notebook PORT=9001 notebook/up.sh
+```
+
+To stop and remove the containers:
+
+```
+NAME=my-notebook notebook/down.sh
+NAME=your-notebook notebook/down.sh
+```
+
+### Where are my notebooks stored?
+
+The `up.sh` creates a Docker volume named after the notebook container with a `-work` suffix, e.g., `my-notebook-work`.
+
+
+### Can multiple notebook containers share the same notebook volume?
+
+Yes. Set the `WORK_VOLUME` environment variable to the same value for each notebook.
+
+```
+NAME=my-notebook PORT=9000 WORK_VOLUME=our-work notebook/up.sh
+NAME=your-notebook PORT=9001 WORK_VOLUME=our-work notebook/up.sh
+```
+
+### How do I run over HTTPS?
+
+To run the notebook server with a self-signed certificate, pass the `--secure` option to the `up.sh` script.  You must also provide a password, which will be used to secure the notebook server.  You can specify the password by setting the `PASSWORD` environment variable, or by passing it to the `up.sh` script.  
+
+```
+PASSWORD=a_secret notebook/up.sh --secure
+
+# or
+notebook/up.sh --secure --password a_secret
+```
+
+### Can I use Let's Encrypt certificate chains?
+
+Sure.  If you want to secure access to publicly addressable notebook containers, you can generate a free certificate using the [Let's Encrypt](https://letsencrypt.org) service.
+
+
+This example includes the `bin/letsencrypt.sh` script, which runs the `letsencrypt` client to create a full-chain certificate and private key, and stores them in a Docker volume.  _Note:_ The script hard codes several `letsencrypt` options, one of which automatically agrees to the Let's Encrypt Terms of Service.
+
+The following command will create a certificate chain and store it in a Docker volume named `mydomain-secrets`.
+
+```
+FQDN=host.mydomain.com EMAIL=myemail@somewhere.com \
+  SECRETS_VOLUME=mydomain-secrets \
+  bin/letsencrypt.sh
+```
+
+Now run `up.sh` with the `--letsencrypt` option.  You must also provide the name of the secrets volume and a password.
+
+```
+PASSWORD=a_secret SECRETS_VOLUME=mydomain-secrets notebook/up.sh --letsencrypt
+
+# or
+notebook/up.sh --letsencrypt --password a_secret --secrets mydomain-secrets
+```
+
+Be aware that Let's Encrypt has a pretty [low rate limit per domain](https://community.letsencrypt.org/t/public-beta-rate-limits/4772/3) at the moment.  You can avoid exhausting your limit by testing against the Let's Encrypt staging servers.  To hit their staging servers, set the environment variable `CERT_SERVER=--staging`.
+
+```
+FQDN=host.mydomain.com EMAIL=myemail@somewhere.com \
+  CERT_SERVER=--staging \
+  bin/letsencrypt.sh
+```
+
+Also, be aware that Let's Encrypt certificates are short lived (90 days).  If you need them for a longer period of time, you'll need to manually setup a cron job to run the renewal steps. (You can reuse the command above.)
+
+### Can I deploy to any Docker Machine host?
+
+Yes, you should be able to deploy to any Docker Machine-controlled host.  To make it easier to get up and running, this example includes scripts to provision Docker Machines to VirtualBox and IBM SoftLayer, but more scripts are welcome!
+
+To create a Docker machine using a VirtualBox VM on local desktop:
+
+```
+bin/vbox.sh mymachine
+```
+
+To create a Docker machine using a virtual device on IBM SoftLayer:
+
+```
+export SOFTLAYER_USER=my_softlayer_username
+export SOFTLAYER_API_KEY=my_softlayer_api_key
+export SOFTLAYER_DOMAIN=my.domain
+
+# Create virtual device
+bin/softlayer.sh myhost
+
+# Add DNS entry (SoftLayer DNS zone must exist for SOFTLAYER_DOMAIN)
+bin/sl-dns.sh myhost
+```
+
+
+## Troubleshooting
+
+### Unable to connect to VirtualBox VM on Mac OS X when using Cisco VPN client.
+
+The Cisco VPN client blocks access to IP addresses that it does not know about, and may block access to a new VM if it is created while the Cisco VPN client is running.
+
+1. Stop Cisco VPN client. (It does not allow modifications to route table).
+2. Run `ifconfig` to list `vboxnet` virtual network devices.
+3. Run `sudo route -nv add -net 192.168.99 -interface vboxnetX`, where X is the number of the virtual device assigned to the VirtualBox VM.
+4. Start Cisco VPN client.

examples/docker-compose/bin/letsencrypt.sh

+#!/bin/bash
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+# Use https://letsencrypt.org to create a certificate for a single domain
+# and store it in a Docker volume.
+
+set -e
+
+# Get domain and email from environment
+[ -z "$FQDN" ] && \
+	echo "ERROR: Must set FQDN environment varable" && \
+  exit 1
+
+[ -z "$EMAIL" ] && \
+	echo "ERROR: Must set EMAIL environment varable" && \
+  exit 1
+
+# letsencrypt certificate server type (default is production).
+# Set `CERT_SERVER=--staging` for staging.
+: ${CERT_SERVER=''}
+
+# Create Docker volume to contain the cert
+: ${SECRETS_VOLUME:=my-notebook-secrets}
+docker volume create --name $SECRETS_VOLUME 1>/dev/null
+# Generate the cert and save it to the Docker volume
+docker run --rm -it \
+  -p 80:80 \
+  -v $SECRETS_VOLUME:/etc/letsencrypt \
+  quay.io/letsencrypt/letsencrypt:latest \
+  certonly \
+  --non-interactive \
+  --keep-until-expiring \
+  --standalone \
+  --standalone-supported-challenges http-01 \
+  --agree-tos \
+  --domain "$FQDN" \
+  --email "$EMAIL" \
+  $CERT_SERVER
+
+# Set permissions so nobody can read the cert and key.
+# Also symlink the certs into the root of the /etc/letsencrypt
+# directory so that the FQDN doesn't have to be known later.
+docker run --rm -it \
+  -v $SECRETS_VOLUME:/etc/letsencrypt \
+  debian:jessie \
+		bash -c "ln -s /etc/letsencrypt/live/$FQDN/* /etc/letsencrypt/ && \
+			find /etc/letsencrypt -type d -exec chmod 755 {} +"

examples/docker-compose/bin/sl-dns.sh

+#!/bin/bash
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+set -e
+
+# User must have slcli installed
+which slcli > /dev/null || (echo "SoftLayer cli not found (pip install softlayer)"; exit 1)
+
+USAGE="Usage: `basename $0` machine_name [domain]"
+E_BADARGS=85
+
+# Machine name is first command line arg
+MACHINE_NAME=$1 && [ -z "$MACHINE_NAME" ] && echo "$USAGE" && exit $E_BADARGS
+
+# Use SOFTLAYER_DOMAIN env var if domain name not set as second arg
+DOMAIN="${2:-$SOFTLAYER_DOMAIN}" && [ -z "$DOMAIN" ] && \
+	echo "Must specify domain or set SOFTLAYER_DOMAIN environment varable" && \
+	echo "$USAGE" && exit $E_BADARGS
+
+IP=$(docker-machine ip "$MACHINE_NAME")
+
+slcli dns record-add $DOMAIN $MACHINE_NAME A $IP

examples/docker-compose/bin/softlayer.sh

+#!/bin/bash
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+# Set default SoftLayer VM settings
+: ${SOFTLAYER_CPU:=4}
+export SOFTLAYER_CPU
+: ${SOFTLAYER_DISK_SIZE:=100}
+export SOFTLAYER_DISK_SIZE
+: ${SOFTLAYER_MEMORY:=4096}
+export SOFTLAYER_MEMORY
+: ${SOFTLAYER_REGION:=wdc01}
+export SOFTLAYER_REGION
+
+docker-machine create --driver softlayer "$@"

examples/docker-compose/bin/vbox.sh

+#!/bin/bash
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+# Set reasonable default VM settings
+: ${VIRTUALBOX_CPUS:=4}
+export VIRTUALBOX_CPUS
+: ${VIRTUALBOX_MEMORY_SIZE:=4096}
+export VIRTUALBOX_MEMORY_SIZE
+
+docker-machine create --driver virtualbox "$@"

examples/docker-compose/notebook/Dockerfile

+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+# Pick your favorite docker-stacks image
+FROM jupyter/minimal-notebook:55d5ca6be183
+
+USER jovyan
+
+# Add permanent pip/conda installs, data files, other user libs here
+# e.g., RUN pip install jupyter_dashboards
+
+USER root
+
+# Add permanent apt-get installs and other root commands here
+# e.g., RUN apt-get install npm nodejs

examples/docker-compose/notebook/build.sh

+#!/bin/bash
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+# Setup environment
+source "$DIR/env.sh"
+
+# Build the notebook image
+docker-compose -f "$DIR/notebook.yml" build

examples/docker-compose/notebook/down.sh

+#!/bin/bash
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+# Setup environment
+source "$DIR/env.sh"
+
+# Bring down the notebook container, using container name as project name
+docker-compose -f "$DIR/notebook.yml" -p "$NAME" down

examples/docker-compose/notebook/env.sh

+#!/bin/bash
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+# Set default values for environment variables required by notebook compose
+# configuration file.
+
+# Container name
+: "${NAME:=my-notebook}"
+export NAME
+
+# Exposed container port
+: ${PORT:=80}
+export PORT
+
+# Container work volume name
+: "${WORK_VOLUME:=$NAME-work}"
+export WORK_VOLUME
+
+# Container secrets volume name
+: "${SECRETS_VOLUME:=$NAME-secrets}"
+export SECRETS_VOLUME

examples/docker-compose/notebook/letsencrypt-notebook.yml

+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+version: "2"
+
+services:
+  notebook:
+    build: .
+    image: my-notebook
+    container_name: ${NAME}
+    volumes:
+     - "work:/home/jovyan/work"
+     - "secrets:/etc/letsencrypt"
+    ports:
+      - "${PORT}:8888"
+    environment:
+      USE_HTTPS: "yes"
+      PASSWORD: ${PASSWORD}
+    command: >
+      start-notebook.sh
+      --NotebookApp.certfile=/etc/letsencrypt/fullchain.pem
+      --NotebookApp.keyfile=/etc/letsencrypt/privkey.pem
+
+volumes:
+  work:
+    external:
+      name: ${WORK_VOLUME}
+  secrets:
+    external:
+      name: ${SECRETS_VOLUME}

examples/docker-compose/notebook/notebook.yml

+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+version: "2"
+
+services:
+  notebook:
+    build: .
+    image: my-notebook
+    container_name: ${NAME}
+    volumes:
+     - "work:/home/jovyan/work"
+    ports:
+      - "${PORT}:8888"
+
+volumes:
+  work:
+    external:
+      name: ${WORK_VOLUME}

examples/docker-compose/notebook/secure-notebook.yml

+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+version: "2"
+
+services:
+  notebook:
+    build: .
+    image: my-notebook
+    container_name: ${NAME}
+    volumes:
+     - "work:/home/jovyan/work"
+    ports:
+      - "${PORT}:8888"
+    environment:
+      USE_HTTPS: "yes"
+      PASSWORD: ${PASSWORD}
+
+volumes:
+  work:
+    external:
+      name: ${WORK_VOLUME}

examples/docker-compose/notebook/up.sh

+#!/bin/bash
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+set -e
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+USAGE="Usage: `basename $0` [--secure | --letsencrypt] [--password PASSWORD] [--secrets SECRETS_VOLUME]"
+
+# Parse args to determine security settings
+SECURE=${SECURE:=no}
+LETSENCRYPT=${LETSENCRYPT:=no}
+while [[ $# > 0 ]]
+do
+key="$1"
+case $key in
+    --secure)
+    SECURE=yes
+    ;;
+    --letsencrypt)
+    LETSENCRYPT=yes
+    ;;
+    --secrets)
+    SECRETS_VOLUME="$2"
+    shift # past argument
+    ;;
+    --password)
+    PASSWORD="$2"
+    export PASSWORD
+    shift # past argument
+    ;;
+    *) # unknown option
+    ;;
+esac
+shift # past argument or value
+done
+
+if [[ "$LETSENCRYPT" == yes || "$SECURE" == yes ]]; then
+  if [ -z "${PASSWORD:+x}" ]; then
+    echo "ERROR: Must set PASSWORD if running in secure mode"
+    echo "$USAGE"
+    exit 1
+  fi
+  if [ "$LETSENCRYPT" == yes ]; then
+    CONFIG=letsencrypt-notebook.yml
+    if [ -z "${SECRETS_VOLUME:+x}" ]; then
+      echo "ERROR: Must set SECRETS_VOLUME if running in letsencrypt mode"
+      echo "$USAGE"
+      exit 1
+    fi
+  else
+    CONFIG=secure-notebook.yml
+  fi
+  export PORT=${PORT:=443}
+else
+  CONFIG=notebook.yml
+  export PORT=${PORT:=80}
+fi
+
+# Setup environment
+source "$DIR/env.sh"
+
+# Create a Docker volume to store notebooks
+docker volume create --name "$WORK_VOLUME"
+
+# Bring up a notebook container, using container name as project name
+echo "Bringing up notebook '$NAME'"
+docker-compose -f "$DIR/$CONFIG" -p "$NAME" up -d
+
+IP=$(docker-machine ip $(docker-machine active))
+echo "Notebook $NAME listening on $IP:$PORT"