Merge pull request #126 from parente/wiki-webhook

Webhook to update docker builds page

README.md

@@ -20,23 +20,22 @@ If this is your first time using Docker or any of the Jupyter projects, do the f
 2. Open the README in one of the folders in this git repository.
 3. Follow the README for that stack.
 
-## Tips and Gotchas
-
-* `tini -- start-notebook.sh` is the default Docker entrypoint-plus-command in every notebook stack. If you plan to modify it any way, be sure to check the *Notebook Options* section of your stack's README to understand the consequences.
-* Check the [Docker recipes wiki page](https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes) attached to this project for information about extending and deploying the Docker images defined here. Add to the wiki if you have relevant information.
-
 ## Stacks, Tags, Versioning, and Progress
 
 Starting with [git commit SHA 9bd33dcc8688](https://github.com/jupyter/docker-stacks/tree/9bd33dcc8688):
 
 * Every folder here on GitHub has an equivalent `jupyter/<stack name>` on Docker Hub.
 * The `latest` tag in each Docker Hub repository tracks the `master` branch `HEAD` reference on GitHub.
-* Any 12-character image tag on Docker Hub refers to a git commit SHA here on GitHub. 
+* Any 12-character image tag on Docker Hub refers to a git commit SHA here on GitHub. See the [Docker build history wiki page](https://github.com/jupyter/docker-stacks/wiki/Docker-build-history) for a table of build details.
 * Stack contents (e.g., new library versions) will be updated upon request via PRs against this project.
 * Users looking to remain on older builds should refer to specific git SHA tagged images in their work, not `latest`.
-* Users who want to know the contents of a specific tagged image on Docker Hub can take its tag and use it as a git SHA to inspect the state of this repo at the time of that image build.
 * For legacy reasons, there are two additional tags named `3.2` and `4.0` on Docker Hub which point to images prior to our versioning scheme switch.
 
+## Other Tips
+
+* `tini -- start-notebook.sh` is the default Docker entrypoint-plus-command in every notebook stack. If you plan to modify it any way, be sure to check the *Notebook Options* section of your stack's README to understand the consequences.
+* Check the [Docker recipes wiki page](https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes) attached to this project for information about extending and deploying the Docker images defined here. Add to the wiki if you have relevant information.
+
 ## Maintainer Workflow
 
 For PRs that impact the definition of one or more stacks:

internal/docker-stacks-webhook/conda_requirements.txt

+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+# NOTE: jupyter_kernel_gateway uses the notebook package. We get a
+# speed up if we pre-install the notebook package using conda since
+# we'll get prebuilt binaries for all its dependencies like pyzmq.
+notebook==4.1

internal/docker-stacks-webhook/docker-stacks-webhook.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# docker-stacks wiki webhook\n",
+    "\n",
+    "Listens for webhook callbacks from Docker Hub. Updates the [docker build history](https://github.com/jupyter/docker-stacks/wiki/Docker-build-history) wiki page in response to completed builds.\n",
+    "\n",
+    "References:\n",
+    "\n",
+    "* https://docs.docker.com/docker-hub/webhooks/\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import datetime as dt\n",
+    "import requests\n",
+    "import os"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Read credentials from the environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "GH_USERNAME = os.getenv('GH_USERNAME')\n",
+    "GH_TOKEN = os.getenv('GH_TOKEN')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Configure git upfront."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "git config --global user.email \"jupyter@googlegroups.com\"\n",
+    "git config --global user.name \"Jupyter Development Team\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Build the templates we need."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "wiki_git_tmpl = 'https://{GH_USERNAME}:{GH_TOKEN}@github.com/jupyter/docker-stacks.wiki.git'\n",
+    "commit_url_tmpl = 'https://github.com/jupyter/docker-stacks/commit/{sha}'\n",
+    "row_tmpl = '|{pushed_at}|[{sha}]({commit_url})|{commit_msg}|\\n'\n",
+    "api_commit_url_tmpl = 'https://api.github.com/repos/jupyter/docker-stacks/commits/{sha}'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "REQUEST = json.dumps({\n",
+    "    'body' : {\n",
+    "       \"push_data\": {\n",
+    "          \"pushed_at\": 1449017033,\n",
+    "          \"images\": [],\n",
+    "          \"tag\": \"9f9907cf1df8\",\n",
+    "          \"pusher\": \"biscarch\"\n",
+    "       },\n",
+    "       \"callback_url\": \"https://registry.hub.docker.com/u/biscarch/webhook-tester-repo/hook/2i5e3gj1bi354asb3f05gchi4ccjg0gas/\",\n",
+    "       \"repository\": {\n",
+    "          \"status\": \"Active\",\n",
+    "          \"description\": \"\",\n",
+    "          \"is_trusted\": False,\n",
+    "          \"full_description\": None,\n",
+    "          \"repo_url\": \"https://registry.hub.docker.com/u/biscarch/webhook-tester-repo/\",\n",
+    "          \"owner\": \"biscarch\",\n",
+    "          \"is_official\": False,\n",
+    "          \"is_private\": False,\n",
+    "          \"name\": \"webhook-tester-repo\",\n",
+    "          \"namespace\": \"biscarch\",\n",
+    "          \"star_count\": 0,\n",
+    "          \"comment_count\": 0,\n",
+    "          \"date_created\": 1449016916,\n",
+    "          \"repo_name\": \"biscarch/webhook-tester-repo\"\n",
+    "       }\n",
+    "    }\n",
+    "})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Read values we need out of the request body."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "body = json.loads(REQUEST)['body']\n",
+    "\n",
+    "tag = body['push_data']['tag']\n",
+    "pushed_at_ts = body['push_data']['pushed_at']\n",
+    "callback_url = body['callback_url']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Validate the request by seeing if the tag is a valid SHA in the docker-stacks repo."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "commit_resp = requests.get(api_commit_url_tmpl.format(sha=tag))\n",
+    "try:\n",
+    "    commit_resp.raise_for_status()\n",
+    "except Exception as ex:\n",
+    "    requests.post(callback_url, json={\n",
+    "        'state': 'failure',\n",
+    "        'description': 'request does not contain a valid sha',\n",
+    "        'context' : 'docker-stacks-webhook',\n",
+    "        'target_url' : 'https://github.com/jupyter/docker-stacks/wiki/Docker-build-history'\n",
+    "    })\n",
+    "    raise ex"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Get a fresh clone of the wiki git repo."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "wiki_git = wiki_git_tmpl.format(GH_USERNAME=GH_USERNAME, GH_TOKEN=GH_TOKEN)\n",
+    "\n",
+    "!rm -rf docker-stacks.wiki\n",
+    "!git clone $wiki_git"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Read the build page markdown."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "with open('docker-stacks.wiki/Docker-build-history.md') as f:\n",
+    "    lines = f.readlines()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Find the start of the table."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "for table_top_i, line in enumerate(lines):\n",
+    "    if line.startswith('|--'):\n",
+    "        break\n",
+    "else:\n",
+    "    requests.post(callback_url, json={\n",
+    "        'state': 'failure',\n",
+    "        'description': 'could not locate table on wiki page',\n",
+    "        'context' : 'docker-stacks-webhook',\n",
+    "        'target_url' : 'https://github.com/jupyter/docker-stacks/wiki/Docker-build-history'\n",
+    "    })\n",
+    "    raise RuntimeError('wiki table missing')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Format the text we want to put into the wiki table row."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "pushed_at_dt = dt.datetime.fromtimestamp(pushed_at_ts)\n",
+    "pushed_at = pushed_at_dt.strftime('%b. %d, %Y')\n",
+    "commit_url = commit_url_tmpl.format(sha=tag)\n",
+    "commit_msg = commit_resp.json()['commit']['message'].replace('\\n', ' ')\n",
+    "row = row_tmpl.format(pushed_at=pushed_at, sha=tag, commit_url=commit_url, commit_msg=commit_msg)\n",
+    "row"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Insert the table row."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "lines.insert(table_top_i+1, row)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Write the file back out."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "with open('docker-stacks.wiki/Docker-build-history.md', 'w') as f:\n",
+    "    f.writelines(lines)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Commit and push."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "!cd docker-stacks.wiki/ && \\\n",
+    "    git add -A && \\\n",
+    "    git commit -m 'Add build $tag' && \\\n",
+    "    git push origin master"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Tell Docker Hub we succeeded."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# POST /tag\n",
+    "resp = requests.post(callback_url, json={\n",
+    "    'state': 'success',\n",
+    "    'description': 'updated docker-stacks wiki build page',\n",
+    "    'context' : 'docker-stacks-webhook',\n",
+    "    'target_url' : 'https://github.com/jupyter/docker-stacks/wiki/Docker-build-history'\n",
+    "})\n",
+    "\n",
+    "print(resp.status_code)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.5.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

internal/docker-stacks-webhook/manifest.yml

+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+---
+applications:
+- name: docker-stacks-webhook
+  memory: 128M
+  instances: 1
+  path: .
+  buildpack: https://github.com/ihuston/python-conda-buildpack
+  command: >
+    jupyter-kernelgateway --KernelGatewayApp.port=$PORT
+    --KernelGatewayApp.ip=0.0.0.0
+    --KernelGatewayApp.api=notebook-http
+    --KernelGatewayApp.seed_uri='./docker-stacks-webhook.ipynb'

internal/docker-stacks-webhook/requirements.txt

+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+git+https://github.com/parente/kernel_gateway@concatenate-cells

internal/docker-stacks-webhook/runtime.txt

+python-3.19.0