Skip to content

J
Jupyter Docker Stacks
Commit b3d786df authored by Kyle Kelley's avatar Kyle Kelley
Browse files
Options

Merge pull request #126 from parente/wiki-webhook 

Webhook to update docker builds page
parents c9428336 6a3761b0
Hide whitespace changes
Inline Side-by-side
Showing with 416 additions and 10 deletions
README.md
View file @ b3d786df
...@@ -20,23 +20,22 @@ If this is your first time using Docker or any of the Jupyter projects, do the f ...@@ -20,23 +20,22 @@ If this is your first time using Docker or any of the Jupyter projects, do the f
2. Open the README in one of the folders in this git repository. 2. Open the README in one of the folders in this git repository.
3. Follow the README for that stack. 3. Follow the README for that stack.
## Tips and Gotchas
* `tini -- start-notebook.sh` is the default Docker entrypoint-plus-command in every notebook stack. If you plan to modify it any way, be sure to check the *Notebook Options* section of your stack's README to understand the consequences.
* Check the [Docker recipes wiki page](https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes) attached to this project for information about extending and deploying the Docker images defined here. Add to the wiki if you have relevant information.
## Stacks, Tags, Versioning, and Progress ## Stacks, Tags, Versioning, and Progress
Starting with [git commit SHA 9bd33dcc8688](https://github.com/jupyter/docker-stacks/tree/9bd33dcc8688): Starting with [git commit SHA 9bd33dcc8688](https://github.com/jupyter/docker-stacks/tree/9bd33dcc8688):
* Every folder here on GitHub has an equivalent `jupyter/<stack name>` on Docker Hub. * Every folder here on GitHub has an equivalent `jupyter/<stack name>` on Docker Hub.
* The `latest` tag in each Docker Hub repository tracks the `master` branch `HEAD` reference on GitHub. * The `latest` tag in each Docker Hub repository tracks the `master` branch `HEAD` reference on GitHub.
* Any 12-character image tag on Docker Hub refers to a git commit SHA here on GitHub. * Any 12-character image tag on Docker Hub refers to a git commit SHA here on GitHub. See the [Docker build history wiki page](https://github.com/jupyter/docker-stacks/wiki/Docker-build-history) for a table of build details.
* Stack contents (e.g., new library versions) will be updated upon request via PRs against this project. * Stack contents (e.g., new library versions) will be updated upon request via PRs against this project.
* Users looking to remain on older builds should refer to specific git SHA tagged images in their work, not `latest`. * Users looking to remain on older builds should refer to specific git SHA tagged images in their work, not `latest`.
* Users who want to know the contents of a specific tagged image on Docker Hub can take its tag and use it as a git SHA to inspect the state of this repo at the time of that image build.
* For legacy reasons, there are two additional tags named `3.2` and `4.0` on Docker Hub which point to images prior to our versioning scheme switch. * For legacy reasons, there are two additional tags named `3.2` and `4.0` on Docker Hub which point to images prior to our versioning scheme switch.
## Other Tips
* `tini -- start-notebook.sh` is the default Docker entrypoint-plus-command in every notebook stack. If you plan to modify it any way, be sure to check the *Notebook Options* section of your stack's README to understand the consequences.
* Check the [Docker recipes wiki page](https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes) attached to this project for information about extending and deploying the Docker images defined here. Add to the wiki if you have relevant information.
## Maintainer Workflow ## Maintainer Workflow
For PRs that impact the definition of one or more stacks: For PRs that impact the definition of one or more stacks:
...@@ -69,4 +68,4 @@ When there's a new stack, do the following **before** trying to `make release-al ...@@ -69,4 +68,4 @@ When there's a new stack, do the following **before** trying to `make release-al
1. Create a new repo in the `jupyter` org on Docker Hub named after the stack folder in the git repo. 1. Create a new repo in the `jupyter` org on Docker Hub named after the stack folder in the git repo.
2. Grant the `stacks` team permission to write to the repo. 2. Grant the `stacks` team permission to write to the repo.
3. Copy/paste the short and long descriptions from one of the other docker-stacks repos on Docker Hub. Modify the appropriate values. 3. Copy/paste the short and long descriptions from one of the other docker-stacks repos on Docker Hub. Modify the appropriate values.
\ No newline at end of file
internal/docker-stacks-webhook/conda_requirements.txt 0 → 100644
View file @ b3d786df
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
# NOTE: jupyter_kernel_gateway uses the notebook package. We get a
# speed up if we pre-install the notebook package using conda since
# we'll get prebuilt binaries for all its dependencies like pyzmq.
notebook==4.1
internal/docker-stacks-webhook/docker-stacks-webhook.ipynb 0 → 100644
View file @ b3d786df
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# docker-stacks wiki webhook\n",
"\n",
"Listens for webhook callbacks from Docker Hub. Updates the [docker build history](https://github.com/jupyter/docker-stacks/wiki/Docker-build-history) wiki page in response to completed builds.\n",
"\n",
"References:\n",
"\n",
"* https://docs.docker.com/docker-hub/webhooks/\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import json\n",
"import datetime as dt\n",
"import requests\n",
"import os"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Read credentials from the environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"GH_USERNAME = os.getenv('GH_USERNAME')\n",
"GH_TOKEN = os.getenv('GH_TOKEN')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Configure git upfront."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%%bash\n",
"git config --global user.email \"jupyter@googlegroups.com\"\n",
"git config --global user.name \"Jupyter Development Team\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Build the templates we need."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"wiki_git_tmpl = 'https://{GH_USERNAME}:{GH_TOKEN}@github.com/jupyter/docker-stacks.wiki.git'\n",
"commit_url_tmpl = 'https://github.com/jupyter/docker-stacks/commit/{sha}'\n",
"row_tmpl = '|{pushed_at}|[{sha}]({commit_url})|{commit_msg}|\\n'\n",
"api_commit_url_tmpl = 'https://api.github.com/repos/jupyter/docker-stacks/commits/{sha}'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"REQUEST = json.dumps({\n",
" 'body' : {\n",
" \"push_data\": {\n",
" \"pushed_at\": 1449017033,\n",
" \"images\": [],\n",
" \"tag\": \"9f9907cf1df8\",\n",
" \"pusher\": \"biscarch\"\n",
" },\n",
" \"callback_url\": \"https://registry.hub.docker.com/u/biscarch/webhook-tester-repo/hook/2i5e3gj1bi354asb3f05gchi4ccjg0gas/\",\n",
" \"repository\": {\n",
" \"status\": \"Active\",\n",
" \"description\": \"\",\n",
" \"is_trusted\": False,\n",
" \"full_description\": None,\n",
" \"repo_url\": \"https://registry.hub.docker.com/u/biscarch/webhook-tester-repo/\",\n",
" \"owner\": \"biscarch\",\n",
" \"is_official\": False,\n",
" \"is_private\": False,\n",
" \"name\": \"webhook-tester-repo\",\n",
" \"namespace\": \"biscarch\",\n",
" \"star_count\": 0,\n",
" \"comment_count\": 0,\n",
" \"date_created\": 1449016916,\n",
" \"repo_name\": \"biscarch/webhook-tester-repo\"\n",
" }\n",
" }\n",
"})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Read values we need out of the request body."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# POST /tag\n",
"body = json.loads(REQUEST)['body']\n",
"\n",
"tag = body['push_data']['tag']\n",
"pushed_at_ts = body['push_data']['pushed_at']\n",
"callback_url = body['callback_url']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Validate the request by seeing if the tag is a valid SHA in the docker-stacks repo."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# POST /tag\n",
"commit_resp = requests.get(api_commit_url_tmpl.format(sha=tag))\n",
"try:\n",
" commit_resp.raise_for_status()\n",
"except Exception as ex:\n",
" requests.post(callback_url, json={\n",
" 'state': 'failure',\n",
" 'description': 'request does not contain a valid sha',\n",
" 'context' : 'docker-stacks-webhook',\n",
" 'target_url' : 'https://github.com/jupyter/docker-stacks/wiki/Docker-build-history'\n",
" })\n",
" raise ex"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get a fresh clone of the wiki git repo."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# POST /tag\n",
"wiki_git = wiki_git_tmpl.format(GH_USERNAME=GH_USERNAME, GH_TOKEN=GH_TOKEN)\n",
"\n",
"!rm -rf docker-stacks.wiki\n",
"!git clone $wiki_git"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Read the build page markdown."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# POST /tag\n",
"with open('docker-stacks.wiki/Docker-build-history.md') as f:\n",
" lines = f.readlines()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Find the start of the table."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# POST /tag\n",
"for table_top_i, line in enumerate(lines):\n",
" if line.startswith('|--'):\n",
" break\n",
"else:\n",
" requests.post(callback_url, json={\n",
" 'state': 'failure',\n",
" 'description': 'could not locate table on wiki page',\n",
" 'context' : 'docker-stacks-webhook',\n",
" 'target_url' : 'https://github.com/jupyter/docker-stacks/wiki/Docker-build-history'\n",
" })\n",
" raise RuntimeError('wiki table missing')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Format the text we want to put into the wiki table row."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# POST /tag\n",
"pushed_at_dt = dt.datetime.fromtimestamp(pushed_at_ts)\n",
"pushed_at = pushed_at_dt.strftime('%b. %d, %Y')\n",
"commit_url = commit_url_tmpl.format(sha=tag)\n",
"commit_msg = commit_resp.json()['commit']['message'].replace('\\n', ' ')\n",
"row = row_tmpl.format(pushed_at=pushed_at, sha=tag, commit_url=commit_url, commit_msg=commit_msg)\n",
"row"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Insert the table row."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# POST /tag\n",
"lines.insert(table_top_i+1, row)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Write the file back out."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# POST /tag\n",
"with open('docker-stacks.wiki/Docker-build-history.md', 'w') as f:\n",
" f.writelines(lines)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Commit and push."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# POST /tag\n",
"!cd docker-stacks.wiki/ && \\\n",
" git add -A && \\\n",
" git commit -m 'Add build $tag' && \\\n",
" git push origin master"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tell Docker Hub we succeeded."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# POST /tag\n",
"resp = requests.post(callback_url, json={\n",
" 'state': 'success',\n",
" 'description': 'updated docker-stacks wiki build page',\n",
" 'context' : 'docker-stacks-webhook',\n",
" 'target_url' : 'https://github.com/jupyter/docker-stacks/wiki/Docker-build-history'\n",
"})\n",
"\n",
"print(resp.status_code)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.1"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
internal/docker-stacks-webhook/manifest.yml 0 → 100644
View file @ b3d786df
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
---
applications:
- name: docker-stacks-webhook
memory: 128M
instances: 1
path: .
buildpack: https://github.com/ihuston/python-conda-buildpack
command: >
jupyter-kernelgateway --KernelGatewayApp.port=$PORT
--KernelGatewayApp.ip=0.0.0.0
--KernelGatewayApp.api=notebook-http
--KernelGatewayApp.seed_uri='./docker-stacks-webhook.ipynb'
internal/docker-stacks-webhook/requirements.txt 0 → 100644
View file @ b3d786df
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
git+https://github.com/parente/kernel_gateway@concatenate-cells
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment