Skip to content

J
Jupyter Docker Stacks
Commit f4bf2ba5 authored by Graham Dumpleton's avatar Graham Dumpleton
Browse files
Options

Merge branch 'master' of github.com:jupyter/docker-stacks into writable-home-directory

parents 4d32b218 239dff32
Hide whitespace changes
Inline Side-by-side
Showing with 427 additions and 11 deletions
all-spark-notebook/Dockerfile
View file @ f4bf2ba5
...@@ -19,7 +19,7 @@ RUN apt-get update && \ ...@@ -19,7 +19,7 @@ RUN apt-get update && \
gcc && apt-get clean && \ gcc && apt-get clean && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
USER $NB_USER USER $NB_UID
# R packages # R packages
RUN conda install --quiet --yes \ RUN conda install --quiet --yes \
......
base-notebook/Dockerfile
View file @ f4bf2ba5
...@@ -53,7 +53,7 @@ RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \ ...@@ -53,7 +53,7 @@ RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
fix-permissions $HOME && \ fix-permissions $HOME && \
fix-permissions $CONDA_DIR fix-permissions $CONDA_DIR
USER $NB_USER USER $NB_UID
# Setup work directory for backward-compatibility # Setup work directory for backward-compatibility
RUN mkdir /home/$NB_USER/work && \ RUN mkdir /home/$NB_USER/work && \
...@@ -105,4 +105,4 @@ COPY jupyter_notebook_config.py /etc/jupyter/ ...@@ -105,4 +105,4 @@ COPY jupyter_notebook_config.py /etc/jupyter/
RUN fix-permissions /etc/jupyter/ RUN fix-permissions /etc/jupyter/
# Switch back to jovyan to avoid accidental container runs as root # Switch back to jovyan to avoid accidental container runs as root
USER $NB_USER USER $NB_UID
base-notebook/Dockerfile.ppc64le
View file @ f4bf2ba5
...@@ -55,7 +55,7 @@ RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \ ...@@ -55,7 +55,7 @@ RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
mkdir -p $CONDA_DIR && \ mkdir -p $CONDA_DIR && \
chown $NB_USER $CONDA_DIR chown $NB_USER $CONDA_DIR
USER $NB_USER USER $NB_UID
# Setup jovyan home directory # Setup jovyan home directory
RUN mkdir /home/$NB_USER/work && \ RUN mkdir /home/$NB_USER/work && \
...@@ -98,4 +98,4 @@ COPY jupyter_notebook_config.py /home/$NB_USER/.jupyter/ ...@@ -98,4 +98,4 @@ COPY jupyter_notebook_config.py /home/$NB_USER/.jupyter/
RUN chown -R $NB_USER:users /home/$NB_USER/.jupyter RUN chown -R $NB_USER:users /home/$NB_USER/.jupyter
# Switch back to jovyan to avoid accidental container runs as root # Switch back to jovyan to avoid accidental container runs as root
USER $NB_USER USER $NB_UID
datascience-notebook/Dockerfile
View file @ f4bf2ba5
...@@ -36,7 +36,7 @@ RUN mkdir /etc/julia && \ ...@@ -36,7 +36,7 @@ RUN mkdir /etc/julia && \
chown $NB_USER $JULIA_PKGDIR && \ chown $NB_USER $JULIA_PKGDIR && \
fix-permissions $JULIA_PKGDIR fix-permissions $JULIA_PKGDIR
USER $NB_USER USER $NB_UID
# R packages including IRKernel which gets installed globally. # R packages including IRKernel which gets installed globally.
RUN conda config --system --append channels r && \ RUN conda config --system --append channels r && \
......
examples/openshift/README.md 0 → 100644
View file @ f4bf2ba5
This example provides templates for deploying the Jupyter Project docker-stacks images to OpenShift.
Prerequsites
------------
Any OpenShift 3 environment. The templates were tested with OpenShift 3.7. It is believed they should work with at least OpenShift 3.6 or later.
Do be aware that the Jupyter Project docker-stacks images are very large. The OpenShift environment you are using must provide sufficient quota on the per user space for images and the file system for running containers. If the quota is too small, the pulling of the images to a node in the OpenShift cluster when deploying them, will fail due to lack of space. Even if the image is able to be run, if the quota is only just larger than the space required for the image, you will not be able to install many packages into the container before running out of space.
OpenShift Online, the public hosted version of OpenShift from Red Hat has a quota of only 3GB for the image and container file system. As a result, only the ``minimal-notebook`` can be started and there is little space remaining to install additional packages. Although OpenShift Online is suitable for demonstrating these templates work, what you can do in that environment will be limited due to the size of the images.
If you want to experiment with using Jupyter Notebooks in an OpenShift environment, you should instead use [Minishift](https://www.openshift.org/minishift/). Minishift provides you the ability to run OpenShift in a virtual machine on your own local computer.
Loading the Templates
---------------------
To load the templates, login to OpenShift from the command line and run:
```
oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/docker-stacks/master/examples/openshift/templates.json
```
This should create the following templates:
```
jupyter-notebook
```
The template can be used from the command line using the ``oc new-app`` command, or from the OpenShift web console by selecting _Add to Project_. This ``README`` is only going to explain deploying from the command line.
Deploying a Notebook
--------------------
To deploy a notebook from the command line using the template, run:
```
oc new-app --template jupyter-notebook
```
The output will be similar to:
```
--> Deploying template "jupyter/jupyter-notebook" to project jupyter
Jupyter Notebook
---------
Template for deploying Jupyter Notebook images.
* With parameters:
* APPLICATION_NAME=notebook
* NOTEBOOK_IMAGE=jupyter/minimal-notebook:latest
* NOTEBOOK_PASSWORD=ded4d7cada554aa48e0db612e1ed1080 # generated
--> Creating resources ...
configmap "notebook-cfg" created
deploymentconfig "notebook" created
route "notebook" created
service "notebook" created
--> Success
Access your application via route 'notebook-jupyter.b9ad.pro-us-east-1.openshiftapps.com'
Run 'oc status' to view your app.
```
When no template parameters are provided, the name of the deployed notebook will be ``notebook``. The image used will be:
```
jupyter/minimal-notebook:latest
```
A password you can use when accessing the notebook will be auto generated and is displayed in the output from running ``oc new-app``.
To see the hostname for accessing the notebook run:
```
oc get routes
```
The output will be similar to:
```
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
notebook notebook-jupyter.abcd.pro-us-east-1.openshiftapps.com notebook 8888-tcp edge/Redirect None
```
A secure route will be used to expose the notebook outside of the OpenShift cluster, so in this case the URL would be:
```
https://notebook-jupyter.abcd.pro-us-east-1.openshiftapps.com/
```
When prompted, enter the password for the notebook.
Passing Template Parameters
---------------------------
To override the name for the notebook, the image used, and the password, you can pass template parameters using the ``--param`` option.
```
oc new-app --template jupyter-notebook \
--param APPLICATION_NAME=mynotebook \
--param NOTEBOOK_IMAGE=jupyter/scipy-notebook:latest \
--param NOTEBOOK_PASSWORD=mypassword
```
You can deploy any of the Jupyter Project docker-stacks images.
* jupyter/base-notebook
* jupyter/r-notebook
* jupyter/minimal-notebook
* jupyter/scipy-notebook
* jupyter/tensorflow-notebook
* jupyter/datascience-notebook
* jupyter/pyspark-notebook
* jupyter/all-spark-notebook
If you don't care what version of the image is used, add the ``:latest`` tag at the end of the image name, otherwise use the hash corresponding to the image version you want to use.
Deleting the Notebook Instance
------------------------------
To delete the notebook instance, run ``oc delete`` using a label selector for the application name.
```
oc delete all,configmap --selector app=mynotebook
```
Adding Persistent Storage
-------------------------
You can upload notebooks and other files using the web interface of the notebook. Any uploaded files or changes you make to them will be lost when the notebook instance is restarted. If you want to save your work, you need to add persistent storage to the notebook. To add persistent storage run:
```
oc set volume dc/mynotebook --add \
--type=pvc --claim-size=1Gi --claim-mode=ReadWriteOnce \
--claim-name mynotebook-data --name data \
--mount-path /home/jovyan/work
```
When you upload any notebooks or data files, place them under the ``work`` directory.
When you have deleted the notebook instance, if using a persistent volume, you will need to delete it in a separate step.
```
oc delete pvc/mynotebook-data
```
Customizing the Configuration
-----------------------------
If you want to set any custom configuration for the notebook, you can edit the config map created by the template.
```
oc edit configmap/mynotebook-cfg
```
The ``data`` field of the config map contains Python code used as the ``jupyter_notebook_config.py`` file.
If you are using a persistent volume, you can also create a configuration file at:
```
/home/jovyan/work/.jupyter/jupyter_notebook_config.py
```
This will be merged at the end of the configuration from the config map.
Because the configuration is Python code, ensure any indenting is correct. Any errors in the configuration file will cause the notebook to fail when starting.
If the error is in the config map, edit it again to fix it and trigged a new deployment if necessary by running:
```
oc rollout latest dc/mynotebook
```
If you make an error in the configuration file stored in the persistent volume, you will need to scale down the notebook so it isn't running.
```
oc scale dc/mynotebook --replicas 0
```
Then run:
```
oc debug dc/mynotebook
```
to run the notebook in debug mode. This will provide you with an interactive terminal session inside a running container, but the notebook will not have been started. Edit the configuration file in the volume to fix any errors and exit the terminal session.
Start up the notebook again.
```
oc scale dc/mynotebook --replicas 1
```
Changing the Notebook Password
------------------------------
The password for the notebook is supplied as a template parameter, or if not supplied will be automatically generated by the template. It will be passed into the container through an environment variable.
If you want to change the password, you can do so by editing the environment variable on the deployment configuration.
```
oc set env dc/mynotebook JUPYTER_NOTEBOOK_PASSWORD=mypassword
```
This will trigger a new deployment so ensure you have downloaded any work if not using a persistent volume.
If using a persistent volume, you could instead setup a password in the file:
```
/home/jovyan/work/.jupyter/jupyter_notebook_config.py
```
as per guidelines in:
* https://jupyter-notebook.readthedocs.io/en/stable/public_server.html
Deploying from a Custom Image
-----------------------------
If you want to deploy a custom variant of the Jupyter Project docker-stacks images, you can replace the image name with that of your own. If the image is not stored on Docker Hub, but some other public image registry, prefix the name of the image with the image registry host details.
If the image is in your OpenShift project, because you imported the image into OpenShift, or used the docker build strategy of OpenShift to build a derived custom image, you can use the name of the image stream for the image name, including any image tag if necessary.
This can be illustrated by first importing an image into the OpenShift project.
```
oc import-image jupyter/datascience-notebook:latest --confirm
```
Then deploy it using the name of the image stream created.
```
oc new-app --template jupyter-notebook \
--param APPLICATION_NAME=mynotebook \
--param NOTEBOOK_IMAGE=datascience-notebook \
--param NOTEBOOK_PASSWORD=mypassword
```
Importing an image into OpenShift before deploying it means that when a notebook is started, the image need only be pulled from the internal OpenShift image registry rather than Docker Hub for each deployment. Because the images are so large, this can speed up deployments when the image hasn't previously been deployed to a node in the OpenShift cluster.
examples/openshift/templates.json 0 → 100644
View file @ f4bf2ba5
{
"kind": "Template",
"apiVersion": "v1",
"metadata": {
"name": "jupyter-notebook",
"annotations": {
"openshift.io/display-name": "Jupyter Notebook",
"description": "Template for deploying Jupyter Notebook images.",
"iconClass": "icon-python",
"tags": "python,jupyter"
}
},
"parameters": [
{
"name": "APPLICATION_NAME",
"value": "notebook",
"required": true
},
{
"name": "NOTEBOOK_IMAGE",
"value": "jupyter/minimal-notebook:latest",
"required": true
},
{
"name": "NOTEBOOK_PASSWORD",
"from": "[a-f0-9]{32}",
"generate": "expression"
}
],
"objects": [
{
"kind": "ConfigMap",
"apiVersion": "v1",
"metadata": {
"name": "${APPLICATION_NAME}-cfg",
"labels": {
"app": "${APPLICATION_NAME}"
}
},
"data": {
"jupyter_notebook_config.py": "import os\n\npassword = os.environ.get('JUPYTER_NOTEBOOK_PASSWORD')\n\nif password:\n import notebook.auth\n c.NotebookApp.password = notebook.auth.passwd(password)\n del password\n del os.environ['JUPYTER_NOTEBOOK_PASSWORD']\n\nimage_config_file = '/home/jovyan/work/.jupyter/jupyter_notebook_config.py'\n\nif os.path.exists(image_config_file):\n with open(image_config_file) as fp:\n exec(compile(fp.read(), image_config_file, 'exec'), globals())\n"
}
},
{
"kind": "DeploymentConfig",
"apiVersion": "v1",
"metadata": {
"name": "${APPLICATION_NAME}",
"labels": {
"app": "${APPLICATION_NAME}"
}
},
"spec": {
"strategy": {
"type": "Recreate"
},
"triggers": [
{
"type": "ConfigChange"
}
],
"replicas": 1,
"selector": {
"app": "${APPLICATION_NAME}",
"deploymentconfig": "${APPLICATION_NAME}"
},
"template": {
"metadata": {
"annotations": {
"alpha.image.policy.openshift.io/resolve-names": "*"
},
"labels": {
"app": "${APPLICATION_NAME}",
"deploymentconfig": "${APPLICATION_NAME}"
}
},
"spec": {
"containers": [
{
"name": "jupyter-notebook",
"image": "${NOTEBOOK_IMAGE}",
"command": [
"start-notebook.sh",
"--config=/home/jovyan/configs/jupyter_notebook_config.py",
"--no-browser",
"--ip=0.0.0.0"
],
"ports": [
{
"containerPort": 8888,
"protocol": "TCP"
}
],
"env": [
{
"name": "JUPYTER_NOTEBOOK_PASSWORD",
"value": "${NOTEBOOK_PASSWORD}"
}
],
"volumeMounts": [
{
"mountPath": "/home/jovyan/configs",
"name": "configs"
}
]
}
],
"securityContext": {
"supplementalGroups": [
100
]
},
"volumes": [
{
"configMap": {
"name": "${APPLICATION_NAME}-cfg"
},
"name": "configs"
}
]
}
}
}
},
{
"kind": "Route",
"apiVersion": "v1",
"metadata": {
"name": "${APPLICATION_NAME}",
"labels": {
"app": "${APPLICATION_NAME}"
}
},
"spec": {
"host": "",
"to": {
"kind": "Service",
"name": "${APPLICATION_NAME}",
"weight": 100
},
"port": {
"targetPort": "8888-tcp"
},
"tls": {
"termination": "edge",
"insecureEdgeTerminationPolicy": "Redirect"
}
}
},
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "${APPLICATION_NAME}",
"labels": {
"app": "${APPLICATION_NAME}"
}
},
"spec": {
"ports": [
{
"name": "8888-tcp",
"protocol": "TCP",
"port": 8888,
"targetPort": 8888
}
],
"selector": {
"app": "${APPLICATION_NAME}",
"deploymentconfig": "${APPLICATION_NAME}"
},
"type": "ClusterIP"
}
}
]
}
minimal-notebook/Dockerfile
View file @ f4bf2ba5
...@@ -32,4 +32,4 @@ RUN apt-get update && apt-get install -yq --no-install-recommends \ ...@@ -32,4 +32,4 @@ RUN apt-get update && apt-get install -yq --no-install-recommends \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
# Switch back to jovyan to avoid accidental container runs as root # Switch back to jovyan to avoid accidental container runs as root
USER $NB_USER USER $NB_UID
pyspark-notebook/Dockerfile
View file @ f4bf2ba5
...@@ -38,4 +38,4 @@ ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip ...@@ -38,4 +38,4 @@ ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip
ENV MESOS_NATIVE_LIBRARY /usr/local/lib/libmesos.so ENV MESOS_NATIVE_LIBRARY /usr/local/lib/libmesos.so
ENV SPARK_OPTS --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info ENV SPARK_OPTS --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info
USER $NB_USER USER $NB_UID
r-notebook/Dockerfile
View file @ f4bf2ba5
...@@ -15,7 +15,7 @@ RUN apt-get update && \ ...@@ -15,7 +15,7 @@ RUN apt-get update && \
gcc && apt-get clean && \ gcc && apt-get clean && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
USER $NB_USER USER $NB_UID
# R packages # R packages
RUN conda install --quiet --yes \ RUN conda install --quiet --yes \
......
scipy-notebook/Dockerfile
View file @ f4bf2ba5
...@@ -12,7 +12,7 @@ RUN apt-get update && \ ...@@ -12,7 +12,7 @@ RUN apt-get update && \
apt-get clean && \ apt-get clean && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
USER $NB_USER USER $NB_UID
# Install Python 3 packages # Install Python 3 packages
# Remove pyqt and qt pulled in for matplotlib since we're only ever going to # Remove pyqt and qt pulled in for matplotlib since we're only ever going to
...@@ -69,4 +69,4 @@ ENV XDG_CACHE_HOME /home/$NB_USER/.cache/ ...@@ -69,4 +69,4 @@ ENV XDG_CACHE_HOME /home/$NB_USER/.cache/
RUN MPLBACKEND=Agg python -c "import matplotlib.pyplot" && \ RUN MPLBACKEND=Agg python -c "import matplotlib.pyplot" && \
fix-permissions /home/$NB_USER fix-permissions /home/$NB_USER
USER $NB_USER USER $NB_UID
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment