Thank you for contributing to the Jupyter Docker Stacks! We review pull requests of new features (e.g., new packages, new scripts, new flags) to balance the value of the images to the Jupyter community with the cost of maintaining the images over time.
Thank you for contributing to the Jupyter Docker Stacks! We review pull requests of new features
(e.g., new packages, new scripts, new flags) to balance the value of the images to the Jupyter
community with the cost of maintaining the images over time.
## Suggesting a New Feature
## Suggesting a New Feature
Please follow the process below to suggest a new feature for inclusion in one of the core stacks:
Please follow the process below to suggest a new feature for inclusion in one of the core stacks:
1.[Open a GitHub issue](https://github.com/jupyter/docker-stacks/issues) describing the feature you'd like to contribute.
1.[Open a GitHub issue](https://github.com/jupyter/docker-stacks/issues) describing the feature
2. Discuss with the maintainers whether the addition makes sense in [one of the core stacks](../using/selecting.md#Core-Stacks), as a [recipe in the documentation](recipes.md), as a [community stack](stacks.md), or as something else entirely.
you'd like to contribute.
2. Discuss with the maintainers whether the addition makes sense in
[one of the core stacks](../using/selecting.md#Core-Stacks), as a
[recipe in the documentation](recipes.md), as a [community stack](stacks.md), or as something
else entirely.
## Selection Criteria
## Selection Criteria
Roughly speaking, we evaluate new features based on the following criteria:
Roughly speaking, we evaluate new features based on the following criteria:
***Usefulness to Jupyter users**: Is the feature generally applicable across domains? Does it work with Jupyter Notebook, JupyterLab, JupyterHub, etc.?
-**Usefulness to Jupyter users**: Is the feature generally applicable across domains? Does it work
***Fit with the image purpose**: Does the feature match the theme of the stack in which it will be added? Would it fit better in a new, community stack?
with Jupyter Notebook, JupyterLab, JupyterHub, etc.?
***Complexity of build / runtime configuration**: How many lines of code does the feature require in one of the Dockerfiles or startup scripts? Does it require new scripts entirely? Do users need to adjust how they use the images?
-**Fit with the image purpose**: Does the feature match the theme of the stack in which it will be
***Impact on image metrics**: How many bytes does the feature and its dependencies add to the image(s)? How many minutes do they add to the build time?
added? Would it fit better in a new, community stack?
***Ability to support the addition**: Can existing maintainers answer user questions and address future build issues? Are the contributors interested in helping with long-term maintenance? Can we write tests to ensure the feature continues to work over time?
-**Complexity of build / runtime configuration**: How many lines of code does the feature require
in one of the Dockerfiles or startup scripts? Does it require new scripts entirely? Do users need
to adjust how they use the images?
-**Impact on image metrics**: How many bytes does the feature and its dependencies add to the
image(s)? How many minutes do they add to the build time?
-**Ability to support the addition**: Can existing maintainers answer user questions and address
future build issues? Are the contributors interested in helping with long-term maintenance? Can we
write tests to ensure the feature continues to work over time?
## Submitting a Pull Request
## Submitting a Pull Request
If there's agreement that the feature belongs in one or more of the core stacks:
If there's agreement that the feature belongs in one or more of the core stacks:
1. Implement the feature in a local clone of the `jupyter/docker-stacks` project.
1. Implement the feature in a local clone of the `jupyter/docker-stacks` project.
2. Please build the image locally before submitting a pull request. Building the image locally shortens the debugging cycle by taking some load off [Travis CI](http://travis-ci.org/), which graciously provides free build services for open source projects like this one. If you use `make`, call:
2. Please build the image locally before submitting a pull request. Building the image locally
shortens the debugging cycle by taking some load off GitHub Actions, which graciously provide
free build services for open source projects like this one. If you use `make`, call:
```bash
```bash
make build/somestack-notebook
make build/somestack-notebook
```
```
3.[Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)(PR) with your changes.
3.[Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)
4. Watch for Travis to report a build success or failure for your PR on GitHub.
(PR) with your changes.
4. Watch for GitHub to report a build success or failure for your PR on GitHub.
5. Discuss changes with the maintainers and address any build issues.
5. Discuss changes with the maintainers and address any build issues.
We actively seek pull requests which update packages already included in the project Dockerfiles. This is a great way for first-time contributors to participate in developing the Jupyter Docker Stacks.
We actively seek pull requests which update packages already included in the project Dockerfiles.
This is a great way for first-time contributors to participate in developing the Jupyter Docker
Stacks.
Please follow the process below to update a package version:
Please follow the process below to update a package version:
1. Locate the Dockerfile containing the library you wish to update (e.g., [base-notebook/Dockerfile](https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile), [scipy-notebook/Dockerfile](https://github.com/jupyter/docker-stacks/blob/master/scipy-notebook/Dockerfile))
1. Locate the Dockerfile containing the library you wish to update (e.g.,
2. Adjust the version number for the package. We prefer to pin the major and minor version number of packages so as to minimize rebuild side-effects when users submit pull requests (PRs). For example, you'll find the Jupyter Notebook package, `notebook`, installed using conda with `notebook=5.4.*`.
3. Please build the image locally before submitting a pull request. Building the image locally shortens the debugging cycle by taking some load off [Travis CI](http://travis-ci.org/), which graciously provides free build services for open source projects like this one. If you use `make`, call:
2. Adjust the version number for the package. We prefer to pin the major and minor version number of
packages so as to minimize rebuild side-effects when users submit pull requests (PRs). For
example, you'll find the Jupyter Notebook package, `notebook`, installed using conda with
`notebook=5.4.*`.
3. Please build the image locally before submitting a pull request. Building the image locally
shortens the debugging cycle by taking some load off GitHub Actions, which graciously provide
free build services for open source projects like this one. If you use `make`, call:
```bash
```bash
make build/somestack-notebook
make build/somestack-notebook
```
```
4.[Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)(PR) with your changes.
4.[Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)
5. Watch for Travis to report a build success or failure for your PR on GitHub.
(PR) with your changes.
6. Discuss changes with the maintainers and address any build issues. Version conflicts are the most common problem. You may need to upgrade additional packages to fix build failures.
5. Watch for GitHub to report a build success or failure for your PR on GitHub.
6. Discuss changes with the maintainers and address any build issues. Version conflicts are the most
common problem. You may need to upgrade additional packages to fix build failures.
## Notes
## Notes
In order to help identifying packages that can be updated you can use the following helper tool.
In order to help identifying packages that can be updated you can use the following helper tool. It
It will list all the packages installed in the `Dockerfile` that can be updated -- dependencies are filtered to focus only on requested packages.
will list all the packages installed in the `Dockerfile` that can be updated -- dependencies are
We love to see the community create and share new Jupyter Docker images. We've put together a [cookiecutter project](https://github.com/jupyter/cookiecutter-docker-stacks) and the documentation below to help you get started defining, building, and sharing your Jupyter environments in Docker. Following these steps will:
We love to see the community create and share new Jupyter Docker images. We've put together a
[cookiecutter project](https://github.com/jupyter/cookiecutter-docker-stacks) and the documentation
1. Setup a project on GitHub containing a Dockerfile based on either the `jupyter/base-notebook` or `jupyter/minimal-notebook` image.
below to help you get started defining, building, and sharing your Jupyter environments in Docker.
2. Configure Travis CI to build and test your image when users submit pull requests to your repository.
Following these steps will:
1. Setup a project on GitHub containing a Dockerfile based on either the `jupyter/base-notebook` or
`jupyter/minimal-notebook` image.
2. Configure GitHub Actions to build and test your image when users submit pull requests to your
repository.
3. Configure Docker Cloud to build and host your images for others to use.
3. Configure Docker Cloud to build and host your images for others to use.
4. Update the [list of community stacks](../using/selecting.html#community-stacks) in this documentation to include your image.
4. Update the [list of community stacks](../using/selecting.html#community-stacks) in this
documentation to include your image.
This approach mirrors how we build and share the core stack images. Feel free to follow it or pave your own path using alternative services and build tools.
This approach mirrors how we build and share the core stack images. Feel free to follow it or pave
your own path using alternative services and build tools.
## Creating a Project
## Creating a Project
...
@@ -17,29 +24,31 @@ First, install [cookiecutter](https://github.com/audreyr/cookiecutter) using pip
...
@@ -17,29 +24,31 @@ First, install [cookiecutter](https://github.com/audreyr/cookiecutter) using pip
pip install cookiecutter # or conda install cookiecutter
pip install cookiecutter # or conda install cookiecutter
```
```
Run the cookiecutter command pointing to the [jupyter/cookiecutter-docker-stacks](https://github.com/jupyter/cookiecutter-docker-stacks) project on GitHub.
Enter a name for your new stack image. This will serve as both the git repository
Enter a name for your new stack image. This will serve as both the git repository name and the part
name and the part of the Docker image name after the slash.
of the Docker image name after the slash.
```
```
stack_name [my-jupyter-stack]:
stack_name [my-jupyter-stack]:
```
```
Enter the user or organization name under which this stack will reside on
Enter the user or organization name under which this stack will reside on Docker Cloud / Hub. You
Docker Cloud / Hub. You must have access to manage this Docker Cloud org in
must have access to manage this Docker Cloud org in order to push images here and setup automated
order to push images here and setup automated builds.
builds.
```
```
stack_org [my-project]:
stack_org [my-project]:
```
```
Select an image from the jupyter/docker-stacks project that will serve as the
Select an image from the jupyter/docker-stacks project that will serve as the base for your new
base for your new image.
image.
```
```
stack_base_image [jupyter/base-notebook]:
stack_base_image [jupyter/base-notebook]:
...
@@ -65,47 +74,66 @@ git push -u origin master
...
@@ -65,47 +74,66 @@ git push -u origin master
## Configuring Travis
## Configuring Travis
Next, link your GitHub project to Travis CI to build your Docker image whenever you or someone else submits a pull request.
> NOTE: This section and the cookiecutter template should be updated to describe using GitHub
> Actions now that jupyter/docker-stacks uses that service.
Next, link your GitHub project to Travis CI to build your Docker image whenever you or someone else
submits a pull request.
1. Visit [https://docs.travis-ci.com/user/getting-started/#To-get-started-with-Travis-CI](https://docs.travis-ci.com/user/getting-started/#To-get-started-with-Travis-CI) and follow the instructions to add the Travis CI application to your GitHub account.
Now, configure Docker Cloud to build your stack image and push it to Docker Hub repository whenever you merge a GitHub pull request to the master branch of your project.
Now, configure Docker Cloud to build your stack image and push it to Docker Hub repository whenever
you merge a GitHub pull request to the master branch of your project.
1. Visit [https://cloud.docker.com/](https://cloud.docker.com/) and login.
1. Visit [https://cloud.docker.com/](https://cloud.docker.com/) and login.
2. Select the account or organization matching the one you entered when prompted with `stack_org` by the cookiecutter.
2. Select the account or organization matching the one you entered when prompted with `stack_org` by
the cookiecutter. 
3. Scroll to the bottom of the page and click **Create repository**.
3. Scroll to the bottom of the page and click **Create repository**.
4. Enter the name of the image matching the one you entered when prompted with `stack_name` by the cookiecutter.
4. Enter the name of the image matching the one you entered when prompted with `stack_name` by the
cookiecutter. 
5. Enter a description for your image.
5. Enter a description for your image.
6. Click **GitHub** under the **Build Settings** and follow the prompts to connect your account if it is not already connected.
6. Click **GitHub** under the **Build Settings** and follow the prompts to connect your account if
7. Select the GitHub organization and repository containing your image definition from the dropdowns.
it is not already connected.
7. Select the GitHub organization and repository containing your image definition from the
dropdowns.
8. Click the **Create and Build** button.
8. Click the **Create and Build** button.
## Defining Your Image
## Defining Your Image
Make edits the Dockerfile in your project to add third-party libraries and configure Jupyter applications. Refer to the Dockerfiles for the core stacks (e.g., [jupyter/datascience-notebook](https://github.com/jupyter/docker-stacks/blob/master/datascience-notebook/Dockerfile)) to get a feel for what's possible and best practices.
Make edits the Dockerfile in your project to add third-party libraries and configure Jupyter
applications. Refer to the Dockerfiles for the core stacks (e.g.,
to get a feel for what's possible and best practices.
[Submit pull requests](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request) to your project repository on GitHub. Ensure your image builds properly on Travis before merging to master. Refer to Docker Cloud for builds of your master branch that you can `docker pull`.
2. Open the `docs/using/selecting.md` source file and locate the **Community Stacks** section.
2. Open the `docs/using/selecting.md` source file and locate the **Community Stacks** section.
3. Add a bullet with a link to your project and a short description of what your Docker image contains.
3. Add a bullet with a link to your project and a short description of what your Docker image
4.[Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)(PR) with your changes. Maintainers will respond and work with you to address any formatting or content issues.
contains.
\ No newline at end of file
4.[Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)
(PR) with your changes. Maintainers will respond and work with you to address any formatting or
We greatly appreciate pull requests that extend the automated tests that vet the basic functionality of the Docker images.
We greatly appreciate pull requests that extend the automated tests that vet the basic functionality
of the Docker images.
## How the Tests Work
## How the Tests Work
Travis executes `make build-test-all` against pull requests submitted to the `jupyter/docker-stacks` repository. This `make` command builds every docker image. After building each image, the `make` command executes `pytest` to run both image-specific tests like those in [base-notebook/test/](https://github.com/jupyter/docker-stacks/tree/master/base-notebook/test) and common tests defined in [test/](https://github.com/jupyter/docker-stacks/tree/master/test). Both kinds of tests make use of global [pytest fixtures](https://docs.pytest.org/en/latest/fixture.html) defined in the [conftest.py](https://github.com/jupyter/docker-stacks/blob/master/conftest.py) file at the root of the projects.
GitHub executes `make build-test-all` against pull requests submitted to the `jupyter/docker-stacks`
repository. This `make` command builds every docker image. After building each image, the `make`
command executes `pytest` to run both image-specific tests like those in
[base-notebook/test/](https://github.com/jupyter/docker-stacks/tree/master/base-notebook/test) and
common tests defined in [test/](https://github.com/jupyter/docker-stacks/tree/master/test). Both
kinds of tests make use of global [pytest fixtures](https://docs.pytest.org/en/latest/fixture.html)
defined in the [conftest.py](https://github.com/jupyter/docker-stacks/blob/master/conftest.py) file
at the root of the projects.
## Contributing New Tests
## Contributing New Tests
Please follow the process below to add new tests:
Please follow the process below to add new tests:
1. If the test should run against every image built, add your test code to one of the modules in [test/](https://github.com/jupyter/docker-stacks/tree/master/test) or create a new module.
1. If the test should run against every image built, add your test code to one of the modules in
2. If your test should run against a single image, add your test code to one of the modules in `some-notebook/test/` or create a new module.
[test/](https://github.com/jupyter/docker-stacks/tree/master/test) or create a new module.
2. If your test should run against a single image, add your test code to one of the modules in
`some-notebook/test/` or create a new module.
3. Build one or more images you intend to test and run the tests locally. If you use `make`, call:
3. Build one or more images you intend to test and run the tests locally. If you use `make`, call:
```bash
```bash
make build/somestack-notebook
make build/somestack-notebook
make test/somestack-notebook
make test/somestack-notebook
```
```
4.[Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)(PR) with your changes.
4.[Submit a pull request](https://github.com/PointCloudLibrary/pcl/wiki/A-step-by-step-guide-on-preparing-and-submitting-a-pull-request)
5. Watch for Travis to report a build success or failure for your PR on GitHub.
(PR) with your changes.
6. Discuss changes with the maintainers and address any issues running the tests on Travis.
5. Watch for GitHub to report a build success or failure for your PR on GitHub.
6. Discuss changes with the maintainers and address any issues running the tests on GitHub.
To build new images on Docker Cloud and publish them to the Docker Hub registry, do the following:
To build new images on Docker Cloud and publish them to the Docker Hub registry, do the following:
1. Make sure Travis is green for a PR.
1. Make sure GitHub Actions status checks pas for the PR.
2. Merge the PR.
2. Merge the PR.
3. Monitor the Docker Cloud build status for each of the stacks, starting with [jupyter/base-notebook](https://cloud.docker.com/app/jupyter/repository/docker/jupyter/base-notebook/general) and ending with [jupyter/all-spark-notebook](https://cloud.docker.com/app/jupyter/repository/docker/jupyter/all-spark-notebook/general). See the [stack hierarchy diagram](../using/selecting.html#image-relationships) for the current, complete build order.
3. Monitor the Docker Cloud build status for each of the stacks, starting with
4. Manually click the retry button next to any build that fails to resume that build and any dependent builds.
5. Try to avoid merging another PR to master until all outstanding builds complete. There's no way at present to propagate the git SHA to build through the Docker Cloud build trigger API. Every build trigger works off of master HEAD.
See the [stack hierarchy diagram](../using/selecting.html#image-relationships) for the current,
complete build order.
4. Manually click the retry button next to any build that fails to resume that build and any
dependent builds.
5. Try to avoid merging another PR to master until all outstanding builds complete. There's no way
at present to propagate the git SHA to build through the Docker Cloud build trigger API. Every
build trigger works off of master HEAD.
## Updating the Ubuntu Base Image
## Updating the Ubuntu Base Image
When there's a security fix in the Ubuntu base image or after some time passes, it's a good idea to update the pinned SHA in the [jupyter/base-notebook Dockerfile](https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile). Submit it as a regular PR and go through the build process. Expect the build to take a while to complete: every image layer will rebuild.
When there's a security fix in the Ubuntu base image or after some time passes, it's a good idea to
Submit it as a regular PR and go through the build process. Expect the build to take a while to
complete: every image layer will rebuild.
## Adding a New Core Image to Docker Cloud
## Adding a New Core Image to Docker Cloud
When there's a new stack definition, do the following before merging the PR with the new stack:
When there's a new stack definition, do the following before merging the PR with the new stack:
1. Ensure the PR includes an update to the stack overview diagram [in the documentation](https://github.com/jupyter/docker-stacks/blob/master/docs/using/selecting.md#image-relationships). The image links to the [blockdiag source](http://interactive.blockdiag.com/) used to create it.
1. Ensure the PR includes an update to the stack overview diagram
2. Ensure the PR updates the Makefile which is used to build the stacks in order on Travis CI.
[in the documentation](https://github.com/jupyter/docker-stacks/blob/master/docs/using/selecting.md#image-relationships).
3. Create a new repository in the `jupyter` org on Docker Cloud named after the stack folder in the git repo.
The image links to the [blockdiag source](http://interactive.blockdiag.com/) used to create it.
2. Ensure the PR updates the Makefile which is used to build the stacks in order on GitHub Actions.
3. Create a new repository in the `jupyter` org on Docker Cloud named after the stack folder in the
git repo.
4. Grant the `stacks` team permission to write to the repo.
4. Grant the `stacks` team permission to write to the repo.
5. Click *Builds* and then *Configure Automated Builds* for the repository.
5. Click _Builds_ and then _Configure Automated Builds_ for the repository.
6. Select `jupyter/docker-stacks` as the source repository.
6. Select `jupyter/docker-stacks` as the source repository.
7. Choose *Build on Docker Cloud's infrastructure using a Small node* unless you have reason to believe a bigger host is required.
7. Choose _Build on Docker Cloud's infrastructure using a Small node_ unless you have reason to
8. Update the *Build Context* in the default build rule to be `/<name-of-the-stack>`.
believe a bigger host is required.
9. Toggle *Autobuild* to disabled unless the stack is a new root stack (e.g., like `jupyter/base-notebook`).
8. Update the _Build Context_ in the default build rule to be `/<name-of-the-stack>`.
9. Toggle _Autobuild_ to disabled unless the stack is a new root stack (e.g., like
`jupyter/base-notebook`).
10. If the new stack depends on the build of another stack in the hierarchy:
10. If the new stack depends on the build of another stack in the hierarchy:
1. Hit *Save* and then click *Configure Automated Builds*.
1. Hit _Save_ and then click _Configure Automated Builds_.
2. At the very bottom, add a build trigger named *Stack hierarchy trigger*.
2. At the very bottom, add a build trigger named _Stack hierarchy trigger_.
3. Copy the build trigger URL.
3. Copy the build trigger URL.
4. Visit the parent repository *Builds* page and click *Configure Automated Builds*.
4. Visit the parent repository _Builds_ page and click _Configure Automated Builds_.
5. Add the URL you copied to the *NEXT_BUILD_TRIGGERS* environment variable comma separated list of URLs, creating that environment variable if it does not already exist.
5. Add the URL you copied to the _NEXT_BUILD_TRIGGERS_ environment variable comma separated list
6. Hit *Save*.
of URLs, creating that environment variable if it does not already exist.
6. Hit _Save_.
11. If the new stack should trigger other dependent builds:
11. If the new stack should trigger other dependent builds:
1. Add an environment variable named *NEXT_BUILD_TRIGGERS*.
1. Add an environment variable named _NEXT_BUILD_TRIGGERS_.
2. Copy the build trigger URLs from the dependent builds into the *NEXT_BUILD_TRIGGERS* comma separated list of URLs.
2. Copy the build trigger URLs from the dependent builds into the _NEXT_BUILD_TRIGGERS_ comma
3. Hit *Save*.
separated list of URLs.
12. Adjust other *NEXT_BUILD_TRIGGERS* values as needed so that the build order matches that in the stack hierarchy diagram.
3. Hit _Save_.
12. Adjust other _NEXT_BUILD_TRIGGERS_ values as needed so that the build order matches that in the
stack hierarchy diagram.
## Adding a New Maintainer Account
## Adding a New Maintainer Account
...
@@ -61,22 +81,22 @@ If automated builds on Docker Cloud have got you down, do the following to push
...
@@ -61,22 +81,22 @@ If automated builds on Docker Cloud have got you down, do the following to push
