navsection: userguide
title: "Customizing Crunch environment using Docker"
...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
-This page describes how to customize the runtime environment (e.g. the programs, libraries, and other dependencies needed to run a job) that a crunch script will be run in using "Docker.":https://www.docker.com/ Docker is a tool for building and running containers that isolate applications from other applications running on the same node. For detailed information about Docker, see the "Docker User Guide.":https://docs.docker.com/userguide/
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+This page describes how to customize the runtime environment (e.g., the programs, libraries, and other dependencies needed to run a job) that a crunch script will be run in using "Docker.":https://www.docker.com/ Docker is a tool for building and running containers that isolate applications from other applications running on the same node. For detailed information about Docker, see the "Docker User Guide.":https://docs.docker.com/userguide/
This page will demonstrate how to:
<notextile>
<pre><code>$ <span class="userinput">docker run --interactive --tty --user root arvados/jobs /bin/bash</span>
-root@a0e8299b59aa:/#
+root@fbf1d0f529d5:/#
</code></pre>
</notextile>
Next, update the package list using @apt-get update@.
<notextile>
-<pre><code>root@a0e8299b59aa:/# <span class="userinput">apt-get update</span>
-Get:1 http://apt.arvados.org wheezy Release.gpg [490 B]
-Get:2 http://apt.arvados.org wheezy Release [1568 B]
-Get:3 http://apt.arvados.org wheezy/main amd64 Packages [34.6 kB]
-Get:4 http://ftp.us.debian.org wheezy Release.gpg [1655 B]
-Get:5 http://ftp.us.debian.org wheezy-updates Release.gpg [836 B]
-Get:6 http://ftp.us.debian.org wheezy Release [168 kB]
-Ign http://apt.arvados.org wheezy/main Translation-en
-Get:7 http://security.debian.org wheezy/updates Release.gpg [836 B]
-Get:8 http://security.debian.org wheezy/updates Release [102 kB]
-Get:9 http://ftp.us.debian.org wheezy-updates Release [124 kB]
-Get:10 http://ftp.us.debian.org wheezy/main amd64 Packages [5841 kB]
-Get:11 http://security.debian.org wheezy/updates/main amd64 Packages [218 kB]
-Get:12 http://security.debian.org wheezy/updates/main Translation-en [123 kB]
-Hit http://ftp.us.debian.org wheezy/main Translation-en
-Hit http://ftp.us.debian.org wheezy-updates/main amd64 Packages/DiffIndex
-Hit http://ftp.us.debian.org wheezy-updates/main Translation-en/DiffIndex
-Fetched 6617 kB in 5s (1209 kB/s)
+<pre><code>root@fbf1d0f529d5:/# apt-get update
+Hit http://security.debian.org jessie/updates InRelease
+Ign http://httpredir.debian.org jessie InRelease
+Ign http://apt.arvados.org jessie InRelease
+Hit http://apt.arvados.org jessie Release.gpg
+Get:1 http://security.debian.org jessie/updates/main amd64 Packages [431 kB]
+Hit http://apt.arvados.org jessie Release
+Hit http://httpredir.debian.org jessie-updates InRelease
+Get:2 http://apt.arvados.org jessie/main amd64 Packages [257 kB]
+Get:3 http://httpredir.debian.org jessie-updates/main amd64 Packages [17.6 kB]
+Hit http://httpredir.debian.org jessie Release.gpg
+Hit http://httpredir.debian.org jessie Release
+Get:4 http://httpredir.debian.org jessie/main amd64 Packages [9049 kB]
+Fetched 9755 kB in 2s (3894 kB/s)
Reading package lists... Done
</code></pre>
</notextile>
In this example, we will install the "R" statistical language Debian package "r-base-core". Use @apt-get install@:
<notextile>
-<pre><code>root@a0e8299b59aa:/# <span class="userinput">apt-get install r-base-core</span>
+<pre><code>root@fbf1d0f529d5:/# <span class="userinput">apt-get install r-base-core</span>
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
[...]
-libxv1 libxxf86dga1 libxxf86vm1 r-base-core r-base-dev r-base-html r-cran-boot r-cran-class r-cran-cluster r-cran-codetools
+ libxxf86vm1 make patch r-base-core r-base-dev r-cran-boot r-cran-class
+ r-cran-cluster r-cran-codetools r-cran-foreign r-cran-kernsmooth
+ r-cran-lattice r-cran-mass r-cran-matrix r-cran-mgcv r-cran-nlme r-cran-nnet
+ r-cran-rpart r-cran-spatial r-cran-survival r-doc-html r-recommended
[...]
Suggested packages:
[...]
The following NEW packages will be installed:
[...]
- libxv1 libxxf86dga1 libxxf86vm1 r-base r-base-core r-base-dev r-base-html r-cran-boot r-cran-class r-cran-cluster
+ libxxf86vm1 make patch r-base-core r-base-dev r-cran-boot r-cran-class
+ r-cran-cluster r-cran-codetools r-cran-foreign r-cran-kernsmooth
+ r-cran-lattice r-cran-mass r-cran-matrix r-cran-mgcv r-cran-nlme r-cran-nnet
+ r-cran-rpart r-cran-spatial r-cran-survival r-doc-html r-recommended
[...]
-0 upgraded, 107 newly installed, 0 to remove and 9 not upgraded.
-Need to get 88.2 MB of archives.
-After this operation, 219 MB of additional disk space will be used.
+0 upgraded, 203 newly installed, 0 to remove and 39 not upgraded.
+Need to get 124 MB of archives.
+After this operation, 334 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
[...]
-Get:85 http://ftp.us.debian.org/debian/ wheezy/main r-base-core amd64 2.15.1-4 [20.6 MB]
-Get:86 http://ftp.us.debian.org/debian/ wheezy/main r-base-dev all 2.15.1-4 [3882 B]
-Get:87 http://ftp.us.debian.org/debian/ wheezy/main r-cran-boot all 1.3-5-1 [472 kB]
+Get:130 http://httpredir.debian.org/debian/ jessie/main r-cran-cluster amd64 1.15.3-1 [475 kB]
+Get:131 http://httpredir.debian.org/debian/ jessie/main r-base-dev all 3.1.1-1 [4018 B]
+Get:132 http://httpredir.debian.org/debian/ jessie/main r-cran-boot all 1.3-13-1 [571 kB]
+Get:133 http://httpredir.debian.org/debian/ jessie/main r-cran-codetools all 0.2-9-1 [45.7 kB]
+Get:134 http://httpredir.debian.org/debian/ jessie/main r-cran-rpart amd64 4.1-8-1 [862 kB]
+Get:135 http://httpredir.debian.org/debian/ jessie/main r-cran-foreign amd64 0.8.61-1 [213 kB]
[...]
-Fetched 88.2 MB in 2min 17s (642 kB/s)
-Extracting templates from packages: 100%
-Preconfiguring packages ...
+Fetched 124 MB in 52s (2380 kB/s)
+debconf: delaying package configuration, since apt-utils is not installed
[...]
-Unpacking r-base-core (from .../r-base-core_2.15.1-4_amd64.deb) ...
+Unpacking r-base-core (3.1.1-1+b2) ...
Selecting previously unselected package r-base-dev.
-Unpacking r-base-dev (from .../r-base-dev_2.15.1-4_all.deb) ...
+Preparing to unpack .../r-base-dev_3.1.1-1_all.deb ...
+Unpacking r-base-dev (3.1.1-1) ...
Selecting previously unselected package r-cran-boot.
-Unpacking r-cran-boot (from .../r-cran-boot_1.3-5-1_all.deb) ...
+Preparing to unpack .../r-cran-boot_1.3-13-1_all.deb ...
+Unpacking r-cran-boot (1.3-13-1) ...
+Selecting previously unselected package r-cran-mass.
[...]
-Setting up r-base-core (2.15.1-4) ...
-Setting R_PAPERSIZE_USER default to 'a4'
+Setting up r-base-core (3.1.1-1+b2) ...
Creating config file /etc/R/Renviron with new version
-Setting up r-base-dev (2.15.1-4) ...
-Setting up r-cran-boot (1.3-5-1) ...
+Setting up r-base-dev (3.1.1-1) ...
+Setting up r-cran-boot (1.3-13-1) ...
+Setting up r-cran-mass (7.3-34-1) ...
+Setting up r-cran-class (7.3-11-1) ...
[...]
</code></pre>
</notextile>
Now we can verify that "R" is installed:
<notextile>
-<pre><code>root@a0e8299b59aa:/# <span class="userinput">R</span>
+<pre><code>root@fbf1d0f529d5:/# <span class="userinput">R</span>
-R version 2.15.1 (2012-06-22) -- "Roasted Marshmallows"
-Copyright (C) 2012 The R Foundation for Statistical Computing
-ISBN 3-900051-07-0
+R version 3.1.1 (2014-07-10) -- "Sock it to Me"
+Copyright (C) 2014 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
h2. Create a new image
-We're now ready to create a new Docker image. First, quit the container, then use @docker commit@ to create a new image from the stopped container. The container id can be found in the default hostname of the container displayed in the prompt, in this case @a0e8299b59aa@:
+We're now ready to create a new Docker image. First, quit the container, then use @docker commit@ to create a new image from the stopped container. The container id can be found in the default hostname of the container displayed in the prompt, in this case @fbf1d0f529d5@:
<notextile>
-<pre><code>root@a0e8299b59aa:/# <span class="userinput">exit</span>
-$ <span class="userinput">docker commit a0e8299b59aa arvados/jobs-with-r</span>
-33ea6b87792364cb9989a149c36a31e5a9c8cf96694ba05f66545ad7b842522e
+<pre><code>root@fbf1d0f529d5:/# <span class="userinput">exit</span>
+$ <span class="userinput">docker commit fbf1d0f529d5 arvados/jobs-with-r</span>
+sha256:2818853ff9f9af5d7f77979803baac9c4710790ad2b84c1a754b02728fdff205
$ <span class="userinput">docker images</span>
-REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
-arvados/jobs-with-r latest 33ea6b877923 43 seconds ago 1.607 GB
-arvados/jobs latest 3132168f2acb 22 hours ago 1.314 GB
+$ docker images |head
+REPOSITORY TAG IMAGE ID CREATED SIZE
+arvados/jobs-with-r latest 2818853ff9f9 9 seconds ago 703.1 MB
+arvados/jobs latest 12b9f859d48c 4 days ago 362 MB
</code></pre>
</notextile>
<notextile>
<pre><code>$ <span class="userinput">arv keep docker arvados/jobs-with-r</span>
-1591M / 1591M 100.0%
-Collection saved as 'Docker image arvados/jobs-with-r:latest 33ea6b877923'
-qr1hi-4zz18-3fk2px2ji25nst2
+703M / 703M 100.0%
+Collection saved as 'Docker image arvados/jobs-with-r:latest 2818853ff9f9'
+qr1hi-4zz18-abcdefghijklmno
$ <span class="userinput">arv keep docker</span>
REPOSITORY TAG IMAGE ID COLLECTION CREATED
-arvados/jobs-with-r latest 33ea6b877923 qr1hi-4zz18-3fk2px2ji25nst2 Thu Oct 16 13:58:53 2014
+arvados/jobs-with-r latest 2818853ff9f9 qr1hi-4zz18-abcdefghijklmno Tue Jan 17 20:35:53 2017
</code></pre>
</notextile>
-You are now able to specify the runtime environment for your program using the @docker_image@ field of the @runtime_constaints@ section of your pipeline components:
-
-<notextile>
-{% code 'example_docker' as javascript %}
-</notextile>
+You are now able to specify the runtime environment for your program using @DockerRequirement@ in your workflow:
-* The @docker_image@ field can be one of: the Docker repository name (as shown above), the Docker image hash, or the Arvados collection portable data hash.
+<pre>
+hints:
+ DockerRequirement:
+ dockerPull: arvados/jobs-with-r
+</pre>
h2. Share Docker images
projects / arvados.git / blobdiff