title: "Customizing the Crunch runtime environment"
...
-This page describes how to customize the runtime environment (e.g. the programs, libraries, and other dependencies needed to run a job) that a crunch script will be run in.
+This page describes how to customize the runtime environment (e.g. the programs, libraries, and other dependencies needed to run a job) that a crunch script will be run in using "Docker.":https://www.docker.com/
+
+This page will demonstrate:
+
+# How to fetch the arvados/jobs Docker image
+# Manually install additional software into the container
+# Create a new custom image
+# Upload that image to Arvados for use by Crunch jobs.
+
{% include 'tutorial_expectations' %}
-h1. Docker
+h2. Fetching a starting image
+
+First, download the latest image from the Docker registry:
+
+<notextile>
+<pre><code>$ <span class="userinput">docker pull arvados/jobs</span>
+Pulling repository arvados/jobs
+3132168f2acb: Download complete
+a42b7f2c59b6: Download complete
+e5afdf26a7ae: Download complete
+5cae48636278: Download complete
+7a4f91b70558: Download complete
+a04a275c1fd6: Download complete
+c433ff206a22: Download complete
+b2e539b45f96: Download complete
+073b2581c6be: Download complete
+593915af19dc: Download complete
+32260b35005e: Download complete
+6e5b860c1cde: Download complete
+95f0bfb43d4d: Download complete
+c7fd77eedb96: Download complete
+0d7685aafd00: Download complete
+</code></pre>
+</notextile>
+
+h2. Installing new packages
+
+Next, enter the container using "docker run", the arvados/jobs image, and the program you want to run (in this case the bash shell).
+
+<notextile>
+<pre><code>$ <span class="userinput">docker run --interactive --tty --user root arvados/jobs /bin/bash</span>
+root@a0e8299b59aa:/#
+</code></pre>
+</notextile>
+
+Next, update the package list using @apt-get update@.
+
+<notextile>
+<pre><code>root@a0e8299b59aa:/# <span class="userinput">apt-get update</span>
+Get:1 http://apt.arvados.org wheezy Release.gpg [490 B]
+Get:2 http://apt.arvados.org wheezy Release [1568 B]
+Get:3 http://apt.arvados.org wheezy/main amd64 Packages [34.6 kB]
+Get:4 http://ftp.us.debian.org wheezy Release.gpg [1655 B]
+Get:5 http://ftp.us.debian.org wheezy-updates Release.gpg [836 B]
+Get:6 http://ftp.us.debian.org wheezy Release [168 kB]
+Ign http://apt.arvados.org wheezy/main Translation-en
+Get:7 http://security.debian.org wheezy/updates Release.gpg [836 B]
+Get:8 http://security.debian.org wheezy/updates Release [102 kB]
+Get:9 http://ftp.us.debian.org wheezy-updates Release [124 kB]
+Get:10 http://ftp.us.debian.org wheezy/main amd64 Packages [5841 kB]
+Get:11 http://security.debian.org wheezy/updates/main amd64 Packages [218 kB]
+Get:12 http://security.debian.org wheezy/updates/main Translation-en [123 kB]
+Hit http://ftp.us.debian.org wheezy/main Translation-en
+Hit http://ftp.us.debian.org wheezy-updates/main amd64 Packages/DiffIndex
+Hit http://ftp.us.debian.org wheezy-updates/main Translation-en/DiffIndex
+Fetched 6617 kB in 5s (1209 kB/s)
+Reading package lists... Done
+</code></pre>
+</notextile>
+
+In this example, we will install the "R" statistical language Debian package "r-base-core". Use @apt-get install@:
+
+<notextile>
+<pre><code>root@a0e8299b59aa:/# <span class="userinput">apt-get install r-base-core</span>
+Reading package lists... Done
+Building dependency tree
+Reading state information... Done
+The following extra packages will be installed:
+ [...]
+libxv1 libxxf86dga1 libxxf86vm1 r-base-core r-base-dev r-base-html r-cran-boot r-cran-class r-cran-cluster r-cran-codetools
+ [...]
+Suggested packages:
+ [...]
+The following NEW packages will be installed:
+ [...]
+ libxv1 libxxf86dga1 libxxf86vm1 r-base r-base-core r-base-dev r-base-html r-cran-boot r-cran-class r-cran-cluster
+ [...]
+0 upgraded, 107 newly installed, 0 to remove and 9 not upgraded.
+Need to get 88.2 MB of archives.
+After this operation, 219 MB of additional disk space will be used.
+Do you want to continue [Y/n]? y
+[...]
+Get:85 http://ftp.us.debian.org/debian/ wheezy/main r-base-core amd64 2.15.1-4 [20.6 MB]
+Get:86 http://ftp.us.debian.org/debian/ wheezy/main r-base-dev all 2.15.1-4 [3882 B]
+Get:87 http://ftp.us.debian.org/debian/ wheezy/main r-cran-boot all 1.3-5-1 [472 kB]
+[...]
+Fetched 88.2 MB in 2min 17s (642 kB/s)
+Extracting templates from packages: 100%
+Preconfiguring packages ...
+[...]
+Unpacking r-base-core (from .../r-base-core_2.15.1-4_amd64.deb) ...
+Selecting previously unselected package r-base-dev.
+Unpacking r-base-dev (from .../r-base-dev_2.15.1-4_all.deb) ...
+Selecting previously unselected package r-cran-boot.
+Unpacking r-cran-boot (from .../r-cran-boot_1.3-5-1_all.deb) ...
+[...]
+Setting up r-base-core (2.15.1-4) ...
+Setting R_PAPERSIZE_USER default to 'a4'
+
+Creating config file /etc/R/Renviron with new version
+Setting up r-base-dev (2.15.1-4) ...
+Setting up r-cran-boot (1.3-5-1) ...
+[...]
+</code></pre>
+</notextile>
+
+Now we can verify that "R" is installed:
+
+<notextile>
+<pre><code>root@a0e8299b59aa:/# <span class="userinput">R</span>
+
+R version 2.15.1 (2012-06-22) -- "Roasted Marshmallows"
+Copyright (C) 2012 The R Foundation for Statistical Computing
+ISBN 3-900051-07-0
+Platform: x86_64-pc-linux-gnu (64-bit)
+
+R is free software and comes with ABSOLUTELY NO WARRANTY.
+You are welcome to redistribute it under certain conditions.
+Type 'license()' or 'licence()' for distribution details.
+
+R is a collaborative project with many contributors.
+Type 'contributors()' for more information and
+'citation()' on how to cite R or R packages in publications.
+
+Type 'demo()' for some demos, 'help()' for on-line help, or
+'help.start()' for an HTML browser interface to help.
+Type 'q()' to quit R.
+
+>
+</code></pre>
+</notextile>
+
+h2. Create a new image
+
+We're now ready to create a new Docker image. First, quit the container, then use @docker commit@ to create a new image from the stopped container. The container id can be found in the default hostname of the container displayed in the prompt, in this case @a0e8299b59aa@:
+
+<notextile>
+<pre><code>root@a0e8299b59aa:/# <span class="userinput">exit</span>
+$ <span class="userinput">docker commit a0e8299b59aa arvados/jobs-with-r</span>
+33ea6b87792364cb9989a149c36a31e5a9c8cf96694ba05f66545ad7b842522e
+$ <span class="userinput">docker images</span>
+REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
+arvados/jobs-with-r latest 33ea6b877923 43 seconds ago 1.607 GB
+arvados/jobs latest 3132168f2acb 22 hours ago 1.314 GB
+</code></pre>
+</notextile>
+
+h2. Upload your image
-Arvados uses "Docker":https://www.docker.com/ images to provide a runtime environment for Crunch jobs.
+Finally, we are ready to upload the new Docker image to Arvados. Use @arv keep docker@ with the image repository name to upload the image. Without arguments, @arv keep docker@ will print out the list of Docker images in Arvados that are available to you.
-This page will demonstrate how to build the arvados/jobs Docker image, manually install additional software into the image to create a new custom image, and upload that image to Arvados for use by Crunch jobs.
+<notextile>
+<pre><code>$ <span class="userinput">arv keep docker arvados/jobs-with-r</span>
+1591M / 1591M 100.0%
+Collection saved as 'Docker image arvados/jobs-with-r:latest 33ea6b877923'
+qr1hi-4zz18-3fk2px2ji25nst2
+$ <span class="userinput">arv keep docker</span>
+REPOSITORY TAG IMAGE ID COLLECTION CREATED
+arvados/jobs-with-r latest 33ea6b877923 qr1hi-4zz18-3fk2px2ji25nst2 Thu Oct 16 13:58:53 2014
+</code></pre>
+</notextile>
-Arvados images are based on "Debian.":https://debian.org In order to build
+You are now able to specify the runtime environment for your program using the @docker_image@ field of the @runtime_constaints@ section of your pipeline components:
-cd ~/arvados/docker
-./build.sh jobs-image
+<notextile>
+{% code 'example_docker' as javascript %}
+</notextile>
projects / arvados.git / commitdiff