X-Git-Url: https://git.arvados.org/arvados.git/blobdiff_plain/1d13d7e2b314a84abe94f00d4c1d787a40f4d7fb..41305b5ac71cc9a306dc654c42c11ffcc4258a47:/doc/user/topics/arv-docker.html.textile.liquid
diff --git a/doc/user/topics/arv-docker.html.textile.liquid b/doc/user/topics/arv-docker.html.textile.liquid
index 46e220f13a..bb1c7dd53e 100644
--- a/doc/user/topics/arv-docker.html.textile.liquid
+++ b/doc/user/topics/arv-docker.html.textile.liquid
@@ -1,187 +1,114 @@
---
layout: default
navsection: userguide
-title: "Customizing the Crunch runtime environment"
+title: "Working with Docker images"
...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
-This page describes how to customize the runtime environment (e.g. the programs, libraries, and other dependencies needed to run a job) that a crunch script will be run in using "Docker.":https://www.docker.com/
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
-This page will demonstrate:
+This page describes how to set up the runtime environment (e.g., the programs, libraries, and other dependencies needed to run a job) that a workflow step will be run in using "Docker.":https://www.docker.com/ Docker is a tool for building and running containers that isolate applications from other applications running on the same node. For detailed information about Docker, see the "Docker User Guide.":https://docs.docker.com/userguide/
-# How to fetch the arvados/jobs Docker image
-# Manually install additional software into the container
-# Create a new custom image
-# Upload that image to Arvados for use by Crunch jobs.
+This page describes:
+# "Create a custom image using a Dockerfile":#create
+# "Uploading an image to Arvados":#upload
+# "Sources of pre-built bioinformatics Docker images":#sources
-{% include 'tutorial_expectations' %}
+{% include 'tutorial_expectations_workstation' %}
-h2. Fetching a starting image
+You also need ensure that "Docker is installed,":https://docs.docker.com/installation/ the Docker daemon is running, and you have permission to access Docker. You can test this by running @docker version@. If you receive a permission denied error, your user account may need to be added to the @docker@ group. If you have root access, you can add yourself to the @docker@ group using @$ sudo addgroup $USER docker@ then log out and log back in again; otherwise consult your local sysadmin.
-First, download the latest image from the Docker registry:
+h2(#create). Create a custom image using a Dockerfile
-
-$ docker pull arvados/jobs
-Pulling repository arvados/jobs
-3132168f2acb: Download complete
-a42b7f2c59b6: Download complete
-e5afdf26a7ae: Download complete
-5cae48636278: Download complete
-7a4f91b70558: Download complete
-a04a275c1fd6: Download complete
-c433ff206a22: Download complete
-b2e539b45f96: Download complete
-073b2581c6be: Download complete
-593915af19dc: Download complete
-32260b35005e: Download complete
-6e5b860c1cde: Download complete
-95f0bfb43d4d: Download complete
-c7fd77eedb96: Download complete
-0d7685aafd00: Download complete
-
$ docker run --interactive --tty --user root arvados/jobs /bin/bash
-root@a0e8299b59aa:/#
+
$ mkdir docker-example-r-base
+$ cd docker-example-r-base
root@a0e8299b59aa:/# apt-get update
-Get:1 http://apt.arvados.org wheezy Release.gpg [490 B]
-Get:2 http://apt.arvados.org wheezy Release [1568 B]
-Get:3 http://apt.arvados.org wheezy/main amd64 Packages [34.6 kB]
-Get:4 http://ftp.us.debian.org wheezy Release.gpg [1655 B]
-Get:5 http://ftp.us.debian.org wheezy-updates Release.gpg [836 B]
-Get:6 http://ftp.us.debian.org wheezy Release [168 kB]
-Ign http://apt.arvados.org wheezy/main Translation-en
-Get:7 http://security.debian.org wheezy/updates Release.gpg [836 B]
-Get:8 http://security.debian.org wheezy/updates Release [102 kB]
-Get:9 http://ftp.us.debian.org wheezy-updates Release [124 kB]
-Get:10 http://ftp.us.debian.org wheezy/main amd64 Packages [5841 kB]
-Get:11 http://security.debian.org wheezy/updates/main amd64 Packages [218 kB]
-Get:12 http://security.debian.org wheezy/updates/main Translation-en [123 kB]
-Hit http://ftp.us.debian.org wheezy/main Translation-en
-Hit http://ftp.us.debian.org wheezy-updates/main amd64 Packages/DiffIndex
-Hit http://ftp.us.debian.org wheezy-updates/main Translation-en/DiffIndex
-Fetched 6617 kB in 5s (1209 kB/s)
-Reading package lists... Done
+
FROM ubuntu:bionic
+RUN apt-get update && apt-get -yq --no-install-recommends install r-base-core
root@a0e8299b59aa:/# apt-get install r-base-core
-Reading package lists... Done
-Building dependency tree
-Reading state information... Done
-The following extra packages will be installed:
- [...]
-libxv1 libxxf86dga1 libxxf86vm1 r-base-core r-base-dev r-base-html r-cran-boot r-cran-class r-cran-cluster r-cran-codetools
- [...]
-Suggested packages:
- [...]
-The following NEW packages will be installed:
- [...]
- libxv1 libxxf86dga1 libxxf86vm1 r-base r-base-core r-base-dev r-base-html r-cran-boot r-cran-class r-cran-cluster
- [...]
-0 upgraded, 107 newly installed, 0 to remove and 9 not upgraded.
-Need to get 88.2 MB of archives.
-After this operation, 219 MB of additional disk space will be used.
-Do you want to continue [Y/n]? y
-[...]
-Get:85 http://ftp.us.debian.org/debian/ wheezy/main r-base-core amd64 2.15.1-4 [20.6 MB]
-Get:86 http://ftp.us.debian.org/debian/ wheezy/main r-base-dev all 2.15.1-4 [3882 B]
-Get:87 http://ftp.us.debian.org/debian/ wheezy/main r-cran-boot all 1.3-5-1 [472 kB]
-[...]
-Fetched 88.2 MB in 2min 17s (642 kB/s)
-Extracting templates from packages: 100%
-Preconfiguring packages ...
-[...]
-Unpacking r-base-core (from .../r-base-core_2.15.1-4_amd64.deb) ...
-Selecting previously unselected package r-base-dev.
-Unpacking r-base-dev (from .../r-base-dev_2.15.1-4_all.deb) ...
-Selecting previously unselected package r-cran-boot.
-Unpacking r-cran-boot (from .../r-cran-boot_1.3-5-1_all.deb) ...
-[...]
-Setting up r-base-core (2.15.1-4) ...
-Setting R_PAPERSIZE_USER default to 'a4'
-
-Creating config file /etc/R/Renviron with new version
-Setting up r-base-dev (2.15.1-4) ...
-Setting up r-cran-boot (1.3-5-1) ...
-[...]
+
docker-example-r-base$ docker build -t docker-example-r-base .
root@a0e8299b59aa:/# R
+
+$ docker run -ti docker-example-r-base
+root@57ec8f8b2663:/# R
-R version 2.15.1 (2012-06-22) -- "Roasted Marshmallows"
-Copyright (C) 2012 The R Foundation for Statistical Computing
-ISBN 3-900051-07-0
+R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
+Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
+
$ arv-keepdocker docker-example-r-base
+2020-06-29 13:48:19 arvados.arv_put[769] INFO: Creating new cache file at /home/peter/.cache/arvados/arv-put/39ddb51ebf6c5fcb3d713b5969466967
+206M / 206M 100.0% 2020-06-29 13:48:21 arvados.arv_put[769] INFO:
-Type 'demo()' for some demos, 'help()' for on-line help, or
-'help.start()' for an HTML browser interface to help.
-Type 'q()' to quit R.
+2020-06-29 13:48:21 arvados.arv_put[769] INFO: Collection saved as 'Docker image docker-example-r-base:latest sha256:edd10'
+zzzzz-4zz18-0tayximqcyb6uf8
->
+$ arv-keepdocker images
+REPOSITORY TAG IMAGE ID COLLECTION CREATED
+docker-example-r-base latest sha256:edd10 zzzzz-4zz18-0tayximqcyb6uf8 Mon Jun 29 17:46:16 2020
+hints: + DockerRequirement: + dockerPull: docker-example-r-base +-h2. Create a new image +h3. Uploading Docker images to a shared project -We're now ready to create a new Docker image. First, quit the container, then use @docker commit@ to create a new image from the stopped container. The container id can be found in the default hostname of the container displayed in the prompt, in this case @a0e8299b59aa@: +Docker images are subject to normal Arvados permissions. If wish to share your Docker image with others you should use @arv-keepdocker@ with the @--project-uuid@ option to add the image to a shared project and ensure that metadata is set correctly.
root@a0e8299b59aa:/# exit
-$ docker commit a0e8299b59aa arvados/jobs-with-r
-33ea6b87792364cb9989a149c36a31e5a9c8cf96694ba05f66545ad7b842522e
-$ docker images
-REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
-arvados/jobs-with-r latest 33ea6b877923 43 seconds ago 1.607 GB
-arvados/jobs latest 3132168f2acb 22 hours ago 1.314 GB
+$ arv-keepdocker docker-example-r-base --project-uuid zzzzz-j7d0g-xxxxxxxxxxxxxxx
$ arv keep docker arvados/jobs-with-r
-1591M / 1591M 100.0%
-Collection saved as 'Docker image arvados/jobs-with-r:latest 33ea6b877923'
-qr1hi-4zz18-3fk2px2ji25nst2
-$ arv keep docker
-REPOSITORY TAG IMAGE ID COLLECTION CREATED
-arvados/jobs-with-r latest 33ea6b877923 qr1hi-4zz18-3fk2px2ji25nst2 Thu Oct 16 13:58:53 2014
-
-