---
layout: default
navsection: installguide
-title: Installing with Docker
+title: Build and install Docker images
...
-{% include 'alert_docker' %}
+This method is intended for evaluation and development on a local workstation. It is not suitable for production use in a cluster deployment.
-h2. Prerequisites:
+h2. Prerequisites
# A GNU/Linux (virtual) machine
-# A working Docker installation
+# A working Docker installation (see "Installing Docker":https://docs.docker.com/installation/)
+# A working Go installation (see "Install the Go tools":https://golang.org/doc/install)
+# A working Ruby installation, with the Bundler gem installed
+
+h3. Install Ruby and Bundler
+
+{% include 'install_ruby_and_bundler' %}
h2. Download the source tree
h2. Building the Arvados Docker containers
-First of all, a suitable @config.yml@ file is needed.
+First of all, a suitable @config.yml@ file is needed. The easiest way to generate one based on the provided @config.yml.example@ file is to run the @build.sh@ script. If no @config.yml@ file exists, it will will prompt for a few inputs, write the @config.yml@ file, and then proceed to build all the Docker containers. If @config.yml@ exists, invoking @build.sh@ will simply build all Docker containers or update those that need to be updated.
+
+Run @build.sh@ without arguments to generate @config.yml@ and build all Docker containers (this will take a while!):
<notextile>
-<pre><code>~$ <span class="userinput">cd arvados/docker</span>
-~$ <span class="userinput">cp config.yml.example config.yml</span>
+<pre><code>
+~$ <span class="userinput">./build.sh</span>
+Generating config.yml.
+Arvados needs to know the email address of the administrative user,
+so that when that user logs in they are automatically made an admin.
+This should be an email address associated with a Google account.
+
+Enter your Google ID email address here:
+
+...
+
+Step 7 : ADD generated/setup.sh /usr/local/bin/setup.sh
+ ---> d7c0e7fdf7ab
+Removing intermediate container f3d81180795d
+Step 8 : CMD ["/usr/bin/supervisord", "-n"]
+ ---> Running in 84c64cb9f0d5
+ ---> d6cbb5002604
+Removing intermediate container 84c64cb9f0d5
+Successfully built d6cbb5002604
+date >shell-image
</code></pre></notextile>
-Edit the @config.yml@ file and fill in values for at a minimum these
-parameters:
+If all goes well, you should now have a number of Docker images built (the image id will be different):
-<pre>
-PUBLIC_KEY_PATH
-API_HOSTNAME
-API_AUTO_ADMIN_USER
-</pre>
+<notextile>
+<pre><code>~$ <span class="userinput">docker.io images</span>
+REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
+arvados/shell latest d6cbb5002604 10 minutes ago 1.613 GB
+arvados/sso latest 377f1fa0108e 11 minutes ago 1.807 GB
+arvados/keep latest ade0e0d2dd00 12 minutes ago 210.8 MB
+arvados/keepproxy latest b39ef0baba02 12 minutes ago 241.6 MB
+arvados/workbench latest b0e4fb6da385 12 minutes ago 2.218 GB
+arvados/doc latest 4b64daec9454 12 minutes ago 1.524 GB
+arvados/compute latest 7f1f5f7faf54 13 minutes ago 1.862 GB
+arvados/slurm latest f5bfd1008e6b 17 minutes ago 1.573 GB
+arvados/api latest 6b93c5f5fc42 17 minutes ago 2.274 GB
+arvados/passenger latest add2d11fdf24 18 minutes ago 1.738 GB
+arvados/base latest 81eaadd0c6f5 22 minutes ago 1.463 GB
+arvados/debian 7.6 f339ce275c01 6 days ago 116.8 MB
+arvados/debian latest f339ce275c01 6 days ago 116.8 MB
+arvados/debian wheezy f339ce275c01 6 days ago 116.8 MB
+crosbymichael/skydock latest e985023521f6 3 months ago 510.7 MB
+crosbymichael/skydns next 79c99a4608ed 3 months ago 525 MB
+crosbymichael/skydns latest 1923ce648d4c 5 months ago 137.5 MB
+</code></pre></notextile>
-Then build the docker containers (this will take a while):
+h2. Updating the Arvados Docker containers
+
+@build.sh@ takes a few optional arguments:
<notextile>
<pre><code>
-~$ <span class="userinput">./build.sh</span>
-...
- ---> 05f0ae429530
-Step 9 : ADD apache2_foreground.sh /etc/apache2/foreground.sh
- ---> 7292b241305a
-Step 10 : CMD ["/etc/apache2/foreground.sh"]
- ---> Running in 82d59061ead8
- ---> 72cee36a9281
-Successfully built 72cee36a9281
-Removing intermediate container 2bc8c98c83c7
-Removing intermediate container 9457483a59cf
-Removing intermediate container 7cc5723df67c
-Removing intermediate container 5cb2cede73de
-Removing intermediate container 0acc147a7f6d
-Removing intermediate container 82d59061ead8
-Removing intermediate container 9c022a467396
-Removing intermediate container 16044441463f
-Removing intermediate container cffbbddd82d1
-date >sso-image
+~$ <span class="userinput"> ./build.sh --help</span>
+
+usage: ./build.sh [options]
+
+Calling ./build.sh without arguments will build all Arvados docker images
+
+./build.sh options:
+ -h, --help Print this help text
+ clean Clear all build information
+ realclean clean and remove all Arvados Docker images except arvados/debian
+ deepclean realclean and remove arvados/debian, crosbymichael/skydns and
+ crosbymichael/skydns Docker images
</code></pre></notextile>
-If all goes well, you should now have a number of docker images built:
+If there has been an update to the Arvados Docker building code, it is safest to rebuild the Arvados Docker images from scratch. All build information can be cleared with the 'clean' option to build.sh:
<notextile>
-<pre><code>~$ <span class="userinput">docker.io images</span>
-REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
-arvados/sso latest 72cee36a9281 11 seconds ago 1.727 GB
-arvados/keep latest c3842f856bcb 56 seconds ago 210.6 MB
-arvados/workbench latest b91aa980597c About a minute ago 2.07 GB
-arvados/doc latest 050e9e6b8213 About a minute ago 1.442 GB
-arvados/api latest 79843d0a8997 About a minute ago 2.112 GB
-arvados/passenger latest 2342a550da7f 2 minutes ago 1.658 GB
-arvados/base latest 68caefd8ea5b 5 minutes ago 1.383 GB
-arvados/debian 7.5 6e32119ffcd0 8 minutes ago 116.8 MB
-arvados/debian latest 6e32119ffcd0 8 minutes ago 116.8 MB
-arvados/debian wheezy 6e32119ffcd0 8 minutes ago 116.8 MB
-</code></pre></notextile>
+<pre><code>~$ <span class="userinput">./build.sh clean</span></code></pre>
+</notextile>
+
+You can also use 'realclean', which does what 'clean' does and in addition removes all Arvados Docker containers and images from your system, with the exception of the arvados/debian image.
+
+<notextile>
+<pre><code>~$ <span class="userinput">./build.sh realclean</span></code></pre>
+</notextile>
+
+Finally, the 'deepclean' option does what 'realclean' does, and also removes the arvados/debian, crosbymichael/skydns and crosbymichael/skydock images.
+
+<notextile>
+<pre><code>~$ <span class="userinput">./build.sh deepclean</span></code></pre>
+</notextile>
h2. Running the Arvados Docker containers
<pre><code>
~$ <span class="userinput">./arvdock -h</span>
-usage: ./arvdock (start|stop|restart|test) [options]
+usage: ./arvdock (start|stop|restart|reset|test) [options]
+
+start run new or restart stopped arvados containers
+stop stop arvados containers
+restart stop and then start arvados containers
+reset stop and delete containers WARNING: this will delete the data inside Arvados!
+test run tests
./arvdock start/stop/restart options:
- -d [port], --doc[=port] Documentation server (default port 9898)
- -w [port], --workbench[=port] Workbench server (default port 9899)
- -s [port], --sso[=port] SSO server (default port 9901)
- -a [port], --api[=port] API server (default port 9900)
- -k, --keep Keep servers
- --ssh Enable SSH access to server containers
- -h, --help Display this help and exit
+ -d[port], --doc[=port] Documentation server (default port 9898)
+ -w[port], --workbench[=port] Workbench server (default port 9899)
+ -s[port], --sso[=port] SSO server (default port 9901)
+ -a[port], --api[=port] API server (default port 9900)
+ -c, --compute Compute nodes (starts 2)
+ -v, --vm Shell server
+ -n, --nameserver Nameserver
+ -k, --keep Keep servers
+ -p, --keepproxy Keepproxy server
+ -h, --help Display this help and exit
If no options are given, the action is applied to all servers.
</pre>
</notextile>
-The @--ssh@ option can be useful to debug issues with the Docker containers; it allows you to ssh into the running containers as the @root@ user, provided you have access to the private key that matches the public key specified in @config.yml@'s PUBLIC_KEY_PATH variable.
+To debug issues with the Docker containers use @docker exec@, for example:
+
+<notextile>
+<pre><code>
+~$ <span class="userinput">docker exec -t -i api_server /bin/bash</span>
+</code></pre></notextile>
-Start the docker containers:
+To start the docker containers:
<notextile>
<pre><code>
~$ <span class="userinput">./arvdock start</span>
-sso_server
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 9901:443 --name sso_server arvados/sso
-api_server
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 9900:443 --name api_server --link sso_server:sso arvados/api
-keep_server_0
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 25107:25107 --name keep_server_0 -v /tmp/tmp.aCSx8Pq6Wb:/dev/keep-0 --link api_server:api arvados/keep
-keep_server_1
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 25108:25107 --name keep_server_1 -v /tmp/tmp.m4OQ9WB73G:/dev/keep-0 --link api_server:api arvados/keep
-doc_server
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 9898:80 --name doc_server arvados/doc
+start_api=false
+start_compute=false
+start_doc=false
+start_keep=false
+start_keepproxy=false
+start_nameserver=false
+start_sso=false
+start_vm=false
+start_workbench=false
+Starting crosbymichael/skydns container...
+skydns
+/usr/bin/docker.io run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain arvados
+89922ec786cbd2098ed6bae205468a675657cdb2db0e0bfdfe76d1e43cb2fe35
+Starting crosbymichael/skydock container...
+skydock
+/usr/bin/docker.io run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain arvados -name skydns
+de6a35bb20fb9063fb97218ba2554974546eed969d43b1f5aa31a1ac9576e802
+Starting container: api_server
+ /usr/bin/docker.io start api_server
+Started container: api_server
+Starting container: compute0
+ /usr/bin/docker.io start compute0
+Started container: compute0
+Starting container: compute1
+ /usr/bin/docker.io start compute1
+Started container: compute1
+Starting container: keep_server_0
+ /usr/bin/docker.io start keep_server_0
+Started container: keep_server_0
+Starting container: keep_server_1
+ /usr/bin/docker.io start keep_server_1
+Started container: keep_server_1
+Starting container: keepproxy_server
+ /usr/bin/docker.io start keepproxy_server
+Started container: keepproxy_server
+Starting container: doc_server
+ /usr/bin/docker.io start doc_server
+Started container: doc_server
+
+******************************************************************
+You can access the Arvados documentation at http://doc.dev.arvados
+******************************************************************
+
+Starting container: shell
+ /usr/bin/docker.io start shell
+Started container: shell
+Starting container: workbench_server
+ /usr/bin/docker.io start workbench_server
+Started container: workbench_server
+
+********************************************************************
+You can access the Arvados workbench at http://workbench.dev.arvados
+********************************************************************
*****************************************************************
-You can access the Arvados documentation at http://localhost:9898
-*****************************************************************
-
-workbench_server
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 9899:80 --name workbench_server --link api_server:api arvados/workbench
+To access Arvados you must add the Arvados nameserver to the top
+of your DNS configuration in /etc/resolv.conf:
+nameserver 172.17.42.1
+******************************************************************
-*****************************************************************
-You can access the Arvados workbench at http://localhost:9899
-*****************************************************************
</code></pre></notextile>
h2. Accessing workbench
Point your browser to the Dockerized workbench:
<notextile>
-<pre><code><span class="userinput">https://localhost:9899</span>
+<pre><code><span class="userinput">http://workbench.dev.arvados</span>
</code></pre>
</notextile>
-Now use the google account you specified as @API_AUTO_ADMIN_USER@ in @config.yml@ to log in. You will be prompted *twice* by your browser that you are accessing a site with an untrusted SSL certificate. This is normal; by default the Arvados Docker installation uses self-signed SSL certificates for the SSO and API servers, respectively.
-
+Now use the google account you specified as @API_AUTO_ADMIN_USER@ in @config.yml@ to log in.
+You will be prompted by your browser that you are accessing a site with an untrusted SSL certificate. This is normal; by default the Arvados Docker installation uses self-signed SSL certificates for the SSO and API servers, respectively. If you use a local SSO server in a Docker container, you will be prompted *twice*. The default is to use the Curoverse SSO server.
projects / arvados.git / blobdiff