---
layout: default
navsection: installguide
-title: Build and install docker images
+title: Build and install Docker images
...
-This installation method is appropriate for testing, evaluation and development on one computer. *For production use, this installation method is not recommended*.
+This method is intended for evaluation and development on a local workstation. It is not suitable for production use in a cluster deployment.
h2. Prerequisites
# A GNU/Linux (virtual) machine
# A working Docker installation (see "Installing Docker":https://docs.docker.com/installation/)
# A working Go installation (see "Install the Go tools":https://golang.org/doc/install)
+# A working Ruby installation (see "Install Ruby and bundler":install-manual-prerequisites-ruby.html)
h2. Download the source tree
date >shell-image
</code></pre></notextile>
-If all goes well, you should now have a number of docker images built:
+If all goes well, you should now have a number of Docker images built (the image id will be different):
<notextile>
<pre><code>~$ <span class="userinput">docker.io images</span>
arvados/shell latest d6cbb5002604 10 minutes ago 1.613 GB
arvados/sso latest 377f1fa0108e 11 minutes ago 1.807 GB
arvados/keep latest ade0e0d2dd00 12 minutes ago 210.8 MB
+arvados/keepproxy latest b39ef0baba02 12 minutes ago 241.6 MB
arvados/workbench latest b0e4fb6da385 12 minutes ago 2.218 GB
arvados/doc latest 4b64daec9454 12 minutes ago 1.524 GB
arvados/compute latest 7f1f5f7faf54 13 minutes ago 1.862 GB
<pre><code>
~$ <span class="userinput">./arvdock -h</span>
-usage: ./arvdock (start|stop|restart|test) [options]
+usage: ./arvdock (start|stop|restart|reset|test) [options]
+
+start run new or restart stopped arvados containers
+stop stop arvados containers
+restart stop and then start arvados containers
+reset stop and delete containers WARNING: this will delete the data inside Arvados!
+test run tests
./arvdock start/stop/restart options:
-d[port], --doc[=port] Documentation server (default port 9898)
-v, --vm Shell server
-n, --nameserver Nameserver
-k, --keep Keep servers
- --ssh Enable SSH access to server containers
+ -p, --keepproxy Keepproxy server
-h, --help Display this help and exit
If no options are given, the action is applied to all servers.
</pre>
</notextile>
-The @--ssh@ option can be useful to debug issues with the Docker containers; it allows you to ssh into the running containers as the @root@ user, provided you have access to the private key that matches the public key specified in @config.yml@'s PUBLIC_KEY_PATH variable.
+To debug issues with the Docker containers use @docker exec@, for example:
+
+<notextile>
+<pre><code>
+~$ <span class="userinput">docker exec -t -i api_server /bin/bash</span>
+</code></pre></notextile>
-Start the docker containers:
+To start the docker containers:
<notextile>
<pre><code>
~$ <span class="userinput">./arvdock start</span>
-sso_server
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 9901:443 --name sso_server arvados/sso
-api_server
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 9900:443 --name api_server --link sso_server:sso arvados/api
-keep_server_0
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 25107:25107 --name keep_server_0 -v /tmp/tmp.aCSx8Pq6Wb:/dev/keep-0 --link api_server:api arvados/keep
-keep_server_1
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 25108:25107 --name keep_server_1 -v /tmp/tmp.m4OQ9WB73G:/dev/keep-0 --link api_server:api arvados/keep
-doc_server
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 9898:80 --name doc_server arvados/doc
+start_api=false
+start_compute=false
+start_doc=false
+start_keep=false
+start_keepproxy=false
+start_nameserver=false
+start_sso=false
+start_vm=false
+start_workbench=false
+Starting crosbymichael/skydns container...
+skydns
+/usr/bin/docker.io run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain arvados
+89922ec786cbd2098ed6bae205468a675657cdb2db0e0bfdfe76d1e43cb2fe35
+Starting crosbymichael/skydock container...
+skydock
+/usr/bin/docker.io run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain arvados -name skydns
+de6a35bb20fb9063fb97218ba2554974546eed969d43b1f5aa31a1ac9576e802
+Starting container: api_server
+ /usr/bin/docker.io start api_server
+Started container: api_server
+Starting container: compute0
+ /usr/bin/docker.io start compute0
+Started container: compute0
+Starting container: compute1
+ /usr/bin/docker.io start compute1
+Started container: compute1
+Starting container: keep_server_0
+ /usr/bin/docker.io start keep_server_0
+Started container: keep_server_0
+Starting container: keep_server_1
+ /usr/bin/docker.io start keep_server_1
+Started container: keep_server_1
+Starting container: keepproxy_server
+ /usr/bin/docker.io start keepproxy_server
+Started container: keepproxy_server
+Starting container: doc_server
+ /usr/bin/docker.io start doc_server
+Started container: doc_server
+
+******************************************************************
+You can access the Arvados documentation at http://doc.dev.arvados
+******************************************************************
+
+Starting container: shell
+ /usr/bin/docker.io start shell
+Started container: shell
+Starting container: workbench_server
+ /usr/bin/docker.io start workbench_server
+Started container: workbench_server
+
+********************************************************************
+You can access the Arvados workbench at http://workbench.dev.arvados
+********************************************************************
*****************************************************************
-You can access the Arvados documentation at http://localhost:9898
-*****************************************************************
+To access Arvados you must add the Arvados nameserver to the top
+of your DNS configuration in /etc/resolv.conf:
+nameserver 172.17.42.1
+******************************************************************
-workbench_server
-Starting container:
- /usr/bin/docker.io run -d -i -t -p 9899:80 --name workbench_server --link api_server:api arvados/workbench
-
-*****************************************************************
-You can access the Arvados workbench at http://localhost:9899
-*****************************************************************
</code></pre></notextile>
h2. Accessing workbench
Point your browser to the Dockerized workbench:
<notextile>
-<pre><code><span class="userinput">https://localhost:9899</span>
+<pre><code><span class="userinput">http://workbench.dev.arvados</span>
</code></pre>
</notextile>
Now use the google account you specified as @API_AUTO_ADMIN_USER@ in @config.yml@ to log in.
You will be prompted by your browser that you are accessing a site with an untrusted SSL certificate. This is normal; by default the Arvados Docker installation uses self-signed SSL certificates for the SSO and API servers, respectively. If you use a local SSO server in a Docker container, you will be prompted *twice*. The default is to use the Curoverse SSO server.
-
-
---
layout: default
navsection: installguide
-title: Install pre-built docker images
+title: Install pre-built Docker images
...
+This method is intended for evaluation and development on a local workstation. It is not suitable for production use in a cluster deployment.
+
{% include 'notebox_begin' %}
-This method is easy, but is not suitable for production use. It is still in development: some features do not work yet.
-* Websockets service is not enabled. This means Workbench auto-refresh and web upload (and some other features) do not work.
-* The node manager is not enabled. Two worker containers are brought up at startup.
-* The automatic network configuration allows you to log in to Workbench from a browser _running on the same host as docker_. Connecting from other hosts will require additional configuration (not covered here).
+* The automatic network configuration allows you to log in to Workbench from a browser _running on the same host as Docker_. Connecting from other hosts requires additional configuration (not covered here).
+* Your data will be stored inside the Docker containers. You may stop and restart the containers without loss, but if you delete the container, your data will be gone.
+* Updating the Arvados software inside the Docker containers is not supported. You may download updated Docker images, but migrating data to updated containers is not yet supported.
{% include 'notebox_end' %}
-This installation method is appropriate for testing and evaluation on one computer. *For production use, this installation method is not recommended*.
-
h2. Prerequisites
-# A GNU/Linux (virtual) machine
+# A GNU/Linux x64 (virtual) machine
# A working Docker installation (see "Installing Docker":https://docs.docker.com/installation/)
# curl
<pre><code>~$ <span class="userinput">\curl -sSL get.arvados.org | bash</span>
</code></pre></notextile>
-This command will download the latest copy of the Arvados docker images. It also gets the arvdock command and saves it in the current working directory. It then uses arvdock to spin up Arvados. Depending on the speed of your internet connection, it can take a while to download the Arvados docker images.
+This command will download the latest build of the Arvados docker images. It also gets the @arvdock@ command and saves it in the current working directory. It then uses @arvdock@ to spin up Arvados. Note that the Arvados Docker images are large and may take a while to download.
+
+If you prefer, you can also download and inspect the installation script before running it. @get.arvados.org@ redirects to "https://raw.githubusercontent.com/curoverse/arvados-dev/master/install/easy-docker-install.sh":https://raw.githubusercontent.com/curoverse/arvados-dev/master/install/easy-docker-install.sh, which is the installation script.
+
+The @arvdock@ command usage is listed here:
+
+<pre>
+usage: ./arvdock (start|stop|restart|reset|test) [options]
+
+start run new or restart stopped arvados containers
+stop stop arvados containers
+restart stop and then start arvados containers
+reset stop and delete containers WARNING: this will delete the data inside Arvados!
+test run tests
+
+./arvdock start/stop/restart options:
+ -d[port], --doc[=port] Documentation server (default port 9898)
+ -w[port], --workbench[=port] Workbench server (default port 9899)
+ -s[port], --sso[=port] SSO server (default port 9901)
+ -a[port], --api[=port] API server (default port 9900)
+ -c, --compute Compute nodes (starts 2)
+ -v, --vm Shell server
+ -n, --nameserver Nameserver
+ -k, --keep Keep servers
+ -p, --keepproxy Keepproxy server
+ -h, --help Display this help and exit
+
+ If no options are given, the action is applied to all servers.
-If you prefer, you can also download the installation script and inspect it before running it. @get.arvados.org@ redirects to "https://raw.githubusercontent.com/curoverse/arvados-dev/master/install/easy-docker-install.sh":https://raw.githubusercontent.com/curoverse/arvados-dev/master/install/easy-docker-install.sh, which is the installation script.
+./arvdock test [testname] [testname] ...
+ By default, all tests are run.
+</pre>
# Install postgres and apache.
RUN apt-get update -qq
RUN apt-get install -qqy \
- procps postgresql postgresql-server-dev-9.1 apache2 slurm-llnl munge \
+ procps postgresql postgresql-server-dev-9.1 slurm-llnl munge \
supervisor sudo libwww-perl libio-socket-ssl-perl libcrypt-ssleay-perl \
libjson-perl cron openssh-server
ADD generated/omniauth.rb /usr/src/arvados/services/api/config/initializers/omniauth.rb
RUN /bin/cp /usr/src/arvados/services/api/config/environments/production.rb.example /usr/src/arvados/services/api/config/environments/production.rb
ADD generated/application.yml /usr/src/arvados/services/api/config/application.yml
-ADD generated/apache2_vhost /etc/apache2/sites-available/arvados
# Configure Rails databases.
ENV RAILS_ENV production
mkdir -p tmp && \
chown www-data:www-data tmp -R
-# Configure Apache and Passenger.
-RUN a2dissite default && \
- a2ensite arvados && \
- a2enmod rewrite && \
- a2enmod ssl && \
- /bin/mkdir /var/run/apache2
-
# Install a token for root
RUN mkdir -p /root/.config/arvados; echo "ARVADOS_API_HOST=api" >> /root/.config/arvados/settings.conf && echo "ARVADOS_API_HOST_INSECURE=yes" >> /root/.config/arvados/settings.conf && echo "ARVADOS_API_TOKEN=$(cat /tmp/superuser_token)" >> /root/.config/arvados/settings.conf && chmod 600 /root/.config/arvados/settings.conf
ADD generated/setup-gitolite.sh /usr/local/bin/setup-gitolite.sh
ADD crunch-dispatch-run.sh /usr/local/bin/crunch-dispatch-run.sh
ADD munge.sh /usr/local/bin/munge.sh
-ADD apache2_foreground.sh /etc/apache2/foreground.sh
+ADD passenger.sh /usr/local/bin/passenger.sh
# Start the supervisor.
CMD ["/usr/bin/supervisord", "-n"]
production:
host: api.dev.arvados
+ git_host: api.dev.arvados
# At minimum, you need a nice long randomly generated secret_token here.
# Use a long string of alphanumeric characters (at least 36).
auto_admin_first_user: true
+ auto_setup_new_users_with_repository: true
+
+ auto_setup_new_users_with_vm_uuid: @@API_HOSTNAME@@-2x53u-csbtkecoa669vkz
+
test:
uuid_prefix: zzzzz
secret_token: <%= rand(2**512).to_s(36) %>
--- /dev/null
+#!/bin/sh
+
+cd /usr/src/arvados/services/api
+export ARVADOS_WEBSOCKETS=1
+export RAILS_ENV=production
+/usr/local/rvm/bin/rvm-exec default bundle exec rake db:migrate
+exec /usr/local/rvm/bin/rvm-exec default bundle exec passenger start -p443 --ssl --ssl-certificate=/etc/ssl/certs/ssl-cert-snakeoil.pem --ssl-certificate-key=/etc/ssl/private/ssl-cert-snakeoil.key
set -x
+if test -f /root/finished_arvados_setup ; then
+ exit
+fi
+
. /etc/profile.d/rvm.sh
export ARVADOS_API_HOST=api
export ARVADOS_API_HOST_INSECURE=yes
export ARVADOS_API_TOKEN=@@API_SUPERUSER_SECRET@@
+export HOME=/root
+
+# Wait for API server to come up.
+while ! arv user current ; do sleep 1 ; done
# Arvados repository object
-all_users_group_uuid="$prefix-j7d0g-fffffffffffffff"
-repo_uuid=`arv --format=uuid repository create --repository '{"name":"arvados","fetch_url":"git@api.dev.arvados:arvados.git","push_url":"git@api.dev.arvados:arvados.git"}'`
+all_users_group_uuid="@@API_HOSTNAME@@-j7d0g-fffffffffffffff"
+
+arv user update --uuid @@API_HOSTNAME@@-tpzed-000000000000000 --user '{"username":"root"}'
+repo_uuid=`arv --format=uuid repository create --repository '{"owner_uuid":"@@API_HOSTNAME@@-tpzed-000000000000000", "name":"arvados"}'`
echo "Arvados repository uuid is $repo_uuid"
read -rd $'\000' newlink <<EOF; arv link create --link "$newlink"
fi
# User repository object
-user_uuid=`arv --format=uuid user current`
-repo_uuid=`arv --format=uuid repository create --repository '{"name":"@@ARVADOS_USER_NAME@@","fetch_url":"git@api.dev.arvados:@@ARVADOS_USER_NAME@@.git","push_url":"git@api.dev.arvados:@@ARVADOS_USER_NAME@@.git"}'`
-echo "User repository uuid is $repo_uuid"
+# user_uuid=`arv --format=uuid user current`
+# repo_uuid=`arv --format=uuid repository create --repository '{"name":"@@ARVADOS_USER_NAME@@","fetch_url":"git@api.dev.arvados:@@ARVADOS_USER_NAME@@.git","push_url":"git@api.dev.arvados:@@ARVADOS_USER_NAME@@.git"}'`
-read -rd $'\000' newlink <<EOF; arv link create --link "$newlink"
-{
- "tail_uuid":"$user_uuid",
- "head_uuid":"$repo_uuid",
- "link_class":"permission",
- "name":"can_write"
-}
-EOF
+# echo "User repository uuid is $repo_uuid"
+
+# read -rd $'\000' newlink <<EOF; arv link create --link "$newlink"
+# {
+# "tail_uuid":"$user_uuid",
+# "head_uuid":"$repo_uuid",
+# "link_class":"permission",
+# "name":"can_write"
+# }
+# EOF
+
+# # Shell machine object
+shell_uuid=`arv --format=uuid virtual_machine create --virtual-machine '{"hostname":"shell"}'`
+arv virtual_machine create --virtual-machine '{"hostname":"shell.dev", "uuid": "@@API_HOSTNAME@@-2x53u-csbtkecoa669vkz"}'
+
+# read -rd $'\000' newlink <<EOF; arv link create --link "$newlink"
+# {
+# "tail_uuid":"$user_uuid",
+# "head_uuid":"$shell_uuid",
+# "link_class":"permission",
+# "name":"can_login",
+# "properties": {"username": "@@ARVADOS_USER_NAME@@"}
+# }
+# EOF
-# Shell machine object
-arv virtual_machine create --virtual-machine '{"hostname":"shell"}'
+touch /root/finished_arvados_setup
command=/usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
autorestart=true
-[program:apache2]
-command=/etc/apache2/foreground.sh
-stopsignal=6
+[program:passenger]
+command=/usr/local/bin/passenger.sh
autorestart=true
[program:munge]
DOCKER=`which docker`
fi
+CURL=`which curl`
+
COMPUTE_COUNTER=0
function usage {
echo >&2
- echo >&2 "usage: $0 (start|stop|restart|test|reset) [options]"
+ echo >&2 "usage: $0 (start|stop|restart|reset|test) [options]"
+ echo >&2
+ echo >&2 "start run new or restart stopped arvados containers"
+ echo >&2 "stop stop arvados containers"
+ echo >&2 "restart stop and then start arvados containers"
+ echo >&2 "reset stop and delete containers WARNING: this will delete the data inside Arvados!"
+ echo >&2 "test run tests"
echo >&2
- echo >&2 "$0 start/stop/restart options:"
+ echo >&2 "$0 options:"
echo >&2 " -d[port], --doc[=port] Documentation server (default port 9898)"
echo >&2 " -w[port], --workbench[=port] Workbench server (default port 9899)"
echo >&2 " -s[port], --sso[=port] SSO server (default port 9901)"
echo "Started container: $container"
fi
- if [[ "$name" == "doc_server" ]]; then
- echo
- echo "******************************************************************"
- echo "You can access the Arvados documentation at http://doc.dev.arvados"
- echo "******************************************************************"
- echo
- fi
-
- if [[ "$name" == "workbench_server" ]]; then
- echo
- echo "********************************************************************"
- echo "You can access the Arvados workbench at http://workbench.dev.arvados"
- echo "********************************************************************"
- echo
- fi
-
-
}
# Create a Docker data volume
echo "To access Arvados you must add the Arvados nameserver to the top"
echo "of your DNS configuration in /etc/resolv.conf:"
echo "nameserver 172.17.42.1"
+ echo
+ echo "Then run '$0 start' again"
echo "******************************************************************"
echo
+ else
+ while ! $CURL -L -f http://workbench.dev.arvados >/dev/null 2>/dev/null ; do
+ echo "Waiting for Arvados to be ready."
+ sleep 1
+ done
+
+ `$DOCKER ps |grep -P "doc_server[^/]" -q`
+ if [[ "$?" == "0" ]]; then
+ echo
+ echo "******************************************************************"
+ echo "You can access the Arvados documentation at http://doc.dev.arvados"
+ echo "******************************************************************"
+ echo
+ fi
+
+ `$DOCKER ps |grep -P "workbench_server[^/]" -q`
+ if [[ "$?" == "0" ]]; then
+ echo
+ echo "********************************************************************"
+ echo "You can access the Arvados workbench at http://workbench.dev.arvados"
+ echo "********************************************************************"
+ echo
+ fi
fi
+
}
function do_stop {
done
}
+if [ "$DOCKER" == '' ]
+then
+ echo "Docker not found. Please install it first."
+ exit 2
+fi
+
+if [ "$CURL" == '' ]
+then
+ echo "Curl not found. Please install it first."
+ exit 3
+fi
+
if [ $# -lt 1 ]
then
usage
export ARVADOS_API_HOST=api
export ARVADOS_API_HOST_INSECURE=yes
export ARVADOS_API_TOKEN=@@API_SUPERUSER_SECRET@@
+export HOME=/root
-arv node create --node {} > /tmp/node.json
+# Wait for API server to come up.
+while ! arv user current ; do sleep 1 ; done
-UUID=`grep \"uuid\" /tmp//node.json |cut -f4 -d\"`
-PING_SECRET=`grep \"ping_secret\" /tmp//node.json |cut -f4 -d\"`
+if ! test -f /root/node.json ; then
+ arv node create --node "{\"hostname\": \"$(hostname)\"}" > /root/node.json
-echo "*/5 * * * * root /usr/bin/curl -k -d ping_secret=$PING_SECRET https://api/arvados/v1/nodes/$UUID/ping" > /etc/cron.d/node_ping
+ # Make sure /dev/fuse permissions are correct (the device appears after fuse is loaded)
+ chmod 1660 /dev/fuse && chgrp fuse /dev/fuse
+fi
-# Send a ping now
-/usr/bin/curl -k -d ping_secret=$PING_SECRET https://api/arvados/v1/nodes/$UUID/ping?ping_secret=$PING_SECRET
+UUID=`grep \"uuid\" /root/node.json |cut -f4 -d\"`
+PING_SECRET=`grep \"ping_secret\" /root/node.json |cut -f4 -d\"`
+
+if ! test -f /etc/cron.d/node_ping ; then
+ echo "*/5 * * * * root /usr/bin/curl -k -d ping_secret=$PING_SECRET https://api/arvados/v1/nodes/$UUID/ping" > /etc/cron.d/node_ping
+fi
-# Just make sure /dev/fuse permissions are correct (the device appears after fuse is loaded)
-chmod 1660 /dev/fuse && chgrp fuse /dev/fuse
+/usr/bin/curl -k -d ping_secret=$PING_SECRET https://api/arvados/v1/nodes/$UUID/ping?ping_secret=$PING_SECRET
projects / arvados.git / commitdiff