--- /dev/null
+ # Arvados-in-a-box
+
+ Self-contained development, demonstration and testing environment for Arvados.
+
+ ## Quick start
+
+ ```
+ $ bin/arvbox reboot localdemo
+ ```
+
+ ## Usage
+
+ ```
+ Arvados-in-a-box
+
+ arvbox (build|start|run|open|shell|ip|stop|reboot|reset|destroy|log|svrestart)
+
+ build <config> build arvbox Docker image
+ start|run <config> start arvbox container
+ open open arvbox workbench in a web browser
+ shell enter arvbox shell
+ ip print arvbox ip address
+ stop stop arvbox container
+ restart <config> stop, then run again
+ reboot <config> stop, build arvbox Docker image, run
+ reset delete arvbox arvados data (be careful!)
+ destroy delete all arvbox code and data (be careful!)
+ log <service> tail log of specified service
+ svrestart <service> restart specified service inside arvbox
+ clone <from> <to> clone an arvbox
+ ```
+
+ ## Requirements
+
+ * Linux 3.x+ and Docker 1.9+
+ * Minimum of 3 GiB of RAM + additional memory to run jobs
+ * Minimum of 3 GiB of disk + storage for actual data
+
+ ## Configs
+
+ ### dev
+ Development configuration. Boots a complete Arvados environment inside the
+ container. The "arvados", "arvado-dev" and "sso-devise-omniauth-provider" code
+ directories along data directories "postgres", "var", "passenger" and "gems"
+ are bind mounted from the host file system for easy access and persistence
+ across container rebuilds. Services are bound to the Docker container's
+ network IP address and can only be accessed on the local host.
+
+ In "dev" mode, you can override the default autogenerated settings of Rails
+ projects by adding "application.yml.override" to any Rails project (sso, api,
+ workbench). This can be used to test out API server settings or point
+ Workbench at an alternate API server.
+
+ ### localdemo
+ Demo configuration. Boots a complete Arvados environment inside the container.
+ Unlike the development configuration, code directories are included in the demo
+ image, and data directories are stored in a separate data volume container.
+ Services are bound to the Docker container's network IP address and can only be
+ accessed on the local host.
+
+ ### test
+ Run the test suite.
+
+ ### publicdev
+ Publicly accessible development configuration. Similar to 'dev' except that
+ service ports are published to the host's IP address and can accessed by anyone
+ who can connect to the host system. WARNING! The public arvbox configuration
+ is NOT SECURE and must not be placed on a public IP address or used for
+ production work.
+
+ ### publicdemo
+ Publicly accessible development configuration. Similar to 'localdemo' except
+ that service ports are published to the host's IP address and can accessed by
+ anyone who can connect to the host system. WARNING! The public arvbox configuration
+ is NOT SECURE and must not be placed on a public IP address or used for
+ production work.
+
+ ## Environment variables
+
+ ### ARVBOX_DOCKER
+ The location of Dockerfile.base and associated files used by "arvbox build".
+ default: result of $(readlink -f $(dirname $0)/../lib/arvbox/docker)
+
+ ### ARVBOX_CONTAINER
+ The name of the Docker container to manipulate.
+ default: arvbox
+
+ ### ARVBOX_BASE
+ The base directory to store persistent data for arvbox containers.
+ default: $HOME/.arvbox
+
+ ### ARVBOX_DATA
+ The base directory to store persistent data for the current container.
+ default: $ARVBOX_BASE/$ARVBOX_CONTAINER
+
+ ### ARVADOS_ROOT
+ The root directory of the Arvados source tree
+ default: $ARVBOX_DATA/arvados
+
+ ### ARVADOS_DEV_ROOT
+ The root directory of the Arvados-dev source tree
+ default: $ARVBOX_DATA/arvados-dev
+
+ ### SSO_ROOT
+ The root directory of the SSO source tree
+ default: $ARVBOX_DATA/sso-devise-omniauth-provider
+
+ ### ARVBOX_PUBLISH_IP
+ The IP address on which to publish services when running in public
+ configuration. Overrides default detection of the host's IP address.
+
+ ## Notes
+
+ Services are designed to install and auto-configure on start or restart. For
+ example, the service script for keepstore always compiles keepstore from source
+ and registers the daemon with the API server.
+
+ Services are run with process supervision, so a service which exits will be
+ restarted. Dependencies between services are handled by repeatedly trying and
+ failing the service script until dependencies are fulfilled (by other service
+ scripts) enabling the service script to complete.
--- /dev/null
+ #!/bin/sh
+
+ set -e
+
+ if ! test -d /sys/fs/cgroup ; then
+ echo "Arvbox requires cgroups to be mounted at /sys/fs/cgroup in order to use"
+ echo "Docker-in-Docker. Older operating systems that put cgroups in other"
+ echo "places (such as /cgroup) are not supported."
+ exit 1
+ fi
+
+ if ! which docker >/dev/null 2>/dev/null ; then
+ echo "Arvbox requires Docker. To install, run the following command as root:"
+ echo "curl -sSL https://get.docker.com/ | sh"
+ exit 1
+ fi
+
+ if test -z "$ARVBOX_DOCKER" ; then
+ if which greadlink >/dev/null 2>/dev/null ; then
+ ARVBOX_DOCKER=$(greadlink -f $(dirname $0)/../lib/arvbox/docker)
+ else
+ ARVBOX_DOCKER=$(readlink -f $(dirname $0)/../lib/arvbox/docker)
+ fi
+ fi
+
+ if test -z "$ARVBOX_CONTAINER" ; then
+ ARVBOX_CONTAINER=arvbox
+ fi
+
+ if test -z "$ARVBOX_BASE" ; then
+ ARVBOX_BASE="$HOME/.arvbox"
+ fi
+
+ if test -z "$ARVBOX_DATA" ; then
+ ARVBOX_DATA="$ARVBOX_BASE/$ARVBOX_CONTAINER"
+ fi
+
+ if test -z "$ARVADOS_ROOT" ; then
+ ARVADOS_ROOT="$ARVBOX_DATA/arvados"
+ fi
+
+ if test -z "$ARVADOS_DEV_ROOT" ; then
+ ARVADOS_DEV_ROOT="$ARVBOX_DATA/arvados-dev"
+ fi
+
+ if test -z "$SSO_ROOT" ; then
+ SSO_ROOT="$ARVBOX_DATA/sso-devise-omniauth-provider"
+ fi
+
+ PG_DATA="$ARVBOX_DATA/postgres"
+ VAR_DATA="$ARVBOX_DATA/var"
+ PASSENGER="$ARVBOX_DATA/passenger"
+ GEMS="$ARVBOX_DATA/gems"
+
+ getip() {
+ docker inspect $ARVBOX_CONTAINER | grep \"IPAddress\" | head -n1 | tr -d ' ":,\n' | cut -c10-
+ }
+
+ updateconf() {
+ if test -f ~/.config/arvados/$ARVBOX_CONTAINER.conf ; then
+ sed "s/ARVADOS_API_HOST=.*/ARVADOS_API_HOST=$(getip):8000/" <$HOME/.config/arvados/$ARVBOX_CONTAINER.conf >$HOME/.config/arvados/$ARVBOX_CONTAINER.conf.tmp
+ mv ~/.config/arvados/$ARVBOX_CONTAINER.conf.tmp ~/.config/arvados/$ARVBOX_CONTAINER.conf
+ else
+ mkdir -p $HOME/.config/arvados
+ cat >$HOME/.config/arvados/$ARVBOX_CONTAINER.conf <<EOF
+ ARVADOS_API_HOST=$(getip):8000
+ ARVADOS_API_TOKEN=
+ ARVADOS_API_HOST_INSECURE=true
+ EOF
+ fi
+ }
+
+ wait_for_arvbox() {
+ FF=/tmp/arvbox-fifo-$$
+ mkfifo $FF
+ docker logs -f $ARVBOX_CONTAINER > $FF &
+ LOGPID=$!
+ while read line ; do
+ echo $line
+ if echo $line | grep "Workbench is running at" >/dev/null ; then
+ kill $LOGPID
+ fi
+ done < $FF
+ rm $FF
+ echo
+ if test -n "$localip" ; then
+ echo "export ARVADOS_API_HOST=$localip:8000"
+ else
+ echo "export ARVADOS_API_HOST=$(getip):8000"
+ fi
+ }
+
+ run() {
+ if docker ps -a | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ echo "Container $ARVBOX_CONTAINER is already running, use stop, restart or reboot"
+ exit 0
+ fi
+
+ if echo "$1" | grep '^public' ; then
+ if test -n "$ARVBOX_PUBLISH_IP" ; then
+ localip=$ARVBOX_PUBLISH_IP
+ else
+ defaultdev=$(/sbin/ip route|awk '/default/ { print $5 }')
+ localip=$(ip addr show $defaultdev | grep 'inet ' | sed 's/ *inet \(.*\)\/.*/\1/')
+ fi
+ iptemp=$(tempfile)
+ echo $localip > $iptemp
+ chmod og+r $iptemp
+ PUBLIC="--volume=$iptemp:/var/run/localip_override
+ --publish=80:80
+ --publish=8000:8000
+ --publish=8900:8900
+ --publish=9001:9001
+ --publish=9002:9002
+ --publish=25100:25100
+ --publish=25107:25107
+ --publish=25108:25108
+ --publish=8001:8001"
+ else
+ PUBLIC=""
+ fi
+
+ if echo "$1" | grep 'demo$' ; then
+ if test -d "$ARVBOX_DATA" ; then
+ echo "It looks like you already have a development container named $ARVBOX_CONTAINER."
+ echo "Set ARVBOX_CONTAINER to set a different name for your demo container"
+ exit 1
+ fi
+
+ if ! (docker ps -a | grep -E "$ARVBOX_CONTAINER-data$" -q) ; then
+ docker create -v /var/lib/postgresql -v /var/lib/arvados --name $ARVBOX_CONTAINER-data arvados/arvbox-demo /bin/true
+ fi
+
+ docker run \
+ --detach \
+ --name=$ARVBOX_CONTAINER \
+ --privileged \
+ --volumes-from $ARVBOX_CONTAINER-data \
+ $PUBLIC \
+ arvados/arvbox-demo
+ updateconf
+ wait_for_arvbox
+ else
+ mkdir -p "$PG_DATA" "$VAR_DATA" "$PASSENGER" "$GEMS"
+
+ if ! test -d "$ARVADOS_ROOT" ; then
+ git clone https://github.com/curoverse/arvados.git "$ARVADOS_ROOT"
+ fi
+ if ! test -d "$SSO_ROOT" ; then
+ git clone https://github.com/curoverse/sso-devise-omniauth-provider.git "$SSO_ROOT"
+ fi
+
+ if test "$1" = test ; then
+ shift
+
+ if ! test -d "$ARVADOS_DEV_ROOT" ; then
+ git clone https://github.com/curoverse/arvados-dev.git "$ARVADOS_DEV_ROOT"
+ fi
+
+ mkdir -p $VAR_DATA/test
+
+ docker run \
+ --detach \
+ --name=$ARVBOX_CONTAINER \
+ --privileged \
+ "--volume=$ARVADOS_ROOT:/usr/src/arvados:rw" \
+ "--volume=$ARVADOS_DEV_ROOT:/usr/src/arvados-dev:rw" \
+ "--volume=$SSO_ROOT:/usr/src/sso:rw" \
+ "--volume=$PG_DATA:/var/lib/postgresql:rw" \
+ "--volume=$VAR_DATA:/var/lib/arvados:rw" \
+ "--volume=$PASSENGER:/var/lib/passenger:rw" \
+ "--volume=$GEMS:/var/lib/gems:rw" \
+ arvados/arvbox-dev \
+ /usr/local/bin/runsvinit -svdir=/etc/test-service
+
+ docker exec -ti \
+ $ARVBOX_CONTAINER \
+ /usr/local/lib/arvbox/runsu.sh \
+ /usr/local/lib/arvbox/waitforpostgres.sh
+
+ docker exec -ti \
+ $ARVBOX_CONTAINER \
+ /usr/local/lib/arvbox/runsu.sh \
+ /var/lib/arvbox/service/sso/run-service --only-setup
+
+ docker exec -ti \
+ $ARVBOX_CONTAINER \
+ /usr/local/lib/arvbox/runsu.sh \
+ /var/lib/arvbox/service/api/run-service --only-setup
+
+ docker exec -ti \
+ $ARVBOX_CONTAINER \
+ /usr/local/lib/arvbox/runsu.sh \
+ /usr/src/arvados-dev/jenkins/run-tests.sh \
+ --temp /var/lib/arvados/test \
+ WORKSPACE=/usr/src/arvados \
+ GEM_HOME=/var/lib/gems \
+ "$@"
+ elif echo "$1" | grep 'dev$' ; then
+ docker run \
+ --detach \
+ --name=$ARVBOX_CONTAINER \
+ --privileged \
+ "--volume=$ARVADOS_ROOT:/usr/src/arvados:rw" \
+ "--volume=$SSO_ROOT:/usr/src/sso:rw" \
+ "--volume=$PG_DATA:/var/lib/postgresql:rw" \
+ "--volume=$VAR_DATA:/var/lib/arvados:rw" \
+ "--volume=$PASSENGER:/var/lib/passenger:rw" \
+ "--volume=$GEMS:/var/lib/gems:rw" \
+ $PUBLIC \
+ arvados/arvbox-dev
+ updateconf
+ wait_for_arvbox
+ echo "The Arvados source code is checked out at: $ARVADOS_ROOT"
+ else
+ echo "Unknown configuration '$1'"
+ fi
+ fi
+ }
+
+ stop() {
+ if docker ps -a --filter "status=running" | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ docker stop $ARVBOX_CONTAINER
+ fi
+
+ VOLUMES=--volumes=true
+ if docker ps -a --filter "status=created" | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ docker rm $VOLUMES $ARVBOX_CONTAINER
+ fi
+ if docker ps -a --filter "status=exited" | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ docker rm $VOLUMES $ARVBOX_CONTAINER
+ fi
+ }
+
+ build() {
+ if ! test -f "$ARVBOX_DOCKER/Dockerfile.base" ; then
+ echo "Could not find Dockerfile ($ARVBOX_DOCKER/Dockerfile.base)"
+ exit 1
+ fi
+ docker build -t arvados/arvbox-base -f "$ARVBOX_DOCKER/Dockerfile.base" "$ARVBOX_DOCKER"
+ if test "$1" = localdemo -o "$1" = publicdemo ; then
+ docker build -t arvados/arvbox-demo -f "$ARVBOX_DOCKER/Dockerfile.demo" "$ARVBOX_DOCKER"
+ else
+ docker build -t arvados/arvbox-dev -f "$ARVBOX_DOCKER/Dockerfile.dev" "$ARVBOX_DOCKER"
+ fi
+ }
+
+ check() {
+ case "$1" in
+ localdemo|publicdemo|dev|publicdev|test)
+ true
+ ;;
+ *)
+ echo "Argument to $subcmd must be one of localdemo, publicdemo, dev, publicdev, test"
+ exit 1
+ ;;
+ esac
+ }
+
+ subcmd="$1"
+ if test -n "$subcmd" ; then
+ shift
+ fi
+ case "$subcmd" in
+ build)
+ check $@
+ build $@
+ ;;
+
+ start|run)
+ check $@
+ run $@
+ ;;
+
+ sh*)
+ docker exec -ti $ARVBOX_CONTAINER /usr/bin/env TERM=$TERM GEM_HOME=/var/lib/gems /bin/bash
+ ;;
+
+ stop)
+ stop
+ ;;
+
+ restart)
+ check $@
+ stop
+ run $@
+ ;;
+
+ reboot)
+ check $@
+ stop
+ build $@
+ run $@
+ ;;
+
+ ip|open)
+ if test "$subcmd" = 'ip' ; then
+ echo $(getip)
+ else
+ xdg-open http://$(getip)
+ fi
+ ;;
+
+ status)
+ echo "Selected: $ARVBOX_CONTAINER"
+ if docker ps -a --filter "status=running" | grep -E "$ARVBOX_CONTAINER$" -q ; then
+ echo "Status: running"
+ echo "IP: $(getip)"
+ else
+ echo "Status: not running"
+ fi
+ if test -d "$ARVBOX_DATA" ; then
+ echo "Data: $ARVBOX_DATA"
+ elif docker ps -a | grep -E "$ARVBOX_CONTAINER-data$" -q ; then
+ echo "Data: $ARVBOX_CONTAINER-data"
+ else
+ echo "Data: none"
+ fi
+ ;;
+
+ reset|destroy)
+ stop
+ if test -d "$ARVBOX_DATA" ; then
+ if test "$subcmd" = destroy ; then
+ if test "$1" != -f ; then
+ echo "WARNING! This will delete your entire arvbox ($ARVBOX_DATA)."
+ echo "Use destroy -f if you really mean it."
+ exit 1
+ fi
+ set -x
+ rm -rf "$ARVBOX_DATA"
+ else
+ if test "$1" != -f ; then
+ echo "WARNING! This will delete your arvbox data ($ARVBOX_DATA)."
+ echo "Code and downloaded packages will be preserved."
+ echo "Use reset -f if you really mean it."
+ exit 1
+ fi
+ set -x
+ rm -rf "$ARVBOX_DATA/postgres"
+ rm -rf "$ARVBOX_DATA/var"
+ fi
+ else
+ if test "$1" != -f ; then
+ echo "WARNING! This will delete your data container $ARVBOX_CONTAINER-data. Use -f if you really mean it."
+ exit 1
+ fi
+ set -x
+ docker rm "$ARVBOX_CONTAINER-data"
+ fi
+ ;;
+
+ log)
+ if test -n "$1" ; then
+ docker exec -ti $ARVBOX_CONTAINER /usr/bin/env TERM=$TERM less --follow-name +GF "/etc/service/$1/log/main/current"
+ else
+ docker exec -ti $ARVBOX_CONTAINER /usr/bin/env TERM=$TERM tail $(docker exec -ti $ARVBOX_CONTAINER find -L /etc -path '/etc/service/*/log/main/current' -printf " %p")
+ fi
+ ;;
+
+ sv)
+ if test -n "$1" ; then
+ docker exec -ti $ARVBOX_CONTAINER sv "$1" "$2"
+ else
+ echo "Usage: $0 $subcmd <service>"
+ echo "Available services:"
+ docker exec -ti $ARVBOX_CONTAINER ls /etc/service
+ fi
+ ;;
+
+ clone)
+ if test -n "$2" ; then
+ cp -r "$ARVBOX_BASE/$1" "$ARVBOX_BASE/$2"
+ echo "Created new arvbox $2"
+ echo "export ARVBOX_CONTAINER=$2"
+ else
+ echo "clone <from> <to> clone an arvbox"
+ echo "available arvboxes: $(ls $ARVBOX_BASE)"
+ fi
+ ;;
+
+ *)
+ echo "Arvados-in-a-box"
+ echo
+ echo "$(basename $0) (build|start|run|open|shell|ip|stop|reboot|reset|destroy|log|svrestart)"
+ echo
+ echo "build <config> build arvbox Docker image"
+ echo "start|run <config> start $ARVBOX_CONTAINER container"
+ echo "open open arvbox workbench in a web browser"
+ echo "shell enter arvbox shell"
+ echo "ip print arvbox ip address"
+ echo "status print some information about current arvbox"
+ echo "stop stop arvbox container"
+ echo "restart <config> stop, then run again"
+ echo "reboot <config> stop, build arvbox Docker image, run"
+ echo "reset delete arvbox arvados data (be careful!)"
+ echo "destroy delete all arvbox code and data (be careful!)"
+ echo "log <service> tail log of specified service"
+ echo "sv <start|stop|restart> <service> change state of service inside arvbox"
+ echo "clone <from> <to> clone an arvbox"
+ ;;
+ esac
--- /dev/null
+ FROM debian:8
+
+ RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get -yq install \
+ postgresql-9.4 git gcc golang-go runit \
+ ruby rake bundler curl libpq-dev \
+ libcurl4-openssl-dev libssl-dev zlib1g-dev libpcre3-dev \
+ openssh-server python-setuptools netcat-traditional \
+ libpython-dev fuse libfuse-dev python-pip python-yaml \
+ pkg-config libattr1-dev python-llfuse python-pycurl \
+ libwww-perl libio-socket-ssl-perl libcrypt-ssleay-perl \
+ libjson-perl nginx gitolite3 lsof python-epydoc graphviz \
+ apt-transport-https ca-certificates slurm-wlm
+
+ VOLUME /var/lib/docker
+ VOLUME /var/log/nginx
+ VOLUME /etc/ssl/private
+
+ RUN apt-key adv --keyserver hkp://pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D || \
+ apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
+
+ RUN mkdir -p /etc/apt/sources.list.d && \
+ echo deb https://apt.dockerproject.org/repo debian-jessie main > /etc/apt/sources.list.d/docker.list && \
+ apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get -yq install docker-engine=1.9.1-0~jessie
+
+ RUN rm -rf /var/lib/postgresql && mkdir -p /var/lib/postgresql
+
+ RUN cd /root && \
+ GOPATH=$PWD go get github.com/curoverse/runsvinit && \
+ install bin/runsvinit /usr/local/bin
+
+ ADD fuse.conf /etc/
+
+ ADD crunch-setup.sh gitolite.rc \
+ keep-setup.sh common.sh createusers.sh \
+ logger runsu.sh waitforpostgres.sh \
+ application_yml_override.py \
+ /usr/local/lib/arvbox/
+
+ # Start the supervisor.
+ CMD ["/usr/local/bin/runsvinit"]
--- /dev/null
+ FROM arvados/arvbox-base
+
+ RUN cd /usr/src && \
+ git clone https://github.com/curoverse/arvados.git && \
+ git clone https://github.com/curoverse/sso-devise-omniauth-provider.git sso
+
+ ADD service/ /var/lib/arvbox/service
+ RUN rmdir /etc/service && ln -sf /var/lib/arvbox/service /etc
+
+ RUN chown -R 1000:1000 /usr/src && /usr/local/lib/arvbox/createusers.sh
+
+ RUN sudo -u arvbox /var/lib/arvbox/service/sso/run-service --only-deps
+ RUN sudo -u arvbox /var/lib/arvbox/service/api/run-service --only-deps
+ RUN sudo -u arvbox /var/lib/arvbox/service/workbench/run-service --only-deps
+ RUN sudo -u arvbox /var/lib/arvbox/service/doc/run-service --only-deps
+ RUN sudo -u arvbox /var/lib/arvbox/service/vm/run-service --only-deps
+ RUN sudo -u arvbox /var/lib/arvbox/service/sdk/run-service
--- /dev/null
+ FROM arvados/arvbox-base
+
+ RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get -yq install \
+ python-virtualenv python3-virtualenv linkchecker xvfb iceweasel
+
+ RUN set -e && \
+ PJS=phantomjs-1.9.7-linux-x86_64 && \
+ curl -L -o/tmp/$PJS.tar.bz2 https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2 && \
+ tar -C /usr/local -xjf /tmp/$PJS.tar.bz2 && \
+ ln -s ../$PJS/bin/phantomjs /usr/local/bin/
+
+ ADD service/ /var/lib/arvbox/service
+ RUN rmdir /etc/service && ln -sf /var/lib/arvbox/service /etc
+
+ RUN mkdir /etc/test-service && ln -sf /var/lib/arvbox/service/postgres /etc/test-service
--- /dev/null
+ #!/usr/bin/env python
+
+ import yaml
+
+ try:
+ with open("application.yml.override") as f:
+ b = yaml.load(f)
+ except IOError:
+ exit()
+
+ with open("application.yml") as f:
+ a = yaml.load(f)
+
+ def recursiveMerge(a, b):
+ if isinstance(a, dict) and isinstance(b, dict):
+ for k in b:
+ print k
+ a[k] = recursiveMerge(a.get(k), b[k])
+ return a
+ else:
+ return b
+
+ with open("application.yml", "w") as f:
+ yaml.dump(recursiveMerge(a, b), f)
--- /dev/null
+
+ if test -s /var/run/localip_override ; then
+ localip=$(cat /var/run/localip_override)
+ else
+ defaultdev=$(/sbin/ip route|awk '/default/ { print $5 }')
+ localip=$(ip addr show $defaultdev | grep 'inet ' | sed 's/ *inet \(.*\)\/.*/\1/')
+ fi
+
+ export GEM_HOME=/var/lib/gems
+ export GEM_PATH=/var/lib/gems
+
+ declare -A services
+ services=(
+ [workbench]=80
+ [api]=8000
+ [sso]=8900
+ [arv-git-httpd]=9001
+ [keep-web]=9002
+ [keepproxy]=25100
+ [keepstore0]=25107
+ [keepstore1]=25108
+ [ssh]=22
+ [doc]=8001
+ )
+
+ if test "$(id arvbox -u 2>/dev/null)" = 0 ; then
+ PGUSER=postgres
+ PGGROUP=postgres
+ else
+ PGUSER=arvbox
+ PGGROUP=arvbox
+ fi
+
+ run_bundler() {
+ if test -f Gemfile.lock ; then
+ frozen=--frozen
+ else
+ frozen=""
+ fi
+ if ! flock /var/lib/arvados/gems.lock bundle install --path $GEM_HOME --local --no-deployment $frozen "$@" ; then
+ flock /var/lib/arvados/gems.lock bundle install --path $GEM_HOME --no-deployment $frozen "$@"
+ fi
+ }
+
+ pip_install() {
+ pushd /var/lib/arvados/pip
+ for p in $(ls http*.tar.gz) ; do
+ if test -f $p ; then
+ ln -sf $p $(echo $p | sed 's/.*%2F\(.*\)/\1/')
+ fi
+ done
+ for p in $(ls http*.whl) ; do
+ if test -f $p ; then
+ ln -sf $p $(echo $p | sed 's/.*%2F\(.*\)/\1/')
+ fi
+ done
+ popd
+
+ if ! pip install --no-index --find-links /var/lib/arvados/pip $1 ; then
+ pip install $1
+ fi
+ }
--- /dev/null
+ #!/bin/bash
+
+ set -e -o pipefail
+
+ if ! grep "^arvbox:" /etc/passwd >/dev/null 2>/dev/null ; then
+ HOSTUID=$(ls -nd /usr/src/arvados | sed 's/ */ /' | cut -d' ' -f4)
+ HOSTGID=$(ls -nd /usr/src/arvados | sed 's/ */ /' | cut -d' ' -f5)
+ FUSEGID=$(ls -nd /dev/fuse | sed 's/ */ /' | cut -d' ' -f5)
+
+ mkdir -p /var/lib/arvados/git /var/lib/gems /var/lib/passenger
+
+ groupadd --gid $HOSTGID --non-unique arvbox
+ groupadd --gid $FUSEGID --non-unique fuse
+ groupadd --gid $HOSTGID --non-unique git
+ useradd --home-dir /var/lib/arvados \
+ --uid $HOSTUID --gid $HOSTGID \
+ --non-unique \
+ --groups docker,fuse \
+ arvbox
+ useradd --home-dir /var/lib/arvados/git --uid $HOSTUID --gid $HOSTGID --non-unique git
+ useradd --groups docker,fuse crunch
+
+ chown arvbox:arvbox -R /usr/local /var/lib/arvados /var/lib/gems \
+ /var/lib/passenger /var/lib/postgresql \
+ /var/lib/nginx /var/log/nginx /etc/ssl/private
+
+ mkdir -p /var/lib/gems/ruby/2.1.0
+ chown arvbox:arvbox -R /var/lib/gems/ruby/2.1.0
+
+ mkdir -p /tmp/crunch0 /tmp/crunch1
+ chown crunch:crunch -R /tmp/crunch0 /tmp/crunch1
+
+ echo "arvbox ALL=(crunch) NOPASSWD: ALL" >> /etc/sudoers
+ fi
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ mkdir -p /var/lib/arvados/gostuff
+ cd /var/lib/arvados/gostuff
+
+ export GOPATH=$PWD
+ mkdir -p "$GOPATH/src/git.curoverse.com"
+ ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+ flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/crunchstat"
+ install bin/crunchstat /usr/local/bin
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export ARVADOS_API_TOKEN=$(cat /usr/src/arvados/services/api/superuser_token)
+ export CRUNCH_JOB_BIN=/usr/src/arvados/sdk/cli/bin/crunch-job
+ export PERLLIB=/usr/src/arvados/sdk/perl/lib
+ export CRUNCH_TMP=/tmp/$1
+ export CRUNCH_DISPATCH_LOCKFILE=/var/lock/$1-dispatch
+ export CRUNCH_JOB_DOCKER_BIN=docker
+ export HOME=/tmp/$1
+
+ cd /usr/src/arvados/services/api
+ exec bundle exec ./script/crunch-dispatch.rb development
--- /dev/null
+ user_allow_other
--- /dev/null
+ # This is based on the default Gitolite configuration file with the following
+ # changes applied as described here:
+ # http://doc.arvados.org/install/install-arv-git-httpd.html
+
+ # configuration variables for gitolite
+
+ # This file is in perl syntax. But you do NOT need to know perl to edit it --
+ # just mind the commas, use single quotes unless you know what you're doing,
+ # and make sure the brackets and braces stay matched up!
+
+ # (Tip: perl allows a comma after the last item in a list also!)
+
+ # HELP for commands can be had by running the command with "-h".
+
+ # HELP for all the other FEATURES can be found in the documentation (look for
+ # "list of non-core programs shipped with gitolite" in the master index) or
+ # directly in the corresponding source file.
+
+ my $repo_aliases;
+ my $aliases_src = "$ENV{HOME}/.gitolite/arvadosaliases.pl";
+ if ($ENV{HOME} && (-e $aliases_src)) {
+ $repo_aliases = do $aliases_src;
+ }
+ $repo_aliases ||= {};
+
+ %RC = (
+
+ REPO_ALIASES => $repo_aliases,
+
+ # ------------------------------------------------------------------
+
+ # default umask gives you perms of '0700'; see the rc file docs for
+ # how/why you might change this
+ UMASK => 0022,
+
+ # look for "git-config" in the documentation
+ GIT_CONFIG_KEYS => '',
+
+ # comment out if you don't need all the extra detail in the logfile
+ LOG_EXTRA => 1,
+ # logging options
+ # 1. leave this section as is for 'normal' gitolite logging (default)
+ # 2. uncomment this line to log ONLY to syslog:
+ # LOG_DEST => 'syslog',
+ # 3. uncomment this line to log to syslog and the normal gitolite log:
+ # LOG_DEST => 'syslog,normal',
+ # 4. prefixing "repo-log," to any of the above will **also** log just the
+ # update records to "gl-log" in the bare repo directory:
+ # LOG_DEST => 'repo-log,normal',
+ # LOG_DEST => 'repo-log,syslog',
+ # LOG_DEST => 'repo-log,syslog,normal',
+
+ # roles. add more roles (like MANAGER, TESTER, ...) here.
+ # WARNING: if you make changes to this hash, you MUST run 'gitolite
+ # compile' afterward, and possibly also 'gitolite trigger POST_COMPILE'
+ ROLES => {
+ READERS => 1,
+ WRITERS => 1,
+ },
+
+ # enable caching (currently only Redis). PLEASE RTFM BEFORE USING!!!
+ # CACHE => 'Redis',
+
+ # ------------------------------------------------------------------
+
+ # rc variables used by various features
+
+ # the 'info' command prints this as additional info, if it is set
+ # SITE_INFO => 'Please see http://blahblah/gitolite for more help',
+
+ # the CpuTime feature uses these
+ # display user, system, and elapsed times to user after each git operation
+ # DISPLAY_CPU_TIME => 1,
+ # display a warning if total CPU times (u, s, cu, cs) crosses this limit
+ # CPU_TIME_WARN_LIMIT => 0.1,
+
+ # the Mirroring feature needs this
+ # HOSTNAME => "foo",
+
+ # TTL for redis cache; PLEASE SEE DOCUMENTATION BEFORE UNCOMMENTING!
+ # CACHE_TTL => 600,
+
+ # ------------------------------------------------------------------
+
+ # suggested locations for site-local gitolite code (see cust.html)
+
+ # this one is managed directly on the server
+ # LOCAL_CODE => "$ENV{HOME}/local",
+
+ # or you can use this, which lets you put everything in a subdirectory
+ # called "local" in your gitolite-admin repo. For a SECURITY WARNING
+ # on this, see http://gitolite.com/gitolite/non-core.html#pushcode
+ # LOCAL_CODE => "$rc{GL_ADMIN_BASE}/local",
+
+ # ------------------------------------------------------------------
+
+ # List of commands and features to enable
+
+ ENABLE => [
+
+ # COMMANDS
+
+ # These are the commands enabled by default
+ 'help',
+ 'desc',
+ 'info',
+ 'perms',
+ 'writable',
+
+ # Uncomment or add new commands here.
+ # 'create',
+ # 'fork',
+ # 'mirror',
+ # 'readme',
+ # 'sskm',
+ # 'D',
+
+ # These FEATURES are enabled by default.
+
+ # essential (unless you're using smart-http mode)
+ 'ssh-authkeys',
+
+ # creates git-config enties from gitolite.conf file entries like 'config foo.bar = baz'
+ 'git-config',
+
+ # creates git-daemon-export-ok files; if you don't use git-daemon, comment this out
+ 'daemon',
+
+ # creates projects.list file; if you don't use gitweb, comment this out
+ 'gitweb',
+
+ # These FEATURES are disabled by default; uncomment to enable. If you
+ # need to add new ones, ask on the mailing list :-)
+
+ # user-visible behaviour
+
+ # prevent wild repos auto-create on fetch/clone
+ # 'no-create-on-read',
+ # no auto-create at all (don't forget to enable the 'create' command!)
+ # 'no-auto-create',
+
+ # access a repo by another (possibly legacy) name
+ 'Alias',
+
+ # give some users direct shell access. See documentation in
+ # sts.html for details on the following two choices.
+ # "Shell $ENV{HOME}/.gitolite.shell-users",
+ # 'Shell alice bob',
+
+ # set default roles from lines like 'option default.roles-1 = ...', etc.
+ # 'set-default-roles',
+
+ # show more detailed messages on deny
+ # 'expand-deny-messages',
+
+ # show a message of the day
+ # 'Motd',
+
+ # system admin stuff
+
+ # enable mirroring (don't forget to set the HOSTNAME too!)
+ # 'Mirroring',
+
+ # allow people to submit pub files with more than one key in them
+ # 'ssh-authkeys-split',
+
+ # selective read control hack
+ # 'partial-copy',
+
+ # manage local, gitolite-controlled, copies of read-only upstream repos
+ # 'upstream',
+
+ # updates 'description' file instead of 'gitweb.description' config item
+ # 'cgit',
+
+ # allow repo-specific hooks to be added
+ # 'repo-specific-hooks',
+
+ # performance, logging, monitoring...
+
+ # be nice
+ # 'renice 10',
+
+ # log CPU times (user, system, cumulative user, cumulative system)
+ # 'CpuTime',
+
+ # syntactic_sugar for gitolite.conf and included files
+
+ # allow backslash-escaped continuation lines in gitolite.conf
+ # 'continuation-lines',
+
+ # create implicit user groups from directory names in keydir/
+ # 'keysubdirs-as-groups',
+
+ # allow simple line-oriented macros
+ # 'macros',
+
+ # Kindergarten mode
+
+ # disallow various things that sensible people shouldn't be doing anyway
+ # 'Kindergarten',
+ ],
+
+ );
+
+ # ------------------------------------------------------------------------------
+ # per perl rules, this should be the last line in such a file:
+ 1;
+
+ # Local variables:
+ # mode: perl
+ # End:
+ # vim: set syn=perl:
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ sleep 2
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ mkdir -p /var/lib/arvados/gostuff
+ cd /var/lib/arvados/gostuff
+
+ export GOPATH=$PWD
+ mkdir -p "$GOPATH/src/git.curoverse.com"
+ ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+ flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/keepstore"
+ install bin/keepstore /usr/local/bin
+
+ mkdir -p /var/lib/arvados/$1
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+ set +e
+ read -rd $'\000' keepservice <<EOF
+ {
+ "service_host":"$localip",
+ "service_port":$2,
+ "service_ssl_flag":false,
+ "service_type":"disk"
+ }
+ EOF
+ set -e
+
+ if test -s /var/lib/arvados/$1-uuid ; then
+ keep_uuid=$(cat /var/lib/arvados/$1-uuid)
+ arv keep_service update --uuid $keep_uuid --keep-service "$keepservice"
+ else
+ UUID=$(arv --format=uuid keep_service create --keep-service "$keepservice")
+ echo $UUID > /var/lib/arvados/$1-uuid
+ fi
+
+ set +e
+ killall -HUP keepproxy
+
+ exec /usr/local/bin/keepstore \
+ -listen=:$2 \
+ -enforce-permissions=true \
+ -blob-signing-key-file=/var/lib/arvados/blob_signing_key \
+ -max-buffers=20 \
+ -volume=/var/lib/arvados/$1
--- /dev/null
+ #!/bin/sh
+ exec svlogd -tt ./main
--- /dev/null
+ # Object files
+ *.o
+ *.ko
+ *.obj
+ *.elf
+
+ # Precompiled Headers
+ *.gch
+ *.pch
+
+ # Libraries
+ *.lib
+ *.a
+ *.la
+ *.lo
+
+ # Shared objects (inc. Windows DLLs)
+ *.dll
+ *.so
+ *.so.*
+ *.dylib
+
+ # Executables
+ *.exe
+ *.out
+ *.app
+ *.i*86
+ *.x86_64
+ *.hex
+
+ # Debug files
+ *.dSYM/
--- /dev/null
+ Copyright (c) 2015, Kosma Moczek
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+ * Neither the name of runit-docker nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
--- /dev/null
+ CFLAGS=-std=c99 -Wall -O2 -fPIC -D_POSIX_SOURCE -D_GNU_SOURCE
+ LDLIBS=-ldl
+
+ PROGNAME=runit-docker
+
+ all: $(PROGNAME).so
+
+ %.so: %.c
+ gcc -shared $(CFLAGS) $(LDLIBS) -o $@ $^
+
+ install: runit-docker.so
+ mkdir -p $(DESTDIR)/sbin
+ mkdir -p $(DESTDIR)/lib
+ install -m 755 $(PROGNAME) $(DESTDIR)/sbin/
+ install -m 755 $(PROGNAME).so $(DESTDIR)/lib/
+
+ clean:
+ $(RM) $(PROGNAME).so
--- /dev/null
+ # runit-docker
+
+ Docker and `runsvdir` don't quite agree on what each signal means, causing
+ TONS of frustration when attempting to use `runsvdir` as init under Docker.
+ `runit-docker` is a plug'n'play adapter library which does signal translation
+ without the overhead and nuisance of running a nanny process.
+
+ ## Features
+
+ * Pressing Ctrl-C does a clean shutdown.
+ * `docker stop` does a clean shutdown.
+
+ Under the hood, `runit-docker` translates `SIGTERM` and `SIGINT` to `SIGHUP`.
+
+ ## Usage
+
+ * Build with `make`, install with `make install`.
+ * Add `CMD ["/sbin/runit-docker"]` to your `Dockerfile`.
+ * Run `debian/rules clean build binary` to build a Debian package.
+
+ ## Author
+
+ runit-docker was written by Kosma Moczek <kosma.moczek@pixers.pl> during a single Scrum
+ planning meeting. Damn meetings.
--- /dev/null
+ runit-docker (1.1) unstable; urgency=low
+
+ * Simplify logic.
+ * Install for SIGINT as well.
+
+ -- Kosma Moczek <kosma@kosma.pl> Mon, 11 May 2015 12:23:59 +0000
+
+ runit-docker (1.0) unstable; urgency=low
+
+ * Initial release
+
+ -- Kosma Moczek <kosma@kosma.pl> Mon, 11 May 2015 12:23:59 +0000
--- /dev/null
+ Source: runit-docker
+ Section: contrib/admin
+ Priority: optional
+ Maintainer: Kosma Moczek <kosma@kosma.pl>
+ Build-Depends: debhelper (>= 9)
+ Standards-Version: 3.9.5
+ Homepage: https://github.com/kosma/runit-docker
+ #Vcs-Git: git://anonscm.debian.org/collab-maint/runit-docker.git
+ #Vcs-Browser: http://anonscm.debian.org/?p=collab-maint/runit-docker.git;a=summary
+
+ Package: runit-docker
+ Architecture: any
+ Depends: ${shlibs:Depends}, ${misc:Depends}
+ Description: painlessly use runit in Docker containers
--- /dev/null
+ Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+ Upstream-Name: runit-docker
+ Source: https://github.com/kosma/runit-docker
+
+ Files: *
+ Copyright: 2015 Kosma Moczek <kosma@kosma.pl>
+ License: MIT
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+ * Neither the name of runit-docker nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--- /dev/null
+ README.md
--- /dev/null
+ #!/usr/bin/make -f
+ # See debhelper(7) (uncomment to enable)
+ # output every command that modifies files on the build system.
+ #DH_VERBOSE = 1
+
+ # see EXAMPLES in dpkg-buildflags(1) and read /usr/share/dpkg/*
+ DPKG_EXPORT_BUILDFLAGS = 1
+ include /usr/share/dpkg/default.mk
+
+ # see FEATURE AREAS in dpkg-buildflags(1)
+ #export DEB_BUILD_MAINT_OPTIONS = hardening=+all
+
+ # see ENVIRONMENT in dpkg-buildflags(1)
+ # package maintainers to append CFLAGS
+ #export DEB_CFLAGS_MAINT_APPEND = -Wall -pedantic
+ # package maintainers to append LDFLAGS
+ #export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed
+
+
+ # main packaging script based on dh7 syntax
+ %:
+ dh $@
+
+ # debmake generated override targets
+ # This is example for Cmake (See http://bugs.debian.org/641051 )
+ #override_dh_auto_configure:
+ # dh_auto_configure -- \
+ # -DCMAKE_LIBRARY_PATH=$(DEB_HOST_MULTIARCH)
+
+
+
+
--- /dev/null
+ 3.0 (quilt)
--- /dev/null
+ #!/bin/sh
+
+ export LD_PRELOAD=/lib/runit-docker.so
+ exec runsvdir /etc/service
--- /dev/null
+ #include <signal.h>
+ #include <dlfcn.h>
+ #include <stdlib.h>
+
+
+ int sigaction(int signum, const struct sigaction *act, struct sigaction *oldact)
+ {
+ static int (*real_sigaction)(int signum, const struct sigaction *act, struct sigaction *oldact) = NULL;
+
+ // Retrieve the real sigaction we just shadowed.
+ if (real_sigaction == NULL) {
+ real_sigaction = (void *) dlsym(RTLD_NEXT, "sigaction");
+ // Prevent further shadowing in children.
+ unsetenv("LD_PRELOAD");
+ }
+
+ if (signum == SIGTERM) {
+ // Skip this handler, it doesn't do what we want.
+ return 0;
+ }
+
+ if (signum == SIGHUP) {
+ // Install this handler for others as well.
+ real_sigaction(SIGTERM, act, oldact);
+ real_sigaction(SIGINT, act, oldact);
+ }
+
+ // Forward the call the the real sigaction.
+ return real_sigaction(signum, act, oldact);
+ }
+
+ // vim: ts=2 sw=2 et
--- /dev/null
+ #!/bin/sh
+
+ HOSTUID=$(ls -nd /usr/src/arvados | sed 's/ */ /' | cut -d' ' -f4)
+ HOSTGID=$(ls -nd /usr/src/arvados | sed 's/ */ /' | cut -d' ' -f5)
+
+ flock /var/lib/arvados/createusers.lock /usr/local/lib/arvbox/createusers.sh
+
+ export HOME=/var/lib/arvados
+
+ if test -z "$1" ; then
+ exec chpst -u arvbox:arvbox:docker $0-service
+ else
+ exec chpst -u arvbox:arvbox:docker $@
+ fi
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -ex -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ cd /usr/src/arvados/services/api
+ export RAILS_ENV=development
+
+ run_bundler --without=development
+ bundle exec passenger start --runtime-check-only --runtime-dir=/var/lib/passenger
+
+ if test "$1" = "--only-deps" ; then
+ exit
+ fi
+
+ set -u
+
+ if ! test -s /var/lib/arvados/api_uuid_prefix ; then
+ ruby -e 'puts "#{rand(2**64).to_s(36)[0,5]}"' > /var/lib/arvados/api_uuid_prefix
+ fi
+ uuid_prefix=$(cat /var/lib/arvados/api_uuid_prefix)
+
+ if ! test -s /var/lib/arvados/api_secret_token ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/api_secret_token
+ fi
+ secret_token=$(cat /var/lib/arvados/api_secret_token)
+
+ if ! test -s /var/lib/arvados/blob_signing_key ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/blob_signing_key
+ fi
+ blob_signing_key=$(cat /var/lib/arvados/blob_signing_key)
+
+ # self signed key will be created by SSO server script.
+ test -s /var/lib/arvados/self-signed.key
+
+ sso_app_secret=$(cat /var/lib/arvados/sso_app_secret)
+
+ if test -s /var/lib/arvados/vm-uuid ; then
+ vm_uuid=$(cat /var/lib/arvados/vm-uuid)
+ else
+ vm_uuid=$uuid_prefix-2x53u-$(ruby -e 'puts rand(2**400).to_s(36)[0,15]')
+ echo $vm_uuid > /var/lib/arvados/vm-uuid
+ fi
+
+ cat >config/application.yml <<EOF
+ development:
+ uuid_prefix: $uuid_prefix
+ secret_token: $secret_token
+ blob_signing_key: $blob_signing_key
+ sso_app_secret: $sso_app_secret
+ sso_app_id: arvados-server
+ sso_provider_url: "https://$localip:${services[sso]}"
+ sso_insecure: true
+ workbench_address: "http://$localip/"
+ git_repo_ssh_base: "git@$localip:"
+ git_repo_https_base: "http://$localip:${services[arv-git-httpd]}/"
+ new_users_are_active: true
+ auto_admin_first_user: true
+ auto_setup_new_users: true
+ auto_setup_new_users_with_vm_uuid: $vm_uuid
+ auto_setup_new_users_with_repository: true
+ default_collection_replication: 1
+ EOF
+
+ (cd config && /usr/local/lib/arvbox/application_yml_override.py)
+
+ if ! test -f /var/lib/arvados/api_database_pw ; then
+ ruby -e 'puts rand(2**128).to_s(36)' > /var/lib/arvados/api_database_pw
+ fi
+ database_pw=$(cat /var/lib/arvados/api_database_pw)
+
+ if ! (psql postgres -c "\du" | grep "^ arvados ") >/dev/null ; then
+ psql postgres -c "create user arvados with password '$database_pw'"
+ psql postgres -c "ALTER USER arvados CREATEDB;"
+ fi
+
+ sed "s/password:.*/password: $database_pw/" <config/database.yml.example >config/database.yml
+
+ if ! test -f /var/lib/arvados/api_database_setup ; then
+ bundle exec rake db:setup
+ touch /var/lib/arvados/api_database_setup
+ fi
+
+ if ! test -s /var/lib/arvados/superuser_token ; then
+ bundle exec ./script/create_superuser_token.rb > /var/lib/arvados/superuser_token
+ fi
+
+ rm -rf tmp
+
+ bundle exec rake db:migrate
+
+ set +u
+ if test "$1" = "--only-setup" ; then
+ exit
+ fi
+
+ ARVADOS_WEBSOCKETS=1 exec bundle exec passenger start --port=${services[api]} \
+ --runtime-dir=/var/lib/passenger \
+ --ssl --ssl-certificate=/var/lib/arvados/self-signed.pem \
+ --ssl-certificate-key=/var/lib/arvados/self-signed.key
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ mkdir -p /var/lib/arvados/gostuff
+ cd /var/lib/arvados/gostuff
+
+ export GOPATH=$PWD
+ mkdir -p "$GOPATH/src/git.curoverse.com"
+ ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+ flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/arv-git-httpd"
+ install bin/arv-git-httpd /usr/local/bin
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export GITOLITE_HTTP_HOME=/var/lib/arvados/git
+ export GL_BYPASS_ACCESS_CHECKS=1
+ export PATH="$PATH:/var/lib/arvados/git/bin"
+ cd ~git
+
+ exec /usr/local/bin/arv-git-httpd \
+ -address=:${services[arv-git-httpd]} \
+ -git-command=/usr/share/gitolite3/gitolite-shell \
+ -repo-root=/var/lib/arvados/git/repositories
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ mkdir -p /var/lib/arvados/gostuff
+ cd /var/lib/arvados/gostuff
+
+ export GOPATH=$PWD
+ mkdir -p "$GOPATH/src/git.curoverse.com"
+ ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+ flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/crunch-run"
+ flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/crunch-dispatch-local"
+ install bin/crunch-run bin/crunch-dispatch-local /usr/local/bin
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+ exec /usr/local/bin/crunch-dispatch-local -crunch-run-command=/usr/local/bin/crunch-run
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/sh
+ exec /usr/local/lib/arvbox/crunch-setup.sh crunch0
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/sh
+ sleep 1
+ exec /usr/local/lib/arvbox/crunch-setup.sh crunch1
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -ex -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ cd /usr/src/arvados/doc
+ run_bundler --without=development
+
+ if test "$1" = "--only-deps" ; then
+ exit
+ fi
+
+ set -u
+
+ cat <<EOF >/var/lib/arvados/doc-nginx.conf
+ worker_processes auto;
+ pid /var/lib/arvados/doc-nginx.pid;
+ daemon off;
+
+ events {
+ worker_connections 64;
+ }
+
+ http {
+ include /etc/nginx/mime.types;
+ default_type application/octet-stream;
+ server {
+ listen ${services[doc]} default_server;
+ listen [::]:${services[doc]} default_server;
+ root /usr/src/arvados/doc/.site;
+ index index.html;
+ server_name _;
+ }
+ }
+ EOF
+
+ bundle exec rake generate baseurl=http://$localip:${services[doc]} arvados_api_host=$localip:${services[api]} arvados_workbench_host=http://$localip
+
+ exec nginx -c /var/lib/arvados/doc-nginx.conf
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ #!/bin/bash
+
+ # Taken from https://github.com/jpetazzo/dind
+
+ exec 2>&1
+
+ # Ensure that all nodes in /dev/mapper correspond to mapped devices currently loaded by the device-mapper kernel driver
+ dmsetup mknodes
+
+ : {LOG:=stdio}
+
+ # First, make sure that cgroups are mounted correctly.
+ CGROUP=/sys/fs/cgroup
+ [ -d $CGROUP ] || mkdir $CGROUP
+
+ if mountpoint -q $CGROUP ; then
+ break
+ else
+ mount -n -t tmpfs -o uid=0,gid=0,mode=0755 cgroup $CGROUP
+ fi
+
+ if ! mountpoint -q $CGROUP ; then
+ echo "Could not find or mount cgroups. Tried /sys/fs/cgroup and /cgroup. Did you use --privileged?"
+ exit 1
+ fi
+
+ if [ -d /sys/kernel/security ] && ! mountpoint -q /sys/kernel/security
+ then
+ mount -t securityfs none /sys/kernel/security || {
+ echo "Could not mount /sys/kernel/security."
+ echo "AppArmor detection and --privileged mode might break."
+ }
+ fi
+
+ # Mount the cgroup hierarchies exactly as they are in the parent system.
+ for SUBSYS in $(cut -d: -f2 /proc/1/cgroup)
+ do
+ [ -d $CGROUP/$SUBSYS ] || mkdir $CGROUP/$SUBSYS
+ mountpoint -q $CGROUP/$SUBSYS ||
+ mount -n -t cgroup -o $SUBSYS cgroup $CGROUP/$SUBSYS
+
+ # The two following sections address a bug which manifests itself
+ # by a cryptic "lxc-start: no ns_cgroup option specified" when
+ # trying to start containers withina container.
+ # The bug seems to appear when the cgroup hierarchies are not
+ # mounted on the exact same directories in the host, and in the
+ # container.
+
+ # Named, control-less cgroups are mounted with "-o name=foo"
+ # (and appear as such under /proc/<pid>/cgroup) but are usually
+ # mounted on a directory named "foo" (without the "name=" prefix).
+ # Systemd and OpenRC (and possibly others) both create such a
+ # cgroup. To avoid the aforementioned bug, we symlink "foo" to
+ # "name=foo". This shouldn't have any adverse effect.
+ echo $SUBSYS | grep -q ^name= && {
+ NAME=$(echo $SUBSYS | sed s/^name=//)
+ ln -s $SUBSYS $CGROUP/$NAME
+ }
+
+ # Likewise, on at least one system, it has been reported that
+ # systemd would mount the CPU and CPU accounting controllers
+ # (respectively "cpu" and "cpuacct") with "-o cpuacct,cpu"
+ # but on a directory called "cpu,cpuacct" (note the inversion
+ # in the order of the groups). This tries to work around it.
+ [ $SUBSYS = cpuacct,cpu ] && ln -s $SUBSYS $CGROUP/cpu,cpuacct
+ done
+
+ # Note: as I write those lines, the LXC userland tools cannot setup
+ # a "sub-container" properly if the "devices" cgroup is not in its
+ # own hierarchy. Let's detect this and issue a warning.
+ grep -q :devices: /proc/1/cgroup ||
+ echo "WARNING: the 'devices' cgroup should be in its own hierarchy."
+ grep -qw devices /proc/1/cgroup ||
+ echo "WARNING: it looks like the 'devices' cgroup is not mounted."
+
+ # Now, close extraneous file descriptors.
+ pushd /proc/self/fd >/dev/null
+ for FD in *
+ do
+ case "$FD" in
+ # Keep stdin/stdout/stderr
+ [012])
+ ;;
+ # Nuke everything else
+ *)
+ eval exec "$FD>&-"
+ ;;
+ esac
+ done
+ popd >/dev/null
+
+
+ # If a pidfile is still around (for example after a container restart),
+ # delete it so that docker can start.
+ rm -rf /var/run/docker.pid
+
+ read pid cmd state ppid pgrp session tty_nr tpgid rest < /proc/self/stat
+ trap "kill -TERM -$pgrp; exit" EXIT TERM KILL SIGKILL SIGTERM SIGQUIT
+
+ if ! docker daemon --storage-driver=overlay $DOCKER_DAEMON_ARGS ; then
+ docker daemon $DOCKER_DAEMON_ARGS
+ fi
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ mkdir -p /var/lib/arvados/git
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+ export USER=git
+ export USERNAME=git
+ export LOGNAME=git
+ export HOME=/var/lib/arvados/git
+
+ cd ~arvbox
+
+ mkdir -p ~arvbox/.ssh ~git/.ssh
+ chmod 0700 ~arvbox/.ssh ~git/.ssh
+
+ if ! test -s ~arvbox/.ssh/id_rsa ; then
+ ssh-keygen -t rsa -P '' -f .ssh/id_rsa
+ cp ~arvbox/.ssh/id_rsa ~arvbox/.ssh/id_rsa.pub ~git/.ssh
+ fi
+
+ if test -s ~arvbox/.ssh/known_hosts ; then
+ ssh-keygen -f ".ssh/known_hosts" -R localhost
+ fi
+
+ if ! test -f /var/lib/arvados/gitolite-setup ; then
+ cd ~git
+
+ # Do a no-op login to populate known_hosts
+ # with the hostkey, so it won't try to ask
+ # about it later.
+ cp .ssh/id_rsa.pub .ssh/authorized_keys
+ ssh -o stricthostkeychecking=no git@localhost true
+ rm .ssh/authorized_keys
+
+ cp /usr/local/lib/arvbox/gitolite.rc .gitolite.rc
+
+ gitolite setup -pk .ssh/id_rsa.pub
+
+ if ! test -d gitolite-admin ; then
+ git clone git@localhost:gitolite-admin
+ fi
+
+ cd gitolite-admin
+ git config user.email arvados
+ git config user.name arvados
+ git config push.default simple
+ git push
+
+ touch /var/lib/arvados/gitolite-setup
+ else
+ # Do a no-op login to populate known_hosts
+ # with the hostkey, so it won't try to ask
+ # about it later. Don't run anything,
+ # get the default gitolite behavior.
+ ssh -o stricthostkeychecking=no git@localhost
+ fi
+
+ prefix=$(arv --format=uuid user current | cut -d- -f1)
+
+ if ! test -s /var/lib/arvados/arvados-git-uuid ; then
+ repo_uuid=$(arv --format=uuid repository create --repository "{\"owner_uuid\":\"$prefix-tpzed-000000000000000\", \"name\":\"arvados\"}")
+ echo $repo_uuid > /var/lib/arvados/arvados-git-uuid
+ fi
+
+ repo_uuid=$(cat /var/lib/arvados/arvados-git-uuid)
+
+ if ! test -s /var/lib/arvados/arvados-git-link-uuid ; then
+ all_users_group_uuid="$prefix-j7d0g-fffffffffffffff"
+
+ set +e
+ read -rd $'\000' newlink <<EOF
+ {
+ "tail_uuid":"$all_users_group_uuid",
+ "head_uuid":"$repo_uuid",
+ "link_class":"permission",
+ "name":"can_read"
+ }
+ EOF
+ set -e
+ link_uuid=$(arv --format=uuid link create --link "$newlink")
+ echo $link_uuid > /var/lib/arvados/arvados-git-link-uuid
+ fi
+
+ if ! test -d /var/lib/arvados/git/repositories/$repo_uuid.git ; then
+ git clone --bare /usr/src/arvados /var/lib/arvados/git/repositories/$repo_uuid.git
+ else
+ git --git-dir=/var/lib/arvados/git/repositories/$repo_uuid.git fetch -f /usr/src/arvados master:master
+ fi
+
+ cd /usr/src/arvados/services/api
+ export RAILS_ENV=development
+
+ git_user_key=$(cat ~git/.ssh/id_rsa.pub)
+
+ cat > config/arvados-clients.yml <<EOF
+ development:
+ gitolite_url: /var/lib/arvados/git/repositories/gitolite-admin.git
+ gitolite_tmp: /var/lib/arvados/git
+ arvados_api_host: $localip:${services[api]}
+ arvados_api_token: "$ARVADOS_API_TOKEN"
+ arvados_api_host_insecure: true
+ gitolite_arvados_git_user_key: "$git_user_key"
+ EOF
+
+ while true ; do
+ bundle exec script/arvados-git-sync.rb development
+ sleep 120
+ done
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ mkdir -p /var/lib/arvados/gostuff
+ cd /var/lib/arvados/gostuff
+
+ export GOPATH=$PWD
+ mkdir -p "$GOPATH/src/git.curoverse.com"
+ ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+ flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/keep-web"
+ install bin/keep-web /usr/local/bin
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+ exec /usr/local/bin/keep-web -trust-all-content -listen=:${services[keep-web]}
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ sleep 2
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ mkdir -p /var/lib/arvados/gostuff
+ cd /var/lib/arvados/gostuff
+
+ export GOPATH=$PWD
+ mkdir -p "$GOPATH/src/git.curoverse.com"
+ ln -sfn "/usr/src/arvados" "$GOPATH/src/git.curoverse.com/arvados.git"
+ flock /var/lib/arvados/gostuff.lock go get -t "git.curoverse.com/arvados.git/services/keepproxy"
+ install bin/keepproxy /usr/local/bin
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+
+ set +e
+ read -rd $'\000' keepservice <<EOF
+ {
+ "service_host":"$localip",
+ "service_port":${services[keepproxy]},
+ "service_ssl_flag":false,
+ "service_type":"proxy"
+ }
+ EOF
+ set -e
+
+ if test -s /var/lib/arvados/keepproxy-uuid ; then
+ keep_uuid=$(cat /var/lib/arvados/keepproxy-uuid)
+ arv keep_service update --uuid $keep_uuid --keep-service "$keepservice"
+ else
+ UUID=$(arv --format=uuid keep_service create --keep-service "$keepservice")
+ echo $UUID > /var/lib/arvados/keepproxy-uuid
+ fi
+
+ exec /usr/local/bin/keepproxy -listen=:${services[keepproxy]}
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+ . /usr/local/lib/arvbox/common.sh
+ exec /usr/local/lib/arvbox/keep-setup.sh keep0 ${services[keepstore0]}
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+ . /usr/local/lib/arvbox/common.sh
+ exec /usr/local/lib/arvbox/keep-setup.sh keep1 ${services[keepstore1]}
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ #!/bin/bash
+
+ flock /var/lib/arvados/createusers.lock /usr/local/lib/arvbox/createusers.sh
+
+ . /usr/local/lib/arvbox/common.sh
+
+ chown -R $PGUSER:$PGGROUP /var/lib/postgresql
+ chown -R $PGUSER:$PGGROUP /var/run/postgresql
+ chown -R $PGUSER:$PGGROUP /etc/postgresql
+ chown -R $PGUSER:$PGGROUP /etc/ssl/private
+
+ exec chpst -u $PGUSER:$PGGROUP $0-service
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ if ! test -d /var/lib/postgresql/9.4/main ; then
+ /usr/lib/postgresql/9.4/bin/initdb -D /var/lib/postgresql/9.4/main
+ sh -c "while ! (psql postgres -c'\du' | grep '^ arvbox ') >/dev/null ; do createuser -s arvbox ; sleep 1 ; done" &
+ fi
+ mkdir -p /var/run/postgresql/9.4-main.pg_stat_tmp
+
+ rm -f /var/lib/postgresql/9.4/main/postmaster.pid
+
+ exec /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ . /usr/local/lib/arvbox/common.sh
+
+ set -eu -o pipefail
+
+ if ! [[ -d /tmp/arvbox-ready ]] ; then
+ echo
+ echo "Arvados-in-a-box starting"
+ echo
+ echo "Note: if this is a fresh arvbox installation, it may take 10-15 minutes (or longer) to download and"
+ echo "install dependencies. Use \"arvbox log\" to monitor the progress of specific services."
+ echo
+ mkdir -p /tmp/arvbox-ready
+ fi
+
+ sleep 3
+
+ waiting=""
+
+ for s in "${!services[@]}"
+ do
+ if ! [[ -f /tmp/arvbox-ready/$s ]] ; then
+ if nc -z localhost ${services[$s]} ; then
+ echo "$s is ready at $localip:${services[$s]}"
+ touch /tmp/arvbox-ready/$s
+ else
+ waiting="$waiting $s"
+ fi
+ fi
+ done
+
+ if ! docker version >/dev/null 2>/dev/null ; then
+ waiting="$waiting docker"
+ fi
+
+ if ! which arv >/dev/null ; then
+ waiting="$waiting sdk"
+ elif ! which arv-get >/dev/null ; then
+ waiting="$waiting sdk"
+ fi
+
+ if ! (ps x | grep -v grep | grep "crunch-dispatch") > /dev/null ; then
+ waiting="$waiting crunch-dispatch"
+ fi
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+
+ vm_ok=0
+ if test -s /var/lib/arvados/vm-uuid -a -s /var/lib/arvados/superuser_token; then
+ vm_uuid=$(cat /var/lib/arvados/vm-uuid)
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+ if (which arv && arv virtual_machine get --uuid $vm_uuid) >/dev/null 2>/dev/null ; then
+ vm_ok=1
+ fi
+ fi
+
+ if test $vm_ok = 0 ; then
+ waiting="$waiting vm"
+ fi
+
+ if ! [[ -z "$waiting" ]] ; then
+ if ps x | grep -v grep | grep "bundle install" > /dev/null; then
+ gemcount=$(ls /var/lib/gems/ruby/2.1.0/gems 2>/dev/null | wc -l)
+
+ gemlockcount=0
+ for l in /usr/src/arvados/services/api/Gemfile.lock \
+ /usr/src/arvados/apps/workbench/Gemfile.lock \
+ /usr/src/sso/Gemfile.lock ; do
+ gc=$(cat $l \
+ | grep -vE "(GEM|PLATFORMS|DEPENDENCIES|$^|remote:|specs:)" \
+ | sed 's/^ *//' | sed 's/(.*)//' | sed 's/ *$//' | sort | uniq | wc -l)
+ gemlockcount=$(($gemlockcount + $gc))
+ done
+ waiting="$waiting (installing ruby gems $gemcount/$gemlockcount)"
+ fi
+
+ if ps x | grep -v grep | grep "c++.*/var/lib/passenger" > /dev/null ; then
+ waiting="$waiting (compiling passenger)"
+ fi
+
+ if ps x | grep -v grep | grep "pip install" > /dev/null; then
+ waiting="$waiting (installing python packages)"
+ fi
+ echo " Waiting for$waiting ..."
+ exit 1
+ fi
+
+ echo
+ echo "Your Arvados-in-a-box is ready!"
+ echo "Workbench is running at http://$localip"
+
+ rm -r /tmp/arvbox-ready
+
+ sv stop ready >/dev/null
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ #!/bin/sh
+ set -e
+
+ /usr/local/lib/arvbox/runsu.sh $0-service
+ sv stop sdk
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ mkdir -p ~/.pip /var/lib/arvados/pip
+ cat > ~/.pip/pip.conf <<EOF
+ [global]
+ download_cache = /var/lib/arvados/pip
+ EOF
+
+ cd /usr/src/arvados/sdk/cli
+ run_bundler --binstubs=$PWD/binstubs
+ ln -sf /usr/src/arvados/sdk/cli/binstubs/arv /usr/local/bin/arv
+
+ cd /usr/src/arvados/sdk/python
+ python setup.py sdist
+ pip_install $(ls dist/arvados-python-client-*.tar.gz | tail -n1)
+
+ cd /usr/src/arvados/services/fuse
+ python setup.py sdist
+ pip_install $(ls dist/arvados_fuse-*.tar.gz | tail -n1)
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ cat > /etc/slurm-llnl/slurm.conf <<EOF
+ ControlMachine=$HOSTNAME
+ ControlAddr=$HOSTNAME
+ AuthType=auth/munge
+ DefaultStorageLoc=/var/log/slurm-llnl
+ SelectType=select/cons_res
+ SelectTypeParameters=CR_CPU_Memory
+ SlurmUser=arvbox
+ SlurmdUser=arvbox
+ SlurmctldPort=7002
+ SlurmctldTimeout=300
+ SlurmdPort=7003
+ SlurmdSpoolDir=/var/tmp/slurmd.spool
+ SlurmdTimeout=300
+ StateSaveLocation=/var/tmp/slurm.state
+ NodeName=$HOSTNAME
+ PartitionName=compute State=UP Default=YES Nodes=$HOSTNAME
+ EOF
+
+ mkdir -p /var/run/munge
+
+ /usr/sbin/munged -f
+
+ exec /usr/sbin/slurmctld -v -D
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ exec /usr/local/lib/arvbox/runsu.sh /usr/sbin/slurmd -v -D
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -eux -o pipefail
+
+ if ! test -d /var/run/sshd ; then
+ mkdir /var/run/sshd
+ chmod 0755 /var/run/sshd
+ fi
+ /usr/sbin/sshd -D
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ /usr/local/lib/arvbox/runsu.sh
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -ex -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ cd /usr/src/sso
+ export RAILS_ENV=development
+
+ run_bundler --without=development
+ bundle exec passenger start --runtime-check-only --runtime-dir=/var/lib/passenger
+
+ if test "$1" = "--only-deps" ; then
+ exit
+ fi
+
+ set -u
+
+ if ! test -s /var/lib/arvados/sso_uuid_prefix ; then
+ ruby -e 'puts "#{rand(2**64).to_s(36)[0,5]}"' > /var/lib/arvados/sso_uuid_prefix
+ fi
+ uuid_prefix=$(cat /var/lib/arvados/sso_uuid_prefix)
+
+ if ! test -s /var/lib/arvados/sso_secret_token ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/sso_secret_token
+ fi
+ secret_token=$(cat /var/lib/arvados/sso_secret_token)
+
+ if ! test -s /var/lib/arvados/self-signed.key ; then
+ openssl req -new -x509 -nodes -out /var/lib/arvados/self-signed.pem -keyout /var/lib/arvados/self-signed.key -days 365 -subj '/CN=localhost'
+ fi
+
+ cat >config/application.yml <<EOF
+ development:
+ uuid_prefix: $uuid_prefix
+ secret_token: $secret_token
+ default_link_url: "http://$localip"
+ allow_account_registration: true
+ EOF
+
+ (cd config && /usr/local/lib/arvbox/application_yml_override.py)
+
+ if ! test -f /var/lib/arvados/sso_database_pw ; then
+ ruby -e 'puts rand(2**128).to_s(36)' > /var/lib/arvados/sso_database_pw
+ fi
+ database_pw=$(cat /var/lib/arvados/sso_database_pw)
+
+ if ! (psql postgres -c "\du" | grep "^ arvados_sso ") >/dev/null ; then
+ psql postgres -c "create user arvados_sso with password '$database_pw'"
+ psql postgres -c "ALTER USER arvados_sso CREATEDB;"
+ fi
+
+ sed "s/password:.*/password: $database_pw/" <config/database.yml.example >config/database.yml
+
+ if ! test -f /var/lib/arvados/sso_database_setup ; then
+ bundle exec rake db:setup
+
+ if ! test -s /var/lib/arvados/sso_app_secret ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/sso_app_secret
+ fi
+ app_secret=$(cat /var/lib/arvados/sso_app_secret)
+
+ bundle exec rails console <<EOF
+ c = Client.new
+ c.name = "joshid"
+ c.app_id = "arvados-server"
+ c.app_secret = "$app_secret"
+ c.save!
+ EOF
+
+ touch /var/lib/arvados/sso_database_setup
+ fi
+
+ rm -rf tmp
+
+ bundle exec rake db:migrate
+
+ set +u
+ if test "$1" = "--only-setup" ; then
+ exit
+ fi
+
+ exec bundle exec passenger start --port=${services[sso]} \
+ --runtime-dir=/var/lib/passenger \
+ --ssl --ssl-certificate=/var/lib/arvados/self-signed.pem \
+ --ssl-certificate-key=/var/lib/arvados/self-signed.key
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ #!/bin/bash
+ set -e
+
+ . /usr/local/lib/arvbox/common.sh
+
+ git config --system "credential.http://$localip:${services[arv-git-httpd]}/.username" none
+ git config --system "credential.http://$localip:${services[arv-git-httpd]}/.helper" '!cred(){ cat >/dev/null; if [ "$1" = get ]; then echo password=$ARVADOS_API_TOKEN; fi; };cred'
+
+ /usr/local/lib/arvbox/runsu.sh $0-service
+
+ cd /usr/src/arvados/services/login-sync
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+ export ARVADOS_VIRTUAL_MACHINE_UUID=$(cat /var/lib/arvados/vm-uuid)
+
+ while true ; do
+ bundle exec arvados-login-sync
+ sleep 120
+ done
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ sleep 2
+ set -ex -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ cd /usr/src/arvados/services/login-sync
+ run_bundler
+
+ if test "$1" = "--only-deps" ; then
+ exit
+ fi
+
+ set -u
+
+ export ARVADOS_API_HOST=$localip:${services[api]}
+ export ARVADOS_API_HOST_INSECURE=1
+ export ARVADOS_API_TOKEN=$(cat /var/lib/arvados/superuser_token)
+ export ARVADOS_VIRTUAL_MACHINE_UUID=$(cat /var/lib/arvados/vm-uuid)
+
+ set +e
+ read -rd $'\000' vm <<EOF
+ {
+ "uuid": "$ARVADOS_VIRTUAL_MACHINE_UUID",
+ "hostname":"$localip"
+ }
+ EOF
+ set -e
+
+ if arv virtual_machine get --uuid $ARVADOS_VIRTUAL_MACHINE_UUID ; then
+ arv virtual_machine update --uuid $ARVADOS_VIRTUAL_MACHINE_UUID --virtual-machine "$vm"
+ else
+ arv virtual_machine create --virtual-machine "$vm"
+ fi
--- /dev/null
--- /dev/null
--- /dev/null
+ /usr/local/lib/arvbox/logger
--- /dev/null
+ #!/bin/sh
+ set -e
+
+ /usr/local/lib/arvbox/runsu.sh $0-service $1
+
+ cd /usr/src/arvados/apps/workbench
+
+ rm -rf tmp
+ mkdir tmp
+ chown arvbox:arvbox tmp
+
+ if test "$1" != "--only-deps" ; then
+ exec bundle exec passenger start --port 80 \
+ --user arvbox --runtime-dir=/var/lib/passenger
+ fi
--- /dev/null
+ #!/bin/bash
+
+ exec 2>&1
+ set -ex -o pipefail
+
+ . /usr/local/lib/arvbox/common.sh
+
+ cd /usr/src/arvados/apps/workbench
+ export RAILS_ENV=development
+
+ run_bundler --without=development
+ bundle exec passenger start --runtime-check-only --runtime-dir=/var/lib/passenger
+
+ if test "$1" = "--only-deps" ; then
+ exit
+ fi
+
+ set -u
+
+ if ! test -s /var/lib/arvados/workbench_secret_token ; then
+ ruby -e 'puts rand(2**400).to_s(36)' > /var/lib/arvados/workbench_secret_token
+ fi
+ secret_token=$(cat /var/lib/arvados/workbench_secret_token)
+
+ if ! test -s self-signed.key ; then
+ openssl req -new -x509 -nodes -out self-signed.pem -keyout self-signed.key -days 365 -subj '/CN=localhost'
+ fi
+
+ cat >config/application.yml <<EOF
+ development:
+ secret_token: $secret_token
+ arvados_login_base: https://$localip:${services[api]}/login
+ arvados_v1_base: https://$localip:${services[api]}/arvados/v1
+ arvados_insecure_https: true
+ keep_web_download_url: http://$localip:${services[keep-web]}/c=%{uuid_or_pdh}
+ keep_web_url: http://$localip:${services[keep-web]}/c=%{uuid_or_pdh}
+ arvados_docsite: http://$localip:${services[doc]}/
+ EOF
+
+ (cd config && /usr/local/lib/arvbox/application_yml_override.py)
--- /dev/null
+ #!/bin/sh
+ while ! psql postgres -c\\du >/dev/null 2>/dev/null ; do
+ sleep 1
+ done
projects / arvados.git / commitdiff