<div class="panel-heading">
<h4 class="panel-title">
<a class="component-detail-panel" data-toggle="collapse" href="#errorDetail">
- <span class="caret"></span> Error: <%= sanitize(wu.runtime_status[:error]) %>
+ <span class="caret"></span> Error: <%= h(wu.runtime_status[:error]) %>
</a>
</h4>
</div>
<div id="errorDetail" class="panel-body panel-collapse collapse">
<% if wu.runtime_status[:errorDetail] %>
- <pre><%= sanitize(wu.runtime_status[:errorDetail]) %></pre>
+ <pre><%= h(wu.runtime_status[:errorDetail]) %></pre>
<% else %>
No detailed information available.
<% end %>
<div class="panel-heading">
<h4 class="panel-title">
<a class="component-detail-panel" data-toggle="collapse" href="#warningDetail">
- <span class="caret"></span> Warning: <%= sanitize(wu.runtime_status[:warning]) %>
+ <span class="caret"></span> Warning: <%= h(wu.runtime_status[:warning]) %>
</a>
</h4>
</div>
<div id="warningDetail" class="panel-body panel-collapse collapse">
<% if wu.runtime_status[:warningDetail] %>
- <pre><%= sanitize(wu.runtime_status[:warningDetail]) %></pre>
+ <pre><%= h(wu.runtime_status[:warningDetail]) %></pre>
<% else %>
No detailed information available.
<% end %>
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Minimum of Ruby 2.3 is required. Ruby 2.5 is recommended.
+Ruby 2.5 or newer is required.
* "Option 1: Install from packages":#packages
* "Option 2: Install with RVM":#rvm
h2(#packages). Option 1: Install from packages
{% include 'notebox_begin' %}
-Future versions of Arvados may require a newer version of Ruby than is packaged with your OS. Using OS packages simplifies initial install, but may complicate upgrades that rely on a newer Ruby. If this is a concern, we recommend using "RVM.":#rvm
+Future versions of Arvados may require a newer version of Ruby than is packaged with your OS. Using OS packages simplifies initial install, but may complicate upgrades that rely on a newer Ruby. If this is a concern, we recommend using "RVM":#rvm.
{% include 'notebox_end' %}
h3. Centos 7
-The Ruby version shipped with Centos 7 is too old. Use "RVM.":#rvm
+The Ruby version shipped with Centos 7 is too old. Use "RVM":#rvm to install Ruby 2.5 or later.
h3. Debian and Ubuntu
-Debian 9 (stretch) and Ubuntu 16.04 (xenial) ship Ruby 2.3, which is sufficient to run Arvados. Later releases have newer versions of Ruby that can also run Arvados.
+Debian 9 (stretch) and Ubuntu 16.04 (xenial) ship Ruby 2.3, which is not supported by Arvados. Use "RVM":#rvm to install Ruby 2.5 or later.
+
+Debian 10 (buster) and Ubuntu 18.04 (bionic) and later ship with Ruby 2.5, which is supported by Arvados.
<notextile>
<pre><code># <span class="userinput">apt-get --no-install-recommends install ruby ruby-dev bundler</span></code></pre>
<div class="releasenotes">
</notextile>
-h2(#master). development master (as of 2020-06-17)
+h2(#master). development master (as of 2020-09-22)
"Upgrading from 2.0.0":#v2_0_0
+h3. Minimum supported Ruby version is now 2.5
+
+The minimum supported Ruby version is now 2.5. If you are running Arvados on Debian 9 or Ubuntu 16.04, you may need to switch to using RVM or upgrade your OS. See "Install Ruby and Bundler":../install/ruby.html for more information.
+
h3. Removing libpam-arvados, replaced with libpam-arvados-go
The Python-based PAM package has been replaced with a version written in Go. See "using PAM for authentication":{{site.baseurl}}/install/setup-login.html#pam for details.
action="store_false", default=True,
help=argparse.SUPPRESS)
+ parser.add_argument("--disable-color", dest="enable_color",
+ action="store_false", default=True,
+ help=argparse.SUPPRESS)
+
parser.add_argument("--disable-js-validation",
action="store_true", default=False,
help=argparse.SUPPRESS)
logger.info("%s reused container %s", self.arvrunner.label(self), response["container_uuid"])
else:
logger.info("%s %s state is %s", self.arvrunner.label(self), response["uuid"], response["state"])
- except Exception:
- logger.exception("%s got an error", self.arvrunner.label(self))
+ except Exception as e:
+ logger.exception("%s error submitting container\n%s", self.arvrunner.label(self), e)
logger.debug("Container request was %s", container_request)
self.output_callback({}, "permanentFail")
"--api=containers",
"--no-log-timestamps",
"--disable-validate",
+ "--disable-color",
"--eval-timeout=%s" % self.arvrunner.eval_timeout,
"--thread-count=%s" % self.arvrunner.thread_count,
"--enable-reuse" if self.enable_reuse else "--disable-reuse",
def arv_executor(self, updated_tool, job_order, runtimeContext, logger=None):
self.debug = runtimeContext.debug
+ logger.info("Using cluster %s (%s)", self.api.config()["ClusterID"], self.api.config()["Services"]["Controller"]["ExternalURL"])
+
updated_tool.visit(self.check_features)
self.project_uuid = runtimeContext.project_uuid
'secret_mounts': {},
'state': 'Committed',
'command': ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256", '--debug', '--on-error=continue',
'/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json'],
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = [
'arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--disable-reuse', "--collection-cache-size=256",
'--debug', '--on-error=continue',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = [
'arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--disable-reuse', "--collection-cache-size=256", '--debug', '--on-error=continue',
'/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json']
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256",
'--debug', '--on-error=stop',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256",
"--output-name="+output_name, '--debug', '--on-error=continue',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256", "--debug",
"--storage-classes=foo", '--on-error=continue',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256", '--debug',
'--on-error=continue',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256",
'--debug', '--on-error=continue',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256",
"--output-tags="+output_tags, '--debug', '--on-error=continue',
'name': 'expect_arvworkflow.cwl#main',
'container_image': '999999999999999999999999999999d3+99',
'command': ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256", '--debug', '--on-error=continue',
'/var/lib/cwl/workflow/expect_arvworkflow.cwl#main', '/var/lib/cwl/cwl.input.json'],
'name': 'a test workflow',
'container_image': "999999999999999999999999999999d3+99",
'command': ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256", '--debug', '--on-error=continue',
'/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json'],
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["owner_uuid"] = project_uuid
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
"--eval-timeout=20", "--thread-count=1",
'--enable-reuse', "--collection-cache-size=256", '--debug',
'--on-error=continue',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=60.0', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=256",
'--debug', '--on-error=continue',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=500",
'--debug', '--on-error=continue',
expect_container = copy.deepcopy(stubs.expect_container_spec)
expect_container["command"] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=20',
'--enable-reuse', "--collection-cache-size=256",
'--debug', '--on-error=continue',
"arv": "http://arvados.org/cwl#",
}
expect_container['command'] = ['arvados-cwl-runner', '--local', '--api=containers',
- '--no-log-timestamps', '--disable-validate',
+ '--no-log-timestamps', '--disable-validate', '--disable-color',
'--eval-timeout=20', '--thread-count=1',
'--enable-reuse', "--collection-cache-size=512", '--debug', '--on-error=continue',
'/var/lib/cwl/workflow.json#main', '/var/lib/cwl/cwl.input.json']
"--api=containers",
"--no-log-timestamps",
"--disable-validate",
+ "--disable-color",
"--eval-timeout=20",
'--thread-count=1',
"--enable-reuse",
# of the docker build root.)
FROM debian:9
-MAINTAINER Ward Vandewege <ward@curoverse.com>
+MAINTAINER Peter Amstutz <peter.amstutz@curii.com>
ENV DEBIAN_FRONTEND noninteractive
-ARG pythoncmd=python
-ARG pipcmd=pip
+ARG pythoncmd=python3
+ARG pipcmd=pip3
RUN apt-get update -q && apt-get install -qy --no-install-recommends \
git ${pythoncmd}-pip ${pythoncmd}-virtualenv ${pythoncmd}-dev libcurl4-gnutls-dev \
# empty collection
pdh = collection.portable_data_hash()
assert (pdh == config.EMPTY_BLOCK_LOCATOR), "Empty collection portable_data_hash did not have expected locator, was %s" % pdh
- logger.info("Using empty collection %s", pdh)
+ logger.debug("Using empty collection %s", pdh)
for c in files:
c.keepref = "%s/%s" % (pdh, c.fn)
noop: # nothing happened ...to the 'spectator' user
id: 1
- uuid: zzzzz-xxxxx-pshmckwoma9plh7
+ uuid: zzzzz-57u5n-pshmckwoma9plh7
owner_uuid: zzzzz-tpzed-000000000000000
object_uuid: zzzzz-tpzed-l1s2piq4t4mps8r
object_owner_uuid: zzzzz-tpzed-000000000000000
event_at: <%= 1.minute.ago.to_s(:db) %>
+ created_at: <%= 1.minute.ago.to_s(:db) %>
admin_changes_repository2: # admin changes repository2, which is owned by active user
id: 2
- uuid: zzzzz-xxxxx-pshmckwoma00002
+ uuid: zzzzz-57u5n-pshmckwoma00002
owner_uuid: zzzzz-tpzed-d9tiejq69daie8f # admin user
object_uuid: zzzzz-2x53u-382brsig8rp3667 # repository foo
object_owner_uuid: zzzzz-tpzed-xurymjxw79nv3jz # active user
+ created_at: <%= 2.minute.ago.to_s(:db) %>
event_at: <%= 2.minute.ago.to_s(:db) %>
event_type: update
admin_changes_specimen: # admin changes specimen owned_by_spectator
id: 3
- uuid: zzzzz-xxxxx-pshmckwoma00003
+ uuid: zzzzz-57u5n-pshmckwoma00003
owner_uuid: zzzzz-tpzed-d9tiejq69daie8f # admin user
object_uuid: zzzzz-2x53u-3b0xxwzlbzxq5yr # specimen owned_by_spectator
object_owner_uuid: zzzzz-tpzed-l1s2piq4t4mps8r # spectator user
+ created_at: <%= 3.minute.ago.to_s(:db) %>
event_at: <%= 3.minute.ago.to_s(:db) %>
event_type: update
system_adds_foo_file: # foo collection added, readable by active through link
id: 4
- uuid: zzzzz-xxxxx-pshmckwoma00004
+ uuid: zzzzz-57u5n-pshmckwoma00004
owner_uuid: zzzzz-tpzed-000000000000000 # system user
object_uuid: zzzzz-4zz18-znfnqtbbv4spc3w # foo file
object_owner_uuid: zzzzz-tpzed-000000000000000 # system user
+ created_at: <%= 4.minute.ago.to_s(:db) %>
event_at: <%= 4.minute.ago.to_s(:db) %>
event_type: create
system_adds_baz: # baz collection added, readable by active and spectator through group 'all users' group membership
id: 5
- uuid: zzzzz-xxxxx-pshmckwoma00005
+ uuid: zzzzz-57u5n-pshmckwoma00005
owner_uuid: zzzzz-tpzed-000000000000000 # system user
object_uuid: zzzzz-4zz18-y9vne9npefyxh8g # baz file
object_owner_uuid: zzzzz-tpzed-000000000000000 # system user
+ created_at: <%= 5.minute.ago.to_s(:db) %>
event_at: <%= 5.minute.ago.to_s(:db) %>
event_type: create
log_owned_by_active:
id: 6
- uuid: zzzzz-xxxxx-pshmckwoma12345
+ uuid: zzzzz-57u5n-pshmckwoma12345
owner_uuid: zzzzz-tpzed-xurymjxw79nv3jz # active user
object_uuid: zzzzz-2x53u-382brsig8rp3667 # repository foo
object_owner_uuid: zzzzz-tpzed-xurymjxw79nv3jz # active user
def assert_no_logs_deleted
logs_before = Log.unscoped.all.count
+ assert logs_before > 0
yield
assert_equal logs_before, Log.unscoped.all.count
end
# but 3 minutes suits our test data better (and is test-worthy in
# that it's expected to work correctly in production).
test 'delete old audit logs with production settings' do
- initial_log_count = Log.unscoped.all.count
+ initial_log_count = remaining_audit_logs.count
+ assert initial_log_count > 0
AuditLogs.delete_old(max_age: 180, max_batch: 100000)
assert_operator remaining_audit_logs.count, :<, initial_log_count
end
test 'delete all audit logs in multiple batches' do
+ assert remaining_audit_logs.count > 2
AuditLogs.delete_old(max_age: 0.00001, max_batch: 2)
assert_equal [], remaining_audit_logs.collect(&:uuid)
end
test 'delete old audit logs in thread' do
- begin
- Rails.configuration.AuditLogs.MaxAge = 20
- Rails.configuration.AuditLogs.MaxDeleteBatch = 100000
- Rails.cache.delete 'AuditLogs'
- initial_log_count = Log.unscoped.all.count + 1
- act_as_system_user do
- Log.create!()
- initial_log_count += 1
- end
- deadline = Time.now + 10
- while remaining_audit_logs.count == initial_log_count
- if Time.now > deadline
- raise "timed out"
- end
- sleep 0.1
+ Rails.configuration.AuditLogs.MaxAge = 20
+ Rails.configuration.AuditLogs.MaxDeleteBatch = 100000
+ Rails.cache.delete 'AuditLogs'
+ initial_audit_log_count = remaining_audit_logs.count
+ assert initial_audit_log_count > 0
+ act_as_system_user do
+ Log.create!()
+ end
+ deadline = Time.now + 10
+ while remaining_audit_logs.count == initial_audit_log_count
+ if Time.now > deadline
+ raise "timed out"
end
- assert_operator remaining_audit_logs.count, :<, initial_log_count
+ sleep 0.1
end
+ assert_operator remaining_audit_logs.count, :<, initial_audit_log_count
end
end
while read line ; do
if [[ $line =~ "ok: down: ready:" ]] ; then
kill $LOGPID
- set +e
- wait $LOGPID 2>/dev/null
- set -e
- else
- echo $line
+ set +e
+ wait $LOGPID 2>/dev/null
+ set -e
+ else
+ echo $line
fi
done < $FF
rm $FF
docker_run_dev() {
docker run \
- "--volume=$ARVADOS_ROOT:/usr/src/arvados:rw" \
+ "--volume=$ARVADOS_ROOT:/usr/src/arvados:rw" \
"--volume=$COMPOSER_ROOT:/usr/src/composer:rw" \
"--volume=$WORKBENCH2_ROOT:/usr/src/workbench2:rw" \
"--volume=$PG_DATA:/var/lib/postgresql:rw" \
"--volume=$NPMCACHE:/var/lib/npm:rw" \
"--volume=$GOSTUFF:/var/lib/gopath:rw" \
"--volume=$RLIBS:/var/lib/Rlibs:rw" \
- --label "org.arvados.arvbox_config=$CONFIG" \
- "$@"
+ --label "org.arvados.arvbox_config=$CONFIG" \
+ "$@"
}
running_config() {
need_setup=1
if docker ps -a --filter "status=running" | grep -E "$ARVBOX_CONTAINER$" -q ; then
- if [[ $(running_config) != "$CONFIG" ]] ; then
- echo "Container $ARVBOX_CONTAINER is '$(running_config)' config but requested '$CONFIG'; use restart or reboot"
- return 1
- fi
+ if [[ $(running_config) != "$CONFIG" ]] ; then
+ echo "Container $ARVBOX_CONTAINER is '$(running_config)' config but requested '$CONFIG'; use restart or reboot"
+ return 1
+ fi
if test "$CONFIG" = test -o "$CONFIG" = devenv ; then
need_setup=0
else
if test -n "$TAG"
then
if test $(echo $TAG | cut -c1-1) != '-' ; then
- TAG=":$TAG"
+ TAG=":$TAG"
shift
else
- if [[ $TAG = '-' ]] ; then
- shift
- fi
+ if [[ $TAG = '-' ]] ; then
+ shift
+ fi
unset TAG
fi
fi
defaultdev=$(/sbin/ip route|awk '/default/ { print $5 }')
localip=$(ip addr show $defaultdev | grep 'inet ' | sed 's/ *inet \(.*\)\/.*/\1/')
fi
- echo "Public arvbox will use address $localip"
+ echo "Public arvbox will use address $localip"
iptemp=$(mktemp)
echo $localip > $iptemp
chmod og+r $iptemp
--publish=8001:8001
--publish=8002:8002
--publish=4202:4202
- --publish=45000-45020:45000-45020"
+ --publish=45000-45020:45000-45020"
else
PUBLIC=""
fi
--name=$ARVBOX_CONTAINER \
--privileged \
--volumes-from $ARVBOX_CONTAINER-data \
- --label "org.arvados.arvbox_config=$CONFIG" \
+ --label "org.arvados.arvbox_config=$CONFIG" \
$PUBLIC \
arvados/arvbox-demo$TAG
updateconf
--detach \
--name=$ARVBOX_CONTAINER \
--privileged \
- "--env=SVDIR=/etc/test-service" \
+ "--env=SVDIR=/etc/test-service" \
arvados/arvbox-dev$TAG
docker exec -ti \
/var/lib/arvbox/service/api/run-service --only-setup
fi
- interactive=""
- if [[ -z "$@" ]] ; then
- interactive=--interactive
- fi
+ interactive=""
+ if [[ -z "$@" ]] ; then
+ interactive=--interactive
+ fi
docker exec -ti \
-e LINES=$(tput lines) \
-e TERM=$TERM \
-e WORKSPACE=/usr/src/arvados \
-e GEM_HOME=/var/lib/gems \
- -e CONFIGSRC=/var/lib/arvados/run_tests \
+ -e CONFIGSRC=/var/lib/arvados/run_tests \
$ARVBOX_CONTAINER \
/usr/local/lib/arvbox/runsu.sh \
/usr/src/arvados/build/run-tests.sh \
--temp /var/lib/arvados/test \
- $interactive \
+ $interactive \
"$@"
elif [[ "$CONFIG" = devenv ]] ; then
- if [[ $need_setup = 1 ]] ; then
- docker_run_dev \
+ if [[ $need_setup = 1 ]] ; then
+ docker_run_dev \
--detach \
- --name=${ARVBOX_CONTAINER} \
- "--env=SVDIR=/etc/devenv-service" \
- "--volume=$HOME:$HOME:rw" \
- --volume=/tmp/.X11-unix:/tmp/.X11-unix:rw \
- arvados/arvbox-dev$TAG
- fi
- exec docker exec --interactive --tty \
- -e LINES=$(tput lines) \
- -e COLUMNS=$(tput cols) \
- -e TERM=$TERM \
- -e "ARVBOX_HOME=$HOME" \
- -e "DISPLAY=$DISPLAY" \
- --workdir=$PWD \
- ${ARVBOX_CONTAINER} \
- /usr/local/lib/arvbox/devenv.sh "$@"
+ --name=${ARVBOX_CONTAINER} \
+ "--env=SVDIR=/etc/devenv-service" \
+ "--volume=$HOME:$HOME:rw" \
+ --volume=/tmp/.X11-unix:/tmp/.X11-unix:rw \
+ arvados/arvbox-dev$TAG
+ fi
+ exec docker exec --interactive --tty \
+ -e LINES=$(tput lines) \
+ -e COLUMNS=$(tput cols) \
+ -e TERM=$TERM \
+ -e "ARVBOX_HOME=$HOME" \
+ -e "DISPLAY=$DISPLAY" \
+ --workdir=$PWD \
+ ${ARVBOX_CONTAINER} \
+ /usr/local/lib/arvbox/devenv.sh "$@"
elif [[ "$CONFIG" =~ dev$ ]] ; then
docker_run_dev \
--detach \
updateconf
wait_for_arvbox
echo "The Arvados source code is checked out at: $ARVADOS_ROOT"
- echo "The Arvados testing root certificate is $VAR_DATA/root-cert.pem"
- if [[ "$(listusers)" =~ ^\{\} ]] ; then
- echo "No users defined, use 'arvbox adduser' to add user logins"
- else
- echo "Use 'arvbox listusers' to see user logins"
- fi
+ echo "The Arvados testing root certificate is $VAR_DATA/root-cert.pem"
+ if [[ "$(listusers)" =~ ^\{\} ]] ; then
+ echo "No users defined, use 'arvbox adduser' to add user logins"
+ else
+ echo "Use 'arvbox listusers' to see user logins"
+ fi
else
echo "Unknown configuration '$CONFIG'"
fi
if test -n "$TAG"
then
if test $(echo $TAG | cut -c1-1) != '-' ; then
- TAG=":$TAG"
+ TAG=":$TAG"
shift
else
unset TAG
fi
if echo "$CONFIG" | grep 'demo$' ; then
- docker pull arvados/arvbox-demo$TAG
+ docker pull arvados/arvbox-demo$TAG
else
- docker pull arvados/arvbox-dev$TAG
+ docker pull arvados/arvbox-dev$TAG
fi
}
sh*)
exec docker exec --interactive --tty \
- -e LINES=$(tput lines) \
- -e COLUMNS=$(tput cols) \
- -e TERM=$TERM \
- -e GEM_HOME=/var/lib/gems \
- $ARVBOX_CONTAINER /bin/bash
+ -e LINES=$(tput lines) \
+ -e COLUMNS=$(tput cols) \
+ -e TERM=$TERM \
+ -e GEM_HOME=/var/lib/gems \
+ $ARVBOX_CONTAINER /bin/bash
;;
ash*)
exec docker exec --interactive --tty \
- -e LINES=$(tput lines) \
- -e COLUMNS=$(tput cols) \
- -e TERM=$TERM \
- -e GEM_HOME=/var/lib/gems \
- -u arvbox \
- -w /usr/src/arvados \
- $ARVBOX_CONTAINER /bin/bash --login
+ -e LINES=$(tput lines) \
+ -e COLUMNS=$(tput cols) \
+ -e TERM=$TERM \
+ -e GEM_HOME=/var/lib/gems \
+ -u arvbox \
+ -w /usr/src/arvados \
+ $ARVBOX_CONTAINER /bin/bash --login
;;
pipe)
update)
check $@
stop
- update $@
+ update $@
run $@
;;
status)
echo "Container: $ARVBOX_CONTAINER"
if docker ps -a --filter "status=running" | grep -E "$ARVBOX_CONTAINER$" -q ; then
- echo "Cluster id: $(getclusterid)"
+ echo "Cluster id: $(getclusterid)"
echo "Status: running"
echo "Container IP: $(getip)"
echo "Published host: $(gethost)"
clone)
if test -n "$2" ; then
- mkdir -p "$ARVBOX_BASE/$2"
+ mkdir -p "$ARVBOX_BASE/$2"
cp -a "$ARVBOX_BASE/$1/passenger" \
- "$ARVBOX_BASE/$1/gems" \
- "$ARVBOX_BASE/$1/pip" \
- "$ARVBOX_BASE/$1/npm" \
- "$ARVBOX_BASE/$1/gopath" \
- "$ARVBOX_BASE/$1/Rlibs" \
- "$ARVBOX_BASE/$1/arvados" \
- "$ARVBOX_BASE/$1/composer" \
- "$ARVBOX_BASE/$1/workbench2" \
- "$ARVBOX_BASE/$2"
+ "$ARVBOX_BASE/$1/gems" \
+ "$ARVBOX_BASE/$1/pip" \
+ "$ARVBOX_BASE/$1/npm" \
+ "$ARVBOX_BASE/$1/gopath" \
+ "$ARVBOX_BASE/$1/Rlibs" \
+ "$ARVBOX_BASE/$1/arvados" \
+ "$ARVBOX_BASE/$1/composer" \
+ "$ARVBOX_BASE/$1/workbench2" \
+ "$ARVBOX_BASE/$2"
echo "Created new arvbox $2"
echo "export ARVBOX_CONTAINER=$2"
else
;;
root-cert)
- CERT=$PWD/${ARVBOX_CONTAINER}-root-cert.crt
- if test -n "$1" ; then
- CERT="$1"
- fi
- docker exec $ARVBOX_CONTAINER cat /var/lib/arvados/root-cert.pem > "$CERT"
- echo "Certificate copied to $CERT"
- ;;
+ CERT=$PWD/${ARVBOX_CONTAINER}-root-cert.crt
+ if test -n "$1" ; then
+ CERT="$1"
+ fi
+ docker exec $ARVBOX_CONTAINER cat /var/lib/arvados/root-cert.pem > "$CERT"
+ echo "Certificate copied to $CERT"
+ ;;
psql)
- exec docker exec -ti $ARVBOX_CONTAINER bash -c 'PGPASSWORD=$(cat /var/lib/arvados/api_database_pw) exec psql --dbname=arvados_development --host=localhost --username=arvados'
- ;;
+ exec docker exec -ti $ARVBOX_CONTAINER bash -c 'PGPASSWORD=$(cat /var/lib/arvados/api_database_pw) exec psql --dbname=arvados_development --host=localhost --username=arvados'
+ ;;
checkpoint)
- exec docker exec -ti $ARVBOX_CONTAINER bash -c 'PGPASSWORD=$(cat /var/lib/arvados/api_database_pw) exec pg_dump --host=localhost --username=arvados --clean arvados_development > /var/lib/arvados/checkpoint.sql'
- ;;
+ exec docker exec -ti $ARVBOX_CONTAINER bash -c 'PGPASSWORD=$(cat /var/lib/arvados/api_database_pw) exec pg_dump --host=localhost --username=arvados --clean arvados_development > /var/lib/arvados/checkpoint.sql'
+ ;;
restore)
- exec docker exec -ti $ARVBOX_CONTAINER bash -c 'PGPASSWORD=$(cat /var/lib/arvados/api_database_pw) exec psql --dbname=arvados_development --host=localhost --username=arvados --quiet --file=/var/lib/arvados/checkpoint.sql'
- ;;
+ exec docker exec -ti $ARVBOX_CONTAINER bash -c 'PGPASSWORD=$(cat /var/lib/arvados/api_database_pw) exec psql --dbname=arvados_development --host=localhost --username=arvados --quiet --file=/var/lib/arvados/checkpoint.sql'
+ ;;
hotreset)
- exec docker exec -i $ARVBOX_CONTAINER /usr/bin/env GEM_HOME=/var/lib/gems /bin/bash - <<EOF
+ exec docker exec -i $ARVBOX_CONTAINER /usr/bin/env GEM_HOME=/var/lib/gems /bin/bash - <<EOF
sv stop api
sv stop controller
sv stop websockets
sv restart keepstore1
sv restart keepproxy
EOF
- ;;
+ ;;
adduser)
- docker exec -ti $ARVBOX_CONTAINER /usr/local/lib/arvbox/edit_users.py /var/lib/arvados/cluster_config.yml.override $(getclusterid) add $@
- docker exec $ARVBOX_CONTAINER sv restart controller
- ;;
+ docker exec -ti $ARVBOX_CONTAINER /usr/local/lib/arvbox/edit_users.py /var/lib/arvados/cluster_config.yml.override $(getclusterid) add $@
+ docker exec $ARVBOX_CONTAINER sv restart controller
+ ;;
removeuser)
- docker exec -ti $ARVBOX_CONTAINER /usr/local/lib/arvbox/edit_users.py /var/lib/arvados/cluster_config.yml.override $(getclusterid) remove $@
- docker exec $ARVBOX_CONTAINER sv restart controller
- ;;
+ docker exec -ti $ARVBOX_CONTAINER /usr/local/lib/arvbox/edit_users.py /var/lib/arvados/cluster_config.yml.override $(getclusterid) remove $@
+ docker exec $ARVBOX_CONTAINER sv restart controller
+ ;;
listusers)
- listusers
- ;;
+ listusers
+ ;;
*)
echo "Arvados-in-a-box https://doc.arvados.org/install/arvbox.html"
echo "build <config> build arvbox Docker image"
echo "reboot <config> stop, build arvbox Docker image, run"
echo "rebuild <config> build arvbox Docker image, no layer cache"
- echo "checkpoint create database backup"
- echo "restore restore checkpoint"
- echo "hotreset reset database and restart API without restarting container"
+ echo "checkpoint create database backup"
+ echo "restore restore checkpoint"
+ echo "hotreset reset database and restart API without restarting container"
echo "reset delete arvbox arvados data (be careful!)"
echo "destroy delete all arvbox code and data (be careful!)"
echo "log <service> tail log of specified service"
echo "cat <files> get contents of files inside arvbox"
echo "pipe run a bash script piped in from stdin"
echo "sv <start|stop|restart> <service> "
- echo " change state of service inside arvbox"
+ echo " change state of service inside arvbox"
echo "clone <from> <to> clone dev arvbox"
- echo "adduser <username> <email>"
- echo " add a user login"
- echo "removeuser <username>"
- echo " remove user login"
- echo "listusers list user logins"
+ echo "adduser <username> <email>"
+ echo " add a user login"
+ echo "removeuser <username>"
+ echo " remove user login"
+ echo "listusers list user logins"
;;
esac
#
# SPDX-License-Identifier: AGPL-3.0
-FROM debian:9
+FROM debian:10
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && \
apt-get -yq --no-install-recommends -o Acquire::Retries=6 install \
- postgresql-9.6 postgresql-contrib-9.6 git build-essential runit curl libpq-dev \
- libcurl4-openssl-dev libssl1.0-dev zlib1g-dev libpcre3-dev libpam-dev \
+ postgresql postgresql-contrib git build-essential runit curl libpq-dev \
+ libcurl4-openssl-dev libssl-dev zlib1g-dev libpcre3-dev libpam-dev \
openssh-server netcat-traditional \
graphviz bzip2 less sudo virtualenv \
- libpython-dev fuse libfuse-dev \
+ fuse libfuse-dev \
pkg-config libattr1-dev \
libwww-perl libio-socket-ssl-perl libcrypt-ssleay-perl \
libjson-perl nginx gitolite3 lsof libreadline-dev \
linkchecker python3-virtualenv python3-venv xvfb iceweasel \
libgnutls28-dev python3-dev vim cadaver cython gnupg dirmngr \
libsecret-1-dev r-base r-cran-testthat libxml2-dev pandoc \
- python3-setuptools python3-pip openjdk-8-jdk bsdmainutils net-tools \
- ruby2.3 ruby-dev bundler shellinabox && \
- apt-get remove -yq libpython-dev libpython-stdlib libpython2.7 libpython2.7-dev \
- libpython2.7-minimal libpython2.7-stdlib python2.7-minimal python2.7 && \
+ python3-setuptools python3-pip default-jdk-headless bsdmainutils net-tools \
+ ruby ruby-dev bundler shellinabox && \
apt-get clean
-ENV RUBYVERSION_MINOR 2.3
-ENV RUBYVERSION 2.3.5
+ENV RUBYVERSION_MINOR 2.5
+ENV RUBYVERSION 2.5.1
# Install Ruby from source
# RUN cd /tmp && \
# rm -rf ruby-${RUBYVERSION}
ENV GEM_HOME /var/lib/gems
-ENV GEM_PATH /var/lib/gems
ENV PATH $PATH:/var/lib/gems/bin
-ENV GOVERSION 1.13.6
+ENV GOVERSION 1.15.2
# Install golang binary
RUN curl -f http://storage.googleapis.com/golang/go${GOVERSION}.linux-amd64.tar.gz | \
rm -f /tmp/8D81803C0EBFCD88.asc
RUN mkdir -p /etc/apt/sources.list.d && \
- echo deb https://download.docker.com/linux/debian/ stretch stable > /etc/apt/sources.list.d/docker.list && \
+ echo deb https://download.docker.com/linux/debian/ buster stable > /etc/apt/sources.list.d/docker.list && \
apt-get update && \
- apt-get -yq --no-install-recommends install docker-ce=17.06.0~ce-0~debian && \
+ apt-get -yq --no-install-recommends install docker-ce=5:19.03.13~3-0~debian-buster && \
apt-get clean
RUN rm -rf /var/lib/postgresql && mkdir -p /var/lib/postgresql
# Start the supervisor.
ENV SVDIR /etc/service
STOPSIGNAL SIGINT
-CMD ["/sbin/runit"]
+CMD ["/etc/runit/2"]
#
# SPDX-License-Identifier: AGPL-3.0
-
+export DEBIAN_FRONTEND=noninteractive
export PATH=${PATH}:/usr/local/go/bin:/var/lib/gems/bin
export GEM_HOME=/var/lib/gems
-export GEM_PATH=/var/lib/gems
export npm_config_cache=/var/lib/npm
export npm_config_cache_min=Infinity
export R_LIBS=/var/lib/Rlibs
run_bundler() {
if test -f Gemfile.lock ; then
+ # The 'gem install bundler line below' is cf.
+ # https://bundler.io/blog/2019/05/14/solutions-for-cant-find-gem-bundler-with-executable-bundle.html,
+ # until we get bundler 2.7.10/3.0.0 or higher
+ gem install bundler --no-document -v "$(grep -A 1 "BUNDLED WITH" Gemfile.lock | tail -n 1|tr -d ' ')"
frozen=--frozen
else
frozen=""
# flock /var/lib/gems/gems.lock gem install --verbose --no-document bundler --version ${bundleversion}
# fi
# fi
- if ! flock /var/lib/gems/gems.lock bundler install --verbose --path $GEM_HOME --local --no-deployment $frozen "$@" ; then
- flock /var/lib/gems/gems.lock bundler install --verbose --path $GEM_HOME --no-deployment $frozen "$@"
+ if ! flock /var/lib/gems/gems.lock bundler install --verbose --local --no-deployment $frozen "$@" ; then
+ flock /var/lib/gems/gems.lock bundler install --verbose --no-deployment $frozen "$@"
fi
}
cat <<EOF > /etc/profile.d/paths.sh
export PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/go/bin:/var/lib/gems/bin:$(ls -d /usr/local/node-*)/bin
export GEM_HOME=/var/lib/gems
-export GEM_PATH=/var/lib/gems
export npm_config_cache=/var/lib/npm
export npm_config_cache_min=Infinity
export R_LIBS=/var/lib/Rlibs
exec 2>&1
set -eux -o pipefail
-PGVERSION=9.6
+PGVERSION=11
if ! test -d /var/lib/postgresql/$PGVERSION/main ; then
/usr/lib/postgresql/$PGVERSION/bin/initdb --locale=en_US.UTF-8 -D /var/lib/postgresql/$PGVERSION/main
export PYCMD=python3
-# Need to install the upstream version of pip because the python-pip package
-# shipped with Debian 9 is patched to change behavior in a way that breaks our
-# use case.
-# See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876145
-# When a non-root user attempts to install system packages, it makes the
-# --ignore-installed flag the default (and there is no way to turn it off),
-# this has the effect of making it very hard to share dependencies shared among
-# multiple packages, because it will blindly install the latest version of each
-# dependency requested by each package, even if a compatible package version is
-# already installed.
-if ! pip3 install --no-index --find-links /var/lib/pip pip==9.0.3 ; then
- pip3 install pip==9.0.3
-fi
-
pip_install wheel
cd /usr/src/arvados/sdk/python
projects / arvados.git / commitdiff