--- /dev/null
+Arvados Code of Conduct
+=======================
+
+The Arvados Project is dedicated to providing a harassment-free experience for
+everyone. We do not tolerate harassment of participants in any form.
+
+This code of conduct applies to all Arvados Project spaces both online and off:
+Gitter chat, Redmine issues, wiki, mailing lists, forums, video chats, and any other
+Arvados spaces. Anyone who violates this code of conduct may be sanctioned or
+expelled from these spaces at the discretion of the Arvados Team.
+
+Some Arvados Project spaces may have additional rules in place, which will be
+made clearly available to participants. Participants are responsible for
+knowing and abiding by these rules.
+
+Harassment includes, but is not limited to:
+
+ - Offensive comments related to gender, gender identity and expression, sexual
+orientation, disability, mental illness, neuro(a)typicality, physical
+appearance, body size, age, race, or religion.
+ - Unwelcome comments regarding a person’s lifestyle choices and practices,
+including those related to food, health, parenting, drugs, and employment.
+ - Deliberate misgendering or use of [dead](https://www.quora.com/What-is-deadnaming/answer/Nancy-C-Walker)
+or rejected names.
+ - Gratuitous or off-topic sexual images or behaviour in spaces where they’re not
+appropriate.
+ - Physical contact and simulated physical contact (eg, textual descriptions like
+“\*hug\*” or “\*backrub\*”) without consent or after a request to stop.
+ - Threats of violence.
+ - Incitement of violence towards any individual, including encouraging a person
+to commit suicide or to engage in self-harm.
+ - Deliberate intimidation.
+ - Stalking or following.
+ - Harassing photography or recording, including logging online activity for
+harassment purposes.
+ - Sustained disruption of discussion.
+ - Unwelcome sexual attention.
+ - Pattern of inappropriate social contact, such as requesting/assuming
+inappropriate levels of intimacy with others
+ - Continued one-on-one communication after requests to cease.
+ - Deliberate “outing” of any aspect of a person’s identity without their consent
+except as necessary to protect vulnerable people from intentional abuse.
+ - Publication of non-harassing private communication.
+
+The Arvados Project prioritizes marginalized people’s safety over privileged
+people’s comfort. The Arvados Leadership Team will not act on complaints regarding:
+
+ - ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’
+ - Reasonable communication of boundaries, such as “leave me alone,” “go away,” or
+“I’m not discussing this with you.”
+ - Communicating in a [tone](http://geekfeminism.wikia.com/wiki/Tone_argument)
+you don’t find congenial
+
+Reporting
+---------
+
+If you are being harassed by a member of the Arvados Project, notice that someone
+else is being harassed, or have any other concerns, please contact the Arvados
+Project Team at contact@arvados.org. If person who is harassing
+you is on the team, they will recuse themselves from handling your incident. We
+will respond as promptly as we can.
+
+This code of conduct applies to Arvados Project spaces, but if you are being
+harassed by a member of Arvados Project outside our spaces, we still want to
+know about it. We will take all good-faith reports of harassment by Arvados Project
+members, especially the Arvados Team, seriously. This includes harassment
+outside our spaces and harassment that took place at any point in time. The
+abuse team reserves the right to exclude people from the Arvados Project based on
+their past behavior, including behavior outside Arvados Project spaces and
+behavior towards people who are not in the Arvados Project.
+
+In order to protect volunteers from abuse and burnout, we reserve the right to
+reject any report we believe to have been made in bad faith. Reports intended
+to silence legitimate criticism may be deleted without response.
+
+We will respect confidentiality requests for the purpose of protecting victims
+of abuse. At our discretion, we may publicly name a person about whom we’ve
+received harassment complaints, or privately warn third parties about them, if
+we believe that doing so will increase the safety of Arvados Project members or
+the general public. We will not name harassment victims without their
+affirmative consent.
+
+Consequences
+------------
+
+Participants asked to stop any harassing behavior are expected to comply
+immediately.
+
+If a participant engages in harassing behavior, the Arvados Team may
+take any action they deem appropriate, up to and including expulsion from all
+Arvados Project spaces and identification of the participant as a harasser to other
+Arvados Project members or the general public.
+
+This anti-harassment policy is based on the [example policy from the Geek
+Feminism wiki](http://geekfeminism.wikia.com/wiki/Community_anti-harassment/Policy),
+created by the Geek Feminism community.
--- /dev/null
+[comment]: # (Copyright © The Arvados Authors. All rights reserved.)
+[comment]: # ()
+[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)
+
+# Contributing
+
+Arvados is free software, which means it is free for all to use, learn
+from, and improve. We encourage contributions from the community that
+improve Arvados for everyone. Some examples of contributions are bug
+reports, bug fixes, new features, and scripts or documentation that help
+with using, administering, or installing Arvados. We also love to
+hear about Arvados success stories.
+
+Those interested in contributing should begin by joining the [Arvados community
+channel](https://gitter.im/arvados/community) and telling us about your interest.
+
+Contributers should also create an account at https://dev.arvados.org
+to be able to create and comment on bug tracker issues. The
+Arvados public bug tracker is located at
+https://dev.arvados.org/projects/arvados/issues .
+
+Contributers may also be interested in the [development road map](https://dev.arvados.org/issues/gantt?utf8=%E2%9C%93&set_filter=1&gantt=1&f%5B%5D=project_id&op%5Bproject_id%5D=%3D&v%5Bproject_id%5D%5B%5D=49&f%5B%5D=&zoom=1).
+
+# Development
+
+Git repositories for primary development are located at
+https://git.arvados.org/ and can also be browsed at
+https://dev.arvados.org/projects/arvados/repository . Every push to
+the master branch is also mirrored to Github at
+https://github.com/arvados/arvados .
+
+Visit [Hacking Arvados](https://dev.arvados.org/projects/arvados/wiki/Hacking) for
+detailed information about setting up an Arvados development
+environment, development process, coding standards, and notes about specific components.
+
+If you wish to build the Arvados documentation from a local git clone, see
+[doc/README.textile](doc/README.textile) for instructions.
+
+# Pull requests
+
+The preferred method for making contributions is through Github pull requests.
+
+This is the general contribution process:
+
+1. Fork the Arvados repository using the Github "Fork" button
+2. Clone your fork, make your changes, commit to your fork.
+3. Every commit message must have a DCO sign-off and every file must have a SPDX license (see below).
+4. Add yourself to the [AUTHORS](AUTHORS) file
+5. When your fork is ready, through Github, Create a Pull Request against `arvados:master`
+6. Notify the core team about your pull request through the [Arvados development
+channel](https://gitter.im/arvados/development) or by other means.
+7. A member of the core team will review the pull request. They may have questions or comments, or request changes.
+8. When the contribution is ready, a member of the core team will
+merge the pull request into the master branch, which will
+automatically resolve the pull request.
+
+The Arvados project does not require a contributor agreement in advance, but does require each commit message include a [Developer Certificate of Origin](https://dev.arvados.org/projects/arvados/wiki/Developer_Certificate_Of_Origin). Please ensure *every git commit message* includes `Arvados-DCO-1.1-Signed-off-by`. If you have already made commits without it, fix them with `git commit --amend` or `git rebase`.
+
+The Developer Certificate of Origin line looks like this:
+
+```
+Arvados-DCO-1.1-Signed-off-by: Joe Smith <joe.smith@example.com>
+```
+
+New files must also include `SPDX-License-Identifier` at the top with one of the three Arvados open source licenses. See [COPYING](COPYING) for details.
+
+# Continuous integration
+
+Continuous integration is hosted at https://ci.arvados.org/
+
+Currently, external contributers cannot trigger builds. We are investigating integration with Github pull requests for the future.
+
+[](https://ci.arvados.org/job/run-tests/)
+
+[](https://goreportcard.com/report/github.com/arvados/arvados)
AGPL-3.0: agpl-3.0.txt
Apache-2.0: apache-2.0.txt
CC-BY-SA-3.0: cc-by-sa-3.0.txt
+
+As a general rule, code in the sdk/ directory is licensed Apache-2.0,
+documentation in the doc/ directory is licensed CC-BY-SA-3.0, and
+everything else is licensed AGPL-3.0.
\ No newline at end of file
[comment]: # ()
[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)
-[Arvados](https://arvados.org) is a free software distributed computing platform
-for bioinformatics, data science, and high throughput analysis of massive data
-sets. Arvados supports a variety of cloud, cluster and HPC environments.
+[](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) | [Installing Arvados](https://doc.arvados.org/install/index.html) | [Installing Client SDKs](https://doc.arvados.org/sdk/index.html) | [Report a bug](https://dev.arvados.org/projects/arvados/issues/new) | [Development and Contributing](CONTRIBUTING.md)
-Arvados consists of:
+<img align="right" src="doc/images/dax.png" height="240px">
-* *Keep*: a petabyte-scale content-addressed distributed storage system for managing and
- storing collections of files, accessible via HTTP and FUSE mount.
+[Arvados](https://arvados.org) is an open source platform for
+managing, processing, and sharing genomic and other large scientific
+and biomedical data. With Arvados, bioinformaticians run and scale
+compute-intensive workflows, developers create biomedical
+applications, and IT administrators manage large compute and storage
+resources.
-* *Crunch*: a Docker-based cluster and HPC workflow engine designed providing
- strong versioning, reproducibilty, and provenance of computations.
+The key components of Arvados are:
-* Related services and components including a web workbench for managing files
- and compute jobs, REST APIs, SDKs, and other tools.
+* *Keep*: Keep is the Arvados storage system for managing and storing large
+collections of files. Keep combines content addressing and a
+distributed storage architecture resulting in both high reliability
+and high throughput. Every file stored in Keep can be accurately
+verified every time it is retrieved. Keep supports the creation of
+collections as a flexible way to define data sets without having to
+re-organize or needlessly copy data. Keep works on a wide range of
+underlying filesystems and object stores.
-## Quick start
+* *Crunch*: Crunch is the orchestration system for running [Common Workflow Language](https://www.commonwl.org) workflows. It is
+designed to maintain data provenance and workflow
+reproducibility. Crunch automatically tracks data inputs and outputs
+through Keep and executes workflow processes in Docker containers. In
+a cloud environment, Crunch optimizes costs by scaling compute on demand.
+
+* *Workbench*: The Workbench web application allows users to interactively access
+Arvados functionality. It is especially helpful for querying and
+browsing data, visualizing provenance, and tracking the progress of
+workflows.
+
+* *Command Line tools*: The command line interface (CLI) provides convenient access to Arvados
+functionality in the Arvados platform from the command line.
+
+* *API and SDKs*: Arvados is designed to be integrated with existing infrastructure. All
+the services in Arvados are accessed through a RESTful API. SDKs are
+available for Python, Go, R, Perl, Ruby, and Java.
+
+# Quick start
To try out Arvados on your local workstation, you can use Arvbox, which
provides Arvados components pre-installed in a Docker container (requires
configure Arvbox to be accessible over a network and for other options see
http://doc.arvados.org/install/arvbox.html for details.
-## Documentation
+# Documentation
-Complete documentation, including a User Guide, Installation documentation and
-API documentation is available at http://doc.arvados.org/
+Complete documentation, including the [User Guide](https://doc.arvados.org/user/index.html), [Installation documentation](https://doc.arvados.org/install/index.html), [Administrator documentation](https://doc.arvados.org/admin/index.html) and
+[API documentation](https://doc.arvados.org/api/index.html) is available at http://doc.arvados.org/
If you wish to build the Arvados documentation from a local git clone, see
-doc/README.textile for instructions.
+[doc/README.textile](doc/README.textile) for instructions.
-## Community
+# Community
[](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
-The [arvados community channel](https://gitter.im/arvados/community)
+The [Arvados community channel](https://gitter.im/arvados/community)
channel at [gitter.im](https://gitter.im) is available for live
discussion and support.
-The
-[Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados)
-is a forum for general discussion, questions, and news about Arvados
-development. The
-[Arvados developer mailing list](http://lists.arvados.org/mailman/listinfo/arvados-dev)
-is a forum for more technical discussion, intended for developers and
-contributors to Arvados.
+The [Arvados developement channel](https://gitter.im/arvados/development)
+channel at [gitter.im](https://gitter.im) is used to coordinate development.
-## Development
+The [Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados)
+is used to announce new versions and other news.
-[](https://ci.arvados.org/job/run-tests/)
-[](https://goreportcard.com/report/github.com/arvados/arvados)
+All participants are expected to abide by the [Arvados Code of Conduct](CODE_OF_CONDUCT.md).
-The Arvados public bug tracker is located at https://dev.arvados.org/projects/arvados/issues
+# Reporting bugs
-Continuous integration is hosted at https://ci.arvados.org/
+[Report a bug](https://dev.arvados.org/projects/arvados/issues/new) on [dev.arvados.org](https://dev.arvados.org).
-Instructions for setting up a development environment and working on specific
-components can be found on the
-["Hacking Arvados" page of the Arvados wiki](https://dev.arvados.org/projects/arvados/wiki/Hacking).
+# Development and Contributing
-## Contributing
+See [CONTRIBUTING](CONTRIBUTING.md) for information about Arvados development and how to contribute to the Arvados project.
-When making a pull request, please ensure *every git commit message* includes a one-line [Developer Certificate of Origin](https://dev.arvados.org/projects/arvados/wiki/Developer_Certificate_Of_Origin). If you have already made commits without it, fix them with `git commit --amend` or `git rebase`.
+The [development road map](https://dev.arvados.org/issues/gantt?utf8=%E2%9C%93&set_filter=1&gantt=1&f%5B%5D=project_id&op%5Bproject_id%5D=%3D&v%5Bproject_id%5D%5B%5D=49&f%5B%5D=&zoom=1) outlines some of the project priorities over the next twelve months.
-## Licensing
+# Licensing
-Arvados is Free Software. See COPYING for information about Arvados Free
-Software licenses.
+Arvados is Free Software. See [COPYING](COPYING) for information about the open source licenses used in Arvados.
# star / unstar the current project
def star
- links = Link.where(tail_uuid: current_user.uuid,
+ links = Link.where(owner_uuid: current_user.uuid,
head_uuid: @object.uuid,
link_class: 'star')
helper_method :my_starred_projects
def my_starred_projects user
return if defined?(@starred_projects) && @starred_projects
- links = Link.filter([['tail_uuid', '=', user.uuid],
+ links = Link.filter([['owner_uuid', 'in', ["#{Rails.configuration.ClusterID}-j7d0g-fffffffffffffff", user.uuid]],
['link_class', '=', 'star'],
['head_uuid', 'is_a', 'arvados#group']]).with_count("none").select(%w(head_uuid))
uuids = links.collect { |x| x.head_uuid }
my_child_containers = my_children.map(&:container_uuid).compact.uniq
grandchildren = {}
my_child_containers.each { |c| grandchildren[c] = []} if my_child_containers.any?
- reqs = ContainerRequest.select(cols).where(requesting_container_uuid: my_child_containers).with_count("none").results if my_child_containers.any?
+ reqs = ContainerRequest.select(cols).where(requesting_container_uuid: my_child_containers).order(["requesting_container_uuid", "uuid"]).with_count("none").results if my_child_containers.any?
reqs.each {|cr| grandchildren[cr.requesting_container_uuid] << cr} if reqs
my_children.each do |cr|
<td style="word-break:break-all;">
<% if @my_vm_logins[vm[:uuid]] %>
<% @my_vm_logins[vm[:uuid]].each do |login| %>
- <code>ssh <%= login %>@<%= vm[:hostname] %>.<%= current_uuid_prefix || 'xyzzy' %></code>
+ <code>ssh <%= login %>@<%= vm[:hostname] %></code>
<% end %>
<% end %>
</td>
<% end %>
</div>
</div>
- <p>In order to access virtual machines using SSH, <%= link_to ssh_keys_user_path(current_user) do%> add an SSH key to your account<%end%> and add a section like this to your SSH configuration file ( <i>~/.ssh/config</i>):</p>
- <pre>Host *.<%= current_uuid_prefix || 'xyzzy' %>
- TCPKeepAlive yes
- ServerAliveInterval 60
- ProxyCommand ssh -p2222 turnout@switchyard.<%= current_api_host || 'xyzzy.arvadosapi.com' %> -x -a $SSH_PROXY_FLAGS %h
- </pre>
+
+<p>In order to access virtual machines using SSH, <%= link_to ssh_keys_user_path(current_user) do%>add an SSH key to your account<%end%>.</p>
+
+<%= raw(Rails.configuration.Workbench.SSHHelpPageHTML) %>
SPDX-License-Identifier: AGPL-3.0 %>
-<p>
-Sample <code>~/.ssh/config</code> section:
-</p>
-
-<pre>
-Host *.arvados
- ProxyCommand ssh -p2222 turnout@switchyard.<%= current_api_host.sub(/:.*$/,"") || 'xyzzy.arvadosapi.com' %> -x -a $SSH_PROXY_FLAGS %h
-<% if @objects.first.andand.current_user_logins.andand.first %>
- User <%= @objects.first.current_user_logins.andand.first %>
-<% end %>
-</pre>
-
-<p>
-Sample login command:
-</p>
-
-<pre>
-ssh <%= @objects.first.andand.hostname.andand.sub('.'+current_api_host.sub(/:.*$/,""),'') or 'vm-hostname' %>.arvados
-</pre>
-
-<p>
- See also:
- <%= link_to raw('Arvados Docs → User Guide → SSH access'),
- "#{Rails.configuration.Workbench.ArvadosDocsite}/user/getting_started/ssh-access-unix.html",
- target: "_blank"%>.
-</p>
+<%= raw(Rails.configuration.Workbench.SSHHelpPageHTML) %>
case "$TARGET" in
centos*)
- fpm_depends+=(git bison make automake gcc gcc-c++)
+ fpm_depends+=(git bison make automake gcc gcc-c++ graphviz)
;;
debian* | ubuntu*)
- fpm_depends+=(git g++ bison zlib1g-dev make)
+ fpm_depends+=(git g++ bison zlib1g-dev make graphviz)
;;
esac
# SPDX-License-Identifier: AGPL-3.0
FROM centos:7
-MAINTAINER Ward Vandewege <ward@curoverse.com>
+MAINTAINER Arvados Package Maintainers <packaging@arvados.org>
# Install dependencies.
RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker libattr-devel nss-devel libcurl-devel which tar unzip scl-utils centos-release-scl postgresql-devel python-devel python-setuptools fuse-devel xz-libs git python-virtualenv wget
RUN git clone --depth 1 git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle
# The version of setuptools that comes with CentOS is way too old
-RUN pip install --upgrade setuptools
+RUN pip install --upgrade 'setuptools<45'
ENV WORKSPACE /arvados
CMD ["scl", "enable", "rh-python36", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos7"]
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip unzip python3-venv python3-dev
# Install virtualenv
-RUN /usr/bin/pip install virtualenv
+RUN /usr/bin/pip install 'virtualenv<20'
# Install RVM
ADD generated/mpapis.asc /tmp/
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip unzip python3-venv python3-dev
# Install virtualenv
-RUN /usr/bin/pip install virtualenv
+RUN /usr/bin/pip install 'virtualenv<20'
# Install RVM
ADD generated/mpapis.asc /tmp/
# SPDX-License-Identifier: AGPL-3.0
FROM ubuntu:xenial
-MAINTAINER Ward Vandewege <ward@curoverse.com>
+MAINTAINER Arvados Package Maintainers <packaging@arvados.org>
ENV DEBIAN_FRONTEND noninteractive
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev libgnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip tzdata python3-venv python3-dev
# Install virtualenv
-RUN /usr/bin/pip install virtualenv
+RUN /usr/bin/pip install 'virtualenv<20'
# Install RVM
ADD generated/mpapis.asc /tmp/
# SPDX-License-Identifier: AGPL-3.0
FROM ubuntu:bionic
-MAINTAINER Ward Vandewege <ward@curoverse.com>
+MAINTAINER Arvados Package Maintainers <packaging@arvados.org>
ENV DEBIAN_FRONTEND noninteractive
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-pip libcurl4-gnutls-dev libgnutls28-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip tzdata python3-venv python3-dev
# Install virtualenv
-RUN /usr/bin/pip install virtualenv
+RUN /usr/bin/pip install 'virtualenv<20'
# Install RVM
ADD generated/mpapis.asc /tmp/
# SPDX-License-Identifier: AGPL-3.0
FROM centos:7
-MAINTAINER Ward Vandewege <wvandewege@veritasgenetics.com>
+MAINTAINER Arvados Package Maintainers <packaging@arvados.org>
# Install dependencies.
RUN yum -q -y install scl-utils centos-release-scl which tar wget
# SPDX-License-Identifier: AGPL-3.0
FROM debian:buster
-MAINTAINER Ward Vandewege <wvandewege@veritasgenetics.com>
+MAINTAINER Arvados Package Maintainers <packaging@arvados.org>
ENV DEBIAN_FRONTEND noninteractive
# SPDX-License-Identifier: AGPL-3.0
FROM debian:stretch
-MAINTAINER Ward Vandewege <wvandewege@veritasgenetics.com>
+MAINTAINER Arvados Package Maintainers <packaging@arvados.org>
ENV DEBIAN_FRONTEND noninteractive
# SPDX-License-Identifier: AGPL-3.0
FROM ubuntu:xenial
-MAINTAINER Ward Vandewege <wvandewege@veritasgenetics.com>
+MAINTAINER Arvados Package Maintainers <packaging@arvados.org>
ENV DEBIAN_FRONTEND noninteractive
# SPDX-License-Identifier: AGPL-3.0
FROM ubuntu:bionic
-MAINTAINER Ward Vandewege <wvandewege@veritasgenetics.com>
+MAINTAINER Arvados Package Maintainers <packaging@arvados.org>
ENV DEBIAN_FRONTEND noninteractive
export RAILS_ENV=production
if ! $COMMAND_PREFIX bundle --version >/dev/null; then
- run_and_report "Installing bundle" $COMMAND_PREFIX gem install bundle
+ run_and_report "Installing bundler" $COMMAND_PREFIX gem install bundler --version 1.17.3
fi
run_and_report "Running bundle install" \
# set to --no-cache-dir to disable pip caching
CACHE_FLAG=
-MAINTAINER="Ward Vandewege <wvandewege@veritasgenetics.com>"
-VENDOR="Veritas Genetics, Inc."
+MAINTAINER="Arvados Package Maintainers <packaging@arvados.org>"
+VENDOR="The Arvados Project"
# End of user configuration
rm -rf dist/*
# Get the latest setuptools
- if ! $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools; then
+ if ! $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'; then
echo "Error, unable to upgrade setuptools with"
- echo " $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools"
+ echo " $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'"
exit 1
fi
# filter a useless warning (when building the cwltest package) from the stderr output
fi
echo "pip version: `build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip --version`"
- if ! build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools; then
+ if ! build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'; then
echo "Error, unable to upgrade setuptools with"
- echo " build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools"
+ echo " build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'"
exit 1
fi
echo "setuptools version: `build/usr/share/$python/dist/$PYTHON_PKG/bin/$python -c 'import setuptools; print(setuptools.__version__)'`"
lib/dispatchcloud/scheduler
lib/dispatchcloud/ssh_executor
lib/dispatchcloud/worker
+lib/mount
lib/service
services/api
services/arv-git-httpd
. "$VENVDIR/bin/activate"
# Needed for run_test_server.py which is used by certain (non-Python) tests.
- pip install --no-cache-dir PyYAML future \
- || fatal "pip install PyYAML failed"
+ pip install --no-cache-dir PyYAML future httplib2 \
+ || fatal "`pip install PyYAML future httplib2` failed"
# Preinstall libcloud if using a fork; otherwise nodemanager "pip
# install" won't pick it up by default.
do_install sdk/cli
do_install sdk/perl
do_install sdk/python pip
+ do_install sdk/python pip3
do_install sdk/ruby
do_install services/api
do_install services/arv-git-httpd go
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+all:
+ @printf "*** note *** due to an xgo limitation, this only works when the working tree is in GOPATH\n\n"
+ go mod download
+ docker build --tag=cgofuse --build-arg=http_proxy="$(http_proxy)" --build-arg=https_proxy="$(https_proxy)" https://github.com/arvados/cgofuse.git
+ go run github.com/karalabe/xgo --image=cgofuse --targets=linux/amd64,linux/386,darwin/amd64,darwin/386,windows/amd64,windows/386 .
+ install arvados-* "$(GOPATH)"/bin/
+ rm --interactive=never arvados-*
"git.arvados.org/arvados.git/lib/cli"
"git.arvados.org/arvados.git/lib/cmd"
+ "git.arvados.org/arvados.git/lib/mount"
)
var (
"user": cli.APICall,
"virtual_machine": cli.APICall,
"workflow": cli.APICall,
+
+ "mount": mount.Command,
})
)
Type=notify
EnvironmentFile=-/etc/arvados/environment
ExecStart=/usr/bin/arvados-controller
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=1
Type=notify
EnvironmentFile=-/etc/arvados/environment
ExecStart=/usr/bin/arvados-dispatch-cloud
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=1
- admin/metrics.html.textile.liquid
- admin/health-checks.html.textile.liquid
- admin/management-token.html.textile.liquid
- - Cloud:
- - admin/storage-classes.html.textile.liquid
- - admin/spot-instances.html.textile.liquid
- - admin/cloudtest.html.textile.liquid
- Data Management:
- admin/collection-versioning.html.textile.liquid
- admin/collection-managed-properties.html.textile.liquid
- admin/keep-balance.html.textile.liquid
- admin/controlling-container-reuse.html.textile.liquid
- admin/logs-table-management.html.textile.liquid
+ - admin/workbench2-vocabulary.html.textile.liquid
+ - admin/storage-classes.html.textile.liquid
+ - Cloud:
+ - admin/spot-instances.html.textile.liquid
+ - admin/cloudtest.html.textile.liquid
- Other:
- install/migrate-docker19.html.textile.liquid
- admin/upgrade-crunch2.html.textile.liquid
- - admin/workbench2-vocabulary.html.textile.liquid
installguide:
- Overview:
- install/index.html.textile.liquid
<script>
function dismissAnnotateNotify() {
- window.sessionStorage.setItem("dismiss-annotate-notify", "true");
+ window.localStorage.setItem("dismiss-annotate-notify", "true");
$('#annotate-notify').attr('style', "display: none;");
}
- if (window.sessionStorage.getItem("dismiss-annotate-notify") === "true") {
+ if (window.localStorage.getItem("dismiss-annotate-notify") === "true") {
dismissAnnotateNotify();
} else {
$('#annotate-notify').attr('style', "display: inline-block;");
This page describes how to enable and configure the collection versioning feature on the API server.
-h3. API Server configuration
+h3. Configuration
-There are 2 configuration settings that control this feature, both go on the @application.yml@ file.
+There are 2 configuration settings in the @Collections@ section of @config.yml@ that control this feature.
-h4. Setting: @collection_versioning@ (Boolean. Default: false)
+<pre>
+ Collections:
+ # If true, enable collection versioning.
+ # When a collection's preserve_version field is true or the current version
+ # is older than the amount of seconds defined on PreserveVersionIfIdle,
+ # a snapshot of the collection's previous state is created and linked to
+ # the current collection.
+ CollectionVersioning: false
-If @true@, collection versioning is enabled, meaning that new version records can be created. Note that if you set @collection_versioning@ to @false@ after being enabled, old versions will still be accessible, but further changes will not be versioned.
+ # This setting control the auto-save aspect of collection versioning, and can be set to:
+ # 0s = auto-create a new version on every update.
+ # -1s = never auto-create new versions.
+ # > 0s = auto-create a new version when older than the specified number of seconds.
+ PreserveVersionIfIdle: -1s
+</pre>
-h4. Setting: @preserve_version_if_idle@ (Numeric. Default: -1)
-
-This setting control the auto-save aspect of collection versioning, and can be set to:
-* @-1@: Never auto-save versions. Only save versions when the client ask for it by setting @preserve_version@ to @true@ on any given collection.
-* @0@: Preserve all versions every time a collection gets a versionable update.
-* @N@ (being N > 0): Preserve version when a collection gets a versionable update after a period of at least N seconds since the last time it was modified.
+Note that if you set @collection_versioning@ to @false@ after being enabled, old versions will still be accessible, but further changes will not be versioned.
h3. Using collection versioning
-"Discussed in the user guide":{{site.baseurl}}/user/topics/collection-versioning.html
\ No newline at end of file
+"Discussed in the user guide":{{site.baseurl}}/user/topics/collection-versioning.html
---
layout: default
navsection: installguide
-title: Migrating Configuration from v1.4 to v1.5
+title: Migrating Configuration from v1.4 to v2.0
...
{% comment %}
{% endcomment %}
{% include 'notebox_begin_warning' %}
-_New installations of Arvados 1.5+ can skip this section_
+_New installations of Arvados 2.0+ can skip this section_
{% include 'notebox_end' %}
-Arvados 1.5 migrates to a centralized configuration file for all components. The centralized Arvados configuration is @/etc/arvados/config.yml@. Components that support the new centralized configuration are listed below. During the migration period, legacy configuration files are still loaded and take precedence over the centralized configuration file.
+Arvados 2.0 migrates to a centralized configuration file for all components. The centralized Arvados configuration is @/etc/arvados/config.yml@. Components that support the new centralized configuration are listed below. During the migration period, legacy configuration files are still loaded and take precedence over the centralized configuration file.
h2. API server
---
layout: default
navsection: admin
-title: Controlling container reuse
+title: Preventing container reuse
...
{% comment %}
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-This page describes how an admin can control container reuse using the @arv@ command. This can be utilized to avoid reusing a completed container without disabling reuse for the corresponding steps in affected workflows. For example, if a container exited successfully but produced bad output, it may not be feasible to update the workflow immediately. Meanwhile, changing the state of the container from @Complete@ to @Cancelled@ will prevent it from being used in subsequent workflows.
+Sometimes a container exited successfully but produced bad output, and re-running the workflow will cause it to re-use the bad container instead of running a new container. One way to deal with this is to re-run the entire workflow with reuse disable. Another way is for the workflow author to tweak the input data or workflow so that on re-run it produces a distinct container request. However, for large or complex workflows both these options may be impractical.
-If a container is in the @Complete@ state, the following @arv@ command will change its state to @Cancelled@, where @xxxxx-xxxxx-xxxxxxxxxxxxxxx@ is the @UUID@ of the container:
+To prevent an individual container from being reused in later workflows, an admin can manually change the state of the bad container record from @Complete@ to @Cancelled@. The following @arv@ command demonstrates how change a container state to @Cancelled@, where @xxxxx-xxxxx-xxxxxxxxxxxxxxx@ is the @UUID@ of the container:
<pre>arv container update -u xxxxx-xxxxx-xxxxxxxxxxxxxxx -c '{"state":"Cancelled"}'</pre>
-
-Use the following command to list all containers that exited with 0 and were then cancelled:
-
-<pre>arv container list --filters='[["state", "=", "Cancelled"], ["exit_code", "=", 0]]'</pre>See the "arv CLI tool overview":{{site.baseurl}}/sdk/cli/index.html for more details about using the @arv@ command.
As a result, this table grows indefinitely, even on sites where policy does not require an audit log; making backups, migrations, and upgrades unnecessarily slow and painful.
-h3. API Server configuration
+h3. Configuration
-To solve the problem mentioned above, the API server offers the possibility to limit the amount of log information stored on the table:
+To solve the problem mentioned above, the @AuditLogs@ section of @config.yml@ offers several options to limit the amount of log information stored on the table:
<pre>
-# Attributes to suppress in events and audit logs. Notably,
-# specifying ["manifest_text"] here typically makes the database
-# smaller and faster.
-#
-# Warning: Using any non-empty value here can have undesirable side
-# effects for any client or component that relies on event logs.
-# Use at your own risk.
-unlogged_attributes: []
+ AuditLogs:
+ # Time to keep audit logs. (An audit log is a row added
+ # to the "logs" table in the PostgreSQL database each time an
+ # Arvados object is created, modified, or deleted.)
+ #
+ # Currently, websocket event notifications rely on audit logs, so
+ # this should not be set lower than 5 minutes.
+ MaxAge: 336h
+
+ # Maximum number of log rows to delete in a single SQL transaction,
+ # to prevent surprises and avoid bad database behavior
+ # (especially the first time the cleanup job runs on an existing
+ # cluster with a huge backlog) a maximum number of rows to
+ # delete in a single transaction.
+ #
+ # If MaxDeleteBatch is 0, log entries will never be
+ # deleted by Arvados. Cleanup can be done by an external process
+ # without affecting any Arvados system processes, as long as very
+ # recent (<5 minutes old) logs are not deleted.
+ #
+ # 100000 is a reasonable batch size for most sites.
+ MaxDeleteBatch: 0
+
+ # Attributes to suppress in events and audit logs. Notably,
+ # specifying {"manifest_text": {}} here typically makes the database
+ # smaller and faster.
+ #
+ # Warning: Using any non-empty value here can have undesirable side
+ # effects for any client or component that relies on event logs.
+ # Use at your own risk.
+ UnloggedAttributes: {}
</pre>
-The above setting affects all events being logged, independently of how much time they will be kept on the database.
-
-<pre>
-# Time to keep audit logs (a row in the log table added each time an
-# Arvados object is created, modified, or deleted) in the PostgreSQL
-# database. Currently, websocket event notifications rely on audit
-# logs, so this should not be set lower than 300 (5 minutes).
-max_audit_log_age: 1209600
-</pre>
-
-...and to prevent surprises and avoid bad database behavior (especially the first time the cleanup job runs on an existing cluster with a huge backlog) a maximum number of rows to delete in a single transaction.
-
-<pre>
-# Maximum number of log rows to delete in a single SQL transaction.
-#
-# If max_audit_log_delete_batch is 0, log entries will never be
-# deleted by Arvados. Cleanup can be done by an external process
-# without affecting any Arvados system processes, as long as very
-# recent (<5 minutes old) logs are not deleted.
-#
-# 100000 is a reasonable batch size for most sites.
-max_audit_log_delete_batch: 0
-</pre>
-
-This feature works when both settings are non-zero, periodically dispatching a background task that deletes all log rows older than @max_audit_log_age@.
-The events being cleaned up by this process don't include job/container stderr logs (they're handled by the existing @delete job/container logs@ rake tasks)
h3. Additional consideration
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-Some Arvados services publish Prometheus/OpenMetrics-compatible metrics at @/metrics@, and some provide additional runtime status at @/status.json@. Metrics can help you understand how components perform under load, find performance bottlenecks, and detect and diagnose problems.
+Some Arvados services publish Prometheus/OpenMetrics-compatible metrics at @/metrics@. Metrics can help you understand how components perform under load, find performance bottlenecks, and detect and diagnose problems.
To access metrics endpoints, services must be configured with a "management token":management-token.html. When accessing a metrics endpoint, prefix the management token with @"Bearer "@ and supply it in the @Authorization@ request header.
-<pre>curl -sfH "Authorization: Bearer your_management_token_goes_here" "https://0.0.0.0:25107/status.json"
+<pre>curl -sfH "Authorization: Bearer your_management_token_goes_here" "https://0.0.0.0:25107/metrics"
</pre>
-h2. Keep-web
+The plain text export format includes "help" messages with a description of each reported metric.
-Keep-web exports metrics at @/metrics@ -- e.g., @https://collections.zzzzz.arvadosapi.com/metrics@.
+When configuring Prometheus, use a @bearer_token@ or @bearer_token_file@ option to authenticate requests.
-table(table table-bordered table-condensed).
-|_. Name|_. Type|_. Description|
-|request_duration_seconds|summary|elapsed time between receiving a request and sending the last byte of the response body (segmented by HTTP request method and response status code)|
-|time_to_status_seconds|summary|elapsed time between receiving a request and sending the HTTP response status code (segmented by HTTP request method and response status code)|
-
-Metrics in the @arvados_keepweb_collectioncache@ namespace report keep-web's internal cache of Arvados collection metadata.
-
-table(table table-bordered table-condensed).
-|_. Name|_. Type|_. Description|
-|arvados_keepweb_collectioncache_requests|counter|cache lookups|
-|arvados_keepweb_collectioncache_api_calls|counter|outgoing API calls|
-|arvados_keepweb_collectioncache_permission_hits|counter|collection-to-permission cache hits|
-|arvados_keepweb_collectioncache_pdh_hits|counter|UUID-to-PDH cache hits|
-|arvados_keepweb_collectioncache_hits|counter|PDH-to-manifest cache hits|
-|arvados_keepweb_collectioncache_cached_manifests|gauge|number of collections in the cache|
-|arvados_keepweb_collectioncache_cached_manifest_bytes|gauge|memory consumed by cached collection manifests|
-
-h2. Keepstore
-
-Keepstore exports metrics at @/status.json@ -- e.g., @http://keep0.zzzzz.arvadosapi.com:25107/status.json@.
-
-h3. Root
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Volumes| array of "volumeStatusEnt":#volumeStatusEnt ||
-|BufferPool| "PoolStatus":#PoolStatus ||
-|PullQueue| "WorkQueueStatus":#WorkQueueStatus ||
-|TrashQueue| "WorkQueueStatus":#WorkQueueStatus ||
-|RequestsCurrent| int ||
-|RequestsMax| int ||
-|Version| string ||
-
-h3(#volumeStatusEnt). volumeStatusEnt
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Label| string||
-|Status| "VolumeStatus":#VolumeStatus ||
-|VolumeStats| "ioStats":#ioStats ||
-
-h3(#VolumeStatus). VolumeStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|MountPoint| string||
-|DeviceNum| uint64||
-|BytesFree| uint64||
-|BytesUsed| uint64||
-
-h3(#ioStats). ioStats
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|Errors| uint64||
-|Ops| uint64||
-|CompareOps| uint64||
-|GetOps| uint64||
-|PutOps| uint64||
-|TouchOps| uint64||
-|InBytes| uint64||
-|OutBytes| uint64||
-
-h3(#PoolStatus). PoolStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|BytesAllocatedCumulative| uint64||
-|BuffersMax| int||
-|BuffersInUse| int||
-
-h3(#WorkQueueStatus). WorkQueueStatus
-
-table(table table-bordered table-condensed).
-|_. Attribute|_. Type|_. Description|
-|InProgress| int||
-|Queued| int||
-
-h3. Example response
-
-<pre>
-{
- "Volumes": [
- {
- "Label": "[UnixVolume /var/lib/arvados/keep0]",
- "Status": {
- "MountPoint": "/var/lib/arvados/keep0",
- "DeviceNum": 65029,
- "BytesFree": 222532972544,
- "BytesUsed": 435456679936
- },
- "InternalStats": {
- "Errors": 0,
- "InBytes": 1111,
- "OutBytes": 0,
- "OpenOps": 1,
- "StatOps": 4,
- "FlockOps": 0,
- "UtimesOps": 0,
- "CreateOps": 0,
- "RenameOps": 0,
- "UnlinkOps": 0,
- "ReaddirOps": 0
- }
- }
- ],
- "BufferPool": {
- "BytesAllocatedCumulative": 67108864,
- "BuffersMax": 20,
- "BuffersInUse": 0
- },
- "PullQueue": {
- "InProgress": 0,
- "Queued": 0
- },
- "TrashQueue": {
- "InProgress": 0,
- "Queued": 0
- },
- "RequestsCurrent": 1,
- "RequestsMax": 40,
- "Version": "dev"
-}
+<pre>scrape_configs:
+ - job_name: keepstore
+ bearer_token: your_management_token_goes_here
+ static_configs:
+ - targets:
+ - "keep0.ClusterID.example.com:25107"
</pre>
-h2. Keep-balance
-
-Keep-balance exports metrics at @/metrics@ -- e.g., @http://keep.zzzzz.arvadosapi.com:9005/metrics@.
-
-table(table table-bordered table-condensed).
-|_. Name|_. Type|_. Description|
-|arvados_keep_total_{replicas,blocks,bytes}|gauge|stored data (stored in backend volumes, whether referenced or not)|
-|arvados_keep_garbage_{replicas,blocks,bytes}|gauge|garbage data (unreferenced, and old enough to trash)|
-|arvados_keep_transient_{replicas,blocks,bytes}|gauge|transient data (unreferenced, but too new to trash)|
-|arvados_keep_overreplicated_{replicas,blocks,bytes}|gauge|overreplicated data (more replicas exist than are needed)|
-|arvados_keep_underreplicated_{replicas,blocks,bytes}|gauge|underreplicated data (fewer replicas exist than are needed)|
-|arvados_keep_lost_{replicas,blocks,bytes}|gauge|lost data (referenced by collections, but not found on any backend volume)|
-|arvados_keep_dedup_block_ratio|gauge|deduplication ratio (block references in collections ÷ distinct blocks referenced)|
-|arvados_keep_dedup_byte_ratio|gauge|deduplication ratio (block references in collections ÷ distinct blocks referenced, weighted by block size)|
-|arvados_keepbalance_get_state_seconds|summary|time to get all collections and keepstore volume indexes for one iteration|
-|arvados_keepbalance_changeset_compute_seconds|summary|time to compute changesets for one iteration|
-|arvados_keepbalance_send_pull_list_seconds|summary|time to send pull lists to all keepstore servers for one iteration|
-|arvados_keepbalance_send_trash_list_seconds|summary|time to send trash lists to all keepstore servers for one iteration|
-|arvados_keepbalance_sweep_seconds|summary|time to complete one iteration|
-
-Each @arvados_keep_@ storage state statistic above is presented as a set of three metrics:
-
-table(table table-bordered table-condensed).
-|*_blocks|distinct block hashes|
-|*_bytes|bytes stored on backend volumes|
-|*_replicas|objects/files stored on backend volumes|
+table(table table-bordered table-condensed table-hover).
+|_. Component|_. Metrics endpoint|
+|arvados-api-server||
+|arvados-controller|✓|
+|arvados-dispatch-cloud|✓|
+|arvados-git-httpd||
+|arvados-node-manager||
+|arvados-ws||
+|composer||
+|keepproxy||
+|keepstore|✓|
+|keep-balance|✓|
+|keep-web|✓|
+|sso-provider||
+|workbench1||
+|workbench2||
h2. Node manager
-The node manager status end point provides a snapshot of internal status at the time of the most recent wishlist update.
+The node manager does not export prometheus-style metrics, but its @/status.json@ endpoint provides a snapshot of internal status at the time of the most recent wishlist update.
+
+<pre>curl -sfH "Authorization: Bearer your_management_token_goes_here" "http://0.0.0.0:8989/status.json"
+</pre>
table(table table-bordered table-condensed).
|_. Attribute|_. Type|_. Description|
The storage classes for each volume are set in the per-volume "keepstore configuration":{{site.baseurl}}/install/install-keepstore.html
<pre>
-Volumes:
- - ... Volume configuration ...
- #
- # If no storage classes are specified, will use [default]
- #
- StorageClasses: null
-
- - ... Volume configuration ...
- #
- # Specify this volume is in the "archival" storage class.
- #
- StorageClasses: [archival]
-
+ Volumes:
+ ClusterID-nyw5e-000000000000000:
+ # This volume is in the "default" storage class.
+ StorageClasses:
+ default: true
+ ClusterID-nyw5e-000000000000001:
+ # Specify this volume is in the "archival" storage class.
+ StorageClasses:
+ archival: true
</pre>
Names of storage classes are internal to the cluster and decided by the administrator. Aside from "default", Arvados currently does not define any standard storage class names.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-The "containers" API is the recommended way to submit compute work to Arvados. It supersedes the "jobs" API, which is end-of-life as of Arvados 1.5.
+The "containers" API is the recommended way to submit compute work to Arvados. It supersedes the "jobs" API, which is end-of-life in Arvados 2.0.
h2. Benefits over the "jobs" API
TODO: extract this information based on git commit messages and generate changelogs / release notes automatically.
{% endcomment %}
-h3(#master). development master (as of 2020-01-16)
+<notextile>
+<div class="releasenotes">
+</notextile>
-h4. "/" prohibited in collection and project names
+h2(#master). development master (as of 2020-02-07)
-(issue "#15836":https://dev.arvados.org/issues/15836) By default, Arvados now rejects new names containing the @/@ character when creating or renaming collections and projects. Previously, these names were permitted, but the resulting objects were invisible in the WebDAV "home" tree. If you prefer, you can restore the previous behavior, and optionally configure a substitution string to make the affected objects accessible via WebDAV. See @ForwardSlashNameSubstitution@ in the "configuration reference":config.html.
+"Upgrading from 2.0.0":#v2_0_0
-h4. Migrating to centralized config.yml
+None in current development master.
-See "Migrating Configuration":config-migration.html for notes on migrating legacy per-component configuration files to the new centralized @/etc/arvados/config.yml@. To ensure a smooth transition, the per-component config files continue to be read, and take precedence over the centralized configuration.
+h2(#v2_0_0). v2.0.0 (2020-02-07)
-h4. Cloud installations only: node manager replaced by arvados-dispatch-cloud
-
-Node manager is deprecated and replaced by @arvados-dispatch-cloud@. No automated config migration is available. Follow the instructions to "install the cloud dispatcher":../install/install-dispatch-cloud.html
+"Upgrading from 1.4":#v1_4_1
-*Only one dispatch process should be running at a time.* If you are migrating a system that currently runs Node manager and @crunch-dispatch-slurm@, it is safest to remove the @crunch-dispatch-slurm@ service entirely before installing @arvados-dispatch-cloud@.
-
-<notextile>
-<pre><code>~$ <span class="userinput">sudo systemctl --now disable crunch-dispatch-slurm</span>
-~$ <span class="userinput">sudo apt-get remove crunch-dispatch-slurm</span>
-</code></pre>
-</notextile>
+Arvados 2.0 is a major upgrade, with many changes. Please read these upgrade notes carefully before you begin.
-h4. Delete "keep_services" records
+h3. Migrating to centralized config.yml
-After all keepproxy and keepstore configurations have been migrated to the centralized configuration file (see below), all keep_services records you added manually during installation should be removed. System logs from keepstore and keepproxy at startup, as well as the output of @arvados-server config-check@, will remind you to do this.
+See "Migrating Configuration":config-migration.html for notes on migrating legacy per-component configuration files to the new centralized @/etc/arvados/config.yml@.
-<notextile><pre><code>$ export ARVADOS_API_HOST=...
-$ export ARVADOS_API_TOKEN=...
-$ arv --format=uuid keep_service list | xargs -n1 arv keep_service delete --uuid
-</code></pre></notextile>
+To ensure a smooth transition, the per-component config files continue to be read, and take precedence over the centralized configuration. Your cluster should continue to function after upgrade but before doing the full configuration migration. However, several services (keepstore, keep-web, keepproxy) require a minimal `/etc/arvados/config.yml` to start:
-Once these old records are removed, @arv keep_service list@ will instead return the services listed under Services/Keepstore/InternalURLs and Services/Keepproxy/ExternalURL in your centralized configuration file.
+<pre>
+Clusters:
+ zzzzz:
+ Services:
+ Controller:
+ ExternalURL: "https://zzzzz.example.com"
+</pre>
-h4. Keep-balance configuration migration
+h3. Keep-balance configuration migration
(feature "#14714":https://dev.arvados.org/issues/14714 ) The keep-balance service can now be configured using the centralized configuration file at @/etc/arvados/config.yml@. The following command line and configuration options have changed.
Please see the "config migration guide":{{site.baseurl}}/admin/config-migration.html and "keep-balance install guide":{{site.baseurl}}/install/install-keep-balance.html for more details.
-h4. Arv-git-httpd configuration migration
+h3. Arv-git-httpd configuration migration
(feature "#14712":https://dev.arvados.org/issues/14712 ) The arv-git-httpd package can now be configured using the centralized configuration file at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available. Please see "arv-git-httpd's config migration guide":{{site.baseurl}}/admin/config-migration.html#arv-git-httpd for more details.
-h4. Keepstore and keep-web configuration migration
+h3. Keepstore and keep-web configuration migration
keepstore and keep-web no longer support configuration via (previously deprecated) command line configuration flags and environment variables.
keepstore now supports the legacy @keepstore.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check the "keepstore config migration notes":{{site.baseurl}}/admin/config-migration.html#keepstore and "keepstore install guide":{{site.baseurl}}/install/install-keepstore.html for more details.
-h4. Keepproxy configuration migration
+h3. Keepproxy configuration migration
(feature "#14715":https://dev.arvados.org/issues/14715 ) Keepproxy can now be configured using the centralized config at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available and the @DisableGet@, @DisablePut@, and @PIDFile@ configuration options are no longer supported. If you are still using the legacy config and @DisableGet@ or @DisablePut@ are set to true or @PIDFile@ has a value, keepproxy will produce an error and fail to start. Please see "keepproxy's config migration guide":{{site.baseurl}}/admin/config-migration.html#keepproxy for more details.
-h4. Jobs API is read-only
-
-(task "#15133":https://dev.arvados.org/issues/15133 ) The legacy 'jobs' API is now read-only. It has been superceded since Arvados 1.1 by containers / container_requests (aka crunch v2). Arvados installations since the end of 2017 (v1.1.0) have probably only used containers, and are unaffected by this change.
-
-So that older Arvados sites don't lose access to legacy records, the API has been converted to read-only. Creating and updating jobs (and related types job_task, pipeline_template and pipeline_instance) is disabled and much of the business logic related has been removed, along with various other code specific to the jobs API. Specifically, the following programs associated with the jobs API have been removed: @crunch-dispatch.rb@, @crunch-job@, @crunchrunner@, @arv-run-pipeline-instance@, @arv-run@.
+h3. Delete "keep_services" records
-h4. No longer stripping ':' from strings in serialized database columns
+After all keepproxy and keepstore configurations have been migrated to the centralized configuration file, all keep_services records you added manually during installation should be removed. System logs from keepstore and keepproxy at startup, as well as the output of @arvados-server config-check@, will remind you to do this.
-(bug "#15311":https://dev.arvados.org/issues/15311 ) Strings read from serialized columns in the database with a leading ':' would have the ':' stripped after loading the record. This behavior existed due to legacy serialization behavior which stored Ruby symbols with a leading ':'. Unfortunately this corrupted fields where the leading ":" was intentional. This behavior has been removed.
+<notextile><pre><code>$ export ARVADOS_API_HOST=...
+$ export ARVADOS_API_TOKEN=...
+$ arv --format=uuid keep_service list | xargs -n1 arv keep_service delete --uuid
+</code></pre></notextile>
-You can test if any records in your database are affected by going to the API server directory and running @bundle exec rake symbols:check@. This will report which records contain fields with a leading ':' that would previously have been stripped. If there are records to be updated, you can update the database using @bundle exec rake symbols:stringify@.
+Once these old records are removed, @arv keep_service list@ will instead return the services listed under Services/Keepstore/InternalURLs and Services/Keepproxy/ExternalURL in your centralized configuration file.
-h4. Enabling Postgres trigram indexes
+h3. Enabling Postgres trigram indexes
Feature "#15106":https://dev.arvados.org/issues/15106 improves the speed and functionality of full text search by introducing trigram indexes on text searchable database columns via a migration. Prior to updating, you must first install the postgresql-contrib package on your system and subsequently run the <code class="userprint">CREATE EXTENSION pg_trgm</code> SQL command on the arvados_production database as a postgres superuser.
Subsequently, the <code class="userinput">psql -d 'arvados_production' -c '\dx'</code> command will display the installed extensions for the arvados_production database. This list should now contain @pg_trgm@.
+h3. New Workbench 2
+
+Workbench 2 is now ready for regular use. Follow the instructions to "install workbench 2":../install/install-workbench2-app.html
+
+h3. New property vocabulary format for Workbench2
-h3(#v1_4_1). v1.4.1 (2019-09-20)
+(feature "#14151":https://dev.arvados.org/issues/14151) Workbench2 supports a new vocabulary format and it isn't compatible with the previous one, please read the "workbench2 vocabulary format admin page":{{site.baseurl}}/admin/workbench2-vocabulary.html for more information.
-h4. Centos7 Python 3 dependency upgraded to rh-python36
+h3. Cloud installations only: node manager replaced by arvados-dispatch-cloud
+
+Node manager is deprecated and replaced by @arvados-dispatch-cloud@. No automated config migration is available. Follow the instructions to "install the cloud dispatcher":../install/install-dispatch-cloud.html
+
+*Only one dispatch process should be running at a time.* If you are migrating a system that currently runs Node manager and @crunch-dispatch-slurm@, it is safest to remove the @crunch-dispatch-slurm@ service entirely before installing @arvados-dispatch-cloud@.
+
+<notextile>
+<pre><code>~$ <span class="userinput">sudo systemctl --now disable crunch-dispatch-slurm</span>
+~$ <span class="userinput">sudo apt-get remove crunch-dispatch-slurm</span>
+</code></pre>
+</notextile>
+
+h3. Jobs API is read-only
+
+(task "#15133":https://dev.arvados.org/issues/15133 ) The legacy 'jobs' API is now read-only. It has been superceded since Arvados 1.1 by containers / container_requests (aka crunch v2). Arvados installations since the end of 2017 (v1.1.0) have probably only used containers, and are unaffected by this change.
+
+So that older Arvados sites don't lose access to legacy records, the API has been converted to read-only. Creating and updating jobs (and related types job_task, pipeline_template and pipeline_instance) is disabled and much of the business logic related has been removed, along with various other code specific to the jobs API. Specifically, the following programs associated with the jobs API have been removed: @crunch-dispatch.rb@, @crunch-job@, @crunchrunner@, @arv-run-pipeline-instance@, @arv-run@.
+
+h3. "/" prohibited in collection and project names
+
+(issue "#15836":https://dev.arvados.org/issues/15836) By default, Arvados now rejects new names containing the @/@ character when creating or renaming collections and projects. Previously, these names were permitted, but the resulting objects were invisible in the WebDAV "home" tree. If you prefer, you can restore the previous behavior, and optionally configure a substitution string to make the affected objects accessible via WebDAV. See @ForwardSlashNameSubstitution@ in the "configuration reference":config.html.
+
+h3. No longer stripping ':' from strings in serialized database columns
+
+(bug "#15311":https://dev.arvados.org/issues/15311 ) Strings read from serialized columns in the database with a leading ':' would have the ':' stripped after loading the record. This behavior existed due to legacy serialization behavior which stored Ruby symbols with a leading ':'. Unfortunately this corrupted fields where the leading ":" was intentional. This behavior has been removed.
+
+You can test if any records in your database are affected by going to the API server directory and running @bundle exec rake symbols:check@. This will report which records contain fields with a leading ':' that would previously have been stripped. If there are records to be updated, you can update the database using @bundle exec rake symbols:stringify@.
+
+h3. Scoped tokens should use PATCH for updates
+
+The API server accepts both PUT and PATCH for updates, but they will be normalized to PATCH by arvados-controller. Scoped tokens should be updated accordingly.
+
+
+
+h2(#v1_4_1). v1.4.1 (2019-09-20)
+
+"Upgrading from 1.4.0":#v1_4_0
+
+h3. Centos7 Python 3 dependency upgraded to rh-python36
The Python 3 dependency for Centos7 Arvados packages was upgraded from rh-python35 to rh-python36.
-h3(#v1_4_0). v1.4.0 (2019-06-05)
+h2(#v1_4_0). v1.4.0 (2019-06-05)
+
+"Upgrading from 1.3.3":#v1_3_3
-h4. Populating the new file_count and file_size_total columns on the collections table
+h3. Populating the new file_count and file_size_total columns on the collections table
As part of story "#14484":https://dev.arvados.org/issues/14484, two new columns were added to the collections table in a database migration. If your installation has a large collections table, this migration may take some time. We've seen it take ~5 minutes on an installation with 250k collections, but your mileage may vary.
The new columns are initialized with a zero value. In order to populate them, it is necessary to run a script called <code class="userinput">populate-file-info-columns-in-collections.rb</code> from the scripts directory of the API server. This can be done out of band, ideally directly after the API server has been upgraded to v1.4.0.
-h4. Stricter collection manifest validation on the API server
+h3. Stricter collection manifest validation on the API server
As a consequence of "#14482":https://dev.arvados.org/issues/14482, the Ruby SDK does a more rigorous collection manifest validation. Collections created after 2015-05 are unlikely to be invalid, however you may check for invalid manifests using the script below.
The script will return a final report enumerating any invalid collection by UUID, with its creation date and error message so you can take the proper correction measures, if needed.
-h4. Python packaging change
+h3. Python packaging change
As part of story "#9945":https://dev.arvados.org/issues/9945, the distribution packaging (deb/rpm) of our Python packages has changed. These packages now include a built-in virtualenv to reduce dependencies on system packages. We have also stopped packaging and publishing backports for all the Python dependencies of our packages, as they are no longer needed.
</pre>
</notextile>
-h4. python-arvados-cwl-runner deb/rpm package now conflicts with python-cwltool deb/rpm package
+h3. python-arvados-cwl-runner deb/rpm package now conflicts with python-cwltool deb/rpm package
As part of story "#9945":https://dev.arvados.org/issues/9945, the distribution packaging (deb/rpm) of our Python packages has changed. The python-arvados-cwl-runner package now includes a version of cwltool. If present, the python-cwltool and cwltool distribution packages will need to be uninstalled before the python-arvados-cwl-runner deb or rpm package can be installed.
-h4. Centos7 Python 3 dependency upgraded to rh-python35
+h3. Centos7 Python 3 dependency upgraded to rh-python35
As part of story "#9945":https://dev.arvados.org/issues/9945, the Python 3 dependency for Centos7 Arvados packages was upgraded from SCL python33 to rh-python35.
-h4. Centos7 package for libpam-arvados depends on the python-pam package, which is available from EPEL
+h3. Centos7 package for libpam-arvados depends on the python-pam package, which is available from EPEL
As part of story "#9945":https://dev.arvados.org/issues/9945, it was discovered that the Centos7 package for libpam-arvados was missing a dependency on the python-pam package, which is available from the EPEL repository. The dependency has been added to the libpam-arvados package. This means that going forward, the EPEL repository will need to be enabled to install libpam-arvados on Centos7.
-h4. New configuration
+h3. New configuration
Arvados is migrating to a centralized configuration file for all components. During the migration, legacy configuration files will continue to be loaded. See "Migrating Configuration":config-migration.html for details.
-h3(#v1_3_3). v1.3.3 (2019-05-14)
+h2(#v1_3_3). v1.3.3 (2019-05-14)
+
+"Upgrading from 1.3.0":#v1_3_0
This release corrects a potential data loss issue, if you are running Arvados 1.3.0 or 1.3.1 we strongly recommended disabling @keep-balance@ until you can upgrade to 1.3.3 or 1.4.0. With keep-balance disabled, there is no chance of data loss.
We've put together a "wiki page":https://dev.arvados.org/projects/arvados/wiki/Recovering_lost_data which outlines how to recover blocks which have been put in the trash, but not yet deleted, as well as how to identify any collections which have missing blocks so that they can be regenerated. The keep-balance component has been enhanced to provide a list of missing blocks and affected collections and we've provided a "utility script":https://github.com/arvados/arvados/blob/master/tools/keep-xref/keep-xref.py which can be used to identify the workflows that generated those collections and who ran those workflows, so that they can be rerun.
-h3(#v1_3_0). v1.3.0 (2018-12-05)
+h2(#v1_3_0). v1.3.0 (2018-12-05)
+
+"Upgrading from 1.2":#v1_2_0
This release includes several database migrations, which will be executed automatically as part of the API server upgrade. On large Arvados installations, these migrations will take a while. We've seen the upgrade take 30 minutes or more on installations with a lot of collections.
Support for the deprecated "jobs" API is broken in this release. Users who rely on it should not upgrade. This will be fixed in an upcoming 1.3.1 patch release, however users are "encouraged to migrate":upgrade-crunch2.html as support for the "jobs" API will be dropped in an upcoming release. Users who are already using the "containers" API are not affected.
-h3(#v1_2_1). v1.2.1 (2018-11-26)
+h2(#v1_2_1). v1.2.1 (2018-11-26)
There are no special upgrade notes for this release.
-h3(#v1_2_0). v1.2.0 (2018-09-05)
+h2(#v1_2_0). v1.2.0 (2018-09-05)
-h4. Regenerate Postgres table statistics
+"Upgrading from 1.1.2 or 1.1.3":#v1_1_2
+
+h3. Regenerate Postgres table statistics
It is recommended to regenerate the table statistics for Postgres after upgrading to v1.2.0. If autovacuum is enabled on your installation, this script would do the trick:
If you also need to do the vacuum, you could adapt the script to run 'vacuum analyze' instead of 'analyze'.
-h4. New component: arvados-controller
+h3. New component: arvados-controller
Commit "db5107dca":https://dev.arvados.org/projects/arvados/repository/revisions/db5107dca adds a new system service, arvados-controller. More detail is available in story "#13496":https://dev.arvados.org/issues/13497.
Verify your setup by confirming that API calls appear in the controller's logs (_e.g._, @journalctl -fu arvados-controller@) while loading a workbench page.
-h3(#v1_1_4). v1.1.4 (2018-04-10)
+h2(#v1_1_4). v1.1.4 (2018-04-10)
+
+"Upgrading from 1.1.3":#v1_1_3
-h4. arvados-cwl-runner regressions (2018-04-05)
+h3. arvados-cwl-runner regressions (2018-04-05)
<strong>Secondary files missing from toplevel workflow inputs</strong>
This bug has been fixed in Arvados release v1.2.0.
-h3(#v1_1_3). v1.1.3 (2018-02-08)
+h2(#v1_1_3). v1.1.3 (2018-02-08)
There are no special upgrade notes for this release.
-h3(#v1_1_2). v1.1.2 (2017-12-22)
+h2(#v1_1_2). v1.1.2 (2017-12-22)
-h4. The minimum version for Postgres is now 9.4 (2017-12-08)
+"Upgrading from 1.1.0 or 1.1.1":#v1_1_0
+
+h3. The minimum version for Postgres is now 9.4 (2017-12-08)
As part of story "#11908":https://dev.arvados.org/issues/11908, commit "8f987a9271":https://dev.arvados.org/projects/arvados/repository/revisions/8f987a9271 introduces a dependency on Postgres 9.4. Previously, Arvados required Postgres 9.3.
*# Install the @rh-postgresql94@ backport package from either Software Collections: http://doc.arvados.org/install/install-postgresql.html or the Postgres developers: https://www.postgresql.org/download/linux/redhat/
*# Restore from the backup using @psql@
-h3(#v1_1_1). v1.1.1 (2017-11-30)
+h2(#v1_1_1). v1.1.1 (2017-11-30)
There are no special upgrade notes for this release.
-h3(#v1_1_0). v1.1.0 (2017-10-24)
+h2(#v1_1_0). v1.1.0 (2017-10-24)
-h4. The minimum version for Postgres is now 9.3 (2017-09-25)
+h3. The minimum version for Postgres is now 9.3 (2017-09-25)
As part of story "#12032":https://dev.arvados.org/issues/12032, commit "68bdf4cbb1":https://dev.arvados.org/projects/arvados/repository/revisions/68bdf4cbb1 introduces a dependency on Postgres 9.3. Previously, Arvados required Postgres 9.1.
*# Install the @rh-postgresql94@ backport package from either Software Collections: http://doc.arvados.org/install/install-postgresql.html or the Postgres developers: https://www.postgresql.org/download/linux/redhat/
*# Restore from the backup using @psql@
-h3(#older). Older versions
+h2(#older). Older versions
-h4. Upgrade slower than usual (2017-06-30)
+h3. Upgrade slower than usual (2017-06-30)
As part of story "#11807":https://dev.arvados.org/issues/11807, commit "55aafbb":https://dev.arvados.org/projects/arvados/repository/revisions/55aafbb converts old "jobs" database records from YAML to JSON, making the upgrade process slower than usual.
* The conversion runs as a database migration, i.e., during the deb/rpm package upgrade process, while your API server is unavailable.
* Expect it to take about 1 minute per 20K jobs that have ever been created/run.
-h4. Service discovery overhead change in keep-web (2017-06-05)
+h3. Service discovery overhead change in keep-web (2017-06-05)
As part of story "#9005":https://dev.arvados.org/issues/9005, commit "cb230b0":https://dev.arvados.org/projects/arvados/repository/revisions/cb230b0 reduces service discovery overhead in keep-web requests.
* When upgrading keep-web _or keepproxy_ to/past this version, make sure to update API server as well. Otherwise, a bad token in a request can cause keep-web to fail future requests until either keep-web restarts or API server gets upgraded.
-h4. Node manager now has an http endpoint for management (2017-04-12)
+h3. Node manager now has an http endpoint for management (2017-04-12)
As part of story "#11349":https://dev.arvados.org/issues/11349, commit "2c094e2":https://dev.arvados.org/projects/arvados/repository/revisions/2c094e2 adds a "management" http server to nodemanager.
port = 8989</pre> (see example configuration files in source:services/nodemanager/doc or https://doc.arvados.org/install/install-nodemanager.html for more info)
* The server responds to @http://{address}:{port}/status.json@ with a summary of how many nodes are in each state (booting, busy, shutdown, etc.)
-h4. New websockets component (2017-03-23)
+h3. New websockets component (2017-03-23)
As part of story "#10766":https://dev.arvados.org/issues/10766, commit "e8cc0d7":https://dev.arvados.org/projects/arvados/repository/revisions/e8cc0d7 replaces puma with arvados-ws as the recommended websocket server.
* See http://doc.arvados.org/install/install-ws.html for install/upgrade instructions.
$ systemctl stop puma
</pre>
-h4. Change of database encoding for hashes and arrays (2017-03-06)
+h3. Change of database encoding for hashes and arrays (2017-03-06)
As part of story "#11168":https://dev.arvados.org/issues/11168, commit "660a614":https://dev.arvados.org/projects/arvados/repository/revisions/660a614 uses JSON instead of YAML to encode hashes and arrays in the database.
* Downgrading past this version is not supported, and is likely to cause errors. If this happens, the solution is to upgrade past this version.
* After upgrading, make sure to restart puma and crunch-dispatch-* processes.
-h4. Docker image format compatibility check (2017-02-03)
+h3. Docker image format compatibility check (2017-02-03)
As part of story "#10969":https://dev.arvados.org/issues/10969, commit "74a9dec":https://dev.arvados.org/projects/arvados/repository/revisions/74a9dec introduces a Docker image format compatibility check: the @arv keep docker@ command prevents users from inadvertently saving docker images that compute nodes won't be able to run.
* If your compute nodes run a version of *docker older than 1.10* you must override the default by adding to your API server configuration (@/etc/arvados/api/application.yml@): <pre><code class="yaml">docker_image_formats: ["v1"]</code></pre>
* Refer to the comments above @docker_image_formats@ in @/var/www/arvados-api/current/config/application.default.yml@ or source:services/api/config/application.default.yml or issue "#10969":https://dev.arvados.org/issues/10969 for more detail.
* *NOTE:* This does *not* include any support for migrating existing Docker images from v1 to v2 format. This will come later: for now, sites running Docker 1.9 or earlier should still *avoid upgrading Docker further than 1.9.*
-h4. Debian and RPM packages now have systemd unit files (2016-09-27)
+h3. Debian and RPM packages now have systemd unit files (2016-09-27)
Several Debian and RPM packages -- keep-balance ("d9eec0b":https://dev.arvados.org/projects/arvados/repository/revisions/d9eec0b), keep-web ("3399e63":https://dev.arvados.org/projects/arvados/repository/revisions/3399e63), keepproxy ("6de67b6":https://dev.arvados.org/projects/arvados/repository/revisions/6de67b6), and arvados-git-httpd ("9e27ddf":https://dev.arvados.org/projects/arvados/repository/revisions/9e27ddf) -- now enable their respective components using systemd. These components prefer YAML configuration files over command line flags ("3bbe1cd":https://dev.arvados.org/projects/arvados/repository/revisions/3bbe1cd).
** keepproxy - /etc/arvados/keepproxy/keepproxy.yml
** arvados-git-httpd - /etc/arvados/arv-git-httpd/arv-git-httpd.yml
-h4. Installation paths for Python modules and script changed (2016-05-31)
+h3. Installation paths for Python modules and script changed (2016-05-31)
Commits "ae72b172c8":https://dev.arvados.org/projects/arvados/repository/revisions/ae72b172c8 and "3aae316c25":https://dev.arvados.org/projects/arvados/repository/revisions/3aae316c25 change the filesystem location where Python modules and scripts are installed.
* Previous packages installed these files to the distribution's preferred path under @/usr/local@ (or the equivalent location in a Software Collection). Now they get installed to a path under @/usr@. This improves compatibility with other Python packages provided by the distribution. See "#9242":https://dev.arvados.org/issues/9242 for more background.
* If you simply import Python modules from scripts, or call Python tools relying on $PATH, you don't need to make any changes. If you have hardcoded full paths to some of these files (e.g., in symbolic links or configuration files), you will need to update those paths after this upgrade.
-h4. Crunchrunner package is required on compute and shell nodes (2016-04-25)
+h3. Crunchrunner package is required on compute and shell nodes (2016-04-25)
Commit "eebcb5e":https://dev.arvados.org/projects/arvados/repository/revisions/eebcb5e requires the crunchrunner package to be installed on compute nodes and shell nodes in order to run CWL workflows.
* On each Debian-based compute node and shell node, run: @sudo apt-get install crunchrunner@
* On each Red Hat-based compute node and shell node, run: @sudo yum install crunchrunner@
-h4. Keep permission signature algorithm change (2016-04-21)
+h3. Keep permission signature algorithm change (2016-04-21)
Commit "3c88abd":https://dev.arvados.org/projects/arvados/repository/revisions/3c88abd changes the Keep permission signature algorithm.
* All software components that generate signatures must be upgraded together. These are: keepstore, API server, keep-block-check, and keep-rsync. For example, if keepstore < 0.1.20160421183420 but API server >= 0.1.20160421183420, clients will not be able to read or write data in Keep.
* Jobs and client operations that are in progress during the upgrade (including arv-put's "resume cache") will fail.
-h4. Workbench's "Getting Started" popup disabled by default (2015-01-05)
+h3. Workbench's "Getting Started" popup disabled by default (2015-01-05)
Commit "e1276d6e":https://dev.arvados.org/projects/arvados/repository/revisions/e1276d6e disables Workbench's "Getting Started" popup by default.
* If you want new users to continue seeing this popup, set @enable_getting_started_popup: true@ in Workbench's @application.yml@ configuration.
-h4. Crunch jobs now have access to Keep-backed writable scratch storage (2015-12-03)
+h3. Crunch jobs now have access to Keep-backed writable scratch storage (2015-12-03)
Commit "5590c9ac":https://dev.arvados.org/projects/arvados/repository/revisions/5590c9ac makes a Keep-backed writable scratch directory available in crunch jobs (see "#7751":https://dev.arvados.org/issues/7751)
* All compute nodes must be upgraded to arvados-fuse >= 0.1.2015112518060 because crunch-job uses some new arv-mount flags (--mount-tmp, --mount-by-pdh) introduced in merge "346a558":https://dev.arvados.org/projects/arvados/repository/revisions/346a558
* Jobs will fail if the API server (in particular crunch-job from the arvados-cli gem) is upgraded without upgrading arvados-fuse on compute nodes.
-h4. Recommended configuration change for keep-web (2015-11-11)
+h3. Recommended configuration change for keep-web (2015-11-11)
Commit "1e2ace5":https://dev.arvados.org/projects/arvados/repository/revisions/1e2ace5 changes recommended config for keep-web (see "#5824":https://dev.arvados.org/issues/5824)
* Workbench config adds @keep_web_download_url@
* More info on the (still beta/non-TOC-linked) "keep-web doc page":http://doc.arvados.org/install/install-keep-web.html
-h4. Stopped containers are now automatically removed on compute nodes (2015-11-04)
+h3. Stopped containers are now automatically removed on compute nodes (2015-11-04)
Commit "1d1c6de":https://dev.arvados.org/projects/arvados/repository/revisions/1d1c6de removes stopped containers (see "#7444":https://dev.arvados.org/issues/7444)
* arvados-docker-cleaner removes _all_ docker containers as soon as they exit, effectively making @docker run@ default to @--rm@. If you run arvados-docker-cleaner on a host that does anything other than run crunch-jobs, and you still want to be able to use @docker start@, read the "new doc page":http://doc.arvados.org/install/install-compute-node.html to learn how to turn this off before upgrading.
-h4. New keep-web service (2015-11-04)
+h3. New keep-web service (2015-11-04)
Commit "21006cf":https://dev.arvados.org/projects/arvados/repository/revisions/21006cf adds a new keep-web service (see "#5824":https://dev.arvados.org/issues/5824).
* Nothing relies on keep-web yet, but early adopters can install it now by following http://doc.arvados.org/install/install-keep-web.html (it is not yet linked in the TOC).
+
+<notextile>
+</div>
+</notextile>
---
layout: default
navsection: admin
-title: Workbench2 Vocabulary Format
+title: User properties vocabulary
...
{% comment %}
{% codeblock as python %}
{% include 'vocabulary_migrate_py' %}
-{% endcodeblock %}
\ No newline at end of file
+{% endcodeblock %}
* Have filters only matching @[["uuid", "in", [...]]@ or @["uuid", "=", "..."]@
* Specify @count=none@
-* If @select@ is specified, it must include @uuid@
* Not specify @limit@, @offset@ or @order@
* Not request more items than the maximum response size
See "permission links":{{site.baseurl}}/api/permission-model.html#links section of the permission model.
+h3. star
+
+A **star** link is a shortcut to a project that is displayed in the user interface (Workbench) as "favorites". Users can mark their own favorites (implemented by creating or deleting **star** links).
+
+An admin can also create **star** links owned by the "All Users" group, these will be displayed to all users that have permission to read the project that has been favorited.
+
+The schema for a star link is:
+
+table(table table-bordered table-condensed).
+|_. Field|_. Value|_. Description|
+|owner_uuid|user or group uuid|Either the user that owns the favorite, or the "All Users" group for public favorites.|
+|head_uuid|project uuid|The project being favorited|
+|link_class|string of value "star"|Indicates this represents a link to a user favorite|
+
+h4. Creating a favorite
+
+@owner_uuid@ is either an individual user, or the "All Users" group. The @head_uuid@ is the project being favorited.
+
+<pre>
+$ arv link create --link '{
+ "owner_uuid": "zzzzz-j7d0g-fffffffffffffff",
+ "head_uuid": "zzzzz-j7d0g-theprojectuuid",
+ "link_class": "star"}'
+</pre>
+
+h4. Deleting a favorite
+
+<pre>
+$ arv link delete --uuid zzzzz-o0j2j-thestarlinkuuid
+</pre>
+
+h4. Listing favorites
+
+To list all 'star' links that will be displayed for a user:
+
+<pre>
+$ arv link list --filters '[
+ ["link_class", "=", "star"],
+ ["owner_uuid", "in", ["zzzzz-j7d0g-fffffffffffffff", "zzzzz-tpzed-currentuseruuid"]]]'
+</pre>
+
h3. tag
-A **tag** link describes an object using an unparsed plain text string. Tags can be used to annotate objects that are not editable, like collections and objects shared as read-only.
+A **tag** link describes an object using an unparsed plain text string. Tags can be used to annotate objects that are not directly editable by the user, like collections and objects shared as read-only.
table(table table-bordered table-condensed).
|_. tail_type→head_type|_. name→head_uuid {properties}|
td.line-numbers {
width: 2em;
}
+
+.releasenotes h2 { margin-top: 1.5em; text-decoration: underline; }
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
+{% include 'notebox_begin' %}
+This section is about installing an Arvados cluster. If you are just looking to install Arvados client tools and libraries, "go to the SDK section.":{{site.baseurl}}/sdk
+{% include 'notebox_end' %}
+
Arvados components run on GNU/Linux systems, and supports AWS, GCP and Azure cloud platforms as well as on-premises installs. Arvados supports Debian and derivatives such as Ubuntu, as well as Red Hat and derivatives such as CentOS. "Arvados is Free Software":{{site.baseurl}}/user/copying/copying.html and self-install installations are not limited in any way. Commercial support and development are also available from "Curii Corporation.":mailto:info@curii.com
Arvados components can be installed and configured in a number of different ways.
---
layout: default
navsection: installguide
-title: Install Workbench2 (beta)
+title: Install Workbench 2
...
{% comment %}
Copyright (C) The Arvados Authors. All rights reserved.
Workbench2 is the web-based user interface for Arvados.
{% include 'notebox_begin' %}
-Workbench2 is the replacement for Arvados Workbench. Workbench2 is currently in <i>beta</i>, it is not yet feature complete.
+Workbench2 is the replacement for Arvados Workbench. Workbench2 is suitable for day-to-day use, but does not yet implement every feature of the traditional Workbench.
{% include 'notebox_end' %}
h2(#configure). Update config.yml
This section documents language bindings for the "Arvados API":{{site.baseurl}}/api and Keep that are available for various programming languages. Not all features are available in every SDK. The most complete SDK is the Python SDK. Note that this section only gives a high level overview of each SDK. Consult the "Arvados API":{{site.baseurl}}/api section for detailed documentation about Arvados API calls available on each resource.
-* "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html
+* "Python SDK":{{site.baseurl}}/sdk/python/sdk-python.html (also includes essential command line tools such as "arv-put" and "arv-get")
* "Command line SDK":{{site.baseurl}}/sdk/cli/install.html ("arv")
* "Go SDK":{{site.baseurl}}/sdk/go/index.html
* "R SDK":{{site.baseurl}}/sdk/R/index.html
github.com/Microsoft/go-winio v0.4.5 // indirect
github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7 // indirect
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 // indirect
+ github.com/arvados/cgofuse v1.2.0
github.com/aws/aws-sdk-go v1.25.30
github.com/coreos/go-oidc v2.1.0+incompatible
github.com/coreos/go-systemd v0.0.0-20180108085132-cc4f39464dc7
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/jmcvetta/randutil v0.0.0-20150817122601-2bb1b664bcff
github.com/julienschmidt/httprouter v1.2.0
+ github.com/karalabe/xgo v0.0.0-20191115072854-c5ccff8648a7 // indirect
github.com/kevinburke/ssh_config v0.0.0-20171013211458-802051befeb5 // indirect
github.com/lib/pq v0.0.0-20171126050459-83612a56d3dd
github.com/marstr/guid v1.1.1-0.20170427235115-8bdf7d1a087c // indirect
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 h1:kFOfPq6dUM1hTo4JG6LR5AXSUEsOjtdm0kw0FtQtMJA=
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
+github.com/arvados/cgofuse v1.2.0 h1:sWgVxyvSFjH965Uc7ReScn/cBl9Jemc9SeUNlEmjRH4=
+github.com/arvados/cgofuse v1.2.0/go.mod h1:79WFV98hrkRHK9XPhh2IGGOwpFSjocsWubgxAs2KhRc=
github.com/arvados/goamz v0.0.0-20190905141525-1bba09f407ef h1:cl7DIRbiAYNqaVxg3CZY8qfZoBOKrj06H/x9SPGaxas=
github.com/arvados/goamz v0.0.0-20190905141525-1bba09f407ef/go.mod h1:rCtgyMmBGEbjTm37fCuBYbNL0IhztiALzo3OB9HyiOM=
github.com/aws/aws-sdk-go v1.25.30 h1:I9qj6zW3mMfsg91e+GMSN/INcaX9tTFvr/l/BAHKaIY=
github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
github.com/coreos/go-systemd v0.0.0-20180108085132-cc4f39464dc7 h1:e3u8KWFMR3irlDo1Z/tL8Hsz1MJmCLkSoX5AZRMKZkg=
github.com/coreos/go-systemd v0.0.0-20180108085132-cc4f39464dc7/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
-github.com/curoverse/goamz v0.0.0-20190905141525-1bba09f407ef h1:k3Q9m06dbTShrR4phl/QNi15ZSPkIwgyQmNvJRcXR3Y=
-github.com/curoverse/goamz v0.0.0-20190905141525-1bba09f407ef/go.mod h1:NUkr+hZ9k+l0cEXg9S7EW8+UIfPkP/hNy2Ga0QVPZ88=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/julienschmidt/httprouter v1.2.0 h1:TDTW5Yz1mjftljbcKqRcrYhd4XeOoI98t+9HbQbYf7g=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
+github.com/karalabe/xgo v0.0.0-20191115072854-c5ccff8648a7 h1:AYzjK/SHz6m6mg5iuFwkrAhCc14jvCpW9d6frC9iDPE=
+github.com/karalabe/xgo v0.0.0-20191115072854-c5ccff8648a7/go.mod h1:iYGcTYIPUvEWhFo6aKUuLchs+AV4ssYdyuBbQJZGcBk=
github.com/kevinburke/ssh_config v0.0.0-20171013211458-802051befeb5 h1:xXn0nBttYwok7DhU4RxqaADEpQn7fEMt5kKc3yoj/n0=
github.com/kevinburke/ssh_config v0.0.0-20171013211458-802051befeb5/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
identification, and does not retrieve any other personal
information.</i>
+ # Workbench screen displayed to inactive users. This is HTML
+ # text that will be incorporated directly onto the page.
InactivePageHTML: |
<img src="/arvados-logo-big.png" style="width: 20%; float: right; padding: 1em;" />
<h3>Hi! You're logged in, but...</h3>
<p>An administrator must activate your account before you can get
any further.</p>
+ # Connecting to Arvados shell VMs tends to be site-specific.
+ # Put any special instructions here. This is HTML text that will
+ # be incorporated directly onto the Workbench page.
+ SSHHelpPageHTML: |
+ <a href="https://doc.arvados.org/user/getting_started/ssh-access-unix.html">Accessing an Arvados VM with SSH</a> (generic instructions).
+ Site configurations vary. Contact your local cluster administrator if you have difficulty accessing an Arvados shell node.
+
# Bypass new (Arvados 1.5) API implementations, and hand off
# requests directly to Rails instead. This can provide a temporary
# workaround for clients that are incompatible with the new API
"Workbench.VocabularyURL": true,
"Workbench.WelcomePageHTML": true,
"Workbench.InactivePageHTML": true,
+ "Workbench.SSHHelpPageHTML": true,
}
func redactUnsafe(m map[string]interface{}, mPrefix, lookupPrefix string) error {
identification, and does not retrieve any other personal
information.</i>
+ # Workbench screen displayed to inactive users. This is HTML
+ # text that will be incorporated directly onto the page.
InactivePageHTML: |
<img src="/arvados-logo-big.png" style="width: 20%; float: right; padding: 1em;" />
<h3>Hi! You're logged in, but...</h3>
<p>An administrator must activate your account before you can get
any further.</p>
+ # Connecting to Arvados shell VMs tends to be site-specific.
+ # Put any special instructions here. This is HTML text that will
+ # be incorporated directly onto the Workbench page.
+ SSHHelpPageHTML: |
+ <a href="https://doc.arvados.org/user/getting_started/ssh-access-unix.html">Accessing an Arvados VM with SSH</a> (generic instructions).
+ Site configurations vary. Contact your local cluster administrator if you have difficulty accessing an Arvados shell node.
+
# Bypass new (Arvados 1.5) API implementations, and hand off
# requests directly to Rails instead. This can provide a temporary
# workaround for clients that are incompatible with the new API
if !remote.Proxy {
continue
}
- remotes[id] = rpc.NewConn(id, &url.URL{Scheme: remote.Scheme, Host: remote.Host}, remote.Insecure, saltedTokenProvider(local, id))
+ conn := rpc.NewConn(id, &url.URL{Scheme: remote.Scheme, Host: remote.Host}, remote.Insecure, saltedTokenProvider(local, id))
+ // Older versions of controller rely on the Via header
+ // to detect loops.
+ conn.SendHeader = http.Header{"Via": {"HTTP/1.1 arvados-controller"}}
+ remotes[id] = conn
}
return &Conn{
// or "" for the local backend.
//
// A non-nil error means all backends failed.
-func (conn *Conn) tryLocalThenRemotes(ctx context.Context, fn func(context.Context, string, backend) error) error {
- if err := fn(ctx, "", conn.local); err == nil || errStatus(err) != http.StatusNotFound {
+func (conn *Conn) tryLocalThenRemotes(ctx context.Context, forwardedFor string, fn func(context.Context, string, backend) error) error {
+ if err := fn(ctx, "", conn.local); err == nil || errStatus(err) != http.StatusNotFound || forwardedFor != "" {
+ // Note: forwardedFor != "" means this request came
+ // from a remote cluster, so we don't take a second
+ // hop. This avoids cycles, redundant calls to a
+ // mutually reachable remote, and use of double-salted
+ // tokens.
return err
}
} else {
// UUID is a PDH
first := make(chan arvados.Collection, 1)
- err := conn.tryLocalThenRemotes(ctx, func(ctx context.Context, remoteID string, be backend) error {
- c, err := be.CollectionGet(ctx, options)
+ err := conn.tryLocalThenRemotes(ctx, options.ForwardedFor, func(ctx context.Context, remoteID string, be backend) error {
+ remoteOpts := options
+ remoteOpts.ForwardedFor = conn.cluster.ClusterID + "-" + options.ForwardedFor
+ c, err := be.CollectionGet(ctx, remoteOpts)
if err != nil {
return err
}
//
// * len(Order)==0
//
-// * Each filter must be either "uuid = ..." or "uuid in [...]".
+// * Each filter is either "uuid = ..." or "uuid in [...]".
//
// * The maximum possible response size (total number of objects that
// could potentially be matched by all of the specified filters)
}
}
- if len(todoByRemote) > 1 {
- if cannotSplit {
- return httpErrorf(http.StatusBadRequest, "cannot execute federated list query: each filter must be either 'uuid = ...' or 'uuid in [...]'")
- }
- if opts.Count != "none" {
- return httpErrorf(http.StatusBadRequest, "cannot execute federated list query unless count==\"none\"")
- }
- if opts.Limit >= 0 || opts.Offset != 0 || len(opts.Order) > 0 {
- return httpErrorf(http.StatusBadRequest, "cannot execute federated list query with limit, offset, or order parameter")
- }
- if max := conn.cluster.API.MaxItemsPerResponse; nUUIDs > max {
- return httpErrorf(http.StatusBadRequest, "cannot execute federated list query because number of UUIDs (%d) exceeds page size limit %d", nUUIDs, max)
- }
- selectingUUID := false
- for _, attr := range opts.Select {
- if attr == "uuid" {
- selectingUUID = true
- }
- }
- if opts.Select != nil && !selectingUUID {
- return httpErrorf(http.StatusBadRequest, "cannot execute federated list query with a select parameter that does not include uuid")
- }
+ if len(todoByRemote) == 0 {
+ return nil
+ }
+ if len(todoByRemote) == 1 && todoByRemote[conn.cluster.ClusterID] != nil {
+ // All UUIDs are local, so proxy a single request. The
+ // generic case has some limitations (see below) which
+ // we don't want to impose on local requests.
+ _, err := fn(ctx, conn.cluster.ClusterID, conn.local, opts)
+ return err
+ }
+ if cannotSplit {
+ return httpErrorf(http.StatusBadRequest, "cannot execute federated list query: each filter must be either 'uuid = ...' or 'uuid in [...]'")
+ }
+ if opts.Count != "none" {
+ return httpErrorf(http.StatusBadRequest, "cannot execute federated list query unless count==\"none\"")
+ }
+ if opts.Limit >= 0 || opts.Offset != 0 || len(opts.Order) > 0 {
+ return httpErrorf(http.StatusBadRequest, "cannot execute federated list query with limit, offset, or order parameter")
+ }
+ if max := conn.cluster.API.MaxItemsPerResponse; nUUIDs > max {
+ return httpErrorf(http.StatusBadRequest, "cannot execute federated list query because number of UUIDs (%d) exceeds page size limit %d", nUUIDs, max)
}
ctx, cancel := context.WithCancel(ctx)
return
}
remoteOpts := opts
+ if remoteOpts.Select != nil {
+ // We always need to select UUIDs to
+ // use the response, even if our
+ // caller doesn't.
+ remoteOpts.Select = append([]string{"uuid"}, remoteOpts.Select...)
+ }
for len(todo) > 0 {
if len(batch) > len(todo) {
// Reduce batch to just the todo's
"context"
"fmt"
"net/http"
+ "reflect"
"sort"
"git.arvados.org/arvados.git/sdk/go/arvados"
break
}
if cl.matchFilters(c, options.Filters) {
+ if reflect.DeepEqual(options.Select, []string{"uuid", "name"}) {
+ c = arvados.Collection{UUID: c.UUID, Name: c.Name}
+ } else if reflect.DeepEqual(options.Select, []string{"name"}) {
+ c = arvados.Collection{Name: c.Name}
+ } else if len(options.Select) > 0 {
+ panic(fmt.Sprintf("not implemented: options=%#v", options))
+ }
resp.Items = append(resp.Items, c)
}
}
offset int
order []string
filters []arvados.Filter
+ selectfields []string
expectUUIDs []string
expectCalls []int // number of API calls to backends
expectStatus int
})
}
+func (s *CollectionListSuite) TestCollectionListOneLocalDeselectingUUID(c *check.C) {
+ s.test(c, listTrial{
+ count: "none",
+ limit: -1,
+ filters: []arvados.Filter{{"uuid", "=", s.uuids[0][0]}},
+ selectfields: []string{"name"},
+ expectUUIDs: []string{""}, // select=name is honored
+ expectCalls: []int{1, 0, 0},
+ })
+}
+
func (s *CollectionListSuite) TestCollectionListOneLocalUsingInOperator(c *check.C) {
s.test(c, listTrial{
count: "none",
})
}
+func (s *CollectionListSuite) TestCollectionListOneRemoteDeselectingUUID(c *check.C) {
+ s.test(c, listTrial{
+ count: "none",
+ limit: -1,
+ filters: []arvados.Filter{{"uuid", "=", s.uuids[1][0]}},
+ selectfields: []string{"name"},
+ expectUUIDs: []string{s.uuids[1][0]}, // uuid is returned, despite not being selected
+ expectCalls: []int{0, 1, 0},
+ })
+}
+
func (s *CollectionListSuite) TestCollectionListOneLocalOneRemote(c *check.C) {
s.test(c, listTrial{
count: "none",
})
}
+func (s *CollectionListSuite) TestCollectionListOneLocalOneRemoteDeselectingUUID(c *check.C) {
+ s.test(c, listTrial{
+ count: "none",
+ limit: -1,
+ filters: []arvados.Filter{{"uuid", "in", []string{s.uuids[0][0], s.uuids[1][0]}}},
+ selectfields: []string{"name"},
+ expectUUIDs: []string{s.uuids[0][0], s.uuids[1][0]}, // uuid is returned, despite not being selected
+ expectCalls: []int{1, 1, 0},
+ })
+}
+
func (s *CollectionListSuite) TestCollectionListTwoRemotes(c *check.C) {
s.test(c, listTrial{
count: "none",
Offset: trial.offset,
Order: trial.order,
Filters: trial.filters,
+ Select: trial.selectfields,
})
if trial.expectStatus != 0 {
c.Assert(err, check.NotNil)
import (
"context"
+ "crypto/tls"
"encoding/json"
+ "io/ioutil"
"net/http"
"net/http/httptest"
"net/url"
c.Check(user.UUID, check.Equals, arvadostest.ActiveUserUUID)
c.Check(user.Authorization.TokenV2(), check.Equals, auth.TokenV2())
}
+
+func (s *HandlerSuite) CheckObjectType(c *check.C, url string, token string, skippedFields map[string]bool) {
+ var proxied, direct map[string]interface{}
+ var err error
+
+ // Get collection from controller
+ req := httptest.NewRequest("GET", url, nil)
+ req.Header.Set("Authorization", "Bearer "+token)
+ resp := httptest.NewRecorder()
+ s.handler.ServeHTTP(resp, req)
+ c.Assert(resp.Code, check.Equals, http.StatusOK,
+ check.Commentf("Wasn't able to get data from the controller at %q", url))
+ err = json.Unmarshal(resp.Body.Bytes(), &proxied)
+ c.Check(err, check.Equals, nil)
+
+ // Get collection directly from RailsAPI
+ client := &http.Client{
+ Transport: &http.Transport{
+ TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
+ },
+ }
+ resp2, err := client.Get(s.cluster.Services.RailsAPI.ExternalURL.String() + url + "/?api_token=" + token)
+ c.Check(err, check.Equals, nil)
+ defer resp2.Body.Close()
+ db, err := ioutil.ReadAll(resp2.Body)
+ c.Check(err, check.Equals, nil)
+ err = json.Unmarshal(db, &direct)
+ c.Check(err, check.Equals, nil)
+
+ // Check that all RailsAPI provided keys exist on the controller response.
+ for k := range direct {
+ if _, ok := skippedFields[k]; ok {
+ continue
+ } else if val, ok := proxied[k]; ok {
+ if direct["kind"] == "arvados#collection" && k == "manifest_text" {
+ // Tokens differ from request to request
+ c.Check(strings.Split(val.(string), "+A")[0], check.Equals, strings.Split(direct[k].(string), "+A")[0])
+ } else {
+ c.Check(val, check.DeepEquals, direct[k],
+ check.Commentf("RailsAPI %s key %q's value %q differs from controller's %q.", direct["kind"], k, direct[k], val))
+ }
+ } else {
+ c.Errorf("%s's key %q missing on controller's response.", direct["kind"], k)
+ }
+ }
+}
+
+func (s *HandlerSuite) TestGetObjects(c *check.C) {
+ // Get the 1st keep service's uuid from the running test server.
+ req := httptest.NewRequest("GET", "/arvados/v1/keep_services/", nil)
+ req.Header.Set("Authorization", "Bearer "+arvadostest.AdminToken)
+ resp := httptest.NewRecorder()
+ s.handler.ServeHTTP(resp, req)
+ c.Assert(resp.Code, check.Equals, http.StatusOK)
+ var ksList arvados.KeepServiceList
+ json.Unmarshal(resp.Body.Bytes(), &ksList)
+ c.Assert(len(ksList.Items), check.Not(check.Equals), 0)
+ ksUUID := ksList.Items[0].UUID
+
+ testCases := map[string]map[string]bool{
+ "api_clients/" + arvadostest.TrustedWorkbenchAPIClientUUID: nil,
+ "api_client_authorizations/" + arvadostest.AdminTokenUUID: nil,
+ "authorized_keys/" + arvadostest.AdminAuthorizedKeysUUID: nil,
+ "collections/" + arvadostest.CollectionWithUniqueWordsUUID: map[string]bool{"href": true},
+ "containers/" + arvadostest.RunningContainerUUID: nil,
+ "container_requests/" + arvadostest.QueuedContainerRequestUUID: nil,
+ "groups/" + arvadostest.AProjectUUID: nil,
+ "keep_services/" + ksUUID: nil,
+ "links/" + arvadostest.ActiveUserCanReadAllUsersLinkUUID: nil,
+ "logs/" + arvadostest.CrunchstatForRunningJobLogUUID: nil,
+ "nodes/" + arvadostest.IdleNodeUUID: nil,
+ "repositories/" + arvadostest.ArvadosRepoUUID: nil,
+ "users/" + arvadostest.ActiveUserUUID: map[string]bool{"href": true},
+ "virtual_machines/" + arvadostest.TestVMUUID: nil,
+ "workflows/" + arvadostest.WorkflowWithDefinitionYAMLUUID: nil,
+ }
+ for url, skippedFields := range testCases {
+ s.CheckObjectType(c, "/arvados/v1/"+url, arvadostest.AdminToken, skippedFields)
+ }
+}
Subsystem: "dispatchcloud",
Name: "instances_total",
Help: "Number of cloud VMs.",
- }, []string{"category"})
+ }, []string{"category", "instance_type"})
reg.MustRegister(wp.mInstances)
wp.mInstancesPrice = prometheus.NewGaugeVec(prometheus.GaugeOpts{
Namespace: "arvados",
wp.mtx.RLock()
defer wp.mtx.RUnlock()
- instances := map[string]int64{}
+ type entKey struct {
+ cat string
+ instType string
+ }
+ instances := map[entKey]int64{}
price := map[string]float64{}
cpu := map[string]int64{}
mem := map[string]int64{}
default:
cat = "idle"
}
- instances[cat]++
+ instances[entKey{cat, wkr.instType.Name}]++
price[cat] += wkr.instType.Price
cpu[cat] += int64(wkr.instType.VCPUs)
mem[cat] += int64(wkr.instType.RAM)
running += int64(len(wkr.running) + len(wkr.starting))
}
for _, cat := range []string{"inuse", "hold", "booting", "unknown", "idle"} {
- wp.mInstances.WithLabelValues(cat).Set(float64(instances[cat]))
wp.mInstancesPrice.WithLabelValues(cat).Set(price[cat])
wp.mVCPUs.WithLabelValues(cat).Set(float64(cpu[cat]))
wp.mMemory.WithLabelValues(cat).Set(float64(mem[cat]))
+ // make sure to reset gauges for non-existing category/nodetype combinations
+ for _, it := range wp.instanceTypes {
+ if _, ok := instances[entKey{cat, it.Name}]; !ok {
+ wp.mInstances.WithLabelValues(cat, it.Name).Set(float64(0))
+ }
+ }
+ }
+ for k, v := range instances {
+ wp.mInstances.WithLabelValues(k.cat, k.instType).Set(float64(v))
}
wp.mContainersRunning.Set(float64(running))
}
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package mount
+
+import (
+ "flag"
+ "io"
+ "log"
+ "net/http"
+ _ "net/http/pprof"
+ "os"
+
+ "git.arvados.org/arvados.git/sdk/go/arvados"
+ "git.arvados.org/arvados.git/sdk/go/arvadosclient"
+ "git.arvados.org/arvados.git/sdk/go/keepclient"
+ "github.com/arvados/cgofuse/fuse"
+)
+
+var Command = &cmd{}
+
+type cmd struct {
+ // ready, if non-nil, will be closed when the mount is
+ // initialized. If ready is non-nil, it RunCommand() should
+ // not be called more than once, or when ready is already
+ // closed.
+ ready chan struct{}
+ // It is safe to call Unmount only after ready has been
+ // closed.
+ Unmount func() (ok bool)
+}
+
+// RunCommand implements the subcommand "mount <path> [fuse options]".
+//
+// The "-d" fuse option (and perhaps other features) ignores the
+// stderr argument and prints to os.Stderr instead.
+func (c *cmd) RunCommand(prog string, args []string, stdin io.Reader, stdout, stderr io.Writer) int {
+ logger := log.New(stderr, prog+" ", 0)
+ flags := flag.NewFlagSet(prog, flag.ContinueOnError)
+ ro := flags.Bool("ro", false, "read-only")
+ experimental := flags.Bool("experimental", false, "acknowledge this is an experimental command, and should not be used in production (required)")
+ blockCache := flags.Int("block-cache", 4, "read cache size (number of 64MiB blocks)")
+ pprof := flags.String("pprof", "", "serve Go profile data at `[addr]:port`")
+ err := flags.Parse(args)
+ if err != nil {
+ logger.Print(err)
+ return 2
+ }
+ if !*experimental {
+ logger.Printf("error: experimental command %q used without --experimental flag", prog)
+ return 2
+ }
+ if *pprof != "" {
+ go func() {
+ log.Println(http.ListenAndServe(*pprof, nil))
+ }()
+ }
+
+ client := arvados.NewClientFromEnv()
+ ac, err := arvadosclient.New(client)
+ if err != nil {
+ logger.Print(err)
+ return 1
+ }
+ kc, err := keepclient.MakeKeepClient(ac)
+ if err != nil {
+ logger.Print(err)
+ return 1
+ }
+ kc.BlockCache = &keepclient.BlockCache{MaxBlocks: *blockCache}
+ host := fuse.NewFileSystemHost(&keepFS{
+ Client: client,
+ KeepClient: kc,
+ ReadOnly: *ro,
+ Uid: os.Getuid(),
+ Gid: os.Getgid(),
+ ready: c.ready,
+ })
+ c.Unmount = host.Unmount
+ ok := host.Mount("", flags.Args())
+ if !ok {
+ return 1
+ }
+ return 0
+}
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package mount
+
+import (
+ "bytes"
+ "encoding/json"
+ "io/ioutil"
+ "os"
+ "time"
+
+ "git.arvados.org/arvados.git/sdk/go/arvadostest"
+ check "gopkg.in/check.v1"
+)
+
+var _ = check.Suite(&CmdSuite{})
+
+type CmdSuite struct {
+ mnt string
+}
+
+func (s *CmdSuite) SetUpTest(c *check.C) {
+ tmpdir, err := ioutil.TempDir("", "")
+ c.Assert(err, check.IsNil)
+ s.mnt = tmpdir
+}
+
+func (s *CmdSuite) TearDownTest(c *check.C) {
+ c.Check(os.RemoveAll(s.mnt), check.IsNil)
+}
+
+func (s *CmdSuite) TestMount(c *check.C) {
+ exited := make(chan int)
+ stdin := bytes.NewBufferString("stdin")
+ stdout := bytes.NewBuffer(nil)
+ stderr := bytes.NewBuffer(nil)
+ mountCmd := cmd{ready: make(chan struct{})}
+ ready := false
+ go func() {
+ exited <- mountCmd.RunCommand("test mount", []string{"--experimental", s.mnt}, stdin, stdout, stderr)
+ }()
+ go func() {
+ <-mountCmd.ready
+ ready = true
+
+ f, err := os.Open(s.mnt + "/by_id/" + arvadostest.FooCollection)
+ if c.Check(err, check.IsNil) {
+ dirnames, err := f.Readdirnames(-1)
+ c.Check(err, check.IsNil)
+ c.Check(dirnames, check.DeepEquals, []string{"foo"})
+ f.Close()
+ }
+
+ buf, err := ioutil.ReadFile(s.mnt + "/by_id/" + arvadostest.FooCollection + "/.arvados#collection")
+ if c.Check(err, check.IsNil) {
+ var m map[string]interface{}
+ err = json.Unmarshal(buf, &m)
+ c.Check(err, check.IsNil)
+ c.Check(m["manifest_text"], check.Matches, `\. acbd.* 0:3:foo\n`)
+ }
+
+ _, err = os.Open(s.mnt + "/by_id/zzzzz-4zz18-does-not-exist")
+ c.Check(os.IsNotExist(err), check.Equals, true)
+
+ ok := mountCmd.Unmount()
+ c.Check(ok, check.Equals, true)
+ }()
+ select {
+ case <-time.After(5 * time.Second):
+ c.Fatal("timed out")
+ case errCode, ok := <-exited:
+ c.Check(ok, check.Equals, true)
+ c.Check(errCode, check.Equals, 0)
+ }
+ c.Check(ready, check.Equals, true)
+ c.Check(stdout.String(), check.Equals, "")
+ // stdin should not have been read
+ c.Check(stdin.String(), check.Equals, "stdin")
+}
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package mount
+
+import (
+ "io"
+ "log"
+ "os"
+ "runtime/debug"
+ "sync"
+
+ "git.arvados.org/arvados.git/sdk/go/arvados"
+ "git.arvados.org/arvados.git/sdk/go/keepclient"
+ "github.com/arvados/cgofuse/fuse"
+)
+
+// sharedFile wraps arvados.File with a sync.Mutex, so fuse can safely
+// use a single filehandle concurrently on behalf of multiple
+// threads/processes.
+type sharedFile struct {
+ arvados.File
+ sync.Mutex
+}
+
+// keepFS implements cgofuse's FileSystemInterface.
+type keepFS struct {
+ fuse.FileSystemBase
+ Client *arvados.Client
+ KeepClient *keepclient.KeepClient
+ ReadOnly bool
+ Uid int
+ Gid int
+
+ root arvados.CustomFileSystem
+ open map[uint64]*sharedFile
+ lastFH uint64
+ sync.RWMutex
+
+ // If non-nil, this channel will be closed by Init() to notify
+ // other goroutines that the mount is ready.
+ ready chan struct{}
+}
+
+var (
+ invalidFH = ^uint64(0)
+)
+
+// newFH wraps f in a sharedFile, adds it to fs's lookup table using a
+// new handle number, and returns the handle number.
+func (fs *keepFS) newFH(f arvados.File) uint64 {
+ fs.Lock()
+ defer fs.Unlock()
+ if fs.open == nil {
+ fs.open = make(map[uint64]*sharedFile)
+ }
+ fs.lastFH++
+ fh := fs.lastFH
+ fs.open[fh] = &sharedFile{File: f}
+ return fh
+}
+
+func (fs *keepFS) lookupFH(fh uint64) *sharedFile {
+ fs.RLock()
+ defer fs.RUnlock()
+ return fs.open[fh]
+}
+
+func (fs *keepFS) Init() {
+ defer fs.debugPanics()
+ fs.root = fs.Client.SiteFileSystem(fs.KeepClient)
+ fs.root.MountProject("home", "")
+ if fs.ready != nil {
+ close(fs.ready)
+ }
+}
+
+func (fs *keepFS) Create(path string, flags int, mode uint32) (errc int, fh uint64) {
+ defer fs.debugPanics()
+ if fs.ReadOnly {
+ return -fuse.EROFS, invalidFH
+ }
+ f, err := fs.root.OpenFile(path, flags|os.O_CREATE, os.FileMode(mode))
+ if err == os.ErrExist {
+ return -fuse.EEXIST, invalidFH
+ } else if err != nil {
+ return -fuse.EINVAL, invalidFH
+ }
+ return 0, fs.newFH(f)
+}
+
+func (fs *keepFS) Open(path string, flags int) (errc int, fh uint64) {
+ defer fs.debugPanics()
+ if fs.ReadOnly && flags&(os.O_RDWR|os.O_WRONLY|os.O_CREATE) != 0 {
+ return -fuse.EROFS, invalidFH
+ }
+ f, err := fs.root.OpenFile(path, flags, 0)
+ if err != nil {
+ return -fuse.ENOENT, invalidFH
+ } else if fi, err := f.Stat(); err != nil {
+ return -fuse.EIO, invalidFH
+ } else if fi.IsDir() {
+ f.Close()
+ return -fuse.EISDIR, invalidFH
+ }
+ return 0, fs.newFH(f)
+}
+
+func (fs *keepFS) Utimens(path string, tmsp []fuse.Timespec) int {
+ defer fs.debugPanics()
+ if fs.ReadOnly {
+ return -fuse.EROFS
+ }
+ f, err := fs.root.OpenFile(path, 0, 0)
+ if err != nil {
+ return fs.errCode(err)
+ }
+ f.Close()
+ return 0
+}
+
+func (fs *keepFS) errCode(err error) int {
+ if os.IsNotExist(err) {
+ return -fuse.ENOENT
+ }
+ switch err {
+ case os.ErrExist:
+ return -fuse.EEXIST
+ case arvados.ErrInvalidArgument:
+ return -fuse.EINVAL
+ case arvados.ErrInvalidOperation:
+ return -fuse.ENOSYS
+ case arvados.ErrDirectoryNotEmpty:
+ return -fuse.ENOTEMPTY
+ case nil:
+ return 0
+ default:
+ return -fuse.EIO
+ }
+}
+
+func (fs *keepFS) Mkdir(path string, mode uint32) int {
+ defer fs.debugPanics()
+ if fs.ReadOnly {
+ return -fuse.EROFS
+ }
+ f, err := fs.root.OpenFile(path, os.O_CREATE|os.O_EXCL, os.FileMode(mode)|os.ModeDir)
+ if err != nil {
+ return fs.errCode(err)
+ }
+ f.Close()
+ return 0
+}
+
+func (fs *keepFS) Opendir(path string) (errc int, fh uint64) {
+ defer fs.debugPanics()
+ f, err := fs.root.OpenFile(path, 0, 0)
+ if err != nil {
+ return fs.errCode(err), invalidFH
+ } else if fi, err := f.Stat(); err != nil {
+ return fs.errCode(err), invalidFH
+ } else if !fi.IsDir() {
+ f.Close()
+ return -fuse.ENOTDIR, invalidFH
+ }
+ return 0, fs.newFH(f)
+}
+
+func (fs *keepFS) Releasedir(path string, fh uint64) (errc int) {
+ defer fs.debugPanics()
+ return fs.Release(path, fh)
+}
+
+func (fs *keepFS) Rmdir(path string) int {
+ defer fs.debugPanics()
+ return fs.errCode(fs.root.Remove(path))
+}
+
+func (fs *keepFS) Release(path string, fh uint64) (errc int) {
+ defer fs.debugPanics()
+ fs.Lock()
+ defer fs.Unlock()
+ defer delete(fs.open, fh)
+ if f := fs.open[fh]; f != nil {
+ err := f.Close()
+ if err != nil {
+ return -fuse.EIO
+ }
+ }
+ return 0
+}
+
+func (fs *keepFS) Rename(oldname, newname string) (errc int) {
+ defer fs.debugPanics()
+ if fs.ReadOnly {
+ return -fuse.EROFS
+ }
+ return fs.errCode(fs.root.Rename(oldname, newname))
+}
+
+func (fs *keepFS) Unlink(path string) (errc int) {
+ defer fs.debugPanics()
+ if fs.ReadOnly {
+ return -fuse.EROFS
+ }
+ return fs.errCode(fs.root.Remove(path))
+}
+
+func (fs *keepFS) Truncate(path string, size int64, fh uint64) (errc int) {
+ defer fs.debugPanics()
+ if fs.ReadOnly {
+ return -fuse.EROFS
+ }
+
+ // Sometimes fh is a valid filehandle and we don't need to
+ // waste a name lookup.
+ if f := fs.lookupFH(fh); f != nil {
+ return fs.errCode(f.Truncate(size))
+ }
+
+ // Other times, fh is invalid and we need to lookup path.
+ f, err := fs.root.OpenFile(path, os.O_RDWR, 0)
+ if err != nil {
+ return fs.errCode(err)
+ }
+ defer f.Close()
+ return fs.errCode(f.Truncate(size))
+}
+
+func (fs *keepFS) Getattr(path string, stat *fuse.Stat_t, fh uint64) (errc int) {
+ defer fs.debugPanics()
+ var fi os.FileInfo
+ var err error
+ if f := fs.lookupFH(fh); f != nil {
+ // Valid filehandle -- ignore path.
+ fi, err = f.Stat()
+ } else {
+ // Invalid filehandle -- lookup path.
+ fi, err = fs.root.Stat(path)
+ }
+ if err != nil {
+ return fs.errCode(err)
+ }
+ fs.fillStat(stat, fi)
+ return 0
+}
+
+func (fs *keepFS) Chmod(path string, mode uint32) (errc int) {
+ if fs.ReadOnly {
+ return -fuse.EROFS
+ }
+ if fi, err := fs.root.Stat(path); err != nil {
+ return fs.errCode(err)
+ } else if mode & ^uint32(fuse.S_IFREG|fuse.S_IFDIR|0777) != 0 {
+ // Refuse to set mode bits other than
+ // regfile/dir/perms
+ return -fuse.ENOSYS
+ } else if (fi.Mode()&os.ModeDir != 0) != (mode&fuse.S_IFDIR != 0) {
+ // Refuse to transform a regular file to a dir, or
+ // vice versa
+ return -fuse.ENOSYS
+ }
+ // As long as the change isn't nonsense, chmod is a no-op,
+ // because we don't save permission bits.
+ return 0
+}
+
+func (fs *keepFS) fillStat(stat *fuse.Stat_t, fi os.FileInfo) {
+ defer fs.debugPanics()
+ var m uint32
+ if fi.IsDir() {
+ m = m | fuse.S_IFDIR
+ } else {
+ m = m | fuse.S_IFREG
+ }
+ m = m | uint32(fi.Mode()&os.ModePerm)
+ stat.Mode = m
+ stat.Nlink = 1
+ stat.Size = fi.Size()
+ t := fuse.NewTimespec(fi.ModTime())
+ stat.Mtim = t
+ stat.Ctim = t
+ stat.Atim = t
+ stat.Birthtim = t
+ stat.Blksize = 1024
+ stat.Blocks = (stat.Size + stat.Blksize - 1) / stat.Blksize
+ if fs.Uid > 0 && int64(fs.Uid) < 1<<31 {
+ stat.Uid = uint32(fs.Uid)
+ }
+ if fs.Gid > 0 && int64(fs.Gid) < 1<<31 {
+ stat.Gid = uint32(fs.Gid)
+ }
+}
+
+func (fs *keepFS) Write(path string, buf []byte, ofst int64, fh uint64) (n int) {
+ defer fs.debugPanics()
+ if fs.ReadOnly {
+ return -fuse.EROFS
+ }
+ f := fs.lookupFH(fh)
+ if f == nil {
+ return -fuse.EBADF
+ }
+ f.Lock()
+ defer f.Unlock()
+ if _, err := f.Seek(ofst, io.SeekStart); err != nil {
+ return fs.errCode(err)
+ }
+ n, err := f.Write(buf)
+ if err != nil {
+ log.Printf("error writing %q: %s", path, err)
+ return fs.errCode(err)
+ }
+ return n
+}
+
+func (fs *keepFS) Read(path string, buf []byte, ofst int64, fh uint64) (n int) {
+ defer fs.debugPanics()
+ f := fs.lookupFH(fh)
+ if f == nil {
+ return -fuse.EBADF
+ }
+ f.Lock()
+ defer f.Unlock()
+ if _, err := f.Seek(ofst, io.SeekStart); err != nil {
+ return fs.errCode(err)
+ }
+ n, err := f.Read(buf)
+ for err == nil && n < len(buf) {
+ // f is an io.Reader ("If some data is available but
+ // not len(p) bytes, Read conventionally returns what
+ // is available instead of waiting for more") -- but
+ // our caller requires us to either fill buf or reach
+ // EOF.
+ done := n
+ n, err = f.Read(buf[done:])
+ n += done
+ }
+ if err != nil && err != io.EOF {
+ log.Printf("error reading %q: %s", path, err)
+ return fs.errCode(err)
+ }
+ return n
+}
+
+func (fs *keepFS) Readdir(path string,
+ fill func(name string, stat *fuse.Stat_t, ofst int64) bool,
+ ofst int64,
+ fh uint64) (errc int) {
+ defer fs.debugPanics()
+ f := fs.lookupFH(fh)
+ if f == nil {
+ return -fuse.EBADF
+ }
+ fill(".", nil, 0)
+ fill("..", nil, 0)
+ var stat fuse.Stat_t
+ fis, err := f.Readdir(-1)
+ if err != nil {
+ return fs.errCode(err)
+ }
+ for _, fi := range fis {
+ fs.fillStat(&stat, fi)
+ fill(fi.Name(), &stat, 0)
+ }
+ return 0
+}
+
+func (fs *keepFS) Fsync(path string, datasync bool, fh uint64) int {
+ defer fs.debugPanics()
+ f := fs.lookupFH(fh)
+ if f == nil {
+ return -fuse.EBADF
+ }
+ return fs.errCode(f.Sync())
+}
+
+func (fs *keepFS) Fsyncdir(path string, datasync bool, fh uint64) int {
+ return fs.Fsync(path, datasync, fh)
+}
+
+// debugPanics (when deferred by keepFS handlers) prints an error and
+// stack trace on stderr when a handler crashes. (Without this,
+// cgofuse recovers from panics silently and returns EIO.)
+func (fs *keepFS) debugPanics() {
+ if err := recover(); err != nil {
+ log.Printf("(%T) %v", err, err)
+ debug.PrintStack()
+ panic(err)
+ }
+}
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package mount
+
+import (
+ "testing"
+
+ "git.arvados.org/arvados.git/sdk/go/arvados"
+ "git.arvados.org/arvados.git/sdk/go/arvadosclient"
+ "git.arvados.org/arvados.git/sdk/go/keepclient"
+ "github.com/arvados/cgofuse/fuse"
+ check "gopkg.in/check.v1"
+)
+
+// Gocheck boilerplate
+func Test(t *testing.T) {
+ check.TestingT(t)
+}
+
+var _ = check.Suite(&FSSuite{})
+
+type FSSuite struct{}
+
+func (*FSSuite) TestFuseInterface(c *check.C) {
+ var _ fuse.FileSystemInterface = &keepFS{}
+}
+
+func (*FSSuite) TestOpendir(c *check.C) {
+ client := arvados.NewClientFromEnv()
+ ac, err := arvadosclient.New(client)
+ c.Assert(err, check.IsNil)
+ kc, err := keepclient.MakeKeepClient(ac)
+ c.Assert(err, check.IsNil)
+
+ var fs fuse.FileSystemInterface = &keepFS{
+ Client: client,
+ KeepClient: kc,
+ }
+ fs.Init()
+ errc, fh := fs.Opendir("/by_id")
+ c.Check(errc, check.Equals, 0)
+ c.Check(fh, check.Not(check.Equals), uint64(0))
+ c.Check(fh, check.Not(check.Equals), invalidFH)
+ errc, fh = fs.Opendir("/bogus")
+ c.Check(errc, check.Equals, -fuse.ENOENT)
+ c.Check(fh, check.Equals, invalidFH)
+}
for req in job_reqs:
tool.requirements.append(req)
- def arv_executor(self, tool, job_order, runtimeContext, logger=None):
+ def arv_executor(self, updated_tool, job_order, runtimeContext, logger=None):
self.debug = runtimeContext.debug
- tool.visit(self.check_features)
+ updated_tool.visit(self.check_features)
self.project_uuid = runtimeContext.project_uuid
self.pipeline = None
raise Exception("--submit-request-uuid requires containers API, but using '{}' api".format(self.work_api))
if not runtimeContext.name:
- runtimeContext.name = self.name = tool.tool.get("label") or tool.metadata.get("label") or os.path.basename(tool.tool["id"])
+ runtimeContext.name = self.name = updated_tool.tool.get("label") or updated_tool.metadata.get("label") or os.path.basename(updated_tool.tool["id"])
# Upload local file references in the job order.
job_order = upload_job_order(self, "%s input" % runtimeContext.name,
- tool, job_order)
+ updated_tool, job_order)
+
+ # the last clause means: if it is a command line tool, and we
+ # are going to wait for the result, and always_submit_runner
+ # is false, then we don't submit a runner process.
submitting = (runtimeContext.update_workflow or
runtimeContext.create_workflow or
(runtimeContext.submit and not
- (tool.tool["class"] == "CommandLineTool" and
+ (updated_tool.tool["class"] == "CommandLineTool" and
runtimeContext.wait and
not runtimeContext.always_submit_runner)))
if submitting:
# Document may have been auto-updated. Reload the original
# document with updating disabled because we want to
- # submit the original document, not the auto-updated one.
- tool = load_tool(tool.tool["id"], loadingContext)
+ # submit the document with its original CWL version, not
+ # the auto-updated one.
+ tool = load_tool(updated_tool.tool["id"], loadingContext)
+ else:
+ tool = updated_tool
# Upload direct dependencies of workflow steps, get back mapping of files to keep references.
# Also uploads docker images.
if runtimeContext.submit:
# Submit a runner job to run the workflow for us.
if self.work_api == "containers":
- if tool.tool["class"] == "CommandLineTool" and runtimeContext.wait and (not runtimeContext.always_submit_runner):
- runtimeContext.runnerjob = tool.tool["id"]
+ if submitting:
+ tool = RunnerContainer(self, updated_tool,
+ tool, loadingContext, runtimeContext.enable_reuse,
+ self.output_name,
+ self.output_tags,
+ submit_runner_ram=runtimeContext.submit_runner_ram,
+ name=runtimeContext.name,
+ on_error=runtimeContext.on_error,
+ submit_runner_image=runtimeContext.submit_runner_image,
+ intermediate_output_ttl=runtimeContext.intermediate_output_ttl,
+ merged_map=merged_map,
+ priority=runtimeContext.priority,
+ secret_store=self.secret_store,
+ collection_cache_size=runtimeContext.collection_cache_size,
+ collection_cache_is_default=self.should_estimate_cache_size)
else:
- tool = RunnerContainer(self, tool, loadingContext, runtimeContext.enable_reuse,
- self.output_name,
- self.output_tags,
- submit_runner_ram=runtimeContext.submit_runner_ram,
- name=runtimeContext.name,
- on_error=runtimeContext.on_error,
- submit_runner_image=runtimeContext.submit_runner_image,
- intermediate_output_ttl=runtimeContext.intermediate_output_ttl,
- merged_map=merged_map,
- priority=runtimeContext.priority,
- secret_store=self.secret_store,
- collection_cache_size=runtimeContext.collection_cache_size,
- collection_cache_is_default=self.should_estimate_cache_size)
+ runtimeContext.runnerjob = tool.tool["id"]
if runtimeContext.cwl_runner_job is not None:
self.uuid = runtimeContext.cwl_runner_job.get('uuid')
"""Base class for runner processes, which submit an instance of
arvados-cwl-runner and wait for the final result."""
- def __init__(self, runner, tool, loadingContext, enable_reuse,
+ def __init__(self, runner, updated_tool,
+ tool, loadingContext, enable_reuse,
output_name, output_tags, submit_runner_ram=0,
name=None, on_error=None, submit_runner_image=None,
intermediate_output_ttl=0, merged_map=None,
collection_cache_is_default=True):
loadingContext = loadingContext.copy()
- loadingContext.metadata = loadingContext.metadata.copy()
- loadingContext.metadata["cwlVersion"] = INTERNAL_VERSION
+ loadingContext.metadata = updated_tool.metadata.copy()
- super(Runner, self).__init__(tool.tool, loadingContext)
+ super(Runner, self).__init__(updated_tool.tool, loadingContext)
self.arvrunner = runner
self.embedded_tool = tool
'bin/arvados-cwl-runner',
],
# Note that arvados/build/run-build-packages.sh looks at this
- # file to determine what version of cwltool and schema-salad to build.
+ # file to determine what version of cwltool and schema-salad to
+ # build.
install_requires=[
'cwltool==1.0.20190831161204',
'schema-salad==4.5.20190815125611',
],
test_suite='tests',
tests_require=[
- 'mock>=1.0',
+ 'mock>=1.0,<4',
'subprocess32>=3.5.1',
],
- zip_safe=True
- )
+ zip_safe=True,
+)
stubs.expect_container_request_uuid + '\n')
self.assertEqual(exited, 0)
+
+ @stubs
+ def test_submit_container_tool(self, stubs):
+ # test for issue #16139
+ exited = arvados_cwl.main(
+ ["--submit", "--no-wait", "--api=containers", "--debug",
+ "tests/tool/tool_with_sf.cwl", "tests/tool/tool_with_sf.yml"],
+ stubs.capture_stdout, sys.stderr, api_client=stubs.api, keep_client=stubs.keep_client)
+
+ self.assertEqual(stubs.capture_stdout.getvalue(),
+ stubs.expect_container_request_uuid + '\n')
+ self.assertEqual(exited, 0)
+
@stubs
def test_submit_container_no_reuse(self, stubs):
exited = arvados_cwl.main(
--- /dev/null
+clipper clupper
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Test case for arvados-cwl-runner
+#
+# Used to test whether scanning a tool file for dependencies (e.g. default
+# value blub.txt) and uploading to Keep works as intended.
+
+class: CommandLineTool
+cwlVersion: v1.0
+requirements:
+ - class: DockerRequirement
+ dockerPull: debian:8
+inputs:
+ - id: x
+ type: File
+ secondaryFiles:
+ - .cat
+ inputBinding:
+ valueFrom: $(self.path).cat
+ position: 1
+outputs: []
+baseCommand: cat
--- /dev/null
+x:
+ class: File
+ location: blub.txt
UUID string `json:"uuid"`
Select []string `json:"select"`
IncludeTrash bool `json:"include_trash"`
+ ForwardedFor string `json:"forwarded_for"`
}
type UntrashOptions struct {
ReplicationDesired *int `json:"replication_desired"`
StorageClassesDesired []string `json:"storage_classes_desired"`
StorageClassesConfirmed []string `json:"storage_classes_confirmed"`
- StorageClassesConfirmedAt time.Time `json:"storage_classes_confirmed_at"`
+ StorageClassesConfirmedAt *time.Time `json:"storage_classes_confirmed_at"`
DeleteAt *time.Time `json:"delete_at"`
IsTrashed bool `json:"is_trashed"`
Properties map[string]interface{} `json:"properties"`
WritableBy []string `json:"writable_by,omitempty"`
+ FileCount int `json:"file_count"`
+ FileSizeTotal int64 `json:"file_size_total"`
+ Version int `json:"version"`
+ PreserveVersion bool `json:"preserve_version"`
+ CurrentVersionUUID string `json:"current_version_uuid"`
+ Description string `json:"description"`
}
func (c Collection) resourceName() string {
VocabularyURL string
WelcomePageHTML string
InactivePageHTML string
+ SSHHelpPageHTML string
}
ForceLegacyAPI14 bool
ActiveTokenUUID = "zzzzz-gj3su-077z32aux8dg2s1"
ActiveTokenV2 = "v2/zzzzz-gj3su-077z32aux8dg2s1/3kg6k6lzmp9kj5cpkcoxie963cmvjahbt2fod9zru30k1jqdmi"
AdminToken = "4axaw8zxe0qm22wa6urpp5nskcne8z88cvbupv653y1njyi05h"
+ AdminTokenUUID = "zzzzz-gj3su-027z32aux8dg2s1"
AnonymousToken = "4kg6k6lzmp9kj4cpkcoxie964cmvjahbt4fod9zru44k4jqdmi"
DataManagerToken = "320mkve8qkswstz7ff61glpk3mhgghmg67wmic7elw4z41pke1"
SystemRootToken = "systemusertesttoken1234567890aoeuidhtnsqjkxbmwvzpy"
WorkflowWithDefinitionYAMLUUID = "zzzzz-7fd4e-validworkfloyml"
CollectionReplicationDesired2Confirmed2UUID = "zzzzz-4zz18-434zv1tnnf2rygp"
+
+ ActiveUserCanReadAllUsersLinkUUID = "zzzzz-o0j2j-ctbysaduejxfrs5"
+
+ TrustedWorkbenchAPIClientUUID = "zzzzz-ozdt8-teyxzyd8qllg11h"
+
+ AdminAuthorizedKeysUUID = "zzzzz-fngyi-12nc9ov4osp8nae"
+
+ CrunchstatForRunningJobLogUUID = "zzzzz-57u5n-tmymyrojrbtnxh1"
+
+ IdleNodeUUID = "zzzzz-7ekkf-2z3mc76g2q73aio"
+
+ TestVMUUID = "zzzzz-2x53u-382brsig8rp3064"
+
+ CollectionWithUniqueWordsUUID = "zzzzz-4zz18-mnt690klmb51aud"
)
// PathologicalManifest : A valid manifest designed to test
func logResponse(w *responseTimer, req *http.Request, lgr *logrus.Entry) {
if tStart, ok := req.Context().Value(&requestTimeContextKey).(time.Time); ok {
tDone := time.Now()
+ writeTime := w.writeTime
+ if !w.wrote {
+ // Empty response body. Header was sent when
+ // handler exited.
+ writeTime = tDone
+ }
lgr = lgr.WithFields(logrus.Fields{
"timeTotal": stats.Duration(tDone.Sub(tStart)),
- "timeToStatus": stats.Duration(w.writeTime.Sub(tStart)),
- "timeWriteBody": stats.Duration(tDone.Sub(w.writeTime)),
+ "timeToStatus": stats.Duration(writeTime.Sub(tStart)),
+ "timeWriteBody": stats.Duration(tDone.Sub(writeTime)),
})
}
respCode := w.WroteStatus()
],
test_suite='tests',
tests_require=['pbr<1.7.0', 'mock>=1.0', 'python-pam'],
- zip_safe=False
- )
+ zip_safe=False,
+)
svc.api_token = token
svc.insecure = insecure
svc.request_id = request_id
+ svc.config = lambda: util.get_config_once(svc)
kwargs['http'].max_request_size = svc._rootDesc.get('maxRequestSize', 0)
kwargs['http'].cache = None
kwargs['http']._request_id = lambda: svc.request_id or util.new_request_id()
rid += chr(c+ord('a')-10)
n = n // 36
return rid
+
+def get_config_once(svc):
+ if not svc._rootDesc.get('resources')['configs']:
+ # Old API server version, no config export endpoint
+ return {}
+ if not hasattr(svc, '_cached_config'):
+ svc._cached_config = svc.configs().get().execute()
+ return svc._cached_config
'Programming Language :: Python :: 3',
],
test_suite='tests',
- tests_require=['pbr<1.7.0', 'mock>=1.0', 'PyYAML'],
+ tests_require=['pbr<1.7.0', 'mock>=1.0,<4', 'PyYAML'],
zip_safe=False
)
return
end
(managed_props.keys - self.properties.keys).each do |key|
- if managed_props[key].has_key?('Value')
- self.properties[key] = managed_props[key]['Value']
- elsif managed_props[key]['Function'].andand == 'original_owner'
+ if managed_props[key]['Function'] == 'original_owner'
self.properties[key] = self.user_owner_uuid
+ elsif managed_props[key]['Value']
+ self.properties[key] = managed_props[key]['Value']
else
logger.warn "Unidentified default property definition '#{key}': #{managed_props[key].inspect}"
end
# (same order as Container#handle_completed). Locking always
# reloads the Container and ContainerRequest records.
c = Container.find_by_uuid(container_uuid)
- c.lock!
+ c.lock! if !c.nil?
self.lock!
- if container_uuid != c.uuid
+ if !c.nil? && container_uuid != c.uuid
# After locking, we've noticed a race, the container_uuid is
# different than the container record we just loaded. This
# can happen if Container#handle_completed scheduled a new
redo
end
- if state == Committed && c.final?
- # The current container is
- act_as_system_user do
- leave_modified_by_user_alone do
- finalize!
+ if !c.nil?
+ if state == Committed && c.final?
+ # The current container is
+ act_as_system_user do
+ leave_modified_by_user_alone do
+ finalize!
+ end
end
end
+ elsif state == Committed
+ # Behave as if the container is cancelled
+ update_attributes!(state: Final)
end
return true
end
# finished/cancelled.
def finalize!
container = Container.find_by_uuid(container_uuid)
- update_collections(container: container)
-
- if container.state == Container::Complete
- log_col = Collection.where(portable_data_hash: container.log).first
- if log_col
- # Need to save collection
- completed_coll = Collection.new(
- owner_uuid: self.owner_uuid,
- name: "Container log for container #{container_uuid}",
- properties: {
- 'type' => 'log',
- 'container_request' => self.uuid,
- 'container_uuid' => container_uuid,
- },
- portable_data_hash: log_col.portable_data_hash,
- manifest_text: log_col.manifest_text)
- completed_coll.save_with_unique_name!
+ if !container.nil?
+ update_collections(container: container)
+
+ if container.state == Container::Complete
+ log_col = Collection.where(portable_data_hash: container.log).first
+ if log_col
+ # Need to save collection
+ completed_coll = Collection.new(
+ owner_uuid: self.owner_uuid,
+ name: "Container log for container #{container_uuid}",
+ properties: {
+ 'type' => 'log',
+ 'container_request' => self.uuid,
+ 'container_uuid' => container_uuid,
+ },
+ portable_data_hash: log_col.portable_data_hash,
+ manifest_text: log_col.manifest_text)
+ completed_coll.save_with_unique_name!
+ end
end
end
-
update_attributes!(state: Final)
end
collections.each do |out_type|
pdh = container.send(out_type)
next if pdh.nil?
+ c = Collection.where(portable_data_hash: pdh).first
+ next if c.nil?
+ manifest = c.manifest_text
+
coll_name = "Container #{out_type} for request #{uuid}"
trash_at = nil
if out_type == 'output'
trash_at = db_current_time + self.output_ttl
end
end
- manifest = Collection.where(portable_data_hash: pdh).first.manifest_text
coll_uuid = self.send(out_type + '_uuid')
coll = coll_uuid.nil? ? nil : Collection.where(uuid: coll_uuid).first
if self.container_count_changed?
errors.add :container_count, "cannot be updated directly."
return false
- else
- self.container_count += 1
- if self.container_uuid_was
- old_container = Container.find_by_uuid(self.container_uuid_was)
- old_logs = Collection.where(portable_data_hash: old_container.log).first
- if old_logs
- log_coll = self.log_uuid.nil? ? nil : Collection.where(uuid: self.log_uuid).first
- if self.log_uuid.nil?
- log_coll = Collection.new(
- owner_uuid: self.owner_uuid,
- name: coll_name = "Container log for request #{uuid}",
- manifest_text: "")
- end
+ end
- # copy logs from old container into CR's log collection
- src = Arv::Collection.new(old_logs.manifest_text)
- dst = Arv::Collection.new(log_coll.manifest_text)
- dst.cp_r("./", "log for container #{old_container.uuid}", src)
- manifest = dst.manifest_text
-
- log_coll.assign_attributes(
- portable_data_hash: Digest::MD5.hexdigest(manifest) + '+' + manifest.bytesize.to_s,
- manifest_text: manifest)
- log_coll.save_with_unique_name!
- self.log_uuid = log_coll.uuid
- end
- end
+ self.container_count += 1
+ return if self.container_uuid_was.nil?
+
+ old_container = Container.find_by_uuid(self.container_uuid_was)
+ return if old_container.nil?
+
+ old_logs = Collection.where(portable_data_hash: old_container.log).first
+ return if old_logs.nil?
+
+ log_coll = self.log_uuid.nil? ? nil : Collection.where(uuid: self.log_uuid).first
+ if self.log_uuid.nil?
+ log_coll = Collection.new(
+ owner_uuid: self.owner_uuid,
+ name: coll_name = "Container log for request #{uuid}",
+ manifest_text: "")
end
+
+ # copy logs from old container into CR's log collection
+ src = Arv::Collection.new(old_logs.manifest_text)
+ dst = Arv::Collection.new(log_coll.manifest_text)
+ dst.cp_r("./", "log for container #{old_container.uuid}", src)
+ manifest = dst.manifest_text
+
+ log_coll.assign_attributes(
+ portable_data_hash: Digest::MD5.hexdigest(manifest) + '+' + manifest.bytesize.to_s,
+ manifest_text: manifest)
+ log_coll.save_with_unique_name!
+ self.log_uuid = log_coll.uuid
end
end
owner_uuid: zzzzz-tpzed-000000000000000
uuid: zzzzz-tpzed-xurymjxw79nv3jz
email: active-user@arvados.local
+ modified_by_client_uuid: zzzzz-ozdt8-teyxzyd8qllg11h
+ modified_by_user_uuid: zzzzz-tpzed-xurymjxw79nv3jz
first_name: Active
last_name: User
identity_url: https://active-user.openid.local
is_active: true
is_admin: false
+ modified_at: 2015-03-26 12:34:56.789000000 Z
username: active
prefs:
profile:
assert_equal log.owner_uuid, project.uuid, "Container log should be copied to #{project.uuid}"
end
+ # This tests bug report #16144
+ test "Request is finalized when its container is completed even when log & output don't exist" do
+ set_user_from_auth :active
+ project = groups(:private)
+ cr = create_minimal_req!(owner_uuid: project.uuid,
+ priority: 1,
+ state: "Committed")
+ assert_equal users(:active).uuid, cr.modified_by_user_uuid
+
+ output_pdh = '1f4b0bc7583c2a7f9102c395f4ffc5e3+45'
+ log_pdh = 'fa7aeb5140e2848d39b416daeef4ffc5+45'
+
+ c = act_as_system_user do
+ c = Container.find_by_uuid(cr.container_uuid)
+ c.update_attributes!(state: Container::Locked)
+ c.update_attributes!(state: Container::Running,
+ output: output_pdh,
+ log: log_pdh)
+ c
+ end
+
+ cr.reload
+ assert_equal "Committed", cr.state
+
+ act_as_system_user do
+ Collection.where(portable_data_hash: output_pdh).delete_all
+ Collection.where(portable_data_hash: log_pdh).delete_all
+ c.update_attributes!(state: Container::Complete)
+ end
+
+ cr.reload
+ assert_equal "Final", cr.state
+ end
+
+ # This tests bug report #16144
+ test "Can destroy CR even if its container doesn't exist" do
+ set_user_from_auth :active
+ project = groups(:private)
+ cr = create_minimal_req!(owner_uuid: project.uuid,
+ priority: 1,
+ state: "Committed")
+ assert_equal users(:active).uuid, cr.modified_by_user_uuid
+
+ c = act_as_system_user do
+ c = Container.find_by_uuid(cr.container_uuid)
+ c.update_attributes!(state: Container::Locked)
+ c.update_attributes!(state: Container::Running)
+ c
+ end
+
+ cr.reload
+ assert_equal "Committed", cr.state
+
+ cr_uuid = cr.uuid
+ act_as_system_user do
+ Container.find_by_uuid(cr.container_uuid).destroy
+ cr.destroy
+ end
+ assert_nil ContainerRequest.find_by_uuid(cr_uuid)
+ end
+
test "Container makes container request, then is cancelled" do
set_user_from_auth :active
cr = create_minimal_req!(priority: 5, state: "Committed", container_count_max: 1)
[Service]
Type=notify
ExecStart=/usr/bin/crunch-dispatch-slurm
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=1
LimitNOFILE=1000000
LLFUSE_VERSION_0 = llfuse.__version__.startswith('0')
-from .fusedir import sanitize_filename, Directory, CollectionDirectory, TmpCollectionDirectory, MagicDirectory, TagsDirectory, ProjectDirectory, SharedDirectory, CollectionDirectoryBase
+from .fusedir import Directory, CollectionDirectory, TmpCollectionDirectory, MagicDirectory, TagsDirectory, ProjectDirectory, SharedDirectory, CollectionDirectoryBase
from .fusefile import StringFile, FuseArvadosFile
_logger = logging.getLogger('arvados.arvados_fuse')
return
e = self.operations.inodes.add_entry(Directory(
- llfuse.ROOT_INODE, self.operations.inodes))
+ llfuse.ROOT_INODE, self.operations.inodes, self.api.config))
dir_args[0] = e.inode
for name in self.args.mount_by_id:
# appear as underscores in the fuse mount.)
_disallowed_filename_characters = re.compile('[\x00/]')
-# '.' and '..' are not reachable if API server is newer than #6277
-def sanitize_filename(dirty):
- """Replace disallowed filename characters with harmless "_"."""
- if dirty is None:
- return None
- elif dirty == '':
- return '_'
- elif dirty == '.':
- return '_'
- elif dirty == '..':
- return '__'
- else:
- return _disallowed_filename_characters.sub('_', dirty)
-
class Directory(FreshBase):
"""Generic directory object, backed by a dict.
and the value referencing a File or Directory object.
"""
- def __init__(self, parent_inode, inodes):
+ def __init__(self, parent_inode, inodes, apiconfig):
"""parent_inode is the integer inode number"""
super(Directory, self).__init__()
raise Exception("parent_inode should be an int")
self.parent_inode = parent_inode
self.inodes = inodes
+ self.apiconfig = apiconfig
self._entries = {}
self._mtime = time.time()
- # Overriden by subclasses to implement logic to update the entries dict
- # when the directory is stale
+ def forward_slash_subst(self):
+ if not hasattr(self, '_fsns'):
+ self._fsns = None
+ config = self.apiconfig()
+ try:
+ self._fsns = config["Collections"]["ForwardSlashNameSubstitution"]
+ except KeyError:
+ # old API server with no FSNS config
+ self._fsns = '_'
+ else:
+ if self._fsns == '' or self._fsns == '/':
+ self._fsns = None
+ return self._fsns
+
+ def unsanitize_filename(self, incoming):
+ """Replace ForwardSlashNameSubstitution value with /"""
+ fsns = self.forward_slash_subst()
+ if isinstance(fsns, str):
+ return incoming.replace(fsns, '/')
+ else:
+ return incoming
+
+ def sanitize_filename(self, dirty):
+ """Replace disallowed filename characters according to
+ ForwardSlashNameSubstitution in self.api_config."""
+ # '.' and '..' are not reachable if API server is newer than #6277
+ if dirty is None:
+ return None
+ elif dirty == '':
+ return '_'
+ elif dirty == '.':
+ return '_'
+ elif dirty == '..':
+ return '__'
+ else:
+ fsns = self.forward_slash_subst()
+ if isinstance(fsns, str):
+ dirty = dirty.replace('/', fsns)
+ return _disallowed_filename_characters.sub('_', dirty)
+
+
+ # Overridden by subclasses to implement logic to update the
+ # entries dict when the directory is stale
@use_counter
def update(self):
pass
self._entries = {}
changed = False
for i in items:
- name = sanitize_filename(fn(i))
+ name = self.sanitize_filename(fn(i))
if name:
if name in oldentries and same(oldentries[name], i):
# move existing directory entry over
"""
- def __init__(self, parent_inode, inodes, collection):
- super(CollectionDirectoryBase, self).__init__(parent_inode, inodes)
+ def __init__(self, parent_inode, inodes, apiconfig, collection):
+ super(CollectionDirectoryBase, self).__init__(parent_inode, inodes, apiconfig)
+ self.apiconfig = apiconfig
self.collection = collection
def new_entry(self, name, item, mtime):
- name = sanitize_filename(name)
+ name = self.sanitize_filename(name)
if hasattr(item, "fuse_entry") and item.fuse_entry is not None:
if item.fuse_entry.dead is not True:
raise Exception("Can only reparent dead inode entry")
item.fuse_entry.dead = False
self._entries[name] = item.fuse_entry
elif isinstance(item, arvados.collection.RichCollectionBase):
- self._entries[name] = self.inodes.add_entry(CollectionDirectoryBase(self.inode, self.inodes, item))
+ self._entries[name] = self.inodes.add_entry(CollectionDirectoryBase(self.inode, self.inodes, self.apiconfig, item))
self._entries[name].populate(mtime)
else:
self._entries[name] = self.inodes.add_entry(FuseArvadosFile(self.inode, item, mtime))
def on_event(self, event, collection, name, item):
if collection == self.collection:
- name = sanitize_filename(name)
+ name = self.sanitize_filename(name)
_logger.debug("collection notify %s %s %s %s", event, collection, name, item)
with llfuse.lock:
if event == arvados.collection.ADD:
"""Represents the root of a directory tree representing a collection."""
def __init__(self, parent_inode, inodes, api, num_retries, collection_record=None, explicit_collection=None):
- super(CollectionDirectory, self).__init__(parent_inode, inodes, None)
+ super(CollectionDirectory, self).__init__(parent_inode, inodes, api.config, None)
self.api = api
self.num_retries = num_retries
self.collection_record_file = None
keep_client=api_client.keep,
num_retries=num_retries)
super(TmpCollectionDirectory, self).__init__(
- parent_inode, inodes, collection)
+ parent_inode, inodes, api_client.config, collection)
self.collection_record_file = None
self.populate(self.mtime())
""".lstrip()
def __init__(self, parent_inode, inodes, api, num_retries, pdh_only=False):
- super(MagicDirectory, self).__init__(parent_inode, inodes)
+ super(MagicDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self.pdh_only = pdh_only
e = self.inodes.add_entry(ProjectDirectory(
self.inode, self.inodes, self.api, self.num_retries, project[u'items'][0]))
else:
+ import sys
e = self.inodes.add_entry(CollectionDirectory(
self.inode, self.inodes, self.api, self.num_retries, k))
"""A special directory that contains as subdirectories all tags visible to the user."""
def __init__(self, parent_inode, inodes, api, num_retries, poll_time=60):
- super(TagsDirectory, self).__init__(parent_inode, inodes)
+ super(TagsDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self._poll = True
def __init__(self, parent_inode, inodes, api, num_retries, tag,
poll=False, poll_time=60):
- super(TagDirectory, self).__init__(parent_inode, inodes)
+ super(TagDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self.tag = tag
def __init__(self, parent_inode, inodes, api, num_retries, project_object,
poll=False, poll_time=60):
- super(ProjectDirectory, self).__init__(parent_inode, inodes)
+ super(ProjectDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self.project_object = project_object
elif self._full_listing or super(ProjectDirectory, self).__contains__(k):
return super(ProjectDirectory, self).__getitem__(k)
with llfuse.lock_released:
+ k2 = self.unsanitize_filename(k)
+ if k2 == k:
+ namefilter = ["name", "=", k]
+ else:
+ namefilter = ["name", "in", [k, k2]]
contents = self.api.groups().list(filters=[["owner_uuid", "=", self.project_uuid],
["group_class", "=", "project"],
- ["name", "=", k]],
- limit=1).execute(num_retries=self.num_retries)["items"]
+ namefilter],
+ limit=2).execute(num_retries=self.num_retries)["items"]
if not contents:
contents = self.api.collections().list(filters=[["owner_uuid", "=", self.project_uuid],
- ["name", "=", k]],
- limit=1).execute(num_retries=self.num_retries)["items"]
+ namefilter],
+ limit=2).execute(num_retries=self.num_retries)["items"]
if contents:
- name = sanitize_filename(self.namefn(contents[0]))
+ if len(contents) > 1 and contents[1]['name'] == k:
+ # If "foo/bar" and "foo[SUBST]bar" both exist, use
+ # "foo[SUBST]bar".
+ contents = [contents[1]]
+ name = self.sanitize_filename(self.namefn(contents[0]))
if name != k:
raise KeyError(k)
return self._add_entry(contents[0], name)
new_attrs = properties.get("new_attributes") or {}
old_attrs["uuid"] = ev["object_uuid"]
new_attrs["uuid"] = ev["object_uuid"]
- old_name = sanitize_filename(self.namefn(old_attrs))
- new_name = sanitize_filename(self.namefn(new_attrs))
+ old_name = self.sanitize_filename(self.namefn(old_attrs))
+ new_name = self.sanitize_filename(self.namefn(new_attrs))
# create events will have a new name, but not an old name
# delete events will have an old name, but not a new name
def __init__(self, parent_inode, inodes, api, num_retries, exclude,
poll=False, poll_time=60):
- super(SharedDirectory, self).__init__(parent_inode, inodes)
+ super(SharedDirectory, self).__init__(parent_inode, inodes, api.config)
self.api = api
self.num_retries = num_retries
self.current_user = api.users().current().execute(num_retries=num_retries)
import arvados_fuse as fuse
from . import run_test_server
+from .integration_test import IntegrationTest
from .mount_test_base import MountTestBase
logger = logging.getLogger('arvados.arv-mount')
llfuse.listdir(os.path.join(self.mounttmp, self.testcollection))
-class FuseUnitTest(unittest.TestCase):
+class SanitizeFilenameTest(MountTestBase):
def test_sanitize_filename(self):
+ pdir = fuse.ProjectDirectory(1, {}, self.api, 0, project_object=self.api.users().current().execute())
acceptable = [
"foo.txt",
".foo",
"//",
]
for f in acceptable:
- self.assertEqual(f, fuse.sanitize_filename(f))
+ self.assertEqual(f, pdir.sanitize_filename(f))
for f in unacceptable:
- self.assertNotEqual(f, fuse.sanitize_filename(f))
+ self.assertNotEqual(f, pdir.sanitize_filename(f))
# The sanitized filename should be the same length, though.
- self.assertEqual(len(f), len(fuse.sanitize_filename(f)))
+ self.assertEqual(len(f), len(pdir.sanitize_filename(f)))
# Special cases
- self.assertEqual("_", fuse.sanitize_filename(""))
- self.assertEqual("_", fuse.sanitize_filename("."))
- self.assertEqual("__", fuse.sanitize_filename(".."))
+ self.assertEqual("_", pdir.sanitize_filename(""))
+ self.assertEqual("_", pdir.sanitize_filename("."))
+ self.assertEqual("__", pdir.sanitize_filename(".."))
class FuseMagicTestPDHOnly(MountTestBase):
def test_with_default_by_id(self):
self.verify_pdh_only(skip_pdh_only=True)
+
+
+class SlashSubstitutionTest(IntegrationTest):
+ mnt_args = [
+ '--read-write',
+ '--mount-home', 'zzz',
+ ]
+
+ def setUp(self):
+ super(SlashSubstitutionTest, self).setUp()
+ self.api = arvados.safeapi.ThreadSafeApiCache(arvados.config.settings())
+ self.api.config = lambda: {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
+ self.testcoll = self.api.collections().create(body={"name": "foo/bar/baz"}).execute()
+ self.testcolleasy = self.api.collections().create(body={"name": "foo-bar-baz"}).execute()
+ self.fusename = 'foo[SLASH]bar[SLASH]baz'
+
+ @IntegrationTest.mount(argv=mnt_args)
+ @mock.patch('arvados.util.get_config_once')
+ def test_slash_substitution_before_listing(self, get_config_once):
+ get_config_once.return_value = {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
+ self.pool_test(os.path.join(self.mnt, 'zzz'), self.fusename)
+ self.checkContents()
+ @staticmethod
+ def _test_slash_substitution_before_listing(self, tmpdir, fusename):
+ with open(os.path.join(tmpdir, 'foo-bar-baz', 'waz'), 'w') as f:
+ f.write('xxx')
+ with open(os.path.join(tmpdir, fusename, 'waz'), 'w') as f:
+ f.write('foo')
+
+ @IntegrationTest.mount(argv=mnt_args)
+ @mock.patch('arvados.util.get_config_once')
+ def test_slash_substitution_after_listing(self, get_config_once):
+ get_config_once.return_value = {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
+ self.pool_test(os.path.join(self.mnt, 'zzz'), self.fusename)
+ self.checkContents()
+ @staticmethod
+ def _test_slash_substitution_after_listing(self, tmpdir, fusename):
+ with open(os.path.join(tmpdir, 'foo-bar-baz', 'waz'), 'w') as f:
+ f.write('xxx')
+ os.listdir(tmpdir)
+ with open(os.path.join(tmpdir, fusename, 'waz'), 'w') as f:
+ f.write('foo')
+
+ def checkContents(self):
+ self.assertRegexpMatches(self.api.collections().get(uuid=self.testcoll['uuid']).execute()['manifest_text'], ' acbd18db') # md5(foo)
+ self.assertRegexpMatches(self.api.collections().get(uuid=self.testcolleasy['uuid']).execute()['manifest_text'], ' f561aaf6') # md5(xxx)
+
+ @IntegrationTest.mount(argv=mnt_args)
+ @mock.patch('arvados.util.get_config_once')
+ def test_slash_substitution_conflict(self, get_config_once):
+ self.testcollconflict = self.api.collections().create(body={"name": self.fusename}).execute()
+ get_config_once.return_value = {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
+ self.pool_test(os.path.join(self.mnt, 'zzz'), self.fusename)
+ self.assertRegexpMatches(self.api.collections().get(uuid=self.testcollconflict['uuid']).execute()['manifest_text'], ' acbd18db') # md5(foo)
+ # foo/bar/baz collection unchanged, because it is masked by foo[SLASH]bar[SLASH]baz
+ self.assertEqual(self.api.collections().get(uuid=self.testcoll['uuid']).execute()['manifest_text'], '')
+ @staticmethod
+ def _test_slash_substitution_conflict(self, tmpdir, fusename):
+ with open(os.path.join(tmpdir, fusename, 'waz'), 'w') as f:
+ f.write('foo')
[Service]
Type=simple
ExecStart=/usr/bin/arvados-health
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=1
[Service]
Type=simple
ExecStart=/usr/bin/keep-balance -commit-pulls -commit-trash
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=10s
Nice=19
"flag"
"fmt"
"io"
+ "net/http"
"os"
"git.arvados.org/arvados.git/lib/config"
options.Dumper = dumper
}
- // Only pass along the version flag, which gets handled in RunCommand
+ // Drop our custom args that would be rejected by the generic
+ // service.Command
args = nil
+ dropFlag := map[string]bool{
+ "once": true,
+ "commit-pulls": true,
+ "commit-trash": true,
+ "dump": true,
+ }
flags.Visit(func(f *flag.Flag) {
- if f.Name == "version" {
+ if !dropFlag[f.Name] {
args = append(args, "-"+f.Name, f.Value.String())
}
})
}
srv := &Server{
+ Handler: http.NotFoundHandler(),
Cluster: cluster,
ArvClient: ac,
RunOptions: options,
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: AGPL-3.0
+
+package main
+
+import (
+ "bytes"
+ "io/ioutil"
+ "net"
+ "net/http"
+ "time"
+
+ check "gopkg.in/check.v1"
+)
+
+var _ = check.Suite(&mainSuite{})
+
+type mainSuite struct{}
+
+func (s *mainSuite) TestVersionFlag(c *check.C) {
+ var stdout, stderr bytes.Buffer
+ runCommand("keep-balance", []string{"-version"}, nil, &stdout, &stderr)
+ c.Check(stderr.String(), check.Equals, "")
+ c.Log(stdout.String())
+}
+
+func (s *mainSuite) TestHTTPServer(c *check.C) {
+ ln, err := net.Listen("tcp", ":0")
+ if err != nil {
+ c.Fatal(err)
+ }
+ _, p, err := net.SplitHostPort(ln.Addr().String())
+ ln.Close()
+ config := "Clusters:\n zzzzz:\n ManagementToken: abcdefg\n Services: {Keepbalance: {InternalURLs: {'http://localhost:" + p + "/': {}}}}\n"
+
+ var stdout bytes.Buffer
+ go runCommand("keep-balance", []string{"-config", "-"}, bytes.NewBufferString(config), &stdout, &stdout)
+ done := make(chan struct{})
+ go func() {
+ defer close(done)
+ for {
+ time.Sleep(time.Second / 10)
+ req, err := http.NewRequest(http.MethodGet, "http://:"+p+"/metrics", nil)
+ if err != nil {
+ c.Fatal(err)
+ return
+ }
+ req.Header.Set("Authorization", "Bearer abcdefg")
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ c.Logf("error %s", err)
+ continue
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ c.Logf("http status %d", resp.StatusCode)
+ continue
+ }
+ buf, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ c.Logf("read body: %s", err)
+ continue
+ }
+ c.Check(string(buf), check.Matches, `(?ms).*arvados_keepbalance_sweep_seconds_sum.*`)
+ return
+ }
+ }()
+ select {
+ case <-done:
+ case <-time.After(time.Second):
+ c.Log(stdout.String())
+ c.Fatal("timeout")
+ }
+
+ // Check non-metrics URL that gets passed through to us from
+ // service.Command
+ req, err := http.NewRequest(http.MethodGet, "http://:"+p+"/not-metrics", nil)
+ c.Assert(err, check.IsNil)
+ resp, err := http.DefaultClient.Do(req)
+ c.Check(err, check.IsNil)
+ defer resp.Body.Close()
+ c.Check(resp.StatusCode, check.Equals, http.StatusNotFound)
+}
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+case "$TARGET" in
+ centos*)
+ fpm_depends+=(mailcap)
+ ;;
+ debian* | ubuntu*)
+ fpm_depends+=(mime-support)
+ ;;
+esac
"git.arvados.org/arvados.git/lib/config"
"git.arvados.org/arvados.git/sdk/go/arvados"
+ "git.arvados.org/arvados.git/sdk/go/arvadosclient"
"git.arvados.org/arvados.git/sdk/go/arvadostest"
"git.arvados.org/arvados.git/sdk/go/auth"
"git.arvados.org/arvados.git/sdk/go/ctxlog"
s.Config = cfg
}
-func (s *UnitSuite) TestKeepClientBlockCache(c *check.C) {
- cfg := newConfig(s.Config)
- cfg.cluster.Collections.WebDAVCache.MaxBlockEntries = 42
- h := handler{Config: cfg}
- c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Not(check.Equals), cfg.cluster.Collections.WebDAVCache.MaxBlockEntries)
- u := mustParseURL("http://keep-web.example/c=" + arvadostest.FooCollection + "/t=" + arvadostest.ActiveToken + "/foo")
- req := &http.Request{
- Method: "GET",
- Host: u.Host,
- URL: u,
- RequestURI: u.RequestURI(),
- }
- resp := httptest.NewRecorder()
- h.ServeHTTP(resp, req)
- c.Check(resp.Code, check.Equals, http.StatusOK)
- c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Equals, cfg.cluster.Collections.WebDAVCache.MaxBlockEntries)
-}
-
func (s *UnitSuite) TestCORSPreflight(c *check.C) {
h := handler{Config: newConfig(s.Config)}
u := mustParseURL("http://keep-web.example/c=" + arvadostest.FooCollection + "/foo")
c.Check(resp.Body.String(), check.Matches, `{"health":"OK"}\n`)
}
+func (s *IntegrationSuite) TestFileContentType(c *check.C) {
+ s.testServer.Config.cluster.Services.WebDAVDownload.ExternalURL.Host = "download.example.com"
+
+ client := s.testServer.Config.Client
+ client.AuthToken = arvadostest.ActiveToken
+ arv, err := arvadosclient.New(&client)
+ c.Assert(err, check.Equals, nil)
+ kc, err := keepclient.MakeKeepClient(arv)
+ c.Assert(err, check.Equals, nil)
+
+ fs, err := (&arvados.Collection{}).FileSystem(&client, kc)
+ c.Assert(err, check.IsNil)
+
+ trials := []struct {
+ filename string
+ content string
+ contentType string
+ }{
+ {"picture.txt", "BMX bikes are small this year\n", "text/plain; charset=utf-8"},
+ {"picture.bmp", "BMX bikes are small this year\n", "image/x-ms-bmp"},
+ {"picture.jpg", "BMX bikes are small this year\n", "image/jpeg"},
+ {"picture1", "BMX bikes are small this year\n", "image/bmp"}, // content sniff; "BM" is the magic signature for .bmp
+ {"picture2", "Cars are small this year\n", "text/plain; charset=utf-8"}, // content sniff
+ }
+ for _, trial := range trials {
+ f, err := fs.OpenFile(trial.filename, os.O_CREATE|os.O_WRONLY, 0777)
+ c.Assert(err, check.IsNil)
+ _, err = f.Write([]byte(trial.content))
+ c.Assert(err, check.IsNil)
+ c.Assert(f.Close(), check.IsNil)
+ }
+ mtxt, err := fs.MarshalManifest(".")
+ c.Assert(err, check.IsNil)
+ var coll arvados.Collection
+ err = client.RequestAndDecode(&coll, "POST", "arvados/v1/collections", nil, map[string]interface{}{
+ "collection": map[string]string{
+ "manifest_text": mtxt,
+ },
+ })
+ c.Assert(err, check.IsNil)
+
+ for _, trial := range trials {
+ u, _ := url.Parse("http://download.example.com/by_id/" + coll.UUID + "/" + trial.filename)
+ req := &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ Header: http.Header{
+ "Authorization": {"Bearer " + client.AuthToken},
+ },
+ }
+ resp := httptest.NewRecorder()
+ s.testServer.Handler.ServeHTTP(resp, req)
+ c.Check(resp.Code, check.Equals, http.StatusOK)
+ c.Check(resp.Header().Get("Content-Type"), check.Equals, trial.contentType)
+ c.Check(resp.Body.String(), check.Equals, trial.content)
+ }
+}
+
+func (s *IntegrationSuite) TestKeepClientBlockCache(c *check.C) {
+ s.testServer.Config.cluster.Collections.WebDAVCache.MaxBlockEntries = 42
+ c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Not(check.Equals), 42)
+ u := mustParseURL("http://keep-web.example/c=" + arvadostest.FooCollection + "/t=" + arvadostest.ActiveToken + "/foo")
+ req := &http.Request{
+ Method: "GET",
+ Host: u.Host,
+ URL: u,
+ RequestURI: u.RequestURI(),
+ }
+ resp := httptest.NewRecorder()
+ s.testServer.Handler.ServeHTTP(resp, req)
+ c.Check(resp.Code, check.Equals, http.StatusOK)
+ c.Check(keepclient.DefaultBlockCache.MaxBlocks, check.Equals, 42)
+}
+
func copyHeader(h http.Header) http.Header {
hc := http.Header{}
for k, v := range h {
[Service]
Type=notify
ExecStart=/usr/bin/keep-web
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=1
import (
"flag"
"fmt"
+ "mime"
"os"
"git.arvados.org/arvados.git/lib/config"
log.Printf("keep-web %s started", version)
+ if ext := ".txt"; mime.TypeByExtension(ext) == "" {
+ log.Warnf("cannot look up MIME type for %q -- this probably means /etc/mime.types is missing -- clients will see incorrect content types", ext)
+ }
+
os.Setenv("ARVADOS_API_HOST", cfg.cluster.Services.Controller.ExternalURL.Host)
srv := &server{Config: cfg}
if err := srv.Start(); err != nil {
[Service]
Type=notify
ExecStart=/usr/bin/keepproxy
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=1
Environment=GOGC=10
Type=notify
ExecStart=/usr/bin/keepstore
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=1
'apache-libcloud==2.5.0',
'subprocess32>=3.5.1',
],
- zip_safe=False
- )
+ zip_safe=False,
+)
[Service]
Type=notify
ExecStart=/usr/bin/arvados-ws
+# Set a reasonable default for the open file limit
+LimitNOFILE=65536
Restart=always
RestartSec=1
ENV GEM_PATH /var/lib/gems
ENV PATH $PATH:/var/lib/gems/bin
-ENV GOVERSION 1.12.7
+ENV GOVERSION 1.13.6
# Install golang binary
RUN curl -f http://storage.googleapis.com/golang/go${GOVERSION}.linux-amd64.tar.gz | \
# Can't use "yarn start", need to run the dev server script
# directly so that the TERM signal from "sv restart" gets to the
# right process.
+export VERSION=$(./version-at-commit.sh)
exec node node_modules/react-scripts-ts/scripts/start.js
],
test_suite='tests',
tests_require=['pbr<1.7.0', 'mock>=1.0'],
- zip_safe=False
- )
+ zip_safe=False,
+)
projects / arvados.git / commitdiff