From: Lucas Di Pentima Date: Wed, 19 Feb 2020 21:41:39 +0000 (-0300) Subject: Merge branch '15781-multi-value-property-search' X-Git-Tag: 2.1.0~292 X-Git-Url: https://git.arvados.org/arvados.git/commitdiff_plain/64e387b2f4f0fe6c4c7bf16232706c7cf194caf0?hp=1d922bf219f74efcd5fda01ed127f7ccae0790c4 Merge branch '15781-multi-value-property-search' Refs #15781 --- diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000000..5345f045ff --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,96 @@ +Arvados Code of Conduct +======================= + +The Arvados Project is dedicated to providing a harassment-free experience for +everyone. We do not tolerate harassment of participants in any form. + +This code of conduct applies to all Arvados Project spaces both online and off: +Gitter chat, Redmine issues, wiki, mailing lists, forums, video chats, and any other +Arvados spaces. Anyone who violates this code of conduct may be sanctioned or +expelled from these spaces at the discretion of the Arvados Team. + +Some Arvados Project spaces may have additional rules in place, which will be +made clearly available to participants. Participants are responsible for +knowing and abiding by these rules. + +Harassment includes, but is not limited to: + + - Offensive comments related to gender, gender identity and expression, sexual +orientation, disability, mental illness, neuro(a)typicality, physical +appearance, body size, age, race, or religion. + - Unwelcome comments regarding a person’s lifestyle choices and practices, +including those related to food, health, parenting, drugs, and employment. + - Deliberate misgendering or use of [dead](https://www.quora.com/What-is-deadnaming/answer/Nancy-C-Walker) +or rejected names. + - Gratuitous or off-topic sexual images or behaviour in spaces where they’re not +appropriate. + - Physical contact and simulated physical contact (eg, textual descriptions like +“\*hug\*” or “\*backrub\*”) without consent or after a request to stop. + - Threats of violence. + - Incitement of violence towards any individual, including encouraging a person +to commit suicide or to engage in self-harm. + - Deliberate intimidation. + - Stalking or following. + - Harassing photography or recording, including logging online activity for +harassment purposes. + - Sustained disruption of discussion. + - Unwelcome sexual attention. + - Pattern of inappropriate social contact, such as requesting/assuming +inappropriate levels of intimacy with others + - Continued one-on-one communication after requests to cease. + - Deliberate “outing” of any aspect of a person’s identity without their consent +except as necessary to protect vulnerable people from intentional abuse. + - Publication of non-harassing private communication. + +The Arvados Project prioritizes marginalized people’s safety over privileged +people’s comfort. The Arvados Leadership Team will not act on complaints regarding: + + - ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’ + - Reasonable communication of boundaries, such as “leave me alone,” “go away,” or +“I’m not discussing this with you.” + - Communicating in a [tone](http://geekfeminism.wikia.com/wiki/Tone_argument) +you don’t find congenial + +Reporting +--------- + +If you are being harassed by a member of the Arvados Project, notice that someone +else is being harassed, or have any other concerns, please contact the Arvados +Project Team at contact@arvados.org. If person who is harassing +you is on the team, they will recuse themselves from handling your incident. We +will respond as promptly as we can. + +This code of conduct applies to Arvados Project spaces, but if you are being +harassed by a member of Arvados Project outside our spaces, we still want to +know about it. We will take all good-faith reports of harassment by Arvados Project +members, especially the Arvados Team, seriously. This includes harassment +outside our spaces and harassment that took place at any point in time. The +abuse team reserves the right to exclude people from the Arvados Project based on +their past behavior, including behavior outside Arvados Project spaces and +behavior towards people who are not in the Arvados Project. + +In order to protect volunteers from abuse and burnout, we reserve the right to +reject any report we believe to have been made in bad faith. Reports intended +to silence legitimate criticism may be deleted without response. + +We will respect confidentiality requests for the purpose of protecting victims +of abuse. At our discretion, we may publicly name a person about whom we’ve +received harassment complaints, or privately warn third parties about them, if +we believe that doing so will increase the safety of Arvados Project members or +the general public. We will not name harassment victims without their +affirmative consent. + +Consequences +------------ + +Participants asked to stop any harassing behavior are expected to comply +immediately. + +If a participant engages in harassing behavior, the Arvados Team may +take any action they deem appropriate, up to and including expulsion from all +Arvados Project spaces and identification of the participant as a harasser to other +Arvados Project members or the general public. + +This anti-harassment policy is based on the [example policy from the Geek +Feminism wiki](http://geekfeminism.wikia.com/wiki/Community_anti-harassment/Policy), +created by the Geek Feminism community. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000..459d7277a5 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,75 @@ +[comment]: # (Copyright © The Arvados Authors. All rights reserved.) +[comment]: # () +[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0) + +# Contributing + +Arvados is free software, which means it is free for all to use, learn +from, and improve. We encourage contributions from the community that +improve Arvados for everyone. Some examples of contributions are bug +reports, bug fixes, new features, and scripts or documentation that help +with using, administering, or installing Arvados. We also love to +hear about Arvados success stories. + +Those interested in contributing should begin by joining the [Arvados community +channel](https://gitter.im/arvados/community) and telling us about your interest. + +Contributers should also create an account at https://dev.arvados.org +to be able to create and comment on bug tracker issues. The +Arvados public bug tracker is located at +https://dev.arvados.org/projects/arvados/issues . + +Contributers may also be interested in the [development road map](https://dev.arvados.org/issues/gantt?utf8=%E2%9C%93&set_filter=1&gantt=1&f%5B%5D=project_id&op%5Bproject_id%5D=%3D&v%5Bproject_id%5D%5B%5D=49&f%5B%5D=&zoom=1). + +# Development + +Git repositories for primary development are located at +https://git.arvados.org/ and can also be browsed at +https://dev.arvados.org/projects/arvados/repository . Every push to +the master branch is also mirrored to Github at +https://github.com/arvados/arvados . + +Visit [Hacking Arvados](https://dev.arvados.org/projects/arvados/wiki/Hacking) for +detailed information about setting up an Arvados development +environment, development process, coding standards, and notes about specific components. + +If you wish to build the Arvados documentation from a local git clone, see +[doc/README.textile](doc/README.textile) for instructions. + +# Pull requests + +The preferred method for making contributions is through Github pull requests. + +This is the general contribution process: + +1. Fork the Arvados repository using the Github "Fork" button +2. Clone your fork, make your changes, commit to your fork. +3. Every commit message must have a DCO sign-off and every file must have a SPDX license (see below). +4. Add yourself to the [AUTHORS](AUTHORS) file +5. When your fork is ready, through Github, Create a Pull Request against `arvados:master` +6. Notify the core team about your pull request through the [Arvados development +channel](https://gitter.im/arvados/development) or by other means. +7. A member of the core team will review the pull request. They may have questions or comments, or request changes. +8. When the contribution is ready, a member of the core team will +merge the pull request into the master branch, which will +automatically resolve the pull request. + +The Arvados project does not require a contributor agreement in advance, but does require each commit message include a [Developer Certificate of Origin](https://dev.arvados.org/projects/arvados/wiki/Developer_Certificate_Of_Origin). Please ensure *every git commit message* includes `Arvados-DCO-1.1-Signed-off-by`. If you have already made commits without it, fix them with `git commit --amend` or `git rebase`. + +The Developer Certificate of Origin line looks like this: + +``` +Arvados-DCO-1.1-Signed-off-by: Joe Smith +``` + +New files must also include `SPDX-License-Identifier` at the top with one of the three Arvados open source licenses. See [COPYING](COPYING) for details. + +# Continuous integration + +Continuous integration is hosted at https://ci.arvados.org/ + +Currently, external contributers cannot trigger builds. We are investigating integration with Github pull requests for the future. + +[![Build Status](https://ci.arvados.org/buildStatus/icon?job=run-tests)](https://ci.arvados.org/job/run-tests/) + +[![Go Report Card](https://goreportcard.com/badge/github.com/arvados/arvados)](https://goreportcard.com/report/github.com/arvados/arvados) diff --git a/COPYING b/COPYING index 61c31397a0..c549d8a7bb 100644 --- a/COPYING +++ b/COPYING @@ -17,3 +17,7 @@ The full license text for each license is available in this directory: AGPL-3.0: agpl-3.0.txt Apache-2.0: apache-2.0.txt CC-BY-SA-3.0: cc-by-sa-3.0.txt + +As a general rule, code in the sdk/ directory is licensed Apache-2.0, +documentation in the doc/ directory is licensed CC-BY-SA-3.0, and +everything else is licensed AGPL-3.0. \ No newline at end of file diff --git a/README.md b/README.md index 08c102557c..fced2eb5b7 100644 --- a/README.md +++ b/README.md @@ -2,22 +2,47 @@ [comment]: # () [comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0) -[Arvados](https://arvados.org) is a free software distributed computing platform -for bioinformatics, data science, and high throughput analysis of massive data -sets. Arvados supports a variety of cloud, cluster and HPC environments. +[![Join the chat at https://gitter.im/arvados/community](https://badges.gitter.im/arvados/community.svg)](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) | [Installing Arvados](https://doc.arvados.org/install/index.html) | [Installing Client SDKs](https://doc.arvados.org/sdk/index.html) | [Report a bug](https://dev.arvados.org/projects/arvados/issues/new) | [Development and Contributing](CONTRIBUTING.md) -Arvados consists of: + -* *Keep*: a petabyte-scale content-addressed distributed storage system for managing and - storing collections of files, accessible via HTTP and FUSE mount. +[Arvados](https://arvados.org) is an open source platform for +managing, processing, and sharing genomic and other large scientific +and biomedical data. With Arvados, bioinformaticians run and scale +compute-intensive workflows, developers create biomedical +applications, and IT administrators manage large compute and storage +resources. -* *Crunch*: a Docker-based cluster and HPC workflow engine designed providing - strong versioning, reproducibilty, and provenance of computations. +The key components of Arvados are: -* Related services and components including a web workbench for managing files - and compute jobs, REST APIs, SDKs, and other tools. +* *Keep*: Keep is the Arvados storage system for managing and storing large +collections of files. Keep combines content addressing and a +distributed storage architecture resulting in both high reliability +and high throughput. Every file stored in Keep can be accurately +verified every time it is retrieved. Keep supports the creation of +collections as a flexible way to define data sets without having to +re-organize or needlessly copy data. Keep works on a wide range of +underlying filesystems and object stores. -## Quick start +* *Crunch*: Crunch is the orchestration system for running [Common Workflow Language](https://www.commonwl.org) workflows. It is +designed to maintain data provenance and workflow +reproducibility. Crunch automatically tracks data inputs and outputs +through Keep and executes workflow processes in Docker containers. In +a cloud environment, Crunch optimizes costs by scaling compute on demand. + +* *Workbench*: The Workbench web application allows users to interactively access +Arvados functionality. It is especially helpful for querying and +browsing data, visualizing provenance, and tracking the progress of +workflows. + +* *Command Line tools*: The command line interface (CLI) provides convenient access to Arvados +functionality in the Arvados platform from the command line. + +* *API and SDKs*: Arvados is designed to be integrated with existing infrastructure. All +the services in Arvados are accessed through a RESTful API. SDKs are +available for Python, Go, R, Perl, Ruby, and Java. + +# Quick start To try out Arvados on your local workstation, you can use Arvbox, which provides Arvados components pre-installed in a Docker container (requires @@ -32,48 +57,40 @@ In this mode you will only be able to connect to Arvbox from the same host. To configure Arvbox to be accessible over a network and for other options see http://doc.arvados.org/install/arvbox.html for details. -## Documentation +# Documentation -Complete documentation, including a User Guide, Installation documentation and -API documentation is available at http://doc.arvados.org/ +Complete documentation, including the [User Guide](https://doc.arvados.org/user/index.html), [Installation documentation](https://doc.arvados.org/install/index.html), [Administrator documentation](https://doc.arvados.org/admin/index.html) and +[API documentation](https://doc.arvados.org/api/index.html) is available at http://doc.arvados.org/ If you wish to build the Arvados documentation from a local git clone, see -doc/README.textile for instructions. +[doc/README.textile](doc/README.textile) for instructions. -## Community +# Community [![Join the chat at https://gitter.im/arvados/community](https://badges.gitter.im/arvados/community.svg)](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -The [arvados community channel](https://gitter.im/arvados/community) +The [Arvados community channel](https://gitter.im/arvados/community) channel at [gitter.im](https://gitter.im) is available for live discussion and support. -The -[Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados) -is a forum for general discussion, questions, and news about Arvados -development. The -[Arvados developer mailing list](http://lists.arvados.org/mailman/listinfo/arvados-dev) -is a forum for more technical discussion, intended for developers and -contributors to Arvados. +The [Arvados developement channel](https://gitter.im/arvados/development) +channel at [gitter.im](https://gitter.im) is used to coordinate development. -## Development +The [Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados) +is used to announce new versions and other news. -[![Build Status](https://ci.arvados.org/buildStatus/icon?job=run-tests)](https://ci.arvados.org/job/run-tests/) -[![Go Report Card](https://goreportcard.com/badge/github.com/arvados/arvados)](https://goreportcard.com/report/github.com/arvados/arvados) +All participants are expected to abide by the [Arvados Code of Conduct](CODE_OF_CONDUCT.md). -The Arvados public bug tracker is located at https://dev.arvados.org/projects/arvados/issues +# Reporting bugs -Continuous integration is hosted at https://ci.arvados.org/ +[Report a bug](https://dev.arvados.org/projects/arvados/issues/new) on [dev.arvados.org](https://dev.arvados.org). -Instructions for setting up a development environment and working on specific -components can be found on the -["Hacking Arvados" page of the Arvados wiki](https://dev.arvados.org/projects/arvados/wiki/Hacking). +# Development and Contributing -## Contributing +See [CONTRIBUTING](CONTRIBUTING.md) for information about Arvados development and how to contribute to the Arvados project. -When making a pull request, please ensure *every git commit message* includes a one-line [Developer Certificate of Origin](https://dev.arvados.org/projects/arvados/wiki/Developer_Certificate_Of_Origin). If you have already made commits without it, fix them with `git commit --amend` or `git rebase`. +The [development road map](https://dev.arvados.org/issues/gantt?utf8=%E2%9C%93&set_filter=1&gantt=1&f%5B%5D=project_id&op%5Bproject_id%5D=%3D&v%5Bproject_id%5D%5B%5D=49&f%5B%5D=&zoom=1) outlines some of the project priorities over the next twelve months. -## Licensing +# Licensing -Arvados is Free Software. See COPYING for information about Arvados Free -Software licenses. +Arvados is Free Software. See [COPYING](COPYING) for information about the open source licenses used in Arvados. diff --git a/apps/workbench/app/controllers/actions_controller.rb b/apps/workbench/app/controllers/actions_controller.rb index 376465ee12..885f539363 100644 --- a/apps/workbench/app/controllers/actions_controller.rb +++ b/apps/workbench/app/controllers/actions_controller.rb @@ -210,7 +210,7 @@ You can try recreating the collection to get a copy with full provenance data." # star / unstar the current project def star - links = Link.where(tail_uuid: current_user.uuid, + links = Link.where(owner_uuid: current_user.uuid, head_uuid: @object.uuid, link_class: 'star') diff --git a/apps/workbench/app/controllers/application_controller.rb b/apps/workbench/app/controllers/application_controller.rb index 63112984ca..5407700615 100644 --- a/apps/workbench/app/controllers/application_controller.rb +++ b/apps/workbench/app/controllers/application_controller.rb @@ -927,7 +927,7 @@ class ApplicationController < ActionController::Base helper_method :my_starred_projects def my_starred_projects user return if defined?(@starred_projects) && @starred_projects - links = Link.filter([['tail_uuid', '=', user.uuid], + links = Link.filter([['owner_uuid', 'in', ["#{Rails.configuration.ClusterID}-j7d0g-fffffffffffffff", user.uuid]], ['link_class', '=', 'star'], ['head_uuid', 'is_a', 'arvados#group']]).with_count("none").select(%w(head_uuid)) uuids = links.collect { |x| x.head_uuid } diff --git a/apps/workbench/app/models/container_work_unit.rb b/apps/workbench/app/models/container_work_unit.rb index faba062d46..292bc3679b 100644 --- a/apps/workbench/app/models/container_work_unit.rb +++ b/apps/workbench/app/models/container_work_unit.rb @@ -29,7 +29,7 @@ class ContainerWorkUnit < ProxyWorkUnit my_child_containers = my_children.map(&:container_uuid).compact.uniq grandchildren = {} my_child_containers.each { |c| grandchildren[c] = []} if my_child_containers.any? - reqs = ContainerRequest.select(cols).where(requesting_container_uuid: my_child_containers).with_count("none").results if my_child_containers.any? + reqs = ContainerRequest.select(cols).where(requesting_container_uuid: my_child_containers).order(["requesting_container_uuid", "uuid"]).with_count("none").results if my_child_containers.any? reqs.each {|cr| grandchildren[cr.requesting_container_uuid] << cr} if reqs my_children.each do |cr| diff --git a/apps/workbench/app/views/users/_virtual_machines.html.erb b/apps/workbench/app/views/users/_virtual_machines.html.erb index 026f016f8c..38458593fa 100644 --- a/apps/workbench/app/views/users/_virtual_machines.html.erb +++ b/apps/workbench/app/views/users/_virtual_machines.html.erb @@ -85,7 +85,7 @@ SPDX-License-Identifier: AGPL-3.0 %> <% if @my_vm_logins[vm[:uuid]] %> <% @my_vm_logins[vm[:uuid]].each do |login| %> - ssh <%= login %>@<%= vm[:hostname] %>.<%= current_uuid_prefix || 'xyzzy' %> + ssh <%= login %>@<%= vm[:hostname] %> <% end %> <% end %> @@ -105,9 +105,7 @@ SPDX-License-Identifier: AGPL-3.0 %> <% end %> -

In order to access virtual machines using SSH, <%= link_to ssh_keys_user_path(current_user) do%> add an SSH key to your account<%end%> and add a section like this to your SSH configuration file ( ~/.ssh/config):

- 
Host *.<%= current_uuid_prefix || 'xyzzy' %>
-      TCPKeepAlive yes
-      ServerAliveInterval 60
-      ProxyCommand ssh -p2222 turnout@switchyard.<%= current_api_host || 'xyzzy.arvadosapi.com' %> -x -a $SSH_PROXY_FLAGS %h
-
+ +

In order to access virtual machines using SSH, <%= link_to ssh_keys_user_path(current_user) do%>add an SSH key to your account<%end%>.

+ +<%= raw(Rails.configuration.Workbench.SSHHelpPageHTML) %> diff --git a/apps/workbench/app/views/virtual_machines/_show_help.html.erb b/apps/workbench/app/views/virtual_machines/_show_help.html.erb index d989663fe9..daf7ba50f4 100644 --- a/apps/workbench/app/views/virtual_machines/_show_help.html.erb +++ b/apps/workbench/app/views/virtual_machines/_show_help.html.erb @@ -2,29 +2,4 @@ SPDX-License-Identifier: AGPL-3.0 %> -

-Sample ~/.ssh/config section: -

- -
-Host *.arvados
-  ProxyCommand ssh -p2222 turnout@switchyard.<%= current_api_host.sub(/:.*$/,"") || 'xyzzy.arvadosapi.com' %> -x -a $SSH_PROXY_FLAGS %h
-<% if @objects.first.andand.current_user_logins.andand.first %>
-  User <%= @objects.first.current_user_logins.andand.first %>
-<% end %>
-
- -

-Sample login command: -

- -
-ssh <%= @objects.first.andand.hostname.andand.sub('.'+current_api_host.sub(/:.*$/,""),'') or 'vm-hostname' %>.arvados
-
- -

- See also: - <%= link_to raw('Arvados Docs → User Guide → SSH access'), - "#{Rails.configuration.Workbench.ArvadosDocsite}/user/getting_started/ssh-access-unix.html", - target: "_blank"%>. -

+<%= raw(Rails.configuration.Workbench.SSHHelpPageHTML) %> diff --git a/apps/workbench/fpm-info.sh b/apps/workbench/fpm-info.sh index bb7c624cae..d6604b7d51 100644 --- a/apps/workbench/fpm-info.sh +++ b/apps/workbench/fpm-info.sh @@ -4,9 +4,9 @@ case "$TARGET" in centos*) - fpm_depends+=(git bison make automake gcc gcc-c++) + fpm_depends+=(git bison make automake gcc gcc-c++ graphviz) ;; debian* | ubuntu*) - fpm_depends+=(git g++ bison zlib1g-dev make) + fpm_depends+=(git g++ bison zlib1g-dev make graphviz) ;; esac diff --git a/build/package-build-dockerfiles/centos7/Dockerfile b/build/package-build-dockerfiles/centos7/Dockerfile index 3a61e64cf4..faaf91f43b 100644 --- a/build/package-build-dockerfiles/centos7/Dockerfile +++ b/build/package-build-dockerfiles/centos7/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM centos:7 -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers # Install dependencies. RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker libattr-devel nss-devel libcurl-devel which tar unzip scl-utils centos-release-scl postgresql-devel python-devel python-setuptools fuse-devel xz-libs git python-virtualenv wget @@ -50,7 +50,7 @@ RUN rpm -ivh epel-release-latest-7.noarch.rpm RUN git clone --depth 1 git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle # The version of setuptools that comes with CentOS is way too old -RUN pip install --upgrade setuptools +RUN pip install --upgrade 'setuptools<45' ENV WORKSPACE /arvados CMD ["scl", "enable", "rh-python36", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos7"] diff --git a/build/package-build-dockerfiles/debian10/Dockerfile b/build/package-build-dockerfiles/debian10/Dockerfile index 20527dd45c..ff86262d38 100644 --- a/build/package-build-dockerfiles/debian10/Dockerfile +++ b/build/package-build-dockerfiles/debian10/Dockerfile @@ -12,7 +12,7 @@ ENV DEBIAN_FRONTEND noninteractive RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip unzip python3-venv python3-dev # Install virtualenv -RUN /usr/bin/pip install virtualenv +RUN /usr/bin/pip install 'virtualenv<20' # Install RVM ADD generated/mpapis.asc /tmp/ diff --git a/build/package-build-dockerfiles/debian9/Dockerfile b/build/package-build-dockerfiles/debian9/Dockerfile index 2aaccf6aa7..257523a72f 100644 --- a/build/package-build-dockerfiles/debian9/Dockerfile +++ b/build/package-build-dockerfiles/debian9/Dockerfile @@ -12,7 +12,7 @@ ENV DEBIAN_FRONTEND noninteractive RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip unzip python3-venv python3-dev # Install virtualenv -RUN /usr/bin/pip install virtualenv +RUN /usr/bin/pip install 'virtualenv<20' # Install RVM ADD generated/mpapis.asc /tmp/ diff --git a/build/package-build-dockerfiles/ubuntu1604/Dockerfile b/build/package-build-dockerfiles/ubuntu1604/Dockerfile index 23fcca07df..6b1304265b 100644 --- a/build/package-build-dockerfiles/ubuntu1604/Dockerfile +++ b/build/package-build-dockerfiles/ubuntu1604/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM ubuntu:xenial -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive @@ -11,7 +11,7 @@ ENV DEBIAN_FRONTEND noninteractive RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev libgnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip tzdata python3-venv python3-dev # Install virtualenv -RUN /usr/bin/pip install virtualenv +RUN /usr/bin/pip install 'virtualenv<20' # Install RVM ADD generated/mpapis.asc /tmp/ diff --git a/build/package-build-dockerfiles/ubuntu1804/Dockerfile b/build/package-build-dockerfiles/ubuntu1804/Dockerfile index 06c21ca3fa..58bff61603 100644 --- a/build/package-build-dockerfiles/ubuntu1804/Dockerfile +++ b/build/package-build-dockerfiles/ubuntu1804/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM ubuntu:bionic -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive @@ -11,7 +11,7 @@ ENV DEBIAN_FRONTEND noninteractive RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-pip libcurl4-gnutls-dev libgnutls28-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip tzdata python3-venv python3-dev # Install virtualenv -RUN /usr/bin/pip install virtualenv +RUN /usr/bin/pip install 'virtualenv<20' # Install RVM ADD generated/mpapis.asc /tmp/ diff --git a/build/package-test-dockerfiles/centos7/Dockerfile b/build/package-test-dockerfiles/centos7/Dockerfile index 6508c86be0..3d68cfc00b 100644 --- a/build/package-test-dockerfiles/centos7/Dockerfile +++ b/build/package-test-dockerfiles/centos7/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM centos:7 -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers # Install dependencies. RUN yum -q -y install scl-utils centos-release-scl which tar wget diff --git a/build/package-test-dockerfiles/debian10/Dockerfile b/build/package-test-dockerfiles/debian10/Dockerfile index 48e4984d58..32996e4a54 100644 --- a/build/package-test-dockerfiles/debian10/Dockerfile +++ b/build/package-test-dockerfiles/debian10/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM debian:buster -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive diff --git a/build/package-test-dockerfiles/debian9/Dockerfile b/build/package-test-dockerfiles/debian9/Dockerfile index 470845ae97..423a9e7c37 100644 --- a/build/package-test-dockerfiles/debian9/Dockerfile +++ b/build/package-test-dockerfiles/debian9/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM debian:stretch -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive diff --git a/build/package-test-dockerfiles/ubuntu1604/Dockerfile b/build/package-test-dockerfiles/ubuntu1604/Dockerfile index c35deebf10..e0432c20ee 100644 --- a/build/package-test-dockerfiles/ubuntu1604/Dockerfile +++ b/build/package-test-dockerfiles/ubuntu1604/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM ubuntu:xenial -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive diff --git a/build/package-test-dockerfiles/ubuntu1804/Dockerfile b/build/package-test-dockerfiles/ubuntu1804/Dockerfile index 60296e6be1..2d4189879e 100644 --- a/build/package-test-dockerfiles/ubuntu1804/Dockerfile +++ b/build/package-test-dockerfiles/ubuntu1804/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM ubuntu:bionic -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive diff --git a/build/rails-package-scripts/postinst.sh b/build/rails-package-scripts/postinst.sh index a06f8a8e90..7ea21848b2 100644 --- a/build/rails-package-scripts/postinst.sh +++ b/build/rails-package-scripts/postinst.sh @@ -199,7 +199,7 @@ configure_version() { export RAILS_ENV=production if ! $COMMAND_PREFIX bundle --version >/dev/null; then - run_and_report "Installing bundle" $COMMAND_PREFIX gem install bundle + run_and_report "Installing bundler" $COMMAND_PREFIX gem install bundler --version 1.17.3 fi run_and_report "Running bundle install" \ diff --git a/build/run-build-packages.sh b/build/run-build-packages.sh index f0f9fccdfb..e38d1fd24e 100755 --- a/build/run-build-packages.sh +++ b/build/run-build-packages.sh @@ -38,8 +38,8 @@ EOF # set to --no-cache-dir to disable pip caching CACHE_FLAG= -MAINTAINER="Ward Vandewege " -VENDOR="Veritas Genetics, Inc." +MAINTAINER="Arvados Package Maintainers " +VENDOR="The Arvados Project" # End of user configuration diff --git a/build/run-library.sh b/build/run-library.sh index 5b1dde46a9..ac5dc718be 100755 --- a/build/run-library.sh +++ b/build/run-library.sh @@ -475,9 +475,9 @@ fpm_build_virtualenv () { rm -rf dist/* # Get the latest setuptools - if ! $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools; then + if ! $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'; then echo "Error, unable to upgrade setuptools with" - echo " $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools" + echo " $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'" exit 1 fi # filter a useless warning (when building the cwltest package) from the stderr output @@ -532,9 +532,9 @@ fpm_build_virtualenv () { fi echo "pip version: `build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip --version`" - if ! build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools; then + if ! build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'; then echo "Error, unable to upgrade setuptools with" - echo " build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools" + echo " build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'" exit 1 fi echo "setuptools version: `build/usr/share/$python/dist/$PYTHON_PKG/bin/$python -c 'import setuptools; print(setuptools.__version__)'`" diff --git a/build/run-tests.sh b/build/run-tests.sh index f21861762d..891faca419 100755 --- a/build/run-tests.sh +++ b/build/run-tests.sh @@ -90,6 +90,7 @@ lib/dispatchcloud/container lib/dispatchcloud/scheduler lib/dispatchcloud/ssh_executor lib/dispatchcloud/worker +lib/mount lib/service services/api services/arv-git-httpd @@ -647,8 +648,8 @@ install_env() { . "$VENVDIR/bin/activate" # Needed for run_test_server.py which is used by certain (non-Python) tests. - pip install --no-cache-dir PyYAML future \ - || fatal "pip install PyYAML failed" + pip install --no-cache-dir PyYAML future httplib2 \ + || fatal "`pip install PyYAML future httplib2` failed" # Preinstall libcloud if using a fork; otherwise nodemanager "pip # install" won't pick it up by default. @@ -1099,6 +1100,7 @@ install_deps() { do_install sdk/cli do_install sdk/perl do_install sdk/python pip + do_install sdk/python pip3 do_install sdk/ruby do_install services/api do_install services/arv-git-httpd go diff --git a/cmd/arvados-client/Makefile b/cmd/arvados-client/Makefile new file mode 100644 index 0000000000..b043fc90e2 --- /dev/null +++ b/cmd/arvados-client/Makefile @@ -0,0 +1,11 @@ +# Copyright (C) The Arvados Authors. All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +all: + @printf "*** note *** due to an xgo limitation, this only works when the working tree is in GOPATH\n\n" + go mod download + docker build --tag=cgofuse --build-arg=http_proxy="$(http_proxy)" --build-arg=https_proxy="$(https_proxy)" https://github.com/arvados/cgofuse.git + go run github.com/karalabe/xgo --image=cgofuse --targets=linux/amd64,linux/386,darwin/amd64,darwin/386,windows/amd64,windows/386 . + install arvados-* "$(GOPATH)"/bin/ + rm --interactive=never arvados-* diff --git a/cmd/arvados-client/cmd.go b/cmd/arvados-client/cmd.go index bc6c7f0021..887bc62bb3 100644 --- a/cmd/arvados-client/cmd.go +++ b/cmd/arvados-client/cmd.go @@ -9,6 +9,7 @@ import ( "git.arvados.org/arvados.git/lib/cli" "git.arvados.org/arvados.git/lib/cmd" + "git.arvados.org/arvados.git/lib/mount" ) var ( @@ -50,6 +51,8 @@ var ( "user": cli.APICall, "virtual_machine": cli.APICall, "workflow": cli.APICall, + + "mount": mount.Command, }) ) diff --git a/cmd/arvados-server/arvados-controller.service b/cmd/arvados-server/arvados-controller.service index e85707473f..1a43d7cd3a 100644 --- a/cmd/arvados-server/arvados-controller.service +++ b/cmd/arvados-server/arvados-controller.service @@ -18,6 +18,8 @@ StartLimitIntervalSec=0 Type=notify EnvironmentFile=-/etc/arvados/environment ExecStart=/usr/bin/arvados-controller +# Set a reasonable default for the open file limit +LimitNOFILE=65536 Restart=always RestartSec=1 diff --git a/cmd/arvados-server/arvados-dispatch-cloud.service b/cmd/arvados-server/arvados-dispatch-cloud.service index aa5cc3b4a5..d3f476b7df 100644 --- a/cmd/arvados-server/arvados-dispatch-cloud.service +++ b/cmd/arvados-server/arvados-dispatch-cloud.service @@ -18,6 +18,8 @@ StartLimitIntervalSec=0 Type=notify EnvironmentFile=-/etc/arvados/environment ExecStart=/usr/bin/arvados-dispatch-cloud +# Set a reasonable default for the open file limit +LimitNOFILE=65536 Restart=always RestartSec=1 diff --git a/doc/_config.yml b/doc/_config.yml index 0f7684f837..01a19c16b9 100644 --- a/doc/_config.yml +++ b/doc/_config.yml @@ -165,20 +165,20 @@ navbar: - admin/metrics.html.textile.liquid - admin/health-checks.html.textile.liquid - admin/management-token.html.textile.liquid - - Cloud: - - admin/storage-classes.html.textile.liquid - - admin/spot-instances.html.textile.liquid - - admin/cloudtest.html.textile.liquid - Data Management: - admin/collection-versioning.html.textile.liquid - admin/collection-managed-properties.html.textile.liquid - admin/keep-balance.html.textile.liquid - admin/controlling-container-reuse.html.textile.liquid - admin/logs-table-management.html.textile.liquid + - admin/workbench2-vocabulary.html.textile.liquid + - admin/storage-classes.html.textile.liquid + - Cloud: + - admin/spot-instances.html.textile.liquid + - admin/cloudtest.html.textile.liquid - Other: - install/migrate-docker19.html.textile.liquid - admin/upgrade-crunch2.html.textile.liquid - - admin/workbench2-vocabulary.html.textile.liquid installguide: - Overview: - install/index.html.textile.liquid diff --git a/doc/_includes/_navbar_top.liquid b/doc/_includes/_navbar_top.liquid index c69a2efbec..38c8e0a1db 100644 --- a/doc/_includes/_navbar_top.liquid +++ b/doc/_includes/_navbar_top.liquid @@ -57,10 +57,10 @@ SPDX-License-Identifier: CC-BY-SA-3.0