From: Lucas Di Pentima Date: Thu, 26 Mar 2020 16:59:19 +0000 (-0300) Subject: Merge branch '16265-security-updates' into dependabot/bundler/apps/workbench/loofah... X-Git-Tag: 2.1.0~262^2~1^2 X-Git-Url: https://git.arvados.org/arvados.git/commitdiff_plain/2f66d4cc05e9442a9bb69969744d0750a02a1ed4?hp=720ac44caf998a136b64b7968253dd68972a69f4 Merge branch '16265-security-updates' into dependabot/bundler/apps/workbench/loofah-2.3.1 --- diff --git a/AUTHORS b/AUTHORS index 9a861a6315..436a504c36 100644 --- a/AUTHORS +++ b/AUTHORS @@ -18,3 +18,4 @@ President and Fellows of Harvard College <*@harvard.edu> Thomas Mooney Chen Chen Veritas Genetics, Inc. <*@veritasgenetics.com> +Curii Corporation, Inc. <*@curii.com> diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000000..5345f045ff --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,96 @@ +Arvados Code of Conduct +======================= + +The Arvados Project is dedicated to providing a harassment-free experience for +everyone. We do not tolerate harassment of participants in any form. + +This code of conduct applies to all Arvados Project spaces both online and off: +Gitter chat, Redmine issues, wiki, mailing lists, forums, video chats, and any other +Arvados spaces. Anyone who violates this code of conduct may be sanctioned or +expelled from these spaces at the discretion of the Arvados Team. + +Some Arvados Project spaces may have additional rules in place, which will be +made clearly available to participants. Participants are responsible for +knowing and abiding by these rules. + +Harassment includes, but is not limited to: + + - Offensive comments related to gender, gender identity and expression, sexual +orientation, disability, mental illness, neuro(a)typicality, physical +appearance, body size, age, race, or religion. + - Unwelcome comments regarding a person’s lifestyle choices and practices, +including those related to food, health, parenting, drugs, and employment. + - Deliberate misgendering or use of [dead](https://www.quora.com/What-is-deadnaming/answer/Nancy-C-Walker) +or rejected names. + - Gratuitous or off-topic sexual images or behaviour in spaces where they’re not +appropriate. + - Physical contact and simulated physical contact (eg, textual descriptions like +“\*hug\*” or “\*backrub\*”) without consent or after a request to stop. + - Threats of violence. + - Incitement of violence towards any individual, including encouraging a person +to commit suicide or to engage in self-harm. + - Deliberate intimidation. + - Stalking or following. + - Harassing photography or recording, including logging online activity for +harassment purposes. + - Sustained disruption of discussion. + - Unwelcome sexual attention. + - Pattern of inappropriate social contact, such as requesting/assuming +inappropriate levels of intimacy with others + - Continued one-on-one communication after requests to cease. + - Deliberate “outing” of any aspect of a person’s identity without their consent +except as necessary to protect vulnerable people from intentional abuse. + - Publication of non-harassing private communication. + +The Arvados Project prioritizes marginalized people’s safety over privileged +people’s comfort. The Arvados Leadership Team will not act on complaints regarding: + + - ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’ + - Reasonable communication of boundaries, such as “leave me alone,” “go away,” or +“I’m not discussing this with you.” + - Communicating in a [tone](http://geekfeminism.wikia.com/wiki/Tone_argument) +you don’t find congenial + +Reporting +--------- + +If you are being harassed by a member of the Arvados Project, notice that someone +else is being harassed, or have any other concerns, please contact the Arvados +Project Team at contact@arvados.org. If person who is harassing +you is on the team, they will recuse themselves from handling your incident. We +will respond as promptly as we can. + +This code of conduct applies to Arvados Project spaces, but if you are being +harassed by a member of Arvados Project outside our spaces, we still want to +know about it. We will take all good-faith reports of harassment by Arvados Project +members, especially the Arvados Team, seriously. This includes harassment +outside our spaces and harassment that took place at any point in time. The +abuse team reserves the right to exclude people from the Arvados Project based on +their past behavior, including behavior outside Arvados Project spaces and +behavior towards people who are not in the Arvados Project. + +In order to protect volunteers from abuse and burnout, we reserve the right to +reject any report we believe to have been made in bad faith. Reports intended +to silence legitimate criticism may be deleted without response. + +We will respect confidentiality requests for the purpose of protecting victims +of abuse. At our discretion, we may publicly name a person about whom we’ve +received harassment complaints, or privately warn third parties about them, if +we believe that doing so will increase the safety of Arvados Project members or +the general public. We will not name harassment victims without their +affirmative consent. + +Consequences +------------ + +Participants asked to stop any harassing behavior are expected to comply +immediately. + +If a participant engages in harassing behavior, the Arvados Team may +take any action they deem appropriate, up to and including expulsion from all +Arvados Project spaces and identification of the participant as a harasser to other +Arvados Project members or the general public. + +This anti-harassment policy is based on the [example policy from the Geek +Feminism wiki](http://geekfeminism.wikia.com/wiki/Community_anti-harassment/Policy), +created by the Geek Feminism community. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000..459d7277a5 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,75 @@ +[comment]: # (Copyright © The Arvados Authors. All rights reserved.) +[comment]: # () +[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0) + +# Contributing + +Arvados is free software, which means it is free for all to use, learn +from, and improve. We encourage contributions from the community that +improve Arvados for everyone. Some examples of contributions are bug +reports, bug fixes, new features, and scripts or documentation that help +with using, administering, or installing Arvados. We also love to +hear about Arvados success stories. + +Those interested in contributing should begin by joining the [Arvados community +channel](https://gitter.im/arvados/community) and telling us about your interest. + +Contributers should also create an account at https://dev.arvados.org +to be able to create and comment on bug tracker issues. The +Arvados public bug tracker is located at +https://dev.arvados.org/projects/arvados/issues . + +Contributers may also be interested in the [development road map](https://dev.arvados.org/issues/gantt?utf8=%E2%9C%93&set_filter=1&gantt=1&f%5B%5D=project_id&op%5Bproject_id%5D=%3D&v%5Bproject_id%5D%5B%5D=49&f%5B%5D=&zoom=1). + +# Development + +Git repositories for primary development are located at +https://git.arvados.org/ and can also be browsed at +https://dev.arvados.org/projects/arvados/repository . Every push to +the master branch is also mirrored to Github at +https://github.com/arvados/arvados . + +Visit [Hacking Arvados](https://dev.arvados.org/projects/arvados/wiki/Hacking) for +detailed information about setting up an Arvados development +environment, development process, coding standards, and notes about specific components. + +If you wish to build the Arvados documentation from a local git clone, see +[doc/README.textile](doc/README.textile) for instructions. + +# Pull requests + +The preferred method for making contributions is through Github pull requests. + +This is the general contribution process: + +1. Fork the Arvados repository using the Github "Fork" button +2. Clone your fork, make your changes, commit to your fork. +3. Every commit message must have a DCO sign-off and every file must have a SPDX license (see below). +4. Add yourself to the [AUTHORS](AUTHORS) file +5. When your fork is ready, through Github, Create a Pull Request against `arvados:master` +6. Notify the core team about your pull request through the [Arvados development +channel](https://gitter.im/arvados/development) or by other means. +7. A member of the core team will review the pull request. They may have questions or comments, or request changes. +8. When the contribution is ready, a member of the core team will +merge the pull request into the master branch, which will +automatically resolve the pull request. + +The Arvados project does not require a contributor agreement in advance, but does require each commit message include a [Developer Certificate of Origin](https://dev.arvados.org/projects/arvados/wiki/Developer_Certificate_Of_Origin). Please ensure *every git commit message* includes `Arvados-DCO-1.1-Signed-off-by`. If you have already made commits without it, fix them with `git commit --amend` or `git rebase`. + +The Developer Certificate of Origin line looks like this: + +``` +Arvados-DCO-1.1-Signed-off-by: Joe Smith +``` + +New files must also include `SPDX-License-Identifier` at the top with one of the three Arvados open source licenses. See [COPYING](COPYING) for details. + +# Continuous integration + +Continuous integration is hosted at https://ci.arvados.org/ + +Currently, external contributers cannot trigger builds. We are investigating integration with Github pull requests for the future. + +[![Build Status](https://ci.arvados.org/buildStatus/icon?job=run-tests)](https://ci.arvados.org/job/run-tests/) + +[![Go Report Card](https://goreportcard.com/badge/github.com/arvados/arvados)](https://goreportcard.com/report/github.com/arvados/arvados) diff --git a/COPYING b/COPYING index 61c31397a0..c549d8a7bb 100644 --- a/COPYING +++ b/COPYING @@ -17,3 +17,7 @@ The full license text for each license is available in this directory: AGPL-3.0: agpl-3.0.txt Apache-2.0: apache-2.0.txt CC-BY-SA-3.0: cc-by-sa-3.0.txt + +As a general rule, code in the sdk/ directory is licensed Apache-2.0, +documentation in the doc/ directory is licensed CC-BY-SA-3.0, and +everything else is licensed AGPL-3.0. \ No newline at end of file diff --git a/README.md b/README.md index 5843bb84da..fced2eb5b7 100644 --- a/README.md +++ b/README.md @@ -2,24 +2,47 @@ [comment]: # () [comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0) -[Arvados](https://arvados.org) is a free software distributed computing platform -for bioinformatics, data science, and high throughput analysis of massive data -sets. Arvados supports a variety of cloud, cluster and HPC environments. +[![Join the chat at https://gitter.im/arvados/community](https://badges.gitter.im/arvados/community.svg)](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) | [Installing Arvados](https://doc.arvados.org/install/index.html) | [Installing Client SDKs](https://doc.arvados.org/sdk/index.html) | [Report a bug](https://dev.arvados.org/projects/arvados/issues/new) | [Development and Contributing](CONTRIBUTING.md) -Arvados consists of: + -* *Keep*: a petabyte-scale content-addressed distributed storage system for managing and - storing collections of files, accessible via HTTP and FUSE mount. +[Arvados](https://arvados.org) is an open source platform for +managing, processing, and sharing genomic and other large scientific +and biomedical data. With Arvados, bioinformaticians run and scale +compute-intensive workflows, developers create biomedical +applications, and IT administrators manage large compute and storage +resources. -* *Crunch*: a Docker-based cluster and HPC workflow engine designed providing - strong versioning, reproducibilty, and provenance of computations. +The key components of Arvados are: -* Related services and components including a web workbench for managing files - and compute jobs, REST APIs, SDKs, and other tools. +* *Keep*: Keep is the Arvados storage system for managing and storing large +collections of files. Keep combines content addressing and a +distributed storage architecture resulting in both high reliability +and high throughput. Every file stored in Keep can be accurately +verified every time it is retrieved. Keep supports the creation of +collections as a flexible way to define data sets without having to +re-organize or needlessly copy data. Keep works on a wide range of +underlying filesystems and object stores. -## Quick start +* *Crunch*: Crunch is the orchestration system for running [Common Workflow Language](https://www.commonwl.org) workflows. It is +designed to maintain data provenance and workflow +reproducibility. Crunch automatically tracks data inputs and outputs +through Keep and executes workflow processes in Docker containers. In +a cloud environment, Crunch optimizes costs by scaling compute on demand. -Veritas Genetics maintains a public installation of Arvados for evaluation and trial use, the [Arvados Playground](https://playground.arvados.org). A Google account is required to log in. +* *Workbench*: The Workbench web application allows users to interactively access +Arvados functionality. It is especially helpful for querying and +browsing data, visualizing provenance, and tracking the progress of +workflows. + +* *Command Line tools*: The command line interface (CLI) provides convenient access to Arvados +functionality in the Arvados platform from the command line. + +* *API and SDKs*: Arvados is designed to be integrated with existing infrastructure. All +the services in Arvados are accessed through a RESTful API. SDKs are +available for Python, Go, R, Perl, Ruby, and Java. + +# Quick start To try out Arvados on your local workstation, you can use Arvbox, which provides Arvados components pre-installed in a Docker container (requires @@ -34,48 +57,40 @@ In this mode you will only be able to connect to Arvbox from the same host. To configure Arvbox to be accessible over a network and for other options see http://doc.arvados.org/install/arvbox.html for details. -## Documentation +# Documentation -Complete documentation, including a User Guide, Installation documentation and -API documentation is available at http://doc.arvados.org/ +Complete documentation, including the [User Guide](https://doc.arvados.org/user/index.html), [Installation documentation](https://doc.arvados.org/install/index.html), [Administrator documentation](https://doc.arvados.org/admin/index.html) and +[API documentation](https://doc.arvados.org/api/index.html) is available at http://doc.arvados.org/ If you wish to build the Arvados documentation from a local git clone, see -doc/README.textile for instructions. +[doc/README.textile](doc/README.textile) for instructions. -## Community +# Community -[![Join the chat at https://gitter.im/curoverse/arvados](https://badges.gitter.im/curoverse/arvados.svg)](https://gitter.im/curoverse/arvados?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) +[![Join the chat at https://gitter.im/arvados/community](https://badges.gitter.im/arvados/community.svg)](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -The [curoverse/arvados channel](https://gitter.im/curoverse/arvados) +The [Arvados community channel](https://gitter.im/arvados/community) channel at [gitter.im](https://gitter.im) is available for live discussion and support. -The -[Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados) -is a forum for general discussion, questions, and news about Arvados -development. The -[Arvados developer mailing list](http://lists.arvados.org/mailman/listinfo/arvados-dev) -is a forum for more technical discussion, intended for developers and -contributors to Arvados. +The [Arvados developement channel](https://gitter.im/arvados/development) +channel at [gitter.im](https://gitter.im) is used to coordinate development. -## Development +The [Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados) +is used to announce new versions and other news. -[![Build Status](https://ci.curoverse.com/buildStatus/icon?job=run-tests)](https://ci.curoverse.com/job/run-tests/) -[![Go Report Card](https://goreportcard.com/badge/github.com/curoverse/arvados)](https://goreportcard.com/report/github.com/curoverse/arvados) +All participants are expected to abide by the [Arvados Code of Conduct](CODE_OF_CONDUCT.md). -The Arvados public bug tracker is located at https://dev.arvados.org/projects/arvados/issues +# Reporting bugs -Continuous integration is hosted at https://ci.curoverse.com/ +[Report a bug](https://dev.arvados.org/projects/arvados/issues/new) on [dev.arvados.org](https://dev.arvados.org). -Instructions for setting up a development environment and working on specific -components can be found on the -["Hacking Arvados" page of the Arvados wiki](https://dev.arvados.org/projects/arvados/wiki/Hacking). +# Development and Contributing -## Contributing +See [CONTRIBUTING](CONTRIBUTING.md) for information about Arvados development and how to contribute to the Arvados project. -When making a pull request, please ensure *every git commit message* includes a one-line [Developer Certificate of Origin](https://dev.arvados.org/projects/arvados/wiki/Developer_Certificate_Of_Origin). If you have already made commits without it, fix them with `git commit --amend` or `git rebase`. +The [development road map](https://dev.arvados.org/issues/gantt?utf8=%E2%9C%93&set_filter=1&gantt=1&f%5B%5D=project_id&op%5Bproject_id%5D=%3D&v%5Bproject_id%5D%5B%5D=49&f%5B%5D=&zoom=1) outlines some of the project priorities over the next twelve months. -## Licensing +# Licensing -Arvados is Free Software. See COPYING for information about Arvados Free -Software licenses. +Arvados is Free Software. See [COPYING](COPYING) for information about the open source licenses used in Arvados. diff --git a/apps/workbench/Gemfile b/apps/workbench/Gemfile index 40cf4a86c0..24bfba383f 100644 --- a/apps/workbench/Gemfile +++ b/apps/workbench/Gemfile @@ -5,9 +5,9 @@ source 'https://rubygems.org' gem 'rails', '~> 5.0.0' -gem 'arvados', git: 'https://github.com/curoverse/arvados.git', glob: 'sdk/ruby/arvados.gemspec' +gem 'arvados', git: 'https://github.com/arvados/arvados.git', glob: 'sdk/ruby/arvados.gemspec' -gem 'activerecord-nulldb-adapter', git: 'https://github.com/curoverse/nulldb' +gem 'activerecord-nulldb-adapter', git: 'https://github.com/arvados/nulldb' gem 'multi_json' gem 'oj' gem 'sass' @@ -74,7 +74,7 @@ gem 'less-rails' # Wiselinks hasn't been updated for many years and it's using deprecated methods # Use our own Wiselinks fork until this PR is accepted: # https://github.com/igor-alexandrov/wiselinks/pull/116 -# gem 'wiselinks', git: 'https://github.com/curoverse/wiselinks.git', branch: 'rails-5.1-compatibility' +# gem 'wiselinks', git: 'https://github.com/arvados/wiselinks.git', branch: 'rails-5.1-compatibility' gem 'sshkey' @@ -98,7 +98,7 @@ gem 'piwik_analytics' gem 'httpclient', '~> 2.5' # This fork has Rails 4 compatible routes -gem 'themes_for_rails', git: 'https://github.com/curoverse/themes_for_rails' +gem 'themes_for_rails', git: 'https://github.com/arvados/themes_for_rails' gem "deep_merge", :require => 'deep_merge/rails_compat' diff --git a/apps/workbench/Gemfile.lock b/apps/workbench/Gemfile.lock index 08ce86a9d6..2af9c8b16f 100644 --- a/apps/workbench/Gemfile.lock +++ b/apps/workbench/Gemfile.lock @@ -1,9 +1,9 @@ GIT - remote: https://github.com/curoverse/arvados.git - revision: dd9f2403f43bcb93da5908ddde57d8c0491bb4c2 + remote: https://github.com/arvados/arvados.git + revision: c210114aa8c77ba0bb8e4d487fc1507b40f9560f glob: sdk/ruby/arvados.gemspec specs: - arvados (1.4.1.20191019025325) + arvados (1.5.0.pre20200114202620) activesupport (>= 3) andand (~> 1.3, >= 1.3.3) arvados-google-api-client (>= 0.7, < 0.8.9) @@ -13,14 +13,14 @@ GIT jwt (>= 0.1.5, < 2) GIT - remote: https://github.com/curoverse/nulldb + remote: https://github.com/arvados/nulldb revision: d8e0073b665acdd2537c5eb15178a60f02f4b413 specs: activerecord-nulldb-adapter (0.3.9) activerecord (>= 2.0.0) GIT - remote: https://github.com/curoverse/themes_for_rails + remote: https://github.com/arvados/themes_for_rails revision: ddf6e592b3b6493ea0c2de7b5d3faa120ed35be0 specs: themes_for_rails (0.5.1) @@ -150,7 +150,7 @@ GEM rails-dom-testing (>= 1, < 3) railties (>= 4.2.0) thor (>= 0.14, < 2.0) - json (2.2.0) + json (2.3.0) jwt (1.5.6) launchy (2.4.3) addressable (~> 2.3) @@ -172,7 +172,7 @@ GEM nokogiri (>= 1.5.9) mail (2.7.1) mini_mime (>= 0.1.1) - memoist (0.16.0) + memoist (0.16.2) metaclass (0.0.4) method_source (0.9.2) mime-types (3.2.2) @@ -195,7 +195,7 @@ GEM net-ssh-gateway (2.0.0) net-ssh (>= 4.0.0) nio4r (2.3.1) - nokogiri (1.10.5) + nokogiri (1.10.8) mini_portile2 (~> 2.4.0) npm-rails (0.2.1) rails (>= 3.2) @@ -213,7 +213,7 @@ GEM cliver (~> 0.3.1) multi_json (~> 1.0) websocket-driver (>= 0.2.0) - public_suffix (4.0.1) + public_suffix (4.0.3) rack (2.0.7) rack-mini-profiler (1.0.2) rack (>= 1.2.0) @@ -247,7 +247,7 @@ GEM method_source rake (>= 0.8.7) thor (>= 0.18.1, < 2.0) - rake (12.3.2) + rake (13.0.1) raphael-rails (2.1.2) rb-fsevent (0.10.3) rb-inotify (0.10.0) @@ -308,7 +308,7 @@ GEM thor (0.20.3) thread_safe (0.3.6) tilt (2.0.9) - tzinfo (1.2.5) + tzinfo (1.2.6) thread_safe (~> 0.1) uglifier (2.7.2) execjs (>= 0.3.0) @@ -375,4 +375,4 @@ DEPENDENCIES uglifier (~> 2.0) BUNDLED WITH - 1.17.3 + 1.11 diff --git a/apps/workbench/app/controllers/actions_controller.rb b/apps/workbench/app/controllers/actions_controller.rb index 376465ee12..885f539363 100644 --- a/apps/workbench/app/controllers/actions_controller.rb +++ b/apps/workbench/app/controllers/actions_controller.rb @@ -210,7 +210,7 @@ You can try recreating the collection to get a copy with full provenance data." # star / unstar the current project def star - links = Link.where(tail_uuid: current_user.uuid, + links = Link.where(owner_uuid: current_user.uuid, head_uuid: @object.uuid, link_class: 'star') diff --git a/apps/workbench/app/controllers/application_controller.rb b/apps/workbench/app/controllers/application_controller.rb index 63112984ca..8d6f897bb6 100644 --- a/apps/workbench/app/controllers/application_controller.rb +++ b/apps/workbench/app/controllers/application_controller.rb @@ -62,6 +62,7 @@ class ApplicationController < ActionController::Base # the browser can't. f.json { render opts.merge(json: {success: false, errors: @errors}) } f.html { render({action: 'error'}.merge(opts)) } + f.all { render({action: 'error', formats: 'text'}.merge(opts)) } end end @@ -927,7 +928,7 @@ class ApplicationController < ActionController::Base helper_method :my_starred_projects def my_starred_projects user return if defined?(@starred_projects) && @starred_projects - links = Link.filter([['tail_uuid', '=', user.uuid], + links = Link.filter([['owner_uuid', 'in', ["#{Rails.configuration.ClusterID}-j7d0g-fffffffffffffff", user.uuid]], ['link_class', '=', 'star'], ['head_uuid', 'is_a', 'arvados#group']]).with_count("none").select(%w(head_uuid)) uuids = links.collect { |x| x.head_uuid } diff --git a/apps/workbench/app/controllers/collections_controller.rb b/apps/workbench/app/controllers/collections_controller.rb index 10e026ae6c..9073d06c17 100644 --- a/apps/workbench/app/controllers/collections_controller.rb +++ b/apps/workbench/app/controllers/collections_controller.rb @@ -343,7 +343,7 @@ class CollectionsController < ApplicationController # Prefer the attachment-only-host when we want an attachment # (and when there is no preview link configured) tmpl = Rails.configuration.Services.WebDAVDownload.ExternalURL.to_s - elsif not Rails.configuration.Workbench.TrustAllContent + elsif not Rails.configuration.Collections.TrustAllContent check_uri = URI.parse(tmpl.sub("*", munged_id)) if opts[:query_token] and (check_uri.host.nil? or ( diff --git a/apps/workbench/app/controllers/users_controller.rb b/apps/workbench/app/controllers/users_controller.rb index febd6e3a1d..27fc12bf4c 100644 --- a/apps/workbench/app/controllers/users_controller.rb +++ b/apps/workbench/app/controllers/users_controller.rb @@ -124,7 +124,7 @@ class UsersController < ApplicationController def show_pane_list if current_user.andand.is_admin - super | %w(Admin) + %w(Admin) | super else super end diff --git a/apps/workbench/app/controllers/virtual_machines_controller.rb b/apps/workbench/app/controllers/virtual_machines_controller.rb index 1427e3cc72..c743773141 100644 --- a/apps/workbench/app/controllers/virtual_machines_controller.rb +++ b/apps/workbench/app/controllers/virtual_machines_controller.rb @@ -25,8 +25,8 @@ class VirtualMachinesController < ApplicationController end def webshell - return render_not_found if Rails.configuration.Workbench.ShellInABoxURL == URI("") - webshell_url = URI(Rails.configuration.Workbench.ShellInABoxURL) + return render_not_found if Rails.configuration.Services.WebShell.ExternalURL == URI("") + webshell_url = URI(Rails.configuration.Services.WebShell.ExternalURL) if webshell_url.host.index("*") != nil webshell_url.host = webshell_url.host.sub("*", @object.hostname) else diff --git a/apps/workbench/app/helpers/application_helper.rb b/apps/workbench/app/helpers/application_helper.rb index 0a872446d5..330d30976f 100644 --- a/apps/workbench/app/helpers/application_helper.rb +++ b/apps/workbench/app/helpers/application_helper.rb @@ -12,7 +12,11 @@ module ApplicationHelper end def current_api_host - "#{Rails.configuration.Services.Controller.ExternalURL.hostname}:#{Rails.configuration.Services.Controller.ExternalURL.port}" + if Rails.configuration.Services.Controller.ExternalURL.port == 443 + "#{Rails.configuration.Services.Controller.ExternalURL.hostname}" + else + "#{Rails.configuration.Services.Controller.ExternalURL.hostname}:#{Rails.configuration.Services.Controller.ExternalURL.port}" + end end def current_uuid_prefix diff --git a/apps/workbench/app/models/container_work_unit.rb b/apps/workbench/app/models/container_work_unit.rb index faba062d46..292bc3679b 100644 --- a/apps/workbench/app/models/container_work_unit.rb +++ b/apps/workbench/app/models/container_work_unit.rb @@ -29,7 +29,7 @@ class ContainerWorkUnit < ProxyWorkUnit my_child_containers = my_children.map(&:container_uuid).compact.uniq grandchildren = {} my_child_containers.each { |c| grandchildren[c] = []} if my_child_containers.any? - reqs = ContainerRequest.select(cols).where(requesting_container_uuid: my_child_containers).with_count("none").results if my_child_containers.any? + reqs = ContainerRequest.select(cols).where(requesting_container_uuid: my_child_containers).order(["requesting_container_uuid", "uuid"]).with_count("none").results if my_child_containers.any? reqs.each {|cr| grandchildren[cr.requesting_container_uuid] << cr} if reqs my_children.each do |cr| diff --git a/apps/workbench/app/views/application/error.text.erb b/apps/workbench/app/views/application/error.text.erb new file mode 100644 index 0000000000..1035182748 --- /dev/null +++ b/apps/workbench/app/views/application/error.text.erb @@ -0,0 +1,11 @@ +<%# Copyright (C) The Arvados Authors. All rights reserved. + +SPDX-License-Identifier: AGPL-3.0 %> + +Oh... fiddlesticks. + +Sorry, I had some trouble handling your request. + +<% if @errors.is_a? Array then @errors.each do |error| %> +<%= error %> +<% end end %> diff --git a/apps/workbench/app/views/users/_setup_popup.html.erb b/apps/workbench/app/views/users/_setup_popup.html.erb index d6f25136c4..4c3a95e87e 100644 --- a/apps/workbench/app/views/users/_setup_popup.html.erb +++ b/apps/workbench/app/views/users/_setup_popup.html.erb @@ -11,7 +11,7 @@ SPDX-License-Identifier: AGPL-3.0 %>
-

Setup Shell Account

+

Setup Account


@@ -48,7 +48,7 @@ SPDX-License-Identifier: AGPL-3.0 %> <% end %>
- +
- +
diff --git a/apps/workbench/app/views/users/_show_admin.html.erb b/apps/workbench/app/views/users/_show_admin.html.erb index ddff79be01..1da22d438f 100644 --- a/apps/workbench/app/views/users/_show_admin.html.erb +++ b/apps/workbench/app/views/users/_show_admin.html.erb @@ -4,35 +4,36 @@ SPDX-License-Identifier: AGPL-3.0 %>
+

- As an admin, you can log in as this user. When you’ve - finished, you will need to log out and log in again with your - own account. + This page enables you to manage users.

-
- <%= button_to "Log in as #{@object.full_name}", sudo_user_url(id: @object.uuid), class: 'btn btn-primary' %> -
-

- As an admin, you can setup a shell account for this user. - The login name is automatically generated from the user's e-mail address. + This button sets up a user. After setup, they will be able use + Arvados. This dialog box also allows you to optionally set up a + shell account for this user. The login name is automatically + generated from the user's e-mail address.

-
- <%= link_to "Setup shell account #{'for ' if @object.full_name.present?} #{@object.full_name}", setup_popup_user_url(id: @object.uuid), {class: 'btn btn-primary', :remote => true, 'data-toggle' => "modal", 'data-target' => '#user-setup-modal-window'} %> -
+ <%= link_to "Setup account #{'for ' if @object.full_name.present?} #{@object.full_name}", setup_popup_user_url(id: @object.uuid), {class: 'btn btn-primary', :remote => true, 'data-toggle' => "modal", 'data-target' => '#user-setup-modal-window'} %> -

+

As an admin, you can deactivate and reset this user. This will remove all repository/VM permissions for the user. If you "setup" the user again, the user will have to sign the user - agreement again. + agreement again. You may also want to reassign data ownership. +

+ + <%= button_to "Deactivate #{@object.full_name}", unsetup_user_url(id: @object.uuid), class: 'btn btn-primary', data: {confirm: "Are you sure you want to deactivate #{@object.full_name}?"} %> + +

+ As an admin, you can log in as this user. When you’ve + finished, you will need to log out and log in again with your + own account.

-
- <%= button_to "Deactivate #{@object.full_name}", unsetup_user_url(id: @object.uuid), class: 'btn btn-primary', data: {confirm: "Are you sure you want to deactivate #{@object.full_name}?"} %> -
+ <%= button_to "Log in as #{@object.full_name}", sudo_user_url(id: @object.uuid), class: 'btn btn-primary' %>
diff --git a/apps/workbench/app/views/users/_virtual_machines.html.erb b/apps/workbench/app/views/users/_virtual_machines.html.erb index 026f016f8c..e2ce5b39bc 100644 --- a/apps/workbench/app/views/users/_virtual_machines.html.erb +++ b/apps/workbench/app/views/users/_virtual_machines.html.erb @@ -85,7 +85,7 @@ SPDX-License-Identifier: AGPL-3.0 %> <% if @my_vm_logins[vm[:uuid]] %> <% @my_vm_logins[vm[:uuid]].each do |login| %> - ssh <%= login %>@<%= vm[:hostname] %>.<%= current_uuid_prefix || 'xyzzy' %> + ssh <%= login %>@<%= vm[:hostname] %><%=Rails.configuration.Workbench.SSHHelpHostSuffix%> <% end %> <% end %> @@ -105,9 +105,7 @@ SPDX-License-Identifier: AGPL-3.0 %> <% end %>
-

In order to access virtual machines using SSH, <%= link_to ssh_keys_user_path(current_user) do%> add an SSH key to your account<%end%> and add a section like this to your SSH configuration file ( ~/.ssh/config):

- 
Host *.<%= current_uuid_prefix || 'xyzzy' %>
-      TCPKeepAlive yes
-      ServerAliveInterval 60
-      ProxyCommand ssh -p2222 turnout@switchyard.<%= current_api_host || 'xyzzy.arvadosapi.com' %> -x -a $SSH_PROXY_FLAGS %h
-
+ +

In order to access virtual machines using SSH, <%= link_to ssh_keys_user_path(current_user) do%>add an SSH key to your account<%end%>.

+ +<%= raw(Rails.configuration.Workbench.SSHHelpPageHTML) %> diff --git a/apps/workbench/app/views/virtual_machines/_show_help.html.erb b/apps/workbench/app/views/virtual_machines/_show_help.html.erb index 204e71a914..daf7ba50f4 100644 --- a/apps/workbench/app/views/virtual_machines/_show_help.html.erb +++ b/apps/workbench/app/views/virtual_machines/_show_help.html.erb @@ -2,29 +2,4 @@ SPDX-License-Identifier: AGPL-3.0 %> -

-Sample ~/.ssh/config section: -

- -
-Host *.arvados
-  ProxyCommand ssh -p2222 turnout@switchyard.<%= current_api_host || 'xyzzy.arvadosapi.com' %> -x -a $SSH_PROXY_FLAGS %h
-<% if @objects.first.andand.current_user_logins.andand.first %>
-  User <%= @objects.first.current_user_logins.andand.first %>
-<% end %>
-
- -

-Sample login command: -

- -
-ssh <%= @objects.first.andand.hostname.andand.sub('.'+current_api_host,'') or 'vm-hostname' %>.arvados
-
- -

- See also: - <%= link_to raw('Arvados Docs → User Guide → SSH access'), - "#{Rails.configuration.Workbench.ArvadosDocsite}/user/getting_started/ssh-access-unix.html", - target: "_blank"%>. -

+<%= raw(Rails.configuration.Workbench.SSHHelpPageHTML) %> diff --git a/apps/workbench/config/application.rb b/apps/workbench/config/application.rb index 514d57196d..e88229b851 100644 --- a/apps/workbench/config/application.rb +++ b/apps/workbench/config/application.rb @@ -19,6 +19,10 @@ require "rails/test_unit/railtie" Bundler.require(:default, Rails.env) +if ENV["ARVADOS_RAILS_LOG_TO_STDOUT"] + Rails.logger = ActiveSupport::TaggedLogging.new(Logger.new(STDOUT)) +end + module ArvadosWorkbench class Application < Rails::Application diff --git a/apps/workbench/fpm-info.sh b/apps/workbench/fpm-info.sh index 22ec1ba14c..d6604b7d51 100644 --- a/apps/workbench/fpm-info.sh +++ b/apps/workbench/fpm-info.sh @@ -4,9 +4,9 @@ case "$TARGET" in centos*) - fpm_depends+=(git) + fpm_depends+=(git bison make automake gcc gcc-c++ graphviz) ;; debian* | ubuntu*) - fpm_depends+=(git g++) + fpm_depends+=(git g++ bison zlib1g-dev make graphviz) ;; esac diff --git a/apps/workbench/test/controllers/collections_controller_test.rb b/apps/workbench/test/controllers/collections_controller_test.rb index 4fce54a8ab..a95b649942 100644 --- a/apps/workbench/test/controllers/collections_controller_test.rb +++ b/apps/workbench/test/controllers/collections_controller_test.rb @@ -532,7 +532,7 @@ class CollectionsControllerTest < ActionController::TestCase end test "Redirect to keep_web_url via #{id_type} when trust_all_content enabled" do - Rails.configuration.Workbench.TrustAllContent = true + Rails.configuration.Collections.TrustAllContent = true setup_for_keep_web('https://collections.example', 'https://download.example') tok = api_token('active') @@ -583,7 +583,7 @@ class CollectionsControllerTest < ActionController::TestCase [false, true].each do |trust_all_content| test "Redirect preview to keep_web_download_url when preview is disabled and trust_all_content is #{trust_all_content}" do - Rails.configuration.Workbench.TrustAllContent = trust_all_content + Rails.configuration.Collections.TrustAllContent = trust_all_content setup_for_keep_web "", 'https://download.example/' tok = api_token('active') id = api_fixture('collections')['w_a_z_file']['uuid'] diff --git a/apps/workbench/test/controllers/projects_controller_test.rb b/apps/workbench/test/controllers/projects_controller_test.rb index dd828952be..27d7dedc91 100644 --- a/apps/workbench/test/controllers/projects_controller_test.rb +++ b/apps/workbench/test/controllers/projects_controller_test.rb @@ -151,7 +151,7 @@ class ProjectsControllerTest < ActionController::TestCase end ['', ' asc', ' desc'].each do |direction| - test "projects#show tab partial orders correctly by #{direction}" do + test "projects#show tab partial orders correctly by created_at#{direction}" do _test_tab_content_order direction end end diff --git a/apps/workbench/test/integration/users_test.rb b/apps/workbench/test/integration/users_test.rb index bad01a1c62..57be9d370d 100644 --- a/apps/workbench/test/integration/users_test.rb +++ b/apps/workbench/test/integration/users_test.rb @@ -77,6 +77,8 @@ class UsersTest < ActionDispatch::IntegrationTest find('a', text: 'Show'). click + click_link 'Attributes' + assert page.has_text? 'modified_by_user_uuid' page.within(:xpath, '//span[@data-name="is_active"]') do assert_equal "false", text, "Expected new user's is_active to be false" @@ -84,7 +86,7 @@ class UsersTest < ActionDispatch::IntegrationTest click_link 'Advanced' click_link 'Metadata' - assert page.has_text? 'can_login' # make sure page is rendered / ready + assert page.has_text? 'can_read' # make sure page is rendered / ready assert page.has_no_text? 'VirtualMachine:' end @@ -103,9 +105,9 @@ class UsersTest < ActionDispatch::IntegrationTest # Setup user click_link 'Admin' - assert page.has_text? 'As an admin, you can setup' + assert page.has_text? 'This button sets up a user' - click_link 'Setup shell account for Active User' + click_link 'Setup account for Active User' within '.modal-content' do find 'label', text: 'Virtual Machine' @@ -113,6 +115,7 @@ class UsersTest < ActionDispatch::IntegrationTest end visit user_url + click_link 'Attributes' assert page.has_text? 'modified_by_client_uuid' click_link 'Advanced' @@ -123,7 +126,7 @@ class UsersTest < ActionDispatch::IntegrationTest # Click on Setup button again and this time also choose a VM click_link 'Admin' - click_link 'Setup shell account for Active User' + click_link 'Setup account for Active User' within '.modal-content' do select("testvm.shell", :from => 'vm_uuid') @@ -132,12 +135,15 @@ class UsersTest < ActionDispatch::IntegrationTest end visit user_url + click_link 'Attributes' find '#Attributes', text: 'modified_by_client_uuid' click_link 'Advanced' click_link 'Metadata' assert page.has_text? 'VirtualMachine: testvm.shell' assert page.has_text? '["test group one", "test-group-two"]' + vm_links = all("a", text: "VirtualMachine:") + assert_equal(2, vm_links.size) end test "unsetup active user" do @@ -155,7 +161,7 @@ class UsersTest < ActionDispatch::IntegrationTest user_url = page.current_url # Verify that is_active is set - find('a,button', text: 'Attributes').click + click_link 'Attributes' assert page.has_text? 'modified_by_user_uuid' page.within(:xpath, '//span[@data-name="is_active"]') do assert_equal "true", text, "Expected user's is_active to be true" @@ -176,6 +182,8 @@ class UsersTest < ActionDispatch::IntegrationTest # poltergeist returns true for confirm(), so we don't need to accept. end + click_link 'Attributes' + # Should now be back in the Attributes tab for the user assert page.has_text? 'modified_by_user_uuid' page.within(:xpath, '//span[@data-name="is_active"]') do @@ -188,7 +196,7 @@ class UsersTest < ActionDispatch::IntegrationTest # setup user again and verify links present click_link 'Admin' - click_link 'Setup shell account for Active User' + click_link 'Setup account for Active User' within '.modal-content' do select("testvm.shell", :from => 'vm_uuid') @@ -196,6 +204,7 @@ class UsersTest < ActionDispatch::IntegrationTest end visit user_url + click_link 'Attributes' assert page.has_text? 'modified_by_client_uuid' click_link 'Advanced' @@ -211,7 +220,7 @@ class UsersTest < ActionDispatch::IntegrationTest # Setup user click_link 'Admin' - assert page.has_text? 'As an admin, you can setup' + assert page.has_text? 'This button sets up a user' click_link 'Add new group' diff --git a/apps/workbench/test/test_helper.rb b/apps/workbench/test/test_helper.rb index 28a7fa5137..84728b8c68 100644 --- a/apps/workbench/test/test_helper.rb +++ b/apps/workbench/test/test_helper.rb @@ -362,6 +362,7 @@ module Minitest n += 1 raise if n > 2 || e.is_a?(Skip) STDERR.puts "Test failed, retrying (##{n})" + ActiveSupport::TestCase.reset_api_fixtures_now retry end rescue *PASSTHROUGH_EXCEPTIONS diff --git a/build/build-dev-docker-jobs-image.sh b/build/build-dev-docker-jobs-image.sh index 52df80f582..a3d439be6f 100755 --- a/build/build-dev-docker-jobs-image.sh +++ b/build/build-dev-docker-jobs-image.sh @@ -16,7 +16,7 @@ Syntax: WORKSPACE=path Path to the Arvados source tree to build packages from CWLTOOL=path (optional) Path to cwltool git repository. SALAD=path (optional) Path to schema_salad git repository. -PYCMD=pythonexec (optional) Specify the python executable to use in the docker image. Defaults to "python". +PYCMD=pythonexec (optional) Specify the python executable to use in the docker image. Defaults to "python3". EOF @@ -36,13 +36,13 @@ fi cd "$WORKSPACE" -py=python +py=python3 pipcmd=pip if [[ -n "$PYCMD" ]] ; then py="$PYCMD" - if [[ $py = python3 ]] ; then - pipcmd=pip3 - fi +fi +if [[ $py = python3 ]] ; then + pipcmd=pip3 fi (cd sdk/python && python setup.py sdist) @@ -79,6 +79,7 @@ if [[ $python_sdk_ts -gt $cwl_runner_ts ]]; then cwl_runner_version=$(cd sdk/python && nohash_version_from_git 1.0) fi -docker build --build-arg sdk=$sdk --build-arg runner=$runner --build-arg salad=$salad --build-arg cwltool=$cwltool --build-arg pythoncmd=$py --build-arg pipcmd=$pipcmd -f "$WORKSPACE/sdk/dev-jobs.dockerfile" -t arvados/jobs:$cwl_runner_version "$WORKSPACE/sdk" +set -x +docker build --no-cache --build-arg sdk=$sdk --build-arg runner=$runner --build-arg salad=$salad --build-arg cwltool=$cwltool --build-arg pythoncmd=$py --build-arg pipcmd=$pipcmd -f "$WORKSPACE/sdk/dev-jobs.dockerfile" -t arvados/jobs:$cwl_runner_version "$WORKSPACE/sdk" echo arv-keepdocker arvados/jobs $cwl_runner_version arv-keepdocker arvados/jobs $cwl_runner_version diff --git a/build/libcloud-pin.sh b/build/libcloud-pin.sh index 65e9be5685..9cc2924e1b 100644 --- a/build/libcloud-pin.sh +++ b/build/libcloud-pin.sh @@ -6,7 +6,7 @@ LIBCLOUD_PIN=2.3.1.dev2 using_fork=true if [[ $using_fork = true ]]; then - LIBCLOUD_PIN_SRC="https://github.com/curoverse/libcloud/archive/apache-libcloud-$LIBCLOUD_PIN.zip" + LIBCLOUD_PIN_SRC="https://github.com/arvados/libcloud/archive/apache-libcloud-$LIBCLOUD_PIN.zip" else LIBCLOUD_PIN_SRC="" fi diff --git a/build/package-build-dockerfiles/Makefile b/build/package-build-dockerfiles/Makefile index 6972415152..818f257525 100644 --- a/build/package-build-dockerfiles/Makefile +++ b/build/package-build-dockerfiles/Makefile @@ -2,7 +2,7 @@ # # SPDX-License-Identifier: AGPL-3.0 -all: centos7/generated debian9/generated ubuntu1604/generated ubuntu1804/generated +all: centos7/generated debian9/generated debian10/generated ubuntu1604/generated ubuntu1804/generated centos7/generated: common-generated-all test -d centos7/generated || mkdir centos7/generated @@ -12,6 +12,11 @@ debian9/generated: common-generated-all test -d debian9/generated || mkdir debian9/generated cp -rlt debian9/generated common-generated/* +debian10/generated: common-generated-all + test -d debian10/generated || mkdir debian10/generated + cp -rlt debian10/generated common-generated/* + + ubuntu1604/generated: common-generated-all test -d ubuntu1604/generated || mkdir ubuntu1604/generated cp -rlt ubuntu1604/generated common-generated/* @@ -20,7 +25,7 @@ ubuntu1804/generated: common-generated-all test -d ubuntu1804/generated || mkdir ubuntu1804/generated cp -rlt ubuntu1804/generated common-generated/* -GOTARBALL=go1.12.7.linux-amd64.tar.gz +GOTARBALL=go1.13.4.linux-amd64.tar.gz NODETARBALL=node-v6.11.2-linux-x64.tar.xz RVMKEY1=mpapis.asc RVMKEY2=pkuczynski.asc diff --git a/build/package-build-dockerfiles/centos7/Dockerfile b/build/package-build-dockerfiles/centos7/Dockerfile index 916c4abbb0..faaf91f43b 100644 --- a/build/package-build-dockerfiles/centos7/Dockerfile +++ b/build/package-build-dockerfiles/centos7/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM centos:7 -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers # Install dependencies. RUN yum -q -y install make automake gcc gcc-c++ libyaml-devel patch readline-devel zlib-devel libffi-devel openssl-devel bzip2 libtool bison sqlite-devel rpm-build git perl-ExtUtils-MakeMaker libattr-devel nss-devel libcurl-devel which tar unzip scl-utils centos-release-scl postgresql-devel python-devel python-setuptools fuse-devel xz-libs git python-virtualenv wget @@ -16,6 +16,7 @@ RUN gpg --import --no-tty /tmp/mpapis.asc && \ curl -L https://get.rvm.io | bash -s stable && \ /usr/local/rvm/bin/rvm install 2.5 && \ /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 && \ /usr/local/rvm/bin/rvm-exec default gem install fpm --version 1.10.2 # Install Bash 4.4.12 // see https://dev.arvados.org/issues/15612 @@ -30,7 +31,7 @@ RUN cd /usr/local/src \ && ln -sf /usr/local/src/bash-4.4.12/bash /bin/bash # Install golang binary -ADD generated/go1.12.7.linux-amd64.tar.gz /usr/local/ +ADD generated/go1.13.4.linux-amd64.tar.gz /usr/local/ RUN ln -s /usr/local/go/bin/go /usr/local/bin/ # Install nodejs and npm @@ -46,10 +47,10 @@ RUN scl enable rh-python36 "easy_install-3.6 pip" && easy_install-2.7 pip RUN wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm RUN rpm -ivh epel-release-latest-7.noarch.rpm -RUN git clone --depth 1 git://git.curoverse.com/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle +RUN git clone --depth 1 git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle # The version of setuptools that comes with CentOS is way too old -RUN pip install --upgrade setuptools +RUN pip install --upgrade 'setuptools<45' ENV WORKSPACE /arvados CMD ["scl", "enable", "rh-python36", "/usr/local/rvm/bin/rvm-exec default bash /jenkins/run-build-packages.sh --target centos7"] diff --git a/build/package-build-dockerfiles/debian10/Dockerfile b/build/package-build-dockerfiles/debian10/Dockerfile new file mode 100644 index 0000000000..ff86262d38 --- /dev/null +++ b/build/package-build-dockerfiles/debian10/Dockerfile @@ -0,0 +1,39 @@ +# Copyright (C) The Arvados Authors. All rights reserved. +# +# SPDX-License-Identifier: AGPL-3.0 + +## dont use debian:10 here since the word 'buster' is used for rvm precompiled binaries +FROM debian:buster +MAINTAINER Ward Vandewege + +ENV DEBIAN_FRONTEND noninteractive + +# Install dependencies. +RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip unzip python3-venv python3-dev + +# Install virtualenv +RUN /usr/bin/pip install 'virtualenv<20' + +# Install RVM +ADD generated/mpapis.asc /tmp/ +ADD generated/pkuczynski.asc /tmp/ +RUN gpg --import --no-tty /tmp/mpapis.asc && \ + gpg --import --no-tty /tmp/pkuczynski.asc && \ + curl -L https://get.rvm.io | bash -s stable && \ + /usr/local/rvm/bin/rvm install 2.5 && \ + /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 && \ + /usr/local/rvm/bin/rvm-exec default gem install fpm --version 1.10.2 + +# Install golang binary +ADD generated/go1.13.4.linux-amd64.tar.gz /usr/local/ +RUN ln -s /usr/local/go/bin/go /usr/local/bin/ + +# Install nodejs and npm +ADD generated/node-v6.11.2-linux-x64.tar.xz /usr/local/ +RUN ln -s /usr/local/node-v6.11.2-linux-x64/bin/* /usr/local/bin/ + +RUN git clone --depth 1 git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle + +ENV WORKSPACE /arvados +CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "debian10"] diff --git a/build/package-build-dockerfiles/debian9/Dockerfile b/build/package-build-dockerfiles/debian9/Dockerfile index c403d79bcc..257523a72f 100644 --- a/build/package-build-dockerfiles/debian9/Dockerfile +++ b/build/package-build-dockerfiles/debian9/Dockerfile @@ -12,7 +12,7 @@ ENV DEBIAN_FRONTEND noninteractive RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev curl git procps libattr1-dev libfuse-dev libgnutls28-dev libpq-dev python-pip unzip python3-venv python3-dev # Install virtualenv -RUN /usr/bin/pip install virtualenv +RUN /usr/bin/pip install 'virtualenv<20' # Install RVM ADD generated/mpapis.asc /tmp/ @@ -22,17 +22,18 @@ RUN gpg --import --no-tty /tmp/mpapis.asc && \ curl -L https://get.rvm.io | bash -s stable && \ /usr/local/rvm/bin/rvm install 2.5 && \ /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 && \ /usr/local/rvm/bin/rvm-exec default gem install fpm --version 1.10.2 # Install golang binary -ADD generated/go1.12.7.linux-amd64.tar.gz /usr/local/ +ADD generated/go1.13.4.linux-amd64.tar.gz /usr/local/ RUN ln -s /usr/local/go/bin/go /usr/local/bin/ # Install nodejs and npm ADD generated/node-v6.11.2-linux-x64.tar.xz /usr/local/ RUN ln -s /usr/local/node-v6.11.2-linux-x64/bin/* /usr/local/bin/ -RUN git clone --depth 1 git://git.curoverse.com/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle +RUN git clone --depth 1 git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle ENV WORKSPACE /arvados CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "debian9"] diff --git a/build/package-build-dockerfiles/ubuntu1604/Dockerfile b/build/package-build-dockerfiles/ubuntu1604/Dockerfile index 90f340e66c..6b1304265b 100644 --- a/build/package-build-dockerfiles/ubuntu1604/Dockerfile +++ b/build/package-build-dockerfiles/ubuntu1604/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM ubuntu:xenial -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive @@ -11,7 +11,7 @@ ENV DEBIAN_FRONTEND noninteractive RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-setuptools python3-pip libcurl4-gnutls-dev libgnutls-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip tzdata python3-venv python3-dev # Install virtualenv -RUN /usr/bin/pip install virtualenv +RUN /usr/bin/pip install 'virtualenv<20' # Install RVM ADD generated/mpapis.asc /tmp/ @@ -21,17 +21,18 @@ RUN gpg --import --no-tty /tmp/mpapis.asc && \ curl -L https://get.rvm.io | bash -s stable && \ /usr/local/rvm/bin/rvm install 2.5 && \ /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 && \ /usr/local/rvm/bin/rvm-exec default gem install fpm --version 1.10.2 # Install golang binary -ADD generated/go1.12.7.linux-amd64.tar.gz /usr/local/ +ADD generated/go1.13.4.linux-amd64.tar.gz /usr/local/ RUN ln -s /usr/local/go/bin/go /usr/local/bin/ # Install nodejs and npm ADD generated/node-v6.11.2-linux-x64.tar.xz /usr/local/ RUN ln -s /usr/local/node-v6.11.2-linux-x64/bin/* /usr/local/bin/ -RUN git clone --depth 1 git://git.curoverse.com/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle +RUN git clone --depth 1 git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle ENV WORKSPACE /arvados CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "ubuntu1604"] diff --git a/build/package-build-dockerfiles/ubuntu1804/Dockerfile b/build/package-build-dockerfiles/ubuntu1804/Dockerfile index 1adff74000..58bff61603 100644 --- a/build/package-build-dockerfiles/ubuntu1804/Dockerfile +++ b/build/package-build-dockerfiles/ubuntu1804/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM ubuntu:bionic -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive @@ -11,7 +11,7 @@ ENV DEBIAN_FRONTEND noninteractive RUN /usr/bin/apt-get update && /usr/bin/apt-get install -q -y python2.7-dev python3 python-setuptools python3-pip libcurl4-gnutls-dev libgnutls28-dev curl git libattr1-dev libfuse-dev libpq-dev python-pip unzip tzdata python3-venv python3-dev # Install virtualenv -RUN /usr/bin/pip install virtualenv +RUN /usr/bin/pip install 'virtualenv<20' # Install RVM ADD generated/mpapis.asc /tmp/ @@ -21,17 +21,18 @@ RUN gpg --import --no-tty /tmp/mpapis.asc && \ curl -L https://get.rvm.io | bash -s stable && \ /usr/local/rvm/bin/rvm install 2.5 && \ /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 && \ /usr/local/rvm/bin/rvm-exec default gem install fpm --version 1.10.2 # Install golang binary -ADD generated/go1.12.7.linux-amd64.tar.gz /usr/local/ +ADD generated/go1.13.4.linux-amd64.tar.gz /usr/local/ RUN ln -s /usr/local/go/bin/go /usr/local/bin/ # Install nodejs and npm ADD generated/node-v6.11.2-linux-x64.tar.xz /usr/local/ RUN ln -s /usr/local/node-v6.11.2-linux-x64/bin/* /usr/local/bin/ -RUN git clone --depth 1 git://git.curoverse.com/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle +RUN git clone --depth 1 git://git.arvados.org/arvados.git /tmp/arvados && cd /tmp/arvados/services/api && /usr/local/rvm/bin/rvm-exec default bundle && cd /tmp/arvados/apps/workbench && /usr/local/rvm/bin/rvm-exec default bundle ENV WORKSPACE /arvados CMD ["/usr/local/rvm/bin/rvm-exec", "default", "bash", "/jenkins/run-build-packages.sh", "--target", "ubuntu1804"] diff --git a/build/package-test-dockerfiles/Makefile b/build/package-test-dockerfiles/Makefile index c7b32968ff..1066750fe5 100644 --- a/build/package-test-dockerfiles/Makefile +++ b/build/package-test-dockerfiles/Makefile @@ -2,7 +2,7 @@ # # SPDX-License-Identifier: AGPL-3.0 -all: centos7/generated debian9/generated ubuntu1604/generated ubuntu1804/generated +all: centos7/generated debian9/generated debian10/generated ubuntu1604/generated ubuntu1804/generated centos7/generated: common-generated-all test -d centos7/generated || mkdir centos7/generated @@ -12,6 +12,10 @@ debian9/generated: common-generated-all test -d debian9/generated || mkdir debian9/generated cp -rlt debian9/generated common-generated/* +debian10/generated: common-generated-all + test -d debian10/generated || mkdir debian10/generated + cp -rlt debian10/generated common-generated/* + ubuntu1604/generated: common-generated-all test -d ubuntu1604/generated || mkdir ubuntu1604/generated cp -rlt ubuntu1604/generated common-generated/* diff --git a/build/package-test-dockerfiles/centos7/Dockerfile b/build/package-test-dockerfiles/centos7/Dockerfile index 49d04aa744..3d68cfc00b 100644 --- a/build/package-test-dockerfiles/centos7/Dockerfile +++ b/build/package-test-dockerfiles/centos7/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM centos:7 -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers # Install dependencies. RUN yum -q -y install scl-utils centos-release-scl which tar wget @@ -16,7 +16,8 @@ RUN touch /var/lib/rpm/* && \ gpg --import --no-tty /tmp/pkuczynski.asc && \ curl -L https://get.rvm.io | bash -s stable && \ /usr/local/rvm/bin/rvm install 2.3 && \ - /usr/local/rvm/bin/rvm alias create default ruby-2.3 + /usr/local/rvm/bin/rvm alias create default ruby-2.3 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 # Install Bash 4.4.12 // see https://dev.arvados.org/issues/15612 RUN cd /usr/local/src \ diff --git a/build/package-test-dockerfiles/debian10/Dockerfile b/build/package-test-dockerfiles/debian10/Dockerfile new file mode 100644 index 0000000000..32996e4a54 --- /dev/null +++ b/build/package-test-dockerfiles/debian10/Dockerfile @@ -0,0 +1,27 @@ +# Copyright (C) The Arvados Authors. All rights reserved. +# +# SPDX-License-Identifier: AGPL-3.0 + +FROM debian:buster +MAINTAINER Arvados Package Maintainers + +ENV DEBIAN_FRONTEND noninteractive + +# Install dependencies +RUN apt-get update && \ + apt-get -y install --no-install-recommends curl ca-certificates gpg procps gpg-agent + +# Install RVM +ADD generated/mpapis.asc /tmp/ +ADD generated/pkuczynski.asc /tmp/ +RUN gpg --import --no-tty /tmp/mpapis.asc && \ + gpg --import --no-tty /tmp/pkuczynski.asc && \ + curl -L https://get.rvm.io | bash -s stable && \ + /usr/local/rvm/bin/rvm install 2.5 && \ + /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 + +# udev daemon can't start in a container, so don't try. +RUN mkdir -p /etc/udev/disabled + +RUN echo "deb file:///arvados/packages/debian10/ /" >>/etc/apt/sources.list diff --git a/build/package-test-dockerfiles/debian9/Dockerfile b/build/package-test-dockerfiles/debian9/Dockerfile index e759c9ce1d..423a9e7c37 100644 --- a/build/package-test-dockerfiles/debian9/Dockerfile +++ b/build/package-test-dockerfiles/debian9/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM debian:stretch -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive @@ -18,7 +18,8 @@ RUN gpg --import --no-tty /tmp/mpapis.asc && \ gpg --import --no-tty /tmp/pkuczynski.asc && \ curl -L https://get.rvm.io | bash -s stable && \ /usr/local/rvm/bin/rvm install 2.5 && \ - /usr/local/rvm/bin/rvm alias create default ruby-2.5 + /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 # udev daemon can't start in a container, so don't try. RUN mkdir -p /etc/udev/disabled diff --git a/build/package-test-dockerfiles/ubuntu1604/Dockerfile b/build/package-test-dockerfiles/ubuntu1604/Dockerfile index 422ee43e06..e0432c20ee 100644 --- a/build/package-test-dockerfiles/ubuntu1604/Dockerfile +++ b/build/package-test-dockerfiles/ubuntu1604/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM ubuntu:xenial -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive @@ -18,7 +18,8 @@ RUN gpg --import --no-tty /tmp/mpapis.asc && \ gpg --import --no-tty /tmp/pkuczynski.asc && \ curl -L https://get.rvm.io | bash -s stable && \ /usr/local/rvm/bin/rvm install 2.5 && \ - /usr/local/rvm/bin/rvm alias create default ruby-2.5 + /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 # udev daemon can't start in a container, so don't try. RUN mkdir -p /etc/udev/disabled diff --git a/build/package-test-dockerfiles/ubuntu1804/Dockerfile b/build/package-test-dockerfiles/ubuntu1804/Dockerfile index 68babe3fd5..2d4189879e 100644 --- a/build/package-test-dockerfiles/ubuntu1804/Dockerfile +++ b/build/package-test-dockerfiles/ubuntu1804/Dockerfile @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0 FROM ubuntu:bionic -MAINTAINER Ward Vandewege +MAINTAINER Arvados Package Maintainers ENV DEBIAN_FRONTEND noninteractive @@ -18,7 +18,8 @@ RUN gpg --import --no-tty /tmp/mpapis.asc && \ gpg --import --no-tty /tmp/pkuczynski.asc && \ curl -L https://get.rvm.io | bash -s stable && \ /usr/local/rvm/bin/rvm install 2.5 && \ - /usr/local/rvm/bin/rvm alias create default ruby-2.5 + /usr/local/rvm/bin/rvm alias create default ruby-2.5 && \ + /usr/local/rvm/bin/rvm-exec default gem install bundler --version 2.0.2 # udev daemon can't start in a container, so don't try. RUN mkdir -p /etc/udev/disabled diff --git a/build/package-testing/deb-common-test-packages.sh b/build/package-testing/deb-common-test-packages.sh index 77017ba970..32fb2009e1 100755 --- a/build/package-testing/deb-common-test-packages.sh +++ b/build/package-testing/deb-common-test-packages.sh @@ -23,7 +23,7 @@ export ARV_PACKAGES_DIR="/arvados/packages/$target" dpkg-query --show > "$ARV_PACKAGES_DIR/$1.before" -apt-get $DASHQQ_UNLESS_DEBUG update +apt-get $DASHQQ_UNLESS_DEBUG --allow-insecure-repositories update apt-get $DASHQQ_UNLESS_DEBUG -y --allow-unauthenticated install "$1" >"$STDOUT_IF_DEBUG" 2>"$STDERR_IF_DEBUG" diff --git a/build/package-testing/test-packages-debian10.sh b/build/package-testing/test-packages-debian10.sh new file mode 120000 index 0000000000..54ce94c357 --- /dev/null +++ b/build/package-testing/test-packages-debian10.sh @@ -0,0 +1 @@ +deb-common-test-packages.sh \ No newline at end of file diff --git a/build/rails-package-scripts/postinst.sh b/build/rails-package-scripts/postinst.sh index 56d55d3276..7ea21848b2 100644 --- a/build/rails-package-scripts/postinst.sh +++ b/build/rails-package-scripts/postinst.sh @@ -186,10 +186,6 @@ configure_version() { setup_confdirs /etc/arvados "$CONFIG_PATH" setup_conffile environments/production.rb environments/production.rb.example \ || true - setup_conffile application.yml application.yml.example || APPLICATION_READY=0 - if [ -n "$RAILSPKG_DATABASE_LOAD_TASK" ]; then - setup_conffile database.yml database.yml.example || DATABASE_READY=0 - fi setup_extra_conffiles echo "... done." @@ -203,7 +199,7 @@ configure_version() { export RAILS_ENV=production if ! $COMMAND_PREFIX bundle --version >/dev/null; then - run_and_report "Installing bundle" $COMMAND_PREFIX gem install bundle + run_and_report "Installing bundler" $COMMAND_PREFIX gem install bundler --version 1.17.3 fi run_and_report "Running bundle install" \ @@ -229,7 +225,7 @@ configure_version() { fi if [ 11 = "$RAILSPKG_SUPPORTS_CONFIG_CHECK$APPLICATION_READY" ]; then - run_and_report "Checking application.yml for completeness" \ + run_and_report "Checking configuration for completeness" \ $COMMAND_PREFIX bundle exec rake config:check || APPLICATION_READY=0 fi @@ -258,9 +254,9 @@ elif [ "$1" = "0" ] || [ "$1" = "1" ] || [ "$1" = "2" ]; then configure_version fi -report_not_ready "$DATABASE_READY" "$CONFIG_PATH/database.yml" if printf '%s\n' "$CONFIG_PATH" | grep -Fqe "sso"; then report_not_ready "$APPLICATION_READY" "$CONFIG_PATH/application.yml" + report_not_ready "$DATABASE_READY" "$CONFIG_PATH/database.yml" else report_not_ready "$APPLICATION_READY" "/etc/arvados/config.yml" fi diff --git a/build/run-build-packages-one-target.sh b/build/run-build-packages-one-target.sh index 378c9bbfa3..f476a9691c 100755 --- a/build/run-build-packages-one-target.sh +++ b/build/run-build-packages-one-target.sh @@ -10,7 +10,7 @@ Syntax: WORKSPACE=/path/to/arvados $(basename $0) [options] --target - Distribution to build packages for (default: debian9) + Distribution to build packages for (default: debian10) --command Build command to execute (default: use built-in Docker image command) --test-packages @@ -21,6 +21,9 @@ Syntax: Build only a specific package --only-test Test only a specific package +--force-build + Build even if the package exists upstream or if it has already been + built locally --force-test Test even if there is no new untested package --build-version @@ -51,13 +54,14 @@ if ! [[ -d "$WORKSPACE" ]]; then fi PARSEDOPTS=$(getopt --name "$0" --longoptions \ - help,debug,test-packages,target:,command:,only-test:,force-test,only-build:,build-version: \ + help,debug,test-packages,target:,command:,only-test:,force-test,only-build:,force-build,build-version: \ -- "" "$@") if [ $? -ne 0 ]; then exit 1 fi -TARGET=debian9 +TARGET=debian10 +FORCE_BUILD=0 COMMAND= DEBUG= @@ -80,6 +84,9 @@ while [ $# -gt 0 ]; do --force-test) FORCE_TEST=true ;; + --force-build) + FORCE_BUILD=1 + ;; --only-build) ONLY_BUILD="$2"; shift ;; @@ -269,6 +276,7 @@ else --env ARVADOS_BUILDING_ITERATION="$ARVADOS_BUILDING_ITERATION" \ --env ARVADOS_DEBUG=$ARVADOS_DEBUG \ --env "ONLY_BUILD=$ONLY_BUILD" \ + --env "FORCE_BUILD=$FORCE_BUILD" \ "$IMAGE" $COMMAND then echo diff --git a/build/run-build-packages-sso.sh b/build/run-build-packages-sso.sh index e7a3aacda3..d8d9b984a0 100755 --- a/build/run-build-packages-sso.sh +++ b/build/run-build-packages-sso.sh @@ -17,7 +17,7 @@ Options: --debug Output debug information (default: false) --target - Distribution to build packages for (default: debian9) + Distribution to build packages for (default: debian10) WORKSPACE=path Path to the Arvados SSO source tree to build packages from @@ -25,7 +25,7 @@ EOF EXITCODE=0 DEBUG=${ARVADOS_DEBUG:-0} -TARGET=debian9 +TARGET=debian10 PARSEDOPTS=$(getopt --name "$0" --longoptions \ help,build-bundle-packages,debug,target: \ diff --git a/build/run-build-packages.sh b/build/run-build-packages.sh index fd313cf10e..4faa1c6b0d 100755 --- a/build/run-build-packages.sh +++ b/build/run-build-packages.sh @@ -19,9 +19,12 @@ Options: --debug Output debug information (default: false) --target - Distribution to build packages for (default: debian9) + Distribution to build packages for (default: debian10) --only-build Build only a specific package (or $ONLY_BUILD from environment) +--force-build + Build even if the package exists upstream or if it has already been + built locally --command Build command to execute (defaults to the run command defined in the Docker image) @@ -35,18 +38,19 @@ EOF # set to --no-cache-dir to disable pip caching CACHE_FLAG= -MAINTAINER="Ward Vandewege " -VENDOR="Veritas Genetics, Inc." +MAINTAINER="Arvados Package Maintainers " +VENDOR="The Arvados Project" # End of user configuration DEBUG=${ARVADOS_DEBUG:-0} +FORCE_BUILD=${FORCE_BUILD:-0} EXITCODE=0 -TARGET=debian9 +TARGET=debian10 COMMAND= PARSEDOPTS=$(getopt --name "$0" --longoptions \ - help,build-bundle-packages,debug,target:,only-build: \ + help,build-bundle-packages,debug,target:,only-build:,force-build \ -- "" "$@") if [ $? -ne 0 ]; then exit 1 @@ -66,6 +70,9 @@ while [ $# -gt 0 ]; do --only-build) ONLY_BUILD="$2"; shift ;; + --force-build) + FORCE_BUILD=1 + ;; --debug) DEBUG=1 ;; @@ -157,14 +164,6 @@ if [[ "$?" != 0 ]]; then exit 1 fi -PYTHON2_FPM_INSTALLER=(--python-easyinstall "$(find_python_program easy_install-$PYTHON2_VERSION easy_install)") -install3=$(find_python_program easy_install-$PYTHON3_VERSION easy_install3 pip-$PYTHON3_VERSION pip3) -if [[ $install3 =~ easy_ ]]; then - PYTHON3_FPM_INSTALLER=(--python-easyinstall "$install3") -else - PYTHON3_FPM_INSTALLER=(--python-pip "$install3") -fi - RUN_BUILD_PACKAGES_PATH="`dirname \"$0\"`" RUN_BUILD_PACKAGES_PATH="`( cd \"$RUN_BUILD_PACKAGES_PATH\" && pwd )`" # absolutized and normalized if [ -z "$RUN_BUILD_PACKAGES_PATH" ] ; then @@ -281,7 +280,6 @@ debug_echo -e "\nPython packages\n" # Go binaries cd $WORKSPACE/packages/$TARGET export GOPATH=$(mktemp -d) -go get github.com/kardianos/govendor package_go_binary cmd/arvados-client arvados-client \ "Arvados command line tool (beta)" package_go_binary cmd/arvados-server arvados-server \ @@ -296,7 +294,7 @@ package_go_binary services/crunch-dispatch-local crunch-dispatch-local \ "Dispatch Crunch containers on the local system" package_go_binary services/crunch-dispatch-slurm crunch-dispatch-slurm \ "Dispatch Crunch containers to a SLURM cluster" -package_go_binary services/crunch-run crunch-run \ +package_go_binary cmd/arvados-server crunch-run \ "Supervise a single Crunch container" package_go_binary services/crunchstat crunchstat \ "Gather cpu/memory/network statistics of running Crunch jobs" @@ -324,8 +322,11 @@ package_go_binary tools/keep-exercise keep-exercise \ # The Python SDK - Should be built first because it's needed by others fpm_build_virtualenv "arvados-python-client" "sdk/python" -# Arvados cwl runner -fpm_build_virtualenv "arvados-cwl-runner" "sdk/cwl" +# The Python SDK - Python3 package +fpm_build_virtualenv "arvados-python-client" "sdk/python" "python3" + +# Arvados cwl runner - Only supports Python3 now +fpm_build_virtualenv "arvados-cwl-runner" "sdk/cwl" "python3" # The PAM module fpm_build_virtualenv "libpam-arvados" "sdk/pam" @@ -339,9 +340,6 @@ fpm_build_virtualenv "arvados-node-manager" "services/nodemanager" # The Arvados crunchstat-summary tool fpm_build_virtualenv "crunchstat-summary" "tools/crunchstat-summary" -# The Python SDK - Python3 package -fpm_build_virtualenv "arvados-python-client" "sdk/python" "python3" - # The Docker image cleaner fpm_build_virtualenv "arvados-docker-cleaner" "services/dockercleaner" "python3" @@ -351,6 +349,8 @@ if [[ -e "$WORKSPACE/cwltest" ]]; then rm -rf "$WORKSPACE/cwltest" fi git clone https://github.com/common-workflow-language/cwltest.git +# last release to support python 2.7 +(cd cwltest && git checkout 1.0.20190906212748) # signal to our build script that we want a cwltest executable installed in /usr/bin/ mkdir cwltest/bin && touch cwltest/bin/cwltest fpm_build_virtualenv "cwltest" "cwltest" diff --git a/build/run-build-test-packages-one-target.sh b/build/run-build-test-packages-one-target.sh index d75e2785ec..1c8296dfca 100755 --- a/build/run-build-test-packages-one-target.sh +++ b/build/run-build-test-packages-one-target.sh @@ -10,7 +10,7 @@ Syntax: WORKSPACE=/path/to/arvados $(basename $0) [options] --target - Distribution to build packages for (default: debian9) + Distribution to build packages for (default: debian10) --upload If the build and test steps are successful, upload the packages to a remote apt repository (default: false) @@ -50,7 +50,7 @@ if [ $? -ne 0 ]; then exit 1 fi -TARGET=debian9 +TARGET=debian10 UPLOAD=0 RC=0 DEBUG= @@ -128,11 +128,11 @@ if [[ "$UPLOAD" != 0 ]]; then if [ ${#failures[@]} -eq 0 ]; then if [[ "$RC" != 0 ]]; then - echo "/usr/local/arvados-dev/jenkins/run_upload_packages_testing.py -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace $WORKSPACE $TARGET" - /usr/local/arvados-dev/jenkins/run_upload_packages_testing.py -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace $WORKSPACE $TARGET + echo "/usr/local/arvados-dev/jenkins/run_upload_packages.py --repo testing -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace $WORKSPACE $TARGET" + /usr/local/arvados-dev/jenkins/run_upload_packages.py --repo testing -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace $WORKSPACE $TARGET else - echo "/usr/local/arvados-dev/jenkins/run_upload_packages.py -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace $WORKSPACE $TARGET" - /usr/local/arvados-dev/jenkins/run_upload_packages.py -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace $WORKSPACE $TARGET + echo "/usr/local/arvados-dev/jenkins/run_upload_packages.py --repo dev -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace $WORKSPACE $TARGET" + /usr/local/arvados-dev/jenkins/run_upload_packages.py --repo dev -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace $WORKSPACE $TARGET fi else echo "Skipping package upload, there were errors building and/or testing the packages" diff --git a/build/run-library.sh b/build/run-library.sh index 95f2ff1452..ac5dc718be 100755 --- a/build/run-library.sh +++ b/build/run-library.sh @@ -40,27 +40,24 @@ EOF format_last_commit_here() { local format="$1"; shift - TZ=UTC git log -n1 --first-parent "--format=format:$format" . + local dir="${1:-.}"; shift + TZ=UTC git log -n1 --first-parent "--format=format:$format" "$dir" } version_from_git() { # Output the version being built, or if we're building a # dev/prerelease, output a version number based on the git log for - # the current working directory. + # the given $subdir. + local minorversion="$1"; shift # unused + local subdir="$1"; shift if [[ -n "$ARVADOS_BUILDING_VERSION" ]]; then echo "$ARVADOS_BUILDING_VERSION" return fi - local git_ts git_hash prefix - if [[ -n "$1" ]] ; then - prefix="$1" - else - prefix="0.1" - fi - - declare $(format_last_commit_here "git_ts=%ct git_hash=%h") - ARVADOS_BUILDING_VERSION="$(git tag -l |sort -V -r |head -n1).$(date -ud "@$git_ts" +%Y%m%d%H%M%S)" + local git_ts git_hash + declare $(format_last_commit_here "git_ts=%ct git_hash=%h" "$subdir") + ARVADOS_BUILDING_VERSION="$($WORKSPACE/build/version-at-commit.sh $git_hash)" echo "$ARVADOS_BUILDING_VERSION" } @@ -73,7 +70,8 @@ nohash_version_from_git() { } timestamp_from_git() { - format_last_commit_here "%ct" + local subdir="$1"; shift + format_last_commit_here "%ct" "$subdir" } handle_python_package () { @@ -108,32 +106,32 @@ calculate_go_package_version() { # to another variable that is passed in as the first argument to this function. # see https://www.gnu.org/software/bash/manual/html_node/Shell-Parameters.html local -n __returnvar="$1"; shift - local src_path="$1"; shift - - mkdir -p "$GOPATH/src/git.curoverse.com" - ln -sfn "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" - (cd "$GOPATH/src/git.curoverse.com/arvados.git" && "$GOPATH/bin/govendor" sync -v) + local oldpwd="$PWD" - cd "$GOPATH/src/git.curoverse.com/arvados.git/$src_path" - local version="$(version_from_git)" - local timestamp="$(timestamp_from_git)" + cd "$WORKSPACE" + go mod download # Update the version number and build a new package if the vendor # bundle has changed, or the command imports anything from the # Arvados SDK and the SDK has changed. - declare -a checkdirs=(vendor) - if grep -qr git.curoverse.com/arvados .; then + declare -a checkdirs=(go.mod go.sum) + while [ -n "$1" ]; do + checkdirs+=("$1") + shift + done + if grep -qr git.arvados.org/arvados .; then checkdirs+=(sdk/go lib) fi + local timestamp=0 for dir in ${checkdirs[@]}; do - cd "$GOPATH/src/git.curoverse.com/arvados.git/$dir" - ts="$(timestamp_from_git)" + cd "$WORKSPACE" + ts="$(timestamp_from_git "$dir")" if [[ "$ts" -gt "$timestamp" ]]; then - version=$(version_from_git) + version=$(version_from_git "" "$dir") timestamp="$ts" fi done - + cd "$oldpwd" __returnvar="$version" } @@ -164,7 +162,7 @@ package_go_binary() { return 1 fi - go get -ldflags "-X git.curoverse.com/arvados.git/lib/cmd.version=${go_package_version} -X main.version=${go_package_version}" "git.curoverse.com/arvados.git/$src_path" + go get -ldflags "-X git.arvados.org/arvados.git/lib/cmd.version=${go_package_version} -X main.version=${go_package_version}" "git.arvados.org/arvados.git/$src_path" local -a switches=() systemd_unit="$WORKSPACE/${src_path}/${prog}.service" @@ -211,20 +209,14 @@ _build_rails_package_scripts() { rails_package_version() { local pkgname="$1"; shift + local srcdir="$1"; shift if [[ -n "$ARVADOS_BUILDING_VERSION" ]]; then echo "$ARVADOS_BUILDING_VERSION" return fi local version="$(version_from_git)" if [ $pkgname = "arvados-api-server" -o $pkgname = "arvados-workbench" ] ; then - local P="$PWD" - cd $WORKSPACE - local arvados_server_version - calculate_go_package_version arvados_server_version cmd/arvados-server - cd $P - if [ $arvados_server_version > $version ] ; then - version=$arvados_server_version - fi + calculate_go_package_version version cmd/arvados-server "$srcdir" fi echo $version } @@ -241,7 +233,7 @@ test_rails_package_presence() { cd $srcdir - local version="$(rails_package_version $pkgname)" + local version="$(rails_package_version "$pkgname" "$srcdir")" cd $tmppwd @@ -319,7 +311,9 @@ test_package_presence() { # sure it gets picked up by the test and/or upload steps. # Get the list of packages from the repos - if [[ "$FORMAT" == "deb" ]]; then + if [[ "$FORCE_BUILD" == "1" ]]; then + echo "Package $full_pkgname build forced with --force-build, building" + elif [[ "$FORMAT" == "deb" ]]; then declare -A dd dd[debian9]=stretch dd[debian10]=buster @@ -373,7 +367,7 @@ handle_rails_package() { local srcdir="$1"; shift cd "$srcdir" local license_path="$1"; shift - local version="$(rails_package_version $pkgname)" + local version="$(rails_package_version "$pkgname" "$srcdir")" echo "$version" >package-build.version local scripts_dir="$(mktemp --tmpdir -d "$pkgname-XXXXXXXX.scripts")" && \ ( @@ -481,9 +475,9 @@ fpm_build_virtualenv () { rm -rf dist/* # Get the latest setuptools - if ! $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools; then + if ! $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'; then echo "Error, unable to upgrade setuptools with" - echo " $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools" + echo " $pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'" exit 1 fi # filter a useless warning (when building the cwltest package) from the stderr output @@ -538,9 +532,9 @@ fpm_build_virtualenv () { fi echo "pip version: `build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip --version`" - if ! build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools; then + if ! build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'; then echo "Error, unable to upgrade setuptools with" - echo " build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U setuptools" + echo " build/usr/share/$python/dist/$PYTHON_PKG/bin/$pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG -U 'setuptools<45'" exit 1 fi echo "setuptools version: `build/usr/share/$python/dist/$PYTHON_PKG/bin/$python -c 'import setuptools; print(setuptools.__version__)'`" diff --git a/build/run-tests.sh b/build/run-tests.sh index 0014547ce5..7a3302de9b 100755 --- a/build/run-tests.sh +++ b/build/run-tests.sh @@ -81,6 +81,7 @@ lib/controller/railsproxy lib/controller/router lib/controller/rpc lib/crunchstat +lib/crunch-run lib/cloud lib/cloud/azure lib/cloud/cloudtest @@ -89,6 +90,7 @@ lib/dispatchcloud/container lib/dispatchcloud/scheduler lib/dispatchcloud/ssh_executor lib/dispatchcloud/worker +lib/mount lib/service services/api services/arv-git-httpd @@ -104,7 +106,6 @@ services/keep-balance services/login-sync services/nodemanager services/nodemanager_integration -services/crunch-run services/crunch-dispatch-local services/crunch-dispatch-slurm services/ws @@ -126,7 +127,7 @@ sdk/go/blockdigest sdk/go/asyncbuf sdk/go/stats sdk/go/crunchrunner -sdk/cwl +sdk/cwl:py3 sdk/R sdk/java-v2 tools/sync-groups @@ -392,7 +393,7 @@ checkpidfile() { checkhealth() { svc="$1" - base=$(python -c "import yaml; print list(yaml.safe_load(file('$ARVADOS_CONFIG'))['Clusters']['zzzzz']['Services']['$1']['InternalURLs'].keys())[0]") + base=$("${VENVDIR}/bin/python" -c "import yaml; print list(yaml.safe_load(file('$ARVADOS_CONFIG'))['Clusters']['zzzzz']['Services']['$1']['InternalURLs'].keys())[0]") url="$base/_health/ping" if ! curl -Ss -H "Authorization: Bearer e687950a23c3a9bceec28c6223a06c79" "${url}" | tee -a /dev/stderr | grep '"OK"'; then echo "${url} failed" @@ -404,7 +405,7 @@ checkdiscoverydoc() { dd="https://${1}/discovery/v1/apis/arvados/v1/rest" if ! (set -o pipefail; curl -fsk "$dd" | grep -q ^{ ); then echo >&2 "ERROR: could not retrieve discovery doc from RailsAPI at $dd" - tail -v $WORKSPACE/services/api/log/test.log + tail -v $WORKSPACE/tmp/railsapi.log return 1 fi echo "${dd} ok" @@ -520,6 +521,10 @@ setup_ruby_environment() { || fatal 'rvm gemset setup' rvm env + (bundle version | grep -q 2.0.2) || gem install bundler -v 2.0.2 + bundle="$(which bundle)" + echo "$bundle" + "$bundle" version | grep 2.0.2 || fatal 'install bundler' else # When our "bundle install"s need to install new gems to # satisfy dependencies, we want them to go where "gem install @@ -545,9 +550,14 @@ setup_ruby_environment() { echo "Will install dependencies to $(gem env gemdir)" echo "Will install arvados gems to $tmpdir_gem_home" echo "Gem search path is GEM_PATH=$GEM_PATH" + bundle="$(gem env gempath | cut -f1 -d:)/bin/bundle" + ( + export HOME=$GEMHOME + ("$bundle" version | grep -q 2.0.2) \ + || gem install --user bundler -v 2.0.2 + "$bundle" version | tee /dev/stderr | grep -q 'version 2' + ) || fatal 'install bundler' fi - bundle config || gem install bundler \ - || fatal 'install bundler' } with_test_gemset() { @@ -631,33 +641,20 @@ initialize() { } install_env() { - ( - set -e - mkdir -p "$GOPATH/src/git.curoverse.com" - if [[ ! -h "$GOPATH/src/git.curoverse.com/arvados.git" ]]; then - for d in \ - "$GOPATH/src/git.curoverse.com/arvados.git/tmp/GOPATH" \ - "$GOPATH/src/git.curoverse.com/arvados.git/tmp" \ - "$GOPATH/src/git.curoverse.com/arvados.git/arvados" \ - "$GOPATH/src/git.curoverse.com/arvados.git"; do - [[ -h "$d" ]] && rm "$d" - [[ -d "$d" ]] && rmdir "$d" - done - fi - ln -vsfT "$WORKSPACE" "$GOPATH/src/git.curoverse.com/arvados.git" - go get -v github.com/kardianos/govendor - cd "$GOPATH/src/git.curoverse.com/arvados.git" - go get -v -d ... - "$GOPATH/bin/govendor" sync - which goimports >/dev/null || go get golang.org/x/tools/cmd/goimports - ) || fatal "Go setup failed" + go mod download || fatal "Go deps failed" + which goimports >/dev/null || go get golang.org/x/tools/cmd/goimports || fatal "Go setup failed" setup_virtualenv "$VENVDIR" --python python2.7 . "$VENVDIR/bin/activate" # Needed for run_test_server.py which is used by certain (non-Python) tests. - pip install --no-cache-dir PyYAML future \ - || fatal "pip install PyYAML failed" + ( + set -e + "${VENVDIR}/bin/pip" install PyYAML + "${VENV3DIR}/bin/pip" install PyYAML + cd "$WORKSPACE/sdk/python" + python setup.py install + ) || fatal "installing PyYAML and sdk/python failed" # Preinstall libcloud if using a fork; otherwise nodemanager "pip # install" won't pick it up by default. @@ -683,11 +680,6 @@ Warning: python3 could not be found. Python 3 tests will be skipped. EOF fi - - if ! which bundler >/dev/null - then - gem install --user-install bundler || fatal 'Could not install bundler' - fi } retry() { @@ -735,7 +727,7 @@ do_test() { stop_services check_arvados_config "$1" ;; - gofmt | govendor | doc | lib/cli | lib/cloud/azure | lib/cloud/ec2 | lib/cloud/cloudtest | lib/cmd | lib/dispatchcloud/ssh_executor | lib/dispatchcloud/worker) + gofmt | doc | lib/cli | lib/cloud/azure | lib/cloud/ec2 | lib/cloud/cloudtest | lib/cmd | lib/dispatchcloud/ssh_executor | lib/dispatchcloud/worker) check_arvados_config "$1" # don't care whether services are running ;; @@ -753,7 +745,7 @@ do_test() { go_ldflags() { version=${ARVADOS_VERSION:-$(git log -n1 --format=%H)-dev} - echo "-X git.curoverse.com/arvados.git/lib/cmd.version=${version} -X main.version=${version}" + echo "-X git.arvados.org/arvados.git/lib/cmd.version=${version} -X main.version=${version}" } do_test_once() { @@ -771,12 +763,12 @@ do_test_once() { then covername="coverage-$(echo "$1" | sed -e 's/\//_/g')" coverflags=("-covermode=count" "-coverprofile=$WORKSPACE/tmp/.$covername.tmp") - # We do "go get -t" here to catch compilation errors + # We do "go install" here to catch compilation errors # before trying "go test". Otherwise, coverage-reporting # mode makes Go show the wrong line numbers when reporting # compilation errors. - go get -ldflags "$(go_ldflags)" -t "git.curoverse.com/arvados.git/$1" && \ - cd "$GOPATH/src/git.curoverse.com/arvados.git/$1" && \ + go install -ldflags "$(go_ldflags)" "$WORKSPACE/$1" && \ + cd "$WORKSPACE/$1" && \ if [[ -n "${testargs[$1]}" ]] then # "go test -check.vv giturl" doesn't work, but this @@ -785,7 +777,7 @@ do_test_once() { else # The above form gets verbose even when testargs is # empty, so use this form in such cases: - go test ${short:+-short} ${coverflags[@]} "git.curoverse.com/arvados.git/$1" + go test ${short:+-short} ${coverflags[@]} "git.arvados.org/arvados.git/$1" fi result=${result:-$?} if [[ -f "$WORKSPACE/tmp/.$covername.tmp" ]] @@ -863,7 +855,7 @@ do_install_once() { result=1 elif [[ "$2" == "go" ]] then - go get -ldflags "$(go_ldflags)" -t "git.curoverse.com/arvados.git/$1" + go install -ldflags "$(go_ldflags)" "$WORKSPACE/$1" elif [[ "$2" == "pip" ]] then # $3 can name a path directory for us to use, including trailing @@ -881,8 +873,8 @@ do_install_once() { cd "$WORKSPACE/$1" \ && "${3}python" setup.py sdist rotate --keep=1 --match .tar.gz \ && cd "$WORKSPACE" \ - && "${3}pip" install --no-cache-dir --quiet "$WORKSPACE/$1/dist"/*.tar.gz \ - && "${3}pip" install --no-cache-dir --quiet --no-deps --ignore-installed "$WORKSPACE/$1/dist"/*.tar.gz + && "${3}pip" install --no-cache-dir "$WORKSPACE/$1/dist"/*.tar.gz \ + && "${3}pip" install --no-cache-dir --no-deps --ignore-installed "$WORKSPACE/$1/dist"/*.tar.gz elif [[ "$2" != "" ]] then "install_$2" @@ -899,11 +891,11 @@ bundle_install_trylocal() { ( set -e echo "(Running bundle install --local. 'could not find package' messages are OK.)" - if ! bundle install --local --no-deployment; then + if ! "$bundle" install --local --no-deployment; then echo "(Running bundle install again, without --local.)" - bundle install --no-deployment + "$bundle" install --no-deployment fi - bundle package --all + "$bundle" package ) } @@ -950,8 +942,10 @@ install_services/login-sync() { install_services/api() { stop_services + check_arvados_config "services/api" cd "$WORKSPACE/services/api" \ - && RAILS_ENV=test bundle_install_trylocal + && RAILS_ENV=test bundle_install_trylocal \ + || return 1 rm -f config/environments/test.rb cp config/environments/test.rb.example config/environments/test.rb @@ -960,7 +954,7 @@ install_services/api() { # database, so that we can drop it. This assumes the current user # is a postgresql superuser. cd "$WORKSPACE/services/api" \ - && test_database=$(python -c "import yaml; print yaml.safe_load(file('$ARVADOS_CONFIG'))['Clusters']['zzzzz']['PostgreSQL']['Connection']['dbname']") \ + && test_database=$("${VENVDIR}/bin/python" -c "import yaml; print yaml.safe_load(file('$ARVADOS_CONFIG'))['Clusters']['zzzzz']['PostgreSQL']['Connection']['dbname']") \ && psql "$test_database" -c "SELECT pg_terminate_backend (pg_stat_activity.pid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = '$test_database';" 2>/dev/null mkdir -p "$WORKSPACE/services/api/tmp/pids" @@ -984,15 +978,16 @@ install_services/api() { && git --git-dir internal.git init \ || return 1 - - (cd "$WORKSPACE/services/api" - export RAILS_ENV=test - if bundle exec rails db:environment:set ; then - bundle exec rake db:drop - fi - bundle exec rake db:setup \ - && bundle exec rake db:fixtures:load - ) + ( + set -e + cd "$WORKSPACE/services/api" + export RAILS_ENV=test + if "$bundle" exec rails db:environment:set ; then + "$bundle" exec rake db:drop + fi + "$bundle" exec rake db:setup + "$bundle" exec rake db:fixtures:load + ) || return 1 } declare -a pythonstuff @@ -1000,7 +995,6 @@ pythonstuff=( sdk/pam sdk/python sdk/python:py3 - sdk/cwl sdk/cwl:py3 services/dockercleaner:py3 services/fuse @@ -1017,7 +1011,7 @@ install_apps/workbench() { cd "$WORKSPACE/apps/workbench" \ && mkdir -p tmp/cache \ && RAILS_ENV=test bundle_install_trylocal \ - && RAILS_ENV=test RAILS_GROUPS=assets bundle exec rake npm:install + && RAILS_ENV=test RAILS_GROUPS=assets "$bundle" exec rake npm:install } test_doc() { @@ -1027,7 +1021,7 @@ test_doc() { ARVADOS_API_HOST=qr1hi.arvadosapi.com # Make sure python-epydoc is installed or the next line won't # do much good! - PYTHONPATH=$WORKSPACE/sdk/python/ bundle exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=https://workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST + PYTHONPATH=$WORKSPACE/sdk/python/ "$bundle" exec rake linkchecker baseurl=file://$WORKSPACE/doc/.site/ arvados_workbench_host=https://workbench.$ARVADOS_API_HOST arvados_api_host=$ARVADOS_API_HOST ) } @@ -1037,36 +1031,15 @@ test_gofmt() { [[ -z "$(gofmt -e -d $dirs | tee -a /dev/stderr)" ]] } -test_govendor() { - ( - set -e - cd "$GOPATH/src/git.curoverse.com/arvados.git" - # Remove cached source dirs in workdir. Otherwise, they will - # not qualify as +missing or +external below, and we won't be - # able to detect that they're missing from vendor/vendor.json. - rm -rf vendor/*/ - go get -v -d ... - "$GOPATH/bin/govendor" sync - if [[ -n $("$GOPATH/bin/govendor" list +unused +missing +external | tee /dev/stderr) ]]; then - echo >&2 "vendor/vendor.json has unused or missing dependencies -- try: - -(export GOPATH=\"${GOPATH}\"; cd \$GOPATH/src/git.curoverse.com/arvados.git && \$GOPATH/bin/govendor add +missing +external && \$GOPATH/bin/govendor remove +unused) - -" - return 1 - fi - ) -} - test_services/api() { rm -f "$WORKSPACE/services/api/git-commit.version" cd "$WORKSPACE/services/api" \ - && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} bundle exec rake test TESTOPTS=\'-v -d\' ${testargs[services/api]} + && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} "$bundle" exec rake test TESTOPTS=\'-v -d\' ${testargs[services/api]} } test_sdk/ruby() { cd "$WORKSPACE/sdk/ruby" \ - && bundle exec rake test TESTOPTS=-v ${testargs[sdk/ruby]} + && "$bundle" exec rake test TESTOPTS=-v ${testargs[sdk/ruby]} } test_sdk/R() { @@ -1079,7 +1052,7 @@ test_sdk/R() { test_sdk/cli() { cd "$WORKSPACE/sdk/cli" \ && mkdir -p /tmp/keep \ - && KEEP_LOCAL_STORE=/tmp/keep bundle exec rake test TESTOPTS=-v ${testargs[sdk/cli]} + && KEEP_LOCAL_STORE=/tmp/keep "$bundle" exec rake test TESTOPTS=-v ${testargs[sdk/cli]} } test_sdk/java-v2() { @@ -1088,7 +1061,7 @@ test_sdk/java-v2() { test_services/login-sync() { cd "$WORKSPACE/services/login-sync" \ - && bundle exec rake test TESTOPTS=-v ${testargs[services/login-sync]} + && "$bundle" exec rake test TESTOPTS=-v ${testargs[services/login-sync]} } test_services/nodemanager_integration() { @@ -1099,31 +1072,31 @@ test_services/nodemanager_integration() { test_apps/workbench_units() { local TASK="test:units" cd "$WORKSPACE/apps/workbench" \ - && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} bundle exec rake ${TASK} TESTOPTS=\'-v -d\' ${testargs[apps/workbench]} ${testargs[apps/workbench_units]} + && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} "$bundle" exec rake ${TASK} TESTOPTS=\'-v -d\' ${testargs[apps/workbench]} ${testargs[apps/workbench_units]} } test_apps/workbench_functionals() { local TASK="test:functionals" cd "$WORKSPACE/apps/workbench" \ - && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} bundle exec rake ${TASK} TESTOPTS=\'-v -d\' ${testargs[apps/workbench]} ${testargs[apps/workbench_functionals]} + && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} "$bundle" exec rake ${TASK} TESTOPTS=\'-v -d\' ${testargs[apps/workbench]} ${testargs[apps/workbench_functionals]} } test_apps/workbench_integration() { local TASK="test:integration" cd "$WORKSPACE/apps/workbench" \ - && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} bundle exec rake ${TASK} TESTOPTS=\'-v -d\' ${testargs[apps/workbench]} ${testargs[apps/workbench_integration]} + && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} "$bundle" exec rake ${TASK} TESTOPTS=\'-v -d\' ${testargs[apps/workbench]} ${testargs[apps/workbench_integration]} } test_apps/workbench_benchmark() { local TASK="test:benchmark" cd "$WORKSPACE/apps/workbench" \ - && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} bundle exec rake ${TASK} ${testargs[apps/workbench_benchmark]} + && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} "$bundle" exec rake ${TASK} ${testargs[apps/workbench_benchmark]} } test_apps/workbench_profile() { local TASK="test:profile" cd "$WORKSPACE/apps/workbench" \ - && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} bundle exec rake ${TASK} ${testargs[apps/workbench_profile]} + && eval env RAILS_ENV=test ${short:+RAILS_TEST_SHORT=1} "$bundle" exec rake ${TASK} ${testargs[apps/workbench_profile]} } install_deps() { @@ -1133,6 +1106,7 @@ install_deps() { do_install sdk/cli do_install sdk/perl do_install sdk/python pip + do_install sdk/python pip "${VENV3DIR}/bin/" do_install sdk/ruby do_install services/api do_install services/arv-git-httpd go @@ -1183,7 +1157,6 @@ test_all() { fi do_test gofmt - do_test govendor do_test doc do_test sdk/ruby do_test sdk/R @@ -1299,7 +1272,8 @@ else ${verb}_${target} ;; *) - testargs["$target"]="${opts}" + argstarget=${target%:py3} + testargs["$argstarget"]="${opts}" tt="${testfuncargs[${target}]}" tt="${tt:-$target}" do_$verb $tt diff --git a/build/version-at-commit.sh b/build/version-at-commit.sh new file mode 100755 index 0000000000..89684cf2ab --- /dev/null +++ b/build/version-at-commit.sh @@ -0,0 +1,48 @@ +#!/bin/bash + +set -e -o pipefail +commit="$1" +versionglob="[0-9].[0-9]*.[0-9]*" +devsuffix=".dev" + +# automatically assign version +# +# handles the following cases: +# +# 1. commit is directly tagged. print that. +# +# 2. commit is on master or a development branch, the nearest tag is older +# than commit where this branch joins master. +# -> take greatest version tag in repo X.Y.Z and assign X.(Y+1).0 +# +# 3. commit is on a release branch, the nearest tag is newer +# than the commit where this branch joins master. +# -> take nearest tag X.Y.Z and assign X.Y.(Z+1) + +tagged=$(git tag --points-at "$commit") + +if [[ -n "$tagged" ]] ; then + echo $tagged +else + # 1. get the nearest tag with 'git describe' + # 2. get the merge base between this commit and master + # 3. if the tag is an ancestor of the merge base, + # (tag is older than merge base) increment minor version + # else, tag is newer than merge base, so increment point version + + nearest_tag=$(git describe --tags --abbrev=0 --match "$versionglob" "$commit") + merge_base=$(git merge-base origin/master "$commit") + + if git merge-base --is-ancestor "$nearest_tag" "$merge_base" ; then + # x.(y+1).0.devTIMESTAMP, where x.y.z is the newest version that does not contain $commit + # grep reads the list of tags (-f) that contain $commit and filters them out (-v) + # this prevents a newer tag from retroactively changing the versions of everything before it + v=$(git tag | grep -vFf <(git tag --contains "$commit") | sort -Vr | head -n1 | perl -pe 's/\.(\d+)\.\d+/".".($1+1).".0"/e') + else + # x.y.(z+1).devTIMESTAMP, where x.y.z is the latest released ancestor of $commit + v=$(echo $nearest_tag | perl -pe 's/(\d+)$/$1+1/e') + fi + isodate=$(TZ=UTC git log -n1 --format=%cd --date=iso "$commit") + ts=$(TZ=UTC date --date="$isodate" "+%Y%m%d%H%M%S") + echo "${v}${devsuffix}${ts}" +fi diff --git a/cmd/arvados-client/Makefile b/cmd/arvados-client/Makefile new file mode 100644 index 0000000000..b043fc90e2 --- /dev/null +++ b/cmd/arvados-client/Makefile @@ -0,0 +1,11 @@ +# Copyright (C) The Arvados Authors. All rights reserved. +# +# SPDX-License-Identifier: Apache-2.0 + +all: + @printf "*** note *** due to an xgo limitation, this only works when the working tree is in GOPATH\n\n" + go mod download + docker build --tag=cgofuse --build-arg=http_proxy="$(http_proxy)" --build-arg=https_proxy="$(https_proxy)" https://github.com/arvados/cgofuse.git + go run github.com/karalabe/xgo --image=cgofuse --targets=linux/amd64,linux/386,darwin/amd64,darwin/386,windows/amd64,windows/386 . + install arvados-* "$(GOPATH)"/bin/ + rm --interactive=never arvados-* diff --git a/cmd/arvados-client/cmd.go b/cmd/arvados-client/cmd.go index 308af5a01a..887bc62bb3 100644 --- a/cmd/arvados-client/cmd.go +++ b/cmd/arvados-client/cmd.go @@ -7,8 +7,9 @@ package main import ( "os" - "git.curoverse.com/arvados.git/lib/cli" - "git.curoverse.com/arvados.git/lib/cmd" + "git.arvados.org/arvados.git/lib/cli" + "git.arvados.org/arvados.git/lib/cmd" + "git.arvados.org/arvados.git/lib/mount" ) var ( @@ -50,6 +51,8 @@ var ( "user": cli.APICall, "virtual_machine": cli.APICall, "workflow": cli.APICall, + + "mount": mount.Command, }) ) diff --git a/cmd/arvados-server/arvados-controller.service b/cmd/arvados-server/arvados-controller.service index e85707473f..1a43d7cd3a 100644 --- a/cmd/arvados-server/arvados-controller.service +++ b/cmd/arvados-server/arvados-controller.service @@ -18,6 +18,8 @@ StartLimitIntervalSec=0 Type=notify EnvironmentFile=-/etc/arvados/environment ExecStart=/usr/bin/arvados-controller +# Set a reasonable default for the open file limit +LimitNOFILE=65536 Restart=always RestartSec=1 diff --git a/cmd/arvados-server/arvados-dispatch-cloud.service b/cmd/arvados-server/arvados-dispatch-cloud.service index aa5cc3b4a5..d3f476b7df 100644 --- a/cmd/arvados-server/arvados-dispatch-cloud.service +++ b/cmd/arvados-server/arvados-dispatch-cloud.service @@ -18,6 +18,8 @@ StartLimitIntervalSec=0 Type=notify EnvironmentFile=-/etc/arvados/environment ExecStart=/usr/bin/arvados-dispatch-cloud +# Set a reasonable default for the open file limit +LimitNOFILE=65536 Restart=always RestartSec=1 diff --git a/cmd/arvados-server/cmd.go b/cmd/arvados-server/cmd.go index 14cd63e46d..a9d927d873 100644 --- a/cmd/arvados-server/cmd.go +++ b/cmd/arvados-server/cmd.go @@ -7,11 +7,13 @@ package main import ( "os" - "git.curoverse.com/arvados.git/lib/cloud/cloudtest" - "git.curoverse.com/arvados.git/lib/cmd" - "git.curoverse.com/arvados.git/lib/config" - "git.curoverse.com/arvados.git/lib/controller" - "git.curoverse.com/arvados.git/lib/dispatchcloud" + "git.arvados.org/arvados.git/lib/boot" + "git.arvados.org/arvados.git/lib/cloud/cloudtest" + "git.arvados.org/arvados.git/lib/cmd" + "git.arvados.org/arvados.git/lib/config" + "git.arvados.org/arvados.git/lib/controller" + "git.arvados.org/arvados.git/lib/crunchrun" + "git.arvados.org/arvados.git/lib/dispatchcloud" ) var ( @@ -20,11 +22,13 @@ var ( "-version": cmd.Version, "--version": cmd.Version, + "boot": boot.Command, "cloudtest": cloudtest.Command, "config-check": config.CheckCommand, "config-dump": config.DumpCommand, "config-defaults": config.DumpDefaultsCommand, "controller": controller.Command, + "crunch-run": crunchrun.Command, "dispatch-cloud": dispatchcloud.Command, }) ) diff --git a/doc/Rakefile b/doc/Rakefile index f1aa3bfce8..623dbd033b 100644 --- a/doc/Rakefile +++ b/doc/Rakefile @@ -3,9 +3,21 @@ # # SPDX-License-Identifier: CC-BY-SA-3.0 +# As a convenience to the documentation writer, you can touch a file +# called 'no-sdk' in the 'doc' directory and it will suppress +# generating the documentation for the SDKs, which (the R docs +# especially) take a fair bit of time and slow down the edit-preview +# cycle. + require "rubygems" require "colorize" +module Zenweb + class Site + @binary_files = %w[png jpg gif eot svg ttf woff2? ico pdf m4a t?gz xlsx] + end +end + task :generate => [ :realclean, 'sdk/python/arvados/index.html', 'sdk/R/arvados/index.html', 'sdk/java-v2/javadoc/index.html' ] do vars = ['baseurl', 'arvados_cluster_uuid', 'arvados_api_host', 'arvados_workbench_host'] vars.each do |v| @@ -15,7 +27,14 @@ task :generate => [ :realclean, 'sdk/python/arvados/index.html', 'sdk/R/arvados/ end end +file ["install/new_cluster_checklist_Azure.xlsx", "install/new_cluster_checklist_AWS.xlsx"] do |t| + cp(t, t) +end + file "sdk/python/arvados/index.html" do |t| + if ENV['NO_SDK'] || File.exists?("no-sdk") + next + end `which epydoc` if $? == 0 STDERR.puts `epydoc --html --parse-only -o sdk/python/arvados ../sdk/python/arvados/ 2>&1` @@ -26,11 +45,16 @@ file "sdk/python/arvados/index.html" do |t| end file "sdk/R/arvados/index.html" do |t| + if ENV['NO_SDK'] || File.exists?("no-sdk") + next + end `which R` if $? == 0 tgt = Dir.pwd Dir.mkdir("sdk/R") Dir.mkdir("sdk/R/arvados") + puts("tgt", tgt) + cp('css/R.css', 'sdk/R/arvados') docfiles = [] Dir.chdir("../sdk/R/") do STDERR.puts `Rscript createDoc.R README.Rmd #{tgt}/sdk/R/README.md 2>&1` @@ -88,6 +112,9 @@ EOF end file "sdk/java-v2/javadoc/index.html" do |t| + if ENV['NO_SDK'] || File.exists?("no-sdk") + next + end `which java` if $? == 0 `which gradle` @@ -99,6 +126,8 @@ file "sdk/java-v2/javadoc/index.html" do |t| Dir.chdir("../sdk/java-v2") do STDERR.puts `gradle javadoc 2>&1` raise if $? != 0 + puts `sed -i "s/@import.*dejavu.css.*//g" build/docs/javadoc/stylesheet.css` + raise if $? != 0 end cp_r("../sdk/java-v2/build/docs/javadoc", "sdk/java-v2") raise if $? != 0 @@ -128,6 +157,7 @@ end require "zenweb/tasks" load "zenweb-textile.rb" load "zenweb-liquid.rb" +load "zenweb-fix-body.rb" task :extra_wirings do $website.pages["sdk/python/python.html.textile.liquid"].depends_on("sdk/python/arvados/index.html") diff --git a/doc/_config.yml b/doc/_config.yml index da7635c1f4..a8394300ea 100644 --- a/doc/_config.yml +++ b/doc/_config.yml @@ -88,9 +88,6 @@ navbar: - R: - sdk/R/index.html.md - sdk/R/arvados/index.html.textile.liquid - - Perl: - - sdk/perl/index.html.textile.liquid - - sdk/perl/example.html.textile.liquid - Ruby: - sdk/ruby/index.html.textile.liquid - sdk/ruby/example.html.textile.liquid @@ -101,6 +98,9 @@ navbar: - Java v1: - sdk/java/index.html.textile.liquid - sdk/java/example.html.textile.liquid + - Perl: + - sdk/perl/index.html.textile.liquid + - sdk/perl/example.html.textile.liquid api: - Concepts: - api/index.html.textile.liquid @@ -114,6 +114,7 @@ navbar: - api/methods/authorized_keys.html.textile.liquid - api/methods/groups.html.textile.liquid - api/methods/users.html.textile.liquid + - api/methods/user_agreements.html.textile.liquid - System resources: - api/methods/keep_services.html.textile.liquid - api/methods/links.html.textile.liquid @@ -128,6 +129,8 @@ navbar: - api/methods/container_requests.html.textile.liquid - api/methods/containers.html.textile.liquid - api/methods/workflows.html.textile.liquid + - Management (admin/system): + - api/dispatch.html.textile.liquid - Jobs engine (legacy): - api/crunch-scripts.html.textile.liquid - api/methods/jobs.html.textile.liquid @@ -148,34 +151,33 @@ navbar: admin: - Topics: - admin/index.html.textile.liquid - - Configuration: - - admin/config.html.textile.liquid - - admin/federation.html.textile.liquid - - Upgrading and migrations: - - admin/upgrading.html.textile.liquid - - admin/config-migration.html.textile.liquid - Users and Groups: - - install/cheat_sheet.html.textile.liquid - - admin/activation.html.textile.liquid + - admin/user-management.html.textile.liquid + - admin/reassign-ownership.html.textile.liquid + - admin/user-management-cli.html.textile.liquid + - admin/group-management.html.textile.liquid + - admin/federation.html.textile.liquid - admin/merge-remote-account.html.textile.liquid - admin/migrating-providers.html.textile.liquid - user/topics/arvados-sync-groups.html.textile.liquid + - admin/scoped-tokens.html.textile.liquid - Monitoring: - - admin/health-checks.html.textile.liquid + - admin/logging.html.textile.liquid - admin/metrics.html.textile.liquid + - admin/health-checks.html.textile.liquid - admin/management-token.html.textile.liquid - - Cloud: - - admin/storage-classes.html.textile.liquid - - admin/spot-instances.html.textile.liquid - - admin/cloudtest.html.textile.liquid - Data Management: - admin/collection-versioning.html.textile.liquid - admin/collection-managed-properties.html.textile.liquid - admin/keep-balance.html.textile.liquid - admin/controlling-container-reuse.html.textile.liquid - admin/logs-table-management.html.textile.liquid + - admin/workbench2-vocabulary.html.textile.liquid + - admin/storage-classes.html.textile.liquid + - Cloud: + - admin/spot-instances.html.textile.liquid + - admin/cloudtest.html.textile.liquid - Other: - - admin/troubleshooting.html.textile.liquid - install/migrate-docker19.html.textile.liquid - admin/upgrade-crunch2.html.textile.liquid installguide: @@ -187,11 +189,14 @@ navbar: - install/arvados-on-kubernetes.html.textile.liquid - Manual installation: - install/install-manual-prerequisites.html.textile.liquid - - install/install-components.html.textile.liquid + - install/packages.html.textile.liquid + - admin/upgrading.html.textile.liquid + - Configuration: + - install/config.html.textile.liquid + - admin/config-migration.html.textile.liquid + - admin/config.html.textile.liquid - Core: - - install/install-postgresql.html.textile.liquid - install/install-api-server.html.textile.liquid - - install/install-controller.html.textile.liquid - Keep: - install/install-keepstore.html.textile.liquid - install/configure-fs-storage.html.textile.liquid @@ -201,21 +206,24 @@ navbar: - install/install-keep-web.html.textile.liquid - install/install-keep-balance.html.textile.liquid - User interface: + - install/setup-login.html.textile.liquid - install/install-sso.html.textile.liquid - install/install-workbench-app.html.textile.liquid - install/install-workbench2-app.html.textile.liquid - install/install-composer.html.textile.liquid - Additional services: - install/install-ws.html.textile.liquid - - install/install-shell-server.html.textile.liquid - install/install-arv-git-httpd.html.textile.liquid - - Containers API support on SLURM: - - install/crunch2-slurm/install-prerequisites.html.textile.liquid - - install/crunch2-slurm/install-slurm.html.textile.liquid + - install/install-shell-server.html.textile.liquid + - Containers API: - install/crunch2-slurm/install-compute-node.html.textile.liquid + - install/install-jobs-image.html.textile.liquid + - install/install-dispatch-cloud.html.textile.liquid - install/crunch2-slurm/install-dispatch.html.textile.liquid - install/crunch2-slurm/install-test.html.textile.liquid - - install/install-nodemanager.html.textile.liquid - - install/install-compute-ping.html.textile.liquid - - Containers API support on cloud (beta): - - install/install-dispatch-cloud.html.textile.liquid + - External dependencies: + - install/install-postgresql.html.textile.liquid + - install/ruby.html.textile.liquid + - install/nginx.html.textile.liquid + - install/google-auth.html.textile.liquid + - install/install-docker.html.textile.liquid diff --git a/doc/_includes/_assign_volume_uuid.liquid b/doc/_includes/_assign_volume_uuid.liquid index cdd0f1a6db..97bc67c293 100644 --- a/doc/_includes/_assign_volume_uuid.liquid +++ b/doc/_includes/_assign_volume_uuid.liquid @@ -4,4 +4,6 @@ Copyright (C) The Arvados Authors. All rights reserved. SPDX-License-Identifier: CC-BY-SA-3.0 {% endcomment %} -Note that each volume has a UUID, like @zzzzz-nyw5e-0123456789abcde@. You assign these manually: replace @zzzzz@ with your cluster ID, and replace @0123456789abcde@ with an arbitrary string of 15 alphanumerics. Once assigned, UUIDs should not be changed. +Note that each volume has a UUID, like @zzzzz-nyw5e-0123456789abcde@. You assign these manually: replace @zzzzz@ with your Cluster ID, and replace @0123456789abcde@ with an arbitrary unique string of 15 alphanumerics. Once assigned, UUIDs should not be changed. + +Essential configuration values are highlighted in red. Remaining parameters are provided for documentation, with their default values. \ No newline at end of file diff --git a/doc/_includes/_example_sdk_go.liquid b/doc/_includes/_example_sdk_go.liquid index cc68b5a0b2..9c84ca0e2a 100644 --- a/doc/_includes/_example_sdk_go.liquid +++ b/doc/_includes/_example_sdk_go.liquid @@ -15,7 +15,7 @@ package main import ( "fmt" - "git.curoverse.com/arvados.git/sdk/go/arvadosclient" + "git.arvados.org/arvados.git/sdk/go/arvadosclient" "log" ) diff --git a/doc/_includes/_install_compute_docker.liquid b/doc/_includes/_install_compute_docker.liquid index 69b49e83cd..63c54aed72 100644 --- a/doc/_includes/_install_compute_docker.liquid +++ b/doc/_includes/_install_compute_docker.liquid @@ -4,76 +4,47 @@ Copyright (C) The Arvados Authors. All rights reserved. SPDX-License-Identifier: CC-BY-SA-3.0 {% endcomment %} -h2. Install Docker +h2(#cgroups). Configure Linux cgroups accounting -Compute nodes must have Docker installed to run containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the "Docker Engine installation documentation":https://docs.docker.com/ for your distribution. - -For Debian-based systems, the Arvados package repository includes a backported @docker.io@ package with a known-good version you can install. - -h2(#configure_docker_daemon). Configure the Docker daemon - -Crunch runs Docker containers with relatively little configuration. You may need to start the Docker daemon with specific options to make sure these jobs run smoothly in your environment. This section highlights options that are useful to most installations. Refer to the "Docker daemon reference":https://docs.docker.com/reference/commandline/daemon/ for complete information about all available options. - -The best way to configure these options varies by distribution. - -* If you're using our backported @docker.io@ package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker.io@. -* If you're using another Debian-based package, you can list these options in the @DOCKER_OPTS@ setting in @/etc/default/docker@. -* On Red Hat-based distributions, you can list these options in the @other_args@ setting in @/etc/sysconfig/docker@. - -h3. Default ulimits - -Docker containers inherit ulimits from the Docker daemon. However, the ulimits for a single Unix daemon may not accommodate a long-running Crunch job. You may want to increase default limits for compute containers by passing @--default-ulimit@ options to the Docker daemon. For example, to allow containers to open 10,000 files, set @--default-ulimit nofile=10000:10000@. +Linux can report what compute resources are used by processes in a specific cgroup or Docker container. Crunch can use these reports to share that information with users running compute work. This can help pipeline authors debug and optimize their workflows. -h3. DNS +To enable cgroups accounting, you must boot Linux with the command line parameters @cgroup_enable=memory swapaccount=1@. -Your containers must be able to resolve the hostname of your API server and any hostnames returned in Keep service records. If these names are not in public DNS records, you may need to specify a DNS resolver for the containers by setting the @--dns@ address to an IP address of an appropriate nameserver. You may specify this option more than once to use multiple nameservers. +After making changes, reboot the system to make these changes effective. -h2. Configure Linux cgroups accounting +h3. Red Hat and CentOS -Linux can report what compute resources are used by processes in a specific cgroup or Docker container. Crunch can use these reports to share that information with users running compute work. This can help pipeline authors debug and optimize their workflows. + +
~$ sudo grubby --update-kernel=ALL --args='cgroup_enable=memory swapaccount=1'
+
+ -To enable cgroups accounting, you must boot Linux with the command line parameters @cgroup_enable=memory swapaccount=1@. +h3. Debian and Ubuntu -On Debian-based systems, open the file @/etc/default/grub@ in an editor. Find where the string @GRUB_CMDLINE_LINUX@ is set. Add @cgroup_enable=memory swapaccount=1@ to that string. Save the file and exit the editor. Then run: +Open the file @/etc/default/grub@ in an editor. Find where the string @GRUB_CMDLINE_LINUX@ is set. Add @cgroup_enable=memory swapaccount=1@ to that string. Save the file and exit the editor. Then run: 
~$ sudo update-grub
-On Red Hat-based systems, run: +h2(#install_docker). Install Docker + +Compute nodes must have Docker installed to run containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the "Docker Engine installation documentation":https://docs.docker.com/install/ for your distribution. + +Make sure Docker is enabled to start on boot: -
~$ sudo grubby --update-kernel=ALL --args='cgroup_enable=memory swapaccount=1'
+
# systemctl enable --now docker
 

 
 
-Finally, reboot the system to make these changes effective.
-
-h2. Create a project for Docker images
+h2(#configure_docker_daemon). Configure the Docker daemon
 
-Here we create a default project for the standard Arvados Docker images, and give all users read access to it. The project is owned by the system user.
+Depending on your anticipated workload or cluster configuration, you may need to tweak Docker options.
 
-
-
~$ uuid_prefix=`arv --format=uuid user current | cut -d- -f1`
-~$ project_uuid=`arv --format=uuid group create --group "{\"owner_uuid\":\"$uuid_prefix-tpzed-000000000000000\", \"group_class\":\"project\", \"name\":\"Arvados Standard Docker Images\"}"`
-~$ echo "Arvados project uuid is '$project_uuid'"
-~$ read -rd $'\000' newlink <<EOF; arv link create --link "$newlink"
-{
- "tail_uuid":"$all_users_group_uuid",
- "head_uuid":"$project_uuid",
- "link_class":"permission",
- "name":"can_read"
-}
-EOF
-

-
-h2. Download and tag the latest arvados/jobs docker image
-
-In order to start workflows from workbench, there needs to be Docker image tagged @arvados/jobs:latest@. The following command downloads the latest arvados/jobs image from Docker Hub, loads it into Keep, and tags it as 'latest'.  In this example @$project_uuid@ should be the UUID of the "Arvados Standard Docker Images" project.
+For information about how to set configuration options for the Docker daemon, see https://docs.docker.com/config/daemon/systemd/
 
-
-
~$ arv-keepdocker --pull arvados/jobs latest --project-uuid $project_uuid
-

+h3. Changing ulimits
 
-If the image needs to be downloaded from Docker Hub, the command can take a few minutes to complete, depending on available network bandwidth.
+Docker containers inherit ulimits from the Docker daemon.  However, the ulimits for a single Unix daemon may not accommodate a long-running Crunch job.  You may want to increase default limits for compute containers by passing @--default-ulimit@ options to the Docker daemon.  For example, to allow containers to open 10,000 files, set @--default-ulimit nofile=10000:10000@.
diff --git a/doc/_includes/_install_compute_fuse.liquid b/doc/_includes/_install_compute_fuse.liquid
index 449c32c266..95679f3fa1 100644
--- a/doc/_includes/_install_compute_fuse.liquid
+++ b/doc/_includes/_install_compute_fuse.liquid
@@ -4,20 +4,14 @@ Copyright (C) The Arvados Authors. All rights reserved.
 SPDX-License-Identifier: CC-BY-SA-3.0
 {% endcomment %}
 
-h2. Configure FUSE
+h2(#fuse). Update fuse.conf
 
 FUSE must be configured with the @user_allow_other@ option enabled for Crunch to set up Keep mounts that are readable by containers.  Install this file as @/etc/fuse.conf@:
 
 
 
-# Set the maximum number of FUSE mounts allowed to non-root users.
-# The default is 1000.
-#
-#mount_max = 1000
-
 # Allow non-root users to specify the 'allow_other' or 'allow_root'
 # mount options.
-#
 user_allow_other
 

 
diff --git a/doc/_includes/_install_debian_key.liquid b/doc/_includes/_install_debian_key.liquid
index 75942c7bcb..e74bba21de 100644
--- a/doc/_includes/_install_debian_key.liquid
+++ b/doc/_includes/_install_debian_key.liquid
@@ -4,7 +4,10 @@ Copyright (C) The Arvados Authors. All rights reserved.
 SPDX-License-Identifier: CC-BY-SA-3.0
 {% endcomment %}
 
+
+
 
-
~$ sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
+
# apt-get --no-install-recommends install gnupg
+# /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
 

 
diff --git a/doc/_includes/_install_docker_cleaner.liquid b/doc/_includes/_install_docker_cleaner.liquid
index 5b0e1551ed..f8e9e049df 100644
--- a/doc/_includes/_install_docker_cleaner.liquid
+++ b/doc/_includes/_install_docker_cleaner.liquid
@@ -4,13 +4,9 @@ Copyright (C) The Arvados Authors. All rights reserved.
 SPDX-License-Identifier: CC-BY-SA-3.0
 {% endcomment %}
 
-h2. Configure the Docker cleaner
+h2(#docker-cleaner). Update docker-cleaner.json
 
-The arvados-docker-cleaner program removes least recently used Docker images as needed to keep disk usage below a configured limit.
-
-{% include 'notebox_begin' %}
-This also removes all containers as soon as they exit, as if they were run with @docker run --rm@. If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or configure it with @"RemoveStoppedContainers":"never"@.
-{% include 'notebox_end' %}
+The @arvados-docker-cleaner@ program removes least recently used Docker images as needed to keep disk usage below a configured limit.
 
 Create a file @/etc/arvados/docker-cleaner/docker-cleaner.json@ in an editor, with the following contents.
 
@@ -24,11 +20,6 @@ Create a file @/etc/arvados/docker-cleaner/docker-cleaner.json@ in an editor, wi
 
 *Choosing a quota:* Most deployments will want a quota that's at least 10G.  From there, a larger quota can help reduce compute overhead by preventing reloading the same Docker image repeatedly, but will leave less space for other files on the same storage (usually Docker volumes).  Make sure the quota is less than the total space available for Docker images.
 
-Restart the service after updating the configuration file.
-
-
-
~$ sudo systemctl restart arvados-docker-cleaner
-

-
-
-*If you are using a different daemon supervisor,* or if you want to test the daemon in a terminal window, run @arvados-docker-cleaner@. Run @arvados-docker-cleaner --help@ for more configuration options.
+{% include 'notebox_begin' %}
+This also removes all containers as soon as they exit, as if they were run with @docker run --rm@. If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or configure it with @"RemoveStoppedContainers":"never"@.
+{% include 'notebox_end' %}
diff --git a/doc/_includes/_install_git_curl.liquid b/doc/_includes/_install_git_curl.liquid
deleted file mode 100644
index 40b95d314d..0000000000
--- a/doc/_includes/_install_git_curl.liquid
+++ /dev/null
@@ -1,19 +0,0 @@
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-On a Debian-based system, install the following packages:
-
-
-
~$ sudo apt-get install git curl
-

-
-
-On a Red Hat-based system, install the following packages:
-
-
-
~$ sudo yum install git curl
-

-
diff --git a/doc/_includes/_install_packages.liquid b/doc/_includes/_install_packages.liquid
new file mode 100644
index 0000000000..bfac32d834
--- /dev/null
+++ b/doc/_includes/_install_packages.liquid
@@ -0,0 +1,24 @@
+{% comment %}
+packages_to_install should be a list
+fallback on arvados_component if not defined
+{% endcomment %}
+
+{% if package_to_install == nil %}
+  {% assign packages_to_install = arvados_component | split: " " %}
+{% endif %}
+
+h2(#install-packages). Install {{packages_to_install | join: " and " }}
+
+h3. Red Hat and Centos
+
+
+
# yum install {{packages_to_install | join: " "}}
+

+
+
+h3. Debian and Ubuntu
+
+
+
# apt-get install {{packages_to_install  | join " "}}
+

+
diff --git a/doc/_includes/_install_postgres_database.liquid b/doc/_includes/_install_postgres_database.liquid
index f4b95ac0be..e93457f188 100644
--- a/doc/_includes/_install_postgres_database.liquid
+++ b/doc/_includes/_install_postgres_database.liquid
@@ -4,29 +4,29 @@ Copyright (C) The Arvados Authors. All rights reserved.
 SPDX-License-Identifier: CC-BY-SA-3.0
 {% endcomment %}
 
-
    
+
      
 
    1. Start a shell for the postgres user:
-
      ~$ sudo -u postgres bash
      
+
      # su postgres
      
 
      2. 
 
    2. Generate a new database password:
-
      $ ruby -e 'puts rand(2**128).to_s(36)'
+
      postgres$ tr -dc 0-9a-zA-Z </dev/urandom | head -c25; echo
 yourgeneratedpassword
 
       Record this.  You'll need it when you set up the Rails server later.
 
      3. 
 
    3. Create a database user with the password you generated:
-  
      $ createuser --encrypted -R -S --pwprompt {{service_role}}
+  
      postgres$ createuser --encrypted --no-createrole --no-superuser --pwprompt {{service_role}}
   Enter password for new role: yourgeneratedpassword
   Enter it again: yourgeneratedpassword
      
 
      4. 
 
    4. Create a database owned by the new user:
-  
      $ createdb {{service_database}} -T template0 -E UTF8 -O {{service_role}}
      
+  
      postgres$ createdb {{service_database}} -T template0 -E UTF8 -O {{service_role}}
      
 
      5. 
 {% if use_contrib %}
 
    5. Enable the pg_trgm extension
-  
      $ psql {{service_database}} -c "CREATE EXTENSION IF NOT EXISTS pg_trgm"
      
+  
      postgres$ psql {{service_database}} -c "CREATE EXTENSION IF NOT EXISTS pg_trgm"
      
 
      6. 
 {% endif %}
 
    6. Exit the postgres user shell:
-  
      $ exit
      
+  
      postgres$ exit
      
 
      7. 
 
    
diff --git a/doc/_includes/_install_rails_command.liquid b/doc/_includes/_install_rails_command.liquid
index 027f64bebb..10c17a0c2d 100644
--- a/doc/_includes/_install_rails_command.liquid
+++ b/doc/_includes/_install_rails_command.liquid
@@ -28,17 +28,6 @@ This template recognizes four variables:
   {% assign railscmd = "bundle exec rails console" %}
 {% endunless %}
 
-Using RVM:
-
-
-
    {{railshost}}~$ cd {{railsdir}}
-{{railshost}}{{railsdir}}$ sudo -u webserver-user RAILS_ENV=production `which rvm-exec` default {{railscmd}}
-{% if railsout %}{{railsout}}
-{% endif %}
    
-    
-
-Not using RVM:
-
 
 
    {{railshost}}~$ cd {{railsdir}}
 {{railshost}}{{railsdir}}$ sudo -u webserver-user RAILS_ENV=production {{railscmd}}
diff --git a/doc/_includes/_install_redhat_key.liquid b/doc/_includes/_install_redhat_key.liquid
index 69cfd5a013..c9430515f1 100644
--- a/doc/_includes/_install_redhat_key.liquid
+++ b/doc/_includes/_install_redhat_key.liquid
@@ -4,12 +4,13 @@ Copyright (C) The Arvados Authors. All rights reserved.
 SPDX-License-Identifier: CC-BY-SA-3.0
 {% endcomment %}
 
-The Curoverse signing key fingerprint is
+The Arvados signing key fingerprint is
 
 
-
    
-pub  2048R/1078ECD7 2010-11-15 Curoverse, Inc Automatic Signing Key 
-      Key fingerprint = B2DA 2991 656E B4A5 0314  CA2B 5716 5911 1078 ECD7
-sub  2048R/5A8C5A93 2010-11-15
+
    pub   rsa2048 2010-11-15 [SC]
+      B2DA 2991 656E B4A5 0314  CA2B 5716 5911 1078 ECD7
+uid           [ unknown] Arvados Automatic Signing Key 
+uid           [ unknown] Curoverse, Inc Automatic Signing Key 
+sub   rsa2048 2010-11-15 [E]
 
    
 
diff --git a/doc/_includes/_install_ruby_and_bundler.liquid b/doc/_includes/_install_ruby_and_bundler.liquid
index a8323f592d..d14e555f89 100644
--- a/doc/_includes/_install_ruby_and_bundler.liquid
+++ b/doc/_includes/_install_ruby_and_bundler.liquid
@@ -4,28 +4,72 @@ Copyright (C) The Arvados Authors. All rights reserved.
 SPDX-License-Identifier: CC-BY-SA-3.0
 {% endcomment %}
 
-Ruby 2.5 is recommended; Ruby 2.3 is also known to work.
+Minimum of Ruby 2.3 is required.  Ruby 2.5 is recommended.
 
-h4(#rvm). *Option 1: Install with RVM*
+* "Option 1: Install from packages":#packages
+* "Option 2: Install with RVM":#rvm
+* "Option 3: Install from source":#fromsource
+
+h2(#packages). Option 1: Install from packages
+
+{% include 'notebox_begin' %}
+Future versions of Arvados may require a newer version of Ruby than is packaged with your OS.  Using OS packages simplifies initial install, but may complicate upgrades that rely on a newer Ruby.  If this is a concern, we recommend using "RVM.":#rvm
+{% include 'notebox_end' %}
+
+h3. Centos 7
+
+The Ruby version shipped with Centos 7 is too old.  Use "RVM.":#rvm
+
+h3. Debian and Ubuntu
+
+Debian 9 (stretch) and Ubuntu 16.04 (xenial) ship Ruby 2.3, which is sufficient to run Arvados.  Later releases have newer versions of Ruby that can also run Arvados.
+
+
+
    # apt-get --no-install-recommends install ruby ruby-dev bundler
    
+    
+
+h2(#rvm). Option 2: Install with RVM
+
+h3. Install gpg and curl
+
+h4. Centos 7
+
+
    +yum install gpg curl which
+
    
+
+h4. Debian and Ubuntu
+
+
    +apt-get --no-install-recommends install gpg curl
+
    
+
+h3. Install RVM
+
+
+
    # gpg --keyserver pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
+\curl -sSL https://get.rvm.io | bash -s stable --ruby=2.5
+
    
+
+To use Ruby installed from RVM, load it in an open shell like this:
 
 
-
    sudo gpg --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
-\curl -sSL https://get.rvm.io | sudo bash -s stable --ruby=2.5
+
    . /usr/local/rvm/scripts/rvm
 
    
 
-Either log out and log back in to activate RVM, or explicitly load it in all open shells like this:
+Alternately you can use @rvm-exec@ (the first parameter is the ruby version to use, or "default"), for example:
 
 
-
    source /usr/local/rvm/scripts/rvm
+
    rvm-exec default rails console
 
    
 
-Once RVM is activated in your shell, install Bundler:
+Finally, install Bundler:
 
 
 
    ~$ gem install bundler
 
    
 
-h4(#fromsource). *Option 2: Install from source*
+h2(#fromsource). Option 3: Install from source
 
 Install prerequisites for Debian 8:
 
diff --git a/doc/_includes/_mount_types.liquid b/doc/_includes/_mount_types.liquid
index de417e1488..86e05be866 100644
--- a/doc/_includes/_mount_types.liquid
+++ b/doc/_includes/_mount_types.liquid
@@ -34,7 +34,7 @@ At container startup, the target path will have the source tree indicated by the
 }
 
    |
 |Temporary directory|@tmp@|@"capacity"@: capacity (in bytes) of the storage device.
-@"device_type"@ (optional, default "network"): one of @{"ram", "ssd", "disk", "network"}@ indicating the acceptable level of performance.
+@"device_type"@ (optional, default "network"): one of @{"ram", "ssd", "disk", "network"}@ indicating the acceptable level of performance. (*note: not yet implemented as of v1.5*)
 At container startup, the target path will be empty. When the container finishes, the content will be discarded. This will be backed by a storage mechanism no slower than the specified type.|
    {
  "kind":"tmp",
  "capacity":100000000000
diff --git a/doc/_includes/_navbar_left.liquid b/doc/_includes/_navbar_left.liquid
index cba6c46e4d..2e18980f23 100644
--- a/doc/_includes/_navbar_left.liquid
+++ b/doc/_includes/_navbar_left.liquid
@@ -4,6 +4,32 @@ Copyright (C) The Arvados Authors. All rights reserved.
 SPDX-License-Identifier: CC-BY-SA-3.0
 {% endcomment %}
 
+{% assign highlighturl = "" %}
+{% for section in site.navbar[page.navsection] %}
+  {% for entry in section %}
+    {% comment %}
+    Want to highlight the current page on the left nav.
+    But some pages have been renamed with a symlink from the old page to the new one.
+    Then the URL won't match.
+    So if the URL doesn't match, as a fallback look for a page with a matching title.
+    {% endcomment %}
+
+    {% for item in entry[1] %}
+      {% if site.pages[item].url == page.url %}
+        {% assign highlighturl = site.pages[item].url %}
+      {% endif %}
+    {% endfor %}
+
+    {% if highlighturl == "" %}
+      {% for item in entry[1] %}
+        {% if site.pages[item].title == page.title %}
+          {% assign highlighturl = site.pages[item].url %}
+        {% endif %}
+      {% endfor %}
+    {% endif %}
+  {% endfor %}
+{% endfor %}
+
 
    
   
    
     
      
@@ -11,10 +37,10 @@ SPDX-License-Identifier: CC-BY-SA-3.0
       {% for entry in section %}
       
    1. {{ entry[0] }}
 	
        
-          {% for item in entry[1] %}        
-          {% assign p = site.pages[item] %}
-          
      1. 
-            {{ p.title }}
        2. 
+          {% for item in entry[1] %}
+            {% assign p = site.pages[item] %}
+            
      2. 
+              {{ p.title }}
        3. 
           {% endfor %}
         
      
         {% endfor %}
diff --git a/doc/_includes/_navbar_top.liquid b/doc/_includes/_navbar_top.liquid
index 39c70471a3..38c8e0a1db 100644
--- a/doc/_includes/_navbar_top.liquid
+++ b/doc/_includes/_navbar_top.liquid
@@ -36,7 +36,7 @@ SPDX-License-Identifier: CC-BY-SA-3.0
         
    2. arvados.org »
      3. 
       
 
-      
      
+      
      
         
      
           
      
             
@@ -48,5 +48,24 @@ SPDX-License-Identifier: CC-BY-SA-3.0
         
       
      
     
      
+
+   
+
+   
+
   
      
 
    
diff --git a/doc/_includes/_restart_api.liquid b/doc/_includes/_restart_api.liquid
new file mode 100644
index 0000000000..c3e0330b87
--- /dev/null
+++ b/doc/_includes/_restart_api.liquid
@@ -0,0 +1,8 @@
+h2(#restart-api). Restart the API server and controller
+
+*Make sure the cluster config file is up to date on the API server host* then restart the API server and controller processes to ensure the configuration changes are visible to the whole cluster.
+
+
+
    # systemctl restart nginx arvados-controller
+
    
+    
diff --git a/doc/_includes/_start_service.liquid b/doc/_includes/_start_service.liquid
new file mode 100644
index 0000000000..27c42c94c9
--- /dev/null
+++ b/doc/_includes/_start_service.liquid
@@ -0,0 +1,15 @@
+h2(#start-service). Start the service
+
+
+
    # systemctl enable --now {{arvados_component}}
+# systemctl status {{arvados_component}}
+[...]
+
    
+    
+
+If @systemctl status@ indicates it is not running, use @journalctl@ to check logs for errors:
+
+
+
    # journalctl -n12 --unit {{arvados_component}}
+
    
+    
diff --git a/doc/_includes/_vocabulary_migrate_py.liquid b/doc/_includes/_vocabulary_migrate_py.liquid
new file mode 120000
index 0000000000..9cd36294fc
--- /dev/null
+++ b/doc/_includes/_vocabulary_migrate_py.liquid
@@ -0,0 +1 @@
+../../tools/vocabulary-migrate/vocabulary-migrate.py
\ No newline at end of file
diff --git a/doc/_includes/_wb2_vocabulary_example.liquid b/doc/_includes/_wb2_vocabulary_example.liquid
new file mode 100644
index 0000000000..ee2ac97ef3
--- /dev/null
+++ b/doc/_includes/_wb2_vocabulary_example.liquid
@@ -0,0 +1,27 @@
+{
+    "strict_tags": false,
+    "tags": {
+        "IDTAGANIMALS": {
+            "strict": false,
+            "labels": [{"label": "Animal" }, {"label": "Creature"}, {"label": "Species"}],
+            "values": {
+                "IDVALANIMALS1": { "labels": [{"label": "Human"}, {"label": "Homo sapiens"}] },
+                "IDVALANIMALS2": { "labels": [{"label": "Dog"}, {"label": "Canis lupus familiaris"}] },
+                "IDVALANIMALS3": { "labels": [{"label": "Elephant"}, {"label": "Loxodonta"}] },
+                "IDVALANIMALS4": { "labels": [{"label": "Eagle"}, {"label": "Haliaeetus leucocephalus"}] }
+            }
+        },
+        "IDTAGCOMMENT": {
+            "labels": [{"label": "Comment"}, {"label": "Suggestion"}]
+        },
+        "IDTAGIMPORTANCES": {
+            "strict": true,
+            "labels": [{"label": "Importance"}, {"label": "Priority"}],
+            "values": {
+                "IDVALIMPORTANCES1": { "labels": [{"label": "Critical"}, {"label": "Urgent"}, {"label": "High"}] },
+                "IDVALIMPORTANCES2": { "labels": [{"label": "Normal"}, {"label": "Moderate"}] },
+                "IDVALIMPORTANCES3": { "labels": [{"label": "Low"}] }
+            }
+        }
+    }
+}
\ No newline at end of file
diff --git a/doc/_layouts/default.html.liquid b/doc/_layouts/default.html.liquid
index 7c6d36ec46..2ce354f060 100644
--- a/doc/_layouts/default.html.liquid
+++ b/doc/_layouts/default.html.liquid
@@ -22,6 +22,9 @@ SPDX-License-Identifier: CC-BY-SA-3.0
     
     
     
+    
+    
+    
     
 
     
@@ -81,7 +86,8 @@ SPDX-License-Identifier: CC-BY-SA-3.0
     {{ content }}
     {% else %}
 
-    
    
+    
    
+
       
    
         {% include 'navbar_left' %}
         
    
@@ -95,8 +101,6 @@ SPDX-License-Identifier: CC-BY-SA-3.0
 
     
    
     {% endif %}
-    
-