doc/install/*.xlsx
sdk/cwl/tests/wf/hello.txt
sdk/cwl/tests/wf/indir1/hello2.txt
-sdk/cwl/tests/chipseq/data/Genomes/*
\ No newline at end of file
+sdk/cwl/tests/chipseq/data/Genomes/*
+CITATION.cff
Thomas Mooney <tmooney@genome.wustl.edu>
Chen Chen <aflyhorse@gmail.com>
Veritas Genetics, Inc. <*@veritasgenetics.com>
-Curii Corporation, Inc. <*@curii.com>
+Curii Corporation <*@curii.com>
Dante Tsang <dante@dantetsang.com>
Codex Genetics Ltd <info@codexgenetics.com>
Bruno P. Kinoshita <brunodepaulak@yahoo.com.br>
--- /dev/null
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+authors:
+- name: "The Arvados Authors"
+- family-names: "Amstutz"
+ given-names: "Peter"
+ orcid: "https://orcid.org/0000-0003-3566-7705"
+- family-names: "Bértoli"
+ given-names: "Javier"
+ family-names: "César"
+ given-names: "Nico"
+- family-names: "Clegg"
+ given-names: "Tom"
+ orcid: "https://orcid.org/0000-0001-6751-2930"
+- family-names: "Di Pentima"
+ given-names: "Lucas"
+ orcid: "https://orcid.org/0000-0002-2807-6854"
+- family-names: "Kutyła"
+ given-names: "Daniel"
+- family-names: "Li"
+ given-names: "Jiayong"
+- family-names: "Smith"
+ given-names: "Stephen"
+- family-names: "Vandewege"
+ given-names: "Ward"
+ orcid: "https://orcid.org/0000-0002-2527-6949"
+- family-names: "Wait Zaranek"
+ given-names: "Alexander"
+ orcid: "https://orcid.org/0000-0002-0415-9655"
+- family-names: "Wait Zaranek"
+ given-names: "Sarah"
+ orcid: "https://orcid.org/0000-0003-4716-9121"
+title: "Arvados"
+abstract: "Arvados is an open source platform for managing, processing, and sharing genomic and other large scientific and biomedical data."
+type: software
+url: "https://github.com/arvados/arvados/"
+doi: 10.5281/zenodo.6382942
h2. Configuration
-Add entries to @InstanceTypes@ that have @Preemptible: true@. Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The @Price@ for preemptible instances is the maximum bid price, the actual price paid is dynamic and will likely be lower. For example:
+First, configure some @InstanceTypes@ that have @Preemptible: true@. For a preemptible instance, @Price@ determines the maximum bid price; the actual price paid is dynamic and will likely be lower.
+
+Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. To do this automatically, use @PreemptiblePriceFactor@ to enable a preemptible version of each listed type, using the given factor to set the maximum bid price relative to the non-preemptible price. Alternatively, you can configure preemptible instance types explicitly. For example, the following two configurations are equivalent:
<pre>
Clusters:
ClusterID:
+ Containers:
+ PreemptiblePriceFactor: 0.8
InstanceTypes:
m4.large:
- Preemptible: false
ProviderType: m4.large
VCPUs: 2
RAM: 8GiB
AddedScratch: 32GB
Price: 0.1
- m4.large.spot:
- Preemptible: true
+</pre>
+
+<pre>
+Clusters:
+ ClusterID:
+ InstanceTypes:
+ m4.large:
ProviderType: m4.large
VCPUs: 2
RAM: 8GiB
AddedScratch: 32GB
Price: 0.1
+ m4.large.preemptible:
+ Preemptible: true
+ ProviderType: m4.large
+ VCPUs: 2
+ RAM: 8GiB
+ AddedScratch: 32GB
+ Price: 0.08
</pre>
Next, you can choose to enable automatic use of preemptible instances:
<pre>
+Clusters:
+ ClusterID:
Containers:
AlwaysUsePreemptibleInstances: true
</pre>
<div class="releasenotes">
</notextile>
-h2(#main). development main (as of 2022-03-08)
+h2(#main). development main (as of 2022-03-??)
-"previous: Upgrading to 2.3.0":#v2_3_0
+h2(#v2_4_0). v2.4.0 (2022-03-??)
+
+"previous: Upgrading to 2.3.1":#v2_3_1
h3. Ubuntu 18.04 Arvados Python packages now depend on python-3.8
The anonymous token configured in @Users.AnonymousUserToken@ must now be 32 characters or longer. This was already the suggestion in the documentation, now it is enforced. The @script/get_anonymous_user_token.rb@ script that was needed to register the anonymous user token in the database has been removed. Registration of the anonymous token is no longer necessary. If the anonymous token in @config.yml@ is specified as a full V2 token, that will now generate a warning - it should be updated to list just the secret (i.e. the part after the last forward slash).
-h3. Preemptible instance types are used automatically, if any are configured
+h3. Preemptible instance support changes
-The default behavior for selecting "preemptible instances":{{site.baseurl}}/admin/spot-instances.html has changed. If your configuration lists any instance types with @Preemptible: true@, all child (non-top-level) containers will automatically be scheduled on preemptible instances. To avoid using preemptible instances except when explicitly requested by clients, add @AlwaysUsePreemptibleInstances: false@ in the @Containers@ config section. (Previously, preemptible instance types were never used unless the configuration specified @UsePreemptibleInstances: true@. That flag has been removed.)
+The @Containers.UsePreemptibleInstances@ option has been renamed to @Containers.AlwaysUsePreemptibleInstances@ and has the same behavior when @true@ and one or more preemptible instances are configured. However, a value of @false@ no longer disables support for preemptible instances, instead users can now enable use of preemptible instances at the level of an individual workflow or workflow step.
-h3. Role groups are visible to all users by default
-
-The permission model has changed such that all role groups are visible to all active users. This enables users to share objects with groups they don't belong to. To preserve the previous behavior, where role groups are only visible to members and admins, add @RoleGroupsVisibleToAll: false@ to the @Users@ section of your configuration file.
+In addition, there is a new configuration option @Containers.PreemptiblePriceFactor@ will automatically add a preemptible instance type corresponding to each regular instance type. See "Using Preemptible instances":spot-instances.html for details.
h3. Default LSF arguments have changed
If you use LSF and your configuration specifies @Containers.LSF.BsubArgumentsList@, you should update it to include the new arguments (@"-R", "select[mem>=%MMB]", ...@, see "configuration reference":{{site.baseurl}}/admin/config.html). Otherwise, containers that are too big to run on any LSF host will remain in the LSF queue instead of being cancelled.
-h3. Previously trashed role groups will be deleted
+h3. Support for NVIDIA CUDA GPUs
-Due to a bug in previous versions, the @DELETE@ operation on a role group caused the group to be flagged as trash in the database, but continue to grant permissions regardless. After upgrading, any role groups that had been trashed this way will be deleted. This might surprise some users if they were relying on permissions that were still in effect due to this bug. Future @DELETE@ operations on a role group will immediately delete the group and revoke the associated permissions.
+Arvados now supports requesting NVIDIA CUDA GPUs for cloud and LSF (Slurm is currently not supported). To be able to request GPU nodes, some additional configuration is needed:
-h3. Users are visible to other users by default
+"Including GPU support in cloud compute node image":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html#nvidia
-When a new user is set up (either via @AutoSetupNewUsers@ config or via Workbench admin interface) the user immediately becomes visible to other users. To revert to the previous behavior, where the administrator must add two users to the same group using the Workbench admin interface in order for the users to see each other, change the new @Users.ActivatedUsersAreVisibleToOthers@ config to @false@.
+"Configure cloud dispatcher for GPU support":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html#GPUsupport
+
+"LSF GPU configuration":{{site.baseurl}}/install/crunch2-lsf/install-dispatch.html
+
+h3. Role groups are visible to all users by default
+
+The permission model has changed such that all role groups are visible to all active users. This enables users to share objects with groups they don't belong to. To preserve the previous behavior, where role groups are only visible to members and admins, add @RoleGroupsVisibleToAll: false@ to the @Users@ section of your configuration file.
+
+h3. Previously trashed role groups will be deleted
+
+Due to a bug in previous versions, the @DELETE@ operation on a role group caused the group to be flagged as trash in the database, but continue to grant permissions regardless. After upgrading, any role groups that had been trashed this way will be deleted. This might surprise some users if they were relying on permissions that were still in effect due to this bug. Future @DELETE@ operations on a role group will immediately delete the group and revoke the associated permissions.
h3. Dedicated keepstore process for each container
* If you already have a robust permanent keepstore infrastructure, you can set @Containers.LocalKeepBlobBuffersPerVCPU@ to 0 to disable this feature and preserve the previous behavior of sending container I/O traffic to your separately provisioned keepstore servers.
* This feature is enabled only if no volumes use @AccessViaHosts@, and no volumes have underlying @Replication@ less than @Collections.DefaultReplication@. If the feature is configured but cannot be enabled due to an incompatible volume configuration, this will be noted in the @crunch-run.txt@ file in the container log.
+h2(#v2_3_1). v2.3.1 (2021-11-24)
+
+"previous: Upgrading to 2.3.0":#v2_3_0
+
+h3. Users are visible to other users by default
+
+When a new user is set up (either via @AutoSetupNewUsers@ config or via Workbench admin interface) the user immediately becomes visible to other users. To revert to the previous behavior, where the administrator must add two users to the same group using the Workbench admin interface in order for the users to see each other, change the new @Users.ActivatedUsersAreVisibleToOthers@ config to @false@.
+
h3. Backend support for vocabulary checking
If your installation uses the vocabulary feature on Workbench2, you will need to update the cluster configuration by moving the vocabulary definition file to the node where @controller@ runs, and set the @API.VocabularyPath@ configuration parameter to the local path where the file was placed.
</code></pre>
</notextile>
-h4. NVIDIA GPU support
+h4(#GPUsupport). NVIDIA GPU support
To specify instance types with NVIDIA GPUs, you must include an additional @CUDA@ section:
# configured, and has no effect on top-level containers.
AlwaysUsePreemptibleInstances: false
+ # Automatically add a preemptible variant for every
+ # non-preemptible entry in InstanceTypes below. The maximum bid
+ # price for the preemptible variant will be the non-preemptible
+ # price multiplied by PreemptiblePriceFactor. If 0, preemptible
+ # variants are not added automatically.
+ #
+ # A price factor of 1.0 is a reasonable starting point.
+ PreemptiblePriceFactor: 0
+
# PEM encoded SSH key (RSA, DSA, or ECDSA) used by the
# cloud dispatcher for executing containers on worker VMs.
# Begins with "-----BEGIN RSA PRIVATE KEY-----\n"
"Containers.MaxDispatchAttempts": false,
"Containers.MaxRetryAttempts": true,
"Containers.MinRetryPeriod": true,
+ "Containers.PreemptiblePriceFactor": false,
"Containers.ReserveExtraRAM": true,
"Containers.RuntimeEngine": true,
"Containers.ShellAccess": true,
}
}
+ // Preprocess/automate some configs
+ for id, cc := range cfg.Clusters {
+ ldr.autofillPreemptible("Clusters."+id, &cc)
+
+ if strings.Count(cc.Users.AnonymousUserToken, "/") == 3 {
+ // V2 token, strip it to just a secret
+ tmp := strings.Split(cc.Users.AnonymousUserToken, "/")
+ cc.Users.AnonymousUserToken = tmp[2]
+ }
+
+ cfg.Clusters[id] = cc
+ }
+
// Check for known mistakes
for id, cc := range cfg.Clusters {
for remote := range cc.RemoteClusters {
return nil, err
}
}
- if strings.Count(cc.Users.AnonymousUserToken, "/") == 3 {
- // V2 token, strip it to just a secret
- tmp := strings.Split(cc.Users.AnonymousUserToken, "/")
- cc.Users.AnonymousUserToken = tmp[2]
- }
}
return &cfg, nil
}
}
}
}
+
+func (ldr *Loader) autofillPreemptible(label string, cc *arvados.Cluster) {
+ if factor := cc.Containers.PreemptiblePriceFactor; factor > 0 {
+ for name, it := range cc.InstanceTypes {
+ if !it.Preemptible {
+ it.Preemptible = true
+ it.Price = it.Price * factor
+ it.Name = name + ".preemptible"
+ if it2, exists := cc.InstanceTypes[it.Name]; exists && it2 != it {
+ ldr.Logger.Warnf("%s.InstanceTypes[%s]: already exists, so not automatically adding a preemptible variant of %s", label, it.Name, name)
+ continue
+ }
+ cc.InstanceTypes[it.Name] = it
+ }
+ }
+ }
+
+}
func (s *LoadSuite) TestNoWarningsForDumpedConfig(c *check.C) {
var logbuf bytes.Buffer
- logger := logrus.New()
- logger.Out = &logbuf
cfg, err := testLoader(c, `
Clusters:
zzzzz:
_, err = ldr.Load()
c.Assert(err, check.ErrorMatches, `there is no default storage class.*`)
}
+
+func (s *LoadSuite) TestPreemptiblePriceFactor(c *check.C) {
+ yaml := `
+Clusters:
+ z1111:
+ InstanceTypes:
+ Type1:
+ RAM: 12345M
+ VCPUs: 8
+ Price: 1.23
+ z2222:
+ Containers:
+ PreemptiblePriceFactor: 0.5
+ InstanceTypes:
+ Type1:
+ RAM: 12345M
+ VCPUs: 8
+ Price: 1.23
+ z3333:
+ Containers:
+ PreemptiblePriceFactor: 0.5
+ InstanceTypes:
+ Type1:
+ RAM: 12345M
+ VCPUs: 8
+ Price: 1.23
+ Type1.preemptible: # higher price than the auto-added variant would use -- should generate warning
+ ProviderType: Type1
+ RAM: 12345M
+ VCPUs: 8
+ Price: 1.23
+ Preemptible: true
+ Type2:
+ RAM: 23456M
+ VCPUs: 16
+ Price: 2.46
+ Type2.preemptible: # identical to the auto-added variant -- so no warning
+ ProviderType: Type2
+ RAM: 23456M
+ VCPUs: 16
+ Price: 1.23
+ Preemptible: true
+`
+ var logbuf bytes.Buffer
+ cfg, err := testLoader(c, yaml, &logbuf).Load()
+ c.Assert(err, check.IsNil)
+ cc, err := cfg.GetCluster("z1111")
+ c.Assert(err, check.IsNil)
+ c.Check(cc.InstanceTypes["Type1"].Price, check.Equals, 1.23)
+ c.Check(cc.InstanceTypes, check.HasLen, 1)
+
+ cc, err = cfg.GetCluster("z2222")
+ c.Assert(err, check.IsNil)
+ c.Check(cc.InstanceTypes["Type1"].Preemptible, check.Equals, false)
+ c.Check(cc.InstanceTypes["Type1"].Price, check.Equals, 1.23)
+ c.Check(cc.InstanceTypes["Type1.preemptible"].Preemptible, check.Equals, true)
+ c.Check(cc.InstanceTypes["Type1.preemptible"].Price, check.Equals, 1.23/2)
+ c.Check(cc.InstanceTypes["Type1.preemptible"].ProviderType, check.Equals, "Type1")
+ c.Check(cc.InstanceTypes, check.HasLen, 2)
+
+ cc, err = cfg.GetCluster("z3333")
+ c.Assert(err, check.IsNil)
+ // Don't overwrite the explicitly configured preemptible variant
+ c.Check(cc.InstanceTypes["Type1.preemptible"].Price, check.Equals, 1.23)
+ c.Check(cc.InstanceTypes, check.HasLen, 4)
+ c.Check(logbuf.String(), check.Matches, `(?ms).*Clusters\.z3333\.InstanceTypes\[Type1\.preemptible\]: already exists, so not automatically adding a preemptible variant of Type1.*`)
+ c.Check(logbuf.String(), check.Not(check.Matches), `(?ms).*Type2\.preemptible.*`)
+ c.Check(logbuf.String(), check.Not(check.Matches), `(?ms).*(z1111|z2222)[^\n]*InstanceTypes.*`)
+}
StaleLockTimeout Duration
SupportedDockerImageFormats StringSet
AlwaysUsePreemptibleInstances bool
+ PreemptiblePriceFactor float64
RuntimeEngine string
LocalKeepBlobBuffersPerVCPU int
LocalKeepLogsToContainerLog string
end
end
- test "Having preemptible_instances=true create a committed child container request and verify the scheduling parameter of its container" do
+ test "AlwaysUsePreemptibleInstances makes child containers preemptible" do
+ Rails.configuration.Containers.AlwaysUsePreemptibleInstances = true
common_attrs = {cwd: "test",
priority: 1,
command: ["echo", "hello"],
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+{%- set curr_tpldir = tpldir %}
+{%- set tpldir = 'arvados' %}
+{%- from "arvados/map.jinja" import arvados with context %}
+{%- set tpldir = curr_tpldir %}
+
+extra_shell_sudo_passwordless_sudo_pkg_installed:
+ pkg.installed:
+ - name: sudo
+
+extra_shell_sudo_passwordless_config_file_managed:
+ file.managed:
+ - name: /etc/sudoers.d/arvados_passwordless
+ - makedirs: true
+ - user: root
+ - group: root
+ - mode: '0440'
+ - replace: false
+ - contents: |
+ # This file managed by Salt, do not edit by hand!!
+ # Allow members of group sudo to execute any command without password
+ %sudo ALL=(ALL:ALL) NOPASSWD:ALL
+ - require:
+ - pkg: extra_shell_sudo_passwordless_sudo_pkg_installed
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+{%- set curr_tpldir = tpldir %}
+{%- set tpldir = 'arvados' %}
+{%- from "arvados/map.jinja" import arvados with context %}
+{%- set tpldir = curr_tpldir %}
+
+extra_shell_sudo_passwordless_sudo_pkg_installed:
+ pkg.installed:
+ - name: sudo
+
+extra_shell_sudo_passwordless_config_file_managed:
+ file.managed:
+ - name: /etc/sudoers.d/arvados_passwordless
+ - makedirs: true
+ - user: root
+ - group: root
+ - mode: '0440'
+ - replace: false
+ - contents: |
+ # This file managed by Salt, do not edit by hand!!
+ # Allow members of group sudo to execute any command without password
+ %sudo ALL=(ALL:ALL) NOPASSWD:ALL
+ - require:
+ - pkg: extra_shell_sudo_passwordless_sudo_pkg_installed
"${f}" > "${P_DIR}"/$(basename "${f}")
done
-if [ "x${TEST}" = "xyes" ] && [ ! -d "${SOURCE_TESTS_DIR}" ]; then
- echo "You requested to run tests, but ${SOURCE_TESTS_DIR} does not exist or is not a directory. Exiting."
- exit 1
-fi
-mkdir -p ${T_DIR}
-# Replace cluster and domain name in the test files
-for f in $(ls "${SOURCE_TESTS_DIR}"/*); do
- FILTERS="s#__CLUSTER__#${CLUSTER}#g;
- s#__CONTROLLER_EXT_SSL_PORT__#${CONTROLLER_EXT_SSL_PORT}#g;
- s#__DOMAIN__#${DOMAIN}#g;
- s#__IP_INT__#${IP_INT}#g;
- s#__INITIAL_USER_EMAIL__#${INITIAL_USER_EMAIL}#g;
- s#__INITIAL_USER_PASSWORD__#${INITIAL_USER_PASSWORD}#g
- s#__INITIAL_USER__#${INITIAL_USER}#g;
- s#__DATABASE_PASSWORD__#${DATABASE_PASSWORD}#g;
- s#__SYSTEM_ROOT_TOKEN__#${SYSTEM_ROOT_TOKEN}#g"
- if [ "$USE_SINGLE_HOSTNAME" = "yes" ]; then
- FILTERS="s#__CLUSTER__.__DOMAIN__#${HOSTNAME_EXT}#g;
- $FILTERS"
+if [ ! -d "${SOURCE_TESTS_DIR}" ]; then
+ echo "WARNING: The tests directory was not copied to \"${SOURCE_TESTS_DIR}\"."
+ if [ "x${TEST}" = "xyes" ]; then
+ echo "WARNING: Disabling tests for this installation."
fi
- sed "$FILTERS" \
- "${f}" > ${T_DIR}/$(basename "${f}")
-done
-chmod 755 ${T_DIR}/run-test.sh
+ TEST="no"
+else
+ mkdir -p ${T_DIR}
+ # Replace cluster and domain name in the test files
+ for f in $(ls "${SOURCE_TESTS_DIR}"/*); do
+ FILTERS="s#__CLUSTER__#${CLUSTER}#g;
+ s#__CONTROLLER_EXT_SSL_PORT__#${CONTROLLER_EXT_SSL_PORT}#g;
+ s#__DOMAIN__#${DOMAIN}#g;
+ s#__IP_INT__#${IP_INT}#g;
+ s#__INITIAL_USER_EMAIL__#${INITIAL_USER_EMAIL}#g;
+ s#__INITIAL_USER_PASSWORD__#${INITIAL_USER_PASSWORD}#g
+ s#__INITIAL_USER__#${INITIAL_USER}#g;
+ s#__DATABASE_PASSWORD__#${DATABASE_PASSWORD}#g;
+ s#__SYSTEM_ROOT_TOKEN__#${SYSTEM_ROOT_TOKEN}#g"
+ if [ "$USE_SINGLE_HOSTNAME" = "yes" ]; then
+ FILTERS="s#__CLUSTER__.__DOMAIN__#${HOSTNAME_EXT}#g;
+ $FILTERS"
+ fi
+ sed "$FILTERS" \
+ "${f}" > ${T_DIR}/$(basename "${f}")
+ done
+ chmod 755 ${T_DIR}/run-test.sh
+fi
# Replace helper state files that differ from the formula's examples
if [ -d "${SOURCE_STATES_DIR}" ]; then
# Same when using self-signed certificates.
SKIP_SNAKE_OIL="dont_add_snakeoil_certs"
fi
- for f in $(ls "${F_DIR}"/extra/extra/*.sls | grep -v ${SKIP_SNAKE_OIL}); do
+ for f in $(ls "${F_DIR}"/extra/extra/*.sls | egrep -v "${SKIP_SNAKE_OIL}|shell_sudo_passwordless"); do
echo " - extra.$(basename ${f} | sed 's/.sls$//g')" >> ${S_DIR}/top.sls
done
# Use byo or self-signed certificates
grep -q "custom_certs" ${S_DIR}/top.sls || echo " - extra.custom_certs" >> ${S_DIR}/top.sls
fi
+ echo " - extra.shell_sudo_passwordless" >> ${S_DIR}/top.sls
echo " - postgres" >> ${S_DIR}/top.sls
echo " - docker.software" >> ${S_DIR}/top.sls
echo " - arvados" >> ${S_DIR}/top.sls
;;
"shell")
# States
+ echo " - extra.shell_sudo_passwordless" >> ${S_DIR}/top.sls
grep -q "docker" ${S_DIR}/top.sls || echo " - docker.software" >> ${S_DIR}/top.sls
grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
# Pillars
projects / arvados.git / commitdiff