git checkout $DASHQ_UNLESS_DEBUG "$COMMIT_HASH"
echo "$COMMIT_HASH" >git-commit.version
- cd "$SRC_BUILD_DIR"
- PKG_VERSION=$(version_from_git)
cd $WORKSPACE/packages/$TARGET
- fpm_build "$WORKSPACE" $SRC_BUILD_DIR/=/usr/local/arvados/src arvados-src 'dir' "$PKG_VERSION" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
+ fpm_build "$WORKSPACE" $SRC_BUILD_DIR/=/usr/local/arvados/src arvados-src 'dir' "$arvados_src_version" "--exclude=usr/local/arvados/src/.git" "--url=https://arvados.org" "--license=GNU Affero General Public License, version 3.0" "--description=The Arvados source code" "--architecture=all"
rm -rf "$SRC_BUILD_DIR"
fi
repo_subdir=${pkgname:0:1}
fi
- repo_pkg_list=$(curl -s -o - http://apt.arvados.org/pool/${D}-dev/main/${repo_subdir}/${pkgname}/)
+ repo_pkg_list=$(curl -s -o - http://apt.arvados.org/${D}/pool/main/${repo_subdir}/${pkgname}/)
echo "${repo_pkg_list}" |grep -q ${full_pkgname}
if [ $? -eq 0 ] ; then
echo "Package $full_pkgname exists upstream, not rebuilding, downloading instead!"
- curl -s -o "$WORKSPACE/packages/$TARGET/${full_pkgname}" http://apt.arvados.org/pool/${D}-dev/main/${repo_subdir}/${pkgname}/${full_pkgname}
+ curl -s -o "$WORKSPACE/packages/$TARGET/${full_pkgname}" http://apt.arvados.org/${D}/pool/main/${repo_subdir}/${pkgname}/${full_pkgname}
return 1
elif test -f "$WORKSPACE/packages/$TARGET/processed/${full_pkgname}" ; then
echo "Package $full_pkgname exists, not rebuilding!"
h3. Centralized (LoginCluster) federation
-If all clusters belong to the same organization, and users in that organization should have access to all the clusters, user management can be simplified by setting the @LoginCluster@ which manages the user database used by all other clusters in the federation. To do this, choose one cluster in the federation which will be the 'login cluster'. Set the the @Login.LoginCluster@ configuration value on all clusters in the federation to the cluster id of the login cluster. After setting @LoginCluster@, restart arvados-api-server and arvados-controller.
+If all clusters belong to the same organization, and users in that organization should have access to all the clusters, user management can be simplified by setting the @LoginCluster@ which manages the user database used by all other clusters in the federation. To do this, choose one cluster in the federation which will be the 'login cluster'. Set the @Login.LoginCluster@ configuration value on all clusters in the federation to the cluster id of the login cluster. After setting @LoginCluster@, restart arvados-api-server and arvados-controller.
<pre>
Clusters:
This page describes how to enable preemptible instances. Preemptible instances typically offer lower cost computation with a tradeoff of lower service guarantees. If a compute node is preempted, Arvados will restart the computation on a new instance.
-Currently Arvados supports preemptible instances using AWS spot instances.
+Currently Arvados supports preemptible instances using AWS and Azure spot instances.
h2. Configuration
-To use preemptible instances, set @UsePreemptibleInstances: true@ and add entries to @InstanceTypes@ with @Preemptible: true@ to @config.yml@. Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The @Price@ for preemptible instances is the maximum bid price, the actual price paid is dynamic and may be lower. For example:
+To use preemptible instances, set @UsePreemptibleInstances: true@ and add entries to @InstanceTypes@ with @Preemptible: true@ to @config.yml@. Typically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. The @Price@ for preemptible instances is the maximum bid price, the actual price paid is dynamic and will likely be lower. For example:
<pre>
Clusters:
When @UsePreemptibleInstances@ is enabled, child containers (workflow steps) will automatically be made preemptible. Note that because preempting the workflow runner would cancel the entire workflow, the workflow runner runs in a reserved (non-preemptible) instance.
-If you are using "arvados-dispatch-cloud":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html no additional configuration is required.
+No additional configuration is required, "arvados-dispatch-cloud":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html will now start preemptible instances where appropriate.
+
+h3. Cost Tracking
+
+Preemptible instances prices are declared at instance request time and defined by the maximum price that the user is willing to pay per hour. By default, this price is the same amount as the on-demand version of each instance type, and this setting is the one that @arvados-dispatch-cloud@ uses for now, as it doesn't include any pricing data to the spot instance request.
+
+For AWS, the real price that a spot instance has at any point in time is discovered at the end of each usage hour, depending on instance demand. For this reason, AWS provides a data feed subscription to get hourly logs, as described on "Amazon's User Guide":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html.
h2. Preemptible instances on AWS
-For general information, see "using Amazon EC2 spot instances":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html .
+For general information, see "using Amazon EC2 spot instances":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html.
h3. Permissions
The account needs to have a service linked role created. This can be done by logging into the AWS account, go to _IAM Management_ → _Roles_ and create the @AWSServiceRoleForEC2Spot@ role by clicking on the @Create@ button, selecting @EC2@ service and @EC2 - Spot Instances@ use case.
-h3. Cost Tracking
+h2. Preemptible instances on Azure
+
+For general information, see "Use Spot VMs in Azure":https://docs.microsoft.com/en-us/azure/virtual-machines/spot-vms.
-Amazon's Spot instances prices are declared at instance request time and defined by the maximum price that the user is willing to pay per hour. By default, this price is the same amount as the on-demand version of each instance type, and this setting is the one that @arvados-dispatch-cloud@ uses for now, as it doesn't include any pricing data to the spot instance request.
+When starting preemptible instances on Azure, Arvados configures the eviction policy to 'delete', with max price set to '-1'. This has the effect that preemptible VMs will not be evicted for pricing reasons. The price paid for the instance will be the current spot price for the VM type, up to a maximum of the price for a standard, non-spot VM of that type.
-The real price that a spot instance has at any point in time is discovered at the end of each usage hour, depending on instance demand. For this reason, AWS provides a data feed subscription to get hourly logs, as described on "Amazon's User Guide":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html.
+Please note that Azure provides no SLA for preemptible instances. Even in this configuration, preemptible instances can still be evicted for capacity reasons. If that happens and a container is aborted, Arvados will try to restart it, subject to the usual retry rules.
+Spot pricing is not available on 'B-series' VMs, those should not be defined in the configuration file with the _Preemptible_ flag set to true. Spot instances have a separate quota pool, make sure you have sufficient quota available.
The @preserve_version@ attribute on collections was originally designed to allow clients to persist a preexisting collection version. This forced clients to make 2 requests if the intention is to "make this set of changes in a new version that will be kept", so we have changed the semantics to do just that: When passing @preserve_version=true@ along with other collection updates, the current version is persisted and also the newly created one will be persisted on the next update.
+h3. System token requirements
+
+System services now log a warning at startup if any of the system tokens (@ManagementToken@, @SystemRootToken@, and @Collections.BlobSigningKey@) are less than 32 characters, or contain characters other than a-z, A-Z, and 0-9. After upgrading, run @arvados-server config-check@ and update your configuration file if needed to resolve any warnings.
+
+The @API.RailsSessionSecretToken@ configuration key has been removed. Delete this entry from your configuration file after upgrading.
+
h3. Centos7 Python 3 dependency upgraded to python3
Now that Python 3 is part of the base repository in CentOS 7, the Python 3 dependency for Centos7 Arvados packages was changed from SCL rh-python36 to python3.
h2. Workbench examples
-Many Arvados Workbench pages, under the the *Advanced* tab, provide examples of API and SDK use for accessing the current resource .
+Many Arvados Workbench pages, under the *Advanced* tab, provide examples of API and SDK use for accessing the current resource .
<a href="{{ site.baseurl }}/sdk/index.html">SDK Reference</a> — Details about the accessing Arvados from various programming languages.
</p>
<p>
- <a href="{{ site.baseurl }}/architecture/index.html">Arvados Architecture</a> — Details about the the Arvados components and architecture.
+ <a href="{{ site.baseurl }}/architecture/index.html">Arvados Architecture</a> — Details about the Arvados components and architecture.
</p>
<p>
- <a href="{{ site.baseurl }}/api/index.html">API Reference</a> — Details about the the Arvados REST API.
+ <a href="{{ site.baseurl }}/api/index.html">API Reference</a> — Details about the Arvados REST API.
</p>
<p>
<a href="{{ site.baseurl }}/admin/index.html">Admin Guide</a> — Details about administering an Arvados cluster.
SPDX-License-Identifier: CC-BY-SA-3.0
{% endcomment %}
-h2. Arados /etc/arvados/config.yml
+h2. Arvados /etc/arvados/config.yml
The configuration file is normally found at @/etc/arvados/config.yml@ and will be referred to as just @config.yml@ in this guide. This configuration file must be kept in sync across every service node in the cluster, but not shell and compute nodes (which do not require config.yml).
# "Introduction":#introduction
# "Create an SSH keypair":#sshkeypair
# "The build script":#building
-# "Build an Azure image":#azure
# "Build an AWS image":#aws
+# "Build an Azure image":#azure
h2(#introduction). Introduction
Output debug information (default: false)
</code></pre></notextile>
-h2(#azure). Build an Azure image
+h2(#aws). Build an AWS image
-<notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-azure.json \
+<notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-aws.json \
--arvados-cluster-id ClusterID \
- --azure-resource-group ResourceGroup \
- --azure-location AzureRegion \
- --azure-sku AzureSKU \
- --azure-secrets-file AzureSecretsFilePath \
+ --aws-profile AWSProfile \
+ --aws-source-ami AMI \
+ --aws-vpc-id VPC \
+ --aws-subnet-id Subnet \
+ --ssh_user admin \
--resolver ResolverIP \
--public-key-file ArvadosDispatchCloudPublicKeyPath
</span>
</code></pre></notextile>
-For @ClusterID@, fill in your cluster ID. The @ResourceGroup@ and @AzureRegion@ (e.g. 'eastus2') should be configured for where you want the compute image to be generated and stored. The @AzureSKU@ is the SKU of the base image to be used, e.g. '18.04-LTS' for Ubuntu 18.04.
-
-@AzureSecretsFilePath@ should be replaced with the path to a shell script that loads the Azure secrets with sufficient permissions to create the image. The file would look like this:
-
-<notextile><pre><code>export ARM_CLIENT_ID=...
-export ARM_CLIENT_SECRET=...
-export ARM_SUBSCRIPTION_ID=...
-export ARM_TENANT_ID=...
-</code></pre></notextile>
-
-These secrets can be generated from the Azure portal, or with the cli using a command like this:
+For @ClusterID@, fill in your cluster ID. The @VPC@ and @Subnet@ should be configured for where you want the compute image to be generated and stored. The @AMI@ is the identifier for the base image to be used. Current AMIs are maintained by "Debian":https://wiki.debian.org/Cloud/AmazonEC2Image/Buster and "Ubuntu":https://cloud-images.ubuntu.com/locator/ec2/.
-<notextile><pre><code>~$ <span class="userinput">az ad sp create-for-rbac --name Packer --password ...</span>
-</code></pre></notextile>
+@AWSProfile@ should be replaced with the name of an AWS profile with sufficient permissions to create the image.
@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
Adding these lines to the @/etc/hosts@ file in the compute node image could be done with a small change to the Packer template and the @scripts/base.sh@ script, which will be left as an exercise for the reader.
-h2(#aws). Build an AWS image
+h2(#azure). Build an Azure image
-<notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-aws.json \
+<notextile><pre><code>~$ <span class="userinput">./build.sh --json-file arvados-images-azure.json \
--arvados-cluster-id ClusterID \
- --aws-profile AWSProfile \
- --aws-source-ami AMI \
- --aws-vpc-id VPC \
- --aws-subnet-id Subnet \
- --ssh_user admin \
+ --azure-resource-group ResourceGroup \
+ --azure-location AzureRegion \
+ --azure-sku AzureSKU \
+ --azure-secrets-file AzureSecretsFilePath \
--resolver ResolverIP \
--public-key-file ArvadosDispatchCloudPublicKeyPath
</span>
</code></pre></notextile>
-For @ClusterID@, fill in your cluster ID. The @VPC@ and @Subnet@ should be configured for where you want the compute image to be generated and stored. The @AMI@ is the identifier for the base image to be used. Current AMIs are maintained by "Debian":https://wiki.debian.org/Cloud/AmazonEC2Image/Buster and "Ubuntu":https://cloud-images.ubuntu.com/locator/ec2/.
+For @ClusterID@, fill in your cluster ID. The @ResourceGroup@ and @AzureRegion@ (e.g. 'eastus2') should be configured for where you want the compute image to be generated and stored. The @AzureSKU@ is the SKU of the base image to be used, e.g. '18.04-LTS' for Ubuntu 18.04.
-@AWSProfile@ should be replaced with the name of an AWS profile with sufficient permissions to create the image.
+@AzureSecretsFilePath@ should be replaced with the path to a shell script that loads the Azure secrets with sufficient permissions to create the image. The file would look like this:
+
+<notextile><pre><code>export ARM_CLIENT_ID=...
+export ARM_CLIENT_SECRET=...
+export ARM_SUBSCRIPTION_ID=...
+export ARM_TENANT_ID=...
+</code></pre></notextile>
+
+These secrets can be generated from the Azure portal, or with the cli using a command like this:
+
+<notextile><pre><code>~$ <span class="userinput">az ad sp create-for-rbac --name Packer --password ...</span>
+</code></pre></notextile>
@ArvadosDispatchCloudPublicKeyPath@ should be replaced with the path to the ssh *public* key file generated in "Create an SSH keypair":#sshkeypair, above.
h4. Minimal configuration example for Amazon EC2
+The <span class="userinput">ImageID</span> value is the compute node image that was built in "the previous section":install-compute-node.html#aws.
+
<notextile>
<pre><code> Containers:
CloudVMs:
- ImageID: ami-01234567890abcdef
+ ImageID: <span class="userinput">ami-01234567890abcdef</span>
Driver: ec2
DriverParameters:
AccessKeyID: XXXXXXXXXXXXXXXXXXXX
Using managed disks:
+The <span class="userinput">ImageID</span> value is the compute node image that was built in "the previous section":install-compute-node.html#azure.
+
<notextile>
<pre><code> Containers:
CloudVMs:
- ImageID: "zzzzz-compute-v1597349873"
+ ImageID: <span class="userinput">"zzzzz-compute-v1597349873"</span>
Driver: azure
# (azure) managed disks: set MaxConcurrentInstanceCreateOps to 20 to avoid timeouts, cf
# https://docs.microsoft.com/en-us/azure/virtual-machines/linux/capture-image
<notextile>
<pre><code> Containers:
CloudVMs:
- ImageID: "shared_image_gallery_image_definition_name"
+ ImageID: <span class="userinput">"shared_image_gallery_image_definition_name"</span>
Driver: azure
DriverParameters:
# Credentials.
Using unmanaged disks (deprecated):
+The <span class="userinput">ImageID</span> value is the compute node image that was built in "the previous section":install-compute-node.html#azure.
+
<notextile>
<pre><code> Containers:
CloudVMs:
- ImageID: "https://zzzzzzzz.blob.core.windows.net/system/Microsoft.Compute/Images/images/zzzzz-compute-osDisk.55555555-5555-5555-5555-555555555555.vhd"
+ ImageID: <span class="userinput">"https://zzzzzzzz.blob.core.windows.net/system/Microsoft.Compute/Images/images/zzzzz-compute-osDisk.55555555-5555-5555-5555-555555555555.vhd"</span>
Driver: azure
DriverParameters:
# Credentials.
<notextile>
<pre><code> SystemRootToken: <span class="userinput">"$system_root_token"</span>
ManagementToken: <span class="userinput">"$management_token"</span>
- API:
- RailsSessionSecretToken: <span class="userinput">"$rails_secret_token"</span>
Collections:
BlobSigningKey: <span class="userinput">"$blob_signing_key"</span>
</code></pre>
These secret tokens are used to authenticate messages between Arvados components.
* @SystemRootToken@ is used by Arvados system services to authenticate as the system (root) user when communicating with the API server.
* @ManagementToken@ is used to authenticate access to system metrics.
-* @API.RailsSessionSecretToken@ is used to sign session cookies.
* @Collections.BlobSigningKey@ is used to control access to Keep blocks.
Each token should be a string of at least 50 alphanumeric characters. You can generate a suitable token with the following command:
h3. Collections download URL
-Downloads links will served from the the URL in @Services.WebDAVDownload.ExternalURL@ . The collection uuid or PDH is put in the URL path.
+Downloads links will served from the URL in @Services.WebDAVDownload.ExternalURL@ . The collection uuid or PDH is put in the URL path.
If blank, serve links to WebDAV with @disposition=attachment@ query param. Unlike preview links, browsers do not render attachments, so there is no risk of XSS.
A shell node runs the @arvados-login-sync@ service to manage user accounts, and typically has Arvados utilities and SDKs pre-installed. Users are allowed to log in and run arbitrary programs. For optimal performance, the Arvados shell server should be on the same LAN as the Arvados cluster.
-Because it _contains secrets_ shell nodes should *not* have a copy of the Arvados @config.yml@.
+Because Arvados @config.yml@ _contains secrets_ it should not *not* be present on shell nodes.
Shell nodes should be separate virtual machines from the VMs running other Arvados services. You may choose to grant root access to users so that they can customize the node, for example, installing new programs. This has security considerations depending on whether a shell node is single-user or multi-user.
Set @ARVADOS_VIRTUAL_MACHINE_UUID@ to the UUID from "Create record for VM":#vm-record
+h3. Standalone cluster
+
<notextile>
<pre>
<code>shellserver:# <span class="userinput">umask 0700; tee /etc/cron.d/arvados-login-sync <<EOF
</pre>
</notextile>
+h3. Part of a LoginCLuster federation
+
+If this cluster is part of a "federation with centralized user management":../admin/federation.html#LoginCluster , the login sync script also needs to be given the host and user token for the login cluster.
+
+<notextile>
+<pre>
+<code>shellserver:# <span class="userinput">umask 0700; tee /etc/cron.d/arvados-login-sync <<EOF
+ARVADOS_API_HOST="<strong>ClusterID.example.com</strong>"
+ARVADOS_API_TOKEN="<strong>xxxxxxxxxxxxxxxxx</strong>"
+LOGINCLUSTER_ARVADOS_API_HOST="<strong>LoginClusterID.example.com</strong>"
+LOGINCLUSTER_ARVADOS_API_TOKEN="<strong>yyyyyyyyyyyyyyyyy</strong>"
+ARVADOS_VIRTUAL_MACHINE_UUID="<strong>zzzzz-2x53u-zzzzzzzzzzzzzzz</strong>"
+*/2 * * * * root arvados-login-sync
+EOF</span></code>
+</pre>
+</notextile>
+
+
h2(#confirm-working). Confirm working installation
A user should be able to log in to the shell server when the following conditions are satisfied:
table(table table-bordered table-condensed).
|_. OS version|_. Command|
-|Debian 10 ("buster")|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ buster main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
-|Ubuntu 18.04 ("bionic")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ bionic main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
-|Ubuntu 16.04 ("xenial")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/ xenial main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
+|Debian 10 ("buster")|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/buster buster main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
+|Ubuntu 20.04 ("focal")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/focal focal main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
+|Ubuntu 18.04 ("bionic")[1]|<notextile><code><span class="userinput">echo "deb http://apt.arvados.org/bionic bionic main" | tee /etc/apt/sources.list.d/arvados.list</span></code></notextile>|
{% include 'notebox_begin' %}
Arvados uses SSL to encrypt communications. Its UI uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
-For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so ypu can add it to your workstation.
+For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so you can add it to your workstation.
Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
Arvados uses SSL to encrypt communications. Its UI uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
-For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so ypu can add it to your workstation.
+For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so you can add it to your workstation.
Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
The SDK is packaged as a JAR named @arvados-java-<version>.jar@, which is published to Maven Central and can be included using Maven, Gradle, or by hand.
-Here is an example @build.gradle@ file that uses the Arados java sdk:
+Here is an example @build.gradle@ file that uses the Arvados java sdk:
<pre>
apply plugin: 'application'
We have already previously registered the WGS workflow and set default input values for this set of the walkthrough.
-Let’s find the the registered WGS Processing Workflow and run it interactively in our newly created project.
+Let’s find the registered WGS Processing Workflow and run it interactively in our newly created project.
# To find the registered workflow, you can search for it in the search box located in the top right corner of the Arvados Workbench by looking for the name “WGS Processing Workflow”.
# Once you have found the registered workflow, you can run it your project by using the <span class="btn btn-sm btn-primary" >Run this workflow..</span> button and selecting your project ("WGS Processing Tutorial") that you set up in Section 3a.
# apt.arvados.org
-deb http://apt.arvados.org/ buster-dev main
+deb http://apt.arvados.org/buster buster-dev main
# apt.arvados.org
-deb http://apt.arvados.org/ buster main
+deb http://apt.arvados.org/buster buster main
# apt.arvados.org
-deb http://apt.arvados.org/ buster-testing main
+deb http://apt.arvados.org/buster buster-testing main
VOLUME /var/lib/docker
RUN mkdir -p /etc/apt/sources.list.d && \
- echo deb http://apt.arvados.org/ jessie main > /etc/apt/sources.list.d/apt.arvados.org.list && \
+ echo deb http://apt.arvados.org/jessie jessie main > /etc/apt/sources.list.d/apt.arvados.org.list && \
apt-get clean && \
apt-get update && \
apt-get install -yq --no-install-recommends -o Acquire::Retries=6 \
createCertificates{},
runPostgreSQL{},
runNginx{},
- runServiceCommand{name: "controller", svc: super.cluster.Services.Controller, depends: []supervisedTask{runPostgreSQL{}}},
+ runServiceCommand{name: "controller", svc: super.cluster.Services.Controller, depends: []supervisedTask{seedDatabase{}}},
runGoProgram{src: "services/arv-git-httpd", svc: super.cluster.Services.GitHTTP},
runGoProgram{src: "services/health", svc: super.cluster.Services.Health},
runGoProgram{src: "services/keepproxy", svc: super.cluster.Services.Keepproxy, depends: []supervisedTask{runPassenger{src: "services/api"}}},
runGoProgram{src: "services/keepstore", svc: super.cluster.Services.Keepstore},
runGoProgram{src: "services/keep-web", svc: super.cluster.Services.WebDAV},
- runServiceCommand{name: "ws", svc: super.cluster.Services.Websocket, depends: []supervisedTask{runPostgreSQL{}}},
+ runServiceCommand{name: "ws", svc: super.cluster.Services.Websocket, depends: []supervisedTask{seedDatabase{}}},
installPassenger{src: "services/api"},
- runPassenger{src: "services/api", varlibdir: "railsapi", svc: super.cluster.Services.RailsAPI, depends: []supervisedTask{createCertificates{}, runPostgreSQL{}, installPassenger{src: "services/api"}}},
- installPassenger{src: "apps/workbench", depends: []supervisedTask{installPassenger{src: "services/api"}}}, // dependency ensures workbench doesn't delay api startup
+ runPassenger{src: "services/api", varlibdir: "railsapi", svc: super.cluster.Services.RailsAPI, depends: []supervisedTask{createCertificates{}, seedDatabase{}, installPassenger{src: "services/api"}}},
+ installPassenger{src: "apps/workbench", depends: []supervisedTask{seedDatabase{}}}, // dependency ensures workbench doesn't delay api install/startup
runPassenger{src: "apps/workbench", varlibdir: "workbench1", svc: super.cluster.Services.Workbench1, depends: []supervisedTask{installPassenger{src: "apps/workbench"}}},
seedDatabase{},
}
if cluster.ManagementToken == "" {
cluster.ManagementToken = randomHexString(64)
}
- if cluster.API.RailsSessionSecretToken == "" {
- cluster.API.RailsSessionSecretToken = randomHexString(64)
- }
if cluster.Collections.BlobSigningKey == "" {
cluster.Collections.BlobSigningKey = randomHexString(64)
}
+ if cluster.Users.AnonymousUserToken == "" {
+ cluster.Users.AnonymousUserToken = randomHexString(64)
+ }
if cluster.Containers.DispatchPrivateKey == "" {
buf, err := ioutil.ReadFile(filepath.Join(super.SourcePath, "lib", "dispatchcloud", "test", "sshkey_dispatch"))
if err != nil {
},
}
+ if instanceType.Preemptible {
+ // Setting maxPrice to -1 is the equivalent of paying spot price, up to the
+ // normal price. This means the node will not be pre-empted for price
+ // reasons. It may still be pre-empted for capacity reasons though. And
+ // Azure offers *no* SLA on spot instances.
+ var maxPrice float64 = -1
+ vmParameters.VirtualMachineProperties.Priority = compute.Spot
+ vmParameters.VirtualMachineProperties.EvictionPolicy = compute.Delete
+ vmParameters.VirtualMachineProperties.BillingProfile = &compute.BillingProfile{MaxPrice: &maxPrice}
+ }
+
vm, err := az.vmClient.createOrUpdate(az.ctx, az.azconfig.ResourceGroup, name, vmParameters)
if err != nil {
// Do some cleanup. Otherwise, an unbounded number of new unused nics and
Price: .02,
Preemptible: false,
},
+ "tinyp": {
+ Name: "tiny",
+ ProviderType: "Standard_D1_v2",
+ VCPUs: 1,
+ RAM: 4000000000,
+ Scratch: 10000000000,
+ Price: .002,
+ Preemptible: true,
+ },
})}
if *live != "" {
var exampleCfg testConfig
c.Check(tags["TestTagName"], check.Equals, "test tag value")
c.Logf("inst.String()=%v Address()=%v Tags()=%v", inst.String(), inst.Address(), tags)
+ instPreemptable, err := ap.Create(cluster.InstanceTypes["tinyp"],
+ img, map[string]string{
+ "TestTagName": "test tag value",
+ }, "umask 0600; echo -n test-file-data >/var/run/test-file", pk)
+
+ c.Assert(err, check.IsNil)
+
+ tags = instPreemptable.Tags()
+ c.Check(tags["TestTagName"], check.Equals, "test tag value")
+ c.Logf("instPreemptable.String()=%v Address()=%v Tags()=%v", instPreemptable.String(), instPreemptable.Address(), tags)
+
}
func (*AzureInstanceSetSuite) TestListInstances(c *check.C) {
in := `
Clusters:
z1234:
- ManagementToken: xyzzy
- SystemRootToken: xyzzy
+ ManagementToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+ SystemRootToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
API:
MaxItemsPerResponse: 1234
+ Collections:
+ BlobSigningKey: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
PostgreSQL:
Connection:
sslmode: require
# serving a single incoming multi-cluster (federated) request.
MaxRequestAmplification: 4
- # RailsSessionSecretToken is a string of alphanumeric characters
- # used by Rails to sign session tokens. IMPORTANT: This is a
- # site secret. It should be at least 50 characters.
- RailsSessionSecretToken: ""
-
# Maximum wall clock time to spend handling an incoming request.
RequestTimeout: 5m
}
`)
cluster, err := testLoadLegacyConfig(content, "-legacy-keepweb-config", c)
- c.Check(err, check.IsNil)
+ c.Assert(err, check.IsNil)
c.Check(cluster.Services.Controller.ExternalURL, check.Equals, arvados.URL{Scheme: "https", Host: "example.com", Path: "/"})
c.Check(cluster.SystemRootToken, check.Equals, "abcdefg")
}
`)
cluster, err := testLoadLegacyConfig(content, "-legacy-keepweb-config", c)
- c.Check(err, check.IsNil)
+ c.Assert(err, check.IsNil)
// The resulting ManagementToken should be the one set up on the test server.
c.Check(cluster.ManagementToken, check.Equals, TestServerManagementToken)
}
content := []byte(fmtKeepproxyConfig("", true))
cluster, err := testLoadLegacyConfig(content, f, c)
- c.Check(err, check.IsNil)
- c.Check(cluster, check.NotNil)
+ c.Assert(err, check.IsNil)
+ c.Assert(cluster, check.NotNil)
c.Check(cluster.Services.Controller.ExternalURL, check.Equals, arvados.URL{Scheme: "https", Host: "example.com", Path: "/"})
c.Check(cluster.SystemRootToken, check.Equals, "abcdefg")
c.Check(cluster.ManagementToken, check.Equals, "xyzzy")
f := "-legacy-git-httpd-config"
cluster, err := testLoadLegacyConfig(content, f, c)
- c.Check(err, check.IsNil)
- c.Check(cluster, check.NotNil)
+ c.Assert(err, check.IsNil)
+ c.Assert(cluster, check.NotNil)
c.Check(cluster.Services.Controller.ExternalURL, check.Equals, arvados.URL{Scheme: "https", Host: "example.com", Path: "/"})
c.Check(cluster.SystemRootToken, check.Equals, "abcdefg")
c.Check(cluster.ManagementToken, check.Equals, "xyzzy")
}
`)
cluster, err := testLoadLegacyConfig(content, "-legacy-git-httpd-config", c)
- c.Check(err, check.IsNil)
+ c.Assert(err, check.IsNil)
// The resulting ManagementToken should be the one set up on the test server.
c.Check(cluster.ManagementToken, check.Equals, TestServerManagementToken)
}
content := []byte(fmtKeepBalanceConfig(""))
cluster, err := testLoadLegacyConfig(content, f, c)
- c.Check(err, check.IsNil)
- c.Check(cluster, check.NotNil)
+ c.Assert(err, check.IsNil)
+ c.Assert(cluster, check.NotNil)
c.Check(cluster.ManagementToken, check.Equals, "xyzzy")
c.Check(cluster.Services.Keepbalance.InternalURLs[arvados.URL{Host: ":80"}], check.Equals, arvados.ServiceInstance{})
c.Check(cluster.Collections.BalanceCollectionBuffers, check.Equals, 1000)
"API.MaxKeepBlobBuffers": false,
"API.MaxRequestAmplification": false,
"API.MaxRequestSize": true,
- "API.RailsSessionSecretToken": false,
"API.RequestTimeout": true,
"API.SendTimeout": true,
"API.WebsocketClientEventQueue": false,
# serving a single incoming multi-cluster (federated) request.
MaxRequestAmplification: 4
- # RailsSessionSecretToken is a string of alphanumeric characters
- # used by Rails to sign session tokens. IMPORTANT: This is a
- # site secret. It should be at least 50 characters.
- RailsSessionSecretToken: ""
-
# Maximum wall clock time to spend handling an incoming request.
RequestTimeout: 5m
"io"
"io/ioutil"
"os"
+ "regexp"
"strings"
"git.arvados.org/arvados.git/sdk/go/arvados"
// Check for known mistakes
for id, cc := range cfg.Clusters {
for _, err = range []error{
+ ldr.checkToken(fmt.Sprintf("Clusters.%s.ManagementToken", id), cc.ManagementToken),
+ ldr.checkToken(fmt.Sprintf("Clusters.%s.SystemRootToken", id), cc.SystemRootToken),
+ ldr.checkToken(fmt.Sprintf("Clusters.%s.Collections.BlobSigningKey", id), cc.Collections.BlobSigningKey),
checkKeyConflict(fmt.Sprintf("Clusters.%s.PostgreSQL.Connection", id), cc.PostgreSQL.Connection),
ldr.checkEmptyKeepstores(cc),
ldr.checkUnlistedKeepstores(cc),
return &cfg, nil
}
+var acceptableTokenRe = regexp.MustCompile(`^[a-zA-Z0-9]+$`)
+var acceptableTokenLength = 32
+
+func (ldr *Loader) checkToken(label, token string) error {
+ if token == "" {
+ ldr.Logger.Warnf("%s: secret token is not set (use %d+ random characters from a-z, A-Z, 0-9)", label, acceptableTokenLength)
+ } else if !acceptableTokenRe.MatchString(token) {
+ return fmt.Errorf("%s: unacceptable characters in token (only a-z, A-Z, 0-9 are acceptable)", label)
+ } else if len(token) < acceptableTokenLength {
+ ldr.Logger.Warnf("%s: token is too short (should be at least %d characters)", label, acceptableTokenLength)
+ }
+ return nil
+}
+
func checkKeyConflict(label string, m map[string]string) error {
saw := map[string]bool{}
for k := range m {
_, err := testLoader(c, `
Clusters:
zzzzz:
+ ManagementToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+ SystemRootToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+ Collections:
+ BlobSigningKey: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
postgresql: {}
BadKey: {}
Containers: {}
err = yaml.Unmarshal(buf, &loaded)
c.Assert(err, check.IsNil)
+ c.Check(logbuf.String(), check.Matches, `(?ms).*SystemRootToken: secret token is not set.*`)
+ c.Check(logbuf.String(), check.Matches, `(?ms).*ManagementToken: secret token is not set.*`)
+ c.Check(logbuf.String(), check.Matches, `(?ms).*Collections.BlobSigningKey: secret token is not set.*`)
+ logbuf.Reset()
loader.logExtraKeys(loaded, supplied, "")
c.Check(logbuf.String(), check.Equals, "")
}
var logbuf bytes.Buffer
logger := logrus.New()
logger.Out = &logbuf
- cfg, err := testLoader(c, `{"Clusters":{"zzzzz":{}}}`, &logbuf).Load()
+ cfg, err := testLoader(c, `
+Clusters:
+ zzzzz:
+ ManagementToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+ SystemRootToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+ Collections:
+ BlobSigningKey: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa`, &logbuf).Load()
c.Assert(err, check.IsNil)
yaml, err := yaml.Marshal(cfg)
c.Assert(err, check.IsNil)
c.Check(logbuf.String(), check.Equals, "")
}
+func (s *LoadSuite) TestUnacceptableTokens(c *check.C) {
+ for _, trial := range []struct {
+ short bool
+ configPath string
+ example string
+ }{
+ {false, "SystemRootToken", "SystemRootToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa_b_c"},
+ {false, "ManagementToken", "ManagementToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa b c"},
+ {false, "ManagementToken", "ManagementToken: \"$aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabc\""},
+ {false, "Collections.BlobSigningKey", "Collections: {BlobSigningKey: \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa⛵\"}"},
+ {true, "SystemRootToken", "SystemRootToken: a_b_c"},
+ {true, "ManagementToken", "ManagementToken: a b c"},
+ {true, "ManagementToken", "ManagementToken: \"$abc\""},
+ {true, "Collections.BlobSigningKey", "Collections: {BlobSigningKey: \"⛵\"}"},
+ } {
+ c.Logf("trying bogus config: %s", trial.example)
+ _, err := testLoader(c, "Clusters:\n zzzzz:\n "+trial.example, nil).Load()
+ if trial.short {
+ c.Check(err, check.ErrorMatches, `Clusters.zzzzz.`+trial.configPath+`: unacceptable characters in token.*`)
+ } else {
+ c.Check(err, check.ErrorMatches, `Clusters.zzzzz.`+trial.configPath+`: unacceptable characters in token.*`)
+ }
+ }
+}
+
func (s *LoadSuite) TestPostgreSQLKeyConflict(c *check.C) {
_, err := testLoader(c, `
Clusters:
MaxKeepBlobBuffers int
MaxRequestAmplification int
MaxRequestSize int
- RailsSessionSecretToken string
RequestTimeout Duration
SendTimeout Duration
WebsocketClientEventQueue int
class KeepWriterThread(threading.Thread):
- TaskFailed = RuntimeError()
+ class TaskFailed(RuntimeError): pass
def __init__(self, queue, data, data_hash, timeout=None):
super(KeepClient.KeepWriterThread, self).__init__()
try:
locator, copies = self.do_task(service, service_root)
except Exception as e:
- if e is not self.TaskFailed:
+ if not isinstance(e, self.TaskFailed):
_logger.exception("Exception in KeepWriterThread")
self.queue.write_fail(service)
else:
self.data_hash,
result['status_code'],
result['body'])
- raise self.TaskFailed
+ raise self.TaskFailed()
_logger.debug("KeepWriterThread %s succeeded %s+%i %s",
str(threading.current_thread()),
"SystemRootToken": auth_token('system_user'),
"API": {
"RequestTimeout": "30s",
- "RailsSessionSecretToken": "e24205c490ac07e028fd5f8a692dcb398bcd654eff1aef5f9fe6891994b18483",
},
"Login": {
"SSO": {
class TransactionFailedError < StandardError
end
- @@config = nil
@@debuglevel = 0
class << self
attr_accessor :debuglevel
@arvados_api_version = opts[:api_version] || 'v1'
- @arvados_api_host = opts[:api_host] ||
- config['ARVADOS_API_HOST'] or
- raise "#{$0}: no :api_host or ENV[ARVADOS_API_HOST] provided."
- @arvados_api_token = opts[:api_token] ||
- config['ARVADOS_API_TOKEN'] or
- raise "#{$0}: no :api_token or ENV[ARVADOS_API_TOKEN] provided."
+ @config = nil
+ [[:api_host, 'ARVADOS_API_HOST'],
+ [:api_token, 'ARVADOS_API_TOKEN']].each do |op, en|
+ if opts[op]
+ config[en] = opts[op]
+ end
+ if !config[en]
+ raise "#{$0}: no :#{op} or ENV[#{en}] provided."
+ end
+ end
if (opts[:suppress_ssl_warnings] or
%w(1 true yes).index(config['ARVADOS_API_HOST_INSECURE'].
# result looks like Arvados::A26949680::Job.
namespace_class.const_set classname, klass
- self.class.class_eval do
- define_method classname.underscore do
- klass
- end
+ self.define_singleton_method classname.underscore do
+ klass
end
end
end
def client
@client ||= Google::APIClient.
- new(:host => @arvados_api_host,
+ new(:host => config["ARVADOS_API_HOST"],
:application_name => @application_name,
:application_version => @application_version.to_s)
end
end
def config(config_file_path="~/.config/arvados/settings.conf")
- return @@config if @@config
+ return @config if @config
# Initialize config settings with environment variables.
config = {}
# Note: If we start using additional configuration settings from
# this file in the future, we might have to read the file anyway
# instead of returning here.
- return (@@config = config)
+ return (@config = config)
end
begin
debuglog "Ignoring error reading #{config_file_path}: #{e}", 0
end
- @@config = config
+ @config = config
end
class Model
:parameters => parameters,
:body_object => body,
:headers => {
- :authorization => 'OAuth2 '+arvados.config['ARVADOS_API_TOKEN']
+ :authorization => 'Bearer '+arvados.config['ARVADOS_API_TOKEN']
})
resp = JSON.parse result.body, :symbolize_names => true
if resp[:errors]
arvcfg.declare_config "API.MaxIndexDatabaseRead", Integer, :max_index_database_read
arvcfg.declare_config "API.MaxItemsPerResponse", Integer, :max_items_per_response
arvcfg.declare_config "API.AsyncPermissionsUpdateInterval", ActiveSupport::Duration, :async_permissions_update_interval
-arvcfg.declare_config "API.RailsSessionSecretToken", NonemptyString, :secret_token
arvcfg.declare_config "Users.AutoSetupNewUsers", Boolean, :auto_setup_new_users
arvcfg.declare_config "Users.AutoSetupNewUsersWithVmUUID", String, :auto_setup_new_users_with_vm_uuid
arvcfg.declare_config "Users.AutoSetupNewUsersWithRepository", Boolean, :auto_setup_new_users_with_repository
# Rails.configuration.API["Blah"]
ConfigLoader.copy_into_config $arvados_config, config
ConfigLoader.copy_into_config $remaining_config, config
- secrets.secret_key_base = $arvados_config["API"]["RailsSessionSecretToken"]
+
+ # We don't rely on cookies for authentication, so instead of
+ # requiring a signing key in config, we assign a new random one at
+ # startup.
+ secrets.secret_key_base = rand(1<<255).to_s(36)
end
1. Add this Arvados repository to your sources list::
- deb http://apt.arvados.org/ jessie main
+ deb http://apt.arvados.org/buster buster main
2. Update your package list.
begin
arv = Arvados.new({ :suppress_ssl_warnings => false })
+ logincluster_arv = Arvados.new({ :api_host => (ENV['LOGINCLUSTER_ARVADOS_API_HOST'] || ENV['ARVADOS_API_HOST']),
+ :api_token => (ENV['LOGINCLUSTER_ARVADOS_API_TOKEN'] || ENV['ARVADOS_API_TOKEN']),
+ :suppress_ssl_warnings => false })
vm_uuid = ENV['ARVADOS_VIRTUAL_MACHINE_UUID']
begin
if !File.exist?(tokenfile)
- user_token = arv.api_client_authorization.create(api_client_authorization: {owner_uuid: l[:user_uuid], api_client_id: 0})
+ user_token = logincluster_arv.api_client_authorization.create(api_client_authorization: {owner_uuid: l[:user_uuid], api_client_id: 0})
f = File.new(tokenfile, 'w')
f.write("ARVADOS_API_HOST=#{ENV['ARVADOS_API_HOST']}\n")
f.write("ARVADOS_API_TOKEN=v2/#{user_token[:uuid]}/#{user_token[:api_token]}\n")
password: ${database_pw}
dbname: arvados_${database_env}
client_encoding: utf8
- API:
- RailsSessionSecretToken: $secret_token
Collections:
BlobSigningKey: $blob_signing_key
DefaultReplication: 1
# Add the arvados apt repository
echo "# apt.arvados.org" |$SUDO tee --append /etc/apt/sources.list.d/apt.arvados.org.list
-echo "deb http://apt.arvados.org/ $LSB_RELEASE_CODENAME${REPOSUFFIX} main" |$SUDO tee --append /etc/apt/sources.list.d/apt.arvados.org.list
+echo "deb http://apt.arvados.org/$LSB_RELEASE_CODENAME $LSB_RELEASE_CODENAME${REPOSUFFIX} main" |$SUDO tee --append /etc/apt/sources.list.d/apt.arvados.org.list
# Add the arvados signing key
cat /tmp/1078ECD7.asc | $SUDO apt-key add -
# Usually there's no need to modify things below this line
# Formulas versions
-ARVADOS_TAG="v1.1.3"
+ARVADOS_TAG="v1.1.4"
POSTGRES_TAG="v0.41.3"
NGINX_TAG="v2.4.0"
DOCKER_TAG="v1.0.0"
projects / arvados.git / commitdiff