.Rproj.user
_version.py
*.bak
+arvados-snakeoil-ca.pem
+.vagrant
h2. Usage
-Set ARVADOS_API_HOST and ARVADOS_API_TOKEN for an admin user or system root token.
+Set ARVADOS_API_HOST to the api server of the cluster for which the report should be generated. ARVADOS_API_TOKEN needs to be a "v2 token":../admin/scoped-tokens.html for an admin user, or a superuser token (e.g. generated with @script/create_superuser_token.rb@). Please note that in a login cluster federation, the token needs to be issued by the login cluster, but the report should be generated against the API server of the cluster for which it is desired. In other words, ARVADOS_API_HOST would point at the satellite cluster for which the report is desired, but ARVADOS_API_TOKEN would be a token that belongs to a login cluster user.
Run the tool with the option @--days@ giving the number of days to report on. It will request activity logs from the API and generate a summary report on standard output.
The certificate will be added under the "Arvados testing" organization as "arvbox testing root CA".
-To access your Arvbox instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage (instructions for Debian/Ubuntu):
+To access your Arvbox instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
-# copy @arvbox-root-cert.pem@ to @/usr/local/share/ca-certificates/@
-# run @/usr/sbin/update-ca-certificates@
+h3. On Debian/Ubuntu:
+
+<notextile>
+<pre><code>cp arvbox-root-cert.pem /usr/local/share/ca-certificates/
+/usr/sbin/update-ca-certificates
+</code></pre>
+</notextile>
+
+h3. On CentOS:
+
+<notextile>
+<pre><code>cp arvbox-root-cert.pem /etc/pki/ca-trust/source/anchors/
+/usr/bin/update-ca-trust
+</code></pre>
+</notextile>
h2. Configs
# "Install Saltstack":#saltstack
# "Single host install using the provision.sh script":#single_host
-# "DNS configuration":#final_steps
+# "Final steps":#final_steps
+## "DNS configuration":#dns_configuration
+## "Install root certificate":#ca_root_certificate
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
</code></pre>
</notextile>
-h2(#final_steps). DNS configuration
+h2(#final_steps). Final configuration steps
+
+h3(#dns_configuration). DNS configuration
After the setup is done, you need to set up your DNS to be able to access the cluster.
</code></pre>
</notextile>
+h3(#ca_root_certificate). Install root certificate
+
+Arvados uses SSL to encrypt communications. Its UI uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
+
+For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so ypu can add it to your workstation.
+
+Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
+
+# Go to the certificate manager in your browser.
+#* In Chrome, this can be found under "Settings → Advanced → Manage Certificates" or by entering @chrome://settings/certificates@ in the URL bar.
+#* In Firefox, this can be found under "Preferences → Privacy & Security" or entering @about:preferences#privacy@ in the URL bar and then choosing "View Certificates...".
+# Select the "Authorities" tab, then press the "Import" button. Choose @arvados-snakeoil-ca.pem@
+
+The certificate will be added under the "Arvados Formula".
+
+To access your Arvados instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
+
+* On Debian/Ubuntu:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /usr/local/share/ca-certificates/
+/usr/sbin/update-ca-certificates
+</code></pre>
+</notextile>
+
+* On CentOS:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /etc/pki/ca-trust/source/anchors/
+/usr/bin/update-ca-trust
+</code></pre>
+</notextile>
+
h2(#initial_user). Initial user and login
At this point you should be able to log into the Arvados cluster.
{% endcomment %}
# "Vagrant":#vagrant
-# "DNS configuration":#final_steps
+# "Final steps":#final_steps
+## "DNS configuration":#dns_configuration
+## "Install root certificate":#ca_root_certificate
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
</code></pre>
</notextile>
-h2(#final_steps). DNS configuration
+h2(#final_steps). Final configuration steps
+
+h3(#dns_configuration). DNS configuration
After the setup is done, you need to set up your DNS to be able to access the cluster.
</code></pre>
</notextile>
+h3(#ca_root_certificate). Install root certificate
+
+Arvados uses SSL to encrypt communications. Its UI uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
+
+For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so ypu can add it to your workstation.
+
+Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
+
+# Go to the certificate manager in your browser.
+#* In Chrome, this can be found under "Settings → Advanced → Manage Certificates" or by entering @chrome://settings/certificates@ in the URL bar.
+#* In Firefox, this can be found under "Preferences → Privacy & Security" or entering @about:preferences#privacy@ in the URL bar and then choosing "View Certificates...".
+# Select the "Authorities" tab, then press the "Import" button. Choose @arvados-snakeoil-ca.pem@
+
+The certificate will be added under the "Arvados Formula".
+
+To access your Arvados instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
+
+* On Debian/Ubuntu:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /usr/local/share/ca-certificates/
+/usr/sbin/update-ca-certificates
+</code></pre>
+</notextile>
+
+* On CentOS:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /etc/pki/ca-trust/source/anchors/
+/usr/bin/update-ca-trust
+</code></pre>
+</notextile>
+
h2(#initial_user). Initial user and login
At this point you should be able to log into the Arvados cluster.
end
end
- api_client_auth.api_token
+ "v2/" + api_client_auth.uuid + "/" + api_client_auth.api_token
end
end
end
class CreateSuperUserTokenTest < ActiveSupport::TestCase
include CreateSuperUserToken
- test "create superuser token twice and expect same resutls" do
+ test "create superuser token twice and expect same results" do
# Create a token with some string
token1 = create_superuser_token 'atesttoken'
assert_not_nil token1
- assert_equal token1, 'atesttoken'
+ assert_match(/atesttoken$/, token1)
# Create token again; this time, we should get the one created earlier
token2 = create_superuser_token
# Create a token with some string
token1 = create_superuser_token 'atesttoken'
assert_not_nil token1
- assert_equal token1, 'atesttoken'
+ assert_match(/\/atesttoken$/, token1)
# Create token again with some other string and expect the existing superuser token back
token2 = create_superuser_token 'someothertokenstring'
assert_equal token1, token2
end
- test "create superuser token twice and expect same results" do
- # Create a token with some string
- token1 = create_superuser_token 'atesttoken'
- assert_not_nil token1
- assert_equal token1, 'atesttoken'
-
- # Create token again with that same superuser token and expect it back
- token2 = create_superuser_token 'atesttoken'
- assert_not_nil token2
- assert_equal token1, token2
- end
-
test "create superuser token and invoke again with some other valid token" do
# Create a token with some string
token1 = create_superuser_token 'atesttoken'
assert_not_nil token1
- assert_equal token1, 'atesttoken'
+ assert_match(/\/atesttoken$/, token1)
su_token = api_client_authorizations("system_user").api_token
token2 = create_superuser_token su_token
- assert_equal token2, su_token
+ assert_equal token2.split('/')[2], su_token
end
test "create superuser token, expire it, and create again" do
# Create a token with some string
token1 = create_superuser_token 'atesttoken'
assert_not_nil token1
- assert_equal token1, 'atesttoken'
+ assert_match(/\/atesttoken$/, token1)
# Expire this token and call create again; expect a new token created
- apiClientAuth = ApiClientAuthorization.where(api_token: token1).first
+ apiClientAuth = ApiClientAuthorization.where(api_token: 'atesttoken').first
+ refute_nil apiClientAuth
Thread.current[:user] = users(:admin)
apiClientAuth.update_attributes expires_at: '2000-10-10'
arv.vm.provision "shell",
path: "provision.sh",
args: [
+ # "--debug",
"--test",
"--vagrant",
"--ssl-port=8443"
-#!/bin/bash
+#!/bin/bash
# Copyright (C) The Arvados Authors. All rights reserved.
#
##########################################################
# Usually there's no need to modify things below this line
+# Formulas versions
+ARVADOS_TAG="v1.1.3"
+POSTGRES_TAG="v0.41.3"
+NGINX_TAG="v2.4.0"
+DOCKER_TAG="v1.0.0"
+LOCALE_TAG="v0.3.4"
+
set -o pipefail
# capture the directory that the script is running from
base:
- ${S_DIR}
- ${F_DIR}/*
- - ${F_DIR}/*/test/salt/states
+ - ${F_DIR}/*/test/salt/states/examples
pillar_roots:
base:
cat > ${S_DIR}/top.sls << EOFTSLS
base:
'*':
- - example_single_host_host_entries
- - example_add_snakeoil_certs
+ - single_host.host_entries
+ - single_host.snakeoil_certs
- locale
- nginx.passenger
- postgres
- postgresql
EOFPSLS
-
# Get the formula and dependencies
cd ${F_DIR} || exit 1
-for f in postgres arvados nginx docker locale; do
- git clone https://github.com/saltstack-formulas/${f}-formula.git
-done
+git clone --branch "${ARVADOS_TAG}" https://github.com/saltstack-formulas/arvados-formula.git
+git clone --branch "${DOCKER_TAG}" https://github.com/saltstack-formulas/docker-formula.git
+git clone --branch "${LOCALE_TAG}" https://github.com/saltstack-formulas/locale-formula.git
+git clone --branch "${NGINX_TAG}" https://github.com/saltstack-formulas/nginx-formula.git
+git clone --branch "${POSTGRES_TAG}" https://github.com/saltstack-formulas/postgres-formula.git
if [ "x${BRANCH}" != "x" ]; then
cd ${F_DIR}/arvados-formula || exit 1
fi
# END FIXME! #16992 Temporary fix for psql call in arvados-api-server
-# If running in a vagrant VM, add default user to docker group
+# Leave a copy of the Arvados CA so the user can copy it where it's required
+echo "Copying the Arvados CA certificate to the installer dir, so you can import it"
+# If running in a vagrant VM, also add default user to docker group
if [ "x${VAGRANT}" = "xyes" ]; then
- usermod -a -G docker vagrant
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem /vagrant
+
+ echo "Adding the vagrant user to the docker group"
+ usermod -a -G docker vagrant
+else
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem ${SCRIPT_DIR}
fi
# Test that the installation finished correctly
tls:
# certificate: ''
# key: ''
- # required to test with snakeoil certs
+ # required to test with arvados-snakeoil certs
insecure: true
### TOKENS
- proxy_set_header: 'X-Real-IP $remote_addr'
- proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- proxy_set_header: 'X-External-Client $external_client'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/__CLUSTER__.__DOMAIN__.error.log
- client_max_body_size: 128m
- client_max_body_size: 64M
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.error.log
- client_max_body_size: 0
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
- add_header: "'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'"
- add_header: "'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'"
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.error.log
- client_max_body_size: 64M
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.error.log
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__"}' ~ "'" }}
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__"}' ~ "'" }}
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
- proxy_set_header: 'Host $http_host'
- proxy_set_header: 'X-Real-IP $remote_addr'
- proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.error.log
export ARVADOS_API_HOST=__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
export ARVADOS_API_HOST_INSECURE=true
+set -o pipefail
+
+# First, validate that the CA is installed and that we can query it with no errors.
+if ! curl -s -o /dev/null https://workbench.${ARVADOS_API_HOST}/users/welcome?return_to=%2F; then
+ echo "The Arvados CA was not correctly installed. Although some components will work,"
+ echo "others won't. Please verify that the CA cert file was installed correctly and"
+ echo "retry running these tests."
+ exit 1
+fi
# https://doc.arvados.org/v2.0/install/install-jobs-image.html
echo "Creating Arvados Standard Docker Images project"
projects / arvados.git / commitdiff