.Rproj.user
_version.py
*.bak
+arvados-snakeoil-ca.pem
+.vagrant
The certificate will be added under the "Arvados testing" organization as "arvbox testing root CA".
-To access your Arvbox instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage (instructions for Debian/Ubuntu):
+To access your Arvbox instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
-# copy @arvbox-root-cert.pem@ to @/usr/local/share/ca-certificates/@
-# run @/usr/sbin/update-ca-certificates@
+h3. On Debian/Ubuntu:
+
+<notextile>
+<pre><code>cp arvbox-root-cert.pem /usr/local/share/ca-certificates/
+/usr/sbin/update-ca-certificates
+</code></pre>
+</notextile>
+
+h3. On CentOS:
+
+<notextile>
+<pre><code>cp arvbox-root-cert.pem /etc/pki/ca-trust/source/anchors/
+/usr/bin/update-ca-trust
+</code></pre>
+</notextile>
h2. Configs
API:
RailsSessionSecretToken: <span class="userinput">"$rails_secret_token"</span>
Collections:
- BlobSigningKey: <span class="userinput">"blob_signing_key"</span>
+ BlobSigningKey: <span class="userinput">"$blob_signing_key"</span>
</code></pre>
</notextile>
-@SystemRootToken@ is used by Arvados system services to authenticate as the system (root) user when communicating with the API server.
+These secret tokens are used to authenticate messages between Arvados components.
+* @SystemRootToken@ is used by Arvados system services to authenticate as the system (root) user when communicating with the API server.
+* @ManagementToken@ is used to authenticate access to system metrics.
+* @API.RailsSessionSecretToken@ is used to sign session cookies.
+* @Collections.BlobSigningKey@ is used to control access to Keep blocks.
-@ManagementToken@ is used to authenticate access to system metrics.
-
-@API.RailsSessionSecretToken@ is required by the API server.
-
-@Collections.BlobSigningKey@ is used to control access to Keep blocks.
-
-You can generate a random token for each of these items at the command line like this:
+Each token should be a string of at least 50 alphanumeric characters. You can generate a suitable token with the following command:
<notextile>
-<pre><code>~$ <span class="userinput">tr -dc 0-9a-zA-Z </dev/urandom | head -c50; echo</span>
+<pre><code>~$ <span class="userinput">tr -dc 0-9a-zA-Z </dev/urandom | head -c50 ; echo</span>
</code></pre>
</notextile>
# "Install Saltstack":#saltstack
# "Single host install using the provision.sh script":#single_host
-# "DNS configuration":#final_steps
+# "Final steps":#final_steps
+## "DNS configuration":#dns_configuration
+## "Install root certificate":#ca_root_certificate
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
</code></pre>
</notextile>
-h2(#final_steps). DNS configuration
+h2(#final_steps). Final configuration steps
+
+h3(#dns_configuration). DNS configuration
After the setup is done, you need to set up your DNS to be able to access the cluster.
</code></pre>
</notextile>
+h3(#ca_root_certificate). Install root certificate
+
+Arvados uses SSL to encrypt communications. Its UI uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
+
+For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so ypu can add it to your workstation.
+
+Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
+
+# Go to the certificate manager in your browser.
+#* In Chrome, this can be found under "Settings → Advanced → Manage Certificates" or by entering @chrome://settings/certificates@ in the URL bar.
+#* In Firefox, this can be found under "Preferences → Privacy & Security" or entering @about:preferences#privacy@ in the URL bar and then choosing "View Certificates...".
+# Select the "Authorities" tab, then press the "Import" button. Choose @arvados-snakeoil-ca.pem@
+
+The certificate will be added under the "Arvados Formula".
+
+To access your Arvados instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
+
+* On Debian/Ubuntu:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /usr/local/share/ca-certificates/
+/usr/sbin/update-ca-certificates
+</code></pre>
+</notextile>
+
+* On CentOS:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /etc/pki/ca-trust/source/anchors/
+/usr/bin/update-ca-trust
+</code></pre>
+</notextile>
+
h2(#initial_user). Initial user and login
At this point you should be able to log into the Arvados cluster.
{% endcomment %}
# "Vagrant":#vagrant
-# "DNS configuration":#final_steps
+# "Final steps":#final_steps
+## "DNS configuration":#dns_configuration
+## "Install root certificate":#ca_root_certificate
# "Initial user and login":#initial_user
# "Test the installed cluster running a simple workflow":#test_install
</code></pre>
</notextile>
-h2(#final_steps). DNS configuration
+h2(#final_steps). Final configuration steps
+
+h3(#dns_configuration). DNS configuration
After the setup is done, you need to set up your DNS to be able to access the cluster.
</code></pre>
</notextile>
+h3(#ca_root_certificate). Install root certificate
+
+Arvados uses SSL to encrypt communications. Its UI uses AJAX which will silently fail if the certificate is not valid or signed by an unknown Certification Authority.
+
+For this reason, the @arvados-formula@ has a helper state to create a root certificate to authorize Arvados services. The @provision.sh@ script will leave a copy of the generated CA's certificate (@arvados-snakeoil-ca.pem@) in the script's directory so ypu can add it to your workstation.
+
+Installing the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.
+
+# Go to the certificate manager in your browser.
+#* In Chrome, this can be found under "Settings → Advanced → Manage Certificates" or by entering @chrome://settings/certificates@ in the URL bar.
+#* In Firefox, this can be found under "Preferences → Privacy & Security" or entering @about:preferences#privacy@ in the URL bar and then choosing "View Certificates...".
+# Select the "Authorities" tab, then press the "Import" button. Choose @arvados-snakeoil-ca.pem@
+
+The certificate will be added under the "Arvados Formula".
+
+To access your Arvados instance using command line clients (such as arv-get and arv-put) without security errors, install the certificate into the OS certificate storage.
+
+* On Debian/Ubuntu:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /usr/local/share/ca-certificates/
+/usr/sbin/update-ca-certificates
+</code></pre>
+</notextile>
+
+* On CentOS:
+
+<notextile>
+<pre><code>cp arvados-root-cert.pem /etc/pki/ca-trust/source/anchors/
+/usr/bin/update-ca-trust
+</code></pre>
+</notextile>
+
h2(#initial_user). Initial user and login
At this point you should be able to log into the Arvados cluster.
Clusters:
xxxxx:
+ # Token used internally by Arvados components to authenticate to
+ # one another. Use a string of at least 50 random alphanumerics.
SystemRootToken: ""
# Token to be included in all healthcheck requests. Disabled by default.
Clusters:
xxxxx:
+ # Token used internally by Arvados components to authenticate to
+ # one another. Use a string of at least 50 random alphanumerics.
SystemRootToken: ""
# Token to be included in all healthcheck requests. Disabled by default.
class CreateSuperUserTokenTest < ActiveSupport::TestCase
include CreateSuperUserToken
- test "create superuser token twice and expect same resutls" do
+ test "create superuser token twice and expect same results" do
# Create a token with some string
token1 = create_superuser_token 'atesttoken'
assert_not_nil token1
- assert_equal token1, 'atesttoken'
+ assert_match(/atesttoken$/, token1)
# Create token again; this time, we should get the one created earlier
token2 = create_superuser_token
# Create a token with some string
token1 = create_superuser_token 'atesttoken'
assert_not_nil token1
- assert_equal token1, 'atesttoken'
+ assert_match(/\/atesttoken$/, token1)
# Create token again with some other string and expect the existing superuser token back
token2 = create_superuser_token 'someothertokenstring'
assert_equal token1, token2
end
- test "create superuser token twice and expect same results" do
- # Create a token with some string
- token1 = create_superuser_token 'atesttoken'
- assert_not_nil token1
- assert_equal token1, 'atesttoken'
-
- # Create token again with that same superuser token and expect it back
- token2 = create_superuser_token 'atesttoken'
- assert_not_nil token2
- assert_equal token1, token2
- end
-
test "create superuser token and invoke again with some other valid token" do
# Create a token with some string
token1 = create_superuser_token 'atesttoken'
assert_not_nil token1
- assert_equal token1, 'atesttoken'
+ assert_match(/\/atesttoken$/, token1)
su_token = api_client_authorizations("system_user").api_token
token2 = create_superuser_token su_token
- assert_equal token2, su_token
+ assert_equal token2.split('/')[2], su_token
end
test "create superuser token, expire it, and create again" do
# Create a token with some string
token1 = create_superuser_token 'atesttoken'
assert_not_nil token1
- assert_equal token1, 'atesttoken'
+ assert_match(/\/atesttoken$/, token1)
# Expire this token and call create again; expect a new token created
- apiClientAuth = ApiClientAuthorization.where(api_token: token1).first
+ apiClientAuth = ApiClientAuthorization.where(api_token: 'atesttoken').first
+ refute_nil apiClientAuth
Thread.current[:user] = users(:admin)
apiClientAuth.update_attributes expires_at: '2000-10-10'
fs.ForwardSlashNameSubstitution(h.Config.cluster.Collections.ForwardSlashNameSubstitution)
var objectNameGiven bool
+ var bucketName string
fspath := "/by_id"
if id := parseCollectionIDFromDNSName(r.Host); id != "" {
fspath += "/" + id
+ bucketName = id
objectNameGiven = strings.Count(strings.TrimSuffix(r.URL.Path, "/"), "/") > 0
} else {
+ bucketName = strings.SplitN(strings.TrimPrefix(r.URL.Path, "/"), "/", 2)[0]
objectNameGiven = strings.Count(strings.TrimSuffix(r.URL.Path, "/"), "/") > 1
}
fspath += r.URL.Path
fmt.Fprintln(w, `<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>`)
} else {
// ListObjects
- h.s3list(w, r, fs)
+ h.s3list(bucketName, w, r, fs)
}
return true
case r.Method == http.MethodGet || r.Method == http.MethodHead:
var errDone = errors.New("done")
-func (h *handler) s3list(w http.ResponseWriter, r *http.Request, fs arvados.CustomFileSystem) {
+func (h *handler) s3list(bucket string, w http.ResponseWriter, r *http.Request, fs arvados.CustomFileSystem) {
var params struct {
- bucket string
delimiter string
marker string
maxKeys int
prefix string
}
- params.bucket = strings.SplitN(r.URL.Path[1:], "/", 2)[0]
params.delimiter = r.FormValue("delimiter")
params.marker = r.FormValue("marker")
if mk, _ := strconv.ParseInt(r.FormValue("max-keys"), 10, 64); mk > 0 && mk < s3MaxKeys {
}
params.prefix = r.FormValue("prefix")
- bucketdir := "by_id/" + params.bucket
+ bucketdir := "by_id/" + bucket
// walkpath is the directory (relative to bucketdir) we need
// to walk: the innermost directory that is guaranteed to
// contain all paths that have the requested prefix. Examples:
}
resp := listResp{
ListResp: s3.ListResp{
- Name: strings.SplitN(r.URL.Path[1:], "/", 2)[0],
+ Name: bucket,
Prefix: params.prefix,
Delimiter: params.delimiter,
Marker: params.marker,
"fmt"
"io/ioutil"
"net/http"
+ "net/http/httptest"
"net/url"
"os"
"os/exec"
c.Assert(fs.Sync(), check.IsNil)
}
+func (s *IntegrationSuite) TestS3VirtualHostStyleRequests(c *check.C) {
+ stage := s.s3setup(c)
+ defer stage.teardown(c)
+ for _, trial := range []struct {
+ url string
+ method string
+ body string
+ responseCode int
+ responseRegexp []string
+ }{
+ {
+ url: "https://" + stage.collbucket.Name + ".example.com/",
+ method: "GET",
+ responseCode: http.StatusOK,
+ responseRegexp: []string{`(?ms).*sailboat\.txt.*`},
+ },
+ {
+ url: "https://" + strings.Replace(stage.coll.PortableDataHash, "+", "-", -1) + ".example.com/",
+ method: "GET",
+ responseCode: http.StatusOK,
+ responseRegexp: []string{`(?ms).*sailboat\.txt.*`},
+ },
+ {
+ url: "https://" + stage.projbucket.Name + ".example.com/?prefix=" + stage.coll.Name + "/&delimiter=/",
+ method: "GET",
+ responseCode: http.StatusOK,
+ responseRegexp: []string{`(?ms).*sailboat\.txt.*`},
+ },
+ {
+ url: "https://" + stage.projbucket.Name + ".example.com/" + stage.coll.Name + "/sailboat.txt",
+ method: "GET",
+ responseCode: http.StatusOK,
+ responseRegexp: []string{`⛵\n`},
+ },
+ {
+ url: "https://" + stage.projbucket.Name + ".example.com/" + stage.coll.Name + "/beep",
+ method: "PUT",
+ body: "boop",
+ responseCode: http.StatusOK,
+ },
+ {
+ url: "https://" + stage.projbucket.Name + ".example.com/" + stage.coll.Name + "/beep",
+ method: "GET",
+ responseCode: http.StatusOK,
+ responseRegexp: []string{`boop`},
+ },
+ } {
+ url, err := url.Parse(trial.url)
+ c.Assert(err, check.IsNil)
+ req, err := http.NewRequest(trial.method, url.String(), bytes.NewReader([]byte(trial.body)))
+ c.Assert(err, check.IsNil)
+ req.Header.Set("Authorization", "AWS "+arvadostest.ActiveTokenV2+":none")
+ rr := httptest.NewRecorder()
+ s.testServer.Server.Handler.ServeHTTP(rr, req)
+ resp := rr.Result()
+ c.Check(resp.StatusCode, check.Equals, trial.responseCode)
+ body, err := ioutil.ReadAll(resp.Body)
+ c.Assert(err, check.IsNil)
+ for _, re := range trial.responseRegexp {
+ c.Check(string(body), check.Matches, re)
+ }
+ }
+}
+
func (s *IntegrationSuite) TestS3GetBucketVersioning(c *check.C) {
stage := s.s3setup(c)
defer stage.teardown(c)
arv.vm.provision "shell",
path: "provision.sh",
args: [
+ # "--debug",
"--test",
"--vagrant",
"--ssl-port=8443"
-#!/bin/bash
+#!/bin/bash
# Copyright (C) The Arvados Authors. All rights reserved.
#
##########################################################
# Usually there's no need to modify things below this line
+# Formulas versions
+ARVADOS_TAG="v1.1.3"
+POSTGRES_TAG="v0.41.3"
+NGINX_TAG="v2.4.0"
+DOCKER_TAG="v1.0.0"
+LOCALE_TAG="v0.3.4"
+
set -o pipefail
# capture the directory that the script is running from
base:
- ${S_DIR}
- ${F_DIR}/*
- - ${F_DIR}/*/test/salt/states
+ - ${F_DIR}/*/test/salt/states/examples
pillar_roots:
base:
cat > ${S_DIR}/top.sls << EOFTSLS
base:
'*':
- - example_single_host_host_entries
- - example_add_snakeoil_certs
+ - single_host.host_entries
+ - single_host.snakeoil_certs
- locale
- nginx.passenger
- postgres
- postgresql
EOFPSLS
-
# Get the formula and dependencies
cd ${F_DIR} || exit 1
-for f in postgres arvados nginx docker locale; do
- git clone https://github.com/saltstack-formulas/${f}-formula.git
-done
+git clone --branch "${ARVADOS_TAG}" https://github.com/saltstack-formulas/arvados-formula.git
+git clone --branch "${DOCKER_TAG}" https://github.com/saltstack-formulas/docker-formula.git
+git clone --branch "${LOCALE_TAG}" https://github.com/saltstack-formulas/locale-formula.git
+git clone --branch "${NGINX_TAG}" https://github.com/saltstack-formulas/nginx-formula.git
+git clone --branch "${POSTGRES_TAG}" https://github.com/saltstack-formulas/postgres-formula.git
if [ "x${BRANCH}" != "x" ]; then
cd ${F_DIR}/arvados-formula || exit 1
fi
# END FIXME! #16992 Temporary fix for psql call in arvados-api-server
-# If running in a vagrant VM, add default user to docker group
+# Leave a copy of the Arvados CA so the user can copy it where it's required
+echo "Copying the Arvados CA certificate to the installer dir, so you can import it"
+# If running in a vagrant VM, also add default user to docker group
if [ "x${VAGRANT}" = "xyes" ]; then
- usermod -a -G docker vagrant
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem /vagrant
+
+ echo "Adding the vagrant user to the docker group"
+ usermod -a -G docker vagrant
+else
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem ${SCRIPT_DIR}
fi
# Test that the installation finished correctly
tls:
# certificate: ''
# key: ''
- # required to test with snakeoil certs
+ # required to test with arvados-snakeoil certs
insecure: true
### TOKENS
- proxy_set_header: 'X-Real-IP $remote_addr'
- proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- proxy_set_header: 'X-External-Client $external_client'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/__CLUSTER__.__DOMAIN__.error.log
- client_max_body_size: 128m
- client_max_body_size: 64M
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.error.log
- client_max_body_size: 0
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
- add_header: "'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'"
- add_header: "'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'"
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.error.log
- client_max_body_size: 64M
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.error.log
- 'if (-f $document_root/maintenance.html)':
- return: 503
- location /config.json:
- - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__"}' ~ "'" }}
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__"}' ~ "'" }}
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
- proxy_set_header: 'Host $http_host'
- proxy_set_header: 'X-Real-IP $remote_addr'
- proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- # - include: 'snippets/letsencrypt.conf'
- - include: 'snippets/snakeoil.conf'
+ - include: 'snippets/arvados-snakeoil.conf'
- access_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.error.log
export ARVADOS_API_HOST=__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
export ARVADOS_API_HOST_INSECURE=true
+set -o pipefail
+
+# First, validate that the CA is installed and that we can query it with no errors.
+if ! curl -s -o /dev/null https://workbench.${ARVADOS_API_HOST}/users/welcome?return_to=%2F; then
+ echo "The Arvados CA was not correctly installed. Although some components will work,"
+ echo "others won't. Please verify that the CA cert file was installed correctly and"
+ echo "retry running these tests."
+ exit 1
+fi
# https://doc.arvados.org/v2.0/install/install-jobs-image.html
echo "Creating Arvados Standard Docker Images project"
projects / arvados.git / commitdiff