## even though credentials are already in .dockercfg
docker login -u arvados
- docker_push arvados/arvbox-dev
- docker_push arvados/arvbox-demo
+ if [[ "$images" =~ dev ]]; then
+ docker_push arvados/arvbox-dev
+ fi
+ if [[ "$images" =~ demo ]]; then
+ docker_push arvados/arvbox-demo
+ fi
title "upload arvados images complete (`timer`)"
else
title "upload arvados images SKIPPED because no --upload option set"
- install/index.html.textile.liquid
- Docker quick start:
- install/arvbox.html.textile.liquid
+ - Installation with Salt:
+ - install/salt.html.textile.liquid
+ - install/salt-vagrant.html.textile.liquid
+ - install/salt-single-host.html.textile.liquid
+ - install/salt-multi-host.html.textile.liquid
- Arvados on Kubernetes:
- install/arvados-on-kubernetes.html.textile.liquid
- install/arvados-on-kubernetes-minikube.html.textile.liquid
<div class="offset1">
table(table table-bordered table-condensed).
|||\5=. Appropriate for|
-||_. Ease of setup|_. Multiuser/networked access|_. Workflow Development and Testing|_. Large Scale Production|_. Development of Arvados|_. Arvados Evaluation|
+||_. Setup difficulty|_. Multiuser/networked access|_. Workflow Development and Testing|_. Large Scale Production|_. Development of Arvados|_. Arvados Evaluation|
|"Arvados-in-a-box":arvbox.html (arvbox)|Easy|no|yes|no|yes|yes|
+|"Installation with Salt":salt-single-host.html (single host)|Easy|no|yes|no|yes|yes|
+|"Installation with Salt":salt-multi-host.html (multi host)|Moderate|yes|yes|yes|yes|yes|
|"Arvados on Kubernetes":arvados-on-kubernetes.html|Easy ^1^|yes|yes ^2^|no ^2^|no|yes|
-|"Manual installation":install-manual-prerequisites.html|Complicated|yes|yes|yes|no|no|
+|"Manual installation":install-manual-prerequisites.html|Hard|yes|yes|yes|no|no|
|"Cluster Operation Subscription supported by Curii":mailto:info@curii.com|N/A ^3^|yes|yes|yes|yes|yes|
</div>
--- /dev/null
+---
+layout: default
+navsection: installguide
+title: Multi host Arvados
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+# "Install Saltstack":#saltstack
+# "Install dependencies":#dependencies
+# "Install Arvados using Saltstack":#saltstack
+# "DNS configuration":#final_steps
+# "Initial user and login":#initial_user
+
+h2(#saltstack). Install Saltstack
+
+If you already have a Saltstack environment you can skip this section.
+
+The simplest way to get Salt up and running on a node is to use the bootstrap script they provide:
+
+<notextile>
+<pre><code>curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
+sudo sh /tmp/bootstrap_salt.sh -XUdfP -x python3
+</code></pre>
+</notextile>
+
+For more information check "Saltstack's documentation":https://docs.saltstack.com/en/latest/topics/installation/index.html
+
+h2(#dependencies). Install dependencies
+
+Arvados depends in a few applications and packages (postgresql, nginx+passenger, ruby) that can also be installed using their respective Saltstack formulas.
+
+The formulas we use are:
+
+* "postgres":https://github.com/saltstack-formulas/postgres-formula.git
+* "nginx":https://github.com/saltstack-formulas/nginx-formula.git
+* "docker":https://github.com/saltstack-formulas/docker-formula.git
+* "locale":https://github.com/saltstack-formulas/locale-formula.git
+
+There are example Salt pillar files for each of those formulas in the "arvados-formula's test/salt/pillar/examples":https://github.com/saltstack-formulas/arvados-formula/tree/master/test/salt/pillar/examples directory. As they are, they allow you to get all the main Arvados components up and running.
+
+h2(#saltstack). Install Arvados using Saltstack
+
+This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+
+The Arvados formula we maintain is located in the Saltstack's community repository of formulas:
+
+* "arvados-formula":https://github.com/saltstack-formulas/arvados-formula.git
+
+The @development@ version lives in our own repository
+
+* "arvados-formula development":https://github.com/arvados/arvados-formula.git
+
+This last one might break from time to time, as we try and add new features. Use with caution.
+
+As much as possible, we try to keep it up to date, with example pillars to help you deploy Arvados.
+
+For those familiar with Saltstack, the process to get it deployed is similar to any other formula:
+
+1. Fork/copy the formula to your Salt master host.
+2. Edit the Arvados, nginx, postgres, locale and docker pillars to match your desired configuration.
+3. Run a @state.apply@ to get it deployed.
+
+h2(#final_steps). DNS configuration
+
+After the setup is done, you need to set up your DNS to be able to access the cluster's nodes.
+
+The simplest way to do this is to add entries in the @/etc/hosts@ file of every host:
+
+<notextile>
+<pre><code>export CLUSTER="arva2"
+export DOMAIN="arv.local"
+
+echo A.B.C.a api ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} >> /etc/hosts
+echo A.B.C.b keep keep.${CLUSTER}.${DOMAIN} >> /etc/hosts
+echo A.B.C.c keep0 keep0.${CLUSTER}.${DOMAIN} >> /etc/hosts
+echo A.B.C.d collections collections.${CLUSTER}.${DOMAIN} >> /etc/hosts
+echo A.B.C.e download download.${CLUSTER}.${DOMAIN} >> /etc/hosts
+echo A.B.C.f ws ws.${CLUSTER}.${DOMAIN} >> /etc/hosts
+echo A.B.C.g workbench workbench.${CLUSTER}.${DOMAIN} >> /etc/hosts
+echo A.B.C.h workbench2 workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+</code></pre>
+</notextile>
+
+Replacing in each case de @A.B.C.x@ IP with the corresponding IP of the node.
+
+If your infrastructure uses another DNS service setup, add the corresponding entries accordingly.
+
+h2(#initial_user). Initial user and login
+
+At this point you should be able to log into the Arvados cluster.
+
+If you did not change the defaults, the initial URL will be:
+
+* https://workbench.arva2.arv.local
+
+or, in general, the url format will be:
+
+* https://workbench.@<cluster>.<domain>@
+
+By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
+
+Assuming you didn't change the defaults, the initial credentials are:
+
+* User: 'admin'
+* Password: 'password'
+* Email: 'admin@arva2.arv.local'
--- /dev/null
+---
+layout: default
+navsection: installguide
+title: Single host Arvados
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+# "Install Saltstack":#saltstack
+# "Single host install using the provision.sh script":#single_host
+# "Local testing Arvados in a Vagrant box":#vagrant
+# "DNS configuration":#final_steps
+# "Initial user and login":#initial_user
+
+h2(#saltstack). Install Saltstack
+
+If you already have a Saltstack environment you can skip this section.
+
+The simplest way to get Salt up and running on a node is to use the bootstrap script they provide:
+
+<notextile>
+<pre><code>curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
+sudo sh /tmp/bootstrap_salt.sh -XUdfP -x python3
+</code></pre>
+</notextile>
+
+For more information check "Saltstack's documentation":https://docs.saltstack.com/en/latest/topics/installation/index.html
+
+h2(#single_host). Single host install using the provision.sh script
+
+This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+
+Use the @provision.sh@ script to deploy Arvados, which is implemented with the @arvados-formula@ in a Saltstack master-less setup:
+
+* edit the variables at the very beginning of the file,
+* run the script as root
+* wait for it to finish
+
+This will install all the main Arvados components to get you up and running. The whole installation procedure takes somewhere between 15 to 60 minutes, depending on the host and your network bandwidth. On a virtual machine with 1 core and 1 GB RAM, it takes ~25 minutes to do the initial install.
+
+If everything goes OK, you'll get some final lines stating something like:
+
+<notextile>
+<pre><code>arvados: Succeeded: 109 (changed=9)
+arvados: Failed: 0
+</code></pre>
+</notextile>
+
+h2(#final_steps). DNS configuration
+
+After the setup is done, you need to set up your DNS to be able to access the cluster.
+
+The simplest way to do this is to edit your @/etc/hosts@ file (as root):
+
+<notextile>
+<pre><code>export CLUSTER="arva2"
+export DOMAIN="arv.local"
+export HOST_IP="127.0.0.2" # This is valid either if installing in your computer directly
+ # or in a Vagrant VM. If you're installing it on a remote host
+ # just change the IP to match that of the host.
+echo "${HOST_IP} api keep keep0 collections download ws workbench workbench2 ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} keep.${CLUSTER}.${DOMAIN} keep0.${CLUSTER}.${DOMAIN} collections.${CLUSTER}.${DOMAIN} download.${CLUSTER}.${DOMAIN} ws.${CLUSTER}.${DOMAIN} workbench.${CLUSTER}.${DOMAIN} workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+</code></pre>
+</notextile>
+
+h2(#initial_user). Initial user and login
+
+At this point you should be able to log into the Arvados cluster.
+
+If you changed nothing in the @provision.sh@ script, the initial URL will be:
+
+* https://workbench.arva2.arv.local
+
+or, in general, the url format will be:
+
+* https://workbench.@<cluster>.<domain>@
+
+By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
+
+Assuming you didn't change these values in the @provision.sh@ script, the initial credentials are:
+
+* User: 'admin'
+* Password: 'password'
+* Email: 'admin@arva2.arv.local'
--- /dev/null
+---
+layout: default
+navsection: installguide
+title: Arvados in a VM with Vagrant
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+# "Vagrant":#vagrant
+# "DNS configuration":#final_steps
+# "Initial user and login":#initial_user
+
+h2(#vagrant). Vagrant
+
+This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+
+A @Vagrantfile@ is provided to install Arvados in a virtual machine on your computer using "Vagrant":https://www.vagrantup.com/.
+
+To get it running, install Vagrant in your computer, edit the variables at the top of the @provision.sh@ script as needed, and run
+
+<notextile>
+<pre><code>vagrant up
+</code></pre>
+</notextile>
+
+If you want to reconfigure the running box, you can just:
+
+1. edit the pillars to suit your needs
+2. run
+
+<notextile>
+<pre><code>vagrant reload --provision
+</code></pre>
+</notextile>
+
+h2(#final_steps). DNS configuration
+
+After the setup is done, you need to set up your DNS to be able to access the cluster.
+
+The simplest way to do this is to edit your @/etc/hosts@ file (as root):
+
+<notextile>
+<pre><code>export CLUSTER="arva2"
+export DOMAIN="arv.local"
+export HOST_IP="127.0.0.2" # This is valid either if installing in your computer directly
+ # or in a Vagrant VM. If you're installing it on a remote host
+ # just change the IP to match that of the host.
+echo "${HOST_IP} api keep keep0 collections download ws workbench workbench2 ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} keep.${CLUSTER}.${DOMAIN} keep0.${CLUSTER}.${DOMAIN} collections.${CLUSTER}.${DOMAIN} download.${CLUSTER}.${DOMAIN} ws.${CLUSTER}.${DOMAIN} workbench.${CLUSTER}.${DOMAIN} workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+</code></pre>
+</notextile>
+
+h2(#initial_user). Initial user and login
+
+At this point you should be able to log into the Arvados cluster.
+
+If you didn't change the defaults, the initial URL will be:
+
+* https://workbench.arva2.arv.local:8443
+
+or, in general, the url format will be:
+
+* https://workbench.@<cluster>.<domain>:8443@
+
+By default, the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.
+
+Assuming you didn't change the defaults, the initial credentials are:
+
+* User: 'admin'
+* Password: 'password'
+* Email: 'admin@arva2.arv.local'
--- /dev/null
+---
+layout: default
+navsection: installguide
+title: Salt prerequisites
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+# "Introduction":#introduction
+# "Choose an installation method":#installmethod
+
+h2(#introduction). Introduction
+
+To ease the installation of the various Arvados components, we have developed a "Saltstack":https://www.saltstack.com/ 's "arvados-formula":https://github.com/saltstack-formulas/arvados-formula which can help you get an Arvados cluster up and running.
+
+Saltstack is a Python-based, open-source software for event-driven IT automation, remote task execution, and configuration management. It can be used in a master/minion setup or master-less.
+
+This is a package-based installation method. The Salt scripts are available from the "tools/salt-install":https://github.com/arvados/arvados/tree/master/tools/salt-install directory in the Arvados git repository.
+
+h2(#installmethod). Choose an installation method
+
+The salt formulas can be used in different ways. Choose one of these three options to install Arvados:
+
+* "Use Vagrant to install Arvados in a virtual machine":salt-vagrant.html
+* "Arvados on a single host":salt-single-host.html
+* "Arvados across multiple hosts":salt-multi-host.html
// it to the router package would cause a circular dependency
// router->arvadostest->ctrlctx->router.)
type RoutableFunc func(ctx context.Context, opts interface{}) (interface{}, error)
+
+type RoutableFuncWrapper func(RoutableFunc) RoutableFunc
+
+// ComposeWrappers(w1, w2, w3, ...) returns a RoutableFuncWrapper that
+// composes w1, w2, w3, ... such that w1 is the outermost wrapper.
+func ComposeWrappers(wraps ...RoutableFuncWrapper) RoutableFuncWrapper {
+ return func(f RoutableFunc) RoutableFunc {
+ for i := len(wraps) - 1; i >= 0; i-- {
+ f = wraps[i](f)
+ }
+ return f
+ }
+}
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: AGPL-3.0
+
+package controller
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "time"
+
+ "git.arvados.org/arvados.git/sdk/go/arvados"
+ "git.arvados.org/arvados.git/sdk/go/arvadostest"
+ "git.arvados.org/arvados.git/sdk/go/ctxlog"
+ "git.arvados.org/arvados.git/sdk/go/httpserver"
+ "github.com/sirupsen/logrus"
+ check "gopkg.in/check.v1"
+)
+
+// Gocheck boilerplate
+var _ = check.Suite(&AuthSuite{})
+
+type AuthSuite struct {
+ log logrus.FieldLogger
+ // testServer and testHandler are the controller being tested,
+ // "zhome".
+ testServer *httpserver.Server
+ testHandler *Handler
+ // remoteServer ("zzzzz") forwards requests to the Rails API
+ // provided by the integration test environment.
+ remoteServer *httpserver.Server
+ // remoteMock ("zmock") appends each incoming request to
+ // remoteMockRequests, and returns 200 with an empty JSON
+ // object.
+ remoteMock *httpserver.Server
+ remoteMockRequests []http.Request
+
+ fakeProvider *arvadostest.OIDCProvider
+}
+
+func (s *AuthSuite) SetUpTest(c *check.C) {
+ s.log = ctxlog.TestLogger(c)
+
+ s.remoteServer = newServerFromIntegrationTestEnv(c)
+ c.Assert(s.remoteServer.Start(), check.IsNil)
+
+ s.remoteMock = newServerFromIntegrationTestEnv(c)
+ s.remoteMock.Server.Handler = http.HandlerFunc(http.NotFound)
+ c.Assert(s.remoteMock.Start(), check.IsNil)
+
+ s.fakeProvider = arvadostest.NewOIDCProvider(c)
+ s.fakeProvider.AuthEmail = "active-user@arvados.local"
+ s.fakeProvider.AuthEmailVerified = true
+ s.fakeProvider.AuthName = "Fake User Name"
+ s.fakeProvider.ValidCode = fmt.Sprintf("abcdefgh-%d", time.Now().Unix())
+ s.fakeProvider.PeopleAPIResponse = map[string]interface{}{}
+ s.fakeProvider.ValidClientID = "test%client$id"
+ s.fakeProvider.ValidClientSecret = "test#client/secret"
+
+ cluster := &arvados.Cluster{
+ ClusterID: "zhome",
+ PostgreSQL: integrationTestCluster().PostgreSQL,
+ ForceLegacyAPI14: forceLegacyAPI14,
+ SystemRootToken: arvadostest.SystemRootToken,
+ }
+ cluster.TLS.Insecure = true
+ cluster.API.MaxItemsPerResponse = 1000
+ cluster.API.MaxRequestAmplification = 4
+ cluster.API.RequestTimeout = arvados.Duration(5 * time.Minute)
+ arvadostest.SetServiceURL(&cluster.Services.RailsAPI, "https://"+os.Getenv("ARVADOS_TEST_API_HOST"))
+ arvadostest.SetServiceURL(&cluster.Services.Controller, "http://localhost/")
+
+ cluster.RemoteClusters = map[string]arvados.RemoteCluster{
+ "zzzzz": {
+ Host: s.remoteServer.Addr,
+ Proxy: true,
+ Scheme: "http",
+ },
+ "zmock": {
+ Host: s.remoteMock.Addr,
+ Proxy: true,
+ Scheme: "http",
+ },
+ "*": {
+ Scheme: "https",
+ },
+ }
+ cluster.Login.OpenIDConnect.Enable = true
+ cluster.Login.OpenIDConnect.Issuer = s.fakeProvider.Issuer.URL
+ cluster.Login.OpenIDConnect.ClientID = s.fakeProvider.ValidClientID
+ cluster.Login.OpenIDConnect.ClientSecret = s.fakeProvider.ValidClientSecret
+ cluster.Login.OpenIDConnect.EmailClaim = "email"
+ cluster.Login.OpenIDConnect.EmailVerifiedClaim = "email_verified"
+
+ s.testHandler = &Handler{Cluster: cluster}
+ s.testServer = newServerFromIntegrationTestEnv(c)
+ s.testServer.Server.Handler = httpserver.HandlerWithContext(
+ ctxlog.Context(context.Background(), s.log),
+ httpserver.AddRequestIDs(httpserver.LogRequests(s.testHandler)))
+ c.Assert(s.testServer.Start(), check.IsNil)
+}
+
+func (s *AuthSuite) TestLocalOIDCAccessToken(c *check.C) {
+ req := httptest.NewRequest("GET", "/arvados/v1/users/current", nil)
+ req.Header.Set("Authorization", "Bearer "+s.fakeProvider.ValidAccessToken())
+ rr := httptest.NewRecorder()
+ s.testServer.Server.Handler.ServeHTTP(rr, req)
+ resp := rr.Result()
+ c.Check(resp.StatusCode, check.Equals, http.StatusOK)
+ var u arvados.User
+ c.Check(json.NewDecoder(resp.Body).Decode(&u), check.IsNil)
+ c.Check(u.UUID, check.Equals, arvadostest.ActiveUserUUID)
+ c.Check(u.OwnerUUID, check.Equals, "zzzzz-tpzed-000000000000000")
+
+ // Request again to exercise cache.
+ req = httptest.NewRequest("GET", "/arvados/v1/users/current", nil)
+ req.Header.Set("Authorization", "Bearer "+s.fakeProvider.ValidAccessToken())
+ rr = httptest.NewRecorder()
+ s.testServer.Server.Handler.ServeHTTP(rr, req)
+ resp = rr.Result()
+ c.Check(resp.StatusCode, check.Equals, http.StatusOK)
+}
return updatedReq, nil
}
+ ctxlog.FromContext(req.Context()).Infof("saltAuthToken: cluster %s token %s remote %s", h.Cluster.ClusterID, creds.Tokens[0], remote)
token, err := auth.SaltToken(creds.Tokens[0], remote)
if err == auth.ErrObsoleteToken {
- // If the token exists in our own database, salt it
- // for the remote. Otherwise, assume it was issued by
- // the remote, and pass it through unmodified.
+ // If the token exists in our own database for our own
+ // user, salt it for the remote. Otherwise, assume it
+ // was issued by the remote, and pass it through
+ // unmodified.
currentUser, ok, err := h.validateAPItoken(req, creds.Tokens[0])
if err != nil {
return nil, err
- } else if !ok {
- // Not ours; pass through unmodified.
+ } else if !ok || strings.HasPrefix(currentUser.UUID, remote) {
+ // Unknown, or cached + belongs to remote;
+ // pass through unmodified.
token = creds.Tokens[0]
} else {
// Found; make V2 version and salt it.
} else if err != nil {
return nil, err
}
+ if strings.HasPrefix(aca.UUID, remoteID) {
+ // We have it cached here, but
+ // the token belongs to the
+ // remote target itself, so
+ // pass it through unmodified.
+ tokens = append(tokens, token)
+ continue
+ }
salted, err := auth.SaltToken(aca.TokenV2(), remoteID)
if err != nil {
return nil, err
"sync"
"time"
+ "git.arvados.org/arvados.git/lib/controller/api"
"git.arvados.org/arvados.git/lib/controller/federation"
+ "git.arvados.org/arvados.git/lib/controller/localdb"
"git.arvados.org/arvados.git/lib/controller/railsproxy"
"git.arvados.org/arvados.git/lib/controller/router"
"git.arvados.org/arvados.git/lib/ctrlctx"
Routes: health.Routes{"ping": func() error { _, err := h.db(context.TODO()); return err }},
})
- rtr := router.New(federation.New(h.Cluster), ctrlctx.WrapCallsInTransactions(h.db))
+ oidcAuthorizer := localdb.OIDCAccessTokenAuthorizer(h.Cluster, h.db)
+ rtr := router.New(federation.New(h.Cluster), api.ComposeWrappers(ctrlctx.WrapCallsInTransactions(h.db), oidcAuthorizer.WrapCalls))
mux.Handle("/arvados/v1/config", rtr)
mux.Handle("/"+arvados.EndpointUserAuthenticate.Path, rtr)
hs := http.NotFoundHandler()
hs = prepend(hs, h.proxyRailsAPI)
hs = h.setupProxyRemoteCluster(hs)
+ hs = prepend(hs, oidcAuthorizer.Middleware)
mux.Handle("/", hs)
h.handlerStack = mux
"context"
"encoding/json"
"io"
+ "io/ioutil"
"math"
"net"
"net/http"
"git.arvados.org/arvados.git/lib/service"
"git.arvados.org/arvados.git/sdk/go/arvados"
"git.arvados.org/arvados.git/sdk/go/arvadosclient"
+ "git.arvados.org/arvados.git/sdk/go/arvadostest"
"git.arvados.org/arvados.git/sdk/go/auth"
"git.arvados.org/arvados.git/sdk/go/ctxlog"
"git.arvados.org/arvados.git/sdk/go/keepclient"
type IntegrationSuite struct {
testClusters map[string]*testCluster
+ oidcprovider *arvadostest.OIDCProvider
}
func (s *IntegrationSuite) SetUpSuite(c *check.C) {
}
cwd, _ := os.Getwd()
+
+ s.oidcprovider = arvadostest.NewOIDCProvider(c)
+ s.oidcprovider.AuthEmail = "user@example.com"
+ s.oidcprovider.AuthEmailVerified = true
+ s.oidcprovider.AuthName = "Example User"
+ s.oidcprovider.ValidClientID = "clientid"
+ s.oidcprovider.ValidClientSecret = "clientsecret"
+
s.testClusters = map[string]*testCluster{
"z1111": nil,
"z2222": nil,
ActivateUsers: true
`
}
+ if id == "z1111" {
+ yaml += `
+ Login:
+ LoginCluster: z1111
+ OpenIDConnect:
+ Enable: true
+ Issuer: ` + s.oidcprovider.Issuer.URL + `
+ ClientID: ` + s.oidcprovider.ValidClientID + `
+ ClientSecret: ` + s.oidcprovider.ValidClientSecret + `
+ EmailClaim: email
+ EmailVerifiedClaim: email_verified
+`
+ } else {
+ yaml += `
+ Login:
+ LoginCluster: z1111
+`
+ }
loader := config.NewLoader(bytes.NewBufferString(yaml), ctxlog.TestLogger(c))
loader.Path = "-"
c.Check(len(outLinks.Items), check.Equals, 1)
}
+
+func (s *IntegrationSuite) TestOIDCAccessTokenAuth(c *check.C) {
+ conn1 := s.conn("z1111")
+ rootctx1, _, _ := s.rootClients("z1111")
+ s.userClients(rootctx1, c, conn1, "z1111", true)
+
+ accesstoken := s.oidcprovider.ValidAccessToken()
+
+ for _, clusterid := range []string{"z1111", "z2222"} {
+ c.Logf("trying clusterid %s", clusterid)
+
+ conn := s.conn(clusterid)
+ ctx, ac, kc := s.clientsWithToken(clusterid, accesstoken)
+
+ var coll arvados.Collection
+
+ // Write some file data and create a collection
+ {
+ fs, err := coll.FileSystem(ac, kc)
+ c.Assert(err, check.IsNil)
+ f, err := fs.OpenFile("test.txt", os.O_CREATE|os.O_RDWR, 0777)
+ c.Assert(err, check.IsNil)
+ _, err = io.WriteString(f, "IntegrationSuite.TestOIDCAccessTokenAuth")
+ c.Assert(err, check.IsNil)
+ err = f.Close()
+ c.Assert(err, check.IsNil)
+ mtxt, err := fs.MarshalManifest(".")
+ c.Assert(err, check.IsNil)
+ coll, err = conn.CollectionCreate(ctx, arvados.CreateOptions{Attrs: map[string]interface{}{
+ "manifest_text": mtxt,
+ }})
+ c.Assert(err, check.IsNil)
+ }
+
+ // Read the collection & file data
+ {
+ user, err := conn.UserGetCurrent(ctx, arvados.GetOptions{})
+ c.Assert(err, check.IsNil)
+ c.Check(user.FullName, check.Equals, "Example User")
+ coll, err = conn.CollectionGet(ctx, arvados.GetOptions{UUID: coll.UUID})
+ c.Assert(err, check.IsNil)
+ c.Check(coll.ManifestText, check.Not(check.Equals), "")
+ fs, err := coll.FileSystem(ac, kc)
+ c.Assert(err, check.IsNil)
+ f, err := fs.Open("test.txt")
+ c.Assert(err, check.IsNil)
+ buf, err := ioutil.ReadAll(f)
+ c.Assert(err, check.IsNil)
+ c.Check(buf, check.DeepEquals, []byte("IntegrationSuite.TestOIDCAccessTokenAuth"))
+ }
+ }
+}
"context"
"crypto/hmac"
"crypto/sha256"
+ "database/sql"
"encoding/base64"
"errors"
"fmt"
+ "io"
"net/http"
"net/url"
"strings"
"text/template"
"time"
+ "git.arvados.org/arvados.git/lib/controller/api"
+ "git.arvados.org/arvados.git/lib/controller/railsproxy"
"git.arvados.org/arvados.git/lib/controller/rpc"
+ "git.arvados.org/arvados.git/lib/ctrlctx"
"git.arvados.org/arvados.git/sdk/go/arvados"
"git.arvados.org/arvados.git/sdk/go/auth"
"git.arvados.org/arvados.git/sdk/go/ctxlog"
"git.arvados.org/arvados.git/sdk/go/httpserver"
"github.com/coreos/go-oidc"
+ lru "github.com/hashicorp/golang-lru"
+ "github.com/jmoiron/sqlx"
+ "github.com/sirupsen/logrus"
"golang.org/x/oauth2"
"google.golang.org/api/option"
"google.golang.org/api/people/v1"
)
+const (
+ tokenCacheSize = 1000
+ tokenCacheNegativeTTL = time.Minute * 5
+ tokenCacheTTL = time.Minute * 10
+)
+
type oidcLoginController struct {
Cluster *arvados.Cluster
RailsProxy *railsProxy
return arvados.APIClientAuthorization{}, httpserver.ErrorWithStatus(errors.New("username/password authentication is not available"), http.StatusBadRequest)
}
+// claimser can decode arbitrary claims into a map. Implemented by
+// *oauth2.IDToken and *oauth2.UserInfo.
+type claimser interface {
+ Claims(interface{}) error
+}
+
// Use a person's token to get all of their email addresses, with the
// primary address at index 0. The provided defaultAddr is always
// included in the returned slice, and is used as the primary if the
// Google API does not indicate one.
-func (ctrl *oidcLoginController) getAuthInfo(ctx context.Context, token *oauth2.Token, idToken *oidc.IDToken) (*rpc.UserSessionAuthInfo, error) {
+func (ctrl *oidcLoginController) getAuthInfo(ctx context.Context, token *oauth2.Token, claimser claimser) (*rpc.UserSessionAuthInfo, error) {
var ret rpc.UserSessionAuthInfo
defer ctxlog.FromContext(ctx).WithField("ret", &ret).Debug("getAuthInfo returned")
var claims map[string]interface{}
- if err := idToken.Claims(&claims); err != nil {
- return nil, fmt.Errorf("error extracting claims from ID token: %s", err)
+ if err := claimser.Claims(&claims); err != nil {
+ return nil, fmt.Errorf("error extracting claims from token: %s", err)
} else if verified, _ := claims[ctrl.EmailVerifiedClaim].(bool); verified || ctrl.EmailVerifiedClaim == "" {
// Fall back to this info if the People API call
// (below) doesn't return a primary && verified email.
fmt.Fprintf(mac, "%x %s %s", s.Time, s.Remote, s.ReturnTo)
return mac.Sum(nil)
}
+
+func OIDCAccessTokenAuthorizer(cluster *arvados.Cluster, getdb func(context.Context) (*sqlx.DB, error)) *oidcTokenAuthorizer {
+ // We want ctrl to be nil if the chosen controller is not a
+ // *oidcLoginController, so we can ignore the 2nd return value
+ // of this type cast.
+ ctrl, _ := chooseLoginController(cluster, railsproxy.NewConn(cluster)).(*oidcLoginController)
+ cache, err := lru.New2Q(tokenCacheSize)
+ if err != nil {
+ panic(err)
+ }
+ return &oidcTokenAuthorizer{
+ ctrl: ctrl,
+ getdb: getdb,
+ cache: cache,
+ }
+}
+
+type oidcTokenAuthorizer struct {
+ ctrl *oidcLoginController
+ getdb func(context.Context) (*sqlx.DB, error)
+ cache *lru.TwoQueueCache
+}
+
+func (ta *oidcTokenAuthorizer) Middleware(w http.ResponseWriter, r *http.Request, next http.Handler) {
+ if ta.ctrl == nil {
+ // Not using a compatible (OIDC) login controller.
+ } else if authhdr := strings.Split(r.Header.Get("Authorization"), " "); len(authhdr) > 1 && (authhdr[0] == "OAuth2" || authhdr[0] == "Bearer") {
+ err := ta.registerToken(r.Context(), authhdr[1])
+ if err != nil {
+ http.Error(w, err.Error(), http.StatusInternalServerError)
+ return
+ }
+ }
+ next.ServeHTTP(w, r)
+}
+
+func (ta *oidcTokenAuthorizer) WrapCalls(origFunc api.RoutableFunc) api.RoutableFunc {
+ if ta.ctrl == nil {
+ // Not using a compatible (OIDC) login controller.
+ return origFunc
+ }
+ return func(ctx context.Context, opts interface{}) (_ interface{}, err error) {
+ creds, ok := auth.FromContext(ctx)
+ if !ok {
+ return origFunc(ctx, opts)
+ }
+ // Check each token in the incoming request. If any
+ // are OAuth2 access tokens, swap them out for Arvados
+ // tokens.
+ for _, tok := range creds.Tokens {
+ err = ta.registerToken(ctx, tok)
+ if err != nil {
+ return nil, err
+ }
+ }
+ return origFunc(ctx, opts)
+ }
+}
+
+// registerToken checks whether tok is a valid OIDC Access Token and,
+// if so, ensures that an api_client_authorizations row exists so that
+// RailsAPI will accept it as an Arvados token.
+func (ta *oidcTokenAuthorizer) registerToken(ctx context.Context, tok string) error {
+ if tok == ta.ctrl.Cluster.SystemRootToken || strings.HasPrefix(tok, "v2/") {
+ return nil
+ }
+ if cached, hit := ta.cache.Get(tok); !hit {
+ // Fall through to database and OIDC provider checks
+ // below
+ } else if exp, ok := cached.(time.Time); ok {
+ // cached negative result (value is expiry time)
+ if time.Now().Before(exp) {
+ return nil
+ } else {
+ ta.cache.Remove(tok)
+ }
+ } else {
+ // cached positive result
+ aca := cached.(arvados.APIClientAuthorization)
+ var expiring bool
+ if aca.ExpiresAt != "" {
+ t, err := time.Parse(time.RFC3339Nano, aca.ExpiresAt)
+ if err != nil {
+ return fmt.Errorf("error parsing expires_at value: %w", err)
+ }
+ expiring = t.Before(time.Now().Add(time.Minute))
+ }
+ if !expiring {
+ return nil
+ }
+ }
+
+ db, err := ta.getdb(ctx)
+ if err != nil {
+ return err
+ }
+ tx, err := db.Beginx()
+ if err != nil {
+ return err
+ }
+ defer tx.Rollback()
+ ctx = ctrlctx.NewWithTransaction(ctx, tx)
+
+ // We use hmac-sha256(accesstoken,systemroottoken) as the
+ // secret part of our own token, and avoid storing the auth
+ // provider's real secret in our database.
+ mac := hmac.New(sha256.New, []byte(ta.ctrl.Cluster.SystemRootToken))
+ io.WriteString(mac, tok)
+ hmac := fmt.Sprintf("%x", mac.Sum(nil))
+
+ var expiring bool
+ err = tx.QueryRowContext(ctx, `select (expires_at is not null and expires_at - interval '1 minute' <= current_timestamp at time zone 'UTC') from api_client_authorizations where api_token=$1`, hmac).Scan(&expiring)
+ if err != nil && err != sql.ErrNoRows {
+ return fmt.Errorf("database error while checking token: %w", err)
+ } else if err == nil && !expiring {
+ // Token is already in the database as an Arvados
+ // token, and isn't about to expire, so we can pass it
+ // through to RailsAPI etc. regardless of whether it's
+ // an OIDC access token.
+ return nil
+ }
+ updating := err == nil
+
+ // Check whether the token is a valid OIDC access token. If
+ // so, swap it out for an Arvados token (creating/updating an
+ // api_client_authorizations row if needed) which downstream
+ // server components will accept.
+ err = ta.ctrl.setup()
+ if err != nil {
+ return fmt.Errorf("error setting up OpenID Connect provider: %s", err)
+ }
+ oauth2Token := &oauth2.Token{
+ AccessToken: tok,
+ }
+ userinfo, err := ta.ctrl.provider.UserInfo(ctx, oauth2.StaticTokenSource(oauth2Token))
+ if err != nil {
+ ta.cache.Add(tok, time.Now().Add(tokenCacheNegativeTTL))
+ return nil
+ }
+ ctxlog.FromContext(ctx).WithField("userinfo", userinfo).Debug("(*oidcTokenAuthorizer)registerToken: got userinfo")
+ authinfo, err := ta.ctrl.getAuthInfo(ctx, oauth2Token, userinfo)
+ if err != nil {
+ return err
+ }
+
+ // Expiry time for our token is one minute longer than our
+ // cache TTL, so we don't pass it through to RailsAPI just as
+ // it's expiring.
+ exp := time.Now().UTC().Add(tokenCacheTTL + time.Minute)
+
+ var aca arvados.APIClientAuthorization
+ if updating {
+ _, err = tx.ExecContext(ctx, `update api_client_authorizations set expires_at=$1 where api_token=$2`, exp, hmac)
+ if err != nil {
+ return fmt.Errorf("error updating token expiry time: %w", err)
+ }
+ ctxlog.FromContext(ctx).WithField("HMAC", hmac).Debug("(*oidcTokenAuthorizer)registerToken: updated api_client_authorizations row")
+ } else {
+ aca, err = createAPIClientAuthorization(ctx, ta.ctrl.RailsProxy, ta.ctrl.Cluster.SystemRootToken, *authinfo)
+ if err != nil {
+ return err
+ }
+ _, err = tx.ExecContext(ctx, `update api_client_authorizations set api_token=$1, expires_at=$2 where uuid=$3`, hmac, exp, aca.UUID)
+ if err != nil {
+ return fmt.Errorf("error adding OIDC access token to database: %w", err)
+ }
+ aca.APIToken = hmac
+ ctxlog.FromContext(ctx).WithFields(logrus.Fields{"UUID": aca.UUID, "HMAC": hmac}).Debug("(*oidcTokenAuthorizer)registerToken: inserted api_client_authorizations row")
+ }
+ err = tx.Commit()
+ if err != nil {
+ return err
+ }
+ ta.cache.Add(tok, aca)
+ return nil
+}
import (
"bytes"
"context"
- "crypto/rand"
- "crypto/rsa"
- "encoding/base64"
"encoding/json"
"fmt"
"net/http"
"git.arvados.org/arvados.git/sdk/go/auth"
"git.arvados.org/arvados.git/sdk/go/ctxlog"
check "gopkg.in/check.v1"
- jose "gopkg.in/square/go-jose.v2"
)
// Gocheck boilerplate
var _ = check.Suite(&OIDCLoginSuite{})
type OIDCLoginSuite struct {
- cluster *arvados.Cluster
- localdb *Conn
- railsSpy *arvadostest.Proxy
- fakeIssuer *httptest.Server
- fakePeopleAPI *httptest.Server
- fakePeopleAPIResponse map[string]interface{}
- issuerKey *rsa.PrivateKey
-
- // expected token request
- validCode string
- validClientID string
- validClientSecret string
- // desired response from token endpoint
- authEmail string
- authEmailVerified bool
- authName string
+ cluster *arvados.Cluster
+ localdb *Conn
+ railsSpy *arvadostest.Proxy
+ fakeProvider *arvadostest.OIDCProvider
}
func (s *OIDCLoginSuite) TearDownSuite(c *check.C) {
}
func (s *OIDCLoginSuite) SetUpTest(c *check.C) {
- var err error
- s.issuerKey, err = rsa.GenerateKey(rand.Reader, 2048)
- c.Assert(err, check.IsNil)
-
- s.authEmail = "active-user@arvados.local"
- s.authEmailVerified = true
- s.authName = "Fake User Name"
- s.fakeIssuer = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
- req.ParseForm()
- c.Logf("fakeIssuer: got req: %s %s %s", req.Method, req.URL, req.Form)
- w.Header().Set("Content-Type", "application/json")
- switch req.URL.Path {
- case "/.well-known/openid-configuration":
- json.NewEncoder(w).Encode(map[string]interface{}{
- "issuer": s.fakeIssuer.URL,
- "authorization_endpoint": s.fakeIssuer.URL + "/auth",
- "token_endpoint": s.fakeIssuer.URL + "/token",
- "jwks_uri": s.fakeIssuer.URL + "/jwks",
- "userinfo_endpoint": s.fakeIssuer.URL + "/userinfo",
- })
- case "/token":
- var clientID, clientSecret string
- auth, _ := base64.StdEncoding.DecodeString(strings.TrimPrefix(req.Header.Get("Authorization"), "Basic "))
- authsplit := strings.Split(string(auth), ":")
- if len(authsplit) == 2 {
- clientID, _ = url.QueryUnescape(authsplit[0])
- clientSecret, _ = url.QueryUnescape(authsplit[1])
- }
- if clientID != s.validClientID || clientSecret != s.validClientSecret {
- c.Logf("fakeIssuer: expected (%q, %q) got (%q, %q)", s.validClientID, s.validClientSecret, clientID, clientSecret)
- w.WriteHeader(http.StatusUnauthorized)
- return
- }
-
- if req.Form.Get("code") != s.validCode || s.validCode == "" {
- w.WriteHeader(http.StatusUnauthorized)
- return
- }
- idToken, _ := json.Marshal(map[string]interface{}{
- "iss": s.fakeIssuer.URL,
- "aud": []string{clientID},
- "sub": "fake-user-id",
- "exp": time.Now().UTC().Add(time.Minute).Unix(),
- "iat": time.Now().UTC().Unix(),
- "nonce": "fake-nonce",
- "email": s.authEmail,
- "email_verified": s.authEmailVerified,
- "name": s.authName,
- "alt_verified": true, // for custom claim tests
- "alt_email": "alt_email@example.com", // for custom claim tests
- "alt_username": "desired-username", // for custom claim tests
- })
- json.NewEncoder(w).Encode(struct {
- AccessToken string `json:"access_token"`
- TokenType string `json:"token_type"`
- RefreshToken string `json:"refresh_token"`
- ExpiresIn int32 `json:"expires_in"`
- IDToken string `json:"id_token"`
- }{
- AccessToken: s.fakeToken(c, []byte("fake access token")),
- TokenType: "Bearer",
- RefreshToken: "test-refresh-token",
- ExpiresIn: 30,
- IDToken: s.fakeToken(c, idToken),
- })
- case "/jwks":
- json.NewEncoder(w).Encode(jose.JSONWebKeySet{
- Keys: []jose.JSONWebKey{
- {Key: s.issuerKey.Public(), Algorithm: string(jose.RS256), KeyID: ""},
- },
- })
- case "/auth":
- w.WriteHeader(http.StatusInternalServerError)
- case "/userinfo":
- w.WriteHeader(http.StatusInternalServerError)
- default:
- w.WriteHeader(http.StatusNotFound)
- }
- }))
- s.validCode = fmt.Sprintf("abcdefgh-%d", time.Now().Unix())
-
- s.fakePeopleAPI = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
- req.ParseForm()
- c.Logf("fakePeopleAPI: got req: %s %s %s", req.Method, req.URL, req.Form)
- w.Header().Set("Content-Type", "application/json")
- switch req.URL.Path {
- case "/v1/people/me":
- if f := req.Form.Get("personFields"); f != "emailAddresses,names" {
- w.WriteHeader(http.StatusBadRequest)
- break
- }
- json.NewEncoder(w).Encode(s.fakePeopleAPIResponse)
- default:
- w.WriteHeader(http.StatusNotFound)
- }
- }))
- s.fakePeopleAPIResponse = map[string]interface{}{}
+ s.fakeProvider = arvadostest.NewOIDCProvider(c)
+ s.fakeProvider.AuthEmail = "active-user@arvados.local"
+ s.fakeProvider.AuthEmailVerified = true
+ s.fakeProvider.AuthName = "Fake User Name"
+ s.fakeProvider.ValidCode = fmt.Sprintf("abcdefgh-%d", time.Now().Unix())
+ s.fakeProvider.PeopleAPIResponse = map[string]interface{}{}
cfg, err := config.NewLoader(nil, ctxlog.TestLogger(c)).Load()
c.Assert(err, check.IsNil)
s.cluster.Login.Google.ClientID = "test%client$id"
s.cluster.Login.Google.ClientSecret = "test#client/secret"
s.cluster.Users.PreferDomainForUsername = "PreferDomainForUsername.example.com"
- s.validClientID = "test%client$id"
- s.validClientSecret = "test#client/secret"
+ s.fakeProvider.ValidClientID = "test%client$id"
+ s.fakeProvider.ValidClientSecret = "test#client/secret"
s.localdb = NewConn(s.cluster)
c.Assert(s.localdb.loginController, check.FitsTypeOf, (*oidcLoginController)(nil))
- s.localdb.loginController.(*oidcLoginController).Issuer = s.fakeIssuer.URL
- s.localdb.loginController.(*oidcLoginController).peopleAPIBasePath = s.fakePeopleAPI.URL
+ s.localdb.loginController.(*oidcLoginController).Issuer = s.fakeProvider.Issuer.URL
+ s.localdb.loginController.(*oidcLoginController).peopleAPIBasePath = s.fakeProvider.PeopleAPI.URL
s.railsSpy = arvadostest.NewProxy(c, s.cluster.Services.RailsAPI)
*s.localdb.railsProxy = *rpc.NewConn(s.cluster.ClusterID, s.railsSpy.URL, true, rpc.PassthroughTokenProvider)
c.Check(err, check.IsNil)
target, err := url.Parse(resp.RedirectLocation)
c.Check(err, check.IsNil)
- issuerURL, _ := url.Parse(s.fakeIssuer.URL)
+ issuerURL, _ := url.Parse(s.fakeProvider.Issuer.URL)
c.Check(target.Host, check.Equals, issuerURL.Host)
q := target.Query()
c.Check(q.Get("client_id"), check.Equals, "test%client$id")
func (s *OIDCLoginSuite) TestGoogleLogin_InvalidState(c *check.C) {
s.startLogin(c)
resp, err := s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: "bogus-state",
})
c.Check(err, check.IsNil)
}
func (s *OIDCLoginSuite) setupPeopleAPIError(c *check.C) {
- s.fakePeopleAPI = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+ s.fakeProvider.PeopleAPI = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusForbidden)
fmt.Fprintln(w, `Error 403: accessNotConfigured`)
}))
- s.localdb.loginController.(*oidcLoginController).peopleAPIBasePath = s.fakePeopleAPI.URL
+ s.localdb.loginController.(*oidcLoginController).peopleAPIBasePath = s.fakeProvider.PeopleAPI.URL
}
func (s *OIDCLoginSuite) TestGoogleLogin_PeopleAPIDisabled(c *check.C) {
s.localdb.loginController.(*oidcLoginController).UseGooglePeopleAPI = false
- s.authEmail = "joe.smith@primary.example.com"
+ s.fakeProvider.AuthEmail = "joe.smith@primary.example.com"
s.setupPeopleAPIError(c)
state := s.startLogin(c)
_, err := s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
c.Check(err, check.IsNil)
s.setupPeopleAPIError(c)
state := s.startLogin(c)
resp, err := s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
c.Check(err, check.IsNil)
func (s *OIDCLoginSuite) TestGenericOIDCLogin(c *check.C) {
s.cluster.Login.Google.Enable = false
s.cluster.Login.OpenIDConnect.Enable = true
- json.Unmarshal([]byte(fmt.Sprintf("%q", s.fakeIssuer.URL)), &s.cluster.Login.OpenIDConnect.Issuer)
+ json.Unmarshal([]byte(fmt.Sprintf("%q", s.fakeProvider.Issuer.URL)), &s.cluster.Login.OpenIDConnect.Issuer)
s.cluster.Login.OpenIDConnect.ClientID = "oidc#client#id"
s.cluster.Login.OpenIDConnect.ClientSecret = "oidc#client#secret"
- s.validClientID = "oidc#client#id"
- s.validClientSecret = "oidc#client#secret"
+ s.fakeProvider.ValidClientID = "oidc#client#id"
+ s.fakeProvider.ValidClientSecret = "oidc#client#secret"
for _, trial := range []struct {
expectEmail string // "" if failure expected
setup func()
expectEmail: "user@oidc.example.com",
setup: func() {
c.Log("=== succeed because email_verified is false but not required")
- s.authEmail = "user@oidc.example.com"
- s.authEmailVerified = false
+ s.fakeProvider.AuthEmail = "user@oidc.example.com"
+ s.fakeProvider.AuthEmailVerified = false
s.cluster.Login.OpenIDConnect.EmailClaim = "email"
s.cluster.Login.OpenIDConnect.EmailVerifiedClaim = ""
s.cluster.Login.OpenIDConnect.UsernameClaim = ""
expectEmail: "",
setup: func() {
c.Log("=== fail because email_verified is false and required")
- s.authEmail = "user@oidc.example.com"
- s.authEmailVerified = false
+ s.fakeProvider.AuthEmail = "user@oidc.example.com"
+ s.fakeProvider.AuthEmailVerified = false
s.cluster.Login.OpenIDConnect.EmailClaim = "email"
s.cluster.Login.OpenIDConnect.EmailVerifiedClaim = "email_verified"
s.cluster.Login.OpenIDConnect.UsernameClaim = ""
expectEmail: "user@oidc.example.com",
setup: func() {
c.Log("=== succeed because email_verified is false but config uses custom 'verified' claim")
- s.authEmail = "user@oidc.example.com"
- s.authEmailVerified = false
+ s.fakeProvider.AuthEmail = "user@oidc.example.com"
+ s.fakeProvider.AuthEmailVerified = false
s.cluster.Login.OpenIDConnect.EmailClaim = "email"
s.cluster.Login.OpenIDConnect.EmailVerifiedClaim = "alt_verified"
s.cluster.Login.OpenIDConnect.UsernameClaim = ""
expectEmail: "alt_email@example.com",
setup: func() {
c.Log("=== succeed with custom 'email' and 'email_verified' claims")
- s.authEmail = "bad@wrong.example.com"
- s.authEmailVerified = false
+ s.fakeProvider.AuthEmail = "bad@wrong.example.com"
+ s.fakeProvider.AuthEmailVerified = false
s.cluster.Login.OpenIDConnect.EmailClaim = "alt_email"
s.cluster.Login.OpenIDConnect.EmailVerifiedClaim = "alt_verified"
s.cluster.Login.OpenIDConnect.UsernameClaim = "alt_username"
state := s.startLogin(c)
resp, err := s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
c.Assert(err, check.IsNil)
func (s *OIDCLoginSuite) TestGoogleLogin_Success(c *check.C) {
state := s.startLogin(c)
resp, err := s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
c.Check(err, check.IsNil)
}
func (s *OIDCLoginSuite) TestGoogleLogin_RealName(c *check.C) {
- s.authEmail = "joe.smith@primary.example.com"
- s.fakePeopleAPIResponse = map[string]interface{}{
+ s.fakeProvider.AuthEmail = "joe.smith@primary.example.com"
+ s.fakeProvider.PeopleAPIResponse = map[string]interface{}{
"names": []map[string]interface{}{
{
"metadata": map[string]interface{}{"primary": false},
}
state := s.startLogin(c)
s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
}
func (s *OIDCLoginSuite) TestGoogleLogin_OIDCRealName(c *check.C) {
- s.authName = "Joe P. Smith"
- s.authEmail = "joe.smith@primary.example.com"
+ s.fakeProvider.AuthName = "Joe P. Smith"
+ s.fakeProvider.AuthEmail = "joe.smith@primary.example.com"
state := s.startLogin(c)
s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
// People API returns some additional email addresses.
func (s *OIDCLoginSuite) TestGoogleLogin_AlternateEmailAddresses(c *check.C) {
- s.authEmail = "joe.smith@primary.example.com"
- s.fakePeopleAPIResponse = map[string]interface{}{
+ s.fakeProvider.AuthEmail = "joe.smith@primary.example.com"
+ s.fakeProvider.PeopleAPIResponse = map[string]interface{}{
"emailAddresses": []map[string]interface{}{
{
"metadata": map[string]interface{}{"verified": true},
}
state := s.startLogin(c)
s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
// Primary address is not the one initially returned by oidc.
func (s *OIDCLoginSuite) TestGoogleLogin_AlternateEmailAddresses_Primary(c *check.C) {
- s.authEmail = "joe.smith@alternate.example.com"
- s.fakePeopleAPIResponse = map[string]interface{}{
+ s.fakeProvider.AuthEmail = "joe.smith@alternate.example.com"
+ s.fakeProvider.PeopleAPIResponse = map[string]interface{}{
"emailAddresses": []map[string]interface{}{
{
"metadata": map[string]interface{}{"verified": true, "primary": true},
}
state := s.startLogin(c)
s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
authinfo := getCallbackAuthInfo(c, s.railsSpy)
}
func (s *OIDCLoginSuite) TestGoogleLogin_NoPrimaryEmailAddress(c *check.C) {
- s.authEmail = "joe.smith@unverified.example.com"
- s.authEmailVerified = false
- s.fakePeopleAPIResponse = map[string]interface{}{
+ s.fakeProvider.AuthEmail = "joe.smith@unverified.example.com"
+ s.fakeProvider.AuthEmailVerified = false
+ s.fakeProvider.PeopleAPIResponse = map[string]interface{}{
"emailAddresses": []map[string]interface{}{
{
"metadata": map[string]interface{}{"verified": true},
}
state := s.startLogin(c)
s.localdb.Login(context.Background(), arvados.LoginOptions{
- Code: s.validCode,
+ Code: s.fakeProvider.ValidCode,
State: state,
})
return
}
-func (s *OIDCLoginSuite) fakeToken(c *check.C, payload []byte) string {
- signer, err := jose.NewSigner(jose.SigningKey{Algorithm: jose.RS256, Key: s.issuerKey}, nil)
- if err != nil {
- c.Error(err)
- }
- object, err := signer.Sign(payload)
- if err != nil {
- c.Error(err)
- }
- t, err := object.CompactSerialize()
- if err != nil {
- c.Error(err)
- }
- c.Logf("fakeToken(%q) == %q", payload, t)
- return t
-}
-
func getCallbackAuthInfo(c *check.C, railsSpy *arvadostest.Proxy) (authinfo rpc.UserSessionAuthInfo) {
for _, dump := range railsSpy.RequestDumps {
c.Logf("spied request: %q", dump)
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package arvadostest
+
+import (
+ "crypto/rand"
+ "crypto/rsa"
+ "encoding/base64"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "net/url"
+ "strings"
+ "time"
+
+ "gopkg.in/check.v1"
+ "gopkg.in/square/go-jose.v2"
+)
+
+type OIDCProvider struct {
+ // expected token request
+ ValidCode string
+ ValidClientID string
+ ValidClientSecret string
+ // desired response from token endpoint
+ AuthEmail string
+ AuthEmailVerified bool
+ AuthName string
+
+ PeopleAPIResponse map[string]interface{}
+
+ key *rsa.PrivateKey
+ Issuer *httptest.Server
+ PeopleAPI *httptest.Server
+ c *check.C
+}
+
+func NewOIDCProvider(c *check.C) *OIDCProvider {
+ p := &OIDCProvider{c: c}
+ var err error
+ p.key, err = rsa.GenerateKey(rand.Reader, 2048)
+ c.Assert(err, check.IsNil)
+ p.Issuer = httptest.NewServer(http.HandlerFunc(p.serveOIDC))
+ p.PeopleAPI = httptest.NewServer(http.HandlerFunc(p.servePeopleAPI))
+ return p
+}
+
+func (p *OIDCProvider) ValidAccessToken() string {
+ return p.fakeToken([]byte("fake access token"))
+}
+
+func (p *OIDCProvider) serveOIDC(w http.ResponseWriter, req *http.Request) {
+ req.ParseForm()
+ p.c.Logf("serveOIDC: got req: %s %s %s", req.Method, req.URL, req.Form)
+ w.Header().Set("Content-Type", "application/json")
+ switch req.URL.Path {
+ case "/.well-known/openid-configuration":
+ json.NewEncoder(w).Encode(map[string]interface{}{
+ "issuer": p.Issuer.URL,
+ "authorization_endpoint": p.Issuer.URL + "/auth",
+ "token_endpoint": p.Issuer.URL + "/token",
+ "jwks_uri": p.Issuer.URL + "/jwks",
+ "userinfo_endpoint": p.Issuer.URL + "/userinfo",
+ })
+ case "/token":
+ var clientID, clientSecret string
+ auth, _ := base64.StdEncoding.DecodeString(strings.TrimPrefix(req.Header.Get("Authorization"), "Basic "))
+ authsplit := strings.Split(string(auth), ":")
+ if len(authsplit) == 2 {
+ clientID, _ = url.QueryUnescape(authsplit[0])
+ clientSecret, _ = url.QueryUnescape(authsplit[1])
+ }
+ if clientID != p.ValidClientID || clientSecret != p.ValidClientSecret {
+ p.c.Logf("OIDCProvider: expected (%q, %q) got (%q, %q)", p.ValidClientID, p.ValidClientSecret, clientID, clientSecret)
+ w.WriteHeader(http.StatusUnauthorized)
+ return
+ }
+
+ if req.Form.Get("code") != p.ValidCode || p.ValidCode == "" {
+ w.WriteHeader(http.StatusUnauthorized)
+ return
+ }
+ idToken, _ := json.Marshal(map[string]interface{}{
+ "iss": p.Issuer.URL,
+ "aud": []string{clientID},
+ "sub": "fake-user-id",
+ "exp": time.Now().UTC().Add(time.Minute).Unix(),
+ "iat": time.Now().UTC().Unix(),
+ "nonce": "fake-nonce",
+ "email": p.AuthEmail,
+ "email_verified": p.AuthEmailVerified,
+ "name": p.AuthName,
+ "alt_verified": true, // for custom claim tests
+ "alt_email": "alt_email@example.com", // for custom claim tests
+ "alt_username": "desired-username", // for custom claim tests
+ })
+ json.NewEncoder(w).Encode(struct {
+ AccessToken string `json:"access_token"`
+ TokenType string `json:"token_type"`
+ RefreshToken string `json:"refresh_token"`
+ ExpiresIn int32 `json:"expires_in"`
+ IDToken string `json:"id_token"`
+ }{
+ AccessToken: p.ValidAccessToken(),
+ TokenType: "Bearer",
+ RefreshToken: "test-refresh-token",
+ ExpiresIn: 30,
+ IDToken: p.fakeToken(idToken),
+ })
+ case "/jwks":
+ json.NewEncoder(w).Encode(jose.JSONWebKeySet{
+ Keys: []jose.JSONWebKey{
+ {Key: p.key.Public(), Algorithm: string(jose.RS256), KeyID: ""},
+ },
+ })
+ case "/auth":
+ w.WriteHeader(http.StatusInternalServerError)
+ case "/userinfo":
+ if authhdr := req.Header.Get("Authorization"); strings.TrimPrefix(authhdr, "Bearer ") != p.ValidAccessToken() {
+ p.c.Logf("OIDCProvider: bad auth %q", authhdr)
+ w.WriteHeader(http.StatusUnauthorized)
+ return
+ }
+ json.NewEncoder(w).Encode(map[string]interface{}{
+ "sub": "fake-user-id",
+ "name": p.AuthName,
+ "given_name": p.AuthName,
+ "family_name": "",
+ "alt_username": "desired-username",
+ "email": p.AuthEmail,
+ "email_verified": p.AuthEmailVerified,
+ })
+ default:
+ w.WriteHeader(http.StatusNotFound)
+ }
+}
+
+func (p *OIDCProvider) servePeopleAPI(w http.ResponseWriter, req *http.Request) {
+ req.ParseForm()
+ p.c.Logf("servePeopleAPI: got req: %s %s %s", req.Method, req.URL, req.Form)
+ w.Header().Set("Content-Type", "application/json")
+ switch req.URL.Path {
+ case "/v1/people/me":
+ if f := req.Form.Get("personFields"); f != "emailAddresses,names" {
+ w.WriteHeader(http.StatusBadRequest)
+ break
+ }
+ json.NewEncoder(w).Encode(p.PeopleAPIResponse)
+ default:
+ w.WriteHeader(http.StatusNotFound)
+ }
+}
+
+func (p *OIDCProvider) fakeToken(payload []byte) string {
+ signer, err := jose.NewSigner(jose.SigningKey{Algorithm: jose.RS256, Key: p.key}, nil)
+ if err != nil {
+ p.c.Error(err)
+ return ""
+ }
+ object, err := signer.Sign(payload)
+ if err != nil {
+ p.c.Error(err)
+ return ""
+ }
+ t, err := object.CompactSerialize()
+ if err != nil {
+ p.c.Error(err)
+ return ""
+ }
+ p.c.Logf("fakeToken(%q) == %q", payload, t)
+ return t
+}
auth = nil
[params["api_token"],
params["oauth_token"],
- env["HTTP_AUTHORIZATION"].andand.match(/(OAuth2|Bearer) ([-\/a-zA-Z0-9]+)/).andand[2],
+ env["HTTP_AUTHORIZATION"].andand.match(/(OAuth2|Bearer) ([!-~]+)/).andand[2],
*reader_tokens,
].each do |supplied|
next if !supplied
return auth
end
+ token_uuid = ''
+ secret = token
+ optional = nil
+
case token[0..2]
when 'v2/'
_, token_uuid, secret, optional = token.split('/')
return auth
end
- token_uuid_prefix = token_uuid[0..4]
- if token_uuid_prefix == Rails.configuration.ClusterID
+ upstream_cluster_id = token_uuid[0..4]
+ if upstream_cluster_id == Rails.configuration.ClusterID
# Token is supposedly issued by local cluster, but if the
# token were valid, we would have been found in the database
# in the above query.
return nil
- elsif token_uuid_prefix.length != 5
+ elsif upstream_cluster_id.length != 5
# malformed
return nil
end
- # Invariant: token_uuid_prefix != Rails.configuration.ClusterID
- #
- # In other words the remaing code in this method below is the
- # case that determines whether to accept a token that was issued
- # by a remote cluster when the token absent or expired in our
- # database. To begin, we need to ask the cluster that issued
- # the token to [re]validate it.
- clnt = ApiClientAuthorization.make_http_client(uuid_prefix: token_uuid_prefix)
-
- host = remote_host(uuid_prefix: token_uuid_prefix)
- if !host
- Rails.logger.warn "remote authentication rejected: no host for #{token_uuid_prefix.inspect}"
+ else
+ # token is not a 'v2' token. It could be just the secret part
+ # ("v1 token") -- or it could be an OpenIDConnect access token,
+ # in which case either (a) the controller will have inserted a
+ # row with api_token = hmac(systemroottoken,oidctoken) before
+ # forwarding it, or (b) we'll have done that ourselves, or (c)
+ # we'll need to ask LoginCluster to validate it for us below,
+ # and then insert a local row for a faster lookup next time.
+ hmac = OpenSSL::HMAC.hexdigest('sha256', Rails.configuration.SystemRootToken, token)
+ auth = ApiClientAuthorization.
+ includes(:user, :api_client).
+ where('api_token in (?, ?) and (expires_at is null or expires_at > CURRENT_TIMESTAMP)', token, hmac).
+ first
+ if auth && auth.user
+ return auth
+ elsif !Rails.configuration.Login.LoginCluster.blank? && Rails.configuration.Login.LoginCluster != Rails.configuration.ClusterID
+ # An unrecognized non-v2 token might be an OIDC Access Token
+ # that can be verified by our login cluster in the code
+ # below. If so, we'll stuff the database with hmac instead of
+ # the real OIDC token.
+ upstream_cluster_id = Rails.configuration.Login.LoginCluster
+ token_uuid = upstream_cluster_id + generate_uuid[5..27]
+ secret = hmac
+ else
return nil
end
+ end
- begin
- remote_user = SafeJSON.load(
- clnt.get_content('https://' + host + '/arvados/v1/users/current',
- {'remote' => Rails.configuration.ClusterID},
- {'Authorization' => 'Bearer ' + token}))
- rescue => e
- Rails.logger.warn "remote authentication with token #{token.inspect} failed: #{e}"
- return nil
- end
+ # Invariant: upstream_cluster_id != Rails.configuration.ClusterID
+ #
+ # In other words the remaining code in this method decides
+ # whether to accept a token that was issued by a remote cluster
+ # when the token is absent or expired in our database. To
+ # begin, we need to ask the cluster that issued the token to
+ # [re]validate it.
+ clnt = ApiClientAuthorization.make_http_client(uuid_prefix: upstream_cluster_id)
+
+ host = remote_host(uuid_prefix: upstream_cluster_id)
+ if !host
+ Rails.logger.warn "remote authentication rejected: no host for #{upstream_cluster_id.inspect}"
+ return nil
+ end
- # Check the response is well formed.
- if !remote_user.is_a?(Hash) || !remote_user['uuid'].is_a?(String)
- Rails.logger.warn "remote authentication rejected: remote_user=#{remote_user.inspect}"
- return nil
- end
+ begin
+ remote_user = SafeJSON.load(
+ clnt.get_content('https://' + host + '/arvados/v1/users/current',
+ {'remote' => Rails.configuration.ClusterID},
+ {'Authorization' => 'Bearer ' + token}))
+ rescue => e
+ Rails.logger.warn "remote authentication with token #{token.inspect} failed: #{e}"
+ return nil
+ end
- remote_user_prefix = remote_user['uuid'][0..4]
+ # Check the response is well formed.
+ if !remote_user.is_a?(Hash) || !remote_user['uuid'].is_a?(String)
+ Rails.logger.warn "remote authentication rejected: remote_user=#{remote_user.inspect}"
+ return nil
+ end
- # Clusters can only authenticate for their own users.
- if remote_user_prefix != token_uuid_prefix
- Rails.logger.warn "remote authentication rejected: claimed remote user #{remote_user_prefix} but token was issued by #{token_uuid_prefix}"
- return nil
- end
+ remote_user_prefix = remote_user['uuid'][0..4]
- # Invariant: remote_user_prefix == token_uuid_prefix
- # therefore: remote_user_prefix != Rails.configuration.ClusterID
+ # Clusters can only authenticate for their own users.
+ if remote_user_prefix != upstream_cluster_id
+ Rails.logger.warn "remote authentication rejected: claimed remote user #{remote_user_prefix} but token was issued by #{upstream_cluster_id}"
+ return nil
+ end
- # Add or update user and token in local database so we can
- # validate subsequent requests faster.
+ # Invariant: remote_user_prefix == upstream_cluster_id
+ # therefore: remote_user_prefix != Rails.configuration.ClusterID
- if remote_user['uuid'][-22..-1] == '-tpzed-anonymouspublic'
- # Special case: map the remote anonymous user to local anonymous user
- remote_user['uuid'] = anonymous_user_uuid
- end
+ # Add or update user and token in local database so we can
+ # validate subsequent requests faster.
- user = User.find_by_uuid(remote_user['uuid'])
+ if remote_user['uuid'][-22..-1] == '-tpzed-anonymouspublic'
+ # Special case: map the remote anonymous user to local anonymous user
+ remote_user['uuid'] = anonymous_user_uuid
+ end
- if !user
- # Create a new record for this user.
- user = User.new(uuid: remote_user['uuid'],
- is_active: false,
- is_admin: false,
- email: remote_user['email'],
- owner_uuid: system_user_uuid)
- user.set_initial_username(requested: remote_user['username'])
- end
+ user = User.find_by_uuid(remote_user['uuid'])
- # Sync user record.
- act_as_system_user do
- %w[first_name last_name email prefs].each do |attr|
- user.send(attr+'=', remote_user[attr])
- end
+ if !user
+ # Create a new record for this user.
+ user = User.new(uuid: remote_user['uuid'],
+ is_active: false,
+ is_admin: false,
+ email: remote_user['email'],
+ owner_uuid: system_user_uuid)
+ user.set_initial_username(requested: remote_user['username'])
+ end
- if remote_user['uuid'][-22..-1] == '-tpzed-000000000000000'
- user.first_name = "root"
- user.last_name = "from cluster #{remote_user_prefix}"
- end
+ # Sync user record.
+ act_as_system_user do
+ %w[first_name last_name email prefs].each do |attr|
+ user.send(attr+'=', remote_user[attr])
+ end
- user.save!
-
- if user.is_invited && !remote_user['is_invited']
- # Remote user is not "invited" state, they should be unsetup, which
- # also makes them inactive.
- user.unsetup
- else
- if !user.is_invited && remote_user['is_invited'] and
- (remote_user_prefix == Rails.configuration.Login.LoginCluster or
- Rails.configuration.Users.AutoSetupNewUsers or
- Rails.configuration.Users.NewUsersAreActive or
- Rails.configuration.RemoteClusters[remote_user_prefix].andand["ActivateUsers"])
- user.setup
- end
+ if remote_user['uuid'][-22..-1] == '-tpzed-000000000000000'
+ user.first_name = "root"
+ user.last_name = "from cluster #{remote_user_prefix}"
+ end
- if !user.is_active && remote_user['is_active'] && user.is_invited and
- (remote_user_prefix == Rails.configuration.Login.LoginCluster or
- Rails.configuration.Users.NewUsersAreActive or
- Rails.configuration.RemoteClusters[remote_user_prefix].andand["ActivateUsers"])
- user.update_attributes!(is_active: true)
- elsif user.is_active && !remote_user['is_active']
- user.update_attributes!(is_active: false)
- end
+ user.save!
+
+ if user.is_invited && !remote_user['is_invited']
+ # Remote user is not "invited" state, they should be unsetup, which
+ # also makes them inactive.
+ user.unsetup
+ else
+ if !user.is_invited && remote_user['is_invited'] and
+ (remote_user_prefix == Rails.configuration.Login.LoginCluster or
+ Rails.configuration.Users.AutoSetupNewUsers or
+ Rails.configuration.Users.NewUsersAreActive or
+ Rails.configuration.RemoteClusters[remote_user_prefix].andand["ActivateUsers"])
+ user.setup
+ end
- if remote_user_prefix == Rails.configuration.Login.LoginCluster and
- user.is_active and
- user.is_admin != remote_user['is_admin']
- # Remote cluster controls our user database, including the
- # admin flag.
- user.update_attributes!(is_admin: remote_user['is_admin'])
- end
+ if !user.is_active && remote_user['is_active'] && user.is_invited and
+ (remote_user_prefix == Rails.configuration.Login.LoginCluster or
+ Rails.configuration.Users.NewUsersAreActive or
+ Rails.configuration.RemoteClusters[remote_user_prefix].andand["ActivateUsers"])
+ user.update_attributes!(is_active: true)
+ elsif user.is_active && !remote_user['is_active']
+ user.update_attributes!(is_active: false)
end
- # We will accept this token (and avoid reloading the user
- # record) for 'RemoteTokenRefresh' (default 5 minutes).
- # Possible todo:
- # Request the actual api_client_auth record from the remote
- # server in case it wants the token to expire sooner.
- auth = ApiClientAuthorization.find_or_create_by(uuid: token_uuid) do |auth|
- auth.user = user
- auth.api_client_id = 0
+ if remote_user_prefix == Rails.configuration.Login.LoginCluster and
+ user.is_active and
+ user.is_admin != remote_user['is_admin']
+ # Remote cluster controls our user database, including the
+ # admin flag.
+ user.update_attributes!(is_admin: remote_user['is_admin'])
end
- auth.update_attributes!(user: user,
- api_token: secret,
- api_client_id: 0,
- expires_at: Time.now + Rails.configuration.Login.RemoteTokenRefresh)
- Rails.logger.debug "cached remote token #{token_uuid} with secret #{secret} in local db"
end
- return auth
- else
- # token is not a 'v2' token
- auth = ApiClientAuthorization.
- includes(:user, :api_client).
- where('api_token=? and (expires_at is null or expires_at > CURRENT_TIMESTAMP)', token).
- first
- if auth && auth.user
- return auth
+
+ # We will accept this token (and avoid reloading the user
+ # record) for 'RemoteTokenRefresh' (default 5 minutes).
+ # Possible todo:
+ # Request the actual api_client_auth record from the remote
+ # server in case it wants the token to expire sooner.
+ auth = ApiClientAuthorization.find_or_create_by(uuid: token_uuid) do |auth|
+ auth.user = user
+ auth.api_client_id = 0
end
+ auth.update_attributes!(user: user,
+ api_token: secret,
+ api_client_id: 0,
+ expires_at: Time.now + Rails.configuration.Login.RemoteTokenRefresh)
+ Rails.logger.debug "cached remote token #{token_uuid} with secret #{secret} in local db"
+ return auth
end
return nil
--- /dev/null
+[comment]: # (Copyright © The Arvados Authors. All rights reserved.)
+[comment]: # ()
+[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)
+
+# Arvados install with Saltstack
+
+##### About
+
+This directory holds a small script to install Arvados on a single node, using the
+[Saltstack arvados-formula](https://github.com/saltstack-formulas/arvados-formula)
+in master-less mode.
+
+The fastest way to get it running is to modify the first lines in the `provision.sh`
+script to suit your needs, copy it in the host where you want to install Arvados
+and run it as root.
+
+There's an example `Vagrantfile` also, to install it in a vagrant box if you want
+to try it locally.
+
+For more information, please read https://doc.arvados.org/v2.1/install/install-using-salt.html
--- /dev/null
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+# Vagrantfile API/syntax version. Don"t touch unless you know what you"re doing!
+VAGRANTFILE_API_VERSION = "2".freeze
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+ config.ssh.insert_key = false
+ config.ssh.forward_x11 = true
+
+ config.vm.define "arvados" do |arv|
+ arv.vm.box = "bento/debian-10"
+ arv.vm.hostname = "arva2.arv.local"
+ # Networking
+ arv.vm.network "forwarded_port", guest: 8443, host: 8443
+ arv.vm.network "forwarded_port", guest: 25100, host: 25100
+ arv.vm.network "forwarded_port", guest: 9002, host: 9002
+ arv.vm.network "forwarded_port", guest: 9000, host: 9000
+ arv.vm.network "forwarded_port", guest: 8900, host: 8900
+ arv.vm.network "forwarded_port", guest: 8002, host: 8002
+ arv.vm.network "forwarded_port", guest: 8001, host: 8001
+ arv.vm.network "forwarded_port", guest: 8000, host: 8000
+ arv.vm.network "forwarded_port", guest: 3001, host: 3001
+ # config.vm.network "private_network", ip: "192.168.33.10"
+ # arv.vm.synced_folder "salt_pillars", "/srv/pillars",
+ # create: true
+ arv.vm.provision "shell",
+ path: "provision.sh",
+ args: [
+ "--vagrant",
+ "--ssl-port=8443"
+ ].join(" ")
+ end
+end
--- /dev/null
+#!/bin/bash -x
+
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: CC-BY-SA-3.0
+
+# If you want to test arvados in a single host, you can run this script, which
+# will install it using salt masterless
+# This script is run by the Vagrant file when you run it with
+#
+# vagrant up
+
+##########################################################
+# This section are the basic parameters to configure the installation
+
+# The 5 letters name you want to give your cluster
+CLUSTER="arva2"
+DOMAIN="arv.local"
+
+INITIAL_USER="admin"
+
+# If not specified, the initial user email will be composed as
+# INITIAL_USER@CLUSTER.DOMAIN
+INITIAL_USER_EMAIL="${INITIAL_USER}@${CLUSTER}.${DOMAIN}"
+INITIAL_USER_PASSWORD="password"
+
+# The example config you want to use. Currently, only "single_host" is
+# available
+CONFIG_DIR="single_host"
+
+# Which release of Arvados repo you want to use
+RELEASE="production"
+# Which version of Arvados you want to install. Defaults to 'latest'
+# in the desired repo
+VERSION="latest"
+
+# Host SSL port where you want to point your browser to access Arvados
+# Defaults to 443 for regular runs, and to 8443 when called in Vagrant.
+# You can point it to another port if desired
+# In Vagrant, make sure it matches what you set in the Vagrantfile
+# HOST_SSL_PORT=443
+
+# This is a arvados-formula setting.
+# If branch is set, the script will switch to it before running salt
+# Usually not needed, only used for testing
+# BRANCH="master"
+
+##########################################################
+# Usually there's no need to modify things below this line
+
+set -o pipefail
+
+usage() {
+ echo >&2
+ echo >&2 "Usage: $0 [-h] [-h]"
+ echo >&2
+ echo >&2 "$0 options:"
+ echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
+ echo >&2 " -p <N>, --ssl-port <N> SSL port to use for the web applications"
+ echo >&2 " -h, --help Display this help and exit"
+ echo >&2
+}
+
+arguments() {
+ # NOTE: This requires GNU getopt (part of the util-linux package on Debian-based distros).
+ TEMP=`getopt -o hvp: \
+ --long help,vagrant,ssl-port: \
+ -n "$0" -- "$@"`
+
+ if [ $? != 0 ] ; then echo "GNU getopt missing? Use -h for help"; exit 1 ; fi
+ # Note the quotes around `$TEMP': they are essential!
+ eval set -- "$TEMP"
+
+ while [ $# -ge 1 ]; do
+ case $1 in
+ -v | --vagrant)
+ VAGRANT="yes"
+ shift
+ ;;
+ -p | --ssl-port)
+ HOST_SSL_PORT=${2}
+ shift 2
+ ;;
+ --)
+ shift
+ break
+ ;;
+ *)
+ usage
+ exit 1
+ ;;
+ esac
+ done
+}
+
+HOST_SSL_PORT=443
+
+arguments $@
+
+# Salt's dir
+## states
+S_DIR="/srv/salt"
+## formulas
+F_DIR="/srv/formulas"
+##pillars
+P_DIR="/srv/pillars"
+
+apt-get update
+apt-get install -y curl git
+
+dpkg -l |grep salt-minion
+if [ ${?} -eq 0 ]; then
+ echo "Salt already installed"
+else
+ curl -L https://bootstrap.saltstack.com -o /tmp/bootstrap_salt.sh
+ sh /tmp/bootstrap_salt.sh -XUdfP -x python3
+ /bin/systemctl disable salt-minion.service
+fi
+
+# Set salt to masterless mode
+cat > /etc/salt/minion << EOFSM
+file_client: local
+file_roots:
+ base:
+ - ${S_DIR}
+ - ${F_DIR}/*
+ - ${F_DIR}/*/test/salt/states
+
+pillar_roots:
+ base:
+ - ${P_DIR}
+EOFSM
+
+mkdir -p ${S_DIR}
+mkdir -p ${F_DIR}
+mkdir -p ${P_DIR}
+
+# States
+cat > ${S_DIR}/top.sls << EOFTSLS
+base:
+ '*':
+ - example_add_snakeoil_certs
+ - locale
+ - nginx.passenger
+ - postgres
+ - docker
+ - arvados
+EOFTSLS
+
+# Pillars
+cat > ${P_DIR}/top.sls << EOFPSLS
+base:
+ '*':
+ - arvados
+ - locale
+ - nginx_api_configuration
+ - nginx_controller_configuration
+ - nginx_keepproxy_configuration
+ - nginx_keepweb_configuration
+ - nginx_passenger
+ - nginx_websocket_configuration
+ - nginx_webshell_configuration
+ - nginx_workbench2_configuration
+ - nginx_workbench_configuration
+ - postgresql
+EOFPSLS
+
+
+# Get the formula and dependencies
+cd ${F_DIR} || exit 1
+for f in postgres arvados nginx docker locale; do
+ git clone https://github.com/saltstack-formulas/${f}-formula.git
+done
+
+if [ "x${BRANCH}" != "x" ]; then
+ cd ${F_DIR}/arvados-formula
+ git checkout -t origin/${BRANCH}
+ cd -
+fi
+
+# sed "s/__DOMAIN__/${DOMAIN}/g; s/__CLUSTER__/${CLUSTER}/g; s/__RELEASE__/${RELEASE}/g; s/__VERSION__/${VERSION}/g" \
+# ${CONFIG_DIR}/arvados_dev.sls > ${P_DIR}/arvados.sls
+
+if [ "x${VAGRANT}" = "xyes" ]; then
+ SOURCE_PILLARS_DIR="/vagrant/${CONFIG_DIR}"
+else
+ SOURCE_PILLARS_DIR="./${CONFIG_DIR}"
+fi
+
+# Replace cluster and domain name in the example pillars
+for f in ${SOURCE_PILLARS_DIR}/*; do
+ # sed "s/example.net/${DOMAIN}/g; s/fixme/${CLUSTER}/g" \
+ sed "s/__DOMAIN__/${DOMAIN}/g;
+ s/__CLUSTER__/${CLUSTER}/g;
+ s/__RELEASE__/${RELEASE}/g;
+ s/__HOST_SSL_PORT__/${HOST_SSL_PORT}/g;
+ s/__GUEST_SSL_PORT__/${GUEST_SSL_PORT}/g;
+ s/__INITIAL_USER__/${INITIAL_USER}/g;
+ s/__INITIAL_USER_EMAIL__/${INITIAL_USER_EMAIL}/g;
+ s/__INITIAL_USER_PASSWORD__/${INITIAL_USER_PASSWORD}/g;
+ s/__VERSION__/${VERSION}/g" \
+ ${f} > ${P_DIR}/$(basename ${f})
+done
+
+# Let's write an /etc/hosts file that points all the hosts to localhost
+
+echo "127.0.0.2 api keep keep0 collections download ws workbench workbench2 ${CLUSTER}.${DOMAIN} api.${CLUSTER}.${DOMAIN} keep.${CLUSTER}.${DOMAIN} keep0.${CLUSTER}.${DOMAIN} collections.${CLUSTER}.${DOMAIN} download.${CLUSTER}.${DOMAIN} ws.${CLUSTER}.${DOMAIN} workbench.${CLUSTER}.${DOMAIN} workbench2.${CLUSTER}.${DOMAIN}" >> /etc/hosts
+
+# FIXME! #16992 Temporary fix for psql call in arvados-api-server
+if [ -e /root/.psqlrc ]; then
+ if ! ( grep 'pset pager off' /root/.psqlrc ); then
+ RESTORE_PSQL="yes"
+ cp /root/.psqlrc /root/.psqlrc.provision.backup
+ fi
+else
+ DELETE_PSQL="yes"
+fi
+
+echo '\pset pager off' >> /root/.psqlrc
+# END FIXME! #16992 Temporary fix for psql call in arvados-api-server
+
+# Now run the install
+salt-call --local state.apply -l debug
+
+# FIXME! #16992 Temporary fix for psql call in arvados-api-server
+if [ "x${DELETE_PSQL}" = "xyes" ]; then
+ echo "Removing .psql file"
+ rm /root/.psqlrc
+fi
+
+if [ "x${RESTORE_PSQL}" = "xyes" ]; then
+ echo "Restroting .psql file"
+ mv -v /root/.psqlrc.provision.backup /root/.psqlrc
+fi
+# END FIXME! #16992 Temporary fix for psql call in arvados-api-server
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+# The variables commented out are the default values that the formula uses.
+# The uncommented values are REQUIRED values. If you don't set them, running
+# this formula will fail.
+arvados:
+ ### GENERAL CONFIG
+ version: '__VERSION__'
+ ## It makes little sense to disable this flag, but you can, if you want :)
+ # use_upstream_repo: true
+
+ ## Repo URL is built with grains values. If desired, it can be completely
+ ## overwritten with the pillar parameter 'repo_url'
+ # repo:
+ # humanname: Arvados Official Repository
+
+ release: __RELEASE__
+
+ ## IMPORTANT!!!!!
+ ## api, workbench and shell require some gems, so you need to make sure ruby
+ ## and deps are installed in order to install and compile the gems.
+ ## We default to `false` in these two variables as it's expected you already
+ ## manage OS packages with some other tool and you don't want us messing up
+ ## with your setup.
+ ruby:
+ ## We set these to `true` here for testing purposes.
+ ## They both default to `false`.
+ manage_ruby: true
+ manage_gems_deps: true
+ # pkg: ruby
+ # gems_deps:
+ # - curl
+ # - g++
+ # - gcc
+ # - git
+ # - libcurl4
+ # - libcurl4-gnutls-dev
+ # - libpq-dev
+ # - libxml2
+ # - libxml2-dev
+ # - make
+ # - python3-dev
+ # - ruby-dev
+ # - zlib1g-dev
+
+ # config:
+ # file: /etc/arvados/config.yml
+ # user: root
+ ## IMPORTANT!!!!!
+ ## If you're intalling any of the rails apps (api, workbench), the group
+ ## should be set to that of the web server, usually `www-data`
+ # group: root
+ # mode: 640
+
+ ### ARVADOS CLUSTER CONFIG
+ cluster:
+ name: __CLUSTER__
+ domain: __DOMAIN__
+
+ database:
+ # max concurrent connections per arvados server daemon
+ # connection_pool_max: 32
+ name: arvados
+ host: 127.0.0.1
+ password: changeme_arvados
+ user: arvados
+ encoding: en_US.utf8
+ client_encoding: UTF8
+
+ tls:
+ # certificate: ''
+ # key: ''
+ # required to test with snakeoil certs
+ insecure: true
+
+ ### TOKENS
+ tokens:
+ system_root: changeme_system_root_token
+ management: changeme_management_token
+ rails_secret: changeme_rails_secret_token
+ anonymous_user: changeme_anonymous_user_token
+
+ ### KEYS
+ secrets:
+ blob_signing_key: changeme_blob_signing_key
+ workbench_secret_key: changeme_workbench_secret_key
+ dispatcher_access_key: changeme_dispatcher_access_key
+ dispatcher_secret_key: changeme_dispatcher_secret_key
+ keep_access_key: changeme_keep_access_key
+ keep_secret_key: changeme_keep_secret_key
+
+ Login:
+ Test:
+ Enable: true
+ Users:
+ __INITIAL_USER__:
+ Email: __INITIAL_USER_EMAIL__
+ Password: __INITIAL_USER_PASSWORD__
+
+ ### VOLUMES
+ ## This should usually match all your `keepstore` instances
+ Volumes:
+ # the volume name will be composed with
+ # <cluster>-nyw5e-<volume>
+ __CLUSTER__-nyw5e-000000000000000:
+ AccessViaHosts:
+ http://keep0.__CLUSTER__.__DOMAIN__:25107:
+ ReadOnly: false
+ Replication: 2
+ Driver: Directory
+ DriverParameters:
+ Root: /tmp
+
+ Users:
+ NewUsersAreActive: true
+ AutoAdminFirstUser: true
+ AutoSetupNewUsers: true
+ AutoSetupNewUsersWithRepository: true
+
+ Services:
+ Controller:
+ ExternalURL: https://__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ InternalURLs:
+ http://127.0.0.2:8003: {}
+ DispatchCloud:
+ InternalURLs:
+ http://__CLUSTER__.__DOMAIN__:9006: {}
+ Keepbalance:
+ InternalURLs:
+ http://__CLUSTER__.__DOMAIN__:9005: {}
+ Keepproxy:
+ ExternalURL: https://keep.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ InternalURLs:
+ http://127.0.0.2:25100: {}
+ Keepstore:
+ InternalURLs:
+ http://keep0.__CLUSTER__.__DOMAIN__:25107: {}
+ RailsAPI:
+ InternalURLs:
+ http://127.0.0.2:8004: {}
+ WebDAV:
+ ExternalURL: https://collections.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ InternalURLs:
+ http://127.0.0.2:9002: {}
+ WebDAVDownload:
+ ExternalURL: https://download.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ WebShell:
+ ExternalURL: https://webshell.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ Websocket:
+ ExternalURL: wss://ws.__CLUSTER__.__DOMAIN__/websocket
+ InternalURLs:
+ http://127.0.0.2:8005: {}
+ Workbench1:
+ ExternalURL: https://workbench.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
+ Workbench2:
+ ExternalURL: https://workbench2.__CLUSTER__.__DOMAIN__:__HOST_SSL_PORT__
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+locale:
+ present:
+ - "en_US.UTF-8 UTF-8"
+ default:
+ # Note: On debian systems don't write the second 'UTF-8' here or you will
+ # experience salt problems like: LookupError: unknown encoding: utf_8_utf_8
+ # Restart the minion after you corrected this!
+ name: 'en_US.UTF-8'
+ requires: 'en_US.UTF-8 UTF-8'
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### ARVADOS
+arvados:
+ config:
+ group: www-data
+
+### NGINX
+nginx:
+ ### SITES
+ servers:
+ managed:
+ arvados_api:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - listen: '127.0.0.2:8004'
+ - server_name: api
+ - root: /var/www/arvados-api/current/public
+ - index: index.html index.htm
+ - access_log: /var/log/nginx/api.__CLUSTER__.__DOMAIN__-upstream.access.log combined
+ - error_log: /var/log/nginx/api.__CLUSTER__.__DOMAIN__-upstream.error.log
+ - passenger_enabled: 'on'
+ - client_max_body_size: 128m
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### NGINX
+nginx:
+ ### SERVER
+ server:
+ config:
+ ### STREAMS
+ http:
+ 'geo $external_client':
+ default: 1
+ '127.0.0.0/8': 0
+ upstream controller_upstream:
+ - server: '127.0.0.2:8003 fail_timeout=10s'
+
+ ### SITES
+ servers:
+ managed:
+ ### DEFAULT
+ arvados_controller_default:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: __CLUSTER__.__DOMAIN__
+ - listen:
+ - 80 default
+ - location /.well-known:
+ - root: /var/www
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ arvados_controller_ssl:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: __CLUSTER__.__DOMAIN__
+ - listen:
+ - __HOST_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /:
+ - proxy_pass: 'http://controller_upstream'
+ - proxy_read_timeout: 300
+ - proxy_connect_timeout: 90
+ - proxy_redirect: 'off'
+ - proxy_set_header: X-Forwarded-Proto https
+ - proxy_set_header: 'Host $http_host'
+ - proxy_set_header: 'X-Real-IP $remote_addr'
+ - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
+ - proxy_set_header: 'X-External-Client $external_client'
+ # - include: 'snippets/letsencrypt.conf'
+ - include: 'snippets/snakeoil.conf'
+ - access_log: /var/log/nginx/__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/__CLUSTER__.__DOMAIN__.error.log
+ - client_max_body_size: 128m
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### NGINX
+nginx:
+ ### SERVER
+ server:
+ config:
+ ### STREAMS
+ http:
+ upstream keepproxy_upstream:
+ - server: '127.0.0.2:25100 fail_timeout=10s'
+
+ servers:
+ managed:
+ ### DEFAULT
+ arvados_keepproxy_default:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: keep.__CLUSTER__.__DOMAIN__
+ - listen:
+ - 80
+ - location /.well-known:
+ - root: /var/www
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ arvados_keepproxy_ssl:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: keep.__CLUSTER__.__DOMAIN__
+ - listen:
+ - __HOST_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /:
+ - proxy_pass: 'http://keepproxy_upstream'
+ - proxy_read_timeout: 90
+ - proxy_connect_timeout: 90
+ - proxy_redirect: 'off'
+ - proxy_set_header: X-Forwarded-Proto https
+ - proxy_set_header: 'Host $http_host'
+ - proxy_set_header: 'X-Real-IP $remote_addr'
+ - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
+ - proxy_buffering: 'off'
+ - client_body_buffer_size: 64M
+ - client_max_body_size: 64M
+ - proxy_http_version: '1.1'
+ - proxy_request_buffering: 'off'
+ # - include: 'snippets/letsencrypt.conf'
+ - include: 'snippets/snakeoil.conf'
+ - access_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.error.log
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### NGINX
+nginx:
+ ### SERVER
+ server:
+ config:
+ ### STREAMS
+ http:
+ upstream collections_downloads_upstream:
+ - server: '127.0.0.2:9002 fail_timeout=10s'
+
+ servers:
+ managed:
+ ### DEFAULT
+ arvados_collections_download_default:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - listen:
+ - 80
+ - location /.well-known:
+ - root: /var/www
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ ### COLLECTIONS / DOWNLOAD
+ arvados_collections_download_ssl:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - listen:
+ - __HOST_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /:
+ - proxy_pass: 'http://collections_downloads_upstream'
+ - proxy_read_timeout: 90
+ - proxy_connect_timeout: 90
+ - proxy_redirect: 'off'
+ - proxy_set_header: X-Forwarded-Proto https
+ - proxy_set_header: 'Host $http_host'
+ - proxy_set_header: 'X-Real-IP $remote_addr'
+ - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
+ - proxy_buffering: 'off'
+ - client_max_body_size: 0
+ - proxy_http_version: '1.1'
+ - proxy_request_buffering: 'off'
+ # - include: 'snippets/letsencrypt.conf'
+ - include: 'snippets/snakeoil.conf'
+ - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### NGINX
+nginx:
+ install_from_phusionpassenger: true
+ lookup:
+ passenger_package: libnginx-mod-http-passenger
+ passenger_config_file: /etc/nginx/conf.d/mod-http-passenger.conf
+
+ ### SERVER
+ server:
+ config:
+ include: 'modules-enabled/*.conf'
+ worker_processes: 4
+
+ ### SITES
+ servers:
+ managed:
+ # Remove default webserver
+ default:
+ enabled: false
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### NGINX
+nginx:
+ ### SERVER
+ server:
+ config:
+
+ ### STREAMS
+ http:
+ upstream webshell_upstream:
+ - server: '127.0.0.2:4200 fail_timeout=10s'
+
+ ### SITES
+ servers:
+ managed:
+ arvados_webshell_default:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: webshell.__CLUSTER__.__DOMAIN__
+ - listen:
+ - 80
+ - location /.well-known:
+ - root: /var/www
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ arvados_webshell_ssl:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: webshell.__CLUSTER__.__DOMAIN__
+ - listen:
+ - __HOST_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /shell.__CLUSTER__.__DOMAIN__:
+ - proxy_pass: 'http://webshell_upstream'
+ - proxy_read_timeout: 90
+ - proxy_connect_timeout: 90
+ - proxy_set_header: 'Host $http_host'
+ - proxy_set_header: 'X-Real-IP $remote_addr'
+ - proxy_set_header: X-Forwarded-Proto https
+ - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
+ - proxy_ssl_session_reuse: 'off'
+
+ - "if ($request_method = 'OPTIONS')":
+ - add_header: "'Access-Control-Allow-Origin' '*'"
+ - add_header: "'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'"
+ - add_header: "'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'"
+ - add_header: "'Access-Control-Max-Age' 1728000"
+ - add_header: "'Content-Type' 'text/plain charset=UTF-8'"
+ - add_header: "'Content-Length' 0"
+ - return: 204
+
+ - "if ($request_method = 'POST')":
+ - add_header: "'Access-Control-Allow-Origin' '*'"
+ - add_header: "'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'"
+ - add_header: "'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'"
+
+ - "if ($request_method = 'GET')":
+ - add_header: "'Access-Control-Allow-Origin' '*'"
+ - add_header: "'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'"
+ - add_header: "'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'"
+
+ # - include: 'snippets/letsencrypt.conf'
+ - include: 'snippets/snakeoil.conf'
+ - access_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.error.log
+
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### NGINX
+nginx:
+ ### SERVER
+ server:
+ config:
+ ### STREAMS
+ http:
+ upstream websocket_upstream:
+ - server: '127.0.0.2:8005 fail_timeout=10s'
+
+ servers:
+ managed:
+ ### DEFAULT
+ arvados_websocket_default:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: ws.__CLUSTER__.__DOMAIN__
+ - listen:
+ - 80
+ - location /.well-known:
+ - root: /var/www
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ arvados_websocket_ssl:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: ws.__CLUSTER__.__DOMAIN__
+ - listen:
+ - __HOST_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /:
+ - proxy_pass: 'http://websocket_upstream'
+ - proxy_read_timeout: 600
+ - proxy_connect_timeout: 90
+ - proxy_redirect: 'off'
+ - proxy_set_header: 'Host $host'
+ - proxy_set_header: 'X-Real-IP $remote_addr'
+ - proxy_set_header: 'Upgrade $http_upgrade'
+ - proxy_set_header: 'Connection "upgrade"'
+ - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
+ - proxy_buffering: 'off'
+ - client_body_buffer_size: 64M
+ - client_max_body_size: 64M
+ - proxy_http_version: '1.1'
+ - proxy_request_buffering: 'off'
+ # - include: 'snippets/letsencrypt.conf'
+ - include: 'snippets/snakeoil.conf'
+ - access_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.error.log
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### ARVADOS
+arvados:
+ config:
+ group: www-data
+
+### NGINX
+nginx:
+ ### SITES
+ servers:
+ managed:
+ ### DEFAULT
+ arvados_workbench2_default:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: workbench2.__CLUSTER__.__DOMAIN__
+ - listen:
+ - 80
+ - location /.well-known:
+ - root: /var/www
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ arvados_workbench2_ssl:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: workbench2.__CLUSTER__.__DOMAIN__
+ - listen:
+ - __HOST_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /:
+ - root: /var/www/arvados-workbench2/workbench2
+ - try_files: '$uri $uri/ /index.html'
+ - 'if (-f $document_root/maintenance.html)':
+ - return: 503
+ - location /config.json:
+ - return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__"}' ~ "'" }}
+ # - include: 'snippets/letsencrypt.conf'
+ - include: 'snippets/snakeoil.conf'
+ - access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### ARVADOS
+arvados:
+ config:
+ group: www-data
+
+### NGINX
+nginx:
+ ### SERVER
+ server:
+ config:
+
+ ### STREAMS
+ http:
+ upstream workbench_upstream:
+ - server: '127.0.0.2:9000 fail_timeout=10s'
+
+ ### SITES
+ servers:
+ managed:
+ ### DEFAULT
+ arvados_workbench_default:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: workbench.__CLUSTER__.__DOMAIN__
+ - listen:
+ - 80
+ - location /.well-known:
+ - root: /var/www
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ arvados_workbench_ssl:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: workbench.__CLUSTER__.__DOMAIN__
+ - listen:
+ - __HOST_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /:
+ - proxy_pass: 'http://workbench_upstream'
+ - proxy_read_timeout: 300
+ - proxy_connect_timeout: 90
+ - proxy_redirect: 'off'
+ - proxy_set_header: X-Forwarded-Proto https
+ - proxy_set_header: 'Host $http_host'
+ - proxy_set_header: 'X-Real-IP $remote_addr'
+ - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
+ # - include: 'snippets/letsencrypt.conf'
+ - include: 'snippets/snakeoil.conf'
+ - access_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.error.log
+
+ arvados_workbench_upstream:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - listen: '127.0.0.2:9000'
+ - server_name: workbench
+ - root: /var/www/arvados-workbench/current/public
+ - index: index.html index.htm
+ - passenger_enabled: 'on'
+ # yamllint disable-line rule:line-length
+ - access_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__-upstream.access.log combined
+ - error_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__-upstream.error.log
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### POSTGRESQL
+postgres:
+ use_upstream_repo: false
+ pkgs_extra:
+ - postgresql-contrib
+ postgresconf: |-
+ listen_addresses = '*' # listen on all interfaces
+ acls:
+ - ['local', 'all', 'postgres', 'peer']
+ - ['local', 'all', 'all', 'peer']
+ - ['host', 'all', 'all', '127.0.0.1/32', 'md5']
+ - ['host', 'all', 'all', '::1/128', 'md5']
+ - ['host', 'arvados', 'arvados', '127.0.0.1/32']
+ users:
+ arvados:
+ ensure: present
+ password: changeme_arvados
+
+ # tablespaces:
+ # arvados_tablespace:
+ # directory: /path/to/some/tbspace/arvados_tbsp
+ # owner: arvados
+
+ databases:
+ arvados:
+ owner: arvados
+ template: template0
+ lc_ctype: en_US.utf8
+ lc_collate: en_US.utf8
+ # tablespace: arvados_tablespace
+ schemas:
+ public:
+ owner: arvados
+ extensions:
+ pg_trgm:
+ if_not_exists: true
+ schema: public
projects / arvados.git / commitdiff