Arvados support for shell nodes allows you to use Arvados permissions to grant Linux shell accounts to users.
-A shell node runs the @arvados-login-sync@ service, and has some additional configuration to make it convenient for users to use Arvados utilites and SDKs. Users are allowed to log in and run arbitrary programs. For optimal performance, the Arvados shell server should be on the same LAN as the Arvados cluster.
+A shell node runs the @arvados-login-sync@ service to manage user accounts, and typically has Arvados utilities and SDKs pre-installed. Users are allowed to log in and run arbitrary programs. For optimal performance, the Arvados shell server should be on the same LAN as the Arvados cluster.
-Because it _contains secrets_ shell nodes should *not* have a copy of the complete @config.yml@. For example, if users have access to the @docker@ daemon, it is trival to gain *root* access to any file on the system. Users sharing a shell node should be implicitly trusted, or not given access to Docker. In more secure environments, the admin should allocate a separate VM for each user.
+Because it _contains secrets_ shell nodes should *not* have a copy of the Arvados @config.yml@.
+
+Shell nodes should be separate virtual machines from the VMs running other Arvados services. You may choose to grant root access to users so that they can customize the node, for example, installing new programs. This has security considerations depending on whether a shell node is single-user or multi-user.
+
+A single-user shell node should be set up so that it only stores Arvados access tokens that belong to that user. In that case, that user can be safely granted root access without compromising other Arvados users.
+
+In the multi-user shell node case, a malicious user with @root@ access could access other user's Arvados tokens. Users should only be given @root@ access on a multi-user shell node if you would trust them them to be Arvados administrators. Be aware that with access to the @docker@ daemon, it is trival to gain *root* access to any file on the system, so giving users @docker@ access should be considered equivalent to @root@ access.
h2(#dependencies). Install Dependecies and SDKs
h2(#vm-record). Create record for VM
-This program makes it possible for Arvados users to log in to the shell server -- subject to permissions assigned by the Arvados administrator -- using the SSH keys they upload to Workbench. It sets up login accounts, updates group membership, and adds users' public keys to the appropriate @authorized_keys@ files.
-
-Create an Arvados virtual_machine object representing this shell server. This will assign a UUID.
+As an admin, create an Arvados virtual_machine object representing this shell server. This will return a uuid.
<notextile>
<pre>
-<code>apiserver:~$ <span class="userinput">arv --format=uuid virtual_machine create --virtual-machine '{"hostname":"<b>your.shell.server.hostname.without.domain</b>"}'</span>
+<code>apiserver:~$ <span class="userinput">arv --format=uuid virtual_machine create --virtual-machine '{"hostname":"<b>shell.ClusterID.example.com</b>"}'</span>
zzzzz-2x53u-zzzzzzzzzzzzzzz</code>
</pre>
</notextile>
-h2(#scoped-token). Create scoped token
+h2(#arvados-login-sync). Install arvados-login-sync
+
+The @arvados-login-sync@ service makes it possible for Arvados users to log in to the shell server. It sets up login accounts, updates group membership, adds each user's SSH public keys to the @~/.ssh/authorized_keys@ file, and adds an Arvados token to @~/.config/arvados/settings.conf@ .
-As an Arvados admin user (such as the system root user), create a "scoped token":{{site.baseurl}}/admin/scoped-tokens.html that is permits only reading login information for this VM. Setting a scope on the token means that even though a user with root access on the shell node can access the token, the token is not usable for admin actions on Arvados.
+Install the @arvados-login-sync@ program from RubyGems.
<notextile>
<pre>
-<code>apiserver:~$ <span class="userinput">arv api_client_authorization create --api-client-authorization '{"scopes":["GET /arvados/v1/virtual_machines/<b>zzzzz-2x53u-zzzzzzzzzzzzzzz</b>/logins"]}'</span>
-{
- ...
- "api_token":"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz",
- ...
-}</code>
+<code>shellserver:# <span class="userinput">gem install arvados-login-sync</span></code>
</pre>
</notextile>
-Note the UUID and the API token output by the above commands: you will need them in a minute.
+h2(#arvados-login-sync). Run arvados-login-sync periodically
-h2(#arvados-login-sync). Install arvados-login-sync
+Create a cron job to run the @arvados-login-sync@ program every 2 minutes. This will synchronize user accounts.
-Install the arvados-login-sync program from RubyGems.
+If this is a single-user shell node, then @ARVADOS_API_TOKEN@ should be a token for that user. See "Create a token for a user":{{site.baseurl}}/admin/user-management-cli.html#create-token .
-<notextile>
-<pre>
-<code>shellserver:# <span class="userinput">gem install arvados-login-sync</span></code>
-</pre>
-</notextile>
+If this is a multi-user shell node, then @ARVADOS_API_TOKEN@ should be an administrator token such as the @SystemRootToken@. See discussion in the "introduction":#introduction about security on multi-user shell nodes.
-Configure cron to run the @arvados-login-sync@ program every 2 minutes.
+Set @ARVADOS_VIRTUAL_MACHINE_UUID@ to the UUID from "Create record for VM":#vm-record
<notextile>
<pre>
-<code>shellserver:# <span class="userinput">umask 077; tee /etc/cron.d/arvados-login-sync <<EOF
+<code>shellserver:# <span class="userinput">umask 0700; tee /etc/cron.d/arvados-login-sync <<EOF
ARVADOS_API_HOST="<strong>ClusterID.example.com</strong>"
-ARVADOS_API_TOKEN="<strong>the_token_you_created_above</strong>"
+ARVADOS_API_TOKEN="<strong>xxxxxxxxxxxxxxxxx</strong>"
ARVADOS_VIRTUAL_MACHINE_UUID="<strong>zzzzz-2x53u-zzzzzzzzzzzzzzz</strong>"
*/2 * * * * root arvados-login-sync
EOF</span></code>
A user should be able to log in to the shell server when the following conditions are satisfied:
-# The user has uploaded an SSH public key: Workbench → Account menu → "SSH keys" item → "Add new SSH key" button.
# As an admin user, you have given the user permission to log in using the Workbench → Admin menu → "Users" item → "Show" button → "Admin" tab → "Setup account" button.
# The cron job has run.
+In order to log in via SSH, the user must also upload an SSH public key. Alternately, if configured, users can log in using "Webshell":install-webshell.html .
+
See also "how to add a VM login permission link at the command line":../admin/user-management-cli.html
location /<span class="userinput">shell.ClusterID</span> {
if ($request_method = 'OPTIONS') {
- add_header 'Access-Control-Allow-Origin' '*';
+ add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
add_header 'Access-Control-Max-Age' 1728000;
h2(#config-pam). Configure pam
-Use a text editor to create a new file @/etc/pam.d/shellinabox@ with the following configuration. Options that need attention are marked in <span class="userinput">red</span>.
+Use a text editor to create a new file @/etc/pam.d/shellinabox@ with the following configuration. Options that need attention are marked in <span class="userinput">red</span>.
<notextile><pre>
# This example is a stock debian "login" file with pam_arvados
session required pam_env.so readenv=1
session required pam_env.so readenv=1 envfile=/etc/default/locale
+# The first argument is the address of the API server. The second
+# argument is this shell node's hostname. The hostname must match the
+# "hostname" field of the virtual_machine record.
auth [success=1 default=ignore] /usr/lib/pam_arvados.so <span class="userinput">ClusterID.example.com</span> <span class="userinput">shell.ClusterID.example.com</span>
+
auth requisite pam_deny.so
auth required pam_permit.so
h2(#confirm-working). Confirm working installation
-A user should be able to log in to the shell server, using webshell via workbench. Please refer to "Accessing an Arvados VM with Webshell":{{site.baseurl}}/user/getting_started/vm-login-with-webshell.html
-
+A user should now be able to log in to the shell server, using webshell via workbench. Please refer to "Accessing an Arvados VM with Webshell":{{site.baseurl}}/user/getting_started/vm-login-with-webshell.html
RUN echo cwl_runner_version $cwl_runner_version python_sdk_version $python_sdk_version
RUN apt-get update -q
-RUN apt-get install -yq --no-install-recommends nodejs \
- python-arvados-python-client=$python_sdk_version \
- python3-arvados-cwl-runner=$cwl_runner_version
+RUN apt-get install -yq --no-install-recommends python3-arvados-cwl-runner=$cwl_runner_version
# use the Python executable from the python-arvados-cwl-runner package
-RUN rm -f /usr/bin/python && ln -s /usr/share/python2.7/dist/python-arvados-python-client/bin/python /usr/bin/python
+RUN rm -f /usr/bin/python && ln -s /usr/share/python3/dist/python3-arvados-cwl-runner/bin/python /usr/bin/python
RUN rm -f /usr/bin/python3 && ln -s /usr/share/python3/dist/python3-arvados-cwl-runner/bin/python /usr/bin/python3
# Install dependencies and set up system.
if err != nil {
return err
}
+ err = super.RunProgram(ctx, "services/api", nil, railsEnv, "bundle", "exec", "./script/get_anonymous_user_token.rb")
+ if err != nil {
+ return err
+ }
return nil
}
if cluster.Collections.BlobSigningKey == "" {
cluster.Collections.BlobSigningKey = randomHexString(64)
}
+ if cluster.Users.AnonymousUserToken == "" {
+ cluster.Users.AnonymousUserToken = randomHexString(64)
+ }
+
if super.ClusterType != "production" && cluster.Containers.DispatchPrivateKey == "" {
buf, err := ioutil.ReadFile(filepath.Join(super.SourcePath, "lib", "dispatchcloud", "test", "sshkey_dispatch"))
if err != nil {
"bytes"
"context"
"encoding/json"
+ "fmt"
"io"
"math"
"net"
}
}
+// Get rpc connection struct initialized to communicate with the
+// specified cluster.
func (s *IntegrationSuite) conn(clusterID string) *rpc.Conn {
return rpc.NewConn(clusterID, s.testClusters[clusterID].controllerURL, true, rpc.PassthroughTokenProvider)
}
+// Return Context, Arvados.Client and keepclient structs initialized
+// to connect to the specified cluster (by clusterID) using with the supplied
+// Arvados token.
func (s *IntegrationSuite) clientsWithToken(clusterID string, token string) (context.Context, *arvados.Client, *keepclient.KeepClient) {
cl := s.testClusters[clusterID].config.Clusters[clusterID]
ctx := auth.NewContext(context.Background(), auth.NewCredentials(token))
return ctx, ac, kc
}
+// Log in as a user called "example", get the user's API token,
+// initialize clients with the API token, set up the user and
+// optionally activate the user. Return client structs for
+// communicating with the cluster on behalf of the 'example' user.
func (s *IntegrationSuite) userClients(rootctx context.Context, c *check.C, conn *rpc.Conn, clusterID string, activate bool) (context.Context, *arvados.Client, *keepclient.KeepClient) {
login, err := conn.UserSessionCreate(rootctx, rpc.UserSessionCreateOptions{
ReturnTo: ",https://example.com",
return ctx, ac, kc
}
+// Return Context, arvados.Client and keepclient structs initialized
+// to communicate with the cluster as the system root user.
func (s *IntegrationSuite) rootClients(clusterID string) (context.Context, *arvados.Client, *keepclient.KeepClient) {
return s.clientsWithToken(clusterID, s.testClusters[clusterID].config.Clusters[clusterID].SystemRootToken)
}
+// Return Context, arvados.Client and keepclient structs initialized
+// to communicate with the cluster as the anonymous user.
+func (s *IntegrationSuite) anonymousClients(clusterID string) (context.Context, *arvados.Client, *keepclient.KeepClient) {
+ return s.clientsWithToken(clusterID, s.testClusters[clusterID].config.Clusters[clusterID].Users.AnonymousUserToken)
+}
+
func (s *IntegrationSuite) TestGetCollectionByPDH(c *check.C) {
conn1 := s.conn("z1111")
rootctx1, _, _ := s.rootClients("z1111")
c.Check(coll.PortableDataHash, check.Equals, pdh)
}
+func (s *IntegrationSuite) TestGetCollectionAsAnonymous(c *check.C) {
+ conn1 := s.conn("z1111")
+ conn3 := s.conn("z3333")
+ rootctx1, rootac1, rootkc1 := s.rootClients("z1111")
+ anonctx3, anonac3, _ := s.anonymousClients("z3333")
+
+ // Make sure anonymous token was set
+ c.Assert(anonac3.AuthToken, check.Not(check.Equals), "")
+
+ // Create the collection to find its PDH (but don't save it
+ // anywhere yet)
+ var coll1 arvados.Collection
+ fs1, err := coll1.FileSystem(rootac1, rootkc1)
+ c.Assert(err, check.IsNil)
+ f, err := fs1.OpenFile("test.txt", os.O_CREATE|os.O_RDWR, 0777)
+ c.Assert(err, check.IsNil)
+ _, err = io.WriteString(f, "IntegrationSuite.TestGetCollectionAsAnonymous")
+ c.Assert(err, check.IsNil)
+ err = f.Close()
+ c.Assert(err, check.IsNil)
+ mtxt, err := fs1.MarshalManifest(".")
+ c.Assert(err, check.IsNil)
+ pdh := arvados.PortableDataHash(mtxt)
+
+ // Save the collection on cluster z1111.
+ coll1, err = conn1.CollectionCreate(rootctx1, arvados.CreateOptions{Attrs: map[string]interface{}{
+ "manifest_text": mtxt,
+ }})
+ c.Assert(err, check.IsNil)
+
+ // Share it with the anonymous users group.
+ var outLink arvados.Link
+ err = rootac1.RequestAndDecode(&outLink, "POST", "/arvados/v1/links", nil,
+ map[string]interface{}{"link": map[string]interface{}{
+ "link_class": "permission",
+ "name": "can_read",
+ "tail_uuid": "z1111-j7d0g-anonymouspublic",
+ "head_uuid": coll1.UUID,
+ },
+ })
+ c.Check(err, check.IsNil)
+
+ // Current user should be z3 anonymous user
+ outUser, err := anonac3.CurrentUser()
+ c.Check(err, check.IsNil)
+ c.Check(outUser.UUID, check.Equals, "z3333-tpzed-anonymouspublic")
+
+ // Get the token uuid
+ var outAuth arvados.APIClientAuthorization
+ err = anonac3.RequestAndDecode(&outAuth, "GET", "/arvados/v1/api_client_authorizations/current", nil, nil)
+ c.Check(err, check.IsNil)
+
+ // Make a v2 token of the z3 anonymous user, and use it on z1
+ _, anonac1, _ := s.clientsWithToken("z1111", fmt.Sprintf("v2/%v/%v", outAuth.UUID, outAuth.APIToken))
+ outUser2, err := anonac1.CurrentUser()
+ c.Check(err, check.IsNil)
+ // z3 anonymous user will be mapped to the z1 anonymous user
+ c.Check(outUser2.UUID, check.Equals, "z1111-tpzed-anonymouspublic")
+
+ // Retrieve the collection (which is on z1) using anonymous from cluster z3333.
+ coll, err := conn3.CollectionGet(anonctx3, arvados.GetOptions{UUID: coll1.UUID})
+ c.Check(err, check.IsNil)
+ c.Check(coll.PortableDataHash, check.Equals, pdh)
+}
+
// Get a token from the login cluster (z1111), use it to submit a
// container request on z2222.
func (s *IntegrationSuite) TestCreateContainerRequestWithFedToken(c *check.C) {
# Add or update user and token in local database so we can
# validate subsequent requests faster.
+ if remote_user['uuid'][-22..-1] == '-tpzed-anonymouspublic'
+ # Special case: map the remote anonymous user to local anonymous user
+ remote_user['uuid'] = anonymous_user_uuid
+ end
+
user = User.find_by_uuid(remote_user['uuid'])
if !user
user.send(attr+'=', remote_user[attr])
end
+ if remote_user['uuid'][-22..-1] == '-tpzed-000000000000000'
+ user.first_name = "root"
+ user.last_name = "from cluster #{remote_user_prefix}"
+ end
+
act_as_system_user do
if user.is_active && !remote_user['is_active']
user.unsetup
act_as_system_user
def create_api_client_auth(supplied_token=nil)
+ supplied_token = Rails.configuration.Users["AnonymousUserToken"]
- # If token is supplied, see if it exists
- if supplied_token
- api_client_auth = ApiClientAuthorization.
- where(api_token: supplied_token).
- first
- if !api_client_auth
- # fall through to create a token
- else
- raise "Token exists, aborting!"
+ if supplied_token.nil? or supplied_token.empty?
+ puts "Users.AnonymousUserToken is empty. Destroying tokens that belong to anonymous."
+ # Token is empty. Destroy any anonymous tokens.
+ ApiClientAuthorization.where(user: anonymous_user).destroy_all
+ return nil
+ end
+
+ attr = {user: anonymous_user,
+ api_client_id: 0,
+ scopes: ['GET /']}
+
+ secret = supplied_token
+
+ if supplied_token[0..2] == 'v2/'
+ _, token_uuid, secret, optional = supplied_token.split('/')
+ if token_uuid[0..4] != Rails.configuration.ClusterID
+ # Belongs to a different cluster.
+ puts supplied_token
+ return nil
end
+ attr[:uuid] = token_uuid
end
- api_client_auth = ApiClientAuthorization.
- new(user: anonymous_user,
- api_client_id: 0,
- expires_at: Time.now + 100.years,
- scopes: ['GET /'],
- api_token: supplied_token)
- api_client_auth.save!
- api_client_auth.reload
+ attr[:api_token] = secret
+
+ api_client_auth = ApiClientAuthorization.where(attr).first
+ if !api_client_auth
+ api_client_auth = ApiClientAuthorization.create!(attr)
+ end
api_client_auth
end
end
# print it to the console
-puts api_client_auth.api_token
+if api_client_auth
+ puts "v2/#{api_client_auth.uuid}/#{api_client_auth.api_token}"
+end
Arvados::V1::SchemaController.any_instance.stubs(:root_url).returns "https://#{@remote_host[0]}"
@stub_status = 200
@stub_content = {
- uuid: 'zbbbb-tpzed-000000000000000',
+ uuid: 'zbbbb-tpzed-000000000000001',
email: 'foo@example.com',
username: 'barney',
is_admin: true,
params: {format: 'json'},
headers: auth(remote: 'zbbbb')
assert_response :success
- assert_equal 'zbbbb-tpzed-000000000000000', json_response['uuid']
+ assert_equal 'zbbbb-tpzed-000000000000001', json_response['uuid']
assert_equal false, json_response['is_admin']
assert_equal false, json_response['is_active']
assert_equal 'foo@example.com', json_response['email']
params: {format: 'json'},
headers: auth(remote: 'zbbbb')
assert_response :success
- assert_equal 'zbbbb-tpzed-000000000000000', json_response['uuid']
+ assert_equal 'zbbbb-tpzed-000000000000001', json_response['uuid']
assert_equal false, json_response['is_admin']
assert_equal false, json_response['is_active']
assert_equal 'foo@example.com', json_response['email']
assert_equal 'barney', json_response['username']
- post '/arvados/v1/users/zbbbb-tpzed-000000000000000/activate',
+ post '/arvados/v1/users/zbbbb-tpzed-000000000000001/activate',
params: {format: 'json'},
headers: auth(remote: 'zbbbb')
assert_response 422
params: {format: 'json'},
headers: auth(remote: 'zbbbb')
assert_response :success
- assert_equal 'zbbbb-tpzed-000000000000000', json_response['uuid']
+ assert_equal 'zbbbb-tpzed-000000000000001', json_response['uuid']
assert_equal false, json_response['is_admin']
assert_equal true, json_response['is_active']
assert_equal 'foo@example.com', json_response['email']
params: {format: 'json'},
headers: auth(remote: 'zbbbb')
assert_response :success
- assert_equal 'zbbbb-tpzed-000000000000000', json_response['uuid']
+ assert_equal 'zbbbb-tpzed-000000000000001', json_response['uuid']
assert_equal true, json_response['is_admin']
assert_equal true, json_response['is_active']
assert_equal 'foo@example.com', json_response['email']
end
end
+ test 'authenticate with remote token, remote user is system user' do
+ @stub_content[:uuid] = 'zbbbb-tpzed-000000000000000'
+ get '/arvados/v1/users/current',
+ params: {format: 'json'},
+ headers: auth(remote: 'zbbbb')
+ assert_equal 'from cluster zbbbb', json_response['last_name']
+ end
+
+ test 'authenticate with remote token, remote user is anonymous user' do
+ @stub_content[:uuid] = 'zbbbb-tpzed-anonymouspublic'
+ get '/arvados/v1/users/current',
+ params: {format: 'json'},
+ headers: auth(remote: 'zbbbb')
+ assert_response :success
+ assert_equal 'zzzzz-tpzed-anonymouspublic', json_response['uuid']
+ end
+
+
end
userdotconfig = File.join(homedir, ".config")
if !File.exist?(userdotconfig)
Dir.mkdir(userdotconfig)
- FileUtils.chown_R(l[:username], nil, userdotconfig)
- File.chmod(0700, userdotconfig)
end
configarvados = File.join(userdotconfig, "arvados")
f.write("ARVADOS_API_HOST=#{ENV['ARVADOS_API_HOST']}\n")
f.write("ARVADOS_API_TOKEN=v2/#{user_token[:uuid]}/#{user_token[:api_token]}\n")
f.close()
- File.chmod(0600, tokenfile)
end
rescue => e
STDERR.puts "Error setting token for #{l[:username]}: #{e}"
end
FileUtils.chown_R(l[:username], nil, userdotssh)
- FileUtils.chown_R(l[:username], nil, configarvados)
+ FileUtils.chown_R(l[:username], nil, userdotconfig)
File.chmod(0700, userdotssh)
+ File.chmod(0700, userdotconfig)
+ File.chmod(0700, configarvados)
File.chmod(0750, homedir)
File.chmod(0600, keysfile)
- File.chmod(0700, configarvados)
+ if File.exist?(tokenfile)
+ File.chmod(0600, tokenfile)
+ end
end
rescue Exception => bang
projects / arvados.git / commitdiff