railties (>= 4)
request_store (~> 1.0)
logstash-event (1.2.02)
- loofah (2.18.0)
+ loofah (2.19.1)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
mail (2.7.1)
net-ssh-gateway (2.0.0)
net-ssh (>= 4.0.0)
nio4r (2.5.8)
- nokogiri (1.13.9)
+ nokogiri (1.13.10)
mini_portile2 (~> 2.8.0)
racc (~> 1.4)
npm-rails (0.2.1)
multi_json (~> 1.0)
websocket-driver (>= 0.2.0)
public_suffix (4.0.6)
- racc (1.6.0)
+ racc (1.6.1)
rack (2.2.4)
rack-mini-profiler (1.0.2)
rack (>= 1.2.0)
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
nokogiri (>= 1.6)
- rails-html-sanitizer (1.4.3)
- loofah (~> 2.3)
+ rails-html-sanitizer (1.4.4)
+ loofah (~> 2.19, >= 2.19.1)
rails-perftest (0.0.7)
railties (5.2.8.1)
actionpack (= 5.2.8.1)
- sdk/index.html.textile.liquid
- Python:
- sdk/python/sdk-python.html.textile.liquid
- - sdk/python/example.html.textile.liquid
+ - sdk/python/api-client.html.textile.liquid
- sdk/python/python.html.textile.liquid
- sdk/python/arvados-fuse.html.textile.liquid
- sdk/python/arvados-cwl-runner.html.textile.liquid
* The name of a role is unique across a single Arvados cluster.
* Roles can be both targets (@head_uuid@) and origins (@tail_uuid@) of permission links.
* By default, all roles are visible to all active users. However, if the configuration entry @Users.RoleGroupsVisibleToAll@ is @false@, visibility is determined by normal permission rules, _i.e._, a role is only visible to users who have that role, and to admins.
+* By default, any user can create a new role. However, if the configuration entry @Users.CanCreateRoleGroups@ is @false@, only admins can create roles.
h3. Access through Roles
--- /dev/null
+---
+layout: default
+navsection: sdk
+navmenu: Python
+title: Arvados API Client
+...
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+{% comment %}
+A note about scope for future authors: This page is meant to be a general guide to using the API client. It is intentionally limited to using the general resource methods as examples, because those are widely available and should be sufficient to give the reader a general understanding of how the API client works. In my opinion we should not cover resource-specific API methods here, and instead prefer to cover them in the cookbook or reference documentation, which have a more appropriate scope. --Brett 2022-12-06
+{% endcomment %}
+
+The Arvados Python SDK provides a complete client interface to the "Arvados API":{{site.baseurl}}/api/index.html. You can use this client interface directly to send requests to your Arvados API server, and many of the higher-level interfaces in the Python SDK accept a client object in their constructor for their use. Any Arvados software you write in Python will likely use these client objects.
+
+This document explains how to instantiate the client object, and how its methods map to the full "Arvados API":{{site.baseurl}}/api/index.html. Refer to the API documentation for full details about all available resources and methods. The rest of the Python SDK documentation after this covers the higher-level interfaces it provides.
+
+h2. Initializing the API client
+
+In the simplest case, you can import the @arvados@ module and call its @api@ method with an API version number:
+
+{% codeblock as python %}
+import arvados
+arv_client = arvados.api('v1')
+{% endcodeblock %}
+
+This will connect to the Arvados API server using the @ARVADOS_API_HOST@, @ARVADOS_API_TOKEN@, and @ARVADOS_API_HOST_INSECURE@ settings from environment variables or @~/.config/arvados/settings.conf@. You can alternatively pass these settings as arguments:
+
+{% codeblock as python %}
+import arvados
+arv_client = arvados.api(
+ 'v1',
+ host='api.arvados.example.com',
+ token='ExampleToken',
+ insecure=False,
+)
+{% endcodeblock %}
+
+Either way, you can now use the @arv_client@ object to send requests to the Arvados API server you specified, using the configured token. The client object queries the API server for its supported API version and methods, so this client object will always support the same API the server does, even when there is a version mismatch between it and the Python SDK.
+
+h2. Resources, methods, and requests
+
+The API client has a method that corresponds to each "type of resource supported by the Arvados API server":{{site.baseurl}}/api/ (listed in the documentation sidebar). You call these methods without any arguments. They return a resource object you use to call a method on that resource type.
+
+Each resource object has a method that corresponds to each API method supported by that resource type. You call these methods with the keyword arguments and values documented in the API reference. They return an API request object.
+
+Each API request object has an @execute()@ method. You may pass a @num_retries@ integer argument to retry the operation that many times, with exponential back-off, in case of temporary errors like network problems. If it ultimately succeeds, it returns the kind of object documented in the API reference for that method. Usually that's a dictionary with details about the object you requested. If there's a problem, it raises an exception.
+
+Putting it all together, basic API requests usually look like:
+
+{% codeblock as python %}
+arv_object = arv_client.resource_type().api_method(
+ argument=...,
+ other_argument=...,
+).execute(num_retries=3)
+{% endcodeblock %}
+
+The following sections detail how to call "common resource methods in the API":{{site.baseurl}}/api/methods.html with more concrete examples. Additional methods may be available on specific resource types.
+
+h2. get method
+
+To fetch a single Arvados object, call the @get@ method of the resource type. You must pass a @uuid@ argument string that identifies the object to fetch. The method returns a dictionary with the object's fields.
+
+{% codeblock as python %}
+# Get a workflow and output its Common Workflow Language definition
+workflow = api.workflows().get(uuid='zzzzz-7fd4e-12345abcde67890').execute()
+print(workflow['definition'])
+{% endcodeblock %}
+
+You can pass a @select@ argument that's a list of field names to return in the included object. Doing this avoids the overhead of de/serializing and transmitting data that you won't use. Skipping a large field over a series of requests can yield a noticeable performance improvement.
+
+{% codeblock as python %}
+# Get a workflow and output its name and description.
+# Don't load the workflow definition, which might be large and we're not going to use.
+workflow = api.workflows().get(
+ uuid='zzzzz-7fd4e-12345abcde67890',
+ select=['name', 'description'],
+).execute()
+print(f"## {workflow['name']} ##\n\n{workflow['description']}")
+
+# ERROR: This raises a KeyError because we didn't load this field in
+# the `select` argument.
+workflow['created_at']
+{% endcodeblock %}
+
+h2. list method
+
+To fetch multiple Arvados objects of the same type, call the @list@ method for that resource type. The list method takes a number of arguments. Refer to the "list method API reference":{{site.baseurl}}/api/methods.html#index for details about them. The method returns a dictionary also documented at the bottom of that section. The most interesting field is @'items'@, which is a list of dictionaries where each one corresponds to an Arvados object that matched your search. To work with a single page of results:
+
+{% codeblock as python %}
+# Output the exit codes of the 10 most recently run containers.
+container_list = arv_client.containers().list(
+ limit=10,
+ order=['finished_at desc'],
+).execute()
+for container in container_list['items']:
+ print(f"{container['uuid']}: {container['exit_code']}")
+{% endcodeblock %}
+
+If you need to retrieve all of the results for a query, you may need to call the @list@ method multiple times: the default @limit@ is 100 items, and the API server will never return more than 1000. The SDK function @arvados.util.keyset_list_all@ can help orchestrate this for you. Call it with the @list@ method you want to query (don't call it yourself!) and the same keyword arguments you would pass to that method. You can control ordering by passing an @order_key@ string that names the field to use, and an @ascending@ bool that indicates whether that key should be sorted in ascending or descending order. The function returns an iterator of dictionaries, where each dictionary corresponds to an Arvados object that matched your query.
+
+{% codeblock as python %}
+# Output all the portable data hashes in a project.
+project_data = set()
+for collection in arvados.util.keyset_list_all(
+ # Note we pass the `list` method without calling it
+ arv_client.collections().list,
+ # The UUID of the project we're searching
+ filters=[['owner_uuid', '=', 'zzzzz-j7d0g-12345abcde67890']],
+):
+ project_data.add(collection['portable_data_hash'])
+print('\n'.join(project_data))
+{% endcodeblock %}
+
+When you list many objects, the following can help improve performance:
+
+* Call the list method with @count='none'@ to avoid the overhead of counting all results with each request.
+* Call the list method with a @select@ argument to only request the data you need. This cuts out some overhead from de/serializing and transferring data you won't use.
+
+h2. create method
+
+To create a new Arvados object, call the @create@ method for that resource type. You must pass a @body@ dictionary with a single item. Its key is the resource type you're creating as a string, and its value is dictionary of data fields for that resource. The method returns a dictionary with the new object's fields.
+
+If the resource type has a @name@ field, you may pass an @ensure_unique_name@ boolean argument. If true, the method will automatically update the name of the new object to make it unique if necessary.
+
+{% codeblock as python %}
+# Create a new project and output its UUID.
+project = arv_client.groups().create(
+ body={
+ 'group': {
+ 'name': 'Python SDK Test Project',
+ 'group_class': 'project',
+ },
+ },
+ ensure_unique_name=True,
+).execute()
+print(project['uuid'])
+{% endcodeblock %}
+
+h2. update method
+
+To modify an existing Arvados object, call the @update@ method for that resource type. You must pass a @uuid@ string argument that identifies the object to update, and a @body@ dictionary with a single item. Its key is the resource type you're creating as a string, and its value is dictionary of data fields to update on the resource. The method returns a dictionary with the updated object's fields.
+
+If the resource type has a @name@ field, you may pass an @ensure_unique_name@ boolean argument. If true, the method will automatically update the name of the new object to make it unique if necessary.
+
+{% codeblock as python %}
+# Update the name of a container request,
+# finalize it to submit it to Crunch for processing,
+# and output its priority.
+submitted_container_request = arv_client.container_requests().update(
+ uuid='zzzzz-xvhdp-12345abcde67890',
+ body={
+ 'container_request': {
+ 'name': 'Container Request Committed by Python SDK',
+ 'state': 'Committed',
+ },
+ },
+ ensure_unique_name=True,
+).execute()
+print(submitted_container_request['priority'])
+{% endcodeblock %}
+
+h2. delete method
+
+To delete an existing Arvados object, call the @delete@ method for that resource type. You must pass a @uuid@ string argument that identifies the object to delete. The method returns a dictionary with the deleted object's fields.
+
+{% codeblock as python %}
+# Delete a collection and output its name
+deleted_collection = arv_client.collections().delete(
+ uuid='zzzzz-4zz18-12345abcde67890',
+).execute()
+print(deleted_collection['name'])
+{% endcodeblock %}
+
+For resource types that support being trashed, you can untrash the object by calling the resource type's @untrash@ method with a @uuid@ argument identifying the object to restore.
+++ /dev/null
----
-layout: default
-navsection: sdk
-navmenu: Python
-title: Examples
-...
-{% comment %}
-Copyright (C) The Arvados Authors. All rights reserved.
-
-SPDX-License-Identifier: CC-BY-SA-3.0
-{% endcomment %}
-
-In these examples, the site prefix is @aaaaa@.
-
-See also the "cookbook":cookbook.html for more complex examples.
-
-h2. Initialize SDK
-
-{% codeblock as python %}
-import arvados
-api = arvados.api("v1")
-{% endcodeblock %}
-
-h2. create
-
-{% codeblock as python %}
-result = api.collections().create(body={"collection": {"name": "create example"}}).execute()
-{% endcodeblock %}
-
-h2. delete
-
-{% codeblock as python %}
-result = api.collections().delete(uuid="aaaaa-4zz18-ccccccccccccccc").execute()
-{% endcodeblock %}
-
-h2. get
-
-{% codeblock as python %}
-result = api.collections().get(uuid="aaaaa-4zz18-ccccccccccccccc").execute()
-{% endcodeblock %}
-
-h2. list
-
-{% codeblock as python %}
-result = api.collections().list(filters=[["uuid", "=", "aaaaa-bbbbb-ccccccccccccccc"]]).execute()
-{% endcodeblock %}
-
-h2. update
-
-{% codeblock as python %}
-result = api.collections().update(uuid="aaaaa-4zz18-ccccccccccccccc", body={"collection": {"name": "update example"}}).execute()
-{% endcodeblock %}
-
-h2. Get current user
-
-{% codeblock as python %}
-result = api.users().current().execute()
-{% endcodeblock %}
-
-h2. Get the User object for the current user
-
-{% codeblock as python %}
-current_user = arvados.api('v1').users().current().execute()
-{% endcodeblock %}
-
-h2. Get the UUID of an object that was retrieved using the SDK
-
-{% codeblock as python %}
-my_uuid = current_user['uuid']
-{% endcodeblock %}
h2. Usage
-Check out the "examples":example.html and "cookbook":cookbook.html
+Check out the "API client overview":api-client.html and "cookbook":cookbook.html.
h3. Notes
Permission level can be one of the following: @can_read@, @can_write@ or @can_manage@, giving the group member read, read/write or managing privileges on the group. For backwards compatibility purposes, if any record omits the third (permission) field, it will default to @can_write@ permission. You can read more about permissions on the "group management admin guide":{{ site.baseurl }}/admin/group-management.html.
+When using @arvados-sync-groups@, consider setting @Users.CanCreateRoleGroups: false@ in your "cluster configuration":{{site.baseurl}}/admin/config.html to prevent users from creating additional groups.
+
h2. Options
The following command line options are supported:
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: AGPL-3.0
+
+package boot
+
+import (
+ "context"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+
+ "git.arvados.org/arvados.git/lib/controller/dblock"
+ "git.arvados.org/arvados.git/lib/ctrlctx"
+ "github.com/sirupsen/logrus"
+)
+
+type railsDatabase struct{}
+
+func (runner railsDatabase) String() string {
+ return "railsDatabase"
+}
+
+// Run checks for and applies any pending Rails database migrations.
+//
+// If running a dev/test environment, and the database is empty, it
+// initializes the database.
+func (runner railsDatabase) Run(ctx context.Context, fail func(error), super *Supervisor) error {
+ err := super.wait(ctx, runPostgreSQL{}, installPassenger{src: "services/api"})
+ if err != nil {
+ return err
+ }
+
+ // determine path to installed rails app or source tree
+ var appdir string
+ if super.ClusterType == "production" {
+ appdir = "/var/lib/arvados/railsapi"
+ } else {
+ appdir = filepath.Join(super.SourcePath, "services/api")
+ }
+
+ // Check for pending migrations before running rake.
+ //
+ // In principle, we could use "rake db:migrate:status" or skip
+ // this check entirely and let "rake db:migrate" be a no-op
+ // most of the time. However, in the most common case when
+ // there are no new migrations, that would add ~2s to startup
+ // time / downtime during service restart.
+
+ todo, err := migrationList(appdir, super.logger)
+ if err != nil {
+ return err
+ }
+
+ // read schema_migrations table (list of migrations already
+ // applied) and remove those entries from todo
+ dbconnector := ctrlctx.DBConnector{PostgreSQL: super.cluster.PostgreSQL}
+ defer dbconnector.Close()
+ db, err := dbconnector.GetDB(ctx)
+ if err != nil {
+ return err
+ }
+ rows, err := db.QueryContext(ctx, `SELECT version FROM schema_migrations`)
+ if err != nil {
+ if super.ClusterType == "production" {
+ return err
+ }
+ super.logger.WithError(err).Info("schema_migrations query failed, trying db:setup")
+ return super.RunProgram(ctx, "services/api", runOptions{env: railsEnv}, "bundle", "exec", "rake", "db:setup")
+ }
+ for rows.Next() {
+ var v string
+ err = rows.Scan(&v)
+ if err != nil {
+ return err
+ }
+ delete(todo, v)
+ }
+ err = rows.Close()
+ if err != nil {
+ return err
+ }
+
+ // if nothing remains in todo, all available migrations are
+ // done, so return without running any [relatively slow]
+ // ruby/rake commands
+ if len(todo) == 0 {
+ return nil
+ }
+
+ super.logger.Infof("%d migrations pending", len(todo))
+ if !dblock.RailsMigrations.Lock(ctx, dbconnector.GetDB) {
+ return context.Canceled
+ }
+ defer dblock.RailsMigrations.Unlock()
+ return super.RunProgram(ctx, appdir, runOptions{env: railsEnv}, "bundle", "exec", "rake", "db:migrate")
+}
+
+func migrationList(dir string, log logrus.FieldLogger) (map[string]bool, error) {
+ todo := map[string]bool{}
+
+ // list versions in db/migrate/{version}_{name}.rb
+ err := fs.WalkDir(os.DirFS(dir), "db/migrate", func(path string, d fs.DirEntry, err error) error {
+ if d.IsDir() {
+ return nil
+ }
+ fnm := d.Name()
+ if !strings.HasSuffix(fnm, ".rb") {
+ log.Warnf("unexpected file in db/migrate dir: %s", fnm)
+ return nil
+ }
+ for i, c := range fnm {
+ if i > 0 && c == '_' {
+ todo[fnm[:i]] = true
+ break
+ }
+ if c < '0' || c > '9' {
+ // non-numeric character before the
+ // first '_' means this is not a
+ // migration
+ log.Warnf("unexpected file in db/migrate dir: %s", fnm)
+ return nil
+ }
+ }
+ return nil
+ })
+ if err != nil {
+ return nil, err
+ }
+ return todo, nil
+}
--- /dev/null
+// Copyright (C) The Arvados Authors. All rights reserved.
+//
+// SPDX-License-Identifier: AGPL-3.0
+
+package boot
+
+import (
+ "bytes"
+
+ "git.arvados.org/arvados.git/lib/config"
+ "git.arvados.org/arvados.git/sdk/go/arvadostest"
+ "git.arvados.org/arvados.git/sdk/go/ctxlog"
+ "gopkg.in/check.v1"
+)
+
+type railsDBSuite struct{}
+
+var _ = check.Suite(&railsDBSuite{})
+
+// Check services/api/db/migrate/*.rb match schema_migrations
+func (s *railsDBSuite) TestMigrationList(c *check.C) {
+ var logbuf bytes.Buffer
+ log := ctxlog.New(&logbuf, "text", "info")
+ todo, err := migrationList("../../services/api", log)
+ c.Check(err, check.IsNil)
+ c.Check(todo["20220804133317"], check.Equals, true)
+ c.Check(logbuf.String(), check.Equals, "")
+
+ cfg, err := config.NewLoader(nil, ctxlog.TestLogger(c)).Load()
+ c.Assert(err, check.IsNil)
+ cluster, err := cfg.GetCluster("")
+ c.Assert(err, check.IsNil)
+ db := arvadostest.DB(c, cluster)
+ rows, err := db.Query(`SELECT version FROM schema_migrations`)
+ for rows.Next() {
+ var v string
+ err = rows.Scan(&v)
+ c.Assert(err, check.IsNil)
+ if !todo[v] {
+ c.Errorf("version is in schema_migrations but not services/api/db/migrate/: %q", v)
+ }
+ delete(todo, v)
+ }
+ err = rows.Close()
+ c.Assert(err, check.IsNil)
+
+ // In the test suite, the database should be fully migrated.
+ // So, if there's anything left in todo here, there is
+ // something wrong with our "db/migrate/*.rb ==
+ // schema_migrations" reasoning.
+ c.Check(todo, check.HasLen, 0)
+}
+++ /dev/null
-// Copyright (C) The Arvados Authors. All rights reserved.
-//
-// SPDX-License-Identifier: AGPL-3.0
-
-package boot
-
-import (
- "context"
-)
-
-// Populate a blank database with arvados tables and seed rows.
-type seedDatabase struct{}
-
-func (seedDatabase) String() string {
- return "seedDatabase"
-}
-
-func (seedDatabase) Run(ctx context.Context, fail func(error), super *Supervisor) error {
- err := super.wait(ctx, runPostgreSQL{}, installPassenger{src: "services/api"})
- if err != nil {
- return err
- }
- if super.ClusterType == "production" {
- return nil
- }
- err = super.RunProgram(ctx, "services/api", runOptions{env: railsEnv}, "bundle", "exec", "rake", "db:setup")
- if err != nil {
- return err
- }
- return nil
-}
createCertificates{},
runPostgreSQL{},
runNginx{},
- runServiceCommand{name: "controller", svc: super.cluster.Services.Controller, depends: []supervisedTask{seedDatabase{}}},
+ railsDatabase{},
+ runServiceCommand{name: "controller", svc: super.cluster.Services.Controller, depends: []supervisedTask{railsDatabase{}}},
runServiceCommand{name: "git-httpd", svc: super.cluster.Services.GitHTTP},
runServiceCommand{name: "health", svc: super.cluster.Services.Health},
runServiceCommand{name: "keepproxy", svc: super.cluster.Services.Keepproxy, depends: []supervisedTask{runPassenger{src: "services/api"}}},
runServiceCommand{name: "keepstore", svc: super.cluster.Services.Keepstore},
runServiceCommand{name: "keep-web", svc: super.cluster.Services.WebDAV},
- runServiceCommand{name: "ws", svc: super.cluster.Services.Websocket, depends: []supervisedTask{seedDatabase{}}},
+ runServiceCommand{name: "ws", svc: super.cluster.Services.Websocket, depends: []supervisedTask{railsDatabase{}}},
installPassenger{src: "services/api", varlibdir: "railsapi"},
- runPassenger{src: "services/api", varlibdir: "railsapi", svc: super.cluster.Services.RailsAPI, depends: []supervisedTask{createCertificates{}, seedDatabase{}, installPassenger{src: "services/api", varlibdir: "railsapi"}}},
- seedDatabase{},
+ runPassenger{src: "services/api", varlibdir: "railsapi", svc: super.cluster.Services.RailsAPI, depends: []supervisedTask{
+ createCertificates{},
+ installPassenger{src: "services/api", varlibdir: "railsapi"},
+ railsDatabase{},
+ }},
}
if !super.NoWorkbench1 {
tasks = append(tasks,
- installPassenger{src: "apps/workbench", varlibdir: "workbench1", depends: []supervisedTask{seedDatabase{}}}, // dependency ensures workbench doesn't delay api install/startup
+ installPassenger{src: "apps/workbench", varlibdir: "workbench1", depends: []supervisedTask{railsDatabase{}}}, // dependency ensures workbench doesn't delay api install/startup
runPassenger{src: "apps/workbench", varlibdir: "workbench1", svc: super.cluster.Services.Workbench1, depends: []supervisedTask{installPassenger{src: "apps/workbench", varlibdir: "workbench1"}}},
)
}
# cluster.
RoleGroupsVisibleToAll: true
+ # If CanCreateRoleGroups is true, regular (non-admin) users can
+ # create new role groups.
+ #
+ # If false, only admins can create new role groups.
+ CanCreateRoleGroups: true
+
# During each period, a log entry with event_type="activity"
# will be recorded for each user who is active during that
# period. The object_uuid attribute will indicate the user's
"Users.AutoSetupNewUsersWithRepository": false,
"Users.AutoSetupNewUsersWithVmUUID": false,
"Users.AutoSetupUsernameBlacklist": false,
+ "Users.CanCreateRoleGroups": true,
"Users.EmailSubjectPrefix": false,
"Users.NewInactiveUserNotificationRecipients": false,
"Users.NewUserNotificationRecipients": false,
KeepBalanceService = &DBLocker{key: 10003} // keep-balance service in periodic-sweep loop
KeepBalanceActive = &DBLocker{key: 10004} // keep-balance sweep in progress (either -once=true or service loop)
Dispatch = &DBLocker{key: 10005} // any dispatcher running
+ RailsMigrations = &DBLocker{key: 10006}
retryDelay = 5 * time.Second
)
"environment": map[string]string{},
"mounts": map[string]arvados.Mount{"/out": {Kind: "tmp", Capacity: 10000}},
"output_path": "/out",
- "runtime_constraints": arvados.RuntimeConstraints{RAM: 100000000, VCPUs: 1},
+ "runtime_constraints": arvados.RuntimeConstraints{RAM: 100000000, VCPUs: 1, KeepCacheRAM: 1 << 26},
"priority": 1,
"state": arvados.ContainerRequestStateCommitted,
}, 0)
"/out": {Kind: "tmp", Capacity: 10000},
},
"output_path": "/out",
- "runtime_constraints": arvados.RuntimeConstraints{RAM: 100000000, VCPUs: 1},
+ "runtime_constraints": arvados.RuntimeConstraints{RAM: 100000000, VCPUs: 1, KeepCacheRAM: 1 << 26},
"priority": 1,
"state": arvados.ContainerRequestStateCommitted,
"container_count_max": 1,
}
if err := db.Ping(); err != nil {
ctxlog.FromContext(ctx).WithError(err).Error("postgresql connect succeeded but ping failed")
+ db.Close()
return nil, errDBConnection
}
dbc.pgdb = db
return db, nil
}
+
+func (dbc *DBConnector) Close() error {
+ dbc.mtx.Lock()
+ defer dbc.mtx.Unlock()
+ var err error
+ if dbc.pgdb != nil {
+ err = dbc.pgdb.Close()
+ dbc.pgdb = nil
+ }
+ return err
+}
if api_client is None:
api_client = arvados.safeapi.ThreadSafeApiCache(
api_params={"model": OrderedJsonModel(), "timeout": arvargs.http_timeout},
- keep_params={"num_retries": 4})
+ keep_params={"num_retries": 4},
+ version='v1',
+ )
keep_client = api_client.keep
# Make an API object now so errors are reported early.
api_client.users().current().execute()
def check_contents(group, wf_uuid):
contents = api.groups().contents(uuid=group["uuid"]).execute()
- if len(contents["items"]) != 3:
- raise Exception("Expected 3 items in "+group["uuid"]+" was "+len(contents["items"]))
+ if len(contents["items"]) != 4:
+ raise Exception("Expected 4 items in "+group["uuid"]+" was "+str(len(contents["items"])))
found = False
for c in contents["items"]:
if not found:
raise Exception("Couldn't find jobs image dependency")
+ found = False
+ for c in contents["items"]:
+ if c["kind"] == "arvados#collection" and c["portable_data_hash"] == "13d3901489516f9986c9685867043d39+61":
+ found = True
+ if not found:
+ raise Exception("Couldn't find collection containing workflow")
+
def test_create():
group = api.groups().create(body={"group": {"name": "test-19070-project-1", "group_class": "project"}}, ensure_unique_name=True).execute()
wf_uuid = wf_uuid.decode("utf-8").strip()
contents = api.groups().contents(uuid=group["uuid"]).execute()
- if len(contents["items"]) != 1:
- raise Exception("Expected 1 items")
+ if len(contents["items"]) != 2:
+ raise Exception("Expected 2 items")
found = False
for c in contents["items"]:
if not found:
raise Exception("Couldn't find workflow")
+ found = False
+ for c in contents["items"]:
+ if c["kind"] == "arvados#collection" and c["portable_data_hash"] == "13d3901489516f9986c9685867043d39+61":
+ found = True
+ if not found:
+ raise Exception("Couldn't find collection containing workflow")
+
# Updating by default will copy missing items
cmd = ["arvados-cwl-runner", "--update-workflow", wf_uuid, "19070-copy-deps.cwl"]
print(" ".join(cmd))
if err != nil {
panic(err)
}
- return clusterID + "-" + infix + "-" + n.Text(36)
+ nstr := n.Text(36)
+ for len(nstr) < 15 {
+ nstr = "0" + nstr
+ }
+ return clusterID + "-" + infix + "-" + nstr
}
PreferDomainForUsername string
UserSetupMailText string
RoleGroupsVisibleToAll bool
+ CanCreateRoleGroups bool
ActivityLoggingPeriod Duration
}
StorageClasses map[string]StorageClassConfig
# Copyright (C) The Arvados Authors. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
+"""Arvados API client
+
+The code in this module builds Arvados API client objects you can use to submit
+Arvados API requests. This includes extending the underlying HTTP client with
+niceties such as caching, X-Request-Id header for tracking, and more. The main
+client constructors are `api` and `api_from_config`.
+"""
from __future__ import absolute_import
from future import standard_library
"""Model class for JSON that preserves the contents' order.
API clients that care about preserving the order of fields in API
- server responses can use this model to do so, like this::
+ server responses can use this model to do so, like this:
from arvados.api import OrderedJsonModel
client = arvados.api('v1', ..., model=OrderedJsonModel())
return None
return cache.SafeHTTPCache(path, max_age=60*60*24*2)
-def api(version=None, cache=True, host=None, token=None, insecure=False,
- request_id=None, timeout=5*60, **kwargs):
- """Return an apiclient Resources object for an Arvados instance.
-
- :version:
- A string naming the version of the Arvados API to use (for
- example, 'v1').
-
- :cache:
- Use a cache (~/.cache/arvados/discovery) for the discovery
- document.
-
- :host:
- The Arvados API server host (and optional :port) to connect to.
-
- :token:
- The authentication token to send with each API call.
-
- :insecure:
- If True, ignore SSL certificate validation errors.
-
- :timeout:
- A timeout value for http requests.
-
- :request_id:
- Default X-Request-Id header value for outgoing requests that
- don't already provide one. If None or omitted, generate a random
+def api_client(
+ version,
+ discoveryServiceUrl,
+ token,
+ *,
+ cache=True,
+ http=None,
+ insecure=False,
+ request_id=None,
+ timeout=5*60,
+ **kwargs,
+):
+ """Build an Arvados API client
+
+ This function returns a `googleapiclient.discovery.Resource` object
+ constructed from the given arguments. This is a relatively low-level
+ interface that requires all the necessary inputs as arguments. Most
+ users will prefer to use `api` which can accept more flexible inputs.
+
+ Arguments:
+
+ version: str
+ : A string naming the version of the Arvados API to use.
+
+ discoveryServiceUrl: str
+ : The URL used to discover APIs passed directly to
+ `googleapiclient.discovery.build`.
+
+ token: str
+ : The authentication token to send with each API call.
+
+ Keyword-only arguments:
+
+ cache: bool
+ : If true, loads the API discovery document from, or saves it to, a cache
+ on disk (located at `~/.cache/arvados/discovery`).
+
+ http: httplib2.Http | None
+ : The HTTP client object the API client object will use to make requests.
+ If not provided, this function will build its own to use. Either way, the
+ object will be patched as part of the build process.
+
+ insecure: bool
+ : If true, ignore SSL certificate validation errors. Default `False`.
+
+ request_id: str | None
+ : Default `X-Request-Id` header value for outgoing requests that
+ don't already provide one. If `None` or omitted, generate a random
ID. When retrying failed requests, the same ID is used on all
attempts.
- Additional keyword arguments will be passed directly to
- `apiclient_discovery.build` if a new Resource object is created.
- If the `discoveryServiceUrl` or `http` keyword arguments are
- missing, this function will set default values for them, based on
- the current Arvados configuration settings.
+ timeout: int
+ : A timeout value for HTTP requests in seconds. Default 300 (5 minutes).
+ Additional keyword arguments will be passed directly to
+ `googleapiclient.discovery.build`.
"""
-
- if not version:
- version = 'v1'
- _logger.info("Using default API version. " +
- "Call arvados.api('%s') instead." %
- version)
- if 'discoveryServiceUrl' in kwargs:
- if host:
- raise ValueError("both discoveryServiceUrl and host provided")
- # Here we can't use a token from environment, config file,
- # etc. Those probably have nothing to do with the host
- # provided by the caller.
- if not token:
- raise ValueError("discoveryServiceUrl provided, but token missing")
- elif host and token:
- pass
- elif not host and not token:
- return api_from_config(
- version=version, cache=cache, timeout=timeout,
- request_id=request_id, **kwargs)
- else:
- # Caller provided one but not the other
- if not host:
- raise ValueError("token argument provided, but host missing.")
- else:
- raise ValueError("host argument provided, but token missing.")
-
- if host:
- # Caller wants us to build the discoveryServiceUrl
- kwargs['discoveryServiceUrl'] = (
- 'https://%s/discovery/v1/apis/{api}/{apiVersion}/rest' % (host,))
-
- if 'http' not in kwargs:
- http_kwargs = {'ca_certs': util.ca_certs_path()}
- if cache:
- http_kwargs['cache'] = http_cache('discovery')
- if insecure:
- http_kwargs['disable_ssl_certificate_validation'] = True
- kwargs['http'] = httplib2.Http(**http_kwargs)
-
- if kwargs['http'].timeout is None:
- kwargs['http'].timeout = timeout
-
- kwargs['http'] = _patch_http_request(kwargs['http'], token)
-
- svc = apiclient_discovery.build('arvados', version, cache_discovery=False, **kwargs)
+ if http is None:
+ http = httplib2.Http(
+ ca_certs=util.ca_certs_path(),
+ cache=http_cache('discovery') if cache else None,
+ disable_ssl_certificate_validation=bool(insecure),
+ )
+ if http.timeout is None:
+ http.timeout = timeout
+ http = _patch_http_request(http, token)
+
+ svc = apiclient_discovery.build(
+ 'arvados', version,
+ cache_discovery=False,
+ discoveryServiceUrl=discoveryServiceUrl,
+ http=http,
+ **kwargs,
+ )
svc.api_token = token
svc.insecure = insecure
svc.request_id = request_id
svc.config = lambda: util.get_config_once(svc)
svc.vocabulary = lambda: util.get_vocabulary_once(svc)
svc.close_connections = types.MethodType(_close_connections, svc)
- kwargs['http'].max_request_size = svc._rootDesc.get('maxRequestSize', 0)
- kwargs['http'].cache = None
- kwargs['http']._request_id = lambda: svc.request_id or util.new_request_id()
+ http.max_request_size = svc._rootDesc.get('maxRequestSize', 0)
+ http.cache = None
+ http._request_id = lambda: svc.request_id or util.new_request_id()
return svc
-def api_from_config(version=None, apiconfig=None, **kwargs):
- """Return an apiclient Resources object enabling access to an Arvados server
- instance.
+def normalize_api_kwargs(
+ version=None,
+ discoveryServiceUrl=None,
+ host=None,
+ token=None,
+ **kwargs,
+):
+ """Validate kwargs from `api` and build kwargs for `api_client`
+
+ This method takes high-level keyword arguments passed to the `api`
+ constructor and normalizes them into a new dictionary that can be passed
+ as keyword arguments to `api_client`. It raises `ValueError` if required
+ arguments are missing or conflict.
- :version:
- A string naming the version of the Arvados REST API to use (for
- example, 'v1').
+ Arguments:
- :apiconfig:
- If provided, this should be a dict-like object (must support the get()
- method) with entries for ARVADOS_API_HOST, ARVADOS_API_TOKEN, and
- optionally ARVADOS_API_HOST_INSECURE. If not provided, use
- arvados.config (which gets these parameters from the environment by
- default.)
+ version: str | None
+ : A string naming the version of the Arvados API to use. If not specified,
+ the code will log a warning and fall back to 'v1'.
- Other keyword arguments such as `cache` will be passed along `api()`
+ discoveryServiceUrl: str | None
+ : The URL used to discover APIs passed directly to
+ `googleapiclient.discovery.build`. It is an error to pass both
+ `discoveryServiceUrl` and `host`.
+ host: str | None
+ : The hostname and optional port number of the Arvados API server. Used to
+ build `discoveryServiceUrl`. It is an error to pass both
+ `discoveryServiceUrl` and `host`.
+
+ token: str
+ : The authentication token to send with each API call.
+
+ Additional keyword arguments will be included in the return value.
+ """
+ if discoveryServiceUrl and host:
+ raise ValueError("both discoveryServiceUrl and host provided")
+ elif discoveryServiceUrl:
+ url_src = "discoveryServiceUrl"
+ elif host:
+ url_src = "host argument"
+ discoveryServiceUrl = 'https://%s/discovery/v1/apis/{api}/{apiVersion}/rest' % (host,)
+ elif token:
+ # This specific error message gets priority for backwards compatibility.
+ raise ValueError("token argument provided, but host missing.")
+ else:
+ raise ValueError("neither discoveryServiceUrl nor host provided")
+ if not token:
+ raise ValueError("%s provided, but token missing" % (url_src,))
+ if not version:
+ version = 'v1'
+ _logger.info(
+ "Using default API version. Call arvados.api(%r) instead.",
+ version,
+ )
+ return {
+ 'discoveryServiceUrl': discoveryServiceUrl,
+ 'token': token,
+ 'version': version,
+ **kwargs,
+ }
+
+def api_kwargs_from_config(version=None, apiconfig=None, **kwargs):
+ """Build `api_client` keyword arguments from configuration
+
+ This function accepts a mapping with Arvados configuration settings like
+ `ARVADOS_API_HOST` and converts them into a mapping of keyword arguments
+ that can be passed to `api_client`. If `ARVADOS_API_HOST` or
+ `ARVADOS_API_TOKEN` are not configured, it raises `ValueError`.
+
+ Arguments:
+
+ version: str | None
+ : A string naming the version of the Arvados API to use. If not specified,
+ the code will log a warning and fall back to 'v1'.
+
+ apiconfig: Mapping[str, str] | None
+ : A mapping with entries for `ARVADOS_API_HOST`, `ARVADOS_API_TOKEN`, and
+ optionally `ARVADOS_API_HOST_INSECURE`. If not provided, calls
+ `arvados.config.settings` to get these parameters from user configuration.
+
+ Additional keyword arguments will be included in the return value.
"""
- # Load from user configuration or environment
if apiconfig is None:
apiconfig = config.settings()
+ missing = " and ".join(
+ key
+ for key in ['ARVADOS_API_HOST', 'ARVADOS_API_TOKEN']
+ if key not in apiconfig
+ )
+ if missing:
+ raise ValueError(
+ "%s not set.\nPlease set in %s or export environment variable." %
+ (missing, config.default_config_file),
+ )
+ return normalize_api_kwargs(
+ version,
+ None,
+ apiconfig['ARVADOS_API_HOST'],
+ apiconfig['ARVADOS_API_TOKEN'],
+ insecure=config.flag_is_true('ARVADOS_API_HOST_INSECURE', apiconfig),
+ **kwargs,
+ )
+
+def api(version=None, cache=True, host=None, token=None, insecure=False,
+ request_id=None, timeout=5*60, *,
+ discoveryServiceUrl=None, **kwargs):
+ """Dynamically build an Arvados API client
+
+ This function provides a high-level "do what I mean" interface to build an
+ Arvados API client object. You can call it with no arguments to build a
+ client from user configuration; pass `host` and `token` arguments just
+ like you would write in user configuration; or pass additional arguments
+ for lower-level control over the client.
+
+ This function returns a `arvados.safeapi.ThreadSafeApiCache`, an
+ API-compatible wrapper around `googleapiclient.discovery.Resource`. If
+ you're handling concurrency yourself and/or your application is very
+ performance-sensitive, consider calling `api_client` directly.
+
+ Arguments:
+
+ version: str | None
+ : A string naming the version of the Arvados API to use. If not specified,
+ the code will log a warning and fall back to 'v1'.
+
+ host: str | None
+ : The hostname and optional port number of the Arvados API server.
+
+ token: str | None
+ : The authentication token to send with each API call.
+
+ discoveryServiceUrl: str | None
+ : The URL used to discover APIs passed directly to
+ `googleapiclient.discovery.build`.
+
+ If `host`, `token`, and `discoveryServiceUrl` are all omitted, `host` and
+ `token` will be loaded from the user's configuration. Otherwise, you must
+ pass `token` and one of `host` or `discoveryServiceUrl`. It is an error to
+ pass both `host` and `discoveryServiceUrl`.
- errors = []
- for x in ['ARVADOS_API_HOST', 'ARVADOS_API_TOKEN']:
- if x not in apiconfig:
- errors.append(x)
- if errors:
- raise ValueError(" and ".join(errors)+" not set.\nPlease set in %s or export environment variable." % config.default_config_file)
- host = apiconfig.get('ARVADOS_API_HOST')
- token = apiconfig.get('ARVADOS_API_TOKEN')
- insecure = config.flag_is_true('ARVADOS_API_HOST_INSECURE', apiconfig)
-
- return api(version=version, host=host, token=token, insecure=insecure, **kwargs)
+ Other arguments are passed directly to `api_client`. See that function's
+ docstring for more information about their meaning.
+ """
+ kwargs.update(
+ cache=cache,
+ insecure=insecure,
+ request_id=request_id,
+ timeout=timeout,
+ )
+ if discoveryServiceUrl or host or token:
+ kwargs.update(normalize_api_kwargs(version, discoveryServiceUrl, host, token))
+ else:
+ kwargs.update(api_kwargs_from_config(version))
+ version = kwargs.pop('version')
+ # We do the import here to avoid a circular import at the top level.
+ from .safeapi import ThreadSafeApiCache
+ return ThreadSafeApiCache({}, {}, kwargs, version)
+
+def api_from_config(version=None, apiconfig=None, **kwargs):
+ """Build an Arvados API client from a configuration mapping
+
+ This function builds an Arvados API client from a mapping with user
+ configuration. It accepts that mapping as an argument, so you can use a
+ configuration that's different from what the user has set up.
+
+ This function returns a `arvados.safeapi.ThreadSafeApiCache`, an
+ API-compatible wrapper around `googleapiclient.discovery.Resource`. If
+ you're handling concurrency yourself and/or your application is very
+ performance-sensitive, consider calling `api_client` directly.
+
+ Arguments:
+
+ version: str | None
+ : A string naming the version of the Arvados API to use. If not specified,
+ the code will log a warning and fall back to 'v1'.
+
+ apiconfig: Mapping[str, str] | None
+ : A mapping with entries for `ARVADOS_API_HOST`, `ARVADOS_API_TOKEN`, and
+ optionally `ARVADOS_API_HOST_INSECURE`. If not provided, calls
+ `arvados.config.settings` to get these parameters from user configuration.
+
+ Other arguments are passed directly to `api_client`. See that function's
+ docstring for more information about their meaning.
+ """
+ return api(**api_kwargs_from_config(version, apiconfig, **kwargs))
@synchronized
def _my_api(self):
if self._api_client is None:
- self._api_client = ThreadSafeApiCache(self._config)
+ self._api_client = ThreadSafeApiCache(self._config, version='v1')
if self._keep_client is None:
self._keep_client = self._api_client.keep
return self._api_client
dockerhash = hash_link_map[collection_uuid]['name']
except KeyError:
dockerhash = '<unknown>'
- name_parts = link['name'].split(':', 1)
+ name_parts = link['name'].rsplit(':', 1)
images.append(_new_image_listing(link, dockerhash, *name_parts))
# Find any image hash links that did not have a corresponding name link,
elif args.tag is None:
args.tag = 'latest'
+ if '/' in args.image:
+ hostport, path = args.image.split('/', 1)
+ if hostport.endswith(':443'):
+ # "docker pull host:443/asdf" transparently removes the
+ # :443 (which is redundant because https is implied) and
+ # after it succeeds "docker images" will list "host/asdf",
+ # not "host:443/asdf". If we strip the :443 then the name
+ # doesn't change underneath us.
+ args.image = '/'.join([hostport[:-4], path])
+
# Pull the image if requested, unless the image is specified as a hash
# that we already have.
if args.pull and not find_image_hashes(args.image):
# Copyright (C) The Arvados Authors. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
+"""Thread-safe wrapper for an Arvados API client
+
+This module provides `ThreadSafeApiCache`, a thread-safe, API-compatible
+Arvados API client.
+"""
from __future__ import absolute_import
from builtins import object
-import copy
import threading
-import arvados
-import arvados.keep as keep
-import arvados.config as config
+from . import api
+from . import config
+from . import keep
+from . import util
class ThreadSafeApiCache(object):
- """Threadsafe wrapper for API objects.
+ """Thread-safe wrapper for an Arvados API client
- This stores and returns a different api object per thread, because httplib2
- which underlies apiclient is not threadsafe.
+ This class takes all the arguments necessary to build a lower-level
+ Arvados API client `googleapiclient.discovery.Resource`, then
+ transparently builds and wraps a unique object per thread. This works
+ around the fact that the client's underlying HTTP client object is not
+ thread-safe.
- """
+ Arguments:
- def __init__(self, apiconfig=None, keep_params={}, api_params={}):
- if apiconfig is None:
- apiconfig = config.settings()
- self.apiconfig = copy.copy(apiconfig)
- self.api_params = api_params
- self.local = threading.local()
+ apiconfig: Mapping[str, str] | None
+ : A mapping with entries for `ARVADOS_API_HOST`, `ARVADOS_API_TOKEN`,
+ and optionally `ARVADOS_API_HOST_INSECURE`. If not provided, uses
+ `arvados.config.settings` to get these parameters from user
+ configuration. You can pass an empty mapping to build the client
+ solely from `api_params`.
+
+ keep_params: Mapping[str, Any]
+ : Keyword arguments used to construct an associated
+ `arvados.keep.KeepClient`.
- # Initialize an API object for this thread before creating
- # KeepClient, this will report if ARVADOS_API_HOST or
- # ARVADOS_API_TOKEN are missing.
- self.localapi()
+ api_params: Mapping[str, Any]
+ : Keyword arguments used to construct each thread's API client. These
+ have the same meaning as in the `arvados.api.api` function.
+ version: str | None
+ : A string naming the version of the Arvados API to use. If not specified,
+ the code will log a warning and fall back to 'v1'.
+ """
+
+ def __init__(self, apiconfig=None, keep_params={}, api_params={}, version=None):
+ if apiconfig or apiconfig is None:
+ self._api_kwargs = api.api_kwargs_from_config(version, apiconfig, **api_params)
+ else:
+ self._api_kwargs = api.normalize_api_kwargs(version, **api_params)
+ self.api_token = self._api_kwargs['token']
+ self.request_id = self._api_kwargs.get('request_id')
+ self.local = threading.local()
self.keep = keep.KeepClient(api_client=self, **keep_params)
def localapi(self):
- if 'api' not in self.local.__dict__:
- self.local.api = arvados.api_from_config('v1', apiconfig=self.apiconfig,
- **self.api_params)
- return self.local.api
+ try:
+ client = self.local.api
+ except AttributeError:
+ client = api.api_client(**self._api_kwargs)
+ client._http._request_id = lambda: self.request_id or util.new_request_id()
+ self.local.api = client
+ return client
def __getattr__(self, name):
# Proxy nonexistent attributes to the thread-local API client.
- if name == "api_token":
- return self.apiconfig['ARVADOS_API_TOKEN']
return getattr(self.localapi(), name)
import socket
import string
import unittest
+import urllib.parse as urlparse
import mock
from . import run_test_server
from apiclient import errors as apiclient_errors
from apiclient import http as apiclient_http
-from arvados.api import OrderedJsonModel, RETRY_DELAY_INITIAL, RETRY_DELAY_BACKOFF, RETRY_COUNT
+from arvados.api import (
+ api_client,
+ normalize_api_kwargs,
+ api_kwargs_from_config,
+ OrderedJsonModel,
+ RETRY_DELAY_INITIAL,
+ RETRY_DELAY_BACKOFF,
+ RETRY_COUNT,
+)
from .arvados_testutil import fake_httplib2_response, queue_with
if not mimetypes.inited:
json.dumps({'errors': errors,
'error_token': '1234567890+12345678'}).encode())
+ def _config_from_environ(self):
+ return {
+ key: value
+ for key, value in os.environ.items()
+ if key.startswith('ARVADOS_API_')
+ }
+
+ def _discoveryServiceUrl(
+ self,
+ host=None,
+ path='/discovery/v1/apis/{api}/{apiVersion}/rest',
+ scheme='https',
+ ):
+ if host is None:
+ host = os.environ['ARVADOS_API_HOST']
+ return urlparse.urlunsplit((scheme, host, path, None, None))
+
def test_new_api_objects_with_cache(self):
clients = [arvados.api('v1', cache=True) for index in [0, 1]]
self.assertIsNot(*clients)
result = api.humans().get(uuid='test').execute()
self.assertEqual(string.hexdigits, ''.join(list(result.keys())))
+ def test_api_is_threadsafe(self):
+ api_kwargs = {
+ 'host': os.environ['ARVADOS_API_HOST'],
+ 'token': os.environ['ARVADOS_API_TOKEN'],
+ 'insecure': True,
+ }
+ config_kwargs = {'apiconfig': os.environ}
+ for api_constructor, kwargs in [
+ (arvados.api, {}),
+ (arvados.api, api_kwargs),
+ (arvados.api_from_config, {}),
+ (arvados.api_from_config, config_kwargs),
+ ]:
+ sub_kwargs = "kwargs" if kwargs else "no kwargs"
+ with self.subTest(f"{api_constructor.__name__} with {sub_kwargs}"):
+ api_client = api_constructor('v1', **kwargs)
+ self.assertTrue(hasattr(api_client, 'localapi'),
+ f"client missing localapi method")
+ self.assertTrue(hasattr(api_client, 'keep'),
+ f"client missing keep attribute")
+
+ def test_api_host_constructor(self):
+ cache = True
+ insecure = True
+ client = arvados.api(
+ 'v1',
+ cache,
+ os.environ['ARVADOS_API_HOST'],
+ os.environ['ARVADOS_API_TOKEN'],
+ insecure,
+ )
+ self.assertEqual(client.api_token, os.environ['ARVADOS_API_TOKEN'],
+ "client constructed with incorrect token")
+
+ def test_api_url_constructor(self):
+ client = arvados.api(
+ 'v1',
+ discoveryServiceUrl=self._discoveryServiceUrl(),
+ token=os.environ['ARVADOS_API_TOKEN'],
+ insecure=True,
+ )
+ self.assertEqual(client.api_token, os.environ['ARVADOS_API_TOKEN'],
+ "client constructed with incorrect token")
+
+ def test_api_bad_args(self):
+ all_kwargs = {
+ 'host': os.environ['ARVADOS_API_HOST'],
+ 'token': os.environ['ARVADOS_API_TOKEN'],
+ 'discoveryServiceUrl': self._discoveryServiceUrl(),
+ }
+ for use_keys in [
+ # Passing only a single key is missing required info
+ *([key] for key in all_kwargs.keys()),
+ # Passing all keys is a conflict
+ list(all_kwargs.keys()),
+ ]:
+ kwargs = {key: all_kwargs[key] for key in use_keys}
+ kwargs_list = ', '.join(use_keys)
+ with self.subTest(f"calling arvados.api with {kwargs_list} fails"), \
+ self.assertRaises(ValueError):
+ arvados.api('v1', insecure=True, **kwargs)
+
+ def test_api_bad_url(self):
+ for bad_kwargs in [
+ {'discoveryServiceUrl': self._discoveryServiceUrl() + '/BadTestURL'},
+ {'version': 'BadTestVersion', 'host': os.environ['ARVADOS_API_HOST']},
+ ]:
+ bad_key = next(iter(bad_kwargs))
+ with self.subTest(f"api fails with bad {bad_key}"), \
+ self.assertRaises(apiclient_errors.UnknownApiNameOrVersion):
+ arvados.api(**bad_kwargs, token='test_api_bad_url', insecure=True)
+
+ def test_normalize_api_good_args(self):
+ for version, discoveryServiceUrl, host in [
+ ('Test1', None, os.environ['ARVADOS_API_HOST']),
+ (None, self._discoveryServiceUrl(), None)
+ ]:
+ argname = 'discoveryServiceUrl' if host is None else 'host'
+ with self.subTest(f"normalize_api_kwargs with {argname}"):
+ actual = normalize_api_kwargs(
+ version,
+ discoveryServiceUrl,
+ host,
+ os.environ['ARVADOS_API_TOKEN'],
+ insecure=True,
+ )
+ self.assertEqual(actual['discoveryServiceUrl'], self._discoveryServiceUrl())
+ self.assertEqual(actual['token'], os.environ['ARVADOS_API_TOKEN'])
+ self.assertEqual(actual['version'], version or 'v1')
+ self.assertTrue(actual['insecure'])
+ self.assertNotIn('host', actual)
+
+ def test_normalize_api_bad_args(self):
+ all_args = (
+ self._discoveryServiceUrl(),
+ os.environ['ARVADOS_API_HOST'],
+ os.environ['ARVADOS_API_TOKEN'],
+ )
+ for arg_index, arg_value in enumerate(all_args):
+ args = [None] * len(all_args)
+ args[arg_index] = arg_value
+ with self.subTest(f"normalize_api_kwargs with only arg #{arg_index + 1}"), \
+ self.assertRaises(ValueError):
+ normalize_api_kwargs('v1', *args)
+ with self.subTest("normalize_api_kwargs with discoveryServiceUrl and host"), \
+ self.assertRaises(ValueError):
+ normalize_api_kwargs('v1', *all_args)
+
+ def test_api_from_config_default(self):
+ client = arvados.api_from_config('v1')
+ self.assertEqual(client.api_token, os.environ['ARVADOS_API_TOKEN'],
+ "client constructed with incorrect token")
+
+ def test_api_from_config_explicit(self):
+ config = self._config_from_environ()
+ client = arvados.api_from_config('v1', config)
+ self.assertEqual(client.api_token, os.environ['ARVADOS_API_TOKEN'],
+ "client constructed with incorrect token")
+
+ def test_api_from_bad_config(self):
+ base_config = self._config_from_environ()
+ for del_key in ['ARVADOS_API_HOST', 'ARVADOS_API_TOKEN']:
+ with self.subTest(f"api_from_config without {del_key} fails"), \
+ self.assertRaises(ValueError):
+ config = dict(base_config)
+ del config[del_key]
+ arvados.api_from_config('v1', config)
+
+ def test_api_kwargs_from_good_config(self):
+ for config in [None, self._config_from_environ()]:
+ conf_type = 'default' if config is None else 'passed'
+ with self.subTest(f"api_kwargs_from_config with {conf_type} config"):
+ version = 'Test1' if config else None
+ actual = api_kwargs_from_config(version, config)
+ self.assertEqual(actual['discoveryServiceUrl'], self._discoveryServiceUrl())
+ self.assertEqual(actual['token'], os.environ['ARVADOS_API_TOKEN'])
+ self.assertEqual(actual['version'], version or 'v1')
+ self.assertTrue(actual['insecure'])
+ self.assertNotIn('host', actual)
+
+ def test_api_kwargs_from_bad_config(self):
+ base_config = self._config_from_environ()
+ for del_key in ['ARVADOS_API_HOST', 'ARVADOS_API_TOKEN']:
+ with self.subTest(f"api_kwargs_from_config without {del_key} fails"), \
+ self.assertRaises(ValueError):
+ config = dict(base_config)
+ del config[del_key]
+ api_kwargs_from_config('v1', config)
+
+ def test_api_client_constructor(self):
+ client = api_client(
+ 'v1',
+ self._discoveryServiceUrl(),
+ os.environ['ARVADOS_API_TOKEN'],
+ insecure=True,
+ )
+ self.assertEqual(client.api_token, os.environ['ARVADOS_API_TOKEN'],
+ "client constructed with incorrect token")
+ self.assertFalse(
+ hasattr(client, 'localapi'),
+ "client has localapi method when it should not be thread-safe",
+ )
+
+ def test_api_client_bad_url(self):
+ all_args = ('v1', self._discoveryServiceUrl(), 'test_api_client_bad_url')
+ for arg_index, arg_value in [
+ (0, 'BadTestVersion'),
+ (1, all_args[1] + '/BadTestURL'),
+ ]:
+ with self.subTest(f"api_client fails with {arg_index}={arg_value!r}"), \
+ self.assertRaises(apiclient_errors.UnknownApiNameOrVersion):
+ args = list(all_args)
+ args[arg_index] = arg_value
+ api_client(*args, insecure=True)
+
class RetryREST(unittest.TestCase):
def setUp(self):
class ArvKeepdockerTestCase(unittest.TestCase, tutil.VersionChecker):
- def run_arv_keepdocker(self, args, err):
+ def run_arv_keepdocker(self, args, err, **kwargs):
sys.argv = ['arv-keepdocker'] + args
log_handler = logging.StreamHandler(err)
arv_keepdocker.logger.addHandler(log_handler)
try:
- return arv_keepdocker.main()
+ return arv_keepdocker.main(**kwargs)
finally:
arv_keepdocker.logger.removeHandler(log_handler)
self.run_arv_keepdocker(['repo:tag'], sys.stderr)
find_image_mock.assert_called_with('repo', 'tag')
+ def test_image_given_as_registry_repo_colon_tag(self):
with self.assertRaises(StopTest), \
mock.patch('arvados.commands.keepdocker.find_one_image_hash',
side_effect=StopTest) as find_image_mock:
self.run_arv_keepdocker(['myreg.example:8888/repo/img:tag'], sys.stderr)
find_image_mock.assert_called_with('myreg.example:8888/repo/img', 'tag')
+ with self.assertRaises(StopTest), \
+ mock.patch('arvados.commands.keepdocker.find_one_image_hash',
+ side_effect=StopTest) as find_image_mock:
+ self.run_arv_keepdocker(['registry.hub.docker.com:443/library/debian:bullseye-slim'], sys.stderr)
+ find_image_mock.assert_called_with('registry.hub.docker.com/library/debian', 'bullseye-slim')
+
def test_image_has_colons(self):
with self.assertRaises(StopTest), \
mock.patch('arvados.commands.keepdocker.find_one_image_hash',
self.run_arv_keepdocker(['[::1]/repo/img'], sys.stderr)
find_image_mock.assert_called_with('[::1]/repo/img', 'latest')
+ with self.assertRaises(StopTest), \
+ mock.patch('arvados.commands.keepdocker.find_one_image_hash',
+ side_effect=StopTest) as find_image_mock:
+ self.run_arv_keepdocker(['[::1]:8888/repo/img:tag'], sys.stderr)
+ find_image_mock.assert_called_with('[::1]:8888/repo/img', 'tag')
+
+ def test_list_images_with_host_and_port(self):
+ api = arvados.api('v1')
+ taglink = api.links().create(body={'link': {
+ 'link_class': 'docker_image_repo+tag',
+ 'name': 'registry.example:1234/repo:latest',
+ 'head_uuid': 'zzzzz-4zz18-1v45jub259sjjgb',
+ }}).execute()
+ try:
+ out = tutil.StringIO()
+ with self.assertRaises(SystemExit):
+ self.run_arv_keepdocker([], sys.stderr, stdout=out)
+ self.assertRegex(out.getvalue(), '\nregistry.example:1234/repo +latest ')
+ finally:
+ api.links().delete(uuid=taglink['uuid']).execute()
+
@mock.patch('arvados.commands.keepdocker.list_images_in_arv',
return_value=[])
@mock.patch('arvados.commands.keepdocker.find_image_hashes',
def test_put_block_replication(self):
self.call_main_on_test_file()
+ arv_put.api_client = None
with mock.patch('arvados.collection.KeepClient.local_store_put') as put_mock:
put_mock.return_value = 'acbd18db4cc2f85cedef654fccc4a4d8+3'
self.call_main_on_test_file(['--replication', '1'])
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+import unittest
+
+import googleapiclient
+
+from arvados import safeapi
+
+from . import run_test_server
+
+class SafeApiTest(run_test_server.TestCaseWithServers):
+ MAIN_SERVER = {}
+
+ def test_constructor(self):
+ env_mapping = {
+ key: value
+ for key, value in os.environ.items()
+ if key.startswith('ARVADOS_API_')
+ }
+ extra_params = {
+ 'timeout': 299,
+ }
+ base_params = {
+ key[12:].lower(): value
+ for key, value in env_mapping.items()
+ }
+ try:
+ base_params['insecure'] = base_params.pop('host_insecure')
+ except KeyError:
+ pass
+ expected_keep_params = {}
+ for config, params, subtest in [
+ (None, {}, "default arguments"),
+ (None, extra_params, "extra params"),
+ (env_mapping, {}, "explicit config"),
+ (env_mapping, extra_params, "explicit config and params"),
+ ({}, base_params, "params only"),
+ ]:
+ with self.subTest(f"test constructor with {subtest}"):
+ expected_timeout = params.get('timeout', 300)
+ expected_params = dict(params)
+ keep_params = dict(expected_keep_params)
+ client = safeapi.ThreadSafeApiCache(config, keep_params, params, 'v1')
+ self.assertTrue(hasattr(client, 'localapi'), "client missing localapi method")
+ self.assertEqual(client.api_token, os.environ['ARVADOS_API_TOKEN'])
+ self.assertEqual(client._http.timeout, expected_timeout)
+ self.assertEqual(params, expected_params,
+ "api_params was modified in-place")
+ self.assertEqual(keep_params, expected_keep_params,
+ "keep_params was modified in-place")
+
+ def test_constructor_no_args(self):
+ client = safeapi.ThreadSafeApiCache()
+ self.assertTrue(hasattr(client, 'localapi'), "client missing localapi method")
+ self.assertEqual(client.api_token, os.environ['ARVADOS_API_TOKEN'])
+ self.assertTrue(client.insecure)
+
+ def test_constructor_bad_version(self):
+ with self.assertRaises(googleapiclient.errors.UnknownApiNameOrVersion):
+ safeapi.ThreadSafeApiCache(version='BadTestVersion')
railties (>= 4)
request_store (~> 1.0)
logstash-event (1.2.02)
- loofah (2.18.0)
+ loofah (2.19.1)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
mail (2.7.1)
multi_json (1.15.0)
multipart-post (2.1.1)
nio4r (2.5.8)
- nokogiri (1.13.7)
+ nokogiri (1.13.10)
mini_portile2 (~> 2.8.0)
racc (~> 1.4)
oj (3.9.2)
pg (1.1.4)
power_assert (1.1.4)
public_suffix (4.0.6)
- racc (1.6.0)
+ racc (1.6.1)
rack (2.2.4)
rack-test (2.0.2)
rack (>= 1.3)
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
nokogiri (>= 1.6)
- rails-html-sanitizer (1.4.3)
- loofah (~> 2.3)
+ rails-html-sanitizer (1.4.4)
+ loofah (~> 2.19, >= 2.19.1)
rails-observers (0.1.5)
activemodel (>= 4.0)
rails-perftest (0.0.7)
end
end
+ def permission_to_create
+ if !super
+ return false
+ elsif group_class == "role" &&
+ !Rails.configuration.Users.CanCreateRoleGroups &&
+ !current_user.andand.is_admin
+ raise PermissionDeniedError.new("this cluster does not allow users to create role groups")
+ else
+ return true
+ end
+ end
+
def permission_to_update
if !super
return false
arvcfg.declare_config "Users.UserNotifierEmailBcc", Hash
arvcfg.declare_config "Users.NewUserNotificationRecipients", Hash, :new_user_notification_recipients, ->(cfg, k, v) { arrayToHash cfg, "Users.NewUserNotificationRecipients", v }
arvcfg.declare_config "Users.NewInactiveUserNotificationRecipients", Hash, :new_inactive_user_notification_recipients, method(:arrayToHash)
+arvcfg.declare_config "Users.CanCreateRoleGroups", Boolean
arvcfg.declare_config "Users.RoleGroupsVisibleToAll", Boolean
arvcfg.declare_config "Login.LoginCluster", String
arvcfg.declare_config "Login.TrustedClients", Hash
assert proj.update_attributes(frozen_by_uuid: users(:active).uuid)
end
end
+
+ [
+ [false, :admin, true],
+ [false, :active, false],
+ [true, :admin, true],
+ [true, :active, true],
+ [true, :inactive, false],
+ ].each do |conf, user, allowed|
+ test "config.Users.CanCreateRoleGroups conf=#{conf}, user=#{user}" do
+ Rails.configuration.Users.CanCreateRoleGroups = conf
+ act_as_user users(user) do
+ if allowed
+ Group.create!(name: 'admin-created', group_class: 'role')
+ else
+ assert_raises(ArvadosModel::PermissionDeniedError) do
+ Group.create!(name: 'user-created', group_class: 'role')
+ end
+ end
+ end
+ end
+ end
end
disk_cache=self.args.disk_cache,
disk_cache_dir=self.args.disk_cache_dir),
'num_retries': self.args.retries,
- })
+ },
+ version='v1',
+ )
except KeyError as e:
self.logger.error("Missing environment: %s", e)
exit(1)
run_test_server.run()
run_test_server.authorize_with("admin")
- self.api = api if api else arvados.safeapi.ThreadSafeApiCache(arvados.config.settings(), keep_params={"block_cache": make_block_cache(self.disk_cache)})
+ self.api = api if api else arvados.safeapi.ThreadSafeApiCache(
+ arvados.config.settings(),
+ keep_params={"block_cache": make_block_cache(self.disk_cache)},
+ version='v1',
+ )
self.llfuse_thread = None
# This is a copy of Mount's method. TODO: Refactor MountTestBase
def setUp(self):
super(SlashSubstitutionTest, self).setUp()
- self.api = arvados.safeapi.ThreadSafeApiCache(arvados.config.settings())
+ self.api = arvados.safeapi.ThreadSafeApiCache(
+ arvados.config.settings(),
+ version='v1',
+ )
self.api.config = lambda: {"Collections": {"ForwardSlashNameSubstitution": "[SLASH]"}}
self.testcoll = self.api.collections().create(body={"name": "foo/bar/baz"}).execute()
self.testcolleasy = self.api.collections().create(body={"name": "foo-bar-baz"}).execute()
def setUp(self):
super(StorageClassesTest, self).setUp()
- self.api = arvados.safeapi.ThreadSafeApiCache(arvados.config.settings())
+ self.api = arvados.safeapi.ThreadSafeApiCache(
+ arvados.config.settings(),
+ version='v1',
+ )
@IntegrationTest.mount(argv=mnt_args)
def test_collection_default_storage_classes(self):
def runTest(self):
settings = arvados.config.settings().copy()
settings["ARVADOS_API_TOKEN"] = run_test_server.fixture("api_client_authorizations")["project_viewer"]["api_token"]
- self.api = arvados.safeapi.ThreadSafeApiCache(settings)
+ self.api = arvados.safeapi.ThreadSafeApiCache(settings, version='v1')
self.make_mount(fuse.CollectionDirectory, collection_record=self.testcollection, enable_write=False)
self.pool.apply(_readonlyCollectionTestHelper, (self.mounttmp,))
projects / arvados.git / commitdiff