8 Copyright (C) The Arvados Authors. All rights reserved.
10 SPDX-License-Identifier: CC-BY-SA-3.0
13 # "Introduction":#introduction
14 # "Working with the current user":#working-with-current-user
15 ## "Fetch the current user":#fetch-current-user
16 ## "List objects shared with the current user":#list-shared-objects
17 # "Working with projects":#working-with-projects
18 ## "Create a project":#create-a-project
19 ## "List the contents of a project":#list-project-contents
20 # "Working with permissions":#working-with-permissions
21 ## "Grant permission to an object":#grant-permission
22 ## "Modify permission on an object":#modify-permission
23 ## "Revoke permission from an object":#revoke-permission
24 # "Working with properties":#working-with-properties
25 ## "Update the properties of an object":#update-properties
26 ## "Translate between vocabulary identifiers and labels":#translating-between-vocabulary-identifiers-and-labels
27 ## "Query the vocabulary definition":#querying-the-vocabulary-definition
28 # "Working with collections":#working-with-collections
29 ## "Load and update an existing collection":#load-collection
30 ## "Create and save a new collection":#create-collection
31 ## "Read a file from a collection":#read-a-file-from-a-collection
32 ## "Download a file from a collection":#download-a-file-from-a-collection
33 ## "Write a file to a collection":#write-a-file-into-a-new-collection
34 ## "Upload a file to a collection":#upload-a-file-into-a-new-collection
35 ## "Delete a file from a collection":#delete-a-file-from-an-existing-collection
36 ## "Delete a directory from a collection recursively":#delete-a-directory-from-a-collection
37 ## "Walk over all files in a collection":#walk-collection
38 ## "Copy a file between collections":#copy-files-from-a-collection-to-another-collection
39 ## "Combine two or more collections":#combine-two-or-more-collections
40 ## "Create a collection sharing link":#sharing-link
41 # "Working with containers and workflow runs":#working-with-containers
42 ## "Get input of a container":#get-input-of-a-container
43 ## "Get input of a CWL workflow run":#get-input-of-a-cwl-workflow
44 ## "Get output of a container":#get-output-of-a-container
45 ## "Get output of a CWL workflow run":#get-output-of-a-cwl-workflow
46 ## "Get logs of a container or CWL workflow run":#get-log-of-a-child-request
47 ## "Get status of a container or CWL workflow run":#get-state-of-a-cwl-workflow
48 ## "List child requests of a container or CWL workflow run":#list-failed-child-requests
49 ## "List child requests of a container request":#list-child-requests-of-container-request
50 # "Working with the container request queue":#working-with-container-request-queue
51 ## "List completed container requests":#list-completed-container-requests
52 ## "Cancel a container request":#cancel-a-container-request
53 ## "Cancel multiple pending container requests":#cancel-all-container-requests
55 h2(#introduction). Introduction
57 This page provides example code to perform various high-level tasks using Arvados' Python SDK. This page assumes you've already read the "API client documentation":{{ site.baseurl }}/sdk/python/api-client.html and understand the basics of using the Python SDK client. You don't have to have the details of every API method memorized, but you should at least be comfortable with the pattern of calling a resource type, API method, and @execute()@, as well as the dictionaries these methods return.
59 The code examples assume you've built the @arv_client@ object by doing something like:
61 {% codeblock as python %}
63 arv_client = arvados.api('v1', ...)
66 These examples work no matter how you call @arvados.api()@, or if you use another constructor from "@arvados.api@ module":{{ site.baseurl }}/sdk/python/arvados/api.html. Just understand that @arv_client@ represents your client object, no matter how you built it.
68 Whenever you see the Ellipsis object @...@ in these examples, that means you may need or want to fill something in. That might be list items, function arguments, or your own code. Comments will provide additional guidance.
70 Whenever you see the example UUID @zzzzz-zzzzz-12345abcde67890@, you should provide your own UUID from input.
72 h2(#working-with-current-user). Working with the current user
74 h3(#fetch-current-user). Fetch the current user
76 The API provides a "dedicated users method named @current@":{{ site.baseurl }}/api/methods/users.html#current. It returns the user object that is authenticated by your current API token. Use this method to get the current user's UUID to use in other API calls, or include user details like name in your output.
78 {% codeblock as python %}
79 current_user = arv_client.users().current().execute()
82 h3(#list-shared-objects). List objects shared with the current user
84 The API provides a "dedicated groups method named @shared@":{{ site.baseurl }}/api/methods/groups.html#shared to do this. Call it like you would any other list method. This example illustrates some popular arguments. Check the API reference for full details of all possible arguments.
86 {% codeblock as python %}
87 for item in arvados.util.keyset_list_all(
88 # Pass the method keyset_list_all will call to retrieve items.
89 # Do not call it yourself.
90 arv_client.groups().shared,
91 # Pass filters to limit what objects are returned.
92 # This example returns only subprojects.
94 ['uuid', 'is_a', 'arvados#group'],
95 ['group_class', '=', 'project'],
97 # Pass order_key and ascending to control how the contents are sorted.
98 # This example lists projects in ascending creation time (the default).
99 order_key='created_at',
102 ... # Work on item as desired
105 h2(#working-with-projects). Working with projects
107 h3(#create-a-project). Create a project
109 A project is represented in the Arvados API as a group with its @group_class@ field set to @"project"@.
111 {% codeblock as python %}
112 new_project = arv_client.groups().create(
115 'group_class': 'project',
116 'name': 'Python SDK Test Project',
117 # owner_uuid can be the UUID for an Arvados user or group.
118 # Specify the UUID of an existing project to make a subproject.
119 # If not specified, the current user is the default owner.
120 'owner_uuid': 'zzzzz-j7d0g-12345abcde67890',
123 ensure_unique_name=True,
127 h3(#list-project-contents). List the contents of a project
129 The API provides a "dedicated groups method named @contents@":{{ site.baseurl }}/api/methods/groups.html#contents to do this. Call it like you would any other list method. This example illustrates some popular arguments. Check the API reference for full details of all possible arguments.
131 {% codeblock as python %}
132 current_user = arv_client.users().current().execute()
133 for item in arvados.util.keyset_list_all(
134 # Pass the method keyset_list_all will call to retrieve items.
135 # Do not call it yourself.
136 arv_client.groups().contents,
137 # The UUID of the project whose contents we're listing.
138 # Pass a user UUID to list their home project.
139 # This example lists the current user's home project.
140 uuid=current_user['uuid'],
141 # Pass filters to limit what objects are returned.
142 # This example returns only subprojects.
144 ['uuid', 'is_a', 'arvados#group'],
145 ['group_class', '=', 'project'],
147 # Pass recursive=True to include results from subprojects in the listing.
149 # Pass include_trash=True to include objects in the listing whose
150 # trashed_at time is passed.
153 ... # Work on item as desired
156 h2(#working-with-permissions). Working with permissions
158 In brief, a permission is represented in Arvados as a link object with the following values:
160 * @link_class@ is @"permission"@.
161 * @name@ is one of @"can_read"@, @"can_write"@, @"can_manage"@, or @"can_login"@.
162 * @tail_uuid@ identifies the user or role group that receives the permission.
163 * @head_uuid@ identifies the Arvados object this permission grants access to.
165 For details, refer to the "Permissions model documentation":{{ site.baseurl }}/api/permission-model.html. Managing permissions is just a matter of ensuring the desired links exist using the standard @create@, @update@, and @delete@ methods.
167 h3(#grant-permission). Grant permission to an object
169 Create a link with values as documented above.
171 {% codeblock as python %}
172 permission = arv_client.links().create(
175 'link_class': 'permission',
176 # Adjust name for the level of permission you want to grant
178 # tail_uuid must identify a user or role group
179 'tail_uuid': 'zzzzz-tpzed-12345abcde67890',
180 # head_uuid can identify any Arvados object
181 'head_uuid': 'zzzzz-4zz18-12345abcde67890',
187 h3(#modify-permission). Modify permission on an object
189 To modify an existing permission—for example, to change its access level—find the existing link object for the permission, then update it with the new values you want. This example shows changing all read-write permissions on a specific collection to read-only. Adjust the filters appropriately to find the permission(s) you want to modify.
191 {% codeblock as python %}
193 for permission in arvados.util.keyset_list_all(
194 # Pass the method keyset_list_all will call to retrieve items.
195 # Do not call it yourself.
196 arv_client.links().list,
198 # You should use this filter for all permission searches,
199 # to exclude other kinds of links.
200 ['link_class', '=', 'permission'],
201 # Add other filters as desired.
202 ['name', '=', 'can_write'],
203 ['head_uuid', '=', 'zzzzz-4zz18-12345abcde67890'],
207 arv_client.links().update(
208 uuid=permission['uuid'],
217 h3(#revoke-permission). Revoke permission from an object
219 To revoke an existing permission, find the existing link object for the permission, then delete it. This example shows revoking one user's permission to log into any virtual machines. Adjust the filters appropriately to find the permission(s) you want to revoke.
221 {% codeblock as python %}
223 for permission in arvados.util.keyset_list_all(
224 # Pass the method keyset_list_all will call to retrieve items.
225 # Do not call it yourself.
226 arv_client.links().list,
228 # You should use this filter for all permission searches,
229 # to exclude other kinds of links.
230 ['link_class', '=', 'permission'],
231 # Add other filters as desired.
232 ['name', '=', 'can_login'],
233 ['tail_uuid', '=', 'zzzzz-tpzed-12345abcde67890'],
237 arv_client.links().delete(
238 uuid=permission['uuid'],
242 h2(#working-with-properties). Working with properties
244 Container requests, collections, groups, and links can have metadata properties set through their @properties@ field. For details, refer to the "Metadata properties API reference":{{ site.baseurl }}/api/properties.html.
246 An Arvados cluster can be configured to use a metadata vocabulary. If this is set up, the vocabulary defines standard identifiers and specific properties and their values. These identifiers can also have more human-friendly aliases. The cluster can also be configured to use the vocabulary strictly, so clients may _only_ set properties on objects that are defined in the vocabulary. For more information about configuring a metadata vocabulary, refer to the "Metadata vocabulary administration documentation":{{ site.baseurl }}/admin/metadata-vocabulary.html.
248 h3(#update-properties). Update the properties of an object
250 To set an object's properties to a new value, just call the resource's @update@ method with a new @properties@ field in the body. If you want to make changes to the current set of properties, @get@ the object, build a new dictionary based on its @properties@ field, then call the resource's @update@ method with your new dictionary as the @properties@. Below is an example for a container request.
252 {% codeblock as python %}
253 container_request = arv_client.container_requests().get(
254 uuid='zzzzz-xvhdp-12345abcde67890',
256 new_properties = dict(container_request['properties'])
257 ... # Make your desired changes to new_proprties
258 container_request = arv_client.container_requests().update(
259 uuid=container_request['uuid'],
261 'container_request': {
262 'properties': new_properties,
268 h3(#translating-between-vocabulary-identifiers-and-labels). Translate between vocabulary identifiers and labels
270 Client software might need to present properties to the user in a human-readable form or take input from the user without requiring them to remember identifiers. The "@Vocabulary.convert_to_labels@":{{ site.baseurl }}/sdk/python/arvados/vocabulary.html#arvados.vocabulary.Vocabulary.convert_to_labels and "@Vocabulary.convert_to_identifiers@":{{ site.baseurl }}/sdk/python/arvados/vocabulary.html#arvados.vocabulary.Vocabulary.convert_to_identifiers methods help with these tasks, respectively.
272 {% codeblock as python %}
273 import arvados.vocabulary
274 vocabulary = arvados.vocabulary.load_vocabulary(arv_client)
276 # The argument should be a mapping of vocabulary keys and values using any
277 # defined aliases, like this:
278 # {'Creature': 'Human', 'Priority': 'Normal'}
279 # The return value will be an analogous mapping where all the aliases have
280 # been translated to identifiers, like this:
281 # {'IDTAGANIMALS': 'IDVALANIMALS2', 'IDTAGIMPORTANCES': 'IDTAGIMPORTANCES1'}
282 properties_by_identifier = vocabulary.convert_to_identifiers({...})
284 # You can use this to set metadata properties on objects that support them.
285 project = arv_client.groups().update(
286 uuid='zzzzz-j7d0g-12345abcde67890',
289 'properties': properties_by_identifier,
294 # You can report properties to the user by their preferred name.
295 print(f"{project['name']} ({project['group_class']} {project['uuid']}) updated with properties:")
296 for key, value in vocabulary.convert_to_labels(project['properties']).items():
297 print(f"↳ {key}: {value}")
300 h3(#querying-the-vocabulary-definition). Query the vocabulary definition
302 The @arvados.vocabulary@ module provides facilities to interact with the "active metadata vocabulary":{{ site.baseurl }}/admin/metadata-vocabulary.html in the system. The "@Vocabulary@ class":{{ site.baseurl }}/sdk/python/arvados/vocabulary.html#arvados.vocabulary.Vocabulary provides a mapping-like view of a cluster's configured vocabulary.
304 {% codeblock as python %}
305 import arvados.vocabulary
306 vocabulary = arvados.vocabulary.load_vocabulary(arv_client)
308 # You can use the vocabulary object to access specific keys and values by
309 # case-insensitive mapping, like this:
310 # vocabulary_value = vocabulary[key_alias][value_alias]
311 # You can also access the `key_aliases` and `value_aliases` mapping
312 # attributes directly to view the entire vocabulary. The example below
313 # writes a plaintext table of the vocabulary.
314 for vocabulary_key in set(vocabulary.key_aliases.values()):
316 vocabulary_key.identifier,
317 vocabulary_key.preferred_label,
318 ', '.join(vocabulary_key.aliases[1:]),
321 for vocabulary_value in set(vocabulary_key.value_aliases.values()):
323 f'↳ {vocabulary_value.identifier}',
324 vocabulary_value.preferred_label,
325 ', '.join(vocabulary_value.aliases[1:]),
330 h2(#working-with-collections). Working with collections
332 The "@arvados.collection.Collection@ class":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.Collection provides a high-level interface to read, create, and update collections. It orchestrates multiple requests to API and Keep so you don't have to worry about the low-level details of keeping everything in sync. It uses threads to make multiple requests to Keep in parallel.
334 This page only shows you how to perform common tasks using the @Collection@ class. To see all the supported constructor arguments and methods, refer to "the @Collection@ class documentation":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.Collection.
336 h3(#load-collection). Load and update an existing collection
338 Construct the @Collection@ class with the UUID of a collection you want to read. You can pass additional constructor arguments as needed.
340 {% codeblock as python %}
341 import arvados.collection
342 collection = arvados.collection.Collection('zzzzz-4zz18-12345abcde67890', ...)
345 If you make changes to the collection and want to update the existing collection, call the "@Collection.save@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.Collection.save:
347 {% codeblock as python %}
351 If you would rather save your changes as a new collection object, call the "@Collection.save_new@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.Collection.save_new. This example illustrates some popular arguments. Check the API reference for full details of all possible arguments.
353 {% codeblock as python %}
355 name='Collection updated by Python SDK',
356 # owner_uuid can be the UUID for an Arvados user or group.
357 # Specify the UUID of a project to add this collection to it.
358 owner_uuid='zzzzz-j7d0g-12345abcde67890',
362 h3(#create-collection). Create and save a new collection
364 Construct the @Collection@ class without an existing collection UUID or manifest text. You can pass additional constructor arguments as needed.
366 {% codeblock as python %}
367 import arvados.collection
368 new_collection = arvados.collection.Collection(...)
371 Usually you'll upload or copy files to the new collection. Once you're done with that and ready to save your changes, call the "@Collection.save_new@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.Collection.save_new. This example illustrates some popular arguments. Check the API reference for full details of all possible arguments.
373 {% codeblock as python %}
374 new_collection.save_new(
375 name='Collection created by Python SDK',
376 # owner_uuid can be the UUID for an Arvados user or group.
377 # Specify the UUID of a project to add this collection to it.
378 owner_uuid='zzzzz-j7d0g-12345abcde67890',
382 h3(#read-a-file-from-a-collection). Read a file from a collection
384 Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. It returns a file-like object that you can use in many of the same ways you would use any other file object. This example prints all non-empty lines from @ExampleFile@ in your collection:
386 {% codeblock as python %}
387 import arvados.collection
388 collection = arvados.collection.Collection(...)
389 with collection.open('ExampleFile') as my_file:
390 # Read from my_file as desired.
391 # This example prints all non-empty lines from the file to stdout.
393 if not line.isspace():
397 h3(#download-a-file-from-a-collection). Download a file from a collection
399 Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. You pass a second mode argument like @'rb'@ to open the file in binary mode. It returns a file-like object that you can use in many of the same ways you would use any other file object. You can pass it as a source to Python's standard "@shutil.copyfileobj@ function":https://docs.python.org/3/library/shutil.html#shutil.copyfileobj to download it. This code downloads @ExampleFile@ from your collection and saves it to the current working directory as @ExampleDownload@:
401 {% codeblock as python %}
402 import arvados.collection
404 collection = arvados.collection.Collection(...)
406 collection.open('ExampleFile', 'rb') as src_file,
407 open('ExampleDownload', 'wb') as dst_file,
409 shutil.copyfileobj(src_file, dst_file)
412 h3(#write-a-file-into-a-new-collection). Write a file to a collection
414 Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. Pass a second mode argument like @'w'@, @'a'@, or @'wb'@ to write a file in the collection. It returns a file-like object that you can use in many of the same ways you would use any other file object. This example writes @Hello, Arvados!@ to a file named @ExampleHello@ in your collection:
416 {% codeblock as python %}
417 import arvados.collection
418 collection = arvados.collection.Collection(...)
419 with collection.open('ExampleFile', 'w') as my_file:
420 # Write to my_file as desired.
421 # This example writes "Hello, Arvados!" to the file.
422 print("Hello, Arvados!", file=my_file)
423 collection.save_new(...) # or collection.save() to update an existing collection
426 h3(#upload-a-file-into-a-new-collection). Upload a file to a collection
428 Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. Pass a second mode argument like @'w'@, @'a'@, or @'wb'@ to write a file in the collection. It returns a file-like object that you can use in many of the same ways you would use any other file object. You can pass it as a destination to Python's standard "@shutil.copyfileobj@ function":https://docs.python.org/3/library/shutil.html#shutil.copyfileobj to upload data from a source file. This example reads @ExampleFile@ from the current working directory and uploads it into your collection as @ExampleUpload@:
430 {% codeblock as python %}
431 import arvados.collection
433 collection = arvados.collection.Collection(...)
435 open('ExampleFile', 'rb') as src_file,
436 collection.open('ExampleUpload', 'wb') as dst_file,
438 shutil.copyfileobj(src_file, dst_file)
439 collection.save_new(...) # or collection.save() to update an existing collection
442 h3(#delete-a-file-from-an-existing-collection). Delete a file from a collection
444 Once you have a @Collection@ object, call the "@Collection.remove@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.Collection.remove with a file path to remove that file or directory from the collection.
446 {% codeblock as python %}
447 import arvados.collection
448 collection = arvados.collection.Collection(...)
449 collection.remove('ExamplePath')
450 collection.save_new(...) # or collection.save() to update an existing collection
453 h3(#delete-a-directory-from-a-collection). Delete a directory from a collection recursively
455 Once you have a @Collection@ object, call the "@Collection.remove@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.Collection.remove with a directory path and @recursive=True@ to delete everything under that directory from the collection.
457 {% codeblock as python %}
458 import arvados.collection
459 collection = arvados.collection.Collection(...)
460 collection.remove('ExampleDirectoryPath', recursive=True)
461 collection.save_new(...) # or collection.save() to update an existing collection
464 h3(#walk-collection). Walk over all files in a collection
466 Once you have a @Collection@ object, you can iterate over it to retrieve the names of all files and streams in it. Streams are like subdirectories: you can open them using the "@Collection.find@ method":{{ site.baseurl }}/sdk/python/python.html, and work with the files in them just like you would in the original collection. This example shows how to combine these techniques to iterate all files in a collection, including its streams.
468 {% codeblock as python %}
469 import arvados.collection
472 root_collection = arvados.collection.Collection(...)
473 # Start work from the base stream.
474 stream_queue = collections.deque(['.'])
476 stream_name = stream_queue.popleft()
477 collection = root_collection.find(stream_name)
478 for item_name in collection:
480 my_file = collection.open(item_name)
481 except IsADirectoryError:
482 # item_name refers to a stream. Queue it to walk later.
483 stream_path = pathlib.Path(stream_name, item_name)
484 stream_queue.append(stream_path.as_posix())
487 ... # Work with my_file as desired
490 h3(#copy-files-from-a-collection-to-another-collection). Copy a file between collections
492 Once you have one or more @Collection@ objects, call the "@Collection.copy@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.copy on the destination collection to copy files to it. This method doesn't re-upload data, so it's very efficient.
494 {% codeblock as python %}
495 import arvados.collection
496 src_collection = arvados.collection.Collection(...)
497 dst_collection = arvados.collection.Collection(...)
499 # The path of the source file or directory to copy
501 # The path where the source file or directory will be copied.
502 # Pass the empty string like this to copy it to the same path.
504 # The collection where the source file or directory comes from.
505 # If not specified, the default is the current collection (so you'll
506 # make multiple copies of the same data in the same collection).
507 source_collection=src_collection,
508 # Pass overwrite=True to force the method to overwrite any data
509 # that already exists at the given path in the current collection.
512 dst_collection.save_new(...) # or dst_collection.save() to update an existing collection
515 h3(#combine-two-or-more-collections). Combine two or more collections
517 You can concatenate manifest texts from multiple collections to create a single collection that contains all the data from the source collections. Note that if multiple source collections have data at the same path, the merged collection will have a single file at that path with concatenated data from the source collections.
519 {% codeblock as python %}
520 import arvados.collection
522 # Retrieve all of the source collection manifest texts
523 src_collection_uuid_list = [
524 'zzzzz-4zz18-111111111111111',
525 'zzzzz-4zz18-222222222222222',
529 arvados.collection.Collection(uuid).manifest_text()
530 for uuid in src_collection_uuid_list
533 # Initialize a new collection object from the concatenated manifest text
534 new_collection = arvados.collection.Collection(''.join(manifest_texts), ...)
536 # Record the new collection in Arvados
537 new_collection.save_new(
538 name='Collection merged by Python SDK',
539 owner_uuid='zzzzz-j7d0g-12345abcde67890',
543 h3(#sharing-link). Create a collection sharing link
545 You can create a sharing link for a collection by creating a new API token that is only allowed to read that collection; then constructing a link to your Keep web server that includes the collection UUID and the new token.
547 {% codeblock as python %}
550 # The UUID of the collection you want to share
551 collection_uuid = 'zzzzz-4zz18-12345abcde67890'
553 sharing_token_scopes = [
554 'GET /arvados/v1/keep_services/accessible',
555 f'GET /arvados/v1/collections/{collection_uuid}',
556 f'GET /arvados/v1/collections/{collection_uuid}/',
558 sharing_token = arv_client.api_client_authorizations().create(
560 'api_client_authorization': {
561 'scopes': sharing_token_scopes,
565 plain_token = sharing_token['api_token']
566 token_parts = plain_token.split('/')
567 if token_parts[0] == 'v2':
568 plain_token = token_parts[2]
570 sharing_url_parts = (
571 # The scheme your Keep web server uses. Change this to 'http' if necessary.
573 # The hostname, and optionally port, your Keep web server uses
574 'collections.zzzzz.example.com',
575 # You shouldn't need to change any other items
576 f'/c={collection_uuid}/t={plain_token}/_/',
580 sharing_url = urllib.parse.urlunsplit(sharing_url_parts)
584 h2(#working-with-containers). Working with containers
586 If you haven't already, start by reading the "Computing with Crunch":{{ site.baseurl }}/api/execution.html guide. It provides a high-level overview of how users submit work to Arvados as container requests; how Arvados dispatches that work to containers; and how Arvados records the association and results back on the original container request record.
588 If you have experience running CWL workflows on Workbench 2, it runs through this same API. When you start that workflow run, Workbench 2 creates a small container request to run a "CWL runner" tool with the specific inputs you gave it. Once Crunch dispatches a container for it, the CWL runner creates additional container requests to run each step of the workflow, and oversees the process until the workflow runs to completion. The UUID of this container is recorded in the @container_uuid@ field of the container request you submitted.
590 The UUID of the CWL runner container is recorded in the @requesting_container_uuid@ field of each container request it creates. You can list container requests with a filter on this field to inspect each step of the workflow individually, as shown below.
592 The next few examples show how to perform a task with a container request generally, and then provide a more specific example of working with a CWL runner container.
594 h3(#get-input-of-a-container). Get input of a container
596 A container request's most varied inputs are recorded in the @mounts@ field, which can include data from Keep, specific collections, Git checkouts, and static files. You might also be interested in the @environment@, @command@, @container_image@, and @secret_mounts@ fields. Refer to the "container requests API documentation":{{ site.baseurl }}/api/methods/container_requests.html for details.
598 {% codeblock as python %}
599 container_request = arv_client.container_requests().get(
600 uuid='zzzzz-xvhdp-12345abcde67890',
602 # From here, you can process any of the container request's input fields.
603 # Below is an example of listing all the mounts.
605 for mount_name, mount_source in container_request['mounts'].items():
607 # These are the fields that define different types of mounts.
608 # Try to collect them all. Just skip any that aren't set.
609 for key in ['kind', 'uuid', 'portable_data_hash', 'commit', 'path']:
611 mount_summary.append(mount_source[key])
614 print(f"{mount_name}: {' '.join(mount_summary)}")
615 if mount_source.get('kind') == 'json':
616 pprint.pprint(mount_source.get('content'))
619 h3(#get-input-of-a-cwl-workflow). Get input of a CWL workflow run
621 When you run a CWL workflow, the CWL inputs are stored in the container request's @mounts@ field as a JSON mount named @/var/lib/cwl/cwl.input.json@.
623 {% codeblock as python %}
624 container_request = arv_client.container_requests().get(
625 uuid='zzzzz-xvhdp-12345abcde67890',
627 cwl_input = container_request['mounts']['/var/lib/cwl/cwl.input.json']['content']
628 ... # Work with the cwl_input dictionary
631 h3(#get-output-of-a-container). Get output of a container
633 A container's output files are saved in a collection. The UUID of that collection is recorded in the @output_uuid@ of the container request, which you can load as you like.
635 {% codeblock as python %}
636 import arvados.collection
637 container_request = arv_client.container_requests().get(
638 uuid='zzzzz-xvhdp-12345abcde67890',
640 container_output = arvados.collection.Collection(
641 container_request.get('output_uuid'),
643 ... # Work with the container_output collection object
646 h3(#get-output-of-a-cwl-workflow). Get output of a CWL workflow run
648 When you run a CWL workflow, the container request's output collection includes a file named @cwl.output.json@ that provides additional information about other files in the output.
650 {% codeblock as python %}
651 import arvados.collection
653 cwl_container_request = arv_client.container_requests().get(
654 uuid='zzzzz-xvhdp-12345abcde67890',
656 cwl_output_collection = arvados.collection.Collection(
657 cwl_container_request['output_uuid'],
659 with cwl_output_collection.open('cwl.output.json') as cwl_output_file:
660 cwl_output = json.load(cwl_output_file)
661 ... # Work with the cwl_output dictionary
664 h3(#get-log-of-a-child-request). Get logs of a container or CWL workflow run
666 A container's log files are saved in a collection. The UUID of that collection is recorded in the @log_uuid@ of the container request, which you can load as you like.
668 {% codeblock as python %}
669 import arvados.collection
670 container_request = arv_client.container_requests().get(
671 uuid='zzzzz-xvhdp-12345abcde67890',
673 log_collection = arvados.collection.Collection(
674 container_request['log_uuid'],
676 # From here, you can process the container's log collection any way you like.
677 # Below is an example that writes the container's stderr to this process' stderr.
680 with log_collection.open('stderr.txt') as containter_stderr:
681 shutil.copyfileobj(container_stderr, sys.stderr)
684 h3(#get-state-of-a-cwl-workflow). Get status of a container or CWL workflow run
686 Workbench shows users a single status badge for container requests. This status is synthesized from different fields on the container request and associated container. This code shows how to do analogous reporting using the Python SDK.
688 {% codeblock as python %}
689 container_request = arv_client.container_requests().get(
690 uuid='zzzzz-xvhdp-12345abcde67890',
692 if container_request['container_uuid'] is None:
693 status = container_request['state']
695 container = arv_client.containers().get(
696 uuid=container_request['container_uuid'],
698 container_state = container['state']
699 if container_state == 'Queued' or container_state == 'Locked':
700 status = "On hold" if container['priority'] == 0 else "Queued"
701 elif container_state == 'Running':
702 if container['runtime_status'].get('error'):
704 elif container['runtime_status'].get('warning'):
707 status = container_state
708 elif container_state == 'Cancelled':
709 status = container_state
710 elif container_state == 'Complete':
711 status = "Completed" if container['exit_code'] == 0 else "Failed"
712 ... # Report status as desired
715 h3(#list-failed-child-requests). List child requests of a container or CWL workflow run
717 When a running container creates a container request to do additional work, the UUID of the source container is recorded in the @requesting_container_uuid@ field of the new container request. You can list container requests with this filter to find requests created by a specific container.
719 {% codeblock as python %}
721 for child_container_requests in arvados.util.keyset_list_all(
722 # Pass the method keyset_list_all will call to retrieve items.
723 # Do not call it yourself.
724 arv_client.container_requests().list,
726 # Note this is a container UUID, *not* a container request UUID
727 ['requesting_container_uuid', '=', 'zzzzz-dz642-12345abcde67890'],
728 # You may add other filters for your listing.
729 # For example, you could filter by 'name' to find specific kinds
730 # of steps of a CWL workflow.
734 ... # Work with each child container request
737 h3(#list-child-requests-of-container-request). List child requests of a container request
739 When a running container creates a container request to do additional work, the UUID of the source container is recorded in the @requesting_container_uuid@ field of the new container request. If all you have is the UUID of a container request, you can get that request, then list container requests with a filter where @requesting_container_uuid@ matches the @container_uuid@ of your request to find all its children.
741 {% codeblock as python %}
743 parent_container_request = arv_client.container_requests().get(
744 uuid='zzzzz-xvhdp-12345abcde67890',
746 parent_container_uuid = parent_container_request['container_uuid']
747 if parent_container_uuid is None:
748 # No container has run for this request yet, so there cannot be child requests.
749 child_container_requests = ()
751 child_container_requests = arvados.util.keyset_list_all(
752 # Pass the method keyset_list_all will call to retrieve items.
753 # Do not call it yourself.
754 arv_client.container_requests().list,
756 ['requesting_container_uuid', '=', parent_container_uuid],
757 # You may add other filters for your listing.
758 # For example, you could filter by 'name' to find specific kinds
759 # of steps of a CWL workflow.
763 for child_container_request in child_container_requests:
764 ... # Work with each child container request
767 With each child container request, you could repeat any of the recipes listed earlier in this section: examine their status, inputs, outputs, logs, and so on.
769 h2(#working-with-container-request-queue). Working with the container request queue
771 h3(#list-completed-container-requests). List completed container requests
773 Completed container requests have their @state@ field set to @"Final"@. You can list container requests with this filter to find completed requests.
775 {% codeblock as python %}
778 time_filter = datetime.datetime.utcnow()
779 time_filter -= datetime.timedelta(days=7)
781 for container_request in arvados.util.keyset_list_all(
782 # Pass the method keyset_list_all will call to retrieve items.
783 # Do not call it yourself.
784 arv_client.container_requests().list,
786 # This is the filter you need to find completed container requests.
787 ['state', '=', 'Final'],
788 # There could be many completed container requests, so you should
789 # provide additional filters. This example limits the listing to
790 # container requests from the past week.
791 ['created_at', '>=', f'{time_filter.isoformat()}Z'],
795 # Work with each container_request as desired.
796 # This example provides a basic status table with the container request
797 # UUID, time the request was created, and time the container finished
800 container_request['uuid'],
801 container_request['created_at'],
802 container_request['modified_at'],
806 h3(#cancel-a-container-request). Cancel a container request
808 To cancel a container request, update it to set its @priority@ field to 0. See the "containers API reference":{{ site.baseurl }}/api/methods/containers.html for details.
810 {% codeblock as python %}
811 cancelled_container_request = arv_client.container_requests().update(
812 uuid='zzzzz-xvhdp-12345abcde67890',
814 'container_request': {
821 h3(#cancel-all-container-requests). Cancel multiple pending container requests
823 If you want to cancel multiple pending container requests, you can list container requests with the @state@ field set to @"Committed"@, a @priority@ greater than zero, and any other filters you like. Then update each container request to set its @priority@ field to 0. See the "containers API reference":{{ site.baseurl }}/api/methods/containers.html for details.
825 {% codeblock as python %}
827 for container_request in arvados.util.keyset_list_all(
828 # Pass the method keyset_list_all will call to retrieve items.
829 # Do not call it yourself.
830 arv_client.container_requests().list,
832 # These are the filters you need to find cancellable container requests.
833 ['state', '=', 'Committed'],
834 ['priority', '>', 0],
835 # You can add other filters as desired.
836 # For example, you might filter on `requesting_container_uuid` to
837 # cancel only steps of one specific workflow.
841 cancelled_container_request = arv_client.container_requests().update(
842 uuid=container_request['uuid'],
844 'container_request': {