## "Upload a file to a collection":#upload-a-file-into-a-new-collection
## "Delete a file from a collection":#delete-a-file-from-an-existing-collection
## "Delete a directory from a collection recursively":#delete-a-directory-from-a-collection
+## "Walk over all files in a collection":#walk-collection
## "Copy a file between collections":#copy-files-from-a-collection-to-another-collection
## "Combine two or more collections":#combine-two-or-more-collections
## "Create a collection sharing link":#sharing-link
# "Working with the container request queue":#working-with-container-request-queue
## "List completed container requests":#list-completed-container-requests
## "Cancel a container request":#cancel-a-container-request
+## "Cancel multiple pending container requests":#cancel-all-container-requests
h2(#introduction). Introduction
{% codeblock as python %}
for item in arvados.util.keyset_list_all(
- # Do *not* call the method here, just pass it.
+ # Pass the method keyset_list_all will call to retrieve items.
+ # Do not call it yourself.
arv_client.groups().shared,
# Pass filters to limit what objects are returned.
# This example returns only subprojects.
{% codeblock as python %}
current_user = arv_client.users().current().execute()
for item in arvados.util.keyset_list_all(
- # Do *not* call the method here, just pass it.
+ # Pass the method keyset_list_all will call to retrieve items.
+ # Do not call it yourself.
arv_client.groups().contents,
# The UUID of the project whose contents we're listing.
# Pass a user UUID to list their home project.
* @tail_uuid@ identifies the user or role group that receives the permission.
* @head_uuid@ identifies the Arvados object this permission grants access to.
-For details, refer to the "Permissions model documentation":{{ site.baseurl }}/api/permission-model.html. Managing permissions is just a matter of ensuring the desired links exist with the standard @create@, @update@, and @delete@ methods.
+For details, refer to the "Permissions model documentation":{{ site.baseurl }}/api/permission-model.html. Managing permissions is just a matter of ensuring the desired links exist using the standard @create@, @update@, and @delete@ methods.
h3(#grant-permission). Grant permission to an object
{% codeblock as python %}
import arvados.util
for permission in arvados.util.keyset_list_all(
- # Do *not* call the method here, just pass it.
+ # Pass the method keyset_list_all will call to retrieve items.
+ # Do not call it yourself.
arv_client.links().list,
filters=[
# You should use this filter for all permission searches,
{% codeblock as python %}
import arvados.util
for permission in arvados.util.keyset_list_all(
- # Do *not* call the method here, just pass it.
+ # Pass the method keyset_list_all will call to retrieve items.
+ # Do not call it yourself.
arv_client.links().list,
filters=[
# You should use this filter for all permission searches,
h3(#download-a-file-from-a-collection). Download a file from a collection
-Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. It returns a file-like object that you can use in many of the same ways you would use any other file object. You can pass it as a source to Python's standard "@shutil.copyfileobj@ function":https://docs.python.org/3/library/shutil.html#shutil.copyfileobj to download it. This code downloads @ExampleFile@ from your collection and saves it to the current working directory as @ExampleDownload@:
+Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. You pass a second mode argument like @'rb'@ to open the file in binary mode. It returns a file-like object that you can use in many of the same ways you would use any other file object. You can pass it as a source to Python's standard "@shutil.copyfileobj@ function":https://docs.python.org/3/library/shutil.html#shutil.copyfileobj to download it. This code downloads @ExampleFile@ from your collection and saves it to the current working directory as @ExampleDownload@:
{% codeblock as python %}
import arvados.collection
import shutil
collection = arvados.collection.Collection(...)
with (
- collection.open('ExampleFile') as src_file,
- open('ExampleDownload', 'w') as dst_file,
+ collection.open('ExampleFile', 'rb') as src_file,
+ open('ExampleDownload', 'wb') as dst_file,
):
shutil.copyfileobj(src_file, dst_file)
{% endcodeblock %}
h3(#write-a-file-into-a-new-collection). Write a file to a collection
-Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. Pass a second mode argument like @'w'@ or @'a'@ to write a file in the collection. It returns a file-like object that you can use in many of the same ways you would use any other file object. This example writes @Hello, Arvados!@ to a file named @ExampleHello@ in your collection:
+Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. Pass a second mode argument like @'w'@, @'a'@, or @'wb'@ to write a file in the collection. It returns a file-like object that you can use in many of the same ways you would use any other file object. This example writes @Hello, Arvados!@ to a file named @ExampleHello@ in your collection:
{% codeblock as python %}
import arvados.collection
collection = arvados.collection.Collection(...)
with collection.open('ExampleFile', 'w') as my_file:
# Write to my_file as desired.
- # This example writes "Hello, world!" to the file.
+ # This example writes "Hello, Arvados!" to the file.
print("Hello, Arvados!", file=my_file)
collection.save_new(...) # or collection.save() to update an existing collection
{% endcodeblock %}
h3(#upload-a-file-into-a-new-collection). Upload a file to a collection
-Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. Pass a second mode argument like @'w'@ or @'a'@ to write a file in the collection. It returns a file-like object that you can use in many of the same ways you would use any other file object. You can pass it as a destination to Python's standard "@shutil.copyfileobj@ function":https://docs.python.org/3/library/shutil.html#shutil.copyfileobj to upload data from a source file. This example reads @ExampleFile@ from the current working directory and uploads it into your collection as @ExampleUpload@:
+Once you have a @Collection@ object, the "@Collection.open@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.open lets you open files from a collection the same way you would open files from disk using Python's built-in @open@ function. Pass a second mode argument like @'w'@, @'a'@, or @'wb'@ to write a file in the collection. It returns a file-like object that you can use in many of the same ways you would use any other file object. You can pass it as a destination to Python's standard "@shutil.copyfileobj@ function":https://docs.python.org/3/library/shutil.html#shutil.copyfileobj to upload data from a source file. This example reads @ExampleFile@ from the current working directory and uploads it into your collection as @ExampleUpload@:
{% codeblock as python %}
import arvados.collection
import shutil
collection = arvados.collection.Collection(...)
with (
- open('ExampleFile') as src_file,
- collection.open('ExampleUpload', 'w') as dst_file,
+ open('ExampleFile', 'rb') as src_file,
+ collection.open('ExampleUpload', 'wb') as dst_file,
):
shutil.copyfileobj(src_file, dst_file)
collection.save_new(...) # or collection.save() to update an existing collection
collection.save_new(...) # or collection.save() to update an existing collection
{% endcodeblock %}
+h3(#walk-collection). Walk over all files in a collection
+
+Once you have a @Collection@ object, you can iterate over it to retrieve the names of all files and streams in it. Streams are like subdirectories: you can open them using the "@Collection.find@ method":{{ site.baseurl }}/sdk/python/python.html, and work with the files in them just like you would in the original collection. This example shows how to combine these techniques to iterate all files in a collection, including its streams.
+
+{% codeblock as python %}
+import arvados.collection
+import collections
+import pathlib
+root_collection = arvados.collection.Collection(...)
+# Start work from the base stream.
+stream_queue = collections.deque([pathlib.PurePosixPath('.')])
+while stream_queue:
+ stream_path = stream_queue.popleft()
+ collection = root_collection.find(str(stream_path))
+ for item_name in collection:
+ try:
+ my_file = collection.open(item_name)
+ except IsADirectoryError:
+ # item_name refers to a stream. Queue it to walk later.
+ stream_queue.append(stream_path / item_name)
+ continue
+ with my_file:
+ ... # Work with my_file as desired
+{% endcodeblock %}
+
h3(#copy-files-from-a-collection-to-another-collection). Copy a file between collections
Once you have one or more @Collection@ objects, call the "@Collection.copy@ method":{{ site.baseurl }}/sdk/python/arvados/collection.html#arvados.collection.RichCollectionBase.copy on the destination collection to copy files to it. This method doesn't re-upload data, so it's very efficient.
pprint.pprint(mount_source.get('content'))
{% endcodeblock %}
-h3(#get-input-of-a-cwl-workflow). Get input of a container or CWL workflow run
+h3(#get-input-of-a-cwl-workflow). Get input of a CWL workflow run
When you run a CWL workflow, the CWL inputs are stored in the container request's @mounts@ field as a JSON mount named @/var/lib/cwl/cwl.input.json@.
{% codeblock as python %}
import arvados.util
for child_container_requests in arvados.util.keyset_list_all(
- # Do *not* call the method here, just pass it.
+ # Pass the method keyset_list_all will call to retrieve items.
+ # Do not call it yourself.
arv_client.container_requests().list,
filters=[
# Note this is a container UUID, *not* a container request UUID
child_container_requests = ()
else:
child_container_requests = arvados.util.keyset_list_all(
- # Do *not* call the method here, just pass it.
+ # Pass the method keyset_list_all will call to retrieve items.
+ # Do not call it yourself.
arv_client.container_requests().list,
filters=[
['requesting_container_uuid', '=', parent_container_uuid],
time_filter -= datetime.timedelta(days=7)
for container_request in arvados.util.keyset_list_all(
- # Do *not* call the method here, just pass it.
+ # Pass the method keyset_list_all will call to retrieve items.
+ # Do not call it yourself.
arv_client.container_requests().list,
filters=[
# This is the filter you need to find completed container requests.
).execute()
{% endcodeblock %}
-p(#cancel-all-container-requests). If you want to cancel many container requests, you can list container requests with the @state@ field set to @"Committed"@, a @priority@ greater than zero, and any other filters you like. Then update each container request in turn.
+h3(#cancel-all-container-requests). Cancel multiple pending container requests
+
+If you want to cancel multiple pending container requests, you can list container requests with the @state@ field set to @"Committed"@, a @priority@ greater than zero, and any other filters you like. Then update each container request to set its @priority@ field to 0. See the "containers API reference":{{ site.baseurl }}/api/methods/containers.html for details.
{% codeblock as python %}
import arvados.util
for container_request in arvados.util.keyset_list_all(
- # Do *not* call the method here, just pass it.
+ # Pass the method keyset_list_all will call to retrieve items.
+ # Do not call it yourself.
arv_client.container_requests().list,
filters=[
# These are the filters you need to find cancellable container requests.
projects / arvados.git / blobdiff