arvados.git
7 years ago11349: Report wishlist size in status["nodes_wish"].
Tom Clegg [Mon, 10 Apr 2017 14:20:31 +0000 (10:20 -0400)]
11349: Report wishlist size in status["nodes_wish"].

7 years ago11332: Use urllib quote/unquote to handle special chars (e.g. '#')
Peter Amstutz [Mon, 10 Apr 2017 14:07:23 +0000 (10:07 -0400)]
11332: Use urllib quote/unquote to handle special chars (e.g. '#')

7 years ago11349: Add management server with /status.json
Tom Clegg [Fri, 7 Apr 2017 19:20:04 +0000 (15:20 -0400)]
11349: Add management server with /status.json

7 years ago8465: Set proper container flags to provide data on stdin.
Peter Amstutz [Fri, 7 Apr 2017 21:06:01 +0000 (17:06 -0400)]
8465: Set proper container flags to provide data on stdin.

7 years agoMerge branch '11237-google-api-python-client' refs #11237
Peter Amstutz [Fri, 7 Apr 2017 19:26:54 +0000 (15:26 -0400)]
Merge branch '11237-google-api-python-client' refs #11237

7 years ago11237: Tweak version dependencies.
Peter Amstutz [Fri, 7 Apr 2017 18:31:21 +0000 (14:31 -0400)]
11237: Tweak version dependencies.

7 years ago11237: Don't build oauth2client on ubuntu1604
Peter Amstutz [Fri, 7 Apr 2017 18:21:35 +0000 (14:21 -0400)]
11237: Don't build oauth2client on ubuntu1604

7 years ago11237: Bump google-api-python-client to 1.6.2 for better compatability across
Peter Amstutz [Fri, 7 Apr 2017 18:18:57 +0000 (14:18 -0400)]
11237: Bump google-api-python-client to 1.6.2 for better compatability across
oauth2client and Python versions.

7 years agoMerge branch '11209-unmount-subtype'
Tom Clegg [Fri, 7 Apr 2017 17:23:57 +0000 (13:23 -0400)]
Merge branch '11209-unmount-subtype'

refs #11209

7 years agoIncrease ping delay in WatchdogActorTest to try and reduce spurious test failures...
Peter Amstutz [Fri, 7 Apr 2017 15:43:56 +0000 (11:43 -0400)]
Increase ping delay in WatchdogActorTest to try and reduce spurious test failures.  no issue #

7 years agoAdd check that gnutls.h is available so pycurl can compile. no issue #
Peter Amstutz [Fri, 7 Apr 2017 15:36:30 +0000 (11:36 -0400)]
Add check that gnutls.h is available so pycurl can compile.  no issue #

7 years agoEnsure chown does not fail on disappearing files and dirs
Javier BĂ©rtoli [Fri, 7 Apr 2017 11:29:21 +0000 (08:29 -0300)]
Ensure chown does not fail on disappearing files and dirs

refs #11363

7 years ago11209: Restrict --unmount* operations to given --subtype.
Tom Clegg [Fri, 7 Apr 2017 07:41:53 +0000 (03:41 -0400)]
11209: Restrict --unmount* operations to given --subtype.

Add warnings about affecting other fuse mounts with --unmount flags.

7 years ago8465: added stderr redirection and tests
radhika [Fri, 7 Apr 2017 00:12:59 +0000 (20:12 -0400)]
8465: added stderr redirection and tests

7 years agoAdd missing test file refs #11423
Peter Amstutz [Thu, 6 Apr 2017 21:23:10 +0000 (17:23 -0400)]
Add missing test file refs #11423

7 years agoMerge branch '8104-pycurl-721'
Tom Clegg [Thu, 6 Apr 2017 20:47:18 +0000 (16:47 -0400)]
Merge branch '8104-pycurl-721'

closes #8104

7 years ago8104: Do not close sockets after giving them to pycurl. Reverts 7b31dd40b31bdbf7473b5...
Tom Clegg [Thu, 6 Apr 2017 20:43:36 +0000 (16:43 -0400)]
8104: Do not close sockets after giving them to pycurl. Reverts 7b31dd40b31bdbf7473b50fd407cd6f636657c69.

7 years ago8104: Drop unnecessary lambda.
Tom Clegg [Thu, 6 Apr 2017 20:20:56 +0000 (16:20 -0400)]
8104: Drop unnecessary lambda.

7 years ago8465: do not wait in the go routine while doing io.Copy of stdin and also invoke...
radhika [Thu, 6 Apr 2017 20:13:38 +0000 (16:13 -0400)]
8465: do not wait in the go routine while doing io.Copy of stdin and also invoke runner.stop on error.

7 years ago8104: Stop building pycurl package for ubuntu1604.
Tom Clegg [Thu, 6 Apr 2017 19:42:36 +0000 (15:42 -0400)]
8104: Stop building pycurl package for ubuntu1604.

7 years agoMerge branch '11423-cwl-collection-dir-ref' closes #11423
Peter Amstutz [Thu, 6 Apr 2017 19:46:18 +0000 (15:46 -0400)]
Merge branch '11423-cwl-collection-dir-ref' closes #11423

7 years ago11423: Restore check to use pdh_path because it is only supposed to match Files
Peter Amstutz [Thu, 6 Apr 2017 19:43:38 +0000 (15:43 -0400)]
11423: Restore check to use pdh_path because it is only supposed to match Files
references.

7 years agoMerge branch '7824-arvls-arvput-collection-api-usage'
Lucas Di Pentima [Thu, 6 Apr 2017 19:17:43 +0000 (16:17 -0300)]
Merge branch '7824-arvls-arvput-collection-api-usage'
Closes #7824

7 years ago7824: Merge branch 'master' into 7824-arvls-arvput-collection-api-usage
Lucas Di Pentima [Thu, 6 Apr 2017 19:15:25 +0000 (16:15 -0300)]
7824: Merge branch 'master' into 7824-arvls-arvput-collection-api-usage

7 years ago8104: Remove pycurl<7.21.5 dependency.
Tom Clegg [Thu, 6 Apr 2017 18:49:29 +0000 (14:49 -0400)]
8104: Remove pycurl<7.21.5 dependency.

7 years ago8104: OPENSOCKETFUNCTION accepts calls from pycurl 7.21.
Tom Clegg [Thu, 6 Apr 2017 18:38:44 +0000 (14:38 -0400)]
8104: OPENSOCKETFUNCTION accepts calls from pycurl 7.21.

7 years ago11308: Close socket after pycurl.
Tom Clegg [Mon, 3 Apr 2017 05:06:08 +0000 (01:06 -0400)]
11308: Close socket after pycurl.

7 years agoMerge branch 'master' into 8465-stdin-redirection
radhika [Thu, 6 Apr 2017 18:28:57 +0000 (14:28 -0400)]
Merge branch 'master' into 8465-stdin-redirection

7 years ago8465: test clean up
radhika [Thu, 6 Apr 2017 18:28:00 +0000 (14:28 -0400)]
8465: test clean up

7 years ago11423: Fix bug checking existence of Directories which refer to root of keep collections.
Peter Amstutz [Thu, 6 Apr 2017 15:58:38 +0000 (11:58 -0400)]
11423: Fix bug checking existence of Directories which refer to root of keep collections.

7 years agoMerge branch '11413-nodemanager-gce-fixes' refs #11413
Peter Amstutz [Thu, 6 Apr 2017 13:05:41 +0000 (09:05 -0400)]
Merge branch '11413-nodemanager-gce-fixes' refs #11413

7 years ago8465: added stdin redirection for json mount.
radhika [Wed, 5 Apr 2017 23:53:28 +0000 (19:53 -0400)]
8465: added stdin redirection for json mount.

7 years ago6304: Avoid possible dns update race condition by writing the temp file with unique...
Lucas Di Pentima [Wed, 5 Apr 2017 17:23:49 +0000 (14:23 -0300)]
6304: Avoid possible dns update race condition by writing the temp file with unique name to avoid collisions.
Catch IO specific exceptions when trying to write the config files.
Added test to assert that no config file is written when the update command is not set up but the config dir is.

7 years ago7824: Added test cases.
Lucas Di Pentima [Tue, 4 Apr 2017 21:08:39 +0000 (18:08 -0300)]
7824: Added test cases.

7 years ago8465: added stdin redirection for collection
radhika [Tue, 4 Apr 2017 20:48:12 +0000 (16:48 -0400)]
8465: added stdin redirection for collection

7 years ago11413: Use getattr() in exception handler.
Peter Amstutz [Tue, 4 Apr 2017 18:03:08 +0000 (14:03 -0400)]
11413: Use getattr() in exception handler.

7 years ago11413: Wrap destroy_node with similar logic to create_node: on exception check
Peter Amstutz [Tue, 4 Apr 2017 15:40:58 +0000 (11:40 -0400)]
11413: Wrap destroy_node with similar logic to create_node: on exception check
the node list to determine if the node was actually destroyed successfully.

7 years agoMerge branch '11100-cr-output-ttl'
Tom Clegg [Tue, 4 Apr 2017 15:39:30 +0000 (11:39 -0400)]
Merge branch '11100-cr-output-ttl'

refs #11100

7 years ago11413: Fix issues with node manager on GCE:
Peter Amstutz [Tue, 4 Apr 2017 15:21:57 +0000 (11:21 -0400)]
11413: Fix issues with node manager on GCE:

* Always override Node.size with CloudSizeWrapper

* Get updated node record before setting metadata to minimize 'Supplied
  fingerprint does not match current metadata fingerprint.' error.

* Use ex_set_node_metadata() instead of issuing request directly.

7 years ago7824: Now a subdirectory can be asked to be listed. Also, enhanced error message...
Lucas Di Pentima [Mon, 3 Apr 2017 21:16:08 +0000 (18:16 -0300)]
7824: Now a subdirectory can be asked to be listed. Also, enhanced error message when asking for a non existant collection.

7 years ago7824: Several improvements:
Lucas Di Pentima [Mon, 3 Apr 2017 20:45:34 +0000 (17:45 -0300)]
7824: Several improvements:
* Added error reporting on cases where the user ask for downloading a nonexisting file/subdir inside a collection.
* Enhanced error message when the user ask to download a non existing collection.
* Improved the collection traversal when downloading a subcollection.
* Exit with an error message when asking for a subcollection name without the trailing '/'.
* Error out if asking to get a subcollection with '-' as destination.
* Test updated.

7 years agoMerge branch '11407-go-sdk-debug' closes #11407
Peter Amstutz [Mon, 3 Apr 2017 20:21:28 +0000 (16:21 -0400)]
Merge branch '11407-go-sdk-debug' closes #11407

7 years ago11407: keepclient checks for ARVADOS_DEBUG and enables debugging.
Peter Amstutz [Mon, 3 Apr 2017 19:26:50 +0000 (15:26 -0400)]
11407: keepclient checks for ARVADOS_DEBUG and enables debugging.

7 years ago7824: Log errors when trying to read the collection specified.
Lucas Di Pentima [Mon, 3 Apr 2017 18:07:48 +0000 (15:07 -0300)]
7824: Log errors when trying to read the collection specified.

7 years ago7824: Closing file when finished writing
Lucas Di Pentima [Mon, 3 Apr 2017 17:52:36 +0000 (14:52 -0300)]
7824: Closing file when finished writing

7 years ago8567: Add workaround for "trap EXIT" in pid 1 bash process. Add comments.
Peter Amstutz [Mon, 3 Apr 2017 13:52:13 +0000 (09:52 -0400)]
8567: Add workaround for "trap EXIT" in pid 1 bash process.  Add comments.

7 years ago11100: Adjust delete_at in before_validation hook instead of validation. Permit chang...
Tom Clegg [Sat, 1 Apr 2017 02:07:19 +0000 (22:07 -0400)]
11100: Adjust delete_at in before_validation hook instead of validation. Permit changing trash_at without extending delete_at.

7 years ago11100: Specify units in output_ttl docs.
Tom Clegg [Fri, 31 Mar 2017 21:26:43 +0000 (17:26 -0400)]
11100: Specify units in output_ttl docs.

7 years ago11402: Disable caching if $HOME/ is not writable, instead of returning a broken cache.
Tom Clegg [Fri, 31 Mar 2017 20:17:19 +0000 (16:17 -0400)]
11402: Disable caching if $HOME/ is not writable, instead of returning a broken cache.

closes #11402

7 years agocloses #11368
radhika [Fri, 31 Mar 2017 18:04:02 +0000 (14:04 -0400)]
closes #11368
Merge branch '11368-collection-file-rename-warning'

7 years ago11368: show error if user tries to rename a collection file with a duplicate name.
radhika [Fri, 31 Mar 2017 15:39:13 +0000 (11:39 -0400)]
11368: show error if user tries to rename a collection file with a duplicate name.

7 years agoMerge branch '11398-nodes-items-available'
Tom Clegg [Fri, 31 Mar 2017 17:54:51 +0000 (13:54 -0400)]
Merge branch '11398-nodes-items-available'

refs #11398

7 years ago11398: Fix missing items_available in nodes#index response.
Tom Clegg [Fri, 31 Mar 2017 17:52:36 +0000 (13:52 -0400)]
11398: Fix missing items_available in nodes#index response.

7 years ago7824: Use logging facility to show error messages.
Lucas Di Pentima [Fri, 31 Mar 2017 16:40:23 +0000 (13:40 -0300)]
7824: Use logging facility to show error messages.

7 years agoMerge branch '9132-dockerclient' closes #9132
Peter Amstutz [Fri, 31 Mar 2017 14:41:55 +0000 (10:41 -0400)]
Merge branch '9132-dockerclient' closes #9132

7 years ago7824: Updated code to use newer Collection APIs, leaving all_streams() and all_files...
Lucas Di Pentima [Fri, 31 Mar 2017 14:24:50 +0000 (11:24 -0300)]
7824: Updated code to use newer Collection APIs, leaving all_streams() and all_files() behind.

7 years ago9132: Fix test errors due to rebase.
Peter Amstutz [Fri, 31 Mar 2017 14:17:23 +0000 (10:17 -0400)]
9132: Fix test errors due to rebase.

7 years agoFix package test for ubuntu1204.
Ward Vandewege [Fri, 31 Mar 2017 13:57:56 +0000 (09:57 -0400)]
Fix package test for ubuntu1204.

refs #10797

7 years ago11100: If caller sets collection.delete_at too early, set it to the earliest allowed...
Tom Clegg [Thu, 30 Mar 2017 22:47:12 +0000 (18:47 -0400)]
11100: If caller sets collection.delete_at too early, set it to the earliest allowed time instead of failing.

7 years ago9132: Fix ContainerCreate to include HostConfig, container UUID as name.
Peter Amstutz [Thu, 30 Mar 2017 22:02:11 +0000 (18:02 -0400)]
9132: Fix ContainerCreate to include HostConfig, container UUID as name.

7 years ago11100: Add comment about delete_at validation race.
Tom Clegg [Thu, 30 Mar 2017 21:11:05 +0000 (17:11 -0400)]
11100: Add comment about delete_at validation race.

7 years ago11100: Clean up permission checks.
Tom Clegg [Thu, 30 Mar 2017 20:56:59 +0000 (16:56 -0400)]
11100: Clean up permission checks.

7 years ago11100: Add container_requests.output_ttl field. Fix validation of output/log changes...
Tom Clegg [Thu, 30 Mar 2017 16:34:12 +0000 (12:34 -0400)]
11100: Add container_requests.output_ttl field. Fix validation of output/log changes on finalized CRs.

7 years ago8567: Log total bytes to migrate
Peter Amstutz [Thu, 30 Mar 2017 19:40:28 +0000 (19:40 +0000)]
8567: Log total bytes to migrate

7 years ago8567: Use arvados.config instead of os.environ. Tweak disk space accounting output.
Peter Amstutz [Thu, 30 Mar 2017 19:14:37 +0000 (19:14 +0000)]
8567: Use arvados.config instead of os.environ.  Tweak disk space accounting output.

7 years ago7824: Moved arv-get code to arvados.commands and replaced bin/arv-get with a stub...
Lucas Di Pentima [Thu, 30 Mar 2017 18:40:41 +0000 (15:40 -0300)]
7824: Moved arv-get code to arvados.commands and replaced bin/arv-get with a stub caller.

7 years ago9132: Fix after rebase
Peter Amstutz [Thu, 30 Mar 2017 18:19:42 +0000 (14:19 -0400)]
9132: Fix after rebase

7 years ago9132: Use ThinDockerClientProxy to overcome the import issue around "github.com/docke...
radhika [Wed, 22 Mar 2017 20:47:44 +0000 (16:47 -0400)]
9132: Use ThinDockerClientProxy to overcome the import issue around "github.com/docker/docker/vendor/golang.org/x/net/context".
Perform most of the test updates needed to use the new API.

7 years ago9132: few more updates to use the new API
Peter Amstutz [Thu, 30 Mar 2017 18:09:28 +0000 (14:09 -0400)]
9132: few more updates to use the new API

7 years ago9132: WIP switching to using official Docker Go client.
Peter Amstutz [Thu, 30 Mar 2017 18:07:01 +0000 (14:07 -0400)]
9132: WIP switching to using official Docker Go client.

7 years agoMerge branch '11332-fix-crunchscript' refs #11332
Peter Amstutz [Thu, 30 Mar 2017 18:32:06 +0000 (14:32 -0400)]
Merge branch '11332-fix-crunchscript' refs #11332

7 years ago7824: Further ordering fixes. Now all arv-ls tests passes.
Lucas Di Pentima [Thu, 30 Mar 2017 18:15:20 +0000 (15:15 -0300)]
7824: Further ordering fixes. Now all arv-ls tests passes.

7 years agoMerge branch '7709-api-rails4' (partial)
Tom Clegg [Thu, 30 Mar 2017 17:41:22 +0000 (13:41 -0400)]
Merge branch '7709-api-rails4' (partial)

refs #7709
refs #11100

7 years ago8567: Improve disk space accounting. Default to use overlay storage driver (but...
Peter Amstutz [Thu, 30 Mar 2017 14:07:52 +0000 (10:07 -0400)]
8567: Improve disk space accounting.  Default to use overlay storage driver (but user can override).

7 years ago7824: Fix upper/lower case ordering on arv-ls.
Lucas Di Pentima [Thu, 30 Mar 2017 15:06:46 +0000 (12:06 -0300)]
7824: Fix upper/lower case ordering on arv-ls.

7 years ago7709: De-duplicate "ensure unique name" implementations.
Tom Clegg [Tue, 21 Mar 2017 15:59:40 +0000 (11:59 -0400)]
7709: De-duplicate "ensure unique name" implementations.

7 years ago7709: Do not set job_readable when it won't be used.
Tom Clegg [Tue, 21 Mar 2017 16:49:06 +0000 (12:49 -0400)]
7709: Do not set job_readable when it won't be used.

7 years ago7709: Leave container_request alone when applying default keep_cache_ram value.
Tom Clegg [Tue, 21 Mar 2017 15:56:17 +0000 (11:56 -0400)]
7709: Leave container_request alone when applying default keep_cache_ram value.

7 years ago7709: Fix tests (count existing links before adding new ones).
Tom Clegg [Tue, 21 Mar 2017 16:01:33 +0000 (12:01 -0400)]
7709: Fix tests (count existing links before adding new ones).

7 years ago11209: Restore missing import.
Tom Clegg [Thu, 30 Mar 2017 14:29:47 +0000 (10:29 -0400)]
11209: Restore missing import.

refs #11209

7 years ago11332-fix-crunchscript: Set ACCEPTLIST_RE because crunch_script alternate entry
Peter Amstutz [Thu, 30 Mar 2017 14:09:33 +0000 (10:09 -0400)]
11332-fix-crunchscript: Set ACCEPTLIST_RE because crunch_script alternate entry
point doesn't go through main.

7 years ago11355: Arvados gems accept json 2.x.
Tom Clegg [Thu, 30 Mar 2017 01:23:24 +0000 (21:23 -0400)]
11355: Arvados gems accept json 2.x.

refs #11355

7 years ago10401: Bump cwltool for packing fix.
Peter Amstutz [Wed, 29 Mar 2017 15:51:45 +0000 (11:51 -0400)]
10401: Bump cwltool for packing fix.

7 years ago10401: Bump schema salad for self-colliding id fix.
Peter Amstutz [Wed, 29 Mar 2017 16:00:07 +0000 (12:00 -0400)]
10401: Bump schema salad for self-colliding id fix.

7 years ago10401: Update crunch script, no longer expands directories by default.
Peter Amstutz [Tue, 28 Mar 2017 13:57:12 +0000 (09:57 -0400)]
10401: Update crunch script, no longer expands directories by default.

7 years ago10401: Fixup file/directory literal handling
Peter Amstutz [Mon, 27 Mar 2017 20:27:32 +0000 (16:27 -0400)]
10401: Fixup file/directory literal handling

7 years ago10401: Only upload file:// identifiers.
Peter Amstutz [Mon, 27 Mar 2017 20:18:34 +0000 (16:18 -0400)]
10401: Only upload file:// identifiers.

7 years ago10401: Bump arvados-python-client version for updates to
Peter Amstutz [Mon, 27 Mar 2017 20:04:55 +0000 (16:04 -0400)]
10401: Bump arvados-python-client version for updates to
arvados.command.run.uploadfiles.

7 years ago10401: Use custom_schema_callback. Bump schema-salad and cwltool versions.
Peter Amstutz [Wed, 29 Mar 2017 15:59:35 +0000 (11:59 -0400)]
10401: Use custom_schema_callback.  Bump schema-salad and cwltool versions.

7 years ago10401: Rework support for uploading Directories (don't assume files are
Peter Amstutz [Fri, 24 Mar 2017 18:55:02 +0000 (14:55 -0400)]
10401: Rework support for uploading Directories (don't assume files are
enumerated).

7 years ago10401: Allow Arvados extensions to be used in "requirements" not just "hints".
Peter Amstutz [Wed, 8 Mar 2017 18:23:38 +0000 (13:23 -0500)]
10401: Allow Arvados extensions to be used in "requirements" not just "hints".

7 years ago10401: Use use_custom_schema feature for extensions.
Peter Amstutz [Tue, 28 Mar 2017 17:57:06 +0000 (13:57 -0400)]
10401: Use use_custom_schema feature for extensions.

7 years agoMerge branch '11235-log-retryable'
Tom Clegg [Wed, 29 Mar 2017 15:51:55 +0000 (11:51 -0400)]
Merge branch '11235-log-retryable'

refs #11235

7 years ago11235: Log a message when a job is interrupted by node failure.
Tom Clegg [Wed, 29 Mar 2017 15:49:49 +0000 (11:49 -0400)]
11235: Log a message when a job is interrupted by node failure.

...and say what's going to happen as a result, even if that is not
"giving up" yet.

7 years agocloses #3821
radhika [Wed, 29 Mar 2017 15:44:42 +0000 (11:44 -0400)]
closes #3821
Merge branch '3821-collection-file-manage'

7 years agoFix invalid records in API collections test fixture and improve error message
Peter Amstutz [Wed, 29 Mar 2017 15:43:20 +0000 (11:43 -0400)]
Fix invalid records in API collections test fixture and improve error message
to include both values.  Incidentally revealed by adding PDH index to
collections table.  refs #11361

7 years ago3821: improve the tooltip on rename file option.
radhika [Wed, 29 Mar 2017 15:28:28 +0000 (11:28 -0400)]
3821: improve the tooltip on rename file option.

7 years agoFix packaging for python-arvados-cwl-runner refs #11333
Peter Amstutz [Wed, 29 Mar 2017 14:54:20 +0000 (10:54 -0400)]
Fix packaging for python-arvados-cwl-runner refs #11333

7 years agoMerge branch '11361-pdh-index'
Tom Clegg [Wed, 29 Mar 2017 04:26:24 +0000 (00:26 -0400)]
Merge branch '11361-pdh-index'

refs #11361

7 years ago11361: Add pdh index to collections table.
Tom Clegg [Tue, 28 Mar 2017 22:01:25 +0000 (18:01 -0400)]
11361: Add pdh index to collections table.