Peter Amstutz [Tue, 14 Feb 2017 21:22:09 +0000 (16:22 -0500)]
6520: Add testcase using sinfo to set crunch_worker_state
Peter Amstutz [Tue, 14 Feb 2017 21:01:21 +0000 (16:01 -0500)]
6520: Add testcase getting wishlist from squeue.
Peter Amstutz [Mon, 6 Feb 2017 19:00:53 +0000 (14:00 -0500)]
6520: Call squeue to include slurm jobs in wishlist. Call sinfo to update
crunch_worker_state in arvados node records.
Tom Clegg [Tue, 14 Feb 2017 20:16:22 +0000 (15:16 -0500)]
11017: When compute nodes use image format v2, prefer migrated docker images.
Peter Amstutz [Tue, 14 Feb 2017 20:02:49 +0000 (15:02 -0500)]
Merge branch 'arvbox-docker-imgfmt' no issue #
Peter Amstutz [Tue, 14 Feb 2017 20:02:26 +0000 (15:02 -0500)]
Update test_with_arvbox script.
Peter Amstutz [Tue, 14 Feb 2017 19:11:20 +0000 (14:11 -0500)]
Set docker_image_formats to v1.
Peter Amstutz [Tue, 14 Feb 2017 17:09:13 +0000 (12:09 -0500)]
Merge branch 'fix-arvbox' no issue #
Peter Amstutz [Tue, 14 Feb 2017 16:05:56 +0000 (11:05 -0500)]
Fix arvbox build.
* Set up fuse groups separately.
* Separate generating superuser token from writing file so as not to write file on failure.
Tom Clegg [Tue, 14 Feb 2017 16:08:49 +0000 (11:08 -0500)]
Merge branch '10701-refactor-dispatch'
closes #10701
Tom Clegg [Tue, 14 Feb 2017 15:50:19 +0000 (10:50 -0500)]
10701: Update error text in test.
Tom Clegg [Tue, 14 Feb 2017 15:42:12 +0000 (10:42 -0500)]
10701: Improve comments and identifiers.
Tom Clegg [Mon, 13 Feb 2017 22:18:11 +0000 (17:18 -0500)]
11017: Add arv-migrate-docker19
Peter Amstutz [Mon, 13 Feb 2017 21:19:52 +0000 (16:19 -0500)]
Merge branch '9397-cwl-initialworkdir-crunchv2' closes #9397
Peter Amstutz [Mon, 13 Feb 2017 20:20:00 +0000 (15:20 -0500)]
9397: Improve container InitialWorkDir test.
Peter Amstutz [Fri, 10 Feb 2017 21:29:02 +0000 (16:29 -0500)]
9397: Fix keep URI splitting. Fix error when container has error log
collection with single file.
Peter Amstutz [Wed, 8 Feb 2017 16:11:42 +0000 (11:11 -0500)]
9397: arvados-cwl-runner implementation of InitialWorkDir on container API.
Peter Amstutz [Mon, 13 Feb 2017 19:34:15 +0000 (14:34 -0500)]
Update schema-salad pin. refs #10993
Peter Amstutz [Mon, 13 Feb 2017 18:03:47 +0000 (13:03 -0500)]
Update cwltool pin. closes #10993
Peter Amstutz [Mon, 13 Feb 2017 17:29:55 +0000 (12:29 -0500)]
Merge branch '9397-prepopulate-output-directory' refs #9397
Peter Amstutz [Mon, 13 Feb 2017 17:01:41 +0000 (12:01 -0500)]
Merge branch '10846-nodemanager-errors' closes #10846
Tom Clegg [Fri, 10 Feb 2017 23:03:10 +0000 (18:03 -0500)]
Merge branch '11088-keep-cache-units'
closes #11088
Tom Clegg [Fri, 10 Feb 2017 22:08:46 +0000 (17:08 -0500)]
Merge branch '11070-ws-listener-problem'
closes #11070
radhika [Fri, 10 Feb 2017 21:36:13 +0000 (16:36 -0500)]
closes #10903
Merge branch '10903-wb-cancel-cascade'
Tom Clegg [Fri, 10 Feb 2017 21:21:37 +0000 (16:21 -0500)]
11088: Fix keep_cache_ram units.
Lucas Di Pentima [Fri, 10 Feb 2017 21:00:22 +0000 (18:00 -0300)]
Merge branch '10956-arvput-pdh-fix'
Closes #10956
Lucas Di Pentima [Fri, 10 Feb 2017 19:18:21 +0000 (16:18 -0300)]
Merge branch '3900-trash-collections-delete-api'
Closes #3900
radhika [Fri, 10 Feb 2017 19:15:11 +0000 (14:15 -0500)]
closes #10945
Merge branch '10945-used-in-section-for-cwl-jobs'
radhika [Fri, 10 Feb 2017 19:08:50 +0000 (14:08 -0500)]
10903: support need_transaction for job and pipeline_instance cancel methods.
Peter Amstutz [Fri, 10 Feb 2017 18:51:42 +0000 (13:51 -0500)]
9397: Fix bind behavior to include Path. Update tests to check container binds
as well as output manifest.
Peter Amstutz [Fri, 10 Feb 2017 18:40:31 +0000 (13:40 -0500)]
9397: Adjust stream normalization to use just digest (no hints) to de-duplicate
block locators.
Tom Clegg [Fri, 10 Feb 2017 18:31:21 +0000 (13:31 -0500)]
10701: Get multiple pages of queue if necessary.
Tom Clegg [Fri, 10 Feb 2017 18:16:47 +0000 (13:16 -0500)]
10701: Fix chan type.
Tom Clegg [Fri, 10 Feb 2017 18:11:52 +0000 (13:11 -0500)]
10701: Add back MinRetryPeriod throttle. Update comments/identifiers.
Lucas Di Pentima [Fri, 10 Feb 2017 18:04:41 +0000 (15:04 -0300)]
3900: Merge branch 'master' into 3900-trash-collections-delete-api
Lucas Di Pentima [Fri, 10 Feb 2017 18:01:45 +0000 (15:01 -0300)]
3900: Flipped item.destroy call for semantics reasons
radhika [Fri, 10 Feb 2017 17:13:09 +0000 (12:13 -0500)]
10945: "Used in jobs" panel placement
Peter Amstutz [Fri, 10 Feb 2017 16:04:23 +0000 (11:04 -0500)]
Merge branch 'master' into origin-9397-prepopulate-output-directory
Peter Amstutz [Fri, 10 Feb 2017 16:03:42 +0000 (11:03 -0500)]
Merge branch '9397-go-manifest' refs #9397
Peter Amstutz [Fri, 10 Feb 2017 16:02:33 +0000 (11:02 -0500)]
9397: Update comment & tests for CollectionFileReader to reflect it is more
lenient in the paths it accepts as a result of updates to implementation of
manifest.FileSegmentIterByName.
Tom Clegg [Fri, 10 Feb 2017 09:56:28 +0000 (04:56 -0500)]
10701: cleanup
Tom Clegg [Fri, 10 Feb 2017 06:25:14 +0000 (01:25 -0500)]
10701: Refactor dispatch library.
radhika [Fri, 10 Feb 2017 00:06:40 +0000 (19:06 -0500)]
10945: show jobs in which this job is used in job#show page.
Peter Amstutz [Thu, 9 Feb 2017 21:58:23 +0000 (16:58 -0500)]
9397: Tweak method documentation for Extract()
Peter Amstutz [Thu, 9 Feb 2017 21:55:10 +0000 (16:55 -0500)]
9397: Update method documentation for Extract()
Lucas Di Pentima [Thu, 9 Feb 2017 21:47:01 +0000 (18:47 -0300)]
10956: Fixed cwl test mocks to match behaviour change on collection class.
Peter Amstutz [Thu, 9 Feb 2017 21:46:21 +0000 (16:46 -0500)]
9397: Fix major bug in firstBlock(). Refactor ManifestTextForPath() to
Extract(). Test returning errors from Extract().
Peter Amstutz [Thu, 9 Feb 2017 19:28:26 +0000 (14:28 -0500)]
9397: Fix collection.go to use blockdigest.LocatorPattern
Peter Amstutz [Thu, 9 Feb 2017 19:28:26 +0000 (14:28 -0500)]
9397: Fix collection.go to use blockdigest.LocatorPattern
Peter Amstutz [Thu, 9 Feb 2017 19:25:19 +0000 (14:25 -0500)]
9397: Use ManifestTextForPath
Peter Amstutz [Thu, 9 Feb 2017 19:11:53 +0000 (14:11 -0500)]
Merge branch '9397-go-manifest' into origin-9397-prepopulate-output-directory
Peter Amstutz [Thu, 9 Feb 2017 19:11:35 +0000 (14:11 -0500)]
9397: Remove conflicting code for easier merge
Peter Amstutz [Thu, 9 Feb 2017 18:46:14 +0000 (13:46 -0500)]
10846: Specify whether to try to resume the node when cancelling shutdown
Peter Amstutz [Thu, 9 Feb 2017 18:25:40 +0000 (13:25 -0500)]
9397: Use idomatic loop form.
Peter Amstutz [Thu, 9 Feb 2017 18:21:28 +0000 (13:21 -0500)]
9397: Move most functions into private API.
* Document ManifestTextForPath().
* Check for invalid file segments, and fix tests that were using invalid file segments(!)
Tom Clegg [Thu, 9 Feb 2017 17:24:20 +0000 (12:24 -0500)]
10701: Tidy up error handling.
Tom Clegg [Thu, 9 Feb 2017 17:15:30 +0000 (12:15 -0500)]
10701: Fix edge case: scheduling_parameters.partitions=[]
Tom Clegg [Thu, 9 Feb 2017 17:11:38 +0000 (12:11 -0500)]
11070: Fix startup race: clients can safely connect before/during pq setup phase.
Tom Clegg [Thu, 9 Feb 2017 09:04:57 +0000 (04:04 -0500)]
11070: Add some logs, ensure shutdown after a pq connection problem.
Tom Clegg [Thu, 9 Feb 2017 08:35:59 +0000 (03:35 -0500)]
Merge branch '10700-dispatch'
refs #10700
Tom Clegg [Thu, 9 Feb 2017 07:28:47 +0000 (02:28 -0500)]
10700: Fix error message.
radhika [Thu, 9 Feb 2017 00:16:57 +0000 (19:16 -0500)]
10903: use cancel with cascade to cancel jobs and pause pipelines.
Lucas Di Pentima [Wed, 8 Feb 2017 23:03:32 +0000 (20:03 -0300)]
10956: When asked for the recently uploaded collection's pdh, arv-put will compute a pdh from the local collection's manifest and compare it with the API server provided version. If they differ, it will log a warning, always returning the API server's version.
Lucas Di Pentima [Wed, 8 Feb 2017 22:37:45 +0000 (19:37 -0300)]
10956: Get PDH from API server's response when saving a collection so that it doesn't have to be calculated when being asked for later on.
Updated tests to reflect this change.
Lucas Di Pentima [Wed, 8 Feb 2017 18:59:18 +0000 (15:59 -0300)]
3900: Reverted changes so that only Collection deletion from a project get deleted via an delete API call.
Lucas Di Pentima [Wed, 8 Feb 2017 16:18:36 +0000 (13:18 -0300)]
3900: When deleting items inside a project, use the delete API. In the special case of trashing collections, first remove them from its parent project.
Peter Amstutz [Mon, 6 Feb 2017 22:16:01 +0000 (17:16 -0500)]
9397: Add manifest normalization and sub-manifest extraction by path.
Introduces "SegmentedManifest" which stores streams -> files -> file segments.
Enables reexport of manifest in normalized form, as well as extraction of
individual files, streams or sets of streams. Also adds binary search for
efficiently determining first block to access for some stream offset.
radhika [Tue, 7 Feb 2017 22:51:13 +0000 (17:51 -0500)]
10903: Add cancel method with cascade to pipeline_instance and update a job's cancel method to accept a cascade parameter.
radhika [Mon, 6 Feb 2017 21:56:38 +0000 (16:56 -0500)]
9397: add pre-population of output directory documentation
radhika [Mon, 6 Feb 2017 03:41:34 +0000 (22:41 -0500)]
Merge branch 'master' into 9397-prepopulate-output-directory
radhika [Mon, 6 Feb 2017 03:40:00 +0000 (22:40 -0500)]
9397: add normalized manifest test
radhika [Mon, 6 Feb 2017 02:03:11 +0000 (21:03 -0500)]
9397: add handling of denormalized manifest
radhika [Mon, 6 Feb 2017 00:14:48 +0000 (19:14 -0500)]
9397: NormalizeManifest
Lucas Di Pentima [Fri, 3 Feb 2017 22:34:49 +0000 (19:34 -0300)]
Merge branch '10968-arvput-resume-progress'
Closes #10968
Lucas Di Pentima [Fri, 3 Feb 2017 22:34:02 +0000 (19:34 -0300)]
10968: Deleted space char at the end of a comment
Tom Clegg [Fri, 3 Feb 2017 21:13:58 +0000 (16:13 -0500)]
Merge branch 'master' into 10797-ruby-2.3
Conflicts:
build/package-build-dockerfiles/Makefile
radhika [Fri, 3 Feb 2017 21:00:33 +0000 (16:00 -0500)]
9397: get manifest segment for a subdir or file using manifest from sdk.
Lucas Di Pentima [Fri, 3 Feb 2017 19:21:24 +0000 (16:21 -0300)]
10968: Added a notification when uploading at least one directory, just to let know the user that the expected bytes count can take some time when trying to upload lots of files.
Tom Clegg [Fri, 3 Feb 2017 18:15:43 +0000 (13:15 -0500)]
Merge branch '10969-check-docker-format'
closes #10969
Lucas Di Pentima [Fri, 3 Feb 2017 17:52:16 +0000 (14:52 -0300)]
10968: Changed the periodic update thread to run every 1 second while arv-put is checking which files to skip, only notifying the user via the progress indicator.
When starts uploading the rest of the files, the update thread returns to the previous behaviour, running once every minute and checkpointing to the cache.
Lucas Di Pentima [Fri, 3 Feb 2017 15:38:52 +0000 (12:38 -0300)]
Merge branch '10932-arvput-slow-resuming'
Closes #10932
Lucas Di Pentima [Fri, 3 Feb 2017 15:09:58 +0000 (12:09 -0300)]
10932: Changed _file_paths from being a list to a set so we're not going to copy it when checking for missing files on local collection on resume start.
Added comments on cache saving explaining why is better to use json.dumps() instead of copy.deepcopy().
radhika [Fri, 3 Feb 2017 00:16:25 +0000 (19:16 -0500)]
9397: Use manifest.FileSegmentForPath to get manifest segment for a file path. Cache collections to avoid fetching
the same collection repeatedly. If no manifest segment found for a mounted path, log that fact.
Peter Amstutz [Thu, 2 Feb 2017 14:41:30 +0000 (09:41 -0500)]
10846: Revise shutdown behavior.
* Remove node blacklisting. Because of arvados node record reuse and assigning
compute node ids based on record uuid, it was possible to boot a new node
with the same id as a previously blacklisted node. Previously blacklisted
'broken' nodes are now considered 'down' when determining if it is necessary
to bring up new nodes.
* Failure to destroy a node is no longer retried by the shutdown actor. A
failure cancels the shutdown. The daemon is expected to schedule a new
shutdown attempt.
* Restored the concept of "cancellable" shutdown, which checks if the node is
still shutdown eligible before actually making the call to shut it down.
* Adjusted mocking behavior to fix tests which were producing suppressed
errors (visible in debug logging but not failing the test) when node sizes
were inconsistent between the wishlist, cloud_node objects, and
ServerCalculator.
Tom Clegg [Thu, 2 Feb 2017 22:32:31 +0000 (17:32 -0500)]
10969: Add docker_image_formats server config, and corresponding check in `arv keep docker`.
Lucas Di Pentima [Thu, 2 Feb 2017 22:10:45 +0000 (19:10 -0300)]
10932: Replaced the use of a list with a set to check if files on local collection are on the local file list, so that the resume start time is greatly reduced.
Also, the save_state method was taking too much time on two operations: deepcopy() and json.dump(). Replaced both with just one call to json.dumps() that's a lot faster than json.dump().
This will improve overall performance on big file collections uploads.
Tom Clegg [Wed, 1 Feb 2017 21:31:06 +0000 (16:31 -0500)]
Merge branch '10990-keep-web-ranges'
closes #10990
Tom Clegg [Wed, 1 Feb 2017 21:29:16 +0000 (16:29 -0500)]
10990: Add Seek tests.
Peter Amstutz [Wed, 1 Feb 2017 17:40:12 +0000 (12:40 -0500)]
10846: Remove duplicate log message suppression by ComputeNodeMonitorActor
radhika [Wed, 1 Feb 2017 16:50:42 +0000 (11:50 -0500)]
9397: Add ExcludeFromOutput to Mount struct.
Tom Clegg [Wed, 1 Feb 2017 16:46:26 +0000 (11:46 -0500)]
Merge branch '10705-dedup-log-message'
refs #10705
Tom Clegg [Wed, 1 Feb 2017 16:38:29 +0000 (11:38 -0500)]
10705: Fix double-logged error.
Tom Clegg [Tue, 31 Jan 2017 21:32:05 +0000 (16:32 -0500)]
Merge branch '10700-dispatch'
closes #10700
refs #10701
closes #10702
closes #10703
closes #10704
Conflicts:
services/crunch-dispatch-slurm/crunch-dispatch-slurm.go
Lucas Di Pentima [Tue, 31 Jan 2017 17:27:03 +0000 (14:27 -0300)]
Merge branch '10694-provenance-graph-consistency'
Closes #10694
Tom Clegg [Tue, 31 Jan 2017 15:52:29 +0000 (10:52 -0500)]
Merge branch '10868-ip-addr-conflict'
refs #10868
Lucas Di Pentima [Tue, 31 Jan 2017 15:13:33 +0000 (12:13 -0300)]
Merge branch '10813-arv-put-six-threads'
Closes #10813
radhika [Tue, 31 Jan 2017 14:39:43 +0000 (09:39 -0500)]
9397: update comments
Tom Clegg [Tue, 31 Jan 2017 00:19:49 +0000 (19:19 -0500)]
10700: Rename squeue identifiers (sqCheck = SqueueChecker{})
Tom Clegg [Mon, 30 Jan 2017 23:32:49 +0000 (18:32 -0500)]
10700: Simplify squeue checker.
Lucas Di Pentima [Mon, 30 Jan 2017 21:29:10 +0000 (18:29 -0300)]
10813: Merge branch 'master' into 10813-arv-put-six-threads