Lucas Di Pentima [Wed, 12 Oct 2016 20:56:42 +0000 (17:56 -0300)]
9463: Commit old arv-put command back, as the new one still have some performance issues when dealing with a lot of files.
Ward Vandewege [Wed, 12 Oct 2016 15:50:37 +0000 (11:50 -0400)]
Do not try to chown_R the gid of .ssh, because we're making an
assumption about the existance of a group with the same name as the
user, and it's irrelevant anyway because the permissions on .ssh are
0700.
No issue #
radhika [Wed, 12 Oct 2016 15:23:37 +0000 (11:23 -0400)]
refs #10144
Merge branch '10144-pipeline-display-perf'
radhika [Wed, 12 Oct 2016 00:27:28 +0000 (20:27 -0400)]
10144: call object_for_dataclass only when attr value is not-nil.
radhika [Tue, 11 Oct 2016 23:07:38 +0000 (19:07 -0400)]
10144: limit find_object_by_uuid update to the case where find is used; this will ensure that the object is not fetch again when show is used.
radhika [Tue, 11 Oct 2016 20:56:05 +0000 (16:56 -0400)]
10144: Conver the link_to_log tests into controller tests so that the #show is executed.
radhika [Tue, 11 Oct 2016 18:22:30 +0000 (14:22 -0400)]
10144: Preload children during (main) component display. The Job.where in pipeline_instance_work_unit is fetching all jobs even if they are not readable.
radhika [Tue, 11 Oct 2016 14:46:45 +0000 (10:46 -0400)]
10144: Rename :by_name as "by_attr" for preload_objects_for_dataclass
radhika [Fri, 7 Oct 2016 19:46:30 +0000 (15:46 -0400)]
10144: Preloading all the collection using one single large query is much slower than making several requests
with a smaller subset of uuids/pdhs when displaying a pipeline instance with many jobs. Hence, make several
requests for 200 collections at a time.
radhika [Thu, 6 Oct 2016 21:45:22 +0000 (17:45 -0400)]
10144: preload work_unit children
radhika [Thu, 6 Oct 2016 17:30:36 +0000 (13:30 -0400)]
10144: preload repositories during work unit component display.
radhika [Thu, 6 Oct 2016 17:12:05 +0000 (13:12 -0400)]
10144: Update link_to_if_arvados_object to use collection_for_pdh when resource_class is a Collection and attrvalue is a pdh.
radhika [Thu, 6 Oct 2016 17:09:17 +0000 (13:09 -0400)]
10144: Replace work_unit -> link_to_log with partial so that preloaded objects are used.
Nico Cesar [Wed, 12 Oct 2016 14:06:34 +0000 (10:06 -0400)]
Merge branch '10228-arvados-git-httpd-service'
closes #10228
Nico Cesar [Wed, 12 Oct 2016 10:57:41 +0000 (06:57 -0400)]
package has a .service missing
refs #10228
Lucas Di Pentima [Tue, 11 Oct 2016 23:18:16 +0000 (20:18 -0300)]
Merge branch '9701-collection-pack-small-files-alt'
Closes #9701 #9463
Lucas Di Pentima [Tue, 11 Oct 2016 23:13:06 +0000 (20:13 -0300)]
9701: Replaced deprecated use of File.exists?() with File.exist()
Lucas Di Pentima [Tue, 11 Oct 2016 23:06:37 +0000 (20:06 -0300)]
9701: Fixed arv-put to pass a couple of pending sdk/cli tests. refs #9463
Lucas Di Pentima [Tue, 11 Oct 2016 17:54:14 +0000 (14:54 -0300)]
9701: Merge branch 'master' into 9701-collection-pack-small-files-alt
Lucas Di Pentima [Tue, 11 Oct 2016 17:52:22 +0000 (14:52 -0300)]
9701: Wrapping actual alloc_bufferblock and delete_bufferblock methods inside synchronized calls, to be able to use them from within already synchronized BlockManager methods.
Peter Amstutz [Tue, 11 Oct 2016 17:30:12 +0000 (13:30 -0400)]
Merge branch '10186-crunch2-slurm-partition' closes #10186
radhika [Tue, 11 Oct 2016 14:10:13 +0000 (10:10 -0400)]
refs #10078
Merge branch '10078-dashboard-perf'
radhika [Tue, 11 Oct 2016 14:08:45 +0000 (10:08 -0400)]
10078: Rename the workbench config param 'show_user_notifications_on_dashboard' as 'show_user_notifications' because they are in topnav, not on dashboard!
radhika [Tue, 11 Oct 2016 14:04:16 +0000 (10:04 -0400)]
Merge branch 'master' into 10078-dashboard-perf
Lucas Di Pentima [Mon, 10 Oct 2016 11:45:31 +0000 (08:45 -0300)]
9701: Added clarifying comments to the small block searching list comprehension.
Lucas Di Pentima [Mon, 10 Oct 2016 11:40:17 +0000 (08:40 -0300)]
9701: Better bufferblock id generation.
Lucas Di Pentima [Mon, 10 Oct 2016 11:33:40 +0000 (08:33 -0300)]
9701: Superfluous variable eliminated
Lucas Di Pentima [Mon, 10 Oct 2016 11:18:05 +0000 (08:18 -0300)]
9701: Use a collection.OrderedDict instead of a simple dict to hold bufferblocks so that the packed files order is consistent. Updated related test.
Ward Vandewege [Sun, 9 Oct 2016 01:21:16 +0000 (21:21 -0400)]
Add the new KeepServiceURIs option in the crunch-dispatch-slurm
configuration file to the documentation.
Update the documentation for the crunch-dispatch-slurm configuration
file to have complete examples and use native YAML syntax.
refs #10041
Ward Vandewege [Sun, 9 Oct 2016 00:38:01 +0000 (20:38 -0400)]
Merge branch '10041-test-arvados-keep-services'
refs #10041
Tom Clegg [Sat, 8 Oct 2016 22:25:54 +0000 (18:25 -0400)]
10041: Set ARVADOS_KEEP_SERVICES from Client.KeepServicesURIs field.
Ward Vandewege [Sat, 8 Oct 2016 20:19:06 +0000 (16:19 -0400)]
Make sure that we can set ARVADOS_KEEP_SERVICES via the
crunch-dispatch-slurm configuration file.
refs #10041 #9550
Ward Vandewege [Sat, 8 Oct 2016 15:24:35 +0000 (11:24 -0400)]
Small naming fix for the config variables.
refs #10078
Ward Vandewege [Sat, 8 Oct 2016 15:24:35 +0000 (11:24 -0400)]
Small naming fix for the config variables.
refs #10078
Lucas Di Pentima [Fri, 7 Oct 2016 20:59:16 +0000 (17:59 -0300)]
9701: Simplifying small bufferblock query
Lucas Di Pentima [Fri, 7 Oct 2016 20:27:41 +0000 (17:27 -0300)]
9701: Set repack_small_blocks() method as @synchronized
Peter Amstutz [Fri, 7 Oct 2016 18:57:05 +0000 (14:57 -0400)]
10186: "partition" container runtime constraint takes array of strings instead
of single name.
radhika [Fri, 7 Oct 2016 18:34:46 +0000 (14:34 -0400)]
refs #10078
Merge branch '10078-dashboard-perf'
radhika [Fri, 7 Oct 2016 17:40:34 +0000 (13:40 -0400)]
10078: Since we are not fetching components for pipeline instances (at least until the performance issues are resolved),
we will no longer see the running and queued jobs listed within each pipeline instance row in dashboard. Hence, update
test to look for these for a container request row instead of for a pipeline instance row.
radhika [Fri, 7 Oct 2016 14:19:48 +0000 (10:19 -0400)]
10078: When fetching recent_processes, do not select 'components' which is not needed in this context. For very large pipelines, this will perform better.
radhika [Mon, 3 Oct 2016 22:58:59 +0000 (18:58 -0400)]
10078: Now that the recent_collections panel includes collections from Home directory as well
(self and other users' for admin user), suppress displaying confusing '(none)' in place of a Home directory.
radhika [Mon, 3 Oct 2016 21:07:43 +0000 (17:07 -0400)]
10078: add configuration parameter to suppress display of user notifications to improve dashboard performance
radhika [Mon, 3 Oct 2016 18:29:00 +0000 (14:29 -0400)]
10078: config parameter to hide / show recent_collections panel in dashboard.
radhika [Sat, 1 Oct 2016 02:54:09 +0000 (22:54 -0400)]
10078: preload collections and links during dashboard display
radhika [Fri, 30 Sep 2016 14:40:41 +0000 (10:40 -0400)]
10078: Update recent_collections so that the collection query it is not executed twice, once in this method and once in show_dashboard for r[:collections].each statement.
radhika [Fri, 30 Sep 2016 14:36:07 +0000 (10:36 -0400)]
10078: Update the Node.filter so that this query is not executed three times: once for nodes.sort_by and twice in compute_node_summary for nodes.select
radhika [Fri, 7 Oct 2016 18:32:32 +0000 (14:32 -0400)]
10078: fix failing test due to new fixture addition
Peter Amstutz [Fri, 7 Oct 2016 18:32:15 +0000 (14:32 -0400)]
Merge branch '8018-container-retry' closes #8018
Peter Amstutz [Fri, 7 Oct 2016 18:30:30 +0000 (14:30 -0400)]
Bump cwltool version for bugfix https://github.com/common-workflow-language/cwltool/pull/206
refs #10203
radhika [Fri, 7 Oct 2016 17:40:34 +0000 (13:40 -0400)]
10078: Since we are not fetching components for pipeline instances (at least until the performance issues are resolved),
we will no longer see the running and queued jobs listed within each pipeline instance row in dashboard. Hence, update
test to look for these for a container request row instead of for a pipeline instance row.
radhika [Fri, 7 Oct 2016 14:19:48 +0000 (10:19 -0400)]
10078: When fetching recent_processes, do not select 'components' which is not needed in this context. For very large pipelines, this will perform better.
Tom Morris [Thu, 6 Oct 2016 21:45:58 +0000 (17:45 -0400)]
Merge branch '10077-workbench-integration-tests' closes #10077
Tom Morris [Thu, 6 Oct 2016 21:43:56 +0000 (17:43 -0400)]
Add explanatory comment about FIXME
Peter Amstutz [Thu, 6 Oct 2016 20:50:16 +0000 (16:50 -0400)]
Enumerate Directory contents in output when --compute-checksum is
provided (needed to pass CWL conformance tests). No issue #
Lucas Di Pentima [Thu, 6 Oct 2016 20:32:35 +0000 (17:32 -0300)]
9701: Fixed a previous test to match new flush() behaviour. Added a new one to check
for small blocks packing.
Lucas Di Pentima [Thu, 6 Oct 2016 20:23:09 +0000 (17:23 -0300)]
9701: Several corrections/enhancements:
* Added a 'sync' parameter on repack_small_blocks() so that blocks commits can
be done in either way depending from where it's called.
* Allow packing small buffer blocks up to a full block on Keep.
* Replaced ArvadosFile's _closed flag with a list of its ArvadosFileWriter objects,
so that it's used as a reference counter.
* Moved ArvadosFile flush behaviour from ArvadosFileWriter.close() method to
ArvadosFile.remove_writer() so that it can decide whether it should commit the
buffer block or repack it with others.
Lucas Di Pentima [Thu, 6 Oct 2016 20:21:49 +0000 (17:21 -0300)]
9701: Reverted the use of the optional 'flush' argument on ArvadosFileWriter.close()
Peter Amstutz [Thu, 6 Oct 2016 18:53:57 +0000 (14:53 -0400)]
10186: Add PartitionRequirement
Peter Amstutz [Thu, 6 Oct 2016 18:38:30 +0000 (14:38 -0400)]
10186: Add support for "partition" in runtime_constraints. Add --all to squeue to ensure that it sees all paritions.
Peter Amstutz [Thu, 6 Oct 2016 16:54:28 +0000 (12:54 -0400)]
8018: Additional assertions checking when container_uuid changes/does not change.
Peter Amstutz [Thu, 6 Oct 2016 15:36:27 +0000 (11:36 -0400)]
Merge branch '9307-cwl-use-tmp-output' closes #9307 closes #9308
Peter Amstutz [Thu, 6 Oct 2016 15:34:34 +0000 (11:34 -0400)]
Merge branch 'master' into 9307-cwl-use-tmp-output
Conflicts:
sdk/cwl/tests/test_job.py
Peter Amstutz [Thu, 6 Oct 2016 15:25:53 +0000 (11:25 -0400)]
9307: Improve crunchrunner error handling. Fix typo.
Peter Amstutz [Wed, 5 Oct 2016 19:55:35 +0000 (15:55 -0400)]
Bump schema-salad & cwltool version for cwl document validation bugfix. no issue #
Lucas Di Pentima [Wed, 5 Oct 2016 18:30:33 +0000 (15:30 -0300)]
9701: Test fixes
Lucas Di Pentima [Wed, 5 Oct 2016 18:20:55 +0000 (15:20 -0300)]
9701: Changes on the Python SDK to allow small file packing on Collection class:
* Added optional flush parameter to ArvadosFileWriter.close().
* Added _closed attribute & related accessors to ArvadosFile to enable BlockManager to query this state.
* For every ArvadosFile close() operation, call BlockManager to search for small blocks for repacking purposes.
* Do a last repacking operation just before exiting, joining the last small blocks even if they don't meet the
minimum size requirement.
There's some pending code cleanup, as BlockManager fiddles directly with ArvadosFile._segments lists, some locking:
issues have to be solved.
Peter Amstutz [Wed, 5 Oct 2016 18:08:52 +0000 (14:08 -0400)]
9307: Bump tests to use v1.0 and make sure that arvados_cwl.add_arv_hints() is called.
Peter Amstutz [Wed, 5 Oct 2016 15:08:40 +0000 (11:08 -0400)]
9308: Set task.keepTmpOutput: true when outputDirType is keep_output_dir.
Peter Amstutz [Wed, 5 Oct 2016 17:52:23 +0000 (13:52 -0400)]
9308: Add test for KeepTmpOutput flag.
Peter Amstutz [Wed, 5 Oct 2016 17:16:54 +0000 (13:16 -0400)]
8018: Add migration script for "container_count" field.
Peter Amstutz [Wed, 5 Oct 2016 15:09:37 +0000 (11:09 -0400)]
Merge branch '10165-cwl-outputs' closes #10165
Peter Amstutz [Tue, 4 Oct 2016 22:30:06 +0000 (18:30 -0400)]
9308: Add 'keepTmpOutput' option to crunchrunner
Peter Amstutz [Tue, 4 Oct 2016 20:24:46 +0000 (16:24 -0400)]
10165: Add --output-name option to set name of output collection.
Peter Amstutz [Tue, 4 Oct 2016 19:56:55 +0000 (15:56 -0400)]
8018: Fix state transition checks. Add test of retry and fix other tests to
not retry.
Peter Amstutz [Tue, 4 Oct 2016 15:06:11 +0000 (11:06 -0400)]
Merge branch '9001-summary-skip-malformed-lines' closes #9001
radhika [Mon, 3 Oct 2016 22:58:59 +0000 (18:58 -0400)]
10078: Now that the recent_collections panel includes collections from Home directory as well
(self and other users' for admin user), suppress displaying confusing '(none)' in place of a Home directory.
radhika [Mon, 3 Oct 2016 21:07:43 +0000 (17:07 -0400)]
10078: add configuration parameter to suppress display of user notifications to improve dashboard performance
Lucas Di Pentima [Mon, 3 Oct 2016 19:44:31 +0000 (16:44 -0300)]
9701: Merge branch '9463-change-arvput-use-collection-class' into 9701-collection-pack-small-files-alt
Peter Amstutz [Mon, 3 Oct 2016 18:48:36 +0000 (14:48 -0400)]
10165: Add test for make_output_collection
radhika [Mon, 3 Oct 2016 18:29:00 +0000 (14:29 -0400)]
10078: config parameter to hide / show recent_collections panel in dashboard.
Peter Amstutz [Mon, 3 Oct 2016 16:59:51 +0000 (12:59 -0400)]
10165: Pass around keep client object instead of creating new client per
Collection.
Peter Amstutz [Fri, 30 Sep 2016 21:35:52 +0000 (17:35 -0400)]
10165: Handle copying complete collection contents into subdirectory.
radhika [Sat, 1 Oct 2016 02:54:09 +0000 (22:54 -0400)]
10078: preload collections and links during dashboard display
Peter Amstutz [Fri, 30 Sep 2016 21:00:34 +0000 (17:00 -0400)]
10165: DRY key cleanup.
Peter Amstutz [Fri, 30 Sep 2016 19:59:51 +0000 (15:59 -0400)]
10165: Add FinalOutputPathMapper. Delete basename/size/listing from output object because that information is captured by the enclosing Collection. Sort keys.
Peter Amstutz [Fri, 30 Sep 2016 18:59:32 +0000 (14:59 -0400)]
Merge branch '9857-cwl-acceptlist-re' refs #9857
Peter Amstutz [Fri, 30 Sep 2016 18:16:32 +0000 (14:16 -0400)]
10165: Always make an output collection when a workflow completes.
Peter Amstutz [Fri, 30 Sep 2016 17:21:56 +0000 (13:21 -0400)]
Merge branch '10032-cwl-spinup' refs #10032
radhika [Fri, 30 Sep 2016 17:16:31 +0000 (13:16 -0400)]
10078: Restore master version
radhika [Fri, 30 Sep 2016 17:12:07 +0000 (13:12 -0400)]
Merge branch 'master' into 10078-dashboard-perf
Peter Amstutz [Fri, 30 Sep 2016 15:58:49 +0000 (11:58 -0400)]
10032: Fix test to include --preserve-entire-environment
Peter Amstutz [Fri, 30 Sep 2016 15:24:53 +0000 (11:24 -0400)]
10032: Update schema salad and cwltool packages to get document validation optimizations.
Peter Amstutz [Fri, 30 Sep 2016 15:24:15 +0000 (11:24 -0400)]
Drill down on metrics on job submission.
radhika [Fri, 30 Sep 2016 14:40:41 +0000 (10:40 -0400)]
10078: Update recent_collections so that the collection query it is not executed twice, once in this method and once in show_dashboard for r[:collections].each statement.
radhika [Fri, 30 Sep 2016 14:36:07 +0000 (10:36 -0400)]
10078: Update the Node.filter so that this query is not executed three times: once for nodes.sort_by and twice in compute_node_summary for nodes.select
Tom Morris [Thu, 29 Sep 2016 16:33:49 +0000 (12:33 -0400)]
Fix negative Capybara assertion/matcher
Tom Morris [Thu, 29 Sep 2016 16:33:40 +0000 (12:33 -0400)]
Fix not present test for Capybara
Tom Morris [Thu, 29 Sep 2016 16:33:27 +0000 (12:33 -0400)]
Fix long wait for Capybara test
Tom Morris [Thu, 29 Sep 2016 16:33:01 +0000 (12:33 -0400)]
Fix row selection logic to not wait
Tom Morris [Thu, 29 Sep 2016 16:33:16 +0000 (12:33 -0400)]
Restore Capybara.default_max_wait_time side effect