Ward Vandewege [Wed, 16 Jul 2014 14:23:52 +0000 (10:23 -0400)]
Whitespace cleanup.
refs #3219
Ward Vandewege [Wed, 16 Jul 2014 14:20:50 +0000 (10:20 -0400)]
Further changes to make pipeline running work reliably.
refs #3219
Ward Vandewege [Tue, 15 Jul 2014 19:28:16 +0000 (15:28 -0400)]
Merge branch 'master' into 3219-further-docker-improvements
Ward Vandewege [Tue, 15 Jul 2014 19:27:04 +0000 (15:27 -0400)]
Add local git repository support.
refs #3219
Brett Smith [Tue, 15 Jul 2014 17:02:03 +0000 (13:02 -0400)]
3195: Add job reuse test.
I wrote this while debugging a user-reported issue. Ultimately the
issue ended up being a bug in the database, but no point letting a
good test go to waste.
Refs #3195.
Brett Smith [Mon, 14 Jul 2014 21:11:16 +0000 (17:11 -0400)]
Merge branch '3195-bugfix-job-min-script-ver'
Closes #3195. There may be other issues here but we need more
information to debug them.
Brett Smith [Tue, 8 Jul 2014 17:01:44 +0000 (13:01 -0400)]
Improve API server's git rev-list calls.
This doesn't completely silence errors, but it makes them less verbose
and prevents git from trying to look up paths, which we never want.
Brett Smith [Tue, 8 Jul 2014 13:49:16 +0000 (09:49 -0400)]
3195: Improve error handling in Job creation API.
* Express the default filters using high-level operators, then convert
them to SQL filters using the same conversion logic users get.
* Raise an ArgumentError when a git version search fails.
* Render normal error pages from ArgumentErrors.
Brett Smith [Mon, 14 Jul 2014 19:10:44 +0000 (15:10 -0400)]
Merge branch '2044-api-users-index-wip'
Refs #2044. Closes #3217.
Brett Smith [Tue, 1 Jul 2014 17:45:49 +0000 (13:45 -0400)]
2044: Improve API test fixtures around projects.
This change adjusts the API server test fixtures to look more like
real data would: items in projects have the project's UUID as their
owner_uuid.
Brett Smith [Mon, 7 Jul 2014 15:01:51 +0000 (11:01 -0400)]
2044: API filters can search on boolean columns.
Brett Smith [Mon, 7 Jul 2014 15:01:30 +0000 (11:01 -0400)]
2044: Non-admins get all users' basic info from index API.
This will support a Workbench feature to let users at the same site
share projects with each other. Adding kind to index requests with
select parameters is necessary to help Workbench understand the
result.
There was extensive discussion on IRC about whether or not e-mail
addresses should be included in this information. We toyed with ideas
like providing an e-mail address checksum, so that you could find
exact but not partial matches. Ultimately I decided that none of
those measures were worth the hassle, because the domain of addresses
at any given site would be small enough that they would easily be
discoverable through brute force.
Ward Vandewege [Mon, 14 Jul 2014 18:30:53 +0000 (14:30 -0400)]
Compute nodes can now run jobs. Also, small fixes all over the place.
refs #3219
Brett Smith [Mon, 14 Jul 2014 17:44:16 +0000 (13:44 -0400)]
3113: Make job task test more robust.
Per Tom's feedback in refs #3113.
I should've made this change before I merged; apologies.
Brett Smith [Mon, 14 Jul 2014 17:33:23 +0000 (13:33 -0400)]
Merge branch '3113-qsequence-serial'
Refs #3113. This is a useful bugfix, but we need to test whether
this fully resolves the issue.
Brett Smith [Wed, 9 Jul 2014 19:04:21 +0000 (15:04 -0400)]
3113: API server assigns task qsequence from a PostgreSQL sequence.
We always wanted monotonically increasing qsequence values. This
implements that.
Because sequences are specific to PostgreSQL, it can't be expressed in
schema.rb. This commit switches our Rails configuration to dump the
database schema in SQL to capture that.
PostgreSQL supports assigning a column's default value from a
sequence, but this doesn't play well with ActiveRecord. Without more
information, it always tries to assign NULL as the column value,
rather than leaving it unassigned to get the default. We have to get
the value from inside the model.
Tom Clegg [Wed, 9 Jul 2014 16:58:31 +0000 (12:58 -0400)]
3113: Remove qsequence assignment shim. Clarify "do not load tasks
queued by a failed task" logic in crunch-job.
Ward Vandewege [Mon, 14 Jul 2014 04:05:24 +0000 (00:05 -0400)]
Compute nodes can run jobs - almost. The final arv-put --raw in the
collation step still fails (file descriptor error?).
refs #3219
Ward Vandewege [Sun, 13 Jul 2014 18:10:04 +0000 (14:10 -0400)]
The --pretty option was removed from arv some time ago. Do not refer to
it in under the 'Advanced' tab in workbench.
no issue #
Ward Vandewege [Sun, 13 Jul 2014 15:04:44 +0000 (11:04 -0400)]
Two compute nodes work at the slurm level. Crunch-dispatch runs.
refs #3219
Ward Vandewege [Thu, 10 Jul 2014 20:34:12 +0000 (16:34 -0400)]
First set of changes to add crunch-dispatch and compute nodes. This is not complete yet.
refs #3219
Tom Clegg [Thu, 10 Jul 2014 19:07:08 +0000 (15:07 -0400)]
Fix broken links to SSH key help page. no issue #
Ward Vandewege [Thu, 10 Jul 2014 16:42:43 +0000 (12:42 -0400)]
Merge branch '3219-further-docker-improvements'
refs #3219
Ward Vandewege [Thu, 10 Jul 2014 16:38:00 +0000 (12:38 -0400)]
Update the documentation to reflect the new, safer build.sh --realclean
behavior.
refs #3219
Ward Vandewege [Thu, 10 Jul 2014 16:20:43 +0000 (12:20 -0400)]
Be smarter about building and the --realclean option:
* remove intermediate containers after a successful build
* --realclean removes only images labelled arvados/*
* --realclean removes only arvados containers
refs #3219
Ward Vandewege [Thu, 10 Jul 2014 15:17:08 +0000 (11:17 -0400)]
Sanitize how we handle the --ssh option in the api docker container.
refs #3219
Ward Vandewege [Thu, 10 Jul 2014 15:12:42 +0000 (11:12 -0400)]
Merge branch '3219-further-docker-improvements'
refs #3219
Ward Vandewege [Thu, 10 Jul 2014 15:07:56 +0000 (11:07 -0400)]
Be consistent about the version of ruby that is installed by rvm and
used by the various containers. Be consistent about using rvm-exec and
bundle exec in every command that uses ruby.
Add documentation and warning about --realclean option to build.sh
refs #3219
Ward Vandewege [Thu, 10 Jul 2014 14:35:04 +0000 (10:35 -0400)]
Whitespace fix.
refs #3219
Ward Vandewege [Wed, 9 Jul 2014 18:41:18 +0000 (14:41 -0400)]
Another tweak to tutorial-job-debug.html.textile.liquid
no issue #
Ward Vandewege [Wed, 9 Jul 2014 18:33:52 +0000 (14:33 -0400)]
Be more explicit that hello-world.py is a script which is expected to fail.
no issue #
Ward Vandewege [Wed, 9 Jul 2014 17:30:29 +0000 (13:30 -0400)]
Improve arv-run-pipeline-instance's logging when it encounters an error.
no issue #
Brett Smith [Wed, 9 Jul 2014 15:18:26 +0000 (11:18 -0400)]
Give API server arv-run-pipeline-instance bugfix.
This commit ensures that crunch-dispatch gets the bugfix in
b5b0f8c9.
No issue #.
Brett Smith [Wed, 9 Jul 2014 01:32:40 +0000 (21:32 -0400)]
3201: Simplify logic to decide "My Projects" is selectable.
Thinking about it on the commute home, I decided that my previous
implementation that inspected action_href (
e007ef71) was too clever by
half. Following the adage that explicit is better than implicit, this
version provides a knob you can turn to just say whether or not "My
Projects" is selectable.
Closes #3201.
radhika [Wed, 9 Jul 2014 00:34:22 +0000 (20:34 -0400)]
closes #3203
Merge branch '3203-workbench-tutorial'
radhika [Wed, 9 Jul 2014 00:23:02 +0000 (20:23 -0400)]
Merge branch 'master' into 3203-workbench-tutorial
Brett Smith [Tue, 8 Jul 2014 22:06:45 +0000 (18:06 -0400)]
3201: Only Projects can be moved to "My Projects."
Before this commit, it was possible to move any object that goes in
folders to "My Projects." If you did, it would be removed from any
project, and owned by the user directly. This is non-obvious. Data
should be removed from projects by using the trash icon on the project
page instead.
Closes #3201.
Brett Smith [Tue, 8 Jul 2014 20:46:29 +0000 (16:46 -0400)]
arv-run-pipeline-instance doesn't submit nil Job parameters.
This is the same principle that we already take with POST parameters,
applied to the body of the Job itself. It immediately prevents
sending nil runtime_constraints, which the API server doesn't accept.
It should also mean that any error messages passed on from the API
server are more likely to be helpful.
No issue # - reported by Abram on IRC.
radhika [Tue, 8 Jul 2014 20:34:56 +0000 (16:34 -0400)]
Merge branch 'master' into 3203-workbench-tutorial
radhika [Tue, 8 Jul 2014 20:34:07 +0000 (16:34 -0400)]
3203: fix doc issues with old dashboard references.
Brett Smith [Tue, 8 Jul 2014 20:00:08 +0000 (16:00 -0400)]
Start API pipeline integration test.
No issue #. I wrote this while debugging the issue in the previous
commit.
radhika [Tue, 8 Jul 2014 19:59:59 +0000 (15:59 -0400)]
3203: fix doc issues with compute references.
Brett Smith [Tue, 8 Jul 2014 19:59:39 +0000 (15:59 -0400)]
3209: arv-run-pipeline-instance sends instance body as JSON.
If we don't encode the body as JSON ourselves, the API client library
will omit empty attributes. This can cause required component
attributes like script_parameters to be lost, so we need to circumvent
that behavior. It also seems safe, since we already do the same
encoding in the update method. Closes #3209.
Peter Amstutz [Tue, 8 Jul 2014 18:57:14 +0000 (14:57 -0400)]
Merge branch 'resumable-output-upload'. Adds signal handling and error recovery
when uploading job output to run-command wrapper. refs #2342
Peter Amstutz [Tue, 8 Jul 2014 18:51:18 +0000 (14:51 -0400)]
Actually use the resume cache
radhika [Tue, 8 Jul 2014 18:44:04 +0000 (14:44 -0400)]
Merge branch 'master' into 3203-workbench-tutorial
radhika [Tue, 8 Jul 2014 18:42:26 +0000 (14:42 -0400)]
3203: Update the pipeline tutorial page to run pipeline from a project.
Peter Amstutz [Tue, 8 Jul 2014 18:29:29 +0000 (14:29 -0400)]
Turns out sys.exit runs finally: blocks.
Peter Amstutz [Tue, 8 Jul 2014 18:24:32 +0000 (14:24 -0400)]
Add debugging
Peter Amstutz [Tue, 8 Jul 2014 18:21:05 +0000 (14:21 -0400)]
Wrong num args
Peter Amstutz [Tue, 8 Jul 2014 18:19:45 +0000 (14:19 -0400)]
SigHandler object
Peter Amstutz [Tue, 8 Jul 2014 18:16:21 +0000 (14:16 -0400)]
Still messing with signals.
Peter Amstutz [Tue, 8 Jul 2014 18:08:28 +0000 (14:08 -0400)]
Fix wrong number of parameters
Peter Amstutz [Tue, 8 Jul 2014 18:06:15 +0000 (14:06 -0400)]
Record signals a different way
Peter Amstutz [Tue, 8 Jul 2014 18:02:26 +0000 (14:02 -0400)]
More work on signal handling
Peter Amstutz [Tue, 8 Jul 2014 17:50:58 +0000 (13:50 -0400)]
Forward signals to process.
radhika [Tue, 8 Jul 2014 17:29:55 +0000 (13:29 -0400)]
3203: Update "Accessing Arvados Workbench" page.
radhika [Tue, 8 Jul 2014 15:27:55 +0000 (11:27 -0400)]
refs #2380 Needed to updated all docs referencing to SSH documentation.
Merge branch '2380-ssh-doc'
radhika [Tue, 8 Jul 2014 15:24:12 +0000 (11:24 -0400)]
2380: ssh doc references are updated to point to the right env.
Peter Amstutz [Tue, 8 Jul 2014 15:12:40 +0000 (11:12 -0400)]
Use resumable upload writing output in run-command
radhika [Tue, 8 Jul 2014 14:15:26 +0000 (10:15 -0400)]
closes #2380
Merge branch '2380-ssh-doc'
radhika [Tue, 8 Jul 2014 14:14:44 +0000 (10:14 -0400)]
Merge branch 'master' into 2380-ssh-doc
Peter Amstutz [Tue, 8 Jul 2014 14:08:26 +0000 (10:08 -0400)]
Merge branch '3206-keepproxy-use-client-token' closes #3206
Brett Smith [Tue, 8 Jul 2014 13:55:23 +0000 (09:55 -0400)]
Merge branch '3078-arv-put-name-links'
Closes #3078, #3176, #3205.
Brett Smith [Mon, 7 Jul 2014 21:52:31 +0000 (17:52 -0400)]
3176: Fix arv-put's resume cache writing.
Refactoring directory creation into _util accidentally broke this:
arv-put was trying to save the cache to a relatively directory that
almost never existed, and then silently ignored the error.
Brett Smith [Mon, 7 Jul 2014 20:48:50 +0000 (16:48 -0400)]
3078: Add arv-put options to add the upload to a Project.
Brett Smith [Mon, 7 Jul 2014 18:29:39 +0000 (14:29 -0400)]
3078: arv-put supports dynamic stderr.
Brett Smith [Mon, 7 Jul 2014 17:10:42 +0000 (13:10 -0400)]
3078: arv-put always writes progress before starting work.
It previously did this when resuming, but there's no reason we
shouldn't do it when starting fresh as well.
radhika [Tue, 8 Jul 2014 13:40:55 +0000 (09:40 -0400)]
Merge branch 'master' into 2380-ssh-doc
radhika [Tue, 8 Jul 2014 13:40:27 +0000 (09:40 -0400)]
2380: update the add key image and text
Peter Amstutz [Tue, 8 Jul 2014 13:18:07 +0000 (09:18 -0400)]
Keepproxy use client-supplied token when forwarding GET and PUT requests.
radhika [Tue, 8 Jul 2014 13:09:02 +0000 (09:09 -0400)]
2380: more updates
radhika [Tue, 8 Jul 2014 12:51:30 +0000 (08:51 -0400)]
2380: Extract common documentation items from Unix and Windows environments into include files.
Peter Amstutz [Tue, 8 Jul 2014 12:47:59 +0000 (08:47 -0400)]
missing paren typo in arvados-bcbio-nextgen.py. refs #2991
Peter Amstutz [Tue, 8 Jul 2014 02:45:29 +0000 (22:45 -0400)]
bc-bio script links gemini reference data. refs #2991
Brett Smith [Tue, 8 Jul 2014 02:48:28 +0000 (22:48 -0400)]
3204: Remove redundant pull-right from Workbench Collections search.
content_for :tab_line_buttons is already wrapped in a div.pull-right.
Doing that a second time forces the contents into an overly-small
box.
Closes #3204.
radhika [Tue, 8 Jul 2014 00:53:36 +0000 (20:53 -0400)]
refs #3140
Revert back the UI to display only one tab instead of 5 content tabs since it is not working when accessing a project with large amounts of data.
Merge branch '3140-undo-project-content-tabs'
radhika [Tue, 8 Jul 2014 00:44:36 +0000 (20:44 -0400)]
3140: revert multiple content tabs work
Peter Amstutz [Mon, 7 Jul 2014 21:00:34 +0000 (17:00 -0400)]
add 'import sys' to arvados-bcbio-nextgen. no issue #
Peter Amstutz [Mon, 7 Jul 2014 20:54:17 +0000 (16:54 -0400)]
Don't use "arv" command wrapper, call arv-get/arv-put directly. Avoids
dependency on arvados-cli gem being properly configured on compute node.
no issue #
radhika [Mon, 7 Jul 2014 20:39:15 +0000 (16:39 -0400)]
2380: one more update
Peter Amstutz [Mon, 7 Jul 2014 20:34:14 +0000 (16:34 -0400)]
Update arvados-cli dependency to correct deployed version. no issue #
Peter Amstutz [Mon, 7 Jul 2014 20:24:47 +0000 (16:24 -0400)]
Merge branch '3123-pipeline-report-job-submit-errors' closes #3123
Peter Amstutz [Mon, 7 Jul 2014 20:24:04 +0000 (16:24 -0400)]
Google API client jsonifies the body, so we don't have to.
radhika [Mon, 7 Jul 2014 20:22:12 +0000 (16:22 -0400)]
2380: rename ssh-access.html.textile.liquid as ssh-access-unix.html.textile.liquid
radhika [Mon, 7 Jul 2014 20:11:08 +0000 (16:11 -0400)]
2380: minor updates
Peter Amstutz [Mon, 7 Jul 2014 19:48:12 +0000 (15:48 -0400)]
Update package version dependencies on arvados-cli to use commit:
7bd2ee2
radhika [Mon, 7 Jul 2014 19:47:41 +0000 (15:47 -0400)]
2380: more updates and also correct the link to unix page from within the windows page
Peter Amstutz [Mon, 7 Jul 2014 19:45:24 +0000 (15:45 -0400)]
arv-run-pipeline-instance sends error message to logs table when job submission
fails. Pipeline instance view in workbench now shows messages for the pipeline
instance (in addition to the jobs) and shows messages when the pipeline is
failed (not only when it is running.)
radhika [Mon, 7 Jul 2014 19:33:30 +0000 (15:33 -0400)]
Merge branch 'master' into 2380-ssh-doc
radhika [Mon, 7 Jul 2014 19:01:29 +0000 (15:01 -0400)]
closes #3140
Merge branch '3140-project-content-tabs'
Ward Vandewege [Mon, 7 Jul 2014 19:04:52 +0000 (15:04 -0400)]
crunch-dispatch: make sure to update the info['slurm_state'] for the
correct node, not just the first one that happens to have a matching
hostname in our database.
no issue #
radhika [Mon, 7 Jul 2014 19:00:25 +0000 (15:00 -0400)]
Merge branch 'master' into 3140-project-content-tabs
radhika [Mon, 7 Jul 2014 18:58:01 +0000 (14:58 -0400)]
Merge branch 'master' into 3140-project-content-tabs
radhika [Mon, 7 Jul 2014 18:51:29 +0000 (14:51 -0400)]
Merge branch 'master' into 2380-ssh-doc
radhika [Mon, 7 Jul 2014 18:51:00 +0000 (14:51 -0400)]
2380: more doc updates
radhika [Mon, 7 Jul 2014 18:36:25 +0000 (14:36 -0400)]
2380: minor doc updates
Peter Amstutz [Mon, 7 Jul 2014 18:22:48 +0000 (14:22 -0400)]
Don't render selection check boxes when using show_file partials from chooser
modal. closes #3103.
radhika [Mon, 7 Jul 2014 18:13:25 +0000 (14:13 -0400)]
2380: minor doc updates
Peter Amstutz [Mon, 7 Jul 2014 18:09:20 +0000 (14:09 -0400)]
Consider script_parameters in determining whether to merge script boxes in
pipeline graph drawing. Also tweaks behavior generating label for
script_parameter boxes to be more compact. closes #3100