radhika [Sat, 18 Oct 2014 00:09:29 +0000 (20:09 -0400)]
Merge branch 'master' into 4054-sharing-email
radhika [Fri, 17 Oct 2014 22:40:28 +0000 (18:40 -0400)]
4054: show email address in sharing tab and update test to verify.
Peter Amstutz [Fri, 17 Oct 2014 20:54:31 +0000 (16:54 -0400)]
Merge branch '3698-invalid-output-of' closes #3698
Ward Vandewege [Fri, 17 Oct 2014 19:47:11 +0000 (15:47 -0400)]
Make sure to use httpclient 2.5.0 or higher for Workbench. Older
versions force the use of SSLv3, which is no longer advisable now that
Poodle is a thing.
No issue #
Peter Amstutz [Fri, 17 Oct 2014 19:36:49 +0000 (15:36 -0400)]
3698: Fix: components keys are symbols, not strings. Tweaked error message to
tell you what the invalid output_of actually is.
Peter Amstutz [Fri, 17 Oct 2014 19:17:01 +0000 (15:17 -0400)]
3698: Add check that output_of refers to an actual component.
Peter Amstutz [Fri, 17 Oct 2014 17:58:50 +0000 (13:58 -0400)]
4083: Searched for everything using strptime and added import _strptime where it was missing. closes #4083
Tom Clegg [Fri, 17 Oct 2014 15:45:33 +0000 (11:45 -0400)]
Merge branch '4229-close-files' closes #4229
Nancy Ouyang [Fri, 17 Oct 2014 13:52:17 +0000 (09:52 -0400)]
added by_id subdirectory notes and arvados#collection
No issue #.
Brett Smith [Fri, 17 Oct 2014 14:22:01 +0000 (10:22 -0400)]
4239: Update Python SDK for google-api-python-client 1.3.
This package renames the module from apiclient to googleapiclient. It
provides a shim apiclient module for compatibility, but it only
exports the appropriate names, not real submodules. Using it requires
changing imports like
import apiclient.errors
to
from apiclient import errors
This approach works (has been tested) with both versions 1.2 and 1.3.
Closes #4239.
Brett Smith [Fri, 17 Oct 2014 12:36:54 +0000 (08:36 -0400)]
4189: Diagnostics suite doesn't try to clear Rails cache.
Refs #4189.
Brett Smith [Thu, 16 Oct 2014 21:23:57 +0000 (17:23 -0400)]
Merge branch '4189-workbench-project-admin-attr-editing-wip'
Closes #4189, #4202.
Brett Smith [Wed, 15 Oct 2014 18:04:12 +0000 (14:04 -0400)]
4189: Workbench lets users with write permission rename objects in projects.
The key part of this commit are the changes to ApplicationHelper.
There's also some refactoring to more consistently answer questions
about whether or not an attribute is editable.
There are also some tests in here that I wrote while I was tracking
down the problem. They seem like a useful addition.
Brett Smith [Wed, 15 Oct 2014 17:29:48 +0000 (13:29 -0400)]
4189: Workbench tests clear more state during teardown.
I'm working on some new tests that pick up on stale state without
these changes.
Brett Smith [Thu, 16 Oct 2014 21:00:39 +0000 (17:00 -0400)]
4189: Document the API server's writable_by field.
Tom Clegg [Thu, 16 Oct 2014 20:17:51 +0000 (16:17 -0400)]
4189: Add writable_by to User API response, so writable_by is available for every owner_uuid.
Tom Clegg [Thu, 16 Oct 2014 20:11:13 +0000 (16:11 -0400)]
4189: Admit in writable_by that an object is writable when its owner_uuid is writable.
Tom Clegg [Thu, 16 Oct 2014 20:09:16 +0000 (16:09 -0400)]
4189: Look up owner's writable_by when API did not provide writable_by for the object in question.
Peter Amstutz [Thu, 16 Oct 2014 19:35:00 +0000 (15:35 -0400)]
Merge branch '3609-arv-ws' refs #3609
Peter Amstutz [Thu, 16 Oct 2014 19:34:25 +0000 (15:34 -0400)]
3609: Fix PollClient for case when there are no filters (get everything).
Peter Amstutz [Thu, 16 Oct 2014 19:24:51 +0000 (15:24 -0400)]
3609: Testing fixups.
Peter Amstutz [Thu, 16 Oct 2014 18:59:18 +0000 (14:59 -0400)]
3609: Back out setting api_version, api_host, and api_insecure on api client
and instead document that caller to arvados.events.subscribe() should pass in a
freshly created api client. Add event variable to indicate when subscription
is a success instead of sleeping. Removed unused "import time". Fixed "web
sockets" -> "websockets".
Tom Clegg [Thu, 16 Oct 2014 18:20:10 +0000 (14:20 -0400)]
4229: Add missing Close() calls
Tom Clegg [Thu, 16 Oct 2014 18:19:51 +0000 (14:19 -0400)]
4229: Get rid of sleep() in test
Peter Amstutz [Thu, 16 Oct 2014 17:20:12 +0000 (13:20 -0400)]
3609: import signal; add_mutually_exclusive_group doesn't take any arguments.
Brett Smith [Thu, 16 Oct 2014 16:47:09 +0000 (12:47 -0400)]
Python SDK is not compatible with apiclient 1.3.
Figuring out why this is needs more investigation. It's something I'd
like to do, but it seems urgent to push this fix so builds can pass
again. No issue #, but see
<https://ci.curoverse.com/job/arvados-api-server/980/>.
Peter Amstutz [Thu, 16 Oct 2014 16:23:41 +0000 (12:23 -0400)]
3609: Add additional api_ fields to api client object so that subscribe() can
create a new api client. Change test to use event variable instead of
sleeping. Fix "Web sockets" -> "Websockets". Make some command line options
mutually exclusive.
Tom Clegg [Thu, 16 Oct 2014 15:51:56 +0000 (11:51 -0400)]
Merge branch '4219-verify-with-hints' closes #4219
Tom Clegg [Thu, 16 Oct 2014 15:34:16 +0000 (11:34 -0400)]
4219: Add test for missing size hint. Dry up "known signature hint" consts.
Peter Amstutz [Thu, 16 Oct 2014 14:43:34 +0000 (10:43 -0400)]
Merge branch 'master' into 3609-arv-ws
Peter Amstutz [Thu, 16 Oct 2014 13:30:48 +0000 (09:30 -0400)]
3609: Prefetch pipeline components to find out which jobs to subscribe to.
Tom Clegg [Thu, 16 Oct 2014 05:29:09 +0000 (01:29 -0400)]
4219: Use a precompiled global regexp for signature verification.
Tom Clegg [Wed, 15 Oct 2014 22:27:35 +0000 (18:27 -0400)]
4219: Accept signatures even if other hints are present in URL.
mishaz [Wed, 15 Oct 2014 21:18:25 +0000 (21:18 +0000)]
Merge branch '4197-remove-enforce-permissions-check-from-index-handler'
Closes #4197
Peter Amstutz [Wed, 15 Oct 2014 21:12:59 +0000 (17:12 -0400)]
3609: PollClient waits on event object instead of time.sleep, fix filters. Also fix --poll-interval and --no-poll
Peter Amstutz [Wed, 15 Oct 2014 21:09:52 +0000 (17:09 -0400)]
3609: Roll back schema change, not appropriate for this branch
mishaz [Tue, 14 Oct 2014 22:11:40 +0000 (22:11 +0000)]
Modified IndexHandler to match TrashHandler and PullHandler
* No longer checks for the enforce-permissions flag.
* Still checks for DataManager auth token.
* The HTTP error returned when we don't find the DataManager auth token is now Unauthorized instead of Forbidden.
Modified tests to check for new behavior.
Tom Clegg [Wed, 15 Oct 2014 20:59:07 +0000 (16:59 -0400)]
Resolve race condition: wait for response before checking mock expectations. refs #3782
Peter Amstutz [Wed, 15 Oct 2014 20:22:26 +0000 (16:22 -0400)]
3609: Add test case for PollClient. arvados.event.subscribe() starts a new
polling thread by default so as to provide the same behavior as websockets.
Add documentation strings to ws.py command line parameters.
Peter Amstutz [Wed, 15 Oct 2014 20:20:04 +0000 (16:20 -0400)]
3609: Fix schema so 'order' parameter of 'list' is an array instead of a string.
Brett Smith [Wed, 15 Oct 2014 13:22:40 +0000 (09:22 -0400)]
4196: arv-keepdocker uses Docker 1.0 pull syntax.
`docker pull -t` was the old way of pulling an image with a specific
tag. Docker 1.0 introduced `docker pull repo:tag`, and Docker 1.2
removes support for `-t` completely. We're well-standardized on 1.1+
at this point, and considering an upgrade to 1.2+, so now's the time
to switch.
Closes #4196.
Brett Smith [Tue, 14 Oct 2014 20:30:21 +0000 (16:30 -0400)]
4126: API server uses fixed a-r-p-i (again).
Refs #4126.
Brett Smith [Tue, 14 Oct 2014 20:15:16 +0000 (16:15 -0400)]
4126: a-r-p-i dereferences parameter values when creating jobs.
Refs #4126. This is required to run jobs.
Brett Smith [Tue, 14 Oct 2014 18:49:49 +0000 (14:49 -0400)]
4126: API server uses fixed a-r-p-i.
Refs #4126.
Peter Amstutz [Tue, 14 Oct 2014 15:41:02 +0000 (11:41 -0400)]
Merge branch '3692-event-bus-fix-and' closes #3692
Peter Amstutz [Tue, 14 Oct 2014 15:33:31 +0000 (11:33 -0400)]
Merge branch '3656-arv-create' closes #3656
Peter Amstutz [Tue, 14 Oct 2014 15:32:38 +0000 (11:32 -0400)]
3656: Delete unused documentation page
Peter Amstutz [Tue, 14 Oct 2014 14:54:51 +0000 (10:54 -0400)]
3656: Alphabetize list of subcommands. Rename tmp -> tmp_file. Small wording change on doc pages.
Peter Amstutz [Tue, 14 Oct 2014 14:40:36 +0000 (10:40 -0400)]
3692: Bug fix for inadequate grouping when constructing selection.
Brett Smith [Tue, 14 Oct 2014 14:19:41 +0000 (10:19 -0400)]
Merge branch '4139-blocking-node-manager-tests-wip'
Refs #2881, #4139. Closes #4184.
Brett Smith [Tue, 14 Oct 2014 14:18:49 +0000 (10:18 -0400)]
4139: Node Manager README links to compute node lifecycle page.
Brett Smith [Mon, 13 Oct 2014 19:19:15 +0000 (15:19 -0400)]
4139: Speed up Node Manager tests.
Previously, the tests would poll interesting mocks, waiting for them
to be called. This introduces significant overhead to the tests, and
they would frequently time out on Jenkins. This modifies the tests to
get more information by blocking on the tested actors, which means
more predictability and less fighting for CPU (typical runtimes for
all the tests improved from 5 seconds to 0.5 seconds on my
workstation).
The downside to this approach is that it ties the tests more tightly
to the underlying actors' implementation. In particular, they
sometimes send a message and block for a response to ensure that any
internal messages generated by the *last* message have been handled.
This is less than ideal, but I don't have a better idea right now.
Peter Amstutz [Fri, 10 Oct 2014 20:47:07 +0000 (16:47 -0400)]
Fix filters, subscribing to components of pipelines.
Peter Amstutz [Fri, 10 Oct 2014 20:14:15 +0000 (16:14 -0400)]
3692: Explicitly incorporate sequence number test into where clause
Peter Amstutz [Fri, 10 Oct 2014 19:43:52 +0000 (15:43 -0400)]
Websocket server side fix, perform database notify in after_save callback on
the log object instead of in log_change on ArvadosBase because crunch-dispatch
was creating Log objects directly and bypassing the notification in log_change.
Peter Amstutz [Fri, 10 Oct 2014 19:45:44 +0000 (15:45 -0400)]
3609: Add some options to control polling fallback, improve error logging a bit.
Peter Amstutz [Fri, 10 Oct 2014 19:43:52 +0000 (15:43 -0400)]
Websocket server side fix, perform database notify in after_save callback on
the log object instead of in log_change on ArvadosBase because crunch-dispatch
was creating Log objects directly and bypassing the notification in log_change.
Peter Amstutz [Fri, 10 Oct 2014 15:25:48 +0000 (11:25 -0400)]
3609: Logging should work for jobs but running into a bug in eventbus
Peter Amstutz [Fri, 10 Oct 2014 14:52:26 +0000 (10:52 -0400)]
3609: --job and --pipeline logging implemented, needs testing
Peter Amstutz [Fri, 10 Oct 2014 14:17:40 +0000 (10:17 -0400)]
3609: Refactoring arv-ws into a reusable command module. Working on adding
ability to monitor pipeline/job log at command line.
radhika [Fri, 10 Oct 2014 13:45:09 +0000 (09:45 -0400)]
closes #4126
Merge branch '4126-preserve-parameter-hash'
Peter Amstutz [Fri, 10 Oct 2014 12:52:27 +0000 (08:52 -0400)]
3692: Fixed test, and fixed the actual bug
Peter Amstutz [Thu, 9 Oct 2014 18:56:47 +0000 (14:56 -0400)]
3656: Add missing file
Brett Smith [Thu, 9 Oct 2014 17:46:51 +0000 (13:46 -0400)]
4139: Add README to Node Manager.
Refs #4139.
Brett Smith [Thu, 9 Oct 2014 17:32:50 +0000 (13:32 -0400)]
4139: Add *.egg to Node Manager's .gitignore.
`python setup.py test` will automatically download dependencies to the
source directory if you don't already have them available in your
environment. Refs #2881, #4139.
Brett Smith [Thu, 9 Oct 2014 17:31:47 +0000 (13:31 -0400)]
4139: Add environment configuration knobs for Node Manager tests.
These are settings I've fiddled with regularly over the course of
development, and now it looks like we're going to need to fiddle them
some more to accommodate Jenkins. I'm exposing them as environment
variables so I can stop messing with the code appropriately.
Refs #4139.
Peter Amstutz [Thu, 9 Oct 2014 17:21:27 +0000 (13:21 -0400)]
3656: Documentation updated to use "arv create".
Peter Amstutz [Thu, 9 Oct 2014 15:37:39 +0000 (11:37 -0400)]
3656: Support additional create parameters on the command line, only open
editor on the object itself, should be less confusing.
Peter Amstutz [Thu, 9 Oct 2014 15:04:43 +0000 (11:04 -0400)]
3656: Add arv-create command. Refactor run_editor to be shared by arv_edit and arv_create.
Peter Amstutz [Thu, 9 Oct 2014 13:33:34 +0000 (09:33 -0400)]
Merge branch '4042-run-command-MxN' closes #4042
Peter Amstutz [Thu, 9 Oct 2014 13:32:36 +0000 (09:32 -0400)]
4042: Typo fixes. Highlight run-command and script_parameters in text. Rename
--job-parameters to --script-parameters and add mention of --dry-run mode.
Brett Smith [Thu, 9 Oct 2014 13:20:38 +0000 (09:20 -0400)]
Update install docs for keep→keepstore rename.
No issue #. Reported on #arvados.
Peter Amstutz [Thu, 9 Oct 2014 13:15:30 +0000 (09:15 -0400)]
Merge branch '3381-job-progress-bar-bug' closes #3381
Peter Amstutz [Thu, 9 Oct 2014 13:14:56 +0000 (09:14 -0400)]
3381: Fix layout comment
radhika [Thu, 9 Oct 2014 12:52:23 +0000 (08:52 -0400)]
4126: undo the hash parameter retention logic for value
radhika [Wed, 8 Oct 2014 21:30:59 +0000 (17:30 -0400)]
Merge branch 'master' into 4126-preserve-parameter-hash
radhika [Wed, 8 Oct 2014 21:28:38 +0000 (17:28 -0400)]
4126: when parameter is a hash, use it as value if nothing else matches.
Brett Smith [Wed, 8 Oct 2014 20:52:44 +0000 (16:52 -0400)]
Merge branch '2881-node-manager'
Closes #2881, #4106.
Brett Smith [Fri, 3 Oct 2014 21:53:57 +0000 (17:53 -0400)]
2881: Add Node Manager service.
Peter Amstutz [Wed, 8 Oct 2014 19:00:42 +0000 (15:00 -0400)]
Updated examples.
Peter Amstutz [Wed, 8 Oct 2014 18:50:12 +0000 (14:50 -0400)]
Merge branch '4042-run-command-MxN' of git.curoverse.com:arvados into 4042-run-command-MxN
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
Peter Amstutz [Wed, 8 Oct 2014 18:49:54 +0000 (14:49 -0400)]
4042: Update documentation samples. Small fix to dry-run to allow supplying custom TASK_KEEPMOUNT.
Peter Amstutz [Wed, 8 Oct 2014 15:12:57 +0000 (11:12 -0400)]
4042: Rename bad reuse of 'p' to 'match' in expand_item. Finish describing
$(task.outdir). Clarify that list functions take a user parameter name. Fix
other spelling and grammatical errors in documentation.
Tom Clegg [Wed, 8 Oct 2014 14:58:31 +0000 (10:58 -0400)]
Merge branch '4044-crunchstat-wait' refs #4044
Tom Clegg [Wed, 8 Oct 2014 14:58:23 +0000 (10:58 -0400)]
4044: Merge branch 'master' into 4044-crunchstat-wait
Tom Clegg [Wed, 8 Oct 2014 14:58:11 +0000 (10:58 -0400)]
4044: Add comments to "continue" statements.
Peter Amstutz [Wed, 8 Oct 2014 14:45:35 +0000 (10:45 -0400)]
3381: Reorganize _running_components to be clearer. Fixing workbench.
Tom Clegg [Wed, 8 Oct 2014 14:05:27 +0000 (10:05 -0400)]
4126: Preserve hash form when populating component parameters.
radhika [Wed, 8 Oct 2014 13:58:44 +0000 (09:58 -0400)]
closes #3990
Merge branch '3990-owner-when-rerunning-pipeline'
radhika [Wed, 8 Oct 2014 13:58:10 +0000 (09:58 -0400)]
3990: minor test update
Tom Clegg [Tue, 7 Oct 2014 20:24:15 +0000 (16:24 -0400)]
4044: Clean up channel and pipe usage.
* Remove code for using one goroutine to copy stdout and stderr from
channels to pipes. Instead, copy stderr from channel to pipe in a
simple goroutine, and use give the child our own Stdout to use (we
don't use it ourselves anyway).
* Rename functions (OutputChannel -> CopyChanToPipe, ReadLineByLine ->
CopyPipeToChan).
* Add "stop" channel to shut down polling loop when child process
exits.
* Check for errors when opening cgroup stats files. Report errors
instead of displaying bogus stats.
* Split main() into run() and a short main() with os.Exit logic, so
deferred functions run regardless of exit path.
* Close dangling filehandle when cgroup_cidfile succeeds.
* Fix slight divide-by-zero opportunity when elapsed==0.
* Fix condition triggering the "could not read cid file" error
message.
radhika [Tue, 7 Oct 2014 18:04:26 +0000 (14:04 -0400)]
Merge branch 'master' into 3990-owner-when-rerunning-pipeline
radhika [Tue, 7 Oct 2014 18:02:35 +0000 (14:02 -0400)]
closes #3882
Merge branch '3882-cancel-already-cancelled-job'
radhika [Tue, 7 Oct 2014 17:55:32 +0000 (13:55 -0400)]
3882: update "after_validation :update_timestamps_when_state_changes" to "before_save :update_timestamps_when_state_changes",
and also add comple more test combinations around cancelled state.
radhika [Tue, 7 Oct 2014 17:29:01 +0000 (13:29 -0400)]
Merge branch 'master' into 3882-cancel-already-cancelled-job
radhika [Tue, 7 Oct 2014 17:26:18 +0000 (13:26 -0400)]
Merge branch 'master' into 3990-owner-when-rerunning-pipeline
radhika [Tue, 7 Oct 2014 17:17:33 +0000 (13:17 -0400)]
3990: rerunning pipeline from within a project with no write permission
Tom Clegg [Tue, 7 Oct 2014 16:04:34 +0000 (12:04 -0400)]
Merge branch '3775-fetch-git-repo' closes #3775
Conflicts:
sdk/cli/bin/crunch-job
radhika [Tue, 7 Oct 2014 14:42:57 +0000 (10:42 -0400)]
Merge branch 'master' into 3990-owner-when-rerunning-pipeline
radhika [Tue, 7 Oct 2014 14:41:30 +0000 (10:41 -0400)]
3990: add integration tests to rerun pipeline.