radhika [Fri, 23 Oct 2015 13:44:47 +0000 (09:44 -0400)]
Merge branch 'master' into 7492-keepproxy-upstream-errors
Brett Smith [Fri, 23 Oct 2015 00:09:37 +0000 (20:09 -0400)]
Merge branch '7587-httplib2-retries-wip'
Refs #7587. Closes #7640.
Brett Smith [Wed, 21 Oct 2015 16:35:45 +0000 (12:35 -0400)]
7587: PySDK retries socket.error exceptions from API requests.
radhika [Thu, 22 Oct 2015 19:32:31 +0000 (15:32 -0400)]
closes #7546
Merge branch '7546-put-retry'
radhika [Thu, 22 Oct 2015 19:31:32 +0000 (15:31 -0400)]
7546: update comment to explain why we do not want to retry when status code is 503.
radhika [Thu, 22 Oct 2015 19:24:43 +0000 (15:24 -0400)]
Merge branch 'master' into 7546-put-retry
radhika [Thu, 22 Oct 2015 17:14:00 +0000 (13:14 -0400)]
refs #7167
Merge branch '7167-keep-rsync'
radhika [Thu, 22 Oct 2015 17:03:18 +0000 (13:03 -0400)]
Merge branch 'master' into 7167-keep-rsync
Brett Smith [Wed, 21 Oct 2015 19:37:53 +0000 (15:37 -0400)]
Fix multiple exception catching in arv-run.
The previous version catches IOError and binds the exception object to
the name OSError. No issue #.
radhika [Wed, 21 Oct 2015 16:23:09 +0000 (12:23 -0400)]
7167: expand the src and dst help messages to list the config parameters that are to be included in the config files.
Bryan Cosca [Wed, 21 Oct 2015 15:36:25 +0000 (11:36 -0400)]
Merge branch '7015-update-user-guide'
closes #7015
Ward Vandewege [Wed, 21 Oct 2015 15:19:46 +0000 (11:19 -0400)]
SSO installation doc fix: to run rails console, you need to be in the
/var/www/arvados-sso/current directory.
closes #7623
Bryan Cosca [Wed, 21 Oct 2015 15:18:52 +0000 (11:18 -0400)]
7015: Removed whitespace
radhika [Wed, 21 Oct 2015 13:00:58 +0000 (09:00 -0400)]
Merge branch 'master' into 7492-keepproxy-upstream-errors
Tom Clegg [Tue, 20 Oct 2015 21:53:27 +0000 (17:53 -0400)]
6358: Fix probe order test logic.
This request order is OK with two threads: thread "0" just took a long
time to make its request.
expect 0 1 2 3 4 5 6 7
got 1 2 3 4 5 0 6 7
The inverse is not OK. This would mean 0 started before any of
1,2,3,4,5 finished.
expect 1 2 3 4 5 0 6 7
got 0 1 2 3 4 5 6 7
refs #6358
Tom Clegg [Tue, 20 Oct 2015 21:52:55 +0000 (17:52 -0400)]
6358: Fix race opportunity in ThreadLimiter.
refs #6358
radhika [Tue, 20 Oct 2015 17:41:29 +0000 (13:41 -0400)]
7546: update some of the failure tests to use keepclient.Retries = 0, so that the tests do not waste too much time
retrying a test that is designed for failures. This update brings down the keepclient_test runtime from 49s to 10s.
radhika [Tue, 20 Oct 2015 17:30:45 +0000 (13:30 -0400)]
Merge branch 'master' into 7546-put-retry
radhika [Tue, 20 Oct 2015 15:41:07 +0000 (11:41 -0400)]
Merge branch 'master' into 7492-keepproxy-upstream-errors
radhika [Tue, 20 Oct 2015 15:40:05 +0000 (11:40 -0400)]
7492: update keep-rsync test with bad blob signing key to expect Forbidden error instead of Block not found error.
radhika [Tue, 20 Oct 2015 15:21:34 +0000 (11:21 -0400)]
7546: also retry when status code is 0, which is the case when a closed connection was used.
Brett Smith [Tue, 20 Oct 2015 15:04:29 +0000 (11:04 -0400)]
Merge branch 'pr/28'
Closes #7324.
Brett Smith [Tue, 20 Oct 2015 15:03:55 +0000 (11:03 -0400)]
Clean redundant except: blocks in run_test_server.
Brett Smith [Tue, 20 Oct 2015 15:02:51 +0000 (11:02 -0400)]
7324: Tighten exception ignoring in run_test_server start_nginx.
We just want to make sure the FIFO's gone. Ignore the OSError that
says "can't remove it because it's already gone," and re-raise all
others.
radhika [Tue, 20 Oct 2015 15:02:19 +0000 (11:02 -0400)]
Merge branch 'master' into 7546-put-retry
radhika [Tue, 20 Oct 2015 14:59:41 +0000 (10:59 -0400)]
7492: better error reporting of upstream errors in keepproxy.
Tom Clegg [Mon, 19 Oct 2015 19:29:28 +0000 (15:29 -0400)]
Merge branch '6358-put-rendezvous' closes #6358
Brett Smith [Mon, 19 Oct 2015 18:40:24 +0000 (14:40 -0400)]
7499: Update development link in Workbench "Getting Started" popup.
Also, de-hyphenate "open source."
Refs #7499.
radhika [Mon, 19 Oct 2015 18:13:25 +0000 (14:13 -0400)]
Merge branch 'master' into 7546-put-retry
radhika [Mon, 19 Oct 2015 18:12:00 +0000 (14:12 -0400)]
refs #7167
Merge branch '7167-keep-rsync'
radhika [Mon, 19 Oct 2015 18:11:41 +0000 (14:11 -0400)]
Merge branch 'master' into 7167-keep-rsync
radhika [Mon, 19 Oct 2015 03:21:58 +0000 (23:21 -0400)]
7546: enhance putReplicas method to retry.
Tom Clegg [Sat, 17 Oct 2015 04:39:25 +0000 (00:39 -0400)]
Merge branch '7173-jessie'
closes #7173
Tom Clegg [Fri, 16 Oct 2015 23:40:22 +0000 (19:40 -0400)]
6358: Test partial ordering with multiple writer threads.
Tom Clegg [Fri, 16 Oct 2015 22:26:14 +0000 (18:26 -0400)]
6358: Fix rendezvous probe order on Put.
Bug #1 was that KeepClient.put() was starting threads in the order
given by roots_map.iteritems(), instead of the order they were
supplied by weighted_service_roots(). This is fixed by using the same
logic get() was using.
Bug #2 was that ThreadLimiter didn't unblock threads in the same order
they were created by put(). This is fixed by adding a "set_sequence"
method to ThreadLimiter to indicate the order in which threads should
be unblocked.
The new test case confirms that put(copies=1) always makes requests in
the correct order.
Bryan Cosca [Fri, 16 Oct 2015 21:07:59 +0000 (17:07 -0400)]
7015: Finished going through user guide
Peter Amstutz [Fri, 16 Oct 2015 15:42:01 +0000 (11:42 -0400)]
Merge branch '6321-slurm-oserror' closes #6321
Peter Amstutz [Fri, 16 Oct 2015 15:40:36 +0000 (11:40 -0400)]
6321: Add note about rationale for retrying on OSError.
Peter Amstutz [Fri, 16 Oct 2015 15:16:49 +0000 (11:16 -0400)]
6321: Add test that OSError is caught from slurm subprocess invocations.
radhika [Fri, 16 Oct 2015 14:23:12 +0000 (10:23 -0400)]
7167: Remove StartKeepWithParams and StopKeepWithParams and make StartKeep and StopKeep with parameters the only exposed funcs.
The update was small enough, about 10 usages in the entire code, and hence did not make sense to postpone it for a "better" time.
Colin Nolan [Fri, 16 Oct 2015 13:15:09 +0000 (14:15 +0100)]
7324: Implemented deletion of previous nginx access log fifo before creation,
as discussed with @jrandall to address issue raised by @brettcs
(see: https://github.com/curoverse/arvados/pull/28#discussion_r39689972).
radhika [Fri, 16 Oct 2015 02:55:03 +0000 (22:55 -0400)]
7167: Break all the code from keep-rsync main method into a separate func so that arg parsing can also be tested.
Rather than using default flag parsing, use FlagSet so that flags can be set multiple times from multiple tests.
Bryan Cosca [Thu, 15 Oct 2015 21:10:19 +0000 (17:10 -0400)]
7015: Checked up to Concurrent Crunch tasks
Tom Clegg [Thu, 15 Oct 2015 20:27:58 +0000 (16:27 -0400)]
Merge branch '7159-keepexercise' refs #7159
Tom Clegg [Thu, 15 Oct 2015 20:25:14 +0000 (16:25 -0400)]
7159: Add keep-exercise
Tom Clegg [Thu, 15 Oct 2015 19:47:11 +0000 (15:47 -0400)]
Merge branch 'wtsi-hgi-feature/arv-view'
refs #7204
Tom Clegg [Thu, 15 Oct 2015 19:42:37 +0000 (15:42 -0400)]
7204: Tweak usage message, add a .shellescape, fix error-reporting error
radhika [Thu, 15 Oct 2015 16:35:32 +0000 (12:35 -0400)]
closes #7167
Merge branch '7167-keep-rsync'
radhika [Thu, 15 Oct 2015 16:35:15 +0000 (12:35 -0400)]
Merge branch 'master' into 7167-keep-rsync
radhika [Thu, 15 Oct 2015 16:32:07 +0000 (12:32 -0400)]
Merge branch 'master' into 7167-keep-rsync
Conflicts:
sdk/go/keepclient/keepclient.go
Tom Clegg [Thu, 15 Oct 2015 16:22:57 +0000 (12:22 -0400)]
7173: Update bundle.
Recent firefox requires recent selenium-webdriver.
Tom Clegg [Thu, 15 Oct 2015 16:21:07 +0000 (12:21 -0400)]
Merge branch '7173-jessie' refs #7173
Tom Clegg [Mon, 31 Aug 2015 19:00:13 +0000 (15:00 -0400)]
7173: Add jessie to supported distro list.
radhika [Thu, 15 Oct 2015 16:18:20 +0000 (12:18 -0400)]
7167: update the progress message to say "est. time remaining" instead of "ETA" which is not accurate in this case.
radhika [Thu, 15 Oct 2015 15:56:36 +0000 (11:56 -0400)]
7167: rename the newly added StopKeepServers as StopKeepWithParams; it now sounds similar to StartKeepWithParams and is self explanatory.
radhika [Thu, 15 Oct 2015 15:08:10 +0000 (11:08 -0400)]
Merge branch '7167-keep-rsync' of git.curoverse.com:arvados into 7167-keep-rsync
Tom Clegg [Thu, 15 Oct 2015 15:02:30 +0000 (11:02 -0400)]
7167: Always pass --num-keep-servers to python wrapper. Update comments.
Tom Clegg [Thu, 15 Oct 2015 14:50:13 +0000 (10:50 -0400)]
7167: Rename conf flags to -src and -dst to match arv-copy. Always add .conf for short form.
Tom Clegg [Thu, 15 Oct 2015 14:48:33 +0000 (10:48 -0400)]
7167: Fix up progress/ETA logging.
Peter Amstutz [Thu, 15 Oct 2015 14:32:26 +0000 (10:32 -0400)]
Merge branch '7491-go-keepclient-retry' closes #7491
Peter Amstutz [Thu, 15 Oct 2015 14:16:27 +0000 (10:16 -0400)]
7491: Remove unnecessary writable service roots from GET tests.
Tom Clegg [Thu, 15 Oct 2015 13:18:43 +0000 (09:18 -0400)]
Merge branch 'master' into wtsi-hgi-feature/arv-view
Conflicts:
sdk/cli/bin/arv
radhika [Thu, 15 Oct 2015 03:01:13 +0000 (23:01 -0400)]
Merge branch 'master' into 7167-keep-rsync
radhika [Thu, 15 Oct 2015 03:00:05 +0000 (23:00 -0400)]
7167: log progress during keep-rsync and several test improvements.
Bryan Cosca [Wed, 14 Oct 2015 21:09:07 +0000 (17:09 -0400)]
7015: Checked up to Working with data sets
Brett Smith [Wed, 14 Oct 2015 20:59:40 +0000 (16:59 -0400)]
API server needs an arvados-cli with crunch-job --docker-bin.
crunch-dispatch was extended to use `crunch-job --docker-bin` in
#6838. This commit simply updates the Gemfile to ensure this
dependency is satisfied. No issue #.
radhika [Wed, 14 Oct 2015 19:15:15 +0000 (15:15 -0400)]
Merge branch 'master' into 7167-keep-rsync
Bryan Cosca [Wed, 14 Oct 2015 19:01:00 +0000 (15:01 -0400)]
7015: Tested up to Accessing an Arvados VM with Webshell
Tom Clegg [Wed, 14 Oct 2015 18:51:34 +0000 (14:51 -0400)]
Merge branch '7167-propagate-error' refs #7167
Tom Clegg [Wed, 14 Oct 2015 18:51:01 +0000 (14:51 -0400)]
Merge branch '7159-clean-index' refs #7159 refs #7168
radhika [Wed, 14 Oct 2015 17:43:54 +0000 (13:43 -0400)]
7167: loadConfig setupKeepclient do only one set at a time.
Peter Amstutz [Wed, 14 Oct 2015 14:49:49 +0000 (10:49 -0400)]
6321: Add OSError to slurm shutdown actor retry conditions, to catch and retry
subprocess errors from the operating system, which is preferrable to crashing
the shutdown actor.
radhika [Wed, 14 Oct 2015 02:27:41 +0000 (22:27 -0400)]
7167: when the config file does not contain '/', use $HOME/.config/arvados/<filename>.
radhika [Wed, 14 Oct 2015 01:47:06 +0000 (21:47 -0400)]
7167: replace keep_existing with num_keep_servers and use it create all required keep servers at once.
radhika [Wed, 14 Oct 2015 01:16:35 +0000 (21:16 -0400)]
7167: replace the keep_existing logic and create all 3 keep servers at once and use the first two as src keepservers and the last one as the dst keep server.
radhika [Tue, 13 Oct 2015 21:01:46 +0000 (17:01 -0400)]
7167: Convert blobSigningKey also into local variable and make necessary changes to accommodate this change.
Remove the New method added in arvadosclient.go and revert MakeArvadosClient to what it was before.
radhika [Tue, 13 Oct 2015 19:46:26 +0000 (15:46 -0400)]
7167: update run_test_servers.py to use action="store_true" instead of converting string to boolean.
Tom Clegg [Tue, 13 Oct 2015 19:33:33 +0000 (15:33 -0400)]
7159: Address golint complaints
Tom Clegg [Tue, 13 Oct 2015 19:33:02 +0000 (15:33 -0400)]
7159: Omit non-Keep blobs from index
radhika [Tue, 13 Oct 2015 17:38:52 +0000 (13:38 -0400)]
Merge branch 'master' into 7167-keep-rsync
radhika [Tue, 13 Oct 2015 17:37:33 +0000 (13:37 -0400)]
7167: Convert most of the globals in keep-sync into locals and update all the code and tests as needed.
Tom Clegg [Tue, 13 Oct 2015 16:11:23 +0000 (12:11 -0400)]
7159: Return benign os.ErrNotExist error from Compare to avoid excessive logs. refs #7159
Tom Clegg [Tue, 13 Oct 2015 15:17:49 +0000 (11:17 -0400)]
7159: Fix error handling when reading full size block. refs #7159
Tom Clegg [Mon, 12 Oct 2015 17:41:04 +0000 (13:41 -0400)]
7167: Propagate read errors to caller. Fixes failing TestTransferShortBuffer.
Tom Clegg [Mon, 12 Oct 2015 16:40:38 +0000 (12:40 -0400)]
Warn about unhandled case if broken node has no ping time. refs #7286
Tom Clegg [Mon, 12 Oct 2015 16:06:47 +0000 (12:06 -0400)]
Merge branch '7159-empty-blob-race' refs #7159
Tom Clegg [Mon, 12 Oct 2015 13:33:24 +0000 (09:33 -0400)]
7159: Shorten race waits during generic tests
Tom Clegg [Fri, 9 Oct 2015 21:09:42 +0000 (17:09 -0400)]
7159: Log when waiting for get/put races
Tom Clegg [Thu, 8 Oct 2015 18:00:00 +0000 (14:00 -0400)]
7159: Exclude new empty blocks from index.
Tom Clegg [Thu, 8 Oct 2015 17:30:18 +0000 (13:30 -0400)]
7159: Test race deadline
Tom Clegg [Thu, 8 Oct 2015 16:52:17 +0000 (12:52 -0400)]
7159: Work around CreateBlob race by polling for updates when a brand new blob is found empty.
radhika [Mon, 12 Oct 2015 14:40:38 +0000 (10:40 -0400)]
7167: break load config logic out of main into loadConfig func and add several tests.
radhika [Mon, 12 Oct 2015 13:13:45 +0000 (09:13 -0400)]
7167: some more error tests such as error getting block from src and error putting block to dst.
radhika [Sat, 10 Oct 2015 00:09:17 +0000 (20:09 -0400)]
7167: stop rsync operation on any errors during Get or Put operations; add additional tests.
radhika [Fri, 9 Oct 2015 22:01:14 +0000 (18:01 -0400)]
Merge branch '7167-keep-rsync-test-setup' into 7167-keep-rsync
radhika [Fri, 9 Oct 2015 22:00:56 +0000 (18:00 -0400)]
Merge branch 'master' into 7167-keep-rsync-test-setup
Conflicts:
sdk/go/keepclient/perms.go
sdk/go/keepclient/perms_test.go
services/keepstore/perms.go
services/keepstore/perms_test.go
Peter Amstutz [Fri, 9 Oct 2015 20:41:52 +0000 (16:41 -0400)]
7491: Fix response body close, error logging.
Peter Amstutz [Fri, 9 Oct 2015 20:06:14 +0000 (16:06 -0400)]
Merge branch 'master' into 7491-go-keepclient-retry
Conflicts:
sdk/go/keepclient/keepclient.go
Peter Amstutz [Fri, 9 Oct 2015 19:52:36 +0000 (15:52 -0400)]
7491: Remove KeepServerError, all errors are BlockNotFound errors for now.
Refactor Get() and Ask() to use common function.
Tom Clegg [Fri, 9 Oct 2015 19:48:28 +0000 (15:48 -0400)]
Merge branch '7491-keepclient-bugs' refs #7491
projects / arvados.git / log