Lucas Di Pentima [Tue, 19 Jul 2016 15:07:22 +0000 (12:07 -0300)]
9463: Using finally: on except: clause to stop the updater thread on error or finished work. When counting the collection size, stopped ignoring items that are not Collection or ArvadosFile, in case some other class is used in the future. When getting block locators (for example when called with --raw), don\'t make sure the blocks are flushed to Keep before asking for segments
Lucas Di Pentima [Thu, 14 Jul 2016 20:53:18 +0000 (17:53 -0300)]
9463: blank spaces cleanup
Lucas Di Pentima [Thu, 14 Jul 2016 20:48:31 +0000 (17:48 -0300)]
9463: New approach on this arv-put revamping: the ArvPutUploadJob class writes to a Collection and start a thread to periodically poll the upload status
Lucas Di Pentima [Fri, 8 Jul 2016 22:23:54 +0000 (19:23 -0300)]
9463: Finished first draft on arv-put command use of Collection class. Also, partial uploads resuming is working. Tests were written to validate this. Refs #9463 #8910
Lucas Di Pentima [Thu, 7 Jul 2016 21:32:50 +0000 (18:32 -0300)]
9463: Fixed some integration tests broken because of the replication_desired addition to Collection
Lucas Di Pentima [Thu, 7 Jul 2016 18:44:16 +0000 (15:44 -0300)]
9463: Added to Collection a way to ask _BlockManager how many copies need to be written when uploading blocks to Keep
Lucas Di Pentima [Thu, 7 Jul 2016 16:19:24 +0000 (13:19 -0300)]
9463: Added desired replication level to collection record on API Server when using Collection class
Lucas Di Pentima [Thu, 7 Jul 2016 03:20:58 +0000 (00:20 -0300)]
9463: Polishing the last details to make the integration tests work ok
Lucas Di Pentima [Tue, 5 Jul 2016 21:53:37 +0000 (18:53 -0300)]
9463: Lots of progress today, resume upload code written, many tests to do!
Lucas Di Pentima [Mon, 4 Jul 2016 23:07:56 +0000 (20:07 -0300)]
9463: Resume cache implementation (WIP)
Lucas Di Pentima [Fri, 1 Jul 2016 22:05:18 +0000 (19:05 -0300)]
9463: Initial coding with tests
Peter Amstutz [Fri, 24 Jun 2016 14:01:31 +0000 (10:01 -0400)]
9488: Add log message to notify users on sucessful websocket reconnect. refs #9488
Tom Clegg [Fri, 24 Jun 2016 14:59:33 +0000 (10:59 -0400)]
Merge branch '9486-filter-nodes'
refs #9486
Tom Clegg [Thu, 23 Jun 2016 16:47:49 +0000 (12:47 -0400)]
9486: Use API to filter nodes by last_ping_at.
Peter Amstutz [Wed, 22 Jun 2016 16:18:49 +0000 (12:18 -0400)]
Update run-build-packages versions for schema_salad and cwltool, refs #8442
Peter Amstutz [Wed, 22 Jun 2016 15:44:03 +0000 (11:44 -0400)]
Merge branch 'origin-8442-cwl-crunch2' closes #8442
Peter Amstutz [Wed, 22 Jun 2016 13:57:58 +0000 (09:57 -0400)]
Merge branch 'master' into origin-8442-cwl-crunch2
Peter Amstutz [Wed, 22 Jun 2016 13:55:12 +0000 (09:55 -0400)]
8442: Code cleanup. Remove dead code. Update comments.
Brett Smith [Wed, 22 Jun 2016 13:46:40 +0000 (09:46 -0400)]
PySDK: Fix spelling of X-Keep-Desired-Replicas header.
This matches what Keep services expect, and other clients use.
No issue #.
Peter Amstutz [Tue, 21 Jun 2016 20:10:46 +0000 (16:10 -0400)]
8442: Fix error reporting on missing Docker image
Peter Amstutz [Tue, 21 Jun 2016 19:57:23 +0000 (15:57 -0400)]
Increase timeout on failing websocket test. refs #9427
Peter Amstutz [Tue, 21 Jun 2016 16:20:48 +0000 (12:20 -0400)]
Merge branch '9427-threaded-websockets' closes #9427
Peter Amstutz [Tue, 21 Jun 2016 16:16:23 +0000 (12:16 -0400)]
8442: Make work_api parameter to ArvCwlRunner constructor optional.
Peter Amstutz [Tue, 21 Jun 2016 16:01:57 +0000 (12:01 -0400)]
8442: cwl-runner crunch script must provide work_api parameter to ArvCwlRunner
constructor.
Peter Amstutz [Tue, 21 Jun 2016 15:55:43 +0000 (11:55 -0400)]
8442: Bugfixes from end-to-end testing.
Peter Amstutz [Tue, 21 Jun 2016 14:56:57 +0000 (10:56 -0400)]
9427: Back out redundant connection pool configuration option and add comment
to database.yml.example about connection pool and websockets.
Peter Amstutz [Tue, 21 Jun 2016 12:22:54 +0000 (08:22 -0400)]
8442: Fix tests. Update comments. Rename jobs -> processes since it doesn't hold just "jobs" any more.
Peter Amstutz [Mon, 20 Jun 2016 21:34:07 +0000 (17:34 -0400)]
8442: Rename --crunch1 and --crunch2 options to --api=jobs and --api=containers
Lucas Di Pentima [Mon, 20 Jun 2016 19:59:38 +0000 (16:59 -0300)]
Merge branch '9180-avoid-overreplication-keepclient'
Closes #9180
Peter Amstutz [Mon, 20 Jun 2016 18:46:54 +0000 (14:46 -0400)]
Merge branch '9353-retry-http-error' closes #9353
Peter Amstutz [Mon, 20 Jun 2016 18:29:21 +0000 (14:29 -0400)]
9427: Add limits for connections, subscriptions, queued notifications, and
database connections to configuration. Check notify queue size on push instead
of on pop and close connection immediately.
Peter Amstutz [Mon, 20 Jun 2016 13:35:21 +0000 (09:35 -0400)]
Add missing arvbox package "virtualenv" refs #9408
Peter Amstutz [Fri, 17 Jun 2016 20:41:03 +0000 (16:41 -0400)]
9427: Add connection limit.
Peter Amstutz [Fri, 17 Jun 2016 20:27:18 +0000 (16:27 -0400)]
Bugfix "arv-ws --id" option, refs #9388.
Peter Amstutz [Fri, 17 Jun 2016 20:12:15 +0000 (16:12 -0400)]
9427: Websockets now uses a thread per connection for database query and
sending outgoing messages. This minimizes interference by slow clients or
clients that are "replaying the world" with a last_log_id far in the past.
radhika [Fri, 17 Jun 2016 15:13:28 +0000 (11:13 -0400)]
closes #9372
Merge branch '9372-container-display'
radhika [Fri, 17 Jun 2016 15:13:12 +0000 (11:13 -0400)]
Merge branch 'master' into 9372-container-display
radhika [Fri, 17 Jun 2016 15:11:48 +0000 (11:11 -0400)]
9372: improve container_work_unit -> uri method to check if @proxied will respond_to table_name instead of using a rescue.
Lucas Di Pentima [Thu, 16 Jun 2016 23:06:21 +0000 (20:06 -0300)]
9180: Changed some of the logic on ThreadLimiter and made unit tests to validate the new behaviour
refs #9180
radhika [Thu, 16 Jun 2016 14:32:21 +0000 (10:32 -0400)]
9372: The children of a Container will be ContainerRequests to which this is the requesting_container_uuid.
radhika [Thu, 16 Jun 2016 12:17:46 +0000 (08:17 -0400)]
9372: variable names and comments.
Brett Smith [Thu, 16 Jun 2016 02:47:05 +0000 (22:47 -0400)]
9408: Restore build-essential and ruby-dev to arvbox.
These were previously being pulled in as Recommends, and they're
necessary to build gems for the demo image.
Refs #9408.
Lucas Di Pentima [Thu, 16 Jun 2016 01:51:07 +0000 (22:51 -0300)]
Merge branch '9417-asserttrue-for-value-comparison'
Closes #9417
Lucas Di Pentima [Wed, 15 Jun 2016 19:27:23 +0000 (16:27 -0300)]
9417: Replaced assertTrue() with assertEqual() on RetryNeedsMultipleServices's tests
Lucas Di Pentima [Wed, 15 Jun 2016 21:45:10 +0000 (18:45 -0300)]
First try, don't think it'll work
Peter Amstutz [Wed, 15 Jun 2016 20:09:22 +0000 (16:09 -0400)]
8442: Generatefiles encode utf-8
Peter Amstutz [Wed, 15 Jun 2016 18:51:14 +0000 (14:51 -0400)]
8442: Submit CommandLineTool containers directly without an intermediate
workflow runner. Handle unsupported feature gracefully.
Peter Amstutz [Wed, 15 Jun 2016 15:23:47 +0000 (11:23 -0400)]
8442: Tweak internal handling of keep: paths, examine exit codes to determine
success/fail.
Peter Amstutz [Mon, 13 Jun 2016 20:42:37 +0000 (16:42 -0400)]
8442: more import fixups, import workflow file correctly.
Peter Amstutz [Mon, 13 Jun 2016 20:28:45 +0000 (16:28 -0400)]
8442: import fixups
Peter Amstutz [Mon, 13 Jun 2016 20:17:45 +0000 (16:17 -0400)]
8442: Debugging container --submit with crunch2
Peter Amstutz [Mon, 13 Jun 2016 19:45:29 +0000 (15:45 -0400)]
8442: Adding --submit support with --crunch2. General refactoring into more/smaller files.
Peter Amstutz [Mon, 13 Jun 2016 14:36:05 +0000 (10:36 -0400)]
8442: Fix message
Peter Amstutz [Thu, 9 Jun 2016 19:16:09 +0000 (15:16 -0400)]
8442: raise UnsupportedRequirement for unsupported features in the conformance
tests. Bump cwltool dependency.
Peter Amstutz [Thu, 9 Jun 2016 12:21:34 +0000 (08:21 -0400)]
8442: Setting up mount points works. Capturing output works.
Peter Amstutz [Wed, 8 Jun 2016 15:45:00 +0000 (11:45 -0400)]
8442: Return PDH for Docker container. Working on setting up mount points.
Peter Amstutz [Wed, 8 Jun 2016 15:15:58 +0000 (11:15 -0400)]
8442: Submit containers Work in progess.
Peter Amstutz [Mon, 6 Jun 2016 19:54:35 +0000 (15:54 -0400)]
8442: Add --crunch1/--crunch2 switch
Peter Amstutz [Mon, 6 Jun 2016 19:24:50 +0000 (15:24 -0400)]
8442: CWL create crunch2 containers WIP
Tom Clegg [Wed, 15 Jun 2016 19:40:58 +0000 (15:40 -0400)]
Merge branch '9395-keep-balance-page-size'
closes #9395
Tom Clegg [Wed, 15 Jun 2016 19:40:28 +0000 (15:40 -0400)]
9395: Fix encoding of integers in query params.
Tom Clegg [Wed, 15 Jun 2016 14:53:01 +0000 (10:53 -0400)]
9395: Explain CollectionBatchSize and CollectionBuffers in -help message.
Tom Clegg [Mon, 13 Jun 2016 14:31:39 +0000 (10:31 -0400)]
9395: Add CollectionBuffers config to keep-balance.
Tom Clegg [Mon, 13 Jun 2016 14:30:23 +0000 (10:30 -0400)]
9395: Add CollectionBatchSize config to keep-balance.
Peter Amstutz [Wed, 15 Jun 2016 19:38:34 +0000 (15:38 -0400)]
Add missing arvbox package "sudo" refs #9408
Peter Amstutz [Wed, 15 Jun 2016 19:15:50 +0000 (15:15 -0400)]
Use "less -R" in "arvbox log" so that color codes from api server logs display
properly. no issue #
Brett Smith [Wed, 15 Jun 2016 18:31:38 +0000 (14:31 -0400)]
Merge branch '9309-centos-7-install-guide-wip'
Refs #9309. Closes #9368.
Brett Smith [Wed, 8 Jun 2016 21:28:24 +0000 (17:28 -0400)]
9309: Add CentOS 7 to Install Guide.
Peter Amstutz [Wed, 15 Jun 2016 18:02:39 +0000 (14:02 -0400)]
Add missing package 'less'. Also reduce crunch-dispatch-local poll interval to 3 seconds. no issue #
radhika [Wed, 15 Jun 2016 17:39:52 +0000 (13:39 -0400)]
Merge branch 'master' into 9372-container-display
Lucas Di Pentima [Wed, 15 Jun 2016 17:35:34 +0000 (14:35 -0300)]
Merge branch '9408-arvbox-dockerfile-pulls-unnecessary-packages'
refs #9408
radhika [Wed, 15 Jun 2016 17:27:43 +0000 (13:27 -0400)]
9372: Corrected dashboard display to list container_requests, not containers as implementd in #9318.
Brett Smith [Wed, 15 Jun 2016 17:04:37 +0000 (13:04 -0400)]
9309: Bump Rails package iteration.
To accommodate systemd support in the package scripts.
Refs #9309.
Brett Smith [Wed, 15 Jun 2016 16:58:31 +0000 (12:58 -0400)]
Package build directories get same owner as $WORKSPACE.
Since this script runs as root inside the Docker container, the
directory created on the previous line is normally owned by root too.
This causes trouble when the package tests run later and try to create
repository information inside the directory. See, e.g.,
<https://ci.curoverse.com/job/build-packages-sso/23/consoleFull> for a
build that failed because of this.
No issue #.
Brett Smith [Wed, 15 Jun 2016 15:52:00 +0000 (11:52 -0400)]
Merge branch '9309-centos-7-rails-scripts-wip'
Refs #9309. Closes #9351, #9345.
Brett Smith [Fri, 3 Jun 2016 19:34:25 +0000 (15:34 -0400)]
9345: Use same script to test all Rails packages.
We want to simplify the SSO test script for a variety of reasons. We
don't want the package to depend on the PostgreSQL server; testing
proper PostgreSQL server configuration is a little out of scope for
this; and it's difficult to execute the test in a CentOS 7 Docker
image. This commit has the SSO server follow the same test strategy
as other Rails servers.
Brett Smith [Fri, 3 Jun 2016 18:50:59 +0000 (14:50 -0400)]
9309: Add systemd support to Rails package scripts.
Lucas Di Pentima [Wed, 15 Jun 2016 13:58:02 +0000 (10:58 -0300)]
9408: Added apt-get flag to avoid installing 'Recommended:' packages, it's better to have an explicit dependency list. Also, added bzip2 that was being pulled implicitly and it's needed.
Tom Clegg [Wed, 15 Jun 2016 13:51:55 +0000 (09:51 -0400)]
Merge branch '9374-go-sdk'
refs #9374
Tom Clegg [Wed, 15 Jun 2016 13:47:59 +0000 (09:47 -0400)]
9374: Remove unused receivers.
Tom Clegg [Tue, 14 Jun 2016 14:30:06 +0000 (10:30 -0400)]
9374: Fix uses of "this" as receiver name.
Tom Clegg [Tue, 14 Jun 2016 14:24:21 +0000 (10:24 -0400)]
9374: Use arvados.Collection instead of own CollectionRecord.
Tom Clegg [Mon, 13 Jun 2016 19:03:50 +0000 (15:03 -0400)]
9374: Propagate API transaction error details from response body to caller.
Tom Clegg [Fri, 10 Jun 2016 03:18:24 +0000 (23:18 -0400)]
9374: Add arvados.APIClientAuthorization.
Tom Clegg [Fri, 10 Jun 2016 03:04:50 +0000 (23:04 -0400)]
9374: Consolidate various Container structs as arvados.Container.
radhika [Tue, 14 Jun 2016 22:11:29 +0000 (18:11 -0400)]
9372: name and description can be edited on containers and container_requests despite the state.
Peter Amstutz [Tue, 14 Jun 2016 20:24:55 +0000 (16:24 -0400)]
Merge branch '9388-websocket-every-notify' closes #9388
Peter Amstutz [Tue, 14 Jun 2016 20:17:14 +0000 (16:17 -0400)]
Merge branch '8442-crunch-run-enable-net' refs #8442
Peter Amstutz [Tue, 14 Jun 2016 19:53:21 +0000 (15:53 -0400)]
9388: Reset sent_ids for consistency
Peter Amstutz [Tue, 14 Jun 2016 19:48:08 +0000 (15:48 -0400)]
9388: Add ws.notify_queue to ensure that notifies occurring during "catch up"
will get delivered in the correct order after catch up completes.
Peter Amstutz [Tue, 14 Jun 2016 17:28:32 +0000 (13:28 -0400)]
9388: Only record log ids sent during "catch up" phase, to avoid duplicates.
radhika [Tue, 14 Jun 2016 17:26:21 +0000 (13:26 -0400)]
9372: test cancel button for jobs and container_requests.
radhika [Tue, 14 Jun 2016 14:15:17 +0000 (10:15 -0400)]
Merge branch 'master' into 9372-container-display
radhika [Tue, 14 Jun 2016 14:14:18 +0000 (10:14 -0400)]
closes #9274
Merge branch '9274-requesting-container-uuid'
Peter Amstutz [Tue, 14 Jun 2016 13:57:54 +0000 (09:57 -0400)]
9353: Don't retry all BaseHTTPError, only retry InvalidInstanceID.NotFound.
Brett Smith [Tue, 14 Jun 2016 13:47:53 +0000 (09:47 -0400)]
Merge branch '9370-crunch2-packages-wip'
Closes #9370, #9382.
Brett Smith [Fri, 10 Jun 2016 20:02:50 +0000 (16:02 -0400)]
9370: Package Crunch v2 dispatch services.
Brett Smith [Fri, 10 Jun 2016 19:57:05 +0000 (15:57 -0400)]
9370: Alphabetize the package_go_binary list.
radhika [Tue, 14 Jun 2016 03:10:51 +0000 (23:10 -0400)]
9372: container display
Peter Amstutz [Mon, 13 Jun 2016 20:31:49 +0000 (16:31 -0400)]
8442: Enable net when API is enabled.