arvados.git
8 years ago9463: Using finally: on except: clause to stop the updater thread on error or finishe...
Lucas Di Pentima [Tue, 19 Jul 2016 15:07:22 +0000 (12:07 -0300)]
9463: Using finally: on except: clause to stop the updater thread on error or finished work. When counting the collection size, stopped ignoring items that are not Collection or ArvadosFile, in case some other class is used in the future. When getting block locators (for example when called with --raw), don\'t make sure the blocks are flushed to Keep before asking for segments

8 years ago9463: blank spaces cleanup
Lucas Di Pentima [Thu, 14 Jul 2016 20:53:18 +0000 (17:53 -0300)]
9463: blank spaces cleanup

8 years ago9463: New approach on this arv-put revamping: the ArvPutUploadJob class writes to...
Lucas Di Pentima [Thu, 14 Jul 2016 20:48:31 +0000 (17:48 -0300)]
9463: New approach on this arv-put revamping: the ArvPutUploadJob class writes to a Collection and start a thread to periodically poll the upload status

8 years ago9463: Finished first draft on arv-put command use of Collection class. Also, partial...
Lucas Di Pentima [Fri, 8 Jul 2016 22:23:54 +0000 (19:23 -0300)]
9463: Finished first draft on arv-put command use of Collection class. Also, partial uploads resuming is working. Tests were written to validate this. Refs #9463 #8910

8 years ago9463: Fixed some integration tests broken because of the replication_desired addition...
Lucas Di Pentima [Thu, 7 Jul 2016 21:32:50 +0000 (18:32 -0300)]
9463: Fixed some integration tests broken because of the replication_desired addition to Collection

8 years ago9463: Added to Collection a way to ask _BlockManager how many copies need to be writt...
Lucas Di Pentima [Thu, 7 Jul 2016 18:44:16 +0000 (15:44 -0300)]
9463: Added to Collection a way to ask _BlockManager how many copies need to be written when uploading blocks to Keep

8 years ago9463: Added desired replication level to collection record on API Server when using...
Lucas Di Pentima [Thu, 7 Jul 2016 16:19:24 +0000 (13:19 -0300)]
9463: Added desired replication level to collection record on API Server when using Collection class

8 years ago9463: Polishing the last details to make the integration tests work ok
Lucas Di Pentima [Thu, 7 Jul 2016 03:20:58 +0000 (00:20 -0300)]
9463: Polishing the last details to make the integration tests work ok

8 years ago9463: Lots of progress today, resume upload code written, many tests to do!
Lucas Di Pentima [Tue, 5 Jul 2016 21:53:37 +0000 (18:53 -0300)]
9463: Lots of progress today, resume upload code written, many tests to do!

8 years ago9463: Resume cache implementation (WIP)
Lucas Di Pentima [Mon, 4 Jul 2016 23:07:56 +0000 (20:07 -0300)]
9463: Resume cache implementation (WIP)

8 years ago9463: Initial coding with tests
Lucas Di Pentima [Fri, 1 Jul 2016 22:05:18 +0000 (19:05 -0300)]
9463: Initial coding with tests

8 years ago9488: Add log message to notify users on sucessful websocket reconnect. refs #9488
Peter Amstutz [Fri, 24 Jun 2016 14:01:31 +0000 (10:01 -0400)]
9488: Add log message to notify users on sucessful websocket reconnect. refs #9488

8 years agoMerge branch '9486-filter-nodes'
Tom Clegg [Fri, 24 Jun 2016 14:59:33 +0000 (10:59 -0400)]
Merge branch '9486-filter-nodes'

refs #9486

8 years ago9486: Use API to filter nodes by last_ping_at.
Tom Clegg [Thu, 23 Jun 2016 16:47:49 +0000 (12:47 -0400)]
9486: Use API to filter nodes by last_ping_at.

8 years agoUpdate run-build-packages versions for schema_salad and cwltool, refs #8442
Peter Amstutz [Wed, 22 Jun 2016 16:18:49 +0000 (12:18 -0400)]
Update run-build-packages versions for schema_salad and cwltool, refs #8442

8 years agoMerge branch 'origin-8442-cwl-crunch2' closes #8442
Peter Amstutz [Wed, 22 Jun 2016 15:44:03 +0000 (11:44 -0400)]
Merge branch 'origin-8442-cwl-crunch2' closes #8442

8 years agoMerge branch 'master' into origin-8442-cwl-crunch2
Peter Amstutz [Wed, 22 Jun 2016 13:57:58 +0000 (09:57 -0400)]
Merge branch 'master' into origin-8442-cwl-crunch2

8 years ago8442: Code cleanup. Remove dead code. Update comments.
Peter Amstutz [Wed, 22 Jun 2016 13:55:12 +0000 (09:55 -0400)]
8442: Code cleanup.  Remove dead code.  Update comments.

8 years agoPySDK: Fix spelling of X-Keep-Desired-Replicas header.
Brett Smith [Wed, 22 Jun 2016 13:46:40 +0000 (09:46 -0400)]
PySDK: Fix spelling of X-Keep-Desired-Replicas header.

This matches what Keep services expect, and other clients use.
No issue #.

8 years ago8442: Fix error reporting on missing Docker image
Peter Amstutz [Tue, 21 Jun 2016 20:10:46 +0000 (16:10 -0400)]
8442: Fix error reporting on missing Docker image

8 years agoIncrease timeout on failing websocket test. refs #9427
Peter Amstutz [Tue, 21 Jun 2016 19:57:23 +0000 (15:57 -0400)]
Increase timeout on failing websocket test.  refs #9427

8 years agoMerge branch '9427-threaded-websockets' closes #9427
Peter Amstutz [Tue, 21 Jun 2016 16:20:48 +0000 (12:20 -0400)]
Merge branch '9427-threaded-websockets' closes #9427

8 years ago8442: Make work_api parameter to ArvCwlRunner constructor optional.
Peter Amstutz [Tue, 21 Jun 2016 16:16:23 +0000 (12:16 -0400)]
8442: Make work_api parameter to ArvCwlRunner constructor optional.

8 years ago8442: cwl-runner crunch script must provide work_api parameter to ArvCwlRunner
Peter Amstutz [Tue, 21 Jun 2016 16:01:57 +0000 (12:01 -0400)]
8442: cwl-runner crunch script must provide work_api parameter to ArvCwlRunner
constructor.

8 years ago8442: Bugfixes from end-to-end testing.
Peter Amstutz [Tue, 21 Jun 2016 15:55:43 +0000 (11:55 -0400)]
8442: Bugfixes from end-to-end testing.

8 years ago9427: Back out redundant connection pool configuration option and add comment
Peter Amstutz [Tue, 21 Jun 2016 14:56:57 +0000 (10:56 -0400)]
9427: Back out redundant connection pool configuration option and add comment
to database.yml.example about connection pool and websockets.

8 years ago8442: Fix tests. Update comments. Rename jobs -> processes since it doesn't hold...
Peter Amstutz [Tue, 21 Jun 2016 12:22:54 +0000 (08:22 -0400)]
8442: Fix tests.  Update comments.  Rename jobs -> processes since it doesn't hold just "jobs" any more.

8 years ago8442: Rename --crunch1 and --crunch2 options to --api=jobs and --api=containers
Peter Amstutz [Mon, 20 Jun 2016 21:34:07 +0000 (17:34 -0400)]
8442: Rename --crunch1 and --crunch2 options to --api=jobs and --api=containers

8 years agoMerge branch '9180-avoid-overreplication-keepclient'
Lucas Di Pentima [Mon, 20 Jun 2016 19:59:38 +0000 (16:59 -0300)]
Merge branch '9180-avoid-overreplication-keepclient'

Closes #9180

8 years agoMerge branch '9353-retry-http-error' closes #9353
Peter Amstutz [Mon, 20 Jun 2016 18:46:54 +0000 (14:46 -0400)]
Merge branch '9353-retry-http-error' closes #9353

8 years ago9427: Add limits for connections, subscriptions, queued notifications, and
Peter Amstutz [Mon, 20 Jun 2016 18:29:21 +0000 (14:29 -0400)]
9427: Add limits for connections, subscriptions, queued notifications, and
database connections to configuration.  Check notify queue size on push instead
of on pop and close connection immediately.

8 years agoAdd missing arvbox package "virtualenv" refs #9408
Peter Amstutz [Mon, 20 Jun 2016 13:35:21 +0000 (09:35 -0400)]
Add missing arvbox package "virtualenv" refs #9408

8 years ago9427: Add connection limit.
Peter Amstutz [Fri, 17 Jun 2016 20:41:03 +0000 (16:41 -0400)]
9427: Add connection limit.

8 years agoBugfix "arv-ws --id" option, refs #9388.
Peter Amstutz [Fri, 17 Jun 2016 20:27:18 +0000 (16:27 -0400)]
Bugfix "arv-ws --id" option, refs #9388.

8 years ago9427: Websockets now uses a thread per connection for database query and
Peter Amstutz [Fri, 17 Jun 2016 20:12:15 +0000 (16:12 -0400)]
9427: Websockets now uses a thread per connection for database query and
sending outgoing messages.  This minimizes interference by slow clients or
clients that are "replaying the world" with a last_log_id far in the past.

8 years agocloses #9372
radhika [Fri, 17 Jun 2016 15:13:28 +0000 (11:13 -0400)]
closes #9372
Merge branch '9372-container-display'

8 years agoMerge branch 'master' into 9372-container-display
radhika [Fri, 17 Jun 2016 15:13:12 +0000 (11:13 -0400)]
Merge branch 'master' into 9372-container-display

8 years ago9372: improve container_work_unit -> uri method to check if @proxied will respond_to...
radhika [Fri, 17 Jun 2016 15:11:48 +0000 (11:11 -0400)]
9372: improve container_work_unit -> uri method to check if @proxied will respond_to table_name instead of using a rescue.

8 years ago9180: Changed some of the logic on ThreadLimiter and made unit tests to validate...
Lucas Di Pentima [Thu, 16 Jun 2016 23:06:21 +0000 (20:06 -0300)]
9180: Changed some of the logic on ThreadLimiter and made unit tests to validate the new behaviour
refs #9180

8 years ago9372: The children of a Container will be ContainerRequests to which this is the...
radhika [Thu, 16 Jun 2016 14:32:21 +0000 (10:32 -0400)]
9372: The children of a Container will be ContainerRequests to which this is the requesting_container_uuid.

8 years ago9372: variable names and comments.
radhika [Thu, 16 Jun 2016 12:17:46 +0000 (08:17 -0400)]
9372: variable names and comments.

8 years ago9408: Restore build-essential and ruby-dev to arvbox.
Brett Smith [Thu, 16 Jun 2016 02:47:05 +0000 (22:47 -0400)]
9408: Restore build-essential and ruby-dev to arvbox.

These were previously being pulled in as Recommends, and they're
necessary to build gems for the demo image.
Refs #9408.

8 years agoMerge branch '9417-asserttrue-for-value-comparison'
Lucas Di Pentima [Thu, 16 Jun 2016 01:51:07 +0000 (22:51 -0300)]
Merge branch '9417-asserttrue-for-value-comparison'

Closes #9417

8 years ago9417: Replaced assertTrue() with assertEqual() on RetryNeedsMultipleServices's tests
Lucas Di Pentima [Wed, 15 Jun 2016 19:27:23 +0000 (16:27 -0300)]
9417: Replaced assertTrue() with assertEqual() on RetryNeedsMultipleServices's tests

8 years agoFirst try, don't think it'll work
Lucas Di Pentima [Wed, 15 Jun 2016 21:45:10 +0000 (18:45 -0300)]
First try, don't think it'll work

8 years ago8442: Generatefiles encode utf-8
Peter Amstutz [Wed, 15 Jun 2016 20:09:22 +0000 (16:09 -0400)]
8442: Generatefiles encode utf-8

8 years ago8442: Submit CommandLineTool containers directly without an intermediate
Peter Amstutz [Wed, 15 Jun 2016 18:51:14 +0000 (14:51 -0400)]
8442: Submit CommandLineTool containers directly without an intermediate
workflow runner.  Handle unsupported feature gracefully.

8 years ago8442: Tweak internal handling of keep: paths, examine exit codes to determine
Peter Amstutz [Wed, 15 Jun 2016 15:23:47 +0000 (11:23 -0400)]
8442: Tweak internal handling of keep: paths, examine exit codes to determine
success/fail.

8 years ago8442: more import fixups, import workflow file correctly.
Peter Amstutz [Mon, 13 Jun 2016 20:42:37 +0000 (16:42 -0400)]
8442: more import fixups, import workflow file correctly.

8 years ago8442: import fixups
Peter Amstutz [Mon, 13 Jun 2016 20:28:45 +0000 (16:28 -0400)]
8442: import fixups

8 years ago8442: Debugging container --submit with crunch2
Peter Amstutz [Mon, 13 Jun 2016 20:17:45 +0000 (16:17 -0400)]
8442: Debugging container --submit with crunch2

8 years ago8442: Adding --submit support with --crunch2. General refactoring into more/smaller...
Peter Amstutz [Mon, 13 Jun 2016 19:45:29 +0000 (15:45 -0400)]
8442: Adding --submit support with --crunch2.  General refactoring into more/smaller files.

8 years ago8442: Fix message
Peter Amstutz [Mon, 13 Jun 2016 14:36:05 +0000 (10:36 -0400)]
8442: Fix message

8 years ago8442: raise UnsupportedRequirement for unsupported features in the conformance
Peter Amstutz [Thu, 9 Jun 2016 19:16:09 +0000 (15:16 -0400)]
8442: raise UnsupportedRequirement for unsupported features in the conformance
tests.  Bump cwltool dependency.

8 years ago8442: Setting up mount points works. Capturing output works.
Peter Amstutz [Thu, 9 Jun 2016 12:21:34 +0000 (08:21 -0400)]
8442: Setting up mount points works.  Capturing output works.

8 years ago8442: Return PDH for Docker container. Working on setting up mount points.
Peter Amstutz [Wed, 8 Jun 2016 15:45:00 +0000 (11:45 -0400)]
8442: Return PDH for Docker container.  Working on setting up mount points.

8 years ago8442: Submit containers Work in progess.
Peter Amstutz [Wed, 8 Jun 2016 15:15:58 +0000 (11:15 -0400)]
8442: Submit containers Work in progess.

8 years ago8442: Add --crunch1/--crunch2 switch
Peter Amstutz [Mon, 6 Jun 2016 19:54:35 +0000 (15:54 -0400)]
8442: Add --crunch1/--crunch2 switch

8 years ago8442: CWL create crunch2 containers WIP
Peter Amstutz [Mon, 6 Jun 2016 19:24:50 +0000 (15:24 -0400)]
8442: CWL create crunch2 containers WIP

8 years agoMerge branch '9395-keep-balance-page-size'
Tom Clegg [Wed, 15 Jun 2016 19:40:58 +0000 (15:40 -0400)]
Merge branch '9395-keep-balance-page-size'

closes #9395

8 years ago9395: Fix encoding of integers in query params.
Tom Clegg [Wed, 15 Jun 2016 19:40:28 +0000 (15:40 -0400)]
9395: Fix encoding of integers in query params.

8 years ago9395: Explain CollectionBatchSize and CollectionBuffers in -help message.
Tom Clegg [Wed, 15 Jun 2016 14:53:01 +0000 (10:53 -0400)]
9395: Explain CollectionBatchSize and CollectionBuffers in -help message.

8 years ago9395: Add CollectionBuffers config to keep-balance.
Tom Clegg [Mon, 13 Jun 2016 14:31:39 +0000 (10:31 -0400)]
9395: Add CollectionBuffers config to keep-balance.

8 years ago9395: Add CollectionBatchSize config to keep-balance.
Tom Clegg [Mon, 13 Jun 2016 14:30:23 +0000 (10:30 -0400)]
9395: Add CollectionBatchSize config to keep-balance.

8 years agoAdd missing arvbox package "sudo" refs #9408
Peter Amstutz [Wed, 15 Jun 2016 19:38:34 +0000 (15:38 -0400)]
Add missing arvbox package "sudo" refs #9408

8 years agoUse "less -R" in "arvbox log" so that color codes from api server logs display
Peter Amstutz [Wed, 15 Jun 2016 19:15:50 +0000 (15:15 -0400)]
Use "less -R" in "arvbox log" so that color codes from api server logs display
properly.  no issue #

8 years agoMerge branch '9309-centos-7-install-guide-wip'
Brett Smith [Wed, 15 Jun 2016 18:31:38 +0000 (14:31 -0400)]
Merge branch '9309-centos-7-install-guide-wip'

Refs #9309.  Closes #9368.

8 years ago9309: Add CentOS 7 to Install Guide.
Brett Smith [Wed, 8 Jun 2016 21:28:24 +0000 (17:28 -0400)]
9309: Add CentOS 7 to Install Guide.

8 years agoAdd missing package 'less'. Also reduce crunch-dispatch-local poll interval to 3...
Peter Amstutz [Wed, 15 Jun 2016 18:02:39 +0000 (14:02 -0400)]
Add missing package 'less'.  Also reduce crunch-dispatch-local poll interval to 3 seconds.  no issue #

8 years agoMerge branch 'master' into 9372-container-display
radhika [Wed, 15 Jun 2016 17:39:52 +0000 (13:39 -0400)]
Merge branch 'master' into 9372-container-display

8 years agoMerge branch '9408-arvbox-dockerfile-pulls-unnecessary-packages'
Lucas Di Pentima [Wed, 15 Jun 2016 17:35:34 +0000 (14:35 -0300)]
Merge branch '9408-arvbox-dockerfile-pulls-unnecessary-packages'

refs #9408

8 years ago9372: Corrected dashboard display to list container_requests, not containers as imple...
radhika [Wed, 15 Jun 2016 17:27:43 +0000 (13:27 -0400)]
9372: Corrected dashboard display to list container_requests, not containers as implementd in #9318.

8 years ago9309: Bump Rails package iteration.
Brett Smith [Wed, 15 Jun 2016 17:04:37 +0000 (13:04 -0400)]
9309: Bump Rails package iteration.

To accommodate systemd support in the package scripts.
Refs #9309.

8 years agoPackage build directories get same owner as $WORKSPACE.
Brett Smith [Wed, 15 Jun 2016 16:58:31 +0000 (12:58 -0400)]
Package build directories get same owner as $WORKSPACE.

Since this script runs as root inside the Docker container, the
directory created on the previous line is normally owned by root too.
This causes trouble when the package tests run later and try to create
repository information inside the directory.  See, e.g.,
<https://ci.curoverse.com/job/build-packages-sso/23/consoleFull> for a
build that failed because of this.

No issue #.

8 years agoMerge branch '9309-centos-7-rails-scripts-wip'
Brett Smith [Wed, 15 Jun 2016 15:52:00 +0000 (11:52 -0400)]
Merge branch '9309-centos-7-rails-scripts-wip'

Refs #9309.  Closes #9351, #9345.

8 years ago9345: Use same script to test all Rails packages.
Brett Smith [Fri, 3 Jun 2016 19:34:25 +0000 (15:34 -0400)]
9345: Use same script to test all Rails packages.

We want to simplify the SSO test script for a variety of reasons.  We
don't want the package to depend on the PostgreSQL server; testing
proper PostgreSQL server configuration is a little out of scope for
this; and it's difficult to execute the test in a CentOS 7 Docker
image.  This commit has the SSO server follow the same test strategy
as other Rails servers.

8 years ago9309: Add systemd support to Rails package scripts.
Brett Smith [Fri, 3 Jun 2016 18:50:59 +0000 (14:50 -0400)]
9309: Add systemd support to Rails package scripts.

8 years ago9408: Added apt-get flag to avoid installing 'Recommended:' packages, it's better...
Lucas Di Pentima [Wed, 15 Jun 2016 13:58:02 +0000 (10:58 -0300)]
9408: Added apt-get flag to avoid installing 'Recommended:' packages, it's better to have an explicit dependency list. Also, added bzip2 that was being pulled implicitly and it's needed.

8 years agoMerge branch '9374-go-sdk'
Tom Clegg [Wed, 15 Jun 2016 13:51:55 +0000 (09:51 -0400)]
Merge branch '9374-go-sdk'

refs #9374

8 years ago9374: Remove unused receivers.
Tom Clegg [Wed, 15 Jun 2016 13:47:59 +0000 (09:47 -0400)]
9374: Remove unused receivers.

8 years ago9374: Fix uses of "this" as receiver name.
Tom Clegg [Tue, 14 Jun 2016 14:30:06 +0000 (10:30 -0400)]
9374: Fix uses of "this" as receiver name.

8 years ago9374: Use arvados.Collection instead of own CollectionRecord.
Tom Clegg [Tue, 14 Jun 2016 14:24:21 +0000 (10:24 -0400)]
9374: Use arvados.Collection instead of own CollectionRecord.

8 years ago9374: Propagate API transaction error details from response body to caller.
Tom Clegg [Mon, 13 Jun 2016 19:03:50 +0000 (15:03 -0400)]
9374: Propagate API transaction error details from response body to caller.

8 years ago9374: Add arvados.APIClientAuthorization.
Tom Clegg [Fri, 10 Jun 2016 03:18:24 +0000 (23:18 -0400)]
9374: Add arvados.APIClientAuthorization.

8 years ago9374: Consolidate various Container structs as arvados.Container.
Tom Clegg [Fri, 10 Jun 2016 03:04:50 +0000 (23:04 -0400)]
9374: Consolidate various Container structs as arvados.Container.

8 years ago9372: name and description can be edited on containers and container_requests despite...
radhika [Tue, 14 Jun 2016 22:11:29 +0000 (18:11 -0400)]
9372: name and description can be edited on containers and container_requests despite the state.

8 years agoMerge branch '9388-websocket-every-notify' closes #9388
Peter Amstutz [Tue, 14 Jun 2016 20:24:55 +0000 (16:24 -0400)]
Merge branch '9388-websocket-every-notify' closes #9388

8 years agoMerge branch '8442-crunch-run-enable-net' refs #8442
Peter Amstutz [Tue, 14 Jun 2016 20:17:14 +0000 (16:17 -0400)]
Merge branch '8442-crunch-run-enable-net' refs #8442

8 years ago9388: Reset sent_ids for consistency
Peter Amstutz [Tue, 14 Jun 2016 19:53:21 +0000 (15:53 -0400)]
9388: Reset sent_ids for consistency

8 years ago9388: Add ws.notify_queue to ensure that notifies occurring during "catch up"
Peter Amstutz [Tue, 14 Jun 2016 19:48:08 +0000 (15:48 -0400)]
9388: Add ws.notify_queue to ensure that notifies occurring during "catch up"
will get delivered in the correct order after catch up completes.

8 years ago9388: Only record log ids sent during "catch up" phase, to avoid duplicates.
Peter Amstutz [Tue, 14 Jun 2016 17:28:32 +0000 (13:28 -0400)]
9388: Only record log ids sent during "catch up" phase, to avoid duplicates.

8 years ago9372: test cancel button for jobs and container_requests.
radhika [Tue, 14 Jun 2016 17:26:21 +0000 (13:26 -0400)]
9372: test cancel button for jobs and container_requests.

8 years agoMerge branch 'master' into 9372-container-display
radhika [Tue, 14 Jun 2016 14:15:17 +0000 (10:15 -0400)]
Merge branch 'master' into 9372-container-display

8 years agocloses #9274
radhika [Tue, 14 Jun 2016 14:14:18 +0000 (10:14 -0400)]
closes #9274
Merge branch '9274-requesting-container-uuid'

8 years ago9353: Don't retry all BaseHTTPError, only retry InvalidInstanceID.NotFound.
Peter Amstutz [Tue, 14 Jun 2016 13:57:54 +0000 (09:57 -0400)]
9353: Don't retry all BaseHTTPError, only retry InvalidInstanceID.NotFound.

8 years agoMerge branch '9370-crunch2-packages-wip'
Brett Smith [Tue, 14 Jun 2016 13:47:53 +0000 (09:47 -0400)]
Merge branch '9370-crunch2-packages-wip'

Closes #9370, #9382.

8 years ago9370: Package Crunch v2 dispatch services.
Brett Smith [Fri, 10 Jun 2016 20:02:50 +0000 (16:02 -0400)]
9370: Package Crunch v2 dispatch services.

8 years ago9370: Alphabetize the package_go_binary list.
Brett Smith [Fri, 10 Jun 2016 19:57:05 +0000 (15:57 -0400)]
9370: Alphabetize the package_go_binary list.

8 years ago9372: container display
radhika [Tue, 14 Jun 2016 03:10:51 +0000 (23:10 -0400)]
9372: container display

8 years ago8442: Enable net when API is enabled.
Peter Amstutz [Mon, 13 Jun 2016 20:31:49 +0000 (16:31 -0400)]
8442: Enable net when API is enabled.