arvados.git
7 years ago9459: arvbox-demo accepts version arguments for Git checkouts. 9459-arvbox-demo-version-arg-wip
Brett Smith [Thu, 30 Jun 2016 21:45:12 +0000 (17:45 -0400)]
9459: arvbox-demo accepts version arguments for Git checkouts.

7 years agoUpdate arvados-cwl-runner cwltool pin to get ruamel.yaml fix (c8f584840591f3a4a0fad3b...
Peter Amstutz [Thu, 30 Jun 2016 17:24:12 +0000 (13:24 -0400)]
Update arvados-cwl-runner cwltool pin to get ruamel.yaml fix (c8f584840591f3a4a0fad3bce6b39f8866cd9dad) refs #9458

7 years agoMerge branch '9513-keepproxy-content-length'
Tom Clegg [Wed, 29 Jun 2016 20:11:41 +0000 (16:11 -0400)]
Merge branch '9513-keepproxy-content-length'

closes #9513

7 years ago9513: Do not set response Content-Length to -1 when returning 411 Length Required.
Tom Clegg [Wed, 29 Jun 2016 17:41:57 +0000 (13:41 -0400)]
9513: Do not set response Content-Length to -1 when returning 411 Length Required.

Add tests for requests with bad/missing Content-Length headers.

7 years agoMerge branch '9518-username-conflict'
Tom Clegg [Wed, 29 Jun 2016 19:55:11 +0000 (15:55 -0400)]
Merge branch '9518-username-conflict'

closes #9518

7 years ago9518: Fix username conflict resolution.
Tom Clegg [Wed, 29 Jun 2016 19:53:27 +0000 (15:53 -0400)]
9518: Fix username conflict resolution.

7 years agocloses #9407
radhika [Wed, 29 Jun 2016 18:03:55 +0000 (14:03 -0400)]
closes #9407
Merge branch '9407-container-log-viewer'

7 years ago9407: container log view
radhika [Fri, 24 Jun 2016 21:16:12 +0000 (17:16 -0400)]
9407: container log view

7 years agoMerge branch '9511-cwl-type-checking' closes #9511
Peter Amstutz [Wed, 29 Jun 2016 17:13:02 +0000 (13:13 -0400)]
Merge branch '9511-cwl-type-checking' closes #9511

7 years ago9511: Bump cwltool dependency to get bugfix (81ff56f239de3ff0d17c8dec98d1a40f1e41cb7d...
Peter Amstutz [Wed, 29 Jun 2016 17:04:47 +0000 (13:04 -0400)]
9511: Bump cwltool dependency to get bugfix (81ff56f239de3ff0d17c8dec98d1a40f1e41cb7d) affecting bcbio run.

7 years agoMerge branch '9442-cwl-runtime-outdir' closes #9442
Peter Amstutz [Tue, 28 Jun 2016 19:50:02 +0000 (15:50 -0400)]
Merge branch '9442-cwl-runtime-outdir' closes #9442

7 years ago9442: Incorporate check for correct $(runtime.outdir) into tests.
Peter Amstutz [Tue, 28 Jun 2016 17:25:13 +0000 (13:25 -0400)]
9442: Incorporate check for correct $(runtime.outdir) into tests.

7 years ago9442: Make cwl-runner crunch script backwards compatible (note-6)
Peter Amstutz [Tue, 28 Jun 2016 17:15:20 +0000 (13:15 -0400)]
9442: Make cwl-runner crunch script backwards compatible (note-6)

7 years ago9442: Fix runtime.outdir for containers.
Peter Amstutz [Tue, 28 Jun 2016 17:15:02 +0000 (13:15 -0400)]
9442: Fix runtime.outdir for containers.

7 years ago9442: Override kwargs["outdir"] so that $(runtime.outdir) is set correctly.
Peter Amstutz [Tue, 28 Jun 2016 14:03:30 +0000 (10:03 -0400)]
9442: Override kwargs["outdir"] so that $(runtime.outdir) is set correctly.

7 years agoMerge branch '9446-refactor-keep-parallel-write-strategy'
Lucas Di Pentima [Mon, 27 Jun 2016 19:56:43 +0000 (16:56 -0300)]
Merge branch '9446-refactor-keep-parallel-write-strategy'

9446: PySDK Keep concurrent write refactoring. Simplified thread management using a thread pool and a queue.
ThreadLimiter not needed anymore, so related tests were rewritten for the new code.
Closes #9446

7 years agoMerge branch '9491-desired-replicas'
Tom Clegg [Mon, 27 Jun 2016 19:13:32 +0000 (15:13 -0400)]
Merge branch '9491-desired-replicas'

closes #9491

7 years ago9491: Fix keepproxy support for X-Keep-Desired-Replicas header.
Tom Clegg [Sat, 25 Jun 2016 01:57:30 +0000 (21:57 -0400)]
9491: Fix keepproxy support for X-Keep-Desired-Replicas header.

7 years ago9491: Put*() return number of replicas actually stored, even if we ended up storing...
Tom Clegg [Sat, 25 Jun 2016 01:55:16 +0000 (21:55 -0400)]
9491: Put*() return number of replicas actually stored, even if we ended up storing more than requested.

7 years agoMerge branch '8470-resolve-container-req'
Tom Clegg [Mon, 27 Jun 2016 19:10:31 +0000 (15:10 -0400)]
Merge branch '8470-resolve-container-req'

closes #8470

7 years ago8470: Resolve docker image hash or tag to collection PDH when creating a Container.
Tom Clegg [Fri, 24 Jun 2016 17:55:08 +0000 (13:55 -0400)]
8470: Resolve docker image hash or tag to collection PDH when creating a Container.

7 years ago8470: Clean up ContainerRequest tests.
Tom Clegg [Fri, 24 Jun 2016 02:28:50 +0000 (22:28 -0400)]
8470: Clean up ContainerRequest tests.

7 years ago8470: Resolve mounts to PDH.
Tom Clegg [Fri, 24 Jun 2016 01:42:51 +0000 (21:42 -0400)]
8470: Resolve mounts to PDH.

7 years ago8470: Resolve runtime_constraints ranges to numbers when satisfying a Container Request.
Tom Clegg [Thu, 23 Jun 2016 18:16:45 +0000 (14:16 -0400)]
8470: Resolve runtime_constraints ranges to numbers when satisfying a Container Request.

7 years ago9446: Applying Peter's review suggestions. refs #9446
Lucas Di Pentima [Mon, 27 Jun 2016 18:36:17 +0000 (15:36 -0300)]
9446: Applying Peter's review suggestions. refs #9446

7 years agoPySDK put() refactoring ready, all local tests worked OK
Lucas Di Pentima [Fri, 24 Jun 2016 22:10:22 +0000 (19:10 -0300)]
PySDK put() refactoring ready, all local tests worked OK

7 years agoSome tests done on new code
Lucas Di Pentima [Fri, 24 Jun 2016 15:34:55 +0000 (12:34 -0300)]
Some tests done on new code

7 years ago9488: Add log message to notify users on sucessful websocket reconnect. refs #9488
Peter Amstutz [Fri, 24 Jun 2016 14:01:31 +0000 (10:01 -0400)]
9488: Add log message to notify users on sucessful websocket reconnect. refs #9488

7 years agoMerge branch '9486-filter-nodes'
Tom Clegg [Fri, 24 Jun 2016 14:59:33 +0000 (10:59 -0400)]
Merge branch '9486-filter-nodes'

refs #9486

7 years agoOne last method before calling it a day
Lucas Di Pentima [Thu, 23 Jun 2016 22:47:19 +0000 (19:47 -0300)]
One last method before calling it a day

7 years agoReady to start testing code
Lucas Di Pentima [Thu, 23 Jun 2016 22:38:23 +0000 (19:38 -0300)]
Ready to start testing code

7 years ago9486: Use API to filter nodes by last_ping_at.
Tom Clegg [Thu, 23 Jun 2016 16:47:49 +0000 (12:47 -0400)]
9486: Use API to filter nodes by last_ping_at.

7 years agoUpdate run-build-packages versions for schema_salad and cwltool, refs #8442
Peter Amstutz [Wed, 22 Jun 2016 16:18:49 +0000 (12:18 -0400)]
Update run-build-packages versions for schema_salad and cwltool, refs #8442

7 years agoMerge branch 'origin-8442-cwl-crunch2' closes #8442
Peter Amstutz [Wed, 22 Jun 2016 15:44:03 +0000 (11:44 -0400)]
Merge branch 'origin-8442-cwl-crunch2' closes #8442

7 years agoMerge branch 'master' into origin-8442-cwl-crunch2
Peter Amstutz [Wed, 22 Jun 2016 13:57:58 +0000 (09:57 -0400)]
Merge branch 'master' into origin-8442-cwl-crunch2

7 years ago8442: Code cleanup. Remove dead code. Update comments.
Peter Amstutz [Wed, 22 Jun 2016 13:55:12 +0000 (09:55 -0400)]
8442: Code cleanup.  Remove dead code.  Update comments.

7 years agoPySDK: Fix spelling of X-Keep-Desired-Replicas header.
Brett Smith [Wed, 22 Jun 2016 13:46:40 +0000 (09:46 -0400)]
PySDK: Fix spelling of X-Keep-Desired-Replicas header.

This matches what Keep services expect, and other clients use.
No issue #.

7 years ago8442: Fix error reporting on missing Docker image
Peter Amstutz [Tue, 21 Jun 2016 20:10:46 +0000 (16:10 -0400)]
8442: Fix error reporting on missing Docker image

7 years agoIncrease timeout on failing websocket test. refs #9427
Peter Amstutz [Tue, 21 Jun 2016 19:57:23 +0000 (15:57 -0400)]
Increase timeout on failing websocket test.  refs #9427

7 years agoMerge branch '9427-threaded-websockets' closes #9427
Peter Amstutz [Tue, 21 Jun 2016 16:20:48 +0000 (12:20 -0400)]
Merge branch '9427-threaded-websockets' closes #9427

7 years ago8442: Make work_api parameter to ArvCwlRunner constructor optional.
Peter Amstutz [Tue, 21 Jun 2016 16:16:23 +0000 (12:16 -0400)]
8442: Make work_api parameter to ArvCwlRunner constructor optional.

7 years ago8442: cwl-runner crunch script must provide work_api parameter to ArvCwlRunner
Peter Amstutz [Tue, 21 Jun 2016 16:01:57 +0000 (12:01 -0400)]
8442: cwl-runner crunch script must provide work_api parameter to ArvCwlRunner
constructor.

7 years ago8442: Bugfixes from end-to-end testing.
Peter Amstutz [Tue, 21 Jun 2016 15:55:43 +0000 (11:55 -0400)]
8442: Bugfixes from end-to-end testing.

7 years ago9427: Back out redundant connection pool configuration option and add comment
Peter Amstutz [Tue, 21 Jun 2016 14:56:57 +0000 (10:56 -0400)]
9427: Back out redundant connection pool configuration option and add comment
to database.yml.example about connection pool and websockets.

7 years ago8442: Fix tests. Update comments. Rename jobs -> processes since it doesn't hold...
Peter Amstutz [Tue, 21 Jun 2016 12:22:54 +0000 (08:22 -0400)]
8442: Fix tests.  Update comments.  Rename jobs -> processes since it doesn't hold just "jobs" any more.

7 years ago8442: Rename --crunch1 and --crunch2 options to --api=jobs and --api=containers
Peter Amstutz [Mon, 20 Jun 2016 21:34:07 +0000 (17:34 -0400)]
8442: Rename --crunch1 and --crunch2 options to --api=jobs and --api=containers

7 years agoMerge branch '9180-avoid-overreplication-keepclient'
Lucas Di Pentima [Mon, 20 Jun 2016 19:59:38 +0000 (16:59 -0300)]
Merge branch '9180-avoid-overreplication-keepclient'

Closes #9180

7 years agoMerge branch '9353-retry-http-error' closes #9353
Peter Amstutz [Mon, 20 Jun 2016 18:46:54 +0000 (14:46 -0400)]
Merge branch '9353-retry-http-error' closes #9353

7 years ago9427: Add limits for connections, subscriptions, queued notifications, and
Peter Amstutz [Mon, 20 Jun 2016 18:29:21 +0000 (14:29 -0400)]
9427: Add limits for connections, subscriptions, queued notifications, and
database connections to configuration.  Check notify queue size on push instead
of on pop and close connection immediately.

7 years agoAdd missing arvbox package "virtualenv" refs #9408
Peter Amstutz [Mon, 20 Jun 2016 13:35:21 +0000 (09:35 -0400)]
Add missing arvbox package "virtualenv" refs #9408

7 years ago9427: Add connection limit.
Peter Amstutz [Fri, 17 Jun 2016 20:41:03 +0000 (16:41 -0400)]
9427: Add connection limit.

7 years agoBugfix "arv-ws --id" option, refs #9388.
Peter Amstutz [Fri, 17 Jun 2016 20:27:18 +0000 (16:27 -0400)]
Bugfix "arv-ws --id" option, refs #9388.

7 years ago9427: Websockets now uses a thread per connection for database query and
Peter Amstutz [Fri, 17 Jun 2016 20:12:15 +0000 (16:12 -0400)]
9427: Websockets now uses a thread per connection for database query and
sending outgoing messages.  This minimizes interference by slow clients or
clients that are "replaying the world" with a last_log_id far in the past.

7 years agocloses #9372
radhika [Fri, 17 Jun 2016 15:13:28 +0000 (11:13 -0400)]
closes #9372
Merge branch '9372-container-display'

7 years agoMerge branch 'master' into 9372-container-display
radhika [Fri, 17 Jun 2016 15:13:12 +0000 (11:13 -0400)]
Merge branch 'master' into 9372-container-display

7 years ago9372: improve container_work_unit -> uri method to check if @proxied will respond_to...
radhika [Fri, 17 Jun 2016 15:11:48 +0000 (11:11 -0400)]
9372: improve container_work_unit -> uri method to check if @proxied will respond_to table_name instead of using a rescue.

7 years ago9180: Changed some of the logic on ThreadLimiter and made unit tests to validate...
Lucas Di Pentima [Thu, 16 Jun 2016 23:06:21 +0000 (20:06 -0300)]
9180: Changed some of the logic on ThreadLimiter and made unit tests to validate the new behaviour
refs #9180

7 years ago9372: The children of a Container will be ContainerRequests to which this is the...
radhika [Thu, 16 Jun 2016 14:32:21 +0000 (10:32 -0400)]
9372: The children of a Container will be ContainerRequests to which this is the requesting_container_uuid.

7 years ago9372: variable names and comments.
radhika [Thu, 16 Jun 2016 12:17:46 +0000 (08:17 -0400)]
9372: variable names and comments.

7 years ago9408: Restore build-essential and ruby-dev to arvbox.
Brett Smith [Thu, 16 Jun 2016 02:47:05 +0000 (22:47 -0400)]
9408: Restore build-essential and ruby-dev to arvbox.

These were previously being pulled in as Recommends, and they're
necessary to build gems for the demo image.
Refs #9408.

7 years agoMerge branch '9417-asserttrue-for-value-comparison'
Lucas Di Pentima [Thu, 16 Jun 2016 01:51:07 +0000 (22:51 -0300)]
Merge branch '9417-asserttrue-for-value-comparison'

Closes #9417

7 years ago9417: Replaced assertTrue() with assertEqual() on RetryNeedsMultipleServices's tests
Lucas Di Pentima [Wed, 15 Jun 2016 19:27:23 +0000 (16:27 -0300)]
9417: Replaced assertTrue() with assertEqual() on RetryNeedsMultipleServices's tests

7 years agoFirst try, don't think it'll work
Lucas Di Pentima [Wed, 15 Jun 2016 21:45:10 +0000 (18:45 -0300)]
First try, don't think it'll work

7 years ago8442: Generatefiles encode utf-8
Peter Amstutz [Wed, 15 Jun 2016 20:09:22 +0000 (16:09 -0400)]
8442: Generatefiles encode utf-8

7 years ago8442: Submit CommandLineTool containers directly without an intermediate
Peter Amstutz [Wed, 15 Jun 2016 18:51:14 +0000 (14:51 -0400)]
8442: Submit CommandLineTool containers directly without an intermediate
workflow runner.  Handle unsupported feature gracefully.

7 years ago8442: Tweak internal handling of keep: paths, examine exit codes to determine
Peter Amstutz [Wed, 15 Jun 2016 15:23:47 +0000 (11:23 -0400)]
8442: Tweak internal handling of keep: paths, examine exit codes to determine
success/fail.

7 years ago8442: more import fixups, import workflow file correctly.
Peter Amstutz [Mon, 13 Jun 2016 20:42:37 +0000 (16:42 -0400)]
8442: more import fixups, import workflow file correctly.

7 years ago8442: import fixups
Peter Amstutz [Mon, 13 Jun 2016 20:28:45 +0000 (16:28 -0400)]
8442: import fixups

7 years ago8442: Debugging container --submit with crunch2
Peter Amstutz [Mon, 13 Jun 2016 20:17:45 +0000 (16:17 -0400)]
8442: Debugging container --submit with crunch2

7 years ago8442: Adding --submit support with --crunch2. General refactoring into more/smaller...
Peter Amstutz [Mon, 13 Jun 2016 19:45:29 +0000 (15:45 -0400)]
8442: Adding --submit support with --crunch2.  General refactoring into more/smaller files.

7 years ago8442: Fix message
Peter Amstutz [Mon, 13 Jun 2016 14:36:05 +0000 (10:36 -0400)]
8442: Fix message

7 years ago8442: raise UnsupportedRequirement for unsupported features in the conformance
Peter Amstutz [Thu, 9 Jun 2016 19:16:09 +0000 (15:16 -0400)]
8442: raise UnsupportedRequirement for unsupported features in the conformance
tests.  Bump cwltool dependency.

7 years ago8442: Setting up mount points works. Capturing output works.
Peter Amstutz [Thu, 9 Jun 2016 12:21:34 +0000 (08:21 -0400)]
8442: Setting up mount points works.  Capturing output works.

7 years ago8442: Return PDH for Docker container. Working on setting up mount points.
Peter Amstutz [Wed, 8 Jun 2016 15:45:00 +0000 (11:45 -0400)]
8442: Return PDH for Docker container.  Working on setting up mount points.

7 years ago8442: Submit containers Work in progess.
Peter Amstutz [Wed, 8 Jun 2016 15:15:58 +0000 (11:15 -0400)]
8442: Submit containers Work in progess.

7 years ago8442: Add --crunch1/--crunch2 switch
Peter Amstutz [Mon, 6 Jun 2016 19:54:35 +0000 (15:54 -0400)]
8442: Add --crunch1/--crunch2 switch

7 years ago8442: CWL create crunch2 containers WIP
Peter Amstutz [Mon, 6 Jun 2016 19:24:50 +0000 (15:24 -0400)]
8442: CWL create crunch2 containers WIP

7 years agoMerge branch '9395-keep-balance-page-size'
Tom Clegg [Wed, 15 Jun 2016 19:40:58 +0000 (15:40 -0400)]
Merge branch '9395-keep-balance-page-size'

closes #9395

7 years ago9395: Fix encoding of integers in query params.
Tom Clegg [Wed, 15 Jun 2016 19:40:28 +0000 (15:40 -0400)]
9395: Fix encoding of integers in query params.

7 years ago9395: Explain CollectionBatchSize and CollectionBuffers in -help message.
Tom Clegg [Wed, 15 Jun 2016 14:53:01 +0000 (10:53 -0400)]
9395: Explain CollectionBatchSize and CollectionBuffers in -help message.

7 years ago9395: Add CollectionBuffers config to keep-balance.
Tom Clegg [Mon, 13 Jun 2016 14:31:39 +0000 (10:31 -0400)]
9395: Add CollectionBuffers config to keep-balance.

7 years ago9395: Add CollectionBatchSize config to keep-balance.
Tom Clegg [Mon, 13 Jun 2016 14:30:23 +0000 (10:30 -0400)]
9395: Add CollectionBatchSize config to keep-balance.

7 years agoAdd missing arvbox package "sudo" refs #9408
Peter Amstutz [Wed, 15 Jun 2016 19:38:34 +0000 (15:38 -0400)]
Add missing arvbox package "sudo" refs #9408

7 years agoUse "less -R" in "arvbox log" so that color codes from api server logs display
Peter Amstutz [Wed, 15 Jun 2016 19:15:50 +0000 (15:15 -0400)]
Use "less -R" in "arvbox log" so that color codes from api server logs display
properly.  no issue #

7 years agoMerge branch '9309-centos-7-install-guide-wip'
Brett Smith [Wed, 15 Jun 2016 18:31:38 +0000 (14:31 -0400)]
Merge branch '9309-centos-7-install-guide-wip'

Refs #9309.  Closes #9368.

7 years ago9309: Add CentOS 7 to Install Guide.
Brett Smith [Wed, 8 Jun 2016 21:28:24 +0000 (17:28 -0400)]
9309: Add CentOS 7 to Install Guide.

7 years agoAdd missing package 'less'. Also reduce crunch-dispatch-local poll interval to 3...
Peter Amstutz [Wed, 15 Jun 2016 18:02:39 +0000 (14:02 -0400)]
Add missing package 'less'.  Also reduce crunch-dispatch-local poll interval to 3 seconds.  no issue #

7 years agoMerge branch 'master' into 9372-container-display
radhika [Wed, 15 Jun 2016 17:39:52 +0000 (13:39 -0400)]
Merge branch 'master' into 9372-container-display

7 years agoMerge branch '9408-arvbox-dockerfile-pulls-unnecessary-packages'
Lucas Di Pentima [Wed, 15 Jun 2016 17:35:34 +0000 (14:35 -0300)]
Merge branch '9408-arvbox-dockerfile-pulls-unnecessary-packages'

refs #9408

7 years ago9372: Corrected dashboard display to list container_requests, not containers as imple...
radhika [Wed, 15 Jun 2016 17:27:43 +0000 (13:27 -0400)]
9372: Corrected dashboard display to list container_requests, not containers as implementd in #9318.

7 years ago9309: Bump Rails package iteration.
Brett Smith [Wed, 15 Jun 2016 17:04:37 +0000 (13:04 -0400)]
9309: Bump Rails package iteration.

To accommodate systemd support in the package scripts.
Refs #9309.

7 years agoPackage build directories get same owner as $WORKSPACE.
Brett Smith [Wed, 15 Jun 2016 16:58:31 +0000 (12:58 -0400)]
Package build directories get same owner as $WORKSPACE.

Since this script runs as root inside the Docker container, the
directory created on the previous line is normally owned by root too.
This causes trouble when the package tests run later and try to create
repository information inside the directory.  See, e.g.,
<https://ci.curoverse.com/job/build-packages-sso/23/consoleFull> for a
build that failed because of this.

No issue #.

7 years agoMerge branch '9309-centos-7-rails-scripts-wip'
Brett Smith [Wed, 15 Jun 2016 15:52:00 +0000 (11:52 -0400)]
Merge branch '9309-centos-7-rails-scripts-wip'

Refs #9309.  Closes #9351, #9345.

7 years ago9345: Use same script to test all Rails packages.
Brett Smith [Fri, 3 Jun 2016 19:34:25 +0000 (15:34 -0400)]
9345: Use same script to test all Rails packages.

We want to simplify the SSO test script for a variety of reasons.  We
don't want the package to depend on the PostgreSQL server; testing
proper PostgreSQL server configuration is a little out of scope for
this; and it's difficult to execute the test in a CentOS 7 Docker
image.  This commit has the SSO server follow the same test strategy
as other Rails servers.

7 years ago9309: Add systemd support to Rails package scripts.
Brett Smith [Fri, 3 Jun 2016 18:50:59 +0000 (14:50 -0400)]
9309: Add systemd support to Rails package scripts.

7 years ago9408: Added apt-get flag to avoid installing 'Recommended:' packages, it's better...
Lucas Di Pentima [Wed, 15 Jun 2016 13:58:02 +0000 (10:58 -0300)]
9408: Added apt-get flag to avoid installing 'Recommended:' packages, it's better to have an explicit dependency list. Also, added bzip2 that was being pulled implicitly and it's needed.

7 years agoMerge branch '9374-go-sdk'
Tom Clegg [Wed, 15 Jun 2016 13:51:55 +0000 (09:51 -0400)]
Merge branch '9374-go-sdk'

refs #9374

7 years ago9374: Remove unused receivers.
Tom Clegg [Wed, 15 Jun 2016 13:47:59 +0000 (09:47 -0400)]
9374: Remove unused receivers.

7 years ago9374: Fix uses of "this" as receiver name.
Tom Clegg [Tue, 14 Jun 2016 14:30:06 +0000 (10:30 -0400)]
9374: Fix uses of "this" as receiver name.

7 years ago9374: Use arvados.Collection instead of own CollectionRecord.
Tom Clegg [Tue, 14 Jun 2016 14:24:21 +0000 (10:24 -0400)]
9374: Use arvados.Collection instead of own CollectionRecord.