arvados.git
6 years ago14510: Bugfix order of precedence for setting collection cache size
Peter Amstutz [Fri, 30 Nov 2018 15:50:20 +0000 (10:50 -0500)]
14510: Bugfix order of precedence for setting collection cache size

Fix a couple other buggy tests.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14510: Don't initialize workflow redundantly.
Peter Amstutz [Fri, 30 Nov 2018 14:25:47 +0000 (09:25 -0500)]
14510: Don't initialize workflow redundantly.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14510: Set collection metadata cache size as 150% of sum of inputs
Peter Amstutz [Wed, 28 Nov 2018 19:42:06 +0000 (14:42 -0500)]
14510: Set collection metadata cache size as 150% of sum of inputs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14510: Estimate collection cache size wip
Peter Amstutz [Tue, 27 Nov 2018 23:13:30 +0000 (18:13 -0500)]
14510: Estimate collection cache size wip

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14510: Update tests/docs for --collection-cache-size
Peter Amstutz [Tue, 27 Nov 2018 22:26:07 +0000 (17:26 -0500)]
14510: Update tests/docs for --collection-cache-size

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14510: Setting collection cache wip
Peter Amstutz [Tue, 27 Nov 2018 20:53:07 +0000 (15:53 -0500)]
14510: Setting collection cache wip

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14510: Perfomance fixes
Peter Amstutz [Tue, 20 Nov 2018 19:33:53 +0000 (14:33 -0500)]
14510: Perfomance fixes

* Add --collection-cache to enable users to workaround cache thrashing

* Limit task queue size.  Release workflow lock when attempting to
  enqueue a task (which now may block).

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoRestore default value of --on-error to 'continue' refs #14440
Peter Amstutz [Tue, 27 Nov 2018 20:54:37 +0000 (15:54 -0500)]
Restore default value of --on-error to 'continue' refs #14440

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14440-fed-user-doc' refs #14440
Peter Amstutz [Tue, 27 Nov 2018 19:18:31 +0000 (14:18 -0500)]
Merge branch '14440-fed-user-doc' refs #14440

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14440: Fix formatting of cwl-run-options table.
Peter Amstutz [Tue, 27 Nov 2018 19:14:54 +0000 (14:14 -0500)]
14440: Fix formatting of cwl-run-options table.

Also expand help text of --on-error option a bit.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14535-ruamel-dependencies'
Nico Cesar [Tue, 27 Nov 2018 18:15:18 +0000 (13:15 -0500)]
Merge branch '14535-ruamel-dependencies'

refs #14535

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <ncesar@veritasgenetics.com>

6 years ago14535: bumping package revision up for python-ruamel.yaml
Nico Cesar [Tue, 27 Nov 2018 18:06:36 +0000 (13:06 -0500)]
14535: bumping package revision up for python-ruamel.yaml

refs #14535

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <ncesar@veritasgenetics.com>

6 years ago14440: Refresh cwl-run-options, add note about dockerCollectionPDH
Peter Amstutz [Tue, 27 Nov 2018 17:47:11 +0000 (12:47 -0500)]
14440: Refresh cwl-run-options, add note about dockerCollectionPDH

Also add note about workbench not displaying remote steps.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14535: add dependencies for ruamel.yaml
Nico Cesar [Tue, 27 Nov 2018 16:27:44 +0000 (11:27 -0500)]
14535: add dependencies for ruamel.yaml

refs #14535

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <ncesar@veritasgenetics.com>

6 years agoMerge branch '14519-wb-textile-issues'
Lucas Di Pentima [Mon, 26 Nov 2018 19:43:29 +0000 (16:43 -0300)]
Merge branch '14519-wb-textile-issues'
Closes #14519

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14519: Adds more table related tags to the whitelist
Lucas Di Pentima [Mon, 26 Nov 2018 19:42:11 +0000 (16:42 -0300)]
14519: Adds more table related tags to the whitelist

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14519: Fixes bug by adding table tags to the sanitizer white list
Lucas Di Pentima [Wed, 21 Nov 2018 22:32:23 +0000 (19:32 -0300)]
14519: Fixes bug by adding table tags to the sanitizer white list

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14519: Test exposing the regression
Lucas Di Pentima [Wed, 21 Nov 2018 22:31:51 +0000 (19:31 -0300)]
14519: Test exposing the regression

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14440: Add discussion of federation features.
Peter Amstutz [Mon, 26 Nov 2018 17:37:07 +0000 (12:37 -0500)]
14440: Add discussion of federation features.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14440: Federated workflow diagram.
Peter Amstutz [Wed, 21 Nov 2018 21:24:31 +0000 (16:24 -0500)]
14440: Federated workflow diagram.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14440: Comments on how federated.cwl works
Peter Amstutz [Wed, 21 Nov 2018 21:23:59 +0000 (16:23 -0500)]
14440: Comments on how federated.cwl works

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14440: Add project
Peter Amstutz [Wed, 21 Nov 2018 19:46:06 +0000 (14:46 -0500)]
14440: Add project

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14440: Add project_uuid to ClusterTarget example
Peter Amstutz [Wed, 21 Nov 2018 18:42:38 +0000 (13:42 -0500)]
14440: Add project_uuid to ClusterTarget example

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14440: Federated workflow docs WIP
Peter Amstutz [Wed, 21 Nov 2018 16:33:40 +0000 (11:33 -0500)]
14440: Federated workflow docs WIP

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14440: Working on docs for federated workflows.
Peter Amstutz [Tue, 20 Nov 2018 22:20:52 +0000 (17:20 -0500)]
14440: Working on docs for federated workflows.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14198-make-builder' refs #14198
Peter Amstutz [Mon, 26 Nov 2018 15:42:02 +0000 (10:42 -0500)]
Merge branch '14198-make-builder' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Restore behavior of creating an empty Builder object
Peter Amstutz [Mon, 26 Nov 2018 15:13:39 +0000 (10:13 -0500)]
14198: Restore behavior of creating an empty Builder object

Used for do_eval() on set_cluster_target().  Calling _init_job() turns
out to have some unintended side effects as well as doing redundant
work.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoFinish fixing setting project_uuid from runtimeContext, fix tests.
Peter Amstutz [Wed, 21 Nov 2018 21:39:59 +0000 (16:39 -0500)]
Finish fixing setting project_uuid from runtimeContext, fix tests.

refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoFix ClusterTarget.project_uuid to correctly set the project uuid
Peter Amstutz [Wed, 21 Nov 2018 19:46:38 +0000 (14:46 -0500)]
Fix ClusterTarget.project_uuid to correctly set the project uuid

refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14345-nginx-proxy-request-buffering'
Tom Clegg [Wed, 21 Nov 2018 19:44:23 +0000 (14:44 -0500)]
Merge branch '14345-nginx-proxy-request-buffering'

refs #14345

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Disable proxy request buffering in Nginx keepproxy example.
Tom Clegg [Wed, 21 Nov 2018 19:34:17 +0000 (14:34 -0500)]
14345: Disable proxy request buffering in Nginx keepproxy example.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14345: Disable Nginx proxy request buffering for keep-web requests.
Tom Clegg [Wed, 21 Nov 2018 19:14:04 +0000 (14:14 -0500)]
14345: Disable Nginx proxy request buffering for keep-web requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoFix --submit-runner-cluster tests refs #14198
Peter Amstutz [Wed, 21 Nov 2018 19:00:04 +0000 (14:00 -0500)]
Fix --submit-runner-cluster tests refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14345-proppatch'
Tom Clegg [Wed, 21 Nov 2018 18:28:00 +0000 (13:28 -0500)]
Merge branch '14345-proppatch'

refs #14345

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14198-fed-testing' refs #14198
Peter Amstutz [Wed, 21 Nov 2018 18:06:00 +0000 (13:06 -0500)]
Merge branch '14198-fed-testing' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Call validate_cluster_target on --submit-runner-cluster
Peter Amstutz [Wed, 21 Nov 2018 17:41:11 +0000 (12:41 -0500)]
14198: Call validate_cluster_target on --submit-runner-cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add check for uuidPrefix if local cluster isn't in the remoteHosts.
Peter Amstutz [Tue, 20 Nov 2018 22:26:17 +0000 (17:26 -0500)]
14198: Add check for uuidPrefix if local cluster isn't in the remoteHosts.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Update build.list for latest cwltool/schema-salad
Peter Amstutz [Tue, 20 Nov 2018 18:30:40 +0000 (13:30 -0500)]
14198: Update build.list for latest cwltool/schema-salad

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix secondaryFile tests
Peter Amstutz [Mon, 19 Nov 2018 20:05:01 +0000 (15:05 -0500)]
14198: Fix secondaryFile tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Tests for hints in different places
Peter Amstutz [Sat, 17 Nov 2018 03:01:45 +0000 (22:01 -0500)]
14198: Tests for hints in different places

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Tests prefer setting ClusterTarget hint on the workflow step.
Peter Amstutz [Fri, 16 Nov 2018 21:39:09 +0000 (16:39 -0500)]
14198: Tests prefer setting ClusterTarget hint on the workflow step.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Test cases for places that ClusterTarget can appear
Peter Amstutz [Fri, 16 Nov 2018 21:18:19 +0000 (16:18 -0500)]
14198: Test cases for places that ClusterTarget can appear

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Set ClusterTarget hint on steps instead of tools
Peter Amstutz [Fri, 16 Nov 2018 15:59:52 +0000 (10:59 -0500)]
14198: Set ClusterTarget hint on steps instead of tools

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Rewrite README
Peter Amstutz [Wed, 14 Nov 2018 19:44:19 +0000 (14:44 -0500)]
14198: Rewrite README

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Connect test cases to outputs
Peter Amstutz [Wed, 14 Nov 2018 04:10:07 +0000 (23:10 -0500)]
14198: Connect test cases to outputs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add threestep-remote
Peter Amstutz [Wed, 14 Nov 2018 03:56:56 +0000 (22:56 -0500)]
14198: Add threestep-remote

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add scatter-gather
Peter Amstutz [Wed, 14 Nov 2018 03:06:36 +0000 (22:06 -0500)]
14198: Add scatter-gather

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add test twostep-remote-copy-to-home
Peter Amstutz [Tue, 13 Nov 2018 22:28:59 +0000 (17:28 -0500)]
14198: Add test twostep-remote-copy-to-home

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix collection PDHs for remote-case
Peter Amstutz [Tue, 13 Nov 2018 21:37:34 +0000 (16:37 -0500)]
14198: Fix collection PDHs for remote-case

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add some basic doc comments.
Peter Amstutz [Fri, 9 Nov 2018 20:48:14 +0000 (15:48 -0500)]
14198: Add some basic doc comments.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add remote-case and twostep-both-remote
Peter Amstutz [Fri, 9 Nov 2018 20:33:05 +0000 (15:33 -0500)]
14198: Add remote-case and twostep-both-remote

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add more twostep tests
Peter Amstutz [Fri, 9 Nov 2018 20:02:50 +0000 (15:02 -0500)]
14198: Add more twostep tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Working on two step test cases
Peter Amstutz [Fri, 9 Nov 2018 15:47:23 +0000 (10:47 -0500)]
14198: Working on two step test cases

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Refactor arvbox federation setup
Peter Amstutz [Fri, 9 Nov 2018 03:19:15 +0000 (22:19 -0500)]
14198: Refactor arvbox federation setup

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Federation feature and testing
Peter Amstutz [Fri, 16 Nov 2018 21:15:11 +0000 (16:15 -0500)]
14198: Federation feature and testing

* Evaluate ClusterTarget in the job step that declares it

* Don't use get_requirement to inherit ClusterTarget.

* Workflow to start and configure a federation of N arvboxes, initialize test user

* Test cases.

* Prepare/scrub before running test

* Use --always-submit-runner

* Tests use isolated inputs.  Return success boolean

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Set arvrunner before calling super constructor
Peter Amstutz [Fri, 16 Nov 2018 19:21:07 +0000 (14:21 -0500)]
14198: Set arvrunner before calling super constructor

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Set arvrunner on ArvadosWorkflowStep
Peter Amstutz [Fri, 16 Nov 2018 19:12:51 +0000 (14:12 -0500)]
14198: Set arvrunner on ArvadosWorkflowStep

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Set fake 'class' so _init_job is happy
Peter Amstutz [Fri, 16 Nov 2018 18:55:49 +0000 (13:55 -0500)]
14198: Set fake 'class' so _init_job is happy

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix jobstep builder init
Peter Amstutz [Fri, 16 Nov 2018 16:44:55 +0000 (11:44 -0500)]
14198: Fix jobstep builder init

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix ruamel.yaml versioning
Peter Amstutz [Fri, 16 Nov 2018 16:28:27 +0000 (11:28 -0500)]
14198: Fix ruamel.yaml versioning

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Construct ArvadosWorkflowStep
Peter Amstutz [Fri, 16 Nov 2018 16:26:51 +0000 (11:26 -0500)]
14198: Construct ArvadosWorkflowStep

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Update version deps for schema-salad, cwltool, ruamel.yaml, requests
Peter Amstutz [Fri, 16 Nov 2018 15:28:24 +0000 (10:28 -0500)]
14198: Update version deps for schema-salad, cwltool, ruamel.yaml, requests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Update cwltool and schema-salad
Peter Amstutz [Wed, 14 Nov 2018 18:22:16 +0000 (13:22 -0500)]
14198: Update cwltool and schema-salad

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add --always-submit-runner
Peter Amstutz [Wed, 7 Nov 2018 21:24:02 +0000 (16:24 -0500)]
14198: Add --always-submit-runner

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add --submit-runner-cluster test
Peter Amstutz [Mon, 5 Nov 2018 22:44:23 +0000 (17:44 -0500)]
14198: Add --submit-runner-cluster test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Unit tests pass again
Peter Amstutz [Mon, 5 Nov 2018 15:16:48 +0000 (10:16 -0500)]
14198: Unit tests pass again

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: test_submit passes
Peter Amstutz [Fri, 2 Nov 2018 21:05:20 +0000 (17:05 -0400)]
14198: test_submit passes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: More test updates
Peter Amstutz [Fri, 2 Nov 2018 20:41:01 +0000 (16:41 -0400)]
14198: More test updates

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Bringing unit tests up to date WIP
Peter Amstutz [Fri, 2 Nov 2018 20:05:29 +0000 (16:05 -0400)]
14198: Bringing unit tests up to date WIP

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add paging on container list, check for valid cluster id
Peter Amstutz [Thu, 1 Nov 2018 18:42:12 +0000 (14:42 -0400)]
14198: Add paging on container list, check for valid cluster id

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Use ArvRuntimeContext.submit_runner_cluster for target cluster
Peter Amstutz [Wed, 31 Oct 2018 17:25:24 +0000 (13:25 -0400)]
14198: Use ArvRuntimeContext.submit_runner_cluster for target cluster

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Fix typo current -> current_container, add copyright header
Peter Amstutz [Thu, 25 Oct 2018 21:21:05 +0000 (17:21 -0400)]
14198: Fix typo current -> current_container, add copyright header

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Log current container
Peter Amstutz [Thu, 25 Oct 2018 21:11:59 +0000 (17:11 -0400)]
14198: Log current container

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Support expressions in TargetCluster[clusterID, ownerUUID]
Peter Amstutz [Thu, 25 Oct 2018 20:14:50 +0000 (16:14 -0400)]
14198: Support expressions in TargetCluster[clusterID, ownerUUID]

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Resolve to Docker images to PDH and set "http://arvados.org/cwl#dockerCollecti...
Peter Amstutz [Thu, 25 Oct 2018 19:19:05 +0000 (15:19 -0400)]
14198: Resolve to Docker images to PDH and set "arvados.org/cwl#dockerCollectionPDH"

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Refactor and add support for --submit-runner-cluster
Peter Amstutz [Thu, 25 Oct 2018 19:02:23 +0000 (15:02 -0400)]
14198: Refactor and add support for --submit-runner-cluster

Rename ArvCwlRunner to ArvCwlExecutor and move into its own file.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Use PDH for container_image instead of docker repo+tag
Peter Amstutz [Thu, 25 Oct 2018 14:36:47 +0000 (10:36 -0400)]
14198: Use PDH for container_image instead of docker repo+tag

Needed to support federated container requests.

This reverts 6ea807b2caf6c934f170b2e4d89c23c4a08ca69c

Based on the commit comment, that change was made to accomodate the
Docker v1 to v2 image format migration (to enable the API server to
select the image with the correct format).  However, the API server
subsequently gained the ability to detect if it needed to substitute a
PDH with a migrated image PDH in commit
a72205728f94f5261b657766e01f5767dc15d4b5 so now we want restore the
original behavior of locally resolving the image PDH and using that in
the container request.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Initial support ClusterTarget hint
Peter Amstutz [Wed, 24 Oct 2018 18:45:21 +0000 (14:45 -0400)]
14198: Initial support ClusterTarget hint

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14345: Support PROPPATCH method.
Tom Clegg [Wed, 21 Nov 2018 16:46:34 +0000 (11:46 -0500)]
14345: Support PROPPATCH method.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14516: when calculating the next available username, make sure to take merged
Ward Vandewege [Tue, 20 Nov 2018 18:45:14 +0000 (13:45 -0500)]
14516: when calculating the next available username, make sure to take merged
accounts into consideration - they have usernames too, and we have a unique
index on username in the users table.

refs #14516

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoCWL arvados-tests.sh uses --api=containers no issue #
Peter Amstutz [Tue, 20 Nov 2018 14:23:25 +0000 (09:23 -0500)]
CWL arvados-tests.sh uses --api=containers no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14497-rack-upgrade'
Lucas Di Pentima [Fri, 16 Nov 2018 16:44:42 +0000 (13:44 -0300)]
Merge branch '14497-rack-upgrade'
Closes #14497

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago14497: Upgrades rack to 1.6.11 on api & workbench.
Lucas Di Pentima [Fri, 16 Nov 2018 15:07:32 +0000 (12:07 -0300)]
14497: Upgrades rack to 1.6.11 on api & workbench.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoAdd --no-tty to apt-key in arvbox dockerfile. no issue #
Peter Amstutz [Wed, 14 Nov 2018 21:35:30 +0000 (16:35 -0500)]
Add --no-tty to apt-key in arvbox dockerfile.  no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14419-concurrent-map-write'
Tom Clegg [Wed, 14 Nov 2018 21:05:34 +0000 (16:05 -0500)]
Merge branch '14419-concurrent-map-write'

fixes #14419

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoThe Dockerfiles for package building and testing on debian 9 seem to want
Ward Vandewege [Wed, 14 Nov 2018 20:57:32 +0000 (15:57 -0500)]
The Dockerfiles for package building and testing on debian 9 seem to want
--no-tty as an argument to gpg calls now.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoMerge branch '14198-request-token' refs #14198
Peter Amstutz [Wed, 14 Nov 2018 19:07:18 +0000 (14:07 -0500)]
Merge branch '14198-request-token' refs #14198

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14198: Add test that runtime token is set from auth for remote user
Peter Amstutz [Wed, 14 Nov 2018 18:53:47 +0000 (13:53 -0500)]
14198: Add test that runtime token is set from auth for remote user

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago14419: Fix concurrent map write.
Tom Clegg [Wed, 14 Nov 2018 18:49:39 +0000 (13:49 -0500)]
14419: Fix concurrent map write.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoUpdate tests to match new template.
Tom Clegg [Tue, 13 Nov 2018 21:56:27 +0000 (16:56 -0500)]
Update tests to match new template.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoFix git command.
Tom Clegg [Tue, 13 Nov 2018 21:19:40 +0000 (16:19 -0500)]
Fix git command.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14198: Strip 4th position when setting runtime_token from provided auth
Peter Amstutz [Tue, 13 Nov 2018 21:18:09 +0000 (16:18 -0500)]
14198: Strip 4th position when setting runtime_token from provided auth

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '14475-update-collection-error'
Tom Clegg [Tue, 13 Nov 2018 21:18:59 +0000 (16:18 -0500)]
Merge branch '14475-update-collection-error'

fixes #14475

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14475: Fix Content-Type header parsing.
Tom Clegg [Tue, 13 Nov 2018 20:49:31 +0000 (15:49 -0500)]
14475: Fix Content-Type header parsing.

"Content-Type: application/x-www-form-urlencoded; charset=UTF-8", for
example, means we should parse the request body as a form.

See https://github.com/golang/go/blob/master/src/net/http/request.go
-> parsePostForm().

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '14397-keepstore-v4-signature'
Tom Clegg [Tue, 13 Nov 2018 19:51:08 +0000 (14:51 -0500)]
Merge branch '14397-keepstore-v4-signature'

fixes #14397

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Add comment about memory implications of providing SHA256.
Tom Clegg [Tue, 13 Nov 2018 16:43:28 +0000 (11:43 -0500)]
14397: Add comment about memory implications of providing SHA256.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14198: Remote user uses current token in runtime_token
Peter Amstutz [Tue, 13 Nov 2018 16:42:19 +0000 (11:42 -0500)]
14198: Remote user uses current token in runtime_token

Can't create a new token for a remote user, so use the token that was
supplied.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoIn the notification e-mail to the user that a shell account has been created,
Ward Vandewege [Wed, 7 Nov 2018 15:58:36 +0000 (10:58 -0500)]
In the notification e-mail to the user that a shell account has been created,
change the title of the e-mail to mention Arvados instead of Curoverse.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago14397: Pass SHA256 to s3 library.
Tom Clegg [Mon, 12 Nov 2018 18:12:28 +0000 (13:12 -0500)]
14397: Pass SHA256 to s3 library.

Otherwise, the s3 library copies the data into another new buffer,
doubling the memory needed for a write operation.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago14397: Use {sha1}-dev as version in dev builds of Go programs.
Tom Clegg [Mon, 12 Nov 2018 15:40:27 +0000 (10:40 -0500)]
14397: Use {sha1}-dev as version in dev builds of Go programs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>