arvados.git
5 years agoMerge branch '14807-prod-blockers'
Tom Clegg [Thu, 21 Mar 2019 20:50:32 +0000 (16:50 -0400)]
Merge branch '14807-prod-blockers'

refs #14807

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14451-crunchstat-summary-cache-ram'
Tom Morris [Thu, 21 Mar 2019 20:02:20 +0000 (16:02 -0400)]
Merge branch '14451-crunchstat-summary-cache-ram'

Fix three small crunchstat-summary issues:
- Use default of 1 if no cores requested. fixes #12026
- Don't recommend RAM or CPU changes based on zero usage. fixes #10570
- Remove extra MB (2**20) factor in Keep cache recommendation.  Fixes #14451

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoImprove wording with direction of recommendation change.
Tom Morris [Fri, 15 Mar 2019 05:11:59 +0000 (01:11 -0400)]
Improve wording with direction of recommendation change.

No issue.

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoUse default of 1 if no cores requested. refs #12026
Tom Morris [Fri, 15 Mar 2019 04:47:38 +0000 (00:47 -0400)]
Use default of 1 if no cores requested. refs #12026

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoDon't recommend RAM or CPU changes based on zero usage. refs #10570
Tom Morris [Fri, 15 Mar 2019 04:10:06 +0000 (00:10 -0400)]
Don't recommend RAM or CPU changes based on zero usage. refs #10570

Never recommend 0 MB RAM. If CPU usage is zero, don't make any
CPU recommendations.

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoFix Keep cache recommendation calculation.
Tom Morris [Fri, 15 Mar 2019 04:03:59 +0000 (00:03 -0400)]
Fix Keep cache recommendation calculation.

Remove extra MB (2**20) factor. Refs #14451

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoSpell MB consistently.
Tom Morris [Fri, 15 Mar 2019 02:11:45 +0000 (22:11 -0400)]
Spell MB consistently.

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoMerge branch '14645-fuse-operations-reporting'
Eric Biagiotti [Thu, 21 Mar 2019 19:52:26 +0000 (15:52 -0400)]
Merge branch '14645-fuse-operations-reporting'

refs #14645

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years agoMerge branch '14939-crunchstat-summary-python3'
Tom Morris [Thu, 21 Mar 2019 19:47:54 +0000 (15:47 -0400)]
Merge branch '14939-crunchstat-summary-python3'

Fixes #14939

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoResolve deprecation warnings
Tom Morris [Thu, 14 Mar 2019 22:47:20 +0000 (18:47 -0400)]
Resolve deprecation warnings

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoFix resource leak warned by Python 3
Tom Morris [Thu, 14 Mar 2019 22:30:52 +0000 (18:30 -0400)]
Fix resource leak warned by Python 3

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoAdd crunchstat-summary to Python 3 tests. refs #14939
Tom Morris [Wed, 13 Mar 2019 23:33:03 +0000 (19:33 -0400)]
Add crunchstat-summary to Python 3 tests. refs #14939

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoFix test Python 3 integer divide difference.
Tom Morris [Wed, 13 Mar 2019 23:26:00 +0000 (19:26 -0400)]
Fix test Python 3 integer divide difference.

Use float for both Python 2 & 3 like the rest of the metrics.
refs #14393

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years agoUpdate crunchstat-summary to Python 3. Refs #14939
Tom Morris [Fri, 8 Mar 2019 23:40:24 +0000 (18:40 -0500)]
Update crunchstat-summary to Python 3. Refs #14939

Arvados-DCO-1.1-Signed-off-by: Tom Morris <tfmorris@veritasgenetics.com>

5 years ago14807: Allow some fake nodes to remain unbroken.
Tom Clegg [Thu, 21 Mar 2019 18:12:54 +0000 (14:12 -0400)]
14807: Allow some fake nodes to remain unbroken.

If every fake node hits a fake failure mode in 200-400 ms, throughput
can fall below 20 containers per second and fail the test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge remote-tracking branch 'origin/master' into 14645-fuse-operations-reporting
Eric Biagiotti [Thu, 21 Mar 2019 18:03:16 +0000 (14:03 -0400)]
Merge remote-tracking branch 'origin/master' into 14645-fuse-operations-reporting

refs #14645

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years ago14645: Updates crunchstat format for fuse ops
Eric Biagiotti [Thu, 21 Mar 2019 14:49:31 +0000 (10:49 -0400)]
14645: Updates crunchstat format for fuse ops

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years agoMerge branch '14873-google-api-client-update'
Lucas Di Pentima [Wed, 20 Mar 2019 20:17:07 +0000 (17:17 -0300)]
Merge branch '14873-google-api-client-update'
Refs #14873

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago14807: Give up if initial keep services list isn't loaded in 60s.
Tom Clegg [Wed, 20 Mar 2019 05:50:10 +0000 (01:50 -0400)]
14807: Give up if initial keep services list isn't loaded in 60s.

Otherwise, MakeKeepClient blocks forever when ARVADOS_* env vars are
bad.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Report queue metrics.
Tom Clegg [Tue, 19 Mar 2019 21:02:05 +0000 (17:02 -0400)]
14807: Report queue metrics.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Add "kill instance" management API.
Tom Clegg [Tue, 19 Mar 2019 17:40:10 +0000 (13:40 -0400)]
14807: Add "kill instance" management API.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Configurable rate limit for cloud provider API calls.
Tom Clegg [Mon, 18 Mar 2019 21:27:40 +0000 (17:27 -0400)]
14807: Configurable rate limit for cloud provider API calls.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Drain instances that crunch-run reports broken.
Tom Clegg [Mon, 18 Mar 2019 20:32:58 +0000 (16:32 -0400)]
14807: Drain instances that crunch-run reports broken.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14807-escalate-sigterm'
Tom Clegg [Wed, 20 Mar 2019 19:24:02 +0000 (15:24 -0400)]
Merge branch '14807-escalate-sigterm'

refs #14807

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Merge branch 'master'
Tom Clegg [Wed, 20 Mar 2019 19:20:57 +0000 (15:20 -0400)]
14807: Merge branch 'master'

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Tweak log message.
Tom Clegg [Wed, 20 Mar 2019 19:20:38 +0000 (15:20 -0400)]
14807: Tweak log message.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14947-run-tests-interactive'
Tom Clegg [Wed, 20 Mar 2019 18:07:06 +0000 (14:07 -0400)]
Merge branch '14947-run-tests-interactive'

closes #14947

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14947: Ignore --only / --skip args in interactive mode.
Tom Clegg [Wed, 20 Mar 2019 17:41:35 +0000 (13:41 -0400)]
14947: Ignore --only / --skip args in interactive mode.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14873: Updates arvados & arvados-cli gems dependency on google-api-client.
Lucas Di Pentima [Wed, 20 Mar 2019 17:39:03 +0000 (14:39 -0300)]
14873: Updates arvados & arvados-cli gems dependency on google-api-client.

Start using our own fork arvados-google-api-client that ask for activesupport
version <5.1 so that it doesn't block the rails 5 upgrade.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago14947: Fix controller install in deps.
Tom Clegg [Wed, 20 Mar 2019 17:37:28 +0000 (13:37 -0400)]
14947: Fix controller install in deps.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14947: Accept test suite options in interactive mode.
Tom Clegg [Wed, 20 Mar 2019 13:28:24 +0000 (09:28 -0400)]
14947: Accept test suite options in interactive mode.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14645: Adds op counts to reporting
Eric Biagiotti [Tue, 19 Mar 2019 20:27:00 +0000 (16:27 -0400)]
14645: Adds op counts to reporting

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years ago14947: Install controller (cmd/arvados-server) for integration tests.
Tom Clegg [Tue, 19 Mar 2019 20:23:55 +0000 (16:23 -0400)]
14947: Install controller (cmd/arvados-server) for integration tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Don't send SIGKILL to crunch-run processes.
Tom Clegg [Tue, 19 Mar 2019 17:28:39 +0000 (13:28 -0400)]
14807: Don't send SIGKILL to crunch-run processes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14645: Adds individual ops time output
Eric Biagiotti [Tue, 19 Mar 2019 17:26:26 +0000 (13:26 -0400)]
14645: Adds individual ops time output

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years ago14807: Add comments to "error indicates success" code.
Tom Clegg [Tue, 19 Mar 2019 15:57:32 +0000 (11:57 -0400)]
14807: Add comments to "error indicates success" code.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14947: Suggest "install deps" as first interactive command.
Tom Clegg [Tue, 19 Mar 2019 14:30:36 +0000 (10:30 -0400)]
14947: Suggest "install deps" as first interactive command.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMove back arvbox version of nodejs to fix composer compile issue
Peter Amstutz [Mon, 18 Mar 2019 20:13:35 +0000 (16:13 -0400)]
Move back arvbox version of nodejs to fix composer compile issue

no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14807: Update tests to new worker/pool interface.
Tom Clegg [Mon, 18 Mar 2019 19:16:25 +0000 (15:16 -0400)]
14807: Update tests to new worker/pool interface.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Lengthen kill delays in stub driver to test term-kill-drain.
Tom Clegg [Mon, 18 Mar 2019 19:08:19 +0000 (15:08 -0400)]
14807: Lengthen kill delays in stub driver to test term-kill-drain.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14322-cwl-uuid-input' refs #14322
Peter Amstutz [Mon, 18 Mar 2019 18:43:31 +0000 (14:43 -0400)]
Merge branch '14322-cwl-uuid-input' refs #14322

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Tweak JSON_KEEP_LOCATOR_REGEXP and comment
Peter Amstutz [Mon, 18 Mar 2019 18:04:19 +0000 (14:04 -0400)]
14322: Tweak JSON_KEEP_LOCATOR_REGEXP and comment

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14807: Escalate sigterm->sigkill->drain instance.
Tom Clegg [Mon, 18 Mar 2019 15:09:09 +0000 (11:09 -0400)]
14807: Escalate sigterm->sigkill->drain instance.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Fail test if a container gets run twice.
Tom Clegg [Mon, 18 Mar 2019 15:44:19 +0000 (11:44 -0400)]
14807: Fail test if a container gets run twice.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14807: Fix unreliable test.
Tom Clegg [Mon, 18 Mar 2019 15:44:07 +0000 (11:44 -0400)]
14807: Fix unreliable test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14977-kill-if-not-in-queue'
Tom Clegg [Mon, 18 Mar 2019 15:11:56 +0000 (11:11 -0400)]
Merge branch '14977-kill-if-not-in-queue'

fixes #14977

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14977: Re-order statements for clarity.
Tom Clegg [Mon, 18 Mar 2019 13:42:27 +0000 (09:42 -0400)]
14977: Re-order statements for clarity.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14977: Don't try fixStaleLocks until worker pool state is loaded.
Tom Clegg [Fri, 15 Mar 2019 20:19:51 +0000 (16:19 -0400)]
14977: Don't try fixStaleLocks until worker pool state is loaded.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14977: Kill containers that don't exist according to controller.
Tom Clegg [Fri, 15 Mar 2019 20:11:48 +0000 (16:11 -0400)]
14977: Kill containers that don't exist according to controller.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14977: Drop containers when the controller says they don't exist.
Tom Clegg [Fri, 15 Mar 2019 19:21:39 +0000 (15:21 -0400)]
14977: Drop containers when the controller says they don't exist.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14322: Fix noisy unicode logging error in tests, add comments
Peter Amstutz [Fri, 15 Mar 2019 14:54:59 +0000 (10:54 -0400)]
14322: Fix noisy unicode logging error in tests, add comments

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Add missing test files
Peter Amstutz [Wed, 13 Mar 2019 21:30:57 +0000 (17:30 -0400)]
14322: Add missing test files

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Workbench links collection uuids in workflow inputs
Peter Amstutz [Wed, 13 Mar 2019 21:26:07 +0000 (17:26 -0400)]
14322: Workbench links collection uuids in workflow inputs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Update comment on bwa-mem-input-mixed.yml
Peter Amstutz [Wed, 13 Mar 2019 21:20:22 +0000 (17:20 -0400)]
14322: Update comment on bwa-mem-input-mixed.yml

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Add additional example input files
Peter Amstutz [Wed, 13 Mar 2019 20:40:33 +0000 (16:40 -0400)]
14322: Add additional example input files

* with uuids
* with both portable data hashes and uuids

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Tweak workbench support from manual testing
Peter Amstutz [Wed, 13 Mar 2019 20:19:37 +0000 (16:19 -0400)]
14322: Tweak workbench support from manual testing

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Update workbench to use new collectionUUID
Peter Amstutz [Wed, 13 Mar 2019 19:52:20 +0000 (15:52 -0400)]
14322: Update workbench to use new collectionUUID

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Fix tests
Peter Amstutz [Wed, 13 Mar 2019 19:25:33 +0000 (15:25 -0400)]
14322: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Tweak documentation to mention uuid keep URI form
Peter Amstutz [Wed, 13 Mar 2019 18:57:53 +0000 (14:57 -0400)]
14322: Tweak documentation to mention uuid keep URI form

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: CollectionFsAccess accepts UUIDs
Peter Amstutz [Wed, 13 Mar 2019 18:57:20 +0000 (14:57 -0400)]
14322: CollectionFsAccess accepts UUIDs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Tests for edge cases
Peter Amstutz [Tue, 12 Mar 2019 20:39:12 +0000 (16:39 -0400)]
14322: Tests for edge cases

Report unknown uuids and mismatches between current collection
PDH (from API server lookup) and location PDH.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Propagate collection UUID to container mount
Peter Amstutz [Mon, 11 Mar 2019 21:42:09 +0000 (17:42 -0400)]
14322: Propagate collection UUID to container mount

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14322: Convert uuids on input to portable data hashes
Peter Amstutz [Mon, 11 Mar 2019 21:02:14 +0000 (17:02 -0400)]
14322: Convert uuids on input to portable data hashes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years agoFix a-c-r printing logs on error refs #14870
Peter Amstutz [Fri, 15 Mar 2019 14:10:15 +0000 (10:10 -0400)]
Fix a-c-r printing logs on error refs #14870

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14947: run-tests: add --interactive, skip deps on --skip-install.
Tom Clegg [Thu, 14 Mar 2019 20:24:18 +0000 (16:24 -0400)]
14947: run-tests: add --interactive, skip deps on --skip-install.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14966-api-test-hang-fix'
Lucas Di Pentima [Thu, 14 Mar 2019 19:42:18 +0000 (16:42 -0300)]
Merge branch '14966-api-test-hang-fix'
Closes #14966

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years agoArvbox updates, no issue #
Peter Amstutz [Thu, 7 Feb 2019 22:07:08 +0000 (17:07 -0500)]
Arvbox updates, no issue #

* Bump Go version, add openjdk.

* Bump node version.  Make sure node is in path.

* Fix $HOME in setup scripts.

* Add "arvbox devenv" command which creates and enters a container
  environment suitable for interactively running development tools

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14966: Implements cleanup on test case teardown.
Lucas Di Pentima [Thu, 14 Mar 2019 18:56:00 +0000 (15:56 -0300)]
14966: Implements cleanup on test case teardown.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago14645: Adds prometheus_client dependency
Eric Biagiotti [Thu, 14 Mar 2019 13:26:29 +0000 (09:26 -0400)]
14645: Adds prometheus_client dependency

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years ago14966: Fixes test.
Lucas Di Pentima [Wed, 13 Mar 2019 23:52:25 +0000 (20:52 -0300)]
14966: Fixes test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years agoMerge remote-tracking branch 'origin/master' into 14645-fuse-operations-reporting
Eric Biagiotti [Wed, 13 Mar 2019 17:49:48 +0000 (13:49 -0400)]
Merge remote-tracking branch 'origin/master' into 14645-fuse-operations-reporting

refs #14645

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years ago14645: Adds FUSE ops time to crunchstat output
Eric Biagiotti [Wed, 13 Mar 2019 17:46:26 +0000 (13:46 -0400)]
14645: Adds FUSE ops time to crunchstat output

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>
Makes stat writing more generic. Now can accommodate different stat counts and types.

5 years ago14645: Adds FUSE ops timing using prometheus
Eric Biagiotti [Wed, 13 Mar 2019 17:43:03 +0000 (13:43 -0400)]
14645: Adds FUSE ops timing using prometheus

Arvados-DCO-1.1-Signed-off-by: Eric Biagiotti <ebiagiotti@veritasgenetics.com>

5 years agoSwitch to stretch-slim as the base image for arvados/jobs, this results in a
Ward Vandewege [Wed, 13 Mar 2019 15:42:30 +0000 (11:42 -0400)]
Switch to stretch-slim as the base image for arvados/jobs, this results in a
smaller image.

arvados/jobs         1.3.1.20190308210512   2a6c5e4993e7        4 days ago          284MB
arvados/jobs         latest                 0cfff29120dd        20 minutes ago      232MB

refs #14819

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years agoMerge branch '13593-async-perm-graph-update'
Lucas Di Pentima [Wed, 13 Mar 2019 15:09:14 +0000 (12:09 -0300)]
Merge branch '13593-async-perm-graph-update'
Closes #13593

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years agoMerge branch '13937-keepstore-prometheus'
Lucas Di Pentima [Tue, 12 Mar 2019 20:32:43 +0000 (17:32 -0300)]
Merge branch '13937-keepstore-prometheus'
Closes #13937

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago13593: Moves & enhances group creation with async=true test.
Lucas Di Pentima [Tue, 12 Mar 2019 18:58:57 +0000 (15:58 -0300)]
13593: Moves & enhances group creation with async=true test.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago13937: Removes the 'any' operation type for a better way of testing driver labels
Lucas Di Pentima [Tue, 12 Mar 2019 18:19:36 +0000 (15:19 -0300)]
13937: Removes the 'any' operation type for a better way of testing driver labels

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago13593: Cleans up old experimental code.
Lucas Di Pentima [Tue, 12 Mar 2019 15:04:49 +0000 (12:04 -0300)]
13593: Cleans up old experimental code.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago13593: Changes default async permission update interval. Clears cache.
Lucas Di Pentima [Tue, 12 Mar 2019 14:38:09 +0000 (11:38 -0300)]
13593: Changes default async permission update interval. Clears cache.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years agoSome build script housekeeping: fix the detection logic for the existence of
Ward Vandewege [Tue, 12 Mar 2019 14:32:05 +0000 (10:32 -0400)]
Some build script housekeeping: fix the detection logic for the existence of
Debian/Ubuntu packages on our apt repositories.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years agoBuild the cwltest package, which lives out of tree.
Ward Vandewege [Tue, 12 Mar 2019 14:31:13 +0000 (10:31 -0400)]
Build the cwltest package, which lives out of tree.

refs #9945

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years ago13937: Removes unnecessary data passing on tests.
Lucas Di Pentima [Tue, 12 Mar 2019 14:10:51 +0000 (11:10 -0300)]
13937: Removes unnecessary data passing on tests.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago13593: Updates API documentation.
Lucas Di Pentima [Tue, 12 Mar 2019 12:51:11 +0000 (09:51 -0300)]
13593: Updates API documentation.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years ago13593: Adds 'async' parameter to groups create & update calls for perms updates.
Lucas Di Pentima [Tue, 12 Mar 2019 12:33:30 +0000 (09:33 -0300)]
13593: Adds 'async' parameter to groups create & update calls for perms updates.

When async=true, the permission graph update will be deferred by the configured
amount of seconds at 'async_permissions_update_interval' (default: 60)

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years agoFix error message.
Tom Clegg [Mon, 11 Mar 2019 20:21:26 +0000 (16:21 -0400)]
Fix error message.

No issue #.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoExplicitly print the version numbers of virtualenv, pip, setuptools and wheel
Ward Vandewege [Mon, 11 Mar 2019 15:04:44 +0000 (11:04 -0400)]
Explicitly print the version numbers of virtualenv, pip, setuptools and wheel
when building Python packages.

refs #14936

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

5 years agoMerge branch '14938-unreliable-test'
Tom Clegg [Mon, 11 Mar 2019 04:34:33 +0000 (00:34 -0400)]
Merge branch '14938-unreliable-test'

fixes #14938

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14938: Fix unreliable test.
Tom Clegg [Mon, 11 Mar 2019 04:32:22 +0000 (00:32 -0400)]
14938: Fix unreliable test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years agoMerge branch '14880-keepdocker-argparse' refs #14880
Peter Amstutz [Fri, 8 Mar 2019 21:05:12 +0000 (16:05 -0500)]
Merge branch '14880-keepdocker-argparse' refs #14880

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14880: Fix help message typo.
Peter Amstutz [Fri, 8 Mar 2019 21:03:03 +0000 (16:03 -0500)]
14880: Fix help message typo.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago13937: Adds tests for operation & I/O counters.
Lucas Di Pentima [Fri, 8 Mar 2019 20:56:17 +0000 (17:56 -0300)]
13937: Adds tests for operation & I/O counters.

Added {"operation": "any"} to opsCounters so that it's easy to check on tests,
because the kind of operation depends on the driver and the tests were
implemented as part of the volume generic tests.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

5 years agoMerge branch '14291-cdc-aws' refs #14291
Peter Amstutz [Fri, 8 Mar 2019 20:19:59 +0000 (15:19 -0500)]
Merge branch '14291-cdc-aws' refs #14291

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: EbsVolumeType → EBSVolumeType
Peter Amstutz [Fri, 8 Mar 2019 20:18:44 +0000 (15:18 -0500)]
14291: EbsVolumeType → EBSVolumeType

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14880: Fix argument parsing issues
Peter Amstutz [Fri, 8 Mar 2019 15:14:27 +0000 (10:14 -0500)]
14880: Fix argument parsing issues

arv-keepdocker passes through some flags to arv-put.  The strategy it
uses is to remove known flags and pass through unknown flags. However
it doesn't know if an unknown flag takes a value or not, so it treats
the values as a positional argument rather than the value
corresponding to the unknown flag.

* Move positional arguments from "keepdocker_parser" to "arg_parser"
* Strip positional arguments (image name, tag) from arguments before
  parsing known keepdocker_parser flags.
* Additionally move a couple of flags that were mistakenly added to
  put.run_opts (and thus appearing in the arv-keepdocker help despite
  making no sense) that should have been in put.upload_opts

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Exclude both "shutting-down" and "terminated" instances.
Peter Amstutz [Thu, 7 Mar 2019 21:53:14 +0000 (16:53 -0500)]
14291: Exclude both "shutting-down" and "terminated" instances.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years ago14291: Generate key fingerprints that work with AWS. Filter out terminated instances
Peter Amstutz [Thu, 7 Mar 2019 21:27:18 +0000 (16:27 -0500)]
14291: Generate key fingerprints that work with AWS.  Filter out terminated instances

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

5 years agoMerge branch '14920-unknown-booting-race'
Tom Clegg [Thu, 7 Mar 2019 19:35:01 +0000 (14:35 -0500)]
Merge branch '14920-unknown-booting-race'

fixes #14920

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14920: Update metrics when worker state changes idle->running.
Tom Clegg [Thu, 7 Mar 2019 19:27:13 +0000 (14:27 -0500)]
14920: Update metrics when worker state changes idle->running.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

5 years ago14291: Add EbsVolumeType, also test fixes
Peter Amstutz [Thu, 7 Mar 2019 19:16:40 +0000 (14:16 -0500)]
14291: Add EbsVolumeType, also test fixes

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>