arvados.git
6 years ago13166: Limit wishlist to (max_nodes - up_nodes)
Lucas Di Pentima [Mon, 19 Mar 2018 14:42:30 +0000 (11:42 -0300)]
13166: Limit wishlist to (max_nodes - up_nodes)

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13166: Order slurm queue by priority
Lucas Di Pentima [Fri, 16 Mar 2018 20:32:18 +0000 (17:32 -0300)]
13166: Order slurm queue by priority

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '13138-request-error' refs #13138
Peter Amstutz [Thu, 15 Mar 2018 19:36:19 +0000 (15:36 -0400)]
Merge branch '13138-request-error' refs #13138

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoFix 'arvbox log' to propagate terminal size, no issue #
Peter Amstutz [Thu, 15 Mar 2018 19:28:02 +0000 (15:28 -0400)]
Fix 'arvbox log' to propagate terminal size, no issue #

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13100-crunch-run-memory'
Tom Clegg [Thu, 15 Mar 2018 19:12:38 +0000 (15:12 -0400)]
Merge branch '13100-crunch-run-memory'

refs #13100
refs #11583

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago11583: 13100: Fix memory leak in test suite.
Tom Clegg [Thu, 15 Mar 2018 16:45:52 +0000 (12:45 -0400)]
11583: 13100: Fix memory leak in test suite.

Something (an unterminated goroutine?) holds a reference to the keep
client, which (in the test suite) holds a reference to the last block
it wrote. Releasing the reference to the block reduces test suite
memory use by about 2.6 GB.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13100: Fix memory leak (64MiB per stream in output/log collection).
Tom Clegg [Thu, 15 Mar 2018 14:57:59 +0000 (10:57 -0400)]
13100: Fix memory leak (64MiB per stream in output/log collection).

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '12552-wf-priority'
Tom Clegg [Wed, 14 Mar 2018 19:10:56 +0000 (15:10 -0400)]
Merge branch '12552-wf-priority'

refs #12552

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13138: Gather arvados API errors under "RequestError" class.
Peter Amstutz [Wed, 14 Mar 2018 18:46:22 +0000 (14:46 -0400)]
13138: Gather arvados API errors under "RequestError" class.

Don't print stack trace for RequestError, because they are expected
conditions when there is an error in the request.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12552: Merge branch 'master' into 12552-wf-priority
Tom Clegg [Wed, 14 Mar 2018 18:15:15 +0000 (14:15 -0400)]
12552: Merge branch 'master' into 12552-wf-priority

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12552: Fix workflow parent priority test.
Tom Clegg [Wed, 14 Mar 2018 18:13:11 +0000 (14:13 -0400)]
12552: Fix workflow parent priority test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13135-cwl-secrets' closes #13135
Peter Amstutz [Wed, 14 Mar 2018 15:09:20 +0000 (11:09 -0400)]
Merge branch '13135-cwl-secrets' closes #13135

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13135: Code cleanup & comments.
Peter Amstutz [Wed, 14 Mar 2018 14:48:58 +0000 (10:48 -0400)]
13135: Code cleanup & comments.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoFix arvbox workbench for updated passenger, refs #12737
Peter Amstutz [Tue, 13 Mar 2018 20:23:17 +0000 (16:23 -0400)]
Fix arvbox workbench for updated passenger, refs #12737

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12552: Prioritize containers according to top-level requests.
Tom Clegg [Tue, 13 Mar 2018 14:44:15 +0000 (10:44 -0400)]
12552: Prioritize containers according to top-level requests.

This change ensures:

* a high-priority container and all of its descendants are prioritized
  ahead of lower-priority containers and their descendants.

* when two top-level containers are requested using equal priority,
  the earlier-submitted container has higher priority.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13135: Fix test
Peter Amstutz [Tue, 13 Mar 2018 17:31:36 +0000 (13:31 -0400)]
13135: Fix test

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13135: Update docs
Peter Amstutz [Tue, 13 Mar 2018 14:05:19 +0000 (10:05 -0400)]
13135: Update docs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13135: Add secrets test to arvados-tests
Peter Amstutz [Mon, 12 Mar 2018 20:45:39 +0000 (16:45 -0400)]
13135: Add secrets test to arvados-tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13135: Secrets not supported with --api=jobs
Peter Amstutz [Mon, 12 Mar 2018 19:18:50 +0000 (15:18 -0400)]
13135: Secrets not supported with --api=jobs

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13135: Update tests for secret_mounts
Peter Amstutz [Mon, 12 Mar 2018 18:37:56 +0000 (14:37 -0400)]
13135: Update tests for secret_mounts

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13135: Remove testing workarounds
Peter Amstutz [Mon, 12 Mar 2018 17:48:57 +0000 (13:48 -0400)]
13135: Remove testing workarounds

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13135: Handle secrets in file literals.
Peter Amstutz [Fri, 9 Mar 2018 21:57:42 +0000 (16:57 -0500)]
13135: Handle secrets in file literals.

Also check for secrets leaking into command line or environment, and
fail if detected.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13135: Secrets support WIP
Peter Amstutz [Fri, 9 Mar 2018 21:32:35 +0000 (16:32 -0500)]
13135: Secrets support WIP

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13134-secret-mounts' closes #13134
Peter Amstutz [Tue, 13 Mar 2018 19:32:14 +0000 (15:32 -0400)]
Merge branch '13134-secret-mounts' closes #13134

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13134: Check for conflicts between secret mounts and regular mounts
Peter Amstutz [Tue, 13 Mar 2018 15:37:03 +0000 (11:37 -0400)]
13134: Check for conflicts between secret mounts and regular mounts

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13134: Use the container token to fetch secret_mounts
Peter Amstutz [Mon, 12 Mar 2018 20:25:08 +0000 (16:25 -0400)]
13134: Use the container token to fetch secret_mounts

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13134: Get secret_mounts from separate API endpoint. Update tests.
Peter Amstutz [Mon, 12 Mar 2018 17:46:46 +0000 (13:46 -0400)]
13134: Get secret_mounts from separate API endpoint.  Update tests.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13134: Support for secret mounts in crunch-run.
Peter Amstutz [Fri, 9 Mar 2018 19:11:28 +0000 (14:11 -0500)]
13134: Support for secret mounts in crunch-run.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13143-secret-mounts'
Tom Clegg [Tue, 13 Mar 2018 17:08:29 +0000 (13:08 -0400)]
Merge branch '13143-secret-mounts'

refs #13143

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '12737-update-rails-wb'
Lucas Di Pentima [Tue, 13 Mar 2018 16:18:08 +0000 (13:18 -0300)]
Merge branch '12737-update-rails-wb'
Refs #12737

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '12369-fuse-driver-doc-updates'
Lucas Di Pentima [Tue, 13 Mar 2018 16:03:50 +0000 (13:03 -0300)]
Merge branch '12369-fuse-driver-doc-updates'
Closes #12369

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12369: Grammar fix
Lucas Di Pentima [Tue, 13 Mar 2018 16:02:51 +0000 (13:02 -0300)]
12369: Grammar fix

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13143: Update find_reusable() invocation in test suite.
Tom Clegg [Tue, 13 Mar 2018 14:47:43 +0000 (10:47 -0400)]
13143: Update find_reusable() invocation in test suite.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12369: Add fuse driver installation notes & update arv-mount tutorial.
Lucas Di Pentima [Mon, 12 Mar 2018 20:52:40 +0000 (17:52 -0300)]
12369: Add fuse driver installation notes & update arv-mount tutorial.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13143: Decode request body secret_mounts properly.
Tom Clegg [Mon, 12 Mar 2018 19:57:43 +0000 (15:57 -0400)]
13143: Decode request body secret_mounts properly.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Remove superfluous check for null hash.
Tom Clegg [Mon, 12 Mar 2018 17:35:53 +0000 (13:35 -0400)]
13143: Remove superfluous check for null hash.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Add missing "secret_mounts scrubbed" test.
Tom Clegg [Mon, 12 Mar 2018 17:07:51 +0000 (13:07 -0400)]
13143: Add missing "secret_mounts scrubbed" test.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Fix same/different-secret_mounts tests.
Tom Clegg [Mon, 12 Mar 2018 15:44:19 +0000 (11:44 -0400)]
13143: Fix same/different-secret_mounts tests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Sort hash keys for more realistic fixture md5.
Tom Clegg [Mon, 12 Mar 2018 15:43:22 +0000 (11:43 -0400)]
13143: Sort hash keys for more realistic fixture md5.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Ignore secret_mounts_md5 from fixtures, too.
Tom Clegg [Mon, 12 Mar 2018 15:43:03 +0000 (11:43 -0400)]
13143: Ignore secret_mounts_md5 from fixtures, too.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Prohibit identical path in mounts + secret_mounts.
Tom Clegg [Mon, 12 Mar 2018 15:23:48 +0000 (11:23 -0400)]
13143: Prohibit identical path in mounts + secret_mounts.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Merge branch 'master' into 13143-secret-mounts
Tom Clegg [Mon, 12 Mar 2018 14:43:26 +0000 (10:43 -0400)]
13143: Merge branch 'master' into 13143-secret-mounts

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Skip some unneeded deep_dup.
Tom Clegg [Mon, 12 Mar 2018 14:42:33 +0000 (10:42 -0400)]
13143: Skip some unneeded deep_dup.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Don't advertise secret_* columns in discovery doc.
Tom Clegg [Mon, 12 Mar 2018 14:42:01 +0000 (10:42 -0400)]
13143: Don't advertise secret_* columns in discovery doc.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Don't load secret_mounts from test fixtures.
Tom Clegg [Mon, 12 Mar 2018 14:24:08 +0000 (10:24 -0400)]
13143: Don't load secret_mounts from test fixtures.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Skip some unneeded database resets.
Tom Clegg [Mon, 12 Mar 2018 14:32:09 +0000 (10:32 -0400)]
13143: Skip some unneeded database resets.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoFix "priority" KeyError in crunch1 refs #12526
Peter Amstutz [Mon, 12 Mar 2018 13:43:40 +0000 (09:43 -0400)]
Fix "priority" KeyError in crunch1 refs #12526

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13143: Do not try to select container secret_mounts column.
Tom Clegg [Fri, 9 Mar 2018 21:47:19 +0000 (16:47 -0500)]
13143: Do not try to select container secret_mounts column.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12737: Try pinning all testing gems
Lucas Di Pentima [Fri, 9 Mar 2018 21:21:52 +0000 (18:21 -0300)]
12737: Try pinning all testing gems

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13143: Add secret_mounts field to containers and container requests.
Tom Clegg [Fri, 9 Mar 2018 20:52:16 +0000 (15:52 -0500)]
13143: Add secret_mounts field to containers and container requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoFix packaging for python-cwltest. refs #13140
Peter Amstutz [Fri, 9 Mar 2018 20:29:22 +0000 (15:29 -0500)]
Fix packaging for python-cwltest.  refs #13140

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12737: Avoid upgrading to selenium 3.x
Lucas Di Pentima [Fri, 9 Mar 2018 19:38:11 +0000 (16:38 -0300)]
12737: Avoid upgrading to selenium 3.x

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12737: Fix benchmark & integration tests
Lucas Di Pentima [Fri, 9 Mar 2018 18:49:03 +0000 (15:49 -0300)]
12737: Fix benchmark & integration tests

Minitest & angularjs-rails gems seem to have incompatible changes
on newer versions.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12737: Update workbench rails to 4.1.16 (latest 4.1.x)
Lucas Di Pentima [Wed, 7 Mar 2018 14:18:35 +0000 (11:18 -0300)]
12737: Update workbench rails to 4.1.16 (latest 4.1.x)

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '12737-update-rails-api'
Lucas Di Pentima [Fri, 9 Mar 2018 15:49:56 +0000 (12:49 -0300)]
Merge branch '12737-update-rails-api'
Refs #12737

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '13078-badconstraints'
Tom Clegg [Fri, 9 Mar 2018 06:40:53 +0000 (01:40 -0500)]
Merge branch '13078-badconstraints'

refs #13078

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13147-arvput-request-id'
Lucas Di Pentima [Thu, 8 Mar 2018 20:04:38 +0000 (17:04 -0300)]
Merge branch '13147-arvput-request-id'
Closes #13147

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoWhen running inside a virtualenv, pip still uses the ~/.cache/pip cache
Ward Vandewege [Thu, 8 Mar 2018 18:32:07 +0000 (13:32 -0500)]
When running inside a virtualenv, pip still uses the ~/.cache/pip cache
directory, unless it is explicitly overridden with XDG_CACHE_HOME.

In the test script, stop using pip's cache directory altogether (part 2).

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoFix arvados-cwl-runner --version argument (again) refs #12526
Peter Amstutz [Thu, 8 Mar 2018 16:14:35 +0000 (11:14 -0500)]
Fix arvados-cwl-runner --version argument (again) refs #12526

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13147: X-Request-Id logging updates when running arv-put.
Lucas Di Pentima [Thu, 8 Mar 2018 15:34:16 +0000 (12:34 -0300)]
13147: X-Request-Id logging updates when running arv-put.

Remove the initial request-id logging when starting the run.
Update the custom formatter so that it shows the request-id only once.
Add related tests.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoFix arvados-cwl-runner --version argument refs #12526
Peter Amstutz [Thu, 8 Mar 2018 15:17:11 +0000 (10:17 -0500)]
Fix arvados-cwl-runner --version argument refs #12526

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoUpdate version of cwltest package refs #13140
Peter Amstutz [Thu, 8 Mar 2018 14:55:25 +0000 (09:55 -0500)]
Update version of cwltest package refs #13140

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13078: Add test case for "release job held because BadConstraints".
Tom Clegg [Wed, 7 Mar 2018 22:54:35 +0000 (17:54 -0500)]
13078: Add test case for "release job held because BadConstraints".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoDocumention: fix spelling of --tempdir argument to arv-migrate-docker19
Ward Vandewege [Wed, 7 Mar 2018 19:40:36 +0000 (14:40 -0500)]
Documention: fix spelling of --tempdir argument to arv-migrate-docker19

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoPython sdk: do not try to do a bitwise operation on a float.
Ward Vandewege [Wed, 7 Mar 2018 19:35:51 +0000 (14:35 -0500)]
Python sdk: do not try to do a bitwise operation on a float.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago13078: Update comment.
Tom Clegg [Wed, 7 Mar 2018 15:09:20 +0000 (10:09 -0500)]
13078: Update comment.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Reduce "feature kludge" poll interval.
Tom Clegg [Wed, 7 Mar 2018 14:43:28 +0000 (09:43 -0500)]
13078: Reduce "feature kludge" poll interval.

Whenever a new node comes online, "scontrol reconfigure" invalidates
all jobs that could be scheduled onto that node *or any other node* so
it's worthwhile to recover from this quickly.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Fix jobs stuck in "held" state in old SLURM versions.
Tom Clegg [Wed, 7 Mar 2018 14:19:40 +0000 (09:19 -0500)]
13078: Fix jobs stuck in "held" state in old SLURM versions.

In SLURM 14 and 15, if a queued job has feature constraints which
become invalid (e.g., when "scontrol reconfigure" clears all node
features), the job is put on hold with priority=0, and it stays in
this state even after the features reappear. "scontrol release"
recovers a job from this state.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Fix ignoring queued jobs with reason=BadConstraints.
Tom Clegg [Wed, 7 Mar 2018 14:17:56 +0000 (09:17 -0500)]
13078: Fix ignoring queued jobs with reason=BadConstraints.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '12526-cwl-priority' closes #12526
Peter Amstutz [Wed, 7 Mar 2018 01:07:33 +0000 (20:07 -0500)]
Merge branch '12526-cwl-priority' closes #12526

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12737: Update API server to rails 4.2.10
Lucas Di Pentima [Tue, 20 Feb 2018 19:16:09 +0000 (16:16 -0300)]
12737: Update API server to rails 4.2.10

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13078: Ignore held jobs with priority=0 (e.g., SLURM 15.x).
Tom Clegg [Tue, 6 Mar 2018 22:14:24 +0000 (17:14 -0500)]
13078: Ignore held jobs with priority=0 (e.g., SLURM 15.x).

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Remove debug log.
Tom Clegg [Tue, 6 Mar 2018 22:11:35 +0000 (17:11 -0500)]
13078: Remove debug log.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13147: Append X-Request-Id on error & debug log messages.
Lucas Di Pentima [Tue, 6 Mar 2018 20:29:14 +0000 (17:29 -0300)]
13147: Append X-Request-Id on error & debug log messages.

Set up a custom formatter that replaces the standard arvados one
when running arv-put, so that it appends the Req-Id on errors.
Replacing 'arvados' log formatter makes all loggers below it (arv_put and
api in this case) use the same format.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13078: Ensure compute0 always has all the instancetype=X features.
Tom Clegg [Tue, 6 Mar 2018 19:52:29 +0000 (14:52 -0500)]
13078: Ensure compute0 always has all the instancetype=X features.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13078-cds-noheader' refs #13078
Peter Amstutz [Tue, 6 Mar 2018 18:49:41 +0000 (13:49 -0500)]
Merge branch '13078-cds-noheader' refs #13078

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13078: Add --noheader to squeue invocation.
Peter Amstutz [Tue, 6 Mar 2018 18:49:10 +0000 (13:49 -0500)]
13078: Add --noheader to squeue invocation.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '12268-build-fix'
Lucas Di Pentima [Tue, 6 Mar 2018 16:44:20 +0000 (13:44 -0300)]
Merge branch '12268-build-fix'
Refs #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12268: Add comment just in case we need to use a forked libcloud again.
Lucas Di Pentima [Tue, 6 Mar 2018 16:43:27 +0000 (13:43 -0300)]
12268: Add comment just in case we need to use a forked libcloud again.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12526: Add constant for default priority and check range.
Peter Amstutz [Tue, 6 Mar 2018 16:36:07 +0000 (11:36 -0500)]
12526: Add constant for default priority and check range.

Also fix a-c-r help text refs #12410

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12526: Fix tests
Peter Amstutz [Fri, 2 Mar 2018 20:05:47 +0000 (15:05 -0500)]
12526: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12526: add --priority for --api=containers
Peter Amstutz [Fri, 2 Mar 2018 14:53:39 +0000 (09:53 -0500)]
12526: add --priority for --api=containers

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12268: Fix package build scripts
Lucas Di Pentima [Tue, 6 Mar 2018 15:56:52 +0000 (12:56 -0300)]
12268: Fix package build scripts

Libcloud now is built from official sources instead of our fork.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12268: Fix merge conflict.
Lucas Di Pentima [Mon, 5 Mar 2018 21:58:17 +0000 (18:58 -0300)]
12268: Fix merge conflict.
Refs #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '12268-libcloud230-official'
Lucas Di Pentima [Mon, 5 Mar 2018 21:49:58 +0000 (18:49 -0300)]
Merge branch '12268-libcloud230-official'
Closes #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13147: Logger call fix.
Lucas Di Pentima [Mon, 5 Mar 2018 21:39:04 +0000 (18:39 -0300)]
13147: Logger call fix.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12268: Remove libcloud fork installation attempt on run-tests.sh
Lucas Di Pentima [Mon, 5 Mar 2018 21:16:03 +0000 (18:16 -0300)]
12268: Remove libcloud fork installation attempt on run-tests.sh

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12268: Update nodemanager dependency on libcloud 2.3.0
Lucas Di Pentima [Mon, 5 Mar 2018 20:20:01 +0000 (17:20 -0300)]
12268: Update nodemanager dependency on libcloud 2.3.0

As we're now using the official release, this update run-test.sh will
try to install from PyPI.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '13140-cwl-conformance' refs #13140
Peter Amstutz [Mon, 5 Mar 2018 18:38:43 +0000 (13:38 -0500)]
Merge branch '13140-cwl-conformance' refs #13140

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoWhen running inside a virtualenv, pip still uses the ~/.cache/pip cache
Ward Vandewege [Sun, 4 Mar 2018 02:58:06 +0000 (21:58 -0500)]
When running inside a virtualenv, pip still uses the ~/.cache/pip cache
directory, unless it is explicitly overridden with XDG_CACHE_HOME.

In the test script, stop using pip's cache directory altogether.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoStop running sdk/R install if sdk/R is not being tested. It is
Ward Vandewege [Sat, 3 Mar 2018 00:40:40 +0000 (19:40 -0500)]
Stop running sdk/R install if sdk/R is not being tested. It is
really expensive on a clean install (~6 minutes).

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago13140: Fix call to docker.get_image()
Peter Amstutz [Fri, 2 Mar 2018 21:12:17 +0000 (16:12 -0500)]
13140: Fix call to docker.get_image()

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12552: Remove superfluous arithmetic.
Tom Clegg [Fri, 2 Mar 2018 19:58:43 +0000 (14:58 -0500)]
12552: Remove superfluous arithmetic.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12552: Fix starvation in test case.
Tom Clegg [Fri, 2 Mar 2018 19:53:05 +0000 (14:53 -0500)]
12552: Fix starvation in test case.

refs #12552

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '12737-arvados-gem-dependency'
Lucas Di Pentima [Fri, 2 Mar 2018 19:22:46 +0000 (16:22 -0300)]
Merge branch '12737-arvados-gem-dependency'
Refs #12737

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13140: Fix tests
Peter Amstutz [Fri, 2 Mar 2018 19:18:10 +0000 (14:18 -0500)]
13140: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoThe crunch-dispatch-slurm binary has a new dependency on the
Ward Vandewege [Fri, 2 Mar 2018 19:04:11 +0000 (14:04 -0500)]
The crunch-dispatch-slurm binary has a new dependency on the
lib/dispatchcloud directory. Reflect that in the build scripts.

refs #13078

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago12737: Fix activesupport dependency from arvados gem.
Lucas Di Pentima [Thu, 1 Mar 2018 19:57:10 +0000 (16:57 -0300)]
12737: Fix activesupport dependency from arvados gem.

Upper version requirement was avoiding gem installation when upgrading rails
to a version >= 4.2.6.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13140: Bump cwltool version
Peter Amstutz [Fri, 2 Mar 2018 15:00:12 +0000 (10:00 -0500)]
13140: Bump cwltool version

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '12552-slurm-priority'
Tom Clegg [Thu, 1 Mar 2018 21:41:07 +0000 (16:41 -0500)]
Merge branch '12552-slurm-priority'

refs #12552

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>