arvados.git
6 years ago13143: Do not try to select container secret_mounts column.
Tom Clegg [Fri, 9 Mar 2018 21:47:19 +0000 (16:47 -0500)]
13143: Do not try to select container secret_mounts column.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13143: Add secret_mounts field to containers and container requests.
Tom Clegg [Fri, 9 Mar 2018 20:52:16 +0000 (15:52 -0500)]
13143: Add secret_mounts field to containers and container requests.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13078-badconstraints'
Tom Clegg [Fri, 9 Mar 2018 06:40:53 +0000 (01:40 -0500)]
Merge branch '13078-badconstraints'

refs #13078

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13147-arvput-request-id'
Lucas Di Pentima [Thu, 8 Mar 2018 20:04:38 +0000 (17:04 -0300)]
Merge branch '13147-arvput-request-id'
Closes #13147

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoWhen running inside a virtualenv, pip still uses the ~/.cache/pip cache
Ward Vandewege [Thu, 8 Mar 2018 18:32:07 +0000 (13:32 -0500)]
When running inside a virtualenv, pip still uses the ~/.cache/pip cache
directory, unless it is explicitly overridden with XDG_CACHE_HOME.

In the test script, stop using pip's cache directory altogether (part 2).

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoFix arvados-cwl-runner --version argument (again) refs #12526
Peter Amstutz [Thu, 8 Mar 2018 16:14:35 +0000 (11:14 -0500)]
Fix arvados-cwl-runner --version argument (again) refs #12526

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13147: X-Request-Id logging updates when running arv-put.
Lucas Di Pentima [Thu, 8 Mar 2018 15:34:16 +0000 (12:34 -0300)]
13147: X-Request-Id logging updates when running arv-put.

Remove the initial request-id logging when starting the run.
Update the custom formatter so that it shows the request-id only once.
Add related tests.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoFix arvados-cwl-runner --version argument refs #12526
Peter Amstutz [Thu, 8 Mar 2018 15:17:11 +0000 (10:17 -0500)]
Fix arvados-cwl-runner --version argument refs #12526

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoUpdate version of cwltest package refs #13140
Peter Amstutz [Thu, 8 Mar 2018 14:55:25 +0000 (09:55 -0500)]
Update version of cwltest package refs #13140

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13078: Add test case for "release job held because BadConstraints".
Tom Clegg [Wed, 7 Mar 2018 22:54:35 +0000 (17:54 -0500)]
13078: Add test case for "release job held because BadConstraints".

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoDocumention: fix spelling of --tempdir argument to arv-migrate-docker19
Ward Vandewege [Wed, 7 Mar 2018 19:40:36 +0000 (14:40 -0500)]
Documention: fix spelling of --tempdir argument to arv-migrate-docker19

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoPython sdk: do not try to do a bitwise operation on a float.
Ward Vandewege [Wed, 7 Mar 2018 19:35:51 +0000 (14:35 -0500)]
Python sdk: do not try to do a bitwise operation on a float.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago13078: Update comment.
Tom Clegg [Wed, 7 Mar 2018 15:09:20 +0000 (10:09 -0500)]
13078: Update comment.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Reduce "feature kludge" poll interval.
Tom Clegg [Wed, 7 Mar 2018 14:43:28 +0000 (09:43 -0500)]
13078: Reduce "feature kludge" poll interval.

Whenever a new node comes online, "scontrol reconfigure" invalidates
all jobs that could be scheduled onto that node *or any other node* so
it's worthwhile to recover from this quickly.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Fix jobs stuck in "held" state in old SLURM versions.
Tom Clegg [Wed, 7 Mar 2018 14:19:40 +0000 (09:19 -0500)]
13078: Fix jobs stuck in "held" state in old SLURM versions.

In SLURM 14 and 15, if a queued job has feature constraints which
become invalid (e.g., when "scontrol reconfigure" clears all node
features), the job is put on hold with priority=0, and it stays in
this state even after the features reappear. "scontrol release"
recovers a job from this state.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Fix ignoring queued jobs with reason=BadConstraints.
Tom Clegg [Wed, 7 Mar 2018 14:17:56 +0000 (09:17 -0500)]
13078: Fix ignoring queued jobs with reason=BadConstraints.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '12526-cwl-priority' closes #12526
Peter Amstutz [Wed, 7 Mar 2018 01:07:33 +0000 (20:07 -0500)]
Merge branch '12526-cwl-priority' closes #12526

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13078: Ignore held jobs with priority=0 (e.g., SLURM 15.x).
Tom Clegg [Tue, 6 Mar 2018 22:14:24 +0000 (17:14 -0500)]
13078: Ignore held jobs with priority=0 (e.g., SLURM 15.x).

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Remove debug log.
Tom Clegg [Tue, 6 Mar 2018 22:11:35 +0000 (17:11 -0500)]
13078: Remove debug log.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13147: Append X-Request-Id on error & debug log messages.
Lucas Di Pentima [Tue, 6 Mar 2018 20:29:14 +0000 (17:29 -0300)]
13147: Append X-Request-Id on error & debug log messages.

Set up a custom formatter that replaces the standard arvados one
when running arv-put, so that it appends the Req-Id on errors.
Replacing 'arvados' log formatter makes all loggers below it (arv_put and
api in this case) use the same format.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13078: Ensure compute0 always has all the instancetype=X features.
Tom Clegg [Tue, 6 Mar 2018 19:52:29 +0000 (14:52 -0500)]
13078: Ensure compute0 always has all the instancetype=X features.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13078-cds-noheader' refs #13078
Peter Amstutz [Tue, 6 Mar 2018 18:49:41 +0000 (13:49 -0500)]
Merge branch '13078-cds-noheader' refs #13078

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13078: Add --noheader to squeue invocation.
Peter Amstutz [Tue, 6 Mar 2018 18:49:10 +0000 (13:49 -0500)]
13078: Add --noheader to squeue invocation.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '12268-build-fix'
Lucas Di Pentima [Tue, 6 Mar 2018 16:44:20 +0000 (13:44 -0300)]
Merge branch '12268-build-fix'
Refs #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12268: Add comment just in case we need to use a forked libcloud again.
Lucas Di Pentima [Tue, 6 Mar 2018 16:43:27 +0000 (13:43 -0300)]
12268: Add comment just in case we need to use a forked libcloud again.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12526: Add constant for default priority and check range.
Peter Amstutz [Tue, 6 Mar 2018 16:36:07 +0000 (11:36 -0500)]
12526: Add constant for default priority and check range.

Also fix a-c-r help text refs #12410

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12526: Fix tests
Peter Amstutz [Fri, 2 Mar 2018 20:05:47 +0000 (15:05 -0500)]
12526: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12526: add --priority for --api=containers
Peter Amstutz [Fri, 2 Mar 2018 14:53:39 +0000 (09:53 -0500)]
12526: add --priority for --api=containers

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12268: Fix package build scripts
Lucas Di Pentima [Tue, 6 Mar 2018 15:56:52 +0000 (12:56 -0300)]
12268: Fix package build scripts

Libcloud now is built from official sources instead of our fork.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12268: Fix merge conflict.
Lucas Di Pentima [Mon, 5 Mar 2018 21:58:17 +0000 (18:58 -0300)]
12268: Fix merge conflict.
Refs #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '12268-libcloud230-official'
Lucas Di Pentima [Mon, 5 Mar 2018 21:49:58 +0000 (18:49 -0300)]
Merge branch '12268-libcloud230-official'
Closes #12268

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13147: Logger call fix.
Lucas Di Pentima [Mon, 5 Mar 2018 21:39:04 +0000 (18:39 -0300)]
13147: Logger call fix.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12268: Remove libcloud fork installation attempt on run-tests.sh
Lucas Di Pentima [Mon, 5 Mar 2018 21:16:03 +0000 (18:16 -0300)]
12268: Remove libcloud fork installation attempt on run-tests.sh

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12268: Update nodemanager dependency on libcloud 2.3.0
Lucas Di Pentima [Mon, 5 Mar 2018 20:20:01 +0000 (17:20 -0300)]
12268: Update nodemanager dependency on libcloud 2.3.0

As we're now using the official release, this update run-test.sh will
try to install from PyPI.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoMerge branch '13140-cwl-conformance' refs #13140
Peter Amstutz [Mon, 5 Mar 2018 18:38:43 +0000 (13:38 -0500)]
Merge branch '13140-cwl-conformance' refs #13140

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoWhen running inside a virtualenv, pip still uses the ~/.cache/pip cache
Ward Vandewege [Sun, 4 Mar 2018 02:58:06 +0000 (21:58 -0500)]
When running inside a virtualenv, pip still uses the ~/.cache/pip cache
directory, unless it is explicitly overridden with XDG_CACHE_HOME.

In the test script, stop using pip's cache directory altogether.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoStop running sdk/R install if sdk/R is not being tested. It is
Ward Vandewege [Sat, 3 Mar 2018 00:40:40 +0000 (19:40 -0500)]
Stop running sdk/R install if sdk/R is not being tested. It is
really expensive on a clean install (~6 minutes).

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago13140: Fix call to docker.get_image()
Peter Amstutz [Fri, 2 Mar 2018 21:12:17 +0000 (16:12 -0500)]
13140: Fix call to docker.get_image()

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12552: Remove superfluous arithmetic.
Tom Clegg [Fri, 2 Mar 2018 19:58:43 +0000 (14:58 -0500)]
12552: Remove superfluous arithmetic.

No issue #

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12552: Fix starvation in test case.
Tom Clegg [Fri, 2 Mar 2018 19:53:05 +0000 (14:53 -0500)]
12552: Fix starvation in test case.

refs #12552

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '12737-arvados-gem-dependency'
Lucas Di Pentima [Fri, 2 Mar 2018 19:22:46 +0000 (16:22 -0300)]
Merge branch '12737-arvados-gem-dependency'
Refs #12737

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13140: Fix tests
Peter Amstutz [Fri, 2 Mar 2018 19:18:10 +0000 (14:18 -0500)]
13140: Fix tests

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoThe crunch-dispatch-slurm binary has a new dependency on the
Ward Vandewege [Fri, 2 Mar 2018 19:04:11 +0000 (14:04 -0500)]
The crunch-dispatch-slurm binary has a new dependency on the
lib/dispatchcloud directory. Reflect that in the build scripts.

refs #13078

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years ago12737: Fix activesupport dependency from arvados gem.
Lucas Di Pentima [Thu, 1 Mar 2018 19:57:10 +0000 (16:57 -0300)]
12737: Fix activesupport dependency from arvados gem.

Upper version requirement was avoiding gem installation when upgrading rails
to a version >= 4.2.6.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13140: Bump cwltool version
Peter Amstutz [Fri, 2 Mar 2018 15:00:12 +0000 (10:00 -0500)]
13140: Bump cwltool version

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '12552-slurm-priority'
Tom Clegg [Thu, 1 Mar 2018 21:41:07 +0000 (16:41 -0500)]
Merge branch '12552-slurm-priority'

refs #12552

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '11850-singlecontainer-max-requirements'
jiayong2 [Wed, 28 Feb 2018 17:47:15 +0000 (17:47 +0000)]
Merge branch '11850-singlecontainer-max-requirements'

closes #11850

Arvados-DCO-1.1-Signed-off-by: Jiayong Li <jiayong@math.mit.edu>

6 years agoMerge branch 'master' into 11850-singlecontainer-max-requirements
jiayong2 [Wed, 28 Feb 2018 17:39:42 +0000 (17:39 +0000)]
Merge branch 'master' into 11850-singlecontainer-max-requirements

Arvados-DCO-1.1-Signed-off-by: Jiayong Li <jiayong@math.mit.edu>

6 years agoMerge branch '13078-slurm-feature-test'
Tom Clegg [Wed, 28 Feb 2018 16:27:29 +0000 (11:27 -0500)]
Merge branch '13078-slurm-feature-test'

refs #13078

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12552: Improve PrioritySpread docs.
Tom Clegg [Wed, 28 Feb 2018 16:27:12 +0000 (11:27 -0500)]
12552: Improve PrioritySpread docs.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Accept "no nodes" as an indication that features are OK.
Tom Clegg [Tue, 27 Feb 2018 22:09:20 +0000 (17:09 -0500)]
13078: Accept "no nodes" as an indication that features are OK.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13078: Use --test-only instead of invalid --gres to check features.
Tom Clegg [Tue, 27 Feb 2018 22:01:19 +0000 (17:01 -0500)]
13078: Use --test-only instead of invalid --gres to check features.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '12315-exclude-child-procs'
Tom Clegg [Tue, 27 Feb 2018 18:05:13 +0000 (13:05 -0500)]
Merge branch '12315-exclude-child-procs'

closes #12315

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12315: Tidy up test params.
Tom Clegg [Tue, 27 Feb 2018 16:08:19 +0000 (11:08 -0500)]
12315: Tidy up test params.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13037-composer-link' closes #13037
Peter Amstutz [Mon, 26 Feb 2018 21:27:08 +0000 (16:27 -0500)]
Merge branch '13037-composer-link' closes #13037

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch 'master' into 11850-singlecontainer-max-requirements
jiayong2 [Mon, 26 Feb 2018 20:11:08 +0000 (20:11 +0000)]
Merge branch 'master' into 11850-singlecontainer-max-requirements

Arvados-DCO-1.1-Signed-off-by: Jiayong Li <jiayong@math.mit.edu>

6 years agoTake sum for outdir requirement in single container
jiayong2 [Mon, 26 Feb 2018 20:06:59 +0000 (20:06 +0000)]
Take sum for outdir requirement in single container

Arvados-DCO-1.1-Signed-off-by: Jiayong Li <jiayong@math.mit.edu>

6 years agoMerge branch '12707-storage-classes-columns'
Lucas Di Pentima [Mon, 26 Feb 2018 18:22:44 +0000 (15:22 -0300)]
Merge branch '12707-storage-classes-columns'
Closes #12707

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12707: Adds default value setting on collection creation.
Lucas Di Pentima [Mon, 26 Feb 2018 17:26:41 +0000 (14:26 -0300)]
12707: Adds default value setting on collection creation.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12552: Fix accidental textile link syntax.
Tom Clegg [Mon, 26 Feb 2018 16:14:18 +0000 (11:14 -0500)]
12552: Fix accidental textile link syntax.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13037: Add hidden api_token to Composer button to skip extra login
Peter Amstutz [Wed, 14 Feb 2018 16:22:57 +0000 (11:22 -0500)]
13037: Add hidden api_token to Composer button to skip extra login

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12552: Document PrioritySpread. Default to 10 if not configured.
Tom Clegg [Mon, 26 Feb 2018 15:42:16 +0000 (10:42 -0500)]
12552: Document PrioritySpread. Default to 10 if not configured.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12552: Skip extra lookup.
Tom Clegg [Mon, 26 Feb 2018 14:46:39 +0000 (09:46 -0500)]
12552: Skip extra lookup.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12552: Add slurm-renice test cases.
Tom Clegg [Mon, 26 Feb 2018 14:46:31 +0000 (09:46 -0500)]
12552: Add slurm-renice test cases.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12707: Some fixes and additions on storage classes:
Lucas Di Pentima [Fri, 23 Feb 2018 18:09:30 +0000 (15:09 -0300)]
12707: Some fixes and additions on storage classes:

* Remove the unnecessary default values from model.
* Fixes and tests on storage classes columns.
* Adds tests for 'exists' filter on jsonb array columns.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago12315: Propagate show_children to infinite scroll pages. Fix & add tests.
Peter Amstutz [Fri, 23 Feb 2018 21:29:06 +0000 (16:29 -0500)]
12315: Propagate show_children to infinite scroll pages.  Fix & add tests.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12315: Add abstract show_children query parameter
Peter Amstutz [Fri, 23 Feb 2018 19:31:40 +0000 (14:31 -0500)]
12315: Add abstract show_children query parameter

Can't use requesting_container_uuid filter directly because it breaks
query of jobs and pipelines.  Add abstract show_children query
parameter.  Default is to hide children processes.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch 'wtsi/13113-acr-collectioncache-retries' refs #13113
Peter Amstutz [Fri, 23 Feb 2018 18:28:50 +0000 (13:28 -0500)]
Merge branch 'wtsi/13113-acr-collectioncache-retries' refs #13113

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch 'wtsi/13093-crunch-dispatch-slurm-add-mem' refs #13093
Peter Amstutz [Fri, 23 Feb 2018 18:26:59 +0000 (13:26 -0500)]
Merge branch 'wtsi/13093-crunch-dispatch-slurm-add-mem' refs #13093

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13064-arv-mount-perf' closes #13064
Peter Amstutz [Fri, 23 Feb 2018 16:15:44 +0000 (11:15 -0500)]
Merge branch '13064-arv-mount-perf' closes #13064

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13064: Add comment about has_ref() and in_use() checks
Peter Amstutz [Fri, 23 Feb 2018 16:15:03 +0000 (11:15 -0500)]
13064: Add comment about has_ref() and in_use() checks

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12707: Add storage classes attributes to collections.
Lucas Di Pentima [Fri, 16 Feb 2018 23:47:49 +0000 (20:47 -0300)]
12707: Add storage classes attributes to collections.

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years ago13064: Explicitly precompile regexes used in manifest parsing
Peter Amstutz [Thu, 22 Feb 2018 20:58:56 +0000 (15:58 -0500)]
13064: Explicitly precompile regexes used in manifest parsing

Seems to improve parsing time roughly 5-10%

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13064: Add __slots__ to ArvadosFile and FuseArvadosFile
Peter Amstutz [Thu, 22 Feb 2018 19:47:42 +0000 (14:47 -0500)]
13064: Add __slots__ to ArvadosFile and FuseArvadosFile

More than 50% decrease in memory footprint for large collections!!!

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13064: Check has_ref before checking in_use
Peter Amstutz [Thu, 22 Feb 2018 19:09:33 +0000 (14:09 -0500)]
13064: Check has_ref before checking in_use

Kernel behavior seems to be that if a file is referenced, its parents
remain referenced too.  This means has_ref() exits early when a
collection is not candidate for eviction.

By contrast, in_use() doesn't increment references on parents, so it
requires a full tree walk to determine if a collection is a candidate
for eviction.

Switching the order of the checks retains correctness but greatly
improves real-world performance.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoadd ReserveExtraRAM config option to crunch-dispatch-slurm
Joshua C. Randall [Thu, 15 Feb 2018 18:50:19 +0000 (18:50 +0000)]
add ReserveExtraRAM config option to crunch-dispatch-slurm

Arvados-DCO-1.1-Signed-off-by: Joshua C. Randall <jcrandall@alum.mit.edu>

6 years agoFix CollectionCache to pass num_retries to CollectionReader
Joshua Randall [Wed, 21 Feb 2018 20:51:56 +0000 (20:51 +0000)]
Fix CollectionCache to pass num_retries to CollectionReader

Arvados-DCO-1.1-Signed-off-by: Joshua C. Randall <jcrandall@alum.mit.edu>

6 years agoMerge branch '11645-keepstore-storageclasses' closes #11645
Peter Amstutz [Wed, 21 Feb 2018 20:46:32 +0000 (15:46 -0500)]
Merge branch '11645-keepstore-storageclasses' closes #11645

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago11645: Add tests that StorageClasses is loaded from config
Peter Amstutz [Wed, 21 Feb 2018 20:42:48 +0000 (15:42 -0500)]
11645: Add tests that StorageClasses is loaded from config

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoFix typos/grammar.
Ward Vandewege [Wed, 21 Feb 2018 20:13:02 +0000 (15:13 -0500)]
Fix typos/grammar.

No issue #

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <wvandewege@veritasgenetics.com>

6 years agoMerge branch '13106-keepclient-slot' closes #13106
Peter Amstutz [Wed, 21 Feb 2018 17:06:04 +0000 (12:06 -0500)]
Merge branch '13106-keepclient-slot' closes #13106

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13106: Add note about the test.
Peter Amstutz [Wed, 21 Feb 2018 17:05:36 +0000 (12:05 -0500)]
13106: Add note about the test.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago11645: Add support for StorageClasses in volume config.
Peter Amstutz [Wed, 21 Feb 2018 16:56:39 +0000 (11:56 -0500)]
11645: Add support for StorageClasses in volume config.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13106: Add test confirming deadlock
Peter Amstutz [Wed, 21 Feb 2018 01:32:44 +0000 (20:32 -0500)]
13106: Add test confirming deadlock

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13106: Ensure cache slot is reset on error
Peter Amstutz [Tue, 20 Feb 2018 21:29:22 +0000 (16:29 -0500)]
13106: Ensure cache slot is reset on error

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12315: Add "exclude child processes" checkbox to /all_processes.
Tom Clegg [Sat, 17 Feb 2018 04:18:17 +0000 (23:18 -0500)]
12315: Add "exclude child processes" checkbox to /all_processes.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13098-assign-hostname'
Tom Clegg [Fri, 16 Feb 2018 19:09:30 +0000 (14:09 -0500)]
Merge branch '13098-assign-hostname'

refs #13098
refs #12199

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13098: 12199: Fix hostname assigned but not saved.
Tom Clegg [Fri, 16 Feb 2018 18:55:24 +0000 (13:55 -0500)]
13098: 12199: Fix hostname assigned but not saved.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13063-update-azure-sdk'
Tom Clegg [Fri, 16 Feb 2018 17:48:34 +0000 (12:48 -0500)]
Merge branch '13063-update-azure-sdk'

refs #13063

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago13063: Use upstream azure-sdk-for-go.
Tom Clegg [Fri, 16 Feb 2018 17:41:48 +0000 (12:41 -0500)]
13063: Use upstream azure-sdk-for-go.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '13090-eval-timeout' closes #13090
Peter Amstutz [Fri, 16 Feb 2018 16:41:01 +0000 (11:41 -0500)]
Merge branch '13090-eval-timeout'  closes #13090

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago13090: Pass through --eval-timeout to container command line.
Peter Amstutz [Fri, 16 Feb 2018 15:53:49 +0000 (10:53 -0500)]
13090: Pass through --eval-timeout to container command line.

Also adds a check so that --ignore-docker-for-reuse can't be used with
containers, refs #12666

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago12552: Adjust SLURM niceness to bring priority into desired order.
Tom Clegg [Fri, 16 Feb 2018 07:29:22 +0000 (02:29 -0500)]
12552: Adjust SLURM niceness to bring priority into desired order.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years ago12552: Add SLURM niceness calculator.
Tom Clegg [Thu, 15 Feb 2018 23:17:52 +0000 (18:17 -0500)]
12552: Add SLURM niceness calculator.

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoAdd missing parameter to update call.
Tom Clegg [Thu, 15 Feb 2018 20:31:33 +0000 (15:31 -0500)]
Add missing parameter to update call.

fixes #12199

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '11876-R-deps' refs #11876
Peter Amstutz [Thu, 15 Feb 2018 19:51:43 +0000 (14:51 -0500)]
Merge branch '11876-R-deps' refs #11876

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years ago11876: Pin R deps that are installed from gitub.
Peter Amstutz [Thu, 15 Feb 2018 18:34:11 +0000 (13:34 -0500)]
11876: Pin R deps that are installed from gitub.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>

6 years agoMerge branch '13063-update-azure-sdk'
Tom Clegg [Thu, 15 Feb 2018 18:38:55 +0000 (13:38 -0500)]
Merge branch '13063-update-azure-sdk'

closes #13063

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tclegg@veritasgenetics.com>

6 years agoMerge branch '11454-remove-button-removal'
Lucas Di Pentima [Thu, 15 Feb 2018 16:53:26 +0000 (13:53 -0300)]
Merge branch '11454-remove-button-removal'
Refs #11454

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>

6 years agoR SDK documentation link fix refs #11876
Peter Amstutz [Thu, 15 Feb 2018 16:07:24 +0000 (11:07 -0500)]
R SDK documentation link fix refs #11876

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pamstutz@veritasgenetics.com>