arvados.git
9 years agoMerge branch 'master' into 6203-collection-perf-api
radhika [Wed, 3 Jun 2015 19:11:20 +0000 (15:11 -0400)]
Merge branch 'master' into 6203-collection-perf-api

9 years ago6203: Do not use Keep::Locator.parse to parse locator in some of the most expensive...
radhika [Wed, 3 Jun 2015 19:08:56 +0000 (15:08 -0400)]
6203: Do not use Keep::Locator.parse to parse locator in some of the most expensive paths.

9 years agoMerge branch '6146-log-squeue-lost-tasks' refs #6146
Tom Clegg [Wed, 3 Jun 2015 13:50:55 +0000 (09:50 -0400)]
Merge branch '6146-log-squeue-lost-tasks' refs #6146

9 years ago6146: Better log message.
Tom Clegg [Sun, 31 May 2015 09:48:22 +0000 (05:48 -0400)]
6146: Better log message.

9 years ago6146: Use new SLURM_JOB_ID env var instead of old SLURM_JOBID
Tom Clegg [Sun, 31 May 2015 09:32:55 +0000 (05:32 -0400)]
6146: Use new SLURM_JOB_ID env var instead of old SLURM_JOBID

9 years ago6146: Improvements to "kill srun process if slurm task disappears" feature:
Tom Clegg [Sun, 31 May 2015 06:02:19 +0000 (02:02 -0400)]
6146: Improvements to "kill srun process if slurm task disappears" feature:

* Log when we notice a process is orphaned.

* Log when we decide to kill an orphaned process.

* Use `squeue --jobs $SLURM_JOBID` so slurm doesn't have to tell us
  about other jobs' tasks.

* Do not kill a process that is still reporting stderr.

* Do not check `squeue` at all if every process has reported stderr
  since the last squeue check. (In such cases, it seems safe to assume
  no children are hung/dead.)

* Use the same timer/interval (15 seconds) for both noticing and
  killing orphaned processes.

9 years agoMerge branch 'master' into 6203-collection-perf-api
radhika [Wed, 3 Jun 2015 02:26:22 +0000 (22:26 -0400)]
Merge branch 'master' into 6203-collection-perf-api

9 years agorefs #6093
radhika [Wed, 3 Jun 2015 02:19:54 +0000 (22:19 -0400)]
refs #6093
Merge branch '6093-refresh-docs'

9 years ago6093: delete the redundant details in "alternate way to add ssh keys" section.
radhika [Wed, 3 Jun 2015 02:17:30 +0000 (22:17 -0400)]
6093: delete the redundant details in "alternate way to add ssh keys" section.

9 years agoMerge branch 'master' into 6093-refresh-docs
radhika [Wed, 3 Jun 2015 02:08:42 +0000 (22:08 -0400)]
Merge branch 'master' into 6093-refresh-docs

9 years agocloses #5930. Merge branch '5930-smalldocfix'
Nancy Ouyang [Wed, 3 Jun 2015 00:09:20 +0000 (20:09 -0400)]
closes #5930. Merge branch '5930-smalldocfix'

9 years ago5930: fixed as per code review
Nancy Ouyang [Wed, 3 Jun 2015 00:08:16 +0000 (20:08 -0400)]
5930: fixed as per code review

9 years agoMerge branch '6194-python-arvfile-large-write' closes #6194
Peter Amstutz [Tue, 2 Jun 2015 20:23:22 +0000 (16:23 -0400)]
Merge branch '6194-python-arvfile-large-write' closes #6194

9 years ago6194: Simplify test_large_write a little bit.
Peter Amstutz [Tue, 2 Jun 2015 20:17:31 +0000 (16:17 -0400)]
6194: Simplify test_large_write a little bit.

9 years agoMerge branch 'master' into 6093-refresh-docs
radhika [Tue, 2 Jun 2015 19:39:41 +0000 (15:39 -0400)]
Merge branch 'master' into 6093-refresh-docs

9 years ago6194: Make splitting loop simpler since [n:n+KEEP_BLOCK_SIZE] returns a short
Peter Amstutz [Tue, 2 Jun 2015 17:32:19 +0000 (13:32 -0400)]
6194: Make splitting loop simpler since [n:n+KEEP_BLOCK_SIZE] returns a short
slice when there isn't KEEP_BLOCK_SIZE data.  Update test.

9 years agoMerge branch 'master' into 6087-collection-timing
radhika [Tue, 2 Jun 2015 13:44:08 +0000 (09:44 -0400)]
Merge branch 'master' into 6087-collection-timing

9 years agoMerge branch 'master' into 6194-python-arvfile-large-write
Peter Amstutz [Mon, 1 Jun 2015 20:57:54 +0000 (16:57 -0400)]
Merge branch 'master' into 6194-python-arvfile-large-write

9 years ago6194: Fix test. Lots of small writes break across blocks differently than one huge
Peter Amstutz [Mon, 1 Jun 2015 20:56:15 +0000 (16:56 -0400)]
6194: Fix test.  Lots of small writes break across blocks differently than one huge
one.

9 years agoMerge branch 'master' into 6087-collection-timing
radhika [Mon, 1 Jun 2015 14:05:56 +0000 (10:05 -0400)]
Merge branch 'master' into 6087-collection-timing

9 years ago6194: Fix typo in invocation of writeto() and use memoryview to avoid copying slices.
Peter Amstutz [Mon, 1 Jun 2015 12:51:35 +0000 (08:51 -0400)]
6194: Fix typo in invocation of writeto() and use memoryview to avoid copying slices.

9 years agoRemove non-existent migration from structure.sql. refs #3036
Tom Clegg [Sun, 31 May 2015 12:39:20 +0000 (08:39 -0400)]
Remove non-existent migration from structure.sql. refs #3036

9 years agoUpdate example dns_server_update_command. refs #6146
Tom Clegg [Sun, 31 May 2015 12:38:52 +0000 (08:38 -0400)]
Update example dns_server_update_command. refs #6146

9 years agoMerge branch '6146-dns-update-command' refs #6146
Tom Clegg [Sun, 31 May 2015 12:29:51 +0000 (08:29 -0400)]
Merge branch '6146-dns-update-command' refs #6146

9 years ago6146: Add dns_server_update_command. Update docs & tests for DNS update hooks.
Tom Clegg [Sun, 31 May 2015 12:23:44 +0000 (08:23 -0400)]
6146: Add dns_server_update_command. Update docs & tests for DNS update hooks.

9 years agoTell tar to read to EOF (even if it detects trailing NULs).
Tom Clegg [Sat, 30 May 2015 02:01:06 +0000 (22:01 -0400)]
Tell tar to read to EOF (even if it detects trailing NULs).

Avoids SIGPIPE when feeding a tarball made with tar -A.

refs #6146 refs #6094

9 years ago6194: Chunk large ArvadosFile writes automatically instead of raising an error.
Peter Amstutz [Fri, 29 May 2015 20:28:37 +0000 (16:28 -0400)]
6194: Chunk large ArvadosFile writes automatically instead of raising an error.

9 years agoMerge branch '6146-ignore-tar-sigpipe' refs #6146 refs #6094
Tom Clegg [Fri, 29 May 2015 18:04:01 +0000 (14:04 -0400)]
Merge branch '6146-ignore-tar-sigpipe' refs #6146 refs #6094

9 years ago6146: Ignore SIGPIPE while feeding data to tar. Rely on close() retval instead.
Tom Clegg [Fri, 29 May 2015 17:32:40 +0000 (13:32 -0400)]
6146: Ignore SIGPIPE while feeding data to tar. Rely on close() retval instead.

9 years agoIn install script, log archive hash before running tar. refs #6146
Tom Clegg [Fri, 29 May 2015 16:02:03 +0000 (12:02 -0400)]
In install script, log archive hash before running tar. refs #6146

9 years agoMerge branch '6146-job-runtime-sanity' refs #6146
Tom Clegg [Fri, 29 May 2015 15:31:43 +0000 (11:31 -0400)]
Merge branch '6146-job-runtime-sanity' refs #6146

9 years ago6146: Exit TEMPFAIL early (without failing the job) if worker nodes cannot run a...
Tom Clegg [Thu, 28 May 2015 21:13:44 +0000 (17:13 -0400)]
6146: Exit TEMPFAIL early (without failing the job) if worker nodes cannot run a trivial command.

This is meant to improve the way we handle a couple of edge cases.

1. A worker node doesn't get bootstrapped properly. It works well
enough to persuade nodemanager and the API server that it's alive and
ready to run jobs, but it can't actually run jobs. This means there's
a bug in the bootstrapping process -- its startup script shouldn't
tell slurm State=RESUME without checking itself -- but even so this
doesn't deserve to fail a job: it's definitely a system problem,
there's zero chance a different job would have gone any differently.

2. A worker node has a hardware problem, or it has fallen off the
network, or something like that, but slurm hasn't yet noticed and set
its state to DOWN, so slurm still uses it to satisfy crunch-dispatch's
"salloc" commands. As above, there's zero chance this could have gone
differently for any other job, so it doesn't make sense to fail the
job.

9 years agoMerge branch 'master' into 6087-collection-timing
radhika [Thu, 28 May 2015 18:53:44 +0000 (14:53 -0400)]
Merge branch 'master' into 6087-collection-timing

9 years agoMerge branch '6146-retry-install' refs #6146
Tom Clegg [Wed, 27 May 2015 20:10:24 +0000 (16:10 -0400)]
Merge branch '6146-retry-install' refs #6146

9 years ago6146: Retry install (max 3 attempts) if install script fails with no error messages.
Tom Clegg [Wed, 27 May 2015 19:48:54 +0000 (15:48 -0400)]
6146: Retry install (max 3 attempts) if install script fails with no error messages.

Also: if install fails, croak() instead of exit(1) so we still get a log file.

9 years agoMerge branch 'master' into 6093-refresh-docs
radhika [Wed, 27 May 2015 19:38:59 +0000 (15:38 -0400)]
Merge branch 'master' into 6093-refresh-docs

Conflicts:
doc/user/tutorials/tutorial-submit-job.html.textile.liquid

9 years agocloses #6057
radhika [Wed, 27 May 2015 19:27:36 +0000 (15:27 -0400)]
closes #6057
Merge branch '6057-public-projects-page'

9 years ago6057: few more minor tweaks
radhika [Wed, 27 May 2015 19:26:20 +0000 (15:26 -0400)]
6057: few more minor tweaks

9 years agoMerge branch '6141-doc-workbench-links' refs #6141
Peter Amstutz [Wed, 27 May 2015 19:11:58 +0000 (15:11 -0400)]
Merge branch '6141-doc-workbench-links' refs #6141

9 years agoMerge branch 'master' into 6057-public-projects-page
radhika [Wed, 27 May 2015 19:03:54 +0000 (15:03 -0400)]
Merge branch 'master' into 6057-public-projects-page

9 years agoMerge branch '6090-docker-use-local-sso' closes #6138
Peter Amstutz [Wed, 27 May 2015 18:23:03 +0000 (14:23 -0400)]
Merge branch '6090-docker-use-local-sso' closes #6138

9 years agoAdd GPG key for RVM installation in the doc.
Ward Vandewege [Wed, 27 May 2015 16:42:44 +0000 (12:42 -0400)]
Add GPG key for RVM installation in the doc.

No issue #

9 years agoMerge branch '6098-full-text-index' refs #6098
Tom Clegg [Wed, 27 May 2015 13:06:10 +0000 (09:06 -0400)]
Merge branch '6098-full-text-index' refs #6098

9 years ago6098: Recreate full text indexes with leading spaces, to persuade Postgres to actuall...
Tom Clegg [Wed, 27 May 2015 13:05:40 +0000 (09:05 -0400)]
6098: Recreate full text indexes with leading spaces, to persuade Postgres to actually use them.

9 years agoMerge branch 'master' into 6057-public-projects-page
radhika [Tue, 26 May 2015 22:58:18 +0000 (18:58 -0400)]
Merge branch 'master' into 6057-public-projects-page

9 years ago6057: if /projects/public is accessed when anonymous config is not enabled, show...
radhika [Tue, 26 May 2015 22:54:13 +0000 (18:54 -0400)]
6057: if /projects/public is accessed when anonymous config is not enabled, show 404.

9 years agoMerge branch 'master' into 6093-refresh-docs
radhika [Tue, 26 May 2015 20:04:41 +0000 (16:04 -0400)]
Merge branch 'master' into 6093-refresh-docs

9 years ago6093: one more
radhika [Tue, 26 May 2015 19:57:11 +0000 (15:57 -0400)]
6093: one more

9 years ago6093: a few more updates
radhika [Tue, 26 May 2015 19:51:44 +0000 (15:51 -0400)]
6093: a few more updates

9 years ago6093: some more doc updates.
radhika [Tue, 26 May 2015 15:56:29 +0000 (11:56 -0400)]
6093: some more doc updates.

9 years agoMerge branch '6094-install-script-sigpipe' refs #6094
Tom Clegg [Tue, 26 May 2015 14:31:56 +0000 (10:31 -0400)]
Merge branch '6094-install-script-sigpipe' refs #6094

9 years ago6094: Propagate install script stderr+stdout to job log.
Tom Clegg [Tue, 26 May 2015 14:18:38 +0000 (10:18 -0400)]
6094: Propagate install script stderr+stdout to job log.

9 years agoMerge branch '6095-arv-copy-preserve-object-order-wip'
Brett Smith [Tue, 26 May 2015 00:32:19 +0000 (20:32 -0400)]
Merge branch '6095-arv-copy-preserve-object-order-wip'

Closes #6095, #6117.

9 years ago6095: arv-copy preserves order of copied JSON.
Brett Smith [Fri, 22 May 2015 21:10:37 +0000 (17:10 -0400)]
6095: arv-copy preserves order of copied JSON.

This means arv-copy no longer loses the order of pipeline template
components, which makes for a nicer presentation in Workbench.

Other Python clients that would like to preserve the order of JSON
responses can use OrderedJsonModel the same way.

9 years ago6093: doc updates
radhika [Mon, 25 May 2015 22:36:49 +0000 (18:36 -0400)]
6093: doc updates

9 years ago6087: Add big-manifest tests, with some finer-grained performance numbers on stderr.
Tom Clegg [Mon, 25 May 2015 17:25:22 +0000 (13:25 -0400)]
6087: Add big-manifest tests, with some finer-grained performance numbers on stderr.

9 years ago6087: Use HTTPClient's compression feature (instead of adding the
Tom Clegg [Mon, 25 May 2015 14:35:02 +0000 (10:35 -0400)]
6087: Use HTTPClient's compression feature (instead of adding the
Content-Encoding header ourselves). Rename config knob to describe
purpose instead of implementation.

9 years ago6087: Compute portable_data_hash only once during check_signatures.
Tom Clegg [Mon, 25 May 2015 14:23:53 +0000 (10:23 -0400)]
6087: Compute portable_data_hash only once during check_signatures.

9 years ago6087: Use app-configured key by default for blob signing and verification.
Tom Clegg [Mon, 25 May 2015 14:19:31 +0000 (10:19 -0400)]
6087: Use app-configured key by default for blob signing and verification.

9 years agoMerge branch '6094-install-script-sigpipe' (early part) refs #6094
Tom Clegg [Fri, 22 May 2015 21:42:17 +0000 (17:42 -0400)]
Merge branch '6094-install-script-sigpipe' (early part) refs #6094

9 years agoMerge branch 'master' into 6057-public-projects-page
radhika [Fri, 22 May 2015 20:40:59 +0000 (16:40 -0400)]
Merge branch 'master' into 6057-public-projects-page

9 years ago6057: add projects/public page, which lists publicly accessible projects.
radhika [Fri, 22 May 2015 20:39:57 +0000 (16:39 -0400)]
6057: add projects/public page, which lists publicly accessible projects.

9 years ago6141: Remove hard-coded "https://" from "https://{{site.arvados_workbench_host}}...
Peter Amstutz [Fri, 22 May 2015 20:13:22 +0000 (16:13 -0400)]
6141: Remove hard-coded "https://" from "https://{{site.arvados_workbench_host}}" and require that arvados_workbench_host include the url scheme instead.

9 years ago6094: Consider arvados_sdk_version (not just script_version) when
Tom Clegg [Fri, 22 May 2015 19:40:29 +0000 (15:40 -0400)]
6094: Consider arvados_sdk_version (not just script_version) when
deciding there's no need to extract or install anything.

9 years ago6094: Avoid SIGPIPE by consuming DATA section even when it's not needed.
Tom Clegg [Fri, 22 May 2015 19:39:00 +0000 (15:39 -0400)]
6094: Avoid SIGPIPE by consuming DATA section even when it's not needed.

9 years ago6138: Added --domain to set ARVADOS_DOMAIN. Removed useless comments in apache2_vhos...
Peter Amstutz [Fri, 22 May 2015 19:32:47 +0000 (15:32 -0400)]
6138: Added --domain to set ARVADOS_DOMAIN.  Removed useless comments in apache2_vhost that mentioned qr1hi.

9 years agoAdd installation instructions for compute nodes; update the installation
Ward Vandewege [Fri, 22 May 2015 14:03:01 +0000 (10:03 -0400)]
Add installation instructions for compute nodes; update the installation
instructions for crunch dispatcher.

No issue #

9 years agoMerge branch '6087-collection-timing' (early part) refs #6087 refs #6092
Tom Clegg [Thu, 21 May 2015 21:15:06 +0000 (17:15 -0400)]
Merge branch '6087-collection-timing' (early part) refs #6087 refs #6092

9 years ago6087: Get database time only once per manifest-signing/verifying event, rather than...
Tom Clegg [Thu, 21 May 2015 20:51:52 +0000 (16:51 -0400)]
6087: Get database time only once per manifest-signing/verifying event, rather than once per locator.

9 years agoUpdate tutorial pipeline page to match new definition.
Brett Smith [Thu, 21 May 2015 20:06:16 +0000 (16:06 -0400)]
Update tutorial pipeline page to match new definition.

No issue #.

9 years ago6090: Docker install uses local SSO server instead of auth.curoverse.com. Also
Peter Amstutz [Thu, 21 May 2015 19:00:10 +0000 (15:00 -0400)]
6090: Docker install uses local SSO server instead of auth.curoverse.com.  Also
clean up references to dev.arvados to use @@ARVADOS_DOMAIN@@.

9 years agoUpdate tutorial pipeline template definition.
Brett Smith [Thu, 21 May 2015 18:23:46 +0000 (14:23 -0400)]
Update tutorial pipeline template definition.

This helps it run out of the box again.
No issue #.

9 years agorefs #6061
radhika [Thu, 21 May 2015 16:09:07 +0000 (12:09 -0400)]
refs #6061
6061: set profiling_enabled to true in workbench application.default.yml for test env.

9 years agoDocumentation fix (quoting) for the 'Add an Arvados repository' page.
Ward Vandewege [Wed, 20 May 2015 15:29:29 +0000 (11:29 -0400)]
Documentation fix (quoting) for the 'Add an Arvados repository' page.

No issue #

9 years ago5748: Remove unused import. refs #5748
Tom Clegg [Wed, 20 May 2015 14:54:00 +0000 (10:54 -0400)]
5748: Remove unused import. refs #5748

9 years ago5748: Remove runtime.GC() obsoleted by bufferpool. refs #5748
Tom Clegg [Wed, 20 May 2015 14:15:30 +0000 (10:15 -0400)]
5748: Remove runtime.GC() obsoleted by bufferpool. refs #5748

9 years agocloses #6064
radhika [Wed, 20 May 2015 13:40:35 +0000 (09:40 -0400)]
closes #6064
Merge branch '6064-collection-output-of-perf-issue'

9 years agointegration_performance tests: add comment describing how to run the tests; also...
radhika [Wed, 20 May 2015 12:13:14 +0000 (08:13 -0400)]
integration_performance tests: add comment describing how to run the tests; also remove the ENV variable which is not needed
since the tests in this directory will not be part of regular test run because it is not a standard rails test directory.

9 years ago6064: rollback cache update so that it can be done in a separate branch.
radhika [Wed, 20 May 2015 03:00:21 +0000 (23:00 -0400)]
6064: rollback cache update so that it can be done in a separate branch.

9 years agoMerge branch 'master' into 6064-collection-output-of-perf-issue
radhika [Wed, 20 May 2015 02:32:25 +0000 (22:32 -0400)]
Merge branch 'master' into 6064-collection-output-of-perf-issue

9 years agoMerge branch '6071-java-sdk-timeouts' closes #6071
Peter Amstutz [Tue, 19 May 2015 20:31:39 +0000 (16:31 -0400)]
Merge branch '6071-java-sdk-timeouts' closes #6071

9 years ago6071: Bump up timeout to 120 seconds.
Peter Amstutz [Tue, 19 May 2015 20:29:36 +0000 (16:29 -0400)]
6071: Bump up timeout to 120 seconds.

9 years agorefs #6061
Radhika Chippada [Tue, 19 May 2015 19:26:07 +0000 (15:26 -0400)]
refs #6061
Merge branch '6061-intg-perf-tests'

9 years agoMerge branch '6073-pycurl-version' refs #6073
Peter Amstutz [Tue, 19 May 2015 19:31:35 +0000 (15:31 -0400)]
Merge branch '6073-pycurl-version' refs #6073

9 years ago6061: add workbench integration_performance test folder.
Radhika Chippada [Tue, 19 May 2015 19:23:34 +0000 (15:23 -0400)]
6061: add workbench integration_performance test folder.

9 years ago6071: Fix URL
Peter Amstutz [Tue, 19 May 2015 19:20:16 +0000 (15:20 -0400)]
6071: Fix URL

9 years ago6071: Refactor project organization a bit.
Peter Amstutz [Tue, 19 May 2015 17:55:57 +0000 (13:55 -0400)]
6071: Refactor project organization a bit.

9 years ago6071: Increase default timeout and enable retry support.
Peter Amstutz [Tue, 19 May 2015 17:52:51 +0000 (13:52 -0400)]
6071: Increase default timeout and enable retry support.

9 years ago6073: Pycurl depends on latest in Arvados SDK setup.py
Peter Amstutz [Tue, 19 May 2015 16:54:37 +0000 (12:54 -0400)]
6073: Pycurl depends on latest in Arvados SDK setup.py

9 years agocloses #4717
Radhika Chippada [Tue, 19 May 2015 03:24:46 +0000 (23:24 -0400)]
closes #4717
Merge branch '4717-read-only-keep-services-flag'

9 years ago4717: rename writableRoots as writableLocalRoots.
Radhika Chippada [Tue, 19 May 2015 03:11:59 +0000 (23:11 -0400)]
4717: rename writableRoots as writableLocalRoots.

9 years agoMerge branch 'master' into 4717-read-only-keep-services-flag
Radhika Chippada [Tue, 19 May 2015 02:22:18 +0000 (22:22 -0400)]
Merge branch 'master' into 4717-read-only-keep-services-flag

9 years agocloses #5720
Radhika Chippada [Tue, 19 May 2015 02:21:33 +0000 (22:21 -0400)]
closes #5720
Merge branch '5720-ajax-loading-error'

9 years ago6064: display only 5 objects for which this collection is the output in source_summary.
Radhika Chippada [Tue, 19 May 2015 02:09:21 +0000 (22:09 -0400)]
6064: display only 5 objects for which this collection is the output in source_summary.

9 years ago5720: Leave existing content in place (and schedule another reload) if a tab-refresh...
Tom Clegg [Mon, 18 May 2015 17:36:27 +0000 (13:36 -0400)]
5720: Leave existing content in place (and schedule another reload) if a tab-refresh gets cancelled.

9 years agoMerge branch '5998-arvdock-bridge-option'
Nico Cesar [Fri, 15 May 2015 21:04:06 +0000 (17:04 -0400)]
Merge branch '5998-arvdock-bridge-option'

closes #5998

9 years ago$start_bridge -> $bridge
Nico Cesar [Fri, 15 May 2015 19:00:21 +0000 (15:00 -0400)]
$start_bridge -> $bridge

options modified

ip command refactored

refs #5998

9 years agoCheck in docker/sso/database.yml.in, forgotten in commit ea7da293b0a10baef18cfc6e107c...
Ward Vandewege [Fri, 15 May 2015 13:46:07 +0000 (09:46 -0400)]
Check in docker/sso/database.yml.in, forgotten in commit ea7da293b0a10baef18cfc6e107c87a9c3b09546.

No issue #

9 years agoMerge branch 'master' into 4717-read-only-keep-services-flag
Radhika Chippada [Fri, 15 May 2015 02:56:12 +0000 (22:56 -0400)]
Merge branch 'master' into 4717-read-only-keep-services-flag

9 years ago4717: use keep_services -> read_only flag in python sdk.
Radhika Chippada [Fri, 15 May 2015 02:55:25 +0000 (22:55 -0400)]
4717: use keep_services -> read_only flag in python sdk.