Tom Clegg [Thu, 21 Jan 2016 19:35:34 +0000 (14:35 -0500)]
8281: Fix KeepClient retry bugs.
get() and put() were both handling all Curl exceptions -- including
timeouts -- by marking the keep service as unusable. For example, if a
single proxy is the only service available, a single timeout was
fatal. This is fixed by setting the retry loop status to None instead
of False after curl exceptions.
put() was repeating its retry loop until it achieved the desired
number of replicas _in a single iteration_. For example, when trying
to store 2 replicas, 6 loop iterations with a single success in each
iteration would result in 6 copies being stored but put() declaring
failure. This is fixed by checking against a cumulative "done" counter
instead of the "copies done in this loop iteration" counter.
Tom Clegg [Thu, 21 Jan 2016 09:01:16 +0000 (04:01 -0500)]
8281: Fix arv-mount ignoring --retries argument when writing file data.
"num_retries" arguments get passed around extensively in arvfile.py
and collection.py in the Python SDK, but ultimately the writing of
file data is done by a _BlockManager which doesn't have any way to
accept that argument or pass it along to a KeepClient, so PUT requests
always use the CollectionWriter's KeepClient's default num_retries.
In arv-mount's case, we have been telling CollectionWriter the
num_retries we want. When CollectionWriter creates a KeepClient,
num_retries gets passed along -- normally this works around the fact
that num_retries gets lost by the _BlockManager layer. However, we
provided our own KeepClient to use instead of letting CollectionWriter
create one, and we forgot to set num_retries on our own KeepClient, so
we weren't retrying PUT requests.
Peter Amstutz [Thu, 21 Jan 2016 20:46:53 +0000 (15:46 -0500)]
6833: Test setting small blobSignatureTtl. Fix earlier fix.
Peter Amstutz [Thu, 21 Jan 2016 16:30:23 +0000 (11:30 -0500)]
6833: Test to confirm that enabling polling on CollectionDirectory causes
tokens to be refreshed.
Peter Amstutz [Thu, 21 Jan 2016 16:29:40 +0000 (11:29 -0500)]
6833: Collection update file block list (to get most recent tokens) even when actual
content hasn't changed.
Tom Clegg [Fri, 15 Jan 2016 21:20:49 +0000 (16:20 -0500)]
6833: Add (most of) a test case for token expiry.
radhika [Thu, 21 Jan 2016 20:25:06 +0000 (15:25 -0500)]
8178: All three currently supported volumes return error when trash-lifetime period is not configured. azure blob and s3 volumes are updated to do so.
Returning an error is causing test failures in unix volume and hence is still a work in progress.
Peter Amstutz [Thu, 21 Jan 2016 19:47:40 +0000 (14:47 -0500)]
Merge branch '8008-test-all-packages' closes #8008
Peter Amstutz [Thu, 21 Jan 2016 19:30:03 +0000 (14:30 -0500)]
8008: Tweak error code reporting. Remove debugging line.
Peter Amstutz [Thu, 21 Jan 2016 19:07:11 +0000 (14:07 -0500)]
Merge branch '8008-package-testing' refs #8008
radhika [Thu, 21 Jan 2016 18:59:36 +0000 (13:59 -0500)]
8178: rename Delete api as Trash; add Untrash to volume interface; add UndeleteHandler and test for this endpoint.
Peter Amstutz [Thu, 21 Jan 2016 18:37:20 +0000 (13:37 -0500)]
7846: Better directory entry invalidation, fixes MagicDirApiError test. Also
fix bug in typo in exception handler.
Brett Smith [Wed, 20 Jan 2016 23:52:36 +0000 (18:52 -0500)]
8008: Quote more variables in package test scripts.
radhika [Wed, 20 Jan 2016 21:15:12 +0000 (16:15 -0500)]
refs #6833
Merge branch '6833-arv-mount-cache-refresh'
radhika [Mon, 11 Jan 2016 14:37:10 +0000 (09:37 -0500)]
6833: get blobSignatureTtl from discovery document and use it to set the poll_time.
radhika [Thu, 7 Jan 2016 18:14:05 +0000 (13:14 -0500)]
6833: add poll_time to CollectionDirectory.
Ward Vandewege [Wed, 20 Jan 2016 18:19:58 +0000 (13:19 -0500)]
Fix typo.
refs #8248
radhika [Wed, 20 Jan 2016 17:44:02 +0000 (12:44 -0500)]
refs #8028
Merge branch '8028-crunch-dispatch-local'
radhika [Wed, 20 Jan 2016 17:43:32 +0000 (12:43 -0500)]
Merge branch 'master' into 8028-crunch-dispatch-local
Conflicts:
jenkins/run-tests.sh
radhika [Wed, 20 Jan 2016 17:37:34 +0000 (12:37 -0500)]
closes #8028
Merge branch '8028-crunch-dispatch-local'
radhika [Wed, 20 Jan 2016 17:33:16 +0000 (12:33 -0500)]
8028: Update the stub based test to use waitGroup.Wait() instead of sleep.
radhika [Wed, 20 Jan 2016 17:05:59 +0000 (12:05 -0500)]
8028: In runningCmds loop, no need to wait for process termination.
radhika [Wed, 20 Jan 2016 16:40:30 +0000 (11:40 -0500)]
8028: add command to waitGroup during run method itself; not during signal handling.
radhika [Wed, 20 Jan 2016 12:16:05 +0000 (07:16 -0500)]
8028: when a signal is received, terminate all running commands and wait in a WaitGroup.
radhika [Wed, 20 Jan 2016 02:40:59 +0000 (21:40 -0500)]
8028: add signal handling to dispatcher.
radhika [Tue, 19 Jan 2016 17:16:51 +0000 (12:16 -0500)]
8028: After getting list of Queued containers, instead of looking for containers.ItemsAvailable, look for len(containers.Items)
radhika [Wed, 13 Jan 2016 16:15:28 +0000 (11:15 -0500)]
8028: crunch-dispatch-local implementation
Peter Amstutz [Wed, 20 Jan 2016 17:10:26 +0000 (12:10 -0500)]
Document some arvbox hw/sw requirements refs #8080
Peter Amstutz [Wed, 20 Jan 2016 17:10:26 +0000 (12:10 -0500)]
Document some arvbox hw/sw requirements refs #8080
Peter Amstutz [Wed, 20 Jan 2016 16:43:10 +0000 (11:43 -0500)]
Enable arvbox user to sudo to crunch user. refs #8080
Peter Amstutz [Wed, 20 Jan 2016 16:43:10 +0000 (11:43 -0500)]
Enable arvbox user to sudo to crunch user. refs #8080
Peter Amstutz [Wed, 20 Jan 2016 16:40:09 +0000 (11:40 -0500)]
Add crunch-run to run-tests refs #7816
Ward Vandewege [Wed, 20 Jan 2016 16:26:16 +0000 (11:26 -0500)]
Merge branch '8248-explain-serialize'
closes #8248
Ward Vandewege [Wed, 20 Jan 2016 16:25:20 +0000 (11:25 -0500)]
Doc improvement: explain in more detail what the -serialize argument to
keepstore does.
closes #8248
Peter Amstutz [Wed, 20 Jan 2016 16:04:51 +0000 (11:04 -0500)]
Merge branch '7816-crunch2-exec' closes #7816
Peter Amstutz [Wed, 20 Jan 2016 15:44:49 +0000 (10:44 -0500)]
6605: arv-copy preferentially uses HTTP git urls.
Peter Amstutz [Wed, 20 Jan 2016 15:38:43 +0000 (10:38 -0500)]
Fix gitolite-shell path refs #8080
Peter Amstutz [Wed, 20 Jan 2016 15:38:43 +0000 (10:38 -0500)]
Fix gitolite-shell path refs #8080
Tom Clegg [Wed, 20 Jan 2016 08:40:46 +0000 (03:40 -0500)]
8123: Ignore crunchstat "caught signal:" log.
Tom Clegg [Wed, 20 Jan 2016 02:12:38 +0000 (21:12 -0500)]
Merge branch '8177-keep-web-trust-all-content-flag' closes #8177
Tom Clegg [Tue, 19 Jan 2016 22:38:05 +0000 (17:38 -0500)]
8177: Offer -trust-all-content option for no-wildcard installs.
Add alert about XSS on intranet sites.
Peter Amstutz [Tue, 19 Jan 2016 22:07:59 +0000 (17:07 -0500)]
7816: CopyReaderToLog renamed to ReadWriteLines. Use Writer instead of Logger
to avoid unnecessary copy.
Tom Clegg [Mon, 18 Jan 2016 21:17:17 +0000 (16:17 -0500)]
8177: Expand trust_all_content comments.
Skip whole check_uri block when trust_all_content.
Fix test name.
radhika [Mon, 18 Jan 2016 05:04:48 +0000 (00:04 -0500)]
8177: add trust_all_content config to Workbench.
Peter Amstutz [Tue, 19 Jan 2016 21:43:54 +0000 (16:43 -0500)]
7816: Rename to crunch-run
Nico Cesar [Tue, 19 Jan 2016 20:17:22 +0000 (15:17 -0500)]
added git_internal_dir init in postinst.sh
Peter Amstutz [Tue, 19 Jan 2016 18:09:31 +0000 (13:09 -0500)]
Merge branch '8080-arvbox' closes #8080
Peter Amstutz [Tue, 19 Jan 2016 18:09:31 +0000 (13:09 -0500)]
Merge branch '8080-arvbox' closes #8080
Tom Clegg [Tue, 19 Jan 2016 04:19:35 +0000 (23:19 -0500)]
Merge branch '7996-keep-web-doc' closes #7996
Peter Amstutz [Mon, 18 Jan 2016 20:33:03 +0000 (15:33 -0500)]
8080: fpm_depends comes before run-build-package commands; fpm_args comes after
run-build package commands. Improve quoting. Use sed -i.
Peter Amstutz [Mon, 18 Jan 2016 19:43:01 +0000 (14:43 -0500)]
8008: Move test files into their own directories. Fix rails package tests to
work with debian & ubuntu. Fix error code checking.
Tom Clegg [Mon, 18 Jan 2016 02:45:46 +0000 (21:45 -0500)]
7996: Add keep-web to install TOC and server/cert lists.
Peter Amstutz [Fri, 15 Jan 2016 21:32:08 +0000 (16:32 -0500)]
7816: Fixes recommended by golint.
Peter Amstutz [Fri, 15 Jan 2016 20:50:33 +0000 (15:50 -0500)]
7816: Change ThrottledLogger Stop() to Close(). Choose finalState once so that
log and container record can't go out of sync. Add comments.
Peter Amstutz [Fri, 15 Jan 2016 15:10:48 +0000 (10:10 -0500)]
8008: Remove workbench git version constraint.
Peter Amstutz [Thu, 14 Jan 2016 21:40:59 +0000 (16:40 -0500)]
8008: SSO package tests pass for all targets.
Peter Amstutz [Thu, 14 Jan 2016 20:45:18 +0000 (15:45 -0500)]
8008: Incorporate SSO package testing into general package testing framework.
Tom Clegg [Thu, 14 Jan 2016 20:20:46 +0000 (15:20 -0500)]
Fix cli "update" example. No issue #
Tom Clegg [Thu, 14 Jan 2016 20:19:48 +0000 (15:19 -0500)]
Merge branch '8189-handle-large-collections-better' refs #8189
Tom Clegg [Thu, 14 Jan 2016 18:15:08 +0000 (13:15 -0500)]
8189: Use large page size only when fetching all pages.
Peter Amstutz [Thu, 14 Jan 2016 16:44:18 +0000 (11:44 -0500)]
8008: Add dependencies on libcurl and postgresql dev packages in order to build
gems.
Tom Clegg [Thu, 14 Jan 2016 16:27:08 +0000 (11:27 -0500)]
8189: Replace MAX_LIMIT const with max_items_per_response config.
Ward Vandewege [Thu, 14 Jan 2016 16:01:04 +0000 (11:01 -0500)]
8189: Fix too-small page size in "fetch all pages" code.
Python SDK and workbench effectively default to the API server's
MAX_LIMIT when requesting a list of objects, in those cases where no
explicit limit is set in the client code.
Peter Amstutz [Wed, 13 Jan 2016 21:41:39 +0000 (16:41 -0500)]
Fixing our fpm rails package building to handle fpm-info.sh.
Peter Amstutz [Wed, 13 Jan 2016 21:21:27 +0000 (16:21 -0500)]
8008: Add git dependency to api server and workbench.
Peter Amstutz [Wed, 13 Jan 2016 20:40:36 +0000 (15:40 -0500)]
8008: Run bundle install and bundle list for api server and workbench.
Peter Amstutz [Wed, 13 Jan 2016 19:50:45 +0000 (14:50 -0500)]
8080: Successfully tests all packages. Produces before/after/diff lists for each package.
Peter Amstutz [Wed, 13 Jan 2016 19:11:30 +0000 (14:11 -0500)]
8008: Test each Arvados package individually.
radhika [Wed, 13 Jan 2016 16:17:11 +0000 (11:17 -0500)]
8028: add services/crunch-dispatch-local
Tom Clegg [Wed, 13 Jan 2016 15:18:17 +0000 (10:18 -0500)]
8123: Fix crunchstat formatting error and resulting crunchstat-summary crash.
Tom Clegg [Thu, 24 Dec 2015 18:51:35 +0000 (13:51 -0500)]
8123: Add option to include stats from child jobs.
Tom Clegg [Wed, 23 Dec 2015 19:09:37 +0000 (14:09 -0500)]
8123: Generate multiple sets of charts when data source is a pipeline instance.
Tom Clegg [Wed, 23 Dec 2015 16:39:14 +0000 (11:39 -0500)]
8123: Add option (--format html) to generate canvasjs charts.
Tom Clegg [Tue, 22 Dec 2015 21:16:15 +0000 (16:16 -0500)]
8123: Aim 5% below GiB memory size boundaries.
Tom Clegg [Mon, 21 Dec 2015 21:53:37 +0000 (16:53 -0500)]
8123: Recommend more economical values for resource_constraints.
Tom Clegg [Mon, 21 Dec 2015 19:42:44 +0000 (14:42 -0500)]
8123: Add --pipeline-instance mode: generate a report for each finished component.
Brett Smith [Tue, 12 Jan 2016 01:23:56 +0000 (20:23 -0500)]
Merge branch '8014-rails-postinst-doc-update-wip'
Refs #8014. Closes #8111.
Brett Smith [Tue, 5 Jan 2016 22:33:11 +0000 (17:33 -0500)]
8014: Install guide uses new Rails postinst scripts for configuration.
Brett Smith [Mon, 11 Jan 2016 22:17:45 +0000 (17:17 -0500)]
Merge branch '8014-rails-postinst-scripts-wip'
Refs #8014. Closes #8036.
Brett Smith [Mon, 11 Jan 2016 22:17:09 +0000 (17:17 -0500)]
8014: Rails upgrade scripts are removed; don't run them in deploy script.
Brett Smith [Mon, 4 Jan 2016 23:02:08 +0000 (18:02 -0500)]
8014: Rails postinst script sets better default access for /etc/arvados.
Brett Smith [Sat, 2 Jan 2016 17:15:53 +0000 (12:15 -0500)]
8014, 8059: Unify Rails package building.
* Introduce a handle_rails_package function, and helpers, to build a
Rails package.
* Generalize all the support scripts into unified postinst, prerm, and
postrm scripts. handle_rails_package builds these into the
packages. See jenkins/rails-package-scripts/README.md for details.
* More error checking throughout (e.g., check for failure when doing
any prep work for Rails packages, or the Workbench packages
specifically, using `set -e` in a subshell).
One behavior change from this unification: before this,
run-build-packages treated --build-bundle-packages to mean "*also*
build bundle packages", while run-build-packages-sso treated the
switch to mean "*only* build bundle packages". This commit declares
run-build-packages to be the winner, and makes run-build-packages-sso
consistent with it.
Brett Smith [Sat, 2 Jan 2016 00:09:08 +0000 (19:09 -0500)]
Fix quoting of local variable assignments throughout.
When you assign a variable directly (e.g., `FOO=bar`), you don't need
to quote the RHS, because the shell doesn't do expansion in that
case. But when you declare and assign a variable (e.g., `local
foo=bar`), you *do* potentially need to quote it, because `local` is a
command and all the normal expansion rules for running commands
applies.
Brett Smith [Fri, 1 Jan 2016 18:38:00 +0000 (13:38 -0500)]
8014: Clean database state detection in arvados-sso postinst.
* Remove unused status code capture.
* Use `grep -q`. Previous versions of the script avoided it because
they piped directly from rake tasks, which would get upset when the
pipe broke. Now that we capture all the output, that's no longer a
concern.
Brett Smith [Fri, 1 Jan 2016 17:26:20 +0000 (12:26 -0500)]
8014: Introduce run_and_report function to arvados-sso postinst.
Use this to DRY up code that says "Doing something... done."
This commit removes some `|| exit $?` that's redundant with `set -e`.
Brett Smith [Fri, 1 Jan 2016 17:17:01 +0000 (12:17 -0500)]
8014: Improve conffile handling in arvados-sso postinst.
* DRY it up into a function.
* DATABASE_READY and APPLICATION_READY care about whether the
corresponding .yml file is usable. Always detect when it is
unmodified from the .yml.example file.
* Build symlinks from /var/www to /etc in more cases where it's safe.
Brett Smith [Fri, 1 Jan 2016 16:10:05 +0000 (11:10 -0500)]
8014: arvados-sso postinst recognizes versioned Nginx service names.
Brett Smith [Fri, 1 Jan 2016 16:02:46 +0000 (11:02 -0500)]
8014: Refactor web service detection warnings in arvados-sso postinst.
Brett Smith [Fri, 1 Jan 2016 15:54:00 +0000 (10:54 -0500)]
8014: Simplify COMMAND_PREFIX-setting code.
Brett Smith [Fri, 1 Jan 2016 15:51:01 +0000 (10:51 -0500)]
8014: Refactor "not fully configured" messages in arvados-sso postinst.
Brett Smith [Fri, 1 Jan 2016 15:41:23 +0000 (10:41 -0500)]
8014: Remove unused $VERSION tracking from arvados-sso postinst.
Brett Smith [Thu, 31 Dec 2015 20:58:18 +0000 (15:58 -0500)]
8014: Remove unused sso-server-upgrade.sh script.
Brett Smith [Sun, 10 Jan 2016 03:34:18 +0000 (22:34 -0500)]
7991: pip build process detects if --no-use-wheel is supported.
Refs #7991.
Brett Smith [Sun, 10 Jan 2016 03:33:34 +0000 (22:33 -0500)]
7991: Update error message for pip build errors.
Refs #7991.
Brett Smith [Fri, 8 Jan 2016 22:20:05 +0000 (17:20 -0500)]
Merge branch '7991-google-api-python-client-perms-wip'
Closes #7991, #8136.
Brett Smith [Fri, 8 Jan 2016 20:58:28 +0000 (15:58 -0500)]
7991: Fix permissions of google-api-python-client egg info files.
Ward Vandewege [Thu, 7 Jan 2016 00:38:28 +0000 (19:38 -0500)]
Make sure not to silence package installation during the test-packages
stage, so that we can see actual failure messages if there is a problem.
No issue #
Peter Amstutz [Wed, 6 Jan 2016 21:38:58 +0000 (16:38 -0500)]
7816: Converted testing to use stubbed out ThinDockerClient instead of real Docker.
Peter Amstutz [Wed, 6 Jan 2016 15:52:07 +0000 (10:52 -0500)]
7816: Crunch2 executor prototype work in progress.
Peter Amstutz [Wed, 6 Jan 2016 15:51:28 +0000 (10:51 -0500)]
7816: Update Go SDK to use correct block locator pattern. Change FileToken
string in manifest to FileStreamSegment struct.