Brett Smith [Thu, 22 May 2014 20:11:10 +0000 (16:11 -0400)]
2752: Move arv-put functionality to arvados.commands.put.
This will make it easier to unit test arv-put, and make it easier for
developers to extend or customize its functionality.
Brett Smith [Thu, 22 May 2014 19:49:31 +0000 (15:49 -0400)]
2752: Move Python tests to dedicated subdirectory.
This better follows Python convention, and will let us build out our
test frameworks without cluttering the root source directories.
This change brings a small change to the way we run tests. Pass the
directory name to the discover command:
$ python -m unittest discover tests
Brett Smith [Fri, 30 May 2014 14:40:07 +0000 (10:40 -0400)]
2752: Add arvados.collections.ResumableCollectionWriter.
This is a subclass of CollectionWriter that only accepts data from the
filesystem. In exchange, it can record its own state, and resume
writing from one of those states. arv-put will use this to make the
user experience nicer if a long upload is interrupted.
Brett Smith [Tue, 20 May 2014 16:14:07 +0000 (12:14 -0400)]
2752: Implement CollectionWriter with a work queue.
This will make it easier to capture and restore state.
Brett Smith [Tue, 20 May 2014 15:29:44 +0000 (11:29 -0400)]
2752: Reuse CollectionWriter.write_file() where possible.
Brett Smith [Tue, 20 May 2014 15:15:00 +0000 (11:15 -0400)]
2752: Add method CollectionWriter.write_file().
Brett Smith [Tue, 20 May 2014 15:12:56 +0000 (11:12 -0400)]
2752: Remove trailing whitespace in arv-put.
Brett Smith [Tue, 20 May 2014 14:25:18 +0000 (10:25 -0400)]
2752: Add tests for CollectionWriter.write_directory_tree().
Brett Smith [Tue, 20 May 2014 13:47:27 +0000 (09:47 -0400)]
2752: Clean up after Python SDK Collections tests.
Brett Smith [Tue, 20 May 2014 13:40:53 +0000 (09:40 -0400)]
2752: Refactor Python SDK test_collections.
This eliminates duplication in setup and teardown code, and clarifies
dependencies between tests a bit.
Brett Smith [Mon, 19 May 2014 18:10:41 +0000 (14:10 -0400)]
2752: Clean up arv-put imports.
Tim Pierce [Fri, 30 May 2014 13:49:26 +0000 (09:49 -0400)]
2755: comment doc fix.
Tim Pierce [Thu, 29 May 2014 21:08:22 +0000 (17:08 -0400)]
2755: code review.
* arv-put returns an unsigned manifest UUID.
* The ArvPutSignedManifest test first confirms that the collection is
not present in the API server. Also asserts that the arv-put command
completed successfully.
* Additional tests for KeepPermissionTestCase for each potential
combination of wrong authorization and signature.
Tom Clegg [Thu, 29 May 2014 20:48:30 +0000 (16:48 -0400)]
2762: Do not try to auto-destroy referring links when destroying a model with no uuid. refs #2762
radhika [Thu, 29 May 2014 20:34:53 +0000 (16:34 -0400)]
Merge branch 'master' into 2903-remove-pi-active-and-success
Ward Vandewege [Thu, 29 May 2014 18:17:23 +0000 (14:17 -0400)]
Switch to using auth helpers instead of messing with Thread.current in the user tests.
Tom Clegg [Thu, 29 May 2014 17:58:02 +0000 (13:58 -0400)]
2754: Change "create pipeline" label to "run", to match others in same workflow.
Peter Amstutz [Thu, 29 May 2014 17:41:23 +0000 (13:41 -0400)]
Another fix to sharing scope to use keep_services/accessible. refs #2766
Ward Vandewege [Thu, 29 May 2014 17:32:59 +0000 (13:32 -0400)]
Provide link to users page in notification e-mails.
Peter Amstutz [Thu, 29 May 2014 17:05:43 +0000 (13:05 -0400)]
Added proper sharing scope to permit actually downloading shared files. refs #2766
Tim Pierce [Thu, 29 May 2014 16:56:01 +0000 (12:56 -0400)]
2755: code review.
* CollectionWriter does not cache manifest_text, and recomputes it each
time it is asked.
* Manifest uuid is computed from a manifest that has been stripped of
permission hints, but not any other kind of locator hint.
Ward Vandewege [Thu, 29 May 2014 15:16:21 +0000 (11:16 -0400)]
Change default for
new_user_notification_recipients
new_inactive_user_notification_recipients
in config/application.default.yml to an empty array, so that it is
obvious how to specify multiple recipients.
Brett Smith [Thu, 29 May 2014 14:52:04 +0000 (10:52 -0400)]
2753: Fix Collection inline file view.
The height limit needs to be applied to the images themselves, rather
than the containing div.
Peter Amstutz [Thu, 29 May 2014 14:18:53 +0000 (10:18 -0400)]
Added rescue AccessForbiddenException to collections controller and view to
disable the "sharing link" feature when the client is untrusted. refs #2766
radhika [Thu, 29 May 2014 14:08:02 +0000 (10:08 -0400)]
2903: during validation methods, set state instead of success
radhika [Thu, 29 May 2014 13:37:02 +0000 (09:37 -0400)]
Merge branch 'master' into 2903-remove-pi-active-and-success
Peter Amstutz [Thu, 29 May 2014 13:32:08 +0000 (09:32 -0400)]
Merge branch '2044-share-button' refs #2766
Peter Amstutz [Thu, 29 May 2014 13:22:43 +0000 (09:22 -0400)]
Go Keep client correctly closes response body on client requests, should fix
lingering connections problem. Also added graceful shutdown on SIGINT to
keepproxy. refs #1885
Ward Vandewege [Thu, 29 May 2014 00:29:12 +0000 (20:29 -0400)]
Merge branch 'master' into 2681-new-inactive-user-notification
radhika [Wed, 28 May 2014 23:12:11 +0000 (19:12 -0400)]
2903: active to state usage
Tim Pierce [Wed, 28 May 2014 21:58:09 +0000 (17:58 -0400)]
2755: add support to arv-put for signed manifests.
When arv-put finishes a stream, the manifest it stores in Keep now has
been stripped of signatures and other variable hints.
test_cmdline.py tests arv-put's handling of the manifest to make sure
that, when permissions are enabled, the manifest in Keep lacks
signatures, and the same manifest returned from the API server includes
signatures.
Refs #2755.
radhika [Wed, 28 May 2014 21:30:21 +0000 (17:30 -0400)]
2903: Remove active and success flags from pipeline_instance. Update all other files as necessary.
radhika [Wed, 28 May 2014 16:58:11 +0000 (12:58 -0400)]
Merge branch 'master' of git.curoverse.com:arvados
radhika [Wed, 28 May 2014 16:56:36 +0000 (12:56 -0400)]
An equivalent of this script is now at services/api/script. This was done as part of Feature #2272.
Tom Clegg [Wed, 28 May 2014 16:44:04 +0000 (12:44 -0400)]
2754: Merge branch '2754-pipeline-template-description' refs #2754
Tom Clegg [Wed, 28 May 2014 16:43:20 +0000 (12:43 -0400)]
2893: Add more tests for bogus serialized hash values.
Tom Clegg [Wed, 28 May 2014 16:34:59 +0000 (12:34 -0400)]
2893: Add test for un-symbolizing existing db content.
Tom Clegg [Wed, 28 May 2014 16:22:18 +0000 (12:22 -0400)]
2893: Prevent symbol keys/values in serialized attributes.
Tom Clegg [Wed, 28 May 2014 16:00:33 +0000 (12:00 -0400)]
2754: Add migration missing from commit:
fe84c84827af6092784408629e4cde0663fe6af8
Peter Amstutz [Wed, 28 May 2014 15:19:37 +0000 (11:19 -0400)]
2044: Fixed CollectionsController#search_scopes to use 'filter' correctly
instead of using 'where' incorrectly. Reverted most changes to
ArvadosModel#ensure_owner_uuid_is_permitted. Added rescue to
collections/show.html.erb when log.event_at is nil.
Ward Vandewege [Wed, 28 May 2014 15:00:07 +0000 (11:00 -0400)]
Do not use symbol keys in the properties of permission links.
Peter Amstutz [Wed, 28 May 2014 14:53:23 +0000 (10:53 -0400)]
Merge branch 'master' into 2044-share-button
Tom Clegg [Wed, 28 May 2014 14:04:55 +0000 (10:04 -0400)]
2892: Fix user agreement routing. closes #2892
Tom Clegg [Wed, 28 May 2014 00:01:22 +0000 (20:01 -0400)]
2862: Use ?format=js so browser does not think js and html views are
interchangeable. closes #2862
Tom Clegg [Tue, 27 May 2014 22:16:39 +0000 (18:16 -0400)]
2760: Merge branch '2760-not-all-objects-in-folders' refs #2760
Peter Amstutz [Tue, 27 May 2014 21:07:40 +0000 (17:07 -0400)]
Merge branch 'master' of git.curoverse.com:arvados
Peter Amstutz [Tue, 27 May 2014 21:07:11 +0000 (17:07 -0400)]
Adjusted timeout on websocket_test from 3 seconds to 4 in an attempt to reduce
the number of spurious timeouts on jenkins.
radhika [Tue, 27 May 2014 20:50:26 +0000 (16:50 -0400)]
Merge branch 'master' of git.curoverse.com:arvados
Peter Amstutz [Tue, 27 May 2014 20:50:09 +0000 (16:50 -0400)]
Merge branch 'master' of git.curoverse.com:arvados #2751
radhika [Tue, 27 May 2014 20:20:57 +0000 (16:20 -0400)]
Merge branch '2756-eventbus-in-workbench' into 2678-owned_by
Conflicts:
services/api/script/crunch-dispatch.rb
radhika [Tue, 27 May 2014 19:54:36 +0000 (15:54 -0400)]
2756: fix error where partial lines were potentially logged
Peter Amstutz [Tue, 27 May 2014 19:50:04 +0000 (15:50 -0400)]
Merge branch 'master' of git.curoverse.com:arvados refs #2751
Peter Amstutz [Tue, 27 May 2014 19:49:37 +0000 (15:49 -0400)]
Merge branch '2751-python-sdk-keep-proxy-support' closes #2751
radhika [Tue, 27 May 2014 19:16:05 +0000 (15:16 -0400)]
Merge branch 'master' into 2756-eventbus-in-workbench
radhika [Tue, 27 May 2014 19:15:26 +0000 (15:15 -0400)]
2756: log span title
radhika [Tue, 27 May 2014 19:11:07 +0000 (15:11 -0400)]
2756: fix typo in helper method signature
radhika [Tue, 27 May 2014 18:59:55 +0000 (14:59 -0400)]
2756: update the ui to show job progress, and links in one column
radhika [Tue, 27 May 2014 17:19:53 +0000 (13:19 -0400)]
2525: add some javadoc comments
radhika [Tue, 27 May 2014 17:02:47 +0000 (13:02 -0400)]
2678: upon Tom's feedback, setting owner_uuid of the pi whether or not the pi belongs to a group.
Tom Clegg [Tue, 27 May 2014 15:43:07 +0000 (11:43 -0400)]
2754: Add pipeline template description attribute.
radhika [Tue, 27 May 2014 15:30:44 +0000 (11:30 -0400)]
Merge branch 'master' into 2525-java-sdk
radhika [Tue, 27 May 2014 15:22:00 +0000 (11:22 -0400)]
Merge branch 'master' into 2678-owned_by
radhika [Tue, 27 May 2014 15:16:22 +0000 (11:16 -0400)]
Merge branch 'master' into 2756-eventbus-in-workbench
radhika [Tue, 27 May 2014 15:15:03 +0000 (11:15 -0400)]
Merge branch 'master' into 2525-java-sdk
radhika [Tue, 27 May 2014 15:12:25 +0000 (11:12 -0400)]
Merge branch 'master' into 2767-doc-updates
Tom Clegg [Tue, 27 May 2014 15:08:25 +0000 (11:08 -0400)]
2760: Exclude system objects like VMs and repos from folder behavior.
Tim Pierce [Tue, 27 May 2014 15:06:42 +0000 (11:06 -0400)]
Merge branch '2755-python-sdk-permissions'
Refs #2755.
radhika [Tue, 27 May 2014 14:47:17 +0000 (10:47 -0400)]
Merge branch 'master' into 2525-java-sdk
Tom Clegg [Tue, 27 May 2014 14:44:56 +0000 (10:44 -0400)]
2760: Merge branch '2760-folder-hierarchy' refs #2760
Peter Amstutz [Tue, 27 May 2014 14:43:04 +0000 (10:43 -0400)]
2751: Test tweak to clear settings after changing environment variables.
Tim Pierce [Tue, 27 May 2014 14:38:29 +0000 (10:38 -0400)]
2865: collect garbage after each GET and PUT.
Quick fix for Keep OOM errors: reclaim memory aggressively.
Fixes #2865.
radhika [Tue, 27 May 2014 14:38:32 +0000 (10:38 -0400)]
Merge branch '2756-eventbus-in-workbench' into 2678-owned_by
radhika [Tue, 27 May 2014 14:37:57 +0000 (10:37 -0400)]
Merge branch 'master' into 2756-eventbus-in-workbench
radhika [Tue, 27 May 2014 14:37:12 +0000 (10:37 -0400)]
Merge branch 'master' into 2678-owned_by
Peter Amstutz [Tue, 27 May 2014 14:36:56 +0000 (10:36 -0400)]
2751: Changed to "except:" to "except Exception:". Refactored environment setup in keep_client_Test to be a bit more robust.
radhika [Tue, 27 May 2014 14:35:03 +0000 (10:35 -0400)]
2678: update a-r-p-i to set the owner_uuid on the newly created job when the pi belongs to a group. also, update crunch-dspatch to create the log lines with the group's owner_uuid.
Peter Amstutz [Tue, 27 May 2014 14:23:12 +0000 (10:23 -0400)]
Merge branch 'master' of git.curoverse.com:arvados refs #1885
Peter Amstutz [Tue, 27 May 2014 14:22:38 +0000 (10:22 -0400)]
1885: Tests and bugfixes for -no-get and -no-put options.
Tom Clegg [Tue, 27 May 2014 14:17:33 +0000 (10:17 -0400)]
2760: Merge branch 'master' into 2760-folder-hierarchy
radhika [Mon, 26 May 2014 17:33:45 +0000 (13:33 -0400)]
Merge branch 'master' into 2756-eventbus-in-workbench
Brett Smith [Mon, 26 May 2014 15:36:11 +0000 (11:36 -0400)]
2863: Remove debugging print from Python KeepClient.
Other tools like crunch-job parse the output of tools like arv-get and
arv-put. Right now crunch-job is saving this debug output in the log
attribute of running jobs, which confuses several other tools like
Workbench.
Closes #2863.
radhika [Sun, 25 May 2014 04:08:15 +0000 (00:08 -0400)]
2756: added a link to job when a job is chosen
radhika [Sat, 24 May 2014 22:23:44 +0000 (18:23 -0400)]
2756: log write
radhika [Sat, 24 May 2014 21:08:06 +0000 (17:08 -0400)]
2756: fix issue from merge conflict
radhika [Sat, 24 May 2014 19:49:06 +0000 (15:49 -0400)]
Merge branch 'master' into 2756-eventbus-in-workbench
Conflicts:
apps/workbench/app/assets/javascripts/pipeline_instances.js
Tim Pierce [Sat, 24 May 2014 18:34:45 +0000 (14:34 -0400)]
2829: add X-Block-Size response header
Responses to successful GET or HEAD requests include an X-Block-Size
header.
radhika [Sat, 24 May 2014 17:38:16 +0000 (13:38 -0400)]
Merge branch 'master' into 2767-doc-updates
Tim Pierce [Sat, 24 May 2014 04:44:06 +0000 (00:44 -0400)]
2755: incorporate code review.
* Unit tests cover all permutations of signature/authorization when
--enforce-permissions=false
* Keep is more forgiving about the structure of locators, permits
locator hints of unknown type (as long as they begin with an uppercase
letter)
* Keep delivers 400 Bad Request for requests that do not match any
route, or are lexically invalid. 404 Not Found only for requests with
a syntactically valid hash not found on disk.
Refs #2755.
Tom Clegg [Fri, 23 May 2014 22:44:35 +0000 (18:44 -0400)]
2760: Merge branch 'master' into 2760-folder-hierarchy
Tom Clegg [Fri, 23 May 2014 21:32:21 +0000 (17:32 -0400)]
2760: Disambiguate "hover" from "selected" style. Do not highlight
non-selectable rows on hover.
Tim Pierce [Wed, 21 May 2014 14:58:18 +0000 (10:58 -0400)]
2755: add support for signed locators in the Python SDK.
* arvados.Keep.put() saves the response body (which may contain a
signed locator) and returns it to the caller.
* arvados.Keep.get() passes the full signed locator to the remote Keep
server. The bare MD5 hash is still used for caching and for
shuffled_service_roots
* run_test_server.run_keep() takes arguments 'blob_signing_key' and
'enforce_permissions', for testing permission signatures in unit
tests.
* test_keep_client: new unit tests for permissions:
- with --enforce-permissions=true:
- GET with a signed locator works
- GET with an unsigned locator fails
- unauthenticated GET fails
- with --enforce-permissions=false:
- GET with a signed locator works
- GET with an unsigned locator works
- unauthenticated GET works
Bug fixes to permission handling in the Keep server:
* Locator hints may appear in any order; be flexible. Parse them in
GetBlockHandler rather than in the REST router.
* Returned locators are terminated with newline (consistent with
Warehouse, and more friendly for human debugging).
* The locator returned from a PUT request always has a size hint.
* The correct Authorization header keyword is "OAuth2", not
"OAuth". D'oh.
* Updated unit tests to accommodate newlines, size hints and OAuth2.
Refs #2755.
Tom Clegg [Fri, 23 May 2014 21:12:33 +0000 (17:12 -0400)]
2760: Merge branch 'master' into 2760-folder-hierarchy
Peter Amstutz [Fri, 23 May 2014 21:04:10 +0000 (17:04 -0400)]
Merge branch 'master' into 2044-share-button refs #2044
Tom Clegg [Fri, 23 May 2014 21:01:08 +0000 (17:01 -0400)]
2760: Use "My Folders" to indicate the root of my folder hierarchy.
Peter Amstutz [Fri, 23 May 2014 20:47:45 +0000 (16:47 -0400)]
Merge branch 'master' into 2751-python-sdk-keep-proxy-support refs #2751
Peter Amstutz [Fri, 23 May 2014 19:37:57 +0000 (15:37 -0400)]
Merge branch '1885-keep-proxy' closes #1885
Peter Amstutz [Fri, 23 May 2014 18:08:53 +0000 (14:08 -0400)]
1885: Added logging of invalid requests. Added logging when the server list is
update. Improved KeepClient error reporting.
Peter Amstutz [Fri, 23 May 2014 14:27:59 +0000 (10:27 -0400)]
Merge branch 'master' into 1885-keep-proxy refs #1885
Peter Amstutz [Fri, 23 May 2014 14:26:31 +0000 (10:26 -0400)]
1885: Fix bug in not returning from the handler on failed authorization.
Improved logging (which uncovered the previous bug.) Tweaked go.sh and tests.
Peter Amstutz [Thu, 22 May 2014 18:51:56 +0000 (14:51 -0400)]
1885: Made ServiceRoots atomically updatable, so that KeepProxy can poll for
changes in the services list without disrupting any active flows.
projects / arvados.git / log