Lucas Di Pentima [Wed, 21 Jun 2017 15:12:18 +0000 (12:12 -0300)]
11789: Unified the exclude logic by removing expected_bytes_for() and moving
the upload list code to a new method.
Updated tests to be in sync with this refactoring.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Lucas Di Pentima [Tue, 20 Jun 2017 20:41:36 +0000 (17:41 -0300)]
11789: Merge branch 'master' into 11789-arvput-exclude-flag
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Lucas Di Pentima [Tue, 20 Jun 2017 20:24:13 +0000 (17:24 -0300)]
11789: Splitted exclude_* arguments on ArvPutUploadJob class. Simplified
the way exclude_paths list is handled.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Tom Clegg [Tue, 20 Jun 2017 20:05:01 +0000 (16:05 -0400)]
Merge branch '10557-setup-cleanup'
refs #10557
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Peter Amstutz [Tue, 20 Jun 2017 16:39:50 +0000 (12:39 -0400)]
Merge branch '8186-nodemanager-ebs' closes #8186
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Tue, 20 Jun 2017 16:38:10 +0000 (12:38 -0400)]
8186: Add comment/warning about EBS volume size limit.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Tue, 20 Jun 2017 14:25:09 +0000 (10:25 -0400)]
8186: Round up ec2 ebs volume sizes. Ensure size is in valid range. Add test for creating ec2 instance with ebs storage.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Mon, 12 Jun 2017 19:43:19 +0000 (19:43 +0000)]
8186: Set virtual device path. Require libcloud 0.20 because 0.18 has mistakes in the ec2 instance size table.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Wed, 14 Jun 2017 13:28:27 +0000 (09:28 -0400)]
8186: Add EBS storage if necessary so there is sufficient scratch space.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Tom Clegg [Tue, 20 Jun 2017 14:45:19 +0000 (10:45 -0400)]
8784: Update arvbox to go1.8.
refs #8784
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Mon, 19 Jun 2017 21:31:19 +0000 (17:31 -0400)]
10557: Tidy up users#setup controller.
Simplify long conditional, and fix bug where admin asks for repo
"username/reponame" but "username/username/reponame" gets created.
This also fixes an unpredictable API: Previously, if params included
{user:{uuid:X,email:Y}}, the setup API would either create a new user
with uuid X and email Y, or set up an existing user (ignoring Y),
depending on whether X was the UUID of an existing user. Now, passing
a "user" hash like this always tries to create a new user with
uuid=X (if given) and email=Y, and returns an error if the given UUID
is already in use.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Tue, 20 Jun 2017 13:45:23 +0000 (09:45 -0400)]
Merge branch '8784-dir-listings'
refs #8784
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Mon, 19 Jun 2017 22:37:13 +0000 (18:37 -0400)]
8784: Replace remaining uses of keepclient.Reader with arvados.File.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Lucas Di Pentima [Mon, 19 Jun 2017 21:06:51 +0000 (18:06 -0300)]
Merge branch '11788-arvput-dir-references'
Closes #11788
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Tom Clegg [Mon, 19 Jun 2017 20:47:56 +0000 (16:47 -0400)]
8784: Merge branch 'master' into 8784-dir-listings
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Lucas Di Pentima [Mon, 19 Jun 2017 19:08:02 +0000 (16:08 -0300)]
11788: Added conditional behaviour when using, or not, a trailing slash on directory references.
Added test to cover all cases & updated help message.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Lucas Di Pentima [Mon, 19 Jun 2017 17:12:41 +0000 (14:12 -0300)]
11778: Added explicit test that checks directory references upload behavior.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Lucas Di Pentima [Mon, 19 Jun 2017 16:49:21 +0000 (13:49 -0300)]
11788: Simple fix to upload directory references as subcollections instead of
uploading their contents directly on the collection's root.
Updated tests.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Peter Amstutz [Mon, 19 Jun 2017 14:29:23 +0000 (10:29 -0400)]
test_with_arvbox determines the correct arvados/jobs image to pull by querying
verison server. refs #11458
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Lucas Di Pentima [Fri, 16 Jun 2017 18:02:44 +0000 (15:02 -0300)]
Merge branch '11557-acr-output-col-perms'
Closes #11557
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Lucas Di Pentima [Fri, 16 Jun 2017 17:12:16 +0000 (14:12 -0300)]
11557: Fixed the way that can_read permission link is created. Updated test.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Lucas Di Pentima [Fri, 16 Jun 2017 16:34:13 +0000 (13:34 -0300)]
11557: Merge branch 'master' into 11557-acr-output-col-perms
Tom Clegg [Fri, 16 Jun 2017 13:43:49 +0000 (09:43 -0400)]
Merge branch '11283-available-slot-query'
closes #11283
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Fri, 16 Jun 2017 13:34:08 +0000 (09:34 -0400)]
11283: Add explanatory comment to SQL query.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Lucas Di Pentima [Thu, 15 Jun 2017 22:29:42 +0000 (19:29 -0300)]
11789: Added '--excluded <pattern>' argument so that files and/or directories can be excluded
from the upload.
Several patterns can be used by passing more than one instance of the --exclude argument.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Tom Clegg [Thu, 15 Jun 2017 21:24:07 +0000 (17:24 -0400)]
8784: Update to Go 1.8.3.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Thu, 15 Jun 2017 21:18:59 +0000 (17:18 -0400)]
8784: Remove useless condition.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Thu, 15 Jun 2017 21:15:34 +0000 (17:15 -0400)]
8784: Show "no files" note if the collection is empty.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Thu, 15 Jun 2017 21:02:20 +0000 (17:02 -0400)]
8784: Improve comments. Fix fake error on range requests.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Thu, 15 Jun 2017 21:01:28 +0000 (17:01 -0400)]
8784: Just pass collection name to serveDirectory, not whole record.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Thu, 15 Jun 2017 20:19:54 +0000 (16:19 -0400)]
Merge branch '11729-cache-miss'
refs #11729
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 21:46:01 +0000 (17:46 -0400)]
11729: Avoid refetching manifest in forceReload case
...if uuid->pdh cache turns out not to be expired.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Lucas Di Pentima [Thu, 15 Jun 2017 16:17:50 +0000 (13:17 -0300)]
11557: Fixed can_read link creation, updated test to check this behavior.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Tom Clegg [Thu, 15 Jun 2017 16:11:13 +0000 (12:11 -0400)]
Merge branch '10557-auto-setup'
refs #10557
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 20:10:07 +0000 (16:10 -0400)]
11283: Fix exec_query() usage for rails4.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 20:05:39 +0000 (16:05 -0400)]
8784: Fix error returned by Readdir() when count > remain.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 19 Apr 2017 14:44:21 +0000 (10:44 -0400)]
11283: Fix "available slot number" query.
Fixes repetitive queries and excessive Postgres and Rails log
messages:
2017-04-10 16:39:09 UTC [4734-1] arvados@arvados_test ERROR: duplicate key value violates unique constraint "index_nodes_on_slot_number"
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_nodes_on_slot_number"
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 15:38:03 +0000 (11:38 -0400)]
10557: Remove redundant hook condition.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 15:26:27 +0000 (11:26 -0400)]
Merge branch '2411-check-copyright'
refs #2411
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Thu, 1 Jun 2017 18:50:17 +0000 (14:50 -0400)]
2411: Add build/check-copyright-notices.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 15:14:37 +0000 (11:14 -0400)]
Merge branch '11803-repo-perms'
refs #11803
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 13:25:14 +0000 (09:25 -0400)]
8784: Use arvados.Collection in cache.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 01:02:17 +0000 (21:02 -0400)]
10557: Always run user setup procedure when is_active becomes true.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 14 Jun 2017 01:01:41 +0000 (21:01 -0400)]
10557: Tidy up some user setup code.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Peter Amstutz [Tue, 13 Jun 2017 19:37:45 +0000 (15:37 -0400)]
test_with_arvbox.sh tags selected arvados/jobs version as latest refs #11458
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Tue, 13 Jun 2017 17:21:23 +0000 (13:21 -0400)]
Add --compute-checksum to test_with_arvbox.sh command line extras refs #11458
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Tom Clegg [Tue, 13 Jun 2017 15:48:49 +0000 (11:48 -0400)]
11803: Get group permissions with 1 query instead of N queries.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Tue, 13 Jun 2017 13:54:34 +0000 (09:54 -0400)]
8784: Show file sizes.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Peter Amstutz [Tue, 13 Jun 2017 14:48:24 +0000 (10:48 -0400)]
Remove apt-get clean when starting frorm bare debian image. no issue #
Tom Clegg [Mon, 12 Jun 2017 20:59:09 +0000 (16:59 -0400)]
8784: Correct wget args, tweak styles.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 7 Jun 2017 21:36:08 +0000 (17:36 -0400)]
8784: Add keep-web directory listings via (*arvados.Collection)FileSystem().
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Peter Amstutz [Tue, 13 Jun 2017 14:31:25 +0000 (10:31 -0400)]
Ensure that /var/cache/apt/archives is present so arvbox apt-get will work. no issue #
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Mon, 12 Jun 2017 14:02:42 +0000 (10:02 -0400)]
Remove node manager integration test wait check for quota set (event ordering), refs #10312
Peter Amstutz [Mon, 12 Jun 2017 13:35:20 +0000 (09:35 -0400)]
Fix integration test again, refs #10312
Peter Amstutz [Mon, 12 Jun 2017 13:12:30 +0000 (09:12 -0400)]
Fix integration test, refs #10312
Peter Amstutz [Sat, 10 Jun 2017 01:30:44 +0000 (21:30 -0400)]
Merge branch '11345-nodemanager-retry-after' refs #11345
Peter Amstutz [Sat, 10 Jun 2017 01:30:29 +0000 (21:30 -0400)]
Merge branch '10847-nodemanager-shutdown' refs #10847
Peter Amstutz [Sat, 10 Jun 2017 01:29:59 +0000 (21:29 -0400)]
Merge branch '10312-nodemanager-quotas' refs #10312
Peter Amstutz [Sat, 10 Jun 2017 01:24:46 +0000 (21:24 -0400)]
Merge branch '11642-null-job-log' refs #11642
Lucas Di Pentima [Fri, 9 Jun 2017 21:11:15 +0000 (18:11 -0300)]
11557: Moved logic to done.done(). Updated related tests.
Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lucas@curoverse.com>
Peter Amstutz [Fri, 9 Jun 2017 20:03:09 +0000 (16:03 -0400)]
11642: Jobs where log is unreadable (but not null) are not offered for reuse
but are taken into account in check for consistent outputs.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Thu, 8 Jun 2017 20:56:10 +0000 (16:56 -0400)]
11345: Clamp retry-after to (0, max_retry_wait). Deindent retry_wrapper a bit for readability.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Fri, 9 Jun 2017 18:25:59 +0000 (14:25 -0400)]
11642: Don't reuse completed jobs where output or log is null or not readable by current user.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Fri, 9 Jun 2017 17:27:51 +0000 (13:27 -0400)]
11642: Increase arv-put retries. Job considered failure if unable to save job log.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
radhika [Fri, 9 Jun 2017 18:12:01 +0000 (14:12 -0400)]
closes #11821
Merge branch '11821-discovery-doc-updates'
radhika [Fri, 9 Jun 2017 15:56:45 +0000 (11:56 -0400)]
11821: add include_trash for collections.index and recursive for groups.contents to discovery doc
Peter Amstutz [Thu, 8 Jun 2017 17:13:58 +0000 (13:13 -0400)]
11345: Fix race-prone test test_issue_slurm_drain_retry
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Tue, 6 Jun 2017 20:22:02 +0000 (16:22 -0400)]
11345: Fix unit tests after refactoring error types.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Tue, 6 Jun 2017 18:05:44 +0000 (14:05 -0400)]
11345: Simplify and consolodate retry for API throttling errors.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
radhika [Thu, 8 Jun 2017 19:34:25 +0000 (15:34 -0400)]
closes #11837
Merge branch '11837-trash-access'
Tom Clegg [Thu, 8 Jun 2017 18:21:21 +0000 (14:21 -0400)]
11837: Fix "include_trash" scope and test case.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
radhika [Thu, 8 Jun 2017 17:15:52 +0000 (13:15 -0400)]
11837: write tests
Peter Amstutz [Thu, 8 Jun 2017 15:55:08 +0000 (11:55 -0400)]
10847: Don't log everything from node manager unless test fails.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
radhika [Wed, 7 Jun 2017 21:18:15 +0000 (17:18 -0400)]
refs #9587
Merge branch '9587-trash-page'
radhika [Wed, 7 Jun 2017 21:16:34 +0000 (17:16 -0400)]
9587: Show "delete_at" timestamp instead of "created_at" timestamp in trash page
Peter Amstutz [Wed, 31 May 2017 19:37:15 +0000 (15:37 -0400)]
10847: Daemon shutdown now stops most actors, only waits for setup actors.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Tom Clegg [Wed, 7 Jun 2017 15:39:22 +0000 (11:39 -0400)]
Merge branch '11809-keep-web-cache'
closes #11809
Tom Clegg [Wed, 7 Jun 2017 15:28:22 +0000 (11:28 -0400)]
11809: Add tests for /status.json.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Peter Amstutz [Wed, 7 Jun 2017 14:57:25 +0000 (10:57 -0400)]
10312: Add example jobs_queue and slurm_queue options to example node manager configurations.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Peter Amstutz [Wed, 7 Jun 2017 14:53:43 +0000 (10:53 -0400)]
10312: Add some comments to node manager integration test.
Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curoverse.com>
Tom Clegg [Wed, 7 Jun 2017 14:51:15 +0000 (10:51 -0400)]
11809: Merge branch 'master' into 11809-keep-web-cache
Tom Clegg [Wed, 7 Jun 2017 14:47:37 +0000 (10:47 -0400)]
11809: Skip lookups in forceReload case. Add forceReload tests.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 7 Jun 2017 14:32:09 +0000 (10:32 -0400)]
11809: Increase default cache sizes.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
radhika [Wed, 7 Jun 2017 14:30:22 +0000 (10:30 -0400)]
closes #11652
Merge branch '11652-recursive-contents-wb'
radhika [Wed, 7 Jun 2017 14:29:30 +0000 (10:29 -0400)]
11652: fix comment on test
Tom Clegg [Wed, 7 Jun 2017 14:27:59 +0000 (10:27 -0400)]
11809: Prune collection cache asynchronously.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 7 Jun 2017 14:27:06 +0000 (10:27 -0400)]
11809: Rename FooEntries -> MaxFooEntries in cache config.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Wed, 7 Jun 2017 14:23:41 +0000 (10:23 -0400)]
11809: Add /status.json handler.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Lucas Di Pentima [Wed, 7 Jun 2017 13:45:53 +0000 (10:45 -0300)]
11557: Merge branch 'master' into 11557-acr-output-col-perms
radhika [Tue, 6 Jun 2017 19:49:34 +0000 (15:49 -0400)]
Merge branch 'master' into 11652-recursive-contents-wb
Tom Clegg [Tue, 6 Jun 2017 19:22:57 +0000 (15:22 -0400)]
Merge branch '11652-recursive-contents'
refs #11652
Tom Clegg [Tue, 6 Jun 2017 19:21:28 +0000 (15:21 -0400)]
11809: More cache tests.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Tue, 6 Jun 2017 19:12:53 +0000 (15:12 -0400)]
11809: Add unit test for cache.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
Tom Clegg [Tue, 6 Jun 2017 17:53:36 +0000 (13:53 -0400)]
11652: Merge branch 'master' into 11652-recursive-contents
Tom Clegg [Tue, 6 Jun 2017 17:53:25 +0000 (13:53 -0400)]
11652: Add tests for non-recursive and home-project cases.
radhika [Tue, 6 Jun 2017 17:06:19 +0000 (13:06 -0400)]
11652: use "recursive=true" in workbench search requests
Tom Clegg [Tue, 6 Jun 2017 15:57:34 +0000 (11:57 -0400)]
11809: Cache permission and collection lookups.
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curoverse.com>
radhika [Tue, 6 Jun 2017 15:17:29 +0000 (11:17 -0400)]
Merge branch 'master' into 11652-recursive-contents
radhika [Tue, 6 Jun 2017 15:07:36 +0000 (11:07 -0400)]
closes #9587
Merge branch '9587-trash-page'
radhika [Tue, 6 Jun 2017 15:01:06 +0000 (11:01 -0400)]
9587: rename pane name to "Recent trash"
projects / arvados.git / log