Radhika Chippada [Thu, 7 May 2015 18:17:55 +0000 (14:17 -0400)]
3454: use configured default docker image when none found in a job's runtime_constraints.
Tom Clegg [Thu, 7 May 2015 17:47:16 +0000 (13:47 -0400)]
Merge branch '5745-serialize-content-only' closes #5745
Tom Clegg [Thu, 7 May 2015 16:48:48 +0000 (12:48 -0400)]
5745: Fix test order dependency (restore mocked methods), tidy up test data.
Tom Clegg [Wed, 6 May 2015 16:56:34 +0000 (12:56 -0400)]
5745: Serialize writes and data reads, but allow concurrent requests
to do read-only non-data operations (like finding existing blocks and
checking free disk space) which are likely to be cached by the OS and
therefore not involve any disk activity.
Also:
* Serialize Touch and Delete.
* Make sure to close and delete tempfiles on write errors.
* Update comments.
Tom Clegg [Thu, 7 May 2015 15:47:36 +0000 (11:47 -0400)]
Merge branch '5523-stats-error' closes #5523
Ward Vandewege [Wed, 6 May 2015 21:31:36 +0000 (17:31 -0400)]
Fix docker image build for SSO server after refactoring work in #5829.
refs #5829
Nancy Ouyang [Wed, 6 May 2015 21:28:05 +0000 (17:28 -0400)]
refs #5493: Merge branch '5493-fix-yellow-gif'
Nancy Ouyang [Wed, 6 May 2015 21:27:22 +0000 (17:27 -0400)]
5493: fix yellow gif
Tom Clegg [Wed, 6 May 2015 21:13:36 +0000 (17:13 -0400)]
Merge branch '5748-keepstore-leak' refs #5748
Tom Clegg [Wed, 6 May 2015 21:10:35 +0000 (17:10 -0400)]
5748: Clean up comments and variable names.
Tom Clegg [Wed, 6 May 2015 17:36:16 +0000 (13:36 -0400)]
5748: Write index data to http.ResponseWriter, instead of using string
concatenation to buffer the entire response.
Brett Smith [Wed, 6 May 2015 20:57:29 +0000 (16:57 -0400)]
Merge branch '5842-node-manager-tighten-successful-boot-criteria-wip'
Closes #5842, #5931.
Brett Smith [Wed, 6 May 2015 19:59:34 +0000 (15:59 -0400)]
5842: Node Manager only considers nodes busy if they're working.
Previously, Node Manager considered any non-idle node as busy,
including down nodes. This causes it to boot replacements for nodes
that are marked "down," even if that's because they're still
bootstrapping. Tighten the busy criteria to avoid booting excess
nodes.
It's easier to make this change now that Node Manager checks that
nodes have a functional Crunch worker state to be considered a
successful bootstrap. This means that any node that's down later is
in an unexpected state, and we should avoid messing with it.
Brett Smith [Wed, 6 May 2015 20:07:33 +0000 (16:07 -0400)]
5842: Node Manager shuts down booted nodes if they can't do compute work.
If a booted node is not able to do work (it's not idle or busy),
consider that a bootstrapping failure and shut it down, just like a
failure to pair with an Arvados node.
Ward Vandewege [Wed, 6 May 2015 20:17:49 +0000 (16:17 -0400)]
Merge branch '5926-parameterize-max-slots'
closes #5926
Ward Vandewege [Wed, 6 May 2015 20:16:53 +0000 (16:16 -0400)]
Improve comment in application.default.yml based on review feedback.
refs #5926
Ward Vandewege [Wed, 6 May 2015 19:46:28 +0000 (15:46 -0400)]
Make MAX_SLOTS a configurable parameter, and rename it to max_compute_nodes.
refs #5926
Tom Clegg [Wed, 6 May 2015 14:09:21 +0000 (10:09 -0400)]
5523: Say "notice" instead of "error" when stats file changes/appears/disappears.
Radhika Chippada [Tue, 5 May 2015 19:35:58 +0000 (15:35 -0400)]
closes #5735
Merge branch '5735-edit-description-box-size'
Radhika Chippada [Tue, 5 May 2015 19:26:07 +0000 (15:26 -0400)]
5735: move tiptitle into nonhtml_options
Radhika Chippada [Tue, 5 May 2015 18:16:38 +0000 (14:16 -0400)]
Merge branch 'master' into 5735-edit-description-box-size
Radhika Chippada [Tue, 5 May 2015 18:12:54 +0000 (14:12 -0400)]
5735: add comment
Radhika Chippada [Tue, 5 May 2015 17:53:28 +0000 (13:53 -0400)]
5735: add a new nonhtml_options parameter to render_editable_attribute method,
which can be used to pass any non data-* custom attributes that are useful in
customizing the rendering of the editable attribute.
Radhika Chippada [Mon, 4 May 2015 23:59:35 +0000 (19:59 -0400)]
5735: use data- for button text and button class custom attributes.
Radhika Chippada [Mon, 4 May 2015 23:11:08 +0000 (19:11 -0400)]
5735: exclude btn* attributes from span tag.
Tom Clegg [Mon, 4 May 2015 20:46:17 +0000 (16:46 -0400)]
Merge branch '5724-blob-signature-ttl' closes #5724
Radhika Chippada [Mon, 4 May 2015 19:50:25 +0000 (15:50 -0400)]
Merge branch 'master' into 5735-edit-description-box-size
Radhika Chippada [Mon, 4 May 2015 19:48:55 +0000 (15:48 -0400)]
5735: use the inline css for description only so that any future inline css declarations are not impacted with this.
Brett Smith [Mon, 4 May 2015 17:07:19 +0000 (13:07 -0400)]
Merge branch '5752-arv-copy-preserve-docker-timestamps-wip'
Closes #5752, #5881.
Brett Smith [Fri, 1 May 2015 18:07:04 +0000 (14:07 -0400)]
5752: arv-copy preserves properties of docker_image links.
The properties include the image_timestamp, which tells Arvados when
the source image was created, rather than the containing collection.
The API server uses this information to determine which image is
newest.
Radhika Chippada [Mon, 4 May 2015 16:28:57 +0000 (12:28 -0400)]
Merge branch 'master' into 5735-edit-description-box-size
Peter Amstutz [Mon, 4 May 2015 15:40:10 +0000 (11:40 -0400)]
Merge branch '5787-run-command-checkin' refs #5787
Peter Amstutz [Mon, 4 May 2015 15:24:27 +0000 (11:24 -0400)]
5787: Change caught_error to last_error in vwd.py to return last exception that was caught. Fix typo.
Tom Clegg [Mon, 4 May 2015 15:18:37 +0000 (11:18 -0400)]
Merge branch '5893-helper-stdin' closes #5893
Tom Clegg [Mon, 4 May 2015 14:27:59 +0000 (10:27 -0400)]
Merge branch '5724-blob-signature-ttl' commit '
6fc44a6' refs #5724
Tom Clegg [Mon, 4 May 2015 14:27:41 +0000 (10:27 -0400)]
5724: Update reference to keepstore flag, now -blob-signature-ttl.
Peter Amstutz [Mon, 4 May 2015 13:07:33 +0000 (09:07 -0400)]
Merge branch '5562-pycurl-upload' closes #5562
Tom Clegg [Sat, 2 May 2015 06:49:44 +0000 (02:49 -0400)]
5893: Remove "disable repo browsing with git2" stuff.
git2 is no longer unreliable, now that we know the silent failures
meant "git credential helpers must consume their stdin".
Tom Clegg [Sat, 2 May 2015 06:16:19 +0000 (02:16 -0400)]
5893: Use git credential helpers for arv-git-httpd tests. Fix workbench helper.
The sporadic "git exited 128" errors -- very common in git2 and rare
in git1 -- seem to have been caused by git getting SIGPIPE when the
credential helper exited without consuming stdin. The solution is for
the credential helper to discard its standard input before exiting.
Tom Clegg [Sat, 2 May 2015 06:09:54 +0000 (02:09 -0400)]
5893: gofmt fixes. Fix some logs to print as strings, not byte arrays.
Tom Clegg [Fri, 1 May 2015 23:47:09 +0000 (19:47 -0400)]
5893: Add realm="git" to WWW-Authenticate header, cf. rfc2617
Peter Amstutz [Fri, 1 May 2015 21:07:33 +0000 (17:07 -0400)]
5787: Fix syntax error
Peter Amstutz [Fri, 1 May 2015 21:02:54 +0000 (17:02 -0400)]
5787: Error writing output will cause task to fail. Tweak documentation page.
Brett Smith [Fri, 1 May 2015 20:48:42 +0000 (16:48 -0400)]
Merge branch '5834-api-max-response-size-wip'
Closes #5834, #5871.
Brett Smith [Thu, 30 Apr 2015 22:23:21 +0000 (18:23 -0400)]
5834: Limit how much manifest text data API server will load for index.
This prevents situations where clients cause the API server to balloon
in memory use by requesting an index including many large manifest
texts. Data Manager has been doing this unwittingly lately.
Radhika Chippada [Fri, 1 May 2015 19:43:35 +0000 (15:43 -0400)]
5735: use inline editing for project description x-editable textarea.
Peter Amstutz [Fri, 1 May 2015 18:43:41 +0000 (14:43 -0400)]
5787: Write files that are pointed to by symlinks that are outside of Keep.
Catch exceptions. Update documentation.
Radhika Chippada [Fri, 1 May 2015 15:36:30 +0000 (11:36 -0400)]
closes #4458
Merge branch '4458-breadcrumbs-in-small-screens'
Radhika Chippada [Fri, 1 May 2015 15:35:17 +0000 (11:35 -0400)]
Merge branch 'master' into 4458-breadcrumbs-in-small-screens
Peter Amstutz [Fri, 1 May 2015 14:49:18 +0000 (10:49 -0400)]
5787: Copy to the right destination name, add docstring, fix syntax errors
Peter Amstutz [Fri, 1 May 2015 14:32:01 +0000 (10:32 -0400)]
5787: run-command uploading uses new collection API and supports symlinks into Keep
Radhika Chippada [Fri, 1 May 2015 14:26:40 +0000 (10:26 -0400)]
closes #5843
Merge branch '5843-subproject-deletion-owner-name-constraint'
Radhika Chippada [Fri, 1 May 2015 14:26:01 +0000 (10:26 -0400)]
Merge branch 'master' into 5843-subproject-deletion-owner-name-constraint
Tom Clegg [Fri, 1 May 2015 07:13:47 +0000 (03:13 -0400)]
5724: gofmt fixes.
Tom Clegg [Fri, 1 May 2015 07:11:46 +0000 (03:11 -0400)]
5724: Rename -permission-* flags to -blob-sign* to match apiserver. Tidy up some time-related code.
Tom Clegg [Fri, 1 May 2015 04:59:20 +0000 (00:59 -0400)]
5724: Add blobSignatureTtl to discovery doc. Add config comment/explanation.
Peter Amstutz [Thu, 30 Apr 2015 21:14:07 +0000 (17:14 -0400)]
5562: keepstub sends "100 Continue" to client. Fix test_write_three_replicas
to return a valid response body instead of None.
Peter Amstutz [Thu, 30 Apr 2015 20:38:24 +0000 (16:38 -0400)]
run-command list correct directory to be uploaded when $(task.cwd) != $(task.outdir)
no issue #
Peter Amstutz [Thu, 30 Apr 2015 18:39:46 +0000 (14:39 -0400)]
5562: Use cStringIO module which is more efficient than StringIO.
Peter Amstutz [Thu, 30 Apr 2015 20:21:49 +0000 (16:21 -0400)]
5562: Add comment about UPLOAD
Peter Amstutz [Thu, 30 Apr 2015 17:55:36 +0000 (13:55 -0400)]
5562: Add support for "Expect: 100-Continue" flow for PUT.
This tells cURL to wait for a "go ahead" from the Keep server (in the form of a
HTTP/1.1 "100 Continue" response) instead of sending the request body
immediately. This allows the server to reject the request if the request is
invalid or the server is read-only, without waiting for the client to send the
entire block.
Uses pycurl.UPLOAD to enable the 'PUT' and 'Expect: 100-continue" behavior.
Use INFILESIZE to sent Content-Length and use READFUNCTION to actually spool
the data.
Peter Amstutz [Thu, 30 Apr 2015 20:07:12 +0000 (16:07 -0400)]
Merge branch '3145-missing-content-length-error' closes #3145
Peter Amstutz [Thu, 30 Apr 2015 18:30:09 +0000 (14:30 -0400)]
Merge branch '5856-read-exact' closes #5856
Peter Amstutz [Thu, 30 Apr 2015 18:21:41 +0000 (14:21 -0400)]
5856: Tweak tests. Fix typo.
Radhika Chippada [Thu, 30 Apr 2015 18:19:42 +0000 (14:19 -0400)]
4458: breadcrumbs now visible in small window sizes; also, use user icon instead of email for my account menu.
Peter Amstutz [Thu, 30 Apr 2015 17:59:22 +0000 (13:59 -0400)]
Return error if Content-Length header is missing instead of panicing.
Peter Amstutz [Thu, 30 Apr 2015 15:18:24 +0000 (11:18 -0400)]
ArvadosFileReader object always tries to return the exact amount of data asked for.
This is to avoid breaking stuff like gzip that assumes it always gets the exact
amount of data unless EOF. (From the Python file docs for read(): Note that
this method may call the underlying C function fread() more than once in an
effort to acquire as close to size bytes as possible.) closes #5856
Tom Clegg [Thu, 30 Apr 2015 15:30:20 +0000 (11:30 -0400)]
Merge branch '5562-pycurl' refs #5562
Tom Clegg [Thu, 30 Apr 2015 15:27:46 +0000 (11:27 -0400)]
5562: Use static method. Fix "TypeError: _socket_open() takes exactly 5 arguments (4 given)".
Radhika Chippada [Thu, 30 Apr 2015 14:59:29 +0000 (10:59 -0400)]
5843: rename objects that contain xx_owner_uuid_name_unique constraint when they are removed from a subproject and the home project already has another object of the same type and name.
Tom Clegg [Thu, 30 Apr 2015 14:25:13 +0000 (10:25 -0400)]
Merge branch '3145-close-early' closes #3145
Tom Clegg [Thu, 30 Apr 2015 14:24:16 +0000 (10:24 -0400)]
Merge branch '5836-remote-api-server-errors-made-obvious' closes #5836
Tom Clegg [Thu, 30 Apr 2015 13:53:15 +0000 (09:53 -0400)]
5562: Use WRITEFUNCTION instead of WRITEDATA for compatibility with older pycurl. refs #5562
Tom Clegg [Wed, 29 Apr 2015 18:56:22 +0000 (14:56 -0400)]
3145: Respond to PUT without reading request body if no volumes are writable.
Tom Clegg [Wed, 29 Apr 2015 17:45:46 +0000 (13:45 -0400)]
Merge branch '5562-pycurl' closes #5562
Tom Clegg [Wed, 29 Apr 2015 15:10:40 +0000 (11:10 -0400)]
5562: Return UA to pool if it worked well enough to get an HTTP status.
Tom Clegg [Wed, 29 Apr 2015 14:57:47 +0000 (10:57 -0400)]
5562: Enable TCP Keepalive for Keep requests. refs #5822
mishaz [Wed, 29 Apr 2015 16:31:00 +0000 (16:31 +0000)]
Renamed RemoteApiServerError to APIServerError and NewRemoteApiServerError to newAPIServerError.
Reworked error messages to feature error details more prominently.
These changes were requested by code review at
https://github.com/curoverse/arvados/pull/17/files
Radhika Chippada [Wed, 29 Apr 2015 13:07:10 +0000 (09:07 -0400)]
closes #5826
Merge branch '5826-unreadable-inputs-error'
Radhika Chippada [Wed, 29 Apr 2015 12:44:49 +0000 (08:44 -0400)]
5826: use present? instead of !empty?
Tom Clegg [Tue, 28 Apr 2015 23:08:45 +0000 (19:08 -0400)]
5562: Add fake keepstore server with configurable problems.
Tom Clegg [Wed, 22 Apr 2015 18:26:43 +0000 (14:26 -0400)]
5562: Use pycurl library (instead of requests) for Keep transactions.
mishaz [Tue, 28 Apr 2015 22:29:27 +0000 (22:29 +0000)]
Added tests to confirm that error messages indicate that error is remote.
Fixed punctuation.
mishaz [Tue, 28 Apr 2015 22:16:46 +0000 (22:16 +0000)]
Rewrote ArvadosApiError as RemoteApiServerError:
* Name change to indicate that it's a remote error, not a local one.
* Rewrote Error() method to explicitly state that the error is remote and list all fields.
* Added server address field and use it in error response.
* Error details are now kept as a string slice rather than as a delimited string.
Some cleanup and gofmt import shuffling.
Radhika Chippada [Tue, 28 Apr 2015 22:03:58 +0000 (18:03 -0400)]
5826: revert the previous update that checked if readable only for pdh or uuid;
instead check as long as a collection datatype is presented with a non-empty string.
Radhika Chippada [Tue, 28 Apr 2015 18:43:34 +0000 (14:43 -0400)]
closes #5622
Merge branch '5622-fix-timeout-pdh-page'
Radhika Chippada [Tue, 28 Apr 2015 18:06:28 +0000 (14:06 -0400)]
5622: when there are too many collections matching a pdh, show only 20 of them.
Peter Amstutz [Tue, 28 Apr 2015 13:46:58 +0000 (09:46 -0400)]
Check if cache slot is not None before trying to test the read flag. refs #5831
Tom Clegg [Tue, 28 Apr 2015 06:10:14 +0000 (02:10 -0400)]
Merge branch '5739-no-sqlite3' closes #5739
Radhika Chippada [Mon, 27 Apr 2015 22:48:09 +0000 (18:48 -0400)]
closes #5826
closes #5827
Merge branch '5826-unreadable-inputs-error'
Radhika Chippada [Mon, 27 Apr 2015 22:30:22 +0000 (18:30 -0400)]
5827: check readable only attrvalue is a string
Brett Smith [Mon, 27 Apr 2015 21:52:35 +0000 (17:52 -0400)]
Merge branch '5490-crunch-tighten-job-reuse-wip'
Closes #3341, #5388, #5490, #5688, #5784.
Brett Smith [Tue, 21 Apr 2015 20:48:42 +0000 (16:48 -0400)]
5490: Require exact matches in API server's default job reuse filters.
The previous default filters would reuse jobs that used any Docker
image with the matching name, and/or Arvados SDK version since the
named commit. User feedback indicates this is surprising, and they
would prefer to have behavior more like the handling around
script_version: only reuse jobs that have the latest version of a
symbolic name, or the exact specified hash.
Brett Smith [Tue, 21 Apr 2015 15:35:01 +0000 (11:35 -0400)]
5490: Test API server finds Docker images with hash names.
This is in the same spirit as the previous, analogous commit for git
branches.
Brett Smith [Tue, 21 Apr 2015 15:23:19 +0000 (11:23 -0400)]
5490: Test API server handling of git hash-like branch names.
If you have a branch that looks like a commit hash, different git subcommands do different things. After extended discussion on IRC, we've decided we'd like the rules to be:
* If a 40-character hex string references a commit, resolve it to that
commit.
* In other cases of ambiguity, prefer branch names over short commit
hashes.
Fortunately, this corresponds to the behavior of `git rev-list` and
our existing implementation. This commit codifies our desired
behavior with tests.
Brett Smith [Thu, 16 Apr 2015 21:33:22 +0000 (17:33 -0400)]
5490: Clean up API server previous job fixtures.
* Every fixture that specified an arvados_sdk_version constraint
should also specify a docker_image constraint, to better match
reality.
* Use the same Docker image name in link and previous job fixture
constraints.
Brett Smith [Thu, 16 Apr 2015 21:32:54 +0000 (17:32 -0400)]
5490: Refactor API server job_reuse_controller_test.
Teach the convenience methods to start from any fixture job as a basis
for submitting new ones and comparing.
Radhika Chippada [Mon, 27 Apr 2015 20:58:10 +0000 (16:58 -0400)]
5826: check readable only when it is a data collection pdf or uuid.
Brett Smith [Mon, 27 Apr 2015 15:23:50 +0000 (11:23 -0400)]
Merge branch '5554-delete-job-log-rows-wip'
Closes #5554, #5798.
Brett Smith [Fri, 24 Apr 2015 21:13:31 +0000 (17:13 -0400)]
5554: Workbench links finished job logs from pipeline instance log tab.
This is an affordance to help users find logs after they're removed
from the API server's logs table. The display is not very great.
We're punting improvements until Workbench generally has better
capabilities to display large logs.
The layout change necessitated updates to the anonymous access tests.
While I was in there, I strengthened the assertions by putting
assertions that check a particular element has loaded before asserting
the nonexistence of other elements. This revealed the JavaScript
issue mentioned in the new comment: the old version of the test was
doing a single `assert_no_text` after clicking a job's log tab, which
only confirmed that we navigated away. We already test that the log
tab exists and is not disabled in this case, so I consider that
sufficient for now.