radhika [Fri, 14 Nov 2014 19:50:59 +0000 (14:50 -0500)]
4194: since LoggingResponseWriter is doing the logging, remove some of the old log statements that are now covered by the LoggingResponseWriter in keepstore handlers.
Tim Pierce [Wed, 12 Nov 2014 21:07:15 +0000 (16:07 -0500)]
3609: fix "NameError: name 'rcode' is not defined"
If run-command raises an exception in the main 'try' loop before any
subprocesses have run, rcode will not be defined and "not any([status !=
0 for status in rcode.values()])" will raise a confusing "NameError:
name 'rcode' is not defined" exception.
Solution is to define rcode before the try/except block and to make sure
'success' is False if rcode is completely empty.
Brett Smith [Tue, 11 Nov 2014 22:23:14 +0000 (17:23 -0500)]
4380: Reorganize arvnodeman.computenode.
This makes the hierarchy a little richer:
* arvnodeman.computenode.driver has all the cloud driver wrappers.
* arvnodeman.computenode.dispatch will be just like that, except it
will consider local dispatch concerns. For example, I'm going to
add a SLURM submodule here to take care of draining.
* arvnodeman.computenode still has utility functions and
ShutdownTimer.
Peter Amstutz [Wed, 12 Nov 2014 16:39:41 +0000 (11:39 -0500)]
3400: Stub out obsolete #links_for and remove result_links and related test.
Set @result_limit and @result_offset to nil unless @fetch_multiple_pages ==
false. Add test for limit > default page size.
Brett Smith [Wed, 12 Nov 2014 14:42:17 +0000 (09:42 -0500)]
3603: PySDK Collection objects support file-like APIs.
This commit adds an open() method to CollectionReader and
CollectionWriter. They mimic the built-in open(), returning objects
that implement as much as the Python file API as I can reasonably
manage.
There are a couple of backwards-incompatible changes to
StreamFileReader here:
* seek() now defaults to SEEK_CUR rather than SEEK_SET behavior.
* readlines() returns a list, and no longer supports decompression.
These changes bring the object closer to a file-like object. We
reviewed our existing Python code, including known Crunch scripts from
users. Nobody was using seek(), and nobody was obviously relying on
the decompression behavior of readlines(). So we believe this is
reasonably safe.
(Contrast name(), which lots of things are using, so we provide a
backward compatibility shim for it.)
Tim Pierce [Tue, 11 Nov 2014 20:59:23 +0000 (15:59 -0500)]
4294: additional unit tests
test_server_calc_returns_min_nodes_for_one_job
* Ensures that the @ServerCalculator@ still returns a wishlist with
_min_nodes_ when there is a job in the queue.
test_server_calc_more_jobs_than_min_nodes
* Tests that when the job queue has between _min_nodes_ and _max_nodes_ in
it, @ServerCalculator@ returns a wishlist exactly as long as the job
queue.
Tim Pierce [Tue, 11 Nov 2014 16:04:19 +0000 (11:04 -0500)]
3857: code review feedback
* Set default timeouts to (20, 300) for proxies and (2, 300) for all
else.
* KeepService:
** Removed unused 'timeout' argument to __init__()
** Trimmed unnecessary exceptions in last_status()
* Tests:
** test_collections.py: changed mock_api_call return value back to
fake_httplib2_response
** Flattened test layers to eliminate KeepClientRetryGetTestMixin and
KeepClientRetryPutTestMixin
** Moved proxy timeout tests to
KeepClientServiceTestCase.test_proxy_get_request and
.test_proxy_put_request
* Removed excess blank lines in docstrings
Tim Pierce [Fri, 7 Nov 2014 14:37:24 +0000 (09:37 -0500)]
4294: added min_nodes config parameter
Added min_nodes configuration setting. The job queue will return a
wishlist with at least min_nodes elements in it, and the node manager
daemon will avoid shutting down a node if it would bring the total below
min_nodes.
* arvnodeman.config sets default min_nodes to 0.
* NodeManagerDaemonActor:
** _nodes_wanted returns at least enough to keep the node count above
min_nodes
** _nodes_excess will not allow the node count to drop below min_nodes
* jobqueue.ServerCalculator.servers_for_queue returns a list with at least
min_nodes elements
* added test cases:
** NodeManagerDaemonActorTestCase.test_shutdown_declined_below_min_nodes
** ServerCalculatorTestCase.test_server_calc_returns_at_least_min_nodes
Peter Amstutz [Fri, 7 Nov 2014 20:48:22 +0000 (15:48 -0500)]
3400: ArvadosResourceList retains api token that is active when it is created.
Tweaked fetch-multiple-pages loop. Made #each_page protected. _paging sets
fetch_multiple_pages(false).
Added unit tests to check that KeepClient.get and KeepClient.put use the
default 'timeout' setting when talking directly to a Keep server, and
the 'proxy_timeout' setting when communicating with a proxy.
Earlier we refactored our API token loading code to load User.current
to check that it was valid. This approach doesn't work when we're
presenting shared Collections, because the token usually won't be
scoped to get user information. Skip the check for this specific
case.
projects / arvados.git / log