Peter Amstutz [Wed, 25 Feb 2015 19:09:09 +0000 (14:09 -0500)]
5310: Use c.get('name') instead of c['name']
* 'name' isn't necessarily present when obj_uuid is a PDH,
src.collections().get(uuid=obj_uuid).execute() may return a synthetic record
without a name field.
Peter Amstutz [Mon, 23 Feb 2015 21:36:09 +0000 (16:36 -0500)]
4520: Coerce unicode strings to ascii in put(). Use result.content (returns
literal result bytes) instead result.text (returns unicode) when processing
Keep responses.
Peter Amstutz [Mon, 23 Feb 2015 20:17:19 +0000 (15:17 -0500)]
4520: manifest_text() is utf-8 encoded by default so it can be safely put() to
Keep. Add test that calling put() with a unicode string raises an error.
Fetching user uuid in arv-copy uses num_retries.
Peter Amstutz [Mon, 23 Feb 2015 19:17:12 +0000 (14:17 -0500)]
4520: Better checking to see if collection already exists at the destination.
Set args.project_uuid to default value (current user uuid) if not set on
command line to simplify code.
Peter Amstutz [Mon, 23 Feb 2015 18:54:47 +0000 (13:54 -0500)]
4520: Refactor code to create the collection record. Also refactored code
which creates Docker metadata links so that copying any collection which
represents a Docker image will also copy over the metadata.
Phil Hodgson [Sat, 21 Feb 2015 09:16:31 +0000 (10:16 +0100)]
4232: revert experimental change to using find? for each of the jobs in a pipeline, rather than simply a where clause: there is no evidence that this switch to find? was helping to speed up anything overall
Brett Smith [Mon, 16 Feb 2015 16:06:41 +0000 (11:06 -0500)]
4138: Prepare Node Manager GCE driver for production.
* Set node metadata in more appropriate places.
* Bridge more differences between GCE and EC2, like the fact that
sizes are listed for each location they're available, and GCE
doesn't provide node boot times.
* Use more infrastructure from BaseComputeNodeDriver to reduce code
duplication.
* Load as many objects as possible at initialization time, to reduce
API overhead of creating nodes.
Brett Smith [Fri, 13 Feb 2015 20:24:04 +0000 (15:24 -0500)]
4138: Revamp Node Manager driver proxying in BaseComputeNodeDriver.
Accessing attributes through a super() proxy does not invoke
__getattr__ on base classes, so the old implementation made it
impossible for subclasses to be agnostic about whether a method was
implemented in BaseComputeNodeDriver or the real libcloud driver.
This version makes that possible. It's also a little nicer because
now the class will report these method names to dir(), hasattr(), etc.
Brett Smith [Thu, 12 Feb 2015 20:53:16 +0000 (15:53 -0500)]
4138: Fix noop Node Manager EC2 driver tests.
The previous tests simply instantiated the driver, then checked that a
mock method was truthy (which it will always be). This makes the test
work as intended.
Brett Smith [Wed, 11 Feb 2015 20:12:37 +0000 (15:12 -0500)]
4138: Simplify Node Manager GCE credential handling.
Because libcloud's GCE driver accepts a key path as a constructor
argument, it's relatively straightforward to put all the constructor
arguments directly in the Node Manager configuration. No need to
parse out JSON.
Tim Pierce [Fri, 23 Jan 2015 22:24:54 +0000 (17:24 -0500)]
4138: GCE fixes
The 'network_id' parameter needs to be delivered as 'location' in GCE.
The ping_url parameter is now delivered in the node metadata as
'pingUrl'.
When creating a new GCE instance, 'name' is a required parameter and
must begin with a letter. The default name is the UUID of the
corresponding Arvados node, prepended with 'arv-'.
Tim Pierce [Wed, 21 Jan 2015 18:06:35 +0000 (13:06 -0500)]
4138: general GCE fixes
* JSON credential file
** GCE credentials are delivered as a JSON string (and the key is formatted as a multi-line RSA private key). Let the GCE config file specify a path to the JSON credential file for simplicity.
* Accept NodeSizes addressed by id or name
** In EC2, NodeSizes are identified by the 'id' field. In GCE they are identified by the 'name' field. Allow the Node Manager config module to accept either.
Tom Clegg [Sat, 14 Feb 2015 21:54:00 +0000 (16:54 -0500)]
Ensure result order is predictable, even if client-provided orders do not specify a complete ordering.
Fixes intermittent test failures. Example (from
https://ci.curoverse.com/job/arvados-api-server/1305/console):
GroupsTest#test_get_all_pages_of_group-owned_objects [/data/1/jenkins/workspace/arvados-api-server/services/api/test/integration/groups_test.rb:31]:
Received 'zzzzz-4zz18-fy296fx3hot09f7' again on page 3.
<nil> expected but was
<true>.
mishaz [Fri, 30 Jan 2015 01:25:11 +0000 (01:25 +0000)]
Now fetch Keep Server Status and record it to the log. Renamed some fields and added a comment for a potential improvement to decrease lock contention.
mishaz [Tue, 27 Jan 2015 01:06:21 +0000 (01:06 +0000)]
Renamed BlockDigest's ToString() to String() to implement fmt.Stringer() interface so that we get more readable error messages when structs contain BlockDigests.
mishaz [Sat, 24 Jan 2015 02:22:01 +0000 (02:22 +0000)]
A bunch of changes, most in response to Peter's review.
Logger:
Edit() and Record() have been replaced with the single Update() method which takes a function as input (suggested by Tom).
lastWrite replaced by nextWriteAllowed, for cleaner logic
Added writeScheduled to reduce the number of writes scheduled and attempted, thereby reducing lock contention
Added sanity-checking of params
A bunch of overdue cleanup
Update documentation to reflect the above changes
Manifest:
Renamed ManifestLine to ManifestStream
Util:
Deleted a lot of crap that proved less useful than I thought.
Moved collection.NumberCollectionsAvailable() to util.NumberItemsAvailable() and made it more generic.
collection:
Just cleanup in response to changes in above packages.
keep:
Switched Mtime from int to int64 to avoid y2038 problems.
Switched approach for avoiding keep proxy from using "accessible" to filtering on service_type = disk.
Cleanup in response to changes in above packages.
loggerutil:
Cleanup in response to changes in logger.
projects / arvados.git / log