mishaz [Wed, 7 Jan 2015 04:16:40 +0000 (04:16 +0000)]
Started focusing on Keep Server responses again. Switched to using blockdigest instead of strings. Added per block info so that we can track block replication across servers.
mishaz [Wed, 24 Dec 2014 20:26:38 +0000 (20:26 +0000)]
Added string copying to try to reduce memory usage, didn't seem to work. Cleaned up logging (and logging logic) so that we only see one line per batch.
mishaz [Wed, 24 Dec 2014 01:36:43 +0000 (01:36 +0000)]
Switched from strings to BlockDigests to hold block digests more efficiently. Started clearing out manifest text once we finished with it. Made profiling conitional on flag (before it crashed if not provided). Added final heap profile once collections were finished.
mishaz [Tue, 23 Dec 2014 23:55:12 +0000 (23:55 +0000)]
Added blockdigest class to store digests more efficiently. This has the nice side effect of reducing how many string slices we use from the SDK, so the large string can get garbage collected once we remove other usages.
mishaz [Tue, 23 Dec 2014 19:33:07 +0000 (19:33 +0000)]
Long overdue checkin of data manager. Current code runs, but uses way too much memory and eventually crashes. This checkin includes heap profiling to track down memory usage.
mishaz [Sat, 22 Nov 2014 00:57:40 +0000 (00:57 +0000)]
Added reporting of disk usage. This is the Collection Storage of each user as described here: https://arvados.org/projects/arvados/wiki/Data_Manager_Design_Doc#Reports-Produced
But it does not include the size of projects owned by the user (projects and subprojects are each reported as their own users)
Brett Smith [Mon, 13 Oct 2014 19:19:15 +0000 (15:19 -0400)]
4139: Speed up Node Manager tests.
Previously, the tests would poll interesting mocks, waiting for them
to be called. This introduces significant overhead to the tests, and
they would frequently time out on Jenkins. This modifies the tests to
get more information by blocking on the tested actors, which means
more predictability and less fighting for CPU (typical runtimes for
all the tests improved from 5 seconds to 0.5 seconds on my
workstation).
The downside to this approach is that it ties the tests more tightly
to the underlying actors' implementation. In particular, they
sometimes send a message and block for a response to ensure that any
internal messages generated by the *last* message have been handled.
This is less than ideal, but I don't have a better idea right now.
mishaz [Fri, 10 Oct 2014 20:27:36 +0000 (20:27 +0000)]
Added tests to check that we're iterating on manifest lines correctly and handling blank lines in manifests.
LineIter will now handle long manifest lines properly and added test to check that we continue to do so.
Peter Amstutz [Fri, 10 Oct 2014 19:43:52 +0000 (15:43 -0400)]
Websocket server side fix, perform database notify in after_save callback on
the log object instead of in log_change on ArvadosBase because crunch-dispatch
was creating Log objects directly and bypassing the notification in log_change.
Brett Smith [Thu, 9 Oct 2014 17:32:50 +0000 (13:32 -0400)]
4139: Add *.egg to Node Manager's .gitignore.
`python setup.py test` will automatically download dependencies to the
source directory if you don't already have them available in your
environment. Refs #2881, #4139.
Brett Smith [Thu, 9 Oct 2014 17:31:47 +0000 (13:31 -0400)]
4139: Add environment configuration knobs for Node Manager tests.
These are settings I've fiddled with regularly over the course of
development, and now it looks like we're going to need to fiddle them
some more to accommodate Jenkins. I'm exposing them as environment
variables so I can stop messing with the code appropriately.
Refs #4139.
Peter Amstutz [Thu, 9 Oct 2014 13:32:36 +0000 (09:32 -0400)]
4042: Typo fixes. Highlight run-command and script_parameters in text. Rename
--job-parameters to --script-parameters and add mention of --dry-run mode.
Peter Amstutz [Wed, 8 Oct 2014 18:50:12 +0000 (14:50 -0400)]
Merge branch '4042-run-command-MxN' of git.curoverse.com:arvados into 4042-run-command-MxN
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
Peter Amstutz [Wed, 8 Oct 2014 15:12:57 +0000 (11:12 -0400)]
4042: Rename bad reuse of 'p' to 'match' in expand_item. Finish describing
$(task.outdir). Clarify that list functions take a user parameter name. Fix
other spelling and grammatical errors in documentation.
Tom Clegg [Tue, 7 Oct 2014 20:24:15 +0000 (16:24 -0400)]
4044: Clean up channel and pipe usage.
* Remove code for using one goroutine to copy stdout and stderr from
channels to pipes. Instead, copy stderr from channel to pipe in a
simple goroutine, and use give the child our own Stdout to use (we
don't use it ourselves anyway).
radhika [Tue, 7 Oct 2014 17:55:32 +0000 (13:55 -0400)]
3882: update "after_validation :update_timestamps_when_state_changes" to "before_save :update_timestamps_when_state_changes",
and also add comple more test combinations around cancelled state.
Tom Clegg [Tue, 7 Oct 2014 02:06:50 +0000 (22:06 -0400)]
3775: Update perlpod. Use items_available. Be more conservative when
using already-cached commit sha1s, expand related comment. Move "skip
if not possible now" checks into subroutines.
Brett Smith [Mon, 6 Oct 2014 21:36:31 +0000 (17:36 -0400)]
3634: Update user setup tests for preserved tab state.
These tests assume that the page reloads after submitting Ajax
dialogs. They started failing when we started preserving tab state.
Update the tests to expressly refresh the page. Future improvements
might find a solution with lower overhead.
Peter Amstutz [Mon, 6 Oct 2014 19:03:37 +0000 (15:03 -0400)]
3381: Merge job_status_label and job_progress into a single job_progress
partial. This partial renders a progress bar if the job is running, otherwise
renders a label with the job state. The progress bar now shows only 'done'
tasks and renders the progress bar in orange if any tasks have failed. Move
"done, failure, running, todo" from panel body to panel heading on
running_component partial. Dashboard now uses job_progress partial with
"scaleby" to indicate pipeline progress more precisely.
projects / arvados.git / log