Tim Pierce [Fri, 13 Jun 2014 14:44:21 +0000 (10:44 -0400)]
2934: add launch_crunch_dispatch to integration test
Added functionality for launching crunch-dispatch in
crunch_dispatch_test.rb while integration tests are running.
Refs #2934.
Tim Pierce [Fri, 13 Jun 2014 14:39:00 +0000 (10:39 -0400)]
Merge branch '2934-limit-crunch-logs' of git.curoverse.com:arvados into 2934-limit-crunch-logs
Fixing broken rebase
Peter Amstutz [Fri, 13 Jun 2014 14:13:39 +0000 (10:13 -0400)]
Merge branch 'origin-2883-job-log-viewer' closes #2883
Peter Amstutz [Fri, 13 Jun 2014 14:10:40 +0000 (10:10 -0400)]
2883: Moved addToLogViewer function out of the template and into assets and
reformatted it properly.
Brett Smith [Thu, 12 Jun 2014 19:45:36 +0000 (15:45 -0400)]
Merge branch '2879-docker-image-installer'
Refs #2879. Closes #2998, #3006.
Brett Smith [Thu, 12 Jun 2014 19:36:18 +0000 (15:36 -0400)]
2879: arv-keepdocker reports a better error for ambiguous hashes.
Refs #2879.
Brett Smith [Thu, 12 Jun 2014 18:51:23 +0000 (14:51 -0400)]
2879: Clarify Docker image search behavior in Keep installer.
Brett Smith [Thu, 12 Jun 2014 18:44:55 +0000 (14:44 -0400)]
2879: Docker Keep installer returns 0 if image already installed.
Brett Smith [Tue, 10 Jun 2014 20:34:01 +0000 (16:34 -0400)]
2879: Add arv-keepdocker command.
This puts a Docker image in Keep, and makes Arvados links to help find
the Collection by the image's names.
Brett Smith [Tue, 10 Jun 2014 15:09:42 +0000 (11:09 -0400)]
2879: arv-put main() returns what it prints.
This will enable other tools to reuse the results.
Brett Smith [Mon, 9 Jun 2014 16:02:01 +0000 (12:02 -0400)]
2879: Move Python is_hex function to arvados.util.
Brett Smith [Mon, 9 Jun 2014 15:13:39 +0000 (11:13 -0400)]
2879: Split arv-put's ArgumentParser into module segments.
This will let us use different pieces as a parent parser for other
tools, which I plan to do for the Docker image uploader.
Brett Smith [Mon, 9 Jun 2014 15:06:20 +0000 (11:06 -0400)]
2879: Factor out arv-put's cache creation to a utility library.
This function will be useful for the Docker image uploader, and
probably other tools as well.
Tom Clegg [Thu, 12 Jun 2014 19:21:59 +0000 (15:21 -0400)]
2755: Merge branch '2755-require-keep-permission' refs #2755
Tom Clegg [Thu, 12 Jun 2014 19:14:22 +0000 (15:14 -0400)]
2755: Fix handling of stripped/orig manifest text.
peter [Thu, 12 Jun 2014 19:11:53 +0000 (19:11 +0000)]
2883: Added filtering for crunchstat, slurm log lines.
Tom Clegg [Thu, 12 Jun 2014 19:00:13 +0000 (15:00 -0400)]
2755: Preserve blob signatures in manifest_text passed to collections.create.
Peter Amstutz [Thu, 12 Jun 2014 18:49:22 +0000 (14:49 -0400)]
Expanded search path for cgroup stats, changed command line interface a bit.
Also adjusted polling interval to every 10 seconds istead of every 1 second.
refs #2882
Peter Amstutz [Thu, 12 Jun 2014 18:18:48 +0000 (14:18 -0400)]
Bugfix in logging which cgroup stats files it uses. refs #2882
Peter Amstutz [Thu, 12 Jun 2014 18:16:19 +0000 (14:16 -0400)]
Bugfix searching for cgroup stats files. refs #2882
Peter Amstutz [Thu, 12 Jun 2014 18:04:08 +0000 (14:04 -0400)]
Crunchstat logs where it reads its stats from. refs #2882
Peter Amstutz [Thu, 12 Jun 2014 17:58:35 +0000 (13:58 -0400)]
Runs crunch script using "stdbuf -o0 -e0" so stdout and stderr are unbuffered,
solves problem of stdout showing up out of order and with incorrect time stamps
in logs. no issue #
Tom Clegg [Thu, 12 Jun 2014 17:36:10 +0000 (13:36 -0400)]
2755: Fix unfinished comment.
Peter Amstutz [Thu, 12 Jun 2014 15:14:03 +0000 (11:14 -0400)]
Made a note about user how permissions are applied when attempting to create a
token for another user. Test for this case already exists. closes #2350.
Tim Pierce [Tue, 10 Jun 2014 14:37:35 +0000 (10:37 -0400)]
2934: add limits for crunch job log generation.
* crunch_limit_log_events_per_job
* crunch_limit_log_event_bytes_per_job
Limit the number of events a job may log, and the total number of
bytes a job may log.
* crunch_log_bytes_per_event
* crunch_log_seconds_between_events
Set minimum logging thresholds: event logs will only be flushed to the
database when at least crunch_log_bytes_per_event have been generated,
or crunch_log_seconds_between_events have passed.
Integration test CrunchDispatchTest exercised creating a job.
Refs #2934.
Peter Amstutz [Thu, 12 Jun 2014 14:23:42 +0000 (10:23 -0400)]
Merge branch 'origin-2882-job-process-stats' refs #2882
Peter Amstutz [Thu, 12 Jun 2014 14:19:18 +0000 (10:19 -0400)]
2882: Search subdirectories of cgroup for stats, because compute0 has a
slightly different cgroup layout from what's on my workstation (wtf?).
Peter Amstutz [Thu, 12 Jun 2014 13:56:52 +0000 (09:56 -0400)]
2883: Removed commented out code related to live loading (saved it to another branch).
peter [Thu, 12 Jun 2014 02:22:44 +0000 (02:22 +0000)]
2883: Adjusted task type column width. Commented out code for working with
live-updating logs, it almost works but there are some complications.
Tom Clegg [Thu, 12 Jun 2014 02:18:24 +0000 (22:18 -0400)]
2755: Add api server config to enable mandatory Keep signatures.
peter [Thu, 12 Jun 2014 01:59:12 +0000 (01:59 +0000)]
2883: Adjust timestamp display based on timezone and locale.
Peter Amstutz [Wed, 11 Jun 2014 21:12:15 +0000 (21:12 +0000)]
2883: Live updating log wip
Peter Amstutz [Wed, 11 Jun 2014 20:13:22 +0000 (20:13 +0000)]
2883: Styling column widths so they don't jump around when filters are turned
on and off.
Peter Amstutz [Wed, 11 Jun 2014 20:02:12 +0000 (20:02 +0000)]
2883: Moved hardcoded styles out into a stylesheet.
Peter Amstutz [Wed, 11 Jun 2014 19:38:44 +0000 (19:38 +0000)]
2883: Log viewer works, supports sorting and filtering.
radhika [Wed, 11 Jun 2014 16:30:24 +0000 (12:30 -0400)]
refs #2871
Merge branch '2871-preload-objects'
radhika [Wed, 11 Jun 2014 16:22:32 +0000 (12:22 -0400)]
2871: fix the typo caught by integration test
radhika [Wed, 11 Jun 2014 15:02:21 +0000 (11:02 -0400)]
closes #2871
Merge branch '2871-preload-objects'
Peter Amstutz [Wed, 11 Jun 2014 02:09:20 +0000 (22:09 -0400)]
2883: Initial work on fetching and parsing job log
Ward Vandewege [Tue, 10 Jun 2014 20:31:59 +0000 (16:31 -0400)]
Bugfix: the discovery doc should not clobber defaults values with the
word 'string'. Instead, make sure the default values are always exposed
as a string, which is something the java google apiclient is sensitive
to.
Refs no issue #
Peter Amstutz [Tue, 10 Jun 2014 20:04:56 +0000 (16:04 -0400)]
Added error-handler for x-editable that extracts Arvados API error field for
display, closes #2819
radhika [Tue, 10 Jun 2014 20:02:02 +0000 (16:02 -0400)]
2871: add andand to prevent error
radhika [Tue, 10 Jun 2014 19:56:07 +0000 (15:56 -0400)]
Merge branch 'master' into 2871-preload-objects
Peter Amstutz [Tue, 10 Jun 2014 19:46:42 +0000 (15:46 -0400)]
Merge branch '2961-load-tab-partials' closes #2961
Ward Vandewege [Tue, 10 Jun 2014 18:59:21 +0000 (14:59 -0400)]
arv-run-pipeline-instance should not keep trying to create jobs when the
API server returns an error on job creation.
This closes #3005
Peter Amstutz [Tue, 10 Jun 2014 18:56:01 +0000 (14:56 -0400)]
Merge branch '2826-simple-go-sdk' closes #2826
Ward Vandewege [Tue, 10 Jun 2014 18:32:09 +0000 (14:32 -0400)]
Fix inconsistent JSON error return behavior on job create.
refs #3004
Brett Smith [Tue, 10 Jun 2014 18:12:31 +0000 (14:12 -0400)]
Merge branch '2880-crunch-dispatch-node-constraints'
Closes #2880, #2976, #2993.
Brett Smith [Fri, 6 Jun 2014 17:18:38 +0000 (13:18 -0400)]
2880: Don't dispatch Jobs until runtime constraints are met.
This retains the same FIFO approach to the Job queue that
crunch-dispatch currently uses, but now when it encounters a Job whose
constraints are not met:
* it may wait for a while to see if the Node Manager makes Nodes
available, if it hasn't done that this hour; and
* it leaves that Job in the queue and tries to process the next one.
See #2880 for further background. The exact parameters of "waiting
for Nodes" will probably need tuning, but that will be easier to do
after it's been in production for a while.
Brett Smith [Fri, 6 Jun 2014 16:03:54 +0000 (12:03 -0400)]
2880: Improve error reporting when crunch-dispatch updates node status.
Brett Smith [Fri, 6 Jun 2014 16:03:08 +0000 (12:03 -0400)]
2880: crunch-dispatch must stop putting symbols in the database.
Peter Amstutz [Tue, 10 Jun 2014 17:31:46 +0000 (13:31 -0400)]
2961: Removed calls to wait_for_ajax, fixes user tests.
Peter Amstutz [Tue, 10 Jun 2014 17:28:13 +0000 (13:28 -0400)]
2961: user-setup-modal-window div is no longer in footer_html because it is
loaded using AJAX.
Peter Amstutz [Tue, 10 Jun 2014 17:26:20 +0000 (13:26 -0400)]
2961: Don't call #render_index from #index on folders_controller because #index
is also used by #choose and the index does not have any tabs.
Peter Amstutz [Tue, 10 Jun 2014 17:23:24 +0000 (13:23 -0400)]
2961: Do not include meta tag for websockets if there is no websockets in the
discovery document.
Peter Amstutz [Tue, 10 Jun 2014 17:22:48 +0000 (13:22 -0400)]
2961: Fixed name of collection on collection show page to fix test.
Peter Amstutz [Tue, 10 Jun 2014 14:20:29 +0000 (10:20 -0400)]
2961: Track valid/up to date state for each panel, only reload panels when they
are out of date (based on event log) and visible to user.
Peter Amstutz [Tue, 10 Jun 2014 14:18:22 +0000 (10:18 -0400)]
2961: Added use count so ajax loading icon is accurate when there are multiple
outstanding ajax requests.
Peter Amstutz [Tue, 10 Jun 2014 14:16:04 +0000 (10:16 -0400)]
2961: Removed dangling close div.
Peter Amstutz [Tue, 10 Jun 2014 14:15:34 +0000 (10:15 -0400)]
2961: Added missing user uuid to session.
Peter Amstutz [Tue, 10 Jun 2014 13:21:02 +0000 (09:21 -0400)]
Merge branch 'master' into 2961-load-tab-partials
Peter Amstutz [Tue, 10 Jun 2014 13:19:39 +0000 (09:19 -0400)]
Fix for running workbench against server that doesn't have 'Job.supplied_script_version' yet. refs #2939
Peter Amstutz [Tue, 10 Jun 2014 13:18:02 +0000 (09:18 -0400)]
2961: Job index needs to call super sometimes, render_index other times.
Peter Amstutz [Tue, 10 Jun 2014 13:08:38 +0000 (09:08 -0400)]
2961: Changed index methods on some controllers to use render_index instead of
super, to skip the default index query logic but still apply the tab partial
logic.
radhika [Tue, 10 Jun 2014 12:58:22 +0000 (08:58 -0400)]
Merge branch 'master' into 2871-preload-objects
radhika [Tue, 10 Jun 2014 12:49:39 +0000 (08:49 -0400)]
2871: trailing whitespace
Peter Amstutz [Mon, 9 Jun 2014 21:05:43 +0000 (17:05 -0400)]
Specify server-side keep-alive ping interval for websockets. no issue #
Peter Amstutz [Mon, 9 Jun 2014 20:57:50 +0000 (16:57 -0400)]
2961: Escaped erb used embedded in javascript. Updated comments in
event_log.js. Removed debug put from ApplicationController.
radhika [Mon, 9 Jun 2014 20:40:39 +0000 (16:40 -0400)]
Merge branch 'master' into 2871-preload-objects
radhika [Mon, 9 Jun 2014 20:39:31 +0000 (16:39 -0400)]
2871: test preload_objects_for_dataclass helper method
Ward Vandewege [Mon, 9 Jun 2014 20:33:01 +0000 (16:33 -0400)]
Add puma to the Gemfile for the API server; we use it for the websockets
server.
no issue #
radhika [Mon, 9 Jun 2014 19:37:39 +0000 (15:37 -0400)]
2871: preload_log_collections tests
radhika [Mon, 9 Jun 2014 18:00:53 +0000 (14:00 -0400)]
2871: more testing
radhika [Mon, 9 Jun 2014 17:24:09 +0000 (13:24 -0400)]
2871: more tests
radhika [Mon, 9 Jun 2014 16:09:56 +0000 (12:09 -0400)]
2871: test invoking methods a second time to retrieve preloaded data
radhika [Mon, 9 Jun 2014 15:43:40 +0000 (11:43 -0400)]
2871: add tests for get_n_objects_of_class helper method
Peter Amstutz [Mon, 9 Jun 2014 15:41:19 +0000 (11:41 -0400)]
2961: Collections show controller now delegates to super to handle tabs
properly.
radhika [Mon, 9 Jun 2014 14:13:16 +0000 (10:13 -0400)]
2871: add tests for preload_collections_for_objects helper method
radhika [Mon, 9 Jun 2014 13:42:04 +0000 (09:42 -0400)]
2871: add input checks for helper methods. start adding functional tests for the helper methods.
Peter Amstutz [Mon, 9 Jun 2014 13:09:20 +0000 (09:09 -0400)]
Merge remote-tracking branch 'origin/master' into 2961-load-tab-partials
Peter Amstutz [Mon, 9 Jun 2014 13:09:09 +0000 (09:09 -0400)]
2961: Save user information as part of the session cookie, eliminates looking
up current user when rendering every page and panel, this means 2x reduction in
latency when the panel only requires 1 API lookup otherwise.
radhika [Sat, 7 Jun 2014 13:03:19 +0000 (09:03 -0400)]
Merge branch 'master' into 2871-preload-objects
Tim Pierce [Fri, 6 Jun 2014 21:37:18 +0000 (17:37 -0400)]
Merge branch '2857-collection-tests'
Closes #2857.
radhika [Fri, 6 Jun 2014 19:57:19 +0000 (15:57 -0400)]
Merge branch 'master' into 2871-preload-objects
Tim Pierce [Fri, 6 Jun 2014 19:48:43 +0000 (15:48 -0400)]
2857: strip signatures from manifests before testing equality.
As Brett observed, the "post" method in these tests mutates its
argument. For a valid test, we must ensure that the source data has not
been transformed into the result data.
For tests which do not care about validating the signatures, we just
strip out the signatures from the result manifest before comparing it to
the source.
Refs #2857.
Peter Amstutz [Fri, 6 Jun 2014 17:45:18 +0000 (13:45 -0400)]
2961: Added checks if we are rendering a tab partial and skips expensive calls
to the API server if not needed to render the requested tab pane.
Tim Pierce [Fri, 6 Jun 2014 15:31:59 +0000 (11:31 -0400)]
Merge branch 'master' into 2857-collection-tests
Peter Amstutz [Fri, 6 Jun 2014 14:40:59 +0000 (10:40 -0400)]
Merge branch '2955-fail-orphan-jobs' closes #2955
Peter Amstutz [Fri, 6 Jun 2014 14:39:39 +0000 (10:39 -0400)]
Reverted change
a0aba6d "crunch-dispatch now sends a clean environment to crunch-job"
Peter Amstutz [Fri, 6 Jun 2014 14:31:18 +0000 (10:31 -0400)]
Merge branch 'master' into 2955-fail-orphan-jobs
Peter Amstutz [Fri, 6 Jun 2014 14:30:06 +0000 (10:30 -0400)]
Reverted change
a0aba6d "crunch-dispatch now sends a clean environment to crunch-job" refs #2882.
Brett Smith [Fri, 6 Jun 2014 13:47:11 +0000 (09:47 -0400)]
Merge branch '2880-compute-ping-stats'
Closes #2967, #2975.
Brett Smith [Thu, 5 Jun 2014 15:38:46 +0000 (11:38 -0400)]
2880: API server saves node statistics from pings.
Crunch will use this information when scheduling Jobs, to satisfy
specified runtime constraints. Getting this data into the pings is
the responsibility of the node's ping script.
Brett Smith [Fri, 6 Jun 2014 13:38:25 +0000 (09:38 -0400)]
2882: Workbench generic create controller adds JSON href attribute.
This is a bugfix commit. This detail got lost when create stopped
calling show to render the object. Refs #2882.
Peter Amstutz [Fri, 6 Jun 2014 02:02:30 +0000 (22:02 -0400)]
2961: Now loads tab panes (past the first one) in the background.
Automatically triggers reload of tab panes when receiving a changed event from
websocket.
Peter Amstutz [Thu, 5 Jun 2014 20:31:18 +0000 (16:31 -0400)]
2961: Removed spurious elementId from subscribeToEventLog(). Added arv-log-event-listener[data-object-type="xxxxx"] to
match on uuid types.
Peter Amstutz [Thu, 5 Jun 2014 19:50:26 +0000 (15:50 -0400)]
Fixes PipelineInstancesControllerTest. application_controller#create now does
the right thing based on requested response format. refs #2882
Tim Pierce [Thu, 5 Jun 2014 19:42:06 +0000 (15:42 -0400)]
2857: fix flaky time-sensitive tests
Allow tests to inject a mock Time object into Blob, in order to generate
consistent behavior for tests that rely on permission signatures.
Closes #2857.
Peter Amstutz [Thu, 5 Jun 2014 18:49:35 +0000 (14:49 -0400)]
Merge branch '2882-job-process-stats' refs #2882
Peter Amstutz [Thu, 5 Jun 2014 18:44:51 +0000 (14:44 -0400)]
2955: Renamed script to "cancel_stale_jobs". Fixed spelling canceled_at -> cancelled_at.