Tom Clegg [Fri, 25 Nov 2016 23:10:16 +0000 (18:10 -0500)]
10586: Fix deadlock in KeepWriterThreadPool.
Previously, an unhandled exception in KeepWriterThread (including
KeepService.put()) would terminate the thread without calling
queue.task_done(), which would cause queue.join() to deadlock.
Additionally, this commit moves all of the "wait until work is needed"
logic from KeepWriterThread into KeepWriterQueue.
Tom Clegg [Thu, 24 Nov 2016 19:18:59 +0000 (14:18 -0500)]
8460: More statistics in status.json: events in/out, reqs received, lowest client queue delay.
Peter Amstutz [Wed, 23 Nov 2016 23:56:55 +0000 (18:56 -0500)]
Update arvados-cwl-runner with compatible dependencies, refs #10583
radhika [Wed, 23 Nov 2016 19:54:37 +0000 (14:54 -0500)]
refs #10290
Merge branch '10290-scheduling-parameters-doc'
radhika [Wed, 23 Nov 2016 19:52:38 +0000 (14:52 -0500)]
10290: scheduling_parameters documentation
Tom Clegg [Wed, 23 Nov 2016 19:36:59 +0000 (14:36 -0500)]
Merge branch '10484-count-clients' closes #10484
Peter Amstutz [Wed, 23 Nov 2016 19:18:24 +0000 (14:18 -0500)]
Update cwltool version with compatible dependencies, refs #10583
Tom Clegg [Wed, 23 Nov 2016 07:49:54 +0000 (02:49 -0500)]
10535: Fix race-to-deadlock bugs in tests.test_events: use threading.Event, not threading.Condition. refs #10535
radhika [Tue, 22 Nov 2016 22:42:28 +0000 (17:42 -0500)]
refs #10524
Merge branch '10524-crunch2-workbench'
radhika [Tue, 22 Nov 2016 22:41:33 +0000 (17:41 -0500)]
10524: update test assertions
radhika [Tue, 22 Nov 2016 21:31:28 +0000 (16:31 -0500)]
Merge branch 'master' into 10293-cwl-cr-output
radhika [Tue, 22 Nov 2016 21:30:45 +0000 (16:30 -0500)]
10293: done_outputs in a try/except block
Peter Amstutz [Tue, 22 Nov 2016 21:26:17 +0000 (16:26 -0500)]
Merge branch '10583-cwl-salad-ver' refs #10583
Peter Amstutz [Tue, 22 Nov 2016 20:25:21 +0000 (15:25 -0500)]
10583: Bump versions.
radhika [Tue, 22 Nov 2016 19:06:59 +0000 (14:06 -0500)]
10293: invoke output_callback in a try/except block
radhika [Tue, 22 Nov 2016 18:38:27 +0000 (13:38 -0500)]
Merge branch 'master' into 10293-cwl-cr-output
radhika [Tue, 22 Nov 2016 18:35:54 +0000 (13:35 -0500)]
closes #10524
Merge branch '10524-crunch2-workbench'
Tom Clegg [Tue, 22 Nov 2016 17:14:47 +0000 (12:14 -0500)]
Merge branch '9897-log-block-prefetch-worker-exceptions' closes #9897
Tom Clegg [Tue, 22 Nov 2016 17:02:29 +0000 (12:02 -0500)]
Merge branch '10473-keepstore-logrus' closes #10473
Lucas Di Pentima [Tue, 22 Nov 2016 16:40:45 +0000 (13:40 -0300)]
5737: Merge branch 'master' into 5737-ruby231
Peter Amstutz [Tue, 22 Nov 2016 16:40:36 +0000 (11:40 -0500)]
Merge branch '10498-arvdocker-project-uuid' refs #10498
Peter Amstutz [Tue, 22 Nov 2016 15:33:01 +0000 (10:33 -0500)]
10498: Handle project_uuid of None in arv_docker_get_image.
Lucas Di Pentima [Tue, 22 Nov 2016 13:37:13 +0000 (10:37 -0300)]
10383: Fixed integration tests that got broken when the resume feature was implemented.
radhika [Tue, 22 Nov 2016 13:20:02 +0000 (08:20 -0500)]
10524: add additional test assertions to disabled_api_test
Colin Nolan [Tue, 22 Nov 2016 10:18:18 +0000 (10:18 +0000)]
9897: Changes as per discussion in #9897.
Peter Amstutz [Tue, 22 Nov 2016 02:08:11 +0000 (21:08 -0500)]
Arvbox sets PATH, GEM_HOME and GEM_PATH in common.sh. refs #10410
Lucas Di Pentima [Mon, 21 Nov 2016 22:28:14 +0000 (19:28 -0300)]
10383: Merge branch 'master' into 10383-arv-put-incremental-upload
Lucas Di Pentima [Mon, 21 Nov 2016 22:23:20 +0000 (19:23 -0300)]
10383: Several fixes/enhancements to arv-put:
* Added --no-cache to avoid creating/loading a cache file.
* Removed unnecessary code when writing empty ArvadosFiles.
* Use absolute path on cache index so that is not relative to the CWD and can be used from anywhere on the system.
Lucas Di Pentima [Mon, 21 Nov 2016 22:22:31 +0000 (19:22 -0300)]
10383: Fixed an error when using Collection.copy() to overwrite an existing ArvadosFile.
Tom Clegg [Mon, 21 Nov 2016 19:22:59 +0000 (14:22 -0500)]
10484: Report current client count.
Tom Clegg [Mon, 21 Nov 2016 18:39:50 +0000 (13:39 -0500)]
Merge branch '10473-keepstore-logrus' closes #10473
Tom Clegg [Sun, 20 Nov 2016 06:44:35 +0000 (01:44 -0500)]
10473: Tune up text-format logging.
Tom Clegg [Sun, 20 Nov 2016 06:39:05 +0000 (01:39 -0500)]
10473: Use fixed-point timestamps and more human-readable field ordering, hide "incoming request" if level<debug.
Tom Clegg [Sun, 20 Nov 2016 05:30:16 +0000 (00:30 -0500)]
10473: Fixed-width timestamps.
Tom Clegg [Fri, 18 Nov 2016 06:49:47 +0000 (01:49 -0500)]
10473: Log requests/responses in JSON format by default; add config option for text format.
radhika [Mon, 21 Nov 2016 18:22:24 +0000 (13:22 -0500)]
10524: update tests to look for 'Run a process' button instead of 'Run a pipeline'
Tom Clegg [Mon, 21 Nov 2016 18:07:30 +0000 (13:07 -0500)]
Merge branch '10484-keepstore-stats' refs #10484
radhika [Mon, 21 Nov 2016 18:02:07 +0000 (13:02 -0500)]
10524: update workbench titles to use "processes" instead of "pipelines and processes" when pipelines are disabled.
Tom Clegg [Thu, 17 Nov 2016 22:24:05 +0000 (17:24 -0500)]
10484: Remove unused volumeStats.
Tom Clegg [Wed, 16 Nov 2016 23:10:32 +0000 (18:10 -0500)]
10484: Test s3 bucket stats.
Tom Clegg [Wed, 16 Nov 2016 22:51:24 +0000 (17:51 -0500)]
10484: Tidy up stats-tracking code into a bucket proxy type.
Tom Clegg [Thu, 10 Nov 2016 20:25:32 +0000 (15:25 -0500)]
10484: Track non-s3 errors by Go type.
Tom Clegg [Thu, 10 Nov 2016 20:21:26 +0000 (15:21 -0500)]
10484: Track s3 errors by response code.
Tom Clegg [Tue, 8 Nov 2016 23:15:20 +0000 (18:15 -0500)]
10484: Report volume IO stats for S3 volumes.
Tom Clegg [Tue, 8 Nov 2016 16:04:08 +0000 (11:04 -0500)]
10484: Serve MemStats at /debug.json instead of /status.json.
Tom Clegg [Mon, 21 Nov 2016 16:06:39 +0000 (11:06 -0500)]
Merge branch '10499-cwl-crunch2-docs' refs #10499
Tom Clegg [Sun, 20 Nov 2016 05:48:46 +0000 (00:48 -0500)]
8460: Move logging setup to log.go, use fixed-width timestamps.
Tom Clegg [Sat, 19 Nov 2016 08:10:24 +0000 (03:10 -0500)]
8460: Report status of outgoing queues.
Tom Clegg [Sat, 19 Nov 2016 07:15:57 +0000 (02:15 -0500)]
8460: Add /status.json
Tom Clegg [Sat, 19 Nov 2016 06:35:01 +0000 (01:35 -0500)]
8460: Inject permChecker from main.
Tom Clegg [Fri, 18 Nov 2016 23:07:15 +0000 (18:07 -0500)]
8460: Fix "send to closed channel" race by using context lib to release goroutines.
Tom Clegg [Fri, 18 Nov 2016 23:00:59 +0000 (18:00 -0500)]
8460: Accept 403 response from API as "not permitted", instead of "unexpected error".
Tom Clegg [Fri, 18 Nov 2016 20:14:47 +0000 (15:14 -0500)]
Merge branch '10470-slurm-error'
refs #10470
Tom Clegg [Fri, 18 Nov 2016 19:04:53 +0000 (14:04 -0500)]
10470: Recognize more slurm error messages.
Example from slurm 14.03.9:
srun: error: _server_read: fd 12 got error or unexpected eof reading header
srun: error: step_launch_notify_io_failure: aborting, io error with slurmstepd on node 0
srun: Job step aborted: Waiting up to 2 seconds for job step to finish.
Javier Bértoli [Fri, 18 Nov 2016 17:20:16 +0000 (14:20 -0300)]
Merge branch '10556-get_anonymous_user_token.rb-multiple-similar-records'
refs #10556
Tom Clegg [Thu, 17 Nov 2016 22:18:38 +0000 (17:18 -0500)]
Merge branch '10498-register-with-chosen-api' closes #10498
Lucas Di Pentima [Thu, 17 Nov 2016 21:59:40 +0000 (18:59 -0300)]
10383: Unused exception removed.
Lucas Di Pentima [Thu, 17 Nov 2016 21:53:54 +0000 (18:53 -0300)]
10383: Tidying up internal collection management and naming.
Tom Clegg [Thu, 17 Nov 2016 21:49:12 +0000 (16:49 -0500)]
10498: Check error message in "incompatible APIs" test.
Tom Clegg [Thu, 17 Nov 2016 21:48:38 +0000 (16:48 -0500)]
10498: Tweak var names and help messages.
Tom Clegg [Thu, 17 Nov 2016 21:33:13 +0000 (16:33 -0500)]
8460: Fix up logging.
Javier Bértoli [Thu, 17 Nov 2016 21:29:12 +0000 (18:29 -0300)]
Update documentation about get_anonymous_user_token.rb to prevent duplicates
refs #10556
Lucas Di Pentima [Thu, 17 Nov 2016 21:13:17 +0000 (18:13 -0300)]
10383: Now --update-collection is not mutually exclusive with --resume and --no-resume.
Also moved update collection setup code into the setup method.
Lucas Di Pentima [Thu, 17 Nov 2016 20:54:15 +0000 (17:54 -0300)]
10383: Removing unnecessary locking, as Collection class is thread safe.
Tom Clegg [Thu, 17 Nov 2016 20:23:01 +0000 (15:23 -0500)]
8460: Reply to unparsable messages with status:400.
Tom Clegg [Thu, 17 Nov 2016 20:10:56 +0000 (15:10 -0500)]
8460: Stop existing nginx server, if any, before starting new.
Tom Clegg [Thu, 17 Nov 2016 19:49:46 +0000 (14:49 -0500)]
8460: Refactor "old events / other messages" mechanism to use the outgoing message queue.
radhika [Thu, 17 Nov 2016 19:21:16 +0000 (14:21 -0500)]
Merge branch 'master' into 10293-cwl-cr-output
Conflicts:
sdk/cwl/arvados_cwl/__init__.py
sdk/cwl/arvados_cwl/arvcontainer.py
Tom Clegg [Thu, 17 Nov 2016 19:10:15 +0000 (14:10 -0500)]
8460: Structured logging.
radhika [Thu, 17 Nov 2016 18:11:08 +0000 (13:11 -0500)]
10293: update documentation with scheduling_parameters, log_uuid and output_uuid.
Lucas Di Pentima [Thu, 17 Nov 2016 17:52:59 +0000 (14:52 -0300)]
10383: Re-done of the resume and update logic:
* Eliminated the restriction to update the same collection once uploaded.
* Greatly simplified and corrected the file resuming decision making.
* Deterministic path traversing and uploading added.
* Remote collection updating is done using efficient Collection.copy() method from cached manifest.
Peter Amstutz [Thu, 17 Nov 2016 16:39:26 +0000 (11:39 -0500)]
Merge branch '10529-cwl-exception-stuck' closes #10529
Tom Clegg [Wed, 16 Nov 2016 23:15:56 +0000 (18:15 -0500)]
8460: Merge branch 'master' into 8460-websocket-go
radhika [Wed, 16 Nov 2016 22:04:23 +0000 (17:04 -0500)]
Merge branch 'master' into 10293-cwl-cr-output
radhika [Wed, 16 Nov 2016 22:03:35 +0000 (17:03 -0500)]
refs #10293
Merge branch '10293-container-request-output-uuid'
radhika [Wed, 16 Nov 2016 22:00:18 +0000 (17:00 -0500)]
Merge branch 'master' into 10293-container-request-output-uuid
radhika [Wed, 16 Nov 2016 21:58:12 +0000 (16:58 -0500)]
10293: add scheduling_parameters to containers controller
Peter Amstutz [Wed, 16 Nov 2016 21:35:08 +0000 (16:35 -0500)]
Permit use_existing in create-and-commit container requests. refs #10216
Tom Clegg [Wed, 16 Nov 2016 21:23:35 +0000 (16:23 -0500)]
10498: --create-workflow and --create-template do the same thing, depending on --api choice.
Also:
* --update-workflow works for pipeline templates, too.
* --update-workflow establishes --api=jobs or --api=containers
depending on the UUID. (If this conflicts with an explicit --api
argument, that is an error.)
* If no --project-uuid is given, the user's UUID is not passed
explicitly with create/update requests. The API will default to the
user's UUID or (when updating an existing object) leaving the
existing object's owner unchanged.
radhika [Wed, 16 Nov 2016 20:58:45 +0000 (15:58 -0500)]
Merge branch '10293-container-request-output-uuid' into 10293-cwl-cr-output
radhika [Wed, 16 Nov 2016 20:58:35 +0000 (15:58 -0500)]
Merge branch 'master' into 10293-cwl-cr-output
radhika [Wed, 16 Nov 2016 20:58:10 +0000 (15:58 -0500)]
Merge branch 'master' into 10293-container-request-output-uuid
radhika [Wed, 16 Nov 2016 20:57:19 +0000 (15:57 -0500)]
closes #10290
Merge branch '10290-container-partitions'
radhika [Wed, 16 Nov 2016 20:33:47 +0000 (15:33 -0500)]
Merge branch 'master' into 10293-cwl-cr-output
radhika [Wed, 16 Nov 2016 20:25:01 +0000 (15:25 -0500)]
10293: update cwl runner
Lucas Di Pentima [Wed, 16 Nov 2016 20:08:58 +0000 (17:08 -0300)]
10383: Using os.walk() to traverse input directories in an deterministic way.
Peter Amstutz [Wed, 16 Nov 2016 20:06:02 +0000 (15:06 -0500)]
10529: Improve handling errors raised when collecting outputs. If state
polling thread crashes, terminate executor.
Tom Clegg [Wed, 16 Nov 2016 19:09:15 +0000 (14:09 -0500)]
8460: Return recent events if last_log_id given.
Tom Clegg [Wed, 16 Nov 2016 19:08:27 +0000 (14:08 -0500)]
8460: Pass tests even if websocket server sends events that do not match our filters.
radhika [Wed, 16 Nov 2016 16:40:55 +0000 (11:40 -0500)]
10290: scheduling_parameters added to container_requests controller.
radhika [Wed, 16 Nov 2016 16:24:00 +0000 (11:24 -0500)]
Merge branch 'master' into 10290-container-partitions
radhika [Wed, 16 Nov 2016 16:23:40 +0000 (11:23 -0500)]
Merge branch '10290-container-partitions' of git.curoverse.com:arvados into 10290-container-partitions
Tom Clegg [Wed, 16 Nov 2016 06:45:41 +0000 (01:45 -0500)]
8460: Retrieve recent logs and send old matching events if last_log_id given.
Tom Clegg [Wed, 16 Nov 2016 06:11:36 +0000 (01:11 -0500)]
8460: Allow session Receive handler to queue multiple messages.
Tom Clegg [Wed, 16 Nov 2016 05:56:46 +0000 (00:56 -0500)]
8460: Support created_at filters.
Tom Clegg [Wed, 16 Nov 2016 04:26:57 +0000 (23:26 -0500)]
8460: Send {"status":200} messages. Bring up ws server for Python SDK tests.
Tom Clegg [Wed, 16 Nov 2016 04:00:58 +0000 (23:00 -0500)]
8460: Log connection stats.
Tom Clegg [Wed, 16 Nov 2016 03:44:33 +0000 (22:44 -0500)]
8460: Cache negative permission.
Tom Clegg [Tue, 15 Nov 2016 21:30:01 +0000 (16:30 -0500)]
8460: Send selected old/new attributes with v0 events.
Peter Amstutz [Tue, 15 Nov 2016 21:10:21 +0000 (16:10 -0500)]
Merge branch '10410-arvbox-ruby' refs #10410