arvados.git
11 years agoMerge branch '1587-fix-test-suites-in-devsandbox'
Tim Pierce [Mon, 9 Dec 2013 23:36:23 +0000 (18:36 -0500)]
Merge branch '1587-fix-test-suites-in-devsandbox'

11 years agoUse body={...} instead of object_type=json.dumps({...})
Tom Clegg [Mon, 9 Dec 2013 19:15:39 +0000 (11:15 -0800)]
Use body={...} instead of object_type=json.dumps({...})

11 years agoAdd example scripts and libraries for bwa, samtools, picard, gatk2
Tom Clegg [Mon, 9 Dec 2013 18:25:48 +0000 (10:25 -0800)]
Add example scripts and libraries for bwa, samtools, picard, gatk2

closes #1598

11 years agoAdding 'run.sh test api' to run API server Rails tests.
Tim Pierce [Fri, 6 Dec 2013 22:49:40 +0000 (17:49 -0500)]
Adding 'run.sh test api' to run API server Rails tests.

11 years agoIgnore config/environments/test.rb and production.rb
Tom Clegg [Fri, 6 Dec 2013 20:45:26 +0000 (12:45 -0800)]
Ignore config/environments/test.rb and production.rb

11 years agoIgnore tmp directory created by cli tests
Tom Clegg [Fri, 6 Dec 2013 20:36:05 +0000 (12:36 -0800)]
Ignore tmp directory created by cli tests

11 years agoAdded 'test' command to run.sh.
Tim Pierce [Fri, 6 Dec 2013 19:46:48 +0000 (14:46 -0500)]
Added 'test' command to run.sh.

11 years agorun.sh: bug fixes
Tim Pierce [Thu, 5 Dec 2013 22:40:02 +0000 (17:40 -0500)]
run.sh: bug fixes

11 years agorun.sh: add "stop" command, also remove old containers before trying to start them
Tim Pierce [Thu, 5 Dec 2013 21:40:02 +0000 (16:40 -0500)]
run.sh: add "stop" command, also remove old containers before trying to start them

11 years agoRemove spurious "if".
Tom Clegg [Thu, 5 Dec 2013 21:27:51 +0000 (13:27 -0800)]
Remove spurious "if".

11 years agoUse some helpful HTTP status codes, not just generic 422.
Tom Clegg [Thu, 5 Dec 2013 21:08:28 +0000 (13:08 -0800)]
Use some helpful HTTP status codes, not just generic 422.

* 403 for PermissionDeniedError
* 401 for UnauthorizedError

11 years agoCheck permission before deleting objects
Tom Clegg [Thu, 5 Dec 2013 21:12:50 +0000 (13:12 -0800)]
Check permission before deleting objects

11 years agoReorganized code to support 'start' command.
Tim Pierce [Thu, 5 Dec 2013 19:26:39 +0000 (14:26 -0500)]
Reorganized code to support 'start' command.

11 years agorun.sh prints instructions/settings for running a test suite.
Tim Pierce [Wed, 4 Dec 2013 22:43:48 +0000 (17:43 -0500)]
run.sh prints instructions/settings for running a test suite.

11 years agoAdd support for ARVADOS_API_PORT.
Tim Pierce [Wed, 4 Dec 2013 20:50:58 +0000 (15:50 -0500)]
Add support for ARVADOS_API_PORT.

11 years agoSuperuser script must be run as ./script/create_superuser_token.rb.
Tim Pierce [Wed, 4 Dec 2013 20:10:09 +0000 (15:10 -0500)]
Superuser script must be run as ./script/create_superuser_token.rb.

11 years agoAPI_SUPERUSER_TOKEN must be named API_SUPERUSER_SECRET to be autogenerated.
Tim Pierce [Wed, 4 Dec 2013 19:35:00 +0000 (14:35 -0500)]
API_SUPERUSER_TOKEN must be named API_SUPERUSER_SECRET to be autogenerated.

11 years agorun.sh bug fixes.
Tim Pierce [Wed, 4 Dec 2013 18:38:20 +0000 (13:38 -0500)]
run.sh bug fixes.

11 years agoMount separate volumes for each keep container launched.
Tim Pierce [Wed, 4 Dec 2013 16:51:12 +0000 (11:51 -0500)]
Mount separate volumes for each keep container launched.

11 years agoAllow overriding of default ports for API, doc, SSO, and workbench
Ward Vandewege [Wed, 4 Dec 2013 02:34:05 +0000 (21:34 -0500)]
Allow overriding of default ports for API, doc, SSO, and workbench
servers when started with run.sh

11 years agoFix failing test by parsing timestamp correctly.
Tom Clegg [Tue, 3 Dec 2013 23:19:22 +0000 (15:19 -0800)]
Fix failing test by parsing timestamp correctly.

11 years agoFix PermissionDeniedError when a non-admin user is the first to log in
Tom Clegg [Tue, 3 Dec 2013 23:17:30 +0000 (15:17 -0800)]
Fix PermissionDeniedError when a non-admin user is the first to log in
from a given ApiClient.

11 years agoChanges to permit keep_server containers to authenticate to api_server.
Tim Pierce [Tue, 3 Dec 2013 22:55:08 +0000 (17:55 -0500)]
Changes to permit keep_server containers to authenticate to api_server.

* Generate an API "superuser token" and deliver this token to the
  Keep server via warehouse.conf.
* Start the keep_server containers with a link to api_server.
* Add config.yml to the dependency chain for the base image (so it will
  be rebuilt if PUBLIC_KEY_PATH is changed).

11 years agoAdd arvados.getjobparam() convenience function.
Tom Clegg [Fri, 29 Nov 2013 19:20:33 +0000 (11:20 -0800)]
Add arvados.getjobparam() convenience function.

11 years agoAccept empty string as root stream name "." in
Tom Clegg [Fri, 29 Nov 2013 16:36:04 +0000 (08:36 -0800)]
Accept empty string as root stream name "." in
arvados.CollectionWriter, to mirror os.path.split()

11 years agoAccept data from a generator in arvados.CollectionWriter.write()
Tom Clegg [Fri, 29 Nov 2013 16:35:04 +0000 (08:35 -0800)]
Accept data from a generator in arvados.CollectionWriter.write()

11 years agoSort filenames when writing multi-level streams in
Tom Clegg [Wed, 27 Nov 2013 18:41:02 +0000 (10:41 -0800)]
Sort filenames when writing multi-level streams in
arvados.CollectionWriter.

11 years agoAdd arvados.util.clear_tmpdir() function.
Tom Clegg [Tue, 26 Nov 2013 22:53:27 +0000 (14:53 -0800)]
Add arvados.util.clear_tmpdir() function.

11 years agoPass through stderr in real time by default in util.run_command()
Tom Clegg [Tue, 26 Nov 2013 22:08:27 +0000 (14:08 -0800)]
Pass through stderr in real time by default in util.run_command()

11 years agoAllow caller to override close_fds in util.run_command()
Tom Clegg [Tue, 26 Nov 2013 20:34:43 +0000 (12:34 -0800)]
Allow caller to override close_fds in util.run_command()

11 years agoUse collection hash instead of entire manifest as symlink target
Tom Clegg [Tue, 26 Nov 2013 16:00:35 +0000 (08:00 -0800)]
Use collection hash instead of entire manifest as symlink target
in arvados.collection_extract(), even when the manifest was passed as
a function argument.

11 years agoOnly require keep volume when a keep container is being started.
Ward Vandewege [Tue, 3 Dec 2013 21:47:26 +0000 (16:47 -0500)]
Only require keep volume when a keep container is being started.

11 years agoMake the sample "Add authorized key" button non-clickable.
Tom Clegg [Mon, 2 Dec 2013 18:52:52 +0000 (10:52 -0800)]
Make the sample "Add authorized key" button non-clickable.

If it behaves like a real button in the browser, users are inclined to
think it *is* a real button, rather than just an image of what to look
for.

Fixes #1664

11 years agoAdd delete button to generic Workbench index page.
Tom Clegg [Mon, 2 Dec 2013 18:49:28 +0000 (10:49 -0800)]
Add delete button to generic Workbench index page.

Fixes #1665

11 years agoRemove Rails default scaffold CSS.
Tom Clegg [Mon, 2 Dec 2013 15:55:55 +0000 (07:55 -0800)]
Remove Rails default scaffold CSS.

11 years agoShow how crunch job tasks get their runtime parameters. refs #1513
Tom Clegg [Thu, 28 Nov 2013 21:09:27 +0000 (13:09 -0800)]
Show how crunch job tasks get their runtime parameters. refs #1513

11 years agoClear TASK_WORK before starting a task.
Tom Clegg [Thu, 28 Nov 2013 21:06:10 +0000 (13:06 -0800)]
Clear TASK_WORK before starting a task.

11 years agoUse new-style Python classes (subclass object), so super() can work
Tom Clegg [Wed, 27 Nov 2013 21:40:51 +0000 (13:40 -0800)]
Use new-style Python classes (subclass object), so super() can work

11 years agoAdd --api, --sso, --workbench, --doc, --keep options to run.sh.
Tim Pierce [Tue, 26 Nov 2013 22:04:50 +0000 (17:04 -0500)]
Add --api, --sso, --workbench, --doc, --keep options to run.sh.
Default is to run all servers.

11 years agoRefactored and updated run.sh:
Ward Vandewege [Tue, 26 Nov 2013 19:43:30 +0000 (14:43 -0500)]
Refactored and updated run.sh:

* support ENABLE_SSH environment variable
* start 2 keep containers (backed by the same filesystem for now)

11 years agobuild.sh has been superseded by Makefile.
Ward Vandewege [Tue, 26 Nov 2013 19:03:49 +0000 (14:03 -0500)]
build.sh has been superseded by Makefile.

11 years agoRename docserver to doc for consistency.
Ward Vandewege [Tue, 26 Nov 2013 19:03:21 +0000 (14:03 -0500)]
Rename docserver to doc for consistency.

11 years agoAdd an SSH server in each docker container.
Ward Vandewege [Tue, 26 Nov 2013 18:57:48 +0000 (13:57 -0500)]
Add an SSH server in each docker container.

It will not be started by default, but can be enabled by setting an
environment (ENABLE_SSH) when starting a container, and is useful for
debugging.

11 years agoDo not require filesystem_uuid in first keep_disks/ping request.
Tom Clegg [Mon, 25 Nov 2013 16:18:51 +0000 (08:18 -0800)]
Do not require filesystem_uuid in first keep_disks/ping request.

Record it if provided, but do not try to do any sanity-checking with
it.

11 years agoUpdate crunch_refresh_trigger config setting name in tests.
Tom Clegg [Mon, 25 Nov 2013 16:16:45 +0000 (08:16 -0800)]
Update crunch_refresh_trigger config setting name in tests.

11 years agoAdd job.cancel API.
Tom Clegg [Mon, 25 Nov 2013 08:05:45 +0000 (00:05 -0800)]
Add job.cancel API.

Same as job.update(job={"cancelled_at":"2112-12-21 21:12"}) but more
convenient.

Closes #1654

11 years agoFix config setting name.
Tom Clegg [Mon, 25 Nov 2013 07:53:55 +0000 (23:53 -0800)]
Fix config setting name.

11 years agoUpdate comments in config/environment/ files.
Tom Clegg [Mon, 25 Nov 2013 07:47:15 +0000 (23:47 -0800)]
Update comments in config/environment/ files.

11 years agoCheck refresh trigger in crunch-job instead of crunch-dispatch.
Tom Clegg [Mon, 25 Nov 2013 07:36:56 +0000 (23:36 -0800)]
Check refresh trigger in crunch-job instead of crunch-dispatch.

Crunch-job now checks the timestamp on the trigger file, rather than
waiting for crunch-dispatch notice that the trigger file appeared and
send HUP to its child processes. This way we don't depend on srun
doing what we want with signals.

See f99ffd6844ae9b88cfd17c8fe6e9fcc85a55d79c

11 years agoAvoid starting new jobs when TERM signal has been received.
Tom Clegg [Mon, 25 Nov 2013 07:31:16 +0000 (23:31 -0800)]
Avoid starting new jobs when TERM signal has been received.

11 years agoRuby 2.0.0-p247 -> p353 (now installed as stable).
Tim Pierce [Sat, 23 Nov 2013 21:48:30 +0000 (16:48 -0500)]
Ruby 2.0.0-p247 -> p353 (now installed as stable).

11 years agoAvoid infinite poll loop triggered by (:running and !(:finished or :cancelled))
Tom Clegg [Sat, 23 Nov 2013 00:09:28 +0000 (16:09 -0800)]
Avoid infinite poll loop triggered by (:running and !(:finished or :cancelled))

11 years agoFix scalar gmtime() usage.
Tom Clegg [Fri, 22 Nov 2013 23:51:39 +0000 (15:51 -0800)]
Fix scalar gmtime() usage.

11 years agoFix missing "my" scope.
Tom Clegg [Fri, 22 Nov 2013 23:47:45 +0000 (15:47 -0800)]
Fix missing "my" scope.

11 years agoAllow users to cancel a running crunch job by updating cancelled_at
Tom Clegg [Fri, 22 Nov 2013 23:34:31 +0000 (15:34 -0800)]
Allow users to cancel a running crunch job by updating cancelled_at
(to something other than nil).

Also:
* Add test fixtures for jobs
* Take care to update the Job object piecemeal during crunch-job
  (avoid overwriting cancelled_at)

11 years agoFix base class of ApiClient model so as_api_response works
Tom Clegg [Fri, 22 Nov 2013 22:10:03 +0000 (14:10 -0800)]
Fix base class of ApiClient model so as_api_response works

11 years agoFix up keep_disks/ping and add tests. refs #1581
Tom Clegg [Fri, 22 Nov 2013 16:28:37 +0000 (08:28 -0800)]
Fix up keep_disks/ping and add tests. refs #1581

11 years agoOnly apply the auto_admin_user configuration rule when no admin users
Ward Vandewege [Fri, 22 Nov 2013 01:05:27 +0000 (20:05 -0500)]
Only apply the auto_admin_user configuration rule when no admin users
exist yet, but exclude the default root system user from that count.
Just in case.

11 years agoOnly apply the auto_admin_user configuration rule when no admin users
Ward Vandewege [Thu, 21 Nov 2013 01:22:48 +0000 (20:22 -0500)]
Only apply the auto_admin_user configuration rule when no admin users
exist yet. Just in case.

11 years agoadd script/create_superuser_token.rb
Tom Clegg [Wed, 20 Nov 2013 23:14:18 +0000 (15:14 -0800)]
add script/create_superuser_token.rb

11 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Ward Vandewege [Wed, 20 Nov 2013 22:22:36 +0000 (17:22 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

11 years ago* Docker API container: add support for the auto_admin_user feature (API server).
Ward Vandewege [Wed, 20 Nov 2013 22:19:24 +0000 (17:19 -0500)]
* Docker API container: add support for the auto_admin_user feature (API server).

This closes #1655

* Default WORKBENCH_INSECURE_HTTPS in config.yml.example to true,
  because we use self-signed certificates in the development environment.

11 years agoConverted README to real markdown.
Tim Pierce [Wed, 20 Nov 2013 22:14:30 +0000 (17:14 -0500)]
Converted README to real markdown.

11 years agoFix dependencies.
Tim Pierce [Wed, 20 Nov 2013 21:59:10 +0000 (16:59 -0500)]
Fix dependencies.

11 years agoImprove Makefile rules.
Tim Pierce [Wed, 20 Nov 2013 18:32:31 +0000 (13:32 -0500)]
Improve Makefile rules.
Start Keep container from run.sh.

11 years agoAdd a config setting: auto_admin_user
Ward Vandewege [Wed, 20 Nov 2013 22:15:49 +0000 (17:15 -0500)]
Add a config setting: auto_admin_user

This setting is optional, and can be configured to the The e-mail
address of the user you would like to become marked as an admin user on
their first login. This is useful during installation.

In the default configuration, authentication happens through the Arvados
SSO server, which uses openid against Google's servers. In that case
this e-mail address should be an address associated with a Google
account.

11 years agoConverted README to real markdown.
Tim Pierce [Wed, 20 Nov 2013 22:14:30 +0000 (17:14 -0500)]
Converted README to real markdown.

11 years agoFix dependencies.
Tim Pierce [Wed, 20 Nov 2013 21:59:10 +0000 (16:59 -0500)]
Fix dependencies.

11 years agoImprove Makefile rules.
Tim Pierce [Wed, 20 Nov 2013 18:32:31 +0000 (13:32 -0500)]
Improve Makefile rules.
Start Keep container from run.sh.

11 years agoMake workbench work with the API server.
Ward Vandewege [Wed, 20 Nov 2013 15:15:07 +0000 (10:15 -0500)]
Make workbench work with the API server.

11 years agoMake sure the API rails app tmp directory is writable by www-data
Ward Vandewege [Wed, 20 Nov 2013 15:08:34 +0000 (10:08 -0500)]
Make sure the API rails app tmp directory is writable by www-data

11 years ago* Mention SSO server to documentation in README
Ward Vandewege [Wed, 20 Nov 2013 04:31:13 +0000 (23:31 -0500)]
* Mention SSO server to documentation in README

* Fix up SSO and API server configuration to allow API and SSO server to communicate

11 years agoAdd threaded Keep.put()
Tom Clegg [Tue, 19 Nov 2013 23:07:21 +0000 (15:07 -0800)]
Add threaded Keep.put()

11 years agoFixed Makefile dependencies, added docker_build script to work around
Tim Pierce [Wed, 20 Nov 2013 00:01:07 +0000 (19:01 -0500)]
Fixed Makefile dependencies, added docker_build script to work around
https://github.com/dotcloud/docker/issues/1875.

11 years agoOptimize docserver/Dockerfile
Ward Vandewege [Tue, 19 Nov 2013 22:08:22 +0000 (17:08 -0500)]
Optimize docserver/Dockerfile

11 years ago* Make arvados/passenger container
Ward Vandewege [Tue, 19 Nov 2013 21:09:14 +0000 (16:09 -0500)]
* Make arvados/passenger container
* Build time optimization of the various containers

11 years agoFix encoding issue so Python clients can store binary data in Keep
Tom Clegg [Tue, 19 Nov 2013 21:03:34 +0000 (13:03 -0800)]
Fix encoding issue so Python clients can store binary data in Keep

http://stackoverflow.com/questions/8705962/python-http-put-with-unencoded-binary-data

11 years agoWarehouse docker image no longer requires local checked-out source.
Tim Pierce [Tue, 19 Nov 2013 20:46:32 +0000 (15:46 -0500)]
Warehouse docker image no longer requires local checked-out source.

11 years agoRefuse to stream responses unless request Accept header has text/plain
Tom Clegg [Tue, 19 Nov 2013 20:19:53 +0000 (12:19 -0800)]
Refuse to stream responses unless request Accept header has text/plain
or */*. Closes #1604

11 years agoSend "Accept: application/json" header with API requests. refs #1604
Tom Clegg [Tue, 19 Nov 2013 20:18:12 +0000 (12:18 -0800)]
Send "Accept: application/json" header with API requests. refs #1604

11 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Tim Pierce [Tue, 19 Nov 2013 20:15:31 +0000 (15:15 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

11 years agoUpdated Makefile for more accurate dependency management.
Tim Pierce [Tue, 19 Nov 2013 20:15:04 +0000 (15:15 -0500)]
Updated Makefile for more accurate dependency management.

11 years agoMove api/job-tasks to api/crunch-scripts and add some explanation
Tom Clegg [Tue, 19 Nov 2013 18:00:49 +0000 (10:00 -0800)]
Move api/job-tasks to api/crunch-scripts and add some explanation

11 years agoFix navorder of API resource pages
Tom Clegg [Tue, 19 Nov 2013 17:21:32 +0000 (09:21 -0800)]
Fix navorder of API resource pages

11 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Tim Pierce [Tue, 19 Nov 2013 17:44:34 +0000 (12:44 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

Conflicts:
docker/base/Dockerfile

11 years agoAdded Makefile to track dependencies for Docker images.
Tim Pierce [Tue, 19 Nov 2013 17:35:58 +0000 (12:35 -0500)]
Added Makefile to track dependencies for Docker images.

11 years agoUpdate the dockserver Dockerfile to take advantage of the doc/Gemfile,
Ward Vandewege [Tue, 19 Nov 2013 17:22:17 +0000 (12:22 -0500)]
Update the dockserver Dockerfile to take advantage of the doc/Gemfile,
as well as the fact that locale generation and the DEBIAN_FRONTEND
environment variable are now set in the base container.

11 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Ward Vandewege [Tue, 19 Nov 2013 17:21:43 +0000 (12:21 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

11 years agoMove UTF-8 locale generation and the DEBIAN_FRONTEND environment setting
Ward Vandewege [Tue, 19 Nov 2013 17:21:14 +0000 (12:21 -0500)]
Move UTF-8 locale generation and the DEBIAN_FRONTEND environment setting
into the base image.

11 years agoFix job script attribute (not script_name), and update script_version
Tom Clegg [Tue, 19 Nov 2013 17:09:51 +0000 (09:09 -0800)]
Fix job script attribute (not script_name), and update script_version
notes to match current code

11 years agoDo not re-attempt failed job tasks, unless there is some reason to
Tom Clegg [Tue, 19 Nov 2013 16:54:50 +0000 (08:54 -0800)]
Do not re-attempt failed job tasks, unless there is some reason to
believe the failure is temporary (i.e., exit value 111 or slurm
failure). closes #1619

11 years agoUpdate development-mode crunch-job notes
Tom Clegg [Tue, 19 Nov 2013 16:45:36 +0000 (08:45 -0800)]
Update development-mode crunch-job notes

11 years agoadd doc/Gemfile
Tom Clegg [Tue, 19 Nov 2013 16:43:21 +0000 (08:43 -0800)]
add doc/Gemfile

11 years agoDocument user#event_stream. closes #1605
Tom Clegg [Tue, 19 Nov 2013 16:42:29 +0000 (08:42 -0800)]
Document user#event_stream. closes #1605

11 years ago* OMNIAUTH_APP_SECRET is no longer used
Ward Vandewege [Tue, 19 Nov 2013 16:25:07 +0000 (11:25 -0500)]
* OMNIAUTH_APP_SECRET is no longer used
* We include an omniauth-to-openid gateway container now ('sso')

11 years agoAdd threading locks in Keep client
Tom Clegg [Tue, 19 Nov 2013 09:04:58 +0000 (01:04 -0800)]
Add threading locks in Keep client

11 years agoremove unused class DataReader
Tom Clegg [Tue, 19 Nov 2013 09:02:46 +0000 (01:02 -0800)]
remove unused class DataReader

11 years agoEnsure script_version matches before satisfying pipeline component with a job.
Tom Clegg [Tue, 19 Nov 2013 08:38:38 +0000 (00:38 -0800)]
Ensure script_version matches before satisfying pipeline component with a job.

11 years agoAdd /users/:uuid/event_stream action to subscribe to per-user channel
Tom Clegg [Tue, 19 Nov 2013 08:37:03 +0000 (00:37 -0800)]
Add /users/:uuid/event_stream action to subscribe to per-user channel

11 years agoSend job log messages to owner_uuid redis channel
Tom Clegg [Tue, 19 Nov 2013 08:16:22 +0000 (00:16 -0800)]
Send job log messages to owner_uuid redis channel