arvados.git
10 years agoOnly apply the auto_admin_user configuration rule when no admin users
Ward Vandewege [Fri, 22 Nov 2013 01:05:27 +0000 (20:05 -0500)]
Only apply the auto_admin_user configuration rule when no admin users
exist yet, but exclude the default root system user from that count.
Just in case.

10 years agoOnly apply the auto_admin_user configuration rule when no admin users
Ward Vandewege [Thu, 21 Nov 2013 01:22:48 +0000 (20:22 -0500)]
Only apply the auto_admin_user configuration rule when no admin users
exist yet. Just in case.

10 years agoadd script/create_superuser_token.rb
Tom Clegg [Wed, 20 Nov 2013 23:14:18 +0000 (15:14 -0800)]
add script/create_superuser_token.rb

10 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Ward Vandewege [Wed, 20 Nov 2013 22:22:36 +0000 (17:22 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

10 years ago* Docker API container: add support for the auto_admin_user feature (API server).
Ward Vandewege [Wed, 20 Nov 2013 22:19:24 +0000 (17:19 -0500)]
* Docker API container: add support for the auto_admin_user feature (API server).

This closes #1655

* Default WORKBENCH_INSECURE_HTTPS in config.yml.example to true,
  because we use self-signed certificates in the development environment.

10 years agoConverted README to real markdown.
Tim Pierce [Wed, 20 Nov 2013 22:14:30 +0000 (17:14 -0500)]
Converted README to real markdown.

10 years agoFix dependencies.
Tim Pierce [Wed, 20 Nov 2013 21:59:10 +0000 (16:59 -0500)]
Fix dependencies.

10 years agoImprove Makefile rules.
Tim Pierce [Wed, 20 Nov 2013 18:32:31 +0000 (13:32 -0500)]
Improve Makefile rules.
Start Keep container from run.sh.

10 years agoAdd a config setting: auto_admin_user
Ward Vandewege [Wed, 20 Nov 2013 22:15:49 +0000 (17:15 -0500)]
Add a config setting: auto_admin_user

This setting is optional, and can be configured to the The e-mail
address of the user you would like to become marked as an admin user on
their first login. This is useful during installation.

In the default configuration, authentication happens through the Arvados
SSO server, which uses openid against Google's servers. In that case
this e-mail address should be an address associated with a Google
account.

10 years agoConverted README to real markdown.
Tim Pierce [Wed, 20 Nov 2013 22:14:30 +0000 (17:14 -0500)]
Converted README to real markdown.

10 years agoFix dependencies.
Tim Pierce [Wed, 20 Nov 2013 21:59:10 +0000 (16:59 -0500)]
Fix dependencies.

10 years agoImprove Makefile rules.
Tim Pierce [Wed, 20 Nov 2013 18:32:31 +0000 (13:32 -0500)]
Improve Makefile rules.
Start Keep container from run.sh.

10 years agoMake workbench work with the API server.
Ward Vandewege [Wed, 20 Nov 2013 15:15:07 +0000 (10:15 -0500)]
Make workbench work with the API server.

10 years agoMake sure the API rails app tmp directory is writable by www-data
Ward Vandewege [Wed, 20 Nov 2013 15:08:34 +0000 (10:08 -0500)]
Make sure the API rails app tmp directory is writable by www-data

10 years ago* Mention SSO server to documentation in README
Ward Vandewege [Wed, 20 Nov 2013 04:31:13 +0000 (23:31 -0500)]
* Mention SSO server to documentation in README

* Fix up SSO and API server configuration to allow API and SSO server to communicate

10 years agoAdd threaded Keep.put()
Tom Clegg [Tue, 19 Nov 2013 23:07:21 +0000 (15:07 -0800)]
Add threaded Keep.put()

10 years agoFixed Makefile dependencies, added docker_build script to work around
Tim Pierce [Wed, 20 Nov 2013 00:01:07 +0000 (19:01 -0500)]
Fixed Makefile dependencies, added docker_build script to work around
https://github.com/dotcloud/docker/issues/1875.

10 years agoOptimize docserver/Dockerfile
Ward Vandewege [Tue, 19 Nov 2013 22:08:22 +0000 (17:08 -0500)]
Optimize docserver/Dockerfile

10 years ago* Make arvados/passenger container
Ward Vandewege [Tue, 19 Nov 2013 21:09:14 +0000 (16:09 -0500)]
* Make arvados/passenger container
* Build time optimization of the various containers

10 years agoFix encoding issue so Python clients can store binary data in Keep
Tom Clegg [Tue, 19 Nov 2013 21:03:34 +0000 (13:03 -0800)]
Fix encoding issue so Python clients can store binary data in Keep

http://stackoverflow.com/questions/8705962/python-http-put-with-unencoded-binary-data

10 years agoWarehouse docker image no longer requires local checked-out source.
Tim Pierce [Tue, 19 Nov 2013 20:46:32 +0000 (15:46 -0500)]
Warehouse docker image no longer requires local checked-out source.

10 years agoRefuse to stream responses unless request Accept header has text/plain
Tom Clegg [Tue, 19 Nov 2013 20:19:53 +0000 (12:19 -0800)]
Refuse to stream responses unless request Accept header has text/plain
or */*. Closes #1604

10 years agoSend "Accept: application/json" header with API requests. refs #1604
Tom Clegg [Tue, 19 Nov 2013 20:18:12 +0000 (12:18 -0800)]
Send "Accept: application/json" header with API requests. refs #1604

10 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Tim Pierce [Tue, 19 Nov 2013 20:15:31 +0000 (15:15 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

10 years agoUpdated Makefile for more accurate dependency management.
Tim Pierce [Tue, 19 Nov 2013 20:15:04 +0000 (15:15 -0500)]
Updated Makefile for more accurate dependency management.

10 years agoMove api/job-tasks to api/crunch-scripts and add some explanation
Tom Clegg [Tue, 19 Nov 2013 18:00:49 +0000 (10:00 -0800)]
Move api/job-tasks to api/crunch-scripts and add some explanation

10 years agoFix navorder of API resource pages
Tom Clegg [Tue, 19 Nov 2013 17:21:32 +0000 (09:21 -0800)]
Fix navorder of API resource pages

10 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Tim Pierce [Tue, 19 Nov 2013 17:44:34 +0000 (12:44 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

Conflicts:
docker/base/Dockerfile

10 years agoAdded Makefile to track dependencies for Docker images.
Tim Pierce [Tue, 19 Nov 2013 17:35:58 +0000 (12:35 -0500)]
Added Makefile to track dependencies for Docker images.

10 years agoUpdate the dockserver Dockerfile to take advantage of the doc/Gemfile,
Ward Vandewege [Tue, 19 Nov 2013 17:22:17 +0000 (12:22 -0500)]
Update the dockserver Dockerfile to take advantage of the doc/Gemfile,
as well as the fact that locale generation and the DEBIAN_FRONTEND
environment variable are now set in the base container.

10 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Ward Vandewege [Tue, 19 Nov 2013 17:21:43 +0000 (12:21 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

10 years agoMove UTF-8 locale generation and the DEBIAN_FRONTEND environment setting
Ward Vandewege [Tue, 19 Nov 2013 17:21:14 +0000 (12:21 -0500)]
Move UTF-8 locale generation and the DEBIAN_FRONTEND environment setting
into the base image.

10 years agoFix job script attribute (not script_name), and update script_version
Tom Clegg [Tue, 19 Nov 2013 17:09:51 +0000 (09:09 -0800)]
Fix job script attribute (not script_name), and update script_version
notes to match current code

10 years agoDo not re-attempt failed job tasks, unless there is some reason to
Tom Clegg [Tue, 19 Nov 2013 16:54:50 +0000 (08:54 -0800)]
Do not re-attempt failed job tasks, unless there is some reason to
believe the failure is temporary (i.e., exit value 111 or slurm
failure). closes #1619

10 years agoUpdate development-mode crunch-job notes
Tom Clegg [Tue, 19 Nov 2013 16:45:36 +0000 (08:45 -0800)]
Update development-mode crunch-job notes

10 years agoadd doc/Gemfile
Tom Clegg [Tue, 19 Nov 2013 16:43:21 +0000 (08:43 -0800)]
add doc/Gemfile

10 years agoDocument user#event_stream. closes #1605
Tom Clegg [Tue, 19 Nov 2013 16:42:29 +0000 (08:42 -0800)]
Document user#event_stream. closes #1605

10 years ago* OMNIAUTH_APP_SECRET is no longer used
Ward Vandewege [Tue, 19 Nov 2013 16:25:07 +0000 (11:25 -0500)]
* OMNIAUTH_APP_SECRET is no longer used
* We include an omniauth-to-openid gateway container now ('sso')

10 years agoAdd threading locks in Keep client
Tom Clegg [Tue, 19 Nov 2013 09:04:58 +0000 (01:04 -0800)]
Add threading locks in Keep client

10 years agoremove unused class DataReader
Tom Clegg [Tue, 19 Nov 2013 09:02:46 +0000 (01:02 -0800)]
remove unused class DataReader

10 years agoEnsure script_version matches before satisfying pipeline component with a job.
Tom Clegg [Tue, 19 Nov 2013 08:38:38 +0000 (00:38 -0800)]
Ensure script_version matches before satisfying pipeline component with a job.

10 years agoAdd /users/:uuid/event_stream action to subscribe to per-user channel
Tom Clegg [Tue, 19 Nov 2013 08:37:03 +0000 (00:37 -0800)]
Add /users/:uuid/event_stream action to subscribe to per-user channel

10 years agoSend job log messages to owner_uuid redis channel
Tom Clegg [Tue, 19 Nov 2013 08:16:22 +0000 (00:16 -0800)]
Send job log messages to owner_uuid redis channel

10 years agoRead pipeline template from local file if --template arg is not a uuid
Tom Clegg [Tue, 19 Nov 2013 00:40:40 +0000 (16:40 -0800)]
Read pipeline template from local file if --template arg is not a uuid

10 years agoun-comment ADD line for api.tar
Ward Vandewege [Tue, 19 Nov 2013 03:30:19 +0000 (22:30 -0500)]
un-comment ADD line for api.tar

10 years agoLet API server use SSO container.
Ward Vandewege [Tue, 19 Nov 2013 03:24:03 +0000 (22:24 -0500)]
Let API server use SSO container.

This doesn't work quite yet - need to sort out inter-container
communication. But it's very close now.

10 years agoFixes for SSO rails app.
Ward Vandewege [Tue, 19 Nov 2013 03:23:44 +0000 (22:23 -0500)]
Fixes for SSO rails app.

10 years agoGenerated files shouldn't have been committed.
Ward Vandewege [Tue, 19 Nov 2013 02:51:38 +0000 (21:51 -0500)]
Generated files shouldn't have been committed.

10 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Ward Vandewege [Mon, 18 Nov 2013 22:25:29 +0000 (17:25 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

Conflicts:
docker/build.sh
docker/config.yml.example

10 years agoMerge branch '1579-keep-server-in-docker' of git.clinicalfuture.com:arvados into...
Tim Pierce [Mon, 18 Nov 2013 21:50:38 +0000 (16:50 -0500)]
Merge branch '1579-keep-server-in-docker' of git.clinicalfuture.com:arvados into 1579-keep-server-in-docker

10 years agoInstall warehouse-apps into arvados/warehouse directly from github source.
Tim Pierce [Mon, 18 Nov 2013 20:06:21 +0000 (15:06 -0500)]
Install warehouse-apps into arvados/warehouse directly from github source.

10 years agoAdd prerequisites for Keep server described in
Tim Pierce [Fri, 15 Nov 2013 23:52:56 +0000 (18:52 -0500)]
Add prerequisites for Keep server described in
https://github.com/clinicalfuture/warehouse-apps/blob/master/libwarehouse-perl/server/INSTALL

10 years agoAdd supervisor to start keepd when container is launched.
Tim Pierce [Thu, 14 Nov 2013 22:54:55 +0000 (17:54 -0500)]
Add supervisor to start keepd when container is launched.
Update README with deploy instructions for Keep.

10 years agoBuild an arvados/warehouse container with locally checked out warehouse-apps
Tim Pierce [Thu, 14 Nov 2013 20:49:18 +0000 (15:49 -0500)]
Build an arvados/warehouse container with locally checked out warehouse-apps
code.

The warehouse-apps repo at git://github.com/clinicalfuture/warehouse-apps.git
must be checked out at the same level as the arvados repo.

10 years agoInitial checkin for building warehouse-apps.
Tim Pierce [Fri, 8 Nov 2013 22:30:59 +0000 (17:30 -0500)]
Initial checkin for building warehouse-apps.

10 years agoDeploy containers with working copy of sources.
Tim Pierce [Thu, 14 Nov 2013 07:10:05 +0000 (02:10 -0500)]
Deploy containers with working copy of sources.

11 years agoRevert anjlab-bootstrap-rails to version 2 because bootstrap-editable
Tom Clegg [Mon, 18 Nov 2013 20:49:13 +0000 (12:49 -0800)]
Revert anjlab-bootstrap-rails to version 2 because bootstrap-editable
is not ready for bootstrap 3 yet.

11 years agoInstall warehouse-apps into arvados/warehouse directly from github source.
Tim Pierce [Mon, 18 Nov 2013 20:06:21 +0000 (15:06 -0500)]
Install warehouse-apps into arvados/warehouse directly from github source.

11 years agoMove @redis.exists and .getrange outside .subscribe block. Otherwise
Tom Clegg [Mon, 18 Nov 2013 19:53:00 +0000 (11:53 -0800)]
Move @redis.exists and .getrange outside .subscribe block. Otherwise
they fail silently and the client does not get any of the buffered log
messages.

11 years agoAdd SSO docker container
Ward Vandewege [Mon, 18 Nov 2013 19:49:21 +0000 (14:49 -0500)]
Add SSO docker container

11 years agoBring the name of the workbench secret token config variable in line.
Ward Vandewege [Mon, 18 Nov 2013 19:47:35 +0000 (14:47 -0500)]
Bring the name of the workbench secret token config variable in line.

11 years agoAvoid Rails cache when polling database for latest node/job state
Tom Clegg [Mon, 18 Nov 2013 19:38:10 +0000 (11:38 -0800)]
Avoid Rails cache when polling database for latest node/job state

11 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Ward Vandewege [Mon, 18 Nov 2013 19:36:38 +0000 (14:36 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

11 years agoAlso auto-generate _SECRET config variables when they are unset, like we
Ward Vandewege [Mon, 18 Nov 2013 19:36:15 +0000 (14:36 -0500)]
Also auto-generate _SECRET config variables when they are unset, like we
do for _PW config variables.

11 years agoAdd Admin/Nodes page
Tom Clegg [Mon, 18 Nov 2013 19:21:30 +0000 (11:21 -0800)]
Add Admin/Nodes page

11 years agoInclude privileged attributes in node#ping response
Tom Clegg [Mon, 18 Nov 2013 19:09:44 +0000 (11:09 -0800)]
Include privileged attributes in node#ping response

11 years agoadd tests for node#ping, fix response when ping_secret is invalid
Tom Clegg [Mon, 18 Nov 2013 18:21:28 +0000 (10:21 -0800)]
add tests for node#ping, fix response when ping_secret is invalid

11 years agoFix crash building api response when user is nil
Tom Clegg [Mon, 18 Nov 2013 18:20:44 +0000 (10:20 -0800)]
Fix crash building api response when user is nil

11 years agoMerge branch '1600-native-keep-client'
Tom Clegg [Mon, 18 Nov 2013 04:58:23 +0000 (20:58 -0800)]
Merge branch '1600-native-keep-client'

11 years agoObey ARVADOS_API_HOST_INSECURE and use the debian trusted CA list if
Tom Clegg [Fri, 15 Nov 2013 16:40:46 +0000 (08:40 -0800)]
Obey ARVADOS_API_HOST_INSECURE and use the debian trusted CA list if
available (instead of skipping certificate validation because Python's
CA list does not include the one we need).

11 years agoAllow all active users to retrieve list of keep nodes.
Tom Clegg [Fri, 15 Nov 2013 16:31:54 +0000 (08:31 -0800)]
Allow all active users to retrieve list of keep nodes.
refs #1599

11 years agoturn on logging if ARVADOS_DEBUG environment variable exists
Tom Clegg [Mon, 18 Nov 2013 04:55:19 +0000 (20:55 -0800)]
turn on logging if ARVADOS_DEBUG environment variable exists

11 years agoUse hash to determine probe order
Tom Clegg [Mon, 18 Nov 2013 04:54:35 +0000 (20:54 -0800)]
Use hash to determine probe order

11 years agoAdd test case for reading and writing small Keep blocks
Tom Clegg [Mon, 18 Nov 2013 04:28:55 +0000 (20:28 -0800)]
Add test case for reading and writing small Keep blocks

11 years agoUse httplib2 to connect to Keep instead of executing whget/whput
Tom Clegg [Mon, 18 Nov 2013 04:26:23 +0000 (20:26 -0800)]
Use httplib2 to connect to Keep instead of executing whget/whput

11 years agoObey ARVADOS_API_HOST_INSECURE and use the debian trusted CA list if
Tom Clegg [Fri, 15 Nov 2013 16:40:46 +0000 (08:40 -0800)]
Obey ARVADOS_API_HOST_INSECURE and use the debian trusted CA list if
available (instead of skipping certificate validation because Python's
CA list does not include the one we need).

11 years agoAdd prerequisites for Keep server described in
Tim Pierce [Fri, 15 Nov 2013 23:52:56 +0000 (18:52 -0500)]
Add prerequisites for Keep server described in
https://github.com/clinicalfuture/warehouse-apps/blob/master/libwarehouse-perl/server/INSTALL

11 years agoImprove error handling for login operations.
Ward Vandewege [Fri, 15 Nov 2013 21:49:35 +0000 (16:49 -0500)]
Improve error handling for login operations.

11 years agoAllow all active users to retrieve list of keep nodes.
Tom Clegg [Fri, 15 Nov 2013 16:31:54 +0000 (08:31 -0800)]
Allow all active users to retrieve list of keep nodes.
refs #1599

11 years agofix error message scraping regexp
Tom Clegg [Fri, 15 Nov 2013 16:13:41 +0000 (08:13 -0800)]
fix error message scraping regexp

11 years agoUse more current slurm environment variable name in error message scraping
Tom Clegg [Fri, 15 Nov 2013 15:51:45 +0000 (07:51 -0800)]
Use more current slurm environment variable name in error message scraping

11 years agofix over-compensation for attempts counter when task fails on suspect node
Tom Clegg [Fri, 15 Nov 2013 15:51:02 +0000 (07:51 -0800)]
fix over-compensation for attempts counter when task fails on suspect node

11 years agoreduce default #attempts per job task from 9 to 2
Tom Clegg [Fri, 15 Nov 2013 15:45:20 +0000 (07:45 -0800)]
reduce default #attempts per job task from 9 to 2

11 years agofix loop label syntax error introduced in 7c4460b82b8abadc5144f758a9f36ce0f42893e8
Tom Clegg [Fri, 15 Nov 2013 08:15:57 +0000 (00:15 -0800)]
fix loop label syntax error introduced in 7c4460b82b8abadc5144f758a9f36ce0f42893e8

11 years agoAdd supervisor to start keepd when container is launched.
Tim Pierce [Thu, 14 Nov 2013 22:54:55 +0000 (17:54 -0500)]
Add supervisor to start keepd when container is launched.
Update README with deploy instructions for Keep.

11 years agoBuild an arvados/warehouse container with locally checked out warehouse-apps
Tim Pierce [Thu, 14 Nov 2013 20:49:18 +0000 (15:49 -0500)]
Build an arvados/warehouse container with locally checked out warehouse-apps
code.

The warehouse-apps repo at git://github.com/clinicalfuture/warehouse-apps.git
must be checked out at the same level as the arvados repo.

11 years agoMerge branch '1579-keep-server-in-docker' of git.clinicalfuture.com:arvados into...
Tim Pierce [Thu, 14 Nov 2013 17:35:48 +0000 (12:35 -0500)]
Merge branch '1579-keep-server-in-docker' of git.clinicalfuture.com:arvados into 1579-keep-server-in-docker

Rebasing 1579-keep-server-in-docker from 1578-api-server-in-docker
(which has much better Docker support to work from).

11 years agoadd discovery document test
Tom Clegg [Thu, 14 Nov 2013 16:50:13 +0000 (08:50 -0800)]
add discovery document test

11 years agodelete cached discovery document during boot, so doc matches running code
Tom Clegg [Thu, 14 Nov 2013 16:49:58 +0000 (08:49 -0800)]
delete cached discovery document during boot, so doc matches running code

11 years agoupdate discovery document generator for rails 3.2
Tom Clegg [Thu, 14 Nov 2013 16:49:02 +0000 (08:49 -0800)]
update discovery document generator for rails 3.2

11 years agoadd generatedAt field to discovery document
Tom Clegg [Thu, 14 Nov 2013 16:48:22 +0000 (08:48 -0800)]
add generatedAt field to discovery document

11 years agoDeploy containers with working copy of sources.
Tim Pierce [Thu, 14 Nov 2013 07:10:05 +0000 (02:10 -0500)]
Deploy containers with working copy of sources.

11 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Tim Pierce [Wed, 13 Nov 2013 18:31:57 +0000 (13:31 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

11 years agoBreak out deploying vs. installation instructions.
Tim Pierce [Wed, 13 Nov 2013 18:31:21 +0000 (13:31 -0500)]
Break out deploying vs. installation instructions.

11 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Ward Vandewege [Wed, 13 Nov 2013 17:59:02 +0000 (12:59 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados

Conflicts:
docker/workbench/Dockerfile

11 years agoAdd workbench server to build.sh and run.sh
Ward Vandewege [Wed, 13 Nov 2013 17:43:20 +0000 (12:43 -0500)]
Add workbench server to build.sh and run.sh

11 years agodocserver container updates:
Ward Vandewege [Wed, 13 Nov 2013 17:42:23 +0000 (12:42 -0500)]
docserver container updates:
* use base arvados Debian image
* simplify Dockerfile

11 years agoFix Dockerfile comments.
Ward Vandewege [Wed, 13 Nov 2013 17:11:57 +0000 (12:11 -0500)]
Fix Dockerfile comments.

11 years agoAdd docker image with workbench server.
Ward Vandewege [Mon, 11 Nov 2013 22:33:12 +0000 (17:33 -0500)]
Add docker image with workbench server.

11 years agoMerge branch 'master' of git.clinicalfuture.com:arvados
Tim Pierce [Wed, 13 Nov 2013 15:04:03 +0000 (10:04 -0500)]
Merge branch 'master' of git.clinicalfuture.com:arvados