Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <ldipentima@veritasgenetics.com>
# $ rake generate baseurl=/example arvados_api_host=example.com
baseurl:
+ current_version:
+ all_versions:
arvados_api_host: localhost
arvados_cluster_uuid: local
arvados_workbench_host: http://localhost
- Cloud:
- admin/storage-classes.html.textile.liquid
- admin/spot-instances.html.textile.liquid
- - Other:
+ - Data Management:
- admin/collection-versioning.html.textile.liquid
+ - admin/collection-managed-properties.html.textile.liquid
+ - Other:
- admin/federation.html.textile.liquid
- admin/controlling-container-reuse.html.textile.liquid
- admin/logs-table-management.html.textile.liquid
Two Arvados Rails servers store data in a PostgreSQL database: the SSO server, and the API server. The API server requires at least version *9.4* of PostgreSQL. Beyond that, you have the flexibility to deploy PostgreSQL any way that the Rails servers will be able to connect to it. Our recommended deployment strategy is:
-* Install PostgreSQL on the the same host as the SSO server, and dedicate that install to hosting the SSO database. This provides the best security for the SSO server, because the database does not have to accept any client connections over the network. Typical load on the SSO server is light enough that deploying both it and its database on the same host does not compromise performance.
+* Install PostgreSQL on the same host as the SSO server, and dedicate that install to hosting the SSO database. This provides the best security for the SSO server, because the database does not have to accept any client connections over the network. Typical load on the SSO server is light enough that deploying both it and its database on the same host does not compromise performance.
* If you want to provide the most scalability for your Arvados cluster, install PostgreSQL for the API server on a dedicated host. This gives you the most flexibility to avoid resource contention, and tune performance separately for the API server and its database. If performance is less of a concern for your installation, you can install PostgreSQL on the API server host directly, as with the SSO server.
Find the section for your distribution below, and follow it to install PostgreSQL on each host where you will deploy it. Then follow the steps in the later section(s) to set up PostgreSQL for the Arvados service(s) that need it.
{% include 'note_python_sc' %}
# Install PostgreSQL:
- <notextile><pre>~$ <span class="userinput">sudo yum install rh-postgresql95</span>
+ <notextile><pre>~$ <span class="userinput">sudo yum install rh-postgresql95 rh-postgresql95-postgresql-contrib</span>
~$ <span class="userinput">scl enable rh-postgresql95 bash</span></pre></notextile>
# Initialize the database:
<notextile><pre>~$ <span class="userinput">sudo postgresql-setup initdb</span></pre></notextile>
Ubuntu 14.04 (Trusty) requires an updated PostgreSQL version, see "the PostgreSQL ubuntu repository":https://www.postgresql.org/download/linux/ubuntu/
# Install PostgreSQL:
- <notextile><pre>~$ <span class="userinput">sudo apt-get install postgresql</span></pre></notextile>
+ <notextile><pre>~$ <span class="userinput">sudo apt-get install postgresql postgresql-contrib</span></pre></notextile>
# "Set up Arvados credentials and databases":#rails_setup for the services that will use this PostgreSQL install.
<a name="rails_setup"></a>
{% assign service_role = "arvados_sso" %}
{% assign service_database = "arvados_sso_production" %}
+ {% assign use_contrib = false %}
{% include 'install_postgres_database' %}
h2(#api). Set up API server credentials and database
{% assign service_role = "arvados" %}
{% assign service_database = "arvados_production" %}
+ {% assign use_contrib = true %}
{% include 'install_postgres_database' %}
InternalURLs: {}
ExternalURL: ""
GitSSH:
+ InternalURLs: {}
ExternalURL: ""
DispatchCloud:
InternalURLs: {}
ExternalURL: "-"
SSO:
+ InternalURLs: {}
ExternalURL: ""
Keepproxy:
InternalURLs: {}
InternalURLs: {}
ExternalURL: "-"
Composer:
+ InternalURLs: {}
ExternalURL: ""
WebShell:
+ InternalURLs: {}
ExternalURL: ""
Workbench1:
InternalURLs: {}
ExternalURL: ""
Workbench2:
+ InternalURLs: {}
ExternalURL: ""
Nodemanager:
InternalURLs: {}
# Interval (seconds) between asynchronous permission view updates. Any
# permission-updating API called with the 'async' parameter schedules a an
# update on the permission view in the future, if not already scheduled.
- AsyncPermissionsUpdateInterval: 20
+ AsyncPermissionsUpdateInterval: 20s
# Maximum number of concurrent outgoing requests to make while
# serving a single incoming multi-cluster (federated) request.
# Interval (seconds) between trash sweeps. During a trash sweep,
# collections are marked as trash if their trash_at time has
# arrived, and deleted if their delete_at time has arrived.
- TrashSweepInterval: 60
+ TrashSweepInterval: 60s
# If true, enable collection versioning.
# When a collection's preserve_version field is true or the current version
# the current collection.
CollectionVersioning: false
- # 0 = auto-create a new version on every update.
- # -1 = never auto-create new versions.
- # > 0 = auto-create a new version when older than the specified number of seconds.
- PreserveVersionIfIdle: -1
+ # 0s = auto-create a new version on every update.
+ # -1s = never auto-create new versions.
+ # > 0s = auto-create a new version when older than the specified number of seconds.
+ PreserveVersionIfIdle: -1s
+ # Managed collection properties. At creation time, if the client didn't
+ # provide the listed keys, they will be automatically populated following
+ # one of the following behaviors:
+ #
+ # * UUID of the user who owns the containing project.
+ # responsible_person_uuid: {function: original_owner, protected: true}
+ #
+ # * Default concrete value.
+ # foo_bar: {value: baz, protected: false}
+ #
+ # If protected is true, only an admin user can modify its value.
+ ManagedProperties: {}
+
Login:
# These settings are provided by your OAuth2 provider (e.g.,
# sso-provider).
# scheduling parameter parameter set.
UsePreemptibleInstances: false
- # Include details about job reuse decisions in the server log. This
- # causes additional database queries to run, so it should not be
- # enabled unless you expect to examine the resulting logs for
- # troubleshooting purposes.
- LogReuseDecisions: false
-
# PEM encoded SSH key (RSA, DSA, or ECDSA) used by the
# (experimental) cloud dispatcher for executing containers on
# worker VMs. Begins with "-----BEGIN RSA PRIVATE KEY-----\n"
LogBytesPerEvent: 4096
LogSecondsBetweenEvents: 1
- # The sample period for throttling logs, in seconds.
- LogThrottlePeriod: 60
+ # The sample period for throttling logs.
+ LogThrottlePeriod: 60s
# Maximum number of bytes that job can log over crunch_log_throttle_period
# before being silenced until the end of the period.
# silenced by throttling are not counted against this total.
LimitLogBytesPerJob: 67108864
- LogPartialLineThrottlePeriod: 5
+ LogPartialLineThrottlePeriod: 5s
- # Container logs are written to Keep and saved in a collection,
- # which is updated periodically while the container runs. This
- # value sets the interval (given in seconds) between collection
- # updates.
- LogUpdatePeriod: 1800
+ # Container logs are written to Keep and saved in a
+ # collection, which is updated periodically while the
+ # container runs. This value sets the interval between
+ # collection updates.
+ LogUpdatePeriod: 30m
# The log collection is also updated when the specified amount of
# log data (given in bytes) is produced in less than one update
# period.
- LogUpdateSize: 33554432
+ LogUpdateSize: 32MiB
SLURM:
Managed:
TimeoutShutdown: 10s
# Worker VM image ID.
- ImageID: ami-01234567890abcdef
+ ImageID: ""
# Tags to add on all resources (VMs, NICs, disks) created by
# the container dispatcher. (Arvados's own tags --
Insecure: false
ActivateUsers: false
SAMPLE:
+ # API endpoint host or host:port; default is {id}.arvadosapi.com
Host: sample.arvadosapi.com
+
+ # Perform a proxy request when a local client requests an
+ # object belonging to this remote.
Proxy: false
+
+ # Default "https". Can be set to "http" for testing.
Scheme: https
+
+ # Disable TLS verify. Can be set to true for testing.
Insecure: false
+
+ # When users present tokens issued by this remote cluster, and
+ # their accounts are active on the remote cluster, activate
+ # them on this cluster too.
ActivateUsers: false
+
+ Workbench:
+ # Workbench1 configs
+ Theme: default
+ ActivationContactLink: mailto:info@arvados.org
+ ArvadosDocsite: https://doc.arvados.org
+ ArvadosPublicDataDocURL: https://playground.arvados.org/projects/public
+ ShowUserAgreementInline: false
+ SecretToken: ""
+ SecretKeyBase: ""
+ RepositoryCache: /var/www/arvados-workbench/current/tmp/git
+ UserProfileFormFields:
+ SAMPLE:
+ Type: text
+ FormFieldTitle: ""
+ FormFieldDescription: ""
+ Required: true
+ UserProfileFormMessage: 'Welcome to Arvados. All <span style="color:red">required fields</span> must be completed before you can proceed.'
+ ApplicationMimetypesWithViewIcon:
+ cwl: {}
+ fasta: {}
+ go: {}
+ javascript: {}
+ json: {}
+ pdf: {}
+ python: {}
+ x-python: {}
+ r: {}
+ rtf: {}
+ sam: {}
+ x-sh: {}
+ vnd.realvnc.bed: {}
+ xml: {}
+ xsl: {}
+ LogViewerMaxBytes: 1M
+ EnablePublicProjectsPage: true
+ EnableGettingStartedPopup: false
+ APIResponseCompression: true
+ APIClientConnectTimeout: 2m
+ APIClientReceiveTimeout: 5m
+ RunningJobLogRecordsToFetch: 2000
+ ShowRecentCollectionsOnDashboard: true
+ ShowUserNotifications: true
+ MultiSiteSearch: false
+ Repositories: true
+ SiteName: Arvados Workbench
+
+ # Workbench2 configs
+ VocabularyURL: ""
+ FileViewersConfigURL: ""
InternalURLs: {}
ExternalURL: ""
GitSSH:
+ InternalURLs: {}
ExternalURL: ""
DispatchCloud:
InternalURLs: {}
ExternalURL: "-"
SSO:
+ InternalURLs: {}
ExternalURL: ""
Keepproxy:
InternalURLs: {}
InternalURLs: {}
ExternalURL: "-"
Composer:
+ InternalURLs: {}
ExternalURL: ""
WebShell:
+ InternalURLs: {}
ExternalURL: ""
Workbench1:
InternalURLs: {}
ExternalURL: ""
Workbench2:
+ InternalURLs: {}
ExternalURL: ""
Nodemanager:
InternalURLs: {}
# Interval (seconds) between asynchronous permission view updates. Any
# permission-updating API called with the 'async' parameter schedules a an
# update on the permission view in the future, if not already scheduled.
- AsyncPermissionsUpdateInterval: 20
+ AsyncPermissionsUpdateInterval: 20s
# Maximum number of concurrent outgoing requests to make while
# serving a single incoming multi-cluster (federated) request.
# Interval (seconds) between trash sweeps. During a trash sweep,
# collections are marked as trash if their trash_at time has
# arrived, and deleted if their delete_at time has arrived.
- TrashSweepInterval: 60
+ TrashSweepInterval: 60s
# If true, enable collection versioning.
# When a collection's preserve_version field is true or the current version
# the current collection.
CollectionVersioning: false
- # 0 = auto-create a new version on every update.
- # -1 = never auto-create new versions.
- # > 0 = auto-create a new version when older than the specified number of seconds.
- PreserveVersionIfIdle: -1
+ # 0s = auto-create a new version on every update.
+ # -1s = never auto-create new versions.
+ # > 0s = auto-create a new version when older than the specified number of seconds.
+ PreserveVersionIfIdle: -1s
+ # Managed collection properties. At creation time, if the client didn't
+ # provide the listed keys, they will be automatically populated following
+ # one of the following behaviors:
+ #
+ # * UUID of the user who owns the containing project.
+ # responsible_person_uuid: {function: original_owner, protected: true}
+ #
+ # * Default concrete value.
+ # foo_bar: {value: baz, protected: false}
+ #
+ # If protected is true, only an admin user can modify its value.
+ ManagedProperties: {}
+
Login:
# These settings are provided by your OAuth2 provider (e.g.,
# sso-provider).
# scheduling parameter parameter set.
UsePreemptibleInstances: false
- # Include details about job reuse decisions in the server log. This
- # causes additional database queries to run, so it should not be
- # enabled unless you expect to examine the resulting logs for
- # troubleshooting purposes.
- LogReuseDecisions: false
-
# PEM encoded SSH key (RSA, DSA, or ECDSA) used by the
# (experimental) cloud dispatcher for executing containers on
# worker VMs. Begins with "-----BEGIN RSA PRIVATE KEY-----\n"
LogBytesPerEvent: 4096
LogSecondsBetweenEvents: 1
- # The sample period for throttling logs, in seconds.
- LogThrottlePeriod: 60
+ # The sample period for throttling logs.
+ LogThrottlePeriod: 60s
# Maximum number of bytes that job can log over crunch_log_throttle_period
# before being silenced until the end of the period.
# silenced by throttling are not counted against this total.
LimitLogBytesPerJob: 67108864
- LogPartialLineThrottlePeriod: 5
+ LogPartialLineThrottlePeriod: 5s
- # Container logs are written to Keep and saved in a collection,
- # which is updated periodically while the container runs. This
- # value sets the interval (given in seconds) between collection
- # updates.
- LogUpdatePeriod: 1800
+ # Container logs are written to Keep and saved in a
+ # collection, which is updated periodically while the
+ # container runs. This value sets the interval between
+ # collection updates.
+ LogUpdatePeriod: 30m
# The log collection is also updated when the specified amount of
# log data (given in bytes) is produced in less than one update
# period.
- LogUpdateSize: 33554432
+ LogUpdateSize: 32MiB
SLURM:
Managed:
TimeoutShutdown: 10s
# Worker VM image ID.
- ImageID: ami-01234567890abcdef
+ ImageID: ""
# Tags to add on all resources (VMs, NICs, disks) created by
# the container dispatcher. (Arvados's own tags --
Insecure: false
ActivateUsers: false
SAMPLE:
+ # API endpoint host or host:port; default is {id}.arvadosapi.com
Host: sample.arvadosapi.com
+
+ # Perform a proxy request when a local client requests an
+ # object belonging to this remote.
Proxy: false
+
+ # Default "https". Can be set to "http" for testing.
Scheme: https
+
+ # Disable TLS verify. Can be set to true for testing.
Insecure: false
+
+ # When users present tokens issued by this remote cluster, and
+ # their accounts are active on the remote cluster, activate
+ # them on this cluster too.
ActivateUsers: false
+
+ Workbench:
+ # Workbench1 configs
+ Theme: default
+ ActivationContactLink: mailto:info@arvados.org
+ ArvadosDocsite: https://doc.arvados.org
+ ArvadosPublicDataDocURL: https://playground.arvados.org/projects/public
+ ShowUserAgreementInline: false
+ SecretToken: ""
+ SecretKeyBase: ""
+ RepositoryCache: /var/www/arvados-workbench/current/tmp/git
+ UserProfileFormFields:
+ SAMPLE:
+ Type: text
+ FormFieldTitle: ""
+ FormFieldDescription: ""
+ Required: true
+ UserProfileFormMessage: 'Welcome to Arvados. All <span style="color:red">required fields</span> must be completed before you can proceed.'
+ ApplicationMimetypesWithViewIcon:
+ cwl: {}
+ fasta: {}
+ go: {}
+ javascript: {}
+ json: {}
+ pdf: {}
+ python: {}
+ x-python: {}
+ r: {}
+ rtf: {}
+ sam: {}
+ x-sh: {}
+ vnd.realvnc.bed: {}
+ xml: {}
+ xsl: {}
+ LogViewerMaxBytes: 1M
+ EnablePublicProjectsPage: true
+ EnableGettingStartedPopup: false
+ APIResponseCompression: true
+ APIClientConnectTimeout: 2m
+ APIClientReceiveTimeout: 5m
+ RunningJobLogRecordsToFetch: 2000
+ ShowRecentCollectionsOnDashboard: true
+ ShowUserNotifications: true
+ MultiSiteSearch: false
+ Repositories: true
+ SiteName: Arvados Workbench
+
+ # Workbench2 configs
+ VocabularyURL: ""
+ FileViewersConfigURL: ""
`)
end
end
+ def user_owner_uuid
+ if self.owner_uuid.nil?
+ return current_user.uuid
+ end
+ owner_class = ArvadosModel.resource_class_for_uuid(self.owner_uuid)
+ if owner_class == User
+ self.owner_uuid
+ else
+ owner_class.find_by_uuid(self.owner_uuid).user_owner_uuid
+ end
+ end
+
def logged_attributes
attributes.except(*Rails.configuration.AuditLogs.UnloggedAttributes)
end
end.map(&:name)
end
+ def self.full_text_coalesce
+ full_text_searchable_columns.collect do |column|
+ is_jsonb = self.columns.select{|x|x.name == column}[0].type == :jsonb
+ cast = (is_jsonb || serialized_attributes[column]) ? '::text' : ''
+ "coalesce(#{column}#{cast},'')"
+ end
+ end
+
+ def self.full_text_trgm
+ "(#{full_text_coalesce.join(" || ' ' || ")})"
+ end
+
def self.full_text_tsvector
parts = full_text_searchable_columns.collect do |column|
is_jsonb = self.columns.select{|x|x.name == column}[0].type == :jsonb
projects / arvados.git / commitdiff