--- /dev/null
+{% comment %}
+Copyright (C) The Arvados Authors. All rights reserved.
+
+SPDX-License-Identifier: CC-BY-SA-3.0
+{% endcomment %}
+
+If you plan to use custom certificates, please set the variable <i>USE_LETSENCRYPT=no</i> and copy your certificates to the directory specified with the variable @CUSTOM_CERTS_DIR@ (usually "./certs") in the remote directory where you copied the @provision.sh@ script. From this dir, the provision script will install the certificates required for the role you're installing.
+
+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>download / collections</i>):
+
+* "controller"
+* "websocket"
+* "workbench"
+* "workbench2"
+* "webshell"
+* "download" # Part of keepweb
+* "collections" # Part of keepweb
+* "keepproxy"
+
+Ie., for 'keepproxy', the script will lookup for
+
+<notextile>
+<pre><code>${CUSTOM_CERTS_DIR}/keepproxy.crt
+${CUSTOM_CERTS_DIR}/keepproxy.key
+</code></pre>
+</notextile>
Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_INT_IP, *_TOKEN</b> and <b>*KEY</b> variables. Those variables will be used to do a search and replace on the <i>pillars/*</i> in place of any matching __VARIABLE__.
-The <i>multi_host</i> include LetsEncrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53. If you will provide custom certificates, please set the variable <i>USE_LETSENCRYPT=no</i>.
+The <i>multi_host</i> example includes Let's Encrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.
+
+{% include 'install_custom_certificates' %}
h3(#further_customization). Further customization of the installation (modifying the salt pillars and states)
Edit the variables in the <i>local.params</i> file. Pay attention to the <b>*_PORT, *_TOKEN</b> and <b>*KEY</b> variables.
+The <i>single_host</i> examples use self-signed SSL certificates, which are deployed using the same mechanism used to deploy custom certificates.
+
+{% include 'install_custom_certificates' %}
+
+If you want to use valid certificates provided by Let's Encrypt, please set the variable <i>USE_LETSENCRYPT=yes</i> and make sure that all the FQDNs that you will use for the public-facing applications (API/controller, Workbench, Keepproxy/Keepweb) are reachable.
+
h3(#single_host_multiple_hostnames). Single host / multiple hostnames (Alternative configuration)
<notextile>
<pre><code>cp local.params.example.single_host_multiple_hostnames local.params
cp -vr /vagrant/tests /home/vagrant/tests;
sed 's#cluster_fixme_or_this_wont_work#harpo#g;
s#domain_fixme_or_this_wont_work#local#g;
- s/#\ BRANCH=\"main\"/\ BRANCH=\"main\"/g;
- s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g' \
+ s#CONTROLLER_EXT_SSL_PORT=443#CONTROLLER_EXT_SSL_PORT=8443#g;
+ s#RELEASE=\"production\"#RELEASE=\"development\"#g;
+ s/# VERSION=.*$/VERSION=\"latest\"/g;
+ s/#\ BRANCH=\"main\"/\ BRANCH=\"main\"/g' \
/vagrant/local.params.example.single_host_multiple_hostnames > /tmp/local.params.single_host_multiple_hostnames"
+
arv.vm.provision "shell",
path: "provision.sh",
args: [
# "--debug",
"--config /tmp/local.params.single_host_multiple_hostnames",
+ "--development",
"--test",
"--vagrant"
].join(" ")
### LETSENCRYPT
letsencrypt:
domainsets:
- __CLUSTER__.__DOMAIN__:
+ controller.__CLUSTER__.__DOMAIN__:
- __CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
- ### SNIPPETS
- snippets:
- __CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
- - ssl_certificate: /etc/letsencrypt/live/__CLUSTER__.__DOMAIN__/fullchain.pem
- - ssl_certificate_key: /etc/letsencrypt/live/__CLUSTER__.__DOMAIN__/privkey.pem
### LETSENCRYPT
letsencrypt:
domainsets:
- keep.__CLUSTER__.__DOMAIN__:
+ keepproxy.__CLUSTER__.__DOMAIN__:
- keep.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
- ### SNIPPETS
- snippets:
- keep.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
- - ssl_certificate: /etc/letsencrypt/live/keep.__CLUSTER__.__DOMAIN__/fullchain.pem
- - ssl_certificate_key: /etc/letsencrypt/live/keep.__CLUSTER__.__DOMAIN__/privkey.pem
collections.__CLUSTER__.__DOMAIN__:
- collections.__CLUSTER__.__DOMAIN__
- '*.collections.__CLUSTER__.__DOMAIN__'
-
-### NGINX
-nginx:
- ### SNIPPETS
- snippets:
- download.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
- - ssl_certificate: /etc/letsencrypt/live/download.__CLUSTER__.__DOMAIN__/fullchain.pem
- - ssl_certificate_key: /etc/letsencrypt/live/download.__CLUSTER__.__DOMAIN__/privkey.pem
- collections.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
- - ssl_certificate: /etc/letsencrypt/live/collections.__CLUSTER__.__DOMAIN__/fullchain.pem
- - ssl_certificate_key: /etc/letsencrypt/live/collections.__CLUSTER__.__DOMAIN__/privkey.pem
domainsets:
webshell.__CLUSTER__.__DOMAIN__:
- webshell.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
- ### SNIPPETS
- snippets:
- webshell.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
- - ssl_certificate: /etc/letsencrypt/live/webshell.__CLUSTER__.__DOMAIN__/fullchain.pem
- - ssl_certificate_key: /etc/letsencrypt/live/webshell.__CLUSTER__.__DOMAIN__/privkey.pem
### LETSENCRYPT
letsencrypt:
domainsets:
- ws.__CLUSTER__.__DOMAIN__:
+ websocket.__CLUSTER__.__DOMAIN__:
- ws.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
- ### SNIPPETS
- snippets:
- ws.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
- - ssl_certificate: /etc/letsencrypt/live/ws.__CLUSTER__.__DOMAIN__/fullchain.pem
- - ssl_certificate_key: /etc/letsencrypt/live/ws.__CLUSTER__.__DOMAIN__/privkey.pem
domainsets:
workbench2.__CLUSTER__.__DOMAIN__:
- workbench2.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
- ### SNIPPETS
- snippets:
- workbench2.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
- - ssl_certificate: /etc/letsencrypt/live/workbench2.__CLUSTER__.__DOMAIN__/fullchain.pem
- - ssl_certificate_key: /etc/letsencrypt/live/workbench2.__CLUSTER__.__DOMAIN__/privkey.pem
domainsets:
workbench.__CLUSTER__.__DOMAIN__:
- workbench.__CLUSTER__.__DOMAIN__
-
-### NGINX
-nginx:
- ### SNIPPETS
- snippets:
- workbench.__CLUSTER__.__DOMAIN___letsencrypt_cert.conf:
- - ssl_certificate: /etc/letsencrypt/live/workbench.__CLUSTER__.__DOMAIN__/fullchain.pem
- - ssl_certificate_key: /etc/letsencrypt/live/workbench.__CLUSTER__.__DOMAIN__/privkey.pem
### SITES
servers:
managed:
- arvados_api:
+ arvados_api.conf:
enabled: true
overwrite: true
config:
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### NGINX
+nginx:
+ servers:
+ managed:
+ ### DEFAULT
+ arvados_collections_default.conf:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: '~^(.*\.)?collections\.__CLUSTER__\.__DOMAIN__'
+ - listen:
+ - 80
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ ### COLLECTIONS
+ arvados_collections_ssl.conf:
+ enabled: true
+ overwrite: true
+ requires:
+ __CERT_REQUIRES__
+ config:
+ - server:
+ - server_name: '~^(.*\.)?collections\.__CLUSTER__\.__DOMAIN__'
+ - listen:
+ - __KEEPWEB_EXT_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /:
+ - proxy_pass: 'http://collections_downloads_upstream'
+ - proxy_read_timeout: 90
+ - proxy_connect_timeout: 90
+ - proxy_redirect: 'off'
+ - proxy_set_header: X-Forwarded-Proto https
+ - proxy_set_header: 'Host $http_host'
+ - proxy_set_header: 'X-Real-IP $remote_addr'
+ - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
+ - proxy_buffering: 'off'
+ - client_max_body_size: 0
+ - proxy_http_version: '1.1'
+ - proxy_request_buffering: 'off'
+ - include: snippets/ssl_hardening_default.conf
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
+ - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
servers:
managed:
### DEFAULT
- arvados_controller_default:
+ arvados_controller_default.conf:
enabled: true
overwrite: true
config:
- server_name: __CLUSTER__.__DOMAIN__
- listen:
- 80 default
+ - location /.well-known:
+ - root: /var/www
- location /:
- return: '301 https://$host$request_uri'
- arvados_controller_ssl:
+ arvados_controller_ssl.conf:
enabled: true
overwrite: true
requires:
- cmd: create-initial-cert-__CLUSTER__.__DOMAIN__-__CLUSTER__.__DOMAIN__
+ __CERT_REQUIRES__
config:
- server:
- server_name: __CLUSTER__.__DOMAIN__
- proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- proxy_set_header: 'X-External-Client $external_client'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
- access_log: /var/log/nginx/controller.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/controller.__CLUSTER__.__DOMAIN__.error.log
- client_max_body_size: 128m
--- /dev/null
+---
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: AGPL-3.0
+
+### NGINX
+nginx:
+ servers:
+ managed:
+ ### DEFAULT
+ arvados_download_default.conf:
+ enabled: true
+ overwrite: true
+ config:
+ - server:
+ - server_name: download.__CLUSTER__.__DOMAIN__
+ - listen:
+ - 80
+ - location /:
+ - return: '301 https://$host$request_uri'
+
+ ### DOWNLOAD
+ arvados_download_ssl.conf:
+ enabled: true
+ overwrite: true
+ requires:
+ __CERT_REQUIRES__
+ config:
+ - server:
+ - server_name: download.__CLUSTER__.__DOMAIN__
+ - listen:
+ - __KEEPWEB_EXT_SSL_PORT__ http2 ssl
+ - index: index.html index.htm
+ - location /:
+ - proxy_pass: 'http://collections_downloads_upstream'
+ - proxy_read_timeout: 90
+ - proxy_connect_timeout: 90
+ - proxy_redirect: 'off'
+ - proxy_set_header: X-Forwarded-Proto https
+ - proxy_set_header: 'Host $http_host'
+ - proxy_set_header: 'X-Real-IP $remote_addr'
+ - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
+ - proxy_buffering: 'off'
+ - client_max_body_size: 0
+ - proxy_http_version: '1.1'
+ - proxy_request_buffering: 'off'
+ - include: snippets/ssl_hardening_default.conf
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
+ - access_log: /var/log/nginx/download.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/download.__CLUSTER__.__DOMAIN__.error.log
servers:
managed:
### DEFAULT
- arvados_keepproxy_default:
+ arvados_keepproxy_default.conf:
enabled: true
overwrite: true
config:
- location /:
- return: '301 https://$host$request_uri'
- arvados_keepproxy_ssl:
+ arvados_keepproxy_ssl.conf:
enabled: true
overwrite: true
requires:
- cmd: create-initial-cert-keep.__CLUSTER__.__DOMAIN__-keep.__CLUSTER__.__DOMAIN__
+ __CERT_REQUIRES__
config:
- server:
- server_name: keep.__CLUSTER__.__DOMAIN__
- listen:
- - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
+ - __KEEP_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /:
- proxy_pass: 'http://keepproxy_upstream'
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/keep.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
- access_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.error.log
#
# SPDX-License-Identifier: AGPL-3.0
+# Keepweb upstream is common to both downloads and collections
### NGINX
nginx:
### SERVER
http:
upstream collections_downloads_upstream:
- server: 'localhost:9002 fail_timeout=10s'
-
- servers:
- managed:
- ### DEFAULT
- arvados_collections_download_default:
- enabled: true
- overwrite: true
- config:
- - server:
- - server_name: '~^((.*\.)?collections|download)\.__CLUSTER__\.__DOMAIN__'
- - listen:
- - 80
- - location /:
- - return: '301 https://$host$request_uri'
-
- ### COLLECTIONS
- arvados_collections_ssl:
- enabled: true
- overwrite: true
- requires:
- cmd: 'create-initial-cert-collections.__CLUSTER__.__DOMAIN__-collections.__CLUSTER__.__DOMAIN__+*.__CLUSTER__.__DOMAIN__'
- config:
- - server:
- - server_name: '*.collections.__CLUSTER__.__DOMAIN__'
- - listen:
- - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- - index: index.html index.htm
- - location /:
- - proxy_pass: 'http://collections_downloads_upstream'
- - proxy_read_timeout: 90
- - proxy_connect_timeout: 90
- - proxy_redirect: 'off'
- - proxy_set_header: X-Forwarded-Proto https
- - proxy_set_header: 'Host $http_host'
- - proxy_set_header: 'X-Real-IP $remote_addr'
- - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- - proxy_buffering: 'off'
- - client_max_body_size: 0
- - proxy_http_version: '1.1'
- - proxy_request_buffering: 'off'
- - include: snippets/ssl_hardening_default.conf
- - include: snippets/collections.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
- - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
- - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
-
- ### DOWNLOAD
- arvados_download_ssl:
- enabled: true
- overwrite: true
- requires:
- cmd: create-initial-cert-download.__CLUSTER__.__DOMAIN__-download.__CLUSTER__.__DOMAIN__
- config:
- - server:
- - server_name: download.__CLUSTER__.__DOMAIN__
- - listen:
- - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- - index: index.html index.htm
- - location /:
- - proxy_pass: 'http://collections_downloads_upstream'
- - proxy_read_timeout: 90
- - proxy_connect_timeout: 90
- - proxy_redirect: 'off'
- - proxy_set_header: X-Forwarded-Proto https
- - proxy_set_header: 'Host $http_host'
- - proxy_set_header: 'X-Real-IP $remote_addr'
- - proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- - proxy_buffering: 'off'
- - client_max_body_size: 0
- - proxy_http_version: '1.1'
- - proxy_request_buffering: 'off'
- - include: snippets/ssl_hardening_default.conf
- - include: snippets/download.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
- - access_log: /var/log/nginx/download.__CLUSTER__.__DOMAIN__.access.log combined
- - error_log: /var/log/nginx/download.__CLUSTER__.__DOMAIN__.error.log
### SITES
servers:
managed:
- arvados_webshell_default:
+ arvados_webshell_default.conf:
enabled: true
overwrite: true
config:
- location /:
- return: '301 https://$host$request_uri'
- arvados_webshell_ssl:
+ arvados_webshell_ssl.conf:
enabled: true
overwrite: true
requires:
- cmd: create-initial-cert-webshell.__CLUSTER__.__DOMAIN__-webshell.__CLUSTER__.__DOMAIN__
+ __CERT_REQUIRES__
config:
- server:
- server_name: webshell.__CLUSTER__.__DOMAIN__
- listen:
- - __CONTROLLER_EXT_SSL_PORT__ http2 ssl
+ - __WEBSHELL_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- location /shell.__CLUSTER__.__DOMAIN__:
- proxy_pass: 'http://webshell_upstream'
- add_header: "'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'"
- include: snippets/ssl_hardening_default.conf
- - include: snippets/webshell.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
- access_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.error.log
servers:
managed:
### DEFAULT
- arvados_websocket_default:
+ arvados_websocket_default.conf:
enabled: true
overwrite: true
config:
- location /:
- return: '301 https://$host$request_uri'
- arvados_websocket_ssl:
+ arvados_websocket_ssl.conf:
enabled: true
overwrite: true
requires:
- cmd: create-initial-cert-ws.__CLUSTER__.__DOMAIN__-ws.__CLUSTER__.__DOMAIN__
+ __CERT_REQUIRES__
config:
- server:
- server_name: ws.__CLUSTER__.__DOMAIN__
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/ws.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
- access_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.error.log
servers:
managed:
### DEFAULT
- arvados_workbench2_default:
+ arvados_workbench2_default.conf:
enabled: true
overwrite: true
config:
- location /:
- return: '301 https://$host$request_uri'
- arvados_workbench2_ssl:
+ arvados_workbench2_ssl.conf:
enabled: true
overwrite: true
requires:
- cmd: create-initial-cert-workbench2.__CLUSTER__.__DOMAIN__-workbench2.__CLUSTER__.__DOMAIN__
+ __CERT_REQUIRES__
config:
- server:
- server_name: workbench2.__CLUSTER__.__DOMAIN__
- location /config.json:
- return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: snippets/ssl_hardening_default.conf
- - include: snippets/workbench2.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
servers:
managed:
### DEFAULT
- arvados_workbench_default:
+ arvados_workbench_default.conf:
enabled: true
overwrite: true
config:
- location /:
- return: '301 https://$host$request_uri'
- arvados_workbench_ssl:
+ arvados_workbench_ssl.conf:
enabled: true
overwrite: true
requires:
- cmd: create-initial-cert-workbench.__CLUSTER__.__DOMAIN__-workbench.__CLUSTER__.__DOMAIN__
+ __CERT_REQUIRES__
config:
- server:
- server_name: workbench.__CLUSTER__.__DOMAIN__
- proxy_set_header: 'X-Real-IP $remote_addr'
- proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/workbench.__CLUSTER__.__DOMAIN___letsencrypt_cert[.]conf
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
- access_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.error.log
tls:
# certificate: ''
# key: ''
- # required to test with arvados-snakeoil certs
- insecure: true
+ # When using arvados-snakeoil certs set insecure: true
+ insecure: false
resources:
virtual_machines:
enabled: true
overwrite: true
requires:
- file: nginx_snippet_arvados-snakeoil.conf
+ __CERT_REQUIRES__
config:
- server:
- server_name: __CLUSTER__.__DOMAIN__
- proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- proxy_set_header: 'X-External-Client $external_client'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/arvados-snakeoil.conf
- - access_log: /var/log/nginx/__CLUSTER__.__DOMAIN__.access.log combined
- - error_log: /var/log/nginx/__CLUSTER__.__DOMAIN__.error.log
+ - ssl_certificate: __CERT_PEM__
+ - ssl_certificate_key: __CERT_KEY__
+ - access_log: /var/log/nginx/controller.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/controller.__CLUSTER__.__DOMAIN__.error.log
- client_max_body_size: 128m
enabled: true
overwrite: true
requires:
- file: nginx_snippet_arvados-snakeoil.conf
+ file: extra_custom_certs_file_copy_arvados-keepproxy.pem
config:
- server:
- server_name: keep.__CLUSTER__.__DOMAIN__
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/arvados-snakeoil.conf
+ - ssl_certificate: /etc/nginx/ssl/arvados-keepproxy.pem
+ - ssl_certificate_key: /etc/nginx/ssl/arvados-keepproxy.key
- access_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/keepproxy.__CLUSTER__.__DOMAIN__.error.log
- return: '301 https://$host$request_uri'
### COLLECTIONS / DOWNLOAD
- arvados_collections_download_ssl.conf:
+ {%- for vh in [
+ 'collections',
+ 'download'
+ ]
+ %}
+ arvados_{{ vh }}.conf:
enabled: true
overwrite: true
requires:
- file: nginx_snippet_arvados-snakeoil.conf
+ file: extra_custom_certs_file_copy_arvados-{{ vh }}.pem
config:
- server:
- - server_name: collections.__CLUSTER__.__DOMAIN__ download.__CLUSTER__.__DOMAIN__
+ - server_name: {{ vh }}.__CLUSTER__.__DOMAIN__
- listen:
- __CONTROLLER_EXT_SSL_PORT__ http2 ssl
- index: index.html index.htm
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/arvados-snakeoil.conf
- - access_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.access.log combined
- - error_log: /var/log/nginx/collections.__CLUSTER__.__DOMAIN__.error.log
+ - ssl_certificate: /etc/nginx/ssl/arvados-{{ vh }}.pem
+ - ssl_certificate_key: /etc/nginx/ssl/arvados-{{ vh }}.key
+ - access_log: /var/log/nginx/{{ vh }}.__CLUSTER__.__DOMAIN__.access.log combined
+ - error_log: /var/log/nginx/{{ vh }}.__CLUSTER__.__DOMAIN__.error.log
+ {%- endfor %}
# replace with the IP address of your resolver
# - resolver: 127.0.0.1
- arvados-snakeoil.conf:
- - ssl_certificate: /etc/ssl/private/arvados-snakeoil-cert.pem
- - ssl_certificate_key: /etc/ssl/private/arvados-snakeoil-cert.key
-
### SITES
servers:
managed:
enabled: true
overwrite: true
requires:
- file: nginx_snippet_arvados-snakeoil.conf
+ file: extra_custom_certs_file_copy_arvados-webshell.pem
config:
- server:
- server_name: webshell.__CLUSTER__.__DOMAIN__
- add_header: "'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'"
{%- endfor %}
- include: snippets/ssl_hardening_default.conf
- - include: snippets/arvados-snakeoil.conf
+ - ssl_certificate: /etc/nginx/ssl/arvados-webshell.pem
+ - ssl_certificate_key: /etc/nginx/ssl/arvados-webshell.key
- access_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/webshell.__CLUSTER__.__DOMAIN__.error.log
enabled: true
overwrite: true
requires:
- file: nginx_snippet_arvados-snakeoil.conf
+ file: extra_custom_certs_file_copy_arvados-websocket.pem
config:
- server:
- server_name: ws.__CLUSTER__.__DOMAIN__
- proxy_http_version: '1.1'
- proxy_request_buffering: 'off'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/arvados-snakeoil.conf
+ - ssl_certificate: /etc/nginx/ssl/arvados-websocket.pem
+ - ssl_certificate_key: /etc/nginx/ssl/arvados-websocket.key
- access_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/ws.__CLUSTER__.__DOMAIN__.error.log
enabled: true
overwrite: true
requires:
- file: nginx_snippet_arvados-snakeoil.conf
+ file: extra_custom_certs_file_copy_arvados-workbench2.pem
config:
- server:
- server_name: workbench2.__CLUSTER__.__DOMAIN__
- location /config.json:
- return: {{ "200 '" ~ '{"API_HOST":"__CLUSTER__.__DOMAIN__:__CONTROLLER_EXT_SSL_PORT__"}' ~ "'" }}
- include: snippets/ssl_hardening_default.conf
- - include: snippets/arvados-snakeoil.conf
+ - ssl_certificate: /etc/nginx/ssl/arvados-workbench2.pem
+ - ssl_certificate_key: /etc/nginx/ssl/arvados-workbench2.key
- access_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench2.__CLUSTER__.__DOMAIN__.error.log
enabled: true
overwrite: true
requires:
- file: nginx_snippet_arvados-snakeoil.conf
+ file: extra_custom_certs_file_copy_arvados-workbench.pem
config:
- server:
- server_name: workbench.__CLUSTER__.__DOMAIN__
- proxy_set_header: 'X-Real-IP $remote_addr'
- proxy_set_header: 'X-Forwarded-For $proxy_add_x_forwarded_for'
- include: snippets/ssl_hardening_default.conf
- - include: snippets/arvados-snakeoil.conf
+ - ssl_certificate: /etc/nginx/ssl/arvados-workbench.pem
+ - ssl_certificate_key: /etc/nginx/ssl/arvados-workbench.key
- access_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.access.log combined
- error_log: /var/log/nginx/workbench.__CLUSTER__.__DOMAIN__.error.log
--- /dev/null
+# Copyright (C) The Arvados Authors. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+{%- set orig_cert_dir = salt['pillar.get']('extra_custom_certs_dir', '/srv/salt/certs') %}
+{%- set dest_cert_dir = '/etc/nginx/ssl' %}
+{%- set certs = salt['pillar.get']('extra_custom_certs', []) %}
+
+extra_custom_certs_file_directory_certs_dir:
+ file.directory:
+ - name: /etc/nginx/ssl
+ - require:
+ - pkg: nginx_install
+
+{%- for cert in certs %}
+ {%- set cert_file = 'arvados-' ~ cert ~ '.pem' %}
+ {#- set csr_file = 'arvados-' ~ cert ~ '.csr' #}
+ {%- set key_file = 'arvados-' ~ cert ~ '.key' %}
+ {% for c in [cert_file, key_file] %}
+extra_custom_certs_file_copy_{{ c }}:
+ file.copy:
+ - name: {{ dest_cert_dir }}/{{ c }}
+ - source: {{ orig_cert_dir }}/{{ c }}
+ - force: true
+ - user: root
+ - group: root
+ - unless: cmp {{ dest_cert_dir }}/{{ c }} {{ orig_cert_dir }}/{{ c }}
+ - require:
+ - file: extra_custom_certs_file_directory_certs_dir
+ {%- endfor %}
+{%- endfor %}
#
# SPDX-License-Identifier: Apache-2.0
+# WARNING: This file is only used for testing purposes, and should not be used
+# in a production environment
+
{%- set curr_tpldir = tpldir %}
{%- set tpldir = 'arvados' %}
{%- from "arvados/map.jinja" import arvados with context %}
{%- set tpldir = curr_tpldir %}
+{%- set orig_cert_dir = salt['pillar.get']('extra_custom_certs_dir', '/srv/salt/certs') %}
+
include:
- nginx.passenger
- nginx.config
# we'll keep it simple here.
{%- set arvados_ca_cert_file = '/etc/ssl/private/arvados-snakeoil-ca.pem' %}
{%- set arvados_ca_key_file = '/etc/ssl/private/arvados-snakeoil-ca.key' %}
-{%- set arvados_cert_file = '/etc/ssl/private/arvados-snakeoil-cert.pem' %}
-{%- set arvados_csr_file = '/etc/ssl/private/arvados-snakeoil-cert.csr' %}
-{%- set arvados_key_file = '/etc/ssl/private/arvados-snakeoil-cert.key' %}
{%- if grains.get('os_family') == 'Debian' %}
{%- set arvados_ca_cert_dest = '/usr/local/share/ca-certificates/arvados-snakeoil-ca.crt' %}
{%- set update_ca_cert = '/usr/sbin/update-ca-certificates' %}
{%- set openssl_conf = '/etc/ssl/openssl.cnf' %}
+
+extra_snakeoil_certs_ssl_cert_pkg_installed:
+ pkg.installed:
+ - name: ssl-cert
+ - require_in:
+ - sls: postgres
+
{%- else %}
{%- set arvados_ca_cert_dest = '/etc/pki/ca-trust/source/anchors/arvados-snakeoil-ca.pem' %}
{%- set update_ca_cert = '/usr/bin/update-ca-trust' %}
{%- set openssl_conf = '/etc/pki/tls/openssl.cnf' %}
+
{%- endif %}
-arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed:
+extra_snakeoil_certs_dependencies_pkg_installed:
pkg.installed:
- pkgs:
- openssl
# random generator, cf
# https://github.com/openssl/openssl/issues/7754
#
-arvados_test_salt_states_examples_single_host_snakeoil_certs_file_comment_etc_openssl_conf:
+extra_snakeoil_certs_file_comment_etc_openssl_conf:
file.comment:
- name: /etc/ssl/openssl.cnf
- regex: ^RANDFILE.*
- onlyif: grep -q ^RANDFILE /etc/ssl/openssl.cnf
- require_in:
- - cmd: arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run
+ - cmd: extra_snakeoil_certs_arvados_snakeoil_ca_cmd_run
-arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run:
+extra_snakeoil_certs_arvados_snakeoil_ca_cmd_run:
# Taken from https://github.com/arvados/arvados/blob/master/tools/arvbox/lib/arvbox/docker/service/certificate/run
cmd.run:
- name: |
- test -f {{ arvados_ca_cert_file }}
- openssl verify -CAfile {{ arvados_ca_cert_file }} {{ arvados_ca_cert_file }}
- require:
- - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed
+ - pkg: extra_snakeoil_certs_dependencies_pkg_installed
+
+# Create independent certs for each vhost
+{%- for vh in [
+ 'collections',
+ 'controller',
+ 'download',
+ 'keepproxy',
+ 'webshell',
+ 'workbench',
+ 'workbench2',
+ 'websocket',
+ ]
+%}
+# We're creating these in a tmp directory, so they're copied to their destination
+# with the `custom_certs` state file, as if using custom certificates.
+{%- set arvados_cert_file = orig_cert_dir ~ '/arvados-' ~ vh ~ '.pem' %}
+{%- set arvados_csr_file = orig_cert_dir ~ '/arvados-' ~ vh ~ '.csr' %}
+{%- set arvados_key_file = orig_cert_dir ~ '/arvados-' ~ vh ~ '.key' %}
-arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_cert_cmd_run:
+extra_snakeoil_certs_arvados_snakeoil_cert_{{ vh }}_cmd_run:
cmd.run:
- name: |
- cat > /tmp/openssl.cnf <<-CNF
+ cat > /tmp/{{ vh }}.openssl.cnf <<-CNF
[req]
default_bits = 2048
prompt = no
default_md = sha256
- req_extensions = rext
distinguished_name = dn
+ req_extensions = rext
+ [rext]
+ subjectAltName = @alt_names
[dn]
C = CC
ST = Some State
L = Some Location
- O = Arvados Formula
- OU = arvados-formula
- CN = {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ O = Arvados Provision Example Single Host / Multiple Hostnames
+ OU = arvados-provision-example-single_host_multiple_hostnames
+ CN = {{ vh }}.{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
emailAddress = admin@{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
- [rext]
- subjectAltName = @alt_names
[alt_names]
{%- for entry in grains.get('ipv4') %}
IP.{{ loop.index }} = {{ entry }}
{%- endfor %}
- {%- for entry in [
- 'keep',
- 'collections',
- 'download',
- 'ws',
- 'workbench',
- 'workbench2',
+ DNS.1 = {{ vh }}.{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ {%- if vh in [
+ 'controller',
+ 'keepproxy',
+ 'websocket'
]
%}
- DNS.{{ loop.index }} = {{ entry }}.{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
- {%- endfor %}
- DNS.7 = {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ {%- if vh == 'controller' %}
+ DNS.2 = {{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ {%- elif vh == 'keepproxy' %}
+ DNS.2 = keep.{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ {%- elif vh == 'websocket' %}
+ DNS.2 = ws.{{ arvados.cluster.name }}.{{ arvados.cluster.domain }}
+ {%- endif %}
+ {%- endif %}
CNF
# The req
openssl req \
- -config /tmp/openssl.cnf \
+ -config /tmp/{{ vh }}.openssl.cnf \
-new \
-nodes \
-sha256 \
-out {{ arvados_csr_file }} \
- -keyout {{ arvados_key_file }} > /tmp/snake_oil_certs.output 2>&1 && \
+ -keyout {{ arvados_key_file }} > /tmp/snakeoil_certs.{{ vh }}.output 2>&1 && \
# The cert
openssl x509 \
-req \
-days 365 \
-in {{ arvados_csr_file }} \
-out {{ arvados_cert_file }} \
- -extfile /tmp/openssl.cnf \
+ -extfile /tmp/{{ vh }}.openssl.cnf \
-extensions rext \
-CA {{ arvados_ca_cert_file }} \
-CAkey {{ arvados_ca_key_file }} \
- test -f {{ arvados_key_file }}
- openssl verify -CAfile {{ arvados_ca_cert_file }} {{ arvados_cert_file }}
- require:
- - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_dependencies_pkg_installed
- - cmd: arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_ca_cmd_run
- # We need this before we can add the nginx's snippet
- - require_in:
- - file: nginx_snippet_arvados-snakeoil.conf
-
-{%- if grains.get('os_family') == 'Debian' %}
-arvados_test_salt_states_examples_single_host_snakeoil_certs_ssl_cert_pkg_installed:
- pkg.installed:
- - name: ssl-cert
+ - pkg: extra_snakeoil_certs_dependencies_pkg_installed
+ - cmd: extra_snakeoil_certs_arvados_snakeoil_ca_cmd_run
- require_in:
- - sls: postgres
+ - file: extra_custom_certs_file_copy_arvados-{{ vh }}.pem
+ - file: extra_custom_certs_file_copy_arvados-{{ vh }}.key
-arvados_test_salt_states_examples_single_host_snakeoil_certs_certs_permissions_cmd_run:
+ {%- if grains.get('os_family') == 'Debian' %}
+extra_snakeoil_certs_certs_permissions_{{ vh}}_cmd_run:
file.managed:
- name: {{ arvados_key_file }}
- owner: root
- group: ssl-cert
- require:
- - cmd: arvados_test_salt_states_examples_single_host_snakeoil_certs_arvados_snake_oil_cert_cmd_run
- - pkg: arvados_test_salt_states_examples_single_host_snakeoil_certs_ssl_cert_pkg_installed
- - require_in:
- - file: nginx_snippet_arvados-snakeoil.conf
-{%- endif %}
+ - cmd: extra_snakeoil_certs_arvados_snakeoil_cert_{{ vh }}_cmd_run
+ - pkg: extra_snakeoil_certs_ssl_cert_pkg_installed
+ {%- endif %}
+{%- endfor %}
tls:
# certificate: ''
# key: ''
- # required to test with arvados-snakeoil certs
+ # When using arvados-snakeoil certs set insecure: true
insecure: true
### TOKENS
SHELL_INT_IP=10.0.0.7
INITIAL_USER="admin"
-INITIAL_USER_PASSWORD="password"
# If not specified, the initial user email will be composed as
# INITIAL_USER@CLUSTER.DOMAIN
# salt formula (https://github.com/saltstack-formulas/letsencrypt-formula) to try to
# automatically obtain and install SSL certificates for your instances or set this
# variable to "no", provide and upload your own certificates to the instances and
-# modify the 'nginx_*' salt pillars accordingly
+# modify the 'nginx_*' salt pillars accordingly (see CUSTOM_CERTS_DIR below)
USE_LETSENCRYPT="yes"
USE_LETSENCRYPT_IAM_USER="yes"
# For collections, we need to obtain a wildcard certificate for
LE_AWS_ACCESS_KEY_ID="AKIABCDEFGHIJKLMNOPQ"
LE_AWS_SECRET_ACCESS_KEY="thisistherandomstringthatisyoursecretkey"
+# If you going to provide your own certificates for Arvados, the provision script can
+# help you deploy them. In order to do that, you need to set `USE_LETSENCRYPT=no` above,
+# and copy the required certificates under the directory specified in the next line.
+# The certs will be copied from this directory by the provision script.
+CUSTOM_CERTS_DIR="./certs"
+# The script expects cert/key files with these basenames (matching the role except for
+# keepweb, which is split in both downoad/collections):
+# "controller"
+# "websocket"
+# "workbench"
+# "workbench2"
+# "webshell"
+# "download" # Part of keepweb
+# "collections" # Part of keepweb
+# "keep" # Keepproxy
+# Ie., 'keep', the script will lookup for
+# ${CUSTOM_CERTS_DIR}/keep.crt
+# ${CUSTOM_CERTS_DIR}/keep.key
+
# The directory to check for the config files (pillars, states) you want to use.
# There are a few examples under 'config_examples'.
# CONFIG_DIR="local_config_dir"
# salt formula (https://github.com/saltstack-formulas/letsencrypt-formula) to try to
# automatically obtain and install SSL certificates for your instances or set this
# variable to "no", provide and upload your own certificates to the instances and
-# modify the 'nginx_*' salt pillars accordingly
+# modify the 'nginx_*' salt pillars accordingly (see CUSTOM_CERTS_DIR below)
USE_LETSENCRYPT="no"
+# If you going to provide your own certificates for Arvados, the provision script can
+# help you deploy them. In order to do that, you need to set `USE_LETSENCRYPT=no` above,
+# and copy the required certificates under the directory specified in the next line.
+# The certs will be copied from this directory by the provision script.
+CUSTOM_CERTS_DIR="./certs"
+# The script expects cert/key files with these basenames (matching the role except for
+# keepweb, which is split in both downoad/collections):
+# "controller"
+# "websocket"
+# "workbench"
+# "workbench2"
+# "webshell"
+# "download" # Part of keepweb
+# "collections" # Part of keepweb
+# "keepproxy"
+# Ie., 'keepproxy', the script will lookup for
+# ${CUSTOM_CERTS_DIR}/keepproxy.crt
+# ${CUSTOM_CERTS_DIR}/keepproxy.key
+
# The directory to check for the config files (pillars, states) you want to use.
# There are a few examples under 'config_examples'.
# CONFIG_DIR="local_config_dir"
echo >&2 " for the selected role/s"
echo >&2 " - writes the resulting files into <dest_dir>"
echo >&2 " -v, --vagrant Run in vagrant and use the /vagrant shared dir"
+ echo >&2 " --development Run in dev mode, using snakeoil certs"
echo >&2
}
fi
TEMP=$(getopt -o c:dhp:r:tv \
- --long config:,debug,dump-config:,help,roles:,test,vagrant \
+ --long config:,debug,development,dump-config:,help,roles:,test,vagrant \
-n "${0}" -- "${@}")
if [ ${?} != 0 ];
DUMP_CONFIG="yes"
shift 2
;;
+ --development)
+ DEV_MODE="yes"
+ shift 1
+ ;;
-r | --roles)
for i in ${2//,/ }
do
done
}
+DEV_MODE="no"
CONFIG_FILE="${SCRIPT_DIR}/local.params"
CONFIG_DIR="local_config_dir"
DUMP_CONFIG="no"
WORKBENCH1_EXT_SSL_PORT=443
WORKBENCH2_EXT_SSL_PORT=3001
+USE_LETSENCRYPT="no"
+CUSTOM_CERTS_DIR="./certs"
+
## These are ARVADOS-related parameters
# For a stable release, change RELEASE "production" and VERSION to the
# package version (including the iteration, e.g. X.Y.Z-1) of the
# States, extra states
if [ -d "${F_DIR}"/extra/extra ]; then
- for f in $(ls "${F_DIR}"/extra/extra/*.sls); do
+ if [ "$DEV_MODE" = "yes" ]; then
+ # In dev mode, we create some snake oil certs that we'll
+ # use as CUSTOM_CERTS, so we don't skip the states file
+ SKIP_SNAKE_OIL="dont_snakeoil_certs"
+ else
+ SKIP_SNAKE_OIL="snakeoil_certs"
+ fi
+ for f in $(ls "${F_DIR}"/extra/extra/*.sls | grep -v ${SKIP_SNAKE_OIL}); do
echo " - extra.$(basename ${f} | sed 's/.sls$//g')" >> ${S_DIR}/top.sls
done
+ # Use custom certs
+ if [ "x${USE_LETSENCRYPT}" != "xyes" ]; then
+ mkdir -p "${F_DIR}"/extra/extra/files
+ fi
fi
# If we want specific roles for a node, just add the desired states
echo " - nginx.passenger" >> ${S_DIR}/top.sls
# Currently, only available on config_examples/multi_host/aws
if [ "x${USE_LETSENCRYPT}" = "xyes" ]; then
- if [ "x${USE_LETSENCRYPT_IAM_USER}" = "xyes" ]; then
- grep -q "aws_credentials" ${S_DIR}/top.sls || echo " - aws_credentials" >> ${S_DIR}/top.sls
+ if [ "x${USE_LETSENCRYPT_IAM_USER}" != "xyes" ]; then
+ grep -q "aws_credentials" ${S_DIR}/top.sls || echo " - extra.aws_credentials" >> ${S_DIR}/top.sls
fi
grep -q "letsencrypt" ${S_DIR}/top.sls || echo " - letsencrypt" >> ${S_DIR}/top.sls
+ else
+ # Use custom certs
+ # Copy certs to formula extra/files
+ # In dev mode, the files will be created and put in the destination directory by the
+ # snakeoil_certs.sls state file
+ mkdir -p /srv/salt/certs
+ cp -rv ${CUSTOM_CERTS_DIR}/* /srv/salt/certs/
+ # We add the custom_certs state
+ grep -q "custom_certs" ${S_DIR}/top.sls || echo " - extra.custom_certs" >> ${S_DIR}/top.sls
fi
+
echo " - postgres" >> ${S_DIR}/top.sls
echo " - docker.software" >> ${S_DIR}/top.sls
echo " - arvados" >> ${S_DIR}/top.sls
echo " - nginx_workbench2_configuration" >> ${P_DIR}/top.sls
echo " - nginx_workbench_configuration" >> ${P_DIR}/top.sls
echo " - postgresql" >> ${P_DIR}/top.sls
+
# Currently, only available on config_examples/multi_host/aws
if [ "x${USE_LETSENCRYPT}" = "xyes" ]; then
- if [ "x${USE_LETSENCRYPT_IAM_USER}" = "xyes" ]; then
+ if [ "x${USE_LETSENCRYPT_IAM_USER}" != "xyes" ]; then
grep -q "aws_credentials" ${P_DIR}/top.sls || echo " - aws_credentials" >> ${P_DIR}/top.sls
fi
grep -q "letsencrypt" ${P_DIR}/top.sls || echo " - letsencrypt" >> ${P_DIR}/top.sls
+
+ # As the pillar differ whether we use LE or custom certs, we need to do a final edition on them
+ for c in controller websocket workbench workbench2 webshell download collections keepproxy; do
+ sed -i "s/__CERT_REQUIRES__/cmd: create-initial-cert-${c}.${CLUSTER}.${DOMAIN}*/g;
+ s#__CERT_PEM__#/etc/letsencrypt/live/${c}.${CLUSTER}.${DOMAIN}/fullchain.pem#g;
+ s#__CERT_KEY__#/etc/letsencrypt/live/${c}.${CLUSTER}.${DOMAIN}/privkey.pem#g" \
+ ${P_DIR}/nginx_${c}_configuration.sls
+ done
+ else
+ # Use custom certs (either dev mode or prod)
+ grep -q "extra_custom_certs" ${P_DIR}/top.sls || echo " - extra_custom_certs" >> ${P_DIR}/top.sls
+ # And add the certs in the custom_certs pillar
+ echo "extra_custom_certs_dir: /srv/salt/certs" > ${P_DIR}/extra_custom_certs.sls
+ echo "extra_custom_certs:" >> ${P_DIR}/extra_custom_certs.sls
+
+ for c in controller websocket workbench workbench2 webshell download collections keepproxy; do
+ grep -q ${c} ${P_DIR}/extra_custom_certs.sls || echo " - ${c}" >> ${P_DIR}/extra_custom_certs.sls
+
+ # As the pillar differ whether we use LE or custom certs, we need to do a final edition on them
+ sed -i "s/__CERT_REQUIRES__/file: extra_custom_certs_file_copy_arvados-${c}.pem/g;
+ s#__CERT_PEM__#/etc/nginx/ssl/arvados-${c}.pem#g;
+ s#__CERT_KEY__#/etc/nginx/ssl/arvados-${c}.key#g" \
+ ${P_DIR}/nginx_${c}_configuration.sls
+ done
fi
else
# If we add individual roles, make sure we add the repo first
grep -q "postgres.client" ${S_DIR}/top.sls || echo " - postgres.client" >> ${S_DIR}/top.sls
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
### If we don't install and run LE before arvados-api-server, it fails and breaks everything
- ### after it so we add this here, as we are, after all, sharing the host for api and controller
+ ### after it. So we add this here as we are, after all, sharing the host for api and controller
# Currently, only available on config_examples/multi_host/aws
if [ "x${USE_LETSENCRYPT}" = "xyes" ]; then
- if [ "x${USE_LETSENCRYPT_IAM_USER}" = "xyes" ]; then
+ if [ "x${USE_LETSENCRYPT_IAM_USER}" != "xyes" ]; then
grep -q "aws_credentials" ${S_DIR}/top.sls || echo " - aws_credentials" >> ${S_DIR}/top.sls
fi
- grep -q "letsencrypt" ${S_DIR}/top.sls || echo " - letsencrypt" >> ${S_DIR}/top.sls
+ grep -q "letsencrypt" ${S_DIR}/top.sls || echo " - letsencrypt" >> ${S_DIR}/top.sls
+ else
+ # Use custom certs
+ cp -v ${CUSTOM_CERTS_DIR}/controller.* "${F_DIR}/extra/extra/files/"
+ # We add the custom_certs state
+ grep -q "custom_certs" ${S_DIR}/top.sls || echo " - extra.custom_certs" >> ${S_DIR}/top.sls
fi
grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
# Pillars
grep -q "nginx.passenger" ${S_DIR}/top.sls || echo " - nginx.passenger" >> ${S_DIR}/top.sls
# Currently, only available on config_examples/multi_host/aws
if [ "x${USE_LETSENCRYPT}" = "xyes" ]; then
- if [ "x${USE_LETSENCRYPT_IAM_USER}" = "xyes" ]; then
+ if [ "x${USE_LETSENCRYPT_IAM_USER}" != "xyes" ]; then
grep -q "aws_credentials" ${S_DIR}/top.sls || echo " - aws_credentials" >> ${S_DIR}/top.sls
fi
grep -q "letsencrypt" ${S_DIR}/top.sls || echo " - letsencrypt" >> ${S_DIR}/top.sls
+ else
+ # Use custom certs, special case for keepweb
+ if [ ${R} = "keepweb" ]; then
+ cp -v ${CUSTOM_CERTS_DIR}/download.* "${F_DIR}/extra/extra/files/"
+ cp -v ${CUSTOM_CERTS_DIR}/collections.* "${F_DIR}/extra/extra/files/"
+ else
+ cp -v ${CUSTOM_CERTS_DIR}/${R}.* "${F_DIR}/extra/extra/files/"
+ fi
+ # We add the custom_certs state
+ grep -q "custom_certs" ${S_DIR}/top.sls || echo " - extra.custom_certs" >> ${S_DIR}/top.sls
+
fi
# webshell role is just a nginx vhost, so it has no state
if [ "${R}" != "webshell" ]; then
- grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
+ grep -q "arvados.${R}" ${S_DIR}/top.sls || echo " - arvados.${R}" >> ${S_DIR}/top.sls
fi
# Pillars
grep -q "nginx_passenger" ${P_DIR}/top.sls || echo " - nginx_passenger" >> ${P_DIR}/top.sls
grep -q "nginx_${R}_configuration" ${P_DIR}/top.sls || echo " - nginx_${R}_configuration" >> ${P_DIR}/top.sls
+ # Special case for keepweb
+ if [ ${R} = "keepweb" ]; then
+ grep -q "nginx_download_configuration" ${P_DIR}/top.sls || echo " - nginx_download_configuration" >> ${P_DIR}/top.sls
+ grep -q "nginx_collections_configuration" ${P_DIR}/top.sls || echo " - nginx_collections_configuration" >> ${P_DIR}/top.sls
+ fi
+
# Currently, only available on config_examples/multi_host/aws
if [ "x${USE_LETSENCRYPT}" = "xyes" ]; then
- if [ "x${USE_LETSENCRYPT_IAM_USER}" = "xyes" ]; then
+ if [ "x${USE_LETSENCRYPT_IAM_USER}" != "xyes" ]; then
grep -q "aws_credentials" ${P_DIR}/top.sls || echo " - aws_credentials" >> ${P_DIR}/top.sls
fi
grep -q "letsencrypt" ${P_DIR}/top.sls || echo " - letsencrypt" >> ${P_DIR}/top.sls
grep -q "letsencrypt_${R}_configuration" ${P_DIR}/top.sls || echo " - letsencrypt_${R}_configuration" >> ${P_DIR}/top.sls
+
+ # As the pillar differ whether we use LE or custom certs, we need to do a final edition on them
+ # Special case for keepweb
+ if [ ${R} = "keepweb" ]; then
+ for kwsub in download collections; do
+ sed -i "s/__CERT_REQUIRES__/cmd: create-initial-cert-${kwsub}.${CLUSTER}.${DOMAIN}*/g;
+ s#__CERT_PEM__#/etc/letsencrypt/live/${kwsub}.${CLUSTER}.${DOMAIN}/fullchain.pem#g;
+ s#__CERT_KEY__#/etc/letsencrypt/live/${kwsub}.${CLUSTER}.${DOMAIN}/privkey.pem#g" \
+ ${P_DIR}/nginx_${kwsub}_configuration.sls
+ done
+ else
+ sed -i "s/__CERT_REQUIRES__/cmd: create-initial-cert-${R}.${CLUSTER}.${DOMAIN}*/g;
+ s#__CERT_PEM__#/etc/letsencrypt/live/${R}.${CLUSTER}.${DOMAIN}/fullchain.pem#g;
+ s#__CERT_KEY__#/etc/letsencrypt/live/${R}.${CLUSTER}.${DOMAIN}/privkey.pem#g" \
+ ${P_DIR}/nginx_${R}_configuration.sls
+ fi
+ else
+ grep -q ${R} ${P_DIR}/extra_custom_certs.sls || echo " - ${R}" >> ${P_DIR}/extra_custom_certs.sls
+
+ # As the pillar differ whether we use LE or custom certs, we need to do a final edition on them
+ # Special case for keepweb
+ if [ ${R} = "keepweb" ]; then
+ for kwsub in download collections; do
+ sed -i "s/__CERT_REQUIRES__/file: extra_custom_certs_file_copy_arvados-${kwsub}.pem/g;
+ s#__CERT_PEM__#/etc/nginx/ssl/arvados-${kwsub}.pem#g;
+ s#__CERT_KEY__#/etc/nginx/ssl/arvados-${kwsub}.key#g" \
+ ${P_DIR}/nginx_${kwsub}_configuration.sls
+ done
+ else
+ sed -i "s/__CERT_REQUIRES__/file: extra_custom_certs_file_copy_arvados-${R}.pem/g;
+ s#__CERT_PEM__#/etc/nginx/ssl/arvados-${R}.pem#g;
+ s#__CERT_KEY__#/etc/nginx/ssl/arvados-${R}.key#g" \
+ ${P_DIR}/nginx_${R}_configuration.sls
+ fi
fi
;;
"shell")
# END FIXME! #16992 Temporary fix for psql call in arvados-api-server
# Leave a copy of the Arvados CA so the user can copy it where it's required
-echo "Copying the Arvados CA certificate to the installer dir, so you can import it"
-# If running in a vagrant VM, also add default user to docker group
-if [ "x${VAGRANT}" = "xyes" ]; then
- cp /etc/ssl/certs/arvados-snakeoil-ca.pem /vagrant/${CLUSTER}.${DOMAIN}-arvados-snakeoil-ca.pem
-
- echo "Adding the vagrant user to the docker group"
- usermod -a -G docker vagrant
-else
- cp /etc/ssl/certs/arvados-snakeoil-ca.pem ${SCRIPT_DIR}/${CLUSTER}.${DOMAIN}-arvados-snakeoil-ca.pem
+if [ "$DEV_MODE" = "yes" ]; then
+ echo "Copying the Arvados CA certificate to the installer dir, so you can import it"
+ # If running in a vagrant VM, also add default user to docker group
+ if [ "x${VAGRANT}" = "xyes" ]; then
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem /vagrant/${CLUSTER}.${DOMAIN}-arvados-snakeoil-ca.pem
+
+ echo "Adding the vagrant user to the docker group"
+ usermod -a -G docker vagrant
+ else
+ cp /etc/ssl/certs/arvados-snakeoil-ca.pem ${SCRIPT_DIR}/${CLUSTER}.${DOMAIN}-arvados-snakeoil-ca.pem
+ fi
fi
# Test that the installation finished correctly
projects / arvados.git / commitdiff